Underscore "_" is a wild card match character in MSSQL

Besides '%', underscore '_' is also a wild card match character in MSSQL. This can lead to some very subtle bug.

Say your table name has underscore in it, e.g., there are 2 tables my_Tablename and my_WeirdTablename. You find table using 'name like %_Tablename', then you will get both tables, instead of the first one. This leads to hidden bugs that are hard to identify.

To fix this issue, use this: 'name like %[_]tablename'. The '_' char is escaped by quoting it with brackets.

Linux - change hostname

http://www.ducea.com/2006/08/07/how-to-change-the-hostname-of-a-linux-system/
Linux Hostname Configuration

Trac wiki formatting

http://trac.edgewall.org/wiki/WikiFormatting
https://trac.tools.ietf.org/wg/enum/trac/wiki/WikiMacros

Testing

Baidu Baike: A/B测试
Wiki: A/B Testing
The Complete Guide to A/B Testing

A List Of Best System Monitoring Applications (GUI & CLI) For Ubuntu 12.04/Linux Mint 13

A List Of Best System Monitoring Applications (GUI & CLI) For Ubuntu 12.04/Linux Mint 13

Find linux distro

You can use any one of the following method to find out your Linux distribution and name:

a) /etc/*-release file. (for distro, may include multiple release files under /etc/)
b) /proc/version file. (for linux version)
c) lsb_release command. (for hardware version)

See http://www.cyberciti.biz/faq/find-linux-distribution-name-version-number/

Cloudera

Try out Cloudera product.

Download quickstart VM for CDH 5.4.x from:
http://www.cloudera.com/content/cloudera/en/downloads.html

Get history report of the CPU/Memory usage on windows

How to get history report of the CPU/Memory usage on windows 2008 R2 (64-bit)

Below are from the above link:

Perfmon in server 2008 really is your best option.

We will use system memory as an example but you can add the processor counters just as easily.

Start>run>perfmon

In the left, expand "Monitoring Tools". Now right click "Performance Monitor" and choose, "New">"Data Collector Set".

Give it a name of "Memory" and click "Next"

Now browse to where you want to save the logged data, lets make it easy and drop it into the root of " and then click "Next"

Choose save and close on the next screen and click "Finish"

Now in the left hand side, expand "Data collector sets" then "User Defined" and click on your new "Memory" one we created.

In the right hand screen double click the "System monitor log"

In this screen, click the "Processor" that's already listed and then click the "Remove" button.

Now click the "Add" button, scroll through the list and click the plus symbol next to "Memory". Click on any one of the things in the list to deselect all of them. We are going to individually select jsut the ones that interest us and judging by what you want to see I would hold down the control key and specifically select,

% Committed Bytes in use

Available MBytes

Pages Input/sec

Pages Output/sec

We can start with these and add more if needed later, click the add button to add them to the right hand side.

Now click "OK" then "Apply" and then "OK" again.

Now on the left hand side in the tree, right click your user defined "Memory" and choose "Start".

On the left, expand "Reports">"User Defined">"Memory">"System Monitor Log.blg"

Clicking on that last one should show you your current monitor is running in the right hand side. If we go back to our data collector set on the left right click and say stop, then click again on the system monitor log, we can now see a graph of what is being collected. This same data is also being saved to a log file in the C root which when we open it shows something similar.

In the steps above where we chose what to see, had we left everything selected for memory, it would have been around 30 things it would graph out making it difficult to read.

From here, run it for a week and check your results. It should give you a nice graphical baseline.

Good luck and let me know if you need any further assistance.

-Jay

---

How to interpret metrics collected by Perfmon.

  - Evaluating Memory and Cache Usage
  - http://www.sqlshack.com/sql-server-memory-performance-metrics-part-1-memory-pagessec-memory-page-faultssec/

For memory usage, the most important counters are pages/sec and page_faults/sec.

pages/sec is the sum of input/sec and output/sec. The normal range is 40-150/sec.

page faults/sec can include both hard and soft page faults. The normal range is 10-15/sec. But it can be as high as 1000/sec sometimes.

To decide if there is a short of memory is more complex than just looking at these numbers. You also should take into consideration the reads/sec etc.

Adobe Flash Player Settings Manager

Adobe Flash Player Settings Manager:

http://www.macromedia.com/support/documentation/en/flashplayer/help/settings_manager09.html

This works across different browsers.

Share a folder between windows and linux

To share a linux folder with windows, just use Samba.

To share a windows folder with linux, you can use cifs-utils. Details are below:

    1) Create windows shared folder [2] on windows machine (say MachineX): C:\Shared\
       - now can access \\MachineX\Shared\ on other machines on the same domain.
    2) Download/Install cifs-utils on linux machine:
       - download from [3].
       - unzip and install:
         bzip2 -d cifs-utils-6.4.tar.bz2
         tar xvf cifs-utils-6.4.tar
         cd cifs-utils-6.4
         ./configure --prefix=/usr \
            --disable-pam \
            --disable-systemd
         make
         sudo make install
       - finally, the command is available on linux by: sudo /sbin/mount.cifs
    3) Mount the shared folder on linux:
       - mount:
         cd /mnt
         sudo mkdir MachineX_Shared
         sudo /sbin/mount.cifs  //[MachineX IP address]/Shared  /mnt/MachineX_Shared -o user=[username]
       - now can access the folder and write to it.

References:

[1] http://www.howtogeek.com/176471/how-to-share-files-between-windows-and-linux/
[2] http://lifehacker.com/288033/mount-a-windows-shared-folder-in-linux
[3] http://www.linuxfromscratch.org/blfs/view/svn/basicnet/cifsutils.html

Use WordPress as CMS

WordPress is a popular blogging software. It has a lot of nice-looking themes. It will be very nice if it can be used as the framework of a website.

For this purpose, one should be able to create custom roles (user groups), members of such roles may or may not be able to post blogs, but can access custom pages once log in. Several things are crucial:

1) create custom role(s).
2) check if a user is logged in, and access user profile such as ID, login name, user name, email, user type. Based on this, one can add custom database tables and build whatever function that is desired.
3) make use of WordPress theme in custom page.

A study on these requirements proves fruitful. Over the weekend, I was able to build a member page for 4 WordPress themes. It's easy to extend to more themes.

The files are prepared and will be uploaded to github. See:

https://github.com/chenx/WordPress_Extension


Summary:

• Shows how to check if a user is logged in, and retrieval user information.
• Shows how to include header/footer of current theme.
• To add custom links, go to Appearance -> Menus -> Custom Links
• To allow user register, see [3] below.
  It seems Wordpress allows each user to have one role only. That's less flexible than .NET membership, which allows each user to have any roles.
• To change default user registeration email, see [4].
• To allow manage user roles, use the Members plugin [5].
• To disable comments/discussion on a per-page basis:
  On Edit Page, click Screen Options and check the Discussion box. Then in Discussion section, uncheck "Allow comments".
• To add a forum, see .
• Note: installed themes are in wp-content/themes/
• Note: to imitate other page's layout, see theme's page.php.
• Note: to customize page title, see wp-includes/post-template.php function get_the_title().

References:

1. http://codex.wordpress.org/Function_Reference/wp_get_current_user
2. https://web-design-weekly.com/snippets/load-a-different-header-in-wordpress/
3. http://www.wpbeginner.com/beginners-guide/how-to-allow-user-registration-on-your-wordpress-site/
4. https://wordpress.org/plugins/welcome-email-editor/
5. https://wordpress.org/plugins/members/
6. wordpress - redirect to target page after log in 

HTML 5移动开发从入门到精通

HTML 5移动开发从入门到精通

This introduces some HTML5 elements.

Javascript elements:

1. selectors api
2. JSON parse/stringify
3. Hashchage event
4. async.defer for script
5. progress for ajax
6. DeviceOrientation
7. touch events

Html5 Tag elements:

1. progress: <progress max="100" value="30"></progress>
2. mark (highlight text): <mark>This is a mark</mark>
3. address (italic format)
4. Colgoup col
5. keygen
6. fieldset/legend
7. ol.li (reversed, start, type)
8. q (quotation)
9. contenteditable div

Image hover effects & Customize css for different devices

I. Simple but effective and look good:

[1] http://designshack.net/articles/css/joshuajohnson-2/
[2] http://designshack.net/tutorialexamples/imagehovers/index.html


II. Customize css for different devices. Example:

/* >= 768. Desktop. */
@media only screen and (min-width:768px){
    #slide_img1, #slide_img2, #slide_img3
    {
        display: block; max-width: 100%; height:auto;
    }
}

/* < 768px. Mobile. */
@media only screen and (max-width:768px){
    #slide_img1, #slide_img2, #slide_img3
    {
        display: block; max-width: 800px; height: auto;
    }
}

[3] http://stackoverflow.com/questions/22839792/bootstrap-3-custom-css-class-depending-of-devices

Normalize.css, an alternative to CSS resets

From [1]:

Normalize.css is a small CSS file that provides better cross-browser consistency in the default styling of HTML elements. It’s a modern, HTML5-ready, alternative to the traditional CSS reset.

From [7]:

A CSS Reset (or “Reset CSS”) is a short, often compressed (minified) set of CSS rules that resets the styling of all HTML elements to a consistent baseline.

In case you didn’t know, every browser has its own default ‘user agent’ stylesheet, that it uses to make unstyled websites appear more legible. For example, most browsers by default make links blue and visited links purple, give tables a certain amount of border and padding, apply variable font-sizes to H1, H2, H3 etc. and a certain amount of padding to almost everything. Ever wondered why Submit buttons look different in every browser?

Obviously this creates a certain amount of headaches for CSS authors, who can’t work out how to make their websites look the same in every browser. (NB: article coming soon about why this is a false notion!)

Using a CSS Reset, CSS authors can force every browser to have all its styles reset to null, thus avoiding cross-browser differences as much as possible.


[1] About normalize.css
[2] https://github.com/necolas/normalize.css/
[3] https://necolas.github.io/normalize.css/
[4] Applying Normalize.css Browser Reset CSS (video)

[5] CSS Tools: Reset CSS
[6] http://cssreset.com 2015’s most popular CSS Reset scripts, all in one place.
[7] What Is A CSS Reset?
[8] http://stackoverflow.com/questions/6887336/what-is-the-difference-between-normalize-css-and-reset-css

Paging with SQL

In MySQL, this can be easily done with:

SELECT * FROM tbl limit [start], [end]

In MSSQL, this can be done as in [1] :

1.Paging rows with Limit (MSSQL 2005 or later)

--VIEWING THE PAGE "2" WITH 5 ROWS
DECLARE @PageNumber AS INT, @RowspPage AS INT
SET @PageNumber = 2
SET @RowspPage = 5 SELECT * FROM (
             SELECT ROW_NUMBER() OVER(ORDER BY ID_EXAMPLE) AS NUMBER,
                    ID_EXAMPLE, NM_EXAMPLE, DT_CREATE FROM TB_EXAMPLE
               ) AS TBL
WHERE NUMBER BETWEEN ((@PageNumber - 1) * @RowspPage + 1) AND (@PageNumber * @RowspPage)
ORDER BY ID_EXAMPLE

2. Paging in SQL Server 2012, with FETCH/OFFSET

--CREATING A PAGING WITH OFFSET and FETCH clauses IN "SQL SERVER 2012"
DECLARE @PageNumber AS INT, @RowspPage AS INT
SET @PageNumber = 2
SET @RowspPage = 10 
SELECT ID_EXAMPLE, NM_EXAMPLE, DT_CREATE
FROM TB_EXAMPLE
ORDER BY ID_EXAMPLE
OFFSET ((@PageNumber - 1) * @RowspPage) ROWS
FETCH NEXT @RowspPage ROWS ONLY;

For performance, 2 is better than 1. Both are better than getting all rows and use specific range.

[1] http://social.technet.microsoft.com/wiki/contents/articles/23811.paging-a-query-with-sql-server.aspx

A little Software Engineering - Collect LOC metrics

To manage one's project, one thing is to collect metrics.

To count line of code in all project files, the CLOC [1] command line utility is easy to use.

For example, you can setup a batch file on windows for this purpose. On windows, you can put the cloc.exe executable in C:\windows\ so it's available anywhere in DOS.  Next, in general you need 2 files: 1) a exclude-list file, 2) a batch file to call the CLOC.exe utility.


-- cloc_exclude_list.txt --

c:\inetpub\wwwroot\my_site\backup
c:\inetpub\wwwroot\my_site\images

(Note you cannot put a back slash after the folder name.)


-- run_cloc.bat --

@echo off
For /f "tokens=2-4 delims=/ " %%a in ('date /t') do (set mydate=%%c-%%a-%%b)
echo Today is: %mydate%
echo.

cloc c:\inetpub\wwwroot\my_site --exclude-list-file=cloc_exclude_list.txt -out=cloc_log\short_%mydate%.txt
cloc c:\inetpub\wwwroot\my_site --exclude-list-file=cloc_exclude_list.txt -out=cloc_log\short_%mydate%.csv --csv

cloc c:\inetpub\wwwroot\my_site --exclude-list-file=cloc_exclude_list.txt --by-file -out=cloc_log\long_%mydate%.txt
cloc c:\inetpub\wwwroot\my_site --exclude-list-file=cloc_exclude_list.txt --by-file -out=cloc_log\long_%mydate%.csv --csv

(This will create 4 output files, for the combination of full/short list and txt/csv format,
prefixed by today's date. So you could setup a schedule task for this to run automatically everyday.)


[1] http://sourceforge.net/projects/cloc/
[2] http://stackoverflow.com/questions/203090/how-to-get-current-datetime-on-windows-command-line-in-a-suitable-format-for-us

Bootstrap

== Summary of today's review of Bootstrap ==

Bootstrap is the most popular HTML, CSS, and JavaScript framework for developing responsive, mobile-first web sites. Bootstrap is completely free to download and use!

• Download from: http://getbootstrap.com
• Documentation on using JavaScript, CSS and Components are also on the above site.
• Folder demo/ contains some nice examples of website templates. Good for quick start.
  • demo/assets/js/docs.min.js creates placeholder image. See examples/theme/ for examples.
  • In demo/examples/, for difference of static top and fixed top, you need to view in mobile phone mode.
  • Good ones for mobile: Carousel, cover, Dashboard, Grid, jumbotron-narrow, jumbotron, navbar-fixed-top, navbar-static-top, navbar, signin, sticky-footer-navbar, sticky-footer, theme.
• Folder tests/ contains some tests of bootstrap.
• Folder backup/ contains the downloaded boostrap zip distribution package.
• Folder download/ contains the downloaded entire website of getbootstrap.com.

References:

1. http://getbootstrap.com
2. http://www.w3schools.com/bootstrap/

Upload timeout issue

This is a repeated scenario.

Now I'm trying to be as thorough as possible in fixing this.

In PHP, this can be related to the following parameters in php.ini (often /etc/php.ini):

max_execution_time = 60  ; seconds
max_input_time = 60         ; seconds
memory_limit = 512M        ; 0.5 GB
post_max_size = 1000M    ; 1 GB
upload_max_filesize = 1000M   ; 1 GB

If the upload is done through a SOAP client [1], then this parameter in php.ini also should be updated:

default_socket_timeout = 60 ; seconds

After this, apache web server needs restart. (e.g., on redhat, sudo /sbin/service httpd restart)


On windows, ASP.NET application, these parameters in web.config needs update [2]:

In system.web (maxRequestLength is in KB):
<httpRuntime maxRequestLength="1048576" executionTimeout="3600" />

And in system.webServer (maxAllowedContentLength is in bytes) (this is needed in IIS7 and later): 
<system.webServer>
   <security>
      <requestFiltering>
         <requestLimits maxAllowedContentLength="1073741824" />
      </requestFiltering>
   </security>
</system.webServer>


References:
[1] http://stackoverflow.com/questions/9403486/error-fetching-http-headers-in-soapclient
[2] http://stackoverflow.com/questions/3853767/maximum-request-length-exceeded

Windows Domain Controller

On domain controller, show users from console:

dsquery user | sort

Show computers:

dsquery computer

Show help by typing "dsquery":

Description: This tool's commands suite allow you to query the directory
according to specified criteria. Each of the following dsquery commands finds
objects of a specific object type, with the exception of dsquery *, which can
query for any type of object:

dsquery computer - finds computers in the directory.
dsquery contact - finds contacts in the directory.
dsquery subnet - finds subnets in the directory.
dsquery group - finds groups in the directory.
dsquery ou - finds organizational units in the directory.
dsquery site - finds sites in the directory.
dsquery server - finds AD DCs/LDS instances in the directory.
dsquery user - finds users in the directory.
dsquery quota - finds quota specifications in the directory.
dsquery partition - finds partitions in the directory.
dsquery * - finds any object in the directory by using a generic LDAP query.

For help on a specific command, type "dsquery <ObjectType> /?" where
<ObjectType> is one of the supported object types shown above.
For example, dsquery ou /?.

Remarks:
The dsquery commands help you find objects in the directory that match
a specified search criterion: the input to dsquery is a search criterion
and the output is a list of objects matching the search. To get the
properties of a specific object, use the dsget commands (dsget /?).

The results from a dsquery command can be piped as input to one of the other
directory service command-line tools, such as dsmod, dsget, dsrm or dsmove.

Commas that are not used as separators in distinguished names must be
escaped with the backslash ("\") character
(for example, "CN=Company\, Inc.,CN=Users,DC=microsoft,DC=com").

Backslashes used in distinguished names must be escaped with a backslash
(for example,
"CN=Sales\\ Latin America,OU=Distribution Lists,DC=microsoft,DC=com").

Examples:
To find all computers that have been inactive for the last four weeks and
remove them from the directory:

        dsquery computer -inactive 4 | dsrm

To find all users in the organizational unit
"ou=Marketing,dc=microsoft,dc=com" and add them to the Marketing Staff group:

        dsquery user ou=Marketing,dc=microsoft,dc=com | dsmod group
        "cn=Marketing Staff,ou=Marketing,dc=microsoft,dc=com" -addmbr

To find all users with names starting with "John" and display his office
number:

        dsquery user -name John* | dsget user -office

To display an arbitrary set of attributes of any given object in the
directory use the dsquery * command. For example, to display the
sAMAccountName, userPrincipalName and department attributes of the object
whose DN is ou=Test,dc=microsoft,dc=com:

        dsquery * ou=Test,dc=microsoft,dc=com -scope base
        -attr sAMAccountName userPrincipalName department

To read all attributes of the object whose DN is ou=Test,dc=microsoft,dc=com:

        dsquery * ou=Test,dc=microsoft,dc=com -scope base -attr *

Directory Service command-line tools help:
dsadd /? - help for adding objects.
dsget /? - help for displaying objects.
dsmod /? - help for modifying objects.
dsmove /? - help for moving objects.
dsquery /? - help for finding objects matching search criteria.
dsrm /? - help for deleting objects.

Webcam setup is easy

It seems setting up a real time online webcam is easy.

An example: http://foscam.us/

The webcam will be IP based, so user can log in and watch anytime online. There is IR mode (Infrared), so you can watch during night. You can control orientation and angle of the camera.

Other choices are like BlueIris and Webcam Surveyor.

Free Git repository

Assembla no longer has free repository. Bitbucket.org allows free repository of team up to 5 members.

Note:
- Git ignore file list: http://git-scm.com/docs/gitignore

Batch script to copy files

The batch script below can be used to automate file copy. You need to modify it to adapt to your needs.

echo off
echo Copy script

REM Source machines.
set SERVERS[0]=machine1
set SERVERS[1]=machine2
set SERVERS[2]=machine3

REM Source folder pattern.
set PATTERN=Folder_Pattern
 
REM Target location root.
set ROOT=machine_target_root
set VERBOSE=1

set "x=0"

:SymLoop

if defined SERVERS[%x%] (
  REM call echo %%SERVERS[%x%]%%
  call set SERVER=%%SERVERS[%x%]%%
  echo %SERVER%

  REM Create target folder.
  mkdir %ROOT%\%SERVER%

  FOR /f "tokens=*" %%i in ('DIR /a:d /b \\%SERVER%\%PATTERN%') DO (
      if VERBOSE=1 (
          ECHO Copy from \\%SERVER%\%%i\* to %ROOT%\%SERVER%\%%i\
      )
      xcopy \\%SERVER%\%%i\* %ROOT%\%SERVER%\%%i\
  )

  set /a "x+=1"
  GOTO :SymLoop
)

:END

Use Java library in C# - IKVM.NET

[1] IKVM.NET
[2] http://en.wikipedia.org/wiki/IKVM.NET

From [2]:

IKVM.NET is an implementation of Java for Mono and the Microsoft .NET Framework. IKVM is free software, distributed under a permissive free software licence.[1]

IKVM.NET includes the following components:

    A Java Virtual Machine implemented in .NET
    A .NET implementation of the Java class libraries
    A tool that translates Java bytecode (JAR files) to .NET IL (DLLs or EXE files).
    Tools that enable Java and .NET interoperability

With IKVM.NET you can run compiled Java code (bytecode) directly on Microsoft .NET or Mono. The bytecode is converted on the fly to CIL and executed.

Therefore if you want to use a java Jar library as C# assembly, you can use IKVM for the conversion.

This is great.

Principle of image search

Principle of image search:

http://www.ruanyifeng.com/blog/2011/07/principle_of_similar_image_search.html
http://www.ruanyifeng.com/blog/2013/03/similar_image_search_part_ii.html

From blog of Ruan YiFeng.

Emscripten: An LLVM-to-JavaScript Compiler

This is very impressive.


Emscripten

Emscripten: An LLVM-to-JavaScript Compiler
Project main site: http://kripken.github.io/emscripten-site/
Wiki site: https://github.com/kripken/emscripten/wiki
Github project: https://github.com/kripken/emscripten


asm.js

An extraordinarily optimizable, low-level subset of JavaScript.
http://asmjs.org/

Fun programming - graphics and animation

http://funprogramming.org/

short programs for graphics and animation.

Standard and nested/hybrid RAID levels

[1] http://en.wikipedia.org/wiki/Standard_RAID_levels

It's good to know about RAID levels 1, 2, 3, 4, 5, 6.

[2] http://en.wikipedia.org/wiki/Nested_RAID_levels

For nested RAID levels (hybrid RAID), which combines 2 or more standard RAID levels, there are:
01, 03, 10, 50, 60, 100.

In [2], it says:

According to manufacturer specifications and official independent benchmarks,^[8][9][10] in most cases RAID 10 provides better throughput and latency than all other RAID levels except RAID 0 (which wins in throughput). Thus, it is the preferable RAID level for I/O-intensive applications such as database, email, and web servers, as well as for any other use requiring high disk performance.^[11]

So RAID 10 seems a widely used configuration.

HTML5 - Video and Canvas

http://html5doctor.com/video-canvas-magic/

HTML5

W3C released HTML5 standard in 10/29/2014 after 8 years of effort.

Html5 provides lots of new features that make it easier to create web applications with rich features.

Notable new features include support for multimedia (canvas, audio, video, svg), real time communication (websocket), local storage, css3, and many new semantic tags.

[1] Baidu Baike: Html5
[2] HTML5 in China大会综述 2011
[3] wiki: Html5
[4] HTML5梦工场

Web Audio

Good source of web audio features

http://labs.music.baidu.com/demo/audiolab/?p=0
http://labs.music.baidu.com/demo/interactions/visualization/
Getting Started with Web Audio API
Web Audio API - High-level JavaScript API for processing and synthesizing audio
http://www.sitepoint.com/5-libraries-html5-audio-api/
http://codegeekz.com/10-javascript-audio-libraries-for-developers/
http://www.smartjava.org/content/exploring-html5-web-audio-visualizing-sound
http://ianreah.com/2013/02/28/Real-time-analysis-of-streaming-audio-data-with-Web-Audio-API.html
https://blog.groupbuddies.com/posts/39-tutorial-html-audio-capture-streaming-to-node-js-no-browser-extensions

https://nusofthq.com/blog/recording-mp3-using-only-html5-and-javascript-recordmp3-js/
https://github.com/icatcher-at/MP3RecorderJS
https://github.com/akrennmair/libmp3lame-js

http://www.patrick-wied.at/blog/how-to-create-audio-visualizations-with-javascript-html
http://www.schillmania.com/projects/soundmanager2/demo/360-player/canvas-visualization.html
http://www.sitepoint.com/3-breakthrough-ways-to-visualize-html5-audio/

Audio: visualization
http://readwrite.com/2008/03/13/the_best_tools_for_visualization


Programming with sound - Introduction

http://blogs.msdn.com/b/dawate/archive/2009/06/22/intro-to-audio-programming-part-1-how-audio-data-is-represented.aspx

Digital audio:
- stereo: 2 channels, for left/right ear
- mono: 1 channel only.
- sampling rate: ~ 44,100/sec, 44.1k Hz. Number of points/second. Other rate: 8/48/96/192K Hz.
- fundamental frequency: Number of oscillations of the wave per sec. 440 Hz - Concert A. 261.25 Hz - Concert C. Higher frequency == higher pitch.
- decibel: amplitude of the wave
- bit depth/bits per sample: 8-bit/16-bit waveform - use 8/16 samples for a waveform. The larger, the better quality.
- multiple tones: sound in real world is overlap of multiple sine waves/tones.

http://blogs.msdn.com/b/dawate/archive/2009/06/23/intro-to-audio-programming-part-2-demystifying-the-wav-format.aspx

- wave format: maybe the most basic sound format. Developed by Microsoft and IBM.
- uncompressed. but cannot go over 4GB, since file size header field is 32-bit unsigned int.
- chunks
- header
- format chunk
- data chunk
- making a wave file.

http://blogs.msdn.com/b/dawate/archive/2009/06/24/intro-to-audio-programming-part-3-synthesizing-simple-wave-audio-using-c.aspx

http://blogs.msdn.com/b/dawate/archive/2009/06/25/intro-to-audio-programming-part-4-algorithms-for-different-sound-waves-in-c.aspx


Pure Data

http://puredata.info/

Pure Data (aka Pd) is an open source visual programming language. Pd enables musicians, visual artists, performers, researchers, and developers to create software graphically, without writing lines of code. Pd is used to process and generate sound, video, 2D/3D graphics, and interface sensors, input devices, and MIDI. Pd can easily work over local and remote networks to integrate wearable technology, motor systems, lighting rigs, and other equipment. Pd is suitable for learning basic multimedia processing and visual programming methods as well as for realizing complex systems for large-scale projects.

Pd is free software and can be downloaded in different versions.

http://en.wikipedia.org/wiki/List_of_audio_programming_languages


Programming with sound - Overview

http://forum.devmaster.net/t/what-is-sound-programming/20773


Unlike most other things, it's real-time stuff (we're talking 0.02ms stuff here, as oposed to video's 17ms; and nobody cares if you miss a frame here or there, but audio must be flawless), and you have to deal with latencies and all sort of fun stuff like that.


Again, it looks simple and the concepts are pretty easy, but for some reason it always ends up being tricky. Thus, most people end up using libraries like FMOD, and *still* end up having issues here and there.

Also, if you want to synthesize audio, there is indeed a whole lot of interesting mathematics and theory you won't see much in other places.


I like to lump "sound programming" into 3 broad (sometimes overlapping) categories.

1) Sound Implementation programmer. This sounds like what you probalby think of. That's the programmer (often an intern ;0) whose job is to sprinkle the code with calls like "PlaySound(SNDTag, Parameters); through the game. (it can be quite a bit more than that, of course). Sound Implementation is usually, but not always, pretty straightforward

2) Sound Engine/Tool programmer. These programmers write the high-level sound engines that perform things like real-time streaming, data file parsing, etc. That requires pretty good knowledge of real-time and system programming. They may need to write entire (friendly and robust) User interfaces that a sound designer or composer will use when creating content. They are the essential link between content and code in content-driven audio development systems.

3) low-level audio signal processing programmer. These programmers typically need to know assembly language like the back of their hands as well as the complex mathematics behind the processing of audio. FFTs, DFT's, MLTs, IIR, FIR filters, data compression/decompression algorithms. That's pretty heavy stuff. And there's a ton of stuff going on out there. People do entire PhD dissertations on things like physical modelling of sound, real-time analysis of music, or environmental modeling..

So there's a lot of "meaty" stuff in game audio programming-- way more so than it may seem on the surface. Video game music and sound design still need a lot of specialized code to make it work
Brian Schmidt
Executive Director
www.GameSoundCon.com5


http://stackoverflow.com/questions/4801690/i-want-to-learn-audio-programming

- analogue sound: magnetic tape.. complex numbers, fourier transformation.
- sampled sound: CD, mp3, Ogg vorbis.. DSP.
- synthesized sound: sound + instrument type.

A programmer's guide to sound (1997)

Create daemon in Java

This can be a windows service or a unix daemon. The implementation might be different depending on platform. So I'm starting to do some research.


1. Apache Commons Daemon [1]

Apache Commons Daemon is a good alternative. It has Procrun [2] for windows services, and Jsvc [3] for unix daemons. It uses less restrictive Apache license, and Apache Tomcat uses it as a part of itself to run on Windows and Linux! To get it work is a bit tricky, but there is an exhaustive article [4] with working example. With Apache Commons Daemon you can now have a custom executable name and icon! You can also get a custom Windows tray monitor with your own name and icon!

Jsvc is a set of libraries and applications for making Java applications run on UNIX more easily.
Jsvc allows the application (e.g. Tomcat) to perform some privileged operations as root (e.g. bind to a port < 1024), and then switch identity to a non-privileged user.It can run on Win32 via the Cygwin emulation layer, however Win32 users may prefer to use procrun instead.

Procrun is a set of applications that allow Windows users to wrap (mostly) Java applications (e.g. Tomcat) as a Windows service. The service can be set to automatically start when the machine boots and will continue to run with no user logged onto the machine.


2. Java Service Wrapper (JSW) [download]

See short introduction in [5]. Also an implementation of it: YAJSW - Yet Another Java Service Wrapper [6]. Cons (?): community version cannot be run on a server.


References:

[1] http://commons.apache.org/daemon/
[2] http://commons.apache.org/daemon/procrun.html
[3] http://commons.apache.org/daemon/jsvc.html
[4] Java as Windows Service with Apache Commons Daemon
[5] Running Java Applications as a Windows Service
[6] YAJSW - Yet Another Java Service Wrapper 
[7] http://stackoverflow.com/questions/68113/how-to-create-a-windows-service-from-java-app
[8] Creating a Java Daemon (System Service) for Debian using Apache Commons Jsvc

Greasemonkey

Greasemonkey can do some interesting tricks on a webpage.

Greasemonkey scripts (Javascript) can alter web page elements, add a div or box to the page, trigger button click events and others.

For example, here is a script that adds a checkbox before the link tag for ever <li> element:


// ==UserScript==
// @name       script name
// @namespace  http://my.homepage/
// @version    0.1
// @description  add marking checkbox.
// @match      http://matched_website.com
// @match      https://matched_website.com
// ==/UserScript==

function changhandler(event) {
    if (event.target.checked) {
        localStorage[event.target.name] = true;
    } else {
        localStorage.removeItem(event.target.name);
    }
}

[].forEach.call(
    document.getElementById('main').getElementsByTagName('li'),
    function(e) {
        var checkBox = document.createElement('input');
        checkBox.type = 'checkbox';
        checkBox.name = e.getElementsByTagName('a')[0].innerHTML;
        checkBox.checked = localStorage[e.getElementsByTagName('a')[0].innerHTML];
        checkBox.addEventListener('change', changhandler, false);
        oj.insertBefore(checkBox, null);
    }
);

[1] http://en.wikipedia.org/wiki/Greasemonkey

Create animated and transparent GIF image

Gifsicle: Very useful tool to manipulate gif, and create animated gif:

http://www.lcdf.org/gifsicle/


== How to make an existing animated gif loop repeatedly ==

Solution involving the Terminal [1]:

1. Download the sources of Gifsicle
2. Unpack the sources and go to the unpacked directory
3. Do ./configure --disable-gifview --disable-gifdiff
4. Do make
5. Do cd src
6. And finally do ./gifsicle -bl /path/to/image.gif

Note: to install, after step 4, do: sudo make install

Solutions using GUI tools (I didn't test any them !):

1. You can try GifBuilder (requires Rosetta to work with Snow Leopard).
2. Pixen also seems to be a Gif Editor.

[1] http://superuser.com/questions/159212/how-do-i-make-an-existing-animated-gif-loop-repeatedly
[2] http://blog.iharder.net/2009/10/22/gif-create-animated-gifs-with-mac-os-x-preview-app/
[3] http://osxdaily.com/2013/01/27/make-a-transparent-image-png-or-gif-easily-with-preview-for-mac-os-x/
[4] http://osxdaily.com/2010/01/24/convert-images-in-mac-os-x-jpg-to-gif-psd-to-jpg-gif-to-jpg-bmp-to-jpg-png-to-pdf-and-more/ 

Compress javascript a.js into a.min.js

Compress javascript a.js into a.min.js can be done on this site: http://jscompress.com/

Perl - extend lib path

See: http://www.perlhowto.com/extending_the_library_path

This is useful when you want to use personal lib for perl modules, which is not on perl path (@INC) and cannot be found.

3 methods are mentioned.

1) The non-invasive method is to use -I switch:

perl -I /home/path/lib -I /usr/another/lib script.pl

2) add path to environment:

# unix, bourne shell
PERL5LIB=/home/path/lib:/usr/another/path/lib; export PERL5LIB

3) add path to perl code:

#!/usr/bin/perl
use lib "/home/path/lib";
use lib "/usr/another/lib";

use MyCustomModule;

General Game Playing

General Game Playing is a course given by Stanford on the application of AI in computer games.

The course material is available here: http://logic.stanford.edu/ggp/chapters/

Some readings today

Here are links to some readings today:

- Java equals()和hashCode()

   Q: 怎样实现正确的equals()方法
   A: 首先，我们需要遵守Java API文档中equals()方法的约定，如下：
    自反性：对于任何非空引用值 x，x.equals(x) 都应返回 true。
    对称性：对于任何非空引用值 x 和 y，当且仅当 y.equals(x) 返回 true 时，x.equals(y) 才应返回 true。
    传递性：对于任何非空引用值 x、y 和 z，如果 x.equals(y) 返回 true，并且 y.equals(z) 返回 true，那么 x.equals(z) 应返回 true。
    一致性：对于任何非空引用值 x 和 y，多次调用 x.equals(y) 始终返回 true 或始终返回 false，前提是对象上 equals 比较中所用的信息没有被修改。

    对于任何非空引用值 x，x.equals(null) 都应返回 false。
    其次，当我们重写equals()方法时 , 不同类型的属性比较方式不同,如下:
    属性是Object类型, 包括集合: 使用equals()方法。
    属性是类型安全的枚举: 使用equals()方法或==运算符(在这种情况下，它们是相同的)。
    属性是可能为空的Object类型: 使用==运算符和equals()方法。
    属性是数组类型: 使用Arrays.equals()方法。
    属性是除float和double之外的基本类型: 使用==运算符。
    属性是float: 使用Float.floatToIntBits方法转化成int,然后使用 ==运算符。
    属性是double: 使用Double.doubleToLongBits方法转化成long , 然后使用==运算符。

  Q: 重写equals()方法为什么一定要重写hashCode()方法
  A: 如果不这样做就会违反Java API中Object类的hashCode()方法的约定，从而导致该类无法很好的用于基于散列的数据结构(HashSet、HashMap、Hashtable、LinkedHashSet、LinkedHashMap等等)。i.e., when search in hashtable based data structure, it'll first find the bucket based on hashCode(), then do compare by equals() to find exact match. Object.hashCode() by default is different for every object. This can't be used when, say, 2 strings "aaa" and "aaa", are different objects so have different hashCode, then you cannot find "aaa" in a container that contains "aaa".


- 揭秘阿里服务互联网金融的关系数据库——OceanBase
  This lacks details though.

-  JS前台加密,java后台解密实现
   This is not hard to decode though since JS is open.

用 Python 绘制音乐图谱

用 Python 绘制音乐图谱
Original post is from http://www.christianpeccei.com/
And Articles by: PyPer

Very nice. Might worth the time to try to reproduce.

Play midi in browser by Javascript only

Midi files usually have very small size, but rich sound effects. HTML5 Audio tag supports wav, mp3 and ogg formats so far.  Here are methods to play midi music in web browser using Javascript only, with no plugin such as QuickTime.

I wrote the one at [7].

Features

• Can specify these in constructor parameter list: midi, target, loop, maxLoop, end_callback.
  - midi: MIDI file path.
  - target: Target html element that this MIDI player is attached to.
  - loop: Optinoal. Whether loop the play. Value is true/false, default is false.
  - maxLoop: Optional. max number of loops to play when loop is true. Negative or 0 means infinite. Default is 1.
  - end_callback: Optional. Callback function when MIDI ends.
    e.g., use this to reset target button value from "stop" back to "play".
• Can specify a debug div, to display debug message: setDebugDiv(debug_div_id).
• Start/stop MIDI by: start(), stop().
• If a MIDI started play, call start() again will stop and then restart from beginning.

This depends on other 5 javascript files (audio.js, midifile.js, replayer.js, stream.js, synth.js) from [2][3], which is a demo of [1]. This is related to [4], which is a powerful tool to play MIDI in browser.

The disadvantage of [2][3] is that it does not have control over how a MIDI file is played: when clicking on the link the file will be started multiple times and sounds chaotic; and there is no loop feature. Both are well handled by MidiPlayer.js here.

Another midi player javascript is in [5], but it cannot play multiple MIDI files at the same time, cannot play a MIDI file automatically after loading the page, and has no loop feature. All are handled by MidiPlayer.js here.

It can be a good idea to add MIDI support to HTML5 Audio tag, because MIDI files have much smaller size than wav/mp3, and the sound effects are very rich.

[1] http://matt.west.co.tt/music/jasmid-midi-synthesis-with-javascript-and-html5-audio/
[2] http://jsspeccy.zxdemo.org/jasmid/
[3] https://github.com/gasman/jasmid
[4] MIDI.js - Sequencing in Javascript.
[5] MIDI.js - The 100% JavaScript MIDI Player using W3C Web Audio
[6] Dynamically generating MIDI in JavaScript
[7]  The MidiPlayer javascript class

Javascript drag/drop event listener

It is a cool feature that file processing is triggered when you drag a file and drop it onto a web page.

For example, the midi player demo page index.html from [1] has this in header javascript:

    if (FileReader){
        function cancelEvent(e){
            e.stopPropagation();
            e.preventDefault();
        }
        document.addEventListener('dragenter', cancelEvent, false);
        document.addEventListener('dragover', cancelEvent, false);
        document.addEventListener('drop', function(e){
            cancelEvent(e);
            for(var i=0;i            var
                file = e.dataTransfer.files[i]
            ;
            if(file.type != 'audio/midi' && file.type != 'audio/mid'){
                continue;
            }
            var
                reader = new FileReader()
            ;
            reader.onload = function(e){
                midiFile = MidiFile(e.target.result);
                synth = Synth(44100);
                replayer = Replayer(midiFile, synth);
                audio = AudioPlayer(replayer);
            };
            reader.readAsBinaryString(file);
            }
        }, false);
    }

What this does is, when you drag and drop a midi file on the page, it'll be read and played.

HTML5 has native support for drag/drop event. See [2].

[1] https://github.com/gasman/jasmid
[2] Native HTML5 Drag and Drop

Javascript OOP

Javascript is a prototypical langaage. Its OOP is achieved by prototype instead of class. Prototype is kind of like decorator patten, that decorates a initially bare object with more functionalities.

Some links to Javascript OOP:

• [1] Introduction to Object-Oriented JavaScript
• [2] JavaScript Classical Inheritance
• [3] JavaScript Private Members
• [4] Class-Based vs. Prototype-Based Languages
• [5] 3 ways to define a JavaScript class
• [6] Javascript - create object - with some depth

The most common syntax for defining a class is [3]:

if (typeof (classA == "undefined")) {

    // Another short-hand way of defining this class:  function classA() { ... }
    var classA = function() {
        this.property1 = 1;  // A public variable. Defined with "this.". To access, use "this.".
        var secret = 2;  // A private variable. Defined with "var". To access, don't use "this.".

        // A private method.
        // Another short-hand way of defining this:  function dec () { ... }
        var dec = function() {
             if (secret > 0) {
                secret -= 1;
            }
        }
       
        // A privileged method. Can access private variables and methods. Accessible to outside.
        // Defined with "this.".
        // Another way of defining this:  function getInfo() { ... }
        this.getInfo = function() {
            return secret + this.property1;
        }     
    }

    // A public method.
    classA.prototype.getColor = function() { return 'green'; }
}

To create an instance you do:

var objectA = new classA();

To inherit from classA you do something like this [2]:

function classB() {
    this.setValue(value);
} 
classB.inherits(classA);

Promiscuous multiple inheritance is possible but hard and may suffer from name collision.

Other related concepts include swiss inheritance, parasitic inheritance etc.

NMap - network scanning tool

The Official Nmap Project Guide to Network Discovery and Security Scanning

Nmap参考指南(Man Page)

Shell scripts to start and stop a server

Now I have a working Python|Authbahn websocket server. To start and stop both there are several steps to do.  The scripts here help to simplify the job.


== start_server.sh ==

export PYTHONPATH=.
export DJANGO_SETTINGS_MODULE=my_project.settings
daemonize  -c /django_projects/my_project  /django_projects/my_project/server/my_server.py

#
# Note here there is a need to setup PYTHONPATH and django project setting, since we have
# customized modules to use in the server.
# After setting up the path, when use daemonize then you need to specify
# the working directory with -c.
#

== stop_server.sh ==

output=`ps ax|grep my_server.py | grep -v grep`
echo killing process my_server.py
if [[ -z $output ]]; then
    echo 'this process doe not exist'
    exit 0
fi

set -- $output
pid=$1

echo killing proces $pid
kill -9 $pid
#sleep 2
#kill -9 $pid >/dev/null 2>&1


#
# Reference:
# [1]
# http://stackoverflow.com/questions/6437602/shell-script-to-get-the-process-id-on-linux
# The backticks allow you to capture the output of a comand in a shell variable.
# The set -- parses the ps output into words, and $2 is the second word on the
# line which happens to be the pid. Then you send a TERM signal, wait a couple
# of seconds for ruby to to shut itself down, then kill it mercilessly if it
# still exists, but throw away any output because most of the time kill -9 will
# complain that the process is already dead.
#
# [2]
# http://www.cyberciti.biz/tips/grepping-ps-output-without-getting-grep.html
# when ps aux | grep something, to avoid getting the line of grep, do either of
# 1) ps aux | grep something | grep -v grep
# 2) ps aux | grep [s]omething
# 3) ps aux | grep '[s]omething'
# for 2) and 3), it's actually a regular expression, [s] matches to s.
#

How to Install and Use Screen on an Ubuntu Cloud Server

Screen is a console application that allows you to use multiple terminal sessions within one window. The program operates within a shell session and acts as a container and manager for other terminal sessions, similar to how a window manager manages windows.

This is like the console version of Mac and Linux's multiple desktops. Very handy.

To run a program (say a server) in a screen session, detach it without killing the session:
     screen -d -m python server/server.py 80

This is like running the "daemonize" command, can make the program an independent daemon process.

[1] How to Install and Use Screen on an Ubuntu Cloud Server

Websocket

Websocket can be used to create real time chat and game services.

Autobahn|Python [3] is a WebSocket / WAMP library for Python 2 (using Twisted) and 3 (using asyncio). It's easy to setup and use.

Websocket and browser support [1]:

WebSocket is a protocol providing full-duplex communications channels over a single TCP connection. The WebSocket protocol was standardized by the IETF as RFC 6455 in 2011, and the WebSocket API in Web IDL is being standardized by the W3C.

WebSocket is designed to be implemented in web browsers and web servers, but it can be used by any client or server application. The WebSocket Protocol is an independent TCP-based protocol. Its only relationship to HTTP is that its handshake is interpreted by HTTP servers as an Upgrade request.[1] The WebSocket protocol makes more interaction between a browser and a website possible, facilitating live content and the creation of real-time games. This is made possible by providing a standardized way for the server to send content to the browser without being solicited by the client, and allowing for messages to be passed back and forth while keeping the connection open. In this way a two-way (bi-directional) ongoing conversation can take place between a browser and the server. A similar effect has been achieved in non-standardized ways using stop-gap technologies such as Comet.

In addition, the communications are done over TCP port number 80, which is of benefit for those environments which block non-web Internet connections using a firewall. The WebSocket protocol is currently supported in most major browsers including Google Chrome, Internet Explorer, Firefox, Safari and Opera. WebSocket also requires web applications on the server to support it.


Turn on websockets in firefox [2]:

1. Type about:config in address bar, and continue by clicking “I’ll be careful, I promise”
2. Set network.websocket.enabled  value to ‘true’ and set network.websocket.override-security-block preferences to ‘true’.
3. Restart Firefox browser.

== Change a websocker server to daemon ==

Each time you can run this to start a websocket server: python ./server.py

To change it to a daemon you can do this:
- add "#!/usr/bin/python" to top of server.py
- install daemonize [4]
- use absolute path of server.py, run this: daemonize /../server.py

Now server.py is started as a daemon. To find out which process it's running as and to shut it down, do:

- ps -aux | grep server.py

This will (usually, if you don't have other process with the name "server") find 2 entries, one for the server.py process, one for the grep command. Pick the former, you can watch its activity by:

- top -p [pid]

To kill it:

- kill -9 [pid]


[1] http://en.wikipedia.org/wiki/WebSocket
[2] http://techdows.com/2010/12/turn-on-websockets-in-firefox-4.html
[3] http://autobahn.ws/
[4] http://software.clapper.org/daemonize/ 
[5] websocket browser support

Monitor Linux (Ubuntu) network traffic in real time

== Monitor incoming network traffic in real time:
[http://manpages.ubuntu.com/manpages/lucid/man1/tcptrack.1.html]

ifstat
iftop
iptraf
tcptrack

== Commands to verify ports:
[https://www.serverpronto.com/accounts/knowledgebase.php?action=displayarticle&id=11]

nmap IP#
nmap localhost
netstat –ntulp

== to verify single port

netstat -nap | grep 

== to list all current rules in iptables

iptables -L

== For opening a TCP port:

iptables -A INPUT  -p tcp –dport -j ACCEPT

== For opening a UDP port:

iptables -A INPUT -p udp –sport   -j ACCEPT

== Save changes:

iptables-save > /etc/iptables.rules

== If you need to disable the firewall temporarily, you can flush all the rules using:

iptables -F

Sqlite3

Sqlite3 is a light-weighted database server. It comes directly with Django.

Here is an introduction to its use: http://www.sqlite.org/sessions/sqlite.html

Some common commands:

- to enter sqlite3 shell: sqlite3 [sqlite3 db filename]
- show all databases: .databases
- show all tables: .tables
- query a table: select * from sqlite_master;
- show all table schema: .schema
- backup database to another file: .backup another_db.db
- exit shell: .q, or .exit

Note that there is only 1 database per database file in Sqlite3. So there is no command like "use database1".

Some notes:

- For sqlite3 database to be writable, it should be writable by the web account, its containing folder should also be writable by the web account.

- If a column in missing in a table, you can do this to add it:
sqlite3 yourdb.db
> drop table bookmarks_bookmark;
> .quit
cd yourpythonproj
python manage.py syncdb
python manage.py runserver

Basically, syncdb cannot change integrate schema once the tables are created. You have to drop that table first (or maybe need to remove the database totally, which can be done by mv the current db.sqlite3 to db.sqlite3.bak and do syncdb).

shell script to concat all files

This shell script runs on *nix and concatenates all files into a single one, including file name before file content. This is useful, for example, when you want to print out all files to read.

#!/bin/bash

#
# Dump multiple files to one text file.
# By: HomeTom. 3/11/2015
#

cd my_dir;  # go to the target file directory.

ct=0;  # keep a counter of file.

for i in README.md manage.py my_dir/*.py;
do
    ct=$(($ct + 1));
    echo;
    echo == File $ct: "$i" ==;  # print file name and its counter.
    echo;
    cat "$i";
done

This second shell script is improved from the above, by adding a table of contents (a list of all files) to the beginning of output.

#!/bin/bash

#
# Dump multiple files to one text file.
# By: HomeTom. 3/11/2015
#

cd my_dir;  # go to target file directory.

# define file list.
arr=(README.md manage.py my_dir/*.py);

# 1) get table of contents.
echo Table of Contents;
echo;
ct=0;  # keep a counter of file.
for i in ${arr[@]};
do
    ct=$(($ct + 1));
    echo == File $ct: "$i";  # print file name and its counter.
done

# 2) get file content.
ct=0;  # keep a counter of file.
for i in ${arr[@]};
do
    ct=$(($ct + 1));
    echo;
    echo == File $ct: "$i" ==;  # print file name and its counter.
    echo;
    cat "$i";
done

Django framework

To tell if you have Django installed with Python:
$ python -c "import django; print(django.get_version())"

Setup Django on bluehost:

[1] Python 2.7 + Django 1.4 on Bluehost
[2] Django on Bluehost in Five Minutes
[3] Django documentation
[4] Django documentation - tutorial

To access database using Django, see:
[5] Writing your first Django app, part 1 - create project and model (database)
[6] Writing your first Django app, part 2 - create admin site superuser
      basically it's by command: $ python manage.py createsuperuser

== Appendix 1. Python 2.7 + Django 1.4 on Bluehost ==

To avoid loss of previous web page, the basic process [1] is copied here:
Monday, April 30, 2012
Python 2.7 + Django 1.4 on Bluehost

Bluehost is a cheap shared hosting provider, that allows to run applications using fastCGI, among others, webapps created with django - python web framework. In this post you will find information how to install the newest (in April 2012) versions of python and django and how to configure them for  bluehost.

First you need to enable ssh access to your bluehost account. Sign in to their control panel, enter Security > SSH/Shell Access and click Manage SSH Access button. Select SSH Access enabled and submit. Now connect via ssh to your bluehost account.

Python
Download, extract and install python 2.7 in your home directory:
mkdir python27
wget http://www.python.org/ftp/python/2.7.2/Python-2.7.2.tgz
tar xzvf Python-2.7.2.tgz
cd Python-2.7.2
./configure -prefix=/homeX/your_username/python27 --enable-unicode=ucs4
make
make install

Rename the python binary (to avoid overriding current system python version - if you're not using default bluehost python for anything else you can skip this step):
mv ~/python27/bin/python ~/python27/bin/python27
Add new python directory with binaries to your PATH environment variable. To do this edit .bashrc file in your HOME directory, for example with vim:
vim ~/.bashrc
and add the line at the end of the .bashrc file:
PATH=/homeX/your_username/python27/bin:$PATH
load the new settings:
source .bashrc
and then test it:
python27
You should see something similar to:
Python 2.7.2 (default, Apr 11 2012, 01:29:09)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-52)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

New python is available for you. Remember to use command python27 instead of python from now on.

Pip
We're going to use pip here - easy to use python package manager. It requires setuptools before it can be installed:

wget http://pypi.python.org/packages/source/s/setuptools/setuptools-0.6c11.tar.gz
tar xzvf setuptools-0.6c11.tar.gz
cd setuptools-0.6c11
python27 setup.py install
cd
wget http://pypi.python.org/packages/source/p/pip/pip-1.1.tar.gz
tar xzvf pip-1.1.tar.gz
cd pip-1.1
python27 setup.py install

Now thanks to our earlier PATH settings pip is available from command line.

Django installation
With pip you can install latest stable Django (in my case 1.4) simply by entering:
pip install Django
Wait for download and installation process to complete. After that django is installed and command django-admin.py available. You will also probably need python driver for mysql database. Install it with pip:
pip install MySQL-python
If you're going to use PosgreSQL install psycopg2 library:
pip install psycopg2

Django configuration
Create django project:
django-admin.py startproject myproject

Decide what url should be used to access your django app. In my case it will be subdirectory:
http://somedomain.com/myproject

Prepare directory in bluehost public_html:
mkdir public_html/myproject
cd public_html/myproject


In this directory create fastcgi file (for example: mysite.fcgi) with content:
#!/homeX/your_username/python27/bin/python27
import sys, os

# Add a custom Python path.
sys.path.insert(0, "/homeX/your_username/python27")
sys.path.insert(13, "/homeX/your_username/myproject")

os.environ['DJANGO_SETTINGS_MODULE'] = 'myproject.settings'
from django.core.servers.fastcgi import runfastcgi
runfastcgi(method="threaded", daemonize="false")

Remember to change /homeX/your_username to your home directory path on bluehost. Change myproject.settings to correct project name. The first line contains path to your custom python binary.

Django fastcgi requires flup so install it with pip:
pip install flup
Also change file permissions to mysite.fcgi:
chmod 0755 mysite.fcgi

Now create .htaccess file in the public_html/myproject directory with content:
AddHandler fcgid-script .fcgi
RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(.*)$ mysite.fcgi/$1 [QSA,L]


That's all. You should see your django start page on: http://somedomain.com/myproject. You can start coding some real stuff in django now. Remember to install every python and django packages using pip - that way they will be available to your django app. Also keep in mind that every django commands like syncdb, collectstatic  etc. should be run using python27 for example:
python27 manage.py syncdb


== Appendix 2. Python/Django setup on Mac ==

-- Intall wget on Mac --

Note, on Mac, to install wget, you need to download from http://ftp.gnu.org/gnu/wget/, and then do:
./configure --with-ssl=openssl
make
make install

See:
[5] http://coolestguidesontheplanet.com/install-and-configure-wget-on-os-x/

-- Install setuptools --

For more recent version of setup tools, follow instruction at:
[6] https://pypi.python.org/pypi/setuptools

You may need superuser privilege:
wget https://bootstrap.pypa.io/ez_setup.py -O - | sudo python

-- Install mysql-python --

When install mysql-python, it may report "EnvironmentError: mysql_config not found". See
[7] http://stackoverflow.com/questions/25459386/mac-os-x-environmenterror-mysql-config-not-found
In that case, do:

> locate mysql_config
# this should report something like: /usr/local/mysql-5.5.18-osx10.6-x86_64/bin/mysql_config.
# then add it to current path:
> PATH=$PATH:/usr/local/mysql-5.5.18-osx10.6-x86_64/bin/
> mysql_config
# this should show that mysql_config is available. Finally:
> sudo pip install mysql-python

-- Install mod_fcgid --

On mac, you may need to install mod_fcgid. Without mod_fcgid, will get this error:
/mysite.fcgi/ was not found on this server.
E.g.: http://stackoverflow.com/questions/24446496/deploying-django-error-mysite-fcgi-was-not-found-on-this-server-in-shared

To install mod_fcgid, you need to install macport first. See:
[8] http://www.dionysopoulos.me/apache-mysql-php-server-on-mac-os-x-with-multiple-simultaneous-php-versions/

To install macport, go to: http://www.macports.org/install.php. Use "pkg" installer.
Then do:
sudo port selfupdate
sudo port install mod_fcgid
sudo cp /opt/local/apache2/modules/mod_fcgid.so /usr/libexec/apache2

Then: sudo vi /etc/apache2/httpd.conf

Find all the LoadModule lines. After the last one add:
LoadModule fcgid_module libexec/apache2/mod_fcgid.so

Then restart apache:
sudo apachectl restart


== Appendix 3. Use Python in the web ==

Use Python in the web: https://docs.python.org/2/howto/webservers.html

Save code below as a.fcgi. This can run as fastcgi.

#!/usr/bin/env python
# -*- coding: UTF-8 -*-

from cgi import escape
import sys, os
from flup.server.fcgi import WSGIServer

def app(environ, start_response):
    start_response('200 OK', [('Content-Type', 'text/html')])

    yield 'Use Python in the web: https://docs.python.org/2/howto/webservers.html'
    yield '<h1>FastCGI Environment</h1>'
    yield '<table>'
    for k, v in sorted(environ.items()):
         yield '<tr><th>%s</th><td>%s</td></tr>' % (escape(k), escape(v))
    yield '</table>'

WSGIServer(app).run()


== Appendix 4. Install rest_framework ==

sudo pip install djangorestframework

怎样尊重一个程序员

怎样尊重一个程序员 (from http://www.yinwang.org)

Why I left Google.

Replace failed RAID controller

OK, here is some experience on building a RAID server and fixing hardware failure. This is quite interesting as software engineers do not often have the chance to work on hardware.

The server uses PERC 6/E card to connect to a disk array. The disks in the disk array have the same physical capacity, say 1TB, but configured as logical drives with large capacity each (which effectively means you combine several physical drives into one logical drive).  PERC stands for PowerEdge RAID Controller. "E" in 6/E stands for External, which means HDDs are in a expansion box separate from the server box, attached to server using a SCSI interface such as MD1000 or MD3000. Correspondingly "I" in 6/I means "Internal", where HDDs are in the same server box. PowerEdge is a server brand of Dell. Relevant servers are listed on this wiki page: List of Dell PowerEdge Servers.

[1] is a very good step by step guide on how to do the configuration on a PERC 6/I card. It is the same for configuring a PERC 6/E card. [2] is a more general and complete guide.

There are several things one can do.

At boot time, press CTRL-R to enter PERC configuration utility. Once in the configuration utility, you can see a list of PERC cards, click "enter" on one to enter its configuration. In the next page "VD Mgmt", press "F2" for actions to choose from, including: Create New VD, Clear Config, Foreign Config (Import/Clear).

In the case the PERC card failed, the speed to access PERC configurations can be painfully slow each operation. Once the new good card is replaced on board, all steps are very much faster. The difference is like 10-30 minutes versus a few seconds.

PERC configuration is written to each RAID disk. In case of a PERC failure, just replace the PERC card and "import" foreign configuration. A configuration is "foreign" means the PERC configuration does not match configuration information on the RAID disks. When a PERC card fails its configuration may become inconsistent with information on the disks, and you may see warning lights on the disks blinking and the status of the lights is inconsistent upon each reboot.

If 1 or 2 disks in array failed, you just pull it out and insert a new disk. RAID should rebuild automatically.

If a large number of disks are in foreign state, you may want to clear foreign configuration and recreate from scratch. Note that for new setup you can "Initialize" the disks to clean them, but for disks with existing data (in the case of recovering from a failed PERC card, where disks with data should keep intact) you should not initialize the disks since you want to preserve the data.

If you need to create new VD (Virtual Disks), you need to choose RAID level, then assign disks to use, and set some options.

It's actually quite a lot of fun.

[1] DELL Tutorial: Create RAID Using PERC 6/i Integrated BIOS Configuration Utility
[2] Dell™ PERC 6/i, PERC 6/E and CERC 6/i User's Guide