Sunday, June 29, 2014

A Python daemon

A simple unix/linux daemon in Python. This is for Python 2.x. The article also contains a link for Python 3.x (link is here).

The python-daemon library is another solution.

Saturday, June 28, 2014

A Perl HTTP Web Server - Full source code

This is the full source code of the HTTP web server written in Perl we discussed in the previous post.

If you visit a page /index.html which contains an image tag <img src="bg.jpg">, the log file of this HTTP web server will be something like:

2014-7-3 3:47:28  server is started
2014-7-3 3:47:28  Listening on [IO::Socket::INET=GLOB(0x7f8bd98bd9f0)] 0.0.0.0:9000 ...
2014-7-3 3:52:40  == incoming connection from [IO::Socket::INET=GLOB(0x7f8bd98bd9f0)] 127.0.0.1:51659 ...
2014-7-3 3:52:40  request file: /index.html
2014-7-3 3:52:40  == incoming connection from [IO::Socket::INET=GLOB(0x7f8bd98bd9f0)] 127.0.0.1:51661 ...
2014-7-3 3:52:40  request file: /bg.jpg


Some notes:
- Here LocalAddr should be "0.0.0.0" instead of "localhost" or "127.0.0.1" if you want remote computers to be able to access it. If it's "localhost" or "127.0.0.1" then you can access it only from local machine.
- Besides using a browser, you can also access the web server using telnet: telnet [ip] [port]
- Use this command to see open ports on local machine: netstat -an | grep "LISTEN"
- On Ubuntu, use this to add a chain rule to iptables firewall to open a port (e.g., see here): iptables -A INPUT -p tcp --dport 9000 -j ACCEPT
- On Mac, besides the firewall in System Preferences -> Security & Privacy -> Firewall, there is another deprecated firewall: ipfw. By default it opens all the ports though. To see its rules, use: sudo ipfw list.
- Here we use a buffer size of 1024 in the recv call. In reality, the GET request length is limited from 2K to 8K bytes for different browsers. For example, this source says: The limit (of HTTP GET Request) is in MSIE and Safari about 2KB, in Opera about 4KB and in Firefox about 8KB. We may thus assume that 8KB is the maximum possible length and that 2KB is a more affordable length to rely on at the server side and that 255 bytes is the safest length to assume that the entire URL will come in.

Note that since HTTP is a stateless protocol, we close the connection immediately after each time we send response. In reality, we may not be able to receive the entire request message in one read. But as a demonstration of concept this serves pretty well.

One may ask why recreate the wheel? The answer is, of course, we don't need another primitive HTTP web server, we already got so many. But if we need a service, which resides on a linux/unix/mac server and listens to a specific port, to accomplish a specific task, then we can easily write a server of our own this way using this template, using our own customized protocol.  Another situation is that if for some reason you don't have a web server, then you can use this template to build one. Of course, this later case rarely happens today.

As for the applicability, this Perl server runs well on linux/unix/mac. For windows, it is easy to build something similar as a windows service, such as a .NET remoting TCP server, to listen on a specific port and accomplish similar tasks.


#
# This script demonstrates a functional HTTP web server in Perl:
# 1) running the Perl HTTP web server as a daemon in background.
# 2) only one copy of the server can run by checking "Proc::PID::File->running()".
# 3) implementation of daemon commands: start, stop, status.
#    e.g. start the web server by: sudo perl dmon_server.pl start.
#
# Note: 
# 1) to run as daemon, "sudo" should be used for non-admin user.
# 2) parameters that can change: $LOG_FILE, $WWWROOT, $USE_OPT,
#     and $localport in functoin do_start().
#
# @By: X.C.
# @Created on: 6/28/2014
# @Last modified: 7/2/2014
#

#!/usr/bin/perl

use strict;
use warnings;
use Getopt::Long;
use Proc::Daemon;
use Proc::PID::File;
use IO::Select;
use IO::Socket;
use URI::Escape;

#
# Location of log file.
#
my $LOG_FILE = "/Users/chenx/tmp/dmon.log";
#
# Location of web root.
#
my $WWWROOT = "/Users/chenx/tmp/wwwroot";

#
# If $USE_OPT = 1, use GetOptions. 
# 0 is better here because if an arg does not start with "--",
# it will be ignored and no usage information is printed.
#
my $USE_OPT = 0; 


my $len = @ARGV;
if ($len == 0) {
    show_usage();
} else {
    if ($USE_OPT) {
        GetOptions(
            "start" => \&do_start,
            "status" => \&show_status,
            "stop" => \&do_stop,
            "help" => \&show_usage
        ) or show_usage();
    } else {
        my $cmd = $ARGV[0];
        if ($cmd eq "start") { do_start(); }
        elsif ($cmd eq "stop") { do_stop(); }
        elsif ($cmd eq "status") { show_status(); }
        else { show_usage(); }
    }
}


#
# 1 at the end of a module means that the module returns true to use/require statements. 
# It can be used to tell if module initialization is successful. 
# Otherwise, use/require will fail.
#
# 1;


sub show_usage {
    if ($USE_OPT) {
        print "Usage: sudo perl $0 --[start|stop|status|help]\n";
    } else {
        print "Usage: sudo $0 [start|stop|status]\n";
    }
    exit(0);
}


sub show_status {
    if (Proc::PID::File->running()) {
        print "daemon is running..\n";
    } else {
        print "daemon is stopped\n";
    }
}


sub do_stop {
    my $pid = Proc::PID::File->running();
    if ($pid == 0) {
        print "daemon is not running\n";
    } else {
        #print "stop daemon now ..\n";
        kill(9, $pid);
        print "daemon is stopped\n";
    }
    do_log("server is stopped");
}


sub do_start {
    print "start daemon now\n";

    Proc::Daemon::Init();

    if (Proc::PID::File->running()) {
        do_log( "A copy of this daemon is already running, exit" );
        exit(0);
    }

    do_log("server is started");

    my ($data, $fh, $data_len); 
    my $localhost = "0.0.0.0";
    my $localport = 9000;
    my $ipc_select = IO::Select->new();
    my $IPC_SOCKET = new IO::Socket::INET(
             Listen  => 5, LocalAddr => $localhost, LocalPort => $localport, Proto => "tcp" );

    $ipc_select->add($IPC_SOCKET);
    do_log( "Listening on [$IPC_SOCKET] $localhost:$localport ..." );
    while (1) {
      if (my @ready = $ipc_select->can_read(.01)) {
        foreach $fh (@ready) {
            if($fh == $IPC_SOCKET) {
                my $new = $IPC_SOCKET->accept;
                $ipc_select->add($new);
                do_log( "== incoming connection from [$fh] " . 
                        $new->peerhost() . ":" . $new->peerport() . " ...");
            } else {
                recv($fh, $data, 1024, 0);
                my $data_len = length($data);
                if ($data_len > 0) { # feedback to client.
                    print $fh http_response($data);
                }
                $ipc_select->remove($fh);
                $fh->close;
            }
        }
      }
    }
}



#
# Implements part of the HTTP protocal:
# - GET command
# - Status code: 200, 400, 404, 500
# - Customized command TEST, with status code -1.
#
# Reference: http://www.w3.org/Protocols/rfc2616/rfc2616.html
#
sub http_response {
    my ($request) = @_;

    my $status = 0;
    my $data = "";
    if ($request =~ m/^GET\s(\S+)\s/) {
        do_log("request file: $1");
        my $file = ($1 eq "/") ? "/index.html" : $1;  # default file under a directory.
        $file = uri_unescape($file); # url_decode() function. E.g. change %20 back to space.

        my $path = "$WWWROOT$file";
        if (-e $path) {
           my $open_ok = 1;
           open my $fh, '<', $path or $open_ok = 0; # die "error opening $path: $!";
           if ($open_ok == 1) {
               $data = do { local $/ = undef; <$fh> };
               $status = 200;
           } else {
               $data = "Internal Server Error";
               $status = 500;
           }
        }
        else { # file not found.
           $data = "Not Found";
           $status = 404;
        }
    } elsif ($request =~ m/^TEST\s/) {
        $status = -1; # for testing purpose of the Perl Web Server.
    } else { # bad request: unknown command.
        $data = "Bad Request";
        $status = 400;
    }


    my $response = "";
    if ($status == 200) {
        my $data_len = length($data);
        $response = "HTTP/1.1 200 OK\nContent-Type:text\nContent-Length:$data_len\n\n$data";
    }
    elsif ($status == 400 || $status == 404 || $status == 500) {
        my $body = "";
        my $data_len = length($body);
        $response = "HTTP/1.1 $status $data\nContent-Type:text\nContent-Length:$data_len\n\n$body";
    }
    else { # status == -1
        my $hdr = "PERL Web Server Received:\n";
        my $data_len = length($hdr) + length($request);
        $response = "HTTP/1.1 200 OK\nContent-Type:text\nContent-Length:$data_len\n\n$hdr$request";
    }

    return $response;
}


sub do_log {
    my ($msg) = @_;

    my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime();
    $year += 1900;
    $mon += 1;

    open FILE, ">>$LOG_FILE" or die "cannot open log file $!\n";
    print FILE "$year-$mon-$mday $hour:$min:$sec  $msg\n";
    close FILE;
}


A Perl HTTP Web Server

This is a simple server written in Perl. It listens on a port, but does not follow the HTTP protocol. It just echoes whatever the client sent in.

Note for examples in this post the value of "LocalAddr" is "localhost", so the servers here are accessible only on local machine. If you want access from a remote machine, you should change it to "0.0.0.0".


# This demonstrates a simple Perl HTTP server.
# A user can connect using: telnet [host] [port], here port = 9000.
# The user will say something, which the server repeats.
# When the user says "bye", the connection is closed.
#
# Modified from: http://www.perlmonks.org/?node_id=49823
# By: X.C.
# Created on: 6/28/2014
# Last modified: 6/28/2014
#
#!/usr/bin/perl -w
use strict;
use IO::Select;
use IO::Socket;

my ($data, $fh);
my $ipc_select = IO::Select->new();
my $IPC_SOCKET = new IO::Socket::INET(Listen    => 5,
                                    LocalAddr => 'localhost',
                                    LocalPort => 9000,
                                    Proto   => "tcp" );


$ipc_select->add($IPC_SOCKET);
print "Listening on Socket [$IPC_SOCKET] ...\n";
while (1) {
    if (my @ready = $ipc_select->can_read(.01)) {
        foreach $fh (@ready) {
            if($fh == $IPC_SOCKET) {
                #add incoming socket to select
                my $new = $IPC_SOCKET->accept;
                $ipc_select->add($new);
                print "== incoming connection...\n";
            } else {
                # Process socket
                my $n = recv($fh, $data, 1024, 0); # this seems to be empty.
                my $data_len = length($data); # $data ends with "\r\n".
                #print "data len:" . length($data) . "\n";
                if ($n || $data_len > 0) {
                    print $fh "Server feedback: $data"; # feedback to client.
                    $data = substr($data, 0, $data_len - 2);
                    print "incoming data: $data\n";
                    #chomp($data); # this won't work, since $data is a buffer not filled with 0s.
                    if (uc($data) eq "BYE") {
                        print "== close connection\n";
                        $ipc_select->remove($fh);
                        $fh->close;
                    }
                } else { # seems this won't get executed ever.
                    $ipc_select->remove($fh);
                    $fh->close;
                }
            }
        }
    }
}

Or it can be simplified into the code below. Now this uses standard HTTP response and is a HTTP web server, so you can actually visit it from a browser using url http://localhost:9000/.


#!/usr/bin/perl -w
use strict;
use IO::Select;
use IO::Socket;

my ($data, $fh);
my $ipc_select = IO::Select->new();
my $IPC_SOCKET = new IO::Socket::INET(
    Listen  => 5, LocalAddr => 'localhost', LocalPort => 9000, Proto => "tcp" );

$ipc_select->add($IPC_SOCKET);
print "Listening on Socket [$IPC_SOCKET] ...\n";
while (1) {
    if (my @ready = $ipc_select->can_read(.01)) {
        foreach $fh (@ready) {
            if($fh == $IPC_SOCKET) {
                my $new = $IPC_SOCKET->accept;
                $ipc_select->add($new);
                print "== incoming connection from [$fh]...\n";
            } else {
                recv($fh, $data, 1024, 0); 

                my $data_len = length($data);
                if ($data_len > 0 && uc($data) ne "BYE\r\n") { # feedback to client.
                    print $fh "HTTP/1.0 200 OK\nContent-Type:text\nContent-Length:$data_len\n\n$data"; 
                } else { 
                    $ipc_select->remove($fh);
                    $fh->close;
                }
            }
        }
    }
}

So far this is run in a console, and is tied to the controlling console. If you type CTRL-C then it's stopped, or when the console is closed it's gone.  However, by combining this with the Perl daemon we discussed in the previous post, we obtain a full-fledged web server written in Perl! This web server runs as a daemon process. You can visit it in a web browser using http://localhost:9000. So far the only thing this web server does is to prepend the client request with "PERL Web Server Received:" and send it back to the client.

To be exact, the only modifications needed to the Perl daemon in the previous post are:

1) add this to the top of file:

use IO::Select;
use IO::Socket;


2) replace the do_start() function with:

sub do_start {
    print "start daemon now\n";

    Proc::Daemon::Init();

    if (Proc::PID::File->running()) {
        do_log( "A copy of this daemon is already running, exit" );
        exit(0);
    }

    my ($data, $fh, $data_len);
    my $ipc_select = IO::Select->new();
    my $IPC_SOCKET = new IO::Socket::INET(
        Listen  => 5, LocalAddr => 'localhost', LocalPort => 9000, Proto => "tcp" );

    $ipc_select->add($IPC_SOCKET);
    do_log( "Listening on Socket [$IPC_SOCKET] ..." );
    while (1) {
      if (my @ready = $ipc_select->can_read(.01)) {
        foreach $fh (@ready) {
            if($fh == $IPC_SOCKET) {
                my $new = $IPC_SOCKET->accept;
                $ipc_select->add($new);
                do_log( "== incoming connection from [$fh]..." );
            } else {
                recv($fh, $data, 1024, 0);
                my $data_len = length($data);
                if ($data_len > 0) { # feedback to client.
                    print $fh http_response($data);
                }
                $ipc_select->remove($fh);
                $fh->close;
            }
        }
      }
    }
}

sub http_response {
    my ($request) = @_;
    my $hdr = "PERL Web Server Received:\n";
    my $data_len = length($hdr) + length($request);

    return "HTTP/1.0 200 OK\nContent-Type:text\nContent-Length:$data_len\n\n$hdr$request";
}

A Perl daemon

A daemon is a process running in the background continuously. It is the linux comparable of a windows service. A daemon or windows service can do many things in the background. If you let it open and listen to a port, it becomes a server, e.g., a web server like Apache or IIS, or a database server like MySQL or MongoDB. For this reason, it is a very interesting and useful thing to learn how to write a daemon or windows service. The installation and/or execution of a linux daemon or windows service usually requires the user to have admin permission.

A Perl daemon can be easily implemented using the Proc::Daemon module [1]. If you don't have this installed, you can do it by:

sudo perl -MCPAN -e shell
cpan[1]> install Proc::Daemon
cpan[2]> install Proc::PID::File

A Perl daemon can be as simple as this (note you will need to run it with "sudo"):

#!/usr/bin/perl
use Proc::Daemon;
use Proc::PID::File;

Proc::Daemon::Init; # Demonize.
if (Proc::PID::File->running()) { exit(0); # Exit if already running, so only 1 instance can run.
for (;;) { 
    # print "do something every 5 seconds..\n"; # This won't print because STDOUT is closed.
    sleep(5); 
}


You cannot really see any console print output since STDOUT is closed by Proc::Daemon. You can see output by printing to a log file, or use the "top" command to see this new process.

Here is a more full-fledged version of a Perl daemon. It allows you to use start/stop/status commands to control and monitor the daemon. A log file is also generated.


#
# This script demonstates:
# 1) running a Perl script as daemon in background, using the Daemon module.
# 2) only one instance of the daemon can run by checking "Proc::PID::File->running()".
# 3) implementation of daemon commands: start, stop, status.
# 4) use of Getopt::Long, $0 (name of this file).
#
# Note:
# 1) to run as daemon, "sudo" should be used for non-admin user.
# 2) parameters that can change: $LOG_FILE, $USE_OPT.
#
# @By: X.C.
# @Created on: 6/28/2014
# @Last modified: 6/28/2014
#

#!/usr/bin/perl

use strict;
use warnings;
use Getopt::Long;
use Proc::Daemon;
use Proc::PID::File;

#
# Location of log file.
#
my $LOG_FILE = "/Users/chenx/tmp/dmon.log";

#
# If $USE_OPT = 1, use GetOptions.
# 0 is better here because if an arg does not start with "--",
# it will be ignored and no usage information is printed.
#
my $USE_OPT = 0;


my $len = @ARGV;
if ($len == 0) {
    show_usage();
} else {
    if ($USE_OPT) {
        GetOptions(
            "start" => \&do_start,
            "status" => \&show_status,
            "stop" => \&do_stop,
            "help" => \&show_usage
        ) or show_usage();
    } else {
        my $cmd = $ARGV[0];
        if ($cmd eq "start") { do_start(); }
        elsif ($cmd eq "stop") { do_stop(); }
        elsif ($cmd eq "status") { show_status(); }
        else { show_usage(); }
    }
}


#
# 1 at the end of a module means that the module returns true to use/require statements.
# It can be used to tell if module initialization is successful.
# Otherwise, use/require will fail.
#
# 1;


sub show_usage {
    if ($USE_OPT) {
        print "Usage: sudo perl $0 --[start|stop|status|help]\n";
    } else {
        print "Usage: sudo $0 [start|stop|status]\n";
    }
    exit(0);
}


sub show_status {
    if (Proc::PID::File->running()) {
        print "daemon is running..\n";
    } else {
        print "daemon is stopped\n";
    }
}


sub do_stop {
    my $pid = Proc::PID::File->running();
    if ($pid == 0) {
        print "daemon is not running\n";
    } else {
        #print "stop daemon now ..\n";
        kill(9, $pid);
        print "daemon is stopped\n";
    }
}


sub do_start {
    print "start daemon now\n";

    Proc::Daemon::Init();

    #
    # To use this, you need to start the daemon with "sudo" to have the permission
    # to PID file. To kill the daemon, "sudo" is also needed.
    #
    if (Proc::PID::File->running()) {
        do_log( "A copy of this daemon is already running, exit" );
        exit(0);
    }

    my $continue = 1;
    $SIG{TERM} = sub { $continue = 0; };

    while ($continue) {
        #print "continue ..\n"; # this won't work, since STDOUT is closed.
        do_log("continuing ..");

        sleep(2);
    }
}


sub do_log {
    my ($msg) = @_;

    my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime();
    $year += 1900;

    open FILE, ">>$LOG_FILE" or die "cannot open log file $!\n";
    print FILE "$year-$mon-$mday $hour:$min:$sec  $msg\n";
    close FILE;
}


Alternatively, you can also implement a daemon without using Proc::Daemon.  An example is below.

This code [2] is back from year 2000. The current Proc::Daemon implements majority of these ideas. These are also the basic requirements of a well-rounded daemon, so it won't become a demon:

1) Fork a child process as the daemon, the exit the parent process. This guarantees the daemon process is free of any controlling process and terminal.
2) Call setsid. This makes the daemon process a session leader, and free of any controlling terminal/shell.
3) Change working dir, usually "/". This prevents the relevant partition where the working dir resides from resisting unmounting by admin.
4) Use umask to cancel the default file creation mode permission inherited from parent process.
5) Close unneeded file handlers, mostly STDIN, STDOUT and STDERR, by redirecting them to /dev/null. This is because a daemon has no associated terminal/shell and has nowhere to write these to.
6) Logging message, so you know what's going on when STDOUT and STDERR are closed.


#!/usr/bin/perl

#
# This demonstrates running a Perl program in background as daemon without using Proc::Daemon.
# More than 1 instance can run at the same time.
# From: [2] 
#

use POSIX qw(setsid);

chdir '/';
umask 0;
open STDIN, '/dev/null';
#open STDOUT, '>/Users/chenx/tmp/dmon.log';
open STDERR, '>/dev/null';

defined(my $pid = fork);
exit if $pid;
setsid;

while(1)
{
    sleep(2);
    #print "Hello...\n";
    do_log();
}

# note: cannot use "log" as it's preserved.
sub do_log {
    open LOGFILE, '>>', '/Users/chenx/tmp/dmon.log' or die "cannot open file"; # $!;
    print LOGFILE "continue on ..\n";
    close LOGFILE;
}


There is an interesting Perl Daemon Contest [3] here, back in 2000. Some of these are pretty complicated and highly practical, such as web proxy, system monitoring and reporting, stock monitoring, even a reversed-engineered printer daemon, and an interesting AI game of rock/scissors/paper played inside the file system.


References:

[1] Proc::Daemon implementation version 0.14
[2] Unix Daemon In Perl Tutorial
[3] Perl Daemon Contest


Thursday, June 26, 2014

Trying out mongoDB

Today I'm trying out mongoDB. MongoDB is an open source NoSQL document database.

Important resources:

http://www.mongodb.org - for any user to get to know it, and download
http://www.mongodb.org/downloads - download, manual (in html and pdf) is at the end of this page
http://docs.mongodb.org/ecosystem/drivers/ - mongoDB drivers for all languages
http://docs.mongodb.org/manual/reference/sql-comparison
http://www.mongodb.com - for enterprise users. Enterprise MongoDB Architecture.

Using MongoDB with ASP.NET Web API - need to install a mongoDB driver for .NET
Create a C# ASP.NET Application on Azure with MongoDB using the MongoLab Add-On

PHP manual on mongoDB - from installation to usage

Introduction to MongoDB for Java, PHP and Python Developers (May, 2012)

MongoDB partly obtain its performance from using memory mapped file. It supports easy replica for high up time. It keeps most important contents in memory and horizontally scale, shard automatically. It uses dynamic schema, and supports many programming languages and platforms. Its customers often have a cluster with over 100 nodes, some over 1000 nodes; data size ranges from TB to PB.

MongoDB installation package is self-contained and quite clean, in that it's a "green" software. The console is like MySQL or Python console in commands, and uses JavaScript as the query language. The comparable to MSSQL is: database - database, table - collection, row - document, column - field, index - index, table joins - embedded documents and linking, aggregate function - aggregation pipeline.

Installation on windows is recommended on Windows Server 2008 or above. Windows XP is not supported in MongoDB version 2.2 or above. Install as a windows service is easy. MongoVUE is a windows client, similar to PhpMyAdmin (actually, a web client instead of windows client) for MySql.

Friday, June 20, 2014

Convert dmg file to iso on windows

To convert dmg file to iso on windows, tools for this include:

PowerISO (commercial), dmg2img (GPL, how to. easy to use).


Move PhpBB from root to subdirectory

When move a PhpBB forum from domain root to a subdirectory, many links would not work any more.

The fix is to go to phpbb_config table in the database, change script_path and cookie_path.

Thursday, June 19, 2014

VirtualBox Windows XP guest boot in safe mode


In this case, F8 won't work. fn+F8 won't work too since here it's a special function such as open itune. The method is:

With the XP guest running.
Go to Start, click on Run. In the box type msconfig and hit enter.
Click on the BOOT.INI tab, Check /SAFEBOOT save and restart.

Once GA are installed, reverse the above and reboot.

See here.

Mac memory management

Mac memory:

Total = free + wired + active + inactive
Used = wired + active + inactive

Wired memory are those that cannot be swapped to the hard drive.

Inactive memory are those no longer used and can be released using the "purge" command.

[1] Free Up Inactive Memory in Mac OS X with Purge Command
[2] What is "wired memory" in Activity Monitor?

Wednesday, June 18, 2014

Oracle Linux in VirtualBox

To install Oracle database via VirtualBox: How to install Oracle Database on Mac OS, or Evaluating Oracle Linux From Inside Oracle VM VirtualBox.

After get the Oracle Linux running, choose Network Adapter setting as "Host-only", then you can download Oracle SQL Developer - a free Oracle client IDE (like phpMyAdmin for MySQL), on your host machine and install it. You can then connect to the Oracle database running in the virtual machine. Note there is also an Oracle SQL Developer running inside the guest, you can copy its connection settings to the one on the host.

Then if on windows you may want to setup ODBC driver for Oracle. Download relevant installation package from here or here. Check this installation guide. The 3 steps are:

1) download and setup Install the ODAC or Oracle Client
2) Setup the SQLNET and TNSNAMES files. Usually SQLNET.ora file does not need change. Change TNSNAMES.ora, mostly this need to change the tnsname, host ip/name, port, and service_name.
3) Create the ODBC Data Source. Note here the "TNS Servie Name" is the same as in TNSNAMES.ora. When test the connection, the username and password are the same as in the property of connection in Oracle SQL Developer.

For Oracle/OL-SQL, see here for a manual.

To find out the version of your Oracle, use this command:
select * from v$version where banner like 'Oracle%';

Share files between OS: FAT, NTFS and exFAT file systems

I want to transfer large files on a USB drive and share on different OS.

Mac OS uses HFS file system. Windows uses FAT and NTFS, and server filesystems like XFS and ZFS. Linux uses Ext2/3/4.

FAT32 files system can share between Windows, Linux and Mac, but does not allow files larger than 4GB.

NTFS can share between Windows and Linux. Mac can read but not write to NTFS. But here is an article on How to manually enable NTFS read and write in OS X [6], or Enable writing to NTFS hard drives on Mac OS X [7] (use NTFS-3G, NTFS-3G patch, and OSXFuse).

The proprietary exFAT file system is designed for Flash drive from Microsoft since Vista SP1, can read/write for Windows and Mac, but not work well for linux [5]. exFAT is supported in Mac OS 10.6.x and later, and Windows Vista/7 and later, for XP there is driver available.

No silver bullet. Let me use exFAT for now to share between windows and mac, and if use linux, install driver [5]. I don't want to worry too much about Linux, since Linux users are supposed to be savvy, and Linux support for exFAT is coming in the open source community.

[1] Can Microsoft’s exFAT file system bridge the gap between OSes?
[2] exFAT Versus FAT32 Versus NTFS
[3] Understanding File-Size Limits on NTFS and FAT
[4] The best ways to format an external drive for Windows and Mac
[5] How to enable exFAT in Ubuntu
[6] How to manually enable NTFS read and write in OS X
[7] Enable writing to NTFS hard drives on Mac OS X
[8] wiki: ExFAT

Tuesday, June 17, 2014

The URL Lister extension of Firefox

What if you open too many tabs in firefox? It's necessary since these are all needed for fast reference. But Firefox memory usage is too high and slows down the machine.

A better way is to use a page to keep all these links. So instead of opening all these tabs, just keep the one page open, kind of like a bookmark feature.

This can be easily done by installing the URL lister extension of Firefox. Afterwards it's available in the Tools submenu.

MSSQL table row and column limit

The max number of columns allowed in MSSQL database is 1024. Wide table can contain up to 30000 columns, but it applies only to sparse columns, meaning most columns should be null and regular/calculated columns number is still 1024, and there is performance issue. This is from this article in MSDN.

Also bytes per row is limited to max 8060 (also from the link above, or here). varchar types can be longer but still will decrease performance. Since max length for any field is 8 bytes, this is 1024*8 = 8192, a little higher than 8060.


Sunday, June 15, 2014

VirtualBox cluster and Hadoop

This is to setup a VirtualBox cluster on Mac and then test Hadoop.

VirtualBox download
(How To Enlarge a Virtual Machine’s Disk in VirtualBox or VMware)
Ubuntu download (use server version)

Running Hadoop on Ubuntu Linux (Single-Node Cluster)
Setup Virtual Hadoop Cluster under Ubuntu with VirtualBox

iPhone development

Bought a book on Beginning iOS Programming, by Nick Harris. This is a very good book imo. It directs you through the entire process of building a Bands application, which stores your favorite music bands, and allows you to search online about band information, find nearby stores in map, and listen to preview samples of the band's tracks on iTunes, it also allows you to share the band's information with other people by email or messages. So by following the book, you instantly build a full functional iPhone application with rather complex functionalities. - I got through the book in 3 weeks, with this finished Bands application on my iPhone.

But when started trying the example project, I found it's using Xcode 5 for iOS 7. The book didn't even mention which version of Xcode it's using.

I'm still running Xcode 4.1 on OS X 10.7.5, a.k.a OS X Lion. 10.8 is Mountain Lion, and 10.9 is Mavericks which was released on 10/22/2013. Mavericks is the first free upgrade of OS X. However there are quite some security media coverage on it.

Without a time machine to backup Lion, should I give up upgrading to Mavericks? After reading many of the terrible feedback on line I don't want to upgrade for now.

Why these major software upgrade prematurely only to replace stable old version and ruin their reputation and cause damage to their users? There is a point that when cross it it's overdone.

Maybe I'll stop working on iOS programming now and just concentrate on Android development.

Or let me try virtual machine. VirtualBox is free on Mac. Try to install VirtualBox and see if I can install OS X 10.8 Mountain Lion  and then upgrade to Maverick. Note that Mountain Lion needs to be 10.8.4 or above to run Xcode, so if it's 10.8.3 then you need to find the upgrade. Yikes! It works!  So now I can run Xcode 5.1 in Mountain Lion from VirtualBox on my Lion iMac!

About sharing files between host and guest, it can use the shared folder option in VirtualBox guest OS setting. This needs to install GuestExtension on the guest machine. The GuestExtension is often included in the OS X installation kit, or can be downloaded separately from http://download.virtualbox.org/virtualbox/. However for Mac guest, there is no way to use shared folder this way, because GuestExtension is not available for Mac. So the alternative is to use file sharing on a network. For this, the guest machine should set network as "Bridged Network" instead of "NAT". Then the host and guest can share files on the local network. The file transfer speed can be as fast as hundreds of MB per second.

To remote desktop to PC from a mac, one needs to install Microsoft Remote Desktop Connection Client for Mac.

To install Mountain Lion on VMWare on PC seems hard. But use VirtualBox is fine.

Friday, June 13, 2014

Android App Development

7/12/2014

Open Android emulator and install apk in command line on a mac:

- Start avd manager:
In /Developer/Applications/Android/android-sdk-macosx/tools, type
android avd

- To install an apk
with the emulator running, go into ../platform-tools, type:
./adb install ~/Downloads/myapp.apk


7/3/2014

Now put the Birthday Calculator project on repository, and created the first release.

The App is built with the setting of Android SDK min version 11 and target version 19. In other words, to install this App, you should have Android 3.0 (Honeycomb, see Wiki: Android version history) or above to be safely compatible.


6/26/2014

- After 3 weeks, at the same time I read Nick Harris's Beginning iOS Programming and finished a Bands App on my iPhone. In comparison the "Android Essentials" book did not impress me. It is hard to read and too crude to make much sense. Anyway, it was published back in 2008, so let's don't push too much expectation on it. The only thing "good" is that it's a thin book. Actually, there are tutorials on line. Once got through several prototype examples, one will understand the basic building blocks and underlying framework, then pick a better book becomes easy.


6/13/2014

- Was able to run example projects, and create empty Android project.
- downloaded book code from http://www.apress.com/9781430210641
  for book Android Essentials, by Chris Haseman. QA76.76 A65 J37 2008
- Android emulator runs too slow. Follow this to make it faster:
  http://stackoverflow.com/questions/1554099/why-is-the-android-emulator-so-slow
  - save this html page as "performance - Why is the Android emulator so slow.pdf"
  Basically:
  Use 2000 MiB for SD Card. Enable Snapshot. Use 1024 for device ram size.
  In AVD manager, Start.. -> check Launch from snapshot and Save to snapshot
  This seems faster but not good enough. So try the next: HAMX acceleration.
- My CPU: Intel Xeon CPU W3530. Support vt, from
  http://ark.intel.com/Products/VirtualizationTechnology
  Tried the one using HAMX acceleration. It's much faster now!
  -> The HAEM.exe is at D:\Android\android-sdk\extras\intel\Hardware_Accelerated_Execution_Manager
- Also found that, once the emulator is on, open new Android app
  does not need to restart the emulator.
- Another online tutorial:
  https://developer.android.com/training/basics/firstapp/index.html
  - this provide info on doing this from command line, v.s. using IDE such as eclipse.
- Same installation procedures work on both Windows and Mac.
  Environment set up on both Windows and Mac.
    

6/12/2014

- Install Eclipse from eclipse.org/downloads/. Use JEE version.
  - 32-bit version does not work.
  - 64-bit version works.
  - tried to create the Hello World application and works.

- Install Android SDK
  - from https://developer.android.com/sdk/index.html
  - run it. Install to D:\Android\android-sdk

- Install Eclipse Plug-In of Android. ADT: Android Developer Tools.
  - From https://developer.android.com/sdk/index.html
    Choose: USE AN EXISTING IDE. It is on page:
    https://developer.android.com/sdk/installing/index.html


6/11/2014

- Borrowed book "Android Essentials" by Chris Haseman from library. It's on web at here.


References:
[1] Develop Apps | Android Developers Homepage
[2] Android ActionBar完全解析,使用官方推荐的最佳导航栏

Blog Archive

Followers