A Python daemon

A simple unix/linux daemon in Python. This is for Python 2.x. The article also contains a link for Python 3.x (link is here).

The python-daemon library is another solution.

A Perl HTTP Web Server - Full source code

This is the full source code of the HTTP web server written in Perl we discussed in the previous post.

If you visit a page /index.html which contains an image tag <img src="bg.jpg">, the log file of this HTTP web server will be something like:

2014-7-3 3:47:28  server is started

2014-7-3 3:47:28  Listening on [IO::Socket::INET=GLOB(0x7f8bd98bd9f0)] 0.0.0.0:9000 ...

2014-7-3 3:52:40  == incoming connection from [IO::Socket::INET=GLOB(0x7f8bd98bd9f0)] 127.0.0.1:51659 ...

2014-7-3 3:52:40  request file: /index.html

2014-7-3 3:52:40  == incoming connection from [IO::Socket::INET=GLOB(0x7f8bd98bd9f0)] 127.0.0.1:51661 ...

2014-7-3 3:52:40  request file: /bg.jpg

Some notes:
- Here LocalAddr should be "0.0.0.0" instead of "localhost" or "127.0.0.1" if you want remote computers to be able to access it. If it's "localhost" or "127.0.0.1" then you can access it only from local machine.
- Besides using a browser, you can also access the web server using telnet: telnet [ip] [port]
- Use this command to see open ports on local machine: netstat -an | grep "LISTEN"
- On Ubuntu, use this to add a chain rule to iptables firewall to open a port (e.g., see here): iptables -A INPUT -p tcp --dport 9000 -j ACCEPT
- On Mac, besides the firewall in System Preferences -> Security & Privacy -> Firewall, there is another deprecated firewall: ipfw. By default it opens all the ports though. To see its rules, use: sudo ipfw list.
- Here we use a buffer size of 1024 in the recv call. In reality, the GET request length is limited from 2K to 8K bytes for different browsers. For example, this source says: The limit (of HTTP GET Request) is in MSIE and Safari about 2KB, in Opera about 4KB and in Firefox about 8KB. We may thus assume that 8KB is the maximum possible length and that 2KB is a more affordable length to rely on at the server side and that 255 bytes is the safest length to assume that the entire URL will come in.

Note that since HTTP is a stateless protocol, we close the connection immediately after each time we send response. In reality, we may not be able to receive the entire request message in one read. But as a demonstration of concept this serves pretty well.

One may ask why recreate the wheel? The answer is, of course, we don't need another primitive HTTP web server, we already got so many. But if we need a service, which resides on a linux/unix/mac server and listens to a specific port, to accomplish a specific task, then we can easily write a server of our own this way using this template, using our own customized protocol.  Another situation is that if for some reason you don't have a web server, then you can use this template to build one. Of course, this later case rarely happens today.

As for the applicability, this Perl server runs well on linux/unix/mac. For windows, it is easy to build something similar as a windows service, such as a .NET remoting TCP server, to listen on a specific port and accomplish similar tasks.


#
# This script demonstrates a functional HTTP web server in Perl:
# 1) running the Perl HTTP web server as a daemon in background.
# 2) only one copy of the server can run by checking "Proc::PID::File->running()".
# 3) implementation of daemon commands: start, stop, status.
#    e.g. start the web server by: sudo perl dmon_server.pl start.
#
# Note: 
# 1) to run as daemon, "sudo" should be used for non-admin user.
# 2) parameters that can change: $LOG_FILE, $WWWROOT, $USE_OPT,
#     and $localport in functoin do_start().
#
# @By: X.C.
# @Created on: 6/28/2014
# @Last modified: 7/2/2014
#

#!/usr/bin/perl

use strict;
use warnings;
use Getopt::Long;
use Proc::Daemon;
use Proc::PID::File;
use IO::Select;
use IO::Socket;
use URI::Escape;

#
# Location of log file.
#
my $LOG_FILE = "/Users/chenx/tmp/dmon.log";
#
# Location of web root.
#
my $WWWROOT = "/Users/chenx/tmp/wwwroot";

#
# If $USE_OPT = 1, use GetOptions. 
# 0 is better here because if an arg does not start with "--",
# it will be ignored and no usage information is printed.
#
my $USE_OPT = 0; 


my $len = @ARGV;
if ($len == 0) {
    show_usage();
} else {
    if ($USE_OPT) {
        GetOptions(
            "start" => \&do_start,
            "status" => \&show_status,
            "stop" => \&do_stop,
            "help" => \&show_usage
        ) or show_usage();
    } else {
        my $cmd = $ARGV[0];
        if ($cmd eq "start") { do_start(); }
        elsif ($cmd eq "stop") { do_stop(); }
        elsif ($cmd eq "status") { show_status(); }
        else { show_usage(); }
    }
}


#
# 1 at the end of a module means that the module returns true to use/require statements. 
# It can be used to tell if module initialization is successful. 
# Otherwise, use/require will fail.
#
# 1;


sub show_usage {
    if ($USE_OPT) {
        print "Usage: sudo perl $0 --[start|stop|status|help]\n";
    } else {
        print "Usage: sudo $0 [start|stop|status]\n";
    }
    exit(0);
}


sub show_status {
    if (Proc::PID::File->running()) {
        print "daemon is running..\n";
    } else {
        print "daemon is stopped\n";
    }
}


sub do_stop {
    my $pid = Proc::PID::File->running();
    if ($pid == 0) {
        print "daemon is not running\n";
    } else {
        #print "stop daemon now ..\n";
        kill(9, $pid);
        print "daemon is stopped\n";
    }
    do_log("server is stopped");
}


sub do_start {
    print "start daemon now\n";

    Proc::Daemon::Init();

    if (Proc::PID::File->running()) {
        do_log( "A copy of this daemon is already running, exit" );
        exit(0);
    }

    do_log("server is started");

    my ($data, $fh, $data_len); 
    my $localhost = "0.0.0.0";
    my $localport = 9000;
    my $ipc_select = IO::Select->new();
    my $IPC_SOCKET = new IO::Socket::INET(
             Listen  => 5, LocalAddr => $localhost, LocalPort => $localport, Proto => "tcp" );

    $ipc_select->add($IPC_SOCKET);
    do_log( "Listening on [$IPC_SOCKET] $localhost:$localport ..." );
    while (1) {
      if (my @ready = $ipc_select->can_read(.01)) {
        foreach $fh (@ready) {
            if($fh == $IPC_SOCKET) {
                my $new = $IPC_SOCKET->accept;
                $ipc_select->add($new);
                do_log( "== incoming connection from [$fh] " . 
                        $new->peerhost() . ":" . $new->peerport() . " ...");
            } else {
                recv($fh, $data, 1024, 0);
                my $data_len = length($data);
                if ($data_len > 0) { # feedback to client.
                    print $fh http_response($data);
                }
                $ipc_select->remove($fh);
                $fh->close;
            }
        }
      }
    }
}



#
# Implements part of the HTTP protocal:
# - GET command
# - Status code: 200, 400, 404, 500
# - Customized command TEST, with status code -1.
#
# Reference: http://www.w3.org/Protocols/rfc2616/rfc2616.html
#
sub http_response {
    my ($request) = @_;

    my $status = 0;
    my $data = "";
    if ($request =~ m/^GET\s(\S+)\s/) {
        do_log("request file: $1");
        my $file = ($1 eq "/") ? "/index.html" : $1;  # default file under a directory.
        $file = uri_unescape($file); # url_decode() function. E.g. change %20 back to space.

        my $path = "$WWWROOT$file";
        if (-e $path) {
           my $open_ok = 1;
           open my $fh, '<', $path or $open_ok = 0; # die "error opening $path: $!";
           if ($open_ok == 1) {
               $data = do { local $/ = undef; <$fh> };
               $status = 200;
           } else {
               $data = "Internal Server Error";
               $status = 500;
           }
        }
        else { # file not found.
           $data = "Not Found";
           $status = 404;
        }
    } elsif ($request =~ m/^TEST\s/) {
        $status = -1; # for testing purpose of the Perl Web Server.
    } else { # bad request: unknown command.
        $data = "Bad Request";
        $status = 400;
    }


    my $response = "";
    if ($status == 200) {
        my $data_len = length($data);
        $response = "HTTP/1.1 200 OK\nContent-Type:text\nContent-Length:$data_len\n\n$data";
    }
    elsif ($status == 400 || $status == 404 || $status == 500) {
        my $body = "";
        my $data_len = length($body);
        $response = "HTTP/1.1 $status $data\nContent-Type:text\nContent-Length:$data_len\n\n$body";
    }
    else { # status == -1
        my $hdr = "PERL Web Server Received:\n";
        my $data_len = length($hdr) + length($request);
        $response = "HTTP/1.1 200 OK\nContent-Type:text\nContent-Length:$data_len\n\n$hdr$request";
    }

    return $response;
}


sub do_log {
    my ($msg) = @_;

    my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime();
    $year += 1900;
    $mon += 1;

    open FILE, ">>$LOG_FILE" or die "cannot open log file $!\n";
    print FILE "$year-$mon-$mday $hour:$min:$sec  $msg\n";
    close FILE;
}

A Perl HTTP Web Server

This is a simple server written in Perl. It listens on a port, but does not follow the HTTP protocol. It just echoes whatever the client sent in.

Note for examples in this post the value of "LocalAddr" is "localhost", so the servers here are accessible only on local machine. If you want access from a remote machine, you should change it to "0.0.0.0".


# This demonstrates a simple Perl HTTP server.
# A user can connect using: telnet [host] [port], here port = 9000.
# The user will say something, which the server repeats.
# When the user says "bye", the connection is closed.
#
# Modified from: http://www.perlmonks.org/?node_id=49823
# By: X.C.
# Created on: 6/28/2014
# Last modified: 6/28/2014
#
#!/usr/bin/perl -w
use strict;
use IO::Select;
use IO::Socket;

my ($data, $fh);
my $ipc_select = IO::Select->new();
my $IPC_SOCKET = new IO::Socket::INET(Listen    => 5,
                                    LocalAddr => 'localhost',
                                    LocalPort => 9000,
                                    Proto   => "tcp" );


$ipc_select->add($IPC_SOCKET);
print "Listening on Socket [$IPC_SOCKET] ...\n";
while (1) {
    if (my @ready = $ipc_select->can_read(.01)) {
        foreach $fh (@ready) {
            if($fh == $IPC_SOCKET) {
                #add incoming socket to select
                my $new = $IPC_SOCKET->accept;
                $ipc_select->add($new);
                print "== incoming connection...\n";
            } else {
                # Process socket
                my $n = recv($fh, $data, 1024, 0); # this seems to be empty.
                my $data_len = length($data); # $data ends with "\r\n".
                #print "data len:" . length($data) . "\n";
                if ($n || $data_len > 0) {
                    print $fh "Server feedback: $data"; # feedback to client.
                    $data = substr($data, 0, $data_len - 2);
                    print "incoming data: $data\n";
                    #chomp($data); # this won't work, since $data is a buffer not filled with 0s.
                    if (uc($data) eq "BYE") {
                        print "== close connection\n";
                        $ipc_select->remove($fh);
                        $fh->close;
                    }
                } else { # seems this won't get executed ever.
                    $ipc_select->remove($fh);
                    $fh->close;
                }
            }
        }
    }
}

Or it can be simplified into the code below. Now this uses standard HTTP response and is a HTTP web server, so you can actually visit it from a browser using url http://localhost:9000/.


#!/usr/bin/perl -w
use strict;
use IO::Select;
use IO::Socket;

my ($data, $fh);
my $ipc_select = IO::Select->new();
my $IPC_SOCKET = new IO::Socket::INET(
    Listen  => 5, LocalAddr => 'localhost', LocalPort => 9000, Proto => "tcp" );

$ipc_select->add($IPC_SOCKET);
print "Listening on Socket [$IPC_SOCKET] ...\n";
while (1) {
    if (my @ready = $ipc_select->can_read(.01)) {
        foreach $fh (@ready) {
            if($fh == $IPC_SOCKET) {
                my $new = $IPC_SOCKET->accept;
                $ipc_select->add($new);
                print "== incoming connection from [$fh]...\n";
            } else {
                recv($fh, $data, 1024, 0); 

                my $data_len = length($data);
                if ($data_len > 0 && uc($data) ne "BYE\r\n") { # feedback to client.
                    print $fh "HTTP/1.0 200 OK\nContent-Type:text\nContent-Length:$data_len\n\n$data"; 
                } else { 
                    $ipc_select->remove($fh);
                    $fh->close;
                }
            }
        }
    }
}

So far this is run in a console, and is tied to the controlling console. If you type CTRL-C then it's stopped, or when the console is closed it's gone.  However, by combining this with the Perl daemon we discussed in the previous post, we obtain a full-fledged web server written in Perl! This web server runs as a daemon process. You can visit it in a web browser using http://localhost:9000. So far the only thing this web server does is to prepend the client request with "PERL Web Server Received:" and send it back to the client.

To be exact, the only modifications needed to the Perl daemon in the previous post are:

1) add this to the top of file:

use IO::Select;
use IO::Socket;


2) replace the do_start() function with:

sub do_start {
    print "start daemon now\n";

    Proc::Daemon::Init();

    if (Proc::PID::File->running()) {
        do_log( "A copy of this daemon is already running, exit" );
        exit(0);
    }

    my ($data, $fh, $data_len);
    my $ipc_select = IO::Select->new();
    my $IPC_SOCKET = new IO::Socket::INET(
        Listen  => 5, LocalAddr => 'localhost', LocalPort => 9000, Proto => "tcp" );

    $ipc_select->add($IPC_SOCKET);
    do_log( "Listening on Socket [$IPC_SOCKET] ..." );
    while (1) {
      if (my @ready = $ipc_select->can_read(.01)) {
        foreach $fh (@ready) {
            if($fh == $IPC_SOCKET) {
                my $new = $IPC_SOCKET->accept;
                $ipc_select->add($new);
                do_log( "== incoming connection from [$fh]..." );
            } else {
                recv($fh, $data, 1024, 0);
                my $data_len = length($data);
                if ($data_len > 0) { # feedback to client.
                    print $fh http_response($data);
                }
                $ipc_select->remove($fh);
                $fh->close;
            }
        }
      }
    }
}

sub http_response {
    my ($request) = @_;
    my $hdr = "PERL Web Server Received:\n";
    my $data_len = length($hdr) + length($request);

    return "HTTP/1.0 200 OK\nContent-Type:text\nContent-Length:$data_len\n\n$hdr$request";
}

A Perl daemon

A daemon is a process running in the background continuously. It is the linux comparable of a windows service. A daemon or windows service can do many things in the background. If you let it open and listen to a port, it becomes a server, e.g., a web server like Apache or IIS, or a database server like MySQL or MongoDB. For this reason, it is a very interesting and useful thing to learn how to write a daemon or windows service. The installation and/or execution of a linux daemon or windows service usually requires the user to have admin permission.

A Perl daemon can be easily implemented using the Proc::Daemon module [1]. If you don't have this installed, you can do it by:

sudo perl -MCPAN -e shell
cpan[1]> install Proc::Daemon
cpan[2]> install Proc::PID::File

A Perl daemon can be as simple as this (note you will need to run it with "sudo"):

#!/usr/bin/perl

use Proc::Daemon;

use Proc::PID::File;

Proc::Daemon::Init; # Demonize.

if (Proc::PID::File->running()) { exit(0); } # Exit if already running, so only 1 instance can run.

for (;;) { 
    # print "do something every 5 seconds..\n"; # This won't print because STDOUT is closed.
    sleep(5); 
}

You cannot really see any console print output since STDOUT is closed by Proc::Daemon. You can see output by printing to a log file, or use the "top" command to see this new process.

Here is a more full-fledged version of a Perl daemon. It allows you to use start/stop/status commands to control and monitor the daemon. A log file is also generated.


#
# This script demonstates:
# 1) running a Perl script as daemon in background, using the Daemon module.
# 2) only one instance of the daemon can run by checking "Proc::PID::File->running()".
# 3) implementation of daemon commands: start, stop, status.
# 4) use of Getopt::Long, $0 (name of this file).
#
# Note:
# 1) to run as daemon, "sudo" should be used for non-admin user.
# 2) parameters that can change: $LOG_FILE, $USE_OPT.
#
# @By: X.C.
# @Created on: 6/28/2014
# @Last modified: 6/28/2014
#

#!/usr/bin/perl

use strict;
use warnings;
use Getopt::Long;
use Proc::Daemon;
use Proc::PID::File;

#
# Location of log file.
#
my $LOG_FILE = "/Users/chenx/tmp/dmon.log";

#
# If $USE_OPT = 1, use GetOptions.
# 0 is better here because if an arg does not start with "--",
# it will be ignored and no usage information is printed.
#
my $USE_OPT = 0;


my $len = @ARGV;
if ($len == 0) {
    show_usage();
} else {
    if ($USE_OPT) {
        GetOptions(
            "start" => \&do_start,
            "status" => \&show_status,
            "stop" => \&do_stop,
            "help" => \&show_usage
        ) or show_usage();
    } else {
        my $cmd = $ARGV[0];
        if ($cmd eq "start") { do_start(); }
        elsif ($cmd eq "stop") { do_stop(); }
        elsif ($cmd eq "status") { show_status(); }
        else { show_usage(); }
    }
}


#
# 1 at the end of a module means that the module returns true to use/require statements.
# It can be used to tell if module initialization is successful.
# Otherwise, use/require will fail.
#
# 1;


sub show_usage {
    if ($USE_OPT) {
        print "Usage: sudo perl $0 --[start|stop|status|help]\n";
    } else {
        print "Usage: sudo $0 [start|stop|status]\n";
    }
    exit(0);

}

sub show_status {

    if (Proc::PID::File->running()) {

        print "daemon is running..\n";

    } else {

        print "daemon is stopped\n";

    }

}

sub do_stop {

    my $pid = Proc::PID::File->running();

    if ($pid == 0) {

        print "daemon is not running\n";

    } else {

        #print "stop daemon now ..\n";

        kill(9, $pid);

        print "daemon is stopped\n";

    }

}

sub do_start {

    print "start daemon now\n";

    Proc::Daemon::Init();

    #

    # To use this, you need to start the daemon with "sudo" to have the permission

    # to PID file. To kill the daemon, "sudo" is also needed.

    #

    if (Proc::PID::File->running()) {

        do_log( "A copy of this daemon is already running, exit" );

        exit(0);

    }

    my $continue = 1;

    $SIG{TERM} = sub { $continue = 0; };

    while ($continue) {

        #print "continue ..\n"; # this won't work, since STDOUT is closed.

        do_log("continuing ..");

        sleep(2);

    }

}

sub do_log {

    my ($msg) = @_;

    my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime();
    $year += 1900;

    open FILE, ">>$LOG_FILE" or die "cannot open log file $!\n";
    print FILE "$year-$mon-$mday $hour:$min:$sec  $msg\n";
    close FILE;

}

Alternatively, you can also implement a daemon without using Proc::Daemon.  An example is below.

This code [2] is back from year 2000. The current Proc::Daemon implements majority of these ideas. These are also the basic requirements of a well-rounded daemon, so it won't become a demon:

1) Fork a child process as the daemon, the exit the parent process. This guarantees the daemon process is free of any controlling process and terminal.

2) Call setsid. This makes the daemon process a session leader, and free of any controlling terminal/shell.

3) Change working dir, usually "/". This prevents the relevant partition where the working dir resides from resisting unmounting by admin.

4) Use umask to cancel the default file creation mode permission inherited from parent process.

5) Close unneeded file handlers, mostly STDIN, STDOUT and STDERR, by redirecting them to /dev/null. This is because a daemon has no associated terminal/shell and has nowhere to write these to.

6) Logging message, so you know what's going on when STDOUT and STDERR are closed.

#!/usr/bin/perl

#

# This demonstrates running a Perl program in background as daemon without using Proc::Daemon.

# More than 1 instance can run at the same time.

# From: [2] 

#

use POSIX qw(setsid);

chdir '/';

umask 0;

open STDIN, '/dev/null';

#open STDOUT, '>/Users/chenx/tmp/dmon.log';

open STDERR, '>/dev/null';

defined(my $pid = fork);

exit if $pid;

setsid;

while(1)

{

    sleep(2);

    #print "Hello...\n";

    do_log();

}

# note: cannot use "log" as it's preserved.

sub do_log {

    open LOGFILE, '>>', '/Users/chenx/tmp/dmon.log' or die "cannot open file"; # $!;

    print LOGFILE "continue on ..\n";

    close LOGFILE;

}

There is an interesting Perl Daemon Contest [3] here, back in 2000. Some of these are pretty complicated and highly practical, such as web proxy, system monitoring and reporting, stock monitoring, even a reversed-engineered printer daemon, and an interesting AI game of rock/scissors/paper played inside the file system.


References:

[1] Proc::Daemon implementation version 0.14
[2] Unix Daemon In Perl Tutorial
[3] Perl Daemon Contest

Trying out mongoDB

Today I'm trying out mongoDB. MongoDB is an open source NoSQL document database.

Important resources:

http://www.mongodb.org - for any user to get to know it, and download
http://www.mongodb.org/downloads - download, manual (in html and pdf) is at the end of this page
http://docs.mongodb.org/ecosystem/drivers/ - mongoDB drivers for all languages
http://docs.mongodb.org/manual/reference/sql-comparison
http://www.mongodb.com - for enterprise users. Enterprise MongoDB Architecture.

Using MongoDB with ASP.NET Web API - need to install a mongoDB driver for .NET
Create a C# ASP.NET Application on Azure with MongoDB using the MongoLab Add-On

PHP manual on mongoDB - from installation to usage

Introduction to MongoDB for Java, PHP and Python Developers (May, 2012)

MongoDB partly obtain its performance from using memory mapped file. It supports easy replica for high up time. It keeps most important contents in memory and horizontally scale, shard automatically. It uses dynamic schema, and supports many programming languages and platforms. Its customers often have a cluster with over 100 nodes, some over 1000 nodes; data size ranges from TB to PB.

MongoDB installation package is self-contained and quite clean, in that it's a "green" software. The console is like MySQL or Python console in commands, and uses JavaScript as the query language. The comparable to MSSQL is: database - database, table - collection, row - document, column - field, index - index, table joins - embedded documents and linking, aggregate function - aggregation pipeline.

Installation on windows is recommended on Windows Server 2008 or above. Windows XP is not supported in MongoDB version 2.2 or above. Install as a windows service is easy. MongoVUE is a windows client, similar to PhpMyAdmin (actually, a web client instead of windows client) for MySql.

Convert dmg file to iso on windows

To convert dmg file to iso on windows, tools for this include:

PowerISO (commercial), dmg2img (GPL, how to. easy to use).

Move PhpBB from root to subdirectory

When move a PhpBB forum from domain root to a subdirectory, many links would not work any more.

The fix is to go to phpbb_config table in the database, change script_path and cookie_path.