A blog on software engineering and project management

A blog on software engineering and project management: http://michaelochurch.wordpress.com/
Teach Yourself Programming in Ten Years - by Peter Norvig.

Where the internet lives

Google releases photos of its data centers. Data centers homepage

"using" keyword in C#, and large file processing

1. Using

The "using" keywork in C# is used to either import a library, or to cause a local variable to be disposed immediately after use.

To design a class that can be used in the using(...) clause, the class needs to implemented the IDisposable interface. This mostly means to implement the Dispose() and Dispose(boolean) methods, and deallocate local resources in the Dispose(boolean) method. See http://msdn.microsoft.com/en-us/library/system.idisposable.aspx.

2. Processing large data file

Processing of large data file may run out of memory if everything is done inside memory, for example, XmlSerializer may do this. The solution is to do the processing chunk by chunk (e.g., line by line, or block by block if no line separator).

For example, processing a file of 13GB will exhaust almost 16GB memory, causes the machine to hang for 30 minutes and fail. Using line by line processing, it takes 15 minutes and works successfully. Of course, for line by line processing, output can use buffering to avoid too many IO which also can be slow.

Another example is when reading a large file, in C/C++, read by line is much faster than read by char. But, for a binary file, you will not be able to read by line.

So processing large file requires careful handling of memory and IO.

Also, when a file is large, for example the 13GB file which does not contain any new line character (so read by line does not work), it can't be open by any common editor on windows including notepad, wordpad or VS.NET studio; it also can't be open on linux by vi. Well, when use vi to open it, it waits and seems there is never an end to the waiting. Search google shows that vi will have difficulty opening file with more than 9070000 character or file of size 2GB. Also for openning large file under 2GB, it will be faster for vi by disabling swap file, syntax parsing or undo history, see How to open big size file using vi editor or Faster loading of large files.

Use Perl to open the file also waits for ever. Actually using Perl it should also work if read as byte stream. Using C or Java to read as byte stream it works immediately.

Below is Java code to read as a byte stream:

import java.lang.*;
import java.io.*;

public class readByteFile {
  public static void main(String[] args) {
    InputStream is = null;
    ByteArrayOutputStream os = null;

    try  {
      File f = new File("filename");
      byte[] b = new byte[100];      // byte buffer.
      is = new FileInputStream(f);
      os = new ByteArrayOutputStream();

      int read = 0;
      int len = 0;
      while ( (read = is.read(b)) != -1 ) {
        os.write(b, 0, read);
        System.out.print(new String(b));
        len += read;
        if (len > 1000) break;
      }

      System.out.println(new String(b));
    } catch (Exception e) {

    } finally {
      try { if (os != null) os.close(); } catch (IOException e) {}
      try { if (is != null) is.close(); } catch (IOException e) {}
    }
  }
}

Or read in C using fgetc():

#include 

int main() {
  FILE * f = fopen("filename", "r");
  char ch;
  long ct = 0;      // char count.
  long line_ct = 0; // line count.

  if (f != NULL) {
    while (1) {
      ch = fgetc(f);
      ct ++;
      if (ch == '\r' || ch == '\n') line_ct ++;

      if (ch == EOF) break;

      putchar(ch);

      if (ct > 1000) break;
      if (line_ct > 5) break;
    }
  }
  printf("\n");
  fclose(f);
  return 0;
}

Tech Jobs Across America

Interesting article and statistics on tech job distribution across American.

A Chinese poem processor

A lot of language games can be done. I am also thinking of doing some Natural Language Processing stuff later.
The code below can be improved significantly in the data processing part. For example, on things such as 1) length of poem sentence, 2) position of target character, 3) tone (平仄), 4) semantic analysis so the mood matches.
More poem source files can be added.
The only tricky thing here so far is handling UTF8 characters.

#
# This script reads Chinese poems and store the sentences into a repository,
# then find poem sentences that start with letters in the given target sentence.
#
# This can be used for some language games, 
# such as forming a poem for somebody's birthday in the form of a "藏头诗".
#
# This script should be saved in utf8 format. 
#
# http://ahinea.com/en/tech/perl-unicode-struggle.html
# http://stackoverflow.com/questions/519309/how-do-i-read-utf-8-with-diamond-operator
# http://stackoverflow.com/questions/9574198/comparing-two-unicode-strings-with-perl
# 
# Chinese poems:
# http://www.shuku.net/novels/mulu/shici.html
#
# 藏头诗 generator: http://www.zhelizhao.com/cangtoushi/
#
# By: HomeTom
# Created on: 2012/09/04
#

require Encode;
use utf8;
use strict;

##################
# Change setting here.
##################

# Data source
my @files = ("tang300.txt", "song100.txt");
# Target sentence
my $target = ("小明生日快乐");

##################

my $cnt = 1;
my $len;
my @chars;
my @first = (); # first char.
#my $char;
my $DEBUG1 = 0;
my $DEBUG2 = 0;
#print "hi\n";

#binmode STDIN, ":utf8";
binmode STDOUT, ":utf8";

##################
# Read data.
##################

my @lines = ();
my @lines2;

foreach my $file (@files) {
  open FILE, $file or die $!;
  @lines2 = <FILE>;
  close FILE;
  push(@lines, @lines2);
}

##################
# Process and analyze data.
##################

foreach my $line (@lines) {
  chomp($line);
  $line = trim($line);
  $line = Encode::decode_utf8($line);
  @chars = split //, $line;

  if ($DEBUG1) { 
    print "$cnt: $line\n"; 
    print "$cnt: "; 
    print "$_." foreach (@chars); 
    print "\n"; 
  }

  #if ($chars[0] ne "") 
  {
    push (@first, $chars[0]);
  }

  $cnt ++;
}

if ($DEBUG2) {
  print "$_\n" foreach (@first);
}

$len = @first;


##################
# Search poem sentences for each word in the target sentence.
##################

my @words = split //, $target;

foreach my $w (@words) {
  print "==$w==\n";
  getLetterLines($w);
}


1;


##################
# Subroutines
##################

sub getLetterLines() {
  my $i;
  my $c;
  my $letter = shift;
  for ($i = 0; $i < $len; $i ++) {
    $c = $first[$i];
    if ($c eq $letter) { print "$i: $lines[$i]\n"; }
  }
  #print "\n";
}


#
# Trim functions, from: http://www.somacon.com/p114.php
#

# Perl trim function to remove whitespace from the start and end of the string
sub trim($)
{
  my $string = shift;
  $string =~ s/^\s+//;
  $string =~ s/\s+$//;
  return $string;
}
# Left trim function to remove leading whitespace
sub ltrim($)
{
  my $string = shift;
  $string =~ s/^\s+//;
  return $string;
}
# Right trim function to remove trailing whitespace
sub rtrim($)
{
  my $string = shift;
  $string =~ s/\s+$//;
  return $string;
}

2001 A Space Odyssey

2001: A Space Odyssey (film).

This is the 1968 sci-fi film made by Stanley Kubrick and Arthur Clarke, famous for its pioneering depiction of technology, artificial intelligence, anthropology and human evolution etc. The supercomputer HAL 9000 is a famous figure of AI, which possess not just strong computational power, but also human-like emotions.

The story is centered around a sentinel rock found on Moon, which sent signal to Jupiter. A mission group is sent from early to investigate the signal destination on Jupiter. Of the crew of 5 humans and HAL, only HAL knew this true purpose. HAL killed 4 crew members in fear that they would shut him down and interfere with the trip mission. One crew survived, shut down HAL, and arrived at the Jupiter.

In my opinion, the 1968 movie is superb in its whimsical sense of sci-fi thoughts. The scenes setup both internally inside spacecraft and externally in the universe are still convincing today. But lack of dialogue and slow pace of storyline may not be liked by some people.

There are 4 episodes of the Space Odyssey series, published as 4 books. The first 2 were made into movies in 1968 (2001 A Space Odyssey) and 1984 (2010 Space Odyssey II). The later two are set in years 2061 and 3001. The 1968 movie is regarded as among the best movies in history. Before these 4 books, Clarke published a short story "The Sentinel" (PDF) in 1951, which he later expanded into the "2001 A Space Odyssey" in 1964, and further rewritten as a movie.

Arthur Clarke's books on the Space Odyssey series can be found here.

The magic squares

The matlab7 book I'm reading uses Durer's matrix as an example. This leads to my finding of the wiki page on magic squares. Sleek, or sick?

Another unrelated but similarly magical description on numbers encrypted in architectures: Secrets In Plain Sight - Art, Architecture & Urban Design, and here on youtube.