Setup email forwarding in linux/unix

Email forwarding can be setup in an email client such as Thunderbird.

Under linux/unix, it can done by admin or by user himself, as explained here.

To do it by the user himself, he only needs to create a file .forward in his account root, and enter the forward email addresses separated by comma or new line. The .forward file should have permission 644.

An example .forward file is:

"|/usr/local/bin/procmail"
example@gmail.com

Java becomes closed source

December 9, 2010: The ASF Resigns From the JCP Executive Committee.

It seems that now Java to Oracle is as C# to Microsoft. Java becomes proprietary language of Oracle. But, C/C++/PHP/Perl/Python/Ruby are still open source.

Since Oracle also owns MySQL, don't know what will happen to MySQL. PostgreSQL is still open source though.

Silicon valley

Here is an introduction to silicon valley.

SQL Injection Attack

One of the servers is hacked. The symptom is that javascript code were inserted into some database tables. When users visit the site, the javascript code would invoke remote site scripts, display a faked virus scan and report (might be a dynamic gif image), and ask the user to download and install a virus removal software. Once the user installs the software, his computer will be infected.

A script is written to search and clean the database based on signature (characteristic substring) in the javascript code. SQL Query log is added to record all the executed queries. Nothing was found. The problem was finally identified by checking the web server visit log. There are visit requests where SQL commands such as UPDATE used in query string parameter value. It takes advantage of the fact that two consecutive SQL statements can be executed one after the other. This SQL injection attack achieved its goal. Checking the request parameter before executing the SQL can catch such problems. Use of stored procedure and avoid this vulnerability.

A perl script is written to analyze the visit log files and extract these attack requests. The attacked page and attacker's IP is found. Whois service is used to locate where the attacker's IP is from. Patches were made to the attacked page and the site.

Below is the perl script to analyze web server visit log. It extract queries using the 'update' command. Actually it's found that command 'select' is also used in other queries. Can change the script to extract those as well.

#!/usr/bin/perl

#
# @Author: Tom
# @Created on: 9/30/2010
# @Last modified: 9/30/2010
# @Usage example: perl Analyzer.pl < ex100920.log > 100920.txt
# All lines containing the substring "1091+update" are extracted.
#

use strict;

my @fields;
my $fields_ct;
my $line_ct = 0;
my $attack_line_ct = 0;
my @attacked_Pages;
my @attack_IPs;
my $v;
my @vs;
my $i;
my $prefix;

my @ascii_table = (
"NUL",
"SOH",
"STX",
"ETX",
"EOT",
"ENQ",
"ACK",
"BEL",
"BS",
"HT",
"LF",
"VT",
"FF",
"CR",
"SO",
"SI",
"DLE",
"DC1",
"DC2",
"DC3",
"DC4",
"NAK",
"SYN",
"ETB",
"CAN",
"EM",
"SUB",
"ESC",
"FS",
"GS",
"RS",
"US",
" ",
"!",
"\"",
"#",
"\$",
"%",
"&",
"'",
"(",
")",
"*",
"+",
",",
"-",
".",
"/",
"0",
"1",
"2",
"3",
"4",
"5",
"6",
"7",
"8",
"9",
":",
";",
"<",
"=",
">",
"?",
"@",
"A",
"B",
"C",
"D",
"E",
"F",
"G",
"H",
"I",
"J",
"K",
"L",
"M",
"N",
"O",
"P",
"Q",
"R",
"S",
"T",
"U",
"V",
"W",
"X",
"Y",
"Z",
"[",
"\\",
"]",
"^",
"_",
"`",
"a",
"b",
"c",
"d",
"e",
"f",
"g",
"h",
"i",
"j",
"k",
"l",
"m",
"n",
"o",
"p",
"q",
"r",
"s",
"t",
"u",
"v",
"w",
"x",
"y",
"z",
"{",
"|",
"}",
"~",
"DEL",
);

while(<>) {
  $line_ct ++;
  if (/^\s+$/) { next; } # ignore empty line.
  if (/id=1091\+update/) {} else { next; } # ignore non-attack lines.
  $attack_line_ct ++;

  print $line_ct . ": \n" ;
  print $_ . "\n"; 
  print "Fields dump: \n";
  @fields = split(' ', $_);
  $fields_ct = @fields;
  for ($i = 0; $i < $fields_ct; $i ++) {
    print "$i: ";
    $v = $fields[$i];
    if ($i == 6) {
      $v =~ s/\%2B/+/g;
      #print "$v\n";
      @vs = split('\+', $v);
      foreach my $u (@vs) {
        if ($u =~ /^varchar\(8000\)/) {
          print "$u";
        } elsif ($u =~ /(cast\()(char\()((\d)+)\)/) {
          print "cast(" . $ascii_table[$3]; 
        } elsif ($u =~ /(char\()((\d)+)\)/) {
          print $ascii_table[$2]; 
        } else {
          print "$u";
        }
        #print "+";
        print " ";
      }
      print "\n";
    } elsif ($i == 5) {
      if (Page_Exists($v) == 0) {
        push(@attacked_Pages, $v);
      }
      print "$v\n";
    } elsif ($i == 9) {
      if (IP_Exists($v) == 0) {
        push(@attack_IPs, $v);
      }
      print "$v\n";
    } else {
      print "$v\n";
    }
  }
  print "\n";
}

print "$attack_line_ct attack requests found.\n";

my $Page_ct = @attacked_Pages;
print "$Page_ct attacked Page(s) found: \n";
foreach my $p (@attacked_Pages) {
  print "$p\n";
}

my $IP_ct = @attack_IPs;
print "$IP_ct attacking IP(s) found: \n";
foreach my $p (@attack_IPs) {
  print "$p\n";
}


sub Page_Exists() {
  my ($ip) = @_;
  foreach my $p (@attacked_Pages) {
    if ($ip eq $p) { return 1; }
  }
  return 0;
}

sub IP_Exists() {
  my ($ip) = @_;
  foreach my $p (@attack_IPs) {
    if ($ip eq $p) { return 1; }
  }
  return 0;
}

For cleaning the database, an ASP script is written.

'
' Remove injected code from database tables.
' Use signature and full_signature to specify the injected code.
' Use request string "doClean=y" to do cleaning.
' If not use doClean, will only show infected rows.
' Tom 9-24-2010
'
Dim signature, full_signature
signature = "</script>" 
full_signature = "<script src=http://.../...js></script>"

Dim doClean
doClean = false
if request("doClean") <> "" then doClean = true

Response.Write("<p>Use doClean request parameter to do cleaning. ")
Response.Write("doClean = " & doClean & "</p>")
call getTables()

function getTables()
    'Response.Write("test()<br>")
    Dim db, rs, sql, count, tbl, infectedTblCount
    infectedTblCount = 0

    sql = "select table_name as Name from INFORMATION_SCHEMA.Tables where TABLE_TYPE ='BASE TABLE'"
    set db = Connect()
    set rs = ExecuteRS(db, sql)
    
    count = 1
    do while not rs.eof
        tbl = rs("Name")
            Response.Write("Table " & count & ". " & tbl & "<br>")
            if getTableColumns(tbl) > 0 then infectedTblCount = infectedTblCount + 1
            count = count + 1
        rs.MoveNext()
    loop
    
    Response.Write("<p>" & infectedTblCount & " tables are infected.</p>")

    call rs.close()
    set rs = nothing
    call db.close()
    set db = nothing    
end function


function getTableColumns(tbl)
    Dim db, rs, sql, col, str, chk

    sql = "select column_name as Name from INFORMATION_SCHEMA.COLUMNS where TABLE_name ='" & tbl & "'"
    set db = Connect()
    set rs = ExecuteRS(db, sql)
    
    count = 0 ' count of infected columns.
    
    do while not rs.eof
        col = rs("Name")
        chk = checkTblColumn(tbl, col)
        str = str & "<li>" & col & chk & "</li>"
        if len(chk) > 0 then 
            count = count + 1
            if doClean then str = str & cleanTblColumn(tbl, col)
        end if
        rs.MoveNext()
    loop
    str = "<ol>" & str & "</ol>"
    Response.Write(str)
    
    getTableColumns = count

    call rs.close()
    set rs = nothing
    call db.close()
    set db = nothing
end function


function checkTblColumn(tbl, col)
    Dim db, rs, sql, val, str

    sql = "select " & col & " as Name from " & tbl
    set db = Connect()
    set rs = ExecuteRS(db, sql)
    
    count = 0   
    str = ""
    do while not rs.eof
        val = rs("Name")
        if InStr(1, val, signature) > 0 then
            str = str & ("<li>Infected row: " & encodeStr(val) & "</li>")
            count = count + 1
        end if
        rs.MoveNext()
    loop
    if count > 0 then
        str = "<font color='red'>" & count & " rows infected</font>" & str
        str = "<ol>" & str & "</ol>"
        'Response.Write(str)
        checkTblColumn = str
    else
        checkTblColumn = ""
    end if

    call rs.close()
    set rs = nothing
    call db.close()
    set db = nothing
end function


function cleanTblColumn(tbl, col)
    Dim db, rs, sql, val, str

    set db = Connect()
    set rs = Server.CreateObject("ADODB.Recordset")
    rs.Open tbl, db, 1, 2, adCmdTableDirect
    
    count = 0   
    str = ""
    do while not rs.eof
        val = rs(col)
        if InStr(1, val, signature) > 0 then
            rs(col) = replace(val, full_signature, "")
            rs.Update()
            count = count + 1
        end if
        rs.MoveNext()
    loop
    if count > 0 then
        str = "<font color='red'>CLEANED " & count & " rows infected</font>" & str
        cleanTblColumn = str
    else
        cleanTblColumn = ""
    end if

    call rs.close()
    set rs = nothing
    call db.close()
    set db = nothing
end function


function encodeStr(s)
    s = replace(s, "<", "&lt;")
    s = replace(s, ">", "&gt;")
    encodeStr = s
end function

BTW, this is a wiki article on SQL injection. Search on google for SQL injection would bring up much more articles. Good and important to know for web developers, for security concern.

Book: Apache Jakarta and Beyond

Book: Apache Jakarta and Beyond - A Java programmer's introduction. By Larne Pekowsky. ISBN 0-321-23771-4 QA76.73.J38P44 2004. 2005.

This book introduces a series of Jakarta tools to be used by Java programmers. Including:

- Ant
- Eclipse
- Testing with JUnit
- Testing web sites with HTTPUnit
- Further web testing with Jakarta Cactus
- Stress Testing with Jakarta JMeter
- Simplifying Bean Development with BeanUtils
- Traversing Hierarchical Data with JXPath
- Database tools: 
  Hsqldb, DBCP, OJB
- Logging
  - Java.util.logging
  - Log4j
- Configuring program options
  - Jakarta CLI (Command-Line Interface)
  - Jakarta Digester (XML-based: object stack, element matching patterns, processing rules)
- Working with Text 1: Regular Expressions
- Working with Text 2: Searching  
- Creating office documents with POI
- Scripting
- Tomcat
- The standard tag library
- Struts: application toolkit/web application framework
- Cocoon: provides a complete XML-based publishing suite, for the generation, manipulation and rendering of XML.

Design Patterns

Design Patterns - Elements of Reusable Object-Oriented Software. This is a classical book. The authors won ACM 2010 SIGSOFT outstanding research award for their contribution to software engineering for. The four authors are classed the GoF (Gang of Four). The design patterns in their book is called the GoF patterns.

Design patterns are solutions abstracted from repeatedly occurring design problems and can be reused in similar situations. Each has a pattern name, associated problem, solution and consequence.

Some design patterns are bounded to languages features, so is easier to implement in some languages than the others. For example, the Template pattern is easy to do in C++ and Java, since C++ provides template and Java provides generics.

In this book, designed patterns are divided into 3 categories based on purpose. The following notes are extracted from the book.

A. Creational

Class:

1. Factory method
Define an interface for creating an object, but let subclasses decide which class to instantiate. It lets a class defer instantiation to subclasses.

Object:

2. Abstract Factory
Provide an interface for creating families of related or dependent objects w/o specifying their concrete classes.

Isn't this the interface concept in C++/Java? Obviously it's related to polymorphism.

3. Builder
Separate the construction of a complex object from its representation so that the same construction process can create different representations.

4. Prototype
Specify the kinds of objects to create using a prototypical instance, and create new objects by copying this prototype.

5. Singleton
Ensure a class only has one instance, and provide a global point of access to it.

B. Structural

Class:

6. Adapter (class)
Convert the interface of a class into another interface clients expect. It lets classes work together that couldn't otherwise because of incompatible interfaces.

Object:

7. Adapter (object)

8. Bridge
Decouple an abstraction from its implementation so that the two can vary independently.

This is the abstraction and encapsulation principles of OOP.

9. Composite
Compose objects into tree structures to represent part-whole hierarchies. It lets clients treat individual objects and compositions of objects uniformly.

OOP uses inheritance for IS-A relationship, and uses composition for HAS-A relationship.

10. Decorator
Attach additional responsibilities to an object dynamically. It provides a flexible alternative to subclassing for extending functionality.

11. Facade
Provide a unified interface to a set of interfaces in a subsystem. Facade defines a higher-level interface that makes the subsystem easier to use.

12. Flyweight
Use sharing to support large numbers of fine-grained objects efficiently.

13. Proxy
Provide a surrogate or placeholder for another object to control access to it.

C. Behavioral

Class:

14. Interpreter
Given a language, define a representation for its grammar along with an interpreter that uses the representation to interpret sentences in the language.

Sounds related to compiler/interpreter.

15. Template method
Define the skeleton of an algorithm in an operation, deferring some steps to subclasses. Template method lets subclasses redefine certain steps of an algorithm w/o changing the algorithm's structure.

Object:

16. Chain of Responsibility
Avoid coupling the sender of a request to its receiver by giving more than one object a chance to handle the request. Chain the receiving objects and pass the request along the chain until an object handles it.

One example to this is to catch a series of exceptions in C++/Java.

17. Command
Encapsulate a request as an object, thereby letting you parameterize clients with different requests, queue or log requests, and support undoable operations.

18. Iterator
Provide a way to access the elements of an aggregate object sequentially w/o exposing its underlying representation.

This occurs abundantly in C++/Java.

19. Mediator
Define an object that encapsulates how a set of objects interact. It promotes loose coupling by keeping objects from referring to each other explicitly, and it lets you vary their interaction independently.

20. Memento
W/o violating encapsulation, capture and externalize an object's internal state so that the object can be restored to this state later.

21. Observer
Define a one-to-many dependency between objects so that when one object changes state, all its dependents are notified and updated automatically.

For example, Java uses wait(), notify() and notifyAll() methods for threads communication.

22. State
Allow an object to alter its behavior when its internal state changes. It will appear to change it class.

One example for this is the workflow state management as in my work.

23. Strategy
Define a family of algorithms, encapsulate each one, and make them interchangeable. It lets the algorithm vary independently from clients that use it.

24. Visitor
Represent an operation to be performed on the elements of an object structure. Visitor lets you define a new operation w/o changing the classes of the elements on which it operates.

The famous MVC is not included in list because it's a combination of multiple patterns. Use Smalltalk MVC as an example. The VC relationship uses the Strategy pattern. MVC also uses Factory method to specify the default controller class for a view, and Decorator pattern to add scrolling to a view.

A new comer at OOD can start with the simplest and most common patterns:
Creational: Abstract Factory, Factory
Structural: Adapter, Composite, Decorator
Behavioral: Observer, Strategy, Template(! Yeah, this is behavioral, not structural)

Seems like I already had experience with at least these design patterns:
Creational: Singleton, Factory
Structural: Adapter, Bridge, Composite
Behavioral: Interpreter, Template, Chain of Responsibility, Iterator, Observer, State

[1] Amazon: Design patterns. By Erich Gamma, Richard Helm, Ralph Johnson, John Vlissides. QA 76.64 .D47 1994.
[2] Wiki - design patterns. Short but comprehensive list of design patterns from different sources.
[3] Design pattern implementation in C# and VB.NET. Good link with real code examples. E.g., load balancer using the Singleton pattern.

Tips on IMS development

Here are some tips on information management system web application development from my work. Most are platform independent and can be easily applied to different platforms and technologies. They can also be applied to non-web applications, and are actually general design tips.

• Menu. The menu can be dynamically generated from database. This allows flexible configuration and update. If there are multiple roles, each can be configured to draw from the pool of menu items. For this first get a javascript menu template, then create it dynamically. For better performance and void regenerate it each time, such menu can be generated once at creation time, then be stored for later use.
  
  
• Roles and permissions. There can be many roles. The permission can be configured this way: for each item that needs permission control, say we need read/write/delete, then can use a string such as "742172". Here each digit stands for the permission setting of one role. The value of a digit is the combined sum of 4/2/1, which stands for read/write/delete, similar to unix.
  
  
• Workflow. A table can be used for this, which records stage and stage transition settings. Each role may have different permissions at different stages. Such a table can easily store these information.
  
  
• Email. If there are emails to be sent, an admin interface can be provided to configure the subject, body and other fields of emails. For specific information, provide some macros that the administrator can use, which will be replaced by program later.
  
  
• HTML Form. Can create a self-defined framework, instead of using that of such as .NET's control. This allows more flexibility, and can be more efficient since .NET's control may keep more information than necessary. To store the state/parameters, such as when you need to ask for confirmation of user, use hidden HTML control.
  
  
• HTML Form fields. The form fields settings can be managed in the database. So there is no need to go to code for update. Everything can be configured from database. Like label management below, adding a new field can automatically insert itself to a table in database with default settings. Given each form and each form field (if needed) a class name, and control display properties from CSS.
  
  
• Label management. An admin interface can be provided for this. When a tag is put on a page, it can automatically check if it's in database already, if not then insert itself in as a unique tag name. This allows domain experts to change it, and frees programmers from such ad-hoc, trivial but time-consuming and distracting chores.
  
  
• The use of coding. Any systematic categorizing paradigms can be implemented by coding for ease of control and ease of versioning. Say we can use the starting code of 001 to stand for version 1, 002 to stand for version 2, etc. This can apply to many things and is highly useful.
  
  
• Use classes as much as possible, decouple business logic from interface files.