Platform independent GUI development

For platform independent GUI development, some choices are Qt, GTK, Java.

[1] GUI development in Perl: wxPerl, Perl/Tk, Perl/Qt and Perl/KDE, gtk2-perl
[2] GTK2 tutorial
[3] Qt: best cross platform GUI Applications
[4] Comparison of GUI development tools for Linux
[5] Qt Project - download installation package and more
[6] Java: Lesson: modifying the look and feel

Online books for Perl

Talk about web client programming, and LWP module.

[1] Web Client Programming in Perl 
[2] Perl and LWP

IT - The Big Picture

== Importance of Daemon/Service and Socket Programming ==

Linux/Unix daemon and windows service are important. They reside in memory as long-running processes, often start up automatically when the machine is booted.

Any piece of software, either a database or web/file/mail/etc server, is a linux/unix daemon or windows service. NoSQL applications Hadoop, mongoDB, memcached, etc., all are daemons/services or make use of it.

For this reason, it is good to write templates for linux/unix daemon and windows service. When need a relevant utility, we can quickly add the functionality into the daemon/service template and make a product.

Socket programming is another technology tightly coupled and integrated with Daemon/Service. This comes from the need of communication between the daemon/service and their clients.  Protocol and port are two key players.  Different servers use different protocols, and listen on different ports.

Most enterprise IT and software jobs today are on information system, used to be C/S, now B/S. Most of them use databases, process data, implement business logic, have back end and front end. Back end use database and web server, front end use web programming languages. Database + Web/Mobile UI + business logic is the mainstream of IT and software jobs today. Technologies outside this realm feels exotic to many programmers.

However, when it comes to creating new software tools, Daemon/Service and Socket programming come into play.

== General Technologies In Wide Use ==

Operating System and Programming Languages are the basics of everything.

Database and Network are another duo on the top list, which we kind of already mentioned.

Compiler techniques is used widely.  If a new domain language is needed for a new software, then a compiler/interpreter is written. LR or LL, or something simpler for specific domains.  Parsing, AST and JIT techniques are used by editors for text highlighting and intellisense, or building the project tree like in Eclipse. Static analysis is used for cross reference, debugging and code refactoring.

Security is an eternal topic. Software engineering is about the practice.

Web, Cloud and Mobile are the applications of these, on which Social media is based. They comprise the notion of high tech for the public. Web and Game are two public industries that make use of a wide variety of computer technologies.

== Domain Technologies ==

Artificial intelligence. In a broader sense, AI also includes Data Mining, Machine Learning, NLP etc. These are often intensive in probability and statistics, parsing theory, and mathematics.

Multimedia applications need Computer graphics, Computer vision, Image Processing and Visualization.

Informatics for different domains, such as Medical informatics, Bioinformatics, GIS etc. These are more like field applications of CS techniques, but some can be very intense.

HPC and scientific computing are often for government and academia projects.

Robotics/Vision generally need support from the government or large corporation.

Embed devices and programming is another broad field, since so many devices use chips internally.

HCI is softer, often correlates with phycology and cognitive science. It plays an important role in the better adoption of technology.

Operation research, this is more on the mathematics side.

== Fundamentals ==

On the basics of the fundamentals are math, discrete math, numerical analysis, computation and automata theory. Probability and Statistics. Algorithms and data structures. 

Then, on the software side, it is Operating system and Programming languages design. On the hardware side, it is Computer Architecture, micro processors, memory, and other device physics in Electrical Engineering.


== Today's Hot Topics == 

It has been PC in the 1980s, Internet in the 1990s, Search in 2000s.

Today, the hypes are: Big data (Cloud, NoSQL, Hadoop, DM, ML, NLP), Mobile, Social media.

Of these, Big data builds the platform through large corporations. Mobile and Social media are the interface to the general public, where business profits are made.


== Conclusion ==

So, based on this big picture, one may ask him/her self, where am I? What do I know, and what do I want to know next, based on job prospects or personal interest?

Write a crawler in Perl

Now improve a web crawler in Perl, based on a script I wrote several years ago. This can be useful if there is a need to download a bundle of files from a website.

The crawl and storage parts are there now. Some general considerations are below. These considerations, of course, are independent of implementation language.

- Crawl
  - keeps links crawled, in queue or hash
  - get header: status code (200, 404 etc.), content-size, content-type, modification time
  - broken links
  - dynamic page
  - special chars in url if create local folder using url path
  - relative url - handled by relevant lib.
  - wait interval between requests
  - robot.txt [11]
  - mime types
  - referrer
  - html parsing
  - header 302 redirect, and redirect level
  - javascript submit/redirect
  - non-stardard tags

- Storage
  - use web structure, or flat (need to resolve file name conflicts)
  - store progress: in file or database: link_queue, non_link_queue, current pointer in link_queue.

- Data Analysis and mining
  - text mining
  - reverse index
  - NLP etc.

- Rank Analysis
  - web link graph
  - page rank


== Compile Perl to executable ==

Seems PAR is a good choice [1][2].


== Other Perl Crawlers ==

[5] is a good introduction on the modules to use to write a crawler in Perl.
[4] is a simple one. [6] seems more involved.

[10] is a good introduction to general principles of web crawler.


== GUI with Perl/Tk ==

With Tk it's easy to make event driven GUI interface [7][8][9].

Tried to install Tk. Type:

sudo Perl -MCPAN -e shell
> install Tk

But there is error that prevents the installation to finish:

t/wm-tcl.t ................... 119/315 
#   Failed test 'attempting to resize a gridded toplevel to a value bigger'
#   at t/wm-tcl.t line 1153.
#          got: '4'
#     expected: '6'

#   Failed test at t/wm-tcl.t line 1155.
#          got: '4'
#     expected: '5'
t/wm-tcl.t ................... 312/315 # Looks like you failed 2 tests of 315.
t/wm-tcl.t ................... Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/315 subtests 
(less 43 skipped subtests: 270 okay)
(31 TODO tests unexpectedly succeeded)

Test Summary Report
-------------------
t/listbox.t                (Wstat: 0 Tests: 537 Failed: 0)
  TODO passed:   320, 322, 328, 502
t/text.t                   (Wstat: 0 Tests: 415 Failed: 0)
  TODO passed:   121
t/wm-tcl.t                 (Wstat: 512 Tests: 315 Failed: 2)
  Failed tests:  160-161
  TODO passed:   64, 86-87, 154-157, 164-165, 171-176, 221-224
                237-239, 264-265, 275-276, 280-283, 300
  Non-zero exit status: 2
t/zzScrolled.t             (Wstat: 0 Tests: 94 Failed: 0)
  TODO passed:   52, 66, 80, 94
Files=74, Tests=4348, 55 wallclock secs ( 0.84 usr  0.25 sys + 14.66 cusr  1.57 csys = 17.32 CPU)
Result: FAIL
Failed 1/74 test programs. 2/4348 subtests failed.
make: *** [test_dynamic] Error 255
  SREZIC/Tk-804.032.tar.gz
  /usr/bin/make test -- NOT OK
//hint// to see the cpan-testers results for installing this module, try:
  reports SREZIC/Tk-804.032.tar.gz
Running make install
  make test had returned bad status, won't install without force
Failed during this command:
 SREZIC/Tk-804.032.tar.gz                     : make_test NO

Only 2 tests failed out of many on a resize issue, shouldn't be serious. So use force option to install and it worked:

sudo perl -fi Tk

References:

[1] Create self-contained Perl executables, Part II
[2] PAR: Perl Archiving Toolkit
[3] Compiling or packaging an executable from perl code on windows

[4] Web scraping with modern perl
[5] Web crawling with Perl
[6] spider.pl - Example Perl program to spider web servers

[7] Tk:UserGuid
[8] Learning Perl/Tk: Graphical User Interfaces with Perl
[9] Book: Mastering Perl/Tk

[10] Wiki: web crawler
[11] Robots Exclusion Standard

Coding Styles

Coding Style for C++
Coding style for Java
Coding style for Python
Coding style guide 

Supporting PHP, ASP, JSP/Servlet and Tomcat in Perl Web Server

Last time we implemented a small but functional HTTP web server in Perl, which works like Apache by serving static contents. When that was done, it became instantly clear how and why a HTTP web server, such as Apache, works that way.  It also became somewhat clear how Apache uses extensions to work with non-static content, such as PHP, JSP, ASP etc.

For example, when a PHP file test.php is requested, in the Perl web server, just call something like this:
system("php test.php");
then grab the output and send it back to the client. The basic principle is as simple as that.

Now I'm looking at JEE, which uses Tomcat application server and a connector module in the middle to work with Apache.  I'm thinking I should be able to extend my Perl web server to work with  Tomcat, and thus the JSP/Servlet/JEE stack as well.

To do this is easy: in a config file, tell the Perl web server which paths should be mapped to Tomcat. Then, when a request coming for that path, establish a TCP client, transfer the request URL to Tomcat, which basically means to send a request to "http://localhost:8080/path", and receive the response from Tomcat server, then send it back to the Perl web server client.

Actually, this can be easily verified by using telnet. Type:
telnet localhost 8080
This will establish a connection session to Tomcat server. Next type this request:
GET /
The Tomcat server will send the index page back, and close the connection.

Basically, what the Perl web server should do is exactly the same. To implement this, we need to be able to code a web client in Perl.  We can either build it from scratch in socket programming, or use the LWP module, as in reference [1]. 

To work with ASP or ASP.NET, the Perl web server can work as a proxy, passing the request to an internal IIS web server, and sends back the response.

This way, the Perl web server in principle can work with any other web technologies.

== Create Web Browser with a GUI ==

Chapter 7 of [1] is on graphical examples in Perl/Tk. This basically demonstrates how to implement your own web browser with a GUI (not just command line interface), similar to firefox or any other popular web browsers.  And if we can do it in Perl/Tk, we can also create the GUI interface in Java or C/C++. Following this way, we can reinvent the entire wheel of the internet world [2][3]. [3] talks about the libwww module of Python, which was written by Tim Berners-Lee, and contains many functions needed by a web application including a browser.

One of the most difficult part of this is the amount of work involved in html/javascript/css parser and renderer. In 2006 Netscape wanted to create a new one from scratch, they failed after 3 years.  Many of current browsers are based on a rendering engine, this is WebKit [6][7] for Safari and Chrome (before 28), blink for Opera and Chrome (28+) [11][12], Gecko for Firefox [10], and Trident for IE [8][9].

A list of browser html rendering engines can be found in [4][5], including Amaya, Blink, Gecho, KHTML, Presto, Tasman, Trident and WebKit.


References:

[1] Web Client Programming in Perl
[2] Where should you start Coding a Web Browser?
[3] W3C Blog: Build Your Own Browser
[4] Comparison of layout engines (HTML)
[5] Web browser engine
[6] The WebKit Open Source Project
[7] Wiki: WebKit
[8] Wiki: Trident (layout engine)
[9] Internet Explorer Architecture
[10] Wiki: Gecko
[11] Wiki: Blink (layout engine)
[12] The Chromium Projects: Blink

Setup Ubuntu in VirtualBox

Now set up Ubuntu in VirtualBox on my windows machine.


== Download and Install Ubuntu ==

Get Ubuntu Desktop from http://www.ubuntu.com/ and install.
Also install VitualBox Guest Addition after this.

-- X sessions and Command startx --
Note: it seems if install Ubuntu desktop in VMWare, it boots into command line. Use this to start desktop: startx

There can be multiple X sessions going simultaneously, and can be accessed using combination keys Ctrl-Alt-X, where X is F1, F2, ... F7. The default seems to be Ctrl-Alt-F7. If you start another X session inside this, then it's Ctrl-Alt-F1. An X session, when not fully loaded, can be killed by Ctrl-Z. See more: man startx.

The two possible problems below are both related to Ubuntu in VM only. The first happened to me.

-- Possible login issue --
It's possible that after you reboot machine, then you won't be able to log in. Each time it returns to the login screen after a few seconds. It might be caused by that your ~/.Xauthority file is owned by root:root. To fix, press Ctrl-Alt-F1 to enter session 1, log in console from there, and type: sudo chown me:me ~/.Xauthority, where "me" is your account name. See [7].

-- 3D acceleration --
3D acceleration by GPU guratantees after speed, otherwise 3D acceleration is done by software and slow. See [8].


== Install LAMP ==

-- Apache2 --
sudo apt-get install apache2

-- PHP --
sudo apt-get install php5

-- PHP Client (not necessary, this is console client) --
sudo apt-get install php5-cli

-- restart apache (not necessary, since it's already restarted) --
Use either of the 2 commands below:
sudo /etc/init.d/apache2 restart
sudo service apache2 restart

-- MySQL --
sudo apt-get install mysql-server libapache2-mod-auth-mysql php5-mysql

Then optionally do the following:
-- activate mysql --
sudo mysql_install_db
-- change root password, and other security settings --
sudo /usr/bin/mysql_secure_installation


== Verify LAMP Installation ==

Now, Apache root is: /var/www/html/. There is already a index.html file there.
This can be visited as http://localhost

Create a php file phptest.php under the root, with content: <?php phpinfo(); ?>
This can be visited as http://localhost/phptest.php

Visit mysql from command line:
> mysql -u root -p


== Network Setup ==

You want to be able to access guest's webserver from host so it's more useful.

In VMWare Player, with networking set to NAT you can access the guest from host using guest's IP:

http://[guest ip]/

However, it seems VirtualBox does not allow this.

Recommendation for VirtualBox is to use Bridged Network, then host can access guest. However the guest loses ability to access Internet.

One way in this case is to use NAT but enable port forwarding, so host will route such request to guest.

To do this (you can change setting with guest running), go to Settings -> Adapter 1 (NAT), click on button "Port Forwarding", and enter this:

Name: Rule 1 (default)
Protocol: TCP
Host IP: (leave blank)
Host Port: 81
Guest IP:  (leave blank, or use the IP found by ifconfig)
Guest Port: 80

Then in host browser, type: http://localhost:81, you will see 10.0.2.15:80 (which is not accessible if visit directly from host).

See:
- https://www.virtualbox.org/manual/ch06.html#natforward
- http://askubuntu.com/questions/353235/how-to-visit-a-site-hosted-by-my-virtual-machine


-- Note 1 --

For multiple machines hosted on VirtualBox, if they are all started, it seems they will have the same IP (found by ifconfig). I don't know how to reset their IP address (like statically), yet. But with Port Forwarding you can visit webservers of all guests from host. For example, forward the second guest's port 80 to port 82 of host.

What's more, guests can see each other. For instance, using the example above, guest 1 can see guest 2 by visit: http://[host IP]:81, and guest 2 can see guest 1 by visit: http://[host IP]:82.

-- Note 2 --

For VMWare, Port Forwarding can be setup for NAT network by using the "virtual network editor".  However, the "virtual network editor" is available only to VMWare Workstation, and not available to VMWare Player.

To add the editor to VMWare Player, you can follow instructions here:
- http://www.eightforums.com/virtualization/5137-how-add-virtual-network-editor-vmware-player.html


== Install JEE Environment ==

-- Java --
See section Other Utilities below.

-- Tomcat --
To install, type:
sudo apt-get install tomcat7
Then should be able to visit http://localhost:8080.

Optionally you can install tomcat7-admin:
sudo apt-get install tomcat7-admin

Alternatively, you can download source from tomcat homepage and build it.

-- Eclipse --
Download from eclipse.com/download, extract it, then move it to a place, e.g., /home/me/Eclipse/eclipse.
And create a soft link on desktop: ln -s /home/me/Eclipse/eclipse/eclipse ~/Desktop/eclipse

== Install Mono.NET ==

See [2][3][4] for more information on Mono. To install type:

sudo apt-get install mono-complete

Install gtk-sharp2 for GTK+ support:

sudo apt-get install gtk-sharp2

Then type this to show version and enter C# shell:

csharp --version

Type this to show mono version, use mcs or gmcs, which is "csc" under windows:

mcs --version

To compile a csharp file hello.cs [4], type:

mcs hello.cs

To compile a gtk-csharp file hello_gtk.cs [4], type:

mac hello_gtk.cs -pkg:gtk-sharp-2.0

To compile a winform file hello_winform.cs [4], type (note it's gmcs, not mcs!):

gmcs hello_winform.cs -pkg:dotnet

To run aspx, install xsp2:
sudo apt-get install mono-xsp2 
This will report:
 * You have an incomplete /etc/xsp2/debian.webapp
 * To fix it, you need to install at least one package for xsp2 (like asp.net2-examples)
This tell us the xsp2 server configuration is in /etc/xsp2. To fix it (note it's actually asp.net-examples):
sudo apt-get install asp.net-examples
Now start xsp2 server. It uses port 8080 by default, but this is in conflict with Tomcat, so use a different port, say 8082. Run the following command from the same folder as hello.aspx:
xsp2 --port 8082
Then visit http://localhost:8082/hello.aspx, it works!
To run xsp2 as a daemon, refer to [11]. Also can run this for more help:
xsp2 --help

Or if install runtime only, type:

sudo apt-get installmono-runtime

== Install memcached ==

sudo apt-get install memcached

Now memcached should have been automatically started under user memcache. This can be verified by one of the following 3 ways:
1) use System Monitor to view all processes to find out.
2) use command: top -u memcache
3) use command: ps -aux | grep memcached. You can see the command of the process is:
    /usr/bin/memcached -m 64 -p 11211 -u memcache -l 127.0.0.1

-- start/stop/restart/status --
sudo service memcached start/stop/restart/status

-- access in console --
telnet localhost 11211
stats


== Other Utilities ==

-- Java --
Type "java" or "javac" will tell you it's not installed, and suggests the packages to install. I choose this:
sudo apt-get install openjdk-7-jdk

After installation, see version: javac -version, java -version

-- Perl --
Type "perl -v". It's installed.

-- Python --
Type "python -V". It's installed.

Try the mandelbrot drawing python script. It reports numpy not installed.
Install NumPy and SciPy (packages for scientific computing with Python):
sudo apt-get install python-numpy
sudo apt-get install python-scipy

Then run the mandelbrot.py file:
python mandelbrot.py
Compile it:
python -m py_compile mandelbrot.py
And run it:
./mandelbroth.pyc

-- Gtk+ --

See [5][6] for more information. To install GTK+ 3.0 type:
sudo apt-get install libgtk-3-dev

-- Make --
Type "make -v", it's installed.

-- Gcc --
Type "gcc -v", it's installed.

-- G++ --
Type "g++ -v", it's installed.

-- Git --
Type "git", it's not installed. To install type:
sudo apt-get install git

-- Svn --
Note installed. To install type:
sudo apt-get install subversion

== Uninstall a package using apt-get ==

Below commands are from [11].

apt-get remove packagename
This will remove the binaries, but not the configuration or data files of the package packagename. It will also leave dependencies installed with it on installation time untouched.

apt-get purge packagename, or
apt-get remove --purge packagename
This will remove about everything regarding the package packagename, but not the dependencies installed with it on installation. Both commands are equivalent.

apt-get autoremove
removes orphaned packages, i.e. installed packages that used to be installed as an dependency, but aren't any longer. Use this after removing a package which had installed dependencies you're no longer interested in.

Or could try this:
dpkg --purge --force-depends application


== Mount a VirtualBox shared folder ==

See [12]. Assume host is windows, guest is Ubuntu.

1) First you need to install Guest Additions for VirtualBox.

2) Next  open Devices -> Shared Folders Settings, and choose a folder on Host to share, say name it "shared_folder".

3) In Ubuntu guest, type these commands:

sudo mkdir /mnt/host/
sudo mount -t vboxsf shared_folder /mnt/host/

That's it.

Finally, you can create a soft link to the shared folder on desktop as shortcut:

ln -s /mnt/host  ~/Desktop/host


References:

[1] How To Install Linux, Apache, MySQL, PHP (LAMP) stack on Ubuntu
[2] AWS .NET SDK on Mono/Linux
[3] mono-project.com
[4] Mono basics
[5] How do I install GTK+ 3.0
[6] The GTK+ project
[7] Can't login into Ubuntu VM
[8] 3D Acceleration with Ubuntu Guests
[9] How To Install Apache Tomcat on Ubuntu 12.04
[10] How to install Mono XSP as a daemon on Debian?
[11] What is the correct way to completely remove an application?
[12] How to mount a VirtualBox shared folder?