Hadoop - The Definitive Guide, 3rd edition.
Just got this book over the weekend. Now reading.
The 2nd edition of this book is available here.
Reference:
[1] 15+ Great Books for Hadoop
Wednesday, November 19, 2014
Wednesday, November 5, 2014
How jQuery is written from scratch?
This probably helps: http://www.mikedoesweb.com/2012/creating-your-own-javascript-library/
This is a minimum javascript library whose grammar is very similar to jQuery: _('id').function(). It helps to understand how jQuery is written from scratch.
I think I can start writing a javascript library similar to jQuery now :).
This is a minimum javascript library whose grammar is very similar to jQuery: _('id').function(). It helps to understand how jQuery is written from scratch.
I think I can start writing a javascript library similar to jQuery now :).
Tuesday, November 4, 2014
Wikipedia management
Wikipedia is a nice open source project. However it may incur a lot of hackers' interests if it is open for edit. Hackers may register large number of spam accounts, create new pages and modify existing pages. This is what's happening to a wikipedia I'm managing right now.
Although quite straightforward, I keep log of these management tips of wikipedia.
The default settings are in wiki/includes/DefaultSettings.php. But changes to settings can be made in wiki/LocalSettings.php, the changes here will override default settings.
1) To change logo, change this line:
$wgLogo = "$wgStylePath/common/images/wiki.png";
2) To disable anonymous user (not logged in) creating new page and editing, add this to the end of LocalSettings:
#
# See: http://www.mediawiki.org/wiki/Manual:User_rights
#
#
# not-logged-in user
#
$wgGroupPermissions['*']['createaccount'] = false; # no new user registration.
//$wgGroupPermissions['*']['read'] = true; # default is true, so no need to add this statement.
$wgGroupPermissions['*']['edit'] = false;
$wgGroupPermissions['*']['createpage'] = false;
#
# logged-in user.
# Add this if you want to disable logged-in users from createpage/edit actions.
#
#$wgGroupPermissions['user']['edit'] = false;
#$wgGroupPermissions['user']['createpage'] = false; // true;
#
# sysop
# If you disabled logged-in users actions, you may want to enable sysop's permission here.
#
#$wgGroupPermissions['sysop']['edit'] = true;
#$wgGroupPermissions['sysop']['createpage'] = true;
After you disabled new user's createaccount ability, you will want to manually create new accounts. You can do this by logging in as sysop, and going to "Special pages -> Login / create account".
3) Email setting. Use $wgSMTP to change email setting: either use Pear Mail or PHP Mail.
Refer to the last post "Fix Bluehost wikipedia email problem".
Although quite straightforward, I keep log of these management tips of wikipedia.
The default settings are in wiki/includes/DefaultSettings.php. But changes to settings can be made in wiki/LocalSettings.php, the changes here will override default settings.
1) To change logo, change this line:
$wgLogo = "$wgStylePath/common/images/wiki.png";
2) To disable anonymous user (not logged in) creating new page and editing, add this to the end of LocalSettings:
#
# See: http://www.mediawiki.org/wiki/Manual:User_rights
#
#
# not-logged-in user
#
$wgGroupPermissions['*']['createaccount'] = false; # no new user registration.
//$wgGroupPermissions['*']['read'] = true; # default is true, so no need to add this statement.
$wgGroupPermissions['*']['edit'] = false;
$wgGroupPermissions['*']['createpage'] = false;
#
# logged-in user.
# Add this if you want to disable logged-in users from createpage/edit actions.
#
#$wgGroupPermissions['user']['edit'] = false;
#$wgGroupPermissions['user']['createpage'] = false; // true;
#
# sysop
# If you disabled logged-in users actions, you may want to enable sysop's permission here.
#
#$wgGroupPermissions['sysop']['edit'] = true;
#$wgGroupPermissions['sysop']['createpage'] = true;
After you disabled new user's createaccount ability, you will want to manually create new accounts. You can do this by logging in as sysop, and going to "Special pages -> Login / create account".
3) Email setting. Use $wgSMTP to change email setting: either use Pear Mail or PHP Mail.
Refer to the last post "Fix Bluehost wikipedia email problem".
Fix Bluehost wikipedia email problem
I'm managing a website hosted by Bluehost.com.
From CPanel I installed wikipedia. However the email function does not work. It says: "bluehost Mailer returned: Failed to add recipient: [email] … Mailer returned: Unknown error in PHP's mail() function."
I followed instruction at http://www.mediawiki.org/wiki/Manual:$wgSMTP to add email SMTP setting to wiki/LocalSettings.php:
Still it does not work. I followed Bluehost's email setting page, use either SSL/TLS setting or non-SSL setting. Still it fails.
Finally I tried to find out where $wgSMTP is used: grep -i wgSMTH */*, and found it being used in /wiki/includes/UserMailer.php. It comes down to the function send(). When $wgSMTP is used, it uses Pear Mail; otherwise, it uses PHP Mail. Since I'm using PHP Mail in a personal project on the same server, I tried to check what's different it uses PHP Mail from my use. On line 359 of UserMailer.php, it has this code:
I print out $headers, it is:
From: CareerAssist Wiki Return-Path: apache@www.careerassist.org Date: Tue, 04 Nov 2014 04:41:29 -0700 Message-ID: X-Mailer: MediaWiki mailer MIME-Version: 1.0 Content-type: text/plain; charset=UTF-8 Content-transfer-encoding: 8bit
Since I'm using the PHP's mail() function in a personal project on the same server and it works well, where I simply use: $headers = "From: [email]";
So I replace the $headers here with this simpler rendition:
Then it works! This also shows it takes the 2nd path here ($safeMode is false).
Thus I got wikipedia email working on Bluehost, with this hack into wikipedia's source code.
From CPanel I installed wikipedia. However the email function does not work. It says: "bluehost Mailer returned: Failed to add recipient: [email] … Mailer returned: Unknown error in PHP's mail() function."
I followed instruction at http://www.mediawiki.org/wiki/Manual:$wgSMTP to add email SMTP setting to wiki/LocalSettings.php:
$wgSMTP = array( 'host' => "mail.example.com", 'IDHost' => "example.com", 'port' => 26, 'auth' => true, 'username' => "my_user_name", 'password' => "my_password" );
Still it does not work. I followed Bluehost's email setting page, use either SSL/TLS setting or non-SSL setting. Still it fails.
Finally I tried to find out where $wgSMTP is used: grep -i wgSMTH */*, and found it being used in /wiki/includes/UserMailer.php. It comes down to the function send(). When $wgSMTP is used, it uses Pear Mail; otherwise, it uses PHP Mail. Since I'm using PHP Mail in a personal project on the same server, I tried to check what's different it uses PHP Mail from my use. On line 359 of UserMailer.php, it has this code:
foreach ( $to as $recip ) {
if ( $safeMode ) {
$sent = mail( $recip, self::quotedPrintable( $subject ), $body, $headers );
} else {
$sent = mail( $recip, self::quotedPrintable( $subject ), $body, $headers, $wgAdditionalMailParams );
}
}I print out $headers, it is:
From: CareerAssist Wiki Return-Path: apache@www.careerassist.org Date: Tue, 04 Nov 2014 04:41:29 -0700 Message-ID: X-Mailer: MediaWiki mailer MIME-Version: 1.0 Content-type: text/plain; charset=UTF-8 Content-transfer-encoding: 8bit
Since I'm using the PHP's mail() function in a personal project on the same server and it works well, where I simply use: $headers = "From: [email]";
So I replace the $headers here with this simpler rendition:
foreach ( $to as $recip ) {
if ( $safeMode ) {
$sent = mail( $recip, self::quotedPrintable( $subject ), $body, $headers );
} else {
$headers = "From: [email]";
$sent = mail( $recip, self::quotedPrintable( $subject ), $body, $headers, $wgAdditionalMailParams );
}
}Then it works! This also shows it takes the 2nd path here ($safeMode is false).
Thus I got wikipedia email working on Bluehost, with this hack into wikipedia's source code.
Saturday, October 25, 2014
Mysql - myisam and innodb
Major differences:
innodb: supports referential integrity (foreign key), transaction.
myisam does not support these 2.
In more details [1]:
innodb: supports referential integrity (foreign key), transaction.
myisam does not support these 2.
In more details [1]:
MYISAM:
INNODB:
- MYISAM supports Table-level Locking
- MyISAM designed for need of speed
- MyISAM does not support foreign keys hence we call MySQL with MYISAM is DBMS
- MyISAM stores its tables, data and indexes in diskspace using separate three different files. (tablename.FRM, tablename.MYD, tablename.MYI)
- MYISAM not supports transaction. You cannot commit and rollback with MYISAM. Once you issue a command it’s done.
- MYISAM supports fulltext search
- You can use MyISAM, if the table is more static with lots of select and less update and delete.
INNODB:
- InnoDB supports Row-level Locking
- InnoDB designed for maximum performance when processing high volume of data
- InnoDB support foreign keys hence we call MySQL with InnoDB is RDBMS
- InnoDB stores its tables and indexes in a tablespace
- InnoDB supports transaction. You can commit and rollback with InnoDB
Tuesday, October 21, 2014
jQuery 1.8 update on ajax call
jQuery ajax call uses different syntax since version 1.8 [1].
Deprecation Notice: The jqXHR.success(), jqXHR.error(), and jqXHR.complete() callbacks are deprecated as of jQuery 1.8 (but still works in 1.9.1). To prepare your code for their eventual removal, use jqXHR.done(), jqXHR.fail(), and jqXHR.always() instead.
Get html in jQuery before 1.8:
$.ajax({
url: 'test.html',
dataType: 'html',
success: function (data, textStatus, xhr)
{
console.log(data);
},
error: function (xhr, textStatus, errorThrown)
{
console.log('error: '+textStatus);
}
});
The code above still works in 1.8. But the code below no longer works in 1.8:
$.post("get_news.php", { id: id }, function(data, status) {
if (status == "success") {
if ( data != '' ) {
o.innerHTML = data;
}
return 1;
} else {
return 1; // ok
}
}, 5);
Get html in jQuery since 1.8:
// cache: false is used to fetch the latest version
$.ajax({
type: 'POST',
url: "test.html",
data: { id: id },
dataType: 'html',
cache: false
})
.done(function(data, textStatus, jqXHR)
{
console.log(data);
})
.fail(function(jqXHR, textStatus, errorThrown)
{
console.log('error: '+textStatus);
});
[1] http://www.sitepoint.com/ajax-jquery-1-8/
Deprecation Notice: The jqXHR.success(), jqXHR.error(), and jqXHR.complete() callbacks are deprecated as of jQuery 1.8 (but still works in 1.9.1). To prepare your code for their eventual removal, use jqXHR.done(), jqXHR.fail(), and jqXHR.always() instead.
Get html in jQuery before 1.8:
$.ajax({
url: 'test.html',
dataType: 'html',
success: function (data, textStatus, xhr)
{
console.log(data);
},
error: function (xhr, textStatus, errorThrown)
{
console.log('error: '+textStatus);
}
});
The code above still works in 1.8. But the code below no longer works in 1.8:
$.post("get_news.php", { id: id }, function(data, status) {
if (status == "success") {
if ( data != '' ) {
o.innerHTML = data;
}
return 1;
} else {
return 1; // ok
}
}, 5);
Get html in jQuery since 1.8:
// cache: false is used to fetch the latest version
$.ajax({
type: 'POST',
url: "test.html",
data: { id: id },
dataType: 'html',
cache: false
})
.done(function(data, textStatus, jqXHR)
{
console.log(data);
})
.fail(function(jqXHR, textStatus, errorThrown)
{
console.log('error: '+textStatus);
});
[1] http://www.sitepoint.com/ajax-jquery-1-8/
Wednesday, October 15, 2014
Nginx v.s. Apache
[1] Blog: Nginx v.s. Apache (May 14th, 2014)
Apache - spawns a new process for each request. In pre-forked mode, each process contains one thread; in worker Multi-process mode (not thread safe), each process contains multiple thread. Apache is process-driven.
Nginx - creates a fixed number of processes at start (a rule of thumb is 1 process each CPU), each process is single-threaded. Each request is handled by a thread. Each thread can handle thousands of requests in sequence. Nginx is event-driven, asynchronous and non-blocking.
Nginx can process php natively. But it does not support Python/Ruby etc. Nginx is much faster in serving static contents. A common configuration is use Nginx as proxy server in front of Apache back end server.
[2] http://www.thegeekstuff.com/2013/11/nginx-vs-apache/
Nginx v.s. Apache:
[3] http://www.wikivs.com/wiki/Apache_vs_nginx
Apache is like Microsoft Word, it has a million options but you only need six. Nginx does those six things, and it does five of them 50 times faster than Apache.
[4] Install Nginx: http://www.thegeekstuff.com/2011/07/install-nginx-from-source/
Summary of installation steps:
- download source code from: here. Or:
- Install Nginx:
./configure --help
./configure
make
make install
Note: if in the step of "./configure" you get a complaint that pcre is not available, then you can either install pcre, or do: "./configure –-without-http_rewrite_module"
- Change default listening port
After this you can query the nginx processes:
Get version:
Read logs:
- Stop Nginx server:
Apache - spawns a new process for each request. In pre-forked mode, each process contains one thread; in worker Multi-process mode (not thread safe), each process contains multiple thread. Apache is process-driven.
Nginx - creates a fixed number of processes at start (a rule of thumb is 1 process each CPU), each process is single-threaded. Each request is handled by a thread. Each thread can handle thousands of requests in sequence. Nginx is event-driven, asynchronous and non-blocking.
Nginx can process php natively. But it does not support Python/Ruby etc. Nginx is much faster in serving static contents. A common configuration is use Nginx as proxy server in front of Apache back end server.
[2] http://www.thegeekstuff.com/2013/11/nginx-vs-apache/
Nginx v.s. Apache:
- As discussed above Nginx is based on event-driven architecture. Apache is based on process-driven architecture. It is interesting to note that Apache in its earliest release was not having multitasking architecture. Later Apache MPM (multi-processing module) was added to achieve this.
- Nginx doesn’t create a new process for a new request. Apache creates a new process for each request.
- In Nginx, memory consumption is very low for serving static pages. But, Apache’s nature of creating new process for each request increases the memory consumption.
- Several benchmarking results indicates that when compared to Apache, Nginx is extremely fast for serving static pages.
- Nginx development started only in 2002. But Apache initial release was in 1995.
- In complex configurations situation, when compared to Nginx, Apache can be configured easily as it comes with lot of configuration features to cover wide range of requirements.
- When compared to Nginx, Apache has excellent documentation.
- In general, Nginx have less components to add more features. But Apache has tons of features and provides lot more functionality than Nginx.
- Nginx do not support Operating Systems like OpenVMS and IBMi. But Apache supports much wider range of Operating Systems.
- Since Nginx comes only with core features that are required for a web server, it is lightweight when compared to Apache.
- The performance and scalability of Nginx is not completely dependent on hardware resources, whereas the performance and scalability of the Apache is dependent on underlying hardware resources like memory and CPU.
[3] http://www.wikivs.com/wiki/Apache_vs_nginx
Apache is like Microsoft Word, it has a million options but you only need six. Nginx does those six things, and it does five of them 50 times faster than Apache.
[4] Install Nginx: http://www.thegeekstuff.com/2011/07/install-nginx-from-source/
Summary of installation steps:
- download source code from: here. Or:
cd wget http://nginx.org/download/nginx-1.0.5.tar.gz tar xvfz nginx-1.0.5.tar.gz cd nginx-1.0.5
- Install Nginx:
./configure --help
./configure
make
make install
Note: if in the step of "./configure" you get a complaint that pcre is not available, then you can either install pcre, or do: "./configure –-without-http_rewrite_module"
- Change default listening port
# vi /usr/local/nginx/conf/nginx.conf
server {
listen 8081;
server_name localhost;
- Start Nginx server:
cd /usr/local/nginx/sbin
./nginx
After this you can query the nginx processes:
# ps -ef | grep -i nginx root 18596 13:16 nginx: master process ./nginx nobody 18597 13:16 nginx: worker process
Get version:
# ./nginx -v nginx: nginx version: nginx/1.0.5
Read logs:
# ls /usr/local/nginx/logs/ access.log error.log nginx.pid
- Stop Nginx server:
cd /usr/local/nginx/sbin ./nginx -s stop
Friday, October 10, 2014
CSS Box Shadow
CSS Box Shadow
Basically you use this css:
div {
box-shadow: 0px 20px 15px -15px rgba(0,0,0,0.49), /* bottom */
10px 0px 15px -15px rgba(0,0,0,0.49), /* right */
-10px 0px 15px -15px rgba(0,0,0,0.49); /* left */
-webkit-box-shadow: 0px 20px 15px -15px rgba(0, 0, 0, 0.49),
10px 0px 15px -15px rgba(0,0,0,0.49),
-10px 0px 15px -15px rgba(0,0,0,0.49);
}
The 5 parameters are:
1. The horizontal offset of the shadow, positive means the shadow will be on the right of the box, a negative offset will put the shadow on the left of the box.
2. The vertical offset of the shadow, a negative one means the box-shadow will be above the box, a positive one means the shadow will be below the box.
3. The blur radius (optional), if set to 0 the shadow will be sharp, the higher the number, the more blurred it will be.
4. The spread radius (optional), positive values increase the size of the shadow, negative values decrease the size. Default is 0 (the shadow is same size as blur).
5. Color
Basically you use this css:
div {
box-shadow: 0px 20px 15px -15px rgba(0,0,0,0.49), /* bottom */
10px 0px 15px -15px rgba(0,0,0,0.49), /* right */
-10px 0px 15px -15px rgba(0,0,0,0.49); /* left */
-webkit-box-shadow: 0px 20px 15px -15px rgba(0, 0, 0, 0.49),
10px 0px 15px -15px rgba(0,0,0,0.49),
-10px 0px 15px -15px rgba(0,0,0,0.49);
}
The 5 parameters are:
1. The horizontal offset of the shadow, positive means the shadow will be on the right of the box, a negative offset will put the shadow on the left of the box.
2. The vertical offset of the shadow, a negative one means the box-shadow will be above the box, a positive one means the shadow will be below the box.
3. The blur radius (optional), if set to 0 the shadow will be sharp, the higher the number, the more blurred it will be.
4. The spread radius (optional), positive values increase the size of the shadow, negative values decrease the size. Default is 0 (the shadow is same size as blur).
5. Color
Wednesday, September 10, 2014
Thursday, August 21, 2014
Count LOC recursively
Count lines of code in a directory recursively:
find . -name '*.java' | xargs wc -l
See [1].
[1] How to count all the lines of code in a directory recursively?
find . -name '*.java' | xargs wc -l
See [1].
[1] How to count all the lines of code in a directory recursively?
Monday, August 18, 2014
Avoid text/image/video width overrun with css
Say you have a html table or div, or in the case of mobile device browser. A long text segment, or image, or video (e.g., from youtube, using iframe), whose width is too long to fit in the parent container, this will break the layout. Since I'm working on a forum compatible with both desktop computer and mobile devices, here are my findings on the solutions.
== text ==
Use this in display:
<pre class="forum">content</pre>
Of course, it does not have to be pre tag. It can be a div too. In css [1]:
pre.forum {
/* Wrap long text with space in the middle. */
white-space: pre-wrap; /* Wrap text as needed. CSS 3. */
white-space: -moz-pre-wrap; /* Mozilla, since 1999 */
white-space: -pre-wrap; /* Opera 4-6 */
white-space: -o-pre-wrap; /* Opera 7 */
word-wrap: break-word; /* Internet Explorer 5.5+, Non standard for webkit */
/* Wrap long text without space in between. */
-ms-word-break: break-all;
word-break: break-all;
-webkit-hyphens: auto;
-moz-hyphens: auto;
-ms-hyphens: auto;
hyphens: auto;
}
This css works for both desktop browser and mobile browser.
== Image ==
Use this in display:
<img src="..." class="media_img">
In css:
img.media_img {
max-width: 100%;
height: auto;
}
Note here use max-width instead of width, because if the image width is less than device width or parent container width (in desktop browser), we want to keep the image width and not expand it. This css works for both desktop browser and mobile browser.
This will always work in Safari. However, this may fail in firefox, if the "width" value is specified in the img tag, e.g.,
<img src="..." width="1500" class="media_img">
The way to fix this, is to wrap the img tag with another div tag, whose width is specified as an absolute value (the max design value):
<div class="media_img_container"> <img src="..." width="1500" class="media_img"> </div>
In css:
div.media_img_container {
width: 1000px;
}
Then the image will be at most 1000px in width, despite the specification of width="1500" in img tag. This also means that for different design width, you need to specify different div container width.
== Video ==
For desktop browser, this usually is not a problem, since video width is often around 500px. For mobile device, this can be a problem since mobile browser width can be only 320px, so the css here can be used. This css will resize the video iframe to always match device width.
In the case the iframe width is less than device width, this will expand the iframe width. If you don't want to do it, maybe you can check the width parameter in the iframe tag and don't apply this css.
Use this in display:
<div class="aspect-ratio"><iframe ...></iframe></div>
In css [2]:
.aspect-ratio {
position: relative;
width: 100%;
height: 0;
padding-bottom: 75%; /* height/width, usually 3/4 on youtube */
}
.aspect-ratio iframe {
position: absolute;
width: 99%;
height: 99%;
left: 0; top: 0;
}
References:
[1] http://kenneth.io/blog/2012/03/04/word-wrapping-hypernation-using-css/
[2] http://fettblog.eu/blog/2013/06/16/preserving-aspect-ratio-for-embedded-iframes/
== text ==
Use this in display:
<pre class="forum">content</pre>
Of course, it does not have to be pre tag. It can be a div too. In css [1]:
pre.forum {
/* Wrap long text with space in the middle. */
white-space: pre-wrap; /* Wrap text as needed. CSS 3. */
white-space: -moz-pre-wrap; /* Mozilla, since 1999 */
white-space: -pre-wrap; /* Opera 4-6 */
white-space: -o-pre-wrap; /* Opera 7 */
word-wrap: break-word; /* Internet Explorer 5.5+, Non standard for webkit */
/* Wrap long text without space in between. */
-ms-word-break: break-all;
word-break: break-all;
-webkit-hyphens: auto;
-moz-hyphens: auto;
-ms-hyphens: auto;
hyphens: auto;
}
This css works for both desktop browser and mobile browser.
== Image ==
Use this in display:
<img src="..." class="media_img">
In css:
img.media_img {
max-width: 100%;
height: auto;
}
Note here use max-width instead of width, because if the image width is less than device width or parent container width (in desktop browser), we want to keep the image width and not expand it. This css works for both desktop browser and mobile browser.
This will always work in Safari. However, this may fail in firefox, if the "width" value is specified in the img tag, e.g.,
<img src="..." width="1500" class="media_img">
The way to fix this, is to wrap the img tag with another div tag, whose width is specified as an absolute value (the max design value):
<div class="media_img_container"> <img src="..." width="1500" class="media_img"> </div>
In css:
div.media_img_container {
width: 1000px;
}
Then the image will be at most 1000px in width, despite the specification of width="1500" in img tag. This also means that for different design width, you need to specify different div container width.
== Video ==
For desktop browser, this usually is not a problem, since video width is often around 500px. For mobile device, this can be a problem since mobile browser width can be only 320px, so the css here can be used. This css will resize the video iframe to always match device width.
In the case the iframe width is less than device width, this will expand the iframe width. If you don't want to do it, maybe you can check the width parameter in the iframe tag and don't apply this css.
Use this in display:
<div class="aspect-ratio"><iframe ...></iframe></div>
In css [2]:
.aspect-ratio {
position: relative;
width: 100%;
height: 0;
padding-bottom: 75%; /* height/width, usually 3/4 on youtube */
}
.aspect-ratio iframe {
position: absolute;
width: 99%;
height: 99%;
left: 0; top: 0;
}
References:
[1] http://kenneth.io/blog/2012/03/04/word-wrapping-hypernation-using-css/
[2] http://fettblog.eu/blog/2013/06/16/preserving-aspect-ratio-for-embedded-iframes/
Thursday, August 7, 2014
Create a website compatible with Mobile device
A template of a html page compatible with mobile device layout is attached at the end of this post.
In the header: <meta name="viewport" content="width=device-width"> means if the agent is a mobile device, then set the page width to the device's width.
In the resizePage() function, if the navigator useragent is detected as a mobile device, then set the image's width and the div's width. Function effectiveDeviceWidth() [1] will return the device screen width for the current orientation (horizontal or vertical). The code in [1] was like this but somehow it does not work with my Android:
function effectiveDeviceWidth() {
var deviceWidth = window.orientation == 0 ? window.screen.width : window.screen.height;
// iOS returns available pixels, Android returns pixels / pixel ratio
// http://www.quirksmode.org/blog/archives/2012/07/more_about_devi.html
if (navigator.userAgent.indexOf('Android') >= 0 && window.devicePixelRatio) {
deviceWidth = deviceWidth / window.devicePixelRatio;
}
return deviceWidth;
}
Note you can see relevant parameters with this javascript:
document.write("width: " + window.screen.width + ", height:" + window.screen.height + ", ratio:" + window.devicePixelRatio + ", orientation:" + window.orientation);
For example, for iPhone:
Vertical: width: 320, height: 480, ratio: 2, orientation: 0
Horizontal: width: 320, height: 480, ratio: 2, orientation:90 or -90
For Android:
Vertical: width: 480, height: 800, ratio: 1.5, orientation: 0
Horizontal: width: 800, height: 480, ratio: 1.5, orientation:90 or -90
Note in general it is hard to get the device size in a way compatible with all the brands. I am using iPhone 7.1 and Android 4.1.2.
The code after this [2] are useful to mobile devices that do not automatically update layout when orientation is changed. This is not an issue for iPhone, but is one for Android.
== HTML Page Template Compatible With Mobile Devices ==
<!DOCTYPE html>
<html>
<head>
<meta name="viewport" content="width=device-width">
<title>Test Site</title>
</head>
<body style="margin: 0px 0px 0px 0px;" onload="resizePage();">
<center>
<img id="img1" src="https://www.google.com/images/srpr/logo11w.png">
<div id="div1">
<h2>About</h2>
<p>This is a site compatible to mobile device layout.</p>
</div>
</center>
<script type="text/javascript">
//
// Reisze page elements according to device width.
// Applies for mobile devices only.
//
function resizePage() {
var uagent = navigator.userAgent.toLowerCase();
if (uagent.search("iphone") > -1 ||
uagent.search("ipod") > -1 ||
uagent.search("ipad") > -1 ||
uagent.search("appletv") > -1 ||
uagent.search("android") > -1 ||
uagent.search("blackberry") > -1 ||
uagent.search("webos") > -1
) {
var w = effectiveDeviceWidth();
document.getElementById("img1").style.width = w + "px";
document.getElementById("div1").style.width = (w - 20) + "px";
}
}
function effectiveDeviceWidth() {
// document.write("width: " + window.screen.width + ", height:" + window.screen.height + ", ratio:" +
// window.devicePixelRatio + ", orientation:" + window.orientation);
var deviceWidth = window.screen.width;
if (navigator.userAgent.toLowerCase().indexOf('android') > -1 && window.devicePixelRatio) {
deviceWidth = deviceWidth / window.devicePixelRatio;
}
return deviceWidth;
}
//
// Resize page if orientation is changed. useful for Android.
//
var previousOrientation = window.orientation;
var checkOrientation = function(){
if(window.orientation !== previousOrientation){
previousOrientation = window.orientation;
resizePage();
}
};
window.addEventListener("resize", checkOrientation, false);
window.addEventListener("orientationchange", checkOrientation, false);
</script>
</body>
</html>
References:
[1] Detect effective horizontal pixel width on a mobile device with Javascript
[2] Detect rotation of Android phone in the browser with javascript
In the header: <meta name="viewport" content="width=device-width"> means if the agent is a mobile device, then set the page width to the device's width.
In the resizePage() function, if the navigator useragent is detected as a mobile device, then set the image's width and the div's width. Function effectiveDeviceWidth() [1] will return the device screen width for the current orientation (horizontal or vertical). The code in [1] was like this but somehow it does not work with my Android:
function effectiveDeviceWidth() {
var deviceWidth = window.orientation == 0 ? window.screen.width : window.screen.height;
// iOS returns available pixels, Android returns pixels / pixel ratio
// http://www.quirksmode.org/blog/archives/2012/07/more_about_devi.html
if (navigator.userAgent.indexOf('Android') >= 0 && window.devicePixelRatio) {
deviceWidth = deviceWidth / window.devicePixelRatio;
}
return deviceWidth;
}
Note you can see relevant parameters with this javascript:
document.write("width: " + window.screen.width + ", height:" + window.screen.height + ", ratio:" + window.devicePixelRatio + ", orientation:" + window.orientation);
For example, for iPhone:
Vertical: width: 320, height: 480, ratio: 2, orientation: 0
Horizontal: width: 320, height: 480, ratio: 2, orientation:90 or -90
For Android:
Vertical: width: 480, height: 800, ratio: 1.5, orientation: 0
Horizontal: width: 800, height: 480, ratio: 1.5, orientation:90 or -90
Note in general it is hard to get the device size in a way compatible with all the brands. I am using iPhone 7.1 and Android 4.1.2.
The code after this [2] are useful to mobile devices that do not automatically update layout when orientation is changed. This is not an issue for iPhone, but is one for Android.
== HTML Page Template Compatible With Mobile Devices ==
<!DOCTYPE html>
<html>
<head>
<meta name="viewport" content="width=device-width">
<title>Test Site</title>
</head>
<body style="margin: 0px 0px 0px 0px;" onload="resizePage();">
<center>
<img id="img1" src="https://www.google.com/images/srpr/logo11w.png">
<div id="div1">
<h2>About</h2>
<p>This is a site compatible to mobile device layout.</p>
</div>
</center>
<script type="text/javascript">
//
// Reisze page elements according to device width.
// Applies for mobile devices only.
//
function resizePage() {
var uagent = navigator.userAgent.toLowerCase();
if (uagent.search("iphone") > -1 ||
uagent.search("ipod") > -1 ||
uagent.search("ipad") > -1 ||
uagent.search("appletv") > -1 ||
uagent.search("android") > -1 ||
uagent.search("blackberry") > -1 ||
uagent.search("webos") > -1
) {
var w = effectiveDeviceWidth();
document.getElementById("img1").style.width = w + "px";
document.getElementById("div1").style.width = (w - 20) + "px";
}
}
function effectiveDeviceWidth() {
// document.write("width: " + window.screen.width + ", height:" + window.screen.height + ", ratio:" +
// window.devicePixelRatio + ", orientation:" + window.orientation);
var deviceWidth = window.screen.width;
if (navigator.userAgent.toLowerCase().indexOf('android') > -1 && window.devicePixelRatio) {
deviceWidth = deviceWidth / window.devicePixelRatio;
}
return deviceWidth;
}
//
// Resize page if orientation is changed. useful for Android.
//
var previousOrientation = window.orientation;
var checkOrientation = function(){
if(window.orientation !== previousOrientation){
previousOrientation = window.orientation;
resizePage();
}
};
window.addEventListener("resize", checkOrientation, false);
window.addEventListener("orientationchange", checkOrientation, false);
</script>
</body>
</html>
References:
[1] Detect effective horizontal pixel width on a mobile device with Javascript
[2] Detect rotation of Android phone in the browser with javascript
WordPress note
== Move WordPress to another folder ==
There is no need to backup entire site and move things. A simple, light-weight procedure is [1]:
1. Go to admin panel: Settings > General
2. Change the two URI addresses, click on "Save Changes". This will show an error page, but it's fine.
3. Open your SSH/FTP client (or other site admin tool) and move the WP folder
4. Now open your blog at the new location, go to Settings > General, click on "Save Changes".
5. Go to Settings > Permalinks, click on "Save Changes", this is supposed to recreate all .htaccess rules.
== Add social media share function ==
The simple script to insert for the sharing panel can be obtained from [2]. It just needs to insert this script anywhere between the body tags in your site.
Say you are using the twenty twelve theme, just edit this file: wp-content/themes/twentytwelve/footer.php
Add the script before the close body tag. That's all.
[1] http://wordpress.org/support/topic/moving-wordpress-to-another-folder
[2] http://jiathis.com/
There is no need to backup entire site and move things. A simple, light-weight procedure is [1]:
1. Go to admin panel: Settings > General
2. Change the two URI addresses, click on "Save Changes". This will show an error page, but it's fine.
3. Open your SSH/FTP client (or other site admin tool) and move the WP folder
4. Now open your blog at the new location, go to Settings > General, click on "Save Changes".
5. Go to Settings > Permalinks, click on "Save Changes", this is supposed to recreate all .htaccess rules.
== Add social media share function ==
The simple script to insert for the sharing panel can be obtained from [2]. It just needs to insert this script anywhere between the body tags in your site.
Say you are using the twenty twelve theme, just edit this file: wp-content/themes/twentytwelve/footer.php
Add the script before the close body tag. That's all.
[1] http://wordpress.org/support/topic/moving-wordpress-to-another-folder
[2] http://jiathis.com/
Wednesday, July 30, 2014
Flash an Android Phone
A friend got an old Android phone to use for Android development testing. It was a Nexus S 4G, Android build version was GWK74 (2.7.3 Gingerbread). So I spent some time to investigate how to do this. It worked and here is the note.
In this case, when boot into fastboot mode, it complains "Fastboot Mode No Boot or Recovery IMG". So need to download the images.
Also during the process, windows needs 2 device drivers to detect the phone: 1) when android is on, need google device driver for Nexus S 4G, which comes with ADT (in my case, it's in D:\Android\android-sdk\extras\google\usb_drive). 2) when android is in fastboot mode, need another driver called "Android Bootloader Interface driver". After a lot of search, this is available with pdaNet [1].
Next, need to download the images. This can be downloaded from Factory Images for Nexus Devices [2]. For Nexus S 4G, the corresponding section is at the end of the page: "Factory Images sojus for Nexus S 4G (d720)". 3 versions are available: 2.3.7 (GWK74), 4.0.4 (IMM76D) and 4.1.1 (JRO03R). The phone previously had 2.3.7 (GWK74), now we want a more modern version, so choose 4.1.1 (JRO03R).
Now, for actual steps to flash the phone, see [3]. Note in the last step 10, the commands are the same as in flash-all.sh and flash-all.bat of the package in [2]. The steps in [3] are copied below. In an established environment (no need to setup any drivers), the steps involved are 5, 6, 7, 9, 10, 11.
1. Install the Android SDK and Eclipse.
Eclipse probably isn’t necessary but it is nice to have.
2. Launch the Android SDK Manager.
3. In the Android SDK manager verify that the Google USB Driver is installed and up to date.
4. Connect the target phone to the PC using USB. Make sure USB Debugging is enabled on the phone.
5. From the command line, display all adb devices
Use the command: adb devices
This command is in the Android SDK platform-tools folder. I have this added to my Path. ADB should list out all attached Android devices.
If no devices are listed, make sure you have the Google Android USB driver installed.
Windows Device Manager should show a device of “Android Composite ADB Interface”.
6. Use ADB to reboot the phone into Fastboot mode
Use the command: adb reboot bootloader
This command is in the Android SDK platform-tools folder.
7. Verify that fastboot can access the phone
Use the command: fastboot devices
This should list out all attached Android devices in Fastboot mode. Notice that the device is no longer visible to ADB. “adb devices” no longer lists the device. However, “fastboot devices” should.
8. Install the Android Bootloader Interface driver if needed
If Fastboot Devices lists the phone, this step is not necessary. If Fastboot does not list the phone or shows
You may need to install the Android Bootloader Interface driver. This can be from the pdaNet.
9. Download and expand the device images to use.
For this phone, Google publishes the standard images at https://developers.google.com/android/nexus/images#sojusgwk74
10. Execute the commands from flash-all.sh to flash the device.
Change to the folder with the expanded images and execute the fastboot commands from the flash-all.sh file.
fastboot flash bootloader bootloader-crespo4g-d720sprlc1.img
fastboot reboot-bootloader
fastboot flash radio radio-crespo4g-d720sprlf2.img
fastboot reboot-bootloader
fastboot -w update image-sojus-jro03r.zip
Note the last step "fastboot -w update image-sojus-jro03r.zip" may fail, possible because it cannot decode the zip file for the images, so do it manually: first uncompress the zip file, which contains the images, then in DOS, use these commands (refer to the end of [4]), which I call step 11:
11. Flash the images.
fastboot flash recovery recovery.img
fastboot flash boot boot.img
fastboot flash userdata userdata.img
fastboot flash system system.img
12. Recovery
In the Android screen, use sound dial to choose "Recovery", then press power button to confirm.
It takes a few minutes to recover from the flashed images, then automatically boot into the new Android 4.1.1 system.
The entire process can be summarized into 3 steps:
1) setup device drivers so computer can detect the phone in both normal and fastboot modes.
2) download image package for your device.
3) use Android SDK tools adb and fastboot to do the flash.
References:
[1] pdaNet
[2] Factory Images for Nexus Devices
[3] Flash Nexus S 4G
[4] http://wiki.cyanogenmod.org/w/Doc:_fastboot_intro
In this case, when boot into fastboot mode, it complains "Fastboot Mode No Boot or Recovery IMG". So need to download the images.
Also during the process, windows needs 2 device drivers to detect the phone: 1) when android is on, need google device driver for Nexus S 4G, which comes with ADT (in my case, it's in D:\Android\android-sdk\extras\google\usb_drive). 2) when android is in fastboot mode, need another driver called "Android Bootloader Interface driver". After a lot of search, this is available with pdaNet [1].
Next, need to download the images. This can be downloaded from Factory Images for Nexus Devices [2]. For Nexus S 4G, the corresponding section is at the end of the page: "Factory Images sojus for Nexus S 4G (d720)". 3 versions are available: 2.3.7 (GWK74), 4.0.4 (IMM76D) and 4.1.1 (JRO03R). The phone previously had 2.3.7 (GWK74), now we want a more modern version, so choose 4.1.1 (JRO03R).
Now, for actual steps to flash the phone, see [3]. Note in the last step 10, the commands are the same as in flash-all.sh and flash-all.bat of the package in [2]. The steps in [3] are copied below. In an established environment (no need to setup any drivers), the steps involved are 5, 6, 7, 9, 10, 11.
1. Install the Android SDK and Eclipse.
Eclipse probably isn’t necessary but it is nice to have.
2. Launch the Android SDK Manager.
3. In the Android SDK manager verify that the Google USB Driver is installed and up to date.
4. Connect the target phone to the PC using USB. Make sure USB Debugging is enabled on the phone.
5. From the command line, display all adb devices
Use the command: adb devices
This command is in the Android SDK platform-tools folder. I have this added to my Path. ADB should list out all attached Android devices.
If no devices are listed, make sure you have the Google Android USB driver installed.
Windows Device Manager should show a device of “Android Composite ADB Interface”.
6. Use ADB to reboot the phone into Fastboot mode
Use the command: adb reboot bootloader
This command is in the Android SDK platform-tools folder.
7. Verify that fastboot can access the phone
Use the command: fastboot devices
This should list out all attached Android devices in Fastboot mode. Notice that the device is no longer visible to ADB. “adb devices” no longer lists the device. However, “fastboot devices” should.
8. Install the Android Bootloader Interface driver if needed
If Fastboot Devices lists the phone, this step is not necessary. If Fastboot does not list the phone or shows
9. Download and expand the device images to use.
For this phone, Google publishes the standard images at https://developers.google.com/android/nexus/images#sojusgwk74
10. Execute the commands from flash-all.sh to flash the device.
Change to the folder with the expanded images and execute the fastboot commands from the flash-all.sh file.
fastboot flash bootloader bootloader-crespo4g-d720sprlc1.img
fastboot reboot-bootloader
fastboot flash radio radio-crespo4g-d720sprlf2.img
fastboot reboot-bootloader
fastboot -w update image-sojus-jro03r.zip
Note the last step "fastboot -w update image-sojus-jro03r.zip" may fail, possible because it cannot decode the zip file for the images, so do it manually: first uncompress the zip file, which contains the images, then in DOS, use these commands (refer to the end of [4]), which I call step 11:
11. Flash the images.
fastboot flash recovery recovery.img
fastboot flash boot boot.img
fastboot flash userdata userdata.img
fastboot flash system system.img
12. Recovery
In the Android screen, use sound dial to choose "Recovery", then press power button to confirm.
It takes a few minutes to recover from the flashed images, then automatically boot into the new Android 4.1.1 system.
The entire process can be summarized into 3 steps:
1) setup device drivers so computer can detect the phone in both normal and fastboot modes.
2) download image package for your device.
3) use Android SDK tools adb and fastboot to do the flash.
References:
[1] pdaNet
[2] Factory Images for Nexus Devices
[3] Flash Nexus S 4G
[4] http://wiki.cyanogenmod.org/w/Doc:_fastboot_intro
Monday, July 28, 2014
Google form and Sign in with linkedin
== Google form ==
Now working with some friends on a registration function.
I just learned Google forms is a convenient tool to create simple registration forms. You can do it from either Google Forms or Google Drive. This form will be linked to a spreadsheet online, such that all records are stored there.
The submission of a form is much better if it 1) sends a confirmation email, and also 2) include a link to edit the submission. It's also good to 3) have a dashboard that displays submitted information, which you often want an interface independent from the spreadsheet. These can all be done with Google Forms API.
1) and 2) need writing a javascript function.
See Email confirmations from Google Forms for how to set up a script triggered by form submission action.
This is my code to include edit link:
function myFunction(e) {
if (typeof e == 'undefined') {
Logger.log("e is undefined");
return;
}
//var userName = e.values[1];
//var userEmail = e.values[2];
var userName = e.namedValues["Name"][0]; // From a field whose name is "Name".
var userEmail = e.namedValues["Email"][0]; // From a field whose name is "Email".
if (userEmail == '') return;
var subject = "Form Submitted";
var form = FormApp.openById('[form id]');
var formResponses = form.getResponses(); // All responses/rows in spreadsheet.
var formResponse = formResponses[formResponses.length-1]; // Get the just submitted item - last row.
//Logger.log("formResponses.length = " + formResponses.length);
var message = "Thank you, " + userName + " for finishing the survey.\n\n";
message += "You can see the current list at [dash board page link]\n\n";
message += "You can edit your information at: " + formResponse.getEditResponseUrl() + "\n\n";
message += "Have a good day.";
MailApp.sendEmail (userEmail, subject, message);
}
Note in the code above, the "form id" must be the id of the form, and not the spreadsheet. The code itself it a code of the spreadsheet.
Here is another piece of code that works equally well, but should be embedded in the form, and not the spreadsheet. This code is better in that it does not need to specify any form id. I prefer this one.
function onFormSubmit(e) {
if (typeof e == 'undefined') {
Logger.log("e is undefined");
return;
}
var itemResponses = e.response.getItemResponses();
/*
for (var i = 0; i < itemResponses.length; i++) {
var itemResponse = itemResponses[i];
Logger.log('Response #%s to the question "%s" was "%s"',
(i + 1).toString(),
itemResponse.getItem().getTitle(),
itemResponse.getResponse());
}
*/
var subject = "Form Submitted";
var userName = itemResponses[0].getResponse();
var userEmail = itemResponses[4].getResponse();
var message = "Thank you, " + userName + " for finishing the survey.\n\n";
message += "You can edit your information at: " + e.response.getEditResponseUrl() + "\n\n";
message += "Have a good day.";
MailApp.sendEmail (userEmail, subject, message);
}
3) Displays submitted information not using the spreadsheet.
See Query a Google Spreadsheet like a Database with Google Visualization API Query Language. This shows how to display a table containing selected spreadsheet columns. The link will be below (replace group id and group id with your values):
https://docs.google.com/spreadsheets/d/[form id]/gviz/tq?tqx=out:html&tq&gid=[group id]
If you want to display only selected columns, e.g., columns A and B, you can specify this with the tq parameter: tq=SELECT+A,B, so the links becomes:
https://docs.google.com/spreadsheets/d/[form id]/gviz/tq?tqx=out:html&tq=SELECT+A,B&gid=[group id]
One concern of giving people this link is security: they can modify the value of tq to see all fields. To overcome this is easy: set up a php page that read in the contents and display, this way the url is hidden. It is also really easy to set up, just 1 line of php code is needed:
<?php
echo file_get_contents("https://docs.google.com/spreadsheets/d/[form id]/gviz/tq?tqx=out:html&tq=SELECT+A,B&gid=[group id]");
?>
== Reliability issue ==
Well, it seems the script in google forms are not always reliably triggered. The above code stops to function without any reason. Searched on line for "google form script trigger not reliable", it seems many other people had similar experience. Free lunch is not always tasty.
== Sign in with Linkedin ==
Say you want people to fill the above form, but not everyone, only those who registered with linked in. So what you do is to set up a page that requires linkedin authentication, then forward people to the above link. For details, see [1][2][3][4].
Following example code in [4]. The code to set up such a page is in appendix.
Note that the Google form itself is not protected by session. So if anyone knows the url of the form, he will be able to register the form. I have not studied about ways to do this. There may not be a way of doing it, since it's not a full-fledged website anyway. One can change the setting of make a Google form private/public or accessible to only some people, that's what you can do if you don't want it public.
References:
[1] Sign In With LinkedIn
[2] Linkedin authentication documentation - Important. [3] below is linked from here.
[3] Linkedin developer network - Register here to get a linkedin application account. Important.
[4] Linkedin authentication code sample in PHP - Useful
Appendix. Authentication with Linkedin.
<?php
// Change these 5 fields.
define('API_KEY', '...');
define('API_SECRET', '...');
define('REDIRECT_URI', 'http://...');
define('SCOPE', ''); //r_fullprofile r_emailaddress rw_nus');
$reg_url = "https://docs.google.com/forms/d/[form id]/viewform?c=0&w=1&usp=mail_form_link";
// You'll probably use a database
session_name('linkedin');
session_start();
$user = fetch('GET', '/v1/people/~:(firstName,lastName)');
// OAuth 2 Control Flow
if (isset($_GET['error'])) {
// LinkedIn returned an error
//print $_GET['error'] . ': ' . $_GET['error_description'];
//exit;
} elseif (isset($_GET['code'])) {
// User authorized your application
if ($_SESSION['state'] == $_GET['state']) {
// Get token so you can make API calls
getAccessToken();
} else {
// CSRF attack? Or did you mix up your states?
exit;
}
} elseif (isset($_GET['logout'])) {
$_SESSION = array();
} elseif (isset($_GET['login'])) {
if ((empty($_SESSION['expires_at'])) || (time() > $_SESSION['expires_at'])) {
// Token has expired, clear the state
$_SESSION = array();
}
if (empty($_SESSION['access_token'])) {
// Start authorization process
getAuthorizationCode();
}
else {
print "?";
}
}
$user = fetch('GET', '/v1/people/~:(firstName,lastName)');
if ($user->firstName == '' && $user->lastName == '') {
print "Please <a href='" . $_SERVER['PHP_SELF'] . "?login=1'>log into linkedin</a> before registration.<br/>";
exit;
} else {
header("Location: $reg_url");
//echo file_get_contents($reg_url);
//exit;
//print "Hello $user->firstName $user->lastName. Click here to go to <a href='$reg_url'>registration form</a>.";
//print "<br/><a href='" . $_SERVER['PHP_SELF'] . "?logout=1'>logout</a>";
exit;
}
function getAuthorizationCode() {
$_SESSION['state'] = uniqid('', true); // unique long string.
$params = array('response_type' => 'code',
'client_id' => API_KEY,
'scope' => SCOPE,
'state' => $_SESSION['state'],
'redirect_uri' => REDIRECT_URI,
);
// Authentication request
$url = 'https://www.linkedin.com/uas/oauth2/authorization?' . http_build_query($params);
// Needed to identify request when it returns to us
$_SESSION['state'] = $params['state'];
// Redirect user to authenticate
header("Location: $url");
exit;
}
function getAccessToken() {
$params = array('grant_type' => 'authorization_code',
'client_id' => API_KEY,
'client_secret' => API_SECRET,
'code' => $_GET['code'],
'redirect_uri' => REDIRECT_URI,
);
// Access Token request
$url = 'https://www.linkedin.com/uas/oauth2/accessToken?' . http_build_query($params);
// Tell streams to make a POST request
$context = stream_context_create(
array('http' =>
array('method' => 'POST',
)
)
);
// Retrieve access token information
$response = file_get_contents($url, false, $context);
// Native PHP object, please
$token = json_decode($response);
// Store access token and expiration time
$_SESSION['access_token'] = $token->access_token; // guard this!
$_SESSION['expires_in'] = $token->expires_in; // relative time (in seconds)
$_SESSION['expires_at'] = time() + $_SESSION['expires_in']; // absolute time
return true;
}
function fetch($method, $resource, $body = '') {
$params = array('oauth2_access_token' => $_SESSION['access_token'],
'format' => 'json',
);
// Need to use HTTPS
$url = 'https://api.linkedin.com' . $resource . '?' . http_build_query($params);
// Tell streams to make a (GET, POST, PUT, or DELETE) request
$context = stream_context_create(
array('http' =>
array('method' => $method,
)
)
);
// Hocus Pocus
$response = file_get_contents($url, false, $context);
// Native PHP object, please
return json_decode($response);
}
?>
Now working with some friends on a registration function.
I just learned Google forms is a convenient tool to create simple registration forms. You can do it from either Google Forms or Google Drive. This form will be linked to a spreadsheet online, such that all records are stored there.
The submission of a form is much better if it 1) sends a confirmation email, and also 2) include a link to edit the submission. It's also good to 3) have a dashboard that displays submitted information, which you often want an interface independent from the spreadsheet. These can all be done with Google Forms API.
1) and 2) need writing a javascript function.
See Email confirmations from Google Forms for how to set up a script triggered by form submission action.
This is my code to include edit link:
function myFunction(e) {
if (typeof e == 'undefined') {
Logger.log("e is undefined");
return;
}
//var userName = e.values[1];
//var userEmail = e.values[2];
var userName = e.namedValues["Name"][0]; // From a field whose name is "Name".
var userEmail = e.namedValues["Email"][0]; // From a field whose name is "Email".
if (userEmail == '') return;
var subject = "Form Submitted";
var form = FormApp.openById('[form id]');
var formResponses = form.getResponses(); // All responses/rows in spreadsheet.
var formResponse = formResponses[formResponses.length-1]; // Get the just submitted item - last row.
//Logger.log("formResponses.length = " + formResponses.length);
var message = "Thank you, " + userName + " for finishing the survey.\n\n";
message += "You can see the current list at [dash board page link]\n\n";
message += "You can edit your information at: " + formResponse.getEditResponseUrl() + "\n\n";
message += "Have a good day.";
MailApp.sendEmail (userEmail, subject, message);
}
Note in the code above, the "form id" must be the id of the form, and not the spreadsheet. The code itself it a code of the spreadsheet.
Here is another piece of code that works equally well, but should be embedded in the form, and not the spreadsheet. This code is better in that it does not need to specify any form id. I prefer this one.
function onFormSubmit(e) {
if (typeof e == 'undefined') {
Logger.log("e is undefined");
return;
}
var itemResponses = e.response.getItemResponses();
/*
for (var i = 0; i < itemResponses.length; i++) {
var itemResponse = itemResponses[i];
Logger.log('Response #%s to the question "%s" was "%s"',
(i + 1).toString(),
itemResponse.getItem().getTitle(),
itemResponse.getResponse());
}
*/
var subject = "Form Submitted";
var userName = itemResponses[0].getResponse();
var userEmail = itemResponses[4].getResponse();
var message = "Thank you, " + userName + " for finishing the survey.\n\n";
message += "You can edit your information at: " + e.response.getEditResponseUrl() + "\n\n";
message += "Have a good day.";
MailApp.sendEmail (userEmail, subject, message);
}
3) Displays submitted information not using the spreadsheet.
See Query a Google Spreadsheet like a Database with Google Visualization API Query Language. This shows how to display a table containing selected spreadsheet columns. The link will be below (replace group id and group id with your values):
https://docs.google.com/spreadsheets/d/[form id]/gviz/tq?tqx=out:html&tq&gid=[group id]
If you want to display only selected columns, e.g., columns A and B, you can specify this with the tq parameter: tq=SELECT+A,B, so the links becomes:
https://docs.google.com/spreadsheets/d/[form id]/gviz/tq?tqx=out:html&tq=SELECT+A,B&gid=[group id]
One concern of giving people this link is security: they can modify the value of tq to see all fields. To overcome this is easy: set up a php page that read in the contents and display, this way the url is hidden. It is also really easy to set up, just 1 line of php code is needed:
<?php
echo file_get_contents("https://docs.google.com/spreadsheets/d/[form id]/gviz/tq?tqx=out:html&tq=SELECT+A,B&gid=[group id]");
?>
== Reliability issue ==
Well, it seems the script in google forms are not always reliably triggered. The above code stops to function without any reason. Searched on line for "google form script trigger not reliable", it seems many other people had similar experience. Free lunch is not always tasty.
== Sign in with Linkedin ==
Say you want people to fill the above form, but not everyone, only those who registered with linked in. So what you do is to set up a page that requires linkedin authentication, then forward people to the above link. For details, see [1][2][3][4].
Following example code in [4]. The code to set up such a page is in appendix.
Note that the Google form itself is not protected by session. So if anyone knows the url of the form, he will be able to register the form. I have not studied about ways to do this. There may not be a way of doing it, since it's not a full-fledged website anyway. One can change the setting of make a Google form private/public or accessible to only some people, that's what you can do if you don't want it public.
References:
[1] Sign In With LinkedIn
[2] Linkedin authentication documentation - Important. [3] below is linked from here.
[3] Linkedin developer network - Register here to get a linkedin application account. Important.
[4] Linkedin authentication code sample in PHP - Useful
Appendix. Authentication with Linkedin.
<?php
// Change these 5 fields.
define('API_KEY', '...');
define('API_SECRET', '...');
define('REDIRECT_URI', 'http://...');
define('SCOPE', ''); //r_fullprofile r_emailaddress rw_nus');
$reg_url = "https://docs.google.com/forms/d/[form id]/viewform?c=0&w=1&usp=mail_form_link";
// You'll probably use a database
session_name('linkedin');
session_start();
$user = fetch('GET', '/v1/people/~:(firstName,lastName)');
// OAuth 2 Control Flow
if (isset($_GET['error'])) {
// LinkedIn returned an error
//print $_GET['error'] . ': ' . $_GET['error_description'];
//exit;
} elseif (isset($_GET['code'])) {
// User authorized your application
if ($_SESSION['state'] == $_GET['state']) {
// Get token so you can make API calls
getAccessToken();
} else {
// CSRF attack? Or did you mix up your states?
exit;
}
} elseif (isset($_GET['logout'])) {
$_SESSION = array();
} elseif (isset($_GET['login'])) {
if ((empty($_SESSION['expires_at'])) || (time() > $_SESSION['expires_at'])) {
// Token has expired, clear the state
$_SESSION = array();
}
if (empty($_SESSION['access_token'])) {
// Start authorization process
getAuthorizationCode();
}
else {
print "?";
}
}
$user = fetch('GET', '/v1/people/~:(firstName,lastName)');
if ($user->firstName == '' && $user->lastName == '') {
print "Please <a href='" . $_SERVER['PHP_SELF'] . "?login=1'>log into linkedin</a> before registration.<br/>";
exit;
} else {
header("Location: $reg_url");
//echo file_get_contents($reg_url);
//exit;
//print "Hello $user->firstName $user->lastName. Click here to go to <a href='$reg_url'>registration form</a>.";
//print "<br/><a href='" . $_SERVER['PHP_SELF'] . "?logout=1'>logout</a>";
exit;
}
function getAuthorizationCode() {
$_SESSION['state'] = uniqid('', true); // unique long string.
$params = array('response_type' => 'code',
'client_id' => API_KEY,
'scope' => SCOPE,
'state' => $_SESSION['state'],
'redirect_uri' => REDIRECT_URI,
);
// Authentication request
$url = 'https://www.linkedin.com/uas/oauth2/authorization?' . http_build_query($params);
// Needed to identify request when it returns to us
$_SESSION['state'] = $params['state'];
// Redirect user to authenticate
header("Location: $url");
exit;
}
function getAccessToken() {
$params = array('grant_type' => 'authorization_code',
'client_id' => API_KEY,
'client_secret' => API_SECRET,
'code' => $_GET['code'],
'redirect_uri' => REDIRECT_URI,
);
// Access Token request
$url = 'https://www.linkedin.com/uas/oauth2/accessToken?' . http_build_query($params);
// Tell streams to make a POST request
$context = stream_context_create(
array('http' =>
array('method' => 'POST',
)
)
);
// Retrieve access token information
$response = file_get_contents($url, false, $context);
// Native PHP object, please
$token = json_decode($response);
// Store access token and expiration time
$_SESSION['access_token'] = $token->access_token; // guard this!
$_SESSION['expires_in'] = $token->expires_in; // relative time (in seconds)
$_SESSION['expires_at'] = time() + $_SESSION['expires_in']; // absolute time
return true;
}
function fetch($method, $resource, $body = '') {
$params = array('oauth2_access_token' => $_SESSION['access_token'],
'format' => 'json',
);
// Need to use HTTPS
$url = 'https://api.linkedin.com' . $resource . '?' . http_build_query($params);
// Tell streams to make a (GET, POST, PUT, or DELETE) request
$context = stream_context_create(
array('http' =>
array('method' => $method,
)
)
);
// Hocus Pocus
$response = file_get_contents($url, false, $context);
// Native PHP object, please
return json_decode($response);
}
?>
Saturday, July 26, 2014
Platform independent GUI development
For platform independent GUI development, some choices are Qt, GTK, Java.
[1] GUI development in Perl: wxPerl, Perl/Tk, Perl/Qt and Perl/KDE, gtk2-perl
[2] GTK2 tutorial
[3] Qt: best cross platform GUI Applications
[4] Comparison of GUI development tools for Linux
[5] Qt Project - download installation package and more
[6] Java: Lesson: modifying the look and feel
[1] GUI development in Perl: wxPerl, Perl/Tk, Perl/Qt and Perl/KDE, gtk2-perl
[2] GTK2 tutorial
[3] Qt: best cross platform GUI Applications
[4] Comparison of GUI development tools for Linux
[5] Qt Project - download installation package and more
[6] Java: Lesson: modifying the look and feel
Friday, July 25, 2014
Online books for Perl
Talk about web client programming, and LWP module.
[1] Web Client Programming in Perl
[2] Perl and LWP
[1] Web Client Programming in Perl
[2] Perl and LWP
Saturday, July 19, 2014
IT - The Big Picture
== Importance of Daemon/Service and Socket Programming ==
Linux/Unix daemon and windows service are important. They reside in memory as long-running processes, often start up automatically when the machine is booted.
Any piece of software, either a database or web/file/mail/etc server, is a linux/unix daemon or windows service. NoSQL applications Hadoop, mongoDB, memcached, etc., all are daemons/services or make use of it.
For this reason, it is good to write templates for linux/unix daemon and windows service. When need a relevant utility, we can quickly add the functionality into the daemon/service template and make a product.
Socket programming is another technology tightly coupled and integrated with Daemon/Service. This comes from the need of communication between the daemon/service and their clients. Protocol and port are two key players. Different servers use different protocols, and listen on different ports.
Multimedia applications need Computer graphics, Computer vision, Image Processing and Visualization.
It has been PC in the 1980s, Internet in the 1990s, Search in 2000s.
Today, the hypes are: Big data (Cloud, NoSQL, Hadoop, DM, ML, NLP), Mobile, Social media.
Of these, Big data builds the platform through large corporations. Mobile and Social media are the interface to the general public, where business profits are made.
== Conclusion ==
So, based on this big picture, one may ask him/her self, where am I? What do I know, and what do I want to know next, based on job prospects or personal interest?
Linux/Unix daemon and windows service are important. They reside in memory as long-running processes, often start up automatically when the machine is booted.
Any piece of software, either a database or web/file/mail/etc server, is a linux/unix daemon or windows service. NoSQL applications Hadoop, mongoDB, memcached, etc., all are daemons/services or make use of it.
For this reason, it is good to write templates for linux/unix daemon and windows service. When need a relevant utility, we can quickly add the functionality into the daemon/service template and make a product.
Socket programming is another technology tightly coupled and integrated with Daemon/Service. This comes from the need of communication between the daemon/service and their clients. Protocol and port are two key players. Different servers use different protocols, and listen on different ports.
Most enterprise IT and software jobs today are on information system, used to be C/S, now B/S. Most of them use databases, process data, implement business logic, have back end and front end. Back end use database and web server, front end use web programming languages. Database + Web/Mobile UI + business logic is the mainstream of IT and software jobs today. Technologies outside this realm feels exotic to many programmers.
However, when it comes to creating new software tools, Daemon/Service and Socket programming come into play.
== General Technologies In Wide Use ==
Operating System and Programming Languages are the basics of everything.
Database and Network are another duo on the top list, which we kind of already mentioned.
Compiler techniques is used widely. If a new domain language is needed for a new software, then a compiler/interpreter is written. LR or LL, or something simpler for specific domains. Parsing, AST and JIT techniques are used by editors for text highlighting and intellisense, or building the project tree like in Eclipse. Static analysis is used for cross reference, debugging and code refactoring.
Security is an eternal topic. Software engineering is about the practice.
Web, Cloud and Mobile are the applications of these, on which Social media is based. They comprise the notion of high tech for the public. Web and Game are two public industries that make use of a wide variety of computer technologies.
== Domain Technologies ==
Artificial intelligence. In a broader sense, AI also includes Data Mining, Machine Learning, NLP etc. These are often intensive in probability and statistics, parsing theory, and mathematics.
Multimedia applications need Computer graphics, Computer vision, Image Processing and Visualization.
Informatics for different domains, such as Medical informatics, Bioinformatics, GIS etc. These are more like field applications of CS techniques, but some can be very intense.
HPC and scientific computing are often for government and academia projects.
Robotics/Vision generally need support from the government or large corporation.
HPC and scientific computing are often for government and academia projects.
Robotics/Vision generally need support from the government or large corporation.
Embed devices and programming is another broad field, since so many devices use chips internally.
HCI is softer, often correlates with phycology and cognitive science. It plays an important role in the better adoption of technology.
Operation research, this is more on the mathematics side.
HCI is softer, often correlates with phycology and cognitive science. It plays an important role in the better adoption of technology.
Operation research, this is more on the mathematics side.
== Fundamentals ==
On the basics of the fundamentals are math, discrete math, numerical analysis, computation and automata theory. Probability and Statistics. Algorithms and data structures.
Then, on the software side, it is Operating system and Programming languages design. On the hardware side, it is Computer Architecture, micro processors, memory, and other device physics in Electrical Engineering.
== Today's Hot Topics ==
== Today's Hot Topics ==
It has been PC in the 1980s, Internet in the 1990s, Search in 2000s.
Today, the hypes are: Big data (Cloud, NoSQL, Hadoop, DM, ML, NLP), Mobile, Social media.
Of these, Big data builds the platform through large corporations. Mobile and Social media are the interface to the general public, where business profits are made.
== Conclusion ==
So, based on this big picture, one may ask him/her self, where am I? What do I know, and what do I want to know next, based on job prospects or personal interest?
Write a crawler in Perl
Now improve a web crawler in Perl, based on a script I wrote several years ago. This can be useful if there is a need to download a bundle of files from a website.
The crawl and storage parts are there now. Some general considerations are below. These considerations, of course, are independent of implementation language.
- Crawl
- keeps links crawled, in queue or hash
- get header: status code (200, 404 etc.), content-size, content-type, modification time
- broken links
- dynamic page
- special chars in url if create local folder using url path
- relative url - handled by relevant lib.
- wait interval between requests
- robot.txt [11]
- mime types
- referrer
- html parsing
- header 302 redirect, and redirect level
- javascript submit/redirect
- non-stardard tags
- Storage
- use web structure, or flat (need to resolve file name conflicts)
- store progress: in file or database: link_queue, non_link_queue, current pointer in link_queue.
- Data Analysis and mining
- text mining
- reverse index
- NLP etc.
- Rank Analysis
- web link graph
- page rank
== Compile Perl to executable ==
Seems PAR is a good choice [1][2].
== Other Perl Crawlers ==
[5] is a good introduction on the modules to use to write a crawler in Perl.
[4] is a simple one. [6] seems more involved.
[10] is a good introduction to general principles of web crawler.
== GUI with Perl/Tk ==
With Tk it's easy to make event driven GUI interface [7][8][9].
Tried to install Tk. Type:
sudo Perl -MCPAN -e shell
> install Tk
But there is error that prevents the installation to finish:
t/wm-tcl.t ................... 119/315
# Failed test 'attempting to resize a gridded toplevel to a value bigger'
# at t/wm-tcl.t line 1153.
# got: '4'
# expected: '6'
# Failed test at t/wm-tcl.t line 1155.
# got: '4'
# expected: '5'
t/wm-tcl.t ................... 312/315 # Looks like you failed 2 tests of 315.
t/wm-tcl.t ................... Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/315 subtests
(less 43 skipped subtests: 270 okay)
(31 TODO tests unexpectedly succeeded)
Test Summary Report
-------------------
t/listbox.t (Wstat: 0 Tests: 537 Failed: 0)
TODO passed: 320, 322, 328, 502
t/text.t (Wstat: 0 Tests: 415 Failed: 0)
TODO passed: 121
t/wm-tcl.t (Wstat: 512 Tests: 315 Failed: 2)
Failed tests: 160-161
TODO passed: 64, 86-87, 154-157, 164-165, 171-176, 221-224
237-239, 264-265, 275-276, 280-283, 300
Non-zero exit status: 2
t/zzScrolled.t (Wstat: 0 Tests: 94 Failed: 0)
TODO passed: 52, 66, 80, 94
Files=74, Tests=4348, 55 wallclock secs ( 0.84 usr 0.25 sys + 14.66 cusr 1.57 csys = 17.32 CPU)
Result: FAIL
Failed 1/74 test programs. 2/4348 subtests failed.
make: *** [test_dynamic] Error 255
SREZIC/Tk-804.032.tar.gz
/usr/bin/make test -- NOT OK
//hint// to see the cpan-testers results for installing this module, try:
reports SREZIC/Tk-804.032.tar.gz
Running make install
make test had returned bad status, won't install without force
Failed during this command:
SREZIC/Tk-804.032.tar.gz : make_test NO
References:
[1] Create self-contained Perl executables, Part II
[2] PAR: Perl Archiving Toolkit
[3] Compiling or packaging an executable from perl code on windows
[4] Web scraping with modern perl
[5] Web crawling with Perl
[6] spider.pl - Example Perl program to spider web servers
[7] Tk:UserGuid
[8] Learning Perl/Tk: Graphical User Interfaces with Perl
[9] Book: Mastering Perl/Tk
[10] Wiki: web crawler
[11] Robots Exclusion Standard
The crawl and storage parts are there now. Some general considerations are below. These considerations, of course, are independent of implementation language.
- Crawl
- keeps links crawled, in queue or hash
- get header: status code (200, 404 etc.), content-size, content-type, modification time
- broken links
- dynamic page
- special chars in url if create local folder using url path
- relative url - handled by relevant lib.
- wait interval between requests
- robot.txt [11]
- mime types
- referrer
- html parsing
- header 302 redirect, and redirect level
- javascript submit/redirect
- non-stardard tags
- Storage
- use web structure, or flat (need to resolve file name conflicts)
- store progress: in file or database: link_queue, non_link_queue, current pointer in link_queue.
- Data Analysis and mining
- text mining
- reverse index
- NLP etc.
- Rank Analysis
- web link graph
- page rank
== Compile Perl to executable ==
Seems PAR is a good choice [1][2].
== Other Perl Crawlers ==
[5] is a good introduction on the modules to use to write a crawler in Perl.
[4] is a simple one. [6] seems more involved.
[10] is a good introduction to general principles of web crawler.
== GUI with Perl/Tk ==
With Tk it's easy to make event driven GUI interface [7][8][9].
Tried to install Tk. Type:
sudo Perl -MCPAN -e shell
> install Tk
But there is error that prevents the installation to finish:
t/wm-tcl.t ................... 119/315
# Failed test 'attempting to resize a gridded toplevel to a value bigger'
# at t/wm-tcl.t line 1153.
# got: '4'
# expected: '6'
# Failed test at t/wm-tcl.t line 1155.
# got: '4'
# expected: '5'
t/wm-tcl.t ................... 312/315 # Looks like you failed 2 tests of 315.
t/wm-tcl.t ................... Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/315 subtests
(less 43 skipped subtests: 270 okay)
(31 TODO tests unexpectedly succeeded)
Test Summary Report
-------------------
t/listbox.t (Wstat: 0 Tests: 537 Failed: 0)
TODO passed: 320, 322, 328, 502
t/text.t (Wstat: 0 Tests: 415 Failed: 0)
TODO passed: 121
t/wm-tcl.t (Wstat: 512 Tests: 315 Failed: 2)
Failed tests: 160-161
TODO passed: 64, 86-87, 154-157, 164-165, 171-176, 221-224
237-239, 264-265, 275-276, 280-283, 300
Non-zero exit status: 2
t/zzScrolled.t (Wstat: 0 Tests: 94 Failed: 0)
TODO passed: 52, 66, 80, 94
Files=74, Tests=4348, 55 wallclock secs ( 0.84 usr 0.25 sys + 14.66 cusr 1.57 csys = 17.32 CPU)
Result: FAIL
Failed 1/74 test programs. 2/4348 subtests failed.
make: *** [test_dynamic] Error 255
SREZIC/Tk-804.032.tar.gz
/usr/bin/make test -- NOT OK
//hint// to see the cpan-testers results for installing this module, try:
reports SREZIC/Tk-804.032.tar.gz
Running make install
make test had returned bad status, won't install without force
Failed during this command:
SREZIC/Tk-804.032.tar.gz : make_test NO
Only 2 tests failed out of many on a resize issue, shouldn't be serious. So use force option to install and it worked:
sudo perl -fi Tk
References:
[1] Create self-contained Perl executables, Part II
[2] PAR: Perl Archiving Toolkit
[3] Compiling or packaging an executable from perl code on windows
[4] Web scraping with modern perl
[5] Web crawling with Perl
[6] spider.pl - Example Perl program to spider web servers
[7] Tk:UserGuid
[8] Learning Perl/Tk: Graphical User Interfaces with Perl
[9] Book: Mastering Perl/Tk
[10] Wiki: web crawler
[11] Robots Exclusion Standard
Friday, July 18, 2014
Monday, July 14, 2014
Supporting PHP, ASP, JSP/Servlet and Tomcat in Perl Web Server
Last time we implemented a small but functional HTTP web server in Perl, which works like Apache by serving static contents. When that was done, it became instantly clear how and why a HTTP web server, such as Apache, works that way. It also became somewhat clear how Apache uses extensions to work with non-static content, such as PHP, JSP, ASP etc.
For example, when a PHP file test.php is requested, in the Perl web server, just call something like this:
system("php test.php");
then grab the output and send it back to the client. The basic principle is as simple as that.
Now I'm looking at JEE, which uses Tomcat application server and a connector module in the middle to work with Apache. I'm thinking I should be able to extend my Perl web server to work with Tomcat, and thus the JSP/Servlet/JEE stack as well.
To do this is easy: in a config file, tell the Perl web server which paths should be mapped to Tomcat. Then, when a request coming for that path, establish a TCP client, transfer the request URL to Tomcat, which basically means to send a request to "http://localhost:8080/path", and receive the response from Tomcat server, then send it back to the Perl web server client.
Actually, this can be easily verified by using telnet. Type:
telnet localhost 8080
This will establish a connection session to Tomcat server. Next type this request:
GET /
The Tomcat server will send the index page back, and close the connection.
Basically, what the Perl web server should do is exactly the same. To implement this, we need to be able to code a web client in Perl. We can either build it from scratch in socket programming, or use the LWP module, as in reference [1].
To work with ASP or ASP.NET, the Perl web server can work as a proxy, passing the request to an internal IIS web server, and sends back the response.
This way, the Perl web server in principle can work with any other web technologies.
== Create Web Browser with a GUI ==
Chapter 7 of [1] is on graphical examples in Perl/Tk. This basically demonstrates how to implement your own web browser with a GUI (not just command line interface), similar to firefox or any other popular web browsers. And if we can do it in Perl/Tk, we can also create the GUI interface in Java or C/C++. Following this way, we can reinvent the entire wheel of the internet world [2][3]. [3] talks about the libwww module of Python, which was written by Tim Berners-Lee, and contains many functions needed by a web application including a browser.
One of the most difficult part of this is the amount of work involved in html/javascript/css parser and renderer. In 2006 Netscape wanted to create a new one from scratch, they failed after 3 years. Many of current browsers are based on a rendering engine, this is WebKit [6][7] for Safari and Chrome (before 28), blink for Opera and Chrome (28+) [11][12], Gecko for Firefox [10], and Trident for IE [8][9].
A list of browser html rendering engines can be found in [4][5], including Amaya, Blink, Gecho, KHTML, Presto, Tasman, Trident and WebKit.
References:
[1] Web Client Programming in Perl
[2] Where should you start Coding a Web Browser?
[3] W3C Blog: Build Your Own Browser
[4] Comparison of layout engines (HTML)
[5] Web browser engine
[6] The WebKit Open Source Project
[7] Wiki: WebKit
[8] Wiki: Trident (layout engine)
[9] Internet Explorer Architecture
[10] Wiki: Gecko
[11] Wiki: Blink (layout engine)
[12] The Chromium Projects: Blink
For example, when a PHP file test.php is requested, in the Perl web server, just call something like this:
system("php test.php");
then grab the output and send it back to the client. The basic principle is as simple as that.
Now I'm looking at JEE, which uses Tomcat application server and a connector module in the middle to work with Apache. I'm thinking I should be able to extend my Perl web server to work with Tomcat, and thus the JSP/Servlet/JEE stack as well.
To do this is easy: in a config file, tell the Perl web server which paths should be mapped to Tomcat. Then, when a request coming for that path, establish a TCP client, transfer the request URL to Tomcat, which basically means to send a request to "http://localhost:8080/path", and receive the response from Tomcat server, then send it back to the Perl web server client.
Actually, this can be easily verified by using telnet. Type:
telnet localhost 8080
This will establish a connection session to Tomcat server. Next type this request:
GET /
The Tomcat server will send the index page back, and close the connection.
Basically, what the Perl web server should do is exactly the same. To implement this, we need to be able to code a web client in Perl. We can either build it from scratch in socket programming, or use the LWP module, as in reference [1].
To work with ASP or ASP.NET, the Perl web server can work as a proxy, passing the request to an internal IIS web server, and sends back the response.
This way, the Perl web server in principle can work with any other web technologies.
== Create Web Browser with a GUI ==
Chapter 7 of [1] is on graphical examples in Perl/Tk. This basically demonstrates how to implement your own web browser with a GUI (not just command line interface), similar to firefox or any other popular web browsers. And if we can do it in Perl/Tk, we can also create the GUI interface in Java or C/C++. Following this way, we can reinvent the entire wheel of the internet world [2][3]. [3] talks about the libwww module of Python, which was written by Tim Berners-Lee, and contains many functions needed by a web application including a browser.
One of the most difficult part of this is the amount of work involved in html/javascript/css parser and renderer. In 2006 Netscape wanted to create a new one from scratch, they failed after 3 years. Many of current browsers are based on a rendering engine, this is WebKit [6][7] for Safari and Chrome (before 28), blink for Opera and Chrome (28+) [11][12], Gecko for Firefox [10], and Trident for IE [8][9].
A list of browser html rendering engines can be found in [4][5], including Amaya, Blink, Gecho, KHTML, Presto, Tasman, Trident and WebKit.
References:
[1] Web Client Programming in Perl
[2] Where should you start Coding a Web Browser?
[3] W3C Blog: Build Your Own Browser
[4] Comparison of layout engines (HTML)
[5] Web browser engine
[6] The WebKit Open Source Project
[7] Wiki: WebKit
[8] Wiki: Trident (layout engine)
[9] Internet Explorer Architecture
[10] Wiki: Gecko
[11] Wiki: Blink (layout engine)
[12] The Chromium Projects: Blink
Friday, July 11, 2014
Setup Ubuntu in VirtualBox
Now set up Ubuntu in VirtualBox on my windows machine.
== Download and Install Ubuntu ==
Get Ubuntu Desktop from http://www.ubuntu.com/ and install.
Also install VitualBox Guest Addition after this.
-- X sessions and Command startx --
Note: it seems if install Ubuntu desktop in VMWare, it boots into command line. Use this to start desktop: startx
There can be multiple X sessions going simultaneously, and can be accessed using combination keys Ctrl-Alt-X, where X is F1, F2, ... F7. The default seems to be Ctrl-Alt-F7. If you start another X session inside this, then it's Ctrl-Alt-F1. An X session, when not fully loaded, can be killed by Ctrl-Z. See more: man startx.
The two possible problems below are both related to Ubuntu in VM only. The first happened to me.
-- Possible login issue --
It's possible that after you reboot machine, then you won't be able to log in. Each time it returns to the login screen after a few seconds. It might be caused by that your ~/.Xauthority file is owned by root:root. To fix, press Ctrl-Alt-F1 to enter session 1, log in console from there, and type: sudo chown me:me ~/.Xauthority, where "me" is your account name. See [7].
-- 3D acceleration --
3D acceleration by GPU guratantees after speed, otherwise 3D acceleration is done by software and slow. See [8].
== Install LAMP ==
-- Apache2 --
sudo apt-get install apache2
-- PHP --
sudo apt-get install php5
-- PHP Client (not necessary, this is console client) --
sudo apt-get install php5-cli
-- restart apache (not necessary, since it's already restarted) --
Use either of the 2 commands below:
sudo /etc/init.d/apache2 restart
sudo service apache2 restart
-- MySQL --
sudo apt-get install mysql-server libapache2-mod-auth-mysql php5-mysql
Then optionally do the following:
-- activate mysql --
sudo mysql_install_db
-- change root password, and other security settings --
sudo /usr/bin/mysql_secure_installation
== Verify LAMP Installation ==
Now, Apache root is: /var/www/html/. There is already a index.html file there.
This can be visited as http://localhost
Create a php file phptest.php under the root, with content: <?php phpinfo(); ?>
This can be visited as http://localhost/phptest.php
Visit mysql from command line:
> mysql -u root -p
== Network Setup ==
You want to be able to access guest's webserver from host so it's more useful.
In VMWare Player, with networking set to NAT you can access the guest from host using guest's IP:
http://[guest ip]/
However, it seems VirtualBox does not allow this.
Recommendation for VirtualBox is to use Bridged Network, then host can access guest. However the guest loses ability to access Internet.
One way in this case is to use NAT but enable port forwarding, so host will route such request to guest.
To do this (you can change setting with guest running), go to Settings -> Adapter 1 (NAT), click on button "Port Forwarding", and enter this:
Name: Rule 1 (default)
Protocol: TCP
Host IP: (leave blank)
Host Port: 81
Guest IP: (leave blank, or use the IP found by ifconfig)
Guest Port: 80
Then in host browser, type: http://localhost:81, you will see 10.0.2.15:80 (which is not accessible if visit directly from host).
See:
- https://www.virtualbox.org/manual/ch06.html#natforward
- http://askubuntu.com/questions/353235/how-to-visit-a-site-hosted-by-my-virtual-machine
-- Note 1 --
For multiple machines hosted on VirtualBox, if they are all started, it seems they will have the same IP (found by ifconfig). I don't know how to reset their IP address (like statically), yet. But with Port Forwarding you can visit webservers of all guests from host. For example, forward the second guest's port 80 to port 82 of host.
What's more, guests can see each other. For instance, using the example above, guest 1 can see guest 2 by visit: http://[host IP]:81, and guest 2 can see guest 1 by visit: http://[host IP]:82.
-- Note 2 --
For VMWare, Port Forwarding can be setup for NAT network by using the "virtual network editor". However, the "virtual network editor" is available only to VMWare Workstation, and not available to VMWare Player.
To add the editor to VMWare Player, you can follow instructions here:
- http://www.eightforums.com/virtualization/5137-how-add-virtual-network-editor-vmware-player.html
== Install JEE Environment ==
-- Java --
See section Other Utilities below.
-- Tomcat --
To install, type:
sudo apt-get install tomcat7
Then should be able to visit http://localhost:8080.
Optionally you can install tomcat7-admin:
sudo apt-get install tomcat7-admin
Alternatively, you can download source from tomcat homepage and build it.
-- Eclipse --
Download from eclipse.com/download, extract it, then move it to a place, e.g., /home/me/Eclipse/eclipse.
And create a soft link on desktop: ln -s /home/me/Eclipse/eclipse/eclipse ~/Desktop/eclipse
== Install memcached ==
sudo apt-get install memcached
Now memcached should have been automatically started under user memcache. This can be verified by one of the following 3 ways:
1) use System Monitor to view all processes to find out.
2) use command: top -u memcache
3) use command: ps -aux | grep memcached. You can see the command of the process is:
/usr/bin/memcached -m 64 -p 11211 -u memcache -l 127.0.0.1
-- start/stop/restart/status --
sudo service memcached start/stop/restart/status
-- access in console --
telnet localhost 11211
stats
== Other Utilities ==
-- Java --
Type "java" or "javac" will tell you it's not installed, and suggests the packages to install. I choose this:
sudo apt-get install openjdk-7-jdk
After installation, see version: javac -version, java -version
-- Perl --
Type "perl -v". It's installed.
-- Python --
Type "python -V". It's installed.
Try the mandelbrot drawing python script. It reports numpy not installed.
Install NumPy and SciPy (packages for scientific computing with Python):
sudo apt-get install python-numpy
sudo apt-get install python-scipy
Then run the mandelbrot.py file:
python mandelbrot.py
Compile it:
python -m py_compile mandelbrot.py
And run it:
./mandelbroth.pyc
-- Gtk+ --
See [5][6] for more information. To install GTK+ 3.0 type:
sudo apt-get install libgtk-3-dev
-- Make --
Type "make -v", it's installed.
-- Gcc --
Type "gcc -v", it's installed.
-- G++ --
Type "g++ -v", it's installed.
-- Git --
Type "git", it's not installed. To install type:
sudo apt-get install git
-- Svn --
Note installed. To install type:
sudo apt-get install subversion
== Uninstall a package using apt-get ==
Below commands are from [11].
apt-get remove packagename
This will remove the binaries, but not the configuration or data files of the package packagename. It will also leave dependencies installed with it on installation time untouched.
apt-get purge packagename, or
apt-get remove --purge packagename
This will remove about everything regarding the package packagename, but not the dependencies installed with it on installation. Both commands are equivalent.
apt-get autoremove
removes orphaned packages, i.e. installed packages that used to be installed as an dependency, but aren't any longer. Use this after removing a package which had installed dependencies you're no longer interested in.
Or could try this:
dpkg --purge --force-depends application
== Mount a VirtualBox shared folder ==
See [12]. Assume host is windows, guest is Ubuntu.
1) First you need to install Guest Additions for VirtualBox.
2) Next open Devices -> Shared Folders Settings, and choose a folder on Host to share, say name it "shared_folder".
3) In Ubuntu guest, type these commands:
sudo mkdir /mnt/host/
sudo mount -t vboxsf shared_folder /mnt/host/
That's it.
Finally, you can create a soft link to the shared folder on desktop as shortcut:
ln -s /mnt/host ~/Desktop/host
References:
[1] How To Install Linux, Apache, MySQL, PHP (LAMP) stack on Ubuntu
[2] AWS .NET SDK on Mono/Linux
[3] mono-project.com
[4] Mono basics
[5] How do I install GTK+ 3.0
[6] The GTK+ project
[7] Can't login into Ubuntu VM
[8] 3D Acceleration with Ubuntu Guests
[9] How To Install Apache Tomcat on Ubuntu 12.04
[10] How to install Mono XSP as a daemon on Debian?
[11] What is the correct way to completely remove an application?
[12] How to mount a VirtualBox shared folder?
== Download and Install Ubuntu ==
Get Ubuntu Desktop from http://www.ubuntu.com/ and install.
Also install VitualBox Guest Addition after this.
-- X sessions and Command startx --
Note: it seems if install Ubuntu desktop in VMWare, it boots into command line. Use this to start desktop: startx
There can be multiple X sessions going simultaneously, and can be accessed using combination keys Ctrl-Alt-X, where X is F1, F2, ... F7. The default seems to be Ctrl-Alt-F7. If you start another X session inside this, then it's Ctrl-Alt-F1. An X session, when not fully loaded, can be killed by Ctrl-Z. See more: man startx.
The two possible problems below are both related to Ubuntu in VM only. The first happened to me.
-- Possible login issue --
It's possible that after you reboot machine, then you won't be able to log in. Each time it returns to the login screen after a few seconds. It might be caused by that your ~/.Xauthority file is owned by root:root. To fix, press Ctrl-Alt-F1 to enter session 1, log in console from there, and type: sudo chown me:me ~/.Xauthority, where "me" is your account name. See [7].
-- 3D acceleration --
3D acceleration by GPU guratantees after speed, otherwise 3D acceleration is done by software and slow. See [8].
== Install LAMP ==
-- Apache2 --
sudo apt-get install apache2
-- PHP --
sudo apt-get install php5
-- PHP Client (not necessary, this is console client) --
sudo apt-get install php5-cli
-- restart apache (not necessary, since it's already restarted) --
Use either of the 2 commands below:
sudo /etc/init.d/apache2 restart
sudo service apache2 restart
-- MySQL --
sudo apt-get install mysql-server libapache2-mod-auth-mysql php5-mysql
Then optionally do the following:
-- activate mysql --
sudo mysql_install_db
-- change root password, and other security settings --
sudo /usr/bin/mysql_secure_installation
== Verify LAMP Installation ==
Now, Apache root is: /var/www/html/. There is already a index.html file there.
This can be visited as http://localhost
Create a php file phptest.php under the root, with content: <?php phpinfo(); ?>
This can be visited as http://localhost/phptest.php
Visit mysql from command line:
> mysql -u root -p
== Network Setup ==
You want to be able to access guest's webserver from host so it's more useful.
In VMWare Player, with networking set to NAT you can access the guest from host using guest's IP:
http://[guest ip]/
However, it seems VirtualBox does not allow this.
Recommendation for VirtualBox is to use Bridged Network, then host can access guest. However the guest loses ability to access Internet.
One way in this case is to use NAT but enable port forwarding, so host will route such request to guest.
To do this (you can change setting with guest running), go to Settings -> Adapter 1 (NAT), click on button "Port Forwarding", and enter this:
Name: Rule 1 (default)
Protocol: TCP
Host IP: (leave blank)
Host Port: 81
Guest IP: (leave blank, or use the IP found by ifconfig)
Guest Port: 80
Then in host browser, type: http://localhost:81, you will see 10.0.2.15:80 (which is not accessible if visit directly from host).
See:
- https://www.virtualbox.org/manual/ch06.html#natforward
- http://askubuntu.com/questions/353235/how-to-visit-a-site-hosted-by-my-virtual-machine
-- Note 1 --
For multiple machines hosted on VirtualBox, if they are all started, it seems they will have the same IP (found by ifconfig). I don't know how to reset their IP address (like statically), yet. But with Port Forwarding you can visit webservers of all guests from host. For example, forward the second guest's port 80 to port 82 of host.
What's more, guests can see each other. For instance, using the example above, guest 1 can see guest 2 by visit: http://[host IP]:81, and guest 2 can see guest 1 by visit: http://[host IP]:82.
-- Note 2 --
For VMWare, Port Forwarding can be setup for NAT network by using the "virtual network editor". However, the "virtual network editor" is available only to VMWare Workstation, and not available to VMWare Player.
To add the editor to VMWare Player, you can follow instructions here:
- http://www.eightforums.com/virtualization/5137-how-add-virtual-network-editor-vmware-player.html
== Install JEE Environment ==
-- Java --
See section Other Utilities below.
-- Tomcat --
To install, type:
sudo apt-get install tomcat7
Then should be able to visit http://localhost:8080.
Optionally you can install tomcat7-admin:
sudo apt-get install tomcat7-admin
Alternatively, you can download source from tomcat homepage and build it.
-- Eclipse --
Download from eclipse.com/download, extract it, then move it to a place, e.g., /home/me/Eclipse/eclipse.
And create a soft link on desktop: ln -s /home/me/Eclipse/eclipse/eclipse ~/Desktop/eclipse
== Install Mono.NET ==
See [2][3][4] for more information on Mono. To install type:
sudo apt-get install mono-complete
Install gtk-sharp2 for GTK+ support:
sudo apt-get install gtk-sharp2
Then type this to show version and enter C# shell:
csharp --version
Type this to show mono version, use mcs or gmcs, which is "csc" under windows:
mcs --version
To compile a csharp file hello.cs [4], type:
mcs hello.cs
To compile a gtk-csharp file hello_gtk.cs [4], type:
mac hello_gtk.cs -pkg:gtk-sharp-2.0
To compile a winform file hello_winform.cs [4], type (note it's gmcs, not mcs!):
gmcs hello_winform.cs -pkg:dotnet
To run aspx, install xsp2:
sudo apt-get install mono-xsp2
This will report:
* You have an incomplete /etc/xsp2/debian.webapp
* To fix it, you need to install at least one package for xsp2 (like asp.net2-examples)
This tell us the xsp2 server configuration is in /etc/xsp2. To fix it (note it's actually asp.net-examples):
sudo apt-get install asp.net-examples
Now start xsp2 server. It uses port 8080 by default, but this is in conflict with Tomcat, so use a different port, say 8082. Run the following command from the same folder as hello.aspx:
xsp2 --port 8082
Then visit http://localhost:8082/hello.aspx, it works!
To run xsp2 as a daemon, refer to [11]. Also can run this for more help:
xsp2 --help
sudo apt-get install mono-xsp2
This will report:
* You have an incomplete /etc/xsp2/debian.webapp
* To fix it, you need to install at least one package for xsp2 (like asp.net2-examples)
This tell us the xsp2 server configuration is in /etc/xsp2. To fix it (note it's actually asp.net-examples):
sudo apt-get install asp.net-examples
Now start xsp2 server. It uses port 8080 by default, but this is in conflict with Tomcat, so use a different port, say 8082. Run the following command from the same folder as hello.aspx:
xsp2 --port 8082
Then visit http://localhost:8082/hello.aspx, it works!
To run xsp2 as a daemon, refer to [11]. Also can run this for more help:
xsp2 --help
Or if install runtime only, type:
sudo apt-get installmono-runtime
sudo apt-get install memcached
Now memcached should have been automatically started under user memcache. This can be verified by one of the following 3 ways:
1) use System Monitor to view all processes to find out.
2) use command: top -u memcache
3) use command: ps -aux | grep memcached. You can see the command of the process is:
/usr/bin/memcached -m 64 -p 11211 -u memcache -l 127.0.0.1
-- start/stop/restart/status --
sudo service memcached start/stop/restart/status
-- access in console --
telnet localhost 11211
stats
== Other Utilities ==
-- Java --
Type "java" or "javac" will tell you it's not installed, and suggests the packages to install. I choose this:
sudo apt-get install openjdk-7-jdk
After installation, see version: javac -version, java -version
-- Perl --
Type "perl -v". It's installed.
-- Python --
Type "python -V". It's installed.
Try the mandelbrot drawing python script. It reports numpy not installed.
Install NumPy and SciPy (packages for scientific computing with Python):
sudo apt-get install python-numpy
sudo apt-get install python-scipy
Then run the mandelbrot.py file:
python mandelbrot.py
Compile it:
python -m py_compile mandelbrot.py
And run it:
./mandelbroth.pyc
-- Gtk+ --
See [5][6] for more information. To install GTK+ 3.0 type:
sudo apt-get install libgtk-3-dev
-- Make --
Type "make -v", it's installed.
-- Gcc --
Type "gcc -v", it's installed.
-- G++ --
Type "g++ -v", it's installed.
-- Git --
Type "git", it's not installed. To install type:
sudo apt-get install git
-- Svn --
Note installed. To install type:
sudo apt-get install subversion
== Uninstall a package using apt-get ==
Below commands are from [11].
apt-get remove packagename
This will remove the binaries, but not the configuration or data files of the package packagename. It will also leave dependencies installed with it on installation time untouched.
apt-get purge packagename, or
apt-get remove --purge packagename
This will remove about everything regarding the package packagename, but not the dependencies installed with it on installation. Both commands are equivalent.
apt-get autoremove
removes orphaned packages, i.e. installed packages that used to be installed as an dependency, but aren't any longer. Use this after removing a package which had installed dependencies you're no longer interested in.
Or could try this:
dpkg --purge --force-depends application
== Mount a VirtualBox shared folder ==
See [12]. Assume host is windows, guest is Ubuntu.
1) First you need to install Guest Additions for VirtualBox.
2) Next open Devices -> Shared Folders Settings, and choose a folder on Host to share, say name it "shared_folder".
3) In Ubuntu guest, type these commands:
sudo mkdir /mnt/host/
sudo mount -t vboxsf shared_folder /mnt/host/
That's it.
Finally, you can create a soft link to the shared folder on desktop as shortcut:
ln -s /mnt/host ~/Desktop/host
References:
[1] How To Install Linux, Apache, MySQL, PHP (LAMP) stack on Ubuntu
[2] AWS .NET SDK on Mono/Linux
[3] mono-project.com
[4] Mono basics
[5] How do I install GTK+ 3.0
[6] The GTK+ project
[7] Can't login into Ubuntu VM
[8] 3D Acceleration with Ubuntu Guests
[9] How To Install Apache Tomcat on Ubuntu 12.04
[10] How to install Mono XSP as a daemon on Debian?
[11] What is the correct way to completely remove an application?
[12] How to mount a VirtualBox shared folder?
Subscribe to:
Posts (Atom)