shell script to concat all files

This shell script runs on *nix and concatenates all files into a single one, including file name before file content. This is useful, for example, when you want to print out all files to read.

#!/bin/bash

#
# Dump multiple files to one text file.
# By: HomeTom. 3/11/2015
#

cd my_dir;  # go to the target file directory.

ct=0;  # keep a counter of file.

for i in README.md manage.py my_dir/*.py;
do
    ct=$(($ct + 1));
    echo;
    echo == File $ct: "$i" ==;  # print file name and its counter.
    echo;
    cat "$i";
done

This second shell script is improved from the above, by adding a table of contents (a list of all files) to the beginning of output.

#!/bin/bash

#
# Dump multiple files to one text file.
# By: HomeTom. 3/11/2015
#

cd my_dir;  # go to target file directory.

# define file list.
arr=(README.md manage.py my_dir/*.py);

# 1) get table of contents.
echo Table of Contents;
echo;
ct=0;  # keep a counter of file.
for i in ${arr[@]};
do
    ct=$(($ct + 1));
    echo == File $ct: "$i";  # print file name and its counter.
done

# 2) get file content.
ct=0;  # keep a counter of file.
for i in ${arr[@]};
do
    ct=$(($ct + 1));
    echo;
    echo == File $ct: "$i" ==;  # print file name and its counter.
    echo;
    cat "$i";
done

DOS Batch file to get the number of rows in all the tables and views in a MSSQL database

Store the code below as a bat file, can obtain the number of rows in all the tables and views.--
--

 @echo off

  set db=Northwind

  FOR /F "skip=2" %%G IN ('sqlcmd -E -S localhost -d %db% -Q "set nocount on; set ansi_warnings off; select TABLE_NAME from INFORMATION_SCHEMA.TABLES order by table_name;"') DO (
    REM echo dump table %%G ..
    FOR /F "skip=2" %%X IN ('sqlcmd -E -S localhost -d %db% -Q "set nocount on; set ansi_nulls on; set ansi_warnings on; select count(*) from %%G;"  -s "," -W') DO (
        echo [%%G]: %%X
    )
  )

DOS batch file to dump all tables/views in a MSSQL database

Store the code below as a bat file: dump_csv.bat.
--

@ECHO OFF
REM This script dumps all tables in the given database in CSV format.
REM Author: X.C. 3/6/2014
CLS

set db=Northwind
set mode=%1

IF NOT "%mode%" == "all" set mode=test

echo -----------------------------------------------------------------
echo This script dumps all tables in the given database in CSV format.
echo Author: X.C. 3/6/2014
echo.
echo Usage: dump_csv.bat [mode]
echo   If mode = all, the entire tables are dumped.
echo   If mode = test, only first row of each table is dumped.
echo -----------------------------------------------------------------
echo.
echo Mode: %mode%
echo Database: %db%
echo.
echo ==Tables to dump==

FOR /F "skip=2" %%G IN ('sqlcmd -E -S localhost -d %db% -Q "set nocount on; set ansi_warnings off; select TABLE_NAME from INFORMATION_SCHEMA.TABLES order by table_name;"') DO Echo %%GG

echo.
choice /m "Do you want to continue "
if errorlevel 2 goto Lexit

echo.

IF "%mode%" == "test" (
  FOR /F "skip=2" %%G IN ('sqlcmd -E -S localhost -d %db% -Q "set nocount on; set ansi_nulls on; set ansi_warnings on; select TABLE_NAME from INFORMATION_SCHEMA.TABLES order by table_name;"') DO (
    echo dump table %%G ..
    sqlcmd -E -S localhost -d %db% -Q "set nocount on; set ansi_warnings off; select top 1 * from %%G;" -o output/%%G.txt -s "," -W
  )
)

IF "%mode%" == "all" (
  FOR /F "skip=2" %%G IN ('sqlcmd -E -S localhost -d %db% -Q "set nocount on; set ansi_nulls on; set ansi_warnings on; select TABLE_NAME from INFORMATION_SCHEMA.TABLES order by table_name;"') DO (
    echo dump table %%G ..
    sqlcmd -E -S localhost -d %db% -Q "set nocount on; set ansi_warnings off; select * from %%G;" -o output/%%G.txt -s "," -W
  )
)

:Lexit
echo.
pause


REM Note 1: If do "set ansi_nulls off; set ansi_warnings off", then view information cannot be retrieved.
REM Note 2: To obtain table values only, use: where TABLE_TYPE = 'BASE TABLE'
REM             To obtain view values only, use: where TABLE_TYPE = 'VIEW'


To dump just one table T_data from a database DB, use:

echo dump table T_data ..
sqlcmd -E -S localhost -d DB -Q "set nocount on; set ansi_warnings off; select * from T_data;" -o output/T_data.txt -s "," -W

To bulk insert data from generated dump files, use the command below. Note the "FIRSTROW" is 1-based and not 0-based. In the dump file, the first row will be column names, the second row is separator line "----", so data starts from the 3rd line.

USE [DB]
GO

BULK INSERT [T_data]
    FROM 'C:\\output\\T_data.txt'
    WITH
    (
        FIRSTROW = 3,
        FIELDTERMINATOR = ',',
        ROWTERMINATOR = '0x0a'
    )
GO

Python code to log in a site

Below is Python code to log in a site and submit a page by post:

"""
# Script to log in to website and store cookies.
# run as: python web_login.py USERNAME PASSWORD
#
# from: http://martinjc.com/2011/06/09/logging-in-to-websites-with-python/
#
# sources of code include:
#
# http://stackoverflow.com/questions/2954381/python-form-post-using-urllib2-also-question-on-saving-using-cookies
# http://stackoverflow.com/questions/301924/python-urllib-urllib2-httplib-confusion
# http://www.voidspace.org.uk/python/articles/cookielib.shtml
#
# mashed together by Martin Chorley
# modified by HomeTom
#
# Licensed under a Creative Commons Attribution ShareAlike 3.0 Unported License.
# http://creativecommons.org/licenses/by-sa/3.0/
"""

import urllib, urllib2
import cookielib
import sys

class WebLogin(object):

    def __init__(self, username, password):
      
        # url for website we want to log in to
        self.base_url = 'http://baseurl.com'
        # login action we want to post data to
        # could be /login or /account/login or something similar
        self.login_action = '/account/login.php'
        # file for storing cookies
        self.cookie_file = 'login.cookies'

        # user provided username and password
        self.username = username
        self.password = password

        # set up a cookie jar to store cookies
        self.cj = cookielib.MozillaCookieJar(self.cookie_file)

        # set up opener to handle cookies, redirects etc
        self.opener = urllib2.build_opener(
            urllib2.HTTPRedirectHandler(),
            urllib2.HTTPHandler(debuglevel=0),
            urllib2.HTTPSHandler(debuglevel=0),
            urllib2.HTTPCookieProcessor(self.cj)
        )

        # pretend we're a web browser and not a python script
        self.opener.addheaders = [('User-agent',
            ('Mozilla/4.0 (compatible; MSIE 6.0; '
            'Windows NT 5.2; .NET CLR 1.1.4322)'))
        ]

        # open the front page of the website to set and save initial cookies
        response = self.opener.open(self.base_url)
        self.cj.save()

        # try and log in to the site
        response = self.login()
        #print response.read()

        data = urllib.urlencode({
            'fieldName1' : 'fieldValue1',
            'fieldName2' : 'fieldValue2',
            'btnSubmit'  : "submit"
        })

        response = self.opener.open("http://baseurl.com/func.php", data)
        print response.read()

      
    # method to do login
    def login(self):

        # parameters for login action
        # may be different for different websites
        # check html source of website for specifics
        login_data = urllib.urlencode({
            'username' : self.username,
            'password' : self.password,
            'btnLogin' : "submit"
        })

        # construct the url
        login_url = self.base_url + self.login_action
        # then open it
        response = self.opener.open(login_url, login_data)
        # save the cookies and return the response
        self.cj.save()
        return response


if __name__ == "__main__":

    args = sys.argv

    # check for username and password
    if len(args) != 3:
        print "Incorrect number of arguments"
        print "Argument pattern: username password"
        exit(1)

    username = args[1]
    password = args[2]

    # initialise and login to the website
    WebLogin(username, password)

A Chinese poem processor

A lot of language games can be done. I am also thinking of doing some Natural Language Processing stuff later.
The code below can be improved significantly in the data processing part. For example, on things such as 1) length of poem sentence, 2) position of target character, 3) tone (平仄), 4) semantic analysis so the mood matches.
More poem source files can be added.
The only tricky thing here so far is handling UTF8 characters.

#
# This script reads Chinese poems and store the sentences into a repository,
# then find poem sentences that start with letters in the given target sentence.
#
# This can be used for some language games, 
# such as forming a poem for somebody's birthday in the form of a "藏头诗".
#
# This script should be saved in utf8 format. 
#
# http://ahinea.com/en/tech/perl-unicode-struggle.html
# http://stackoverflow.com/questions/519309/how-do-i-read-utf-8-with-diamond-operator
# http://stackoverflow.com/questions/9574198/comparing-two-unicode-strings-with-perl
# 
# Chinese poems:
# http://www.shuku.net/novels/mulu/shici.html
#
# 藏头诗 generator: http://www.zhelizhao.com/cangtoushi/
#
# By: HomeTom
# Created on: 2012/09/04
#

require Encode;
use utf8;
use strict;

##################
# Change setting here.
##################

# Data source
my @files = ("tang300.txt", "song100.txt");
# Target sentence
my $target = ("小明生日快乐");

##################

my $cnt = 1;
my $len;
my @chars;
my @first = (); # first char.
#my $char;
my $DEBUG1 = 0;
my $DEBUG2 = 0;
#print "hi\n";

#binmode STDIN, ":utf8";
binmode STDOUT, ":utf8";

##################
# Read data.
##################

my @lines = ();
my @lines2;

foreach my $file (@files) {
  open FILE, $file or die $!;
  @lines2 = <FILE>;
  close FILE;
  push(@lines, @lines2);
}

##################
# Process and analyze data.
##################

foreach my $line (@lines) {
  chomp($line);
  $line = trim($line);
  $line = Encode::decode_utf8($line);
  @chars = split //, $line;

  if ($DEBUG1) { 
    print "$cnt: $line\n"; 
    print "$cnt: "; 
    print "$_." foreach (@chars); 
    print "\n"; 
  }

  #if ($chars[0] ne "") 
  {
    push (@first, $chars[0]);
  }

  $cnt ++;
}

if ($DEBUG2) {
  print "$_\n" foreach (@first);
}

$len = @first;


##################
# Search poem sentences for each word in the target sentence.
##################

my @words = split //, $target;

foreach my $w (@words) {
  print "==$w==\n";
  getLetterLines($w);
}


1;


##################
# Subroutines
##################

sub getLetterLines() {
  my $i;
  my $c;
  my $letter = shift;
  for ($i = 0; $i < $len; $i ++) {
    $c = $first[$i];
    if ($c eq $letter) { print "$i: $lines[$i]\n"; }
  }
  #print "\n";
}


#
# Trim functions, from: http://www.somacon.com/p114.php
#

# Perl trim function to remove whitespace from the start and end of the string
sub trim($)
{
  my $string = shift;
  $string =~ s/^\s+//;
  $string =~ s/\s+$//;
  return $string;
}
# Left trim function to remove leading whitespace
sub ltrim($)
{
  my $string = shift;
  $string =~ s/^\s+//;
  return $string;
}
# Right trim function to remove trailing whitespace
sub rtrim($)
{
  my $string = shift;
  $string =~ s/\s+$//;
  return $string;
}

Some vbscript utilities

Here are some vbscript code examples.

--Table of contents--
1. Send email
2. Get computer name
3. Connect to database and execute query
4. Restart a windows service
5. Use of a log file
--

1. Send email

Sub sendEmail(ByRef text)
  Dim objMessage
  Set objMessage = CreateObject("CDO.Message")
  objMessage.Subject = "..."
  objMessage.From = "..."
  objMessage.To = "...@..."
  objMessage.TextBody = "...text..."

  objMessage.Configuration.Fields.Item ("http://schemas.microsoft.com/cdo/configuration/sendusing")=2 
  objMessage.Configuration.Fields.Item ("http://schemas.microsoft.com/cdo/configuration/smtpserver") = "..." 
  objMessage.Configuration.Fields.Item ("http://schemas.microsoft.com/cdo/configuration/smtpserverport") =25 
  objMessage.Configuration.Fields.Update

  objMessage.Send
End Sub

2. Get computer name

Dim objShell, strComputer
Set objShell = CreateObject("WScript.Shell")
strComputer = objShell.ExpandEnvironmentStrings( "%COMPUTERNAME%" )
Set objShell = nothing

3. Connect to database and execute query

Option Explicit

on error resume next
Set objShell = CreateObject("WScript.Shell")

Set objConnection = CreateObject("ADODB.Connection")
objConnection.Open "Provider=SQLOLEDB.1;Data Source=localhost;Initial Catalog=<db>","<username>","<password>"

if err.number <> 0 then
    WScript.Echo "error " & hex(err.number) & ": " & err.description 
else
    WScript.Echo "value = " & getVal()
    objConnection.close
end if
on error goto 0

Set objConnection = Nothing

Set objShell = nothing
WScript.Quit(0)

Function getVal()
    Dim sql, strWatchFile
    sql = "select val from table"
    set strWatchFile = objConnection.Execute( sql )
    if err.number <> 0 then
        WScript.Echo hex(err.number) & vbcrlf & err.description 
        getVal = -1
        Exit Function
    else
        getVal = strWatchFile(0).value
        strWatchFile.close
    end if
    set strWatchFile=nothing
End Function

4. Restart a windows service (by calling the batch file below)

objShell.Run "restart.bat ""<service name>""", 0, true

The three parameters are:
  1) string: shell command, a batch file in this case (plus its parameter).
  2) int: 1 - show window, 0 - hide window.
  3) boolean: true - wait until the shell command ends, false - do not wait.

5. Use of a log file

Dim objFileSystem, objLogFile, logFileName, useLog

useLog = True
logFileName = "C:\mylog.log"
Set objShell = CreateObject("WScript.Shell")

openLog()
writeLog("hello world")
closeLog()

Set objShell = nothing
WScript.Quit(0)

Sub openLog()
  If NOT useLog Then Exit Sub
  Const OPEN_FILE_FOR_APPENDING = 8
  Set objFileSystem = CreateObject("Scripting.fileSystemObject")
  If NOT objFileSystem.FileExists(logFileName) Then
    Set objLogFile = objFileSystem.CreateTextFile(logFileName, TRUE)
  Else
    Set objLogFile = objFileSystem.OpenTextFile(logFileName, OPEN_FILE_FOR_APPENDING)
  End If
End Sub

Sub writeLog(txt)
  WScript.Echo txt
  If useLog Then objLogFile.WriteLine(date & " " & time & ": " & txt)
End Sub

Sub closeLog()
  If NOT useLog Then Exit Sub
  objLogFile.Close
  Set objLogFile = Nothing
  Set objFileSystem = Nothing
End Sub

Restart windows service in a batch file

::
:: usage: restart.bat <service name>
:: Note quotation marks can be used if the service name contains space.
::
:: This batch script restarts a windows service (given as a parameter of the batch file).
:: It does this by:
:: 1) issue a stop command;
:: 2) check the status of the service, go back to 1) if it is not stopped yet;
:: 3) when the service has been stopped, restart it.
::
:: Note: the provided service MUST exist, otherwise it will get into an infinite loop.
::
:: References:
:: [1] http://serverfault.com/questions/25081/how-do-i-restart-a-windows-service-from-a-script
:: [2] http://boards.straightdope.com/sdmb/showthread.php?t=458812
:: [3] http://www.robvanderwoude.com/errorlevel.php
::
::

@ECHO OFF

if [%1]==[] goto end

:stop
sc stop %1

rem cause a ~2 seconds sleep before checking the service state
ping 127.0.0.1 -n 2 -w 1000 > nul

sc query %1 | find /I "STATE" | find "STOPPED"

if errorlevel 1 goto :stop
goto :start

:start
sc start %1

:end

Command line utilities

Many GUI functions have command line counterparts. This makes it possible to write scripts to automate a lot of tasks. The ones here are for the windows platform, specifically here we are talking about windows host script (*.vbs).

1) For SVN, slik svn is a command line version.

2) The sqlcmd command enables running SQL script from command line.

E.g., this runs eg.sql from command line.
sqlcmd -S localhost -d database_name -i eg.sql

This lists all databases on localhost:
sqlcmd -S localhost -i list_databases.sql

list_databases.sql:
select distinct db_name(database_id) AS DATABASE_NAME from sys.master_files group by database_id;
GO

So if you want to do something to certain databases in a server, you can use this to get a list of databases, and check which ones are you need, then construct sql command dynamically.

Just enter sqlcmd, you enter the interactive mode.

3) Build a visual studio solution from command line:

"c:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\devenv.com" c:\projects\Project1\Project1.sln /build Debug /out

Or:

C:\Windows\Microsoft.NET\Framework64\v3.5\MSBuild.exe c:\projects\Project1\Project1.sln

Script can invoke other executables. Executables (e.g. windows service) can invoke scripts. Both can also capture each other's output. This makes rich interaction.

Windows service uses socket in usual. It can use windows remoting (TcpClientChannel) as well.

The following installs a windows service:

C:\Projects\wsService1\wsService1\bin\Debug>InstallUtil /LogToConsole=true wsService1.exe

Adding the /u switch will uninstall it.

When install the windows service under a user account, you will be prompted for the account name and password. The name should be [domain_name]\account_name. If it is a local user (i.e., no domain_name), then it should be .\account_name. Not doing this causes failure.

4) The WMIC command.

The WMIC command is a powerful command line tool to get all kinds of system information, from running processes to OS related. See here for examples.

5) FileSystemObject.copyFolder and XCOPY

FileSystemObject.copyFolder fails when some target files are readonly, this is so even when the OverwriteExisting argument is set to true. The solution is to use XCOPY instead. You may want to use XCOPY this way:

start /WAIT /B XCOPY %1 %2 /R /Y /E /H /Q

/WAIT - wait until this finishes.
/B - do not open another dos window.
For the rest, use XCOPY /? for details. See here for more details.