This are my notes in the fields of computer science and technology. Everything is written with ABSOLUTE NO WARRANTY of fitness for any purpose. Of course, feel free to comment anything.

Thursday, December 31, 2009

ssh autocompletion in tcsh

I wrote the following for my .tcshrc to get some autocompletion for ssh.
set hostlist=(`cat ~/.ssh/known_hosts | cut -f 1 -d ' ' | cut -f 1 -d ',' | uniq`)
complete ssh 'p/1/$hostlist/' 'p/2/c/'

Saturday, December 12, 2009

TestDisk

Hard disk failure recovery tool; tries to repair partition tables, boot sectors, MBR; allows to read and copy files from lost partitions; an undelete feature is also included.

Homepage: http://www.cgsecurity.org/wiki/TestDisk
Developer: Christophe Grenier
Version: 6.11 (April 19, 2009)
License: open source free software
Platform: Linux/Windows/Mac

Friday, December 11, 2009

Inkscape

Vector graphics program, similar to commercial packages like Adobe Illustrator, with advanced editing features and importing/exporting in a lot of different formats (e.g. PDF). The native format is the open format SGV.

Homepage: http://www.inkscape.org/
Developer: The Inkscape Team
Version: 0.47 (November 21, 2009)
License: open source free software
Platform: Linux/Windows/Mac

Sunday, October 11, 2009

Starting ssh-agent from .tcshrc

While connecting to a server using SSH, you need either to provide a password or to use a public-private key pair to identify yourself. The latter is interesting, as you don't have to write your password each time. A copy of the public key must be saved in the computer acting as ssh server under ".ssh/authorized_keys", the private key stays only on the client computer.

However for security reasons it is strongly reccomended to set a password on the key file. Otherwise anyone who can steal your key somehow (like accessing your computer physically), can steal your identity. But if your key file has a password, then you would have to write the password each time, so it wouldn't be much better than using passwords instead of keys in the first time. Luckily there is a solution: ssh-agent and ssh-add. ssh-agent runs as a deamon and remembers the password to your key. ssh-add allows you to tell ssh-agent the password.

As ssh-agent keep running when you log off your session, you may add some lines to your shell startup script, to test if ssh-agent is running, otherwise to start it and ask you the password (via ssh-add). In the github guides there is a script for bash (). I adapted it to use it in tcsh, which I use as a shell. Here is it (I added these lines at the bottom of my .tcshrc file, only *before* setting the prompt explicitely, as I used it to distinguish interactive sessions from non-interactive ones):

# start ssh agent if not started yet
set SSH_ENV = "$HOME/.ssh/environment"
unset SSH_AGENT_RUNNING
if (-f $SSH_ENV) then
source $SSH_ENV
ps $SSH_AGENT_PID > /dev/null && set SSH_AGENT_RUNNING
endif
if ($?prompt && ! $?SSH_AGENT_RUNNING) then
echo "Initializing SSH agent..."
/usr/bin/ssh-agent | sed 's/^echo/#echo/' >! $SSH_ENV
chmod 600 $SSH_ENV
source $SSH_ENV
echo "done: ssh-agent is now running, PID $SSH_AGENT_PID"
/usr/bin/ssh-add
endif

Sunday, September 13, 2009

My virtual linux server

In the windows installation on my laptop, I also run a linux server in background. Here is how I setup my virtual linux server.

As a virtualization software I installed Sun VirtualBox. I created a virtual machine. I gave it a little less than the half of my memory and turned on support for AMD-V extensions (as I did also in my host machine BIOS). As HD controller I selected SATA (the default is IDE). For networking I use the NAT configuration. I forwarded the SSH port as described in the virtual box manual (backlinux is the name of my virtual machine):
VBoxManage setextradata "backlinux" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/guestssh/Protocol" TCP
VBoxManage setextradata "backlinux" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/guestssh/GuestPort" 22
VBoxManage setextradata "backlinux" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/guestssh/HostPort" 2222
Then I followed the suggestion of Sebastian Bauer's blog: I installed the ntwind software's hidden start tool (homepage) and created two one-liner batch files that I use to start and stop the virtual machine. There are also more sofisticated approaches to run the machine as a windows service, however I found this very simple and I chose it for this reason.

Here is the start.bat file: (paths are specific to my machine)
"c:\Program Files\standalone\hstart\hstart" /NOCONSOLE ""c:\Program Files\Sun\VirtualBox\vboxheadless" -s backlinux -vrdp off"
while the stop.bat file contains:
"c:\Program Files\Sun\VirtualBox\VBoxManage.exe" controlvm backlinux savestate
I created two hard disk images for the machine, called backlinux.vdi and backlinux.home.vdi. I set both to dynamical 8 Gb. I later attached backlinux.home to /home. I keep /home on a separate disk, so that I can make use of the snapshots feature of Virtualbox, without loosing the data in /home. To do this I changed the type of backlinux.home.vdi to write-through using:
"c:\Program Files\Sun\VirtualBox\VBoxManage.exe" modifyhd backlinux.home.vdi --type writethrough
Then I installed ubuntu-server (jaunty, i.e. 9.04) with the option "minimal virtual machine" (press F4 by the installation) and I added some software packages I need/like using aptitude (for example: libncurses-dev, libcairo2-dev, ruby1.9, git-core, gitk ... and relative dependences). The machine has no x-server installed, but I have an X server installed under Windows (Xming), so I can start also graphical software using X-forwarding.

To allow the virtual machine to access the host disk (C:), I installed the VBoxLinuxGuestAdditions from the shell script in the virtual CD image (.iso) provided with virtual box (you need to install linux-headers package for your kernel for that). Then I created a shared folder "c" using the virtual box gui. In Linux then I mounted the folder, creating a /C folder, adding this line to /etc/fstab:
c    /C    vboxsf
As last step, I created two icons starting from NewTux.svg of Wikimedia Commons (author: Larry Ewing). I resized it to a 48px square and overlayed a green triangle (start icon) and a red square (stop icon). I created then links to my batch files to my desktop and assigned the icons to the links.

Here is the result:

VirtualBox

There are several available platform virtual machines available. I generally use the Sun VirtualBox. I usually run a Linux server in my Windows-running laptop, using a virtual hard disk for it and accessing it with PuTTY and WinSCP.

VirtualBox is able to run a virtual machine also without an output terminal/window (headless). I use this for my Linux server.

Another possibility it offers is to run a machine booting from a real partition instead of a virtual hard disk. However this makes things quite difficult and is an experimental feature. The hardware found by the operating system will be different in a virtual box and booting as host, so this will lead to several problems. For example I was unable to run a deployed Vista from Linux and I could only run a deployed Ubuntu in Windows with command line interface.

Developer: Sun
Version: 3.0.6 (September 9, 2009)
Platform: Cross-platform

PuTTY

A terminal emulator using ssh or telnet connections to remote servers. I always use this for ssh connections from a Windows machine to a Linux machine (usually using WinSCP for file exchanges) - including my virtual linux server I run in background.

In Windows the program settings (including the saved sessions) are stored in the registry at the position: Computer\HKEY_CURRENT_USER\Software\SimonTatham\PuTTY .

Developer: Simon Thatam
Version: 0.60 (April 29, 2007)
Platform: cross-platform
License: open source

Saturday, September 12, 2009

DVD43

In order to rip/transcode DVD you own, for example to see the videos on your mobile device, this tool may be useful. It circumvents copy protection and zone settings. It may be illegal under certain circumstances and jurisdictions, however the tool itself is probably legal, if you don't engage in llegal activities with it.

Developer: unknown, maybe for legal reasons (?) [whois information]
Version: 4.4.1 (June 29, 2009)
License: closed source freeware
Platform: Windows

HandBrake

This software comes in hand, when you want to upload videos to a mobile phone or media player. It allows to transcode easily to mp4 format with a good control of options, encoder to use, resizing...

Here is my settings file (user_presets.xml), which I use to convert files to display them in my phone:
<?xml version="1.0"?>
<ArrayOfPreset xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<Preset>
<Level>0</Level>
<Name>Mobile phone</Name>
<Query> -f mp4 -w 400 -e ffmpeg -q 0.5 -a 1 -E faac -B 160 -R 0 -6 stereo -D 1 -m -v </Query>
</Preset>
</ArrayOfPreset>
My settings are: ffmpeg encoding, mp4 container format, aac stereo sound, 50% constant quality, resize to 400px width. Note that to be able to create a preset under Vista you need to start the application with administrative rights.

Platform: cross-platform
Version: 0.9.3 (Nov 23, 2008)
License: open source freeware

VLC media player

This media player allows to reproduce a wide spectrum of media files. It is extremely flexible and has a lot of advanced features, probably more than any other media player. The graphical interface (which is one of the different interfaces it has) is nevertheless very simple and lightweight.

You can select the GUI language using Ctrl+P and changing the first option in the dialog.

Homepage: http://www.videolan.org/vlc/
Developer: VideoLAN project
Version: 1.0.1 (July 27, 2009)
Platform: cross-platform
License: freeware, open source

SyncToy

As a simple backup strategy under Windows, I synchronize my important folders with a copy on an external drive. To do that I am using SyncToy, a free-to-download tool from Microsoft.

Download link: http://www.microsoft.com/downloads/[...]
Developer: Microsoft
Version: 2.0 (August 15, 2008)
Platform: Windows
License: freeware, closed source

Windows Alpha Blender

This tool allows to set the tranparency of a windows using a shortcut to increase (default: Ctrl+F11; my setting: Ctrl+<) or decrease (default: Ctrl+F12; my setting: Ctrl+Shift+<) the alpha transparency value of the currently active window. It is not very sofisticated, but it can be useful sometimes.

Personal settings (ie the keyboard shortcuts) are kept in the registry.

Homepage: http://www.vaultmate.com/freewaregifts.php
Developer: VaultMate Software
Version: 1.2 (June 7, 2008)
License: freeware, closed source
Platform: Windows

Colors in grep

To turn on colors in grep, you may add --color=auto to the GREP_OPTIONS environment variable. The way to do it is of course shell-dependant. E.g. add the following to your shell rc file:
export GREP_OPTIONS='--color=auto'

Unlocker

This tool allows to identify the process, which is currently locking a given file or directory and if desidered to force unlocking of the file from one or all processes. It works as a shell extension in explorer. Note that under Vista the program requires a confirmation to be allowed to access a file (if the UAC system is on).

Developer: Cedrick Collomb
Version: 1.8.7 (May 1st, 2008)
Platform: Windows
License: freeware, closed source

WinSCP

WinSCP is the perfect windows tool to connect to remote servers and exchange (download/upload) files or edit them. It is a SFTP and FTP client.

There is a standalone version and an installer based setup; if using the latter, beware that it will ask you install additional software using the OpenCandy system. However at least the default is not to install any additional software.

I like to store the preferences in ini file instead of the registry, so that I can export them easily to other computers. To turn on the ini preferences storage: in WinSCP login dialog select "preferences", then "Other general options: Preferences", "Storage" and change the "configuration storage" value to "INI file". The default location of the ini file under Windows Vista will be: "C:\Users\\AppData\Local\VirtualStore\Program Files\WinSCP". Under older Windows version it will be directly in the application directory (C:\program files\WinSCP).

Version: 4.2.3 beta (August 18, 2009)
Platform: Windows
License: freeware

ls colors

add an alias to your shell rc file:
alias ls "ls --color=auto"

Friday, September 11, 2009

Gimp

Gimp is great if you cannot afford on do not want to pay a lot of money for professional software (i.e. Photoshop). According to some it *is* a professional level software, according to other not yet; however it has more than enough features for amateur photo manipulation and other image related tasks (e.g. icon editing, image conversion, resizing etc). The first version was realeased in 1996 by two computer science students (Kimball and Mattis, currently Google engineers).

It is quite complicated to use and probably extreme complicated to master, however it is intuitive enough for simple tasks. Since version 2.6 they invested in an effort to make the interface easier to use (see Wikipedia article). An user manual and documentation page is also available.

If you want a different language than your operating system, here is the howto. In my Vista laptop I wanted the english version, so I did the following: I created a .bat file in C:/users/program files/GIMP2.0/bin (you need admin privileges to write it there) with the following content:
set lang=en
start gimp-2.6.exe
Then I changed the startmenu entry to point to the bat file instead of the exe file (other settings: execute minimized, use icon of the .exe file).

Name: Gimp
Version: 2.6.7 (August 14, 2009)
License: free, open-source software
Platform: cross-platform

JDiskReport

There are for sure a lot of disk space analysis tools, but I liked this German Java application a lot: JDiskReport. It is relatively fast and displays a lot of additional information. The space usage or number of files information can be grouped by location on the file system, but also by file size, modification time or file extension. All information can be displayed as pie chart, histogram or detailed list.

Name: JDiskReport
Developer: Karsten Lentzsch (JGoodies, www.jgoodies.com)
Current version: 1.3.1 (July 30, 2008)
License: freeware, partially closed source; open source release planned [see here]
Platform: cross-platform

Tuesday, August 11, 2009

git: write a single commit to file

Write the commit to file:
git diff-tree <commit> -p --pretty > <filename>
You can also use git format-patch, but that is more specialized for patches to be submitted by email.

Apply the commit from the file:
git apply <filename>
Here is an excerpt from the git diff-tree manual page:
When "git-diff-index", "git-diff-tree", or "git-diff-files"
are run with a -p option, "git diff" without the --raw option,
or "git log" with the "-p" option, they do not produce the output
described above; instead they produce a patch file. You can
customize the creation of such patches via the GIT_EXTERNAL_DIFF
and the GIT_DIFF_OPTS environment variables

Monday, August 10, 2009

Scite: add a vertical line at column 80

A tip for the SciTE editor: this is interesting if you want your code to be easy to print and display without horizontal scrolling.

Add the following to the global or user properties file:
edge.column=80
edge.mode=1
edge.colour=#C0DCC0
This is an excerpt of the SciTE documentation for the edge property:
Indicates long lines. The default edge.mode, 0, does not
indicate long lines. An edge.mode of 1 uses a vertical
line to indicate the specified column and an edge.mode
of 2 changes the background colour of characters beyond
that column. For proportional fonts, an edge.mode of 2
is more useful than 1.

Sunday, August 9, 2009

Compact guide to less commands

Movement

Arrows, page up/down, home/end keys all work as expected. In addition the following are sometimes useful:
d/uMove half page forward / backward
20gMove to line 20
20%Move to the 20% point
ma/'aset markpoint "a" / move to markpoint "a"
(also instead of "a" any other lowcase letter)

Search

/patternsearch pattern forward
?patternsearch pattern backward
nnext match
Nprevious match
/!patternsearch non-matching line
Esc-uturn match highlighting on/off

Misc

=show info about the file and current position
! commandexecute command and show its output
s filenamesave to file (if the input was a pipe)

GIT_PAGER and LESS colors

I was wondering why, working on two different machines, I was able to display git-log and git-diff colors on one machine and not on the other, despite the same git configuration file ~/.gitconfig:

[color]
diff = auto
status = auto
branch = auto
log = auto

and no particular settings regarding colors in <repository>/.git/config

Although I was not using a Mac, the solution was in this blog post: that is, setting the GIT_PAGER variable to cat was already solving the problem; GIT_PAGER sets which program should be used by git to display for example the diff and log output (default less).

The actual problem was in less and how it outputs ANSI color ESC codes. A solution, similarly to what suggested in that post, was to set the LESS variable to -R and leave GIT_PAGER unset, as it originally was. Regarding the other options suggested in the blog post for LESS, -X and -e have no influence on color display, and -R is probably better than -r; in particular, this in an extract of the less manual, at the -r/-R section, explaining why:
-r or --raw-control-chars

Causes "raw" control characters to be displayed.
[...] Warning: when the -r option is used, less cannot
keep track of the actual appearance of the screen
(since this depends on how the screen responds to
each type of control character). Thus, various display
problems may result, such as long lines being split
in the wrong place.

-R or --RAW-CONTROL-CHARS

Like -r, but only ANSI "color" escape sequences are
output in "raw" form. Unlike -r, the screen appearance
is maintained correctly in most cases. [...]

Thursday, July 23, 2009

Conversion of CR+LF to LF

The following will convert all files also in subdirectories
exept for those starting with a dot (skips also such directories).
This works if dos2unix is available, of course; otherwise use sed.

find ^.* -exec dos2unix '{}' \;

I needed it to convert the content of a git repository (that's
why I skipped the . files: I didn't want .git/* files to be included).
Of course you can write better matching chriteria, but this worked
fine for my case.

Thursday, July 16, 2009

Tiny implementation of a suffix array in Ruby

class SuffixArray

attr_reader :suf, :string

def initialize(string)
@string = string
@suf = (0..string.size-1).sort_by{|i|@string[i..-1]}
end

end

Monday, April 20, 2009

Count distinct values in all columns of a table

The following rake task will show the number of different values of each column of a given table in the database. I wrote it to see statistics on which columns are really used.
namespace :db do
desc "Count distinct values for each non empty column of a table (TABLE=xxx) "+
"optionally using a condition (WHERE=\"yyy\")"
task :count_distincts => :environment do
raise "Specify option TABLE=<table_name>" unless ENV["TABLE"]
puts "Distinct values in table #{ENV['TABLE']}:"
puts "(condition: where #{ENV['WHERE']})" if ENV['WHERE']
c = ActiveRecord::Base.connection
columns = c.select_all("describe #{ENV['TABLE']};").map{|a|a["Field"]}
columns.each do |col|
sql = "select count(distinct #{col}) as c from #{ENV['TABLE']}"
sql << " where #{ENV['WHERE']}" if ENV['WHERE']
n = c.select_one(sql)['c']
puts "#{col}: #{n}" unless n == "0"
STDOUT.flush
end
end
end

Thursday, April 16, 2009

assert.h

Assertions: you express your expectations in a certain position of the code, and this way you avoid bugs that would otherwise very difficult to trace.

In C use the standard library macro assert. You must #include <assert.h>, then you assert(expression); in the code, and if the expression is evaluating to 0 / false, the program aborts with an error msg printed on sterr that tells you which assertion fails and where it is (file/line).

It is also possible to turn assertions off, adding a #define NDEBUG before the #include <assert.h>.

Tuesday, April 14, 2009

Why main is int and not void.

The standard says so. And it actually makes a difference: here is a detailed discussion about this topic: http://users.aber.ac.uk/auj/voidmain.shtml

Why do you have to pass scanf the address of the variable to write in?

As I was learning C the first time, I remember I was asking this myself... I would have preferred it to return the scanned values, something like:

myVar = scanf("%s") /* don't do this :) */

but of course, in this case (1) you could assign only 1 variable, (2) you would have not had the return value (number of read items)...

Here is some discussion of it: ...

Turn colors on in vi/vim

If the color syntax highlighting is off, you can turn it on by editing (or creating) ~/.vimrc, adding the following line:

:syn on

Monday, April 13, 2009

square root

When I have some time, I will have a look to this algorithm to calculate the square root of a number:
    int sqrt(int num) {
int op = num;
int res = 0;
int one = 1 << 14; // The second-to-top bit is set: 1L<<30 for long


// "one" starts at the highest power of four <= the argument.
while (one > op)
one >>= 2;

while (one != 0) {

if (op >= res + one) {
op -= res + one;
res += one << 1;
}

res >>= 1;
one >>= 2;
}
return res;
}

(from Wikipedia)

Saturday, April 11, 2009

XSLT

XSLT: take an XML doc and, through a sort of "stylesheet", make something with the data (another XML doc, or perhaps a XHTML doc). I don't know this language but I guess it's rather useful (although I have no use for it at the moment).

related stuff: libxml, libxml2, libxslt, xsltproc

Links:
- specifications@w3: http://www.w3.org/TR/xslt


PragProg media: Expression Engine Techniques

I often have a look at the list of media of "PragProg" (http://www.pragprog.com/categories/all?sort=pubdate). The newest title is the screencast serie: "Expression Engine Techniques"

Well I definitely don't need it, anyway I'm always curious, so here is the info I collected (mainly from Wiki) about the topic:

ExpressionEngine http://expressionengine.com/ is a CMS, developer: EllisLab, there is a free version, and two paid ones. A "2.0" is expected in 2009, will be based on CodeIgniter, which is a PHP framework.  


capistrano using git

The cap deploy:update task when you are using git runs the following git command: 

git checkout -q -b deploy 3fe75somehash.....

meaning : 

- creates a new branch named "deploy" (what happens if there is already one called like that?)
- the source of the branch is the commit identified by the given hash
- -q option is "quiet mode"

rsync and symbolic links

some possible behaviours of rsync (from the man page):

(1) SKIP:
default case => symlinks are simply not followed

(2) COPY THE LINK:
"symlinks are recreated with the same target on the destination"
rsync --links
also: rsync --archive implies --links

(3) FOLLOW THE LINK
rsync --copy-links

safe/unsafe:

relevant for case 3 is the "safe" vs. "unsafe" difference which I did not understand good.
This is what is written: "An example where this might be used is a web site mirror that wishes ensure the rsync module they copy does not include symbolic links to /etc/passwd in the public section of the site. Using --copy-unsafe-links will cause any links to be copied as the file they point to on the destination. Using --safe-links will cause unsafe links to be ommitted altogether."

Always back up on a dedicated partition or disk!

My home dir on a certain server was backed up by another company. I set under my home a soft link to another location (several Gb of data). What I didn't know: their backup script was rync-based *with* the option --copy-unsafe-links. Rsync followed the link and backed up tons of stuff clugging the backup hard disk. They used the same hard disk and the same partition also for another function: email server. So that softlink disrupted the email server! At first I though I was to be blamed, but after a second thought, I understood, that it's not 100% true: they should have *never* run the mail server on the same partition where they backed up that stuff!


Ruby and GUIs

Is there a good ruby GUI toolkit?

I want to write a simple app, with a simple DB backend, just to keep some notes. As I want it to run on my laptop, I don't want any webserver running, so no rails app (and also because I want to do some gui programming, it's years I am only working on console or web apps). 

I started earlier this morning my Google quest to answer this question, I am not so far yet. I am no expert in GUIs. 

Some disorganized thoughts:

- Java of course is a good choice, isn't it? Yes I want to learn Java again (I learned it 10 years ago, and not using it a long time, so I guess my knowledge is totally out-of-date). But now I want just to write a little small app in that simple little lovely ruby language. 

- I guess most Win apps are based on "native" widgets. For that maybe you need VS or the like and I am in this moment in no mood to be a MS fan. And I want something cross-platform. 

- There a libraries, and like always in IT a lot of names and acronymes, just to confuse stupid newbies like me. QT is one, I think, then there are others (I think some GUI library-flame is also the reason why Gnome and KDE are 2 different desktops, isn't it?). So I guess you use one of that with some good ruby bindings and you are on it. Right?

- I read of Shoes: very simple, maybe good idea, but probably just for beginners, I don't know if I want to waste my time with it. Looks like is just something for kids learning programming? 

Blog Archive

About Me

My photo
Hamburg, Hamburg, Germany
Former molecular biologist and web developer (Rails) and currently research scientist in bioinformatics.