Richard Penman

Code paste

2011-12-17T00:00:00+09:00

Use this service for pasting code with syntax highlighting and even test execution:
http://codepad.org

How to make web2py faster

2011-11-26T00:00:00+09:00

Avoid regenerating tables:

  
db = DAL('sqlite://storage.sqlite', migrate=runonce)

Create indexes:

  
db.executesql('CREATE INDEX IF NOT EXISTS {table}_index ON {table} (id);'.format(table=table))

Compile byte code in admin

Serve static files direct from server

Move as much logic as possible from models into controllers and modules

Disable session if possible

Cache database queries and controller functions:

  
@cache(request.env.path_info, time_expire=5, cache_model=cache.ram)
def index():
    db().select(db.log.ALL, cache=(cache.ram, 60))

web2py with uwsgi

2011-11-25T00:00:00+09:00

I found Apache used too much memory so changed to uwsgi:

# build uwsgi  
hg clone http://projects.unbit.it/hg/uwsgi  
cd uwsgi  
make -f Makefile.Py27  
cd ..

# get web2py  
wget http://www.web2py.com/examples/static/web2py_src.zip  
unzip web2py_src.zip  
cd web2py  
# run web2py rocket server to setup password  
python2.7 web2py.py  
# set your password then kill the server  
cp parameters_8080.py parameters_PORT.py  
cd ..

# start uwsgi with 32 MB of memory, 24 threads, 1 process  
./uwsgi/uwsgi --http=127.0.0.1:PORT --pythonpath /path/web2py --module wsgihandler -d /path/uwsgi.log -t 20 --async 24 --ugreen --limit-as 32 -r --no-orphans -M -p 1 --touch-reload /path/uwsgireload.txt --reload-on-rss 50  

# reload server
touch /path/uwsgireload.txt

Let webfaction serve your static content

2011-11-25T00:00:00+09:00

Create a static symbolic link application:

Name: web2py_static  
App category: Symbolic link  
App type: Symbolic link to static-only app  
App doc: This creates a symlink from ~/webapps/ to theabsolute path specified in the extra info field.

Then in your website mount this at /static

Now requests to static files will be handled directly by webfaction and never reach your app.

How to determine what Linux is installed

2011-11-25T00:00:00+09:00

Kernel version:

uname -r  
2.6.18-238.12.1.el5PAE

Distribution name:

cat /etc/issue  
CentOS release 5.7 (Final)

32 or 64 bit:

getconf LONG_BIT  
32

How to create an image mosaic

2011-11-18T00:00:00+09:00

For a birthday present I generated an image mosaic using the metapixel command.
Firstly get atleast 1000 images in [base_image_folder] and sub-directories. Then process the data with:

metapixel-prepare --width=64 --height=96 -r [base_image_folder] [tmp_thumbnails_dir]

This processing will take a while.Then generate the mosaic with:

metapixel -w 48 -h 64 -s 8 --metapixel [image_pattern_for_mosaic] [output_image_file] --library [tmp_thumbnails_dir]

This should only take a few minutes. For different resolutions change the size flag.

Automatically fix rotation of images

2011-11-18T00:00:00+09:00

My camera stores the orientation of the images in the header, but this is not recognized by most programs.
You can automatically fix the image rotations with:

jhead -autorot

And to recursively rotate all images:

find . -name "*.JPG" -exec jhead -autorot -exonly {} \;

GoDaddy domains with webfaction hosting

2011-11-15T00:00:00+09:00

In your GoDaddy account select Set Nameservers and then I have specific name serversfor my domains(Documented here)

Enter webfaction's domain name servers (Documented here):

ns1.webfaction.com  
ns2.webfaction.com  
ns3.webfaction.com  
ns4.webfaction.com

The docs say to wait atleast 2 hours but I found my website worked immediately.

Downloading data from GAE

2011-11-11T00:00:00+09:00

As documented here you can download your data into a blob:

appcfg.py download_data --application=[app-id] --filename=[output dump file] --url=http://[app-id].appspot.com/_ah/remote_api

And then upload to another app:

appcfg.py upload_data --application=[app-id] --kind=[kind] --filename=[input dump file]

Or to work with structured data first generate the bulkloader file:

appcfg.py create_bulkloader_config --filename=bulkloader.yaml --url=http://[app-id].appspot.com/_ah/remote_api

and make sure to set the connectors to CSV

Then download the data:

appcfg.py download_data--application=[app-id] --config_file=bulkloader.yaml --filename=[csv file] --db_filename=progress.sql --kind=[which entity]

--db_filename is important for large uploads to avoid re-uploading the same data

Deploying code with rsync

2011-11-10T00:00:00+09:00

I use rsync to deploy code to my web server:

rsync --recursive --compress \  
--verbose --progress --stats \  
--times --perms --delete \  
--exclude "*~" --exclude "*.pyc" \  
. login@server:path

This command will transfer the current directory and all sub-directories and files to my server, but only transfer modified files to save time and bandwidth.
verbose, progress, and stats display useful output
times / perms ensure the file timestamps / permissions are the same on my server
delete will delete remote files when deleted locally
exclude temporary files

Convert DVD to AVI

2011-11-06T00:00:00+09:00

I tried a few programs to burn a DVD and found acidrip works best.

Remember:

set size to around 1000 for each hour
turn scale off
detect the correct crop size
preview results

Code syntax highlighting

2011-11-06T00:00:00+09:00

I use this when I need to syntax highlight code snippets:
http://tohtml.com

Writing in Korean

2011-11-03T00:00:00+09:00

Install ibus, ibus-m17n, ibus-hangul

Restart system - I couldn't use ibus in Chrome or choose the Romaji keyboard option until restarting

Ctrl-Space to activate language. Change keyboard to Romaji for English keyboard.

Scim also has a Korean package but I couldn't get it to activate reliably.

Set linux date

2011-08-20T00:00:00+09:00

Use this graphical tool to set the timezone:

sudo dpkg-reconfigure tzdata

Or set the time directly with:

sudo date MMDDhhmmYYYY

Left hand mouse

2011-07-31T00:00:00+09:00

One of the buttons on my laptop broke so to avoid that key I switched to using a left hand mouse. Here are the codes for each key:

xmodmap -pp  
There are 12 pointer buttons defined.

    Physical        Button
     Button          Code
        1              1
        2              2
        3              3
        4              4
        5              5
        6              6
        7              7
        8              8
        9              9
       10             10
       11             11
       12             12

And here I switch the left and right buttons:

xmodmap -e "pointer = 3 2 1 4 5 6 7 8 9 11 12 13"

chrome user agent

2011-05-20T00:00:00+09:00

Chrome does not currently have a setting or extension to set the user-agent.

You can however define it with a command line argument:

chromium-browser --user-agent="richard"

Working with large CSV files

2011-05-02T00:00:00+09:00

When working with large CSV files in Python I sometimes get this:

> Traceback (most recent call last):  
File "stats.py", line 12, in   
for i, row in enumerate(csv.reader(open(filename))):  
_csv.Error: field larger than field limit (131072)

This means the CSV file contains more content between delimiters that allowed by default. To raise this limit use:

import sys  
import csv  
csv.field_size_limit(sys.maxint)

Change keyboards

2011-03-17T00:00:00+09:00

I have been testing how fast I can get with the DVORAK keyboard. To change keyboard to DVORAK:

setxkbmap dvorak

And back to QWERTY again:

setxkbmap us

My vimrc

2011-02-12T00:00:00+09:00

set encoding=utf8  
  
set textwidth=0  
set tabstop=4 " Number of spaces to expand tab (messy)  
set softtabstop=4 " Number of spaces for insert for tab  
set shiftwidth=4 " Number of spaces for << or >>  
set expandtab  
  
set ruler  
set autoindent  
set nocompatible " use Vim defaults (much better!)  
set bs=2 " allow backspacing over everything in insert mode  
set ai " always set autoindenting on  
set vb " turn off bell  
set guifont=Monospace\ 12 " determined with 'set gfn'  
syntax on " switch syntax highlighting on  
  
" searching options  
set nowrapscan  
set ignorecase  
  
" use ALT key to treat wrapped text as new line when traversing  
map gj  
map gk  
imap gki  
imap gji  
  
" disable indentation for pasting with f2  
nnoremap :set invpaste paste?  
set pastetoggle=  
set showmode  
  
" for working with Windows line endings  
autocmd BufReadPost * nested  
\ if !exists('b:reload_dos') && !&binary && &ff=='unix' && (0 < search('\r$', 'nc')) |  
\ let b:reload_dos = 1 |  
\ e ++ff=dos |  
\ endif  

" Protect large files from sourcing and other overhead.  
if !exists("my_auto_commands_loaded")  
let my_auto_commands_loaded = 1  
" Large files are > 10M  
" Set options:  
" eventignore+=FileType (no syntax highlighting etc  
" assumes FileType always on)  
" noswapfile (save copy of file)  
" bufhidden=unload (save memory when other file is viewed)  
" undolevels=-1 (no undo possible)  
let g:LargeFile = 1024 * 1024 * 10  
augroup LargeFile  
autocmd BufReadPre * let f=expand("") | if getfsize(f) > g:LargeFile | set eventignore+=FileType | setlocal noswapfile bufhidden=unload undolevels=-1 | else | set eventignore-=FileType | endif  
augroup END  
endif

Format harddrive

2010-11-09T00:00:00+09:00

First find hard drive name:

ls -l /dev/disk/by-id/

And then format harddrive name:

sudo mkfs.ext3 /dev/sdb1 -L "Buffalo"

Amazon EC2

2010-11-09T00:00:00+09:00

Steps for setting up a free ec2 micro account:

Sign up for a free ec2 account
- You will need to create a new AWS account if you already have one
Log into theAWS Management Console
Create a micro Linux account - choose an EBS backedAMI(Official Ubuntu AMI's are uploaded by user 099720109477)
Make sure security group has SSH access
Download pem file (can store in same directory or ~/.ssh/identity - ~/.ssh/id_dsa for newer ssh versions)
chmod 400 [KEY].pem
From folder of pem file connect with: ssh -i [KEY].pem ec2-user@[ID].compute-1.amazonaws.com (user name may be root or ubuntu, depending on instance)
Send files with: scp -i [KEY].pem file [USER]@[ID].compute-1.amazonaws.com:
Mount filesystem: sshfs [USER]@[ID].compute-1.amazonaws.com: ~/mnt -o IdentityFile=[KEY].pem
Unmount: fusermount -u ~/mnt

Grep-fu

2010-08-31T00:00:00+09:00

How to display list of files with given extension with given text:

grep -lr --include="*.py" "firefox_" *

Installing Crunchbang

2010-07-11T00:00:00+09:00

Download Crunchbang development torrent
Format USB as FAT32
Download recent version of unetbootin
Create Debian Unstable HDMedia environment on USB
Copy Crunchbang ISO to USB at root
Boot from USB to install

Post install:
* Add account to sudoers with visudo * Add http://ftp.debian.org/debian/ and http://www.debian-multimedia.org/ to /etc/apt/sources.list * Add "setxkbmap us &" to ~/.config/openbox/autostart.sh to set US keyboard * install ttf-* packages for foreign fonts (Chinese: arphic-uming, Korean: unfonts-extra) * Install vim, codecs, etc * Customize .vimrc, .bashrc

iso install

2010-07-06T00:00:00+09:00

Find path to USB with: ls -l /dev/disk/by-id/usb
Format USB to FAT32: sudo umount /dev/sdx && sudo mkfs.vfat -I -n "custom name" /dev/sdX
Replug USB
Write ISO to usb with: sudo dd if=/path/to/iso of=/dev/sdX bs=4M;sync (or use UNetbootin)
Make USB first in BIOS boot order (hit F2 or Del at startup)

OCR for Linux

2010-07-01T00:00:00+09:00

I tried a few solutions to extract text out of a clear image containing alan0@hotmail.com:

Install with:

sudo apt-get install gocr tesseract ocrad

Gocr

gocr -i email.png   
aIan0hdmaiI.com

Tesseract

convert email.png email.tif # requires tif  
tesseract email.tif out  
cat out.txt  
Ina rrykeeg a

Ocrad

convert email.png email.ppm # requires p[bgp]m  
ocrad email.ppm   
alano_no_mall.

Upscaling the image slightly improved the results but still none were correct.
Disappointing ...

web2py on webfaction

2010-06-30T00:00:00+09:00

In Panel create a custom script and then paste this URL: http://wiki.webfaction.com/wiki/Web2py-LatestSource

Tor on Ubuntu

2010-06-28T00:00:00+09:00

I found Tor works with privoxy but not the recommended polipo on Ubuntu 9.10. To run tor first add the following to /etc/apt/sources.list:

deb http://deb.torproject.org/torproject.org [distribution] main

Now install Tor and the proxy server with:

sudo aptitude install tor privoxy
sudo vim /etc/tor/torrc and modifyMaxCircuitDirtiness
sudo vim /etc/privoxy/config
comment out: logfile logfile
find: listen-address 127.0.0.1:8118
add (on next line): forward-socks4a / localhost:9050 . (including the period)
sudo /etc/init.d/tor restart && sudo /etc/init.d/privoxy restart

Now set 127.0.0.1:8118 as proxy (eg in firefox) to access the Tor network.

GAE domains

2010-05-29T00:00:00+09:00

How to register a domain with godaddy and enable free hosting on Google:

Purchase domain from godaddy
- This will use the default godaddy nameservers (ns35.domaincontrol.com, ns36.domaincontrol.com, etc)
Set up Google Apps for domain
- Complete signup form and request cname value to verify domain
Login to godaddy domain manager (advanced)
- Add www forwarding to get naked domain to work (use masking)
- Delete all the default cname values, which point to various services
- Add google.com cname value and use default TTL
- Add cnameghs.google.com with alias www
Add www sub domain through GAE dashboard
To enable email use this service
Had to wait 10 minutes before service was active

SSH timeouts

2010-05-03T00:00:00+09:00

Add this to ~/.ssh/config to avoid ssh timeouts (create file if does not exist):

Host *  
ServerAliveInterval 240

Memcached on webfaction

2010-05-01T00:00:00+09:00

Choose memcached from custom install scripts and set extra info to maximum number of MB (eg 2)
This executes:

memcached start -d -l 127.0.0.1 -m 5 -P $PWD/memcached.pid -p XXXXX

Take note of port used. Then can be accessed with gluon.contrib.memcache

Call url with CRON

2010-05-01T00:00:00+09:00

Call URL periodically with CRON, discard HTML, and send all output (including errors) to file:

crontab -e   
0 * * * * wget URL -O /dev/null 2>> ~/cron.log

Komodo

2010-03-17T00:00:00+09:00

Run current file with Python:

%(python) "%F"

screen

2009-09-26T00:00:00+09:00

Use ssh to login to server and start screen:

ssh -t $1 'screen -dRR'

Common commands for manipulating screen:

screen -S main # create new screen session named main  
^a d # log out of screen, but leave running  
^a c # new window  
^a k # close window  
^a n # next window  
^a p # previous window

Configuration file to paste in ~/.screenrc:

hardstatus on  
hardstatus alwayslastline  
startup_message off  
termcapinfo xterm ti@:te@  
hardstatus string "%{= kG}%-w%{.rW}%n %t%{-}%+w %=%{..G} %H %{..Y} %m/%d %C%a "  
screen -t bash1 1  
screen -t bash2 2  
screen -t bash3 3  
screen -t bash4 4

Good reference

mercurial

2009-09-26T00:00:00+09:00

My .hgrc:

[ui]  
username = Richard Baron Penman
verbose = True
merge = internal:merge

Common commands:

hg clone URL # checkout  
hg pull # update  
hg commit -m "..." # save to local  
hg rollback # undo commit  
hg push # save to remote

Essential Firefox settings

2009-05-20T00:00:00+09:00

Install these extensions:

Set home page to Google.

Enable master password.

Regular expressions

2008-10-30T00:00:00+09:00

Some useful regular expressions

Email regex:

[a-zA-Z0-9._-]+@(([a-zA-Z0-9_-]{2,99}\.)+[a-zA-Z]{2,4})|((25[0-5]|2[0-4]\d|1\d\d|[1-9]\d|[1-9])\.
(25[0-5]|2[0-4]\d|1\d\d|[1-9]\d|[1-9])\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]\d|[1-9])\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]\d|[1-9]))

URL regex:

[a-zA-Z]{3,4}[sS]?://((([\w\d\-]+\.)+[ a-zA-Z]{2,4})|((25[0-5]|2[0-4]\d|1\d\d|[1-9]\d|[1-9])\.
(25[0-5]|2[0-4]\d|1\d\d|[1-9]\d|[1-9])\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]\d|[1-9])\.
(25[0-5]|2[0-4]\d|1\d\d|[1-9]\d|[1-9])))((\?|/)[\w/=+#_~&:;%\-\?\.]*)

IP reg exp:

(25[0-5]|2[0-4]\d|1\d\d|[1-9]\d|[1-9])\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]\d|[1-9])\.
(25[0-5]|2[0-4]\d|1\d\d|[1-9]\d|[1-9])\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]\d|[1-9])

Automate sudo password

2008-10-03T00:00:00+09:00

I am tired of typing my password to add new packages with Synaptic.

One solution is to edit the sudoers list with sudo visudo. To automate the sudo password for all programs add:

%yourgroup ALL = NOPASSWD: ALL

But this is excessive, so to automate the password for just a particular program provide its path like this:

%yourgroup ALL = NOPASSWD: /usr/sbin/synaptic

Now I can add new packages with Synaptic without being prompted for a password.

Fun with Festival

2008-06-01T00:00:00+09:00

Festival is a light weight text to speech program originally developed at The University of Edinburgh.

From the command line

echo "hello world" | festival --tts

From a file

echo "hello world" > file  
festival --tts file

file conversion

2008-05-28T00:00:00+09:00

Open source developers have focussed a lot of energy on the plumbing. Here are some examples for manipulating file types:

Convert between video formats

ffmpeg -i input.flv output.mpg

Convert between image formats

convert input.png output.jpeg

Join together audio files

sox in_1.wav in_2.wav out.wav

Remove sound from video

mencoder -ovc copy -nosound video.avi -o video_nosound.avi

Add sound to video

mencoder -ovc copy -audiofile soundtrack.mp3 -oac copy video_nosound.avi -o video_sound.avi

firefox proxy

2008-04-05T00:00:00+09:00

If your web browsing is restricted you can use a remote server to get round it. In the firefox network settings choose manual proxy and a port number. Then at the command line type:

ssh -XfN -D PORT -p 22 user@server

Now when you surf the packets will be redirected through this ssh tunnel.

Password protect grub

2008-03-30T00:00:00+09:00

By default with Ubuntu the user can use grub to login with root privileges WITHOUT a password, which I don't feel comfortable with. This is how I password protected grub to prevent changing the boot process:

grub  
grub> md5crypt  

Password: YOUR_PASSWORD  
Encrypted: abcdef  

grub> quit

Now copy the encrypted string into the start of the grub file:

sudo vim /boot/grub/menu.lst  
password --md5 abcdef  
lock

After saving change the permissions to prevent other uses viewing the file:

sudo chmod 600 /boot/grub/menu.lst

Recover password

2008-03-29T00:00:00+09:00

If you forget your login password it can be recovered through grub. At boot enter the grub menu and choose single. This will get you a root command prompt. Now run passwd to set a new password. Now anyone can do this if they get access to your machine so you should password protect grub. I will write how to do this later.

Email from the command line

2008-03-01T00:00:00+09:00

Send email from the command line:

echo "message" | mutt -s "subject" -x address@host.com

Passwordless ssh

2008-02-28T00:00:00+09:00

It becomes a hassle hopping between servers with ssh if you have to type the password each time. To enable passwordless ssh go:

ssh-keygen -t dsa (ENTER to skip password)  
scp ~/.ssh/id_dsa.pub user@other_server

And on the other server:

cat ~/id_dsa.pub >> ~/.ssh/authorized_keys  
rm ~/id_dsa.pub

Depending on the server may need to set permissions so no one can view the file:

chmod 600 ~/.ssh/authorized_keys
chmod 700 ~/.ssh
chmod 750 ~

Now you can hop between servers without a password.

Extracting tar balls

2008-02-16T00:00:00+09:00

I often forget the flags:

tar -xvvf file.tar  
tar -xvvzf file.tar.gz  
tar -zxvf file.tgz  
tar xjvf file.tar.bz2

Moving and svn repository

2008-02-15T00:00:00+09:00

To move an svn repository first dump it to a file with:

svnadmin dump /path/to/repository > svn.dump

Now to create the new repository go:

svnadmin create new_repository_name  
svnadmin load new_repository_name < svn.dump

Mediawiki upload settings

2008-01-30T00:00:00+09:00

By default the Mediawiki upload settings are too restrictive because they deny most filetypes. And changing the file extension is not enough because Mediawiki also checks the mime type. As a quick and dirty to allow what you want open LocalSettings.php and add:

$wgStrictFileExtensions = false;  
$wgCheckFileExtensions = false;  
$wgVerifyMimeType = false;

Mediawiki printing

2008-01-20T00:00:00+09:00

By default mediawiki includes header and footer information, which looks unprofessional when printing. To disable this open up _skins/common/commonPrint.css and add:

#footer {  
    display: none;  
}  
.printfooter {  
    display: none;  
}  
#bodyContent {  
    font-family: courier;  
}

Mediawiki sidebar

2008-01-15T00:00:00+09:00

To change the Mediawiki sidebar login as sysop and edit the page MediaWiki:Sidebar. Note that the default sysop login name is WikiSysop.

Mediawiki refresh

2008-01-12T00:00:00+09:00

To force Mediawiki to refresh anything it has cached just update LocalSettings.php:

touch /var/lib/mediawiki/LocalSettings.php

Mediawiki disable new accounts

2008-01-10T00:00:00+09:00

For privacy I recently made my wiki (Mediawiki 1.7) private and disabled new account creations. To do this I opened LocalSettings.php and added:

$wgGroupPermissions['*']['createaccount'] = false;  
$wgGroupPermissions['*']['read'] = false;

The problem is this prevents users logging in. I finally figured out I need to make the login page an exception for anonymous users with:

$wgWhitelistRead = array("Special:Userlogin");

First post!

2008-01-02T00:00:00+09:00

This blog will serve as a repository of solutions to technical problems I faced.
Currently I use the following technologies often:

Python
Linux (Ubuntu)
jQuery
web2py
Qt