Ultra High

Weird problems with ‘curl -d’ and lighttpd

I’m trying to test a Google Checkout response handler for a project I’m working on. Rather than putting through sandbox orders I’m just trying to post a message using Curl but for some reason POSTing anything longer than about a kilobyte just doesn’t show up at the server, running lighttpd 1.4. Enabling some debugging showed that curl is sending an HTTP 100 header.

User-Agent: curl/7.18.0 (i486-pc-linux-gnu) libcurl/7.18.0 OpenSSL/0.9.8g zlib/1.2.3.3 libidn/1.1
Host: asap
Accept: */*
Content-Length: 1116
Content-Type: application/x-www-form-urlencoded
Expect: 100-continue

There’s a bug in lighttpd 1.4 which means that this ‘Expect’ header is not handled properly. It won’t be fixed in v1.4 either.

The quick workaround for this bug is to call curl with the option “-H ‘Expect: ‘” to disable the header.

By the way, I also didn’t know how to post a file of data with curl - the answer is

Command-line face detection

This post explains how to:

1. Take a facial portrait and detect the position of the face
2. Cut a facial portrait down the center and remove half of the picture so that kids can fill it in themselves.
3. Print a massive amount of JPEGs at once by putting them in a PDF.

I’m in the Eastern Cape, South Africa now, working with orphans and vulnerable children. Alex and I are spending some of our time on art projects in remote rural areas, and one of the projects is an idea we stole from an orphanage in Cape Town: take a digital portrait of a child’s face, crop it down the center, print it and let them draw the other side of the face. Like this one Alex did:

Half-face portrait

The first time we did this, we went out to the rural area, took pictures of about 16 kids and then spent an hour or two processing the pictures and printing. The processing involved:

1. Importing the pictures to Picassa, straightening some of them and cropping others. (Yes, I know Picassa is not Free/Libre, but F-spot (in Ubuntu Hardy) is dog slow to display pictures and doesn’t have the straighten function).
2. Exporting to a directory, then opening each file in GIMP and cropping the right-or-left hand side of the face away.
3. Combine all the JPEG images into a PDF so they’re easy to print.

The second time, we did the project at a school, for 60+ pupils. The straightening/cropping in Picassa took about ten minutes (since most of the pictures didn’t need much work). The open-crop-save-close process in GIMP took about thirty seconds per picture and was both repetitive and highly mouse intensive so that we both got hand cramps after a while.

So, after watching Alex do the process for a second class at the school, I decided there must be a better way: automatic face detection. Lo-and-behold, five minutes of Googling got me to Torch3Vision, an image recognition toolkit with built-in face detection. It definitely works, but it takes quite a little setting up, so here’s a guide.

1. Download Torch3Vision and un-tar it: tar -zxf Torch3vision2.1.tgz
2. Build Torch3vision: cp Linux_i686.cfg.vision2.1 Linux_i686.cfg ./torch3make
3. Build the vison examples for face detection:cd vision2.1/examples/facedetect/
   ../../../torch3make *.cc

So now we have a working set of face-detection programmes. The command line interface isn’t too friendly, so they take a little playing around. For starters, the binaries on my Ubuntu system don’t read JPEG images (although the code seems to be there, the build system is non-standard and didn’t automatically pick up my jpeg libraries. So, I needed to convert my images to PPM format, which is one of those image formats that no-one uses but somehow is the lowest common denominator for image processing command line apps. I use the program ‘jpegtopnm’ from package ‘netpbm’.

Of the three facial detection programs available, I found ‘mlpcascadescan’ to be the most effective and quickest, although they all have similar interfaces so this will basically be the same for all of them. We need to pass the source image and the model file, and we tell it to write the face position and to save a drawing with the face detected:

This command takes about 20s to run on my creaky old laptop, and creates two files. One is a greyscale visualization of the face detected (the original image was colour):

Face detected, more or less

The other file ‘andy.pos’ contains the results of face detection. Line one is the number of detections, then each line has format x y w h, very easy to parse.

I played around with the step-factors in the x and y directions to shave a second or so off the face detection routine, the values I chose were 0.1 and 0.2 respectively (I don’t need any accuracy in the y direction really, since my use is to cut the face down the middle).

Then, since these are portrait photographs, I can speed up face detection by setting a minimum size for the face. I experimented and one sixth of the total image width gave good results - any larger and the face detection would fail with a crash. Adding this constraint provides better than 10X speed up, since the algorithm doesn’t waste time searching for small faces.

So now here’s the final face detection command

And finally, as promised, I’ll tell you how to blank-out one side of the face: of course, using Image Magick. Using the ‘chop’ or ‘crop’ commands didn’t work for this purpose, where I wanted the image to keep it’s dimensions but have one half just be white. So I decided to draw a white rectangle over half of the picture.  I apply the image manipulation to the original JPEG file, not the temporary PPM file that I used to detect the face position.

And here’s the final result:

The script I am using to tie this all together.

After processing all the portraits, I run a quick script to convert the jpegs to PDF and then join them into one master PDF file that I can easily print. The JPEG-PDF conversion uses Image Magick again (convert -rotate 90 file.jpg file.pdf). Joining together many PDFs into one document is easy with ‘pdfjoin’ from package ‘pdfjam’ (pdfjoin $tempfiles –outfile jpg2pdf.pdf). See the final jpg2pdf script.

But perhaps more enjoyable is to see the result after letting my limited creative talents loose:

A work of staggering complexity.

Africa

Having finished my PhD, Alex and I have moved to South Africa for a year to volunteer with a fantastic organization that supports community-based care of HIV/AIDS affected children. I’m intending to keep Ultrahigh around for posting geeky stuff, but we have also started a new blog about our travels and work in Africa. It’s called Live Free or Braai Hard and you can find it at http://africa.potatoriot.com.

Rsync backups with secure ssh key

After today’s Debian/Ubuntu OpenSSH fiasco I decided to tighten security in my backup system. I added a single-purpose SSH key which only allows access to rsync on the backup source hosts.

The main logic in the program is the same as my earlier attempt, although I added a check that the machine I am about to rsync from is the right one (this would fail due to ssh keys anyway). The ping check I previously did before starting a backup was insufficient if my laptop was on my home LAN, where NAT gave it the same IP address as my home gateway machine.

The other new part is the ssh authentication improvement. In order to use that, you need to create a single purpose ssh key on the backup server:

Then edit the public part of the key (~/.ssh/rsync.pub) to limit the commands that can be run with this key. You would add this to the front of the key, with no spaces (the whole key should be on one line too):

This comes before the ssh-dss with a space before ssh-dss.

You will need to install this key in ~/.ssh/authorized_keys on the backup source machines, and also on those machines install the ‘secure-rsync’ wrapper. The secure rsync wrapper is from Barry O’Donovan, who has a thorough page explaining a very similar (but better!) backup system. The path in the public ssh key should correspond to this file. All this wrapper does is check that the command is a safe rsync command and start rsync:

#!/bin/sh
case "$SSH_ORIGINAL_COMMAND" in
    *\&* | *\;* | *\|*)
        echo "Access denied"
        ;;
    rsync\ --server*)
        $SSH_ORIGINAL_COMMAND
        ;;
    hostname)
        $SSH_ORIGINAL_COMMAND
        ;;
    *)
        echo "Access denied"
        ;;
esac

Then you can use the ‘mybackup’ script to run the backups, with a simple config file that is explained in the script comments.

Building ROOT for Debian Lenny (testing)

Debian Lenny is currently the ‘testing’ version, but it is pretty stable and that’s what my workstation runs at work. However the annoying bug I had before with libroot-python-dev is still present (short version: the package wants (python < 2.5) but the version that ships with Lenny and Ubuntu Hardy is 2.5.)

Anyway, the easiest way around it is to build the packages from source. So add this to your sources.list

Then do these commands

• mkdir temp/root/
• cd temp/root
• apt-get source libroot-python-dev (this downloads the source)
• cd root-system-5.17.07

Now we come up against another problem. The maxdb packages aren’t available in Lenny (I’m not sure why they were removed) but the source packages depend on them, so you’ll need to work around that. If the packages weren’t broken, you could do apt-get build-dep libroot-plugin-python. Instead, you can get the list of packages you need from debian/control. Here’s the dependencies:

Now you have to edit the Debian control files to remove the maxdb dependencies.

Edit debian/rules, and

• Delete the ‘–enable-maxdb‘ line. (You can’t comment it out in place, but you could move it down a few lines and comment it out there)
• Comment out lines 35-45, just leaving ‘SAPDB = –disable-sapdb‘

Edit debian/control to remove the dependency on libsqlod-dev in the first stanza and delete the entire root-plugin-maxdb stanza.

If you like, you can edit debian/changelog and give it a new version number. The version extension must contain a digit, e.g.

Avoid this gotcha: I’d symlinked /usr/bin/gfortran to /usr/bin/gfortran4.3 for another project. ROOT’s config/Makefile.linux checks that gcc and gfortran are the same version, so I had to undo that symlink.

Build the packages:

My build failed with this error but all of the packages seem to be there (including root-plugin-asimage, so I don’t know what it’s complaining about):

dpkg-genchanges -b >../root-system_5.17.07-1-ultrahigh1_i386.changes
dpkg-genchanges: binary-only upload - not including any source code
dpkg-genchanges: failure: cannot fstat file ../root-plugin-asimage_5.17.07-1-ultrahigh_i386.deb: No such file or directory
dpkg-buildpackage: failure: dpkg-genchanges gave error exit status 2
And if you are too lazy to build the packages and you trust me (why would you!), you can download my packages which should install OK.

Voice Over IP on Hardy

In a separate post I talk about how Pulseaudio has broken some proprietary applications on Ubuntu Hardy. What’s worse is that many of the open source voice over IP clients (SIP, in particular) are broken on Hardy, which is a shame since Pulseaudio has been in the works for a while now and most major distributions are shipping it by default these days.

Here’s a list of the SIP clients that I have tried.

Ekiga: Just doesn’t do any sound in the call when using ALSA-via-pulseaudio (a real shame since this is the most GNOME-flavoured app and I’d really expect it to work on Ubuntu).

Gizmo: Ringing/Dialling noises work if you set ‘paplay %s’ in the “Use command to play sounds” option. Audio in call is missing when using ALSA-via-pulseaudio. Won’t start with OSS using padsp.

OpenWengo: This is barely maintained at the moment, it’s sad. And it’s tied into the Wengo side of things which seems to have fallen by the wayside (incoming SIP via voip.wengo.fr doesn’t work). I’d like to see the client return, open to multiple VOIP providers since it’s pretty solid software. But the microphone doesn’t seem to work in calls with pulseaudio at the moment!

Twinkle: Confusing interface, but it does work. Set it to use OSS for audio input and output, set the ringing to use ALSA default device (which is pulseaudio) and start the program with ‘padsp twinkle’. Not ideal since you lose a lot of the benefits of Pulseaudio, but at least it works. Wish we had v1.2 in Hardy which figures out which network to use automatically - at the moment I need to restart the program if I move from wired to wireless.

Empathy: Cool idea but SIP support isn’t really ready yet. The SIP configuration dialog is insufficient, it doesn’t offer all the fields you need to login to something like voxalot.com or FWD. Actually, that’s not Empathy’s fault but the “Mission Control” configuration app. Also, I’d like to get an audio ringing sound for incoming calls, and I can’t add SIP contacts via the main interface (it silently fails). I think Empathy is basically alpha software so I’m looking forward to these issues getting fixed - it has the promise of being the ultimate IM/SIP/Video client for GNOME.

Linphone: Works pretty well! Use the default ALSA device (which is pulse) and you can take/recieve calls and the ringer works. Great! I have a few issues with the interface: I’d love to see a tray icon and better visual ringing notification, but this is a solid app that actually works. Good stuff.

Pulseaudio problems on Hardy

So Ubuntu 8.04 (Hardy) came out, and it rocks a lot. By default it enables the pulseaudio sound server, which also rocks but has caused a few of the ‘usual suspect’ proprietary apps to fail even more than usual.

Skype

First: Skype. Skype is the bane of my existence, and judging from the way their developers rail against change on the forums, Linux users like me are the bane of theirs. No sooner had they fixed their three-year avoidance of ALSA than we all move to a new system and theirs breaks. Well, actuallyour new system was supposed to work fine with ALSA apps, except they seem to have used ALSA in a weird way, which means that when you redirect your ALSA input and output to pulseaudio, Skype just ignores it and tries to access your hardware directly. Which breaks, because pulseaudio is using your hardware.

One fix proffered on the Ubuntu forums is to make pusleaudio use ALSA’s dmix, that way your pulse apps will have this chain app->pulse->dmix->hardware,whilst broken apps like Skype can be set to use app->dmix->hardware. And this fix does make Skype work OK… but it messes up everything else like audio players because pulse-over-dmix is crappy and uses all your CPU.

So here’s my fix for broken apps, and no, it’s not ideal. Guess what? It never will be while Skype is broken!

Run the Hardy stock setup with ALSA-over-pulse, but manually run pasuspender when you need to take/make a call. I have this script which is called from a button on my panel which I press.

# temporarily pause pulseaudio so I can use Skype or whatever
pasuspender — /usr/bin/zenity –info –text “Click OK to resume Pulseaudio”

Works OK, although you won’t hear Skype ringing.

Flash

Next up, Flash. Flash on Linux just sucks, absolutely. In Hardy, as of this moment, if you install the library that allows Flash to do audio output over pulseaudio, the Flash will crash every time you do something unusual like, say, open a frickin’ hilarious YouTube video, your browser will crash. Awesome huh! In fact, you can alleviate this by installing nspluginwrapper, in which case you will just have a gray square instead of your YouTube video. Also awesome.

My workaround for this is… just hit refresh a billion times until Flash stops sucking and plays the stupid video. It’s some sort of race condition between Flash and libflashsupport, at least that’s what the Pulseaudio developers claim. Lame!

Every Voice-over-IP App, ever

WTF guys! We all knew Pulse was coming for over a year… so why is it so hard to find a SIP client that works in Hardy? Check out my other post for listings of what’s broken about the different clients. Short answer: Linphone works well, Twinkle can be coaxed into working.

One Keyboard, Two Computers: x2x Over SSH

I just set up my laptop at work as a docked second screen next to my main workstation (yeah, I know, procrastination central). I wanted to use the network to control the second machine (laptop) from the nice keyboard of my workstation. I knew I could do it because I use to use x2x when we had the remote HiRes station here (4 old junker PCs with one or two screens each, two keyboard/mice setups).

For some reason I was trying to use synergy, which is completely insecure. The only advice offered on the synergy site about securing it was setting up port forwarding via ssh first, then connecting over those ports. Yuck.

Anyway, since ssh automatically forwards X11 connections, it’s dead easy to use x2x for this task in a secure way (with all the usual nice stuff like ssh key authentication too).

On my wokstation, with my laptop to the right, I run the command
ssh -X laptop "x2x -east -to :0"

That’s it. Run the mouse over to the right hand side of teh workstation screen, it travels over to the laptop screen. Even X middle-click cut-n-paste works. Sweet.

Oh, well, one more cool thing, if you have sshfs installed (and fuse, and have added yourself to the fuse group, and re-logged in, and loaded the fuse module…), you can do this to automatically cross-mount the laptop home directory on your workstation:

#!/bin/sh
# Tunnel a x2x connection to a second machine, presumed to be at
# the right of this machine's screen.

host="$1"
mount_dir="$HOME/$host"
direction="east"  # west means the OTHER screen is to the left

echo "Creating $mount_dir"
mkdir -p $mount_dir
echo "Mounting remote machine at $mount_dir"
sshfs cactus: $mount_dir

echo "Connecting to remote machine for x2x over ssh"
ssh -X $host "x2x -$direction -to :0"

# after ctrl-C killing above session, clean up sshfs stuff
# sshfs automatically unmounts when you ctrl-C out of above command
echo "Removing mount point"
rmdir $mount_dir

Thank you to the developers of FUSE, sshfs and x2x… You rock!

The Pastels

I just discovered the Pastels, in particular the best-of Truckload of Trouble. Fantastic!

I’m going to look at the whole C86 ’scene’ since they are supposed to be so representive of that sound, in particular the first Primal Scream album Sonic Flower Groove seems interesting. It is supposed to be very Byrds-esque and apparently was a major influence on early Stone Roses.

Until today I only really knew Primal Scream from after Screamadelica onwards. I can’t decide if this album cover is more or less embarrassing for them than the confederate flag on Give Out.

Making a shared data folder in Linux

I wanted to set the file permissions properly on our shared music folder (accessed by Samba/scp/directly on the computer by different users). Basically, I want anyone in the ‘data’ group able to read and write everything.

First, I created the data group.
sudo groupadd data

#!/bin/bash

DIR=/data/music

echo "Changing Group ownership to 'data'"
chgrp -R data $DIR
echo "Changing permissions of files to ug=rw,a=r"
chmod -R 664 $DIR
echo "Changing permissions of directories to a+rx, g+rwx, g+s"
find $DIR -type d -exec chmod a+rx,g+rwx,g+s '{}' \;

[data]
path = /data
available = yes
browsable = yes
public = yes
writable = yes
create mask = 664
directory mask = 775

drwxrwsr-x 2 user1 data 4096 2008-02-15 11:29 temp
-rw-rw-r-- 1 user1 data    0 2008-02-15 11:29 test