Planet Belfast Linux

Fizz buzz

2010-03-08T22:32:37+00:00

Inspired by a conversation about interview coding tasks from a list I'm on, I present the following - I considered it too long to email there. It took me longer than I expected to write; my x86 assembly is quite rusty. I'm not claiming it's pretty, but it fits in a single sector and most of the overhead is actually ELF structures.

; nasm -f elf fizzbuzz.asm
; ld -melf_i386 -s -o fizzbuzz fizzbuzz.o
; ./fizzbuzz

section .data

fizz	db	" fizz"
fizzlen	equ	$ - fizz
buzz	db	" buzz"
buzzlen	equ	$ - buzz
num	db	"   "
numend	equ	$ - 1
numlen	equ	$ - num
nl	db	0xa
nllen	equ	$ - nl

curnum	db 1

section .text

	global _start

_start:
	mov ax, [curnum]
	call printnum

	mov ax, [curnum]
	mov cx, 3
	xor dx, dx
	div cx
	cmp dx, 0
	jnz notfizz

	mov edx, fizzlen
	mov ecx, fizz
	call printstr

notfizz:
	mov ax, [curnum]
	mov cx, 5
	xor dx, dx
	div cx
	cmp dx, 0
	jnz notbuzz

	mov edx, buzzlen
	mov ecx, buzz
	call printstr

notbuzz:
	mov edx, nllen
	mov ecx, nl
	call printstr

	inc BYTE [curnum]
	cmp BYTE [curnum], 100
	jle _start

	xor ebx, ebx
	mov eax, 1
	int 0x80

printnum:
	mov edi, numend
	mov cx, 10
p1:
	xor edx, edx
	div cx
	add dx, '0'
	mov [edi], dl
	dec edi
	cmp ax, 0
	jne p1

	mov ecx, num
	mov edx, numlen
printstr:
	mov ebx, 1
	mov eax, 4
	int 0x80
	ret

Meta: Rant about rants about PGP keysigning problems

2010-03-03T22:15:23+00:00

(This has ended up longer than I intended, largely because I felt I should then get into why. I'm aware I haven't got into all the nuances, so I hope readers familiar with the area will appreciate this is the compact version.)

Thorsten had a rant last week about PGP keysigning problems. He apologises for his tone, but that's not the issue I take with his rant.

It starts "Keysigning is useless". And yet his complaints seem to be:

Dealing with the private half of your GPG key securely involves some faff (in this case booting with a live CD and having to set things up ready to keysign).
He doesn't get on with caff.
People reject email from machines with invalid HELOs and perform other anti spam measures on ISP access ranges (I'm not clear if it's just greylisting or outright rejects as that's not made clear).
PGP/MIME is a protocol violation (yes, but it's much better than inline OpenPGP. Unless you have to deal with RT, which mangles it. *sigh*)

None of these seem to actually be about keysigning being useless. The process of doing it, maybe, though he misses the main valid rant about this I'd have, which is that most mass keysignings don't actually allow you to accurately verify the identity of other participants unless you already know them reasonably well. (The LCA2010 keysigning and DebConf5 in Helsinki spring to mind as 2 good examples of bad keysignings I've attended, but speaking to others suggests it's far from an isolated thing.)

Torsten does say that he'll continue to do keysigning on a per-person basis, so it doesn't sound like he's completely given up. I'm posting this largely so other Debian related people don't get the idea that it's not important to think about keysigning.

Why should we care?

Firstly, let's clarify what I mean when I sign someone else's key. If I sign your key then I think that I believe you hold the private part of a key that has your name and an email address I believe I can use to contact you on it. It means I have seen government issued ID that matches that name. It also means that I have interacted with you (and watched others interact with you) under that persona. In short I am happy that the key is a reasonable digital representation of your identity - something signed by it either comes from you or has involved the key being compromised or you coerced into using it against your will.

Why is this useful?

It gets useful thanks to the web of trust; ie the idea that there are a bunch of people I trust partially to sign other keys, and if enough of them have signed a key then I can have a reasonable expectation that the key belongs to the person I want to talk to. Which means I might be prepared to send private data to them. Or Debian might be prepared to accept an upload from them. Which, when you're dealing with a community that spans the planet and where most of the contributors haven't met each other, is pretty freakin' useful - I, as part of Debian's keyring team, don't need to personally be able to identify every Debian developer. All I need to do is be able to trust other DDs to be able to do so. (Though maybe I'm missing out on something here - perhaps Debian should be paying for Gunnar and me to travel the world verifying fingerprints. \o/)

(I still do mass keysignings btw. I'm picky about which keys I actually sign - this is in no way intended as a slight against those I don't, but a mass keysigning at least lets me know that the people involved are happy to exchange fingerprints. Though, FWIW, I normally have ID on me and frequently have fingerprint slips, so if you know me and want me to sign your key/want to sign mine then by all means ask me when you see me!)

Valentine's CISS

2010-02-25T12:28:49+00:00

Printers. Love them or hate them, you still have to feed them Ink (or Toner) cartridges.
These are expensive little beasts to keep running - it has been commented that printer ink is expensive, and to give to an idea just how expensive it is:

Printer Ink is 7 times more expensive than Dom Perignon.
Printer Ink is more expensive than the most expensive perfumes.
Printer Ink is more expensive than human blood.

Or if you want to see the scale, here is an often posted image (attribution unknown): Update, found the original source at Gizmodo from Nov 2006.

Like many people I had found the relative comfort of 3rd-party or remanufactured Ink cartriges which brink the cost per cart down from around £3 (instead of £9) for my particular model.

However, as I was installing the last of my replacement carts, before having to order more, imagine my horror when the magenta cart simply failed to work. Nothing - printer refused to accept it, thankfully my old cart had a dribble of ink left and was able to convince the printer to keep going while I got my order in for more.

Next step, the online store where I order my carts, SVP, typed in my printer model in the search box as they recommend and the first hit wasn't my usual multipack of 3rd party R265 carts - no, it was a CISS (Continuous Ink Supply System). Interesting.

Here is the page: http://svp.co.uk/product/ciss_for_epson_r265_r360_rx560_printers_mte058

Intrigued, I read the install manual they have on the page and thought it looked easy enough to try. And so I bought one - couldn't hurt - it cost the same as a complete set of carts and would last 10 times longer on the first fill.

The device arrived a few days later, I sat on it a few days more, then got stuck in. I took some photos of the completed install and I have to say I am very impressed with it.

I originally had the Inkwells on top of the printer, but I found it was putting out way too much ink - blobs of the stuff - and I figured gravity was playing a part. Placing it down beside the printer saw the ink flow backwards, so I taped a few empty DVD cases together to get the right approximate height beside the printer and placed it there.

I have printed the equivalent of 20 full A4 colour pages at photo quality - quality is excellent and although the computer thinks the carts are now half full (or half empty), the evidence above shows just how much money I am going to save even in the short term.

If you are feeling the cost of Ink is too high (who doesn't?) and if you can find a well reviewed CISS system for your printer, I would encourage you to give it a go.

What’s changed?

2010-02-22T11:48:09+00:00

Matt Johnston commented recently that the recent surge in activity in the community side of the local tech & business scene could be “the ‘real’ end of the ‘Troubles’?”. It’s definitely a positive thing, I’m delighted the next generation of technologists in Northern Ireland has a growing & diverse community around them. Something that was sorely lacking in my formative years, where it seemed that the only exposure to technology was from inside the technology firm you worked in. [and I'm a committed technologist, not a 9-5 salary man]. So what’s changed?

Many of us are of a similar age, all in full-time IT roles from the mid-90’s onwards, some for much longer. Is it the relatively recent additions that have invigorated us? People like the hyper-active Andy McMillen, or what’s caused “the old guard” like Matt to push on with xcake & startvi, or Colm & Norbert to persevere with MobileMondayBelfast, or Darryl & the first Open Coffee Belfast?

Surely none of us would admit to letting Northern Ireland’s previous problems get in the way of the way we lead our lives?

So what changed? How do we make sure we don’t loose momentum?

How much would your life changed if you had the community & adventure surrounding you 10-15 years ago when you first discovered your passion for technology could pay the bills? Would you have endured the 10 years in big, faceless corporations? [how did we get brain washed into thinking that the best IT career involved one of 3 or 4 companies in NI?]

Would I still be doing what I’m doing now? Probably, but probably not for who I’m doing it for. And I hope I would have had a much interesting & independent path here.

Sinful admission

2010-01-27T09:43:55+00:00

Knowing full well that it will cause many of my readers to tut and roll their eyes at me I derive slight pleasure from confessing that I have now eaten at McDonalds on 6 continents. To make it worse I only did so in Asia and Australia so I could say I had.

It does lead me into a slightly more valid ramble. I'm not particularly bothered about eating on my own. I like food (even if I'm picky about what I like), but I prefer it with company. This means I'm not particularly great about being organised to cook complicated things when I'm on my own; largely the whole thing just turns into dealing with hunger. It also mean I'm bad at finding nice places to eat out while travelling. Which, when you're travelling for a month to some interesting places, is a bit of a bugger. I've got a bit better at actually going to nice places, and trying not to bolt my food into me so I can get out ASAP but instead enjoy it. A book helps. Also places that aren't quite as busy (which if you have no agenda is nicely achieved by letting your body clock desync from the world around you). Unfortunately I didn't really do so until after Hong Kong. Guess I'll just have to go back at some point...

(More on my travels at some point, maybe. I've started the tortuous journey home now.)

Boot time sound problems with QuickCam Pro 9000 webcam

2010-01-17T22:21:30+00:00

I've had a Logitech QuickCam Pro 9000 for some time now, and it works well, but I did have one bizarre problem with it when I used it with my main desktop machine (running Debian (Sid)). Namely, that if I had the camera plugged in (usb) at boot time the sound on the computer did not work, if you plugged it in after boot, everything was fine.

This was pretty irritating because if I forgot I would often have dozens of windows open and ready for work before I realised.

Normally this happens because the cards are loaded in the wrong sequence by udev. But, if you listed the sound cards with

cat /proc/asound/cards

the main card wasn't just in the wrong order, it simply wasn't there. I tried comparing modules loaded with and without the camera (at boot) and manually loading the differences, but this did not help. I tried forcing the index to be zero on the correct card, but this also did not help.

In the end, I made the following edit to /etc/modprobe.d/alsa-base.conf

# Keep USB (webcam from being loaded as first card)
options snd_usb_audio index=-2

and this did the trick. So if like me you were searching for the answer to this, I hope it helps.

Prettifying Debian boot output

2010-01-07T04:40:21+00:00

Being a bit of a freak I actually tend to read my boot output. Well, when I'm not using that time to go grab a cup of coffee first thing on a Monday morning. However I don't always find Debian's default output as clear as I'd like (especially when I haven't had that coffee yet, or when I'm half watching a remote machine boot over its serial console). ISTR that RedHat had much nicer boot output (I'm talking about text based here, not the fancy Plymouth stuff) with a row of OK/FAILED etc down the right hand side that made it nice and obvious what was happening. Of course we can do that with Debian, at least for packages that use the lsb-base logging functions. Drop the following into /etc/lsb-base-logging.sh and enjoy a slightly prettier boot. I'm slowly filing wishlist bugs with patches for those bits and pieces I use that don't use lsb-base for their init script logging.

# Colour our init scripts output

# int log_end_message (int exitstatus)
log_end_msg () {
    # If no arguments were passed, return
    if [ -z "${1:-}" ]; then
        return 1
    fi

    retval=$1

    log_end_msg_pre "$@"

    # Only do the fancy stuff if we have an appropriate terminal
    # and if /usr is already mounted
    if log_use_fancy_output; then
        RED=`$TPUT setaf 1`
        GREEN=`$TPUT setaf 2`
        YELLOW=`$TPUT setaf 3`
        NORMAL=`$TPUT sgr0`
        $TPUT hpa $((`$TPUT cols` - 12))
    else
        RED=''
        GREEN=''
        YELLOW=''
        NORMAL=''
    fi

    if [ $1 -eq 0 ]; then
        /bin/echo -e " [   ${GREEN}OK${NORMAL}   ]"
    elif [ $1 -eq 255 ]; then
        /bin/echo -e " [${YELLOW}WARNING!${NORMAL}]"
    else
        /bin/echo -e " [ ${RED}FAILED${NORMAL} ]"
    fi
    log_end_msg_post "$@"
    return $retval
}

log_action_end_msg () {
    log_action_end_msg_pre "$@"
    if [ -z "${2:-}" ]; then
        end=""
    else
        end=" ($2)"
    fi

    /bin/echo -n "${end}"

    # Only do the fancy stuff if we have an appropriate terminal
    # and if /usr is already mounted
    if log_use_fancy_output; then
        RED=`$TPUT setaf 1`
        BLUE=`$TPUT setaf 4`
        NORMAL=`$TPUT sgr0`
        $TPUT hpa $((`$TPUT cols` - 12))
    else
        RED=''
        BLUE=''
        NORMAL=''
    fi


    if [ $1 -eq 0 ]; then
        /bin/echo -e " [  ${BLUE}DONE${NORMAL}  ]"
    else
        /bin/echo -e " [ ${RED}FAILED${NORMAL} ]"
    fi
    log_action_end_msg_post "$@"
}

(This is mostly for my own future reference, because I keep trying to search for details on it and not finding exactly what I want. I forget where I even found the basis for this, though there are lots of similar snippets out there. I've not tried it with a concurrency based boot so it may well look horrible under that.)

Parsing the Doom Wad File

2010-01-07T01:12:31+00:00

So it was a fine Christmas, I was at home with family armed only with 2 seasons of xFiles and my laptop added to the factI didn’t upload my ssh key to my laptop so hacking on Crules was out the window. So it got me thinking on the main goals for my Crules Scripting Language, one of which ( the main one ) is that it should be embeddable into existing C/C++ applications like Lua or Python or even C# (mono) if your using its recently added reflection api!

So why not make a game engine using my language with crules as a scripting language so I can see what the API _SHOULD_ look like for the client. So i choose doom since i am terrible with graphics, i can use the doom.wad and concentrate on programming; So in the past i have written two separate 2D game engines one was a Java 2D Monkey Island clone i done for university and it was terrible since Java is a terrible platform in my opinion, then my 2nd was a port of it over to C++/SDL and the code was terrible so don’t bother asking me about it.

Anyway’s so the long and short i am writing doom engine from scratch, my original idea was to work with an existing game engine like Crystal Space or Doomsday Engine though, Crystal Space i was put off since it already uses Python and i don’t really feel like trawling though lots of code and forking it essentially would be really more work than its worth. Doomsday Engine is the cream of the crop of Doom Ports at the moment, all others are practically the same in essence using the same ancient code and hacks from throughout the 90’s. And really the technology has changed somewhat since then. Though Doomsday Engine is pretty amazing and works well, it seems to need a massive cleanup and i don’t mean to be nasty on the developers or anything they have done a fantastic job! And they have really made the code much cleaner than any other port about! Plus the lead developer has been really nice to me . Anyway’s Games have always been a great passion to me and really doom can’t be that hard to re-implement from scratch and really it isn’t i already have a basic game engine working using OpenGL and SDL{ _ttf, _mixer }. With audio, and fps counter and input bla bla. But i am just concentrating on differentiating the wad sprite formats etc while revising for exams and working on crules and many other bits and pieces. So in the end i’ll just stick in an extract of the log of my game engine when it starts parsing the doom.wad and explain a little on how to parse the wad!

debug: main.c:28 -> Trying to start crldoom wad: data/doom.wad!
debug: doom.c:147 -> parsing wad file <data/doom.wad>!
debug: doom.c:80 -> is a wad file :: <IWAD>!
log: doom.c:155 -> wad file data/doom.wad has length 12408292!
debug: doom.c:27 -> debug seeking…
debug: doom.c:30 -> reading….
debug: doom.c:34 -> parsing…
log: doom.c:49 -> header:: ident -> 1145132873, n_lumps -> 2306, infotableofs -> 12371396!
log: doom.c:62 -> directory:: filepos -> 12, size -> 10752, name -> 1497451600, ident -> PLAYPAL!
log: doom.c:62 -> directory:: filepos -> 10764, size -> 8704, name -> 1330401091, ident -> COLORMAP!
log: doom.c:62 -> directory:: filepos -> 19468, size -> 4000, name -> 1329876549, ident -> ENDOOM!
log: doom.c:62 -> directory:: filepos -> 23468, size -> 6854, name -> 1330464068, ident -> DEMO1!
log: doom.c:62 -> directory:: filepos -> 30324, size -> 9402, name -> 1330464068, ident -> DEMO2!
log: doom.c:62 -> directory:: filepos -> 39728, size -> 15466, name -> 1330464068, ident -> DEMO3!
log: doom.c:62 -> directory:: filepos -> 55196, size -> 3286, name -> 1330464068, ident -> DEMO4!
log: doom.c:62 -> directory:: filepos -> 58484, size -> 0, name -> 827142469, ident -> E1M1!
log: doom.c:62 -> directory:: filepos -> 58484, size -> 1430, name -> 1313425492, ident -> THINGS!
log: doom.c:62 -> directory:: filepos -> 59916, size -> 6804, name -> 1162758476, ident -> LINEDEFS!
log: doom.c:62 -> directory:: filepos -> 66720, size -> 19980, name -> 1162103123, ident -> SIDEDEFS!
log: doom.c:62 -> directory:: filepos -> 86700, size -> 1880, name -> 1414677846, ident -> VERTEXES!
log: doom.c:62 -> directory:: filepos -> 88580, size -> 8964, name -> 1397179731, ident -> SEGS!
log: doom.c:62 -> directory:: filepos -> 97544, size -> 956, name -> 1128616787, ident -> SSECTORS!
log: doom.c:62 -> directory:: filepos -> 98500, size -> 6664, name -> 1162104654, ident -> NODES!
log: doom.c:62 -> directory:: filepos -> 105164, size -> 2288, name -> 1413694803, ident -> SECTORS!
log: doom.c:62 -> directory:: filepos -> 107452, size -> 968, name -> 1162495314, ident -> REJECT!
log: doom.c:62 -> directory:: filepos -> 108420, size -> 6948, name -> 1129270338, ident -> BLOCKMAP!

So this is just some of the output of the code i had written it parses out whats called the ‘directories’ in the file and then i also have functions to parse out whats called the ‘lumps’ from these directories the lumps are the actual data. I’ll explain more detail when i talk about how to parse this. And how to convert the music lumps to proper midi so you can play it in SDL_mixer or even in Totem if in in gnome.

So what do we need, if you have yourself a Doom.wad file lying about i am sure if you go through your old boxes you’ll find yourself a copy of doom, or if you cant find it you can find it on torrent sites but you didn’t read that here . Thing is I have bought the game like 3/4 times I am sure hehe. Lets get started.

In the Doom wad there is whats called the ‘HEADER’ which is of length 12 bytes which contains 3 * 4-byte integers which contain the data we care about, so lets do that lets make some code (i just wrote this code in like 10 min to illustrate the idea its ok does the job!):

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#define WAD_HEADER_LENGTH 12
#define WAD_DIRECTORY_LENGTH 16
unsigned long parse_int( unsigned char *p )
{
return ( (unsigned long) p[0]
| (unsigned long) p[1] << 8
| (unsigned long) p[2] << 16
| (unsigned long) p[3] << 24
) ;
}
int main( int argc, char *argv[] )
{
const char* wad_file= "/home/redbrain/workspace/doom-dev/crldoom/data/doom.wad";
FILE* wad_fd;
if( !(wad_fd= fopen(wad_file, "rb")) )
{
fprintf(stderr, "error opening <%s>!\n", wad_file);
return EXIT_FAILURE;
}
unsigned char header_buffer[ WAD_HEADER_LENGTH ];
fread( header_buffer, WAD_HEADER_LENGTH, 1, wad_fd );
char* wad_ident= (char*) header_buffer;
wad_ident[ 4 ]= ‘\0‘;
if( strncmp( wad_ident, "IWAD", 4 ) )
{
fprintf(stderr, "invalid wad header type <%s>!\n", wad_ident );
return EXIT_FAILURE;
}
else
{
printf("doom wad is a <%s>!\n", wad_ident );
}
unsigned long wad_length= 0;
fseek( wad_fd, 0, SEEK_END );
wad_length= ftell( wad_fd );
unsigned long directory_offset= parse_int( header_buffer+8 );
unsigned long number_lumps= parse_int( header_buffer+4 );
printf("wad directory offset <%lu> with <%lu> lumps!\n",
directory_offset, number_lumps );
unsigned long t_ofs= directory_offset; unsigned long lump_count= 0;
while( t_ofs <= ( wad_length -- WAD_DIRECTORY_LENGTH ) )
{
fseek( wad_fd, t_ofs, SEEK_SET );
unsigned char *directory_buffer= (unsigned char*)
malloc( sizeof(char) * WAD_DIRECTORY_LENGTH );
fread( directory_buffer, WAD_DIRECTORY_LENGTH, 1, wad_fd );
unsigned long filepos= parse_int( directory_buffer );
unsigned long size= parse_int( directory_buffer+4 );
char *directory_ident= strdup( (char*) directory_buffer+8 );
directory_ident[ strlen(directory_ident) ]= ‘\0‘;
printf("directory name <%s> at offset <%lu> with size <%lu>!\n",
directory_ident, filepos, size );
free( directory_ident ); free( directory_buffer );
t_ofs += WAD_DIRECTORY_LENGTH; lump_count++;
}
printf("directorys parsed <%lu> total lumps <%lu>!\n",
lump_count, number_lumps );
fclose( wad_fd );
return 0;
}

So lets compile and test this code <remember to change the path to your doom wad>:

gcc doom_wad_test.c

./a.out

So right what does any of this code mean i just chucked it up there, i wrote it up very quickly from scratch. So as i stated earlier there is a wad header which is a 12 bytes in length which i refer to as ‘WAD_HEADER_LENGTH’ which contains 3 * 4-byte integers. So from the code we have fopen’d the wad_file and i use fread to read in the 12 bytes in a 12 byte ‘unsigned char[]‘ buffer. So to read out the integers in a neutral endian way i found a nice parse_int function in some of the wad documentation i found and in old doom source code which was GPL so it was like…. yoink… but its pretty easy to understand if you know your representations well enough but i wont go into detail.

The first integer is the IWAD or PWAD string, IWAD is what you will find mostly its the deployment version of the wad, the PWAD is a patch wad and very few of them are still around and i think it has similar structure. Anyways, you parse out the first integer like this

char* wad_ident= (char*) header_buffer;
wad_ident[ 4 ]= ‘\0‘;

Remember to add null string ‘\0′, so we don’t get buffer overflow! This string is always a length of 4 so no need to mess about! And then the next integer is the number of lumps in the wad the lumps are the actual binary data like music and graphics. Third int is the ‘directory offset’ which contains pointers to where the lumps are their idents and the size of the lumps which have lengths of 16 ‘WAD_DIRECTORY_LENGTH’ until the end of the wad file! so note i find the length of the file and do:

while( t_ofs <= ( wad_length -- WAD_DIRECTORY_LENGTH ) )
{
…
}

So how do we parse out these directories? We move to the directory offset and then read the 16 bytes into a buffer, which is 4 integers this time. But still contains only 3 things means the ident strings can be longer or can be with some padding between each directory lump.

unsigned long filepos= parse_int( directory_buffer );
unsigned long size= parse_int( directory_buffer+4 );
char *directory_ident= strdup( (char*) directory_buffer+8 );
directory_ident[ strlen(directory_ident) ]= ‘\0‘;

Now we know the file offset of the lump the size of the lump and the ident of the lump and we make sure to add the null terminator again to avoid buffer overflow. And then we can parse out the lumps simply just fseek and fread Done its so simple and seems like there would be more to it, yet when you think of it is very much common sense there really isn’t that many ways to pack data into a file like this! If your interested i’ll be talking about it soon… on how to convert the D_E1M* lumps which are music lumps to compatible midi tracks. And finally on how BSP works for the levels and maybe how to draw them in C/SDL/OpenGL… if you ask i prefer C over C++, yeah when it comes to building this game engine there have been times i though woo having an object would be really nice here but then i realise well no because that’s what the scripting engine is going to do… Mainly due to the fact i dont like C++ syntax it feels messy to me and C is just nice and simple

Finally I just want to wrap up with saying John Carmack has to be one of my biggest Hero’s in Computer Science along with Migel De Icaza, Linus Torvalds, Various GCC hackers… like Ian Lance Taylor But John Carmack in particular since he made Doom! But also he attended the University of Missouri–Kansas City for two semesters before withdrawing to work as a freelance programmer. Which i think that is brilliant! Since University does not teach students to be serious programmers or technology well very few do anyway and i think its great he had the balls to do that and so did Migel De Icaza and they have both done some of the most amazing things in computer science!

Anyways if your interested in Crules or CrlDoom see: http://crules.org code for crldoom is over @ http://code.redbrain.co.uk/cgit.cgi/crldoom

This is a project i love Classic Doom 3 it was Doom re-implemented ontop of the Doom3 Engine what more do you want!!!

Michael

2010-01-01T01:12:36+00:00

Michael stands in the rain, as drops fall from his brow to the ground,
He waits patiently, his steel grey eyes surveying his battlefield,
It is a strange one, strewn with grass and flowers and many a mound,
And rank and file of stones each entreating that He should yield.

Scattered here and there, his kinfolk stand, but none as Michael.
He is the soldier, with breastplate and shield, and sword by his side,
Not drawn since he came to be here; he stands as a stoic sentinel,
Patient, benevolent, the fire in his spirit not worn with boastful pride.

Michael stands in the rain, as he has stood in the blistering Sun,
And when stars wheel above him; when snow coated, but not numb,
Rather with exultant wings renewed, he stands vigilant for the One.
Waiting for his ancient foe, the firstborn seraph, who does not come.

Virtual folders with Dovecot and Debian

2009-12-31T15:06:09+00:00

I use client side virtual folders a bit for my mail. Specifically, I tag messages with IMAP flags like todo and important, and then in Icedove / Thunderbird, I set up a special folder as a saved search which shows message that are either unseen, or marked todo in my inbox. It works rather well, and I use the same set-up on my laptop, and work and home desktop machines.

But it's not very useful on my phone, which doesn't allow such sophisticated client side behaviour. My phone mail applications shows the most recent 25 messages in a folder, but there are times when it would be really useful to look up messages that are labeled as important but rather old. It would be time consuming to look through the older messages, and difficult to find the one I want anyway.

As a result, I've been looking at the possibility of using virtual server side folders using dovecot on my Debian mail server. I was put off by the documentation which left a lot of questions unanswered.

Here's how I did it on Debian. First of all edit the config file /etc/dovecot/dovecot.conf, back up this file first, so you can restore working behaviour if something goes wrong.

#
# You have to add the default namespace
# which is normally NOT added explicitly before
#
namespace private {
prefix =
separator = /
# the next line is very specific to where you keep your mail
location = mbox:~/Mail/:INBOX=/var/mail/%u
list = yes
inbox = yes
subscriptions = yes
hidden = no
}

#
# Then add the virtual namespace
#
namespace private {
prefix = virtual/
separator = /
# pick where the virtual folders will be
location = virtual:~/Mail/virtual
list = yes
inbox = no
subscriptions = yes
hidden = no
}

You must also add the virtual folder plugin.

##
## IMAP specific settings
##

protocol imap {

# ... you need to enable the plugin
mail_plugins = virtual

Now restart dovecot and check your normal folders are working.

/etc/init.d/dovecot restart

Note that I found dovecot will generally not serve physical folders correctly if the virtual mail folder (even if empty) does not exist. I consider this a bug, but one that needs to be worked around, at least for me.

If that's all done and working you can begin to create virtual folders. I created two directories within my ~/Mail/virtual folders; which were inbox-todo and inbox-important respectively. Inside each I put the following files.

# ~/Mail/virtual/inbox-todo/dovecot-virtual
INBOX
OR (OR (OR KEYWORD $TODO KEYWORD todo) KEYWORD $label4) unseen

which shows all unseen and mail labelled todo in my inbox and

# ~/Mail/virtual/inbox-important/dovecot-virtual
INBOX
OR (OR KEYWORD $IMPORTANT KEYWORD important) KEYWORD $label1

which shows only important mail in my inbox.

It seems to be working, my normal folders appear to be working perfectly correctly (but I'll know better in a couple more hours/days); my phone has successfully subscribed to the two virtual folders, though the folder list shows a number of files which I'm certain it should not, again, this looks like a dovecot bug to be honest.

Neat find of the day: inputlirc

2009-12-24T00:19:57+00:00

I've recently been fixing up my VDR setup to work with FreeSat and make it brother/parent friendly. I've applied the EPG patches to get the 7 day guide, setup an autologging in user under gdm with vdr-sxfe running and that left getting the remote working. For some reason my old serial dongle wasn't happy with lirc - it got detected ok, and would show some signal when buttons were pressed but didn't work with the old config. The entire hardware of the box has changed, so it seems likely something isn't quite right (in particular the lirc drivers spew out warnings about SMP bits so I should probably try the dongle under a single core setup to rule that out, but there's also a move to 64 bit involved).

The easy solution to have something sorted for Christmas was to pickup a cheap remote from eBay. This ended up being a Cyberlink remote + USB dongle combo. Worked just fine when plugged in, turning up as a normal input device and the obvious keys doing the obvious things. I wanted all the keys to work though, as I'd got used to having a lot of the VDR functions instantly accessible rather than having to work my way through the menus. Various searches suggested I'd need to use LIRC to access the odder keys. That seemed a lot of hassle for something that was doing the decoding itself. Some playing with xev turned up keycodes for a number of the keys, but there were still a few missing (and important ones at that, such as Red/Green/Yellow/Blue). Further digging found me a suggestion of an Xorg keyboard map that would map the KEY_RED etc from the evdev device into something workable under X. And then I found inputlirc via the Debian package. This is really bloody neat - point it at an evdev device and it will present all of the KEY_* codes out as lirc keys. If you pass the -g parameter it makes sure the key presses only go to lirc as well. Exactly what I want and a doddle to setup - no messing with a big configuration file, just edit /etc/defaults/inputlirc to point to the correct /dev/input/by-id/ file, add the -g to the options in that file and restart.

Now the main remaining task is to get it working with BBC/ITV HD.

Fixing g1 Accelerometer

2009-12-22T22:56:23+00:00

I had noted before that my accelerometer had broken on my Android g1 phone, so that one axis was out by about 40 degrees. Despite lots of "helpful" suggestions from my friends on how to fix it (you know who you are), nothing worked. This did.

Connect to the phone with adb (from the Android SDK), in the following way.

./adb start-server
./adb shell
# cd /data/misc/
# mv amkd_set.txt amkd_old.txt
# killall amkd

Now it works.

Cyanogen Android ROM on the g1

2009-12-15T21:37:25+00:00

I've had my Google development phone, the g1, for some time now. I haven't had the luxury of time to write anything for it, but I mainly wanted it to try out Android anyway. I've reviewed the phone before, and again after some canonical firmware upgrades.

I use the truly excellent K9 application for mail, it has good support for self signed certificates, now has IMAP push support and is generally excellent. However, it stores all the mail on the shockingly limited internal memory on the device. That, and upgrades to things like Google Maps, adding truly excellent new functionality, left me constantly looking for applications to remove.

This is why in the end I decided to try Cyanogen's ROMs. Since I have a development phone, I didn't need to root it, and just followed the relevant instructions (in truth, I couldn't be bothered to downgrade the OS to root it first).

Here are some observations about the new ROM:

Apps2sd is amazing.
I have the whole pleasure of trying different apps all over again, without sweating about every byte. I don't have to worry about how much data is in my contacts (whether I assign them icons), my emails, and so on. I have plenty of room. I was delighted to be able to install DocumentsToGo. Which makes the phone much more useful for work emails. Loads of great apps I had to remove have been reinstated, and I can play with others, like the awesome Google Googles.
Extra workspaces
There are five workspaces, making for more widget playroom. I now have a calendar app taking up a whole workspace with the events to come. Excellent.
It fixes several problems I had with MMS functionality.
- It fakes a variety of user agents, meaning that a test video message I sent myself on o2 finally worked, for the first time ever.
- The stock ROM allows you to prevent data access when roaming, which is good. But it also doesn't fetch MMS when roaming, which is (for me) a nuisance, and these are usually on a different tariff system. So when you receive an MMS on roaming, you end up enabling all data access to quickly receive the MMS, and then turn it off again. The Cyanogen ROM has an option to retrieve MMS on roaming.
UI feels snappier
USB tethering
can be enabled, which JustWorks (TM) with Debian. Excellent.

On the downside, I have had some reset problems, but admittedly I have sometimes been pushing the phone very hard indeed to test it. And the battery life on the g1 is still awful. I know Noodles has solved the problem by not actually using his phone :-), but I want to use mine.

Another minor problem I encountered some weeks ago was the accelerometer suddenly starting serious misreporting on one axis. This problem seems to be becoming less severe, but even reinstalling the stock and then cyanogen ROM did not fix it. However, note I did not wipe the user data.

Cyanogen has made my phone fun to have again. And I will still replace it when a new Android handset comes out that I really like, but a lot of the urgency has gone. I'll certainly buy him a beer for Christmas.

Crules and Intermediate Representation

2009-12-03T02:47:00+00:00

If you follow my blog you may have noticed i have been quiet and thats because i’ve been working on http://crules.org Its the new homepage for my project Crules, its got some good content on it but nothing really on the language yet. Though today i spend a very long time on the HACKING article And i explain a lot on how Intermediate Representation’s work and 3-address code.

So lets just take an extract since i am lazy: So in compilers or interpreters how do we represent:

(( x + y ) -- ( ( x + y ) * ( x -- y ) ) ) + ( ( x + y ) * ( x -- y ) )

Lets see it in a diagram first then in IR code:

Expression

So what is this? Well early on when i was getting into compiler construction when doing semantic analysis you will need to find a way of representing these semantics for your language, and you will find generally that there are 3 things you need to know: DESTINATION OPERAND A and OPERAND B. This is what’s called 3-address code. This is what sets compilers apart and can make or break them! Its what gives implementations flexibility and the ability to ‘control the flow‘ of execution! Any good compiler book will give you more detailed discussion on this topic its my main interest in compilers since this is what gives an implementation its logic. So lets give it a syntax we can write down to illustrate how mine works: it just so happens a LISP ‘ish syntax can show this very well!

( ( IDENTIFIER ( TYPE ) ) => ( OPERAND A ) => ( OPERAND B ) );

Since the parser is precedence aware it will construct the tree as shown in a symbol language which looks like this if it was output ( outputting the IR like this is a future TODO ); It may look a little like GIMPLE but this IR I have as I will demonstrate more examples can encompass much much more.. ( this is a 3-address code by the way )

( ( NIL ( OP_ADD ) )
=> ( ( ( NIL ( OP_SUBTRACT ) )
=> ( ( ( NIL ( OP_ADD ) )
=> ( ( ( NIL ( SYMBOL_ACCESS ) )
=> ( ‘x’ )
=> ( NIL )
); )
=> ( ( ( NIL ( SYMBOL_ACCESS ) )
=> ( ‘y’ )
=> ( NIL )
); )
) ; )
=> ( ( ( NIL ( OP_MULTIPLY ) )
=> ( ( ( NIL ( OP_ADD ) )
=> ( ( ( NIL ( SYMBOL_ACCESS ) )
=> ( ‘x’ )
=> ( NIL )
); )
=> ( ( ( NIL ( SYMBOL_ACCESS ) )
=> ( ‘y’ )
=> ( NIL )
); )
); )
=> ( ( ( NIL ( OP_SUBTRACT ) )
=> ( ( ( NIL ( SYMBOL_ACCESS ) )
=> ( ‘x’ )
=> ( NIL )
); )
=> ( ( ( NIL ( SYMBOL_ACCESS ) )
=> ( ‘y’ )
=> ( NIL )
); )
); )
); )
); )
=> ( ( ( NIL ( OP_MULTIPLY ) )
=> ( ( ( NIL ( OP_ADD ) )
=> ( ( ( NIL ( SYMBOL_ACCESS ) )
=> ( ‘x’ )
=> ( NIL )
); )
=> ( ( ( NIL ( SYMBOL_ACCESS ) )
=> ( ‘y’ )
=> ( NIL )
); )
); )
=> ( ( ( NIL ( OP_SUBTRACT ) )
=> ( ( ( NIL ( SYMBOL_ACCESS ) )
=> ( ‘x’ )
=> ( NIL )
); )
=> ( ( ( NIL ( SYMBOL_ACCESS ) )
=> ( ‘y’ )
=> ( NIL )
); )
); )
); )
);

busy!

Though one big note its is ok for me to leave the representation like this since in my interpreter i just evaluate this at run time but how does this affect things like code-generators like proper Compilers? Well its quite simple in a sane compiler like GCC ( although most compilers will follow this idiom i am sure) introduce another layer for IR called RTL ( Register Transfer Language ). The wikipedia article lacks and most compiler books overlook this section very much since, most books are written by the academics and generally stay away from the back-end of compilers since that is the actual hard part since a code-gen is very specific to the instruction set your targeting and register allocation algorithms are very difficult and is a huge research project. Its said in computer science the most difficult things to build software wise is a good JIT or any Code-Generator for that matter. Anyways back to the problem lets take a smaller expression for sleep’s sake the expression:

unsigned int retval = ( x * 2 ) / ( 7 + y );

Will generate something like:

t1 = x * 2;
t2 = 7 + y;
t3 = t1 / t2;
retval = t3;

Anyways you’ll notice in my the IR keywords such as SYMBOL_ACCESS or SYMBOL_ITEM or OP_ADD etc, these are what’s called OP_Codes and are generally represented by some symbolic hex values since its cheap to check against for the implementation to figure out what to do with symbol ‘x’; So ok thats a quick and dirty intro to Intermediate Representation’s!

I think this lolcat will sum up your feelings at the end of reading this blog post:

And i started getting out my old grunge cd’s which include my Silverchair collection and this is on of my favourite bands of all time and this is a pretty good song!

DIY laptop SSD "upgrade"

2009-11-29T17:32:57+00:00

I'm due a new laptop; my Portege R200 is over 4 years old now, I find it much slower than my desktop (no surprise; P-M 1.2GHz/1.2GB vs Core 2 2GHz/4GB), it's had a new battery and it has an annoying whine on the screen unless you press the side in just the right way. It's still better for long periods of use than my EEE 901, which is why I keep it. In particular I'm off to LCA2010 in January, along with some associated travel beforehand, and I'll want more than the EEE for that, but probably not a brand new laptop. Which meant that when the R200 started throwing disk errors recently I wasn't particularly happy.

Having decided it wasn't worth buying a new 1.8" drive I looked for alternatives. eBay offered plenty of options for 1.8" IDE to CF adaptors and I picked up a Kingston 32GB CF card from eBuyer. I copied across the data from the old drive using a USB adaptor, frobbed grub sufficiently that I got it installed on the CF, then swapped out the drive for the adaptor + CF. I had to do the swap a few times; the first I forgot to have a kernel with ext2 support (the HDD was ext3, but I went back to ext2 for the CF). The second I'd done my usual trick of forgetting to populate /dev with things like console and sda* (ie enough to get to the point where udev will run). And then grub needed some prodding to boot without intervention. I got there in the end and I'm currently writing this from said laptop.

I've made a few changes to the Debian install in an attempt to make things smoother; basically the same tricks people have been using on EEEs or other slow SSD devices. No swap, Iceweasel synchronous toolkit writes disabled, filesystems mounted with noatime, /tmp on tmpfs, various daemons that I don't really use disabled. It's still noticably slower than with the HDD, particularly on writes. However it's quieter and as long as it lasts for the next 3 months I'm happy (plus after that I can reuse the 32G CF card somewhere else, which I couldn't have done with a 1.8" HDD as easily).

Now, back to eying the Lenovo X200s (ugly, trackpoint but 1440x900 display, good battery life) and Toshiba R600 (slower, lower res but pretty and trackpad).as potential replacements. Maybe there'll be something even shinier in 12" by next year...

Intel C++ Complilers Linux

2009-11-28T16:15:27+00:00

So if you know me you’ll know i think compilers are cool… yeah i know i am weird lol. So any-way’s this is something i never really got around to until last night. Which was setting up the Intel C/C++ compilers in my Computer which is running Debian. If you ever look at using compilers other than GCC such as like Sun Studio and Intel C++ you may find its very messy feeling. What i mean is they have like a nasty install.sh, and to be quite honest they fail lol. Since they generally only cared about creating RPM’s and hoping your running Red Hat since most customers who would buy these compilers would be running a Red Hat server farm. So if your project is using Autoconf and Automake you dont have to worry about a thing, once the compilers are installed it will be one quick minor change to your configure.ac. And your all set!

First off go and download the 750mb tarball of the Intel C compilers over at: http://software.intel.com/en-us/intel-compilers/

% tar zxvf l_cproc_p_11.1.059_ia32.tgz

% cd l_cproc_p_11.1.059_ia32

% ./install.sh

# follow all the instructions it complained i didn’t have java installed but you don’t need it. Once your finished it should all be installed to /opt/intel/Compiler

% PATH=$PATH:/opt/intel/Compiler/11.0/069/bin/ia32

% LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/intel/Compiler/11.1/059/lib/ia32/:/usr/lib/

You should be able to run ‘icc’ but you will probably get the error: ” /opt/intel/Compiler/11.1/059/bin/ia32/mcpcom: error while loading shared libraries: libstdc++.so.5: cannot open shared object file: No such file or directory

compilation aborted for test.c (code 127)”

So you need to install an older libstdc++ on debian it is:

% aptitude install libstdc++5

That should be it lets give it a test:

int main( )
{
;
return 0;
}

% icc test.c

Should work now! If your project uses a Configure.ac lets make a quick change:

Change your AC_PROG_CC to:
AC_PROG_CC([icc gcc cc cl])

This is the order in which the configure should search for each compiler. Since you need GCC installed to use Intel C++ you need to put ICC at the top of the list! Hope this helps someone!

C++ Templates for XCode 3.2

2009-11-19T16:17:02+00:00

Here are a couple of Project templates I made for XCode 3.2. One for a WxWidgets based project (wxMac 2.8), and one for a GLUT project. Hope they work.

GLUT C++ Project – Initializes GLUT and creates a scene containing a cube which can be rotated using the keyboard arrow keys. toggle an axis widget using ‘w’, toggle fullscreen using ‘o’ and exit using ESC. The FPS is displayed in the titlebar.
WxWidgets C++ Project – Initializes a WxWidgets based app with an empty frame.

三年間

2009-11-18T16:09:23+00:00

Karen and I have now been living in Japan for three years. When we decided to move here we thought it would be an adventure, and it has been. We’ve had some problems, we’ve had some fun, and we’ve had lots of fun-problem combinations. I think we made the right choice.

More about LHC black holes

2009-11-15T23:20:41+00:00

Last week I wrote a little about the size of black holes, and incidentally discussed very primitive calculations I did on the lifespan of any black hole created by the LHC.

A few days later, this interesting article showed the results of professional physicists on just how little such little black holes could grow, in some cases even if their lifetime was not restricted. Enjoy.

Playing with an e-reader

2009-11-15T14:25:09+00:00

I've always preferred dead tree to reading things on screen; I just find it easier. I've tried reading fiction in the past off a laptop and just didn't find it as enjoyable an experience; whether it was the form factor (hard to curl up with a laptop, even a netbook), the quality of the screen or the interface I don't know. Equally with technical documentation if it's something I'm using a lot I prefer a printed copy to flick through. However e-ink based e-readers are becoming much more common and affordable and I figured I should give the whole ebook experience another try.

I spent 3 weeks in October in the Bay Area on work, so I borrowed my Dad's Sony PR505 for the trip. First problem is that it's picky about USB charging - works fine if attached to a computer, but using a Blackberry charger looks like it's fine but then results in what looks like a hanging reader. However once charged it lasts for ages - I charged it before I went and only hooked it up to my laptop once during the trip to transfer some new content onto it. The form factor is also quite good; a bit heavier than a book, but not excessively so. Holdable in one hand, big enough screen that I wasn't squinting at it. Slightly too big physically to easily go in a coat pocket the way a paperback would unfortunately, but perfectly fine for taking to read at breakfast every morning in the hotel.

The screen was also much better than I'd expected. I knew the technology was different and supposed to be easier to read than traditional LCDs, but I was sceptical. Overall I had no problems with it. I was even quite impressed with the Sony's zoom function - one of the PDFs I was reading was too small when view full screen, so I zoomed in so it was readable. The nice feature was that then the "next page" button gave me the next bit of text to read, rather than the actual next page, so it remained very easy to navigate through as I read. A small touch, but very welcome.

This isn't intended to be a review of the Sony reader; that just happened to be the model I was easily able to borrow. I happened to see a Kindle in use on the plane and I was struck by how much bulkier it seemed, though having looked at the relative specs it seems this is entirely due to the keyboard rather than any difference in screen size. The Nook was also announced during my trip and I'd quite like a play with that as it looks quite nice.

Despite my positive experience with the Sony I'm not planning to go out and get one just yet. And that's the lack of sensibly priced content. When I bought my first mp3 player I could take my exising CD collection, rip it and be able to play it on my new device. This meant I got an immediate benefit of having my entire music collection with me all the time, just by buying the player. If I buy an e-reader then in order to get all my existing books on it I have to go and buy them again. What's worse is they'll cost me the same or more than I paid for the paperbacks. I can't go and exchange my Pratchett collection for the electronic versions for a nominal fee. I can't easily scan them in myself and produce some decent ebooks. I can't even go and buy the entire set for £20; I'd have to spend something like ten times that. With mp3s I can continue to buy the real item and also have it on my portable player. Or, while albums still don't seem to be much cheaper electronically than on CD, there is at least the ability to buy a single track if that's all I want. Books don't have a comparison. I'm not going to want to buy a single chapter, am I?

There are some sites out there that can provide cheaper ebooks - Rachel Willmer runs ebookprice.info which lets you compare pricing from different vendors. There's also Project Gutenberg if you're looking for out of copyright books. Finally Peter Corlett pointed me at the Baen Free Library, which I haven't downloaded anything from but will definitely investigate at some point. I still maintain that none of these are enough and that content provision will continue to be a hinderance for e-reader mass adoption until there is some fundemental change in the way its provision is handled.

Nethack on OS X 10.6 (Snow Leopard)

2009-11-13T14:47:16+00:00

O.K. So, this is kind of a followup to a previous post I did about how I like to setup Nethack on my Mac. However, more recently, some of the patches I had used stopped working on later versions of Mac OS (10.5 and 10.6). So I spent a bit of time looking into this today and I got some positive results. (update: I think the problem I had with Menucolor compiling was that it didn’t like the regexp method – I changed this to simply use the wildcard matching.)

Nethack with Status Colors and Menu Colors

Unfortunately, because the two patches I’m applying have resulting conflicts- it meant manually applying the patches to the source code in some cases (also: I’m lazy). I, therefore, don’t have an elegant method of actually applying the two patches that I can pass on. What I am doing instead is providing two things: The complete refined archive of the patched Nethack source which should be ready-to-compile on OS 10.6; And also the .diff file produced from a vanilla copy of the source and my own patched version. I will also provide my new updated nethackrc file. (ok… 3 things )

Here’s some details about the setup this provides:

The Status Colors patch is applied. This is a configurable patch allowing you to color code status effects – hunger, blindness, HP, Gold, AC, etc. It’s very, very cool!
Menucolors patch applied. This is a configurable patch that let’s you apply colors to popup menus. e.g. inventory items can be red for cursed, green for blessed, etc. Again, this patch is very neat!
The initial configuration for getting Nethack to build on mac has been taken care of.

So Here are the links to get everything up and running. I should say at this point that this stuff works on my machine and YMMV. If something is broken and you want my to take a look at it just leave a comment of send an email – I’ll try to fix it if I have time. You may also want to dig through the changes in the patch.

I hope this is useful for someone who wants a nice shiny setup of Nethack on Snow Leopard.

Black Hole Radius, or How I came to love the LHC

2009-11-09T14:58:39+00:00

I allowed myself to become rather distracted by my second years last week as the class was finishing. They were talking about an episode of Horizon that discussed General Relativity and theories of Quantum Gravity. What followed was a free ranging discussion on the nature of infinity, mentioned briefly in the program. But we also talked about the nature of a black hole and its size.

It's surprisingly easy to calculate this with reasonably elementary maths and physics. I first did this when I was about 17 (how very sad) using classical physics equations, and was astounded to discover that even so, the answer was correct (I checked it in the Encyclopedia Britannica in the library at the time).

Here is Newton's universal law of gravitation, between two bodies. It describes the force F between two bodies that are r metres apart. Let's take the one with mass M to be the black hole. G is a small (though mysterious) constant.

You can work out the energy needed to escape the black hole using the old stand by equation that work done is the force times distance traveled against that force, but that only works with a constant force, this force will change as we move, so we need to use the big daddy of multiplication, integration.

Specifically, we will work out the energy needed to escape from the event horizon, the surface at which the escape velocity is the speed of light, which is c (299,792,458 m/s). So the energy will be given by moving my little mass m from the radius of the event horizon, let's call is R to infinity, to show we have broken away.

Now, this should just balance the kinetic energy possessed by my little mass m traveling at the speed of light.

and if we rearrange for R we get that

In other words, the radius of the event horizon, the bit we think of as the "hole" is dependent entirely upon the mass of the object. Please note this is based on a very simple model of a non rotating black hole. Nevertheless we can do some nice calculations from this.

The Sun, would have to be compressed from its diameter of about 700 million kilometres into a radius of just under 3 kilometres. The Earth's mass would need to be compressed so much to form a black hole you would need to squeeze its radius of over 6 thousand kilometres into a radius of around 9 millimetres. That's how dense we're talking here.

We can also consider the radius as described by the contained energy of the black hole, since we know that

and so, replacing our M in our above equation we get

Wow. Remember c is a big number, taking it to the power of four is a lot. So why do this? There's been a lot of speculation about the possibility the Large Hadron Collider (LHC) could create a black hole. This has caused a fair degree of panic, and at least one suicide. It's a physicist's dream that a black hole might be created. I just looked up the "high" energies used by the LHC, and high is a relative term. It plans to bash protons together with 7 TeV (Tera electron volts) of energy each, or lead nucleii with 574 TeV each, let's take the latter. Just how much energy is that in a collision? Well, doubling and converting to good old Joules gives 184 micro Joules. That's really not a lot, 184 millionth's of a Joule. A 100W light bulb uses 100 Joules each and every second. How big would the radius of such a black hole that might form be, from that energy? Check the maths, because so far I haven't but I get.

metres

which is 0.000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,003 metres wide (I hope, I didn't double check the zeros either), which is probably not the planet swallower of people's imagination (but this is fun). But the problem is people think this tiddler will grow very rapidly, but that's because they don't know about Hawking Radiation. This is an interesting quantum effect that means black holes aren't really black, they do emit a little radiation. Large holes would gather surrounding matter faster than their low radiation rate, but small holes have the opposite situation, they radiate more rapidly. The maths for all that is pretty complex, and you need to make lots of assumptions, but the time taken for our little black hole to "evaporate" is (hurriedly calculated by me)

a tiny, tiny fraction of a second. Even allowing for the ambient temperature and some fall in of matter, this little baby is not in equilibrium, it's not getting mass fast enough to accumulate more. It's safe*.

* All disclaimers apply. No liability is assumed for foolish unvalidated experiments done by you or other members of your species. Do not attempt to create black holes in your garage. Any subsequent destruction of your civilization, planet or solar system is at your own risk, and any "EPIC FAIL" signs placed by aliens on the remains is not due to me or my calculation. No calculations have been done on the matter of strange matter either. If you break the planet / system / galaxy or universe you own all the parts.

Whoops

2009-11-07T03:46:40+00:00

So yeah i have lost a fair bit of content but i had some server trouble and reverted to an old backup but at least the other people i host on here didn’t seem to loose anything so that was a huge weight off my mind anyways its nearly 4 am and its nearly all setup now just need a quick bit of exim tinkering and various bits of pluming to be checked out! So hopefully it will be fine!

Eat the Jam!

2009-11-01T17:46:26+00:00

Ladies and gentlemen, May I present “Eat the Jam!” The only game which combines a high powered pop culture quiz with a taxing conserve based forfeit.

Vidzone working at last!

2009-10-20T22:57:09+00:00

Success at last. I've written before about my problems getting vidzone working, and tonight while taking a breather from some work I tried the Oracle of Google again, and I found a solution. As I suspected, it wasn't a network problem at all.

Kudos to lyonix, who suggested that deleting a corrupt cache for Vidzone from the game data utility often works. It worked perfectly for me, (note even removing and reinstalling the software doesn't affect that).

Perl’s XML::Twig

2009-10-13T16:24:20+00:00

I was asked to “Free the code” from my XML parsing experiment , so I will post some here. It may be a bit disappointing though, since these are only some short scripts, and they’re a bit ugly. I’ll explain the Perl one today, and do the Haskell sometime soon.

I was playing with Jim Breen’s Japanese dictionary and I wanted to make a list of the first kanji component in each entry. I wanted one result for each entry, so I used “(none)” if the entry has no kanji part. This is not a difficult problem, although XML makes it as slow and memory intensive as many difficult problems.

use XML::Twig;
my @keb = (); # for the results

sub entry {
    my ($t, $e) = @_;
    my $kt = "(none)";
    if (my $k = $e->first_child("k_ele")) {
        if(my $keb = $k->first_child("keb")) {
            $kt = $keb->text();
        }
    }
    $e->purge;
    push @keb, $kt;
}

my $twig = XML::Twig->new(
    twig_handlers => { entry => \&entry }
);
$twig->parsefile($ARGV[0]);
$twig->purge;

# now the results are in @keb

Using XML::Twig is quite simple. When I create the parser I tell it how to handle the elements I care about, and in this case I only care about “entry” elements. When the parser finds an entry, it calls my entry subroutine, passing the entry’s object as the second parameter, $e. Inside the entry routine I can use DOM-style methods on $e to extract the data I want. Notice that I call $e->purge when I’ve got the data out. This tells the parser that I won’t need that element again, so it can free the memory. This is how XML::Twig manages to parse a file that most other modules can’t.

What to wear?

2009-10-13T06:17:08+00:00

I tend to dress quite casually - unless there's a good reason to do otherwise I'll be in jeans and a t-shirt, or something similar. I'm comfortable in it and I take the view point that in general people shouldn't be making assumptions based on what I'm wearing. Of course they do, and while this can be infuriating at times it can also be amusing. The man at the RBoS who witnessed me signing the personal DD guarantee form for Black Cat obviously couldn't reconcile my appearance with what I was there to do, for example.

I'd never thought about this from the other side until a few weeks ago. I spoke to someone who explained how difficult it had become to conduct business meetings with other companies with the increased level of business casual. In fact on occasion different branches of the same company that he'd be meeting with would have different dress codes, so he'd go to something involving casual dress in the morning and something with full business suit attire in the afternoon and end up feeling over or under dressed.

This isn't likely to make me change my own behaviour in the immediate future (I don't have to interact with external parties as part of my current job). It did help me realise that there was another explanation for awkward behaviour when I'm my usual scruffy self that wasn't just about judging on appearances though.

(All of the above blatantly obvious once you think about it, but it took that conversation for me to do so.)

Android 1.6 (Donut) initial thoughts

2009-10-08T00:40:33+00:00

I installed donut earlier this week. This is on my G1 devphone, so I don't recommend you use the htc page otherwise. This time I didn't bother going through all the nonsense with the tools, I just mounted the phone on USB and slapped the image across one by one into the SD card as update.zip.

This is not as significant an upgrade as cupcake was but nevertheless, it's improved the phone in a number of key ways.

the search widget now searches phone content as well internet content, this works very smoothly;
the camera / camcorder apps are now easy to switch between, and much faster;
the gallery is quicker, but only after it takes a complete age to index/whatever;
the market application is hugely improved, and as a bonus, lots of content that was previously unavailable on the devphone is now listed;
the speech synthesis stuff is installed, but I don't really see it does much just yet;
the battery monitor app is very useful, especially since the g1 battery life sucks so badly.

My battery life is, by the way, 29% taken up by wifi (forgot to switch it off today), 18% by display, 14% cell standby, and so on. 8% was taken up by voice calls.

Be warned, the new upgrade seems to suck up more space, and takes a long time to start-up after first reboot.

You can see a full list of the changes here, and though it's modest, I think it's a nice upgrade in lots of ways. At this time, it's hard to know how much limitations in the G1 experience are from the hardware or the OS, I'm looking forward to seeing how android behaves on new hardware.

Gallium3D on AROS

2009-10-08T00:10:38+00:00

Seems that Gallium 3D acceleration is being ported to AROS. Haiku, another non-mainstream OS, has some support already, and of course, Linux and Windows have always been target platforms for Gallium.

At this rate, Gallium might just be the thing we’ve been missing, that can bring 3D features to all platforms, in the same way that all platforms have been able to do decent 2D for a long time. If so, this will be a very welcome change to the IT landscape.

GUI Kernel OOPs Dialogs

2009-10-07T10:49:12+00:00

Hmm. I’m running debian/KDE in virtualbox. A lot of syslog entries just popped up from the systray, quickly followed by some analysis/explanation: the kernel just panicked, and I can submit a bug report.

This is highly cool. Last I knew, kernels just locked up on panic. But to keep going, present nice GUI prompts, and let me submit bug reports over the net? Very nice.

Union Hand-Roasted sumatra (dark insight)

2009-10-06T23:15:55+00:00

Brand: Union hand-roasted
Bean: sumatra, extra fancy
Region/Producer: Gayo Mountain Co-Operative, Aceh.
Roast:
Grind: Cafetière

Test brew: Espresso
Test Equipment: Kenwood Cremissimo

Full bodied, deep rich taste, minimal bitter aftertaste, but not overly sweet. Very impressed. 4/5

Beating down the XML

2009-10-06T18:04:38+00:00

XML is still a huge mess, but at least now I have managed to get a few programs that can handle it with reasonable-ish memory requirements.

For Perl, as I had thought, the XML::Twig module gave me a pleasant interface and was able to easily handle the document.

For Haskell it was a little bit trickier. I used the SAX parser in HaXml, but it is not like a regular SAX parser, since Haskell is so unlike any regular language. The parser returns a lazy list of SAX events, so I had to make sure I processed the list without evaluating the whole thing into memory.

Now that I’ve dealt with the memory issue it appears that I have a speed issue to deal with next.

Compassion in the Martial Arts

2009-10-04T23:49:00+00:00

I recently had a conversation with some people who wondered if there could be any place for compassion in the martial arts in general, and certainly in Iaido, or Japanese swordsmanship in particular.

It may be useful to start by considering the idea of Bushido, or "the Way of the Warrior". A lot of people have heard about Bushido, but often through a very distorted perspective. This is not least because the ideas have been smeared by the inexcusable behaviour of some Japanese in the Second World War. These individuals were not, by the way, samurai. That class had long since been dissolved in the Meiji Restoration.

The closest analogous concept for us in the West is that of Chivalry and the word has similar linguistic roots. Now, we do not believe that Christian knights always behaved in a way that was above reproach, do we? Certainly not, but we do not throw out the baby with the bath water. The legend of Arthur is a romanticised and idealised idea of chivalry but one that has inspired many to better conduct.

Bushido lists seven virtues (some say the seven folds in a hakama represent these), which I will borrow the listing of from wikipedia rather than mess with the kanji myself:

Rectitude (義, gi^?)
Courage (勇, yuu^?)
Benevolence (仁, jin^?)
Respect (礼, rei^?)
Honesty (誠, makoto or 信 shin^?)
Honor (誉, yo^?)
Loyalty (忠, chuu^?)

Do you find any of this list surprising? It might temper your view of how a warrior might see himself. Perhaps you did not expect respect and benevolence to make such a list.

I think it's a nice list to consider with a martial art like aikido which is a modern budo that seeks to avoid gratuitous damage to the "opponent". For the practitioners of such an art, it provides a compass for our behaviour. I constantly emphasise the importance of compassion in aikido when I teach, and strive to embody it in practice. There are actually pragmatic and selfish reasons why this behaviour is martially more effective; you do not damage joints you need to effect control; you do not provoke responses you need to contain and so on. Some might argue if compassion with self-interest is still compassion. The concept of cause and effect, or karma provides a strong incentive for acting appropriately, isn't it a bonus that it is not merely "right" but also effective?

But the sword is not a weapon designed to control or subdue but to kill. There is a Japanese word katsujinken, the sword that gives life. What can that mean? It can have many meanings. It could mean to not use the sword when not necessary, to fight without fighting.

Compassion requires the love of others. I feel I have long understood that. It has taken me longer to realise it also implies the love of one's self, and I still struggle with the consequences of that. Compassion can manifest itself in many ways. Exercising compassion may require hard and apparently brutal choices to avoid greater suffering caused by another path. Such choices can be be traumatic for the one who acts as well as to those whom it appears to affect the most. Reconsider the list of seven virtues in that context if you will. The samurai were required to take quick, just decisions, and see them through, with all that entailed.

To assist another as they commit Seppuku was (it is now illegal in Japan) an act of respect, benevolence and compassion. There are many stories of samurai acting as seconds for both their closest friends, and their defeated enemies to shorten their suffering. To regretfully, but decisively take a life to protect one's own can be an act of compassion (to one's self), and in many cases as an act to protect the lives of many innocent others.

A final thought from the modern era. If you found yourself in a hijacked plane post 2001, with the split second opportunity and means to kill the hijacker, would it not be a compassionate act to swiftly and decisively kill that person before they could take any action to kill so many others? Might it not even be an act of compassion towards that person and their family?

If you seek mastery of the sword, seek first sincerity of the heart, for the former is but a reflection of the latter.

Iwakura Yoshinori

Derren Brown tells porky pies, get over it

2009-10-04T22:14:45+00:00

Derren Brown has been back on our screens recently with a series of big events. It's brought him a lot of publicity, and a lot of the public comment has been amusing to say the least. I've been watching everyone comment on the TV and Radio that he's a trickster, and his explanations of how he does things are often bogus.

Yes.... Yes... That's the point. He's a magician, a showman, he tells you right up front at the start of the show. Misdirection, magic, showmanship and more. It's entertainment. He is the first to make it clear that what he does is a trick. He doesn't claim otherwise. If you can't bring yourself to enjoy that, I humbly suggest you watch something else. And while we are on it, I'm relieved people see it's a trick. I think David Blaine's closeup magic is among the most impressive I have ever seen, and I shudder when those around him don't even seem to consider it's just a trick.

Personally, I love Derren, I love his cheeky smug as hell smile as he gets away with it. I enjoy his faked discomfiture. I enjoy it when he fakes near success in tricks to make them more convincing. I enjoy calling him names as I laugh as he lies through his teeth to the audience. I have casually studied a bit of magic in my time, and he is a fine magician. Many big stage magicians rely on assistants, who actually do all the hard work, but he is clearly the architect of most of his own tricks. I'm reading his book and it is informative and very amusing to me, and I very much respect him as a fellow in fight against (rampant) irrationality (a little bit can be a fine thing).

Sadly I've missed a few of the "events", since it clashes with my Iaido class and I wasn't organised enough to record it. I'm slowly catching up on 4OD. But I did see the program on how he stole the lottery. And I'm a mathematician, and I laughed and laughed at it. I didn't believe a word of it, but even the deceit was cleverly convincing and it's not trivial to say exactly what he really did.

He claims he averaged the results of a crowd of people picking lottery numbers. There's a consequence to this. Consider this: how many ways can 50 people pick a number that averages to 1? How many ways can they pick numbers that average to 30? If you understand this point you'll see the possible "answers" from the crowd have an odd distribution. But the cover story was very amusing in its own right. You don't want to know how he really did it, it'll be shockingly dull.

So yes. We know he didn't show the numbers in advance of the draw. Yes, we know his explanation is nonsense. It's a trick. That's the point. Enjoy it and smile along with his insufferable smugness .

Virgin Media, Netgear and Vidzone

2009-10-04T21:53:48+00:00

The PS3 has an free network application called Vidzone. I downloaded it when it originally came out some time ago, but it never actually worked. It would load, show playlists, adverts, and the seize up the whole console when I tried to actually play videos. It's a shame, because it looked like a nice app for putting on some music.

Since then the PS3 itself has had a big firmware upgrade to 3.0.1 and Vidzone was upgraded to 1.0.4. Worth trying again I figured. Well, initially I couldn't get the application to crash the console anymore (I did later!), but certainly it wouldn't play videos. It did show adverts, even video adverts, playlists, everything but actually play videos. Which is weird to say the least. So to be honest, it looks like the problem is somewhere at the PS3/Vidzone end, since absolutely all other network functionality works on the PS3.

But, as a precaution I thought it was worth checking out that old bugbear, the router firmware, and there are some hints that's an issue. I'm a Virgin Media customer at the moment, for complex reasons I don't really have a choice at the moment, and they supplied a cable modem and router when they connected me (rather late as it happened). When I looked at the router firmware, it's shockingly old. So I clicked on the link for the knowledge base and downloaded the latest firmware. It failed to upload, no explanation of why. I tried every intermediate release, which took quite a while to do. They all failed.

So now I'm suspicious and confirm that, yes... Virgin are specifically blocking updates, even though many, many bugs have been fixed in the new releases. I find this pretty puzzling.

I phoned their tech support to try and confirm this in person. I suggested that I would have to buy my own router if I wanted to get round these problems, and they informed me I wouldn't be supported. I put it to them that I seem to have a choice between being supported with no actual support and no means to help myself, or getting new hardware, being officially unsupported but able to help myself. He put me on hold to check it out, a minute later the line went dead.

So if you're out there googling for "vidzone doesn't work", this might be part of the reason, but who knows? It seems it could be fixed at the PS3 end in any case. But it's a cautionary tale that the stock VM hardware might come with lots of problems you can't fix.

XML is a huge mess

2009-09-29T16:27:28+00:00

I have a 39 MB XML file that I wanted to process. I wasn’t expecting it to be so difficult. Writing the code, in multiple languages, was not difficult. But running the programs was a big problem.

My first attempt was a simple Haskell program, but I had to kill it after it ate over 1.3 GB (yes, 1.3 GB) of ram!

Haskell’s strings are known to be memory hogs, and the HaXml module I was using was making them even worse by not sensible decoding the UTF-8 text correctly. I decided to write a leaner Haskell program later, and switch to Perl to get the job done.

At this point I also decided to set a limit to the amount of memory the programs could consume. For a 39 MB file I hoped that 10 times that would be enough, so I rounded up and set the limit at 512 MB.

But Perl, using the XML::LibXML module, couldn’t process the file with that memory limit. I also ran a quick one-liner in Erlang, just to watch it crash out of memory too. I’m going to try some other languages to see if I can find one that can work in 512 MB.

My next useful step is to try the XML::Twig module in Perl. I’ve had good experiences with it before. It won’t be as fast as LibXML, but it probably has the best chance of surviving within my 512 MB limit. For Haskell, I think I’ll have to resort to a SAX style parser.

Amazon 1 - 0 Waterstones, aka this morning's rant.

2009-09-29T10:24:10+00:00

SWMBO asked me to order a book for her this morning, so being the tightwad that I am, I go hunting for the ISBN and the cheapest place to buy it.
Amazon has it for £17.24 and Waterstone's for £18.49 (but if I order through quidco, then I receive another 8% cashback, making Waterstone's cheaper).

Waterstone's need me to create an account, fair enough and state "Also, please be aware that passwords are case sensitive, and must be a minimum of seven characters long and at least one character must be numeric." Again, all fairly standard.

So each time I try to enter a password which meets this criteria I'm hit with a javascript popup that claims: "Your new password must be at least 7 characters long and contain at least one digit."

"But it is!", I yell in frustration.

Using the Firefox "Web Developer" plugin I unhide the password boxes and see that my password doesn't have digits... wtf! I enter the digits again and find there is another hidden rule:

<input name="newPassword" type="password"
id="fPassword" size="40" maxlength="16" value="" />

Yes, an upper limit of 16 characters on the password.

Right ok, make a password of 16 chars or under and move on...

Add name, postcode to the next form and it finds my address ok - click continue to (I presume) proceed to the CC entry form and voila!

Well not quite.

"There has been a problem processing this request
Please use the refresh button on your browser to try again.

Thank you."

A few refreshes later, it is apparent that Waterstone's has no intention of working, sod it, for 20p more I can have less hassle at Amazon.

Class::Accessor can has “has”

2009-09-20T15:20:58+00:00

I maintain the Class::Accessor module. It appears to be used a lot, but the API is a bit ugly. In YAPC::Asia the ugly API was criticised in at least three different talks, and each time it was compared to the fashionable Moose API.

In one of these talks JRockway asked Shawn Moore how to turn a bad API into a good API, so I’m going to try that: adding antlers to Class::Accessor!

So now instead of writing:

package Foo;
use base qw(Class::Accessor);
Foo->mk_accessors(qw(alpha beta gamma));

If you prefer Moose-style you can write:

package Foo;
use Class::Accessor "antlers";
has alpha => ( is => "rw" );
has beta  => ( is => "rw" );
has gamma => ( is => "rw" );

The original API is still available, and everything is the same underneath.

Snow Leopard can DIAF

2009-09-16T21:50:12+00:00

Hurray! Apple fanboys should be pleased with the release of OS X, version 10.6. Snow Leopard panders to their desire for an all Apple platform. For the rest of us sensible users, Apple have screwed the pooch.

Okay, we have new shiny toys like Quicktime X and – well, that’s about the height of noticeable changes. I’m all for enhancing the user experience, but in this case, in my opinion, it has come at the expense of developer comfort.

What I’m referring to here is the lack of Java support in the new XCode 3.2. I’ve always felt that XCode was a fairly elegant environment for writing all C/C++, Java and Objective-C projects. So- why, Apple, would you arbitrarily drop support for a language? Out of spite? You didn’t even drop support – you just made it frigging irritating to use. Do you enjoy kicking your users in the nuts with each new release?

I’m not even surprised at this stage. I guess I could add the XCode templates back in manually but I’m more in favour of adopting a more portable command line build process… It seems the best way to get away from Apple’s user abuse.

It’s alive!

2009-09-14T17:53:19+00:00

My blog: it’s alive!

I don’t post very often, but I’m going to try to change that. Is this my fifth attempt?

This time, to give myself a goal, I joined the Perl Ironman Challenge and I will try to blog at least once a week about Perl. So…

Perl: it’s alive!

There have been lots of reports over the last few years about Perl being dead. Those reports upset a lot of Perl mongers, and I didn’t fully understand that. Perl was not a family member, friend, or pet; so why the strong emotion? It was never really “alive”, so how did it “die”? And all these upset people were still using Perl, so they kept it breathing. And there were many more Perl users who weren’t upset, maybe because they never heard about the death.

It seems to me that Perl never died: it just became unfashionable for a while. And during the unfashionable period Perl did have some self-image issues, and maybe a lot of misdirected energy. But being unfashionable isn’t life-threatening.

Windows updates

2009-09-10T14:25:28+00:00

I am forced to use a Windows based laptop in work and one of the more frustrating features is the updates that get pushed out by the IT ’services’ group. Of course, updates are a good thing, but we do them during downtime when it won’t disrupt actual work. Here are the things I most hate about these Windows updates:

Do it now, or else.

There are updates available for your computer. You can install them now or they will automatically be installed in 15 minutes. After installation your computer will be restarted.

I’m sorry, were you busy?

As if point 1 didn’t irritate me enough (which is most certainly does), updates frequently occur right in the middle of a large task. Copying a large file or set of files is doomed to failure. In other cases the updates will simply interrupt anything you happened to be doing, forcing you to break your concentration.

Update Dependencies

OK, I’ll install your updates already! Tick, tick, tick… *reboot*
Your updates have been installed. There are new updates to install.

Yes, some updates rely on the presence of previous updates so it takes a couple of runs to get them all installed. which also means a couple of reboots; a couple more minutes of your precious day.

And one of my personal favourites: The “Your computer must be restarted dialog” which always stays on top of other windows. In case you forget.

Jedi training in Belfast?

2009-09-01T08:43:47+00:00

Ok, ok, bear with me here .

Allen Baird and Rory O'Connor will be offering a personal development seminar in Belfast on the 17th October, but with a twist. Like a previous event that Allen has offered, the whole thing is steeped in metaphors related to the Star Wars movies, in order to put over some serious ideas in an accessible and fun way.

The event will explore lots of concepts like non violent communication, body language, assertiveness among others. I will be offering a slot at the end with a little aikido, to explain how it physically embodies some of the other concepts in resolving conflict with a minimum of force. And I'll be helping John Donaldson demonstrate some iaido, and finally we'll have some fun (I hope) with the lightsabers Allen and Rory are giving out to to the delegates as part of the package.

And then, we will go home and rethink our lives.

Campaign for an apology for treatment of Alan Turing

2009-08-31T17:09:27+00:00

The website for Number 10 Downing Street has a new petition calling for an apology for the treatment which led Alan Turing to commit suicide at the age of 41, rather than submit to having his homosexuality "cured" by chemical castration.

Turing was, not only a founding father of Computer Science, but a leading member of the dedicated team at Bletchley Park, who decoded the Nazi Enigma code. This work saved thousands of lives.

As reported on the BBC, the originator of the petition, John Graham-Cumming, is not only suggesting an apology, but a posthumous knighthood for Turing. It would seem the least his country could do for him, after all he did for his country, mathematics and computing.

If you're a UK resident, please consider signing the petition.

Python, GCC and Crules :-)

2009-08-25T17:28:33+00:00

Hey guys, I’ve been pretty quiet this summer i guess so as it may seem, but I’ve been working frigging hard. I’m starting to get things coming together! So what am i doing? I’ve been doing a little freelance server work, I’ve been working on CRULES which is coming along nicely a lot of the back-end is working pretty nice now and i just need to keep working at it, but i am also trying to start a new front-end into GCC. So what does that mean, so in GCC the gnu compiler collection its a collection of compilers i am writing a small paper on the architecture of GCC and how to build a new front-end but more on that later, but what i mean is each language in GCC be it ADA, Fortran, C, C++, Java they are all front-ends to GCC, that is the part which does the semantic analysis and parsing of the pain text source code into the GIMPLE tree for middle end optimization from there code-generation.

So i am making a Python compiler(front-end), well starting too its going quite slow because i am in the middle of finishing up some automake work, and crules definitions. And GCC is quite complex in its development but its quite amazing and the most amazing project i have ever seen. I mean its development consists of many little files in different languages that aren’t even front-ends in GCC. For example a .md machine description is written in LISP that describes the architecture of an OS or Processor this makes it easier to port GCC to new systems, GIMPLE the IR was a language called SIMPLE but its GNU/Simple = GIMPLE, this gives the IR a better formalization, lang.opt files for command line options so you dont have to do any command line parsing in your front-end thats up to the gcc driver program( I’ll explain this some other time), Make-lang.in GNU/Make on how your code should be compiled, config-lang.in its in bash but does all the configuration on how the front-end is build with respect to Stage-x as in the boot-strap if it requires libraries to be built like JAVA needs libjava or if its an optional language.

The force!

Thats a lot of stuff people probably don’t care about! But its interesting what other project uses a cool a setup as that! I mean LISP, there isn’t even a lisp compiler in there but there its a bit of a one in the backend. There are some exciting things coming from GCC at the moment, LTO or graphite springs to mind, they aren’t ready yet, but LTO mean Link time optimization, so when you compile your program with gcc in whatever language, you compile each file to object code individually but you use gcc to link each object code and library together instead of using LD explicitly because it works nicer, so on Link time optimization when your linking all the code together another pass of the optimizations goes over a huge IR of all the object code to re-structure of remove dead code etc all the usual optimizations. And with OPEN/MP becoming pretty popular graphite you can pass flags like -fparallelize-all to extract loops even dependence loops to push in open/mp pragmas to make fully parallelized code without having to lift a finger. I’m working quite slowly to write 3 papers in total, a long thesis on Crules you can check it in my git-repos over at http://code.redbrain.co.uk and i am working on a thesis on the python-front-end as-well as a little paper on the gcc architecture. There is a tagged branch for a front-end-skeleton here on my git-repos: http://code.redbrain.co.uk/cgit.cgi/gcc-python/

So there is a lot of work there. And i chose to do a python front-end due to the fact everyone can say python is an awesome language no-one can deny that, it has been very well designed; language design is something a lot of people can over-look or take too far and python was a good mix in my opinion. What i mean is if you spend too long on language design your language will become very messy and hard for users to become comfortable with, something that has happened Haskell in my opinion although the way non-determinism is very clever in how Haskell has implemented it. On the other hand if you don’t spend enough time designing your language its going to be very messy and not much re-use of syntax. One thing to remember in design there is always a reason for the smallest of things be it the reason of using an identifier to using a ‘.’ for access or ‘- >;’ for pointer access, keywords have to make sense etc.

I was invited to join the graphite project i posted a quick patch to fix a part of their build-system, but i am not sure what i can really contribute because i would need to work full-time on it because its been going for some time now. Some of the work is pretty exciting. I am going to see later if i can maby merge the actual python parser from python into my GCC front-end because this would make a very nice setup to mirror the language features of python to an extend but there will be differences an GNU’isms because the mere fact that this will be compiled handling things like IO will rely on linking against libraries and the python pre-processor functions like import and from etc will be fiddly to work but i should be able to take alot of work from the gnuJava front-end as there are similarities in how certain things work.

Something to Distract you!

Anyways from my goggling i haven’t been able to find an implementation of python that is compiled properly i mean natively compiled, some people mix the up the fact that python does create byte-code like java think its compiled code and it really isn’t. I remember at university a lecturer told us that compiling JAVA code you were creating Machine code to run natively on a computer. And he got the pipeline so wrong in creating a binary. They always made it seem that a compiler was a piece of magic that made a binary and java is NOT a compiled language unless you use GCJ. I personally find linkers are a piece of magic these days a lot of the know-how in how linkers are build is quite complicated in some ways. Although what you are essentially doing is taking the object code output from an assembler and creating an ELF format file with the object code in the correct segments and data in the top segment etc. And on windows its a COFF format i think, but even at that what are you doing and what does object code look-like is it literally the binary format of instructions in order from the corresponding assembly code?!

I would love to understand more on how that all works, but i mean an object code file is a not a plain text binary file so yeah i even posted to comp.compilers a thread on manging the jit you can look for it in the archives but its something that is still quite hidden, and only a few people who implemented it understand, i see how it all partly works because of the libjit project. like the defintions of an a.out and ELF were quite understandable i guess. But with a jit if your able to for example jit a function like:

int multiply( int x, int y )
{
return x*y;
}

If you are able to make target assembly code for this, and you assemble it and link it properly you have a binary how do you actually execute it in a useful way for an interpreter?! I mean do you keep adding new functions into this image and use dlopen functions and treat the symbols like a dynamic library and execute them that way through dlopen? I mean this is what the LLVM project hides from the users very well but anyways enough of that.

Anyways thats what I’ve been doing for a summer what have you been up to?! You can follow some of my stuff on several mailing lists like bison,autoconf, automake, gcc-help, gcc,comp.compilers. I helped a guy a while back on his bison parser, bison is a very useful tool, but for doing more free-form grammars is a pain I’ve found and i like to think i know YACC pretty well now from using it for about a year.

By the way I’ll be posting a wiki links on crules soon for example on how the language works etc and why but i would really like it if people read my thesis when its finished because i designed this language from a very specific set of reasons in the beginning for a specific application, but its become a fully multi-paradigm language in its design now, and this is what my thesis on it describes.

Anyways i’m going back to get some work done and i have BLUG tomorrow! Oh hey i found a new favourite band bought their album as soon as i heard this, its a polish metal band they are probably the a new Nightwish i guess! They are very good!

I think thats enough for now i have some photos to upload but i’ll do that next time

Choosing a development hosting solution

2009-08-10T16:02:00+00:00

I'm considering changing the development environment we use to host our projects in work and I'm not alone. The reason why will become more clear further down. To be honest this article is partly to help me work out what my priorities for this are, and to appeal to the lazy web for any other suggestions.

So first of all, here are the things I'm looking for in this project.

These are essential:

must be free software;
must have flexible issue tracking with user defined fields;
must be able to handle multiple projects;
must be extremely transparent to non developer users;
must support svn and git and graphical front ends to them;
must support some kind of announcement system.

These are desireable:

should integrate in some way to mailing lists;
should support ad hoc tar bar downloads from scm;
should be easy to search issues;
should graphically depict progress on releases;
should have documentation (wiki like) integration;
should handles news and download areas;
should be able to exchange data (like issues) on projects;
should actively maintained;
should automate most sysadmin activity, account creation etc.;
should be packaged for Debian ideally.

The contenders thus far are:

Savane, which we are currently using;
Fusion Forge;
Redmine;
Trac.

Let's take these one at a time.

Savane

Savane currently supports all the essential features, with the exception of git support. Being a fork from the original SourceForge software it works in a similar way. A web front end (in PHP), with back end functionality (in Perl) that creates shell users and groups, interfaces with mailman and so on.

But there are some problems. First of all the official project seems to have ground to a complete halt. Fixes that were submitted by users are not applied, there seems to be no forward momentum whatsoever. This is a problem, the offical version needs patched (modestly) to run on PHP 5, and the code base is a mess, with a few pages still requiring register_globals to be on. Bad.

However, another fork was taken some time ago which addresses all these problems including git support, well actually, the code base is still probably in need or work, but most apps of a certain age have this problem. What I'd like to see in Savane is better graphical tools to monitor project progress, better documentation features and incidentally better Debian packaging, but I've signed up to the fork to work on the latter on the first instance (this looks like it will be quite a bit of work, it's non Debian policy compliant in many ways right now). The new fork is going to be forked again by the way, into a Python version. Could be interesting.

Fusion Forge

Previously known, or descended from GForge, Fusion Forge is another fork of the original SourceForge source code. It is well packaged for Debian (naturally, it runs Debian's own code hosting environment, Alioth) and well maintained. It has many of the features of Savane, which probably makes it puzzling why I didn't choose it in the first place. Well, there's one reason why, in my opinion the front pages of FusionForge are rather un-user friendly, I mean what the heck is "Code Snippets" doing there? They feel very aimed at developers, which is great... but I need a nice straightforward interface for less savvy users. Looking today, the navigation still feels it's just too developer centric, but as these sites double as developer sites and user sites (to acquire the software, report bugs and so on), that's not great. I really don't know yet whether Fusion Forge has better graphical tools than Savane.

In a conversation about this last week, Noodles suggested I should look at the ease with which the interface could be changed. A sensible suggestion I shall follow up.

Redmine

Written in Ruby on Rails and advocated by my work colleage Paul Vitty, Redmine is clean and elegant looking. It shows all the signs of benefiting from being a later generation project and has a plugin architecture that seems excellent too. It has easy clean wiki integration, graphical road maps and Gannt charts; and these things make a difference. I don't think it's as remotely scalable as the design of the first two systems, which isn't a huge problem for me, but I mention it in passing.

On the other hand, it seems to have little or no automated integration into the backend, user and group creation, mailing lists and so on. on the other hand it has a very rich set of plug ins, so it may already have such support or it may be possible to implement it.

Trac

Written in Python, Trac feels like Redmine lite, and that's in the wrong direction for me. It explicitly does not have multiple project support and that's a deal breaker for me.

I haven't come to a strong conclusion about this so far. I note John (whom I linked above) notes that Redmine is difficult to install and upgrade because of Ruby, this is a hassle I just don't need, but he also thinks it may be the best of the field. For me, it is probably more trouble free to migrate to the fork of Savane or to FusionForge. More thought needed, and your thoughts are most welcome!

Violence and Gender

2009-08-10T15:17:35+00:00

Recently, I've spoken to a few people about an English tourist having his genitals set alight in Crete. Apparently he was exposing himself to a local girl and fondling her.

If so, his behaviour was just plain disgraceful, and regardless of the justice of the situation as defined by the law, a stern reminder to him and others that actions have consequences, no matter how fair those may or may not be.

But what interests me most is the almost universal mirthful reaction to this story. It reminds me of the furore around John Wayne Bobbit, which was also regarded as all but universally hilarious by both genders. That relationship was very abusive, and hardly hilarious in its own right. Bobbit seems to have a record of physical as well as other abuse doled out to his partners, but what really interests me is the gender polarity of the amusing violence.

Suppose a woman abused her husband in a relationship; and in turn, one night when she was drunk he took a scalpel and removed one of her breasts. Who's laughing now?

Suppose it was a British girl in Crete, annoying a man; again who laughs if he sets her on fire? It's interesting how we are programed to trivialise and laugh concerning violence against men that would be considered an utter disgrace and conversation stopper against a woman.

Back from DebConf9

2009-08-02T19:57:39+00:00

Finally back home post DebConf9; felt reasonably productive and I have a lengthy todo list which I've even written down and started work on. More on that when any of them come to fruition.

Very much enjoying the return to easily accessible decent tea and salad/other vegetables. I haven't eaten any fish since my return and hopefully won't for at least a few days. Also I feel the need for an alcohol free week so my liver can recover. Back to work tomorrow - what is it I do again? Also, bets on how my cow orkers will react to my purple hair?

Update – Interpreter CRULES & Jamedo

2009-07-27T23:51:42+00:00

So hey, its been a little longer than i intended to update my blog! But i’ve been busy, so i have got rid of my personal work being on repositories all over the show. So I have made:

http://code.redbrain.co.uk

There will be public git access soon, but you can download the snapshot tarballs of the branches. And I have a wiki too there isn’t anything on there yet but it will give me room to stop putting long tutorial article’s as pages on my blog to proper wiki article’s:

http://wiki.redbrain.co.uk

So Compilers, Interpreters and programming languages are my interests… *yawn! So let me show you my interpreter pipeline for my interpreted language ‘CRULES’:

Crules

So I have to admit that’s the most impressive art I ever done (I know I’m lame ). Its my pipeline for my programming language, but I guess a lot of programming languages would follow this kind of model/architecture. So it got me thinking this is nearly the basics for any semi-serious interpreter or ‘a’ language so lets talk though it.

I am still not going to talk about the details of my language because its not ready yet. Other than I am doing fun things like, so far the only data-structure I let the developer have is lists, so if you do something like:

% foo := { 2,3,4,5,6,67,7}

#is a list and strings are lists!

% str := “bla”

#is the same as

% str := { ‘b’, ‘l’, ‘a’ }

It works a lot around the shell so you can also do:

% foo := `ls`

#it produces a delimited list of strings with the delimited as carriage return or new line etc.

Though note I am going to implement an api for arrays. But not yet, because lists are going to cause problems unless I implemented a table/array of accessors to the elements in the list so you can do fast access to the elements because it doesn’t scale no matter what anyone says when say you are doing image processing on a 1024×1024 list… accessing the elements isn’t going to scale.

Anyway so lets get back to looking into the pipeline of it: So first a user creates a program lets look at the helloworld.crl:

rule main()

{

echo “Hello World!”

foo := 5 + 2 – 7

}

all := main;

So first my interpreter will start lexing this using LEX, into tokens like: rule, main ‘(’ ‘)’…. and then the parser starts applying the grammar implemented using YACC. So I start producing symbols to build a symbol table the data-structure looks like this its basicly a more high-level 3 address code:

struct symtab {
const char *identifier;
uint8_t symType;
uint8_t op_a_t;
uint8_t op_b_t;
union {
_SYM_LIST_ *list;
float flt;
int32_t integer;
char ch;
//sym link to function call
struct symtab *syms;
} op_a;
union {
_SYM_LIST_ *list;
float flt;
int32_t integer;
//params to a func_call
struct symstack *syms;
struct symtab *sy;
} op_b;
struct symtab *next;
} __attribute__((aligned));

So first it creates the rule definition! For now just think as rules as entry points to a program so you can add more identifiers to that ‘all’ list not just main. It has a deeper meaning but I don’t want to talk or show this until I have the interpreter and language definitions more concrete!

So the the rule ‘main’ creates a new symbol like this:

//the symbol takes the identifer of the rule identifier

symbol->identifer = ‘main’

//its a rule symbol

symbol->symType= _SYM _RULE _

symbol union_a is the link to the first symbol in that block of statements and union_b is NULL because I don’t need it until I have rule dependencies it needs to list.

symbol->next= NULL, because it is just a definition

Then each expression like ‘ echo “Hello World” ‘, creates a symbol and its next = the next expression like the ‘ foo := 5 + 2 – 7 ‘. This creates an interesting one, because this can’t be reduced to a 3-address code until you split it up, for now I just assume the developer wants it to run as 5+2=7 then 7-7=0; But I am going to add precedence later. So you can be more expressive to have nested expressions in brackets to be executed first etc..

But yeah this expression is split up into 3 symbols for now, its an over all variable Assignment but it depends an expression of 5+2 as a temporary symbol then this temporary symbol – 7 then foo = this new temporary symbol and I can push these dead symbols to the garbage collector.

When a symbol is allocated, it is allocated memory of a preallocated memory heap of default 32MB, it can be extended or shortened if needs be, but it allows me to control and see how much memory is being occupied at anyone time in the runtime of the interpreter. Any symbols to be kept such as rule definitions, function definitions or variables are kept on one massive stack. So ok, that’s a problem if you need to search for them, the answer I found was having 3 other tables that link to the symbols in the stack. When any link is broke it becomes garbage, so when I have a new variable re-assignment I can break that link and remove the entry in the variable table it still exists in the symbol stack but the Garbage collector is always called on EXIT and the exit of any block statements like loops functions etc, so when you pop your way out of the stack you can have local variables instead of everything global and you have conflicts which will cause a lot of problems. Also the garbage collector is implemented as a pthread so you have have to pause for several seconds waiting for it to free the garbage. I haven’t done this yet because its going to be very awkward to keep the concurrency, the pthread mutex api should handle a lot for me but its still going to be complicated. I have thought I might like to try implementing a lot of different internal using OPEN/MP to make it even speedier, but that won’t be for a while.

And then I have been playing with LLVM to create a JIT execution system, you can see my posts on ‘Managing the JIT’ on comp.compilers, I have posted there quite a few times before, its a great list of basically all programming language, compiler and interpreter experts in the world!

So I have a variable table that links to the variables in the symbol stack and the same for the rules and functions. This speed up searching for specific identifiers.

So that’s the basics of parsing into symbols I guess, it pains me to say but many people thing a programming language is mainly a parser is so wrong the parser just lets you build your symbols the execution and management of these symbols is much, much more complicated. That’s not to say implementing a simple programming language is difficult, the problem I see a lot is people are scared of doing it because the vocabulary is huge! I mean if you start working at an interpreter from first principle’s I’ll guarantee if you have half a brain you’ll come to a similar architecture as this.

I found it difficult at the start when I knew anything in a programming language can always be brought to 3 things and I got really confused when people talked about symbol tables, 3 address code, IR (intermediate representation) and parse trees or syntax tree’s. I think a lot of developers have their own prefered meanings to them, but it is mainly a barrier of academics trying to make the subject more complicated than needs be, although when you go at this from first principle’s you’ll will soon learn .

So thats the main focus of my personal work but I am now also soon to be an approved GNU/gcc and GNU/automake developer! Can’t wait, got my FSF paper work signed and sent back it may arrive in Boston soon I guess but yeah I am excited. I am updating the automake documentation to show the best way to handle multiple LEX and YACC in one program. Its something only one of two people do but its a problem none the less and to make it portable is the actual problem. But I have the solutions to it now .

Got your attention

Thing is I would really love it if I had an AIX/FreeBSD/Solaris machine to do more testing on because now that I am doing work on GCC it would be so helpful to have more machines to do testing. But i’m ok for now, so far in GCC I am doing regression testing to find a regression I can tackle, its extremely difficult some are defunct on the bugzilla some are ancient and not perused. And the rest is very active but for MIPS or SPARC, and I don’t have a scratchbox setup to even test that yet. Would just be great to get my hands on even an old ibook for a power-pc cpu or an old IBM p-server. If anyone knows of any available I would be really grateful!

Ok this has become a very long post so I’ll shut up! . See you guys from Beflast LUG on Wednesday! One last thing i want to link you to 5 of my favourite bands and they are all off Jamendo:

Steganography, prevention before is better that detection after

2009-07-27T19:53:24+00:00

New Scientist recently ran an article about steganography. If you don't already know, steganography is essentially encryption with a difference. Specifically, encryption is usually obvious. It may be that the data Alice sends via email to Bob with public key cryptography is entirely secure from eves-dropping by Eve (pun intended, sorry), but Eve will know data is being sent that she might be interested in. Steganography, by contrast, seeks to hide the encrypted data so Eve is not aware of its very existence.

It's a very ancient idea, stretching back to ancient Greece. In modern times a common way to perform the trick is to hide data in an image. One of my more gifted undergraduate students did a final year project on this with me. We used a known password as a seed for a pseudo-random number generator to determine which pixels of the image we would embed the data in. By playing with the least significant bit of one colour in randomly spaced pixels, you can very effectively hide data.

The New Scientist article suggests that if you detect the steganography, and if we obtain the computer of the suspect and if they have carelessly wiped the software, there might be traces that tell you this was done. Now let's remember the whole point of steganography is that the first step is improbable, you most likely won't detect it.

The issue is, in today's geopolitical situation, reasonably serious. It has been suggested (see the wikipedia article I linked above), that such techniques were used to exchange data on site like ebay to plan major terrorist attacks. With lots of analysis software only playing with known algorithms, or relying on comparing modified images with the original (where the original may not be available) what can such a major website do to prevent such abuse? Well, I thought an approach would be to essentially employ the same techniques with random data. That is, randomly poking data into bits in pixels here and there will, up to a certain point, not affect image clarity to the naked eye, but unless the encrypted data is loaded with huge amounts of error correcting code, it will destroy the payload. You could easily automatically run such a filter over uploaded data. I'm sure similar approaches would work for digital sound.

Receiving Ukemi

2009-07-25T01:04:00+00:00

For those not acquainted by martial arts. The title is a bit of a weak play on words, since "ukemi" means something like "receiving through the body". Uke is generally the receiver of a technique and thus the attacker in aikido, whereas the nage is the thrower to use one possible word.

Ukemi is important in Aikido. You will spend literally half your training time attacking and then "receiving" the technique, and all of these aspects can be lumped under the title of ukemi. It's a difficult balance to get right in aikido. It is learned in a very cooperative nature, and so the idea (for some) is be an "appropriate" attacker for each training partner, tailoring your attack to each. Thus, for a student on their first night, you might actually move the student's arms to help them learn the "throw".

Issues tend to arise more in pairings between non beginners. Some advocate that as an uke you can learn aikido by blending completely with the nage. I personally prefer an approach of attacking in a way I believe is probably appropriate for a "normal" attacker, and then protecting myself and blending in the aftermath. I'm probably slightly above average height, weight and strength and so a good model of an attacker, and perhaps somewhat harder to throw.

Having said that, I do not prevent people from throwing me. I just don't strive to throw myself, blending so completely that every throw seems to "work" perfectly no matter what.

Recently I received a fair amount of verbal and non verbal (expressions and so on) criticism from a student of about a year's experience about the "awkwardness" of my ukemi. I think in retrospect I should have indicated that unhelpful, non constructive criticism is as unpleasant and counterproductive for me as to anyone else. I did however, rather unhelpfully, suggest "you might want to consider that I am not the whole problem". I was exasperated at the fact that this person thought I was trying to stop her from throwing me, when in fact I was trying to help her explore where things were going well and where they were not. The exasperation was not helped by my conviction (rightly or wrongly) that if I really wanted to stop her from throwing me, I could have done so very easily.

The reality is, there are very, very few people to whom I would give concerted "awkwardness" to stop them throwing me, and those are among the people whose aikido I respect the very most, and it would be something I would do rarely, and as a gift to that person (and because it can be a great deal of fun once you are comfortable with another person!). If you're reading this, you know who you are! If I'm trying to be "awkward", you'll know about it.

When I hit 3rd Kyu (around the middle of the white belt grades), I took a decision I simply wasn't going to worry about awkward ukes any more. It's my problem to deal with the uke, not theirs. And now a good few years later (about nine) I find myself teaching Aikido, I emphasise this point to all the students in our club, and it's very much our philosophy. Nevertheless, there are times I am training with another when I don't "finish" a technique when I feel their ukemi doesn't allow it. Those are times when I feel I would have to be profoundly unkind to do so, or potentially injure someone, especially when the person is (in my opinion) unaware of their own danger. Mostly however, I believe in the importance of finishing, providing I'm not putting my ego above the uke's safety.

But I believe awkward (but realistic) ukemi is a gift to receive gratefully, that some people I train with, though I like them very much personally, are so intent on blending perfectly with everything I do before I do it, that I am deprived of a chance to learn from my mistakes. I suppose I should have, with humble sincerity, explained this to my training partner. Better luck next time.

Richard Feynman Lectures

2009-07-25T00:03:10+00:00

I hold Richard Feynman in huge regard. He was a fascinating human being, a Nobel laureate physicist, his research in physics was second to none. But he was also a legendary lecturer, in both the fields of physics, and perhaps surprisingly computer science. And even more, he was an exceptionally well rounded person, a gifted artist, an amateur safe cracker and more besides. I own a copy of his lectures on Physics, bought for me by my Mum who felt (probably correctly) that no-one else would buy an item that sounded so boring, though it was on my Amazon wish list. (Incidentally, I think some pages touch on issues like the paradox I presented on crashing cars, I haven't had the leisure to study this more closely).

Recently it was announced (and one of my students kindly wrote to tell me) that Bill Gates had bought up the rights to his lecturers and was making them available. I do praise Bill Gates for his philanthropy, and would have praised him for this, but regrettably, the lectures are only available with Silverlight, and so it's another of a long line of Trojan horses to ensure we buy into a new proprietary standard from Microsoft. A huge shame.

In my last, marathon article, I talked a little about models of reality. A point I didn't make is that we have trouble accepting that; no matter how much we dislike aspects of reality, they remain the same despite that. Feynman encapsulated this beautifully in this YouTube snippet of his QED lectures (which I had showed to my final year students). I have attempted a limited transcript below, but you should hear it in Feynman's excellent good humoured Brooklyn accent for full effect.

And then there's the ... kind of thing which you don't understand. Meaning "I don't believe it, it's crazy, it's the kind of thing I won't accept."

Eh. The other part well... this kind, I hope you'll come along with me and you'll have to accept it because it's the way nature works. If you want to know the way nature works, we looked at it, carefully, [...unsure of this bit...] that's the way it works.

You don't like it..., go somewhere else!

To another universe! Where the rules are simpler, philosophically more pleasing, more psychologically easy. I can't help it! OK! If I'm going to tell you honestly what the world looks like to the... human beings who have struggled as hard as they can to understand it, I can only tell you what it looks like.

And I cannot make it any simpler, I'm not going to do this, I'm not going to simplify it, and I'm not going to fake it. I'm not going to tell you it's something like a ball bearing inside a spring, it isn't.

So I'm going to tell you what it really is like, and if you don't like it, that's too bad.

If you'd like to hear more from this fascinating man, can I suggest more YouTube videos showing an old BBC interview with him:

Home directory on SD card?

2009-07-18T17:26:54+00:00

I'm off to DebConf next Thursday. I'm looking forward to it; I could do with a change of scenery and some time with friends. I'm having my usual "What do I need to bring?" worries and I got to wondering about which laptop to bring - since last year I've gained an EEE 901, but I still have my Toshiba R200.

In general I'm using the EEE these days when I'm not using my desktop - it's much more convenient to throw in an overnight bag. However the keyboard and screen are too small for hard core use. Which suggests I should take the R200. Except the EEE battery life is great and a small laptop is handy for use in talks. So I think I'll bring both. Which led me to think about the fact I don't use the R200 that much these days and why that is. And it's partly about all the various bits that live in my home directory that mean switching machine is a sort of context shift. That applies to my desktops too.

And then I had a thought. Both laptops have built in SD readers. So do both of the desktops I regularly use. Why not just put my home directory on an SD card? These are machines that I don't log into remotely, so the card being removed when I'm not in front of the machine isn't a problem (it means I have to log out, but I think that's acceptable). I'm not totally sure of the speed of the readers in the various machines, but I guess the best way to find out if it's doable is to try it.

So, various things to ponder:

Card size. Do all my readers support SDHC? In which case I should get an 8G or 16G card. Otherwise 1G is probably the safest maximum size? I suppose I can order one of each; they'll get used somewhere even if it's not for this.
Card make. I don't want something that's going to die after a week of use; I'll try to ensure cache directories and similar are symlinks to a local piece of storage but I don't have a feel for the number of writes my home directory normally sees. I like Crucial for RAM. How's their flash? Integral? Or just bite the bullet and accept Sandisk are going to be best, if pricey?
Filesystem. ext3? Or will the journal kill the card faster - I understood this was less of a concern these days. Are ext4 or btrfs ready for this sort of use? Perhaps this is the right time to try; I can keep backups on every machine that uses the card easily enough.
Crypto. I may as well encrypt the card for security as I doubt that'll end up being the bottleneck for access. Is dm-crypt the right thing? libpam-mount looks like it might let me tie things together in a simplish fashion.
Union mount? It might be nice to have a basic home directory on every machine so I can login even if I don't have my card with me. Or have local configuration bits specific to each machine. Perhaps something for a bit further down the line - I'm not sure any of the unionfs options are in mainline kernels yet?

I suppose I'll do some card manufacturer investigation and try and get a couple of cards ordered in time to play with over DebConf.

Models, Perception, Science, Religion, Martial Arts

2009-07-18T13:42:43+00:00

This is quite a long and detailed article, if you have no vague interest in meta-physics or philosophy and associated ideas, it may hold little value for you. You have been warned!

I doubt that any human being lives directly in reality. None of us has an exact understanding of reality and generally I suspect that's a good thing. Those human beings who are gifted with an ability to see reality more uncloaked often pay a heavy cost for that, you can see this from reading biographies of our greatest scientists and artists. So in fact, we compose models of reality in our mind. Actually our very brains use models to simplify the massive processing required from our senses, magicians exploit the limitation of these models all the time to entertain us. Martial artists can exploit them for self defence.

Models usually start off simply, along the principles of Occam's Razor. So for example when we are young we observe the Sun rising and setting in the sky and we take it as the simplest, and reasonable explanation that the Sun is moving around the Earth and not the other way round. The stars appear as a fixed background that wheels around the Earth, reinforcing the idea of a geocentric universe. But they are not uniformly distributed, and now we hit a factor that can often run contrary to Occam's razor; humanity's ability to discern patterns in the environment. It's an important ability, and lies at the heart of the innate mathematical ability that defines us a species as much as our gift of language. But it often misfires. It leads us to see agents where there are none, and we are predisposed to suspect and fear agents that are essentially like us - anthropomorphic. This is also, incidentally, the root of our fondness of conspiracy theories although the irony of that cartoon appealing to one agency about our intrinsic ability to perceive agency should be considered.

We begin to construct theories as to the distribution of the stars, and we prefer solutions of order and agency to randomness. So we see constellations, and we name them by appearance. But why do the stars look like these things? It must have a meaning, so we build stories bringing all of these things together. And then, we notice things that wander against the stellar background, and indeed today we still use the word planet, derived from the greek observation of this wandering. These must be great, special things. And to these we attach godhood. An increasingly, some might say ludicrously complex model of the world emerges.

This pattern was, of course, repeated across the world. In the western world we have been significantly shaped by the beliefs of the Egyptians, Jews, Greeks, Romans and the old Norse models of the world. Most of those where themselves derived from others. Each became progressively more complex and intricate as time progressed. Depending on your perspective, Christianity is often blamed or praised for being the greatest influence on our world model today in the west, but in fact I would argue that the Greeks have that dubious honour. And to illustrate how ideas from old models are often patently false but very hard to leave behind, many (perhaps you) still believe Aristotle that there are five senses when there are more and we still use words like "quintessential" without much thought as to what they actually meant. Indeed what we think of as Christianity today is heavily influenced by Greek philosophy.

The ideological battle within us between the simple and complex, rich models continued. Most of the world moved towards belief in a single God, laughing at old beliefs of many spirits or many gods. Without commenting on the validity of these religions, it's worth noting that if for example we look at very early Christianity, it was a very simple model of the world. In the centuries that have followed more and more layers of tradition have been tacked on to make a richer, more complex cosmology. Every so often, a schism occurs and a faction tries to return to fundamental simplicity, but usually every branch continues to grow in complexity thereafter.

Returning to physics and meta-physics, for a time, we knew the Earth was a sphere, and then forgot again in favour of flatness as a simpler model. Many suspected the Earth was still round, and then apparently we discovered it (though of course we were mistaken in fact). But the idea of the geocentric universe was generally still very secure, as was our model of humanity being the very purpose of the universe itself.

When Galileo constructed his telescope he used it to look at the greatest planet in our Solar System, named after the greatest Roman God, Jupiter. He saw what anyone alive can see today with a cheap telescope or binoculars, up to four dots near the great planet. Sometimes not all these dots were visible. It was extremely difficult to understand what could be going on. If everything orbited the Earth then the motion of these dots must be exceptionally complicated. But there was a much simpler solution, that these were moons of Jupiter, orbiting the distant planet directly. Four moons, the largest of the numerous moons of Jupiter and still collectively named for Galileo today. To us, living within a well established (more accurate) model of the Solar System that we have today this seems of no consequence, but this is an example of that overused phrase a paradigm shift. In other words, it required Galileo to make that leap that the model of the world espoused by Aristotle was quite simply wrong. There is an enormous mental inertia to be overcome in such an act; we become very attached to our models, they are nothing short of our perception of the reality we exist in. Remember, one branch of the Christian church only pardoned Galileo for this "heresy" in 1992! But when we can and do embrace the improved model, other deepening of understanding can rapidly follow. So for example, once we accept that moons can directly orbit an entity other than the Earth, we begin to question all the assumptions of what goes around the Earth.

And so in a short time, we move from a model, that began so simply and became so complex, where the Sun, Moon, planets and stars revolve around our beloved Earth, to a much simpler model where only the Moon keeps us company in this way. There is massive resistance because of our huge investment in our model; and also because it diminishes us as a species, and our home. But when we accept the truth, or at least our improved model of it, we come to an understanding of the astonishing grandeur of the universe, so much greater than we could ever have believed before.

It's happened again since of course, odd little inconsistencies in the Newtonian model of the universe emerged. The pattern observed by Kuhn appears again and again. We greatly resist the evidence against our current model. The model is again, our very perception of reality. Eventually the evidence mounts up so much that we can no longer ignore it; it has to be explained. The explanation that eventually comes is again, shattering intellectually to us. And in fact that is getting worse.

Specifically, most people still happily live with the Newtonian model of reality (even most scientists). That's because it's an astonishingly successful approximation to reality, it works brilliantly well in almost all situations. As I write, near the 40th anniversary of the Apollo 11 mission I should note that Newtonian mechanics alone were sufficient to land a man on the Moon. (Incidentally, note that the names of the missions that took us there, first Mercury, then Gemini, then Apollo in the great Saturn V rocket show how we drag our old obsolete models with us centuries later). However, we know the Newtonian model to be wrong. We know that General Relativity and Quantum Mechanics are better models, and unfortunately, they are not simpler. In fact they are so complex, we fight our mind and brain, evolved to understand a Newtonian world, as we try to encompass these theories.

And for the first time physicists are facing a shocking possible addition to reality, one that rocks them to the core in the same way the incompleteness theorem did for mathematicians. We may simply not have a sufficiently evolved brain to totally perceive reality. Perhaps all our models are such a simplification of reality that in the same way a dog cannot understand general relativity or quantum electro-dynamics, we shall never be able to understand the true Theory of Everything if it exists. Certainly it is harder than ever to train young humans to the summit of current understanding in their twenties, when their brains are still supple enough to probe reality (few great discoveries in science and mathematics were made by older people).

But suppose for a moment that the theory of everything exists, and is simple. Many people believe this would give us full predictive power over all the emergent behaviour that arises (i.e. literally everything, including love, music and art). Of course that doesn't reckon with, just to begin with, chaos theory which actually doesn't mean what most lay people think it means, especially since Jeff Goldblum propounded it in Jurassic Park in a way, that to me, sounded much more like the totally different catastrophe theory. (By the way, following that film, I have watched the total bemusement on the face of a kindly elder pure mathematics professor as he was asked by a prospective student if he was a "chaotician".)

But actually, and coming to martial arts in this, the truth can often be safely hidden in plain view. Most of us will never find it or believe it. Note I include myself in "us". As human beings, we actually love complexity and tradition and can't accept simple truth any more than we can understand complex mind bending theories. For now, I set aside the fact that some will always use complexity and tradition to exploit us by hiding reality from us, it's quite enough to deal with self-deception at the moment.

When a new student sits on the side of the mats watching an instructor demonstrate aikido (for the sake of argument) they are looking at two human beings with flawed models of reality attempting to do their best to demonstrate an underlying reality. No matter how hard they try, or what they know, it will not be perfect. They fight the interpretation of what the other is, what they are doing, what they themselves are and what they are doing. In his recently published memoirs Alan Ruddock discusses how the mental aspects of this for each of us are shown by analogy in the old Chinese book Journey to the West introduced to many of us by the Monkey TV series.

Add to this mix that the student, actually whether they are a beginner or not, views the whole proceedings through their own flawed model and perception. What they see is rarely (possibly never) what is exactly happening. Watching a particular "throw" the student can see a start with two people standing, and a finish with one lying on the ground. The arms are used to cause the throw to "happen" and so a very simple model is that the person still standing at the end essentially struck the other person to force them on the ground. I've watched this very process happen in a room filled with beginning students, who after witnessing a fluid gentle throw that brought no harm to the "victim", proceed to stand with their equally inexperienced partner and more or less try to knock them onto the ground in a very crude approximation of the throw they watched.

So it is for beginners, but actually with experience it's just the depth of the misconceptions that alters. For instance, person "Anne" throws person "Barry" while person "Clare" watches. Suppose to simplify things that Anne does a perfect (if such a thing exists) aikido throw on Barry who is gripping both of Anne's wrists with his hands. She moves in complete harmony with Barry so well that Barry cannot resist and is thrown perfectly. As Anne moves - Barry, who is gripping her moves too - and his body position relative to hers changes, this in turn changes his grip and as a consequence he rotates Anne's arms. Anne neither opposes nor amplifies this, she merely continues to move where she can till Barry loses his balance and falls.

But Clare, watching from the sidelines, knows it cannot be that simple. Anne must know a special trick or two. She has so much to watch. So much information to assimilate into her model: how Anne's face moves, her arms, her legs. After all, Anne is "doing the throw" to Barry, so Anne must be doing the special things. She sees that Anne's arms rotate as she performs the throw. She cannot possibly perceive that actually it is Barry that is doing this, and even if she could, even if her highly sophisticated mind could believe it, is is likely that the primitive model most of us have will prevent her from being able to accept it.

And so Clare learns to perform this throw by moving her arms. It's a flawed model, but actually it may not be too far from the truth, so mainly it works. The times it doesn't, well these are just aberrations that can be put down to other factors (just as Kuhn has observed in science). The longer Clare trains this way, the more deeply ingrained this flawed model becomes. She starts to tack on little adjustments that help correct for the times it diverges from reality. She learns a host of tricks to deal with times that this causes the "throw" to begin to fail, so she can "make it work".

Perhaps one day Clare starts to teach others. She teaches them honestly, sincerely, as best as she can, but from a seriously flawed model. The model propagates to many other people. It's an interesting example that memetics doesn't guarantee that "good" memes or models of the world survive and prosper. The correct model Anne tried to demonstrate is astonishingly simple, much simpler than the one Clare has assimilated. But now, the intellectual investment Clare has made makes it unlikely that she will ever really see the true model. Worse yet, the model she has learned has crept into her every body movement. Even if she could somehow intellectually grasp the correct model, she must fight every "lower" part of her body to effect it.

Perhaps there can be seen here the parallels between these threads. That models spread rather like religions whatever their nature is. Loyalty to a personality or a concept can be laudable, but it can prevent people from questioning things for themselves, preferring to attach themselves to the models espoused by others. But however much we may respect another human being and attach value to their beliefs, I believe it's our responsibility to ourselves, and to them, to remember that they are fundamentally like us, imperfect, doing the best they can. If they falsely believe something however sincerely and we come to believe it too out of loyalty, we only increase their attachment to their incorrect model, and the chance they will never move from it.

So again, even leaving aside the malevolent attempts by some people to deliberately deceive ourselves (and a yet deeper discussion would examine our beliefs about such motivations as we are predisposed to see malevolence where there is none), our very nature is our greatest enemy. We resist perceiving reality as it really is, we each have rival models of reality which leads us to clash with each other when we "cannot see things from each other's point of view". We literally cannot. We naturally embellish models to make them more complex than they need to be, we enjoy it. We struggle to believe things could be simple. We often conflate the ideas of "simple" and "easy". Simple things can be hard to do.

All we can do is make a massive effort to see things as they are, as individuals, or as as close as we can. In the Japanese martial arts, we would say that we should strive for shoshin, "beginner's mind". We need to constantly challenge our assumptions about the world and other people. In many senses it's a very Buddhist philosophy. It's arguable whether it is always an act of kindness to help others refine their models. It may be, in extreme cases, an act of violence to attempt to wrest someone from a model they are so deeply attached to. And yet some of those models can themselves be damaging to the individuals that hold them and those around them. Few would argue that attempting to refine the model of a man who despises and may injure people of a given race because of his deeply but incorrectly held views would be a bad thing. But incorrect can be a hard thing to judge. None of us is perfect, none of us perceives reality as it is. Is it right for an atheist who is absolutely convinced of the absence of a God to try and disturb the world model of a terminally ill believer who derives much happiness from their model, whether it is correct or not?

Hello, Lifeforms!

2009-07-12T16:42:01+00:00

Let’s start with the results, so everyone’s clear what this is about.

UmS6D0aW62+q!'%S2 (perfection:   0%) Evolving        2nd generation.
aUKahChc.5,e&8:?1 (perfection:   0%) Evolving        3rd generation.
bYJo$F?O2;aTV'q'? (perfection:   0%) Evolving        4th generation.
P]N0+dC=p-GVh<p1E (perfection:   0%) Evolving        5th generation.
P]N0+dC=p-GPQ>+6b (perfection:   0%) Evolving        6th generation.
Q?T1LHOp=LSj*VUr0 (perfection:   0%) Evolving        7th generation.
ob3<2Ih=p-CO^f#`5 (perfection:   0%) Evolving        8th generation.
dK8:SCn^omSj*VO ` (perfection:   0%) Evolving        9th generation.
X@Z'H<f.)T?f&(i6. (perfection:   6%) Evolving       10th generation.
g&H0+)DG^J%GV(\(l (perfection:   0%) Evolving       11th generation.
dK9K3Vf.)T?f&`Kfq (perfection:   6%) Evolving       12th generation.
8@Zm`,jZ-*jf38":; (perfection:  12%) Evolving       13th generation.
g&l:7O^q0jGA2r2,& (perfection:  12%) Evolving       14th generation.
j^4'N.:L0fV.9_Psi (perfection:  18%) Evolving       15th generation.
co2sL:BLffGA2((Y! (perfection:  18%) Evolving       16th generation.
MHX<oUL$EP`<o(\si (perfection:  18%) Evolving       17th generation.
g9l#bkrL$+e &>Tsr (perfection:  24%) Evolving       18th generation.
Pibho,MYilkEo(.sj (perfection:  29%) Evolving       19th generation.
cel/opLRP2aEerTs! (perfection:  35%) Evolving       20th generation.
HR(so ELrfef/\-s& (perfection:  41%) Evolving       21st generation.
ce>l9dr3iPef?r2sc (perfection:  41%) Evolving       22nd generation.
HRl#) EaibkforTs! (perfection:  47%) Evolving       23rd generation.
Ho9ho, LJ-?fY>ms! (perfection:  53%) Evolving       24th generation.
He^j`,YLrfeHFlms! (perfection:  53%) Evolving       25th generation.
HRli*/ Lrfe &rms! (perfection:  59%) Evolving       26th generation.
HelBNd Lrfe &rms! (perfection:  65%) Evolving       27th generation.
HeGlo,jLifGf&rm]! (perfection:  71%) Evolving       28th generation.
ce4lod >iWeforms! (perfection:  71%) Evolving       29th generation.
H&lho, Lif@forms! (perfection:  82%) Evolving       30th generation.
HRllo, >ifeforms! (perfection:  88%) Evolving       31st generation.
Hello,:Lif@forms! (perfection:  88%) Evolving       32nd generation.
He9lo, Lifeforms! (perfection:  94%) Evolving       33rd generation.
Hello, LifeformN! (perfection:  94%) Evolving       34th generation.
HelloV Lifeforms! (perfection:  94%) Evolving       35th generation.
Hello, LifefKrms! (perfection:  94%) Evolving       36th generation.
Evolved to 100% perfection. Concluding.
 
35,000 individuals sacrificed their lives
so that their species could eventually say:
 
        Hello, Lifeforms!

The Challenge

So… the other day, Steve drew my attention to this craigslist “to all those who think themselves a programmer” challenge. To cut a long story short, the advert included a programming challenge, and mentioned that a genetic algorithm that could say Hello World using the sacrificial efforts of billions of evolving entities would be one of the winners.

Now, it just so happens that I wrote a pretty fun Genetic Programming app as my first foray into linux C++ programming back in the mid 90’s, and had lost the code for it. So, last night, I rewrote a cut-down version of it in Python.

#!/usr/bin/env python2.6
#
# Copyright (c) 2009 Lee Braiden <lee.b@irukado.org>
# Released under the GNU General Public License, version 3.
# http://www.gnu.org/licenses/gpl.html for details.
#
 
import sys
from random import randint, sample
 
class Mirror:
	"""Stores certain details that individuals must analyse
	and evolve themselves against.  Named after the
	psychology concept of being a mirror for an individual's
	own thoughts."""
	def __init__(self, destiny):
		"""Takes a string, 'destiny', which creatures must
		evolve to say."""
		self.UPPER = max([ ord(x) for x in destiny ])
		self.LOWER = min([ ord(x) for x in destiny ])
		self.destiny = destiny
 
	def chromosome_len(self):
		"""Returns the number of chromosomes in an individual's
		genes.  In other words, this is the length of the message
		characters should evolve to say."""
		return len(self.destiny)
 
	def range(self):
		"""A quick (readability-)helper function, which allows looping
		over each gene."""
		return range(0, self.chromosome_len())
 
class Individual:
	"""Represents an individual beastie which will be created, can
	breed, can mutate, can attempt to solve the problem at hand,
	and can be tested for fitness, among other things."""
	def __init__(self, chromosome_len, mutation_rarity):
		self._genes =  " " * chromosome_len
		self.fitness = 0
		self.mutation_rarity = mutation_rarity
 
	def _gene_val(self, mirror):
		"""Create a random gene value"""
		return chr(randint(mirror.LOWER, mirror.UPPER))
 
	def _randomise_genes(self, mirror):
		"""Initialise all of the individual's genes randomly.  Used
		during creation, rather than breeding"""
		self._genes = "".join([ self._gene_val(mirror) for idx in mirror.range() ])
 
	def _choose_gene(self):
		"""Choose an individual gene to target."""
		return randint(0, len(self._genes)-1)
 
	def _freaky(self):
		"""Roll a dice, and see if this individual is (un)lucky enough
		to gain a mutation."""
		return randint(0, self.mutation_rarity) == 0
 
	@classmethod
	def conjure(cls, mirror, mutation_rarity):
		"""Create an individual from scratch."""
		i = cls(len(mirror.destiny), mutation_rarity)
		i._randomise_genes(mirror)
		return i
 
	@classmethod
	def breed(cls, mirror, mom, dad):
		"""Breed an individual from two parents."""
		mutation_rarity = float(mom.mutation_rarity + dad.mutation_rarity) / 2.0
		kid = cls(len(mirror.destiny), mutation_rarity)
		x = kid._choose_gene()
		kid._genes = "".join(mom._genes[:x]) + "".join(dad._genes[x:])
		if kid._freaky():
			kid._mutate(mirror)
		return kid
 
	def _mutate(self, mirror):
		"""Mutate a random gene"""
		orig_genes = self._genes
		idx = self._choose_gene()
		els = [ x for x in self._genes ]
		els[idx] = self._gene_val(mirror)
		self._genes = "".join(els)
 
	def self_evaluate(self, mirror):
		"""Evaluate 'fitness', or the ability to solve the given
		problem.  In other words, measure how close the individual
		is to the correct message text, and assign a score
		accordingly."""
		fitness = max_fitness = float(len(self._genes))
 
		for idx in range(0, len(self._genes)):
			if self._genes[idx] == mirror.destiny[idx]:
				continue
			else:
				fitness -= 1
 
		self.fitness = 100.0 * (fitness / max_fitness)
 
	def __cmp__(self, competitor):
		"""Comparison function, used for sorting populations to find
		the fittest."""
		if self.fitness > competitor.fitness:
			return 1
		elif self.fitness == competitor.fitness:
			return 0
		else:
			return -1
 
class World:
	"""Represents a world populated by Individuals, which breed
	together, subject to natural selection, to reach an evolutionary
	goal."""
	def __init__(self, mirror, population_size=1000, mutation_rarity=1000, fitness_required=100.0):
		"""Initialise a population of population_size members, mutating
		one in every mutation_rarity breeds.  Evolution will stop when
		fitness reaches fitness_required percent."""
		self._mirror = mirror
		self._population = [ Individual.conjure(mirror, mutation_rarity) for i in range(0, population_size) ]
		self.sacrifice_count = 0
		self._fitness_required = 100.0
		self._mutation_rarity = mutation_rarity
 
	def _natural_select(self, exclusion_list=[]):
		"""Select two individuals from a population, choosing the most
		'fit' for breeding."""
		a,b = sample(self._population, 2)
		if (a > b):
			return a
		else:
			return b
 
	def _breeding_pairs(self):
		"""Select enough breeding pairs from a population to create a
		new population of equal size, assuming one child per couple."""
		bp = []
		pop_size = len(self._population)
 
		for i in range(0, pop_size):
			mom = self._natural_select()
			dad = self._natural_select([mom])
			bp.append((mom,dad))
 
		return bp
 
	def _breed_new_generation(self, mirror):
		"""Create a new generation, breeding it from the older
		generation using natural selection."""
		new_gen = []
		bp = self._breeding_pairs()
		for mom,dad in bp:
			kid = Individual.breed(mirror, mom,dad)
			new_gen.append(kid)
 
		self.sacrifice_count += len(self._population)
		self._population = new_gen
 
	def _analyse_population(self):
		"""Evaluate the genetic fitness of the entire population."""
		for i in self._population:
			i.self_evaluate(self._mirror)
 
		self._population.sort()
		most_civilised = self._population[0]
 
		return most_civilised
 
	def evolve_civilisation(self):
		"""Evolve this civilisation until the required fitness_level
		is reached."""
		generation_count = 1
		last_str_len = 0
		while(generation_count):
			most_civilised = self._analyse_population()
			gen_ord = ordinal(generation_count + 1)
 
			if most_civilised.fitness >= self._fitness_required:
				print "Evolved to %3.0f%% perfection. Concluding." % most_civilised.fitness
				return most_civilised
			else:
				print "%s (perfection: %3.0f%%) Evolving % 10s generation." % (most_civilised._genes, most_civilised.fitness, gen_ord)
				self._breed_new_generation(self._mirror)
 
			generation_count += 1
 
# a couple of helper functions #############
 
def ordinal(n):
	"""borrowed from John Machin's python-list post.  Appends an
	ordinal suffix to a number.  For example, 1 becomes 1st,
	2 becomes 2nd, etc."""
	if 10 <= n % 100 < 20:
		return str(n) + 'th'
	else:
		return  str(n) + {1 : 'st', 2 : 'nd', 3 : 'rd'}.get(n % 10, "th")
 
 
def PrintLargeNumber(n,width=2,delim=',',decimal='.'):
	"""Converts a float to a string with appropriately placed commas.
 
	Floats will be shown with 'width' digits right of the decimal.
	'delim' specifies the thousands delimiter.
	'decimal' specifies the decimal character.
 
	Copyright 2007 Regents of the University of California
	Written by David Isaacson at the University of California,
	Davis BSD License"""
	if width >= 0: s = "%.*f" %(width,n)
	else: s = str(n)
	dec = s.find(decimal)
	if dec == -1: dec = len(s)
	threes = int((dec-1)/3) # we don't need a comma at the start
	for i in xrange(threes):
		loc = dec-3*(i+1)
		s = s[:loc] + delim + s[loc:]
	return s
 
############################################
 
def evolve_from_nothingness(msg):
	"""Evolve an individual capable of saying the given msg."""
	world = World(Mirror(msg))
	greeter = world.evolve_civilisation()
 
	large_num_str = PrintLargeNumber(world.sacrifice_count, width=0)
 
	print "\n%s individuals sacrificed their lives\nso that their species could eventually say:\n\n\t%s\n\n" % (large_num_str, greeter._genes)
 
 
if __name__ == "__main__":
	evolve_from_nothingness("Hello, Lifeforms!")
 
	sys.stdout.write("Press return: ")
	sys.stdout.flush()
	sys.stdin.readline()

All in all, it’s pretty sweet. It evolves fairly constantly, straight to the answer, in less than 40 generations. Completes in a few seconds as well.

I’ve no doubt the SF Bay guys would find problems with this. They’re into pointlessly obfuscated code, FP, and lambdas, for instance. I’m into pointedly unobfuscated code, and using FP only when it makes things much clearer. And optimisation is for compilers long after your algorithms and portability have done their thing. That said, most of this IS off the top of my head, and I’m sure there are better ways to do it, so let me know if there are obvious (or not so obvious) improvements that can be made.

Anyway… for me, this is what programming’s all about. In what other fields can you create worlds, create lifeforms, prove that evolution works, test yourself, frustrate yourself, and enjoy seeing your project finally work, all in the space of a few hours?

I should probably thank the python guys for a lot of that rapid development; the C++ version took much longer. Then again, it had a virtual CPU and datatyped I/O etc. I think it was threaded, too, if I remember correctly. It also took hours to work out that the way to get 4 out of two 2s was to add them. I might try that again someday, with a tweaked CPU/instruction set/program model.

All your base are belong to us

2009-07-07T16:45:07+00:00

We were all amused by the bad translation in the “All your base” meme some years ago. I was thinking about it recently (for other reasons) and wondered how Google translate would handle the same Japanese source material: 「君達の基地は、全てCATSがいただいた」

Google said: “You are our base, all you CATS”. I contributed a better translation.

I thought the original phrase would be entertaining because it contains a Japanese verb that Google mistranslates brilliantly: いただく。

「いただく」 means “receive” or “accept”. But it’s a humble verb, so the receiver has a lower status than the giver and so receives with gratitude. I suspect that Oliver Twist would use a form of this verb when asking for more gruel.

If you were the receiver and used this verb you would most likely use it in the form of 「いただきます」(the same verb with the normal polite ending). And this is where Google translate gets really confused. Because, unless you frequently get gifts from important people or work in a shop, the most common situation where you would use this word is at the start of a meal: you’ve just been given some food, and you’re thankful, so you say “I (gratefully) accept (this food)”. It’s slightly idiomatic, although the meaning is clear. It’s a bit like “saying grace”. But a popular idiomatic expression used in that situation by many English speakers is (the French phrase) “Bon appétit”. Therefore Google always (as far as I can tell) translates 「いただきます」 as “Bon appétit”.

That really doesn’t work well if the main verb in your sentence is “receive”. One example, appropriate at this time of day, would be 「お休みをいただきます」, for which I would say “thankfully I’ll get some rest”. Google says “Rest for a bon appétit”.

Mono in Debian

2009-06-29T12:20:32+00:00

I note that the controversy of Mono in Debian which reached fever pitch with the inclusion of Tomboy as part of the default Gnome applications rages on, and now RMS has entered the fray.

I respect Stallman a lot. I'm a fellow of FSFE. His positions are usually well thought and argued, though I can't say I agree with him on all of them, and I don't agree with this one.

On the issue of the plain usefulness of Mono apps; well some of them are just excellent. I use f-spot for managing my photo collection and have to say no other application I've used comes close. Tomboy is itself a considerable improvement on the almost useless sticky notes applet that's been in Gnome for some time; though I've yet to work out what minimum I need for ssh synchronization (to be fair, I've hardly tried).

And I should declare an interest. I thought Gnome-Do was absolutely excellent when I stumbled upon it, and I helped in a modest way to initially package it for Debian (specifically I updated the Ubuntu stuff for the plugins package and prepared that for its initial import to the archive), albeit I haven't worked on it since. It's fairly clear to me that implementing this in, for example, C++ looks like it could be a lot more work. I can't say I've studied it in total depth, but that was my feeling when working with the code.

Anyway, never mind all the issues of sheer practicality; Jo Shields of the Debian Mono team, wrote an excellent rebuttal to some of the nonsense that was posted about Mono. It makes excellent reading. It is calmly delivered (with an undertone of quite justified and controlled anger), carefully argued and a cautionary tale about some of the nonsense we in the Free Software community can get wrapped up in, it's value extends well beyond the current debacle.

Swap my memory!

2009-06-23T16:26:58+00:00

So its been a while since i have updated, while i was offline i guess, i have been moving to a new server! I have had a few teething problems but i think its all going fine now!

I decided for a VPS off: http://www.vpsville.ca/index

I got the CITY plan Its very powerful because its running in some kind of container VPS, i get to see the full ‘$ cat /proc/cpuinfo’. And it 8 core’s of the new Intel core i7! But i only have 256mb ram which was the teething problem so i turned off innodb in mysql and limited php memory usage to help compensate. It was nice Debian Server has all the mysql.conf and php.conf files all documenated very well so you generaly have an idea what your changing.

This is a very useful link for mysql tunning: http://howtoforge.com/tuning-mysql-performance-with-mysqltuner

——– General Statistics ————————————————–
[--] Skipped version check for MySQLTuner script
[OK] Currently running supported MySQL version 5.0.75-0ubuntu10.2
[OK] Operating on 32-bit architecture with less than 2GB RAM

——– Storage Engine Statistics ——————————————-
[--] Status: +Archive -BDB -Federated -InnoDB -ISAM -NDBCluster
[--] Data in MyISAM tables: 940K (Tables: 32)
[--] Data in MEMORY tables: 0B (Tables: 1)
[!!] Total fragmented tables: 2

——– Performance Metrics ————————————————-
[--] Up for: 2h 45m 48s (6K q [0.612 qps], 214 conn, TX: 11M, RX: 1M)
[--] Reads / Writes: 74% / 26%
[--] Total buffers: 58.0M global + 2.6M per thread (100 max threads)
[!!] Maximum possible memory usage: 320.5M (125% of installed RAM)
[OK] Slow queries: 0% (0/6K)
[OK] Highest usage of available connections: 4% (4/100)
[OK] Key buffer size / total MyISAM indexes: 16.0M/331.0K
[OK] Key buffer hit rate: 99.7% (29K cached / 87 reads)
[OK] Query cache efficiency: 58.5% (2K cached / 4K selects)
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 291 sorts)
[!!] Temporary tables created on disk: 30% (103 on disk / 339 total)
[OK] Thread cache hit rate: 98% (4 created / 214 connections)
[OK] Table cache hit rate: 89% (50 open / 56 opened)
[OK] Open file limit used: 9% (98/1K)
[OK] Table locks acquired immediately: 100% (3K immediate / 3K locks)

——– Recommendations —————————————————–
General recommendations:
Run OPTIMIZE TABLE to defragment tables for better performance
MySQL started within last 24 hours – recommendations may be inaccurate
Reduce your overall MySQL memory footprint for system stability
Enable the slow query log to troubleshoot bad queries
When making adjustments, make tmp_table_size/max_heap_table_size equal
Reduce your SELECT DISTINCT queries without LIMIT clauses
Variables to adjust:
*** MySQL’s maximum memory usage is dangerously high ***
*** Add RAM before increasing MySQL buffer variables ***
tmp_table_size (> 32M)
max_heap_table_size (> 16M)

That’s the kind of output i got in my first go so i start going though each of these recommendations: This link shows you each of the different variables you can change: http://dev.mysql.com/doc/refman/5.1/en/innodb-configuration.html, I think you can change a lot of Apache memory variables but i haven’t looked into this yet!

But i also decided to make a swap file, this is very easy in Linux if your system is all running directly off a rootfs you can:

$ dd if=/dev/zero of=/myswapfile bs=1024 count=1048576

#this should make you a 1G file which we can use for a Swap file!

$ mkswap /myswapfile

$ swapon /myswapfile

#this should work but in my VPS i keep getting ’swapon: swapfile: Operation not permitted’

#you can also add this to your fstab like so:

$ echo “/myswapfile none swap sw 0 0″ >> /etc/fstab

So yeah that’s quite useful to know! I have to say i have upgraded my wordpress to 2.8 and very unhappy to be honest, there are a lot of little bits and pieces annoying me 2.7.1 was a very good release! But this 2.8 is very memory hungry, but from what i read is that they have introduced a new API for widgets, plugins and themes. So i guess there will be problems for a while!

Anyways what i am hosting on my VPS is 4 sites:

http://redbrain.co.uk

http://brendanmullan.com

http://team-qcg.com

http://animeequation.com

They are the websites so i run: mySQL, Exim mail server(because noodles recommended it to me), Apache (might test light httpd), PHP plus various php plugins, git (using those python-tools): http://scie.nti.st/2007/11/14/hosting-git-repositories-the-easy-and-secure-way

I have to say anyone who works with me will know i hate SVN and CVS. Mainly because they are just retarded and your locked in too much and if the server goes down, then what the hell are you meant to do. My main reason for sticking with Linux was the fact now your no longer locked into software. A lot of people i know in the windows world use windows but use ‘cracked’ copies of it, as-well as photoshop, games, office etc… all downloaded etc. Using free software alternatives means your no longer reliant on these kinds of software and you know whats available and you learn alot more about your computer and why!

So anyways i thought i would post a screenshot of my current desktop i think it looks pretty nice tbh! I was using Xmonad for ages there, its a great window manager written in haskell its great because i run a server on my work desktop! And then i still need a way to run firefox, email and emacs. So this was very useful, and its very lightweight and usable. Other tiling window managers are light but very complicated!

my laptop

So i am also giving Haskell another go, i got the book ‘Real World Haskell’, its actually pretty good there is a helluva lot of learn, but its going to take a while before i’ll be seriously able to say i can write a mail client in haskell rofl! I would seriously like to make a GHS right now a gcc implementation of Haskell, because i have started the gcc-front-end documentation with a friend.

http://gcc.gnu.org/wiki/FrontEnd/skeleton

Problem is were kind of stuck right now, i posted most of it all so far and put up the code, etc but there have been quite a few changes in the latest gcc-4.5-snapshots so trying to keep up with them. But the real problem is i posted the code etc, but posting source code on a wiki page… its not that nice! So this is another point where the gcc-developers on oftc.net have said should really get my FSF-copyright assignment document stuff done, because i have my automake work and now gcc, and they would like to see official documentation in the GCC-manual in the next releases i think this would really help GCC.

If you working in industry like me, you’ll find the only real thing keeping compilers/interpreters and programming languages alive in large business driven companies is the idea of parallelism, this is quite old right now but its only really getting together now! I also have a book on OPEN/MP and know some of the basics of it in C/C++. Its very impressive the implementation in GCC and the Glibc. There is only a little theory to learn about OPENMP because actualy coding with it, is very simple! But you wont understand the code until you learn a little about shared memory systems and threading on this! Because OPENMP at the end of the day is starting threads for faster execution behind the scene. But it does all the work for accessing and manipulating memory without breaking your code for you! So in that sense its awesome

So yeah i also joined http://wakoopa.com/redbrain Its quite an interesting idea! And also pointing out that: http://outlawpla.net/ This is a Linux outlaws Planet, it should be cools i am hoping to get my blog on it as Dan and Fab pointed out in the show yesterday!

Anyways spent most of my time reading tbh recently so i havent been coding much, re-writing my OS from-scratch again i am starting to understand the i386 kind of now, and i want to structure my kernel correctly and make sure its as similar to just coding with libc as much as possible so i dont implement a stupidly named API of mad functions that only i understand! But i have done bits and pieces of cleaning my simple-programming language. Need to finish that tbh, there isn’t much more work for me to do to demonstrate the basics for it all. But been doing reading and writing takes alot of time when i work around my Job which takes up most of my time and i haven’t had a chance to get a break to do serious work at work, been documenating and transferring knowledge of work systems on Unix to new guys which is taking some time as they dont have a background in Linux or UNIX so its kind of frustrating to say the least.

Anyways i better head on looking forward to the BLUG meet on Wednesday! See you there guys!

no-space

Going to DebConf 9

2009-06-22T20:35:47+00:00

I've known I was going for a while, but only finally booked the holiday off work and had it approved last week (unfortunately they're not giving me the time for free like last year. :( ). I'll be there for DebConf proper (ie 23rd until 31st). In the unlikely event anyone else is flying DUB<>MAD my flights are:

2009-07-23 06:00 (DUB) -> 09:30 (MAD) FR 7158
2009-07-31 20:15 (MAD) -> 21:50 (DUB) EI 595

I'm on the 16:25 Talgo from Madrid and then the 09:25 back on the 31st. Looks like I'll have company from the train booking page.

Also I've been doing the first draft of the room allocation. If you're expecting to stay at DebConf organised accommodation you should check your name is on this list with the expected dates, and email rooms@debconf.org if it's not.

Roberts WM201 Internet Radio

2009-06-13T23:23:39+00:00

I recently bought a Roberts WM201 Internet Radio. I've used it for a while now and thought I'd post my thoughts.

First of all, I was looking for a radio meeting certain criteria, it needed wifi, I wanted it to pack a reasonable punch since it would essentially be my main music source, it needed to support upnp media servers. I also wanted it to have an integrated transformer so that it would not have a bulky mains lead since I wanted it for my rather small kitchen. Finally I wanted it to be semi portable, so I could move it from room to room without too much fuss.

The WM201 meets all these criteria, and is based on the Reciva technology that has been well received by a few of my friends, notably Paddy and Noodles. The radio is a pretty good size, not too large and not too small, and feels really solid. It has a wired network port as well as wireless capability which is great. For complex reasons, when it first arrived I had no internet connection (no gasps, I was making do with 3G hookups). That being the case I knew I wouldn't be able to get the internet radio functionality itself working. But I figured I'd set up a local lan and get the mediatomb server on my development machine working. I was able to hook up to the LAN and enter the WPA password, but it just would not play with any functionality whatsoever if it doesn't see the servers it expects to. Now Noodles has suggested my geek privileges should be revoked for not working out how much of the internet I had to fake to get it to work. He may well have a point! In my defense I had plenty of other issues to deal with instead.

About a week later I got my net connection, albeit temporarily since some work was needed on the cable. So while the network was up I was finally able to get into all the functionality, and I was really impressed, the small display and control is really intuitive and the shipped remote control is excellent. When it came time for the external network connection to be severed again, I quickly switched the device over to streaming media from a playlist on mediatomb, but interestingly it still gave that up midway when the external connection went down. Dumb, but forgivable.

So again, what's good? It offers brilliant sound, and more volume than I could wish for. It's easy to browse through the huge array of stations, and for things like the BBC stations, it has a good interface to the "listen again" service. It works seamlessly with mediatomb on my PC. All excellent. A minor grumble is that it's not easy to switch briefly from a radio station to the playlist on mediatomb and back again, you have to go through all the menus every time. The number of stations is so huge, finding them can be a little slow, but you can as you would expect, save them to a preferred list. I hit a problem with that; my saved BBC stations have spontaneously stopped working, just showing endless retrying messages. When I go back through the menus it's all fine. Odd.

The radio becomes better yet when you use the Reciva portal to set up your "stuff", a list of your preferred stations and podcasts. Obviously it's much easier to do this on a web page, and then you simply register your radio. Now (it seemed to require a hard power cycle for me) the radio has an extra "My Stuff" menu which gives really easy access to your favourite stations and allows you to quickly select podcasts, far faster than navigating on the radio. An odd note, if you for example navigate through "listen again" in the normal way, you can fast forward, pause and rewind the playing media. But if the same stream is selected from the podcast menu in "My Stuff", you can't. A slight annoyance.

I'd really recommend the device overall, it's great. Incidentally the cheapest prices I could find were on Amazon by some distance, but time and time again, I would select a seller and only at the final hurdle be told they wouldn't ship to Northern Ireland. I've complained about this before, it would be useful to know rather sooner that I'm wasting my time. Anyway, I found a simple way to work out which sellers ship to Northern Ireland and find the cheapest of those. I selected one from each and every seller on Amazon. Then when I went to checkout I removed all those that caused complaints. It was then easy to find the cheapest remaining seller. Much faster than trying them one at a time.

Self Defence and Over Confidence

2009-06-13T14:49:19+00:00

Some years ago, I saw an article on the local news showing some girls being schooled in martial arts, Karate in this case, in order to defend themselves from disreputable men (no jokes about tautologies please). It was interesting to note that many felt after about an hour that they were suddenly capable of fending off any male attacker. In particular, and I've heard this view expressed by many female non martial artists, it was put forward that a swift kick between the legs would solve all problems.

Now men have exercised undue dominance in world affairs for many thousands of years, and certainly this is true in the military arena. For all that time the basic design has been the same regarding external placement of genitals, and yet the "swift kick between the legs" did not bring about a fundamental change in gender balance. In other words, it's often not just as simple to obtain victory by that means. If you're a woman, you shouldn't count on that being your winning strategy. If it were really that simple there would be much less tragic history of men's ill treatment of women.

It was interesting then to read about a current scheme in India for training sex workers to defend themselves with Karate. I'm not against any of these schemes, and at least this seems to spend some substantial quantity of time in a training programme. Still, from the article we see:

"I think I can easily handle one man at any given time. I face physical abuse on a daily basis and have been abused and been beaten up by my clients many times."

"A thug once stripped me of my clothes and told me to run naked. From now on, I think nobody can do that to me. I will kick him."

Right. Where I have concerns is that, to quote the great Harry Callahan: "A man's got to know his limitations", and nowadays that should probably be amended to "person". I have spent several thousand hours training in dojos with all sorts of folks of all backgrounds, shapes, sizes and available genders. Over this time I've trained with people who've studied many different martial arts. I would never make such a sweeping statement, and I'm a six foot tall male (ape descendant). You never know what the other guy/girl knows, has as a weapon, and so on. Complacency and overconfidence are dangerous. Self esteem is great, but believing that you just have to kick an assailant once to resolve the issue may well be wishful thinking.

In the Book of Five Rings (Go Rin No Sho) the legendary Miyamoto Musashi warns the reader that when a person has no training, they at least act naturally, without hesitation, for good or for ill. Once you set down the pathway of a fighting art, your goal is to learn new reflexes that become your automatic reactions, but initially, the result can be quite negative, since conscious thought is required, and can cause the novice to "freeze" under pressure. Dabbling in martial arts may be much less effective that people believe.