We have a nice Swan Heated Tray courtesy of my Mum. It's useful for lots of things, heating plates for dinner and then placing serving dishes on it, or for keeping a stack of pancakes warm on Sunday. Unfortunately it stopped working recently. The red power LED still lit when power was applied but no heating. Google produced no answers (which is why I'm writing this for anyone following a similar trail). The helpline couldn't help, and theoretically the tray was under warranty, but with no receipt we couldn't follow their advice to return it to the store. So I had to fix it myself.

You should obviously think twice before messing around with something (a) electrical and (b) which generates large amounts of heat. Please don't kill yourself or burn your house down, that will make us both feel really bad.

The tray is fitted with triangular screws which reinforces my comments above, but not having previously purchased some triangular screwdrivers, these were next to be acquired. I figured they might come in useful for something else in the future.

Taking the device apart shows that basically it's quite simple, there are some blocks through which elements do the heating, a lot of glass fibre (so wear gloves) to protect the underneath from the blocks.

There are also a number of polythene covers to hold the corners of the blocks, though it seems a few were missing. There was no obvious fuse much to my surprise and irritation. A bit of testing with a continuity tester showed that a particular loop of cable was no longer doing its job. I pulled it out, and pulled back some insulated sheathing to reveal the culprit fuse.

Replacement fuses can be obtained here. I teased upon the crimps with a precision screw driver and fitted the replacement, crimping it very firmly back in place. Then it was a matter of putting it all back together and testing it was appropriate safety measures in case of problems. All working again.

For my day job I build storage systems. A lot of what I do at present involves caring a lot about how different OSes deal with things like new LUNs being presented from a SCSI target, or errors along a subset of the available paths to a device.

It will come as no surprise to you to discover that they all suck (for values of all equal to Linux, Solaris, Windows and VMWare). New LUNs are particularly annoying. I'm in the situation that creation and removal of a LUN is exceptionally easy.

Hmmm. Maybe I need to back up here a bit first. SCSI has the concept of a target (think, device, eg hard drive). Each target can present multiple logical units. Each of these is assigned a number - a Logical Unit Number. Most devices (a hard drive, or a CDROM drive) will present a single LUN. A storage array will tend to present multiple LUNs; one for each volume that is exported to the host. At the host level each LUN really just looks like a separate device (for Linux /dev/sda and /dev/sdb may well be separate LUNs on the same array, rather than 2 separate arrays/hard drives, for example. At the block device level you don't care about the difference usually).

Anyway. For various reasons I end up adding and removing LUNs quite often. And there are ways for the array to indicate that this has happened to the host (the UNIT ATTENTION/REPORT LUNS DATA CHANGED check condition seems to be favoured these days, as a complete Fibre Channel LIP can be disruptive). What I'd like to happen in that case is the host to pick up the check condition and drop and/or add the devices that have changed. Instead everything wants a manual rescan. rescan-scsi-bus tends to be simplest for Linux. Windows wants a manual refresh in Disk Administrator. VMWare a "Rescan HBAs" from vSphere. Solaris a "devfsadm -C" and possibly a "cfgadm -al" first. And all of these can be temperamental about picking up the changes.

We've done a lot about hotplug for the desktop user experience, without doing the same level for the server experience. I appreciate that there are situations that you don't want your server to reconfigure things without being told to, but the current situation can be detrimental (for example Linux multipathing will hold a device open even after it's disappeared and is returning an "INVALID LUN" response; it would be much better if it could cleanly close that device and wait for it to return). Storage is capable of being much more than just a single block device these days, and it's a shame that nothing seems to deal fully with that fact.

(Yes, yes, I should write and submit patches, but I appreciate that there's not always a simple answer, nor necessarily an answer that works for all situations automatically. Plus, y'know, not enough hours in the day and I hope you all appreciate I've taken a break from watching BSG to write this.)

One of our services has been around for a while, a realy long time.  It used to get developed in production, there is an awful lot of work involved in making the app self-contained, to where it could be brought up in a VM and run without access to production or some kinds of fake supporting environment.  There’s lots of stuff hard coded in the app (like database server names/ip etc), and indeed, and there’s a lot of code designed to handle inaccessible database servers in some kind of graceful manor.

We’ve been taking bite sized chunks of all of this over the last few years, we’re on the home straight.

One of the handy tricks we used to get this application to be better self-contained was avoid changing all of the database access layer (hint, there isn’t one) and just use iptables to redirect requests to production database servers to either local empty database schema on the VM, or shared database servers with realistic amounts of data.

We manage our database pools (master-dbs.example.com, slave-dbs.example.com, other-dataset.example.com etc) using DNS (PowerDNS with MySQL back end), in production, if you make a DNS request for master-dbs.example.com, you will get 3+ IPs back, one of which will be in your datacentre, the others will be other datacentres, the app has logic for selecting the local DB first, and using an offsite DB if there is some kind of connection issue.  We also mark databases as offline by prepending the relevant record in MySQL with OUTOF, so that a request for master-dbs.example.com will return only 2 IPs, and a DNS request for OUTOFmaster-dbs.example.com will return any DB servers marked out of service.

Why am I telling you all of this?  Well, it’s just not very straight forward for us to update a single config file and have the entire app start using a different database server. Fear not, our production databases aren’t actually accessible from the dev environments.

But what we can do is easily identify the IP:PORT combinations that an application server will try and connect to.  And once we know that it’s pretty trivial to generate a set of iptables statements that will quietly divert that traffic elsewhere.

Here’s a little ruby that generates some iptables statements to divert access to remote, production, databases to local ports, where you can either use ssh port-forwarding to forward on to a shared set of development databases, or to several local empty-schema MySQL instances:

require “rubygems”
require ‘socket’

# map FQDNs to local ports
fqdn_port = Hash.new
fqdn_port["master-dbs.example.com"] = 3311
fqdn_port["slave-dbs.example.com"] = 3312
fqdn_port["other-dataset.example.com"] = 3314

fqdn_port.each do |fqdn, port|
puts “#”
puts “# #{fqdn}”
# addressess for this FQDN
fqdn_addr = Array.new

# get the addresses for the FQDN
addr = TCPSocket.gethostbyname(fqdn)
addr[3, addr.length].each { |ip| fqdn_addr << ip }

addr = TCPSocket.gethostbyname(‘OUTOF’ + fqdn)
addr[3, addr.length].each { |ip| fqdn_addr << ip }

fqdn_addr.each do |ip|
puts “iptables -t nat -A OUTPUT -p tcp -d #{ip} –dport 3306 -j DNAT –to 127.0.0.1:#{fqdn_port[fqdn]}”
end
end

And yes, this only generates the statements, just pipe the output into bash if you want the commands actually run.  Want to see what it’s going to do?  Just run it.  Simples.

I think I'm currently supposed to be out buying a TV at the moment. Or something else expensive. Instead I'm lying on the sofa listening to Whale and drinking tea. That's much better in my opinion, but it reminded me that I hadn't mentioned that I bought a projector.

I had the loan of a projector for a while when I was in Belfast. It was a huge thing that made a lot of noise, but was pretty cool for watching films with. When I moved to the US I decided not to bother with a TV to start with - using my laptop did just fine for most things. Except films. They really benefit from a bigger screen. Especially if you want to watch them with someone else. So I started looking at pico projectors, because I wanted something small and cute that I could throw in a bag with my tiny laptop. At the time the best the pico projectors could do was 840x480, which I felt wasn't really that great. However back in January TI announced their DLP Pico HD chipset, offering a resolution 1280x800.

I kept a look out for projectors using this to appear, and eventually, in July, Amazon claimed to have availability of the Vivtek Qumi. So I bought one. And I'm very happy with it. I'm not a heavy user, and there are some niggles, but it's small and does exactly what I want. In low light conditions it'll happily throw an image all the way across the room, which is more than it's rated at. There's a mini-HDMI connector on the back, so it's a doddle to plug it into my laptop using the supplied HDMI to mini-HDMI cable. The laptop auto-detects the device and extends the desktop appropriate, as you'd expect.

It'll also do media playback itself - there's a USB host port and a micro SD slot on the back. This works ok, and the included remote means you can easily set the projector somewhere above your head and still be able to easily control it. Unfortunately the built in speakers are fairly useless. There's a 3.5mm socket for external speakers, but having to plug something else in detracts from the convenience factor of the built-in media player. Also my unit had the power switch installed upside down (the little red line indicating the power is on actually shows up when it's off), but that's the only complaint I've got about the build quality. There's a little neoprene case to store the thing in as well.

So, er, yeah. I think I can continue to make do without a normal TV for a while longer and avoid the nightmare that I suspect are the shops on Black Friday.

So the other day, was exciting something i will always remember; becoming a detective. What you say, yes well i will try to keep this mostly anonymous as i think that’s more appropriate due to the circumstances. Well a close friend of mine created a website for buying and selling cars in a foreign country which his brother lives in. So all going well website is fine bla bla, I just helped my friend system admin for the server maintaining apache, mysql, postfix etc.. the website was fairly busy.

But there was fake accounts created over time posting fake ads selling cars but most of which in-fact all were very obvious scams but one of which ads was created 5 times on this website from 5 different users with 5 different emails but each listing was exactly the same. Long and short of it one person fell for this listing and ended up wiring ~£8000 to this scammer. So the victim contacted the local police which contacted us to try and find as much details as possible on the scammer like emails ip’s etc.

As the scammer was trying to get more money out of the victim the police suggested to play along with the scammer so we can catch him in the act. But my job was to find out as much information as you can. And by god the amount of information you can figure out about someone based of server logs is scary if your clever.

So we cross referenced the times which the scammer signed up to the website each time, to get his ip and his email and checking the post data to be 100% sure we had the right ip. There were 5 instances we had to go through. 3 of which ips would origionate from exactly the same place which was a major city and isp the other 2 were in a more local rural town. And if you take more attention to the dates the first 2 sign-ups by this scammer originated from this city and occurred around June then nothing until October or so where the next 3 sign-ups occurred so this could give rise to the idea that this person moved house. We know its the same scammer each time when the listing is exactly the same each time and the trace on the ip go to the same places each time.

So yes this was an interesting day but very very interesting what you can figure out about people from their IP.

 

You should probably ignore this post. I'm just venting. I'll be better after a nice cup of tea.

Things that are causing me to fume about the fact Gnome Shell just hit Debian/Testing:

• Spacefun came back. Even on GDM3. Just fucking die already.
• I had to reboot to get bluetooth working again.
• /desktop/gnome/shell/windows/workspaces_only_on_primary set by default? SRSLY?
• It failed to carry over my previous monitor settings.
• I'm pretty sure I don't have a latent desire for a machine running OS X, whatever the GNOME devs might think.
• How many mouse clicks to get to the list of applications?
• Er, why have you changed my default apps? (Why is clicking a link in a terminal opening Epiphany rather than a new tab in Iceweasel?)

I update my testing boxes (work + home laptops) almost every day. It rarely breaks, and certainly when it does I accept that's what I get for doing rolling upgrades. I can't remember the last time I did an upgrade that actually made me angry.

Also I suspect this thing is going to have a complete fit on my binary nVidia/hacked up DisplayLink configuration at work (the DisplayLink side refuses to do 3D for starters). Perhaps better not to upgrade there until I have a sufficient block of free time.

Maybe it's time to go back to evilwm. I only stopped because I wanted a dock for wifi/bluetooth etc applets on my laptop that didn't get hidden when I fullscreened things. Implementing _NET_WM_STRUT might make that doable...

(I'm sure some of this is just dealing with the change but it's a bit bloody difficult to deal with a complete change in user interface that hasn't even managed to carry across settings from the old one.)

So i plan to try and do an update of me every 2 weeks on my blog to get more into routine in my life. For the last few years i realized i was very depressed and frustrated, from university i would go in and do my mathematics side of my degree and go to my Computer Science classes and hope to learn from these to understand what the updates in my debian system meant like changes to libc for security reasons then once 2nd year hit, it became very obvious that university wasn’t going to teach me the skills i wanted or hoped. So i took things into my own hands and threw my self into ~4 years of solid learning on my own and university kind of took a back step.

I have no regrets in the choices that i made,  doing what i have done has opened so many more doors for me in my life than anything else has in my life, including university. These days so many students go to university especially science related topics and dont get a related job from it in the end. I can say for me i know i am good at programming, i know i love it; i know i want to do this for the rest of my life as a career  not many people at my age can say anything like that about themselves. For me i never wanted to blame university to not get where i wanted to be in life. If it wasn’t for me taking so much time to learn on my own i wouldn’t have become a decent GCC hacker with server access etc. Mentored by Ian Taylor a well know free software hacker. I took my until june this year to break a lot of depression i had in my life and to accept my failing at university i felt like i had let down my whole family so much when i failed a module, it was when i tried the absolute hardest i possibly could at an applied math module at university and it didn’t pay off and there didn’t seem to be anyone to help you or talk to about it. That was in my 2nd year and that was when i really pushed my self into open source and free software.

It feels good to just talk about it and it doesnt make me feel sad anymore, i am back at university this year i should have been finished now but i failed all my modules last year it really took the biscuit if i am honest, i realised even more so that there is no chance students can learn anything useful from university, well at least from my university if you compare what others do. But yeah the long and short of it is i only need to pass 4 modules now this year to finish up my degree and i will be finished. Its hard for me but i will do it there is so much i want to do this year so i am pushing my self into alot of work to keep my mind active and my self in a routine.

 

It scares me how much power your mind has over your body, when getting over all my depression i feel so much better in every day life. Its like i can see again but anyways enough of me blabing on i have lots of gccpy updates to post so i will do that over the coming weeks.

I use the Debian operating system on several computers. My "main" computer (Imladris) runs Debian unstable (Sid) while the others mostly run on testing. I've been anticipating the Gnome 3 upgrade for some time, mainly because of the switch to Gnome Shell which is a completely new way of using the desktop. I had played with Gnome Shell a while ago, and was kind of impressed and worried by it in equal measure, I decided it wasn't ready for prime time so stopped using it. Naturally I assumed it would be much more impressive upon release; especially since Debian is not (by far) the first GNU/Linux distribution to include Shell.

A while ago a big upgrade came through on imladris, and it was clear it was the Gnome 3 upgrade. I share this computer with three other users, two of which are children for whom I have implemented password less login (locally only). I can only say I think Gnome have significantly mishandled the upgrade. Here are some reasons why.

Login is seriously slow

The display manager can take up to a whole minute to display the list of users (and often doesn't display the icons). There are some bug reports about a possible race condition that causes this, but seriously on a reasonable spec computer this is unacceptably slow. The same problems occurs when switching user.

I couldn't login

My, admittedly old user account simply wouldn't launch a working desktop. I had to (at a command prompt) delete configuration directories to get my account working again.

Absolutely zero support for the user in transition

So the average user does the upgrade and suddenly their entire desktop has changed. But when they first login there will be some guidance about where everything is gone... right? No. Having already used Shell, I knew, but I had to try and show everyone else how to use the machine again. It's not that spectacularly intuitive.

Actually, a lot of functions have just gone

There's a huge removal of existing functionality. All your carefully tweaked panels: gone. All your applets: gone. And bizarrely often with no working alternative.

Not friendly for children

It was possible to set up a Gnome 2 account to make it easy for kids. Low res graphics, and big panels with big select icons. The new paradigm completely ignores all that in favour of a sleek minimalist environment which is probably not that easy for young children to understand.

Dictatorial design choices

It's been decided that we don't need minimise buttons or maximise buttons. It's been decided not to honour old desktop backgrounds. It's been decided not to honour existing resolution settings. It's been decided not to show anything on the Desktop (much to the confusion of many users). It's been decided we can't right click on the desktop.

Some of this kind of nonsense is exactly why I don't like some other operating systems who believe they know what's best for you with Messianic Zeal (I'm looking at you Apple).

All in all I find this transition very disappointing. There are lots of basic things no-one seems to have thought of, and years of desktop customisation have been swept away with an extraordinary arrogance. Don't get me wrong, I support the idea of trying a new Desktop paradigm: but, for instance, if people used to have applets on their desktop for the weather, or for system monitoring, it's because they needed it. Rolling out a new desktop that simply ignores these things in favour of how some people thing everyone should use their desktop is exasperating.

I'm seriously hoping that Gnome Shell improves significantly and fast. I won't hold my breath.

I’m siting on my flight home from my first devopsdays in Goteborg, so firstly, many thanks to the awesome Patrick Debois, Ulf & many many others that put the effort in to organising the conference, and everybody that turned up and made the event so worth while! My primary reason for going was to hear other people’s experience with configuration management and general ops deployment experience. (I’m in the process of adding puppet to our large legacy LAMP stack)

I kind of expected to be the fuddy duddy in the room (my group runs 4 SaaS services, our largest is a LAMP+JBoss SIP stack, a Solaris/Tomcat/Oracle/Coherence stack, a Linux/Tomcat/MySQL stack and a Apache/Weblogic/Cognos/Oracle stack, all hosted on our own hardware, how retro), so I was prepared to hear stories of how easy it was to deploy services built on modern interpreted stacks to the cloud, but I was pleasantly surprised to hear that plenty of people are using java application servers of all shapes & sizes in production. I was less pleased to hear, but somewhat comforted, that everybody running java stacks in production is suffering pain somewhere (damn, no silver bullet to take home).

 

Deployment Pain

Lots of people were good enough to share their success & horror stories about how their current java stacks get into production, some of the recurring topics:

Orchestration

I think this deserved a talk or open space on it’s own, but John E. Vincent covered chunks of this in his great tools talk, and it came up in the “deploying java artifacts” open space.

I’ve got some take away reading to do about tools like Apache Whirr & UrbanCode’s deployment & configuration tools, but everybody has similar problems, needing a controlled, reliable method of automating the pre & post deployment steps (traffic bleed off, deploy, service verification, data load, back in service) and managing the service availability during the deployment (or managing the stress on systems affected by the post deployment steps)

Hot/Cold deployments?

In general, hot deployments never seem to work as planned reliably, hot deployments are highly desirable for some services due to session requirements, but most people observed that hot deployments are prone to problems, leaking memory on many occasions, leading to hot deployments being something you can only get away with a few times, if at all depending on your memory overhead.
The guys from zeroturnaround demoed their latest jrebel/liverebel tools. JRebel is a developer focused tool that allows a jar to be hot updated in a running JaS, for quicker iterative java development. LiveRebel is built on similar technology but aimed at use in production, to do hot updates (I’m not sure how this differs from the hot deployments of war & ear etc, but that’s a gap in my JaS understanding)

war/ear or exploded webapps directory?

Currently we do both, and each have their pros & cons, exploded webapp directories have a tendency to build up undocumented cruft essential to the smooth running, and war/ear deploys have a tendency to break your heart with environment issues (what do you mean we need a new build to use a different database server!?)

For our next service going into production, we need to be able to vary the number of tomcats running on a physical host, each running on a different port, to support this we’ve extended our existing in_service hook, which in a our simpler environments just lets the load balancer know that this host is now good to take traffic, now it will build out the multiple tomcat CATALINA_HOME trees from scratch, going as far as grabbing the ant & tomcat tarballs required (version numbers pulled from central config db, allowing per host overrides for piloting versions on individual machines), the aim here is 2 fold, have a clearly documented process for building a working CATALINA_HOME and be able to dynamically vary our tomcat count without lots of manual preparation required)

Environment & Configuration

Lots of issues & lots of different solutions to this one, best case, war file ships with 3 environments configured, default to production, over ride on the command line for other environments (down side, production passwords are in the artefacts & therefore in source control).

There was some discussion over externalising the config, various methods (XML includes in comtext.xml/server.xml) and providing an standard-ish API to get/set properties (some commercial JaS already do this).

Horror stories included the deploy process having a start to explode the war, stop, remove the war, fix the config, restart. Another involved a post restart data load that took 30-40min before the tomcat was ready for traffic again.

General concencus was that involving dev in more of the ops deployment pain helped hilight areas that needed some improvement.

Config/Properties APIs

There was a little bit of discussion around APIs for managing configs in a running AS, some of the commercial AS already have this, but their was little support around the room for single vendor solutions, although most agreed that practically nobody changes AS after initial selection, the desire for a single tool was focused on having a single tool to gain momentum instead of fragmented tooling.

No Ops in the Java Serverlet Steering committee?

I missed the exact names of the standards & people involved, I’ll update this if you have specifics
One of the participants in the java artifacts open space is on the Java Community Process mailing list, he pointed out that their was practically no one representing ops in the ML, some of the proposed changes for horrified ops people in the room.

the platform/application split

Some ardent supporters of deploying nothing but packages, people using FPM and other tools to build RPMs and other packages of the AS and another package for application.

Part of this also comes down to your orchestration tools and how you run your CM, Ramon of Hyves hilighted that they don’t run puppet continuously, they run once a day, due to orchestration requirements and scale (they have 3,000+ application server in production).

Most people agree that the CM goes as far as preparing the AS for an application to be dropped in, although most of these environments ran a single AS per host.

I was due for another Google interview mail it seems. I have to say I wasn't expecting it, but this week I had a follow up to my polite mail from 6 months ago that said "No thanks, I'm not looking" asking if it was still the case.

Normally I welcome this little bit of ego stroking; it's always nice to be wanted. Except that's not really the case, is it? It's an invitation to interview for something, not any indication that you've done more than tick some initial boxes. Google mails inevitably ask me if I'd like to work in SRE. It's always SRE. No one ever emails and asks if I want to work on self-replicating nanobots that will roam Mars searching for the perfect spot to build a beach house.

And that's where things fall down. If someone currently has a job, then emailing them out of the blue to ask if they want to come and interview for something vague is hoping that they're either looking, and just haven't come to you yet, or not looking but unhappy enough with their current role that all they needed to start was an email asking them to submit a recent CV.

For the former, if you're Google, do you really think that person doesn't know where to find you? For the latter, you're being quite presumptuous, aren't you? The act of updating my CV my be some effort (actually it's usually not, because the stuff that's not on it is the stuff I can't talk about because it's not released yet, or stuff that's specific and thus wouldn't go on a CV for a vague job spec). Even if it's not the act of interviewing is potentially a waste of time for both of us, if the role isn't clear.

One argument used is that people will be placed according to the skills they show during the interview process. That's fine from the employers point of view, but if you're actively trying to get some interest from people who are gainfully employed you really need to grab their attention somehow. I can't remember the last time I had an unsolicited email interview offer that actually wowed me, or indeed even showed more than a passing sign of tailoring a spec to my profile.

When I was running Black Cat I made a point of always replying to unsolicited CVs. How polite I was depended on how the covering emails were worded (a Word document with nothing else was likely to get short shrift, something well targeted in a Linux friendly format would normally get some comment about how we weren't hiring and were unlikely to be, but if that changed it would be mentioned on the website), but I felt people deserved a reply - I have been disappointed by not receiving responses myself to what I considered well targeted job inquiries.

So far I've so far taken the same approach with mails from corporate recruiters (less so with recruiters that are associated with recruitment firms, rather than directly with the companies they are hiring for), but I'm starting to feel like changing that stance. Candidates are told to tailor CVs to the role being applied for, provide a decent cover letter, and in general make companies want to talk to them. Companies who are sending out recruitment emails should be held to the same standard. Even assuming you do a basic phone screen first, I can probably expect to need to take a day off work assuming that goes well. You need to convince me I can justify that before I'm going to feel like engaging at all.

(And if I'm honest, based on what I've seen so far, it's unlikely to happen. All of the things I've considered have come from conversations with people I know directly about companies they own or work for, never some random contact via email. I try hard not to think of recruitment mail as spam, but I can how that line of thought follows through.)

I should apologize to Google here. They got mentioned as an example, but I don't think they're particularly bad. I did interview with them at one point, and made the decision not to continue that process after deciding a different, more certain, path was better for me. So I've displayed interest. And in response to my reply today of "I know where to find you, so please assume I'll do so if I change my mind." they've said they'll make a note on their records.

I’ve just rebuilt my laptop (a combination of McAfee Whole Disk Encryption slowing the current build down & a Crucial ReadSSD 128Gb that was too cheap to resist forced me to, honest guv), so it’s time to refresh & re-document the essential software list:

1. Windows 7 Professional 64bit
2. VistaSwitcher (better alt-tab)
3. WindowSpace (snap windows to screen edges & other windows, extended keyboard support for moving/resizing)
4. Launchy
5. Thunderbird6
   1. Lightning (required for work calendars)
   2. OBET
   3. Provider for Google Calendar (so I can see my personal calendar)
   4. Google Contacts (sync sync sync)
   5. Mail Redirect (bounce/redirect email to a ticketing system)
   6. Nostalgy (move/copy mail to different folders from the keyboard)
   7. Phoenity Shredder or Littlebird (the default theme is a bit slow, these are lighter and quicker)
   8. Hacked BlunderDelay & mailnews.sendInBackground=true
6. Chrome + Xmarks
7. Xmarks for IE
8. Evernote
9. Dropbox & Dropbox Folder Sync
10. PuTTY (remember to export HKEY_CURRENT_USER\Software\SimonTatham\PuTTY\Sessions)
11. WinSCP
12. Pidgin + OTR
13. gVim
14. Cisco AnyConnect (main work VPN)
15. Cisco VPNClient (backup & OOB VPN)

I think that’s it for now.

Those who know me will know that there have been two important events occurring in my life this week. 1) Tiling my kitchen, and 2) Trying to buy a HP TouchPad.

As part of tiling the kitchen, significant other wanted to replace all the standard white wall sockets with Stainless Steel ones to match the rest of the kitchen colour scheme. I was safely over half way through this task, having disconnected the socket mains supply, when a van pulled up bearing HP TouchPad goodness. Thus the next 30 minutes of my life was accounted for.

Upon returning to the kitchen tasks I came to the Cooker 40A switch. I disconnected the ring main from the switch and proceeded to move the wires so that I kept the supply and load pairs apart when BUZZZZZZ and 240 Volts of high energy goodness shot through my hand and up my arm.

Yes folks, TouchPad caused me to forget to flip the cooker ring circuit in the fuse box.

Cue me jumping back, dropping tools and collapsing to the floor in a bout of Arrrrgh.

Wife said "Stop it, you'll scare the kids.".

Kids run in asking what happened and wife explains that Daddy electrocuted himself.

Six year old Lauren asked "Could you see his bones?".

Based on the messages we have received from family and friends, I’m sure that if Star Trek’s transporter technology worked they would without hesitation beam us out of Tokyo.

But despite the best efforts of many so-called “news” media, I don’t feel afraid to be in Tokyo. I’m not unnamed Starfleet crewman #2. With the tricorder information I have here I am confident that my continued exploration of Tokyo is much safer than being dematerialized, where I would risk being trapped in a pattern buffer for 70 years, or cloned by inadvertent reflection off the ionosphere.

I’m glad to say that the radiation threatening my life, just like the transporter, does not yet exist.

For some years now, basically since Aimee became old enough to use a computer, I have had a need for decent multi-user sound. Specifically I would often have intricate work open in multiple work spaces on my desktop, and Aimee would want to do some artwork.

I guess Aimee was about two when she started using my computer a fair bit, and I immediately had a separate account for her with lower resolution graphics, easier menus, simple shortcuts and the like. I also had a graphics tablet for her, which she mastered very quickly. Another important reason to have a separate account was that if stuff got really badly messed up I could just nuke the account and start again, and she couldn't really hurt my configuration.

As you would imagine, lots of the excellent free software for children, like GCompris, and Tuxpaint is very rich in its sound effects, and sometimes the sound is not just desirable, but essential for the activities. So a very annoying persistent problem has been that, once in a while, something in the sound stack in my login would stop Aimee's sound from working.

Now I share my main PC with two other people, Tamsin and Aimee, and soon Matilda too, so this recurrent problem is more of an issue, it's also very hard to nail down. Despite protests to the contrary the default ALSA setup still has this problem; ConsoleKit on its own doesn't seem to get it quite right. I tried PulseAudio for a while, and generally it was an improvement, but the problem did still occur sometimes. I even made sure all the users were members of the right groups including pulse-rt.

Then I found this entry in the PulseAudio FAQ.

Sound doesn't work when switching users

PulseAudio works with a single user, but when an additional user logs in (fast user switching), sound/audio does not work for the additional user.

Check that no users are part of the "audio" group.

In simple setups (e.g. singe user, without PulseAudio), users must be a member of the "audio" group to access the sound devices (/dev/snd/* (which have group "audio" write permissions)). Switching users will not automatically stop programs using those sound devices though, so those sound devices will not be accessible to a new (faster user switched) user's programs.

By removing all users from the "audio" group (the PulseAudio server still runs in the "audio" group), PulseAudio is able manage access to sound devices (/dev/snd/*) amongst multiple users with the help of ConsoleKit.

It would never have occurred to me to remove the users from the audio group, but doing so seems to have solved the problem. I almost don't want to say that, because every "solution" up to now has been partial, but so far no problems, so maybe this will be the fix.

I have been asked by a few friends to document how I did this. All the information is derived from elsewhere and its currency may be limited. Trying to root and flash phones is potentially dangerous. You need to take your time and consider what you are doing carefully. If you break your phone, you own the pieces. This worked for me, but I can make no guarantees. You get the idea.

Read the whole article before you start please.

The HTC Desire is a lovely phone, but it has two pretty annoying flaws. First of all the battery life isn't great, but that's not what this article is about, and secondly, the internal memory is very limited. I thought when I got the phone that Froyo would save me because it had apps2sd. Apps2sd allows you to move applications from the internal memory to the SD card, but it has some serious limitations:

• apps with widgets and some other bits won't work properly from SD
• many apps only partially move to SD
• some huge apps won't move at all (I'm looking at you Google).

The ROM images I was getting, from Google to HTC to O2 were out of date, and they clearly didn't even care about fixing some significant issues (like the broken authentication in the HTC Peep program). This wouldn't matter because you could install a decent twitter client if you had the memory... oh... you get the idea. I was rapidly having to remove apps hand over fist with every upgrade, and my Daughter was complaining about their absence. So I decided to sort it out.

Some ROM images have the rather different data2sd. This allows you to treat part of your SD card as the internal memory of the phone. This makes a crucial difference, no messing about partially moving to "SD", but allowing a large amount of memory to be treated as internal.

Step Zero: You will need

1. This process will take some time, during which you won't really be able to use your phone. Make sure you have time. If it works you will have to do some work setting some things back up, it will to some extent be like having a new phone; some work can be done to minimise this. See below.
2. For this to work you will need a half decent SD card. Mine is a class 6, 8G card. The class information is written in the card in a number with a circle on it, and has to do with the speed of the card. Class 2 will apparently be painful. Class 4 is apparently fine. But I already had class 6. Get a decent card.
3. Some means of mounting the card on your PC, usually an SD to micro SD adapter.
4. Possibly a blank CD, and some spare Hard Disc space for backups.

Step One: Backup Your Phone.... Really

Even if this all works it will be like having a new phone to some extent, so expect to do some setup again. If you don't have time, don't start. Backup your phone. I had Backup PRO which I used to backup everything. I did this to the SD card this time, but actually, I should have done it online instead (or as well). I'll explain why later. Backup PRO wasn't free, but it was cheap and has been more than worth it for me. I've used it several times.

Now backup your SD card. Either mount it as a drive or take it out of the phone and put it in your adapter or whatever. Copy everything to your PC as files (in the past I've used dd, but this is not needed).

Step Two: Root Your Phone

At some points in the past this has been very tricky, but this was easy this time. I went to the Unrevoked website, and downloaded the software to flash the phone. Follow the instructions carefully and read all the guidance. Click on the Desire, and your OS, and download what's needed (some extra drivers for Windows please note). I was using Debian GNU/Linux at the time.

Turn on "USB Debugging" (Menu >> Settings >> Applications >> Development) on the phone, and then plug the phone in (leave it as charge only when it prompts you). Run the software, wait, and in a few minutes your phone should reboot with ClockworkMod and root access. Note some people suggest you need to run the program on your PC as root, I did need to do so.

Step Three: Check you can access recover mode

For some reason I couldn't do this with the volume buttons on power on, so I did the following. Power off the phone. Hold the "back" (hardware button) down. Now press power on. You will get to the Bootloader screen. By using the power button you can run the Bootloader. This brings another menu and (after some patience) allows you to move up and down with the volume buttons until you pick "Recovery". Now press the power button again. When the phone reboots it will be into recovery mode. Peruse the options, see that you can navigate with the optical trackball and the back button.

Step Four: Possibly install a ROM manager

Now you can pick an alternative ROM of your choice. But this article following on below is specific to the Supernova ROM.

I installed ROM Manager (and it's cheap but not free upgrade) to allow me to download ROMs and flash them, and I flashed Cyanogen... but I didn't like it. I did it for old time's sake and much as I tried to bring myself to like it I found I missed the HTC Sense stuff (that surprised me). You can experiment with all of this. I strongly recommend you opt to wipe user data on a major ROM change, the phone will probably hang if you don't when you reboot, and you'll have to get into recovery mode anyway and do it there.

Step Five: Get the ROM files

The ROM I went for was Supernova since essentially it's a good, HTC Sense oriented, Gingerbread based ROM with the data2sd extra. In other words, you get newer Android goodies with much the same user experience, but don't have to worry so much about the memory. You need to sign up to the website and then go to the download links and get the ROM and data2sd installer. Copy them to your hard disc for now.

Step Six: Prepare your SD card

For the data2sd to work, you need to prepare your SD card with a FAT32 partition (for general use) followed by an ext4 partition (for use as internal memory). Don't panic if you're not a Linux user. There is a way to do this for you.

In Debian, I installed gparted. I then put the SD card in its adapter and in the machine. The machine may mount it automatically... make sure you unmount it before proceeding. Run gparted and follow on below.

If you are using another OS, get the GParted live CD.

Follow these instructions carefully.

Make sure the GParted is accessing the correct device in the pull down before you start, make sure the disc space in front of you looks correct. YOU DO NOT WANT TO ACCIDENTALLY REPARTITION YOUR COMPUTER'S HARD DISC. TAKE YOUR TIME. Personally I went for about 7000 MB on FAT32 and the remainder for my ext4 partition.

Get out of GParted, back into your regular environment (close GParted, reboot or whatever). Now copy your SD card backup (remember that, right?) back onto the card. Finally copy the two ZIP files from the Supernova website (the ROM and data2sd installer) into the root of the SD card. Dismount the card and put it in your phone.

Step Seven: Install the ROM and data2sd bits

I recommend you read and follow the official instructions carefully from here in. Note I didn't bother with the radio code because I was confident it was already very recent. Follow the instructions very carefully to be sure the data2sd will work correctly... note there are a few very specific things you must do and must not do in between boots.. Basically you need to use reovey mode to do a factory reset, navigate to the ROM ZIP and install, reboot, change a few settings, back to recovery, navigate to the data2sd ZIP, install, reboot.

Step Eight: Restore as needed

You should now have loads of space in internal memory. Check in Menu >> Settings >> Applications >> Storage.

I then put in my Google credentials and restored everything after downloading Backup PRO again. Because my backup was on the SD and it was copying to SD, it was slow. I recommend using the online option. Be patient, if you are restoring call logs and SMS messages it will take time, do not navigate away. Wait for it to finish and immediately restart. Think twice about copying "settings", I always worry it will cause the newer ROM to cease. Your mileage may vary.

The thing that finally made me install Chromium? The refusal of Google+ to work with Firefox 3.5 (from Debian/testing). Well played Google. Well played.

Update: Please note I didn't say "switch to Chromium", it was just installing it at all. I've also pulled in FF 5 from experimental which seems to be working nicely - I hope that it can make its way into unstable and thus to testing soon.

Today, two of my friends independently sent me a story about Tau Day which I had hitherto never heard of. One of them asked for me comment about whether this had any point to it. At first I thought the article was just mathematical trolling, thought about it a bit more, thought there might be a real point to it, thought some more and concluded it seemed rather silly.

The argument is about whether the mathematical constant pi, would be better being replaced throughout mathematics with an alternative tau, which is just twice pi (in other words, replacing pi everywhere with a half of this tau). It's suggested that formulae with tau will be more simple.

Basic Geometry

So this is all about the fact that pi was defined historically as the ratio of the circumference to the diameter of the circle, a very old classical reference stemming back to Greek geometry (incidentally pi is also known as Archimedes' constant since he attempted to calculate an approximation to it). Once upon a time, the formula used in schools would have been:



related the circumference C to the diameter D. But generally now, we use the radius r rather than the diameter. And so that gives us (for circumference and area):



The argument for tau begins by observing the extra 2 in the first formula, and wouldn't be nicer if we just defined tau to be twice pi so that these formula would be so much nicer. Would they?



Set aside for the moment the fact that pi is probably the most recognisable Greek letter in the world that speaks languages based on the Latin alphabet (aside from those that are, or appear to be the same). Set aside the fact that tau is used for other specific purposes in much of modern Mathematics, and in particular in the discipline of Topology. The first formula might be nicer, but the second one is probably worse, and by enough to make the improvement of the first rather parlous. OK. But the article talks about this being the problem behind radians, so maybe that's where we get the big gain. Let's explore that.

Radians instead of Degrees

There's nothing particularly clever about using degrees. It's an arbitrary choice (360 degrees in a circle) that probably owes a lot to do with historical factors in one civilisation. It is true that when you start to do some significant mathematics with degrees, it starts to look quite unwieldy. The classic two formulae to consider are the length of an arc and area of a sector.

Suppose we have a circle of radius r and we want to work out the length of an arc (a part of the circumference) where the angle subtending this arc is theta degrees (don't panic, no more Greek to come). Then in degrees the formula will be:



The reason why is that the fraction on the left is the fraction of the relevant angle out of all the angle available, multiplied by the total arc length available (the whole circumference). The formula is not beautiful, and the similar formula for sector area is also a big ugly.



You will note that in both cases there is a 360 on the bottom of the fraction and a 2 pi on the top. This looks like nature's way of trying to tell us something. What would happen if we used an unit of angle so that, instead of having 360 of them in a circle, we had 2 pi of them in a circle (proponents of tau will just say tau of them in a circle)? The formula, derived using the same logic, become much nicer.




So we get



Now these are beautiful, elegant formulae, and the underpinning of why radians (the unit of angle we are talking about here) are used instead of degrees in much of higher mathematics, the formula are much simpler (particularly true when using calculus). Also, look at that first formula, it has all the resonance of F = ma. 1 unit of arc length is found in a circle of radius 1 unit with an angle of 1 radian. So beautiful is this that it used as the definition of the radian in many books. So far, so good. Did we really need tau to produce these? Does it matter that it it tau and not pi that cancels out? I can't see why.

Fourier Series

Another example owes to the work of Fourier, who showed that repeating patterns can be broken into sums of the most basic repeating functions, the ones that are most simple are the sine and cosine functions. These are used to model waves of any sort which are of course ubiquitous in nature. It turns out you can build up more odd shapes like triangular and square waves out of these sinusoidal ones. Being able to do this is important in many aspects of Science and Engineering, and eventually this theory leads to all sorts of cool stuff like the way data is compressed in photos and more.



This graph is labelled in degrees, so you can imagine if we change it to radians as we should then the 360 would become 2 pi. This is where again, proponents of tau will argue that replacing the 360 with a simple tau makes things easier. And it does, if all you want to do is to label that diagram. But the foundation of Fourier theory is building functions up in combinations of these:



The formulae you need to be able to deal with to do this are (among others):



which I grant, strike fear into the hearts of many. But they don't look nicer with tau (and are a little more awkward):



Remember, the people arguing for tau are claiming it simplifies formulae, not making them look worse.

Euler's Identity

Finally, I cannot leave this without talking about Euler's identity considered by most mathematicians (including myself) to be one of the most beautiful results in Mathematics.



This result can be written in a few ways, but this way is very commonly used. This is because in this form you can see how this identity connects the five most important numbers of Mathematics: 0, 1, pi, i and e. With tau, it just doesn't have the same beauty:



so I will stick with pi. Thanks all the same.

This is for my own reference more than anything else; periodically I find myself needing to boot DOS. Usually because I want to upgrade a BIOS and the upgrade tool only works with Windows or DOS. Yes, I'm aware of flashrom, but the scary messages about toasting laptops means I'm much happier going the DOS route there. So I need a writable medium with a DOS boot image. The easy way to do it is to find a DOS floppy image and write that to a USB stick with dd, but it's basically a one off and means you can't fit a whole lot on the image. I wanted to do a native USB boot. The following did the trick for me.

Firstly, this is all on a Debian testing box, in particular with dosfstools (3.0.9-1), mbr (1.1.11-4) and syslinux (2:4.04+dfsg-2). I don't think I'm using anything particularly new from these tools, so I suspect Debian stable will work just fine.

In the below my USB stick is on /dev/sdb, I've got a copy of FreeDOS in ~/FreeDOS/ (I extracted the files from a balder10.img floppy image that's easy enough to find, but all you need is kernel.sys and command.com), and I wanted things neatly in their own subdirectories so I could try and keep the key for repeated use and know which bits I wanted to keep and which were transient. I also added a copy of the Debian Installer for convenience; it would need a copy of the netinst ISO dropped into the root to actually be useful, as there's not enough to do a full netboot from the wider Internet using just the HD initrd.

    # Create a single bootable LBA VFAT partition spanning entire stick.
    echo 0,,C,* | sfdisk /dev/sdb
    # Format as VFAT and give it a disk label.
    mkfs.vfat -n 'DOS BOOT' /dev/sdb1
    # Install an MBR
    install-mbr /dev/sdb
    # Mount it and create some directories for our files.
    mount /dev/sdb1 /media/DOS\ BOOT/
    cd /media/DOS\ BOOT/
    mkdir boot
    mkdir boot/syslinux/
    mkdir FreeDOS
    # Copy the chain.c32 syslinux tool over.
    cp /usr/lib/syslinux/chain.c32 boot/syslinux/
    # Copy FreeDOS over and move the important bits to /
    cp ~/FreeDOS/* FreeDOS/
    mv FreeDOS/command.com .
    mv FreeDOS/autoexec.bat .
    mv FreeDOS/fdconfig.sys .

    # Add the Debian Installer
    mkdir boot/d-i/
    cd boot/d-i/
    wget http://cdn.debian.net/debian/dists/squeeze/main/installer-amd64/current/images/hd-media/initrd.gz
    wget http://cdn.debian.net/debian/dists/squeeze/main/installer-amd64/current/images/hd-media/vmlinuz

    # Install syslinux
    syslinux -d boot /dev/sdb1

I made a boot/syslinux.help file containing:

    Available boot options are: FreeDOS d-i

and a boot/syslinux.cfg with:

    prompt 1
    display syslinux.help
    default FreeDOS

    label FreeDOS
        COM32 /boot/syslinux/chain.c32
        APPEND freedos=/FreeDOS/kernel.sys

    label d-i
        KERNEL /boot/d-i/vmlinuz
        INITRD /boot/d-i/initrd.gz

Initial testing was done using kvm which was a lot easier than constantly rebooting my laptop.

Hopefully this is of help to someone. The only neat bit that I didn't see elsewhere when I was looking was the use of chain.c32 to load kernel.sys rather than having to use a FreeDOS provided boot sector image.

This should probably be an official FAQ, but a) I wanted to rant a bit more than is probably acceptable for something "official" and b) the sort of person this information is directed at never bloody reads keyring.debian.org, which is the logical place for it.

Who are keyring-maint?

Currently Gunnar Wolf (good cop) and Jonathan McDowell (bad cop). Previous keyring maintainers include Igor Grobman & James Troup.

I'd like to be a DM/DD. Do I send you my key?

No. You go through the DebianMaintainer or NM processes. Then the DM team or DAM tell us to add your key to the appropriate keyring.

I'd like to replace my DM/DD key in the Debian keyring. What should I do?

Read the instructions at http://keyring.debian.org/replacing_keys.html

I have a new key that isn't signed by anyone else, will you accept it?

No. Did you read http://keyring.debian.org/replacing_keys.html ?

I've got a single DD signature on my new key. That's enough, right?

Not unless your old key has been lost and you're getting a different DD to request the replacement for you (and if they're prepared to ask for a key replacement we'll wonder why they're not prepared to sign the new key too).

Did you read http://keyring.debian.org/replacing_keys.html ?

I'm still really confused about how I should request a key replacement. Help?

Try reading https://rt.debian.org/Ticket/Display.html?id=3141 (which just happens to be a recent decent example). Clear subject line (I'd have added a real name too, but it's still fairly clear), full fingerprint of the old and new keys, inline signed so RT doesn't mangle it. New key signed by old key and 3 other DDs. Request signed by old key.

That RT link needs a login. I don't have one.

Have you tried reading up on the Debian RT system? There's a generic read only login that'll get you access.

That's too hard. Can't you just give me the details?

Damnit. It appears the read-only login details are currently disabled due to misuse (one wonders how). Try reading http://wiki.debian.org/rt.debian.org

Why are you using RT? Isn't bugs.debian.org more appropriate?

We need the ability to for people to contact us is in a private fashion, for example if they need to us to remove a key because it's been lost or compromised. We could only use RT for that purpose and use bugs.d.o for things that can be public, but this way all the information is in one place and we get to make the call about when it becomes a publicly viewable ticket.

What's with jetring? Should I send you a jetring changeset?

jetring is a tool written by Joey Hess that used to be used to manage the Debian Maintainers keyring. keyring-maint borrowed a number of good ideas from jetring but don't use it at all. We ignore jetring changesets.

So you just want key fingerprints, not attached keys?

Yes. Of course you have to make sure your key is actually on a public keyserver so we can get it. the.earth.li is a good choice (because Jonathan runs it and thus pays more attention to it), but subkeys.pgp.net or pool.sks-keyservers.net are also commonly used.

My key has expired and I want to update the key expiry date. I should email RT asking for this to be done, right?

No, you should send the updated key via HKP to keyring.debian.org. You can do this with

"gpg --keyserver keyring.debian.org --send-key <keyid>"

Obviously replace <keyid> with your own key ID.

I tried to send an entirely new key via HKP to keyring.debian.org, but I can't see it there. What gives?

keyring.debian.org only accepts updates to keys it already knows about. That means you can send updated expiry dates, new uids and new signatures to your existing key, but not an entirely new key.

I sent my updated key via HKP to keyring.debian.org and can see it's updated there, but the Debian archive processing tools (eg dak) don't seem to recognize the update. Why not?

The updates sent via HKP are folded back into the HKP server automatically every 15 minutes or so. They are folded into the live Debian keyrings on a manual basis, at least once a month.

This means if your key has an expiry date then you probably want to update your key at least a month before it expires.

Where can I find these live Debian keyrings?

They're what's available via rsync from keyring.debian.org::keyrings/keyrings/

This is canonical location for the current Debian Developers and Debian Maintainers keyrings.

What about the debian-keyring package?

This is a convenience package of the keyrings. It's usually the most out of date. We update it sporadically and try to ensure that the version shipped with a stable Debian release is current at the point of release. It is not used by any of the official Debian infrastructure.

Why don't you automatically update my key in the live keyring when I send an update via HKP?

We think that automatic updates of keys that allow uploads to Debian are a bad thing and that invoking a human eye at some step of the process is a useful sanity check.

Paranoid much?

Never enough.

How are updates to the keyring tracked?

We use bzr to maintain the keyring, with a separate file per key that can then be easily combined into the various keyrings. You can see the repository at:

http://bzr.debian.org/scm/loggerhead/keyring/debian-keyring/changes

Note that this is only updated when a keyring is pushed to live; the working tree may contain details of compromised keys and thus isn't public.

What's with the whole replacement of 1024 bit keys?

2 things. Firstly 1024 bit keys tend to use SHA1 as a hash algorithm, which has been shown to be weaker than expected. While we're not aware of active exploits against this updating all of the keys Debian uses is not a trivial process and it's wiser to get it done /before/ there's a known issue. Secondly computing power has moved on and we feel that upgrading to larger key sizes is prudent.

Elliptic curve cryptography (ECC) keys look like the future. Can I use one for Debian?

No, not at present. When there are tools that are part of a Debian stable release that support them we'll look into it, after discussion with the major users of the keyring (DSA, ftpmaster, the secretary).

I'm sad to report that I won't be attending DebConf11. I'd have loved to go, but unfortunately I couldn't get the time off work due to various schedule conflicts. I'll have to find another excuse to make it to Bosnia at some point.

Hey guys time for an overview of Gccpy,:

Gccpy is a Python front-end to gcc from my last years GSOC (2010), I was mentored by Ian Lance Taylor who continues to be a great inspiration to me with his work on gccgo and his help on gcc-help to the community of users and developers.

An overview of what the project aims to achieve is creating an AOT compiled version of Python using GCC as a framework for middle-end, back-end optimization as well as protable code-generation. Creating AOT languages has been generally aimed for more ‘low-level’ languages such as C/C++/Fortran where the language requires strong typing and other kinds declarative features; which gives rise to much less dynamic features which languages like Python/PHP/Perl take for granted. The reason these more ‘high-level’  languages are able to do such things is due to the fact they are generally implemented as interpreted languages and this allows for much of this dynamic logic to take place at runtime when a program is passed through their respective interpreters. But gccpy tackles this by generating code which allows for and efficient runtime as well as the dynamic features required.

Personally I feel this is a very strong and important style of implementing dynamic languages; which needs to be shown and proven to be an effective and strong way of implementing new and up-coming languages and why GCC is the right platform to do so.

So ok this was the overview of my Gsoc application, but i think its about the best i can word it. But first lets discuss how this actually works over several blog posts i will demonstrate some of the basic ideas which lets us compile python.  From the title of this post i will quickly demonstrate the ideas behind dynamic typing:

Lets look at how we can easily generate code for some C code for example:

We could generate an IR of:

And the i386 code of:

This is where some interpreters/runtimes start to try and become much more like a ‘virtual machine’ like Java they implement their language by having a runtime which runs code that is in a virtual inscrution set. So when they parse their language with a front-end they generate this virtual instruction set for the given program but then they ‘compile/assemble’ this to bytecode which is a similar akin to C where we assemble the target code to an object code before linking into an executeable format. But really the byte code is nothing more than a binary form of the instruction set to optimize execution of the instruction set.

An example why many people belive generating efficient code for dynamic languages can be difficult is take for example:

So what happens here in an abstract point of view you can’t assume anything about this code due to dynamic typing compared to something like

Having storage specifiers insantly makes this set of code much more declaritive and gives rise to many more assumptions able to be made; which in turn gives a compiler more ‘hints’ on what it can do to generate code. Of course in the example above this is just a hypothetical language just to demonstrate the idea. So to implement dynamic typing we have to analyse what it actually is.

Lets take a normal/regular python session:

So what is actually happening now line by line by showing what each identifier is assigned to what data.

So why is this actually a problem, traditionly take for example code like:

When a c-compiler would run over that code it would give all manar of warnings about type conversion and invalid types being assigned. But why is this since a compiler will want to generate efficient code it will reserve the space valid for an integer on the stack which on an i386 32bit processor would be 32 bits usualy and would asuse subl $4, %esp to have space on the stack for that integer, but the problem arises if we were to then want to put in data which is greater than the size previously allocated for the given initial data. So you will have overflow and corruption of data. So how can you combat that to make dynamic typing work, the method or approach i have taken for gccpy takes much inspiration in how object orientation works. Every piece of static data give in a program is wraped into a gpy_object_t structure at runtime so in turn every type in gccpy is implemented via a gpy_object_t type, so for example the previous python session could be represented in GIMPLE via something like:

The basic idea how dynamic typing is not what an identifier with a specified storage specifier holds its what an identifier points to. So when `x = fold_integer (1)` we should look at what the gpy_object_t structure looks like currently its in many ways similar to how PY_object works in the cpython implemetation but is a little more specific and streamlined to gccpy’s needs.

This structure is quite open to be used in many areas of how gccpy works but what we are interested in is the:

This is the part where it stores the static data defined be it an integer or a class defined in the source code.

This structure is whats used to hold the object_state it holds the object type identifier as a string the reference count for the garbage collector the pointer to a structure in memory which is the actual data for example int or FILE * etc, and also a pointer to the objects definition structure. Each object has its own definition and each definition requires several hooks:

It has the identifier the size of the sturcture of which holds the actual data the initilization hook which returns the object state when you initialize an object the destroy hook for the garbage collector, a print hook for the print keyword to print the data. Now things get more interesting, the binary protocol is whats used to allow for dynamic binary operators so things like:

Can allow for mixed type binary operations, by having hooks for each type of binary operation be it addition subtraction etc. Finaly there is a table of member methods which allows for dot accesors like:

As append and index are both member methods to the builtin type List.

Hey all i am sorry for the last post i have many posts drafted to be posted within the next 2-3 weeks where i am going to discuss my project GccPy as there has been alot happening with it recently, i am working on a paper which shows everything about it so far but i want to have some posts demonstrating the basic principles of compilers and in deed creating an Ahead of Time version of Python on top of GCC and how this actually works. I personally find it a pretty cool topic but first i will am just in a bit of rant mood and its about whats happened since my last blog post:

Well anyway i’m currently in an awkward position i feel i may leave university but i will see because its simply causing me so much problems in my day to day life, I’ve got onto Google summer of code 2011 yet again to work on Gccpy with Ian lance Taylor but i may focus a little more on working on the Gcc middle-end to make some of my work a little easier. But the problem i am having at the moment with university is i actually failed all my modules last semester which may sound awful. Esp the one in compiler development i am writing a complaint about this at the moment due to the fact i actually took my own time to create a compiler specificity for this language they developed which works i cant emphasize this enough it actually works and demonstrates an IR properly designed and works well and doesn’t segv if you put in a syntax/grammar error i didnt even have to do this for the module.

I guess it was my own fault where i answered the exam using x86 code and more normal DAG graphs and IR’s to demonstrate things to answer the questions where they wanted this _FAKE_ code and _WIERD_ ideas to demonstrate Lexical Analysis. I failed my Math one but i kind of expected this one because i just didnt care anymore the lectureer just had a problem with me and wouldnt put any work online as i wasnt able to attend for a week or so and just gave me awkard looks when i went to class and was just sooo condecending the whole time to me i tried hard but it was just awful then the other exam concurrent programming where your forced to learn off all these Java constructs to implement monitors thats all fine 100% in all practicals but exam your just forced to learn off meaning less facts and stuff where it really means nothing and all the practicals are only worth 10% of the module.

So yeah didnt go well but i feel very strongly about my compiler development module so i am going to write a complaint about it and send it in, the problem being is i am just made to feel like one of the students who just goes out and doesnt do work. I am so far away from that statement its insane, i am 100x more dedicated and practical than anyone in my university thats why after Gsoc 2011 i have another interview with Google which will be interesting. I am just so increibly sick of feeling useless, i just feel like giving up at the moment. Its destroyed nearly all my confidence recently, if it wasnt for getting Gsoc 2011 for my own project i dont know what i would do sometimes.

Its not just those exams that annoyed me even a few years ago i remember studying months i mean months for an exam and lost by like 5% i was so gobsmacked. Its just got to this stage where absolutly everything i have done in university has just annoyed me so much i cant concentrate at all, when you consider absolutly everything is soooo niche and acedemic its insane in computer science esp in open-source and linux its such a hands on subject. It scares me that everyone in my university couldnt actually design a project and implement any kind of system level project even though they are meant to have studied it. Take the operating systems module is literately take a book and re-wrtie paragraphs of crap down in an exam the exact same went for networking.

Basically i found this comment on a website http://ihateuniversity.com/2004/03/why-i-hate-university-life/:

University seems to suck all the life out of learning. The assignments are meaningless, the lectures are boring and the people are all mindless. I have no time anymore to read a book…a real book, worth reading. I’m too busy with my nose buried in a textbook learning meaningless facts to pass another meaningless test, to earn a meaningless number entitled “grade”.

Sorry for the rant but i just have been made to feel very useless at the moment and personally i just feel if i was to just quit university to persue my gccpy project over summer and have this paper finished and apply to jobs based on this project as it demonstrates everything i know and understand and how i designed it all and implemented it. Also GCC hackers are a very scarce resource so i can even taylor alot of middle end trickery to pop up specific things for a project using plugins etc. Which i think could be a usefull thing for a company to have but i have literly no confidence in my self anymore i in the UK university has become this Class system and if you didnt pass your made to feel like a lesser person. But its kind of irritating i’ve tried to get help from my university but because i do Mathematics and Computer Science there are like zero people who do this course esp my set of modules i choose i get like no help or no one to talk to coupled with the fact my advisor of studies just retired and the new guy doesn’t really know.

The thing that has annoyed me the most i feel almost ashamed of myself to talk to my friends on identi.ca i just need to prove myself and i just dont know what to persue but i know right now i am pretty much miserable when i worked full time for SAP i was happy they offered me a PHd positions for goodness sake. I must be good at something when so few get that as well as Gsoc twice in a row on the same project which i started myself. I’ve set my self the goal of at Oggcamp comming up i will have the Linux outlaws testsuite for Gccpy working which it pretty much is and to demonstrate its awesomeness to everyone i care about.

Dont worry i will be posting up from now over the next few weeks much more frequently every time i write a blog post i feel much better about myself. I a lot to offer the world and its time for me to do it.

PS: Sorry for the mountains of spelling errors! xD and i prob wont leave uni its just how i feel i talked some stuff over and feel alot better i just need to stay calm and get though it all xD

I don't normally mention the PGP keyserver I've been working on for the past 9 years, but there are a few keyserver related comments I've been meaning to make that prompt me to do so this time.

onak is the keyserver powering the.earth.li (AKA wwwkeys.uk.pgp.net). Back in February I finally enabled hkps (HKP over SSL) access to it as I promised dkg I would do back at DebConf10. I also turned on IPv6.

Most of the active keyservers out there are running SKS. I think there might be a few OpenPKSD instances as well. If you're running SKS please ensure that you have at least one address in your mailsync file, so that you send key updates to the non SKS keyservers out there - it seems that SKS requires the receiving keyserver to send out updates via email, rather than each server sending out any updates it processes to its mailsync peers. Feel free to add pgp-public-keys at earth.li - no need to contact me to do so (but you can drop me a line if you want me to send you mail updates back).

Also in the past there was a pgp-keyserver-folk mailing list, originally on flame.org, then on alt.org. Both of these have been dead for a while. I created a local pgp-keyserver-folk some time ago, but never publicised it. I believe the SKS lists are fairly active, but limited to that community, so it would be nice to have somewhere that was keyserver agnostic that admins could talk to each other.

Anyway. onak 0.4.0. Largely fixes to keyd, which is a backend daemon that handles talking to the key database. This is particularly useful for large/busy keyservers. Also a couple of fixes that make the db4 backend a lot more reliable (especially with queries that return lots of results). And various other minor cleanups. If you're one of the 3 people running it you probably want to upgrade; updated packages have already been uploaded to Debian unstable.

(This is part of a series of posts on Why Linux?)

This is probably a non-obvious reason to run Linux to most people. One of the complaints I've heard in the past is the lack of support for Linux. It's not really relevant to running it on a work desktop (where support is presumably from some central IT organisation), but I believe it's a red herring. Support in the Free software world can exceed that in the proprietary by an impressive margin. And that can turn out to be useful at work, even if you do suppose the existence of central IT.

The support I've received for Linux and applications under it has been, on numerous occasions, excellent in a way I feel I could never expect from the commercial software world. And when it isn't, I have the choice of whomever I want to try and fix my problems, rather than being stuck with no options.

I have found that if I am polite, and provide useful information, and patient, I can talk directly to the developers who make the software I use. If I have a patch they will often take it from me, or use it as a basis for the final fix. If they are no longer available, or too busy, then because I have the source I can find someone else to look at the problem, or do so myself.

The chances of most large proprietary software companies giving me the time of day is low. I don't understand why small businesses think that Windows is a safer choice due to being able to get support from Microsoft.

Again using experience gained at Black Cat, at one point we had a problem with IPv6 routes being updated. Signs pointed to Quagga, Simon collected relevant information and got in contact with the Quagga devs. They investigated, realised it wasn't a problem with their code, and then proceeded to find the issue in the Linux kernel, produce the fix and pass it upstream. Within hours of us reporting the problem. I'll accept that's an exceptional response that can be matched within the proprietary software world, but not without a hefty support contract and I doubt you would have got a response other than "Not our fault, talk to the other supplier", or maybe a workaround at best.

More recently, I had a conversation about Linux and discard support, with one of the people who'd worked on it. Very interesting from my PoV, useful to my work (I ended up fixing something in our code base that led to the Linux support Just Working with our product) and I believe was useful to that developer too, in terms of providing some insight to the other side of the problem. I don't get that kind of access to proprietary software developers - while it could be argued that I know a lot of Free software people, I also know a lot of other programmers and they're just not able to be so open about things, so both sides lose out.

Or my new laptop's ethernet wasn't supported by the kernel in squeeze. I filed a bug requesting the driver be added (and pointing to the patch). The week I wrote the first draft of this article I saw confirmation that this had been done, and will be part of the final squeeze release. I think that's pretty impressive. (Thanks, Ben.)

If you were not aware - O2 last week reduced the price of the O2 Joggler from £149.99 to £99.99 to £49.99.  Nothing remarkable in price reductions, however what is remarkable is what you actually get for your money.

The O2 Joggler is a silent 7" touchscreen device with Intel Atom Z520 running at 1.3Ghz, 512Mb ram, 1GB internal flash storage with additional storage available via an external USB port. What makes the device really exciting is that it also has a 1GB ethernet (Realtec 8168) and Wifi.

Looking at that - I know you're all thinking "linux box" - but conveniently, the default operating system on it is based on Ubuntu 8.04 and busybox, The frontend is a custom flash driven UI developed by OpenPeak (makers of the Jogger which is rebadged by O2).

There are plenty of clever individuals over at http://hackthejoggler.freeforums.org/and in the #mer channel on freenode who are working to allow the Joggler to run other Operating Systems such as Android, Ubuntu Netbook Edition, MID, even Windows and OSX.

However, the standard OS supplied is also pretty powerful and allows significant customisation.  You can enable telnet just by turning it on with a custom USB stick plugged in. To do some of the things I'll be describing here you will need to have started with that.

Once you can telnet in, the world opens up and you can do lots of things that you would expect* to be able to do in a linux system. (* except run a web browser.... at this time - we don't have web access with the stock OS due to the custom flash GUI interface.)

Lets install some useful utilities, how about perl, terminfo, irc clients, bit torrent, rsync, ssh/scp ?   Yes, I know what you're thinking - overnight silent downloads - no need to leave PC on... nice...

I've built several of these utilities with installation instructions and uploaded them over here: 
  http://pgregg.com/projects/joggler/

Feel free to download, examine, install etc.  Read the README on the download page - it explains how to do it. Most of the packs include an install.sh script that will provide an installation safety net - it won't overwrite existing files or libraries.

Usual disclaimers apply - you try these applications at your own risk. I accept no responsibility if you manage to brick, fry or otherwise trash your joggler.

Comments, suggestions for other applications are always welcome.

Update: Thanks to NP - seems I was a little too brutal with the library stripping to keep the download sizes small and I missed some required libraries (that I mistakenly thought were in the standard Joggler distro).   I have rebuilt the following packages as they were missing some libs: rtorrent, rsync, sudo,ssh
The screen package has been rebuilt too to add one final tweak (to the installer script) so it can be used by non-root users - the only difference you need to do is run the command: chmod 666 /dev/ptmp /dev/tty

I've been looking for some home backup solutions over the past couple of months. This has led me down both the do-it-yourself route and buying a ready-made solution.

One of my requirements was that I wanted the solution to be more than just storage - otherwise I would have purchased a straight NAS box from the likes of Qnap, Netgear or if feeling rich Drobo. Most of these dedicated NAS boxes can be "rooted" to allow ssh access , however their CPUs are generally underpowered for general purpose use.

Other requirements were that I wanted a reasonably small form factor and to be able to use at least 4 SATA hard drives, preferably with hot swap ability. Hardware raid was not a requirement because I intended on using a Linux distribution with mdadm software raid.

In the end, I ended up building two boxes.
The first, a home build, based on the CFI A7879 chassis with a Gigabyte GA-D525TUD Dual Core Atom Mini-ITX Board.

Thankfully, that is exactly how it looked before the quake. Nothing was damaged.

It's my birthday, so a week ago I bought myself a present:

The office is 4.5 miles from the BART station, so up until now I've been taking a bus. Unfortunately the timings aren't that frequent (every 45-60 minutes). Plus I haven't been doing any exercise since I moved out here. Now it seems to be getting drier I figured it was time to sort out a bike. I wanted something flexible, and have heard nothing but good things from the Brompton owners I know, so that's what I ended up with. I'm very happy with it; despite my lack of recent exercise it's pretty easy to ride. I've actually found it doesn't take any longer than taking the bus, and is sometimes quicker, because I avoid having all of the waiting around for it.

I bought it from Warm Planet Bikes, who were incredibly helpful. They had a selection in store that they let me try (and quite a few other models of folding bikes too) and when I made my decision it turned out they could do the customizations I wanted in shop. So rather than me having to order it and wait a month I was able to cycle away that same evening! Full marks to Matt for staying late to sort me out (plus he's a Brompton owner too).

I've been out here for 6 months now. At the point in time I was coming out I was pointed at Andrew Pollock's On bootstrapping oneself in the US. I figure some extra data may be useful for anyone considering the same. I moved from the UK rather than Australia.

Banking

I sorted out my US banking long before I ever moved here; in fact I kicked the process off shortly after I accepted the move offer. Matthew had recommended the HSBC approach, so I did that. This meant when I arrived I had a credit card, debit card and check book. They also allow me to do really easy online transfers between my US and UK accounts.

Mobile phone

My mobile phone was something else that was dealt with before the move. I bought a Simple Mobile SIM off eBay. I thought I'd end up moving to a contract once I had a credit record here, but actually the $60/month unlimited tariff seems to be fairly reasonable. Their customer service leaves a lot to be desired, but I've sadly come to expect that from phone companies. I've got a Google Voice account which is the number I've been giving out, so it's easily for me to switch providers (or redirect things to a VOIP setup for when I'm back in the UK, but I've got a writeup to do for that at a later point in time).

Finding somewhere to live

One of my concerns was that it would be hard to find somewhere to live at short notice without any credit history or SSN. This turned out to be ill founded. Both of the managed apartment complexes I looked at were ok with the fact I'd just moved to the country and seemed used to such a setup. The place I went with is privately leased and was a bit more wary, but after a conversation with my US manager (who'd managed me even when I was in the UK) it was fine.

Social Security Number

AKA existing as an entity as far as the US is concern. I was told to wait at least 10 working days after entry before even applying for this, or I wouldn't be in the system and it could cause delays. I did so and the card turned up within about a week (I had been told up to 2 weeks). Which was good, because work were having problems figuring out how to pay me without one.

You do seem to need the SSN for everything. The only times I can remember needing my National Insurance number in the UK is for new jobs and my E111 application.

Driving licence

I haven't ended up buying a car here; I pretty much don't need one most of the time and I've signed up to ZipCar for easy short term rentals - they're happy to take a UK licence for sign up. However I figured getting a licence here was a good idea in case I did decide to get a car, and even just for ID (I kept having to point people at the DOB on my UK licence - everywhere here seems to card). Applying for a licence ideally relies an SSN, so I did so a few days after I'd got that. I did the car + motorbike written tests and got my learner's permit. I applied for the car theory test the same day, but the first available slot was about 6 weeks later. During this period I did the CMSP basic RiderCourse, which meant that when I passed the car test I was also fully entitled to ride a motorbike here.

Other useful notes

As I live in the Bay Area I got myself a ClipperCard - it's like Oyster except without the clever discounts. Works on more agencies though and helps avoid having to sort out change. For broadband I went with Sonic.net - I'm in their ADSL2 coverage area so I'm getting a reasonable 17Mb/s connection. The other options were AT&T ADSL (slower) or Comcast cable (evil). I've had zero problems with Sonic; It Just Works.

Also I sort of ended up with a trial run thanks to DebConf10 being in New York. Due to the way timings worked out this took place only a few weeks before my move to SF was scheduled. My banking and phone were already in place by this point, so it allowed me to check both were working ok. They were, and I didn't really expect them not to be, but it was a nice safety net to have for the move proper.

Overall the main issue I had was that a lot of things were dependant on other things; get the SSN, get a learner's permit, apply for the practical test, wait for the licence to actually turn up etc. It probably took over 3 months for everything to actually happen and come through in the post. I can tell when I actually started to exist in the system because Capital One being junk mailing me about whether I want a credit card with them...

I did my first upgrade of a lenny box to squeeze today; a test server in work. All went pretty smoothly but I had a handful of things I had to frob manually that I thought I'd write up here:

• collectd SNMP plugin needs MIBs or it won't start: installed snmp-mibs-downloader from non-free
• autofs starts before nis (#470573): added ypbind to the Required-Start: line in /etc/init.d/autofs and re-ran insserv
• NFS automounts now using NFSv4 instead of v3 and user/group mapping setting everything to nobody: Set NEED_IDMAPD=yes in /etc/default/nfs-common and ensure Domain in /etc/idmapd.conf is set to the correct domain name (look in /var/log/daemon.log for lines from rpc.idmapd saying "does not map into domain" to find out what this should be if you don't know).

I think that's pretty smooth overall; kudos to all those involved. I've a few more boxes to upgrade, but they're all more likely to have people complaining at me if there are hiccups so they'll have to wait until I have a suitable block of time set aside.

(This is part of a series of posts on Why Linux?)

I think of myself as reasonably pragmatic in my approach to Free/Open Source software. I don't get worked up over which set of language people want to use. I use devices that require binary firmware to be downloaded to them (because just because I can't see it doesn't mean it doesn't exist). I have non-free in my sources.list.

And yet, talking to other Linux users these days, I realize I'm much more of a Freedom nut job than average. I want the source, be it for a driver, a minor widget, or a full app. I don't buy nVidia. I will sacrifice a degree of functionality in order to get Free. And while I think WINE is an excellent piece of software, I think the best end result is that it's no longer necessary, not that it's a perfect implementation of the ABI.

How does any of this help justify my use of Linux in the work place? As previously mentioned, I'm a developer. Most developers don't operate in a vacuum; they have to inter-operate with other ecosystems. And usually somewhere along the line there's a failure to document exactly how something is handled, or an ambiguity about what exact choice might be taken. If I have access to the source then I can check that out for myself. If I don't, I have to guess. As an example, a long time ago I was involved in writing a serial console driver for QNX. There came a point where the behaviour wasn't quite as we'd expect. Although the organisation had a license for the source, I wasn't allowed to look at it. Instead I had to come up with a series of suitable questions that someone who could look at the source could answer without violating any NDAs. If I'd been able to look at the source directly we'd have all saved a lot of time. And that's an example where someone could look at the source, rather than having to make a bunch of guesses and instrument tests to see which was right.

Access to the Linux source has helped me in other commercial contexts too. At Black Cat we were able to take advantage of patches like grsecurity in order to tighten up shell account boxes. I wrote the IPv6 support for l2tpns, because we had access to the source and could. I've been able to look at the source to understand exactly what SCSI responses are sent in certain circumstances too (or understand exactly what the error that a user land test program was getting back meant).

Also I'm a big believe in Linus' Law. I do think that good Free software is much better than proprietary software (there's some really bad Free software out there though, I'm not disputing that). The fact that smart people can look at it and scratch whatever their itch is means that we get a gradual process of improvement that can't be ignored. Equally as long as someone has an interest in the software, end users can't be left high and dry by organisations abandoning still users applications. I think that should be a powerful driver to business to look towards Free software.

(Before my more astute readers point it out; yes, I am employed writing non-free software. See the first sentence. One day I'll find a job working on Free software that ticks enough of the other boxes to be viable.)

I love revision control. I love the ability to track changes over time, whether that be to see why I changed something in the past, or to see why a particular thing has stopped working, or to see if a particular thing is fixed in a more recent version than the one I'm using.

However I have a few opinions about the use of revision control that are obviously not shared by other people. Here are a few of them:

• One change per changeset.

  The only argument I can see against this is laziness. Changesets are cheap. Checking in multiple things in a single go makes it hard to work out exactly which piece of code fixes which problem. I'm fine with a big initial drops of code if logically it all needs to go together, but changesets that bundle up half a dozen different fixes piss me off.

• Descriptive changeset comments.

  Don't make me guess what you changed. Tell me. Bug numbers are not sufficient (though including them is really helpful).

• Comments in the changeset, not per file.

  I've only seen this with BitKeeper; you can have per file comments and then an overall changeset comment. At first I thought this was quite neat, because you can explain each part of a change. Now it just annoys me, because I want the relevant detail in one place rather than having to drill down to a per file level to figure out what's going on.

• The tree should always compile.

  There are people I respect who are all for checking in all the time throughout development no matter what the status. I have to disagree, at least for anything that's available to other people. The tree should always compile. This avoids pissing off your coworkers (especially if they're in a different timezone) and means you can do things like git bisect more easily. Plus it shows you've at least done minimal testing.

• Don't hide your tree.

  I like centralised locations for master trees. It means I can make an educated guess about where to look first for information about changes. Trees that live in obscure network shares or, worse, someone's home directory aren't helpful. While I may not always agree with the choice of VCS for the centralised service as long as it's actually fit for purpose I think it makes much more sense to use it than go off on a separate path that's less obvious for others to find.

Opinion Piece

I was googling (as a verb) and came across a rather peculiar message at the bottom of Google's search results:

In response to a complaint we received under the US Digital Millennium Copyright Act, we have removed 1 result(s) from this page. If you wish, you may read the DMCA complaint that caused the removal(s) at ChillingEffects.org.

Interesting - never saw that before!

Following the link to Chilling Effects shows a copy of the complaint which has some interesting text in it.

Experts-Exchange makes a detailed itemisation of their registered Copyrights, none of which I find objectionable, however, the complaint then goes on to list several issues against the Defendant, the first and most egregious of which is:

a direct "copy and paste job" lifting the content of Plaintiff's question and answer forums and inserting them onto AllFAQ's website. AllFAQ's question and "Solutions" are verbatim to Experts-Exchange's questions and "Accepted Solutions;"

From this Experts Exchange is accusing allfaq.org of Copyright infringement against Experts Exchange owned Copyright.

At first glance, this might seem fully justified - but look at what they are claiming copyright on.  Experts Exchange are assuming copyright ownership of content that you, and I, and all their users create by asking and answering questions on their web site.

I looked at Experts Exchange's Terms of Use and could not find any agreement that users were assigning their rights and copyrights to Experts Exchange. The relevant paragraph is:

"5. Content License

EXPERTS EXCHANGE enables Members to post problems or questions, proposed solutions or answers, information, comments and other content ("Your Content") to its Site. When you post Your Content to the Site, you understand and agree that Your Content can be viewed and used by other Members who visit the Site with or without attribution.

You represent and warrant that you own or otherwise control all of the rights to Your Content and that use of Your Content by EXPERTS EXCHANGE and its affiliates will not infringe upon or violate the rights of any third party. Before you use EXPERTS EXCHANGE Services to post any information or content that is protected by intellectual property laws, you shall have acquired the legal right to do so from the owner or authorized licensee of such information or content.

By registering with EXPERTS EXCHANGE and posting Your Content on the Site, you hereby: (i) grant EXPERTS EXCHANGE a non-exclusive, perpetual, irrevocable, unrestricted, transferable, fully sub-licensable, worldwide, royalty-free license to use, distribute, display, reproduce, perform, modify, adapt, publish, translate and create derivative works from Your Content in any form, media or technology, whether now-known or hereafter developed; (ii) grant EXPERTS EXCHANGE and its affiliates and sub-licensees the right to use the Member Name that you submit with Your Content for purposes of attribution; (iii) authorize EXPERTS EXCHANGE to assert and prosecute claims against any third-party making any unauthorized use of Your Content, including any use that violates this User Agreement ("Third-Party Claims"); and (iv) appoint EXPERTS EXCHANGE as your attorney-in-fact for the purpose of asserting and prosecuting Third-Party Claims. If you do not wish to have Your Content attributed to you, then you must notify EXPERTS EXCHANGE at customer_service@experts-exchange.com.

Experts Exchange acknowledges that the copyright belongs to the author as "Your Content" and that by posting you are granting them extensive licenses to use that content. You are not assigning your copyright to Experts Exchange.

Now I am glad that their ToU does not attempt to wrest copyright ownership from its rightful owner, that is right and proper.

allfaq.org is demonstrably guilty of screen-scraping the Experts Exchange web site and I do not condone those actions at all. However, looking at what they copied - it was the Title, Question and Accepted Solution text - the copyright of 100% of that is with the original authors, and not Experts Exchange.

Thus, in my opinion, this complaint against allfaq.org is without merit and should be dismissed.

It would also appear that Experts Exchange has also abused the provisions of the DMCA in forcing Google to remove the content. Google should restore the links.

And finally, Experts Exchange should implement some technical measures to prevent automated scraping. Find better ways to improve your search ranking, and if your competition beats you don't ask your own members how to do better SEO; be told by them that you have no Copyright Claims on the content; and then proceed to file DMCA take down notices when you know you have no (copy)right.

Following up on my recent post on the engineers toobox, I’ve just rebuilt my Apple MacBook (newer, bigger hard disk was the perfect opportunity for a fresh Snow Leopard install and to fix some annoying iPhoto index & thumbnail corruption), so here’s my list of essentials for my MacBook, in no particular order:

• Evernote
• SpanningSync
• Dropbox
• Last.fm
• Xmarks for Safari
• Xmarks for Firefox
• Panic Coda
• Thunderbird 3.x
  • British Dictionary
  • Enigmail
  • Google Contacts
  • Mail Redirect
  • Nostalgy
  • Mnenhy (view a custom set of headers)
  • Display Remote Images automatically
  • Custom Archive folder (one big bucket please)
• Adium
• VMWare Fusion
• iPhoto
• iMovie
• QuickSilver
• Microsoft Office
• Google Chrome
• Flickr Uploader
• Skype
• Google Picasa
• Get iPlayer Automator
• Cyberduck
• MacVim
• Spotify
• iSquint
• AudioHub
• VisualHub
• SuperSync
• TweetDeck
• ClickToFlash
• Growl

Happy new year everyone I hope your all well and having a nice fresh start. I am going through a bit of a rough patch this last few months its been fairly difficult to deal with things, and I’ve found myself going though some depression because of it all. Mainly because of University this is my final year i just wanted to finish it so i can go out and work properly again but the sheer frustration i have  with my university is nearly destroying me. Its nothing to do with the cliché of Tuition fee’s or anything like that its with the Lectures the modules and content being taught and how its being taught.

So for example i thought i would go and take the Compiler Module there last  semester, found out the lecturer has absolutely no clue. I mean i now have redbrain@gcc.gnu.org i post _loads_ of help to   bison-help gcc-help comp.compilers am a project lead on the python front-end for gcc and am working in my own ideas for front and middle end interaction for gcc. I’d like to think i know how to build   compiler since i have write an  interpreter for a dynamic language I’ve designed i’ve write a compiler from scratch for a basic language called Mod that outputs i386 code has  code ready to handle  optimisations a well formed IR called mod-dot. Then i start this module we get given an absolutely terrible compiler for this Mod language written in Java that outputs some insane weird   instruction set  that cant be assembled or linked… And the laxer and parser code is unbelievably disgusting and is written clearly by someone who doesn’t know how to build a compiler along with the  fact our lecturer thinks you _need_ EBNF to handle this very basic dialect of modular we call mod. So i tough yeah i will write my own from scratch and i did its over at code.redbrain.co.uk. Its 10x  clearer cleaner and easier to understand, and remember this is a lecturer trying to teach compiler development not only was the parser code hard and very ugly they never go into anything like DFA or  LR or LALR or decent regular expressions for the lexer. Its this hugely protracted view of how a compiler could work and in the end teaches nothing but scares students. And then the backend to this  compiler was truly awful it has pretty much Zero IR structure in it using nothing but statics to in a hackish way get the job done. We had 3 assignments to extend this compiler i pretty much ignored  them and just worked at my own since it did the job 10x better and more clearly and easier and it output code you could run! Anyone onto my Mathematics module, all i can say is thank flip this is my  last mathematics module i will _ever_ do i my life i mean when i started my Degree in Mathematics and Computer Science i liked Math in high school it was the only subject i was good at i mean i  enjoyed solving the problems. But in University all i have done is work at pretty much utterly random topic after topic with little or no coherence between anything so you literal would learn off module  for the exam forget everything move onto the next one. And i will _never_ use anything have learned from it again, i was hoping with mathematics i would learn decent logic and problem solving skills but i have learnt nothing. all i want to do is erase the name Chebsev the Russian mathematician out of my brain for the utterly random crap of his i have had to learn. I mean i try going to lectures but i usually don’t bother any-more it depresses me so much i don’t sleep at all these days the its bothered me so much to the fact i struggle to get motivated to work on my personal gcc stuff.

I mean what do people want degree’s for. I mean i am not stupid far from it i just want a life and at the moment i don’t have one. I mean i have been contacted by Google and Cisco for interviews for which i didn’t even apply for!!! And all of that has been down to working in open source with Gcc and absolutely _nothing_ to do with my degree i seriously doubt i will _ever_ use anything of my degree ever again.

Then my 3rd module at the moment is concurrent programming it was ok but seriously the lecturers at my uni need to grow up absolutely everything in my uni in the computer science dept is 20% practical at max then 80% theory. So i mean wtf people come out not able to program or know how anything works.

I know people in my computer science classes don’t know what ram is what hard drives look like and how to even build a simple desktop computer… Then even worse everything is windows the word Unix doesn’t get edge way unless its MacOS then programming languages Jezz don’t get me started. Everyone has this knee jerk reaction to the word C and start arguing like a gaggle of geese. I mean C is my fav language its simple it forces people to understand their computer, but in uni is Java or nothing and to be honest Java is absolutly killing university graduates creating drones who have absolutly no idea how a computer works or how to build software.

If universities were to teach C along side a decent i386 assembler to learn an instruction set then into a project each year of university students would get experience and get a portfolio of work and abilities and could do a project in anything that would interest them and if they learn C they are _not_ limited into what they could do since if you learn Java you will find moving to something else a complete struggle for a long time since you learn so many bad habits.

Right now i have exams coming up which is what i hate we do out winter semester then have exams after Christmas it makes you not able to enjoy the holiday what so ever but the thing that kills me the most i know i am better at anyone in my compiler module there only are like 15 of us and not one of them barley handed in their assignment but i know some ass kisser will get like 90% in the module and i will come out with like %50 making me look like an ass since i am redbrain@gcc.gnu.org but the problem is if you look at the exam is nothing but these utterly random questions directly related to this Java compiler they produced that is just a mess. I mean the worst part is those people will be able to focus and revise for it but i cant i have always have to do 90% revision on my math modules and like 1 nights revision for computer science Math has just been so incredibly difficult simply because is incredibly random and pointless. Plus the lecture i had for my math module had his head so far up his ass its been insane he has not even once been remotly helpful to me putting assignments online or anything i have had so many assignments to do this semester i was stuggling to keep my head above water and because i do 2 subjects unlike most people i am the only person in my situation and i get absolutely no help and there is absolutely no standard across the board.

I really don’t understand why people put so much faith in degrees when i graduate i will come out with a plethora of open source experience and actual code people can look at an peruse which many people already do use my work as a reference in the real world.

And thing is i just want to be free of this degree its really killing me karma at the moment i just want to be able to rest properly. I am so on edge and i really don’t mix well with people in my class most really don’t care about a computers more into parting and just revising crap for exams and really have zero understanding of real world applications.

Yeah so that leads me into my concurrent programming module i mean it started off ok but i mean it was sooo java centric it was silly in how you can program monitors etc but i mean still it was so Java centric that to reuse what he was doing outside of it would be difficult to say the least so basically i was crap then the last part of the module which makes up one third of it was to do with some dudes PHD project this super niche idea of modelling concurrent processes which in the end was just finite state machines but just retarded let me find it http://www.doc.ic.ac.uk/~jnm/book/ i mean this book was just a heap of crap academics are ruining software i mean they absolutely have no clue outside of their super niche world absolutely nothing in there is helpful to anyone outside of Java land.

I mean at university you should be learning more general skills and abilities to be able to adapt and not be sand-boxed into these real niches where you cant escape.

I always said to myself i don’t want to blame university for everything that’s why i have done so much of my own work and projects and open source because then i actually have work people already use and can see and judge me based on that i just hope when i get out of here people will judge me based on that not my degree because at this stage i could have got a 1st but my marks are at 2:1 2:2 ish since the maths modules kill me so much since i do half a maths degree and half a computer science degree its not meshed into one thing. Right now all i want is to get a 3rd do min work on my degree to keep my self happy because right now i am just so depressed i cant sleep properly and it keeps making me sick.

I am just happy i have my friends and keeping things into perspective the odd time is nice since i have been out of routine not on identi.ca that much not keeping up with Linux outlaws which i really miss and i really miss my Linux outlaws friends. One of the happiest times of my life was in Liverpool with everyone i hope your all doing well sorry for this super rant i just really needed to get this down on paper to let it all out so i can go for a walk I’m feeling a lot better now.

PS: sorry for grammar and spelling mistakes i just needed to get this all off my chest i have some cool things to show in the next while after this few weeks are over me.

(This is part of a series of posts on Why Linux?)

I find Linux more flexible. Maybe that's the familiarity showing, maybe it's about the package management, but it's a powerful reason for me to use it.

For example, a couple of years ago I wanted to try out some iSCSI stuff against a SAN. Of course I have test boxes available I can do this on, but this was just to try out a few bits and pieces rather than anything more concrete. So I installed open-iscsi on my desktop and was able to merrily do the tests I wanted with very little additional work.

Or I wanted to try out some BitKeeper to git conversion work recently. I wasn't sure how much resource it would take on a build server, and didn't want to tie things up there. So I ran it on my desktop overnight, where I could easily setup the appropriate environment and wouldn't impact on anyone else's resources.

Problems talking to dodgy hardware? Linux is much better about giving you some idea what's going on, without needing to install extra software. I had a workmate grappling with an old USB music player recently; hooking it up to her Windows laptop wasn't providing a lot of joy so I attached it to my Linux box and was able to see that it did identify ok, but was disconnecting randomly at times too.

Want to script querying an AD server for the current employee list and displaying who's joined and who's left since the last time you did so? I found that easy enough with the common Linux LDAP tools. I'm sure it's doable under Windows too, but I'm not sure it would be quite so simple. For bonus points add graphviz into the mix for automatic organisation charts (modulo accuracy of the AD data).

This flexibility is something that helps me do my job. Sure, as I mentioned above I do have access to test boxes that I can use for this, but being able to do it on my desktop can be useful - for example if I'm offline, or on a slow network connection, or just geographically distant from my test machines so network latency is higher than I'd like.

(Also, it's something that makes a Linux box a really great test box. I'm lucky in that I have a mix of OSes available to me for testing, but the one that I use most often is the Debian box. Much easier to get and install decent diagnosis tools for it that can give me packet level dumps, or do really odd stuff that turns out to be really useful.)

(This is part of a series of posts on Why Linux?)

I've run a number of distros in my time. I ended up on Debian near the end of 1999, and part of the drive for that was the number of packages available in one centralised location. Decent package management is a definite strength of Linux (or FreeBSD) over proprietary operating systems. It derives from the freedom aspect, but means you can end up with one source for all (or most) of your software, that's compiled against the same set of libraries, with one way to track what owns what.

This may not seem like a big thing, especially if you're a hobbyist or are coming from a Windows background. Reinstalling is often seen as a necessary regular requirement. Personally I've got better things to do with my time. If I want to try out a piece of software I want to be able to install it safe in the knowledge I know exactly what files it owns and where there are. And I want it to be able to tell me what other common components it needs that I might not already have. Then if I decide it's not for me I can cleanly remove it and anything else it pulled in that I no longer need.

Don't underestimate this. This is useful on all of my machines. I can query the version number of everything installed. I can check for updates with one command (no need for every piece of installed software to have its own updater implementation). Software can share libraries correctly rather than stashing their own private copies, meaning I get bug fixes and security updates. (Yes, sometimes authors bundle even in the Linux world. Stop it.)

I'm a developer. I tend to interact with a lot of different systems, of different types. It's really handy to have access to a wide range of tools to help me with that, know that there's legally no problem with me installing them, be able to do so with a single command and, should they turn out to be unsuitable, know I can cleanly remove them with another single command. This is a definite win in the work context.

Equally I've been a sysadmin for multiple machines at once. Being able to login to each of them and check that everything is up to date is damn handy. Being able to easily install software for customers tends to make you popular too. And being able to rebuild boxes (or build additional boxes to share load) with the same setup is a lot easier with a decent package manager too.

And, to pre-empt any responses about how a lot of this is possible under, say, Windows, yes, it is. I've spent some time in the past building packages for commercial deployment using Novadigm's Radia tool. I'm aware that Windows integral package management has also got better over time. I still think dpkg/apt (or rpm/yum) is far more powerful. And, for the end user, mostly easier as well - distros are building pre-prepared packages for you, rather than you having to do it yourself like with Radia.

Russell complains about locked down phones and horrible telcos, in particular about not getting a discount on your monthly contract if you don't get a phone with it.

This hasn't been my experience, either in the UK or since I moved to the US. In the UK I ended up on an O2 Simplicity (month-by-month) plan which provided more minutes, SMSes and data allowance that I needed for £20/month (note that I only use the data for the phone, I didn't tether it to my laptop). Originally I chose this because I wasn't sure about coverage where I lived (that's why I was changing provider), but it turned out to be a pretty good deal, saving me at least £10/month over a contract that I'd have been tied into. When the G1 was launched I wasn't interested in moving to T-Mobile, who I knew had no 3G coverage outside of Belfast, so I ended up with one off eBay (received as a gift) and kept my O2 contract.

When I moved to the US I signed up to Simple Mobile mainly because I could get a SIM from eBay before I left the UK, and it was PAYG (so the fact I'd no credit record didn't matter) but still included unlimited voice/SMS/data. Significantly more expensive at $60/month than I was used to paying in the UK, but seemed to be the going rate even for a contract.

Then the G2 launched back in October. I resisted for 2 or 3 weeks, then decided it had to be mine. The G1's battery was even worse than it had been (to be fair it had lasted 2 years), and although Cyanogen provided Android 2.2 the hardware isn't really up to it. I decided to go with T-Mobile; Simple Mobile use their network, so I knew the coverage would be fine, and I figured a contract was probably a good way to help get a credit record here.

Except, the pricing was a bit weird. $200 for the phone with a 2 year $80/month contract or $500 for the phone with an identical contract but no tie in and $60/month. Er, what? I pay up front and I save $180 and I can walk away whenever I want? Ok.

As it turned out this was the smart choice. Firstly $60/month means $60/month plus taxes[0], so I was paying more than I paid Simple Mobile. I figured I could bear that for a few months to get the credit history, plus the free network unlock after 3 months. Except then it became clear that international SMS wasn't included in the unlimited SMS (it is with Simple). Most of my SMS is international. Now, T-Mobile have a $5/month bolt on to cover that, but not if you're on their flexpay scheme because they found themselves unable to verify your SSN. So I cancelled the contract after the first month and moved back to Simple. I didn't even need to unlock the phone due to the fact it's the same network (though I have now in preparation for my trip back to the UK over Christmas). Surprisingly T-Mobile didn't try and keep me by sorting out the international SMS bolt on. I guess US mobile customers are used to being screwed over (certainly the pricing suggests that).

Er, sorry, that turned into a bit of a T-Mobile rant. My original point was that all of my recent mobile contracts have been month by month, not involved a subsidised phone, and saved me money over being tied in. And even if they hadn't my experiences with the flexibility offered by not being tied in (worries about coverage, discovering the deal isn't as good as you thought) mean that I'm pretty much convinced that contract-free is the way to go anyway.

[0] Dear America, for all your complaints about VAT it's not really a lot different from sales tax and at least the prices in shops/online actually include it. Also it's the same everywhere in the country.

(This is part of a series of posts on Why Linux?)

Linux, to me as an end user, is cheap. Even taking into account the fact most PCs come with a Windows licence included it's still cheaper for me to run Linux. I paid Steve minimal amounts for my first set of Debian install CDs. These days I can burn my own netinst CDs and pull the rest over the internet. I legally have access to a tremendous range of excellent software, for nothing. All the apps I need are available without having to shell out more. How is that not awesome? I get free updates, both for bugs and also for major features. I'm not left with the option of paying lots of money for the latest and greatest, or dealing with an unsupported old release with known bugs.

The counter argument from my Windows using friends is often about how they didn't pay for their Windows updates nor their copy of Office. I'm unimpressed with anyone who tells me Windows is a better option, but is unprepared to pay for it. If you have to illegally obtain it in order for it to compete with Linux then you're not really comparing on equal terms, are you? Also don't tell me that Free software takes away jobs from software engineers and then pirate software, eh?

Cost isn't just about the money though. I've put many hours into being involved in Debian. I've provided project resources when I was in a position to do so. I've contributed to the Linux kernel. Not quite the same as paying for it, but I think does indicate that I'm trying to give back a little too. I also accept that at an organisational level the basic cost of the software licences is often negligible compared to things like hardware, training and support.

I still think cost is a compelling argument for the home user, and for decisions at an organisational level. As mentioned I realise there are issues with training and support, but I don't believe these costs are any higher than for alternative OSes. Linux also makes it remarkably easy to remotely administer machines, and perform common actions across an entire installed estate, without needing extra bolt ons from 3rd parties.

Cost doesn't provide sufficient justification for an individual desktop in an organisation that has site licences for an alternative however (and in fact running Linux requires extra work on my part to do the install and maintenance compared to allowing central IT to manage my machine). So that's not a good enough reason.

How interesting, today I learned that Derek Burgess, the PhD supervisor of my PhD supervisor (Brian McMaster) was himself supervised by Frank Smithies. With a little help from the Mathematical Genealogy Project this has helped me work out my academic "parentage".

1. Colin Turner, Queen's University of Belfast, 1997
2. Brian McMaster, Queen's University of Belfast, 1972
3. Derek Burgess, University of Cambridge, 1951
4. Frank Smithies, University of Cambridge, 1937
5. G. H. Hardy, University of Cambridge
6. Edmund Whittaker, University of Cambridge, 1895
7. Andrew Forsyth, University of Cambridge, 1881
8. Arthur Cayley, University of Oxford / University College Dublin / Universiteit Leiden, 1864,1865,1875
9. William Hopkins, University of Cambridge, 1830 (Note his many famous students)
10. Adam Sedgwick, University of Cambridge, 1811
11. Thomas Jones, University of Cambridge, 1782
12. Thomas Postlethwaite, University of Cambridge, 1756
13. Stephen Whisson, University of Cambridge, 1742
14. Walter Taylor, University of Cambridge, 1723
15. Robert Smith, University of Cambridge, 1715
16. Roger Cotes, University of Cambridge, 1706
17. Isaac Newton, University of Cambridge, 1668
18. Isaac Barrow, University of Cambridge, 1652
19. Vincenzio Viviani
20. Galileo Galilei, University of Pisa

I think I'll stop there. I found this absolutely fascinating, many notable figures, and Hopkins supervised many famous figures. It's a shame I could never hope to live up to such a line!

This is a whimsical storyline I knocked out this morning. If you don't know much about HE (Higher Education) or the QAA it may not mean anything to you. Indeed, you might need to know a little bit about the Metatron. and some other Judeo/Christian theology to follow this. Absolutely no offence is intended to anyone!

---

There was a polite knock on the door. The Metatron looked up from his desk and bade the guest to enter. And so Michael, Field Marshal of the Heavenly Host entered the room, a troubled look on his normally serene face.

“Peter says we have an unusual guest at the gates. He claims to be from something called the … QAA”, he said.
“The QAA?”
“Yes, some sort of Quality Assurance Agency.”
“And why are they here?”
“Apparently”, and here Michael paused for a moment, “they want to audit just how heavenly Heaven is...”
“What?”
“They say that there are some metrics which cast some doubt on the issue apparently.”
“But Heaven is the definition of perfection. How could anything be found wanting?”

Michael looked wretched for a moment.

“He's asking if this is so, how was it that we had... the Incident...?”
“What?”
“Well,... he's saying that if everything here is so great and perfect how come we have a … what were the words he used again? Oh yes.... a retention problem.”
“Retention?”
“Apparently yes, this is one of the metrics they use. They are apparently concerned that our “First Output” showed an “attrition rate” of a third”.
“Did you tell him about Free Will? That it's up to sentient creatures to choose their own path?”
“Yes”
“And?”
“He says everyone uses that excuse.”

The Metatron paused while he considered this.
“OK, so what next?”
“Apparently they want to see the figures to see if this is a one time “blip”, they are asking for our “Academic Plan” for the next output.”
“Oh dear...”
“Is that a problem?”

The Metatron thumbed through some documents on his desk.

“Apparently we only intend to “retain” 144,000 of our second output”.
“Out of how many?”
“Well, sort of, um, 100 billion and counting so far”.
“Isn't that an attrition rate of over 99.999%.”
“Yes,... do you think that will present a problem?”

The two angels considered for a moment, after a while, the Metatron cleared his throat again.

“Do you think we could classify purgatory as an exit award?”

After my earlier post Referrer and Comment spammers are a PITA I came up with two mod_perl plugins to Apache and an "apache level" firewall.

The reason for the apache-level firewall is two-fold.  There is no direct way for the Apache user to manipulate an iptables chain (as it doesn't run as root), and second; I was not happy with suid root access or other forms of message passing to a daemon which would manipulate the firewall for me.

Architecture is thus, in httpd.conf place the following two lines:

PerlPreConnectionHandler PGREGG::httpBLBlock
PerlLogHandler PGREGG::httpBLLog

The first tells apache to run the handler in my httpBLBlock.pm module when a connection is received (before the request has been sent by the client).  In this handler, I am simply looking for a filename matching that IP in a directory that is writable by the apache user.  The contents of the file are a SCORE:httpBL_answer:[LIST].  Based on this, the module checks the mtime of the filename is in the last SCORE days, then the firewall is in effect. If so, we simply tell apache to drop the connection.  If the file has expired, we delete the file.

The second line is more interesting, and what creates the firewall filenames. In order to not impede the general speed of request handling, processing is performed in the Logging section of the Apache process. Our module is called by apache after the response has been sent, but before the access_log entry has been written.  In our module we perform the http:BL API call and compute the above SCORE based upon the Threat* level and Age* of the API response. (* both Threat and Age are octets in the DNS lookup).  We merely discount the Threat down to zero based on the Age (0-255) where an entry 255 days old reduces the SCORE to zero.
If the SCORE is larger than our trigger level (3) then we create the firewall filename, log the entry in our own httpbl.log and return Apache2::Const::FORBIDDEN.  This causes Apache to not log the entry in the normal access_log.  Otherwise, if all is ok, we return Apache2::Const::OK and Apache logs the hit as normal.

I have a bit of code tidy up, restructure the config/firewall directory and pull some common code out to a shared module before I can release to the world.

An interesting side effect to publishing the last story out through Planet PHP and other news sources along with the Project Honey Pot image is that when browsers viewed those sources, they all asked for the image off my server. In several cases, these were known spammer, Comment spammer, and other abusers. My server then created the firewall entry blocking them before they were able to follow the links back to my server.
 
I have been reading up more on Apache Bucket Brigades in an attempt to allow the firewall filter to be placed immediately after the request has been received and allow a custom response to the browser. This may help an otherwise unsuspecting user if their machine had been trojaned. I don't mind admitting I'm thoroughly confused right now :)

This shouldn't be news to anyone - but Referrer and Comment spammers are a real pain in the a*se.  Polluting my web logs and making any meaningful log analysis problematic.

So, I now have an itch to scratch and I'm going to do something about it. I would encourage you, the reader, to do something about it too.

Firstly, get yourself over to Project Honey Pot and read up on the project.  If you can, set up a Honey Pot or two yourself. Also be sure to read about the http:BL - this works along similar lines to the DNS blacklists used for Email spammers.

Next, I'm going to write a general Apache mod_perl module which will provide integration (lookup) to the http:BL and allow the user to "action"* the abusers.  Minimally, it will prevent the normal apache log files from being polluted by diverting the log entries to a httpbl logfile.

* "action" - To provide flexibility, I'm thinking of running an external script with the IP of the abuser.  The script can then perform any action you wish. The one I'm going for is an iptables firewall block.

Comments and suggestions welcome.

Project Honey Pot has implementations for several languages, including PHP and Perl (the languages that mean most to me).  There may be an implementation for your Web application so you might not be interested in what I'm doing at all :)

I have both these phones. The HTC Desire is my personal phone, and runs Android (in this case Froyo, Android 2.2). I've been using Android for some time now, and regular readers will know I haven't been shy about commenting on its problems in the early releases. I also have, and have had for about the last month, an Apple iPhone 4, running iOS 4, for work. I've been promising some people a comparison.

Disclosure: I'm not really an Apple fan. From the moment I started to play with some in QUB, I disliked the single button, the cotton wool interface that kept you from "harming yourself", or doing anything deep. But I know that has changed somewhat, the latest Apple computers, while still having that fluffy exterior now have a decent operating system underneath. So I'm going to try to be as fair as I can be.

First a comment about the build quality. Both phones feel very similar both in the hand and in the pocket, the iPhone feels like it has a better build quality, but then you do expect solid hardware from Apple. Both phones have a button on the top to "wake" the device. Both have volume controls on the side. The iPhone has a nice feature of a slider button that mutes the device at one go; the Desire requires you to put the volume slider to zero (there are other ways, I know). The iPhone has one big button at the bottom, reminiscent of the one button mouse, and this is an area where the Desire wins hands down, with more physical buttons including the search, back button, the menu button, the home button and the optical trackball.

A note on the intuitiveness of the interface. Apple claims that their interface is so intuitive you don't need any instructions. I must say I find the Android (Desire) interface more intuitive, and when you start that phone for the first time, it walks you through the basics. Very helpful for beginners.

Anyway, time is short and some people are waiting on this review, so here's a potted comparison.

Feature	iPhone	Desire
Basic Interface	Uncluttered, but uninformative, no widgets, no live wall paper, no active folders. Dull, one size fits all. Switching between tasks and back again is inelegant. Notification of outstanding items is cleaner than standard Android.	Very rich, combinations of apps on the desktop, widgets and all the things mentioned by their absence for the iPhone. Much more personalised. Task switching, particularly the back button, is much more elegant. HTC Sense is nicer.
Phone	Disastrous. Frequently won't connect calls when my Desire will. There seems to be something else at play here and I've reported it. Both phones are on the same network BTW. It's more awkward to change numbers on the fly and many other things. But it's very pretty.	Much improved in Android over the last few versions, the ease of dialling, changing numbers is much better. Finding contacts to dial is much easier and faster. Oh, did I mention it works?
Voicemail	Fancy. Asks me to set it up every single time I turn on the phone. Recently while travelling, I couldn't pick up a voicemail because of this for about an hour, by which time I was sitting with the caller.	Not Fancy. Works all the time.
Workplace	The stock mail client is very pretty, and for example, links to appointments easily (but makes it hard to see if you are free). It has limited threading support which is really nice.	The Android exchange support is, in my opinion, superior. It lacks threading, but does have follow up support, which I take to be vastly more important.
Apps	Legendary, but there are relatively few free quality apps. For example, I struggled (still haven't) found a decent calculator (not the built in one) that is free. I find the market app rather clunky. Can't find any decent external exchange apps that work.	Many problems with the Market were fixed in Froyo, the apps available seem to be broader in nature, and many more are free (my perception). Choice of several exchange apps, more fully featured than iOS.
Software Keyboard	Simple, elegant, but frustratingly difficult to type complex content, having to change layouts all the time.	More cluttered, but actually as easy to use, better word prediction, less switching between layouts.
Battery Life	Initially winning hands down, but now hogging battery like no tomorrow, can't make it through an average day. I don't know what's causing the problem and so I'm just deleting apps all over the show.	Vastly improved over other phones, still an issue, but actually appreciating it more after the iPhone
Music	Very pretty. iTunes integration. This is also the problem. A cheap player I bought for my Daughter allows me to just dump music on it and it works. What I had to go through to get Music onto the iPhone because of my unusual setup, well, it wasn't easy. Oh, and by the way. iTunes sucks. I mean really... disastrous, but with no alternative.	Bulk device, you can just copy the music on and it works. Plays music just as well as the iPhone, in fact better because the former occaisionally and inexplicably stops. Wide variety of music players.
Video		Flash.
Web	Browser is probably prettier than Android's and allows more Tabs, seems to be slower though.	Native Flash is an advantage here too.
Notifications	Really dreadful, and a well known problem in the Apple community. Poorly handled, and when they pop up, and you go to use the phone the notification is just gone.	Elegant system that allows multiple notifications each of which take you straight to the issue. Persist, unobtrusively, until dismissed.
Calendaring / Time	Automatically setting the time to the wrong time since the clocks changed. Manually fixing this makes calender entries wrong. Setting it back to automatic makes the time wrong again.	Google Calendar back end more open than Exchange. Exchange functionality built in too. Minor quibble, cannot change the colour of the Exchange calendar. Date / Time works. Minor quibble, on a non rooted device you can't use ntp for ultra correct times. Can't on the iPhone either as far as I know.
Oddities	My laptop supplies power out of USB while it's off. I use this to charge my Desire if need be. The iPhone requires the whole machine to be on for it to charge the phone.	The power connector on my Desire seems to be a bit stretched, so if I'm not careful, it's not being charged. The Desire asks, when plugged in, whether it should charge, act as a disk, do internet tethering etc..

I'm honestly struggling to find an area where the iPhone wins hands down against its competitor. I can't think of one. I imagine if the iPhone is the only smart phone you are used to, it seems miraculous. It probably seemed that way against Android 1.1. But Android has grown up now, and it makes the iPhone look just stupid by comparison. I couldn't recommend an iPhone to anyone. Sorry.

By the way, I fully accept that perhaps when I get used to the iPhone I might come to love it more, but I'll be surprised.

One of the things I love about the Wii is the Virtual Console. It lets me play a lot of the games I grew up with without the need to sacrifice several power sockets, or swap out all the RF connectors when I want to switch console. So over the weekend I played a bit of Mario World and Zelda: Link to the Past. Awesome!

But then I downloaded Super Castlevania IV and tried to play it… “Mode not supported.”

“Uh, oh!” I guess it doesn’t like my TV, or perhaps the connector – even though I’m already using the Nintendo Composite cable. Surely the console should do the necessary work to send the correct signal?

I looked online and found a fix – but here’s the strange thing: The fix isn’t something like “Try using an old Co-ax cable,” or, “Change the Wii’s ratio to 4:3.” The actual fix is much better:

Start the game and wait for the ‘Mode not supported’ message.
Press the Wii’s home button.
Open the game’s Operation Manual.
Press the following keys together: “Z-A-2″
A chime will sound if you entered it correctly.
The game will now display on your HDTV.

…

Really?… I mean, that’s awesome! It worked, but this felt more like putting in a cheat code than solving a hardware issue. If you decide in the end that you don’t really like ‘being able to see the game’ you can go back in and disable the fix with the combo ‘Z-A-1′.

Today I read some really crufty code. The whole application is a mess of String conversions. This is because every class in the model only contains String members. Regardless of whether the data is a number, a boolean, a currency value or a date. *Everything* is a String. This makes it very hard to visually (or programmatically) parse what is happening in a piece of logical code because:

1) A lot of methods take Strings as Arguments.
2) The compiler can’t do any useful type checking. Everything is a String.
3) Things like dates and currencies can have multiple representations. Strings provide no abstraction.
4) Logic is shrouded in conversion and other housekeeping.

I don’t which moron had the bright idea of storing everything as a String. They’ve probably been promoted to principle engineer or technologist now and don’t have to deal with the stink they created a decade ago.

Of course, bad code exists. Sometimes it’s a kludge to get round a language limitation, or maybe it’s a speed hack.

This is just depressingly bad design. There was never a time when Java only supported Strings. I won’t even accept serialisation as an excuse because a) Java has been doing that for years and b) If it’s done manually then it should be done once, not peppered thoughout the business logic.

It feels as though someone read a bunch of Design Pattern books, rushed through an example implementation as quickly as possible, and released it as a polished framework.

And people just accept it like it’s no big deal. Worse, some people seem to consider it flexible design.

At a certain point on the Bureaucratic scale, you may start using Outlook for more than simply the exchange of messages: Tasks/Todo lists, Appointments/Calendar, Notes.

Too bad Outlook isn’t very good at anything it does.

Here are a couple of things I’d like to say Outlook provides to make life easier (I use Outlook 2003, not by choice):

• I flag messages and categorize them quite often. One would imagine that when a message  is flagged that it becomes a ToDo item and is added to the task list. If a flagged email has a follow up date, shouldn’t it appear in the calendar? I guess integration of different services was never a primary goal.
• I’d like folders to be able to tell me the number of flagged emails. Currently they only show total or unread (but not both). I often use folders to store actionable bulk mail. It would be nice to see how many tasks I’ve woken up to.
• When you flag email in a personal folder it does not show up un the ‘Follow Up’ search folder. I guess this is just a bug?

Apple have unveiled the shiny new iPod Nano range and they are sexy! Half the size of the previous model and sporting the same screen technology as the touch devices. Wonderful!

What I find surprising, though, is how many of my technically adept colleagues seem to be disgusted (or at least perturbed) at the disappearance of the ClickWheel.

Don’t get me wrong: The ClickWheel was a marvellous idea and worked really well… five years ago. It was a solution to the problem of providing a sleek touch based scrolling interface in the days before cheap multi-touch screens. This was also the time when the big iPods needed to fit a mechanical hard disk in the back – so there was this area on the face that needed to be filled with something eye-catching. Yes, the ClickWheel was great- I’ll continue to use one until my 30GB iPod Video decides to die.

But it’s 2010 now. We have solid state drives that can cram tons of information into a really tiny space. We have multi-touch interfaces that are just really beautiful and slick to use. Let’s enjoy the memories we shared with the ClickWheel as we lay it to rest and look forward to new innovations to come. Live in the now.

An important person in the company (all praise the company!) has just attempted to send me an updated security certificate by email.

Oh, no, no, no, my friend! Surely you know that I’m running Micros~1 Outlook as my office mail client?

You see: Outlook knows what’s good for you, and apparently, security certificates are not. Indeed, Outlook will not allow me to access this file, or even give me the option of “Don’t worry, I know what I’m doing.”

[Here's a fun fact: The certificate was *sent* *using* *Outlook*. So, Outlook will happily send files that it deems to be unsafe, but the poor sap on the other end won't get to use it.]

Now, I tend to place a degree of value on others’ time. So it pains me to have to reply to someone saying “Hi, Outlook won’t let me open your attachment. Can you jump through hoops X, Y, Z to get it to me?”

Outlook used to simply be a poor email client. Now it has become actively counterproductive.

Hey guys, so its been a while I’ve got a lot of stuff done and many core things working in Gccpy now more links on how to get up and running with it soon.

I am asking for help from you or from your friend.. anyone, all you need is:

• 5 minutes
• text editor
• basic knowledge in python

I am trying to build up a test suite for this compiler and i would love to have it all built from scratch for this project rather than taking something from CPython where i think licenses will clash and i want to avoid anything like that. So what does this all mean?

I need people to write up their favourite python snippets, but there are requirements: what you can do is use anything EXCEPT any imports. Imports are something which will be handled soon, and isn’t that important  yet for the core language implementation.

So I have set-up this mailing list and i would love it if as many of you would join: http://crules.org/cgi-bin/mailman/listinfo/gccpy

So join the mailing list or simply post a message if you prefer and we will try to remember to cc you with in. One thing i would ask is only submit a small test case if you are ok with this being pretty much un-licensed because GNU projects require copy approval for you to submit code, but we can negate this by going though me just submit this to me under a http://sam.zoy.org/wtfpl/ (Whatever you want to do license) . This just saves any pain of licensing, you will of course still be attributed! Leave your name and an email within the test-case file, since really that’s all we care about as hackers anyway we don’t care about licensing so long as we have some attribution. To give you a flavour of how simple these test cases can be lets look at one i made earlier:

Things for testing the expression syntax handling or calling functions, parameter passing like keywords parameters and positional parameters, class’s anything! Even if it is as small as one expression and a print for the result it is more than enough for a test case! The reason I want as many people involved is so i can see what features/trends matter most important to python users and then I can focus on making them extra awesome. And if it only takes you 5 minutes to write a small piece of python code and send an email while you get your name within the Gcc sources the more the better:).

When you create a test case simply send an email to the mailing list with the subject “Test Case <name>”. You can submit as many as you like the more the better  and the funnier the better…. * looks at Jezra ! @yamatt i am reusing that python while loop you sent me ages ago to freak out my processor! Thanks so much every one, watch this space in the next week for links on how to use and see your test case running in Gccpy, so you can compile your python code to an executable!

So remember the mailing list is over here. http://crules.org/cgi-bin/mailman/listinfo/gccpy

Well, it looks like Vodafone UK royally messed up the timing of the HTC Desire OTA Update yesterday.  While most owners were eagerly expecting the announced Android 2.2 (FroYo) update that HTC have been pushing out, Vodafone decided to push out a 2.1-update1 which only provides Vodafone branding, apps, a few bugs, and even a couple of "adult" related bookmarks to everyone - and leaving them on Android 2.1 (Eclair).

Needless to say - Users are not happy at all. *Vodafone appears to have moved the thread here. [2010/08/06] Moved again to here (is Vodafone trying to hide the complaints?).

In work, we (coworkers and I) now have a total of 8 HTC Desires (out of 16 people) - even two iPhone users have converted! A few have been espousing the wonders of FroYo on their phone (some had rooted, and did it themselves, others had bought unbranded phones and got the stock HTC FroYo upgrade). I was jealous(ish) and wanted it myself.

Given Vodafone's actions yesterday, it was likely that the FroYo update from Vodafone was minimum several weeks away, if not 3 months (as was intimated on the Vodafone forum - end of October) - it was also noted that the Vodafone FroYo update would include the Vodafone 360 branding and software.

Thus, last night it was time to embark on the adventure of flashing my HTC Desire to a stock image direct from HTC. I reasoned that my goal should be to flash HTC's Android 2.1 (Eclair) image to the phone, and once there, the normal software update process should take me to 2.2.

And so it began...

I knew that I needed to create a gold card because the Vodafone image doesn't let you install non-Vodafone images. This I discovered as I tried to simply apply a stock 2.2 download and using boot recovery update.zip - all attempts met with a complaint of a "Fingerprint error".

I ended up at this page:
  [TUT]Complete upgrading guide(root, unroot, flashing ROM & updates)

and followed Post #3 which details a) How to make a Gold Card* . Ignore the rest of #3
and then Post #4 - the "unrooting" guide.

You then need to find a download of the correct stock HTC Bravo WWE image ROM - I chose the second WWE ROM from this link (140MB download):
  [ROM] Official HTC Desire RUU ROMS and OTA Update URLs
  RUU_Bravo_HTC_WWE_1.21.405.2_Radio_32.36.00.28U_4.06.00.02_2_release_126984_signed.exe

Proceed through the rest of Post #4 "How to Flash ROM" with the Goldcard inside your Desire, and plugged into your PC.

The phone will take 5-10 minutes to complete upgrading (or downgrading) to HTC Stock 2.1 image. When it reboots you will have to go through all the original setup sequences you did when you first unboxed your phone.

You can then perform a Software Update Check and you should find you have a Android 2.2 FroYo update (90Mb) waiting for you.  Proceed and let it do its thing.

Once done, welcome to FroYo.

All future updates will come direct from HTC - not from Vodafone, and you won't ever have the Vodafone 360 branding rubbish foisted upon you.


* My GoldCard creation had a bit of a hiccup, in that it turns out that the 4GB Samsung card which came with my Desire does not work as a Goldcard despite formatting and following the instructions to the letter.  Trying an old 1GB Sandisk I had resulted in a good goldcard.

Aside from the goldcard hiccup - this all went surprisingly smoothly and painlessly.

Good luck.

Disclaimer: If you try any of the above - it is all your responsibility. I take no responsibility should you brick or damage your phone.

I wrote this essay almost eight years ago. I'm just republishing it in a web format.

In many eastern religions, and indeed their occidental counterparts, great emphasis is placed in reaching states of personal enlightenment or self-improvement. In Zen, this is often approached from several angles (See D.T. Suzuki [2]), but perhaps the most well known is that of zazen meditation. In this practice the mind is encouraged to have no attachments, it may move freely, neither forced to consider thoughts, not forced to leave them.

This state of mind could be described by the word mushin meaning literally “no mind”. In this state the mind is truly focused on the present moment, not anticipating the future and not pondering on the past. The martial arts of Japan have long been looked upon as an alternative, or perhaps at least parallel, means of progressing to this goal. In Budo this can be thought of as a form of meditation in motion, always acting in the present.

This prevents fearful anticipation of the future and also misinterpreting such future events. Although the idea of a budo is that of a way without destination, this goal of mushin should be constantly striven for. Perhaps an intermediary goal is that of fudoshin. This means "immovable mind" although this translation fails to carry the positive implications in this word. For me this means that the mind is immovable by outside influence, or to put it another way the mind can only be “moved” when one desires it.

As the beginner knows nothing about either his body posture or the positioning of his sword, neither does his mind stop any- where within him. If a man striked at him with the sword, he simply meets the attack without anything in mind. As he studies various things and is taught the diverse ways of how to take a stance, the manner of grasping his sword and how where to put his mind, his mind stops in many places. Now if he wants to strike at an opponent, he is extraordinarily discomforted. Later, as days pass and time piles up, in accordance with his practice, neither the postures of his body nor the ways of grasping the sword are weighed in his mind. His mind simply becomes as it was in the beginning when he knew nothing and had yet to be taught anything at all.

extract from the Unfettered Mind, Takuan Soho

So here in "The Mysterious Record of Immovable Wisdom", a letter written by the famous monk Takuan Soho to the equally famous swordsman Yagyu Munenori, Soho advises that a person with no martial experience may be better off than those who embark upon such a path. The person with no such training at least behaves naturally, even if possibly foolishly. Their mind may be affected by many outside influences, but at least they themselves do not cause it to "freeze".

When one begins training in a martial way, one often begins by learning basic skills and body movements and then, of course, waza. When attempting to refine performance of a kata or waza, one’s mind may focus on the future - in the hope of attaining the perfect form - or it may focus on the past, in the recollections of past successes or failures.

Unfortunately these moments of locked focus, the states of "fushin" or frozen mind, however brief cause the body which is forever locked in the present to be deprived of command. In severe cases then, fushin is manifested by the locking in position of the whole body.

How then can we move towards this state of fudoshin, and thus ultimately to mushin? At this stage in my journey it seems that the following are true, whether during the execution of a kata, waza or a more free action.

• One most resist the urge to anticipate the future except by observing the present. It follows that ultimately waza should not be "selected" but eventually will arise without conscious thought (mushin).
• One must avoid lingering in the past. Recollections of past encounters and their success or failure are false as the past moment will never truly match the present.
• One should not let your eyes and mind become locked on the opponents eyes, their weapon, fist or any other point. Gaze should take it the whole person as well and the mind should observe all things.

  Some teachers say that you should always stare at the enemy’s weapons, or at his eyes, or at his feet. This is not a good idea because it fixes your spirit in place and is easily read by an experienced fighter.

  - extract from the Book of Five Rings - Minamoto Musashi
  
  
• Forced aggression may cause the mind to focus too greatly on the future desire for victory and thus ironically deprive the body of the mind in the present to help obtain it. One should not delay the cut, or throw, but neither should one rush it in too great a desire to bring the future more quickly. Indeed it seems from the books "Zen and the art of archery" (see [1]) and the "Unfettered mind" (see [4]) that we wish to obtain a state in which the cut happens because it is natural or the throw happens because nothing else can happen.
• Such considerations occur outside the dojo when considering our progress along the Way or in other matters. Of course we must always strive to improve and progress, but if we are too desperate we risk placing our mind in the future and not the here and now. Similarly we must have the patience and some tolerance about our state of current imperfection. Focusing too much on the failures of the past denies us the chance to observe ourselves in the present to make us better in the future.

These comments are of course, immensely subjective. However these thoughts map my current way of thinking a short way into a long journey and at this time it seems the best way for me to continue to refine myself.

References

[1] Herrigel, Eugen: Zen in the Art of Archery, Arkana 1988.
[2] Suzuki, D.T.: Zen Buddhism: Selected Writings of D.T. Suzuki, Double- day 1996.
[3] Musashi, Minamoto: The Martial Artist’s Book of Five Rings, (trans. Stephen F. Kaufman), Tuttle 1994.
[4] Soho, Takuan: The Unfettered Mind, (trans. William Scott Wilson) Kodansha 1995.

For many years now, I've been aware of the Cthulhu Mythos, since it infuses lots of popular culture. Indeed, Dread Cthulhu was recently blamed for the horizon oil spill, a theory that I rather enjoyed.

Despite this, I hadn't actually read any of H.P. Lovecraft's books directly. Last week and this, I decided to try out an ebook reader on my phone, specifically the rather excellent Aldiko for android. It's a nice application. I downloaded two of Lovecraft's books; "The Call of Cthulhu" and the "At the Mountains of Madness".

Lovecraft has been, mostly after his death, immensely influential, with references to his work appearing in many modern authors' work (e.g. Stephen King, Neil Gaiman). He is famous for the idea of "Cosmic Horror". The idea that the world is not truly rational, and understandable, but nightmarish, chaotic and hostile or at best indifferent to human nature.

I found "The Call of Cthulhu" rather unpleasant to read in parts, not because of the supposed horror, but because of the unpleasant racism implicit and explicit within the book. It does need to be remembered that everyone is a product of their times however, and I find the implicit racism difficult to accept in the works of J.R.R. Tolkien too. But the book was interesting.

The second book, "At the Mountains of Madness" was more mature and less objectionable.

But I couldn't entirely share the horror of the protagonists. Some of their discoveries were genuinely horrifying (usually acts committed by human beings) but much of the rest would be wondrous. They are horrified by the discovery of non-Euclidean geometry in architecture. Non Euclidean geometry does not frighten me, and indeed Lovecraft understood it wasn't the true nature of the universe. But seeing clearly non Euclidean architecture in a "normal" setting is wondrous, terrifying perhaps briefly, but not horrific. Much of the horror just comes from the primitive frame of reference, either scientific, or religious or both, of the protagonists.

And this is the bit I really can't understand, since Lovecraft was a failed astronomer, so he would have known that the universe is, to the best of our knowledge, indifferent to our existence, and that the laws of physics themselves could see our annihilation in so many ways. Indeed, apart from a (probably literal) Deus Ex Machina solution, every scientist knows that the ultimate future of humanity is doomed... Is that "Cosmic Horror"?

By the way, since the world is Sherlock Holmes mad at the moment, I recommend A Study In Emerald (PDF), a Hugo Award winning short story that is an interesting cross over of the Sherlock Holmes and Cthulhu mythos written by Neil Gaiman.

Have a look at this documentation for a VerifyEvent in SWT:

void VerifyListener() {…}.verifyText(VerifyEvent e)
Sent when the text is about to be modified.

A verify event occurs after the user has done something to modify the text (typically typed a key), but before the text is modified. The doit field in the verify event indicates whether or not to modify the text.

Wait, what?

What we basically want is a simple predicate function that returns true if the Contents of a Text Field are still valid after a change. What SWT gives us is this awkward event model where we have to modify the event that has just been generated. (In my opinion, events should be immutable, since they represent a historical occurence.) Next SWT will have a peek in the event to see how to proceed.

SWT is full of weird bagbiters like this.

From all the work i have been doing on compilers recently i have started to realise a lot of people won’t know how to compile up GCC or even how that works. Since how are you meant to compile a compiler written in a language that itself implements. Well the technique is called bootstrapping. But this is something people try and think its more complicated that it usually is. To understand how it works we have to think how was GCC built; well its as simple as it NEED’s a C compiler . But how did that first c-compiler get there in the first place, well it goes back years of people made the first c compilers in Assembler and then it bootstrapped up!

Within GCC when you start the build we bootstrap 3 times, which means we build it 3 times but why you ask? The generated code for the first compiler run, is by the host compile so the compiler itself cannot be trusted, then the 2^nd is done with the newly built compiler and the 3^rd for good measure to guarantee we’ve compiled with the new compiler and GCC itself should be running off its own code. Then you can look at something like the GHC the Glasgow Haskell compiler which is written in its own language, but the thing they don’t say is that it is written in a some-what simplified dialect of Haskell and bootstraps off a small amount of C code if you don’t already have a Haskell implementation already present. Which is similar to the bootstrapping languages of GCC which use C90 to maintain language portability.

Then you can look at something like what if someone writes a new operating system how do you port a compiler and libc over to the new system. Well this is done by a mixture of cross complication and complicated kernel work which can take quite some time to get going correctly.

But I digress as usual now onto a little tutorial how do we compile GCC well yes it uses the ./configure, make, make install build system. But it doesn’t quite work like this so we do it slightly different.

First to run experimental I prefer to use git:

# to get my gccpy work use this git repo it merges with gcc master at least weekly into master

$ git clone git://crules.org:gcc-dev.git

# or run the gcc git mirror:

$ git clone git://gcc.gnu.org/git/gcc.git

Now compiling time:

$ aptitude install libgmp3-dev libmpfr-dev

# GCC now requires libmpc which you may need to compile and install from:

# http://www.multiprecision.org/index.php?prog=mpc

$ cd gcc

$ mkdir build

$ cd build

Useful configure options

$ ../configure

–enable-gold | use the GOLD linker

–disable-bootstrap | useful for gcc developers not recommended for users or for your first build

–enable-languages=c,c++,python,gccgo …

….

–with-gmp-lib=/usr/local/lib

–with-mpfr-lib=/usr/local/include

-with-mpc/mpfr …. the same optiosn to point to different library locations.

Now to build:

$ make

$ make install

And you should be good to go. We have to do our build in a separate directory due to the size of GCC some components don’t like being built directly.

I was having an enjoyable early Summer BBQ chat with Mark Kerr and Jonathan (Noodles) McDowell a few weeks ago. We know each other from the days of Fidonet's greatest glory. Fidonet still exists, but back then the internet did not have a presence in normal people's lives, academics I worked with at Queen's only selectively had email access, and that through the dark arts of Kermit. Back then a few of us ran Bulletin Board Systems, over plain vanilla modems with all sorts of speeds, 1200 bps transfer rates were common.

The systems were held together as nodes in Fidonet, and the hundreds or thousands of users on each system could send a form of email to other users in the same system, or even users across the world. Every message was piggy backed on the daily or nightly phone calls from one system to a neighbouring system. Mark and I would call some odd systems, like that of Joaquim Homrighausen and Mats Wallin in Sweden, and we would happily carry any mail going for Sweden or coming back.

Everyone was sharing a bit of the cost. As such, it was considered the height of poor etiquette to reply to a message tens of lines long, quoting the whole message with a "me too" at the top or the bottom. You were expecting someone else to pay for transmitting the whole original message again, just for your extra words. You were expecting in echomail (the Fidonet equivalent of newsgroups, or mailing lists) that hundreds or thousands or tens of thousands of systems would store all that information on all those hard disks.

No, it was correct to trim a few salient lines from the original, and put your reply below it. That's enough for everyone to see what is going on. Every mail editor in existence showed the quoted text either with a ">" character to the left, or sometimes a "CT>" to indicate the initials of who wrote it.

These trimmed, quoted conversations allowed the full flow of the conversation to be followed by anyone without referring back to every word uttered by the original poster, and showed the coherence of the thread in a way that is hard to do in any other way. Google Wave is perhaps a modern attempt at the same idea.

But than people grew indifferent to bandwidth because there was so much, so they never bothered to trim their replies. You can expect in a modern mailing list to find each reply containing complete copies of all the previous correspondence to that point, which is already stored in the list, shown in a nice threaded view in your editor. Some new email programs (I'm looking at you, Outlook), made it almost impossible for people to do anything other than "top post", because they don't easily show quoted text.

And here I am, an old dinosaur who still tries to stick to the old ways, because I believe it's easier to read properly quoted replies... but increasingly, you can't have conversations with people with braindead mail software because they just can't do it. And they haven't even seen it before, they can't understand it. They complain when you don't top post your replies. Like the stereotypical cantankerous codger sitting in his porch watching the kids mess on his lawn I want to tell them that no, they are the ones doing it in the stupid new fangled way. We were here first.

So maybe we should just give up... those few of us left, but then I was reminded (by Noodles) that now email is being shipped around mobile devices all the old arguments of bandwidth and storage are increasingly relevant again. So I ask the open question, is there any point in trying to stick to the "old ways"? Thoughts on a postcard please.

In days gone by, any computer guy worth his salt had a collection of boot floppies, 5.25″ & 3.5″, containing a mix of MS-DOS, DR-DOS, Toms Root Boot & Norton tools. These days passed and the next set of essentials was boot cd-r, containing BartPE, RIPLinux, Knoppix etc. People quickly switched to carrying these tools USB sticks, smaller, easier to change, great when the dodgy PC you were trying to breathe life into supported USB booting.

I think there’s a better way, based on the last 3 days of hell spent setting up what should have been identical touchscreen machines (no cd, slow USB interfaces)

Your new toolkit is a cheap laptop, with a big hard disk, running the following:

1. Your favourite Linux distro (I’ve used Ubuntu for this laptop)
2. tftpd, dhcpd & dnsmasq setup for PXE booting other machines from this laptop (FOG uses dhcpd for all it’s automatic DHCP magic, use dnsmasq for simple local DNS, required for Unattended)
3. FOG Cloning System
4. Unattended Windows 2000/XP/2003 Network Install System
5. CloneZilla PXE Image (for good measure)
6. RIPLinux PXE Image

Why?  USB booting stills seems troublesome, installing Windows from flash seems very slow.  Nearly everything supports PXE these days, if it has a built in ethernet port, it’s pretty much guaranteed to support PXE booting.  There is nothing like the feeling of being able to image a machine into FOG over a 1Gb crossover cable in a matter of minutes.  Got everything working? image it and walk away, safe in the knowledge that if somebody comes along and breaks things, you can image it back in minutes, instead of having to do another clean install and build all your updates & software back on top.

There’s a little bit of plain in getting all of separate packages to run from the one /tftpboot/pxelinux.cfg/default, but it’s just a matter of careful copy & paste from the canned configs.

just incase anybody was interested, the slides from my BarCamp Belfast talk are here:

• Google Docs
• SlideShare (you can download the PPT from there as well)

The more I go through this cycle the more it amazes me that this is the development paradigm that won out. That back in the golden era people sat down and decided that this was the model of the future.

For large applications it’s a major pain – much time spent compiling, starting up the program anew and returning it to the state you need in order to test your latest fix. Even for a fairly trivial change or a bit of experimentation, this is a huge timesink.

And yet there is an alternative approach – incremental compilation, à la Lisp. When the Lisp environment hits a bug it pauses. It shows you the location of the bug, it gives you a full stack trace at the time of breakage. You can inspect just about everything. Here’s the clincher: you can fix the code while the debugger is waiting. Then you can load that fixed code right back into the running program and continue on your merry way.

Let me repeat that last bit: You compile only the code that needed fixed and load it into the running program. The running program then continues using the new code. No stopping, compiling, running, getting back to where you were, debugging.

This is special. It’s not even that new. Lot’s of hip, young, dynamic languages are attempting to copy it. Let’s now, in 2010, make an attempt to catch up to the technology of 50 years ago. We owe the founding hackers that much, right?