Crafting KML from Garmin GPS Data

Note: This entry has been restored from old archives.

This entry explores my journey from getting a Garmin eTrex Vista HCx GPS unit to generating a KML file that shows information I’ve gathered on my GPS via the Google Maps website (and, in theory, also Google Earth). It’s only lightly technical, more a review of the process than an instructional reference. I expect, and hope, there are more efficient ways to go about doing what I’ve done here.

I mentioned a little while ago that a recent addition to my toybox is a Garmin eTrex Vista HCx (AmazonUK). I’ve used this a lot over the last few months and am very happy with the purchase. The primary reason for having the device is that we like to ramble around the countryside a lot. A stint of Rogaining and Orienteering in high-school left me with very good map navigation skills, which have served me well. But when trekking around, especially moving at speed, referring to a paper map regularly can be a bit of an encumbrance. Enter the GPS, it fits in my hand and always tells me exactly where I am. There’s one important thing to be aware of though: it’d really not be so much use without a good map set.

The map set I have is Garmin TOPO Great Britain (AmazonUK). I don’t have much to compare it to in GPS-land. What I can say is that it is better for walkers than all the online mapping I’ve seen, except for those that use present sections of the Ordnance Survey maps (like MultiMap used to at high zoom levels, but it just seems to use Google Map data now). Additionally, this Garmin map is actually derived from the Ordnance Survey data. This is a huge bonus for walkers. Any UK rambler should know the greatness of the OS Explorer and Landranger maps sets. With the TOPO Great Britain maps on your GPS you’ll have the great feature that contours shown on the GPS unit are the same as those on the OS maps — making it very easy to place yourself on an OS map.

An OS map you say? Yes, I don’t consider the GPS to be a complete replacement. First, in deference to screen size, the maps are nowhere near as detailed as the OS maps. They have roads, water features, contours, and some terrain information — but they lack all the additional details found on the OS maps. The most important thing lacking on the Garmin map is full coverage of all the footpaths, bridleways, and byways marked on the OS maps. That sucks quite a bit.

The other obvious great features of this GPS unit are that it can store the path of your travels (you can save several tracks, I did this to map ski slopes and lifts individually in Ylläs), and you can mark waypoints for locations of interest. Then you can move this data between the unit and a PC. Walking geek heaven!

It isn’t all plain sailing though. Garmin sell their GPS units, and then want to make a ton more money on maps. As it comes the GPS isn’t a lot of use for walkers, it has only main roads loaded. So as well as spending more than 200 quid on the unit you’ll need to spend a further 100 quid for the UK maps. Ouch! There are a lot of complaints to be found on the web about this particular point, people don’t realise that the maps the Garmin units come with are rather crap. This seems to be a factor in a lot of negative reviews for the various eTrex units.

The eTrex Vista HCx I got is top of it’s line, you can spend a little less by getting one without a barometric altimeter and compass. Such as the Garmin eTrex Legend HCx, or one of the Cx models which are slower and less accurate than the HCx ones (but have longer battery life). Garmin actually provides a nice feature comparison tool on their site, here’s the eTrex models with SD card slots compared. The altimeter and compass are features that I’m a bit iffy about anyway, after three months of use. The altimeter needs regular recalibration and since you almost always have a, more accurate, 3D GPS lock it is redundant, and the compass doesn’t seem as stable as a “real” one.

You’ll also need to spend a little extra on a microSD/transFlash card. You don’t need a huge one though, I’m using a 256MB card and it fits the entire set of maps covering greater London in a fraction of it’s space. You could probably fit the entire map set in a 1GB card (a total guess). There’s not much point buying a new one that’s smaller than though, they’re as cheap as chips (har har) now. I picked up a 1GB one for about 8 quid (they’re 6 now) and put it into another device to free up the 256MB card.

The other stormy waters are around their MapSource software (“free” with the unit, thankfully) and their documentation is pretty crap. It’s isn’t abysmal, but as someone extremely familiar with the world of software my opinion is that it is firmly in the massive ranks of the “mediocre”. Of course, I’m also a Linux geek so using their software has an added disadvantage for me. I’d have to reboot into Windows to use it, maybe it’ll work in Wine? Maybe I’d rather be doing other things with my time?

Luckily for us Linux dweebs there are plenty of industrious monkeys out there pushing their valuable hours into great software! The biggest and hairiest monkey is Google. Their online maps setup is geek heaven, and Google Earth is software bling at it’s finest. Now, Google Earth can supposedly hook up with my GPS unit and make sweet love… if I pay for the non-free version that is. I don’t have anything against paying for this, but have refrained thus-far since I’ve had terrible problems getting Google Earth to even work on my Ubuntu or Debian systems. sigh

So, I need a bridge. This bridge is given to the world in the form of gpsbabel. It speaks several map-data and GPS-device languages fluently! Brilliant!

What I typically want to do with my GPS data is push it from the device to a format I can use to display the data on the Google Maps website. The desired format is Google’s “KML” (tutorial/examples), the same file format that drives Google Earth.

For purposes of demonstration the rest of this post uses my GPS unit with a load of tracks and waypoints on it I saved while I was in Finland, the goal is to get a neat map of the area I visited out of the Garmin and onto Google. I had other data on the device though, so I chose a lat-long point from one of my Finland waypoints and only grabbed data within a 200 mile radius of that point. To do this we employ the ‘garmin‘ driver to suck the data, the ‘radius‘ driver to filter out unwanted points, and the ‘kml‘ driver to poop the data out to a nice KML file:

:; gpsbabel -i garmin -f /dev/ttyUSB0 -w -t 
            -x radius,lon=24.171808,lat=67.604049,distance=200 
            -o kml,points=0 -F gpsbabel.kml

Note the ‘points=0‘, if you don’t do this you get a “waypoint” for every point in your tracklog. If you don’t need these it is best to avoid them, my KML file was 137kB without the points and 34MB with them, ouch!

Nothing is simple though. Now I want to prettify my KML. This involved going through by hand and editing the marker names and descriptions, and adding in CDATA HTML sections to embed links and images for waypoints. The former is mostly necessary since the GPS doesn’t let names be longer than 12 chars, not enough, and the transfer maps non-ascii characters (i.e. Unicode) to ‘$’. Note that for names you can use XML-permitted entities like & but not most HTML ones, for ä you can’t use ä you’ll need to use Unicode.

Now I want my own waypoint icons and path colours. This requires removing the styles set by gpsbabel and defining a set of your own. For example I want a style for markers that show hotels, I define this:

    <Style id="waypoint_hotel">
      <IconStyle>
        <scale>1.2</scale>
        <Icon>
          <href>http://maps.google.com/mapfiles/ms/micons/lodging.png</href>
        </Icon>
      </IconStyle>
    </Style>

There’s a useful link for the “standard” Google Maps icon set here.

I also want to define my own line widths and colours for different classes of track. Good to distinguish between walking routes and bus routes, especially good in this case for the different grades of ski slope! Here’s a sample line style:

    <Style id="track_ski_easy">
      <LineStyle>
        <color>aaff3333</color>
        <width>4</width>
      </LineStyle>
    </Style>

Note the <color> is not as the HTML-familiar would expect! The hex fields are ‘aabbggrr’, that’s aa=alpha, bb=blue, gg=green, rr=red. So a 50% transparent cyan line is actually 80ffff00. Such a simple thing, yet it breaks my rrggbb indoctrinated brain!

Oops! Height and distance data embedded in KML description fields by gpsbabel is in miles and feet, I hate miles and feet! Maybe gpsbabel can be asked to use metric units but I don’t see an obvious way to tell it to do so. (Update: Thanks to the “Chief (GPS)Babel-head” (see comments) I now know that the ‘kml‘ driver has a ‘units‘ argument, i.e. ‘units=m‘ — I didn’t RTFM well enough.) Anyway, I’ve just put loads of effort into customising my KML, so time to do it with perl, yeah, sure, ewww. shrug

perl -ne 
  'if (/(d*.d*) mi/)
      { $x = $1 * 0.3048; s/$1 mi/$x km/; print $_; }
   else
      { print $_; }' 
  YllasAkasSkiHoliday-26.kml > YllasAkasSkiHoliday-27.kml

It took me a whole evening to work through all this and get my final KML. I hope it’ll be easier next time around. It certainly isn’t for everyone.

The best way to describe all the details is by example, so check out my full Ylläs Ski Trip KML for the gory internal details, and here’s a link to my Ylläs mapping data displayed by Google Maps. The iframe to the right (assuming you support iframes) is zoomed in on the part of the map data showing the ski slopes and lifts that I mapped. The dark grey lines are the lifts (recycle signs mark the bottom of the lift), the green (very easy) and blue (easy) lines are the slopes. It’s cool how my weaving back and forth (at speed) on the slopes is captured so well by the GPS!

In conclusion:

  1. It might look like horrid process, it might have taken a lot of effort, but I did enjoy it.
  2. That said, I hope that Google Earth can do this WYSIWYG style or something. That’d make this sort of thing much more friendly.
  3. Joy, one more dependency out of the way so I might get that write-up of the Yllä ski trip finished.
  4. Don’t forget than you can use gpsbabel to upload data from KML files to your GPS too!
  5. I wanted a topo map of Finland for the Garmin, I found it, but it makes the GB Topo map look cheap! At least you can buy it in parts, at the link given it is 99 euro per CD, and Finland is covered by 6 CDs.

Animeats

Note: This entry has been restored from old archives.

Hm, before Mary posted this I’d never even considered that someone old enough to talk wouldn’t know “Meat is made from animals.” After reading this story, and the one linking to it, it begins to sound normal. I didn’t grow up on a farm, but I did grow up in a restaurant — so maybe my world was different. After all, in butchers and the like it’s normal to see very animal-like carcasses hanging around, literally. But even thinking of the humble whole chicken found in most shops, how can anyone mistake that as something that wasn’t once walking around? Then again, I’m not four years old.

Maybe it’s an American, land of plastic food, thing? Or a city thing? But even in Sydney the animalness of meat seems clear. In inner Sydney butchers (the good ones) carcasses are hanging, and chicken feet are on display (OK, maybe western Sydney for the latter).

I call beef “cow” and pork “pig” and have for a very long time I think. We had laying hens back home and the link to what was on our plates was unmistakable, I’d have thought. Only once did we try to eat one of our own chickens… that was mostly because I wanted to. I killed it and, IIRC, plucked it (Mum might have) and Mum cleaned it. It was baked but, while tasty, turned out to be rather tough. His name was “Elvis”. My sister was rather upset, you see, I killed the wrong rooster. I killed the one she’d named. (But she never went vego!)

We didn’t try to repeat this experiment with other excess roosters I killed. They fertilised the occasional lucky tree. One rooster had to be killed twice, I stuffed up the first attempt. For the brief period between his killings this rooster was dubbed Lazarus. This is the time I learnt that it was better to use the axe than try to break the neck (there’s a technique to that that I didn’t know at the time). In a note of defence, to the person keeping laying hens roosters are an inconvenience, one is OK (you need a succession after all) but they can be mean to the hens and are noisy.

I hope that in the future I have the chance to do it properly. I’m better informed now, I know that a rooster as old as the one we roasted (in it’s second year) isn’t right for roasting, but is good for, say, Cock-au-Vin.

Django Forum Software

Note: This entry has been restored from old archives.

Update 2008-02-25 20:24: I’ve only made small progress on examining each of the options below as I haven’t had much time for personal projects in the last couple of months. An important point to note is that some of these didn’t work with Django-trunk, so certainly check that if you’re not working with the development version (conversely, some might not work with 0.96 I guess.) Note, from the comments, Antti Kaihola has created a “Django Forum Apps Comparison” on the django wiki, it’s well worth checking that out as it’ll be more complete and more up to date than my list.

I’ve been toying with a community site recently. The logical starting point for the code behind it is discussion functionality. The idea is to take some existing code that does “forums” well and use this as the kernel that the community site is built around. Nothing is ever so easy though, the ‘net and developer communities have grown so huge that we’re not talking a needle in a haystack but finding the right needle in a haystack full of needles. There’s a scattering of good needles, but a lot of blunt of downright broken ones.

I’ve locked myself into Django now, but even having already determined the web framework to use (i.e. choosing a smaller haystack) the task isn’t trivial. I think that a general issue here is this exact experience I’m having, people give up eventually, try to spin their own, get big ideas, publish their half baked code, and make it just a little harder for the next person trying to find pre-existing functionality. It doesn’t help that Django’s relatively young, I don’t make life easy for myself sometimes!

Here’s my small attempt to work against the trend. I’ve gathered a list of the Django forum/discussion software I’ve found and associated each project with any further useful information that might help the decision of which to use. I’ve filtered out anything that was clearly broken, but don’t know the level of completeness of everything listed here.

Also, I’m sure there’s other projects out there I’ve missed! I’ll update this page if I find anything new.

I’m not recommending any of these, maybe that’ll come later once I’ve actually decided which one to work with.

Ross’s Django Forum (the django-forum)

“Simple Django Forum Component”

This seems to be the most linked-to Django forum software out there. It seems basic, but I haven’t tested it or seen a demo. I’ve seen various blog comments commenting on either it’s “lack of features” or “ease of installation and use” — how many “features” do people really need?

Jonathan’s Django Forum

Almost got this one mixed up with django-forum above. At least it has a demo site! The demo looks like it has all the basic expected components working.

counterpoint

“forum written with django”

Very little information about this out there, the code is available though. I’ve seen a comment on a blog post from Nov 29th 2007 that says “missing much functionality”, it’s a young project though and that was a month ago.

snapboard

“Python Forum/Bulletin-Board for Django”
“SNAPboard: S(imple), N(imble), A(ttractive), P(ython) board”

In a way this one seems to do the best job of selling its self. The Google Code frontpage and wiki (documentation) are good. However activity on the project has dropped off and a new maintainer has come on board recently (December 11th according to a forum post).

Django Discussion (django-discussion)

“A generic discussion application for Django”

I’m guessing this is the same author as the second item in this list, pretty unlikely coincidence otherwise? Anyway, the code seems to be different, so a different project?

  • Google Code: http://code.google.com/p/django-discussion/
  • Owner: Jonathan Buchanan (Same author as the second forum in the list I’m guessing, but the code seems very different.)
  • Initial checkin: 2007-03-15
  • Latest update: 2007-12-08 (only 9 changes since creation, but recently active at least)

Diamanda “MyghtyBoard”

“Diamanda Wiki and Forum”

Diamanda isn’t only a “forum”, it’s a site-builder toolkit that contains a complete “forum” subcomponent called “MyghtyBoard”. The forum seems to possess all the expected features. Development started quite some time ago and was regular up until early 2007, reached a degree feature completeness maybe? A recent “bug report” is “rewrite code for better look and use” with the response “I will in time :)” (6 days ago).

Sphene Community Tools

“Django Forum Application and Django Wiki Application”

Like Diamanda, this is a toolkit that happens to include a forum. I’m assuming that the forum on their site represents a demo of SCT in action. This project looks like the most active of the lot and seems to be quite well documented.

ENDE

That’s the lot for the moment.

Cinnamon Square on Church Street, Rickmansworth

Note: This entry has been restored from old archives.

Cinnamon Square
Cinnamon Square

Coffee House: Cinnamon Square
Address: 9 Church Street, Rickmansworth, WD3 1BX
Rating: New Orleans equivalent (Sydney-Coffee Rating System)
Website | Map

Rewrite 2007-11-30: The espresso improves and I partake of it more often, a general re-write.

It isn’t entirely fair to rate Cinnamon Square as just a “coffee house”, their raison d’être is given by their catchphrase: “the theatre of baking”. That said, they have the distinction of providing the best espresso in Rickmansworth so I feel Cinnamon Square belongs here since they’re “my local”. Unfortunately “best espresso in Rickmansworth” is not, on it’s own, a great qualification. The competition is generally atrocious, although a couple of the Italian restaurants serve an excellent shot (but are not really accessible for the causal espresso).

The gory details: A Cinnamon Square espresso is high standard but not brilliant, rating at New Orleans to TBA equivalence. Cinnamon Square even comes close to filling the role in my life that both filled back in Sydney, being a short walk away from where I live and where I usually work. We have espresso at Cinnamon Square every Saturday we’re in town, when I’m working from home (permanently these days) I pop in once or twice every day, if they were open on Sunday it’d be every day of the week! Alas, the best local coffee place doesn’t follow New Orleans’s virtually “always open” hours.

Update 2007-12-30: Cinnamon square is now open on Sunday!

The length of the pour is usually appropriate but with too-frequent “fill the cup” efforts, and crema is normally full and firm. The coffee tends to the sour-bitter ends of the spectrum, but certainly not far and it is quite good. I don’t know the origin or age of the roast but it is fresh ground (the least you should expect these days). There’s potential for truly excellent espresso here, possibly just some grind, machine, and roast tweaks away. The most significant problem is barista training, which is usual for places that aren’t primarily coffee houses. Sometimes the head hasn’t been packed well enough and the volume of the pour swings between just-right and full-up (luckily the demitasse are small so full-up isn’t as bad as it could be).

The espresso covered, I can’t finish without mentioning the pusscakes[1] Their namesake product is evilly delicious, they’ve even won a “great taste award” for the “Sweet Fermented Bun” (aka “Cinnamon Square”). I have a hard time resisting these every time I go into the shop, especially since I love cinnamon, but alas a ball of sugary starches isn’t going to work out on my nutrition spreadsheet. The Cinnamon Squares keep good company with a selection of danishes, cupcakes, and other delights — we’ve never had a dud. They do well in the savoury department too, with beautiful breads and a range of lunches, the goat-cheese focaccia is excellent (but beware: their focaccia’s are huge). We don’t eat much bread but when we do fancy some this is where we go, they do a good range of large and small loaves and bake regularly.

If you’re in Rickmansworth and have a hankering for an espresso you can’t do better than Cinnamon Square, and you absolutely must try their namesake at least once. It’s also worth visiting just to see the cute little heritage-listed 500-year-old building they’re in — where even I bang my head on a padded rafter and can smugly think to myself: “Ho ho ho, I’m so tall.”[2]

Cinnamon Square, be there or be without a square!


[1] Pusscake: A term I picked up in my youth when labouring for a paver. Often I would be sent to the bakery “for pusscakes”, this pretty much meant anything sticky and sweet, but especially those containing cream and/or custard.

[2] I’m told that this reflects the fact that the average height was significantly lower 500 years ago, and you do find lintels low enough for me to bang into surprisingly often in old buildings in the UK. I’m only 5’9″ barefoot.

Neronet

Note: This entry has been restored from old archives.

Coffee House: Caffé Nero, Rickmansworth
Address: 80 High Street, Rickmansworth, WD3 1AQ
Rating: Below even the depths of Gloria Jean’s (Sydney-Coffee Rating System)
Chain Website | Map (Hey, why’s the Ricky satellite imagery suddenly a decade old?! The building I live in is a sandpit!)

Update 2007-12-30: Very recently a sign appeared in the window of the local Nero outlet informing passers-by that Nero is switching over to BT OpenZone. I’m not a huge fan of BT OpenZone but they are my roaming wifi provider because everywhere I’ve been (US, Sydney, Perth, UK, throughout Europe) I can usually find a place that my OpenZone account works (almost every hotel I’ve stayed in uses a provider that partners with OpenZone). Of course, OpenZone aren’t the only provider to have extensive worldwide partnerships like this (T-Mobile do, and most APs in Europe I use are actually T-Mobile). Anyway, the important point is that OpenZone has a far more convenient pricing structure than “Surf and Sip”, importantly this includes a no-upfront-fee pay-per-minute account type. Even though minutes are expensive here in the UK (why? no idea, because they can be I guess) this makes “Neronet” far more useful for the casual low-frequency user. Now, they just need to elevate the average quality of their espresso above “chain store”, hah.

The only “‘net Café” in Rickmansworth is a Nero outlet. Even though the coffee is pretty terrible I’d be happy to sit here for an hour or two and tap away on the laptop, mainly because the chairs are comfy. There’s a big problem though: the cost. A day-ticket costs 10 quid, which is the cost of about 5 coffees and is the lowest price ‘net access ticket you can get. A month costs 40 quid (twice as much as I pay for my home 8Mbit ADSL and telephone combined), and a yearly access ticket is 200 quid. The access provision company is “Surf and Sip(TM)” and, on the prior-to-payment café web pages, I can’t find any listing of the outlets where I can get connected to them. I’d bet it’s probably only available in Nero outlets, and outside of Ricky I never go to Nero.

Coffee notes about Nero in Ricky: If you get the right person you can get a barely drinkable espresso, that’s one staff member in about ten. I only drink Americanos here, watering down bad espresso can make it not insult my mouth at least. I used to have coffee here most mornings (simply to get the caffeine fix) but now that I’m working from home I go to Cinnamon Square instead. On the Sydney-coffee-rating scale this place is below Gloria Jean’s. They do have some decent panini though, so on Sundays Kat and I tend to have coffee and a panini for breakfast here (there’s not really anywhere else to go).

I’d really rather have a Starbucks in town, the coffee is a little better and the ‘net access is a little cheaper (but still not very well priced).

I’m almost tempted to try the local Wetherspoon’s pub for ‘net access. The coffee will probably be undrinkable, but they give you 30 minutes of free ‘net access with each drink you buy (and a coffee only costs about a quid there). But I can’t really bring myself to step into a pub before midday, even one with coffee and a breakfast menu. Also, the Penn is a pretty bodgey chainpub that I wouldn’t normally wander into at any time of day.

It doesn’t help that it is Sunday and the only things open are Nero, the Penn, and an Italian place we don’t go to.

So, 1.5 years into living in Ricky and Sundays still suck and (legally sound) out-of-home ‘net access is still a myth. I thought it might have picked up a bit by now. Unfortunately this is a town of rich semi-to-fully geriatric professionals (many retirees I’d guess) and breeders… so there’s probably not a lot of market for a bit of modernisation.

If it fit into my visa provisions I’d seriously consider trying to pick up a little café on the high street that shut down a while back. This Nero place is absolutely packed, the coffee and food are both dull but there are no other options (hey, even I’m sitting here).

I wonder about WiMax. They say it’s long range, if I ran an AP from my balcony what sort of cover could I get in buildings. How much does line-of-sight matter? How much does WiMax gear cost?

Bad Weather

Note: This entry has been restored from old archives.

The last two days have been somewhat joyous in a less than traditional sense. Two whole days with all computers shut off! OK, so not that much different from our recent holiday without work/computers, but more relaxing.

The reason for the title is twofold, firstly the weather here really is rather shit. It’s England! What should I expect? Christmas day was chilly and wet, at least on boxing day there was a little sunlight. Dreaming of a white Christmas around London? Not much chance these days it would seem. This is my third in the UK, the first was white thanks only to a heavy frost, there was a little snow around the period but it wasn’t so cold that I didn’t spend the day on my bike in Wendover Woods. Last year it was cold at least. This year it isn’t even chilly, there isn’t one 0 or sub-zero day predicted in the entire last week of the month! Today has a predicted minimum of 9 — I can quite comfortably wear just a thin t-shirt under an unbuttoned jacket. Oh well.

The other bad storm is one of 2007’s old favourites, the Zhelatin/Storm/Nuwar “worm”. After somewhat of a lull in seeing emails from this network I suddenly got one on the 23rd, as I mentioned on Monday.

This turned out to be the first of many as the network pumped out a full-scale assault capitalising on the jovial season, both Christmas and New Year. Taking advantage in two ways I think: 1) people probably are sending a lot of stupid email right now so it may be more likely that people follow the evil links, 2) A lot of people, including those in the security industry and the IT-shops responsible for maintaining corporate security, are on holiday so the “good guys” may have a slower response time.

The latter point is worth some thought. I’m sure it has been discussed before: computers don’t have holidays, crims take advantage of holidays, most normal people let their guard down on holidays. Good news for botnet herders. As I mentioned earlier in the week the malware payload wasn’t detected by any of the large-market-share AV engines, the biggest player to detect some of the samples I tried was Kaspersky (finding accurate market-share figures is difficult, suggestions on the net for KAV are between 5 and 1 percent). As has now been clearly established, I’d think, the malware writers test against the biggest AV engines. We can get a good picture of which engines they’re testing with by rounding up as many of these jolly-Storms as possible and scanning them to see which engines, when loaded with a pre-mailout database, detect close to 0% of the samples. The list you’ll find isn’t all that surprising.

It’d be really nice to have a good statistic on the size of the botnet on December 20th versus the size on January 7th. But all botnet size estimates are generally a product of bad guesstimation, we can’t expect anyone to know the numbers except the ones in control.

I’m becoming more pessimistic about the situation as time goes on. The concept of a “virus filter” product seems to have been proven fatally flawed. Whether detection takes place via signatures or “heuristics” (in my opinion this is little more than complicated signatures) the approach is reactive. Either to specific malware or to specific exploits, the latter gets a lot of press as “generic” detection usually classified as “heuristic” but in the end is just reactive detection taken from a different angle. AV engines do have their place, but they’re not a solution — certainly not anymore. A small thought, and privacy advocates would hate this thought, is that maybe the AV vendors need to make their software 100%-report-to-base. Try to take some of the testing ability away from the criminals? Could this even be workable, what information could you report to base that’d help? How long would it be before the bad guys subverted the process or simply circumvented it… probably not long. sigh

I guess this is why the security industry is diversifying into more elements of command and control, maybe there is some light at the end of the tunnel? Of course is it likely that anything of this sort is best done at-or-below the OS level, thus by the OS vendor, but when Microsoft tried to do this for Vista there was an all-out cry of foul from the AV industry! Protecting themselves, or protecting users from the likelihood that Microsoft would get it wrong? A bit of both I expect.

In this direction lot of noise was made about one thing in the last year that to me smells like a load of of bollocks: virtualisation. It’s a very neat geek-toy that has spawned both it’s own industry around maintaining systems and has been co-opted by the security industry in a way that stinks of “silver-bullet”. The former works for me, but I think we want to keep in mind that virutalisation used this way is just an evolutionary step. Virtualisation for robustness/etc is a neat replacement for things like telnettable power supplies and Dell DRAC (remote administration) hardware. Security tends to be fitted in from a perspective of keeping an eye on things from the outside. We like this image because it works fairly well with physical-world security systems. My guess is that it isn’t going to work out quite as neatly or easily as hoped when it comes to anti-malware. I think the best anti-virutalisation FUD I’ve seen came from Theo, of OpenBSD fame.

[Update: In case it isn’t as blindly obvious as I thought, I agree with Theo de Raadt’s FUD (though I don’t understand why anybody thinks my agreement or labelling matters). sigh “FUD” is a just TLA, please attach less emotion to it Internet randoms. I’m wasting my time since the complaint I received was clearly derived purely from the sight of the TLA and the context ignored. Anyway, FUD = “Fear, Uncertainty, and Doubt” and in my mind is a mere function of marketing. Negative marketing based on perceived flaws in the security sphere is a case of FUD (since this is what it causes), sometimes for good (being informative), sometimes for bad (being misleading). Pro-virtualisation-for security people will label de Raadt’s opinion as FUD in the traditional sense, but I bag up what they see as smelly manure and feed it to my roses. I apologise for going against the grain of the TLA and upsetting a poor sensitive soul or two. To repeat: I, in my non-expert opinion, am more convinced by Theo’s FUD than the FUD from the other side of the argument. If it makes you feel better execute a mental s/FUD/marketing/g or just go away.]

Still, we have to grasp at what straws present themselves. (Remembering to let go of the ones that have burnt all the way down to our fingers.) I try to remind myself that entirely giving up hope is not the correct response. Especially while people are profiting from criminal acts that take advantage of the industry’s current failure to adequately deal with the problem.

At this moment, given a corporate network to run and short of “running with scissors”, I’d be focusing attention on environment control. Mostly meaning various approaches to controlled execution. I don’t think it’s an easy path, but does anyone expect a solution to really be “easy”? Hah! There’s a strong chance it’d just turn into another reactive scene, say we allow IE to run, fine, then malware runs it’s own code as part of IE. (Through one of virtually limitless vectors, from buffer buffer overflows inserting actual machine code to simple exploitation of design flaws in JS/VBS/Flash/plugin-X/technology-Y.) What about the much-maligned (at least it is in OSS/FSF circles) TPM approach? (Maybe just simplified virtualisation that’ll just come with a heap of it’s own new flaws.)

Network segregation should offer some relief and damage control. Do users really always need to access email/web from the same machine they access the IRS/HMRC/etc database from? At least if there is an infection (inevitable?) it can only go so far. This is heading into DLP territory though, which is a different problem and mostly the bugs that need to be fixed are in process and people.

Have we given up on user education yet? It’s bloody difficult, but I hope not. We can’t really expect people to always do the right thing, just as we can’t always expect programmers who know they should use validate all user data to always remember to do so (humans tend to be lazy by preference!). That said, the situation is certainly worse if they don’t even know what the right/wrong things are!

It’s easy to become despondent. I’m certainly not all that happy with the industry that I, in a small way, am part of. Taken as a whole the last year or two has been pretty abysmal. Surely things can only improve from this point?

Storm Worm Vigenère

Note: This entry has been restored from old archives.

A small hobby of mine to pick apart JavaScript/ECMA obfuscation such as that used by the Zhelatin/Storm/Nuwar “worm”. My usual approach, which is certainly inefficient, is to grok the actual code by translating it to Perl. I’ve written about this before in “Someone Doesn’t Like Kaspersky“.

I don’t usually have time, after wasting much in the process of grokking, to write about these critters and I don’t expect that to change much! Time is so hard to come by! But after looking at some of the code with recent Storm mailings I think it’s worth noting the evolution.

The previous obfuscation I’ve written about is simple application of “xor encryption”, and much of what I’ve seen has been a variation on this at a similar level of simplicity.

The basic xor case worked along the lines of the following pattern.

    function decode(A,B) {
        ...
        eval(C);
    }
    decode(ciphertext,key);

In this case the key (and thus ciphertext) value was randomly generated for different visits to the page. In the decode function B is applied byte-by-byte to A to gain the plaintext C. Usually this processing was xor (^) and was further complicated with a URI decode or something of that ilk.

The sample I have looked at most recently has the following form.

    function X(Y) {
        ...
        eval(Z);
    }
    X(payload);

The key differences are that the function name (X) is now a variable and the obvious key input is gone, which hints at something. What’s changed inside the code? Well, working from the final decrypt up to the start of the function, this is what happens (somewhat simplified, but this is the core pattern):

  1. An array of 8 bytes is used as a key to shift the values in the input array in the manner of a classic Vigenère cipher applied mod-256).
  2. The key array is obtained be encoding a 32 bit value (i.e. 2309737967) to hex (0x89ABCDEF) and using the ASCII value of each hex digit to populate the key array ([56, 57, 65, 66, 67, 68, 69, 70).
  3. The 32 bit value is obtained by condensing an array of 256 integers (array256) and the text of the decode function (funcText) into an integer! The method iterates over characters in funcText using the byte values as lookup indexes in array256. Complete detail: key=0xFFFFFFFF; then for i in 0 to length(funcText) do:
    key=(array256[(key^funcText[i]) & 0xFF] ^ ((key >> 8) & 0xFFFFFF))
  4. The text of the decode function is obtained with arguments.callee.toString(), which has non-word chars stripped out and is converted to all-caps. Thus the importance of the function name X as an input parameter to the obfuscation, it doesn’t stop there as the text for the rest of the function body is also part of this key material and is full of randomised variable names. As you may have guessed, is is the random function and variable names that change from one downloading of the script to another — rather than just the xor key.
  5. The array of 256 integers is generated from a simple algorithm with a seed value, no need to detail it I think. It’s worth observing that between the different downloads of the script I saw the effective seed value didn’t change so this array remained constant.

Certainly much more complicated than the old xor code! But, I’d hope, a waste of time — since AV suspicious-script detection should work off generic patterns visible in the script from inspection rather than relying on the variable details. Still, only 3 AV engines on virustotal.com thought this script was worth noting as “generic obfuscated HTML”, but I don’t know what script/browser components they have enabled so I wouldn’t trust these out-of-context results. Many AV products exhibit different, usually more paranoid, behaviour when scanning in-browser data and HTTP at the gateway. And, looking at the whole Storm picture, this little snippet of code is just part of the delivery mechanism, it’s more important that the actual browser exploits and malware executables are caught!

Anyway, back to the script, this thing unwraps like a matryoshka doll. The plaintext is the same algorithm over again with new randomly generated function/variable names and a new ciphertext. The new ciphertext is much shorter though and after decoding we’re finished with this sample. The end result is javascript that generates a script DOM element and appends this to the document.

    var script = document.createElement("script");

    script.setAttribute("language", "JavaScript");
    script.setAttribute("src", "<nasty_local_url>");

    document.body.appendChild(script);

The most interesting item is the sample is this use of arguments.callee.toString() as key material. No doubt a direct defence against the usual malware-researcher practice of changing the final eval into an alert to expose the plaintext. While an admirable attempt at making life harder for researchers it’s not difficult to circumvent, just create a new variable assigned to the text “function X(Y) { ... }” and use this in place of the arguments.callee.toString() and good old alert should do it’s usual trick (then unwrap the next shell of the matryoshka). (Yes, “function” all that are included, though braces/punctuation don’t matter in the samples I have since an s/W//g is applied to the text)

The other “new technology” here is intriguing but not remarkable, using Vigenère instead of xor seems a curiosity more than a real advance (they’re certainly not doing it to hide the tell-tale use of the xor operator in a loop since they use xor in the key generation loops). Honestly, is looks just like some geek having fun, like me… but in this case we have a bad geek. Tut tut.

I’ve put a de-obfuscated and commented version of the script code up as well as a page containing active JavaScript that demonstrates the code. (Don’t worry, the active page’s payload is just an “alert” call!)

Christmas Storm

Note: This entry has been restored from old archives.

It’s been a while since I’ve had a Zhelatin/Storm/Nuwar mail get through to my inbox. Just in time for Christmas I get a shiny new one! It wishes me “Merry Christmas Dude” and provides a suitable URL for the season, no suspicious IP address link for this special occasion.

This one is a little different to previous efforts I’ve looked at. The embedded javascript isn’t malicious at all, in fact it is JSnow v0.2 complete with copyright notice. Snow! Joy! Is our favourite bot-net wishing us all a good Christmas out of good old fashioned social benevolence? Ha, fat chance! The page displays for us a set of scantily clad Mrs Clauses, enticing us to click on them for more. The link is to stripshow.exe, just less than 50% of the scanners on virustotal.com detect this at the moment. The list of ones that miss is conspicuously a round-up of the set with the largest market-share (interspersed with the ones that simply suck), this shouldn’t be any surprise these days.

It doesn’t stop there though, in a further effort the page embeds a javascript
in a I-Frame. And behold! We see the expected obfuscation code. So, in the end this isn’t really much different to previous sightings. I guess this strategy is still paying off for the crims behind it. It’s a sad indictment against the state of Internet security and security awareness that even after so many months this seemingly still works.

This time the javascript obfuscation is far more complex than others I’ve seen. Rather than a couple of simple translations we have several loops employing shifts and a variety of other bitwise operators (didn’t even know ECMA had an LSR operator). I guess they’ve invested some of their research time into this aspect of their code. At the moment only three of the virustotal.com scanners have anything to say about this and that’s just something along the lines of “generic obfuscated HTML”.

I wish people an infection-free Christmas. Have a good one.

Malignity?

Note: This entry has been restored from old archives.

Why malignity? More than 5 years ago I registered malignity.net and now, honestly, I don’t remember why I chose “malignity”. Maybe I was going through one of my phases of angst-ridden annoyance at society. Or it could have just been contrived evilness designed to somehow upset the apple-cart of the Establishment (not that it could care less). In reality I rarely feel truely malignant, and in history it isn’t a common theme in my life. Sure, society still annoys me, I still turn my nose at anything with a whiff of Establishment about it, but my reactions are more reflective, more latently sociopathic. Malignity? If I was malignant in my reaction to the world about me we’d be talking Fight Club, not mere grumpiness.

In short: I’m redirecting to the more logical URL of http://yvan.seth.id.au/. Honestly, that makes far more sense. I’m not going to consider finding a new word to inaccurately describe myself and my babble, my name is correct and sufficient and unlikely to change (neither is the fact that I’m Australian). Anyway, a quest for a new “word” would require the dictionary-of-eye-wateringly-long-words, or looking at new and obscure top-level-domains.

I haven’t decided whether or not I’ll renew the domain in the future, however malignity.net doesn’t expire until mid-2012 so there is some time to mull over that one! I have no intention of trying to change over the huge number of emails tied to the domain anyway, so I expect it is a keeper — just in a reduced capacity. For now all all old URLs will just return 301 to yvan.seth.id.au, but I expect I’ll turn that off in a month or two. (In other words: if you care, update your bookmarks or readers.)


Now for the waffle. Best served with bannana, maple syrup, and double cream.

Even for people like myself, who really don’t have much time for the whole “Christmas thing”, this time of year is marked with indelible ink. When it comes to traditional Christmas and New Year I can take it or leave it. To me, one is a solstice festival commandeered by popular religion. Christianity owned it for centuries, and Commerce seems to be the major stakeholder now. The other is a mere side-effect of the ridiculous concepts of “clock” and “calendar”, I try not to think about time too much as it upsets me[1]. That’s how I feel, but it does not represent how I think anyone else should feel! It’s been a long while since I last thought the world should conform to my point of view. What a mess that would be!

Now I write, delete, write, and then finally suppress a sequence of words that tries to adequately describe my position on Christmas. It’s too difficult for me to explain in a succinct manner. I’m not against it, not for it. It is just a concept after all. What gets my hackles up is other people, and that is a truly endless source of material which can all be boiled down to “they are not me”.

Me? I can’t escape the season, it’d be like trying to run away from my own shadow. My family did the Christmas thing and even took a day or two off, and as a restaurant family time-off in the high-season is a big thing. So from before I remember it was part of my life. Religion wasn’t though, the only dose of religion in my childhood was a devout Christian (Methodist maybe, I’m not sure) baby-sitter who didn’t really push the subject but read us a lot of old testament stories for bedtime. That and my father’s occasional claims of being Catholic, which I always thought a bit absurd. He had it whipped into him by nuns or something though, corporal punishment does tend to drive the message home[2].

Anyway, despite the usual Grinch-like posturng, there’s a Christmas shaped hole in my year and what better to fill it with than Christmas? But what shape is this hole? It isn’t relaxation shaped, it isn’t shaped like a glass of beer, nor a church. Maybe it should be family shaped, it probably was once but eight years of being nowhere near family at this time of year has morphed it into something else. The shape is one of reflection and speculation. I have to laugh at myself on thinking this: no religion yet I immerse myself in quiet contemplation of the metaphysical.

I don’t travel at this time of year, it’s the worst time to try. I don’t even go out much, for much the same reason. I stay at home and try to get some of the wrinkles ironed out of my thoughts. Small things I’d normally be forced to dismiss get more time. Small things such as “malignity”, I’ve been uncomfortable with it for quite some time now. I thought about it for a good couple of hours just now and decided there’s no reason not to switch to my namesake domain. So it is done.

I’ll have more time to myself than usual this year. Traditionally I work on the on-days over the period, but the calendar is inconvenient this time so I’ll probably have the entire block of 11 days as a break. Kathlene on the other hand has to work on the on-days, bummer. So, what’s going to happen? More reflection and speculation or will hoped-for personal-productivity actually occur? This time right now is certainly the former.

I have a very long list of things to dwell upon, haven’t we all?


[1] For a sociological headache read up on the history “standard time”. Especially daylight savings and the insane emotion and politics around the subject. Physics is another issue entirely.

[2] Proof: In year-1 I was smacked by the principal for biting a classmate on the back (he pushed into the line!) and I’ve not bitten anyone since (well, not against their will).

Foggy Day

Note: This entry has been restored from old archives.

Into the fog
Into the fog

Today it was cold and foggy. What did we do? We popped out to the supermarket
to get some goodies: cheese, salami, and crusty bread. We stuck some hot water and a couple of bags of Earl Grey into a thermos. We hopped up to the next station on the tube. We wandered to the centre of Chorleywood Common. We had a picnic! It’s nice to see the world a bit differently.

Interested people might like this little collection of
photos from our foggy wander.

I’ve got a lot of things to get “written down” but, as is my continual predicament, it costs me a great effort to get things from my head to “paper”. I’m not going to finish the day-by-day summaries from Finland, instead I’ve been trying to codify a more complete “write up” of the trip into a sequence of words. It grows like a monster! I try to cut down the length and behold, it grows! I seriously lack the art of brevity. Additionally I’m intending to write reviews for three restaurants in Äkäslompolo. I have at least three recipes I need to finish, though they’ll doubtless end up in the folder with 10 or so others that gathered too much dust. I have a couple of entries I need to complete on local produce, local to the Rickmansworth area specifically. The list goes on!

Beauty
Beauty

A pile of randomly conceived chains of thought gathering dust. And always the qustion: why bother? Much effort, many ungainly sequences of words, a very small and anonymous audience. Believe me, it isn’t through a belief that I’m improving the content of the ‘net in any way. I really don’t care much for such high minded claptrap. The truth is that it is entirely self-centred, what human act isn’t? I derive an unlikely level of enjoyment from the effort, though I ridiculously feel much angst over the dust gatherers.

This “time of year” may help me a little on this front. What a mess the calendar has made! Both Christmas and New Year public holidays mid-week! So I’ll probably take the unusual, for me, route of just logging it all as “leave”. Although, as ever, work is a self-perpetuating to-do list with variable urgency.

There’s also two (semi)personal “tech” projects I want to get some time in on. It’s going to have to be one or the other, which will win, which lose?

There’s non-tech projects galore… this is the conundrum commonly referred to as “life”.

Decisions!

I’m leafing through “The River Cottage Year“. Inspiring! Depressing! Where is my garden? Where, for that matter, is my cow?