Yvan Seth's Hole in the Internet

Backporting an Ubuntu/Debian Package - spatialite for Jaunty

Wed, 21 Oct 2009 22:03:00 GMT

There are a lot more "Debian", aka Ubuntu, users around the place these days. In general it seems many newer users have been spoilt by Ubuntu's regular release schedule. Back in the old days we sometimes waited years between Debian releases, if there was something you needed that wasn't in your antique but "stable" Debian you'd have to use backports, if you were lucky enough to find what you wanted there, backport a package for yourself (thanks to the Debian package build tools being mostly friendly), or just compile from source.

Perhaps self-backporting has become an anachrnism? Still, I find myself doing it from time to time. Right now I want spatialite because I'm playing with GeoDjango and don't want to mess with PostgreSQL or MySQL – I'm just fiddling. In general on the 'net you find people are just grabbing precompiled binary tarballs or, shudder, the game are self-compiling and doing a make install … won't somebody think about the package management system!

Still another approach is to just upgrade to "testing" (or is it "unstable" - I think "stable" Ubuntu is closer to what we might have called "testing" Debian back in the day.) But the koala isn't fully baked yet, so why risk that? Backporting the karmic package is safer, and probably quicker. (Yes, yes, I know there's only 10 more days of baking to go.)

Here's a recipe to help avoid unbaked koalas:

sudo -s
# Put the Karmic 'universe' deb-src line into your sources
echo "deb-src http://gb.archive.ubuntu.com/ubuntu/ karmic universe" > \
    /etc/apt/sources.list.d/karmic-universe.list
# Install a few pre-requsites 
# you may find you need more than this
# but you'll definitely need at least the following
apt-get install autotools-dev debhelper doxygen dpatch dpkg-dev fakeroot \
    libgeos-dev libsqlite3-dev quilt ruby ruby-dev sharutils swig
exit

# Get, compile, and install a newer libgeos
apt-get source libgeos-dev
cd geos-3.1.0
dpkg-buildpackage -rfakeroot
cd ..
sudo dpkg -i libgeos-3.1.0_3.1.0-1_i386.deb \
    libgeos-dev_3.1.0-1_i386.deb libgeos-c1_3.1.0-1_i386.deb

# Get, compile, and install libproj
apt-get source libproj-dev
cd proj-4.6.1
dpkg-buildpackage -rfakeroot
cd ..
sudo dpkg -i libproj0_4.6.1-5_i386.deb \
    libproj-dev_4.6.1-5_i386.deb  proj-data_4.6.1-5_i386.deb

# Get, compile, and install spatialite - see a pattern emerging?
apt-get source spatialite
cd spatialite-2.3.0
dpkg-buildpackage -rfakeroot
cd ..
sudo dpkg -i libspatialite2_2.3.0-1_i386.deb  spatialite-bin_2.3.0-1_i386.deb

The advantage of this approach is that your additional bleeding edge software is installed "properly". The package management system is aware of it, and when the koala is baked and you upgrade you'll be certain to get any important updates.

Of course, the sequence above makes it look like plain sailing. The reality of the process is that you start by adding the deb-src line, then you get the spatialite source, then you try to build it. The build fails with a long list of the packages it needs as build dependencies. You install each of the build dependencies as required, grabbing some from the "unstable" source repository when needed (as for libgeos and libproj above.) In this case it all came from universe, so my restrictive deb-src line was fine, but usually you'll find you need things from "main" or even "multiverse" as well. Be warned, if you start needing to upgrade things like your libc it is time to stop and either venture the dist-upgrade or resort to make install (or try hacking the Debian package source to lower a dependency version;)

This post is a little tongue-in-cheek in some ways. I wouldn't expect people, developers especially, to stick with "stable" religiously - nothing ventured nothing gained! But I decided to pollute the 'net with these thoughts after coming across some young developers who had never seen the apt-get source command, let alone built a Debian package (die-hard Ubuntu fans who're young enough that Ubuntu was the first "Debian" they installed.) No, don't risk the koala I shouted! It is not the right way! (Then I hobbled back up to my hermit hole while the young fellows pitied the poor old crazy man.)

I do think backporting is usually a better approach than risking a premature system upgrade though. That said, we live in a pretty bleeding-edge world these days. I myself use most of my direct dependencies from SVN trunks, stuff releases - they're old and dusty, and a lot of these newfangled frameworks/technologies/whathaveyou even recommend this.

Also, I'd be dishonest if I didn't mention that sometimes when trying to backport something from proper Debian/Ubuntu package sources you can end up chasing your tail for ages. Quickly leading to frustration, and giving up, followed by either a make install or a dist-upgrade … ah, technology.

For the interested, after doing the above the rest of the process to get my Django sqlite database primed for geospatial data was simply:

wget http://www.gaia-gis.it/spatialite/init_spatialite-2.3.sql.gz
gunzip init_spatialite-2.3.sql.gz
spatialite mydatabase.db  < init_spatialite-2.3.sql
./manage.py syncdb

Oh, you want the fancy Open Street Maps editor widget too do you?

apt-get source libogdi3.2
sudo apt-get install python-gdal gdal-bin libgdal1-1.5.0 libogdi3.2
cd ogdi-dfsg-3.2.0~beta2
dpkg-buildpackage -rfakeroot
cd ..
sudo dpkg -i libogdi3.2_3.2.0~beta2-4_i386.deb ogdi-bin_3.2.0~beta2-4_i386.deb
sudo apt-get install python-gdal
./manage.py shell
Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41) 
[GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from django.contrib.gis.gdal import SpatialReference
>>> from django.contrib.gis.utils import add_postgis_srs
>>> add_postgis_srs(SpatialReference(900913))
>>> from django.contrib.gis import gdal
>>> gdal.HAS_GDAL
True

w00t!

std::endl

Thu, 03 Jul 2008 23:17:00 GMT

All these years I've been making regular use of good old std::endl without realising that it is a templated function! We live in a crazy world. Somewhere in the back of my head I guess I'd just assumed it was simply platform-dependant constant… not quite.

I came across this merry discovery when implementing a logger, since re-inventing the wheel is always so much fun. It isn't as bad as it seems, the "logger" is basically just a matter of encapsulating the relevant utility code, weighing in at less than 500 lines of which more than half are API comments.

Anyway, I'd like my logger to act like a std::ostream, although I don't want it to be a std::ostream really (preferring to avoid inheriting from such beasts without a really good reason.) The functionality is simple, it wraps a set of std::ostreams to send the output to. The important part is that it implements log levels and and log message classes (which you can register and name if you wish.) At any one time there is a log threshold and a set of active classes. To keep things really simple it has a single output method that takes only a std::string ref.

Now I want to use it with good old << since that is comfortable and not unexpected so I implement:

template <typename T>
Log& operator<<(Log & log, T const & t) {
    std::ostringstream oss;
    oss << t;
    log.output(oss.str());
    return log;
}

This seems to work swimmingly until I try the likes of log << std::endl, then it fails to compile. Eh, what on earth is std::endl that the template won't catch it? A peak into the ostream C++ header tells us it is a function pointer:

  template<typename _CharT, typename _Traits>
    inline basic_ostream<_CharT, _Traits>&.
    endl(basic_ostream<_CharT, _Traits>& __os)
    { return flush(__os.put(__os.widen('\n'))); }

Egads! So I implement this (bear with me):

Log& Ephedrine::operator<<(Log & log, std::ostream& (*fn)(std::ostream&)) {
    std::ostringstream oss;
    fn(oss);
    log.output(oss.str());
    return log;
}

And now I can do this:

g_log << Log::NORMAL << "Normal log output.  A number is " << 2.123 << std::endl;
g_log << Log::VERBOSE << "Verbose log output." << std::endl;

Joy.

But there's more than one worm in this can. Note that in the definition above it calls flush on the ostream. In the doxygen comments it notes "This manipulator is often mistakenly used when a simple newline is desired, leading to poor buffering performance." On checking the standard there it is too, this is std::endl by definition. I've never used ostreams, thus never std::endl, in performance critical code but I'll have to keep this in mind if I ever do! In all of the C++ books I've read, a fairly large number, I don't recall ever seeing this noted anywhere.

This brings into mind a question of design. I've avoided being an std::ostream yet gone and used std::endl, and std::endl is supposed to flush the stream (by definition, it's in the standard, see section 27.6.2.7.) My current implementation doesn't flush the streams, so what should I do?

A) Not care.
B) Reimplement as Log::endl (which will ultimately call std::endl and thus end up flushing the streams!)
C) Create a special case specifically for std::endl.

I don't like the first option, not my way of doing things by preference (alas, it has been known to happen.) The second option is easy to implement but will feel unnatural to the user (a moot point, I'm the sole user!) The third option smells "wrong," but I already decided to do it didn't I, and I went one step worse and broke the guarantee for std::endl (albeit probably mostly unknown.)

Is C really enough though? No, it turns out. Aside from breaking the flush guarantee it also doesn't go quite far enough. Think about std::hex and similar modifiers. By converting them to a string (an empty string) before writing to the underlying std::ostreams their meaning is lost. My original implementation above is, in essence, an abomination.

In the end I settled on E: templates. What I should have done in the first place, but avoided since I was hung up on pimpl at the time (templates and pimpl don't mix well.)

template <typename T>
void Log::output(T const & t) {
    ... /prep/
        BOOST_FOREACH(std::ostream & os, m_osList) {
            os << t;
        }
    ...
}

Bitten by the python

Mon, 16 Jun 2008 17:10:00 GMT

I play with the snake from time to time, Python having now almost entirely supplanted Perl as my hak-n-slash language. But I'm certainly still learning as I go. Today I've been trying to work out what's gone wrong with something I'm doing. I thought I'd done something strange with inheritance, not understanding some case where inheritance causes data to become static (in the sense of static members in C++.) What was really the problem is that I didn't realise that default parameters to members are kept and reused, and if you go and use a parameter you can end up with state being held on to where you don't (well, I didn't) expect it.

I've hit this at least once before I think, it is vaguely familiar. Anyway, it's a bit of a tricky gotcha as far as I'm concerned. Intuitive? I wouldn't say so. Here's the example:

#!/usr/bin/python

class Parent:
    def __init__(self, stuff = []):
        self.stuff = stuff
    def doIt(self):
        print " ".join(self.stuff)
    def addStuff(self, stuff):
        self.stuff.append(stuff)

class Child(Parent):
    def __init__(self, stuff = []):
        Parent.__init__(self)
    def doIt(self):
        self.addStuff("foo")
        Parent.doIt(self)

c1 = Child()
c1.doIt()
c2 = Child()
c2.doIt()
c3 = Child()
c3.doIt()

Execute this and we see:

foo
foo foo
foo foo foo

What? Why is it accumulating "foo"s? The answer is in the "stuff = []" argument to __init__. What is going on, as far as I can garner from the documentation, is that that default argument (a list) is being instantiated once and then kept for future use. What's more, I'm assigning self.stuff to this, which is kept as a reference and doesn't create a copy. So I have ended up with a static value for self.stuff, well, within the functionality of this code – it isn't entirely congruous to a static member.

How to fix it? I don't know the definitive approach, but here's a couple of ways. To achieve complete deep copying of the argument whether it be default or passed in use copy.deepcopy:

#!/usr/bin/python
import copy

class Parent:
    def __init__(self, stuff = []):
        self.stuff = copy.deepcopy(stuff)
...

Alternatively, you could use copy.copy for a shallow copy and I guess that might be analogous to this:

#!/usr/bin/python

class Parent:
    def __init__(self, stuff = []):
        self.stuff = []
        self.stuff.extend(stuff)
...

I'm sure there are great uses for this behaviour in Python. What are they? Something more fundamental than "neat tricks" involving incremental/changing default state? Am I the only one to think this somewhat of a gotcha?

Now that I've worked out what was wrong I can retrospectively build the right Google magic to get straight to the answer.

No time to read up on more casual chatter about it though. The documented formula is something like:

#!/usr/bin/python

class Parent:
    def __init__(self, stuff = None):
        if stuff is None:
            stuff = []
        self.stuff = stuff
...

Don't store references to boost::shared_ptr

Wed, 09 Apr 2008 22:57:00 GMT

I should have known better, in fact I'm pretty sure I did know better, but it is so easy to fall into the trap of using reference members as often as possible. Actually, in other code I've not done this, so I'll call it a "typo", yeah. :)

No time to go into details with examples and all that, best to keep things simple anyway.

Avoid storing auto/smart/shared pointer references in the name of avoiding premature optimisation. You always have to think: will this be destroyed before this is destroyed. If it takes more than 10 seconds to work that out then steer clear of the reference path! Hoping it'll all be OK is asking for trouble. Especially if you don't own the calling code!

A shared pointer instance stores little state, so just copy the damn thing. This way the target instance will be destroyed when nothing needs it anymore, which is the whole point. Better a few copies than having things vanish earlier than you expected because you tried to game the system.

My new personal rule is: always copy shared/auto/smart pointers.

If the copy is a problem it'll show up in profiling later and that's when you work out how to fix it.

(Alas there isn't an easy way around avoiding circular references.)