It seems Google haven’t really studied the book though, as they’re not making this happen naturally or with minimum effort — virally, in modern business speak. If Google Chrome Frame installed in IE for some site as readily as flash installs, this would be a hugely significant release. Seems they’ve missed a trick here.

Some time ago, I blogged about Pluggable Django and How I hate repetiting myself. That was mostly about current but messy code I was using, which allowed me to automatically add decoupled apps to a django project without having modify the rest of my django project each time. Apps were auto-detected, and loaded, including templates etc.

Since then, I’ve been re-working this code a little. It hasn’t been high priority, since django is not a tool I’ve needed on current projects, but I did manage to get something up on github a few days back:

• Blitzen Project
• Blitzen App

So basically, these are git repositories for “django-admin startproject” and “./manage.py startapp” replacements. To start a project, just clone the Blitzen project url, and to create an app within that, clone the Blitzen app url. Note that most of the new code here is in a directory called quickening, which was a previous working title for this stuff.

So what does it do? Well, currently, not as much as my old code, in that it doesn’t really handle templates etc. just yet (the code might be there for that, but it’s untested). Things you’ll notice:

• settings.py is a lot more manageable. Most of the complex config is moved into the library code and called
• Load order of middleware is worked out automatically, by specifying dependencies. Code for this is very straightforward, in quickening/deps.py, with the dependencies currently in quickening/apps.py
• If I recall correctly, app dependencies aren’t handled yet, but the middleware dependency code was written to also support automatic dependency resolution for apps, including specifying dependencies within an app itself, rather than in the project that hosts an app. More work on this is probably required.
• there is no default SECRET_KEY any more. Instead, secret keys are auto-generated and kept in files which are ignored by git, meaning that when you checkout the code, or push the code to a webserver, each new instance gets its own key automatically.
• sqlite is configured by default

Ideas, suggestions, flames, code, etc. welcome

Apparently they tried to talk Riverbank Computing into releasing the PyQt bindings as LGPL, but Riverbank wouldn’t cooperate. Presumably Riverbank wanted Nokia to pay them a stack of money, forgetting that Nokia, owning Qt, could undercut them by just rewriting the thing themselves.

Either way… this is nothing but good news for Python+Qt fans I think. Not only do we get new, LGPL bindings, but they’re official, and will probably be better synced with official Qt releases. Who knows? They might even be documented

Oh, and it’s written in python

UmS6D0aW62+q!'%S2 (perfection:   0%) Evolving        2nd generation.
aUKahChc.5,e&8:?1 (perfection:   0%) Evolving        3rd generation.
bYJo$F?O2;aTV'q'? (perfection:   0%) Evolving        4th generation.
P]N0+dC=p-GVh<p1E (perfection:   0%) Evolving        5th generation.
P]N0+dC=p-GPQ>+6b (perfection:   0%) Evolving        6th generation.
Q?T1LHOp=LSj*VUr0 (perfection:   0%) Evolving        7th generation.
ob3<2Ih=p-CO^f#`5 (perfection:   0%) Evolving        8th generation.
dK8:SCn^omSj*VO ` (perfection:   0%) Evolving        9th generation.
X@Z'H<f.)T?f&(i6. (perfection:   6%) Evolving       10th generation.
g&H0+)DG^J%GV(\(l (perfection:   0%) Evolving       11th generation.
dK9K3Vf.)T?f&`Kfq (perfection:   6%) Evolving       12th generation.
8@Zm`,jZ-*jf38":; (perfection:  12%) Evolving       13th generation.
g&l:7O^q0jGA2r2,& (perfection:  12%) Evolving       14th generation.
j^4'N.:L0fV.9_Psi (perfection:  18%) Evolving       15th generation.
co2sL:BLffGA2((Y! (perfection:  18%) Evolving       16th generation.
MHX<oUL$EP`<o(\si (perfection:  18%) Evolving       17th generation.
g9l#bkrL$+e &>Tsr (perfection:  24%) Evolving       18th generation.
Pibho,MYilkEo(.sj (perfection:  29%) Evolving       19th generation.
cel/opLRP2aEerTs! (perfection:  35%) Evolving       20th generation.
HR(so ELrfef/\-s& (perfection:  41%) Evolving       21st generation.
ce>l9dr3iPef?r2sc (perfection:  41%) Evolving       22nd generation.
HRl#) EaibkforTs! (perfection:  47%) Evolving       23rd generation.
Ho9ho, LJ-?fY>ms! (perfection:  53%) Evolving       24th generation.
He^j`,YLrfeHFlms! (perfection:  53%) Evolving       25th generation.
HRli*/ Lrfe &rms! (perfection:  59%) Evolving       26th generation.
HelBNd Lrfe &rms! (perfection:  65%) Evolving       27th generation.
HeGlo,jLifGf&rm]! (perfection:  71%) Evolving       28th generation.
ce4lod >iWeforms! (perfection:  71%) Evolving       29th generation.
H&lho, Lif@forms! (perfection:  82%) Evolving       30th generation.
HRllo, >ifeforms! (perfection:  88%) Evolving       31st generation.
Hello,:Lif@forms! (perfection:  88%) Evolving       32nd generation.
He9lo, Lifeforms! (perfection:  94%) Evolving       33rd generation.
Hello, LifeformN! (perfection:  94%) Evolving       34th generation.
HelloV Lifeforms! (perfection:  94%) Evolving       35th generation.
Hello, LifefKrms! (perfection:  94%) Evolving       36th generation.
Evolved to 100% perfection. Concluding.
 
35,000 individuals sacrificed their lives
so that their species could eventually say:
 
        Hello, Lifeforms!

The Challenge

So… the other day, Steve drew my attention to this craigslist “to all those who think themselves a programmer” challenge. To cut a long story short, the advert included a programming challenge, and mentioned that a genetic algorithm that could say Hello World using the sacrificial efforts of billions of evolving entities would be one of the winners.

Now, it just so happens that I wrote a pretty fun Genetic Programming app as my first foray into linux C++ programming back in the mid 90’s, and had lost the code for it. So, last night, I rewrote a cut-down version of it in Python.

#!/usr/bin/env python2.6
#
# Copyright (c) 2009 Lee Braiden <lee.b@irukado.org>
# Released under the GNU General Public License, version 3.
# http://www.gnu.org/licenses/gpl.html for details.
#
 
import sys
from random import randint, sample
 
class Mirror:
	"""Stores certain details that individuals must analyse
	and evolve themselves against.  Named after the
	psychology concept of being a mirror for an individual's
	own thoughts."""
	def __init__(self, destiny):
		"""Takes a string, 'destiny', which creatures must
		evolve to say."""
		self.UPPER = max([ ord(x) for x in destiny ])
		self.LOWER = min([ ord(x) for x in destiny ])
		self.destiny = destiny
 
	def chromosome_len(self):
		"""Returns the number of chromosomes in an individual's
		genes.  In other words, this is the length of the message
		characters should evolve to say."""
		return len(self.destiny)
 
	def range(self):
		"""A quick (readability-)helper function, which allows looping
		over each gene."""
		return range(0, self.chromosome_len())
 
class Individual:
	"""Represents an individual beastie which will be created, can
	breed, can mutate, can attempt to solve the problem at hand,
	and can be tested for fitness, among other things."""
	def __init__(self, chromosome_len, mutation_rarity):
		self._genes =  " " * chromosome_len
		self.fitness = 0
		self.mutation_rarity = mutation_rarity
 
	def _gene_val(self, mirror):
		"""Create a random gene value"""
		return chr(randint(mirror.LOWER, mirror.UPPER))
 
	def _randomise_genes(self, mirror):
		"""Initialise all of the individual's genes randomly.  Used
		during creation, rather than breeding"""
		self._genes = "".join([ self._gene_val(mirror) for idx in mirror.range() ])
 
	def _choose_gene(self):
		"""Choose an individual gene to target."""
		return randint(0, len(self._genes)-1)
 
	def _freaky(self):
		"""Roll a dice, and see if this individual is (un)lucky enough
		to gain a mutation."""
		return randint(0, self.mutation_rarity) == 0
 
	@classmethod
	def conjure(cls, mirror, mutation_rarity):
		"""Create an individual from scratch."""
		i = cls(len(mirror.destiny), mutation_rarity)
		i._randomise_genes(mirror)
		return i
 
	@classmethod
	def breed(cls, mirror, mom, dad):
		"""Breed an individual from two parents."""
		mutation_rarity = float(mom.mutation_rarity + dad.mutation_rarity) / 2.0
		kid = cls(len(mirror.destiny), mutation_rarity)
		x = kid._choose_gene()
		kid._genes = "".join(mom._genes[:x]) + "".join(dad._genes[x:])
		if kid._freaky():
			kid._mutate(mirror)
		return kid
 
	def _mutate(self, mirror):
		"""Mutate a random gene"""
		orig_genes = self._genes
		idx = self._choose_gene()
		els = [ x for x in self._genes ]
		els[idx] = self._gene_val(mirror)
		self._genes = "".join(els)
 
	def self_evaluate(self, mirror):
		"""Evaluate 'fitness', or the ability to solve the given
		problem.  In other words, measure how close the individual
		is to the correct message text, and assign a score
		accordingly."""
		fitness = max_fitness = float(len(self._genes))
 
		for idx in range(0, len(self._genes)):
			if self._genes[idx] == mirror.destiny[idx]:
				continue
			else:
				fitness -= 1
 
		self.fitness = 100.0 * (fitness / max_fitness)
 
	def __cmp__(self, competitor):
		"""Comparison function, used for sorting populations to find
		the fittest."""
		if self.fitness > competitor.fitness:
			return 1
		elif self.fitness == competitor.fitness:
			return 0
		else:
			return -1
 
class World:
	"""Represents a world populated by Individuals, which breed
	together, subject to natural selection, to reach an evolutionary
	goal."""
	def __init__(self, mirror, population_size=1000, mutation_rarity=1000, fitness_required=100.0):
		"""Initialise a population of population_size members, mutating
		one in every mutation_rarity breeds.  Evolution will stop when
		fitness reaches fitness_required percent."""
		self._mirror = mirror
		self._population = [ Individual.conjure(mirror, mutation_rarity) for i in range(0, population_size) ]
		self.sacrifice_count = 0
		self._fitness_required = 100.0
		self._mutation_rarity = mutation_rarity
 
	def _natural_select(self, exclusion_list=[]):
		"""Select two individuals from a population, choosing the most
		'fit' for breeding."""
		a,b = sample(self._population, 2)
		if (a > b):
			return a
		else:
			return b
 
	def _breeding_pairs(self):
		"""Select enough breeding pairs from a population to create a
		new population of equal size, assuming one child per couple."""
		bp = []
		pop_size = len(self._population)
 
		for i in range(0, pop_size):
			mom = self._natural_select()
			dad = self._natural_select([mom])
			bp.append((mom,dad))
 
		return bp
 
	def _breed_new_generation(self, mirror):
		"""Create a new generation, breeding it from the older
		generation using natural selection."""
		new_gen = []
		bp = self._breeding_pairs()
		for mom,dad in bp:
			kid = Individual.breed(mirror, mom,dad)
			new_gen.append(kid)
 
		self.sacrifice_count += len(self._population)
		self._population = new_gen
 
	def _analyse_population(self):
		"""Evaluate the genetic fitness of the entire population."""
		for i in self._population:
			i.self_evaluate(self._mirror)
 
		self._population.sort()
		most_civilised = self._population[0]
 
		return most_civilised
 
	def evolve_civilisation(self):
		"""Evolve this civilisation until the required fitness_level
		is reached."""
		generation_count = 1
		last_str_len = 0
		while(generation_count):
			most_civilised = self._analyse_population()
			gen_ord = ordinal(generation_count + 1)
 
			if most_civilised.fitness >= self._fitness_required:
				print "Evolved to %3.0f%% perfection. Concluding." % most_civilised.fitness
				return most_civilised
			else:
				print "%s (perfection: %3.0f%%) Evolving % 10s generation." % (most_civilised._genes, most_civilised.fitness, gen_ord)
				self._breed_new_generation(self._mirror)
 
			generation_count += 1
 
# a couple of helper functions #############
 
def ordinal(n):
	"""borrowed from John Machin's python-list post.  Appends an
	ordinal suffix to a number.  For example, 1 becomes 1st,
	2 becomes 2nd, etc."""
	if 10 <= n % 100 < 20:
		return str(n) + 'th'
	else:
		return  str(n) + {1 : 'st', 2 : 'nd', 3 : 'rd'}.get(n % 10, "th")
 
 
def PrintLargeNumber(n,width=2,delim=',',decimal='.'):
	"""Converts a float to a string with appropriately placed commas.
 
	Floats will be shown with 'width' digits right of the decimal.
	'delim' specifies the thousands delimiter.
	'decimal' specifies the decimal character.
 
	Copyright 2007 Regents of the University of California
	Written by David Isaacson at the University of California,
	Davis BSD License"""
	if width >= 0: s = "%.*f" %(width,n)
	else: s = str(n)
	dec = s.find(decimal)
	if dec == -1: dec = len(s)
	threes = int((dec-1)/3) # we don't need a comma at the start
	for i in xrange(threes):
		loc = dec-3*(i+1)
		s = s[:loc] + delim + s[loc:]
	return s
 
############################################
 
def evolve_from_nothingness(msg):
	"""Evolve an individual capable of saying the given msg."""
	world = World(Mirror(msg))
	greeter = world.evolve_civilisation()
 
	large_num_str = PrintLargeNumber(world.sacrifice_count, width=0)
 
	print "\n%s individuals sacrificed their lives\nso that their species could eventually say:\n\n\t%s\n\n" % (large_num_str, greeter._genes)
 
 
if __name__ == "__main__":
	evolve_from_nothingness("Hello, Lifeforms!")
 
	sys.stdout.write("Press return: ")
	sys.stdout.flush()
	sys.stdin.readline()

All in all, it’s pretty sweet. It evolves fairly constantly, straight to the answer, in less than 40 generations. Completes in a few seconds as well.

I’ve no doubt the SF Bay guys would find problems with this. They’re into pointlessly obfuscated code, FP, and lambdas, for instance. I’m into pointedly unobfuscated code, and using FP only when it makes things much clearer. And optimisation is for compilers long after your algorithms and portability have done their thing. That said, most of this IS off the top of my head, and I’m sure there are better ways to do it, so let me know if there are obvious (or not so obvious) improvements that can be made.

Anyway… for me, this is what programming’s all about. In what other fields can you create worlds, create lifeforms, prove that evolution works, test yourself, frustrate yourself, and enjoy seeing your project finally work, all in the space of a few hours?

I should probably thank the python guys for a lot of that rapid development; the C++ version took much longer. Then again, it had a virtual CPU and datatyped I/O etc. I think it was threaded, too, if I remember correctly. It also took hours to work out that the way to get 4 out of two 2s was to add them. I might try that again someday, with a tweaked CPU/instruction set/program model.

I’ve been holding off on playing with TG2 too much, since it wasn’t final, but now I’m gonna have a good play. It’ll be interesting to compare this with django, cherrypy 3, etc.

Working through this in python is a very useful exercise; it’s teaching me lots about the most efficient ways of doing things in that language, and with the standard library tools, some of which I rarely find use for otherwise.

Anyway, together with some other things mentioned in the preface, Derivations seems like a very promising book. Except for one thing:

One particular respect in which the book departs requires some defense here, I think: the book employs hexadecimal numerals.

So it’s going to explain complex math, but it’s going to do it… in hexadecimal.

Not sure whether to laugh or cry. But I’ll see how it goes. I don’t entirely disagree with the idea that, “If one wishes to reach hexadecimal ground, one must leap.” It’s just that I don’t really see the point of becoming THAT fluent in hex, when understanding the basic concept of bases, and knowing how to convert from dec<>hex<>oct<>bin, that 2⁵ is 32, and that 2³² is roughly ~4 billion is almost certainly enough. Is this book really suggesting I learn to do all my math in hex? And if so, is there any real benefit? It does seem to suggest that math is more beautiful in hex, but I’ve never heard of anything special about base-16, in relation to laws of physics, for instance.

Well, find out next time, same bat-time, same bat-channel, I guess

So I’m deploying my first real django app to a live server, with modwsgi. ModWSGI itself isn’t great, as it insists on a single python version per apache server (and a global WSGIPythonHome rather than per vhost), thereby negating most of the benefits of virtualenv [update: seems I can use site.addsitedir() to get around this]. But my real problem with this deployment has not been modwsgi — that’s the nice, smooth part.

MySQL table name length limits

The problem is that django behaves entirely differently when deployed under modwsgi from testing under runserver(_plus).

First off, I did the obvious: copy the site over to the server, and change the database config from a local SQLite file (used for development) to a MySQL database on the live server. The next step is to run:

./manage.py syncdb

to initialise the database tables. This led to the first problem, which was a huge one. Django creates table names that are too long for MySQL (the limit is 64 characters, I believe) — the most common open source DB around! As a result, it simply doesn’t work with that database backend:

File "/var/lib/python-support/python2.6/django/core/management/commands/syncdb.py", line 80, in handle_noargs
cursor.execute(statement)
File "/var/lib/python-support/python2.6/django/db/backends/util.py", line 19, in execute
return self.cursor.execute(sql, params)
File "/var/lib/python-support/python2.6/django/db/backends/mysql/base.py", line 83, in execute
return self.cursor.execute(query, args)
File "/var/lib/python-support/python2.6/MySQLdb/cursors.py", line 166, in execute
self.errorhandler(self, exc, value)
File "/var/lib/python-support/python2.6/MySQLdb/connections.py", line 35, in defaulterrorhandler
raise errorclass, errorvalue
_mysql_exceptions.OperationalError: (1059, "Identifier name 'preferred_beneficiary_id_refs_basebeneficiary_ptr_id_24644201ef7702ef' is too long")


The only solution seems to be shortening the names of all your models, which is a bit useless after you’ve just completed testing your app and are ready to deploy it, or using another backend. I’ve read that the problem also exists with PostgreSQL. Surprisingly, it existed in Oracle too, much more severely, but that has been fixed:

Naming issues

Oracle imposes a name length limit of 30 characters. To accommodate this, the backend truncates database identifiers to fit, replacing the final four characters of the truncated name with a repeatable MD5 hash value.

I don’t have (or want) Oracle available of course, so that leaves a file-based backend for now, and re-naming lots of core models as soon as I get a bit of time. Not fun. A simple assertion check in the Django code against my model names while I developed the thing (or a few unit tests by Django developers) would have saved me all this deployment hassle.

But that’s just the start of the trouble, it seems.

Middleware ordering

I’m also seeing lots of problems related to middleware and and (probably) to their load-order. Lovely errors like “ModWSGIRequest.user is not defined”, which is pretty fundamental to Django, considering that just about every view uses the .user attribute from its first request argument, and that every template in a dynamic app that depends on the logged-in user also depends on it.

All of that would be fine, if it was my mistake. The thing that gets me is, this load-order stuff was almost entirely copied from django’s (and satchmo’s) examples, and worked fine in the development test server. If anything, I’d expect the test server (especially with debugging enabled) to be the more fussy target.

I’m still getting to the bottom of all this, and I’m not optimistic about solving it soon, since, as others have pointed out, There is no canonical reference for django middleware order, even though the docs often warn that it’s very important. Different examples give different orders (or say things like “must be last”, despite other middleware having the same vague requirement).

Then there are lovely issues with satchmo. Satchmo is a whole other beast though, so I won’t get into those right now.

Suffice to say, deploying django after developing is not at all a straightforward matter of uploading code and initialising or updating the live database. If I ever develop with django again, I’ll almost certainly use a development server with a setup identical to the live one. Runserver just doesn’t cut it. In reality, I’m very much thinking of Pylons or Turbogears for future projects.

• IO stuff reimplemented in C
• new garbage collector
• faster Unicode handling
• computed gotos (which I’ve never heard of, but apparently offers 20% speedup on supported compilers)

Things like this keep shocking me in python’s release notes though:

- Issue #4707: round(x, n) now returns an integer if x is an integer.
Previously it returned a float.

and from Python 3.0.1’s NEWS:

- Issue #4998: Decimal no longer subclasses from or is registered to
numbers.Real. Instead, it is registered to numbers.Number so that
isinstance(d, Number) will work.

In a release of the C standard library or something, changes like that would mean API and ABI breakage. With python’s flexible typing, it’s not so much of an issue, I guess. Still, it would be nice if someone had sat down and thought about the math a bit more, and been sure about what was right from the beginning. Working out if Decimals should be Real numbers (or, indeed, if Reals should be Numbers) shouldn’t be this hard, should it?

Hmm. I was sure Java would have gotten this right, but it seems not:

class java.lang.Number
* class java.lang.Double
* class java.lang.Float
* class java.lang.Integer
* class java.lang.Long

So longs aren’t integers, and Floats and Doubles are unrelated, except for being some kind of Number? Oh well. It’s not like math is important in computer science, right?

Anyway. I’m trying to keep track of Python 3.x, but until a lot more libraries are available that run with it, it’s mostly a far off dream. Shame, as I’ve been dying to dump 2.x in favor of the new Unicode stuff