class A(object):
    def __init__(self):
        super(self.__class__, self).__init__()

class B(A):
    def __init__(self):
        super(B, self).__init__()

The surprise comes when I try to use my sub-class, B. Instantiating B() blows up the stack with:

RuntimeError: maximum recursion depth exceeded while calling a Python object.

What?

According to the Python 2.7.2 standard library documentation, super “return[s] a proxy object that delegates method calls to a parent or sibling class of type.” So in the case of single inheritance, it delegates access to the super class, it does not return an instance of the super class. In the example above, this means that when you instantiate B, the follow happens:

1. enter B.__init__()
2. call super on B and call __init__ on the proxy object
3. enter A.__init__()
4. call super on self.__class__ and call __init__ on the proxy object

The problem is that when we get to step four, self still refers to our instance of B, so calling super points back to A again. In technical terms: Ka-bloom.

TL;DR: super(self.__class__, self) may look like a neat trick, but it’s the end of the line for sub-classing.

Further reading: Raymond Hettinger’s excellent blog post on super provides a great overview of super and shows off the improved Python 3 syntax, which removes the need to write the class name as part of the super statement. I was really pleased to find the Python standard library documentation links directly to it.

PyMock makes this somewhat simpler by providing a way of “recording” what you want your mock objects to return, and the going into “play back” mode to run the test. It provides a way to mock methods, attributes and generators on Python objects.

Things I’m sure of following the talk:

* The idea of mocking objects seems like a powerful way to isolate pieces of existing code for the purpose of testing.
* Showing a slide full of code, with near-zero explanation, for only a few seconds before moving on, isn’t long enough for me to grok it; judging by the under-the-breath comments from those around me, I’m not alone in this.
* Unit tests are hideously ugly when compared to the beauty of doctests.

McMillan is the developer of the “fixture”:http://code.google.com/p/fixture/ package, which provides a way of loading and referencing test data for your application. fixture also allows you to write tests against fixture objects using either a mixin or a decorator[1]. Fixture supports loading data into SQLAlchemy and SQLObject; CSV(comma separated value) and possible Django support are also under consideration.

Fixture is a relatively young project, but it does appear to be useful for testing data-driven applications.

fn1. For testing with “nose”:http://python.org/pypi/nose.

In response to running into these problems with testing ZEO, Jim developed a library called the Network Gateway Interface (NGI). The goal of NGI is that application code doesn’t touch socket code. NGI splits the network application into two pieces: the application side and the implementation side. The application side doesn’t need to be thread aware (or even safe) and uses events to communicate with the implementation side. And the implementation side handles the actual socket or transport work.

Jim’s talk used the example of a simple application, z2monitoring, to demonstrate the NGI. The goal of z2monitoring was to monitor and get status from a running Zope 2 application at the time when status is needed most: when the application is bogged down, responding to requests. When testing with NGI, you create a special testing connection, such as a TextConnection (which just uses plain text). The connection conforms to the IConnection interface, which also provides a way to set the connection handler. The connection handler (IConnectionHandler, natch), is used to respond to events like input, connection close and exceptions.

In addition to the async, event-driven interface, zc.ngi provides a simple, blocking file-like API. This is convenient for developing simple, single-task applications.

Jim spent a short part of his talk discussing about the benefits of non-determinism, such as the existing ZEO implementation. One of these was the “opportunity for the unexpected.” Of course, Jim also noted (with an unclear amount of irony) that production is also a form of testing.

NGI 1.0 was released just before PyCon, and this will probably be the last release. [Un]fortunately, Jim recently found that Twisted provides facilities for this style of programming and testing. Given this, zc.ngi will probably only live on (if at all) as a simplified facade over Twisted’s facilities.

* How fast is fast enough?
** Don’t prematurely optimize
** *Know* where the bottlenecks are, and optimize those specifically?
* Orders of magnitude: static (httpd), dynamic (python), db-queried
* Even 40 req/s in 3.4m pages/day
* Hundreds to low thousands of dynamic page views is usually good enough
* Scaling isn’t about the language, it’s about:
** DRY: cache!
** share nothing
* built a sample photo-app, FlickrKillr, for demonstration purposes
** preloaded with 100k’s users, 10-20 photos each
* first iteration: CGI
** roughly 23 requests/second
** problems:
*** loading Python interpreter for each request
*** all resources initialized for each request (inc. db connection)
** possible remedies:
*** run a Python web server (long-running process)
*** make one db connection per thread instead of request
** other remedies:
*** fastcgi
*** snakelets, twisted.web, RhubarbTart
*** mod_python
* second iteration: python app server (CherryPy used for this demo)
** roughly 139 requests per second
** problems
*** global interpreter lock — can only utilize one core on a dual core machine
*** sessions in the database — prefer an in-memory session store
** remedies:
*** run multiple instances of CherryPy (overcode GIL)
*** but then we need to balance with something like nginx
** other options
*** cherrypy in mod_python
* version 3: load balancing with nginx
** 217 requests/sec
** outstanding problems
*** static files read from disk every time
*** and they’re being read/written from python
** solutions:
*** memcached
*** combine with memcached w/ nginx
* version 4: caching
** 616 req/sec (benchmarking w/ homegrown tool)
** 1750 req/sec (benchmarking w/ ab)
* other notes:
** don’t forget to index
** without an index, the fourth iteration falls down to 28 requests/sec

http://www.polimetrix.com/pycon/demo.tar.gz

But the other things, about what works and what doesn’t, just isn’t a suprise. I wish I could remember the examples through the lack-of-coffee haze, but I was suprised by the similarity — I would have thought “money changes everything” (to quote the Cindi). And two points that hit home: teachers (in the aggregate, I presume) haven’t had a pay raise of more than 1% since the 1990-1991 school year, and that the way to get students to learn is to get them to build things and explain what they built to other students.

I was exposed to the latter theory in my Software Engineering course at “IPFW”:http://ipfw.edu. Dr. Modesitt, on the first day of class, presented the idea that there are five levels of learning, with lecturing being the least effective, up to building as the second most effective, and teaching others being the most effective. Now Dr. Modesitt used this to set the stage for how the class would be structured, with most of the work occuring at those upper two levels. And that class was like drinking from the firehose, but I _did_ learn a lot. So although it was in a completely different context, I can attest that it really works.

It was also interesting to hear that Goldberg had built a course/school system on Zope 2 (hmm, that “sounds familiar”:http://tech.canterburyschool.org/tech/STOA), and that she’s currently working on rewriting it in Zope 3. And in building that from an “instructional” perspective versus “content management” perspective (which is what STOA(Student Teacher Online Applications) definitely has). The new tool, CourseCloud, uses XML to represent exercises and instructional content, with the goal being to present the exercises in multiple contexts.

fn1. What? I had two hours of sleep the night before, and wound up buying Age of Empires III for my MacBook at Fry’s and stayed up, well, later than planned.

And after lunch, it’s fun with Python on the Web. The first afternoon talk was “Ian Bicking’s”:http://ianbicking.org/ _Everything You Wanted to Know About WSGI but were Afraid to Ask_. OK, that’s just my title; it’s really “WSGI: An Introduction”:http://blog.ianbicking.org/going-to-pycon-2007.html. “WSGI”:http://en.wikipedia.org/wiki/WSGI is a specification for connecting pieces of web applications in Python. It’s something (like Eggs) that I can conceptually understand as being important. But (like Eggs) I have a hard time really groking it.

WSGI provides for this idea known as Middleware, which allows you to wrap your applications in, uh, other applications. Ian actually had a good litmus test for what is middleware: if you code is acting as both a client *and* a server, then by definition it is middleware. Ian also spent some time discussing the idea of “microframeworks”, pieces of software that don’t provide this full stack but can be combined in interesting ways to build something bigger. So yeah, WSGI: darn cool.

Following the WSGI talk was the Python Web Framework Panel, with representatives from Zope, CherryPy, Pylons, Django, TurboGears, Pyjamas, Spyce and Nevow[1]. Nothing really ground-breaking, but Bob Brewer of CherryPy did make an amusing comment: “Reinventing the wheel isn’t so bad if you’re really interested in wheels.” What did sort of annoy me is the multitude of comments from creators about “looking at Zope and running away screaming.” Look, I agree that Zope isn’t for everyone. I also agree with Phillip: “where Zope leads, Python follows”:http://dirtsimple.org/2007/01/where-zope-leads-python-follows.html. I don’t know when Zope became the Python web community’s collective bitch.

Now, off to my friend Vern’s talk about teaching with Python (and dumping that tired hag “hello, world”), then a nap, lightning talks and finally Shawn and I make our annual pilgrimage to Fry’s.

fn1. Apologies if I missed one; this is from memory.

“Philip J. Eby”:http://dirtsimple.org/, the mastermind behind things like PEAK and Python Eggs, wrote a “blog post”:http://dirtsimple.org/2005/08/chandler-begins-recovery-from-xml.html last month titled “Chandler Begins Recovery from XML”:http://dirtsimple.org/2005/08/chandler-begins-recovery-from-xml.html This follows his self-described rant from late last year, “Python is not Java”:http://dirtsimple.org/2004/12/python-is-not-java.html where he took developers to task for, among other things, turning to XML as the solution to all your data and configuration woes. The gist was, it might work for Java, but when mixed with Python it’s nothing but a boat anchor. So how is “Chandler”:http://osafoundation.org “recovering” from XML? By dumping it. Their system for extending Chandler, parcels, previously used an XML file to define extension points and connections (roughly — I won’t claim really deep knowledge here). The new system, championed by PJE, uses Python syntax and code — descriptors, registrations, etc — to accomplish the same thing. PJE’s argument, as I read it, hinges not on the idea that XML is inherently evil, but rather that using XML is often a sign of over-engineering. As a believer in YAGNI (Ya Ain’t Gonna Need It) in software development, I can agree with that.

So what does this have to do with ccPublisher 2, and more importantly the delayed Developer Preview packages? Let me address the two parts of that question in sequence.

First, what does it have to do with ccPublisher 2? A major design goal of ccPublisher 2 is enabling third-party contributions, in the form of extensions and derivative applications. We’re doing this in a number of ways, including basic things like improved documentation. A major tactic, though, is the use of loosely coupled pieces of code that are intentionally ignorant about one another. For example, an MP3 file contains metadata in the form of ID3 tags. The object that wraps the generic file doesn’t know this, but it knows it can say “Hey, all you components — anyone know anything about this here file-thingy?” and an adapter object will respond with everything it knows. So in theory (and in practice, actually — this mostly works already) you can swap out or add objects that respond without major surgery. A huge improvement over the ccPublisher 1 codebase. All these bits of code are tied together by XML files that describe subscriptions, adapters and interfaces. I chose the ZCML format, developed as part of the Zope3 project, because I was familiar with it, and because I was reasonably confident I could use code from Zope3 to make my life easier. And it turns out I was right — ZCML was reasonably easy to separate from Zope3. It’s also made life somewhat easier, and it will let non-coders who need customized metadata fields to add them relatively easily (note that I haven’t actually decided if non-coders will actually need to do this, it’s just the easiest rationalization right now).

So after reading Philip’s rant(s) and background on deprecating XML configuration files in Chandler, I started thinking about the suitability of ZCML for the task at hand. ZCML makes a lot of sense for Zope3 — a big advantage (in my mind) of Zope3 over previous versions is that (in theory) you can take existing classes that model data or behavior and use them in Zope without making them Zope-specific. In that case moving the configuration and registration into external files helps with that goal. ccPublisher doesn’t have that goal or that baggage — anything used in ccPublisher will probably be ccPublisher-ized in some way. I’m not convinced that ZCML is the wrong choice for ccPublisher, but the talk has had the effect of making me think about it more now than I did earlier.

Now, on to the second question — why the delay. Well, it turns out that ZCML makes life a bit more difficult when packaging your code. Linux wasn’t a problem — you just use distutils and specify a recursive-include in the MANIFEST.in. Windows is a different story — we’re using py2exe, which means there are two problems: first, py2exe ignores the MANIFEST.in when finding modules to include. This makes a certain perverse sense, but it still bites you in the ass. After hacking up a script to include the ZCML along side the Python byte-code, though, you [I] realize something — the byte code is in a ZIP file, and your code doesn’t traverse into ZIP files (ala “PEP 302″:http://www.python.org/peps/pep-0302.html) to retrieve the ZCML resources properly. Additionally, even though you can set up a dummy tree along side library.zip containing the ZCML, the Python pathing makes things, well, ugly. Really ugly. Sigh.

So ccPublisher 2 Developer Preview is slightly delayed on Windows while we make some retrofits to the code. The solution I’ve decided on is “Python Eggs”:http://peak.telecommunity.com/DevCenter/PythonEggs. Eggs let you package your Python code, make explicit declarations about dependencies and (most importantly for this situation) “access non-code resources”:http://peak.telecommunity.com/DevCenter/PythonEggs#accessing-package-resources stored in the package.

So interestly, PJE appears to have the ability to spark concern as well as solve weird edge-case problems.