If you get a UnicodeDecodeError from the cc.engine (you'll see this if it's
running in the foreground) when you try to access the http://host:9080/license/
then it's likely that the install of python you are using is set to use ASCII
as it's default output.  You can change this to UTF-8 by creating the file
/usr/lib/python<version>/sitecustomize.py and adding these lines:
  import sys
  sys.setdefaultencoding("utf-8")

This always struck me as a bit inelegant—having to muck with something outside my application directory. After all, this belief that the application should be self-contained is the reason I use zc.buildout and share Jim’s belief in the evil of the system Python. Like a lot of inelegant things, though, it never rose quite to the level of annoyance needed to motivate me to do it right.

Today I was working on moving the license engine to a different server² and ran into this problem again. I decided to dig in and see if I could track it down. In fact I did track down the initial problem—I was making a comparison between an encoded Unicode string and without specifying an explicit codec to use for the decode. Unfortunately once I fixed that I found it was turtles all the way down.

Turns out the default Zope 3 page template machinery uses StringIO to collect the output. StringIO uses, uh, strings—strings with the default system encoding. Reading the module documentation, it would appear that mixing String and Unicode input in your StringIO will cause this sort of issue.

Andres suggested marking my templates as UTF-8 XML using something like:

 < ?xml version="1.0" encoding="UTF-8" ?>

but even after doing this and fixing the resulting entity errors, there’s still obviously some 8 bit Strings leaking into the output. In conversations on IRC the question was then asked: “is there a reason you don’t want a reasonable system wide encoding if your locale can support it?”

I guess not³.

UPDATE Martijn has a tangentially related post which sheds some light on why Python does/should ship with ascii as the default codec. At least people smarter than me have problems with this sort of thing, too.

¹ Yes, I may be a bit biased—I wrote the Zope3/Grok implementation. Of course, I wrote the previous implementation, too, and I can say without a doubt it was… “sub-optimal”.

² We’re doing a lot of shuffling lately to complete a 32 to 64 bit conversion; see the CC Labs blog post for the harrowing details.

³ So the warning remains.

Someday I’ll write up my thoughts on distributed version control and “convention versus configuration”, which seem to overlap in this deployment. But not today.

Last week I was looking at the upcoming features in Subversion 1.5. Asheesh had pointed out the merge tracking feature, which sounded lovely. And it probably will be. The thing I discovered, though, is that you can get it today in the form of svnmerge.

Sure, it’ll be great to get this feature into the core application (and an interactive mode ala darcs would be slick, too), but to paraphrase Scarlett O’Hara, “with god as my witness, I will never svn merge again”.

Java Applets Reborn from Dion Almaer on Vimeo.

So the most interesting things (to me):

• The plugin itself is mostly written in Java. I love these sort of recursive language exercises (see PyPy), if nothing else for their turtles all the way down effect. But I suppose this actually makes a certain degree of sense: if you have a fixed set of resources and you’re putting lots of them towards improving the JVM/JRE, you can take advantage of those improvements by writing your plugin in Java. They don’t talk about the architecture of the old plugin much, but I wouldn’t be terribly surprised to find out the native-code “shim” used to communicate between the browser and the JVM is easier to maintain than the old plugin was.
• Applets run in a completely independent process (not just thread).

• An applet can demand to run with a particular JRE version. They label this “enterprise support.” I’d call it “sanity support.”

  None of this really speaks to whether it’s too little, too late. It seems like a good move on Sun’s part, especially given the recent attention on rich internet applications (RIAs) has focused (not necessarily undeservedly) on Flash, Adobe AIR (the vapor-ware jokes write themselves, don’t they?) and Microsoft Silverlight. The amount of press and attention may indicate this is an area whose time has come. And while I admit having a begrudging soft spot for Java, it seems that companies (especially capital-”E” “enterprise” companies) who already have lots of in house Java expertise could benefit from being able to extend that knowledge down to the desktop. Maybe.

  ]]> http://yergler.net/blog/2007/10/22/sun-rewrites-java-browser-plugin/feed/ http://yergler.net/blog/2007/03/06/for-some-definition-of-online/ http://yergler.net/blog/2007/03/06/for-some-definition-of-online/#comments Tue, 06 Mar 2007 14:08:27 +0000 Nathan Y. http://yergler.net/blog/2007/03/06/for-some-definition-of-online/ Starting yesterday afternoon I found that I couldn’t commit code to Creative Commons’ Subversion repository at Sourceforge. “Just another intermittent outage,” I thought. I even took a look at the site status page, saw that Subversion was “online” and figured it’d be resolved by the morning.

  Well this morning it’s still down, so I took another look. Yup, still online:

  

  And then you scroll down and find the truth:

  

  So can someone explain to me how “completely inaccessible due to a massive DDOS attack” (my words) is in any way equivalent to “online”?

  ]]> http://yergler.net/blog/2007/03/06/for-some-definition-of-online/feed/ http://yergler.net/blog/2007/02/23/pycon-day-1-morning-sessions/ http://yergler.net/blog/2007/02/23/pycon-day-1-morning-sessions/#comments Fri, 23 Feb 2007 18:36:19 +0000 Nathan Y. http://yergler.net/blog/2007/02/23/pycon-day-1-morning-sessions/ This morning following the “keynote”, I attended the “State fo Zope Panel”. As Jim pointed out at the start (and as I remember from past years), this is typically part one of the Jim Show (part two being about Zope 3 or the Zope Component Architecture). This year the panel consists of four developers, with someone representing Zope 3, Zope 2, Plone and Zope writ large (including the Zope Foundation). After hearing about the state of Plone, Zope 2, and Zope 3, Jim spoke about the Zope Foundation. I was particularly suprised to that there are only 48 committer members in the foundation. Given that I’m one of them, well… wow¹. I appreciated Jim’s frankess about the challenges the Foundation faces, as well as his thoughts on Zope as a Pluggable Application versus Zope as a Library.

  Based on the questions to the panel, there still seems to be lots of angst over the future of Zope 2 with respect to Zope 3. Jim made some deceptively sane comments about Zope 2 and Zope 3 last year², and it seems like people still want to know “when will Zope 3 be the real Zope” which implies the corollary, “when will Zope 2 die?” I think this agnst stems from the desire not to invest in a “dead” technology, or to invest too early in an immature one. But the answer just doesn’t matter. Zope 2 is appropriate for use cases which Zope 3 is not, and vice versa.

  ---

  Following the Zope Panel I attended Ernesto Posse’s talk on writing parsers using grammar transformation. Posse set out to implement the LL(1) parsing algorithm based on its simplicity and speed. However, given the limited expressiveness of an LL(1) grammar, his solution (called aperiot) was to writer a parser grammar transformation which transforms a non-LL(1) grammar to an LL(1) grammar. Aperiot allows you to generate a parser as a Python module from a give grammar definition.

  Grammar parsing is one of those topics which I find academically interesting, but which I have a real problem following in the course of a talk. Especially with 2 hours of sleep. But from what I was able to absorb, I think aperiot is something I’d investigate if I needed to write a parser.

  ---

  The final morning talk I attended was an overview of developing desktop applications with Dabo. Ed presented on Dabo two years ago at PyCon is DC; I remember it well because I was in the same track as him³, presenting about cross-platform development with wxPython. If I recall correctly, Ed’s talk in DC talked a lot about the automated database to UI development. Today’s talk focused on how Dabo wraps the wxPython API to make it more “Python-ic”. Ed has a really good point: wxPython is powerful, and it looks great on each platform, because it uses native widgets. But when it comes to the API, knowing how C++ works is unfortunately a benefit, because it’s C++ roots show through⁴.

  After some slides presenting an overview of Dabo’s API, Ed presented a deceptively simple example program. The example loads an image, and allows the user to rotate, resize, stretch, clip, etc. Image handling is a particular token of pain in wxPython, so the demo really resonated with me. Particularly cool points:

  • Loading an image is as simple as assigning a file path to the picture property of an image control; Dabo handles the loading process.
  • Python objects can act as “data sources”; more accurately, Dabo has a really flexible binding method, which allows you to do things like create a drop-down that changes the image clipping mode (this is really simple, but hey, saving 4 lines of code is saving 4 lines of code).

    For his final image demo, Ed demonstrated adding drag and drop support to the image control. Drag and drop is… well, it still doesn’t work exactly right under some edge cases in ccPublisher or ccLookup, and that code’s been reworked half a dozen times with special exceptions and handlers. I think the total code to add support was something like 3 or 4 lines. If it works half as well as the demo, that’s really cool.

    Ed wrapped up his talk with a demonstration of creating a wxPython grid bound to a database. Holy crap. I’m not sure exactly when I’ll use Dabo, but the next time I’m doing cross-platform desktop development, dabodev.com will be one of my first stops.

    ---

    ¹ I say “wow” only because my contributions have been sparse and sporadic, to put it generously.

    ² Zope 3 doesn’t need to replace Zope 2, maybe it never will, or maybe there should be something called Zope 5 that just builds Zope 2 as an application on the Zope 3 library… or something like that. It was a year ago, and I have been to Rio since then.

    ³ It was sort of the year of the desktop development talk, if I recall correctly.

    ⁴ Admittedly they show through to a much smaller degree now, but there are still moment

    ]]> http://yergler.net/blog/2007/02/23/pycon-day-1-morning-sessions/feed/ http://yergler.net/blog/2007/02/13/snowed-in/ http://yergler.net/blog/2007/02/13/snowed-in/#comments Tue, 13 Feb 2007 22:54:45 +0000 Nathan Y. http://yergler.net/blog/2007/02/13/snowed-in/ So Fort Wayne and the surrounding county is under a “Level 2 Snow Emergency” until further notice. This basically means that if you park on the street, you will be towed. If you drive, you better have a damn good reason. So I’m doing neither. I’m sitting at home coding, of course.

    I gave up the two-machine (desktop and laptop) dichotomy last year, selling my iMac on eBay when I found it just wasn’t getting the use I thought it needed to. In the interim I’ve had moments where I thought that was exactly the right decision and moments where I just wasn’t sure. Usually I came down on the “it’d probably be nice to have a desktop” side: sure, there are all sorts of problems with keeping things in sync, but the (relatively) cheap storage for things like audio and video means I don’t have to pay a premium for a big ass laptop drive.

    Over the weekend I received a check for some consulting work I did, so yesterday I went to my local Apple dealer and bought a new iMac. I have admittedly mixed feelings about Mac OS X (see Mark’s thoughts, which mirror mine to a certain extent), but I figured I either have to pay the Microsoft Tax or the Apple Tax, so I may as well have a machine I find aesthetically pleasing that will make a great Ubuntu box¹.

    Anyway, I haven’t installed Ubuntu yet, but I was playing with Dashcode a bit this afternoon. So if you like ccMixter (and I think you should), check out the ccMixter Editors’ Picks widget.

    

    When you install it in the Dashboard it lets you easily play the latest Editors’ Picks from ccMixter. And if you want more information, just click the ”>” button and it’ll open the selected track at ccMixter.org.

    ---

    ¹ And it’s not an HP.

    ]]> http://yergler.net/blog/2007/02/13/snowed-in/feed/ http://yergler.net/blog/2007/01/30/the-little-things/ http://yergler.net/blog/2007/01/30/the-little-things/#comments Tue, 30 Jan 2007 13:00:21 +0000 Nathan Y. http://yergler.net/blog/2007/01/30/the-little-things/ I’m knee deep in several projects right now, some using darcs and some using subversion for version control. There are things I like about both, but that’s a different topic. There’s really only one thing I loathe about Subversion, and it looks like this:

    
    $ svn propget svn:externals .
      svn: Inconsistent line ending style
    

    Arghhh!

    Insert rant about how the hell line endings get inconsistent on something Subversion should have complete control over here.

    ]]> http://yergler.net/blog/2007/01/30/the-little-things/feed/ http://yergler.net/blog/2007/01/10/wiki-licensing-for-moinmoin/ http://yergler.net/blog/2007/01/10/wiki-licensing-for-moinmoin/#comments Wed, 10 Jan 2007 22:11:10 +0000 Nathan Y. http://yergler.net/blog/2007/01/10/wiki-licensing-for-moinmoin/ I’ve been using MoinMoin for managing the non-blog content of yergler.net for some time now, and its mostly been a great choice for me. Just enough through-the-web management to make maintaining the content less burdensome. One feature I noticed it was missing (compared to MediaWiki, which we use at work) is Creative Commons license support. After I started to do some digging I found that there is support for a wiki license page. If enabled, this adds some text to the editor pages notifying contributors that their content will be subject to the site license.

    

    So over the weekend I put together MoinLicense. MoinLicense is a pair of actions and a macro for MoinMoin which enables license selection. In addition to supporting a site-wide license, MoinLicense allows you to select a particular license for any page. Details, download information and known issues are in the wiki, natch.

    ]]> http://yergler.net/blog/2007/01/10/wiki-licensing-for-moinmoin/feed/ http://yergler.net/blog/2006/12/15/deploying-python-code-support-services/ http://yergler.net/blog/2006/12/15/deploying-python-code-support-services/#comments Fri, 15 Dec 2006 22:00:12 +0000 Nathan Y. http://yergler.net/blog/2006/12/15/deploying-python-code-support-services/ Just over a month ago I blogged about my first experience with zc.buildout. For those that don’t remember (shame on you!), zc.buildout is a Python tool developed by Jim Fulton of Zope fame for constructing software installations. “Buildouts”, if you will. The attraction to this over, say, just distutils or just setuptools is that you can go from a bare-bones Python installation and source code checkout (or tar-ball) to a fully functional application, with dependencies and the trimmings, in a predictable, sane way. The fact that dependencies are installed in a local folder makes this especially appealing for deploying code on hosts where you don’t have write access to site-packages. But enough re-hashing my last zc.buildout post.

    As I mentioned in my previous post, I’ve been using zc.buildout mostly to deploy the web applications that power Creative Commons’ web services These are written using CherryPy, and were previously invoked as simple CGI scripts. However, as usage grew, we needed to move them to independent processes, which Apache proxied to using mod_rewrite. zc.buildout provided a great way to generate the wrapper script for running that process, but early this week we ran into a problem: we didn’t have a good way to detect if the process died for some reason and needed to be restarted.

    I decided that a simple approach was best for the initial go-round, and wrote a small shell script that looks at the pid file, checks if it’s still running and if not starts the process. Put this in a cron job that runs every 10 minutes or so, and you have a poor man’s watchdog¹. After writing two of these, I realized that this is something that should really be automated as part of my software installation: I always want to have a “am I up?” check script available, and so my first custom zc.buildout recipe, build_script, is born.

    build_script is a recipe that takes a template file, fills in values using Python string formatting and marks it as executable. It has one template included, paster-check that we’re using to check on a paster process, and you can use templates that are part of the egg itself or are part of your codebase (via the template_dir setting).

    You can find the code for build_script itself in Creative Commons’ subversion repository (web view), and an example of how we’re using it for our web services is here.

    Suggestions, comments, questions all welcome, as are additional templates that might be useful to include in the egg itself.

    ¹ I’m sure there’s a better, more robust way to do this, but this is where we’re comfortable on the down time v. implementation time continuum. That said, if someone knows of an easy, more robust way, of course I’d like to hear about it.

    ]]> http://yergler.net/blog/2006/12/15/deploying-python-code-support-services/feed/