for the record

the python CSV package is not as useful as it could be. no unicode support, can't handle all sorts of useful things like switching out the line delimiter (unless I'm missing something), and frequently just falls over for no reason.

I always try to use it, because modularity, and DRY, and yet I end up just writing up a quick csv parser myself for the job at hand like 60% of the time. Makes me think I should look for other CSV parsing packages, or just write my own.

more python SUDS advice

Everybody knows the a SOAP API is a horrid thing to work with, even a well designed one. But everybody also knows that SOAP is going to be around for a while.
Recently, while working with the ExactTarget SOAP API in Python, I figured out another helpful trick with SUDS, which is the main python SOAP wrapper.

Let me preface this by saying that suds is not a great library; particularly, it was written by and for people from another era of web development and does not stand up to today's needs. Why doesn't somebody write a better one? because everyone who does modern web dev hates SOAP in the first place.

But anyway, one annoying feature of suds is that when you create an object from its object factory, it always fills in all properties with an empty string, even those that are optional and have sane defaults. So essentially, it breaks all objects right out of the gate, forcing you to go through one by one and choose those sane defaults explicitly.

However, there's another way - for any properties that are giving you trouble in this fashion, just delete them after creating the object:
del object.AnnoyingProperty

Suds will then not send the property, allowing the API server to choose the sane default.

facepalms: 6

ipython and the django shell: strange scoping errors

this is a minor issue that has annoying repercussions. on most versions of django, if you use ipython and start it with ./manage.py shell, you cannot define global variables and then use them in local functions. it gets ugly quick. more info here:
https://github.com/ipython/ipython/issues/62

patch is here:
https://github.com/django/django/pull/512/files

and it definitely fixes the issue.

python httplib2 7.x and ssl

this is not a completely new issue, and the internet helped me find the solution in about 5 minutes, but it was not very clear or obvious so I thought I'd write it down succinctly for posterity:

Python's httplib2 package did not validate ssl certificates in versions < 7.0, and started to from 7.0 and up. Unfortunately, it ships with its own set of trusted ssl certificates which comprise only a medium-sized subset of all of your favorite sites' certs (for instance, wikipedia fails when pinged at https://en.wikipedia.org!). The error looks like this:

  File "/home/deploy/.virtualenvs/sandbox.koaladeal.com/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1597, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)

  File "/home/deploy/.virtualenvs/sandbox.koaladeal.com/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1345, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)

  File "/home/deploy/.virtualenvs/sandbox.koaladeal.com/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1281, in _conn_request
    conn.connect()

  File "/home/deploy/.virtualenvs/sandbox.koaladeal.com/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1036, in connect
    raise SSLHandshakeError(e)

SSLHandshakeError: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

 So the solution is to use ubuntu's system default cert file, which lives at /etc/ssl/certs/ca-certificates.crt. I ended up overwriting the one that shipped with httplib2, which fixes the problem globally, but fails if we ever reinstall httplib2. Hence this blog post for posterity. If anyone knows of another way to globally install a new certificates file for httplib2 without changing the package itself, I'm all ears. I might just onboard the package to our project (but then I have to remember never to pip install it...).

by the way, this page was most helpful.

facepalms: 3

celery + djcelery problem with virtualenv and virtualenvwrapper

this is a tricky one - out of nowhere, production env boxes started failing at celery startup with:



ImportError: cannot import name current_app
when importing djcelery. Versions of celery were fine.
It turns out if you import celery and run
import celery.current_app you'll see the real problem, which is that the virtualenv binary is out of sync with the new python binary from a recent security update - specifically, os.urandom has been changed/removed.

if you have virtualenvwrapper, and you let $ENV=YOUR_ENVIRONEMNT_NAME
So the answer is:


deactivate (in case an env is running)
cd ~/$ENV_HOME (.virtualenvs, for me)
rm $ENV/bin/python
virtualenv $ENV


this will rebuild your python binary with the correct python post-security fix, without losing any other packages. happy hacking!

another python suds tip

If you are having trouble creating nested XML for array objects like this:

<configurations>
 <configuration>
   stuff
 </configuration>
</configurations>

while using suds' Factory methods, you should try to create the struct from scratch, by passing whatever method is appropriate a list of dictionaries or whatever nested structure applies:

[{'Configuration':obj},{'Configuration':obj}]

facepalms: a million

suds empty tag issue

I just posted a really nice Stack Overflow solution about this:
http://stackoverflow.com/questions/9388180/suds-generates-empty-elements-how-to-remove-them
The general point is that although Suds is an awesome python library that lets you connect to SOAP clients (I mean, really? welcome to the 21st century, people) with relative ease, it has the bad habit of adding empty tags for optional properties of objects. This tends to confuse (poorly written) API endpoints.

facepalms: 6

app engine and appcfg import errors

If you get an error that looks like the below, be sure that you are importing the correct version of django(which is to say, that if you are using the builtin version, be sure you are not also importing another version of django from your sitepackages). Appcfg automatically follows all symlinks and packages up everything on your pythonpath, and the import order in production is different (actually reversed!) from that in development.

I solved this problem by deactivating my virtualenv whenever I deploy.

facepalms: 8 (3 hours of nonsense for a 1-line fix!)

Traceback (most recent call last): File "/base/python27_runtime/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 193, in Handle result = handler(self._environ, self._StartResponse) File "/base/python27_runtime/python27_lib/versions/third_party/django-1.2/django/core/handlers/wsgi.py", line 232, in __call__ self.load_middleware() File "/base/python27_runtime/python27_lib/versions/third_party/django-1.2/django/core/handlers/base.py", line 40, in load_middleware mod = import_module(mw_module) File "/base/python27_runtime/python27_lib/versions/third_party/django-1.2/django/utils/importlib.py", line 35, in import_module __import__(name) File "/base/python27_runtime/python27_lib/versions/third_party/django-1.2/django/middleware/transaction.py", line 1, in from django.db import transaction File "/base/python27_runtime/python27_lib/versions/third_party/django-1.2/django/db/__init__.py", line 77, in connection = connections[DEFAULT_DB_ALIAS] File "/base/python27_runtime/python27_lib/versions/third_party/django-1.2/django/db/utils.py", line 92, in __getitem__ backend = load_backend(db['ENGINE']) File "/base/python27_runtime/python27_lib/versions/third_party/django-1.2/django/db/utils.py", line 50, in load_backend raise ImproperlyConfigured(error_msg) ImproperlyConfigured: 'google.appengine.ext.django.backends.rdbms' isn't an available database backend. Try using django.db.backends.XXX, where XXX is one of: 'dummy', 'mysql', 'oracle', 'postgresql', 'postgresql_psycopg2', 'sqlite3' Error was: cannot import name backends

Dynamic Databases in Django

My current (paid) project has me managing a separate database for every client in Django. This has been a great challenge. Since 1.2, Django has had multidb support, so that's not hard - the hard part is all of the edge cases.

For instance, we want to be able to add clients. On the fly. We plan to have many - like more than 20. So we certainly don't want to have our database definition in settings, all written out like the Django tutorials. At the very least, a loop over db names.

def add_db_to_databases(DATABASES,name):

if name in DATABASES:

return DATABASES

DATABASES[name] = {

'HOST': 'localhost',

'PORT': '',

'NAME': name,

'USER': '',

'PASSWORD': '',

'ENGINE': '',

'OPTIONS': {

'autocommit': True,

}

}

return DATABASES

for name in pro_dbs.names:

DATABASES = add_db_to_databases(DATABASES,name)

What I did there is take the names from another python file, which contains a simple python list of names.

I needed to be able to add clients on the fly. This is the hard part; as of yet I have two stumbling blocks with only partial workarounds.

1. I'd love to have the database names in a database themselves. Soooo much better than reading the python file with the names into a list, appending the new name to the list, than writing back to the file. But how to load from a database in the django settings file itself? It's been engineered not to allowed that.
2. I'd love to be able to update settings without restarting the server. You can do certain things in that vein by messing with django.conf.settings, but it's unclear how well that'll hold up under multithreading.

All in all, not a facepalm worthy subject, but very interesting.

web scraping and microformats

This proved very helpful to me. It's a short, concise explanation of how to extract data from the structured markup of a website in python. It's a no-brainer, but sometimes you need someone to show you the no-brainer before it becomes one for you.

The bottom line is that html is a string. It's also a tree. You can parse it either way. Microformats and the semantic web aren't only useful to big spidering companies like the goog; they're useful to all of us.

upgrading to ubuntu natty

annoyingness right off the bat. maybe a 7 on the facepalm scale.

1. the login screen allows you to choose between unity and classic, but only after you click the login name to get the password box out. before then, no indication of any possible choices on the bottom bar.

2. loaded up classic, but compiz settings were off.

3. my ATI graphics seem choppy (ouch).

4. virtualenvwrapper acting funky - specifically,

ImportError: No module named virtualenvwrapper.hook_loader

even though things generally seemed to work. Wow, I'm going to have to figure out how to inline code on blogspot here.

Anyways, the problem turned out to be that natty installed python 2.7.1 as default. Nobody warned me! I added the line

export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python2.6

To send everybody back to python2.6. Problem solved.

Unity isn't super annoying or buggy yet, but I'll give a shoutout if it becomes so.