redis vs rabbit with django celery

if you're planning on putting a django celery app into heavy production use, your message queue matters. At SimpleRelevance we used RabbitMQ for months, since it's probably the most famous of the available options.
It took about 8 seconds to restart out celery workers, but that was fine. In general it was fairly reliable, and the way it routed tasks to different celery workers on different boxes often felt like magic.

However, getting results of a task back using rabbit as a result backend was a different story - it often took minutes, even hanging the box it was on. And these weren't big results either.

So for the record here, we switched to Redis. Not only is restarting about 3X faster, but as a results backend it also wins - no more hanging, and results come back as soon as they're ready. My sysops also tells me it was much easier to install and configure.
boom.

----
update!
actually it turns out redis starts to perform very badly when faced with a deep queue in our production environment. So the optimal setup for us turns out to be RabbitMQ for the queue handling, and Redis for the result backend.

"Error in service module" Ubuntu pam login fail

This was a strange one. One of the newer computers at the lab where I moonlight, running Ubuntu 11.04 (yes, I probably should upgrade to the next LTS, I guess), suddenly stopped logging in. This was preceded by a pink screen and a bunch of errors about the harddrive (so they tell me; I wasn't there. oh, the lives of not IT folk - like living underwater).
So now, whenever they logged in through the GUI, nothing happened - click the user, type the password, straight back to login screen.
So I ctrl alt F1 to TTY1, and log in, and the only error I got was:
"error in service module". Not helpful.
Googling that was helpful in that it started to point the blame at PAM, or Pluggable Authentication Modules, which I now know way too much about.

But anyway I ended up booting into recovery mode which got me past the login, and then overwriting the /etc/init/tty1.conf to skip login even when not in recovery mode following this advice, which allowed me to access the internet and external drives on the machine.

Then I realized that ubuntu logs everything, and I checked /var/log/auth.log, which mentioned a bunch of missing files in /etc/pam.d/ and other fun places. All such fun directories were empty. very strange. So I copied over all of the files from another ubuntu machine into said directories, including a bunch of .so files, and what do you know, login worked. That simple.

Moral of the story - read the logs when something goes wrong that you don't understand. Ubuntu writes a lot of logs, and there's a reason for that. I know I made it seem easy here but it probably took 2-3 hours to do all of the above. facepalms: 4

another python suds tip

If you are having trouble creating nested XML for array objects like this:

<configurations>
 <configuration>
   stuff
 </configuration>
</configurations>

while using suds' Factory methods, you should try to create the struct from scratch, by passing whatever method is appropriate a list of dictionaries or whatever nested structure applies:

[{'Configuration':obj},{'Configuration':obj}]

facepalms: a million

suds empty tag issue

I just posted a really nice Stack Overflow solution about this:
http://stackoverflow.com/questions/9388180/suds-generates-empty-elements-how-to-remove-them
The general point is that although Suds is an awesome python library that lets you connect to SOAP clients (I mean, really? welcome to the 21st century, people) with relative ease, it has the bad habit of adding empty tags for optional properties of objects. This tends to confuse (poorly written) API endpoints.

facepalms: 6

django celery with remote worker nodes

I set up rabbitmq and celery. Plenty of good tutorials online about how to do that. Then I wanted a bunch of different linode boxen all running the same django project, with the following setup:

1 server running mysql and nothing else

1 server running nginx serving http requests and routing tasks to rabbitmq / celery

1 server running rabbitmq and celery and django

N boxes running django and celery

Turns out, it's easy!

• All of the above hook into the mysql server by setting the HOST and PORT settings in the django settings.
• Each slave celery box uses an environment variable to take care of any individual settings it might need, but in general each of them uses django-celery's BROKER_HOST and BROKER_PORT options to connect to the rabbitmq server.
• using fabric makes deploying code to all of them fairly simple

Believe it or not, rabbitmq effortlessly figures out who's got a free worker between all of your boxes and just does it.

app engine and appcfg import errors

If you get an error that looks like the below, be sure that you are importing the correct version of django(which is to say, that if you are using the builtin version, be sure you are not also importing another version of django from your sitepackages). Appcfg automatically follows all symlinks and packages up everything on your pythonpath, and the import order in production is different (actually reversed!) from that in development.

I solved this problem by deactivating my virtualenv whenever I deploy.

facepalms: 8 (3 hours of nonsense for a 1-line fix!)

Traceback (most recent call last): File "/base/python27_runtime/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 193, in Handle result = handler(self._environ, self._StartResponse) File "/base/python27_runtime/python27_lib/versions/third_party/django-1.2/django/core/handlers/wsgi.py", line 232, in __call__ self.load_middleware() File "/base/python27_runtime/python27_lib/versions/third_party/django-1.2/django/core/handlers/base.py", line 40, in load_middleware mod = import_module(mw_module) File "/base/python27_runtime/python27_lib/versions/third_party/django-1.2/django/utils/importlib.py", line 35, in import_module __import__(name) File "/base/python27_runtime/python27_lib/versions/third_party/django-1.2/django/middleware/transaction.py", line 1, in from django.db import transaction File "/base/python27_runtime/python27_lib/versions/third_party/django-1.2/django/db/__init__.py", line 77, in connection = connections[DEFAULT_DB_ALIAS] File "/base/python27_runtime/python27_lib/versions/third_party/django-1.2/django/db/utils.py", line 92, in __getitem__ backend = load_backend(db['ENGINE']) File "/base/python27_runtime/python27_lib/versions/third_party/django-1.2/django/db/utils.py", line 50, in load_backend raise ImproperlyConfigured(error_msg) ImproperlyConfigured: 'google.appengine.ext.django.backends.rdbms' isn't an available database backend. Try using django.db.backends.XXX, where XXX is one of: 'dummy', 'mysql', 'oracle', 'postgresql', 'postgresql_psycopg2', 'sqlite3' Error was: cannot import name backends

app engine and datastore keys

say you know the key name's string of a Model instance and you want to fetch it from the datastore. It goes like this:

Model.get(db.Key.from_path("Model",key_name))

why so complicated? why can't you just Model.get(key_name)? Couldn't tell you, but was getting inexplicable errors that way.

facepalms: 4