Friday, March 30, 2012

django celery with remote worker nodes

I set up rabbitmq and celery. Plenty of good tutorials online about how to do that. Then I wanted a bunch of different linode boxen all running the same django project, with the following setup:

1 server running mysql and nothing else
1 server running nginx serving http requests and routing tasks to rabbitmq / celery
1 server running rabbitmq and celery and django
N boxes running django and celery

Turns out, it's easy!
  • All of the above hook into the mysql server by setting the HOST and PORT settings in the django settings.
  • Each slave celery box uses an environment variable to take care of any individual settings it might need, but in general each of them uses django-celery's BROKER_HOST and BROKER_PORT options to connect to the rabbitmq server.
  • using fabric makes deploying code to all of them fairly simple
Believe it or not, rabbitmq effortlessly figures out who's got a free worker between all of your boxes and just does it.