1 server running mysql and nothing else
1 server running nginx serving http requests and routing tasks to rabbitmq / celery
1 server running rabbitmq and celery and django
N boxes running django and celery
Turns out, it's easy!
- All of the above hook into the mysql server by setting the HOST and PORT settings in the django settings.
- Each slave celery box uses an environment variable to take care of any individual settings it might need, but in general each of them uses django-celery's BROKER_HOST and BROKER_PORT options to connect to the rabbitmq server.
- using fabric makes deploying code to all of them fairly simple
Believe it or not, rabbitmq effortlessly figures out who's got a free worker between all of your boxes and just does it.