django celery with remote worker nodes

I set up rabbitmq and celery. Plenty of good tutorials online about how to do that. Then I wanted a bunch of different linode boxen all running the same django project, with the following setup:

1 server running mysql and nothing else

1 server running nginx serving http requests and routing tasks to rabbitmq / celery

1 server running rabbitmq and celery and django

N boxes running django and celery

Turns out, it's easy!

• All of the above hook into the mysql server by setting the HOST and PORT settings in the django settings.
• Each slave celery box uses an environment variable to take care of any individual settings it might need, but in general each of them uses django-celery's BROKER_HOST and BROKER_PORT options to connect to the rabbitmq server.
• using fabric makes deploying code to all of them fairly simple

Believe it or not, rabbitmq effortlessly figures out who's got a free worker between all of your boxes and just does it.

Porting Postgres to google cloud SQL

I know it's quite a peculiar problem - you have a postgres database, and suddenly you're migrating to Google App Engine and you want to use the new Google Cloud SQL (look it up, it's public and it's pretty cool).

Anyway, you need to port the old data. Google cloud sql is cool except for a big problem - sql dump imports have 0 error reporting - if it fails, it just turns red and tells you that an unknown error occurred. So here's how I made it work:

1. dump the postgres stuff selecting the tables you want with a couple extra options on:
2. pg_dump --column-inserts --data-only POSTGRES_DATABASE_NAME -t TABLE_NAME -t ANOTHER_TABLE_NAME -f NEW_FILE_NAME.sql [note: you need to have psql privileges already here].
3. delete the top lines of the dump file created in 2) until the first "insert" line.
4. load it into mysql locally, where you can catch any errors:
5. mysql -u USER -p DATABASE_NAME < NEW_FILE_NAME
6. dump it from the local mysql:
7. mysqldump -u USERNAME -p --add-drop-table MYSQL_DATABASE_NAME TABLE_NAME ANOTHER TABLE_NAME> FIXED_SQL_DUMP_FILE.sql
8. add as the first line in the new dumpfile: "use DATABASE_NAME;" (ignore the quotes, add the name of the database you want the data loaded into on google).
9. Now you can load this new file into a google cloud storage bucket using their web browser gui and from there import it into cloud sql.
10. pray, as you wait for the stupid thing with no error reporting to turn green.

facepalms: 7

rails fields_for, object sorting

So what I needed to do was select a subset of one model which happened to have a key into another, sort it by an attribute of the other model, and display a list of the results in a form.
here's the query to find, filter, join, and order:

@user_categories = Category.joins(:user_categories).where('user_categories.user_id' => current_user.id).order("name asc")

and here's the bit of the form:

<%= f.fields_for :user_categories, @user_categories do |user_category| %>

and then to access the objects:

<%= user_category.label :name, user_category.object.name %>

Things to notice:
the fields_for uses a parameter :user_categories which references the name of the model, followed by the object which is actually a list of objects of type Category, so I'm assuming this is to let rails know how to treat it. Also, note that to reference the actual objects, you have to use model_class.object.method. the model_class is actually some sort of form builder helper before the call to .object.

This only took an hour to figure out - #facepalms = 3.