Tuesday, March 18, 2014

how to tell if an API works well before you write the integration

I connect with APIs for a living, or at least for part of a living. I do it all the time.
Often the ops team will ask me - how long to do an integration with ESP X, or Analytics firm Y? short of actually doing the integration, it used to be hard to give a good answer to this question. Now, tho, I have the perfect heuristic - it's never failed me.

Just take a look at the API docs. doesn't really matter what they say; if they're well formatted, easy to navigate, and well thought through, API integration will be a breeze. That simple.

Something like this makes an easy heuristic:

  • no API docs - refuse to do the job
  • API docs not available online but emailed to you in pdf form (or shared as gdoc): 40-50 hours
  • API docs in one monolithic html page: 30-40 hours (obviously all of this depends on depth of integration too)
  • docs have a formatted nav bar at left and some readable styling: 20-30 hours
  • docs have the above in addition to "web 3.0" style formatting, and maybe some helpful examples: 10-20 hours
  • docs have the above plus toggleable examples in different programming languages, maybe a link to wrapper libraries, are well written, and code blocks are formatted with special code block formatting: 1-10 hours.
you can use this, free of charge :)
best part of it is that I taught the ops team how to do this, and now they know exactly what estimate I'll give before they even ask me. Pretty soon they won't even need me at all, what with services like zapier.

Tuesday, February 4, 2014

for the record

the python CSV package is not as useful as it could be. no unicode support, can't handle all sorts of useful things like switching out the line delimiter (unless I'm missing something), and frequently just falls over for no reason.

I always try to use it, because modularity, and DRY, and yet I end up just writing up a quick csv parser myself for the job at hand like 60% of the time. Makes me think I should look for other CSV parsing packages, or just write my own.

Wednesday, October 30, 2013

more python SUDS advice

Everybody knows the a SOAP API is a horrid thing to work with, even a well designed one. But everybody also knows that SOAP is going to be around for a while.
Recently, while working with the ExactTarget SOAP API in Python, I figured out another helpful trick with SUDS, which is the main python SOAP wrapper.

Let me preface this by saying that suds is not a great library; particularly, it was written by and for people from another era of web development and does not stand up to today's needs. Why doesn't somebody write a better one? because everyone who does modern web dev hates SOAP in the first place.

But anyway, one annoying feature of suds is that when you create an object from its object factory, it always fills in all properties with an empty string, even those that are optional and have sane defaults. So essentially, it breaks all objects right out of the gate, forcing you to go through one by one and choose those sane defaults explicitly.

However, there's another way - for any properties that are giving you trouble in this fashion, just delete them after creating the object:
del object.AnnoyingProperty

Suds will then not send the property, allowing the API server to choose the sane default.

facepalms: 6

Friday, October 18, 2013

How to do get a good security audit for your startup

Recently at SimpleRelevance we decided it was time for a security audit on our website and especially our dashboard. We have quite a bit of client information that we would never want to share with the world.

Security audits are kind of like STD testing - even when you feel 100% fine, it's better safe than sorry.
Unfortunately, also like STD testing, if you cheap out, you won't get tested for everything.

So our problem was - how do we not get ripped off, while still ensuring a comprehensive audit - we're talking more than just simple pentests here. It's so hard to verify that you've gotten a good security audit, since the company can just come back and say "we didn't find any issues - your system is bulletproof!", and you can't prove that negative. In the end we did a lot of research and interviews and chose based on the data at hand. But since then I've thought of a fun way that might work even better.


1) through research, find N>1 companies that you'd consider paying for research.
2) bargain each one down (you were going to do this anyway, right? We ended up paying about 65% of the original ask for the company we chose).
3) tell each one: "I'll make you a deal. I'll pay you full price on the test, if you agree to the stipulation that I'm making this same deal with another security company, and whoever finds fewer or less important security holes doesn't earn any money, and whoever finds the most gets the full price [plus 10% if you need to sweeten it]".
4) Some of the companies are going to say no. Decide which company is most excited by the idea of the deal - they are clearly the winner, so congrats - you found the best one! Tell them none of the others would take the deal, so unfortunately you can't do it, but you'll still pay them the discounted rate you'd already agreed upon for a full audit, and you applaud their style.

Tuesday, June 11, 2013

ipython and the django shell: strange scoping errors

this is a minor issue that has annoying repercussions. on most versions of django, if you use ipython and start it with ./manage.py shell, you cannot define global variables and then use them in local functions. it gets ugly quick. more info here:
https://github.com/ipython/ipython/issues/62

patch is here:
https://github.com/django/django/pull/512/files

and it definitely fixes the issue.

Tuesday, January 15, 2013

python httplib2 7.x and ssl

this is not a completely new issue, and the internet helped me find the solution in about 5 minutes, but it was not very clear or obvious so I thought I'd write it down succinctly for posterity:

Python's httplib2 package did not validate ssl certificates in versions < 7.0, and started to from 7.0 and up. Unfortunately, it ships with its own set of trusted ssl certificates which comprise only a medium-sized subset of all of your favorite sites' certs (for instance, wikipedia fails when pinged at https://en.wikipedia.org!). The error looks like this:

  File "/home/deploy/.virtualenvs/sandbox.koaladeal.com/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1597, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)

  File "/home/deploy/.virtualenvs/sandbox.koaladeal.com/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1345, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)

  File "/home/deploy/.virtualenvs/sandbox.koaladeal.com/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1281, in _conn_request
    conn.connect()

  File "/home/deploy/.virtualenvs/sandbox.koaladeal.com/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1036, in connect
    raise SSLHandshakeError(e)

SSLHandshakeError: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

 So the solution is to use ubuntu's system default cert file, which lives at /etc/ssl/certs/ca-certificates.crt. I ended up overwriting the one that shipped with httplib2, which fixes the problem globally, but fails if we ever reinstall httplib2. Hence this blog post for posterity. If anyone knows of another way to globally install a new certificates file for httplib2 without changing the package itself, I'm all ears. I might just onboard the package to our project (but then I have to remember never to pip install it...).

by the way, this page was most helpful.

facepalms: 3

Tuesday, November 20, 2012

how to not run your business

Inline image 1


tell me - if you're going to pay for hosting for a corporate wordpress blog, what's the chance you're not going to use your corporate email account and what's the chance you're not going to host the blog at your corporate url? in other words, for what business would this error message not come up?

None. Facepalms: 5 (horrifically hilarious, luckily not a problem I have to deal with today.)