g.raphaelli's weblog

Flickr.API 0.4.3

2010-01-25T01:41:15Z

Aaron kindly informed me of a problem with Flickr.API where calling flickr.urls.lookupUser with the required 'url' parameter returns the the contents of the resource pointed to by that url, instead of the expected API response. The problem is easily demonstrated using flickrcall.py, introduced in this post:

$ flickrcall.py flickr.urls.lookupUser "url=http://www.flickr.com/photos/graph"

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">

<html>
<head>
...

The trouble lies in the Flickr.API.Request init:

def __init__(self, url='http://api.flickr.com/services/rest/', **args):
    urllib2.Request.__init__(self, url=url)
    ...

Clearly I didn't anticipate 'url' as a valid argument - if you pass it in through **args it overwrites the default url parameter. Worse, if you wanted to use a different api endpoint, you couldn't even call this method with the required url parameter as python will complain about multiple values for the same 'url' argument. Here's a simple demonstration of what's happening:

>>> def foo(bar="baz", **kwargs): print bar

>>> foo()
baz

>>> foo("bar")
bar

>>> foo(bar="bar")
bar

>>> foo(**{"bar":"bar"})
bar

>>> foo("bar", **{"bar":"baz"})
TypeError: foo() got multiple values for keyword argument 'bar'

>>> foo(bar="bar", **{"bar":"baz"})
TypeError: foo() got multiple values for keyword argument 'bar'

For 0.4.3, I've simply changed the url keyword parameter to apiurl to make things work as I haven't thought of any backwards compatible way to fix this. Now we get the right response:

$ flickrcall.py flickr.urls.lookupUser "url=http://www.flickr.com/photos/graph"
<?xml version="1.0" encoding="utf-8" ?>
<rsp stat="ok">
<user id="83127823@N00">
	<username>raphco</username>
</user>
</rsp>

I'm hoping that unless you work at Flickr (we're hiring!), you're not likely to change the endpoint url and this change should only fix this particular problem for you.

Streaming Flickr Uploads in Python

2009-12-01T08:58:20Z

Flickr has supported up to 500 MB uploads since April. Unfortunately, uploading even a 100 MB file (easily less than 30 sec of high def video) with Flickr.API, possibly after reading this or this, uses about about 100 MB of memory, because the entire file is first read() before being sent off to Flickr. top reports this for a 94 MB video upload:

  PID COMMAND     #TH #PRTS #MREGS RPRVT  RSHRD  RSIZE  VSIZE
  627 python        1    20    133   98M   256K   101M   118M

A quick search of pypi turned up Poster, a library for Streaming HTTP uploads and multipart/form-data encoding. After hooking this up, the same 94 MB video upload is reported as:

  PID COMMAND     #TH #PRTS #MREGS RPRVT  RSHRD  RSIZE  VSIZE
  631 python        1    20    132 4904K   256K  7524K    24M

Notice the private memory usage drop from 98 MB to under 5 MB. The code to get it done looks like this:

import poster, poster.streaminghttp
poster.streaminghttp.register_openers()

def flop(fn):
    def flopped(a, **kwargs):
        (c, d) = fn(a, **kwargs)
        return (d,c)
    return flopped

flop swaps the (body generator, headers) tuple returned by Poster's multipart_encode into (headers, body) that Flickr.API's execute_request is expecting as seen in the next code block. multipart_encode can receive a custom boundary but we're basically just going to ignore that here. Then, when executing the upload request, this new form encoding can be used like:

photo = open(file, 'rb')
upload_request = Flickr.API.Request(url="http://api.flickr.com/services/upload",
    auth_token=token, photo=photo)
upload_response = api.execute_request(upload_request, sign=True,
    encode=flop(poster.encode.multipart_encode))

Now Poster takes care of transferring the file in 8KB chunks, instead of loading the whole file into memory.

Flickr.API Helper Script

2009-08-12T07:40:35Z

Here is a [formerly] little script I use to test out Flickr API calls: flickrcall.py. It grew by about 200 lines getting it ready for posting here and it's still not pretty. It does work though, so use it like this:

Create a config file (optional, you can specify the key and secret on the command line)

$ cat > ~/.flickr.cfg
[api]
key = your key
secret = your secret
^D

Make the call

$ flickrcall.py flickr.test.echo foo=bar -j -b

that will return the familiar:

{"api_key": {"_content":" your api key"},
 "foo":{"_content":"bar"}
 "nojsoncallback":{"_content":"1"},
 "method":{"_content":"flickr.test.echo"},
 "format":{"_content":"json"},
 "stat":"ok"}

Need to make an authenticated call but don't have a token yet? Pass in -t:

$ flickrcall.py -t -j -b -p write flickr.contacts.getList page=1 per_page=5
auth me:  http://flickr.com/services/auth/...
done [y]:
try auth_token= acquired_auth_token next time for write permissions

Some desktop auth action later you have a token for use in subsequent calls. Of course, you also get the response you were (hopefully) after:

{u'stat': u'ok', u'contacts': {u'perpage': 1000, u'pages': 1, u'contact': [ ....

I often just run ipython -i flickrcall.py ... to get an interactive session, the 'rsp' and 'r' variables contain the response object and the whole response as a string so you can mess with parsing or otherwise interrogate Flickr's response.

Finally, flickrcall can be used as a library

>>> import flickrcall
>>> flickrcall.main("flickr.test.echo", "-n", "foo=bar", bar="foo").read()
'<?xml version="1.0" encoding="utf-8" ?>
<rsp stat="ok">
<api_key> your_api_key </api_key>
<foo>bar</foo>
<bar>foo</bar>
<method>flickr.test.echo</method>
</rsp>'

Here is the full usage:

$ flickrcall.py -h
Usage: flickrcall.py [options] api.method [key=val ... ]

Options:
  -h, --help            show this help message and exit
  -c CONFIG, --config=CONFIG
                        flickr config file
  --no-sign             don't sign request
  -k KEY, --key=KEY     Flickr API key
  -s SECRET, --secret=SECRET
                        Flickr API secret
  -n, --no-secret       don't use the Flickr API secret
  -p PERMS, --permissions=PERMS
                        permissions
  -t, --get-auth-token  get an authentication token first
  -b, --nojsoncallback  nojsoncallback=1
  -j, --json            request response in json
  -x, --xml             request response in xml (rest)
  -D, --debug           print some details about what's going on

A few things that could be added another day:

Optional checkToken
Loading/Saving auth tokens in the config file
Multiple API Key config file support

Flickr.API 0.4.2

2009-08-02T04:03:56Z

I've uploaded a minor update of Flickr.API to pypi. This release addresses the deprecation warning on importing the md5 module under python2.6 by using the hashlib module where available. This is a good opportunity to expand on the pydoc bundled in Flickr.API so here goes.

Making a simple call

flickr.test.echo is a simple place to start and a good way to make sure that everything is set up correctly. It echoes back any parameters you provide.

import Flickr.API
api = Flickr.API.API(key, secret=None)
test_rsp = api.execute_method(method='flickr.test.echo', sign=False)

that returns:

<?xml version="1.0" encoding="utf-8" ?>
<rsp stat="ok">
<api_key> your api key </api_key>
<method>flickr.test.echo</method>
</rsp>

A few things to note here:

the default response format from flickr is xml (called 'rest' on http://www.flickr.com/services/api/response.rest.html)
test_rsp is a file-like addinfourl instance, like that returned by urllib2.urlopen(). That means:
test_rsp has a read() method, great for passing into ElementTree.parse() or json.load()
test_rsp has an HTTP response code (test_rsp.code), which should be checked before you even bother to parse the response contents
Every response from flickr is wrapped in rsp tags. Be sure to check the value of stat after every call

Adding some parameters to request a json response looks like:

test_json_rsp = api.execute_method(
    method='flickr.test.echo',
    args={'format':'json', 'nojsoncallback':1},
    sign=False)

That returns:

{"api_key": {"_content":" your api key"},
 "nojsoncallback":{"_content":"1"},
 "method":{"_content":"flickr.test.echo"},
 "format":{"_content":"json"},
 "stat":"ok"}

There are numerous xml and json handling libraries in python so handling either response type is a matter of preference.

Putting it together

Putting these basic calls together to create a real application using the Flickr API should be pretty straightforward now. Let's say you'd like to write a command-line uploading utility. The Flickr API Upload Documentation specifies that uploading photos requires write permission on a user's account. This means we need to authenticate the user and get our application approved with write access. Desktop Auth lays out the approach (starting with step #3):

Request a frob

api = Flickr.API.API(key, secret)
frob_request = Flickr.API.Request(method='flickr.auth.getFrob')
frob_rsp = api.execute_request(frob_request)
if frob_rsp.code == 200:
    frob_rsp_et = xml.etree.ElementTree.parse(frob_rsp)
    if frob_rsp_et.getroot().get('stat') == 'ok':
        frob = frob_rsp_et.findtext('frob')

This introduces the other way to make an api call with Flickr.API. Here, we create a Flickr.API.Request object as frob_request and pass that into api.execute_request. Note that now we provide the secret instead of None - the secret is required for generating api call signatures. api.execute_method is simply a convenience method that does exactly the same thing. Flickr.API.Request provides more options for configuration as it is a urllib2.Request object. This means you can configure a proxy server, add your own headers, etc. Also, execute_method() takes arguments as a dict (args={}), originally intended to mirror the way Flickr::API works. execute_request uses the more pythonic **kwargs for arguments.

We also see here some of the application-level error handling that needs to be done. Note that execute_request could throw any of the exceptions urllib2.urlopen() does so that should be wrapped in a try/except and handled appropriately.

Create a login link

The get_authurl method makes this step easy. Continuing with the command line uploader, here we display the link and wait until the user confirms that it visited that url. For web authentication, you don't get the frob until after authorization happens so you would just leave out the frob paramater.

auth_url = api.get_authurl('write', frob=frob)
print "auth me:  %s" % (auth_url,)
input = raw_input("done [y]: ")
if input.lower() not in ('', 'y', 'Y'):
    sys.exit()

Convert Frob to Token

token_rsp = api.execute_request(Flickr.API.Request(
    method='flickr.auth.getToken', frob=frob, format='json', nojsoncallback=1)
)

if token_rsp.code == 200:
    token_rsp_json = simplejson.load(token_rsp)
    if token_rsp_json['stat'] == 'ok':
        token = str(token_rsp_json['auth']['token']['_content'])

Here we see how to handle a json response. Now that we have the token, we can make any authenticated calls by passing auth_token=token in the list of arguments.

Make an authenticated call

There are two methods for uploading photos provided in Flickr.API. Like execute_method, execute_upload is a convenience method that calls execute_request with the appropriate Flickr.API.Request object. That looks like:

photo = open('photo.jpg', 'rb')
upload_response = api.execute_upload(
    filename='photo.jpg', 
    args={'auth_token':token, 'title':'test upload', 'photo':photo})

If you need to build your own request object, that could look like:

photo = open('photo.jpg', 'rb')
upload_request = Flickr.API.Request(
    url="http://api.flickr.com/services/upload",
    auth_token=token, title='test upload', photo=photo)
upload_response = api.execute_request(upload_request,
    sign=True, encode=Flickr.API.encode_multipart_formdata)

Either way, the upload response contains the unique photo id assigned by Flickr to that photo. This response is always XML, even if you try to specify the response format. The url to that photo is: http://www.flickr.com/photos/nsid/photoid or just: http://www.flickr.com/photo.gne?id=photoid.

That's it, go create.

libyahoo2 python bindings

2009-04-08T03:37:34Z

I have wanted Python bindings for libyahoo2 for a long time and finally sat down with Ned Batchedler's Pycon '09 slides, A Whirlwind Excursion through Python C Extensions, and wrote some basic ones available at:

http://bitbucket.org/graphaelli/ymsgrclient

Thanks Ned for the great presentation. He's right that it's just an introduction as things like calling python from C couldn't even be covered. I would recommend that aspiring module writers review Ned's presentation, move on to Extending and Embedding the Python Interpreter, and then bookmark the crap out of the Python/C API Reference Manual.

Back to the bindings - a simple bot that will parrot your words back to you looks like this:

import YmsgrClient

class ParrotBot(YmsgrClient.YmsgrClient):
    def got_im(self, who, msg, timestamp):
        self.send_im(who, msg)

b = ParrotBot(username, password)
b.login(YmsgrClient.STATUS_AVAILABLE)

There are a number of rough edges and simply bad ideas in the code but it is sufficient for writing a single account client. We've been using this for our bot at work for a few days without problems [or segfaults]. What we have had is an increase in bot functionality, as the barrier to writing few lines of Python and restarting the bot is much lower than writing a bunch of C and recompiling repeatedly until it works [or doesn't, giving up, and rolling it all back].

Some things I'd like to do immediately:

Convert the data structures and functions lifted directly from the libyahoo2 sample client to Python objects and expose them
Redo the login poll loop or at least allow callbacks to be scheduled somehow
Add Documentation
Throw better exceptions and use the logging module properly

Some things that someone should do eventually:

Add Webcam support and sample usage documentation
Test functionality on Windows

Contributions are welcome. Constructive criticism is greatly appreciated.

Circumventing security policy with gearman

2009-01-12T10:45:44Z

Gearman can be handy for defeating security in situations like the one pictured:

Here, Host A can initiate connections to Host B but Host B is blocked by a firewall/router ACL/etc from initiating communications with Host A.

With gearman, you can call a service on Host A from Host B with a setup like this:

Run a gearmand job server on host B (or any host that A and B can both reach).
Register a gearman worker running on Host A with that job server.
Submit work from a client running on Host B to that job server.

This idea can be taken another step by chaining workers together such that calling abc() as depicted above creates a job that Host C, which can't reach or be reached by Host B at all, can ultimately execute. That kind of setup makes it increasingly difficult to track an individual task's real status but it can be handy in a pinch.

Gearmand Rewrite Released

2009-01-11T01:57:53Z

There is some big news for Gearman fans out today - the first release of the C-rewrite of the gearmand job server has been announced.

Gearman is an excellent framework for farming out tasks to pools of machines, parallelizing tasks, and making calls between programming languages that speak gearman. Like memcached, it's extremely easy to get started using it. Once you do start using it, you'll never understand how you lived without it.

The new release does not provide python bindings but the code in sixapart's svn should be compatible with the new server*. The sample code provided with the both the perl and C versions is pretty trivial as they simply echo or reverse some strings. To pass interesting data between client and worker you'll need to create your own convention. YAML or simply JSON is very handy for this.

Let's take a contrived example. Of course, error handling will be omitted for clarity.

This is a gearman worker that expects a list of urls and will do something to each of them.

""" A Sample Gearman Worker """
import logging

from functools import wraps
import os
import simplejson as json
import urllib

from gearman import GearmanWorker

def job_in(fn):
    """ Decorates worker functions by calling them with a job's arguments """
    @wraps(fn)
    def new(job):
        # do something with the job object
        return fn(job.arg)
    return new

def json_in(fn):
    """ Decorates a function that may be called with a
        JSON-formatted string but expects a python object """
    @wraps(fn)
    def new(arg):
        # convert the args in JSON to a python object
        arg = json.loads(arg)
        return fn(arg)
    return new

@job_in
@json_in
def fetch(urls):
    success = 0
    for url in urls:
        logging.debug("fetching %s" % url)
        # do something with the url
        success += 1

    return json.dumps({'fetched': success})

worker = GearmanWorker(jobservers)
worker.register_function("fetch", fetch)
worker.work()

A client for this worker can look like:

import simplejson as json
from gearman import GearmanClient, Task

urls = ['http://www.flickr.com', 'http://www.yahoo.com']

client = GearmanClient(jobservers)
response = client.do_task(Task("fetch", arg=json.dumps(urls)))

print "%i urls fetched successfully" % json.loads(response)['fetched']

While this example is still quite simple, it does illustrate the idea of the convention necessary for passing real information between clients and workers. The python libraries for gearman also support tasksets which are a set of tasks submitted at once to a job server for parallel execution. This is a simple yet powerful way to speed up work and make the most of hardware investments.

I hope that this rewrite of gearmand renews interest in the python community to enhance the current bindings. I'd be very interested in a gearman protocol implementation for twisted (and might just start working on it soon).

* on Mac OSX 10.4.11 I'm getting a bus error from gearmand after successfully running a job. I haven't tracked it down or submitted a bug yet. update: bug reported via IRC (yes, I'm gilad on freenode) and fixed with a two line patch. A new release should be announced shortly.

Dr. Dog, Gomez, The Black Keys at The Arena, Brisbane

2009-01-07T03:42:20Z

Dr. Dog, Gomez, and The Black Keys played The Arena last night. It was the best rock show I've been to in a long time.

Dr. Dog opened and I'm pretty sure they played Worst Trip, The Ark, The Beach, The Old Days, and a few others. They were easily my favorite new-to-me band of 2008. I first heard them on yrock on xpn and soon after bought their new album, Fate. They rocked.

While I went to the show to see the opening act, the second opener Gomez turned out to be awesome. I'm definitely going to need to check out some more of their work.

The Black Keys headlined and were incredible - those two guys had so much energy. They're touring a bit more in Australia and then heading back to my hometown area. They'll be at the Electric Factory in Philly in Feb - word is they sold out the TLA so quickly the show got moved.

Ganglia for MySQL Metrics

2009-08-07T23:46:46Z

I just made some minor updates to the metric collection module I use for monitoring MySQL with Ganglia. It collects over 100 metrics and includes a basic report like:

This blog is the only thing using this MySQL server and I don't generate much traffic (hi mom!)

It works either as a ganglia 3.1-style gmond python module or as a gmetric script, for use with any decent version of ganglia. These are the relevant files:

the module: mysql.py
sample module configuration file: mysql.pyconf
ganglia frontend report: mysql_query_report.php
dependency module, for parsing InnoDB Status: DBUtil.py

Comments and improvements are welcome.

Packaging Python Modules

2009-08-07T23:48:13Z

Tools like EasyInstall and PIP are excellent for frequently changing environments. I use virtualenv extensively in my development environments where I can easy_install or pip -E install away. These tools, however, are not appropriate for production environments. In production, repeatability and minimal dependency on development tools are essential. When rolling out hundreds of servers, installing MySQL-python bindings with easy_install would require setting up a build environment on each host. The operating system's package manager is sufficient for managing the installation of these modules.

What if you need different versions of the same package on a particular host? Assuming there are two applications running on a host that require different versions of the same package: move one of the applications to a separate host or VM. If one application needs two versions of a particular package: fix the application. In an emergency, running an application out of a temporary virtualenv can do the job at the expense of operational headaches.

I run Redhat based systems so RPMs are the way to go for me. python setup.py bdist_rpm is handy but there are a number of things I don't like about it:

Packages are not python-versioned. pyOpenSSL-0.8-1.i386.rpm does not indicate which python it is built against. Also, what if you want that package for both python2.4 and python2.5?
Similarly, I prefer that python library packages be clearly identified in the list of installed packages. A 'python-' prefix is effective. rrdtool-1.2.27-5.i386.rpm is the application package, python25-rrdtool.i386.rpm are the python2.5 bindings.

Since those things are not easy to override in the stock bdist_rpm, I use a lightly modified distutils/command/bdist_rpm.py arbitrarily named bdist_rpm_ver.py. Building production packages is usually as easy as:

easy_install --build-directory ~/rpm/BUILD/ --editable <pkg>
cd ~/rpm/BUILD/<pkg>
pythonXX setup.py bdist_rpm_ver --fix-python --binary-only

This certainly is not perfect yet - sometimes I have to break down and do a --spec-only and tweak things by hand. Fortunately, this happens once and only takes a few minutes to fix and then I have a stable package ready for deployment. Also, I handle dependencies separately from the rpms currently but they can be added in step 3 with --requires, --conflicts, etc.

I'm interested in hearing more about:

I’ve been meaning to write a post on why I think using system packaging for libraries is counter-productive, but that’ll wait for another time.

Ian Bicking - A Few Corrections To “On Packaging”.

In my experience, system packaging is the way to go for simple, repeatable installations to a single host or 1000s of hosts.

Prelude

2009-08-07T23:47:49Z

Zine 0.1.1 is installed and I'm really going to do this thing [ with the words and the writing ].