g.raphaelli's weblog

Entries tagged “python”

Flickr.API 0.4.3

written by g, on Jan 25, 2010 11:26:00 AM.

Aaron kindly informed me of a problem with Flickr.API where calling flickr.urls.lookupUser with the required 'url' parameter returns the the contents of the resource pointed to by that url, instead of the expected API response. The problem is easily demonstrated using flickrcall.py, introduced in this post:
$ flickrcall.py flickr.urls.lookupUser "url=http://www.flickr.com/photos/graph"

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">

<html>
<head>
...
The trouble lies in the Flickr.API.Request init:
def __init__(self, url='http://api.flickr.com/services/rest/', **args):
    urllib2.Request.__init__(self, url=url)
    ...
Clearly I didn't anticipate 'url' as a valid argument - if you pass it in through **args it overwrites the default url parameter. Worse, if you wanted to use a different api endpoint, you couldn't even call this method with the required url parameter as python will complain about multiple values for the same 'url' argument. Here's a simple demonstration of what's happening:
>>> def foo(bar="baz", **kwargs): print bar

>>> foo()
baz

>>> foo("bar")
bar

>>> foo(bar="bar")
bar

>>> foo(**{"bar":"bar"})
bar

>>> foo("bar", **{"bar":"baz"})
TypeError: foo() got multiple values for keyword argument 'bar'

>>> foo(bar="bar", **{"bar":"baz"})
TypeError: foo() got multiple values for keyword argument 'bar'
For 0.4.3, I've simply changed the url keyword parameter to apiurl to make things work as I haven't thought of any backwards compatible way to fix this. Now we get the right response:
$ flickrcall.py flickr.urls.lookupUser "url=http://www.flickr.com/photos/graph"
<?xml version="1.0" encoding="utf-8" ?>
<rsp stat="ok">
<user id="83127823@N00">
	<username>raphco</username>
</user>
</rsp>
I'm hoping that unless you work at Flickr (we're hiring!), you're not likely to change the endpoint url and this change should only fix this particular problem for you.

Streaming Flickr Uploads in Python

written by g, on Oct 7, 2009 5:29:00 PM.

Flickr has supported up to 500 MB uploads since April. Unfortunately, uploading even a 100 MB file (easily less than 30 sec of high def video) with Flickr.API, possibly after reading this or this, uses about about 100 MB of memory, because the entire file is first read() before being sent off to Flickr. top reports this for a 94 MB video upload:
  PID COMMAND     #TH #PRTS #MREGS RPRVT  RSHRD  RSIZE  VSIZE
  627 python        1    20    133   98M   256K   101M   118M 
A quick search of pypi turned up Poster, a library for Streaming HTTP uploads and multipart/form-data encoding. After hooking this up, the same 94 MB video upload is reported as:
  PID COMMAND     #TH #PRTS #MREGS RPRVT  RSHRD  RSIZE  VSIZE
  631 python        1    20    132 4904K   256K  7524K    24M 
Notice the private memory usage drop from 98 MB to under 5 MB. The code to get it done looks like this:
import poster, poster.streaminghttp
poster.streaminghttp.register_openers()

def flop(fn):
    def flopped(a, **kwargs):
        (c, d) = fn(a, **kwargs)
        return (d,c)
    return flopped
flop swaps the (body generator, headers) tuple returned by Poster's multipart_encode into (headers, body) that Flickr.API's execute_request is expecting as seen in the next code block. multipart_encode can receive a custom boundary but we're basically just going to ignore that here. Then, when executing the upload request, this new form encoding can be used like:
photo = open(file, 'rb')
upload_request = Flickr.API.Request(url="http://api.flickr.com/services/upload",
    auth_token=token, photo=photo)
upload_response = api.execute_request(upload_request, sign=True,
    encode=flop(poster.encode.multipart_encode))
Now Poster takes care of transferring the file in 8KB chunks, instead of loading the whole file into memory.

Flickr.API Helper Script

written by g, on Aug 7, 2009 11:23:00 AM.

Here is a [formerly] little script I use to test out Flickr API calls: flickrcall.py. It grew by about 200 lines getting it ready for posting here and it's still not pretty. It does work though, so use it like this:

Create a config file (optional, you can specify the key and secret on the command line)

$ cat > ~/.flickr.cfg
[api]
key = your key
secret = your secret
^D

Make the call

$ flickrcall.py flickr.test.echo foo=bar -j -b

that will return the familiar:

{"api_key": {"_content":" your api key"},
 "foo":{"_content":"bar"}
 "nojsoncallback":{"_content":"1"},
 "method":{"_content":"flickr.test.echo"},
 "format":{"_content":"json"},
 "stat":"ok"}

Need to make an authenticated call but don't have a token yet? Pass in -t:

$ flickrcall.py -t -j -b -p write flickr.contacts.getList page=1 per_page=5
auth me:  http://flickr.com/services/auth/...
done [y]:
try auth_token= acquired_auth_token next time for write permissions

Some desktop auth action later you have a token for use in subsequent calls. Of course, you also get the response you were (hopefully) after:

{u'stat': u'ok', u'contacts': {u'perpage': 1000, u'pages': 1, u'contact': [ ....

I often just run ipython -i flickrcall.py ... to get an interactive session, the 'rsp' and 'r' variables contain the response object and the whole response as a string so you can mess with parsing or otherwise interrogate Flickr's response.

Finally, flickrcall can be used as a library

>>> import flickrcall
>>> flickrcall.main("flickr.test.echo", "-n", "foo=bar", bar="foo").read()
'<?xml version="1.0" encoding="utf-8" ?>
<rsp stat="ok">
<api_key> your_api_key </api_key>
<foo>bar</foo>
<bar>foo</bar>
<method>flickr.test.echo</method>
</rsp>'

Here is the full usage:

$ flickrcall.py -h
Usage: flickrcall.py [options] api.method [key=val ... ]

Options:
  -h, --help            show this help message and exit
  -c CONFIG, --config=CONFIG
                        flickr config file
  --no-sign             don't sign request
  -k KEY, --key=KEY     Flickr API key
  -s SECRET, --secret=SECRET
                        Flickr API secret
  -n, --no-secret       don't use the Flickr API secret
  -p PERMS, --permissions=PERMS
                        permissions
  -t, --get-auth-token  get an authentication token first
  -b, --nojsoncallback  nojsoncallback=1
  -j, --json            request response in json
  -x, --xml             request response in xml (rest)
  -D, --debug           print some details about what's going on

A few things that could be added another day:

  • Optional checkToken
  • Loading/Saving auth tokens in the config file
  • Multiple API Key config file support

Flickr.API 0.4.2

written by g, on Aug 2, 2009 11:40:00 AM.

I've uploaded a minor update of Flickr.API to pypi. This release addresses the deprecation warning on importing the md5 module under python2.6 by using the hashlib module where available. This is a good opportunity to expand on the pydoc bundled in Flickr.API so here goes.

Making a simple call

flickr.test.echo is a simple place to start and a good way to make sure that everything is set up correctly. It echoes back any parameters you provide.

import Flickr.API
api = Flickr.API.API(key, secret=None)
test_rsp = api.execute_method(method='flickr.test.echo', sign=False)

that returns:

<?xml version="1.0" encoding="utf-8" ?>
<rsp stat="ok">
<api_key> your api key </api_key>
<method>flickr.test.echo</method>
</rsp>
A few things to note here:
  • the default response format from flickr is xml (called 'rest' on http://www.flickr.com/services/api/response.rest.html)
  • test_rsp is a file-like addinfourl instance, like that returned by urllib2.urlopen(). That means:
  • test_rsp has a read() method, great for passing into ElementTree.parse() or json.load()
  • test_rsp has an HTTP response code (test_rsp.code), which should be checked before you even bother to parse the response contents
  • Every response from flickr is wrapped in rsp tags. Be sure to check the value of stat after every call

Adding some parameters to request a json response looks like:

test_json_rsp = api.execute_method(
    method='flickr.test.echo',
    args={'format':'json', 'nojsoncallback':1},
    sign=False)

That returns:

{"api_key": {"_content":" your api key"},
 "nojsoncallback":{"_content":"1"},
 "method":{"_content":"flickr.test.echo"},
 "format":{"_content":"json"},
 "stat":"ok"}

There are numerous xml and json handling libraries in python so handling either response type is a matter of preference.

Putting it together

Putting these basic calls together to create a real application using the Flickr API should be pretty straightforward now. Let's say you'd like to write a command-line uploading utility. The Flickr API Upload Documentation specifies that uploading photos requires write permission on a user's account. This means we need to authenticate the user and get our application approved with write access. Desktop Auth lays out the approach (starting with step #3):

Request a frob

api = Flickr.API.API(key, secret)
frob_request = Flickr.API.Request(method='flickr.auth.getFrob')
frob_rsp = api.execute_request(frob_request)
if frob_rsp.code == 200:
    frob_rsp_et = xml.etree.ElementTree.parse(frob_rsp)
    if frob_rsp_et.getroot().get('stat') == 'ok':
        frob = frob_rsp_et.findtext('frob')

This introduces the other way to make an api call with Flickr.API. Here, we create a Flickr.API.Request object as frob_request and pass that into api.execute_request. Note that now we provide the secret instead of None - the secret is required for generating api call signatures. api.execute_method is simply a convenience method that does exactly the same thing. Flickr.API.Request provides more options for configuration as it is a urllib2.Request object. This means you can configure a proxy server, add your own headers, etc. Also, execute_method() takes arguments as a dict (args={}), originally intended to mirror the way Flickr::API works. execute_request uses the more pythonic **kwargs for arguments.

We also see here some of the application-level error handling that needs to be done. Note that execute_request could throw any of the exceptions urllib2.urlopen() does so that should be wrapped in a try/except and handled appropriately.

Create a login link

The get_authurl method makes this step easy. Continuing with the command line uploader, here we display the link and wait until the user confirms that it visited that url. For web authentication, you don't get the frob until after authorization happens so you would just leave out the frob paramater.

auth_url = api.get_authurl('write', frob=frob)
print "auth me:  %s" % (auth_url,)
input = raw_input("done [y]: ")
if input.lower() not in ('', 'y', 'Y'):
    sys.exit()

Convert Frob to Token

token_rsp = api.execute_request(Flickr.API.Request(
    method='flickr.auth.getToken', frob=frob, format='json', nojsoncallback=1)
)

if token_rsp.code == 200:
    token_rsp_json = simplejson.load(token_rsp)
    if token_rsp_json['stat'] == 'ok':
        token = str(token_rsp_json['auth']['token']['_content'])

Here we see how to handle a json response. Now that we have the token, we can make any authenticated calls by passing auth_token=token in the list of arguments.

Make an authenticated call

There are two methods for uploading photos provided in Flickr.API. Like execute_method, execute_upload is a convenience method that calls execute_request with the appropriate Flickr.API.Request object. That looks like:

photo = open('photo.jpg', 'rb')
upload_response = api.execute_upload(
    filename='photo.jpg', 
    args={'auth_token':token, 'title':'test upload', 'photo':photo})

If you need to build your own request object, that could look like:

photo = open('photo.jpg', 'rb')
upload_request = Flickr.API.Request(
    url="http://api.flickr.com/services/upload",
    auth_token=token, title='test upload', photo=photo)
upload_response = api.execute_request(upload_request,
    sign=True, encode=Flickr.API.encode_multipart_formdata)

Either way, the upload response contains the unique photo id assigned by Flickr to that photo. This response is always XML, even if you try to specify the response format. The url to that photo is: http://www.flickr.com/photos/nsid/photoid or just: http://www.flickr.com/photo.gne?id=photoid.

That's it, go create.

libyahoo2 python bindings

written by g, on Apr 3, 2009 9:23:00 AM.

I have wanted Python bindings for libyahoo2 for a long time and finally sat down with Ned Batchedler's Pycon '09 slides, A Whirlwind Excursion through Python C Extensions, and wrote some basic ones available at:

Thanks Ned for the great presentation. He's right that it's just an introduction as things like calling python from C couldn't even be covered. I would recommend that aspiring module writers review Ned's presentation, move on to Extending and Embedding the Python Interpreter, and then bookmark the crap out of the Python/C API Reference Manual.

Back to the bindings - a simple bot that will parrot your words back to you looks like this:

import YmsgrClient

class ParrotBot(YmsgrClient.YmsgrClient):
    def got_im(self, who, msg, timestamp):
        self.send_im(who, msg)

b = ParrotBot(username, password)
b.login(YmsgrClient.STATUS_AVAILABLE)

There are a number of rough edges and simply bad ideas in the code but it is sufficient for writing a single account client. We've been using this for our bot at work for a few days without problems [or segfaults]. What we have had is an increase in bot functionality, as the barrier to writing few lines of Python and restarting the bot is much lower than writing a bunch of C and recompiling repeatedly until it works [or doesn't, giving up, and rolling it all back].

Some things I'd like to do immediately:

  • Convert the data structures and functions lifted directly from the libyahoo2 sample client to Python objects and expose them
  • Redo the login poll loop or at least allow callbacks to be scheduled somehow
  • Add Documentation
  • Throw better exceptions and use the logging module properly
Some things that someone should do eventually:
  • Add Webcam support and sample usage documentation
  • Test functionality on Windows

Contributions are welcome. Constructive criticism is greatly appreciated.