Magic Methods and the 26 Line Json Client

Magic Methods and the 26 Line Json Client

Python has a beautiful and terse syntax. However, the magic of Python is that it gives you the ability to provide your own implementations for the syntax of the language allowing you to do truly magicial things.

I discovered that using magic methods I could implement a reasonable json client in 26 lines of python complete with auth. The vast majority of the programatic web involves making an http request to a descriptive url/endpoint with params to fetch data. So to get your friends from Facebook you would make an http call to 'https://graph.facebook.com/me/friends' that returns json. I wanted to write a client that translated native python into that or any descriptive url:

# Magically call https://graph.facebook.com/me/friends?limit=10
client = JsonClient('https://graph.facebook.com', auth)
friends = client.me.friends(limit=10)
for f in friends:
   print f

Turns out there is a very elegant way in Python to accomplish a basic implementation within 26 lines. You can then make it more robust without much more coding. So in the interest of brevity I will present the basic implementation taking advantage of the 'requests' and 'json' libraries.

Let's talk about syntax vs implementation for a second. Pretend you have a fictitious class called User such that you can do this.

u = User(name='Brad', age=31)
print u.name, u.age

So User is a class that has two attributes name and age, and the instance of the class lets you access them via '.name' and '.age'. Ignore how you might want to implement that class and realize 'u.name' means get me the attribute from u called 'name'. So logically what happens is that Python has to check and see if u has an attribute called 'name' and then return it or error. So imagine a world in which you could intercept that attribute lookup and return whatever you wanted, and that fantastic world is named Python.

When you define a class, you can create a 'magic method' called __getattr__ that has the argument 'name' that lets you intercept the request for the attribute and return whatever you want

class User(object):
    def __getattr__(self, name):
       return 'you reqested %s and got this instead' % name

This is immediatley powerful. The method __getattr__ could make a database call to find the attribute, perform and return a calculation, or in the case of our JsonClient it will just record what was requested and return itself instead (more on this later).

So Python lets you intercept the getting of an attribute. But then what is the difference between asking for the attribute or calling a function? Syntactically it's just the presence of parenthesis with optional parameters, but logically it's exactly the same except there could be arguments. Python does not dissapoint and also lets you intercept a function call with another magic method named __call__ that may have arguments.

class User(object):
    def __call__(self, **kwargs):
       return 'you made a function call with %s' % kwargs

So let's look as what we want to happen again:

client.me.friends.pizzas.cheese(limit=10)

Should request the url https://graph.facebook.com/me/friends/pizzas/cheese?limit=10

So if we break it down syntactically it's instance 'client', get atttribute 'me', get attribute 'friends', get atribute 'pizzas', get attribute 'cheese', function with keyword arg limit=10. So the magical JsonClient class winds up looking like this:

import requests
import json


class JsonClient(object):
    def __init__(self, url, auth=None):
        self.api_url = url
        self.auth = auth
        self.url = self.api_url

    def __getattr__(self, name):
        self.url +='/%s' % name
        return self

    def __call__(self, **kwargs):
        print 'invoking %s with %s' % (self.url, kwargs)
        http_params = {'params': kwargs}

        if self.auth:
            http_params['auth'] = self.auth

        r = requests.get(self.url, **http_params)
        self.url = self.api_url  # reset url
        r.raise_for_status()

        return json.loads(r.content)

Our class JsonClient has a contructor that takes the api_url like 'https://graph.facebook.com' or 'https://api.twitter.com/1.1' for the endpoint, and the requests auth object (because nearly everything requires auth) and binds them to the instance 'self' in __init__. The class then implements __getattr__ to intercept the syntax to get an attribute but instead of doing a lookup, it steals the name and concatinates it to self.url with the '/' and returns itself with 'self'. So multiple __getattr__ invocations becomes a recursive call that appends to the api_url.

When you're at the end of all the attribute gets, you then intercept the function call to finally execute the http request against the url. The url was built via __getattr__ recusion, and the function parameters can then be used as the http request parameters. The code in __call__ then makes the request with r = requests.get(self.url, **http_params), resets the url, asks the response to raise an exception if the request was bad with r.raise_for_status(), otherwise parse the json string and return the data.

This really embodies the power of Python. So much time and effort was put into making a simple and elegant language, but the power in Python is that the language lets you reuse it's own syntactic beauty by giving you the power to define its meaning with language features like magic methods.

Abuse this power and you'll write code no one will ever understand but master this power and you'll write dynamic code that truly seems to accomplish magical things.

Brad WillardComment