Thursday, February 26, 2009

7 Questions for Jeff Lindsay on Web Hooks

Web Hooks is an emerging concept in the web services community whereby developers register callback scripts to events on remote applications for asynchronous execution.   Jeff Lindsay, since coining the moniker, serves as the driving force behind helping people understand how to best use it.  His slide presentation on the "Programmable World of Tomorrow" has motivated scores of people to incorporate the technology into their own projects.  

I asked Jeff about his thoughts on the emergence of a platform that's not really new at all.

1. You've certainly been asked this a thousand times and will be asked to do it a million more, but explain the concept behind web hooks and its implementation in today's web services.
Web hooks are user-defined callbacks over HTTP. They're intended to, in a sense, "jailbreak" our web applications to become more extensible, customizable, and ultimately more useful. I think of web hooks as the STDOUT of web applications, where the STDIN are web APIs. Together they create a simple input/output model for the web, similar to what was needed for Unix command pipelining. Pipes allowed users to compose and orchestrate new functionality by chaining together commands. Web hooks will achieve something similar by bringing a new level of event-based programmability to the web.

Like web APIs, vendors have to opt-in on providing this kind of infrastructure. Fortunately, web hooks are not hard to implement at the core. They're just making inline HTTP requests on the backend at significant events to a URL specified by the user. In most programming environments, this can be implemented with a single line of code, without needing a special library or external dependency. There are obvious scalability issues with making a blocking request to the greater web, so most implementations will be more complex than that. However, conceptually, they really are that simple.

2. Since callbacks aren't an original technology concept, as was the case for AJAX, how does the community build momentum for its use, adoption and evolution so it can achieve critical mass, as was the case for AJAX?
I think AJAX achieved critical mass from two things: having a very well-known and obvious example use-case (Gmail and Google Maps), and having a name to reference the design pattern. Those two things are all you need to seed a new technology like that. From there, word spreads, more examples show up, people talk about it online and at conferences, and libraries pop up to make it even easier to accomplish. The adoption of AJAX happened very quickly.

The AJAX story provides a nice template for web hook adoption, but unfortunately, AJAX is a much more user-facing technology. You can see it very clearly when it's used. Web hooks are not only less obvious, but they conceptually require you to step back and think of how they affect the web as an ecosystem, not just a particular website you love to use. 

Perhaps another technology to compare the adoption of web hooks to is REST. You've always been able to do REST APIs, just like you've always been able to do web hooks or HTTP callbacks. Whereas AJAX required functionality to finally be exposed in all the popular browsers. Unlike AJAX, after REST was coined and explained, it a while for it to become the dominant way to implement web APIs. This might have been because it was such a developer-facing technology. 

However, both REST and AJAX grew from having a name to talk about them in discourse, and examples to inspire and instruct others how they can be used. Web hooks are a name, and over the past three years since they got a name, a number of examples have shown up. I think the best way to build momentum is to keep talking about them and thinking about the issues and implications, but more importantly, building out the infrastructure by actually implementing them.

3. What are some misconceptions about the process of registering callbacks?  Have there been inappropriate applications of the concept?
Some of the original cases of web hooks were framed as notification mechanisms. While notification is a major use-case of web hooks and describes how they were being used in those cases, it doesn't describe why they were being used. The PayPal Instant Payment Notification mechanism, maybe one of the oldest instances of a web hook, was described as a notification, but the purpose was really about integration. If all you wanted was notification, you could get an email to tell you a payment was made. But IPN was useful for integrating the rest of your software system with PayPal's payment processing.

I don't think there have been inappropriate applications, although one requirement people sometimes forget is that web hooks are user-defined. This can be a hard distinction to make, but is why I don't consider linkbacks to really be web hooks. And again, a lot of people are thinking of web hooks for just notifications and push, which I don't think is entirely counter productive, but it falls short of the bigger picture. 

4. XMPP is the darling of the media right now, gaining lots of traction within the development community and a decent amount of attention from the mainstream press.  Web hooks has a slight advantage in being a clever off-the-cuff branding idea not being bound to a formal spec.  How can we ensure development with web hooks isn't swept under the rug by people's fascination with following the Pied Piper of Hamelin, as it were, and lose focus on using the right tool for the right job?
Definitely XMPP is a strong contender for real-time message passing. It was designed for that. It only just so happens that web hooks can be used as an alternative to XMPP for server-to-server communication. XMPP comes with a lot more out of the box that makes it better suited for many cases, and web hooks requires more work to compete with some of those features. To me, though, it's a little bit like comparing apples to oranges.

I used to compare web hooks to XMPP to bootstrap the existing conceptual model there. I also used to describe web hooks as push over HTTP. I've since realized that could easily pigeonhole web hooks. Now I try and emphasize the functional extensibility it provides more than the push aspect. After all, that's why I fell in love with web hooks in the first place. Can you build a plugin framework in your web app with XMPP? 

I think the strength behind web hooks, at least in relation to the grander vision, is that it's not only simpler, but you're already using its protocol. To play the role of STDOUT for the web, it would be silly for every web application to have an XMPP stack alongside their HTTP stack. If we're going to have a common "output infrastructure" on the web, web hooks make too much sense.

That said, I think XMPP and web hooks will coexist and even work together in the future of the programmable web. 

5. The evolution of distributed computing has gone from complex binary remoting to crude screen scraping, to structured web APIs, to object-oriented JavaScript and cross-domain access, to now extensible triggered execution.  How survivable is this iteration in The Programmable Web? 

That's an interesting question, although it seems all those forms of "distributed computing" are still around and used in some way today. The common property of all the previous technologies, including the many not mentioned, seem to be that they're request-oriented. While web hooks are based on making a request, they're not invoked by making a request, but by a relevant event. In this way, they bring event-driven programming to distributed computing.

Off the top of my head, I can't think of existing technology at the application level for event-driven distributed programming. At least in the context of the web, I see web hooks being the foundation for more specific standards and technologies that promote this paradigm. When technology is a platform like that, like the web itself, it can sometimes be more survivable than people want! For example, I'm still surprised email hasn't been replaced with a better system. :)

6. There are certainly going to be cases where web programmers are going to want to tap existing apps that don't have publish formal APIs.  Without reverting to the hacky days of using mammoth regular expressions or feigning server-side event frameworks, what ecosystem exists in these cases?
Honestly, I don't see anything wrong with hacky solutions sometimes. Either they lead to a more formal solution (like screen scraping to web APIs), or they remain the best solution for a particular problem (like CGI handling the proper use of POST instead of the web server itself). 

Sometimes you can package up hacky solutions as a nice library (the way AJAX is used now), or even as a service. For example, although screen scraping is outdated, not everybody has a web API, making it the only viable solution in some cases. Dapper's Dapp Factory lets you easily create a scrape-based feed of any site, automating the need for tedious regular expressions. 

These kinds of transformer services seem like a great thing to have in our programmable web ecosystem. I can only imagine that web hooks will encourage more of them, allowing web programmers to easily interact with other systems, like even XMPP for example. Gnip lets you consume XMPP data streams using web hook callbacks. I'd just as much like to see a web service that gives you an HTTP endpoint for posting into XMPP. 

7. You've become the de facto champion of this movement.  What are some of the barriers - technological, political, social, etc. - you can see going forward with this means of data access?
It seemed like the biggest hurdle originally was getting people to wrap their heads around this idea. I would always talk about it in the abstract and go on about all the implications of what was essentially one line of code. I think there are enough fairly well-known examples now that it's easier for people to join the party. Even then, the general perception of what's possible is going to be limited by the examples. 

Like AJAX, you can't just build a popular example of AJAX without it being a useful tool itself. I can build all the web hook prototypes I want, but it's not until the Googles and Facebooks implement them in a useful way that people will really see the value. Until then, we get incremental boosts by the smaller companies like Gnip, GitHub, and others. I've started working or talking with these guys to get them involved in a collective conversation around web hooks, so we can work out issues standing in front of adoption. 

The issues people come up with are usually security and scalability related. As it turns out, some of these issues have been solved by these guys already doing it. So I'm trying to get more of them to share best practices and publicize their use of web hooks. This way people can start seeing the different ways they can be used. For example, the Facebook Platform, although pretty complicated and full of their own technology, is still at the core based on web hooks. They call out to a user-defined external web application and integrate that with their application. That's quite a radically different use of web hooks compared to the way people think of them in relation to XMPP. 

Moving forward, I think we're going to see more libraries and tools that have solutions to scalability and security built-in. I've started one project called Hookah that I'm hoping to get released soon. It provides basic callback request queuing and management of callback URLs so you really can implement web hooks with a single line of code for each event. We're also starting to see similar helper libraries for frameworks like Django and Rails. 

Eventually we'll be seeing specs for doing specific things on top of web hooks. One of the first things on my list of standards to look into is the way in which you register and manage callbacks in a programmatic way. Many web hook providers use a web interface to manage your callback URLs. We'll see some neat things happen when you can manage them via APIs so that tools can set callbacks with services on your behalf. 

Anyway, one of the reasons I'm so attached to the idea of web hooks is that I see a lot of long-term potential. Especially when you integrate them into other visions of the future, like the Web of Things. When you combine the Programmable Web with the Web of Things, you get a world of Programmable Things. 

That's where I'd like to see this end up.

Thanks, Jeff!  :-)


Post a Comment

Links to this post:

Create a Link

<< Home

This page is powered by Blogger. Isn't yours?

Subscribe to Posts [Atom]