Happstack and Streaming: Part 1: Introduction


The question: is it possible to use Happstack to serve streaming data?

Spoiler alert: the answer is “no”. At least, not with the current version of Happstack (0.4.1). However, exploring the reasons why it isn’t possible sheds some light on what changes you’d need to make to Happstack to make it work. This five-part series will explore that topic in some detail, including code that almost-but-not-quite does what we want.

What is streaming, anyway?

For this discussion, I’m talking about generating a series of bytes in real time from a web application server and sending those bytes to a web browser as part of a single HTTP response message.

This is different from simply sending a large file to the browser. In that case, file already exists on the disk, and can simply be read from and sent across the network as quickly as the client can keep up. TCP automatically takes care of all the ugly details of how you do that reliably; from the application server’s perspective, you send the whole thing in one giant chunk and let the network worry about the rest.

Here, however, the bytes we want to send aren’t known ahead of time. The use case I have in mind is a browser-based game where the current game state changes continuously. It’d be nice to have the server send all those updates as part of a single response, with the browser acting on updates as they are received. The alternative is to have the browser repeatedly poll for updates, sending a brand new request each time it wants the latest information. Polling is more complicated to implement because the browser now needs to worry about timing its update requests, instead of just letting the server send them when an update is ready. (Recall that HTTP uses a client/server, request/response model, preventing the server from sending data to the browser without the browser first asking for it.)

The game scenario is what prevents the server from knowing what bytes will be sent to the browser beforehand, since the game state depend on what moves the players make as the game progresses. To stream updates, the server needs to start sending the response before it knows what all the data that will be included in it are.

Thus the question: if we use Happstack to write the web application server implementing the game, is streaming updates over a single response even possible?

Happstack model

The core part of any application written for Happstack is a function that takes the request from the browser and returns a Response object that gets sent back to the browser. (Granted, that’s a very simplified version of what you pass to simpleHTTP, but it’s good enough for our discussion here.) Happstack takes care of all the details of actually sending and receiving data over the network, converting things into Haskell objects, and the like.

In Happstack 0.4.1, there are two kinds of Responses, only the first of which is of concern to us. (The second kind is optimized for sending existing a preexisting file to the browser, which precisely not what we want.) A Response is really just a (lazy) ByteString, along with HTTP metadata like a status code and a set of headers. Happstack provides a lot of tools to help set it all up, but ultimately the application is responsible for providing the ByteString to be sent to the browser. Once we give Happstack that ByteString, it’s out of the application’s hands.

At first glance, this seems to preclude us from returning any kind of real-time stream. After all, how can the application possibly return a value that it can’t compute until some point in the future? That doesn’t just sound difficult; it sounds physically impossible.

Actually, it turns out that’s the easy part, as we’ll see in Part 2.

Comments Off