Accessing the Twitter Streaming API from Adobe Flex/AIR

Over the weekend I discovered an interesting new hobby, so being the geek that I am I wrote a mobile application to support it. I found that I really like watching the pictures people post on Twitter in real-time. This was based around hurricane #irene so seeing the incoming photos was awesome, although I suppose it could also be applied to even more interesting endeavors like #friskyfriday (NSFW) ;) In the application I wrote I wanted to connect to the Twitter live stream instead of using search. Although it doesn't send you the full firehose of tweets, I wanted everything in real-time for geek-ability.

I won't post all the gory details of getting this application up and running, but there were two things that anyone developing for this API will need to do, and here they are:

1. Connect to the Streaming API

I had tried to do this in the past and had much more luck using curl or Java to coax the tweets out of the Streaming API, but this time I wanted to do it all in ActionScript as I didn't want to keep track of anything in a database. I wanted real-time streaming of data as it came in, and nothing historical. To do this I used the URLStream class. URLStream opens a HTTP request and then downloads the received data as it comes. URLStream is especially suited to this type of request as the download does not have to finish before you can access the downloaded bytes. This means two things: 1) You can read the bytes as they come and maintain the connection to keep receiving bytes, and 2) you must be super careful, as the Streaming API sends chunks of data. This means that the last chunk you received from Twitter might not be a complete object. You can imagine what this does to the parser (more on this later).

The Streaming API current supports both basic and OAuth authentication. Twitter says that basic authentication is deprecated, so use at your own risk. This article will use basic auth as OAuth and Twitter has been done to death on other blogs.

While setting up the stream controller, we will create a property that will hold our URLStream. This must exist in the scope of the controller as all of the events will interact with this stream as the URLRequest is processed.

private var stream:URLStream;

After instantiating the class we need to set up our URLStream. This is accomplished by defining a new request, setting the URI we want to hit, setting the basic auth parameters and finally adding event handlers to react to the data streaming in or any errors that might occur.

public function startStream():void {
  stream = new URLStream();
  var request:URLRequest;
      
  request = new URLRequest("http://stream.twitter.com/1/statuses/filter.json?track=&include_entities=1");
  
  // set up basic auth
  var encoder:Base64Encoder = new Base64Encoder();
  encoder.encode(appController.username + ":" + appController.password);
  var credentials:String = encoder.toString();
  var authHeader:URLRequestHeader = new URLRequestHeader("Authorization","Basic " + credentials);
  //add the header to request
  request.requestHeaders.push(authHeader);
  request.method = URLRequestMethod.POST;
  
  // in a real app I would also recommend handlers for SecurityErrorEvent.SECURITY_ERROR, IOErrorEvent.IO_ERROR at a minimum
  stream.addEventListener(ProgressEvent.PROGRESS, progressHandler);
  
  try {
    stream.load(request);
  } catch (error:Error) {
    trace("Bad URL.");
  }
}

A few notes about the code above... First, there is a track parameter being passed but it is currently empty. This allows you to filter your request. For my example I let the user enter this in on a prior view and passed it in via an application controller. Second, we are only listening for a single event, ProgressEvent.PROGRESS. In the real application I suggest you listen for at least security errors and input errors. You can also listen for HTTP status codes. This is important and Twitter would like for you to watch those codes and throttle your requests accordingly. Please make sure to do this or they will terminate your stream. Lastly, please note that we are requesting the response to be JSON. The streaming API only allows you to request JSON. Next, we need to define our handler. The one I am concerned with for this tutorial is the progress handler (it handles the ProgressEvent.PROGRESS event). When a progress event is received you can check the Stream object to see if enough bytes have been loaded to process the stream's data. In this application I didn't care how much data had been sent as I was more concerned with if the data available had any complete tweets (JSON objects) in it. To do this I fire off an event to another controller which processes the data.

private function progressHandler(event:ProgressEvent):void {
  // read the bytes
  var x:ByteArray = new ByteArray();
  stream.readBytes(x, 0, stream.bytesAvailable);
  
  // create an event to create the picTweets from the JSON
  var te:TweetEvent = new TweetEvent( TweetEvent.NEW_TWEET_RECEIVED );
  te.json = x.toString();
  // I'm using Swiz to dispatch my events - perhaps you should be also ;)
  dispatcher.dispatchEvent(te);
  
  trace("Progress Event Handled.");
}

We read the bytes from the stream as a ByteArray. The ByteArray data is then sent across to the other controller as a string using a custom event. The custom event looks like this:

package events
{
  import flash.events.Event;
  
  public class TweetEvent extends Event
  {
    public static const NEW_TWEET_RECEIVED:String = "events.TweetEvent.NEW_TWEET_RECEIVED";
    
    public var json:String;
    
    public function TweetEvent(type:String, bubbles:Boolean=true, cancelable:Boolean=false)
    {
      super(type, bubbles, cancelable);
    }
  }
}

2. Parse the resulting data

The handler for this event takes the JSON response and adds it to any other text left over in the buffer from prior requests. We then need to parse the JSON text to look for complete objects to deserialize. Twitter made this easy. They send along a newline character at the end of each JSON object. Simply split() the JSON text on newline and you'll have an array of objects. By trying to deserialize any of these objects using the JSON class (available from the as3corelib package) we can create tweet objects. Any text that can't be deserialized will be put back into the String variable that is holding the JSON data to be parsed with the next data stream response.

public function handleNewTweet(json:String):void {
  // add the new json to the text storage - this is a private var for this controller
  jsonText += json;
  
  // now parse through the text for newline chars, which denote the end of a JSON object
  var aJSON:Array = jsonText.split(/\n/);
  
  
  if ( aJSON.length ) {
    // clear the jsonText for appending any partial objects
    jsonText = '';
    
    for ( var i:String in aJSON ) {
      // try to deserialize, if we error, we dont have a complete object in that array, add it back to the queue (it will always be the last item in the array that fails, if any)
      // deserialize the JSON
      if ( aJSON[i].length ) {
        try {
          var tweet:Object = JSON.decode(aJSON[i]);
          
          var j:String;
          
          // HERE IS WHERE YOU CAN ACTUALLY PROCESS THE TWEET
          
        } catch ( e:Error ) {
          // here is where we add the partial object back into the jsonText variable - will be appended to the next response
          if ( aJSON[i].length && aJSON[i].indexOf('\r') != 0 ) { jsonText += aJSON[i]; }
        }
      }
    }
  }
}

For my purposes I just parsed the entities for media and displayed it within my application. You have the full status object to play with for your application. Have fun, and if you use this drop me a comment so I can check it out!