Fast XML parsing with Ruby

One of the first things I needed to do while building the GeeQE iPhone application was process the CC data dump from Stack Overflow. The dump contains XML files representing tables from Stack Overflow with the largest file being posts.xml weighing in at 1.2G as of September. I decided it would be pretty easy to use Ruby to parse the XML and load the data into MySQL so I went about finding the right parser for the job.

If you haven't processed large amounts of XML before one thing to realize is that you don't want to use a DOM parser because it is going to load the entire XML structure into memory. What you want is a SAX parser that can work on the XML stream as it comes in. With this in mind I started looking around and quickly found an older benchmark post that gave me an educated guess that the LibXML library was going to be the fastest parser for Ruby. After figuring out how to use it I decided to also give a couple other libraries a shot to see how they stacked up, the other two I looked at were REXML and Nokogiri.

Continue reading

RFID Reader USB Prototyping Kit

I recently won a programming contest that netted me a gift card for ThinkGeek and not knowing what else to do I strolled the site looking for something interesting to use the gift card on. Eventually I ran into the RFID Experimentation Kit they have and decided that was what I needed. I have been wanting to play around with RFID for a while and this kit turned out to be pretty nice for tinkering.

Continue reading

iPhone Windowed HTTP Live Streaming Server

After some more work I have enhanced the HTTP segmenter and uploading script from my iPhone streaming using AWS S3 and Cloudfront post. I added a number of features and tried to pull together some of the ideas from the comments. I'll go over some of the features here and there is a full list of configuration options on the HTTP Live Video Streaming server project page and the source is available at the github repo.

So the major changes I have added are:

  • Yaml based configuration file. See the project page for a complete list of options.
  • Ability to transfer segments via copy, ftp, scp and s3.
  • Added the ability to do variable bitrate streams.
  • Added re-streaming support.
  • Added logging to a file and better debug output.

The variable bitrate streams where done by using pipes. I haven't done a large amount of testing but it seems to work fairly well. I am able to stream a live HD video source into 3 different bitrates on a fairly old PC. Here are a couple clips I created to show the progressive enhancement in action, you probably want to switch to the HD version of the video and watch it full screen to get the bet view:


The configuration file will allow for any number of encoding options or transfer options and they can be put together in a number of different ways. Here are a couple examples of both, see the example configuration files for more.

An encoder example:

ep_128k:
  ffmpeg_command: "ffmpeg -er 4 -y -i %s -f mpegts -acodec libmp3lame -ar 48000 -ab 64k -s 320x240 -vcodec libx264 -b 128k -flags +loop -cmp +chroma -partitions +parti4x4+partp8x8+partb8x8 -subq 5 -trellis 1 -refs 1 -coder 0 -me_range 16 -keyint_min 25 -sc_threshold 40 -i_qfactor 0.71 -bt 128k -maxrate 128k -bufsize 128k -rc_eq 'blurCplx^(1-qComp)' -qcomp 0.6 -qmin 10 -qmax 51 -qdiff 4 -level 30 -aspect 320:240 -g 30 -async 2 - | %s %s %s %s %s"
  bandwidth: 128000

Transfer configuration example:

ftp_dev:
  transfer_type: 'ftp'
  remote_host: '192.168.1.1'
  user_name: 'user'
  password: 'pass'
  directory: 'html/streamingvideo'

As a final note on changes, you are no longer able to use the segmenter without the script now really. If you want to do that you should use the original version of the segmenter source.

Please note that there is still some work to be done on the script to be complete. If I have time my next enhancement will be to add encryption and I will probably try to test builds on other distributions (maybe attempt to create segmenter binaries).

iPhone Windowed HTTP Live Streaming Using Amazon S3 and Cloudfront Proof of Concept

This post should be seen as a proof of concept. I'm working on creating a more concise and easier to use package of everything covered here but I felt like getting the knowledge out sooner rather than later would be of help to people looking for a way to do this. If you are interested keep an eye on the HTTP live video stream segementer and distributor project page as well as the github git repository.

After my post on using FFMpeg and an open source segmenter to create videos for the iPhone that conform to the HTTP live streaming protocol I decided to see if I could get the same segmenter to work on a live stream. As it turns out it didn't take much modification to work.

If you are looking for something you can buy out of the box it appears that Akamai is doing iPhone video streaming now. I believe that the following solution using Amazon S3 and Cloudfront is probably as good as what Akamai can offer but it may be a better choice if you don't want to have to maintain the configuration.

I put together a quick diagram of the process of transferring the video stream from source to final destination that will hopefully help people understand the full picture before jumping into the details:

HTTP Live Streaming Diagram

Continue reading

Streaming Video Between QuickTime Broadcaster and VLC

In my attempt to stream live video to my iPhone I ran into an issue with the USB QuickCam I have. Instead of fighting the problem I decided to turn to the iSight camera on a macbook but I quickly found that there isn't a great way to get the video stream to another machine. The best way that I have found to stream from the iSight to a second machine is to use a combination of QuickTime Broadcaster and VLC. In this post I detail how I got it set up and working.

Continue reading

iPhone HTTP Streaming with FFMpeg and an Open Source Segmenter

With the release of the iPhone OS 3 update came the ability to do live streaming. There are a few types of streaming and each requires a certain encoding and segmentation. I've put together a cheat sheet on how I went about building a static stream using FFMpeg and an example segmenter that someone has posted. I'm not covering windowed streams in this post but if you are thinking about implementing a windowed stream the following will help you make a step in that direction.

Before getting started it is best to read over the Apple documentation on HTTP live streaming. Start out with the iPhone streaming media overview. This document covers the basics of how the streaming works and has some nice diagrams.

If you want even more information after reading the overview you can take a look at the HTTP Live streaming draft proposal that was submitted to the IETF by Apple. It covers the streaming protocol in complete detail and has examples of the stream file format for reference.

Once you are ready to start grab a decent quality video clip to use. If you don't have one handy I found a nice list of downloadable HD clips in various formats for testing.

Continue reading

Upgrading to Fedora 11 from Fedora 10

It is time again to upgrade if you are using Fedora. Fedora 11 was released a few days ago and contains some nice enhancements.

For people who can a complete re-install is probably best. One reason for that is the inclusion of ext4 in Fedora11. You won't get the benefit of ext4 unless you do a fresh install or upgrade from ext3 to ext4. If you read the upgrade guide that Fedora produces it recommends not doing an upgrade.

Continue reading

10 Tips for BlackBerry Development

Over time I've gathered up some nice tips I think would help get someone started developing applications for the BlackBerry so I decided to pull some of the more interesting ones together into the following list.

  1. The Eclipse plugin makes life much easier so use it. If you are used to Intellij IDEA then try out the Intellij IDEA keymap for Eclipse
  2. Some information about signing keys:
    • You don't have to get signing keys to make a deployable application
    • They are inexpensive
    • If you do get the signing keys you will have access to libraries that you otherwise can't use
    • You will need the JDE installed even if you have the Eclipse plugin installed as explained here
  3. When sending apps to the background:
  4. Install the MDS emulator. You need the MDS emulator if you plan to do anything that requires a network connection.
  5. If you need SOAP support use kSOAP. There is a good DDJ article with examples on how to use kSOAP in a mobile setting.
  6. Use Display.getWidth() to make your custom items look correct on any device. Keep the following in mind:
    • Make any static graphics you use that need to span an entire screen as large as the largest device size
    • I have created a list of BlackBerry device screen sizes
  7. The BlackBerry has an embeddable browser. There is a good example post on how to use it.
  8. Spend time to make a splash screen. The knowledge base has a good article on making a splash screen.
  9. Make good looking icons and make sure to also define the rollover icon.
  10. If you need charts in your application take advantage of the device always being connected to the internet and give Google Charts a try.

How to create iPhone wireframes with Inkscape

While developing ideas for iPhone applications I've played around with just using Interface Builder to stub things out. This works reasonably well but I know how to use Interface Builder so that makes a difference. If you are designing an application and want to stick with graphical tools only you are in luck because Yahoo has produced a set of wireframing SVG stencils for the iPhone that are very nice.

Continue reading

BlackBerry screen and icon sizes

For some reason the old link I had that contained display dimensions for BlackBerry devices is resulting in a 404 now. When you are doing BlackBerry development it is nice to be able to go to one page that just shows you the screen sizes instead of having to look at each device page so I don't know why they removed this page. After looking around a little I couldn't find a consolidated list that has just screen sizes so I figured I would put one together that replicates the old link:

BlackBerry device model Display screen size Icon image size
BlackBerry Pearl 8100 240 x 260 pixels 60 x 55 pixels Icon theme
48 x 36 pixels Zen theme
BlackBerry 7100 Series 240 x 260 pixels 60 x 55 pixels Icon theme
48 x 36 pixels Zen theme
BlackBerry Pearl 8220 240 x 320 pixels 46 x 46 pixels
BlackBerry Curve 8300 320 x 240 pixels 53 x 48 pixels Icon theme
48 x 36 pixels Zen theme
BlackBerry Curve 8900 480 x 360 pixels Unknown
BlackBerry 8800 Series 320 x 240 pixels 53 x 48 pixels Icon theme
48 x 36 pixels Zen theme
BlackBerry 8700 Series 320 x 240 pixels 53 x 48 pixels Icon theme
48 x 36 pixels Zen theme
BlackBerry Bold 9000 480 x 320 pixels 80 x 80 pixels
BlackBerry Storm 9500 360 x 480 pixels 76 x 76 pixels Home screen
24 x 24 pixels Shortcut bar

This data comes from the BlackBerry devices page.