IONCANNON » iphone

iPad Streaming Video and More

carson — Tue, 06 Apr 2010 11:01:33 +0000

I've updated the configuration examples in the open source segmenter project to reflect Apple's recommended stream bitrates for iPad video streaming, added a few fixes and a few new features. If you are interested in streaming video on the iPad, iPhone or iPod Touch and haven't done so yet you it may help to start with my post on windowed streaming on for the iPhone, then read about iPhone HTTP streaming with FFMpeg and the open source segmenter and finally check out the iPad, iPhone, and iPod Touch live video streaming project page.

Here is a demo of the iPad streaming video created with the segmenter (I tried to show the progressive upgrade happening but it happens very quickly since the iPad is on WIFI, I also show that you can scrub without any issues and if you look in the background you can see the server log displaying entries as the segments are downloaded):

If you want to view the demo yourself I've created a demo for the iPad, iPhone, and iPod Touch. Note that those are two links, one for the iPad version and one for the iPhone/iPod Touch version. The main difference is the size of the video.

I used the open source video Big Buck Bunny (the 1920×1080 ogg version) for the above demos.

If you are interested in more details on what changed read on or skip to the bottom if you want to see what I'll be working towards in future versions of the segmenter.

Over the past few weeks I learned that it is important that each segment starts with an IDR frame. To accomplish this I thought setting the gop so that the segment time % (gop size / frame rate) = 0 would work but I haven't completely convinced myself of this yet. An example of would be a segment size of 10 seconds % (300 gop size / 30 frame rate) = 0. This should insure that each segment starts with the correct i-frame and while I have looked over the resulting segments with a hex editor and believe it works I still get errors when using certain tools on the individual segments that makes me think it isn't working like I think it should. Either way I have included updates to the gop size in the example configuration files. The gop size is controlled by the -g option for FFMpeg and in the examples I have set it so that for a 10 second segment with the given frame rate the gop size makes sure each segment starts correctly. If you want to know more you can dig into the resulting segments and use this forum post on how to extract an i-frame, this list of mpeg headers to verify that each has the correct i-frame at the start.

The Apple streaming tech notes where informative in a few other ways as well. The tech notes contain the supported h264 profiles for each device. For the iPhone/iPod Touch baseline level 3.0 is supported while the iPad supports baseline level 3.1. You can find out more about h264 encoding levels with FFMpeg and also review FFMpeg x264 encoding guide, FFMpeg option mappings and information on encoding with the x264 codec for more information.

Another bit of information disclosed in the Apple tech notes is the existence of a media stream validation tool. This tool can be downloaded from the Apple iPhone developer site and has to be run on OS X. Apple recommends that you use this tool to validate any streams that you create.

One other note that I have recently ran across is that Apple seems to be rejecting native apps that have video streams without a fall back audio stream. The correct way of generating the fallback 64k audio only stream is something that is lacking in the current version of the segmenter but I hope to fix that soon. It won't matter for those using the HTML5 video tag however.

If you are using HTML5 video tags it is important not to use my older posts as a guide for the iPad since I use a now outdated example. Those examples still work on the iPhone but they do not work on the iPad. Here is an updated example that will work on both the iPhone and iPad:


  
    Video Test

Finally I am currently using the latest version of FFMpeg to do tests and it seems to have become more reliable for me. The current version in git seems to be pretty good at producing usable videos from a variety of source videos.

The following is a quick summary of the recent changes I have made:

Applied an audio patch for the segmenter from Scott Kidder. This patch skips any video processing if there is no video stream.
Modified sample configs with newer FFMpeg string, gop sizes, frame rates and other parameters to match the Apple recomended values for iPhone/iPod Touch with aspect ratios of 4:3 and 6:9 on cell or wifi and iPad with aspect ratios of 4:3 and 6:9 on cell or wifi
Added configuration sanity checks
Made gems not required if not using those features that need them
Added fix for deprecation warning from newest version of libavformat
Made termination work better when killed with SIGINT/ctrl-c
Fixed some minor issues with index file format

The following are enhancements I'm planning on making:

I plan on creating a howto on getting everything set up on EC2.
Dig more into verification that each segment starts with the correct I-frame.
I'm going to work on refactoring everything so that instead of creating files and then transferring them it streams them all. This makes more sense with doing encryption as just another part of the pipeline.
Add ability to do encryption. I have a proof of concept working but I decided I need to get everything set up in one continuous stream before integrating encryption into the mix.
I had thoughtabout using the ruby daemon utils to wrap the segmenter but I decided it probably isn't necessary at this point.

iPhone Windowed HTTP Live Streaming Server

carson — Mon, 27 Jul 2009 11:22:54 +0000

After some more work I have enhanced the HTTP segmenter and uploading script from my iPhone streaming using AWS S3 and Cloudfront post. I added a number of features and tried to pull together some of the ideas from the comments. I'll go over some of the features here and there is a full list of configuration options on the HTTP Live Video Streaming server project page and the source is available at the github repo.

So the major changes I have added are:

Yaml based configuration file. See the project page for a complete list of options.
Ability to transfer segments via copy, ftp, scp and s3.
Added the ability to do variable bitrate streams.
Added re-streaming support.
Added logging to a file and better debug output.

The variable bitrate streams where done by using pipes. I haven't done a large amount of testing but it seems to work fairly well. I am able to stream a live HD video source into 3 different bitrates on a fairly old PC. Here are a couple clips I created to show the progressive enhancement in action, you probably want to switch to the HD version of the video and watch it full screen to get the bet view:

The configuration file will allow for any number of encoding options or transfer options and they can be put together in a number of different ways. Here are a couple examples of both, see the example configuration files for more.

An encoder example:

ep_128k:

  ffmpeg_command: "ffmpeg -er 4 -y -i %s -f mpegts -acodec libmp3lame -ar 48000 -ab 64k -s 320×240 -vcodec libx264 -b 128k -flags +loop -cmp +chroma -partitions +parti4x4+partp8x8+partb8x8 -subq 5 -trellis 1 -refs 1 -coder 0 -me_range 16 -keyint_min 25 -sc_threshold 40 -i_qfactor 0.71 -bt 128k -maxrate 128k -bufsize 128k -rc_eq 'blurCplx^(1-qComp)' -qcomp 0.6 -qmin 10 -qmax 51 -qdiff 4 -level 30 -aspect 320:240 -g 30 -async 2 – | %s %s %s %s %s"

  bandwidth: 128000

Transfer configuration example:

ftp_dev:

  transfer_type: 'ftp'

  remote_host: '192.168.1.1'

  user_name: 'user'

  password: 'pass'

  directory: 'html/streamingvideo'

As a final note on changes, you are no longer able to use the segmenter without the script now really. If you want to do that you should use the original version of the segmenter source.

Please note that there is still some work to be done on the script to be complete. If I have time my next enhancement will be to add encryption and I will probably try to test builds on other distributions (maybe attempt to create segmenter binaries).

iPhone Windowed HTTP Live Streaming Using Amazon S3 and Cloudfront Proof of Concept

carson — Mon, 06 Jul 2009 02:07:37 +0000

This post should be seen as a proof of concept. I'm working on creating a more concise and easier to use package of everything covered here but I felt like getting the knowledge out sooner rather than later would be of help to people looking for a way to do this. If you are interested keep an eye on the HTTP live video stream segementer and distributor project page as well as the github git repository.

After my post on using FFMpeg and an open source segmenter to create videos for the iPhone that conform to the HTTP live streaming protocol I decided to see if I could get the same segmenter to work on a live stream. As it turns out it didn't take much modification to work.

If you are looking for something you can buy out of the box it appears that Akamai is doing iPhone video streaming now. I believe that the following solution using Amazon S3 and Cloudfront is probably as good as what Akamai can offer but it may be a better choice if you don't want to have to maintain the configuration.

I put together a quick diagram of the process of transferring the video stream from source to final destination that will hopefully help people understand the full picture before jumping into the details:

Please note that except for the video stream all of the following was done using Fedora 11. I believe it could work on Windows or OS X but I haven't had time to test it on either.

Step 1: Find a suitable video source

This is an important part and can take more work than it seems like it should. I started out trying to stream video from a USB QuickCam but for some reason the resulting stream wasn't correctly formatted even after going through the FFMpeg transcoding. I then turned to the iSight camera on a macbook using QuickTime Broadcaster to VLC streaming. The resulting stream from the iSight camera works well.

The easiest way I found to experiment with finding a good stream is to dump a short clip out to a file then use the instructions in my previous post to test.

It is also possible to do all of the following steps using a non-live video stream. In fact, due to the flexibility of what can be input into FFMpeg, someone could pull a stream from somewhere like Ustream TV or Livestream and rebroadcast it. Doing so would open up the door for live streaming to the iPhone using just Safari.

Step 2: Set up modified segmenter

To handle the live stream I had to modify the segmenter in two ways. The first was required to bypass an issue with the input coming in as a pipe. For those interested here is the modified section of code:

ret = av_write_frame(output_context, &packet);

    if (ret < 0)

    {

      fprintf(stderr, "Could not write frame of stream: %d\n", ret);

      av_free_packet(&packet);

      //break; ****** removed for streaming support *****

    }

    else if (ret > 0)

    {

      fprintf(stderr, "End of stream requested\n");

      av_free_packet(&packet);

      break;

    }

The second modification was more extensive. The original segmenter wrote the index file out for each segment as the segment ended. Instead of writing the index to disk I need the index information to push to S3 as well as to know when the segment itself is ready to be pushed. I could have used a S3 library and stuck everything into the C code but instead I decided that it made more sense to save the stream to disk then push the index information to another process.

I do the transfer of information over a TCP socket connection from the segmenter to the upload process. A side effect of doing this is that it will allow for the upload process to take input from multiple transcode and segmenters at the same time. This should make for easy variable rate configurations where the transcoding can take advantage of multiple machines.

Here is the modified section of code:

int write_index_file(constchar index[],constunsignedint segment_duration,constchar output_directory[],constchar output_prefix[],constchar http_prefix[],constunsignedint first_segment,constunsignedint last_segment,constint end,constchar bucket_name[],constchar key_prefix[])
{
char buffer[1024 * 10];

  memset(buffer, 0,sizeof(char)* 1024 * 10);

  sprintf(buffer,"%s, %s, %d, %s, %s, %d, %d, %d, %s, %s", index, output_directory, segment_duration, output_prefix, http_prefix, first_segment, last_segment, end, bucket_name, key_prefix);
  fprintf(stderr, "Sending: %s\n", buffer);
  int sock;

  if ((sock = socket(PF_INET, SOCK_STREAM, IPPROTO_TCP)) < 0)

  {

    fprintf(stderr, "Could not open socket.");

    return -1;

  }
  const char *serverIP = "127.0.0.1";  

  int serverPort = 10234;
  struct sockaddr_in serverAddress;

  memset(&serverAddress, 0, sizeof(serverAddress));

  serverAddress.sin_family      = AF_INET;

  serverAddress.sin_addr.s_addr = inet_addr(serverIP);

  serverAddress.sin_port        = htons(serverPort);
  if (connect(sock, (struct sockaddr *) &serverAddress, sizeof(serverAddress)) < 0)

  {

    fprintf(stderr, "Could not connect to socket.");

    return -1;

  }
  int buffer_len = strlen(buffer);

  if (send(sock, buffer, buffer_len, 0) != buffer_len)

  {

    fprintf(stderr, "Could not send command.");

    return -1;

  }
  close(sock);
  return 0;

}

The above function assumes that the upload server lives on the same machine as the segmenter at this point. Also note that the command line arguments for the segmenter have grown to include a S3 bucket name and a S3 key prefix:

Usage: live_upload        

input MPEG-TS file – For the live streaming use you want this to be a pipe so it should be -
segment duration in seconds – How long to make each segment of video
output directory – Where the video segments live before they are transfered
output MPEG-TS file prefix – The prefix of the video file
output m3u8 index file – The name of the m3u8 index file
http prefix – The prefix of the URL where the segments are ultimately located
bucket name – The S3 bucket name that the segments and index will be stored in
key prefix – The S3 key that the segments and index should be prefixed with

Download the Makefile and segmenter source and compile if you want to follow step 4.

Step 3: Transfer the segments

The modified segmenter is now ready to push the index information to a process that will in turn upload the index as well as the stream segment. In this case I chose to push the segments and index to S3 but that isn't the only option possible. Furthermore I've stuck Cloudfront in front of the segments so they can be cached closer to the destination. Letting the index files be cached by Cloudfront could be done but care would need to be taken to make sure the index isn't cached for longer than the segment duration.

For the upload server I'm using Ruby and the Rightscale AWS gem to push the segments and the index files to S3. Here is the complete code to do the upload server (this file is called s3server.rb in the git repository):

require'thread'
require'socket'
require'ftools'
require'rubygems'
require'right_aws'
AWS_S3_ID="your s3 id"

AWS_S3_KEY="your s3 private key"
def create_index(segment_duration, output_prefix, http_prefix, first_segment, last_segment, stream_end)

  File.open("tmp.index.m3u8", 'w') do |index_file| 

    index_file.write("#EXTM3U\n")

    index_file.write("#EXT-X-TARGETDURATION:#{segment_duration}\n")

    index_file.write("#EXT-X-MEDIA-SEQUENCE:#{last_segment >= 5 ? last_segment-4 : 1}\n")
  first_segment.upto(last_segment) do | segment_index |

    if segment_index > last_segment - 5

      index_file.write("#EXTINF:#{segment_duration}\n")

      index_file.write("#{http_prefix}#{output_prefix}-%05u.ts\n" % segment_index)

    end

  end
    index_file.write("#EXT-X-ENDLIST") if stream_end

  end

end
def push_to_s3(index, output_directory, bucket_name, key_prefix, output_prefix, last_segment)

  s3 = RightAws::S3Interface.new(AWS_S3_ID, AWS_S3_KEY)
  video_filename = "#{output_directory}/#{output_prefix}-%05u.ts" % last_segment

  puts "Pushing #{video_filename} to s3://#{bucket_name}/#{key_prefix}/#{output_prefix}-%05u.ts" % last_segment

  s3.put(bucket_name, "#{key_prefix}/#{output_prefix}-%05u.ts" % last_segment, File.open(video_filename), {'x-amz-acl' => 'public-read', 'content-type' => 'video/MP2T'})

  puts "Done pushing video file"

  puts "Pushing tmp.index.m3u8 to s3://#{bucket_name}/#{key_prefix}/#{index}"

  s3.put(bucket_name, key_prefix + "/" + index, File.open("tmp.index.m3u8"), {'x-amz-acl' => 'public-read', 'content-type' => 'video/MP2T'})

  puts "Done pushing index file"

end
queue = Queue.new
server_thread = Thread.new do

  server = TCPServer.new('0.0.0.0', 10234)

  while (session = server.accept)

    input = session.gets

    queue << input

    session.close

  end    

end
upload_thread = Thread.new do

  while (value = queue.pop)

    (index, output_directory, segment_duration, output_prefix, http_prefix, first_segment, last_segment, stream_end, bucket_name, key_prefix) = value.strip.split(%r{,\s*})

    if last_segment.to_i > 0

      create_index(segment_duration.to_i, output_prefix, http_prefix, first_segment.to_i, last_segment.to_i, stream_end.to_i == 1)

      push_to_s3(index, output_directory, bucket_name, key_prefix, output_prefix, last_segment.to_i)

    end

  end

end
server_thread.join

To use the above code you will need to put your S3 credentials in place for the values AWS_S3_ID and AWS_S3_KEY.

Step 4: Test it

Set up a HTML file that points to the streaming index file:

<html>
<head>
<title>Video Test</title>
<meta name="viewport" content="width=320; initial-scale=1.0; maximum-scale=1.0; user-scalable=0;"/>
</head>
<body style="background-color:#FFFFFF; ">
<center>
width='150' height='150' src="http://s3.amazonaws.com/ionlivestream/stream0001/stream-128k.m3u8" />
</center>
</body>
</html>

Note that the index file is coming from S3 directly and not from Cloudfront to keep it from being cached and a stale version being served. In case it helps, the format for the source index location in this example is: http://s3.amazonaws.com///
Start the upload script. Once started It will sit and wait for input from the segmenter:

ruby s3server.rb

As requests are made the script will dump output to stdout describing what it is doing.
Configure and start your source video stream. In my case I need to start the VLC to QuickTime Broadcaster connection for the iSight.
Run FFMpeg against the source video stream and pipe the resulting transcoded output into the segmenter. For my stream I used the following command:

ffmpeg -v 0 -i http://192.168.132.101:8080 -f mpegts -acodec libmp3lame -ar 48000 -ab 64k -s 320×240 -vcodec libx264 -b 128k -flags +loop -cmp +chroma -partitions +parti4x4+partp8x8+partb8x8 -subq 5 -trellis 1 -refs 1 -coder 0 -me_range 16 -keyint_min 25 -sc_threshold 40 -i_qfactor 0.71 -bt 200k -maxrate 128k -bufsize 128k rc_eq 'blurCplx^(1-qComp)' -qcomp 0.6 -qmin 10 -qmax 51 -qdiff 4 -level 30 -aspect 320:240 -g 30 -async 2 – | live_upload – 10 /tmp/ sample_128k stream-128k.m3u8 http://d3vmly3syseqo9.cloudfront.net/stream0001/ ionlivestream stream0001
Point safari on your iPhone to the HTML file created in #1 and hit play. After buffering starts you should see the live video with a segment or two delay.

Other information

Again, this is just a proof of concept so there are a number of things lacking. Here are a list of enhancements I could imagine people would find useful:

Variable bitrate segments
Easier and more flexible configuration
Use EC2 for encoding from one source stream
Pluggable transfer to a website using FTP or SCP instead of S3

I thought some about the cost breakdown of doing this using S3 and Cloudfront. The following is a quick calculation on what it might cost for a variable rate stream using the cost of S3 and Cloudfront today.

Some assumptions:
Everyone always finishes the entire stream when they start it.
Variable rates: 128kbps, 256kbps, 364kbps
Video length: 5 minute video
Client streams: 100

30 segments for each stream (5 minutes = 300 seconds / 10 second intervals)

128kbps x 5 minutes = 4.8MB
256kbps x 5 minutes = 9.6MB
364kbps x 5 minutes = 13.65MB

% of each stream rate
25% client streams @ 364kbps = 25 * 13.65MB = 341.25MB
50% client streams @ 256kbps = 50 * 9.6MB = 480MB
25% client streams @ 128kbps = 25 * 4.8MB = 128MB

3 stream index files + 1 variable index file
30 segments * 3 streams = 90 index puts + 90 stream segment puts

S3 put cost:
$0.10 per GB – all data transfer in
$0.01 per 1,000 PUT, COPY, POST, or LIST requests

$0.01 * (180/1000) = $0.0018 + $0.10 * (4.8+9.6+13.65)/1000 = $0.002805 = $0.004605

S3 storage cost:
$0.15 per GB – first 50 TB / month of storage used

$0.15 * (4.8+9.6+13.65)/1000 = $0.0042075

S3 transfer cost:
$0.17 per GB – first 10 TB / month data transfer out
$0.01 per 10,000 GET and all other requests

The index gets pulled from S3 every time, the streams come from Cloudfront
Assumes the transfer to Cloudfront happens 3 times per stream
$0.17 * (3 * (4.8+9.6+13.65)) / 1000 = $0.0143055
$0.01 * (100 * 30) + (30 * 3) / 10000 = $0.00309

Cloudfront transfer cost:
$0.17 per GB – first 10 TB / month data transfer out
$0.01 per 10,000 GET requests

$0.17 * (341.25 + 480 + 128) / 1000 = $0.1613708
$0.01 * (30 * 100) / 10000 = $0.003

Rough total cost: $0.1933833

So for 100 streams of 5 minutes worth of video you would be looking at something around 20 cents.

iPhone HTTP Streaming with FFMpeg and an Open Source Segmenter

carson — Sun, 28 Jun 2009 23:43:45 +0000

With the release of the iPhone OS 3 update came the ability to do live streaming. There are a few types of streaming and each requires a certain encoding and segmentation. I've put together a cheat sheet on how I went about building a static stream using FFMpeg and an example segmenter that someone has posted. I'm not covering windowed streams in this post but if you are thinking about implementing a windowed stream the following will help you make a step in that direction.

Before getting started it is best to read over the Apple documentation on HTTP live streaming. Start out with the iPhone streaming media overview. This document covers the basics of how the streaming works and has some nice diagrams.

If you want even more information after reading the overview you can take a look at the HTTP Live streaming draft proposal that was submitted to the IETF by Apple. It covers the streaming protocol in complete detail and has examples of the stream file format for reference.

Once you are ready to start grab a decent quality video clip to use. If you don't have one handy I found a nice list of downloadable HD clips in various formats for testing.

Step 1: Grab the latest version of FFMpeg

You may be able to get away with anything after FFMpeg 0.5 but you might as well pull down a more recent version. The FFMpeg download page has instructions on getting the latest version. I pulled the version I used out of git.

I used the following command to configure FFMpeg:


configure --enable-gpl --enable-nonfree --enable-pthreads --enable-libfaac --enable-libfaad --enable-libmp3lame --enable-libx264

One of the main things to note is the --enable-libx264 flag.

Step 2: Encode your video for the iPhone

Once you have a working version of FFMpeg it is time to create an X264 encoded stream that will work with the iPhone. There are a few things to note before diving in:

The supported bitrates for streaming are: 100 Kbps to 1.6 Mbps
The suggested bitrates for streaming are*:
- Low – 96 Kbps video, 64 Kbps audio
- Medium – 256 Kbps video, 64 Kbps audio
- High – 800 Kbps video, 64 Kbps audio
The iPhone screen size is: 480×320

* See step 7 for more information on what I think are better bitrates.

Taking all that into account someone on the iPhone developer forums suggested the following and it works well for me:

ffmpeg -i  -f mpegts -acodec libmp3lame -ar 48000 -ab 64k -s 320×240 -vcodec libx264 -b 96k -flags +loop -cmp +chroma -partitions +parti4x4+partp8x8+partb8x8 -subq 5 -trellis 1 -refs 1 -coder 0 -me_range 16 -keyint_min 25 -sc_threshold 40 -i_qfactor 0.71 -bt 200k -maxrate 96k -bufsize 96k -rc_eq 'blurCplx^(1-qComp)' -qcomp 0.6 -qmin 10 -qmax 51 -qdiff 4 -level 30 -aspect 320:240 -g 30 -async 2 

If you want some more detail on some of these commands check out the X264 encoding guide and in general the FFMpeg documentation to see what all the flags mean.

Note that I have the bitrate set to 96k in the above example. That can be changed to fit your needs. Use the script that I have created later in the post or just make sure you change the -b, -maxrate, and -bufsize values.

Step 3: Download and build the segmenter

Now you have a complete video but you don't want to toss the entire thing up or you wouldn't be reading about HTTP streaming. What you need is a way to segment the video stream into smaller chunks. You can download Apple's segmenter (see the overview above for more information on where to find it) or you can download one created by the forum user corp186.

There is an SVN repository set up for the segmenter source. It is only a couple files and it is easy to build. The trouble you may run into is that the Makefile that it comes with won't build the binary correctly. Don't worry it just takes some extra link flags to make it work. The following is what I needed in the Makefile to get it to build on my system:

all:

        gcc -Wall -g segmenter.c -o segmenter -lavformat -lavcodec -lavutil -lbz2 -lm -lz -lfaac -lmp3lame -lx264 -lfaad
clean:

        rm segmenter

After compiling the segmenter you are ready to create your first HTTP streaming content.

The format of the segmenter command is:

segmenter     

Following is an example used to create a stream from a video file created with the above FFMpeg command split into 10 second intervals:

segmenter sample_low.ts 10 sample_low stream_low.m3u8 http://www.ioncannon.net/

Step 4: Prepare the HTTP server

At this point you should have a set of files that represent the stream and a stream definition file. Those files can be uploaded to a web server at this point but there is another important step to take that ensures they will be download correctly and that is setting up mime types. There are two mime types that are important for the streaming content:

.m3u8 application/x-mpegURL

.ts video/MP2T

If you are using Apache you would want to add the following to your httpd.conf file:

AddType application/x-mpegURL .m3u8

AddType video/MP2T .ts

If you are using lighttpd you would want to put this in your configuration file (if you have other mime types defined make sure you just add these and don't set them):

mimetype.assign = ( ".m3u8" => "application/x-mpegURL", ".ts" => "video/MP2T" )

Step 5: Test the stream

The video is encoded for the iPhone, segmented for streaming, and the server is configured. The only thing left to do is test the stream and the fastest way to do that is to use the new HTML5 video tag. Here is an example of how to set it up:

<html>

  <head>

    <title>Video Test</title>

    <meta name="viewport" content="width=320; initial-scale=1.0; maximum-scale=1.0; user-scalable=0;"/>

  </head>

  <body style="background-color:#FFFFFF; ">

    <center>

      width='150' height='150' src="stream-128k.m3u8" />

    </center>

  </body>

</html>

If everything has been done correctly you should see the video.

If you want to test the stream out in an application then download the MoviePlayer iPhone demo application from the iPhone developer site. Build and run it in the simulator or put it on an actual phone and then type the URL in for the server you uploaded your stream to.

That is all there is to building a single static HTTP stream. A good number of steps but if you have some experience using FFMpeg it isn't too hard to set up. The only pitfalls I ran into revolve around trying to segment the stream without the segmeter code. I don't know enough about how the segmentation works to know why this is so difficult to do but I believe it could have something to do with synchronization points in the stream. Of course when you stray from the path the stream just doesn't work and you get a generic error message so that is just my best guess. I'll also guess that Apple may tighten up the player over time and make it work better with miss-formatted streams.

Step 6: Automating the stream encoding and segmentation

Here is a little script I put together that first encodes an input file and then segments it into 10 second chunks:

#!/bin/sh
BR=800k
ffmpeg -i $1 -f mpegts -acodec libmp3lame -ar 48000 -ab 64k -s 320×240 -vcodec libx264 -b $BR -flags +loop -cmp +chroma -partitions +parti4x4+partp8x8+partb8x8 -subq 5 -trellis 1 -refs 1 -coder 0 -me_range 16 -keyint_min 25 -sc_threshold 40 -i_qfactor 0.71 -bt 200k -maxrate $BR -bufsize $BR -rc_eq 'blurCplx^(1-qComp)' -qcomp 0.6 -qmin 10 -qmax 51 -qdiff 4 -level 30 -aspect 320:240 -g 30 -async 2 sample_$BR_pre.ts
segmenter sample_$BR_pre.ts 10 sample_$BR stream-$BR.m3u8 http://www.ioncannon.net/
rm -f sample_$BR_pre.ts

The script could use some work but it does a good enough job for testing.

Step 7: Create a variable rate HTTP stream

Once you have creating a single stream down you need to try out creating a variable bitrate stream. There isn't much to it, just create different bitrate encoded streams and link to their stream definition files in a separate stream definition file. Here is an example:

#EXTM3U

#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=96000

http://192.168.132.15/ipv/stream-96k.m3u8

#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=256000

http://192.168.132.15/ipv/stream-256k.m3u8

#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=800000
http://192.168.132.15/ipv/stream-800k.m3u8

I gave the above a try using both the iPhone's 3G connection and a WIFI connection. The following log shows the two different attempts (first 3G then WIFI):


6.x.x.x ioncannon.net - [20:49:13] "GET /varpl.m3u8 HTTP/1.1" 304 0 "-" "..."
6.x.x.x ioncannon.net - [20:49:14] "GET /varpl.m3u8 HTTP/1.1" 206 288 "-" "..."
6.x.x.x ioncannon.net - [20:49:15] "GET /varpl.m3u8 HTTP/1.1" 200 288 "-" "..."
6.x.x.x ioncannon.net - [20:49:16] "GET /stream-96k.m3u8 HTTP/1.1" 200 719 "-" "..."
6.x.x.x ioncannon.net - [20:49:18] "GET /s_96k-00001.ts HTTP/1.1" 200 334828 "-" "..."
6.x.x.x ioncannon.net - [20:49:21] "GET /s_96k-00002.ts HTTP/1.1" 200 377880 "-" "..."
6.x.x.x ioncannon.net - [20:49:30] "GET /s_96k-00003.ts HTTP/1.1" 200 383520 "-" "..."
6.x.x.x ioncannon.net - [20:49:32] "GET /stream-256k.m3u8 HTTP/1.1" 200 730 "-" "..."
6.x.x.x ioncannon.net - [20:49:39] "GET /s_256k-00003.ts HTTP/1.1" 200 716844 "-" "..."
6.x.x.x ioncannon.net - [20:49:49] "GET /s_256k-00004.ts HTTP/1.1" 200 705564 "-" "..."
6.x.x.x ioncannon.net - [20:49:57] "GET /stream-96k.m3u8 HTTP/1.1" 200 719 "-" "..."
6.x.x.x ioncannon.net - [20:49:59] "GET /s_96k-00004.ts HTTP/1.1" 200 368668 "-" "..."
6.x.x.x ioncannon.net - [20:50:03] "GET /s_96k-00005.ts HTTP/1.1" 200 371300 "-" "..."
6.x.x.x ioncannon.net - [20:50:13] "GET /s_96k-00006.ts HTTP/1.1" 200 398936 "-" "..."
6.x.x.x ioncannon.net - [20:50:16] "GET /stream-256k.m3u8 HTTP/1.1" 200 730 "-" "..."
6.x.x.x ioncannon.net - [20:50:22] "GET /s_256k-00006.ts HTTP/1.1" 200 758016 "-" "..."
6.x.x.x ioncannon.net - [20:50:36] "GET /s_256k-00007.ts HTTP/1.1" 200 737524 "-" "..."
6.x.x.x ioncannon.net - [20:50:40] "GET /s_256k-00008.ts HTTP/1.1" 200 773244 "-" "..."
6.x.x.x ioncannon.net - [20:50:46] "GET /s_256k-00009.ts HTTP/1.1" 200 717032 "-" "..."
6.x.x.x ioncannon.net - [20:50:57] "GET /s_256k-00010.ts HTTP/1.1" 200 768920 "-" "..."
6.x.x.x ioncannon.net - [20:51:06] "GET /s_256k-00011.ts HTTP/1.1" 200 611000 "-" "..."

1.x.x.x ioncannon.net - [20:52:23] "GET /varpl.m3u8 HTTP/1.1" 304 0 "-" "..."
1.x.x.x ioncannon.net - [20:52:24] "GET /varpl.m3u8 HTTP/1.1" 206 288 "-" "..."
1.x.x.x ioncannon.net - [20:52:25] "GET /varpl.m3u8 HTTP/1.1" 200 288 "-" "..."
1.x.x.x ioncannon.net - [20:52:25] "GET /stream-96k.m3u8 HTTP/1.1" 200 719 "-" "..."
1.x.x.x ioncannon.net - [20:52:26] "GET /s_96k-00001.ts HTTP/1.1" 200 334828 "-" "..."
1.x.x.x ioncannon.net - [20:52:27] "GET /s_96k-00002.ts HTTP/1.1" 200 377880 "-" "..."
1.x.x.x ioncannon.net - [20:52:28] "GET /stream-800k.m3u8 HTTP/1.1" 200 730 "-" "..."
1.x.x.x ioncannon.net - [20:52:31] "GET /s_800k-00002.ts HTTP/1.1" 200 1774156 "-" "..."
1.x.x.x ioncannon.net - [20:52:34] "GET /s_800k-00003.ts HTTP/1.1" 200 1916096 "-" "..."
1.x.x.x ioncannon.net - [20:52:38] "GET /s_800k-00004.ts HTTP/1.1" 200 1831872 "-" "..."
1.x.x.x ioncannon.net - [20:52:41] "GET /s_800k-00005.ts HTTP/1.1" 200 1831496 "-" "..."
1.x.x.x ioncannon.net - [20:52:46] "GET /s_800k-00006.ts HTTP/1.1" 200 1967608 "-" "..."
1.x.x.x ioncannon.net - [20:52:50] "GET /s_800k-00007.ts HTTP/1.1" 200 1676208 "-" "..."
1.x.x.x ioncannon.net - [20:52:54] "GET /s_800k-00008.ts HTTP/1.1" 200 2094132 "-" "..."
1.x.x.x ioncannon.net - [20:52:58] "GET /s_800k-00009.ts HTTP/1.1" 200 1860260 "-" "..."
1.x.x.x ioncannon.net - [20:53:08] "GET /s_800k-00010.ts HTTP/1.1" 200 2008404 "-" "..."
1.x.x.x ioncannon.net - [20:53:19] "GET /s_800k-00011.ts HTTP/1.1" 200 1400224 "-" "..."

Notice that there is a decent bit of indecisiveness on the part of what stream to pick when using 3G. For my test it actually caused the player to pause while it switched from the 256k stream back to the 96k stream. The stream on the WIFI connection starts out low but then jumps right to the highest quality and stays there. Overall it seems like the variable rate streaming works decently and again Apple may be able to tweak it down the road to get even better results.

The bitrate jump between 96k and 256k is probably too large even though that is what Apple seems to recommend. I believe with some testing a better set of bitrates could be found. The video quality of the 256k bitrate looks pretty good so I would say that 96k, 128k and 384k would potentially be a better choice.

Some parting notes

If you are interested in how the segmenter works you can find out more on how to use libavformat at the following resources: an older libavformat tutorial, some sample libavformat code, How to Write a Video Player in Less Than 1000 Lines, and more sample libavformat code.

The next step for this is to do a windowed live stream. I've done a little experimenting so far and with a modified segmeter I can generate a live stream. I will need to heavily modify the segmeter to get a live windowed stream so it may take a little while to get it done. My intent of course will be to combine the modifications with something fun like S3 and cloudfront since I believe that would be a sweat combination.

How to create iPhone wireframes with Inkscape

carson — Thu, 16 Apr 2009 01:03:01 +0000

While developing ideas for iPhone applications I've played around with just using Interface Builder to stub things out. This works reasonably well but I know how to use Interface Builder so that makes a difference. If you are designing an application and want to stick with graphical tools only you are in luck because Yahoo has produced a set of wireframing SVG stencils for the iPhone that are very nice.

This is what the complete iPhone UI template set looks like:

As you can see it has all of the standard UI elements of the iPhone.

Using the stencils is easy. Open the SVG file with Inkscape and then create a new empty project. Select the UI element you want and copy it into your project:

Knowing some of the Inkscape shortcut keys helps a lot in creating your wireframe. Being able to align and move elements in z-ordering is much easier if you know the correct shortcut keys to use.

Here are a few example images that I made using Inkscape for a demo application:

If you are new to wireframing it is worth reading the 20 steps to better wireframing post by Think Vitamin. One of the 20 steps is not to worry about making your wireframes look exactly like the final product. If you need it to look perfect then using Interface Builder is probably going to work out better.

As a side note I wish Inkscape had a resource for stencils like OmniGraffle does. There are any number of people who publish stencils on their blogs and there is even a site specifically for OmniGraffle stencils. I would add stencils to my list of tips for creating diagrams using Inkscape if it existed. Right now it is seems like the best option is to look for SVG formatted stencils. The gstencil format doesn't seem too complicated so maybe someone will eventually find a way to convert the contents into SVG where they could then be used in Inkscape.

Update: If you have Adobe Fireworks or have money to spend on doing this you should check out this video on building mockups for the iphone using Adobe Fireworks.

Converting videos (flv,wmv,avi,etc) into a format that will work with the iPhone/iTunes

carson — Sun, 28 Dec 2008 02:37:20 +0000

I have had an iPhone for a while now and I keep running into instances where I want to have videos outside of youtube in some format that I can watch on the device. These include windows WMV formatted videos from PDC as well as FVL formatted videos on Flex. I finally broke down and found a working solution to convert pretty much any video into a iPhone/iTunes format using mencoder.

Here is the script I came up with to convert a video:

#!/bin/sh

BN=`basename $1 .flv`

mencoder $1 -ofps 23.976 -ovc lavc -oac copy -o /tmp/test.$BN.avi

mencoder /tmp/test.$BN.avi -o togo/$BN.mp4 -vf scale=480:-10,harddup -lavfopts format=mp4 -faacopts mpeg=4:object=2:raw:br=128 -oac faac -ovc x264 -sws 9 -x264encopts nocabac:level_idc=30:bframes=0:global_header:threads=auto:subq=5:frameref=6:partitions=all:trellis=1:chroma_me:me=umh:bitrate=500 -of lavf

rm /tmp/test.$BN.avi

One thing to notice is that I have the file format hard coded in the basename argument. I did this because I didn't need the script to be that generic and I just change it when I move from one format to another. For some reason I found that converting the video with the first statement works best. Without that I had a number of audio sync issues as well as some outright conversion failures .

The options for the first command are

-ofps output frames per second
-ovc lavc output codec is lavc
-oac copy output audio is a copy of the input
-o the output file

The options for the second command are

-o the output file
-vf scale=480:-10,harddup video filter that scales to 480 width and rounds the height to the closest 10/16th width as well as forcing duplicate frames to be encoded in the output
-lavfopts format=mp4 set the video format options to mp4
-faacopts mpeg=4:object=2:raw:br=128 set the audio options to mp4
-oac faac set the audio output format to faac
-ovc x264 set the output to x264 encoding
-sws 9 set the scaler options to lanczos
-x264encopts nocabac:level_idc=30:bframes=0:global_header:threads=auto:subq=5:frameref=6:partitions=all:trellis=1:chroma_me:me=umh:bitrate=500 sets the 265 enocoding options
-of lavf set the output video format

After converting the video all you need to to do is drag it onto the iTunes Library bar and it will then import it and make it available to sync to your iPhone, touch or iPod.