You may wonder why you would want to do this. I'll give you at least two reasons. First there is speed. The configuration I ended up with will boot in about 14 seconds. Almost all of that time is spent waiting on the AWS DHCP server for an IP address. The second reason is security. There is little on this system that you don't need. The majority of what you need is provided by a single application that you can easily monitor for patches. That leaves you to worry about only the tools you must have to get your job done. I'll also include a couple downsides. First this isn't going to be easy for everyone and it can be fairly complicated. The second issue you will run into is that there is nothing in this distribution. If you need something like PHP you might spend a long time building all the support you need. There isn't much to be done about the complexity if you want to do this all yourself. Selection of tools can help you lessen the impact of the bare nature of the system, more on that in a future post.

I picked TTYLinux because that is what I'm most familiar with. I have used it a number of times and I believe it may be the most basic of all the small Linux distros. It really is bare and the large majority of the features are provided by BusyBox. Before attempting to build your own instance it may help to read some of the TTYLinux documentation. Although I haven't tried I assume that other small distros like DSL would probably work with the same general instructions.

A couple important prerequisites are needed before starting this. First look at the How_To_Build_ttylinux.txt file that comes with the TTYLinux source. It contains a lot of information about how you go about building the system. Next look at my post on compiling the Linux kernel for EC2 because you will need to do that for this TTYLinux instance. Finally you need to have to have gcc, g++ and yacc available on the machine you are going to use to build on.

I have also created a public AMI that anyone can use with the name TTYLinuxBase and AMI ID of ami-0cfe0b65. Be sure to boot it with the correct hd0 kernel for your region. The password for root on this AMI is just password so be sure to change it when you fire it up. I have created the AMI so that the EBS volume gets deleted when it is terminated.

1. Grab the source from the TTYLinux source page:

   wget http://minimalinux.org/ttylinux/Download/ttylinux-src-mp3.tar.bz2
   tar xvjf ttylinux-src-mp3.tar.bz2
   

2. The kernel that comes with the current version of TTYLinux isn't new enough to work on EC2 so the configuration and build scripts need to be changed to get a more recent version that will. The first step is to create a kernel configuration and put it in the correct location:

   ttylinux-src-mp3/config/platform_pc/kernel-2.6.35.4-i686.cfg
   

   You can create this configuration by doing a make menuconfig on the kernel you want to use and then saving the kernel configuration to a file. You may also download the kernel config I used:

   wget http://www.ioncannon.net/examples/ec2ttylinux/kernel-2.6.35.4-i686.cfg
   cp kernel-2.6.35.4-i686.cfg ttylinux-src-mp3/config/platform_pc/kernel-2.6.35.4-i686.cfg
   

   There is a lot of the kernel that can be pruned from the kernel configuration so if you do this by hand take time to turn things off that you won't need. Doing so will speed up the kernel compile and make the system boot faster. The above kernel configuration is pruned down quite a bit.

3. Copy the packages configuration to one that matches the kernel version:

   cp ttylinux-src-mp3/config/platform_pc/packages-2.11-2.6.30.5.txt ttylinux-src-mp3/config/platform_pc/packages-2.11-2.6.35.4.txt
   

4. Modify the kernel build script to apply the XSAVE patch. Start by adding the following line to line 132 of the ttylinux-src-mp3/scripts/build-kernel.sh build script:

   patch -p1 < /tmp/kernel.patch
   

   You should end up with the following around it:

   echo -n "b." >&${CONSOLE_FD}
   cp "${kcfg}" "linux-${kver}/.config"
   cd "linux-${kver}"
   patch -p1 < /tmp/kernel.patch
   sed --in-place scripts/mod/sumversion.c \
           --expression="s|<string.h>| <limits.h>\n#include <string.h>|"
   source "${TTYLINUX_XTOOL_DIR}/_xbt_env_set"
   

   Create the patch file /tmp/kernel.patch with the following in it:

   --- a/arch/x86/xen/enlighten.c  2010-08-05 20:35:13.000000000 -0400
   +++ b/arch/x86/xen/enlighten.c  2010-08-05 20:35:22.000000000 -0400
   @@ -776,6 +776,7 @@
    {
    	cr4 &= ~X86_CR4_PGE;
    	cr4 &= ~X86_CR4_PSE;
   +	cr4 &= ~X86_CR4_OSXSAVE;
   
    	native_write_cr4(cr4);
    }
   

   Note that the above patch file has tabs in it. Make sure there is a tab before each cr4 line and the native_write_cr4 line or download the kernel patch file I created.

5. Select the correct build target in the ttylinux-src-mp3/ttylinux.dist-config.sh file. Find the TTYLINUX_TARGET lines, comment them all out and then add the following line:

   TTYLINUX_TARGET="i686:pc:2.11-2.6.35.4"
   

6. Set up the cross compile tools build directory:

   cp -Rp ttylinux-src-mp3/cross-tools-2.11-2.6.30.5/ ttylinux-src-mp3/cross-tools-2.11-2.6.35.4/
   

   Find XBT_KERNEL in the script ttylinux-src-mp3/cross-tools-2.11-2.6.35.4/_scripts/xbt-versions.sh and change it to:

   XBT_KERNEL="linux-2.6.35.4"
   

7. Build the cross compile tools:

   cd ttylinux-src-mp3/cross-tools-2.11-2.6.35.4
   make setup
   make dload
   make i686
   cd ..
   

8. Build the entire TTYLinux distro:

   make dist
   

   From this point the instructions are just like the last few posts I have made. The distribution created in here needs to be transferred to an EBS volume and made into an AMI.

9. Using a temporary EC2 instance create a volume that will be used to boot the install and give it a file system:

   ec2-create-volume -z us-east-1a -s 1
   ec2-attach-volume volume-id -i instance-id -d /dev/sdh
   mkfs.ext3 /dev/sdh
   

10. Transfer the TTYLinux distribution image that was created to the temporary EC2 instance. Then copy the image to the boot volume:

    mkdir /mnt/dest
    mkdir /mnt/src
    mount /dev/sdh /mnt/dest
    mount -o loop img/file_sys-i686-11.2.img /mnt/src
    cd /mnt/src
    find . | cpio -pvd /mnt/dest
    cd -
    cp config/boot/* /mnt/dest/boot/
    umount /mnt/src
    

    In the above the boot volume is mounted, the TTYLinux distribution image is mounted via a loop device and then all the files are copied from the image to the boot volume. I do a copy here because the default boot image that the TTYLinux build creates is only 8M and the smallest EBS volume you can create is 1G so it is better to use what you have to have.

11. While the boot volume is mounted there are a number of things that need to be configured that are different than some of the previous examples of booting custom systems in EC2. First set the password for root by using chroot:

    chroot /mnt/dest/ passwd
    

    Next change the root device in fstab:

    cat <<EOF > /mnt/dest/etc/fstab
    /dev/xvda1    /            ext3       defaults                      0 0
    tmpfs         /dev         tmpfs      noauto                        0 0
    devpts        /dev/pts     devpts     gid=5,mode=0620               0 0
    tmpfs         /dev/shm     tmpfs      rw,noexec,nosuid,size=24k     0 0
    proc          /proc        proc       noauto                        0 0
    sysfs         /sys         sysfs      noauto                        0 0
    EOF
    

    Then modify the inittab to only create one terminal:

    cat <<EOF > /mnt/dest/etc/inittab
    ::sysinit:/etc/rc.d/rc.sysinit
    
    tty1::respawn:/sbin/getty 38400 tty1
    
    ::ctrlaltdel:/sbin/reboot
    ::shutdown:/etc/rc.d/rc.sysdone
    EOF
    

    Then configure the network to start on boot:

    cat <<EOF > /mnt/dest/etc/sysconfig/network-scripts/ifcfg-eth0
    ENABLE=yes
    NAME=Ethernet
    IPADDRESS=192.168.1.20
    CIDRLEN=24
    NETWORK=192.168.1.0
    NETMASK=255.255.255.0
    GATEWAY=192.168.1.1
    BROADCAST=192.168.1.255
    DHCP=yes
    EOF
    

    Finally, If you want to get rid of the warnings about setting the hardware clock you can edit /mnt/dest/etc/rc.d/rc.sysdone and comment out the part that tries to set it:

    # Disabled for XenU use
    #if [[ "$(uname -m)" != armv5tej* ]]; then
    #     action $"syncing hardware clock to system time" hwclock ${HWCLOCKARGS}
    #fi
    

    and you will also want to do the same to /mnt/dest/etc/rc.d/rc.sysinit:

    # Disabled for XenU
    #if [[ "$(uname -m)" != armv5tej* ]]; then hwclock ${HWCLOCKARGS}; fi
    

12. To tell pv-grub what to boot you will need a /boot/grub/menu.lst file with the following in it:

    mkdir /mnt/dest/boot/grub
    
    cat <<EOF > /mnt/dest/boot/grub/menu.lst
    default 0
    timeout 0
    title TTYOS
            root (hd0)
            kernel /boot/vmlinuz root=/dev/xvda1
    EOF
    

    Unmount the boot image:

    umount /mnt/dest/
    

13. Snapshot the volume and register it as an AMI:

    ec2-create-snapshot -d "Volume Description" volume-id
    ec2-register -n "AMIName" -d "AMI Description" --root-device-name /dev/sda1 -b /dev/sda1=snap-id:1:true
    

14. Boot it using the hd0 kernel for your region (in my case that is aki-407d9529):

    ec2-run-instances -z us-east-1a -g your-group -k your-keypair -n 1 --kernel pv-grub-kernel-id ami-from-step-13
    

There are a few final notes that might be interesting. The smallest you can create is 1G and that is about 950M too large. This is probably not a real issue since you will most likely want space to put your application but it is interesting to note. The instructions assume you are building a i386 instance but they are almost the same for a 64 bit instance.

With the kernel config I provide you will see boot times from start to init in about 0.3 seconds. That is pretty fast. From init to login is fast as well but depends completely on how long it takes to get an IP from the DHCP server. This type of system could potentially boot in just a second or two if it didn't have to wait for any AWS parts.

A few people have created TTY addons to make compiling code for TTYLinux easier. You may want to check those out. Baring that you will find instructions on building anything with the cross compiling system in the TTYLinux howto documentation. Of course you may be able to bypass any pain there by compiling static binaries or even using something like Java.

It is important to have a recent kernel since most of what is needed to get a kernel to work on EC2 is now incorporated into the source. These instructions assume the latest kernel is 2.6.35.4 and I've used them with 2.6.35 as well but keep that in mind since the one patch that is required could eventually be merged in. Before getting started it may help to read over how to compile the Linux kernel normally and then my post on running CentOS 5.5 on EC2 using pv-grub.

Before you begin you will need a place to build the kernel. For these instructions I used an EC2 instance to build the kernel but you don't have to. I also installed the kernel on the same EC2 instance when I was done. The AMI I used was Amazon's EBS boot starter ( ami-b232d0db : amazon/getting-started-with-ebs-boot ).

The following steps go over building and installing the kernel in detail:

1. Download the latest Linux kernel or the version I'm using:

   wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.35.4.tar.bz2
   

2. Configure the source to run on EC2:

   make menuconfig
   

   You will need to make sure the following options are set in the configuration:

   • "Processor type and features" -> "High Memory Support" -> Make sure it is set to 64GB
   • "Processor type and features" -> "PAE (Physical Address Extension) Support" -> enable
   • "Processor type and features" -> "Paravirtualized guest support" -> enable
   • "Processor type and features" -> "Paravirtualized guest support" -> "Xen guest support" -> enable
   • "Device Drivers" -> "Block devices" -> "Xen virtual block device support" -> enable either as a module or built in
   • "Device Drivers" -> "Network device support" -> "Xen network device frontend driver" -> enable either as a module or built in

   
   

   If you want you can make the device drivers modules but you have to have them so it is probably best to just compile them into the kernel itself. If you want to compare your config file with the one I used you can download mine here: kernel-2.6.35.4-i686.config.

   The starter AMI I used needed ncurses development libraries and gcc installed for menuconfig to work:

   yum install ncurses-devel gcc
   

   If you want to know a little more about what is being enabled in this step see the "Building with domU support" section of XenParavirtOps.

3. Apply the following patch to disable XSAVE:

   --- a/arch/x86/xen/enlighten.c	2010-08-05 20:35:13.000000000 -0400
   +++ b/arch/x86/xen/enlighten.c	2010-08-05 20:35:22.000000000 -0400
   @@ -776,6 +776,7 @@
    {
    	cr4 &= ~X86_CR4_PGE;
    	cr4 &= ~X86_CR4_PSE;
   +	cr4 &= ~X86_CR4_OSXSAVE;
   
    	native_write_cr4(cr4);
    }
   
   cd /path/to/root/of/kernel/source
   
   patch -p1 < /tmp/kernel.patch
   

   Note that the above patch file has tabs in it. Make sure there is a tab before each cr4 line and the native_write_cr4 line. If you want to you can download a copy of the patch with the tabs in it to be sure.

4. Build the kernel and install it if you are on the same machine you want to install it on. If you need help compiling the kernel refer to the kernel compile howto.
   
   

   After this step you have a kernel, modules and initrd that you can use. The remaining steps go over using it.

5. Configure the /boot/grub/menu.lst file on the target AMI to use the new kernel, the following is an example of the contents of the file:

   default 0
   timeout 1
   title Test
        root (hd0)
        kernel /boot/vmlinuz-2.6.35.4 root=/dev/xvda1
        initrd /boot/initrd-2.6.35.4.img
   

   Note that the root device here is /dev/xvda1 instead of /dev/sda1. This is caused by the XSAVE patch.

6. Verify that your /etc/fstab file is correct. If your previous root device was /dev/sda1 it is going to be /dev/xvda1 now. The contents of the fstab file I used follow:

   /dev/xvda1                              /                       ext3    defaults 1 1
   /dev/mapper/swapVG-swapFS               swap                    swap    defaults 0 0
   /dev/mapper/storageVG-storageFS         /mnt                    ext3    defaults 0 0
   none                                    /dev/pts                devpts  gid=5,mode=620 0 0
   none                                    /dev/shm                tmpfs   defaults 0 0
   none                                    /proc                   proc    defaults 0 0
   none                                    /sys                    sysfs   defaults 0 0
   

7. Make a snapshot of the volume and register it as an AMI:

   ec2-create-snapshot -d "Snapshot Description" volume-id
   ec2-register -n "CustomKernel" -d "Custom kernel AMI" --root-device-name /dev/sda1 -b /dev/sda1=snap-id:15:true
   

   Note that the devices here are /dev/sda1 and not /dev/xvda1. That is a little confusing but the AWS system doesn't see the devices in the same way your AMI will once it is booted.

8. Start the instance. In my case using the hd0 pv-grub kernel:

   ec2-run-instances -z us-east-1a -g your-group -k your-keypair -n 1 --kernel aki ami-from-step-7
   

If all goes well you should be able to run dmesg and see a boot message something like the following at the top:

Reserving virtual address space above 0xf5800000
Linux version 2.6.35.4 (root@domU) (gcc version 4.1.2 20070925 (Red Hat 4.1.2-33)) #2 SMP Mon Aug 23 20:00:01 EDT 2010
BIOS-provided physical RAM map:
 Xen: 0000000000000000 - 00000000000a0000 (usable)
 Xen: 00000000000a0000 - 0000000000100000 (reserved)
 Xen: 0000000000100000 - 000000006a400000 (usable)
NX (Execute Disable) protection: active
...

With the ability to create a custom kernel for EC2 the next step is to prune the OS itself down to the bare minimum.

Some of what follows is exactly the same as my post about installing CentOS 5.5 on EC2 with the stock kernel. Once you get the hang of it you can install just about anything Linux based to EC2. I've broken this post into two parts to try and separate the generic transfer information from the specifics of an example. The first part that goes over the basics of what needs to be done to transfer any VirtualBox or VMWare box to EC2. The second part is an example of transferring an Ubuntu Server install to from VirtualBox to EC2.

The following steps should work for any Linux OS but the main sticking point is that the kernel needs to be compatible with EC2 (see the anouncment for a list of some distros that have compatible kernels and remember you can always compile by hand as well).

1. Make sure your VirtualBox or VMWare install has a kernel that is able to boot on EC2.
2. Make sure you have grub installed and set up your /boot/grub/menu.lst file to point to the correct Xen kernel and initrd file. See the Ubuntu instructions that follow for an example menu.lst.
3. Make sure you have your /boot directory in the correct place. If your disk is just one big partition you shouldn't need to do anything (use the hd0 PV-Grub kernel in the last step when booting). If you have a partition for /boot you will need to make sure it is the first partition on the disk and you will also need to copy the directory into /boot/boot so that the menu.lst file would be in /boot/boot/grub/menu.lst after the copy. For an example of this see the Ubuntu instructions that follow (use the hd00 PV-Grub kernel in the last step when booting).
4. Export the image to a raw form. This includes the partition table and everything. For VMWare you can use qemu-img to convert the VMDK to a raw image with the following command:

      qemu-img convert -O raw vmware-image.vmdk myosimage.raw
   

   Or if you are using VirtualBox you would use the VBoxManage command (I'm currently using VirtualBox 3.2.6 and since the VBoxManage command has changed before that might be important for those reading this at a later date):

      VBoxManage internalcommands converttoraw myosimage.vdi myosimage.img
   

   In both cases you should be able to run fdisk against the resulting raw image file and see a partition table:

      fdisk -lu myosimage.img
   

5. Create an EBS volume that is the same size or larger than the raw disk image created in step 4 and then attach it to a running EC2 instance:

   ec2-create-volume -z us-east-1a -s 2
   ec2-attach-volume volume-id -i instance-id -d /dev/sdh
   

   Note: The instance you attach the volume to is just a place used to copy the image file onto the volume. There is no need to be too picky about the type of instance it is as long as you have access to the dd command that should be all you need.

6. Transfer the exported VirtualBox or VMWare disk image to the running EC2 instance created in step 5 and then copy it to the volume with the following dd command (this example assumes the volume is attached to the /dev/sdh device):

   dd if=myosimage.img of=/dev/sdh bs=10M
   

   Note: It may make sense to compress your raw image before transferring it since any free space will be represented in the raw file and will compress down a lot.

7. Make a snapshot of the volume:

   ec2-create-snapshot -d "Volume Description" volume-id
   

8. Register the snapshot as a new AMI:

   ec2-register -n "AMIName" -d "AMI Description" --root-device-name /dev/sda2 -b /dev/sda=snap-id:2:true
   

9. Boot the new AMI using the correct PV-Grub kernel. The correct kernel will depend on how your partition structure. For more information see the AWS PV-Grub documentation. The main thing to know is that if you have /boot on its own partition use the hd00 kernel otherwise use the hd0 kernel:

   ec2-run-instances -z us-east-1a -g your-group -k your-keypair -n 1 --kernel pv-grub-kernel-id ami-from-step-19
   

The following is a specific example of converting an existing Ubuntu 10.04 LTS Server installed on VirtualBox to a bootable EC2 AMI. Note that steps 1 to 4 of the following create my "existing" Ubuntu Server so skip those if you have something that already exists. The last portion of this will look just like the above generic steps:

1. Use the Ubuntu Server ISO to install Ubuntu on a fresh VirtualBox VM. For this example I used 512M of memory and a 2G disk. This is what my VirtualBox setup looked like:
   
   

   

2. Because I wanted /boot to be on its own partition I used the manual partition creation option:
   
   

   

   You don't have to do this but I wanted these instructions to align with my previous post for CentOS 5.5. In this case I made the first partition mount at /boot and the second partition was everything else mounted at / and the result looked like:
   

   

   Notice that I didn't include any swap, that is something you probably want to put on the EC2 instance's ephemeral storage.

3. For Ubuntu you are prompted to install other software after the base install and setting up a user. Make sure you install OpenSSH server:
   
   

   

4. When prompted to install grub answer yes.
5. At this point you should have a bootable Ubuntu system either from following the previous steps or from an existing bootable install. You will need to install the libuuid-perl package so that the EC2 compatible Xen kernels will be installable:

      sudo apt-get install libuuid-perl
   

6. Grab a Xen kernel and initrd that are compatible with EC2:

   wget http://ftp.debian.org/debian/pool/main/l/linux-2.6/linux-base_2.6.32-20_all.deb
   wget http://ftp.debian.org/debian/pool/main/l/linux-2.6/linux-image-2.6.32-5-xen-686_2.6.32-20_i386.deb
   
   sudo dpkg -i linux-base_2.6.32-20_all.deb linux-image-2.6.32-5-xen-686_2.6.32-20_i386.deb
   

   Notes: I was unable to find a compatible kernel that was apt-get installable from Ubuntu and that is why I grab the two above from the Debian site. Also, If you want to continue to boot the system outside of EC2 then make sure to edit the /boot/grub/grub.cfg file so that it has the correct kernel set to boot by default.

7. Create a menu.lst file that points to the correct kernel and initrd that were just installed:

      sudo vi /boot/grub/menu.lst
   

   For this example mine contains:

   default 0
   timeout 1
   title UBEC2
           root (hd0,0)
           kernel /boot/vmlinuz-2.6.32-5-xen-686 root=/dev/xvda2
           initrd /boot/initrd.img-2.6.32-5-xen-686
   

   Notes: The grub root command is pointing to hard disk 0 and partition 0 because I put the boot directory on the first partition. The root parameter for the kernel is pointing to /dev/xvda2 and not /dev/sda2 because the kernel patch to disable XSAVE changes the names of the devices.

8. Copy everything from /boot into /boot/boot, this is just the easiest way of making things work since EC2 PV-Grub looks for the menu.lst file /boot/boot/grub/ when you specify the hd00 kernel:

   sudo cp -Rp /boot/ /boot/boot
   

9. Shut the VirtualBox system down and extract the hard drive image with the following command:

   VBoxManage internalcommands converttoraw ~/.VirtualBox/HardDisks/YourHardDiskName.vdi /tmp/myosimage.img
   

   Notes: The raw image that results will be the full size of the disk as it was seen by VirtualBox so if you have a lot of free disk space on your image you will probably want to compress it before you transfer it to EC2. At this point the commands are exactly as they are for the generic instructions above.

10. Start a temporary EC2 instance and transfer your image:

    ec2-run-instances -z us-east-1a -g your-group -k your-keypair -n 1 ami-84db39ed
    

11. Create an EBS volume of the correct size to put your image onto and attach it to the previously created temporary EC2 instance:

    ec2-create-volume -z us-east-1a -s 2
    ec2-attach-volume volume-id -i instance-id -d /dev/sdh
    

12. Transfer the image to the volume:

    dd if=myosimage.img of=/dev/sdh bs=10M
    

13. Create a snapshot of the volume:

    ec2-create-snapshot -d "Volume Description" volume-id
    

14. Register the snapshot as an AMI:

    ec2-register -n "AMIName" -d "AMI Description" --root-device-name /dev/sda2 -b /dev/sda=snap-id:2:true
    

15. Boot the AMI:

    ec2-run-instances -z us-east-1a -g your-group -k your-keypair -n 1 --kernel pv-grub-kernel-id ami-from-step-19
    

    Note: Because the partition table contains /boot on hd00 I used the kernel named aki-4c7d9525 for the east region.

Once you get the hang of transferring the image these steps aren't as complicated as they seem. It may help to imagine the extracted disk image as a physical hard drive that you are moving around. The main pain points are selecting the correct PV-Grub kernel and making sure your grub menu.lst file is in the correct location on the partition.

For those who might just be interested in booting a custom kernel using EC2 pv-grub I will try to produce a few more posts with more details on that but for now here are the main things to know:

• The pv-grup kernels named with hd00 will look on the first partition of the registered device in the /boot/boot/grub directory for a menu.lst file. Use this type of kernel if you create want to use a partitioned disk.
• The pv-grup kernels named with hd0 will look on the registered device in the /boot/grub directory for a menu.lst file. Use this type of kernel if you don't have a partition on your disk.
• You won't get anything meaningful back from the boot attempt if your grub menu.lst file is in the wrong place or is not valid. See the end of the post for what a pv-grub error message looks like and some tips on what to do if you see it.
• The kernel you use does matter but the current mainline Linux kernel (2.6.35) contains everything you need except for a small change to turn off XSAVE. The main thing to know is that not every distribution may have made the change needed to work on EC2.
• I have tried non-Linux kernels to no avail. See the end of the post for a little more information.

A lot of what follows is similar, both steps and concepts, to the "from scratch" section of my post on Fedora 12 on EC2 using a root EBS. I've also bundled all the instance building commands up into one script (centos5.5.sh). If you want to use that script then do 1 and 2 of what follows, make sure to change the password used for root in the script and then pick back up at 18. The following steps should not be taken as the only way to do this but more of a recipe:

1. Start an EC2 instance that has yum on it to be used as a setup box. A RedHat based box, Fedora or CentOS will work best unless you want to install yum. For the following steps I used a Fedora 8 based EC2 node.

   ec2-run-instances -z us-east-1a -g your-group -k your-keypair -n 1 ami-84db39ed
   

2. Create a new EBS volume to install to and map it to the running instance from step 1. Your volume should be greater than 2G for a base install. I mapped this new volume to the /dev/sdh device on the setup machine so you will notice that in the following steps (if you are using the script you will want to make sure you map to /dev/sdh as well):

   ec2-create-volume -z us-east-1a -s 2
   ec2-attach-volume volume-id -i instance-id -d /dev/sdh
   

3. Create a partion table using fdisk on the volume you are going to install to.

   I created both a /boot and / partion on /dev/sdh1 and /dev/sdh2 respecivly. I also made the /dev/sdh1 partition active so it is exactly as it would be if it had been installed on a real machine.

   Note that this step is optional but I am going to include it because I think it makes for a more natural setup and is more in line with what you would get if you did a VirtualBox install and then transfered the image.

4. Format your partition(s) and mount them into /mnt. For me that was done with the following:

   echo "y" | mkfs.ext3 /dev/sdh1
   echo "y" | mkfs.ext3 /dev/sdh2
   mount /dev/sdh2 /mnt
   mkdir /mnt/boot
   mkdir /mnt/dev
   mkdir /mnt/proc
   mkdir /mnt/etc
   mount /dev/sdh1 /mnt/boot
   mount -t proc none /mnt/proc
   

5. Create a base device setup for the new instance:

   for i in console null zero ; do /sbin/MAKEDEV -d /mnt/dev -x $i ; done
   

6. Create a base fstab file in /mnt/etc/fstab. The following is the one I used:

   /dev/sda1               /boot                   ext3    defaults 1 1
   /dev/sda2               /                       ext3    defaults 1 2
   none                    /dev/pts                devpts  gid=5,mode=620 0 0
   none                    /dev/shm                tmpfs   defaults 0 0
   none                    /proc                   proc    defaults 0 0
   none                    /sys                    sysfs   defaults 0 0
   /dev/sdc1               /mnt                    ext3    defaults 0 0
   /dev/sdc2               swap                    swap    defaults 0 0
   

7. Create the yum repo configuration, prepare for the yum install and then install the base OS onto the new volume.
   
   

   The following is the yum configuration file I used:

   [main]
   cachedir=/var/cache/yum
   debuglevel=2
   logfile=/var/log/yum.log
   exclude=*-debuginfo
   gpgcheck=0
   obsoletes=1
   reposdir=/dev/null
   
   [os]
   name=CentOS 5.5 - i386 - OS
   mirrorlist=http://mirrorlist.centos.org/?release=5&arch=i386&repo=os
   enabled=1
   
   [updates]
   name=CentOS 5.5 - i386 - Updates
   mirrorlist=http://mirrorlist.centos.org/?release=5&arch=i386&repo=updates
   enabled=1
   

   The following command will install the base of Cent OS 5.5 into /mnt (note that I created the above config file as /tmp/yumec2.conf):

   yum -c /tmp/yumec2.conf --installroot=/mnt -y groupinstall Base
   

8. Install sshd, grub, the Cent OS Xen kernel and then clean the repo to free up disk space:

   yum -c /tmp/yumec2.conf --installroot=/mnt -y install openssh-server
   yum -c /tmp/yumec2.conf --installroot=/mnt -y install grub
   yum -c /tmp/yumec2.conf --installroot=/mnt -y install kernel-xen.i686
   
   yum -c /tmp/yumec2.conf --installroot=/mnt -y clean packages
   

9. Disable DNS checks and allow root to log in via SSH:

   echo "UseDNS no" >> /mnt/etc/ssh/sshd_config
   echo "PermitRootLogin yes" >> /mnt/etc/ssh/sshd_config
   

10. Set up networking by creating the /mnt/etc/sysconfig/network file. The contents for this example are:

    NETWORKING=yes
    

    As well as the /mnt/etc/sysconfig/network-scripts/ifcfg-eth0 file. The contents for this example are:

    DEVICE=eth0
    BOOTPROTO=dhcp
    ONBOOT=yes
    TYPE=Ethernet
    USERCTL=yes
    PEERDNS=yes
    IPV6INIT=no
    

11. I'm not sure if this is needed still but in the past there have been some /dev file missing on boot so I always add the following to the startup script to make sure they are available. The first two are the random devices and the last three are where the ephimeral drive is usually mapped:

    echo "/sbin/MAKEDEV /dev/urandom" >> /mnt/etc/rc.sysinit
    echo "/sbin/MAKEDEV /dev/random" >> /mnt/etc/rc.sysinit
    echo "/sbin/MAKEDEV /dev/sdc" >> /mnt/etc/rc.sysinit
    echo "/sbin/MAKEDEV /dev/sdc1" >> /mnt/etc/rc.sysinit
    echo "/sbin/MAKEDEV /dev/sdc2" >> /mnt/etc/rc.sysinit
    

12. Change the root password for the new instance. This is optional as you could create scripts to download your SSH key from EC2 but for these instructions setting the root password is the easiest:

    chroot /mnt
    pwconv
    passwd
    exit
    

13. Change the network settings so that the NetworkManager is off and network is on

    chroot /mnt chkconfig --level 2345 NetworkManager off
    chroot /mnt chkconfig --level 2345 network on
    

14. Disable a few things that are enabled by default but won't do any good for an EC2 instance:

    chroot /mnt chkconfig --level 2345 avahi-daemon off
    chroot /mnt chkconfig --level 2345 firstboot off
    

15. The stock CentOS Xen initrd doesn't load the Xen block or net drivers and those are required to boot. I unpacked the installed initrd and created a modified version by hand using the following commands (note that as soon as the CentOS Xen kernel version changes this will stop functioning):

    cp /mnt/boot/initrd-2.6.18-194.8.1.el5xen.img /mnt/boot/initrd-2.6.18-194.8.1.el5xen.img.orig
    mkdir /tmp/initrdextract
    cd /tmp/initrdextract
    gzip -dc /mnt/boot/initrd-2.6.18-194.8.1.el5xen.img | cpio -id
    cp /mnt/lib/modules/2.6.18-194.8.1.el5xen/kernel/drivers/xen/blkfront/xenblk.ko lib
    cp /mnt/lib/modules/2.6.18-194.8.1.el5xen/kernel/drivers/xen/netfront/xennet.ko lib
    chmod -x lib/xenblk.ko
    chmod -x lib/xennet.ko
    cat <<EOL > init
    #!/bin/nash
    
    mount -t proc /proc /proc
    setquiet
    echo Mounting proc filesystem
    echo Mounting sysfs filesystem
    mount -t sysfs /sys /sys
    echo Creating /dev
    mount -o mode=0755 -t tmpfs /dev /dev
    mkdir /dev/pts
    mount -t devpts -o gid=5,mode=620 /dev/pts /dev/pts
    mkdir /dev/shm
    mkdir /dev/mapper
    echo Creating initial device nodes
    mknod /dev/null c 1 3
    mknod /dev/zero c 1 5
    mknod /dev/urandom c 1 9
    mknod /dev/systty c 4 0
    mknod /dev/tty c 5 0
    mknod /dev/console c 5 1
    mknod /dev/ptmx c 5 2
    mknod /dev/rtc c 10 135
    mknod /dev/tty0 c 4 0
    mknod /dev/tty1 c 4 1
    mknod /dev/tty2 c 4 2
    mknod /dev/tty3 c 4 3
    mknod /dev/tty4 c 4 4
    mknod /dev/tty5 c 4 5
    mknod /dev/tty6 c 4 6
    mknod /dev/tty7 c 4 7
    mknod /dev/tty8 c 4 8
    mknod /dev/tty9 c 4 9
    mknod /dev/tty10 c 4 10
    mknod /dev/tty11 c 4 11
    mknod /dev/tty12 c 4 12
    mknod /dev/ttyS0 c 4 64
    mknod /dev/ttyS1 c 4 65
    mknod /dev/ttyS2 c 4 66
    mknod /dev/ttyS3 c 4 67
    echo Setting up hotplug.
    hotplug
    echo Creating block device nodes.
    mkblkdevs
    echo "Loading jbd.ko module"
    insmod /lib/jbd.ko
    echo "Loading ext3.ko module"
    insmod /lib/ext3.ko
    echo "Loading xenblk.ko module"
    insmod /lib/xenblk.ko
    echo "Loading xennet.ko module"
    insmod /lib/xennet.ko
    mkblkdevs
    echo Scanning and configuring dmraid supported devices
    resume /dev/sdc2
    echo Creating root device.
    mkrootdev -t ext3 -o defaults,ro /dev/sda1
    echo Mounting root filesystem.
    mount /sysroot
    echo Setting up other filesystems.
    setuproot
    echo Switching to new root and running init.
    switchroot
    EOL
    find ./ | cpio -H newc -o | gzip > /mnt/boot/initrd-2.6.18-194.8.1.el5xen.img
    cd -
    

16. Install grub on the new instance, move the boot directory into a subdirectory and create a grub menu.lst file that points to the CentOS kernel and initrd file:

    chroot /mnt grub-install /dev/sdh
    
    mkdir /mnt/boot/boot/
    mv /mnt/boot/* /mnt/boot/boot/ 2> /dev/null > /dev/null
    

    Put the following in /mnt/boot/boot/grub/menu.lst (note that as soon as the CentOS Xen kernel version changes this will be incorrect):

    default 0
    timeout 1
    title CentOS5.5
         root (hd0,0)
         kernel /boot/vmlinuz-2.6.18-194.8.1.el5xen root=/dev/sda2
         initrd /boot/initrd-2.6.18-194.8.1.el5xen.img
    

    Note that this goes in /mnt/boot/boot/grub and that isn't the normal spot you would expect it in. This is where the AWS EC2 pv-grub expects to find the file on the first partition and moving the boot directory around just keeps everything in line with those expectations.

17. Make sure everything is written to disk and unmount the volume. At this point you have a CentOS 5.5 install that is almost ready to boot.

    sync
    umount /mnt/proc
    umount /mnt/boot
    umount /mnt
    

18. Make a snapshot of the volume you just installed to, you will need to volume ID that came from step 2:

    ec2-create-snapshot -d "Volume Description" volume-id
    

19. Use the snapshot from step 18 along with the ec2-register command to register your instance:

    ec2-register -n "AMIName" -d "AMI Description" --root-device-name /dev/sda2 -b /dev/sda=snap-id:2:true
    

    There are a number of things to take note of with the above command:

    1. Running this command will result in output something like: IMAGE ami-a5ae9bb
    2. The -b option can now assign a snapshot to a block device, the options in this example tell EC2 to generate 2G of space for the snapshot and to delete the volume it creates from the snapshot if the instance terminates. If you plan to use an instance long term you should replace that true at the end with a false to keep EC2 from deleting the volume when the instance terminates.
    3. Notice that the -b option is assigning the snapshot to the device and not to a partition of the device, that is /dev/sda instead of /dev/sda1. You can still assign a snapshot directly to a partition but now you can also assign a block device to a raw partitioned disk. Because I created the partition table earlier the snapshot is the raw disk device here.
    4. Also note that we are missing the kernel configuration option. As of this post using it with a pv-grub kernel causes the register command to fail. It isn't a big issue but just keep that in mind when you fire the AMI up otherwise it won't boot with the correct pv-grub kernel.

    
    

20. Start an instance of the fresh CentOS 5.5 install. One key thing here is picking the correct pv-grub kernel to boot from. There are currently 4 different kernels at each location, see the Enabling User Provided Kernels in Amazon EC2 document for a full list of kernels in each availability zone. In this case because the root disk was created with a partition table I used the "ec2-public-images/pv-grub-hd00-V1.01-i386.gz.manifest.xml" kernel to boot with (on US-East-1 that is kernel id aki-4c7d9525). For example:

    ec2-run-instances -z us-east-1a -g your-group -k your-keypair -n 1 --kernel pv-grub-kernel-id ami-from-step-19
    

Tips on debugging the boot process

If your instance won't boot you can use the ec2-get-console-output command to get the console output created from the pv-grub boot process. If your console output ends up like the following there are a number of things you may have done wrong.

• You may have selected the wrong kernel and it is trying to boot from a non-existant partition. Make sure you are using the correct pv-grub kernel hd0 vs hd00.
• You forgot to install grub or installed grub in the wrong place. Make sure you have either /boot/grub/menu.lst or /boot/boot/grub/menu.lst
• You have a bad menu.lst file. One mistake I made was giving a boot item a title with a space in it. Make the menu.lst as simple as you can until you get it to boot.

    Xen Minimal OS!
  start_info: 0xb10000(VA)
    nr_pages: 0x6a400
  shared_inf: 0x002f9000(MA)
     pt_base: 0xb13000(VA)
nr_pt_frames: 0x9
    mfn_list: 0x967000(VA)
   mod_start: 0x0(VA)
     mod_len: 0
       flags: 0x0
    cmd_line:  root=/dev/sda1 ro 4
  stack:      0x946780-0x966780
MM: Init
      _text: 0x0(VA)
     _etext: 0x621f5(VA)
   _erodata: 0x76000(VA)
     _edata: 0x7b6d4(VA)
stack start: 0x946780(VA)
       _end: 0x966d34(VA)
  start_pfn: b1f
    max_pfn: 6a400
Mapping memory range 0xc00000 - 0x6a400000
setting 0x0-0x76000 readonly
skipped 0x1000
MM: Initialise page allocator for e6c000(e6c000)-0(6a400000)
MM: done
Demand map pfns at 6a401000-7a401000.
Heap resides at 7a402000-ba402000.
Initialising timer interface
Initialising console ... done.
gnttab_table mapped at 0x6a401000.
Initialising scheduler
Thread "Idle": pointer: 0x7a402008, stack: 0x6a030000
Initialising xenbus
Thread "xenstore": pointer: 0x7a402478, stack: 0x6a040000
Dummy main: start_info=0x966880
Thread "main": pointer: 0x7a4028e8, stack: 0x6a050000
"main" "root=/dev/sda1" "ro" "4"
vbd 2048 is hd0
******************* BLKFRONT for device/vbd/2048 **********

backend at /local/domain/0/backend/vbd/2111/2048
Failed to read /local/domain/0/backend/vbd/2111/2048/feature-barrier.
Failed to read /local/domain/0/backend/vbd/2111/2048/feature-flush-cache.
12582912 sectors of 0 bytes
**************************
vbd 2051 is hd1
******************* BLKFRONT for device/vbd/2051 **********

backend at /local/domain/0/backend/vbd/2111/2051
Failed to read /local/domain/0/backend/vbd/2111/2051/feature-barrier.
Failed to read /local/domain/0/backend/vbd/2111/2051/feature-flush-cache.
1835008 sectors of 0 bytes
**************************

    [H
    [J

    GNU GRUB  version 0.97  (1740800K lower / 0K upper memory)

       [ Minimal BASH-like line editing is supported.   For

         the   first   word,  TAB  lists  possible  command

         completions.  Anywhere else TAB lists the possible

         completions of a device/filename. ]

grubdom>
    [9;10H

Booting non-Linux OSes with EC2

I have attempted both FreeBSD and NetBSD in particular with no luck.

FreeBSD is tricky because it really wants to use its loader and while you can do that with the grub chainloader command it results in a grub error from EC2 about needing to load the kernel before booting:

root (hd0,1)

 Filesystem type unknown, partition type 0xa5

chainloader +1

Error 8: Kernel must be loaded before booting

Press any key to continue...

I was also able to try a modified version of FreeBSD that should boot without the loader but with that I get an error claiming the kernel isn't bziped:

root (hd0,1,a)

 Filesystem type is ufs2, partition type 0xa5

kernel /boot/loader

xc_dom_probe_bzimage_kernel: kernel is not a bzImage
ERROR Invalid kernel: xc_dom_find_loader: no loader found

xc_dom_core.c:523: panic: xc_dom_find_loader: no loader found
xc_dom_parse_image returned -1

Error 9: Unknown boot failure

Press any key to continue...

For NetBSD the result is actually a completely blank console log so I assume it causes some catastrophic failure that keeps the EC2 system from even being able to pull back a log.

The first thing to note is that they are only supporting 64 bit systems officially. Having said that it isn't too hard to modify the code to make it work on a 32 bit system although it may turn out that such early modifications are missing some fundamental bits on why they were only support 64 bit systems. I'm going to assume at first that you are using a 64 bit system and then end with what you need if you are still using a 32 bit system.

I don't actually have a 64 bit system myself so I used an EC2 instance for the following instructions. To do the same start with Amazon's Basic 64-bit Fedora Core 8 (AMI Id: ami-86db39ef) instance (note that this is EBS backed so you will end up with an EBS volume after you start it) and then upgrade to Fedora 12 using my previous instructions on building a EBS bootable Fedora 12 instance. You will need to remove a few packages to get the 64 bit version of Fedora 8 to upgrade that I didn't have to do for the 32 bit version, here are all the commands you need to get to a running 64 bit Fedora 12 instance (the entire upgrade takes about 20 minutes):

# Fedora 8 to Fedora 10
yum -y remove dmraid-1.0.0.rc14-4.fc8.i386 dmraid-1.0.0.rc14-4.fc8.i386 curl-7.18.2-7.fc8.i386
yum clean all
rpm -Uhv http://archive.kernel.org/fedora-archive/releases/10/Fedora/i386/os/Packages/fedora-release-10-1.noarch.rpm http://archive.kernel.org/fedora-archive/releases/10/Fedora/i386/os/Packages/fedora-release-notes-10.0.0-1.noarch.rpm
yum -y update

# Fedora 10 to Fedora 11
yum -y remove gpm-1.20.5-2.fc10.i386
yum clean all
rpm -Uvh http://mirrors.usc.edu/pub/linux/distributions/fedora/linux/releases/11/Fedora/i386/os/Packages/fedora-release-11-1.noarch.rpm http://mirrors.usc.edu/pub/linux/distributions/fedora/linux/releases/11/Fedora/i386/os/Packages/fedora-release-notes-11.0.0-2.fc11.noarch.rpm
yum -y update

# Fedora 11 to Fedora 12
yum -y remove cryptsetup-luks-1.0.6-7.fc11.i586
yum clean all
rpm -Uvh http://mirrors.kernel.org/fedora/releases/12/Fedora/i386/os/Packages/fedora-release-notes-12.0.0-4.fc12.noarch.rpm http://mirrors.kernel.org/fedora/releases/12/Fedora/i386/os/Packages/fedora-release-12-1.noarch.rpm
yum -y update

# Make sure the basics are installed
yum -y install gcc-c++ git

To start with there are some prerequisites you need. This can be taken care of in one command with yum:

yum -y install git cmake boost pcre-devel libicu-devel libmcrypt-devel oniguruma-devel mysql-devel gd-devel boost-devel libxml2-devel libcap-devel binutils-devel flex bison expat-devel

Next create a directory to hold everything in, change into that directory and create another directory to hold the customized libraries needed to compile HipHop PHP:

mkdir hiphop
cd hiphop
mkdir local

Next it is time to pull down the HipHop PHP source along with the source for some libraries it depends on (these all go into the hiphop directory created above):

git clone git://github.com/facebook/hiphop-php.git

wget "http://downloads.sourceforge.net/project/re2c/re2c/0.13.5/re2c-0.13.5.tar.gz?use_mirror=cdnetworks-us-2"
wget "http://www.threadingbuildingblocks.org/uploads/77/142/2.2/tbb22_20090809oss_src.tgz"
wget http://curl.haxx.se/download/curl-7.20.0.tar.bz2
wget http://www.monkey.org/~provos/libevent-1.4.13-stable.tar.gz

tar xvjf curl-7.20.0.tar.bz2
tar xvzf libevent-1.4.13-stable.tar.gz
tar xvzf re2c-0.13.5.tar.gz
tar xvzf tbb22_20090809oss_src.tgz

Next the customized patches get applied to some of the library sources and each is built to install in the custom directory:

export CMAKE_PREFIX_PATH=`pwd`/local

cd tbb22_20090809oss
gmake
cp -Rp include/tbb/ /usr/include/
cp `pwd`/build/*_release/*.so /usr/lib/
ldconfig
cd ..

cd re2c-0.13.5
./configure --prefix=`pwd`/../local
make install
cd ..

cd libevent-1.4.13-stable
cp ../hiphop-php/src/third_party/libevent.fb-changes.diff .
patch < libevent.fb-changes.diff
./configure --prefix=`pwd`/../local
make install
cd ..

cd curl-7.20.0
cp ../hiphop-php/src/third_party/libcurl.fb-changes.diff .
patch -p0 < libcurl.fb-changes.diff
./configure --prefix=`pwd`/../local
make install
cd ..

There is one problem at this point that requires a little surgery on the HipHop PHP source itself. There is more about this in issue #6 and once it gets fixed this won't need to be done.

cd hiphop-php
echo "#ifndef LHASH" >> src/cpp/ext/ext_openssl.h
echo "#define LHASH LHASH_OF(CONF_VALUE)" >> src/cpp/ext/ext_openssl.h
echo "#endif" >> src/cpp/ext/ext_openssl.h

And at last it is time to compile HipHop PHP itself:

git submodule init
git submodule update
export HPHP_HOME=`pwd`
export HPHP_LIB=`pwd`/bin
cmake .
make

It takes about 20 minutes to compile everything. Once the compile is done you are ready to roll. Check out the running HipHop wiki page to learn how to run the resulting binary. One important thing to note is that you need to make sure you have the correct environment variables set when you go to compile things. I created a little file I can source with the following in it:

export HPHP_BASE=<path to the first directory>
export CMAKE_PREFIX_PATH=$HPHP_BASE/local
export HPHP_HOME=$HPHP_BASE/hiphop-php
export HPHP_LIB=$HPHP_HOME/bin

For those who just want it to go I've put all of the above into one script that can be found here. If you are going from Fedora 8 to Fedora 12 on an EC2 node you can get a script for that here.

Now if you want to do this on a 32 bit Fedora 12 install you will need to modify the source first. The easiest way I know of doing this is to look at this commit log or clone my version that can be found here:

git clone git://github.com/carsonmcdonald/hiphop-php.git

Please note that my version my not be up to date and the modifications to get the source to build on the 32 bit system may not be 100% correct. My goal was to get it to build and run on a 32 bit system but I don't have the time to very much more than that.

I'm going to assume you already understand how to do things like create instances, create EBS volumes and ssh into your running instance using key based authentication. I use the AWS management console for a lot of what follows with the exception of needing to register the AMI and for that you will need the Amazon EC2 API Tools and Amazon EC2 AMI Tools

There are two ways to get to a bootable EBS backed Fedora 12 instance and they start off the same. The first thing to do is fire up the AMI named "Basic Fedora Core 8 (AMI Id: ami-84db39ed)" that is provided by Amazon.

Once the Fedora Core 8 EC2 instance is ready ssh into it. Fedora 12 requries a newer version of RPM to install so you now need to upgrade the instance to Fedora 10. This is pretty easy and can be done by following my instructions on upgrading from Fedora 9 to Fedora 10 (don't worry about skipping 9 it will work). Here are the commands needed to do the upgrade:

yum clean all
rpm -Uhv http://archive.kernel.org/fedora-archive/releases/10/Fedora/i386/os/Packages/fedora-release-10-1.noarch.rpm http://archive.kernel.org/fedora-archive/releases/10/Fedora/i386/os/Packages/fedora-release-notes-10.0.0-1.noarch.rpm
yum -y update

After a few minutes the instance will be upgraded and ready for the next step. This is where the two paths diverge depending on how you want the final product constructed. The options are to install Fedora 12 on a freshly minted volume or continue upgrading the instance you just created.

Upgrade path

I will start with the upgrade path since that is probably the easier of the two although may leave you with a messier instances after it is done. The next step for the upgrade path is to do what I outline in upgrading from Fedora 10 to Fedora 11 and upgrading from Fedora 11 to Fedora 12. Here are the commands all in one place to make it easy:

yum clean all
rpm -Uvh http://mirrors.usc.edu/pub/linux/distributions/fedora/linux/releases/11/Fedora/i386/os/Packages/fedora-release-11-1.noarch.rpm http://mirrors.usc.edu/pub/linux/distributions/fedora/linux/releases/11/Fedora/i386/os/Packages/fedora-release-notes-11.0.0-2.fc11.noarch.rpm
yum -y update
yum clean all
rpm -Uvh http://mirrors.kernel.org/fedora/releases/12/Fedora/i386/os/Packages/fedora-release-notes-12.0.0-4.fc12.noarch.rpm http://mirrors.kernel.org/fedora/releases/12/Fedora/i386/os/Packages/fedora-release-12-1.noarch.rpm
yum -y update

Once you have everything upgraded to Fedora 12 you will have a 15G root partition that has less than 2G used. This may not suite your needs very well if you really don't need that extra 13G but thankfully if you want to shrink the root EBS partition you can.

I found some instructions in this article on EBS backed AMIs that describes using the following command to copy the entire file system over. Assuming you have created a smaller volume and attached it to the instance as sdh you should be able to do something like the following to copy everything to the new volume:

mkfs.ext3 /dev/sdh
mount /dev/sdh /mnt
tar cpS / | cpipe -vt -b 1024 | gzip -c | tar zxpS -C /mnt
rm -rf /mnt/mnt/*
rm -rf /mnt/proc/*
umount /mnt

One thing to note in the above is that the entire sdh drive is formatted for the file system (you will actually get a prompt asking if that is ok). As far as I can tell this is the way it has to be or the instance will not boot correctly. I assume this is because the root device is hidden behind a partition already as /dev/sda1 and so shouldn't have a second partition table.

Skip to the common part now to learn how to make the final bootable AMI.

From scratch path

This path is similar to and mostly an update/extension to my post on creating a Fedora 7 AMI setup. I'm going to leave out most of the details and just provide you with a script that will take an empty volume (assumed to be attached as /dev/sdh) and turn it into a bootable EBS backed Fedora 12 volume. Download the script createfedora12bootebs.sh instead of trying to cut and paste the following, it gets formatted in such a way as to lose a newline that is important. Please note that you will need at least 1G of space on the given volume.

#!/bin/sh

echo "y" | mkfs.ext3 /dev/sdh
mount /dev/sdh /mnt

mkdir /mnt/dev
mkdir /mnt/proc
mkdir /mnt/etc

for i in console null zero ; do /sbin/MAKEDEV -d /mnt/dev -x $i ; done

cat <<EOL > /mnt/etc/fstab
/dev/sda1               /                       ext3    defaults 1 1
none                    /dev/pts                devpts  gid=5,mode=620 0 0
none                    /dev/shm                tmpfs   defaults 0 0
none                    /proc                   proc    defaults 0 0
none                    /sys                    sysfs   defaults 0 0
/dev/sdc1               /mnt                    ext3    defaults 0 0
/dev/sdc2               swap                    swap    defaults 0 0
EOL

mount -t proc none /mnt/proc

cat <<EOL > /tmp/yumec2.conf
[main]
cachedir=/var/cache/yum
debuglevel=2
logfile=/var/log/yum.log
exclude=*-debuginfo
gpgcheck=0
obsoletes=1
reposdir=/dev/null

[base]
name=Fedora 12 – i386 – Base
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=fedora-12&arch=i386
enabled=1

[updates-released]
name=Fedora 12 – i386 – Released Updates
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=updates-released-f12&arch=i386
enabled=1
EOL

yum -c /tmp/yumec2.conf --installroot=/mnt -y groupinstall Base
yum -c /tmp/yumec2.conf --installroot=/mnt -y install openssh-server

yum -c /tmp/yumec2.conf --installroot=/mnt -y clean packages

echo "UseDNS no" >> /mnt/etc/ssh/sshd_config
echo "PermitRootLogin without-password" >> /mnt/etc/ssh/sshd_config

cp /etc/rc.local /mnt/etc/
cp /etc/sysconfig/network /mnt/etc/sysconfig/network
cp /etc/sysconfig/network-scripts/ifcfg-eth0 /mnt/etc/sysconfig/network-scripts/ifcfg-eth0
cp /usr/local/sbin/* /mnt/usr/local/sbin/
cp -Rp /lib/modules/2.6.21.7-2.fc8xen/ /mnt/lib/modules/

echo "/sbin/MAKEDEV /dev/urandom" >> /mnt/etc/rc.sysinit
echo "/sbin/MAKEDEV /dev/random" >> /mnt/etc/rc.sysinit
echo "/sbin/MAKEDEV /dev/sdc" >> /mnt/etc/rc.sysinit
echo "/sbin/MAKEDEV /dev/sdc1" >> /mnt/etc/rc.sysinit
echo "/sbin/MAKEDEV /dev/sdc2" >> /mnt/etc/rc.sysinit

cat <<EOF >> /mnt/etc/rc.sysinit
# The following will partition the local drive and set up swap
cat <<EOL | fdisk /dev/sdc
n
p
1
1
+140G
n
p
2

w
EOL
mkswap /dev/sdc2
EOF

mv /mnt/lib/tls /mnt/lib/tls.disabled
echo "hwcap 0 nosegneg" >> /mnt/etc/ld.so.conf.d/kernelcap-2.6.21.7-2.fc8.conf

chroot /mnt chkconfig --level 2345 NetworkManager off
chroot /mnt chkconfig --level 2345 network on

sync
umount /mnt/proc
umount /mnt

Common wrap up

At this point you will need to create a snapshot of the volume that was created for one of the paths above. Once the snapshot is available you will need to then register the snapshot as an AMI that is bootable from EBS. To do that you would issue something like the following command substituting the correct data in where it relates to your volume and snapshot.

ec2-register -n "AMIName" -d "AMI Description" --block-device-mapping /dev/sdc=ephemeral0 --snapshot your-snapname --architecture i386 --kernel aki-a71cf9ce --ramdisk ari-a51cf9cc

One thing to note in this command is the –block-device-mapping option. That option is what gives you access to the local drive on your node once it is booted. This gives you extra storage for things you don't need to keep after the life of the running node. In the from scratch option I'm turning part of the local drive into swap as well as creating a partition that could be used as a large temporary storage. If you want to know more details on the ephermeral storage look at this post

After all that you should have a bootable EBS backed Fedora 12 install to work with.

When I first started down the path of using EC2 I thought I would grab the source each time I wanted to build. I quickly ran into a snag however because it took forever to sync the source and download the Ubuntu repo. Once I had the initial sync of the source I decided I would copy it all to an EBS volume and keep that volume up to date. Using EBS to store the source feels better too since I assume Google expects people to be syncing changes only as opposed to pulling the entire source tree down every time they want to build.

I started out by finding this Ubuntu AMI for a base to work from. For the most efficient compile times I ended up using the High CPU (c1.medium) instance. I started with the default small instance but it was just too slow. With the high cpu instance you are looking at about 45 minutes to build the OS after you have the source synced for the first time and if you add building Chromium in there you are looking at around 55 additional minutes. All told you can have a complete build in less than 2 hours even if there are some source updates needed. For EBS you need a 3G volume for the Chrome OS source plus Ubuntu package repo and a 4G volume for the Chromium source.

I've bundled everything up into one script called builder.sh and if you don't care to know the details you can download it and give it a try. There are some things you need to know about it however even if you don't want to follow all the details. First there are some of the assumptions made by the script:

1. The EC2 AMI ami-ccf615a5 image is used on the node it is run on.
2. If you are storing the source then you have attached the EBS volumes and you have initialized a filesystem on both with mkfs.ext3
3. If you are storing the source then you have attached the EBS volumes and indicated what devices they are using OSVOLDEV and BROWSERVOLDEV in the following config section
4. You are running the script as the root user

Next is a small configuration area at the top of the script that lets it know what devices the two volumes will be mounted on. If you don't intend on having stored source then you don't need to worry about setting them to anything as long as what they are set to doesn't match a real device.

There are a few command line options that will let you tune the script if you want to skip parts of the build process:

• --skip-sync – This will skip trying to sync any of the repos. If you aren't using EBS to store the source the script will fail if you use this option since there will be nothing to build.
• --skip-chrome-build – This will skip trying to build chrome. You have to build chrome at least once for the chrome os build to work.
• --skip-chrome-os-build – This will skip trying to build chrome os.

Here are a few other important things to make note of:

• The build script adds a user named "gogo" that can be used to log in if you don't want to use your Google login.
• The build script sets the password for root access to "gogo".
• Running the build script should result in an image named /mnt/builder/.usb.img.bz when everything is done. This is a compressed bootable image that can be written to a USB key (use something like dd if=usb.img of=/dev/usbkeydevice bs=10M).
• I could imagine the same instructions and script being re-purposed for building on a non-EC2 system too.
• The script can be run multiple times on the same node and will skip the parts that should only be done once.
• There are probably ways to speed up the build to make it even faster. With some more effort the build for the browser could be kicked off while the source for the OS was still be synced. I also wondered if using a 64 bit EC2 node and creating a large ramdisk would help but I didn't want to fiddle with 64 bit builds of Chrome, I may revisit if I find time.
• I was tempted to build an AMI that included all the prerequisites but it only takes 5 minutes to pull all of those together so I decided not to. It could streamline things to do that but I'm not sure that it is worth the effort.

If you are looking for more information about the script, places where it might make sense to modify the script and the build process in general then read on.

To get started there are a number of prerequisites that need to be installed. These are the required parts for both Chromium and Chromium OS. One note here is that there is some issue with lighttpd on Ubuntu that causes it to die while the apt-get is downloading all the packages so I had to resort to installing Apache as well. I make sure lighttpd isn't running since the prerequisites install it.

The next step is to create a user that will actually run the build. It turns out to be important that this user is not root so that is why there are some hoops being jumped here. The two source volumes get mounted into the user's home directory here as well. The last part of this section builds a filesystem to be used as temporary storage for output images. The way they build is set up it will push the final images into the source tree and because that is stored on EBS it could end up eating up a lot of room, there is more to this later in the Chrome OS build section.

The next sections each sync either source or the apt-get repository. The first is syncing the apt-get repository using apt-mirror. In case you missed it the mirror was defined in the prerequisites section above.

The next is the Chromium OS source repository.

There are a few extra parts to syncing the Chromium browser. One of those extras is to make sure the third party test suites don't get synced since they are large.

Building the Chromium browser. This isn't needed every time. When it is done it sticks the browser in place for the Chromium OS build.

Finally the place where the OS actually gets compiled. The majority of this is right out of the build instructions. One thing to understand here is that the build process creates a chroot environment and then maps the source into that before the build starts. The build is done inside the chroot environment so the build script has to create a temporary script with all the commands that need to run there. One other note here is that the images filesystem created above has to be mapped into the chroot environment and I did that by setting it up as a loopback device then mounting it from inside the chroot environment. After the build is complete I can then get to the filesystem outside of the chroot environment to grab the final image. If you want to change the test user, the system password or get rid of either of those options this is the section to change.

A final bit of glue looks for command line arguments ties everything together.

Reading over the comments on the Voxilla post you will see some concern about how cost effective putting Asterisk on EC2 would be. Even if the cost is an issue for normal use I think Asterisk on EC2 could work for bursts of outgoing calls or even temporary conferencing systems. Part of what I wanted to do was find the least resistant path to getting started so I went with Trixbox since it has a lot of tools pre-installed and support for Gizmo5 that was very easy to set up. The key with Gizmo5 is that it is cheap, works with Asterisk via SIP and you can have incoming calls for free from a land line so it is easy to test cheaply.

To start with I'll point out that the Voxilla guys have done what I'm about to go over here in a different way. A portion of what I do matches up with the Voxilla post but I'm using the AWS console. There is now even a Voxilla public AMI available for people who don't want to do anything but fire up an Asterisk node.

The steps for this install break down as follows:

Install Trixbox under VirtualBox

Start by installing Trixbox CE under VirtualBox. I'm currently using VirtualBox 2.1.4 but older versions will probably work equally as well. Grab the Trixbox CE ISO and create a 2G partition to do the install. Here is the VirtualBox configuration I used:

(Click the image to see a larger version)

The install only takes a few minutes:

(Click the image to see a larger version)

Then when you are done you should let it boot:

(Click the image to see a larger version)

After doing the Trixbox install you should follow instructions to extract the image from the VDI and create as an AMI. You should end with a bootable AMI that will start TrixBox once you are finished.

Create a security group to allow Asterisk traffic

The following closely matches what the Voxilla article has except for the use of AWS Management Console everywhere. I'm not really concerned with having a static IP assigned to the node so I've skipped that part. Depending on how you want to use the system you may not need one at all.

Start by going into the "security groups" option off the main console page:

(Click the image to see a larger version)

Create a new group that will contain the security rules. I've named mine "Trixbox":

There are 5 rules that are needed. They are ssh, http, udp ports 10000 to 20000, tcp 5060 to 5061, and udp 5060 to 5060. Here is what my security group looks like:

(Click the image to see a larger version)

Note that in the above I'm opening everything up to the world for each entry but you probably want to restrict things more based on where the traffic will be coming from. At the very least if you open the http port to the world make sure to change all the default passwords.

Now you are ready to launch the instance. Find the "Launch Instances" button on the main console page:

Find the AMI that was created from the TrixBox install:

(Click the image to see a larger version)

Configure the instance for launch:

(Click the image to see a larger version)

Notice that the advanced area is open and there is a specialized kernel (aki-9b00e5f2) that was selected. This is from the Voxilla article and the kernel has a higher frequency clock that makes audio lag less.

Once you have hit launch and the instance has started you should be able to ssh into it and verify that Asterisk is running if you wish. You can actually do all the configuration through the web interface so you don't have to log in if you have faith that everything started as it should.

Configure Asterisk with the web console

You should now be able to put the running instance name into your browser and bring up the web user console.

(Click the image to see a larger version)

There are a few initial changes that need to be made to get the box running smoothly so switch to maintenance mode by clicking the "maint" link on the top right. You will need to use the default username and password (see the TrixBox documentation):

(Click the image to see a larger version)

Because the external IP is nated you will need to edit one of the configuration files to contain the external IP. Grab the external IP of your instance by pinging the hostname. From the main maintenance screen pick PBX then "Config file editor". Find the filename in the list of configuration files named "sip_general_custom.conf" and click it.

Put the following information in the entry box, remember to use the instance IP:

(Click the image to see a larger version)

If you want to find out more about SIP and NAT then check out Asterisk SIP and NAT as well as Asterisk SIP externip.

Now you should have the NAT configuration working. One thing to take note of is that any change you make will require a reload of the Asterisk system. Don't worry though you still need to add extensions before anything is useful and you can reload after that.

Follow the FreePBX adding extensions guide to set up a few extensions.

I also went ahead and made one of the extensions the default inbound for testing:

At this point you should be able to use a VOIP phone to connect to your node and access voicemail or call from one extension to another.

Use the web console to configure Gizmo5 support

At this point you could just use the system for VOIP calls between extensions but that wouldn't be much fun. You really need a VOIP gateway to be able to call out and get calls in. This is where Gizmo5 comes in. You will need a Gizmo5 account before you proceed.

The setup with Trixbox is just a few clicks. Go to PBX, Gizmo5 and then enter your Gizmo5 login information:

(Click the image to see a larger version)

After this you probably want to set up the outbound route:

(Click the image to see a larger version)

You can test the Gizmo5 integration by calling one of their free access numbers. You will first need to log in and get your Gizmo5 SIP number:

(Click the image to see a larger version)

I'm using VirtualBox 2.1.0 so some of the following commands may not work with older versions. I learned the hard way that they have changed a number of tools for VirtualBox and some of the older tools where probably easier to use and documented better. I installed the package I was using from an ISO image and then started trying to extract the part that I needed from the VDI that was created.

My first attempt at extracting the partition required me to convert my dynamic VDI into a static image. To dump a dynamic VDI into a static image you run this command:

I thought I could find the image by hand in the VDI after I had it in a raw format. There were a number of hints that I found that made me think I could just pull the partition out without much of a problem: VirtualBox and forensics tools and a forum post. However I found that just looking around wasn't easy enough to find where the partition started so I moved on to trying to find something else that could scan the disk and find it.

I rand into TestDisk and gave it a try. When it would scan the disk it found the /boot partition but for some reason it wasn't finding the root partition so I moved on.

I then took a look at the format for VDI disks to see if it was possible to pull it out given the header information with a simple program but that looked like it would be a lot of work so it was back to square one.

Along the way I happened to came across information about an undocumented command to export raw disk image. This turned out to be the break I needed because running the following command will result in only the disk image itself without any VirtualBox residue:

At this point things became a lot easier. There were multiple partitions on the resulting disk image but I only needed the / partition. To extract the root partition I first listed the partitions with this command:

This output the following for my image:

To figure out where the root partition starts I just multiplied the start sector by the number of bytes per sector: 208845 * 512 = 106928640

I then did a quick test to make sure I had the correct partition:

This looked good so I extracted the partition from the disk and did a filesystem check on it:

Extracting the partition you want is about 80% of the battle. Getting it to run under Xen after extraction is just a matter of fixing anything that was left out because the install was done under a "real" machine.

I add a nosegneg ld.so.conf directive and move /lib/tls directory out of the way first:

Next the base device entries needed to be created:

I then removed the disk label from the partition using e2label:

Because I was sending this image to EC2 I recreated the fstab with the following entries that are specific to the way EC2 allocates disks to a node:

The finally, again because I was going to EC2 I added a few scripts and created rc.local to let me in when the instance was started.

This seems to be a fairly easy process now that I have done it from start to finish once.

I'm going to assume you have already created your EBS volume, if you haven't you can learn more about that from the docs. You will need to make sure ruby is installed on the AMI you are going to use and that the RightScale AWS gem is installed as well.

The following script grabs the instance id from the EC2 metadata URL. It then uses the RightScale EC2 calls to attach the volume to the current EC2 instance. After the attach call it may take a few seconds for the volume to become ready so the script sleeps for a few seconds before calling out to the system to mount the device. One enhancement that is obvious here would be to have the script make a RightScale EC2 call to determine when the volume is really ready and then continue after that.

I called the script mountlogs.rb and call it out of the local startup script /etc/rc.local so it mounts the disk on startup. This seems to work pretty well as a stopgap until Amazon puts in place a way to auto-attach EBS volumes to instance creation.