Installing Cent OS 5.5 on EC2 with the Cent OS 5.5 Kernel

Amazon recently introduced the ability to boot a custom kernel using pv-grub on EC2. This opens the door for all kinds of interesting ideas that I've been thinking about for a while, like seeing if I can boot right into a web server and skip all that extra junk that comes with Linux distributions, but that is just me. The main door it is going to open for most people interested in EC2 will be the ability to upgrade the kernel that comes with their distribution. That brings us to how to install Cent OS 5.5 on EC2 and use the kernel that is part of the distribution.

For those who might just be interested in booting a custom kernel using EC2 pv-grub I will try to produce a few more posts with more details on that but for now here are the main things to know:

  • The pv-grup kernels named with hd00 will look on the first partition of the registered device in the /boot/boot/grub directory for a menu.lst file. Use this type of kernel if you create want to use a partitioned disk.
  • The pv-grup kernels named with hd0 will look on the registered device in the /boot/grub directory for a menu.lst file. Use this type of kernel if you don't have a partition on your disk.
  • You won't get anything meaningful back from the boot attempt if your grub menu.lst file is in the wrong place or is not valid. See the end of the post for what a pv-grub error message looks like and some tips on what to do if you see it.
  • The kernel you use does matter but the current mainline Linux kernel (2.6.35) contains everything you need except for a small change to turn off XSAVE. The main thing to know is that not every distribution may have made the change needed to work on EC2.
  • I have tried non-Linux kernels to no avail. See the end of the post for a little more information.

A lot of what follows is similar, both steps and concepts, to the "from scratch" section of my post on Fedora 12 on EC2 using a root EBS. I've also bundled all the instance building commands up into one script (centos5.5.sh). If you want to use that script then do 1 and 2 of what follows, make sure to change the password used for root in the script and then pick back up at 18. The following steps should not be taken as the only way to do this but more of a recipe:

  1. Start an EC2 instance that has yum on it to be used as a setup box. A RedHat based box, Fedora or CentOS will work best unless you want to install yum. For the following steps I used a Fedora 8 based EC2 node.

    ec2-run-instances -z us-east-1a -g your-group -k your-keypair -n 1 ami-84db39ed
    
  2. Create a new EBS volume to install to and map it to the running instance from step 1. Your volume should be greater than 2G for a base install. I mapped this new volume to the /dev/sdh device on the setup machine so you will notice that in the following steps (if you are using the script you will want to make sure you map to /dev/sdh as well):

    ec2-create-volume -z us-east-1a -s 2
    ec2-attach-volume volume-id -i instance-id -d /dev/sdh
    
  3. Create a partion table using fdisk on the volume you are going to install to.

    I created both a /boot and / partion on /dev/sdh1 and /dev/sdh2 respecivly. I also made the /dev/sdh1 partition active so it is exactly as it would be if it had been installed on a real machine.

    Note that this step is optional but I am going to include it because I think it makes for a more natural setup and is more in line with what you would get if you did a VirtualBox install and then transfered the image.

  4. Format your partition(s) and mount them into /mnt. For me that was done with the following:

    echo "y" | mkfs.ext3 /dev/sdh1
    echo "y" | mkfs.ext3 /dev/sdh2
    mount /dev/sdh2 /mnt
    mkdir /mnt/boot
    mkdir /mnt/dev
    mkdir /mnt/proc
    mkdir /mnt/etc
    mount /dev/sdh1 /mnt/boot
    mount -t proc none /mnt/proc
    
  5. Create a base device setup for the new instance:

    for i in console null zero ; do /sbin/MAKEDEV -d /mnt/dev -x $i ; done
    
  6. Create a base fstab file in /mnt/etc/fstab. The following is the one I used:

    /dev/sda1               /boot                   ext3    defaults 1 1
    /dev/sda2               /                       ext3    defaults 1 2
    none                    /dev/pts                devpts  gid=5,mode=620 0 0
    none                    /dev/shm                tmpfs   defaults 0 0
    none                    /proc                   proc    defaults 0 0
    none                    /sys                    sysfs   defaults 0 0
    /dev/sdc1               /mnt                    ext3    defaults 0 0
    /dev/sdc2               swap                    swap    defaults 0 0
    
  7. Create the yum repo configuration, prepare for the yum install and then install the base OS onto the new volume.

    The following is the yum configuration file I used:

    [main]
    cachedir=/var/cache/yum
    debuglevel=2
    logfile=/var/log/yum.log
    exclude=*-debuginfo
    gpgcheck=0
    obsoletes=1
    reposdir=/dev/null
    
    [os]
    name=CentOS 5.5 - i386 - OS
    mirrorlist=http://mirrorlist.centos.org/?release=5&arch=i386&repo=os
    enabled=1
    
    [updates]
    name=CentOS 5.5 - i386 - Updates
    mirrorlist=http://mirrorlist.centos.org/?release=5&arch=i386&repo=updates
    enabled=1
    

    The following command will install the base of Cent OS 5.5 into /mnt (note that I created the above config file as /tmp/yumec2.conf):

    yum -c /tmp/yumec2.conf --installroot=/mnt -y groupinstall Base
    
  8. Install sshd, grub, the Cent OS Xen kernel and then clean the repo to free up disk space:

    yum -c /tmp/yumec2.conf --installroot=/mnt -y install openssh-server
    yum -c /tmp/yumec2.conf --installroot=/mnt -y install grub
    yum -c /tmp/yumec2.conf --installroot=/mnt -y install kernel-xen.i686
    
    yum -c /tmp/yumec2.conf --installroot=/mnt -y clean packages
    
  9. Disable DNS checks and allow root to log in via SSH:

    echo "UseDNS no" >> /mnt/etc/ssh/sshd_config
    echo "PermitRootLogin yes" >> /mnt/etc/ssh/sshd_config
    
  10. Set up networking by creating the /mnt/etc/sysconfig/network file. The contents for this example are:

    NETWORKING=yes
    

    As well as the /mnt/etc/sysconfig/network-scripts/ifcfg-eth0 file. The contents for this example are:

    DEVICE=eth0
    BOOTPROTO=dhcp
    ONBOOT=yes
    TYPE=Ethernet
    USERCTL=yes
    PEERDNS=yes
    IPV6INIT=no
    
  11. I'm not sure if this is needed still but in the past there have been some /dev file missing on boot so I always add the following to the startup script to make sure they are available. The first two are the random devices and the last three are where the ephimeral drive is usually mapped:

    echo "/sbin/MAKEDEV /dev/urandom" >> /mnt/etc/rc.sysinit
    echo "/sbin/MAKEDEV /dev/random" >> /mnt/etc/rc.sysinit
    echo "/sbin/MAKEDEV /dev/sdc" >> /mnt/etc/rc.sysinit
    echo "/sbin/MAKEDEV /dev/sdc1" >> /mnt/etc/rc.sysinit
    echo "/sbin/MAKEDEV /dev/sdc2" >> /mnt/etc/rc.sysinit
    
  12. Change the root password for the new instance. This is optional as you could create scripts to download your SSH key from EC2 but for these instructions setting the root password is the easiest:

    chroot /mnt
    pwconv
    passwd
    exit
    
  13. Change the network settings so that the NetworkManager is off and network is on

    chroot /mnt chkconfig --level 2345 NetworkManager off
    chroot /mnt chkconfig --level 2345 network on
    
  14. Disable a few things that are enabled by default but won't do any good for an EC2 instance:

    chroot /mnt chkconfig --level 2345 avahi-daemon off
    chroot /mnt chkconfig --level 2345 firstboot off
    
  15. The stock CentOS Xen initrd doesn't load the Xen block or net drivers and those are required to boot. I unpacked the installed initrd and created a modified version by hand using the following commands (note that as soon as the CentOS Xen kernel version changes this will stop functioning):

    cp /mnt/boot/initrd-2.6.18-194.8.1.el5xen.img /mnt/boot/initrd-2.6.18-194.8.1.el5xen.img.orig
    mkdir /tmp/initrdextract
    cd /tmp/initrdextract
    gzip -dc /mnt/boot/initrd-2.6.18-194.8.1.el5xen.img | cpio -id
    cp /mnt/lib/modules/2.6.18-194.8.1.el5xen/kernel/drivers/xen/blkfront/xenblk.ko lib
    cp /mnt/lib/modules/2.6.18-194.8.1.el5xen/kernel/drivers/xen/netfront/xennet.ko lib
    chmod -x lib/xenblk.ko
    chmod -x lib/xennet.ko
    cat <<EOL > init
    #!/bin/nash
    
    mount -t proc /proc /proc
    setquiet
    echo Mounting proc filesystem
    echo Mounting sysfs filesystem
    mount -t sysfs /sys /sys
    echo Creating /dev
    mount -o mode=0755 -t tmpfs /dev /dev
    mkdir /dev/pts
    mount -t devpts -o gid=5,mode=620 /dev/pts /dev/pts
    mkdir /dev/shm
    mkdir /dev/mapper
    echo Creating initial device nodes
    mknod /dev/null c 1 3
    mknod /dev/zero c 1 5
    mknod /dev/urandom c 1 9
    mknod /dev/systty c 4 0
    mknod /dev/tty c 5 0
    mknod /dev/console c 5 1
    mknod /dev/ptmx c 5 2
    mknod /dev/rtc c 10 135
    mknod /dev/tty0 c 4 0
    mknod /dev/tty1 c 4 1
    mknod /dev/tty2 c 4 2
    mknod /dev/tty3 c 4 3
    mknod /dev/tty4 c 4 4
    mknod /dev/tty5 c 4 5
    mknod /dev/tty6 c 4 6
    mknod /dev/tty7 c 4 7
    mknod /dev/tty8 c 4 8
    mknod /dev/tty9 c 4 9
    mknod /dev/tty10 c 4 10
    mknod /dev/tty11 c 4 11
    mknod /dev/tty12 c 4 12
    mknod /dev/ttyS0 c 4 64
    mknod /dev/ttyS1 c 4 65
    mknod /dev/ttyS2 c 4 66
    mknod /dev/ttyS3 c 4 67
    echo Setting up hotplug.
    hotplug
    echo Creating block device nodes.
    mkblkdevs
    echo "Loading jbd.ko module"
    insmod /lib/jbd.ko
    echo "Loading ext3.ko module"
    insmod /lib/ext3.ko
    echo "Loading xenblk.ko module"
    insmod /lib/xenblk.ko
    echo "Loading xennet.ko module"
    insmod /lib/xennet.ko
    mkblkdevs
    echo Scanning and configuring dmraid supported devices
    resume /dev/sdc2
    echo Creating root device.
    mkrootdev -t ext3 -o defaults,ro /dev/sda1
    echo Mounting root filesystem.
    mount /sysroot
    echo Setting up other filesystems.
    setuproot
    echo Switching to new root and running init.
    switchroot
    EOL
    find ./ | cpio -H newc -o | gzip > /mnt/boot/initrd-2.6.18-194.8.1.el5xen.img
    cd -
    
  16. Install grub on the new instance, move the boot directory into a subdirectory and create a grub menu.lst file that points to the CentOS kernel and initrd file:

    chroot /mnt grub-install /dev/sdh
    
    mkdir /mnt/boot/boot/
    mv /mnt/boot/* /mnt/boot/boot/ 2> /dev/null > /dev/null
    

    Put the following in /mnt/boot/boot/grub/menu.lst (note that as soon as the CentOS Xen kernel version changes this will be incorrect):

    default 0
    timeout 1
    title CentOS5.5
         root (hd0,0)
         kernel /boot/vmlinuz-2.6.18-194.8.1.el5xen root=/dev/sda2
         initrd /boot/initrd-2.6.18-194.8.1.el5xen.img
    

    Note that this goes in /mnt/boot/boot/grub and that isn't the normal spot you would expect it in. This is where the AWS EC2 pv-grub expects to find the file on the first partition and moving the boot directory around just keeps everything in line with those expectations.

  17. Make sure everything is written to disk and unmount the volume. At this point you have a CentOS 5.5 install that is almost ready to boot.

    sync
    umount /mnt/proc
    umount /mnt/boot
    umount /mnt
    
  18. Make a snapshot of the volume you just installed to, you will need to volume ID that came from step 2:

    ec2-create-snapshot -d "Volume Description" volume-id
    
  19. Use the snapshot from step 18 along with the ec2-register command to register your instance:

    ec2-register -n "AMIName" -d "AMI Description" --root-device-name /dev/sda2 -b /dev/sda=snap-id:2:true
    

    There are a number of things to take note of with the above command:

    1. Running this command will result in output something like: IMAGE ami-a5ae9bb
    2. The -b option can now assign a snapshot to a block device, the options in this example tell EC2 to generate 2G of space for the snapshot and to delete the volume it creates from the snapshot if the instance terminates. If you plan to use an instance long term you should replace that true at the end with a false to keep EC2 from deleting the volume when the instance terminates.
    3. Notice that the -b option is assigning the snapshot to the device and not to a partition of the device, that is /dev/sda instead of /dev/sda1. You can still assign a snapshot directly to a partition but now you can also assign a block device to a raw partitioned disk. Because I created the partition table earlier the snapshot is the raw disk device here.
    4. Also note that we are missing the kernel configuration option. As of this post using it with a pv-grub kernel causes the register command to fail. It isn't a big issue but just keep that in mind when you fire the AMI up otherwise it won't boot with the correct pv-grub kernel.

  20. Start an instance of the fresh CentOS 5.5 install. One key thing here is picking the correct pv-grub kernel to boot from. There are currently 4 different kernels at each location, see the Enabling User Provided Kernels in Amazon EC2 document for a full list of kernels in each availability zone. In this case because the root disk was created with a partition table I used the "ec2-public-images/pv-grub-hd00-V1.01-i386.gz.manifest.xml" kernel to boot with (on US-East-1 that is kernel id aki-4c7d9525). For example:

    ec2-run-instances -z us-east-1a -g your-group -k your-keypair -n 1 --kernel pv-grub-kernel-id ami-from-step-19
    

Tips on debugging the boot process

If your instance won't boot you can use the ec2-get-console-output command to get the console output created from the pv-grub boot process. If your console output ends up like the following there are a number of things you may have done wrong.

  • You may have selected the wrong kernel and it is trying to boot from a non-existant partition. Make sure you are using the correct pv-grub kernel hd0 vs hd00.
  • You forgot to install grub or installed grub in the wrong place. Make sure you have either /boot/grub/menu.lst or /boot/boot/grub/menu.lst
  • You have a bad menu.lst file. One mistake I made was giving a boot item a title with a space in it. Make the menu.lst as simple as you can until you get it to boot.
    Xen Minimal OS!
  start_info: 0xb10000(VA)
    nr_pages: 0x6a400
  shared_inf: 0x002f9000(MA)
     pt_base: 0xb13000(VA)
nr_pt_frames: 0x9
    mfn_list: 0x967000(VA)
   mod_start: 0x0(VA)
     mod_len: 0
       flags: 0x0
    cmd_line:  root=/dev/sda1 ro 4
  stack:      0x946780-0x966780
MM: Init
      _text: 0x0(VA)
     _etext: 0x621f5(VA)
   _erodata: 0x76000(VA)
     _edata: 0x7b6d4(VA)
stack start: 0x946780(VA)
       _end: 0x966d34(VA)
  start_pfn: b1f
    max_pfn: 6a400
Mapping memory range 0xc00000 - 0x6a400000
setting 0x0-0x76000 readonly
skipped 0x1000
MM: Initialise page allocator for e6c000(e6c000)-0(6a400000)
MM: done
Demand map pfns at 6a401000-7a401000.
Heap resides at 7a402000-ba402000.
Initialising timer interface
Initialising console ... done.
gnttab_table mapped at 0x6a401000.
Initialising scheduler
Thread "Idle": pointer: 0x7a402008, stack: 0x6a030000
Initialising xenbus
Thread "xenstore": pointer: 0x7a402478, stack: 0x6a040000
Dummy main: start_info=0x966880
Thread "main": pointer: 0x7a4028e8, stack: 0x6a050000
"main" "root=/dev/sda1" "ro" "4"
vbd 2048 is hd0
******************* BLKFRONT for device/vbd/2048 **********


backend at /local/domain/0/backend/vbd/2111/2048
Failed to read /local/domain/0/backend/vbd/2111/2048/feature-barrier.
Failed to read /local/domain/0/backend/vbd/2111/2048/feature-flush-cache.
12582912 sectors of 0 bytes
**************************
vbd 2051 is hd1
******************* BLKFRONT for device/vbd/2051 **********


backend at /local/domain/0/backend/vbd/2111/2051
Failed to read /local/domain/0/backend/vbd/2111/2051/feature-barrier.
Failed to read /local/domain/0/backend/vbd/2111/2051/feature-flush-cache.
1835008 sectors of 0 bytes
**************************

    [H
    [J

    GNU GRUB  version 0.97  (1740800K lower / 0K upper memory)



       [ Minimal BASH-like line editing is supported.   For

         the   first   word,  TAB  lists  possible  command

         completions.  Anywhere else TAB lists the possible

         completions of a device/filename. ]




grubdom>
    [9;10H

Booting non-Linux OSes with EC2

I have attempted both FreeBSD and NetBSD in particular with no luck.

FreeBSD is tricky because it really wants to use its loader and while you can do that with the grub chainloader command it results in a grub error from EC2 about needing to load the kernel before booting:

root (hd0,1)

 Filesystem type unknown, partition type 0xa5

chainloader +1





Error 8: Kernel must be loaded before booting



Press any key to continue...

I was also able to try a modified version of FreeBSD that should boot without the loader but with that I get an error claiming the kernel isn't bziped:

root (hd0,1,a)

 Filesystem type is ufs2, partition type 0xa5

kernel /boot/loader



xc_dom_probe_bzimage_kernel: kernel is not a bzImage
ERROR Invalid kernel: xc_dom_find_loader: no loader found

xc_dom_core.c:523: panic: xc_dom_find_loader: no loader found
xc_dom_parse_image returned -1



Error 9: Unknown boot failure



Press any key to continue...

For NetBSD the result is actually a completely blank console log so I assume it causes some catastrophic failure that keeps the EC2 system from even being able to pull back a log.

Leave a Reply

Your email address will not be published. Required fields are marked *