You may wonder why you would want to do this. I'll give you at least two reasons. First there is speed. The configuration I ended up with will boot in about 14 seconds. Almost all of that time is spent waiting on the AWS DHCP server for an IP address. The second reason is security. There is little on this system that you don't need. The majority of what you need is provided by a single application that you can easily monitor for patches. That leaves you to worry about only the tools you must have to get your job done. I'll also include a couple downsides. First this isn't going to be easy for everyone and it can be fairly complicated. The second issue you will run into is that there is nothing in this distribution. If you need something like PHP you might spend a long time building all the support you need. There isn't much to be done about the complexity if you want to do this all yourself. Selection of tools can help you lessen the impact of the bare nature of the system, more on that in a future post.

I picked TTYLinux because that is what I'm most familiar with. I have used it a number of times and I believe it may be the most basic of all the small Linux distros. It really is bare and the large majority of the features are provided by BusyBox. Before attempting to build your own instance it may help to read some of the TTYLinux documentation. Although I haven't tried I assume that other small distros like DSL would probably work with the same general instructions.

A couple important prerequisites are needed before starting this. First look at the How_To_Build_ttylinux.txt file that comes with the TTYLinux source. It contains a lot of information about how you go about building the system. Next look at my post on compiling the Linux kernel for EC2 because you will need to do that for this TTYLinux instance. Finally you need to have to have gcc, g++ and yacc available on the machine you are going to use to build on.

I have also created a public AMI that anyone can use with the name TTYLinuxBase and AMI ID of ami-0cfe0b65. Be sure to boot it with the correct hd0 kernel for your region. The password for root on this AMI is just password so be sure to change it when you fire it up. I have created the AMI so that the EBS volume gets deleted when it is terminated.

1. Grab the source from the TTYLinux source page:

   wget http://minimalinux.org/ttylinux/Download/ttylinux-src-mp3.tar.bz2
   tar xvjf ttylinux-src-mp3.tar.bz2
   

2. The kernel that comes with the current version of TTYLinux isn't new enough to work on EC2 so the configuration and build scripts need to be changed to get a more recent version that will. The first step is to create a kernel configuration and put it in the correct location:

   ttylinux-src-mp3/config/platform_pc/kernel-2.6.35.4-i686.cfg
   

   You can create this configuration by doing a make menuconfig on the kernel you want to use and then saving the kernel configuration to a file. You may also download the kernel config I used:

   wget http://www.ioncannon.net/examples/ec2ttylinux/kernel-2.6.35.4-i686.cfg
   cp kernel-2.6.35.4-i686.cfg ttylinux-src-mp3/config/platform_pc/kernel-2.6.35.4-i686.cfg
   

   There is a lot of the kernel that can be pruned from the kernel configuration so if you do this by hand take time to turn things off that you won't need. Doing so will speed up the kernel compile and make the system boot faster. The above kernel configuration is pruned down quite a bit.

3. Copy the packages configuration to one that matches the kernel version:

   cp ttylinux-src-mp3/config/platform_pc/packages-2.11-2.6.30.5.txt ttylinux-src-mp3/config/platform_pc/packages-2.11-2.6.35.4.txt
   

4. Modify the kernel build script to apply the XSAVE patch. Start by adding the following line to line 132 of the ttylinux-src-mp3/scripts/build-kernel.sh build script:

   patch -p1 < /tmp/kernel.patch
   

   You should end up with the following around it:

   echo -n "b." >&${CONSOLE_FD}
   cp "${kcfg}" "linux-${kver}/.config"
   cd "linux-${kver}"
   patch -p1 < /tmp/kernel.patch
   sed --in-place scripts/mod/sumversion.c \
           --expression="s|<string.h>| <limits.h>\n#include <string.h>|"
   source "${TTYLINUX_XTOOL_DIR}/_xbt_env_set"
   

   Create the patch file /tmp/kernel.patch with the following in it:

   --- a/arch/x86/xen/enlighten.c  2010-08-05 20:35:13.000000000 -0400
   +++ b/arch/x86/xen/enlighten.c  2010-08-05 20:35:22.000000000 -0400
   @@ -776,6 +776,7 @@
    {
    	cr4 &= ~X86_CR4_PGE;
    	cr4 &= ~X86_CR4_PSE;
   +	cr4 &= ~X86_CR4_OSXSAVE;
   
    	native_write_cr4(cr4);
    }
   

   Note that the above patch file has tabs in it. Make sure there is a tab before each cr4 line and the native_write_cr4 line or download the kernel patch file I created.

5. Select the correct build target in the ttylinux-src-mp3/ttylinux.dist-config.sh file. Find the TTYLINUX_TARGET lines, comment them all out and then add the following line:

   TTYLINUX_TARGET="i686:pc:2.11-2.6.35.4"
   

6. Set up the cross compile tools build directory:

   cp -Rp ttylinux-src-mp3/cross-tools-2.11-2.6.30.5/ ttylinux-src-mp3/cross-tools-2.11-2.6.35.4/
   

   Find XBT_KERNEL in the script ttylinux-src-mp3/cross-tools-2.11-2.6.35.4/_scripts/xbt-versions.sh and change it to:

   XBT_KERNEL="linux-2.6.35.4"
   

7. Build the cross compile tools:

   cd ttylinux-src-mp3/cross-tools-2.11-2.6.35.4
   make setup
   make dload
   make i686
   cd ..
   

8. Build the entire TTYLinux distro:

   make dist
   

   From this point the instructions are just like the last few posts I have made. The distribution created in here needs to be transferred to an EBS volume and made into an AMI.

9. Using a temporary EC2 instance create a volume that will be used to boot the install and give it a file system:

   ec2-create-volume -z us-east-1a -s 1
   ec2-attach-volume volume-id -i instance-id -d /dev/sdh
   mkfs.ext3 /dev/sdh
   

10. Transfer the TTYLinux distribution image that was created to the temporary EC2 instance. Then copy the image to the boot volume:

    mkdir /mnt/dest
    mkdir /mnt/src
    mount /dev/sdh /mnt/dest
    mount -o loop img/file_sys-i686-11.2.img /mnt/src
    cd /mnt/src
    find . | cpio -pvd /mnt/dest
    cd -
    cp config/boot/* /mnt/dest/boot/
    umount /mnt/src
    

    In the above the boot volume is mounted, the TTYLinux distribution image is mounted via a loop device and then all the files are copied from the image to the boot volume. I do a copy here because the default boot image that the TTYLinux build creates is only 8M and the smallest EBS volume you can create is 1G so it is better to use what you have to have.

11. While the boot volume is mounted there are a number of things that need to be configured that are different than some of the previous examples of booting custom systems in EC2. First set the password for root by using chroot:

    chroot /mnt/dest/ passwd
    

    Next change the root device in fstab:

    cat <<EOF > /mnt/dest/etc/fstab
    /dev/xvda1    /            ext3       defaults                      0 0
    tmpfs         /dev         tmpfs      noauto                        0 0
    devpts        /dev/pts     devpts     gid=5,mode=0620               0 0
    tmpfs         /dev/shm     tmpfs      rw,noexec,nosuid,size=24k     0 0
    proc          /proc        proc       noauto                        0 0
    sysfs         /sys         sysfs      noauto                        0 0
    EOF
    

    Then modify the inittab to only create one terminal:

    cat <<EOF > /mnt/dest/etc/inittab
    ::sysinit:/etc/rc.d/rc.sysinit
    
    tty1::respawn:/sbin/getty 38400 tty1
    
    ::ctrlaltdel:/sbin/reboot
    ::shutdown:/etc/rc.d/rc.sysdone
    EOF
    

    Then configure the network to start on boot:

    cat <<EOF > /mnt/dest/etc/sysconfig/network-scripts/ifcfg-eth0
    ENABLE=yes
    NAME=Ethernet
    IPADDRESS=192.168.1.20
    CIDRLEN=24
    NETWORK=192.168.1.0
    NETMASK=255.255.255.0
    GATEWAY=192.168.1.1
    BROADCAST=192.168.1.255
    DHCP=yes
    EOF
    

    Finally, If you want to get rid of the warnings about setting the hardware clock you can edit /mnt/dest/etc/rc.d/rc.sysdone and comment out the part that tries to set it:

    # Disabled for XenU use
    #if [[ "$(uname -m)" != armv5tej* ]]; then
    #     action $"syncing hardware clock to system time" hwclock ${HWCLOCKARGS}
    #fi
    

    and you will also want to do the same to /mnt/dest/etc/rc.d/rc.sysinit:

    # Disabled for XenU
    #if [[ "$(uname -m)" != armv5tej* ]]; then hwclock ${HWCLOCKARGS}; fi
    

12. To tell pv-grub what to boot you will need a /boot/grub/menu.lst file with the following in it:

    mkdir /mnt/dest/boot/grub
    
    cat <<EOF > /mnt/dest/boot/grub/menu.lst
    default 0
    timeout 0
    title TTYOS
            root (hd0)
            kernel /boot/vmlinuz root=/dev/xvda1
    EOF
    

    Unmount the boot image:

    umount /mnt/dest/
    

13. Snapshot the volume and register it as an AMI:

    ec2-create-snapshot -d "Volume Description" volume-id
    ec2-register -n "AMIName" -d "AMI Description" --root-device-name /dev/sda1 -b /dev/sda1=snap-id:1:true
    

14. Boot it using the hd0 kernel for your region (in my case that is aki-407d9529):

    ec2-run-instances -z us-east-1a -g your-group -k your-keypair -n 1 --kernel pv-grub-kernel-id ami-from-step-13
    

There are a few final notes that might be interesting. The smallest you can create is 1G and that is about 950M too large. This is probably not a real issue since you will most likely want space to put your application but it is interesting to note. The instructions assume you are building a i386 instance but they are almost the same for a 64 bit instance.

With the kernel config I provide you will see boot times from start to init in about 0.3 seconds. That is pretty fast. From init to login is fast as well but depends completely on how long it takes to get an IP from the DHCP server. This type of system could potentially boot in just a second or two if it didn't have to wait for any AWS parts.

A few people have created TTY addons to make compiling code for TTYLinux easier. You may want to check those out. Baring that you will find instructions on building anything with the cross compiling system in the TTYLinux howto documentation. Of course you may be able to bypass any pain there by compiling static binaries or even using something like Java.

It is important to have a recent kernel since most of what is needed to get a kernel to work on EC2 is now incorporated into the source. These instructions assume the latest kernel is 2.6.35.4 and I've used them with 2.6.35 as well but keep that in mind since the one patch that is required could eventually be merged in. Before getting started it may help to read over how to compile the Linux kernel normally and then my post on running CentOS 5.5 on EC2 using pv-grub.

Before you begin you will need a place to build the kernel. For these instructions I used an EC2 instance to build the kernel but you don't have to. I also installed the kernel on the same EC2 instance when I was done. The AMI I used was Amazon's EBS boot starter ( ami-b232d0db : amazon/getting-started-with-ebs-boot ).

The following steps go over building and installing the kernel in detail:

1. Download the latest Linux kernel or the version I'm using:

   wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.35.4.tar.bz2
   

2. Configure the source to run on EC2:

   make menuconfig
   

   You will need to make sure the following options are set in the configuration:

   • "Processor type and features" -> "High Memory Support" -> Make sure it is set to 64GB
   • "Processor type and features" -> "PAE (Physical Address Extension) Support" -> enable
   • "Processor type and features" -> "Paravirtualized guest support" -> enable
   • "Processor type and features" -> "Paravirtualized guest support" -> "Xen guest support" -> enable
   • "Device Drivers" -> "Block devices" -> "Xen virtual block device support" -> enable either as a module or built in
   • "Device Drivers" -> "Network device support" -> "Xen network device frontend driver" -> enable either as a module or built in

   
   

   If you want you can make the device drivers modules but you have to have them so it is probably best to just compile them into the kernel itself. If you want to compare your config file with the one I used you can download mine here: kernel-2.6.35.4-i686.config.

   The starter AMI I used needed ncurses development libraries and gcc installed for menuconfig to work:

   yum install ncurses-devel gcc
   

   If you want to know a little more about what is being enabled in this step see the "Building with domU support" section of XenParavirtOps.

3. Apply the following patch to disable XSAVE:

   --- a/arch/x86/xen/enlighten.c	2010-08-05 20:35:13.000000000 -0400
   +++ b/arch/x86/xen/enlighten.c	2010-08-05 20:35:22.000000000 -0400
   @@ -776,6 +776,7 @@
    {
    	cr4 &= ~X86_CR4_PGE;
    	cr4 &= ~X86_CR4_PSE;
   +	cr4 &= ~X86_CR4_OSXSAVE;
   
    	native_write_cr4(cr4);
    }
   
   cd /path/to/root/of/kernel/source
   
   patch -p1 < /tmp/kernel.patch
   

   Note that the above patch file has tabs in it. Make sure there is a tab before each cr4 line and the native_write_cr4 line. If you want to you can download a copy of the patch with the tabs in it to be sure.

4. Build the kernel and install it if you are on the same machine you want to install it on. If you need help compiling the kernel refer to the kernel compile howto.
   
   

   After this step you have a kernel, modules and initrd that you can use. The remaining steps go over using it.

5. Configure the /boot/grub/menu.lst file on the target AMI to use the new kernel, the following is an example of the contents of the file:

   default 0
   timeout 1
   title Test
        root (hd0)
        kernel /boot/vmlinuz-2.6.35.4 root=/dev/xvda1
        initrd /boot/initrd-2.6.35.4.img
   

   Note that the root device here is /dev/xvda1 instead of /dev/sda1. This is caused by the XSAVE patch.

6. Verify that your /etc/fstab file is correct. If your previous root device was /dev/sda1 it is going to be /dev/xvda1 now. The contents of the fstab file I used follow:

   /dev/xvda1                              /                       ext3    defaults 1 1
   /dev/mapper/swapVG-swapFS               swap                    swap    defaults 0 0
   /dev/mapper/storageVG-storageFS         /mnt                    ext3    defaults 0 0
   none                                    /dev/pts                devpts  gid=5,mode=620 0 0
   none                                    /dev/shm                tmpfs   defaults 0 0
   none                                    /proc                   proc    defaults 0 0
   none                                    /sys                    sysfs   defaults 0 0
   

7. Make a snapshot of the volume and register it as an AMI:

   ec2-create-snapshot -d "Snapshot Description" volume-id
   ec2-register -n "CustomKernel" -d "Custom kernel AMI" --root-device-name /dev/sda1 -b /dev/sda1=snap-id:15:true
   

   Note that the devices here are /dev/sda1 and not /dev/xvda1. That is a little confusing but the AWS system doesn't see the devices in the same way your AMI will once it is booted.

8. Start the instance. In my case using the hd0 pv-grub kernel:

   ec2-run-instances -z us-east-1a -g your-group -k your-keypair -n 1 --kernel aki ami-from-step-7
   

If all goes well you should be able to run dmesg and see a boot message something like the following at the top:

Reserving virtual address space above 0xf5800000
Linux version 2.6.35.4 (root@domU) (gcc version 4.1.2 20070925 (Red Hat 4.1.2-33)) #2 SMP Mon Aug 23 20:00:01 EDT 2010
BIOS-provided physical RAM map:
 Xen: 0000000000000000 - 00000000000a0000 (usable)
 Xen: 00000000000a0000 - 0000000000100000 (reserved)
 Xen: 0000000000100000 - 000000006a400000 (usable)
NX (Execute Disable) protection: active
...

With the ability to create a custom kernel for EC2 the next step is to prune the OS itself down to the bare minimum.

For those who might just be interested in booting a custom kernel using EC2 pv-grub I will try to produce a few more posts with more details on that but for now here are the main things to know:

• The pv-grup kernels named with hd00 will look on the first partition of the registered device in the /boot/boot/grub directory for a menu.lst file. Use this type of kernel if you create want to use a partitioned disk.
• The pv-grup kernels named with hd0 will look on the registered device in the /boot/grub directory for a menu.lst file. Use this type of kernel if you don't have a partition on your disk.
• You won't get anything meaningful back from the boot attempt if your grub menu.lst file is in the wrong place or is not valid. See the end of the post for what a pv-grub error message looks like and some tips on what to do if you see it.
• The kernel you use does matter but the current mainline Linux kernel (2.6.35) contains everything you need except for a small change to turn off XSAVE. The main thing to know is that not every distribution may have made the change needed to work on EC2.
• I have tried non-Linux kernels to no avail. See the end of the post for a little more information.

A lot of what follows is similar, both steps and concepts, to the "from scratch" section of my post on Fedora 12 on EC2 using a root EBS. I've also bundled all the instance building commands up into one script (centos5.5.sh). If you want to use that script then do 1 and 2 of what follows, make sure to change the password used for root in the script and then pick back up at 18. The following steps should not be taken as the only way to do this but more of a recipe:

1. Start an EC2 instance that has yum on it to be used as a setup box. A RedHat based box, Fedora or CentOS will work best unless you want to install yum. For the following steps I used a Fedora 8 based EC2 node.

   ec2-run-instances -z us-east-1a -g your-group -k your-keypair -n 1 ami-84db39ed
   

2. Create a new EBS volume to install to and map it to the running instance from step 1. Your volume should be greater than 2G for a base install. I mapped this new volume to the /dev/sdh device on the setup machine so you will notice that in the following steps (if you are using the script you will want to make sure you map to /dev/sdh as well):

   ec2-create-volume -z us-east-1a -s 2
   ec2-attach-volume volume-id -i instance-id -d /dev/sdh
   

3. Create a partion table using fdisk on the volume you are going to install to.

   I created both a /boot and / partion on /dev/sdh1 and /dev/sdh2 respecivly. I also made the /dev/sdh1 partition active so it is exactly as it would be if it had been installed on a real machine.

   Note that this step is optional but I am going to include it because I think it makes for a more natural setup and is more in line with what you would get if you did a VirtualBox install and then transfered the image.

4. Format your partition(s) and mount them into /mnt. For me that was done with the following:

   echo "y" | mkfs.ext3 /dev/sdh1
   echo "y" | mkfs.ext3 /dev/sdh2
   mount /dev/sdh2 /mnt
   mkdir /mnt/boot
   mkdir /mnt/dev
   mkdir /mnt/proc
   mkdir /mnt/etc
   mount /dev/sdh1 /mnt/boot
   mount -t proc none /mnt/proc
   

5. Create a base device setup for the new instance:

   for i in console null zero ; do /sbin/MAKEDEV -d /mnt/dev -x $i ; done
   

6. Create a base fstab file in /mnt/etc/fstab. The following is the one I used:

   /dev/sda1               /boot                   ext3    defaults 1 1
   /dev/sda2               /                       ext3    defaults 1 2
   none                    /dev/pts                devpts  gid=5,mode=620 0 0
   none                    /dev/shm                tmpfs   defaults 0 0
   none                    /proc                   proc    defaults 0 0
   none                    /sys                    sysfs   defaults 0 0
   /dev/sdc1               /mnt                    ext3    defaults 0 0
   /dev/sdc2               swap                    swap    defaults 0 0
   

7. Create the yum repo configuration, prepare for the yum install and then install the base OS onto the new volume.
   
   

   The following is the yum configuration file I used:

   [main]
   cachedir=/var/cache/yum
   debuglevel=2
   logfile=/var/log/yum.log
   exclude=*-debuginfo
   gpgcheck=0
   obsoletes=1
   reposdir=/dev/null
   
   [os]
   name=CentOS 5.5 - i386 - OS
   mirrorlist=http://mirrorlist.centos.org/?release=5&arch=i386&repo=os
   enabled=1
   
   [updates]
   name=CentOS 5.5 - i386 - Updates
   mirrorlist=http://mirrorlist.centos.org/?release=5&arch=i386&repo=updates
   enabled=1
   

   The following command will install the base of Cent OS 5.5 into /mnt (note that I created the above config file as /tmp/yumec2.conf):

   yum -c /tmp/yumec2.conf --installroot=/mnt -y groupinstall Base
   

8. Install sshd, grub, the Cent OS Xen kernel and then clean the repo to free up disk space:

   yum -c /tmp/yumec2.conf --installroot=/mnt -y install openssh-server
   yum -c /tmp/yumec2.conf --installroot=/mnt -y install grub
   yum -c /tmp/yumec2.conf --installroot=/mnt -y install kernel-xen.i686
   
   yum -c /tmp/yumec2.conf --installroot=/mnt -y clean packages
   

9. Disable DNS checks and allow root to log in via SSH:

   echo "UseDNS no" >> /mnt/etc/ssh/sshd_config
   echo "PermitRootLogin yes" >> /mnt/etc/ssh/sshd_config
   

10. Set up networking by creating the /mnt/etc/sysconfig/network file. The contents for this example are:

    NETWORKING=yes
    

    As well as the /mnt/etc/sysconfig/network-scripts/ifcfg-eth0 file. The contents for this example are:

    DEVICE=eth0
    BOOTPROTO=dhcp
    ONBOOT=yes
    TYPE=Ethernet
    USERCTL=yes
    PEERDNS=yes
    IPV6INIT=no
    

11. I'm not sure if this is needed still but in the past there have been some /dev file missing on boot so I always add the following to the startup script to make sure they are available. The first two are the random devices and the last three are where the ephimeral drive is usually mapped:

    echo "/sbin/MAKEDEV /dev/urandom" >> /mnt/etc/rc.sysinit
    echo "/sbin/MAKEDEV /dev/random" >> /mnt/etc/rc.sysinit
    echo "/sbin/MAKEDEV /dev/sdc" >> /mnt/etc/rc.sysinit
    echo "/sbin/MAKEDEV /dev/sdc1" >> /mnt/etc/rc.sysinit
    echo "/sbin/MAKEDEV /dev/sdc2" >> /mnt/etc/rc.sysinit
    

12. Change the root password for the new instance. This is optional as you could create scripts to download your SSH key from EC2 but for these instructions setting the root password is the easiest:

    chroot /mnt
    pwconv
    passwd
    exit
    

13. Change the network settings so that the NetworkManager is off and network is on

    chroot /mnt chkconfig --level 2345 NetworkManager off
    chroot /mnt chkconfig --level 2345 network on
    

14. Disable a few things that are enabled by default but won't do any good for an EC2 instance:

    chroot /mnt chkconfig --level 2345 avahi-daemon off
    chroot /mnt chkconfig --level 2345 firstboot off
    

15. The stock CentOS Xen initrd doesn't load the Xen block or net drivers and those are required to boot. I unpacked the installed initrd and created a modified version by hand using the following commands (note that as soon as the CentOS Xen kernel version changes this will stop functioning):

    cp /mnt/boot/initrd-2.6.18-194.8.1.el5xen.img /mnt/boot/initrd-2.6.18-194.8.1.el5xen.img.orig
    mkdir /tmp/initrdextract
    cd /tmp/initrdextract
    gzip -dc /mnt/boot/initrd-2.6.18-194.8.1.el5xen.img | cpio -id
    cp /mnt/lib/modules/2.6.18-194.8.1.el5xen/kernel/drivers/xen/blkfront/xenblk.ko lib
    cp /mnt/lib/modules/2.6.18-194.8.1.el5xen/kernel/drivers/xen/netfront/xennet.ko lib
    chmod -x lib/xenblk.ko
    chmod -x lib/xennet.ko
    cat <<EOL > init
    #!/bin/nash
    
    mount -t proc /proc /proc
    setquiet
    echo Mounting proc filesystem
    echo Mounting sysfs filesystem
    mount -t sysfs /sys /sys
    echo Creating /dev
    mount -o mode=0755 -t tmpfs /dev /dev
    mkdir /dev/pts
    mount -t devpts -o gid=5,mode=620 /dev/pts /dev/pts
    mkdir /dev/shm
    mkdir /dev/mapper
    echo Creating initial device nodes
    mknod /dev/null c 1 3
    mknod /dev/zero c 1 5
    mknod /dev/urandom c 1 9
    mknod /dev/systty c 4 0
    mknod /dev/tty c 5 0
    mknod /dev/console c 5 1
    mknod /dev/ptmx c 5 2
    mknod /dev/rtc c 10 135
    mknod /dev/tty0 c 4 0
    mknod /dev/tty1 c 4 1
    mknod /dev/tty2 c 4 2
    mknod /dev/tty3 c 4 3
    mknod /dev/tty4 c 4 4
    mknod /dev/tty5 c 4 5
    mknod /dev/tty6 c 4 6
    mknod /dev/tty7 c 4 7
    mknod /dev/tty8 c 4 8
    mknod /dev/tty9 c 4 9
    mknod /dev/tty10 c 4 10
    mknod /dev/tty11 c 4 11
    mknod /dev/tty12 c 4 12
    mknod /dev/ttyS0 c 4 64
    mknod /dev/ttyS1 c 4 65
    mknod /dev/ttyS2 c 4 66
    mknod /dev/ttyS3 c 4 67
    echo Setting up hotplug.
    hotplug
    echo Creating block device nodes.
    mkblkdevs
    echo "Loading jbd.ko module"
    insmod /lib/jbd.ko
    echo "Loading ext3.ko module"
    insmod /lib/ext3.ko
    echo "Loading xenblk.ko module"
    insmod /lib/xenblk.ko
    echo "Loading xennet.ko module"
    insmod /lib/xennet.ko
    mkblkdevs
    echo Scanning and configuring dmraid supported devices
    resume /dev/sdc2
    echo Creating root device.
    mkrootdev -t ext3 -o defaults,ro /dev/sda1
    echo Mounting root filesystem.
    mount /sysroot
    echo Setting up other filesystems.
    setuproot
    echo Switching to new root and running init.
    switchroot
    EOL
    find ./ | cpio -H newc -o | gzip > /mnt/boot/initrd-2.6.18-194.8.1.el5xen.img
    cd -
    

16. Install grub on the new instance, move the boot directory into a subdirectory and create a grub menu.lst file that points to the CentOS kernel and initrd file:

    chroot /mnt grub-install /dev/sdh
    
    mkdir /mnt/boot/boot/
    mv /mnt/boot/* /mnt/boot/boot/ 2> /dev/null > /dev/null
    

    Put the following in /mnt/boot/boot/grub/menu.lst (note that as soon as the CentOS Xen kernel version changes this will be incorrect):

    default 0
    timeout 1
    title CentOS5.5
         root (hd0,0)
         kernel /boot/vmlinuz-2.6.18-194.8.1.el5xen root=/dev/sda2
         initrd /boot/initrd-2.6.18-194.8.1.el5xen.img
    

    Note that this goes in /mnt/boot/boot/grub and that isn't the normal spot you would expect it in. This is where the AWS EC2 pv-grub expects to find the file on the first partition and moving the boot directory around just keeps everything in line with those expectations.

17. Make sure everything is written to disk and unmount the volume. At this point you have a CentOS 5.5 install that is almost ready to boot.

    sync
    umount /mnt/proc
    umount /mnt/boot
    umount /mnt
    

18. Make a snapshot of the volume you just installed to, you will need to volume ID that came from step 2:

    ec2-create-snapshot -d "Volume Description" volume-id
    

19. Use the snapshot from step 18 along with the ec2-register command to register your instance:

    ec2-register -n "AMIName" -d "AMI Description" --root-device-name /dev/sda2 -b /dev/sda=snap-id:2:true
    

    There are a number of things to take note of with the above command:

    1. Running this command will result in output something like: IMAGE ami-a5ae9bb
    2. The -b option can now assign a snapshot to a block device, the options in this example tell EC2 to generate 2G of space for the snapshot and to delete the volume it creates from the snapshot if the instance terminates. If you plan to use an instance long term you should replace that true at the end with a false to keep EC2 from deleting the volume when the instance terminates.
    3. Notice that the -b option is assigning the snapshot to the device and not to a partition of the device, that is /dev/sda instead of /dev/sda1. You can still assign a snapshot directly to a partition but now you can also assign a block device to a raw partitioned disk. Because I created the partition table earlier the snapshot is the raw disk device here.
    4. Also note that we are missing the kernel configuration option. As of this post using it with a pv-grub kernel causes the register command to fail. It isn't a big issue but just keep that in mind when you fire the AMI up otherwise it won't boot with the correct pv-grub kernel.

    
    

20. Start an instance of the fresh CentOS 5.5 install. One key thing here is picking the correct pv-grub kernel to boot from. There are currently 4 different kernels at each location, see the Enabling User Provided Kernels in Amazon EC2 document for a full list of kernels in each availability zone. In this case because the root disk was created with a partition table I used the "ec2-public-images/pv-grub-hd00-V1.01-i386.gz.manifest.xml" kernel to boot with (on US-East-1 that is kernel id aki-4c7d9525). For example:

    ec2-run-instances -z us-east-1a -g your-group -k your-keypair -n 1 --kernel pv-grub-kernel-id ami-from-step-19
    

Tips on debugging the boot process

If your instance won't boot you can use the ec2-get-console-output command to get the console output created from the pv-grub boot process. If your console output ends up like the following there are a number of things you may have done wrong.

• You may have selected the wrong kernel and it is trying to boot from a non-existant partition. Make sure you are using the correct pv-grub kernel hd0 vs hd00.
• You forgot to install grub or installed grub in the wrong place. Make sure you have either /boot/grub/menu.lst or /boot/boot/grub/menu.lst
• You have a bad menu.lst file. One mistake I made was giving a boot item a title with a space in it. Make the menu.lst as simple as you can until you get it to boot.

    Xen Minimal OS!
  start_info: 0xb10000(VA)
    nr_pages: 0x6a400
  shared_inf: 0x002f9000(MA)
     pt_base: 0xb13000(VA)
nr_pt_frames: 0x9
    mfn_list: 0x967000(VA)
   mod_start: 0x0(VA)
     mod_len: 0
       flags: 0x0
    cmd_line:  root=/dev/sda1 ro 4
  stack:      0x946780-0x966780
MM: Init
      _text: 0x0(VA)
     _etext: 0x621f5(VA)
   _erodata: 0x76000(VA)
     _edata: 0x7b6d4(VA)
stack start: 0x946780(VA)
       _end: 0x966d34(VA)
  start_pfn: b1f
    max_pfn: 6a400
Mapping memory range 0xc00000 - 0x6a400000
setting 0x0-0x76000 readonly
skipped 0x1000
MM: Initialise page allocator for e6c000(e6c000)-0(6a400000)
MM: done
Demand map pfns at 6a401000-7a401000.
Heap resides at 7a402000-ba402000.
Initialising timer interface
Initialising console ... done.
gnttab_table mapped at 0x6a401000.
Initialising scheduler
Thread "Idle": pointer: 0x7a402008, stack: 0x6a030000
Initialising xenbus
Thread "xenstore": pointer: 0x7a402478, stack: 0x6a040000
Dummy main: start_info=0x966880
Thread "main": pointer: 0x7a4028e8, stack: 0x6a050000
"main" "root=/dev/sda1" "ro" "4"
vbd 2048 is hd0
******************* BLKFRONT for device/vbd/2048 **********

backend at /local/domain/0/backend/vbd/2111/2048
Failed to read /local/domain/0/backend/vbd/2111/2048/feature-barrier.
Failed to read /local/domain/0/backend/vbd/2111/2048/feature-flush-cache.
12582912 sectors of 0 bytes
**************************
vbd 2051 is hd1
******************* BLKFRONT for device/vbd/2051 **********

backend at /local/domain/0/backend/vbd/2111/2051
Failed to read /local/domain/0/backend/vbd/2111/2051/feature-barrier.
Failed to read /local/domain/0/backend/vbd/2111/2051/feature-flush-cache.
1835008 sectors of 0 bytes
**************************

    [H
    [J

    GNU GRUB  version 0.97  (1740800K lower / 0K upper memory)

       [ Minimal BASH-like line editing is supported.   For

         the   first   word,  TAB  lists  possible  command

         completions.  Anywhere else TAB lists the possible

         completions of a device/filename. ]

grubdom>
    [9;10H

Booting non-Linux OSes with EC2

I have attempted both FreeBSD and NetBSD in particular with no luck.

FreeBSD is tricky because it really wants to use its loader and while you can do that with the grub chainloader command it results in a grub error from EC2 about needing to load the kernel before booting:

root (hd0,1)

 Filesystem type unknown, partition type 0xa5

chainloader +1

Error 8: Kernel must be loaded before booting

Press any key to continue...

I was also able to try a modified version of FreeBSD that should boot without the loader but with that I get an error claiming the kernel isn't bziped:

root (hd0,1,a)

 Filesystem type is ufs2, partition type 0xa5

kernel /boot/loader

xc_dom_probe_bzimage_kernel: kernel is not a bzImage
ERROR Invalid kernel: xc_dom_find_loader: no loader found

xc_dom_core.c:523: panic: xc_dom_find_loader: no loader found
xc_dom_parse_image returned -1

Error 9: Unknown boot failure

Press any key to continue...

For NetBSD the result is actually a completely blank console log so I assume it causes some catastrophic failure that keeps the EC2 system from even being able to pull back a log.

The Aptana Air plugin supports developing Adobe Air applications using HTML and Javascript. It even support the 2.0 release of Air that is currently in beta. Aptana uses the Eclipse framework as an editor so if you are familure with Eclipse it will be even easier to use.

I started by downloading and installing the latest version of the Air runtime. Next I grabbed the Air SDK, the SDK doesn't come with the plugin so it is something you have to get directly from the Air developers site. After getting the SDK unpacked I installed the latest Aptana core release. Once the core is installed there is a big plugin button on the startup screen that currently has Air listed.

The install went smoothly except for a few issues. The first one I ran into was very noticeable since it kept any dialog buttons from working when they were clicked although they did work when I clicked them and then hit enter or navigated to them with the keyboard. Luckily someone has already figured out that there is an issue with Eclipse and GTK+ that is the cause (even though the post is for Ubuntu the same problem and solution worked for me on Fedora). The fix is to set the GDK_NATIVE_WINDOWS variable before running the Aptana binary:

The next thing I noticed was the application.xml descriptor that Aptana created didn't generate correctly. It needs to start with the correct xmlns or the following error will be thrown on run: "invalid application descriptor: descriptor version does not match runtime version". To fix this check the version of the Air SDK by running the following command:

For the version of the Air SDK I downloaded the correct xmlns was http://ns.adobe.com/air/application/1.5 so I needed the following application tag:

Once I had that working I was able to compile and execute a demo application. I was also able to create an Air application package from within Aptana using File > Export > Adobe AIR > Adobe AIR Package. Before creating the Air package I had to create a signing certificate. Creating the certificate can be done within Aptana too but because I had not yet fixed the above button issue I created a cert on the command line with the Air SDK and then imported it. To create the Air signing certificate from the command line I used the adt command from the SDK:

Remember the password that gets used to generate the certificate because it will have to be used before a package is signed.

Finally Adobe has a lot of information on developing Air applications on their Air devnet site. The Air ajax section is especially important.

• Updated window managers Gnome 2.28, KDE 4.3 and Fedora Moblin
• Delta RPM support
• i686 as the base architecture
• Lots of virtualization changes: KSM, KVM huge page support, KVM QCow2 performance improvements, KVM Stable Guest ABI, libguestfs, Virtual network management and improved virtual privileges to name a few
• An easier to use bug reporting interface Abrt 1.0
• Better Webcam Support

You can find the complete list of Fedora 12 enhancements as well if you want more details.

I'm again starting with PreUpdate since it worked well last time. The steps are pretty much the same as last time but I did have more problems after the upgrade:

1. You have to be at Fedora11 before you try this. If you aren't there follow the steps to get to Fedora11.
2. yum clean all
3. yum udpate
4. Make sure to back up your xorg.conf since it can disappear with the upgrade.
5. preupgrade-cli "Fedora 12 (Constantine)" The total download for the upgrade from Fedora 11 to Fedora 12 was 1.1GB for me
6. After I rebooted the first time I got an error that the /boot directory didn't have enough space. It turns out I needed about 27M of free space so I ended up having to delete old kernels until I had 28M of free space.
7. reboot and wait

I lucked out again and didn't have to remove anything to fix dependency issues. I recompiled my existing NVIDIA driver but I ran into an issue with the nouveau NVIDIA driver that comes with Fedora 12. I needed to remove the nouveau package, rebuild initrd and reboot before I could build the NVIDIA driver:

If that is too much work you can also get the NVIDIA driver from atrpms but you will still need to remove the nouveau driver first. I also needed to pull down the Fedora 12 version of Virtualbox but so far that was all.

If you prefer the yum upgrade option here are the steps for that as well:

1. yum clean all
2. yum update
3. rpm -Uvh http://mirrors.kernel.org/fedora/releases/12/Fedora/i386/os/Packages/fedora-release-notes-12.0.0-4.fc12.noarch.rpm http://mirrors.kernel.org/fedora/releases/12/Fedora/i386/os/Packages/fedora-release-12-1.noarch.rpm
4. yum clean all
5. yum -y update
6. You may need to resolve dependencies and then do another yum -y update
7. reboot

I had to resolve dependencies to get this to work. I had to remove tigervnc-server-1.0.0-2.fc11.i586 and VirtualBox-3.0.10_54097_fedora11-1.i586. The resulting update was about 1.4G so it took a little while to apply.

I've also put together a few videos and screenshots if you want to get a quick preview of what the different versions available look like. You can also view the videos on Youtube: Moblin, KDE and Gnome

For people who can a complete re-install is probably best. One reason for that is the inclusion of ext4 in Fedora11. You won't get the benefit of ext4 unless you do a fresh install or upgrade from ext3 to ext4. If you read the upgrade guide that Fedora produces it recommends not doing an upgrade.

This time around I decided to go with PreUpdate right off the bat and it worked great. Here are the condensed steps. I think this will be the last time I do an upgrade post since they have it down to almost nothing now.

1. You have to be at Fedora10 before you try this. If you aren't there follow the steps to get to Fedora10.
2. yum clean all
3. yum udpate
4. preupgrade-cli "Fedora 11 (Leonidas)"
5. reboot and wait

It seems like the days of conflicts before upgrades are gone now so that is a good thing. I didn't have to remove anything to make dependencies work out. The only fallout I had from this upgrade was my NVIDIA configuration for xorg got wiped and I had to pull it from backup. So take note to back up your xorg.conf if you have a custom one.

Tags: linux, fedora

I'm using FreeRADIUS version 2.0.3 so some of these issues may be fixed down the road. The first thing you should do is compile FreeRADIUS and get it working using the normal users file. After you have done that and successfully tested queries to the server you can recompile to build Oracle in.

I used the Oracle Instant client again. I've used the Oracle instant client a number of times now and I can't believe it took them so long to release their SDK in this type of paired down package.

This should be all you need to add to the configure command to enable the Oracle driver:

However that didn't work for me. Instead I had to go into the RLM Oracle driver directory and run the configure command from there:

This created the Makefile but then that still wasn't correct. I had to modify the includes and libraries so they matched the correct location:

After making these changes I could then do a make and make install. You can verify that the module is installed by looking for the module file named rlm_sql_oracle.a in your lib directory (in my case /usr/local/lib/). After you have verified that the module is compiled and in place you are ready to move on to the configuration.

The first thing to do is load the provided Oracle schema. That schema can be found in: freeradius-server-2.0.3/raddb/sql/oracle

Next read over the RLM SQL configuration information to get a general idea of what is going on in the configuration files and how FreeRADIUS uses the queries to find the correct information for a given request.

The following steps are needed to configure the Oracle access and have FreeRADIUS use that configuration for data (I assume that you have installed with a base of /usr/local):

1. Edit /usr/local/etc/raddb/sql.conf set database = "oracle", set the server, login, password, and radius_db values. The following is an example of the connection information needed:
   # Connection info:
   server = "127.0.0.1"
   login = "username"
   password = "password"

   # Database table configuration for everything except Oracle
   #radius_db = "radius"
   # If you are using Oracle then use this instead
   radius_db = "(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=127.0.0.1)(PORT=1521))(CONNECT_DATA=(SID=MYDB01)))"

2. Search for the following and uncomment the SQL load line in the file /usr/local/etc/raddb/sites-enabled/default as follows:
   # See "Authorization Queries" in sql.conf
     sql

   Note: This is something that I didn't find in the documentation. I believe that is due to the documentation being for an older version and this being a new requirement.

3. Load sample data into the database:
   INSERT INTO radusergroup VALUES(radusergroup_seq.NEXTVAL, 'dynamic', 'fredf');
   INSERT INTO radcheck VALUES(radcheck_seq.NEXTVAL, 'fredf', 'Cleartext-Password', ':=', 'wilma');
   INSERT INTO radreply VALUES(radreply_seq.NEXTVAL, 'fredf', 'Framed-IP-Address', ':=', '1.2.3.4');
   INSERT INTO radgroupreply VALUES(radgroupreply_seq.NEXTVAL, 'dynamic', 'Framed-Compression', ':=', 'Van-Jacobsen-TCP-IP');
4. Start the daemon in debug mode:
   radius -X

   If you don't have the oracle libraries in your path you will need to start radius with the correct LD path entry like this:

   LD_LIBRARY_PATH=<path to oracle instant client> radiusd -X
5. Send a test query:
   radtest fredf wilma localhost 0 radpassword

   On the console for radius -X you will see debug and you should receive a valid response from the test that looks like this:

   User-Name = "fredf"
   User-Password = "wilma"
   NAS-IP-Address = 127.0.0.1
   NAS-Port = 0
   Framed-IP-Address = 1.2.3.4

At this point you are ready to load your data into FreeRADIUS via Oracle.

Other Notes:

Depending on how large your configuration values are the provided schema may not give you enough room to store everything. You may need to alter the tables to increase the space available for values like this:

ALTER TABLE radreply MODIFY value varchar(128);
]]> http://www.ioncannon.net/system-administration/136/freeradius-with-oracle/feed/ 3 http://www.ioncannon.net/system-administration/134/netcat-examples/ http://www.ioncannon.net/system-administration/134/netcat-examples/#comments Thu, 24 Apr 2008 13:18:41 +0000 carson http://www.ioncannon.net/?p=134 I recently got a new work PC and was worried that stuck somewhere in the 40G hard drive of the old PC was something I would one day need. The new PC had 300G of space so I figured I would just copy the entire drive over and keep it forever. This isn't the most difficult task in the world and I actually started out using ssh to transfer the image.

To transfer the drive using ssh I was using the following command from the new PC:

ssh -c blowfish old-pc-ip "dd if=/dev/hda" > dd of=old-pc.hda

This worked fine but the old PC seemed to be having a hard time keeping up. I did some adjusting to the block size of the transfers using dd but that didn't seem to help. That is when I decided to give netcat a try.

On the old PC side I ran:

dd if=/dev/hda | nc -l 10001

and on the new PC side I ran:

nc old-pc-ip 10001 | dd of=old-pc.hda

This worked like a charm and transfered the drive about twice as fast as ssh using blowfish encryption.

Just for reference I was using dd here to give myself more control over block sizes, skip any read errors, and at times I was actually trying to just transfer parts of the drive instead of the entire thing using the seek and count options. Another useful trick with dd is that you can find out the current amount transfered and rate by sending it a USR1 signal with kill. After poking around I actually found another nice utility that you can stick in the stream call pipe viewer that is able to give you a nice display of the count instead of having to send signals to dd.

Another great use for netcat that I ran into recently was setting up a ppp tunnel between two machines. Again at first I started by using ssh:

pppd updetach noauth passive pty "ssh remote-host-ip -lroot -o Batchmode=yes pppd nodetach notty noauth" ipparam vpn 192.168.77.1:192.168.77.2

This worked between two machines that were already connected by a network but my real goal was build the tunnel over a device that wouldn't work with ssh traffic. I turned to netcat again.

On the initiating side I ran this:

/usr/sbin/pppd connect-delay 30000 updetach noauth passive pty "echo connect-585 | nc device-ip 2000" ipparam root 192.168.77.1:192.168.77.2

Here I send the traffic to a device in between that forms a connection over a non-network link to the end point machine. I don't need netcat on the endpoint because the communication channel is not a network. On the end point I run this:

/usr/sbin/pppd nodetach notty noauth

In this case netcat saved me from having to write some intermediate code that would communicate with the device and just pipe the bits through.

]]> http://www.ioncannon.net/system-administration/134/netcat-examples/feed/ 0 http://www.ioncannon.net/system-administration/128/how-to-create-a-fedora-7-instance-for-ec2/ http://www.ioncannon.net/system-administration/128/how-to-create-a-fedora-7-instance-for-ec2/#comments Sat, 02 Jun 2007 15:38:34 +0000 carson http://www.ioncannon.net/uncategorized/128/how-to-create-a-fedora-7-instance-for-ec2/ Now that Fedora 7 is out I figured it was time to update the EC2 instance howto. It is almost exactly the same as creating a FC6 instance so if you want to know the details you can reference that article.

Here is an updated script for creating the AMI the only change between this and the one for FC6 is the yum repo and the image name:

#!/bin/sh

dd if=/dev/zero of=fedora7-i386.img bs=1M count=1 seek=1024
/sbin/mke2fs -F -j fedora7-i386.img

mount -o loop fedora7-i386.img /mnt

mkdir /mnt/dev
mkdir /mnt/proc
mkdir /mnt/etc
for i in console null zero ; do /sbin/MAKEDEV -d /mnt/dev -x $i ; done

cat <<EOL > /mnt/etc/fstab
/dev/sda1               /                       ext3    defaults 1 1
none                    /dev/pts                devpts  gid=5,mode=620 0 0
none                    /dev/shm                tmpfs   defaults 0 0
none                    /proc                   proc    defaults 0 0
none                    /sys                    sysfs   defaults 0 0
/dev/sda2               /mnt                    ext3    defaults 1 2
/dev/sda3               swap                    swap    defaults 0 0
EOL

mount -t proc none /mnt/proc

cat <<EOL > /tmp/yumec2.conf
[main]
cachedir=/var/cache/yum
debuglevel=2
logfile=/var/log/yum.log
exclude=*-debuginfo
gpgcheck=0
obsoletes=1
reposdir=/dev/null

[base]
name=Fedora Core 6 – i386 – Base
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=fedora-7&arch=i386
enabled=1

[updates-released]
name=Fedora Core 6 – i386 – Released Updates
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=updates-released-f7&arch=i386
enabled=1
EOL

yum -c /tmp/yumec2.conf –installroot=/mnt -y groupinstall Base

yum -c /tmp/yumec2.conf –installroot=/mnt -y clean packages

mv /mnt/lib/tls /mnt/lib/tls-disabled

cat <<EOL >> /mnt/etc/rc.local
if [ ! -d /root/.ssh ] ; then
        mkdir -p /root/.ssh
        chmod 700 /root/.ssh
fi
# Fetch public key using HTTP
curl http://169.254.169.254/1.0/meta-data/public-keys/0/openssl > /tmp/my-key
if [ $? -eq 0 ] ; then
        cat /tmp/my-key >> /root/.ssh/authorized_keys
        chmod 600 /root/.ssh/authorized_keys
        rm /tmp/my-key
fi
# or fetch public key using the file in the ephemeral store:
if [ -e /mnt/openssh_id.pub ] ; then
        cat /mnt/openssh_id.pub >> /root/.ssh/authorized_keys
        chmod 600 /root/.ssh/authorized_keys
fi
EOL

cat <<EOL >> /mnt/etc/ssh/sshd_config
UseDNS  no
PermitRootLogin without-password
EOL

cat <<EOL > /mnt/etc/sysconfig/network
NETWORKING=yes
HOSTNAME=localhost.localdomain
EOL

cat <<EOL > /mnt/etc/sysconfig/network-scripts/ifcfg-eth0
ONBOOT=yes
DEVICE=eth0
BOOTPROTO=dhcp
EOL

sync
umount /mnt/proc
umount /mnt

Tags: ec2, fedora 7

]]> http://www.ioncannon.net/system-administration/128/how-to-create-a-fedora-7-instance-for-ec2/feed/ 3 http://www.ioncannon.net/system-administration/119/admin-shared-mysql/ http://www.ioncannon.net/system-administration/119/admin-shared-mysql/#comments Mon, 29 Jan 2007 11:09:19 +0000 carson http://www.ioncannon.net/system-administration/119/admin-shared-mysql/ I just finished reading a post to the Media Temple blog about their MySQL problems . I think it is an excellent example of what happens when you only have one side of the house trying to fix a problem. The post leaves out some details but they make it clear that they believe their problems were caused by badly written apps hammering the database. It sounds like they tried very hard to fix the issues on the hardware and MySQL side but couldn't so have switched the way they are provisioning the database systems to more isolate the problem sites. The moral of that story is that even when you are smart you can't always fix software problems on the systems side.

The Media Temple guys don't go into any great detail on their current shared MySQL system but I would think that if nothing else they ran into the problems listed in this post: performance of complex queries. At some point you just have too many people trying to hit your database for any one person to achieve efficiency.

Tags: mysql, administration

]]> http://www.ioncannon.net/system-administration/119/admin-shared-mysql/feed/ 0