This article explains how to resolve some of the common errors that you may encounter when bundling an AMI ( Amazon Machine Image) of an AWS EC2 instance. It is indeed a very good practice bundling your AMI frequently as the latest updates of your app gets stored directly to S3. So even if your instance gets crashed you can launch another instance of your latest ami and get your site back live in minutes!.
But bundling can be a big hassle if we get stuck with rsync errors during the process. There are quite a few cases where rsync fails during bundling.
1) No space left on the device
This usually happens when we specify the size of the / partition with the -s parameter that specifies size, in MB of the image file to create. The maximum size is 10240 MB. Removing the “-s” flag and letting the system determine the size of the image itself will fix it. Also the bundling process will try to include all softlinks ( if any ) to other partitions of larger size ( greater than 10GB ). This would try creating an image file larger than 10GB size and hence throw bundle errors. So check for any softlinks before bundling.
2) Size of /tmp
By Default ec2-bundle-vol will try to bundle the image in /tmp. So if the size of /tmp is small this will throw bundling errors too. Specifying “-d /mnt” to point to the ephemeral drive rather than bundling the image in /tmp by default is a fix for this.
3) rsync warning: some files vanished before they could be transferred
This one is quite an interesting error as it is a warning that some files are being deleted while the bundling process is happening. In my case I had an instance with around 2GB free space in / partition. Also there was a 100GB EBS mounted on /home_ebs. I tried bundling the instance by excluding the 100GB EBS, but still it gave rsync error.
[shell]# ec2-bundle-vol -r i386 -d /mnt -p “myami” -u xxxxxxxxx -k /mnt/pk-xxxxxxxxxxxxxxxx.pem -c /mnt/cert-xxxxxxxxxxxxx.pem -s 10240 -e /home_ebs –debug [/shell]
Copying / into the image file /mnt/myami…
Excluding:
/sys
/proc
/dev/pts
/proc/sys/fs/binfmt_misc
/dev
/media
/mnt
/proc
/sys
/home_ebs
/mnt/myami.
/mnt/img-mnt
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.003113 seconds, 337 MB/s
mke2fs 1.39 (20-June-2010)
file has vanished: “/var/log/dcpumon/toplog.1276842601”
file has vanished: “/var/log/dcpumon/toplog.1276842901”
file has vanished: “/var/log/dcpumon/toplog.1276843201”
rsync warning: some files vanished before they could be transferred (code 24) at main.c(892) [sender=2.6.8]
Executing: umount -d /mnt/img-mnt
ERROR: execution failed: “rsync -rlpgoD -t -r -S -l –exclude /sys –exclude /proc –exclude /dev/pts –exclude /proc/sys/fs/binfmt_misc –exclude /dev –exclude /media –exclude /mnt –exclude /proc –exclude /sys –exclude /home_ebs –exclude /mnt/myami –exclude /mnt/img-mnt -X /* /mnt/img-mnt 2>&1 > /dev/null”
Running Bundle Volume from within a running installation may be problematic because partially written files may get copied into the AMI. To minimize the risk the best option is to stop all non essential services and try rebundling. In my case I removed the /var/log/dcpumon directory and went for a rebudling. That did the trick !.
Please note that using the “–debug” option with ec2-bundle-vol command will be of immense help in troubleshooting bundling errors. Hope this helps anyone out there ! 🙂
Associated Links
