There are times when you might receive certain warning messages from amazon.
“Hello,
We have noticed that one or more of your instances are running on a host degraded due to hardware failure.
i-111111
The host needs to undergo maintenance and will be taken down. Your instances will be terminated at this point. We recommend that you launch replacement instances and start migrating to them.”
Any data stored on the ephemeral store, /mnt, will be lost in this process but data bundled into the AMI and uploaded into S3 will still be available. So the only work around is to launch a new instance and attach all your EBS volumes to this new instance. This article explains how to replace a degraded instance among a pool of instances behind an ELB ( Elastic Load Balancer ).
suppose you have two instances i-111111 and i-222222 behind an ELB named TestBalancer.
And the instance i-111111 gets degraded due to an hardware faillure. At this point you won’t be able to do anything with the instance , not even a reboot will be of help.
Remember, it is always a good practice to bundle your ami to the most latest one as you may add new applications or services to your instances quite frequently. This avoids the last minute panic of launching an instance from your old ami and updating all the recently configured services or applications in the newly launched instance. This will be very helpful if you are into Autoscaling.
1) Launch a new instance from the most latest ami , say ami-xxxxxx
You can launch a new instance either from Elasticfox ( Mozilla Firefox extension for managing your Amazon EC2 account ) or from the CLI as shown below.
[shell]
ec2-run-instances –region REGION ami-xxxxxx -k ec2-keypair -t Instance Type
-z Zone -g security group
[/shell]
This will create a new instance , say i-333333.
2) Deregister both the instances from the ELB.
[shell]elb-deregister-instances-from-lb TestBalancer –headers –instances i-111111,
i-222222
[/shell]
3) Register the instances i-222222 and i-333333 ( the newly launched instance ) with the ELB.
[shell]elb-register-instances-with-lb TestBalancer –headers –instances i-222222, i-333333
[/shell]
4) Verify if the instances are in service with the load balancer
[shell] elb-describe-instance-health TestBalancer
INSTANCE-ID i-222222 InService
INSTANCE-ID i-333333 InService
[/shell]
If they are InService as shown above, then no worries.
done ! Now you are back live with the new instance as a replacement for the degraded one. I would advise one to always store their important data in EBS volumes and automate daily or weekly snapshots of the volumes so that in such worst cases you just need to launch a new instance of the latest bundled ami and attach the EBS volumes to it.
Associated Links
