Correct way to handle automatic mounting of volumes when restarting instances
Background#
- Operating System: Amazon Linux 2023
- Instance Type: t2.micro
I wanted to attach a new volume to my EC2 instance, but when I restarted the instance, the mounted volume disappeared. So, I searched online for how to automatically mount it upon reboot. I tried modifying the /etc/fstab file and then restarted, but I couldn't connect via SSH. After checking the logs, I realized that I had entered emergency mode... and I couldn't even access the interactive interface. "Clearly, I messed up on the first try."
Initially, I thought it was a problem with my proxy, but even after disabling it, I still couldn't connect.
I checked the system logs.
But AWS also provided a solution for failed startups, and there was a small line of text below.
The result:
Initially, I kept looking at the documentation, trying to find out which type of instance would support it, and even changed the instance type, but I still couldn't connect... Finally, I found a method to recover by temporarily starting a separate instance for recovery and modifying the damaged file.
Recovery Method#
Basically, it involves these steps:
- Stop the failed instance.
- Detach the root volume from the failed instance.
- Create a new EC2 instance in the same region.
- Attach the root volume to the new instance.
- Modify the
/etc/fstabfile.
Stop the Instance#
Just wait for it to stop. The purpose is to detach the volume.
Detach the Volume#
Find the volume mounted on the root ("/") directory and click "Detach".
Start a New Instance#
Configure it with the simplest settings, as the purpose is only to attach the previously detached volume.
Remember to specify the subnet (to select the same availability zone as the volume, for example, mine is us-west-1b).
Attach the Volume#
In the AWS management console, attach the previously detached volume to the new instance, and SSH into the new instance to mount it.
Here, you can see the content that we mistakenly modified earlier.
Use vim to directly modify the fstab file, detach the volume after making the necessary changes, and then mount it back to the previously non-starting instance. Make sure to enter "xvda" as the name when mounting it back (the default root volume name I see in AWS).
Finally, start the old instance, and the newly created instance can be deleted.
Success!