The long history of EBS in providing storage to EC2 instances taught us that users need consistent performance. With that in mind, in 2012 AWS introduced EBS Provisioned IOPS or IO1 Volumes and EBS optimised instances. They allow you to specify a consistent IOPS rate when you create the volume.
Up until 2017 developers had no ability to modify the configurations of live volumes so AWS came through for their clients and released Elastic Block Store (EBS). This meant Developers could now interact dynamically with Elastic Volumes and being able to alter live volumes in real time makes the whole process of AWS interaction much more productive.
EC2 instances use EBS volume as virtual hard drive disk on AWS platform. Customers find it attractive because they do not need to use and maintain on-premise hardware equipment which gets difficult to maintain through long period of time. They offer a variety of processor, storage, networking, operating systems, and purchase models.
You can use them as servers to run wide spectrum of software for your needs. In our use case we have performed downsizing of EBS root volume for Talend remote engine and EBS data volume for Tableau server. These two use cases have some specific needs which need to be accomplished to succeed with downsizing of EBS volume.
EBS volume modification
Official AWS documentation offers step-by-step process for increasing volume size, modifying IOPS capacity and change of volume type. It is a straightforward process that does not take a lot of time to perform even in PROD environment. This process requires minimal downtime and has a minimal chance for volume malfunction.
If you are interested in reading more about this, you can visit the following links:
– Modify an EBS volume using Elastic Volumes
– Extend a Linux file system after resizing a volume
Downsize issues
Down-sizing or “shrinking” EBS volume is not a type of process that is covered in AWS official documentation, and it can be specific from one use case to another. However, the links above do not cover the topic on how to create a smaller volume from the existing volume.
There are several articles on EBS volume “shrinking”, but some of them were not straightforward as it looked at first glance. This was the most helpful one:
– Decrease the size of EBS volume in your EC2 instance
My investigation process led me on a path of searching for errors while performing EBS “shrinking”. There are numerous posts on this matter and this blog post is not big enough to cover them all.
Use case #1: Talend remote engine root volume
- shut down EC2-talend-remote-engine
- Create AMI snapshot for EC2-talend-remote-engine
Create an AMI from an Amazon EC2 Instance
- turn on EC2-talend-remote-engine You do not want to have long downtime in PROD environment.
- Create new test instance from that AMI and name it EC2-talend-remote-engine-test-instance You want to have parallel test instance running where you can perform “shrinking” of EBS root volume and not affect original one in any way.
- When test instance is up, connect to it through SSH and execute:
sudo /opt/TalendRemoteEngine/bin/stop
For my use case, I did not want both instances communicating with Talend studio and Talend cloud at the same time. For other use cases with different technologies, it is better to stop the running software to avoid any interference with the original instance.
- create new smaller volume and name it EC2-talend-remote-engine-test-instance-volume Create an Amazon EBS volume
- stop EC2 instance EC2-talend-remote-engine-test-instance and attach talend-remote-engine-test-instance-volume to EC2-talend-remote-engine-test-instance This is the best practice which avoids unintentional malfunction of EBS volume. That could happen if you attach it while the EC2 instance is in running state.
- start instance EC2-talend-remote-engine-test-instance
- connect with PuTTY to EC2-talend-remote-engine-test-instance
- format new volume talend-remote-engine-test-instance-volume with command
sudo mkfs -t ext4 /dev/xvdf
This part depends on the name that the EBS volume AWS gives you. If you go to EC2 -> Instances -> choose instance -> Storage you can see the name of your instance.
Example: /dev/xvdf
If you run command lsblk you can see your volume name.
- mount the new volume with commands:
- create directory
sudo mkdir /mnt/new-volume
- mount talend-remote-engine-test-instance-volume with
sudo mount /dev/xvdf /mnt/new-volume
- check if volume is mounted
df -h
- copy data from old root volume to the new volume
sudo rsync -aHAXxSPn/ /mnt/new-volume/
- run command to compare volumes
diff -r / /mnt/new-volume/
- run command:
sudo grub-install --root-directory=/mnt/new-volume/ --force /dev/xvdf OR
sudo grub2-install --root-directory=/mnt/new-volume/ --force /dev/xvdf
- unmount new volume with command:
sudo umount /mnt/new-volume
- run command
blkid
Check UUID and LABEL for the old volume. Copy UUID and LABEL somewhere so you can paste it later
- run command
sudo tune2fs -U COPIED_UUID /dev/xvdf
You need the UUID and LABEL from the old volume to be the same as the new one. If not, then the EC2 instance will reject the new one when it is attached in the next steps:
- run command
sudo e2label /dev/xvda1
With this command you can check the label, but you can also check the label while running command: blkid
- Run command
sudo e2label /dev/xvdf /
In the root volume the label is “/”, so you will need to copy that one on the new volume.
- stop ec2 instance EC2-talend-remote-engine-test-instance
- detach old and new volume
- attach new volume on /dev/xvda where the previous root volume was (it will probably offer /dev/sdb to attach but you need to change it to /dev/xvda)
- start instance and see if everything looks good
- if everything went well you can attach that volume as root volume on original instance.
Use case #2: Tableau server data volume
- shut down EC2-tableau server
- Create AMI snapshot for EC2-tableau server for backup
Create an AMI from an Amazon EC2 Instance
- Go to Tableau server Web UI and stop Tableau server
- turn on EC2-tableau server
Tableau server has very delicate and complex services that run “under the hood”. In our communication with the Tableau support team, we have not received step-by-step solution. They advised us to reinstall the entire software and did not provide any solution to the Gateway issue in the Web UI which occurs when you are trying to rsync data from volume to volume. The only solution left was to do it with downtime on PROD.
Tableau server was stopped and EC2 instance was running so data could be rsynced.
- create new smaller volume and name its tableau-server-data-volume-new Create an Amazon EBS volume
- connect with PuTTY to EC2-tableau server
- format new volume tableau-server-data-volume-new with command
sudo mkfs -t ext4 /dev/xvdf
This part depends on the name AWS gives your EBS volume. If you go to EC2 -> Instances -> choose instance -> Storage you can see your instance name.
Example: /dev/xvdf
If you run command lsblk you can see your volume name.
- mount the new volume with commands:
- create directory
sudo mkdir /data_TabServ_new/
- mount tableau-server-data-volume-new with
sudo mount /dev/xvdf /data_TabServ_new/
- check if volume is mounted
df -h
- copy data from old root volume to the new volume
sudo rsync -aHAXxSPn /data_TabServ/ /data_TabServ_new/
- run command to compare volumes
diff -r /data_TabServ/ /data_TabServ_new/
Precautions
Upsizing can and should be done gradually. AWS offers upsize with step-by-step instructions as we mentioned. CloudWatch service can be used to monitor metrics to determine what to do with EBS volume that needs modification.
Example would be:
- Original EBS volume size: 50GB
- Gradual expansion to 75GB -> 100GB -> 150GB
- Monitor CPU, Disk size and Memory through CloudWatch with each increase to determine what suits your client the best
- Downsizing should be treated as a stopgap and not a permanent solution.
Conclusion
EBS volumes are a great solution from AWS because they offer excellent scaling options and they offer good overview in their official documentation as we mentioned before. This was a step-by-step solution for our two cases so I hope it will be helpful if you end up with this type of assignment. Best practice would be to warn the client that upsizing is easy and downsizing tricky.
The easiest and most painless solution is to create useful metrics in CloudWatch to regularly monitor the state of our EC2 instances and their EBS volumes so if some issues arise, it is easy to scale it gradually. Big leaps are not recommended unless they are necessary. If downsizing or “shrinking” is necessary, you can try to use this blog post or the multitude of others written on this theme and decide what is best for your use case.