I do R&D work on spot GPU's on AWS and have been, so far, launching them from a 50GB AMI that I create every time I need to shut off the instance. This has been working pretty well, except that startup times/accessing data on the AMI takes a while at first access because the AMI has to be downloaded from S3 and unpacked (I think this is how it works).
The reason I need to persist the image is that I want my configuration for python packages, as well as the packages themselves to be persisted, and for data to be saved locally in an uncompressed format. So, I can't just mount an NFS to solve this problem (although all of my data/code resides on an NFS in some format, and I just move the data over to the EBS once launched to decompress it.
For some reason EFS is slow with dealing with lots of small files, but thats a separate issue). It seems that EC2 doesn't allow using an EBS volume as root volume for spot instances.
Is there a way to achieve the above mentioned goals without purchasing the on-demand instance?
