16

I've been thinking about a Jenkins job to check the pricing for the agent boxes we spin up; every so often the price spikes and it'll be an hour or two before anyone notices that no new agents are coming up, and then we have to go in manually and check the spot price and adjust accordingly, or switch zones, etc.

My first instinct was that the best route is to have an hourly Jenkins job that runs a aws ec2 describe-spot-instance-requests and checks for failed requests (and then slacks us on a failure). But I'm wondering if there's a cleaner method that involves actually comparing hard prices (and could therefore tell us exactly what's off, and by how much), rather than looking at successful/failed requests.

Anyone set up something similar? How'd you do it?

Pierre.Vriens
  • 7,225
  • 14
  • 39
  • 84
Alex
  • 4,612
  • 6
  • 29
  • 49

3 Answers3

15

Spotted an open source tool called autospotting that just might help with this:

Once enabled on an existing on-demand AutoScaling group, it launches an EC2 spot instance that is cheaper, at least as large and configured identically to your current on-demand instances. As soon as the new instance is ready, it is added to the group and an on-demand instance is detached from the group and terminated.

We have this in our to-do pipeline, will be able to add more context once we finish that.

Update:

Another tool that was recently demo'ed in a conference was mapbox/spotswap

This works slightly differently. It monitors a normal ASG with either On-Demand or Reserved instances and then, if scale arises, bids and provisions spot instances of similar compute level on a separate ASG.

Hashfyre
  • 441
  • 3
  • 5
6

I would personally consider a model like this:

Timed Lambdas -> Checks spot price -> Push to ElastiCache

Then when you need instances:

Timed lambdas -> Pulls spot price from ElastiCache, sets it as environment variable on your Machine where you spin up IaC from -> This is parsed as argument to IaC code and pushes out the spot price

You could set some tolerances, too, within the lambdas (i.e. 10, 25, 50% increases based on importance) and a hard cap of on-demand, for example. It's also a great place to build the logic to handle, for example, finding the cheapest AZ, finding the relatively cheapest spot price (2xt2.medium vs t2.large), etc.

Henry
  • 516
  • 3
  • 4
4

Let me give you a tool-agnostic way of trying to do this.

every so often the price spikes and it'll be an hour or two before anyone notices that no new agents are coming up, and then we have to go in manually and check the spot price and adjust accordingly, or switch zones, etc

We faced the same problem in the infrastructure which we're building. So, we had if-else style blocks to set the bid price, depending on the on-demand price of the instance.

AWS has an API for getting the on-demand price of an instance. We used this Python wrapper for the purpose.

So, once we got the on-demand price (let's say X), we plugged in if-else style blocks, which are 0.4*X, 0.6*X, 0.8*X, X, which means we are trying for a bid price in the range 40%, 60%, 80% of the on-demand price. If all fails, then we are falling back to creating on-demand instances.

Also, as this is independent of AWS's current spot pricing, we never pay a price above the on-demand price.

But, if you're looking for a way to do it on-the-fly, then Hashfyre's solution should be the way to go.

Dawny33
  • 2,816
  • 3
  • 24
  • 62