I am using func to perform parallel commands on our servers.
The other day, we had an issue when a service restart of puppet via func made all our severs hit our puppetmaster at the same time.
My question: How can I execute the same exact command on a set of servers while adding a delay before it's executed on the individual servers?
E.g.: random_delay && service puppet restart
I am interested in the random_delay part of the command.
Jist:
sleep $((random % $MAXWAIT)) where $MAXWAIT is the maximum desired delay in seconds.
Your terminal's implementation of random may differ, for example:
echo $((RANDOM));
3540
echo $((RANDOM % 12));
9
echo 0.$[ $((RANDOM % 1000)) + 1 ]
0.536
Bash sleep a number of milliseconds:
sleep 0.$[ $((RANDOM % 1000)) + 1 ]
sleep $((RANDOM))
The RANDOM will return a value between 0 and 32767
If you need to set a lower and upper limit to your sleep you need to define two other variables as follows:
MINWAIT=10
MAXWAIT=30
sleep $((MINWAIT+RANDOM % (MAXWAIT-MINWAIT)))
0 */12 * * * root perl -e 'sleep int(rand(43200))' && service puppet restart
This will sleep a random amount between 0 seconds and 12 hours (43200 = 12 x 60 x 60) every 12 hours
I quite like S19N's innovative, yet less than ideal solution. I only say it's less than ideal, as it's still largely non-deterministic when stuff will actually happen. I'd much rather be able to guarantee when stuff will happen, and what stuff will happen when it does.
Puppet orchestration is actually a hard problem.
One of the "best-practice" solutions is to use MCollective which will not only allow you to configure when puppet runs on your cluster of machines, but you can also use it for other similar orchestration problems.
