1

I am having troubles getting a cronjob on ubuntu 16.04 digital ocean to work.

I programmed a python spider, which i want to run every 5 minutes. In order to run this spider I made a script runmyspider.sh (chmod +x) with the command:

scrapy runspider aspider.py

Now I want to call this script via Cron. 

*/5 * * * * sh /scripts/runmyspider.sh 2>&1 /scripts/spider.log

However, the spider never runs (I can see that no changes in the database have been made, if I execute the file manually the changes happen)

What am I doing wrong here? I have set up a cron multiple times before, but this time I seem to get an error into it...

Thanks for all your advice!

Tom
  • 143

1 Answers1

0
  1. Always use absolute paths to avoid problems.

/bin/sh (or /usr/bin/?)

/usr/bin/scrapy (or wherever it is, you can use which scrapy to find the binary)

and most importantly add script path for the aspider.py file for scrapy to use, I guess it could be /scripts/aspider.py ?

My first guess is that cron is running, finds sh, which finds scrapy which doesn't find the py file.

  1. Don't overwrite the logfile, add to it:

*/5 * * * * /bin/sh /scripts/runmyspider.sh >> /scripts/spider.log 2>&1

Gnudiff
  • 533