How to debug a cron job that runs the script, but the script does not seem to finish?

advertisements

I have a cron job scrape.sh that look like this:

#!/bin/bash
touch rage
cd /etc/myproject/scraper
scrapy crawl foosite --set FEED_URI=../feeds/foosite.xml --set FEED_FORMAT=xml
scrapy crawl barsite --set FEED_URI=../feeds/barsite.xml --set FEED_FORMAT=xml

When it executes the file rage does get created and judging from my syslog it does run as root, so permissions shouldn't be a problem.

May  6 17:35:01 server CRON[10233]: (root) CMD (/etc/myproject/scraper/scrape.sh)
May  6 17:40:01 server CRON[17804]: (root) CMD (/etc/myproject/scraper/scrape.sh)

When I run scrape.sh it executes as expected and puts the foosite.xml file in the ../feeds directory, the directory exist and is empty when the cron jobs starts. What can I do to solve this issue?


  1. If I were going to guess the problem it was an environment issue (e.g. scrapy is not in the path).
  2. To debug, make sure your cron job is sending the standard out and standard error to a log file/and or syslog