I have a cron job
scrape.sh that look like this:
#!/bin/bash touch rage cd /etc/myproject/scraper scrapy crawl foosite --set FEED_URI=../feeds/foosite.xml --set FEED_FORMAT=xml scrapy crawl barsite --set FEED_URI=../feeds/barsite.xml --set FEED_FORMAT=xml
When it executes the file rage does get created and judging from my syslog it does run as root, so permissions shouldn't be a problem.
May 6 17:35:01 server CRON: (root) CMD (/etc/myproject/scraper/scrape.sh) May 6 17:40:01 server CRON: (root) CMD (/etc/myproject/scraper/scrape.sh)
When I run
scrape.sh it executes as expected and puts the
foosite.xml file in the
../feeds directory, the directory exist and is empty when the cron jobs starts. What can I do to solve this issue?
- If I were going to guess the problem it was an environment issue (e.g. scrapy is not in the path).
- To debug, make sure your cron job is sending the standard out and standard error to a log file/and or syslog