Scrapyd jobid value inside spider

Question

Framework Scrapy - Scrapyd server.

I have some problem with getting jobid value inside the spider.

After post data to http://localhost:6800/schedule.json the response is

status = ok
jobid = bc2096406b3011e1a2d0005056c00008

But I need use this jobid inside the current spider during the process. It can be used for open {jobid}.log file or other dynamic reasons.

class SomeSpider(BaseSpider):
    name = "some"
    start_urls = ["http://www.example.com/"]
    def parse(self, response):
        items = []
        for val in values:
            item = SomeItem()
            item['jobid'] = self.jobid # ???!
            items.append(item)
        return items

But I see this jobid only after the task is finihed :( Thanks!

score 10 · Answer 1 · edited Nov 29 '15 at 13:27

10

You can get it from the SCRAPY_JOB environment variable:

os.environ['SCRAPY_JOB']

edited Nov 29 '15 at 13:27

alecxe

462,703
120
1,088
1,195

answered Jan 08 '15 at 06:32

Hassan Raza

3,025
22
35

1

This is now `os.environ['SHUB_JOBKEY']` – scharfmn Nov 10 '20 at 06:09

score 6 · Accepted Answer · answered Mar 11 '12 at 13:48

6

I guess there is an easier way, but you can extract job id from command line args. IIRC, scrapyd launches a spider giving it a jobid in parameters. Just explore sys.args where you need jobid.

answered Mar 11 '12 at 13:48

warvariuc

57,116
41
173
227

1

All genius is easy ;) Thanks, mate! Some example: `if (len(sys.argv)>2): if ('_job' in sys.argv[3]): self.jobid = sys.argv[3].rsplit('=')` – lenriq Mar 11 '12 at 20:31
@Maxim, glad it worked. Please, don't forget to accept and upvote answers that worked for you. – warvariuc Mar 12 '12 at 05:33
It requeres 15 points of reputation. I'll back to this post after some growth ;) Thank you. – lenriq Mar 12 '12 at 12:53
10

You can also get it from the `SCRAPY_JOB` environment variable: `os.environ['SCRAPY_JOB']` – Pablo Hoffman Apr 01 '12 at 03:58
1

@PabloHoffman what happens if we have multiple schedules running? I get the jobids but I am not sure if they will be correct every time. – AliBZ Jan 27 '14 at 20:39
This is now os.environ['SHUB_JOBKEY'] – scharfmn Nov 10 '20 at 06:09

score 1 · Answer 3 · answered Apr 12 '21 at 08:26

In the spider.py -->

class SomeSpider(BaseSpider):
    name = "some"
    start_urls = ["http://www.example.com/"]

    def __init__(self, *args, **kwargs):
        super(SomeSpider, self).__init__(*args, **kwargs)
        self.jobid = kwargs.get('_job')

    def parse(self, response):
        items = []
        for val in values:
           item = SomeItem()
           item['jobid'] = self.jobid # ???!
           items.append(item)
        return items

Scrapyd jobid value inside spider

3 Answers3

Linked