Scrapy - xpath with a regular expression

Question

I'm scraping a page with some divs with the id = Content_Main_ some random number but can't get their content using the following xpath because the result is always empty, what I'm doing wrong?

//div[re:test(@id, '([Content_Main_]+\d{5}[0-9])')]

/div[@id='Content_Main_4']
/div[@id='Content_Main_5']
/div[@id='Content_Main_8']

This should help..https://stackoverflow.com/questions/21405267/xpath-using-regex-in-contains-function — Kamal, Jun 26 '19 at 03:09

gangabass · Accepted Answer · 2019-06-26T10:15:05.480

0

I think you need something like (starting with Content_Main_ and next a digit):

//div[re:test(@id, 'Content_Main_\d+')]

UPDATE To select divs ending with a number you need:

//div[re:test(@id, 'Content_Main_\d+$')]

edited Jun 26 '19 at 10:15

answered Jun 26 '19 at 05:18

gangabass

10,607
2
23
35

I think I'm almost there, now the response is not empty but some divs I don't need are coming too, how can I change the expression to ignore divs like I'd = Content_Main_31_userControl_lbl_title and get only the ones who start with Content_Main_ and ends right after the number (Content_Main_4, Content_Main_5, etc)? – Hana Iku Jun 26 '19 at 10:10

score 0 · Answer 2 · edited Feb 14 '21 at 10:16

0

You are searching exactly five digits after

text(Content_Main_)
\d{5}

Search 5 digits example : 12345; 76543....

make it like:

\d -- if it is always one digit
or
\d+ - if it can have one or more digit
or
\d* - if it may/may not have digit

edited Feb 14 '21 at 10:16

Samsul Islam

2,581
2
17
23

answered Jun 26 '19 at 11:31

Astha Kohli

17
2

Scrapy - xpath with a regular expression

2 Answers2