0

I am trying to get all the followers name from this website https://www.quora.com/profile/Karan-Bansal-3/followers

Since the whole page is not loaded at once, I am using this everytime in a loop : 

driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

Now since I can't select all the element at once, I am trying to use indexing to find the element in the loop.

people = driver.find_element_by_xpath("//div[@class='pagedlist_item'][i]/*/div[@class='ObjectCard-header']/a[@class='user']")

Here as you can see, I am trying to give the indexing using [i] which clearly doesn't work and in place of it, if I give [1] or any number it works well. So how can I select the element one by one.

Code snippet : 

i=1
target = open(filename,'w')
driver.get('https://www.quora.com/profile/Karan-Bansal-3/followers')
while True:
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
people = driver.find_element_by_xpath("//div[@class='pagedlist_item'][i]/*/div[@class='ObjectCard-header']/a[@class='user']")
target.write(people.text)
target.write("\n")
i = i+1
kjhughes
  • 106,133
  • 27
  • 181
  • 240
Arkham
  • 69
  • 1
  • 7
  • Are you replacing the variable "i" by its value? – derloopkat May 19 '16 at 12:35
  • @derloopkat I have shown that `i` is 1 initially and I know XPATH is not taking the variable `[i]`, so how can it be selected then because I need all the elements and it can only be done in a loop which will increment the value and so will the index. – Arkham May 19 '16 at 12:39

3 Answers3

2

Use string concatenation in the hosting language so that i is evaluated before constructing the XPath. Otherwise, [i] is a predicate testing for the presence of an i element. You didn't state what your hosting language is, but assuming string concatenation is "string" + "string":

 "//div[@class='pagedlist_item'][" + i + "]/*/div[@class='ObjectCard-header']/a[@class='user']"

See also: How to pass variable parameter into XPath expression?

Update: Ok, so you're hosting XPath in Python.

You can use + to concatenate above if you first cast i to a string via str(i),

 "//div[@class='pagedlist_item'][" + str(i) + "]/*/div[@class='ObjectCard-header']/a[@class='user']"

or you can use format() as is used in the link I provided:

 "//div[@class='pagedlist_item'][{}]/*/div[@class='ObjectCard-header']/a[@class='user']".format(i)

either way, place the above constructed XPath expressions into your call to find_element_by_xpath() and your problem should be solved.

Caution: Do not use this approach with untrusted values for i or you could open your code to XPath injection attacks.

Community
  • 1
  • 1
kjhughes
  • 106,133
  • 27
  • 181
  • 240
1

Firstly cast index/i variable to string using str(index) and then try using below:

content = rows.xpath('//div[@class="LookupHelpDesc"]['+index+']//text()').extract_first()

always use single quote.

Sajan Kaundal
  • 41
  • 2
0

The solution is to first convert the index to a string.

index = str(i)
people = driver.find_element_by_xpath("//div[@class='pagedlist_item'][" + index + "]/*/div[@class='ObjectCard-header']/a[@class='user']")
i++
Arkham
  • 69
  • 1
  • 7