0

I am trying to get a specific list of webelements from a websites. All of these elements have the exact same xpath, except one number of course. When I am trying to extract all of these elements with the following code, it works perfectly, unless the list contains more than 99 items. If the number in the xpath is above 99, Selenium will just not locate the element at, even though (I have tested it) the xpath stays the exact same.

I have already tried to single out the items, but Selenium acts like the 100th item and all above just do not exist.

I have singled out the following code (simplefied):

String xpath="/html/body/div[1]/div/div[4]/div[2]/div[1]/div/div/div/div/section/div/div/div[2]/section/ol/div[%d]/div/li/div[2]/div/div[1]"; 

private void getTextFromElements(WebDriver driver){
     for(int i=0; i<200; i++){
         if(driver.findElements(By.xpath(String.format(xpath, i)).size()!=0){
             System.out.println(driver.findElement(By.xpath(String.format(xpath, i))).getText());
         }else{
              break;
         }
     }
}

While it prints out the first 99 items perfectly, it throws the exception, that Selenium cannot locate the element as soon as it reaches the 100th element.

This is an example URL, from which I try to extract the data: link.

wittn
  • 298
  • 5
  • 16
  • is your url public?can you share? – KunduK Jul 19 '19 at 18:21
  • Did you try to scroll to the element? Additionally I would suggest to use String.format to build the xpath. – AndiCover Jul 19 '19 at 18:22
  • Please elaborate and explain at least with application screenshots if possible. URL will be great – Krunal Patel Jul 19 '19 at 20:13
  • @KunduK As one example, I am trying to retrieve all songs from the playlist of the following URL: [link](https://open.spotify.com/playlist/7jTiAazSl7oo2WRk4dpFbz) – wittn Jul 19 '19 at 22:43
  • @AndiCover As far as I know, scrolling the element requires finding it first. Please correct me if I am wrong, I have not worked a lot with Selenium. Also, the String.format was very useful, I did not know about it yet. – wittn Jul 19 '19 at 22:48
  • @KrunalPatel I'm happy to share the URL: [link](https://open.spotify.com/playlist/7jTiAazSl7oo2WRk4dpFbz) – wittn Jul 19 '19 at 22:52

2 Answers2

2

As I checked in your code it has default max limit. So you have to scroll down once after you reach to 99. Please refer attachment which i extract from your web. You can only capture elements which are available in your web page. Rest of the element will display upon your scroll down option. there are 209 total items in your list but you cannot get them because they are not visible at the moment.

enter image description here

For scroll down

WebDriver driver = new ChromeDriver();
JavascriptExecutor js = (JavascriptExecutor)driver;
js.executeScript("window.scrollBy(0, 250)", "");

please refer Scroll Element into View with Selenium

Devdun
  • 747
  • 5
  • 14
  • Unfortunaly, even with the provided code it throws the same error; Selenium can't locate the element. As I looked at the article you provided, I discovered, that your code does work for firefox, but not for chrome, which I am using (I apologize for not mentioning this fact earlier). Given that, I would use the code: ((JavascriptExecutor) driver).executeScript("arguments[0].scrollIntoView(true);", element); Eloborating, this means, that I'd have to locate the element - it is known that I am no able to. So is there any solution to this (except switching to firefox, which I cannot). – wittn Jul 20 '19 at 22:57
  • @DFJ I'm happy to hear that your code works in firefox. So answer is yes definitely you can use chrome as well. So your issue may be related with chrome driver config. can you share your config details here? I'll give a try to help. – Devdun Jul 21 '19 at 02:21
  • So, what you mean is that the cod which I mentioned in the comment above, actually works in chrome if the configuration is right? Or do you mean the code, which works for firefx, works in chrome as well with the right configuration? Also, I am happy to share the config details, unfortunaly, I do not know, how to read them out, otherwise I would add them to the question above. – wittn Jul 21 '19 at 09:48
  • @DFJ I guess you already got the answer that the problem was that it has a limit of 100 records in a playlist and dynamically loads when you scroll down to the last element shown on a list. – Krunal Patel Jul 23 '19 at 16:27
  • @KrunalPatel Yes, but unfortunaly, because the code Devdun provided works only for Firefox and I cannot figure out how to do it in Chrome without locating the element first, my problem is nt solved yet. – wittn Jul 23 '19 at 17:25
  • @DFJ see the solution posted this is what i found when explored further – Krunal Patel Jul 23 '19 at 20:53
0

I just rechecked the problem is XPath. it is different for both the browser.

For Example, let's suppose you want to go to a 1st element of that list then xpath for firefox is /html/body/div[1]/div/div[4]/div[2]/div[1]/div/div/div/div/section/div/div/div[2]/section/ol/div[1]

which in your case you are using but when it comes to the chrome it is /html/body/div[1]/div/div[5]/div[2]/div[1]/div/div/div/div/section/div/div/div[2]/section/ol/div[1]

somehow 3rd div tag in the path has index 4 for firefox and 5 for chrome. I know this is weird and rare. Try changing XPath for chrome and let me know if it still remains unsolved. if this worked then you need to configure your code in a way that it takes XPath according to the browser.

here is chrome's xpath match screenshot

chrome path

Krunal Patel
  • 257
  • 3
  • 10
  • Unfortunaly this does not work. With the xpath given by you, Selenium will not find any elements, not even those below 100. – wittn Jul 24 '19 at 20:23
  • please open link in chrome hit F12 and the ctrl+F and the copy xpath of firefox it won't highlight the element giving zero matches and change the div index mentioned to 5 from 4 it will give one match i hope this helps – Krunal Patel Jul 24 '19 at 20:44
  • I thinkt this is really weird, for me it shows the xpath I provided (see the link I provided) and I can extract items below 100 with my xpath, which means it must be working, just not for items above 100. [link](http://prntscr.com/ojlqgh) – wittn Jul 24 '19 at 21:07
  • Try the step i mentioned that in chrome replace 4 with 5 it will highlight element trust me – Krunal Patel Jul 25 '19 at 03:12
  • I added a screenshot to my answer for chrome please check with XPath that highlight same element – Krunal Patel Jul 26 '19 at 14:14
  • Yes, I already tried the step, but for reasons I won't understand Selenium won't find the elements at all with this XPath, also unfortunaly Chrome does not highlight the element either [link](http://prntscr.com/okpitl) – wittn Jul 27 '19 at 08:45