-1

Example site:

https://listen.tidal.com/view/pages/single-module-page/b4104df9-ee72-4c3d-a7b4-30fa61faf56d/5/889e7c2a-e3be-472b-9801-bf6b40655050/1?artistId=8992

I use css to fetch rows like:

div[role=row]

so the code is like

dirver.findElementsByCss(div[role=row])

and the number is only the number of displayed, aka, visible rows, not whole rows. How to make a script that can take all rows?

1)

I've tried zoom out like here, failure:

  • this zooms out in terms of css, not browser zoom out, and it is useless in this case

  • zoom out has its limits

2)

scrolling down seems quite not optimall and not quite unprecise way of doing things like this

I am using ChromeDriver for Crome 77 and python

user2678074
  • 768
  • 3
  • 9
  • 22

2 Answers2

1

Since this table is using React, the elements will not even exist on the page at all unless you scroll down to them. I know you mentioned that scrolling is not an optimal solution, but if you need all the rows in the table, scrolling is the only choice here.

I've implemented a sample ScrollDown method that uses coordinates to scroll down a page. The code is in C# but the idea is the same for other languages.

To scroll using Javascript, you will need to provide coordinates to scroll to. If you wish to scroll multiple times -- for example, scroll down, get table rows, continue scrolling down & getting table rows until reaching end of table -- you will need to update the yCoordinate that you are scrolling with. Since scrolling is an up/down process, you can use the same xCoordinate every time.

Regarding increment -- this variable refers to how MUCH you want to scroll at once. increment should be equal to the number of pixels you wish to scroll. If you want to scroll 3 rows at once, and each row is 100px tall, you will need to set increment to 300.

Since I do not know the size or layout of your page, I cannot provide a more detailed method for you. I recommend implementing this method & experimenting with different x/y coordinates for scrolling to get a personalized solution for your test application.

public void ScrollDown(int startX, int startY, int increment, int numberOfScrolls) 
{
    // cast webdriver to javascript executor
    var executor = ((IJavaScriptExecutor) driver);

    // keep track of current y coordinate
    // since we are scrolling down, y coordinate will change with each scroll
    var currentYCoordinate = startY;

    // scroll down in a loop
    for (int i = 0; i < numberOfScrolls; i++)
    {
        // scroll down
        executor.ExecuteScript("window.scrollTo({startX}, {currentYCoordinate});");

        // todo: implement any FindElement methods to get the new elements which have been scrolled into view.

        // update y coordinate since we just scrolled down
        currentYCoordinate = currentYCoordinate + increment;

     }
}
CEH
  • 5,701
  • 2
  • 16
  • 40
-2

For sites which has grids that implements lazy loading. - You may be able to access the data by using the grid components methods. - or you can use devtools>network to check for request/response which yields data and try to replicate the request.

In your example, the data you are after is on pagedList>items. enter image description here

You should be able to, theoretically, replicate this api call, using a python rest API library. Hope this helps.

zer0gr4v
  • 62
  • 3