How to get information from a button to create url using selenium python?

Question

I am new to webscraping/selenium/html & using a website to get all my past order details in a dataframe.

So far I am able to pull out all the details within the order but I want to run loop on each order so that rather than manually going into each order I automatically fetch details from each order.

But to get into each order I have not been able to extract information from click button (i.e "View Details" in this website) to construct url for each order.

For example this is the order below:

When I inspect on View Details I get below highlight:

My attempt to get the details to create the url which is like: (xyz.com/account/orders/124994123/85405514)

from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Chrome('path/chromedriver.exe')
driver.get("https://www.URL.COM") 

urls = driver.find_elements(By.CSS_SELECTOR, ".css-901oao.css-bfa6kz.r-tdq03u.r-1oke55r.r-1b43r93.r-majxgm")

orders_list = []

for tag in urls:
    orders_list.append(tag.get_attribute('href'))
    
print(orders_list)

# output
[]

When I use tag.text then I get below results:

urls = driver.find_elements(By.CSS_SELECTOR, ".css-901oao.css-bfa6kz.r-tdq03u.r-1oke55r.r-1b43r93.r-majxgm")
# .css-1dbjc4n.r-1awozwy.r-14lw9ot.r-cdmcib.r-18grtmt.r-5kkj8d.r-1777fci.r-ymttw5.r-1f1sjgu

orders_list = []

for tag in urls:
    orders_list.append(tag.text)
    
print(orders_list)

['View Details', 'View Details', 'View Details', 'View Details', 'View Details', 'View Details', 'View Details', 'View Details', 'View Details', 'View Details', 'View Details', 'View Details', 'View Details', 'View Details', 'View Details', 'View Details', 'View Details', 'View Details', 'View Details', 'View Details']

But haven't been able to pull the information that can take me to the subsequent webpage of order details.

UPDATE: outer html of View Details button

<div dir="auto" class="css-901oao css-bfa6kz r-tdq03u r-1oke55r r-1b43r93 r-majxgm">View Details</div>

html of webpage after clicking on View Details: github link for html

`# output []` can be explained, cause you are looking for attribute `href` from the div which contains `View details` and that div does not have any any href. — cruisepandey, Nov 21 '21 at 16:12
`['View Details', 'View Details', 'View Details', 'View Details',` can also be explained since you are targeting the div that contains `View details`. — cruisepandey, Nov 21 '21 at 16:17
so how can I get `url link` from **each button** on this webpage ??? — ViSa, Nov 21 '21 at 16:18
when you click on view detail, what do you see ? Can you share HTML of the same ? — cruisepandey, Nov 21 '21 at 16:19
Also Please share HTML in text format, here are the steps `Press F12 in Chrome -> go to element section -> then right click on the element you want to share the outer HTML - > select copy and then outer HTML` — cruisepandey, Nov 21 '21 at 16:21
I have added `outerhtml` at the bottom of the post in the `UPDATE` section but this outerhtml seems pretty same as the `div` I have shared in image. I am not sure if I did this correctly or not. — ViSa, Nov 21 '21 at 16:29
No, I meant to ask what changes do you see in HTML? when you click on view details? You also will have to write code for click on view details. — cruisepandey, Nov 21 '21 at 16:40
well then it takes me to the new webpage that has order details. It is the same page for which earlier I asked question in the SO post which has line by line items with prices etc. Would you need html of new webpage ? — ViSa, Nov 21 '21 at 16:44
new webpage within same windows ? or in new windows? yes need to see HTML code — cruisepandey, Nov 21 '21 at 16:49
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/239432/discussion-between-visa-and-cruisepandey). — ViSa, Nov 21 '21 at 17:13
You do not need necessarily to extract the link in order to navigate to the View Details page. Selenium can click for you(press that link). Did you try to tag.click()? — Paulus, Nov 21 '21 at 17:59
@Paulus, I didn't try that earlier but after reading ur comment when I tried this: `urls = driver.find_elements(By.CSS_SELECTOR, ".css-901oao.css-bfa6kz.r-tdq03u.r-1oke55r.r-1b43r93.r-majxgm") orders_list = [] for tag in urls[1]: tag.click()` I got an error: `TypeError: 'WebElement' object is not iterable` — ViSa, Nov 21 '21 at 18:09
why you added index there? `for tag in urls[1]` Leave it as it was: for tag in urls: tag.click() — Paulus, Nov 21 '21 at 18:38
when I do it without index then I got this error: `ElementClickInterceptedException: Message: element click intercepted: Element
...
is not clickable at point (888, 9). Other element would receive the click:
...
(Session info: chrome=96.0.4664.45)`. I think this could be a relevant post for this now: https://stackoverflow.com/questions/57632563/elementclickinterceptedexception-message-element-click-intercepted-element-l — ViSa, Nov 21 '21 at 18:49
This error means, that while Selenium was trying to click on View Details, another Html element(pop-up, loading screen, etc) appeared on top of the View Details link and covered the link so selenium could not click on your element. Since you cannot provide the link here, you need to check what is covering View Details(Seems like some header element => `
`) — Paulus, Nov 21 '21 at 19:05
oohh .. I am not sure how to handle that because there shouldn't be any popups once logged in. Can there be an issue due to scrolling the window as well & it may not be able to find `View Details` if it is scrolled down on the page ? — ViSa, Nov 22 '21 at 06:06
I think I did & since there is `View Details` button for each order and that makes it more than 30-50 such buttons on the page, so if I will run them in loop then I am not sure if each of them will be visibly present on screen if it is necessary for them to be displayed. — ViSa, Nov 22 '21 at 19:08
Ok, I see. It's not a pop-up, it's a website's header which is covering the View Details. Does your script maximize the screen before going to the website like below? `driver = webdriver.Chrome('path/chromedriver.exe') driver.maximize_window() driver.get("https://www.URL.COM")` — Paulus, Nov 22 '21 at 19:09
No I haven't used `driver.maximize_window()` so far. Should I do that ?? — ViSa, Nov 22 '21 at 19:11
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/239461/discussion-between-paulus-and-visa). — Paulus, Nov 22 '21 at 19:15

How to get information from a button to create url using selenium python?

0 Answers0