0

INSPECT NETWORK SCREENSHOT

Website Link: https://www.yes.com.kh/support/find-shop

My code for now just manage to transfer the location name. It is very hard to get the longitude and latitude because of the code is in network file. Any ideas guys? This is so hard.

 #situation: -no elements in page source
 # - use inspect to find the page source
 # - in the same div class 

import selenium 
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from webdriver_manager.chrome import ChromeDriverManager
#from selenium.webdriver.common.keys import Keys
import pandas as pd

#install a webdriver
driver = webdriver.Chrome(ChromeDriverManager().install())

#input link for the web driver to access
driver.get('https://www.yes.com.kh/support/find-shop')

locations=WebDriverWait(driver,20).until(EC.presence_of_all_elements_located((By.XPATH,'//div[@class="item blog"]//div[@class="d-title"]')))

This is my output. It just manage to capture all the data on the website but there are other data when you try to load more in all the provinces listed.

                                   Location Name
0                            Tit Heng Phone Shop
1                               Nina2 Phone Shop
2                                 G B Phone Shop
3                             Rathana Phone Shop
4                    Chet Traprek Hou Phone Shop
5                            Siv Heng Phone Shop
6                                D Na Phone Shop
7                       Heang Heang 2 Phone Shop
8                              Rayuth Phone Shop
9                               Virak Phone Shop
10                         yes hub Sihanoukville
11  yes hub Sihanouk Ville International Airport
12                                  Kim Nai Houy
13                          Kim Sreng phone shop
14                                   Por Sokleng
15                            yes hub Battambang
16                       Yorn Kim Err Phone Shop
17                                   Yorn Bunyan
18                           Yon Hour Phone Shop
19                           Yin Fuoy Phone Shop
20                                yes hub Poipet
21                        yes hub Serei Sorphorn
22                                    Sam Sophea
23                                 Kheang Lyhour
24                                   Rith Sovann
25                             yes hub siem reap
26       yes hub Siem Reap International Airport
27                            Vannak Phone Shopp
28                           Rattanak Phone Shop
29                          Hun Rithy Phone Shop
30                                   yes hub kep
31                                 yes hub takeo
32                          yes hub kampong speu
33                       Khoun Phalla Phone Shop
34                          Pheng Nai phone shop
35                     Pheng Chhunleng phon shop
36                       Cheang Srong phone shop
37                                yes hub Kampot
38                                Thov Vannarith
39                                Preap Samphors
40                          yes hub kampong cham
41                         Pea Vibuth Phone shop
42                       Oeng Sopanha Phone Shop
43                          Sorn Naro phone shop
44                         Ka tangkea phone shop
45                                yes hub pursat
46                            yes hub svay rieng
47                         Kim Veasna phone shop
48                       Lim Sorphorn phone shop
49                         Mao Sithan phone shop
50                         Yuk Samorn phone shop
51                              yes hub koh kong
52                                yes hub pailin
53                        Veng Hong 2 Phone Shop
54                             Veasna Phone Shop
55                           Van Seam Phone Shop
56                          Srey Neth Phone Shop
57                          yes hub Kampong Thom
58                       yes hub Kampong Chhnang
59                             yes hub prey veng
bob
  • 79
  • 2
  • 7
  • 2
    So you want to get the data from calling `/shops?lang=en&limit=5&list=web&type=&ehub=true`? This is calling an API, doing that via a UI automation tool makes it unnecessary hard. What you could do (a bit of effort though): Export the curl of that request into Postman, export the request from Postman as Python, and use that in your code, see https://pastebin.com/Zux5kJRQ – Christian Baumann Oct 09 '20 at 06:14
  • I don't get it. Use what in my code? And what is the function of pastebin? – bob Oct 09 '20 at 07:13

1 Answers1

2

You don't need to use selenium on this. You can get what you wanted thru API then use json parser on response. Something like this: enter image description here

Millie Anne Volante
  • 171
  • 5
  • can you elaborate more? In a step by step process? By the what is the link to the json parser – bob Oct 09 '20 at 07:08
  • here's the link for json parser https://jsonpath.herokuapp.com/ As you mentioned, the latitude and longitude is in network, get that response. Then you search online for json parser library that you may use depending on the language you're gonna use. The link I gave is for parsing it manually, one by one only. If you want to automate it, you should use a library. Okay? – Millie Anne Volante Oct 09 '20 at 07:12
  • this is really helpful Millie! But, can you guide me how to export thru an API? – bob Oct 09 '20 at 07:20
  • Sorry, I am not sure what you are asking. If you just want to copy the response, then just copy it from the Network tab. If you want to use python in api requests, you can install 'requests' module (https://stackoverflow.com/questions/30362600/how-to-install-requests-module-in-python-3-4-instead-of-2-7). But I think you have to look for beginners tutorial online before doing this automation stuff. – Millie Anne Volante Oct 09 '20 at 07:55
  • how did you copy from the network beautifully? When I copied mine it became compact. I copied from Network Response and paste it directly to the jsonpath. Thx Millie – bob Oct 09 '20 at 08:07
  • I just use this site to beautify it https://jsonlint.com/ :) – Millie Anne Volante Oct 09 '20 at 08:17