0

Hi

I'm trying to scrap some data from a website the data is displayed in a chart ( the data is currency prices over years)

I was able to get the XHR Request and the API link for the JSON data but when I open the response in the ( network tab or in a new tab ) the data is not completely displayed but in the chart the data is represented.

the api link

  1. searched about the problem and I found this post which says that the dev-tools truncates long network response I tried the solution but the same problem is still happening .

  2. tried to use wget to download them but it didn't help same issue appeared.

I'm opening the link in a separate tab on Brave browser (also tried Firefox) I don't know what's the problem Can you please help me ?!

John_Migla
  • 3
  • 2

1 Answers1

0

You can scrape that API endpoint (with Python) like below:

import requests
import pandas as pd

pd.set_option('display.max_columns', None, 'display.max_colwidth', None)

headers = {
    'content-type': 'application/json',
    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36'
}

r = requests.get('https://sy-exchange-rates-iwi3arxhhq-uc.a.run.app/api/rates?from=Thu,%2031%20Mar%202011%2021:00:00%20GMT&to=Mon,%2023%20Jan%202023%2002:03:23%20GMT&name=USD&source=liranews.info,sp-today.com,dei-sy.com&city=damascus', headers=headers)

df = pd.json_normalize(r.json())
print(df)

Result in terminal:

    timestamp   source  city    name    buy sell
0   2021-01-03T07:00:00Z    sp-today.com    damascus    USD 2855    2880
1   2021-01-03T07:00:00Z    liranews.info   damascus    USD 2855    2880
2   2021-01-03T07:00:00Z    dei-sy.com  damascus    USD 2845    2855
3   2021-01-03T08:00:00Z    sp-today.com    damascus    USD 2855    2880
4   2021-01-03T08:00:00Z    liranews.info   damascus    USD 2855    2880
... ... ... ... ... ... ...
50772   2023-01-22T22:00:00Z    liranews.info   damascus    USD 6625    6685
50773   2023-01-22T23:00:00Z    sp-today.com    damascus    USD 6625    6685
50774   2023-01-22T23:00:00Z    liranews.info   damascus    USD 6625    6685
50775   2023-01-23T01:00:00Z    sp-today.com    damascus    USD 6625    6685
50776   2023-01-23T01:00:00Z    liranews.info   damascus    USD 6625    6685
50777 rows × 6 columns
Barry the Platipus
  • 9,594
  • 2
  • 6
  • 30
  • see the first index have a timestamp of 2021 but in the chart there is data from 2011 which is not being returned here ! https://www.karamshaar.com/exchange-rates open the link and click on max – John_Migla Jan 23 '23 at 22:53
  • Thanks for your answer problem was solved ! I figured out that it's not making api calls to get this data It was a static data which was being loaded when the page rendered ! anyways thanks for your contribution ! – John_Migla Jan 23 '23 at 23:43
  • Glad to head that @John_Migla. Indeed, the data for previous years (restricted to a few datapoints) is loaded in page initially, and can be found in the script tag containing ` // by default select last 11 days only.`. – Barry the Platipus Jan 24 '23 at 12:20