Scraping a table based on the option value from a drop down

Question

I am trying to scrape values from a table after changing the option value from the dropdown. I found this stack overflow post scraping a response from a selected option in dropdown list, but I still haven't managed to get it working.

The site I'm trying to scrape is: https://www.myfxbook.com/forex-market/correlation

I'm trying to get the "5-minutes" chart but it still returns the default table of "1 day".

import requests
import pandas as pd
url = "https://www.myfxbook.com/forex-market/correlation"
with requests.Session() as session:
    response = session.get(url)
    soup = BeautifulSoup(response.content)
    data ={"timeScales": "5"}
    response = session.post(url, data=data)
    soup = BeautifulSoup(response.content)

pd.read_html(str(soup))

score 1 · Accepted Answer · answered Mar 03 '21 at 23:49

To scrape the Table, you can send a POST request to here by adding the correct headers and data

To select a different "Timeframe" from the dropdown, modify the value of timeScale (below) in the data dictionary. It seems that the values should be based on minutes, for example, to select the "5 minutes" table, use "timeScale": "5", or to select the "1 week" table, use timeScale: 10080.

In your example, to select the "5 day" table:

import requests
from bs4 import BeautifulSoup


headers = {
    "x-requested-with": "XMLHttpRequest",
    "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.190 Safari/537.36",
    "referer": "https://www.myfxbook.com/",
}

data = {
    "colSymbols": "8,9,10,6,7,1,4,2,5,3",
    "rowSymbols": "8,47,9,10,1234,11,103,12,46,1245,6,13,14,15,17,18,7,2114,19,20,21,22,1246,23,1,1233,107,24,25,4,2872,137,48,1236,1247,2012,2,1863,3240,26,49,27,28,2090,131,5,29,5779,31,34,3,36,37,38,2076,40,41,42,43,45,3005,3473,50,2115,2603,2119,1815,2521,51,12755,5435,5079,10064,1893",
    # Change the value of `timeScale` to get the different Timeframe, the value should be by minutes, for example to get 1 week, use `10080`
    "timeScale": "5",
    "z": "0.6367404250506281",
}

response = requests.post(
    "https://www.myfxbook.com/updateCorrelationSymbolMenu.json",
    headers=headers,
    data=data,
).json()

table_html = response["content"]["marketCorrelationTable"]
soup = BeautifulSoup(table_html, "html.parser")

Scraping a table based on the option value from a drop down

1 Answers1