scraping from each table based on dates

Question

I'm trying to scrape data https://www.bps.go.id/indicator/3/1/25/inflasi-umum-.html from 1995-to this day,is there any way i can do it? i'm stuck because each year have spesific table and different html. Thank you in advance

Answer 1

First, extract the options value from the select tag to get the url for each year :

import requests
from bs4 import BeautifulSoup
import pandas as pd

baseUrl = "https://www.bps.go.id"
dateFrom = 1995
dateTo = 2019

#get the options 
r = requests.get(f"{baseUrl}/indicator/3/1/25/inflasi-umum-.html")
soup = BeautifulSoup(r.text, "html.parser")
years = dict([
    (t.text, t["value"]) 
    for t in soup.find("select").findAll("option") 
    if t.get("value")
])

And then iterate through your range for each year, and use pandas to extract the table so you have a dictionnary with key as year and Dataframe as value :

#iterate through years
data = {}
ranges = range(dateFrom, dateTo + 1)
for n in ranges:
  print(f"get data for year {n}")
  r = requests.get(f"{baseUrl}{years[str(n)]}")
  table = pd.read_html(r.text)
  data[str(n)] = table[2]

print(data)

Try this on repl.it