Web scraping dynamic webpage with R

Question

my goal is to get data from this site : https://www.insee.fr/fr/recherche?q=Emploi-Population+active+en+2018&taille=20&debut=0, especially id links of different items.
I know that GET function doesn't work because it's dynamic and needed to be process by javascript (same that Web Scraping dynamic webpage Python). So i get info via inspector mode of my browser and found a POST query with the url.

Here is a reproductible example :

library(httr)

body <- list(q="Emploi-Population%20active%20en%202018",
             start="0",
             sortFields=data.frame(field="score",order="desc"),
             filters=data.frame(NULL),
             rows="50",
             facetsQuery=data.frame(NULL))

TMP   <- httr::POST(url = "http://www.insee.fr/fr/solr/consultation?q=Emploi-Population%20active%20en%202018",
              body = body,
              config = config(http_version=1.1),
              encode = "json",verbose())

Note that a i have to put http instead of https because i get nothing otherwise (My proxy is correctly configured and rstudio can connect to the internet).
All i get is a nice 500 error. Any Idea of what i miss ?

score 1 · Answer 1 · answered May 31 '22 at 16:34

You can change the q param and remove it from your url. I would use https and remove your config line to avoid the curl fetch error. However, the below, adapted to return 100 results, still works.

library(httr)

body <- list(
  q = "Emploi-Population active en 2018",
  start = "0",
  sortFields = data.frame(field = "score", order = "desc"),
  rows = "100"
)

TMP <- httr::POST(
  url = "http://www.insee.fr/fr/solr/consultation",
  body = body,
  config = config(http_version = 1.1),
  encode = "json", verbose()
)

data <- fromJSON(content(TMP, type = "text"))

print(data$documents$titre)

It's working now here, thanks a lot. I just have to add `config = use_proxy(url=myurl,port=myport)` to make it work. — Renaud Oger, Jun 01 '22 at 08:38

score 0 · Accepted Answer · answered May 31 '22 at 16:48

I found that passing the json as a string worked fine:

library(httr)

json <- paste0('{"q":"Emploi-Population active en 2018 ",',
            '"start":"0","sortFields":[{"field":"score","order":"desc"}],',
            '"filters":[],"rows":"20","facetsQuery":[]}')

url <- paste0('https://www.insee.fr/fr/solr/consultation?q=Emploi-Population',
               '%20active%20en%202018%20')

res <- POST(url, body = json, content_type_json())
output <- content(res)

Now output is a massive list, but here for example are the document titles:

sapply(output$documents, function(x) x$titre)
#>  [1] "Emploi-Population active en 2018"                                                                          
#>  [2] "Emploi – Population active"                                                                                
#>  [3] "Dossier complet"                                                                                           
#>  [4] "Base du dossier complet"                                                                                   
#>  [5] "Emploi-Population active en 2017"                                                                          
#>  [6] "Comparateur de territoire"                                                                                 
#>  [7] "Emploi – Population active"                                                                                
#>  [8] "L'essentiel sur... les entreprises"                                                                        
#>  [9] "Emploi - population active en 2014"                                                                        
#> [10] "Population active"                                                                                         
#> [11] "Emploi salarié et non salarié par activité"                                                                
#> [12] "Évolution de l'emploi"                                                                                     
#> [13] "Logements, individus, activité, mobilités scolaires et professionnelles, migrations résidentielles en 2018"
#> [14] "Emploi selon le sexe et l’âge"                                                                             
#> [15] "Statut d’emploi et type de contrat selon le sexe et l’âge"                                                 
#> [16] "Sous-emploi selon le sexe et l’âge"                                                                        
#> [17] "Emploi salarié par secteur"                                                                                
#> [18] "Fiche - Chômage"                                                                                           
#> [19] "Emploi-Activité en 2018"                                                                                   
#> [20] "Activité professionnelle des individus : lieu de travail localisé à la zone d'emploi en 2017"

^{Created on 2022-05-31 by the reprex package (v2.0.1)}

It's all good now, thanks a lot. I like the json as a string directly from the browser. I just have to add `config = use_proxy(url=myurl,port=myport)` to make it work. — Renaud Oger, Jun 01 '22 at 08:39

Web scraping dynamic webpage with R

2 Answers2