1

I am trying to scrape the following page with R:

http://monitoramento.sema.mt.gov.br/simlam/ListarTituloImovelRural.aspx?modeloTitulo=117

using the code "17942" as the trial input in "Número do CAR" as I know that works (see pic below).

As far as I can tell aspx are webforms using ajax requests to return data using a mix of javascript and html (apologies if I have this wrong).

From other stack overflows I have seen that others have used Selenium (e.g. Scrape "aspx" page with R), however from inspecting the page it seems to be obtaining data through a POST request.

I have tried getting the data using:

search1 <- "http://monitoramento.sema.mt.gov.br/simlam/ListarTituloImovelRural.aspx?modeloTitulo=117"
rawdata <- POST(search1,encode="form",add_headers(`txtNumeroTitulo` =  17821
                                   ,`hdnCodigoPagina`  = "869E318BBE656CB74E76744CD3026"
                                   ), progress()) 
d_content <- content(rawdata,"text") %>%
  read_html() %>%
  html_text()

Just as a really basic attempt to get something out I could look through, but the data I am hoping to get (see picture) is not there. I wonder if the issue is that the POST request on the site has request headers as well as headers within the form data, which my request is not getting right.

Any help greatly appreciated!

screenshot of webform

samalevy
  • 81
  • 8
  • 1
    I haven't used Selenium, but I doubt you should pass the values as headers. https://stackoverflow.com/a/31303228/1714 implies you should use a `data` parameter. But I think it's going to require a lot of effort to get it to work, as I suspect your `hdnCodigoPagina` parameter will change every time. I suspect (could be wrong) that you'd have to fetch the page with the form, extract the `hdnCodigoPagina` value from that, and pass it back with your `txtNumeroTitulo`. – Hobo Sep 29 '21 at 11:31
  • Thanks for sharing that post and the insights about `hdnCodigoPagina`, very helpful! Let's see if I can make it work – samalevy Oct 05 '21 at 13:28

0 Answers0