I am quite new to web scraping and early on I found myself in a ditch. I wish to scrape Google search results in R. I use the following piece of code:
library(RCurl)
library(XML)
url <- 'https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=something%20random'
doc <- htmlParse(getURL(url))
attrs <- xpathApply(doc, "//h3//a[@href]")
The problem is that attrs turns out to be NULL. If I save the webpage to a .txt I do find the urls that I am looking for, so something is clearly wrong here. In other posts I find that scraping HTTPS could be the issue, however since there is no login requirement here or anything complicated my guess would be that this is not the problem.
