1

I am scraping an Airbnb page using rvest.

My objective is to get the number of listings of a user (on the lower left-hand side of the web page) as well as the links for each listing.

However, it seems that Airbnb is blocking access to the source or something. I am a bit lost..

1) Using SelectorGadget and rvest, I have identified the node I'm interested in. Here is my entire code:

library(rvest)
URL = "https://www.airbnb.com/users/show/..."
--> put any user id instead of ...
source = read_html(URL)
source %>% html_nodes(".row-space-3") %>% .[[1]] %>% html_text()

And here is my (disappointing) output:

[1] "\n          "

Looking for the webpage source code I should get "Listings (2)" - here it is:

<div class="listings row-space-2 row-space-top-4">
     <h2 class="row-space-3">
          Listings
          <small>(2)</small>
     </h2>

What is happening?

PS:

2) I noticed that when I try to get the source code by brute force with XML THERE IS A WHOLE SECTION MISSING if compared to the source code on Chrome or Firefox

library(XML)
library(RCurl)
URL = "https://www.airbnb.com/users/show/..." 
parsed <- htmlParse(getURL(URL),asText=TRUE,encoding = "UTF-8")
dstudeba
  • 8,878
  • 3
  • 32
  • 41
Phelk
  • 19
  • 4
  • Do you have to be signed in to an account to see the number of listings on a user page? I don't see that even for users whose profiles I click through from their listings. If that's the issue, it could explain why rvest isn't seeing the listings. – Sam Firke Nov 10 '15 at 13:54
  • Oh ok, it might be precisely that, thank you!! I can see it on my browser because I am already logged in, but you are right, if not logged it this section is invisible even on the browser... does that mean that I should access airbnb's website through R with some kind of request with my profile and password? – Phelk Nov 10 '15 at 15:34
  • Yes, see http://stackoverflow.com/questions/24723606/scrape-password-protected-website-in-r for two approaches to performing a login browser action while data gathering with R. – Sam Firke Nov 10 '15 at 16:09

0 Answers0