URL gets truncated with httr::GET vs xmlParse

Question

I am trying to request an XML document with two different methods (xmlParse and httr::GET) and expect the response to be the same. The response I get with xmlParse is what I expect but with httr::GET my request URL gets truncated at some point.

An example:

require(httr)
require(XML)
require(rvest)

term <- "alopecia areata"
request <- paste0("http://eutils.ncbi.nlm.nih.gov/entrez/eutils/egquery.fcgi?term=",term)  

#requesting URL with XML
xml_response <- xmlParse(request)

xml_response %>%
        xml_nodes(xpath = "//Result/Term") %>%
        xml_text

This returns, as it should

[1] "alopecia areata"

Now for httr

httr_response <- GET(request)
httr_content <- content(httr_response)

httr_content %>%
        xml_nodes(xpath = "//Result/Term") %>%
        xml_text

This returns

[1] "alopecia"

What's interesting: if we check the httr_response element for the requested URL, it's correct. Only the response is wrong.

> httr_response$request$opts$url

[1] "http://eutils.ncbi.nlm.nih.gov/entrez/eutils/egquery.fcgi?term=alopecia areata"

> httr_response$url

[1] "http://eutils.ncbi.nlm.nih.gov/gquery?term=alopecia&retmode=xml"

So at some point my query term got truncated. If the whole request is put into a browser by hand, it behaves as expected.

Any suggestions how to resolve this would be would be greatly appreciated.

You know `XML` has XPath functions and `xmlValue()`, right? You don't really need three packages to get the xml text — Rich Scriven, Feb 28 '15 at 13:18
Yeah, but I am using rvest functions in the rest of my code and are used to it. — Christian Traupel, Mar 01 '15 at 13:56

score 5 · Accepted Answer · edited May 23 '17 at 11:59

5

You can try replacing the space in your URL by a+ to prevent it from being truncated:

httr_response <- GET(gsub(" ","+",request))
httr_content <- content(httr_response)

httr_content %>%
        xml_nodes(xpath = "//Result/Term") %>%
        xml_text 

#[1] "alopecia areata"

More info about spaces and URLs here

edited May 23 '17 at 11:59

Community

1
1

answered Feb 28 '15 at 11:58

NicE

21,165
3
51
68

URL gets truncated with httr::GET vs xmlParse

1 Answers1