0

I am trying to download an XML file of journal article records and create a dataset for further interrogation in R. I'm completely new to XML and quite novice at R. I cobbled together some code using bits of code from 2 sources: GoogleScholarXScraper and Extracting records from pubMed

library(RCurl)
library(XML)
library(stringr)

#Search terms
SearchString<-"cancer+small+cell+non+lung+survival+plastic"
mySearch<-str_c("http://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=pubmed&term=",SearchString,"&usehistory=y",sep="",collapse=NULL)

#Seach
pub.esearch<-getURL(mySearch)

#Extract QueryKey and WebEnv
pub.esearch<-xmlTreeParse(pub.esearch,asText=TRUE)
key<-as.numeric(xmlValue(pub.esearch[["doc"]][["eSearchResult"]][["QueryKey"]]))
env<-xmlValue(pub.esearch[["doc"]][["eSearchResult"]][["WebEnv"]])

#Fetch Records
myFetch<-str_c("http://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=pubmed&WebEnv=",env,"&retmode=xml&query_key=",key)
pub.efetch<-getURL(myFetch)
myxml<-xmlTreeParse(pub.efetch,asText=TRUE,useInternalNodes=TRUE)

#Create dataset of article characteristics #This doesn't work
pub.data<-NULL
pub.data<-data.frame(
  journal <- xpathSApply(myxml,"//PubmedArticle/MedlineCitation/MedlineJournalInfo/MedlineTA", xmlValue),
  abstract<- xpathSApply(myxml,"//PubmedArticle/MedlineCitation/Article/Abstract/AbstractText",xmlValue),
  affiliation<-xpathSApply(myxml,"//PubmedArticle/MedlineCitation/Article/Affiliation", xmlValue),
  year<-xpathSApply(myxml,"//PubmedArticle/MedlineCitation/Article/Journal/JournalIssue/PubDate/Year", xmlValue)
  ,stringsAsFactors=FALSE)

The main problem I seem to have is that my returned XML file is not completely uniformly structured. For example, some references have a node structure like this:

- <Abstract>
<AbstractText>The Wilms' tumor gene... </AbstractText>

Whilst some have labels and are like this

- <Abstract>
<AbstractText Label="BACKGROUND &#38; AIMS" NlmCategory="OBJECTIVE">Some background text.</AbstractText>
<AbstractText Label="METHODS" NlmCategory="METHODS"> Some text on methods.</AbstractText>

When I extract the 'AbstactText' I am hoping to get 24 rows of data back (there are 24 records when I run this made up search today), but xpathSApply returns all labels within 'AbstactText' as individual elements of my dataframe. Is there a way to collapse the XML structure in this instance/Ignore the labels? Is there a way to make xpathSApply return 'NA' when nothing is found at end of a path? I am aware of xmlToDataFrame, which sounds like it should fit the bill, but whenever I try to use this it doesn't seem to give me anything sensible.

Thanks for your help

DavidT85
  • 123
  • 1
  • 1
  • 5

1 Answers1

1

I am unsure as to which you want however:

xpathSApply(myxml,"//*/AbstractText[@Label]")

will get the nodes with labels (keeping all attributes etc).

xpathSApply(myxml,"//*/AbstractText[not(@Label)]",xmlValue)

will get the nodes without labels.

EDIT:

test<-xpathApply(myxml,"//*/Abstract",xmlValue)

> length(test)
[1] 24

may give you what you want

EDIT:

to get affiliation, year etc padded with NA's

dumfun<-function(x,xstr){
res<-xpathSApply(x,xstr,xmlValue)
if(length(res)==0){
out<-NA
}else{
out<-res
}
out
}

xpathSApply(myxml,"//*/Article",dumfun,xstr='./Affiliation')
xpathSApply(myxml,"//*/Article",dumfun,xstr='./Journal/JournalIssue/PubDate/Year')
shhhhimhuntingrabbits
  • 7,397
  • 2
  • 23
  • 23
  • Thank you, that work great. I'm still unable to form the dataset becuase for some entries some values are unavailable, e.g. 2 records don't report an affiliation. any thoughts on how to get 'NA' returned? – DavidT85 Jul 26 '12 at 11:32