Running a loop on multiple xml files in one folder in R

Question

I am trying to run this script as a loop function as I have over 200 files in a folder and I am trying to produce one CSV file at the end listing all of the data that I need to extract.

I have tried various ways of running this in a loop e.g. In R, how to extracting two values from XML file, looping over 5603 files and write to table

Whenever I do try these different options I get errors like:

Error: XML content does not seem to be XML or permission denied.

However, when I run the code selecting just one file it runs fine. These errors only seems to occur when I try convert it into a loop function for multiple files in a single folder.

Here is the original code used for a single file:

doc<-xmlParse("//file/path/32460004.xml")
xmldf <- xmlToDataFrame(nodes = getNodeSet(doc, "//BatRecord"))

df1 <- data.frame(xmldf)
df1 <- separate(df1, xmldf.DateTime, into = c("Date", "Time"), sep = " ")
df1$Lat <- substr(xmldf$GPS,4,12)
df1$Long <- substr(xmldf$GPS,13,25)
df_final <- data.frame(df1$xmldf.Filename, df1$Date, df1$Time, df1$xmldf.Duration, df1$xmldf.Temperature, df1$Lat, df1$Long)
colnames(df_final) <- c("Filename", "Date", "Time", "Call Duration", "Temperature", "Lat", "Long")

write.csv(df_final, "//file/path/test_file.csv")

Here is a link to some example files:

https://drive.google.com/drive/folders/1ZvmOEWhzlWHRl2GxZrbYY9y7YSZ5j9Fj?usp=sharing

Any help is appreciated.

Here is my version details:

platform       x86_64-w64-mingw32          
arch           x86_64                      
os             mingw32                     
system         x86_64, mingw32             
status                                     
major          3                           
minor          6.3                         
year           2020                        
month          02                          
day            29                          
svn rev        77875                       
language       R                           
version.string R version 3.6.3 (2020-02-29)
nickname       Holding the Windsock

Rentrop · Accepted Answer · 2020-04-10T14:44:02.020

This should work using tidyverse and xml2.

require(tidyverse)
require(xml2)

### Put all your xml files in a vector
my_files <- list.files("path/to/your/xml/files", full.names = TRUE)

### Read function to transform them to tibble (similar to data.frame)
read_my_xml <- function(x, path = "//BatRecord") {
  tmp <- read_xml(x) # read the xml file
  tmp <- tmp %>% 
    xml_find_first(path) %>% # select the //BatRecord node
    xml_children # select all children of that node

  # this extracts the text of all children 
  # aka the text between the > TEXT </ Tags
  out <- tmp %>% xml_text 
  # Takes the names of the tags <NAME> ... </NAME>
  names(out) <- tmp %>% xml_name
  # Turns out to tibble - see https://stackoverflow.com/q/40036207/3301344
  bind_rows(out)
}

### Read the files as data

dat <- map_df(my_files, read_my_xml) # map_df is similar to a loop + binding it to one tibble

### To the transformation

dat %>% 
  separate(DateTime, into = c("Date", "Time"), sep = " ") %>% 
  mutate(Lat = substr(GPS,4,12), Long = substr(GPS,13,25)) %>% 
  write_csv("wherever/you/want/file.txt")

this is great. it works. thank you. is it possible to give an explanation as to what the code for the function to transform them to tibble is actually doing? — DFinch, Apr 10 '20 at 14:37
What if the XML file also contains grandchildren, great-grandchildren, and more? — classy_BLINK, Jan 06 '22 at 06:31

Running a loop on multiple xml files in one folder in R

1 Answers1