0

I'm trying to import a huge dataset (to do analysis later), but I only want 4 of the rows (they are categorized by utility, and I'm only interested in a few of them).

Issue is: the formatting is bizarre AND huge, and I keep getting errors about not having enough column names.

Formatting is as follows: Each row is a utility region, and each column is every hour from Summer 2015 to present along with its electricity usage, so there are a LOT of columns (the number of hours from 2015 to present x 2).

I've tried the following: 

> data<-read.table("C:\\Users\\EBA.txt",header=true,nrows=150)
Error in read.table("C:\\Users\\EBA.txt",  : 
  object 'true' not found
> data<-read.table("C:\\Users\\EBA.txt",header=TRUE,nrows=150)
Error in read.table("C:\\Users\\EBA.txt",  : 
  more columns than column names
> data<-read.table("C:\\Users\\EBA.txt",header=TRUE,sep=",")
#cancelled here because it froze my computer

> data<-read.table("C:\\Users\\EBA.txt",header=TRUE,sep=",]")
Error in scan(file, what = "", sep = sep, quote = quote, nlines = 1, quiet = TRUE,  : 
  invalid 'sep' value: must be one byte
> data<-read.table("C:\\Users\\EBA.txt",header=TRUE,sep=",")
Error in read.table("C:\\Users\\EBA.txt",  : 
  more columns than column names

The dataset I'm looking at lives here ("US Electric System Operating Data", https://www.eia.gov/opendata/bulkfiles.php"

Ultimately, I'm going to want to use the dates and electricity usage, so reading it in a usage form and converting from ISO time would be also be great - thanks!

Community
  • 1
  • 1
  • 3
    `read.table` reads tabular files, not JSON. Have a look at https://stackoverflow.com/questions/2617600/importing-data-from-a-json-file-into-r#2617823 – HubertL Aug 25 '17 at 19:52
  • Did you tag with data.table deliberately? It appears nowhere in the question body... anyway, I'll go ahead and remove it. – Frank Aug 25 '17 at 19:55
  • @HubertL I had no idea JSONs existed! Any favorite packages to manipulate them? This format is totally whacky. Thanks for the help by the way. –  Aug 25 '17 at 20:11

2 Answers2

0

Use the library data.table or the library h2o

For using the data.table library(a fastest way to load big dataset) you only have to use the command fread instead read.table

library(data.table), 
data<-fread("C:\\Users\\EBA.txt")

Another option, maybe the fastest one, is to use H2O:

library(h2o)
h2o.init(nthreads=-1)
data<-h2o.importFile("C:\\Users\\EBA.txt")

Sure that with this 2 codes you will be able to do it!

Jesus
  • 462
  • 6
  • 13
0

This ended up working for me: 

wholedata<-fromJSON(file="EBA.json")
print(wholedata)
wholedata_data_frame<-as.data.frame(wholedata)