21

I am using R to process Census data which uses really long numeric GEOIDs to identify the geographies. The issue I am facing is when writing out the processed data using write_csv (from the readr package) it is writing these GEOIDs in scientific notation. Is there a way to get around this?

Note: I am able to toggle scientific notation display on R console using by setting the scipen option to a sufficiently large value. But this setting does not seem to extend into the readr library.

Here is a toy dataset:

library(dplyr)
library(readr) # which is the package with write_csv
(tbl_df(data.frame(GEOID = seq(from=60150001022000, to=60150001022005, 1))))
Source: local data frame [6 x 1]

           GEOID
1 60150001022000
2 60150001022001
3 60150001022002
4 60150001022003
5 60150001022004
6 60150001022005

write_csv((tbl_df(data.frame(GEOID = seq(from=60150001022000, to=60150001022005, 1)))), "test.csv")

This is what I am getting currently. I am looking for a way to get the same numbers as above:

GEOID
6.02E+13
6.02E+13
6.02E+13
6.02E+13
6.02E+13
6.02E+13
IRTFM
  • 258,963
  • 21
  • 364
  • 487
sriramn
  • 2,338
  • 4
  • 35
  • 45
  • Could you prepare a small [reproducible example](http://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducible-example) to illustrate the problem? Are you sure you want to treat those as numeric values? Perhaps convert them to character/factor values? – MrFlick May 20 '15 at 05:41
  • I would like to continue using numeric type. It would be good to know how to suppress the scientific notation for file writes. – sriramn May 20 '15 at 05:49

6 Answers6

14

I'd rather suggest recoding such columns to type int, because if so write_* won't use scientific number encoding anymore. To convert all numeric columns in one pass (e.g. in case you're dealing with a counts matrix), you could do: 

require(dplyr)    
tbl_df = mutate_if(tbl_df, is.numeric, as.integer)
Holger Brandl
  • 10,634
  • 3
  • 64
  • 63
8

I wrote a pull request with a patch to improve the control of scientific notation in write_csv.

With this patch you would would have a int_use_scientific=FALSE argument in write_csv that would solve your issue. Hopefully it will be eventually merged.

zeehio
  • 4,023
  • 2
  • 34
  • 48
5

It would probably be safer to use character values:

X <- tbl_df(data.frame(GEOID = as.character(seq(from=60150001022000, to=60150001022005))))

 write_csv(X, "test.csv")

It's a bit ironic that the write_csv function does coerce some of its output to character values, but not numeric columns. Only if a column passes the is.object test will it be coerced. There does not appear to be a switch to throw that will preserve maximal precision. The write.table and its offspring write.csv functions have several switches that allow suppression of quotes and other settings that allow tailoring of output but write_csv has very little of such.

You can trick write_csv into thinking that a numeric column is something more complex and this does result in the as.character output, albeit with quotes.

 class(X[[1]])<- c("num", "numeric")
 vapply(X, is.object, logical(1))
#GEOID 
# TRUE 

 write_csv(X, "")
#[1] #"\"GEOID\"\n\"60150001022000\"\n\"60150001022001\"\n\"60150001022002\"\n\"60150001022003\"\n\"60150001022004\"\n\"60150001022005\"\n"

As a matter of best practices I do not agree with your choice of insisting that ID-variables remain numeric. There is too much violence that can be applied to that storage mode for an object. You do not need any of the arithmetic operations for an ID-variable.

IRTFM
  • 258,963
  • 21
  • 364
  • 487
2

Use bit64, it is a S3 class for vectors for 64bit Integers

library(dplyr)
library(readr)
options(digits = 22)
tbl_df <- data.frame(GEOID = seq(from=60150001022000, to=60150001022005, 1))
> tbl_df
           GEOID
1 60150001022000
2 60150001022001
3 60150001022002
4 60150001022003
5 60150001022004
6 60150001022005

library(bit64)
tbl_df$GEOID <- as.integer64(tbl_df$GEOID)
write_csv(tbl_df,'test.csv')

If you read this data again in R, it will assign the correct datatype.

dfr <- read_csv('test.csv')
> dfr
Source: local data frame [6 x 1]

           GEOID
1 60150001022000
2 60150001022001
3 60150001022002
4 60150001022003
5 60150001022004
6 60150001022005 

> str(tbl_df)
'data.frame':   6 obs. of  1 variable:  
Classes ‘tbl_df’, ‘tbl’ and 'data.frame':   6 obs. of  1 variable:
 $ GEOID: num  6.02e+13 6.02e+13 6.02e+13 6.02e+13 6.02e+13 ...

Hope this helps. I opened the csv in a text editor, the numbers had "" around them. But it still worked.

Aks
  • 932
  • 2
  • 17
  • 32
1

I would suggest you use

write.csv((tbl_df(data.frame(GEOID = seq(from=60150001022000, to=60150001022005, 1)))), "test.csv")

instead of 

write_csv((tbl_df(data.frame(GEOID = seq(from=60150001022000, to=60150001022005, 1)))), "test.csv")

If I open test.csv it opens the file in Excel. Excel turns it into scientific notation. When I right click and open with notepad it looks good and I see the original numbers without scientific notation.

MichaelVE
  • 1,304
  • 10
  • 15
  • Thanks! I am trying to migrate to the `readr` package and am looking to see how I can do it within that library. – sriramn May 20 '15 at 16:15
0
 require(dplyr)    
 tbl_df = mutate_if(tbl_df, is.numeric, as.integer)

I think it would be safer to recode those columns into string while exporting the CSV in such a case

MarioHo
  • 31
  • 1
  • 5