2

I'm working with the rnoaa() package to get some historical weather data and am running into problems with retrieving data that says is available, but will not return.

In order for this reproducible example to work you will first need a token from http://www.ncdc.noaa.gov/cdo-web/token

Setup:

options(noaakey = "KEY_EMAILED_TO_YOU")
library(rnoaa)

Check what type of data is available:

ncdc_datatypes(stationid = "GHCND:US009052008", datasetid='GHCND')

Output:

$meta
  offset count limit
1      1     4    25

$data
Source: local data frame [4 x 5]

     mindate    maxdate                                      name datacoverage    id
       (chr)      (chr)                                     (chr)        (int) (chr)
1 1781-01-01 2015-10-30              Precipitation (tenths of mm)            1  PRCP
2 1857-01-18 2015-10-29                           Snow depth (mm)            1  SNWD
3 1763-01-01 2015-10-30 Maximum temperature (tenths of degrees C)            1  TMAX
4 1763-01-01 2015-10-30 Minimum temperature (tenths of degrees C)            1  TMIN

attr(,"class")
[1] "ncdc_datatypes"
ncdc(stationid = "GHCND:US009052008", datasetid='GHCND', datatypeid = 'PRCP', startdate = "1900-01-01", enddate = "1900-12-30")

Notice that the minimum data available for PRCP is 1781. So Let me try and pull data from just the year 1900 as it should be available.

Try and pull data from 1900:

ncdc(stationid = "GHCND:US009052008", datasetid='GHCND', datatypeid = 'PRCP', startdate = "1900-01-01", enddate = "1900-12-30")

Output: 

$meta
$meta$totalCount
NULL

$meta$pageCount
NULL

$meta$offset
NULL


$data
Source: local data frame [0 x 0]


attr(,"class")
[1] "ncdc_data"
Warning message:
In check_response(temp) : Sorry, no data found
www
  • 38,575
  • 12
  • 48
  • 84
Vedda
  • 7,066
  • 6
  • 42
  • 77
  • According [to this page](http://www1.ncdc.noaa.gov/pub/data/globaldatabank/monthly/stage2/day2month/ghcnd/INVENTORY_GHCND_monthly_stage2), the data for Sioux Falls (ENVIRON._CANADA) are available for the period 2008-2015. –  Nov 04 '15 at 01:32
  • @Pascal thanks for your reply, but I'm looking for a way to verify the available years within R since I have many stations to check. Any idea ? – Vedda Nov 04 '15 at 01:57
  • In my question I stated that I'm running into problems with data that says is available but when I que the database it says is not available. By asking for the type of data available I am asking to verify years available. Sorry if that was not clear. – Vedda Nov 04 '15 at 02:10

1 Answers1

1

One way:

sta <- "US009052008"
input <- paste0("ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/daily/all/",sta,".dly")

output <- read.fwf(input, n = -1,
                   widths = c(11,4,2,4), 
                   col.names = c("ID", "YEAR", "MONTH", "ELEMENT"))

out <- split(output, output$ELEMENT)

foo <- function(x){
  y1 <- head(x[,c("YEAR", "MONTH")], 1)
  y2 <- tail(x[,c("YEAR", "MONTH")], 1)

  paste(month.abb[y1$MONTH], y1$YEAR, "-", month.abb[y2$MONTH], y2$YEAR)
}
do.call(rbind, lapply(out, foo))
#       [,1]                 
# PRCP "Oct 2008 - Oct 2015"
# SNWD "Dec 2009 - Oct 2015"
# TMAX "Oct 2008 - Oct 2015"
# TMIN "Oct 2008 - Oct 2015"
  • Thanks for this. Boy do I feel like an idiot as I didn't know what to do with the *.dly files and the `read.fwf` works perfectly in this way. You just killed two birds with one stone, so thanks! Can you explain how you knew what to do with *.dly files and setting the `widths`? – Vedda Nov 04 '15 at 04:30
  • I got all info in the [README file](http://www1.ncdc.noaa.gov/pub/data/ghcn/daily/readme.txt), section **III. FORMAT OF DATA FILES (".dly" FILES)**. You can get the lengths by converting the number of columns. Then, for ID, columns are 1-11, so length is 11. For YEAR, columns are 12-15, so length is 4. And so on. –  Nov 04 '15 at 04:38
  • I've never run into fixed width formats. I've got some reading to do. Thanks! – Vedda Nov 04 '15 at 05:02
  • If you want to read the whole file, you need `read.fwf(input, c(11,4,2,4, rep(c(5,1,1,1), 31)))`. –  Nov 04 '15 at 05:21
  • Great thanks for the hint. That's what I was wondering how to do also. – Vedda Nov 04 '15 at 06:29