extracting Australia BOM weather data programmatically with R

Question

Here http://www.bom.gov.au/climate/data/ I can enter a substation number, say 009572; choose the variable (say Temperature) and its type (say Maximum). Clicking "get data" brings me to a page with a link "All years of data". Click it, and you got a zip file. I am aware of this questions, but here I don't have a direct link to a zip file. Can something be done to automate weather data extraction from the Australian Bureau Of Meteorology website with R?

Answer 1

I had the same question and this S.O. question was one of the first pages to come up. After further searching I found the R package Bomrang (https://github.com/ropensci/bomrang) that:

Provides functions to interface with Australian Government Bureau of Meteorology (BOM) data, fetching data and returning a tidy data frame of précis forecasts, current weather data from stations, ag information bulletins, historical weather data and downloading and importing radar or satellite imagery.

Bomrang is apart of rOpenSci and is actively developed. It has a good set of functions:

Several functions are provided by bomrang to retrieve Australian Bureau of Meteorology (BOM) data. A family of functions retrieve weather data and return tidy data frames;

get_precis_forecast(), which retrieves the précis (short) forecast;
get_current_weather(), which fetches the current weather for a given station;
get_ag_bulletin(), which retrieves the agriculture bulletin;
get_weather_bulletin(), which retrieves the BOM 0900 or 1500 bulletins;
get_coastal_forecast(), which returns coastal waters forecasts; and
get_historical(), which retrieves historical daily observations for a given station.

A second group of functions retrieve information pertaining to satellite and radar imagery,

get_available_imagery();
the satellite imagery itself, get_satellite_imagery();
get_available_radar(); and
the radar imagery itself, get_radar_imagery().

The function get_historical() seems to do what OP is needing. For example, to get the historical daily rainfall from a weather station in Sydney is as easy as:

> rain_066062 <- bomrang::get_historical(stationid = 066062,
+                         type = 'rain',
+                         meta = T)
> head(rain_066062)
$`meta`
# A tibble: 1 x 10
   site name                        lat   lon start      end        years percent AWS   ncc_obs_code
  <int> <chr>                     <dbl> <dbl> <date>     <date>     <dbl>   <int> <chr> <chr>       
1 66062 SYDNEY (OBSERVATORY HILL) -33.9  151. 1858-07-01 2018-11-01  160.     100 Y     136         

$historical_data
      Product_code Station_number Year Month Day Rainfall Period Quality
1       IDCJAC0009          66062 1858     1   1       NA     NA        
2       IDCJAC0009          66062 1858     1   2       NA     NA        
3       IDCJAC0009          66062 1858     1   3       NA     NA        
4       IDCJAC0009          66062 1858     1   4       NA     NA        
5       IDCJAC0009          66062 1858     1   5       NA     NA      
<<SNIP>>

Another nice feature is if you have the longitude and latitude of a place of interest, get_historical() will find the nearest weather station to that location.

To install from CRAN:

install.packages("bomrang")

Or install the development version from Github:

if (!require("remotes")) {
  install.packages("remotes", repos = "http://cran.rstudio.com/")
  library("remotes")
}

install_github("ropensci/bomrang", build_vignettes = TRUE)

Answer 2

Here's the code that I have done to download instantly and it also resolves your p_c problem. You can improve the function if you want and post.

#daily code = 136
#monthy code = 139

bomdata<- function(station,code){
for(i in 1: length(station)){
p.url<-paste("http://www.bom.gov.au/jsp/ncc/cdio/weatherData/av?p_stn_num=",station[i],"&p_display_type=availableYears&p_nccObsCode=",code,sep ="")
download.file(p.url,"test.txt")
filelist <- list.files(pattern = ".txt")
foo<- file(filelist,"r")
text<- suppressWarnings(readLines(foo))
close(foo)
l<- regexpr(":",text[1])
m<- unlist(gregexpr(",", text[1], perl = TRUE))
pc<- substr(text[1],l[[1]]+1,l[[1]]+(m[2]-(l[[1]]+1)))
url<-paste("http://www.bom.gov.au/jsp/ncc/cdio/weatherData/av?p_display_type=dailyZippedDataFile&p_stn_num=",station[i],"&p_c=",pc,"&p_nccObsCode=",code,"&p_startYear=2013", sep ="")
suppressWarnings(download.file(url,paste(station[i],".zip",sep= ""), mode = "wb"))
unlink("test.txt")
 }
}

Example

bomdata(073137,136)

Answer 3

You can try this, it is a code sequence used by metvurst package. metvurst

## SET URL FOR DATA DOWNLOAD
url <- "http://www.bom.gov.au/ntc/IDO70004/IDO70004_"

## YEARS TO BE DOWNLOADED
yr <- 1993:2012

## READ DATA FOR ALL YEARS FROM URL INTO LIST
fijilst <- lapply(seq(yr), function(i) {
read.csv(paste(url, yr[i], ".csv", sep = ""), na.strings = c(-9999, 999))
})

Answer 4

While I still can't see how to do this with download.file(), the following almost does the job provided Chrome's "Ask where to save each file before downloading" is unticked.

system(paste('"C:/Documents and Settings/UserName/Local Settings/Application Data/Google/Chrome/Application/chrome.exe"',
         '-url http://www.bom.gov.au/jsp/ncc/cdio/weatherData/av?p_display_type=dailyZippedDataFile&p_stn_num=009572&p_c=-18465084&p_nccObsCode=136'), wait = FALSE)

Then I could use paste0() and loop through various station numbers if I knew what p_c=-18465084 means and how it changes from station to station.