Homogenize time series type, frequency and length

Question

I am trying to bind together a number of variables that I created previously. Nevertheless, the starting and ending dates of the time-series are not equal amongst the variables. The way I am trying to do so is via:

data.start <- c(1960,1)
data.end <- c(2018,2)
data.out <- window(cbind(gdp.log.ger, interest.ger, inflation, inflation.exp.ger),start = data.start, 
                   end = data.end)
write.table(data.out,file = 'InputData/rstar.data.ge.csv', sep = ',',
            col.names = c("gdp.log","inflation","inflation.expectations", "interest"),
            quote = FALSE, na = '.', row.names = FALSE)

The error that I receive:

Error in window.default(cbind(gdp.log.ger, interest.ger, inflation, inflation.exp.ger),  : 
  'start' cannot be after 'end'
In addition: Warning messages:
1: In cbind(gdp.log.ger, interest.ger, inflation, inflation.exp.ger) :
  number of rows of result is not a multiple of vector length (arg 1)
2: In window.default(cbind(gdp.log.ger, interest.ger, inflation, inflation.exp.ger),  :
  'end' value not changed

Could this be due to the fact that start and end dates are not equal amongst the different time-series? Note that date.start and date.end have been set to the earliest and latest observations.

PS. In order to reproduce the problem:

#------------------------------------------------------------------------------#
# File:        prepare.rstar.data.ger.R
#
# Description: This file prepares the data for Germany to use in the    
#              HLW methodology.
#------------------------------------------------------------------------------#
setwd("/Users/seanbagcik/Dropbox/Master Thesis (2017 - 2018)/R-Code") #set working directory

rm(list = ls()) # clear workspace

if (!require("tis")) {install.packages("tis"); library('tis')} # Load time series library

if (!require("seasonal")) {install.packages("seasonal"); library('seasonal')}
Sys.setenv(X13_PATH = "/Library/Frameworks/R.framework/Versions/3.3/Resources/library/x13binary/bin")

# library('forecast') # for seasonal adjustment
# install.packages("forecast"); 

#------------------------------------------------------------------------------#
# Import raw data: GDP
#------------------------------------------------------------------------------#
gdp.start <- c(1991,1) # type "double"
gdp.end   <- c(2017,4)

gdp.raw <- "rawData/germany_gdp.csv"
gdp.table <- read.table(gdp.raw, skip = 1, header = F, sep = ',', stringsAsFactors = F)
gdp.ger <- ts(gdp.table[,2], start = gdp.start, frequency = 4) # time-series representation

#------------------------------------------------------------------------------#
# Import raw data: inflation
#------------------------------------------------------------------------------#
inflation.start <- c(1960,1)
inflation.end <- c(2018,1)

inflation.raw <- "rawData/germany_inflation.csv"
inflation.table <- read.table(inflation.raw, skip = 1, header = F, sep = ',', stringsAsFactors = F)
inflation.ger <- ts(inflation.table[,2], start = inflation.start, frequency = 4)

inflation.seasadj.ger <- final(seas(as.ts(naWindow(inflation.ger),freq=4))) # seasonal adjustment
inflation.seasadj.ger <- as.tis(cpi,start=inflation.start,tif='quarterly')

# Measure inflation expectations: 4-quarter moving average of past inflation:
inflation.exp.ger <- (inflation.seasadj + Lag(inflation.seasadj, k=1) + Lag(inflation.seasadj, k=2) +
                             Lag(inflation.seasadj, k=3))/4 

#------------------------------------------------------------------------------#
# inflation.fit <- auto.arima(inflation, ic = 'aic') # fit ARIMA model
# plot(forecast(inflation.fit,h=20)) # forecasting
# inflation.seasadj <- seasadj(decompose(inflation.fit, 'multiplicative'))
# inflation.ge <- 400*log(cpi/Lag(cpi, k=1)) # create annual inflation series
#------------------------------------------------------------------------------#

#------------------------------------------------------------------------------#
# Import raw data: short-term nominal interest rate
#------------------------------------------------------------------------------#  
interest.start <- c(1960,2)
interest.end <- c(2018,2)

interest.raw <- 'rawData/germany_interest.csv'
interest.table <- read.table(interest.raw, skip = 1, header = F, sep = ',', stringsAsFactors = F)
interest.m <- ts(interest.table[,2], start = interest.start, frequency = 12) # monthly time-series

interest <- convert(interest.m, tif ="quarterly", observed ="averaged") # monthly to quaterly frequency
interest <- final(seas(as.ts(naWindow(interest),freq=4))) # seasonal adjustment
interest <- as.tis(interest,start=interest.start,tif='quarterly')

interest.ger <- 100*((1+interest/36000)^365 -1) #  365-day annualized basis

#------------------------------------------------------------------------------#
# Prepare Data
#------------------------------------------------------------------------------#

# Take log of real GDP
gdp.log.ger <- log(gdp.ger)

#------------------------------------------------------------------------------#
# Output Data
#------------------------------------------------------------------------------#
data.start <- c(1960,1)
data.end <- c(2018,2)
data.out <- window(cbind(gdp.log.ger, inflation.seasadj.ger, inflation.exp.ger, interest.ger),
                   start = data.start, end = data.end)
write.table(data.out,file = 'InputData/rstar.data.ge.csv', sep = ',',
            col.names = c("gdp.log","inflation","inflation.expectations", "interest"),
            quote = FALSE, na = '.', row.names = FALSE)

With data sets: R-Data

@SirSaleh Original variable is the GDP of Germany named gdp.ger, then the log() is taken. — Sean, May 16 '18 at 12:50
It's hard to say without having `gdp.log.ger`, `interest.ger`, `inflation` and `inflation.exp.ger`. — AkselA, May 16 '18 at 12:56
@AkselA I've edited the question. Does this provide you with enough information? — Sean, May 16 '18 at 13:00
Not quite. I'd like to have the actual objects, or a subset that still produces the same error. I.e a [mcve] — AkselA, May 16 '18 at 13:01
@AkselA Then I have to somehow link you to the material. I was hoping that by reading the error you would know what could be the issue. — Sean, May 16 '18 at 13:06
Maybe someone can, but your chances of getting help greatly increases if you can make so we can reproduce your error on our end, and debug from there. If you're uneasy about publishing sensitive data, there are [ways to get around that](https://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducible-example/45763480?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa). — AkselA, May 16 '18 at 13:10
@AkselA Thanks. Please give me some time to read that post and I'll come back. Can I still receive an answer if I leave this post unanswered for a couple of hours? — Sean, May 16 '18 at 13:23
I'll stay around. Can't tell how many others will give it a look-in, but probably a few. Questions tend to stay on the [r] front page for at least a couple of hours. — AkselA, May 16 '18 at 13:27
There seems to be a small issue. The file `germany_gdp.csv` is 0 bytes. — AkselA, May 16 '18 at 14:35
Are you using R 3.3 by any chance. I'm having trouble with `seasonal`, it doesn't seem to have been updated for R 3.4. — AkselA, May 16 '18 at 15:12
If you could `save(gdp.log.ger, interest.ger, inflation, inflation.exp.ger, file="data_ger.RData")` and upload that file, I won't have to deal with `seasonal`. — AkselA, May 16 '18 at 15:13
@AkselA I am running it on a Mac. But I had issues as wel, before updating R and RStudio to its latests version. Excellent, link to file is now updated within the question — Sean, May 16 '18 at 15:20
@AkselA I might have found the problem. The first problem is that the GDP time-series was not in the same format as the rest; I fixed this now. Now the error has indeed changed, indicating that the time-series have different frequencies. So in order to solve this, I will probably need to cut out observations such that the starting and ending dates match, correct? — Sean, May 16 '18 at 15:28

AkselA · Accepted Answer · 2018-05-17T00:22:23.447

1

The core issue, as you found out, is that the four time series' are of quite different types, frequencies and lengths. So the solution was to homogenize, but that was a bit more work than I had anticipated. Normally these things can be automated more.

I opted to turn the quarterly time series' into monthly, instead of the other way around. As it is they're only padded with NA, but spline/linear/locf interpolation is quite straight forward.

Edit: With some added tinkering I managed to simplify things quite a bit

library(tis)
library(zoo)
library(xts)
library(devtools)

source_gist("https://gist.github.com/AkselA/942097c99bfa22ddc2e3d68d8a198ab8",
  filename="data_ger.r")

# homogenize data types (all zoo yearmon)
gdp.log.ger.z <- zoo(gdp.log.ger)
index(gdp.log.ger.z) <- as.yearmon(index(gdp.log.ger.z))
inflation.seasadj.ger.z <- as.zooreg(inflation.seasadj.ger, class="yearmon")
inflation.exp.ger.z <- as.zooreg(inflation.exp.ger, class="yearmon")
interest.ger.z <- as.zooreg(interest.ger, class="yearmon")

# quick and dirty merge, brings everything to monthly
mrg <- merge(gdp.log.ger.z, inflation.seasadj.ger.z, 
             inflation.exp.ger.z, interest.ger.z)
mrg <- na.approx(mrg) 
colnames(mrg) <- c("gdp.log", "inflation", "inflation.expectations", "interest")

# aggregate to quarterly
mrg.q <- aggregate(mrg, by=yearqtr, FUN=mean)
rownames(mrg.q) <- NULL

# crop all NA at beginning and end
be <- max(apply(mrg.q, 2, function(x) min(which(!is.na(x)))))
en <- min(apply(mrg.q, 2, function(x) max(which(!is.na(x)))))
mrg.q <- mrg.q[be:en,]

# write csv
write.csv(ll.z, file="data.csv", quote=FALSE, na=".", row.names=FALSE)

# plot
e <- local({ 
   mtext <- function(...) graphics::mtext(..., cex = 0.8) 
   environment(plot.zoo) <-  environment() 
}) 
with(e, plot.zoo)(mrg.q, oma=c(2, 0, 2, 0), cex.axis=0.8)

edited May 17 '18 at 00:22

answered May 16 '18 at 17:53

AkselA

8,153
2
21
34

Amazing. I am surprised that it was such a complicated problem. Nevertheless, the HLW methodology, the one I am supposed to use, takes in quarterly data. All variables except interest were downloaded as quarterly data. So now I have a .csv file that is almost perfect! How can I fix that last step. – Sean May 16 '18 at 18:26
I'd take a look at `?aggregate.zoo`. – AkselA May 16 '18 at 18:32
Part of the complication was those `tis` time series. And the mix of monthly and quarterly data. Would probably have been easier to covert to quarterly straight away, if that's what you need. – AkselA May 16 '18 at 18:38
Interest was converted via: interest.m <- ts(interest.table[,2], start = interest.start, frequency = 12); interest <- convert(interest.m, tif ="quarterly", observed ="averaged") – Sean May 16 '18 at 18:39
Would it be a solution as well if I replace all as.tis() for ts() instead of mixing those within the code? – Sean May 16 '18 at 18:40
I did just that and it delivered me with a .csv file as wel. But with gaps in it. – Sean May 16 '18 at 18:45
I'm more comfortable working within the `zoo` framework. It's flexible, well featured and documented. Straight `ts` is a bit limited, but works well together with `zoo` and `xts`. `tis` I'm just not very familiar with. – AkselA May 16 '18 at 18:47
Also, if you [mark the answer as accepted](https://meta.stackexchange.com/questions/5234/how-does-accepting-an-answer-work?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa), you'll let others know that the problem has been solved – AkselA May 16 '18 at 18:51
`as.zoo(gdp.log.ger)` would be enough to convert it to zoo. Also there is a `tis` method for `as.zooreg` so `as.zooreg(inflation.seasadj.ger, class = "yearmon")` would work. Similarly for the others. – G. Grothendieck May 16 '18 at 22:25
@G.Grothendieck: I thought so as well, but I couldn't make it work. – AkselA May 16 '18 at 22:33
You must have tried something else since `identical(as.zoo(gdp.log.ger), gdp.log.ger.z)` is TRUE. – G. Grothendieck May 16 '18 at 22:40
@G.Grothendieck: I tried `as.zoo()` and `zoo()`, but I don't think I tried `as.zooreg()` on the `tis` objects. – AkselA May 16 '18 at 22:44

score 0 · Answer 2 · answered May 16 '18 at 18:48

This appeared to work:

#------------------------------------------------------------------------------#
# File:        prepare.rstar.data.ger.R
#
# Description: This file prepares the data for Germany to use in the    
#              HLW methodology.
#------------------------------------------------------------------------------#
setwd("/Users/seanbagcik/Dropbox/Master Thesis (2017 - 2018)/R-Code") #set working directory

rm(list = ls()) # clear workspace

if (!require("tis")) {install.packages("tis"); library('tis')} # Load time series library

Sys.setenv(X13_PATH = "/Library/Frameworks/R.framework/Versions/3.3/Resources/library/x13binary/bin")
if (!require("seasonal")) {install.packages("seasonal"); library('seasonal')}

# library('forecast')
# install.packages("forecast"); 

#------------------------------------------------------------------------------#
# Import raw data: GDP
#------------------------------------------------------------------------------#
gdp.start <- c(1991,1) # type "double"
gdp.end   <- c(2017,4)

gdp.raw <- "rawData/germany_gdp.csv"
gdp.table <- read.table(gdp.raw, skip = 1, header = F, sep = ',', stringsAsFactors = F)
gdp.ger <- ts(gdp.table[,2], start = gdp.start, frequency = 4) # time-series representation

#------------------------------------------------------------------------------#
# Import raw data: inflation
#------------------------------------------------------------------------------#
inflation.start <- c(1960,1)
inflation.end <- c(2018,1)

inflation.raw <- "rawData/germany_inflation.csv"
inflation.table <- read.table(inflation.raw, skip = 1, header = F, sep = ',', stringsAsFactors = F)
inflation.ger <- ts(inflation.table[,2], start = inflation.start, frequency = 4)

inflation.seasadj.ger <- final(seas(as.ts(naWindow(inflation.ger),freq=4))) # seasonal adjustment
inflation.seasadj.ger <- ts(inflation.seasadj.ger, start = inflation.start, frequency = 4)

# Measure inflation expectations: 4-quarter moving average of past inflation:
inflation.exp.ger <- (inflation.seasadj.ger + Lag(inflation.seasadj.ger, k=1) + 
                        Lag(inflation.seasadj.ger, k=2) + Lag(inflation.seasadj.ger, k=3))/4 

#------------------------------------------------------------------------------#
# inflation.fit <- auto.arima(inflation, ic = 'aic') # fit ARIMA model
# plot(forecast(inflation.fit,h=20)) # forecasting
# inflation.seasadj <- seasadj(decompose(inflation.fit, 'multiplicative'))
# inflation.ge <- 400*log(cpi/Lag(cpi, k=1)) # create annual inflation series
#------------------------------------------------------------------------------#

#------------------------------------------------------------------------------#
# Import raw data: short-term nominal interest rate
#------------------------------------------------------------------------------#  
interest.start <- c(1960,2)
interest.end <- c(2018,2)

interest.raw <- 'rawData/germany_interest.csv'
interest.table <- read.table(interest.raw, skip = 1, header = F, sep = ',', stringsAsFactors = F)
interest.m <- ts(interest.table[,2], start = interest.start, frequency = 12) # monthly time-series

interest <- convert(interest.m, tif ="quarterly", observed ="averaged") # monthly to quaterly frequency
interest <- final(seas(as.ts(naWindow(interest),freq=4))) # seasonal adjustment
interest <- ts(interest, start = interest.start, frequency = 4)

interest.ger <- 100*((1+interest/36000)^365 -1) #  365-day annualized basis

#------------------------------------------------------------------------------#
# Prepare Data
#------------------------------------------------------------------------------#

# Take log of real GDP
gdp.log.ger <- log(gdp.ger)

#------------------------------------------------------------------------------#
# Output Data
#------------------------------------------------------------------------------#
# save(gdp.log.ger, inflation.seasadj.ger, inflation.exp.ger, interest.ger, file="data_ger.RData")

data.start <- c(1960,1)
data.end <- c(2018,2)
data.out <- window(cbind(gdp.log.ger, inflation.seasadj.ger, inflation.exp.ger, interest.ger),
                   start = data.start, end = data.end)
write.table(data.out,file = 'InputData/rstar.data.ge.csv', sep = ',',
            col.names = c("gdp.log","inflation","inflation.expectations", "interest"),
           quote = FALSE, na = '.', row.names = FALSE)

DATA

How can I check if the seasonal adjustment worked, such that inflation.seasadj.ger is an inflation time-series without seasonality? — Sean, May 16 '18 at 18:52
That sounds like something you should open a new question for. — AkselA, May 16 '18 at 18:59

Homogenize time series type, frequency and length

2 Answers2