R: index() or index.xts() changes the values of Date of a time series, why?

Question

I would like to extract the dates from a time series obtained using getSymbols but when I used the index / index.xts function the returned dates appear to be one day earlier. I cannot understand why this behavior happens in the following code.

However, the intended behavior is to obtain a list of Date object corresponding to the one in the original time series.

Here is the code, note the last date of the time series SPY is 24 Aug 2012 but the last value from the index(SPY) call is 23 Aug 2012:

getSymbols("SPY")

tail(SPY)

    SPY.Open SPY.High SPY.Low SPY.Close SPY.Volume SPY.Adjusted
2012-08-17   142.23   142.30  141.86    142.18   90813700       142.18
2012-08-20   141.98   142.22  141.59    142.19   78255700       142.19
2012-08-21   142.54   143.09  141.45    141.76  105581100       141.76
2012-08-22   141.40   142.05  141.07    141.82  132999200       141.82
2012-08-23   141.47   141.48  140.44    140.66  111406800       140.66
2012-08-24   140.31   141.83  140.22    141.51   99431500       141.51 

tail(index(SPY))

[1] "2012-08-16" "2012-08-19" "2012-08-20" "2012-08-21" "2012-08-22" "2012-08-23"

tail(index.xts(SPY))

[1] "2012-08-16" "2012-08-19" "2012-08-20" "2012-08-21" "2012-08-22" "2012-08-23"

Thank you to everybody that could reply to my post.

Additional info on the session

>sessionInfo()
R version 2.15.1 (2012-06-22)
Platform: i386-pc-mingw32/i386 (32-bit)

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] rbenchmark_0.3.1             fGarch_2110.80.1            
 [3] fBasics_2160.81              MASS_7.3-20                 
 [5] timeSeries_2160.95           timeDate_2160.95            
 [7] tseries_0.10-29              quadprog_1.5-4              
 [9] PerformanceAnalytics_1.0.4.4 quantstrat_0.6.8            
[11] blotter_0.8.10               FinancialInstrument_0.15.2  
[13] quantmod_0.3-17              TTR_0.21-1                  
[15] Defaults_1.1-1               xts_0.8-6                   
[17] zoo_1.7-7                    lubridate_1.1.0             
[19] stringr_0.6.1                plyr_1.7.1                  
[21] XML_3.9-4.1                 

loaded via a namespace (and not attached):
 [1] colorspace_1.1-1   dichromat_1.2-4    digest_0.5.2       ggplot2_0.9.1     
 [5] grid_2.15.1        labeling_0.2       lattice_0.20-6     memoise_0.1       
 [9] munsell_0.3        proto_0.3-9.2      RColorBrewer_1.0-5 reshape2_1.2.1    
[13] scales_0.2.1       stabledist_0.6-4   tools_2.15.1     


> getDefaults(getSymbols)
NULL
> getSymbolLookup("SPY")
NULL
> showSymbols()
 SPY     GSPC      IBM      XLF      XLP      XLE      XLY      XLV      XLI 
 "yahoo"  "yahoo"  "yahoo"  "yahoo"  "yahoo"  "yahoo"  "yahoo"  "yahoo"  "yahoo" 
     XLB      XLK      XLU      IEF     AAPL      DIA     MSFT      IWM      EEM 
 "yahoo"  "yahoo"  "yahoo"  "yahoo"  "yahoo"  "yahoo"  "yahoo"  "yahoo"  "yahoo" 
     EFA      GLD      AGG      HYG      FXE      FXY      VXX      VXZ      HIG 
 "yahoo"  "yahoo"  "yahoo"  "yahoo"  "yahoo"  "yahoo"  "yahoo"  "yahoo"  "yahoo" 
    VTI      VEU      VNQ      DBC      XAU     gold     Gold STOXX50E     GOLD 
"yahoo"  "yahoo"  "yahoo"  "yahoo"  "oanda"  "oanda"  "oanda"  "yahoo"  "yahoo" 
     VIX  DEXUSEU   EURUSD  DEXKOUS    EUR=X    INR=X 
 "yahoo"   "FRED"  "oanda"   "FRED"  "yahoo"  "yahoo"

Also note that I have installed some R code from the Systematic Investor blog, systematicinvestor.wordpress.com, by using the commands

setInternet2(TRUE)
con = gzcon(url('systematicportfolio.com/sit.gz', 'rb'))
source(con) 
close(con)

SOLUTION PLUS ADDITIONAL QUESTION

GSee user found the answer (Thank you!) pointing out that in my session I have masked index.xts. So a solution is to call xts:::index.xts(SPY) instead of just index.xts(SPY) to override the masking. In fact the command

> tail(xts:::index.xts(SPY))

returns the right answer

[1] "2012-08-17" "2012-08-20" "2012-08-21" "2012-08-22" "2012-08-23" "2012-08-24" –

The answer now prompted another question: given below the code for the "masking/overriding" index.xts function (that returns the wrong answer moving the dates one day earlier):

> index.xts
function (
x           # XTS object
)
{
temp = attr(x, 'index')
class(temp)='POSIXct'   
if( attr(x, '.indexCLASS')[1] == 'Date')
    temp = as.Date(temp)
return(temp)
}

Why is this function returning the wrong results when called as tail(index.xts(SPY)) ? What is wrong with the code of this index.xts function?

Compare the two output (the first one is wrong while the second provide the right answer):

tail(index.xts(SPY))

[1] "2012-08-16" "2012-08-19" "2012-08-20" "2012-08-21" "2012-08-22" "2012-08-23"

tail(xts:::index.xts(SPY))
[1] "2012-08-17" "2012-08-20" "2012-08-21" "2012-08-22" "2012-08-23" "2012-08-24"

Thanks again for your time and attention.

I cannot reproduce. Please add the output of `sessionInfo()` to your question. Also, show the output of `getDefaults(getSymbols)` and `getSymbolLookup("SPY")` if either of them are not `NULL` and the output of `showSymbols()` — GSee, Aug 25 '12 at 18:51
Thank you GSee for replying to my message, I have updated it with the additional information about the session data — Filippo Neri, Aug 25 '12 at 19:03
It looks like you may have masked `index.xts`. What happens if you use `tail(xts:::index.xts(SPY))`? — GSee, Aug 25 '12 at 19:15
also note that I have installed some R code from the Systematic Investor blog, http://systematicinvestor.wordpress.com/, by using the command setInternet2(TRUE); con = gzcon(url('http://www.systematicportfolio.com/sit.gz', 'rb')); source(con); close(con) — Filippo Neri, Aug 25 '12 at 19:16
GSee, you have found the problem! here is the output > tail(xts:::index.xts(SPY)) [1] "2012-08-17" "2012-08-20" "2012-08-21" "2012-08-22" "2012-08-23" "2012-08-24" — Filippo Neri, Aug 25 '12 at 19:19
GSee, I have appended at the post a further question about the code of the "wrong" index.xts, if you feel like having a look. Thanks! — Filippo Neri, Aug 25 '12 at 19:40
To further confirm. The R code from the Systematic Investor blog, systematicinvestor.wordpress.com redefines the index.xts function causing the wrong answer. — Filippo Neri, Aug 25 '12 at 19:46
I'm using xts (0.9-7), and invoking the index() command seems to get the date/time objects wrong when I do not set the TZ environment variable. Explained here: http://stackoverflow.com/questions/6374874/how-to-change-the-default-timezone-in-r Changing my timezone to GMT helped me get the times right. It was '' on my Ubuntu system. In some cases, a lag of even one day can occur, and corrupt your dataset, therefore I think it's a good idea to check if you're getting the right times with xts objects. The strange thing is, the objects may display differently before/after using index or .index* fun — exa, Feb 24 '14 at 23:46

GSee · Accepted Answer · 2012-08-25T22:07:09.860

7

This issue is that you have run some code that masks the index.xts method from the xts package. In the comments of your OP, you told us you ran this code

setInternet2(TRUE)
con = gzcon(url('http://www.systematicportfolio.com/sit.gz', 'rb'))
source(con)
close(con)

After running that code, index.xts has been redefined (actually, masked) to be

> index.xts
function
(
        x                       # XTS object
)
{
        temp = attr(x, 'index')
        class(temp)='POSIXct' 

        if( attr(x, '.indexCLASS')[1] == 'Date')
                temp = as.Date(temp)
        return(temp)
}

If you compare that to the xts:::index.xts function that is being masked, you can see that it is quite different. So, you can no longer expect the same results.

If you download the code from systematicportfolio.com/sit.gz, and look at it, you'll see this comment

###############################################################################
# Fast alternative to index(x) for XTS object
# NOTE index.xts is the same name as the index function in the XTS package
###############################################################################

So, it appears that the author did this intentionally in hopes of making it faster. Unfortunately, he has also made it less robust.

This code actually gives me the same result as xts:::index.xts, so I'm not 100% certain of why you're getting different results using it. But, my guess is that it has to do with changing the class from numeric to POSIXct and then converting back to Date

Anyway, I'd probably run

rm(index.xts)

after sourcing that code to remove this "enhancement."

edited Aug 25 '12 at 22:07

answered Aug 25 '12 at 19:52

GSee

48,880
13
125
145

4

+1 and great example of why _not_ doing packages sucks. Which I tried before to tell the author of `sit.gz`, but seemingly without too much success. – Dirk Eddelbuettel Aug 25 '12 at 19:57
@DirkEddelbuettel, Several months ago, I asked nicely for that author to make a package as well. He said he would... – GSee Aug 25 '12 at 20:01
Thank you for your replies Gsee and Dirk, could you point out what is wrong with the index.xts code in the sit.gz? I have appended the code in the main post. I cannot understand what is wrong with it. – Filippo Neri Aug 25 '12 at 20:03
yes GSee I have seen your guess. Thank you. However I would like to understand exactly what is wrong with the code of index.xts (from sit.gz). Because if the problem is due to class conversions then it is a problem of R and not of sit.gz. – Filippo Neri Aug 25 '12 at 20:10
1

Simply, that is not the right way to convert `numeric` to `POSIXct`. There is an `as.POSIXct` function for that. – GSee Aug 25 '12 at 20:11
Thank you GSee. Very detailed and precise answer. – Filippo Neri Aug 25 '12 at 20:16
2

`xts:::index.xts` accounts for timezone differences, while the "faster" code does not. That's why some people can't replicate this behavior. – Joshua Ulrich Aug 25 '12 at 21:28

score 0 · Answer 2 · answered Aug 25 '12 at 18:59

It might be worth opening a clean copy of R. I get (with R 2.15.0):

> require(quantmod)
> getSymbols("SPY")
[1] "SPY"
> tail(SPY)
           SPY.Open SPY.High SPY.Low SPY.Close SPY.Volume SPY.Adjusted
2012-08-17   142.23   142.30  141.86    142.18   90813700       142.18
2012-08-20   141.98   142.22  141.59    142.19   78255700       142.19
2012-08-21   142.54   143.09  141.45    141.76  105581100       141.76
2012-08-22   141.40   142.05  141.07    141.82  132999200       141.82
2012-08-23   141.47   141.48  140.44    140.66  111406800       140.66
2012-08-24   140.31   141.83  140.22    141.51   99431500       141.51
> index(tail(SPY))
[1] "2012-08-17" "2012-08-20" "2012-08-21" "2012-08-22" "2012-08-23" "2012-08-24"
> tail(index(SPY))
[1] "2012-08-17" "2012-08-20" "2012-08-21" "2012-08-22" "2012-08-23" "2012-08-24"
> tail(index.xts(SPY))
Error in tail(index.xts(SPY)) : could not find function "index.xts"

Hi Henry, thank you for trying my code. I agree with your suggestion, but I would like to understand what happen with my code to avoid that the same problem happens again in the future. I have updated my post with data on my session, if it may be of help in understanding what caused the problem. — Filippo Neri, Aug 25 '12 at 19:13

R: index() or index.xts() changes the values of Date of a time series, why?

2 Answers2

Linked