7

I can not resolve why error in simple creation of xts object

xts(rep(0, NROW(TICK.NYSE)), order.by = index(TICK.NYSE))
Error in xts(rep(0, NROW(TICK.NYSE)), order.by = index(TICK.NYSE)) : 
  order.by requires an appropriate time-based object

appeared while this was working perfectly 14 days ago when I last used the same code (since then the only difference is that TICK.NYSE grow in length since data was added since then).

More details below:

> Sys.getenv("TZ")
[1] "EST"
> tail(xts(rep(0, NROW(TICK.NYSE)), order.by = index(TICK.NYSE)))
Error in xts(rep(0, NROW(TICK.NYSE)), order.by = index(TICK.NYSE)) : 
  order.by requires an appropriate time-based object
> class(index(TICK.NYSE["T09:30/T09:31"]))
[1] "POSIXct"
> tail(xts(rep(0, NROW(tail(TICK.NYSE))), order.by = index(tail(TICK.NYSE))))
Error in xts(rep(0, NROW(tail(TICK.NYSE))), order.by = index(tail(TICK.NYSE))) : 
  order.by requires an appropriate time-based object
> tail(TICK.NYSE)
                    TICK-NYSE.Open TICK-NYSE.High TICK-NYSE.Low TICK-NYSE.Close
2012-03-15 14:54:00           -278            -89          -299             -89
2012-03-15 14:55:00            -89            427           -89             201
2012-03-15 14:56:00            201            318            30             234
2012-03-15 14:57:00            234            242          -222             -64
2012-03-15 14:58:00            -64            346           -82             346
2012-03-15 14:59:00            346            525            36             525
                    TICK-NYSE.Volume TICK-NYSE.WAP TICK-NYSE.hasGaps
2012-03-15 14:54:00                0             0                 0
2012-03-15 14:55:00                0             0                 0
2012-03-15 14:56:00                0             0                 0
2012-03-15 14:57:00                0             0                 0
2012-03-15 14:58:00                0             0                 0
2012-03-15 14:59:00                0             0                 0
                    TICK-NYSE.Count
2012-03-15 14:54:00              31
2012-03-15 14:55:00              31
2012-03-15 14:56:00              31
2012-03-15 14:57:00              31
2012-03-15 14:58:00              29
2012-03-15 14:59:00              30
> str(TICK.NYSE)
An ‘xts’ object from 2011-01-18 09:30:00 to 2012-03-15 14:59:00 containing:
  Data: num [1:114090, 1:8] -5 -144 24 -148 -184 -77 20 121 111 -60 ...
 - attr(*, "dimnames")=List of 2
  ..$ : NULL
  ..$ : chr [1:8] "TICK-NYSE.Open" "TICK-NYSE.High" "TICK-NYSE.Low" "TICK-NYSE.Close"     ...
  Indexed by objects of class: [POSIXct,POSIXt] TZ: 
  xts Attributes:  
List of 4
 $ from   : chr "20110119  23:59:59"
 $ to     : chr "20110124  23:59:59"
 $ src    : chr "IB"
 $ updated: POSIXct[1:1], format: "2012-01-19 02:34:52"
> str(index(TICK.NYSE))
Class 'POSIXct'  atomic [1:114090] 1.3e+09 1.3e+09 1.3e+09 1.3e+09 1.3e+09 ...
  ..- attr(*, "tzone")= chr ""
  ..- attr(*, "tclass")= chr [1:2] "POSIXct" "POSIXt"
> Sys.getenv("TZ")
[1] "EST"
> tail(index(TICK.NYSE))
[1] "2012-03-15 14:54:00 EST" "2012-03-15 14:55:00 EST"
[3] "2012-03-15 14:56:00 EST" "2012-03-15 14:57:00 EST"
[5] "2012-03-15 14:58:00 EST" "2012-03-15 14:59:00 EST"
> head(index(TICK.NYSE))
[1] "2011-01-18 09:30:00 EST" "2011-01-18 09:31:00 EST"
[3] "2011-01-18 09:32:00 EST" "2011-01-18 09:33:00 EST"
[5] "2011-01-18 09:34:00 EST" "2011-01-18 09:35:00 EST"
> Sys.info()
                                      sysname 
                                      "Linux" 
                                      release 
                           "3.0.0-16-generic" 
                                      version 
"#28-Ubuntu SMP Fri Jan 27 17:44:39 UTC 2012" 
> sessionInfo()
R version 2.14.1 (2011-12-22)
Platform: x86_64-pc-linux-gnu (64-bit)

locale:
 [1] LC_CTYPE=en_US.UTF-8          LC_NUMERIC=C                 
 [3] LC_TIME=en_US.UTF-8           LC_COLLATE=en_US.UTF-8       
 [5] LC_MONETARY=en_US.UTF-8       LC_MESSAGES=en_US.UTF-8      
 [7] LC_PAPER=en_US.UTF-8          LC_NAME=en_US.UTF-8          
 [9] LC_ADDRESS=en_US.UTF-8        LC_TELEPHONE=en_US.UTF-8     
[11] LC_MEASUREMENT=en_US.UTF-8    LC_IDENTIFICATION=en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] lattice_0.20-0               multicore_0.1-7             
 [3] doSNOW_1.0.5                 snow_0.3-8                  
 [5] doRedis_1.0.4                rredis_1.6.3                
 [7] foreach_1.3.2                codetools_0.2-8             
 [9] iterators_1.0.5              PerformanceAnalytics_1.0.3.3
[11] quantstrat_0.6.1             blotter_0.8.4               
[13] twsInstrument_1.3-3          FinancialInstrument_0.10.6  
[15] IBrokers_0.9-6               quantmod_0.3-18             
[17] TTR_0.21-0                   xts_0.8-4                   
[19] Defaults_1.1-1               strucchange_1.4-6           
[21] sandwich_2.2-8               zoo_1.7-7                   
[23] rj_1.0.2-5                  

loaded via a namespace (and not attached):
[1] grid_2.14.1  tools_2.14.1
> dput(tail(TICK.NYSE))
structure(c(-385, -213, -42, -334, -233, -111, -121, 20, -14, 
-125, -73, 265, -583, -269, -426, -520, -443, -440, -213, -42, 
-334, -233, -111, 119, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 
0, 0, 0, 0, 0, 31, 31, 30, 30, 31, 31), class = c("xts", "zoo"
), .indexCLASS = c("POSIXct", "POSIXt"), .indexTZ = "", 
from = "20110119  23:59:59", to = "20110124  23:59:59", src = "IB", 
updated = structure(1326958492.96405, class =  c("POSIXct", 
"POSIXt")), index = structure(c(1331927640, 1331927700, 1331927760, 
1331927820, 1331927880, 1331927940), tzone = "", tclass = c("POSIXct", 
"POSIXt")), .Dim = c(6L, 8L), .Dimnames = list(NULL, c("TICK-NYSE.Open", 
"TICK-NYSE.High", "TICK-NYSE.Low", "TICK-NYSE.Close", "TICK-NYSE.Volume", 
"TICK-NYSE.WAP", "TICK-NYSE.hasGaps", "TICK-NYSE.Count")))

How is this possible? Using function xts::timeBased returns FALSE if I copy the source of the function from xtime::timeBased as timeBased_test and use it returns TRUE... Strange.

> timeBased(index(TICK.NYSE))
[1] FALSE
> 
> sapply(c("Date", "POSIXct", "chron", "dates", "times", 
+               "timeDate", "yearmon", "yearqtr", "xtime"), function(xx)     inherits(index(TICK.NYSE), 
+                   xx))
    Date  POSIXct    chron    dates    times timeDate  yearmon  yearqtr 
   FALSE     TRUE    FALSE    FALSE    FALSE    FALSE    FALSE    FALSE 
   xtime 
   FALSE 
> any(sapply(c("Date", "POSIXct", "chron", "dates", "times", 
+               "timeDate", "yearmon", "yearqtr", "xtime"), function(xx)     inherits(index(TICK.NYSE), 
+                   xx)))
[1] TRUE
> timeBased_test <- function (x) 
+ {
+   if (!any(sapply(c("Date", "POSIXct", "chron", "dates", "times", 
+                           "timeDate", "yearmon", "yearqtr",     "xtime"), function(xx) inherits(x, 
+                               xx)))) {
+       FALSE
+   }
+   else TRUE
+ }
> 
> timeBased_test(index(TICK.NYSE))
[1] TRUE

OK, so, I think I found the reason for this. In xts in file period.R I can find this piece of code:

`is.timeBased` <- `timeBased` <-
function(x) {
if (!any(sapply(c("Date", "POSIXt", "chron", "dates", "times", 
        "timeDate", "yearmon", "yearqtr", "xtime"), function(xx) inherits(x, 
        xx)))) {
        FALSE
    } else TRUE
}

And this is, I guess, actually being called inside of xts. I thought it is xtime:timeBased being called which would be fine since that one has "POSIXct":

timeBased <- function (x) 
{
    if (!any(sapply(c("Date", "POSIXct", "chron", "dates", "times", 
        "timeDate", "yearmon", "yearqtr", "xtime"), function(xx) inherits(x, 
        xx)))) {
        FALSE
    }
    else TRUE
}

What I do not understand here? Since checking inheritance of POSIXct works fine (returns TRUE) while POSIXt dos not (returns FALSE).

Samo
  • 2,065
  • 20
  • 41
  • There was a time change in the United States due to daylight saving last weekend. Maybe your data contains some invalid day/time? – flodel Mar 18 '12 at 12:11
  • Can you provide the output of `dput(tail(TICK.NYSE))`? @flodel: good guess, but the error still occurs with the last 6 observations. – Joshua Ulrich Mar 18 '12 at 13:23
  • complete shot in the dark, but does it help to use `Sys.setenv("America/New_York")` instead of `"EST"`? BTW, when I download the same data with `twsInstrument:::reqTBBOhistory`, I don't get the error. – GSee Mar 18 '12 at 16:37
  • @flodel: thanks, yes, good idea. Will investigate. Although, my wild guess would be, that if I take index (order by) from existing functioning xts object while creating new one I should be doing fine. – Samo Mar 18 '12 at 17:55
  • @josh: Thnx. Added dput. – Samo Mar 18 '12 at 17:55
  • @GSee: Garrett thnx. Changed to Sys.setenv(TZ="America/New_York") but error still exists... I got the data exactly like you described, using your great twsinstrument package twsInstrument:::reqTBBOhistory. I update data every day. – Samo Mar 18 '12 at 18:00
  • Looking at source of xts this puzzles me even more. If I understand this correctly, the error comes from here xts::xts if(!timeBased(order.by)) stop("order.by requires an appropriate time-based object") And xtime::timeBased checks inheritance timeBased <- function (x) { if (!any(sapply(c("Date", "POSIXct", "chron", "dates", "times", "timeDate", "yearmon", "yearqtr", "xtime"), function(xx) inherits(x, xx)))) { FALSE } else TRUE } since I have > class(TICK.NYSE) [1] "xts" "zoo" > class(index(TICK.NYSE)) [1] "POSIXct" – Samo Mar 18 '12 at 18:23
  • I wonder why your output from `str(index(TICK.NYSE))` looks so different than what you get when you do that on the dput of the tail. – GSee Mar 18 '12 at 18:55
  • @Samo, so you probably have a different data file for each day, right? If so, can you load one day at a time and see if the problem occurs regardless of which day(s) of data you load? – GSee Mar 19 '12 at 00:15
  • @Garrett, thnx for the idea. Yes, I have data file for each day. I have tried for several dates (not all systematically, but a sample) loading one day at a time and the problem persists. > getSymbols("TICK.NYSE", src="FI", dir=paste(baseDir, "/TRADES", sep=""), from="2012-03-16", to="2012-03-16") [1] "TICK.NYSE" > tail(xts(rep(0, NROW(TICK.NYSE)), order.by = index(TICK.NYSE))) Error in xts(rep(0, NROW(TICK.NYSE)), order.by = index(TICK.NYSE)) : order.by requires an appropriate time-based object No matter which day I load, the problem persists... – Samo Mar 19 '12 at 06:51
  • Problem still persists... It mysteriously worked after I updated my linux systems but after another restart the problem is still there. – Samo Mar 20 '12 at 07:55

1 Answers1

3

Ok, Samo, that's fixable with this

indexClass(TICK.NYSE) <- c("POSIXt", "POSIXct")

because timeBased checks to see if it inherits "POSIXt", but doesn't check for "POSIXct"

There's at least one other place that has this problem in xts. xts:::.drop.time only works if indexClass(x)[1] == "POSIXt"

GSee
  • 48,880
  • 13
  • 125
  • 145
  • Garrett, thanks. Tried it, it doesn't work unfortunately. > load(paste(baseDir, "/TRADES/TICK.NYSE/", "2011.01.18", ".TICK.NYSE.rda", sep="")) > > indexClass(TICK.NYSE) <- c("POSIXt", "POSIXct") > > tail(xts(rep(0, NROW(TICK.NYSE)), order.by=index(TICK.NYSE))) Error in xts(rep(0, NROW(TICK.NYSE)), order.by = index(TICK.NYSE)) : order.by requires an appropriate time-based object > class(index(TICK.NYSE)) [1] "POSIXct" > indexClass(TICK.NYSE) [1] "POSIXt" "POSIXct" > class(index(TICK.NYSE)) <- c("POSIXt", "POSIXct") > tail(xts(rep(0, NROW(TICK.NYSE)), order.by=index(TICK.NYSE))) Error... – Samo Mar 20 '12 at 15:54
  • Well, I've tried and tried, but I cannot recreate the error even with your exact data that you e-mailed me. I do not understand why `inherits(index(TICK.NYSE), "POSIXt")` does not return `TRUE` for you. (it does for me) – GSee Mar 20 '12 at 16:25