Objective (using Redis):
- Cache data (as an object: example
dict
ofstring/pandas dataframe
) by date - Retrieve data between date range (say 20101010 - 20151231)
- Combine attributes of data between this date range (I'd like Redis to do this for me than combine it in python logic)
Example:
20101010: { "a" : dataframe_A1, "b" : "dataframe_B1 },
20101011: { "a" : dataframe_A2, "b" : "dataframe_B2 }
Output:
{"a" : dataframe_A1 concatenated with dataframe_A2, "b": dataframe_B1 concatenated with dataframe_B2}
One solution I've found is to use ZADD/ZRANGEBYSCORE
technique, where I create an index
on date in YYYYMMDD
format, giving it a natural set
ordering. This makes it trivial to retrieve data in a range (example: 20101010 - 20151231)
(using: https://redis.io/topics/indexes)
Is there a way retrieve data in a combined format from Redis efficiently (#3) - see my example output above.
What sequence of operations/commands would most efficiently get me this information?
Note:
- The dict above is just an example. It can be an object as well, with a way to combine its attributes.
- Dataframes can be stored using
to_msgpack
format into Redis(How to set/get pandas.DataFrame to/from redis?)
Thanks