How to apply threshold and paging on financial series data coming from Alpha Vantage API?

Question

I am using Alpha Vantage API to fetch the financial market data and store it in mongo. My mongo scheme is this:

import * as mongoose from 'mongoose'

export interface Symbol extends mongoose.Document {
    code: string,
    data_frame: object[],
    createdAt: Date,
    updatedAt: Date
}

const symbolSchema = new mongoose.Schema({
    code:{
        type: String,
        required: true,
        unique: true
    },
    data_frame:{
        type: [Object],
        default: {}
    },
    createdAt:{
        type: Date,
        default: Date.now()
    },
    updatedAt:{
        type: Date
    }
}, {timestamps: true})

export const Symbol = mongoose.model<Symbol>('Symbol', symbolSchema)

This is how data is presented by the Alpha Vantage API:

[
    {
        "2021-07-02": {
            "1. open": "17.39",
            "2. high": "17.63",
            "3. low": "17.25",
            "4. close": "17.43",
            "5. adjusted close": "17.43",
            "6. volume": "24228000",
            "7. dividend amount": "0.0000",
            "8. split coefficient": "1.0"
        },
        "2021-07-01": {
            "1. open": "16.86",
            "2. high": "17.32",
            "3. low": "16.82",
            "4. close": "17.2",
            "5. adjusted close": "17.2",
            "6. volume": "39101898",
            "7. dividend amount": "0.0000",
            "8. split coefficient": "1.0"
        },
        "2021-06-30": {
            "1. open": "17.15",
            "2. high": "17.46",
            "3. low": "17.02",
            "4. close": "17.07",
            "5. adjusted close": "17.07",
            "6. volume": "28807699",
            "7. dividend amount": "0.0000",
            "8. split coefficient": "1.0"
...

]

I'm storing the historical data as an object, just as it comes from the Alpha Vantage API. I'm having trouble skipping and limiting queries from my own API. In other words, I would like to receive only x historical data and be able to apply paging, but I am not able to.

Before, I was using Python, and I saved the Alpha Vantage historical data as an array of numbers. I used the following code snippet and was able to apply the document limitation. Could someone help me?

model.findOne({"code": req.query.code}, {data_frame: {$slice:-req.query.limit}})
                    .then(obj=>res.json(200, obj.data_frame))
                    .catch(next)

I tried the following commands without success:

model.aggregate([
                    { $match: { "code": req.query.code } },
                    { $unwind: "$data_frame" },
                    { $limit: 2 }
                ])
                    .then(obj=>res.json(200, obj))
                    .catch(next)

and this..

model.find({"code": req.query.code})
                    .slice('data_frame', 2)
                    .then(obj=>res.json(200, obj))
                    .catch(next)

score 0 · Answer 1 · answered Jul 05 '21 at 09:25

0

Basically, you have to update the page every time you go back and next.

let page = 0
let itemsPerPage = 2

    model.aggregate([
                        { $match: { "code": req.query.code } },
                        { $unwind: "$data_frame" },
                        { $limit: itemsPerPage },
                        { $skip:itemsPerPage*page}
                    ])
                        .then(obj=>res.json(200, obj))
                        .catch(next)

answered Jul 05 '21 at 09:25

Hanuman Singh

196
1
5

That didn't work for me. The data_frame attribute contains a JSON object as mentioned in the question above. Doing this way, it continues to return all the contents of the data_frame and not just the last `x` data. – Maykon Meneghel Jul 05 '21 at 12:08

score 0 · Answer 2 · answered Jul 05 '21 at 18:24

Okay, I solved this problem as follows:

My error was storing the data captured by the Alpha Vantage API. So, from these captured data, I applied the following commands, before saving to mongo, as mentioned here:

let documents = []
Object.entries(data["Time Series (Daily)"]).forEach(function (data) {
  const index_date = {date: Date.parse(data[0])}
  Object.entries(index_date).forEach(([key, value]) => { data[1][key] = value })
  documents.push(data[1])
})

Thus, the saved data looked like this:

"data_frame": {
   "1. open": "113.41",
   "2. high": "114.36",
   "3. low": "110.76",
   "4. close": "111.28",
   "5. adjusted close": "111.28",
   "6. volume": "22014500",
   "7. dividend amount": "0.0000",
   "8. split coefficient": "1.0",
   "date": 1625097600000
 }

And then with the @Hanuman Singh response itself you can limit the query and apply paging.

How to apply threshold and paging on financial series data coming from Alpha Vantage API?

2 Answers2