Batch Fetch in Firebase functions with node-fetch

Question

I'm trying to batch fetch from the iexcloud api, a string of 100 symbols at a time, using firebase functions. My first fetch responds and writes to the DB correctly. On my second call I constantly get "Client network socket disconnected before secure TLS connection was established " .. not sure what I'm doing wrong here:

const functions = require("firebase-functions");
const fetch = require("node-fetch");
const admin = require("firebase-admin");


admin.initializeApp();
const db = admin.firestore();


exports.scheduledFunctionCrontab = functions.pubsub
    .schedule("0 09-17 * * 1-5")
    .timeZone("America/New_York") // Users can choose timezone - default is America/Los_Angeles
    .onRun(async () => {
        const promises = [];
        const positions = await admin.firestore().collection("positions");
        const tempDoc = [];
        await positions.get().then((querySnapshot) => {
            querySnapshot.forEach((doc) => {
                tempDoc.push(doc.id);
            });
            return tempDoc;
        });

    
        // Let's break this into groups of 100.
        let positionsObject = tempDoc.reduce((resultArray, item, index) => {
            let perChunk = 100; // items per chunk
            const chunkIndex = Math.floor(index / perChunk);
            if (!resultArray[chunkIndex]) {
                resultArray[chunkIndex] = []; // start a new chunk
            }
            resultArray[chunkIndex].push(item);
            return resultArray;
        }, []);

        async function databaseWrite(json) {
...
        // writing to the DB
...
        }


        async function getCurrentPrice() {
            for (const symbols of positionsObject) {
                let ourPositionsString = symbols.toString();

                const API_Call = `https://sandbox.iexapis.com/stable/stock/market/batch?symbols=${ourPositionsString}&types=quote&token=TOKEN`;

                const fetchResponse = await fetch(API_Call, {
                    method: "GET",
                    agent: false,
                });
                const json = await fetchResponse.json();
                await databaseWrite(json);

            }
        }
        getCurrentPrice();

        return Promise.all(promises);
    });

databaseWrite works, and the API_Call loops successfully with a new ourPostitionsString of the correct 100 positions, but the 2nd fetch response is always socket disconneceted..

score 0 · Answer 1 · answered Jun 23 '22 at 12:42

A large number of write operations on Firestore can be performed in three ways:
1.Perform each individual write operation in sequence.
2.Using batched write operations.
3.Performing individual write operations in parallel.

The fastest and most efficient way to perform bulk data writes on Firestore is by performing parallel individual write operations. For bulk data entry, use a server client library with parallelized individual writes. You should use a server client library for bulk data operations and not a mobile/web SDK.

Batched writes perform better than serialized writes but not better than parallel writes. Batched writes create a BatchedWrite object by calling batch(), until it has a maximum capacity of 500 documents, and then write it to Firestore. The solution counts every operation which is made to the batch and after the limit is reached a new batch is created and pushed to the batchArray. After all updates are completed the code loops through the batchArray and commits every batch which is inside the array. It is important to count every operation set(), update(), delete() which is made to the batch because they all count to the 500 operation limit.

Have a look at the threads below for the reference on Firestore bulk writes:

[1] : https://firebase.google.com/docs/firestore/quotas
[2]: https://firebase.google.com/docs/firestore/client/libraries#server_client_libraries
[3]: https://stackoverflow.com/a/58897275/15803365

Batch Fetch in Firebase functions with node-fetch

1 Answers1