I realise that there is a duplicate here: Execute more than 500 operations at once in Firestore Database, but the accepted answer there uses TypeScript in the accepted answer which doesn't work for me.
I'm fetching some data from a REST API which returns an JSON array of ~4000 objects. I want to save all of these objects into a collection on the Firestore database.
So, I'm trying to run a set of multiple batch updates.
I have some code which tries to link together some link some promises together in a for loop, taking some data from an external source:
exports.getRawData = functions.https.onRequest((req, res) => {
fetch('https://www.example.com', {
method: 'POST',
headers: {
'Content-Type': 'application/x-www-form-urlencoded'
},
body: qs.stringify(body)
})
.then(response => response.json())
.then(data =>
fetch(`https://www.example.com/Endpoint?StartDate=${req.query.startDate}&EndDate=${req.query.endDate}`, {
method: 'GET',
headers: {
'Content-Type': 'application/json',
'Authorization': `Bearer ${data.access_token}`
}
})
)
.then(response => response.json())
.then(newData => {
for (let i = 0, p = Promise.resolve(); i < 10; i++) {
p = p.then(_ => new Promise(resolve =>
{
console.log(i);
const batch = db.batch()
let sliced = newData.slice(i * 40, (i+1) * 40)
for (let j = 0; j < sliced.length; j++) {
let docRef = db.collection("SessionAttendance").doc()
batch.set(docRef, sliced[j])
}
batch.commit()
resolve();
}
));
}
})
.then(() => {return res.send('OK')})
.catch(err => console.log('Err: ' + err))
})
Weirdly this code doesn't always give the same error. Sometimes it says:
Function execution took 3420 ms, finished with status: 'connection error'
I've read that this error usually happens because I have some unreturned Promises, so perhaps I have some of those.
Also, on some deploys, it returns this error:
Function execution took 60002 ms, finished with status: 'timeout'
And then it just keeps running over and over again.
I've tried quite a few different ways of solving this problem, but none of them seem to work.
I also tried this block of code:
.then(newData => {
const batches = _.chunk(newData, 20)
.map(postSnapshots => {
const writeBatch = db.batch();
postSnapshots.forEach(post => {
const docRef = db.collection("SessionAttendance").doc()
// console.log('Writing ', post.id, ' in feed ', followerId);
writeBatch.set(docRef, post);
});
return writeBatch.commit();
});
return Promise.all(batches);
})
It gives the same error as above:
Function execution took 3635 ms, finished with status: 'connection error'
