0

I'm saving different data on different databases on Redis, with the same key. For example at any time this script runs, I have the keys test_1 and test_2 on databases 0...100 in Redis. What I'm trying to do, is for each database, get the keys that have the same name, and save the data to a file. Here's a snippet of what I've tried. 

const redis = require('redis');
const client = redis.createClient();
const fs = require('fs');

for (let i = 1; i <= 100; i++) {
    client.select(i, function(err,res){
        client.mget(['test_1', 'test_2'], function(err,res){
            let results = JSON.parse(res);

            fs.appendFileSync('testfile.json', results);
        });
    });
}

I've also tried wrapping this in an async function and also wrapping await client.mget in an async function aswell, but nothing seems to work.

Ali Heikal
  • 3,790
  • 3
  • 18
  • 24
KM_
  • 41
  • 14

1 Answers1

1

You can't make async calls like this in a for loop. What you're doing at the moment is telling node to run 100 select statements, it's firing those off and they will all return at different times, then they'll each start their own mget and eventually you might get them attempt to append to the file. This isn't what you want at all, even if it were to work you don't get any reasonable order to your file either.

There are a number of ways you could do this, I'd say the simplest is probably with something like a promise library since you want to fire off a bunch of async requests, aggregate the results, and then write a file.

In this case I'd do something such as the below. Short disclaimer, I've not run the code, but I believe this should roughly work with a couple of tweaks.

const bluebird = require('bluebird');
const redis = require('redis');
const client = redis.createClient();

// Use bluebird to convert client.select and client.mget into
// promise returning functions rather than callback funcs.
// you could do this with the built in node utils, but we need
// some of bluebirds helper functions later on.
const rSelect = bluebird.promisify(client.select);
const rMget = bluebird.promisify(client.mget);

// Get 0..100 array, there are tons of ways to do this...
const dbIds = Array.from(Array(100).keys());

const operations = dbIds.map(id => {
    return rSelect(i)
        .then(res => {
            return rMget(['test_1', 'test_2']);
        });
});

// You now have an array of promises, one for each database id (0..100).
// We can now execute these promises using a bluebird helper.
// Here I'm using map, if you need them done in a specific order you could
// use mapSeries. You could also use reduce if you'd prefer writing it in
// a reducer style, but that doesn't seem to be what you want.
return bluebird.map(operations)
    .then(allResponses => {
        // "allResponses" now contains the response of every operation.
        // lets write those out to a file.
        var result = JSON.parse(allResponses);
        fs.appendFileSync('testfile.json', result);
    })

In the above example I use the Bluebirdjs library, you're free to use any you like but this should get the job done. It's a bit easier to work with bluebird for iterative async processes than callbacks in my experience as it provides some nice helper functions.

Elliot Blackburn
  • 3,759
  • 1
  • 23
  • 42
  • Tried your approach, but I get two errors. One error based on the loop and one error at the end. Check this [pastebin](https://pastebin.com/amSBNivf). – KM_ Jul 04 '19 at 07:25
  • In that case you might find it quicker and easier to wrap the functions in Promises yourself. See https://stackoverflow.com/questions/22519784/how-do-i-convert-an-existing-callback-api-to-promises if you want to use the bluebirdjs library you _might_ need to use "new bluebird" rather than "new Promise". You should be able to fix the final error with a bit of debugging, that doesn't seem like an issue with my proposed solution, more that I didn't line up a type somewhere. – Elliot Blackburn Jul 04 '19 at 15:09