1

I have an iteration that can take up to hours to complete. Example:

    do{
        //this is an api action
        let response = await fetch_some_data;
        // other database action 
        await perform_operation();
        next = response.next;
    }while(next);

I am assuming that the operation doesn't times out. But I don't know it exactly. Any kind of explanation of nodejs satisfying this condition is highly appreciated. Thanks.

Update: The actual development code is as under:

    const Shopify = require('shopify-api-node');
    const shopServices = require('../../../../services/shop_services/shop');
    const { create } = require('../../../../controllers/products/Products');

    exports.initiate = async (redis_client) => {

        redis_client.lpop(['sync'], async function (err, reply) {
            
            if (reply === null) {
                console.log("Queue Empty");
                return true;
            }

            let data = JSON.parse(reply),
                shopservices = new shopServices(data),
                
                shop_data = await shopservices.get()
                    .catch(error => {
                        console.log(error);
                    });

                const shopify = new Shopify({
                    shopName: shop_data.name,
                    accessToken: shop_data.access_token,
                    apiVersion: '2020-04',
                    autoLimit: false,
                    timeout: 60 * 1000
                });

                let params = { limit: 250 };
                do {
                    try {

                        let response = await shopify.product.list(params);

                        if (await create(response, shop_data)) {
                            console.log(`${data.current}`);
                        };

                        data.current += data.offset;
                        params = response.nextPageParameters;

                    } catch (error) {
                        console.log("here");
                        console.log(error);
                        params = false;

                    };
                } while (params);


        });

    }

Everything is working fine till now. I am just making sure that the execution will ever happen in node or not. This function is call by a cron every minute, and data for processing is provided by queue data.

Tekraj Shrestha
  • 1,228
  • 3
  • 20
  • 48
  • I think it would be dependent on the framework you are using on the server. e.g. express: https://stackoverflow.com/questions/21708208/express-js-response-timeout also koa: https://bitbucket.org/atlassian/koa-request-timeout/src/master/ – akaphenom Jul 14 '20 at 12:03
  • @akaphenom I am using core js for making a worker without any framework. – Tekraj Shrestha Jul 14 '20 at 12:47
  • then i think you need to manage that yourself... i am not sure how to do that. But if you post your server code someone (or possibly me) might be able to guide you – akaphenom Jul 15 '20 at 17:16
  • @akaphenom I have updated the code, but the gist is still the same. If you have time please have a look at it. Thanks – Tekraj Shrestha Jul 16 '20 at 11:12
  • I stopped reading at *This function is call by a cron every minute*. If it takes longer then 1 min to complete i.e *take up to hours to complete* then you going to have concurrency issues or hundreds of instances running, dont use cron. – Lawrence Cherone Jul 16 '20 at 11:17
  • @LawrenceCherone What am I supposed to do then ? Well concurrency would be actually nice for me, if it would not effect server to the optimum level. Faster the queue is empty, better for me. – Tekraj Shrestha Jul 16 '20 at 11:32
  • @LawrenceCherone Also I could implement limits up-to 5 simultaneous cron to be running in parallel. If not i will leave the queue as it is without performing pop. So after the currently running cron process is decreased to 4 i can add another cron process. Would it solve the hundreds of instances running issue ? – Tekraj Shrestha Jul 16 '20 at 12:01
  • i am certain i wouldn't use HTTP for long running operations. You are better off using some asynchornous message passing library (e.g. Kafka, SQS, RabbitMQ) and setup channels/topics in each direction (IN and OUT) and tie messages off with UUIDs. – akaphenom Jul 16 '20 at 17:12
  • @akaphenom But this is not http, it is simply a worker. After implementing kafka, SQS or RabbitMQ also i need to process those data in node or other language which will also take same amount of time. – Tekraj Shrestha Jul 16 '20 at 17:28

0 Answers0