How to stop Node API cluster server properly?

Question

I'm just learning to use clusters in my Koa + TypeScript API. I found a blog post and took references from it.

// server.ts

import {app} from './app' // Koa

export function server() {
  const http = app.listen(8000, async () => {
    console.info(`Listening at 8000`)
  })
  return http
}

// main.ts

async function main() {
  try {
    // Create or migrate
    await knex.migrate.latest()
    return server()
  } catch (err) {
    logger.error('Failed to start: ', err.stack)
    process.exit(1)
  }
}

// Clustering
const numCPUs = cpus().length

if (cluster.isMaster) {
  logger.info(`${numCPUs} CPUs available`)
  for (let i = 0; i < numCPUs; i++) {
    cluster.fork()
  }

  cluster.on('online', worker => {
    logger.info(`Worker ${worker.process.pid} is online`)
  })

  cluster.on('disconnect', worker => {
    logger.error(`${worker.process.pid} disconnect!`)
    cluster.fork()
  })

  /** I do not think I need this block, right?
   *
  cluster.on('exit', (worker, code, signal) => {
    logger.info(
      `Worker ${worker.process.pid} died with code: ${code} and signal: ${signal}`,
    )
    logger.info('Starting a new worker...')
    cluster.fork()
  })
   */
} else {
  main()
}

And I have this graceful shutdown function:

// Graceful shutdown
async function shutdown() {
  try {
    await knex.destroy().then(() => {
      logger.warn('Destroyed Knex pool')
    })
    // Make sure we close down within 30 seconds
    const killtimer = setTimeout(() => {
      process.exit(1)
    }, 30000)
    // But don't keep the process open just for that!
    killtimer.unref()
    server().close(() => {
      logger.warn('Closed server')
      process.exit()
    })
    // Let the maste know w're dead.
    // return cluster.worker.disconnect()
  } catch (e) {
    logger.error(`Error during disconnection: ${e.stack}`)
    process.exit(1)
  }
}

function awaitHardStop() {
  const timeout = SHUTDOWN_TIMEOUT ? +SHUTDOWN_TIMEOUT : 1000 * 30
  return setTimeout(() => {
    logger.error(`Process did not terminate within ${timeout}ms. Stopping now!`)
    process.nextTick(process.exit(1))
  }, timeout)
}

// Catch unhandling unexpected exceptions
process.on('uncaughtException', e => {
  logger.error(`Uncaught exception: ${e.message}`)
  process.nextTick(process.exit(1))
})

// Catch unhandling rejected promises
process.on('unhandledRejection', (reason, promise) => {
  logger.error(`Unhandled rejection at ${promise}, reason: ${reason}`)
  process.nextTick(process.exit(1))
})

process.on('SIGINT', async () => {
  logger.warn(`[ SIGNAL ] - SIGINT`)
  const timer = awaitHardStop()
  await shutdown()
  clearTimeout(timer)
})

process.on('SIGTERM', async () => {
  logger.warn(`[ SIGNAL ] - SIGTERM`)
  const timer = awaitHardStop()
  await shutdown()
  clearTimeout(timer)
})

process.on('SIGUSR2', async () => {
  logger.warn(`[ SIGNAL ] - SIGUSR2`)
  const timer = awaitHardStop()
  await shutdown()
  clearTimeout(timer)
})

Issue is I got this when I stop with Ctrl+c:

Error: listen EADDRINUSE: address already in use :::8000
    at Server.setupListenHandle [as _listen2] (net.js:1313:16)
    at listenInCluster (net.js:1361:12)
    at Server.listen (net.js:1447:7)
    at Application.listen (/home/usr/workspace/koa-ts/app/node_modules/koa/lib/application.js:82:19)
    at Object.server (/home/usr/workspace/koa-ts/app/src/core/server.ts:7:20)
    at /home/usr/workspace/koa-ts/app/src/main.ts:71:5
    at Generator.next (<anonymous>)
    at fulfilled (/home/usr/workspace/koa-ts/app/src/main.ts:24:58)
{"level":50,"time":1597173528053,"pid":284878,"hostname":"usr-PC","msg":"Uncaught exception: listen EADDRINUSE: address already in use :::8000"}

So I tried adding this line in main.ts just below server().close():

    ...
    // Let the maste know w're dead.
    return cluster.worker.disconnect()
    ...

Then I got this when Ctrl+c:

Error: listen EADDRINUSE: address already in use :::8000
    at Server.setupListenHandle [as _listen2] (net.js:1313:16)
    at listenInCluster (net.js:1361:12)
    at Server.listen (net.js:1447:7)
    at Application.listen (/home/usr/workspace/koa-ts/app/node_modules/koa/lib/application.js:82:19)
    at Object.server (/home/usr/workspace/koa-ts/app/src/core/server.ts:7:20)
    at /home/usr/workspace/koa-ts/app/src/main.ts:71:5
    at Generator.next (<anonymous>)
    at fulfilled (/home/usr/workspace/koa-ts/app/src/main.ts:24:58)
{"level":50,"time":1597173528053,"pid":284878,"hostname":"usr-PC","msg":"Uncaught exception: listen EADDRINUSE: address already in use :::8000"}
{"level":50,"time":1597173554364,"pid":285843,"hostname":"usr-PC","msg":"Error during disconnection: TypeError: Cannot read property 'disconnect' of undefined\n    at /home/usr/workspace/koa-ts/app/src/main.ts:76:27\n    at Generator.next (<anonymous>)\n    at fulfilled (/home/usr/workspace/koa-ts/app/src/main.ts:24:58)"}
{"level":50,"time":1597173554364,"pid":285843,"hostname":"usr-PC","msg":"Error during disconnection: TypeError: Cannot read property 'disconnect' of undefined\n    at /home/usr/workspace/koa-ts/app/src/main.ts:76:27\n    at Generator.next (<anonymous>)\n    at fulfilled (/home/usr/workspace/koa-ts/app/src/main.ts:24:58)"}
llsi/app/src/main.ts:24:58)"}
{"level":50,"time":1597173997176,"pid":287724,"hostname":"usr-PC","msg":"Error during disconnection: TypeError: Cannot read property 'disconnect' of undefined\n    at /home/usr/workspace/koa-ts/app/src/main.ts:76:27\n    at Generator.next (<anonymous>)\n    at fulfilled (/home/usr/workspace/koa-ts/app/src/main.ts:24:58)"}

Is it normal or am I doing something wrong here?

I am using Node (v12.18.3).

Ref: http://bisaga.com/blog/programming/create-a-node-cluster-with-koa-and-typescript/

Answer 1

Doing a cleanup action just before Node.js exits

Check out what was suggested here. I was also looking for a way to do cleanup on any form of process termination... You should basically only do the termination if cleanup is true. Also I suggest you look into a process manager. For example check out PM2, it allows you to start your application in a cluster easily and manages the different worker processes for you!