Setting Secrets from AWS Secrets manager in Node.JS

Question

Before top-level await becomes a thing, loading secrets asynchronously from AWS Secrets Manager upon startup is a bit of a pain. I'm wondering if anyone has a better solution than what I currently have.

Upon starting up my Node.JS server I'm loading all secrets from AWS Secrets manager and setting them in config files where I have a mix of hardcoded variables and secrets. Here's an example:

In aws.js

import AWS from 'aws-sdk';
const region = "eu-north-1";

AWS.config.setPromisesDependency();

const client = new AWS.SecretsManager({
    region
});

export const getSecret = async(secretName) => {
    const data = await client.getSecretValue({SecretId: secretName}).promise();
    return data.SecretString;
}

Then in sendgridConfig.js

import { getSecret } from "./aws";

export default async() => {
    const secret = JSON.parse(await getSecret("sendgridSecret"));
    return {
        APIKey: secret.sendgridKey,
        fromEmail: "some@email.com",
        toEmail: "some@email.com"
    }
}

Then in some file where the config is used:

import { sendgridConfig } from "./sendgridConfig";

const myFunc = () => {
    const sendgridConf = await sendgridConfig();
    ... do stuff with config ...
}

This works okay in async functions, but what if I'd like to use the same setup in non-async functions where I use my hardcoded variables? Then the secrets haven't been fetched yet, and I can't use them. Also I have to always await the secrets. IMO a good solution in the future could be top level await, where upon booting the server, the server will await the secrets from AWS before proceeding. I guess I could find a way to block the main thread and set the secrets, but that feels kind of hacky.

Does anyone have a better solution?

Answer 1

So I ended up doing the following. First I'm setting the non-async config variables in an exported object literal. Then I'm assigning values to the object literal in the sendgridConfigAsync IIFE (doesn't have to be an IFEE). That way I don't have to await the config promise. As long as the app awaits the IIFE on startup, the keys will be assigned before being accessed.

In sendgridConfig.js

import { getSecret } from "./aws";

export const sendgridConfig = {
    emailFrom: process.env.sendgridFromEmail,
    emailTo: process.env.sendgridToEmail
}

export const sendgridConfigAsync = (async() => {
    const secret = JSON.parse(await getSecret("Sendgrid-dev"));
    sendgridConfig.sendgridKey = secret.key;
})()

Then in the main config file _index.js where I import all the config files.

import { sendgridConfigAsync } from "./sendgrid";
import { twilioConfigAsync } from "./twilio";
import { appConfigAsync } from "./app";

export const setAsyncConfig = async() => {
    await Promise.all([
        appConfigAsync,
        sendgridConfigAsync,
        twilioConfigAsync
    ]);
}

Then in the main index.js file I'm awaiting the setAsyncConfig function first. I did also rebuild the app somewhat in order to control all function invocations and promise resolving in the desired order.

import { servicesConnect } from "../src/service/_index.js";
import { setAsyncConfig } from '$config';
import { asyncHandler } from "./middleware/async";
import { middleware } from "./middleware/_index";
import { initJobs } from "./jobs/_index"
import http from 'http';

async function startServer() {
    await setAsyncConfig();
    await servicesConnect();
    await initJobs();
    app.use(middleware);

    app.server = http.createServer(app);

    app.server.listen(appConfig.port);
    console.log(`Started on port ${app.server.address().port}`);
}

asyncHandler(startServer());

Answer 2

Yup I have the same problem. Once you start with a promise, all dependencies down the line require await.

One solution is to do your awaits and only after that have all your downstream code run. Requires a slightly different software architecture.

E.g.

export const getSecretAndThenDoStuff = async(secretName) => {
    const data = await client.getSecretValue({SecretId: secretName}).promise();
    // instead of return data.SecretString;
    runYourCodeThatNeedsSecret(data);
}

Answer 3

A more generic solution to the top-level await that I tend to use:

async function main() {
  // Do whatever you want with await here
}

main();

Clean and simple.