How to upload multiple files in nodejs to AWS S3 and save file url into database?

Question

Hi i need to upload multiple images at a time on s3. Currently i am using express-fileupload to upload single image on AWS, and i want to use same approach to make it upload multiple files to s3 and update images array with urls on mongodb.

My schema property:

const ServiceSchema = new mongoose.Schema(
{
    photo: [
        {
            type: String,
            default: 'no-photo.jpg',
        },
    ],
});
module.exports = mongoose.model('Service', ServiceSchema);

My Controller:

// @desc        Upload photo for service
// @route       PUT /api/v1/services/:id/photo
// @access      Private
exports.servicePhotoUpload = asyncHandler(async (req, res, next) => {
const service = await Service.findById(req.params.id);

if (!service) {
    return next(new ErrorResponse(`Service not found with id of ${req.params.id}`, 404));
}

// Make sure user adding service is business owner
if (service.user.toString() !== req.user.id && req.user.role !== 'admin') {
    return next(
        new ErrorResponse(
            `User ${req.user.id} is not authorized to update this service to business ${service._id}`,
            401
        )
    );
}

// File Upload validation
if (!req.files) {
    return next(new ErrorResponse(`Please upload a file.`, 400));
}

const file = req.files.file;

// Make sure it is a valid image file
if (!file.mimetype.startsWith('image')) {
    return next(new ErrorResponse(`Please upload a valid image file.`, 400));
}

//Check File Size
if (file.size > process.env.MAX_FILE_UPLOAD) {
    return next(
        new ErrorResponse(
            `Please upload an image less then ${process.env.MAX_FILE_UPLOAD / 1024}KB in size.`,
            400
        )
    );
}

// Create custom filename
file.name = `service-uploads/servicePhoto_${service._id}${path.parse(file.name).ext}`;

uploadToS3({
    fileData: req.files.file.data,
    fileName: file.name,
})
    .then(async (result) => {
        console.log('Success Result: ', result);

        await Service.findByIdAndUpdate(service._id, { photo: result.Location });

        return res
            .status(200)
            .json({ success: true, message: 'Service photo added successfully', url:    result.Location });
    })
    .catch((err) => {
        console.log(err);
        return next(new ErrorResponse('Failed to upload file to S3', 500));
    });
  });

My Utility File to upload File to S3:

const AWS = require('aws-sdk');

const uploadToS3 = (options) => {
// Set the AWS Configuration
AWS.config.update({
    accessKeyId: process.env.AWS_S3_ACCESS_KEY,
    secretAccessKey: process.env.AWS_S3_SECRET_KEY,
    region: 'us-east-2',
});

// Create S3 service object
const s3 = new AWS.S3({ apiVersion: '2006-03-01' });

// Setting up S3 upload parameters
const params = {
    Bucket: 'toolbox-uploads',
    Key: options.fileName, // File name you want to save as in S3
    Body: options.fileData, //
};

// Return S3 uploading function as a promise so return url can be handled properly
return s3.upload(params).promise();
};

module.exports = uploadToS3;

My Router:

const express = require('express');
const {
 servicePhotoUpload
} = require('../controllers/service');

const Service = require('../models/Service');

router.route('/:id/photo').put(protect, authorize('publisher', 'business', 'admin'),  servicePhotoUpload);
 module.exports = router;

This above code is workng 100%.

I am bit confused as there were different approach and none worked for me from google and stack overflow and none of them is getting return url and saving into database.

I want to make separate utility file to upload multiple files to 3 same as i did for single files to use them anywhere. That file should return uploaded urls so i can update my database. I have tried multer-s3 but no solution works for me.

Answer 1

This approach might be different for you but that is how I was able to resolve the same issue.

First you'll need

• Multer
• multer-s3
• aws-sdk

I made a FileUpload class that handles both single and multi-upload (I also needed to be able to upload pdf and video files) and this is the code in my constructor, note that I also specified the s3-bucket in question from aws.

this.s3 = new AWS.S3({
        accessKeyId: process.env.S3_ACCESS_KEY_ID,
        secretAccessKey: process.env.S3_SECRET_KEY,
        Bucket: 'name_of_s3_bucket',
    });

I created a method called upload in the class. Code below

 upload(path, type) {
    let ext = 'jpeg';
    const multerFilter = (req, file, cb) => {
        if (type === 'image') {
            if (file.mimetype.startsWith(this.type)) {
                cb(null, true);
            } else {
                cb(
                    new AppError(
                        'Not an Image! Please upload only images',
                        400
                    ),
                    false
                );
            }
        } else if (type === 'pdf') {
            ext = 'pdf';
            const isPdf = file.mimetype.split('/')[1];
            if (isPdf.startsWith(this.type)) {
                cb(null, true);
            } else {
                cb(
                    new AppError('Not a pdf! Please upload only pdf', 400),
                    false
                );
            }
        }
    };

    const upload = multer({
        storage: multers3({
            acl: 'public-read',
            s3: this.s3,
            bucket: 'name_of_s3_bucket',
            metadata: function (req, file, cb) {
                cb(null, { fieldName: file.fieldname });
            },
            key: function (req, file, cb) {
                let filename = `user-${
                    req.user.id
                }/${path}/${uuid.v4()}-${Date.now()}.${ext}`;
                // eslint-disable-next-line camelcase
                const paths_with_sub_folders = [
                    'auditions',
                    'biography',
                    'movies',
                ];
                if (paths_with_sub_folders.includes(path)) {
                    filename = `user-${req.user.id}/${path}/${
                        req.params.id
                    }/${uuid.v4()}-${Date.now()}.${ext}`;
                }
                cb(null, filename);
            },
        }),
        fileFilter: multerFilter,
        limits: {
            fileSize: 5000000,
        },
    });

    return upload;
}

To consume the above, I import the class into any controller that I needed an upload feature and called the following.

Side Note : Ignore the paths code (It was just a way to generate unique file name for the files)

const upload = new FileUpload('image').upload('profile-images', 'image');
exports.uploadUserPhoto = upload.array('photos', 10);

I then used the uploadUserPhoto as a middleware before calling the following

exports.addToDB = catchAsync(async (req, res, next) => {
if (!req.files) return next();
req.body.photos = [];
Promise.all(
    req.files.map(async (file, i) => {
        req.body.photos.push(file.key);
    })
);

next();

});

On a high-level overview, this is the flow, First, upload your photos to s3 and get the req.files, then look through that req.files object passing them into an array field on your req object then finally save them on your DB.

NOTE: You must promisify the req.file loop since the task is asynchrnous

My final router looked like this

router
.route('/:id')
.put(uploadUserPhoto, addToDB, updateProfile)

Answer 2

Item.js

Your model can have a field called images thats type array.

const mongoose = require("mongoose");

const ItemSchema = mongoose.Schema({
  images: {
    type: [],
  },
});

module.exports = mongoose.model("Items", ItemSchema);

You map through the array of object and only extract the data you want to store, in this example it is the key which is the unique name given to every image thats uploaded.

route.js

router.post("/", verify, upload.array("image"), async (req, res) => {
  
  const { files } = req;
  const images = [];
  files.map((file) => {
    images.push(file.key);
  });

  try {
    new Item({
      images,
    }).save();
    res.status(200).send({message: "saved images to db"})
  }catch(err){
    res.status(400).send({message: err})
  }
  
});

Let me know if this does what you wanted