2

I'm trying to send a blob image, but I'm getting Error: Unexpected end of form using multer with Serverless Framework.

From console.log enter image description here

My understanding is I have to append it to FormData before sending it in the body, but I haven't been able to get backend to accept file without crashing

    uploadImage(imageData: File) {
        console.log('IMAGE DATA', imageData);
        let formData = new FormData();
        formData.append('file', imageData, 'file.png');
        let headers = new HttpHeaders();
        headers.append('Content-Type', 'multipart/form-data');
        headers.append('Accept', 'application/json');
        let options = { headers: headers };

        const api = environment.slsLocal + '/add-image';
        const req = new HttpRequest('PUT', api, formData, options);

        return this.http.request(req);
    }

backend


const multerMemoryStorage = multer.memoryStorage();
const multerUploadInMemory = multer({
    storage: multerMemoryStorage
});

router.put(
    '/add-image',
    multerUploadInMemory.single('file'),
    async (req, res: Response) => {
        try {
            if (!req.file || !req.file.buffer) {
                throw new Error('File or buffer not found');
            }

            console.log(`Upload Successful!`);

            res.send({
                message: 'file uploaded'
            });
        } catch (e) {
            console.error(`ERROR: ${e.message}`);

            res.status(500).send({
                message: e.message
            });
        }

        console.log(`Upload Successful!`);

        return res.status(200).json({ test: 'success' });
    }
);

app.ts

import cors from 'cors';
import express from 'express';
import routers from './routes';
const app = express();
import bodyParser from 'body-parser';

app.use(cors({ maxAge: 43200 }));
app.use(
    express.json({
        verify: (req: any, res: express.Response, buf: Buffer) => {
            req.rawBody = buf;
        }
    })
);

app.use('/appRoutes', routers.appRouter);
app.use(
    bodyParser.urlencoded({
        extended: true  // also tried extended:false
    })
);

export default app;

enter image description here

From my understanding with serverless framework I have to install npm i serverless-apigw-binary

and add

    apigwBinary:
      types: #list of mime-types
        - 'image/png'

to the custom section of the serverless template yaml file. The end goal is not to save to storage like S3, but to send the image to discord.

What am I missing? I appreciate any help!

user6680
  • 79
  • 6
  • 34
  • 78
  • you're missing a file processing middleware on the server, see [Access file upload formData in Express](https://stackoverflow.com/questions/65655500/access-file-upload-formdata-in-express) – traynor Jan 13 '23 at 13:47
  • If I add `multer().single('file')` as middleware on request then backend crashes when sending file. I'm wondering if it's because I'm using aws serverless ExpressJS – user6680 Jan 13 '23 at 17:04
  • try removing `headers` completely, just send `formData`, you also might add error handler to see what's the problem. also, try including extension to the filename, for example `myImage.jpg` – traynor Jan 13 '23 at 18:03
  • I tried removing headers all the way and just sending formData. then tried ```formData.append('name', 'myImage.png');```, but same issue. I get empty {} in backend – user6680 Jan 13 '23 at 18:21
  • the request should be fine now, so try here: [Uploading image to amazon s3 using multer-s3 nodejs](https://stackoverflow.com/questions/40494050/uploading-image-to-amazon-s3-using-multer-s3-nodejs), missed that aws tag.. – traynor Jan 13 '23 at 18:31
  • This seems applicable only to S3. I'm not trying to upload to S3. My end goal is sending the blob image to discord. I'm not trying to store it anywhere. – user6680 Jan 14 '23 at 02:23
  • well, that's quite different from what you asked.. you should include that in your post, as well as all the code used, if you hope to get help.. So, you can read it in memory as a buffer and then pass it along, see: https://stackoverflow.com/questions/62147839/multer-returns-req-file-as-undefined-and-req-file-location-as-location-undefine – traynor Jan 14 '23 at 10:04
  • You're right I should of clarified. I updated my post + added more details. I also modified it to reflect your last suggestion. I now get ```Error: Unexpected end of form```. Some people saying bodyParser is the issue but I already have that. I tried it with ```extended: false``` and ```extended: true```, but same issue. I do feel like I'm closer though because if I changed the formData in angular to not match what multerUploadInMemory.single('file') is looking for in angular service ie ```formData.append('file1', imageData, 'file.png');``` I get ```MulterError: Unexpected field``` – user6680 Jan 14 '23 at 16:22
  • in angular, don't include `headers`, try: `const req = new HttpRequest('PUT', api, formData);` – traynor Jan 14 '23 at 19:37
  • I know you mentioned earlier to remove it completely. I added back when I was trying things. I just removed it again with current code and only have ```const req = new HttpRequest('PUT', api, formData);``` now, but I still get ```Error: Unexpected end of form``` – user6680 Jan 14 '23 at 21:47
  • This is usally an error where the ending boundary is not specified or complete. For whatever reason, the boundary seems malformed. It is normally not advisable to set the `Content-Type` header as you did, which the browsers set automatically when they encounter `FormData` but for testing reasons, could you try settting `headers.append('Content-type', 'multipart/form-data; boundary=XXX')` to explicitly set the boundary and see what happens? – ibrahim tanyalcin Jan 15 '23 at 23:36
  • 1
    I tried that, but I get ```Unexpected end of form``` still. I did manage to get it working though. Thanks for the suggestion though. @traynor thanks also for trying to help me figure this out. This worked for me https://stackoverflow.com/a/75129572/4350389 – user6680 Jan 16 '23 at 01:38

4 Answers4

0

I recently encountered something similar in a react native app. I was trying to send a local file to an api but it wasn't working. turns out you need to convert the blob file into a base64 string before sending it. What I had in my app, took in a local file path, converted that into a blob, went through a blobToBase64 function, and then I called the api with that string. That ended up working for me.

I have this code snippet to help you but this is tsx so I don't know if it'll work for angular.

function blobToBase64(blob: Blob) {
    return new Promise((resolve, reject) => {
      const reader = new FileReader();
      reader.onerror = reject;
      reader.onload = () => {
        resolve(reader.result as string);
      };
      reader.readAsDataURL(blob);
    });
  }

Hope this helps!

Fabi
  • 21
  • 2
  • I have a solution that converts it to base64 and when I send it to expressjs it says `request entity too large`. I looked into it and from what I learned is that sending base64 is much too inefficient size-wise. I should be able to send as a blob. – user6680 Jan 13 '23 at 00:31
  • Yeah I noticed the inefficiency too as it takes multiple seconds for me to get a response from the api but it was the only thing that worked for me at the time. Anyways, I apologize my answer wasn't of any help. – Fabi Jan 13 '23 at 00:36
  • 1
    I appreciate your input, but it's just not what I'm looking for. – user6680 Jan 13 '23 at 00:47
0

You can convert your Blob to a File using

new File([blob], "filename")

and then you should be able pass that file to your existing uploadImage method.

GreyBeardedGeek
  • 29,460
  • 2
  • 47
  • 67
  • With that change backend from original post above crashes and says ```Error: Unexpected end of form``` Could be something to do with using serverless framework + expressJS or ```multerUploadInMemory``` middleware, but not really sure. – user6680 Jan 15 '23 at 19:04
0

Looks like you are passing Blob instead of File based on your console.log(). So you should convert Blob to a File before calling the server. You can change your frontend code like this:

uploadImage(imageData: File) {
  // Convert Blob to File
  const file = new File([imageData], "file_name", { type: imageData.type });

  let formData = new FormData();
  formData.append('file', file, 'file.png');

  const api = environment.slsLocal + '/add-image';
  return this.http.put(api, formData);
}

Note: For more info about converting Blob to File, you can check this StackOverflow question.

NeNaD
  • 18,172
  • 8
  • 47
  • 89
  • Maybe has something to do with using Serverless Framework with expressJS or perhaps it's the ```multerUploadInMemory``` middleware, but with that change backend crashes and says ```Error: Unexpected end of form```. So something is still off. – user6680 Jan 15 '23 at 19:05
  • Hmmm... Can you try to update your bodyParser and JSON parser like this: `app.use(express.json({ limit: '50mb' })); app.use(express.urlencoded({ extended: true, limit: '50mb' }));` – NeNaD Jan 15 '23 at 19:31
  • I replaced ```app.use(bodyParser.urlencoded({ extended: true }));``` in app.ts from original post above with what you suggested, but same error – user6680 Jan 15 '23 at 19:35
  • Are you still setting Content-Type headers in the frontend? If yes, can you remove all custom headers from the request in the frontend? – NeNaD Jan 15 '23 at 19:37
  • Headers have been removed, but that didn't change error. Here's a screenshot of error: https://i.postimg.cc/XNfGPrgb/image.png – user6680 Jan 15 '23 at 19:41
  • Hmmm, just found out that when converting Blob to File, you should also specify the `type` in order for it to work properly. I updated my answer. Can you try it again? – NeNaD Jan 15 '23 at 20:33
  • I updated angular service like so ```const file = new File([imageData], "file_name", { type: imageData.type });```, but same error. – user6680 Jan 15 '23 at 20:50
  • Just a sec.... I see that your are sending PUT request in the fontend... But in the backend, you defined the endpoint with POST. Can you try using POST in the frontend? – NeNaD Jan 15 '23 at 21:01
  • I jumped between the POST and PUT during testing. It's PUT on frontend and backend. I updated my original post to reflect that. That's not the issue. Sorry for the confusion. – user6680 Jan 15 '23 at 21:31
  • Then I don't know man... I updated my answer one more time... Try to use `this.http.put` instead of `this.http.request`, maybe it would handle something differently internally... Also, check this StackOverflow question too: https://stackoverflow.com/questions/72544409/unexpected-end-of-form-error-when-using-multer – NeNaD Jan 15 '23 at 22:09
0

The thing that got it working for me was this article. There might be something different about using Express through Serverless Framework so things like mutler and express-fileupload might not work. Or could be because it's an AWS Lambda function. I don't know this for sure though. I just know I never got it working. This article was the only thing that worked for Serverless Framework + Express.

I also had to install version 0.0.3 of busboy ie npm i busboy@0.0.3. The newer version didn't work for busboy. Newer version was saying Busboy is not a constructor

Since I'm sending the file to discord and not S3 like this article does, I had to tweak the parser.event part in this part of the article for the handler.ts

export const uploadImageRoute = async (
    event: any,
    context: Context
): Promise<ProxyResult> => {
    const parsedEvent: any = await parser(event);
    await sendImageToDiscord(parsedEvent.body.file);
    const response = {
        statusCode: 200,
        body: JSON.stringify('file sent successfully')
    };
    return response;
};

comes in as a Buffer which I was able to send as a file like this

const fs = require('fs-extra');
const cwd = process.cwd();
const { Webhook } = require('discord-webhook-node');

const webhook = new Webhook('<discord-webhook-url>');

export async function sendImageToDiscord(arrayBuffer) {
    var buffer = Buffer.from(arrayBuffer, 'base64');
    const newFileName = 'nodejs.png';
    await fs.writeFile(`./${newFileName}`, buffer, 'utf-8').then(() => {
        webhook.sendFile(`${cwd}/${newFileName}`);
    });
}
});

I hope this helps someone!

user6680
  • 79
  • 6
  • 34
  • 78