body parser is breaking base64 string

Question

I am sending a base64 string from my mobile app to my Express server via POST request. The image is sent to a Google client endpoint, but it is returning an error: Provided image is not valid

code: 400, 0|index | errors: [ 0|index | { 0|index | message: 'Provided image is not valid.', 0|index | domain: 'global', 0|index | reason: 'badRequest' 0|index | } 0|index | ]

I think it has to do with the body-parser breaking my stringified body from client side to server endpoint.

On the client:

const body = typeof data === 'string' ? data : JSON.stringify(data); // data is base64 string;

const response = await fetch(path, {
      method,
      body,
      headers: {
        ...headers,
      },
    });

Here is a gist of base64 from the client: https://gist.github.com/lucksp/0a831684c34271424309096172ccb79a#file-clientbase64-js

Here is sample log:

You can see the highlighted string includes text of: CZjs+EJ+/+I.

Now, here is the log of the same asset on the server:

You can see the highlighted string includes text of CZjs EJ / IC instead. the +'s have been stripped.

Here is my Express server code.

const app = express();
app.use(bodyParser.json({ limit: '20mb' }));
app.use(
  bodyParser.urlencoded({
    limit: '20mb',
    extended: true,
    parameterLimit: 50000,
  })
);
app.use(bodyParser.text({ limit: '200mb' }));
async function callPrediction({
  endpoint,
  data,
}: {
  endpoint?: string | null;
  data: string;
}) {
const auth = new GoogleAuth({
    scopes: 'https://www.googleapis.com/auth/cloud-platform',
  });
  const client = await auth.getClient();
const body = JSON.stringify({
    instances: [
      {
        content: data,
      },
    ],
    parameters: {
      confidenceThreshold: 0.5,
      maxPredictions: 5,
    },
  });

  const res = await client.request({
    method: 'POST',
    url,
    body,
    headers: { 'Content-Type': 'application/json' },
  });

  return res;
};

app.post('/verify', async (req, res) => {
const results = await callPrediction({
    endpoint: endpoints[0].name,
    data: req.body, // comes in as a stringified JSON. Also fails if parsing here.
  });
}

Am I doing something wrong with bodyParser or is it the base64?Am I doing something wrong with bodyParser or is it the base64?

What's really odd is that the same base64 sent from Postman works fine.

Please don't post code, logs and error messages as images. Can you provide a [mcve]? Where is the body parser? — jabaa, Jun 10 '23 at 23:36
this might help https://stackoverflow.com/a/22131822/3807365 — IT goldman, Jun 10 '23 at 23:42
@jabaa I updated the snippet to include `app.use(bodyParser)` implementations — Phil Lucks, Jun 10 '23 at 23:46
I added images because it's easier to highlight the difference — Phil Lucks, Jun 10 '23 at 23:47
I can't reproduce the problem and I am not going to type that data. Can you provide a [mcve]? — jabaa, Jun 11 '23 at 00:11
You should be able to copy/paste from your logs to the question. No need to retype. Images are heavily frowned upon here because they are not friendly to view, are terrible on mobile, can't be searched, can't be copied and pasted into answers, etc... — jfriend00, Jun 11 '23 at 00:22
For body-parser to do its job appropriately, you need to make sure that the data you are sending is in the proper content format and the `content-type` header you are sending matches it 100%. If that's the case, then only the correct body-parser middleware will read and parse your data on the server and you should get back what you expect. You show code for two different http requests and it's unclear to me which one you're having a problem with. — jfriend00, Jun 11 '23 at 03:01
yes @jfriend00, I am sending the POST to `app.post('/verify')` endpoint. This takes my data from client (mobile app), and then calls `callPrediction ` which is a wrapper around a Google Cloud API and takes the `data` as an argument. I have verified that the base64 data is broken when reading data in the `post` . — Phil Lucks, Jun 11 '23 at 04:54
@jfriend00 I did try adding the `type: 'application/json'` to the `bodyParser.json` and `bodyParser.urlencoded`, but now the request body is ` req.body: {}` — Phil Lucks, Jun 12 '23 at 03:07

Watachiaieto · Answer 1 · 2023-06-11T16:22:42.167

1

I am going to make the guess that it is converting the spaces into plus signs. What if you turn the image into a binary blob and turn it into a hex string instead, and then back into binary on the other side?

EDIT: I looked at your strings more closely, and found more examples of the spaces becoming pluses. I have a feeling that urlencode is the function that is doing that.

It is as I suspected... https://github.com/expressjs/body-parser/issues/316 There might be a setting you can change somewhere, but I don't use the bodyparser. I think you should try using a base16 string instead. It contains hex, and none of those are special characters.

edited Jun 11 '23 at 16:22

answered Jun 11 '23 at 05:01

Watachiaieto

417
3
10

Thanks for the response @watachiaieto but the plus character from the client is escaping to spaces on the server. I have also tried express.json() instead of bodyParser with the same result – Phil Lucks Jun 12 '23 at 02:33
https://en.wikipedia.org/wiki/Query_string In this link it has a section on URL encoding. It states that ' ' => '+', and that '+' => '%2B', etc etc etc... Maybe ensure it is properly encoding all the characters they describe, and then ensure that on the decode side it properly decodes. I don't have your setup, and have never used it, but I have done encoding before and sometimes different encoders don't get the memo ;) – Watachiaieto Jun 12 '23 at 03:12
ok, will look into this...What's really odd is that the same base64 sent from Postman works fine. – Phil Lucks Jun 12 '23 at 03:30
Have you debugged and check that the strings match that they send? There coudl also even possibly be a secondary layer where is does further encoding or something strange. Can you look at the raw data on the server as it receives it? – Watachiaieto Jun 12 '23 at 06:37

body parser is breaking base64 string

1 Answers1