3

I wrote the following code to upload a file to my Google Drive account. It works just fine. Then i tried to add some upload progress tracker but i ran into some issues. It works but the progress goes almost instantly to 100%, way before the upload is completed.

export async function uploadFile(auth) {
  const fileMetadata = {
    name: FILENAME,
  };
  const fileWrite = fs.createReadStream(path.join(homedir(), 'Pictures', FILENAME));
  const fileSize = fs.statSync(path.join(homedir(), 'Pictures', FILENAME)).size;
  const media = {
    mimeType: FILETYPE,
    body: fileWrite,
  };
  const drive = google.drive({ version: 'v3', auth });
  const res = await drive.files.create({
    fileMetadata,
    media,
    fields: 'id',
  }, {
    onUploadProgress: (evt) => {
      const progress = (evt.bytesRead / fileSize) * 100;
      console.log(progress);
    },
  });
  console.log(res.data);
}

It uploads the file i want, it shows the progress and it returns the file id. The problem is that the upload progress is wrong. I tried to upload a 100mb file and, with a 20mbps upload connection, it only took 1.8 seconds for the progress to be completed. The promise file upload resolved after 14 seconds

tehhowch
  • 9,645
  • 4
  • 24
  • 42
Pier Brown
  • 31
  • 1
  • I suspect it's a confusion over onUploadProgress. I can't find any documentation for that callback, but I guess it's only appropriate for resumable uploads. – pinoyyid Aug 10 '18 at 21:19
  • Possibly related ([`onUploadProgress`](https://google.github.io/google-api-nodejs-client/interfaces/_lib_api_.methodoptions.html#onuploadprogress) is an Axios thing): https://stackoverflow.com/questions/44893946/axios-upload-progress-for-multiple-file-uploads https://github.com/axios/axios/pull/423 – tehhowch Aug 12 '18 at 19:56
  • Also see https://github.com/axios/axios/issues/639 – tehhowch Aug 12 '18 at 20:06
  • did you find any solution ? – AHmedRef Aug 28 '18 at 14:22

2 Answers2

1

NEW ANSWER (Old answer issue fixed here)

The problem might be caused by not handling the read stream correctly with Axios. Instead of passing the read stream directly to axios.put(), you should pipe the read stream to the PassThrough stream and provide it to axios.put().

import axios from 'axios';
import { google } from 'googleapis';
import fs from 'fs';
import path from 'path';
import { PassThrough } from 'stream';

export async function uploadFile(auth) {
  const fileMetadata = {
    name: FILENAME,
  };
  const filePath = path.join(homedir(), 'Pictures', FILENAME);
  const fileSize = fs.statSync(filePath).size;
  const media = {
    mimeType: FILETYPE,
  };

  const drive = google.drive({ version: 'v3', auth });
  try {
    const uploadUrlRes = await drive.files.create({
      requestBody: fileMetadata,
      media: media,
      fields: 'id',
      uploadType: 'resumable',
    });

    const uploadUrl = uploadUrlRes.data;
    const fileWrite = fs.createReadStream(filePath);
    const passThrough = new PassThrough();
    fileWrite.pipe(passThrough);

    const axiosConfig = {
      headers: {
        'Content-Type': media.mimeType,
        'Content-Length': fileSize,
      },
      onUploadProgress: (evt) => {
        const progress = Math.round((evt.loaded / fileSize) * 100);
        console.log(progress);
      },
    };

    const res = await axios.put(uploadUrl, passThrough, axiosConfig);
    console.log(res.data);
  } catch (error) {
    console.error('Error uploading file:', error);
  }
}

OLD ANSWER.

Using Axios to handle the upload and its progress. First create a resumable upload session with Google Drive API by calling drive.files.create() with the uploadType parameter set to 'resumable'. This will return a resumable session URI, which can use with Axios to upload the file.

Now, configure Axios with the appropriate headers and onUploadProgress event handler. We then use the axios.put() method to upload the file to the resumable session URI.

import axios from 'axios';
import { google } from 'googleapis';
import fs from 'fs';
import path from 'path';

export async function uploadFile(auth) {
  const fileMetadata = {
    name: FILENAME,
  };
  const fileWrite = fs.createReadStream(path.join(homedir(), 'Pictures', FILENAME));
  const fileSize = fs.statSync(path.join(homedir(), 'Pictures', FILENAME)).size;
  const media = {
    mimeType: FILETYPE,
    body: fileWrite,
  };

  const drive = google.drive({ version: 'v3', auth });
  try {
    const uploadUrlRes = await drive.files.create({
      requestBody: fileMetadata,
      media: media,
      fields: 'id',
      uploadType: 'resumable'
    });

    const uploadUrl = uploadUrlRes.data;
    const axiosConfig = {
      headers: {
        'Content-Type': media.mimeType,
        'Content-Length': fileSize,
      },
      onUploadProgress: (evt) => {
        const progress = Math.round((evt.loaded / fileSize) * 100);
        console.log(progress);
      },
    };

    const res = await axios.put(uploadUrl, fileWrite, axiosConfig);
    console.log(res.data);
  } catch (error) {
    console.error('Error uploading file:', error);
  }
}
BIS Tech
  • 17,000
  • 12
  • 99
  • 148
  • Hi, which version of "googleapis" do you use ? Because i have "^118.0.0" in my project and i can't see any "axios" options inside "drive_v3.Options". So where should i define the "axiosInstance" ? – ChaTho Apr 21 '23 at 13:45
  • you have to install axios. npm install axios – BIS Tech Apr 21 '23 at 14:43
  • It's already installed the version of "^1.3.6". – ChaTho Apr 21 '23 at 15:24
  • It's not even defined as an option in googleapis package. What does it change to install axios ? @BIS Tech – ChaTho Apr 22 '23 at 14:05
  • You're correct that the googleapis package doesn't have an explicit option for Axios. but you can still use Axios to upload files to Google Drive and track progress. try my updated code. – BIS Tech Apr 22 '23 at 18:03
  • Thanks for reply, i tried both "multipart" and "resumable" uploadTypes of api with axios but onUploadProgress only runs 2 times while file uploading. And it logs "1" in first and "100" in last log according to your calculated log example above. So i tried to upload bigger files (like 100-200 MB) to see the progress slowly. But logs didn't change. Also the second log runs after first log immediately. Whereas the file has not even uploaded, calculated log says "100" after 1-2 seconds of upload proccess. What's wrong with it ? @BIS Tech – ChaTho Apr 23 '23 at 09:51
  • It seems like the issue you're facing is still related to the progress event being triggered too fast. The problem might be caused by not handling the read stream correctly with Axios. Instead of passing the read stream directly to axios.put(), you should pipe the read stream to the PassThrough stream and provide it to axios.put(). – BIS Tech Apr 24 '23 at 08:42
  • Actually, it is so weird. onUploadProgress works perfect in frontend (react js) while uploading the file to backend (express js). But however, it doesn't work as expected in express js. I also tried your last recommend but nothing changed, all logs are the same as before. I really don't want to waste your time anymore, thank you so much for your effort. I appreciate. :)) – ChaTho Apr 24 '23 at 18:09
0
const fileSize =Number(file.size) //file.size obtain from multer in my case
    const res = await drive.files.create({
      .....
      }, {
        onUploadProgress: (evt) => {
          const progress=Math.round((evt.bytesRead / fileSize) * 100);
          console.log(progress);
        },
      });

This code works for me

Juan Carlos Ajahuana Quispe
  • 41
  • 1
  • 4