1

I am new to AWS. I want to make a hash of the stream before it gets stored on s3 bucket. But, When I call 'getHashFromStream()' before 'await this.s3.upload()' the file stored on s3 is always empty. And when I remove 'getHashFromStream()' it works properly. I don't know what breaks the process.

Code :


public async storeToS3(
    key: string,
    stream: any
  ): Promise<{ key: string; versionId: string; size?: number }> {
    const gzip = zlib.createGzip();
    //Whn I call getHashFromStream() It will store empty file to s3 Bucket even If the file hash data.
    const hash = await this.getHashFromStream(stream);
    await this.s3
      .upload({
        Key: key,
        Metadata: { syncStartTime: this.syncStartTime.toString() },
        Body: stream.pipe(gzip),
        Bucket: this.dataBucketName,
        ContentEncoding: 'gzip',
        StorageClass: 'STANDARD_IA',
      })
      .promise();

    const newItemResult = await this.listVersions(key, ':', 1);

    if (newItemResult.Versions) {
      const Hash = hash;
      const itemId = newItemResult.Versions[0]?.Key;
      const serviceAccountId = newItemResult.Versions[0].Key?.split(
        ':',
        1
      ).toString();
      const versionId = newItemResult.Versions[0]?.VersionId;

      if (
        Hash === undefined ||
        itemId === undefined ||
        serviceAccountId === undefined ||
        versionId === undefined
      ) {
        throw new Error(
          `required variable is undefined ${Hash} ${itemId} ${serviceAccountId} ${versionId}`
        );
      }
      const uniqueKey = {
        [`${itemId}`]: { S: versionId }, //Map value
      };

      //add to UniqueData Table
      await this.uniqueDataTable.addItem({
        Hash: Hash,
        itemId: itemId,
        serviceAccountId: serviceAccountId,
        uniqueKey: uniqueKey,
        versionId: versionId,
      });
      return {
        key,
        versionId: newItemResult.Versions[0].VersionId!,
        size: newItemResult.Versions[0].Size,
      };
    } else {
      throw new Error(`Object: ${key} list versions returned undefined.`);
    }
  }

getHashFromStream() function :

public getHashFromStream = async (stream: any): Promise<string> =>
    new Promise((resolve, reject) => {
      const hash = crypto.createHash('sha256');
      stream.on('error', reject);
      stream.on('data', (chunk: any) => hash.update(chunk));
      stream.on('end', () => resolve(hash.digest('hex')));
    });

Please Let me know if I am missing any information.

JIGNESH PATEL
  • 367
  • 4
  • 17
  • 1
    In all probability the stream is consumed by the hash function and, therefore, nothing is left to send to S3. Perhaps see if [this answer](https://stackoverflow.com/questions/54348804/node-js-copy-a-stream-into-a-file-without-consuming) helps you. – Eben Roux Aug 02 '21 at 06:39
  • @EbenRoux, Yes, I think you are right. But, I have also tried to copy stream value to stream1 like below. But, same issue : ``` const stream1: any = stream; ``` can you please tell me that how can I copy stream to stream1 so that I can create hash from stream1. – JIGNESH PATEL Aug 02 '21 at 08:56

0 Answers0