0

I have thousands of files that I want to GZIP together to make sending them more efficient. I used MergeContent, but that creates zip files, not GZIP. The system on the other side is only looking for GZIP. I can use CompressContent to create a single GZIP file, but that's not efficient for sending across the network. Also I need to preserve headers on the individual files which is why I wanted to use MergeContent.

I could write the files to disk as flowfile packages, run a script, pick up the result, then send it, but I would think I can do that in NiFi without writing to disk.

Any suggestions?

Ginger
  • 29
  • 4

1 Answers1

0

You are confusing compression with archiving.

Tar or Zip is method of archiving 1 or more input files into a single output file. E.g. file1.txt, file2.txt and file3.txt are separate files that are archived into files.tar. When you unpack the archive, you get all 3 files back as they were. An archive is not necessarily compressed.

GZIP is a method of compression, with the goal of reducing the size of the file. It takes 1 input, compresses it, and gives 1 output. E.g. You input file1.txt which is 100Kb, you compress it, you get file1.txt.gz which is 3Kb.

MergeContent is merging, thus is can produce archives like ZIP and TAR. It is not compressing.

CompressContent is compressing, thus it can produce compressed files like GZIP. It is not merging.

If you want to combine many files into a compressed archive like a tar.gz then you can use MergeContent (tar) > CompressContent (gzip). This will first archive all of the input FlowFiles into a tar file, and then GZIP compress the tar into a tar.gz.

See this answer for more detail on compression vs archiving: Difference between archiving and compression

(Note: MergeContent has an optional Compression flag when using it to create ZIPs, so in that one specific use-case it can also apply some compression to the archive, but it is only for zip)

Sdairs
  • 1,912
  • 1
  • 13
  • 13
  • Thank you for the clear explaination! I think I got confused because one of my colleagues was very vague. I tried MergeContent (using Binary Concatenation and a high number for maximum entries), then CompressContent to create a GZIP file. That seems to be what I need. – Ginger Mar 01 '22 at 15:53