We currently use a lambda function to move files (csv, zip or gzip) that are landed into an S3 bucket connected to an sFTP to another S3 bucket for loading into snowflake.
Some of the files we receive are in *.zipx archive format. We are facing difficulties uncompressing the files as the python standard libraries such as zipfile and they don't seem to support zipx. Also, lambda has a 10GB limit. Research bears no fruit because zipx is not a very common format. We don't have an option to convince the provider to give the file in a different format.
We have a hack that writes the file to local disk, unzips and then we rewrite it to S3. However, we are trying to do this in an automated solution so when the file lands, there is no human intervention.
We looked at the standard python libraries such as zipfile, perhaps leveraging our sftp provider, etc. No luck, however. Any ideas on how to support ZIPX?
