I have a Matillion pipeline which downloads a file from a SFTP server and stores the file in Azure blob container called encryptedfiles.
The file is encrypted via OpenSSL and I am unsure what steps to take to decrypt the file and store it in a different container, called decryptedfiles.
One idea that comes to mind is to use either a bash or a Python script to do this, but i am unsure how to get started.
What is the best approach?
You can use the Python component to decrypt the files. You will need to use the python azure blob storage libraries to access the file and download it to the matillion server. Then use python SSL libraries to decrypt it, and then put it back on blob storage. Remember to remove the file(s) from the matillion server when done (use /TMP folder).
visit How to decrypt OpenSSL AES-encrypted files in Python?
The downside to this approach is that it forces you to transfer the data onto the matillion server, so if the data is on the smaller side, this will work fine. If the data is very large, you will need to break it into chunks, and use the file iterator component to feed chunks into the python script
