Read JSON file from Azure Blob Storage using bash script and write back to another file in blob storage

Question

I have following JSON file (product.json) stored in Azure Blob storage. Is it possible to write bash script to read this file from blob storage make some changes and write back to another blob container . The output file I would like where following changes should occur:

Replace "dev" with "qa"

{
    "ds_type": "saas_app",
    "ds_engine": "xxxx",
    "ds_display_name": "xxxx",
    "logo_url": "xxxx/xxxx.png",
    "base_template_path": "xxxx/xxxx/xxxx.cds.json",
    "authentication": {
        "type": "oauth",
        "client_id": "xxxx",
        "client_secret": "xxxx",
        "scope": ["crm.objects.contacts.read", "crm.objects.owners.read"],
        "grant_type": "authorization_code",
        "oauth_base_url": "https://xxxx",
        "oauth_api_url": "https://xxxx",
        "redirect_uri": "https://xxxx-dev.xxxx.com/code",
        "auth_url": "oauth/authorize",
        "token_url": "oauth/v1/token"
    }
}

There isn't any problem to modify your JSON with an external utility like `jq`, but how would you get the JSON from the blob? Have you tried anything? — Fravadona, Dec 17 '21 at 10:33
If you just want to transform JSON, I'd look into some PaaS service like Azure Data Factory https://www.sqlservercentral.com/blogs/transforming-json-data-with-the-help-of-azure-data-factory-part-3 — silent, Dec 17 '21 at 10:43
@Fravadona I have not tried anything yet, I get the json file using az storage blob download command — RahulKumar Surati, Dec 17 '21 at 11:23
@RahulKumarSurati Is it one time scenario or event-triggered? — Jacek Kuliś, Dec 17 '21 at 15:21
incase for workround you want to with powershell you can refrer this link this is same related to your requirement : https://stackoverflow.com/questions/66609333/read-json-file-from-azure-blob-storage-using-powershell-script-and-write-back-to?rq=1 — RahulKumarShaw, Dec 21 '21 at 05:20
for bash command you can refer this : https://stackoverflow.com/questions/525592/find-and-replace-inside-a-text-file-from-a-bash-command?rq=1 — RahulKumarShaw, Dec 21 '21 at 05:29
@RahulKumarShaw-MT your answer is good but is it possible to change text without download the json file? — RahulKumar Surati, Dec 22 '21 at 06:14
Yes I have that solution as well . in that case you have to mount the Azure blob Storage. Updatating in an answer in few min. — RahulKumarShaw, Dec 22 '21 at 06:19
If the answer was helpful, Please [Accept it as an Answer](https://meta.stackexchange.com/questions/5234/how-does-accepting-an-answer-work), so that others who encounter the same issue can find this solution and fix their problem. — Ansuman Bal, Dec 30 '21 at 11:36
Hello @RahulKumarSurati.you can Upvote and accept it as answer( click on the check mark beside the answer to toggle it from greyed out to filled in.). This can be beneficial to other community members. Thank you — RahulKumarShaw, Feb 04 '22 at 17:12

RahulKumarShaw · Answer 1 · 2021-12-22T06:31:55.323

Please follow the below step to achieve your requirement.

I am using Ubuntu VM (Version 18.04)

Step 1: Install AZ CLI module using this below command

curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash

Step 2 : Download Blob file from Storage Account to your local. Below is the picture of container and blob which will be download in local.

az storage blob download --container-name "your container name" --file "Location of file where you want to copy data" --name "Blob Name" --account-name "Storage Account name" --account-key "Storage Account Access Key"

Step 3 : Check the content of the file using cat command

cat ansuman.json

Step 4 : Run the following command to make changes in your json file

replace "dev" "qa" -- package.json

Step 5: Now to final upload the file in Another Container.

az storage blob upload --container-name "your conatiner name where you want to upload" --file "location of local file that is going to upload" --account-name "Storage account name" --account-key "Storage Account Access Key"

Reference : https://learn.microsoft.com/en-us/cli/azure/storage/blob?view=azure-cli-latest#az_storage_blob_upload

Update

Please follow the below step for changes in JSON file without downloading in local.

Step 1: Configure the Microsoft package repository.

wget https://packages.microsoft.com/config/ubuntu/18.04/packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.deb
sudo apt-get update

Step 2: Install Blobfuse on your Linux Machine (I am using Ubuntu version 18.04)

sudo apt-get install blobfuse

Step 3: Prepare for Mounting Use an SSD as a temporary path.

sudo mkdir /mnt/resource/blobfusetmp -p
sudo chown <youruserofVM> /mnt/resource/blobfusetmp

Step 4: Create this file using:

touch ~/fuse_connection.cfg

Step 5: Configure your storage account credentials in. fuse_connection.cfg file.

accountName myaccount
accountKey storageaccesskey
containerName mycontainer

Step 6: Once you've created and edited this file, make sure to restrict access so no other users can read it.

chmod 600 ~/fuse_connection.cfg

Step 7: Create an empty directory for mounting.

mkdir ~/mycontainer

Step 8: To mount blobfuse, run the following command with your user. This command mounts the container specified in 'fuse_connection.cfg' onto the location '/mycontainer`

sudo blobfuse ~/mycontainer --tmp-path=/mnt/resource/blobfusetmp  --config-file=fuse_connection.cfg -o attr_timeout=240 -o entry_timeout=240 -o negative_timeout=120

Step 9: Run the following command to make changes in your json file