How can I append JSON data to an existing JSON file stored in Azure blob storage through python?

Question

I've been looking around the web to append data to an existing JSON file in azure storage, I also check on this post, but it didn't help. I have millions of JSON records coming in real-time which are available in python list and I want to append those JSON records to an existing JSON file in azure blob. Though my main data source is KafkaConsumer, and I'm consuming data from Kafka topic and I want that data into azure storage as JSON format. As, I'm using python and I don't want to read/write on my local hard disk, I just want like if I have list of JSON records I can directly append to JSON file which already in azure container. Can anyone help me out or give some references, it will be pleasure for me. Thanks

In start I only upload JSON file using upload_blob function and then I tried [append_block](https://azuresdkdocs.blob.core.windows.net/$web/python/azure-storage-blob/12.0.0/azure.storage.blob.html#azure.storage.blob.BlobClient.append_block) function on this JSON file, but it gives authentication error as this . "ErrorCode:AuthenticationFailed Error:None AuthenticationErrorDetail:The MAC signature found in the HTTP request 'hW87ugUtVXulSjA4ZpI6jc6vLU+tjj4KKM7/uWE2J6w=' is not the same as any computed signature. Server used following string to sign: 'PUT 1043 application/octet-stream" — zawster, Dec 11 '21 at 17:47
I'm doing authentication when I create conection and it didn't give any issue when I create connection. I only create BlobServiceClient connection using account _url and account key after this I'm not doing any authentication but when I append to blob it gives authentication issue. — zawster, Dec 13 '21 at 05:32
`blob_client.append_block(json_record, sys.getsizeof(json_record))` This is what I am trying to append into the existing blob. — zawster, Dec 13 '21 at 19:54
If the answer is helpful for you, you can accept it as answer( click on the check mark beside the answer to toggle it from greyed out to filled in.). This can be beneficial to other community members. Thank you — ShrutiJoshi-MT, Dec 15 '21 at 14:20

score 2 · Accepted Answer · answered Dec 14 '21 at 10:19

2

I tried in my system able to append the data to existing file, I taken the dummy json data for testing purpose you can pass the your json data

from azure.storage.blob import AppendBlobService
import json

def append_data_to_blob(data):
  service = AppendBlobService(account_name="appendblobex", 
            account_key="key")
  data1 = {}
  data1['hi'] = 'hello'
  json_data = json.dumps(data1)
  data = json.dumps(data1)
  print(data1)
  try:
    service.append_blob_from_text(container_name="test", blob_name="test1", text = data)
  except:
     #To create the blob and append data
    #service.create_blob(container_name="test", blob_name="test1")
    service.append_blob_from_text(container_name="test", blob_name="test1", text = data)
  print('Data Appended to Blob Successfully.')


append_data_to_blob("data")

OUTPUT

Data appended in the azure storage file after download open the file and view the data

answered Dec 14 '21 at 10:19

ShrutiJoshi-MT

1,622
1
4
9

I'm not able to import AppendBlobService, it is showing Module not found. – zawster Dec 16 '21 at 08:15
try with pip install `azure-storage --upgrade` and again try with import statement – ShrutiJoshi-MT Dec 16 '21 at 08:44
If the answer is helpful for you, you can accept it as answer( click on the check mark beside the answer to toggle it from greyed out to filled in.). This can be beneficial to other community members. Thank you – ShrutiJoshi-MT Dec 17 '21 at 11:26
Okay thanks @ShrutiJoshi-MT for your cooperation, definitely it was very helpful for me. – zawster Dec 18 '21 at 10:40
@ShrutiJoshi-MT can we access this appendblob from databricks. As per the MS documention, we cannot read the append blobs in databricks. – SanjanaSanju Jun 29 '22 at 19:09

How can I append JSON data to an existing JSON file stored in Azure blob storage through python?

1 Answers1