Bulk Upload/Import of JSON files to Azure Cosmos DB from .NET Core

Question

I'm generating some JSON files in .NET Core app. I want to import a json file to Azure Cosmos DB as soon as it is created.

Is there some way to achieve it from a .NET Core code.

Thanks in advance!

Answer 1

According to my test, if you want to use Cosmos DB Bulk insert in .Net core application, you need to use the .Net CosmosDB SDK V3 and its version must be larger than 3.4.0. For more details, please refer to the document.

My.json file

[{
        "id": "1",
        "name": "test1",
        "age": "20"
    }, {
        "id": "2",
        "name": "test2",
        "age": "21"
    }, {
        "id": "3",
        "name": "test3",
        "age": "22"
    }, {
        "id": "4",
        "name": "test4",
        "age": "23"
    },
    {
        "id": "5",
        "name": "test5",
        "age": "24"
    }, {
        "id": "6",
        "name": "test6",
        "age": "25"
    }, {
        "id": "7",
        "name": "test7",
        "age": "26"
    }, {
        "id": "8",
        "name": "test8",
        "age": "27"
    }
]

My code

 private const string EndpointUrl = "";
        private const string AuthorizationKey = "";
        private const string DatabaseName = "testbulk";
        private const string ContainerName = "items";
        async static Task Main(string[] args)
        {
            string json = File.ReadAllText(@"E:\test.json");
            
            List<Item> lists = JsonConvert.DeserializeObject<List<Item>>(json);

            CosmosClientOptions options = new CosmosClientOptions() { AllowBulkExecution = true };
            CosmosClient cosmosClient = new CosmosClient(EndpointUrl, AuthorizationKey,options);
            
            Database database = await cosmosClient.CreateDatabaseIfNotExistsAsync(DatabaseName);
            Console.WriteLine(database.Id);
            Container container = await database.CreateContainerIfNotExistsAsync(ContainerName, "/id");
            Console.WriteLine(container.Id);
            
            List<Task> tasks = new List<Task>();
            foreach (Item item in lists)
            {
                tasks.Add(container.CreateItemAsync(item, new PartitionKey(item.Id))
                    .ContinueWith((Task<ItemResponse<Item>> task) =>
                    {


                        Console.WriteLine("Status: " + task.Result.StatusCode + "    Resource: "+ task.Result.Resource.Id );



                    }));


            }
            await Task.WhenAll(tasks);
}
class Item {

        [JsonProperty(PropertyName = "id")]
        public string Id { get; set; }

        [JsonProperty(PropertyName = "name")]
        public string Name { get; set; }

        [JsonProperty(PropertyName = "age")]
        public string Age { get; set; }
    }

For more details about how to develop your application, please refer to the blog

---

Update

I run my code in .Net 4.6.1 console application.

My code

 class Program
    {

        private const string EndpointUrl = "";
        private const string AuthorizationKey = "";
        private const string DatabaseName = "testbulk";
        private const string ContainerName = "items";
        async static Task Main(string[] args)
        {
            string json = File.ReadAllText(@"E:\test.json");

            List<Item> lists = JsonConvert.DeserializeObject<List<Item>>(json);

            CosmosClientOptions options = new CosmosClientOptions() { AllowBulkExecution = true };
            CosmosClient cosmosClient = new CosmosClient(EndpointUrl, AuthorizationKey, options);

            Database database = await cosmosClient.CreateDatabaseIfNotExistsAsync(DatabaseName);
            Console.WriteLine(database.Id);
            Container container = await database.CreateContainerIfNotExistsAsync(ContainerName, "/id");
            Console.WriteLine(container.Id);

            List<Task> tasks = new List<Task>();
            foreach (Item item in lists)
            {
                tasks.Add(container.CreateItemAsync(item, new PartitionKey(item.Id))
                    .ContinueWith((Task<ItemResponse<Item>> task) =>
                    {


                        Console.WriteLine("Status: " + task.Result.StatusCode + "    Resource: " + task.Result.Resource.Id);



                    }));


            }
            await Task.WhenAll(tasks);
            Console.ReadLine();
        }
        class Item
        {

            [JsonProperty(PropertyName = "id")]
            public string Id { get; set; }

            [JsonProperty(PropertyName = "name")]
            public string Name { get; set; }

            [JsonProperty(PropertyName = "age")]
            public string Age { get; set; }
        }
    }

My package.json

<?xml version="1.0" encoding="utf-8"?>
<packages>
  <package id="Microsoft.Azure.Cosmos" version="3.4.1" targetFramework="net461" />
  <package id="Newtonsoft.Json" version="10.0.2" targetFramework="net461" />
  <package id="System.Buffers" version="4.4.0" targetFramework="net461" />
  <package id="System.Configuration.ConfigurationManager" version="4.5.0" targetFramework="net461" />
  <package id="System.Memory" version="4.5.1" targetFramework="net461" />
  <package id="System.Numerics.Vectors" version="4.4.0" targetFramework="net461" />
  <package id="System.Runtime.CompilerServices.Unsafe" version="4.5.1" targetFramework="net461" />
  <package id="System.Security.AccessControl" version="4.5.0" targetFramework="net461" />
  <package id="System.Security.Permissions" version="4.5.0" targetFramework="net461" />
  <package id="System.Security.Principal.Windows" version="4.5.0" targetFramework="net461" />
  <package id="System.ServiceModel.Primitives" version="4.5.0" targetFramework="net461" />
  <package id="System.Threading.Tasks.Extensions" version="4.5.1" targetFramework="net461" />
  <package id="System.ValueTuple" version="4.5.0" targetFramework="net461" />
</packages>

Answer 2

Here is how you can handle this scenario architecturally which will be scalable and as per the guideline:

• From you .Net Core aplication , when you are creating the json file , store it in Azure blob storage
• As a next step , you create an Azure Function with the Azure blob storage trigger, you can check this link for reference:

https://learn.microsoft.com/en-us/azure/azure-functions/functions-create-storage-blob-triggered-function

• From azure function , you can configure to use BulkImport .net library for updating the records in cosmos db.

    BulkImportResponse bulkImportResponse = await bulkExecutor.BulkImportAsync(
      documents: documentsToImportInBatch,
      enableUpsert: true,
      disableAutomaticIdGeneration: true,
      maxConcurrencyPerPartitionKeyRange: null,
      maxInMemorySortingBatchSize: null,
      cancellationToken: token);

This .NET library provides two overloads of the bulk import API - one that accepts a list of serialized JSON documents and the other that accepts a list of deserialized POCO documents. To learn more about the definitions of each of these overloaded methods, refer to the API documentation.

Check this link for further reference:

https://learn.microsoft.com/en-us/azure/cosmos-db/bulk-executor-dot-net

Above software design meet the expectation for availability, scalability and Maintainability.

Feel free to tag me in your conversation, hope it helps.