Split huge Json objects for saving into database

Question

I have a JSON file which contains around 20000 objects.

I would like to split the file into parts each containing a smaller number of object but without using any external libraries.

My current json

  [{"alt_party_id_type":"xyz","first_name":"child1ss","status":"1","dob":"2014-10-02 
     00:00:00.0","last_name":"childSs"},
    {"alt_party_id_type":"xyz","first_name":"suga","status":"1","dob":"2014-11-05 
      00:00:00.0","last_name":"test"},
           {"alt_party_id_type":"xyz","first_name":"test4a","status":"1","dob":"2000-11-05 
       00:00:00.0","last_name":"test4s"},
             {"alt_party_id_type":"xyz","first_name":"demo56","status":"0","dob":"2000-11-04 
    00:00:00.0","last_name":"Demo5"}, 
  {"alt_party_id_type":"xyz","first_name":"testsss","status":"1","dob":"1900-01-01 
     00:00:00.0","last_name":"testssssssssss"}, 
    {"alt_party_id_type":"xyz","first_name":"Demo1234","status":"0","dob":"2014-11-21 
  00:00:00.0","last_name":"Demo1"}, 
   {"alt_party_id_type":"xyz","first_name":"demo2433","status":"1","dob":"2014-11-13 
   00:00:00.0","last_name":"demo222"}, 
 {"alt_party_id_type":"xyz","first_name":"demo333","status":"0","dob":"2014-11-12 
00:00:00.0","last_name":"demo344"}, 
{"alt_party_id_type":"xyz","first_name":"Student","status":"1","dob":"2001-12-03 
 00:00:00.0","last_name":"StudentTest"}]

if split I need it to look like below
    
    
      - [
                {
                    "alt_party_id_type": "xyz",
                    "first_name": "demo56",
                    "status": "0",
                    "dob":"2000-11-04 
         00: 00: 00.0","last_name":"Demo5"}, 
         {
                        "alt_party_id_type": "xyz",
                        "first_name": "testsss",
                        "status": "1",
                        "dob":"1900-01-01 
             00: 00: 00.0","last_name":"testssssssssss"}, 
            {
                            "alt_party_id_type": "xyz",
                            "first_name": "Demo1234",
                            "status": "0",
                            "dob":"2014-11-21 
            00: 00: 00.0","last_name":"Demo1"}]

Will this work if we split this into a 100 files, each with 200 objects in an array and feed those in one by one to the API ? — Arthur Klezovich, Nov 07 '21 at 17:28
I don't think so. It is failing. Not sure on what approach is suitable — John Miller, Nov 07 '21 at 17:33
Gateway timeout after trying to process 100 array objects through postman — John Miller, Nov 07 '21 at 17:44
This is part of data migration from json to database using API — John Miller, Nov 07 '21 at 17:45
> I tried running them through postman of around 100 array of objects but it failed How exactly did it fail ? — Arthur Klezovich, Nov 07 '21 at 17:48
Basically we connect to different api's of another apps internally which needs to be updated in the table . IT threw timeout error when it failed to response. I just ran around with 25 objects and I got the response — John Miller, Nov 07 '21 at 17:55
OK, so you just need to break it down into batches of around 25 objects and them import them individually. This gives us about 800 batches. Can be done manually more or less or by using Gson to split the whole thing — Arthur Klezovich, Nov 07 '21 at 18:01
So I have the 20000 objects in .json file and how can I split it — John Miller, Nov 07 '21 at 18:09
Either manually copy pasting parts of it into new files. Or programmatically by reading it into RAM and writing it out to multiple files and sending it. — Arthur Klezovich, Nov 07 '21 at 18:11
Manual could be bit tough as there are around 150 .json files with 20000 object each. — John Miller, Nov 07 '21 at 18:14
Then, let's go for an automated solution https://stackoverflow.com/questions/49273660/how-to-cut-up-large-json-file-into-chunks-and-sort-using-gson — Arthur Klezovich, Nov 07 '21 at 18:17
Can I try using Jq streaming. Not sure. I am just figuring out — John Miller, Nov 07 '21 at 18:25
"As this is a standalone program I want to do it without any Json library" Whut. — David Conrad, Nov 07 '21 at 22:05
It's not hard to code something to split it into individual objects. If you can read the whole string (good for a few hundred MB) you could then post just a few items at a time. If you wanted to read the file via a stream you need to work a bit harder, say buffer 1MB at a time. — dave110022, Nov 09 '21 at 11:08

score 1 · Answer 1 · answered Nov 07 '21 at 17:33

1

Since your objects are simple (no sub objects or arrays) you can take your json string and just search for the next "{" then the next "}" and extract the json for that object. Do it in a loop, incrementing your position in a string. Package as many as you like with "[" and "]" and you have split your array. Process each package within the loop.

answered Nov 07 '21 at 17:33

dave110022

64
5

Actually some other files do have some nested object. Like array inside another – John Miller Nov 07 '21 at 17:43
See revised answer – dave110022 Nov 08 '21 at 11:59

score 0 · Answer 2 · answered Nov 07 '21 at 17:33

What I'd do is write a program, which does the following ...

Parse the big JSON file without loading it into memory like this
Then you read out the data from it in increments of 100-200 and send to the API

You can also do the file split manually by using this approach and then import via Postman

Split huge Json objects for saving into database

2 Answers2