How to merge multiple json files into one json file using python

Question

I have several json files that I would like to merge into one json file. The json files are in the following format:

file1:

{"seconds":112,"label":"1","value":{"time":15,"period":1576,"values":[1,2,3]},"shape":{"type":1,"values":[1,2,3]},"size":{"type":1,"values":[-9,10,20]}}

{"seconds":115,"label":"2","value":{"time":20,"period":882,"values":[11,82,8]},"shape":{"type":1,"values":[99,8,3]},"size":{"type":3,"values":[89,65,-20]}}

file2:

{"seconds":532,"label":"12","value":{"time":90,"period":4328,"values":[11,2,32]},"shape":{"type":1,"values":[112,2,32]},"size":{"type":1,"values":[-23,9420,20]}} 

{"seconds":115,"label":"2","value":{"time":20,"period":882,"values":[11,82,8]},"shape":{"type":1,"values":[942,6,332]},"size":{"type":32,"values":[49,125,-590]}}

that I would like to merge into one json file to be like the following:

{"seconds":112,"label":"1","value":{"time":15,"period":1576,"values":[1,2,3]},"shape":{"type":1,"values":[1,2,3]},"size":{"type":1,"values":[-9,10,20]}}

{"seconds":115,"label":"2","value":{"time":20,"period":882,"values":[11,82,8]},"shape":{"type":1,"values":[99,8,3]},"size":{"type":3,"values":[89,65,-20]}}

{"seconds":532,"label":"12","value":{"time":90,"period":4328,"values":[11,2,32]},"shape":{"type":1,"values":[112,2,32]},"size":{"type":1,"values":[-23,9420,20]}}

{"seconds":115,"label":"2","value":{"time":20,"period":882,"values":[11,82,8]},"shape":{"type":1,"values":[942,6,332]},"size":{"type":32,"values":[49,125,-590]}}

I have tried:

import json
import glob

result = []
for f in glob.glob("*.json"):
    with open(f, "rb") as infile:
        result.append(json.load(infile))

with open("merge.json", "wb") as outfile:
     json.dump(result, outfile)

But that gave me 'raise JSONDecodeError("Extra data", s, end) json.decoder.JSONDecodeError: Extra data: line 2 column 1 (char 333)' Thank you so much for your help!

@EnoGergurin Not an empty line, just to have each string be on its own line would be ideal. — , Jun 25 '20 at 20:51
@EnoGerguri I tried that, and got 'TypeError: a bytes-like object is required, not 'str'' — , Jun 25 '20 at 20:54
It`s not enough to read the raw files and join them together? https://www.tutorialspoint.com/How-to-merge-multiple-files-into-a-new-file-using-Python — Namba, Jun 25 '20 at 20:56
I agree with @Namba, just concatenate the files. Since it looks like the data is json lines format, you can completely ignore the json nature and treat them as lines of plain text. So standard unix command-line tools are sufficient. — Hitobat, Jun 25 '20 at 21:05
Does this answer your question? [Python concatenate text files](https://stackoverflow.com/questions/13613336/python-concatenate-text-files) — Eno Gerguri, Jun 25 '20 at 21:05

score 0 · Answer 1 · answered Jun 25 '20 at 20:54

The main problem seems to be that your json files are not valid. You need to put [] around your objects. Here is how these files should be:

file1.json

[
    {
      "seconds": 112,
      "label": "1",
      "value": {
        "time": 15,
        "period": 1576,
        "values": [
          1,
          2,
          3
        ]
      },
      "shape": {
        "type": 1,
        "values": [
          1,
          2,
          3
        ]
      },
      "size": {
        "type": 1,
        "values": [
          -9,
          10,
          20
        ]
      }
    },
    {
      "seconds": 115,
      "label": "2",
      "value": {
        "time": 20,
        "period": 882,
        "values": [
          11,
          82,
          8
        ]
      },
      "shape": {
        "type": 1,
        "values": [
          99,
          8,
          3
        ]
      },
      "size": {
        "type": 3,
        "values": [
          89,
          65,
          -20
        ]
      }
    }
  ]

file2.json

[
  {
    "seconds": 532,
    "label": "12",
    "value": {
      "time": 90,
      "period": 4328,
      "values": [
        11,
        2,
        32
      ]
    },
    "shape": {
      "type": 1,
      "values": [
        112,
        2,
        32
      ]
    },
    "size": {
      "type": 1,
      "values": [
        -23,
        9420,
        20
      ]
    }
  },
  {
    "seconds": 115,
    "label": "2",
    "value": {
      "time": 20,
      "period": 882,
      "values": [
        11,
        82,
        8
      ]
    },
    "shape": {
      "type": 1,
      "values": [
        942,
        6,
        332
      ]
    },
    "size": {
      "type": 32,
      "values": [
        49,
        125,
        -590
      ]
    }
  }
]

Also, there are minor errors in your python code. This should work:

import json
import glob

result = []
for f in glob.glob("*.json"):
    with open(f, "r") as infile:
        result += json.load(infile)

with open("merge.json", "w") as outfile:
    json.dump(result, outfile)

Note that you should use += instead of append since append will add an entire list of dictionaries as a new item in a list.

score 0 · Answer 2 · answered Jun 25 '20 at 21:05

file1.json and file2.json should be a list with dictionary object in it like this

[{"seconds":112,"label":"1","value":{"time":15,"period":1576,"values":[1,2,3]},"shape":{"type":1,"values":[1,2,3]},"size":{"type":1,"values":[-9,10,20]}},
{"seconds":115,"label":"2","value":{"time":20,"period":882,"values":[11,82,8]},"shape":{"type":1,"values":[99,8,3]},"size":{"type":3,"values":[89,65,-20]}}]

then you can use below code use extend instead of append

files=['file1.json','file2.json',...]

def merge_JsonFiles(filename):
        result = list()
        for f1 in filename:
            with open(f1, 'r') as infile:
                result.extend(json.load(infile))
    
        with open('counseling3.json', 'w') as output_file:
            json.dump(result, output_file)
    
    merge_JsonFiles(files)

How to merge multiple json files into one json file using python

2 Answers2