How to append data to a json file?

Question

I'm trying to create a function that would add entries to a json file. Eventually, I want a file that looks like

[{"name" = "name1", "url" = "url1"}, {"name" = "name2", "url" = "url2"}]

etc. This is what I have:

def add(args):
    with open(DATA_FILENAME, mode='r', encoding='utf-8') as feedsjson:
        feeds = json.load(feedsjson)
    with open(DATA_FILENAME, mode='w', encoding='utf-8') as feedsjson:
        entry = {}
        entry['name'] = args.name
        entry['url'] = args.url
        json.dump(entry, feedsjson)

This does create an entry such as {"name"="some name", "url"="some url"}. But, if I use this add function again, with different name and url, the first one gets overwritten. What do I need to do to get a second (third...) entry appended to the first one?

EDIT: The first answers and comments to this question have pointed out the obvious fact that I am not using feeds in the write block. I don't see how to do that, though. For example, the following apparently will not do:

with open(DATA_FILENAME, mode='a+', encoding='utf-8') as feedsjson:
    feeds = json.load(feedsjson)
    entry = {}
    entry['name'] = args.name
    entry['url'] = args.url
    json.dump(entry, feeds)

Answer 1

json might not be the best choice for on-disk formats; The trouble it has with appending data is a good example of why this might be. Specifically, json objects have a syntax that means the whole object must be read and parsed in order to understand any part of it.

Fortunately, there are lots of other options. A particularly simple one is CSV; which is supported well by python's standard library. The biggest downside is that it only works well for text; it requires additional action on the part of the programmer to convert the values to numbers or other formats, if needed.

Another option which does not have this limitation is to use a sqlite database, which also has built-in support in python. This would probably be a bigger departure from the code you already have, but it more naturally supports the 'modify a little bit' model you are apparently trying to build.

Answer 2

You probably want to use a JSON list instead of a dictionary as the toplevel element.

So, initialize the file with an empty list:

with open(DATA_FILENAME, mode='w', encoding='utf-8') as f:
    json.dump([], f)

Then, you can append new entries to this list:

with open(DATA_FILENAME, mode='w', encoding='utf-8') as feedsjson:
    entry = {'name': args.name, 'url': args.url}
    feeds.append(entry)
    json.dump(feeds, feedsjson)

Note that this will be slow to execute because you will rewrite the full contents of the file every time you call add. If you are calling it in a loop, consider adding all the feeds to a list in advance, then writing the list out in one go.

Answer 3

Append entry to the file contents if file exists, otherwise append the entry to an empty list and write in in the file:

a = []
if not os.path.isfile(fname):
    a.append(entry)
    with open(fname, mode='w') as f:
        f.write(json.dumps(a, indent=2))
else:
    with open(fname) as feedsjson:
        feeds = json.load(feedsjson)

    feeds.append(entry)
    with open(fname, mode='w') as f:
        f.write(json.dumps(feeds, indent=2))

Answer 4

Using a instead of w should let you update the file instead of creating a new one/overwriting everything in the existing file.

See this answer for a difference in the modes.

Answer 5

One possible solution is do the concatenation manually, here is some useful code:

import json
def append_to_json(_dict,path): 
    with open(path, 'ab+') as f:
        f.seek(0,2)                                #Go to the end of file    
        if f.tell() == 0 :                         #Check if file is empty
            f.write(json.dumps([_dict]).encode())  #If empty, write an array
        else :
            f.seek(-1,2)           
            f.truncate()                           #Remove the last character, open the array
            f.write(' , '.encode())                #Write the separator
            f.write(json.dumps(_dict).encode())    #Dump the dictionary
            f.write(']'.encode())                  #Close the array

You should be careful when editing the file outside the script not add any spacing at the end.

Answer 6

this, work for me :

with open('file.json', 'a') as outfile:
    outfile.write(json.dumps(data))
    outfile.write(",")
    outfile.close()

Answer 7

I have some code which is similar, but does not rewrite the entire contents each time. This is meant to run periodically and append a JSON entry at the end of an array.

If the file doesn't exist yet, it creates it and dumps the JSON into an array. If the file has already been created, it goes to the end, replaces the ] with a , drops the new JSON object in, and then closes it up again with another ]

# Append JSON object to output file JSON array
fname = "somefile.txt"
if os.path.isfile(fname):
    # File exists
    with open(fname, 'a+') as outfile:
        outfile.seek(-1, os.SEEK_END)
        outfile.truncate()
        outfile.write(',')
        json.dump(data_dict, outfile)
        outfile.write(']')
else: 
    # Create file
    with open(fname, 'w') as outfile:
        array = []
        array.append(data_dict)
        json.dump(array, outfile)

Answer 8

You aren't ever writing anything to do with the data you read in. Do you want to be adding the data structure in feeds to the new one you're creating?

Or perhaps you want to open the file in append mode open(filename, 'a') and then add your string, by writing the string produced by json.dumps instead of using json.dump - but nneonneo points out that this would be invalid json.

Answer 9

import jsonlines

object1 = {
               "name": "name1",
               "url": "url1"
          }

object2 = {
               "name": "name2",
               "url": "url2"
          }   


# filename.jsonl is the name of the file
with jsonlines.open("filename.jsonl", "a") as writer:   # for writing
    writer.write(object1)
    writer.write(object2)

with jsonlines.open('filename.jsonl') as reader:      # for reading
    for obj in reader:
        print(obj)             

visit for more info https://jsonlines.readthedocs.io/en/latest/

Answer 10

You can simply import the data from the source file, read it, and save what you want to append to a variable. Then open the destination file, assign the list data inside to a new variable (presumably this will all be valid JSON), then use the 'append' function on this list variable and append the first variable to it. Viola, you have appended to the JSON list. Now just overwrite your destination file with the newly appended list (as JSON).

The 'a' mode in your 'open' function will not work here because it will just tack everything on to the end of the file, which will make it non-valid JSON format.

Answer 11

let's say you have the following dicts

d1 = {'a': 'apple'}
d2 = {'b': 'banana'}
d3 = {'c': 'carrot'}

you can turn this into a combined json like this:

master_json = str(json.dumps(d1))[:-1]+', '+str(json.dumps(d2))[1:-1]+', '+str(json.dumps(d3))[1:]

therefore, code to append to a json file will look like below:

        dict_list = [d1, d2, d3]
        for i, d in enumerate(d_list):
            if i == 0:
                #first dict
                start = str(json.dumps(d))[:-1]
                with open(str_file_name, mode='w') as f:
                    f.write(start)
            else:
                with open(str_file_name, mode='a') as f:
                    if i != (len(dict_list) - 1):
                        #middle dicts
                        mid = ','+str(json.dumps(d))[1:-1]
                        f.write(mid)
                    else:
                        #last dict
                        end = ','+str(json.dumps(d))[1:]
                        f.write(end)

Answer 12

Use json.load, json.update, json.dump like so for instance:

    with open('data.json', mode='r') as file:
        data = json.load(file)

        new_record = {key: {inner_key:content}} # Whatever your structure is
        data.update(new_record)

    with open('data.json', mode='w') as file:
        data.dump(data, file, indent=4)