JSON file isn't finished writing to by the time I load it, behave BDD

Question

My program is writing to a JSON file, and then loading, reading, andPOSTing it. The writing part is being done by the behave BDD.

# writing to the JSON file is done by behave
data = json.load(open('results.json', 'r'))
r = requests.post(MyAPIEndpoint, json=data)

I'm running into an issue since the writing is not being completed before I begin loading. (It's missing the closing [ after the final {.)

HOOK-ERROR in after_all: ValueError: Expecting object: line 2 column 2501 (char 2502)

Is there a way of getting by this, either by changing something with my call to behave's __main__ or by a change in how or when I'm loading the JSON file?

score 1 · Answer 1 · answered Apr 20 '18 at 22:53

i think the problem here is little mixed, in one part, you can wait to the file finish to be written, and close it when is not in use, you can do that inside your code or somwthing like this check if a file is open in Python

In other part, for the computer, the data is data, don't will analysis that, i means, you know where is the error, because you analise that, when you think, so for you is obvs, but for the computer isn't obvs, how much errores there is?, and where or the structure of it?, is there all the data we need?, for a computer is hard know all of this, you will need write a program to can deduce all of this data and check it.

If your program uses multiples results, i think the better ways is use temp files, so you can freely create one, write, check when is ready and use it, and don't will care if you have other similar process using it.

Other way, check if the json is valid before call it, Python: validate and format JSON files, when is valid load it.

Hope can help.

Cya.

Yeah, you're right. Even with analyzing behave's code, this is difficult to debug and in my opinion overkill. As for loading the JSON file, I've moved the reading to the top level of my program, rather than at the level of the behave framework. — natn2323, Apr 25 '18 at 19:00

score 1 · Answer 2 · answered Apr 20 '18 at 23:35

One way to address this problem is to change your file format from being JSON at the top level to newline-delimited JSON (NDJSON), also called line-delimited JSON (LDJSON) or JSON lines (JSONL).

https://en.wikipedia.org/wiki/JSON_streaming#Line-delimited_JSON

For example, this JSON file:

{
  "widgets": [
    {"name": "widget1", "color": "red"},
    {"name": "widget2", "color": "green"},
    {"name": "widget3", "color": "blue"}
  ]
}

Would become this NDJSON file:

{"name": "widget1", "color": "red"}
{"name": "widget2", "color": "green"}
{"name": "widget3", "color": "blue"}

It's especially useful in the context of streaming data, which kind of sounds like the use case you have where you might have one process writing to a file continuously while another is reading it.

You could then read the NDJSON file like so:

import json
from pprint import pprint

with open('widgets.json') as f:
    all_lines = [json.loads(l) for l in f.readlines()]
    all_data = {'widgets': all_lines}
    pprint(all_data)

Output:

{'widgets': [{'color': 'red', 'name': 'widget1'},
             {'color': 'green', 'name': 'widget2'},
             {'color': 'blue', 'name': 'widget3'}]}

After reading your explanation, strangely enough it still gives the same error when I try to open the file as before. Per my framework, my workaround is to read the file in my "main" method after behave is finished writing, instead of within behave's after_all() function (at which point, writing is seemingly incomplete). — natn2323, Apr 25 '18 at 19:00

JSON file isn't finished writing to by the time I load it, behave BDD

2 Answers2