How to extract JSON strings from text file?

Question

I need to post-process text file which has normal text and JSON strings like example below:

Result.txt:

Name: Sham
good student
('{"result": {"School name": "abc", roll no": 2, "Class": "7B", "Marks": '
 '310.55, "class_average_percentage": 81.523, "percentage": 91'}}')

Name: Ram
need to improve
('{"result": {"School name": "abc", "roll no": 3, "Class": "8A", "Marks": '
 '250.89, "class_average_percentage": 86.23, "percentage": 68'}}')

Name: Roy
average student
('{"result": {"School name": "abc", "roll no": 9, "Class": "7B", "Marks": '
 '298.45, "class_average_percentage": 81.523, "percentage": 73'}}')

How to parse file to assign JSON strings to corresponding names?

Example:

Sham = ('{"School name": "abc", "result": {"roll no": 2, "Class": "7B", "Marks": '
        '310.55, "class_average_percentage": 81.523, "percentage": 91'}}')

my suggestion is to use `regex` to extract the information you need (i.e. student name, "good student", etc). After that, you can try using `ast.literal_eval` https://stackoverflow.com/questions/988228/convert-a-string-representation-of-a-dictionary-to-a-dictionary to convert the strings to json — Thu Ya Kyaw, Apr 12 '22 at 07:04
Have you considered loading the data (once it's been fixed) into a dictionary keyed on student name? — DarkKnight, Apr 12 '22 at 07:41
@IamThuYa If its a json why use regex????? A simple deserialize should do the job. regex is not a good solution to this — Josip Juros, Apr 12 '22 at 08:39
Even if you were able parse the file and get the JSON strings, you shouldn't be trying to dynamically create variables. One reason why is because there is no way for the rest of your code to know what variable names to use. — martineau, Apr 12 '22 at 08:40
@JosipJuros it is not a json file. It's a text file with strings that have json format — Thu Ya Kyaw, Apr 12 '22 at 10:52

martineau · Answer 1 · 2022-04-12T11:40:01.780

The JSON in the text file is not in the proper format. The ends of each of the supposed JSON format strings, such as … 91'}}' and … 68'}}' are invalid. It should be like this: … 91}}'.

Here's the corrected file:

Name: Sham
good student
('{"result": {"School name": "abc", "roll no": 2, "Class": "7B", "Marks": '
 '310.55, "class_average_percentage": 81.523, "percentage": 91}}')

Name: Ram
need to improve
('{"result": {"School name": "abc", "roll no": 3, "Class": "8A", "Marks": '
 '250.89, "class_average_percentage": 86.23, "percentage": 68}}')

Name: Roy
average student
('{"result": {"School name": "abc", "roll no": 9, "Class": "7B", "Marks": '
 '298.45, "class_average_percentage": 81.523, "percentage": 73}}')

I was able to parse the fixed version of your Result.txt shown above via something called a FSM (Finite-State-Machine) because they're able to parse patterns such as what's in the contents of this file. In a nutshell, they're are always in one of a finite number of states and change or transition from one state to another based on the input they're fed iteratively (such as lines of a text file).

To handle the JSON data, the code gathers the lines together comprising the JSON string which appears to be in Python literal string format, then it joins them all and applies ast.literal_eval() to them to obtain the string value. It then uses json.loads() to convert that into a Python dictionary, and then converts a nested dictionary in that intermediate result back into a JSON string using json.dumps() which it adds the student_info dictionary being built.

While at that point it would be possible to use the student's name to create a variable and assign the JSON string to it, that is generally considered a poor practice — so instead, the code creates a dictionary named student_info to hold the results. The keys of the dictionary are the student's name and value associated with each on it the JSON format string in the file.

from ast import literal_eval
import json


filepath = './Result.txt'
student_info = {}

with open(filepath) as file:
    state = 0
    while True:
        try:
            line = next(file)
        except StopIteration:  # End-Of-File
            break

        if not (line := line.strip()):  # Skip blank lines.
            continue

        if state == 0:
            if line.startswith('Name:'):  # First line of student data?
                name = line.split()[1]
                json_lines = []
                state = 1

        elif state == 1:
            if line.startswith('('):  # First JSON line?
                json_lines.append(line)
                state = 2

        elif state == 2:
            json_lines.append(line)
            if line.endswith(')'):  # Last JSON line?
                json_str = literal_eval(''.join(json_lines))
                result_dict = json.loads(json_str)
                as_json = json.dumps(result_dict['result'])
                student_info[name] = f'({repr(as_json)})'
                state = 0

for student, json_str in student_info.items():
    print(f'{student}: {json_str}')

Printed results:

Sham: ('{"School name": "abc", "roll no": 2, "Class": "7B", "Marks": 310.55, "class_average_percentage": 81.523, "percentage": 91}')
Ram: ('{"School name": "abc", "roll no": 3, "Class": "8A", "Marks": 250.89, "class_average_percentage": 86.23, "percentage": 68}')
Roy: ('{"School name": "abc", "roll no": 9, "Class": "7B", "Marks": 298.45, "class_average_percentage": 81.523, "percentage": 73}')

score 0 · Answer 2 · answered Apr 12 '22 at 12:38

The input file has a critical fault in that it's missing a double quote. So let's fix that giving:

Name: Sham
good student
('{"result": {"School name": "abc", "roll no": 2, "Class": "7B", "Marks": '
 '310.55, "class_average_percentage": 81.523, "percentage": 91'}}')

Name: Ram
need to improve
('{"result": {"School name": "abc", "roll no": 3, "Class": "8A", "Marks": '
 '250.89, "class_average_percentage": 86.23, "percentage": 68'}}')

Name: Roy
average student
('{"result": {"School name": "abc", "roll no": 9, "Class": "7B", "Marks": '
 '298.45, "class_average_percentage": 81.523, "percentage": 73'}}')

The structure of the file seems to be that the data are spread over groups of 5 lines. The first has the student name, the second is ignored, the significant data are split across two lines then there is a blank line.

The data are not valid JSON due to extraneous single-quotes but it looks as though they can simply be removed. This leads us to:

import json

name = None
result_dictionary = dict()
js = None

with open('Result.txt') as results:
    for line in map(str.strip, results):
        if line.startswith('Name'):
            _, name = line.split()
        elif line.startswith('('):
            js = line[2:]
        else:
            if js:
                js = (js + line[:-1]).replace("'", "")
                result_dictionary[name] = json.loads(js)
                js = None
                

print(json.dumps(result_dictionary, indent=2))

Output:

{
  "Sham": {
    "result": {
      "School name": "abc",
      "roll no": 2,
      "Class": "7B",
      "Marks": 310.55,
      "class_average_percentage": 81.523,
      "percentage": 91
    }
  },
  "Ram": {
    "result": {
      "School name": "abc",
      "roll no": 3,
      "Class": "8A",
      "Marks": 250.89,
      "class_average_percentage": 86.23,
      "percentage": 68
    }
  },
  "Roy": {
    "result": {
      "School name": "abc",
      "roll no": 9,
      "Class": "7B",
      "Marks": 298.45,
      "class_average_percentage": 81.523,
      "percentage": 73
    }
  }
}

This allows simple access to each student's data by name

How to extract JSON strings from text file?

2 Answers2