-3

I have a json file (1m+ rows)

[{...},
{...},
{...},
{...}]

I want to remove the square brackets at start and end and also each commas at end of line. I came across a sed command that will add it but I want to do the opposite?

sed '1s/^/[/;$!s/$/,/;$s/$/]/' file

Ideally, it will take in an input file and output it to a new file...

expected output:

{...}
{...}
{...}
{...}

Any ideas?

Saffik
  • 911
  • 4
  • 19
  • 45
  • Just load the json file using ```json``` module & start writing to the file each dict object. – sushanth May 20 '20 at 14:07
  • 1
    `sed 's/^\[//g; s/\]$//g; s/,$//g' file` – George Vasiliou May 20 '20 at 14:15
  • The [XY](http://meta.stackexchange.com/a/66378) problem is asking about your attempted solution rather than your actual problem. – Cyrus May 20 '20 at 14:25
  • I’m voting to close this question because it asks for a solution in either of two unrelated programming languages. That's not how [so] works. – oguz ismail May 20 '20 at 14:28
  • @oguzismail - I asked if this is possible using python? How is this unrelated? – Saffik May 20 '20 at 14:47
  • You didn't ask that though, you showed your attempt in sed, and added python to tags and the title. That -to me- implies that you're asking someone to write a program for you. That's highly discouraged here – oguz ismail May 20 '20 at 15:00
  • I do apologize if it came across like that but you could just suggest it rather than be dismissive and threaten to close this post. – Saffik May 20 '20 at 15:06
  • 1
    @oguzismail Reference for your claim? I can't imagine that is the case, and I cannot find reference to it anywhere in the Help Center. I cannot imagine being unable to ask a question about Django+Javascript, PHP+Javascript, Python+C, etc. You linked to the StackOverflow main homepage, not a definite "do not" page. Seems like it's your personal preference to not have a question share multiple tags... – felipe May 20 '20 at 15:27
  • @Felipe does the question contain any code written in python? No. Does it ask for a python solution? Yes. That's a downvote from me to both the question and answer proposing a solution using python, and a close vote. If you think I'm in the wrong here, create a post at [meta] about it. I might have chosen wrong words/phrases to describe the problem with OP, but that doesn't make me less right – oguz ismail May 20 '20 at 16:13

2 Answers2

1
import json

data = json.loads('[{"key": "value"}, {"key": "value"}]')

file = ""
for dictionary in data:
    file += f"{dictionary}\n"

with open('file.txt', 'w') as f:
    f.write(file)

Inside file.txt you will see:

{'key': 'value'}
{'key': 'value'}
felipe
  • 7,324
  • 2
  • 28
  • 37
  • I think your solution is great! - If possible, any chance you could edit it slightly so it takes input (opens a file e.g `input.json`) and does its thing and outputs to a new file `output.txt` – Saffik May 20 '20 at 15:49
  • You should be able to adapt [this](https://stackoverflow.com/a/20199213/1305461) with the answer above to do what you ultimately need it to do. :) – felipe May 21 '20 at 17:16
1

If you generate the file using python, of course writing it in your desired format from the first place, is better than first writing it wrong, and then patching it up using sed.

But, just to show you it's easy in sed:

sed 's/^\[//;s/,$//;s/\]$//'

I also agree with @oguzismail that you seem to be asking for someone else to write your program (like I just did for you). This is discouraged because the purpose of stackoverflow is to spread knowledge. Most people don't learn very well when presented with a ready to use solution to their problem.

Huub van Eijndhoven
  • 41
  • 4
  • Any idea what this would be? If I run your command `SED file > file2` i get an error message: `sed: regex input buffer length larger than INT_MAX` (the file has a couple of million lines.) – Saffik May 21 '20 at 08:44
  • Did you test by running it on a small example file (I did, and it runs perfectly.) ? Does your real file have new-line separated lines ? Or is your real file actually totally different ?... – Huub van Eijndhoven May 22 '20 at 15:50
  • My real file has 7.1million rows, maybe....far too many for SED to work. :( – Saffik May 22 '20 at 19:47
  • No, sed can work with any file size. The problem is (absense of a) line separator. Did you open your file with an editor, and look at it? Does it have "lines" separated by "new-line" characters? – Huub van Eijndhoven May 23 '20 at 06:53
  • I just ran it on a test file of 13 million lines. It works and finished in a few secs. Again: your problem is NOT file size. But LINE size. – Huub van Eijndhoven May 23 '20 at 07:01