How to split text file into multiple files and extract filename from line prefix?

Question

I have a simple log file with content like:

1504007980.039:{"key":"valueA"}
1504007990.359:{"key":"valueB", "key2": "valueC"}
...

That I'd like to output to multiple files that each have as content the JSON part that comes after the timestamp. So I would get as a result the files:

1504007980039.json
1504007990359.json
...

This is similar to How to split one text file into multiple *.txt files? but the name of the file should be extracted from each line (and remove an extra dot), and not generated via an index

Preferably I'd want a one-liner that can be executed in bash.

*with valid JSON context* ? show how should look that valid content in regard to your input — RomanPerekhrest, Aug 29 '17 at 12:44
Ah, sorry, no need to validate actually anything: the data that comes after the timestamp is known to be valid. — ttiurani, Aug 29 '17 at 12:45

Ed Morton · Accepted Answer · 2017-08-29T15:05:06.397

2

Since you aren't using GNU awk you need to close output files as you go to avoid the "too many open files" error. To avoid that and issues around specific values in your JSON and issues related to undefined behavior during output redirection, this is what you need:

awk '{
    fname = $0
    sub(/\./,"",fname)
    sub(/:.*/,".json",fname)
    sub(/[^:]+:/,"")
    print >> fname
    close(fname)
}' file

You can of course squeeze it onto 1 line if you see some benefit to that:

awk '{f=$0;sub(/\./,"",f);sub(/:.*/,".json",f);sub(/[^:]+:/,"");print>>f;close(f)}' file

edited Aug 29 '17 at 15:05

answered Aug 29 '17 at 14:51

Ed Morton

188,023
17
78
185

Do you mean to say that without the "close(fname)" I'll generate into too many lingering file handles? – ttiurani Aug 29 '17 at 14:58
Exactly. gawk handles that for you as you go but you're not using gawk so you will have to deal with that issue. – Ed Morton Aug 29 '17 at 14:59
1

Thanks a lot for this! I don't have the large log file yet, but would have run into that open files problem later on. So this is now accepted. – ttiurani Aug 29 '17 at 15:18

RomanPerekhrest · Answer 2 · 2017-08-29T15:13:25.570

1

awk solution:

awk '{ idx=index($0,":"); fn=substr($0,1,idx-1)".json"; sub(/\./,"",fn); 
       print substr($0,idx+1) > fn; close(fn) }' input.log

idx=index($0,":") - capturing index of the 1st :
fn=substr($0,1,idx-1)".json" - preparing filename

Viewing results (for 2 sample lines from the question):

for f in *.json; do echo "$f"; cat "$f"; echo; done

The output (filename -> content):

1504007980039.json
{"key":"valueA"}

1504007990359.json
{"key":"valueB"}

edited Aug 29 '17 at 15:13

answered Aug 29 '17 at 12:53

RomanPerekhrest

88,541
4
65
105

@RomanPerekhrest: The `cat` usage can be removed here with a file re-direction `echo "$( < ${f} )"` – Inian Aug 29 '17 at 13:05

How to split text file into multiple files and extract filename from line prefix?

2 Answers2