convert bash `ls` output to json array

Question

Is it possible to use a bash script to format the output of the ls to a json array? To be valid json, all names of the dirs and files need to be wrapped in double quotes, seperated by a comma, and the entire thing needs to be wrapped in square brackets. I.e. convert:

jeroen@jeroen-ubuntu:~/Desktop$ ls
foo.txt bar baz

to

[ "foo.txt", "bar", "baz" ]

edit: I strongly prefer something that works across all my Linux servers; hence rather not depend on python, but have a pure bash solution.

You're using the wrong tool for the job. If you are worried about python not being available, use perl. It should be on almost all linux servers. — gpojd, Apr 19 '12 at 18:36
Anyone know how to do this via the json manipulation tool jq? http://stedolan.github.io/jq/manual/ — rektide, Apr 19 '13 at 06:42
Now there's [jc](https://github.com/kellyjonbrazil/jc) from [@Kelly Brazil](https://stackoverflow.com/users/12303989/kelly-brazil), which is actually nice. Strange that [his reply got deleted](https://stackoverflow.com/a/58646499/537554). — ryenus, Oct 08 '21 at 03:08

score 34 · Answer 1 · edited Dec 24 '19 at 15:06

34

If you know that no filename contains newlines, use jq:

ls | jq -R -s -c 'split("\n")[:-1]'

Short explanation of the flags to jq:

-R treats the input as string instead of JSON
-s joins all lines into an array
-c creates a compact output
[:-1] removes the last empty string in the output array

This requires version 1.4 or later of jq. Try this if it doesn't work for you:

ls | jq -R '[.]' | jq -s -c 'add'

edited Dec 24 '19 at 15:06

Harm Berntsen

33
4

answered Sep 02 '15 at 13:26

bowel

357
1
3
2

error: split is not defined split("\n") ^^^^^ 1 compile error – Jose Ricardo Bustos M. Sep 02 '15 at 13:33
1

I have tried both `ls | jq -R -s -c 'split("\n")'` and `ls | jq -R '[.]' | jq -s -c 'add'`. The former is having an extra empty string "" added into the array. So the latter is a better solution. – Devy Sep 20 '16 at 18:57
This can be simplified to `ls | jq --raw-input | jq --slurp --compact-output`. Or if you prefer the short syntax: `ls | jq -R | jq -s -c` – StudioLE Jul 23 '23 at 09:50

score 30 · Answer 2 · answered Apr 19 '12 at 18:28

30

Yes, but the corner cases and Unicode handling will drive you up the wall. Better to delegate to a scripting language that supports it natively.

$ ls
あ  a  "a"  à  a b  私
$ python -c 'import os, json; print json.dumps(os.listdir("."))'
["\u00e0", "\"a\"", "\u79c1", "a b", "\u3042", "a"]

answered Apr 19 '12 at 18:28

Ignacio Vazquez-Abrams

776,304
153
1,341
1,358

Thanks, but really prefer something that generally works on any Linux server; so I rather have a pure Bash solution. – Jeroen Ooms Apr 19 '12 at 18:30
9

99.8% of the Linux servers out there have Python available; admittedly you may need to use simplejson though, for older versions of Python. – Ignacio Vazquez-Abrams Apr 19 '12 at 18:31
1

A server without python = Hell. You'll probably get a perl solution soon too. – KurzedMetal Apr 19 '12 at 18:35
Yeah not everyone has that. And I don't want to ask all users of my software to install additional repositories to be able to run a script. – Jeroen Ooms Apr 19 '12 at 18:45
So then provide links directly to the packages. – Ignacio Vazquez-Abrams Apr 19 '12 at 18:47

Tronix117 · Answer 3 · 2012-04-19T19:14:36.857

18

Hello you can do that with sed and awk:

ls | awk ' BEGIN { ORS = ""; print "["; } { print "\/\@"$0"\/\@"; } END { print "]"; }' | sed "s^\"^\\\\\"^g;s^\/\@\/\@^\", \"^g;s^\/\@^\"^g"

EDIT: updated to solve the problem with " and spaces. I use /@ as replacement pattern for ", since / is not a valid character for filename.

edited Apr 19 '12 at 19:14

answered Apr 19 '12 at 18:38

Tronix117

1,985
14
19

Note that this solution does not properly escape double-quotes which may be in the name, e.g. a file named `foo\"bar` comes out as `"foo"bar"` instead of `"foo\"bar"`. – Phrogz Apr 19 '12 at 18:50
also directories with a space in the name are problematic. – Jeroen Ooms Apr 19 '12 at 18:57
1

**-1**. This also suffers from the classic [Parsing LS](http://mywiki.wooledge.org/ParsingLs) problem. – ghoti Dec 19 '13 at 14:38
Could you elaborate on what exactly your awk and sed statements are doing? Specifically, the second print statement in your awk and the 3 sed statements. Here, `s^\/\@^\"^g`, I see that you're escaping the `/@` but I fail to see how you're inserting the commas in between all the entries piped from `ls`. – Sergio Feb 25 '14 at 19:54

score 14 · Accepted Answer · answered Apr 19 '12 at 19:23

14

Use perl as the encoder; it's guaranteed to be non-buggy, is everywhere, and with pipes, it's still reasonably clean:

ls | perl -e 'use JSON; @in=grep(s/\n$//, <>); print encode_json(\@in)."\n";'

answered Apr 19 '12 at 19:23

Dave

10,964
3
32
54

9

**-1**. This suffers from the classic [Parsing LS](http://mywiki.wooledge.org/ParsingLs) problem. Better to do it in a for loop and avoid the pipe entirely. See Glenn Jackman's answer for the correct approach. – ghoti Dec 19 '13 at 14:48
3

" it's guaranteed to be non-buggy, is everywhere".... except for you have to install another module.. So its not everywhere as such.. Sorry but ideal solutions involve already available components – Angry 84 Dec 31 '15 at 01:43
Confirmed, the module is not installed on Fedora 35 using the default config. `Can't locate JSON.pm in @INC (you may need to install the JSON module) (@INC contains: /usr/local/lib64/perl5/5.30 /usr/local/share/perl5/5.30 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5) at -e line 1. BEGIN failed--compilation aborted at -e line 1.` # +1 to the jq answer and simple to install on Fedora `sudo dnf -y install jq` – Stef Jan 30 '22 at 15:58

Adeel Ahmad · Answer 5 · 2016-02-17T09:19:27.077

2

Most of the Linux machine already has python. all you have to do is:

python -c 'import os, json; print json.dumps(os.listdir("/yourdirectory"))'

This is for . directory , you can add any path.

edited Feb 17 '16 at 09:19

answered Apr 08 '15 at 12:16

Adeel Ahmad

1,671
1
17
22

4

Your answer seems to be an exact duplicate of http://stackoverflow.com/a/10234470/817632 – Olga Aug 27 '15 at 13:24
How would I change this to follow directories also? – Ricky Barnett Feb 15 '16 at 10:35
you can do this for different directories using python -c 'import os, json; print json.dumps(os.listdir("/mydirectory/temp"))' – Adeel Ahmad Feb 17 '16 at 09:18
How do we handle this for only files? – Jayant Bhawal Jul 05 '16 at 14:22

TaoJoannes · Answer 6 · 2012-04-19T19:05:04.720

1

Should be pretty easy.

$ cat ls2json.bash
#!/bin/bash
echo -n '['
for FILE in $(ls | sed -e 's/"/\\"/g')
do
echo -n \"${FILE}\",
done
echo -en \\b']'

then run:

$ ./ls2json.bash > json.out

but python would be even easier

import os
directory = '/some/dir'
ls = os.listdir(directory)
dirstring = str(ls)
print dirstring.replace("'",'"')

edited Apr 19 '12 at 19:05

answered Apr 19 '12 at 18:32

TaoJoannes

594
5
14

Could you add the comma that seperates two strings? – Jeroen Ooms Apr 19 '12 at 18:34
oh, yeah, forgot in translation, fixed, problem there is that there will be a trailing comma. hmm. – TaoJoannes Apr 19 '12 at 18:35
1

Well the tricky thing is that there should not be a comma after the last element, right before the ] – Jeroen Ooms Apr 19 '12 at 18:37
You need to set a variable `set FIRST=true`, and at the end of the loop, set it to `false`. Output a comma before FILE when FIRST is false. – Lou Franco Apr 19 '12 at 18:41
easy fix, echo -en \\b ']', added – TaoJoannes Apr 19 '12 at 18:45
Note that this solution does not properly escape double-quotes which may be in the name, e.g. a file named `foo\"bar` comes out as `"foo"bar"` instead of `"foo\"bar"`. – Phrogz Apr 19 '12 at 18:50
true, it would give you "Stupid"Filename.txt" in that case. quick fix with sed. – TaoJoannes Apr 19 '12 at 19:04
**-1** for the the [Parsing LS](http://mywiki.wooledge.org/ParsingLs) problem. I had hope when I saw the for loop, but this way misses the mark. – ghoti Dec 19 '13 at 14:45

score 1 · Answer 7 · answered Apr 19 '12 at 18:35

1

Here's a bash line

echo '[' ; ls --format=commas|sed -e 's/^/\"/'|sed -e 's/,$/\",/'|sed -e 's/\([^,]\)$/\1\"\]/'|sed -e 's/, /\", \"/g'

Won't properly deal with ", \ or some commas in the name of the file. Also, if ls puts newlines between filenames, so will this.

answered Apr 19 '12 at 18:35

Lou Franco

87,846
14
132
192

Also won't deal with tabs, newlines or other control characters in filenames. – ghoti Dec 19 '13 at 14:42

score 1 · Answer 8 · edited Jul 25 '18 at 14:57

I was also searching for a way to output a Linux folder / file tree to some JSON or XML file. Why not use this simple terminal command:

$ tree --dirsfirst --noreport -n -X -i -s -D -f -o my.xml

so, just the linux tree command, and config your own parameters. Here -X gives XML output! For me, that's OK, and i guess there's some script to convert XML to JSON ..

NOTE: I think this covers the same question.

score 0 · Answer 9 · answered Apr 27 '22 at 14:06

0

Here's an elegant one-liner solution that doesn't rely on jq:

echo '[ "'"$(echo "$list" | sed ':a;N;$!ba;s/\n/", "/g')"'" ]'

$list here is a newline-separated string.

answered Apr 27 '22 at 14:06

silkfire

24,585
15
82
105

1

There's a lot more escaping than this needed to transform all possible filenames into valid JSON. Filenames can contain tab characters; they can contain literal newlines; they can contain literal quotes; they can contain backslashes; etc, etc, etc. – Charles Duffy Aug 12 '22 at 13:15

Marc Michel · Answer 10 · 2022-08-12T13:16:13.263

Using gnu column (i.e. doesn't work on OSX)

ls -ldG * --time-style=long-iso | column -t -n "$PWD" -N mod,links,user,size,date,time,name -J

Output :

{
   "/home/pouet": [
      {"mod":"-rwxr-xr-x", "links":"1", "user":"pouet", "size":"21978", "date":"2022-08-12", "time":"11:47", "name":"file1"},
      {"mod":"-rw-r--r--", "links":"1", "user":"pouet", "size":"2634", "date":"2022-06-20", "time":"11:14", "name":"file2"}
   ]
}

score 0 · Answer 11 · answered Jul 23 '23 at 09:57

For a simpler solution using jq:

ls | jq --raw-input | jq --slurp --compact-output

or if you prefer the shorter syntax:

ls | jq -R | jq -s -c

From the manual

https://jqlang.github.io/jq/manual/#Invokingjq

--raw-input / -R

Don't parse the input as JSON. Instead, each line of text is passed to the filter as a string. If combined with --slurp, then the entire input is passed to the filter as a single long string.

--slurp / -s

Instead of running the filter for each JSON object in the input, read the entire input stream into a large array and run the filter just once.

--compact-output / -c

By default, jq pretty-prints JSON output. Using this option will result in more compact output by instead putting each JSON object on a single line.

score -1 · Answer 12 · answered Apr 19 '12 at 18:23

-1

Personnaly, I would code script that would run the command ls, send the output to a file of you choice while parsing the output to make format it to a valid JSON format.

I'm sure that a simple Bash file will do the work.

Bash ouput

answered Apr 19 '12 at 18:23

Erwald

2,118
2
14
20

score -1 · Answer 13 · answered Apr 19 '12 at 18:28

-1

Can't you use a python script like this?

myOutput = subprocess.check_output["ls"]
output = ["+str(e)+" for e in myOutput]
return output

I didn't check if it works, but you can find the specification here

answered Apr 19 '12 at 18:28

DonCallisto

29,419
9
72
100

score -3 · Answer 14 · answered Apr 19 '12 at 18:41

-3

Don't use bash, use a scripting language. Untested perl example:

use JSON;
my @ls_output = `ls`; ## probably better to use a perl module to do this, like DirHandle
print encode_json( @ls_output );

answered Apr 19 '12 at 18:41

gpojd

22,558
8
42
71

convert bash `ls` output to json array

14 Answers14

Linked

Related