Quick ls command

Question

I've got to get a directory listing that contains about 2 million files, but when I do an ls command on it nothing comes back. I've waited 3 hours. I've tried ls | tee directory.txt, but that seems to hang forever.

I assume the server is doing a lot of inode sorting. Is there any way to speed up the ls command to just get a directory listing of filenames? I don't care about size, dates, permission or the like at this time.

score 52 · Answer 1 · edited Feb 24 '22 at 10:42

52

ls -U

will do the ls without sorting.

Another source of slowness is --color. On some linux machines, there is a convenience alias which adds --color=auto' to the ls call, making it look up file attributes for each file found (slow), to color the display. This can be avoided by ls -U --color=never or \ls -U.

edited Feb 24 '22 at 10:42

j13r

2,576
2
21
28

answered Sep 02 '08 at 18:25

Paul Tomblin

179,021
58
319
408

1

Do you know if `ls -U|sort` is faster than `ls`? – User1 Oct 20 '10 at 14:21
1

I don't know. I doubt it, because sort can't complete until it's seen all the records, whether it's done in a separate program in in `ls`. But the only way to find out is to test it. – Paul Tomblin Oct 20 '10 at 14:35
4

Note: on some systems, `ls -f` is equivalent to `ls -aU`; i.e., include _all_ files (even those whose names begin with ‘.’) and don’t sort. And on _some_ systems, `-f` is ***the*** option to suppress sorting, and `-U` does something else (or nothing). – Scott - Слава Україні Mar 05 '13 at 15:43
1

Does not work on BSD. On BSD `-U` sorts by file creation time. – rustyx Jul 20 '18 at 13:37

score 22 · Answer 2 · answered Nov 18 '15 at 16:37

22

I have a directory with 4 million files in it and the only way I got ls to spit out files immediately without a lot of churning first was

ls -1U

answered Nov 18 '15 at 16:37

stu

8,461
18
74
112

3

Saved my day! Thanks! – dschu Nov 04 '16 at 14:58
Absolutely crucial for a huge folder on a network-mounted drive (such as Google Drive on google-drive-ocamlfuse) – masterxilo Jan 22 '21 at 20:43
ls -1f seems a lot better than ls -1U for me. They are both similar in output speed, but ls -1U seems un-interuptable. – Ben Farmer Dec 06 '21 at 23:24
1

uninterruptable? it's writing output to a terminal, any attempt to cancel/ctrl-c/etc would have more to do with your terminal than with ls. – stu Dec 08 '21 at 01:01

score 15 · Answer 3 · edited Oct 28 '14 at 17:50

15

Try using:

find . -type f -maxdepth 1

This will only list the files in the directory, leave out the -type f argument if you want to list files and directories.

edited Oct 28 '14 at 17:50

the Tin Man

158,662
42
215
303

answered Sep 02 '08 at 18:23

Ryan Ahearn

7,886
7
51
56

score 9 · Answer 4 · answered Nov 02 '14 at 13:38

This question seems to be interesting and I was going through multiple answers that were posted. To understand the efficiency of the answers posted, I have executed them on 2 million files and found the results as below.

$ time tar cvf /dev/null . &> /tmp/file-count

real    37m16.553s
user    0m11.525s
sys     0m41.291s

------------------------------------------------------

$ time echo ./* &> /tmp/file-count

real    0m50.808s
user    0m49.291s
sys     0m1.404s

------------------------------------------------------

$ time ls &> /tmp/file-count

real    0m42.167s
user    0m40.323s
sys     0m1.648s

------------------------------------------------------

$ time find . &> /tmp/file-count

real    0m2.738s
user    0m1.044s
sys     0m1.684s

------------------------------------------------------

$ time ls -U &> /tmp/file-count

real    0m2.494s
user    0m0.848s
sys     0m1.452s


------------------------------------------------------

$ time ls -f &> /tmp/file-count

real    0m2.313s
user    0m0.856s
sys     0m1.448s

------------------------------------------------------

To summarize the results

ls -f command ran a bit faster than ls -U. Disabling color might have caused this improvement.
find command ran third with an average speed of 2.738 seconds.
Running just ls took 42.16 seconds. Here in my system ls is an alias for ls --color=auto
Using shell expansion feature with echo ./* ran for 50.80 seconds.
And the tar based solution took about 37 miuntes.

All tests were done seperately when system was in idle condition.

One important thing to note here is that the file lists are not printed in the terminal rather they were redirected to a file and the file count was calculated later with wc command. Commands ran too slow if the outputs where printed on the screen.

Any ideas why this happens ?

the terminal is slow, is has to scroll and do formatting, file writes go to block devices, and in reality, they go to the page cache first, so you're really just writing to memory, which is quicker than a terminal. — stu, Aug 08 '21 at 13:13

score 6 · Answer 5 · edited Aug 27 '13 at 11:51

6

This would be the fastest option AFAIK: ls -1 -f.

-1 (No columns)
-f (No sorting)

edited Aug 27 '13 at 11:51

toro2k

19,020
7
64
71

answered Aug 27 '13 at 11:31

plasmafire

61
1
1

This works for both `macOS` (BSD) and `Linux` – TiLogic Jun 03 '21 at 13:55

score 6 · Answer 6 · answered Nov 01 '14 at 16:09

Using

ls -1 -f

is about 10 times faster and it is easy to do (I tested with 1 million files, but my original problem had 6 800 000 000 files)

But in my case I needed to check if some specific directory contains more than 10 000 files. If there were more than 10 000 files, I am not anymore interested that how many files there is. I just quit the program so that it will run faster and wont try to read the rest one-by-one. If there are less than 10 000, I will print the exact amount. Speed of my program is quite similar to ls -1 -f if you specify bigger value for parameter than amount of files.

You can use my program find_if_more.pl in current directory by typing:

find_if_more.pl 999999999

If you are just interested if there are more than n files, script will finish faster than ls -1 -f with very large amount of files.

#!/usr/bin/perl
    use warnings;
    my ($maxcount) = @ARGV;
    my $dir = '.';
    $filecount = 0;
    if (not defined $maxcount) {
      die "Need maxcount\n";
    }
    opendir(DIR, $dir) or die $!;
    while (my $file = readdir(DIR)) {
        $filecount = $filecount + 1;
        last if $filecount> $maxcount
    }
    print $filecount;
    closedir(DIR);
    exit 0;

score 5 · Answer 7 · edited Oct 28 '14 at 17:50

5

You can redirect output and run the ls process in the background.

ls > myls.txt &

This would allow you to go on about your business while its running. It wouldn't lock up your shell.

Not sure about what options are for running ls and getting less data back. You could always run man ls to check.

edited Oct 28 '14 at 17:50

the Tin Man

158,662
42
215
303

answered Sep 02 '08 at 18:23

Eric

107
4

score 4 · Answer 8 · answered Sep 01 '10 at 16:03

This is probably not a helpful answer, but if you don't have find you may be able to make do with tar

$ tar cvf /dev/null .

I am told by people older than me that, "back in the day", single-user and recovery environments were a lot more limited than they are nowadays. That's where this trick comes from.

score 3 · Answer 9 · answered Sep 02 '08 at 18:24

3

I'm assuming you are using GNU ls? try

\ls

It will unalias the usual ls (ls --color=auto).

answered Sep 02 '08 at 18:24

wbkang

2,136
1
15
22

True, coloring is the usual culprit for me: when coloring, `ls` tries to determine type and mode of each directory entry, resulting in lots of `stat(2)` calls, thus in loads of disk activity. – Ruslan May 27 '17 at 05:38

score 2 · Answer 10 · answered Sep 02 '08 at 20:17

If a process "doesn't come back", I recommend strace to analyze how a process is interacting with the operating system.

In case of ls:

$strace ls

you would have seen that it reads all directory entries (getdents(2)) before it actually outputs anything. (sorting… as it was already mentioned here)

score 1 · Answer 11 · answered Oct 30 '09 at 16:01

Some followup: You don't mention what OS you're running on, which would help indicate which version of ls you're using. This probably isn't a 'bash' question as much as an ls question. My guess is that you're using GNU ls, which has some features that are useful in some contexts, but kill you on big directories.

GNU ls Trying to have prettier arranging of columns. GNU ls tries to do a smart arrange of all the filenames. In a huge directory, this will take some time, and memory.

To 'fix' this, you can try:

ls -1 # no columns at all

find BSD ls someplace, http://www.freebsd.org/cgi/cvsweb.cgi/src/bin/ls/ and use that on your big directories.

Use other tools, such as find

score 1 · Answer 12 · edited Oct 28 '14 at 17:47

1

There are several ways to get a list of files:

Use this command to get a list without sorting:

ls -U

or send the list of files to a file by using:

ls /Folder/path > ~/Desktop/List.txt

edited Oct 28 '14 at 17:47

the Tin Man

158,662
42
215
303

answered Oct 10 '14 at 08:50

Pouya

109
1
8

score 1 · Answer 13 · edited Oct 28 '14 at 17:51

1

How about find ./ -type f (which will find all files in the currently directory)? Take off the -type f to find everything.

edited Oct 28 '14 at 17:51

the Tin Man

158,662
42
215
303

answered Sep 02 '08 at 18:20

Mark Biek

146,731
54
156
201

This would find files in the current directory, and also in any subdirectories. – mwfearnley Oct 11 '17 at 15:46

score 1 · Answer 14 · answered Sep 02 '08 at 18:23

1

Things to try:

Check ls isn't aliased?

alias ls

Perhaps try find instead?

find . \( -type d -name . -prune \) -o \( -type f -print \)

Hope this helps.

answered Sep 02 '08 at 18:23

toolkit

49,809
17
109
135

score 1 · Answer 15 · edited Oct 28 '14 at 17:49

1

What partition type are you using?

Having millions of small files in one directory it might be a good idea to use JFS or ReiserFS which have better performance with many small sized files.

edited Oct 28 '14 at 17:49

the Tin Man

158,662
42
215
303

answered Sep 02 '08 at 18:56

Tanj

1,354
1
15
25

score 0 · Answer 16 · edited Oct 12 '17 at 02:56

0

You should provide information about what operating system and the type of filesystem you are using. On certain flavours of UNIX and certain filesystems you might be able to use the commands ff and ncheck as alternatives.

edited Oct 12 '17 at 02:56

mwfearnley

3,303
2
34
35

answered Sep 02 '08 at 19:55

tonylo

3,311
3
28
27

score 0 · Answer 17 · answered Apr 27 '20 at 22:51

0

I had a directory with timestamps in the file names. I wanted to check the date of the latest file and found find . -type f -maxdepth 1 | sort | tail -n 1 to be about twice as fast as ls -alh.

answered Apr 27 '20 at 22:51

blacksite

12,086
10
64
109

score -1 · Answer 18 · answered Sep 03 '08 at 02:38

-1

Lots of other good solutions here, but in the interest of completeness:

echo *

answered Sep 03 '08 at 02:38

jj33

7,543
2
38
42

3

With 2 million files, that is likely to return only a "command line too long" error. – richq Sep 08 '08 at 07:35

score -2 · Answer 19 · answered Sep 16 '08 at 19:00

-2

You can also make use of xargs. Just pipe the output of ls through xargs.

ls | xargs

If that doesn't work and the find examples above aren't working, try piping them to xargs as it can help the memory usage that might be causing your problems.

answered Sep 16 '08 at 19:00

Jim

1,217
10
16

Quick ls command

19 Answers19

Linked