What's the fastest way to count files?

Question

We just installed a mass filer device that we want to store data from our suppliers on. They FTP files to it every day and right now, we've about 60,000+ files.

Doing find . -name '*TXT' -exec ls '{}' \; | wc -l will work albeit very slowly.

Is there a faster way to count files?

score 7 · Accepted Answer · answered Aug 21 '12 at 16:21

7

Why not just

find . -name '*TXT' | wc -l

? Your current command is unnecessarily spawning ls for each file, and that process spawning in itself will be very slow. A quick test on my system would suggest a 40x speed up.

answered Aug 21 '12 at 16:21

Brian Agnew

268,207
37
334
440

Brian, your explanation makes complete sense. No wonder it's so slow. Thank you. Accepted and voted up. – Chris Aug 21 '12 at 16:24
Be carefull as filenames containing new line charcters would count as more than 1 – Marco Carlo Moriggi Jul 21 '23 at 12:28
I agree with you, but I don't think I've ever seen a filename in the wild that has a newline – Brian Agnew Aug 04 '23 at 13:45

geirha · Answer 2 · 2023-08-02T08:31:25.653

3

With GNU find, this might be slightly faster:

find . -name '*.TXT' -printf x | wc -m

Or with POSIX

find . -name '*.TXT' -exec printf x%.s {} + | wc -m

edited Aug 02 '23 at 08:31

answered Aug 21 '12 at 16:45

geirha

5,801
1
30
35

almost correct and performant, but the correct flag to use to count characters is `wc -m` – Marco Carlo Moriggi Jul 21 '23 at 12:40
This approach also counts the current directory. To avoid this, add `-mindepth 1`. Another tip: the correct flag to use to count characters is `wc -m`, as `wc -c` counts bytes (almost the same for ascii chars, but can lead to unexpected results if not properly handled) – Marco Carlo Moriggi Jul 21 '23 at 12:47
1

@MarcoCarloMoriggi Fair enough, might as well count chars rather than bytes, but it does not count the current directory, since current directory (`.`) does not match `*.TXT`. – geirha Aug 02 '23 at 08:34

score 1 · Answer 3 · answered Aug 21 '12 at 16:20

1

How about

find . -name '*TXT' | wc -l

answered Aug 21 '12 at 16:20

Stephane Rouberol

4,286
19
18

glenn jackman · Answer 4 · 2013-06-28T19:37:13.403

1

with bash:

shopt -s globstar nullglob

and then, either

files=(**/*.TXT)
nfiles=${#files[@]}

or

nfiles=$( printf "%s\n" **/*.TXT | wc -l )

edited Jun 28 '13 at 19:37

answered Aug 21 '12 at 19:54

glenn jackman

238,783
38
220
352

score 0 · Answer 5 · answered Jul 21 '23 at 12:52

There is plenty of questions about counting files. Just for naming a few:

Trying to summarize as much as I can

use this approach:

find . -mindepth 1 -name '*TXT' -printf x | wc -m

The explanantion is simple: print an "x" for each item found in the current directory (recoursively, so if you don't need to recurse into inner subfolder, simply add -maxdepth 1), then count the characters. This way you no longer have to deal with odd file names and will get a huge improvement in time and memory.

If you are not yet conviced see the explanation of the command on explainshell

What's the fastest way to count files?

5 Answers5

Trying to summarize as much as I can