how to couple xargs with pdftotext converter to search inside multiple pdf files

Question

I am making a script which is supposed to search inside all the pdf files in a directory. I have found one converted named "pdftotext" which enables me to use grep on pef files, but I am able to run it only with one file. When I want to run it over all the files present in directory then it fails. Any suggestions ?

This works:for a single file

pdftotext my_file.pdf - | grep 'hot'

This fails: for searching pdf files and converting to text and greping

SHELL PROMPT>find ~/.personal/tips -type f -iname "*" | grep -i "*.pdf" | xargs pdftotext |grep admin
pdftotext version 3.00
Copyright 1996-2004 Glyph & Cog, LLC
Usage: pdftotext [options] <PDF-file> [<text-file>]
  -f <int>          : first page to convert
  -l <int>          : last page to convert
  -layout           : maintain original physical layout
  -raw              : keep strings in content stream order
  -htmlmeta         : generate a simple HTML file, including the meta information
  -enc <string>     : output text encoding name
  -eol <string>     : output end-of-line convention (unix, dos, or mac)
  -nopgbrk          : don't insert page breaks between pages
  -opw <string>     : owner password (for encrypted files)
  -upw <string>     : user password (for encrypted files)
  -q                : don't print any messages or errors
  -cfg <string>     : configuration file to use in place of .xpdfrc
  -v                : print copyright and version info
  -h                : print usage information
  -help             : print usage information
  --help            : print usage information
  -?                : print usage information
SHELL PROMPT 139>

Answer 1

xargs is the wrong tool for this job: find does everything you need built-in.

find ~/.personal/tips \
    -type f \
    -iname "*.pdf" \
    -exec pdftotext '{}' - ';' \
  | grep hot

---

That said, if you did want to use xargs for some reason, correct usage would look something like...

find ~/.personal/tips \
    -type f \
    -iname "*.pdf" \
    -print0 \
  | xargs -0 -J % -n 1 pdftotext % - \
  | grep hot

Note that:

• The find command uses -print0 to NUL-delimit its output
• The xargs command uses -0 to NUL-delimit its input (which also turns off some behavior which would lead to incorrect handling of filenames with whitespace in their names, literal quote characters, etc).
• The xargs command uses -n 1 to call pdftotext once per file
• The xargs command uses -J % to specify a sigil for where the replacement should happen, and uses that % in the pdftotext command line appropriately.

Answer 2

find . -name '*.pdf' -print0 | xargs -0 -n1 -I '{}' pdftotext '{}' -

By default, xargs will try to fit as many lines on the command line for pdftotext as possible. You don't want that. What you want is one file per invocation followed by '-'. This you can achieve with -n1 (limit to one argument per invocation) and -I '{}' (make {} to be the placeholder for where the argument will fit).

The -print0 option to find coupled with the -0 options to xargs makes both use '\0' (null bytes) instead of newlines ('\n') as argument separators.

Xargs with -n1 and -I{} used like this is pretty much semantically equivalent to the find -exec as recommended by Charles Duffy. Xargs has the advantage that can make use of multicore processors (it can run multiple instances of pdftotext at a time; you can configure how many with the -P switch).

Answer 3

This is a Linux question thus primarily how to use the command line to search all pdf files for "hot" in Linux.

For windows users you would need a slightly different syntax using for or forfiles to recurse the directories with for example something like :-

forfiles /P "C:\Users\WDAGUtilityAccount\Desktop\SandBox" /S /M *.pdf /C "cmd /c pdftotext @file  - |find /I \" hot \"

However that would generate reams of mixed output including many pdf errors mixed in with valid outputs such as

Syntax Warning: Invalid Font Weight
Syntax Warning: Invalid Font Weight
identifies hot (frequently executed) bytecode sequences, records
their time in hot loops. Even in dynamically typed languages, we
....
.....

However there is a much simpler method and that is (1st ensuring you installed a pdf iFilter) simply add "hot" to a file search, so here we find there are 26 results in all the sandbox folders.

Answer 4

An answer for concatenate all codes found, by regex, inside each pdf and rename each pdf filename with those codes found.

Examples of codes corresponding to the regexp in the shell to be searched in the PDF files

• File1.pdf:X123456
• File1.pdf:A1234567
• File2.pdf:X003456
• File2.pdf:A0034567

So File1 and File2 files will be renamed:

• X123456_A1234567_File1.pdf
• X003456_A0034567_File2.pdf

The filename batch find_codes_in_pdf_and_rename.sh

To be executed chmod +x find_codes_in_pdf_and_rename.sh

Execution with output to the screen and log (The sed is to be readable under Windows with CR+LF). ./find_codes_in_pdf_and_rename.sh 2>&1 | tee | sed -u 's/$/\r/' 2>&1 | tee find_codes_in_pdf_and_rename.sh_$(date "+%Y_%m_%d_%Hh_%M_%S").log

#!/bin/bash -e


PrevFile=""
PrevCodes=""
mycmd1=""
mycmd2=""

DIRPrevFile="."
DIRFile="."

BASEFile=""

# look for files where the extension is pdf
# -print0 to have character zero to manage file name with space

find /my_path/ -iname "*.pdf" -print0 |  
# head for debug only two files, -z for print0 
# # head -z -n 2 |  
# sort, -z for print0 
sort -z| 
# exclude filename with code yet in filename, -z for print0
grep -z -v   -E   ".*[\s\.\/][A-Z][0-9]{6,7}.*" | 
# list filename:code
xargs -0 pdfgrep  -i  --only-matching  --with-filename -e "([A-Z]{1}[0-9]{6,7})"  2>&1 |
# exclude  "pdfgrep: Could not open"
tee| grep -v "pdfgrep: Could not open" |
# exclude empty lines
grep -v -e '^$' |
# find path of filename in regexp code group 1 
# and code in regexp code group 3 
# and keep only that in the list with the character ':' at the middle. 
# It's partially redundant if pdfgrep works well with --only-matching
sed --regexp-extended -e   's/(.):(.*)([A-Z][0-9]{6,7})(.*)/\1:\3/gm' |
uniq| {
   while read line
   do
       File=$( echo "$line" |cut -d\: -f1 )
       code=$( echo "$line" |cut -d\: -f2 )

       #echo File $File
       #echo code $code

       if [ "$PrevFile" == "" ]
       then
           PrevFile=$File
       fi

       if [ "$PrevFile" == "$File" ] && [ -n "$PrevCodes" ]
       then
           # concatenate all previous code to current code for the same filename 
           PrevCodes="${PrevCodes} ${code}"
       else
           PrevCodes=$code
       fi
       # uniques codes
       PrevCodes=$(echo  $PrevCodes | tr ' ' '\n' | sort | uniq | tr '\n' ' ')
  
       # echo $PrevCodes
       DIRPrevFile=`dirname "${PrevFile}"`
       DIRFile=`dirname "${File}"`
       #echo $DIRPrevFile   
   
       if [ "${DIRPrevFile}/${PrevFile}" != "${DIRFile}/${File}" ]
       then
           # computed at the previous loop of filename
           # echo "MVFake ${mycmd1}" "${mycmd2}"
           set -x
           mv "${mycmd1}" "${mycmd2}"
           set +x
        fi
   
        # to remove old PDF extension
        BASEFile=$(echo `basename  "${File}" .pdf` )
   
        # mycmd1: old filename
        mycmd1="$File"
   
        # concatenate all codes with the old filename, and replace . and space with _
        target=$(echo "${PrevCodes} ${BASEFile}" | sed "s/[ .]/_/g" ) 
        mycmd2=$(echo "${DIRPrevFile}/${target}.pdf" )

        PrevFile=$File
    done
    # echo "MVFake ${mycmd1}" "${mycmd2}"
    set -x
    mv "${mycmd1}" "${mycmd2}"
    set +x
}