Search within a directory structure using find, sed and awk

Question

This series of commands edits idea.js and exports it as idea.csv;

sed -n '/^get\.idea/s/^.*(\(.*\)).*/\1/ p' idea.js | 
awk -F, 'BEGIN {print "idea, description";OFS=","} (NR % 2 ) == 1
         {save=$2} (NR % 2) == 0 {print save, $2}' > idea.csv

And what I need to do is run exactly the same commands but using find to search in multiple files with the same idea.js name which are in multiples directories, exporting each of those files in the same directory that idea.js was found.

I tried in many ways but I get different errors.

Here the result is not what's expected:

find . -type f -name "idea.js" \( \ 
    -exec sed -n '/^get\.idea/s/^.*(\(.*\)).*/\1/ p' idea.js  {} \; -o \
    -exec true \; \) \
    -exec awk -F, 'BEGIN {print "idea, description";OFS=","} (NR % 2 ) == 1 
                   {save=$2} (NR % 2) == 0 {print save, $2}' > idea.csv {} \;

And here:

find . -type f -name "idea.js" \( 
    -exec sed -n '/^get\.idea/s/^.*(\(.*\)).*/\1/ p' idea.js | 
    awk -F, 'BEGIN {print "idea, description";OFS=","} (NR % 2 ) == 1
            {save=$2} (NR % 2) ==0 {print save, $2}' > idea.csv {} \)

I get this error:

    idea, description
awk: cannot open {} (No such file or directory)
find: missing argument to `-exec'
Try 'find --help' for more information.

I would like learn how to do this directly from the command line, and as a script, to be executed this way:

$ sh script.sh

I appreciate the help.

{} are replaced with the match. Why are they placed after the output redirection (>)? They should be in the sed command (instead of the idea.js, that does not contain the path. Also, what does the '(' stand for? See here https://stackoverflow.com/questions/307015/how-do-i-include-a-pipe-in-my-linux-find-exec-command#307154 how to use pipes in -exec — Nico202, May 06 '18 at 22:24
I don't have really the adequate knowledge, I'm just experiment with the commands, can you tell me how can I do it well? — Carlos, May 06 '18 at 22:28
This should be a better strting point: find . -type f -name "idea.js" -exec sh -c "sed -n '/^get\.idea/s/^.*($.*$).*/\1/ p' {} | awk -F, 'BEGIN {print "idea, description";OFS=","} (NR % 2 ) == 1 {save=$2} (NR % 2) ==0 {print save, $2}' > idea.csv" \; — Nico202, May 06 '18 at 22:33
I tried your code and report this error many times maybe the times the command finds the idea.js file: `description;OFS=,} (NR % 2 ) == 1 {save=} (NR % 2) ==0 {print save, }' > idea.csv: 1: description;OFS=,} (NR % 2 ) == 1 {save=} (NR % 2) ==0 {print save, }' > idea.csv: Syntax error: Unterminated quoted string` Maybe some scape character missing? — Carlos, May 06 '18 at 22:42
`find . -type f -name "idea.js" -exec sh -c "sed -n '/^get\.idea/s/^.*($.*$).*/\1/ p' {} | awk -F, 'BEGIN {print \"idea, description\";OFS=\",\"} (NR % 2 ) == 1 {save=\$2} (NR % 2) ==0 {print save, \$2}'" \;` ? — Nico202, May 06 '18 at 23:05
You never need sed when you're using awk. Post some concise, testable sample input and expected output and then I expect someone will be able to help you. — Ed Morton, May 07 '18 at 00:38
The last way is near, I set the output like this: `find . -type f -name "idea.js" -exec sh -c "sed -n '/^get\.idea/s/^.*($.*$).*/\1/ p' {} | awk -F, 'BEGIN {print \"idea, description\";OFS=\",\"} (NR % 2 ) == 1 {save=\$2} (NR % 2) ==0 {print save, \$2}'" > idea.csv \; ` And I get an unique file with all found results in the path where was executed the command. I tryed to put before the double quotes and doesn't work either. I need to generate a file idea.csv for each file idea.js found. — Carlos, May 07 '18 at 00:39
@EdMorton Like that the command throws the expected results and the find command as having been set by Nico202 is near, only missing to set the output in the correct place to generate all idea.csv files for each idea.js file found. Right? — Carlos, May 07 '18 at 00:44
What you're doing right now, though you **may** be able to get the expected output from it, is just a hack. So far you're getting help to write valid syntax for a command but it's the wrong command for whatever it is you're trying do do. See [ask] and in particular see the part about providing a [mcve]. Once you do that then we'll be able to help you get the right answer. — Ed Morton, May 07 '18 at 00:47
@Nico202, the advice you're offering has serious security issues. `{}` should **never** be part of a string passed to `sh -c` for substitution prior to the shell's invocation -- when it is, code embedded inside the paths identified by `find` can be parsed and executed. If someone ran `d=$'tmp/$(rm -rf ~)\'$(rm -rf ~)\''; mkdir -p "$d" && touch "$d/idea.js`, then a future invocation of the OP's code (with `find` covering that directory tree) would be a time bomb. It's only safe to substitute filenames *out-of-band* from the script to execute. — Charles Duffy, May 10 '18 at 17:29
That is, instead of `find . -type f -exec sh -c 'echo {}' ';'` (which is dangerous), one should use `find . -type f -exec sh -c 'echo "$1"' _ {} \;` (which is safe), or `find . -type f -exec sh -c 'for arg; do echo "$arg"; done' _ {} +` (which is both safe and more efficient, since it doesn't start extra shells). [UsingFind](http://mywiki.wooledge.org/UsingFind) covers this is more detail at the end of section 6 ("Complex Actions"). — Charles Duffy, May 10 '18 at 17:31

Kusalananda · Answer 1 · 2018-05-10T17:34:30.103

find . -type f -name idea.js -exec sh -c '
    for idea do
        sed -n "/^get\.idea/s/^.*(\(.*\)).*/\1/ p" "$idea" | 
        awk -F, "BEGIN {print \"idea, description\";OFS=\",\"} (NR % 2 ) == 1
                 {save=\$2} (NR % 2) == 0 {print save, \$2}" >"${idea%.js}.csv"
    done' sh {} +

This takes your shell code snippet more or less unchanged (except that I've arranged so that the single quoting in the code is now double quoting, and the output filename is computed from the input filename by replacing the extension), and executes it through find. find will supply the code with a list of pathnames for idea.js files, and the code will loop over these.

An alternative would be to put your code in a shell script:

#!/bin/sh

for idea do
    sed -n '/^get\.idea/s/^.*(\(.*\)).*/\1/ p' "$idea" | 
    awk -F, 'BEGIN {print "idea, description";OFS=","} (NR % 2 ) == 1
             {save=$2} (NR % 2) == 0 {print save, $2}' > "${idea%.js}.cvs"
done

and then call this script for all the found pathnames:

find . -type f -name idea.js -exec ./script.sh {} +

Good approach in general, but the single quotes inside a single-quoted string (in the first example) won't work so well. Second approach is of course safe. — Charles Duffy, May 10 '18 at 17:34

Search within a directory structure using find, sed and awk

1 Answers1