1

I have a bash script which is called from a nodejs http-server upon a GET request from a web-browser. The output from the bash script is returned to the browser via the http-server.

The purpose of the bash script, is to search a file for the search words supplied by the web-browser and return the search result to the http-server, which in turn send the result to the web-browser.

The bash script receive the following arguments:

CS/NCS --> Case-sensitive search or not.  
OR/AND --> 'AND' or 'OR' search.  
one or more searchwords  

E.g.

'search_index.sh CS AND searchword1 searchword2'  
'search_index.sh NCS OR searchword1 searchword2 searchword3'  
 

Egrep is used to search for words in the file.

In line 26/29 an 'OR' search is performed. Egrep is called with the 'OR' search string. This works perfectly. The search result is shown in the web-browser.

For doing an 'AND' search I'm building an egrep search string from line 33 and forward. In line 50 the egrep search string is executed by the eval command.

If I run the script manually it is working fine. I can see the search result written to the console.

When run by the http-server (web-browser), I do not see anything returned to the web-browser

Is there another way to execute the egrep search string (line 50), that won't breake the return of the search result to the http-server?

1       #!/bin/bash
3       # search_index.sh v0.1 (p) april 2021
4       PARAMS=("$@")
5       T=1
6       declare -a SEARCH
7       for WORD in "${PARAMS[@]}"
8       do
9           if [[ $T -eq 1 ]]; then
10              CS_TYPE=$WORD
11              ((T=T+1))
12              continue
13          fi
14          if [[ $T -eq 2 ]]; then
15              LOG_TYPE=$WORD
16              ((T=T+1))
17              continue
18          fi
19          SEARCH+=("$WORD ")
20      done
21      INDEXFILE="/home/user/search/file.idx"
22      if [[ "$LOG_TYPE" = "OR" ]]; then
23          printf -v SEARCH_STR "%s" "${SEARCH[@]}"
24          SEARCH_STR=$( echo $SEARCH_STR | tr ' ' '|' )
25          if [[ "$CS_TYPE" = "NCS"  ]]; then
26              egrep --ignore-case $SEARCH_STR $INDEXFILE
27              exit
28          else
29              egrep $SEARCH_STR $INDEXFILE
30              exit
31          fi
32      else
33          if [[ "$CS_TYPE" = "NCS"  ]]; then
34              GREP_VAR="egrep --ignore-case "
35          else
36              GREP_VAR="egrep "
37          fi
38          I=0
39          for POS in "${SEARCH[@]}"
40          do
41              if [[ $I -eq 0 ]]; then
42                  GREP_STRING="$GREP_VAR ${SEARCH[$I]} $INDEXFILE "
43                  ((I=I+1))
44                  continue
45              else
46                  GREP_STRING+="| $GREP_VAR ${SEARCH[$I]} "
47                  ((I=I+1))
48              fi
49          done
50      eval $GREP_STRING
51      fi

1       // server.js
2       // HTTP-server for the bash search script ('search_index.sh').
3       // Example call: http://192.168.88.10:8888?SEARCH=NCS AND search_word1 search_word2
4       const { createServer } = require("http");
5       const { parse } = require("url");
6       const spawn = require('child_process').spawn;
7       function onRequest(request, response) {
8           var params = parse(request.url,true).query;
9           const proc = spawn('search_index.sh', [params.SEARCH]);
10          let output = '';                      
11          proc.stdout.on('data', (chunk) => {
12              output += chunk.toString();
13          });                                                                                 
14          proc.on('exit', () => {
15              response.writeHeader(200, {'Content-Type': 'text/plain'});                                                                              
16              response.end(output);
17              console.log(output);
18          });
19      };
20      createServer(onRequest).listen(8888);

UPDATE: The server code below using execSync is working. Thanks to Taylor G. for the hint.

1 const { createServer } = require("http"); // http.createServe
2 const { parse } = require("url"); // url.parse
3 const { execSync } = require('child_process');

4 function onRequest(request, response) {

5    let result = execSync('search_index.sh').toString();
6    response.writeHeader(200, {'Content-Type': 'text/plain'});
7    response.end(result);

8 };
9 createServer(onRequest).listen(8118);
3rlands
  • 13
  • 1
  • 4
  • Don't know if it's the problem, but you should always quote the variable: `eval "$GREP_STRING"` – Barmar Apr 15 '21 at 22:55
  • Change `eval` to `echo` to see the command that it's trying to execute. – Barmar Apr 15 '21 at 22:55
  • Building commands in variables (with or without `eval`) is easy to get wrong. I'd just build the search string in a variable, not the entire command. – Gordon Davisson Apr 15 '21 at 23:07
  • @Barmar If I run the script manually it is working fine. I can see the search result written to the console. – 3rlands Apr 15 '21 at 23:54
  • @GordonDavisson I have a problem doing that with the piping to the next egrep. Lines 41-48. Any hints? – 3rlands Apr 15 '21 at 23:56
  • @Barmar I have tried eval "$GREP_STRING". It make no difference. – 3rlands Apr 15 '21 at 23:57
  • @3rlands I didn't think it was the problem, just a general recommendation. – Barmar Apr 16 '21 at 00:26
  • Building commands for use with `eval` is dangerous, especially when the input comes from the internet. In this case, if the user sends something like `dummyword|rm -rf /` as one of the words it may cause your script to delete everything it has access to on the server. – psmears Apr 17 '21 at 22:50
  • @psmears I'm aware of that. The website is not public and in a controlled environment. – 3rlands Apr 18 '21 at 08:59

1 Answers1

0

You can generate AND egrep result on-the-fly, rather than constructing command and using eval at the end.

Try replacing line 33-50 in your script with the following. See if it would work with your server.

I=0
RES=$(cat "$INDEXFILE")
for POS in "${SEARCH[@]}"
do
    if [[ "$CS_TYPE" = "NCS"  ]]; then
        RES=$(echo "$RES" | egrep --ignore-case ${SEARCH[$I]})
    else
        RES=$(echo "$RES" | egrep ${SEARCH[$I]})
    fi
    ((I=I+1))
done
echo "$RES"
Taylor G.
  • 661
  • 3
  • 10
  • Hi Taylor, Now I DO get results back to the browser. But (1) It takes a long time cycling through the whole file (137644 lines) like that - 8.18 s. (2) When run in the terminal, I only get the search results displayed - fine. But when called from the browser via the http-server, I get ALL 137644 lines displayed in the browser (21.07 MB). Regards, – 3rlands Apr 17 '21 at 09:27
  • I'm not familiar with nodejs but I read your server.js file, you seemed to execute bash script with spawn. The documentation of spawn says "spawns the child process **asynchronously**, without blocking the Node.js event loop". My guess is the bash script was only partially executed and the result was then sent back to browser, which is why you got nothing from eval (as whole long pipe takes time to execute), and got whole file from my code (as cat to get local file is fast). – Taylor G. Apr 17 '21 at 17:13
  • I found this [thread](https://stackoverflow.com/questions/22337446/how-to-wait-for-a-child-process-to-finish-in-node-js), which suggests using **exec-sync** will let script finish before continue proceeding. Give that a shot and see if it would make difference. – Taylor G. Apr 17 '21 at 17:16
  • I will take a look at the **exec-sync** method. – 3rlands Apr 17 '21 at 18:56
  • Using execSync in the nodeJs http server solved the problem - even with eval! Thank You @Taylor G. for the hint. – 3rlands Apr 17 '21 at 22:32