0

I am trying to loop through the file.txt which has domains like %.google.com and use curl with URL crt.sh to extract HTTPS websites subdomains from certificate.

Unfortunately it throws an error

parse error: Invalid numeric literal at line 1, column 20

script.sh

for i in $(cat file.txt); do echo""; echo "crtsh $i"; echo ""; curl -s https://crt.sh/?q\=$1\&output\=json | jq -r '.[].name_value' | sed 's/\*\.//g' | sort -u; echo ""; done
U880D
  • 8,601
  • 6
  • 24
  • 40
Dos_Kid
  • 37
  • 5
  • Are you trying to [read a file line by line](https://stackoverflow.com/questions/10929453/read-a-file-line-by-line-assigning-the-value-to-a-variable)? – Wiktor Stribiżew Feb 28 '20 at 08:33
  • 2
    This is a `jq` error, meaning the "JSON" is not a correct JSON. For example: `echo '[{"name_value": 3x2}]' | jq -r '.[].name_value'`. Inspect what exactly you are getting from your `curl`. – Amadan Feb 28 '20 at 08:35
  • @WiktorStribiżew yes, i have a list of domains which i need to feed to this ```curl -s https://crt.sh/?q\=$1\&output\=json | jq -r '.[].name_value' | sed 's/\*\.//g' | sort -u``` so i'm trying to add a loop – Dos_Kid Feb 28 '20 at 08:37

1 Answers1

0

Since you are trying to read the input file line by line, you need to change your approach slightly.

The following example is working in my environment and has some enhancements for debugging and to get a better understanding. 

#!/bin/bash

while IFS= read -r line; do
  echo"";

  # Added for easier debugging and understanding
  URL="${line}";
  echo ${URL};

  echo "";

  # Added switches for easier debugging
  #   -i, include the HTTP-header in the output
  #   -w, display http_code on stdout after a completed and successful operation
  curl --silent -x "localhost:3128" --include 'https://crt.sh/?q=${URL}&output=json' --write-out "\n\n%{http_code}\n"; # | jq -r '.[].name_value' | sed 's/\*\.//g' | sort -u;

  curl -s 'https://crt.sh/?q=${URL}&output=json' | jq -r '.[].name_value' | sed 's/\*\.//g' | sort -u;

  echo "";
done < URLs.txt

It would also be possible to change it to an one-liner.

U880D
  • 8,601
  • 6
  • 24
  • 40