How to check the validity of a list of URLs in a file?

Question

Trying curl a list of urls to confirm they are active or not. I'd like to separate active from non-active but I'm stuck. I'm super new to bash scripting, as I'm sure you can tell. cat function into the url doesn't appear to work correctly. Separating the contents also doesn't work correctly. A large potion of the urls will go into nv.txt but I know they are active urls.

Thank you for the help in advance

#!/bin/bash
url=$(cat non-ssl-list2.txt)
if curl --output /dev/null --silent --head --fail "$url"
then
    echo $url Exists > v.txt
else
    echo "$url Does Not Exist" > nv.txt
fi

https://hastebin.com/tuqekocate.bash

(code kept getting chopped off so I put it into hastebin).

https://hastebin.com/nogasiyeyo.css (sample of list)

What's the question? What's the problem? Are you getting an error? If so, what's the error? If this is your complete script, then you're missing a `fi` to end your `if-else`. — lurker, Feb 19 '18 at 19:56
Sorry, struggling with adding code into the post. I added a hastebin url to display the code — Jonathan Mann, Feb 19 '18 at 20:03
@JonathanMann did the answer work ? From your latest pastebin, I see that, except `pauliandco.com`, everything else `exists`. Note, you need to remove the whitespace from the beginning of the line. Use `sed -i 's/^ //g' inputfile` to do that. — iamauser, Feb 19 '18 at 21:03
@iamauser running it now, it's moving slow. Giant list and a slow server. I'll keep you posted! — Jonathan Mann, Feb 19 '18 at 21:52

codeforester · Answer 1 · 2018-02-19T20:56:08.250

1

The issue in your code is here:

url=$(cat non-ssl-list2.txt)

this slurps all lines of your input file into a single string. That's not what you want, right?

You can loop through the urls in your file line by line and invoke curl to test if the URL is valid:

while read -r url; do                    # loop through the array
  if curl -s --head "$url" > /dev/null; then  # check URL validity
    printf '%s\n' "$url exists"
  else
    printf '%s\n' "$url does not exist"
  fi
done < non-ssl-list2.txt

See these related posts:

edited Feb 19 '18 at 20:56

answered Feb 19 '18 at 20:09

codeforester

39,467
16
112
140

I tried to use this and added "> list.txt" to the right of non-ssl-list2.txt. It's displays all the urls from non-ssl-list2.txt as not existing (almost all are active). – Jonathan Mann Feb 19 '18 at 20:38
It worked for me. Add a sample of your input file to the question so that we can figure out the issue. – codeforester Feb 19 '18 at 20:38
1

Add it to the question so that other people can see it easily. – codeforester Feb 19 '18 at 20:44
added hastebin url with a sample – Jonathan Mann Feb 19 '18 at 20:50
Leading spaces in your file was the issue. I corrected the code. Please see the updated answer. – codeforester Feb 19 '18 at 20:56

iamauser · Accepted Answer · 2018-02-19T20:18:35.430

One can use && and || operators to print to an output file based on the exit status of the curl command.

$ curl -k -s -o /dev/null "${url}" && echo "${url} : Exists" > 1.txt || echo "${url} : Does not exist" > 2.txt

For a list,

#!/bin/bash
> 1.txt  # Create empty text file
> 2.txt  # 
while IFS= read -r url
do
  curl -k -s -o /dev/null "${url}" && echo "${url} : Exists" >> 1.txt || echo "${url}: Does not exist" >> 2.txt
done < /path/to/list.txt

How to check the validity of a list of URLs in a file?

2 Answers2