Script works with file input but not stdin

Question

this really has me stumped. Here is what I am trying to do:

I try to pipe an article from newsboat to a script. This script should then extract the Title and Url from the article.

Here is an example article:

Feed: NYT > Home Page
Title: Hit Pause on Brett Kavanaugh
Author: THE EDITORIAL BOARD
Link: https://www.nytimes.com/2018/09/26/opinion/kavanaugh-supreme-court-hearing-delay.html?partner=rss&emc=rss
Date: Thu, 27 Sep 2018 01:58:11 +0200

The integrity of the Supreme Court is at stake.

The article gets piped with a macro from newsboat:

macro R pipe-to "cat | ~/.scripts/newsboat_extract"  

Here is the working script:

#!/bin/bash

cat > ~/newsboat         #I do not really need this file, so if I can cut out saving to a file, I would prefer to

title="$(awk -F: '/^Title:/{for(i=2;i<=NF;++i)print $i}' ~/newsboat)"
url="$(awk -F: '/^Link:/{print $2 ":" $3}' ~/newsboat)"
printf '%s\n' "$title" "$url" >> newsboat_result

This delivers the expected output:

Hit Pause on Brett Kavanaugh
https://www.nytimes.com/2018/09/26/opinion/kavanaugh-supreme-court-hearing-delay.html?partner=rss&emc=rss

I would like to avoid saving to a file. However, saving to a variable does - for whatever reason - not work: And this is the script that is not working!

#!/bin/bash

article=$(cat)

title="$(awk -F: '/^Title:/{for(i=2;i<=NF;++i)print $i}' "$article")"
url="$(awk -F: '/^Link:/{print $2 ":" $3}' "$article")"
printf '%s\n' "$title" "$url" >> newsboat_result

the output turns to this:

#empty line
#empty line

I have completely no idea why the script would behave like this. It must have something to do how the variable is stored, right?

Any ideas? - I am pretty new at bash scripting and awk, so thankful also for any comments on how to solve this problem more efficiently.

"""""""""""" " SOLUTION " """"""""""""

This did it, thank you!

#!/bin/bash

article=$(cat "${1:--}")

title="$(awk -F: '/^Title:/{for(i=2;i<=NF;++i)print $i}' <<< "$article")"
url="$(awk -F: '/^Link:/{print $2 ":" $3}' <<< "$article")"
printf '%s\n' "$title" "$url" >> newsboat_result

Answer 1

In your script, you are assuming that $ARTICLE is a plain file and you are making several operations on it. First you read it with cat and store the content in ~/newsboat, then you read it again with awk to extract the title, then you read it a third time to extract the URL.

This can't work with standard input; it can only be read once.

A quick fix is to work on the copy of it you made in the first operation:

#!/bin/bash

article=$1
feed_copy=~/newsboat
cat "${article:--}" > "$feed_copy"     # Use stdin if parameter is not provided

title="$(awk -F: '/^Title:/ { for(i=2; i<=NF; ++i) print $i }' "$feed_copy")"
url="$(awk -F: '/^Link:/ { print $2 ":" $3 }' "$feed_copy")"

printf '%s\n' "$title" "$url" >> "$feed_copy"

Not tested, obviously, but that should work.

Notes:

• reserve uppercase variable names for environment variables (this is a mere convention)
• you should almost always quote your variables (cat "$article", not cat $article) unless you know what you are doing
• avoid echo, use printf

There are other enhancements that could be made to this script but sorry, I lack the time.

---

[edit] Since you don't actually need the ~/newsboat file, here is a updated version that follows Charles Duffy's suggestion:

#!/bin/bash

feed_copy=$(cat "${1:--}")
title="$(awk -F: '/^Title:/ { for(i=2; i<=NF; ++i) print $i }' <<< "$feed_copy")"
url="$(awk -F: '/^Link:/ {print $2 ":" $3}' <<< "$feed_copy")"
printf '%s\n' "$title" "$url"