BASH: add prefix to each line on curl output

Question

I'm trying to figure how to add a prefix at the beggining of each line where the lines are the output of curl:

I have the following command and output:

curl -s https://www.url.com/ | grep -Eo '/news/[^ >]+'

/news/category/title-new-bla-bla
/news/category2/title-new-bla-bla2
/news/category3/title-new-bla-bla3
/news/category4/title-new-bla-bla4

I would to add to my command, maybe using sed or another tool, the prefix https://www.url.com/ to each line as follows:

https://www.url.com/news/category/title-new-bla-bla
https://www.url.com/news/category2/title-new-bla-bla2
https://www.url.com/news/category3/title-new-bla-bla3
https://www.url.com/news/category4/title-new-bla-bla4

Any help would me amazing please!

[related](https://stackoverflow.com/questions/2099471/add-a-prefix-string-to-beginning-of-each-line) — Nahuel Fouilleul, Jul 13 '23 at 11:54
Does this answer your question? [How can I prepend a string to the beginning of each line in a file?](https://stackoverflow.com/questions/13586349/how-can-i-prepend-a-string-to-the-beginning-of-each-line-in-a-file) — user1934428, Jul 13 '23 at 12:17
Please [edit] your question to show the output of `curl -s https://www.url.com/` that you want to produce your shown final expected output from so we can use that to test a solution against. — Ed Morton, Jul 15 '23 at 21:25

Antonio Petricca · Accepted Answer · 2023-07-13T11:57:12.683

2

Here is a simple solution:

  curl -s https://www.url.com/ \
| grep -Eo '/news/[^ >]+' \
| xargs -I "{}" echo "https://www.url.com{}"'

edited Jul 13 '23 at 11:57

answered Jul 13 '23 at 11:51

Antonio Petricca

8,891
5
36
74

score 2 · Answer 2 · answered Jul 13 '23 at 12:02

Using sed

$ sed -En '\~^/news/~s~^~https://www.url.com~p' <(curl -s https://www.url.com/)
https://www.url.com/news/category/title-new-bla-bla
https://www.url.com/news/category2/title-new-bla-bla2
https://www.url.com/news/category3/title-new-bla-bla3
https://www.url.com/news/category4/title-new-bla-bla4

anubhava · Answer 3 · 2023-07-13T11:55:03.490

1

You can use sed instead of grep:

curl -s https://www.url.com/ | 
sed -nE 's~.*(/news/[^ >]+).*~https://www.url.com\1~p'

This sed will search for the same pattern /news/[^ >]+ in a line and in replacement it prefixes each line with https://www.url.com.

edited Jul 13 '23 at 11:55

answered Jul 13 '23 at 11:49

anubhava

761,203
64
569
643

1

This won't skip the non-matching lines. – choroba Jul 13 '23 at 11:50
Valid point @choroba, fixed now – anubhava Jul 13 '23 at 11:51
Thanks a lot @choroba! amazing sed pattern by the way!. my goal is to create patterns like that – cristobal veas Jul 13 '23 at 12:06

score 0 · Answer 4 · answered Jul 14 '23 at 13:31

0

we have:

curl -s https://www.url.com | grep -Eo '/news/[^ >]+'

then, we can add a sed:

curl -s https://www.url.com | grep -Eo '/news[^ >]+' \
| sed 's|^|https://www.url.com/|'

answered Jul 14 '23 at 13:31

Marcelo Guedes

1,419
11
10

score 0 · Answer 5 · answered Jul 15 '23 at 21:21

0

Since you are using GNU grep I assume you also have GNU awk in which case you can use multi-char RS and RT:

curl -s https://www.url.com/ |
awk -v RS='/news/[^ >]+' 'RT{print "https://www.url.com/" RT}'

answered Jul 15 '23 at 21:21

Ed Morton

188,023
17
78
185

BASH: add prefix to each line on curl output

5 Answers5