Unable to make the mentioned regular expression to work in sed command

Question

I am trying to make the following regular expressions to work in sed command in bash.

^[^<]?(https?:\/\/(www\.)?[-a-zA-Z0-9@:%._\+~#=]{1,256}\.[a-zA-Z0-9()]{1,6}\b([-a-zA-Z0-9()@:%_\+.~#?&\/\/=]*))[^>]?$

I know the regular expression is correct and it is working as I expected. So; there is no help needed with that. I tested it on online regular expressions tester and it is working as per my expectations.

Please find the demo of the above regex in here.

My requirement: I want to enclose every url inside <>. If the url is already enclosed; then append it to the result as can be seen in the above regex link.

Sample Input:(in file named website.txt)

// List of all legal urls
https://www.google.com/
https://www.fakesite.co.in
https://www.fakesite.co.uk
<https://www.fakesite.co.uk>
<https://www.google.com/>

Expected Output:(in the file named output.txt)

<https://www.google.com/> // Please notice every url is enclosed in the <>.
<https://www.fakesite.co.in>
<https://www.fakesite.co.uk>
<https://www.fakesite.co.uk> // Please notice if the url is already enclosed in <> then it is appended as it is.
<https://www.google.com/>

What I tried in sed:

1. Since I'm not well-versed in bash commands; so previously I was not able to capture the group properly in sed but after reading this answer; I figured out that we need to escape the parenthesis to be able to capture it.

2. Somewhere; I read that look-arounds are not supported in sed(GNU based) so I removed lookarounds too; but that also didn't worked. If it doesn't support look-arounds then I used this regex and it served my purpose.

3. Then; this is my latest try with sed command:

   sed 's@^[^<]?(https?://(?:www\.)?[-a-zA-Z0-9@:%._\+~#=]{1,256}\.[a-zA-Z0-9()]{1,6}\b(?:[-a-zA-Z0-9()@:%_\+.~#?&/=]*))[^>]?$@<\1>@gm;t;d' websites.txt > output.txt
   

My exact problem:

How can I make the above command to work properly. If you'll run the command sample I attached above in point-3; you'd see it is not replacing the contents properly. It is just dumping the contents of websites.txt to output.txt. But in regex demo; attached above it is working properly i.e. enclosing all the unenclosed websites inside <>. Any suggestions would be helpful. I preferably want it in sed but if it is possible can I convert the above command in awk also? If you can please help me with that too; I'll be highly obliged. Thanks

Answer 1

After working for long, I made my sed command to work. Below is the command which worked.

sed -E 's@^[^<]?(https?://(www\.)?[-a-zA-Z0-9@:%._\+~#=]{1,256}\.[a-zA-Z0-9()]{1,6}\b([-a-zA-Z0-9()@:%_\+.~#?&=]*))[^>]?$@<\1>@gm;t' websites.txt > output.txt

You can find the sample implementation of the command in here.

Since, the regex has already fulfilled the requirement of the person for whom I'm writing this requirement for; I needed to get help only regarding the command syntax (although any improvements are heartily welcomed); I want the command to work with the same regular expression pattern.

Things which I was unaware previously and learnt now:

1. I didn't knew anything about -E flag. Now I know; that -E uses POSIX "extended" syntax ("ERE"). Thanks to @GordonDavisson and @Sundeep. Further reading.

2. I didn't know with clarity that sed doesn't supports look-around. But now I know sed doesn't support look-around. Thanks to @dmitri-chubarov. Further reading

3. I didn't knew sed doesn't support non-capturing groups too. Thanks to @Sundeep for solving this part. Further Reading

4. I didn't knew about GNU sed as a specific command line tool. Thanks to @oguzismail for this. Further reading.

Answer 2

With respect to the command in your answer:

sed -E 's@^[^<]?(https?://(www\.)?[-a-zA-Z0-9@:%._\+~#=]{1,256}\.[a-zA-Z0-9()]{1,6}\b([-a-zA-Z0-9()@:%_\+.~#?&=]*))[^>]?$@<\1>@gm;t'

Here's a few notes:

Your posted sample input has 1 URL per line so AFAIK the gm;t at the end of your sed command is doing nothing useful so either your input is inadequate or your script is wrong.

The hard-coded ranges a-z, A-Z, and 0-9 include different characters in different locales. If you meant to include all (and only) lower case letters, upper case letters, and digits then you should replace a-zA-Z0-9 with the POSIX character class [:alnum:]. So either change to use a locale-independent character class or specify the locale you need on your command line depending in your requirements for which characters to match in your regexp.

Like most characters, the character + is literal inside a bracket expression so it shouldn't be escaped - change \+ to just +.

The bracket expression [^<]? means "1 or 0 occurrences of any character that is not a <" and similarly for [^>]? so if your "url" contained random characters at the start/end it'd be accepted, e.g.:

echo 'xhttp://foo.bar%' | sed -E 's@^[^<]?(https?://(www\.)?[-a-zA-Z0-9@:%._\+~#=]{1,256}\.[a-zA-Z0-9()]{1,6}\b([-a-zA-Z0-9()@:%_\+.~#?&=]*))[^>]?$@<\1>@gm;t'
<http://foo.bar%>

I think you meant to use <? and >? instead of [^<]? and [^>]?.

Your regexp would allow a "url" that has no letters:

echo 'http://=.9' | gsed -E 's@^[^<]?(https?://(www\.)?[-a-zA-Z0-9@:%._\+~#=]{1,256}\.[a-zA-Z0-9()]{1,6}\b([-a-zA-Z0-9()@:%_\+.~#?&=]*))[^>]?$@<\1>@gm;t'
<http://=.9>

If you edit your question to provide more truly representative sample input and expected output (including cases you do not want to match) then we can help you BUT based on a quick google of what a valid URL is it looks like there are several valid URLs that'd be disallowed by your regexp and several invalid ones that'd be allowed so you might want to ask about that in a question tagged with url or similar (with the tags you currently have we can help you implement your regexp but there may be better people to help with defining your regexp).

Answer 3

If the input file is just a comment followed by a list of URLs, try:

sed '1d;s/^[^<]/<&/;s/[^>]$/&>/' websites.txt

Output:

<https://www.google.com/>
<https://www.fakesite.co.in>
<https://www.fakesite.co.uk>
<https://www.fakesite.co.uk>
<https://www.google.com/>