Echo valid certificate file from single line variable with bash

Question

I have a certificate stored in an environment variable on a single line, like this:

$EXAMPLE=-----BEGIN CERTIFICATE----- line1 line2 line3 etc.. -----END CERTIFICATE-----

How can I go from this environment var to a file, where the lines are all converted to newlines, except for the -----BEGIN CERTIFICATE----- and -----END CERTIFICATE----- parts?

I know I can do this:

$ echo "$EXAMPLE" | tr ' ' '\n' > ca.pem

It will replace all spaces to newlines, but this will result in:

-----BEGIN
CERTIFICATE-----
line1
line2
line3
etc..
-----END
CERTIFICATE-----

Almost there... but the BEGIN and END lines with '-----' should not break. Anyone has a clean solution for this? Doesn't matter if it is sed, awk, grep, whatever :)

Thanks!

Possible duplicate of [How can I have a newline in a string in sh?](https://stackoverflow.com/questions/3005963/how-can-i-have-a-newline-in-a-string-in-sh) — tripleee, Dec 09 '18 at 16:57
The question as articulated doesn't directly ask about that, but it seems like the ultimate solution to this problem. Why would you want to represent the string in the wrong format and then fix it at runtime? — tripleee, Dec 09 '18 at 17:00
@triplee - I agree it's better to represent the string in the right format right away, but to put things in perspective: the certificate is pasted in an input field in one of our continuous integration tools. Because the field is an input field and not a textarea, it will convert everything to 1 line. So it's easier to fix it with a script than to do it manually. — 24creative, Dec 10 '18 at 08:50

Ed Morton · Answer 1 · 2018-12-09T18:57:29.207

3

Don't use all upper case for non-exported variable names to avoid conflicting with exported and/or built in names. With GNU awk for the 3rd arg to match() and \S shorthand for [^[:space:]]:

$ example='-----BEGIN CERTIFICATE----- line1 line2 line3 -----END CERTIFICATE-----'
$ printf '%s\n' "$example" |
awk 'match($0,/^(\S+ \S+)(.*)(\S+ \S+)$/,a){gsub(/ /,ORS,a[2]); print a[1] a[2] a[3]}'
-----BEGIN CERTIFICATE-----
line1
line2
line3
-----END CERTIFICATE-----

The above script will work for any input. You should try to come up with input that scripts might fail on to be able to test any proposed solution against to see if it actually works or not as it's trivial to come up with a script that produces the expected output from one specific sample input and far harder to come up with a solution that works in all cases.

For instance, one important and obvious example to test with is:

$ example='-----BEGIN CERTIFICATE----- -----BEGIN CERTIFICATE----- -----END CERTIFICATE----- -----END CERTIFICATE-----'

i.e. where within the 4-line certificate itself there just happens to coincidentally/unfortunately be these strings:

line 1 = -----BEGIN
line 2 = CERTIFICATE-----
line 3 = -----END
line 4 = CERTIFICATE-----

The output should be:

$ printf '%s\n' "$example" |
awk 'match($0,/^(\S+ \S+)(.*)(\S+ \S+)$/,a){gsub(/ /,ORS,a[2]); print a[1] a[2] a[3]}'
-----BEGIN CERTIFICATE-----
-----BEGIN
CERTIFICATE-----
-----END
CERTIFICATE-----
-----END CERTIFICATE-----

so if any proposed solution can't handle that correctly then don't use that solution.

None of the other currently posted answers handle it correctly:

Ravinder's awk:

$ printf '%s\n' "$example" |
awk '
match($0,/- .* -/){
  val=substr($0,RSTART,RLENGTH)
  gsub(/- | -/,"",val)
  gsub(OFS,ORS,val)
  print substr($0,1,RSTART) ORS val ORS substr($0,RSTART+RLENGTH-1)
}'
-----BEGIN CERTIFICATE-----
-----BEGIN
CERTIFICATE---------END
CERTIFICATE-----
-----END CERTIFICATE-----

saichovsky's seds plus pipe:

$ printf '%s\n' "$example" |
sed 's/- /-\n/g; s/ -/\n-/g' | sed '/CERTIFICATE/! s/ /\n/g'
-----BEGIN CERTIFICATE-----
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
-----END CERTIFICATE-----

anubhava's perl:

$ perl -pe 's/(?:BEGIN|END) CERTIFICATE(*SKIP)(*F)|\h+/\n/g' <<< "$example"
-----BEGIN CERTIFICATE-----
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
-----END CERTIFICATE-----

edited Dec 09 '18 at 18:57

answered Dec 09 '18 at 16:55

Ed Morton

188,023
17
78
185

1

Noticing your input of `example='-----BEGIN CERTIFICATE----- -----BEGIN CERTIFICATE----- -----END CERTIFICATE----- -----END CERTIFICATE-----'` I have fine tuned my regex to address this as well. You may please edit your answer – anubhava Dec 10 '18 at 02:43
Thanks Ed for your extensive answer! It is indeed better to use lowercase for non exported variables, but in this case it is exported by one of our continuous integration tools. We use the certificate for deployments, and it is pasted in a (masked) input field that converts it all to a single line. That's why we needed a script to convert it back into a valid file again, but I was struggling with the 'regex' to keep the BEGIN / END lines intact. – 24creative Dec 10 '18 at 09:06
You are right by the way that there could be more -----BEGIN CERTIFICATE----- or end lines within the variable. Sometimes a single certificate file can contain multiple certificates, so actually we need a solution that will not break those lines anywhere in the variable. On all the other lines, a space should be converted into a newline. – 24creative Dec 10 '18 at 09:08
Obviously that implies very different input and requirements to what's stated in your question. That' is disappointing to hear since I put a fair bit of effort into answering your original question. How do you plan to identify those case where, for example, the string `-----BEGIN` followed by `CERTIFICATE-----` exists in your actual certificate or are you just going to cross your fingers and hope it doesn't happen (until it does, of course)? – Ed Morton Dec 10 '18 at 09:44
Hi Ed, fair point, but it depends how you interpret the question. I tried to keep my example as simple as possible, so maybe I should've explained it a little bit better with an actual certificate example, my apologies! My intended question was: how can I convert the variable into a valid certificate file while keeping ..BEGIN.. and ..END.. lines intact. They don't necessarily have to be the first or last lines as (you actually made me realize) there can be multiple occurrences of those lines in the variable. – 24creative Dec 10 '18 at 11:10
The use case here is an SSL certificate by the way and with the way those are generated it is impossible for the certificate itself to contain "-----BEGIN" or something like that in the actual certificate. I should've put more effort in my question to explain this – 24creative Dec 10 '18 at 11:11
Sorry, but none of these worked for me. I get my certificate text from a Hashi-Corp Vault using a Jenkins Plugin into an env var. What did work for me is described in the answer I added here: https://stackoverflow.com/a/61043494/2279082 – ArielGro Apr 05 '20 at 13:40
@ArielGro None of them worked for you because you're not trying to do what the OP is trying to do and which the posted solutions do. You misread the question. – Ed Morton Apr 05 '20 at 13:45
@EdMorton - I read the question quite well. When I tried to echo a Jenkins Env Var (I got from a Hashi-Corp Vault using a plugin) into a file, it ended up as a **_single line_**, which led me to search for a solution and got me here - where you helped the OP to get a single line into a multi-line. I hope this makes more sense – ArielGro Apr 06 '20 at 06:36
That's not what this question is about, it's about not splitting a string at every space. Unlike the OP in this question your variable didn't contain just a single line, it contained multiple lines. If it had contained a single line then `printf '%s' "$var"` would have printed it as a single line because that's what that `printf` statement does - prints a string as-is. – Ed Morton Apr 06 '20 at 12:51
1

@EdMorton - Agreed, I have removed my answer – ArielGro Apr 06 '20 at 13:50

anubhava · Answer 2 · 2018-12-10T10:04:23.897

2

Using perl you can use a regex with (*SKIP)(*F):

s='-----BEGIN CERTIFICATE----- line1 line2 line3 -----END CERTIFICATE-----'
perl -pe 's/-+(?:BEGIN|END) CERTIFICATE-+(*SKIP)(*F)|\h+/\n/g' <<< "$s"

-----BEGIN CERTIFICATE-----
line1
line2
line3
-----END CERTIFICATE-----

Regex ...(*SKIP)(*F)|\h+ will skip starting -BEGIN CERTIFICATE- or ending -END CERTIFICATE- while matching 1+ horizontal whitespaces to be replaced by newline.

In case perl is not available here is an awk solution (little verbose as I wanted to make it work on non-GNU awk as well):

awk '{
   out=""
   for (i=1; i<=NF; i++)
      out = sprintf("%s%s%s", out, $i, 
      (i<NF && $i~/^-+(BEGIN|END)$/ && $(i+1)~/^CERTIFICATE-+$/ ? " " : "\n"))
   printf "%s", out
}' <<< "$s"

edited Dec 10 '18 at 10:04

answered Dec 09 '18 at 16:50

anubhava

761,203
64
569
643

1

Thanks @anubhava for your reply. We can use perl, so your perl regex is a pretty clean solution. Your answer before the edit was actually better :)! When there are more -----BEGIN CERTIFICATE----- or -----END CERTIFICATE----- lines in the variable, they should not break anywhere in the output to keep the certificate valid. – 24creative Dec 10 '18 at 09:17
ok fair point. I have reverted my changes to allow space ***anywhere** between `BEGIN CERTIFICATE` or `END CERTIFICATE`. – anubhava Dec 10 '18 at 10:05

RavinderSingh13 · Answer 3 · 2018-12-09T17:21:11.897

Could you please try following once(considering that your Input_file is same as shown samples).

EXAMPLE="-----BEGIN CERTIFICATE----- line1 line2 line3 -----END CERTIFICATE-----"
echo "$EXAMPLE" | awk '
match($0,/- .* -/){
  val=substr($0,RSTART,RLENGTH)
  gsub(/- | -/,"",val)
  gsub(OFS,ORS,val)
  print substr($0,1,RSTART) ORS val ORS substr($0,RSTART+RLENGTH-1)
}'

Explanation: Adding explanation of above code now.

echo "$EXAMPLE" | awk '          ##Printing value of example variable and passing its output to awk command then.
match($0,/- .* -/){              ##using match to match REGEX to match it from - .* to - if match found then do following.
  val=substr($0,RSTART,RLENGTH)  ##creating val here whose value is substring is starting from RSTART to RLENGTH.
  gsub(/- | -/,"",val)           ##Using gsub to substitute to substitute - space OR space - with NULL in variable val.
  gsub(OFS,ORS,val)              ##Substituting OFS with ORS in variable val here.
  print substr($0,1,RSTART) ORS val ORS substr($0,RSTART+RLENGTH-1)  ##Printing substring from 1 to RSTART then val and substring RSTART+RLENGTH-1 with ORS values.
}'

170730350 · Answer 4 · 2018-12-09T17:35:52.870

-2

echo $EXAMPLE | sed 's/- /-\n/g; s/ -/\n-/g' | sed '/CERTIFICATE/! s/ /\n/g'

should give you what you want.

edited Dec 09 '18 at 17:35

answered Dec 09 '18 at 17:02

170730350

590
1
8
22

Echo valid certificate file from single line variable with bash

4 Answers4