Replace space with newline except space wrapped by special characters

Question

In shell, I have a variable storing private key, but every newline character in original private key is replaced with a space now, so the variable looks like:

PK="-----BEGIN RSA PRIVATE KEY----- aaa bbb ccc ... zzz -----END RSA PRIVATE KEY-----"

Now I need to redirect the string to a file, and I need to replace space with newline character. But if I simply run echo $PK | tr " " "\n" > key, spaces in BEGIN RSA PRIVATE KEY and END RSA PRIVATE KEY will also be replaced with newline, so I get a strange file looks like:

-----BEGIN
RSA
PRIVATE
KEY-----
aaa
bbb
ccc
...
zzz
-----END
RSA
PRIVATE
KEY-----

So my question is how to replace space with newline in the key body and keep space inside ----- unchanged. What I expect to get is:

-----BEGIN RSA PRIVATE KEY-----
aaa
bbb
ccc
...
zzz
-----END RSA PRIVATE KEY-----

There are no spaces in a RSA key's body...?!? All you need is a newline after `-----BEGIN RSA PRIVATE KEY-----` and before `-----END RSA PRIVATE KEY-----`. — DevSolar, Apr 25 '18 at 07:19
@DevSolar Thanks. And how can I add a newline after `-----BEGIN RSA PRIVATE KEY-----` and before `-----END RSA PRIVATE KEY-----`...? — norshtein, Apr 25 '18 at 07:29
maybe you get space because you forgot to double quote variables as command argument like in `echo $PK | tr " " "\n" > key`; must be `echo "$PK"`, other issues you may encounter is globbing. `PK="a * b";` compare `echo $PK` and `echo "$PK"` — Nahuel Fouilleul, Apr 25 '18 at 07:30
@NahuelFouilleul Yes, you are right, I don't know the difference between `echo "$PK"` and `echo $PK` before, sorry for that... — norshtein, Apr 25 '18 at 07:41

score 7 · Accepted Answer · answered Apr 25 '18 at 07:32

7

While this is not the exact answer you're looking for, I think this is what you're really trying to do:

To capture the private key in a variable, do so:

PK="$(cat ~/.id_rsa)" # or whatever command you're getting the key from

Then, to dump it into a file:

echo "$PK" > key # notice the additional quotes

The additional quotes when echoing back the variable preserve the newlines captured in the input. No sed/awk/tr magic required.

reference: Capturing multiple line output into a Bash variable

answered Apr 25 '18 at 07:32

ffledgling

11,502
8
47
69

1

Thanks! But the variable is produced by another application, I can only access this variable, and I can do nothing with the original private key... – norshtein Apr 25 '18 at 07:35
If the other application is doing it's job correctly, all you have to do is `echo "$PK"` instead of `echo $PK` and it should work. You don't need to get the original key itself, the `cat` is just as an example. – ffledgling Apr 25 '18 at 07:36
@norshteintooma: And if the other application is *not* doing its job correctly, send a bug report. ;-) Anyway, the *variable* is created by another application? How does *that* work? Are you sure the PK is not actually the *output* of another application (like ffledgling showcased with `cat` here)? – DevSolar Apr 25 '18 at 07:37
@DevSolar In fact, I'm using concourse, and I provide the private key string via a config file, then concourse will export string as an environment variable inside the container, and I do my jobs inside the container. – norshtein Apr 25 '18 at 07:46
@norshteintooma: Ah, *environment* variables. I was stuck for a moment trying to imagine how another application would magically "inject" a variable into your script. ;-) Nevermind. Good you found your (simple) solution. – DevSolar Apr 25 '18 at 07:50

anubhava · Answer 2 · 2018-04-25T07:35:11.743

6

Using gnu-awk, you can do this:

awk -v RS='-+(BEGIN|END) RSA PRIVATE KEY-+' '{
gsub(/ +/, "\n"); ORS=RT} 1' <<< "$PK"

-----BEGIN RSA PRIVATE KEY-----
aaa
bbb
ccc
...
zzz
-----END RSA PRIVATE KEY-----

Using -v RS='-+(BEGIN|END) RSA PRIVATE KEY-+' we set input record separator as BEGIN or END markers.
Using gsub we replace all spaces with newlines in the main body
Using ORS=RT we put header and footer back in output.

Alternative gnu sed solution:

sed -E 's/(-+(BEGIN|END) RSA PRIVATE KEY-+) *| +/\1\n/g' <<< "$PK"

-----BEGIN RSA PRIVATE KEY-----
aaa
bbb
ccc
...
zzz
-----END RSA PRIVATE KEY-----

edited Apr 25 '18 at 07:35

answered Apr 25 '18 at 07:28

anubhava

761,203
64
569
643

2

Could you explain the sed regex and substitution? The regex `(-+(BEGIN|END) RSA PRIVATE KEY-+) *| +`, will match (and group) the strings `-----BEGIN RSA PRIVATE KEY----` or `-----END RSA PRIVATE KEY----` (with one or more leading or trailing hyphens), and any number of spaces after (` *`) or it will match one or more spaces (` +`). In the replacement part (`\1\n`), the first group (`\1`) will kept as such (retaining the BEGIN/END tags), and the `\n` is the actual replacement for all the spaces matched by the ` +` in the match regex I guess? – akarollil Feb 11 '20 at 19:36
Indeed, you have understood and explained it really well. – anubhava Feb 11 '20 at 19:44

Replace space with newline except space wrapped by special characters

2 Answers2