How can I keep the leading white spaces while removing last field in awk?

Question

I want to create a template of a config file containing credentials dynamically. The copy should keep the structure but with the values replaced by empty strings (i.e. ""). I use awk for this purpose.

awk -v dq="\"" '{ if($NF ~ /^[&|\*|\n]/ || $1 == $NF) print $0; else {$NF=""; print $0 dq dq;} }' .config.yaml >> .config_temp.yaml

The problem is in the else statement. When I set $NF="" the leading white space are not printed anymore. If I do not set the last field to an empty string, I do not observe this behavior but obviously do not receive the intended output (see below).

How can I set the last field as an empty string while keeping the leading white spaces?

I don't want to manually add a number of white spaces since the intent varies.

config.yaml (Input)


DEVELOPMENT: &development
  <<: *common

  check_access_token: False

  database:
    mongodb:
      database: test
      hostname: localhost
      port: 27017
      username: ""
      password: ""

      collection:
        col_1: test_1
        col_2: test_2
        col_3: test_3
        col_4: test_4

conf_temp.yaml (actual output)


DEVELOPMENT: &development
  <<: *common

check_access_token: ""

  database:
    mongodb:
database: ""
hostname: ""
port: ""
username: ""
password: ""

      collection:
property: ""
ctrl_voc: ""
form: ""
user: ""

Expected output


DEVELOPMENT: &development
  <<: *common

check_access_token: ""

  database:
    mongodb:
      database: ""
      hostname: ""
      port: ""
      username: ""
      password: ""

      collection:
        property: ""
        ctrl_voc: ""
        form: ""
        user: ""

Edit (after Sundeep's replay)

Thank you for your answer. It almost works as I expect. However, I do not receive the same output as you. If I call

 awk -F'[ ]' -v dq="\"" 'NF>1 && $NF !~ /^[*&]|:$/{$NF = dq dq} 1' .conf.yaml

I receive the following output:

DEVELOPMENT: &development
  <<: *common

  check_access_token: ""

  ""                             <--
    ""                           <--
      database: ""
      hostname: ""
      port: ""
      username: ""
      password: ""

      ""                         <--
        property: ""
        ctrl_voc: ""
        form: ""
        user: ""

The indention is as expected but the keys of the upper levels are replaced by quotes (see arrows).

I receive the same output, if I use your second suggestion with sed.

Sundeep · Accepted Answer · 2020-01-22T08:44:16.623

1

See Default field separator for awk to understand what happens when FS has default value or set to single space character.

You can avoid it by using some other means to convey single space, like [ ]

$ awk -F'[ ]' -v dq="\"" 'NF>1 && $NF !~ /^[*&]|:$/{$NF = dq dq} 1' ip.txt
DEVELOPMENT: &development
  <<: *common

  check_access_token: ""

  database:
    mongodb:
      database: ""
      hostname: ""
      port: ""
      username: ""
      password: ""

      collection:
        col_1: ""
        col_2: ""
        col_3: ""
        col_4: ""

NF>1 to avoid changing empty lines
$NF !~ /^[*&]|:$/ check if last field doesn't start with * or & or doesn't end with :
if both the above conditions are satisfied, set last field to ""
1 is idiomatic way to print contents of $0

For the given sample, you can also use:

sed '/:$/! s/ [^*&][^ ]*$/ ""/' ip.txt

edited Jan 22 '20 at 08:44

answered Jan 22 '20 at 08:37

Sundeep

23,246
2
28
103

I do not receive the same output as you. See the extension of my question. – Molitoris Jan 23 '20 at 07:49
perhaps you have dos style line endings? what is the output you get for `file .config.yaml`? if it is dos ending, then using `:\r$` instead of `:$` would help.. see also: https://stackoverflow.com/questions/45772525/why-does-my-tool-output-overwrite-itself-and-how-do-i-fix-it – Sundeep Jan 23 '20 at 08:09

kvantour · Answer 2 · 2020-01-22T10:43:48.670

Question: How can I update a field without changing the original field-separators.

According to the awk POSIX standard, when you update a field using $i = expr it causes the value of $0 to be recomputed, with the fields being separated by the value of OFS.

For any field separator which is not an ere, the solution is simple. Changing field n is done as:

awk 'BEGIN{FS=OFS="string"}
     {$n="new_value"}
     { ... }' file

For other field-separators, this is a bit more problematic:

If FS=" " (default value), any spacing before and after the record are ignored, any combination of tabs and fields are used
If FS="ere" an Extended regular expression, you do not really know what the field separator is going to be. If FS="fo*, it can be anything from f to fooooooo.

In POSIX awk, you need to do do some nasty manipulations:

awk 'BEGIN{FS="ere"}
     # split original record
     { split($0,a,FS) }
     # update field value
     { a[n]="new_value" }
     # rebuild record
     {
       match($0,$1); rec=substr($0,1,RSTART-1); t=substr($0,RSTART+RLENGTH)
       for(i=1;i<NF;i++) {
          match(t,$(i+1)); rec = rec a[i] substr(t,1,RSTART-1)
          t=substr(t,RSTART+RLENGTH)
       }
       $0 = rec a[NF] t
     }
     { ... }' file

In GNU awk, you can make use of split in a bit more generic way as it has an extension to save the original separators:

awk 'BEGIN{FS="ere"}
     # split original record
     { split($0,a,FS,f) }
     # update field value
     { a[n]="new_value" }
     # rebuild record
     { rec=f[0]; for(i=1;i<=NF;i++) rec=rec a[i] f[i]; $0 = rec }
     { ... }' file

General comment: GNU awk could really benefit from a routine that reverses the split command and does a combine

How can I keep the leading white spaces while removing last field in awk?

Edit (after Sundeep's replay)

2 Answers2

Linked

Related