Trouble with regex and sed in Fabric

Question

I am writing a fabfile, and I am having trouble with a replacement. I have already checked using regex101, and it shows that my regex works (both with and without the 'extended' switch). However, when I run the fabfile, there is no replacement. The regex is:

ALLOWED_HOSTS\ ?=\ ?\[([^]]+)\]

It is supposed to match any ALLOWED_HOSTS in a Django settings.py-file, from empty definitions to multiliners. The minimum text to match is

ALLOWED_HOSTS = []

A multiline text to match could look like this:

ALLOWED_HOSTS = [
    'django-stage.somedomain.com',
    'django-deployment.somedomain.com',
    'localhost',
]

In other words, the regex should match a string containing 'ALLOWED_HOSTS', followed by 0 or 1 spaces, followed by an equal sign '=', followed by 0 or 1 spaces, followed by a forward bracket '[', followed by any text (including line breaks), followed by a backward bracket ']'. However, when the fabfile is run, the lines above are not changed. The full function in the fabfile:

def _update_settings(source_folder, site_name):
    settings_path = source_folder + '/appname/settings.py'
    sed(settings_path, "DEBUG = True", "DEBUG = False")
    sed(settings_path,
        'ALLOWED_HOSTS\ ?=\ ?\[([^]]+)\]',
        f'ALLOWED_HOSTS = ["{site_name}"]'
    )

When run, the output is:

[django-stage.somedomain.com] run: sed -i.bak -r -e 's/ALLOWED_HOSTS\ ?=\ ?[([^]]+)]/ALLOWED_HOSTS = ["django-stage.somedomain.com"]/g' "$(echo /srv/django-stage.somedomain.com/source/appname/settings.py)"

Any help much appreciated.

`sed` operates on a line by line basis. You may tell it to read the whole file in before, then, you will be able to match across lines there. See [replacing multiple line pattern in sed](https://stackoverflow.com/questions/8164604/replacing-multiple-line-pattern-in-sed) — Wiktor Stribiżew, Feb 11 '19 at 13:03
why dont you use fabric and django integration? http://docs.fabfile.org/en/1.14/api/contrib/django.html — Tiago Gomes, Feb 11 '19 at 13:14
@wiktor-stribiżew; good tip. The only problem is that I am not really using sed, but rather fabric.contrib.files.sed, which makes it difficult to apply the multiline solution, see also https://docs.fabfile.org/en/1.14/api/contrib/files.html — ElToro1966, Feb 11 '19 at 16:45
The most important thing to understand about regexp is this - there is no such thing as **a regexp**. There are BREs, and EREs, and PCREs, and different tools have delimiters and backreferences, etc. that they support and some given various arguments support different regexp flavors and/or support single line vs multi-line, etc. Using some online tool like regex101 to try to verify a regexp that you actually intend to use in sed or any other command-line tool is pointless since the regexp flavors each accepts will vary. — Ed Morton, Feb 11 '19 at 18:13
@tiago-gomex: The sed is part of a deployment script that - among other things - changes the Django settings-file. It must be changed in-place, because there may be restarts after the deployment (etc). — ElToro1966, Feb 11 '19 at 18:17
@ed-morton: regex101 takes into acoount different flavours in the sense that it you can choose ECMA, PCRE, Python or Golang flavours (It is also the recommended tool by StackOverflow for sorting out regex expressions btw). — ElToro1966, Feb 11 '19 at 18:23
@ElToro1966 that's fine but based on the many, many questions posted here that start with "I know my regex is correct as I validated it on regex101 but..." it's not actually useful for creating/verifying regexps to use in command-line UNIX tools. — Ed Morton, Feb 11 '19 at 18:31
Heeded @EdMorton's advice and tested sed on the server. Also looked into multiple line manipulation in sed as mentioned as set out in the sed manual Ch.6 - https://www.gnu.org/software/sed/manual/sed.pdf, and came up with the following expression: sed -r -e "/./{H;\$!d} ; x ; s/ALLOWED_HOSTS\ ?=\ ?\[([^]]+)\]/ALLOWED_HOSTS = \[\'django-stage.somedomain.com\'\]/" test.txt Now the challenge is to make it work in Fabric. Using the run command should do it. — ElToro1966, Feb 12 '19 at 13:38
Correction: The sed should read sed -e '1h;2,$H;$!d;g' -re "s/ALLOWED_HOSTS\ ?=\ ?[([^]]*)]/ALLOWED_HOSTS = \['django-stage.somedomain.com'\]/" test.txt — ElToro1966, Feb 12 '19 at 15:55

score 0 · Answer 1 · answered Feb 12 '19 at 16:52

I think I'll take the liberty of answering my own question as I've got the sript above working now. The function needs to look like this:

def _update_settings(source_folder, site_name):
    settings_path = source_folder + '/appname/settings.py'
    sed(settings_path, "DEBUG = True", "DEBUG = False")
    run(
        "sed -e '1h;2,$H;$!d;g' -i.bak -re 's/"
        + "ALLOWED_HOSTS\ ?=\ ?\[([^]]*)\]/"
        + "ALLOWED_HOSTS = \[\"" + site_name + "\"\]/' "
        + settings_path
        )

The meaning of -e '1h;2,$H;$!d;g' is emminently explained by @antak in "How can I use sed to replace a multi-line string?" .

Switches:

-r: The -r switch is for extended regex (also -E or --regexp-extended).

-e: Separates commands.

-i: Replace in-place. i.bak means that the original file is backed-up to filename.bak.

As for the regex pattern:

ALLOWED_HOSTS\ ?=\ ?\[([^]]*)\]

ALLOWED_HOSTS matches the characters ALLOWED_HOSTS literally (case sensitive)
\ ? matches the space character literally
? Quantifier — Matches between zero and one times, as many times as possible
= matches the equal sign literally
\ ? matches the space character literally
? Quantifier — Matches between zero and one times
\[ matches the left square bracket literally
1st Capturing Group ([^]]*)
Match a single character not present in the list [^]]*, that is, a character that is not a right square bracket
The * quantifier — Matches between zero and unlimited times, as many times as possible
] in the expression ^] matches the right square bracket character literally
\] matches the right square bracket character literally

The output from running the fabfile (site_name = somedomain):

[somedomain] run: sed -e '1h;2,$H;$!d;g' -i.bak -re 's/ALLOWED_HOSTS\ ?=\ ?\[([^]]*)\]/ALLOWED_HOSTS = \["somedomain"\]/' /path_to_settings_file/settings.py

Trouble with regex and sed in Fabric

1 Answers1