How would you search for data between two parentheses on many new lines

Question

Tried looking this up all over stack overflow with no dice. I have this test file

declare -a IP_LIST=(
  'x.x.x.x/24'
  'x.x.x.x/24'
  'x.x.x.x/24'
  'x.x.x.x/24'
  'x.x.x.x/24'
  'x.x.x.x/28'
)

I need to grab the IPs in the middle of this and throw them into an array (the rest of the file is just more bash code)

Tried using something like the following but it just returns none

list=re.search("IP_LIST=\(\n[\n.]*\)", out1)

Does `out1` contain full file content? – anubhava Dec 06 '21 at 17:48 — anubhava, Dec 06 '21 at 17:48
Note that this part `[\n.]*` repeats a newline or a dot. – The fourth bird Dec 06 '21 at 17:50 — The fourth bird, Dec 06 '21 at 17:50
out1 does include the entire text of the file – the_lonely_doge Dec 06 '21 at 17:57 — the_lonely_doge, Dec 06 '21 at 17:57
try this pattern: 'IP_LIST(.|\n)*\)' – Eric Dec 06 '21 at 18:01 — Eric, Dec 06 '21 at 18:01

score 3 · Answer 1 · edited Dec 06 '21 at 20:46

Using Pypi regex module, you can make use of \G regex:

(?:^declare -a IP_LIST=\(|(?!^)\G)\s*'([^']+)'(?=[^()]*\))

RegEx Demo

Code:

import regex

rx = r"(?:^declare -a IP_LIST=\(|(?!^)\G)\s*'([^']+)'(?=[^()]*\))"

s = ("declare -a IP_LIST=(\n"
    "  'x.x.x.x/24'\n"
    "  'x.x.x.x/24'\n"
    "  'x.x.x.x/24'\n"
    "  'x.x.x.x/24'\n"
    "  'x.x.x.x/24'\n"
    "  'x.x.x.x/28'\n"
    ")")

print (regex.findall(rx, s))

Output:

['x.x.x.x/24', 'x.x.x.x/24', 'x.x.x.x/24', 'x.x.x.x/24', 'x.x.x.x/24', 'x.x.x.x/28']

RegEx Details:

(?:: Start non-capture group
- ^declare -a IP_LIST=\(: Match declare -a IP_LIST=( at the start of a line
- |: OR
- (?!^)\G: Start with the position from end of the previous match
): End non-capture group
\s*: Match optional whitespace characters
': Match opening '
([^']+): Match text between quotes and capture it group #1
': Match closing '
(?=[^()]*\)): Positive lookahead to ensure that we have a closing ) ahead of the current position without matching ( or ) in between

Nice answer. I thought about using `findall` too but I struggled with the repetition because the standard `re` module doesn't have `\G`. — Matthias, Dec 06 '21 at 21:13

Matthias · Accepted Answer · 2021-12-06T21:04:47.667

If you don't mind using multiple steps this could be a solution.

out1 = """declare -a IP_LIST=(
    'x.x.x.x/24'
    'x.x.x.x/24'
    'x.x.x.x/24'
    'x.x.x.x/24'
    'x.x.x.x/24'
    'x.x.x.x/28'
)"""

data = re.search(r"IP_LIST=\(((\s*'([x./0-9]+)')+)", out1) # , re.MULTILINE)
print(data.group(1))

You might want to remove the "x" in [x./0-9] for a real ip address.

This will give you

'x.x.x.x/24'
'x.x.x.x/24'
'x.x.x.x/24'
'x.x.x.x/24'
'x.x.x.x/24'
'x.x.x.x/28'

Based on that result we can continue with simple string operations

result = [ip.strip("' ") for ip in data.group(1).strip().split('\n')]
print(result)

The result now is ['x.x.x.x/24', 'x.x.x.x/24', 'x.x.x.x/24', 'x.x.x.x/24', 'x.x.x.x/24', 'x.x.x.x/28'].

If you want to remove the subnet too then change the line where you create the result to

result = [ip.split('/')[0].strip("' ") for ip in data.group(1).strip().split('\n')]

To explain the regular expression as requested in the comment:

We have IP_LIST=\(((\s*'([x./0-9]+)')+). IP_LIST=\( matches "IP_LIST=(". This leaves us with ((\s*'([x./0-9]+)')+). The outer parentheses are there to define the group that will contain the text we want to get later. Take those away and we have (\s*'([x./0-9]+)')+.

Let's first focus on \s*'([x./0-9]+)'. \s* matches a bunch of potential whitespaces (space, tab, newline, ...). Then we have a literal '. The following group defines a set of characters: [x./0-9]+ (where 0-9 defines the numbers/characters from 0-9). As you can see all those characters are used in the ip addresses from your example (as I said, the "x" is due to the example data and might be dropped in the final code). The characters in this set can and will be repeated multiple times so we add the plus sign: [x./0-9]+. The final character is again a literal '.

Now comes the fun part. We have just explained \s*'([x./0-9]+)' which defines some whitespaces followed by an ip address. This might be repeated several times over multiple lines. So we wrap this in parentheses and add a + to allow repetitions of this part. Now we're back to ((\s*'([x./0-9]+)')+).

This works perfectly! My main issue was the regex on this one and targeting it across multiple lines. Since that is the case and I don't like to just grab code off of here. Would you be able to explain how you got to that regex match? — the_lonely_doge, Dec 06 '21 at 18:30
I just added the explanation. I thought I better do that at once. I had [another answer with regular expressions](https://stackoverflow.com/questions/5020906/python-convert-camel-case-to-space-delimited-using-regex-and-taking-acronyms-in/9283563#9283563) where I decided to add the explanation several months later and trying to understand my own regex after that time was no fun. — Matthias, Dec 06 '21 at 21:08
Regex is giving me hell so I appreciate your response. Now I have to edit this to exclude or skip anything matching the style of a comment like `#This IP is from virginia` which are beside a lot of these IP's — the_lonely_doge, Dec 06 '21 at 21:52

How would you search for data between two parentheses on many new lines

2 Answers2