Extract all substrings between two markers for a very long string

Question

This is a continuation of the question Extract all substrings between two markers. The answers by @Daweo and @Tim Biegeleisen works for small strings.

But for very large strings regular expressions doesn't seem to work. This could be because of a of a limit on string length as seen below:

>>> import re
>>> teststr = "&marker1\nThe String that I want /\n&marker1\nAnother string that I want /\n"
>>> for i in range(0, 23):
...    teststr += teststr # creating a very long string here
... 
>>> len(teststr)
603979776
>>> found = re.findall(r"\&marker1\n(.*?)/\n", newstr)
>>> len(found)
46
>>> found
['The String that I want ', 'Another string that I want ', 'The String that I want ', 'Another string that I want ', 'The String that I want ', 'Another string that I want ', 'The String that I want ', 'Another string that I want ', 'The String that I want ', 'Another string that I want ', 'The String that I want ', 'Another string that I want ', 'The String that I want ', 'Another string that I want ', 'The String that I want ', 'Another string that I want ', 'The String that I want ', 'Another string that I want ', 'The String that I want ', 'Another string that I want ', 'The String that I want ', 'Another string that I want ', 'The String that I want ', 'Another string that I want ', 'The String that I want ', 'Another string that I want ', 'The String that I want ', 'Another string that I want ', 'The String that I want ', 'Another string that I want ', 'The String that I want ', 'Another string that I want ', 'The String that I want ', 'Another string that I want ', 'The String that I want ', 'Another string that I want ', 'The String that I want ', 'Another string that I want ', 'The String that I want ', 'Another string that I want ', 'The String that I want ', 'Another string that I want ', 'The String that I want ', 'Another string that I want ', 'The String that I want ', 'Another string that I want ']

What could I do to resolve this and find all occurrences between the makers start="&maker1" and end="/\n" ? What is the maximum string length that re can handle?

Works on my machine, at least when I replaced `newstr` by `teststr`. — Ronald, Jun 12 '20 at 11:27
It works on my home machine, with Python 3.8.3. `len(found)` is printed as `16777216`. — Błotosmętek, Jun 12 '20 at 12:19
Still works ;-) I suppose pushing it to the limits depends on the memory of your system? But what is the error you get? — Ronald, Jun 12 '20 at 12:25

score 0 · Accepted Answer · answered Jun 12 '20 at 11:43

I couldn't get re.findall to work. Now I do use re but to find the location of markers and extract the substrings manually.

locs_start = [match.start() for match in re.finditer("\&marker1", mylongstring)]
locs_end = [match.start() for match in re.finditer("/\n", mylongstring)]

substrings = []
for i in range(0, len(locs_start)):
    substrings.append(mylongstring[locs_start[i]:locs_end[i]+1])

Extract all substrings between two markers for a very long string

1 Answers1