Python library to generate regular expressions

Question

Is there any lib out there that can take a text (like a html document) and a list of strings (like the name of some products) and then find a pattern in the list of strings and generate a regular expression that would extract all the strings in the text (html document) that match the pattern it found?

For example, given the following html:

<table>
  <tr>
    <td>Product 1</td>
    <td>Product 2</td>
    <td>Product 3</td>
    <td>Product 4</td>
    <td>Product 5</td>
    <td>Product 6</td>
    <td>Product 7</td>
    <td>Product 8</td>
  </tr>
</table>

and the following list of strings:

['Product 1', 'Product 2', 'Product 3']

I'd like a function that would build a regex like the following:

'<td>(.*?)</td>'

and then extract all the information from the html that match the regex. In this case, the output would be:

['Product 1', 'Product 2', 'Product 3', 'Product 4', 'Product 5', 'Product 6', 'Product 7', 'Product 8']

CLARIFICATION:

I'd like the function to look at the surrounding of the samples, not at the samples themselves. So, for example, if the html was:

<tr>
  <td>Word</td>
  <td>More words</td>
  <td>101</td>
  <td>-1-0-1-</td>
</tr>

and the samples ['Word', 'More words'] I'd like it to extract:

['Word', 'More words', '101', '-1-0-1-']

Answer 1

Your requirement is at the same time very specific and very general.

I don't think you would ever find any library for your purpose unless you write your own.

On the other hand, if you spend too much time writing regex, you could use some GUI tools to help you build them, like: http://www.regular-expressions.info/regexmagic.html

However, if you need to extract data from html documents only, you should consider using an html parser, it should make things a lot easier.

I recommend beautifulsoup for parsing html document in python: https://pypi.python.org/pypi/beautifulsoup4/4.2.1

Answer 2

I'm pretty sure the answer to this question in the general case (without being pedantic) is no. The problem is that an arbitrary text, together with an arbitrary set of substrings of that text, do not rigorously define a single regular expression.

As a couple people have mentioned, a function could simply return .* for every set of inputs. Or it could return, for input strings ['desired', 'input', 'strings'], the regex

'(desired)+|(input)+|(strings)+'

Or plenty of other trivially correct but wholly useless results.

The issue you're facing is that in order to build a regex, you need to rigorously define it. And to do that, you need to describe the desired expression using language as expressive as the regex language you're working in... a string plus a list of substrings is not sufficient (just look at all the options a tool like RegexMagic needs to compute regular expressions in a limited environment!). In practical terms, this means that you need the regular expression you want, in order to compute it efficiently.

---

Of course, you could always go the million-monkeys route and attempt to evolve an appropriate regex somehow, but you're still going to have the problem of requiring a huge sample input of text + expected output in order to get a viable expression. Plus it'll take ages to run and probably be bloated six ways from Sunday with useless detritus. You'd likely be better off writing it yourself.

Answer 3

I had a similar problem. Pyparsing is a great tool to do exactly as you said.

https://github.com/pyparsing/pyparsing

It allows you to build expressions much list a regex but much more flexible. The site has some good examples.

Here is a quick script for the problem you posed above:

from pyparsing import *
cell_contents = []
results = []
text_string="""<table>
<tr>
     <td>Product 1</td>
     <td>Product 2</td>
     <td>Product 3</td>
     <td>Product 4</td>
     <td>Product 5</td>
     <td>Product 6</td>
     <td>Product 7</td>
     <td>Product 8</td>
</tr>
</table>"""

text_string = text_string.splitlines()
for line in text_string:
    anchorStart,anchorEnd = makeHTMLTags("td")
    table_cell = anchorStart + SkipTo(anchorEnd).setResultsName("contents") + anchorEnd
    for tokens,start,end in table_cell.scanString(line):
        cell_contents = ''.join(tokens.contents)
        results.append(cell_contents)

for i in results:
    print i

Answer 4

Try this:

https://github.com/noprompt/frak

It's written in Clojure, and there are no guarantees what it outputs is the most concise expression, but seems to have some potential

Answer 5

Perhaps it would be better to use a Python HTML parser that supports XPATHs (see this related question), look for the bits of interest in the HTML code, and then record their XPATHs - or at least the ones shared by more than one of the examples?

Answer 6

const table = document.querySelector("table");
const rows = table.querySelectorAll("tr");

let array = [];

for (const row of rows) {
  const cells = row.querySelectorAll("td");
  let rowArray = [];
  for (const cell of cells) {
    rowArray.push(cell.textContent);
  }
  array.push(rowArray);
}

console.log(array);

Answer 7

Rather than generating a regex, how about using a more general regex? If your data is constrained to the inner text of an element that does not itself contain elements, then this regex used with re.findall will yield a list of tuples where each tuple is (tagname, text):

r'<(?P<tag>[^>]*)>([^<>]+?)</(?P=tag)>'

You could then extract just the text from each tuple easily.