2

I have a string which could be arbitrary long say

s = 'Choose from millions of possibilities on Shaadi.com. Create your profile, search&contact; your special one.RegisterFree\xa0\xa0\xa0unsubscribing reply to this mail\xa0\n and 09times and this is limited time offer! and this is For free so you are saving cash'

I have a list of spam words which could be like

p_words = ['cash', 'for free', 'limited time offer']

All I want to know if there pattern exists in the input text and how many times?

It becomes simple when it has just one word

import re
p = re.compile(''.join[p_words])  # correct me if I am wrong here
m = p.match(s)

but it could be a bi-gram, tri-gram or n-gram

How do we approach this?

daydreamer
  • 87,243
  • 191
  • 450
  • 722
  • If you need case-insensitive keywords search see [my answer that shows the difference between full Unicode casefolding and mere `.lower()` method](http://stackoverflow.com/a/9030564/4279) – jfs Feb 15 '12 at 01:34

3 Answers3

4
p = re.compile('|'.join(re.escape(w) for w in p_words))

p will then match any of the strings in p_words.

Amber
  • 507,862
  • 82
  • 626
  • 550
  • There's no reason it wouldn't. – Amber Feb 15 '12 at 00:55
  • @J.F.Sebastian It does if you use `re.finditer` or `re.findall` with the resulting regex. – Amber Feb 15 '12 at 04:07
  • @Amber: `re.find*` enumerate the matches but still by themselves do not answer "how many times". – jfs Feb 15 '12 at 04:36
  • @J.F.Sebastian - Given that the OP was already using regex for their problem when not targeting ngrams, it seems logical to assume that they were able to implement the counting portion as necessary for their purposes. – Amber Feb 15 '12 at 06:17
2

If the text and number of words is not very large you could start with, example:

d = {w: s.count(w) for w in p_words if w in s}
# -> {'cash': 1, 'limited time offer': 1}

You could compare its performance with:

import re
from collections import Counter

p = re.compile('|'.join(map(re.escape, p_words)))
d = Counter(p.findall(s))
# -> Counter({'limited time offer': 2, 'cash': 2})

For reference compare its speed with fgrep. It should be fast at matching multiple strings in the input stream:

$ grep -F -o -f  patternlist.txt largetextfile.txt  | sort | uniq -c

Output

  2 cash
  2 limited time offer
jfs
  • 399,953
  • 195
  • 994
  • 1,670
  • Thank you @J.F.Sebastian for educating me that there exists Counter, I was not aware of it – daydreamer Feb 15 '12 at 19:59
  • @daydreamer: note `Counter` is not the fastest if you use this algorithm: [performance comparison](http://stackoverflow.com/a/2525617/4279) – jfs Feb 15 '12 at 20:15
1

Regexes use the '|' separator. Replace spaces in each case with something like '\W+', which matches non-letters, and I think you're good to go.

zpmorgan
  • 138
  • 1
  • 5