NLTK RegexpTokenizer: Regex to retain just characters in Random text

Question

I used tokenizer = RegexpTokenizer(r'\w+') which retains alphanumeric characters But how do I combine a regular expression to remove every other element retaining just characters greater than length 2

Below is one row in the dataframe which contains random text

0 [ANOTHER 2'' F/P SAMPLE 01:52 ...A13232 / AS OUTPUT MSG...

Do you think `RegexpTokenizer(r'\w{3,}')` ? – jezrael Oct 25 '17 at 05:37 — jezrael, Oct 25 '17 at 05:37
If only things were this simple! It worked. – Hackerds Oct 25 '17 at 05:40 — Hackerds, Oct 25 '17 at 05:40

score 5 · Accepted Answer · answered Oct 25 '17 at 05:41

5

I think you need for find words with len>2:

RegexpTokenizer(r'\w{3,}')

Or if need only letters:

RegexpTokenizer(r'[a-zA-Z]{3,}')

answered Oct 25 '17 at 05:41

jezrael

822,522
95
1,334
1,252

Both work fine. – Hackerds Oct 25 '17 at 05:43
Super, you are welcome! – jezrael Oct 25 '17 at 05:43

NLTK RegexpTokenizer: Regex to retain just characters in Random text

1 Answers1