0

I would need to remove all words (or replace them with spaces) in strings that have non-alphabetic characters (except hyphens and apostrophes) in the middle in R. Could anyone kindly help? Thanks.

e.g.
str = "he@llo wor*ld i'm using state-of-the-art technologies it's i4u"
expected output " i'm using state-of-the-art technologies it's "




I have tried the following regex.

lines <- c("i'm",
           'gas-lighting',
           "i'm gas-lighting",
           "i-love-you",
           "i@u",
           "b2b",
           "i'm gas-lighting u i@u b2b")

gsub("\\w+[^a-z'-]+\\w+", " ", lines) 
[1] "i'm"          "gas-lighting" "i' -lighting" "i-love-you"   " "            
" "            "i' -     "

The problem is the space between words? Tried to skip space.

gsub("\\w+[^a-z\\s'-]+\\w+", " ", lines)**  
[1] "i'm"          "gas-lighting" "i' -lighting" "i-love-you"   " "            
" "            "i' -     "

It wouldn't skip the spaces? Expected the following strings. 

[1] "i'm"          "gas-lighting" "i'm gas-lighting" "i-love-you"   " "            
" "            "i'm gas-lighting u    "




Update 2: OK, this works fine so far.

> lines <- c("i'm",
+            'gas-lighting',
+            "i'm gas-lighting",
+            "i-love-you",
+            "i@u",
+            "b2b",
+            "i'm gas-lighting u and you and you i@u b2b",
+            " he@llo wor$ld how*are&you ")
>
> # split a string at spaces then remove the words 
> # that contain any non-alphabetic characters (excpet "-", "'")
> # then paste them together (separate them with spaces)
> unlist(lapply(lines, function(line){
+   words <- unlist(strsplit(line, "\\s+"))
+   words <- words[!grepl("[^a-z'-]", words, perl=TRUE)]
+   paste(words, collapse=" ")}))
[1] "i'm"                                "gas-lighting"                      
[3] "i'm gas-lighting"                   "i-love-you"                        
[5] ""                                   ""                                  
[7] "i'm gas-lighting u and you and you" ""

Update 1: So far I am using the following regex.

> # replace word at the beginning of a string
> lines <- gsub("^\\s*\\w*[^a-z'-]+\\w*", " ", lines); lines
[1] "i'm"                     "gas-lighting"            "i'm gas-lighting"        "i-love-you"             
[5] " "                       " "                       "i'm gas-lighting u i@u "
> # replace word at the end of a string
> lines <- gsub("\\s[a-z]+[^a-z'-]+\\w*$", " ", lines); lines 
[1] "i'm"                     "gas-lighting"            "i'm gas-lighting"        "i-love-you"             
[5] " "                       " "                       "i'm gas-lighting u i@u "
> # replace words between spaces
> gsub("\\s\\w*[^a-z'-]+\\w*\\s", " ", lines)
[1] "i'm"                 "gas-lighting"        "i'm gas-lighting"    "i-love-you"          " "                  
[6] " "                   "i'm gas-lighting u "
nov05
  • 151
  • 2
  • 11
  • did you try`\b` to pick up word boundaries? – bob1 Jan 27 '19 at 03:13
  • From [here](https://stackoverflow.com/questions/24395382/r-code-removing-words-containing), you can try `gsub('\\S*[@*0-9]\\S*', '', x)`. – A. Suliman Jan 27 '19 at 05:34
  • Thanks. I would need to remove all kinds of non-alphabetic characters, such as !@#$%^&*() even non-English letters. – nov05 Jan 27 '19 at 06:25

2 Answers2

0

I came up with an indirect way, but it worked.

library(tidyverse)

str = "he@llo wor*ld i'm using state-of-the-art technologies it's i4u"

##Break the string based on spaces
break_1 <- (str_split(str, pattern = "\\s"))

##Find the good words and put them in a vector
good_words <- unlist(break_1)[!sapply(break_1,
                                      function(i)str_detect(i,pattern = "[^(Aa-zZ|\\-|')]"))]

##Merge the vector
merged_vector <- paste0(good_words, collapse = " ")
merged_vector
Henry Cyranka
  • 2,970
  • 1
  • 16
  • 21
  • Thanks. I have thought about it too. I have large text files (200 Mb per file) to process. Not sure how much the performance would be affected. – nov05 Jan 27 '19 at 03:44
0

As a variation of Harro Cyranka with grepl

paste0(sapply(break_1, function(x) x[!grepl("[^Aa-zZ|'|-]", x)]), collapse = " ")
Brian Syzdek
  • 873
  • 6
  • 10