How to clean dirty text using java

Question

I am working on collecting data from twitter and making processing on it, but i have the problem that: text is dirty,

example :

String dirtyText="this*is#a*&very_dirty&String";

example :

String dirtyText="All f dis happnd bcause u gave ur time, talent n passion.";

please i want it as simple as possible.

your input dirty string does not match with your output dirty string? — Kick Buttowski, Apr 02 '15 at 16:16
For us to suggest a solution, you must tell us what your requirements mean by "clean" — Samuel, Apr 02 '15 at 16:18
First one might be doable by removing non alphabetic character. But there is no "simple" way of doing the second one — ortis, Apr 02 '15 at 16:19
i want to analyze people text in twitter (tokenize,POS) but if text in that form, this will not work or if there is another way to deal with it — mohamed nagah, Apr 02 '15 at 17:58
@KickButtowski i get text to analyze and use it in my project ,, — mohamed nagah, Apr 02 '15 at 18:58
@Samuel clean means convert text : (this*is#a*&very_dirty&String --> this is very dirty string) ..... (All f dis happnd --> all of this happened) — mohamed nagah, Apr 02 '15 at 18:59

score 0 · Accepted Answer · edited May 23 '17 at 12:06

This is not an easy problem to solve. All f dis happnd could be "cleaned" to produce All *of* this happened or All *if* this happened. For the first example, you can merely replace all non-alphabetic characters with spaces. See this question for how to do that.

Otherwise I think you would need a natural language processor, or at the very least a spell checker. To guess what a Tweet should be in correct english is an extremely complex problem to solve. Take a look at Jazzy for an open source spell checker.

Daham Mustafa · Answer 2 · 2021-08-19T16:46:44.340

public class CleaningDirtText { /* * remove leading and trailing spaces, and split our words into a String array. * The split method allows you to break apart text on a given delimiter. In this * case, we chose to use the regular expression \W, which represents anything * that is not a word character: / private static final String dirtyText = "thisis#a*&very_dirty&String";

public static void main(String[] args) {
    System.out.println(dirtyText);
    String[] words =    dirtyText.toLowerCase().trim().split("[\\W\\d]+");
    // System.out.println(preparedText);
    //String[] cleanText = dirtyText.toLowerCase().trim().split("[\\W]+");
    for (String clean : words) {
        System.out.print(clean + " ");
    }
}

}

Why are you posting your [answer](https://stackoverflow.com/a/68839324/6670491) twice? Can you [edit](https://stackoverflow.com/posts/68839282/edit) and explain why your code can solve OP's issue? — HardcoreGamer, Aug 19 '21 at 06:08

score 0 · Answer 3 · answered Aug 18 '21 at 21:06

public class CleaningDirtText { private static final String dirtyText = "thisis#a&very_dirty&String";

public static void main(String[] args) {
    /*
     * remove leading and trailing spaces, and split our words into a String array.
     * The split method allows you to break apart text on a given delimiter. In this
     * case, we chose to use the regular expression \\W, which represents anything
     * that is not a word character:
     */

    System.out.println(dirtyText);
    String[] words = dirtyText.toLowerCase().trim().split("[\\W\\d]+");
    for (int i = 0; i < words.length; i++) {
        System.out.print(words[i]);
    }
    System.out.println("\nsee the cleand text:-");
    for (String clean : words) {
        System.out.print(clean + " ");
    }
}

}

How to clean dirty text using java

3 Answers3