Replace duplicate senteces in text

Question

I have a big list of sentences, some of them are similar to each other but a bit different. something like:

[word1] [word2] [word3]

[word1] [word3]

[word1] [word2] [word3] [word4]

I would like to delete "duplicates" and get only one sentence. Just asking if it is possible in java?

I have list of 10k sentences similar do each other and I would like to get ~1k without duplicates(some of them have 5 copies, some 20) one for each sentence — Skillzone, Nov 04 '16 at 11:01

score 0 · Answer 1 · answered Nov 04 '16 at 11:00

0

you can do it like this

for (int i = 0; i < words.length; i++)
{
    for (int j = 0; j < words.length; j++)
    {
         if (words[i].equals(words[j]))
         {
         if (i != j)
         words[i] = "";

         }
     }
}

answered Nov 04 '16 at 11:00

Sandeep Singh

21
3

Better approach is to create a new array with the result instead of overwriting the current array with empty string – B001ᛦ Nov 04 '16 at 11:03

score 0 · Answer 2 · answered Nov 04 '16 at 11:11

add the list to a Set.. set will not have duplicates.. refer below code..

    List<String> collectionWithDuplicates = new ArrayList<>();

    Set<String> collectionWithoutDuplicates = new HashSet<>();

    collectionWithoutDuplicates.addAll(collectionWithDuplicates);

score 0 · Answer 3 · answered Nov 04 '16 at 11:31

0

In Java 8

 List<String> newList = oldList.stream().distinct().collect(Collectors.toList());

answered Nov 04 '16 at 11:31

Eritrean

15,851
3
22
28

Replace duplicate senteces in text

3 Answers3