java - Hashset contain duplicates?

Question

I am trying to extract all the text from a PDF and store it inside a HashSet. As I know, HashSet does not contain duplicates so it will ignore the duplicates when I extract them. However, when I print out the results of the hash, I noticed there's duplicate blank space in it.

I want to insert the hash values into my table in MySQL but it has a primary key constraint so that gives me some trouble. Is there a way I could remove entirely all sorts of duplicate in my hash?

My code to extract the text :

public static void main(String[] args) throws Exception {
      String path ="D:/PDF/searchable.pdf";
        HashSet<String> uniqueWords = new HashSet<>();
        try (PDDocument document = PDDocument.load(new File(path))) {

            if (!document.isEncrypted()) {

                PDFTextStripper tStripper = new PDFTextStripper();
                String pdfFileInText = tStripper.getText(document);
                String lines[] = pdfFileInText.split("\\r?\\n");
                for (String line : lines) {
                    String[] words = line.split(" ");

                    for (String word : words) {
                        uniqueWords.add(word);

                    }

                }
              System.out.println(uniqueWords);

            }
        } catch (IOException e){
            System.err.println("Exception while trying to read pdf document - " + e);
        }
        Object[] words =  uniqueWords.toArray();
        System.out.println(words[1].toString());

        MysqlAccess connection=new MysqlAccess();

        for(int i = 1 ; i <= words.length - 1 ; i++ ) {

            connection.readDataBase(path, words[i].toString());

        }

        System.out.println("Completed");

    }

}

This is my hash:

[, highlight, of, Even, copy, file,, or, ., ,, 1, reader,, different, D, F, ll, link, ea, This, ed, document, V, P, ability, regardless, g, d, text., e, b, a, n, o, web, l, footnote., should, Most, IDRH, selection, text-searchable, positioning, u, s, what, r, PDF., happens, er, y, x, to, body, single, ca, te, together, ti, th, would, when, be, Text-Searchable, document,, text, isn't, such, kinds, sh, co, ld, font,, example, ch, this, attempt, have, t,, Notice,, contained, from, re, text.1, page,, style, page., able, if, is, You, standard, PDF, your, as, readers, you, the, in, main, an, iz]

If they are unique, why does it throws " Duplicate entry for key PRIMARY" when I try to insert into a primary key column?

Any suggestion would be appreciated.

Your input might have also contained things like `,`, spaces, tabs and so on. — Jai, Oct 10 '18 at 02:55
Could `reader,` be a word? You don't appear to handle punctuation. — dave, Oct 10 '18 at 02:55
`1, reader,, different` is most likely `1`, `reader,` and `different` instead of `1`, `reader`, and `different`. — Jai, Oct 10 '18 at 02:59
@Jai So they are four different words, Should still be able to insert into my db — TomCold, Oct 10 '18 at 03:00
Your database might have a different notion of uniqueness. For example, it might treat `foo` and `FOO` as the same value. The error message should tell you exactly where it's failing. — shmosel, Oct 10 '18 at 03:02
@Jai Nope, it will insert some data halfway and eventually throws error Duplicate entry for key Primary — TomCold, Oct 10 '18 at 03:05
@shmosel So even though the hashset is meant to ignore duplicates, it still stores Foo and foo but the database primary reads it as duplicates? — TomCold, Oct 10 '18 at 03:07
Your update to your question indicates the non-uniqueness occurs when you try and store the words in the DB. Which suggests that your `Set` and your DB have different meanings for uniqueness. Most likely related to case sensitivity. — dave, Oct 10 '18 at 03:10
@dave I believe so. Then it's gonna be a big problem for me to sort — TomCold, Oct 10 '18 at 03:13
You can use a case-insensitive set: `new TreeSet<>(String.CASE_INSENSITIVE_ORDER)` — shmosel, Oct 10 '18 at 03:17
I have a SQL statement INSERT query to insert hashset data into the db, would there be a way to ignore duplicate? — TomCold, Oct 10 '18 at 03:20
@shmosel I just did that and it works. But does it mean if I have Fish and fish, it only inserts whichever comes first and ignore the second one? — TomCold, Oct 10 '18 at 03:23
Yes. Unless you [define the column](https://stackoverflow.com/q/6448825/1553851) to be case-sensitive. — shmosel, Oct 10 '18 at 03:27
@shmosel So I guess this solves the problem? Could anyone post this as answer and I'll mark it. — TomCold, Oct 10 '18 at 03:29

Udaya Shankara Gandhi Thalabat · Accepted Answer · 2018-10-10T03:25:31.543

HashSet doesn't allow any duplicate to be entered to it.

Here is the add(E e) method description of HashSet class:

public boolean add(E e)

Adds the specified element to this set if it is not already present. More formally, adds the specified element e to this set if this set contains no element e2 such that (e==null ? e2==null : e.equals(e2)). If this set already contains the element, the call leaves the set unchanged and returns false.

In your case, you are be getting string array with strings having single space and strings with multiple spaces while you are calling split method on pdfFileInText causing to have your HashSet data structure to have both single spaced string and multi-spaced string also. But while inserting to the database somewhere the string is being trimmed causing the duplicate entry.

To elaborate more on this please look into the below code snippet:

public class TestHashSetUniqueness {
public static void main(String[] args) {
    HashSet<String> hashSet = new HashSet<String>();
    String oneSpace = " ";
    String twoSpaces = "  ";

    hashSet.add(oneSpace);
    hashSet.add(twoSpaces);

    // Here HashSet size is 2 as it is treating string objects oneSpace
    // and twoSpaces as two different strings.
    System.out.println("HashSet size without trim() : "+hashSet.size());

    hashSet.clear();
    hashSet.add(oneSpace.trim());
    hashSet.add(twoSpaces.trim());

    // As we are trimming(removing the excess spaces) spaces in the strings
    // causing our HashSet to have only one element there by avoiding duplicates
    System.out.println("HashSet size with trim() : "+hashSet.size());
}

}

So while adding your strings to HashSet call trim() on the string to fix your issue.

I hope this answers your question.

Does it ignore duplicates for lowercase and uppercase like Fish and fish? My SQL won't let me insert if these two elements are in the hashset — TomCold, Oct 10 '18 at 03:18
No it doesn't. For a `HashSet` the uniqueness criteria is the result of `String::equals` — Stephen C, Oct 10 '18 at 03:29
To filter duplicates based on its case, you need to use TreeSet with CASE_INSENSITIVE_ORDER comparator. Please go through the below link to get more idea. https://stackoverflow.com/questions/24558456/java-setstring-equality-ignore-case — Udaya Shankara Gandhi Thalabat, Oct 10 '18 at 03:31

java - Hashset contain duplicates?

1 Answers1