Given an Array of strings how do I Remove Duplicates?

Question

I would like to know how to remove duplicate strings from a container, but ignore word differences from trailing punctuation.

For example given these strings:

Why do do we we here here?

I would like to get this output:

Why do we here?

Answer 1

The algorithm:

1. While Reading a word is successful, do:
2. If End of file, quit.
3. If word list is empty, push back word.
4. else begin
   Search word list for the word.
5. if word doesn't exist, push back the word.
   end else (step 4)
6. end (while reading a word)

Use std::string for your word. This allows you to do the following:

std::string word;
while (data_file >> word)
{
}

Use std::vector to contain your words (although you could use std::list as well). The std::vector grows dynamically so you don't have to worry about reallocation if you picked the wrong size.
To append to std::vector, use the push_back method.

To compare std::string, use operator==:

std::string new_word;
std::vector<std::string> word_list;
//...
if (word_list[index] == new_word)
{
  continue;
}

Answer 2

So you have said you know how to tokenize a string. (If you don't spend some time here: https://stackoverflow.com/a/38595708/2642059) So I'm going to assume that we're given a vector<string> foo which contains words with possibly trailing punctuation.

for(auto it = cbegin(foo); it != cend(foo); ++it) {
    if(none_of(next(it), cend(foo), [&](const auto& i) {
                                                         const auto finish = mismatch(cbegin(*it), cend(*it), cbegin(i), cend(i));
                                                         return (finish.first == cend(*it) || !isalnum(*finish.first)) && (finish.second == cend(i) || !isalnum(*finish.second));
                                                        })) {
        cout << *it << ' ';
    }
}

Live Example

It's worth noting here that you haven't given us rules on how to handle words like: "down", "down-vote", and "downvote" This algorithm presumes that the 1^st 2 are equal. You also haven't given us rules for how to handle: "Why do, do we we here, here?" This algorithm always returns the final repetition, so the output would be "Why do we here?"

If the presumptions made by this algorithm are not totally to your liking leave me a comment and we'll work on getting you comfortable with this code to where you can make the adjustments that you need.