Count unique words in a string in C++

Question

I want to count how many unique words are in string 's' where punctuations and newline character (\n) separates each word. So far I've used the logical or operator to check how many wordSeparators are in the string, and added 1 to the result to get the number of words in string s.

My current code returns 12 as the number of word. Since 'ab', 'AB', 'aB', 'Ab' (and same for 'zzzz') are all same and not unique, how can I ignore the variants of a word? I followed the link: http://www.cplusplus.com/reference/algorithm/unique/, but the reference counts unique item in a vector. But, I am using string and not vector.

Here is my code:

#include <iostream>
#include <string>
using namespace std;

bool isWordSeparator(char & c) {

    return c == ' ' || c == '-' || c == '\n' || c == '?' || c == '.' || c == ','
    || c == '?' || c == '!' || c == ':' || c == ';';
}

int countWords(string s) {
    int wordCount = 0;

    if (s.empty()) {
    return 0;
    }

    for (int x = 0; x < s.length(); x++) {
    if (isWordSeparator(s.at(x))) {
            wordCount++;

    return wordCount+1;

int main() {
    string s = "ab\nAb!aB?AB:ab.AB;ab\nAB\nZZZZ zzzz Zzzz\nzzzz";
    int number_of_words = countWords(s);

    cout << "Number of Words: " << number_of_words  << endl;

    return 0;

}

Answer 1

You could create a set of strings, save the position of the last separator (starting with 0) and use substring to extract the word, then insert it into the set. When done just return the set's size.

You could make the whole operation easier by using string::split - it tokenizes the string for you. All you have to do is insert all of the elements in the returned array to the set and again return it's size.

Edit: as per comments, you need a custom comparator to ignore case for comparisons.

Answer 2

What you need to make your code case-insensitive is tolower().
You can apply it to your original string using std::transform:

std::transform(s.begin(), s.end(), s.begin(), ::tolower);

I should add however that your current code is much closer to C than to C++, perhaps you should check out what standard library has to offer.

I suggest istringstream + istream_iterator for tokenizing and either unique_copy or set for getting rid of the duplicates, like this: https://ideone.com/nb4BEH

Answer 3

While splitting the string into words, insert all words into a std::set. This will get rid of the duplicates. Then it's just a matter of calling set::size() to get the number of unique words.

I'm using the boost::split() function from the boost string algorithm library in my solution, because is almost standard nowadays. Explanations in the comments in code...

#include <iostream>
#include <string>
#include <set>
#include <boost/algorithm/string.hpp>
using namespace std;

// Function suggested by user 'mshrbkv':
bool isWordSeparator(char c) {
    return std::isspace(c) || std::ispunct(c);
}

// This is used to make the set case-insensitive.
// Alternatively you could call boost::to_lower() to make the
// string all lowercase before calling boost::split(). 
struct IgnoreCaseCompare { 
    bool operator()( const std::string& a, const std::string& b ) const {
        return boost::ilexicographical_compare( a, b );
    }
};

int main()
{
    string s = "ab\nAb!aB?AB:ab.AB;ab\nAB\nZZZZ zzzz Zzzz\nzzzz";

    // Define a set that will contain only unique strings, ignoring case.
    set< string, IgnoreCaseCompare > words;

    // Split the string by using your isWordSeparator function
    // to define the delimiters. token_compress_on collapses multiple
    // consecutive delimiters into only one. 
    boost::split( words, s, isWordSeparator, boost::token_compress_on );

    // Now the set contains only the unique words.
    cout << "Number of Words: " << words.size() << endl;
    for( auto& w : words )
        cout << w << endl;

    return 0;
}

Demo: http://coliru.stacked-crooked.com/a/a3b51a6c6a3b4ee8

Answer 4

First of all I'd suggest rewriting isWordSeparator like this:

bool isWordSeparator(char c) {
    return std::isspace(c) || std::ispunct(c);
}

since your current implementation doesn't handle all the punctuation and space, like \t or +.

Also, incrementing wordCount when isWordSeparator is true is incorrect for example if you have something like ?!.

So, a less error-prone approach would be to substitute all separators by space and then iterate words inserting them into an (unordered) set:

#include <iterator>
#include <unordered_set>
#include <algorithm>
#include <cctype>
#include <sstream>

int countWords(std::string s) {
    std::transform(s.begin(), s.end(), s.begin(), [](char c) { 
        if (isWordSeparator(c)) {
            return ' ';
        }

        return std::tolower(c);
    });

    std::unordered_set<std::string> uniqWords;

    std::stringstream ss(s);
    std::copy(std::istream_iterator<std::string>(ss), std::istream_iterator<std::string(), std::inserter(uniqWords));

    return uniqWords.size();
}

Answer 5

You can consider SQLite c++ wrapper