C++ getline adding spaces

Question

I have been trying to fix this problem for days and I can't get it. Basically my code is supposed to read a .csv file produced by wmic and save it to a struct. I can read the data and it is being stored, but the data has an extra space after each character. I have tried switching to the Unicode versions of the functions and using wide strings but they only messed up the data even more (they turned a "n" into a "ÿ").

Here is the code that I think is the issue:

system("wmic product get name,version,installdate,vendor /format:csv > product.txt");

std::ifstream infoFile("./program.txt"); // The file wmic wrote in csv format.

if(infoFile.is_open())
{
    std::string line;
    int lineNum = 0;

    while(getline(infoFile, line))
    {
        lineNum++;
        std::cout << "\nLine #" << lineNum << ":" << std::endl;

        Program temp;
        std::istringstream lineStream(line);
        std::string cell;
        int counter = 0;
        int cellNum = 0;

        while(getline(linestream, cell, ','))
        {
            cellNum++;
            std::cout << "\nCell #" << cellNum << ":" << cell << std::endl;

            switch(counter)
            {
            case 0:
                break;
            case 1:
                temp.installDate = cell;
                break;
            case 2:
                temp.name = cell;
                break;
            case 3:
                temp.vendor = cell;
                break;
            case 4:
                temp.version = cell;
                break;
            default:
                std::cout << "GetProductInfo(): Invalid switch value: " << counter << std::endl;
                break;
            }
            counter++;
        }

        information->push_back(temp); // Vector to save all of the programs.
    }

    infoFile.close();
}
else
{
    std::cout << "GetProductInfo(): Failed to open the input file." << std::endl;
    return 1;
}

return 0;
}

Edit: O.K., I am trying to write the BOM (FF FE 0D 00 0A) as it wasn't being written before. I am writing a char array with the hex values but there is a extra 0x0D being added (FF FE 0D 00 0D 0A). It is also saving internal variables with the extra spaces. That might not be an issue as I can modify my code to account for it but that wouldn't be optimal. Any ideas?

Edit2: So I guess I don't need the BOM. My main problem now is just reading the UTF-16LE file and saving the data to a struct without the extra spaces. I need some help doing it the right way as I would like to figure out how to prevent this in the future. Thanks for you help everyone, this bug is critical.

Answer 1

This smelled a lot like a text encoding problem, so I went ahead and tried running the command you provided, and sure enough, the output file is encoded in UCS16LE. (That's 16-bit chars, little-endian.) Try opening the file in a hex editor to see what it actually looks like.

You were on the right path when trying to use wide strings, but dealing with Unicode can be tricky. The next few paragraphs will give you some tips on how to deal with this the hard way, but if you need a quick and easy solution, jump to the end.

There's two thing to be careful of. First, make sure you're also using the wide streams, like wcout. It's worth casting each character to an int to double-check that there isn't a problem with the output formatting.

Second, the format of wcout, wstring, etc, is not standard. On some compilers it's 2 bytes per char, and on others it's 4. You can usually change this in your compiler settings. C++11 also provides std::u16string and std::u32string, which are more explicit about their size.

Reading Unicode text can unfortunately be quite a bit of a hassle with the C++ library, because even if you have the right string size, you need to deal with BOMs and endian formats, not to mention canonicalization.

There's libraries to help with this, but the simplest solution might just be to open the txt file in Notepad, choose Save As, then choose an encoding you're more comfortable with, like ANSI.

Edit: If you're not happy with the quick and dirty solution, and you don't want to use a better Unicode library, you can do this with the standard library, but only if you're using a compiler that supports C++11, such as Visual Studio 2012.

C++11 added some codecvt facets to handle converting between different Unicode file types. This should suit your purpose, but the underlying design of this part of the library was designed in the days or yore, and can be rather difficult to understand. Hold on to your pants.

Below the line where you open your ifstream, add this code:

infoFile.imbue(std::locale(infoFile.getloc(), new std::codecvt_utf16<char, 0x10FFFF, std::consume_header>));

I know that looks a bit scary. What it's doing is making a "locale" from a copy of the existing locale, then adding a "facet" to the locale which handles the format conversion.

"Locales" handle a whole bunch of stuff, mostly related to localization (such as how to punctuate currency, eg "100.00" vs "100,00"). Each of the rules in the locale is called a facet. In the C++ standard library, file encoding is treated as one of these facets.

(Background: In retrospect, it probably wasn't a very wise idea to mix file encoding up with localization, but at the time this part of the library was designed, file encoding was typically dictated by the language of the program, so that's how we got into this situation.)

So the locale constructor above is taking a copy of the default locale created by the file stream as its first parameter, and the second parameter is the new facet to use.

codecvt_utf16 is a facet for converting to and from utf-16. The first parameter is the "wide" type, which is to say, the type used by the program, rather than the type used in the byte stream. I specified char here, and that works with Visual Studio, but it's not actually valid according to the standard. I'll get to that later.

The second parameter is the maximum Unicode value you want to accept without throwing an error, and for the foreseeable future, 0x10FFFF represents the largest Unicode character.

The final parameter is a bitmask that changes the behaviour of the facet. I thought std::consume_header would be particularly useful for you, since wmic outputs a BOM (at least on my machine). This will consume that BOM, and chose whether to treat it as a little- or big-endian stream depending on what it gets.

You'll also notice that I'm creating the facet on the stack with new, but I'm not calling delete anywhere. This is not a very safe way to design a library in modern C++, but like I said, locales are a rather old part of the library.

Rest assured that you don't need to delete this facet. This isn't really documented very well (since locales are so rarely used in practice), but a default-constructed facet will be automatically deleted by the locale it's attached to.

Now, remember how I said it's not valid to use char as the wide type? The standard says you have to use whcar_t, char16_t or char32_t, and if you want to support non-ASCII characters, you'll definitely want to do this. The easiest way to make this valid would be to use wchar_t, change ifstream, string, cout, and istringstream to wifstream, wstring, wcout, and wistringstream, then make sure your strings/char constants have an L in front of them, like so:

std::wcout << L"\nLine #" << lineNum << L":" << line << std::endl;

Those are all the changes you need in order to use wide strings. However, also beware that the Windows console cannot handle non-ANSI characters, so if you try to output such a character (when I ran the code I hit a ™ character), the wcout stream will be invalidated and stop outputting anything. If you're outputting to a file, this shouldn't be a problem.

You can probably tell that I'm not particularly thrilled about this part of the standard library. In practice, most people who want to use Unicode will use a different library (like the ones I mentioned in the comments), or roll their own encoders/decoders.

Answer 2

If your data doesn't have any spaces you need you can use my example:

std::string s = "test, delim, ";
std::string delims = ", ";

size_t pos = 0;
std::string token;

while((pos=s.find(delimiter))!=std::string::npos)) 
{ token = s.substr(0,pos);
  std::cout<<token<<std::endl;
  s.erase(0, pos + delimiter.length());
}
std::cout<<s<<std::endl //last word

Alternatively, you can use strtok from cstring library. You can also check my question, it is pretty the same: strtok() analogue in C++

Answer 3

If the data has an extra space after every character, I suppose it means it also has an extra space after a regular space.

Thus you can safely erase every space (every char, actually) which does not have another space right before it. This assumes you do not have two white spaces in a row in the original data, but if you do, you just need one extra flag to take care of that.

So your code could become something like this:

while(getline(infoFile, line))
{
    int lsize = line.size(), at = 1;
    for(int i = 1; i < lsize; ++i)
        if(line[i-1] == ' ') line[at++] = line[i];
        // if there is no space behind it, skip it, it is a broken space itself!
    line.resize(at);

    lineNum++;
    // std::cout << "\nLine #"...

I realize this is not completelly ideal as you're not actually stopping the core problem from happening, but considering you've been trying for days, this at least effectively mitigates the problem by fixing it after it has happened.

Check the live demo.

Answer 4

In my case, I solved the problem by changing the encoding to utf8 using Notepad++.

1. From Encoding menu:

2. Click on utf8 to do the change, and save: