how to properly apply tolower() on German capital letters Ä, Ö, Ü, ẞ in C++

Question

I am a bit confused since I have opened a question, I would like to be a bit more specific here.

I have numerous files that contain German letters mostly in iso-8859-15 or UTF-8 encoding. In order to process them it is mandatory to transform all letters to lowercase.

For example I have a file (encoded in iso-8859-15 ) that contains:

Dr. Rose in M. Das sogen. Baptisterium zu Winland, eins der im Art. "Baukunst" (S. 496) erwähnten Rundgebäude in Grönland, soll nach Palfreys "History of New England" eine von dem Gouverneur Arnold um 1670 erbaute Windmühle sein. Vgl. Gust. Storm in den "Jahrbüchern der königlichen Gesellschaft für nordische Altertumskunde in Kopenhagen" 1887, S. 296.

Ää Öö Üü ẞß Örebro

Text Ää Öö Üü ẞß Örebro should become: ää öö üü ßß örebro.

However, tolower() does not seem to apply on capital letters such as Ä, Ö, Ü, ẞ eventhough i tried forcing locale as mentioned in this SO post

Here is the same code as posted in my other question:

std::vector<std::string> tokens;
std::string filename = "10223-8.txt";
//std::string filename = "test-UTF8.txt";
std::ifstream inFile;

//std::setlocale(LC_ALL, "en_US.iso88591");
//std::setlocale(LC_ALL, "de_DE.iso88591");
//std::setlocale(LC_ALL, "en_US.iso88591");
//std::locale::global(std::locale(""));

inFile.open(filename);
if (!inFile) { std::cerr << "Failed to open file" << std::endl; exit(1); }

std::string s = "";
std::string line;
while( (inFile.good()) && std::getline(inFile, line) ) {
    s.append(line + "\n");
}
inFile.close();

std::cout << s << std::endl;

//std::setlocale(LC_ALL, "de_DE.iso88591");
for (unsigned int i = 0; i < s.length(); ++i) {
    if (std::ispunct(s[i]) || std::isdigit(s[i]))
            s[i] = ' ';
    if (std::isupper(s[i]))
            s[i] = std::tolower(s[i]);
            //s[i] = std::tolower(s[i]);
            //s[i] = std::tolower(s[i], std::locale("de_DE.utf8"))
}

std::cout << s << std::endl;

//tokenize string
std::istringstream iss(s);
tokens.clear();
tokens = {std::istream_iterator<std::string>{iss}, std::istream_iterator<std::string>{}};

//PROCESS TOKENS...

Its really frustrating and there are not many paradigms regarding the usage of <locale>.

So, apart from the main problem with my code, here are some questions:

1. Do I have to apply some sort of custom locale in other functions too (isupper(), ispunct()...)?
2. Do I need de_DE locale enabled or installed in my linux env to process string's chars correctly?
3. Is it safe to process, in the same way, text as std::string that is extracted from files with different encoding (iso-8859-15 or UTF-8)?

EDIT: Konrad Rudolph answer works fine only for UTF-8 files. It does not work for iso-8859-15 which translates to the initial problem posted here: How to apply functions on text files with different encoding in c++

Answer 1

Use std::ctype::tolower, not std::tolower:

#include <iostream>
#include <locale>

int main() {
    std::locale::global(std::locale("de_DE.UTF-8"));
    std::wcout.imbue(std::locale());
    auto& f = std::use_facet<std::ctype<wchar_t>>(std::locale());
    std::wstring str = L"Ää Öö Üü ẞß Örebro";
    f.tolower(&str[0], &str[0] + str.size());
    std::wcout << "'" << str << "'\n";
}

Rather than setting a global locale, you could also create a local locale (heh):

std::locale loc("de_DE.UTF-8");
std::wcout.imbue(loc);
auto& f = std::use_facet<std::ctype<wchar_t>>(loc);

This compiles and “works”. On my system, it correctly converts the umlauts but it fails to handle the capital-ß (not surprisingly, to be honest).

Furthermore, note the limitations of this function: it can only perform 1-to-1 character conversions. In previous versions of the Unicode standard, the correct uppercase transformation of “ß” was “SS”. std::ctype::toupper explicitly never supported this.