fgetws can't read non-English characters on Linux

Question

I have a basic C program that reads some lines from a text file containing hundreds of lines in its working directory. Here is the code:

#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#include <ctype.h>
#include <string.h>
#include <locale.h>
#include <wchar.h>
#include <wctype.h>
#include <unistd.h>

int main(int argc, const char * argv[]) {
    srand((unsigned)time(0));
    char *nameFileName = "MaleNames.txt";
    wchar_t line[100];
    wchar_t **nameLines = malloc(sizeof(wchar_t*) * 2000);
    int numNameLines = 0;
    FILE *nameFile = fopen(nameFileName, "r");
    while (fgetws(line, 100, nameFile) != NULL) {
        nameLines[numNameLines] = malloc(sizeof(wchar_t) * 100);
        wcsncpy(nameLines[numNameLines], line, 100);
        numNameLines++;
    }
    fclose(nameFile);

    wchar_t *name = nameLines[rand() % numNameLines];
    name[wcslen(name) - 1] = '\0';
    wprintf(L"%ls", name);

    int i;
    for (i = 0; i < numNameLines; i++) {
        free(nameLines[i]);
    }
    free(nameLines);
    return 0;
}

It basically reads my text file (defined as a macro, it exists at the working directory) line by line. Rest is irrelevant. It runs perfect and as expected on my Mac (with llvm/Xcode). When I try to compile (nothing fancy, again, gcc main.c) and run it on a Linux server, it either:

• Exists with error code 2 (meaning no lines are read).
• Reads only first 3 lines from my file with hundreds of lines.

What causes this indeterministic (and incorrect) behavior? I've tried commenting out the first line (random seed) and compile again, it always exits with return code 2.

What is the relation between the random methods and reading a file, and why I'm getting this behavior?

UPDATE: I've fixed malloc to sizeof(wchar_t) * 100 from sizeof(wchar_t) * 50. It didn't change anything. My lines are about 15 characters at most, and there are much less than 2000 lines (it is guaranteed).

UPDATE 2:

• I've compiled with -Wall, no issues.
• I've compiled with -Werror, no issues.
• I've run valgrind didn't find any leaks too.
• I've debugged with gdb, it just doesn't enter the while loop (fgetws call returns 0).

UPDATE 3: I'm getting a floating point exception on Linux, as numNameLines is zero.

UPDATE 4: I verify that I have read permissions on MaleNames.txt.

UPDATE 5: I've found that accented, non-English characters (e.g. Â) cause problems while reading lines. fgetws halts on them. I've tried setting locale (both setlocale(LC_ALL, "en.UTF-8"); and setlocale(LC_ALL, "tr.UTF-8"); separately) but didn't work.

Answer 1

fgetws() is attempting to read up to 100 wide characters. The malloc() call in the loop allocates 50 wide characters.

The wcscpy() call copies all the wide characters read. If more than 50 wide characters have been read (including the terminating nul) then wcscpy() will overrun the allocated buffer. That results in undefined behaviour.

Instead of multiplying by 50 in the loop, multiply by 100. Or, better yet, compute the length of string read and use that.

Independently of the above, your code will also overrun a buffer if the file contains more than 2000 lines. Your loop needs to check for that.

A number of the functions in your code can fail, and will return a value to indicate that. Your code is not checking for any such failures.

Your code running under OS X is happenstance. The behaviour is undefined, which means there is potential to fail on any host system, when built with any compiler. Appearing to run correctly on one system, and failing on another system, is actually a valid set of responses to undefined behaviour.

Answer 2

Found the solution. It was all about the locale, from the beginning. After experimenting and hours of research, I've stumbled upon this: http://cboard.cprogramming.com/c-programming/142780-arrays-accented-characters.html#post1066035

#include < locale.h >

setlocale(LC_ALL, "");

Setting locale to empty string solved my problem instantly.