utf8 aware strncpy

Question

I find it hard to believe I'm the first person to run into this problem but searched for quite some time and didn't find a solution to this.

I'd like to use strncpy but have it be UTF8 aware so it doesn't partially write a utf8 code-point into the destination string.

Otherwise you can never be sure that the resulting string is valid UTF8, even if you know the source is (when the source string is larger than the max length).

Validating the resulting string can work but if this is to be called a lot it would be better to have a strncpy function that checks for it.

glib has g_utf8_strncpy but this copies a certain number of unicode chars, whereas Im looking for a copy function that limits by the byte length.

To be clear, by "utf8 aware", I mean that it should not exceed the limit of the destination buffer and it must never copy only part of a utf-8 code-point. (Given valid utf-8 input must never result in having invalid utf-8 output).

---

Note:

Some replies have pointed out that strncpy nulls all bytes and that it wont ensure zero termination, in retrospect I should have asked for a utf8 aware strlcpy, however at the time I didn't know of the existence of this function.

Answer 1

I've tested this on many sample UTF8 strings with multi-byte characters. If the source is too long, it does a reverse search of it (starts at the null terminator) and works backward to find the last full UTF8 character which can fit into the destination buffer. It always ensures the destination is null terminated.

char* utf8cpy(char* dst, const char* src, size_t sizeDest )
{
    if( sizeDest ){
        size_t sizeSrc = strlen(src); // number of bytes not including null
        while( sizeSrc >= sizeDest ){

            const char* lastByte = src + sizeSrc; // Initially, pointing to the null terminator.
            while( lastByte-- > src )
                if((*lastByte & 0xC0) != 0x80) // Found the initial byte of the (potentially) multi-byte character (or found null).
                    break;

            sizeSrc = lastByte - src;
        }
        memcpy(dst, src, sizeSrc);
        dst[sizeSrc] = '\0';
    }
    return dst;
}

Answer 2

I'm not sure what you mean by UTF-8 aware; strncpy copies bytes, not characters, and the size of the buffer is given in bytes as well. If what you mean is that it will only copy complete UTF-8 characters, stopping, for example, if there isn't room for the next character, I'm not aware of such a function, but it shouldn't be too hard to write:

int
utf8Size( char ch )
{
    static int const sizeTable[] =
    {
        //  ...
    };
    return sizeTable( static_cast<unsigned char>( ch ) )
}

char*
stru8ncpy( char* dest, char* source, int n )
{
    while ( *source != '\0' && utf8Size( *source ) < n ) {
        n -= utf8Size( *source );
        switch ( utf8Size( ch ) ) {
        case 6:
            *dest ++ = *source ++;
        case 5:
            *dest ++ = *source ++;
        case 4:
            *dest ++ = *source ++;
        case 3:
            *dest ++ = *source ++;
        case 2:
            *dest ++ = *source ++;
        case 1:
            *dest ++ = *source ++;
            break;
        default:
            throw IllegalUTF8();
        }
    }
    *dest = '\0';
    return dest;
}

(The contents of the table in utf8Size are a bit painful to generate, but this is a function you'll be using a lot if you're dealing with UTF-8, and you only have to do it once.)

Answer 3

To reply to own question, heres the C function I ended up with (Not using C++ for this project):

Notes: - Realize this is not a clone of strncpy for utf8, its more like strlcpy from openbsd. - utf8_skip_data copied from glib's gutf8.c - It doesn't validate the utf8 - which is what I intended.

Hope this is useful to others and interested in feedback, but please no pedantic zealot's about NULL termination behavior unless its an actual bug, or misleading/incorrect behavior.

Thanks to James Kanze who provided the basis for this, but was incomplete and C++ (I need a C version).

static const size_t utf8_skip_data[256] = {
    1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
    1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
    1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
    1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
    1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
    1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
    2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,
    3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,4,4,4,4,4,4,4,4,5,5,5,5,6,6,1,1
};

char *strlcpy_utf8(char *dst, const char *src, size_t maxncpy)
{
    char *dst_r = dst;
    size_t utf8_size;

    if (maxncpy > 0) {
        while (*src != '\0' && (utf8_size = utf8_skip_data[*((unsigned char *)src)]) < maxncpy) {
            maxncpy -= utf8_size;
            switch (utf8_size) {
                case 6: *dst ++ = *src ++;
                case 5: *dst ++ = *src ++;
                case 4: *dst ++ = *src ++;
                case 3: *dst ++ = *src ++;
                case 2: *dst ++ = *src ++;
                case 1: *dst ++ = *src ++;
            }
        }
        *dst= '\0';
    }
    return dst_r;
}

Answer 4

strncpy() is a terrible function:

1. If there is insufficient space, the resulting string will not be nul terminated.
2. If there is enough space, the remainder is filled with NULs. This can be painful if the target string is very big.

Even if the characters stay in the ASCII range (0x7f and below), the resulting string will not be what you want. In the UTF-8 case it might be not nul-terminated and end in an invalid UTF-8 sequence.

Best advice is to avoid strncpy().

EDIT: ad 1):

#include <stdio.h>
#include <string.h>

int main (void)
{
char buff [4];

strncpy (buff, "hello world!\n", sizeof buff );
printf("%s\n", buff );

return 0;
}

Agreed, the buffer will not be overrun. But the result is still unwanted. strncpy() solves only part of the problem. It is misleading and unwanted.

UPDATE(2012-10-31): Since this is a nasty problem, I decided to hack my own version, mimicking the ugly strncpy() behavior. The return value is the number of characters copied, though..

#include <stdio.h>
#include <string.h>

size_t utf8ncpy(char *dst, char *src, size_t todo);
static int cnt_utf8(unsigned ch, size_t len);

static int cnt_utf8(unsigned ch, size_t len)
{
if (!len) return 0;

if ((ch & 0x80) == 0x00) return 1;
else if ((ch & 0xe0) == 0xc0) return 2;
else if ((ch & 0xf0) == 0xe0) return 3;
else if ((ch & 0xf8) == 0xf0) return 4;
else if ((ch & 0xfc) == 0xf8) return 5;
else if ((ch & 0xfe) == 0xfc) return 6;
else return -1; /* Default (Not in the spec) */
}

size_t utf8ncpy(char *dst, char *src, size_t todo)
{
size_t done, idx, chunk, srclen;

srclen = strlen(src);
for(done=idx=0; idx < srclen; idx+=chunk) {
        int ret;
        for (chunk=0; done+chunk < todo; chunk++) {
                ret = cnt_utf8( src[idx+chunk], srclen - (idx+chunk) );
                if (ret ==1) continue;  /* Normal character: collect it into chunk */
                if (ret < 0) continue;  /* Bad stuff: treat as normal char */
                if (ret ==0) break;     /* EOF */
                if (!chunk) chunk = ret;/* an UTF8 multibyte character */
                else ret = 1;           /* we allready collected a number (chunk) of normal characters */
                break;
                }
        if (ret > 1 && done+chunk > todo) break;
        if (done+chunk > todo) chunk = todo - done;
        if (!chunk) break;
        memcpy( dst+done, src+idx, chunk);
        done += chunk;
        if (ret < 1) break;
        }
        /* This is part of the dreaded strncpy() behavior:
        ** pad the destination string with NULs
        ** upto its intended size
        */
if (done < todo) memset(dst+done, 0, todo-done);
return done;
}

int main(void)
{
char *string = "Hell\xc3\xb6 \xf1\x82\x82\x82, world\xc2\xa1!";
char buffer[30];
unsigned result, len;

for (len = sizeof buffer-1; len < sizeof buffer; len -=3) {
        result = utf8ncpy(buffer, string, len);
        /* remove the following line to get the REAL strncpy() behaviour */
        buffer[result] = 0;
        printf("Chop @%u\n", len );
        printf("Org:[%s]\n", string );
        printf("Res:%u\n", result );
        printf("New:[%s]\n", buffer );
        }

return 0;
}

Answer 5

Here is a C++ solution:

u8string.h:

#ifndef U8STRING_H
#define U8STRING_H 1
#include <stddef.h>
#ifdef __cplusplus
extern "C" {
#endif

/**
 * Copies the first few characters of the UTF-8-encoded string pointed to by
 * \p src into \p dest_buf, as many UTF-8-encoded characters as can be written in
 * <code>dest_buf_len - 1</code> bytes or until the NUL terminator of the string
 * pointed to by \p str is reached.
 *
 * The string of bytes that are written into \p dest_buf is NUL terminated
 * if \p dest_buf_len is greater than 0.
 *
 * \returns \p dest_buf
 */
char * u8slbcpy(char *dest_buf, const char *src, size_t dest_buf_len);

#ifdef __cplusplus
}
#endif
#endif

u8slbcpy.cpp:

#include "u8string.h"

#include <cstring>
#include <utf8.h>

char * u8slbcpy(char *dest_buf, const char *src, size_t dest_buf_len)
{
    if (dest_buf_len <= 0) {
        return dest_buf;
    } else if (dest_buf_len == 1) {
        dest_buf[0] = '\0';
        return dest_buf;
    }

    size_t num_bytes_remaining = dest_buf_len - 1;
    utf8::unchecked::iterator<const char *> it(src);
    const char * prev_base = src;
    while (*it++ != '\0') {
        const char *base = it.base();
        ptrdiff_t diff = (base - prev_base);
        if (num_bytes_remaining < diff) {
            break;
        }
        num_bytes_remaining -= diff;
        prev_base = base;
    }

    size_t n = dest_buf_len - 1 - num_bytes_remaining;
    std::memmove(dest_buf, src, n);
    dest_buf[n] = '\0';

    return dest_buf;
}

The function u8slbcpy() has a C interface, but it is implemented in C++. My implementation uses the header-only UTF8-CPP library.

I think that this is pretty much what you are looking for, but note that there is still the problem that one or more combining characters might not be copied if the combining characters apply to the n^th character (itself not a combining character) and the destination buffer is just large enough to store the UTF-8 encoding of characters 1 through n, but not the combining characters of character n. In this case, the bytes representing characters 1 through n are written, but none of the combining characters of n are. In effect, you could say that the n^th character is partially written.

Answer 6

To comment on the above answer "strncpy() is a terrible function:". I hate to even comment on such blanket statements at the expense of creating yet another internet programming jihad, but will anyhow since statements like this are misleading to those that might come here to look for answers.

Okay maybe C string functions are "old school". Maybe all strings in C/C++ should be in some kind of smart containers, etc., maybe one should use C++ instead of C (when you have a choice), these are more of a preference and an argument for other topics.

I came here looking for a UTF-8 strncpy() my self. Not that I couldn't make one (the encoding is IMHO simple and elegant) but wanted to see how others made theirs and perhaps find a optimized in ASM one.

To the "gods gift" of the programming world people, put your hubris aside for a moment and look at some facts.

There is nothing wrong with "strncpy()", or any other of the similar functions with the same side effects and issues like "_snprintf()", etc.

I say: "strncpy() is not terrible", but rather "terrible programmers use it terribly".

What is "terrible" is not knowing the rules. Furthermore on the whole subject because of security (like buffer overrun) and program stability implications, there wouldn't be a need for example Microsoft to add to it's CRT lib "Safe String Functions" if the rules were just followed.

The main ones:

1. "sizeof()" returns the length of a static string w/terminator.
2. "strlen()" returns the length of string w/o terminator.
3. Most if no all "n" functions just clamp to 'n' with out adding a terminator.
4. There is implicit ambiguity on what "buffer size" is in functions that require and input buffer size. I.E. The "(char *pszBuffer, int iBufferSize)" types. Safer to assume the worst and pass a size one less then the actual buffer size, and adding a terminator at the end to be sure.
5. For string inputs, buffers, etc., set and use a reasonable size limit based on expected average and maximum. To hopefully avoid input truncation, and to eliminate buffer overruns period.

This is how I personally handle such things, and other rules that are just to be known and practiced.

A handy macro for static string size:

// Size of a string with out terminator
#define SIZESTR(x) (sizeof(x) - 1)

When declaring local/stack string buffers:

A) The size for example limited to 1023+1 for terminator to allow for strings up to 1023 chars in length.

B) I'm initializing the the string to zero in length, plus terminating at the very end to cover a possible 'n' truncation.

char szBuffer[1024]; szBuffer[0] = szBuffer[SIZESTR(szBuffer)] = 0;

Alternately one could do just: char szBuffer[1024] = {0}; of course but then there is some performance implication for a compiler generated "memset() like call to zero the whole buffer. It makes things cleaner for debugging though, and I prefer this style for static (vs local/stack) strings buffers.

Now a "strncpy()" following the rules:

char szBuffer[1024]; szBuffer[0] = szBuffer[SIZESTR(szBuffer)] = 0; 
strncpy(szBuffer, pszSomeInput, SIZESTR(szBuffer));

There are other "rules" and issues of course, but these are the main ones that come to mind. You just got to know how the lib functions work and to use safe practices like this.

Finally in my project I use ICU anyhow so I decided to go with it and use the macros in "utf8.h" to make my own "strncpy()".