freebcp: "Unicode data is odd byte size for column. Should be even byte size"

Question

This file works fine (UTF-8):

$ cat ok.txt
291054  Ţawī Rifā

This file causes an error (UTF-8):

$ cat bad.txt
291054  Ţawī Rifā‘

Here's the message:

$ freebcp 'DB.dbo.table' in bad.txt ... -c
Starting copy...
Msg 20050, Level 4
Attempt to convert data stopped by syntax error in source field

Msg 4895, Level 16, State 2
Server '...', Line 1
    Unicode data is odd byte size for column 2. Should be even byte size.
Msg 20018, Level 16
General SQL Server error: Check messages from the SQL Server

The only difference is the last character, which is unicode 2018 (left single quotation mark)

Any idea what is causing this error?

The SQL Server uses UTF-16LE (though TDS starts with UCS-2LE and switches over I believe)

The column in question is nvarchar(200)

Here's the packet sent right before the error:

packet.c:741:Sending packet
0000 07 01 00 56 00 00 01 00-81 02 00 00 00 00 00 08 |...V.... ........|
0010 00 38 09 67 00 65 00 6f-00 6e 00 61 00 6d 00 65 |.8.g.e.o .n.a.m.e|
0020 00 69 00 64 00 00 00 00-00 09 00 e7 90 01 09 04 |.i.d.... ...ç....|
0030 d0 00 34 04 6e 00 61 00-6d 00 65 00 d1 ee 70 04 |Ð.4.n.a. m.e.Ñîp.|
0040 00 13 00 62 01 61 00 77-00 2b 01 20 00 52 00 69 |...b.a.w .+. .R.i|
0050 00 66 00 01 01 18      -                        |.f....|

Answer 1

Update: This issue has apparently been fixed in FreeTDS v1.00.16, released 2016-11-04.

---

I can reproduce your issue using FreeTDS v1.00.15. It definitely looks like a bug in freebcp that causes it to fail when the last character of a text field has a Unicode code point of the form U+20xx. (Thanks to @srutzky for correcting my conclusion as to the cause.) As you noted, this works ...

291054  Ţawī Rifā

... and this fails ...

291054  Ţawī Rifā‘

... but I found that this also works:

291054  Ţawī Rifā‘x

So, an ugly workaround would be to run a script against your input file that would append a low-order non-space Unicode character to each text field (e.g., x which is U+0078, as in the last example above), use freebcp to upload the data, and then run an UPDATE statement against the imported rows to strip off the extra character.

Personally, I would be inclined to switch from FreeTDS to Microsoft's SQL Server ODBC Driver for Linux, which includes the bcp and sqlcmd utilities when installed using the instructions described here:

https://gallery.technet.microsoft.com/scriptcenter/SQLCMD-and-BCP-for-Ubuntu-c88a28cc

I just tested it under Xubuntu 16.04, and although I had to tweak the procedure a bit to use libssl.so.1.0.0 instead of libssl.so.0.9.8 (and the same for libcrypto), once I got it installed the bcp utility from Microsoft succeeded where freebcp failed.

If the SQL Server ODBC Driver for Linux will not work on a Mac then another alternative would be to use the Microsoft JDBC Driver 6.0 for SQL Server and a little bit of Java code, like this:

connectionUrl = "jdbc:sqlserver://servername:49242"
        + ";databaseName=myDb"
        + ";integratedSecurity=false";
String myUserid = "sa", myPassword = "whatever";

String dataFileSpec = "C:/Users/Gord/Desktop/bad.txt";
try (
        Connection conn = DriverManager.getConnection(connectionUrl, myUserid, myPassword);
        SQLServerBulkCSVFileRecord fileRecord = new SQLServerBulkCSVFileRecord(dataFileSpec, "UTF-8", "\t", false);
        SQLServerBulkCopy bulkCopy = new SQLServerBulkCopy(conn)) {
    fileRecord.addColumnMetadata(1, "col1", java.sql.Types.NVARCHAR, 50, 0);
    fileRecord.addColumnMetadata(2, "col2", java.sql.Types.NVARCHAR, 50, 0);
    bulkCopy.setDestinationTableName("dbo.freebcptest");
    bulkCopy.writeToServer(fileRecord);
} catch (Exception e) {
    e.printStackTrace(System.err);
}

Answer 2

This issue has nothing to do with UTF-8 given that the data being transmitted, as shown in the transmission packet (bottom of the question) is UTF-16 Little Endian (just as SQL Server would be expecting). And it is perfectly good UTF-16LE, all except for the missing final byte, just like the error message implies.

The problem is most likely a minor bug in freetds that incorrectly applies logic meant to strip off trailing spaces from variable length string fields. There are no trailing spaces, you say? Well, if it hadn't gotten chopped off then it would be a little clearer (but, if it hadn't gotten chopped off there wouldn't be this error). So, let's look at what the packet to see if we can reconstruct it.

The error in the data is probably being overlooked because the packet contains an even number of bytes. But not all fields are double-byte, so it doesn't need to be an even number. If we know what the good data is (prior to the error), then we can find a starting point in the data and move forwards. It is best to start with Ţ as it will hopefully be above the 255 / FF value and hence take 2 bytes. Anything below will have a 00 and many of the characters have that on both sides. While we should be able to assume Little Endian encoding, it is best to know for certain. To that end, we need at least one character that has two non-00 bytes, and bytes that are different (one of the character is 01 for both bytes and that does not help determine ordering). The first character of this string field, Ţ, confirms this as it is Code Point 0162 yet shows up as 62 01 in the packet.

Below are the characters, in the same order as the packet, their UTF-16 LE values, and a link to their full details. The first character's byte sequence of 62 01 gives us our starting point, and so we can ignore the initial 00 13 00 of line 0040 (they have been removed in the copy below for readability). Please note that the "translation" shown to the right does not interpret Unicode, so the 2-byte sequence of 62 01 is displayed as 62 by itself (i.e. lower-case Latin "b") and 01 by itself (i.e. non-printable character; displayed as ".").

0040 xx xx xx 62 01 61 00 77-00 2b 01 20 00 52 00 69 |...b.a.w .+. .R.i|  
0050 00 66 00 01 01 18 ??   -                        |.f....|

• Ţ -- 62 01 -- http://unicode-table.com/en/0162/
• a -- 61 00 -- http://unicode-table.com/en/0061/
• w -- 77 00 -- http://unicode-table.com/en/0077/
• ī -- 2B 01 -- http://unicode-table.com/en/012B/
•   -- 20 00 -- http://unicode-table.com/en/0020/
• R -- 52 00 -- http://unicode-table.com/en/0052/
• i -- 69 00 -- http://unicode-table.com/en/0069/
• f -- 66 00 -- http://unicode-table.com/en/0066/
• ā -- 01 01 -- http://unicode-table.com/en/0101/
• ‘ -- 18 20 -- http://unicode-table.com/en/2018/

As you can see, the last character is really 18 20 (i.e. a byte-swapped 20 18 due to the Little Endian encoding), not 01 18 as it might appear if reading the packet starting at the end. Somehow, the final byte -- hex 20 -- is missing, hence the Unicode data is odd byte size error.

Now, 20 by itself, or followed by 00, is a space. This would explain why @GordThompson was able to get it working by adding an additional character to the end (the final character was no longer trimmable). This could be further proven by ending with another character that is a U+20xx Code Point. For example, if I am correct about this, then ending with ⁄ -- Fraction Slash U+2044 -- would have the same error, while ending with ⅄ -- Turned Sans-Serif Capital Y U+2144 -- even with the ‘ just before it, should work just fine (@GordThompson was kind enough to prove that ending with ⅄ did work, and that ending with ⁄ resulted the same error).

If the input file is null (i.e. 00) terminated, then it could simply be the 20 00 ending sequence that does it, in which case ending with a newline might fix it. This can also be proven by testing a file with two lines: line 1 is the existing row from bad.txt, and line 2 is a line that should work. For example:

291054  Ţawī Rifā‘
999999  test row, yo!

If the two-line file shown directly above works, that proves that it is the combination of a U+20xx Code Point and that Code Point being the last character (of the transmission more than of the file) that exposes the bug. BUT, if this two-line file also gets the error, then it proves that having a U+20xx Code Point as the last character of a string field is the issue (and it would be reasonable to assume that this error would happen even if the string field were not the final field of the row, since the null terminator for the transmission has already been ruled out in this case).

It seems like either this is a bug with freetds / freebcp, or perhaps there is a configuration option to not have it attempt trimming trailing spaces, or maybe a way to get it to see this field as being NCHAR instead of NVARCHAR.

UPDATE

Both @GordThompson and the O.P. (@NeilMcGuigan) have tested and confirmed that this issue exists regardless of where the string field is in the file: in the middle of a row, at the end of the row, on the last row, and not on the last row. Hence it is a general issue.

And in fact, I found the source code and it makes sense that the issue would happen since there is no consideration for multi-byte character sets. I will file an Issue on the GitHub repository. The source for the rtrim function is here:

https://github.com/FreeTDS/freetds/blob/master/src/dblib/bcp.c#L2267

---

Regarding this statement:

The SQL Server uses UTF-16LE (though TDS starts with UCS-2LE and switches over I believe)

From an encoding stand-point, there is really no difference between UCS-2 and UTF-16. The byte sequences are identical. The only difference is in the interpretation of Surrogate Pairs (i.e. Code Points above U+FFFF / 65535). UCS-2 has the Code Points used to construct Surrogate Pairs reserved, but there was no implementation at that time of any Surrogate Pairs. UTF-16 simply added the implementation of the Surrogate Pairs in order to create Supplementary Characters. Hence, SQL Server stores and retrieves UTF-16 LE data without a problem. The only issue is that the built-in functions don't know how to interpret Surrogate Pairs unless the Collation ends with _SC (for Supplementary Characters), and those Collations were introduced in SQL Server 2012.

Answer 3

This might be an encoding issue of the source file.

As you are using non-standard characters, the source file should be unicode by itself probably. Other encodings use a differing count of bytes (one up to three) to encode one single character. E.g. your Unicode 2018 is 0xE2 0x80 0x98 in UTF-8.

Your packet ends with .R.i.f....| while there should be your ā‘. And the error shows Server '...', Line 1.

Try to find out the encoding of your source file (look at big and little endian too) and try to convert your file to a sure unicode format.

Answer 4

This might solve it:

inf your /etc/freetds/freetds.conf

add:

client charset = UTF-8

also found this about the flag use utf-16

use utf-16 Instead of using UCS-2 for database wide character encoding use UTF-16. Newer Windows versions use this encoding instead of UCS-2. This could result in some issues if clients assume that a character is always 2 bytes.