17

I'm not too familiar with the encoding that Microsoft Word uses. If someone where to save a .doc or .docx file from Word, what is the standard encoding that is used?

I'm guessing it's not UTF-8 as the resulting text (pasted in a UTF-8 encoded text file) does not honour certain punctuation (e.g quotes).

For example, an opening Word 'smart quote' when pasted in a UTF-8 text file, results in an ì symbol. If Word does indeed encode in UTF-8, then how does Word attempt to render the actual UTF-8 character?

Edit

After doing a little digging, I can see that a Microsoft Word .docx file is actually a compressed format. Unzipping it results in a number of .xml files to be unpacked.

However, the inability for a UTF-8 encoded text file to honour these 'smart' quotes is still perplexing. Any enlightening information would be helpful.

shennan
  • 10,798
  • 5
  • 44
  • 79
  • 2
    I'm not able to reproduce that behavior. Could be an issue of your text editor? You can check with a hex editor such as HxD and see what actually gets written to the text file. The ['LEFT DOUBLE QUOTATION MARK' (U+201C)](http://www.fileformat.info/info/unicode/char/201C/index.htm) should be encoded using the following *three* bytes: `E2 80 9C` – Dirk Vollmar Jul 13 '16 at 07:47
  • I don't know what text editor you are using to edit that xml file. And my best bet right now, would be that it does a bad job. Are you using Notepad++ ? If not, I would download and install Notepad++. It's very transparent in use of encodings. – bvdb Dec 16 '18 at 23:41
  • as far as I know, notepad++ is a simple text editor that does not have capabilities for reading binary files such as pdf or docx files. It is like asking your English speaker friend Mr Notepad to index or translate Russian text. It's a different alphabet/language altogether (different character encoding) – rags2riches-prog Dec 04 '19 at 09:14

1 Answers1

13

These days a docx file is really a bunch of compressed xml files. One of these files, is the document.xml file, which starts with the following line (i.e. an xml prolog):

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

As you can see, it's an UTF-8 encoding.

EDIT

UTF-8 supports the full set of Unicode characters. Just for the sake of completeness, that does not mean that all UTF-8 characters can actually be used in an xml file. Even a CDATA block has its limitations. But having said all that, storing an ` or an ì isn't a problem.

And more importantly, the file format does not really have anything to do with copy-paste behavior of the application itself.

Nevertheless, here's how word would store an ` and ì symbol.

xml and hex

CORRECTION

A bit confusing, but I just realized that by "smart quote" you probably refer to the mechanism that Word has to represent the curly quotes. In my previous answer I thought you meant "backticks", which is a different thing. - Sorry for the confusion.

Well, anyway, here are the unicodes for these smart quotes:

the UTF smart quotes

Let's put them in a simple UTF-8 encoded text file. The result is not that spectacular:

  • U+2018 is encoded in UTF-8 as E2 80 98
  • U+2019 is encoded in UTF-8 as E2 80 99
  • U+201C is encoded in UTF-8 as E2 80 9C
  • U+201D is encoded in UTF-8 as E2 80 9D

So, I went 1 step further and put them in a word file. I entered a line with regular quotes, and one with smart quotes.

“ this is a test “ 
“ this is another test ”

And then, I saved the thing and looked how it was stored in Word's xml structure. And actually it is exactly stored as expected.

enter image description here

Community
  • 1
  • 1
bvdb
  • 22,839
  • 10
  • 110
  • 123
  • Exactly what I was looking for. I'm working on a Find & Replace PowerShell for DOCX files and the encoding was set to ASCII by default, causing "?" symbols when after storing it back into the XML file. The other Encoding type that seemed to work was OEM, but I wasn't sure if that was right. Duh, it's at the top of the XML file. Thanks. :D – JoePC Jun 21 '19 at 15:28
  • I find that just changing the extension of a `.doc` file to `.zip` doesn't work (this does work for `.docx`). How can I see the XML components of a `.doc` file? – Zach Smith Oct 03 '19 at 07:51
  • 1
    @ZachSmith Even though still sometimes being used today, the `.doc` is actually an outdated format. The `.doc` format has been around since the early '90s, and back then "open formats" weren't a thing yet. So, `.doc` is a binary format and does not use xml internally. It is totally different. Actually,`docx` has been around since 2007, so you would think that most people would have switched by now. But still ... – bvdb Oct 03 '19 at 20:31