3

I'm using dulwich (a Python library) to access a git repository. When I use get_object to retrieve a commit, it has a number of attributes. One of those is author. When I retrieve this attribute, I get bytes and so the attribute is an an unknown encoding.

Is there an encoding I can safely assume? Does git translate all the metadata to utf-8 before storing it? If it doesn't, how do I know which encoding to use to decode the bytes?

Omnifarious
  • 54,333
  • 19
  • 131
  • 194
  • Presumably the [current `i18n.commitEncoding` value](https://git-scm.com/docs/git-commit#_discussion) holds the correct value (default is UTF-8). – Martijn Pieters Jun 12 '18 at 21:15
  • @MartijnPieters - Is the value of this config option propagated when you clone a repository, or do you have to remember to set it everywhere when you use a repository that has chosen an unusual encoding? If it's the latter, I'm going to start encoding everything I send to github as UTF-16 and watch the hilarity ensue. – Omnifarious Jun 12 '18 at 21:24
  • I see that the commit object actually has a `encoding` header, copied from the `.git/config` setting. This is part of the `objects` data, so no hilarity, sorry. – Martijn Pieters Jun 12 '18 at 21:32
  • @MartijnPieters - I was noticing that, but it was `None` so I thought maybe it was meaningless. I guess maybe that means I can assume the default utf-8 for that particular commit. – Omnifarious Jun 12 '18 at 21:35
  • @MartijnPieters - Anyway, it looks like you've probably answered my question. So maybe you should format your comment as an answer and I can accept it. :-) – Omnifarious Jun 12 '18 at 21:41
  • 1
    I'm doing a few additional tests; I've not been able to actually get my commit messages recoded to UTF-16 just yet.. – Martijn Pieters Jun 12 '18 at 21:42
  • @MartijnPieters - How about EBCDIC? ;-) – Omnifarious Jun 12 '18 at 21:43

1 Answers1

2

Metadata is supposed to be encoded with the value set by the i18n.commitEncoding config value; whenever a commit is created the current value is copied into the 'encoding' header on the object, if set; the default value is UTF-8.

That encoding value is available on Dulwitch objects as the '.encoding' attribute; if it is None then i18n.commitEncoding was not explicitly set and you can use UTF-8 as the default.

However! The actual data stored simply follows whatever bytes where handed to git and no re-coding takes place. The configuration value is purely informational. So you need to take into account that an incorrect codec was used, if you are going to use object.encoding or 'utf8' as the codec, use a sensible error handler or fallback strategy.

Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
  • Thank you for doing the research necessary to figure this out. It's really disappointing to learn that no recoding is done. I should still try uploading an EBCDIC encoded repo that's honest with its encoding flag and see what GitHub does with it. ;-) – Omnifarious Jun 12 '18 at 21:52