4

We read in the C18 standard:

5.1.1.2 Translation phases

The precedence among the syntax rules of translation is specified by the following phases.

  1. Physical source file multibyte characters are mapped, in an implementation-defined manner, to the source character set (introducing new-line characters for end-of-line indicators) if necessary.

Meaning that the source file character set is decoded and mapped to the source character set.

But then you can read:

5.2.1 Character sets

Two sets of characters and their associated collating sequences shall be defined: the set in which source files are written (the source character set), and the set interpreted in the execution environment (the execution character set).

Meaning that the source file character set is the source character set.

So the question is: which one did I understand wrong, or which one is actually wrong?

EDIT: Actually I was wrong. See my answer below.

Pep
  • 625
  • 4
  • 19
  • 1
    The standard doesn't actually use the term "source file character set". It seems to me that the physical format of the source file isn't defined by the standard at all, but by the implementation, and therefore it doesn't enter into 5.2.1. – Nate Eldredge Aug 23 '20 at 15:07
  • But 5.2.1 seems to say that the set in which the source files are written becomes the source character set, which to me doesn't make any sense. – Pep Aug 23 '20 at 15:13
  • I would understand that informally: the programmer must be able to write the program in the source character set, in that they must be able to somehow use characters like `+`, `*`, etc. But what the implementation actually stores in the file system may be different, and the translation back into the source character set is what 5.1.1.2 refers to. – Nate Eldredge Aug 23 '20 at 15:17
  • Actually, the characters you point out are members of the basic source character set, so you can always use them. The problem is that you don't know what the source character set of the implementation is, unless 5.2.1 is right, in which case, the programmer decides what is the source character set. And that makes no sense at all, as one of the restrictions is that basic source character set members should be coded using a single byte; what if you decide to use a UTF-16 source file then? That's why I think 5.2.1 is wrong or I might be understanding it wrong. – Pep Aug 23 '20 at 15:39
  • I don't see how you conclude that 5.2.1 allows the programmer to decide, although that might be true in practice. But nothing in the standard requires that a character have a unique encoding, and 5.1.1.2 allows the translation of multibyte characters "in an implementation-defined manner" which is certainly flexible enough to allow for UTF-16. – rici Aug 24 '20 at 02:04

3 Answers3

1

Meaning that the source file character set is decoded and mapped to the source character set.

No, it does not mean that. My take is that the source is already assumed to be written in the source character set - how exactly would it make sense to "map the source character set to the source character set"? Either they are part of the set or they aren't. If you pick the wrong encoding for your source code, it will simply be rejected before the preprocessing even starts.

Translation phase 1 does two things not quite related to this at all:

  • Resolves trigraphs, which are standardized multibyte sequences.

  • Map multibyte characters into the source character set (defined in 5.2.1).

    The source character set consists of the basic character set which is essentially the Latin alphabet plus various common symbols (5.2.1/3), and an extended character set, which is locale- and implemention-specific.

    The definition of multibyte characters is found at 5.2.1.2:

    The source character set may contain multibyte characters, used to represent members of the extended character set. The execution character set may also contain multibyte characters, which need not have the same encoding as for the source character set.

    Meaning various locale-specific oddball special cases, such as locale-specific trigraphs.

All of this multibyte madness goes back to the first standardization in 1990 - according to anecdotes from those who were part of that committee, this was because members from various European countries weren't able to use various symbols on their national keyboards.

(I'm not sure how widespread the AltGr key on such keyboards was at the time. It remains a key subject to some serious button mashing when writing C on non-English keyboards anyway, to get access to {}[] symbols etc.)

Lundin
  • 195,001
  • 40
  • 254
  • 396
  • It's not "map the source character set to the source character set". It's "map the source **file** character set to the source character set". – Pep Aug 24 '20 at 09:00
  • @PepeDeTicher That's the same thing. The compiler will assume one character set and if you give it a file in a different one, it will not understand it. – Lundin Aug 24 '20 at 09:03
  • Not so. High-range characters in string constants go through. – Joshua Aug 24 '20 at 14:09
1

Well, after all it seems I was wrong. After contacting David Keaton, from the WG14 group (they are in charge of the C standard), I got this clarifying reply:

There is a subtle distinction. The source character set is the character set in which source files are written. However, the source character set is just the list of characters available, which does not say anything about the encoding.

Phase 1 maps the multibyte encoding of the source character set onto the abstract source characters themselves.

In other words, a character that looks like this:

<byte 1><byte 2>

is mapped to this:

<character 1>

The first is an encoding that represents a character in the source character set in which the program was written. The second is the abstract character in the source character set.

Pep
  • 625
  • 4
  • 19
-2

You have encountered cross compiling, where a program is compiled on one architecture and executed on another architecture and these architectures have different character sets.

5.1.1.2 is active early in read, where the input file is converted into the compiler's single character set, which clearly must contain all of the characters required by a C program.

However when cross compiling, the execution character set may be different. 5.2.1 is allowing for this possibility. When the compiler emits code, it must translate all character and string constants to the target platform's character set. On modern platforms, this is a no-op, but on some ancient platforms it wasn't.

Joshua
  • 40,822
  • 8
  • 72
  • 132
  • Not cross compiling, actually. Just two clauses from the standard that seem to contradict each other. To me, 5.2.1 makes no sense, but then, why is it written as it is? – Pep Aug 23 '20 at 15:41
  • @PepeDeTicher: 5.12.1 is for cross compiling. – Joshua Aug 23 '20 at 15:45
  • There's not a single paragraph in the standard talking about cross compilig, as this is something with nothing to do with the language specification. – Pep Aug 23 '20 at 15:50
  • @pepe: That's true, but the standard was written at a time when EBCDIC was still in common use (as opposed to now, when it's in uncommon use). It wasn't necessary for the standard to explain that; it only needed to contemplate the possibility that a program composed in EBCDIC could be compiled into something which would run in ASCII (or vice versa). I agree that the language is slightly sloppy, but it's not at all easy to get it right, which is why there are three different definitions of "character" in the definitions section of the standard. – rici Aug 24 '20 at 02:00
  • @rici: Believe it or not I had PETSCEII in mind not EBCDIC but same difference I guess. – Joshua Aug 24 '20 at 02:03
  • This has absolutely nothing to do with cross compiling. The question is about _the source character set_ in phase 1, not to be confused with _the execution character set_ (phase 5 of the translation phases). – Lundin Aug 24 '20 at 08:59