Deprecate implicit conversions between char8_t and char16_t, char32_t, or wchar_t

Contents

1. Revision history

1.1. Changes since R1

R0 of the paper was seen by SG16, with the following poll results:

P3695R1: Recommend deprecating conversions between char and the charN_t types.

• Attendees: 10
• No objection to unanimous dissent.

P3695R1: Recommend deprecating conversions between char8_t and wchar_t.

• Attendees: 10
• No objection to unanimous consent.

P3695R1: Recommend deprecating conversions between char16_t and char32_t.

• Attendees: 10

• SF	F	N	A	SA
  0	0	3	7	0

• Consensus against.

Consequently, the following changes were made:

• also deprecated conversion between char8_t and wchar_t; see §3.3. What about char and wchar_t?
• changed title and abstract to reflect this new direction
• rewrote §3.2. What about char16_t and char32_t?
• expanded note on tautology warnings in §2.1. It's not hypothetical. This really happens.
• added §3.6. Why not make these conversions narrowing?
• restructured §6. Wording and added editorial notes

1.2. Changes since R0

• limited deprecation to conversions involving char8_t; see §3.2. What about char16_t and char32_t?
• rebased §6. Wording on [N5014]

2. Introduction

Implicit conversions between char8_t and char32_t invite bugs:

Conversions "the other way" (e.g. char32_t → char8_t) are obviously bug-prone too because information is lost, but such bugs can already be caught by all major compilers' warnings, and they are problematic for the same reason as int → short, not because of anything specific to character types. The listed bugs are interesting precisely because no information is lost.

2.1. It's not hypothetical. This really happens.

These kinds of bugs are not far-fetched hypotheticals either; I have written such bugs myself, and have had them contributed to my syntax highlighter [µlight], which makes extensive use of char8_t and char32_t. Very early in development, I have realized how dangerous these implicit conversions are, so most functions in the style of is_umlaut have a deleted overload:

2.2. The underlying problem

The underlying problem is that char8_t == char32_t is Car == Banana. In general, it is meaningless to compare code units with different encodings.

To be fair, Unicode character types aren't strictly required to store Unicode code units. However, that is their primary purpose, and the assumption holds true for any Unicode character-literal and string-literal.

3. Scope

I propose to deprecate implicit conversions between char8_t and char16_t, char32_t, or wchar_t. As demonstrated above, these are extremely bug-prone. Conversions between char16_t and char32_t are not affected.

3.1. What about "safe" comparisons?

In comparisons between code units, certain ranges of code points yield the expected result. For example, u8'x' == U'x' is true because all Unicode encodings are ASCII-compatible, so the numeric value of anything in the basic latin block (≤ U+007F) will have the same single-code-unit value in UTF-8, UTF-16, and UTF-32.

However, even those should be deprecated because:

• Keeping these valid would essentially leak implementation details of UTF-8 into the set of implicit conversions in the C++ core language, which seems like unclean design.
• To rely on this "feature", the developer needs to memorize which code points are "safe to use". It is not obvious whether c == U'€' or c == U'$' are always safe (hint: the latter one is), and it's quite likely that someone uses this "feature" accidentally.
• It would make this "feature" (or lack thereof) harder to teach than it needs to be. The rule can be very simple: char8_t and some other character types cannot be converted to one another. Simple rules are easy to teach.

3.2. What about char16_t and char32_t?

Following some negative feedback on [ClangWarning], the proposal no longer seeks to deprecate conversions between char16_t and char32_t. While these conversions are not guaranteed to be meaningful, there are no false positives in comparisons of UTF-16 and UTF-32 code units, and the comparison is quite likely to be correct.

It is possible to have false negatives when searching for a UTF-32 code unit outside the Basic Multilingual Plane (BMP) in UTF-16 text. However, these searches are tautologically false because values ≥ 0x10000 cannot (usually) be represented by char16_t, so compilers may catch some of them already.

It also also much less likely that char16_t ↔ char32_t conversions actually manifest as a bug. An application that only uses, say, Basic Latin characters and German or Norwegian umlauts can use char16_t and char32_t interchangeably. By contrast, mixing char8_t with other Unicode character types will almost certainly blow up in the user's face if the application processes any kind of non-ASCII text.

Last but not least, UTF-8 is becoming the "default encoding", especially on the web, while UTF-16 is increasingly becoming a "legacy encoding". This makes it unattractive to raise warnings for char16_t when the surrounding code may exist mostly for compatibility purposes, and C++ users are not interested in sinking much time into its maintenance. Substantially more code may be affected by a char16_t ↔ char32_t deprecation because both types were introduced in C++11, unlikely char8_t, which was added in C++20. See also [WikipediaEncodingPopularity]:

Recently it has become clear that the overhead of translating from/to UTF-8 on input and output, and dealing with potential encoding errors in the input UTF-8, overwhelms any benefits UTF-16 could offer. So newer software systems are starting to use UTF-8. The default string primitive used in newer programming languages, such as Go, Julia, Rust and Swift 5, assume UTF-8 encoding. PyPy also uses UTF-8 for its strings, and Python is looking into storing all strings in UTF-8. Microsoft now recommends the use of UTF-8 for applications using the Windows API, while continuing to maintain a legacy "Unicode" (meaning UTF-16) interface.

In summary, in char16_t ↔ char32_t comparisons, there are no false positives, the only false negatives are tautologically false (warnings exist), bugs are unlikely to manifest because code points outside the BMP are relatively uncommon, and if deprecation warnings were raised, that may happen in low-priority legacy code.

3.3. What about char and wchar_t?

As recommended by SG16, I propose to leave char intact, but deprecate conversions between char8_t and wchar_t. Since wchar_t almost certainly has a different encoding than char8_t, this conversions is as problematic as the char16_t and char32_t conversions.

The following conversions are not deprecated:

• char ↔ char8_t is not deprecated because the encoding of both sides is likely UTF-8, in which case the conversion is obviously safe. Substantial amounts of code may already rely on this.
• wchar_t ↔ char16_t and wchar_t ↔ char32_t can also be safe, depending on the platform. Windows-only code can likely treat wchar_t and char16_t interchangeably, and Linux-only code may treat wchar_t and char32_t interchangeably.

Furthermore, deprecating any conversion from char to other character types is a bad idea, and was unanimously recommended against by SG16. In some code bases, char is used purely for ASCII characters and strings. In such code bases, comparing char to any other character type is always correct, assuming that an ASCII-compatible encoding is used everywhere.

It may also be possible to deprecate conversions with char depending on ordinary literal encoding, but char is not necessarily using literal encoding, and doing so would invite non-portable code that fails to compile on e.g. EBCDIC platforms, to the great surprise of the author.

3.4. What about conversions with integers?

It is quite common to compare character types to integer types. For example, we may write c <= 0x7f to check whether a character falls into the basic latin block. There is nothing exceptionally bug-prone about comparing with say, 0x00A0 instead of U'\u00A0', so we are not interested in deprecating character/integer conversions.

3.5. What comes after deprecation?

The goal is to eventually remove these conversions entirely. Since the behavior is easily detected (§5. Implementation experience) and easily replaced (§4.1. Replacement for deprecated behavior), removal should be feasible within one or two revisions of the language.

Furthermore, I don't believe that having "tombstone behavior" would be necessary. That is, allowing the conversion to happen but making the program ill-formed if it happens. The reason is that char8_t, char16_t, and char32_t rarely appear in overload sets that include types that are not characters.

3.6. Why not make these conversions narrowing?

Another possible option (instead of deprecation or following deprecation) is to make the affected char8_t conversions narrowing conversions. This would make char32_t{c} for some char8_t c ill-formed, but the implicit conversion from char8_t to char32_t would remain valid.

There are multiple problems with this approach, which is why it is not proposed:

• char8_t to char32_t is a widening conversion, making the term "narrowing conversion" comically misleading.
• A long time has passed since C++11, and there is a lot of code using list-initialization now. This means that the "blast radius" of the change may still be quite large. If we accept that a non-trivial amount of warnings is raised in existing code, this half-measure seems unattractive.
• A lot of the problematic cases are not initialization, but comparisons as shown in §2. Introduction. Narrowing conversions play no role in equality comparison or in the usual arithmetic conversions.

4. Impact on existing code

It is not trivial to estimate how much code would be affected by a deprecation like this. However, that is ultimately not what makes or breaks this proposal. The goal is not to deprecate a rarely used feature to give it new meaning, like array[0,1] prior to [P1161R3].

The goal is to deprecate a bug-prone and harmful feature to make the language safer.

The longer we wait, the more mistakes will be made using char8_t and other types. C++ will undoubtedly get improved support for the Unicode character types over time, making them used more frequently, so we better deal with this problem now than never.

4.1. Replacement for deprecated behavior

If the new deprecation warnings spot a bug like in §2. Introduction, some work will be required to fix it, but the deprecation will have done its job.

If the comparison is obviously safe, such as c == U'0' with char8_t c, the resolution is usually trivial, like c == u8'0'. This could even be done automatically with tools like clang-tidy.

5. Implementation experience

Corentin Jabot has recently implemented a -Wcharacter-conversion warning in Clang ([ClangWarning]), which is enabled by default. You can test this at [CompilerExplorer].

However the warning is more conservative than the proposed deprecation; it does not warn on "safe comparisons" (§3.1. What about "safe" comparisons?).

6. Wording

The following changes are relative to [N5014].

[conv.integral]

Change [conv.integral] paragraph 1 as follows, and split it into two paragraphs:

[expr.arith.conv]

Change [expr.arith.conv] paragraph 1 as follows:

[expr.static.cast]

Immediately prior to [expr.static.cast] paragraph 5, insert a new paragraph:

Do not change [expr.static.cast] paragraph 5; it is cited here for reference:

Otherwise, an expression $E$ can be explicitly converted to a type T if there is an implicit conversion sequence ([over.best.ics]) from $E$ to T, […]. […], the result object is direct-initialized from $E$.

[depr.conv.unicode]

Insert a new subclause in [depr] between [depr.local] and [depr.capture.this], containing a single paragraph:

7. References