ASCII character utilities

Contents

1. Introduction

Testing whether a character falls into a specific subset of ASCII characters or performing some simple transformations are common tasks in text processing. For example, applications may need to check if identifiers are comprised of alphanumeric ASCII characters or underscores; Unicode properties are not relevant to this task, and usually, neither are locales.

Unfortunately, these common and simple tasks are only supported through functions in the <cctype> and <locale> headers, such as:

Especially the <cctype> functions are ridden with problems:

1. There is no support for Unicode character types (char8_t, char16_t, and char32_t).
2. These functions are not constexpr, but performing basic characters tests would be useful at compile time.
3. There are distinct function names for char and wchar_t such as std::isalnum and std::iswalnum, making generic programming more difficult.
4. If char is signed, these functions can easily result in undefined behavior because the input must be representable as unsigned char or be EOF. If char represents a UTF-8 code unit, passing any non-ASCII code unit into these functions has undefined behavior.
5. These functions violate the zero-overhead principle by also handling an EOF input, and in many use cases, EOF will never be passed into these functions anyway. The caller can easily deal with EOF themselves.
6. The return type of charater tests is int, where a nonzero return value indicates that a test succeeded. This is very unnatural in C++, where bool is more idiomatic.
7. Some functions use the currently installed "C" locale, which makes their use questionable for high-performance tasks because each invocation is typically an opaque call that checks the current locale.

We propose lightweight replacement functions which address all these problems.

1.1. Can't you implement this trivially yourself?

It is worth noting that some of the functions can be implemented very easily by the user. For example, existing code may already use a check like c >= '0' && c <= '9' to test for ASCII digits, and our proposed is_ascii_digit does just that.

However, not all of the proposed functions are this simple. For example, checking whether a char is an ASCII punctuation character ('#', '?', etc.) would require lots of separate checks done naively. In the standard library, it can be efficiently implemented using a 128-bit or 256-bit bitset.

Even if all proposed functions were trivial to implement, working with ASCII characters is such an overwhelmingly common use case that it's worth supporting in the standard library.

2. Design

All proposed functions are constexpr, locale-independent, overloaded (i.e. no separate name for separate input types), and accept any character type (char, wchar_t, char8_t, char16_t, and char32_t). Furthermore, all function names contain ascii to raise awareness for the fact that these functions do not handle Unicode characters. A user would expect is_upper(U'Ä') to be true, but is_ascii_upper(U'Ä') to be false.

character-type means that there exists an overload set where this placeholder is replaced with each of the character types. This design is more consistent with std::from_chars and <cmath> functions than say, template<class Char>. Equivalent functions could also be added to C, if there is interest. This signature also allows the use with types that are convertible to a specific character type.

2.1. List of proposed functions

Find below a list of proposed functions. Note that the character set notation [...] is taken from RegEx.

2.2. is_ascii

This additional function is mainly useful for checking if a character "is ASCII", i.e. falls into the basic latin block, before performing an ASCII-only evaluation.

2.3. base parameter in is_ascii_digit

Similar to std::to_chars, std::is_ascii_digit can also take a base parameter:

If base ≤ 10, the range of valid ASCII digit character is simply limited. For greater base, a subset of alphabetic characters is also accepted, starting with 'a' or 'A'. Such a function is useful when parsing numbers with a base of choice, which is what std::to_chars does, for example.

2.4. is_ascii_bit and is_ascii_octal_digit

C++ and various other programming languages support binary and octal literals, so it seems like an arbitrary choice to only have dedicated overloads for (hexa)decimal digits. is_ascii_bit may be especially useful, such as when dealing with bit-strings like one of the std::bitset constructors.

In conclusion, we may as well have functions for bases 2, 8, 10, and 16; they're not doing much harm, they're trivial to implement, and some users may find them useful.

2.5. Case-insensitive comparison functions

As shown in the table above, we also propose the case-insensitive comparison functions.

2.6. Why no function objects?

For case-insensitive comparisons and for character tests in general, function objects may be convenient because they can be more easily used in algorithms:

However, there is no reason why is_ascii_digit needs to be a function object. It is not a customization point, but a plain function. Furthermore, defining function objects for this purpose may be obsoleted by [P3312R1] Overload Set Types.

2.7. What to do for ASCII-incompatible char and wchar_t

Not every ordinary and wide character encoding is ASCII-compatible, such as EBCDIC, Shift-JIS, and (defunct) ISO-646, i.e. code units ≤ 0x7f do not represent the same characters as ASCII.

This begs the question: what should is_ascii_digit('0') do on an EBCDIC platform, where this call is is_ascii_digit(char(0xf0)) ? We have three options, discussed below.

2.7.1. Conditionally supported char overloads

We could mandate that the ordinary literal encoding is an ASCII superset for the char overload to exist. This would force a cast (to char8_t) to use the functions on EBCDIC platforms. It is not clear how implementations would treat Shift-JIS; GCC assumes '\\' == '¥' to be true, so this option may not be enough to alleviate the awkwardness of is_ascii_punctuation('¥').

Also, this option is not very useful. It is reasonable to have UTF-8 data stored in a char[] on EBCDIC platforms, and having to perform casts to char8_t would be awkward.

2.7.2. Transcode char to ASCII

We could transcode from the ordinary literal encoding to ASCII and produce an answer for the result of that transcoding. This would be a greater burden for implementations, especially on EBCDIC platforms. The benefit is that is_ascii_digit('0') is always true, although is_ascii_digit(char(0x30)) may not be. However, is_ascii_digit(char8_t(0x30)) is always true.

It probably does not solve the is_ascii_punctuation('¥') case, as implementers may keep transcoding '¥' and '\\' in the same way. It would also give incorrect answers for stateful encodings. There are EBCDIC control characters that do not have an ASCII equivalent, so if we were to do conversions, we would have to decide what, for example, is_ascii_control('\u008B') should produce.

2.7.3. Treat the input as ASCII, regardless of the literal encoding

This is our proposed behavior.

The most simple option is to ignore literal encoding entirely, and assume that char inputs are ASCII-encoded. The greatest downside is that depending on encoding, is_ascii_digit('0') may be false, which may be surprising to the user. However, the main purpose of these functions is to be called with characters taken from ASCII text, so what results they yield when passing literals is not so important.

There are use cases for this behavior on EBCDIC platforms. A lot of protocols (HTTP, POP) and file formats (JSON, XML) are ASCII/UTF-8-based and need to be supported on EBCDIC systems, making these functions universally useful, especially as <cctype> functions cannot easily be used to deal with ASCII on these platforms.

Ultimately, do we want functions to deal with ASCII or the literal encoding? If we want them to be a general way to query the ordinary literal encoding, is_ascii is a terrible name, and finding a more general name would prove difficult.

2.8. What if the input is a non-ASCII code unit?

Text input is rarely guaranteed to be pure ASCII, i.e. some code units may be > 0x7f. However, we're still interested in ASCII characters within that input. For example, we may

• parse pure ASCII numbers like 123 in a UTF-8 JSON (or other config) file,
• trim ASCII whitespace in HTTP headers, which are encoded with ISO-8859-1,
• parse ASCII-alphanumeric variable names in Lua scripts, where non-ASCII characters can appear (comments, string),
• ...

It is possible (and expected) that the user calls say, is_ascii_digit(U'ö'), at least indirectly. For the sake of convenience, all proposed functions should handle such inputs by

• returning false in the case of all testing functions, and
• applying an identity transformation in transformation/case-insensitive comparison functions.

The proposed behavior also works excellently with any ASCII-compatible encoding, such as UTF-8. Surrogate code units in UTF-8 are all greater than 0x7F, so if we implement say, is_ascii_digit naively by checking c >= '0' && c <= '9', it "just works".

2.9. Why not accept any integer type?

Some people argue that a test like is_ascii_digit('0') is a purely numerical test using the ASCII table, and so passing is_ascii_digit(0x30) should also be valid.

However, this permissive interface would invite bugs. For example, c - '0' is the difference between ASCII characters, not an ASCII character, so passing it into is_ascii_digit would be nonsensical. Static type systems exist for a reason: to protect us from stupid mistakes. While char, char32_t etc. are not required to be ASCII-encoded, they are at least characters, so passing them into our functions is likely something the user intended to do, which we cannot say with confidence about int, unsigned int, etc.

Additionally, if we allowed passing signed integers, we may want to make the behavior erroneous or undefined for negative inputs because is_ascii_digit(-1'000'000) is most likely a developer mistake. Our interface is very simple: it has a wide contract and almost all functions are noexcept. Let's keep it that way!

Lastly, even proponents of passing integer types would not want is_ascii_digit(true) to be valid.

2.10. ASCII case-insensitive views and case transformation algorithms

Ignoring or transforming ASCII case in algorithms is a fairly common problem. Therefore, it may be useful to provide views such as std::views::ascii_lower, algorithms like std::ranges::equal_ascii_case_insensitive, etc.

While case transformations can be implemented naively using std::transform, dedicated functions would allow an efficient vectorized implementation for contiguous ranges, which can be many times faster ([AvoidCharByChar], [AVX-512CaseConv]) Similarly, a case-insensitive comparison function can be vectorized. In fact, POSIX's strncasecmp has been heavily optimized in glibc ([AVX2strncasecmp]), and providing range-based interfaces would allow delegating to these heavily optimized functions.

We intend to propose such utilities in a future paper or revision of this paper. Currently, this proposal is focused exclusively on operations involving character types.

2.11. Why just ASCII?

It may be tempting to generalize the proposed utilities beyond ASCII, e.g. to UTF-8. However, this is not proposed for multiple reasons:

• You cannot pass char8_t into a UTF-8 is_upper function and expect meaningful results. In general, operations on variable-length encodings require sequences of code units. The interface we propose only makes sense for ASCII.
• Unicode utilities are tremendously more complex than ASCII utilities. Some Unicode case conversions even require multi-code-point changes.

3. Implementation experience

A naive implementation of all proposed functions can be found at [CompilerExplorer], although these are implemented as function templates, not as overload sets (as proposed).

A more advanced implementation of some functions can be found in [µlight]. Character tests can be optimized using 128-bit or 256-bit bitsets.

4. Wording

The wording changes are relative to [N5008].

In subclause [version.syn], update the synopsis as follows:

In Clause [text], append a new subclause:

5. References