1 High-Level Overview

This paper introduces views and ranges for transcoding between UTF formats:

static_assert((u8"🙂" | views::to_utf32 | ranges::to<u32string>()) == U"🙂");

It handles errors by replacing invalid subsequences with �:

static_assert((u8"🙂" | views::take(3) | to_utf32 | ranges::to<std::u32string>()) == U"�");

And by providing or_error views that provide std::expected:

static_assert(
  *(u8"🙂" | views::take(3) | views::to_utf32_or_error).begin() ==
  unexpected{utf_transcoding_error::truncated_utf8_sequence});

2 UTF Primer

If you’re already familiar with Unicode, you can skip this section.

The Unicode standard maps abstract characters to code points in the Unicode codespace from 0 to 0x10FFFF. Unicode text forms a coded character sequence, “an ordered sequence of one or more code points.” [Definitions]

The simplest way of encoding code points is UTF-32, which encodes code points as a sequence of 32-bit unsigned integers. The building blocks of an encoding are code units, and UTF-32 has the most direct mapping between code points and code units.

Any values greater than 0x10FFFF are rejected by validators for being outside the range of valid Unicode.

Next is UTF-16, which exists for the historical reason that the Unicode codespace used to top out at 0xFFFF. Code points outside this range are represented using surrogates, a reserved area in codespace which allows combining the low 10 bits of two code units to form a single code point.

UTF-16 is rendered invalid by improper use of surrogates: a high surrogate not followed by a low surrogate or a low surrogate not preceded by a high surrogate. Note that the presence of any surrogate code points in UTF-32 is also invalid.

Finally, UTF-8, the most ubiquitous and most complex encoding. This uses 8-bit code units. If the high bit of the code unit is unset, the code unit represents its ASCII equivalent for backwards compatibility. Otherwise the code unit is either a start byte, which describes how long the subsequence is (two to four bytes long), or a continuation byte, which fills out the subsequence with the remaining data.

UTF-8 code unit sequences can be invalid for many reasons, such as a start byte not followed by the correct number of continuation bytes, or a UTF-8 subsequence that encodes a surrogate.

Transcoding in this context refers to the conversion of characters between these three encodings.

3 Existing Standard UTF Interfaces in C and C++

3.1 C

C contains an alphabet soup of transcoding functions in <stdlib.h>, <wchar.h>, and <uchar.h>. [Null-terminated multibyte strings]

This paper doesn’t fully litigate these functions’ flaws (see WG14 [N2902] for a more detailed explanation). Some of the issues users encounter include reliance on an internal global conversion state, reliance on the current setting of the global C locale, optimization barriers in one-code-unit-at-a-time function calls, and inadequate error handling that does not support replacement of invalid subsequences with � as specified by Unicode.

Example:

setlocale(LC_ALL, "en_US.utf8");
char c[5] = {0};
const char16_t* w = u"\xd83d\xdd74";
mbstate_t state;
memset(&state, 0, sizeof(state));
c16rtomb(c, w[0], &state);
c16rtomb(c, w[1], &state);
const char* e = "\xf0\x9f\x95\xb4";
assert(strcmp(c, e) == 0);

3.2 C++

C++’s existing transcoding functionality, other than the aforementioned functions it inherits from C, consists of the set of std::codecvt facets provided in <locale> and <codecvt>.

Example:

std::wstring_convert<std::codecvt_utf8<char32_t>, char32_t> conv;
std::string c = conv.to_bytes(U"🙂");
assert(c == "\xf0\x9f\x99\x82");

All of the Unicode-specific functionality in this header was deprecated in C++17, and [P2871R3] and [P2873R2] finally remove most of it in C++26. There are many concerns about these interfaces, particularly with respect to safety.

These functions throw exceptions on encountering invalid UTF. Unicode functions that use exceptions for error handling are a well-known footgun because users consistently invoke them on untrusted user input without handling the exceptions properly, leading to denial-of-service vulnerabilities.

An example of this anti-pattern (although not involving these specific functions) can be found in [CVE-2007-3917], where a multiplayer RPG server could be crashed by malicious users sending invalid UTF. Below is the patch: [wesnoth]

- msg = font::word_wrap_text(msg,font::SIZE_SMALL,map_outside_area().w*3/4);
+ try {
+     // We've had a joker who send an invalid utf-8 message to crash clients
+     // so now catch the exception and ignore the message.
+     msg = font::word_wrap_text(msg,font::SIZE_SMALL,map_outside_area().w*3/4);
+ } catch (utils::invalid_utf8_exception&) {
+     LOG_STREAM(err, engine) << "Invalid utf-8 found, chat message is ignored.\n";
+     return;
+ }

Because it doesn’t use exceptions, the functionality proposed by this paper can serve as a safe, modern replacement for the deprecated and removed codecvt facets.

4 Replacing Ill-Formed Subsequences with “�”

When a transcoder encounters an invalid subsequence, the modern best practice is to replace it in the output with one or more � characters (U+FFFD, REPLACEMENT CHARACTER). The methodology for doing so is described in §3.9.6 of the Unicode Standard v17.0, Substitution of Maximal Subparts [Substitution].

For UTF-32 and UTF-16, each invalid code unit is replaced by an individual � character.

For UTF-8, the same rule applies except if “a sequence of two or three bytes is a truncated version of a sequence which is otherwise well-formed to that point.” In the latter case, the full two-to-three byte subsequence is replaced by a single � character.

For example, UTF-8 encodes 🙂 as 0xF0 0x9F 0x99 0x82.

If that sequence of bytes is truncated to just 0xF0 0x9F 0x99, it becomes a single � replacement character.

On the other hand, if the first byte of the four-byte sequence is changed from 0xF0 to 0xFF, then it’s replaced by four replacement characters, ����, because no valid UTF-8 subsequence begins with 0xFF.

More subtly, the subsequence 0xED 0xA0 must be replaced with two replacement characters, ��, because any continuation of that subsequence can only result in a surrogate code point, so it can’t prefix any valid subsequence.

Each of the proposed to_utfN_view views adheres to this specification. The to_utfN_as_error views also use this scheme but produce unexpected<utf_transcoding_error> values instead of replacement characters.

5 Design Overview

5.1 Transcoding Views

Invoking begin() or end() on a transcoding view constructs an instance of an exposition-only to_utf_view::iterator type.

The to_utf_view::iterator stores an iterator pointing to the start of the character it’s transcoding, and a back-pointer to the underlying range in order to bounds check its beginning and end (which is required for correctness, not just safety).

The to_utf_view::iterator maintains a small buffer (buf_) containing the current character, transcoded into the target encoding. If the underlying range models forward_range, the buffer may additionally contain the transcoded code units of one or more characters following the current one: an implementation is permitted to transcode a whole chunk of input at a time (for example, using SIMD instructions) and serve subsequent increments — or, when iterating backward, decrements — out of the buffer. If the underlying range is single-pass, the buffer contains the code units of exactly one character (between one and four code units), because reading ahead in a single-pass range is destructive and therefore observable.

It also maintains an index (buf_index_) into this buffer, which it increments or decrements when operator++ or operator-- is invoked, respectively. If it runs out of code units in the buffer, it reads more elements from the underlying view. operator* provides the current element of the buffer.

Below is an approximate block diagram of the iterator. Bold lines denote actual data members of the iterator; dashed lines are just function calls.

The to_utf_view::iterator is converting the string Qϕ学𡪇 from UTF-8 to UTF-16. The user has iterated the view to the first UTF-16 code unit of the fourth character. current_ points to the start of the fourth character in the input. buf_ contains both UTF-16 code units of the fourth character; buf_index_ keeps track of the fact that we’re currently pointing to the first one. If we invoke operator++ on the to_utf_view::iterator, it will increment buf_index_ to point to the second code unit. On the other hand, if we invoke operator--, it will notice that buf_index_ is already at the beginning and move backward from the fourth character to the third character by invoking read-reverse(). The read() and read-reverse() functions contain most of the actual transcoding logic, updating current_ and filling buf_ up with the transcoded characters.

(The diagram depicts the minimal buffer, holding one character at a time. As described above, an implementation may instead fill buf_ with the code units of several consecutive characters when the underlying range is multipass.)

Iterating a bidirectional transcoding view backwards produces, in reverse order, the exact same sequence of characters or expected values as are produced by iterating the view forwards.

5.1.1 utf_transcoding_error

Each transcoding view, like to_utf8_view, which produces a range of char8_t and handles errors by substituting � replacement characters, has a corresponding _or_error equivalent, like to_utf8_view_or_error, which produces a range of expected<char8_t, utf_transcoding_error> and handles errors by substituting unexpected<utf_transcoding_error>s.

utf_transcoding_error is an enumeration whose enumerators are:

• truncated_utf8_sequence
  • An ill-formed subsequence that matches the beginning of some well-formed sequence.
  • Example invalid code unit sequence: UTF-8 0xE1 0x80.
• unpaired_high_surrogate
  • Example invalid code unit sequence: UTF-16 0xD800.
• unpaired_low_surrogate
  • Example invalid code unit sequence: UTF-16 0xDC00.
• unexpected_utf8_continuation_byte
  • Example invalid code unit sequence: UTF-8 0x80.
• overlong
  • An overlong UTF-8 encoding.
  • Example invalid code unit sequence: UTF-8 0xE0 0x80.
• encoded_surrogate
  • Applies to both UTF-8 and UTF-32.
  • Example invalid code unit sequence: UTF-8 0xED 0xA0, UTF-32 0x0000D800.
• out_of_range
  • Applies to both UTF-8 and UTF-32
  • In UTF-8, this applies to 0xF4 if it is followed by a continuation byte greater than 0x8F
  • In UTF-32, this is any code unit greater than 0x10FFFF
  • Example invalid code unit sequence: UTF-8 0xF4 0x90, UTF-32 0x110000.
• invalid_utf8_leading_byte
  • In UTF-8, this applies to 0xC0-0xC1 and 0xF5-0xFF.
  • Example invalid code unit sequence: UTF-8 0xC0.

An alternative approach to minimize the number of enumerators could merge truncated_utf8_sequence with unpaired_high_surrogate and merge unexpected_utf8_continuation_byte with unpaired_low_surrogate, but based on feedback, splitting these up seems to be preferred.

The table below compares the error handling behavior of the to_utf16 and to_utf16_or_error views on various sample UTF-8 inputs from the “Substitution of Maximal Subparts” section of the Unicode standard: [SubstitutionExamples]

5.2 Code Unit Views

SG16 has a goal to ensure that C++ standard library functions that expect UTF-encoded input do not accept parameters of type char or wchar_t, whose encodings are implementation-defined, and instead use char8_t, char16_t, and char32_t. These views follow that pattern.

Because virtually all UTF-8 text processed by C++ is stored in char (and similarly for UTF-16 and wchar_t), this means that we need a terse way to smooth over the transition for users. To do so, this paper introduces views for casting between character types: as_char, as_wchar_t, as_char8_t, as_char16_t, and as_char32_t.

These are syntactic sugar for producing a std::ranges::transform_view with an exposition-only transformation functor that performs the needed cast.

6 Additional Examples

6.1 Transcoding a UTF-8 String Literal to a std::u32string

std::u32string hello_world =
  u8"こんにちは世界"sv | std::views::to_utf32 | std::ranges::to<std::u32string>();

6.2 Sanitizing Potentially Invalid Unicode

Note that transcoding to and from the same encoding is not a no-op; it must maintain the invariant that the output of a transcoding view is always valid UTF.

template <typename CharT>
std::basic_string<CharT> sanitize(CharT const* str) {
  return std::null_term(str) | std::views::to_utf<CharT> | std::ranges::to<std::basic_string<CharT>>();
}

6.3 Returning the Final Non-ASCII Code Point in a String, Transcoding Backwards Lazily

std::optional<char32_t> last_nonascii(std::ranges::view auto str) {
  for (auto c : str | std::views::to_utf32 | std::views::reverse
                    | std::views::filter([](char32_t c) { return c > 0x7f; })) {
    return c;
  }
  return std::nullopt;
}

6.4 Transcoding Strings and Throwing a Descriptive Exception on Invalid UTF

(This assumes a reflection-based enum_to_string function.)

template <typename FromChar, typename ToChar>
std::basic_string<ToChar> transcode_or_throw(std::basic_string_view<FromChar> input) {
  std::basic_string<ToChar> result;
  auto view = input | std::views::to_utf_or_error<ToChar>;
  for (auto it = view.begin(), end = view.end(); it != end; ++it) {
    if ((*it).has_value()) {
      result.push_back(**it);
    } else {
      throw std::runtime_error("error at position " +
                               std::to_string(it.base() - input.begin()) + ": " +
                               enum_to_string((*it).error()));
    }
  }
  return result;
}

  // prints: "error at position 2: truncated_utf8_sequence"
  transcode_or_throw<char8_t, char16_t>(
    u8"hi🙂"sv | std::views::take(5) | std::ranges::to<std::u8string>());

6.5 Changing the Suits of Unicode Playing Card Characters

enum class suit : std::uint8_t {
  spades = 0xA,
  hearts = 0xB,
  diamonds = 0xC,
  clubs = 0xD
};

// Unicode playing card characters are laid out such that changing the second least
// significant nibble changes the suit, e.g.
// U+1F0A1 PLAYING CARD ACE OF SPADES
// U+1F0B1 PLAYING CARD ACE OF HEARTS
constexpr char32_t change_playing_card_suit(char32_t card, suit s) {
  if (U'\N{PLAYING CARD ACE OF SPADES}' <= card && card <= U'\N{PLAYING CARD KING OF CLUBS}') {
    return (card & ~(0xF << 4)) | (static_cast<std::uint8_t>(s) << 4);
  }
  return card;
}

void change_playing_card_suits() {
  std::u8string_view const spades = u8"🂡🂢🂣🂤🂥🂦🂧🂨🂩🂪🂫🂭🂮";
  std::u8string const hearts =
    spades |
    to_utf32 |
    std::views::transform(std::bind_back(change_playing_card_suit, suit::hearts)) |
    to_utf8 |
    std::ranges::to<std::u8string>();
  assert(hearts == u8"🂱🂲🂳🂴🂵🂶🂷🂸🂹🂺🂻🂽🂾");
}

6.6 Handling Byte Offsets and Endianness

Say we want to handle a set of bytes in a message starting at offset N with length K that is UTF16BE text:

std::u8string parse_message_subset(
    std::span<std::byte> message, std::size_t offset, std::size_t length) {
  return std::span{message.begin() + offset, message.begin() + offset + length}
         | std::views::chunk(2)
         | std::views::transform(
             [](const auto chunk) {
               std::array<std::byte, 2> a{};
               std::ranges::copy(chunk, a.begin());
               return std::bit_cast<std::uint16_t>(a);
             })
         | std::views::from_big_endian
         | std::views::as_char16_t
         | std::views::to_utf8
         | std::ranges::to<std::u8string>();
}

Note that this depends on P4030R0 “Endian Views” for std::views::from_big_endian.

6.7 Transcoding into a buffer of a fixed number of code units without truncating code points

template <typename T>
constexpr bool is_continuation(T c) {
  if constexpr (std::is_same_v<decltype(c), char8_t>) {
    return (c & 0xC0) == 0x80;
  } else if constexpr (std::is_same_v<decltype(c), char16_t>) {
    return c >= 0xDC00 && c <= 0xDFFF;
  } else {
    return false;
  }
}

template <typename FromType, typename ToType, std::size_t N>
constexpr std::inplace_vector<ToType, N> transcode_truncating_correctly(
    std::basic_string_view<FromType> input) {
  std::inplace_vector<ToType, N> output;
  for (auto code_point_view : input
       | std::views::to_utf<ToType>
       | std::views::chunk_by([](auto, auto b) { return is_continuation(b); })) {
    if (std::ranges::distance(code_point_view)
        > static_cast<std::ptrdiff_t>(output.max_size() - output.size()))
      break;
    std::ranges::copy(code_point_view, std::back_insert_iterator{output});
  }
  return output;
}

6.8 Performing code unit substitutions on cuneiform strings

// Adapted from an ICU unit test:
// https://github.com/unicode-org/icu/blob/649262a75ecddb15a0e58d71f637a8a32eaabd43/icu4c/source/test/intltest/utfiteratortest.cpp#L1205-L1228
std::u16string zamin = u"𒀭𒎏𒄈𒋢𒍠𒊩";
auto view = zamin | std::views::to_utf32;
auto it = view.begin();
++it;
auto ningirsuBegin = it.base();
std::advance(it, 3);
auto ningirsuEnd = it.base();
zamin.replace(ningirsuBegin, ningirsuEnd, u"𒊺𒉀");
assert(std::ranges::equal(zamin, u"𒀭𒊺𒉀𒍠𒊩"sv));

7 Dependencies

The code unit views depend on [P3117R1] “Extending Conditionally Borrowed”.

8 Implementation Experience

The most recent revision of this paper has a reference implementation called beman.utf_view available on GitHub, which is a fork of Jonathan Wakely’s implementation of P2728R6 as an implementation detail for libstdc++. It is part of the Beman project.

Versions of the interfaces provided by previous revisions of this paper have also been implemented, and re-implemented, several times over the last 5 years or so, as part of a proposed (but not yet accepted!) Boost library, Boost.Text. Boost.Text has hundreds of stars on GitHub.

Both libraries have comprehensive tests.

9 Wording

9.1 Additional Helper Concepts

Add the following to 25.5.2 [range.utility.helpers]:

  template<class T>
    concept code-unit =
      same_as<remove_cv_t<T>, char8_t> || same_as<remove_cv_t<T>, char16_t> || same_as<remove_cv_t<T>, char32_t>;

9.2 Header <ranges> synopsis

Add the following to 25.2 [ranges.syn], after the as_input_view entries:

  // [range.transcoding], transcoding views
  enum class utf_transcoding_error : unspecified;
  enum class to_utf_view_error_kind : unspecified;

  template<code-unit ToType>
  struct to_utf_tag_t;

  template<code-unit ToType>
  constexpr to_utf_tag_t<ToType> to_utf_tag{};

  using to_utf8_tag_t = to_utf_tag_t<char8_t>;
  constexpr to_utf8_tag_t to_utf8_tag{};

  using to_utf16_tag_t = to_utf_tag_t<char16_t>;
  constexpr to_utf16_tag_t to_utf16_tag{};

  using to_utf32_tag_t = to_utf_tag_t<char32_t>;
  constexpr to_utf32_tag_t to_utf32_tag{};

  template<input_range V, to_utf_view_error_kind E, code-unit ToType>
    requires view<V> && code-unit<range_value_t<V>>
  class to_utf_view;

  template<class V, to_utf_view_error_kind E, class ToType>
    constexpr bool enable_borrowed_range<to_utf_view<V, E, ToType>> =
      enable_borrowed_range<V>;

  namespace views {
    template<code-unit ToType> inline constexpr unspecified to_utf = unspecified;
    template<code-unit ToType> inline constexpr unspecified to_utf_or_error = unspecified;
    inline constexpr unspecified to_utf8 = unspecified;
    inline constexpr unspecified to_utf8_or_error = unspecified;
    inline constexpr unspecified to_utf16 = unspecified;
    inline constexpr unspecified to_utf16_or_error = unspecified;
    inline constexpr unspecified to_utf32 = unspecified;
    inline constexpr unspecified to_utf32_or_error = unspecified;
  }

  // [range.codeunitadaptor], code unit adaptors
  namespace views {
    inline constexpr unspecified as_char = unspecified;
    inline constexpr unspecified as_wchar_t = unspecified;
    inline constexpr unspecified as_char8_t = unspecified;
    inline constexpr unspecified as_char16_t = unspecified;
    inline constexpr unspecified as_char32_t = unspecified;
  }

9.3 Transcoding views

Add the following subclause to 25.7 [range.adaptors]:

25.7.? Transcoding views [range.transcoding]

25.7.?.1 Overview [range.transcoding.overview]

to_utf_view produces a view of the UTF code units transcoded from the elements of a utf-range. It transcodes from UTF-N to UTF-M, where N and M are each one of 8, 16, or 32. N may equal M. to_utf_view’s ToType template parameter is based on a mapping between character types and UTF encodings, which is that that char8_t corresponds to UTF-8, char16_t corresponds to UTF-16, and char32_t corresponds to UTF-32. If its to_utf_view_error_kind constant template parameter is expected, it produces a view of expected<charN_t, utf_transcoding_error> where invalid input subsequences result in errors.

The names views::to_utf<ToType>, views::to_utf_or_error<ToType>, views::to_utf8, views::to_utf8_or_error, views::to_utf16, views::to_utf16_or_error, views::to_utf32, and views::to_utf32_or_error denote range adaptor objects ([range.adaptor.object]).

views::to_utf<ToType> is equivalent to views::to_utf8 if ToType is char8_t, views::to_utf16 if ToType is char16_t, and views::to_utf32 if ToType is char32_t, and similarly for views::to_utf_or_error.

Let views::to_utfN denote any of the aforementioned range adaptor objects, let Char be its corresponding character type, and let Error be its corresponding to_utf_view_error_kind. Let E be an expression and let T be remove_cvref_t<decltype((E))>. If decltype((E)) does not model utf-range, or if T is an array of char8_t, char16_t, or char32_t, to_utfN(E) is ill-formed. Otherwise, the expression to_utfN(E) is expression-equivalent to:

• If E is a specialization of empty_view ([range.empty.view]):

  • If Error is to_utf_view_error_kind::replacement, then empty_view<Char>{}.

  • Otherwise, empty_view<expected<Char, utf_trancoding_error>>{}.

• Otherwise, if the type of E is a (possibly cv-qualified) specialization of to_utf_view, then to_utf_view(E.base(), cw<Error>, to_utf_tag<Char>).

• Otherwise, if the type of E is cv subrange<to_utf_view::iterator, to_utf_view::iterator, subrange_kind::unsized> for some specialization of to_utf_view, then to_utf_view(subrange(E.begin().base(), E.end().base()), cw<Error>, to_utf_tag<Char>).

• Otherwise, to_utf_view(E, cw<Error>, to_utf_tag<Char>).

25.7.?.2 Enumeration utf_transcoding_error [range.transcoding.error.transcoding]

enum class utf_transcoding_error : unspecified {
  truncated_utf8_sequence,
  unpaired_high_surrogate,
  unpaired_low_surrogate,
  unexpected_utf8_continuation_byte,
  overlong,
  encoded_surrogate,
  out_of_range,
  invalid_utf8_leading_byte
};

25.7.?.3 Enumeration to_utf_view_error_kind [range.transcoding.error.kind]

enum class to_utf_view_error_kind : unspecified {
  replacement,
  expected
};

25.7.?.4 UTF Tags [range.transcoding.tags]

template<code-unit ToType>
struct to_utf_tag_t {
  explicit to_utf_tag_t() = default;
};

template<code-unit ToType>
constexpr to_utf_tag_t<ToType> to_utf_tag{};

using to_utf8_tag_t = to_utf_tag_t<char8_t>;

constexpr to_utf8_tag_t to_utf8_tag{};

using to_utf16_tag_t = to_utf_tag_t<char16_t>;

constexpr to_utf16_tag_t to_utf16_tag{};

using to_utf32_tag_t = to_utf_tag_t<char32_t>;

constexpr to_utf32_tag_t to_utf32_tag{};

25.7.?.5 Class template to_utf_view [range.transcoding.view]

template<input_range V, to_utf_view_error_kind E, code-unit ToType>
  requires view<V> && code-unit<range_value_t<V>>
class to_utf_view : public view_interface<to_utf_view<V, E, ToType>> {
private:
  template<bool>
  struct iterator; // exposition only
  template<bool>
  struct sentinel; // exposition only

  V base_ = V(); // exposition only

public:
  constexpr to_utf_view() requires default_initializable<V> = default;
  template <auto E2>
    constexpr explicit to_utf_view(V base, constant_wrapper<E2, to_utf_view_error_kind>, to_utf_tag_t<ToType>)
      requires (constant_wrapper<E2, to_utf_view_error_kind>::value == E);

  constexpr V base() const& requires copy_constructible<V> { return base_; }
  constexpr V base() && { return std::move(base_); }

  constexpr iterator<false> begin();
  constexpr iterator<true> begin() const requires range<const V> && ((same_as<range_value_t<V>, char32_t>) || (!forward_range<const V>))
  {
    if constexpr (bidirectional_range<const V>) {
      return iterator<true>(begin(base_), begin(base_), end(base_));
    } else {
      return iterator<true>(begin(base_), end(base_));
    }
  }

  constexpr sentinel<false> end() { return sentinel<false>(end(base_)); }
  constexpr iterator<false> end() requires common_range<V>
  {
    if constexpr (bidirectional_range<V>) {
      return iterator<false>(begin(base_), end(base_), end(base_));
    } else {
      return iterator<false>(end(base_), end(base_));
    }
  }
  constexpr sentinel<true> end() const requires range<const V>
  {
    return sentinel<true>(end(base_));
  }
  constexpr iterator<true> end() const requires common_range<const V>
  {
    if constexpr (bidirectional_range<const V>) {
      return iterator<true>(begin(base_), end(base_), end(base_));
    } else {
      return iterator<true>(end(base_), end(base_));
    }
  }

  constexpr bool empty() const { return empty(base_); }

  constexpr size_t size()
    requires sized_range<V> && same_as<char32_t, range_value_t<V>> && same_as<char32_t, ToType>
  {
    return size(base_);
  }

  constexpr auto reserve_hint() requires approximately_sized_range<V>;
  constexpr auto reserve_hint() const requires approximately_sized_range<const V>;
};

template<class R, auto E2, code-unit ToType>
  to_utf_view(R&&, constant_wrapper<E2, to_utf_view_error_kind>, to_utf_tag_t<ToType>)
    -> to_utf_view<views::all_t<R>, constant_wrapper<E2, to_utf_view_error_kind>::value, ToType>;

template <class V, to_utf_view_error_kind E, class ToType>
  inline constexpr bool enable_borrowed_range<to_utf_view<V, E, ToType>> = enable_borrowed_range<V>;

template <auto E2>
  constexpr explicit to_utf_view(V base, constant_wrapper<E2, to_utf_view_error_kind>, to_utf_tag_t<ToType>)
    requires (constant_wrapper<E2, to_utf_view_error_kind>::value == E);

Effects: Initializes base_ with std::move(base).

constexpr iterator begin();

Returns: {*this, std::ranges::begin(base_)}

constexpr auto reserve_hint() requires approximately_sized_range<V>;

Returns: The result is implementation-defined.

constexpr auto reserve_hint() const requires approximately_sized_range<const V>;

Returns: The result is implementation-defined.

25.7.?.6 Class to_utf_view::iterator [range.transcoding.iterator]

template<input_range V, to_utf_view_error_kind E, code-unit ToType>
  requires view<V> && code-unit<range_value_t<V>>
template<bool Const>
class to_utf_view<V, E, ToType>::iterator {
private:
  using Base = maybe-const<Const, V>; // exposition only

public:
  using iterator_concept = see below;
  using iterator_category = see below;  // not always present
  using value_type = conditional_t<E == to_utf_view_error_kind::expected, expected<ToType, utf_transcoding_error>, ToType>;
  using reference_type = value_type;
  using difference_type = ptrdiff_t;

private:
  iterator_t<Base> begin_{}; // exposition only, present only if
                                 //   bidirectional_range<Base> is true
  iterator_t<Base> current_{}; // exposition only
  sentinel_t<Base> end_; // exposition only

  inplace_vector<value_type, buffer-capacity> buf_{}; // exposition only

  ptrdiff_t buf_index_{}; // exposition only
  size_t to_increment_{}; // exposition only

  template<input_range V2, to_utf_view_error_kind E2, code-unit ToType2>
    requires view<V2> && code-unit<range_value_t<V2>>
  friend class to_utf_view; // exposition only

public:
  constexpr iterator() requires default_initializable<iterator_t<Base>> = default;

  constexpr iterator(iterator const&) requires copyable<iterator_t<Base>> = default;
  constexpr iterator& operator=(iterator const&) requires copyable<iterator_t<Base>> = default;
  constexpr iterator(iterator&&) = default;
  constexpr iterator& operator=(iterator&&) = default;

private:
  constexpr iterator(iterator_t<Base> begin, iterator_t<Base> current, sentinel_t<Base> end) // exposition only
    requires bidirectional_range<Base>
      : begin_(std::move(begin)), current_(std::move(current)), end_(end) {
    if (current_ != end())
      read();
  }

  constexpr iterator(iterator_t<Base> current, sentinel_t<Base> end) // exposition only
    requires (!bidirectional_range<Base>)
      : current_(std::move(current)), end_(end) {
    if (current_ != end())
      read();
    else if constexpr (!forward_range<Base>) {
      buf_index_ = -1;
    }
  }

public:
  constexpr iterator_t<Base> base() const
    requires forward_range<Base>;

  constexpr value_type operator*() const;

  constexpr iterator& operator++() requires (E == to_utf_view_error_kind::expected)
  {
    if (!success()) {
      if constexpr (is_same_v<ToType, char8_t>) {
        advance-one();
        advance-one();
      }
    }
    advance-one();
    return *this;
  }

  constexpr iterator& operator++() requires (E == to_utf_view_error_kind::replacement)
  {
    advance-one();
    return *this;
  }

  constexpr auto operator++(int) {
    if constexpr (is_same_v<iterator_concept, input_iterator_tag>) {
      ++*this;
    } else {
      auto retval = *this;
      ++*this;
      return retval;
    }
  }

  constexpr iterator& operator--() requires bidirectional_range<Base>
  {
    if (!buf_index_) {
      read-reverse();
    } else {
      --buf_index_;
      if constexpr (E == to_utf_view_error_kind::expected && is_same_v<ToType, char8_t>) {
        if (!success())
          buf_index_ -= 2;
      }
    }
    return *this;
  }

  constexpr iterator operator--(int) requires bidirectional_range<Base>
  {
    auto retval = *this;
    --*this;
    return retval;
  }

  friend constexpr bool operator==(const iterator& lhs, const iterator& rhs)
    requires equality_comparable<iterator_t<Base>>;

private:
  constexpr sentinel_t<Base> end() const { // exposition only
    return end_;
  }

  constexpr expected<void, utf_transcoding_error> success() const noexcept requires(E == to_utf_view_error_kind::expected); // exposition only

  constexpr void advance-one() // exposition only
  {
    ++buf_index_;
    if (buf_index_ == buf_.size()) {
      if constexpr (forward_range<Base>) {
        buf_index_ = 0;
        advance(current_, to_increment_);
      }
      if (current_ != end()) {
        read();
      } else if constexpr (!forward_range<Base>) {
        buf_index_ = -1;
      }
    }
  }

  constexpr void read(); // exposition only

  constexpr void read-reverse(); // exposition only
};

buffer-capacity is an unspecified constant that is at least 4 / sizeof(ToType). If Base does not model forward_range, buffer-capacity is 4 / sizeof(ToType).

to_utf_view::iterator::iterator_concept is defined as follows:

• If V models bidirectional_range, then iterator_concept is bidirectional_iterator_tag.
• Otherwise, if V models forward_range, then iterator_concept is forward_iterator_tag.
• Otherwise, iterator_concept is input_iterator_tag.

The member typedef-name iterator_category is defined if and only if V models forward_range.

In that case, to_utf_view::iterator::iterator_category is defined as follows:

• Let C denote the type iterator_traits<iterator_t<V>>::iterator_category.
• If C models derived_from<bidirectional_iterator_tag>, then iterator_category denotes bidirectional_iterator_tag.
• Otherwise, if C models derived_from<forward_iterator_tag>, then iterator_category denotes forward_iterator_tag.
• Otherwise, iterator_category denotes C.

constexpr iterator_t<Base> base() const
  requires forward_range<Base>;

Returns: If *this is at the end of the range being adapted, an iterator equal to the end of the range being adapted. Otherwise, an iterator pointing to the first code unit of the current input subsequence.

friend constexpr bool operator==(const iterator& lhs, const iterator& rhs)
  requires equality_comparable<iterator_t<Base>>;

Returns: If Base models forward_range, true if and only if either lhs and rhs are both at the end of the range being adapted, or the current input subsequences of lhs and rhs begin at the same position in the underlying range and *lhs and *rhs denote the code unit at the same offset within the transcoded code units of that input subsequence. Otherwise, lhs.current_ == rhs.current_ && lhs.buf_index_ == rhs.buf_index_.

constexpr value_type operator*() const;

Returns: Either buf_[buf_index_], or, if E is to_utf_view_error_kind::expected and !success(), then unexpected{success().error()}

constexpr expected<void, utf_transcoding_error> success() const noexcept requires(E == to_utf_view_error_kind::expected); // exposition only

Returns:

• If from-type is char8_t:

  • If the current input subsequence is a code unit between 0x80 and 0xBF, returns unexpected_utf8_continuation_byte.
  • If the current input subsequence is a code unit between 0xC0 and 0xC2, or between 0xF5 and 0xFF, returns invalid_utf8_leading_byte.
  • If the current input subsequence is the code unit 0xE0, and the subsequent input subsequence is a code unit between 0x80 and 0x9F; or if the current input subsequence is the code unit 0xF0, and the subsequent input subsequence is a code unit between 0x80 and 0x8F; then returns overlong.
  • If the current input subsequence is the code unit 0xED, and the subsequent input subsequence is a code unit between 0xA0 and 0xBF, then returns encoded_surrogate.
  • If the the current input subsequence is the code unit 0xF4, and the subsequent input subsequence is a code unit between 0x90 and 0xBF, then returns out_of_range.
  • Otherwise, if the current input subsequence is invalid UTF-8, begins with a code unit between 0xC2 and 0xF4, and there exists some hypothetical sequence of code units which would make the current input subsequence well-formed if concatenated to the end of it, then returns truncated_utf8_sequence.

• If from-type is char16_t:

  • If the current input subsequence is a code unit between 0xD800 and 0xDBFF, returns unpaired_high_surrogate.
  • If the current input subsequence is a code unit between 0xDC00 and 0xDFFF, returns unpaired_low_surrogate.

• If from-type is char32_t:

  • If the current input subsequence is between 0xD800 and 0xDFFF, returns encoded_surrogate.
  • If the current input subsequence is between 0x110000 and 0xFFFFFFFF, returns out_of_range.

Otherwise, returns expected<void, utf_transcoding_error>().

constexpr void read(); // exposition only

Effects:

Let n be a number of consecutive input subsequences, chosen by the implementation, such that:

• n >= 1;
• n does not exceed the number of input subsequences remaining in the underlying range, starting at position current_;
• the total number of code units produced by transcoding those n input subsequences as described below is at most buffer-capacity; and
• if forward_range<Base> is not modeled, n == 1.

Clears buf_. Then, for each of the n consecutive input subsequences starting at position current_, in order: decodes the input subsequence into a code point c, using the UTF encoding corresponding to from-type, setting c to U+FFFD if the input subsequence is ill-formed; and appends the code units of c, encoded in the UTF encoding corresponding to ToType, to buf_. Sets to_increment_ to the total number of code units comprising those n input subsequences, and sets buf_index_ to 0. If forward_range<Base> is modeled, current_ is set to the position it had before read was called.

constexpr void read-reverse(); // exposition only

Effects:

Let n be a number of consecutive input subsequences, chosen by the implementation, such that:

• n >= 1;
• n does not exceed the number of input subsequences in the underlying range preceding position current_; and
• the total number of code units produced by transcoding those n input subsequences as described below is at most buffer-capacity.

Clears buf_. Then, for each of the n consecutive input subsequences ending at position current_, in order: decodes the input subsequence into a code point c, using the UTF encoding corresponding to from-type, setting c to U+FFFD if the input subsequence is ill-formed; and appends the code units of c, encoded in the UTF encoding corresponding to ToType, to buf_. Sets to_increment_ to the total number of code units comprising those n input subsequences, and sets current_ to the position of the beginning of the first of those n input subsequences. Sets buf_index_ to the index in buf_ of the first code unit produced by transcoding the last of those n input subsequences if E is to_utf_view_error_kind::expected and that input subsequence is ill-formed, and to buf_.size() - 1 otherwise.

25.7.?.7 Class to_utf_view::sentinel [range.transcoding.sentinel]

template<input_range V, to_utf_view_error_kind E, code-unit ToType>
  requires view<V> && code-unit<range_value_t<V>>
template<bool Const>
struct to_utf_view<V, E, ToType>::sentinel {
private:
  using Base = maybe-const<Const, V>; // exposition only

  sentinel_t<Base> end_ = sentinel_t<Base>(); // exposition only

public:
  sentinel() = default;
  constexpr explicit sentinel(sentinel_t<Base> end) : end_{end} {}
  constexpr explicit sentinel(sentinel<!Const> i)
    requires Const && convertible_to<sentinel_t<V>, sentinel_t<Base>>
    : end_{i.end_} {}

  constexpr sentinel_t<Base> base() const { return end_; }

  template<bool OtherConst>
  requires sentinel_for<sentinel_t<Base>, iterator_t<maybe-const<OtherConst, V>>>
  friend constexpr bool operator==(const iterator<OtherConst>& x, const sentinel& y) {
    if constexpr (forward_range<Base>) {
      return x.current_ == y.end_;
    } else {
      return x.current_ == y.end_ && x.buf_index_ == -1;
    }
  }
};

9.4 Code unit adaptors

Add the following subclause to 25.7 [range.adaptors]:

25.7.? Code unit adaptors [range.codeunitadaptor]

template<class T>
struct implicit-cast-to { // exposition only/
  constexpr T operator()(auto x) const noexcept { return x; }
};

The names as_char, as_wchar_t, as_char8_t, as_char16_t, and as_char32_t denote range adaptor objects ([range.adaptor.object]). Let as_charT denote any one of as_char, as_wchar_t, as_char8_t, as_char16_t, and as_char32_t. Let Char be the corresponding character type for as_charT, let E be an expression and let T be remove_cvref_t<decltype((E))>. If ranges::range_reference_t<T> does not model convertible_to<Char>, or if T is an array, as_charT(E) is ill-formed. Otherwise, the expression as_charT(E) is expression-equivalent to:

• If T is a specialization of empty_view ([range.empty.view]), then empty_view<Char>{}.

• Otherwise, ranges::transform_view(std::views::all(E), implicit-cast-to<Char>{}).

[Example 1:

std::vector<int> path_as_ints = {U'C', U':', U'\x00010000'};
std::filesystem::path path = path_as_ints | as_char32_t | std::ranges::to<std::u32string>();
const auto& native_path = path.native();
if (native_path != std::wstring{L'C', L':', L'\xD800', L'\xDC00'}) {
  return false;
}

— end example]

9.5 Feature test macro

Add the following macro definition to 17.3.2 [version.syn], header <version> synopsis, with the value selected by the editor to reflect the date of adoption of this paper:

#define __cpp_lib_unicode_transcoding 20XXXXL // also in <ranges>

10 Design Discussion and Alternatives

10.1 CPO Rejection of String Literals

String literals are arrays of char types that include a null terminator:

static_assert(std::is_same_v<std::remove_reference_t<decltype("foo")>, const char[4]>);
static_assert(std::ranges::equal("foo", std::array{'f', 'o', 'o', '\0'}));

Because they are ranges, a naive implementation of the to_utfN CPO would result in null terminators in the output:

u8"foo" | to_utf32 | std::ranges::to<u32string>()
// results in a std::u32string of length 4 containing U'f', U'o', U'o', U'\0'

To avoid this situation, the to_utfN CPOs reject all inputs that are arrays of char, as do the as_charT casting CPOs.

10.2 The _or_error Views Are Basis Operations for Other Error Handling Behaviors

You can use the _or_error view to implement the same behavior that the non-std::expected-based views have.

For example, foo | std::views::to_utf8 has the same output as:

foo
  | std::views::to_utf8_or_error
  | std::views::transform(
      [](std::expected<char8_t, std::utf_transcoding_error> c)
        -> std::inplace_vector<char8_t, 3>
      {
        if (c.has_value()) {
          return {c.value()};
        } else {
          // U+FFFD
          return {u8'\xEF', u8'\xBF', u8'\xBD'};
        }
      })
  | std::views::join

You can also substitute a different replacement character by changing the result of the else clause, or add exception-based error handling by throwing at that point.

10.3 Why We Don’t Cache begin()

When we invoke begin(), constructing the transcoding iterator may read a bounded number of elements from the underlying view — up to four if it’s transcoding from UTF-8 and buffering a single code point, or up to the (constant) capacity of its internal buffer if it transcodes a chunk of input at a time. A previous revision of this paper implemented begin() caching, based on the idea that iterating the underlying range could have unbounded complexity.

However, Tim Song pointed to the wording in [iterator.requirements.general] stating that “All the categories of iterators require only those functions that are realizable for a given category in constant time (amortized).” This means that we should be making the assumption that the underlying iterator operations used by begin() are “amortized constant time” (in a hand-wavey sense). Tim also pointed out that transcoding from UTF is equivalent to views::adjacent<4>, which doesn’t cache.

Based on this reasoning, the transcoding views don’t cache begin().

10.4 Optimizing for Double-Transcoding

In generic code, it’s possible to introduce transcoding views that wrap other transcoding views:

void foo(std::ranges::view auto v) {
#ifdef _MSC_VER
  windows_function(v | std::views::to_utf16);
#endif
  // ...
}

int main(int, char const* argv[]) {
  foo(std::null_term(argv[1]) | std::views::as_char8_t | std::views::to_utf32);
}

In the above example, naively, foo would create a to_utf16_view wrapping a to_utf32_view. However, the to_utfN CPOs detect this situation and elide the to_utf32_view, creating the to_utf16_view so that it directly wraps the view produced by as_char8_t.

There’s precedent for this kind of approach in the views::reverse CPO, which simply gives back the original underlying view if it detects that it’s reversing another reverse_view.

10.5 .base_code_units()

10.5.1 Proposal

I received feedback that it could be useful to provide a .base_code_units() member function on the transcoding iterator which would give out a range of iterators from the underlying range delimiting the code units that make up the current code point.

Since we can’t give out iterators to the underlying range if it’s a (non-forward) input range, it’s also been suggested that in this case, .base_code_units() would still be available, but would give out iterators to a special cache that’s stored in the iterator.

To quote from a reflector email discussing this suggestion:

I think it would be useful to differentiate access to the (complete) underlying range vs access to the input code unit sequence for the current character. Obviously, access to the complete underlying range isn’t possible for input iterators, but access to the current input code unit sequence is (with the caching approach described above is). The iterators could expose this interface:

    // Forward+ iterators only; returns an iterator into the underlying range.
    constexpr const iterator_t<Base>& base() const & noexcept requires forward_range<Base> { ... }
    constexpr iterator_t<Base> base() && requires forward_range<Base> { ... }

    // Input+ iterators; returns a subrange containing the input code units for the current character.
    // References the input code unit sequence cache for input iterators.
    // References the underlying range otherwise.
    constexpr subrange<...> base_code_units() const noexcept { ... }

Unlike base(), base_code_units() would not necessarily contain iterators for the underlying range (e.g., in the case of a caching input iterator).

Note that the choice to provide .base_code_units() for input ranges affects ABI since the size of the transcoding iterator depends on whether it contains the cache.

10.5.2 Precedent

[P0244R2] provides transcoding iterators with a .base_range() member function that provide this range, although its input iterator functionality is implemented using special caching iterators that have shared ownership of a cache, instead of by storing the cached range in the iterator itself.

ICU provides multiple analogous APIs. The most directly comparable one is the .stringView() member function on the UnsafeCodeUnits transcoding iterator, which provides a std::basic_string_view containing the underlying code units for the current code point. UnsafeCodeUnits also provides .begin() and .end() member functions which give out the same range. Unlike the proposed .base_code_units() member function, neither of these APIs provide support for input iterators; .stringView() is only enabled when the base range is contiguous, and .begin() and .end() are only enabled if it’s a forward range.

10.5.3 Lifetime Issues

Here’s an example of a function where the use of .base_code_units() subtly introduces UB when the function is passed an input range.

This is a run-length-encoder that prints a count of the number of consecutive times it’s seen a code point, followed by the code units making up that code point:

void print_runs(std::ranges::range auto text) {
  auto utf_view = text | std::views::to_utf32;
  auto it = utf_view.begin();
  while (it != utf_view.end()) {
    auto units = it.base_code_units();
    char32_t code_point = *it;
    int count = 1;
    ++it;
    while (it != utf_view.end() && *it == code_point) {
      ++count;
      ++it;
    }
    std::print(
      "{}x{::#x} ", count,
      units | std::views::transform([](char8_t c) { return (std::uint8_t)c; } ));
  }
  std::println("");
}

When invoked with u8"ⒶⒶⒶⒷⒸ"sv, it prints:

3x[0xe2, 0x92, 0xb6] 1x[0xe2, 0x92, 0xb7] 1x[0xe2, 0x92, 0xb8]

When invoked with u8"ⒶⒶⒶⒷⒸ"sv | std::views::as_input, it invokes library undefined behavior and prints corrupted output. Worse, the UB here isn’t caught by AddressSanitizer or UndefinedBehaviorSanitizer because the invalidated auto units range points into the same, valid, transcoding iterator, whose cache simply contains the values for the subsequent code point, so the corrupted output is not automatically diagnosable.

With ICU’s APIs, on the other hand, this would fail to compile, because ICU only provides them for forward ranges.

I think this footgun would show up frequently.

In response, it was suggested that the above lifetime issue could be addressed by changing the return type of .base_code_units() to something like std::inplace_vector<char8_t, 4>.

That creates a different lifetime problem. Consider this example. The Unicode Tags block is intended for use in flag emojis but has been used for LLM prompt injections. Say a user writes the following function, which divides the stream of characters into Tags and non-Tags, and also imagine that they have a custom sink type that accepts iterator pairs rather than ranges:

constexpr bool is_tag(char32_t c) { return (c & ~0x7F) == 0xE0000; }

void partition_tags(std::ranges::range auto text, sink non_tags, sink tags) {
  auto utf_view = text | std::views::to_utf32;
  for (auto it = utf_view.begin(); it != utf_view.end(); ++it) {
    (is_tag(*it) ? tags : non_tags).consume(
      it.base_code_units().begin(), it.base_code_units().end());
  }
}

Again, this works perfectly well when partition_tags is passed a forward range, but then when it’s passed an input range, because each call to .base_code_units() returns a separate temporary std::inplace_vector, it.base_code_units().begin() and it.base_code_units().end() now point to different objects, so the function invokes UB.

10.5.4 Survey of Range Adaptors that Downgrade to Input

Some range adaptors downgrade forward ranges into input ranges: these are, to my understanding, views::as_input, views::cache_latest, views::join, and views::join_with.

[range.as.input.overview] states, “This is useful to avoid overhead that can be necessary to provide support for the operations needed for greater iterator strength.” This use case is potentially relevant for transcoding views, since the size of the iterator may be greater with a stronger iterator category. For example, bidirectional transcoding iterators need to store the begin iterator from the underlying range to avoid overrunning the beginning when transcoding backwards, but forward iterators don’t need it.

But implementing .base_code_units() for input views would actually cause views::as_input to increase the transcoding iterator’s overhead relative to its forward-iterator implementation, because the iterator would need to contain an additional code unit cache.

views::as_input was introduced by [P3725R3], “Filter View Extensions for Safer Use,” and, rather than avoiding overhead, its main motivation was composition with std::views::filter in order to avoid pitfalls related to mutating through a filter.

This is potentially relevant to transcoding, in that someone might write a filter-view pipeline on characters. Say a user wants to print the UTF-8 code units for all the non-ASCII code points in a range. That would look like this:

void print_nonascii_code_points_and_code_units(std::ranges::range auto text) {
  auto print_code_point{
    [](char32_t code_point, auto code_unit_range) {
    std::println(
      "{:#x} = {::#x}", static_cast<std::uint32_t>(code_point),
      code_unit_range | std::views::transform([](char8_t c) { return (std::uint8_t)c; }));
    }};
  auto code_points = text
                     | std::views::filter([](char8_t c) { return c >= 0x80; })
                     | std::views::to_utf32;
  for (auto it = code_points.begin(); it != code_points.end(); ++it) {
    print_code_point(*it, it.base_code_units());
  }
}

A user following the [P3725R3] guidance might insert a views::as_input adaptor into the pipeline before std::views::filter, which would continue to compile and work if we provided .base_code_units() for input ranges, but which would cause print_nonascii_code_points_and_code_units to fail to compile if we didn’t.

But views::as_input isn’t strictly necessary here. And we already need to teach users that inserting views::as_input before std::views::filter will, in rare cases, cause some uses of .base() to fail to compile. To demonstrate why this isn’t a novelty, consider the following example:

struct Task { int priority; };

bool submit_batch(std::ranges::range auto batch);

// Submit the high-priority tasks in batches; on a transient failure, hand the
// remaining high-priority tasks to the retry queue.
void submit_high_priority_tasks(std::vector<Task>& tasks) {
  auto high = tasks | std::views::filter([](Task const& t) { return t.priority > 100; });
  auto batches = high | std::views::chunk(BATCH_SIZE);
  for (auto it = batches.begin(); it != batches.end(); ++it) {
    if (!submit_batch(*it)) {
      requeue(std::ranges::subrange(it.base(), high.end()));
      return;
    }
  }
}

This works as written, but if views::as_input is inserted in front of views::filter, the call to it.base() fails to compile because std::ranges::chunk_view’s iterator doesn’t provide .base() for input views. But views::as_input is unnecessary here as well.

Furthermore, it’s worth noting that the list of plausible reasons to apply a filter_view on code units as opposed to code points is extremely short; ordinarily, doing so risks corrupting the output.

Moving on to views::cache_latest: that one is an adaptor with a niche use case and no special relevance to transcoding.

views::join is directly relevant, since it’s common for users to want to reassemble a text string that had previously been broken up into separate parts before transcoding it. views::join_with is also potentially relevant, since users may want to transcode text after having used views::join_with to add separators to it.

It’s important to note that in the common case, views::join and views::join_with do not downgrade from forward to input. They only do so if the range of ranges it’s given is a range of prvalue ranges.

For example, the views::join adaptor in the following example does not downgrade:

void print_errors(std::ranges::range auto packets) {
  auto print_code_units =
    [](std::ranges::range auto code_unit) {
      std::println("{::#x}",
                   code_unit
                   | std::views::transform([](char8_t c) { return (std::uint8_t)c; }));
    };
  auto utf_view = packets
                | std::views::join
                | std::views::to_utf32_or_error;
  for (auto it = utf_view.begin(); it != utf_view.end(); ++it) {
    if (!(*it).has_value()) {
      print_code_units(it.base_code_units());
    }
  }
}

But, if the packets need to be decrypted before transcoding, and the user alters the pipeline like so:

+ std::u8string decrypt(std::u8string_view packet) {
+   return packet
+          | std::views::transform(
+              [](char8_t c) {
+                return static_cast<char8_t>(c ^ 0x55);
+              })
+          | std::ranges::to<std::u8string>();
+ }

void print_errors(std::ranges::range auto packets) {
  auto print_code_units =
    [](std::ranges::range auto code_unit) {
      std::println("{::#x}",
                   code_unit
                   | std::views::transform([](char8_t c) { return (std::uint8_t)c; }));
    };
  auto utf_view = packets
+               | std::views::transform(decrypt)
                | std::views::join
                | std::views::to_utf32_or_error;
  for (auto it = utf_view.begin(); it != utf_view.end(); ++it) {
    if (!(*it).has_value()) {
      print_code_units(it.base_code_units());
    }
  }
}

Then it downgrades.

10.5.5 Alternatives to .base_code_units() for Users

10.5.5.1 Forward Ranges

For forward ranges, it.base_code_units() is equivalent to std::ranges::subrange(it.base(), std::ranges::next(it).base()).

The expression above raised concerns about the fact that its use in a loop would mean performing two operator++ operations on every loop iteration, but that can be mitigated by simply caching the previous iterator while iterating forwards:

auto prev_base = it.base();
++it;
auto code_units = std::ranges::subrange(prev_base, it.base());

10.5.5.2 Input Ranges

For input ranges, since the transcoding view doesn’t provide .base(), the workaround involves making a copy of the input range in order to get a forward range.

10.5.5.2.1 Copying the entire range

If it’s viable to copy the entire range, you can simply insert a std::ranges::to<std::u8string>() into the range pipeline.

void print_utf8_code_points_and_code_units(std::ranges::range auto text) {
  auto print_code_point{
    [](char32_t code_point, auto code_unit_range) {
    std::println(
      "{:#x} = {::#x}", static_cast<std::uint32_t>(code_point),
      code_unit_range | std::views::transform([](char8_t c) { return (std::uint8_t)c; }));
    }};
  auto code_points = text
                     | std::ranges::to<std::u8string>()
                     | std::views::to_utf32;
  for (auto it = code_points.begin(); it != code_points.end(); ++it) {
    print_code_point(*it, std::ranges::subrange(it.base(), std::ranges::next(it).base()));
  }
}

When invoked with u8"AΩ€😀b"sv | std::views::as_input, this prints:

0x41 = [0x41]
0x3a9 = [0xce, 0xa9]
0x20ac = [0xe2, 0x82, 0xac]
0x1f600 = [0xf0, 0x9f, 0x98, 0x80]
0x62 = [0x62]

10.5.5.2.2 Avoiding copying

Although it requires rolling your own segmentation, it is possible to iterate over an input view’s code unit subsequences with additional refactoring:

constexpr bool is_utf8_continuation(char8_t c) { return (c & 0xC0) == 0x80; }

void print_utf16_and_utf8_code_units_per_code_point(std::ranges::range auto text) {
  auto print_code_point{
    [](auto u16_view, auto u8_view) {
      std::println(
        "{::#x} = {::#x}",
        u16_view | std::views::transform([](char16_t c) { return (std::uint16_t)c; }),
        u8_view | std::views::transform([](char8_t c) { return (std::uint8_t)c; }));
    }};
  auto it = text.begin();
  std::u8string code_point;
  while (it != text.end()) {
    code_point.clear();
    code_point.push_back(*it);
    ++it;
    it = std::ranges::find_if(std::move(it), text.end(), [&](char8_t c) {
      if (!is_utf8_continuation(c)) return true;
      code_point.push_back(c);
      return false;
    });
    print_code_point(code_point | std::views::to_utf16, code_point);
  }
}

When invoked with u8"AΩ€😀b"sv | std::views::as_input, this prints:

[0x41] = [0x41]
[0x3a9] = [0xce, 0xa9]
[0x20ac] = [0xe2, 0x82, 0xac]
[0xd83d, 0xde00] = [0xf0, 0x9f, 0x98, 0x80]
[0x62] = [0x62]

Clearly, this isn’t an ideal user experience, but it only applies to users who have an input range that is too large to copy, and also need to access the underlying code unit sequence for a code point. In my opinion, preserving the ergonomics of that use case is not worth the tradeoff of introducing the safety footgun demonstrated by the RLE example above.

10.5.6 Implementation

An experimental implementation of .base_code_units() is available on the enolan_basecodeunits2 branch of beman.utf_view.

10.6 SIMD support

The wording has been updated to allow implementations to read code points in chunks rather than one at a time, which enables support for SIMD.

To my knowledge, this would be the first view that reads its input in chunks for reasons of performance rather than correctness.

The transcoding iterator must be increased in size in order to fit a larger buffer which contains the output of the SIMD transcoding kernel. On the other hand, there is no need to store the input buffer in the iterator.

The decision of which output buffer size to select is left up to the implementation, but once chosen, an implementation can’t change it without breaking ABI. (This also implies that, as a process note, it’s not possible for us to standardize a version of this facility that doesn’t allow SIMD and then patch SIMD support onto it in a future paper.)

The invariant of .base() is that it points to the beginning of the code unit range for the current code point in the underlying view. Transcoding more than one code point at a time slightly complicates the implementation of .base() relative to the scalar implementation. For example, an implementation might want to store an iterator to the beginning of the current chunk in the underlying view and then, when .base() is invoked, iterate it forward from the beginning of the chunk to the start of the current code point. This means that, whereas the scalar implementation of .base() is a simple accessor to a data member, the chunked implementation may need to perform CHUNK_SIZE iterator increments internally (which is still O(1)).

Since it’s challenging to implement fully conformant Substitution of Maximal Subparts error handling in a SIMD transcoding kernel, we expect that implementers will add a validation step to their SIMD transcoding kernels and fall back to the serial implementation on invalid UTF. To fully benefit from the fast path, you need valid input.

SIMD support is enabled for forward ranges only. Here is an example of why the chunking behavior breaks input ranges. Say we have a video game that reads the player’s name but only allows space for five code points:

std::u32string get_player_name() {
  std::ranges::subrange input_view(
      std::istreambuf_iterator<char>(std::cin),
      std::istreambuf_iterator<char>{});
  // get 5 code points of input
  return input_view
         | beman::utf_view::as_char8_t
         | beman::utf_view::to_utf32
         | std::views::take(5)
         | std::ranges::to<std::u32string>();
}

(This is actually somewhat more complicated to do than depicted here.)

With a chunked implementation, the user types “GAM3R” but then needs to keep typing until the chunk gets filled up, only for the rest of the chunk to get discarded.

The reference implementation’s benchmark transcodes the unicode_lipsum corpora from UTF-16 to UTF-8. The SIMD implementation uses a prototype kernel written against C++26 std::simd and a buffer capacity of 64 code units. For comparison, the last column is a single bulk simdutf call over the whole corpus, with no view involved. Numbers are GiB/s of input consumed (GCC 16.1, -O3 -march=native, x86-64 AVX2, AMD Ryzen 9 5950X).

In the current prototype, we see >2x speedup on the most favorable case, which is ASCII input, and either parity or small speedups on other corpuses. (Note that the prototype is currently in an incomplete state, and only implements the UTF16-UTF8 direction, so results with other directions may differ). 64 is the minimum buffer size at which we get favorable results, for this particular transcoding direction, in my prototype. simdutf smokes us, mainly due to benefiting from the bulk API, and we can’t approach its speed with a view; we would need to do an algorithm to achieve comparable performance.*

To put these numbers into perspective, the article text of English Wikipedia is roughly 40 GiB in UTF-8, or about 80 GiB in UTF-16, and so would take approximately 43 seconds to transcode on a single core with the scalar implementation and 19 seconds to transcode with SIMD, assuming its properties are roughly similar to the Latin corpus above.

* (This is because a view delivers its output one code unit at a time: every element costs an iterator increment, a dereference, and a buffer-index check, no matter how cheaply the buffer was filled. SIMD accelerates only the buffer refill; once that cost is amortized away, throughput is bounded by the per-element delivery loop — on the Latin row above, the SIMD view is already running at about two cycles per code unit, which is the cost of the loop itself rather than of transcoding. A bulk API has no per-element step at all: it reads and writes entire vectors. Closing the gap therefore requires an interface that writes directly to an output range, not a faster kernel inside the view.)

11 Changelog

11.1 Changes since R13

• Fix typo in Table 3-9 of error handling diagram pointed out during SG16 review (ED A0 had been misspelled as E0 A0).
• Add design discussion about .base_code_units().
• Loosen the wording to admit chunked (e.g. SIMD) implementations when the underlying range models forward_range: make buf_’s capacity an unspecified constant, allow read and read-reverse to transcode several input subsequences per invocation, and respecify base() and iterator equality positionally instead of in terms of the exposition-only members. Make operator-- skip an ill-formed input subsequence’s replacement-character code units as a unit in _or_error views, mirroring operator++, since with chunking they can now appear in the middle of buf_. Specify explicitly that read-reverse leaves current_ at the beginning of the first input subsequence it transcoded. Add corresponding design discussion.

11.2 Changes since R12

• Add example: transcoding into a fixed-size buffer without truncating code points.
• Add example: using to_utf32 and base() for code point-aware substring replacement on cuneiform text, adapted from an ICU unit test.
• Give utf_transcoding_error and to_utf_view_error_kind unspecified underlying types, following the precedent of memory_order, launch, and assertion_kind. Fixing the underlying type (to int or bool) is an unnecessary constraint; an unspecified underlying type gives implementations freedom to choose a type suited to their layout and ABI needs.
• Remove base() from the transcoding iterator when the base range is an input range and not a forward range.
• Add views::as_char and views::as_wchar_t; it’s also useful to get UTF-encoded strings from charN_t types back into char/wchar_t as well as having the reverse direction.
• Fix subclause numbers in wording sections to use 25.7 consistently, matching the current working paper’s numbering for [range.adaptors].
• Add wording for changes to the <ranges> header synopsis in [ranges.syn].
• Expand “CTP” abbreviation to “constant template parameter” in [range.transcoding.overview].
• Add missing “Otherwise” before expression-equivalent clauses in [range.transcoding.overview] and [range.codeunitadaptor].

11.3 Changes since R11

• Refactor use of std::nontype to use std::constant_wrapper instead per [P3948R1].
• Remove back-pointer to parent view in iterator, restoring R7’s three-iterator design and borrowedness, based on precedent from views::adjacent<N>; and based on the fact that, since R10 implements the double-transcode optimization in the CPO, we no longer need the innermost-iter machinery that overcomplicated the R7 design.

11.4 Changes since R10

• Fix the wording around rejecting arrays for the code unit adaptors
• Add example for handling byte offsets and endianness

11.5 Changes since R9

• Replace bool OrError CTP with to_utf_view_error_kind enum class like ranges::subrange_kind
• Replace to_utfX_view classes with a single to_utf_view class with annoying constructor tags to make CTAD work, per SG9 feedback from Kona 2025, and remove related obsolete design discussion
• Fix a bug where the value type of empty_view was not set properly in the CPO when using an _or_error CPO
• Remove section stating that the concept of a CPO template is a “novelty” after it was pointed out in Kona 2025 that it has precedent in views::adjacent_transform<N> and elsewhere
• Don’t cache begin()
• Reject arrays of charN_t in the CPOs
• Implement double-transcode optimization in the CPOs

11.6 Changes since R8

• Fix const correctness bug in wording relating to noncopyable input iterators
• Use a std::inplace_vector instead of a std::array for buf_, allowing us to eliminate buf_last_ exposition-only member
• Simplify the implementation of to-utf-view-impl’s operators
• Rename utf-iterator to to-utf-view-impl::iterator
• Add a to-utf-view-impl::sentinel type
• Cache begin()
• Clean up wording
• Add reserve_hint() member functions
• Fix bug in operator== for input iterators
• Add additional design discussion
• Add size() member function when transcoding from and to UTF-32

11.7 Changes since R7

• Add playing card example.
• Remove iterator_interface from utf-iterator.
• Replace code unit views with range adaptor closure objects that are expression-equivalent to P3117 transform_view.
• Move null_sentinel and null_term into P3705.
• Remove std::uc namespace and replace it with std::ranges and std::ranges::views.
• Remove library EB.
• Change iterator to use a back-pointer to its parent view, removing borrowedness and quadratic stack size growth.
• Remove support for char and wchar_t.
• Rewrite most of the verbiage and add new diagrams.

11.8 Changes since R6

• Fix a bug in null_sentinel_t causing it not to satisfy sentinel_for by changing its operator== to return bool.
• Fix a bug in null_sentinel_t where it did not support non-copyable input iterators by having operator== take input iterators by reference.
• Rename as_utfN to to_utfN to emphasize that a conversion is taking place and to contrast with the code unit views, which remain named as_charN_t.
• Refactor utf_view into an exposition-only utf-view-impl class used as an implementation detail of separate to_utf8_view, to_utf16_view, and to_utf32_view classes, addressing broken deduction guides in the previous revision.
• Remove project_view and copy most of its implementation into separate char8_view, char16_view, and char32_view classes, addressing broken deduction guides in the previous revision.
• Change utf_iterator to an exposition-only member class of utf-view-impl.
• Eliminate iterator unpacking mechanism and replace it with an alternative solution to the problem of transcoding ranges wrapping other transcoding ranges. This simplifies the API at the expense of removing the transcoding iterator’s begin() and end() member functions and losing the ability to implement unpacking for user-defined UTF iterators.
• Remove std::uc::format.
• Make all concepts exposition-only.
• Remove utf_transcoding_error_handler mechanism.
• Introduce new error handling mechanism based on a new utf_transcoding_error enumeration which is returned by an success() member function of the transcoding view’s iterator.
• Remove ability to pass pointers to range adaptor closure objects, which violated the restriction that only ranges may be passed to range adaptor closure objects.
• Remove std::format and std::ostream functionality. It doesn’t make sense for this mechanism to be the only way we have to format/output char8_t; we can revisit this functionality when we have already figured out how to support e.g. std::u8string.
• Replace code examples with new ones reflecting API changes.
• Provide a reference implementation.

11.9 Changes since R5

• Simplify the complicated constraint on the comparison operator for null_sentinel_t.
• Introduce ranges::project_view, and implement charN_views in terms of that.
• Convert the utfN_views to aliases, rather than individual classes.

11.10 Changes since R4

• Replace unpacking_owning_view with unpacking_view, and use it to do unpacking, rather than sometimes doing the unpacking in the adaptor.
• Ensure const and non-const overloads for begin and end in all views.
• Move null_sentinel_t to std, remove its base member function, and make it useful for more than just pointers, based on SG-9 guidance.

11.11 Changes since R3

• Changed the definition of the code_unit concept, and added as_charN_t adaptors.
• Removed the utility functions and Unicode-related constants, except replacement_character.
• Changed the constraint on utf_iterator slightly.
• Change null_sentinel_t back to being Unicode-specific.

11.12 Changes since R2

• Add noexcept where appropriate.
• Remove non-essential constants and utility functions, and elaborate on the usage of the ones that remain.
• Note differences from similar elements proposed in [P1629R1].
• Extend the examples slightly.
• Correct an error in the description of the view adaptors’ semantics, and provide several examples of their use.

11.13 Changes since R1

• Reintroduce the transcoding-from-a-buffer example.
• Generalize null_sentinel_t to a non-Unicode-specific facility.
• In utility functions that search for ill-formed encoding, take a range argument instead of a pair of iterator arguments.
• Replace utf{8,16,32}_view with a single utf_view.

11.14 Changes since R0

• When naming code points in interfaces, use char32_t.
• When naming code units in interfaces, use charN_t.
• Remove each eager algorithm, leaving in its corresponding view.
• Remove all the output iterators.
• Change template parameters to utfN_view to the types of the from-range, instead of the types of the transcoding iterators used to implement the view.
• Remove all make-functions.
• Replace the misbegotten as_utfN() functions with the as_utfN view adaptors that should have been there all along.
• Add missing utf_transcoding_error_handler concept.
• Turn unpack_iterator_and_sentinel into a CPO.
• Lower the UTF iterator concepts from bidirectional to input.

12 Relevant Polls/Minutes

12.1 SG16 review of P2728R13 on 2026-05-27 (Telecon)

• Minutes

No polls were taken during this review.

12.2 SG16 review of P2728R12 on 2026-05-13 (Telecon)

• Minutes

No polls were taken during this review.

12.3 SG9 review of P2728R9 on 2025-11-06 during Kona 2025

• Minutes

POLL: We want to remove the null terminator of char arrays (if present) so that u8"abc" | to_utf32 (and to_utf8, to_utf16) do not include the null terminator in the resulting output.

Attendance: 10 (2 abstentions)

Author Position: A, F

Outcome: Consensus against

POLL: We want to ban char arrays as input to to_utfX to prevent accidental inclusion of the null terminator of a string literal.

Attendance: 10 (2 abstentions)

Author Position: F, SF

Outcome: Strong consensus in favor

ACTION ITEM: Figure out whether we need to cache begin().

POLL: Simplify the to_utfX_view classes by just having a single templated view class similar to scan_view (with potentially annoying constructor tags to make CTAD work) and focus on usability with the CPOs only.

Attendance: 9 (0 abstentions)

Author Position: N

Outcome: Consensus in favor

POLL: We want to optimize nested to_utf_views.

Attendance: 9 (0 abstentions)

Author Position: N

Outcome: Consensus in favor

ACTION ITEM: Come up with more examples where nested to_utf_views occur in practice to explore whether a designated CPO approach is feasible.

12.4 Unofficial SG9 review of P2728R7 during Wrocław 2024

SG9 members provided unofficial guidance that the .success() member function on the utf-iterator wasn’t workable and encouraged providing views with std::expected as a value type.

12.5 SG16 review of P2728R6 on 2023-09-13 (Telecon)

• Minutes

No polls were taken during this review.

12.6 SG16 review of P2728R6 on 2023-08-23 (Telecon)

• Minutes

No polls were taken during this review.

12.7 SG9 review of D2728R4 on 2023-06-12 during Varna 2023

• Minutes

POLL: utf_iterator should be a separate type and not nested within utf_view

Attendance: 8 (3 abstentions)

# of Authors: 1

Author Position: F

Outcome: Weak consensus in favor

SA: Having a separate type complexifies the API

12.8 SG16 review of P2728R0 on 2023-04-12 (Telecon)

• Minutes

POLL: SG16 would like to see a version of P2728 without eager algorithms.

Attendance: 10 (3 abstentions)

Outcome: Consensus in favor

POLL: UTF transcoding interfaces provided by the C++ standard library should operate on charN_t types, with support for other types provided by adapters, possibly with a special case for char and wchar_t when their associated literal encodings are UTF.

Attendance: 9 (2 abstentions)

Outcome: Strong consensus in favor

Author’s note: More commentary on this poll is provided in the section “Discussion of whether transcoding views should accept ranges of char and wchar_t”. But note here that the authors doubt the viability of “a special case for char and wchar_t when their associated literal encodings are UTF”, since making the evaluation of a concept change based on the literal encoding seems like a flaky move; the literal encoding can change TU to TU.

12.9 SG16 review of P2728R0 on 2023-03-22 (Telecon)

• Minutes

No polls were taken during this review.

POLL: char32_t should be used as the Unicode code point type within the C++ standard library implementations of Unicode algorithms.

Attendance: 9 (2 abstentions)

Outcome: Strong consensus in favor

13 Special Thanks

Zach Laine, for writing revisions one through six of the paper and implementing Boost.Text.

Jonathan Wakely, for implementing P2728R6, and design guidance.

Robert Leahy and Gašper Ažman, for design guidance.

The Beman Project, for helping support the reference implementation.

14 References