No Endian Views

Contents

1. Introduction

[P4030R1] Endian Views introduces Endianness adaptors std::views::from_little_endian, std::views::to_­little_endian, std::views::from_big_endian, and std::views::to_big_endian. I argue that these are not suitable for standardization because they are unfit for the intended use case, for the reasons described below.

1.1. Endian Views are of limited use to serialization

1.1.1. The serialization pipeline

Serialization and deserialization is a pipeline with several steps. Let's take deserialization of UTF-32 code points for example:

CONSUME       →  REINTERPRET  →  ADJUST ENDIAN
std::byte[4]     uint32_t        uint32_t

The key problem with Endian Views is that they are intended to assist in this pipeline, but they only cover the last transformation step, i.e. ADJUST ENDIAN, even though these steps are virtually always performed in one go.

1.1.2. Contrived motivation

[P4030R1] §2 Before/After Tables contains a motivating example involving

constexpr vector<uint32_t> utf16be_to_utf32be( const vector<uint16_t>& utf16be_data) { /* ... */ }

This signature is contrived because it doesn't make sense for the user to hold a vector<uint16_t> where the Endianness of the elements is incorrect (at least not if Endian Views are used); nothing can be meaningfully done with that data, other than

• shooting yourself in the foot by forgetting that the data is not usable due to having the wrong Endianness yet,
• making it usable by swapping Endianness (but why didn't you do that sooner?), or
• serializing the data without Endian swap, but then Endian Views were neither used for deserialization nor serialization.

The comparison table the paper should be showing is:

… where endian_swap can be trivially implemented by the user as:

This comparison table makes a few things readily apparent:

• Endian Views provide incredibly low value because they are just a shorthand for wrapping a trivial-to-implement utility function in views::transform.

  There is one minor exception to this: if the original data is contiguous and the Endianness conversion is a no-op, the result is also contiguous, whereas transform_view never produces a contiguous range.

• Endian Views address just a small part of the pipeline; the benefits are marginal if not more of the pipeline is covered by the feature.
• Both before and after, the code expresses the overall idea of loading Big-Endian data in several pieces like chunking, reinterpretation, and Endian swapping. Neither side is expressive.

A significant improvement over the left column would look like:

1.1.3. Problems with range checks

However, both with this version and all the previous ones, there is an issue with needing to check each iteration over a char32_t element whether at least four bytes remain in the range. If we didn't use views whatsoever, we could easily check whether the amount of bytes is a multiple of four once in the function, and perform the conversion in bulk unchecked for the remainder of the function.

Whether the compiler can factor out this range check is questionable, and it could significantly worsen performance of using views if it cannot. [P4030R1] does not investigate this problem.

1.2. Endian Views are a pessimization

Another major problem with Endian Views is that their intended usage is pessimized. Consider a typical use case of serializing a span<const char32_t> as UTF-32LE in a file:

Due to how widespread Little-Endian architectures are, the else branch can also arguably be omitted by most users in favor of a two-line function, where one line is:

Endian Views provide negative value for this use case because not only would it require a lot more effort to use them here, they would also drop the contiguity of the data when transforming from char32_t to an array<byte, 4>, despite the fact that Endian Views try to preserve contiguity.

Curiously, the terms performance, optimization, or throughput, do not appear once in [P4030R1] as if this subject were not worth discussing. This would not be that big of a problem

• if encoding of text wasn't a primary advertised use case (where we should at least care about our low-level utilities performing well), and
• if the baseline wasn't a single call to std::fwrite (which Endian Views can only perform worse than; the question is just how much worse).

1.3. Endian Views work poorly for floating-point

The design of Endian Views is centered around integral types; from_little_endian and other view types can only be used if the element type of the transformed range is integral. However, serialization is perfectly reasonable for other types, such as std::float32_t. For example, the STL file format is essentially a sequence of 32-bit floating-point values.

With the current design, the user would have to add another two steps to the pipeline which bit-cast std::float32_t to std::uint32_t (and back) to be able to use Endian Views. This seems hostile and makes the feature even less attractive than it already is.

Even if this problem were resolved by making Endian Views accept anything trivially copyable, the more pressing issue is that not every floating-point bit pattern is valid for the type, and thus some may result in undefined behavior when returned from a function. This gives us a few options:

• Accept that Endian Views for float can sometimes result in undefined behavior. Deal with it 😎
• Don't accept floating-point types ([P4030R1]).
• Accept only those floating-point types where byte-swapping is always well-defined.
• Don't try to hold invalid floating-point values in the first place. That is, use the views::from_bytes<float, endian::big> approach mentioned above.

The first option reduces safety, the second option is unergonomic, and the third option is clunky and makes float serialization non-portable.

The last option doesn't have any of these problems, and it demonstrates that [P4030R1] artificially creates a problem that cannot be solved without unpleasant trade-offs.

2. Proposed action

Do not pursue the design of [P4030R1].

If the design is pursued, [P4030R1] needs substantial work:

• Provide proper design discussion of a view that converts directly from bytes versus a view that only does the Endian swap. Just because one can be composed from the other doesn't mean the smaller (Endian swap) utility is automatically better, just like composing std::fwrite from std::putchar isn't a good idea, even though it may be academically satisfying.
• Provide proper discussion of performance considerations. Currently, there is no mention of performance whatsoever.
• Discuss the intended use case and the existing practice in the targeted domains. That is, do established C++ libraries actually use views for their serialization purposes, and do those views use an Endian swap in the pipeline? Is that technique a creative invention of [P4030R1] or already established?
• Discuss how much benefit Endian Views provide as compared to wrapping a simple utility function in views::transform. The paper currently does not even consider the possibility of a std::from_big_endian utility function rather than views::from_big_endian.

That doesn't mean that the overarching problem of serialization and deserialization with Endian transformation is not worth solving. It is worth solving, but not using Endian Views.

3. References