Design problems with P2964R5 (User-defined element types in std::simd)

Contents

1. Introduction

[P2964R5] enables the use of user-defined types in std::simd::basic_vec. While this idea is useful in principle, there are some crucial design problems that are either addressed the wrong way or not discussed whatsoever.

2. Opt-in vs. opt-out

One critical question is whether support for simd::vec<T> should be opt-in or opt-out. The paper currently makes it opt-out, while I suggest to make it opt-in.

2.1. False portrayal of vec<T> as an opt-in

§5.1 Naming vec<T> is already an opt-in makes the claim:

No type is vectorized inadvertently. A type enters simd only when a programmer writes vec<T> or basic_vec<T, Abi> in their own source. That act is itself the opt-in: the user has explicitly asked for the vectorized form of T.

It is true that the user opts into using vec<T> by writing vec<T>. However, this opt-in is totally irrelevant to the discussion. We care about whether library authors opt into vec<T> being usable for a type T they define.

The user also opts into details of overload sets by taking a function pointer or by reflecting on library declarations. By the author's logic, it is perfectly fine for all functions in the standard library to be addressable because the user has opted into taking function pointers when writing '&' in &std::sqrtf.

2.2. Structure of arrays

[P2964R5] has a motivational example in §3.5 Compound Types:

Small compound types that fit in 16 bytes can be vectorized as atomic units, enabling structure-of-arrays patterns, or packet processing of multiple values simultaneously.

// Coordinate pairs vec<std::pair<int, int>> coordinates;

This raises the question whether simd::vec itself should have structure-of-arrays layout. The key issue is that for some types, the better layout for SIMD is structure-of-arrays. Making vec<T> have structure-of-arrays layout could be especially sensible for heterogeneous structures, such as a struct number_with_unit { float; enum; }, where it may be much better to perform parallel operations on all the floats and then on all enums, rather than interleaving them.

Only the library author can effectively decide whether for their type T, vec<T> should be a structure-of-arrays or array-of-structures, and this decision is case-by-case. Making that decision after users have started using vec<T> is an ABI break.

The dilemma of structure-of-arrays layout is also somewhat of an argument against [P2964R5] as a whole: while a struct { float; int; } may be eligible for use in simd::vec, the elementwise operations may perform much worse than operations on struct { vec<float>; vec<int>; }. More generally, it may be best in most cases to break everything up into simd::vec of fundamental types, rather than putting large, complex types into simd::vec.

Due to these performance considerations, the library author's position may be

Don't ever put this type into simd::vec. I don't want to support this, and it performs badly anyway.

This position can be hard to take retroactively when users have already begun putting the type into simd::vec and the mechanism is opt-out. A key insight is that with an opt-in mechanism, the author decides how best to use their types with SIMD — the expert makes a conscious decision. With an opt-out mechanism, the user makes that decision irrespective of the library author's opinion, and it's not clear whether the author even has a position on this because an opt-out design doesn't require taking any position (which may prove to be a mistake when it's already too late).

2.3. Bit-packed and byte-packed types

For some types, such as unsigned _BitInt(1) (currently a Clang extension and C23 feature that for 1-bit integers), the layout provided by vec<unsigned _BitInt(1)> without special-casing would be the same as for an unsigned _BitInt(1)[N]; that is, one bit of value per byte.

There are several issues with this:

• simd::basic_vec now needs to handle padding bits properly.
• A bit-packed layout may be better, especially when vec<unsigned _BitInt(1)> is used for its bitwise operations (&, |, etc.). In that case, any operation is doing pointless work on 7 padding bits.
• unsigned _BitInt(1) satisfies the requirements for types to implicitly opt into vec<unsigned _BitInt(1)>, so changing the layout to be bit-packed later is an ABI break.

While unsigned _BitInt(1) can be special-cased by the standard library implementation, this is not something that the user, a third-party library, or anyone else who is not the implementation can do. §7 Customization Points provides no form of layout customization. If there later arrived a customization mechanism for making vec<unsigned _BitInt(1)> bit-packed, that would constitute an ABI break if utilized.

Similarly, there can be types that are larger than the information they carry. For example, scoped enumerations by default have an underlying type of int (usually 4 bytes large), but often only some of the enumerators are actually relevant, so the effective value range may fit into 8 bits or less. [P2964R5] makes any such enumerations implicitly opt into the vec<int32_t> layout, and changing that to be a vec<unsigned char> layout later is an ABI break.

2.4. Elevating ABI breaks to API breaks

[P2964R5] also makes it so that by customization being opt-out, ABI breaks are sometimes elevated to API breaks.

For example, if the user defines a type

… where the alignas(4) might exist for performance reasons, later removing that alignas(4) is an API break of vec<rgb>. This happens because [P2964R5] requires the size of a type to be a power of two, and the size is changed from 4 to 3, so any uses of vec<rgb> become ill-formed.

While breaking ABI is often fine in template-only/header-only libraries, making existing user code suddenly ill-formed is never fine. [P2964R5] imposes the burden of opting out of simd::vec support for any defined type on the library author, rather than letting them retain control by default.

[P2964R5] has the following to conclude about this situation:

We believe the consequence is nonetheless acceptable. The break is diagnosed at compile time, which is preferable to the silent run-time mismatch that vector<rgb> produces, since at least the problem is flagged. Also, it arises only for types whose size may change, where the author is the party best placed to know whether that is likely and to use disable_vectorization accordingly.

The problem with this line of reasoning is that continuously adding mechanisms that library authors have to consider opting out of is obviously unsustainable. Imagine if the library author had to

• opt out of operator== support for every struct because they may change the behavior later,
• opt out of constexpr support for every function because they may do some runtime work in it later, even if the function doesn't make runtime-only calls now,
• opt out of noexcept support for every function because they may want to throw exceptions in the future, even if they don't do it now,
• …

Opt-out mechanisms should be reserved for where opt-out is extremely unlikely and where authors are likely aware of how and when to opt out. Neither of these conditions appears to be satisfied because

• structure-of-arrays layout, bit-packed layout, avoiding future API breaks, and likely other reasons commonly motivate opting out, and
• it's unlikely that every library author who defines an enum or struct is aware that they might need to opt out of SIMD support for a variety of reasons.

2.5. False parallels to containers

§5.3 The library-boundary caveat also draws a false parallel to other containers:

In none of these cases does the standard library place any obligation on the consumer of rgb, and in none does it offer the consumer any protection. std::array has no opt-out and no means of recording that an aggregate use has fixed a type's layout, because altering a type's size is treated as the responsibility of the type's author, not of any container or view that merely stores it. array<rgb> continues to compile until the author changes the size, after which it fails silently on mismatched data, and this is accepted as correct.

Firstly, std::array and other containers store elements directly and provide references to those stored elements. By comparison, simd::vec deliberately does not do this so that bit-packed layouts, structure-of-arrays layout, collapsing the padding bytes between long doubles and other things are possible. Thus, the parallel to std::array is false because simd::vec is not merely an aggregation.

Secondly, containers such as std::array<T> depend on T being movable, copyable, etc. to also provide those properties for the container. This is no different from a user who is copying T around, outside of any container. By comparison, vec<T> depends on hyper-specific layout properties like the size being a power of two, where neither the user nor the library author might be aware that some opt-in has taken place when defining T.

2.6. Opt-out is not too much effort

An underlying premise of [P2964R5] is that it would be too much effort to opt out vec support, even though it's not stated as the key motivation. The premise is visible in the following:

The list [of types that opt out], moreover, has no natural bound, since it must enumerate the entire open set that the constraints previously admitted without enumeration: every strong typedef, every enumeration, every small aggregate, and types such as array, pair, tuple, and duration, together with any small trivially copyable type defined in future. Opt-out enumerates a small number of exceptions and allows the constraints to govern the remainder. Opt-in maintains an unbounded list of inclusions while rendering the constraints inert.

In either case (opt-in or opt-out), the set of types for which action needs to be taken has no bound: For opt-in, every struct and every enum is potentially a type for which a library author needs to opt out of vec support because they may change layout details later.

Furthermore, while there are many types that are eligible for SIMD use, the amount which users would plausibly want to put into a vec is likely a tiny fraction of those types. There are many enums and structs that happen to have the right size, but don't have any operator overloads, so vec<T> could do nothing more than a array<T> anyway. Opting into simd::vec support only starts making sense once there are operations to be performed elementwise, and adding a one-liner opt-in is a very small burden for a type that has hundreds or thousands of lines of arithmetic operations defined.

2.7. No mechanism for deprecation

A problem that further compounds an opt-out design is that there is no way to deprecate existing simd::vec support. If the library author decides that they don't want simd::vec<T> to be allowed but users already make use of that, they have no means of deprecating those existing uses of simd::vec<T>:

• Putting [[deprecated]] on the disable_vectorization<T> specialization doesn't accomplish anything by itself.
• Putting [[deprecated]] onto the simd_operator functions for operation customization also doesn't apply deprecation to the actual operator overloads.

At least, this issue is solvable without any major downside: it is possible for the standard library to detect the presence of [[deprecated]] attributes using C++26 Reflection and to define the corresponding operator overloads and specializations as deprecated.

3. Operation customization

3.1. Customizing math functions

Another major problem with [P2964R5] is the way that operation customization is performed. In §7 Customization Points, for example, the mechanism for customizing operator+ between simd::vec relies on std::plus. Such function objects are sufficient for customizing any operators. However, this mechanism is only necessary if the user wants to avoid the default behavior, which is that operations are performed elementwise using the scalar operation.

The problem is that there are vastly more operations that may need to be customized, and the paper provides no exploration for how that should be done. For example, if the user defines a function abs(T) for their type T, should simd::abs not automatically perform the elementwise operation? The paper uses chrono::duration as an example of a type that may be supported due to implicit opt-in, but simd::abs would not automatically call std::abs(std::chrono::duration).

This problem becomes especially apparent when considering user-defined floating-point types like float8_t (e.g. for use in AI applications). In addition to providing operator overloads for a float8_t type, the user would likely implement large amounts of math functions for float8_t as well. Ultimately, the problem should be addressed holistically in such a way that puts all the following into relation:

• std::abs for floating-point and complex types.
• std::simd::abs for floating-point and complex types.
• Future abs functions for a Quantities and Units library.
• std::abs for additional library types like chrono::duration.
• User-defined abs functions outside the std namespace.
• Customizable math functions like std::math::abs (see [P4188R0]).

3.2. No obvious solution

We appear to have an ever-growing and incoherent collection of mathematical functions, and [P2964R5] compounds this issue. It seems inevitable that any vec<custom_float> would need math function support, so simply ignoring this problem is not viable. There also appears to be no obvious solution:

• If the user is intended to rely on future customizable std::math functions, the user is severely limited in their customization. [P4188R0] only suggests a handful of math functions to be customizable, but simd::vec<float> already supports vastly more, and any user-defined floating-point type may also need to support vastly more.
• If the user is intended to provide their own abs overloads to customize SIMD math, this requires a large amount of boilerplate because there is no nice elementwise by default behavior. This also raises the question whether we need elementwise by default for operator overloads in the first place. Whether the user needs to define 95 boilerplate elementwise math functions where func(simd::vec<T>) calls func(T) and avoids 5 boilerplate operator overloads, or defines 100 boilerplate functions makes very little difference. If [P2964R5] necessitates a mountain of boilerplate, we may as well simplify the overall design instead of reducing the size of the mountain by a few percent.
• If simd::sqrt were intended to behave elementwise by default, just like operator+ between simd::vec behaves elementwise by default, that would be a consistent design, but it would bypass and conflict with the more general mechanism of std::math functions. In any case, this possibility is not discussed at all in the paper.

4. Call for action

[P2964R5] should do the following:

• Revert to opt-in simd::vec support for user-defined types.
• Discuss the possibility of deprecating simd::vec specializations and operators for user-defined types (§2.7. No mechanism for deprecation).
• Halt progress on [P2964R5] until [P4188R0] has been seen by LEWG and until its relation to SIMD operation customization is clear.

5. References