Document number: N4041
Date: 2014-05-23
Project: Programming Language C++, Library Working Group
Reply-to: Jonathan Wakely <cxx@kayari.org>

Concerns with changing existing types in Technical Specifications

Introduction

In Issaquah there were proposals targeting a TS which enhanced existing types in the Standard Library, in particular N3857 which modifies std::future, N3916 which modifies std::function, std::promise and std::packaged_task, and N3920 which modifies std::shared_ptr.

The Library and Library Evolution Working Groups met in a joint session to discuss whether a TS should specify changes to the C++ standard, as a list of additions to the base specification, or whether to copy the whole specification into the TS and create a duplicate type in namespace std::experimental which could then be modified independently.

Following straw polls the decision was taken to allow a TS to change existing types in namespace std, so that users who opt in to use the TS (via some implementation defined means) get the enhanced versions without needing to change any code.

Impact on Standard Library implementations

The straw polls taken in Issaquah were done with very few of the standard library implementers in the room (most of them were in another room doing ballot resolution for the Filesystem TS, which ironically could have benefited from the decision had it been made months earlier, by allowing a std::fstream to be constructed from a std::experimental::path object).

The decision made following the straw polls means for an implementation to ship a TS they must change existing classes in their implementation, which has the potential to make silent ABI changes to users' programs. An implementation can still choose to not declare std::function when users enable support for a TS, instead declaring a different type in an inline namespace, for example std::__libfund::function. This ensures that some uses of std::function will result in different name mangling and so it will be detectable when two translation units have been compiled inconsistently (one with TS support enabled and one without). However, name mangling does not typically encode anything about a function's return type or about the type of global variables, and the mangled name of a class is not affected by the type of one of its members changing from std::function to std::__libfund::function due to the use of a compiler switch or macro definition. This means when users opt in to the TS they can introduce silent ODR violations that typically result in run-time (rather than compile-time or link-time) errors.

This affects linking to libraries that use the changed types in their API, possibly including the Standard Library itself. On some platforms libstdc++ uses an extern std::function<void()> object in the implementation of std::call_once. If a user called std::call_once in a translation unit built with TS support enabled the resulting object file would attempt to invoke member functions on the global variable as though it was of type std::__libfund::function, whereas the actual variable would be of type std::function, resulting in an ODR violation. This could be worked around by the library implementer, for example by using a different global variable when TS support is enabled, and possibly a different std::call_once, but this means the changes are not restricted to just the components altered by the TS, but to other, seemingly unrelated parts of the implementation. It also doesn't avoid all ODR violations anyway, e.g. if the use of std::call_once is in an inline function or a template that is used from two translation units, one with TS support enabled and one without.

These issues can make it complicated for implementers to provide TS support, so that users have to wait longer before they have access to an implementation, which then makes it less likely the committee will get useful and timely feedback on the TS contents.

Impact on third-party implementations

If implementing a TS requires changes to namespace std then it can only be done by the "owner" of namespace std, i.e. the standard library implementation. That makes it impossible to have conforming third-party implementations from groups such as Boost. There is no way for Boost to inject a modified std::future into the implementation's <future> header, so it's necessary to replace the entire header, but that might break other standard headers which rely on implementation details expected to be found in <future>. This was a real problem for the Boost.TR1 implementation, and the author of that library has expressed a very strong preference for not requiring changes to std in any specification except the C++ standard itself.

It is widely acknowledged that it's very difficult to provide third-party implementations of the Standard Library because a number of C++11 library features rely on compiler support, meaning the compiler and library are more tightly coupled. There is no need for that to be the case for a TS though. It makes a lot of sense to allow, and even encourage, third-party implementations of experimental features. Boost already implements the changes to std::shared_ptr that are specified in the Library Fundamentals TS already and has non-standard extensions to its std::future implementation similar to those in the Concurrency TS. It is impossible for Boost to provide these as conforming implementations of the TS if the changes must be made to types in std rather than as separate types in namespace std::experimental.

If it is harder for third-party implementations to provide a working version of the TS then that also makes it harder for users to get their hands on an implementation and again makes it less likely we will get timely feedback on the TS.

Impact on users

If a Standard Library implementation changes the definition of a type when a compiler option or macro is used that is arguably no worse than other ABI breaking options such as -fpack-struct, but the difference is that we're encouraging users to try out TS features in the hope we'll get feedback to help decide whether to standardise the features later.

std::function is a vocabulary type, so it's bad to have two separate versions of it, but the very fact it is a vocabulary type makes it more likely to have been used in APIs and so the potential for ODR violations makes it necessary for some users to recompile their entire codebase. The larger the codebase the more likely it is that something, somewhere will be affected by an ABI change in non-obvious ways (i.e. not resulting in a compile-time or link-time error). However recompiling the entire codebase is just not feasible for most large code bases, and feedback from large scale users is valuable.

Users can still get similar ODR violations if the TS requires a new type, but then they must make source code changes. Code changes mean build tools will typically cause recompilation of all affected files, or the inability to make changes to a third-party library will make it obvious that the library isn't compatible with the new type. Requiring code changes (not just recompilation) makes opt in more explicit and requires a more conscious decision. It's important to note that users can't benefit from these new TS features without some code changes anyway.

Conclusion

I don't really have one, but I think the question of whether to require changes to existing types deserves more attention. If altering existing types makes it harder for implementors to safely ship the TS, or for users to experiment with the contents of namespace std::experimental in real code, then the committee and the community miss out on feedback about the contents of the TS.

Acknowledgements

Thanks to John Maddock, Alisdair Meredith, and Jeffrey Yasskin for sharing their thoughts on this subject.