P3904R3
When paths go WTF: making formatting lossless

"Compatibility means deliberately repeating other people’s mistakes." − David Wheeler

1. Introduction

[P2845], adopted in C++26, added formatting support for std::filesystem::path, addressing encoding issues and making formatting of a path as an ordinary (char) string lossless except for one case, unpaired surrogates on Windows. This paper proposes addressing this case and making such formatting lossless by default via the WTF-8 encoding ([WTF-8]). This will improve consistency in path handling between Windows and POSIX platforms and align with the design of std::format where the default formatting is normally lossless.

2. Changes since R2

• Narrowed the losslessness claim to formatting of a path as an ordinary (char) string and clarified that transcoding to a wide (wchar_t) string on POSIX is implementation-defined and out of scope.

3. Changes since R1

• Added the "Wording" section.

4. Changes since R0

• Added the "Implementation experience" section.

5. Motivation

[P2845] made it possible to format and print Unicode paths, even on Windows, which historically had problems because of legacy code pages. For example

std::print("{}\n", std::filesystem::path(L"Шчучыншчына"));

is correctly formatted and printed on Windows when the literal encoding is UTF-8 regardless of the Active Code Page.

However, paths are not guaranteed to be valid Unicode or even text. In general they are just sequences of bytes (or 16-bit values on Windows) which often but not always contain Unicode text, quoting [WIN32-FILEIO]:

the file system treats path and file names as an opaque sequence of WCHARs

This is also true on POSIX ([PEP383]):

File names, environment variables, and command line arguments are defined as being character data in POSIX; the C APIs however allow passing arbitrary bytes - whether these conform to a certain encoding or not.

Arbitrary paths are formatted on POSIX such that there is no data loss. Unfortunately this is not the case on Windows, for example:

auto p1 = std::filesystem::path(L"\xD800"); // a lone surrogate
auto p2 = std::filesystem::path(L"\xD801"); // another lone surrogate
auto s1 = std::format("{}\n", p1); // s1 == "�"
auto s2 = std::format("{}\n", p2); // s2 == "�"

Apart from being inconsistent between platforms, this makes it impossible to reliably round trip paths. For example, p1 and p2 above are two distinct paths that are formatted as the same string. This may result in a silent data loss and is remarkably different from other standard formatters such as the ones for floating point numbers which are specifically designed to round trip.

For comparison, on POSIX formatting of arbitrary paths including the ones that are not valid Unicode works as expected and is lossless:

auto p = std::filesystem::path("\x80");
auto s = std::format("{}\n", p); // s == "\x80"

This paper focuses on formatting a path as an ordinary (char) string, which is the common case. Formatting a path as a wide (wchar_t) string when the native encoding uses char, as on POSIX, requires transcoding that [P2845] leaves implementation-defined and that is therefore not guaranteed to be lossless. This direction is outside the scope of this paper.

6. Proposal

The current paper proposes preventing data loss and formatting ill-formed UTF-16 paths using WTF-8 (Wobbly Transformation Format − 8-bit) which is "a superset of UTF-8 that can losslessly represent arbitrary sequences of 16-bit code unit (even if ill-formed in UTF-16) but preserves the other well-formedness constraints of UTF-8." ([WTF-8])

std::format("{}\n", std::filesystem::path(L"\xD800"));

"�"

"\xED\xA0\x80"

std::format("{}\n", std::filesystem::path(L"\xD801"));

"�"

"\xED\xA0\x81"

This will enable round trip of paths from char strings which is currently not possible. The API for the read path of the round trip will be proposed by a separate paper.

At the same time this will preserve the observable behavior for std::print when printing to a terminal. For example:

std::print("{}\n", std::filesystem::path(L"\xD800"));

will still print

�

on implementations that follow the recommended practice from [ostream.formatted.print]:

Recommended practice: For vprint_unicode, if invoking the native Unicode API requires transcoding, implementations should substitute invalid code units with U+FFFD REPLACEMENT CHARACTER per the Unicode Standard, Chapter 3.9 U+FFFD Substitution in Conversion.

WTF-8 is used to handle invalid UTF-16 in paths and other system APIs in Rust ([RUST-OSSTRING]) and Node.js libuv ([LIBUV]). Python also handles this but with a different mechanism ([PEP383]).

7. Wording

Modify the value of the feature-test macro __cpp_lib_format_path in "Header <version> synopsis" [version.syn] and "Header <filesystem> synopsis" [fs.filesystem.syn]:

#define __cpp_lib_format_path 202506L**placeholder**	// also in <filesystem>

Modify the Effects of the format member function in "Formatting support functions" [fs.path.fmtr.funcs] as indicated:

template<class FormatContext>
  typename FormatContext::iterator
    format(const filesystem::path& p, FormatContext& ctx) const;

Effects: Let s be p.generic_string<filesystem::path::value_type>() if the g option is used, otherwise p.native(). Writes s into ctx.out(), adjusted according to the path-format-spec. If charT is char, path::value_type is wchar_t, and the literal encoding is UTF-8, then the escaped path is transcoded from the native encoding for wide character strings to UTF-8 with maximal subparts of ill-formed subsequences substituted with U+FFFD REPLACEMENT CHARACTER per the Unicode Standard, Chapter 3.9 U+FFFD Substitution in Conversion , with each unpaired surrogate code unit with value u (a code unit in the range [0xD800, 0xDFFF] that is not part of a valid surrogate pair) encoded as the three bytes 0xE0 | (u >> 12), 0x80 | ((u >> 6) & 0x3F), and 0x80 | (u & 0x3F) . If charT and path::value_type are the same, then no transcoding is performed. Otherwise, transcoding is implementation-defined.

[Note: Special encoding of unpaired surrogates makes this transcoding lossless. — end note]

Returns: An iterator past the end of the output range.

8. Implementation experience

The proposal has been implemented in {fmt} where the default std::filesystem::path representation is now lossless.