Default executor

ISO/IEC JTC1 SC22 WG21 N4039 - 2014-05-23

Adam Berkan (aberkan@google.com)

Chris Mysen (mysen@google.com)

Hans Boehm (hboehm@google.com)

Overview

So far the standard has been quiet about whether there is a default executor (See n3785 for more about executors).  A well-written library that wishes to be flexible should take an executor as a parameter.  Unfortunately there is today no good default for that library.  Programmers want a way to run tasks asynchronously without having to create new executors.

This problem is compounded by the fact that thread pools are often much faster (and use fewer resources) than a thread-per-task executor.  Many systems (e.g. Windows) provide a good system thread pool which would be a great default.  We would like to make it easy to use that thread pool, but a thread pool is generally limited in size and is likely limited in queue depth.  This adds new constraints to tasks running here, including the number of concurrently blocked threads.  There should be a way to specify a real thread-per-task executor for when it is necessary to avoid these constraints.

An executor that reuses threads could also be faster than a thread-per-task executor, but it would leak thread locals between tasks.  Many users would be willing to trade off the leakage for the performance, but this makes it complicated to use thread locals with these executors.

Another related issue is describing where a std::async task runs asynchronously.  If we have a default executor, it should be easy to run an async task there.  It has been proposed many times (e.g. N3970) to let std::async take an optional executor as an argument.  But if no executor is specified, should it run on the std::default_executor?  If the default executor has more constraints than a thread-per-task executor, those limitations would then also apply to async.  This has the potential to break existing code.  Still, most users of async would probably prefer to use a fast thread pool instead of spinning up a thread.

Beyond all of this (and not discussed in depth here), there’s the looming promise of fibers, and it’s uncertain how fibers might be integrated into the standard, and how they would affect executors.

Design Options

We’ve been discussing a number of proposals;  they all have pros and cons.  They all center around 3 new standard executors:

We don’t think any specific favorite proposal is obviously the best, but would like to hear what others think.  Here’s our list:

1. Do Nothing

We always have the option of not providing any default executor.  We still provide an explicit std::thread_executor, and force users to explicitly use it.  If they want to use the fast system thread pool they use the non portable my_system::thread_pool_executor.

Pros:

Cons:

2. Provide a std::default_executor (thread per task) and std::default_pool_executor

We would provide all three executors in the standard, with default_executor being the same as thread_executor (except maybe allowing for thread reuse, i.e. thread-local sharing).

We could also rename default_pool_executor to default_fast_executor, to emphasize why you’d use it, rather than the implementation details.

If std::async is extended to take an executor as parameter, there’s now a standardized easy way to use the system’s thread pool.  If an executor is not specified, it runs on default_executor.

Pros:

Cons:

3. Provide the two new executors (thread pool & thread per task), and let the user specify at compile time which should be the std::default_executor.

The standard doesn’t really discuss compile time options, but we could figure out a way to allow the default executor to be selected at compile time.  This allows an implementation to default to the system thread pool, but allows the escape valve of forcing thread-per-task if you need it.

This comes in two varieties:

  1. std::async doesn’t change, continues to create a new thread per task
  2. std::async by default uses std::default_executor to create new tasks

Pros:

Cons:

4. Add std::default_executor but with weak enough guarantees that can be implemented by a system thread pool

Making default_executor compatible with thread pools will strongly encourage use of system thread pools.  We still provide std::thread_executor as an escape valve, which will work for any existing code.

This on also comes in two varieties:

  1. std::async doesn’t change, continues to create a new thread per task
  2. std::async by default uses std::default_executor to create new tasks

Pros:

Cons:

Conclusion

We think that (1) & (2) are probably not great because they will tend to keep users away from the system thread pools.

(3) is interesting, and the most flexible solution.  It allows advanced users to run exactly the code they want, and it encourages all libraries to specify what constraints they have on executors.  In practice most users would move to using system thread pools by default, but we don’t break any existing code.  It also provides a hook for other executors to come in the future.  All that said, it introduces an “at compile time” concept that’s hard to explain in standardese, and many libraries will never specify their executor constraints, which forces users to either run with thread-per-task, or run without correctness guarantees.

(4 B) would be great, as it would move everyone to system thread pools unless they explicitly move away from it.  Unfortunately it would add new constraints to std::async, which would change existing behavior.  If async is being changed or replaced in the future it might make sense to use the default_executor, but doing that now would be difficult.

(4 A) may be the most reasonable compromise.  It makes the default executor fairly weak in guarantees, but still a good default for users who “don’t care”.  It is tricky to nail down exactly what the guarantees should be, but I suspect people who pay attention to the guarantees will prefer explicitly specifying their executor anyway.  Unfortunately it is verbose to use the thread pool with std::async, but it doesn’t change any existing behavior.