Tracking issue for RFC 2033: Experimentally add coroutines to Rust

[aturon]

RFC.

This is an experimental RFC, which means that we have enough confidence in the overall direction that we're willing to land an early implementation to gain experience. However, a complete RFC will be required before any stabilization.

This issue tracks the initial implementation.

related issues

See also:

• Tracking Issue for gen blocks and functions #117078

[alexcrichton]

cc #43076, an initial implementation

[mitranim]

Copied from #43076:

---

I'm using this branch for stream-heavy data processing. By streams I mean iterators with blocking FS calls. Because Generator is missing an Iterator or IntoIterator implementation, you must call your own wrapper. Zoxc kindly provided an example, but it's quite unergonomic. Consider:

Python:

def my_iter(iter):
    for value in iter:
        yield value

Rust with generators:

fn my_iter<A, I: Iterator<Item=A>>(iter: I) -> impl Iterator<Item=A> {
    gen_to_iter(move || {
        for value in iter {
            yield value;
        }
    })
}

Two extra steps: inner closure + wrapper, and, worse, you have to write the wrapper yourself. We should be able to do better.

TL:DR: There should be a built-in solution for Generator→Iterator.

[silene]

I was a bit surprised that, during the RFC discussion, links to the C++ world seemed to reference documents dating back from 2015. There have been some progress since then. The latest draft TS for coroutines in C++ is n4680. I guess the content of that draft TS will be discussed again when the complete RFC for Rust's coroutines is worded, so here are some of the salient points.

First, it envisions coroutines in a way similar to what this experimental RFC proposes, that is, they are stackless state machines. A function is a coroutine if and only if its body contains the co_await keyword somewhere (or co_yield which is just syntactic sugar for co_await, or co_return). Any occurrence of co_await in the body marks a suspension point where control is returned to the caller.

The object passed to co_await should provide three methods. The first one tells the state machine whether the suspension should be skipped and the coroutine immediately resumed (kind of a degenerate case). The second method is executed before returning control to the caller; it is meant to be used for chaining asynchronous tasks, handling recursive calls, etc. The third method is executed once the coroutine is resumed, e.g. to construct the value returned by the co_await expression. When implementing most generators, these three methods would have trivial bodies, respectively { return false; }, {}, and {}.

Various customization mechanisms are also provided. They tell how to construct the object received by the caller, how to allocate the local variables of the state machine, what to do at the start of the coroutine (e.g. immediately suspend), what to do at the end, what do to in case of an unhandled exception, what to do with the value passed to co_yield or co_return (how yielded values are passed back to the caller is completely controlled by the code).

[arielb1]

One subtle point that came up is how we handle the partially-empty boxes created inside of box statements with respect to OIBITs/borrows.

For example, if we have something like:

fn foo(...) -> Foo<...> {}
fn bar(...) -> Bar<...> {}
box (foo(...), yield, bar(...))

Then at the yield point, the generator obviously contains a live Foo<...> for OIBIT and borrow purposes. It also contains a semi-empty Box<(Foo<...>, (), Bar<...>)>, and we have to decide whether we should have that mean that it is to be treated like it contains a Box, just the Foo<...>, or something else.

[masonk]

I might be missing something in the RFC, but based on the definition of resume in the Generator struct, and the given examples, it looks like these generators don't have two way communication. Ideally this language construct would allow us to yield values out and resume values into the generator.

Here's an example of implementing the async/await pattern using coroutines in ES6. The generator yields Promises and the coroutine resumes the generator with the unwrapped value of a Promise each time the Promise completes. There is no way this pattern could have been implemented without the two-way communication.

Rust has a problem here because what's the type of resume? In the ES6 example, the generator always yields out some kind of Promise and is always resumed with the unwrapped value of the Promise. However the contained type changes on each line. In other words, first it yields a Promise<X> and is resumed with an X, and then it yields a Promise<Y> and is resumed with a Y. I can imagine various ways of declaring that this generator first yields a Wrapper<X> and then a Wrapper<Y>, and expects to be resumed with an X and then a Y, but I can't imagine how the compiler will prove that this is what happens when the code runs.

TL;DR:
yield value is the less interesting half. It has the potential to be a much more ergonomic way to build an Iterator, but nothing more.

let resumedValue = yield value; is the fun half. It's what turns on the unique flow control possibilities of coroutines.

(Here are some more very interesting ideas for how to use two-way coroutines.)

[mikeyhew]

@arielb1

Then at the yield point, the generator obviously contains a live Foo<...> for OIBIT and borrow purposes. It also contains a semi-empty Box<(Foo<...>, (), Bar<...>)>, and we have to decide whether we should have that mean that it is to be treated like it contains a Box, just the Foo<...>, or something else.

I don't know what you mean by "OIBIT". But at the yield point, you do not have a Box<(Foo<...>, (), Bar<...>)> yet. You have a <Box<(Foo<...>, (), Bar<...>)> as Boxed>::Place and a Foo<...> that would need to be dropped if the generator were dropped before resuming.