fn get_default<K, V>(
map: &mut HashMap<K,V>,
key: K,
) -> &mut V {
match map.get_mut(&key) {
Some(value) => value,
None => {
map.insert(key, V::default());
map.get_mut(&key).unwrap()
}
}
}
I won’t go into details about this particular example, but this is a well-known problem referred to as “problem case 3”.
You may know that there are ongoing efforts (for many years now!) to fix the borrow checker to “just work” for this example and others. That work has been dubbed “Polonius” and talked about in great detail by Niko Matsakis and others. Niko has a series of posts talking about Polonius and its predecessor NLL, as well as a couple more recent posts on the most recent approach to implementing Polonius-style borrow checking.
We’re getting very close to stabilizing the new iteration of the borrow checker, which has been coined “Polonius Alpha” (thanks in large part to lqd), and I want to talk today about one little hiccup that was discovered where Polonius Alpha is unsound when using impl Trait (and associated types, but we’ll get there). I want to walk through the initial problem that was found, how it is more comprehensive than originally thought, and then the solution.
I won’t go into too many details of how Polonius works under the hood; for that, you should go read Niko’s latest couple blog posts, since they’re very good. Instead, I’m going to try to give only the details that matter to explain the problem and the solution.
The first thing to talk about is liveness. Put simply, a variable is live if it could be used again:
fn foo() {
let x = String::from("test"); //
let y = &mut x; // -- x is live
call_a(); // | -- y is live
call_b(); // | |
drop(y); // | --
call_c(); // |
} // -- (x is implicitly dropped)
Of course, it’s a bit more complicated than that: we don’t just think about live variables, but live regions (which are sort of the internal representation of a “lifetime”). For the above example, while y is live, so is a mutable borrow of x. So, because y is live at call_b(), we could not, for example, use x because y is later used.
This (and more) is of course available on stable today with NLL: when program flow is linear, borrow checking is “easy”. However, with Polonius Alpha, liveness calculation is extended to take into account branches within a function, too. And further, liveness is combined with known lifetime-outlives relationships to reduce false-positive borrow-check errors. However, incorrect liveness information can result in Polonius disregarding needed outlives requirements, leading to missed borrow-check errors and unsound programs being accepted. To read into the specifics of how liveness is used, Niko shares much more detail in his blog posts.
So, how is liveness calculated anyways? Let’s start with two simple examples:
let x: &'a u8;
let z: MyStruct::<'b>;
The first, we saw this above: as long as x is live, so is 'a. The second is also pretty easy: as long as z is live, 'b is too.
What about opaque types? (You may know them as impl Trait.)
fn rpit_lt<'a, 'b>(x: &'a (), y: &'b mut ()) -> impl Sized + 'a + use<'a, 'b> {
x
}
fn rpit_static<'a, 'b>(x: &'a (), y: &'b mut ()) -> impl Sized + 'static + use<'a, 'b> {
()
}
NOTE: I’m using an explicit
usebound to demonstrate the lifetime captures of the RPIT (return position impl trait) but starting in edition 2024, these captures are implicit.
The first signature basically says “the opaque type can capture any variable with either 'a or 'b, but it must outlive 'a”. The second says “the opaque type can capture any variable with 'a or 'b, but it must outlive 'static”. Also note that in neither example do we know if 'b outlives 'a.
Now, if this is confusing to you, then you’re in good company. How can an opaque type be “allowed to capture a set of lifetimes” but simultaneously be required to outlive only one? Why is this useful to be able to say anyways?
For the question of “why do we need use bounds”, I would direct you to read this blog post. tl;dr we want you to be able to be precise in the lifetimes that can be captured in an impl Trait, particularly when the default is to capture everything.
For the first question, consider the following snippet:
let x = &();
let y = &mut ();
let a = rpit_lt(x, y);
let b = rpit_lt(x, y);
drop(a);
drop(b);
Now, think about this: by default, if we don’t have the 'a outlives bound, then the compiler must assume that both the lifetimes of x and y end up in the opaque type. So, from a compiler perspective, it’s not much different than
let a = (x, y);
let b = (x, y);
drop(a);
drop(b);
Both a and b contain the same mutable reference and because you drop a while b is live, this results in a borrow-check error. The 'a outlives bound makes the compiler treat it more like:
let a = (x,);
let b = (x,);
drop(a);
drop(b);
Only a shared reference is in both a and b, so the drop order doesn’t matter and this compiles fine.
The 'static bound says that the opaque type cannot capture any lifetimes, so the compiler treats it roughly like:
let a = ();
let b = ();
drop(a);
drop(b);
Yes, you could achieve the same effect by changing the
usebound to not name the lifetimes that can’t be captured, but remember that this captures list is often implicit.
So, intuitively, let’s think about liveness and opaque types:
let a = rpit_lt::<'x, 'y>(..);
let b = rpit_static::<'x, 'y>(..);
For a, because the opaque type outlives 'x, then we only need to treat that as live. For b, because the opaque type outlives neither 'x or 'y, then we don’t need to treat either as live.
That’s how it works on stable today (as of #116733). That’s simple enough, right? Well, consider this variant:
fn rpit<'a, 'b>(x: &'a mut &'b ()) -> impl Sized + 'a + use<'a, 'b> {
x
}
I want to point out that the opaque type captures both 'a and 'b because of the nested references. Because there is an implied 'b: 'a outlives relationship, the outlives bound holds.
Let’s look at a minimal example of how you can use (a slight variant of) this function to create an unsoundness with Polonius Alpha today:
trait Swap: Sized {
fn swap(self, other: Self);
}
impl<T> Swap for &mut T {
fn swap(self, other: Self) {
std::mem::swap(self, other);
}
}
fn hide_ref<'a, 'b, T: 'static>(x: &'a mut &'b T) -> impl Swap + 'a {
x
}
fn dangle_ref() -> &'static [i32; 3] {
let mut res = &[4, 5, 6];
let x = [1, 2, 3];
hide_ref(&mut res).swap(hide_ref(&mut &x));
res
}
Let me point out the problem: in dangle_ref, we return a 'static reference to a local value, which can then be read for a use-after-free.
On stable, this fails as expected: we don’t treat 'b as live, but it doesn’t matter because NLL doesn’t combine liveness with outlives relationships to perform borrow-checking (at least, not in a way that matters for this example). But Polonius does.
For the details on why Polonius needs liveness, I’d suggest that you dig into Niko’s blog posts. In short, Polonius requires that for an outlives bound to be meaningful, variables must be live when the outlives bounds are required.
So, the fix here is to treat 'b as live too. I’ll talk about how we actually do that later in the post, and the rules that we use. But, in short: any lifetime must be considered live if it could be captured; and, we can identify that precise set by considering both the item definition and also where clauses in scope.
Oh, if only it was that simple. It turns out we also have to handle: associated types, rustc’s odd opaque representation, RPITITs, and extra bounds from where clauses.
So, impl Trait has this problem because they can have an outlives bound. Actually, associated types have a similar ability:
trait MyTrait {
type Assoc<'a, 'b: 'a>: 'a
where
Self: 'a;
fn changes<'a, 'b>(&'a self, x: &'b mut u8) -> Self::Assoc<'a, 'b>;
}
Surely you can see the parallel:
let t = make_something_that_impls_mytrait();
let x = &mut ();
let a = t.changes(x);
let b = t.changes(x);
drop(a);
drop(b);
We want this to compile, because of the 'a outlives bound on MyTrait::Assoc. We know that the type cannot contain x (because it has 'b which doesn’t outlive 'a); so in the example, there are no overlapping mutable references.
impl Trait in rustc is weirdAnother piece of complexity in this problem stems from how impl Trait is represented in rustc.
If you have:
fn foo<'a, 'b, T>(x: &'a mut &'b T) -> impl Sized + 'a + use<'a>;
That really looks something like:
opaque foo_opaque<'a1>: Sized + 'a1;
fn foo<'a, 'b, T>(x: &'a mut &'b T) -> foo::<'a, 'b, T>::foo_opaque<'a>;
There are two important considerations about the new “opaque” definition: 1) It gets the parent lifetimes copied to itself (but not type or const params). 2) All the information available to the parent (i.e. where clauses and the function signature) don’t get copied.
The latter is particularly important, because the inputs and output of a function can provide implied bounds because those types must be well-formed. For the above, &'a mut &'b T must have been constructed, so callers of foo must prove that 'b: 'a (also that T: 'b) and foo can just assume that holds.
So far when we’ve discussed RPITs, they’ve been present in freestanding functions. Turns out, RPITs in traits have a quirk: they are represented internally as (generic) associated types. Imagine you have a definition:
trait MyTrait {
fn foo<'a>() -> impl OtherTrait + 'a;
}
That is represented as:
trait MyTrait {
type FooRet<'a>: OtherTrait + 'a;
fn foo<'a>() -> Self::FooRet<'a>;
}
Now, since we already need to apply a fix for associated types (see above), that seems fine. Right? Of course it’s not that simple! But we’ll get there later.
where clausesLet’s start with another example:
fn rpit<'a>(x: &'a mut ()) -> impl Sized + use<'a> {
()
}
This is an error today (and with Polonius on nightly, to be fair):
let x = &mut ();
let a = rpit(x);
let b = rpit(x);
drop(a);
drop(b);
This is an error for the same reasons we talked about before: the opaque type could contain the mutable reference. However, there is a nightly feature that allows you to name opaque types to use them in where clauses (return type notation, RTN). With that, you can add additional information about the opaque type:
fn foo() where rpit(..): 'static {
let x = &mut ();
let a = rpit(x);
let b = rpit(x);
drop(a);
drop(b);
}
Note: The RTN feature is most useful when used for trait methods, but the idea translates the same here.
We ideally would let this compile as-if the 'static outlives bound was on the impl Trait itself.
Of course, you don’t need RTN to add additional bounds for associated types, so this is something we need to consider even on stable today:
trait MyTrait {
type Assoc<'a>
where
Self: 'a;
fn foo<'a>(&'a mut self) -> Self::Assoc<'a>;
}
fn bar<T: MyTrait>(mut t: T)
where
for<'a> T::Assoc<'a>: 'static,
{
let a = t.foo();
let b = t.foo();
}
We’ve gone through what the source of the unsoundness is and the key considerations for what we need to keep in mind when fixing it. So, let’s talk about the solution.
Remember, given an opaque type, we want to know what lifetimes can be contained inside it and we need to consider those live for borrowck (and elsewhere).
Let’s go back to this example:
fn rpit<'a, 'b>(x: &'a mut &'b ()) -> impl Sized + 'a + use<'a, 'b> {
x
}
Starting with the 'a outlives bound: this tells us that the opaque type can only contain lifetimes that we know outlive 'a. Or, put differently, any params that outlive 'a can be contained within the opaque type (with a caveat below). And so, we must mark these lifetimes as live. This is the root of the unsoundness: we don’t do that properly today.
Note: It’s not entirely true that we have to know all the params that outlive
'ato be sound; we only need to know all the params that the function knows to outlive'a(otherwise the function would not allow that to be contained in the opaque type).
Now that caveat: use<'a, 'b> is the explicit precise capture list, but you can imagine that more often this is implicitly calculated. In either case, this gives us an upper bound on the params that we need to consider live; we don’t need to consider all params that outlive 'a, only those that can be captured.
So, how do we identify params that can be captured and outlive 'a?
This is where &'a mut &'b () comes in. Or, really, the entire function signature and its where clauses. As mentioned previously, this function’s input argument tells us that 'b: 'a. We combine these implied bounds from the function’s signature with the explicit where clauses. Fortunately, there is existing code within the compiler to get all the explicit and implied bounds for a given item (in this case, for a given function).
Of course, there is a major consideration here: the opaque type has that weird representation where lifetime params are duplicated and where clauses are not copied. The fix is relatively simple in theory: we map the opaque’s parameters back to the parent function, find the known outlives relationship there, and then map back.
So, to know what params to consider live on rpit, we do the following:
1) See that there is an outlives bound on 'a.
2) Map that lifetime to the parent function.
3) For each param on the function (here 'a and 'b), use the where clauses and implied bounds to test if we know that it outlives 'a. In this case, both do.
4) Map this larger set back to captured params on opaque (in this case, both 'a and 'b are captured).
5) Substitute the actual used values for the params.
6) Mark all lifetimes in these values as live.
Let’s go through a much more complicated example. I’m adding a bunch of complexity here not just to the example, but to the explanation too. This much more closely matches the actual representation and implementation, but also means there are concepts I haven’t otherwise covered. I’ll do my best to explain:
fn rpit2<'a, 'b, 'c, 'd: 'b, U: 'a>(
x: &'a &'b u8,
y: &'c u8,
z: U,
) -> impl Sized + 'a + use<'a, 'b, 'c, U> {
(x, z)
}
Which gets represented and used like:
opaque rpit2_opaque<'a1, 'b1, 'c1>: Sized + 'a1;
fn rpit2_lowered<'a, 'b, 'c, 'd: 'b, U: 'a>(
x: &'a &'b u8,
y: &'c u8,
z: U,
) -> rpit2::<'a, 'b, 'c, 'd, U>::rpit2_opaque<'a, 'b, 'c> {
(x, z)
}
let s: OtherStruct<'_> = ...;
let out = rpit2(&&0, &0, s); // out has a type of:
// `rpit2::<'?1, '?2, '?3, '?4, OtherStruct::<'?5>>::rpit2_opaque<'?6, '?7, '?8>`
// `'?X` is a *lifetime variable* and `?Y` is a *type variable*
1) We see that there is a 'a1 outlives bound on rpit2_opaque.
2) Map this back to rpit2: 'a1 matches 'a.
3) Check outlives relationships:
- 'a: 'a trivially holds
- 'b: 'a holds because of &'a &'b u8
- 'c: 'a does not hold
- 'd: 'a holds because of the 'd: 'b bound and 'b: 'a
- U: 'a holds because of the U: 'a bound
4) Map back to the captured opaque params:
- 'a => 'a1
- 'b => 'b1
- 'd => not captured
- U => U
5) Substitute params with actual values:
- 'a1 => '?6
- 'b1 => '?7
- U => OtherStruct::<'?5>
6) Mark lifetimes '?6, '?7, '?5 as live
Before this change, we only would have marked '?6 as live.
I also want to point out that the above can be generalized a bit: given some lifetime on the opaque, we can find the params that must be considered live (steps 2-4). This will be important later, but for now let’s assume this is the generalization we want and work to apply it to other places.
Note: Although RPITITs are internally represented as associated types, the information to calculate all the outlives relationships (namely the where clauses and the well-formed function signature) is not available. Therefore, the algorithm for calculating live params for RPITITs is the same as for RPITs.
Implementing this logic for associated types is easy.
Fortunately, there is existing code within the compiler to get all the explicit and implied bounds for a given item.
This code just works for them.
trait MyTrait {
type Assoc<'a, 'b: 'a, T: 'a>;
}
We can ask “what params outlive 'a” and get back 'a, 'b, and T. We don’t need to do any lifetime mapping tricks or filtering for captured params; those only matter for RPITs.
Warning: this section is as complicated (if not more) as finding live params on the opaque or associated type definition.
Do you remember how you write today:
fn bar<T: MyTrait>(mut t: T)
where
for<'a> T::Assoc<'a>: 'static,
{
...
}
And expect that T::Assoc<'a> can’t actually contain anything that uses the lifetime param. However, here are many other examples to consider:
trait MyTrait { type Assoc<'a, 'b: 'a, 'c>; }
fn fn1<'x, 'y, 'z, T: MyTrait>() where
T::Assoc<'x, 'y, 'z>: 'static,
fn fn2<'x, 'y, 'z, T: MyTrait>() where
T::Assoc<'x, 'y, 'z>: 'x,
fn fn3<'x, 'y, 'z, T: MyTrait>() where
T::Assoc<'x, 'y, 'z>: 'z,
fn fn4<'x, T: MyTrait>() where
T::Assoc<'x, 'x, 'x>: 'x,
fn fn5<'x, 'y, T: MyTrait>() where
for<'h> T::Assoc<'x, 'y, 'h>: 'h,
fn fn6<'x, T: MyTrait>() where
for<'h> T::Assoc<'h, 'x, 'x>: 'x,
fn fn7<'x, T: MyTrait>() where
for<'h> T::Assoc<'h, 'h, 'h>: 'x,
fn fn8<T: MyTrait>() where
for<'h, 'i, 'j> T::Assoc<'h, 'i, 'j>: 'h,
fn fn9<T: MyTrait>() where
for<'h, 'i, 'j> T::Assoc<'h, 'i, 'j>: 'h,
for<'h, 'i, 'j> T::Assoc<'h, 'i, 'j>: 'j,
fn fn10<T: MyTrait>() where
for<'h> T::Assoc<'h, 'h, 'h>: 'h,
fn fn11<T: MyTrait>() where
for<'h> T::Assoc<'h, 'h, 'h>: 'static,
fn fn12<T: MyTrait>() where
for<'h, 'i, 'j> T::Assoc<'h, 'i, 'j>: 'static,
That’s a lot! But generally, there are a couple different questions to consider here:
for<...>?for<...> (or 'static)?So, let’s work through these, and see if we can develop an intuition about the general rule here. We’ll start with actually backing up a bit, and thinking about what the implementation of the associated type could look like. All of these are “valid”:
impl MyTrait for Foo {
type Assoc<'a, 'b, 'c> = (); // No lifetimes used
type Assoc<'a, 'b, 'c> = (&'a &'b (), &'c ()); // All used
type Assoc<'a, 'b, 'c> = &'a &'b (); // Only some
type Assoc<'a, 'b, 'c> = &'c (); // Only one
}
If you have a bound like for<'h, 'i, 'j> T::Assoc<'h, 'i, 'j>: 'h, that rules out the second and fourth definitions, because 'c: 'a doesn’t hold. It does not rule out the third, because we know that 'b: 'a holds because of the trait definition.
Now, what about something like fn2 above with the bound T::Assoc<'x, 'y, 'z>: 'x? Well, free lifetimes (i.e. defined on the function, not within the function) are always live. So, when all the lifetimes contained are free, then they are all live. What about a bound like for<'h> T::Assoc<'x, 'h, 'x>: 'x? Let’s think about what that could mean if the impl definition looks like the third above:
fn foo<'x>()
where
for<'h> T::Assoc<'x, 'h, 'x>: 'x,
{
let val: T::Assoc<'x, '_, 'x> = &&(); // &'?1 &'?2 ();
}
Here, '?2 must outlive '?1 because of the well-formedness of the right side. However, '?1 must equal 'x because of the impl. Therefore, '?2 must outlive 'x and also be live.
What about this example, with the fourth impl definition:
fn bar<'a, 'b>(a: &'a &'a (), b: &'b ()) -> <Foo as MyTrait>::Assoc<'a, 'a, 'b> {
b
}
fn foo<'x>()
where
for<'h, 'i> <Foo as MyTrait>::Assoc<'h, 'i, 'x>: 'x,
{
let a = &mut &();
let b: &'x () = &();
let val2 = bar(a, b);
let val3 = bar(a, b);
drop(val2);
drop(val3);
}
In this case, because of the outlives bound we know that neither 'h and 'i can be in the associated type. So, we should be safe to only treat 'x as live (which it already must be because it is free). However, we don’t do that today (on stable); we treat everything as live. So, that isn’t changed: when the outlives bound is a free variable, we don’t restrict the things that could be live.
So, next example. We’ll look at fn10 from above:
fn fn10<T: MyTrait>() where
for<'h> T::Assoc<'h, 'h, 'h>: 'h,
This means that all three lifetimes may appear in the associated type, and therefore must be considered live.
Of course, if we have for<'h, 'i, 'j> T::Assoc<'h, 'i, 'j>: 'h, then 'h is obviously allowed to be contained in the associated type; but so is 'i, because we know from the trait that the second lifetime outlives the first.
Note: This doesn’t quite work today, because you will get an error that we don’t know that
'i: 'h. But, we eventually want to make this work.
So, let’s work through a general algorithm here, for the for<'h, 'i, 'j> T::Assoc<'h, 'i, 'j>: 'h where clause:
1) Identify the outlived lifetime. If it is 'static, there are no live lifetimes. If it is free, all lifetimes are live.
- The outlived lifetime is 'h.
2) For each param in the where clause for the associated type, if it matches the outlived lifetime then map it to the lifetime on the associated type.
- 'h in T::Assoc<'h, 'i, 'j> corresponds with 'a on the associated type.
3) For each lifetime on the associated type that may be live, find the transitive params that could also be live.
- This uses the generalized algorithm from steps 2-4 earlier.
- Here, that expanded set is 'a and 'b.
4) Substitute the actual values.
- For T::Assoc<'?1, '?2, '?3>, map 'a => '?1 and 'b => '?2.
Simple enough, right?
What about when there are multiple where clauses?
fn fn9<T: MyTrait>() where
for<'h, 'i, 'j> T::Assoc<'h, 'i, 'j>: 'h,
for<'h, 'i, 'j> T::Assoc<'h, 'i, 'j>: 'j,
This answer is actually simple: each where clause independently restricts the set of live regions.
To reiterate and summarize the solution:
I’m hoping I didn’t lose you. This is complicated, and there are some nuances that I didn’t talk about (for example, free standing type aliases are also affected, albeit similar to associated types). The PR with this fix is up, if you’re curious to take a look. You might be interested to see some of the 8 new tests added that demonstrate the complexity (on top of the three existing ones), or read through the actual implementation (it’s hopefully commented well enough).
As a final point: calculating liveness correctly for associated types and opaque types does have positive consequences outside the borrow-checker. For example, there are open issues for where we incorrectly over-approximate live variables, because attempting to be more correct was clearly unsound. The fix presented here allows us to properly calculate liveness and allow more code to compile (correctly).
With all hope, this will be landed soon and Polonius Alpha will be on its way to stabilization!
]]>Let’s start with the basics. What am I writing this post for? For a few reasons, actually. First, many lifetime errors have changed on nightly (and will hit stable in August), so I want to introduce them a bit, because I think they’re really cool. Second, I want to give a little history of how we got here, because I think it’s always important to look at things in retrospect. And third, along the way, I want to shout out all the work done by various contributors to get this enabled. I personally have done very little of the work here; I’m mostly going to be telling a story written by others, and they deserve all the credit.
Also note, while there are a handful of things that will now compile that didn’t before, I won’t really talk about them here. The list is small and most people won’t hit them. The stabilization PR has a couple examples.
I’ll start by just including a snippet of the stabilization report, which gives a brief summary of the motivation of this change, which might help frame the current text.
Over time, the Rust borrow checker has become “smarter” and thus allowed more programs to compile. There have been three different implementations: AST borrowck, MIR borrowck, and polonius (well, in progress). Additionally, there is the “lexical region resolver”, which (roughly) solves the constraints generated through HIR typeck. It is not a full borrow checker, but does emit some errors. The AST borrowck was the original implementation of the borrow checker and was part of the initially stabilized Rust 1.0. In mid 2017, work began to implement the current MIR borrow checker and that effort ompleted by the end of 2017, for the most part. During 2018, efforts were made to migrate away from the AST borrow checker to the MIR borrow checker - eventually culminating into “migrate” mode - where HIR typeck with lexical region resolving following by MIR borrow checking - being active by default in the 2018 edition. In early 2019, migrate mode was turned on by default in the 2015 edition as well, but with MIR borrowck errors emitted as warnings. By late 2019, these warnings were upgraded to full errors. This was followed by the complete removal of the AST borrow checker. In the period since, various errors emitted by the MIR borrow checker have been improved to the point that they are mostly the same or better than those emitted by the lexical region resolver. While there do remain some degradations in errors (tracked under the NLL-diagnostics tag, those are sufficiently small and rare enough that increased flexibility of MIR borrow check-only is now a worthwhile tradeoff.
Likely the biggest difference that will be noticed after this change will be to the error messages you get regarding lifetime errors. To showcase some of the improvements, let me give a few examples. Ultimately, I won’t be able to give a comprehensive overview (there were over 900 rustc test files changed!), but hopefully these examples can provide some representation. For these examples, I’ll give a code snippet, the output on stable 1.61 and on the current nightly.
The first error change highlights nicely the difference in how the compiler processes lifetime errors before and after this change.
Given the following snippet:
fn transmute_lifetime<'a, 'b, T>(t: &'a (T,)) -> &'b T {
match (&t,) {
((u,),) => u,
}
}
fn main() {
let y = Box::new((42,));
let x = transmute_lifetime(&y);
}
Stable emits the following error:
error[E0495]: cannot infer an appropriate lifetime due to conflicting requirements
--> src/main.rs:2:11
|
2 | match (&t,) {
| ^^^^^
|
note: first, the lifetime cannot outlive the lifetime `'a` as defined here...
--> src/main.rs:1:23
|
1 | fn transmute_lifetime<'a, 'b, T>(t: &'a (T,)) -> &'b T {
| ^^
note: ...so that the types are compatible
--> src/main.rs:2:11
|
2 | match (&t,) {
| ^^^^^
= note: expected `(&&(T,),)`
found `(&&'a (T,),)`
note: but, the lifetime must be valid for the lifetime `'b` as defined here...
--> src/main.rs:1:27
|
1 | fn transmute_lifetime<'a, 'b, T>(t: &'a (T,)) -> &'b T {
| ^^
note: ...so that reference does not outlive borrowed content
--> src/main.rs:3:20
|
3 | ((u,),) => u,
| ^
For more information about this error, try `rustc --explain E0495`.
Note that the primary location of the error points at the scrutinee ((&t,)) of the match. Also note that this error is pretty verbose and takes up a whopping 30 lines. On nightly:
error: lifetime may not live long enough
--> src/main.rs:3:20
|
1 | fn transmute_lifetime<'a, 'b, T>(t: &'a (T,)) -> &'b T {
| -- -- lifetime `'b` defined here
| |
| lifetime `'a` defined here
2 | match (&t,) {
3 | ((u,),) => u,
| ^ function was supposed to return data with lifetime `'b` but it is returning data with lifetime `'a`
|
= help: consider adding the following bound: `'a: 'b`
This is quite different. First, it’s much more terse in output; only taking up 12 lines. Second, the error points primarily at u, which actually is what get returned by the match expression. Finally, we get an actual helpful suggestion to add the 'a: 'b bound, which actually does fix this code. On the other hand, we do lose the “expected XX found XX” note, but it’s not super helpful in this case anyways. Also note that we lose the error code (E0495), which is unfortunate as this might be able to point the user to a more detained explanation.
Here’s a slightly different code snippet:
pub fn opt_str<'a>(maybestr: &'a Option<String>) -> &'static str {
if maybestr.is_none() {
"(none)"
} else {
let s: &'a str = maybestr.as_ref().unwrap();
s
}
}
The error on stable isn’t too bad and fairly straightforward:
error[E0312]: lifetime of reference outlives lifetime of borrowed content...
--> src/lib.rs:6:9
|
6 | s
| ^
|
= note: ...the reference is valid for the static lifetime...
note: ...but the borrowed content is only valid for the lifetime `'a` as defined here
--> src/lib.rs:1:16
|
1 | pub fn opt_str<'a>(maybestr: &'a Option<String>) -> &'static str {
| ^^
For more information about this error, try `rustc --explain E0312`.
And on nightly:
error: lifetime may not live long enough
--> src/lib.rs:6:9
|
1 | pub fn opt_str<'a>(maybestr: &'a Option<String>) -> &'static str {
| -- lifetime `'a` defined here
...
6 | s
| ^ returning this value requires that `'a` must outlive `'static`
We give all the same infomration in a slightly shorter error. We also lose the error code (E0312), which again is unfortunate. As you might notice though, what were two very different error outputs before look a bit more similar now: we do slightly less arbibtrary grouping.
This next error is a little less straightforward. In fact, this is a case we actually regress a bit, but not too bad.
use std::sync::Mutex;
struct MyString<'a> {
data: &'a str,
}
fn i_want_static_closure<F>(a: F)
where
F: Fn() + 'static,
{
}
fn print_string<'a>(s: Mutex<MyString<'a>>) {
i_want_static_closure(move || {
println!("{}", s.lock().unwrap().data);
});
}
Before I show the errors, let me just say: there’s a lot going on here. However, the crux of the error is the 'static' bound on F on line 7. So…stable:
error[E0477]: the type `[closure@src/lib.rs:11:27: 13:6]` does not fulfill the required lifetime
--> src/lib.rs:11:5
|
11 | i_want_static_closure(move || {
| ^^^^^^^^^^^^^^^^^^^^^
|
note: type must satisfy the static lifetime as required by this binding
--> src/lib.rs:7:15
|
7 | F: Fn() + 'static,
| ^^^^^^^
For more information about this error, try `rustc --explain E0477`.
Okay, great. This is exactly what we want: The closure is a problem, and it’s because we don’t satisfy that 'static' bound. Nightly:
error[E0521]: borrowed data escapes outside of function
--> src/lib.rs:11:5
|
10 | fn print_string<'a>(s: Mutex<MyString<'a>>) {
| -- - `s` is a reference that is only valid in the function body
| |
| lifetime `'a` defined here
11 | / i_want_static_closure(move || {
12 | | println!("{}", s.lock().unwrap().data);
13 | | });
| | ^
| | |
| |______`s` escapes the function body here
| argument requires that `'a` must outlive `'static`
|
= note: requirement occurs because of the type `Mutex<MyString<'_>>`, which makes the generic argument `MyString<'_>` invariant
= note: the struct `Mutex<T>` is invariant over the parameter `T`
= help: see <https://doc.rust-lang.org/nomicon/subtyping.html> for more information about variance
For more information about this error, try `rustc --explain E0521`.
Well…this isn’t great. We at least get the “'a must outlive 'static”, which might give some clue to the problem. But, we also get a couple notes about variance. This is not really helpful here - it doesn’t have any effect on the error (even though it is true).
Okay, let’s pick out an error that actually does stem from variance and how it has changed. We’ll have this be the last one.
struct SomeStruct<T>(*mut T);
fn foo<'min, 'max: 'min>(v: SomeStruct<&'max ()>) -> SomeStruct<&'min ()> {
v
}
Just quickly, I want to point out the *mut T in SomeStruct. If this was only T, this snippet wouldn’t have any problem. See if you can find out why from the errors (I did give a hint above). On stable:
error[E0308]: mismatched types
--> src/lib.rs:4:5
|
4 | v
| ^ lifetime mismatch
|
= note: expected struct `SomeStruct<&'min ()>`
found struct `SomeStruct<&'max ()>`
note: the lifetime `'min` as defined here...
--> src/lib.rs:3:8
|
3 | fn foo<'min, 'max: 'min>(v: SomeStruct<&'max ()>) -> SomeStruct<&'min ()> {
| ^^^^
note: ...does not necessarily outlive the lifetime `'max` as defined here
--> src/lib.rs:3:14
|
3 | fn foo<'min, 'max: 'min>(v: SomeStruct<&'max ()>) -> SomeStruct<&'min ()> {
| ^^^^
For more information about this error, try `rustc --explain E0308`.
So…this error is okay. We get told that we’re expecting SomeStruct<&'min ()> and that 'min doesn’t outlive 'max. But we don’t really know why. There’s an error code, but it’s pretty generic. Let’s look at nightly:
error: lifetime may not live long enough
--> src/lib.rs:4:5
|
3 | fn foo<'min, 'max: 'min>(v: SomeStruct<&'max ()>) -> SomeStruct<&'min ()> {
| ---- ---- lifetime `'max` defined here
| |
| lifetime `'min` defined here
4 | v
| ^ function was supposed to return data with lifetime `'max` but it is returning data with lifetime `'min`
|
= help: consider adding the following bound: `'min: 'max`
= note: requirement occurs because of the type `SomeStruct<&()>`, which makes the generic argument `&()` invariant
= note: the struct `SomeStruct<T>` is invariant over the parameter `T`
= help: see <https://doc.rust-lang.org/nomicon/subtyping.html> for more information about variance
Wow, that has a lot more information (and it’s also a bit more terse). And we get a suggestion (that works)! Does this help you understand why the code snippet is problematic? We can’t substitute SomeStruct<&'max ()> for SomeStruct<&'min ()> in the return type, because T is invariant. If T was covariant, we could.
I’m not sure if this was too much - or too little. It’s also most just pasting a few code snippets and their errors, with a bit of commentary. Hopefully it at least scratches the surface of the types of lifetime error differences you might see on nightly compared to stable. If you’re interested in checking out how lifetime errors in rustc test suite have changed, feel free to check out the PR or take a look at the short list of diagnostic regressions.
So, for this section, I’m going to work backwards. This is primarily because it’s just easier to recall recent work than older work. I want to just briefly point out some of the major stepping stones towards this stabilization and shout out the various contributors that pushed to make this happen.
The stabilization PR had 985 changed files. Most of those were test-related. It’s impractical to expect a reviewer to go through all of those changes and “sign off” on them all. Adding to that, with the way the rustc test suite is set up, changing test error output requires some manual intervention, so it would be helpful to make such a big PR be as mechanical as possible.
Over the past couple months @marmeladema and I have worked to go through all the test error differences, file issues for those where diagnostics have regressed (subjectively), and prepare the tests for a purely mechanical stabilization PR.
@m-ou-se also triaged the crater run for the stabilization PR to give the stabilization a clean bill of health.
There are three parts to this.
First, @marmeladema and @Aaron1011 made some great PRs over the last couple months to reduce the number of diagnostic regressions.
Second, one of the last “big questions” prior to this stabilization was regarding to mutable_borrow_reservation_conflict lint (this actually came up in the crater run mentioned above). Without going into details, there were concerns that allowing the linted pattern would disallow some optimizations. It accidentally got stabilized in Rust 2018 and the thought was maybe to disallow it after full NLL stabilization. The lang team decided against this and the lint was removed.
Finally, the last bit of known behavior difference with the full stabilization concerned coercion with match statements and the order of match expressions. Actually, in migrate mode, the coercion here fails. Fully enabling NLL allows the pattern to compile, but only in one ordering. While we do want the coercion to compile eventually, we chose for now to just fallback to migrate mode behavior of disallowing the coercion in either ordering.
Up until mid last year, progress on enabling NLL had mostly stalled. The biggest and most daunting blocker being a slew of hideus and generic “higher-ranked subtype error” for a large number of lifetime errors. @matthewjasper did some fantastic work in a branch that @lqd rebased and made a PR for. It didn’t solve all the errors, but it provided a framework to solve most of the remaining problems. @lqd followed up with another PR which fixed more diagnostics.
Then, @Aaron1011 made a long series of PRs addressing various diagnostics. At this point, NLL was very close to being “ready” and just needed a few loose ends tied up.
In 2018, migrate mode was added and enabled for the Rust 2018 edition by @pnkfelix. However, if code was rejected by NLL, but allowed by the AST borrowck, issues were only emitted as warnings. @spastorino also made the AST borrowck not run at all under feature(nll).
Over the period of 2019, lots of shifting happening in the migration from the AST borrowck to the MIR borrowck (NLL). In March, @matthewjasper enabled migrate mode in Rust 2015, but with warning emitted instead of errors. @chrisvittal removed the mostly unused -Zborrowck=compare flag. Then @Centril made a series of PRs that upgraded migrate mode NLL warnings to errors in Rust 2018 and Rust 2015, then finally removed the unused AST borrowck.
In the latter half of 2017 and into 2018, work was done to implement the MIR borrowck and bring it to feature and performance parity. @nikomatsakis envisioned most of the borrow checker and did a large portion of the implementation work. However @Nashenas88 made a few PRs to prepare the compiler for Niko’s branch. @pnkfelix and @shepmaster did great work tracking progress. @nnethercote did some amazing work speeding up the implementation. It’s very possible I’ve missed more, so if you see someone missing from this list, please DM me and I’ll correct it.
That basically concludes this post. As always, this turned out a bit different from how I imagined it in my mind. But I do think it’s important to recount the history of the implementation and give credit to those that did the work to get where we are today. As a side effect, maybe a few people might find it interesting or get some use from going over a couple lifetime error differences.
As a side note, I do welcome feedback on these. I tend to make my blog posts more “informal” and “prose-like”, versus a more rigid structure. The downside of this is that they can be less “beginner-friendly”, in a sense. I would love to know if people enjoy these or not.
]]>Before I go further, if you don’t know what GATs (generic associated types) are, then I recommend reading this blog post from August of last year.
A bit over a year ago, the traits working group started to talk seriously about stabilizing GATs. I and others worked to close issues and move that forward. In June last year, I estimated October for stabilization. Well, turns out there were some things that we needed to do first (besides fixing bugs), like implementing an outlives lint and changing the location of where clauses on GATs. In December, once again, I said “we’re close.” Around Feburary or March, we felt like we were finally in a place to stabilize. Except for one small thing: this blog post.
The goal of this blog post is to try to imagine a “shiny future” where we have GATs. It’s to consider a couple of the patterns that people want GATs for, how we might want to integrate GATs into standard library traits, and a few issues that don’t work today that we want to work in the future. We want this so that we have a future to strive for after stabilization. There are known shortcomings with GATs right now, but that’s okay. They’re already powerful and, as you’ll see, we have some thoughts on how to make them more powerful and ergonomic in the future.
Like I said, it’s been hard to write this. Partially because it’s a lot, but also because it’s difficult to think about the future and try to consider the intersection of a several different patterns. This isn’t to say that I alone have come up with these thoughts: much of this has been talked about within the traits working group. A huge shoutout to Niko Matsakis for an unenumerable amount of feedback, banter, and guidance.
Well, I’ve finally finished this post. To go with this, I’ve posted a stabilization PR for GATs to Github. It’s happening, y’all.
To start with, I want to cover a pattern involving traits with GATs more-or-less representing the “Self” type. I’ll start by introducing two separate but related examples. After I introduce them, I’ll explain a bit about how they are similiar. Finally, I’ll discuss a bit how a full “higher-kinded types” feature might solve this.
The first example roughly follows the Collection trait introduced by Niko Matsakis in a series of blog posts in 2016. Here I’ve made two changes. First, the type parameter T on Collection has been changed to an associated type - which more closely matches real collections like Vec or Option. Second, the Family trait has been removed also in favor of just an associated type on the Collection trait. Here’s what the new trait looks like an an impl for Vec:
trait Collection {
type Item;
type Member<T>: Collection<Item = T>;
type Iter<'iter>: Iterator<Item = &'iter Self::Item>
where
Self: 'iter;
fn empty() -> Self;
fn add(&mut self, value: Self::Item);
fn iter<'iter>(&'iter self) -> Self::Iter<'iter>;
}
impl<T> Collection for Vec<T> {
type Item = T;
type Member<U> = Vec<U>;
type Iter<'iter> = std::slice::Iter<'iter, T> where T: 'iter;
fn empty() -> Self {
Vec::new()
}
fn add(&mut self, value: T) {
self.push(value)
}
fn iter<'iter>(&'iter self) -> Self::Iter<'iter> {
use std::ops::Deref;
self.deref().iter()
}
}
You could also imagine you might have a function like
fn floatify<C>(ints: &C) -> C::Member<f32>
where
C: Collection<Item=i32>,
{
let mut res = C::Member::<f32>::empty();
let mut iter = ints.iter();
while let Some(v) = iter.next() {
res.add(*v as f32);
}
res
}
The second example revolves around mimicking the Functor typeclass from Haskell and functional programming. In Haskell, a Functor allows you to map the element(s) inside some type of container into something else. In Rust, imagine Option::map or Result::map, but generic over many types. There have been a few blog posts published on this topic, so again I’ll just put here a final trait and an impl (this time for Option):
trait Functor {
type Inner;
type This<B>: Functor;
fn map<F, B>(self, f: F) -> Self::This<B>
where
F: FnOnce(Self::Inner) -> B;
}
impl<T> Functor for Option<T> {
type Inner = T;
type This<B> = Option<B>;
fn map<F, B>(self, f: F) -> Self::This<B>
where
F: FnOnce(Self::Inner) -> B {
self.map(f)
}
}
If you compare these two traits, Collection and Functor, you might realize that they look oddly similar. Both have an associated item for the “item inside” (Item and Inner) and an associated item that represents the current type, but wrapping a generic item type (Member<T> and This<B>).
However, if you think about the way these traits are set up, you might notice a potential problem. There is nothing stopping someone from writing an impl like (assuming Result: Functor holds):
impl<T> Functor for Option<T> {
type Inner = T;
type This<B> = Result<B, ()>;
fn map<F, B>(self, f: F) -> Self::This<B>
where
F: FnOnce(Self::Inner) -> B {
self.map(f).ok_or(())
}
}
Namely, we expect that This<B> to be equal to Option<B> (the current container type with a new item type), but instead it is a Result! Maybe this is okay, but it’s certainly unexpected.
So, how can we solve this? This problem originates mostly because we’re trying to emulate higher-kinded types.
I don’t want to focus on the specific syntax that I’m using (because I’m basically making it up on the spot with little thought and it’s not very relevant), but here’s how you might write Functor with higher-kinded types:
trait Functor where Self<Inner> {
fn map<F, B>(self, f: F) -> Self<B>
where
F: FnOnce(Inner) -> B;
}
impl<T> Functor for Option<T> {
fn map<F, B>(self, f: F) -> Self<B>
where
F: FnOnce(T) -> B {
self.map(f)
}
}
Now, you might be asking how this relates to GATs. And, well, the answer is: it doesn’t, really. The solution I show above is pretty far from the GATs feature itself. But I wanted to cover this for two reasons: 1) The initial examples are likely patterns you will see with GATs, even with the limitation described. 2) In theory, the fundamental compiler features needed to support GATs are likely the fundamentals of the ones needed to be able to “real” higher-kinded types - or maybe something similar but more Rust-like.
IteratorThere are many cases today where people want to do something with Iterator, Fn, Deref, etc. that isn’t yet possible. Instead, what they require is really a GAT version of these traits - often with a lifetime parameter.
The question then becomes: what is the best way to make GAT versions of these traits available? I want to explore here Iterator, since that is arguably the most well-known example case. I want to look first at the pattern that requires GATs. Then, I want to introduce a LendingIterator, and explain why might be a less-than-ideal solution. I’ll show a bit about a couple of the things that it might take to replace Iterator with a GATified version (in a backwards-compatible manner). And I’ll end off the section with a somewhat crazy thought I was actually able to get working as I was writing this blog post.
SelfLet’s start by looking at what the current Iterator trait looks like:
trait Iterator {
type Item;
fn next(&mut self) -> Self::Item;
}
Pretty simple, but I’ve obviously left out a large chunk of methods with default implementations, like for_each, map, or peekable. I’ve left them out for simplicity, but they are important and I’ll talk about that in a bit.
To introduce the pattern that requires GATs, I’ll use the std::io::BufRead trait. This is implemented most clearly by the std::io::BufReader struct. In short, this trait represents a type that implements std::io::Read but also keeps an internal buffer, allowing it to be able to do extra things, like returning line by line.
I want to focus on a single method on BufRead:
pub trait BufRead: Read {
fn lines(self) -> Lines<Self>;
}
This function returns a struct Lines which implements Iterator with an Item of Result<String>. Let’s imagine it gets used like:
fn main() {
let r: BufReader<_> = ...;
let mut lines = r.lines();
while let Some(line) = lines.next() {
println!("{line}");
}
}
Now what’s wrong with this (aside from me being able to use a for loop - I’ll get there). It might seem like a philosophical question, but there really is something suboptimal: even though we only look at one line at a time, we allocate a new String for each one. Of course, this is the exact reason we need GATs. If we wanted to be able to reuse the same memory between calls to next, then we would have to ensure that nobody else hold a reference to that memory. But with the current definition of Iterator::next, the return time can outlive the call.
LendingIteratorLet’s introduce a Iterator-like trait that could handle this correctly:
trait LendingIterator {
type Item<'a> where Self: 'a;
fn next(&mut self) -> Self::Item<'_>;
}
Here, I’ve defined a new trait, LendingIterator, that mirrors Iterator, except for a new lifetime 'a on Item. It does exactly what we want:
struct Lines<B: BufRead> { ... } // Stores the state needed to read lines
impl<B: BufRead> LendingIterator for Lines<B> {
type Item<'a> = &'a str;
// The lifetime of the return type is tied to the lifetime of the call
fn next(&mut self) -> Self::Item<'_> { ... }
}
fn read_two_lines<B: BufRead>(mut lines: Lines<B>) {
while let Some(line) = lines.next() {
// `line` is only a `&str`
// The reference is dropped before the next call
println!("{line}");
}
// We also store two lines at a time:
let line1 = lines.next();
let line2 = lines.next(); // <- Error: the lifetime from `line1` is still active
println!("{line1} {line2}");
}
So, this looks great. Why is this imperfect? Well, remember all those methods on iterator with default implementations? We now need to rewrite those. Also, any APIs that take Iterator but don’t read more than one Item at a time would need to be to changed or duplicated to take LendingIterator.
Iterator in a LendingIterator waySo, maybe we don’t want a completely separate LendingIterator trait from Iterator. What can we do instead? Well, what if we just added a 'a parameter to Iterator::Item. Let’s go through some of the different problems that would have to be worked through.
To start, I want to actually just introduce some code, which will help me explain things after.
fn from_iter<A, I>(mut iter: I) -> Vec<A>
where
I: Iterator<Item = A>,
{
let mut v: Vec<I::Item> = vec![];
while let Some(item) = iter.next() {
v.push(item);
}
v
}
This is completely fine. In fact, it’s mostly the same as the FromIterator implementation for Vec.
Let me also show what this would look like with a LendingIterator. Keep in mind that we want this to be just Iterator with a lifetime parameter on Iterator::Item. I’m keeping them separate here to keep comparisons easy to understand.
fn from_iter<A, I>(mut iter: I) -> Vec<A>
where
I: for<'a> LendingIterator<Item<'a> = A>, // (1)
{
let mut v/*: Vec<I::Item>*/ = vec![]; // (2)
while let Some(item) = iter.next() {
v.push(item);
}
v
}
This also works, but differs from the Iterator case in two places, which I’ll talk about next.
Note: It’s nice to see that borrow checker knows that the item being iterated over doesn’t capture the self lifetime from next; with this, we can store item in the Vec without problems. This makes intuitive sense and it’s good to see that this isn’t a problem we have to work through.
I: for<'a> LendingIterator<Item<'a> = A>Let’s first look at the line labeled (1) above. What is this for<'a> business? Well, we have to have some way to denote the where clause “the item being iterated over is A, regardless of the lifetime passed.” In other words, the type being iterated over can’t name the lifetime passed. Therefore, it acts exactly like a non-generic associated type.
Now, let’s remember that we want to replace the definition of Iterator with the definition of LendingIterator. So, we need to make I: Iterator<Item = A> “desugar” into I: for<'a> Iterator<Item<'a> = A>. Semantically, this is actually fairly straightforward.
Iterator::ItemNow let’s look at the line labeled (2).
In the Iterator example, we can name the type of the item being iterated by doing I::Item. With LendingIterator though, we can’t do this, because I::Item isn’t a type, it’s a type constructor. To get an actual type, we have to use a lifetime, like I::Item<'static>. This analogous in that you can’t just write let x: Vec;, you have to write something like let x: Vec<()>;.
Now, you might say “oh, but you could just write Vec<A> or Vec<I::Item<'static>> and it will be the same” and you would be correct: That would be fine. But if only it was so simple.
Again, let’s remember the goal of converting Iterator. We can’t suddently just break all the code out there. So, we have to figure out how to make v: Vec<I::Item> work, even if Iterator::Item has a lifetime parameter. For the example above, we could just handwave and see that we know that it doesn’t matter what lifetime we pass to I::Item, since A can’t bind it. So, we could “desugar” this to either I::Item<'static>, some higher-ranked for<'a> I::Item<'a>, or maybe some “empty” lifetime I::Item<'empty>.
Let’s look at this example:
fn from_iter<I>(mut iter: I) -> Vec<I::Item>
where
I: LendingIterator,
{
...
}
Well, this is more complicated. The code as written doesn’t make much sense. Of course, if this were just a “clean” LendingIterator trait, then we just maybe want to return Vec<I::Item<'static>> and all is well and good (we have to pass some lifetime to the GAT). But, if we imagine that we just took Iterator and added a lifetime parameter, it becomes clear that we need some sort of implicit I: for<'a> Iterator<Item<'a> = A> for Iterator.
But now if Iterator::Item implicitly can’t capture a lifetime, we have to allow code to opt-in to that. What might the syntax for that look like? To be honest, I’m not really sure. To extend upon this, you likely don’t want all code using traits with GATs to have to opt-in to being able to allow the GATs to name a lifetime. So, Iterator would probably be special in some way.
I would love to hear thoughts on how you would like to interact with a GATified Iterator.
As I was writing this blog post, I played around with an idea that was brought up at one point or another. I couldn’t get it to work then, but was able to get something to work, at least a little bit this time.
So, I’m going to spitball this potentially crazy idea, if for no other reason than to let people go “what the heck is he thinking” and pick it apart.
So bear with me here. What if we do have two traits, Iterator and LendingIterator, but they look something like this:
pub trait LendingIterator {
type Item<'a> where Self: 'a;
fn next(&mut self) -> Option<Self::Item<'_>>;
}
pub trait Iterator {
type Item;
fn next(&mut self) -> Option<Self::Item>;
}
impl<'i, I: Iterator + 'i> LendingIterator for I {
type Item<'a> = I::Item where I: 'a;
fn next(&mut self) -> Option<Self::Item<'i>> {
Iterator::next(self)
}
}
In words, every implementation of Iterator automatically implements LendingIterator with the items equal.
I think this mostly works.
Here are couple examples of how you might use these:
fn print_items<I>(mut iter: I)
where
I: LendingIterator,
for<'a> I::Item<'a>: std::fmt::Debug,
{
while let Some(item) = iter.next() {
println!("{item:?}");
}
}
fn collect_items<I>(mut iter: I) -> Vec<I::Item>
where
I: Iterator,
{
let mut v = vec![];
while let Some(item) = iter.next() {
v.push(item);
}
v
}
Importantly, you could change the top function to use Iterator and it works (though it accepts fewer types, since we accepted all Iterator types already because of the blanket impl). You cannot change the bottom function to use LendingIterator, since that would end up with overlapping mutable borrows.
There are couple pain points here that I’ve found as I played with it a bit; let me go through them.
First, you can’t actually call next() on a type that implements Iterator without disambiguating the next function of Iterator and LendingIterator. The “simple” solution is to rename LendingIterator::next to something like lend_next. I’ve tried making the defintion of Iterator be Iterator: LendingIterator, but haven’t gotten it to work.
The other pain point (which might actually be good), is that all (already existing) functions don’t automatically get to take a LendingIterator. I think changing from I: Iterator to I: LendingIterator is a backwards compatible change, but haven’t worked through it to be sure. Related to this, if your impl of LendingIterator just so happens to not name the lifetime parameter on Item, you can’t use a function that takes Iterator. But again, maybe this is the right thing.
And, of course, this doesn’t generalize to other traits with GATs.
So, yeah. This was my crazy revelation while writing this. Here’s a playground link where I’ve added some other examples, if you’re interested in picking it apart: https://play.rust-lang.org/?version=nightly&mode=debug&edition=2021&gist=7647fd487f1733fe97fa55c38e4071ab
for<'a> Self: 'a problemIt’s interesting, just earlier this week an absolutely beautiful blog post was published by Sabrina Jewson (link) that mostly covered this problem, and gave some good and interesting solutions.
I’ll briefly restate the problem, but if you’re interested in a more “user-friendly” explantation defintion go check out Sabrina’s post; it’s great. I’ll use the same LendingIterator example Sabrina uses, the problem definitely pops up in even more subtle places.
So, the problem in the LendingIterator comes from the interaction of two pieces of code.
First, there is the Self: 'this bound on the Item associated type:
pub trait LendingIterator {
type Item<'this> where Self: 'this;
// ^^^^^^^^^^^ this one
...
}
Second, we encounter a piece of code where we reference an GAT using a lifetime from a HRTB:
fn print_items<I>(mut iter: I)
where
I: LendingIterator,
for<'a> I::Item<'a>: Debug, // here
{
while let Some(item) = iter.next() {
println!("{item:?}");
}
}
As noted in Sabrina’s post, the problem comes because we try to prove that for<'a> I: 'a, meaning that we require for any lifetime that we pick, I must outlive it. This isn’t really right, because ultimately, we don’t need to prove that I can outlive any lifetime in this context. We only need to prove that I outlives any lifetime that we could have provided if the GAT was valid (in this case, this is only true if I: 'a, which makes for<'a> I: 'a trivially provable).
This is a bug. And one that is fixable in an almost certainly backwards-compatible manner. But it’s also tough to fix. So it’s not something we want to block stabilization on.
To give a brief teaser to what a fix might look like, you might want to check out Niko Matsakis a-mir-formality. The key part of this formalism of the Rust type system (that differs from the current rustc and Chalk solvers), is the introduction of the “ensures” clause and the separation from an “implication” clause. If I understand the model correctly, I think this is crucial to being able to model and solve this problem.
Right now, you can’t use GATs with dyn. In other words, the following doesn’t work:
fn print_items(items: &mut dyn for<'a> LendingIterator<Item<'a> = &'a str>) {
for item in items {
println!("{item}");
}
}
We expect this to work just fine one day, but there are still some implementation design work to be done.
Hopefully this post provides a little bit of a glimmer of a shiny future with GATs. I wanted to make this post more “glamorous” but really just fell short on time. I might publish a part 2 of this at some point in the coming months; we’ll see.
Ultimately though, I don’t think I can do justice to the many different use cases of GATs and the power they provide. If you’d like to get a sense of the many projects that are “waiting” for GATs, just take a scroll through the tracking issue and see the numerous issues from external projects linking to it. I can’t wait to see the types of projects people build with them.
If you feel like I’ve missed anything, please feel to reach out.
]]>So, first, if you aren’t familiar with the traits working group, the tl;dr is that it’s traditonally been dedicated maintaining and improving rustc’s trait system design and implementation (including writing a new trait solver, Chalk). In recent times, the scope has, in practice, broadened a bit to include the typesystem overall. Stayed tuned for fun things about this soon.
The last real “public” post from the traits working group was at the end of our last sprint…in July 17 of 2020. We certainly have been busy since then though. This post will mainly focus on the work done in 2021, but I will do my best to call out notable points from the latter half of 2020.
There will be roughly two kinds of work highlighted here: that which falls under a lang team initiative and that which doesn’t, but aligns with longer-term goals of the traits working group. I originally wanted this post to be a larger overview of the different projects and work, but in the interest of just “getting this out”, I’ll likely keep things fairly brief in explanations. If you’re interested in any of the work you see here, I urge you to either look through the cited pull requests or issues, to browse the initiatve repositories, or to come join us on Zulip and have a chat.
If you find something I’ve missed here, or any other mistake, please feel free to message me on Zulip; I really don’t want someone to feel like I’ve left them out intentially and I want them to get the credit they deserve. With that being said, let’s get started.
(The issue numbers referenced in this post refer to issues on the rustc repository).
For each of these initiatives, I’ll give a brief snippet as to what the initiative is for, as well as list the people who have contributed (also listing pull requests made and issues fixed) to each. Importantly, there are certainly going to be people that I miss here, either because they haven’t directly contributed code (but have been involved in discussion or design) or because they help out tangentially (making PR rollups, adding regression tests, etc.). I do want to specifically thank them now; that work is important too.
I also want to specifically call out one particular person who will be underrepresented below, but deserves recognition: Niko Matsakis. Though he hasn’t made many pull requests for these, he serves as the lang team liason for basically all of them and is instrumental in design and implementation discussions for this work. He also does amazing work co-leading the traits working group with me.
For a nice introduction to what GATs are, I’ll refer you to a blog post I wrote last August. However, in a few words: it’s a really neat and powerful language feature that is nearing stabilization. Essentially, it allows you to have generics on associated types in traits (surpisingly given the name). Since the blog post in August, there have been many bug and diagnostic fixes made. We missed the “next couple months” hope for stabilization from the blog post, but that’s okay. While I won’t cover all the work done this year (which is siginficant too), you’ll see that 2021 was a busy year in terms of GAT work. We expecting to likely stabilize next release cycle (well, that nightly).
b-naber PRs: #79554, #82272, #90801 Issues: #67510, #68648 #68649, #68650, #68652, #74684, #76535, #79422, #80433, #81801, #81961, #81862, #84439, #90612
lcnr PRs: #80558 Issues: #69184, #80766
BoxyUwU PRs: #81911 Issues: #75415, #79666
Jack Huey (jackh726) PRs: #86993, #87244, #87281, #84622, #87478, #85499, #89914, #87900, #88846, #88441, #89823, #91849, #89970, #88336, #87478, #84623, #90076, #90887, #91853, #92118, #92191 Issues: #76407, #76826, #78113, #81487, #81823, #84931, #85921, #86787, #87429, #87762, #88360, #88459, #89639, #91036, #90888, #91348
Giacomo Stevanato (SkiFire13) PRs: #85375 Issues: #85347
Yuki Okushi (JohnTitor) PRs: #86505 Issues: #86483
Oli Scherer (oli-obk) PRs: #89229 Issues: #87258, #88595
Audun Halland (audunhalland) PRs: #89341 Issues: #89188
Unattributed #79949, #79636, #78671, #70303, #70304, #71176, #81712, #79768, #87750, #88287, #88405, #90014
This feature essentially allows you to do something like
trait AwesomeTrait {}
type MyType = impl AwesomeTrait;
Basically, rather than requiring you to specify the exact type for MyType, it can be inferred from how it’s used in the module it’s defined. This means to users of e.g. a library it’s “opaque” - all you know is that it impls AwesomeTrait. This is also especially nice when the type can’t be named, like for closures.
Santiago Pastorino (spastorino) PRs: #86118, #87141, #87501 Issues: #73481, #77179, #85113
Oli Scherer (oli-obk) PRs: #87287 #89229, #87107, #87200, #87287, #82898, #87587, #89045, #89024, #90376 Issues: #74280, #88595
b-naber PRs: #85755 Issues: #83190, #78450
tmiasko PRs: #81008 Issues: #80998
Esteban Kuber (estebank) PRs: #83954 Issues: #83613
Niko Matsakis (nikomatsakis) PRs: #84701 #86437
Unattributed #63355, #63591, #65384, #69323, #86201, #88287, #90014
Have you ever tried to do something like only to have rustc yell at you?
trait Foo {}
trait Bar: Foo {}
let bar_type: &dyn Bar = ...;
let foo_type: &dyn Foo = bar_type;
Well, with this feature, the above code just works.
Charles Lew (crlf0710) PRs: #86264, #86291, #86461, #86475, #88135, #90536 Issues: #89190, #86324, #90177
For this, I’ll actually give on example how we expect this will be used in the standard library. Currently, the Error trait is defined in std. It would be nice to move in to core. However, we want to write the following impls:
impl From<&str> for Box<dyn Error> {}
impl<E> From<E> for Box<dyn Error> where E: Error {}
But we can’t, because we don’t know that &str: Error will never hold (the way coherence in Rust works, upstream crates are allowed to add new impls). However, if we could add the following to core, it would be fine:
impl !Error for &str {}
This feature allows that.
Santiago Pastorino (spastorino) PRs: #90104
Without getting into the nitty-gritty details, there was a bug related to incremental compilation and associated type projection caching. Fixing that bug led to some severe performance problems in some cases, so there was additional followup work to fix that too.
Aaron1011: #85382, #85868, #88945, #88994, #89125, #89831, #90423, #92041 lcnr: #84944 the8472: #91186
This work fixed a large class of ICEs that have existed since Rust’s infancy. The gist of it is that the following code would have problems.
fn function<F, T>(f: F)
where
F: for<'a> FnOnce(<T as Trait<'a>>::Assoc),
T: for<'b> Trait<'b> {}
Basically, because of the for<'a>, the <T as Trait<'a>>::Assoc type could normalize in parts of rustc, but not in others.
jackh726: #85499, #90801, #90017, #89285, #88441, #86993
This is mostly an internal implementation detail in rustc. Essentially, it changes the way we track “bound variables”, like the 'a in for<'a> T: Trait<'a>. Changing the way we track these didn’t, on its own, fix anything. However, it did identify some places where we were doing the wrong things and should hopefully make rustc more resilient to these bugs in the future. (It also likely is a prerequiste for Chalk integration.)
jackh726: #76814, #83870, #83944, #84377, #84559
For those unfamiliar, Chalk is essentially an implementation of the Rust trait system using a prolog-like solver. While progress here has somewhat slowed as we focus a bit on some rustc things (like GATs, TAITs, etc.), Chalk - and the fundamental idea of “formalizing” the type system and trait solver - remains a strong goal of the traits working group. I’ve tried to list all the people that contributed to Chalk in the latter half of 2020 and in 2021.
0xflotus, Aaron1011, Areredify, AzureMarker, daboross, HKalbasi, JakobDegen, LeSeulArtichaut, Veykril, basavesh, crlf0710, davidbarsky, detrumi, ectastic-morse, eggyal, ehuss, firefighterduck, flodiebold, henrikhorluck, jackh726, josh65536, lf-, matthewjasper, memoryleak47, memoryruins, nathanwit, nikomatsakis, nrc, pierwill, scalexm, spastorino, super-tuple, vandenheuvel
Again, I wish I had made this post earlier in the year. There’s already so much happening in 2022, and not talking about it is very difficult (though always feel free to reach out on Zulip if you’re interested). However, I really wanted to give credit to all the people above who have done great work.
]]>This is a response to the Rust call for blog posts for 2021. Now, I do co-lead the traits working group and have contributed to Chalk a fair bit over the past year. But, I do want to put a disclaimer that the points I may make in this post are my own and I’m not speaking in any “official” capacity here. I wanted to make this post in case others find my thoughts useful. And, frankly, I figured it would be fun.
Well, onto the contents of the post itself.
Okay, let’s start with some background. My day job is actually being a graduate student studying bioinformatics. In this field, it’s really a mixed bag of programming languages that you see. Python and bash/shell scripts are extremely common for data munging, processing, etc. Python also is used often for machine learning. R is used often for plotting, but also some data analysis. Anytime you need more raw performance, you see a lot of C or sometimes C++. In the work I do, I also heavily use Javscript/Typescript for web development, with a mixing of Kotlin. I use Rust for one (unreleased) project, but that’s all so far.
A lot of the libraries and software for bioinformatics are used because other people use them; they’ve been peer-reviewed and published, and as more people use them, the more of the de facto standard they become. This isn’t bad necessarily: there are plenty of libraries and software that are well-maintained, with easily-accesible source code, that does exactly what you need it to do. But, there are also plenty of libraries and software that isn’t well-maintained, the source is hard to find, or you need it to do something different.
Now, you might be asking: “how does this relate to Rust?” Well, in a way, it does and it doesn’t. In some sense, the conclusion from the past couple paragraphs is “I want more libraries and software for bioinformatics written in Rust.” But why? If you think the answer is “because I like the language and like writing in it”, you would be partially correct. But, really, it’s because that I think Rust has a lot to offer here:
Python and bash are generally okay for data munging and processing, but sometimes you really just want some type safety. R is generally excellent for plotting, but…well I’m not even going to try to list all the issues here. As far as data analysis tools written in R: have you ever had to hunt down documentation, unsure of what exactly that function does? Something like rustdoc would be great here. Performance-sensitive libraries and software are where I feel like Rust will shine the most; I would probably be preaching to the choir here if I started to list off reasons why. And finally, for web development related work, particularly server-side, it would be great to have better predictability for the memory and CPU requirements needed.
Eek, that was a longer background than I expected. I guess the tldr is: I like Rust and I think it’s a language that offers a lot of promise for many different types of applications. But, at least for bioinformatics, there’s more work to be done to build up libraries.
Now it’s time for the meat and potatoes: what I want to see from the Rust programming language in 2021. I already mentioned in my background that I feel like more libraries are needed in Rust, particularly around bioinformatics. But, in my opinion, that will come in time, and there’s not much that can be done other than making the language something that the people writing those libraries want to write in.
What I do want to talk about can maybe be summed up into a single word: stability. This is very much a loaded word, but in this post, I’ll talk about two separate parts of this.
I recently saw a reddit post of someone asking about Rust’s stability. But, in their post, they also said a different word: mature. What exactly is a mature language? A mature library? Something that doesn’t add many new features? Well, then surely C++ wouldn’t be called mature then (since there are numerous new features in C++20, for example). One that doesn’t have a lot of bugs? Well, good luck.
One way you could imagine maturity is like this: If the maintainers of the language or library stopped working on it today, how bad would that be. Under this definition, you can imagine something like the C language being very mature; the core language hasn’t change much in a long time. A library that has been written and does everything it’s supposed to without (more than perhaps a couple) bugs is probably considered mature. C++11? Probably mature. C++20? Probably not. Maturity is relative and it’s subjective.
Maturity in a language is important. If you ask the question: “is this language going to be around in 20 years”, the list of languages that you would feel a confident “yes” for is probably small. In my opinion, Rust isn’t quite there, yet. But what are some things that can be done to bring us there?
First, let’s think about the standard library. I want to make sure before I go forward that the number of different APIs in std doesn’t exactly indicate it’s maturity or not: a standard library with “all the batteries” doesn’t mean it’s stable and a standard library without them doesn’t mean it’s not. Instead, std maturity is more about having better knowledge of what programming patterns are used often enough that they should be in std; and importantly, what shouldn’t be. There are plenty of unstable library features where it’s up in the air whether they should be stabilized or removed. And there are plenty of “wanted” features that just aren’t in std yet. This is not to say there won’t always be some API that someone wants, but I do feel like we’re still not quite “mature”. Quick point though, I would say that 98% of the current std is excellent and probably falls under the “mature” category.
Next, we can think about language features. There are number of features people
want to see: GATs, const generics, specialization all come to mind. In some form
or another, these each have partial implementations. There are other areas of
the language where the design hasn’t even been decided on (see the RFC
repository). Maturity here means pushing these language features through. It
means stabilizing pieces that work (e.g. min_const_generics or
min_specialization). It means working through the design of interacting with
bits and pieces of the language that today are somewhat “magic” (e.g.
traits/vtables or dynamically sized types). Of course, this can’t all happen in
2021, but that’s okay. It’s much better to get these designs right than to get
them fast. In 20 years, it isn’t going to matter if we had these features in
2021 or 2023.
Finally, libraries. A lot of libraries right now are 0.*; i.e. they aren’t
committing to semver compatibility. Part of this is due to the previous two
points, but part of it isn’t. I would really like to see more libraries hit
1.0, but again this will come in time.
Before I continue with this, I do want to point out that Niko Matsakis previously wrote a great blog post covering some of this earlier this year. As he put it, “the basic idea is to refactor the compiler into a set of independent libraries, all knit together by the query system.” In my view, this is super important. In the blog post, he covers two points:
First, librarification can mean that tools can use the components of the rust compiler itself to do analysis. For example, rust-analyzer could share the type checker with rustc. This is great because it means that the tools to do static analysis, code refactoring, etc. all become better or easier to implement. You wouldn’t have to worry “am I covering all the types here” because the types you use are the types the compiler uses.
More importantly, in my opinion, librarification also means that each individial library is more accesible than everything as a whole. The rustc codebase as it is now is…intimdating. There’s been a huge surge of work fairly recently to help make it more accesible, but at least for now, the compiler is still all together. Making the compiler more accessible to contributors is especially important when you consider Rust in 20 years. It’s a much better idea to have people be able to pick up parts of the compiler and contribute than to get stuck thinking of the compiler as a monolith. This in particular is near and dear to my heart, because we have had so much success with this with Chalk; this past year, we’ve had so much support, especially from people that don’t work on the rustc codebase at all. I feel like it’s a model that really works well.
Let’s talk about Chalk though. For a quick summary for those who may not now: Chalk is an implementation and definition of the Rust trait system using a PROLOG-like logic solver. The plan is to eventually integrate it into the compiler as the trait solver. Currently, there is a very experimental integration into rustc. My goal for Rust in 2021 is to get this integration to a place that people can start playing with it. I’m definitely not committing to anything here, but I’m super hopeful for this. And, I would love to start seeing other bits of the compiler start to be split into libraries.
Phew. That ended up being a lot. There’s more I could say. There’s so much praise I could give to the Rust language and the people that contribute their time and energy to it, whether it be to the compiler, the standard library, the docs, the libraries, the infrastructure, the community. At the end of the day, I want to see this language grow. I want to see it mature. And ultimately, 2021 is going to be another year of that. It’s been 5 years of Rust, so here’s to another 20+ more. 🥂
]]>