h1 Safe Pin Projections Through View Types
- 2022-03-04

“Pinning” is one of the harder concepts to wrap your head around when writing async Rust. The idea is that we can mark something as: “This won’t change memory addresses from this point forward”. This allows self-referential pointers to work, which are needed for async borrowing over await points. Though that’s what it’s useful for. In practice, how we use it is (generally) through one of three techniques:

• stack pinning ¹: this puts an object on the stack and ensures it doesn’t move.
• heap pinning: using Box::pin to pin a type on the heap to ensure it doesn’t move. This is often used as an alternative to stack pinning.
• pin projections: convert from coarse: “This whole type is pinned” to more fine-grained: “Actually only these fields on the type need to be pinned”.

“stack pinning” is the less common of the two operations, but we’re on a path to adding first-class support to it via the core::pin::pin! macro. For “pin projection” we don’t have a path towards inclusion yet, and we’re left relying on ecosystem solutions such as the pin-project crate² (see footnote for praise).

The status quo is not bad at all, but it’d be a shame if this is where we stopped. Pinning is useful for so much more than just futures, and perhaps by making it more accessible people could experiment with it more. In this post I want to share the beginnings of an idea I have for how we might be able to add pin projections as a first-class language feature to Rust.

What are pin projections?

As we mentioned: pin projections convert from coarse “this whole type is pinned” to more fine-grained: “Actually only these fields on the type need to be pinned”.

There’s a thorough explainer about this on the std::pin docs in the stdlib. But sometimes it’s better to show rather than tell. So let’s write a quick example using the pin-project crate. Let’s create a sleep future which uses some internal timer (work with me here ³), and guarantees that it panics if it’s polled again after it’s completed:

// Our main API which creates a future which wakes up after <dur>
pub fn sleep(dur: Duration) -> Sleep {
    Sleep {
        timer: Timer::after(dur),
        completed: false,
    }
}

// Our future struct. The `timer` implements `Future`,
// and is the type we want to keep fixed in memory.
// But the `completed` field doesn't need to be pinned,
// so we don't mark it with "pin".
#[pin_project]
pub struct Sleep {
    #[pin]
    timer: Timer,
    completed: bool,
}

impl Future for Sleep {
    type Output = Instant;

    fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
        assert!(!self.completed, "future polled after completing");

        // This is where the pin projection happens. We go from a `Pin<&mut Self>`
        // to a new type which can be thought of as:
        // `{ timer: Pin<&mut Timer, completed: bool }`
        let this = self.project();
        match this.timer.poll(cx) {
            Poll::Ready(instant) => {
                *this.completed = true;
                Poll::Ready(instant.into())
            }
            Poll::Pending => Poll::Pending,
        }
    }
}

If you haven’t seen any manual futures code before, that can be a lot to take in. It’s a goal of the Rust Async WG to make it so people need to write as little Pin code as possible to use async Rust. So if this overwhelming to you / you’re unsure what’s going on. Please don’t feel pressured you need to learn this.

But for folks who are curious: the interesting part here is the call to self.project(). This method is provided by the pin-project crate, and allows us to convert from Pin<&mut Self> to a generated type where only certain fields are pinned. I like to visualize it for myself like this:

// before projecting
self: Pin<&mut Sleep { timer: Timer, completed: bool }>

// after projecting
self: &mut Sleep { timer: Pin<&mut Timer>, completed: bool }>

Both provide the guarantee that types which need to be pinned in fact are pinned. But the “after projecting” example does so on a per-field basis, rather than locking the whole thing in-place. And if the whole struct is locked in place, it means we can’t mutate fields on it. Which in turn means we could never mark completed as true. So we have to project to write stateful futures.

View Types

In their post “view types for Rust” Niko talks about a new language construct called “view types” which can create a finer-grained interface for borrowing values. The whole post is worth a read, but to recap the example:

// This is the code we start with

struct WonkaShipmentManifest {
    bars: Vec<ChocolateBar>,
    golden_tickets: Vec<usize>,
}

impl WonkaShipmentManifest {
    fn should_insert_ticket(&self, index: usize) -> bool {
        self.golden_tickets.contains(&index)
    }
}

impl WonkaShipmentManifest {
    fn prepare_shipment(self) -> Vec<WrappedChocolateBar> {
        let mut result = vec![];
        for (bar, i) in self.bars.into_iter().zip(0..) {
            let opt_ticket = if self.should_insert_ticket(i) {
                Some(GoldenTicket::new())
            } else {
                None
            };
            result.push(bar.into_wrapped(opt_ticket));
        }
        result
    }
}

And it yields this error:

error[E0382]: borrow of partially moved value: `self`
   --> src/lib.rs:16:33
    |
15  |         for (bar, i) in self.bars.into_iter().zip(0..) {
    |                                   ----------- `self.bars` partially moved due to this method call
16  |             let opt_ticket = if self.should_insert_ticket(i) {
    |                                 ^^^^ value borrowed here after partial move

However if we change the definition of should_insert_ticket to be more fine-grained, we can tell the compiler that our borrows do not in fact overlap, which makes the borrow checker happy.

impl WonkaChocolateFactory {
    fn should_insert_ticket(&{ golden_tickets } self, index: usize) -> bool {
                          // ^ this is the "view type"
        self.golden_tickets.contains(&index)
    }
}

To me the core insight of “view types” is that by encoding more details in type signatures, the compiler is able to make better decisions about what’s allowed or not. Who’d have guessed?

View types for pin projection

So far we’ve looked at “pin projection” and “view types” individually. But what happens if we put them together? Could we express pin projections using view types? I think we might! “view types” as proposed by Niko are a way to teach the compiler about lifetimes in finer detail. I think we could apply the same logic to use view types to teach the compiler about pinning in finer detail too.

⚠ So before we dive in I want to remind folks again that this is just an idea. I’m not on the lang team. I am in not in any position to make decisions about this. I also don’t consider myself an expert in pin semantics, so I might have gotten things wrong. I am not saying that we should prioritize work on this. I’m sharing thoughts here in the hopes of, well, progressing the language. And sometimes putting half-baked ideas into writing can help others refine them into something practical later on. ⚠

Anyway, let’s take our earlier example and try and fit pin projections on it using view types. Taking it very literally we could imagine the signature of Future::poll to be possible to express like this:

// base
fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> { .. }

// using view types. `timer` is pinned.
fn poll(&mut{ timer, completed } self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> { .. }

Okay okay, this is a super literal application of “what if view types would just do the pin projection”. But bear with me, we’ll go over all details in a second. For now let’s just assume this works, and slot it into the earlier example:

pub fn sleep(dur: Duration) -> Sleep { .. }

pub struct Sleep {
    timer: Timer,
    completed: bool,
}

impl Future for Sleep {
    type Output = Instant;

    fn poll(&mut{ timer, completed } self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
         // ^ we're using the view type here
        assert!(!completed, "future polled after completing");
        match timer.poll(cx) {
            Poll::Ready(instant) => {
                *completed = true;
                Poll::Ready(instant.into())
            }
            Poll::Pending => Poll::Pending,
        }
    }
}

I think overall this gets close to the feel we want pin projections to have. Instead of our base type we narrow it down into its parts. But it’s probably not everything we want to do yet, and it certainly doesn’t cover all aspects of pin projections yet.

Ergonomics

In my opinion the Future::poll signature above is rather noisy; the view type might have helped simplify the function body and struct declration, but it’s definitely not helped the function signature. And somewhat annoyingly: we don’t have any more information than we started with. We can’t actually tell which fields are pinned!

fn poll(&mut{ timer, completed } self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {

Perhaps the compiler can be clever enough to infer that all !Unpin types in a view type should remain pinned when viewed. But even if it can, type signatures are as much for humans as they are for the compiler. And this tells us very little about what is pinned. Instead I think it’d be clearer if this type of lowering required contextual pin annotations:

fn poll(&mut{ pin timer, completed } self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {

Now we can tell that the timer is pinned, but completed is not. We finally have all the detail we need, but the signature is even more overwhelming than it was. I think it’s worth asking at this point whether there’s anything we could remove. If we look at the projected type, we can see there are three parts to it which tell us what “self” is supposed to look like:

   &mut { pin timer, completed } self: Pin<&mut Self>
// ^                             ^     ^
// |                             |     This only exists if `self` is this type.
// |                             This is about a `self` param.
// We have a mutable view type with these fields.

There are three parts here, which if we assume we can remove one, we can have three different permutations:

// view type + self
fn poll(&mut{ pin timer, completed } self, cx: &mut Context<'_>) -> Poll<Self::Output> {
//                                       ^ no right-hand `Pin<&mut Self>`

// view type + self
fn poll(&mut{ pin timer, completed } pin self, cx: &mut Context<'_>) -> Poll<Self::Output> {
//                                       ^ `Pin<&mut Self>` replaced by `pin self`

// view type + arbitrary Self-type
fn poll(&mut{ pin timer, completed }: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
//                                  ^ no left-hand `self` type

// self + arbitrary Self-type
fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
// ... no view type :(

None of these feel particularly great, but they do provide us with some insight. Pin<&mut Self> can be useful to know _when_ this implementation is available. self` is actually never used since we immediately destructure, but it is useful to know that this is a method and not a free-function. The view type is what makes pin projections possible, so we want to keep that somehow.

I’m not sure which attributes make most sense to highlight. The original design keeps all properties, but feels overwhelming. On reviewing this post, my co-worker Eric suggested we try and take “view types” proposal, performing destructuring like we do in patterns, dropping the left-hand variable declaration entirely:

// self + arbitrary Self-type + view type
fn poll(self: Pin<&mut Self { pin timer, completed }>, cx: &mut Context<'_>) -> Poll<Self::Output> {

// self + pin Self-type + view type
fn poll(self: pin Self { pin timer, completed }, cx: &mut Context<'_>) -> Poll<Self::Output> {

// view type + self
fn poll(self: &mut Self { pin timer, completed }, cx: &mut Context<'_>) -> Poll<Self::Output> {

// view type
fn poll(&mut Self { pin timer, completed }, cx: &mut Context<'_>) -> Poll<Self::Output> {

I think we’ll need to keep the self param to not create confusion with Self methods ⁴. But it’s hard to say which option out of these would be the best. The second one is probably the simplest. But the first one conveys more information. Luckily we don’t need to make any decisions now, so we can just settle for having looked at the space a little.

Pin projections and drop

Something we haven’t covered yet is that when pin projecting there are Drop requirements. Put bluntly, the signature of std::drop::Drop is wrong. If we’d known about “pinning” before Rust 1.0 we likely would’ve designed the Drop trait differently.

The reason why Drop and pinning conflict is because once something has been pinned it should pinned forever after ⁵. But Drop takes &mut self, even if a value has previously been pinned, violating this requirement. And so when we implement Drop for our async types we have to make sure to always convert &mut self back to self: Pin<&mut Self>:

impl Drop for Type {
    fn drop(&mut self) {
        // `new_unchecked` is okay because we know this value is never used
        // again after being dropped.
        inner_drop(unsafe { Pin::new_unchecked(self)});
        fn inner_drop(this: Pin<&mut Type>) {
            // Actual drop code goes here.
        }
    }
}

The pin-project crate covers this case for us. It disallows implementing Drop manually, and requires we implement pin_project::PinnedDrop instead, which has the right signature:

use std::pin::Pin;
use pin_project::{pin_project, pinned_drop};

#[pin_project(PinnedDrop)]
struct PrintOnDrop {
    #[pin]
    field: u8,
}

#[pinned_drop]
impl PinnedDrop for PrintOnDrop {
    fn drop(self: Pin<&mut Self>) {
        println!("Dropping: {}", self.field);
    }
}

fn main() {
    let _x = PrintOnDrop { field: 50 };
}

If we want to make pinning more accessible, then we cannot just skip over the interactions with Drop. I’m not sure whether this allows us to trigger soundness bugs using purely safe Rust - but I suspect the answer to that is likely irrelevant. It’s really easy to get lost in technicalities that we lose sight of the wider point: Pinning has too many gotchas to use without pin-project. And it’s hard to tease apart to which degree the issues are with pinning as a concept, or mistakes we’ve made in the past in the language.

Perhaps “mistakes” sounds too harsh. After all, Pin was introduced after Rust 1.0, so how could we have gotten this right? And while that’s true, I think that perspective is a bit of a dead-end. If the issues are with Pin, then logically we’ll seek to fix Pin. And realizing we’re unable to fix Pin, we’ll naturally seek to limit the usage of it. But rather than thinking about pin as “introducing issues into Rust”, what if we instead think of pin as “surfacing issues we have in Rust”. Pin is a fundamental type in Rust. Transformative even. It’s worth pondering what we could do with it if it was better integrated into the language.

Maybe we should bite the bullet and actually fix our Drop impl. The main reason why this hasn’t been done is because it’s backwards incompatible ⁶. But likely the most important reason after that is because it would grossly complicate the ergonomics of implementing Drop for people. Perhaps that calculation changes if the ergonomics or pinning were less bad. Say we took one of the ergo examples and applied it to our earlier drop example:

struct PrintOnDrop {
    field: u8,
}

impl Drop for PrintOnDrop {
    // We've dropped `self: Pin<&mut Self>` from our signature
    // because we immediately destructure it into its parts
    // inside the type signature anyway.
    fn drop(self: pin Self { pin field }) {
        println!("Dropping: {}", field);
    }
}

fn main() {
    let _x = PrintOnDrop { field: 50 };
}

I don’t think this looks noticably worse than today’s drop trait. And hold onto your tin-foil caps, I think this could possibly hint at another set of optimizations we might be able to perform. What if for structs where all types are Unpin we could create whole-sale projections. u8 here doesn’t actually need to be pinned, it’s just for show. So what if we actually were able to write that:

struct PrintOnDrop {
    field: u8,
}

impl Drop for PrintOnDrop {
    // This is a short-hand for a destructuring of `Pin<&mut Self>`
    // where all fields are `Unpin`, requiring no pinning and thus
    // no destructuring.
    fn drop(&mut self) {
        println!("Dropping: {}", self.field);
    }
}

fn main() {
    let _x = PrintOnDrop { field: 50 };
}

destructuring can be nice. That would allow us to write &mut self as sugar for self: Pin<&mut Self> where Self: Unpin. Meaning: for all sound implementations of Drop everything would continue working as they do today. The only difference is that the trait declaration of Drop would change. And we’d catch potential unsound invocations of Drop, requiring them to be sound at the function signature level. This includes pin-project’s use as well, but we’d likely be able to coordinate roll-out, deprecation, etc. in the crate.

This is not unlike how &mut self is syntactic sugar for &mut self: &mut Self already. We could make it so that we can lower Pin<&mut Self> to Self if it’s always safe to do so. Anyway. That’s probably enough speculation for this section. To close it off, so far we haven’t yet figured out how to introduce async Drop into the language. One option mentioned so far been to extend Drop with poll methods. However way we go about this, we’ll have to be looking more closely at Drop and Pin soon anyway.

#[repr(packed)]

There are other considerations such as not being able to pin a #[repr(packed)] type. And honestly: I’m not super sure how we can fix all of this. Maybe we should raise #[repr(packed)] into the type-system and make Pin<T> where T: !marker::Packed. Or perhaps pin projections should magick their way into knowing not to do this. I’m not sure. But this is something to figure out if we want to remove the unsafe {} marker for pin projections.

Unpin

So far we haven’t talked about Unpin at all. I mean, unless you’ve read the footnotes. But I assume that’s nobody, so here’s a recap: Unpin means that a type simply doesn’t care about Pin at all. Moving it around in memory won’t cause it to spontaniously combust, so we can just move it in and out of Pin wrappers whenever we like. This is true for most types ever created. Only types which are self-referential (hold pointers to fields contained within itself) are marked as !Unpin. Which is true for many futures created using async {}, because that’s what borrows across .await points desugar into.

Unpin is an auto-trait: meaning it’s implemented for a type if all the fields of the type implement it. But unlike auto-trait such as Send and Sync, it’s entirely safe to implement. And this causes some issues.

This means that even if our struct contains types which are !Unpin, we can still manually implement Unpin on it:

use std::marker::PhantomPinned;
use std::marker::Unpin;

// `Foo` holds an `!Unpin` type, which makes
// `Foo: !Unpin`.
struct Foo {
    _marker: PhantomPinned
}

// Oh nevermind, we can just mark it as `Unpin` anyway.
impl Unpin for Foo {}

I believe the reasoning is that since “pin projections” require unsafe, upholding the validity of Unpin should just be done there. But that’s an issue if we’re looking for ways to actually make pin projections safe to perform. Upholding invariants shouldn’t be done at the projection site; it needs to be done on declaration!

The pin-project crate works around this by disallowing safe implementations of Unpin and instead introducing their own UnsafeUnpin trait. This flips the safety requirement back to the Unpin implementation, requiring invariants are upheld by the implementer of the trait.

use pin_project::{pin_project, UnsafeUnpin};

#[pin_project(UnsafeUnpin)]
struct Struct<K, V> {
    #[pin]
    field_1: K,
    field_2: V,
}

// Implementing `Unpin` is unsafe here.
unsafe impl<K, V> UnsafeUnpin for Struct<K, V> where K: Unpin + Clone {}

// Which means calling `struct.project()` later on can be safe.

In order for pin projections to be safe, manual implementations of Unpin need to be unsafe. This is because someone needs to be responsible for correctly implementing the invariants; and it either needs to be where we perform the projection, or where we declare types as Unpin. “both” is annoying. “neither” is impossible ⁷.

It seems that if we want to make pin projections a first-class construct in Rust, we’ll have to mark Unpin as unsafe. Annoyingly this would be a change to the stdlib, and we don’t have a process like “editions” where we can flip a switch on changes like these. At least not yet.

Making this happen wouldn’t be easy. And we’d need to assess whether the impact justifies going down this path. But maybe it does. And maybe we should.

Pin keyword in other places?

At the start of this post we mentioned that stack-pinning is gaining first-class support in Rust via the core::pin::pin! macro. But later on in the post we also discuss the possibility of improving pin projection semantics by introducing a pin keyword in the view type. It’s worth asking: could a pin keyword be useful in other places too?

I think, maybe? Even without view types, we could imagine all function signatures which currently take self: Pin<&mut Self> could probably be simplified to take pin self instead:

// Before
pub trait Future {
    type Output;
    fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output>;
    //            ^ Using an "arbitrary self type"
}

// After
pub trait Future {
    type Output;
    fn poll(self: pin Self, cx: &mut Context<'_>) -> Poll<Self::Output>;
    //            ^ Replaced by a `pin` keyword.
}

Additionally, for stack pinning we could imagine we could replace the use of core::pin::pin! with pin-bindings:

// Before
fn block_on<T>(fut: impl Future<Output = T>) -> T {
    let mut fut = pin::pin!(fut);
    ...
}

// Using `let pin` bindings in the function body.
fn block_on<T>(fut: impl Future<Output = T>) -> T {
    let pin fut = fut;
    ...
}

// Using `pin` in the function signature.
fn block_on<T>(pin fut: impl Future<Output = T>) -> T {
    ...
}

I don’t think this looks particularly out of place. It might be nice even. But the real question to ask is: is this useful? We have so many things we could be working on that we have to choose what to prioritize. Right now “pin” is probably misunderstood and under-used because it doesn’t play well with the language. But if we fixed those issues, would we see people use pin for more things ⁸? It’s a bit of a chicken/egg problem; though we do need to have some an answer in order to justify spending time on this.

Conclusion

In this post I’ve shown what pin projections are, what view types are, how the two could be combined, ergonomics considerations, possible interactions with Drop, and interactions with Unpin. In general it seems there are 3 questions to answer?:

• Can we use view types to create pin projections? What should that look like?
• How can we fix the signature of Drop?
• How can we lift #[repr(Packed)] into the type system so we can disallow it to be pinned?
• If we want to make pin projections safe, can we mark Unpin as unsafe?

Have I missed anything? Are there any other parts to consider? I’m not sure. I also want to point out that we don’t need a single unifying answer to solve all of these questions. It may be that each item has a different solution; but once we solve all of them we can start to realize safe pin projections at the lang level. As I’ve mentioned a few times now, I’d be keen to hear from others.

Anyway, this is just some stuff I started thinking of after having read Niko’s view types blog post midway through writing a bunch of futures by hand. I’m not sure whether delving deeper into this idea is worth it at all right now. As I mentioned pinning has potential, but we’re trying hard to make users interact with it as little as possible. Kind of like how we have unsafe, but we don’t really want people interacting with it much if we can help it, even if the stakes are different.

From what I’ve heard the Linux kernel folks are making use of pinning for some of the things they’re doing already. I’m not sure what they’re doing it for, but it’d be fascinating to learn why. If people are thinking of improving the ergonomics of unsafe to make it less hard to use. Perhaps it’d be worth to do the same for pinning as well?

Thanks to Ryan Levick and Eric Holk for helping review and correct this post prior to publishing. Niko Matsakis for entertaining my delirious ramblings about this on Zulip a few days ago. And Boats, with who I’m pretty sure I’ve discussed something like this before, even though time is a lie and at this point I no longer remember how long ago that may have been.