Allow `codec::Encoder` and `codec::Decoder` to take any `IoBuf[Mut]`

[code-withAshish]

I’m working on a use case that involves interacting with RESP (Redis Serialization Protocol), and I’m using the redis-protocol crate—specifically the BytesFrame type.

I’m integrating this with compio’s codec API, whose Decoder trait is defined as:

fn decode(&mut self, buf: &[u8]) -> Result<Item, Error>;

The redis_protocol::decode_bytes API expects a &Bytes, so in my current implementation I’m forced to convert the incoming &[u8] into an owned Bytes on every call:

impl Decoder<BytesFrame> for RespCodec {
    type Error = RedisProtocolError;

    fn decode(&mut self, buf: &[u8]) -> Result<BytesFrame, Self::Error> {
        let buffer = Bytes::copy_from_slice(buf);

        match decode_bytes(&buffer) {
            Ok(Some((frame, _))) => Ok(frame),
            Ok(None) => {
                // not enough data to decode a full frame
                todo!()
            }
            Err(e) => Err(e),
        }
    }
}

This works functionally, but it requires copying the entire input buffer into a new Bytes allocation on every decode, which is potentially expensive—especially for a streaming protocol like RESP.

My questions are:

1. Is this copying unavoidable given the Decoder trait’s &[u8] input and owned Item output?
2. Is there a more idiomatic or zero-copy way to integrate redis-protocol’s decode_bytes with compio’s framing/codec infrastructure?
3. Does compio provide a higher-level framing or buffer abstraction (e.g. based on BytesMut) that would allow incremental decoding and buffer advancement, rather than re-decoding from a copied slice each time?

In other words, am I fundamentally constrained by the Decoder trait’s signature here, or is there a better approach I should be using for RESP-style streaming decoding?

[Vaider7]

+1

I also faced the same problem. I'm trying to port tokio-postgres to compio. tokio-postgres internally uses protocol that works with Bytes/BytesMut, and the only one way for now is to copy the whole buffer

Also I have a question about Decoder signature. It looks like

pub trait Decoder<Item> {
    type Error: From<Error>;

    // Required method
    fn decode(&mut self, buf: &[u8]) -> Result<Item, Self::Error>;
}

But there's common case when there's not enough data to decode a message, so it would be nice to be able to return Ok(None) in this case. For now there's only one way to declare Decoder<Option<Item>>, and it propagates at Framed, where instead of having Framed<Out = Item> you should declare Framed<Out = Option<Item>> which I believe is not proper way

[George-Miao]

@Vaider7 Validity of frame should be handled by Framer. Decoder assume it gets enough data to decode one frame, otherwise it will be a hard error.

See https://docs.rs/compio/latest/compio/io/framed/frame/trait.Framer.html#tymethod.extract. It gives you exactly what you want. Do notice that there is an ongoing effort to refactor En/Decoder to support arbitrary IoBufMut, but that requires some major refactoring in compio-buf first.

[Vaider7]

Ah, I see it now. I missed the Framer part and was wondering what was going on 😅

Thank you so much!

[briancarbone]

@Vaider7 Validity of frame should be handled by Framer. Decoder assume it gets enough data to decode one frame, otherwise it will be a hard error.

See https://docs.rs/compio/latest/compio/io/framed/frame/trait.Framer.html#tymethod.extract. It gives you exactly what you want. Do notice that there is an ongoing effort to refactor En/Decoder to support arbitrary IoBufMut, but that requires some major refactoring in compio-buf first.

Looking forward to this... a mutable reference would be very useful.

[George-Miao]

Looking forward to this... a mutable reference would be very useful.

Hmm. You mean you want mutable reference to the buffer in extract?

[briancarbone]

In the decoder, actually. simd_json takes a mutable reference to a buffer in order to rewrite escapes represented by two characters into the correct byte.

[George-Miao]

@briancarbone I see. I think this should be fine. I will add this to the PR as well. I have finished most of the PR, but unfortunately it was on the computer at home, and I won't be able to get back until three weeks later :(