DATAGRAM frames MUST be protected with either 0-RTT or 1-RTT keys. <del> Application protocols using datagrams are responsible for defining the semantics of the Datagram Data field, and how it is parsed.  If the application protocol supports the coexistence of multiple entities using datagrams inside a single QUIC connection, it can use a mechanism to allow demultiplexing between them.  For example, using datagrams with HTTP/3 involves prepending a flow identifier to all datagrams, see I-D.schinazi-quic-h3-datagram. </del> Note that while the max_datagram_frame_size transport parameter places a limit on the maximum size of DATAGRAM frames, that limit can be further reduced by the max_packet_size transport parameter, and by 5.1.  Although DATAGRAM frames are not retransmitted upon loss detection, they are ack-eliciting (I-D.ietf-quic-recovery).  Receivers SHOULD support delaying ACK frames (within the limits specified by a DATAGRAM frame that was thought to be lost could at a later point be received and acknowledged. <del> 5.2. </del>  DATAGRAM frames do not provide any explicit flow control signaling, and do not contribute to any per-flow or connection-wide data limit. frames are inherently unreliable, they MAY be dropped by the receiver if the receiver cannot process them. <del> 5.3. </del>  DATAGRAM frames employ the QUIC connection's congestion controller. As a result, a connection might be unable to send a DATAGRAM frame

I would like this section to say something about priorities, as emerged in the issue. Something to the effect of: "QUIC Implementations SHOULD present an API to applications to assign relative priorities to DATAGRAM frames with respect to each other and to QUIC streams. This can be done by the API conceptually assigning DATAGRAM frames to an existing QUIC stream and inheriting its priority, or assigning a priority in the DATAGRAM send API itself."
the first sentence of NAME suggestion sounds ok. I don't like the second one much (sorry). edit: clarification - the first sentence is very much like RFC 9000's "A QUIC implementation SHOULD provide ways in which an application can indicate the relative priority of streams"
I'm not married to the second sentence.
Piling on to agree here: I'm OK with adding the first sentence, though I think the second one is too specific so I'd prefer the second sentence wasn't added
Added the first sentence about priorities. Please check!
I finally sat down and read this draft (nice work, BTW) and was somewhat sad to learn that QUIC won't natively support stream multiplexing for datagrams. It is true that applications can implement this if they choose. But one nice thing about QUIC is that it takes care of this machinery on behalf of the application. This is a somewhat aesthetic concern, but it feels awkward to have one layer of stream multiplexing in QUIC and then another layer in the application. It would be ugly in our implementation, at least. Moving on to non-aesthetic issues: I don't have a full grasp of all the use cases for DATAGRAM, but if stream multiplexing is a common requirement, it would be good to move this into the transport. Applications that don't need this are free to do it all over one stream at a tiny loss in wire efficiency, or we could have a stream-less version if people are deeply concerned about one byte per frame.
The argument for taking it out effectively identified that there is little point in putting something in the transport that the transport can't help with
While HOL blocking advantages don't apply here, I'm not sure that it's true that the transport couldn't help. For example, a stream construct would make it much easier to handle priority of DATAGRAMs in the send buffer. There are other ways to solve this, but with streams we could use existing QUIC priority APIs.
In Montreal I also argued for a design that could be used for streaming, but was out numbered at the time. NAME I agree with your opinion that it would be best for the transport to provide this kind of interface, rather than requiring every app/protocol to implement it on top of this extension. I think the best path forward would be make a separate draft for unreliable streams.
I guess I find this prioritization example confusing because the transport protocol doesn't include an expression of the priority on the wire. How does telling the remote endpoint the DATAGRAM flow ID (as we referred to it in previous discussions) help with things that are a concern for transport RX.
As with reliable streams, all the priority negotiation is at the application layer and indeed, whether the multiplexing is in the transport or application doesn't matter. However, in the reliable case it is certainly useful for the transport to have the notion of streams so that it can send the highest-priority streams first. It is certainly possible to build a QUIC API that accepts datagram priority in the absence of transport streams, but it seems a lot more convenient to reuse the existing semantics. If I'm in the rough here, this is not going to break the protocol. But it feels like a recipe for writing a bunch more code.
I do understand what you're saying and it sounds valid but I'm attempting to channel some counter arguments back here. One main services the transport provides for streams is accounting of data stream data flow windows and stream IDs. DATAGRAM is attractive to applications that don't require the reliability that streams provide and the lack of a flow ID means it also doesn't come with the overhead of accounting mechanisms that aren't so useful for single-use atomic messages. I do think DATAGRAM draft would benefit from some clarification about the expectations of prioritizing DATAGRAM data, especially against STREAMS. But I don't think the flow ID is a requirement for that.
A QUIC implementation could easily provide exactly the same API for managing datagram priority that it does for streams; neither need to be specified in detail by the draft, let alone present on the wire. I don't think adding a field to the wire representation could possibly save any code here; it's more complexity for every implementer, not less.
I agree with NAME point about not requiring every higher level protocol that needs it to separately implement flow IDs. The spec provides flow IDs. What if I am designing some new protocol to run on top of QUIC, other than HTTP/3, and I also have the need for multiple datagram flows? If the facility is in QUIC datagrams, then the QUIC library my protocol's implementors will be using is going to provide it to them for free. If the facility is in HTTP/3, few or no HTTP/3 implementations are going to allow just using bits of HTTP/3 in isolation from the rest of it. So it would force the implementors of my higher level protocol to reimplement this facility (or else I might just decide to base my protocol on top of HTTP/3 instead of QUIC – but that then gets away from the idea that QUIC is usable for more than just HTTP.) Couldn't the stuff about flow ID management in the HTTP/3 datagram spec just get moved to this spec instead?
One of the earlier drafts did have them combined exactly like that. The consensus of the group was to keep them separate. The main issue is that there is nothing at the transport layer that the flow ID influences, only the application. There’s nothing saying APIs can’t make this convenient to use, etc, but it doesn’t necessarily belong in the fundamental wire image of the datagram.
To satisfy the flow ID allocation requirement of H3-datagram all that is needed is something like How flow IDs are used is up to the application. And at that point, you're going to really benefit from well-formed application protocol and or/application-mapping. For example, there is no single answer to whether flow IDs are required nor how to bind flow IDs. So, IMO there really not a generic service that a QUIC library can provide to applications.
I personally disagree. The API for a library does more than just implement the protocol. It provides efficient ways to build application logic on top of protocol behavior too. Having the flow-ID at the protocol/library layer allows the API to provide a per-flow abstraction object, similar to a Stream object. This can allow for the app to associate different logic/context to different datagram flows (e.g. register a different receive callback for audio & video flows). IMO, this will be a very common usage on top of datagrams and will end up being reimplemented for every app/protocol built on top of datagrams. One of the primary points of the library/API to to implement logic once so that it may be reused for many different apps. Additionally, I don't believe this kind of abstraction layer should be in the library/API unless it's also in the protocol. The goal of the API is to be cross-compatible with other implementations, generically. If my implementation has some custom flow-ID logic, but no other API does, what's the likelihood of interop issues?
I could implement the same behaviour by registering a predicate to the QUIC library e.g. if N bytes of the QUIC DATAGRAM payload matches this pattern, visit this callback function. I don't have that today on the receive side but I do have that for purging pending items in my send queue, see URL I agree. If I added my predicate-style generic method to my QUIC library, I could build on top of it to support the H3-Datagram flow IDs as a first-class API property. It's the job of the application mapping implementation to write the correct predicate, and any additional logic that may be needed such as only accepting flows that have been permitted by some other form of negotiation (such as a CONNECT-UDP method). Just adding a flow ID field leaves open a lot of questions, and I don't think they can be answered commonly across all the potential applications.
From an API perspective, it is certainly useful to have the transport layer help demultiplex and put different flows in different send/receive queues in the application. To that end, I totally agree with you, NAME However, the diversity of use cases for tracking datagram flows, and the possible ways that this could be usefully exposed to the application, means that building in one interpretation of demultiplexing based on a flow ID without a use for the transport itself may be unnecessarily limiting. This is a common issue for how to do demultiplexing of application flows on top of a transport. In TAPS, we use protocol framers for the same thing, as a way for the application to tell the transport API "I want to separate messages based on this, and pull out these fields as special markers (like a flow ID)". (URL) I imagine that a QUIC API should follow this model if it wants to make it easy for datagram applications to demultiplex without having to build their own demux layers.
I confess to having a primarily aesthetic objection to having two different layers of stream multiplexing in the stack, but I've been trying to figure out a reason streams might matter for the transport. I certainly can't come up with a killer app for transport streams. The best I can find are purely send-side, so in theory they can be done with sufficiently rich APIs rather than something on the wire. I guess the question is whether or not this is too much load to put on the QUIC API: Priority. Without a transport-layer stream, the datagram API will have to specify priority on a per-datagram basis. Ordering. The draft should probably say a little more about ordering expectations, but obviously there are no firm guarantees here. I can certainly imagine QUIC reordering datagrams for its own reasons (e.g. sending a smaller packet first based on space available), and it might be good for the application to specify that a certain subset of datagrams SHOULD be delivered in order. To do this without transport-layer datagram streams, the API would have to provide some sort of datagram identifier that could be ordered with respect to other datagram identifiers. It sounds clunky, if this use case could conceivably matter to anyone. Of course, the higher-assurance way to do ordered delivery is to put a sequence number in the application datagram frame and discard OOO datagrams at the receiver. but it would be nice if QUIC didn't make things worse for these applications by arbitrary reordering. Martin
Let's not conflate "things it would be useful for a QUIC implementation to expose in its API" with "things that need to be mandated as part of the wire format." There can be discussion of options (not requirements) for the former in the text. Adding extra constraints to the wire format complicates all implementations, and makes applications with different requirements awkward.
Purely API-level stream identifiers would work fine for this, as I . This would be pretty convenient and I think is a sensible thing for the draft to suggest (but not require). The datagram extension as currently written is useful in part because it is so content-agnostic; there's plenty of room for more specialized extensions in the future if certain applications would benefit from a more complex wire format.
Yes, I agree that would solve the problem.
I like this status quo. I've been mulling whether in fact all streaming an multiplexing could be done top of a QUIC connection used just for datagrams without loss of QOS. It's certainly not the path taken so far, but I like this datagram extension being pretty minimal to keep the future wider open.
NAME I'm not sure that's the case here, the original design had an optional flow ID specified by the frame type, IIRC.
I tend to agree that this is something that almost every user of datagrams is going to need to specify, and being able to interoperate multiple different protocols using datagrams on a single QUIC connection has come up quite a bit. We handled this before by punting it to H3 and saying that you need H3 to be involved to get that mux-ing, but it's been noted in this thread that you might not always have H3 involved. Previously, this led us down into a conversation before we had figured out what alpn meant for QUIC, but at this point I think we've moved past that enough to not want each possible permutation of protocols in use/defined by a new alpn string to have a slightly different mechanism for handling flow identifiers, some of which might be compatible. If we think that introduces too much complexity/other problems to tackle, then right now we've just moved that complexity into H3, which means we (a) still need to solve it, but (b) now everyone else cannot take advantage of that solution.
To NAME TAPS comment, I think the question about having an additional [de]mux layer in between is the same as punting it to H3 -- we still need to figure out how to make that work, the question is just where do we group that effort and who can take advantage of it once that work has been done. If everyone always uses datagrams + demux-protocol, then that seems like a not-unreasonable outcome too.
A design decision having been present in a prior draft does not cause it to not be a source of avoidable complication. As a user of datagrams, I do not need this.
Hmm, I think the key to the statement above was the optionality of the flow ID, meaning that for use cases that did not require a flow ID, there was no additional space space taken and minimal complexity, since the behavior was exactly as it is today without flow IDs.
IIRC, support for both modes was mandatory for implementations, i.e. if you negotiated support for the extension during the handshake then you must accept flow IDs. That means it's additional implementation complexity.
NAME if both modes were brought back, then support for the flow ID extension could be made optional for implementations, so those who didn't want to implement it didn't have to. However, I think in practice, most implementations would implement it if it existed. Optional features in protocols, if sufficiently widely implemented, end up becoming de facto mandatory. And if the most common higher level protocols were to require flow IDs (e.g. H3 datagrams), then almost all implementations would implement it even if it were technically optional.
In the latest H3 QUIC DATAGRAM draft at URL (bolding is mine): provide a flow identifier allocation service. That service will allow applications co-located with HTTP/3 to request a unique flow identifier that they can subsequently use for their own purposes. The HTTP/3 implementation will then parse the flow identifier of incoming DATAGRAM frames and use it to deliver the frame to the appropriate application. I think this is an important point as to why the Flow ID should be moved to this spec instead as NAME suggests. Every single application that utilizes DATAGRAM must decide if it wants to support being "co-located" with others, and if it does it will more-than-likely be with HTTP/3 and use the Flow IDs specified there. That may be sufficient to keep from there being incompatible approaches, but the more we can do to minimize diverging implementations the better IMO. By adding it as an OPTIONAL section to this spec instead gives it increased visibility for implementers. It doesn't restrict those that definitely do not need to make use of it, but it does provide a strong reference point for those that do and is relevant to their non-HTTP/3 applications. There seems to be some consensus in this thread that such a compromise is acceptable?
I was quite surprised that this spec doesn't specify a mandatory flow ID field. It is inconsistent with the main QUIC spec where having a lot of different flows through one QUIC connection is one of the main features. I think that the flow ID must be a mandatory field in this spec, to make all the QUIC based protocols extensible by default, just to let people to mux a few different QUIC based protocols into one QUIC connection easier.
I'm still happy with having no flow ID in the frame. I've not heard a strong concrete case where the wire format needs this information. I'd like to circle back to the points on priority. QUIC is purposefully quiet on the point of signalling stream priority, this is something that every application mapping will have to reinvent, and we are happy with that design decision. Many potential clients for a feature doesn't mean we need to provide it.
I tried to put together a list of all the scenarios that I think would make use of unreliable data delivery. I came up with the following: VPN/tunneling Audio/video streaming Gaming App-layer keep alives Status updates (e.g. latest temp reading) For every one of these scenarios I can imagine some level of multiplexing being used. For that reason, I personally think QUIC should provide the (optional) multiplexing functionality natively and not require the application layers to continually reinvent the wheel. To NAME comparison to priorities, I disagree with the analogy. I don't think that all QUIC-based application protocols actually need priorities, and even few need to explicitly signal them to the peer; so keeping priority signals out of the QUIC layer makes sense. In contrast, multiplexing (whether over streams or datagrams) is a core functionality for QUIC, and I strongly believe that should extend to unreliable data delivery. So, since most application protocols would use it (IMO), we should include (as an option) a flow ID in datagrams to support multiplexing.
I agree there are application use cases that can benefit from different logical flows. But the question that is at hand is, what services does the transport provide to those flows. There has to be a reason that it exists there. Do you want flow concurrency limits, do you want per-flow buffering etc Of the adopted work I am aware of in the IETF, to make multiplexing actually work, you need an application-layer signalling mechanism. This has been achieved using HTTP/3, to suit how HTTP uses QUIC. Are the needs of other application types similar?I think a lot of the work people anticipate needs doing can't be commonly solved; we risk tying up progress on DATAGRAM by designing an abstraction for loose requirements.
I agree there are different ways to possibly treat/view multiplexed unreliable delivery. On one extreme it is a fire and forget flow identifier and (unfragmented) byte blob; At the other extremely you have "streams" with unreliable delivery, and everything that goes with it (flow ID FC; per-flow FC; fragmentation/reassembly?). I do not think the datagram extension should encompass unreliable "streams". That is a far more complicated topic and should be left to it's own extension. I do believe this extension should encompass the first scenario. Just allow for an optional flow ID. As far as on the wire efficiency, it's cheapest to do it at the QUIC layer. We just use a separate code point for datagrams with flow ID and those without (as we had originally, right?). For the application layer to implement such optional behavior would require more space. Perhaps another way to look at i:. If enough people feel strongly about wanting this, but we can't get agreement here, would you really prefer a completely different extension (and associated draft) just to duplicate what's here but adds the flow ID? IMO, that seems like overkill. Worst case, I'd prefer to have two separate TPs in this draft to indicate support/negotiation for the two different types of datagram frames.
Nick, this is a good spectrum of options / requirements. Thanks for pulling on this thread. For the record, this is my individual opinion based on my own individual implementation experience. I feel DATAGRAM is close to done and revisiting a core design decision at a late stage is unfortunate. But if its the right thing to do then it pays to really probe this and make a decision we can agree to never revisit. I don't know if everyone would agree with your pick on the spectrum. Establishing consensus on what a transport Flow ID should provide is important. Just adding a varint to the payload of the current DATAGRAM frame is easy. An application mapping can copy the blueprint of HTTP/3 DATAGRAM. I will highlight that providing more options adds some complexity to application mappings. You'll be asking them to have to decide if they want to use a flow ID or not. For example, iff a flow-ID containing transport frame were added, then H3 DATAGRAM needs to be updated to prohibit 2 of the frame types; or MASQUE need to be updated to do the same. On balance, I think the pros/cons of the current design are evenly balanced with the alternative (plus its knock on effects).
One option is to define a new frame (?) and associated TP, and just keep it logically separate from the frame and TP. HTTP/3 could continue to only use the frame/functionality, and H3/QUIC implementations could get away with not ever supporting the multiplexed variant. If we went with that approach, would you prefer a completely separate draft for that? IMO, that would entail a lot of duplication of the non-framing specific text.
I think that because there is so much overlap between the two types that, whether they are in one document or two, aapplication mappings and implementers would face the same challenge either way. So, personally, I remain to be convinced that such a design is needed. But if there is consensus to do so, a single document for both - that covers the tradeoffs - is a better way to spec it.
Makes sense. A lot of this just comes down to folks' personal opinions. As the owner of a generic QUIC stack (and not any HTTP specific code) I prefer to support it at the QUIC layer, where we already have all the multiplexing abstraction code. I also really like the "implement once" property as well. IMO it generally leads to a cleaner API design and fewer bugs overall. One final data point: I have had discussion with teams at Microsoft that have non-HTTP based protocols that are looking to move to QUIC, and I do believe they would benefit from native multiplexed unreliable delivery support in QUIC. Obviously they (and any protocol) can do it themselves, but it doesn't mean they should.
Understood. As a maintainer of a generic QUIC stack I want to know if the only service people want from me is basically variable-integer flow ID handling or if they want more. I can expose varint handling helper functions but more complicated things are harder.
As someone working on multiple applications that use DATAGRAM but not H3, I want to ensure that they all can be composed together on one connection compatibly and easily. If I implement the Flow portion of the H3 DATAGRAM independently in each application then I will also have to implement my own shim layer to manage the Flow IDs compatibly when the underlying QUIC library doesn't provide it, and hope that H3-supporting libraries also expose compatible APIs for multiplexing with HTTP. None of that is very difficult really, but it seems that a likely byproduct is pressure on DATAGRAM supporting QUIC libraries to provide the Flow ID counters in a central place independent from any H3 support. I'm not suggesting that it must be added here, but a clear (if small) specification of Flow IDs separate from the H3 DATAGRAM spec could really aid in ensuring compatible implementations and application APIs for all QUIC DATAGRAM libraries that intend to support multiple applications over one connection.
Multiplexing different application mappings inside a single QUIC connection has a bunch of complexity not confidend to DATAGRAM. For instance, QUIC provides stream IDs but the way that H3 chooses to use them means that it can't be multiplexed with other applications without some form of shim. MASQUE is one such shim, which links flow ID allocation to HTTP signalling. There's other ways to do it this of course. It sounds like NAME is asking for something similar to QUICs Stream ID management. This covers (among other things) stream ID allocation and enforcement of stream ID usage, which is a transport interoperability feature. To do the same for DATAGRAM flow IDs would require additional signalling like "maxinitialflowd" TP, and MAX_FLOWS frame. This might be something people want but I don't really know. However, even if a connection-wide transport flow ID manager existed then, just like with stream IDs, there would still be competition between application protocols that want to multiplex themselves together in the shared flow ID space. The hard problem would remain unsolved. The difficulties with sharing stream ID and flow ID space is already manifest in the WebTransport group. Demonstrating that even pooling together multiple instances of the same application mapping has challenges.
With an individual hat on - I agree with NAME on pretty much all of his points. There's definitely advantages to having a flow id at the transport, and it seems pretty straightforward at first glance. That being said there's a lot of complexity with actually managing those flow ids in a generic way. Once we put them in the transport we also have to be in the business of managing those flow ids. I agree that many applications will want to have some sort of flow id -- but I can also foresee those applications being rather opinionated about how those flow ids should behave, and I'd rather not have the transport try to meet those requirements in the generic extension.
I am better appreciating the complexity of adding even an optional basic transport-managed Flow ID, I'd rather see DATAGRAM be a simple standard as-is. If there's consensus, the simplest resolution to this issue might be to just take the paragraph mentioning HTTP/3's flow identifier and make it a ## section to draw more attention to it for application and transport implementers to consider. Possibly even expanding the text to highlight the choice applications will have to make if they want to share a connection with other DATAGRAM apps.
Discussed at IETF 110: Conversation in the room proposed that we point to H3 datagram as an example of what upper layers can do and indicate that other ALPNs should add something similar. Still didn't belong in the QUIC datagram itself, and if we find that many ALPNs repeat a pattern, they can share another extension / definition document.
I think it would be great if the text of the spec could include some brief rationale explaining the reasons for the decision reached. I think it would be helpful for readers to give some brief explanation of why the spec is the way it is on this topic, instead of having to dig through PRs / mailing list archives / etc in order to discover that.
Personally, I think an effective way of framing the rationale that NAME is seeking is a direct comparison to streams (as set out in transport section 2 and 3). Datagram is another application-data carrying primitive, highlighting the core differences could help avoid people needing to ask these questions when they puzzle about how to write application mappings. The suggestion that Eric made would be a piece of such framing.
LGTM

DATAGRAM frames MUST be protected with either 0-RTT or 1-RTT keys.  Note that while the max_datagram_frame_size transport parameter places a limit on the maximum size of DATAGRAM frames, that limit can be further reduced by the max_packet_size transport parameter, and by 5.1. <ins> DATAGRAM frames belong to a QUIC connection as a whole, and are not strongly associated with any stream ID at the QUIC layer.  However, it is expected that applications will want to differentiate between specific DATAGRAM frames by using identifiers, such as for logical flows of datagrams or to distinguish between different kinds of datagrams. Identifiers used to multiplex different kinds of datagrams, or flows of datagrams, are the responsibility of the application protocol running over QUIC to define.  The application defines the semantics of the Datagram Data field and how it is parsed. If the application needs to support the coexistence of multiple flows of datagrams, one recommended pattern is to use a variable-length integer at the beginning of the Datagram Data field. QUIC implementations SHOULD present an API to applications to assign relative priorities to DATAGRAM frames with respect to each other and to QUIC streams. 5.2. </ins> Although DATAGRAM frames are not retransmitted upon loss detection, they are ack-eliciting (I-D.ietf-quic-recovery).  Receivers SHOULD support delaying ACK frames (within the limits specified by a DATAGRAM frame that was thought to be lost could at a later point be received and acknowledged.  <ins> 5.3. </ins> DATAGRAM frames do not provide any explicit flow control signaling, and do not contribute to any per-flow or connection-wide data limit. frames are inherently unreliable, they MAY be dropped by the receiver if the receiver cannot process them.  <ins> 5.4. </ins> DATAGRAM frames employ the QUIC connection's congestion controller. As a result, a connection might be unable to send a DATAGRAM frame


If a sender detects that a packet containing a specific DATAGRAM frame might have been lost, the implementation MAY notify the <del> application that it believes the datagram was lost.  Similarly, if a packet containing a DATAGRAM frame is acknowledged, the implementation MAY notify the application that the datagram was successfully transmitted and received.  Note that, due to reordering, a DATAGRAM frame that was thought to be lost could at a later point be received and acknowledged. </del>  5.3.

An issue came up in discussions around WebTransport: URL The problem is that datagram frames may be acknowledged for the sake of congestion control, but dropped for the sake of flow control. A receiver with a full datagram buffer would have two choices: Drop any packets with datagram frames and do not acknowledge them. Drop the datagram frames but acknowledge the packets. The first approach seems terrible because it drop any other frames bundled with the datagram (ex. STREAM) and cause retransmissions. It would also be treated as network congestion by the sender, causing limited bandwidth for otherwise unrelated STREAM frames. The second approach has ramifications for any implementations that MAY expose datagram acknowledgements. Specifically, the sender would believe that the datagram has been received (although not necessary processed), when in reality it could have been dropped and will never be processed.
I suggest that we remove the recommendation that datagram acknowledgements can be exposed to the application, and possibly even prohibit it.
Alternatively, this problem could be avoided if there was explicit flow control for datagrams. This would avoid a full datagram buffer and may have other benefits.
Adding flow control to DATAGRAM seems like a problem for use cases that don't want that, like tunneling of UDP-based transports that provide their own flow control
I may be wrong, but this seems very implementation specific, but IMO flow control is unnecessary so long as datagrams are indicated to the application layer inline (no buffering). For instance, MsQuic only acknowledges a packet with a datagram frame if it's able to successfully process it (datagram frame processing ). Additionally, "processing" a datagram frame includes indicating it up (inline indication ) to the application layer. There is absolutely no buffering at the datagram frame layer in MsQuic; so flow control is not a really issue. It's possible the application layer then does some buffering and would require flow control, but that's completely up to the application layer. If there is a scenario where the application layer may drop data (due to buffering constraints or something else) it should be up to the application layer to decide if the peer needs to know that information; and if so, inform them appropriately at their layer.
Surely that could be addressed by specifying arbitrarily large or infinite flow control limits? Would still complicate the spec, which seems unfortunate. That's a really cool solution. It's not a small constraint on the receiver's implementation, but it's certainly a case where we forbidding application-visible ACKs would be a harmful restriction.
Inline datagram processing is a cool idea and removes the need for flow control. Unfortunately, it's the sender that exposes acknowledgements to the application, but it's the receiver that would need to process and acknowledge datagrams inline. The sender would not be able to assume the receiver uses a particular implementation.
The capability to do inline indications on the receive side could be negotiated with the transport parameter. "Hey, I receive datagrams, and I don't buffer them." Then the sender side can decide how they want to handle ACKs based on that state. Though, it's not great that they still have to handle the scenario of "I do buffer, and I might throw them away" so long as any peer they communicate with might buffer. Personally, I'd just like to say that we require guaranteed app delivery for acknowledged QUIC packets with datagram frames, and then let QUIC implementations sort it out, but I realize that's not easily done for most implementations without significant refactoring. But if we don't do this, I do see it as a major bummer that buffering on the receiver side could effectively make QUIC layer acknowledgements worthless to the app (which IMO was the best advantage to this solution).
My only other thought would be to have a new frame (e.g. DATAGRAM_DROPPED) that a receiver sends if it drops DATAGRAM frame; but to accurately identify which that were dropped, each datagram would need a unique identifier and that further complicates all this. So I definitely wouldn't prefer this solution.
Or, in more explicit terms, "My ACKs guarantee delivery of datagram frames to the application." Is this any different from handling the scenario of a peer that doesn't support datagram frames at all?
I agree with NAME earlier comment that this sounds very implementation-specific. Our implementation also would not drop any datagram frames that have been received and ack'ed, since we process the data immediately and send it inline. I'd almost prefer to just have a SHOULD NOT drop datagram frames if they've been received and ack'ed. This will certainly influence API design, and means that you'll have a rougher time if you just try to imitate a UDP socket buffer.
Yes, there is a difference. If the peer doesn't support datagram frames at all, the protocol might not work at all, and the app layer will just reject the connection outright. If the app supports datagram frames, but it's implementation may or may not buffer. In that case, depending on the app protocol, it may or may not then need additional app layer acknowledgements or accounting. For instance, perhaps some game protocol that has a fire-and-forget "current player state" datagram payload could care less about acknowledgements, so it doesn't care if buffering drops a frame. All it needs is datagram support.
I would also prefer this approach.
For an implementation that doesn't expose the contents of QUIC packets to an application, exposing the ACKs has little value. The flow control issue is then a little moot. I would push back on any strict requirements as mentioned, it presupposes that implementations will provide a single style of processing or API. SHOULD NOT seems acceptable because I would drop them based on some criteria and the text probably encourages me to document this aspect.
In the context of WebTransport, datagrams are processed at the Javascript level. Browsers are understandably not going to execute and block on inline Javascript callbacks for every received datagram.
Why can't the app layer reject the connection outright if the peer's transport parameters indicate that it does not support reliable acknowledgement for datagrams, just as it would if they indicate no support for datagrams at all? An app which doesn't care about ACKs obviously wouldn't do so, but those that do, can. Or they could apply app-layer mitigations, but that's more options, not fewer.
+1 to URL . There is a process hop between the QUIC code and the JavaScript code, and the QUIC code needs to send ACKs quickly (especially if that packet contains a STREAM frame next to a DATAGRAM frame) so it's simply not possible to wait on JavaScript before sending the ACK. You could buffer datagrams indefinitely after ACking them to ensure that the application gets them, but that goes against the entire philosophy of DATAGRAMs by increasing latency unboundedly. "SHOULD NOT drop DATAGRAM frames" sounds completely unrealistic to me.
NAME You don't need to expose how a datagram frame buffer from the app is put in a QUIC packet in order to give them information back about the receipt (or not) of an acknowledgement of their data. MsQuic allows the app to give a context pointer along with the buffer they send. We then indicate acknowledgement or suspected loss information back to the app for that context (event ). NAME and NAME I totally agree that it's unreasonable to expect every layer above QUIC in the WebTransport scenario to do inline delivery, but that isn't the only scenario, and having that one scenario's requirement for buffering shouldn't make us throw our hands up and say no scenarios will be able to use a zero buffering approach. Protocols directly on top of QUIC should be able to do whatever they want. The "SHOULD NOT drop DATAGRAM frames" purely at the QUIC layer is not unrealistic IMO. What the layers on top of QUIC do are totally up to them to define and implement. WebTransport datagrams processed at the Javascript layer will need application layer acknowledgements if the protocol requires info on exactly which datagrams were delivered.
Thanks Nick, that makes more sense. I do wonder though how useful it acks are if you're talking to my endpoint who might, in some conditions, drop the occasional DATAGRAM on the floor.
NAME I was thinking of this in terms of WebTransport since that was really top of mind. I agree that different applications have different needs, and some might be able to do this inline ACKing more than others. That said, I'm really not sure what the point of this SHOULD NOT is, it's like saying "an IP router SHOULD NOT drop IP packets" - most router implementors know that they shouldn't drop packets unless they have a good reason such as having nowhere to put the packet.
NAME I do agree somewhat with that sentiment, but it's a bit different because the transport is saying "Yes, I got the packet and was able to successfully process it.". If it's possible/allowable for the transport to then just drop it on the floor, is there really any reason to acknowledge datagram frames at all, beyond just another RTT estimate? I just feel it would be a big shame that we have all this ACK machinery built into QUIC, but it gets wasted on datagrams. Not only that, it has to be rebuilt completely by the app to duplicate all the work QUIC already does.
NAME the way I see it, there are only two ways to perform queueing when you have one input and one output: option 1: you buffer things indefinitely forever option 2: you don't I really don't see any other way. I think you're suggesting that transports should buffer DATAGRAMs infinitely if the application is busy, and that doesn't sound like a good idea to me.
There are two questions here: who owns the buffers -- QUIC, or the application; and what are the application's reliability requirements. I would argue that for maximizing generality of a DATAGRAM receiver implementation -- so that multiple application behaviors are possible -- you want the application to own these buffers. Whether these buffers are implemented in QUIC or at the application or somewhere in between doesn't matter; it is meant for the application. For an application that wants some notion of reliability, there are two choices in how a receiver manages its buffers: it signals delivery out of the buffers and/or loss (ACKs/NACKs/gaps); it signals buffer size explicitly to avoid any loss (flow control) An app that sits on top of QUIC DATAGRAMs thus has two choices: use explicit Application ACKs to signal receipt (and gaps or NACKs to signal loss), since the sender might have caused the application buffer to overflow; use explicit flow control to disallow application buffer overflowing. Doing this has the nice property that QUIC ACKs can then be treated as application ACKs, since application flow control will ensure that this data does not get dropped at the receiver between QUIC and the app. In either case, it's a decision that is both application- and implementation-dependent. An app that cannot know the buffer size will have to rely on using explicit application ACKs. An app that is ok with data loss can allow loss to happen, and may or may not signal this back. An app that wants to avoid buffer overflow at the application buffers can reuse QUIC ACKs as application ACKs but will need to ensure that application flow control is in place. Implementations ought to expose DATAGRAM QUIC ACKs to the application, so that applications have a choice of using that signal. However, the application is ultimately responsible for using them correctly.
NAME the way I read your summary of the issue (which is great, btw. Thanks!) it seems that not only do ACKs need to be exposed to the application, but a receiver can't drop acknowledged datagrams without informing the application, right? Otherwise, how would the app be able to reliably implement it's own flow control along with QUIC ACKs? A sender who receives an ACK of a datagram would assume that datagram is taking up buffer on the peer application; but if QUIC dropped it before it received that buffer, then both application sides would get out of sync in their view of the FC window. The sender could eventually stall out because of this, right?
"Instant processing" model is really nice, and makes sense if you're in the same process as the QUIC stack, but across a privilege separation boundary (web app/browser, userspace/kernel, etc), this is not feasible, so I don't think this should be allowed at least by default.  this is technically possible, and it is in fact how WebSocket API works; if your web app is too slow at processing incoming WebSocket messages, your server will just crash a browser tab remotely by overflowing the task queue, so most people agree this was a big mistake.
After reading through this thread I may have missed it, but has anyone considered simply having the spec say something along the lines of: "If an ACK frame has been sent and then due to implementation specific buffering or resource constraints it is unable to process the corresponding DATAGRAM frame, the application MUST be notified of the locally dropped frames so that it has the option to support additional signaling if necessary." This will ensure that only for any implementation where this case is possible, the application developer/operator will be made aware of such a possibility and can decide how best to handle it either through configuration or custom application logic.
What could a reasonable application do with that notification, aside from assert that it doesn't happen?
Many things are possible with the notification, the ability to recognize when the dropped packets are not due to network congestion could be used to do custom throughput signaling back to the sender that incorporates other local signals such as queue depth or processing rate. The underlying implementation might also expose dynamic controls on the cache or queue size that the application can adjust. It could also back off other memory or processor heavy operations in order to prioritize capacity to not drop future packets. By ensuring that the ACK signaling only indicates congestion it isolates the responsibility of additional flow control signaling to just implementations that support it, only having to support notifying applications in such an event where frames are dropped locally.
Dropping datagrams ought to be similar to resetting a stream. It is pretty much the same semantic.
NAME can you elaborate? Resetting a stream is a non-recoverable situation for a stream, whereas dropping datagrams is recoverable.
Resetting streams is defined as an application triggered message: application says it cannot proceed with the stream and provides an application defined code and reason. I think the application saying "I am dropping some of the datagrams that I received" has similar semantics.
I see. It would make me sad to turn a recoverable error into an unrecoverable one though, because some applications may not want that.
Yes, that's where the analogy with reset breaks. The datagram dropped message should be informative, not change the state of the connection or the datagram streams.
The difficulty I have with this concept is that I don't know what I'd expose to the application. My datagram implementation basically keeps a fixed-size queue of frames. So if I dropped some all I can tell an application is that something got dropped. Adding more information about what was dropped increases complexity. The library already doesn't expose information about packets to the application, and so far things have worked out OK.
I tend to agree that there isn't much a QUIC stack that queues datagrams can do here. The best it could do it indicate how many it dropped, but that's likely not enough. If that data needs to be retransmitted then (at the app layer) they would likely need more detailed info (such as an app identifier for each dropped packet). I think the best the draft can do here is mention that if the QUIC stack drops datagrams frames and also acknowledges them, then those acks cannot be leveraged by the application layer. It might be nice to have an additional TP that allows behavior this to be negotiated, but it's not strictly necessary.
" If that data needs to be retransmitted then (at the app layer) " ... then the app needs to run a transport on top of datagrams. Or use regular streams.
+1, if you need reliability then datagrams are not the droids you're looking for
The more I think on this, the more it just seems similar to the existing notion that we have with QUIC streams; transport delivery != application end-to-end delivery & processing. It's like email read-reciepts: just because I opened the email doesn't mean I read it, digested it and have a response. The doc might benefit from doing a better job of explaining this notion and the interop concerns that might arise. For instance, stating that it is possible for receiving implementations to acknowledge DATAGRAM-bearing packets while providing no guarantee that the DATAGRAMS are presented to the application. Valid reasons are flow control or any other arbitrary reason the receiver decides. Application's that desire end-to-end loss-detection need to be aware of this possibility and build measures that accomodate it. As Jana points out, there are some signals an implementation MAY chose to provide to make that easier. If they don't, applications can vote with their feet.
Adding text to clarify that receipt of an ACK for a packet containing a DATAGRAM frame does not mean that the application got the frame sounds like the right solution here.
Text is good, but I am still a bit sad that we don't have some way to leverage the QUIC ACKs in a general sense (not implementation specific).

If a sender detects that a packet containing a specific DATAGRAM frame might have been lost, the implementation MAY notify the  <ins> application that it believes the datagram was lost. Similarly, if a packet containing a DATAGRAM frame is acknowledged, the implementation MAY notify the sender application that the datagram was successfully transmitted and received.  Due to reordering, this can include a DATAGRAM frame that was thought to be lost, but which at a later point was received and acknowledged.  It is important to note that acknowledgement of a DATAGRAM frame only indicates that the transport-layer handling on the receiver processed the frame, and does not guarantee that the application on the receiver successfully processed the data.  Thus, this signal SHOULD NOT replace application-layer signals that indicate successful processing. </ins> 5.3.


timing of these acknowledgements is not used for loss recovery. As with any ack-eliciting frame, when a sender suspects that a packet <del> containing only DATAGRAM frames has been lost, it MAY send probe packets to elicit a faster acknowledgement as described in Section 6.2.4 of RFC9002. </del>  If a sender detects that a packet containing a specific DATAGRAM frame might have been lost, the implementation MAY notify the

RFC 9002 doesn't equivocate regarding whether probes are sent: ack-eliciting packet in the packet number space as a probe. This text uses "MAY", which strongly suggests that probes are discretionary. That's not true.

timing of these acknowledgements is not used for loss recovery. As with any ack-eliciting frame, when a sender suspects that a packet  <ins> containing only DATAGRAM frames has been lost, it sends probe packets to elicit a faster acknowledgement as described in Section 6.2.4 of RFC9002. </ins> If a sender detects that a packet containing a specific DATAGRAM frame might have been lost, the implementation MAY notify the


is important to note that acknowledgement of a DATAGRAM frame only indicates that the transport-layer handling on the receiver processed the frame, and does not guarantee that the application on the <del> receiver successfully processed the data.  Thus, this signal SHOULD NOT replace application-layer signals that indicate successful </del>  processing. 5.3.

The use of "SHOULD NOT" implies that a) this is discretionary, and b) that applications might be able to rely on it in some cases. I don't believe that either is true unless the protocol were specified entirely differently. (And that alternative specification wouldn't be practical anyway.)

is important to note that acknowledgement of a DATAGRAM frame only indicates that the transport-layer handling on the receiver processed the frame, and does not guarantee that the application on the  <ins> receiver successfully processed the data.  Thus, this signal cannot replace application-layer signals that indicate successful </ins> processing. 5.3.


handshake, which has a basic packet loss retransmission timer. This can allow loss recovery to occur more quickly for QUIC data. <del> QUIC datagrams, while unreliable, can support acknowledgements, allowing applications to be aware of whether a datagram was successfully received. </del>  <del> QUIC datagrams are subject to QUIC congestion control, allowing applications to avoid implementing their own. These reductions in connection latency, and application insight into the delivery of datagrams, can be useful for optimizing audio/video streaming applications, gaming applications, and other real-time network applications. </del>  Unreliable QUIC datagrams can also be used to implement an IP packet tunnel over QUIC, such as for a Virtual Private Network (VPN).

Drop the text on acknowledgments. Then modify the congestion control thing to make the benefit clearer. A unified congestion controller can be more efficient.
-- URL This is not an application-layer signal. While the peer might have received the packet and processed it at the QUIC layer, this does not guarantee that the DATAGRAM frame contents were processed by the application. The draft already acknowledges this in . However, the treatment in the motivation could be misleading. Consider dropping this item.
-- URL Isn't it the case that they are acknowledged? (This one is just a nit, the next issue will be more serious.)

handshake, which has a basic packet loss retransmission timer. This can allow loss recovery to occur more quickly for QUIC data.  <ins> QUIC datagrams are subject to QUIC congestion control.  Providing a single congestion control for both reliable and unreliable data can be more effective and efficient. </ins>  <ins> These features can be useful for optimizing audio/video streaming applications, gaming applications, and other real-time network applications. </ins> Unreliable QUIC datagrams can also be used to implement an IP packet tunnel over QUIC, such as for a Virtual Private Network (VPN).


transmitted with the DATAGRAM frame, like the STREAM frame, MUST be protected either by 0-RTT or 1-RTT keys.  7. This document registers a new value in the QUIC Transport Parameter

I put this in Security Considerations, but I can see that is another good choice (perhaps a better one).
I agree this could also be its own subsection of section 5, but given that this really is just reiterating an existing requirement, it's also OK where it is.
Application protocols that use QUIC are required to define how 0-RTT is used. This draft doesn't really change that, but it does add to what 0-RTT can carry. In an abundance of caution, it should link to .

transmitted with the DATAGRAM frame, like the STREAM frame, MUST be protected either by 0-RTT or 1-RTT keys. <ins> Application protocols that allow DATAGRAM frames to be sent in 0-RTT require a profile that defines acceptable use of 0-RTT; see Section 5.6 of RFC9001. </ins> 7. This document registers a new value in the QUIC Transport Parameter


it RFC9002.  The sender implementation MUST either delay sending the frame until the controller allows it or drop the frame without sending it (at which point it MAY notify the application). <del> Implementations that use packet pacing SHOULD support delaying the transmission of DATAGRAM frames for at least the time it takes to send the paced packets allowed by the congestion controller to avoid dropping frames excessively. </del>  Implementations can optionally support allowing the application to specify a sending expiration time, beyond which a congestion-

I've dramatically simplified what we say for pacing now, to just call it out and the fact that it can add slight delays. NAME NAME thoughts?
I like the latest text, it's clear about things that are required, but it doesn't provide advice that we haven't tested. LGTM.
I am having trouble parsing this. I think that this is saying that if you have a pacer, then an endpoint might delay DATAGRAM frames to avoid overriding the sending rate of the pacer. However, this seems to imply that the maximum delay imposed here is the inter-packet interval of the pacer. I think that is a bad assumption, even if it might be a simplifying assumption that some implementations might choose to adopt. In some cases, the pacer interval will be too slow for the application (for low BDP, high RTT connections especially) or in some cases the pacer interval might be short enough to allow multiple intervals before dropping the packet (for high BDP, short RTT connections). The goal should just be to avoid deferring DATAGRAM sending indefinitely, with controls over any delays being available to applications. Maybe:
I think the goal here was to prevent implementations naively dropping DATAGRAM frames anytime they couldn't be sent immediately, which would be a pathologically bad case when a packet pacer is being used. But good point that this should be clarified.
Early in our VPN implementation, we sent QUIC datagrams without any queueing, assuming that if a datagram gets dropped, then we exceeded the cwnd (and thus the estimated channel capacity), so it would likely not reach the peer anyways. In practice, this turned out to be a terrible idea that lead to underutilization of the channel; our implementation always does pacing, so if you don't queue the datagram, it will get dropped unless it just happens to fit into the current pacing quantum. Because pacing is RECOMMENDED for QUIC, I believe some form of queueing should be RECOMMENDED for datagrams, otherwise the users risk running into the same hard-to-debug problem that we had.
I think adding a note about pacing to the "Congestion Control" subsection makes sense to me.
+1, if you're in the bucket of implementations that have insight into their pacing algorithm, it should be reasonable to expect that you'd avoid dropping your own datagrams unless you make a queue and then build up too much in that queue.
Small wording nits around "allows it" twice, but I don't have a radically better wording handy so looks good :)

it RFC9002.  The sender implementation MUST either delay sending the frame until the controller allows it or drop the frame without sending it (at which point it MAY notify the application).  <ins> Implementations that use packet pacing (Section 7.7 of RFC9002) can also delay the sending of DATAGRAM frames to maintain consistent packet pacing. </ins> Implementations can optionally support allowing the application to specify a sending expiration time, beyond which a congestion-


require a profile that defines acceptable use of 0-RTT; see Section 5.6 of RFC9001.  7. This document registers a new value in the QUIC Transport Parameter

Thank you!
Rephrasing what I mentioned at the mic, imagine a scenario where an application uses DATAGRAM to send a single fixed message ("fire the missile"). An adversary on path can start selectively dropping packets and checking to see whether or not they're retransmitted to learn whether or not this special message was sent. (Retransmission detection could be done by looking at the size of the QUIC packet carrying the DATAGRAM, for example.) I don't claim this is easy to do in practice, or useful, but I think it does raise interesting questions about how this new frame affects QUIC's security posture. Perhaps some text in the security considerations is needed? (Issue copied from , by NAME on 2019-11-19)
from NAME on 2020-02-21: How would that be different from streams in ordinary QUIC? Here you can also drop packets and look for retransmissions of a specific size? If there is overlap, this issue belongs to QUIC transport security considerations in general.
I think the key here is that, in dropping the special datagram frame and not seeing anything retransmitted, one could learn something about what was sent.
If we take my recommendation in to always at least send out a PING frame/packet for PTOs or losses of DATAGRAM frame/packets then, it may mitigate this leakage some. There would always be something sent out on these selective losses. I'm not sure how hard it would be to determine if the lost packet was a DATAGRAM or not with this behavior. My guess is that it would be implementation dependent.
PING doesn't really cut it here, because presence and size matter.
Does the attacker have a way of knowing that the packet that was dropped contained a DATAGRAM, vs some other non-retransmitted frame, such as PADDING or PING or ACK?
Probably not, though I would say that the retransmission logic for those three frames could be different enough that the choice of frame might leak. Nothing much doing here, though we might note that retransmission logic might expose information that can be used for traffic analysis.

require a profile that defines acceptable use of 0-RTT; see Section 5.6 of RFC9001. <ins> The use of DATAGRAM frames might be detectable by an adversary on path that is capable of dropping packets.  Since DATAGRAM frames do not use transport-level retransmission, connections that use DATAGRAM frames might be distinguished from other frames using the different response to loss. </ins> 7. This document registers a new value in the QUIC Transport Parameter


DATAGRAM frames contain the following fields: <del> A variable-length integer specifying the length of the datagram in bytes.  This field is present only when the LEN bit is set.  If the LEN bit is not set, the datagram data extends to the end of the QUIC packet.  Note that empty (i.e., zero-length) datagrams are allowed. </del>  The bytes of the datagram to be delivered.

and a bit of editorializing of thee text to match RFC 9000.
If you're receptive to this suggestion I am happy to file a PR
Oh wow I hadn't realized that we never updated to the new style. A PR would be lovely please!

DATAGRAM frames contain the following fields:  <ins> A variable-length integer specifying the length of the Datagram Data field in bytes.  This field is present only when the LEN bit is set to 1.  When the LEN bit is set to 0, the Datagram Data field extends to the end of the QUIC packet.  Note that empty (i.e., zero-length) datagrams are allowed. </ins> The bytes of the datagram to be delivered.


Although DATAGRAM frames are not retransmitted upon loss detection, they are ack-eliciting (RFC9002).  Receivers SHOULD support delaying <del> ACK frames (within the limits specified by max_ack_delay) in reponse </del>  to receiving packets that only contain DATAGRAM frames, since the <del> timing of these acknowledgements is not used for loss recovery. </del>  As with any ack-eliciting frame, when a sender suspects that a packet containing only DATAGRAM frames has been lost, it sends probe packets

This was a suggestion from NAME
Can someone explain why the ack from Datagram-only packets are not used for loss recovery? I would appreciate if someone can point me to an existing discussion or explanation. Thanks.
That's because datagram-only packets don't contain any frames that need to be retransmitted - so there's no recovery to be done. In other words: what was lost doesn't need to be retransmitted. Therefore it's not urgent to let the peer know that this packet was lost because the peer isn't rushing to retransmit anything.
Side note: the last sentence is not correct: the peer does not know that the packet only contains Datagram frames, because the packet has been lost. Back to the original text quoted above. The ack for a Datagram-only packet can still be use to detect loss of other packets, therefore it may be used for loss recovery. In my opinion it should be treated the same as the other ack-eliciting packets. Can we improve he sentence?
In that sentence, the peer was the endpoint who sent the datagram-only packet, so they know what frames it contains. Apologies if that was unclear. Those aren't mutually exclusive. If you receive a train of many datagram-only packets, it is useful to delay acks. If you receive a datagram-only packet whose packet number is more than one apart from the previous packet you saw, then it's indicative of a lost (or reordered) packet so you can act on that to trigger a more urgent ack, regardless of the frame content of the latest received packet. But it's possible we can improve the phrasing in the draft. Do you have a proposal in mind?
It feels like the key realization chain here is: DATAGRAM-only packets have nothing to retransmit if they're lost Therefore, senders don't need to keep the payload of these packets around in case they need to retransmit them Therefore, receipt of an acknowledgement for a DATAGRAM-only packet frees minimal resources on the sender's side Therefore, it's not urgent for receivers to signal that they successfully received a DATAGRAM-only packet So perhaps something like: >Receivers SHOULD support delaying ACK frames (within the limits specified by maxackdelay) in response to receiving packets that only contain DATAGRAM frames, since the sender takes no action if these packets are temporarily unacknowledged. Receivers SHOULD continue to send ACK frames when conditions indicate a packet might be lost, since the packet's payload is unknown to the receiver, and when dictated by maxackdelay or other protocol components. Note also "reponse" => "response"
Thanks. That is more clear.
Seems strictly more clear, nice!This is an editorial change that clarifies the intent of the text.

Although DATAGRAM frames are not retransmitted upon loss detection, they are ack-eliciting (RFC9002).  Receivers SHOULD support delaying  <ins> ACK frames (within the limits specified by max_ack_delay) in response </ins> to receiving packets that only contain DATAGRAM frames, since the  <ins> sender takes no action if these packets are temporarily unacknowledged.  Receivers will continue to send ACK frames when conditions indicate a packet might be lost, since the packet's payload is unknown to the receiver, and when dictated by max_ack_delay or other protocol components. </ins> As with any ack-eliciting frame, when a sender suspects that a packet containing only DATAGRAM frames has been lost, it sends probe packets


5.1. DATAGRAM frames belong to a QUIC connection as a whole, and are not <del> strongly associated with any stream ID at the QUIC layer.  However, it is expected that applications will want to differentiate between </del>  specific DATAGRAM frames by using identifiers, such as for logical flows of datagrams or to distinguish between different kinds of datagrams.

From list discussion: What does "strongly associated" mean in this context? We should remove "strongly".
I agree, removing strongly seems best
Thank you!

5.1. DATAGRAM frames belong to a QUIC connection as a whole, and are not  <ins> associated with any stream ID at the QUIC layer.  However, it is expected that applications will want to differentiate between </ins> specific DATAGRAM frames by using identifiers, such as for logical flows of datagrams or to distinguish between different kinds of datagrams.


places a limit on the maximum size of DATAGRAM frames, that limit can be further reduced by the max_packet_size transport parameter and the Maximum Transmission Unit (MTU) of the path between endpoints. <del> DATAGRAM frames cannot be fragmented, therefore application protocols need to handle cases where the maximum datagram size is limited by other factors. </del>  5.1.

could also do s/,/; but whatever.
I'm not qualified to handle this level of copy-editing, this sounds like something best suited for the RFC editor.

places a limit on the maximum size of DATAGRAM frames, that limit can be further reduced by the max_packet_size transport parameter and the Maximum Transmission Unit (MTU) of the path between endpoints.  <ins> DATAGRAM frames cannot be fragmented; therefore, application protocols need to handle cases where the maximum datagram size is limited by other factors. </ins> 5.1.


The QUIC Transport Protocol RFC9000 provides a secure, multiplexed connection for transmitting reliable streams of application data. <del> Reliability within QUIC is performed on a per-stream basis, so some frame types are not eligible for retransmission. </del>  Some applications, particularly those that need to transmit real-time data, prefer to transmit data unreliably.  In the past, these This document defines two new DATAGRAM QUIC frame types, which carry application data without requiring retransmissions. <del> Note that DATAGRAM frames are only meant for unreliable transmissions.  Reliable transmission is already supported by QUIC via STREAM frames. </del> Discussion of this work is encouraged to happen on the QUIC IETF mailing list quic@ietf.org [3] or on the GitHub repository which contains the draft: https://github.com/quicwg/datagram [4].

In the introduction, it says "Reliability within QUIC is performed on a per-stream basis, so some frame types are not eligible for retransmission" Is it really per-stream? Some frames are retransmitted even though they aren't associated with a stream. Really, the detection mechanism is packet-based, the retransmission decision is made on a per-frame basis, and for stream-specific frames there is the added consideration of stream state (i.e. do not resend if reset) I suggest replacing the sentence above with "Some QUIC frame types are not eligible for retransmission."
Do you have examples? It's common for an action to be attempted again, but not for the frame to be retransmitted. That's not quite right. QUIC does not retransmit stream frames, it instead looks at stream state to see what data needs to be retransmitted - there's an important conceptual difference there: let's say you had 100 bytes left in your packet and therefore sent 98 bytes of stream data and that got lost but now you have a full 1200 payload available, you shouldn't retransmit that 100 byte STREAM frame and instead create a new STREAM frame with as much data as you want to send. states:
So I agree with everything you're saying factually; this is an editorial, semantic gripe about what it means that reliability is on a per-stream basis and what it means that a frame is retransmitted. It's true that, for example, MAX_DATA frames aren't resent verbatim, but RFC 9000 addresses them in a section called "retransmission of information" due to the common-sense meaning of the term. What are you trying to communicate by saying that reliability is "on a per-stream basis"?
I agree that this sentence is confusing. Perhaps we just remove it entirely? NAME NAME thoughts?
NAME NAME made a proposal with
LGTM modulo one commentLGTM

The QUIC Transport Protocol RFC9000 provides a secure, multiplexed connection for transmitting reliable streams of application data.  <ins> QUIC uses various frame types to transmit data within packets, and each frame type defines whether or not the data it contains will be retransmitted.  Streams of reliable application data are sent using STREAM frames. </ins> Some applications, particularly those that need to transmit real-time data, prefer to transmit data unreliably.  In the past, these This document defines two new DATAGRAM QUIC frame types, which carry application data without requiring retransmissions.  Discussion of this work is encouraged to happen on the QUIC IETF mailing list quic@ietf.org [3] or on the GitHub repository which contains the draft: https://github.com/quicwg/datagram [4].


deliver the data to the application immediately, as long as it is able to process the frame and can store the contents in memory. <del> DATAGRAM frames MUST be protected with either 0-RTT or 1-RTT keys. </del>  Note that while the max_datagram_frame_size transport parameter places a limit on the maximum size of DATAGRAM frames, that limit can

Section 5: says DATAGRAM frames MUST be protected with either 0-RTT or 1-RTT keys. I would suggest to add the "otherwise what happens" part. One/two like explanation will be helpful and I think that will improve the understanding of this requirement.
Almost all QUIC frame types are like this. I would suggest saying "Like STREAM frames, ..." to make this obvious for the rationale.

deliver the data to the application immediately, as long as it is able to process the frame and can store the contents in memory.  <ins> Like STREAM frames, DATAGRAM frames contain application data and MUST be protected with either 0-RTT or 1-RTT keys. </ins> Note that while the max_datagram_frame_size transport parameter places a limit on the maximum size of DATAGRAM frames, that limit can


willing to receive such frames on this connection. An endpoint MUST NOT send DATAGRAM frames until it has received the <del> max_datagram_frame_size transport parameter with a non-zero value. An endpoint MUST NOT send DATAGRAM frames that are larger than the max_datagram_frame_size value it has received from its peer.  An endpoint that receives a DATAGRAM frame when it has not indicated </del>  support via the transport parameter MUST terminate the connection with an error of type PROTOCOL_VIOLATION.  Similarly, an endpoint that receives a DATAGRAM frame that is larger than the value it sent

Section 3 says: “An endpoint MUST NOT send DATAGRAM frames until it has received the maxdatagramframesize transport parameter with a non-zero value.” I guess you assume that having received maxdatagramframesize in a previous connection, when 0-RTT is used, counts but interpreting this MUST strictly, you cannot send datagrams in 0-RTT. Maybe that can be clarified. Also if datagrams are used with 0-RTT but for some reason are not supported by the server anymore, it would actually be useful to have a more specific error message if the connection is closed by the server.
This is already discussed in Section 3: That's not how 0-RTT works in QUIC. If the server wishes to no longer support datagrams, it MUST reject 0-RTT data and therefore it won't see those DATAGRAM frames.
NAME given David's explanation, are we OK to close this one?
I know that this is explained in section 3, but what I'm saying is that the MUST NOT in the text I cited contradicts this text (if you interpret the MUST NOT strictly but maybe I'm to strict here). So to be more clear one could say: "... MUST NOT send DATAGRAM frames until it has received the maxdatagramframesize transport parameter with a non-zero value in the handshake, or a previous handshake if 0-RTT is used." Regarding the server not supporting datagram in 0-RTT anymore, my assumption was that if the server does not have any state if datagram was used but does not support datagram anymore for some reason, it would receive a 0-RTT with a datagram and probably close the connection based on this text: "An endpoint that receives a DATAGRAM frame when it has not indicated support via the transport parameter MUST terminate the connection with an error of type PROTOCOLVIOLATION." I guess you are saying if 0-RTT and datagram is used the server MUST remember that datagram was negotiated? However, there is also this text: "If a client stores the value of the maxdatagramframesize transport parameter with their 0-RTT state, they MUST validate that the new value of the maxdatagramframesize transport parameter sent by the server in the handshake is greater or equal to the stored value; if not, the client MUST terminate the connection with error PROTOCOL_VIOLATION." So this would mean larger datagram are accepted in 0-RTT but then if the server indicates a smaller value for 1-RTT, the connection needs to be terminated? Why? Also if datagram was used in the previous connection, then 0-RTT is used without datagrams, the server can still not indicate in the handshake that is not supporting datagram anymore? All I'm saying is that these things could be further clarified to avoid any doubt.
Adding "or a previous handshake if 0-RTT is used" is fine by me. Your assumption is incorrect. If servers used to support datagram and they accept 0-RTT, they must also support datagram in the new connection. That's spelled out here: Clarifications are always good. Can you send a PR?

willing to receive such frames on this connection. An endpoint MUST NOT send DATAGRAM frames until it has received the  <ins> max_datagram_frame_size transport parameter with a non-zero value during the handshake (or during a previous handshake if 0-RTT is used).  An endpoint MUST NOT send DATAGRAM frames that are larger than the max_datagram_frame_size value it has received from its peer. An endpoint that receives a DATAGRAM frame when it has not indicated </ins> support via the transport parameter MUST terminate the connection with an error of type PROTOCOL_VIOLATION.  Similarly, an endpoint that receives a DATAGRAM frame that is larger than the value it sent


7. Clients that act upon a request to rate limit are potentially re- <del> identifiable (see Section 7.1 of DNS-PRIVACY) because they react to </del>  information that might only be given to them.  Note that this might apply to other fields too (e.g.  Retry-After).

RFC6973 Section 5.2.1 would be a better reference.
Ok, thanks URL

7. Clients that act upon a request to rate limit are potentially re-  <ins> identifiable (see Section 5.2.1 of PRIVACY) because they react to </ins> information that might only be given to them.  Note that this might apply to other fields too (e.g.  Retry-After).


The RateLimit-Policy field (see ratelimit-policy-field), might contain information on the associated time window. <del> This field MUST NOT occur multiple times and can be sent in a trailer section. </del>  3.2. These examples show multiple policies being returned: <del> This field MUST NOT occur multiple times and can be sent in a trailer section. </del>  3.3. The field is an Item and its value is a non-negative Integer expressed in service-limit.  Parameters are not allowed. <del> This field MUST NOT occur multiple times and can be sent in a trailer section. </del>  Clients MUST NOT assume that a positive RateLimit-Remaining field value is a guarantee that further requests will be served. it mitigates the risk related to thundering herd when too many clients are serviced with the same timestamp. <del> This field MUST NOT occur multiple times and can be sent in a trailer section. </del>  An example of RateLimit-Reset field use is below.

According to URL there is no need to specify that information.
This text appears in relation to Item fields. By their nature Items can only occur on one field line (the proper terminology, btw), so this requirement is extraneous, and should be removed.
... and it's also on RateLimit-Policy, even though that is a List field.

The RateLimit-Policy field (see ratelimit-policy-field), might contain information on the associated time window.  <ins> This field can be sent in a trailer section. </ins> 3.2. These examples show multiple policies being returned:  <ins> This field can be sent in a trailer section. </ins> 3.3. The field is an Item and its value is a non-negative Integer expressed in service-limit.  Parameters are not allowed.  <ins> This field can be sent in a trailer section. </ins> Clients MUST NOT assume that a positive RateLimit-Remaining field value is a guarantee that further requests will be served. it mitigates the risk related to thundering herd when too many clients are serviced with the same timestamp.  <ins> This field can be sent in a trailer section. </ins> An example of RateLimit-Reset field use is below.


5.2. <del> As is the ordinary case for HTTP caching (HTTP-CACHING), a response with RateLimit fields might be cached and re-used for subsequent requests.  A cached response containing RateLimit fields does not modify quota counters but could contain stale information.  Clients interested in determining the freshness of the RateLimit fields could rely on fields such as the Date header field and on the time window of a quota policy. </del>  6.

It would be better to say that rate limiting information in what we used to call non-firsthand responses (basically, anything with an Age header) should be ignored.
Is the following text ok?
Looks good, just a few suggested tweaks:

5.2.  <ins> HTTP-CACHING defines how responses can be stored and reused for subsequent requests, including those with RateLimit fields.  Because the information in RateLimit fields on a cached response may not be current, they SHOULD be ignored on responses that come from cache (i.e., those with a positive current_age; see Section 4.2.3 of HTTP- CACHING). </ins> 6.


Clients MUST NOT assume that a positive RateLimit-Remaining field value is a guarantee that further requests will be served. <del> A low RateLimit-Remaining field value is like a yellow traffic-light for either the number of requests issued in the time window or the request throughput: the red light may arrive suddenly (see providing- ratelimit-fields). </del>  For example:

This should be specified without resorting to a metaphor that some may find confusing.
I'm ok to removing this line.

Clients MUST NOT assume that a positive RateLimit-Remaining field value is a guarantee that further requests will be served.  <ins> When the value of RateLimit-Remaining is low, it indicates that the server may soon throttle the client (see providing-ratelimit-fields). </ins> For example:


exceeds that limit, it MAY not be served. The field is an Item and its value is a non-negative Integer referred <del> to as the "expiring-limit".  Parameters are not allowed. </del>  The expiring-limit MUST be set to the service limit that is closest to reaching its limit, and the associated time window MUST either be: client. The field is an Item and its value is a non-negative Integer <del> expressed in service-limit.  Parameters are not allowed. </del>  This field can be sent in a trailer section. it mitigates the risk related to thundering herd when too many clients are serviced with the same timestamp.  This field can be sent in a trailer section. An example of RateLimit-Reset field use is below.

Parameters are not allowed, and if they appear they MUST be ignored.
Why not just 'If Parameters appear they MUST be ignored'?
NAME we wanted to stress the fact that servers are not expected to provide parameters, otherwise some clients could just consider the whole field value malformed and ignore it.
The normal way to do that is something like:
Ok, I'll rebase this PR tomorrow :) Late night now in Italy. Thanks for all this support :)
In a few places, the specification says "Parameters are not allowed" but does not spell out what should happen if they appear.

exceeds that limit, it MAY not be served. The field is an Item and its value is a non-negative Integer referred  <ins> to as the "expiring-limit".  This specification does not define Parameters for this field.  If they appear, they MUST be ignored. </ins> The expiring-limit MUST be set to the service limit that is closest to reaching its limit, and the associated time window MUST either be: client. The field is an Item and its value is a non-negative Integer  <ins> expressed in service-limit.  This specification does not define Parameters for this field.  If they appear, they MUST be ignored. </ins> This field can be sent in a trailer section. it mitigates the risk related to thundering herd when too many clients are serviced with the same timestamp. <ins> This specification does not define Parameters for this field.  If they appear, they MUST be ignored. </ins> This field can be sent in a trailer section. An example of RateLimit-Reset field use is below.


Servers' behavior may be subject to other conditions like the one shown in the example from service-limit. <del> A client MUST validate the RateLimit fields before using them and check if there are significant discrepancies with the expected ones. This includes a RateLimit-Reset field moment too far in the future (e.g. similarly to receiving "Retry-after: 1000000") or a service- limit too high. </del>  A client receiving RateLimit fields MUST NOT assume that future responses will contain the same RateLimit fields, or any RateLimit passed by a misconfigured server; or a high RateLimit-Reset field value could inhibit clients to <del> contact the server. Clients MUST validate the received values to mitigate those risks. </del>  7.

no normative language in sec- -
This doesn't seem practical; how would validation help? What are they looking for?
For example, implementations are expected to validate received values like Or The same goes for other fields, e.g. A similar practice, in a different context though, is mentioned in URL Hints are welcome!
Considering the objective of the RateLimit headers is to allow a client to avoid being throttled, the consequence of a client ignoring the RateLimit-* headers is just a client getting throttled. If we believe that a client may need to protect itself against malicious values injected into these headers, then I it should be sufficient to give clients the permission to ignore the headers if they believe the values are not reasonable. Suggest wording like: However, considering that processing these headers is optional for a client anyway, I'm not sure this comment is absolutely necessary.
Part of the problem here is that "valid" is being used in different senses: to indicate that it's a valid Structured Field to indicate that it's semantically valid (e.g., "too big") For the latter, since it's so fuzzy (at least to date), I'd suggest using something like "reasonable" instead.
This seems quite arbitrary for a MUST; what is "too far in the future" and "too high"?
Agree, this is an tricky point. I think an implementation: MUST define internal threeshold of allowed values; MUST validate ratelimit fields according to those thresholds. I think we could borrow some text from Signature. See
OK, but RFC2119 language is for interoperability, and this doesn't provide any; different implementations (or deployments) are going to have different values, and that may cause issues. It'd be better to defines precise limits if you think this is important enough to merit a MUST. Personally, I'd remove the 2119 language and just give guidelines about what implementations should consider when they're looking for abuse.
Ok, great!
Addressed in

Servers' behavior may be subject to other conditions like the one shown in the example from service-limit.  <ins> A client is responsible for ensuring that RateLimit field values returned cause reasonable client behavior with respect to throughput and latency (see sec-resource-exhaustion and sec-dos). </ins> A client receiving RateLimit fields MUST NOT assume that future responses will contain the same RateLimit fields, or any RateLimit passed by a misconfigured server; or a high RateLimit-Reset field value could inhibit clients to  <ins> contact the server (e.g. similarly to receiving "Retry-after: 1000000"). To mitigate this risk, clients can set thresholds that they consider reasonable in terms of quota units, time window, concurrent requests or throughput, and define a consistent behavior when the RateLimit exceed those thresholds.  For example this means capping the maximum number of request per second, or implementing retries when the RateLimit-Reset exceeds ten minutes. The considerations above are not limited to RateLimit fields, but apply to all fields affecting how clients behave in subsequent requests (e.g.  Retry-After). </ins> 7.


3.3. <del> The "RateLimit-Remaining" field response field indicates the remaining quota units defined in service-limit associated with the client. </del>  The field is an Item and its value is a non-negative Integer expressed in service-limit.  This specification does not define 3.4. The "RateLimit-Reset" field response field indicates the number of <del> seconds until the quota resets. </del>  The field is a non-negative Integer compatible with the delay-seconds rule, because:

clarify that we return only the values related to the quota policy identified by the expiring-limit
If there are multiple policies in effect, which is reflected in these fields? RateLimit-Limit has advice about this, but these do not.
Ok, we should clearly state that all the three basic fields return the values associated to the expiring-limit.

3.3.  <ins> The "RateLimit-Remaining" response field indicates the remaining quota units associated to the expiring-limit. </ins> The field is an Item and its value is a non-negative Integer expressed in service-limit.  This specification does not define 3.4. The "RateLimit-Reset" field response field indicates the number of  <ins> seconds until the quota associated to the expiring-limit resets. </ins> The field is a non-negative Integer compatible with the delay-seconds rule, because:


4. A server uses the RateLimit header fields to communicate its quota <del> policies.  Sending the RateLimit-Limit and RateLimit-Reset header fields is REQUIRED; sending RateLimit-Remaining header field is RECOMMENDED. </del>  A server MAY return RateLimit header fields independently of the response status code.  This includes on throttled responses.  This

NAME suggestion on server behavior. Thx! -
In section 4 it says, In section 4.1 it says, These statements seem contradictory. I do not see how we can declare as REQUIRED considering many APIs do not and will not implement these headers. It seems to me that if a client receives a header they can choose to either count the requests they make within the time window or they can monitor the . With neither of those values it would be difficult to do something useful. How would you feel about words like this:
NAME we are open to clarifications. The rationale for the current behavior is in URL I think RateLimit-Reset is always REQUIRED.
NAME In theory someone could use RateLimit-Limit to provide a limited number of "free" requests as a trial for their API. They may not choose to have a reset window and just require payment before allocating more calls. This is a bit of an edge case use of the headers, but might be just enough of a justification to limit constraints that we add. Having said this, I still don't understand how we can make any of these headers required considering there are so many existing APIs that don't return these headers today. How does a developer decide if they MUST return these headers for their API? Assuming the "free trial" scenario is not considered a valid scenario for this header, then how would you feel about these words?
SGTM.

4. A server uses the RateLimit header fields to communicate its quota  <ins> policies.  A response that includes the RateLimit-Limit header field MUST also include the RateLimit-Reset.  It MAY also include a RateLimit-Remaining header field. </ins> A server MAY return RateLimit header fields independently of the response status code.  This includes on throttled responses.  This


informative. The field is a non-empty List of Items.  Each item is a quota-policy.  This field can convey the time window associated with the expiring- limit, as shown in this example: These examples show multiple policies being returned:  This field cannot appear in a trailer section. 3.3.

clarify that quota policies are unique in policy.
If server defines two policies, e.g. and the response header contains, How do I know if I can make 2 more requests, or download 2 more bytes? Is having multiple policies with different quota units not allowed?
Good catch! The list items MUST be unique. I will specify this for now. WDYT about switching Policy to a Dictionary? This requires some tweak though e.g. prefixing the units with . cc: NAME
I'd use a short name for the policy, so that it could be referred to separately; e.g.,

informative. The field is a non-empty List of Items.  Each item is a quota-policy. <ins> Two quota policies MUST NOT be associated with the same quota units value. </ins> This field can convey the time window associated with the expiring- limit, as shown in this example: These examples show multiple policies being returned: <ins> An example of invalid header field value with two policies associated with the same quota units: </ins> This field cannot appear in a trailer section. 3.3.


available in that moment.  Nonetheless servers MAY decide to send the "RateLimit" fields in a trailer section.  5. This section documents the considerations advised in Section 16.3.3 with the expected ones.  This includes a "RateLimit-Reset" moment too far in the future or a "service-limit" too high.  Malformed "RateLimit" fields MAY be ignored. A client SHOULD NOT exceed the "quota-units" expressed in "RateLimit-

servers are not forced to always return ratelimit fields clients must not assume the presence of ratelimit fields performance hints for providers
Waiting for NAME then :)
NAME PTAL :)
Looks great to me. The intent is nice and clear.
The spec to define header inter-dependencies and returning policies [x] Limit -> Reset [x] Limit, Remaining, Reset Do I have to always return all headers? No Do I have to expose all headers or I can only expose a subset? Limit,Reset are REQUIRED, Remaining is OPTIONAL Do we need to add some considerations on client implementation? Already present.
clarifies that clients MUST NOT assume a server always returns fields
Comment by Erik. In https://ietf-wg-URL You state ‘A server MAY use one or more "RateLimit" response fields defined in this document to communicate its quota policies.’ but it does not clearly define how they would be used in isolation or how a client should respond to a limited set of headers (other than discussed earlier above) My feeling is RateLimit-Limit can be optional and RateLimit-Remaining and RateLimit-Reset can be used as a pair. RateLimit-Reset would only have value in isolation if sent with a 429 response code, on other responses it would be meaningless.
It could also make sense to return Reset with other status codes such as and , so I think we don't really want to mandate a specific relation other than, at most, saying that unsuccessful requests (for a broad definition of those, possibly including and at a minimum) with this field are expected to be unsuccessful until that time elapses.
After a couple of years of adoption, my experience is the following: implementations use all 3 fields. Though theoretically one could just return and , I never found an implementation doing this :heavycheckmark: implementations don't know Remaining, and just return Limit and Reset with a fixed window (eg. 1 second). We can then recommend the following: Limit, Reset: MUST Remaining: SHOULD Since https://ietf-wg-URL I now think that we shouldn't mess with status codes. NAME if this sounds reasonable to you, I can provide a PR
Yes, it sounds good, thank you!
Looks good!

available in that moment.  Nonetheless servers MAY decide to send the "RateLimit" fields in a trailer section. <ins> 4.1. Servers are not required to return "RateLimit" fields in every response, and clients need to take this into account.  For example, an implementer concerned with performance might provide "RateLimit" fields only when a given quota is going to expire. Implementers concerned with response fields' size, might take into account their ratio with respect to the payload data, or use header- compression http features such as HPACK. </ins> 5. This section documents the considerations advised in Section 16.3.3 with the expected ones.  This includes a "RateLimit-Reset" moment too far in the future or a "service-limit" too high. <ins> A client receiving "RateLimit" fields MUST NOT assume that subsequent responses contain the same "RateLimit" fields, or any "RateLimit" fields at all. </ins> Malformed "RateLimit" fields MAY be ignored. A client SHOULD NOT exceed the "quota-units" expressed in "RateLimit-