New transport parameters can be registered according to the rules in iana-transport-parameters.  8. Address validation is used by QUIC to avoid being used for a traffic A peer has migrated to a different network when the endpoint had disabled migration.  The cryptographic handshake failed.  A range of 256 values is reserved for carrying error codes specific to the cryptographic handshake that is used.  Codes for errors occurring when TLS is

I’d prefer to always close the connection instead of silently discarding the frames. The problem with discarding frames is that while it doesn’t really hurt with “normal” TLS 1.3, there might be TLS extensions which require the reliable delivery of post-Handshake messages. Furthermore, acknowledging a packet means that a peer has processsed all the frames. Silently dropping frames seems like an awkward exception to that rule.
NAME Good point. That seems like a good principle to apply here.
We discussed this in Tokyo (), and there are basically 3 options here: Implement proper flow control for CRYPTO frames Specify a buffer size (50k is the example from the Tokyo discussion) Drop CRYPTO frames if an endpoint has to buffer too many My sense of the room was that the 3rd option was the preferred option for now. When we have TLS extensions where we care about reliable delivery of post-handshake messages, that sounds like a good time to add proper flow control. I don't like picking a fixed buffer size for the same reason as using timers for discarding keys. If we pick a buffer size now, how do we know that it will be big enough for whatever future TLS extensions might get used with QUIC (or future key exchanges)? I don't think that this breaks a rule of acknowledging packets means that the peer has processed all frames in that packet. For example, if an endpoint receives a packet containing stream data that is a retransmission of previously received data, that endpoint in effect is dropping that STREAM frame. Here, the CRYPTO frame is being processed (and the endpoint is immediately choosing to do nothing based on it).
As NAME said, this was discussed in Tokyo and I believe included in the recent consensus call on the list, so I'd prefer to land this and gain some implementation experience, even if I might personally prefer a different solution. As NAME states, there are 3 options, and the WG has repeatedly said no to flow control and a number of people didn't like a fixed limit on the entire connection, so we ended up with the 3rd option.
I don't think this is a fair characterization. For the sender, it doesn't really matter if it was this copy of the data, or an earlier STREAM frame containing exactly the same data. The ACK signals that the data was delivered to the application, and that's all that matters. Exactly the opposite. Processing the CRYPTO frame would mean to deliver it to the TLS stack. Here it's dropped one layer below that. Everywhere else in the spec, we use "processing" in the sense that the data has been acted upon, not in the sense that the frame was merely parsed.
Is this buffering covering the internal TLS state engine, or is that separate?
This is buffering in the QUIC layer (specifically buffering out of order data that can't be delivered to the TLS stack) - the TLS stack probably has its own internal buffers, but that's opaque to QUIC.
As NAME says, the TLS stack will be expecting data to be delivered in order. If it can't keep up, or data arrives waaay out of order, that is when the transport needs to decide what to do. (That is why this text belongs in the transport doc and not the TLS one, in case people were wondering.)
NAME Is there anything else I need to do to get this PR merged?
Consider the following situation. The client sends a 0-RTT handshake. The server accepts the handshake, installs the 1-RTT keys and sends its ServerHello...Finished flight. The client can now send a bunch of post-handshake messages to the server which the server cannot process until it receives ClientFinished, so it has to buffer them. We should bound that buffer somehow.
(Client post-handshake messages don't really exist today, but they might in the future, e.g., with URL) Another incarnation of this problem is if the client receives a huge window of CRYPTO frames into the future but not the first byte, so it can't process anything yet. With STREAM frames, we have flow control. With CRYPTO frames during the handshake, we know the handshake itself is bounded. Post-handshake, we don't know a priori the server won't send lots and lots of tickets. Of course, realistically, it won't send that many in a row, so one could get away with just picking a bound. But this is much more of an ad-hoc bound than the handshake. It also may send lots of tickets over the course of a connection, so the bound is in how far ahead it can get, rather than the handshake which can impose a stronger total bound. This then starts looking an awful lot like flow control...
NAME Isn't the expected behavior of the server to drop the 1-RTT packets containing the post-handshake messages until it receives the ClientFinished (that activates the 1-RTT read key)? To put it another way, my understanding is that you can have any size of buffer you want to better cope with packet reordering, but that happens per-packet rather than per-frame (because it's not just the post-handshake messages that cannot be trusted prior to receiving ClientFinished), and that the buffer can be arbitrary sized because you can drop it to trigger a retransmit. Am I missing something?
NAME I agree that the problem exists. However, in my view, the memory exhaustion issue does not get resolved just by having a window for the CRYTO frames. My understanding that most TLS stacks buffers the octets of a handshake message until it receives a complete one. That means that the TLS stack is expected to buffer at most 16MB of junk even if we introduce window to the CRYPTO frames. Considering that, I think it would be preferable to at first discuss introducing bounds for the TLS messages in the TLS working group, then consider how to apply that to QUIC. Until then, each QUIC stack or TLS stack can (continue to) have its own internal limit.
I don't think that's an accurate reduction. TLS stacks usually impose ad-hoc limits tighter than the maximum 2^24, sometimes as a function of the maximum certificate size the caller configures. It's unfortunate that it's ad-hoc but different consumers do indeed tolerate different certificate sizes. It's plausible to do that analysis per-message and to do it per-flight since the set of messages in a flight is one of a few fixed patterns. Post-handshake is very different. The pattern of messages is no longer fixed. Rather it is a property of how the application uses TLS. Indeed the QUIC spec even calls out applications choosing to send tickets at various points during the connection. Thus TLS-level limits, ad-hoc or spec'd, are not sufficient in this case. QUIC needs to do its share of the work. (There's a reason I dislike post-handshake messages so much. :-) People always forget this complexity.)
Re the client Finished case, I guess that pulls in the mess of stuff around there. The spec is a little unclear what a server does in between server and client Finished. In particular, if doing 0-RTT the server needs to at least be willing to send half-RTT data. This text suggests that, since there's a binder, processing out of order is okay. (Unless I'm reading it wrong?) URL I agree that, if you believe the server just shouldn't process 1RTT packets before client Finished at all, the buffering issue is solved. Though it does leave it unable to read the ACKs to half-RTT data which is differently weird. :-) I think the general post-handshake issue is probably the more isolated instance. They're the same problem in terms of buffering, but one has a confounding factor around whether it's actually an issue.
NAME Oh I missed it. Thank you for pointing that out. I had assumed that TLS stacks always expose the 1-RTT read key to the QUIC stacks acting as a server when it receives ClientFinished. I'd be curious to know if there are TLS stacks that takes an alternative approach. I assume that issue is unrelated to 0-RTT, right? IIUC the advise to the QUIC stacks is that until you know that the peer can process a packet, you might want to coalesce the packet you send with another packet that uses a lower-epoch key; then you can have a weak confirmation that the packet has reached the peer. Having such signal is beneficial to not give false information to the congestion controller though the loss recovery code needs to assume that the packet might have been lost, because a stack is allowed to drop a packet that is was unable to decrypt. I agree. I think it is preferable to fix the issue. Maybe I should rephrase my argument to that there should be limits in both layers (i.e. limit on the post-handshake message size in the TLS layer, and the window size in the QUIC layer), because having limit only on one layer does not prevent excessive buffering on the other layer. The two limits could either be a shared single value, two values shared using one extension, or two values shared using different mechanisms.
I'm very much inclined to leave this "unfixed", but to leave a warning that sending large post-handshake messages risks connection failure. That was what I had assumed too, until I thought through the nasty interactions with partial Handshake flights and 0-RTT. That is, the TLS stack makes secrets available at the point that it would normally be installing them for its own use. In that way, a server wouldn't install the 1-RTT read keys until it had processed the finished from the client, exactly as you say. My current patch (which is still under review) releases secrets as soon as possible to avoid the problem where the server can't read Handshake acknowledgements because it is receiving 0-RTT. The main consequence of that is that you need a separate API where the TLS stack tells the QUIC stack what it is willing to send and receive (well, I had a ton of fun fixing tests as well, but that's just because we were a little lazy with certain things in the past).
Right, this issue originally came up as we were thinking through key availability vs key install times. The problem is, as you note, it's weird to allow sending data you cannot read ACKs for. (Conversely, it's weird, perhaps fatally so, to allow receiving data you cannot send ACKs for.) Though it's not the only scenario. The other side may send a burst of tickets. The point is that's not a sufficient warning. This is either a problem or a nitpick depending on the solution. Sending an overly large message, in general, risks connection failure. TLS stacks bound the size there based on some ad-hoc analysis that no sane NewSessionTicket (or other message) will ever exceed such-and-such bytes based on what goes in it. TLS stacks typically buffer up a message before processing, so it makes sense that the TLS stack is responsible for buffering it. But that's distinct from sending many post-handshake messages. TLS over QUIC must also account for that due to the lack of flow control. TLS over TCP does not have this problem because TCP provides flow control. If I try to send a million 100-byte TLS messages over TCP, I'll quickly run into TCP's flow control. QUIC, however, doesn't have such flow control for CRYPTO frames. Instead it relies on a limit from the crypto protocol: URL Although the TLS stack natively bounds a single message rather than total data, it is easy to compute a total limit during the handshake because there are only a fixed handful of messages. Thus the advice there works.... except for post-handshake messages. Thus the issue. Realistically, it's probably fine to handwaive and say: The bound is not total data ever sent at the encryption level, but amount of data ahead of the read position. Pick some ad-hoc limit for post-handshake stuff, enough to fit a bunch of NSTs. Yeah, don't send too many NSTs in a row. You can send a bunch of them over the lifetime of the connection, but space them out a bit so you're sure the receiver isn't waiting on the first byte, having buffered the rest up. Probably you won't end up in a situation where the first NST packet keeps getting lost and as you queue up more and more NSTs over the lifetime of the connection. That would be strange. But it also smells weird, so perhaps proper flow control (toss post-hs stuff onto stream 0?) would be better instead?
Thank you for pointing that out. I forgot about that exception. So while that is the only exception picotls has in terms of key installation timing, I agree that it's merely about having one exception or two... I am totally fine with handwaving, but in case we are to have flow control, I'd prefer having flow control for CRYPTO streams in all epochs. They could all use a fixed-size window that would be defined in the specification. I prefer that because IMO having flow control for every stream (i.e. all the ordinary streams + all the 4 CRYPTO streams) is a simplification.
FWIW we had a similar discussion about stream 0 flow control in URL and during the stream 0 design team discussions.
The problem here is that CRYPTO is not flow controlled. Endpoints can cause arbitrary buffering of CRYPTO frame data. This has very little to do with the handshake (though NAME notes that we have a similar concern for CRYPTO in Initial and Handshake, that is bounded by the handshake timers). Proposal is to allow endpoints to maintain a buffer of any size they choose. If they receive a TLS message that does not fit into that buffer, they can drop all subsequent TLS data. The effect of this is to disable TLS from that point onward. Currently, we don't depend on this frame for anything other that NewSessionTicket, for which loss has no real consequence. We will assign a minimum buffer size that implementations are required to support, probably something small like 4k. A higher limit is possible. If an endpoint finds that this buffer is overrun they MAY discard all CRYPTO frames with higher offsets, or they MAY drop the connection with a code.
In Tokyo:
Please don't close issues until AFTER consensus is declared (not just called).
Minor nits.I think that this is good. Thanks for doing the hard work Nick. I believe that this issue can be resolved without chair involvement as it is a conclusion from the Tokyo meeting.I think this is definitely in a gray area. Generally, I view the action of sending an ACK for a packet as saying "I've received the data you sent me, and I don't need that data any more. Don't retransmit it." It's really up to the receiver to decide what that means. The sender can then always safely assume the data doesn't need to be retransmitted. It can't know when (or if) the data ever gets delivered to the next layer; whether that's the application sitting on top of a stream or TLS on top of the crypto stream. Only by receiving data from that next layer in response can the original sender know for sure.

New transport parameters can be registered according to the rules in iana-transport-parameters. <ins> 7.4. Implementations need to maintain a buffer of CRYPTO data received out of order.  Because there is no flow control of CRYPTO frames, an endpoint could potentially force its peer to buffer an unbounded amount of data. Implementations MUST support buffering at least 4096 bytes of data received in CRYPTO frames out of order.  Endpoints MAY choose to allow more data to be buffered during the handshake.  A larger limit during the handshake could allow for larger keys or credentials to be exchanged.  An endpoint's buffer size does not need to remain constant during the life of the connection. Being unable to buffer CRYPTO frames during the handshake can lead to a connection failure.  If an endpoint's buffer is exceeded during the handshake, it can expand its buffer temporarily to complete the handshake.  If an endpoint does not expand its buffer, it MUST close the connection with a CRYPTO_BUFFER_EXCEEDED error code. Once the handshake completes, if an endpoint is unable to buffer all data in a CRYPTO frame, it MAY discard that CRYPTO frame and all CRYPTO frames received in the future, or it MAY close the connection with an CRYPTO_BUFFER_EXCEEDED error code.  Packets containing discarded CRYPTO frames MUST be acknowledged because the packet has been received and processed by the transport even though the CRYPTO frame was discarded. </ins> 8. Address validation is used by QUIC to avoid being used for a traffic A peer has migrated to a different network when the endpoint had disabled migration. <ins> An endpoint has received more data in CRYPTO frames than it can buffer. </ins> The cryptographic handshake failed.  A range of 256 values is reserved for carrying error codes specific to the cryptographic handshake that is used.  Codes for errors occurring when TLS is


endpoint does not support connection migration (migration).  Peers of an endpoint that sets this transport parameter MUST NOT send any packets, including probing packets (probing), from a local <del> address other than that used to perform the handshake.  This parameter is a zero-length value. </del>  The server's preferred address is used to effect a change in server address at the end of the handshake, as described in

Some infrastructures are likely to load balance on port as well as address.
"Address" has been taken to mean IP and port in other contexts, but no harm in being extra precise here.

endpoint does not support connection migration (migration).  Peers of an endpoint that sets this transport parameter MUST NOT send any packets, including probing packets (probing), from a local  <ins> address or port other than that used to perform the handshake. This parameter is a zero-length value. </ins> The server's preferred address is used to effect a change in server address at the end of the handshake, as described in


can be established by consecutive PTOs, but since the PTO timer is reset when a new ack-eliciting packet is sent, an explicit duration must be used to account for those cases where PTOs do not occur or <del> are substantially delayed.  This duration is the equivalent of kPersistentCongestionThreshold consecutive PTOs, and is computed as follows: ~~~ (smoothed_rtt + 4 * rttvar + max_ack_delay) * ((2 ^ kPersistentCongestionThreshold) - 1) ~~~ </del>  For example, assume: smoothed_rtt = 1 rttvar = 0 max_ack_delay = 0 <del> kPersistentCongestionThreshold = 2 </del>  If an eck-eliciting packet is sent at time = 0, the following scenario would illustrate persistent congestion: The first three packets are determined to be lost when the ACK of packet 4 is received at t=8.  The congestion period is calculated as the time between the oldest and newest lost packets: (3 - 0) = 3. <del> The duration for persistent congestion is equal to: (1 * ((2 ^ kPersistentCongestionThreshold) - 1)) = 3.  Because the threshold was </del>  reached and because none of the packets between the oldest and the newest packets are acknowledged, the network is considered to have experienced persistent congestion.

This simplifies the definition of kPersistentThreshold from an exponent to a PTO multiplier.
I think let's keep that separate. I think might end up with different outcomes anyways.
Curious, why wasn't this listed under draft-20 C.1? URL
Oversight. Fixed now in URL . Thanks!
Both Linux and Windows TCP arm a single PTO for TLP and then arm the RTO. The recommendation in RACK draft URL is also for doing a single TLP. If we keep the value as 2 then we need to add a justification statement for deviating from TCP. It would be good to find out if Yuchung has a any data on why Linux settled on a single TLP (PTO).
The recovery draft currently says MUST send 1, and MAY send up to 2 (URL). Is that not good enough?
The issue is about the length of the persistent congestion period. If the -16 draft recommended 1 TLP before the first RTO instead of 2 TLPs before RTO(See URL), then the natural persistent congestion time period would be 2 PTO instead of 3 PTO. As an editorial issue, as Jana and I discussed yesterday, we should make kPersistentCongestionThreshold a direct constant, not an exponent - 1, because it's making this more confusing to think about.
Last time I talked to Yuchung, I remember him saying they chose 1 somewhat arbitrarily. I'm going to argue for 2 (3 * PTO) as a good number because it provides enough time for the PTO timer to go off twice now that the PTO has exponential backoff. This is in some ways more similar to 1 TLP and then 1RTO in terms of network load. Additionally, I believe making it only 1 would allow a single tail packet to be lost and persistent congestion to be declared if there was some RTT variation? That certainly seems undesirable. Maybe the rationale for why this value was chosen should be clarified further?
facepalm NAME of course. (Also, FWIW, PR makes this a simple multiplier, so let's just use that language.) So I think I may have the opposite concern to NAME The mechanisms we are talking about are subtly different, so let's just talk about behavior instead of the mechanisms. TCP RTO uses a minRTO while PTO does not. As a result, in the common case, persistent congestion for TCP will be declared at minRTO (or later). We risk declaring persistent congestion too soon with QUIC, even with 3 PTOs, since minRTO is at least 200ms for TCP, AFAIK. QUIC might be too aggressive in declaring persistent congestion. This might be actually bad for performance. Anecdotally, I remember that Yuchung had tried turning on and found that it increased spurious RTOs. While spurious PTOs are not bad in QUIC, we don't yet have a way to detect spuriously declared persistent congestion. This makes me wonder if we should use a min value for declaring persistent congestion. I think this is justifiable as based on best current practice that we know in TCP and that declaring congestion too soon might hurt throughput. (Note that a min is still unnecessary for QUIC PTO because spurious PTOs have only minor bandwidth cost.) As a strawman, I'll propose that we set the persistent congestion time period to max(PTO, 200ms). Why PTO and not 3 * PTO? Because I believe (and someone can prove me wrong) that TCP actually does use a single RTO period for declaring an RTO, despite potentially sending TLPs in that time period. From the RACK draft, my understanding is that sending TLPs in TCP does not actually change the RTO timer, which runs for the same amount of time (from the last ack received in practice or the earliest outstanding data).
I edited the title to indicate 2 or 3 instead of 1 or 2, now that has landed. Yes, TLPs in TCP do not change RTO. One of the reasons TLP was added in the way it was is that it doesn't really interact with RTO at all, which made it convenient. My understanding is that TCP uses 200ms because it's 2x the expected delayed ACK timer? If so, given we're including maxackdelay in the PTO value, shouldn't that be sufficient(assuming we keep a 2x or larger multiplier on PTO)?
The TCP minRTO is 1 sec per RFC 6298, but Linux uses 200 ms if I'm not mistaken. I could be wrong, but don't think it was based on the delayed ack timer, since the delayed ack timer value is typically 200ms. (I'll confirm this.) My high order point is that we probably do not want QUIC to declare persistent congestion (and collapse cwnd) sooner than TCP currently does in the network. Using only the PTO values might cause us to do that, and it might cause unnecessary throughput degradation. IMO we don't lose anything by declaring persistent congestion similar to TCP. PTOs still fire the way they always do, and recovery continues the way it always does. Other machinery (such as when to call a connection or path dead for instance) can continue to use PTOs, but we just declare persistent congestion based on what we know TCP behavior to be in the wild.
Well the problem with those fixed constants is that they don't work in DC for example. This has been a big pain point with TCP where we apply different minrto configs for DC and Internet. I actually liked the current QUIC definition of persistent congestion because it was dependent on actual path RTT than on some fixed value. NAME the value is not 3PTO because there is exponential doubling? Even if maxackDelay is advertised as 0 by receiver then QUIC is effectively declaring persistent congestion after Srtt + 2Srtt + 4Srtt? So 5 Srtt and that seems long enough?
Discussed in London; best to bring this up at a full IETF meeting.
Montreal: close with no action
Currently the recovery draft 19 requires the sender to either reducing the cwnd to initial cwnd OR pace the send after idle. Idle is defined as "it ceases to send data and has no bytes in flight" . This will mean that just a very small delay in the sending application thread will cause QUIC to reset its cwnd when not pacing. IMO we need to define an interval for this which could be a function of kGranularity or as PTO. FWIW in TCP the idle period for cwnd restart is defined as one retransmission timeout (reference URL Section 4.1).
However, this still has the problem that if I don't pace and I have a window of 100 packets, I can burst 100 packets into the network. This seems suboptimal from both the sender and the network's perspective. Now that I think about it, I think the concept of limiting the size of a burst is the key here, not so much idleness, and I think defining idle more accurately is both not necessary and likely to be difficult. Ideally you would send no more than max(10, BW * kTimerGranularity) packets into the network at once.
NAME pacing or limiting burst is the right idea. TCP RFC limits the burst to IW but it says do that after an idle period of an RTO. Presumably the authors and community considered that period to be small enough that network capacity / conditions hadn't changed drastically. Implementation wise how would we do this without a timer? If not pacing, when would one decide to cut cwnd back to restart cwnd (say IW)?
Your interpretation of RFC5661 looks correct and is different from what's in the QUIC recovery doc. But the implications are that it's ok to send an unlimited sized burst into the network as long as you only drained the pipe very recently. I think that recommendation is a poor one unless the window sizes are fairly small(in which case they'd be close to IW10 anyway). The sender is bursting packets into the network, which previously did not cause loss because they were ack-clocked, but one or more buffers on the network doesn't have enough space for the full CWND. It seems likely this could cause losses for other flows sharing the link as well. This could happen frequently in HTTP applications such as DASH video playback. NewCWV is already referenced for decaying CWND after idle. I could change the text to say nothing about reducing CWND to IW upon full idle and instead say you shouldn't burst more than IW into the network at once ever, and senders may use pacing, ACK-clocking, and/or timers to accomplish that? URL may also be relevant to this topic?
I guess the bigger question is what are we attempting to document in this draft. Is it whats in the TCP standard RFCs OR what most current TCP implementations do OR what the Linux kernel implementation does? By documenting NewReno instead of CUBIC we've chosen to not document the state-of-the-art and erred on the side of simplicity. In the spirit of that I think handling idle period and app limited states are optimizations for better network citizenship on cetrtain type of networks and not a hard requirement. For this particular issue I'd like to point out that the Windows TCP implementation has worked fine without handling the idle period (not capping cwnd burst and neither pacing). The idle cwnd restart knob in Windows is off by defualt and I believe Google TCP turns it off on Linux as well. Yes RFC 7661 is relevant and I think I cited it on the application limited case handling issue. I think there is a fine balance between managing burst size and taking advantage of TSO like optimizations. For example in WAN and DC bulk flows it might better to burst higher than IW with TSO for CPU efficiency than to pace and on the Internet pacing might be better. But I like your direction on this since it sidesteps what is meant by "idle". How about framing it as a suggestion and not a requirement. Something along the lines of: Bursts of more than IW into the network at once may cause losses. Implemementations should consider using pacing, ACK-clocking, and/or timers to minimize bursts. Paging NAME to this issue as well.
My goal is to document the best practices and reference existing RFCs whenever possible. And yes, my understanding is that everyone disables slow start after idle, so I'd like to come up with something I think is more likely to be deployed. When QUIC was started, Cubic wasn't an RFC, which is the other reason we documented Reno instead of Cubic, FYI. I'll write up a PR based on your suggestion and hopefully we can get some text we all think is acceptable.
RFC7661 is already cited in the section, and it was added there to defer to it so we didn't have to delve into details in this doc. I think what we're really discussing here is whether we need the MUST in the sentence "When sending data after becoming idle, a sender MUST reset its congestion window to the initial congestion window". I'm in agreement with what I'm sensing here -- that we don't need a requirement, but we can discuss the implications of various decisions here and current best practices. Higher-level note, yeah, we always wanted to document current best practices, rather than what was in TCP or even linux TCP. I think most things came from linux TCP because that was what we were familiar with. FWIW, agreed that Cubic wasn't an RFC at the time, but it was close to becoming one, and we did discuss it. We decided that it was too complex and adds a ton more complexity to this document. We wanted to have something simple but good enough that implementations could have as a starting point, and then move on to more esoteric algorithms as deployments needed. (I'll disagree that Cubic is the best known CC at the moment -- unless you're talking about DC environments, it's not.)
Thanks for the discussion, I updated based on this, though I'm a bit concerned that SHOULD is too weak in regards to bursts into the network. Then again, if that's what Windows is doing, maybe we have to accept the reality of current deployments?
Given that I think that the general guidance for QUIC is "use pacing", I think a "MUST reset to IW if not paced" is actually the right thing to do to support that pacing gets implemented with QUIC from the start.
NAME pacing may be implemented and controlled outside of the transport. I have two new concerns about text around bursting packets on network: Many CDNs and edge deployments will tweak IW to be a higher value. In Windows TCP we allow config of up to 64 MSS. So if the network admin knows what they are doing and can swallow bursts then we shouldn't discourage the practice. UDP segmentation offload is reality now and for high throughput flows using hardware offload to reduce CPU will mean larger bursts trading off against pacing. We shouldn't preclude this either.
Discussed in Montreal; people to review the PR.
Discussed in Cupertino, NAME to review and determine whether that addresses (or partially addresses) this issue.
NAME NAME This issue was supposed to be but it was waiting for feedback. It's been long enough -- can we close this issue?
This looks good, but I think we need to move a lot of the rationale back into the normative text from the pseudocode. You can do that in this PR or as we

can be established by consecutive PTOs, but since the PTO timer is reset when a new ack-eliciting packet is sent, an explicit duration must be used to account for those cases where PTOs do not occur or  <ins> are substantially delayed.  This duration is computed as follows: </ins> For example, assume: smoothed_rtt = 1 rttvar = 0 max_ack_delay = 0  <ins> kPersistentCongestionThreshold = 3 </ins> If an eck-eliciting packet is sent at time = 0, the following scenario would illustrate persistent congestion: The first three packets are determined to be lost when the ACK of packet 4 is received at t=8.  The congestion period is calculated as the time between the oldest and newest lost packets: (3 - 0) = 3.  <ins> The duration for persistent congestion is equal to: (1 * kPersistentCongestionThreshold) = 3.  Because the threshold was </ins> reached and because none of the packets between the oldest and the newest packets are acknowledged, the network is considered to have experienced persistent congestion.


TRANSPORT is used. The quic_transport_parameters extension is carried in the ClientHello <del> and the EncryptedExtensions messages during the handshake. </del>  While the transport parameters are technically available prior to the completion of the handshake, they cannot be fully trusted until the

All of them have defaults.
If you mean "Is sending TP mandatory", I don't think so. An endpoint that doesn't receive a TP assumes the default value. An endpoint that uses a default value does not need to advertise it.
The issue could have been more precisely worded. The question was about the entire QUIC transport parameters extension. Individual parameters are still clearly optional. We should be very crisp about that. Up until -18, this was mandatory, but we don't need the version parameters any more.
One nice property of making the TLS transport parameters extensions mandatory is that it helps mitigate cross-protocol attacks: QUIC servers can abort the connection if it's missing and TLS over TCP servers can abort if the extension is present.
I think we can call the entire extension as an option. As to NAME point, I'm not sure if it helps in practice. TLS 1.3 stacks that run on top of TCP does not have (and it's doubtful if all of them would have) the capability to detect and raise an error when it sees the QUIC TP extension.
Our defense for cross-protocol attacks is ALPN. Then key separation, and probably other things. While layers of defense is generally a good idea, transport parameters don't need to be another secondary layer.
I agree with NAME and NAME points. This was more of a belt-and-suspenders approach. That said, I still think every single deployment will actually send transport parameters, therefore requiring the extension to be present sounds simpler to implement. But I don't feel too strongly about this.
The question arose when we realized that you can have a perfectly functional stack with the defaults. And it was non- trivial sending any. However, if the client doesn't send any, it prevents the server from sending any. That seems bad.
That argument is more compelling than mine.
Discussed in Prague; no pushback against making them mandatory.
NAME can you make that into a PR?
There already is a PR, it just wasn't tagged properly as closing this issue.

TRANSPORT is used. The quic_transport_parameters extension is carried in the ClientHello  <ins> and the EncryptedExtensions messages during the handshake.  Endpoints MUST send the quic_transport_parameters extension; endpoints that receive ClientHello or EncryptedExtensions messages without the quic_transport_parameters extension MUST terminate the TLS handshake with a fatal missing_extension alert (an error of 0x16d). </ins> While the transport parameters are technically available prior to the completion of the handshake, they cannot be fully trusted until the


The layout of a Version Negotiation packet is: The value in the Unused field is selected randomly by the server.  The Version field of a Version Negotiation packet MUST be set to 0x00000000.

This must be ignored, and should be set to 1.
I've had conversations with a number of people who are interested in identifying QUIC using the QUIC bit. Only one packet does not have the QUIC bit always set to 1 currently, and that's version negotiation. URL Given we have no use for the extra bits in VN, I'm inclined to specify implementations must set it to 1. If we think we'll one day want to claw back use of the QUIC bit and allow it to be 0, then this is a poor plan, but I don't think that will end up being practical on the public internet. A related question is whether we want to put the QUIC bit in the invariants. It's clearly not technically necessary, but it would recognize the ossification I anticipate.
I have a different proposal: allow those that don't depend on this bit to toggle it. That is, if you are certain that you don't need this bit set to 1, set it to 0 occasionally.
That could work. It comes down to whether the WG thinks the QUIC bit is a bug or feature, as well as whether enough people will toggle it that it won't be ossified anyway.
NAME I like how you frame the issue. I think that the QUIC bit should either be defined in the invariants draft, or be greased to avoid it getting ossified. And my weak preference goes to greasing the bit, so that we could have maximum flexibility in the future versions of QUIC.
I'm a bit concerned that relying on greasing is wishful thinking, here and elsewhere. Mostly this has been bypassed with header protection. All it takes is one dominant implementation to be lazy about it. Think older IE browsers vs. standards.
The reason the second bit was fixed to 1 in QUICv1 was to allow multiplexing with other protocols. Greasing it would defeat that and would cause problems with some Google servers that use it for demultiplexing. Given that these demultiplexing concerns are unlikely to go away, it does make sense to add it to the invariants.
I suspect VN packets will be pretty rare in the internet before we roll out v2, so we are in real danger of 50% of these packets failing because they have QUIC set to zero. The most straightforward solution is to just codify the ossification that will happen anyway. NAME is this multiplexing mostly an issue for Initial Packets? Because the way to free up the QUIC bit could be to grease it in the Short Header. No one will implement something that drops 50% of 1RTT packets.
NAME good point - most of the use-cases we have today are servers doing the demultiplexing, and VN is currently only server-to-client. I'm less worried about VN and more about the Internet ossifying "QUIC == the second bit is 1" without it being in the invariants. Greasing the second bit in client-to-server short headers would make it harder to demultiplex on the server without keeping track of all the issued connection IDs, and I don't think we want to make the server demultiplexer stateful.
I agree that the long header packets sent by the client need to have the QUIC bit set. I also agree that there's less need to have the bit set in the long header packets sent by the server, though I think it's possible to argue that we should not have different requirements based on the direction of the packets, because QUIC does not have a distinguish a client address tuple and a server address tuple (i.e. an endpoint can use a one address tuple for accepting connections and also for initiating connections as a client). For short header packets, I think we agree that the bit is not required. Then the question is, assuming that we decide to kill the bit, should we grease it, or rather, change it to a reserve bit? I suggest the latter because changing the definition to "reserved" and covering the bit using header protection is the strongest way of preserving the freedom to use the bit in future versions of QUIC. It can also be considered as a simplification, because there would be not need to "randomize" the bit for greasing; endpoints can rely on the header protection to flip the bits. FWIW, current definition of the short header first byte is: . This would change to if we are going to change the QUIC bit to a reserved bit.
To me, it's clear that we should either (1) add it to the invariants or (2) bring it under header protection for server-generated packets only. (2) is somewhat inelegant, but we get to keep using the bits. Are there any privacy implications to there being a "client bit"?
I would oppose to such a design, because that disables a reverse proxy from using a single port for accepting connections and also for initiating connections, assuming that either the proxy or a middlebox in front of the proxy relies on the QUIC bit to route the packets. Greasing would be fine, because it's not something to be required for every endpoint to implement. The bit would be a good indicator to police P2P traffic for ISPs...
I’m in favor of always setting the bit, and if not, symmetric header protection. There are significant symmetric use cases, including NAT-punching p2p. Making it 1 permanently lets QUIC co-exist, eg on a single client 443 port, and it avoids a lot of potential problems over a bit that is not needed.
IIRC, the bit was a result of us wanting to demux with various other protocols, and as such, the entire purpose of the bit is to allow identification of QUIC. This may not be a feature we like, but it is a feature. I don't think we should be talking about the bit, but about co-existence with the other protocols. We discussed this, NAME made a presentation about it (in Montreal, I think), and we agreed to co-exist. While it can be seen as a natural consequence of setting the QUIC bit, I think the question we want to ask is whether we ought to make this co-existence an invariant. If so, what is the minimal set of packets that need to be demux-able with these other protocols. (We've been over this before, but I don't think we discussed whether it was adequate to demux only for the long header.)
FWIW most of the discussion happen in New York. See URL
NAME thanks for jogging my memory. It seems that we didn't agree on something specific there.
Discussed in Prague; proposal to encourage senders in v1 to set to ; retain requirement in invariants to ignore.
I realize there exist a need to clarify the text regarding ECN on the relation between IP packets and QUIC packets. As QUIC tracks QUIC packets, and not IP packets, this relation when coaleing occurs needs to be clarified. What I think is needed are: In Section 6.8: Using Explicit Congestion Notification should be updated to be explicit about when it talks about IP packet and when about QUIC packets. State explicitly that for coalesed QUIC packets, the IP packet's ECN field value is reported for each QUIC packet.
Sounds good. I'm sure that we'd appreciate getting a PR for this.
So how do we deal with the different packet number spaces here. I see two options: Either say that we totally ignore Initial packets in the perspective of ECN handling. Which basically delays the ECN verification until one sees ACKs for the 0- or 1-RTT packets. Or one requires one set of counters per packet number space. What are peoples preference here?
I'm thinking that you count separately, in the interests of keeping ACK accounting separate (and consistent). But no special handling, or disabling ECN for different phases. You can still use Initial/Handshake to verify that the path supports ECN.
Agreed, each PN space has its own set of counters, and an ACK frame sent in a PN space conveys the counters of that PN space.

The layout of a Version Negotiation packet is: The value in the Unused field is selected randomly by the server. <ins> Clients MUST ignore the value of this field.  Servers SHOULD set the most significant bit of this field (0x40) to 1 so that Version Negotiation packets appear to have the Fixed Bit field. </ins> The Version field of a Version Negotiation packet MUST be set to 0x00000000.


Additionally, each code of the format "0x1f * N + 0x21" for integer values of N (that is, "0x21", "0x40", ..., through <del> "0x3FFFFFFFFFFFFFFE") MUST NOT be assigned by IANA. </del>  10.4. Additionally, each code of the format "0x1f * N + 0x21" for integer values of N (that is, "0x21", "0x40", ..., through <del> "0x3FFFFFFFFFFFFFFE") MUST NOT be assigned by IANA. </del>  10.5. Additionally, each code of the format "0x1f * N + 0x21" for integer values of N (that is, "0x21", "0x40", ..., through <del> "0x3FFFFFFFFFFFFFFE") MUST NOT be assigned by IANA. </del>  11.  References

This was fun, it turns out there were actually 3 pair of instances where the were added. I wonder if this was super meta level protest against the endianess-swap from back in the day...
Draft 19 has rendering issue for the reserved frame type in section 10.4

Additionally, each code of the format "0x1f * N + 0x21" for integer values of N (that is, "0x21", "0x40", ..., through  <ins> "0x3FFFFFFFFFFFFFFE") MUST NOT be assigned by IANA. </ins> 10.4. Additionally, each code of the format "0x1f * N + 0x21" for integer values of N (that is, "0x21", "0x40", ..., through  <ins> "0x3FFFFFFFFFFFFFFE") MUST NOT be assigned by IANA. </ins> 10.5. Additionally, each code of the format "0x1f * N + 0x21" for integer values of N (that is, "0x21", "0x40", ..., through  <ins> "0x3FFFFFFFFFFFFFFE") MUST NOT be assigned by IANA. </ins> 11.  References


A stateless reset is provided as an option of last resort for an endpoint that does not have access to the state of a connection.  A crash or outage might result in peers continuing to send data to an <del> endpoint that is unable to properly continue the connection.  A stateless reset is not appropriate for signaling error conditions. </del>  An endpoint that wishes to communicate a fatal connection error MUST use a CONNECTION_CLOSE frame if it has sufficient state to do so. packet either cannot be associated with a connection, cannot be decrypted, or is marked as a duplicate packet.  The endpoint MUST then compare the last 16 bytes of the packet with all Stateless Reset <del> Tokens that are associated with connection IDs that are currently in use.  This includes Stateless Reset Tokens from NEW_CONNECTION_ID frames and the server's transport parameters.  An endpoint MUST NOT check for any Stateless Reset Tokens associated with connection IDs it has not used or for connection IDs that have been retired. </del>  If the last 16 bytes of the packet values are identical to a Stateless Reset Token, the endpoint MUST enter the draining period A stateless reset token is used in verifying a stateless reset, see stateless-reset.  This parameter is a sequence of 16 bytes. <del> This transport parameter is only sent by a server. </del>  The maximum packet size parameter is an integer value that limits the size of packets that the endpoint is willing to receive.  This

I know that we made this a design issue, and these do change 2119 language, but those normative requirements were implied anyway. That is, the text used cannot vs. MUST NOT.
I was also wondering about the rate limit in SR - I too thought it was there. But now, after thinking a bit, the rate limit was probably removed because it is a DoS vector. It can force state to maintained where it is not wanted.
This is a weird statement to make. It is absolutely the place of the spec to do that! If what you mean is that this is entirely optional, then I'm happy to make it a MAY. My concern is not so much what level of force the spec uses, just that the level of force is clear.
Before I merge this, NAME NAME might want to check that I'm not making technical changes. (There is new 2119 language, but my view is that this is converting from implied to express, but I want to give a chance to run a consensus call if you think that's appropriate.)
Those sorts of conversions have surprised people before, and is design anyway, so let's do run that consensus call. If it's ready to go, add and we'll run it up the virtual flagpole.
NAME take a look at what I just proposed.
While there are some prohibitions, and an explicit MAY for responding to Long Headers, there is no normative statement for the usual use case for Stateless Reset. I believe it's a SHOULD, but maybe someone thinks it's a MUST. Relatedly, in 10.4.1, MUST we attempt to detect Stateless Reset, or MAY endpoints just drop after decrypt failure? At the end of 10.4.2, MUST we not reuse connection IDs if there's a static key? (I think the answer is yes)
Thanks, NAME this LGTM.Thank you, this addresses the original issue. One nit below.

A stateless reset is provided as an option of last resort for an endpoint that does not have access to the state of a connection.  A crash or outage might result in peers continuing to send data to an  <ins> endpoint that is unable to properly continue the connection.  An endpoint MAY send a stateless reset in response to receiving a packet that it cannot associate with an active connection. A stateless reset is not appropriate for signaling error conditions. </ins> An endpoint that wishes to communicate a fatal connection error MUST use a CONNECTION_CLOSE frame if it has sufficient state to do so. packet either cannot be associated with a connection, cannot be decrypted, or is marked as a duplicate packet.  The endpoint MUST then compare the last 16 bytes of the packet with all Stateless Reset  <ins> Tokens that are associated with connection IDs that the endpoint recently used to send packets from the IP address and port on which the datagram is received.  This includes Stateless Reset Tokens from NEW_CONNECTION_ID frames and the server's transport parameters.  An endpoint MUST NOT check for any Stateless Reset Tokens associated with connection IDs it has not used or for connection IDs that have been retired. </ins> If the last 16 bytes of the packet values are identical to a Stateless Reset Token, the endpoint MUST enter the draining period A stateless reset token is used in verifying a stateless reset, see stateless-reset.  This parameter is a sequence of 16 bytes.  <ins> This transport parameter MUST NOT be sent by a client, but MAY be sent by a server.  A server that does not send this transport parameter cannot use stateless reset (stateless-reset) for the connection ID negotiated during the handshake. </ins> The maximum packet size parameter is an integer value that limits the size of packets that the endpoint is willing to receive.  This


6.3. <del> For a server to use a new version in the future, clients must </del>  correctly handle unsupported versions.  To help ensure this, a server <del> SHOULD include a reserved version (see versions) while generating a </del>  Version Negotiation packet. The design of version negotiation permits a server to avoid maintaining state for packets that it rejects in this fashion. <del> A client MAY send a packet using a reserved version number.  This can be used to solicit a list of supported versions from a server. </del>  7.

Section 6.3 says And then Section 15 defines two kinds of "reserved" versions, which makes the paragraph above ambiguous. ... Is the intent in Section 6.3 that a server SHOULD include a version that is reserved for forcing version negotiation (i.e., matches 0x?a?a?a?a), or SHOULD a server include a version reserved for future IETF consensus documents (i.e., matching 0x0000????). Or both?
In my opinion having an implementation that advertises possible future standards versions without actually supporting them seems wrong, what happens if the other side actually supports this version? This seems like it could deadlock or look like a bidding down attack in case the client moves to a lower version to resolve the deadlock. On the other hand, announcing standards versions helps to not ossify around the 0x?a?a?a?a pattern but to me announcing standards versions does not seem like a solution to this.
I think that some of the discussion recently has pushed us away from reserving versions for IETF use. Fixing that would make this a design issue, of course. I get the editorial point though. Doubling up on the use of "reserved" is unfortunate.
looks fine

6.3.  <ins> For a server to use a new version in the future, clients need to </ins> correctly handle unsupported versions.  To help ensure this, a server  <ins> SHOULD include a version that is reserved for forcing version negotiation (0x?a?a?a?a as defined in versions) when generating a </ins> Version Negotiation packet. The design of version negotiation permits a server to avoid maintaining state for packets that it rejects in this fashion.  <ins> A client MAY send a packet using a version that is reserved for forcing version negotiation.  This can be used to solicit a list of supported versions from a server. </ins> 7.


is received (that is, from the "Data Recvd" state).  Similarly, it is possible for remaining stream data to arrive after receiving a RESET_STREAM frame (the "Reset Recvd" state).  An implementation is <del> free to manage this situation as it chooses.  Sending RESET_STREAM means that an endpoint cannot guarantee delivery of stream data; however there is no requirement that stream data not be delivered if a RESET_STREAM is received.  An implementation MAY interrupt delivery of stream data, discard any data that was not consumed, and signal the receipt of the RESET_STREAM immediately.  Alternatively, the RESET_STREAM signal might be suppressed or withheld if stream data is completely received and is buffered to be read by the application. In the latter case, the receiving part of the stream transitions from "Reset Recvd" to "Data Recvd". </del>  Once the application has been delivered the signal indicating that the stream was reset, the receiving part of the stream transitions to

For NAME
This is talking about the case of being in "Data Recvd" and therefore ignoring RESET_STREAM. Should the last sentence instead read: "In the latter case, the receiving part of the stream remains in "Data Recvd" ?
That sounds right.

is received (that is, from the "Data Recvd" state).  Similarly, it is possible for remaining stream data to arrive after receiving a RESET_STREAM frame (the "Reset Recvd" state).  An implementation is  <ins> free to manage this situation as it chooses. Sending RESET_STREAM means that an endpoint cannot guarantee delivery of stream data; however there is no requirement that stream data not be delivered if a RESET_STREAM is received.  An implementation MAY interrupt delivery of stream data, discard any data that was not consumed, and signal the receipt of the RESET_STREAM.  A RESET_STREAM signal might be suppressed or withheld if stream data is completely received and is buffered to be read by the application.  If the RESET_STREAM is suppressed, the receiving part of the stream remains in "Data Recvd". </ins> Once the application has been delivered the signal indicating that the stream was reset, the receiving part of the stream transitions to


unpredictable values.  The last 16 bytes of the datagram contain a Stateless Reset Token. <del> An endpoint that receives a packet where removal of packet protection fails MUST check the last 16 bytes of that packet.  If the last 16 bytes of the packet are identical to a stateless reset token corresponding to a packet that was recently sent, the endpoint MUST NOT send any further packets; all state for the connection can then be discarded. </del> To entities other than its intended recipient, a stateless reset will be appear to be a packet with a short header.  For the packet to appear as valid, the Unpredictable Bits field needs to include at functions defined in QUIC-TLS, packets that are smaller than 21 bytes are never valid.  An endpoint MAY send a stateless reset in response to a packet with a <del> long header.  This would not be effective if the stateless reset token was not yet available to a peer.  In this QUIC version, packets with a long header are only used during connection establishment. Because the stateless reset token is not available until connection establishment is complete or near completion, ignoring an unknown packet with a long header might be more effective. </del>  An endpoint cannot determine the Source Connection ID from a packet with a short header, therefore it cannot set the Destination 10.4.1. <del> An endpoint detects a potential stateless reset when a incoming packet with a short header either cannot be associated with a connection, cannot be decrypted, or is marked as a duplicate packet. The endpoint then compares the last 16 bytes of the packet with the Stateless Reset Token provided by its peer, either in a NEW_CONNECTION_ID frame or the server's transport parameters.  If these values are identical, the endpoint MUST enter the draining period and not send any further packets on this connection.  If the </del>  comparison fails, the packet can be discarded. 10.4.2. Revealing the Stateless Reset Token allows any entity to terminate the connection, so a value can only be used once.  This method for choosing the Stateless Reset Token means that the combination of <del> connection ID and static key cannot occur for another connection.  A denial of service attack is possible if the same connection ID is </del>  used by instances that share a static key, or if an attacker can cause a packet to be routed to an instance that has no state but the same static key (see reset-oracle).  A connection ID from a <del> connection that is reset by revealing the Stateless Reset Token cannot be reused for new connections at nodes that share a static key. </del>  Note that Stateless Reset packets do not have any cryptographic protection.

We don't want to prohibit the use of long headers in other versions - in case they want to share the stateless reset design, but there is no sense in sending long headers.
While I agree with the issue, the PR says that other versions might want to reset with a long header and therefore the receiver must inspect long headers anyway. I can see this, but given that, I think it is pointless to require the current version to only send resets for short headers. A general advice ought to suffice since the MUST is not enforceable anyway.
The requirement exists to protect those packets from detection.
Please don't close issues until AFTER consensus is declared (not just called).
I suspect Martin was confused because this issue was tagged has-proposal 15 days ago.
Right. The trigger is , not .
But it's understandable Martin would have assumed this was one of the issues that was sent out in the batch of issues a week and a half ago, given this has been tagged with has-proposal for 15 days.

unpredictable values.  The last 16 bytes of the datagram contain a Stateless Reset Token.  To entities other than its intended recipient, a stateless reset will be appear to be a packet with a short header.  For the packet to appear as valid, the Unpredictable Bits field needs to include at functions defined in QUIC-TLS, packets that are smaller than 21 bytes are never valid. <ins> Endpoints MUST send stateless reset packets formatted as a packet with a short header.  However, endpoints MUST treat any packet ending in a valid stateless reset token as a stateless reset, as other QUIC versions might allow the use of a long header. </ins> An endpoint MAY send a stateless reset in response to a packet with a  <ins> long header.  Sending a stateless reset is not effective prior to the stateless reset token being available to a peer.  In this QUIC version, packets with a long header are only used during connection establishment.  Because the stateless reset token is not available until connection establishment is complete or near completion, ignoring an unknown packet with a long header might be as effective than sending a stateless reset. </ins> An endpoint cannot determine the Source Connection ID from a packet with a short header, therefore it cannot set the Destination 10.4.1.  <ins> An endpoint detects a potential stateless reset when an incoming packet either cannot be associated with a connection, cannot be decrypted, or is marked as a duplicate packet.  The endpoint MUST then compare the last 16 bytes of the packet with all Stateless Reset Tokens that are associated with connection IDs that are currently in use.  This includes Stateless Reset Tokens from NEW_CONNECTION_ID frames and the server's transport parameters.  An endpoint MUST NOT check for any Stateless Reset Tokens associated with connection IDs it has not used or for connection IDs that have been retired. If the last 16 bytes of the packet values are identical to a Stateless Reset Token, the endpoint MUST enter the draining period and not send any further packets on this connection.  If the </ins> comparison fails, the packet can be discarded. 10.4.2. Revealing the Stateless Reset Token allows any entity to terminate the connection, so a value can only be used once.  This method for choosing the Stateless Reset Token means that the combination of  <ins> connection ID and static key MUST NOT be used for another connection. A denial of service attack is possible if the same connection ID is </ins> used by instances that share a static key, or if an attacker can cause a packet to be routed to an instance that has no state but the same static key (see reset-oracle).  A connection ID from a  <ins> connection that is reset by revealing the Stateless Reset Token MUST NOT be reused for new connections at nodes that share a static key. </ins> Note that Stateless Reset packets do not have any cryptographic protection.


7.9. <del> A sender becomes idle if it ceases to send data and has no bytes in flight.  A sender's congestion window MUST NOT increase while it is idle. </del>  <del> When sending data after becoming idle, a sender MUST reset its congestion window to the initial congestion window (see Section 4.1 of RFC5681), unless it paces the sending of packets.  A sender MAY retain its congestion window if it paces the sending of any packets in excess of the initial congestion window. A sender MAY implement alternate mechanisms to update its congestion window after idle periods, such as those proposed for TCP in RFC7661. 7.10. The congestion window should not be increased in slow start or congestion avoidance when it is not fully utilized.  The congestion window could be under-utilized due to insufficient application data or flow control credit. </del>  A sender that paces packets (see pacing) might delay sending packets and not fully utilize the congestion window due to this delay.  A sender should not consider itself application limited if it would have fully utilized the congestion window without pacing delay.  8. 8.1.

Combines the section on sending after idle and application limited because they share a lot in common. Praveen, I think this is better, but tell me if you'd like to add/change something else? and
I'll take this in, and open an issue for defining "under-utilized cwnd"
changed the discussion in recovery from idle behavior and app limited periods to periods of under-utilized cwnd. This is great, but a precise definition of when the cwnd goes under-utilized is still missing.
Defining "under-utilised cwnd", seems conceptually the right thing, but we struggled with that term and the term "fully utilized the congestion window" when writing RFC7661, because ACKs can arrive in bursts and apps can do their work in bursts: Does that mean it's more than some threshold for some period time?
NAME : That is a question. Maybe we can get away from that by saying that a sender only increases its cwnd if it knows that it was fully utilized since the last cwnd change? Meaning that at some point since the last cwnd change, inflight >= cwnd happened at least once. I have a separate question, one that was being discussed in , and I think is still left open. If we need the entire cwnd to be used before increasing it, we lose out on 1 RTT of growth during slow start. Both Windows and Linux TCP compensate for this by doing something like increase it when > 1/2 cwnd is used... we need to write something here that's at least as performant, and not just "when inflight >= cwnd".
NAME thoughts?
This seems to be digging in the correct place. I think we would agree the aim of some form of cwnd-validation is to stop cwnd growth when there is no growth of inflight data by the upper layer. And I think we'd agree tussle comes when the CC is trying to expand the cwnd - because the upper layer has more data to send, but that doesn't mean cwnd was necessarily fully utilised last RTT. When would in-flight > cwnd ?
NAME Good point about the in_flight > 1/2 CWND check in Linux. In BBR, that only applies to the first doubling of IW, and after that it's fully bandwidth based, FWIW. We could add a similar constraint that this rule can be applied when the window was not fully used at the time of sending, but was at the time of acknowledgement as long as the window is < 2*IW? One clear goal of mine is I don't want the handshake alone increasing the congestion window(assuming IW10 and a moderately sized handshake).
Putting this discussion into a concrete algorithm: -Keep an in-flight high watermark, InFlightHigh, that's reset on each CWND change. -When considering growing the CWND, only let it go as large as InFlightHigh2. -Justification for the "2": it allows base-2 exponential growth, which is the fastest growth we ever want. EDIT: expire on CWND change might not work. I think InFlightHigh expiry is the hard part here.
Possible expiration events: persistent congestion, ssthresh changes, app idle.
Discussed in London, we'd like to avoid specifying this in pseudocode and the exact math, as it may be implementation dependent, but instead focus on improving the existing text.
Currently the recovery draft 19 requires the sender to either reducing the cwnd to initial cwnd OR pace the send after idle. Idle is defined as "it ceases to send data and has no bytes in flight" . This will mean that just a very small delay in the sending application thread will cause QUIC to reset its cwnd when not pacing. IMO we need to define an interval for this which could be a function of kGranularity or as PTO. FWIW in TCP the idle period for cwnd restart is defined as one retransmission timeout (reference URL Section 4.1).
However, this still has the problem that if I don't pace and I have a window of 100 packets, I can burst 100 packets into the network. This seems suboptimal from both the sender and the network's perspective. Now that I think about it, I think the concept of limiting the size of a burst is the key here, not so much idleness, and I think defining idle more accurately is both not necessary and likely to be difficult. Ideally you would send no more than max(10, BW * kTimerGranularity) packets into the network at once.
NAME pacing or limiting burst is the right idea. TCP RFC limits the burst to IW but it says do that after an idle period of an RTO. Presumably the authors and community considered that period to be small enough that network capacity / conditions hadn't changed drastically. Implementation wise how would we do this without a timer? If not pacing, when would one decide to cut cwnd back to restart cwnd (say IW)?
Your interpretation of RFC5661 looks correct and is different from what's in the QUIC recovery doc. But the implications are that it's ok to send an unlimited sized burst into the network as long as you only drained the pipe very recently. I think that recommendation is a poor one unless the window sizes are fairly small(in which case they'd be close to IW10 anyway). The sender is bursting packets into the network, which previously did not cause loss because they were ack-clocked, but one or more buffers on the network doesn't have enough space for the full CWND. It seems likely this could cause losses for other flows sharing the link as well. This could happen frequently in HTTP applications such as DASH video playback. NewCWV is already referenced for decaying CWND after idle. I could change the text to say nothing about reducing CWND to IW upon full idle and instead say you shouldn't burst more than IW into the network at once ever, and senders may use pacing, ACK-clocking, and/or timers to accomplish that? URL may also be relevant to this topic?
I guess the bigger question is what are we attempting to document in this draft. Is it whats in the TCP standard RFCs OR what most current TCP implementations do OR what the Linux kernel implementation does? By documenting NewReno instead of CUBIC we've chosen to not document the state-of-the-art and erred on the side of simplicity. In the spirit of that I think handling idle period and app limited states are optimizations for better network citizenship on cetrtain type of networks and not a hard requirement. For this particular issue I'd like to point out that the Windows TCP implementation has worked fine without handling the idle period (not capping cwnd burst and neither pacing). The idle cwnd restart knob in Windows is off by defualt and I believe Google TCP turns it off on Linux as well. Yes RFC 7661 is relevant and I think I cited it on the application limited case handling issue. I think there is a fine balance between managing burst size and taking advantage of TSO like optimizations. For example in WAN and DC bulk flows it might better to burst higher than IW with TSO for CPU efficiency than to pace and on the Internet pacing might be better. But I like your direction on this since it sidesteps what is meant by "idle". How about framing it as a suggestion and not a requirement. Something along the lines of: Bursts of more than IW into the network at once may cause losses. Implemementations should consider using pacing, ACK-clocking, and/or timers to minimize bursts. Paging NAME to this issue as well.
My goal is to document the best practices and reference existing RFCs whenever possible. And yes, my understanding is that everyone disables slow start after idle, so I'd like to come up with something I think is more likely to be deployed. When QUIC was started, Cubic wasn't an RFC, which is the other reason we documented Reno instead of Cubic, FYI. I'll write up a PR based on your suggestion and hopefully we can get some text we all think is acceptable.
RFC7661 is already cited in the section, and it was added there to defer to it so we didn't have to delve into details in this doc. I think what we're really discussing here is whether we need the MUST in the sentence "When sending data after becoming idle, a sender MUST reset its congestion window to the initial congestion window". I'm in agreement with what I'm sensing here -- that we don't need a requirement, but we can discuss the implications of various decisions here and current best practices. Higher-level note, yeah, we always wanted to document current best practices, rather than what was in TCP or even linux TCP. I think most things came from linux TCP because that was what we were familiar with. FWIW, agreed that Cubic wasn't an RFC at the time, but it was close to becoming one, and we did discuss it. We decided that it was too complex and adds a ton more complexity to this document. We wanted to have something simple but good enough that implementations could have as a starting point, and then move on to more esoteric algorithms as deployments needed. (I'll disagree that Cubic is the best known CC at the moment -- unless you're talking about DC environments, it's not.)
Thanks for the discussion, I updated based on this, though I'm a bit concerned that SHOULD is too weak in regards to bursts into the network. Then again, if that's what Windows is doing, maybe we have to accept the reality of current deployments?
Given that I think that the general guidance for QUIC is "use pacing", I think a "MUST reset to IW if not paced" is actually the right thing to do to support that pacing gets implemented with QUIC from the start.
NAME pacing may be implemented and controlled outside of the transport. I have two new concerns about text around bursting packets on network: Many CDNs and edge deployments will tweak IW to be a higher value. In Windows TCP we allow config of up to 64 MSS. So if the network admin knows what they are doing and can swallow bursts then we shouldn't discourage the practice. UDP segmentation offload is reality now and for high throughput flows using hardware offload to reduce CPU will mean larger bursts trading off against pacing. We shouldn't preclude this either.
Discussed in Montreal; people to review the PR.
Discussed in Cupertino, NAME to review and determine whether that addresses (or partially addresses) this issue.
NAME NAME This issue was supposed to be but it was waiting for feedback. It's been long enough -- can we close this issue?
The text in recovery draft 19 says "The congestion window should not be increased in slow start or congestion avoidance when it is not fully utilized". This is a deviation from how congestion control works in deployed TCP implementation. We need a crisp definition of what is meant by "full utilization". I also have concern that this will make QUICs performance for bulk transfers poorer than TCP because application I/O patterns and scheduling of the sending thread will affect cwnd growth. After making the definition crisp and consensus that we need to include such a provision, the pseudo code will need to be updated to handle this as well.
NAME You're right, this needs to be made crisp and we need to add pseudo-code to handle this. Do you have a strawman proposal? How does Windows TCP do currently to handle growth during app limited periods? The naive approach doesn't work, since it prevents growth in the second RTT during slow start. I don't remember anymore what Linux TCP does, but I can look into that.
FWIW, my read of the Linux code tells me that at ack receipt time, the sender considers itself app-limited if in slow-start and cwnd > 2 x inflight, or if in cong-avoid and cwnd > inflight.
Jana, your reading matches my memory, but I thought there was an extra "fq-pacing quantum" added to inflight in the second case?
So there is no standard mechanism defined for this in any of the TCP RFCs especially the ones describing modern congestion control like CUBIC and DCTCP. Instead of providing a particular algorithm I propose we leave the handling of this to the implementations? Maybe we can refer implementers to URL for possible ideas.
I’m not sure, but I guess RFC4960 & SCTP implementation may be useful in this case.
I found this in RFC4960(URL): "When cwnd is less than or equal to ssthresh, an SCTP endpoint MUST use the slow-start algorithm to increase cwnd only if the current congestion window is being fully utilized, an incoming SACK advances the Cumulative TSN Ack Point, and the data sender is not in Fast Recovery. Only when these three conditions are met can the cwnd be increased; otherwise, the cwnd MUST not be increased." The current QUIC recovery text tries to clarify what fully utilized means, particularly in the presence of pacing.
I'm quite familiar with RFC 4960, and it does not have anything that is particularly helpful here. NAME I didn't see an fq-quantum in the code, but I could also be missing something.
Thanks for this -- I think it's the right direction. Some comments to try and herd this text some more.

7.9.  <ins> A congestion window that is under-utilized SHOULD NOT be increased in either slow start or congestion avoidance.  This can happen due to insufficient application data or flow control credit. </ins>  <ins> A sender MAY use the pipeACK method described in section 4.3 of RFC7661 to determine if the congestion window is sufficiently utilized. </ins> A sender that paces packets (see pacing) might delay sending packets and not fully utilize the congestion window due to this delay.  A sender should not consider itself application limited if it would have fully utilized the congestion window without pacing delay. <ins> Bursting more than an intial window's worth of data into the network might cause short-term congestion and losses.  Implemementations SHOULD either use pacing or reduce their congestion window to limit such bursts. A sender MAY implement alternate mechanisms to update its congestion window after periods of under-utilization, such as those proposed for TCP in RFC7661. </ins> 8. 8.1.


streams (push-streams).  Other stream types can be defined by extensions to HTTP/3; see extensions for more details. <del> Both clients and servers SHOULD send a value of three or greater for the QUIC transport parameter "initial_max_uni_streams". </del>  If the stream header indicates a stream type which is not supported by the recipient, the remainder of the stream cannot be consumed as

This goes someway to fixing and . Setting low values for unidirectional stream transport parameters means that HTTP/3 connections get stuck early. Although client and server can dance around, sending BLOCKED frames and increasing limits, we should clearly highlight the problem and suggest some sensible, bare minimum, values.
I'm happy to take it out. However, I wouldn't call it misbehaviour to exercise protocol features. The unidirectional stream extension capabilities seem dead on arrival if there is no way for either peer to express their intent.
If the transport parameter is not set, HTTP 3 will just stall. That's because the HTTP3 implementations are supposed to send the "Settings" before sending any request or response, but they can only do that if there are flow control credits for Uni streams. Of course, the HTTP3 spec says that "Both clients and servers SHOULD send a value of three or greater for the QUIC transport parameter initialmaxunistreams." But it fails to mention that there should be some sensible value for . The bug was found during interop testing between and . It really is a spec bug. Section 3.1 of the HTTP3 spec has text stating that "it is recommended that be no smaller than 100, so as to not unnecessarily limit parallelism." I think that we need similar text in section 3.2.
I agree that we should probably recommend sending a non-zero value, but the connection is not supposed to stall in either case. If a peer sends 0 in the transport parameters, it will have to grant flow control credit via MAX_DATA frames.
NAME the problem is that if the number of credit is zero, the peer cannot even open the stream. It is not supposed to send a null length stream frame. And if the stream is not even open, the generic transport has no way to know that it should send a MAXSTREAMDATA frame on stream 2 or 3. An H3 conscious transport might know that, but a generic one not so much -- you don't need credits on stream 2 or 3 for HTTP 1.x or for DNS over HTTP. It seems much preferable to just mandate non zero values in the transport parameters.
NAME Why can't you open the stream? There's nothing preventing you from opening a stream that you don't have flow control credit for. You could send a zero-length STREAM frame, or even better, you could send a STREAM_BLOCKED frame.
NAME I suppose you could send a zero length STREAM frame, although I am pretty sure one of spec says that nodes should not send zero-length stream frames because DOS attacks. STREAMBLOCKED has a different meaning -- it does not actually open the stream. I suppose that STREAMDATA_BLOCKED would work. In any case, asking for a sensible initial limit seems much simpler.
STREAMDATABLOCKED is what I meant, sorry. Too many BLOCKED frames. STREAMBLOCKED doesn't even exist... I think this is a general issue with either picoquic or ats, and not H3 specific. No matter what the application protocol, a peer can send you 0-values for stream counts and for flow control limits, and the protocol should still work (although slower, since the MAX frames need to be exchanged). There should never be a deadlock (unless the peer is refusing to grant credits, even at a later point in time).
It is not a "dead" lock. If picoquic does not have credits, it will just wait. If the peer somehow wakes up and provide credit, transmission will resume. In my mind, the transmission of the various "BLOCKED" frames is just optional.
This is slightly related to and . IMO there is no easy fix because even if you grant some initial values, there is nothing forcing the endpoint to use the first available unidirectional streams for control streams. The temptation is to set these transport limits quite high in order to just avoid the issue but that allows the sending endpoint to overwhelming the receiver with a lot of data (negating the value of the parameters). An alternative is to set a conservative value for , say 8 bytes. This would allow the sender to open a uni stream with a type, the receiver can use this to determine if/how to grant further credits.
In Tokyo 2019, we discussed (Avoid creating QPACK streams when unnecessary) and came upon the conclusion: Until such a change is applied to the HTTP and QPACK specs, we rely on the guidance in section 3.2: the QUIC transport parameter "initialmaxunistreams". There is an implicit assumption in this present state that assumes these three streams will be used for the "critical streams" - Control stream, QPACK Encoder Stream and QPACK Decoder stream. The outcome of the Tokyo discussion didn't seem to me to identify the impact on initialmaxunistreams. HTTP/3 reserves code points for GREASING but doesn't provide much guidance on their usage. It says "MUST ignore" and CAN send reserved stream types. In contrast Transport is a little stronger and says that: Not sending GREASE negates its powers. GREASE streams are not the only consumer of unidirectional credit. It seems feasible that an endpoint might optimise for some use cases and decide push on 2-out-of-three streams it was given the credit for. This depends on receiving MAXPUSHID on the control stream, of course. The control stream unfortunately has dubious language: the connection. When is the beginning of beginning and when is the end of beginning? Mike somewhat answered this question in (Default settings in HTTP): Is the end the beginning once SETTINGS have been sent? PR was an attempt at scurry along the spec change wanted. Created in response to hitting an interop issue causes by the old requirement to "MUST create one QPACK Encoder and Decoder stream" conflicting with the "SHOULD allow initial max 3". However, the issue I see with the way the specs would look like with this specification is that there is not much determinism on how an endpoint should use MAXSTREAMS and how it can the unidirectional streams it makes available will be used. For example, a client advertises 3 initial max uni streams, likewise a server advertises 3 a server creates a 2 grease streams and 1 encoder stream Recall the Tokyo outcome was "creating them is not forbidden even if the setting prevents their use" and by default value of SETTINGSQPACKMAXTABLECAPACITY is 0 if you never receive a SETTING that tells you otherwise the client expects 3 "critical streams", so banks the difference (2) and gives unidirectional stream credit by sending a MAXSTREAMS frame of value 5 the client opens a grease stream, a control stream, and 1 encoder stream the client sends SETTINGSQPACKMAXTABLECAPACITY = Y the client sends MAXPUSHID = 3 the client sends a request with static compression the server sends 3 PUSHPROMISES and attempts to push 3 resources. the server wishes to open N unidirectional extension streams the server sends STREAMSBLOCKED with the value of 5 the client sends a request with dynamic compression the server wishes to open a decoder and sends STREAMSBLOCKED with the value of 5 a) What value of MAXSTREAMS is suitable for a client to send? b) if the value is < N+2 what is a suitable rate to send STREAMSBLOCKED? c) will this just devolve to a cycle of one endpoint sending STREAMSBLOCKED and the other echoing back MAX_STREAMS (value +1) until the other side shuts up?
It seems to me that this is the worst case, and that if we think it's fine then every other questions can be considered fine. My position is that having this worst case is fine. Many types of greasing (e.g., version negotiation, retry, HelloRetryRequest) introduces one extra roundtrip. So having one extra round-trip when opening one greasing stream sounds not unreasonable.
I'm ok with that but it also risks affecting all unidirectional stream types. Server push somewhat skirts the issue by providing a signal (counting PUSHPROMISEs) that enhances the clients knowledge. QPACK is part way there because non-zero SETTINGSQPACKMAXTABLECAPACITY implies that some credit from both ends is required. Perhaps we could summarise some guidance for people about the RTT incurred by these streams and ways to avoid them. A totally different alternative that might turn out in practice easier is to simply set maxinitialunistreams to a very large number, and set the initialmaxstreamdatauni to a low number (8 bytes would be enough to peak at the stream type). This way the client has far more insight into what the other side is attempting to do and can exercise stream control as suited to its needs. All while avoiding this weird cycle.
Agreed. Recommending "at least three, but more is better because there could be other types of streams" makes sense to me.
I think that this has resolved into an editorial-level conclusion. (That I agree with.)
I'll create a PR
Realistically, I think an implementation ought to have a higher number than three here, but the actual requirement for interop will be three much of the time, so I think this is good guidance. Note that STREAM_BLOCKED doesn't get sent over and over -- if the peer decides not to give you more credit and that means you can't use your extensions, tough luck.A minor nit, but LGTM otherwise.

streams (push-streams).  Other stream types can be defined by extensions to HTTP/3; see extensions for more details.  <ins> The performance of HTTP/3 connections in the early phase of their lifetime is sensitive to the creation and exchange of data on unidirectional streams.  Endpoints that set low values for the QUIC transport parameters "initial_max_uni_streams" and "initial_max_stream_data_uni" will increase the chance that the remote peer reaches the limit early and becomes blocked.  In particular, the value chosen for "initial_max_uni_streams" should consider that remote peers may wish to exercise reserved stream behaviour (stream-grease).  To reduce the likelihood of blocking, both clients and servers SHOULD send a value of three or greater for the QUIC transport parameter "initial_max_uni_streams", and a value of 1,024 or greater for the QUIC transport parameter "initial_max_stream_data_uni". </ins> If the stream header indicates a stream type which is not supported by the recipient, the remainder of the stream cannot be consumed as


In order to quickly complete the handshake and avoid spurious retransmissions due to crypto retransmission timeouts, crypto packets SHOULD use a very short ack delay, such as the local timer <del> granularity.  ACK frames MAY be sent immediately when the crypto </del>  stack indicates all data for that packet number space has been received.

Changes MAY to SHOULD.
Note that stacks generally don't indicate that all data is available, they just provide newer keys, or provide data that should be protected with those newer keys. But the spirit of this is fine.
On receiving the INITIAL packet, the server passes the CRYPTO data to TLS which then does a bunch of things like computing handshake key, generate INITIAL data etc and then passes it to QUIC. From measurements, it takes ~10ms before the server sends out the INITIAL packet and ACK for client's INITIAL packet. Now, the first RTT measurement doesn't include ack delay. For a very low RTT connection (two machines connected back to back via ethernet), this causes the initial RTT ~ 10ms, which is too high for such a connection. Even after subsequent RTT samples of 0ms, would take a long time to normalize the high initial value. Is there a reason why we don't subtract ack_delay from measured RTT for the INITIAL packet?
The reason to not subtract ack_delay on the first ACK is it could allow the peer to manipulate the SRTT, which may be particularly dangerous at the beginning of a connection. As a mitigation, the current text in 4.1 of Recovery says: "In order to quickly complete the handshake and avoid spurious retransmissions due to crypto retransmission timeouts, crypto packets SHOULD use a very short ack delay, such as the local timer granularity. ACK frames MAY be sent immediately when the crypto stack indicates all data for that packet number space has been received." URL We could add some text indicating that sending an ACK immediately improves the first RTT sample if that seems helpful?
PR is a substantial rewrite of the RTT estimation section, so it may be worth reading that to see if it provides the rationale you were looking for.
This is an interesting point. In addition to having an inflated RTT, there's also the risk of the crypto timer firing for long RTTs. In quicly, we don't delay acks during the handshake to avoid these issues. I wonder if this is worth mentioning in the draft.
NAME we already have :
That "MAY" is odd. You can always ACK immediately. Whether or not you "SHOULD" varies, but in this case maybe we SHOULD RECOMMEND an immediate ACK rather than having it be OPTIONAL.
I believe the MAY is only possible if the crypto stack provides a clear signal that it's received all crypto data in Initial or Handshake packets? I wasn't sure how likely that was to be available, which is why it's a MAY, but I'm also happy to make it a SHOULD. The text used to say you should always ACK crypto packets immediately, but a client might want to wait a short period of time and ACK the servers Initial in a coalesced packet with the client's Handshake ACK/CFIN, so it was adjusted to " a very short ack delay, such as the local timer granularity".
Agrees to what NAME says. "SHOULD if the crypto stack provides such signal" is what I'd favor to see in the text, I think the current text implies that.
NAME : indeed, thanks for pointing me to the text. I'm in agreement as well on turning the MAY into a SHOULD. I'll send a PR.
Sending an ACK immediately for client's INITIAL might be helpful in estimating RTT correctly but it has the disadvantage of not being able to coalesce the ACK with server's INITIAL CRYPTO. Some implementations might even coalesce the Handshake packet.
NAME : that is already covered by the current text.

In order to quickly complete the handshake and avoid spurious retransmissions due to crypto retransmission timeouts, crypto packets SHOULD use a very short ack delay, such as the local timer  <ins> granularity.  ACK frames SHOULD be sent immediately when the crypto </ins> stack indicates all data for that packet number space has been received.


sustained increase in the actual RTT, but the smoothed SRTT has not yet caught up.  Implementations MAY experiment with absolute thresholds, thresholds from previous connections, adaptive thresholds, or including RTT variance.  Smaller thresholds reduce reordering resilience and

PTO works out fine, but loss_delay becomes weird. We should probably handle that.
In some environments, Kerberos integration (for server and client authentication) might be preferable to authentication via server certificates and SPNEGO.
Outside of this WG charter. A version of QUIC could be defined which does this, hypothetically, but our charter is to use TLS as the handshake for the version of QUIC we define.
One small suggestion, but LG

sustained increase in the actual RTT, but the smoothed SRTT has not yet caught up. <ins> An endpoint might consistently record RTT samples as 0 in extremely low latency networks, leading to a smoothed_rtt of 0.  Consequently, the endpoint could declare all earlier packets as lost immediately upon receiving an acknowledgement for a later packet.  That is, the endpoint would not provide any reordering tolerance.  To avoid declaring packets as lost too early, the time threshold MUST be set to at least kGranularity (defined in ld-consts-of-interest). </ins> Implementations MAY experiment with absolute thresholds, thresholds from previous connections, adaptive thresholds, or including RTT variance.  Smaller thresholds reduce reordering resilience and


frame is the offset of the next byte that would be sent. The first byte in the stream has an offset of 0.  The largest offset <del> delivered on a stream - the sum of the offset and data length - MUST be less than 2^62. </del>  19.9.

The text here had a MUST prohibiting data+offset from exceeding 2^62-1. But that is redundant with flow control, which can't express a limit greater than this anyway. So just point to FLOWCONTROLERROR.
Sec 19.8 says: The first byte in the stream has an offset of 0. The largest offset delivered on a stream - the sum of the offset and data length - MUST be less than 2^62. It certainly possible to express a STREAM frame that violates this rule. What action should we take?
I don't know if we ever wrote this down, but I think that we might say if you catch a peer violating the spec, then you can send an error of any type you like. But for this case, PROTOCOL_VIOLATION is entirely appropriate (as with many others of this sort).
Not just offset, but offset + content size of stream frame, otherwise there is no fixed limit on stream sizes, which is unfortunate.

frame is the offset of the next byte that would be sent. The first byte in the stream has an offset of 0.  The largest offset  <ins> delivered on a stream - the sum of the offset and data length - cannot exceed 2^62-1, as it is not possible to provide flow control credit for that data.  Receipt of a frame that exceeds this limit will be treated as a connection error of type FLOW_CONTROL_ERROR. </ins> 19.9.


least 1200 bytes. The crypto retransmission timer is not set if the time threshold <del> time-threshold loss detection timer is set.  When the crypto retransmission timer is active, the probe timer (pto) is not active. </del>  6.2.1.

Adds some explanation about why it's better to arm/fire the time threshold alarm before the crypto retransmission timeout. Hopefully clarifies comments from
Thanks for the explanation. It's much clearer. Can i ask for one more clarification? In DetectLostPackets, shouldn't the for-loop have been something like this:
Thanks for catching the pseudocode error in DetectLostPackets. PTAL.
lgtm. thank you!
In both SetLossDetectionTimer() and OnLossDetectionTimeout(), the losstime based timer setup and loss detection happens before the crypto loss recovery logics. That seems at the cross road with the idea that crypto packets having higher priority in loss recovery. In fact in OnLossDetectionTimerou(), if losstime is present and we go into DetectLostPackets(), then we will only decect loss of one packet number space between Initial and Handshake. Plus the losstime used to set the timer is the earliest among all possible values, which means even if only one of Initial and Handshake space has inflight packets, we may still not retransmit all the inflight packets when this timer fires. So this current code really doesn't follow the idea that when crypto is inflight and loss timer fires, all inflight crypto needs to be retransmitted. If we change the code so that crypto is handled first, we can also reduce the states need to be maintained and simplify some of the functions. In current draft there are 3 losstimes, one per packet number space. By always handling crypto first, we will need such loss_time for AppData space only, and no need to have the helper function to GetEarliestLossTime(). (I'm more than happy to cook up a PR if people think this makes sense.)
It's been a few days. I sent PR .
This was introduced as a fix to URL, and fixed in URL I don't think we should revert that fix.
+1 to NAME comment. Timers are necessary when you have no information about what might be lost, but if you do have information, there's no reason not to use that instead.
I sent out an editorial PR to clarify why time based loss detection has higher priority.

least 1200 bytes. The crypto retransmission timer is not set if the time threshold  <ins> time-threshold loss detection timer is set.  The time threshold loss detection timer is expected to both expire earlier than the crypto retransmission timeout and be less likely to spuriously retransmit data.  The Initial and Handshake packet number spaces will typically contain a small number of packets, so losses are less likely to be detected using packet-threshold loss detection. When the crypto retransmission timer is active, the probe timer (pto) is not active. </ins> 6.2.1.


6.3.2. When a PTO timer expires, the sender MUST send one ack-eliciting <del> packet as a probe.  A sender MAY send up to two ack-eliciting packets, to avoid an expensive consecutive PTO expiration due to a single packet loss. </del>  Consecutive PTO periods increase exponentially, and as a result, connection recovery latency increases exponentially as packets

Editorial fix for This seems like the minimal fix for the existing text, though I wonder if it's the correct one.
I think this fix is fine, but it's inadequate. I'll take this in since it's useful.
Can you clarify why it's inadequate on the still open issue?
NAME primarily because it does not correct the pseudocode, but I've also made some textual adjustments in
As discussed with NAME 6.2.2.2 is awkward, because in the first paragraph it says And then the very last paragraph changes that by saying What is super unclear is that the final paragraph apparently means to imply that when there is no new or unACKed data to send, nothing should be send, i.e., it invalidates the SHOULD in the first paragraph. Related to that, the pseudo code still uses , which is also inaccurate, since it's really "at least one (unless you don't have new or unACKed data) but can also send two". Good luck finding a good name for that function :-)
The pseudocode now uses SendOneOrTwoPackets(), but the last paragraph pretty clearly violates the previous MUST, which is very bad. The simplest fix would be to change the MUST to a "SHOULD unless otherwise specified", but I'm increasingly thinking we should stick with the MUST and if you get into a situation you have nothing to send, require sending something ack-eliciting to ensure in-flight packets are removed from flight. Thoughts?

6.3.2. When a PTO timer expires, the sender MUST send one ack-eliciting  <ins> packet as a probe, unless there is nothing to send.  A sender MAY send up to two ack-eliciting packets, to avoid an expensive consecutive PTO expiration due to a single packet loss. </ins> Consecutive PTO periods increase exponentially, and as a result, connection recovery latency increases exponentially as packets