6.3.2. <del> When a PTO timer expires, the sender MUST send one ack-eliciting packet as a probe, unless there is nothing to send.  A sender MAY send up to two ack-eliciting packets, to avoid an expensive consecutive PTO expiration due to a single packet loss. </del>  Consecutive PTO periods increase exponentially, and as a result, connection recovery latency increases exponentially as packets Sending different payloads each time reduces the chances of spurious retransmission. <del> When a PTO timer expires, new or previously-sent data may not be available to send and packets may still be in flight.  A sender can be blocked from sending new data in the future if packets are left in flight.  Under these conditions, a sender SHOULD mark any packets still in flight as lost.  If a sender wishes to establish delivery of packets still in flight, it MAY send an ack-eliciting packet and re- arm the PTO timer instead. </del> 6.3.3. Delivery or loss of packets in flight is established when an ACK

This PR adds text and pseudo-code to clarify what an endpoint should do when PTO fires and there is no data or other ack-eliciting frames to send.
NAME I've done s/endpoint/sender. Let's deal with this separately. On the SHOULD, thinking about this, yes, I think I agree with you. I'll modify the PR, let's see how that looks.
Ok, modified, take a look
As written now, this swaps a SHOULD and a MAY, so I'm changing it to design.
As discussed with NAME 6.2.2.2 is awkward, because in the first paragraph it says And then the very last paragraph changes that by saying What is super unclear is that the final paragraph apparently means to imply that when there is no new or unACKed data to send, nothing should be send, i.e., it invalidates the SHOULD in the first paragraph. Related to that, the pseudo code still uses , which is also inaccurate, since it's really "at least one (unless you don't have new or unACKed data) but can also send two". Good luck finding a good name for that function :-)
The pseudocode now uses SendOneOrTwoPackets(), but the last paragraph pretty clearly violates the previous MUST, which is very bad. The simplest fix would be to change the MUST to a "SHOULD unless otherwise specified", but I'm increasingly thinking we should stick with the MUST and if you get into a situation you have nothing to send, require sending something ack-eliciting to ensure in-flight packets are removed from flight. Thoughts?

6.3.2.  <ins> When a PTO timer expires, a sender MUST send at least one ack- eliciting packet as a probe, unless there is no data available to send.  An endpoint MAY send up to two ack-eliciting packets, to avoid an expensive consecutive PTO expiration due to a single packet loss. It is possible that the sender has no new or previously-sent data to send.  As an example, consider the following sequence of events: new application data is sent in a STREAM frame, deemed lost, then retransmitted in a new packet, and then the original transmission is acknowledged.  In the absence of any new application data, a PTO timer expiration now would find the sender with no new or previously- sent data to send. When there is no data to send, the sender SHOULD send a PING or other ack-eliciting frame in a single packet, re-arming the PTO timer. Alternatively, instead of sending an ack-eliciting packet, the sender MAY mark any packets still in flight as lost.  Doing so avoids sending an additional packet, but increases the risk that loss is declared too aggressively, resulting in an unnecessary rate reduction by the congestion controller. </ins> Consecutive PTO periods increase exponentially, and as a result, connection recovery latency increases exponentially as packets Sending different payloads each time reduces the chances of spurious retransmission.  6.3.3. Delivery or loss of packets in flight is established when an ACK


packets MUST define a mechanism that is robust against version downgrade attacks.  22. 22.1.

I went to and found that the offending text was already removed, so we're most of the way there. I added a note to the Security Considerations about not letting clients dictate your routing decisions.
In Section 7.2, the spec makes an implicit assumption that the CID of the Initial packet is used for routing: makes this even more explicit: However, in the discussion of , part of the conclusion was that, to protect from malicious clients, load balancers cannot safely permit client-chosen fields to influence the routing of their packets to a particular server. IIRC, this was also discussed in New York. Did we mean that? If so, we have conflicting text.
I'd say sometimes it's safe and sometimes it's not and there are pros and cons of different approaches. You mention conflicting text, can you point out the text that conflicts with the sections you quoted above in this issue?
I tend to agree with Ian, but an Initial that follows a stateless reset would have to be routed consistently. I assume this reset driven routing cannot (trivially) be abused due to integrity checks in the reset, but I don't recall the details.
Conflicting as in, we have text (quoted) which contradicts my understanding of our decision. I think if we decide to go with what we'd discussed before (don't trust the CID of Initial/0-RTT), we should add some Security Considerations text about that.
NAME I presume you mean Retry. The server can specify any CID in the Retry, but I'd expect the routing to be achieved by IP:port consistency throughout the handshake.
I mean Retry, of course. You cannot assume IP/port consistency because the routing will in some cases happen behind an external load balancer that exposes a single public IP and routes to a specific internal IP based on CID.
If a middlebox can tell the difference between and original and a Retry driven Initial packet, it would be possible configure the middlebox to a) route randomly on first initial, or b) route consistently on first initial, and for both a) and b) route consistently after a retry. In the case b) the middlebox could force a retry when it thinks there is overload either accidentally, or through an attack. If the Retry is verified properly, it is not possible to forge a successful CID in this case. All of this should be possible, and trivial to do, but need some reading up to see what is the current state of affairs. I think middleboxes might not be able to detect a Retry driven initial but endpoints are since a Retry can only happen once.
Nah, I think a retry would not make sense in the above, or least not where you end up with the same external loadbalancer and public IP. In that case the load balancer could just internally swap endpoint. So NAME is probably right about IP/port stability. EDIT: disagreeing with myself again: if the load balancer is distributed, but has a single IP, and if is mostly stateless and depends on the CID for actual routing, it does indeed need a Retry even if the IP/port is the same.
My answer is that both are true and that they can co-exist. While the server can't rely on the randomized connection ID being good, it might choose to allow routing based on that value under normal circumstances. The recommendation that a client not tamper with fields that might alter routing is a sensible one. As is the recommendation that a server not attach too much significance to values chosen by clients.
NAME : I don't think there is any text that suggests a server "not attach too much significance to values chosen by clients". Do folks think it might be useful to add to Security Considerations?
Obviously I think we should. :-) If we think there's a place for both, then let's give both some airtime in the spec.
Discussed in Tokyo; add Security Considerations text about an attacker's ability to influence server selection, but don't normatively forbid the use of the ODCID in routing decisions.
I think the point NAME is making is a sensible one. The explicit text that Mike has quoted is implying that servers SHOULD use Client-chosen CID for routing. After all, this purpose is explicitly stated, without any caveats, right there in the spec. I would rather see that quoted text removed altogether. It is enough to state that "A client SHOULD select an initial Destination Connection ID length long enough to satisfy the minimum size requirements for every QUIC version it supports". This is accurate and not misleading.
Fix NAME suggestions and we're good.

packets MUST define a mechanism that is robust against version downgrade attacks. <ins> 21.10. Deployments should limit the ability of an attacker to target a new connection to a particular server instance.  This means that client- controlled fields, such as the initial Destination Connection ID used on Initial and 0-RTT packets SHOULD NOT be used by themselves to make routing decisions.  Ideally, routing decisions are made independently of client-selected values; a Source Connection ID can be selected to route later packets to the same server. </ins> 22. 22.1.


have 1-RTT keys.  If the crypto retransmission timer expires before the client has 1-RTT keys, it is possible that the client may not have any crypto data to retransmit.  However, the client MUST send a <del> new packet, containing only PING or PADDING frames if necessary, to allow the server to continue sending data.  If Handshake keys are available to the client, it MUST send a Handshake packet, and otherwise it MUST send an Initial packet in a UDP datagram of at least 1200 bytes. </del>  The crypto retransmission timer is not set if the time threshold time-threshold loss detection timer is set.  The time threshold loss

That would actually resolve itself, because at some point, the client would send a Handshake packet with data in it, and cause all the outstanding PADDING packets to be acked. If all the client's packets are lost and the server is stuck in deadlock, the handshake is doomed anyway.
It's probably worth acknowledging the edge case and just saying that this protocol doesn't do anything about that. Note that the case we're concerned about should involve the server sending tons of data, which gives it plenty of opportunity to provide the client with an ACK that will unstick it.
LGTM, once you've added the note about congestion.
Still holding on a note about congestion. I will merge and open another issue if that doesn't happen soon-ish.
I'll do that today.
I added description of why I believe this is not a problem. Conveniently, the logic is simpler than I previously stated, because of the fast Initial -> Handshake transition when the client gets Handshake keys.
Now that I've started a PR to combine the crypto handshake timer and PTO, I'm concerned I chose the wrong fix for in this PR, and it would be better to allow PING in Initial. In the existing PTO case when there are bytes in flight, but nothing to send, we say "When there is no data to send, the sender SHOULD send a PING or other ack-eliciting frame in a single packet, re-arming the PTO timer." If we allow PING in Initial, that text will work fine with a combined timer, since all Initials must be in padded datagrams. Otherwise, combining them adds a slightly odd special case. NAME I'd like your thoughts on this in particular.
NAME Good catch! It's useful to go back to the principle we have in place: if we want a PTO running, we need to have an ack-eliciting packet outstanding. I don't think there's a strong enough argument to create an exception here, so I agree that we should change this to always sending a PING when the timer goes off and there's nothing to send. Since the title here is quite clearly the opposite of that, in the interest of transparency, I propose that you close this PR in favor of a new one that says send PING (and stack this new PR on top of ).
Um. Ok. This is attached to an issue that is labeled and now has consensus declared on it with this PR. I'm fried right now, but let's discuss this before doing anything.
In my view the point of not sending PING early is to reduce the complexity of the early packet number spaces and allow special code to handle these. The fewer types of packets and ACK's here the better. However, if PING ends up simplifying this logic, then certainly there is an argument for that. The question then is: would PING' complicate ACK logic in early packet number spaces? That would only really be the case if there were no ACK's otherwise since PING's are easy to handle. OTOH - if it is only a matter of cleaning up two paragraphs in the spec it may not be worthwhile to allow early PING. An exception may be the natural thing if the PTO logic is in fact separate anyway in the different PN spaces.
Note that I didn't include this in the changelog for -21. It seems entirely editorial to my reading.
The says: PING frames are forbidden in both Initial and Handshake packets.
The smallest change is to only send PADDING without a PING, since the purpose here is to allow the server to send, not for the client to receive an ACK. The PING doesn't really add any value.
Sorry for the churn. I sent out the smallest fix possible. We can also change the transport doc to allow PING in Initial and Handshake. Functionally, I don't think it matters for recovery, but both are reasonable options in my opinion.
There's this weird edge case where you might run out of cwnd, because PADDING counts towards in-flight but is not ack-eliciting, but it seems safe to ignore that one. If you don't receive and acknowledgment for that many packets right at the beginning of the connection, something is going seriously wrong anyway.

have 1-RTT keys.  If the crypto retransmission timer expires before the client has 1-RTT keys, it is possible that the client may not have any crypto data to retransmit.  However, the client MUST send a  <ins> new packet, containing only PADDING frames if necessary, to allow the server to continue sending data.  If Handshake keys are available to the client, it MUST send a Handshake packet, and otherwise it MUST send an Initial packet in a UDP datagram of at least 1200 bytes. Because packets only containing PADDING do not elicit an acknowledgement, they may never be acknowledged, but they are removed from bytes in flight when the client gets Handshake keys and the Initial keys are discarded. </ins> The crypto retransmission timer is not set if the time threshold time-threshold loss detection timer is set.  The time threshold loss


promises to never delay acknowledgments of an ack-eliciting packet by more than the indicated value.  If it does, any excess accrues to the RTT estimate and could result in spurious retransmissions from the <del> peer. </del>  5.

Let's do this right and set the value to 0 initially in the transport doc. The "crypto packet" hack is cute, but it's another special case, where this doesn't need a special case, just an initial value.
I think this PR is orthogonal to the question what the default value for this TP is. We use default values to allow peers to skip sending certain transport parameters. This PR is about initializing the maxackdelay variable in the recovery code before we even received the TPs.
NAME I think that using max-ack-delay 0 for crypto packets (to be precise, Initial and Hanshake packets) is not a hack. It's preferable. The problem of respecting TP.maxackdelay for Initial and Handshake packets, once receiving TP from the peer, is that loss recovery would kick in seconds later than they can (note: peer is required to send ACK ASAP).
I see what you mean. I'm still concerned about the special case-y nature of this. Can we segregate by packet number space instead? Is there any reason not to just say that this only applies to Initial and Handshake packets? By my rough reckoning, these should all be acknowledged immediately. Having the transport parameters only apply to 1-RTT (and maybe 0-RTT, which I haven't thought through) is probably right.
I think that just saying that is fine.
I agree. I'll update my PR.
I updated the PR such that maxackdelay is only used for the 1-RTT packet number space.
Note that the ack frames is also not defined before TP's are ready: >The value of the ACK Delay field is scaled by multiplying the encoded value by 2 to the power of the value of the ackdelay_exponent transport parameter set by the sender of the ACK frame.
NAME That is not correct, see URL
OK, but given the latest commit , for Initial, Handshake, does it even make sense to define the exponent for that value rather the specifying ack.ack_delay must be zero for those packets?
NAME This is true, there's no sense in defining an ackdelayexponent for Initial and Handshake packets if we're going to ignore the ACK delay anyway. Should be safe to recommend to just send a 0 for these packet number spaces. I updated the PR accordingly. Since this now also touches the transport document, I guess we'll have to apply the late-stage consensus process to get this merged.
The main optimization the handshake timer makes is the assumption that maxackdelay does not need to be waited for before firing. However, it also uses a completely different equation for a timeout (2 SRTT vs SRTT + 4RTTVar). We could fix the maxackdelay difference by defaulting maxackdelay to 0(), like almost all other transport parameters. If there are no RTT measurements or estimates, we could use 1 second. If there are no RTT measurements, but there is an estimate, we could either use the PTO formula and assume RTTVar is 1/2 RTT(=> 3*RTT), or we could recommend estimating RTTVar as well as RTT. The first approach seems more practical. Another notable difference is the advice to retransmit all the handshake packets instead of 1 or 2, but given the current 3x amplification limit, if a server sends 2 packets in response to a ClientHello, they can't even retransmit both packets, so advising that they retransmit all handshake packets is not indicative of real-world behavior. If the server only has 1 handshake packet, then there's no change. The client should always be able to fit its CRYPTO data into a single packet, unless there are large client certs. If there are large client certs, one would expect time or packet threshold loss recovery to be faster and more effective than timer based retransmissions. And at that point, the client should have at least one RTT sample. The goal of this issue is to make the recovery mechanisms simpler, as well as including RTTVar in the handshake timer, as suggests.
Yes -- I was heading in this direction with , and it is how we're implementing recovery in quicly. I realized that we have a false separation between the crypto retransmission timer and the PTO timers. Using only the PTO timer basically is the most sensible approach. We need to ensure that maxackdelay is used only after handshake, and there's some care required there, but that's still much better than what we have now.
The maxackdelay is advertised in the transport parameters. The recovery text assumes that this value is available right from the beginning of a connection.
I thought this sentence indicated it was 25ms until a value is received: "The maximum ack delay is communicated in the "maxackdelay" transport parameter and the default value is 25ms.", but I agree it could be clearer, ie: "and the default value and the value prior to receiving the transport parameters is 25ms."? As a related point, I think it turns out the maxackdelay doesn't come into play until the transport params are available, because "crypto packets SHOULD use a very short ack delay, such as the local timer granularity." URL
It's not used for sending, but we're using it when calculating the RTT.
Hmm, maybe we should default it to 0 before it's been received then? Or should we just specify it's 25ms until transport params have been received?
I'd argue that we should use 0 until receiving TP. We require endpoints to send ACKs for crypto packets immediately. That means that the ack-delay value found in the ACK packet is something that the peer could not control. It is my understanding that we consider these uncontrollable delays induced within an endpoint as part of RTT.
NAME : It's not about the RTT, it's about the PTO, or crypto timeout to be more precise. It's reasonable to continue using maxackdelay for RTT computations, since it caps ackdelay (ackdelay should be 0 or something very small for those anyways). But when computing the crypto timeout, we don't want to include the maxackdelay for the reasons you cite. The draft currently does not have the maxackdelay being used for the crypto retransmission timer, which was an oversight. We'll clean that up separately, but fortuitously, this means that there's no maxackdelay in the pseudocode or text to change. I don't mind initializing it to 0 or any value, but it won't have any real effect on endpoint behavior based on the draft. maxack_delay doesn't kick in until after the handshake is roughly complete.
The crypto retransmission timer doesn't use maxackdelay intentionally, see: URL "In order to quickly complete the handshake and avoid spurious retransmissions due to crypto retransmission timeouts, crypto packets SHOULD use a very short ack delay, such as the local timer granularity. ACK frames SHOULD be sent immediately when the crypto stack indicates all data for that packet number space has been received.
Same applies to
NAME Which why the spec says the following:
NAME the PR for this changed the transport doc. Can you check whether we need to run a consensus call?
The PR is already merged, so closing.
Still LGTM.

promises to never delay acknowledgments of an ack-eliciting packet by more than the indicated value.  If it does, any excess accrues to the RTT estimate and could result in spurious retransmissions from the  <ins> peer.  For Initial and Handshake packets, a max_ack_delay of 0 is used. </ins> 5.


value to seed the RTT estimator for a subsequent connection attempt to the server. <del> 6.2.2. When packet protection keys are discarded (see Section 4.9 of QUIC- TLS), all packets that were sent with those keys can no longer be acknowledged because their acknowledgements cannot be processed anymore.  The sender MUST discard all recovery state associated with those packets and MUST remove them from the count of bytes in flight. Endpoints stop sending and receiving Initial packets once they start exchanging Handshake packets (see Section 17.2.2.1 of QUIC- TRANSPORT).  At this point, recovery state for all in-flight Initial packets is discarded. When 0-RTT is rejected, recovery state for all in-flight 0-RTT packets is discarded. If a server accepts 0-RTT, but does not buffer 0-RTT packets that arrive before Initial packets, early 0-RTT packets will be declared lost, but that is expected to be infrequent. It is expected that keys are discarded after packets encrypted with them would be acknowledged or declared lost.  Initial secrets however might be destroyed sooner, as soon as handshake keys are available (see Section 4.10 of QUIC-TLS). </del> 6.3. A Probe Timeout (PTO) triggers a probe packet when ack-eliciting data 6.4.  The majority of constants were derived from best common practices among widely deployed TCP implementations on the internet. Exceptions follow.

Move the "Discarding Keys and Packet State" section up a level and out from underneath the Crypto Handshake Timer. This section in no way references the crypto handshake timer, so there's no reason to put it there. It's possible this should be up-leveled out of loss recovery as well? The preceding "Retry and Version Negotiation" has a similar problem, so I'd like suggestions on where to move that too.

value to seed the RTT estimator for a subsequent connection attempt to the server.  6.3. A Probe Timeout (PTO) triggers a probe packet when ack-eliciting data 6.4. <ins> When packet protection keys are discarded (see Section 4.9 of QUIC- TLS), all packets that were sent with those keys can no longer be acknowledged because their acknowledgements cannot be processed anymore.  The sender MUST discard all recovery state associated with those packets and MUST remove them from the count of bytes in flight. Endpoints stop sending and receiving Initial packets once they start exchanging Handshake packets (see Section 17.2.2.1 of QUIC- TRANSPORT).  At this point, recovery state for all in-flight Initial packets is discarded. When 0-RTT is rejected, recovery state for all in-flight 0-RTT packets is discarded. If a server accepts 0-RTT, but does not buffer 0-RTT packets that arrive before Initial packets, early 0-RTT packets will be declared lost, but that is expected to be infrequent. It is expected that keys are discarded after packets encrypted with them would be acknowledged or declared lost.  Initial secrets however might be destroyed sooner, as soon as handshake keys are available (see Section 4.10 of QUIC-TLS). 6.5. </ins> The majority of constants were derived from best common practices among widely deployed TCP implementations on the internet. Exceptions follow.


4.4. <del> HTTP/3 server push is similar to what is described in HTTP/2 HTTP2, but uses different mechanisms. </del>  Each server push is identified by a unique Push ID.  This Push ID is used in a single PUSH_PROMISE frame (see frame-push-promise) which 6.2.2.  A push stream is indicated by a stream type of "0x01", followed by the Push ID of the promise that it fulfills, encoded as a variable- length integer.  The remaining data on this stream consists of HTTP/3

The transition to "Push Streams" has no preceding mention of server push. This adds a reference to the correct section of RFC7540.
If mention of server push is desired before mention of push stream, would it be simpler to move the final sentence of the paragraph to the start?
Maybe this would be better if section 4 was moved after section 6? Then the concepts would be explained before going into the framing. I'd also like to reorder/renumber all the push frame types so they're all adjacent to one another and in a sensible order. Is there a reason they're numbered as they are? If not, I'll open an issue to renumber them.
There's already an issue to reorder some of the sections, so I'm happy to consider this as part of the same discussion. As to the numbering, the primary reason is maintaining the same frame IDs for things that stayed over from HTTP/2 -- PUSHPROMISE has the same ID, CANCELPUSH has the ID that HTTP/2 used for RSTSTREAM, and DUPLICATEPUSH is brand new and so pushes into unallocated space.
Given that is in now, do you still see this as a problem?
I think the Server Push section still lacks a simple explanation of what the push does. A sentence like what I added in this PR at the beginning would help that. Currently the Server Push section goes directly into mechanics. URL "HTTP/3 server push is similar to what is described in HTTP/2 {{!HTTP2}}, but uses different mechanisms. Each server push is identified by a unique Push ID. ..." I'd add more to the first paragraph in case people don't want to read the H2 draft.
I think I'd prefer for the reference to be to 4.4 of the current draft instead, since that does point to RFC7540 and has more context as well.
Whoops -- which also doesn't have the two-sentence explanation of what push is. That's up in the intro. Let me copy that over to 4.4.
Thanks for the added text Mike, it looks great. I'm happy with this, so it's up to you to decide when this is ready to merge.

4.4.  <ins> Server push is an interaction mode introduced in HTTP/2 HTTP2 which permits a server to push a request-response exchange to a client in anticipation of the client making the indicated request.  This trades off network usage against a potential latency gain.  HTTP/3 server push is similar to what is described in HTTP/2 HTTP2, but uses different mechanisms. </ins> Each server push is identified by a unique Push ID.  This Push ID is used in a single PUSH_PROMISE frame (see frame-push-promise) which 6.2.2. <ins> Server push is an optional feature introduced in HTTP/2 that allows a server to initiate a response before a request has been made.  See server-push for more details. </ins> A push stream is indicated by a stream type of "0x01", followed by the Push ID of the promise that it fulfills, encoded as a variable- length integer.  The remaining data on this stream consists of HTTP/3


Any packet containing only one or more ACK frame(s). <del> Packets are considered in-flight when they have been sent and neither acknowledged nor declared lost, and they are not ACK-only. </del>  All frames besides ACK or PADDING are considered ack-eliciting. 3.1.3.  QUIC ACKs contain information that is similar to TCP SACK, but QUIC does not allow any acked packet to be reneged, greatly simplifying implementations on both sides and reducing memory pressure on the sender. <del> 3.1.4. </del>  QUIC supports many ACK ranges, opposed to TCP's 3 SACK ranges.  In high loss environments, this speeds recovery, reduces spurious retransmits, and ensures forward progress without relying on timeouts. <del> 3.1.5. </del>  QUIC endpoints measure the delay incurred between when a packet is received and when the corresponding acknowledgment is sent, allowing 6.1.2. <del> Once a later packet has been acknowledged, an endpoint SHOULD declare an earlier packet lost if it was sent a threshold amount of time in the past.  The time threshold is computed as kTimeThreshold * max(SRTT, latest_RTT).  If packets sent prior to the largest acknowledged packet cannot yet be declared lost, then a timer SHOULD be set for the remaining time. The RECOMMENDED time threshold (kTimeThreshold), expressed as a round-trip time multiplier, is 9/8. </del>  Using max(SRTT, latest_RTT) protects from the two following cases: sustained increase in the actual RTT, but the smoothed SRTT has not yet caught up. <del> An endpoint might consistently record RTT samples as 0 in extremely low latency networks, leading to a smoothed_rtt of 0.  Consequently, the endpoint could declare all earlier packets as lost immediately upon receiving an acknowledgement for a later packet.  That is, the endpoint would not provide any reordering tolerance.  To avoid declaring packets as lost too early, the time threshold MUST be set to at least kGranularity (defined in ld-consts-of-interest). </del>  Implementations MAY experiment with absolute thresholds, thresholds from previous connections, adaptive thresholds, or including RTT QUIC begins every connection in slow start and exits slow start upon loss or upon increase in the ECN-CE counter.  QUIC re-enters slow <del> start anytime the congestion window is less than ssthresh, which typically only occurs after an PTO.  While in slow start, QUIC increases the congestion window by the number of bytes acknowledged when each acknowledgment is processed. </del>  7.3.

From a full read en route to Prague.
Thanks Jana, suggestions taken and "Clearer Loss Epoch" rewritten.

Any packet containing only one or more ACK frame(s).  <ins> Packets are considered in-flight when they have been sent and are not ACK-only, and they are not acknowledged, declared lost, or abandoned along with old keys. </ins> All frames besides ACK or PADDING are considered ack-eliciting. 3.1.3. <ins> QUIC ends a loss epoch when a packet sent after loss is declared is acknowledged.  TCP waits for the gap in the sequence number space to be filled, and so if a segment is lost multiple times in a row, the loss epoch may not end for several round trips.  Because both should reduce their congestion windows only once per epoch, QUIC will do it correctly once for every round trip that experiences loss, while TCP may only do it once across multiple round trips. 3.1.4. </ins> QUIC ACKs contain information that is similar to TCP SACK, but QUIC does not allow any acked packet to be reneged, greatly simplifying implementations on both sides and reducing memory pressure on the sender.  <ins> 3.1.5. </ins> QUIC supports many ACK ranges, opposed to TCP's 3 SACK ranges.  In high loss environments, this speeds recovery, reduces spurious retransmits, and ensures forward progress without relying on timeouts.  <ins> 3.1.6. </ins> QUIC endpoints measure the delay incurred between when a packet is received and when the corresponding acknowledgment is sent, allowing 6.1.2.  <ins> Once a later packet packet within the same packet number space has been acknowledged, an endpoint SHOULD declare an earlier packet lost if it was sent a threshold amount of time in the past.  To avoid declaring packets as lost too early, this time threshold MUST be set to at least kGranularity.  The time threshold is: ~~~ kTimeThreshold * max(SRTT, latest_RTT, kGranularity) ~~~ If packets sent prior to the largest acknowledged packet cannot yet be declared lost, then a timer SHOULD be set for the remaining time. </ins> Using max(SRTT, latest_RTT) protects from the two following cases: sustained increase in the actual RTT, but the smoothed SRTT has not yet caught up.  <ins> The RECOMMENDED time threshold (kTimeThreshold), expressed as a round-trip time multiplier, is 9/8. </ins> Implementations MAY experiment with absolute thresholds, thresholds from previous connections, adaptive thresholds, or including RTT QUIC begins every connection in slow start and exits slow start upon loss or upon increase in the ECN-CE counter.  QUIC re-enters slow  <ins> start anytime the congestion window is less than ssthresh, which only occurs after persistent congestion is declared.  While in slow start, QUIC increases the congestion window by the number of bytes acknowledged when each acknowledgment is processed. </ins> 7.3.


PUSH_PROMISE frames are not part of the response; see server-push for more details.  The "chunked" transfer encoding defined in Section 4.1 of RFC7230 MUST NOT be used.

This arises from a different discussion I had about the message exchange definition in section 5.1. The description is correct but glosses over the detail of QPACK, which is mentioned very briefly later in section 5.1.1. For complete accuracy, I would argue that a single message exchange may also contain one or more QPACK encoder instructions sent on the encoder stream.

PUSH_PROMISE frames are not part of the response; see server-push for more details. <ins> The HEADERS and PUSH_PROMISE frames might reference updates to the QPACK dynamic table.  While these updates are not directly part of the message exchange, they must be received and processed before the message can be consumed.  See header-formatting for more details. </ins> The "chunked" transfer encoding defined in Section 4.1 of RFC7230 MUST NOT be used.


versions without retaining state.  Though either the Initial packet or the Version Negotiation packet that is sent in response could be lost, the client will send new packets until it successfully receives <del> a response or it abandons the connection attempt. </del>  A server MAY limit the number of Version Negotiation packets it sends.  For instance, a server that is able to recognize packets as

Just in case it wasn't perfectly clear from the choice of words.
I know how you feel. We'll be defining "the" eventually. I don't know where the line is, but I think that it helps in this case to be doubly explicit.
I'm wondering how one could possibly misinterpret "abandoning" here. Furthermore, I don't think it's relevant for the server here if the client actually discarded the connection state, as long as it doesn't any more packets.
I don't think we need to spend a lot of energy here. I agree that this doesn't seem necessary, but I'm not going to fight it either.
The draft currently reads: "When a client receives a Version Negotiation packet, it MUST abandon the current connection attempt." However, it does not define what it is to abandon a connection attempt. Is it an abrupt termination without writing a connection close? That probably makes the most sense since the server is meant to be stateless at this stage.
Do we really need to define "abandon"? And if so, don't we need to define what "discarding" means?LGTM

versions without retaining state.  Though either the Initial packet or the Version Negotiation packet that is sent in response could be lost, the client will send new packets until it successfully receives  <ins> a response or it abandons the connection attempt.  As a result, the client discards all state for the connection and does not send any more packets on the connection. </ins> A server MAY limit the number of Version Negotiation packets it sends.  For instance, a server that is able to recognize packets as


6.9. <del> A congestion window that is under-utilized SHOULD NOT be increased in either slow start or congestion avoidance.  This can happen due to insufficient application data or flow control credit. </del>  A sender MAY use the pipeACK method described in section 4.3 of RFC7661 to determine if the congestion window is sufficiently sender should not consider itself application limited if it would have fully utilized the congestion window without pacing delay. <del> Bursting more than an initial window's worth of data into the network might cause short-term congestion and losses.  Implemementations SHOULD either use pacing or reduce their congestion window to limit such bursts. </del>  A sender MAY implement alternate mechanisms to update its congestion window after periods of under-utilization, such as those proposed for

I believe this also fixes the remaining portion of
When an ACK arrives that acknowledges N packets (where N could be very large), QUIC will increase the congestion window by more than 1MSS per congestion window during Congestion Avoidance causing bursty send. This could also affect the slow start. if (congestionwindow < ssthresh): // Slow start. congestionwindow += URL else: // Congestion avoidance. congestionwindow += kMaxDatagramSize * URL / congestionwindow One could argue that QUIC ACKs every other packet, but there ACK packet could get lost or ACK for ACK packet could get lost. The draft is probably assuming packet pacing but not all implementations support that. I discussed this with Jana and he thinks that we should add something in pseudo code or some specific text in the draft.
PR added more specific text on avoiding bursts into the network, FYI.
Please note that PR addresses the issue of app-limited or pacing-limited cases to not increase cwnd when it is under utilized. My concern is cwnd being inflated due to an ACK frame containing ACKs for large number of packets.
NAME : The problem here is that we don't talk about burstiness that can occur when pacing is not in place, and a stretch-ack is received. PRR handles this, but we don't do PRR in the draft.
No we need to spend lots of time on Reno as everybody appears to be doing Cubic or BBR?
Cubic has this problem. BBR cannot be assumed.
Except Cubic opens the window based on time spent, not just ack received
I don't get the scenario. It seems that NAME is worried about receiving an ACK for a large number of packets. This is not a congestion window problem, because it happens even if the congestion windows size remains constant. The root cause is that the ACK of N packets now reduces the number of packet in flow by N. So this is strictly about pacing, not about congestion control.
NAME yes, and it can and still does burst because of stretch acks
Not just stretch Acks -- deliberate Ack compression by the network happens too. Very few alternative to pacing.
URL describes
Discussed in Cupertino. NAME to prepare a PR that hopefully can incorporate some ideas from RFC3465 to add safety features for non-pacing stacks.
But 3465 does not solve the burst problem. It limits the growth of the window, but does not do pacing of any kind.
I think we can change to something like this
NAME I think we need to do 2MSS clamping per ACK received and not per packet acked. I am thinking that we can remove OnPacketAckedCC and instead do OnAckReceivedCC. Pseudocode change something like this:
I think we can keep but just move cwnd update to as you propose.
Yup, that's an important missing fix, though it needs to be applied only when not pacing. Additionally, helps with burstiness in general. NAME : I would make a variable, and apply the min here when not pacing.
The proposed change has the effect of reducing the effective Window if ACKs are compressed. Say you have a window of 12 packets, acks are delayed, and a single ACK acknowledges 12 packets. Now you have a window of 2 packets, your transmission rate is 6 times slower than before, and it will take you 10 RTT to recover. Is that really what you want?
NAME This is what ABC (RFC 3465) does for TCP. ACK compression has side-effects, and the way to deal with it is, as you say, to pace packets out. This reduction in performance is only when the endpoint does not pace, and it is the only way to limit bursts in that case.
Also, ACK compression does not necessarily imply ACK pruning. I have traces in which 1 RTT worth of ACKs is compressed and delivered back to back. In that case, the proposed algorithm will do nothing: you will get 2 MSS added for each ACK, back to back within less than 1 ms.
FYI. FreeBSD implements rfc3465 (and enabled by default) URL Linux removed tcpabc implementation but saying it does in a different way: URL Considering FreeBSD doesn't have pacing by default, I think current proposed algorithm is not worse (or aggressive) than what tcp implementation does.
NAME I am working on the PR for pseudocode changes.
I agree that when not pacing, using an ABC limit is appropriate. FWIW Windows TCP was using a limit of 4 and not 2 for a long time and now uses 8 which is pretty close to IW10 burst in slow start phase.
For the WG: The proposed PR attempts to include a mechanism to limit CWND increase during slow start when not pacing. I'll note that by itself, it does not guarantee we follow the existing MUST: "Implementations MUST either use pacing or limit such bursts to the initial congestion window, which is recommended to be the minimum of 10 maxdatagramsize and max(2 maxdatagramsize, 14720)), where maxdatagramsize is the current maximum size of a datagram for the connection, not including UDP or IP overhead." I don't think we should add this to the pseudocode, because we say SHOULD pace packets and the pseudocode currently only covers SHOULDs, not MAYs. Adding all MAYs to the pseudocode is too complex. I don't have a concern about adding a reference to the existing RFC as a mechanism that MAY help limit bursts during slow start. I'd like a decision from the WG on whether to include this change and if so, some criteria on what should be included going forward.
I agree with the comment that burst from ACK compression or Stretch-ACKs result in bursts. I think the ways to change that is implement a maximum burst size or use pacing. The bursts do not appear a CC function to me.
Bringing conversation in from the list: With regards to , one of the things we found in testing was that it could (a) still allow an implementation to be vulnerable to too much bursting (the original problem), but more importantly (b) locked receivers into an ack pattern of acking every other packet. My main concern is that this causes problems if receivers want to experiment with and change acking strategies going forward in way that otherwise would be fine. One way to look at the problem described by this issue is that the existing text has a minor underspecification, in which it has the notion of a burst limit but says nothing about the frequency of such bursts. Two "bursts" of 10MSS that are sent within a couple nanoseconds are really one burst that's too big. Another way of fixing this is to remove even the specification of the 10MSS limit if we really don't want too much text about the non-pacing cases. Essentially, what this comes down to is: pace, or even if you don't pace, you need to not send bursts above some frequency; and if you don't set some pacing timers, you'll likely underutilize the link.
I'm going to put this out there: the answer is MUST pace. You might formalize this by saying the number of packets sent in any interval cannot exceed by more than the initial cwnd (i.e., the 10MSS). That allows for bursting, but requires a period of quiet either preceding or following any burst.
NAME Yes, it does largely reduce to "implementations MUST pace" (even if the pacing algorithm isn't very nuanced, in which case you'll perform poorly). The formalization of the rate, one option for which was specified in , I think is the heart of what should be done here.
There are two routes that we can take: Specify clearly what 'delay` refers to in the below statement, which is done in OR Change MUST pace to SHOULD and remove the 10 MSS burst size limit and provide general recommendation if an implementation doesn't pace. Something like:
I'm sympathetic to saying MUST pace, and I like the formatlization that NAME suggests. I've heard some pushback on saying MUST pace in the past, but it might be worth giving it a shot again. NAME NAME NAME : What do you think?
That said... I will note that NAME 's formalization allows for a burst at the end of an RTT. Consider for example when a single ACK is received that acknowledges an entire window. That would be legal by this definition. I think that's ok.
Discussed in ZRH. Proposed resolution is to define and REQUIRE pacing.
NAME I think requiring pacing makes good sense, with some formalization of what that means.
To clarify the status here, my understanding is that the proposal is to close this design issue with no action. The discussions have identified editorial improvements that will be addressed independently.
Let's continue the editorial discussion on .
Closing this issue and continuing discussion on
This needs to run through the consensus call procedure, so we'll keep it open for another week.
"Bursting more than an intial window’s worth of data into the network might cause short-term congestion and losses." • Is that really "more than...", I think the evidence was that a larger IW (as in QUIC) can lead to congestion. • We could explain burst size and scenarios, or could simply say something like: "Bursting an initial window’s worth of data into the network can cause short-term congestion and losses for some path conditions." (This also fixes /intial/initial)?
I think we should address this in PR
Linux pacing has an initial burst of 10 packets, followed by pacing, so this is intended to allow for that style of sending. In which case, I think "more than" is the right term? I think the evidence you have in mind is some data presented early on that pacing + IW32 had lower losses than no pacing and IW10? That gap largely is driven by the large initial burst out of quiescence without pacing, which would be mitigated by sending a 10 packet burst and pacing the rest(or doing some form of slow start after idle).
Ah, my reading of the text led me in a different direction - I think we can now talk. ( I know Google shared data on IW10 and also later on other IW proposals. I also think I have seen other analysis that suggested that a larger IW can have an impact - IW10, or larger - can have collateral damage to flows sharing a bottleneck. In this case, because there is no ACK-clocking. ) Could I propose describing limiting the max burst size and enabling pacing for any remaining packets sent above this using the remaining (cwnd-maxburst). My reading is that the current text sets the maxburst size to IW. That is certainly one choice, and could be an upper limit. I am less clear that IW (the startup cwnd) should always be the same as the maximum burst size. Some SCTP implementation uses 4 packets, and I suspect that there could be cases where someone proposed a larger IW, but retained a max_burst that is lower than the increased IW.
I think making IW the max burst size, rather than the recommended burst size is sensible. I've certainly seen real-world traces where bursting 4 packets would not have caused loss, but 10 did. Text suggestions on the PR very welcome.
NAME I think you might be talking about a general max burst size, where the recovery draft (and NAME is trying to talk about an "initial" window's size -- the size of a burst after quiescence. Is that right? I think we need to do some wordsmiting, but I want to make sure that we're talking about the same thing first.
Is there such a thing. I'm wondering: if you use pacing, you still want to cluster your transmissions so you don't get too frequent timer interrupts unless the network adapter is capable of pacing. So how do you decide the granularity of bursts when pacing?
That's a different thing - pacing shouldn't mean you can't have (smallish) bursts.
In response to Jana: I think it would be a good step to first make sure we understand the same by IW - which is initial cwnd, and somewhat dependent on RTT - and a function of the capacity; and maxburst; which isn't related to RTT, but could be related to when it is helpful to "pace". As a starting point, RFC4960 recommends maxburst as 4. Would we agree for instance, largest(max_burst) <= IW?
The current text(before my outstanding PR) does not mention out of quiescence anymore. However, upon re-reading the text, I think it could be clearer, so I updated it in PR .
Gorry: Yes, I agree that micro-bursts ought to be smaller than IW. That's sensible, I think, but we really don't have a best practice for micro-bursts that we can point to (I just remembered that I had written a with mallman 14 years ago on trying to specify micro-burst behavior!). Yes, RFC 4960 uses a simple micro-burst of 4 segments, but that was based on RFC 3390, which used to limit the initial window to at most 4 segments. That's changed now, but I'd be wary of reflecting that change onto micro-bursts, because the analysis is quite different. So while I agree that largest(maxburst) <= IW, I think talking about maxburst adds something new to what the draft is currently talking about. The draft is simply using IW for both connection start-up and noting that as a known upper bound for bursts. That is not the same as specifying maxburst however. I think we ought to say something for senders coming out of an app limited period. (Perhaps we should say that bursting SHOULD be limited in the absence of pacing and add a note to various past maxburst limits without specifying a new one?) I'm not keen to specify a max_burst here (but I'm not sure if that's a tenable position).
I'm not advocating a max burst size of 4 (although I do believe this is safe, other sizes larger may also be safe), however my thought was that the draft infers IW as a safe size to burst - I don't know that this is always true - especially if we look to the future use of a larger IW. You could (I think you do) argue that IW is an upper bound to this safe burst size without pacing. I will try just a little more: • I think we should limit the cwnd when coming out of an app-limited period. • I think we should limit a burst <= some size or that pacing is needed, or if not then it MUST (!) reduce to a safe IW. Does that help?
changed the discussion in recovery from idle behavior and app limited periods to periods of under-utilized cwnd. This is great, but a precise definition of when the cwnd goes under-utilized is still missing.
Defining "under-utilised cwnd", seems conceptually the right thing, but we struggled with that term and the term "fully utilized the congestion window" when writing RFC7661, because ACKs can arrive in bursts and apps can do their work in bursts: Does that mean it's more than some threshold for some period time?
NAME : That is a question. Maybe we can get away from that by saying that a sender only increases its cwnd if it knows that it was fully utilized since the last cwnd change? Meaning that at some point since the last cwnd change, inflight >= cwnd happened at least once. I have a separate question, one that was being discussed in , and I think is still left open. If we need the entire cwnd to be used before increasing it, we lose out on 1 RTT of growth during slow start. Both Windows and Linux TCP compensate for this by doing something like increase it when > 1/2 cwnd is used... we need to write something here that's at least as performant, and not just "when inflight >= cwnd".
NAME thoughts?
This seems to be digging in the correct place. I think we would agree the aim of some form of cwnd-validation is to stop cwnd growth when there is no growth of inflight data by the upper layer. And I think we'd agree tussle comes when the CC is trying to expand the cwnd - because the upper layer has more data to send, but that doesn't mean cwnd was necessarily fully utilised last RTT. When would in-flight > cwnd ?
NAME Good point about the in_flight > 1/2 CWND check in Linux. In BBR, that only applies to the first doubling of IW, and after that it's fully bandwidth based, FWIW. We could add a similar constraint that this rule can be applied when the window was not fully used at the time of sending, but was at the time of acknowledgement as long as the window is < 2*IW? One clear goal of mine is I don't want the handshake alone increasing the congestion window(assuming IW10 and a moderately sized handshake).
Putting this discussion into a concrete algorithm: -Keep an in-flight high watermark, InFlightHigh, that's reset on each CWND change. -When considering growing the CWND, only let it go as large as InFlightHigh2. -Justification for the "2": it allows base-2 exponential growth, which is the fastest growth we ever want. EDIT: expire on CWND change might not work. I think InFlightHigh expiry is the hard part here.
Possible expiration events: persistent congestion, ssthresh changes, app idle.
Discussed in London, we'd like to avoid specifying this in pseudocode and the exact math, as it may be implementation dependent, but instead focus on improving the existing text.
Currently the recovery draft 19 requires the sender to either reducing the cwnd to initial cwnd OR pace the send after idle. Idle is defined as "it ceases to send data and has no bytes in flight" . This will mean that just a very small delay in the sending application thread will cause QUIC to reset its cwnd when not pacing. IMO we need to define an interval for this which could be a function of kGranularity or as PTO. FWIW in TCP the idle period for cwnd restart is defined as one retransmission timeout (reference URL Section 4.1).
However, this still has the problem that if I don't pace and I have a window of 100 packets, I can burst 100 packets into the network. This seems suboptimal from both the sender and the network's perspective. Now that I think about it, I think the concept of limiting the size of a burst is the key here, not so much idleness, and I think defining idle more accurately is both not necessary and likely to be difficult. Ideally you would send no more than max(10, BW * kTimerGranularity) packets into the network at once.
NAME pacing or limiting burst is the right idea. TCP RFC limits the burst to IW but it says do that after an idle period of an RTO. Presumably the authors and community considered that period to be small enough that network capacity / conditions hadn't changed drastically. Implementation wise how would we do this without a timer? If not pacing, when would one decide to cut cwnd back to restart cwnd (say IW)?
Your interpretation of RFC5661 looks correct and is different from what's in the QUIC recovery doc. But the implications are that it's ok to send an unlimited sized burst into the network as long as you only drained the pipe very recently. I think that recommendation is a poor one unless the window sizes are fairly small(in which case they'd be close to IW10 anyway). The sender is bursting packets into the network, which previously did not cause loss because they were ack-clocked, but one or more buffers on the network doesn't have enough space for the full CWND. It seems likely this could cause losses for other flows sharing the link as well. This could happen frequently in HTTP applications such as DASH video playback. NewCWV is already referenced for decaying CWND after idle. I could change the text to say nothing about reducing CWND to IW upon full idle and instead say you shouldn't burst more than IW into the network at once ever, and senders may use pacing, ACK-clocking, and/or timers to accomplish that? URL may also be relevant to this topic?
I guess the bigger question is what are we attempting to document in this draft. Is it whats in the TCP standard RFCs OR what most current TCP implementations do OR what the Linux kernel implementation does? By documenting NewReno instead of CUBIC we've chosen to not document the state-of-the-art and erred on the side of simplicity. In the spirit of that I think handling idle period and app limited states are optimizations for better network citizenship on cetrtain type of networks and not a hard requirement. For this particular issue I'd like to point out that the Windows TCP implementation has worked fine without handling the idle period (not capping cwnd burst and neither pacing). The idle cwnd restart knob in Windows is off by defualt and I believe Google TCP turns it off on Linux as well. Yes RFC 7661 is relevant and I think I cited it on the application limited case handling issue. I think there is a fine balance between managing burst size and taking advantage of TSO like optimizations. For example in WAN and DC bulk flows it might better to burst higher than IW with TSO for CPU efficiency than to pace and on the Internet pacing might be better. But I like your direction on this since it sidesteps what is meant by "idle". How about framing it as a suggestion and not a requirement. Something along the lines of: Bursts of more than IW into the network at once may cause losses. Implemementations should consider using pacing, ACK-clocking, and/or timers to minimize bursts. Paging NAME to this issue as well.
My goal is to document the best practices and reference existing RFCs whenever possible. And yes, my understanding is that everyone disables slow start after idle, so I'd like to come up with something I think is more likely to be deployed. When QUIC was started, Cubic wasn't an RFC, which is the other reason we documented Reno instead of Cubic, FYI. I'll write up a PR based on your suggestion and hopefully we can get some text we all think is acceptable.
RFC7661 is already cited in the section, and it was added there to defer to it so we didn't have to delve into details in this doc. I think what we're really discussing here is whether we need the MUST in the sentence "When sending data after becoming idle, a sender MUST reset its congestion window to the initial congestion window". I'm in agreement with what I'm sensing here -- that we don't need a requirement, but we can discuss the implications of various decisions here and current best practices. Higher-level note, yeah, we always wanted to document current best practices, rather than what was in TCP or even linux TCP. I think most things came from linux TCP because that was what we were familiar with. FWIW, agreed that Cubic wasn't an RFC at the time, but it was close to becoming one, and we did discuss it. We decided that it was too complex and adds a ton more complexity to this document. We wanted to have something simple but good enough that implementations could have as a starting point, and then move on to more esoteric algorithms as deployments needed. (I'll disagree that Cubic is the best known CC at the moment -- unless you're talking about DC environments, it's not.)
Thanks for the discussion, I updated based on this, though I'm a bit concerned that SHOULD is too weak in regards to bursts into the network. Then again, if that's what Windows is doing, maybe we have to accept the reality of current deployments?
Given that I think that the general guidance for QUIC is "use pacing", I think a "MUST reset to IW if not paced" is actually the right thing to do to support that pacing gets implemented with QUIC from the start.
NAME pacing may be implemented and controlled outside of the transport. I have two new concerns about text around bursting packets on network: Many CDNs and edge deployments will tweak IW to be a higher value. In Windows TCP we allow config of up to 64 MSS. So if the network admin knows what they are doing and can swallow bursts then we shouldn't discourage the practice. UDP segmentation offload is reality now and for high throughput flows using hardware offload to reduce CPU will mean larger bursts trading off against pacing. We shouldn't preclude this either.
Discussed in Montreal; people to review the PR.
Discussed in Cupertino, NAME to review and determine whether that addresses (or partially addresses) this issue.
NAME NAME This issue was supposed to be but it was waiting for feedback. It's been long enough -- can we close this issue?

6.9.  <ins> When bytes in flight is smaller than the congestion window and sending is not pacing limited, the congestion window is under- utilized.  When this occurs, the congestion window SHOULD NOT be increased in either slow start or congestion avoidance.  This can happen due to insufficient application data or flow control credit. </ins> A sender MAY use the pipeACK method described in section 4.3 of RFC7661 to determine if the congestion window is sufficiently sender should not consider itself application limited if it would have fully utilized the congestion window without pacing delay.  <ins> Sending multiple packets into the network without any delay between them creates a packet burst that might cause short-term congestion and losses.  Implementations SHOULD either use pacing or reduce their congestion window to limit such bursts to minimum of 10 * kMaxDatagramSize and max(2* kMaxDatagramSize, 14720)), the same as the recommended initial congestion window. </ins> A sender MAY implement alternate mechanisms to update its congestion window after periods of under-utilization, such as those proposed for


endpoint's RTT samples, and rttvar is the endpoint's estimated variance in the RTT samples. <del> smoothed_rtt uses path latency after adjusting RTT samples for peer- reported host delays (host-delay).  A peer limits any delay in </del>  sending an acknowledgement for an ack-eliciting packet to no greater <del> than the advertised max_ack_delay transport parameter.  Consequently, when a peer reports an Ack Delay that is greater than its max_ack_delay, the delay is attributed to reasons out of the peer's control, such as scheduler latency at the peer or loss of previous ACK frames.  Any delays beyond the peer's max_ack_delay are therefore considered effectively part of path delay and incorporated into the smoothed_rtt estimate. </del>  When adjusting an RTT sample using peer-reported acknowledgement delays, an endpoint:  MUST use the lesser of the value reported in Ack Delay field of the ACK frame and the peer's max_ack_delay transport parameter (host-delay).

This should have been part of .
NAME I applied your suggestions. I think this PR should be marked as editorial.
The new "MUST" threw me, but it's not really new, is it?
That one was introduced in already.
Seems reasonable to me. Throughout here, should we be using "peer" or "receiver"? (The suggestions I make here are not issues with this change, but I'm sure that NAME or NAME will find them a home.)Thanks for fixing this.

endpoint's RTT samples, and rttvar is the endpoint's estimated variance in the RTT samples.  <ins> The calculation of smoothed_rtt uses path latency after adjusting RTT samples for host delays (host-delay).  For packets sent in the ApplicationData packet number space, a peer limits any delay in </ins> sending an acknowledgement for an ack-eliciting packet to no greater  <ins> than the value it advertised in the max_ack_delay transport parameter.  Consequently, when a peer reports an Ack Delay that is greater than its max_ack_delay, the delay is attributed to reasons out of the peer's control, such as scheduler latency at the peer or loss of previous ACK frames.  Any delays beyond the peer's max_ack_delay are therefore considered effectively part of path delay and incorporated into the smoothed_rtt estimate. </ins> When adjusting an RTT sample using peer-reported acknowledgement delays, an endpoint: <ins> MUST ignore the Ack Delay field of the ACK frame for packets sent in the Initial and Handshake packet number space. </ins> MUST use the lesser of the value reported in Ack Delay field of the ACK frame and the peer's max_ack_delay transport parameter (host-delay).


A variable-length integer encoding of the Stream ID of the stream being terminated. <del> A 16-bit application protocol error code (see app-error-codes) which indicates why the stream is being closed. </del>  A variable-length integer indicating the final size of the stream by the RESET_STREAM sender, in unit of bytes. A variable-length integer carrying the Stream ID of the stream being ignored. <del> A 16-bit, application-specified reason the sender is ignoring the stream (see app-error-codes). </del>  19.6. CONNECTION_CLOSE frames contain the following fields: <del> A 16-bit error code which indicates the reason for closing this connection.  A CONNECTION_CLOSE frame of type 0x1c uses codes from the space defined in error-codes.  A CONNECTION_CLOSE frame of type 0x1d uses codes from the application protocol error code space; see </del>  A variable-length integer encoding the type of frame that triggered the error.  A value of 0 (equivalent to the mention of 20. <del> QUIC error codes are 16-bit unsigned integers. </del>  This section lists the defined QUIC transport error codes that may be used in a CONNECTION_CLOSE frame.  These errors apply to the entire 20.1. <del> Application protocol error codes are 16-bit unsigned integers, but </del>  the management of application error codes are left to application protocols.  Application protocol error codes are used for the RESET_STREAM frame (frame-reset-stream), the STOP_SENDING frame IANA [SHALL add/has added] a registry for "QUIC Transport Error Codes" under a "QUIC Protocol" heading. <del> The "QUIC Transport Error Codes" registry governs a 16-bit space. This space is split into two spaces that are governed by different policies.  Values with the first byte in the range 0x00 to 0xfe (in hexadecimal) are assigned via the Specification Required policy RFC8126.  Values with the first byte 0xff are reserved for Private Use RFC8126. </del>  Registrations MUST include the following fields: The numeric value of the assignment (registrations will be between <del> 0x0000 and 0xfeff). </del>  A short mnemonic for the parameter. A reference to a publicly available specification for the value.  The initial contents of this registry are shown in iana-error-table. <del> Values from 0xFF00 to 0xFFFF are reserved for Private Use RFC8126. </del>

Changes CONNECTIONCLOSE, STOPSENDING, and RESET_STREAM to varint error codes. Also fixes an error where the spec said the transport parameter registry is 16 bits, but transport parameters are now varints as well.
For consistency you probably want to update HTTP too, e.g. there are some references to 16-bit error code there too URL
Thanks for pointing that out NAME http updated.
Thanks Mike, I think I fixed all your suggestions.
Currently there are three spots in the QUIC transport spec which used fixed length(16-bit) integers: Connection Close error code Application Protocol Error Code in Reset Stream Application Error Code in Stop Sending This typically makes reset stream and stop sending frames 1 byte larger than they need to be, which could be slightly annoying if one is churning through a huge number of streams. It creates some unnecessary oddness in the HTTP/3 spec, where the HTTPMALFORMEDFRAME(0x01XX) is describing a frame type that's a varint. Admittedly, there would still be a limit of 2^60 being expressible, but it's easy to avoid that. I think all 3 fields would be better as a varint, given our recent decisions to make HTTP/3 all varints.
I would support a move to varints. I've been wanting to support more than 16-bit application error codes, for non-http protocols.
We discussed this in , and closed it because nobody felt strongly either way, if I remember correctly.
I think preferences may have changed, and NAME did have a preference for varints here anyways. Since nobody had a strong preference either way in , it's probably worth having people voice their opinions if they now believe more strongly than they did before.
I'm of the opinion that varint is OK. We can crib text from other varint usages about IANA registries and such, so the change is cheap for the spec perspective. The HTTPMALFORMEDFRAME thing would need a solution. I've never been a big fan of the concept, but anything would do (apart from the 2^60 thing, which would take 1/4 of the entire space for bad frames, which is excessive; I hope that's just an error).
I also think that varint is OK. It gives application protocols to use up to 62-bit space for error codes rather than enforcing them to use 16-bit space. The downside is that the CONNECTIONCLOSE frame could become one byte bigger, but I do not think that is a real cost. Re HTTPMALFORMEDFRAME, my preference goes to either a) recommend endpoints expressing the offending frame type in the reason phrase, or b) change the definition of application-close frame (i.e. CONNECTIONCLOSE with type=0x1d) to carry a blob, and define in the HTTP/3 spec. that the blob would be a tuple of error-code (varint), offending-frame-type (varint), and a reason phrase1. My view is that 2^60 is a dirty hack. It works, but I'd argue that it is a sign that the HTTP draft doing something unexpected by the transport design. Therefore, I'm arguing to do what is expected by the transport, or change the design of the transport to meet the requirement of HTTP. 1: If we are to allow the application-close frame to carry a blob, the HTTP draft can also consider adding max-processed-stream-id to the blob, so that the client can replay the requests that have not yet been processed even in case of a protocol violation - a feature we have had in HTTP/2.
For HTTPMALFORMEDFRAME, I think NAME PR that removes it and replaces it with a small number of specific error codes is like a good way to go instead of the 2^60 hack. NAME Are you happy with that PR? I'm going to write a varint PR with the assumption or something like it is likely to land, so it will be transport-only.
FWIW I just fixed the merge conflict on
NAME works for me. My complaint is that HTTP/3 should not "hack" QUIC. Eliminating HTTPMALFORMEDFRAME is a way of achieving that, and I do not have a strong preference on communicating the malformed frame ID.
Thank you for the work!Thanks for adding the registry text. LGTM.LGTM

A variable-length integer encoding of the Stream ID of the stream being terminated.  <ins> A variable-length integer containing the application protocol error code (see app-error-codes) which indicates why the stream is being closed. </ins> A variable-length integer indicating the final size of the stream by the RESET_STREAM sender, in unit of bytes. A variable-length integer carrying the Stream ID of the stream being ignored.  <ins> A variable-length integer containing the application-specified reason the sender is ignoring the stream (see app-error-codes). </ins> 19.6. CONNECTION_CLOSE frames contain the following fields:  <ins> A variable length integer error code which indicates the reason for closing this connection.  A CONNECTION_CLOSE frame of type 0x1c uses codes from the space defined in error-codes.  A CONNECTION_CLOSE frame of type 0x1d uses codes from the application protocol error code space; see </ins> A variable-length integer encoding the type of frame that triggered the error.  A value of 0 (equivalent to the mention of 20.  <ins> QUIC error codes are 62-bit unsigned integers. </ins> This section lists the defined QUIC transport error codes that may be used in a CONNECTION_CLOSE frame.  These errors apply to the entire 20.1.  <ins> Application protocol error codes are 62-bit unsigned integers, but </ins> the management of application error codes are left to application protocols.  Application protocol error codes are used for the RESET_STREAM frame (frame-reset-stream), the STOP_SENDING frame IANA [SHALL add/has added] a registry for "QUIC Transport Error Codes" under a "QUIC Protocol" heading.  <ins> The "QUIC Transport Error Codes" registry governs a 62-bit space. This space is split into three spaces that are governed by different policies.  Values between 0x00 and 0x3f (in hexadecimal) are assigned via the Standards Action or IESG Review policies RFC8126.  Values from 0x40 to 0x3fff operate on the Specification Required policy RFC8126.  All other values are assigned to Private Use RFC8126. </ins> Registrations MUST include the following fields: The numeric value of the assignment (registrations will be between  <ins> 0x0000 and 0x3fff). </ins> A short mnemonic for the parameter. A reference to a publicly available specification for the value. <ins> The nominated expert(s) verify that a specification exists and is readily accessible.  Expert(s) are encouraged to be biased towards approving registrations unless they are abusive, frivolous, or actively harmful (not merely aesthetically displeasing, or architecturally dubious). </ins> The initial contents of this registry are shown in iana-error-table.


changes, the initial RTT SHOULD be set to 500ms, resulting in a 1 second initial handshake timeout as recommended in RFC6298.  When a crypto packet is sent, the sender MUST set a timer for twice the smoothed RTT.  This timer MUST be updated when a new crypto packet is sent and when an acknowledgement is received which computes

Using PATHCHALLENGE/PATHRESPONSE I'm fairly happy with the text, but not that happy with the location. Especially after the recently filed issues(ie: ) on resetting the congestion controller on path changes, I think we may need a section in recovery on that.
Do we really want to keep the requirement to respond with a PATH_RESPONSE immediately? This doesn't talk about paths. Now, we might infer from this that the RTT to a peer isn't likely to dramatically change when paths change, but we'd want to be saying that directly. How does this sit with the change in ?
On 2.: if you roam from 5G to 3G you could see significant latency spikes. Phasing out 3G networks and keeping 2G could move traffic to 2G EDGE in areas of thin coverage. Traffic could also move to narrow band IoT (if it is possible to run QUIC on these at all) with MUCH higher latency. Endpoints could choose stay on narrow band most of the time to save battery and perhaps online costs, but use 5G when a device wakes up for real (I could see a use case for that). Though of course might be served with separate connections for such extreme transitions.
I think this is OK as long as we limit how many path responses we send immediately? Realistically, it's not that different from a generate_204, but it's something to be aware of. I think we need to add a whole section on paths and connection migration in recovery. See I can do that in this PR if people want? decreases the value of this, since you may only see one half of the path. It might still be better than nothing? Initial RTT doesn't need to be correct, but getting it within a factor of 2 is helpful.
NAME I don't think you meant 2367
Thanks David, fixed.
I'm split on this. While I understand the urge to optimize here, I'm not sure that documenting it here is necessary. Saying that an endpoint MAY use this signal gives permission, but does so without noting the potential issues. At a minimum, you would need to talk about the problems that are lurking ... especially in light of , as I've noted in . Which leads me to wondering if it's really worth documenting this here without more experimentation and data.
I'm a fan of forbidding it's use as an actual RTT measurement, which would be one reason to say something?
I'd prefer that it's NOT forbidden. I'm fine with waiting for experimentation.
I changed this to SHOULD NOT from MUST NOT to allow some experimentation.
Section 9.4 contains a mixture of transport (handling migration, setup, etc) and congestion control functions (choice of path parameters - RTO, window, etc). The abstract suggests that these are a part of QUIC Recovery ID, which seems correct to me. So, I would like to discuss if these are intentionally a part of the transport spec, rather than the Loss Detection & Congestion Control Spec.? Specifically, para 1,2 : "... SHOULD NOT contribute to congestion control or RTT estimation for the new path." and "... SHOULD immediately reset ...." - Where the ID could note whatever transport implementors need to be aware of, but point normatively to the QUIC Recovery ID for RFC2119 language. Para 3: "An endpoint MUST NOT return to the send rate..." seems a reasonable thing to note, but also, I think, should point to QUIC Recovery ID.
Discussed in London: NAME to work with NAME to propose an editorial PR
The target is to fix this issue with the same PR as
Discussed today with editors and we believe that these issues have been rectified in the numerous edits that section has received since this issue was opened.
Should the recovery draft explain how to gather RTT samples during connection migration using PATHCHALLENGE and PATHRESPONSE? This is explained in: URL
Is there any expectation that the response is sent immediately?
Section 8.4:
If PR lands, we'd need to restrict this to cases when the PATHRESPONSE arrived on the same path the PATHCHALLENGE was sent on.
NAME don't we already have text about that?
NAME removes that text. NAME is right, we'll need to caveat this at a minimum. I think it gets a bit tricky to use the PATH_CHALLENGE/RESPONSE for RTT if the RESPONSE can arrive on any path. The endpoint does not know what path will be used for data/acks, and it would have to assume that the path for CHALLENGE/RESPONSE will be the one used for data/acks, if it is to set its RTT estimate. Perhaps we could say that an endpoint can use the CHALLENGE/RESPONSE RTT as a hint. This will need some more thought.
How about "MAY use the delay between sending a PATHCHALLENGE and receiving a PATHRESPONSE to seed initial_rtt for a new path, but MUST not be considered an RTT sample."?
Discussed with Ian in London: I'm okay with just using the PATHCHALLENGE/PATHRESPONSE RTT as the initial RTT.
Thanks NAME can you review the outstanding PR?

changes, the initial RTT SHOULD be set to 500ms, resulting in a 1 second initial handshake timeout as recommended in RFC6298. <ins> A connection MAY use the delay between sending a PATH_CHALLENGE and receiving a PATH_RESPONSE to seed initial_rtt for a new path, but the delay SHOULD NOT be considered an RTT sample. </ins> When a crypto packet is sent, the sender MUST set a timer for twice the smoothed RTT.  This timer MUST be updated when a new crypto packet is sent and when an acknowledgement is received which computes


use an CONNECTION_CLOSE frame with an appropriate error code to signal closure. <del> If the connection has been successfully established, endpoints MUST send any CONNECTION_CLOSE frames in a 1-RTT packet.  Prior to connection establishment a peer might not have 1-RTT keys, so endpoints SHOULD send CONNECTION_CLOSE frames in a Handshake packet. If the endpoint does not have Handshake keys, or it is not certain that the peer has Handshake keys, it MAY send CONNECTION_CLOSE frames in an Initial packet.  If multiple packets are sent, they can be coalesced (see packet-coalesce) to facilitate retransmission. </del>  10.4.

Clarifies that if the server wants the client to process a connection close, it SHOULD send one in Initial and Handshake. It seemed odd to me to say MAY send a CONNECTION_CLOSE in Initial, so I changed that to a SHOULD. I can revert that and the other SHOULD, but I think the current PR makes more sense. Came up in discussion on and I believe may be necessary to fully address . If not, I can open another issue.
We might just delete this tomorrow, but this is strictly better than what we have now.
Isn't the rule here that you always send it in the highest level?
The intent is for the peer to be able to read the packet, thus the recommendation to use a lower level if you aren't sure that they have the right keys. If they have the right keys, then they will ignore the Initial packet, if they don't then it's better to signal the error than have them keep trying. I think that this is OK. Do you have a specific concern with this design? Or the specific text?
I'm not sure. It seems like there are two policies: Always use level N until you are sure that the other side has level N+1 Always use the highest level. I take you to be proposing the first of these, which, I believe, means "use the highest level where you have received packets from the peer in (or zero)". This text may say that, but all this stuff about certain is kind of hard to parse. Rather than making people reason it out, I think we should provide them with a clear rule (for instance the one above).
An endpoint could logically migrate to the next packet number space / encryption level for new transmissions while still keeping the previous level for retransmissions. If the connection dies before retransmission the peer might not be able to read latest crypto level even if this is where you normally send new data. It is easier and safer to send CLOSE on both channels rather than retransmitting the handshake history.
I will await a PR then.
I had to fix a bug in which Picoquic was sending a connection close in handshake while the peer would not have the handshake key when receiving the message... URL
Tokyo conclusion: the principle here is that we want to use the highest level we know the peer to be able to read. Reassign to editorial to a) include this principle, and b) make the text clearer.
NAME NAME : This issue is marked editorial, but it's borderline design. The PR that fixes it () introduces 2119 language and behavior that is new. I'll remove the editorial tag, but I'm not marking it as design. Can you advise on what to do next?
Will triage as design. Thanks.
NAME ?
Yes? The proposal is to close with no action on the basis of text already incorporated in .
There wasn't a consensus call. Will do one next round.
Fixed by .
LGTM for the most part. A few minor suggestions.

use an CONNECTION_CLOSE frame with an appropriate error code to signal closure.  <ins> When sending CONNECTION_CLOSE, the goal is to ensure that the peer will process the frame.  Generally, this means sending the frame in a packet with the highest level of packet protection to avoid the packet being discarded.  However, during the handshake, it is possible that more advanced packet protection keys are not available to the peer, so the frame MAY be replicated in a packet that uses a lower packet protection level. After the handshake is confirmed, an endpoint MUST send any CONNECTION_CLOSE frames in a 1-RTT packet.  Prior to handshake confirmation, the peer might not have 1-RTT keys, so the endpoint SHOULD send CONNECTION_CLOSE frames in a Handshake packet.  If the endpoint does not have Handshake keys, it SHOULD send CONNECTION_CLOSE frames in an Initial packet. A client will always know whether the server has Handshake keys (see discard-initial), but it is possible that a server does not know whether the client has Handshake keys.  Under these circumstances, a server SHOULD send a CONNECTION_CLOSE frame in both Handshake and Initial packets to ensure that at least one of them is processable by the client.  These packets can be coalesced into a single UDP datagram (see packet-coalesce). </ins> 10.4.


value provided by its peer. Definitions for each of the defined transport parameters are included <del> in transport-parameter-definitions.  An endpoint MUST treat receipt of a transport parameter with an invalid value as a connection error of type TRANSPORT_PARAMETER_ERROR.  Any given parameter MUST appear at most once in a given transport parameters extension.  An endpoint MUST treat receipt of duplicate transport parameters as a connection error of type TRANSPORT_PARAMETER_ERROR. </del>  A server MUST include the original_connection_id transport parameter (transport-parameter-definitions) if it sent a Retry packet to enable

Changes a receiver requirement to, in , "senders: don't do that; receivers: you're not required to police but we'd really like it if you did."
I would prefer a middle ground, where I only MUST check for TPs I support. I can manage that, but tracking 65K items for duplicates is a bit much
Actually processing duplicates read to unpredictable behavior
I like this PR as currently worded. Checking should be encouraged but not mandated, even for known transport parameters.
The spin here is interesting. An endpoint should be able to track supported transport parameters. Also remember that once we start to approach 300 transport parameters, there is no more room in the client's Initial packet for other stuff (like keys). The server parameters don't have the same constraint, but you might reasonably assume that the same limit is coming.
I think I'm fine with what the PR says now. Though I'm also fine to see something like "the receiver SHOULD treat duplicates of TP that the endpoint supports as a connection error" being added. The issue with "MAY validate" in general is that it is sometimes hard for smaller players to enforce, because they'd be risking interoperability unless other players also enforce the behavior to the extent that the endpoints violating the requirement would never be shipped.
Perhaps a middleground could be: endpoints SHOULD validate (instead of the current MAY)? I like either as long as it's not a MUST validate.
the only thing I don't like about SHOULD validate for supported parameters is that if there are two copies the actual behavior is somewhat ambiguous. If I get two values for the same TP, which should I observe? Much cleaner to demand enforcement for supported TPs, IMO.
I could live in a world that requires MUST validation of the small set of IANA registered non-GREASE transport parameters (and tangentially HTTP/3 SETTINGS parameters). I don't want to "MUST validate" a humungous set of parameters.
If it's MUST not send and SHOULD validate, then I believe there's no need to define what happens if multiple are sent, because it's explicitly forbidden.
Ok, I've changed it to SHOULD error if dups are received, and I've explicitly called out MUST NOT send.
Although I'd prefer a MUST for the non-proprietary TPs, I can live with the current PR.
Unlike HTTP/3 settings, receipt of : Why the inconsistency? The checks for duplicated transport (or settings) parameters are expensive, because one has to maintain an arbitrarily-sized set of possibly large numbers. I propose we switch the duplicate transport parameters enforcement to "MAY treat as error."
I'm fine with MUST not send duplicates, MAY treat receipt as an error.
Fine by me.
Yup, seems sensible to me.

value provided by its peer. Definitions for each of the defined transport parameters are included  <ins> in transport-parameter-definitions. An endpoint MUST treat receipt of a transport parameter with an invalid value as a connection error of type TRANSPORT_PARAMETER_ERROR. An endpoint MUST NOT send a parameter more than once in a given transport parameters extension.  An endpoint SHOULD treat receipt of duplicate transport parameters as a connection error of type TRANSPORT_PARAMETER_ERROR. </ins> A server MUST include the original_connection_id transport parameter (transport-parameter-definitions) if it sent a Retry packet to enable