The sequence number specified in a RETIRE_CONNECTION_ID frame MUST NOT refer to the Destination Connection ID field of the packet in which the frame is contained.  The peer MAY treat this as a <del> connection error of type FRAME_ENCODING_ERROR. </del>  An endpoint cannot send this frame if it was provided with a zero- length connection ID by its peer.  An endpoint that provides a zero-

This seems like another instance of deciding the scope of "context." Is the packet that the frame arrived in part of the connection context, or is it part of looking at the bytes?
NAME Even if the packet header is part of the context, that still doesn't help. The RETIRECONNECTIONID frame only carries the sequence number, not the actual connection ID. You have to access connection-level information to detect this error, which makes the FRAMEENCODINGERROR a poor choice here.
+1 to what NAME says. In addition this is a "MAY send" error, therefore I do not think we need to be cautious in trying to list all the possible error codes.
Our principle regarding error codes states that only those errors that are detectable without accessing connection state should result in a FRAMEENCODINGERROR. Since the RETIRECONNECTIONID frame contains only the sequence number of the connection ID, but not the actual connection ID itself, this should be a different error. Looking through the list of error codes, it seems like we don't have any more specific error code here, so this would be a PROTOCOL_VIOLATION.
Seems OK.
As the proposed PR indicates, this would be a design change.
Agreed, this seems OK.
As I noted in the PR, I agree that PROTOCOL_VIOLATION seems like a more appropriate error. In addition to NAME point, this is a semantic error, not one of encoding (which suggests syntactic).
It seems we have a proposed resolution, marking as such.
PR was less than a year ago, that's when this error code went from PROTOCOLVIOLATION to FRAMEENCODINGERROR_
I agree that this seems like a more appropriate error. In addition to NAME 's point, I will not that this is a semantic error, not one of encoding (which suggests syntactic).This seems like a more correct error, but I also don't have strong opinions about it.

The sequence number specified in a RETIRE_CONNECTION_ID frame MUST NOT refer to the Destination Connection ID field of the packet in which the frame is contained.  The peer MAY treat this as a  <ins> connection error of type PROTOCOL_VIOLATION. </ins> An endpoint cannot send this frame if it was provided with a zero- length connection ID by its peer.  An endpoint that provides a zero-


value.  This Destination Connection ID MUST be at least 8 bytes in length.  Until a packet is received from the server, the client MUST use the same Destination Connection ID value on all packets in this <del> connection.  This Destination Connection ID is used to determine packet protection keys for Initial packets. </del>  The client populates the Source Connection ID field with a value of its choosing and sets the Source Connection ID Length field to

This is better defined elsewhere, but it doesn't hurt to repeat it.
Initial or Retry packet from the server, the client populates the Destination Connection ID field with an unpredictable value. This Destination Connection ID MUST be at least 8 bytes in length. Until a packet is received from the server, the client MUST use the same Destination Connection ID value on all packets in this connection. This Destination Connection ID is used to determine packet protection keys for Initial packets. I had understood that you changed keys for the later initial packets when DCID changed. No?
That is correct. I guess the confusion comes from the "This" in the last sentence. We should make clear which CID it is referring to.
Depends why it changed. Retry starts a new connection attempt, with a new DCID, and new Initial keys. But the Initial keys don't change when the server's Initial provides a new SCID.

value.  This Destination Connection ID MUST be at least 8 bytes in length.  Until a packet is received from the server, the client MUST use the same Destination Connection ID value on all packets in this  <ins> connection. The Destination Connection ID field from the first Initial packet sent by a client is used to determine packet protection keys for Initial packets.  These keys change after receiving a Retry packet; see Section 5.2 of QUIC-TLS. </ins> The client populates the Source Connection ID field with a value of its choosing and sets the Source Connection ID Length field to


a client is optionally authenticated, <del> every connection produces distinct and unrelated keys, </del>  keying material is usable for packet protection for both 0-RTT <del> and 1-RTT packets, and 1-RTT keys have forward secrecy </del>  authenticated values for transport parameters of both endpoints, and confidentiality protection for server transport parameters

This claim isn't necessary, as the statement about suitability is what matters here.
S 7. Claims 1-RTT keys have forward secrecy This is only true if you only resume with PSK-(EC)DHE, but I don't see a requirement for this. So we need to either require this or change the text.
This obsession with forward secrecy is a hang-up from Google QUIC. I think that we should instead say that 1-RTT keys are derived from the completed cryptographic handshake instead.

a client is optionally authenticated,  <ins> every connection produces distinct and unrelated keys, and </ins> keying material is usable for packet protection for both 0-RTT  <ins> and 1-RTT packets </ins> authenticated values for transport parameters of both endpoints, and confidentiality protection for server transport parameters


Packets with a supported version, or no version field, are matched to a connection using the connection ID or - for packets with zero- <del> length connection IDs - the local address and port.  If the packet doesn't match an existing connection, the server continues below. </del>  If the packet is an Initial packet fully conforming with the specification, the server proceeds with the handshake (handshake).

Not saying explicitly that was a small oversight.
S 5.2.2 says: Packets with a supported version, or no version field, are matched to a connection using the connection ID or - for packets with zero-length connection IDs - the local address and port. If the packet doesn't match an existing connection, the server continues below. So, this tells us what to do if we don't match, but not if we do match. I know this is a nit, but I think there should be some kind of pointer after the first sentence.
LGTM

Packets with a supported version, or no version field, are matched to a connection using the connection ID or - for packets with zero-  <ins> length connection IDs - the local address and port.  These packets are processed using the selected connection, otherwise the server continues below. </ins> If the packet is an Initial packet fully conforming with the specification, the server proceeds with the handshake (handshake).


Supported Version. Version Negotiation packets do not use integrity or confidentiality <del> protection.  Specific QUIC versions define mechanisms to authenticate the packet as part of the connection establishment process. </del>  An endpoint MUST include the value from the Source Connection ID field of the packet it receives in the Destination Connection ID The Version Negotiation packet described in this document is not integrity-protected; it only has modest protection against insertion <del> by off-path attackers.  QUIC versions MUST define a mechanism that authenticates the values it contains. </del>  8.

This clarifies that it is a mechanism in the target QUIC version that authenticates the information. This also lifts the strong requirement on authenticating the contents of Version Negotiation. QUIC version 1 does not do this and so does not comply with this requirement. The intent was always that only information that is used needs to be authenticated in this way. (This is a tiny bit of evasion as Version Negotiation is used in QUIC version 1 as a signal that a version is not supported, which results in clients abandoning connections.)
This is for -invariants-09: Section 6 says: Version Negotiation packets do not use integrity or confidentiality protection. A specific QUIC version might authenticate the packet as part of its connection establishment process. Is the second not contradicting what is said in previous paragraph: A Version Negotiation packet contains no other fields. So to my understanding the authentication that the paragraph above talks about needs to be done in another QUIC packet, it can't be done in a QUIC packet that is Version Negotiation. Can this be made more explicit or do I misunderstand and can you then clarify why this is not a contradiction?
I don't see this as contradictory. The sentence here: Implies that the version fields might be authenticated. Indeed, the document says that they MUST be authenticated (somehow). It doesn't say how. However, this ... means that it can't use any fields that are included in a Version Negotiation packet to do that. If you look at the current proposal for , that uses transport parameters.
Ok, then I would request a very minor tweak in this sentence: A specific QUIC version might authenticate the packet as part of its connection establishment process. So that it is explicit that is in another packet(s) than the Version negotiation packet. For example: A specific QUIC version might authenticate the version negotiation packet as part of its connection establishment process outside of this packet.
I have made a slightly larger tweak.

Supported Version. Version Negotiation packets do not use integrity or confidentiality  <ins> protection.  Specific QUIC versions might include protocol elements that allow endpoints to detect modification or corruption in the set of supported versions. </ins> An endpoint MUST include the value from the Source Connection ID field of the packet it receives in the Destination Connection ID The Version Negotiation packet described in this document is not integrity-protected; it only has modest protection against insertion  <ins> by off-path attackers.  An endpoint MUST authenticate the contents of a Version Negotiation packet if it attempts a different QUIC version as a result. </ins> 8.


As keys at a given encryption level become available to TLS, TLS indicates to QUIC that reading or writing keys at that encryption <del> level are available.  These events are not asynchronous; they always occur immediately after TLS is provided with new handshake bytes, or after TLS produces handshake bytes. </del>  TLS provides QUIC with three items as a new encryption level becomes available: 5. As with TLS over TCP, QUIC protects packets with keys derived from <del> the TLS handshake, using the AEAD algorithm negotiated by TLS. </del>  5.1. separation between QUIC and TLS; see key-diversity. The KDF used for initial secrets is always the HKDF-Expand-Label <del> function from TLS 1.3 (see initial-secrets). </del>  5.2. <del> Initial packets are protected with a secret derived from the Destination Connection ID field from the client's Initial packet. Specifically: </del>  <del> The hash function for HKDF when deriving initial secrets and keys is SHA-256 SHA. </del>  The connection ID used with HKDF-Expand-Label is the Destination Connection ID in the Initial packet sent by the client.  This will be packet after receiving a Retry packet, where the Destination Connection ID is selected by the server. <del> The value of initial_salt is a 20-byte sequence shown in the figure in hexadecimal notation.  Future versions of QUIC SHOULD generate a new salt value, thus ensuring that the keys are different for each version of QUIC.  This prevents a middlebox that only recognizes one version of QUIC from seeing or modifying the contents of packets from future versions. </del>  The HKDF-Expand-Label function defined in TLS 1.3 MUST be used for Initial packets even where the TLS versions offered do not include TLS 1.3. <del> The secrets used for protecting Initial packets change when a server sends a Retry packet to use the connection ID value selected by the server.  The secrets do not change when a client changes the </del>  Destination Connection ID it uses in response to an Initial packet from the server. packet; the client has to rely on the exchange that included the Retry packet for that property. <del> test-vectors contains test vectors for packet encryption. </del>  5.3. using the TLS_AES_128_GCM_SHA256 cipher suite, the AEAD_AES_128_GCM function is used. <del> Packets are protected prior to applying header protection (header- protect).  The unprotected packet header is part of the associated data (A).  When removing packet protection, an endpoint first removes the header protection. All QUIC packets other than Version Negotiation are protected with an AEAD algorithm (AEAD).  Prior to establishing a shared secret, packets are protected with AEAD_AES_128_GCM.  Retry packets use the AEAD for integrity protection only, as described in retry-integrity. Initial packets use a key derived from the Destination Connection ID in the client's first Initial packet; see initial-secrets.  This provides protection against off-path attackers and robustness against QUIC-version-unaware middleboxes, but not against on-path attackers. </del> QUIC can use any of the cipher suites defined in TLS13 with the exception of TLS_AES_128_CCM_8_SHA256.  A cipher suite MUST NOT be negotiated unless a header protection scheme is defined for the deploy a new cipher suite.  This also applies to TLS_AES_128_CCM_8_SHA256.  The key and IV for the packet are computed as described in protection-keys.  The nonce, N, is formed by combining the packet protection IV with the packet number.  The 62 bits of the

As pointed out in , the drafts assume that the traffic keys are generated synchronously. But that's not what we see in the wild. In fact, clients often need to spend time while verifying the certificate, and some if not most of the clients do receive and send QUIC packets while they do so. This PR acknowledges the existence of such implementation, recommending them to buffer the undecryptable packets while they calculate the next key, to prevent loss events that would kill the slow start. It also allows endpoints to postpone arming the PTO timer when it knows that the peer is going to send a handshake message.
NAME Thank you for your comments. I think the two new commits address your concerns. PTAL.
NAME Thank you for your help in moving this PR forward. I'm a bit hesitant to call this PR editorial though, previously endpoints were expected to process TLS handshake messages synchronously, now we have a MAY that allows them to process them in an asynchronous manner (while waiting for TLS to provide new keys).
To make the change of status clear, the change to probing keeps probes for application data disabled until handshake confirmation rather than handshake completion. This is based on the conclusions in . I think that PR is an incremental change in that direction, but more is needed to complete the set of changes proposed there. My suggestion would be to get agreement on the overall strategy, as proposed. Then we can take this PR and follow up with more comprehensive/systemic fixes.
The text here (aside from needing a rebase) is OK for TLS. It's the asynchronous certificate validation stuff that bothers me and that can be done as the next step (a step for which it would be good to have this PR merged).
Let's not duplicate existing requirements on ACK behavior in recovery, given we decided to move it to transport and that text is still getting cleaned up.
When reported ACK Delay would cause the RTT sample to go below minrtt, we ignore the ACK Delay. In general, I think that this provides the right incentives to report delays accurately. If the true minimum RTT is lower than the observed value, you add a few samples that are inflated by up to maxackdelay. URL the handshake, where we don't have an effective cap on ACK Delay, so we effectively have unbound inflation. And minrtt is likely not well established. Should we discard these samples instead?
I think this is a potential issue, but I'd like to see this addressed in with some text to advise this is a potential issue that MAY be solved by ignoring RTT samples.
Happy to have this rolled into as long as we can do so expediently. I don't want a design issue floating around after this next batch of consensus calls. NAME do you think that something could be done?
I tend to think that if we are going to have some advice, it would be something like "MAY ignore the first ACK sample for Handshake and ApplicationData packet number spaces." IIRC, NAME suggested something like that somewhere. Though, if we are going to have something like that, I think we also need to state that using a bad RTT sample might still better than relying on initial RTT of 333ms.
I've added some text in (lines 351 - 356) that should capture this point.
Thanks NAME that new text works for me.
I'm splitting this from () because the set of changes are disjoint from the current PR for that issue and I think having both issues in one is making it harder to follow. (The rest of this text is an edit of my last post on that issue). The issue is that ACKs for Handshake and 1-RTT packets could be delayed because the peer doesn't have the keys(either because it takes time to derive them, or it's missing crypto data to derive the next key). The first RTT sample in Handshake and 1-RTT packet spaces could have substantial delay in excess of the 'no ack delay' in Handshake or maxackdelay for 1-RTT, which inflates SRTT and RTTVar. If there's no loss, this fixes itself without harm, which is nice. And it's not as bad for ApplicationData, since that will have a non-0 maxackdelay, which may be enough to absorb the endpoint delay generating keys or doing cert verification, though it won't typically be enough to adjust for a packet being lost and the data being PTO'd and then that making keys available. In , Jana's suggested changing ackdelay to including the time packets were buffered as well as and ignoring maxackdelay for the first sample in a PN space. I think this is a sensible mitigation, though it doesn't guarantee the issue will not occur. If no Initial ACKs have been received and the ACK with the large ackdelay is the first ACK of the connection, then you still can't use ack_delay. The one change we've found mitigates this issue entirely is coalescing packets from the next PN space(Handshake or 1RTT) and then processing the full coalesced packet before processing all buffered packets and sending an ACK. That ensures there's one new packet being acknowledged and you get a non-delayed RTT sample. It doesn't seem onerous to add a "SHOULD process all packets in the current datagram and all buffered packets before sending an ACK to avoid generating multiple inflated RTT samples" to me?
Repeating this here, so that the PR comments don't get lost: This is about a sender having to buffer packets that might contain ACK frames. Usually, this is because the sender doesn't have the keys necessary, so they hold the packets until they do. When processing the ACK frame, some time might have passed. So we now have 5 times in play: when the largest acknowledged packet was a) sent, and b) received; and when the packet containing the ACK was c) sent, d) buffered, and e) processed. The critical difference here is that we now recognize the intentional delays at the sender in addition to intentional delays at the receiver. The receiver calculates and transmit and sends that in ACK Delay. We have a commitment that . That doesn't need to change. The spec currently assumes that the sender calculates RTT as (or to include the cap on receiver delay). That is, the time between when the ACK is processed and when the largest acknowledged was sent, less the ACK Delay. But that's wrong. It's that we care about. So all we need to say is that if the sender buffers packets for processing, then it needs to calculate RTT based on the time that the packet entered the buffer. That is, we capture any intentional delays added by the sender. Then we don't have to worry about the maxackdelay cap, because receiver behaviour always follows the contract.
Sorry, I should correct that last bit: We only have to worry about the maxackdelay cap when the packet that is being acknowledged might have been buffered. Thus, it makes sense to remove the cap for packets that might have been buffered (i.e., those that are sent prior to handshake confirmation).
Also from the PR. I had never considered the case when the sender has buffered a packet that contains an ACK frame until you pointed it out. The PR was intended to deal with the case when a peer receives packets it can't immediately decrypt and acknowledge. But I added a sentence about subtracting the amount of time buffered when processing a buffered packet containing an ACK. It can only happen for a server that hasn't reached handshake confirmed, so it's fairly rare.
It seems to me that this issue is an alternative resolution to one specific part of what is being proposed as the resolution to . Specifically, quoting from URL, the approach proposed here is an alternative to the line being emphasized below. 1-RTT data until the handshake is complete (Section 6.2.1). This is not quite adequate, since it still allows a client to arm a PTO timer on 1-RTT data on sending a CFIN and 1-RTT data, before ensuring that the server has received the CFIN. The proposed solution here is to change this condition from "handshake complete" to "handshake confirmed". Meaning that an endpoint only arms a PTO timer on 0-RTT or 1-RTT data when it knows that the peer has the keys necessary to read and write in 1-RTT, and can therefore respond with an ACK. missing subtle deadlock issues. the data appropriately. ackdelay is the amount of the time the information (regardless of that being the packet itself or the acknowledgement) being buffered intentionally. For this, we need the following: and the ack being buffered) confirmation. Those are the ones that might have been buffered at the peer until keys were ready. 1-RTT packets, instead of dropping them, as dropping packets has significant impact on congestion control. Note that this matches what those implementations already do. The proposed resolution in recommends the sender to retain one bit of constant for each packet it sends, and use it to see if ackdelay should be capped. The resolution proposed here is to retain a mutable bit for each packet number space, and flip that bit when the seder receives the first ack. Until that bit is flipped, the sender does not cap ack_delay. As correctly pointed out in the , this approach relies on the receiver processing all the buffered packets at once then sending an ACK. I think that both approaches work, and do not have a strong preference among the two. I'd also note both approaches would work correctly once we adopt the basis of the latter approach; i.e. recommend the receiver to process all buffered packets before sending an ACK. Therefore, it seems to me that the only question is if we want to have that recommendation. If the answer is yes, then both approaches would work, and the choice between the approaches becomes an editorial preference (i.e. write down either or both).
I actually updated the PR to align more closely with the original proposal after NAME pointed out that they weren't identical if all buffered packets weren't processed at once(which is something I'd implicitly assumed). I could go either way on that, but I do think it's easier to only special case the first ACK for a PN space rather than store a bit per packet. It also mitigates the issue for those that don't follow this new MAY. However, one difference from the above is this doesn't rely on handshake confirmation, since this is just as much a problem for Handshake packets as it is 1-RTT packets, so I think we need to use a more general statement about the sender not knowing if the receiver has the keys to process the packets.
NAME (swapping paragraphs being quoted) I am happy to see the issue addressed for all epochs if that's the preference of others. FWIW I tend to think that we should use SHOULD, especially if we are going to also cover the delay of Handshake ACKs. Max-ack-delay of Handshake packets are assumed to be zero. Therefore, we would see the problem on any environment where the time required for processing a TLS handshake message is relatively greater than the path delay. Including datacenter networks where RTT is small, or slow IoT devices.
Let's mark this as design and run the process.
During the handshake, there are moments when an endpoint receives a packet from its peer and calculates the traffic keys of the next PN space. To give some examples, a server receives an Initial packet containing ClientHello, and derives the Handshake keys. Or, a client receives the Handshake packets, verifies the handshake transcript (including the certificate chain) and derives the 1-RTT keys. While deriving the traffic keys of the next PN space, an endpoint might receive packets belonging to the next PN space. Note that the process of deriving the traffic keys might take some time. I see Chrome running on a PC spending ~100ms for verifying the certificate chain, when it is launched from the command line along with the URL to be fetched. I presume that such launch-to-fetch-URL behavior would not be uncommon on PCs and handheld devices. Another (more) extreme case would be when OCSP is involved. In these cases, the time required for deriving the new keys could take more than a handful of RTTs. Hence the following questions, that might be worth clarifying in the specifications: Q1. How should an endpoint behave when it takes time to derive the traffic keys of the next PN space? Q1.1. Should the endpoint continue ACKing packets of the current PN space? Q1.2. Can the endpoint discard packets of the next PN space, as it can expect those packets to be retransmitted? Q2. How should the peer behave? Re Q1.1, I think that the answer is yes, it should. Such an endpoint should calculate the next set of traffic keys asynchronously. It should continue running the QUIC transport for the existing PN space. To give an example, while the client verifies the certificate chain, it should ACK and continue ACKing the Handshake packets that it receives. Re Q1.2, I think the answer is no. An endpoint SHOULD buffer packets of the next PN space while it calculates the traffic keys for that PN space. This is because the peer cannot distinguish the endpoint dropping packets vs. packets being dropped by the network. Dropping the packets lead to the peer's congestion controller exiting the startup phase. That is going to have a negative effect on performance. Re Q2, I would argue that, ideally, the peer SHOULD refrain from arming the PTO timer for the packets of the next PN space, when it knows that the endpoint has all the information to derive the traffic keys of the next PN space, and knows that the endpoint would send something once those traffic keys are derived. To give an example, a server SHOULD NOT arm the PTO timer for 0.5 RTT data until it receives ClientFinished. At the very least, we should caution the server to not send Handshake(PING) packets alongside the 1-RTT probe, as ACKs for Handshake(PING) packets would reset the PTO timer.
We encountered a problem similar to the one described here in our stack. The latter part of a coalesced packet was being dropped because we had to spend time validating the certificate. We ultimately added a small amount of buffering to address this. And I have debugged a case where handshakes were failing due to the interaction between PTO timers in different packet number spaces. For that, I don't have a good solution yet; I'm not convinced that deferring PTO is the right answer. But I don't think that this requires anything in specifications, aside from perhaps a caution, noting that we already have a caution in . That already suggests and permits (though doesn't recommend) buffering. We could add a note that failing to buffer legitimate packets could lead to marking delivered packets as lost unnecessarily and performance degradation
I would argue that a probe packet is expected to be sent when an endpoint does not receive a response that it expects. That's necessary because it is the only way to guarantee forward progress (and therefore ). But in this particular case that we are discussing, the ball is on the side of the peer. The peer is expected to respond with a new TLS handshake message contained by a QUIC packet using the next PN space. To paraphrase, sending packets before knowing that the peer is ready to decrypt them is a performance optimization. Doing that once is fine, but having a MUST for repeating that is questionable. While I am happy to discuss the nuance, I do think that it needs to be something other than "MUST send a probe." Note also that an endpoint sending PTOs in this case would in turn require the peer to have a larger buffer for storing undecryptable packets in order to avoid buffer overflow (that leads to performance degradation). Aside from relaxing the MUST pointed out above, I think I'd be fine with just adding notes. Though I might argue that it is more important than the existing note that discusses about an "exceptional" case; some endpoints would almost always need time to calculate the next keys.
FWIW, states that: as keys at a given encryption level become available to TLS, TLS indicates to QUIC that reading or writing keys at that encryption level are available. These events are not asynchronous; they always occur immediately after TLS is provided with new handshake bytes, or after TLS produces handshake bytes. Part of the problem is that this is untrue. TLS stacks sometimes take time in providing the net set of traffic keys. And QUIC stacks do not even pretend that TLS stacks are acting in a synchronous manner. Instead, they intentionally send ACKs for Handshake packets that are received later than the 1-RTT packets they buffer. This is a behavior that is observable from the peer, and leads to the exit of slow start depending on how you design the loss recovery. I think we might want to use normative words stating what the expectations are.
Thanks for filing this. I think I mostly understand the issue, and I think this is mostly a correctness issue with Chrome. That being said, we may want to add some caution that this is a potential issue and to avoid handshake operations(ie: cert verification on the client and remote handshaking on the server) from excessively delaying acknowledgements. I've seen lots of cases when the cert is spread over 3 packets and Chrome ACKs the first two packets before the cert is verified, so this doesn't seem to be a case of key availability. Google servers don't have cert compression enabled yet, so many certs don't fit into 1 or 2 packets. The recovery draft already says that peers don't arm for ApplicationData(including 0.5RTT) before the handshake is confirmed. Knowing the peer has the information to derive the keys isn't really enough, since you'd also have to know the peer has received a packet of the next encryption level. As a result, I can't think of any changes to this part of the PTO logic I'd want to make.
NAME I agree that the there is no design changes necessary to the recovery draft. The crux of the problem is the TLS draft. It currently states: However, contrary to what the TLS draft says, Chrome (and IIUC what NAME says, Firefox also) processes the TLS handshake messages containing the certificate chain in an asynchronous manner. Consider the following example, where a web browser receives four packets: Handshake (PN=1, EncryptedExtensions) Handshake (PN=2, Certificate, CertificateVerify, Finished) 1-RTT (PN=3, NewSessionTicket) (30ms later) receives Handshake (PN=4, PTO) The TLS draft suggests that client would be processing PN=3 before PN=4, because the browser would process the TLS handshake transcript synchronously and therefore it would have the 1-RTT keys when it recognizes PN=3 for the first time. But the reality is that, if the browser takes 100ms to verify the certificate chain, the client buffers PN=3, acking PN=4 before PN=3, even though the interval between PN=3 and PN=4 is greater than maxackdelay. As can be seen, this is an observable behavior from the server-side. Therefore, we need to fix the TLS draft at least (assuming that we agree that browsers are doing the correct thing).
One more note is that the RTT samples can be greatly inflated if a packet(ie: the Initial) is lost and the PTO fires to retransmit the data, and that causes the previously buffered Handshake packets to be acknowledged. Coalescing an ack-eliciting Handshake packet with the Initial typically mitigates this, as long as it's processed before the buffered packets. Since coalescing is already an optimization in the draft, it may be worth a note that it can prevent inflated RTT samples. I considered an algorithm to discard these RTT samples, but filtering RTT samples is a dark art I'd rather not put into the recovery spec unless it's absolutely necessary. We've observed some cases where this causes the handshake to timeout because the server's PTO is so long, and our handshake timeout is fairly short(9 or 10 seconds).
The discussion about RTT estimate is interesting. I'm not sure about the Initial vs. Handshake case, but when the client spends X milliseconds (e.g. X=100) verifying the certificate chain at the same time buffering 0.5 RTT data it receives, then the server would obtain an RTT sample of milliseconds (e.g., 75 ms). That's not great. One way of resolving the issue would be to not cap ackdelay using maxackdelay for ApplicationData packets that were sent during the handshake. The rationale for not activating PTO for 1-RTT packets during the handshake is that the peer might not immediately have the necessary keys to process those packets. We assume the peer to buffer such packets. When the sender has such an assumption, it does not really make sense to cap the ackdelay. I think doing something like that would be fine, even though I would not push on recommending such a strategy.
NAME raises a good point about inflated RTTs. NAME point about maxackdelay actually exposes a grey area in the current semantics of that parameter. It is expected to be the maximum delay that the endpoint introduces. Other delay is supposed to be considered to be part of the "path". This is a situation where that assumption doesn't hold well. Separately, reporting this delay as part of ack_delay assumes that the implementation timestamps the packet when received at the endpoint. Meaning that if the endpoint actually does the cert verification synchronously, and does not read from the socket until that is complete, then it will need to read kernel receive times. (This also assumes that an endpoint is expected to timestamp the packet on receipt, not on decryption, which is very much how most stacks will do it, but there's not much difference at the moment in the common case between these two times.) I'm tempted to say that we don't try to solve this in the protocol yet. Let's experiment with strategies that we think might work in implementations, share notes, and come back to the spec when we have a good answer. For the time being, we expect that the delays at the clients should not be super common, and even when it is the case, a couple of RTT samples should not significantly sway the RTT. I'm ok with that.
NAME I think we do not need require endpoints doing that. We do not even require endpoints to verify certificate chain asynchronously, meaning that we allow endpoints to simplify their implementation at the risk of performance penalty. I'd argue that not capping ack_delay during the handshake is strictly better than capping the value all the time, assuming that the client's clock is not drifting. That is because the server would have a better RTT estimate when the client reports the correct numbers. If the client fails to do so, it'd be the same as status quo, where RTT estimate would become larger than the actual value. All that said, I am totally fine with leaving this an open area that endpoints can experiment.
Summarizing my thoughts after discussing this with NAME and thinking about this with NAME Apologies for being verbose, but I'm trying to summarize from the top. Problem: At a high level, the problem we are discussing here is that an endpoint buffers packets, because it isn't able to read them or write acks for them, and it might take a while to dequeue the buffer. As a result, the acks can be sent much later than when the packets were received at the endpoint, causing the ackdelay to potentially be much higher than maxackdelay. This inflates the smoothedrtt and rttvariance, inflating the PTO, which can cause handshakes to fail (if a "handshake timeout" period is employed). The problem is that the receiver endpoint can take arbitrarily long to respond. The ackdelay variable is meant to signal precisely this -- we tout it as an important distinction from TCP -- and it is useful here, except that it is limited to maxackdelay in the various RTT estimations. This was done deliberately, allowing for any additional delays to become part of the path delay, with the expectation that such additional delays are outside of the control of the QUIC endpoint and are likely to be recurring. While that remains true, the issue that we are discussing here is an extreme case that occurs during the handshake and is not expected to be recurring. Proposed Solution: We use the ack delay signal during the handshake, and acknowledge that we might need to treat ack delays reported during the handshake period as slightly special. Specifically, we can ignore the maxackdelay, but we still limit the RTT sample to >= minrtt. An endpoint does this until the handshake is confirmed_. That is, until the endpoint is certain that the peer has 1RTT read and write keys and therefore does not have to buffer any more packets for lack of keys. Also: Separately, the recovery spec currently says don't arm PTO on ApplicationData until . This should be . This is because handshake completes at a client when it sends a CFIN, after which the client can send 1-RTT data. The CFIN however might get lost and the server might buffer the 1-RTT data until it receives a retransmission of the CFIN. Changing this to makes it so that no PTO is armed on ApplicationData until there's no Initial or Handshake data left in flight.
Regarding NAME last point, I've changed the moment that the client starts arming PTO timer for ApplicationData PN space from handshake complete to handshake comfirmed. As Jana states, that is the correct thing to do based on the rationale that we have been using to explain why it should not be armed.
FWIW, I was really concerned about that last change. Aside from it making this a non-editorial change, the way that we avoid handshake deadlock depends very much on HANDSHAKEDONE being followed by probes. But as the server confirms and completes the handshake at the same time, I've convinced myself that this is OK. (That this is what our stack did, largely by accident, is vindication of sorts...) I would encourage people to sit down and run the analysis on that change to convince themselves that this is not reintroducing the handshake deadlock. The other part of the proposed solution needs to be slightly different. NAME proposes that we ignore prior to confirming the handshake. I think that's good, but it is not completely correct, and nor would it be sufficient. We need to ignore for acknowledgements of packets that were sent prior to handshake confirmation. Those are the ones that might have been buffered until keys were ready. In order for that to work, we have two new requirements on packet processing for buffered packets: endpoints MUST process and acknowledge (or discard) any packets that they might have buffered during the handshake within of completing the handshake endpoints MUST report an ACK Delay that is based on when a packet is received, not when it was processed
Regarding , I think NAME comment is accurate with possibly one slight modification. We might want to say that ACK Delay is the amount of the time the information (regardless of that being the packet itself or the acknowledgement) being buffered intentionally. The observation here is that any unintentional delay (e.g., decryption time, payload processing time) should be considered part of RTT, because that is unavoidable. And the added bonus of defining this way is that then there would be no change required for QUIC stacks that synchronously process the TLS handshake messages.
Just FWIW, I might point out that we are actually making the design streamlined, by improving consistency in how we handle ack delay. The new rule is: The receiver reports the sum of the controlled delays by using the delay field of an ACK frame. When the sender knows what the maximum of that controlled delay is, the sender ... uses that value (i.e. maxackdelay) to arm the PTO timer, and uses that value (i.e. maxackdelay) to calculate the RTT, because if the value of the delay field is greater than the known maximum, the difference is considered as uncontrolled delay and therefore has to be accounted as part of the path delay. When the sender does not know what the maximum of that controlled delay is, the sender ... does not arm the PTO timer, because it does not know when it can expect an ACK to arrive, and calculates the RTT without the capability of correcting misreported controlled delay. It's simply that up until now, we have misused max-ack-delay in the last bullet point case.
This is a good outcome, and I agree with Kazuho. However, I am going to request that someone (not me preferably) take this to the mailing list to confirm with a wider group.
NAME : I agree with what you've spelled out above (). It is close to what I had in mind, but I didn't do a good job of even spelling out fully what I was thinking.
Posting here, FTR. This got no pushback and support, FWIW. Note that this resolution makes this issue a design one, so I'm removing the editorial tag.
NAME noted:
This should be marked design.
I'm trying to fully digest everything above, so I apologize if I missed something important. The idea that you wouldn't acknowledge packets that can be processed while waiting for keys for queued packets is clearly unrealistic for a variety of reasons, so we should definitely remove any text that assumes otherwise. I'm unclear about why moving from handshake complete to confirmed is an improvement, given the example you cite seems to imply the server is retransmitting unnecessary data due to client delays, and for the server complete and confirmed happen at almost identical times. Prior to CFIN or 1-RTT being acknowledged, the client knows the server has only one of the two keys. PTO will retransmit the CFIN on the first PTO and the 1-RTT data on a subsequent PTO(assuming the first PTO doesn't send both). I agree we expect HANDSHAKEDONE to make this irrelevant in practice, but I can't reason about why this change is an improvement based on the problem description. Can someone lay that out for me? I believe we should make normative changes to transport along the lines of S2. Specifically, I think we should say receivers SHOULD include the delay from the time an undecryptable packet is buffered to the time it's acknowledged in the ackdelay, as well as processing all buffered packets for a given encryption level before sending an ACK. Ideally, a receiver would process buffered packets after the full datagram being processed as well, since that allows senders to fully mitigate this issue, but we could leave that to a description, rather than normative text if people prefer. This provides more information to the sender, which it can ignore if it wants, and is much better than sending an Ack Delay of 0us in that case. It's a bit odd that we say a receiver SHOULD set the ack delay field to 0 for Initial and Handshake packets today.
I think it is because the client cannot make an assumption on how long it would take for the server to verify the client's TLS handshake messages being sent using Handshake packets. Important thing to note here is that it could not just be Finished. It might include client certificates. Theoretically, client's handshake transcript might be too large so that it requires multiple round-trips to be sent. Regarding ack_delay, I might not sure if I fully follow. But I think I agree that it would become reasonable for an endpoint to send a Handshake Ack with a non-zero ack-delay, if it buffered the handshake packet that was undecryptable at the moment that packet was received, and later processed it. It is a natural outcome of us changing the definition of ack-delay to the sum of undecryptable buffering delay and acknowledgement delay. As Initial packets would always be decryptable, I think "SHOULD set ack delay to 0" rule can remain for Initial packets.
Specifically, is the concern is that the client will have it's Handshake data acknowledged and will spuriously PTO 1-RTT packets a few times over while waiting for the server to make 1-RTT receive keys available? Is this a problem that's been observed? It seems like it would in practice require client certs, since verifying only a CFIN should be very fast. Making the change to wait for handshake confirmed to PTO 1-RTT data doesn't seem harmful given the existence of HANDSHAKEDONE, but I'm also not sure it's going to matter much in practice. It also adds an exception to the current rule that PTO is armed when the client isn't sure the server has completed address validation or there are bytes in flight. If the Handshake data has been acknowledged, but the 1-RTT data hasn't, you'd have bytes in flight but not arm the PTO if I understand correctly. Given there's still a SHOULD about sending packets from multiple PN spaces on PTO, including Handshake + 1RTT, I think this change will perform well, but I do have apprehension about seemingly small but subtle changes like this late in the process. Agreed, the potential for delay is only for the first Handshake and 1-RTT ACKs. My summary: The issue is that ACKs for Handshake and 1-RTT packets could be delayed because the peer doesn't have the keys(either because it takes time to derive them, or it's missing crypto data to derive the next key). This creates two potential issues(neither of which effects correctness, but could be suboptimal): Packets may be acknowledged later than expected, causing the peer to spuriously PTO. The first RTT sample in Handshake and 1-RTT packet spaces could have substantial delay in excess of the 'no ack delay' in Handshake or maxackdelay for 1-RTT, which inflates SRTT and RTTVar. Changing when to arm PTO for 1-RTT data mitigates one case of the 1st issue(the server being slow to make 1-RTT receive keys available), but I don't believe it helps otherwise with the issue, such as delays in generating Handshake keys. Are there any other design changes(besides complete->confirmed) being considered or suggested for the first issue that I'm missing? If there's no loss, the second issue fixes itself without harm, which is nice. And it's not as bad for ApplicationData, since that will have a non-0 maxackdelay, which may be enough to absorb the endpoint delay generating keys or doing cert verification, though it won't typically be enough to adjust for a packet being lost and the data being PTO'd and then that making keys available. I think Jana's suggestion ackdelay to including the time packets were buffered is sensible, and ignoring maxackdelay for the first sample in a PN space seems relatively safe given at least one prior sample. But what if there are no Initial ACKs received and ACK with the large ack_delay is the first ACK of the connection? I guess you're stuck with one bad sample and hope the next is better? The one change we've found mitigates this issue entirely is coalescing packets from the next PN space(Handshake or 1RTT) and then processing the full coalesced packet before processing all buffered packets and sending an ACK. That ensures there's one new packet being acknowledged and you get a non-delayed RTT sample. It doesn't seem onerous to add a "SHOULD process all packets in the current datagram and all buffered packets before sending an ACK to avoid generating multiple inflated RTT samples" to me?
NAME : I'm yet to process the messages above -- but this issue was covered in .
NAME I think the issue is about principles. I tend to agree that it would be fine for an endpoint to arm PTO timer for 1-RTT packets, when it knows that the peer is going to be able to handle them. However, if we are going to talk about such cases, then I might point out there are other similar cases that are missing in the draft, that might be as important as the case you point out: A server can arm 1-RTT PTO timer when an abbreviated TLS handshake (a.k.a. resumption) is being used. This is because a server to assume that the client would obtain 1-RTT read keys almost immediately. A client can arm PTO timer for 0-RTT packets, assuming that the server would be resuming the connection. My view is that all these are "optimizations" that an endpoint might attempt, based on the knowledge of the form of the TLS handshake being used. But they are just optimizations. I think it would be fine to talk about such optimizations, but at the same time think that we should provide safe and principled rules as baseline, so that endpoints following the recommendations do not end up sending bunch of unnecessary packets. Yeah, I tend to think this as a bug fix. Existing PTO rules seem to assume that the client would spend time in processing the Handshake packets, but the server would not. But the reality is more nuanced, as pointed out by the client-certificate case as well as the examples shown above. And in case where the assumption is too aggressive (the client-cert case), we might see a number of unnecessary packets being sent. All that said, I do think that we should try to minimize the impact of the changes that we introduce to the drafts. That's why I think that something along the lines of "SHOULD NOT arm PTO timer for ApplicationData packets until the handshake is confirmed, unless the sender can assume that the peer would be capable of processing ApplicationData packets when they arrive." Clients arming PTO for 1-RTT packets when client-certificate is not used would fall into this exception, and therefore the design change would not affect the existing QUIC stacks, assuming that they do not use client certificates. Stacks that do use client certificates have to have this fix applied.
NAME That's correct. The resolution proposed by NAME does not mitigate that issue. By including queuing delay of unencryptable packets into ackdelay, the proposed resolution improves the quality of RTT samples. But as you say, it does not mitigate the case where that would be the first sample. I'm not sure if doing that mitigates the problem. The proposed resolution recommends endpoints to send ACK for a handshake message before spending time to process it. As an example, a client would ack the Handshake packets before starting to verify the certificate chain. And after that, it would send a Handshake packet (that contains ClientFinished only, no ack will be included), and a 1-RTT packet acking the 0.5-RTT data sent by the server. If the Handshake ack reached the server, then the server would have a good minrtt estimate. If it did not, then, the next sample that the server would receive is the ack for the 0.5-RTT packet(s). As the servers do not arm the PTO timer for 0.5-RTT data, it is hard to have a protocol mechanism that prevents the first 1-RTT ACKs sent by the client from having a ack delay field that reflects the amount of time spent for verifying the certificate chain.
I created a separate issue() and PR() for the inflated RTT problem, since the solution seems fairly disjoint.
I agree that if you have both sets of keys, the transport should send an ACK and not wait for all async crypto processing to complete. This is similar to receiving a GET request at a proxy. The proxy acknowledges the GET immediately even if it has to go to a backend to respond. But that doesn't prevent greatly inflated RTT samples when the keys aren't immediately available.
It's also above, but the inflight PR only attempts to address part of the problem, not the inflated RTT issue that I mentioned halfway through the issue. Also, I'm finding it confusing to track what problems are trying to be solved and what ones are already largely solved, so I was hoping restating my understanding would allow you to correct me if I'm misunderstanding something.
I think we should flag this as design and run the process. It's hard to pin down exactly what changes here substantively, but it's worth getting the eyes. I'd like to bring this slightly tricky issue under the wg's eyes. Discussions so far have yielded a strategy that I propose further below. This summarizes conversations in 3821 and 3874 (and the outcomes might touch 3889 ). The purpose of this email is to see if we can agree on the strategies outlined here, so we can move forward on the PR there and/or other associated issues and PRs. Problems: During the handshake, an endpoint can receive packets (carrying data or acknowledgements) that it cannot decrypt because the requisite keys are not yet available. This can happen because the crypto machinery at the endpoint is waiting for earlier packets to be received to yield the keys. Similarly, a client might not be able to read 1-RTT packets received from the server because it is waiting for certificate validation to complete. The assumption in the TLS specification is that when an endpoint is busy doing something (certificate validation or key generation for instance), it won't be able to respond to received packets with acknowledgements (Section ). Contrary to the assumption in the spec, Chrome and Firefox both process the certificate chain asynchronously, and therefore acks handshake packets immediately, even the ones that arrive after they have started verifying the certificate chain. But they buffer 1-RTT packets, and therefore the issue exists for 1-RTT packets. As a result, there's potential delay, sometimes significant, in responding to some packets during the handshake. There are two side effects of this. S1. First, this means that an endpoint that runs a PTO timer and retransmits data for 0-RTT and 1-RTT packets while the handshake is still in progress might be doing this quite unnecessarily. The peer may well have received this data and have queued it, waiting to be able to decrypt it. S2. Second, acks for these packets that are buffered and then later decrypted and read can be sent much later than when the packets were received at the peer, causing the ackdelay to potentially be much higher than the advertised maxackdelay. The endpoint ignores ackdelay larger than maxackdelay, and this inflates the smoothedrtt and rttvariance, potentially inflating the PTO significantly (which can cause handshakes to fail if the endpoint employs a "handshake timeout" period). Proposed Solutions: S1. To solve the first problem, we already have a condition in the recovery draft to not arm the PTO on 1-RTT data until the handshake is complete (Section ). This is not quite adequate, since it still allows a client to arm a PTO timer on 1-RTT data on sending a CFIN and 1-RTT data, before ensuring that the server has received the CFIN. The proposed solution here is to change this condition from "handshake complete" to "handshake confirmed". Meaning that an endpoint only arms a PTO timer on 0-RTT or 1-RTT data when it knows that the peer has the keys necessary to read and write in 1-RTT, and can therefore respond with an ACK. We think this holds up, but we'd like to get this under everyone's eyes, to ensure that the discussion isn't missing subtle deadlock issues. S2. To solve the second problem, the proposal is to use ackdelay as reported by the endpoint that buffers the data appropriately. ackdelay is the amount of the time the information (regardless of that being the packet itself or the acknowledgement) being buffered intentionally. For this, we need the following: endpoints MUST report an ackdelay that is the sum of the controlled delay (i.e. the duration of the data and the ack being buffered) endpoints ignore maxack_delay for acknowledgements of packets that were sent prior to handshake confirmation. Those are the ones that might have been buffered at the peer until keys were ready. In addition, we explicitly recommend endpoints that asynchronously process handshake messages to buffer 1-RTT packets, instead of dropping them, as dropping packets has significant impact on congestion control. Note that this matches what those implementations already do. Thoughts? jana

As keys at a given encryption level become available to TLS, TLS indicates to QUIC that reading or writing keys at that encryption  <ins> level are available.  While generating these keys, an endpoint SHOULD buffer received packets marked as protected by the keys being generated, and process them once those keys become available.  If the keys are generated asynchronously, an endpoint MAY continue responding to the received packets that were processable while waiting for TLS to provide these keys. </ins> TLS provides QUIC with three items as a new encryption level becomes available: 5. As with TLS over TCP, QUIC protects packets with keys derived from  <ins> the TLS handshake, using the AEAD algorithm AEAD negotiated by TLS. QUIC packets have varying protections depending on their type: Version Negotiation packets have no cryptographic protection. Retry packets use AEAD_AES_128_GCM to provide protection against accidental modification or insertion by off-path adversaries; see retry-integrity. Initial packets use AEAD_AES_128_GCM with keys derived from the Destination Connection ID field of the first Initial packet sent by the client; see initial-secrets. All other packets have strong cryptographic protections for confidentiality and integrity, using keys and algorithms negotiated by TLS. This section describes how packet protection is applied to Handshake packets, 0-RTT packets, and 1-RTT packets.  The same packet protection process is applied to Initial packets.  However, as it is trivial to determine the keys used for Initial packets, these packets are not considered to have confidentiality or integrity protection. Retry packets use a fixed key and so similarly lack confidentiality and integrity protection. </ins> 5.1. separation between QUIC and TLS; see key-diversity. The KDF used for initial secrets is always the HKDF-Expand-Label  <ins> function from TLS 1.3; see initial-secrets. </ins> 5.2.  <ins> Initial packets apply the packet protection process, but use a secret derived from the Destination Connection ID field from the client's first Initial packet. </ins>  <ins> This secret is determined by using HKDF-Extract (see Section 2.2 of HKDF) with a salt of 0xafbfec289993d24c9e9786f19c6111e04390a899 and a IKM of the Destination Connection ID field.  This produces an intermediate pseudorandom key (PRK) that is used to derive two separate secrets for sending and receiving. The secret used by clients to construct Initial packets uses the PRK and the label "client in" as input to the HKDF-Expand-Label function to produce a 32 byte secret; packets constructed by the server use the same process with the label "server in".  The hash function for HKDF when deriving initial secrets and keys is SHA-256 SHA. This process in pseudocode is: </ins> The connection ID used with HKDF-Expand-Label is the Destination Connection ID in the Initial packet sent by the client.  This will be packet after receiving a Retry packet, where the Destination Connection ID is selected by the server.  <ins> Future versions of QUIC SHOULD generate a new salt value, thus ensuring that the keys are different for each version of QUIC.  This prevents a middlebox that recognizes only one version of QUIC from seeing or modifying the contents of packets from future versions. </ins> The HKDF-Expand-Label function defined in TLS 1.3 MUST be used for Initial packets even where the TLS versions offered do not include TLS 1.3.  <ins> The secrets used for constructing Initial packets change when a server sends a Retry packet to use the connection ID value selected by the server.  The secrets do not change when a client changes the </ins> Destination Connection ID it uses in response to an Initial packet from the server. packet; the client has to rely on the exchange that included the Retry packet for that property.  <ins> test-vectors contains sample Initial packets. </ins> 5.3. using the TLS_AES_128_GCM_SHA256 cipher suite, the AEAD_AES_128_GCM function is used.  QUIC can use any of the cipher suites defined in TLS13 with the exception of TLS_AES_128_CCM_8_SHA256.  A cipher suite MUST NOT be negotiated unless a header protection scheme is defined for the deploy a new cipher suite.  This also applies to TLS_AES_128_CCM_8_SHA256. <ins> When constructing packets, the AEAD function is applied prior to applying header protection; see header-protect.  The unprotected packet header is part of the associated data (A).  When processing packets, an endpoint first removes the header protection. </ins> The key and IV for the packet are computed as described in protection-keys.  The nonce, N, is formed by combining the packet protection IV with the packet number.  The 62 bits of the


4.6.  To communicate their willingness to process 0-RTT data, servers send a NewSessionTicket message that contains the "early_data" extension <del> with a max_early_data_size of 0xffffffff; the amount of data which the client can send in 0-RTT is controlled by the "initial_max_data" transport parameter supplied by the server.  Servers MUST NOT send the "early_data" extension with a max_early_data_size set to any value other than 0xffffffff.  A client MUST treat receipt of a NewSessionTicket that contains an "early_data" extension with any other value as a connection error of type PROTOCOL_VIOLATION. </del>  A client that wishes to send 0-RTT packets uses the "early_data" <del> extension in the ClientHello message of a subsequent handshake (see Section 4.2.10 of TLS13).  It then sends the application data in 0-RTT packets. </del>  <del> 4.7. </del>  A server accepts 0-RTT by sending an early_data extension in the EncryptedExtensions (see Section 4.2.10 of TLS13).  The server then configuration.  The client therefore MUST reset the state of all streams, including application state bound to those streams. <del> A client MAY attempt to send 0-RTT again if it receives a Retry or Version Negotiation packet.  These packets do not signify rejection of 0-RTT. </del>  <del> 4.8. </del>  When a server receives a ClientHello with the "early_data" extension, it has to decide whether to accept or reject early data from the different requirements for determining whether to accept or reject early data. <del> 4.9. </del>  In TLS over TCP, the HelloRetryRequest feature (see Section 4.1.4 of TLS13) can be used to correct a client's incorrect KeyShare extension use the Retry feature (see Section 8.1 of QUIC-TRANSPORT). HelloRetryRequest is still used to request key shares. <del> 4.10. </del>  If TLS experiences an error, it generates an appropriate alert as defined in Section 6 of TLS13. handshake_failure (0x128 in QUIC).  Endpoints MAY use a generic error code to avoid possibly exposing confidential information. <del> 4.11. </del>  After QUIC moves to a new encryption level, packet protection keys for previous encryption levels can be discarded.  This occurs several and retransmissions of data in CRYPTO frames are sent at a previous encryption level.  These packets MAY also include PADDING frames. <del> 4.11.1. </del>  Packets protected with Initial secrets (initial-secrets) are not authenticated, meaning that an attacker could spoof packets with the This results in abandoning loss recovery state for the Initial encryption level and ignoring any outstanding Initial packets. <del> 4.11.2. </del>  An endpoint MUST discard its handshake keys when the TLS handshake is confirmed (handshake-confirmed).  The server MUST send a HANDSHAKE_DONE frame as soon as it completes the handshake. <del> 4.11.3. </del>  0-RTT and 1-RTT packets share the same packet number space, and clients do not send 0-RTT packets after sending a 1-RTT packet

This adds a short introductory section to the three sections that deal with 0-RTT. That might help contextualize and help contain the spaghetti structure that this section seems to have.
Thanks Nick. I took it in a slightly different direction, but I think that I captured this.
One high-order bit: Using 0-RTT as a noun doesn't sound right, though we do use it informally in that sense. My suggestion use of "The 0-RTT handshake delay feature" or "0-RTT delay before sending application data" as appropriate.This change looks good.

4.6. <ins> The 0-RTT feature in QUIC allows a client to send application data before the handshake is complete.  This is made possible by reusing negotiated parameters from a previous connection.  To enable this, 0-RTT depends on the client remembering critical parameters and providing the server with a TLS session ticket that allows the server to recover the same information. This information includes parameters that determine TLS state, as governed by TLS13, QUIC transport parameters, the chosen application protocol, and any information the application protocol might need; see app-0rtt.  This information determines how 0-RTT packets and their contents are formed. To ensure that the same information is available to both endpoints, all information used to establish 0-RTT comes from the same connection.  Endpoints cannot selectively disregard information that might alter the sending or processing of 0-RTT. TLS13 sets a limit of 7 days on the time between the original connection and any attempt to use 0-RTT.  There are other constraints on 0-RTT usage, notably those caused by the potential exposure to replay attack; see replay. 4.6.1. </ins> To communicate their willingness to process 0-RTT data, servers send a NewSessionTicket message that contains the "early_data" extension  <ins> with a max_early_data_size of 0xffffffff.  The TLS max_early_data_size parameter is not used in QUIC.  The amount of data which the client can send in 0-RTT is controlled by the "initial_max_data" transport parameter supplied by the server. Servers MUST NOT send the "early_data" extension with a max_early_data_size set to any value other than 0xffffffff.  A client MUST treat receipt of a NewSessionTicket that contains an "early_data" extension with any other value as a connection error of type PROTOCOL_VIOLATION. </ins> A client that wishes to send 0-RTT packets uses the "early_data"  <ins> extension in the ClientHello message of a subsequent handshake; see Section 4.2.10 of TLS13.  It then sends application data in 0-RTT packets. </ins>  <ins> A client that attempts 0-RTT might also provide an address validation token if the server has sent a NEW_TOKEN frame; see Section 8.1 of QUIC-TRANSPORT. 4.6.2. </ins> A server accepts 0-RTT by sending an early_data extension in the EncryptedExtensions (see Section 4.2.10 of TLS13).  The server then configuration.  The client therefore MUST reset the state of all streams, including application state bound to those streams.  <ins> A client MAY reattempt 0-RTT if it receives a Retry or Version Negotiation packet.  These packets do not signify rejection of 0-RTT. </ins>  <ins> 4.6.3. </ins> When a server receives a ClientHello with the "early_data" extension, it has to decide whether to accept or reject early data from the different requirements for determining whether to accept or reject early data.  <ins> 4.7. </ins> In TLS over TCP, the HelloRetryRequest feature (see Section 4.1.4 of TLS13) can be used to correct a client's incorrect KeyShare extension use the Retry feature (see Section 8.1 of QUIC-TRANSPORT). HelloRetryRequest is still used to request key shares.  <ins> 4.8. </ins> If TLS experiences an error, it generates an appropriate alert as defined in Section 6 of TLS13. handshake_failure (0x128 in QUIC).  Endpoints MAY use a generic error code to avoid possibly exposing confidential information.  <ins> 4.9. </ins> After QUIC moves to a new encryption level, packet protection keys for previous encryption levels can be discarded.  This occurs several and retransmissions of data in CRYPTO frames are sent at a previous encryption level.  These packets MAY also include PADDING frames.  <ins> 4.9.1. </ins> Packets protected with Initial secrets (initial-secrets) are not authenticated, meaning that an attacker could spoof packets with the This results in abandoning loss recovery state for the Initial encryption level and ignoring any outstanding Initial packets.  <ins> 4.9.2. </ins> An endpoint MUST discard its handshake keys when the TLS handshake is confirmed (handshake-confirmed).  The server MUST send a HANDSHAKE_DONE frame as soon as it completes the handshake.  <ins> 4.9.3. </ins> 0-RTT and 1-RTT packets share the same packet number space, and clients do not send 0-RTT packets after sending a 1-RTT packet


acceptable provided that the features of TLS 1.3 that are used by QUIC are supported by the newer version. <del> A badly configured TLS implementation could negotiate TLS 1.2 or another older version of TLS.  An endpoint MUST terminate the connection if a version of TLS older than 1.3 is negotiated. </del>  4.3.

Just in case this wasn't blindingly obvious.
The TLS spec says: version of TLS. An endpoint MUST terminate the connection if a version of TLS older than 1.3 is negotiated. I would propose that we forbid clients from offering 1.2 or older [and maybe require servers to enforce this].

acceptable provided that the features of TLS 1.3 that are used by QUIC are supported by the newer version.  <ins> Clients MUST NOT offer TLS versions older than 1.3.  A badly configured TLS implementation could negotiate TLS 1.2 or another older version of TLS.  An endpoint MUST terminate the connection if a version of TLS older than 1.3 is negotiated. </ins> 4.3.


1. HTTP semantics SEMANTICS are used for a broad range of services on <del> the Internet.  These semantics have most commonly been used with two different TCP mappings, HTTP/1.1 and HTTP/2.  HTTP/3 supports the same semantics over a new transport protocol, QUIC. </del>  1.1. <del> HTTP/1.1 HTTP11 is a TCP mapping which uses whitespace-delimited text fields to convey HTTP messages.  While these exchanges are human- readable, using whitespace for message formatting leads to parsing complexity and motivates tolerance of variant behavior.  Because HTTP/1.x does not include a multiplexing layer, multiple TCP connections are often used to service requests in parallel.  However, that has a negative impact on congestion control and network efficiency, since TCP does not share congestion control across multiple connections. </del>  HTTP/2 HTTP2 introduced a binary framing and multiplexing layer to improve latency without modifying the transport layer.  However,

The introduction is oddly narrow about HTTP/1.1 (and h2) being a TCP mapping, even though HTTP/1.x is commonly used on many different transports and session-layers. In fact, it was specifically designed to not be dependent on TCP so that it could be implemented over every network and every security layer. A simple solution would be to delete that as a description, since it is neither accurate nor useful. It's fine to say that HTTP/3 is specific to QUIC.

1. HTTP semantics SEMANTICS are used for a broad range of services on  <ins> the Internet.  These semantics have most commonly been used with HTTP/1.1, over a variety of transport and session layers, and with HTTP/2 over TLS.  HTTP/3 supports the same semantics over a new transport protocol, QUIC. </ins> 1.1.  <ins> HTTP/1.1 HTTP11 uses whitespace-delimited text fields to convey HTTP messages.  While these exchanges are human-readable, using whitespace for message formatting leads to parsing complexity and excessive tolerance of variant behavior.  Because HTTP/1.x does not include a multiplexing layer, multiple TCP connections are often used to service requests in parallel.  However, that has a negative impact on congestion control and network efficiency, since TCP does not share congestion control across multiple connections. </ins> HTTP/2 HTTP2 introduced a binary framing and multiplexing layer to improve latency without modifying the transport layer.  However,


When a stream is closed, this indicates the end of an HTTP message. Because some messages are large or unbounded, endpoints SHOULD begin processing partial HTTP messages once enough of the message has been <del> received to make progress.  If a client stream terminates without enough of the HTTP message to provide a complete response, the server SHOULD abort its response with the error code H3_REQUEST_INCOMPLETE. </del>  A server can send a complete response prior to the client sending an entire request if the response does not depend on any portion of the result of having their request terminated abruptly, though clients can always discard responses at their discretion for other reasons. If the server sends a partial or complete response but does not abort <del> reading, clients SHOULD continue sending the body of the request and close the stream normally. </del>  4.1.1.

client stream should be clearly written as client initiated stream the word which pair of send response is receive request, so its better to be does not abort recieving request here for clearify.
First one is good. The second one is intended to align with Section 2.4 of the transport draft, where "abort reading" is a defined operation on a stream; maybe "does not abort reading the request"?
NAME > "does not abort reading the request" SGTM too

When a stream is closed, this indicates the end of an HTTP message. Because some messages are large or unbounded, endpoints SHOULD begin processing partial HTTP messages once enough of the message has been  <ins> received to make progress.  If a client-initiated stream terminates without enough of the HTTP message to provide a complete response, the server SHOULD abort its response with the error code H3_REQUEST_INCOMPLETE. </ins> A server can send a complete response prior to the client sending an entire request if the response does not depend on any portion of the result of having their request terminated abruptly, though clients can always discard responses at their discretion for other reasons. If the server sends a partial or complete response but does not abort  <ins> reading the request, clients SHOULD continue sending the body of the request and close the stream normally. </ins> 4.1.1.


The RECOMMENDED value for kPersistentCongestionThreshold is 3, which is approximately equivalent to two TLPs before an RTO in TCP.  This duration is computed as follows: For example, assume:

Adds a SHOULD NOT and clarifies that persistent congestion is across PN spaces. I was thinking that this is how it would be implemented, but I suspect this'll end up as a design change.
So a slight rewording might help, but you would have to disqualify losses associated with packets sent prior to getting an RTT estimate, as noted. But then RTT estimates are poor to start with (the first packet to establish a NAT binding can add noticeable delays, for instance).
Can we just forbid persistent congestion detection for the Initial and the Handshake PN space? That would be easier to reason about than the RTT estimate you had a while ago, when you sent the first packet.
NAME I'm sot sure if that would be sufficient as a fix, because packets belonging to ApplicationData PN space are also sent before an RTT sample is obtained (e.g., 0-RTT packets, 0.5-RTT data).
Right, this is what is about. Maybe we should say that only packets sent after handshake completion should be considered for persistent congestion detection?
NAME I agree that using handshake completion (or confirmation) would be better than status quo, though it is still questionable if that's good enough. IIUC, there are two issues: there is no continuity between initial RTT and real RTT the quality of RTT estimate is bad during the early stages of a connection Using handshake completion (or confirmation) as the signal concentrates on fixing the former, but does not address the second concern. If we are to make a design change, I think it would be beneficial to spend some time looking into addressing both of the issues. PS. Note that PTO taking RTTVAR into account does not work as a counterargument here, because we are using RTTVAR in an exceptional way (i.e. to estimate PTO of the past).
Yeah, this is tricky. Can we move the discussion to the issue please?
Based on the discussion we've had, don't we need a more comprehensive set of changes that deal with persistent congestion, sending probe packets, and calculating RTT?
NAME I think that depends on the following two aspects: First, the accuracy we want to have in the definition of persistent congestion (see URL). If we think of persistent congestion as a ballpark figure that resembles a blackhole period of somewhere around 3 PTO, during which an endpoint would have sent at least 2 or 3 packets, then I think the change proposed in this PR is sufficient. Second, if your concern is about endpoints not arming PTO for ApplicationData and that becoming an edge of persistent congestion, then I'd assume that the wort case that we might want to discuss is when all the following conditions are met: the TLS handshake transcript send by the server does not fit in 3 datagrams (therefore the server would have RTT estimate when sending the first 0.5 RTT data) client spends long time in verifying the certificate chain both the 1-RTT packets that carrried 0.5 RTT data and the first 1-RTT data (i.e. the one containing HANDSHAKE_DONE) gets lost Under such scenario, persistent congestion would be declared. But then, the condition matches the definition of the previous paragraph. Therefore, I think we can call this a non-issue, if we are fine with what's written in the previous paragraph.
I think we might need to do something to avoid or at least caution against terrible RTT samples, but that's being discussed in
This is separable from , and I wouldn't mix the two. I agree with NAME that the design issue here is about whether 3 x PTO is about time, about packets, or both. That needs consensus. This PR is good if consensus is on the issue is that PTO is (mostly) about time. We should discuss that on .
NAME : That's probably a good idea.
I added a few lines of pseudocode and a comment. Hopefully that's helpful.
FWIW this is what mvfst (and perhaps others?) has been doing since persistent congestion was implemented, since at the time it seemed silly to calculate the period using the initial RTT. However, at the time we still had the handshake timeout and so persistent congestion did not even apply to initial and handshake packets (see ). More recently our deployment has relied on PTOs for the handshake and thus this check, and there are no obvious problems with it on the Internet. I am in favor of this change even if it is perhaps not exhaustive in its coverage of unlikely pathological cases.
My reaction is the same as NAME I had to think a few times over to make sure that this was adequate, and I believe it is. It covers an egregious case explicitly, which is important.
I'm merging this so Jana can pull any conflicts into
A sender sends a tail of 10 packets, packet number 1 to 10. It now receives an acknowledgement for packets 5 and 10, which establishes the loss of all other packets. It's not clear how to perform the check for persistent congestion now. Several options come to mind: check if the period between 6 and 9 is longer than 3PTO check if either the period between packet 1 and 4 or the period between packet 6 and 9 is longer than 3PTO The ACK doesn't establish loss of all tail packets, so there's no persistent congestion. In an offline discussion, both NAME and NAME expressed preferences for different options here. I'm not sure what the right thing to do is here, but at least the draft should be clearer on what exactly establishes persistent congestion and what doesn't.
Consider the case where the sender receives an acknowledgement for packets from 5 to 10, and the period between 1 and 4 was longer than 3*PTO. IIUC, this is clearly a persistent congestion. The original example is strictly worse than that, and I think it would make sense to declare persistent congestion in the case too. In other words, I think that 3 is incorrect. That said, I do not have a strong opinion regarding if we should specifically recommend either of 1 or 2, considering that the logic (or the time threshold) being used to detect persistent congestion is merely a recommendation.
FWIW, we had some discussion regarding the design rationale behind persistent congestion in , and it seemed to me that it was mostly about the amount of time the sender cannot make any progress.
seems also related
I think we should keep it simple, which IMO is do nothing. Or if we want to add a recommendation, we can say that you start with the largest lost packet and try to detect Persistent congestion until either you have found 1 occurrence or reached the end of list.
This is fixed in , and is an editorial fix.
Consider the following case: endpoint sends Initial after 1 second, endpoint sends Initial (PTO) after 1ms, the endpoint receives an ACK for Initial (PTO) At this point, the endpoint would declare persistent congestion because the duration of congestion (1 seconds) is greater than the threshold (i.e. where smoothedrtt is 1ms, rttvar is 1ms, and maxackdelay is zero). The problem is that determining persistent congestion based on SRTT is not a good idea until the RTT estimate becomes stable. I think the easiest fix (which is beneficial at this late stage) would be to disable persistent congestion until the handshake concludes. [[Spawned off from as a dedicated issue]]
My interpretation here is that the single packet loss does not result in persistent congestion as it spans precisely one packet. That changes of course if there are two packets sent on PTO and the first of those is lost as the losses span multiple packets over a longer period.
NAME is right. The example should be: endpoint sends Initial (PN 1) after 1 second, endpoint sends two Initials on a PTO (PN 2 and PN 3) after 1ms, the endpoint receives an ACK for PN 3 Now we have a loss spanning over 1s, which is >> PTO.
I don't think persistent congestion can be enabled until there is an RTT sample, because smoothed_rtt and rttvar don't have measured values. I wrote an editorial PR to that effect, so assuming that's sufficient, this is an editorial issue.
NAME Thank you for the PR. As I've stated in the PR review comment, per my understanding, the problem is about applying the first RTT sample to the probes sent before obtaining that RTT sample. Because the difference between the first RTT sample and initial RTT could be huge, using the first RTT sample to estimate how many probes have been sent based on the initial RTT does not work well.
I agree that the PR doesn't solve this problem. Conservatively, you could avoid using losses of any packets that were sent before an RTT sample was received as legitimate for persistent congestion, and this might not be particularly hard to implement. You need one variable that is set on the first RTT measurement (largest PN sent for each space), and ensure that all PNs considered when detecting persistent congestion are greater than that number (per PN space). Not doing persistent congestion before the handshake would be quite an aggressive departure from what our intent is.
I agree with Jana that disabling persistent congestion before the handshake would be quite aggressive and quite different from TCP, so I'd prefer to avoid that. I was thinking persistent congestion was a timer that would be started when a packet is sent, but it's a time interval, so I agree the PR is not sufficient as is. I'll update it along the lines of Jana's suggestion, which is what I intended.
So, I understand the disconnect now after discussing with NAME This issue is not just about before an initial RTT sample, it's about the case where the network RTT were to similarly reduce later in the connection. In such a case, when loss also happens, the sender could retroactively declare persistent congestion based on the new RTT estimate, while the previous RTT estimate might have caused fewer than 3 PTOs to fire. Specifically, the sender might have sent fewer than 3 packets as PTO responses into the network before marking the network as having persistent congestion. This seems like a corner case to me that may not be necessary to fix (beyond what is being fixed in ). But I want to hear others' opinions.
NAME Thank you for summarizing the issue. Yes, my point was that the current definition of persistent congestion is inaccurate if it based on the rationale that the endpoint would have had occasions to send 3 probe packets, because the endpoint might have had a larger RTT estimate before receiving the ACK that triggers the persistent congestion check. That said, in the current computation, we do have the guarantee that the endpoint would have sent at least 2 probe packets, because PTO never decreases by more than 25% when obtaining just one RTT sample. If we think that is enough, I'm fine with going with what we have now in .
Thanks for reading the updated PR. Assuming we go with something along the lines of , I think this ends up as a design change, unless others think this is just a clarification of existing text?
I think that this issue is a design issue and also that the current state of is a design change.
This is a design issue. Based on what NAME notes in URL, the design issue here is about whether 3 x PTO is about time, about packets, or both. That needs consensus. As it stands, if it's mostly about time, then fixes this issue. I am happy with that fix.
My understanding was the point of changing from PTOs to persistent congestion was to make it about time. If we think the change in RTT caused by the ACK which triggers the persistent congestion check is an issue, we should ignore that sample when doing the determination. But I'm not convinced the extra complexity is worth it.
NAME NAME Thank you for sharing the rationale behind the design. I am convinced that the fix in is fine.
is now merged, but I realized that it has a slight problem. The text and the pseudo-code don't agree. Since this has gone through consensus, I'm continuing this discussion on .
The wording around persistent congestion sounds across pnspaces: But DetectLostPackets function is per pnspace and persistent congestion checking is invoked from there. So if DetectLostPackets find all packets of one pnspace within a persistent congestion window are lost but packets from other pnspace within this time window are not lost, is this persistent congestion?
I think it should only be for ApplicationData, and say that we expect the handshake timeout to be smaller than the persistent congestion threshold, so it does not need to apply to Initial or Handshake spaces. Prior to the persistent congestion concept, RTOs(and then consecutive PTOs) were used as a signal of persistent congestion, and the RTO and PTO alarm are only active when the crypto handshake alarm is not, so I think saying it's ApplicationData only is keeping with the original design.
Oh i see your point. Basically crypto timer would have been fired earlier than persistent congestion. So this is for AppData only and will apply to AppData only anyway, then I shouldn't have to worry about packets from more than on pn_space are in the persistent congestion window.
It took me quite a while to confirm that "is first rtt sample" is good. It's slightly more expansive than what I had implemented (which records This has the effect of including packets sent prior to getting that sample. But as this is only packets sent within a single RTT, it's guaranteed to be less than a PTO.

The RECOMMENDED value for kPersistentCongestionThreshold is 3, which is approximately equivalent to two TLPs before an RTO in TCP. <ins> The persistent congestion period SHOULD NOT start until there is at least one RTT sample.  Prior to an RTT sample, the duration cannot be correctly calculated.  Waiting for one RTT sample also avoids spuriously declaring persistent congestion when the initial RTT is larger than the actual RTT. </ins> This duration is computed as follows: For example, assume:


with a given connection ID always arrive at an instance that has connection state, unless that connection is no longer active.  In the case of a cluster that uses dynamic load balancing, it's <del> possible that a change in load balancer configuration could happen while an active instance retains connection state; even if an </del>  instance retains connection state, the change in routing and resulting stateless reset will result in the connection being terminated.  If there is no chance in the packet being routed to the <del> correct instance, it is better to send a stateless reset than wait for connections to time out.  However, this is acceptable only if the routing cannot be influenced by an attacker. </del>  21.10.

It's not that great that this text never said this so directly before.
NAME points out that the obvious solution to is for the node to respond to the attempted probe with a stateless reset. Clients would see a Stateless Reset while attempting to probe and not use the new path. That was, in fact, part of the original design IIRC. However, the reason we removed that was that doing so opens up an attack. An attacker can take a packet, rewrite the source address to an IP range the server always rejects, and get back a valid SRT for that CID. It can then inject the SRT toward the client to destroy the connection at any time, albeit only until the CID is rotated out. As far as I can tell, this attack isn't documented anywhere. The right fix is to note that if servers receive packets that would be associated with an existing connection but would not permit migration, they SHOULD drop the packet and MUST NOT generate a stateless reset.
I think that we do have this text. A in fact: The subtle distinction here is not that you are unable to respond, but instead unwilling. Why would you send a stateless reset when you are pretending that the route doesn't work? (Sorry, this just seems like self-inflicted harm.)
Yeah, but this is specifically not a different instance. The defense in that case is to use different keys, so the generated stateless reset isn't valid for the original connection. The section in question could perhaps be augmented to talk about behavior when an instance receives a packet associated with an active connection, but from a source that it considers unacceptable for that connection.
In the case of unsupported migration, the thinking is that a stateless reset would cause the path validation to fail in one round trip(opposed to a timeout), which is very useful if it's the only path the client can use. I think it's worth saying something about this case, but I'd I'm not sure we need normative language. If a server thinks the best course of action is to terminate the connection in this case, then sending a stateless reset is still the right thing to do.
NAME So stateless reset is a wrong mechanism, but you seem to be asking why refuse to serve traffic to a set of CIDRs if they seem to be reachable. The answer could be (a) contractual/legal reasons, (b) performance reasons (you know it will be poor).
I think the normative language is that a valid Stateless Reset MUST NOT be sent, while a connection with that CID could still be active. This covers both far-away-server Stateless Reset Oracle text quoted by NAME and the same-server Stateless Reset Oracle mentioned by NAME here. I am happy to provide a PR for this.
That's a good addition.
If the packet arrives at the right connection, there's no need to send a stateless reset at all. If a server wants to close the connection, it should just send a connection close, which is likely to be smaller and doesn't have any Stateless Reset Oracle issues.
The client could still have its original network attachment intact and could be just probing of it can migrate to the second attachment. Sending a PATHRESPONCE with CONNECTIONCLOSE will kill the connection instead of telling the client "don't migrate".
Agreed, I had in mind the case when disable migration was also sent. I think there are a number of use cases and options here, and the key thing is to ensure servers understand the implications of their choices.
I'm a bit confused in this discussion. Stateless Resets have had connection-level scope so far, and using that signal for invalid paths would be an overload. Not that I'm opposed to this on the face of it, I am just noting that as a significant departure. FWIW, I'm on board with the change to saying "a valid Stateless Reset MUST NOT be sent, while a connection with that CID could still be active". That is the right response I think.
I took a swing at this.

with a given connection ID always arrive at an instance that has connection state, unless that connection is no longer active. <ins> More generally, servers MUST NOT generate a stateless reset if a connection with the corresponding connection ID could be active on any endpoint using the same static key. </ins> In the case of a cluster that uses dynamic load balancing, it's  <ins> possible that a change in load balancer configuration could occur while an active instance retains connection state.  Even if an </ins> instance retains connection state, the change in routing and resulting stateless reset will result in the connection being terminated.  If there is no chance in the packet being routed to the  <ins> correct instance in this new configuration, it is better to send a stateless reset than wait for connections to time out.  However, this is acceptable only if the routing cannot be influenced by an attacker. </ins> 21.10.