migrates away from a local address SHOULD retire all connection IDs used on that address once it no longer plans to use that address.  5.2. Incoming packets are classified on receipt.  Packets can either be The sequence number assigned to the connection ID by the sender. See issue-cid.  An 8-bit unsigned integer containing the length of the connection ID.  Values less than 4 and greater than 18 are invalid and MUST be treated as a connection error of type PROTOCOL_VIOLATION. for different connection IDs, the endpoint MAY treat that receipt as a connection error of type PROTOCOL_VIOLATION.  19.16. An endpoint sends a RETIRE_CONNECTION_ID frame (type=0x19) to

Adds a new (optional) field, 'Retire Prior To' (placeholder name), to the NEWCONNECTIONID frame, which allows a sender to request that its peer immediately retire all connection IDs up to a certain sequence number. Once the peer acknowledges this packet, it has at most 3 PTO to follow through or risk the sender closing the connection.
First, I don't see the frame type rearrange as a very large change at all. Second, I do think the atomic assurance is important, and it's just an oversight that we aren't explicitly require Sequence Number to be greater. I didn't think of that scenario. Without the atomic assurance, I worry about the scenario where an endpoint retires all the CIDs without replacing any. It gets a lot more complicated if you have separate frames. What conditions are considered protocol violations? In order to replace all existing CIDs, the order of the frames in a single packet starts to make a difference. You have to make sure you have the NEWCONNECTIONID frame followed by the RETIREALLCONNECTIONSIDSWITHSEQUENCENUMBER_BEFORE frame.
I think I agree with NAME that we should refrain from renumbering frames (again). One of the issues regarding the proposed renumbering is that the two CONNECTIONCLOSE frame types would not have the same MSB bits; it could be a concern for certain stacks. OTOH, it is also my view that we do not need a new frame nor a new frame type. How about just making the "Retire Prior To" a mandatory field of the NEWCONNECTIONID frame? The value can be set to zero when retirement is not necessary. We do not expect a lot of NEWCONNECTION_ID to be inflight; spending one additional byte is IMO a non-issue.
+1 to NAME suggestion of making it mandatory and not renumbering frame types.
Thanks NAME NAME NAME for your feedback. I have made some changes, making the Retire Prior To field required, so we only have one frame again, added more 2119 language, cleaned up the text on ignoring reductions in the Retire Prior To field, and made it a protocol violation to have Retire Prior To greater than or equal to Sequence Number. NAME on your request to require the difference between the Retire Prior To field and Sequence Number field to be no greater than activeconnectionid_limit, that would mean the sender has to keep track of both what's the highest retired sequence number retired by the peer and what it wants locally, and the max of those two values in this field. I'm not sure the extra complexity for keep track of that is worth the cost. What benefit do you see in that requirement? NAME on the topic of the handshake and preferred address CIDs, I thought we already had text that implicitly assigned them sequence numbers. Is that not the case? What text beyond that would you like me to add?
Thanks for writing this up! Specific to this text (see additional discussion on the issue): I don't think that we need any requirement about the distance between RetirePriorTo/SequenceNumber activeconnectionid_limit. Setting a super high value would be a bit nonsensical, so some non-normative text explaining why doing that is a complete waste of everyone's time would be good, but it doesn't actually break anything. Saying "hey this is silly, consider that limit when you set this" is probably good enough just to make implementors aware of the foot gun. To the point about handshake and preferred address CIDs, I think it would be nice to have some text just as a reminder along the lines of "don't forget these have sequence numbers too" to help folks avoid confusion.
I agree that atomicity is important here, if we want to avoid the situation where all connections IDs are retired at the same time. Adding the field to the NEWCONNECTIONID frame as NAME suggested seems to be the most straightforward way to do so. I'm not a big fan of the 3 PTO requirement here. I don't see why it shouldn't be possible to retire the connections immediately when receiving the frame. There's a DoS risk here: if an attacker delays a packet with a connection ID that was retired afterwards, it can cause either a stateless reset or even a PROTOCOLVIOLATION. The better way to enforce the retiring of the connection IDs is to look at packet numbers: The old connection IDs MUST NOT be used in packets with higher packet numbers than the packet that acknowledged the NEWCONNECTION_ID frame.
NAME Am I correct in understanding that this is the way key update works? Assuming that is the case, my +1 goes here.
I think that's an implementation decision. You could certainly enforce this by requiring that . This doesn't stop a peer from acknowledging the NEWCONNECTIONID frame, but never acting on it though. Another way to implement this would be to enforce that . This would be a somewhat stricter way, since it doesn't allow the peer to ignore the frame (without inducing artificial packet loss).
I think this solves the problem since I really dislike that a delayed packet, 3PTO or otherwise, can take down a connection. However, I also dislike that this is coupled strongly to ACK behaviour. I'm not sure how key updates work but assuming it does not use ACK, ACK handling can run asynchronously in a separate task, and indeed this is how I would architect a high performance implementation. Then one could send a dedicated frame to acknowledge receiving the retire frame, but that is pointless since any packet using the new CID would have that effect. So for the time being I don't have a good answer to this. Altogether, if this facility opens up to DoS attacks and prevents ACK handling from running asynchronously, I think it does more harm than good and should at least be moved to v2 giving more time to consider the implications.
NAME the problem is that the sequence can be sparse. The client isn't required to use the CIDs in order any more, IIRC; so you could have four outstanding CIDs, but your four are sequence numbers (2, 127, 324, 325). Nick wants to expire everything prior to 300 and he's tired of waiting for you to use and discard the two early ones.
NAME I'm all for making it immediate and adding stronger language. That's exactly what I wanted to do in the first place. Initially, I was getting push back though, so I relaxed it a bit. I agree it's a bit too wishy-washy in its current form. Unless anyone has reasons why they can't or shouldn't update immediately I will go ahead and strengthen the text.
I think the main reason not to be totally immediate is when you have some distributed processing generating packets. You might not be able to guarantee that every thread/process generating packets can stop using those CIDs immediately on receipt of the signal that the issuer wants them gone. However, we already went through that with the RETIRE_CID discussion -- sending RETIRE is supposed to be the signal that you've hit your sync point and will not use them any more.
Brainstorming times when it would be valuable to not change immediately: Privacy, a client that wants to avoid linkability on different links shouldn't have to change all of its CIDs regardless of link immediately, since that would trivially link those different CIDs If we have a server behind a load balancer and it wants to rotate keys, having every client connected to that individual server change CIDs right away could betray a bunch of information I don't know if we need to go as far as recommending that clients wait some amount of time, but we could potentially (in QUIC LB maybe?) recommend that servers rotate their clients not all at once. And clients might want to change on different interfaces at different times. I do like the idea of having a packet number cutoff when the client retires the CID to simplify things on the other end.
IIRC, the conclusion we reached on the CID design team was that: When the client sends RETIRE, that's a statement that it has purged that CID and SRT; the CID will never be used again, and the SRT won't be recognized. The server can immediately declare that CID void upon receipt. If an earlier packet got delayed, it will trigger a SR, because the server doesn't recognize the CID any more. This is also true if the client messed up and sends a later packet with the retired CID; the server will discard or reset it. The client will receive the server's reset, but not recognize it as such because the SRT isn't one that it knows any more. The important thing is that the client has already deleted the SRT before sending the RETIRE and the server can safely purge state immediately upon receipt. In this PR, I think we have the right model by letting the client have a little time to make sure those requirements are true before sending the RETIRE. You MUST retire the CIDs, but you're not obligated to do so instantly.
In line of NAME comments and what I have said earlier: Implicit retirement of "Prior To" has the issues mentioned with distributed ACK processing. But the problem is in fact worse: For almost the same reason that the sender wants to retired CID's, the receiver may want to be either migrating internally to another host, or process different parts of a connection on different servers. It can be detrimental to such deployments to have a force expiration. Therefore I think Mike is right in tying this to RETIRECID. Perhaps the right model is to use RETIRECID as the sync point regardless of whether this is done voluntarily or motivated by a "Prior To" flush. Mike suggests the sender should defer RETIRECID until safe, but I think the receiver should make that call because it might have a 10 minute periodic load balancer flush or something completely different while the sender can only estimate based on RTT and the sender also wants to forget the CID ASAP while the receiver can track CID's algorithmically. The recommendation could be to invalidate CIDs from RETIRECID frames no earlier than 3PTO and possibly longer if it suits the deployment. Receiving a non-zero "Prior To" field SHOULD cause the CIDs to be flushed and retired with RETIRECID as soon as practical. The issuer of "Prior To" MAY close a connection with the error CONNECTIONID_TIMEOUT but deployments SHOULD avoid doing that aggressively unless critical to infrastructure since clients may not be able or willing to updatee CIDs fast. Note that there can be race conditions where both endpoints want to retire CID's because they are internally migrating. This can lead to problems if a CID is tied to the peers CID indirectly via source IP. We chose to have asymmetrical path migration due the complexities of this. I start to think we could some nasty corner cases here.
I'm OK with "SHOULD immediately" but I still think having an endpoint close the connection with protocol violation if a packet is delayed is a security risk for the reasons I detailed in the issue.
Personally, I don't think there is a big difference between explicitly closing the connection VS just removing the CIDs from the routing lookup, which would cause a stateless reset to be sent if the peer continues to use it. I like the explicit nature of the close, because it will be easier to debug if there are bugs; though I do concede that if the peer did locally retire the CIDs, but somehow the packets/frames just took an extreme amount of time to get there, the connection isn't penalized. Perhaps a vote? Thumbs up if you prefer connection close with protocol violation. Thumbs down if you prefer the CIDs to be removed, resulting in a stateless reset instead.
NAME As NAME has pointed out in URL, "having the use of a CID you've asked to retire trigger a protocol violation is a security risk. It allows an attacker to close the connection by delaying a valid packet." If it is the case that we would have the same issue in sending a STATELESS_RESET (I assume that is what you are suggesting), I think we need to fix the issue, rather than choosing one between the two designs that both have the same security concern.
PS. One way of resolving the security concern would be to require the use of Stateless Reset (instead of a CONNECTION_CLOSE), and also require that Stateless Reset Tokens for CID above "Retire Prior To" to be different from the tokens below or equal to "Retire Prior To." Note: the latter requirement is necessary because states that "the same Stateless Reset Token MAY be used for multiple connection IDs on the same connection."
I agree with NAME If we want to close the connection (which I still think we shouldn't mandate but I think we can allow) then we should use stateless resets, mandate each CID has a distinct reset token, and mandate that endpoints ignore reset tokens for CIDs they have retired. That'll prevent an attacker from closing the connection by delaying a packet.
That goes back to my suggestion of using epochs ("prior to" being the simplest such) because you might not want to store a reset token for every CID, only for every epoch.
I agree with NAME and NAME NAME In general, each CID is associated with its own reset token. This allows for a straightforward stateless implementation of a stateless reset token generator. I don't think it would be a good idea to revisit that principle.
NAME no you are probably right.
Thanks. All your arguments make sense. I have updated the text to say the connections IDs should just be removed after the 3 PTO, instead of closing the connection. I also added some text warning against reuse of the stateless reset tokens.
I'm not sure I know what "remove the connection ID" means on the receiver. I would remove any mention of 3 PTO as well. Simply state that if a sender keeps using a CID that its peer has asked to retire then packets can get misrouted or dropped.
NAME by "remove the connection ID" I meant that the connection IDs would be implicitly treated as retired. I'm not sure how best to word that section of the text. Suggestions are welcome. Additionally, I don't think many folks (though I'm not among them) would like the removal of the 3 PTO text. Several people have argued for allowing continued use of the CIDs for a short while.
I think the text needs to be more clear on whether RETIRE CID must be send for those requested to be retired, or if that is implicit.
NAME we have the following text already: Is there something else you'd like added?
"The peer SHOULD stop using all the connection IDs as soon as possible and then retire the connection IDs by stop sending a RETIRECONNECTIONID frame for each connection ID." "retire immediately" is a bit too strong as several have commented.
This is true, but the text currently allows for the SRTs associated with various CIDs to be identical. This becomes problematic if you're forcing retirement and want to stop recognizing some of the CIDs previously issued. However, that's a problem we already have and cover in the text: This text seems to effectively exclude silently dropping some CIDs that share a SRT with a currently-active one. It might be worth mentioning that explicitly, if we recommend a silent forget approach.
This is looking good (modulo the one extra paragraph at the end)!
I think the protocol should allow for the originator of a CID to retire it and once that packet has been acknowledged by the peer, remove it from its lookup table. I have a scenario where I want to move a connection from one "partition" to another "partition" on the same machine. I use some bits from the CID to indicate the partition ID. My problem is, I've already sent out a CID to the peer indicating the old partition ID. If I change, I'm going to get a cross-partition lookup perf-hit for the rest of time on this connection. So, I'd like to be able to pick a new partition ID, generate a new CID and send out a NEWCONNID and a RETIRECONNID for the old CID. Once the peer acknowledges this, I remove the old CID from my look up. With this design, if I need to change partitions for any reason, I only pay a perf penalty for 1 or 2 RTT. Another option instead of supporting RETIRE for my own, could be a REPLACECONNID [orignalsequencenumber]. That might be a little less disruptive for implementations to manage. Just copy over the old one with the new one and go about your business.
Would URL meet your needs, other than the fact that it's an extension rather than core protocol functionality?
NAME it might but I don't like the idea that it's a request. If the peer doesn't do this, it is causing two partitions to have cross-partition access for an extended period of time. I don't want the peer to have that type of control. With the extension, even if it claims support, what's stopping it from not actually implementing it? If the act of acknowledging the packet allows me to drop the CID, then they have a great incentive to do it properly. Otherwise the connection dies. And finally, yes, I want it in the spec. Not optional. Back to the whole problem of I don't want the peer to control my perf.
I'd be happy with adding a "If it's acknowledged and a packet with an equal or larger packet number uses the CID, you MAY close the connection." ie: A receiver MUST honor it, the sender MAY enforce it.
That would definitely make things better, but only if they actually supported the extension.
This goes back to a lot of the core concept around vending a CID and having that be a commitment that you're willing for the other side to use it. If we're going to add a bunch of extra machinery around it, maybe it's worth reopening the original conversation? Alternately, it also seems like a lot of these issues would be resolved by a "here's a bunch of new ones, please kill all of the old ones", since in almost all of these situations we're saying "I encoded something in the old ones that is no longer valid or helpful".
NAME that's why I kind of like the idea of REPLACECONNID. Just replace the bytes from whatever CID was previously shared.
The mechanism for REPLACECONNID is not complicated, but I'm not yet convinced about its need. So, if we treat the two partitions as two servers, a commonly used strategy for migrating load is to send a GOAWAY to the client and have new connections be routed to the new server. Is this not adequate for you, NAME
The partition can be on the same physical machine (different processor/numa node). Changing the partition ID allows for dynamic rebalancing of load across these more easily.
I'm a bit torn here - I don't think internal process optimisation should be on the wire, but with 96 cores and counting, and possibly hardware optimized queue delivery, there is probably a reasonable case for it. There may also be a case of handover on the internal network. But this is only a use case consideration. I haven't considered whether NAME proposal is the right way to do it.
Allowing for an endpoint to edit/replace existing CIDs can allow all sorts of dynamic rebalancing logic on the LB side of things. Perhaps server ID is changed for a back end server and it wants to update all CIDs to encode the new ID appropriately. The current design of give something to the peer and you're stuck with it forever, until they decide to stop using it seems overly restrictive. Additionally, if we support inplace CID replacement, the actual implementation logic to support this should be extremely trivial; just memcpy over the old one with the new one for the most part. The complexity only comes from endpoints that want to actually replace their own CIDs. They would need to have synchronization logic for it and so on. Endpoints that never need to do this don't accrue any additional cost.
Even if you do a REPLACE for future IDs, you can't currently require that the client stop using the current CID. So given that you have to wait an arbitrary amount of time for connections to "drain" anyways, I'll go back to my earlier comment: why not send GOAWAYs for connections you want to shed to the new partition? If you really want to have dynamic balancing, it seems to me that there'd be more to gain by forcing the client to stop using the existing CID and using a new one. You would want to REPLACE IDs as well, but without forcing the client to move off the current CID, most connections won't move since most connections are unlikely to use a new CID unless the client is migrating (an uncommon event).
Why can't I require the client update the CIDs? As discussed earlier in the comments, the design would be that the client MUST update and if the server receives an ACK for a packet with the REPLACE_CID and is still using an old CID, it will terminate the connection. HTTP/3 is not my only QUIC protocol, but either way, I want a QUIC solution, not a higher layer protocol solution since QUIC is the layer doing this load balancing.
NAME I think your use case is real, and the implementation work client side seems quite small, but it is one more CID feature and I'm not sure what fraction of implementations want to use this? I can imagine we'd use it on the server side if it was present, though.
As I've pointed out, I don't think the mechanism you're asking for is adequately useful for the problem you're trying to solve. I'm not opposed to solving the problem, which is why I'm trying to understand the problem itself. As I said, you would still want a mechanism to replace the existing CID, without which you are stuck draining open connections that aren't changing CID through the life of the connection. I expect this to be fairly common -- most clients are unlikely to change destination CIDs unless there's migration involved. Given this, I still don't see how simply replacing future CIDs is adequate. NAME : Do you think the proposed mechanism, which does not replace the current CID, is useful? Where do you think you might use it?
There is also the concern of conflating migration with targeted load balancing. Even if you did replace the current CID, what happens if the client migrates frequently. All future CID's would be drawn from a pool of CID's that might point to the wrong endpoint process. This means you would have to split up the CID's into two parts: an address and a sequence number, both appearing random externally. You would then change the address part, but leave the sequence selection to the client. This appears to be rather overengineered. Alternatively the current CID could be replaced post migration which might just be few roundtrips of sub-optimal performance before hitting the proper destination, under the assumption that migration is is not that frequent. This also shows that having the server initiate a migration (which is not currently possibly) is insufficient unless the client future CID's are tied to a specific server path.
NAME The mechanism would have to include a way to request/force the client to change CID ASAP to be useful. Only a way to retire outstanding CIDs is fairly useless in my opinion. I assume the mechanism to enforce this would be a GOAWAY followed by a connection close containing a FAILEDTOROTATE_CID error if the client didn't change in time.
Discussed in London; reservations about whether this is desirable / necessary, but NAME to make a proposal. Atomic nature a particular concern, as is the mandatory nature of this.
REPLACECONNID sounds simple, though we need to remember that this frame might arrive out-of-order. That means that this frame needs to have it's own sequence number and the receiver needs to retain the largest sequence number for each of the Connection ID that it maintains...
REPLACECONNID is interesting, but we need to go through the rest of the issues surrounding changing that base assumption. It's potentially worth working through the fallout from doing that and seeing how icky it gets.
I think we should not do this as this stage, thinking of how messy it was to evolve path migration. Also multi-path might complicate things further. Suggest v2.
I agree, that because of reordering, we can't require immediate usage of the new CID after the packet/frame is acknowledged. Instead, perhaps a 1-RTT timer, after the REPLACECONNID is acknowledged, then I just remove the old CID from my routing table, which means those packets would then be dropped. On the side that receives the REPLACECONNID, it would have to ensure that it replaces immediately though, because if it continues to use the old CID after ACK'ing will eventually get a stateless reset. I have several performance problems I am trying to solve with this right now, which I am trying to push a solution in v1. If the solution to this is ever optional (different version or extension) then it's not going to be a valid solution, I would have to go back to the drawing board.
I started writing up a PR for REPLACECONNID and I realized that the frame would end up being pretty much exactly that the same as NEWCONNID. So, one way we could support REPLACECONNID semantics would be a smaller change to allow for NEWCONNID to overwrite a CID sequence number. Practically that's exactly what REPLACECONNID would do, but adds a new frame and a good bit of new text. So, would folks rather change NEWCONNID to allow for overwriting, or add a completely separate frame?
I am not sure if that is the case. Consider the case where the sender first decides to replace CID at sequence N with value X, then decides to replace that CID with value Y. I think we need a way to distinguish if which of replace-to-X replace-to-Y was sent later. So, we need a sequence number for each of the CID that is being replaced.
If it only takes effect if the CID is still current it avoids a lookup to scan all queued CIDs. This would also address NAME concern I believe.
NAME It doesn't, because NEWCONNECTIONID specifies the CID being replaced by using the sequence number, not the current value of the CID being assigned for that sequence number.
Yes, but there is a proposal to allow for a special case where the CID is used explicitly. I'm not sure that would cleanly map to the same frame given that, as you say, the other CIDs use numbers. Also, it wouldn't allow a global reroute if a process moves. All other pending CIDs might be void. Hence a flush all CIDs if current CID match might be needed.
REPLACECONNID would have the exact same reorder problem. One solution is that the sender must not update a connection ID until the previous packet/frame was acknowledged. The other possibility would that receiver keep track of the packet number the connection ID was received in. Always use the CID in the most recent packet number.
I’d really like to avoid doing any of these. This would create a lot of additional complexity.
I'd prefer to have the sender have the responsibility, and in that case, only implementations that actually want to change their CID gets the extra complexity.
I do not think that either of the two would work considering the fact that an endpoint is permitted to retransmit out-of-date frames as they are. Consider the case where an endpoint first sends packet(pn=x,REPLACECONNCID(seq=y,newvalue=a)), then sends packet(pn=y,REPLACECONNID(seq=y,newvalue=b)). If pn=x is deemed lost, then the endpoint is permitted to send(pn=z, REPLACECONNCID(seq=y,new_value=a)), where z is greater than y.
Again, if the responsibility is on the sender side, it's up to the sender on how to enforce it. If you support the feature, you cannot blindly retransmit these frames.
After talking to a few folks, I'm now leaning towards the following solution: Update NEWCONNID frame to have two frame types (or we can add a new, separate frame) where the second frame type indicates the presence of an additional field: (we can bike shed the name). This new field indicates the sequence number of a CID that is being replaced by this new CID (which has a new sequence number too). The sender sends this frame for any/all outstanding, non-retired CIDs that is needs to replace. The receiver is expected to quickly, if not immediately respond with a RETIRECONNID for all the CIDs that the sender is replacing. Once the NEWCONNID frame is ACKed the sender may drop all the retired CIDs from it's routing table after some time (3 RTT?) even if the peer hasn't retired them; OR it can send a HTTP goaway. One note on the CID flow control: Even if the maximum number of CIDs have been given to the peer, an endpoint is allowed to retire one or all of them with this NEWCONNID frame, and the peer is expected to accept these new CIDs; since they are replacing the old ones.
This seems sensible. Why not always specify RetireSequenceNumber? This is nice because it further incentivizes peers to provide them in order, since they're retired in order. If we wanted, this would make enforcing the active connection ID limits trivial(NAME ), because a receiver of NEWCONNECTIONID would compare the incoming ID to the RetireSequenceNumber and if the gap is larger than activeconnectionid_limit, close the connection.
I think the ACK optimisation should be avoided. A CID is retired when it is retired, not before. This also allows peers to opt in to their own schedule. They might have a complex distributed pipeline that is not synced with ACK logic. It is the senders headache since it is trying to force its own optimisation onto the peer. While I can see the overall idea, I'm a bit concerned about race conditions here. If you retire 5 CID's the might not all take effect immediately so when the first one is retired, the next available CID is used even if it is scheduled to retire. Forcing the peer to sync on this requires processing multiple frames concurrently assuming they all land in the same packet for a start.
My primary goal of this is to force the peer to change. I'm OK if it takes a little time, but I need them to change. If they don't I will kill the connection. If we don't have any enforcement, then peers have no reason not to just ignore this and effectively nullifies the feature.
You can get around the race condition by assigning an epoch to CIDs and retire an entire epoch at once. Obviously you'd need to prime with new CID's on a new epoch first, but you can do that in the same frame as the one retiring an epoch. This might also work better on multi-path going forward.
Thanks for writing this up, NAME I think this is going in a good direction. Ideally, RetireSequenceNumber would either mean "SHOULD retire as soon as possible" or "MUST retire", we can debate that as necessary. To NAME comment, there still needs to be a way to increase the number of CIDs that are active (especially at the beginning of a connection), but I think having the field always be present would be fine. I think the key idea here is that, by asking the other side to retire the CIDs, we can manage to avoid any ACK-tracking or other draining scheme. Instead, we request the peer to use the existing mechanism to remove the old CIDs and all of the logic currently in place for that just works for this too. Instead of asking the sender to track the ACK for NEWCONNID, I'd rather have it just wait for the CID to be retired, and if it becomes untenable to do that, it can close the connection (either with no error, or we can define an error code saying "you didn't retire, please connect again").
Yes, I think that is fine. It is just hard to place a hard limit on that.
Why does the replacement need to be coupled to the retirement request? Why not say "please retire all with seqno lower (or equal to) X", then provide a new connection ID?
NAME I presume you mean NAME proposal (URL). I think that can be considered simpler to the RETIRECONNID proposal, because "retire all with seqno lower than X" frame removes the need to retain per-CID sequence numbers.
NAME my main problem with that design is the CID flow control. If an endpoint has currently maxed out the peers max number of CIDs, and it wants to replace them all, what then? If you don't have a single frame to both retire and replace, then it looks like we have a problem. You obviously can't just retire all outstanding CIDs. Then they have no CIDs to use. You also can't just give them more CIDs first, because the limit has been reached, so they will ignore the new ones.
This is one way to implement the epoch I was referring to in the above.
NAME you can have two epochs: old and new each with a limit. This allows you to replenish CID's in new epoch before retiring the old.
NAME I think I'd prefer to have a "please retire" come with a replacement since otherwise there's not any guarantee that they'll be replaced. Having them in the same frame also makes it easy to replace ones without depending on ordering between frames of a "retire" and then "add new one" if you're already at the limit (as NAME said).
There is never any guarantee that the connection ID will be replaced. Coupling them just complicates the design. The idea that you can provide replacements without blowing the limit is strange. You are still providing extra connection IDs. For the case that NAME cares about, the low water mark is probably fine because you will want to send the retirement before filling up the active connection ID set. I think that it is enough to provide a retirement signal and then wait for the peer to retire old ones and replace them on a regular cadence. Worst case, that means that you have to wait a round trip; but that's not going to be a problem for the RSS scenario Nick described.
You're providing an extra ID and have the ability to retire at the same time, so you can easily avoid blowing the limit. This might actually simplify my implementation, because I could store a fixed number of a CIDs in a ringbuffer of the size I specify instead of a hashmap.
I also think the simplest solution to implement is the NEWCONNID frame that replaces an existing CID. NAME if the is always there, what do you set to it when adding a new CID that isn't actually replacing anything?
Set RetireSequenceNumber to 0?
Is that not the first CID exchanged during the handshake? We need to be able to replace that one.
We could make the semantics: Retire CIDs under this number?
I think I prefer the semantics of retiring everything up to a stated offset. That avoids messing with sequence numbers of sequence numbers, which just seems cleaner. The field needn't be always present; the value can only advance, so if you've ever seen a value, you need to retire everything below that value. Corner case to consider: What if I've received CIDs 1,2,3,5,8 and I get a =7? Do I need to retire 4 and 6 that haven't arrived yet, or do I need to retire them upon receipt?
NAME I think you need to retire 4 and 6 immediately. Overall, I think this proposal is nice. At the cost of a single varint, we achieve clearer connection ID management semantics and the ability to retire CIDs proactively. I can see this being pretty useful. One note is that this means a client really has to support changing CIDs, whereas at the moment a client could support it during the handshake and never again. I don't think that's overly onerous, but it's worth pointing out and may be a pro or a con depending upon one's perspective.
I agree with NAME that you need to retire everything (and practically, remember the limit in case you get a heavily reordered packet offering the old one later). If folks are happy with this design, I can write up a PR.
Thanks for writing up NAME I think either of the following two approaches could be a good path forward (discussing here rather than on the PR): The current PR, each new connection ID comes with a "retire CIDs before sequence number X", which keeps us from having issues around holes in the set of valid sequence numbers for a CID. This is kind of nice, because we can be a little looser with how we handle being at the max number of CIDs and needing to send some more/add new ones. Example: I want to retire all the CIDs that are outstanding. I've issued 1, 2, 3, 4 in NCID frames and we've got the handshake at 0. I send a single packet with: Note that sending for all could be specified as valid or invalid, although if valid it doesn't really help with requiring replacement. We'd also then want to consider writing a bunch of other somewhat messy text to require retiring them in order from oldest to newest, which seems somewhat icky. A new frame for just the retire side, called something like . This would be a request, and an endpoint receiving it SHOULD retire immediately, along with a warning that the other endpoint may choose to close the connection if it doesn't. The endpoint also SHOULD move to a new CID if the one it's currently using is requested to be retired. It's probably worth being a little bit careful about how strongly we make not moving a protocol violation, because it seems like it wouldn't be too difficult for an endpoint whose peer had to migrate off the one it was currently using to end up receiving packets from the old CID for a good while/draining period. If we do (2), in order to have the guarantee that they're replaced, we should just include that all frames MUST also be accompanied by a frame, and have the receiver close the connection if that's not the case. (Timing/[not] keeping things in the same packet strikes me as a part of the text here that needs to be written carefully.) Same example from above: In my mind, the common text between both approaches is: The endpoint being asked to retire CIDs SHOULD retire them immediately, and move to a new CID if necessary. If not, the other end MAY close the connection (plus note about not being too aggressive here to handle draining). The endpoint asking to retire them MUST replace them. If not, the other end closes. How we handle "bad" sequence numbers being sent (too low, also a little weird to send a number higher than the peer's limit away from what's active). How you handle when you're at the max specified by the peer's limit Any preferred address and handshake CIDs should probably explicitly be called out as affected, although they do have sequence numbers If someone requests to retire CIDs that you haven't seen yet, you respond to frames containing them by immediately retiring them. This doesn't add too much state since you just need to keep a single counter of the highest invalid/lowest valid sequence number.
I don’t think we should have dependencies on other frames, then I’d rather have a single frame. It would mess up retransmission logic etc. On 5 June 2019 at 19.21.11, Eric Kinnear (EMAIL) wrote: If we do (2), in order to have the guarantee that they're replaced, we should just include that allPLEASERETIRECONNECTIONID frames MUST also be accompanied by a NEWCONNECTION_ID frame, and have the receiver close the connection if that's not the case. (Timing/[not] keeping things in the same packet strikes me as a part of the text here that needs to be written carefully.)
NAME why couldn't we make it a "MUST immediately retire" instead of a SHOULD? What reasons would an endpoint have to ACK the packets but not retire the CIDs at the same time?
NAME in some implementations the ACK logic is not next to the connection ID management logic. For some of these implementations it would be very hard to ensure that the ACK isn't sent with the CID you'd like to retire. "MUST immediately retire" would add significant complexity to those implementations for very limited benefit. Also, having the use of a CID you've asked to retire trigger a protocol violation is a security risk. It allows an attacker to close the connection by delaying a valid packet. I like the idea of adding a field in NEWCONNECTIONID frames, but we should make it advisory and not mandatory. Sending it tells the peer that if they keep sending with this CID packets might not get through at some point. But I don't think we should have MUSTs, immediatelys or protocol violations involved. Is there any reason that wouldn't serve the needs of your use case?
I argued about the same as NAME in the associated PR.
I'm going to try to answer all the comments/questions from both the Issue and the PR here, so as to not clutter up the PR. It now seems that most people actually prefer the immediate switch of the CIDs instead of allowing for delay. I agree key update is a good example / pattern to follow here. You get a packet with the NEWCONNECTIONID frame to replace existing CIDs, you locally update all CIDs and then you mark the packet for ACK. NAME I understand that some implementations might have extra difficulties, but I'm not sure why it would be that much different from key update. Additionally, you only ACK a packet after it has been fully processed, so if you process the NEWCONNECTIONID frame before the "packet fully processed state" and update CIDs there, then there shouldn't be an order of operations problem.
As NAME said: Previously this had been proposed as a way to think about it, and I think it's probably the right balance to strike. You don't work to hurt the client if they don't switch right away, they may have reasons for a slight delay there. A client knows that the request from the other side serves as fair warning that the CID may not work indefinitely. It's still up to the client to use RETIRECONNECTIONID and all of the draining logic there still applies -- we don't need to reinvent additional signals for that.
Immediate is a little strong. There are valid implementation reasons that an "immediate" switch might not be possible -- or at least impossible to guarantee. The immediacy guarantees are that when the implementation sends RETIRE, it has completed whatever is needed to ensure everything is done. An ACK doesn't logically carry the same guarantees -- it just guarantees that the consumer knows about the request. It can be mandatory for the client to retire the CID, enforced by permitting the issuer to kill the connection if it's not done soon enough, but mandatory immediate retirement probably doesn't work.
Just to pile on here. If there is an implementation that is distributed, with multiple entities producing packets (which is entirely possible with some coordination), then retirement requests can take a little time to propagate. I think that the ACK here should be taken to mean, as NAME says, I got your request, not coupled so tightly to compliance. I don't believe that we have any other such things in the transport. STOPSENDING is perhaps the closest analogue, but that doesn't require that you send a RESETSTREAM in the packet that acknowledges the STOP_SENDING.
I don't have a strong opinion on this, but the new discard keys PR that landed has that shape: URL Later packet numbers not using a given key is fairly similar to later packet numbers not using a given connection ID. Enforcement seems comparably or slightly more difficult, so one could go with MUST retire/MAY close the connection. NAME Once you're talking about distributed peers, I think we're well in the space of multi-machine multipath? Or is there some other use case you have in mind?
NAME I think this is a little bit different, since we have a mechanism to retire CIDs and it this is a request to use that mechanism, not an implicit retirement of the CID? It seems like it would be fine to require that after a CID is retired that later packet numbers not use it (I think that's already a requirement?), but doing anything based on the ACK of the "please retire" message doesn't seem like the right path forward here.
NAME we have strengthened the text, so that it's no longer a request, but a requirement that the peer retire the CID. I also agree with NAME that this is very similar to key update, and if an implementation can synchronize keys and packet numbers, I see no reason why CIDs and packet numbers also cannot be synchronized. Practically, for every packet you send, you pick a CID, a key and a packet number to use at about the same time. The only scenario that I've heard that gives me some pause is if the peer is currently trying to probe on a new path when you are trying to force the CIDs to be retired. It might be easier for the probing logic to continue using the CID it started the probe with. But I'm still not completely convinced. For one, what if that new path has a MUCH bigger RTT than the old one? That would make it a lot more likely it could hit the 3 PTO timer on the primary path. Also, I see no reason why you couldn't immediately switch the CID used for probing as well.
I'm fine with the "please retire" semantics, but wanted to point out an existing case in the QUIC specs we say you must retire something in a similar circumstance. I hate timers, so I suspect my implementation will do that even if the spec doesn't require it, because I believe it's more straightforward to implement. If you supply a CID with sequence number X and I've specified that I'm only going to store Y sequence numbers via activeconnectionid_limit, can you assume the peer is unlikely/unwilling to use CIDs with sequence number < X-Y(modulo reordering). That would be true for my implementation, but possibly it's also related to the 3PTO issue?
I don't think this is analogous to key updates. Due to the design of key updates, when you receive a packet with a new key phase, you need to install the new keys, otherwise you just can't decrypt it. The requirement to start sending using those new keys is not much of a stretch. Here, when you ask me to retire a CID, there's nothing I actually need to do to keep parsing packets. Asking the peer to retire a CID is closer to STOP_SENDING than it is to key updates. That said, our implementation will probably retire the CID immediately, but I really do not want text in the spec about closing the connection after some timer, because that opens up new avenues for attack.
NAME what attacks did you have in mind? Obviously, our implementation should not do that if we think it creates new attacks.
With the current text, we aren't explicitly closing the connection, but just implicitly retiring the CIDs after some time, if the peer doesn't do it themselves. So long as the new CIDs have different stateless reset tokens than the CIDs they replaced, I don't see any way for an attacker to cause the connection to be killed.
NAME Consider a MITM attacker that can intercept packets (prevent them from arriving now, and save them for replay later). Assume the server says "please retire CID X", the client switches from CID X to CID Y, and the attacker intercepts the last packet the client sent with CID X. Then the attacker waits 4 PTO and sends that packet to the server. If the server used the same reset token for both CID X and CID Y, then the server will send a stateless reset and the client will close the connection upon receiving it. There is no text in the spec today that says the server MUST pick different stateless reset tokens for different CIDs. Therefore, the ability to request to retire CIDs does open an avenue of attack similar to TCP RST. If we want to have a safe mechanism to request retiring CIDs, then we have two solutions: (1) we require the peer to retire the CID immediately (2) we make this a best-effort request with no enforcement of actually retiring the CID As discussed earlier, (1) is not practical. What does "immediately" mean when it takes time for packets to travel between endpoints? If it means send it on the same packet that carries the ACK, what happens when that packet gets lost? Do I need to send RETIRECID on every single packet in this window until one gets ACKed? I think this quickly because a mess full of corner-cases with security vulnerabilities at play, and I'm not confident we can get to the bottom of it rapidly. Option (2) is similar to STOPSENDING. It's a way of telling the peer that they should do something because what their sending won't do anything useful. NAME I understand why you'd rather have (1) but I don't think the benefits you'll get from it outweigh the security risks. Even if we pick (2), I think everyone will implement this as retiring the CID either in their next outgoing packet or maybe at worst a couple packets later. And that gets you what you want without adding security risks.
NAME I agree with you. My question to you is: what in the current text do you not like? You mentioned before that you'd like me to remove the 3-PTO text. I'm not sure I can do that. If I remove that text, then how quickly is the endpoint requiring the retirement supposed to wait before just retiring the CIDs any ways? If you're saying it's never allowed to retire them anyways, then that's an attack vector by the peer, IMO.
You can start retiring when you receive RETIRECONNECTIONID.
NAME I would phrase the text as:
What if the peer never retires the CIDs? You're saying I'm never allowed to eventually drop the CIDs anyways? The way you have the text, the feature is optional, as long as you're willing to accept a possible performance hit.
I like STOP_SENDING as an analogy overall. NAME I made a suggestion on the PR about what the client should do if it receives a stateless reset token for a retired connection ID. I think that may both allows the sender to immediately retire and provides the peer some advice on what to do if it sees a reset that's associated with an old CID? This approach basically allows the server to wait as long as they think is necessary, and if they really need to stop using the CID, they can send a stateless reset to force it.
NAME That is the catch-22, but retiring them before receiving a retired frame is somewhat pointless. Closing the connection is a security hazard.
I'd say, if you have to wait for the retire frame to remain operational. If you give up, you should use a reset not a close to avoid the security risk.
NAME I think we're all agreed now that explicitly closing the connection is not the answer, because of the security concerns with a delayed packet. But a stateless reset doesn't have those security concerns. Receiving a stateless reset sent in response to a packet using the retired CID only effects the peer if they haven't actually retired the CID yet. NAME STOPSENDING isn't quite the same. When I send a STOPSENDING frame, I've already made the decision to start dropping everything else I receive for that stream on the floor. So by the time the peer receives the frame, a whole flight of data might have been dropped. And we're all OK with that, because it doesn't effect the overall connection state. That's definitely not the case with connection IDs. If I were to start dropping all packets with those retired CIDs first, then best case I lose an entire flight of packets. Much worse if my packet was lost as well. In my scenario, I need to dump these CIDs right now, but the peer is still using them, so I need to give them some time to make the change. I keep the CIDs in my lookup/routing tables and send the frame out telling them to retire. Eventually they acknowledge the frame, so I know they got the new CIDs and the requirement to retire the old ones. This is where we seem to start having differing views. I cannot allow a peer to go on forever using the CIDs I told them to retire. Continued usage of those CIDs is causing all kinds of performance issues, with cross processor locking/contention, for all connections on the effected processors. Since this scenario should be infrequent, I'm OK to accept the perf hit for a while, but eventually enough is enough. I want the text to be clear that if a peer doesn't switch CIDs, then the connection will go away eventually.
NAME yes, but I suggest that you wait for retire frame unless you run out of patience rather than closing you eyes and counting to three. If you get tired of waiting, you can start to reset, but that may depend on the endpoint more than a specific 3-PTO timeout. Hence it goes both ways: receiver should make an best effort to be timely, and sender should have a grown ups patience.
NAME Here is part of your original description of this issue: If the peer keeps using a CID you want to retire, the only impact is performance. So this is in line with what I proposed: keep this feature a SHOULD that is not enforced. Any attempt at enforcing this will open up security risks.
The more I think about this problem, the more pointless it seems to send a stateless reset, especially one that should explicitly be ignored, and one which the sender already knows will have that effect. It's not like that implementation that receives a stateless reset will say "oh bummer, I forgot, I better hurry up!". No, either it is not implemented, or more time is required. A reset just becomes another way of retransmitting a NEW CID frame with a prior to field. So ultimately I think NAME is right in stating "a Should that is not enforced".
NAME can you explain the security risk of responding to an old CID with a stateless reset, sometime after the NEWCONNECTIONID frame was acknowledged? NAME if the peer continues to use the old CID after acknowledging the NEWCONNECTIONID frame, why would it assume the "sender already knows it will have that effect"? The peer doesn't seem to be doing as expected already, why assume anything else at that point? If the peer is ignoring the stateless reset AND continuing to use the old CID, I don't care if they mess things up for themselves. It has solved the performance problem affecting ALL connections on the related processors.
NAME the issue is that an attacker can close the connection if the stateless reset tokens are the same for different CIDs. Perhaps this wording gets you what you want: This allows you to implement the enforcement you want while keeping the spec written in such a way that most implementations don't accidentally allow this attack by being overly enthusiastic in enforcing this.
NAME looks good to me. I will update the PR. Thanks for the proposed text.
Technically, the SRTs don't all have to have been distinct, there just has to be zero overlap between the Retire Prior To set and the still-valid ones. So you could do as NAME has alluded to earlier, using the same SRT for a period of time, then forcing the retirement of all of those because you changed some input to your generation function, and everything you issue new has moved to a different SRT.
Not Martin, but I think this could as easily be different processes/partitions on the same machine. With UDP, they could all be sending from the same source port, after all. Regardless, I think 3PTO is a totally reasonable period of time to expect them to get their coordination sorted out.
Latest changes are looking great, thanks for driving this NAME Let's merge and we can update any further editorial nits when we next do a pass.

migrates away from a local address SHOULD retire all connection IDs used on that address once it no longer plans to use that address. <ins> An endpoint can request that its peer retire connection IDs by sending a NEW_CONNECTION_ID frame with an increased Retire Prior To field.  Upon receipt, the peer SHOULD retire the corresponding connection IDs and send the corresponding RETIRE_CONNECTION_ID frames in a timely manner.  Failing to do so can cause packets to be delayed, lost, or cause the original endpoint to send a stateless reset in response to a connection ID it can no longer route correctly. An endpoint MAY discard a connection ID for which retirement has been requested once an interval of no less than 3 PTO has elapsed since an acknowledgement is received for the NEW_CONNECTION_ID frame requesting that retirement.  Subsequent incoming packets using that connection ID could elicit a response with the corresponding stateless reset token. </ins> 5.2. Incoming packets are classified on receipt.  Packets can either be The sequence number assigned to the connection ID by the sender. See issue-cid. <ins> A variable-length integer indicating which connection IDs should be retired.  See retiring-cids. </ins> An 8-bit unsigned integer containing the length of the connection ID.  Values less than 4 and greater than 18 are invalid and MUST be treated as a connection error of type PROTOCOL_VIOLATION. for different connection IDs, the endpoint MAY treat that receipt as a connection error of type PROTOCOL_VIOLATION. <ins> The Retire Prior To field is a request for the peer to retire all connection IDs with a sequence number less than the specified value. This includes the initial and preferred_address transport parameter connection IDs.  The peer SHOULD retire the corresponding connection IDs and send the corresponding RETIRE_CONNECTION_ID frames in a timely manner. The Retire Prior To field MUST be less than or equal to the Sequence Number field.  Receiving a value greater than the Sequence Number MUST be treated as a connection error of type PROTOCOL_VIOLATION. Once a sender indicates a Retire Prior To value, smaller values sent in subsequent NEW_CONNECTION_ID frames have no effect.  A receiver MUST ignore any Retire Prior To fields that do not increase the largest received Retire Prior To value. </ins> 19.16. An endpoint sends a RETIRE_CONNECTION_ID frame (type=0x19) to


It is RECOMMENDED that "initial_max_bidi_streams" be no smaller than 100, so as to not unnecessarily limit parallelism. <del> HTTP/3 does not use server-initiated bidirectional streams; clients MUST omit or specify a value of zero for the QUIC transport parameter "initial_max_bidi_streams". </del>  6.2.

per London discussion. Removes the mandate that clients send a TP of zero for and replaces it with an error on stream open unless there's an extension permitting it.
There are presently several kinds of very similar error when creating streams: HTTPUNKNOWNSTREAMTYPE MAY be used to tell a peer to STOPSENDING on a stream type that is unknown by the local peer. It is not a connection error. HTTPWRONGSTREAMCOUNT MUST be used if a peer tries to open more than one critical stream (control, QPACK). Connection error. HTTPWRONGSTREAMDIRECTION MUST be used if a client tries to open a push stream. More generally this could be used if a peer attempts to open a stream type that is not allowed to, but push is the only example today. Connection error. The suggestion is to rationalize these into a single error code - HTTPSTREAMCREATIONERROR - that can be used in STOPSENDING or CLOSECONNECTION. The impact of merging COUNT and DIRECTION is minimal because implementations fundamentally have to open streams correctly in order to interop. PS. The eagle eyes will notice I omitted the HTTPDUPLICATE_PUSH creation error. This is on the chopping block in . I class it differently because it is about parsing information after the stream type.
Should a client respond with STREAMCREATIONERROR when it sees the server opening a bidirectional stream? At the moment the designated error code is HTTPGENERALPROTOCOLERROR, but STREAMCREATIONERROR might be a better fit. Or if we are not going to use STREAMCREATIONERROR for server-initiated bidirectional streams, I think we should better rename it to UNIDIRECTIONALSTREAMCREATIONERROR to avoid confusion.
This relates to some earlier discussion on URL and its now-merged PR URL We only recently added text to specify the use of HTTPGENERALPROTOCOLERROR in this case, I think it fits to modify this to STREAMCREATION_ERROR.
Changed in URL
I don't think that we need the MUST here. An extension might want to use those streams. It should be enough to note that these aren't used.
Would the scope of this change include adding text on how core HTTP/3 implementation should treat the reception of data on these streams?
It probably has to, yes. That's something we generally do with most of our MUST statements anyway.
This might be a transport question, but how should a client respond to a server trying to open a server-initiated bidirectional stream? Assuming that the client's initialmaxstreamsuni is zero, a server would send a STREAMSBLOCKED frame. In response, the client would send a MAXSTREAMS frame with the value set to zero. However, IIUC, this MAXSTREAMS frame merely indicates that the resource is temporary unavailable. That means that the server would be blocked. To rephrase, the transport does not provide a way for an endpoint to reject a stream, without letting the peer to actually send a STREAM frame. Are we fine with that? If we think it as a problem, maybe we should change the definition of initialmaxstreams TP. For example, we could specify that the omission of the initialmaxstreams TP indicates rejection of the corresponding type of streams being opened (instead of interpreting the omission as "zero").
I don't think the transport should police what streams are opened beyond managing resource consumption. It may well be, in some other app protocol, that a server is free to open bidi streams, but only if they start with a certain header, or none at all. The app protocol should make both endpoints agree and what is permitted and police accordingly. An application layer could have credits similar to MAX_STREAMS to permit opening certain streams with or without a header. The transport need not deal with this.
Right. The transport only needs to concern itself with policing stream limits. The question here is more about how the use of h3 constrains future extension. As Mikkel says, if there is a use for the server-initiated streams, we shouldn't constrain that.
On the other hand, an extension could modify this requirement, too. Much of what's in the base spec is for the unextended protocol, and I think this is an instance of that. However, this is a MUST which we don't permit the server to enforce, which argues for making it a SHOULD instead. Maybe SHOULD omit/zero, "unless an extension which defines their use is being offered."?
MUST not accept any bidirectional streams that are not explicitly permitted by extensions. I'm not sure about stream header formats here - it would be easy to create conflicts across extensions.
I'd suggest we define those to be the same format as client-initiated ones, and use a column in IANA registry for listing which party can be the initiator. This way, extensions get a uniform way of defining those streams.
Discussed in London; spec shouldn't prohibit or require any particular value, but should say that these streams have no meaning in the protocol and can only be used by negotiation. If you get one without having negotiated their use, it's a connection error.

It is RECOMMENDED that "initial_max_bidi_streams" be no smaller than 100, so as to not unnecessarily limit parallelism.  <ins> HTTP/3 does not use server-initiated bidirectional streams, though an extension could define a use for these streams.  Clients MUST treat receipt of a server-initiated bidirectional stream as a connection error of type HTTP_GENERAL_PROTOCOL_ERROR unless such an extension has been negotiated. </ins> 6.2.


being used, the initial value of each server setting is the value used in the previous session.  Clients MUST store the settings the server provided in the session being resumed and MUST comply with <del> stored settings until the current server settings are received. </del>  A server can remember the settings that it advertised, or store an integrity-protected copy of the values in the ticket and recover the A server MAY accept 0-RTT and subsequently provide different settings in its SETTINGS frame.  If 0-RTT data is accepted by the server, its SETTINGS frame MUST NOT reduce any limits or alter any values that <del> might be violated by the client with its 0-RTT data. </del>  7.2.6.

My issue is the following: The current HTTP/3 specifications say that a SETTINGS frame must be sent as the first frame of the control stream directly following QUIC connection establishment. Depending on how connection establishment is defined, this could result in some awkward client-sided behavior. There are two ways to interpret connection establishment as far as I can tell: The moment the server has 1-RTT send-keys available (i.e., after Client Hello) On receipt of the client finished TLS packet In the first case, the server can specify its settings before the client makes its first request. In the second case however, the client can already send its first request before having received the server's settings. To handle this case, the specifications say the following: Now say for example that a client wants to use Firefox's prioritization scheme and connection establishment is defined in the second way (receipt of client finished packet). Ideally it would use placeholders to structure the dependency tree, but as it has not yet received a settings frame, it is not allowed to do so (default placeholder count is 0). This gives the client two options from what I can tell: Wait for the settings frame to arrive (and therefore block, which isn't exactly elegant) Use a fallback prioritization scheme and reprioritize requests later when the settings frame has been received and it allows the placeholders the client wants to use (which we'd also ideally like to avoid). As neither of these options are preferred, the importance of the definition of connection establishment is highlighted even more. My request is, if a clear answer has already been thought out, to add a more verbose description of what connection establishment actually means in this situation.
I agree that this is an unnecessary complexity, and I think might solve the issue.
The first is true -- the server SHOULD send its SETTINGS frame immediately. However, any packet can be delayed, so the client can't count on having seen SETTINGS prior to being able to send a request. This constrains the use of placeholders, header compression, extensions, etc. The current design permits the implementation to choose -- block on the SETTINGS frame (which used to be required) or send requests under the (very conservative) default constraints and do better once SETTINGS arrives. Our decision was to open up this flexibility of design rather than requiring either behavior. Do you have suggestions on how to make this clearer?
In my opinion, adding something like you just said to the specifications would be perfect. Given that section 7.2.5.2 already mentions default settings values, it might not be bad to just expand it there. For example: could be appended with something along the lines of: I think the most important part here is being a little more verbose about the edge case instead of just implying its existence by mentioning the default values for settings. If anyone has any other thoughts about it I'd love to hear them.

being used, the initial value of each server setting is the value used in the previous session.  Clients MUST store the settings the server provided in the session being resumed and MUST comply with  <ins> stored settings until the current server settings are received.  A client can use these initial values to send requests before the server's SETTINGS frame has arrived.  This removes the need for a client to wait for the SETTINGS frame before sending requests. </ins> A server can remember the settings that it advertised, or store an integrity-protected copy of the values in the ticket and recover the A server MAY accept 0-RTT and subsequently provide different settings in its SETTINGS frame.  If 0-RTT data is accepted by the server, its SETTINGS frame MUST NOT reduce any limits or alter any values that  <ins> might be violated by the client with its 0-RTT data.  The server MAY omit settings from its SETTINGS frame which are unchanged from the initial value. </ins> 7.2.6.


parameters "initial_max_stream_data_bidi_local".  An HTTP/3 server SHOULD send non-zero values for the QUIC transport parameters "initial_max_stream_data_bidi_remote" and "initial_max_bidi_streams". <del> It is recommended that "initial_max_bidi_streams" be no smaller than </del>  100, so as to not unnecessarily limit parallelism. HTTP/3 does not use server-initiated bidirectional streams; clients

IIUC, RFC 2119 keywords used in RFC 7540 need to be converted to uppercase words in HTTP/3, because HTTP/3 invokes RFC 8174.
That was never an interoperability requirement, just a suggestion. So I think that this should be reverted. If it isn't, using the active voice might be better.

That was never an interoperability requirement, just a suggestion. So I think that this should be reverted. Originally posted by NAME in URL Look at a better way to phrase this guidance, while still retaining the guidance. (Note this text has also been touched by other PRs in the meantime.)

parameters "initial_max_stream_data_bidi_local".  An HTTP/3 server SHOULD send non-zero values for the QUIC transport parameters "initial_max_stream_data_bidi_remote" and "initial_max_bidi_streams".  <ins> It is RECOMMENDED that "initial_max_bidi_streams" be no smaller than </ins> 100, so as to not unnecessarily limit parallelism. HTTP/3 does not use server-initiated bidirectional streams; clients


structure of the dependency tree changes as PRIORITY frames modify the dependency links between other prioritized elements.  All dependent streams are allocated an integer weight between 1 and 256 (inclusive), derived by adding one to the weight expressed in the PRIORITY frame. dependent on the orphan placeholder; pushes are dependent on the client request on which the PUSH_PROMISE frame was sent. <del> Requests may override the default initial values by including a PRIORITY frame (see frame-priority) at the beginning of the stream. These priorities can be updated by sending a PRIORITY frame on the control stream. </del>  4.3.1. server push using a Push ID (see frame-push-promise), or a placeholder using a Placeholder ID (see placeholders). <del> When a client initiates a request, a PRIORITY frame MAY be sent as the first frame of the stream, creating a dependency on an existing element.  In order to ensure that prioritization is processed in a consistent order, any subsequent PRIORITY frames for that request MUST be sent on the control stream.  A PRIORITY frame received after other frames on a request stream MUST be treated as a connection error of type HTTP_UNEXPECTED_FRAME. If, by the time a new request stream is opened, its priority information has already been received via the control stream, the PRIORITY frame sent on the request stream MUST be ignored. </del>  The PRIORITY frame payload has the following fields: A two-bit field indicating the type of element being prioritized <del> (see prioritized-element-types).  When sent on a request stream, this MUST be set to "11".  When sent on the control stream, this MUST NOT be set to "11". </del>  A two-bit field indicating the type of element being depended on <del> (see element-dependency-types). </del>  <del> A four-bit field which MUST be zero when sent and has no semantic </del>  value on receipt. A variable-length integer that identifies the element being prioritized.  Depending on the value of Prioritized Type, this contains the Stream ID of a request stream, the Push ID of a <del> promised resource, a Placeholder ID of a placeholder, or is absent. </del>  A variable-length integer that identifies the element on which a dependency is being expressed.  Depending on the value of prioritized element (see HTTP2, Section 5.3).  Add one to the value to obtain a weight between 1 and 256. <del> The values for the Prioritized Element Type (prioritized-element- types) and Element Dependency Type (element-dependency-types) imply the interpretation of the associated Element ID fields. </del>  Note that unlike in HTTP2, the root of the tree cannot be referenced using a Stream ID of 0, as in QUIC stream 0 carries a valid HTTP of type HTTP_MALFORMED_FRAME.  The following situations are examples of invalid PRIORITY frames: <del> A PRIORITY frame sent on a request stream with the Prioritized Element Type set to any value other than "11" A PRIORITY frame sent on a request stream which expresses a dependency on a request with a greater Stream ID than the current stream A PRIORITY frame sent on a control stream with the Prioritized Element Type set to "11" </del>  A PRIORITY frame which claims to reference a request, but the associated ID does not identify a client-initiated bidirectional client is not yet permitted to open MUST be treated as a connection error of type HTTP_LIMIT_EXCEEDED. <del> A PRIORITY frame received on any stream other than a request or control stream MUST be treated as a connection error of type HTTP_WRONG_STREAM. </del>  PRIORITY frames received by a client MUST be treated as a connection error of type HTTP_UNEXPECTED_FRAME.

Limits the PRIORITY frame to be only sent on the control stream, and brings back the Exclusive Flag. The exclusive flag was previously removed in
Right now we have a SETTINGS frame that can only be sent once (in each direction), and if it is sent it has to be sent at a very precise point (the start of the stream) or the connection is bust. For these reasons, it strikes me that we could simplify things, in line with today's HTTP/3 design, by unframing settings and placing them as the control stream header. For illustration this would look something like below, where Identifier and Value can be sent multiple times. For Simplifies some parts of error handling. The HTTPMISSINGSETTINGS snowflake becomes redundant Not possible to send multiple SETTINGS on the control stream to trigger HTTPUNEXPECTEDFRAME Not possible to send a SETTINGS frame on the wrong stream to trigger HTTPWRONGSTREAM A peer that is happy with defaults can send 00 (2 bytes) rather than 040 (3 bytes). This is a tinsy tiny benefit Forces endpoints to commit to SETTINGS very early (this is of dubious benefit if SETTINGS only loosen values) Against We already have implementations that handle SETTINGS just fine Keeping the frame allows us change today's design to allow multiple SETTINGS in the future. The arguments for and against seem to net out evenly. Changing control stream seems like a cleaner design to me but shipping is a feature :tm: .
aprops NAME One way to remove [HTTPMISSINGSETTINGS] is to allow not sending the SETTINGS frame at all. By that I meant that if first thing on the control stream isn't SETTINGS, then SETTINGS is not coming, use defaults
While the proposal is interesting, I think my preference goes to retaining the current design. I agree that HTTPMISSINGSETTINGS is unnecessary, it is my view that it should be merged to HTTPUNEXPECTEDFRAME. But once it gets done, it would be my view that the complexity is indifferent; there is no difference in the encoding-side (just sending different series of octets). On the decoding-side, it either having a state that receives series of special octets or a state that receives a particular frame. That said, the reason I prefer using a frame for carrying the settings is because it makes the design consistent. Having a layer that is used for "frame"-ing all the information being exchanged is a good thing. IMO the use of HTTPUNEXPECTEDFRAME or HTTPWRONGSTREAM in relation to SETTINGS frame is fine, because use of these errors are the design pattern we use for other frames too. For example, HTTPUNEXPECTEDFRAME is used to indicate error when the peer sends a DATA frame before a HEADERS frame. Or transmission of a HEADERS frame on the control stream triggers the HTTPWRONGSTREAM error. I prefer using that same design pattern for everything, rather than creating different rules for the way things are being sent in each state.
I remembered that we touched on some of this in URL Thanks for the feedback NAME Push streams break this pattern but c'est la vie. I think I'm happy to kill this issue/proposal.
If we took this proposal, we'd be cementing a new pattern: Unidirectional streams might have a header following the type. Push already does (and in London we rejected a proposal to make Push ID behave the way SETTINGS does instead, ). I like that pattern, and alongside this would remove the concept of frame-that-can-only-be-first. That's decidedly appealing. However, we're close to late-stage and this isn't demonstrably better, just arguably cleaner.
I think that Push Streams is a good argument for pushing this change, although I might argue the Push Streams header is not a length-value structure. I think I'd prefer reusing the type-length-value structure (i.e. frames) for sending SETTINGS rather than defining it's own length-value structure just for one purpose. (not that I care much, but just my two cents)
I believe the group did not want to make this change so I am closing the issue. NAME please reopen if I am mistaken.
The additional complexities of the HTTP/3 prioritization scheme, compared to that of HTTP/2, is that PRIORITY frames might arrive out-of-order between the control stream and the request streams, as well as the complexities that generate from the premise that out-of-order handling is necessary. Our previous perception has been that it's worth the complexity, but I am becoming doubtful if it's necessary. IIUC, the need for sending a PRIORITY frame on the request stream is to handle the rare-case (like 1%) where a packet carrying the control frame is being lost. But if we are to require a server to assign the most deferred priority (i.e. weight=0) to a request for which the PRIORITY frame is yet to be received, there would be a guarantee that the performance would be no worse compared to HTTP/2 over TCP even when packet loss is involved, as NAME points out in URL To rephrase, sending PRIORITY frames on request streams is an "optimization" for a rare case instead of it being a necessity to achieve comparable or better performance to HTTP/2. Therefore, I am suggesting to stop sending PRIORITY frames on the request stream. The benefit of stopping that is that then we can have the guarantee that all the PRIORITY frames would be received by the server in the order they were sent by the client. Then, thanks to the in-order delivery guarantee, I think we can revive the "exclusive" bit , revert the introduction of the "placeholders" concept in place of idle streams.
I just lost a morr elegant reply so this might sound curt. The proposal seems to make tree entry operations more deterministic. However, tree management like pruning and shuffling based on stream closure seems non-determinisitc. H2 has problems in this regard already. Some H3 implementation experience is already bringing to light designs that have two world views of stream state: one for QUIC transport and one for H3.
I'll attempt to summarize the issue: HTTP/3 priorities are different, but not better, and this creates implementation work and technical debt? Previous issue introducing placeholders:
NAME Yeah. I think we share the view that the approach proposed will have the same pros and cons that the prioritization scheme of H2 has. I think that's a good thing, because it helps people use the same approach when using the prioritization scheme.
NAME I disagree with the view. Our task is to port HTTP/2 semantics to HTTP/3. This is how you can do it, with less divergence than we have in the draft now. So it lessens the work to be done, implementation-wise.
This is a possibility, presuming that we take . We needed PRIORITY on the request stream so that streams opening before the PRIORITY frame arrives don't erroneously starve other resources until the PRIORITY frame arrives. plus reverting initial priority would allow bringing back exclusive dependencies; placeholders were separately discussed with HTTPbis as an enhancement to incorporate into HTTP/3, so I think a decision to revert that would be handled separately.
NAME Yes. The simplification is based on . I also agree that we can discuss about placeholders separately.
I have a nagging feeling, the QUIC streams can close in non-deterministic order from the remote peer perspective. This risks a H3 priority scheme that looks similar to H2 on paper but behaves quite differently. In the worst case this requires holding more state on the server in order to increase the chances that the client and server have similar global views.
While I agree that the the streams are being closed in a different manner, it is my view that things would generally work better in HTTP/3, because closure of streams would be notified earlier to the client (due to having less HoLB). Earlier synchronization of states lead to less need for retaining information of closed streams.
NAME sorry for the confusion, I was attempting to say that HTTP/3 priorities were different from HTTP/2 priorities, and that was a negative. I just corrected my comment to say HTTP/3. NAME your concerns are well-founded, but I think that's an existing problem and not a result of a recent change? I previously mentioned(verbally) that tree maintenance should be a longer period of time(ie: 3 MSL URL ?), but I realize even that may not be sufficient for QUIC and instead we should use something like 3*PTO? I also think attaching streams to streams is inherently dangerous and much more dangerous in QUIC than HTTP/2, which is why I want to change HTTP/3 priorities to never have streams depend upon streams.
I also think we might be conflating some issues here... to me, streams-depending-on-streams + do we need placeholders is a separate discussion from the original statement of this issue. I am a bit hesitant to agree with this proposal, as I am worried about potential HOL blocking on the control stream, see also (a potential solution which was discarded in London). As mentioned above, this might be rare, but -any- loss on the control stream would impact priorities. Thinking of long fat networks and the CDN/proxy use case (of which it's still a bit unclear if we want to incorporate that into the HTTP/3 prioritization design), this might have a bigger impact than anticipated (though should only be occasional). I myself was quite happy with just changing default behaviour (with ) in the case of potential re-ordering and don't really see the need for this change. The argument that it allows re-introducing exclusive priorities (uttered elsewhere) is true, but as also uttered elsewhere, I don't really see the need for that feature (was only used by Chrome for H2 afaict) and is better replaced with something like "implicit fifo" e.g., or .
NAME the way I'm currently viewing it is that there are two tracks: The QUIC WG is gamely plugging along on its charter to replicate HTTP/2 functionality in HTTP/3 modulo changes required by the nature of the QUIC transport. In the meantime, the HTTP WG might come up with a way to replace/improve the one in HTTP/2. When they tell us what they want to use, we'll do it. With my editor's hat on, only the first is my business unless and until the second one concludes. As an individual, I can and will go to HTTPbis and advocate for something better. But on this side of the fence, I still have to make progress on the first in case the second one derails. , reverting initial PRIORITY, and bringing back Exclusive dependencies gets us back closer to HTTP/2 equivalence and had general support at the interim. So that's our just-in-case position while we see what HTTPbis decides.
NAME I think those two tracks make sense, though I do think along the way we're making some choices about the best way to change H2 priorities to be more suitable to QUIC and what compromises are and aren't acceptable, which is by its nature a series of compromises. For example, it seems a defaulting request weight of 1 or 0 would be a more HTTP/2 like solution than adding a default placeholder? But possibly it doesn't matter that much? Are there outstanding PRs for the rest of the migration back to more H2-like priorities?
NAME I do not think it matters much. And I think that servers might want to play some game here, for example giving higher default precedence to responses that carry CSS files until a PRIORITY frame is being received. That said, I think I prefer the design principle to be "provide something that is guaranteed to be at worst as good as HTTP/2." Giving least precedence to requests for which the client-specified priority is yet to be known is a natural way to achieve that. Just filed . I think that and are the remaining changes that we need to make, in order to allow browsers use the same prioritization scheme as the ones they've used in HTTP/2 (with the exception being that idle prioritized streams are replaced by placeholders).
Thanks NAME reviewing now
Thanks, this LGTM

structure of the dependency tree changes as PRIORITY frames modify the dependency links between other prioritized elements. <ins> An exclusive flag allows for the insertion of a new level of dependencies.  The exclusive flag causes the prioritized element to become the sole dependency of its parent, causing other dependencies to become dependent on the exclusive element. </ins> All dependent streams are allocated an integer weight between 1 and 256 (inclusive), derived by adding one to the weight expressed in the PRIORITY frame. dependent on the orphan placeholder; pushes are dependent on the client request on which the PUSH_PROMISE frame was sent.  <ins> Priorities can be updated by sending a PRIORITY frame (see frame- priority) on the control stream. </ins> 4.3.1. server push using a Push ID (see frame-push-promise), or a placeholder using a Placeholder ID (see placeholders).  <ins> In order to ensure that prioritization is processed in a consistent order, PRIORITY frames MUST be sent on the control stream. </ins> The PRIORITY frame payload has the following fields: A two-bit field indicating the type of element being prioritized  <ins> (see priority-element-types).  This MUST NOT be set to "11". </ins> A two-bit field indicating the type of element being depended on  <ins> (see priority-element-types). A single-bit flag indicating that the dependency is exclusive (see priority). </ins>  <ins> A three-bit field which MUST be zero when sent and has no semantic </ins> value on receipt. A variable-length integer that identifies the element being prioritized.  Depending on the value of Prioritized Type, this contains the Stream ID of a request stream, the Push ID of a  <ins> promised resource, or a Placeholder ID of a placeholder. </ins> A variable-length integer that identifies the element on which a dependency is being expressed.  Depending on the value of prioritized element (see HTTP2, Section 5.3).  Add one to the value to obtain a weight between 1 and 256.  <ins> The values for the Prioritized Element Type and Element Dependency Type (priority-element-types) imply the interpretation of the associated Element ID fields. </ins> Note that unlike in HTTP2, the root of the tree cannot be referenced using a Stream ID of 0, as in QUIC stream 0 carries a valid HTTP of type HTTP_MALFORMED_FRAME.  The following situations are examples of invalid PRIORITY frames:  <ins> A PRIORITY frame with the Prioritized Element Type set to "11". </ins> A PRIORITY frame which claims to reference a request, but the associated ID does not identify a client-initiated bidirectional client is not yet permitted to open MUST be treated as a connection error of type HTTP_LIMIT_EXCEEDED.  <ins> A PRIORITY frame received on any stream other than the control stream MUST be treated as a connection error of type HTTP_WRONG_STREAM. </ins> PRIORITY frames received by a client MUST be treated as a connection error of type HTTP_UNEXPECTED_FRAME.


any of which it can use when sending packets.  When the endpoint wishes to remove a connection ID from use, it sends a RETIRE_CONNECTION_ID frame to its peer.  Sending a <del> RETIRE_CONNECTION_ID frame indicates that the connection ID won't be used again and requests that the peer replace it with a new </del>  connection ID using a NEW_CONNECTION_ID frame. As discussed in migration-linkability, each connection ID MUST be 13.1.1. <del> An endpoint MUST NOT send more than one packet containing only an ACK frame per received packet that contains frames other than ACK and PADDING frames.  An endpoint MUST NOT send a packet containing only an ACK frame in response to a packet containing only ACK or PADDING frames, even if there are packet gaps which precede the received packet.  This prevents an indefinite feedback loop of ACKs.  The endpoint MUST however acknowledge packets containing only ACK or PADDING frames when sending ACK frames in response to other packets. </del>  Packets containing PADDING frames are considered to be in flight for congestion control purposes QUIC-RECOVERY.  Sending only PADDING that other frames are sent in addition to PADDING frames to elicit acknowledgments from the receiver.  The receiver's delayed acknowledgment timer SHOULD NOT exceed the current RTT estimate or the value it indicates in the "max_ack_delay" transport parameter.  This ensures an acknowledgment is sent at least Strategies and implications of the frequency of generating acknowledgments are discussed in more detail in QUIC-RECOVERY.  To limit ACK Ranges (see ack-ranges) to those that have not yet been received by the sender, the receiver SHOULD track which ACK frames have been acknowledged by its peer.  The receiver SHOULD exclude already acknowledged packets from future ACK frames whenever these <del> packets would unnecessarily contribute to the ACK frame size. Because ACK frames are not sent in response to ACK-only packets, a receiver that is only sending ACK frames will only receive acknowledgements for its packets if the sender includes them in packets with non-ACK frames.  A sender SHOULD bundle ACK frames with other frames when possible. </del>  To limit receiver state or the size of ACK frames, a receiver MAY limit the number of ACK Ranges it sends.  A receiver can do this even the receiver SHOULD repeatedly acknowledge newly received packets in preference to packets received in the past. <del> An endpoint SHOULD treat receipt of an acknowledgment for a packet it did not send as a connection error of type PROTOCOL_VIOLATION, if it is able to detect the condition. 13.1.2. </del>  ACK frames MUST only be carried in a packet that has the same packet number space as the packet being ACKed (see packet-protected).  For

MUST NOT bundle a PING with all packets that would otherwise be ACK-only. If there's anything else discussed in London that I missed, please tell me.
Isn't this just moving up one level in the inception layer? Now you must not send ACK+PING too often to prevent an indefinite feedback loop? Why not just say that an ACK should not ACK / PAD only frames for more than a fraction of the packets to avoid feedback loops and to ensure eventual reduction of gap accounting?
NAME My past experience has been that it's better if the sender of ACKs is in control of how often they want an ACK in response. There are a number of packet tracking/ACKing algorithms which never require ACKs of ACK-only packets, and I think we should accommodate those. Making the receiver of ACKs guess how often to ACK ACKs is a problem that's impossible to get right, and as NAME said on the issue, one can never rely on any receiver behavior regarding when they'll ACK ACKs, so it's more robust if the sender is in control. I will add normative text about not including a PING with every ACK.
OK, that makes sense.
I've now removed ACK-only from the transport text, and replaced it with ACK-eliciting.
The transport spec says in section 13.1.1: "An endpoint MUST NOT send a packet containing only an ACK frame in response to a packet containing only ACK or PADDING frames, even if there are packet gaps which precede the received packet. This prevents an indefinite feedback loop of ACKs." I argue that this MUST NOT should be changed to SHOULD NOT. Fist, the infinite feedback can be prevented if, for example, an implementation only sends an ACK for N packets received, so there is really no need to be that strict. Second, ACK of ACK are generally useful. They enable trimming the grow of ACK lists in a controlled way. In fact, I found that problem in interop tests. Some implementations would send thousands of data packets without sending a single ACK of ACK. That means the sender is stuck with a large ACK dashboard. Yes, it could be trimmed with timers, but that's hardly a controlled process. I fixed this by sending an ACK + PING packet whenever there the unacked ACK list is too long, but that's a hack.
I'd argue that ACK+PING isn't a hack, but instead a simple way for the sender, whose state we're trying to limit, to balance the amount of state it stores vs the extra ACKs being received. Getting the peer to implement the correct ACK behavior by sending a single extra byte is much simpler and more robust in my experience than relying on the receiver to implement an "ACK every 10/20/etc ACKs" policy in a way that every sender is happy with. This could likely use more text and there's still an outstanding editorial issue to move most of the ACK sending text that's in recovery into transport, so it's all in one place, since I believe there are currently a few subtle inconsistencies.
Completely agree. Ian hits on the key point here. In the case of an endpoint that would like to limit the amount of state it stores, it's highly desirable for THAT endpoint to be in control. ACK+PING, satisfies that. Relying on ACKs of ACKs from the peer does not, sadly.
Agreed with NAME Irrespective of what we recommend here, note that a sender of ACKs cannot assume what the peer will do -- when the peer will send an ACK for ACKs, I mean -- so, it will end up having to send a periodic PING anyways. Piggybacking a PING periodically (once an RTT, for example) is useful for RTT measurement as well, since ACK of ACKs do not get used for RTT (see for more discussion on this point).
An alternative would be allowing sending ACKs-of-ACKs, if you are sending non-(ACK|PADDING) frames. So, maybe: "An endpoint MUST NOT send a packet containing only ACK or PADDING frames in response to a packet containing only ACK or PADDING frames, if the previous packet it sent contained only ACK or PADDING frames."
As discussed in London, NAME to write a PR with some advisory text
So many comments, so much opportunity for this to get worse if we don't stop.Minor nit, but looks good.Text looks good, just noticed a few minor whitespace nits while reading through.

any of which it can use when sending packets.  When the endpoint wishes to remove a connection ID from use, it sends a RETIRE_CONNECTION_ID frame to its peer.  Sending a  <ins> RETIRE_CONNECTION_ID frame indicates that the connection ID will not be used again and requests that the peer replace it with a new </ins> connection ID using a NEW_CONNECTION_ID frame. As discussed in migration-linkability, each connection ID MUST be 13.1.1.  <ins> An endpoint sends ACK frames to acknowledge packets it has received and processed. Packets containing only ACK frames are not congestion controlled, so there are limits on how frequently they can be sent.  An endpoint MUST NOT send more than one ACK-frame-only packet in response to receiving an ACK-eliciting packet (one containing frames other than ACK and/or PADDING).  An endpoint MUST NOT send a packet containing only an ACK frame in response to a non-ACK-eliciting packet (one containing only ACK and/or PADDING frames), even if there are packet gaps which precede the received packet.  Limiting ACK frames avoids an infinite feedback loop of acknowledgements, which could prevent the connection from ever becoming idle.  However, the endpoint acknowledges non-ACK-eliciting packets when it sends an ACK frame. </ins> Packets containing PADDING frames are considered to be in flight for congestion control purposes QUIC-RECOVERY.  Sending only PADDING that other frames are sent in addition to PADDING frames to elicit acknowledgments from the receiver. <ins> An endpoint that is only sending ACK frames will not receive acknowledgments from its peer unless those acknowledgements are included in packets with ACK-eliciting frames.  An endpoint SHOULD bundle ACK frames with other frames when there are new ACK-eliciting packets to acknowledge.  When only non-ACK-eliciting packets need to be acknowledged, an endpoint MAY wait until an ACK-eliciting packet has been received to bundle an ACK frame with outgoing frames. An endpoint SHOULD treat receipt of an acknowledgment for a packet it did not send as a connection error of type PROTOCOL_VIOLATION, if it is able to detect the condition. </ins> The receiver's delayed acknowledgment timer SHOULD NOT exceed the current RTT estimate or the value it indicates in the "max_ack_delay" transport parameter.  This ensures an acknowledgment is sent at least Strategies and implications of the frequency of generating acknowledgments are discussed in more detail in QUIC-RECOVERY. <ins> 13.1.2. </ins> To limit ACK Ranges (see ack-ranges) to those that have not yet been received by the sender, the receiver SHOULD track which ACK frames have been acknowledged by its peer.  The receiver SHOULD exclude already acknowledged packets from future ACK frames whenever these  <ins> packets would unnecessarily contribute to the ACK frame size.  When the receiver is only sending non-ACK-eliciting packets, it can bundle a PING or other small ACK-eliciting frame with a fraction of them, such as once per round trip, to enable dropping unnecessary ACK ranges and any state for previously sent packets.  The receiver MUST NOT bundle an ACK-elicing frame, such as a PING, with all packets that would otherwise be non-ACK-eliciting, in order to avoid an infinite feedback loop of acknowledgements. </ins> To limit receiver state or the size of ACK frames, a receiver MAY limit the number of ACK Ranges it sends.  A receiver can do this even the receiver SHOULD repeatedly acknowledge newly received packets in preference to packets received in the past.  <ins> 13.1.3. </ins> ACK frames MUST only be carried in a packet that has the same packet number space as the packet being ACKed (see packet-protected).  For


future connections.  The client includes this token in Initial packets to provide address validation in a future connection.  The client MUST include the token in all Initial packets it sends, unless <del> a Retry or NEW_TOKEN frame replaces the token with a newer one.  The client MUST NOT use the token provided in a Retry for future connections.  Servers MAY discard any Initial packet that does not carry the expected token. </del>  A token SHOULD be constructed for the server to easily distinguish it from tokens that are sent in Retry packets as they are carried in the

The client only replaces a token sent in Initial packets when receiving a Retry. NEW_TOKEN is an orthogonal mechanism to provide tickets for subsequent connections, and doesn't have any effect on the current connection.

future connections.  The client includes this token in Initial packets to provide address validation in a future connection.  The client MUST include the token in all Initial packets it sends, unless  <ins> a Retry replaces the token with a newer one.  The client MUST NOT use the token provided in a Retry for future connections.  Servers MAY discard any Initial packet that does not carry the expected token. </ins> A token SHOULD be constructed for the server to easily distinguish it from tokens that are sent in Retry packets as they are carried in the