independent of other connections that are not part of the same Connection Group. <del> Connection establishment is only a local operation for a Datagram transport (e.g., UDP(-Lite)), which serves to simplify the local send/receive functions and to filter the traffic for the specified addresses and ports RFC8085. </del>  Once Initiate has been called, the Selection Properties and Endpoint information are immutable (i.e, an application is not able to later arch), in which the necessary protocol handshakes are conducted so that the transport system can select which set to use. <del> This document structures the candidates for racing as a tree as terminological convention.  While a a tree structure is not the only way in which racing can be implemented, it does ease the illustration of how racing works. </del>  The most simple example of this process might involve identifying the single IP address to which the implementation wishes to connect, original request.  New connection attempts that involve transmitting data on the network ought not to be started after another leaf node has already successfully completed, because the connection as a whole <del> has now been established.  An implementation may choose to let certain handshakes and negotiations complete in order to gather metrics to influence future connections.  Keeping additional connections is generally not recommended since those attempts were slower to connect and may exhibit less desirable properties. </del>  4.4.1. 5.2. <del> Similar to sending, Receiving a Message is determined by the top- </del>  level protocol in the established Protocol Stack.  The main <del> difference with Receiving is that the size and boundaries of the </del>  Message are not known beforehand.  The application can communicate in its Receive action the parameters for the Message, which can help the Transport Services implementation know how much data to deliver and complete Message, the implementation should wait until an entire Message (datagram, stream, or frame) is read before delivering any Message content to the application.  This requires the implementation <del> to understand where messages end, either via a supplied deframer or because the top-level protocol in the established Protocol Stack preserves message boundaries.  If the top-level protocol only supports a byte-stream and no framers were supported, the application can control the flow of received data by specifying the minimum number of bytes of Message content it wants to receive at one time. </del>  If a Connection finishes before a requested Receive action can be satisfied, the Transport Services API should deliver any partial purposes of the Transport Services API, these are ways for applications or application frameworks to define their own Message parsing to be included within a Connection's Protocol Stack.  As an <del> example, TLS is exposed as a protocol natively supported by the Transport Services API, even though it could also serve the purpose of framing data over TCP. </del>  Most Message Framers fall into one of two categories: Upon receiving this event, a framer implementation is responsible for performing any necessary transformations and sending the resulting data back to the Message Framer, which will in turn send it to the <del> next protocol.  Implementations SHOULD ensure that there is a way to pass the original data through without copying to improve performance. </del>  To provide an example, a simple protocol that adds a length as a header would receive the "NewSentMessage" event, create a data If the Protocol Stack includes a transport protocol that supports multipath connectivity, the Transport Services implementation should <del> also inform the Protocol Instance of potentially new paths that </del>  become permissible based on the "multipath" Selection Property and the "multipath-policy" Connection Property choices made by the application.  A protocol can then establish new subflows over new between paths, the update to the set of available paths can trigger the connection to be migrated. <del> In case of Pooled Connections pooled-connections, the Transport </del>  Services implementation may add connections over new paths to the pool if permissible based on the multipath policy and Selection <del> Properties.  In case a previously used path becomes unavailable, the transport system may disconnect all connections that require this path, but should not disconnect the pooled connection object exposed to the application.  The strategy to do so is implementation- specific, but should be consistent with the behavior of multipath transports. </del>  8.

replace DG by CL. Changed one oddity: /as to/ to /about/
"Connection establishment is only a local operation for a Datagram transport (e.g., UDP(-Lite))": I think the criterion is not Datagram transports but "transports without a handshake" or "connectionless protocols" as it is used later in the document.
I wrote, and I suggest we use "connectionless protocols" explicitly mentioning examples of UDP and UDP-Lite when used as a transport without a handshake.
To whoever plans to address this: I checked occurrences of the word "datagram" in the architecture and API drafts, and I found (only) one sentence that needs fixing on similar grounds, I think, in the API draft: in Section 8.1.4 "Timeout for keep alive packets". Surely the guidance regarding this matter in the UDP usage RFC is related to the fact that UDP(-Lite) is connectionless, not that it's datagram based.
Starting to fix this.
This LGTM

independent of other connections that are not part of the same Connection Group.  <ins> Connection establishment is only a local operation for a connectionless protocols, which serves to simplify the local send/ receive functions and to filter the traffic for the specified addresses and ports RFC8085 (for example using UDP or UDP-Lite transport without a connection handshake procedure). </ins> Once Initiate has been called, the Selection Properties and Endpoint information are immutable (i.e, an application is not able to later arch), in which the necessary protocol handshakes are conducted so that the transport system can select which set to use.  <ins> As a terminological convention, this document structures the candidates for racing as a tree.  While a tree structure is not the only way in which racing can be implemented, it does ease the illustration of how racing works. </ins> The most simple example of this process might involve identifying the single IP address to which the implementation wishes to connect, original request.  New connection attempts that involve transmitting data on the network ought not to be started after another leaf node has already successfully completed, because the connection as a whole  <ins> has now been established.  An implementation could choose to let certain handshakes and negotiations complete to gather metrics that influence future connections.  Keeping additional connections is generally not recommended, because those attempts were slower to connect and may exhibit less desirable properties. </ins> 4.4.1. 5.2.  <ins> Similar to sending, receiving a Message is determined by the top- </ins> level protocol in the established Protocol Stack.  The main  <ins> difference with receiving is that the size and boundaries of the </ins> Message are not known beforehand.  The application can communicate in its Receive action the parameters for the Message, which can help the Transport Services implementation know how much data to deliver and complete Message, the implementation should wait until an entire Message (datagram, stream, or frame) is read before delivering any Message content to the application.  This requires the implementation  <ins> to understand where messages end, either via a supplied or because the top-level protocol in the established Protocol Stack preserves message boundaries.  If the top-level protocol only supports a byte- stream and no framers were supported, the application can control the flow of received data by specifying the minimum number of bytes of Message content it wants to receive at one time. </ins> If a Connection finishes before a requested Receive action can be satisfied, the Transport Services API should deliver any partial purposes of the Transport Services API, these are ways for applications or application frameworks to define their own Message parsing to be included within a Connection's Protocol Stack.  As an  <ins> example, TLS is exposed as a protocol that is natively supported by the Transport Services API, even though it could also serve the purpose of framing data over TCP. </ins> Most Message Framers fall into one of two categories: Upon receiving this event, a framer implementation is responsible for performing any necessary transformations and sending the resulting data back to the Message Framer, which will in turn send it to the  <ins> next protocol.  To improve performance, implementations SHOULD ensure that there is a way to pass the original data through without copying. </ins> To provide an example, a simple protocol that adds a length as a header would receive the "NewSentMessage" event, create a data If the Protocol Stack includes a transport protocol that supports multipath connectivity, the Transport Services implementation should  <ins> also inform the Protocol Instance about potentially new paths that </ins> become permissible based on the "multipath" Selection Property and the "multipath-policy" Connection Property choices made by the application.  A protocol can then establish new subflows over new between paths, the update to the set of available paths can trigger the connection to be migrated.  <ins> In the case of a Pooled Connection pooled-connections, the Transport </ins> Services implementation may add connections over new paths to the pool if permissible based on the multipath policy and Selection  <ins> Properties.  In the case that a previously used path becomes unavailable, the transport system may disconnect all connections that require this path, but should not disconnect the pooled connection object exposed to the application.  The strategy to do so is implementation-specific, but should be consistent with the behavior of multipath transports. </ins> 8.


This only takes effect when the transport uses a network layer that supports this functionality.  When it does take effect, setting this property to true will cause the sender to avoid network-layer source <del> frgementation.  When using IPv4, this will result in the Don't </del>  Fragment bit being set in the IP header. Attempts to send a message with this property that result in a size provide segmentation of messages larger than the maximum size permitted by the network layer, and also to avoid network-layer source fragmentation of messages.  When running over IPv4, setting <del> this property to true can result in a sending endpount setting the </del>  Don't Fragment bit in the IPv4 header of packets generated by the <del> transport layer.  An attempt to send a message that results in a size greater than the transport's current estimate of its maximum packet size (singularTransmissionMsgMaxLen) will result in a SendError. This only takes effect when the transport and network layer support this functionality. </del>  9.2.

This seems otherwise inconsistent: for the noFragmentation property, we say that, in case of IPv4, setting this will result in setting the "Don't Fragment" bit. the noSegmentation property is described as avoiding fragmentation AND avoiding segmentation. So logically, this should also result in setting the "Don't Fragment" bit in case of IPv4. Generally, I'm not sure how an implementer should interpret the "can" here, in contrast with the "will" in the "Don't Fragment" case above... the textual difference seems to hint that this could be implemented differently, but then, how?

This only takes effect when the transport uses a network layer that supports this functionality.  When it does take effect, setting this property to true will cause the sender to avoid network-layer source  <ins> fragmentation.  When using IPv4, this will result in the Don't </ins> Fragment bit being set in the IP header. Attempts to send a message with this property that result in a size provide segmentation of messages larger than the maximum size permitted by the network layer, and also to avoid network-layer source fragmentation of messages.  When running over IPv4, setting  <ins> this property to true will result in a sending endpount setting the </ins> Don't Fragment bit in the IPv4 header of packets generated by the  <ins> transport layer. An attempt to send a message that results in a size greater than the transport's current estimate of its maximum packet size ("singularTransmissionMsgMaxLen") will result in a "SendError".  This only takes effect when the transport and network layer support this functionality. </ins> 9.2.


Services implementation MUST NOT automatically fall back from secure protocols to insecure protocols, or to weaker versions of secure protocols.  A Transport Services implementation MAY allow <del> applications to explicitly specify that fallback to a specific other version of a protocol , e.g., to allow fallback to TLS 1.2 if TLS 1.3 is not available. </del>  3.4. Applications need to ensure that they use security APIs appropriately.  In cases where applications use an interface to provide sensitive keying material, e.g., access to private keys or <del> copies of pre-shared keys (PSKs), key use needs to be validated.  For example, applications ought not to use PSK material created for the Encapsulating Security Protocol (ESP, part of IPsec) RFC4303 with QUIC, and applications ought not to use private keys intended for server authentication as keys for client authentication. </del>  A Transport Services system must not automatically fall back from secure protocols to insecure protocols, or to weaker versions of secure protocols (see equivalence).  For example, if an application requests a specific version of TLS, but the desired version of TLS is <del> not available, its connection will fail.  Applications are thus responsible for implementing security protocol fallback or version fallback by creating multiple Connections, if so desired. Alternatively, the Transport Services API MAY allow applications to specify that fallback to a specific other version of a protocol is allowed by the Transport Services system. </del>

We specifically left normative language for the security stuff.
From Watson Ladd: One minor nit in particular concerns the example of TLS fallback. It's not the case that one needs to carry out a fallback to TLS 1.2 if TLS 1.3 is available, and indeed that shouldn't be done. Ordinary TLS version negotiation should work just fine. A deeper nit concerns the way security is supposed to work in this system. It's true that e.g. both TLS and IKEv2 can use X509 certificates. It is not the case they use them the same way as they aren't authenticating the same thing. It is very much not the case that X509 validation involving trust verfication callbacks by the application works correctly, even conceptually or in most libraries, and the WebPKI will have very different additional requirements than other PKIs. It gets even more complicated when we consider how origin authority actually works in the web. I'm confused by the warnings on cross-protocol use of key material. For a PSK one really does want to link it to the protocol, but the same isn't true for X509 certificates across TLS and QUIC: you're supposed to use the same ones. I'm not sure how effective handling this all over to the application will be, or how the API will effectively make clear what connection the material will be used with in a racing environment.
These -arch sections contain normative language… didn't we try to get rid of all normative language in -arch?

Services implementation MUST NOT automatically fall back from secure protocols to insecure protocols, or to weaker versions of secure protocols.  A Transport Services implementation MAY allow  <ins> applications to explicitly specify which versions of a protocol ought to be permitted, e.g., to allow a minimum version of TLS 1.2 in case TLS 1.3 is not available. </ins> 3.4. Applications need to ensure that they use security APIs appropriately.  In cases where applications use an interface to provide sensitive keying material, e.g., access to private keys or  <ins> copies of pre-shared keys (PSKs), key use needs to be validated and scoped to the intended protocols and roles.  For example, if an application provides a certificate to only be used as client authentication for outbound TLS and QUIC connections, the Transport Services system MUST NOT use this automatically in other contexts (such as server authentication for inbound connections, or in other another security protocol handshake that is not equivalent to TLS). </ins> A Transport Services system must not automatically fall back from secure protocols to insecure protocols, or to weaker versions of secure protocols (see equivalence).  For example, if an application requests a specific version of TLS, but the desired version of TLS is  <ins> not available, its connection will fail.  The Transport Services API MAY allow applications to specify minimum versions that are allowed to be used by the Transport Services system. </ins>


the capabilities of the protocol.  Once an application specifies its Transport Properties, the transport system matches the required and prohibited properties against the transport features of the available <del> protocols. </del>  In the following cases, failure should be detected during pre- establishment: section is the algorithm defining which of these options to try, when, and in what order. <del> During Candidate Gathering, an implementation first excludes all protocols and paths that match a Prohibit or do not match all Require properties.  Then, the implementation will sort branches according to Preferred properties, Avoided properties, and possibly other criteria. </del>  4.1. highest available capacity, based on the observed maximum throughput. <del> Implementations process the Properties in the following order: Prohibit, Require, Prefer, Avoid.  If Selection Properties contain any prohibited properties, the implementation should first purge branches containing nodes with these properties.  For required properties, it should only keep branches that satisfy these requirements.  Finally, it should order the branches according to the preferred properties, and finally use any avoided properties as a tiebreaker.  When ordering branches, an implementation can give more weight to properties that the application has explicitly set, than to the properties that are default. </del>  The available protocols and paths on a specific system and in a specific context can change; therefore, the result of sorting and the

Quoting from the end of the Section 4 overview, the earliest point in the Implementation Draft where Preference Levels are explicitly mentioned: While the capitalization of "Prohibit" and "Require" does somewhat help to indicate their role as a special term, the sentence does not scan particularly well to any reader that is yet unfamiliar with the notion of Preference Levels from the API draft. At the very least it would help to better contextualize the sentence. Suggestion 1: It could also help to not explicitly refer to any Preference Levels yet. Suggestion 2: This could be sufficient for the Section 4 overview, since Section 4.1.3 "Sorting Branches" later goes into more detail about the different properties and their handling, even if the wording is a bit terse:

the capabilities of the protocol.  Once an application specifies its Transport Properties, the transport system matches the required and prohibited properties against the transport features of the available  <ins> protocols (see Section 6.2 of I-D.ietf-taps-interface for the definition of property preferences). </ins> In the following cases, failure should be detected during pre- establishment: section is the algorithm defining which of these options to try, when, and in what order.  <ins> During Candidate Gathering, an implementation prunes and sorts branches according to the Selection Property preferences (Section 6.2 of I-D.ietf-taps-interface.  It first excludes all protocols and paths that match a Prohibit property or do not match all Require properties.  Then it will sort branches according to Preferred properties, Avoided properties, and possibly other criteria. </ins> 4.1. highest available capacity, based on the observed maximum throughput.  <ins> Implementations process the Properties (Section 6.2 of I-D.ietf-taps- interface) in the following order: Prohibit, Require, Prefer, Avoid. If Selection Properties contain any prohibited properties, the implementation should first purge branches containing nodes with these properties.  For required properties, it should only keep branches that satisfy these requirements.  Finally, it should order the branches according to the preferred properties, and finally use any avoided properties as a tiebreaker.  When ordering branches, an implementation can give more weight to properties that the application has explicitly set, than to the properties that are default. </ins> The available protocols and paths on a specific system and in a specific context can change; therefore, the result of sorting and the


The process of establishing a network connection begins when an application expresses intent to communicate with a Remote Endpoint by <del> calling Initiate.  (At this point, any constraints or requirements the application may have on the connection are available from pre- establishment.)  The process can be considered complete once there is at least one Protocol Stack that has completed any required setup to the point that it can transmit and receive the application's data. </del>  Connection establishment is divided into two top-level steps: Candidate Gathering, to identify the paths, protocols, and endpoints The rest of this section will use a notation scheme to represent this tree.  The parent (or trunk) node of the tree will be represented by <del> a single integer, such as "1".  Each child of that node will have an integer that identifies it, from 1 to the number of children.  That child node will be uniquely identified by concatenating its integer to it's parents identifier with a dot in between, such as "1.1" and "1.2".  Each node will be summarized by a tuple of three elements: endpoint, path (labeled here by interface), and protocol.  The above example can now be written more succinctly as: </del>  When an implementation is asked to establish a single connection, only one of the leaf nodes in the candidate set is needed to transfer "Capacity Profile": An implementation can use the Capacity Profile to prefer paths that match an application's expected traffic pattern.  This match will use cached performance estimates, see <del> performance-caches: Scavenger: Prefer paths with the highest expected available capacity, but minimising impact on other traffic, based on the observed maximum throughput; </del>  Low Latency/Interactive: Prefer paths with the lowest expected Round Trip Time, based on observed round trip time estimates; Round Trip Time, but can tolerate delay variation; Constant-Rate Streaming: Prefer paths that are expected to <del> satisy the requested Stream Send or Stream Receive Bitrate, </del>  based on the observed maximum throughput; Capacity-Seeking: Prefer adapting to paths to determine the it to each new leaf node that is started and for which a protocol instance supporting 0-RTT is being used. <del> It is also possible that Protocol Stacks within a particular leaf node use 0-RTT handshakes without any safely replayable application data.  For example, TCP Fast Open could use a Client Hello from TLS </del>  as its 0-RTT data, shortening the cumulative handshake time. 0-RTT handshakes often rely on previous state, such as TCP Fast Open can support cloning new Connection objects without a new explicit handshake. <del> Protocols also define a notion of Data Unit.  Possible values for Data Unit are: </del>  Byte-stream.  Byte-stream protocols do not define any Message boundaries of their own apart from the end of a stream in each API mappings for UDP are as follows: <del> UDP connections represent a pair of specific IP addresses and </del>  ports on two hosts. CONNECT.UDP.  Calling "Initiate" on a UDP Connection causes it to

I'm having difficult parsing "At this point, any constraints or requirements the application may have on the connection are available from pre-establishment." I'm not even sure what meaning is intended. Available? To what and for what? Is this really a forest? I.e., is a possible when, for instance, you have a choice of two hostnames? If so, there should be an example illustrating that. The last example in the section should be augmented to show multiple hostnames as described by the preceding paragraph. Additionally, I think stating that there is a clear/natural dependency relationship between branching operations such that later operations depend on earlier decisions could simplify and clarify the justification for the suggested order of operations. "Scavenger" implies LEDBAT to me: it will take whatever is currently not being used but will defer to everything else. I'm not sure that notion aligns with the preference described. Is the intent of: that a protocol higher in the stack (or a framer?) could use 0-RTT even if the application doesn't want 0-RTT? Are there any security or privacy implications to this? Should there be a mechanism to disable this entirely? I do not understand the syntax employed here. Is in: a global object or something? It appears to be generating a Start event with the Connection object to the system, but isn't the framer the receiver of such an event? (Nit: isn't Event<> with angle brackets the syntax here?) doesn't seem quite right. Protocols possess a property of application-layer data unit, usually as an intrinsic property that is a direct consequence of the protocol design. They don't really define it so much as end up with it. Maybe this is too much of a nit, but it reads weirdly. The use of "unconnected" (or "connectionless") directly adjacent to the phrase "UDP connections" is driving me batty. I understand that the TAPS interface provides only a Connection object, but we should not ever use lowercase-c "connection" to refer to UDP, and be clear when we mean the uppercase-C "Connection" object in relation to an underlying UDP transport.
I rust just read this as "The Implementation now has all constrains it needs to know for attempting to establish a connection in the form of Preconnection parameters (endpoints, security parameters and transport properties) and can now continue." I would imagine that if 0-RTT is set to prohibit that applies to the entire stack (including framers). This is a bit confusing, but basically, as far as I understand, there is the MessageFramer which is part of the TAPS Implementation (this creates the events and has the methods MakeConnectionReady, MakeConnectionClosed etc.) and there is the framer implementation (which is the part the users of the TAPS implementation write themselves, this receives the events and then calls the methods on the MessageFramer). It might be good to find another term for MessageFramer to make that difference more clear.
WFM

The process of establishing a network connection begins when an application expresses intent to communicate with a Remote Endpoint by  <ins> calling Initiate, at which point the Preconnection object contains all constraints or requirements the application has configured for the connection.  The establishment process can be considered complete once there is at least one Protocol Stack that has completed any required setup to the point that it can transmit and receive the application's data. </ins> Connection establishment is divided into two top-level steps: Candidate Gathering, to identify the paths, protocols, and endpoints The rest of this section will use a notation scheme to represent this tree.  The parent (or trunk) node of the tree will be represented by  <ins> a single integer, such as "1". ("1" is used assuming that this is the first connection made by the system; future connections created by the application would allocate numbers in an increasing manner.) Each child of that node will have an integer that identifies it, from 1 to the number of children.  That child node will be uniquely identified by concatenating its integer to it's parents identifier with a dot in between, such as "1.1" and "1.2".  Each node will be summarized by a tuple of three elements: endpoint, path (labeled here by interface), and protocol.  The above example can now be written more succinctly as: </ins> When an implementation is asked to establish a single connection, only one of the leaf nodes in the candidate set is needed to transfer "Capacity Profile": An implementation can use the Capacity Profile to prefer paths that match an application's expected traffic pattern.  This match will use cached performance estimates, see  <ins> performance-caches.  Some examples of path preferences based on Capacity Profiles include: </ins> Low Latency/Interactive: Prefer paths with the lowest expected Round Trip Time, based on observed round trip time estimates; Round Trip Time, but can tolerate delay variation; Constant-Rate Streaming: Prefer paths that are expected to  <ins> satisfy the requested Stream Send or Stream Receive Bitrate, </ins> based on the observed maximum throughput; Capacity-Seeking: Prefer adapting to paths to determine the it to each new leaf node that is started and for which a protocol instance supporting 0-RTT is being used.  <ins> It is also possible for Protocol Stacks within a particular leaf node to use 0-RTT handshakes without any safely replayable application data if a protocol in the stack has idempotent handshake data to send.  For example, TCP Fast Open could use a Client Hello from TLS </ins> as its 0-RTT data, shortening the cumulative handshake time. 0-RTT handshakes often rely on previous state, such as TCP Fast Open can support cloning new Connection objects without a new explicit handshake.  <ins> Protocols also have a notion of Data Unit.  Possible values for Data Unit are: </ins> Byte-stream.  Byte-stream protocols do not define any Message boundaries of their own apart from the end of a stream in each API mappings for UDP are as follows:  <ins> UDP Connections represent a pair of specific IP addresses and </ins> ports on two hosts. CONNECT.UDP.  Calling "Initiate" on a UDP Connection causes it to


The implementation stores these properties as a part of the Preconnection object for use during connection establishment.  For Selection Properties that are not provided by the application, the <del> implementation must use the default values specified in the Transport </del>  Services API (I-D.ietf-taps-interface). 3.1. Upon receiving this event, a framer implementation is responsible for performing any necessary transformations and sending the resulting data back to the Message Framer, which will in turn send it to the <del> next protocol.  To improve performance, implementations SHOULD ensure </del>  that there is a way to pass the original data through without copying. that a new incoming stream should be represented as a new Connection Object by the Transport Services system.  A new SCTP stream is created by sending an SCTP message with a new stream id.  Thus, to <del> implement stream mapping, the Transport Services API MUST provide a </del>  newly created Connection Object to the application upon the reception of such a message.  The necessary semantics to implement a Transport Services system Close and Abort primitives are provided by the stream reconfiguration (reset) procedure described in RFC6525.  This also allows to re-use a stream id after resetting ("closing") the stream. To implement this functionality, SCTP stream reconfiguration RFC6525 <del> MUST be supported by both the client and the server side. </del>  <del> To avoid head-of-line blocking, stream mapping SHOULD only be </del>  implemented when both sides support message interleaving RFC8260. This allows a sender to schedule transmissions between multiple streams without risking that transmission of a large message on one To avoid conflicts between stream ids, the following procedure is recommended: the first Connection, for which the SCTP association has <del> been created, MUST always use stream id zero.  All additional </del>  Connections are assigned to unused stream ids in growing order.  To avoid a conflict when both endpoints map new Connections <del> simultaneously, the peer which initiated association MUST use even stream ids whereas the remote side MUST map its Connections to odd </del>  stream ids.  Both sides maintain a status map of the assigned stream <del> ids.  Generally, new streams SHOULD consume the lowest available </del>  (even or odd, depending on the side) stream id; this rule is relevant when lower ids become available because Connection objects associated with the streams are closed.

Two items here: Normative MUST? I guess this is just a guidance document so doesn't need anything normative. There's some later "should" as well which I would generally ask the same question for. but... here we have some NORMATIVE language. That SHOULD should probably be non-capitalised to align with earlier usage. (and so on later in the document, e.g. in the SCTP description)
The first one can just be "uses the default values", without normative text. Second one can be lowercase as well. SCTP has normative SHOULDs and MUSTs that can be removed.

The implementation stores these properties as a part of the Preconnection object for use during connection establishment.  For Selection Properties that are not provided by the application, the  <ins> implementation uses the default values specified in the Transport </ins> Services API (I-D.ietf-taps-interface). 3.1. Upon receiving this event, a framer implementation is responsible for performing any necessary transformations and sending the resulting data back to the Message Framer, which will in turn send it to the  <ins> next protocol.  To improve performance, implementations should ensure </ins> that there is a way to pass the original data through without copying. that a new incoming stream should be represented as a new Connection Object by the Transport Services system.  A new SCTP stream is created by sending an SCTP message with a new stream id.  Thus, to  <ins> implement stream mapping, the Transport Services API must provide a </ins> newly created Connection Object to the application upon the reception of such a message.  The necessary semantics to implement a Transport Services system Close and Abort primitives are provided by the stream reconfiguration (reset) procedure described in RFC6525.  This also allows to re-use a stream id after resetting ("closing") the stream. To implement this functionality, SCTP stream reconfiguration RFC6525  <ins> must be supported by both the client and the server side. </ins>  <ins> To avoid head-of-line blocking, stream mapping should only be </ins> implemented when both sides support message interleaving RFC8260. This allows a sender to schedule transmissions between multiple streams without risking that transmission of a large message on one To avoid conflicts between stream ids, the following procedure is recommended: the first Connection, for which the SCTP association has  <ins> been created, must always use stream id zero.  All additional </ins> Connections are assigned to unused stream ids in growing order.  To avoid a conflict when both endpoints map new Connections  <ins> simultaneously, the peer which initiated association must use even stream ids whereas the remote side must map its Connections to odd </ins> stream ids.  Both sides maintain a status map of the assigned stream  <ins> ids.  Generally, new streams should consume the lowest available </ins> (even or odd, depending on the side) stream id; this rule is relevant when lower ids become available because Connection objects associated with the streams are closed.


4. <del> [EDITOR'S NOTE: write three paragraph summary here; see https://github.com/taps-api/drafts/issues/93] </del>  In the following sections, we describe the details of application interaction with Objects through Actions and Events in each phase of <del> a connection, following the phases described in TAPS-ARCH. </del>  5. Endpoint or a Remote Endpoint in cases where this is required, for example with some NAT traversal protocols (see rendezvous). <del> [NOTE: the API needs MUST be explicit about when name resolution occurs, since the act of resolving a name leaks information, and there may be security implications if this happens unexpectedly.] </del> 5.2. A Preconnection object holds parameters reflecting the application's connections.  This list can be passed to a peer via a signalling protocol, such as SIP or WebRTC, to configure the remote. <del> [NOTE: This API is sufficient for TCP-style simultaneous open, but should be considered experimental for ICE-like protocols.] </del> 6.4. Groups of Connections can be created using Clone action:

There are still a few s and similar scattered throughout the document. Find them and fix them, or move them to post-London issues.

4.  <ins> The Transport Services Interface is the basic common abstract application programming interface to the Transport Services Architecture defined in TAPS-ARCH.  An application primarily interacts with this interface through two objects, Preconnections and Connections.  A Preconnection represents a set of parameters and constraints on the selection and configuration of paths and protocols to establish a connection with a remote endpoint.  A Connection represents a transport protocol stack on which data can be sent to and received from a remote endpoint.  Connections can be created from Preconnections in three ways: by initiating the Preconnection (i.e., actively opening, as in a client), through listening on the Preconnection (i.e., passively opening, as in a server), or rendezvousing on the Preconnection (i.e. peer to peer establishment). Once a Connection is established, data can be sent on it in the form of Messages.  The interface supports the preservation of message boundaries both via explicit protocol stack support, and via application support through a deframing callback which finds message boundaries in a stream.  Messages are received asynchronously through a callback registered by the application.  Error and other notifications also happen asynchronously on the Connection. </ins> In the following sections, we describe the details of application interaction with Objects through Actions and Events in each phase of  <ins> a connection, following the phases described in TAPS-ARCH.  An example interface sketch in Go is shown in appendix-api-sketch. </ins> 5. Endpoint or a Remote Endpoint in cases where this is required, for example with some NAT traversal protocols (see rendezvous).  5.2. A Preconnection object holds parameters reflecting the application's connections.  This list can be passed to a peer via a signalling protocol, such as SIP or WebRTC, to configure the remote.  6.4. Groups of Connections can be created using Clone action:


Once in use, UDP throws "soft errors" (ERROR.UDP(-Lite)) upon receiving ICMP notifications indicating failures in the network.  LISTEN.UDP.  Calling "Listen" for UDP binds a local port and prepares it to receive inbound UDP datagrams from peers.

First cut at new text
UDP, ConnectionError: Would those soft errors contain additional information about the ICMP error code, and can an application expect to find the (partial) original datagram in the error?
This is a good question! - on the one hand, the additional info can be valuable - for example to validate an ICMP message, as suggested for validating ICMP for a DPLPMTUD probe - but on the other hand, this info is often not provided by existing APIs.
I figure that a practical thing to state would be that the information is provided if it is accessible on a platform. Having text that describes this as matter of implementation quality would help TAPS implementers push for (eg. MSG_ERRQUEUE style) provisions in their platforms.
If the system makes this information available, we could mention that you could pass it up.
See a proposal in
The pointer to the PR with the specific proposal to address this issue got a "thumbs up" from the NAME who wrote the review. So, this PR should have closed this issue. Doing it manually.

Once in use, UDP throws "soft errors" (ERROR.UDP(-Lite)) upon receiving ICMP notifications indicating failures in the network. <ins> Additional information can also be provided when this is available to the system (e.g.  ICMP error code and quoted packet header). </ins> LISTEN.UDP.  Calling "Listen" for UDP binds a local port and prepares it to receive inbound UDP datagrams from peers.


Stop action is performed on the Listener object. The ConnectionReceived Event occurs when a Remote Endpoint has <del> established a transport-layer connection to this Listener (for Connection-oriented transport protocols), or when the first Message has been received from the Remote Endpoint (for Connectionless protocols), causing a new Connection to be created.  The resulting Connection is contained within the ConnectionReceived Event, and is ready to use as soon as it is passed to the application via the event. </del>  If the caller wants to rate-limit the number of inbound Connections that will be delivered, it can set a cap using Calling Clone on a Connection yields a Connection Group containing two Connections: the parent Connection on which Clone was called, and a resulting cloned Connection.  The new Connection is actively <del> openend, and it will send a Ready Event or an EstablishmentError Event.  Calling Clone on any of these Connections adds another Connection to the Connection Group.  Connections in a Connection Group share all Connection Properties except "connPriority" (see conn-priority), and these Connection Properties are entangled: Changing one of the Connection Properties on one Connection in the Connection Group automatically changes the Connection Property for all others.  For example, changing "connTimeout" (see conn-timeout) on one Connection in a Connection Group will automatically make the same change to this Connection Property for all other Connections in the Connection Group.  Like all other Properties, "connPriority" is copied to the new Connection when calling Clone(), but in this case, a later change to the "connPriority" on one Connection does not change it on the other Connections in the same Connection Group. </del>  The optional "connectionProperties" parameter allows passing Transport Properties that control the behavior of the underlying

I added text to ConnectionReceived about this. I didn't additionally explicitly state this in the text about clone to avoid complicating things - since similarly, the text describing Initiate() doesn't talk about ConnectionReceived.
If is called, is an unsolicited event generated at the peer? I assume so, but can't find it documented.
Do you want this to happen specifically upon calling clone(), or is it good enough if it happens upon the first arrival of data on the new Connection? The latter could be specified (and I'd have to check if it isn't already). However, in the first case, we could specify that this should happen if possible, but I don't think it's a behaviour that can be promised. For a multi-streaming protocol, URL() just means "from now on, we have a new Connection, for which we're going to use a new stream" - so I don't think that anything's immediately going to happen on the wire just due to clone() in this case.
I'd expect a event to be delivered before an event indicating arrival of data on the new connection. That could be immediately before, if there's no explicit on-the-wire signalling that the new connection has been created.
Ah, ok! I agree it should happen as you describe it, and I'll take care of this: I'll check if this is already covered (and if so, if the text should be more explicit about it). If it isn't covered, I'll add text to state this.

Stop action is performed on the Listener object. The ConnectionReceived Event occurs when a Remote Endpoint has  <ins> established or cloned (e.g., by creating a new stream in a multi- stream transport; see groups) a transport-layer connection to this Listener (for Connection-oriented transport protocols), or when the first Message has been received from the Remote Endpoint (for Connectionless protocols or streams of a multi-streaming transport), causing a new Connection to be created.  The resulting Connection is contained within the ConnectionReceived Event, and is ready to use as soon as it is passed to the application via the event. </ins> If the caller wants to rate-limit the number of inbound Connections that will be delivered, it can set a cap using Calling Clone on a Connection yields a Connection Group containing two Connections: the parent Connection on which Clone was called, and a resulting cloned Connection.  The new Connection is actively  <ins> openend, and it will locally send a Ready Event or an EstablishmentError Event.  Calling Clone on any of these Connections adds another Connection to the Connection Group.  Connections in a Connection Group share all Connection Properties except "connPriority" (see conn-priority), and these Connection Properties are entangled: Changing one of the Connection Properties on one Connection in the Connection Group automatically changes the Connection Property for all others.  For example, changing "connTimeout" (see conn-timeout) on one Connection in a Connection Group will automatically make the same change to this Connection Property for all other Connections in the Connection Group.  Like all other Properties, "connPriority" is copied to the new Connection when calling Clone(), but in this case, a later change to the "connPriority" on one Connection does not change it on the other Connections in the same Connection Group. </ins> The optional "connectionProperties" parameter allows passing Transport Properties that control the behavior of the underlying


either as a complete Message in the underlying protocol, or in multiple parts.  5.1. The effect of the application sending a Message is determined by the

NAME I think I disagree that this needs an API change, but rather a note that a messageData object can be empty. At least in the data objects we work with, they can support 0-length data blobs.
NAME I agree, you're right, this is better.
Is the content of messages always bytes? I certainly got that impression, but didn't see it explicitly either. (The "protocols that expose byte-streams" have that property visible in their name). When framers are used a lot, I can envision that at some point a message might not use data bytes at all any more, as all information in the underlying message is being dissected into protocol specific properties.
To give a bit of more context here: The protocol(s) I have in mind to implement in terms of TAPS is the CoAP family, which is currently specified for UDP, TCP and WebSockets transports. TCP and WebSockets would work well with framers that get data down to messages, but a first "layer" of framers would still produce different styles of messages -- concretely, the native UDP datagrams contain bytes [something, code, message-id, token, options, payload] whereas after a layer of de-framing TCP, what is left is [code, token, options, payload] plus an additional property (token length) that is extracted from a nibble in the framing. To make these messages usable more universally (i.e. independently of their transport), it'd make sense to de-frame these again (including de-framing the UDP messages) into decoding even more, moving code, token and options into properties. In the end, that will often result in messages that look like empty messages with a lot of properties -- is that OK? Would they even necessarily have a zero-length payload (byte-style message data)? Is the message's data just a (maybe special, maybe not) property of the message?
Messages can certainly be empty, and instead only contain Message Properties stored in the Message Context. We can make it clear that this is possible in section 5.2 of implementation.
LGTM. NAME says that text will be needed in the API draft to state that Messages can be empty. I'm not sure this is really necessary to add there; it wouldn't hurt, but it also doesn't get in the way of merging this PR.This text looks fine to me, but as discussed some complementing text about the possibility of a messageData object being empty will be needed in the API draft.

either as a complete Message in the underlying protocol, or in multiple parts. <ins> Messages themselves generally consist of bytes passed in the messageData parameter intended to be processed at an application layer.  However, Message objects presented through the API can carry associated Message Properties passed through the messageContext parameter.  When these are Protocol Specific Properties, they can include metadata that exists separately from a byte encoding.  For example, these Properties can include name-value pairs of information, like HTTP header fields.  In such cases, Messages might be "empty", insofar as they contain zero bytes in the messageData parameter, but can still include data in the messageContext that is interpreted by the Protocol Stack. </ins> 5.1. The effect of the application sending a Message is determined by the


the main Connection implementation that delivers events to the custom framer implementation whenever data is ready to be parsed or framed.  The Transport Services implementation needs to ensure that all of the events and actions taken on a Message Framer are synchronized to ensure consistent behavior.  For example, some of the actions defined

Framers: The paragraphs in Defining Message Framers and later might be formal language or pseudocode, I can't tell. It might help if the introduction stated something around names and signatures of signals and functions being expressed in pseudocode, and that identifiers in implementations are expected to be similar to those but following the conventions of the language
We should reference section 1.1 of the API document which defines the notation used.

the main Connection implementation that delivers events to the custom framer implementation whenever data is ready to be parsed or framed. <ins> The API examples in this section use the notation conventions for the Transport Services API defined in Section 1.1 of I-D.ietf-taps- interface. </ins> The Transport Services implementation needs to ensure that all of the events and actions taken on a Message Framer are synchronized to ensure consistent behavior.  For example, some of the actions defined


Calling "AbortGroup" on a UDP Connection (ABORT.UDP(-Lite)) is identical to calling "Close" on this Connection and on all <del> Connections in the same ConnectionGroup.  ## UDP-Lite </del>  Connectedness: Connectionless UDP-Lite, such as "msgChecksumLen", "fullChecksumSend", "recvChecksumLen", and "fullChecksumRecv". <del> 10.4. </del>  Connectedness: Connectionless A UDP Multicast Receive Connection is ready once the system has received traffic for the appropriate group and port. <del> UDP Multicast Receive Connections generate an InitiateError if Initiate is called. Once in use, UDP throws "soft errors" (ERROR.UDP(-Lite)) upon receiving ICMP notifications indicating failures in the network. </del>  LISTEN.UDP.  Calling "Listen" for UDP Multicast Receive binds a local port, prepares it to receive inbound UDP datagrams from (ABORT.UDP(-Lite)) is identical to calling "Close" on this Connection and on all Connections in the same ConnectionGroup. <del> 10.5. </del>  Connectedness: Connected

(More out of curiosity, with no expectation that this would find its way into the text unless you think I've hit a bug): In UDP Multicast Receive, which kind of ICMP notifications can come in there that would constitute a ConnectionError -- given that this transport does not send (and would thus usually not trigger any)?
Section 10.4 for multicast receive mentions ICMP errors. You could have errors on initiate for joining a multicast group, etc, but it likely doesn't make sense to have ICMP errors mentioned here.
Suggest we can close this issue.
LGTMSeems like we do not loose anything by removing this. The change WFM.

Calling "AbortGroup" on a UDP Connection (ABORT.UDP(-Lite)) is identical to calling "Close" on this Connection and on all  <ins> Connections in the same ConnectionGroup. 10.4. </ins> Connectedness: Connectionless UDP-Lite, such as "msgChecksumLen", "fullChecksumSend", "recvChecksumLen", and "fullChecksumRecv".  <ins> 10.5. </ins> Connectedness: Connectionless A UDP Multicast Receive Connection is ready once the system has received traffic for the appropriate group and port.  <ins> UDP Multicast Receive Connections generate an InitiateError indicating that joining a multicast group failed if Initiate is called. </ins> LISTEN.UDP.  Calling "Listen" for UDP Multicast Receive binds a local port, prepares it to receive inbound UDP datagrams from (ABORT.UDP(-Lite)) is identical to calling "Close" on this Connection and on all Connections in the same ConnectionGroup.  <ins> 10.6. </ins> Connectedness: Connected


resolution, when support mechanisms such as TURN or ICE are used to establish connectivity, if protocols or paths are raised, or if a path fails and fallback or re-establishment is supported in the <del> Transport Services system. </del>  Applications should also take care to not assume that all data received using the Transport Services API is always complete or well-

"In effect, each leaf node will send the same early application data, yet encoded (encrypted) differently on the wire.": There has been a warning on the privacy implications of using the same token on different paths; doesn't sending an identical 0-RTT message racingly on multiple transports cause similar correlation issues?
This may be better to put in the privacy and security considerations section of the API document, particularly that doing cleartext 0-RTT data that races can be recognized on the path (as opposed to being in TLS which fixes this).
Similarly, connection racing overall can let a server observe that a client has multiple addresses and what they are, etc.
LGTM; note that I left a tiny comment.

resolution, when support mechanisms such as TURN or ICE are used to establish connectivity, if protocols or paths are raised, or if a path fails and fallback or re-establishment is supported in the  <ins> Transport Services system.  Applications should take special care when using 0-RTT session resumption (see prop-0rtt), as early data sent across multiple paths during connection establishment may reveal information that can be used to correlate endpoints on these paths. </ins> Applications should also take care to not assume that all data received using the Transport Services API is always complete or well-


use of temporary addresses, the resulting Connection cannot use IPv4, because temporary addresses do not exist in IPv4.  Second, temporary local addresses might involve trading off privacy for performance. <del> For instance, temporary addresses can interfere with resumption mechanisms that some protocols rely on to reduce initial latency. </del>  6.2.14.

Adding this issue from Fernando via the TAPS list: The terminology employed in this subsection seems a bit confusing, or at least deviates from what we normally use in v6-circles. I suggest taking a look at RFC7721 -- certainly not flawless, but it was 6man's shot to try to agree on terminology. For example, I don't think we use "permanent" and "stable" interchangeably. e.g. the "Local" in "useTemporaryLocalAddress" might be misleading to some. -- e.g. could be interpreted as related with link-locals or ULAs. Aside, I believe there's much more to dig regarding address properties. e.g., we have given it a shot at: URL e.g., the choice of ipv6 addresses has more than one dimension... and there are tradeoffs involved, which IMHO deserve discussion. And aside from the regular address systems normally configure, if one were to specify an API one would probably want to provide a mechanism for apps to be able to request sole/single-use addresses (.e.g., one address per app or container).
"Local" has a meaning in the TAPS context, I don't think we need to change it for universalization with the full set of IPv6 terminology. There's also not much point in going all the way down the rabbithole for this particular property, given that we don't do so for any other -- this document is meant as a universal interface to networking for the future, not an unbounded treatise on the future of networking. That said, the linked draft does seem to address the issues here and it's fairly alive, so no harm in adding an informative reference to it.
I agree with the comments of NAME and NAME

use of temporary addresses, the resulting Connection cannot use IPv4, because temporary addresses do not exist in IPv4.  Second, temporary local addresses might involve trading off privacy for performance.  <ins> For instance, temporary addresses (e.g., RFC8981) can interfere with resumption mechanisms that some protocols rely on to reduce initial latency. </ins> 6.2.14.


application's data. Connection establishment is divided into two top-level steps: <del> Candidate Gathering, to identify the paths, protocols, and endpoints to use, and Candidate Racing (see Section 4.2.2 of I-D.ietf-taps- arch), in which the necessary protocol handshakes are conducted so that the transport system can select which set to use. </del>  For ease of illustration, this document structures the candidates for <del> racing as a tree.  This is not meant to restrict implementations from structuring racing candidates differently. </del>  The most simple example of this process might involve identifying the single IP address to which the implementation wishes to connect, section is the algorithm defining which of these options to try, when, and in what order. <del> During Candidate Gathering, an implementation prunes and sorts branches according to the Selection Property preferences (Section 6.2 of I-D.ietf-taps-interface.  It first excludes all protocols and paths that match a Prohibit property or do not match all Require properties.  Then it will sort branches according to Preferred properties, Avoided properties, and possibly other criteria. </del>  4.1.

Raising this issue as part of my review of the the Implementation Draft. To a reader who understands "protocol racing" in the "Happy Eyeballs" sense of near-simultaneous connection establishment attempts, parts of Section 4.1 are confusing. It appears as if "real racing" actually only occurs between Derived Endpoints and not e.g., among Alternate Paths, where we instead have "failover after timeout": Section 4.1.1.2 However, in Section 4.1.1.3 "Protocol Options", the "rank[ing] and attempt[ing]" of options is also referred to as "racing": It seems clear that this kind of information is only useful when connection attempts are not in fact "raced" (nearly) in parallel, but are only conducted in succession, where it makes sense to rank the candidates accordingly. This confusion is only cleared up in Section 4.3, where it says: So it appears as if (in partial departure from the Happy Eyeballs RFC) the term "racing" here is meant to include staggered or failover scenarios, somewhat undermining the original metaphor. Suggestions: Explain the extended scope of the "racing" term earlier Perhaps replace some uses the term "racing" with "failover" or even just "ranking"
Mention that "staggered" is the most common form of racing The main thing with racing is that the first one to succeed wins We should add forward references to sections here: We should also explain in this paragraph that the "Staggered" approach is generally the default for racing, and the first one to finish wins.
LGTM, thanks!

application's data. Connection establishment is divided into two top-level steps:  <ins> Candidate Gathering (defined in Section 4.2.1 of I-D.ietf-taps-arch), to identify the paths, protocols, and endpoints to use (see gathering); and Candidate Racing (defined in Section 4.2.2 of I- D.ietf-taps-arch), in which the necessary protocol handshakes are conducted so that the transport system can select which set to use (see racing).  Candidate Racing involves attempting multiple options for connection establishment, and choosing the first option to succeed as the Protocol Stack to use for the connection.  These attempts are usually staggered, starting each next option after a delay, but they can also be performed in parallel or only after waiting for failures. </ins> For ease of illustration, this document structures the candidates for  <ins> racing as a tree (see tree-structure).  This is not meant to restrict implementations from structuring racing candidates differently. </ins> The most simple example of this process might involve identifying the single IP address to which the implementation wishes to connect, section is the algorithm defining which of these options to try, when, and in what order.  <ins> During Candidate Gathering (gathering), an implementation prunes and sorts branches according to the Selection Property preferences (Section 6.2 of I-D.ietf-taps-interface.  It first excludes all protocols and paths that match a Prohibit property or do not match all Require properties.  Then it will sort branches according to Preferred properties, Avoided properties, and possibly other criteria. </ins> 4.1.


of the tree has successfully completed negotiation with the Remote Endpoint, or else all nodes of the tree have failed to connect.  The first leaf node to complete its connection is then used by the <del> application to send and receive data. </del>  Successes and failures of a given attempt should be reported up to parent nodes (towards the trunk of the tree).  For example, in the 4.4.1. <del> Implementations may select the criteria by which a leaf node is considered to be successfully connected differently on a per-protocol basis.  If the only protocol being used is a transport protocol with a clear handshake, like TCP, then the obvious choice is to declare that node "connected" when the last packet of the three-way handshake has been received.  If the only protocol being used is an </del>  connectionless protocol, like UDP, the implementation may consider the node fully "connected" the moment it determines a route is present, before sending any packets on the network, see further quickly, but TLS negotiation is either very slow or fails altogether in particular network conditions or to a particular endpoint.  To avoid the issue of TLS possibly failing, the implementation should <del> not generate a Ready event for the Connection until TLS is established. </del>  If all of the leaf nodes fail to connect during racing, i.e. none of the configurations that satisfy all requirements given in the Transport Properties actually work over the available paths, then the <del> transport system should notify the application with an InitiateError event.  An InitiateError event should also be generated in case the transport system finds no usable candidates to race. </del>  4.5. called without Messages being handed over, it cannot be guaranteed that the Remote Endpoint will have any way to know about this, and hence a passive endpoint's ConnectionReceived event might not be <del> called until data is received.  Instead, calling the </del>  ConnectionReceived event could be delayed until the first Message arrives. However, this can cause a problem if a specific peer is not reachable over the network using the connectionless protocol, or data cannot be exchanged with the peer for any other reason.  To mitigate this, an <del> application can use a Message Framer message-framers on top of a </del>  connectionless protocol to only mark a specific connection attempt as ready when some data has been received, or after some application- level handshake has been performed. The most basic mapping for sending a Message is an abstraction of datagrams, in which the transport protocol naturally deals in <del> discrete packets.  Each Message here corresponds to a single datagram.  Generally, these will be short enough that sending and receiving will always use a complete Message. </del>  <del> For protocols that expose byte-streams, the only delineation provided by the protocol is the end of the stream in a given direction.  Each Message in this case corresponds to the entire stream of bytes in a direction.  These Messages may be quite long, in which case they can be sent in multiple parts. </del>  Protocols that provide the framing (such as length-value protocols, <del> or protocols that use delimiters) may support Message sizes that do not fit within a single datagram.  Each Message for framing protocols corresponds to a single frame, which may be sent either as a complete Message in the underlying protocol, or in multiple parts. </del>  5.1. 5.1.1.  "msgLifetime": this should be implemented by removing the Message from the queue of pending Messages after the Lifetime has expired. A queue of pending Messages within the transport system complete Message, the implementation should wait until an entire Message (datagram, stream, or frame) is read before delivering any Message content to the application.  This requires the implementation <del> to understand where messages end, either via a supplied or because the top-level protocol in the established Protocol Stack preserves message boundaries.  If the top-level protocol only supports a byte- stream and no framers were supported, the application can control the flow of received data by specifying the minimum number of bytes of Message content it wants to receive at one time. </del>  If a Connection finishes before a requested Receive action can be satisfied, the Transport Services API should deliver any partial connection. If the application has 0-RTT data to send during handshake(s), it <del> needs to provide this data before the handshakes have begun.  When racing, this means that the data should be provided before the process of connection establishment has begun.  If the application wants to send 0-RTT data, it must indicate this to the implementation by setting the "Safely Replayable" send parameter to true when sending the data.  In general, 0-RTT data may be replayed (for example, if a TCP SYN contains data, and the SYN is retransmitted, the data will be retransmitted as well but may be considered as a new connection instead of a retransmission).  Also, when racing connections, different leaf nodes have the opportunity to send the same data independently.  If data is truly safely replayable, this should be permissible. </del>  Once the application has provided its 0-RTT data, a Transport Services implementation should keep a copy of this data and provide implementation, but an implementation ought to allow custom Message Framers to be defined by the application or some other piece of software.  This section describes one possible API for defining <del> Message Framers as an example. </del>  6.1.

NB: This reviewer may be unreasonably biased against unnecessary mention of event-driven interaction patterns. Viewed in concert with the other pre-requisite TAPS documents, the text of this draft is quite readable and represents a useful collection of implementation considerations for various components of a transport services system. My biggest point of concern is the fluctuating level of detail between the different Sections, leading to speculations about who might have written what. Some Sections remain quite general and avoid making too many assumptions about the underlying software architecture. Others discuss a specific implementation approach in some detail. In particular Section 6 (Message Framers) stands out from the rest of the text. Language that relates to event-driven interaction patterns is almost exclusively found in this Section, where different types of events are "generated", "delivered", "received" or "handled" by and between various components. In the rare cases where events are mentioned in the other Sections, they are often associated with other verbs like "notify", "cause", "provoke" or even "call" (see, e.g., Section 4.5 or Section 8). Indeed, the text in most of these other Sections would require only very minor edits to avoid any reference to an event-driven architecture altogether (examples below). It is interesting that the same can not easily be said of Section 6, alone. Overall, the text could benefit from an edit pass that normalizes language and (use of) terms and adds a few more references to related documents. Other than that, I have few issues with it. Below are a few remarks and suggestions to cherry-pick from, as usual. Quoting from 4.4.1. Determining Successful Establishment: Not only is the term "Ready event" used without any sort of context or introduction, it is the first point in the text where familiarity with a specific event type is implied. The intent of the whole sentence could be much better conveyed without this terminological baggage. Suggestion: From later in the section, quoting: While it is reasonable to mention the specific error type here (although a reference to the document where it is defined is still missing), wouldn't the following suffice? Quoting from 4.5. Establishing multiplexed connections: Shouldn't events rather be "emitted" or "triggered" instead of (ambiguously) "called"? It is gratifying to observe that Sections 4.6 and 4.7 on the other hand entirely avoid specific references to an event-based architecture, instead speaking of, e.g., "passing" Connection objects to the application. Quoting from 6. Implementing Message Framers: Given that this section is the only place in the text where concepts and language centered around an event-based implementation are consistently used, it would be appropriate to point this out in its introduction: Quoting 4.4.1. Determining Successful Establishment: The adverb "differently" is too far removed from "select", becoming ambiguous and could be wrongly understood to modify "connected" instead. Suggestion: Quoting 4.6. Handling connectionless protocols: The reference to the section is ungrammatical. About 5. Implementing Sending and Receiving Data: Might it not be helpful to conclude the introduction with some examples: UDP Datagrams, TCP streams, HTTP responses? About 5.1.1. Message Properties: The list of Message Properties is given without any sort of context or introduction. What's the story here, is this an exhaustive list or just a list of those properties that can be adjusted to modify the send and receive behavior? Quoting 5.2. Sending Messages: That sentence is exceedingly hard to parse. Quoting 5.3. Handling of data for fast-open protocols: Possibly intentional redundant wording, but unfortunately distracting.
LGMTThanks for taking care of the edits! Just added a nit and a question. I assume you deliberately skipped the comment from the issue about event-based terminology in Section 6, not thinking it was needed as we reference the API draft for the terminology?

of the tree has successfully completed negotiation with the Remote Endpoint, or else all nodes of the tree have failed to connect.  The first leaf node to complete its connection is then used by the  <ins> application to send and receive data.  This is signalled to the application using the Ready event in the API (Section 7.1 of I- D.ietf-taps-interface). </ins> Successes and failures of a given attempt should be reported up to parent nodes (towards the trunk of the tree).  For example, in the 4.4.1.  <ins> On a per-protocol basis, implementations may select different criteria by which a leaf node is considered to be successfully connected.  If the only protocol being used is a transport protocol with a clear handshake, like TCP, then the obvious choice is to declare that node "connected" when the last packet of the three-way handshake has been received.  If the only protocol being used is an </ins> connectionless protocol, like UDP, the implementation may consider the node fully "connected" the moment it determines a route is present, before sending any packets on the network, see further quickly, but TLS negotiation is either very slow or fails altogether in particular network conditions or to a particular endpoint.  To avoid the issue of TLS possibly failing, the implementation should  <ins> not generate a Ready event for the Connection until the TLS handshake is complete. </ins> If all of the leaf nodes fail to connect during racing, i.e. none of the configurations that satisfy all requirements given in the Transport Properties actually work over the available paths, then the  <ins> transport system should report an InitiateError to the application. An InitiateError event should also be generated in case the transport system finds no usable candidates to race. </ins> 4.5. called without Messages being handed over, it cannot be guaranteed that the Remote Endpoint will have any way to know about this, and hence a passive endpoint's ConnectionReceived event might not be  <ins> delivered until data is received.  Instead, delivering the </ins> ConnectionReceived event could be delayed until the first Message arrives. However, this can cause a problem if a specific peer is not reachable over the network using the connectionless protocol, or data cannot be exchanged with the peer for any other reason.  To mitigate this, an  <ins> application can use a Message Framer (message-framers) on top of a </ins> connectionless protocol to only mark a specific connection attempt as ready when some data has been received, or after some application- level handshake has been performed. The most basic mapping for sending a Message is an abstraction of datagrams, in which the transport protocol naturally deals in  <ins> discrete packets (such as UDP).  Each Message here corresponds to a single datagram.  Generally, these will be short enough that sending and receiving will always use a complete Message. </ins>  <ins> For protocols that expose byte-streams (such as TCP), the only delineation provided by the protocol is the end of the stream in a given direction.  Each Message in this case corresponds to the entire stream of bytes in a direction.  These Messages may be quite long, in which case they can be sent in multiple parts. </ins> Protocols that provide the framing (such as length-value protocols,  <ins> or protocols that use delimiters like HTTP/1.1) may support Message sizes that do not fit within a single datagram.  Each Message for framing protocols corresponds to a single frame, which may be sent either as a complete Message in the underlying protocol, or in multiple parts. </ins> 5.1. 5.1.1. <ins> The API allows various properties to be associated with each Message, which should be implemented as discussed below. </ins> "msgLifetime": this should be implemented by removing the Message from the queue of pending Messages after the Lifetime has expired. A queue of pending Messages within the transport system complete Message, the implementation should wait until an entire Message (datagram, stream, or frame) is read before delivering any Message content to the application.  This requires the implementation  <ins> to understand where messages end, either via a supplied Message Framer or because the top-level protocol in the established Protocol Stack preserves message boundaries.  If the top-level protocol only supports a byte-stream and no framers were supported, the application can control the flow of received data by specifying the minimum number of bytes of Message content it wants to receive at one time. </ins> If a Connection finishes before a requested Receive action can be satisfied, the Transport Services API should deliver any partial connection. If the application has 0-RTT data to send during handshake(s), it  <ins> needs to provide this data before the handshakes have begun.  If the application wants to send 0-RTT data, it must indicate this to the implementation by setting the "Safely Replayable" send parameter to true when sending the data.  In general, 0-RTT data may be replayed (for example, if a TCP SYN contains data, and the SYN is retransmitted, the data will be retransmitted as well but may be considered as a new connection instead of a retransmission).  When racing connections, different leaf nodes have the opportunity to send the same data independently.  If data is truly safely replayable, this is permissible. </ins> Once the application has provided its 0-RTT data, a Transport Services implementation should keep a copy of this data and provide implementation, but an implementation ought to allow custom Message Framers to be defined by the application or some other piece of software.  This section describes one possible API for defining  <ins> Message Framers, as an example. </ins> 6.1.


system objects that may be temporarily unavailable. Separate from events, callbacks are also provided for asynchronous <del> interactions with the Transport Services API, which are not directly </del>  related to events on the network or network interfaces. 2.2.

Clarified one use of word "API" comes just after discussion of the socket API. Tried to fix specialised features to say they define additional requirements. Minor changes to wording, with no intended changes to substance.
LGTM but I made one very small comment

system objects that may be temporarily unavailable. Separate from events, callbacks are also provided for asynchronous  <ins> interactions with the Transport Services API that are not directly </ins> related to events on the network or network interfaces. 2.2.


The effect of the application sending a Message is determined by the top-level protocol in the established Protocol Stack.  That is, if the top-level protocol provides an abstraction of framed messages <del> over a connection, the application will be able to send multiple Messages on that connection, even if the framing protocol is built on a byte-stream protocol like TCP. </del>  5.1.1.1. if the application only wants to receive a complete Message, the implementation should wait until an entire Message (datagram, stream, or frame) is read before delivering any Message content to the <del> application.  Alternatively, the application can specify the minimum number of bytes of Message content it wants to receive (which may be just a single byte) to control the flow of received data. </del>  If a Connection becomes finished before a requested Receive action can be satisfied, the implementation should deliver any partial the TCP state can be closed abruptly, leading to a RST being sent to the peer. <del> Without a layer of framing above TCP, the cleanest abstraction for sending and receiving Messages over TCP is to treat each direction of the stream of bytes as a single Message, terminated by a FIN.  That means that if the application can make several Send calls to enqueue subsequent data chunks for the same Message to continue writing, but marking the Message complete corresponds to closing the sending stream.  Similarly, when the application receives the final portion of a Message, it knows that the receiving stream has been closed. </del>  9.2. the underlying protocol stack does not support message boundary preservation, and no deframer was supplied by the application <del> In this case, the Message object passed to Received contains an indication that the object received is partial, the byte offset of the data in the partial Message within the full Message, an indication whether this is the last (highest-offset) partial Message in the full Message, and an optional reference to the full Message it belongs to. Note that in the degenerate case - no message boundary preservation and no deframing - the entire connection is represented as one large message of indeterminate length. </del>  A ReceiveError occurs when data is received by the underlying Protocol Stack that cannot be fully retrieved or deframed, or when

The previous text didn't support sending multiple messages across vanilla TCP without framers/deframers or a top-level protocol in the stack that would de-frame. This change allows sending multiple messages across vanilla TCP and handing them over as bytes to an application that, by itself, detects message boundaries from the chunks of bytes that it gets.
It seems we never came to consensus over the main issue here - we did have consensus that it's fine to have framers and deframers. It seems that the crux of the matter is the following: Tommy and you want: a sender saying "the frame ends here" will terminate a TCP connection; so if you really really want to hand over multiple chunks of data from a stream, then you should use one huge message and only terminate when finished with everything. I don't want that, for the reason I explained so often under the heading "Application-Framed Bytestream": the sending application doesn't know whether it gets vanilla TCP or e.g. SCTP. If it gets SCTP, messages can be sent as messages, with all their properties: lifetime, unordered delivery ... but this can only work if the sender knows that messages are messages. If the transport system gets TCP, however, it must be able to send the messages directly on the bytestream. Either way this is fine as long as the receiving application is able to determine boundaries in the chunk of data that it gets: in case of SCTP, it really does get messages, in case of TCP, it gets a bytestream. By requiring "atomic" message transmission from one end to the other, and not being able to send multiple messages on a single TCP connection, this functionality becomes unavailable. If this seems confusing, please consider taking another look at the animations, e.g. here: URL If you want a longer, more detailed explanation, please read: URL If you don't believe this can work, please download and test NEAT. I don't understand all this "a TCP connection represents a large message business". It's a strange limit that gets in the way of efficiently falling back to vanilla TCP.
our lack of consensus seems to me to come from a giant pile of subtle misunderstandings: nope. Well, at least I don't want this. This would be silly. There's no guarantee that one side of a connection using this interface will be connected to another side of a connection also using the interface. Message sends are passed down to the underlying protocol stack atomically, so that stacks that do indeed support boundary preservation have boundaries. If you get TCP, it'll ignore the boundaries, and hope the other side provides deframing. (If a sender absolutely requires atomicity without relying on receiver-side deframing, it could prohibit the selection of protocol stacks that don't support it, of course.) But we don't require atomic message transmission. We require an API which presents atomic message transmission to the application when it is available, and allows (asks) for the application to help provide this when it is not. Here's what this looks like when a sender writes multiple messages on a stack not supporting boundary preservation to a receiver not running this interface: The receiver gets a pile or bytes in order and hands them to an app layer parser, just like in TCP. Here's what this looks like when a TCP only-sender writes three app-layer-delimited messages to a stream to a receiver running this interface with deframers: The receiver pushed the deframe logic down into the interface, and got to simplify its own workflow -- every callback has an atomic message. Here's what this looks like in the case this PR seems to care about: TCP, no deframer: Just like the TCP case above. Each event is just like a stream call in TCP. (I can, indeed, see why you'd want to provide length hints to to support this, and I think we should add that.) I believe it can work, that's not the question at all. I just don't think it's a very useful abstraction if a goal of building a new interface is moving application development away from stream-centric assumptions in order to allow it to take advantage of new transports that provide better-than-single-stream abstractions. There's no limit here. To translate this interface into bytestream thinking: a partial message is simply a section of a bytestream with a running offset (which can itself be calculated trivially by the stack or the system). When a receiver doesn't have enough information to atomize messages, it presents bytestream(s) as partials.
So in this comment, I agree with absolutely everything. Just two comments: Great!! But I was under the strong impression that this is what Tommy wants because of things that he wrote, both in the text and in issue . Fantastic if I misunderstood all that, but then my suggested text changes should be fine. All I want is to allow: 1) sending many messages (multiple "message over" markers from the application) over a single TCP connection 2) receiving blocks of data (similar to normal TCP data reception today) from which an application determines where messages begin and end. Just for the record, that didn't come from me - I suspect it was from Tommy. Anyway, it makes sense to me as well.
hm i think that might have been me. such is the nature of the pile of confusions...
with the caveat that these might be completely lost on send, yep, absolutely we need to allow this. yep, with the caveat that we do not yet have a way to pass the max length the application wants to receive (but can easily add this). Hm, I think you still need the "this is a partial message" indicator on the receive (which of course the application knows, but allowing the "partial message" marker to be omitted when the underlying transport / deframe stack doesn't have any boundary information seems like a contract violation to me.
Yes, of course. Same as happens today. Yes, although the way I was envisaging this working was that the application provides a de-framing function to the stack, and that de-framing function determines where the messages begin and end, so that what's returned are complete messages. That way, you get a message based abstraction even if running over a byte-stream transport like TCP.
I agree with the caveats you list above, that's all fine. About this: The point is that the receiving application must be able to know if it did get a complete message. When it does, the transport system can say so, so there you get your contract. When it does not, the receiver-side transport system may not know about it because it doesn't know about sender-supported message boundaries in the first place (the first of your two mentioned caveats). If it got the data as a TCP stream, then at best, it can say "perhaps this isn't a complete message, I'm not sure", but it certainly can't provide the other things your text requires in this case (whether it's a partial message, a reference to the complete message, ..). If we phrase it as I suggest, I think we break no contract and we promise as much as we can promise.
AHHHH okay. Actually, I think this should be a three-state indicator: (1) This is a complete message (known because the deframer said so, or the transport stack did). (2) This is a partial message containing a section of a message with a known message boundary (made partial for local buffering reasons, either by the underlying protocol stack or the deframer) (3) This is a partial message containing a data with no definite message boundary (i.e. stream protocol and no deframer)...
CONVERGED! I agree with the 3-state indicator proposal :)
NAME I like your last list of three, but I will still point out that over any stream protocol with no deframer, there will be a definite end to received data (the equivalent of a FIN). My previous comments about FINs, etc, were just to point out that it is possible to map a stream this way. If we're doing HTTP/2 or QUIC streams, we may indeed want the end of the message to correspond to the end of the stream, since HTTP over HTTP/2 generally will do one request per stream identifier. However, I had also mentioned that it is possible to have the stream protocol ignore any claimed message boundaries, but this means that the message boundaries are understood to be lost when mapped into the bytes on the wire.
NAME yes..... I think that makes sense
NAME - this sounds fine! (phew!) So we agree that a sender can send 50 small messages over a single connection (each marked with "end of message"), right? And when the transport system ends up using vanilla TCP, and the receiver-side application then wants to obtain the messages, it must provide a deframer (note that the "must" here is new to me, but I think it's fine and probably even a good idea in order to do cooler things later).
Well, yes... but the application won't know what the end is until it sees the FIN. so s/no definite message boundary/no known message boundary/ in state 3? (git logistics question: NAME do you want to update this branch (which will automatically update the PR) with receive length and three-state partials, or shall we land this PR then do those in another?)
I'll update the PR next (first time I'm doing this - sorry for being clumsy)
This looks good to me now. I'll do a separate PR to add receive size to the API.
Ah, sorry, I forgot this to do this (receive size); thanks!
URL
Will land both of these; open subsequent issues if we need to tweak the text, but IMO this is certainly -00 quality...

The effect of the application sending a Message is determined by the top-level protocol in the established Protocol Stack.  That is, if the top-level protocol provides an abstraction of framed messages  <ins> over a connection, the receiving application will be able to obtain multiple Messages on that connection, even if the framing protocol is built on a byte-stream protocol like TCP. </ins> 5.1.1.1. if the application only wants to receive a complete Message, the implementation should wait until an entire Message (datagram, stream, or frame) is read before delivering any Message content to the  <ins> application.  This requires the implementation to understand where messages end, either via a supplied deframer or because the top-level protocol in the established Protocol Stack preserves message boundaries; if, on the other hand, the top-level protocol only supports a byte-stream and no deframers were supported, the application must specify the minimum number of bytes of Message content it wants to receive (which may be just a single byte) to control the flow of received data. </ins> If a Connection becomes finished before a requested Receive action can be satisfied, the implementation should deliver any partial the TCP state can be closed abruptly, leading to a RST being sent to the peer.  <ins> Without a layer of framing (a top-level protocol in the established Protocol Stack that preserves message boundaries, or an application- supplied deframer) on top of TCP, the receiver side of the transport system implementation can only treat the incoming stream of bytes as a single Message, terminated by a FIN when the Remote Endpoint closes the Connection. </ins> 9.2. the underlying protocol stack does not support message boundary preservation, and no deframer was supplied by the application  <ins> The Message object passed to Received will indicate one of the following: this is a complete message; this is a partial message containing a section of a message with a known message boundary (made partial for local buffering reasons, either by the underlying protocol stack or the deframer).  In this case, the Message object passed to Received may contain the byte offset of the data in the partial Message within the full Message, an indication whether this is the last (highest-offset) partial Message in the full Message, and an optional reference to the full Message it belongs to; or this is a partial message containing data with no definite message boundary, i.e. the only known message boundary is given by termination of the Connection Note that in the absence of message boundary preservation and without deframing, the entire connection is represented as one large message of indeterminate length. </ins> A ReceiveError occurs when data is received by the underlying Protocol Stack that cannot be fully retrieved or deframed, or when


Preference)".  Instances take a sequence of values each valid for the corresponding value type. <del> Array: Denoted []Type, an instance takes a value for each of zero or more elements in a sequence of the given Type.  An array may be of fixed or variable length. </del>  Collection: An unordered grouping of one or more values of the same type. The following API calls can be used to configure a Preconnection before calling Initiate(): <del> "RemoteSpecifier.WithMulticastGroupIPv4(GroupAddress) RemoteSpecifier.WithMulticastGroupIPv6(GroupAddress) RemoteSpecifier.WithPort(PortNumber) RemoteSpecifier.WithTTL(TTL) " </del> Calling Listen() on a Preconnection with a multicast group specified on the Remote Endpoint will join the multicast group to receive messages.  This Listener will create one Connection for each Remote MessagContext on the messages received (see msg-ctx for further details). <del> "LocalSpecifier.WithSingleSourceMulticastGroupIPv4(GroupAddress, SourceAddress) LocalSpecifier.WithSingleSourceMulticastGroupIPv6(GroupAddress, SourceAddress) LocalSpecifier.WithAnySourceMulticastGroupIPv4(GroupAddress) LocalSpecifier.WithAnySourceMulticastGroupIPv6(GroupAddress) LocalSpecifier.WithPort(PortNumber) LocalSpecifier.WithTTL(TTL) " </del> Calling Rendezvous() on a Preconnection with an any-source multicast group address as the Remote Endpoint will join the multicast group, and also indicates that the resulting connection can be used to send Rendezvous() on a Preconnection with a source-specific multicast group address as the Local Endpoint results in an EstablishmentError. <del> "RemoteSpecifier.WithMulticastGroupIPv4(GroupAddress) RemoteSpecifier.WithMulticastGroupIPv6(GroupAddress) RemoteSpecifier.WithPort(PortNumber) RemoteSpecifier.WithTTL(TTL) LocalSpecifier.WithAnySourceMulticastGroupIPv4(GroupAddress) LocalSpecifier.WithAnySourceMulticastGroupIPv6(GroupAddress) LocalSpecifier.WithPort(PortNumber) LocalSpecifier.WithTTL(TTL) " </del> See multicast-examples for more examples. 6.1.2. address other than the one initially used, it is necessary to set the "advertisesAltaddr" property (see altaddr below). <del> Setting this property to "Active", can have privacy implications: It </del>  enables the transport to establish connectivity using alternate paths that might result in users being linkable across the multiple paths, even if the "advertisesAltaddr" property (see altaddr below) is set

Review the use of and markdown in the API document, and make it consistent.
A fantastic PR! The best ever!

Preference)".  Instances take a sequence of values each valid for the corresponding value type.  <ins> Array: Denoted "[]Type", an instance takes a value for each of zero or more elements in a sequence of the given Type.  An array may be of fixed or variable length. </ins> Collection: An unordered grouping of one or more values of the same type. The following API calls can be used to configure a Preconnection before calling Initiate():  Calling Listen() on a Preconnection with a multicast group specified on the Remote Endpoint will join the multicast group to receive messages.  This Listener will create one Connection for each Remote MessagContext on the messages received (see msg-ctx for further details).  Calling Rendezvous() on a Preconnection with an any-source multicast group address as the Remote Endpoint will join the multicast group, and also indicates that the resulting connection can be used to send Rendezvous() on a Preconnection with a source-specific multicast group address as the Local Endpoint results in an EstablishmentError.  See multicast-examples for more examples. 6.1.2. address other than the one initially used, it is necessary to set the "advertisesAltaddr" property (see altaddr below).  <ins> Setting this property to "Active" can have privacy implications: It </ins> enables the transport to establish connectivity using alternate paths that might result in users being linkable across the multiple paths, even if the "advertisesAltaddr" property (see altaddr below) is set


MessagContext on the messages received (see msg-ctx for further details).  Calling Rendezvous() on a Preconnection with an any-source multicast group address as the Remote Endpoint will join the multicast group, and also indicates that the resulting connection can be used to send Rendezvous() on a Preconnection with a source-specific multicast group address as the Local Endpoint results in an EstablishmentError.  See multicast-examples for more examples. 6.1.2.

Fix-up multicast
Fixing the sequence of some examples vs. the text describing them, and referring to examples ("The following API calls can be used ...")
Also a great PR! :) Seriously, thanks!

MessagContext on the messages received (see msg-ctx for further details). <ins> The following API calls can be used to configure a Preconnection before calling Listen(): </ins> Calling Rendezvous() on a Preconnection with an any-source multicast group address as the Remote Endpoint will join the multicast group, and also indicates that the resulting connection can be used to send Rendezvous() on a Preconnection with a source-specific multicast group address as the Local Endpoint results in an EstablishmentError. <ins> The following API calls can be used to configure a Preconnection before calling Rendezvous(): </ins> See multicast-examples for more examples. 6.1.2.