without requiring changes to the applications.  The specified API follows the Transport Services Architecture by providing asynchronous, atomic transmission of messages.  It is intended to <del> replace the traditional BSD sockets API as the common interface to the transport layer, in an environment where endpoints could select from multiple interfaces and potential transport protocols. </del>  1. <del> This document specifies a modern abstract application programming interface (API) atop the high-level architecture for transport services defined in I-D.ietf-taps-arch.  The Transport Services Architecture supports asynchronous, atomic transmission of messages over transport protocols and network paths dynamically selected at runtime, in environments where an endpoint selects from multiple interfaces and potential transport protocols. </del>  Applications that adopt this interface will benefit from a wide set of transport features that can evolve over time.  This protocol- This API derives specific path and protocol selection properties and supported transport features from the analysis provided in RFC8095, <del> RFC8923, and RFC8922.  The design encourages implementations underneath the interface to dynamically choose a transport protocol depending on an application's choices rather than statically binding applications to a protocol at compile time.  Nevertheless, the Transport Services API also provides applications with a way to override transport selection and instantiate a specific stack, e.g., to support servers wishing to listen to a specific protocol. However, forcing a specific transport stack choice is discouraged for general use, because it can reduce portability. </del>  1.1. Asynchronous Connection establishment, transmission, and reception.  This allows concurrent operations during establishment and event-driven application interactions with the transport <del> layer, in line with developments in modern platforms and programming languages; </del>  Selection between alternate network paths, using additional information about the networks over which a connection can operate

Reconsider words like "atop" and "underneath" when talking about interface boundaries. (Also, I don't know what it means to be atop an architecture.)
There are several places where the language in this document, once published as an RFC, will not age well. Search for occurrences of the word "modern" and consider writing less to motivate and more to define. Reconsider the use of terms like "traditional".
In 3.1 consider "It is important for applications using a Transport Services system to be robust to the" instead of using REQUIRED. At 3.2, if this were a requirements document for the Transport Services system, the SHOULD in the first paragraph might make sense. But if its an architecture document, or better, an overview and introduction. It would be better to simply say "is" rather than SHOULD. (And in a requirements document, I would argue that this SHOULD should be a MUST).
I think 3.1 probably needs a little more unpacking, this is a little abstarct in what is needed to say required: "Applications using a Transport Services system interface are REQUIRED to be robust to the automated selection provided by the system, where the automated selection is constrained by the requirements and preferences expressed by the application." We could therefore loose the /REQUIRED/. I'll suggest this as a starting point for 3.1 - the topic here is not an interoperability requirement of the protocol on the wire, it is about whether the method needs to have the feature to satisfy the design goals: "It is therefore is RECOMMENDED that a Transport Services system permits more specialized protocol features to be used."
this will be fixed with any reasonable fix to
Early art-art review of draft-ietf-taps-architecture My first observation and request is that the group rethink the separation of the architecture and interface documents. It is a warning-sign that the interface document is a normative reference for the architecture document. As the documents are currently constructed, a new reader cannot grasp the architecture without inferring some of it through reading the interface. Consider reformulating the architecture document as an overview - introduce principles, concepts, and terminology, but don't attempt to be normative. (There are signs that this may already be, or have been, a path the document was being pushed towards). If nothing else, rename it, and acknowledge that the actual architecture is specified using the interface definition. Note that the first sentence of the Overview in the interface document also highlights the current structural problem.
I don't entirely agree with this ART-ART comment: I think I might well agree with moving more of the description to this Arch document, but as discussed before, I also think it useful here to set the actual requirements for the architecture, and to define the architecture. It may need to be much clearer that this architecture consists of more the API.
For completeness, this is the first statement of the interface document review: This document reads fairly well. I have some structural concerns (see the early review of the architecture document). In particular I think this document contains the actual definition of the architecture and some of that is inferential.
Discussion at September 2021 interim: pass through this document with a view toward making it a high level introduction, not an architecture definition: "An Introduction to the Transport Services Architecture". remove normative language from this document (moving it to API, where necessary). move details not useful in a high level intro to API.
I disagree with this conclusion at first pass—we did a specific analysis and pass on this over a year ago, with PR . The normative language and the standards-track aspect was very intentional and something the WG had consensus on. The point is that the API is a specific incarnation of the TAPS architecture, but there are overarching architectural properties that go beyond a specific abstract API.
I completely agree with Tommy here, and recommend we fix this in . We should also of course check for any stray text or missing concepts. I think it would also be useful to somehow better explain that in figure 3 the API function is ONE part of the system.
See linked PR first please to see if this resolves this?
A PR was merged that addressed important text issues with separation of requirements and overview, and should more clearly set out that the requirements are for the system. The question of whether the overall TAPS systems requirements belong in this ID; the API; or a separate RFC needs some discussion based on the review, before this can be closed.
Many good text improvements in the PR NAME but I noticed two smaller updates in the PR that I think were incorrect. Fixed in
The review comment showed were many places where the wording of architecture can be improved. There are some high level principles that I think should remain in the architecture document. The decision in today's interim is to review the architecture to get this more into a shape that is valuable as a reference, and then check the updated draft.
I now suggest the issues are addressed and we close this issue without moving text between documents
I agree with that; closing.
Small fix for Gorry's comment suggested.

without requiring changes to the applications.  The specified API follows the Transport Services Architecture by providing asynchronous, atomic transmission of messages.  It is intended to  <ins> replace the BSD sockets API as the common interface to the transport layer, in an environment where endpoints could select from multiple interfaces and potential transport protocols. </ins> 1.  <ins> This document specifies an abstract application programming interface (API) that specifies the interface component of the high-level architecture for transport services defined in I-D.ietf-taps-arch. The Transport Services Architecture supports asynchronous, atomic transmission of messages over transport protocols and network paths dynamically selected at runtime, in environments where an endpoint selects from multiple interfaces and potential transport protocols. </ins> Applications that adopt this interface will benefit from a wide set of transport features that can evolve over time.  This protocol- This API derives specific path and protocol selection properties and supported transport features from the analysis provided in RFC8095,  <ins> RFC8923, and RFC8922.  The design encourages implementations that provide the interface to dynamically choose a transport protocol rather than statically binding applications to a protocol at compile time.  Nevertheless, the Transport Services API also provides applications with a way to override transport selection and instantiate a specific stack, e.g., to support servers wishing to listen to a specific protocol.  However, forcing a specific transport stack choice is discouraged for general use, because it can reduce portability. </ins> 1.1. Asynchronous Connection establishment, transmission, and reception.  This allows concurrent operations during establishment and event-driven application interactions with the transport  <ins> layer; </ins> Selection between alternate network paths, using additional information about the networks over which a connection can operate


asynchronous interface using sockets generally uses a try-and-fail model.  If the application wants to read, but data has not yet been received from the peer, the call to read will fail.  The application <del> then waits and can try again later. In contrast to sockets, all interaction using the Transport Services API is expected to be asynchronous, and use an event-driven model (see events).  For example, an application first completes a call to receive new data from the connection.  When delivered data becomes available, this data is delivered to the application using asynchronous events that contain the data.  Error handling is also asynchronous; a failure to send results in an asynchronous send error as an event. </del>  The Transport Services API also delivers events regarding the lifetime of a connection and changes in the available network links,

Be clear that the API is asynchronous (not the whole arch); and reword the read example to also be asynchronous.
In section 2.1 "its call to read will not complete immediately" is easy to misread. The call to read does in fact return immediately. Any read data is returned asynchronously. Please rework this sentence.

asynchronous interface using sockets generally uses a try-and-fail model.  If the application wants to read, but data has not yet been received from the peer, the call to read will fail.  The application  <ins> then waits and can try again later.  In contrast to sockets, all interaction using the Transport Services API is expected to be asynchronous and use an event-driven model (see events).  For example, an application first issues a call to receive new data from the connection.  When delivered data becomes available, this data is delivered to the application using asynchronous events that contain the data.  Error handling is also asynchronous; a failure to send data results in an asynchronous error event. </ins> The Transport Services API also delivers events regarding the lifetime of a connection and changes in the available network links,


concepts presents the Transport Services architecture diagram and defines the concepts that are used by both the API and implementation documents.  The Preconnection allows applications <del> to configure Connection Properties, and the Connection represents an object that can be used to send and receive Messages. </del>  <del> A Connection is an abstraction that represents the communication. If the transport services interface selects a protocol such as TCP for the communication, a Connection will correspond to an underlying protocol connection.  If however a protocol such as UDP is selected, the Connection remains an abstraction at the end points. </del>  1.3. Connection: A Connection object represents one or more active transport protocol instances that can send and/or receive Messages <del> between local and remote systems.  It holds state pertaining to the underlying transport protocol instances and any ongoing data transfers.  This represents, for example, an active Connection in a connection-oriented protocol such as TCP, or a fully-specified 5-tuple for a connectionless protocol such as UDP.  It can also represent a pool of transport protocol instances, e.g., a set of TCP and QUIC connections to equivalent endpoints, or a stream of a multi-streaming transport protocol instance.  Connections can be created from a Preconnection or by a Listener. </del>  Listener: A Listener object accepts incoming transport protocol connections from remote systems and generates corresponding

Trying to with a hint upfront that connections can be UDP, TCP or any other transport protocol instance... and then later explain about a connection that can remain an abstraction at the end points for a UDP example.
The last bullet in the Overview is out of place - the others are an overview of the document, but this one is a description of a concept.

concepts presents the Transport Services architecture diagram and defines the concepts that are used by both the API and implementation documents.  The Preconnection allows applications  <ins> to configure Connection Properties. </ins>  <ins> concepts also presents how an abstract Connection is used to select a transport protocol instance such as TCP, UDP, or another transport.  The Connection represents an object that can be used to send and receive Messages. </ins> 1.3. Connection: A Connection object represents one or more active transport protocol instances that can send and/or receive Messages  <ins> between local and remote systems.  It is an abstraction that represents the communication.  The Connection object holds state pertaining to the underlying transport protocol instances and any ongoing data transfers.  For example, an active Connection can represent a connection-oriented protocol such as TCP, or can represent a fully-specified 5-tuple for a connectionless protocol such as UDP, where the Connection remains an abstraction at the end points.  It can also represent a pool of transport protocol instances, e.g., a set of TCP and QUIC connections to equivalent endpoints, or a stream of a multi-streaming transport protocol instance.  Connections can be created from a Preconnection or by a Listener. </ins> Listener: A Listener object accepts incoming transport protocol connections from remote systems and generates corresponding


Any functionality that is common across multiple transport protocols SHOULD be made accessible through a unified set of Transport Services <del> API calls.  As a baseline, any Transport Services API MUST allow </del>  access to the minimal set of features offered by transport protocols RFC8923.

Quote from the API draft: "The present API covers all elements of this section except Notification of Excessive Retransmissions (early warning below abortion threshold)." So... shouldn't this MUST be a SHOULD?
Michael will merge this after finishing the PR that updates -interface minset appendix.

Any functionality that is common across multiple transport protocols SHOULD be made accessible through a unified set of Transport Services  <ins> API calls.  As a baseline, any Transport Services API SHOULD allow </ins> access to the minimal set of features offered by transport protocols RFC8923.


protocol features to applications for network communication, the Transport Services architecture.  The Transport Services Application Programming Interface (API) is based on an asynchronous, event-driven <del> interaction pattern.  This uses messages for representing data </del>  transfer to applications, and describes how implementations can use multiple IP addresses, multiple protocols, and multiple paths, and provide multiple application streams.  This document further defines how applications can use them effectively. The goal of the Transport Services architecture is to provide a <del> flexible, and reusable architecture that provides a common interface </del>  for transport protocols.  As applications adopt this interface, they will benefit from a wide set of transport features that can evolve over time, and ensure that the system providing the interface can failure, or some other metric.  Beyond these, if the API for the system remains the same over time, new protocols and features can be added to the system's implementation without requiring changes in <del> applications for adoption.  Similarly this can provide a common basis for utilising information about a a network path or interface, </del>  enabling evolution below the transport layer. The normative requirements described in this section allow Transport An application can specify constraints and preferences for the protocols, features, and network interfaces it will use via <del> Properties.  Properties are used by an applicatioon to declares its </del>  preferences for how the transport service should operate at each <del> stage of the lifetime of a connection.  Transport Properties are </del>  subdivided into Selection Properties, which specify which paths and protocol stacks can be used and are preferred by the application; Connection Properties, which inform decisions made during connection <del> establishment and to fine-tune the established connection and Message Properties, set on indvidual Messages. </del>  It is RECOMMENDED that the Transport Services API offers properties that are common to multiple transport protocols.  This enables the A Transport Services implementation can select Protocol Stacks based on the Selection and Connection Properties communicated by the <del> application and any security parameters.  If two different Protocol Stacks can be safely swapped, or raced in parallel (see racing), then they are considered to be "equivalent".  Equivalent Protocol Stacks are defined as stacks that can provide the same Transport Properties and interface expectations as requested by the application. </del>  The following two examples show non-equivalent Protocol Stacks: 4. <del> This Section and the remainder of this document describe the </del>  architecture non-normatively.  The concepts defined in this document are intended primarily for use in the documents and specifications that describe the Transport Services architecture and API.  While the

Nicer text throughout to introduce key things consistently; firm-up architecture v interface; introduce properties; resolve some NiTs.
Accepted adjustments from Tommy and merged.
I'll have to read the document yet another time, perhaps, but after the few times I went through it for this review, I'm still not clear on how/whether connections are grouped because of actions from the remote end(s). If so, how does the user learn about the grouping? Does it have to poll Connection.GroupedConnections() every time its told of a new Connection?
I think the answer is that an application may be unaware whether incoming Connections are grouped or not. To get the information, polling Connection.GroupedConnections() every time a new Connection arrives sounds just about right. But grouping can take various forms... the most common that I think is on everyone's mind is multi-streaming, where the arrival of a new stream is clear to the underlying stack. In this case, even when a stream is exposed as a Connection, the application can learn that it's really grouped together with others by polling as you say. Another alternative for grouping is to couple the congestion control of connections ([RFC 8699], or see [1] for how it can be done with TCP) - e.g., when they traverse the same tunnel, or when statistics give us a hint that there is a common bottleneck on the path [RFC 8382], [2]. In this case, the grouping is done one-sided. It's also possible that calling clone() really has no effect in the stack, because neither a multi-streaming capable transport nor such coupling mechanisms are available; the Connections are then still "grouped" in that changing a property of one is expected to affect all the others, but this is irrelevant to the peer when the Connections really are, e.g., separate TCP connections. All in all, I don't think a text change is needed. [1] Safiqul Islam, Michael Welzl, Kristian Hiorth, David Hayes, Grenville Armitage, Stein Gjessing: "ctrlTCP: Reducing Latency through Coupled, Heterogeneous Multi-Flow TCP Congestion Control", IEEE INFOCOM Global Internet Symposium (GI) workshop (GI 2018), Honolulu, HI, April 2018. [2] David Hayes, Michael Welzl, Simone Ferlin, David Ros, Safiqul Islam: "Online Identification of Groups of Flows Sharing a Network Bottleneck", IEEE/ACM Transactions on Networking 28(5), pp. 2229-2242, Print ISSN: 1063-6692, Online ISSN: 1558-2566 October 2020. DOI 10.1109/TNET.2020.3007346.
A Connection Group is only a set of Connections that share properties and caches at the sender. I don't think text is needed.
Where does the document state that a connection group is a concept only known to the sender?
Aha! I think that text may be needed. Thanks!
Yes and "aha" - I am happy to add that a connection group is a concept at the sender to arch, I add in a pending PR to address "cloning". I suggest MW does this for interface.
In 3.1 consider "It is important for applications using a Transport Services system to be robust to the" instead of using REQUIRED. At 3.2, if this were a requirements document for the Transport Services system, the SHOULD in the first paragraph might make sense. But if its an architecture document, or better, an overview and introduction. It would be better to simply say "is" rather than SHOULD. (And in a requirements document, I would argue that this SHOULD should be a MUST).
I think 3.1 probably needs a little more unpacking, this is a little abstarct in what is needed to say required: "Applications using a Transport Services system interface are REQUIRED to be robust to the automated selection provided by the system, where the automated selection is constrained by the requirements and preferences expressed by the application." We could therefore loose the /REQUIRED/. I'll suggest this as a starting point for 3.1 - the topic here is not an interoperability requirement of the protocol on the wire, it is about whether the method needs to have the feature to satisfy the design goals: "It is therefore is RECOMMENDED that a Transport Services system permits more specialized protocol features to be used."
this will be fixed with any reasonable fix to
Make it clearer in 3.1 that you are introducing a concept named Properties. A sentence that starts "Properties are" would really help.
Expected fixed in linked PR
The (arch) document currently says (just before section 3.1) "The rest of this document describes the architecture non-normatively", but then goes on to use a bunch of 2119/8174 language (much of which really shouldn't be - please try eliminating all of them).
this will be fixed with any reasonable fix to
See lined PR first, before discussing please.
Early art-art review of draft-ietf-taps-architecture My first observation and request is that the group rethink the separation of the architecture and interface documents. It is a warning-sign that the interface document is a normative reference for the architecture document. As the documents are currently constructed, a new reader cannot grasp the architecture without inferring some of it through reading the interface. Consider reformulating the architecture document as an overview - introduce principles, concepts, and terminology, but don't attempt to be normative. (There are signs that this may already be, or have been, a path the document was being pushed towards). If nothing else, rename it, and acknowledge that the actual architecture is specified using the interface definition. Note that the first sentence of the Overview in the interface document also highlights the current structural problem.
I don't entirely agree with this ART-ART comment: I think I might well agree with moving more of the description to this Arch document, but as discussed before, I also think it useful here to set the actual requirements for the architecture, and to define the architecture. It may need to be much clearer that this architecture consists of more the API.
For completeness, this is the first statement of the interface document review: This document reads fairly well. I have some structural concerns (see the early review of the architecture document). In particular I think this document contains the actual definition of the architecture and some of that is inferential.
Discussion at September 2021 interim: pass through this document with a view toward making it a high level introduction, not an architecture definition: "An Introduction to the Transport Services Architecture". remove normative language from this document (moving it to API, where necessary). move details not useful in a high level intro to API.
I disagree with this conclusion at first pass—we did a specific analysis and pass on this over a year ago, with PR . The normative language and the standards-track aspect was very intentional and something the WG had consensus on. The point is that the API is a specific incarnation of the TAPS architecture, but there are overarching architectural properties that go beyond a specific abstract API.
I completely agree with Tommy here, and recommend we fix this in . We should also of course check for any stray text or missing concepts. I think it would also be useful to somehow better explain that in figure 3 the API function is ONE part of the system.
See linked PR first please to see if this resolves this?
A PR was merged that addressed important text issues with separation of requirements and overview, and should more clearly set out that the requirements are for the system. The question of whether the overall TAPS systems requirements belong in this ID; the API; or a separate RFC needs some discussion based on the review, before this can be closed.
Many good text improvements in the PR NAME but I noticed two smaller updates in the PR that I think were incorrect. Fixed in
The review comment showed were many places where the wording of architecture can be improved. There are some high level principles that I think should remain in the architecture document. The decision in today's interim is to review the architecture to get this more into a shape that is valuable as a reference, and then check the updated draft.
I now suggest the issues are addressed and we close this issue without moving text between documents
I agree with that; closing.

protocol features to applications for network communication, the Transport Services architecture.  The Transport Services Application Programming Interface (API) is based on an asynchronous, event-driven  <ins> interaction pattern.  This API uses messages for representing data </ins> transfer to applications, and describes how implementations can use multiple IP addresses, multiple protocols, and multiple paths, and provide multiple application streams.  This document further defines how applications can use them effectively. The goal of the Transport Services architecture is to provide a  <ins> flexible and reusable architecture that provides a common interface </ins> for transport protocols.  As applications adopt this interface, they will benefit from a wide set of transport features that can evolve over time, and ensure that the system providing the interface can failure, or some other metric.  Beyond these, if the API for the system remains the same over time, new protocols and features can be added to the system's implementation without requiring changes in  <ins> applications for adoption.  Similarly, this can provide a common basis for utilizing information about a network path or interface, </ins> enabling evolution below the transport layer. The normative requirements described in this section allow Transport An application can specify constraints and preferences for the protocols, features, and network interfaces it will use via  <ins> Properties.  Properties are used by an application to declare its </ins> preferences for how the transport service should operate at each  <ins> stage in the lifetime of a connection.  Transport Properties are </ins> subdivided into Selection Properties, which specify which paths and protocol stacks can be used and are preferred by the application; Connection Properties, which inform decisions made during connection  <ins> establishment and fine-tune the established connection; and Message Properties, set on individual Messages. </ins> It is RECOMMENDED that the Transport Services API offers properties that are common to multiple transport protocols.  This enables the A Transport Services implementation can select Protocol Stacks based on the Selection and Connection Properties communicated by the  <ins> application, along with any security parameters.  If two different Protocol Stacks can be safely swapped, or raced in parallel (see racing), then they are considered to be "equivalent".  Equivalent Protocol Stacks are defined as stacks that can provide the same Transport Properties and interface expectations as requested by the application. </ins> The following two examples show non-equivalent Protocol Stacks: 4.  <ins> This section and the remainder of this document describe the </ins> architecture non-normatively.  The concepts defined in this document are intended primarily for use in the documents and specifications that describe the Transport Services architecture and API.  While the


4.1.3. <del> Selection: The Selection Properties consist of the properties that an application can set to influence the selection of paths between the local and remote systems, to influence the selection of transport protocols, or to configure the behavior of generic transport protocol features.  These properties can take the form of requirements, prohibitions, or preferences.  Examples of properties that influence path selection include the interface </del>  type (such as a Wi-Fi connection, or a Cellular LTE connection), requirements around the largest Message that can be sent, or preferences for throughput and latency.  Examples of properties 4.1.8. <del> A Connection Group is a set of Connections at the sender that shares properties and caches.  For multiplexing transport protocols, only Connections within the same Connection Group are allowed to be multiplexed together. </del>  When the API clones an existing Connection, this adds a new Connection to the Connection Group.  A change to one of the

Noted two small edits in Gorry's update that I think were incorrect. Properties erroneously removed in "Selection Properties" Connection groups do not only exist at the Sender.
Early art-art review of draft-ietf-taps-architecture My first observation and request is that the group rethink the separation of the architecture and interface documents. It is a warning-sign that the interface document is a normative reference for the architecture document. As the documents are currently constructed, a new reader cannot grasp the architecture without inferring some of it through reading the interface. Consider reformulating the architecture document as an overview - introduce principles, concepts, and terminology, but don't attempt to be normative. (There are signs that this may already be, or have been, a path the document was being pushed towards). If nothing else, rename it, and acknowledge that the actual architecture is specified using the interface definition. Note that the first sentence of the Overview in the interface document also highlights the current structural problem.
I don't entirely agree with this ART-ART comment: I think I might well agree with moving more of the description to this Arch document, but as discussed before, I also think it useful here to set the actual requirements for the architecture, and to define the architecture. It may need to be much clearer that this architecture consists of more the API.
For completeness, this is the first statement of the interface document review: This document reads fairly well. I have some structural concerns (see the early review of the architecture document). In particular I think this document contains the actual definition of the architecture and some of that is inferential.
Discussion at September 2021 interim: pass through this document with a view toward making it a high level introduction, not an architecture definition: "An Introduction to the Transport Services Architecture". remove normative language from this document (moving it to API, where necessary). move details not useful in a high level intro to API.
I disagree with this conclusion at first pass—we did a specific analysis and pass on this over a year ago, with PR . The normative language and the standards-track aspect was very intentional and something the WG had consensus on. The point is that the API is a specific incarnation of the TAPS architecture, but there are overarching architectural properties that go beyond a specific abstract API.
I completely agree with Tommy here, and recommend we fix this in . We should also of course check for any stray text or missing concepts. I think it would also be useful to somehow better explain that in figure 3 the API function is ONE part of the system.
See linked PR first please to see if this resolves this?
A PR was merged that addressed important text issues with separation of requirements and overview, and should more clearly set out that the requirements are for the system. The question of whether the overall TAPS systems requirements belong in this ID; the API; or a separate RFC needs some discussion based on the review, before this can be closed.
Many good text improvements in the PR NAME but I noticed two smaller updates in the PR that I think were incorrect. Fixed in
The review comment showed were many places where the wording of architecture can be improved. There are some high level principles that I think should remain in the architecture document. The decision in today's interim is to review the architecture to get this more into a shape that is valuable as a reference, and then check the updated draft.
I now suggest the issues are addressed and we close this issue without moving text between documents
I agree with that; closing.

4.1.3.  <ins> Selection Properties: The Selection Properties consist of the properties that an application can set to influence the selection of paths between the local and remote systems, to influence the selection of transport protocols, or to configure the behavior of generic transport protocol features.  These properties can take the form of requirements, prohibitions, or preferences.  Examples of properties that influence path selection include the interface </ins> type (such as a Wi-Fi connection, or a Cellular LTE connection), requirements around the largest Message that can be sent, or preferences for throughput and latency.  Examples of properties 4.1.8.  <ins> A Connection Group is a set of Connections that shares properties and caches.  A Group represents state for managing Connections within a single application, and does not require end-to-end protocol signaling.  For multiplexing transport protocols, only Connections within the same Connection Group are allowed to be multiplexed together. </ins> When the API clones an existing Connection, this adds a new Connection to the Connection Group.  A change to one of the


for incoming Connections, and to which it will attempt to establish connections.  An application that uses Rendezvous() to establish a peer-to-peer connection in the presence of NATs will configure the Preconnection object with at least one a Local Endpoint that supports NAT binding

It feels to me like there's a bit of a gap between how software ought to discover Endpoint Object identifiers for Remote Endpoints versus Local Endpoints. Remote Endpoint identifiers are likely to be defined by the kind of software being used acting on some user input (web browser requesting a page, DNS server recursing, etc.), and these identifiers are resolved through existing internet architecture. Importantly, there's not really any way to check a valid input would work until for a Remote Endpoint. This isn't really true for identifiers on Local Endpoints. While it's true that discovering these things is system-specific, a TAPS API implementation ought to provide a vehicle for discovering what inputs are valid. Two reasons I think it's reasonable to support discovery of local addresses and interfaces include: It'd be unfortunate for TAPS-consuming software to rely on system-specific APIs to implement a UX for configuring that software. Given diversity of public cloud offerings and wide use of multi-cloud deployments, it's reasonable to expect folks to want to dynamically discover what local network addresses and interfaces exist in their environment. If the intent is to not define an API to discover these identifiers, the document should be clarified to discuss what Errors can be returned by and Actions on Local Endpoints. (This is coming from a perspective that "error early, error often" is a best practice, which is admittedly subjective.) If there is agreement to add (for example) and Actions on a Local Endpoint, these should be documented. Further, the and Actions on a Local Endpoint should be described to return an Error when the supplied input would not exist in the results returned by the corresponding call at the time the Action was called. In either case, it would be useful to mention that a successful invocation of a Action is not a guarantee that initiating the precondition will succeed.
This all makes sense to me, and sounds like some things should indeed be added. I'm a little afraid that adding "GetInterface" could cause a rat tail of "GetWhateverElse" ideas... is there a good way to keep that from happening? Something more generic, perhaps - "GetLocalSystemProperties", and explain that such properties could, e.g., contain a list of interfaces? Regarding early failure, I agree with your suggestions too; note that, even though a Preconnection itself is just a passive data structure (or Object holding data), the call to create it can already fail (some details are in section 3.1 of the implementation draft). I think that's the earliest type of failure that we specify (and since this is an abstract API, I'm not sure if we should start specifying that calls to define an Endpoint (which is where "WithInterface" and such are found) could fail too - the endpoints could just be a 100% passive data structure that's handed over, there might not be any calls involved... so I think it's enough to specify failure of the Preconnection creation).
I think that's a fair concern, and I think a API could work. Two concerns with a single API entry point for Local Endpoint identifier discovery: It provides no means for enumerating interface(s) by address(es) or vice versa without some argument signifying how the caller wants the data returned. It's more likely to cause future API and ABI versioning headaches for implementers when new identifiers are deemed useful for discovery, and require changing the returned data structure. I'm not sure how to effectively allay concerns about adding a billion discovery APIs. I don't think discovery is needed for Transport Properties: I see that as a TAPS API implementer's documentation problem (and consumer's reading-the-documentation problem), and maybe that's worth a sentence of rationale somewhere. I also do not think that a service port discovery API is particularly useful in modern systems (and there are many service discovery protocols to choose from for the cases they are). would then have to reference a yet-to-be-defined local transport concept; since a Local Endpoint is effectively defined by a 3-tuple of interface, address, and port, I think it's unlikely new discovery APIs would be requested / added frequently. Great point! If the system changes between discovery and such that a previously-valid Preconnection is now invalid, the inverse is also possible. Happy to scope this issue only to the point of discovery APIs.
Rendezvous lets you enumerate local endpoint candidates via Preconnection.Resolve(). In general, you shouldn't be enumerating the local endpoints for client-to-server connections, as it will limit the ability to do happy eyeballs, etc.
We can add clarity in the existing rendezvous section to explain that this list could be used for other purposes, but people should be careful.
I'm considering cases like proxies, where interface choice might be important for routing / peering / cost decisions. I agree it's generally not something one would want to do for clients-with-people-behind-them. Thanks for pointing out this is possible with Rendezvous!
I'm trying to write some text to resolve this and as it turns out I'm a bit confused by what Preconnection.Resolve() will return: the set of local and remote endpoints supplied to the preconnection to attempt to rendezvous using; or the set of all potential local endpoints (if none has been specified).

for incoming Connections, and to which it will attempt to establish connections. <ins> Note that the set of LocalEndpoints returned by Resolve() might or might not contain information about all possible local interfaces; it is valid only for a Rendezvous happening at the same time as the resolution.  Care should be taken in using these values in any other context. </ins> An application that uses Rendezvous() to establish a peer-to-peer connection in the presence of NATs will configure the Preconnection object with at least one a Local Endpoint that supports NAT binding


changes to the application. requirements explains the fundamental requirements for a Transport <del> Services API.  These principles are intended to make sure that </del>  transport protocols can continue to be enhanced and evolve without requiring significant changes by application developers. <del> concepts presents the Transport Services architecture diagram and defines the concepts that are used by both the API I-D.ietf-taps- interface and implementation guidelines I-D.ietf-taps-impl.  The Preconnection allows applications to configure Connection Properties. </del>  concepts also presents how an abstract Connection is used to select a transport protocol instance such as TCP, UDP, or another transport.  The Connection represents an object that can be used <del> to send and receive Messages. </del>  1.3. the implementation below the API. The Transport Services API I-D.ietf-taps-interface defines the <del> interface for an application to create network connections and transfer data.  It combines interfaces for multiple interaction patterns into a unified whole.  By combining name resolution with connection establishment and data transfer in a single API, it allows for more flexible implementations to provide path and transport protocol agility on the application's behalf. </del>  The Transport Services implementation I-D.ietf-taps-impl implements the transport layer protocols and other functions needed to send and 2.2. The Socket API provides a message interface for datagram protocols <del> like UDP, but provide an unstructured stream abstraction for TCP. While TCP does indeed provide the ability to send and receive data as streams, most applications need to interpret structure within these streams.  For example, HTTP/1.1 uses character delimiters to segment messages over a stream RFC7230; TLS record headers carry a version, content type, and length RFC8446; and HTTP/2 uses frames to segment its headers and bodies RFC7540. </del>  The Transport Services API represents data as messages, so that it more closely matches the way applications use the network.  Providing the ability to associate deadlines with messages, for applications that care about timing; <del> the ability to provide control of reliability, choosing which messages to retransmit when there is packet loss, and how best to make use of the data that arrived; the ability to manage dependencies between messages, when a Transport Services implementation could decide to not deliver a message, either following packet loss or because it has missed a deadline.  In particular, this can avoid (re-)sending data that relies on a previous transmission that was never received. </del>  the ability to automatically assign messages and connections to underlying transport connections to utilize multi-streaming and inform how transport protocols use a network path or interface. There are several ways the Transport Services system can offer <del> flexibility to an application: it can provide access to multiple sets of protocols and protocol features; it can use these protocols across </del>  multiple paths that could have different performance and functional characteristics; and it can communicate with different remote systems to optimize performance, robustness to failure, or some other metric. not. A Transport Services system makes decisions that select protocols and <del> interfaces.  In normal use, a Transport Services system SHOULD result in consistent protocol and interface selection decisions for the same network conditions given the same set of Properties.  This is intended to provide predictable outcomes to the application using the API. </del>  4. information about the received Message, such as metadata generated at the receiver and information signalled by the Remote Endpoint. For example, a Message can be marked with a Message Property <del> indicating that it is the final message on a connection if the peer sent a TCP FIN. </del>  Send: The action to transmit a Message over a Connection to the Remote Endpoint.  The interface to Send can accept Message phase (racing). Candidate Protocol Stack: One Protocol Stack that can be used by <del> an application for a connection, of which there can be several. Candidate Protocol Stacks are identified during the gathering phase (gathering) and are started during the racing phase (racing). </del>  System Policy: Represents the input from an operating system or other global preferences that can constrain or influence how an 4.2.2. Connection establishment attempts for a set of candidates may be <del> performed simultaneously, synchronously, serially, or some </del>  combination of all of these.  We refer to this process as racing, borrowing terminology from Happy Eyeballs RFC8305. protocol state, cached path state, and heuristics, may be shared (e.g. across multiple connections in an application).  This provides efficiency and convenience for the application, since the Transport <del> Services implementation can automatically optimize behavior. </del>  There are several reasons, however, that an application might want to explicitly isolate some Connections.  These reasons include:

Thought to address all listed (and related) issues
These are bunched together, and likely could be addressed by one PR: Section 2, right below Figure 2: "The Transport Services API [I-D.ietf-taps-interface] defines the interface for an application to create network connections and transfer data." => Should this be "Connections" instead of "network connections" ? Section 2.1, paragraph 2, last line: "... made explicit in sockets." seems strange. Should this be "in the sockets API." ? Section 2.2 paragraph 1, this sentence: "While TCP does indeed provide the ability to send and receive data as streams, most applications need to interpret structure within these streams." Two issues here. First, I would switch to singular because TCP is not a multi-streaming protocol. I.e., change this to: "While TCP does indeed provide the ability to send and receive data as a stream, most applications need to interpret structure within this stream." Second, is "stream" a known concept? I don't think it's defined in this document, and I'm a little unsure because, even when using singular, someone might read "stream" and think of "one stream out of several, in a multi-streaming protocol". Maybe this is nit-picking, but if you agree, I would prefer to replace "stream" with "data stream", or having a definition of "stream" and capitalizing it (but then the definition would have to appear before the first use of this term...) GF: Data Stream Seems good. Next paragraph: double "provide" in: "Providing a message-based abstraction provides many benefits, such as:" - I suggest to replace the second with "has". In the following item list, I would remove the third item: "the ability to manage dependencies between messages, ..." While it is true that a message-based abstraction enables this, a reader of this document might get the impression that the TAPS system defined in the interface draft offers this specific functionality. It really doesn't, it would still have to be implemented on top of it. For this reason, this item strikes me as a bit misleading, and I think it should be removed. Section 3, paragraph 2: "it can provide access to multiple sets of protocols and protocol features" - remove "sets of", this adds an unnecessary level of indirection here. I guess the transport property profiles in interface, section B.2 would be such sets, but still... Last paragraph before section 3.2: "Applications using a Transport Services system interface are REQUIRED to be robust to the automated selection provided by the system." - why say "system interface" here all of a sudden? This gives an impression as if there were two things: the API, and then perhaps also a separate "system interface". I'd rather replace this with "API". Last paragraph of section 3.4: "In normal use, a Transport Services system SHOULD result in consistent protocol and interface selection decisions for the same network conditions given the same set of Properties." Isn't this too restrictive? It prohibits a Transport Services system from improving. I would change this to: "In normal use, a given version of a Transport Services system SHOULD result in consistent protocol and interface selection decisions for the same network conditions given the same set of Properties." Section 4.1.5: "For example, a Message can be marked with a Message Property indicating that it is the final message on a connection if the peer sent a TCP FIN." I would stop this sentence after "on a connection." - or is it just me misunderstanding something? I thought (and -interface section 9.1.3.5 seems to agree with me) that the idea of "final message for TCP" really is that setting the property means that THIS message that I'm sending gets a FIN flag. Why would a FIN from the peer make me set this property? FWIW I could go on sending more data, and use this property much later? Section 4.1.7, item 1: "end of stream in the ..." - see above for a discussion of the word "stream". Section 4.2, item 5: "One Protocol Stack that can be used by an application for a connection, of which there can be several." => I believe it should be "Connection" here, not "connection". Section 4.2.2, first sentence: "or some combination of all of these" - should this be "or using some combination of all of these" ? This is mentioned in another Issue: Section 1.2, item 4: About the last sentence in item 4, see: URL One other review comment might need discussion prior to resolving... and will be in the next issue

changes to the application. requirements explains the fundamental requirements for a Transport  <ins> Services system.  These principles are intended to make sure that </ins> transport protocols can continue to be enhanced and evolve without requiring significant changes by application developers.  <ins> concepts presents a diagram showing the Transport Services architecture and defines the concepts that are used by both the API I-D.ietf-taps-interface and implementation guidelines I- D.ietf-taps-impl.  The Preconnection allows applications to configure Connection Properties. </ins> concepts also presents how an abstract Connection is used to select a transport protocol instance such as TCP, UDP, or another transport.  The Connection represents an object that can be used  <ins> to send and receive messages. </ins> 1.3. the implementation below the API. The Transport Services API I-D.ietf-taps-interface defines the  <ins> interface for an application to create Connections and transfer data. It combines interfaces for multiple interaction patterns into a unified whole.  By combining name resolution with connection establishment and data transfer in a single API, it allows for more flexible implementations to provide path and transport protocol agility on the application's behalf. </ins> The Transport Services implementation I-D.ietf-taps-impl implements the transport layer protocols and other functions needed to send and 2.2. The Socket API provides a message interface for datagram protocols  <ins> like UDP, but provides an unstructured stream abstraction for TCP. While TCP has the ability to send and receive data as a byte-stream, most applications need to interpret structure within this byte- stream.  For example, HTTP/1.1 uses character delimiters to segment messages over a byte-stream RFC7230; TLS record headers carry a version, content type, and length RFC8446; and HTTP/2 uses frames to segment its headers and bodies RFC7540. </ins> The Transport Services API represents data as messages, so that it more closely matches the way applications use the network.  Providing the ability to associate deadlines with messages, for applications that care about timing;  <ins> the ability control reliability, which messages to retransmit when there is packet loss, and how best to make use of the data that arrived; </ins> the ability to automatically assign messages and connections to underlying transport connections to utilize multi-streaming and inform how transport protocols use a network path or interface. There are several ways the Transport Services system can offer  <ins> flexibility to an application: it can provide access to transport protocols and protocol features; it can use these protocols across </ins> multiple paths that could have different performance and functional characteristics; and it can communicate with different remote systems to optimize performance, robustness to failure, or some other metric. not. A Transport Services system makes decisions that select protocols and  <ins> interfaces.  In normal use, a given version of a Transport Services system SHOULD result in consistent protocol and interface selection decisions for the same network conditions given the same set of Properties.  This is intended to provide predictable outcomes to the application using the API. </ins> 4. information about the received Message, such as metadata generated at the receiver and information signalled by the Remote Endpoint. For example, a Message can be marked with a Message Property  <ins> indicating that it is the final message on a connection. </ins> Send: The action to transmit a Message over a Connection to the Remote Endpoint.  The interface to Send can accept Message phase (racing). Candidate Protocol Stack: One Protocol Stack that can be used by  <ins> an application for a Connection, which there can be several candidates.  Candidate Protocol Stacks are identified during the gathering phase (gathering) and are started during the racing phase (racing). </ins> System Policy: Represents the input from an operating system or other global preferences that can constrain or influence how an 4.2.2. Connection establishment attempts for a set of candidates may be  <ins> performed simultaneously, synchronously, serially, or using some </ins> combination of all of these.  We refer to this process as racing, borrowing terminology from Happy Eyeballs RFC8305. protocol state, cached path state, and heuristics, may be shared (e.g. across multiple connections in an application).  This provides efficiency and convenience for the application, since the Transport  <ins> Services system can automatically optimize behavior. </ins> There are several reasons, however, that an application might want to explicitly isolate some Connections.  These reasons include:


Specific Properties can significantly reduce the flexibility of a transport services implementation to make appropriate selection and configuration choices.  Therefore, it is RECOMMENDED that Protocol <del> Properties are normnally used for properties common across different protocols and that Protocol Specific Properties are only used where specific protocols or properties are necessary. </del>  The application can set and query Connection Properties on a per- Connection basis.  Connection Properties that are not read-only can

Be firm, but avoid MUST NOT apply across different protocols TO ADDRESS
This was discussed at the Interim and final text approved by MW.
In section 8's third paragraph, the MUST NOT is not a good use of 2119/8174. Are you trying to say protocol specific property names must not be reused across different protocols?
I note that the current text was: "Protocol Specific Properties are defined in a transport- and implementation-specific way, and MUST NOT apply across different protocols. Too much reliance by an application on Protocol Specific Properties can significantly reduce the flexibility of a transport services implementation."
Decision: avoid this language, say "architecture expects that ... "
Say this is important but only recommend.

Specific Properties can significantly reduce the flexibility of a transport services implementation to make appropriate selection and configuration choices.  Therefore, it is RECOMMENDED that Protocol  <ins> Properties are used for properties common across different protocols and that Protocol Specific Properties are only used where specific protocols or properties are necessary. </ins> The application can set and query Connection Properties on a per- Connection basis.  Connection Properties that are not read-only can


match all Protocol Selection Properties that the application specified on the Preconnection. <del> For Connections that are Established: Information concerning the </del>  path(s) used by the Protocol Stack.  This can be derived from local PVD information, measurements by the Protocol Stack, or other sources.  For example, a TAPS system that is configured to

Rewording the path information with respect to PVD to reflect Info concerning the path(s) used by the Protocol Stack.
Addressed MW comments in this PR
At the last bullet before section 8.1, are you RFC7556 being pointed to as an example way one might derive properties? If so, adjust the language to make it clear that its an example.
I was thinking this was a useful example: a system that is configured to receive and process MPVD information, as described in RFC7556, can utilise this to inform the system policy that is later used by a transport system implementation, e.g. to provide additional information about paths when performing candidate gathering.
Gorry will write a PR for review.

match all Protocol Selection Properties that the application specified on the Preconnection.  <ins> For Connections that are Established: information concerning the </ins> path(s) used by the Protocol Stack.  This can be derived from local PVD information, measurements by the Protocol Stack, or other sources.  For example, a TAPS system that is configured to


6.2.7. <del> FullChecksumSend </del>  Preference 6.2.8. <del> FullChecksumRecv </del>  Preference

As discussed at the last interim, but cleaning up the references to use the real property names too.
Yes, because it's just a quote of the minset RFC's name :) can't change that anymore
Works for me. I'm not hugely sold on "end systems" given my recent experience with the word "system" in arch - but it likely is quite OK here.

6.2.7.  <ins> fullChecksumSend </ins> Preference 6.2.8.  <ins> fullChecksumRecv </ins> Preference


according to the Connection Group Transmission Scheduler property (conn-scheduler).  See priority-in-taps for more.  8. During pre-establishment and after establishment, connections can be This event notifies the application when at least one of the paths underlying a Connection has changed.  Changes occur on a single path when the PMTU changes as well as when multiple paths are used and <del> paths are added or removed, or a handover has been performed. </del>  9.

This makes the following changes: Add , , , and actions to existing connections Update the description of the event to note that it's generated if the set of local endpoints available changes Update the peer-to-peer example to show how these can be used to support Trickle ICE
For multipath transport protocols, the transport can automatically manage the set of local and remote endpoints its using, and the application doesn't need to be aware of this. For non-multipath transport protocols used peer-to-peer, the peers need to be aware of changes to the set of local and remote endpoints, so they can trigger updates to the NAT binding discovery. This can't be done automatically, since to implement Trickle ICE, it's necessary to pass the updated set of candidates to the peer via an out-of-band signalling channel, before passing them in to the connection to probe connectivity. Accordingly, we need: A way to add/remove remote endpoints on an existing connection A way to add/remove local endpoints on an existing connection A way to detect when the set of local endpoints has changed To implement Trickle ICE, adding and removing remote endpoints has to be an explicit action, since changes to the set of remote endpoints are communicated to the peer via out-of-band signalling. Similarly, when implementing Trickle ICE, changes to the set of local endpoints have to be communicated to the peer via out-of-band signalling, so the application has to be notified about them. It could be possible to automatically add/remove local endpoints to an existing connection, if the Preconnection left the local endpoints unspecified. But if the set of local endpoints was specified, the application will need to make a choice which ones it adds/removes when the set changes.
I would really like to see an example of how an application that relies on ICE learns about new candidates as they are discovered - the callback pattern is not clear to me yet.
See also
It feels to me like there's a bit of a gap between how software ought to discover Endpoint Object identifiers for Remote Endpoints versus Local Endpoints. Remote Endpoint identifiers are likely to be defined by the kind of software being used acting on some user input (web browser requesting a page, DNS server recursing, etc.), and these identifiers are resolved through existing internet architecture. Importantly, there's not really any way to check a valid input would work until for a Remote Endpoint. This isn't really true for identifiers on Local Endpoints. While it's true that discovering these things is system-specific, a TAPS API implementation ought to provide a vehicle for discovering what inputs are valid. Two reasons I think it's reasonable to support discovery of local addresses and interfaces include: It'd be unfortunate for TAPS-consuming software to rely on system-specific APIs to implement a UX for configuring that software. Given diversity of public cloud offerings and wide use of multi-cloud deployments, it's reasonable to expect folks to want to dynamically discover what local network addresses and interfaces exist in their environment. If the intent is to not define an API to discover these identifiers, the document should be clarified to discuss what Errors can be returned by and Actions on Local Endpoints. (This is coming from a perspective that "error early, error often" is a best practice, which is admittedly subjective.) If there is agreement to add (for example) and Actions on a Local Endpoint, these should be documented. Further, the and Actions on a Local Endpoint should be described to return an Error when the supplied input would not exist in the results returned by the corresponding call at the time the Action was called. In either case, it would be useful to mention that a successful invocation of a Action is not a guarantee that initiating the precondition will succeed.
This all makes sense to me, and sounds like some things should indeed be added. I'm a little afraid that adding "GetInterface" could cause a rat tail of "GetWhateverElse" ideas... is there a good way to keep that from happening? Something more generic, perhaps - "GetLocalSystemProperties", and explain that such properties could, e.g., contain a list of interfaces? Regarding early failure, I agree with your suggestions too; note that, even though a Preconnection itself is just a passive data structure (or Object holding data), the call to create it can already fail (some details are in section 3.1 of the implementation draft). I think that's the earliest type of failure that we specify (and since this is an abstract API, I'm not sure if we should start specifying that calls to define an Endpoint (which is where "WithInterface" and such are found) could fail too - the endpoints could just be a 100% passive data structure that's handed over, there might not be any calls involved... so I think it's enough to specify failure of the Preconnection creation).
I think that's a fair concern, and I think a API could work. Two concerns with a single API entry point for Local Endpoint identifier discovery: It provides no means for enumerating interface(s) by address(es) or vice versa without some argument signifying how the caller wants the data returned. It's more likely to cause future API and ABI versioning headaches for implementers when new identifiers are deemed useful for discovery, and require changing the returned data structure. I'm not sure how to effectively allay concerns about adding a billion discovery APIs. I don't think discovery is needed for Transport Properties: I see that as a TAPS API implementer's documentation problem (and consumer's reading-the-documentation problem), and maybe that's worth a sentence of rationale somewhere. I also do not think that a service port discovery API is particularly useful in modern systems (and there are many service discovery protocols to choose from for the cases they are). would then have to reference a yet-to-be-defined local transport concept; since a Local Endpoint is effectively defined by a 3-tuple of interface, address, and port, I think it's unlikely new discovery APIs would be requested / added frequently. Great point! If the system changes between discovery and such that a previously-valid Preconnection is now invalid, the inverse is also possible. Happy to scope this issue only to the point of discovery APIs.
Rendezvous lets you enumerate local endpoint candidates via Preconnection.Resolve(). In general, you shouldn't be enumerating the local endpoints for client-to-server connections, as it will limit the ability to do happy eyeballs, etc.
We can add clarity in the existing rendezvous section to explain that this list could be used for other purposes, but people should be careful.
I'm considering cases like proxies, where interface choice might be important for routing / peering / cost decisions. I agree it's generally not something one would want to do for clients-with-people-behind-them. Thanks for pointing out this is possible with Rendezvous!
I'm trying to write some text to resolve this and as it turns out I'm a bit confused by what Preconnection.Resolve() will return: the set of local and remote endpoints supplied to the preconnection to attempt to rendezvous using; or the set of all potential local endpoints (if none has been specified).
LGTM, thanks a lot for doing this!

according to the Connection Group Transmission Scheduler property (conn-scheduler).  See priority-in-taps for more. <ins> 7.5. Transport protocols that are explicitly multipath aware are expected to automatically manage the set of Remote Endpoints that they are communicating with, and the paths to those endpoints.  A "PathChange<>" event, described in conn-path-change, will be generated when the path changes. In some cases, however, it is necessary to explicitly indicate to a Connection that a new remote endpoint has become available for use, or to indicate that some remote endpoint is no longer available. This is most common in the case of peer to peer connections using Trickle ICE RFC8838. The "AddRemote()" action can be used to add one or more new remote endpoints to a Connection: Endpoints that are already known to the Connection are ignored.  A call to "AddRemote()" makes the new remote endpoints available to the connection, but whether the Connection makes use of those endpoints will depend on the underlying transport protocol. Similarly, the "RemoveRemote()" action can be used to tell a connection to stop using one or more remote endpoints: Removing all known remote endpoints can have the effect of aborting the connection.  The effect of removing the active remote endpoint(s) depends on the underlying transport: multipath aware transports might be able to switch to a new path if other reachable remote endpoints exist, or the connection might abort. Similarly, the "AddLocal()" and "RemoveLocal()" actions can be used to add and remove local endpoints to/from a Connection. </ins> 8. During pre-establishment and after establishment, connections can be This event notifies the application when at least one of the paths underlying a Connection has changed.  Changes occur on a single path when the PMTU changes as well as when multiple paths are used and  <ins> paths are added or removed, the set of local endpoints changes, or a handover has been performed. </ins> 9.


efficiency and convenience for the application, since the Transport Services system can automatically optimize behavior. <del> There are several reasons, however, that an application might want to explicitly isolate some Connections.  These reasons include: </del>  Privacy concerns about re-using cached protocol state that can <del> lead to linkability.  Sensitive state may include TLS session state RFC8446 and HTTP cookies RFC6265. </del>  Privacy concerns about allowing Connections to multiplex together, which can tell a Remote Endpoint that all of the Connections are <del> coming from the same application (for example, when Connections are multiplexed HTTP/2 or QUIC streams). Performance concerns about Connections introducing head-of-line blocking due to multiplexing or needing to share state on a single thread. The Transport Services API can allow applications to explicitly define Connection Contexts that force separation of Cached State and Protocol Stacks.  For example, a web browser application might use Connection Contexts with separate caches for different tabs in the browser to decrease linkability. </del>  5.

Addressed MW comments; and also tidied several successive similar examples into one para.
Section 4.2.3, item 3: "Performance concerns about Connections introducing head-of-line blocking due to multiplexing or needing to share state on a single thread." I don't think application programmers should decide against using functions such as Connection Grouping because they try to optimize their performance in this way. If it performs poorly, the Transport Services system shouldn't do it. I know this isn't always black and white... I'm guessing that this was written with HTTP/2 in mind. Still ... if a mechanism is judged as generally performing better most of the time, the system should probably use it. If it doesn't perform better e.g. depending on the length of transfers, which may be known from HTTP header, then maybe the Transport Services system is the ideal place to dynamically make such a decision? Anyway - I don't think that we should indirectly suggest to application writers that internal functions chosen by the Transport Services system on the basis of high-level information like "these Connections belong together" might in fact perform poorly, and hence they may be better off not using them if they want the best performance for their application. So, I would remove this item.
Sure, applications might do many things and some decisions to multiplex multiple flows using a single protocol instantiation might be less performance, however, I would suggest removing this bullet: " * Performance concerns about Connections introducing head-of-line blocking due to multiplexing or needing to share state on a single thread.", because I think it creates confusion as Michael suggests. A developer understanding complex tradeoffs (such as web client browsers or NFS/SMB servers) will consider this anyway, others should not be worried.

efficiency and convenience for the application, since the Transport Services system can automatically optimize behavior.  <ins> The Transport Services API can allow applications to explicitly define Connection Contexts that force separation of Cached State and Protocol Stacks.  For example, a web browser application could use Connection Contexts with separate caches when implementing different tabs.  Possible reasons to isolate Connections using separate Connection Contexts include: </ins> Privacy concerns about re-using cached protocol state that can  <ins> lead to linkability.  Sensitive state could include TLS session state RFC8446 and HTTP cookies RFC6265.  These concerns could be addressed using Connection Contexts with separate caches, such as for different browser tabs. </ins> Privacy concerns about allowing Connections to multiplex together, which can tell a Remote Endpoint that all of the Connections are  <ins> coming from the same application.  Using Connection Contexts avoids the Connections being multiplexed in a HTTP/2 or QUIC stream. </ins> 5.


Boolean: Instances take the value "true" or "false". <del> Integer: Instances take positive or negative numeric integer values, or sometimes special non-numeric (symbolic) values. </del>  <del> Numeric: Instances take positive or negative numeric values, or sometimes special non-numeric (symbolic) values. </del>  Enumeration: A family of types in which each instance takes one of a fixed, predefined set of values specific to a given enumerated recvChecksumLen <del> Integer (non-negative with special value "Full Coverage") </del>  <del> Full Coverage </del>  <del> This property specifies the minimum number of bytes in a received message that need to be covered by a checksum.  A receiving endpoint will not forward messages that have less coverage to the application. The application is responsible for handling any corruption within the non-protected part of the message RFC8085.  A special value of 0 means that a received packet may also have a zero checksum field. </del>  8.1.2. connTimeout <del> Numeric, with special value "Disabled" </del>  <del> Disabled </del>  <del> This property specifies how long to wait before deciding that an active Connection has failed when trying to reliably deliver data to the Remote Endpoint.  Adjusting this Property will only take effect when the underlying stack supports reliability.  The special value "Disabled" means that no timeout is scheduled. </del>  8.1.4. keepAliveTimeout <del> Numeric, with special value "Disabled" </del>  Implementation-defined A Transport Services API can request a protocol that supports sending <del> keep alive packets keep-alive.  This property specifies the maximum length of time an idle connection (one for which no transport packets have been sent) should wait before the Local Endpoint sends a keep- alive packet to the Remote Endpoint.  Adjusting this Property will only take effect when the underlying stack supports sending keep- alive packets.  Guidance on setting this value for datagram transports is provided in RFC8085.  A value greater than the connection timeout (conn-timeout), or the special value "Disabled", will disable the sending of keep-alive packets. </del>  8.1.5. minSendRate / minRecvRate / maxSendRate / maxRecvRate <del> Numeric (with special value "Unlimited") / Numeric (with special value "Unlimited") / Numeric (with special value "Unlimited") / Numeric (with special value "Unlimited") </del>  <del> Unlimited / Unlimited / Unlimited / Unlimited </del>  <del> This property specifies an upper-bound rate that a transfer is not expected to exceed (even if flow control and congestion control allow higher rates), and/or a lower-bound rate below which the application does not deem it will be useful.  These are specified in bits per second.  The special value "Unlimited" indicates that no bound is specified. </del>  8.1.9. groupConnLimit <del> Numeric (with special value "Unlimited") </del>  <del> Unlimited </del>  <del> This property controls the number of Connections that can be accepted from a peer as new members of the Connection's group.  Similar to SetNewConnectionLimit(), this limits the number of ConnectionReceived Events that will occur, but constrained to the group of the Connection associated with this property.  For a multi-streaming transport, this limits the number of allowed streams. </del>  8.1.10. msgChecksumLen <del> Integer (non-negative with special value "Full Coverage") </del>  <del> Full Coverage </del>  <del> This property specifies the minimum length of the section of a sent Message, starting from byte 0, that the application requires to be delivered without corruption due to lower layer errors.  It is used to specify options for simple integrity protection via checksums.  A value of 0 means that no checksum needs to be calculated, and "Full Coverage" means that the entire Message needs to be protected by a checksum.  Only "Full Coverage" is guaranteed, any other requests are advisory, which may result in "Full Coverage" being applied. </del>  9.1.3.7.

This replacement for PR was discussed at the last interim.
5 from this . Slightly modified as a result of . Should “Type” line also include (non-negative) like s9.1.3.2 does? Just trying to be matchy-matchy.
3 from this . This issue was addressed via . I included this only for completeness. Close at your leisure. I read this definition a bunch. Are there two special values? The Type line says “Full Coverage” is special, but then the description says “0” is special too. Maybe look at s91.3.6 for inspiration?
This issue on coverage is resolved in the latest.
I cringe at systems that define Integers to be integers and some other enumerated things ("special values"). Is it necessary to define things this way? I understand it's currently typical to have this behavior in the APIs of the protocols this system is attempting to abstract, but is it impossible to stop propagating that kind of type confusion? This API could move the things special values are being used for into separate arguments for example.
This only appears to be used for checksum coverage; don't overload integer for this application.
Use type enumerations instead. Some languages let you do this. Otherwise you just pick an out-of-range integern.
The "special value" also shows up for numeric in 8.1.3, 8.1.4, 8.1.8 and 8.1.9
We discussed this in the last interim, but I don't think the tuple-based approach in works well, and prefer the type enumeration suggestion NAME made. That is, don't say: rather say:
ACK - already in my todo list...
In the interface draft, section 1.1: Do we need to conflate numeric types and enumerations, or can numbers be numbers?
From interim: add a clarification on what this is about, why it (symbolic values) is there.
Fixed by Reviewer: Sean Turner Review result: Has Issues This is an early sector review. Based on the ARTART early reviews of this I-D and the -arch I-D, there is no doubt going to need to be another review after the authors get done restructuring both I-Ds. BTW: Theresa thank you for the heads up about that restructuring. It forced me to read the -arch and -impl I-Ds, but that probably helped me not make silly requests of the authors/WG. tl;dr: I picked "has issues" knowing there is a rewrite on the way and I would like to reserve final judgement until then. I apologize upfront because I am going to ramble a bit before I get to the specifics: The taps I-Ds are about “a protocol-independent Transport Services Application Programming Interface (API).” I took a pretty liberal approach when reviewing this I-D from a security perspective because these I-Ds are somewhat different than the “normal” protocol I-Ds. As noted in the -arch I-D: these I-Ds do not specify “specific protocols or algorithms” … gasp. But, that’s probably okay in this context. I wondered what you would say about an API: Apps need to use the API properly! check: see -arch I-D Implementation/Library needs to be from trust source! check: see -interface I-D - Apps need to use the right keys at the right time! check: see -arch I-D - Avoid fallback/downgrade to insecure protocols! check: see -arch I-D and -impl I-D - Deal with 0-RTT! check: see -impl I-D - Address privacy aspects! check: see -interface I-D I mean maybe you could add something about: randomness for keys - but that better already be in the security protocol specifications so I do not think it is absolutely needed here. following good programming techniques - but I am not sure where you would point nor whether it really makes sense to do so. I think I found what I was looking for is somewhere in the stack of I-Ds. Am I worried somebody can implement this wrong. Sure. But, not any more than I would be for any other protocol. Now on to the specifics: NOTE: I tried not repeat anything from the ARTART review. 0) s6: maybe this wording would be better? For these two sentences are you trying to say that MUST else if? OLD: At least one Local Endpoint MUST be specified if the Preconnection is used to Listen() for incoming Connections, but the list of Local Endpoints MAY be empty if the Preconnection is used to Initiate() connections. NEW: At least one Remote Endpoint MUST be specified if the Preconnection is used to Initiate() Connections, but the list of Remote Endpoints MAY be empty if the Preconnection is used to Listen() for incoming Connections 1) s6.3.1 I know this is an example, but can we replace: SecurityParameters.Set(supported-group, "secp256k1") with SecurityParameters.Set(supported-group, "secp256r1") r1 the current MTI for TLS. k1 is for bitcoin :) Or, is this where I get a bitcoin because I caught the shiny object? :) 2) s6.3.2: Is this like checking the certificate status? 3) s8.1.1- I read this definition a bunch. Are there two special values? The Type line says “Full Coverage” is special, but then the description says “0” is special too. Maybe look at s91.3.6 for inspiration? 4) s8.1.2: Any reason it’s not called connPriority? Seems arbitrary to drop the end of the word when connTimeout is even longer. 5) s8.1.2-should “Type” line also include (non-negative) like s8.1.1 and s9.1.3.2 do? 6) s8.1.3/.4: The Numeric values specifies seconds, minutes, hours, or millennia? 7) s8.1.6: don’t you need to specify the Type? I guess Enumeration is what you’re after? 8) s8.1.11.2/.3/.4: Are these sizes in bytes too like 8.1.11.1? 9) s8.2.1: RFC 5482 allows granularity of minutes or seconds don’t you need to say which it is here? 10) s8.2.2: Is there some reason it’s not called tcp.userTimeoutEnabled? 11) s8.2.3: Is there some reason it’s not called tcp.userTimeoutChangable? 12) s9.1.3.2: Why 100? 13) s9.1.3.2: Why shorten to msrPrio when msgOrdered is almost as long and safelyReplayable is longer? 14) A.1: Could the caution about freeing memory be generalized? 15) What follows are the I-D nits references to check. Some of these might be on purpose: == Outdated reference: A later version (-11) exists of draft-ietf-taps-arch-10 ** Obsolete normative reference: RFC 4941 (Obsoleted by RFC 8981) == Outdated reference: A later version (-06) exists of draft-ietf-httpbis-priority-03 == Outdated reference: A later version (-10) exists of draft-ietf-taps-impl-09 -- Obsolete informational reference (is this intentional?): RFC 5245 (Obsoleted by RFC 8445, RFC 8839) -- Obsolete informational reference (is this intentional?): RFC 5766 (Obsoleted by RFC 8656)

Boolean: Instances take the value "true" or "false".  <ins> Integer: Instances take positive or negative integer values. </ins>  <ins> Numeric: Instances take positive or negative real number values. </ins> Enumeration: A family of types in which each instance takes one of a fixed, predefined set of values specific to a given enumerated recvChecksumLen  <ins> Integer or "Full Coverage" </ins>  <ins> "Full Coverage" </ins>  <ins> If this property is an Integer, it specifies the minimum number of bytes in a received message that need to be covered by a checksum.  A receiving endpoint will not forward messages that have less coverage to the application.  The application is responsible for handling any corruption within the non-protected part of the message RFC8085.  A special value of 0 means that a received packet may also have a zero checksum field. </ins> 8.1.2. connTimeout  <ins> Numeric or "Disabled" </ins>  <ins> "Disabled" </ins>  <ins> If this property is Numeric, it specifies how long to wait before deciding that an active Connection has failed when trying to reliably deliver data to the Remote Endpoint.  Adjusting this Property will only take effect when the underlying stack supports reliability.  If this property has the enumerated value "Disabled", it means that no timeout is scheduled. </ins> 8.1.4. keepAliveTimeout  <ins> Numeric or "Disabled" </ins> Implementation-defined A Transport Services API can request a protocol that supports sending  <ins> keep alive packets keep-alive.  If this property is an Integer, it specifies the maximum length of time an idle connection (one for which no transport packets have been sent) should wait before the Local Endpoint sends a keep-alive packet to the Remote Endpoint. Adjusting this Property will only take effect when the underlying stack supports sending keep-alive packets.  Guidance on setting this value for datagram transports is provided in RFC8085.  A value greater than the connection timeout (conn-timeout) or the enumerated value "Disabled" will disable the sending of keep-alive packets. </ins> 8.1.5. minSendRate / minRecvRate / maxSendRate / maxRecvRate  <ins> Numeric or "Unlimited" / Numeric or "Unlimited" / Numeric or "Unlimited" / Numeric or "Unlimited" </ins>  <ins> "Unlimited" / "Unlimited" / "Unlimited" / "Unlimited" </ins>  <ins> Integer values of this property specify an upper-bound rate that a transfer is not expected to exceed (even if flow control and congestion control allow higher rates), and/or a lower-bound rate below which the application does not deem it will be useful.  These are specified in bits per second.  The enumerated value "Unlimited" indicates that no bound is specified. </ins> 8.1.9. groupConnLimit  <ins> Numeric or "Unlimited" </ins>  <ins> "Unlimited" </ins>  <ins> If this property is an Integer, it controls the number of Connections that can be accepted from a peer as new members of the Connection's group.  Similar to SetNewConnectionLimit(), this limits the number of ConnectionReceived Events that will occur, but constrained to the group of the Connection associated with this property.  For a multi- streaming transport, this limits the number of allowed streams. </ins> 8.1.10. msgChecksumLen  <ins> Integer or "Full Coverage" </ins>  <ins> "Full Coverage" </ins>  <ins> If this property is an Integer, it specifies the minimum length of the section of a sent Message, starting from byte 0, that the application requires to be delivered without corruption due to lower layer errors.  It is used to specify options for simple integrity protection via checksums.  A value of 0 means that no checksum needs to be calculated, and the enumerated value "Full Coverage" means that the entire Message needs to be protected by a checksum.  Only "Full Coverage" is guaranteed, any other requests are advisory, which may result in "Full Coverage" being applied. </ins> 9.1.3.7.


In the following cases, failure should be detected during pre- establishment: <del> A request by an application for Protocol Properties that include requirements or prohibitions that cannot be satisfied by any of the available protocols.  For example, if an application requires "Configure Reliability per Message", but no such feature is available in any protocol the host running the transport system on the host running the transport system this should result in an error, e.g., when SCTP is not supported by the operating system. </del>  A request by an application for Protocol Properties that are in conflict with each other, i.e., the required and prohibited 7.1. <del> For protocols that employ request/response pairs and do not require in-order delivery of the responses, like HTTP, the transport implementation may distribute interactions across several underlying transport connections.  For these kinds of protocols, implementations may hide the connection management and only expose a single Connection object and the individual requests/responses as messages. These Pooled Connections can use multiple connections or multiple streams of multi-streaming connections between endpoints, as long as all of these satisfy the requirements, and prohibitions specified in the Selection Properties of the Pooled Connection.  This enables implementations to realize transparent connection coalescing, connection migration, and to perform per-message endpoint and path selection by choosing among these underlying connections. </del>  7.2.

So implementation talks about pooled connections - I must have blinked and missed that, obviously it is different to a connection group - or is it? If I am asking a silly question then I suspect we need just a few lines of text to demuddle me.
LGTM, thanks!

In the following cases, failure should be detected during pre- establishment:  <ins> A request by an application for Protocol Properties that cannot be satisfied by any of the available protocols.  For example, if an application requires "Configure Reliability per Message", but no such feature is available in any protocol the host running the transport system on the host running the transport system this should result in an error, e.g., when SCTP is not supported by the operating system. </ins> A request by an application for Protocol Properties that are in conflict with each other, i.e., the required and prohibited 7.1.  <ins> For applications that do not need in-order delivery of Messages, the Transport Services implementation may distribute Messages of a single Connection across several underlying transport connections or multiple streams of multi-streaming connections between endpoints, as long as all of these satisfy the Selection Properties.  The Transport Services implementation will then hide this connection management and only expose a single Connection object, which we here call a "Pooled Connection".  This is in contrast to Connection Groups, which explicitly expose combined treatment of Connections, giving the application control over multiplexing, for example. Pooled Connections can be useful when the application using the Transport Services system implements a protocol such as HTTP, which employs request/response pairs and does not require in-order delivery of responses.  This enables implementations of Transport Services systems to realize transparent connection coalescing, connection migration, and to perform per-message endpoint and path selection by choosing among multiple underlying connections. </ins> 7.2.


Parameters.  Action and Event parameters whose names are suffixed with a question mark are optional. <del> Actions associated with no Object are Actions on the API; they are equivalent to Actions on a per-application global context. </del>  Events are sent to the application or application-supplied code (e.g. framers, see framing) for processing; the details of event processing same properties. Properties can be added to a MessageContext object only before the <del> context is used for sending.  Once a messageContext has been used with a Send call, it is invalid to modify any of its properties. </del>  The Message Properties could be inconsistent with the properties of the Protocol Stacks underlying the Connection on which a given

There are several places where the document says "Changes to a after func() has been called do not have any connection on existing s". This feels like an opportunity for more clarity about what a really is. The document uses the word Object, and maybe there's an assumption that there's a shared understanding of what the properties of an Object are. Being explicit with something like "An Object is a collection of properties that implementations of the API read and use..." to avoid the "here's a chunk of memory that can get passed into the API that it can operate on, and expect the caller to see the effects of the operation" interpretation. I think what the current text is trying to say is that the user of the API should assume the implementation of the API made its own private copy of the Object at the point of the given func() call and that it will not see any changes to the copy the user has at any later point. But then there's some text that looks like "Once a has been used with a func(), it is invalid to modify any of its properties." So...

Parameters.  Action and Event parameters whose names are suffixed with a question mark are optional.  <ins> Objects that are passed as parameters to Actions use call-by-value behavior.  Actions associated with no Object are Actions on the API; they are equivalent to Actions on a per-application global context. </ins> Events are sent to the application or application-supplied code (e.g. framers, see framing) for processing; the details of event processing same properties. Properties can be added to a MessageContext object only before the  <ins> context is used for sending.  Once a MessageContext has been used with a Send call, further modifications to the MessageContext object do not have any effect on this Send call. </ins> The Message Properties could be inconsistent with the properties of the Protocol Stacks underlying the Connection on which a given


transport connection.  Endpoints can be Local Endpoints or Remote Endpoints, and respectively represent an identity that the application uses for the source or destination of a connection. <del> An Endpoint can be specified at various levels of abstraction, and an Endpoint at a higher level of abstraction (such as a hostname) can be resolved to more concrete identities (such as IP addresses). </del>  Remote Endpoint: The Remote Endpoint represents the application's identifier for a peer that can participate in a transport

This is the only place multicast considerations might fit in -arch. We don't discuss anycast at all, and I think it's largely orthogonal to transport (indeed, people seem to be running stateful/connection-oriented transports to anycast endpoints just fine in the Internet, even though it "doesn't work" :) )
Consider an explicit discussion of multicast/anycast considerations in this document.

transport connection.  Endpoints can be Local Endpoints or Remote Endpoints, and respectively represent an identity that the application uses for the source or destination of a connection.  <ins> An Endpoint can be specified at various levels of abstraction.  An Endpoint at a higher level of abstraction (such as a hostname) can be resolved to more concrete identities (such as IP addresses). An endpoint may also represent a multicast group, in which case it selects a multicast transport for communication. </ins> Remote Endpoint: The Remote Endpoint represents the application's identifier for a peer that can participate in a transport


Hostname (string): <del> Port (a 16-bit integer) or a Service (string) that maps to a port: </del>  IP address (IPv4 or IPv6 address):

Would there be any benefit of having something like "WithServiceName" to qualify a specifier for use with NAPTR/SRV? Or is the thinking that the resolution of the thing handed down through WithHostname is restricted to A/AAAA?
yes, probably.
We do have this for mDNS services in Network.framework, and internally have tested with SRV. These are useful indeed.
Actually, doesn't WithService do this? Should we clarify that this can use SRV as opposed to just looking up in /etc/services.
Interim: Yes, this is a nice solution. Let's mention how to use withService for SRV types in the text.

Hostname (string):  <ins> Port (a 16-bit integer): Service (an identifier that maps to a port; either a the name of a well-known service, or a DNS SRV service name to be resolved): </ins> IP address (IPv4 or IPv6 address):


Message Properties can be set and queried using the Message Context: <del> To get or set Message Properties, the optional scope parameter is left empty.  To get or set meta-data for a Framer, the application has to pass a reference to this Framer as the scope parameter. </del>  For MessageContexts returned by send Events (see send-events) and receive Events (see receive-events), the application can query 9.1.2.2. <del> When sending Messages, applications can add Framer-specific key/value pairs to a MessageContext (msg-ctx).  This mechanism can be used, for example, to set the type of a Message for a TLV format.  The namespace of values is custom for each unique Message Framer. </del>  When an application receives a MessageContext in a Receive event, it can also look to see if a value was set by a specific Message Framer.

The other option is to tell people to prepend the name of the framer to the property name.
Is the "scope" used for message contexts (9.1.1) ever anything but a framer? If not, could this section just say framer?
We believe that it is the same; if so, "scope" should probably indeed be named "framer". Tommy, could you please check?
I find this a little strange: given that we do have optional parameters, what's the benefit of first introducing the "add" and "get" calls as if the "framer" parameter didn't exist, and then later changing that? It would seem more consistent to me to introduce it where you had the scope parameter before, called "framer", and use a forward reference to say that framers will be explained later.

Message Properties can be set and queried using the Message Context:  <ins> These Message Properties may be generic properties or Protocol Specific Properties. </ins> For MessageContexts returned by send Events (see send-events) and receive Events (see receive-events), the application can query 9.1.2.2.  <ins> When sending Messages, applications can add Framer-specific properties to a MessageContext (msg-ctx).  In order to set these properties, the "add" and "get" actions on the MessageContext.  To avoid naming conflicts, the property names SHOULD be prefixed with a namespace referencing the framer implementation or the protocol it implements as described in property-names. This mechanism can be used, for example, to set the type of a Message for a TLV format.  The namespace of values is custom for each unique Message Framer. </ins> When an application receives a MessageContext in a Receive event, it can also look to see if a value was set by a specific Message Framer.