4. <del> **Note to Reviewers^^ - this section includes some material that may be tutorial for some readers.  We can decide how to say that, if the tutorial material is worth keeping.  Spencer thought it was worth including, because it provides a contrast to the material on QUIC, which is significantly less tutorial, unless you participated in the QUIC working group. Because networking resources are shared between users, a good place to start our discussion is how contention between users, and mechanisms to resolve that contention in ways that are "fair" between users, impact streaming media users.  These topics are closely tied to transport protocol behaviors. As noted in sec-abr, Adaptive Bitrate response strategies such as HLS RFC8216 or DASH DASH are attempting to respond to changing path characteristics, and underlying transport protocols are also attempting to respond to changing path characteristics. For most of the history of the Internet, these transport protocols, described in udp-behavior and tcp-behavior, have had relatively consistent behaviors that have changed slowly, if at all, over time. Newly standardized transport protocols like QUIC I-D.ietf-quic- transport can behave differently from existing transport protocols, and these behaviors may evolve over time more rapidly than currently- used transport protocols. For this reason, we have included a description of how the path characteristics that streaming media providers may see are likely to evolve over time. 4.1. For most of the history of the Internet, we have trusted UDP-based applications to limit their impact on other users.  One of the strategies used was to use UDP for simple query-response application protocols, such as DNS, which is often used to send a single-packet request to look up the IP address for a DNS name, and return a single-packet response containing the IP address.  Although it is possible to saturate a path between a DNS client and DNS server with DNS requests, in practice, that was rare enough that DNS included few mechanisms to resolve contention between DNS users and other users (whether they are also using DNS, or using other application protocols). In recent times, the usage of UDP-based applications that were not simple query-response protocols has grown substantially, and since UDP does not provide any feedback mechanism to senders to help limit impacts on other users, application-level protocols such as RTP RFC3550 have been responsible for the decisions that TCP-based applications have delegated to TCP - what to send, how much to send, and when to send it.  So, the way some UDP-based applications interact with other users has changed. It's also worth pointing out that because UDP has no transport-layer feedback mechanisms, UDP-based applications that send and receive substantial amounts of information are expected to provide their own feedback mechanisms.  This expectation is most recently codified in Best Current Practice RFC8085. RTP relies on RTCP Sender and Receiver Reports RFC3550 as its own feedback mechanism, and even includes Circuit Breakers for Unicast RTP Sessions RFC8083 for situations when normal RTP congestion control has not been able to react sufficiently to RTP flows sending at rates that result in sustained packet loss. The notion of "Circuit Breakers" has also been applied to other UDP applications in RFC8084, such as tunneling packets over UDP that are potentially not congestion-controlled (for example, "Encapsulating MPLS in UDP", as described in RFC7510).  If streaming media is carried in tunnels encapsulated in UDP, these media streams may encounter "tripped circuit breakers", with resulting user-visible impacts. 4.2. For most of the history of the Internet, we have trusted the TCP protocol to limit the impact of applications that sent a significant number of packets, in either or both directions, on other users. Although early versions of TCP were not particularly good at limiting this impact RFC0793, the addition of Slow Start and Congestion Avoidance, as described in RFC2001, were critical in allowing TCP- based applications to "use as much bandwidth as possible, but to avoid using more bandwidth than was possible".  Although dozens of RFCs have been written refining TCP decisions about what to send, how much to send, and when to send it, since 1988 Jacobson-Karels the signals available for TCP senders remained unchanged - end-to-end acknowledgments for packets that were successfully sent and received, and packet timeouts for packets that were not. The success of the largely TCP-based Internet is evidence that the mechanisms TCP used to achieve equilibrium quickly, at a point where TCP senders do not interfere with other TCP senders for sustained periods of time, have been largely successful.  The Internet continued to work even when the specific mechanisms used to reach equilibrium changed over time.  Because TCP provides a common tool to avoid contention, as some TCP-based applications like FTP were largely replaced by other TCP-based applications like HTTP, the transport behavior remained consistent. In recent times, the TCP goal of probing for available bandwidth, and "backing off" when a network path is saturated, has been supplanted by the goal of avoiding growing queues along network paths, which prevent TCP senders from reacting quickly when a network path is saturated.  Congestion control mechanisms such as COPA COPA and BBR I-D.cardwell-iccrg-bbr-congestion-control make these decisions based on measured path delays, assuming that if the measured path delay is increasing, the sender is injecting packets onto the network path faster than the receiver can accept them, so the sender should adjust its sending rate accordingly. Although TCP protocol behavior has changed over time, the common practice of implementing TCP as part of an operating system kernel has acted to limit how quickly TCP behavior can change.  Even with the widespread use of automated operating system update installation on many end-user systems, streaming media providers could have a reasonable expectation that they could understand TCP transport protocol behaviors, and that those behaviors would remain relatively stable in the short term. 4.3. The QUIC protocol, developed from a proprietary protocol into an IETF standards-track protocol I-D.ietf-quic-transport, turns many of the statements made in udp-behavior and tcp-behavior on their heads. Although QUIC provides an alternative to the TCP and UDP transport protocols, QUIC is itself encapsulated in UDP.  As noted elsewhere in this document, the QUIC protocol encrypts almost all of its transport parameters, and all of its payload, so any intermediaries that network operators may be using to troubleshoot HTTP streaming media performance issues, perform analytics, or even intercept exchanges in current applications will not work for QUIC-based applications without making changes to their networks. While QUIC is designed as a general-purpose transport protocol, and can carry different application-layer protocols, the current standardized mapping is for HTTP/3 I-D.ietf-quic-http, which describes how QUIC transport features are used for HTTP.  The convention is for HTTP/3 to run over UDP port 443 Port443 but this is not a strict requirement. When HTTP/3 is encapsulated in QUIC, which is then encapsulated in UDP, streaming operators (and network operators) might see UDP traffic patterns that are similar to HTTP(S) over TCP.  Since earlier versions of HTTP(S) rely on TCP, UDP ports may be blocked for any port numbers that are not commonly used, such as UDP 53 for DNS. Even when UDP ports are not blocked and HTTP/3 can flow, streaming operators (and network operators) may severely rate-limit this traffic because they do not expect to see legitimate high-bandwidth traffic such as streaming media over the UDP ports that HTTP/3 is using. As noted in tcp-behavior, there is increasing interest in transport protocol behaviors that responds to delay measurements, instead of responding to packet loss.  These behaviors may deliver improved user experience, but in some cases have not responded to sustained packet loss, which exhausts available buffers along the end-to-end path that may affect other users sharing that path.  The QUIC protocol provides a set of congestion control hooks that can be use for algorithm agility, and I-D.ietf-quic-recovery defines a basic algorithm with transport behavior that is roughly similar to TCP NewReno RFC6582. However, QUIC senders can and do unilaterally chose to use different algorithms such as loss-based CUBIC RFC8312, delay-based COPA or BBR, or even something completely different We do have experience with deploying new congestion controllers without melting the Internet (CUBIC is one example), but the point mentioned in tcp-behavior about TCP being implemented in operating system kernels is also different with QUIC.  Although QUIC can be implemented in operating system kernels, one of the design goals when this work was chartered was "QUIC is expected to support rapid, distributed development and testing of features", and to meet this expectation, many implementers have chosen to implement QUIC in user space, outside the operating system kernel, and to even distribute QUIC with applications. The decision to deploy a new version of QUIC is relatively uncontrolled, compared to other widely used transport protocols, and this can include new transport behaviors that appear without much notice except to the QUIC endpoints.  At IETF 105, Christian Huitema and Brian Trammell presented a talk on "Congestion Defense in Depth" CDiD, that explored potential concerns about new QUIC congestion controllers being broadly deployed without the testing and instrumentation that current major content providers routinely include.  The sense of the room at IETF 105 was that the current major content providers understood what is at stake when they deploy new congestion controllers, but this presentation, and the related discussion in TSVAREA minutes from IETF 105 (tsvarea-105, are still worth a look for new and rapidly growing content providers. It is worth considering that if TCP-based HTTP traffic and UDP-based HTTP/3 traffic are allowed to enter operator networks on roughly equal terms, questions of fairness and contention will be heavily dependent on interactions between the congestion controllers in use for TCP-base HTTP traffic and UDP-based HTTP/3 traffic. More broadly, I-D.ietf-quic-manageability discusses manageability of the QUIC transport protocol, focusing on the implications of QUIC's design and wire image on network operations involving QUIC traffic. It discusses what network operators can consider in some detail. 5. </del> "Encrypted Media" has at least three meanings: Media encrypted at the application layer, typically using some Both "End-to-End" and "Hop-by-Hop" media encryption have implications for streaming operators. <del> 5.1. </del>  The use of strong encryption does provide confidentiality for encrypted streaming media, from the sender to either an intermediary ID I-D.ietf-quic-invariants.  For these reasons, HTTP/3 is significantly more "opaque" than HTTPS with HTTP/1 or HTTP/2. <del> 5.2. </del>  Although the IETF has put considerable emphasis on end-to-end streaming media encryption, there are still important use cases that consideration are in RFC8825 (for WebRTC) and I-D.ietf-quic- manageability (for HTTP/3 and QUIC). <del> 5.3. </del>  "End-to-end" media encryption offers the potential of providing privacy for streaming media consumers, with the idea being that if an delivered as a routine practice, rather than in exceptional cases for content that was considered "sensitive". <del> 6. </del>  This document requires no actions from IANA. <del> 7. </del>  This document introduces no new security issues. <del> 8. Thanks to Mark Nottingham, Glenn Deen, Dave Oran, Aaron Falk, Kyle Rose, Leslie Daigle, Lucas Pardue, Matt Stock, Alexandre Gouaillard, and Mike English for their very helpful reviews and comments. </del>

Reverts ietf-wg-mops/draft-ietf-mops-streaming-opcons (still learning github)

4.  "Encrypted Media" has at least three meanings: Media encrypted at the application layer, typically using some Both "End-to-End" and "Hop-by-Hop" media encryption have implications for streaming operators.  <ins> 4.1. </ins> The use of strong encryption does provide confidentiality for encrypted streaming media, from the sender to either an intermediary ID I-D.ietf-quic-invariants.  For these reasons, HTTP/3 is significantly more "opaque" than HTTPS with HTTP/1 or HTTP/2.  <ins> 4.2. </ins> Although the IETF has put considerable emphasis on end-to-end streaming media encryption, there are still important use cases that consideration are in RFC8825 (for WebRTC) and I-D.ietf-quic- manageability (for HTTP/3 and QUIC).  <ins> 4.3. </ins> "End-to-end" media encryption offers the potential of providing privacy for streaming media consumers, with the idea being that if an delivered as a routine practice, rather than in exceptional cases for content that was considered "sensitive".  <ins> 5. </ins> This document requires no actions from IANA.  <ins> 6. </ins> This document introduces no new security issues.


This document provides an overview of operational networking issues that pertain to quality of experience in streaming of video and other <del> high-bitrate media over the internet. </del>  1. 3. Streaming media latency refers to the "glass-to-glass" time duration, <del> which is the delay between the real-life occurence of an event and </del>  the streamed media being appropriately displayed on an end user's device.  Note that this is different from the network latency (defined as the time for a packet to cross a network from one end to Because adaptive application-level response strategies are typically using application-level protocols, these mechanisms are affected by transport-level protocol behaviors, and the application-level <del> feedback loop is interacting with a transport-level feedback loop, ase described in idle-time and hol-blocking. </del>  4.3.1. players, or whether current media content caching is providing the best experience for viewers.  4.5. In contrast to segmented delivery, several applications use UDP or

I think we should wait until the [CTA-5004] reference link is resolved to merge this PR.
I made this change in the CMCD branch. We can merge it when we're ready.
Will resolve conflicts and merge.
We could reasonably start with standardized mechanisms, and could also give other examples. Also provide links to specifications that provide guidance on what to measure.
Replace: In addition to measurements media players use to guide their segment-by-segment adaptive streaming requests, streaming media providers may also rely on measurements collected from media players to provide analytics that can be used for decisions such as whether the adaptive encoding bitrates in use are the best ones to provide to media players, or whether current media content caching is providing the best experience for viewers. With: In addition to measurements media players use to guide their segment-by-segment adaptive streaming requests, streaming media providers may also rely on measurements collected from media players to provide analytics that can be used for decisions such as whether the adaptive encoding bitrates in use are the best ones to provide to media players, or whether current media content caching is providing the best experience for viewers. To that effect, the Consumer Technology Association (CTA) who owns the Web Application Video Ecosystem (WAVE) project has published two important specifications. CTA-2066 specifies a set of media player events, properties, quality of experience (QoE) metrics and associated terminology for representing streaming media quality of experience across systems, media players and analytics vendors. While all these events, properties, metrics and associated terminology is used across a number of proprietary analytics and measurement solutions, they were used in slightly (or vastly) different ways that led to interoperability issues. CTA-2066 attempts to address this issue by defining a common terminology as well as how each metric should be computed for consistent reporting. Many assumes that the CDNs have a holistic view into the health and performance of the streaming clients. However, this is not the case. The CDNs produce millions of log lines per second across hundreds of thousands of clients and they have no concept of a "session" as a client would have, so CDNs are decoupled from the metrics the clients generate and report. A CDN cannot tell which request belongs to which playback session, the duration of any media object, the bitrate, or whether any of the clients have stalled and are rebuffering or are about to stall and will rebuffer. The consequence of this decoupling is that a CDN cannot prioritize delivery for when the client needs it most, prefetch content, or trigger alerts when the network itself may be underperforming. One approach to couple the CDN to the playback sessions is for the clients to communicate standardized media-relevant information to the CDNs while they are fetching data. CTA-5004 was developed exactly for this purpose.
CTA-2066: URL CTA-5004: URL
This should be addressed by . Ali, please confirm this, and let me know when we can change the reference URL for [CTA-5004].
URL is fixed now, so please use URL
Looks good, and I asked CTA to fix the link and they did, so please use instead: URL

This document provides an overview of operational networking issues that pertain to quality of experience in streaming of video and other  <ins> high-bitrate media over the Internet. </ins> 1. 3. Streaming media latency refers to the "glass-to-glass" time duration,  <ins> which is the delay between the real-life occurrence of an event and </ins> the streamed media being appropriately displayed on an end user's device.  Note that this is different from the network latency (defined as the time for a packet to cross a network from one end to Because adaptive application-level response strategies are typically using application-level protocols, these mechanisms are affected by transport-level protocol behaviors, and the application-level  <ins> feedback loop is interacting with a transport-level feedback loop, as described in idle-time and hol-blocking. </ins> 4.3.1. players, or whether current media content caching is providing the best experience for viewers. <ins> In addition to measurements media players use to guide their segment- by-segment adaptive streaming requests, streaming media providers may also rely on measurements collected from media players to provide analytics that can be used for decisions such as whether the adaptive encoding bitrates in use are the best ones to provide to media players, or whether current media content caching is providing the best experience for viewers.  To that effect, the Consumer Technology Association (CTA) who owns the Web Application Video Ecosystem (WAVE) project has published two important specifications. 4.4.1. CTA-2066 specifies a set of media player events, properties, quality of experience (QoE) metrics and associated terminology for representing streaming media quality of experience across systems, media players and analytics vendors.  While all these events, properties, metrics and associated terminology is used across a number of proprietary analytics and measurement solutions, they were used in slightly (or vastly) different ways that led to interoperability issues.  CTA-2066 attempts to address this issue by defining a common terminology as well as how each metric should be computed for consistent reporting. 4.4.2. Many assumes that the CDNs have a holistic view into the health and performance of the streaming clients.  However, this is not the case. The CDNs produce millions of log lines per second across hundreds of thousands of clients and they have no concept of a "session" as a client would have, so CDNs are decoupled from the metrics the clients generate and report.  A CDN cannot tell which request belongs to which playback session, the duration of any media object, the bitrate, or whether any of the clients have stalled and are rebuffering or are about to stall and will rebuffer.  The consequence of this decoupling is that a CDN cannot prioritize delivery for when the client needs it most, prefetch content, or trigger alerts when the network itself may be underperforming.  One approach to couple the CDN to the playback sessions is for the clients to communicate standardized media-relevant information to the CDNs while they are fetching data.  CTA-5004 was developed exactly for this purpose. </ins> 4.5. In contrast to segmented delivery, several applications use UDP or


projections can be used to help with e.g. power consumption with routing architectures that provide for modular scalability.  3. 3.1.

title: Operational Considerations for Streaming Media abbrev: Media Streaming Ops docname: draft-jholland-mops-taxonomy-01 date: 2020-01-17 category: info ipr: trust200902 area: Ops workgroup: Mops keyword: Internet-Draft standalone: yes pi: [toc, sortrefs, symrefs] author: ins: J. Holland name: Jake Holland org: Akamai Technologies, Inc. street: 150 Broadway city: Cambridge, MA 02144 country: United States of America email: EMAIL informative: CVNI: target: URL title: "Cisco Visual Networking Index: Forecast and Trends, 2017–2022 White Paper" author: org: "Cisco Systems, Inc." date: 2019-02-27 DASH: title: "Information technology -- Dynamic adaptive streaming over HTTP (DASH) -- Part 1: Media presentation description and segment formats" seriesinfo: "ISO/IEC": 23009-1:2014 date: 2014-05 MSOD: title: "Media Services On Demand: Encoder Best Practices" author: org: "Akamai Technologies, Inc." target: URL date: 2019 RFC2309: RFC3168: RFC5594: RFC5762: RFC6190: RFC8033: RFC8216: I-D.han-iccrg-arvr-transport-problem: --- abstract This document provides an overview of operational networking issues that pertain to quality of experience in delivery of video and other high-bitrate media over the internet. --- middle #Introduction {#intro} As the internet has grown, an increasingly large share of the traffic delivered to end users has become video. Estimates put the total share of internet video traffic at 75% in 2019, expected to grow to 82% by 2022. What's more, this estimate projects the gross volume of video traffic will more than double during this time, based on a compound annual growth rate continuing at 34% (from Appendix D of {{CVNI}}). In many contexts, video traffic can be handled transparently as generic application-level traffic. However, as the volume of video traffic continues to grow, it's becoming increasingly important to consider the effects of network design decisions on application-level performance, with considerations for the impact on video delivery. This document aims to provide a taxonomy of networking issues as they relate to quality of experience in internet video delivery. The focus is on capturing characteristics of video delivery that have surprised network designers or transport experts without specific video expertise, since these highlight key differences between common assumptions in existing networking documents and observations of video delivery issues in practice. Making specific recommendations for mitigating these issues is out of scope, though some existing mitigations are mentioned in passing. The intent is to provide a point of reference for future solution proposals to use in describing how new technologies address or avoid these existing observed problems. #Bandwidth Provisioning ##Scaling Requirements for Media Delivery {#scaling} ###Video Bitrates Video bit-rate selection depends on many variables. Different providers give different guidelines, but an equation that approximately matches the bandwidth requirement estimates from several video providers is given in {{MSOD}}: Height and width are in pixels, and frame rate in frames per second. The actual bit-rate required for a specific video will also depend on the codec used and some other characteristics of the video itself, such as the frequency of high-detail motion, which may influence the compressability of the content, but this equation provides a rough estimate. Here are a few common resolutions used for video content, with their minimum per-user bandwidth requirements according to this formula: Approximate Bit-rate for 60fps | -----+----------------+------------------------------- 3 Mbps 8 Mbps 18 Mbps 70 Mbps ###Virtual Reality Bitrates TBD: Reference and/or adapt content from expired work-in-progress {{I-D.han-iccrg-arvr-transport-problem}}. The punchline is that it starts at a bare minimum of 22 Mbps mean with a 130 Mbps peak rate, up to 3.3 Gbps mean with 38 Gbps peak for high-end technology. ##Path Requirements The bit-rate requirements in {{scaling}} are per end-user actively consuming a media feed, so in the worst case, the bit-rate demands can be multiplied by the number of simultaneous users to find the bandwidth requirements for a router on the delivery path with that number of users downstream. For example, at a node with 10,000 downstream users simultaneously consuming video streams, approximately up to 180 Gbps would be necessary in order for all of them to get 1080p resolution at 60 fps. However, when there is some overlap in the feeds being consumed by end users, it is sometimes possible to reduce the bandwidth provisioning requirements for the network by performing some kind of replication within the network. This can be achieved via object caching with delivery of replicated objects over individual connections, and/or by packet-level replication using multicast. To the extent that replication of popular content can be performed, bandwidth requirements at peering or ingest points can be reduced to as low as a per-feed requirement instead of a per-user requirement. ##Caching Systems TBD: pros, cons, tradeoffs of caching designs at different locations within the network? Peak vs. average provisioning, and effects on peering point congestion under peak load? Provisioning issues for caching systems? ##Predictable Usage Profiles TBD: insert charts showing historical relative data usage patterns with error bars by time of day in consumer networks? Cross-ref vs. video quality by time of day in practice for some case study? Not sure if there's a good way to capture a generalized insight here, but it seems worth making the point that demand projections can be used to help with e.g. power consumption with routing architectures that provide for modular scalability. ##Unpredictable Usage Profiles Although TCP/IP has been used with a number of widely used applications that have symmetric bandwidth requirements (similar bandwidth requirements in each direction between endpoints), many widely-used Internet applications operate in client-server roles, with asymmetric bandwidth requirements. A common example might be an HTTP GET operation, where a client sends a relatively small HTTP GET request for a resource to an HTTP server, and often receives a significantly larger response carrying the requested resource. When HTTP is commonly used to stream movie-length video, the ratio between response size and request size can become quite large. For this reason, operators may pay more attention to downstream bandwidth utilization when planning and managing capacity. In addition, operators have been able to deploy access networks for end users using underlying technologies that are inherently asymetric, favoring downstream bandwidth (e.g. ADSL, cellular technologies, most IEEE 802.11 variants), assuming that users will need less upstream bandwidth than downstream bandwidth. This strategy usually works, except when it does not, because application bandwidth usage patterns have changed. One example of this type of change was when peer-to-peer file sharing applications gained popularity in the early 2000s. To take one well-documented case ({{RFC5594}}), the Bittorrent application created "swarms" of hosts, uploading and downloading files to each other, rather than communicating with a server. Bittorrent favored peers who uploaded as much as they downloaded, so that new Bittorrent users had an incentive to significantly increased their upstream bandwidth utilization. One example of this type of change was when peer-to-peer file sharing applications gained popularity in the early 2000s. To take one well-documented case ({{RFC5594}}), the Bittorrent application created "swarms" of hosts, uploading and downloading files to each other, rather than communicating with a server. Bittorrent favored peers who uploaded as much as they downloaded, so that new Bittorrent users had an incentive to significantly increased their upstream bandwidth utilization. The combination of the large volume of "torrents" and the peer-to-peer characteristic of swarm transfers meant that end user hosts were suddenly uploading higher volumes of traffic to more destinations than was the case before Bittorrent. This caused at least one large ISP to attempt to "throttle" these transfers, to mitigate the load that these hosts placed on their network. These efforts were met by increased use of encryption in Bittorrent, similar to an arms race, and set off discussions about "Net Neutrality" and calls for regulatory action. Especially as end users increase use of video-based social networking applications, it will be helpful for access network providers to watch for applications that encourage end users to upload significant amounts of content. #Adaptive Bit Rate ##Overview Adaptive Bit-Rate (ABR) is a sort of application-level congestion response strategy in which the receiving media player attempts to detect the available bandwidth of the network path by experiment or by observing the successful application-layer download speed, then chooses a video bitrate that fits within that bandwidth, typically adjusting as changes in available bandwidth occur in the network. The choice of bit-rate occurs within the context of optimizing for some metric monitored by the video player, such as highest achievable video quality, or lowest rate of expected rebuffering events. ##Segmented Delivery ABR strategies are commonly implemented by video players using HLS {{RFC8216}} or DASH {{DASH}} to perform a reliable segment delivery of video data over HTTP. Different player implementations and receiving devices use different strategies, often proprietary algorithms, to perform the bit-rate selection and available bandwidth estimation. This kind of bandwidth-detection system can experience trouble in several ways that can be affected by networking design choices. ###Idle Time Between Segments When the bit-rate selection is successfully chosen below the available capacity of the network path, the response to a segment request will complete in less absolute time than the video bit-rate speed. The resulting idle time within the connection carrying the segments has a few surprising consequences: Mobile flow-bandwidth spectrum and timing mapping. TCP Slow-start when restarting after idle requires multiple RTTs to re-establish a throughput at the network's available capacity. On high-RTT paths or with small enough segments, this can produce a falsely low application-visible measurement of the available network capacity. ###Head of Line Blocking In the event of a lost packet on a TCP connection with SACK support (a common case for segmented delivery in practice), loss of a packet can provide a confusing bandwidth signal to the receiving application. Because of the sliding window in TCP, many packets may be accepted by the receiver without being available to the application until the missing packet arrives. Upon arrival of the one missing packet after retransmit, the receiver will suddenly get access to a lot of data at the same time. To a receiver measuring bytes received per unit time at the application layer, and interpreting it as an estimate of the available network bandwidth, this appears as a high jitter in the goodput measurement. Active Queue Management (AQM) systems such as PIE {{RFC8033}} or variants of RED {{RFC2309}}} that induce early random loss under congestion can mitigate this by using ECN {{RFC3168}} where available. ECN provides a congestion signal and induce a similar backoff in flows that use Explicit Congestion Notification-capable transport, but by avoiding loss avoids inducing head-of-line blocking effects in TCP connections. ##Unreliable Transport In contrast to segmented delivery, several applications use UDP or unreliable SCTP to deliver RTP or raw TS-formatted video. Under congestion and loss, this approach generally experiences more video artifacts with fewer delay or head of line blocking effects. Often one of the key goals is to reduce latency, to better support applications like video conferencing, or for other live-action video with interactive components, such as some sporting events. Congestion avoidance strategies for this kind of deployment vary widely in practice, ranging from some streams that are entirely unresponsive to using feedback signaling to change encoder settings (as in {{RFC5762}}), or to use fewer enhancement layers (as in {{RFC6190}}), to proprietary methods for detecting quality of experience issues and cutting off video. Note to RFC Editor: Please remove this section before publication TBD: suggestion from mic at IETF 106 (Mark Nottingham): dive into the different constraints coming from different parts of the network or distribution channels. (regarding questions about how to describe the disconnect between demand vs. capacity, while keeping good archival value.) URL TBD: suggestion from mic at IETF 106 (Dave Oran + Glenn Deen responding): pre-placement for many use cases is useful--distinguish between live vs. cacheable. "People assume high-demand == live, but not always true" with popular netflix example. (Glenn): something about latency requirements for cached vs. streaming on live vs. pre-recorded content, and breaking requirements into 2 separate charts. also: "Standardized ladder" for adaptive bit rate rates suggested, declined as out of scope. URL TBD: suggestion at the mic from IETF 106 (Aaron Falk): include industry standard metrics from citations, some standard scoping metrics may be already defined. URL #IANA Considerations This document requires no actions from IANA. #Security Considerations This document introduces no new security issues. --- back
One nit suggested, go ahead and push with that fix, and thanks!

projections can be used to help with e.g. power consumption with routing architectures that provide for modular scalability. <ins> 2.5. Although TCP/IP has been used with a number of widely used applications that have symmetric bandwidth requirements (similar bandwidth requirements in each direction between endpoints), many widely-used Internet applications operate in client-server roles, with asymmetric bandwidth requirements.  A common example might be an HTTP GET operation, where a client sends a relatively small HTTP GET request for a resource to an HTTP server, and often receives a significantly larger response carrying the requested resource.  When HTTP is commonly used to stream movie-length video, the ratio between response size and request size can become quite large. For this reason, operators may pay more attention to downstream bandwidth utilization when planning and managing capacity.  In addition, operators have been able to deploy access networks for end users using underlying technologies that are inherently asymetric, favoring downstream bandwidth (e.g.  ADSL, cellular technologies, most IEEE 802.11 variants), assuming that users will need less upstream bandwidth than downstream bandwidth.  This strategy usually works, except when it does not, because application bandwidth usage patterns have changed. One example of this type of change was when peer-to-peer file sharing applications gained popularity in the early 2000s.  To take one well- documented case (RFC5594), the Bittorrent application created "swarms" of hosts, uploading and downloading files to each other, rather than communicating with a server.  Bittorrent favored peers who uploaded as much as they downloaded, so that new Bittorrent users had an incentive to significantly increase their upstream bandwidth utilization. One example of this type of change was when peer-to-peer file sharing applications gained popularity in the early 2000s.  To take one well- documented case (RFC5594), the Bittorrent application created "swarms" of hosts, uploading and downloading files to each other, rather than communicating with a server.  Bittorrent favored peers who uploaded as much as they downloaded, so that new Bittorrent users had an incentive to significantly increase their upstream bandwidth utilization. The combination of the large volume of "torrents" and the peer-to- peer characteristic of swarm transfers meant that end user hosts were suddenly uploading higher volumes of traffic to more destinations than was the case before Bittorrent.  This caused at least one large ISP to attempt to "throttle" these transfers, to mitigate the load that these hosts placed on their network.  These efforts were met by increased use of encryption in Bittorrent, similar to an arms race, and set off discussions about "Net Neutrality" and calls for regulatory action. Especially as end users increase use of video-based social networking applications, it will be helpful for access network providers to watch for users who are uploading significant amounts of content. </ins> 3. 3.1.


4.5. <del> In contrast to segmented delivery, several applications use UDP or unreliable SCTP to deliver RTP or raw TS-formatted video. </del>  Under congestion and loss, this approach generally experiences more video artifacts with fewer delay or head-of-line blocking effects. applications like videoconferencing, or for other live-action video with interactive components, such as some sporting events. <del> Congestion avoidance strategies for this kind of deployment vary widely in practice, ranging from some streams that are entirely unresponsive to using feedback signaling to change encoder settings (as in RFC5762), or to use fewer enhancement layers (as in RFC6190), to proprietary methods for detecting quality of experience issues and cutting off video. </del>  5. <del> **Note to Reviewers^^ - this section includes some material that may be tutorial for some readers.  We can decide how to say that, if the tutorial material is worth keeping.  Spencer thought it was worth including, because it provides a contrast to the material on QUIC, which is significantly less tutorial, unless you participated in the QUIC working group. </del>  Because networking resources are shared between users, a good place to start our discussion is how contention between users, and

Also clarified relationship with UDP Behaviors section
I think this is ready to merge. Opinions?
I was more concerned about overlap when I opened , but what's in the section on Unreliable Transport and what's in the section on UDP behaviors seems to read pretty well to me. Opinions?
SRT, TS-Multicast over UDP, etc.
I THINK this is requesting a change in the "Unreliable Transport" section.
I think this one is also ready to close. Opinions?

4.5.  <ins> In contrast to segmented delivery, several applications use unreliable UDP or SCTP with its "partial reliability" extension RFC3758 to deliver Media encapsulated in RTP RFC3550 or raw MPEG Transport Stream ("MPEG-TS")-formatted video MPEG-TS, when the media is being delivered in situations such as broadcast and live streaming, that better tolerate occasional packet loss without retransmission. </ins> Under congestion and loss, this approach generally experiences more video artifacts with fewer delay or head-of-line blocking effects. applications like videoconferencing, or for other live-action video with interactive components, such as some sporting events.  <ins> The Secure Reliable Transport protocol SRT also uses UDP in an effort to achieve lower latency for streaming media, although it adds reliability at the application layer. Congestion avoidance strategies for deployments using unreliable transport protocols vary widely in practice, ranging from being entirely unresponsive to congestion, to using feedback signaling to change encoder settings (as in RFC5762), to using fewer enhancement layers (as in RFC6190), to using proprietary methods to detect "quality of experience" issues and turn off video in order to allow less bandwidth-intensive media such as audio to be delivered. More details about congestion avoidance strategies used with unreliable transport protocols are included in udp-behavior. </ins> 5.  <ins> *Note to Reviewers* This section includes some material on UDP and TCP that may be tutorial for some readers.  We can decide how to explain that, if the working group feels that this tutorial material is worth keeping. Spencer thought it was worth including, because it provides a contrast to the material on QUIC, which is significantly less tutorial, unless the reader participated in the QUIC working group. </ins> Because networking resources are shared between users, a good place to start our discussion is how contention between users, and


network operators may be using to troubleshoot HTTP streaming media performance issues, perform analytics, or even intercept exchanges in current applications will not work for QUIC-based applications <del> without making changes to their networks. </del>  While QUIC is designed as a general-purpose transport protocol, and can carry different application-layer protocols, the current and some carriers exploited this information in attempts to improve transport-layer performance RFC3135.  The most recent standardized version of HTTPS, HTTP/3 I-D.ietf-quic-http, uses the QUIC protocol <del> RFC9000 as its transport layer.  QUIC relies only on the TLS 1.3 initial handshake for key exchange RFC9001, and encrypts almost all transport parameters, with the exception of a few invariant fields. In the QUIC short header, the only transport-level parameter which is sent "in the clear" is the destination connection ID RFC8999.  For </del>  these reasons, HTTP/3 is significantly more "opaque" than HTTPS with HTTP/1 or HTTP/2. request and response headers and identify the media content being streamed. <del> "End-to-end" media encryption became much more widespread in the years since RFC7258 was issued, which identified pervasive monitoring </del>  as a much greater threat than previously appreciated.  After the Snowden disclosures, many content providers made the decision to use HTTPS protection - HTTP over TLS - for most or all content being delivered as a routine practice, rather than in exceptional cases for content that was considered "sensitive".  7. This document requires no actions from IANA.

I did add a few paragraphs of text, but don't expect it to be too controversial. Any thoughts?
Needs to be edited and possibly some text migrated.

network operators may be using to troubleshoot HTTP streaming media performance issues, perform analytics, or even intercept exchanges in current applications will not work for QUIC-based applications  <ins> without making changes to their networks. stream-encrypt-media describes the implications of media encryption in more detail. </ins> While QUIC is designed as a general-purpose transport protocol, and can carry different application-layer protocols, the current and some carriers exploited this information in attempts to improve transport-layer performance RFC3135.  The most recent standardized version of HTTPS, HTTP/3 I-D.ietf-quic-http, uses the QUIC protocol  <ins> RFC9000 as its transport layer.  QUIC relies on the TLS 1.3 initial handshake RFC8446 only for key exchange RFC9001, and encrypts almost all transport parameters itself, with the exception of a few invariant header fields.  In the QUIC short header, the only transport-level parameter which is sent "in the clear" is the Destination Connection ID RFC8999, and even in the QUIC long header, the only transport-level parameters sent "in the clear" are the Version, Destination Connection ID, and Source Connection ID.  For </ins> these reasons, HTTP/3 is significantly more "opaque" than HTTPS with HTTP/1 or HTTP/2. request and response headers and identify the media content being streamed.  <ins> "End-to-end" media encryption has become much more widespread in the years since the IETF issued "Pervasive Monitoring Is an Attack" RFC7258 as a Best Current Practice, describing pervasive monitoring </ins> as a much greater threat than previously appreciated.  After the Snowden disclosures, many content providers made the decision to use HTTPS protection - HTTP over TLS - for most or all content being delivered as a routine practice, rather than in exceptional cases for content that was considered "sensitive". <ins> Unfortunately, as noted in RFC7258, there is no way to prevent pervasive monitoring by an "attacker", while allowing monitoring by a more benign entity who "only" wants to use DPI to examine HTTP requests and responses in order to provide a better user experience. If a modern encrypted transport protocol is used for end-to-end media encryption, intermediary streaming operators are unable to examine transport and application protocol behavior.  As described in hop-by- hop-encrypt, only an intermediary streaming operator who is explicitly authorized to examine packet payloads, rather than intercepting packets and examining them without authorization, can continue these practices. RFC7258 said that "The IETF will strive to produce specifications that mitigate pervasive monitoring attacks", so streaming operators should expect the IETF's direction toward preventing unauthorized monitoring of IETF protocols to continue for the forseeable future. </ins> 7. This document requires no actions from IANA.


consider the effects of network design decisions on application-level performance, with considerations for the impact on video delivery. <del> This document aims to provide a taxonomy of networking issues as they relate to quality of experience in internet video delivery.  The focus is on capturing characteristics of video delivery that have surprised network designers or transport experts without specific video expertise, since these highlight key differences between common </del>  assumptions in existing networking documents and observations of video delivery issues in practice. <del> Making specific recommendations for mitigating these issues is out of scope, though some existing mitigations are mentioned in passing. The intent is to provide a point of reference for future solution proposals to use in describing how new technologies address or avoid these existing observed problems. </del>  1.1. 1.1.2. <del> Contributions are solicited regarding issues and considerations that have an impact on media streaming operations. Please note that contributions may be merged and substantially edited, and as a reminder, please carefully consider the Note Well before contributing: https://datatracker.ietf.org/submit/note-well/ [4] Contributions can be emailed to mops@ietf.org, submitted as issues to the issue tracker of the repository in venue, or emailed to the document authors at draft-ietf-mops-streaming-opcons@ietf.org. Contributors describing an issue not yet addressed in the draft are requested to provide the following information, where applicable: a suggested title or name for the issue a long-term pointer to the best reference describing the issue a short description of the nature of the issue and its impact on media quality of service, including: where in the network this issue has root causes who can detect this issue when it occurs an overview of the issue's known prevalence in practice.  pointers to write-ups of high-profile incidents are a plus. a list of known mitigation techniques, with (for each known mitigation): a name for the mitigation technique a long-term pointer to the best reference describing it a short description of the technique: what it does where in the network it operates an overview of the tradeoffs involved-how and why it's helpful, what it costs. supplemental information about the technique's deployment prevalence and status 1.1.3. </del> Presentations: IETF 105 BOF: <del> https://www.youtube.com/watch?v=4G3YBVmn9Eo&t=47m21s [5] </del>  IETF 106 meeting: <del> https://www.youtube.com/watch?v=4_k340xT2jM&t=7m23s [6] </del>  MOPS Interim Meeting 2020-04-15: <del> https://www.youtube.com/watch?v=QExiajdC0IY&t=10m25s [7] </del>  IETF 108 meeting: <del> https://www.youtube.com/watch?v=ZaRsk0y3O9k&t=2m48s [8] </del>  MOPS 2020-10-30 Interim meeting: <del> https://www.youtube.com/watch?v=vDZKspv4LXw&t=17m15s [9] </del>  2. bitrate. Here are a few common resolutions used for video content, with <del> typical ranges of bitrate for the two most popular video codecs </del>  Encodings. 2.1.2. [3] https://mailarchive.ietf.org/arch/browse/mops/ <del> [4] https://datatracker.ietf.org/submit/note-well/ [5] https://www.youtube.com/watch?v=4G3YBVmn9Eo&t=47m21s </del>  <del> [6] https://www.youtube.com/watch?v=4_k340xT2jM&t=7m23s </del>  <del> [7] https://www.youtube.com/watch?v=QExiajdC0IY&t=10m25s </del>  <del> [8] https://www.youtube.com/watch?v=ZaRsk0y3O9k&t=2m48s </del>  <del> [9] https://www.youtube.com/watch?v=vDZKspv4LXw&t=17m15s </del>

Merged after discussion with NAME ...
This was a brillant ideq, and has worked in other groups, but I can't remember any contribution that was made using this template ...
We should probably talk about how to update This document aims to provide a taxonomy of networking issues as they relate to quality of experience in internet video delivery. The focus is on capturing characteristics of video delivery that have surprised network designers or transport experts without specific video expertise, since these highlight key differences between common assumptions in existing networking documents and observations of video delivery issues in practice. NAME might suggest this (but didn't put it in -05): This document focuses is on capturing characteristics of video delivery that have surprised network designers or transport experts without specific video expertise, since these highlight key differences between common assumptions in existing networking documents and observations of video delivery issues in practice. Thoughts from NAME or NAME
Sure, that works for me.

consider the effects of network design decisions on application-level performance, with considerations for the impact on video delivery.  <ins> This document examines networking issues as they relate to quality of experience in internet video delivery.  The focus is on capturing capturing characteristics of video delivery that have surprised network designers or transport experts without specific video expertise, since these highlight key differences between common </ins> assumptions in existing networking documents and observations of video delivery issues in practice.  <ins> Making specific recommendations on operational practices aimed at mitigating these issues is out of scope, though some existing mitigations are mentioned in passing.  The intent is to provide a point of reference for future solution proposals to use in describing how new technologies address or avoid these existing observed problems. </ins> 1.1. 1.1.2.  Presentations: IETF 105 BOF:  <ins> https://www.youtube.com/watch?v=4G3YBVmn9Eo&t=47m21s [4] </ins> IETF 106 meeting:  <ins> https://www.youtube.com/watch?v=4_k340xT2jM&t=7m23s [5] </ins> MOPS Interim Meeting 2020-04-15:  <ins> https://www.youtube.com/watch?v=QExiajdC0IY&t=10m25s [6] </ins> IETF 108 meeting:  <ins> https://www.youtube.com/watch?v=ZaRsk0y3O9k&t=2m48s [7] </ins> MOPS 2020-10-30 Interim meeting:  <ins> https://www.youtube.com/watch?v=vDZKspv4LXw&t=17m15s [8] </ins> 2. bitrate. Here are a few common resolutions used for video content, with  <ins> typical ranges of bitrates for the two most popular video codecs </ins> Encodings. 2.1.2. [3] https://mailarchive.ietf.org/arch/browse/mops/  <ins> [4] https://www.youtube.com/watch?v=4G3YBVmn9Eo&t=47m21s </ins>  <ins> [5] https://www.youtube.com/watch?v=4_k340xT2jM&t=7m23s </ins>  <ins> [6] https://www.youtube.com/watch?v=QExiajdC0IY&t=10m25s </ins>  <ins> [7] https://www.youtube.com/watch?v=ZaRsk0y3O9k&t=2m48s </ins>  <ins> [8] https://www.youtube.com/watch?v=vDZKspv4LXw&t=17m15s </ins>


5. <del> *Note to Reviewers* This section includes some material on UDP and TCP that may be tutorial for some readers.  We can decide how to explain that, if the working group feels that this tutorial material is worth keeping. Spencer thought it was worth including, because it provides a contrast to the material on QUIC, which is significantly less tutorial, unless the reader participated in the QUIC working group. </del> Because networking resources are shared between users, a good place to start our discussion is how contention between users, and mechanisms to resolve that contention in ways that are "fair" between

Remove this material from Sec 5: This section includes some material on UDP and TCP that may be tutorial for some readers. We can decide how to explain that, if the working group feels that this tutorial material is worth keeping. Spencer thought it was worth including, because it provides a contrast to the material on QUIC, which is significantly less tutorial, unless the reader participated in the QUIC working group.

5.  Because networking resources are shared between users, a good place to start our discussion is how contention between users, and mechanisms to resolve that contention in ways that are "fair" between


some metric monitored by the client, such as highest achievable video quality or lowest chances for a rebuffering event (playback stall).  This kind of bandwidth-measurement system can experience trouble in <del> several ways that can be affected by networking design choices. Because adaptive application-level response strategies are typically using application-level protocols, these mechanisms are affected by transport-level protocol behaviors, and the application-level feedback loop is interacting with a transport-level feedback loop, as described in idle-time and hol-blocking. </del>  <del> 4.3.1. </del>  When the bitrate selection is chosen substantially below the available capacity of the network path, the response to a segment TCP slow-start when restarting after idle requires multiple RTTs to re-establish a throughput at the network's available capacity. When the active transmission time for segments is substantially <del> shorter than the time between segments leaving an idle gap between segments that triggers a restart of TCP slow-start, the estimate of the successful download speed coming from the application- visible receive rate on the socket can thus end up much lower than the actual available network capacity, preventing a shift to the most appropriate bitrate.  RFC7661 provides some mitigations for this effect at the TCP transport layer, for senders who anticipate a high incidence of this problem. </del>  Mobile flow-bandwidth spectrum and timing mapping can be impacted by idle time in some networks.  The carrier capacity assigned to a would be achievable with a steadier transmission in the same network. <del> Some receive-side ABR algorithms such as ELASTIC are designed to try to avoid this effect.  Another way to mitigate this effect is by the help of two simultaneous TCP connections is explained in MMSys11 for Microsoft Smooth Streaming.  In some cases, the system-level TCP slow-start restart can be disabled OReilly-HPBN. </del>  <del> 4.3.2. </del>  In the event of a lost packet on a TCP connection with SACK support (a common case for segmented delivery in practice), loss of a packet To a receiver measuring bytes received per unit time at the application layer, and interpreting it as an estimate of the available network bandwidth, this appears as a high jitter in the <del> goodput measurement. </del>  It's worth noting that more modern transport protocols such as QUIC have mitigation of head-of-line blocking as a protocol design goal. See quic-behavior for more details. <del> 4.4. </del>  In addition to measurements media players use to guide their segment- by-segment adaptive streaming requests, streaming media providers may Association (CTA) who owns the Web Application Video Ecosystem (WAVE) project has published two important specifications. <del> 4.4.1. </del>  CTA-2066 specifies a set of media player events, properties, quality of experience (QoE) metrics and associated terminology for defining a common terminology as well as how each metric should be computed for consistent reporting. <del> 4.4.2. </del>  Many assumes that the CDNs have a holistic view into the health and performance of the streaming clients.  However, this is not the case. standardized media-relevant information to the CDNs while they are fetching data.  CTA-5004 was developed exactly for this purpose. <del> 4.5. </del>  In contrast to segmented delivery, several applications use unreliable UDP or SCTP with its "partial reliability" extension

This is in existing text, but I'd put a paragraph break after "Some receive-side ABR algorithms such as {{ELASTIC}} are designed to try to avoid this effect." and before "Another way to mitigate this effect is by the help of two simultaneous TCP connections, as explained in {{MMSys11}} for Microsoft Smooth Streaming. In some cases, the system-level TCP slow-start restart can also be disabled, for example as described in {{OReilly-HPBN}}."
Works for me. Updated...
The editors are 2:1 in favor of dropping ECN. Spencer is graciously in the rough. We should be talking more broadly about what happens when you lose any packets, If ECN belongs in the draft, it belongs somewhere else. "Coming improvements in transport behaviors" more broadly - we're talking about BBR and about end-to-end transport encryption now. Maybe we should talk about things like edge computing, etc ... but NAME will talk to the working group about this before we start a major refactor.
The point of these sections (4.3.1 Idle Time Between Segments and 4.3.2 Head of Line Blocking) was to explain how and why your bandwidth measurements can go wrong when you're trying to do ABR things with your bandwidth measurements. By name this seems like perhaps a better fit in the 4.4 Measurement Collection section, there might be a reason to merge some of the discussion there or reorganize the 2 sections together.
We maybe should also add a section along these lines on the wide variability in bitrates as things move, both for wi-fi and LTE and 5g. (or paths in general, with those as examples?)
Note also: "Refactor" here means provide a coherent story for why your ABR measurements can go wrong and what you can do about it.

some metric monitored by the client, such as highest achievable video quality or lowest chances for a rebuffering event (playback stall). <ins> 4.4. </ins> This kind of bandwidth-measurement system can experience trouble in  <ins> several ways that are affected by networking issues.  Because adaptive application-level response strategies are often using rates as observed by the application layer, there are sometimes inscrutable transport-level protocol behaviors that can produce surprising measurement values when the application-level feedback loop is interacting with a transport-level feedback loop. A few specific examples of surprising phenomena that affect bitrate detection measurements are described in the following subsections. As these examples will demonstrate, it's common to encounter cases that can deliver application level measurements that are too low, too high, and (possibly) correct but varying more quickly than a lab- tested selection algorithm might expect. These effects and others that cause transport behavior to diverge from lab modeling can sometimes have a significant impact on ABR bitrate selection and on user quality of experience, especially where players use naive measurement strategies and selection algorithms that don't account for the likelihood of bandwidth measurements that diverge from the true path capacity. </ins>  <ins> 4.4.1. </ins> When the bitrate selection is chosen substantially below the available capacity of the network path, the response to a segment TCP slow-start when restarting after idle requires multiple RTTs to re-establish a throughput at the network's available capacity. When the active transmission time for segments is substantially  <ins> shorter than the time between segments, leaving an idle gap between segments that triggers a restart of TCP slow-start, the estimate of the successful download speed coming from the application-visible receive rate on the socket can thus end up much lower than the actual available network capacity.  This in turn can prevent a shift to the most appropriate bitrate.  RFC7661 provides some mitigations for this effect at the TCP transport layer, for senders who anticipate a high incidence of this problem. </ins> Mobile flow-bandwidth spectrum and timing mapping can be impacted by idle time in some networks.  The carrier capacity assigned to a would be achievable with a steadier transmission in the same network.  <ins> Some receiver-side ABR algorithms such as ELASTIC are designed to try to avoid this effect. </ins>  <ins> Another way to mitigate this effect is by the help of two simultaneous TCP connections, as explained in MMSys11 for Microsoft Smooth Streaming.  In some cases, the system-level TCP slow-start restart can also be disabled, for example as described in OReilly- HPBN. 4.4.2. </ins> In the event of a lost packet on a TCP connection with SACK support (a common case for segmented delivery in practice), loss of a packet To a receiver measuring bytes received per unit time at the application layer, and interpreting it as an estimate of the available network bandwidth, this appears as a high jitter in the  <ins> goodput measurement.  This can appear as a stall of some time, followed by a sudden leap that can far exceed the actual capacity of the transport path from the server when the hole in the received data is filled by a later retransmission. </ins> It's worth noting that more modern transport protocols such as QUIC have mitigation of head-of-line blocking as a protocol design goal. See quic-behavior for more details.  <ins> 4.4.3. As many end devices have moved to wireless connectivity for the final hop (Wi-Fi, 5G, or LTE), new problems in bandwidth detction have emerged from radio interference and signal strength effects. Each of these technologies can experience sudden changes in capacity as the end user device moves from place to place and encounters new sources of interference.  Microwave ovens, for example, can cause a throughput degradation of more than a factor of 2 while active Micro. 5G and LTE likewise can easily see rate variation by a factor of 2 or more over a span of seconds as users move around. These swings in actual transport capacity can result in user experience issues that can be exacerbated by insufficiently responsive ABR algorithms. 4.5. </ins> In addition to measurements media players use to guide their segment- by-segment adaptive streaming requests, streaming media providers may Association (CTA) who owns the Web Application Video Ecosystem (WAVE) project has published two important specifications.  <ins> 4.5.1. </ins> CTA-2066 specifies a set of media player events, properties, quality of experience (QoE) metrics and associated terminology for defining a common terminology as well as how each metric should be computed for consistent reporting.  <ins> 4.5.2. </ins> Many assumes that the CDNs have a holistic view into the health and performance of the streaming clients.  However, this is not the case. standardized media-relevant information to the CDNs while they are fetching data.  CTA-5004 was developed exactly for this purpose.  <ins> 4.6. </ins> In contrast to segmented delivery, several applications use unreliable UDP or SCTP with its "partial reliability" extension


this is relatively routine, but in other cases, it is more difficult (sec-unpredict, sec-extreme).  2.4. Historical data shows that users consume more video and videos at

…cy streaming)
This reads nicely to me. I'm not asking about this for today (it's I-D cutoff day, after all), but is there any reference that could be provided for the user to learn more?
Not really, I think I just know a guy who knows a guy who says this happened or something...
From Will - Caching Systems May be worth mentioning that concurrent gaming downloads nowadays drive larger traffic peaks than live or VOD streams. Additionally, something missing from this section is the impact of chunk transfer of aggregating responses. These are being used increasingly for low latency delivery. The caching system is receiving the object in real-time while simultaneously forwarding it to the client. The object size is not known until the transmission is complete. Much of caching infrastructure failover and redundancy is predicated on knowing the size of the object you are transmitting. Distribution tiers have a hard time separating a gap in data from a failure of an upstream component. Setting appropriate time-outs becomes problematic.
This is a true and fine point, but I'm not sure it needs mention in a video-centric doc, except maybe how this impacts video delivery from the shared resources, but I think the congestion control sort of takes that. I'll give it some thought, but I don't think it's really specific to caching or what we get out of raising it here, so I'm leaning toward not doing anything in this section on this point.
I'll follow up with a separate question about the other half of this comment, I'm not quite sure what it's saying. But it's a good point that talking about some non-intuitive failure modes might be handy here.
Hi, NAME - any update here?
Not yet. I did send a message hoping to learn enough about the failure modes and what you can do about them to write something meaningful here, but there are some vacation-related delays and once people are back I may have to ping again.

this is relatively routine, but in other cases, it is more difficult (sec-unpredict, sec-extreme). <ins> And as with other parts of the ecosystem, new technology brings new challenges.  For example, with the emergence of ultra-low-latency streaming, responses have to start streaming to the end user while still being transmitted to the cache, and while the cache does not yet know the size of the object.  Some of the popular caching systems were designed around cache footprint and had deeply ingrained assumptions about knowing the size of objects that are being stored, so the change in design requirements in long-established systems caused some errors in production.  Incidents occurred where a transmission error in the connection from the upstream source to the cache could result in the cache holding a truncated segment and transmitting it to the end user's device.  In this case, players rendering the stream often had the video freeze until the player was reset.  In some cases the truncated object was even cached that way and served later to other players as well, causing continued stalls at the same spot in the video for all players playing the segment delivered from that cache node. </ins> 2.4. Historical data shows that users consume more video and videos at


jump over the years due to increased consumption of entertainment as well as conversational video. <del> TBD: insert charts showing historical relative data usage patterns with error bars by time of day in consumer networks? TBD: Cross-ref vs. video quality by time of day in practice for some case study?  Not sure if there's a good way to capture a generalized insight here, but it seems worth making the point that demand projections can be used to help with e.g. power consumption with routing architectures that provide for modular scalability. </del> 2.5. Although TCP/IP has been used with a number of widely used

I don't remember where this suggestion came from, but it was on the list in the 107 interim slides, so I figured it should be tracked: URL
Maybe refer to Cisco VNI? URL
Cisco VNI link is already a 404, so ... Checking with RFC Editor to see whether they can handle SVG yet.
The point of this is trying to show internet scale growth of video traffic over time, both by proportion and by gross volume, not really about including pictures. This maybe is already covered by our intro paragraph: "Estimates put the total share of internet video traffic at 75% in 2019, expected to grow to 82% by 2022. This estimate projects the gross volume of video traffic will more than double during this time, based on a compound annual growth rate continuing at 34% (from Appendix D of [CVNI])." We should consider just closing this issue.
Related point (perhaps a new issue?): this info may stop being available--prior reports often came from DPI at IXPs or similar to monitor traffic, but since the dawn of ubiquitous encryption this info will stop being available, and we'll (probably?) only have reports from servers.
We're thinking this is Won't Fix - at a minimum, this isn't information we can update in the future.
Pulled these out of the text of the draft as part of this issue, from the bottom of section "2.4 Predictable Usage Profiles": TBD: insert charts showing historical relative data usage patterns with error bars by time of day in consumer networks? TBD: Cross-ref vs. video quality by time of day in practice for some case study? Not sure if there's a good way to capture a generalized insight here, but it seems worth making the point that demand projections can be used to help with e.g. power consumption with routing architectures that provide for modular scalability.

jump over the years due to increased consumption of entertainment as well as conversational video.  2.5. Although TCP/IP has been used with a number of widely used


4.1.  Adaptive BitRate (ABR) is a sort of application-level response strategy in which the streaming client attempts to detect the available bandwidth of the network path by observing the successful 4.4.  This kind of bandwidth-measurement system can experience trouble in several ways that are affected by networking issues.  Because adaptive application-level response strategies are often using rates that don't account for the likelihood of bandwidth measurements that diverge from the true path capacity. <del> 4.4.1. </del>  When the bitrate selection is chosen substantially below the available capacity of the network path, the response to a segment restart can also be disabled, for example as described in OReilly- HPBN. <del> 4.4.2. </del>  In the event of a lost packet on a TCP connection with SACK support (a common case for segmented delivery in practice), loss of a packet have mitigation of head-of-line blocking as a protocol design goal. See quic-behavior for more details. <del> 4.4.3. </del>  As many end devices have moved to wireless connectivity for the final hop (Wi-Fi, 5G, or LTE), new problems in bandwidth detction have experience issues that can be exacerbated by insufficiently responsive ABR algorithms. <del> 4.5. </del>  In addition to measurements media players use to guide their segment- by-segment adaptive streaming requests, streaming media providers may Association (CTA) who owns the Web Application Video Ecosystem (WAVE) project has published two important specifications. <del> 4.5.1. </del>  CTA-2066 specifies a set of media player events, properties, quality of experience (QoE) metrics and associated terminology for defining a common terminology as well as how each metric should be computed for consistent reporting. <del> 4.5.2. </del>  Many assumes that the CDNs have a holistic view into the health and performance of the streaming clients.  However, this is not the case. standardized media-relevant information to the CDNs while they are fetching data.  CTA-5004 was developed exactly for this purpose. <del> 4.6. </del>  In contrast to segmented delivery, several applications use unreliable UDP or SCTP with its "partial reliability" extension

Sorry for the delay on this! Here's a rebased copy of URL we can work from if there's still in adding this section.
NAME my apologies on this - I told NAME and NAME I'd work on this, just before coming down with the plague. Thank you for rebasing the proposal, and yes, we are still interested in adding this section (speaking for the editors, not for the working group). We're in the I-D submission blackout zone now, so let's continue discussion, and we can submit something after IETF 111 opens back up.
I had a few comments in the prior pr, but I'm not sure I quite got across what I meant. I drafted an alternative proposal as a quick straw man example of what I meant, maybe we could do some merger that includes some content along these lines? I was also thinking to put it in the ABR section instead of the bandwidth section, probably. Not sure if this goes too far?
NAME now that we're using Github Actions, it looks like "First-time contributors need a maintainer to approve running workflows", and we actually caught problems when using CircleCI, so we probably want to do that, don't we? If so - who are the maintainers who should be approving this? I'm guessing that this has something to do with URL, but of course, I'm not sure ...
NAME the current slide deck for next week's MOPS session has the issue associated with this PR as our last blocking issue before WGLC, so if we (you, me, NAME NAME can close on this before the meeting, we can update the slide deck and tell the WG that we're just cleaning up for a couple of weeks after IETF 111, and then going forward. Does that make sense to you?
NAME That makes sense to me! I will, unfortunately, not be able to attend IETF 111 due to a previously scheduled commitment, but I will attempt to integrate NAME feedback here beforehand so you can bring that message to the WG session.
Sorry for the late push! We've had even less Internet connectivity than I had expected. Let me know what you think of this synthesis, and whether a section on segmentation and keyframes is worth including. Thanks!
NAME I agree that this probably needs another editorial pass to deduplicate some of the merged text and tighten up the style, but I've tried to clean up all the other loose ends so you can merge this if/when you're ready. Let me know if you need me to make any additional changes in this PR. Thanks!
Thanks Mike, this is fantastic. Much obliged. We'll discuss in the next editor's session, and likely at least one of us will make an editorial pass before long, I expect. (NAME I believe you're still the owner--want to take a final pass? I think another set of eyes would be invaluable here, and your judgement on these matters is impeccable, IMHO...)
NAME has questionable judgement about "impeccable taste", but I should be able to do that ...
Since we're going to be doing a final editorial pass anyway, I'm merging this PR and closing the issue now. I don't see us making significant changes to the current proposed text - as Jake says, it looks quite good. Thanks, NAME for your work on this!
Originally proposed by Matt Stock here: URL
We THINK this is different from general DNS performance impacts, when you do client side ad insertion for URLs that the media provider is handing you based on the specific customer who is looking at the media. The actual problem is probably media player-specific, but but we can probably say something useful in a sentence or two.
BTW: comments are welcomed on whether or not we should acknowledge the existence of DNS-based ad-blockers and their relevance to this issue in this document.
So, now that I've slept on this (sort of), I was leaning toward adding at least a sentence or two about DNS-based ad-blockers, because that was a topic I THOUGHT I understood, but apparently not in much depth. ISTM that the question I SHOULD have asked, is "that's a fascinating question. Is there anything that a media provider can do differently, if we describe the issue?" ISTM that if what we say would be actionable for the designated audience, it makes sense to say it in the draft.
If a player implementer is aware of the potential for trouble from a change in hostname for a stream, they could run that request a little early as a mitigation to give time for the DNS to settle out, or to do any kind of pre-request to an upcoming non-recently-used hostname or something. Also the operator could chose to run server-side ads instead. (That might not be an exhaustive list, but it's at least a couple examples.) Maybe also worth mentioning it can be more of an issue if you're running a tight latency budget. But yeah, I think it's at least somewhat actionable for operators if the issue is on their radar.
Conclusion way back when was that we'd add text if we got to it with a deadline that we missed long ago. So this is won'tfix.
Describe in general terms segments and frame types, tradeoffs associated with segment size and keyframe interval selection. (related to idle-time and caching) Thanks NAME for the suggestion as part of URL
This is about low-latency and where you can start rendering in a media stream. Mention and explain briefly? This isn't quite about the NETWORK, but it's about caching, which is probably in the network. Keyframes can also be used for thumbnails when you hover over something. More keyframes gives you more flexibility in where you can start rendering, but is also larger and comes with a bandwidth and path occupancy cost.
Insert this material in "For further reading" (}?
This is now addressed in the reading list section. Issue can be closed.
My thought here was that stream characteristics like GOP sizing place a constraint on switching opportunities for renditions (ABR) and ad insertion, which can have an impact on bandwidth utilization, etc. I thought some explanatory text might help pull those sections together a bit more, but I'm fine with it just being covered in the further reading, too.
It is not very easy to correctly discuss the trade-offs between the key frame internal, segment size, encoding quality, impact on the storage, request rate and all that. There are detailed studies that do an excellent job on explaining these trade-offs, so referring the readers to these studies is better IMO.
Closing this issue as the further reading section identifies the relevant studies.
Advertising is an important part of the video delivery landscape and has an impact on network design and operational considerations in streaming video, for sure. NAME gave a pull request that's a decent start here: URL There are some unanswered comments in that pull request, but we should consider adding such a section and refining it even in the absence of responses.
Ads are evil :) Could the IETF ban them?
NAME will review proposed text in PR

4.1. <ins> A simple model of video playback can be described as a video stream consumer, a buffer, and a transport mechanism that fills the buffer. The consumption rate is fairly static and is represented by the content bitrate.  The size of the buffer is also commonly a fixed size.  The fill process needs to be at least fast enough to ensure that the buffer is never empty, however it also can have significant complexity when things like personalization or ad workflows are introduced. The challenges in filling the buffer in a timely way fall into two broad categories: 1. content selection and 2. content variation. Content selection comprises all of the steps needed to determine which content variation to offer the client.  Content variation is the number of content options that exist at any given selection point.  A common example, easily visualized, is Adaptive BitRate (ABR), described in more detail below.  The mechanism used to select the bitrate is part of the content selection, and the content variation are all of the different bitrate renditions. </ins> Adaptive BitRate (ABR) is a sort of application-level response strategy in which the streaming client attempts to detect the available bandwidth of the network path by observing the successful 4.4. <ins> A variety of business models exist for producers of streaming media. Some content providers derive the majority of the revenue associated with streaming media directly from consumer subscriptions or one-time purchases.  Others derive the majority of their streaming media associated revenue from advertising.  Many content providers derive income from a mix of these and other sources of funding.  The inclusion of advertising alongside or interspersed with streaming media content is therefore common in today's media landscape. Some commonly used forms of advertising can introduce potential user experience issues for a media stream.  This section provides a very brief overview of a complex and evolving space, but a complete coverage of the potential issues is out of scope for this document. The same techniques used to allow a media player to switch between renditions of different bitrates at segment or chunk boundaries can also be used to enable the dynamic insertion of advertisements. Ads may be inserted either with Client Side Ad Insertion (CSAI) or Server Side Ad Insertion (SSAI).  In CSAI, the ABR manifest will generally include links to an external ad server for some segments of the media stream, while in SSAI the server will remain the same during advertisements, but will include media segments that contain the advertising.  In SSAI, the media segments may or may not be sourced from an external ad server like with CSAI. In general, the more targeted the ad request is, the more requests the ad service needs to be able to handle concurrently.  If connectivity is poor to the ad service, this can cause rebuffering even if the underlying video assets (both content and ads) are able to be accessed quickly.  The less targeted, the more likely the ad requests can be consolidated and can leverage the same caching techniques as the video content. In some cases, especially with SSAI, advertising space in a stream is reserved for a specific advertiser and can be integrated with the video so that the segments share the same encoding properties such as bitrate, dynamic range, and resolution.  However, in many cases ad servers integrate with a Supply Side Platform (SSP) that offers advertising space in real-time auctions via an Ad Exchange, with bids for the advertising space coming from Demand Side Platforms (DSPs) that collect money from advertisers for delivering the advertisements.  Most such Ad Exchanges use application-level protocol specifications published by the Interactive Advertising Bureau IAB-ADS, an industry trade organization. This ecosystem balances several competing objectives, and integrating with it naively can produce surprising user experience results.  For example, ad server provisioning and/or the bitrate of the ad segments might be different from that of the main video, either of which can sometimes result in video stalls.  For another example, since the inserted ads are often produced independently they might have a different base volume level than the main video, which can make for a jarring user experience. Additionally, this market historically has had incidents of ad fraud (misreporting of ad delivery to end users for financial gain).  As a mitigation for concerns driven by those incidents, some SSPs have required the use of players with features like reporting of ad delivery, or providing information that can be used for user tracking.  Some of these and other measures have raised privacy concerns for end users. In general this is a rapidly developing space with many considerations, and media streaming operators engaged in advertising may need to research these and other concerns to find solutions that meet their user experience, user privacy, and financial goals.  For further reading on mitigations, BAP has published some standards and best practices based on user experience research. 4.5. </ins> This kind of bandwidth-measurement system can experience trouble in several ways that are affected by networking issues.  Because adaptive application-level response strategies are often using rates that don't account for the likelihood of bandwidth measurements that diverge from the true path capacity.  <ins> 4.5.1. </ins> When the bitrate selection is chosen substantially below the available capacity of the network path, the response to a segment restart can also be disabled, for example as described in OReilly- HPBN.  <ins> 4.5.2. </ins> In the event of a lost packet on a TCP connection with SACK support (a common case for segmented delivery in practice), loss of a packet have mitigation of head-of-line blocking as a protocol design goal. See quic-behavior for more details.  <ins> 4.5.3. </ins> As many end devices have moved to wireless connectivity for the final hop (Wi-Fi, 5G, or LTE), new problems in bandwidth detction have experience issues that can be exacerbated by insufficiently responsive ABR algorithms.  <ins> 4.6. </ins> In addition to measurements media players use to guide their segment- by-segment adaptive streaming requests, streaming media providers may Association (CTA) who owns the Web Application Video Ecosystem (WAVE) project has published two important specifications.  <ins> 4.6.1. </ins> CTA-2066 specifies a set of media player events, properties, quality of experience (QoE) metrics and associated terminology for defining a common terminology as well as how each metric should be computed for consistent reporting.  <ins> 4.6.2. </ins> Many assumes that the CDNs have a holistic view into the health and performance of the streaming clients.  However, this is not the case. standardized media-relevant information to the CDNs while they are fetching data.  CTA-5004 was developed exactly for this purpose.  <ins> 4.7. </ins> In contrast to segmented delivery, several applications use unreliable UDP or SCTP with its "partial reliability" extension


2.2.  The bitrate requirements in scaling are per end-user actively consuming a media feed, so in the worst case, the bitrate demands can be multiplied by the number of simultaneous users to find the bandwidth requirements at peering or ingest points can be reduced to as low as a per-feed requirement instead of a per-user requirement. <del> 2.3. </del>  When demand for content is relatively predictable, and especially when that content is relatively static, caching content close to at the same spot in the video for all players playing the segment delivered from that cache node. <del> 2.4. </del>  Historical data shows that users consume more video and videos at higher bitrates than they did in the past on their connected devices. jump over the years due to increased consumption of entertainment as well as conversational video. <del> 2.5. </del>  Although TCP/IP has been used with a number of widely used applications that have symmetric bandwidth requirements (similar watch for increasing numbers of end users uploading significant amounts of content. <del> 2.6. </del>  The causes of unpredictable usage described in sec-unpredict were more or less the result of human choices, but we were reminded during RFCs have been written refining TCP decisions about what to send, how much to send, and when to send it, since 1988 Jacobson-Karels the signals available for TCP senders remained unchanged - end-to-end <del> acknowledgments for packets that were successfully sent and received, and packet timeouts for packets that were not. </del>  The success of the largely TCP-based Internet is evidence that the mechanisms TCP used to achieve equilibrium quickly, at a point where

This definitely needs more work!
NAME and NAME - I have three notes in the section I've added. I'll hit this again later in the week, but if you can express opinions about any of them, that would be very helpful.
NAME In the call we discussed this and we feel like we don't need to say anything about this in addition to what we already said in Section titled "Bitrate Detection Challenges". So, maybe we can drop and close this?
I actually like NAME suggestions - let me spend a couple of hours on this, and then you two can either merge it or reject it.
NAME and NAME see what you think. I'm also ok with closing as wontfix, but wanted to give it a try.
NAME and NAME any thoughts on this PR? I tried to reflect Jake's high-level comments, which were even more helpful and actionable than usual ...
This reads much better than the initial version, thanks.
Pulling the TBD out of the draft text and capturing it here instead. Suggestion from mic at IETF 106 (Mark Nottingham) in response to call for help on capturing the mismatch between supply and demand of bandwidth with regard to video: Dive into the different constraints coming from different parts of the network or distribution channels. (regarding questions about how to describe the disconnect between demand vs. capacity, while keeping good archival value.) "These are big numbers, but the internet has grown over time, and CDN capacities have grown over time. Is it last mile? Probably not--Is it middle mile? Maybe that's more interesting. It it CPU? Is it network cards? Is it the economics of your peering relationships? Unpicking those bits might help a bit. URL
For many, poor Wifi is the culprit. For many others, the ISP aggregation links are the culprit during the prime times (since most such links are oversubscribed/oversold). For some, the CDN issues create problems and for rare cases, the origin drowns and everybody gets upset. For certain environments like student dorms, campus networks, libraries, airport cafes, it is just lack of bandwidth for the power users.
Thanks, this version reads much better to me. I have no objections.

2.2. <ins> Even when the bandwidth requirements for video streams along a path are well understood, additional analysis is required to understand the contraints on bandwidth at various points in the network.  This analysis is necessary because media servers may react to bandwith constraints using two independent feedback loops: Media servers often respond to application-level feedback from the media player that indicates a bottleneck link somewhere along the path, by adjusting the amount of media that the media server will send to the media player in a given timeframe.  This is described in greater detail in sec-abr. Media servers also typically implement transport protocols with capacity-seeking congestion controllers that probe for bandwidth, and adjust the sending rate based on transport mechanisms.  This is described in greater detail in sec-trans. The result is that these two (potentially competing) "helpful" mechanisms each respond to the same bottleneck with no coordination between themselves, so that each is unaware of actions taken by the other, and this can result in a quality of experience for users that is significantly lower than what could have been achieved. In one example, if a media server overestimates the available handwidth to the media player, the transport protocol detects loss due to congestion, and reduces its sending window size per round trip, the media server adapts to application-level feedback from the media player, and reduces its own sending rate, the transport protocol sends media at the new, lower rate, and confirms that this new, lower rate is "safe", because no transport-level loss is occuring, but because the media server continues to send at the new, lower rate, the transport protocol's maximum sending rate is now limited by the amount of information the media server queues for transmission, so the transport protocol can't probe for available path bandwidth by sending at a higher rate. In order to avoid these types of situations, which can potentially affect all the users whose streaming media traverses a bottleneck link, there are several possible mitigations that streaming operators can use, but the first step toward mitigating a problem is knowing when that problem occurs. 2.2.1. There are many reasons why path characteristics might change suddenly, for example, "cross traffic" that traverses part of the path, especially if this traffic is "inelastic", and does not, itself, respond to indications of path congestion. routing changes, which can happen in normal operation, especially if the new path now includes path segments that are more heavily loaded, offer lower total bandwidth, or simply cover more distance. Recognizing that a path carrying streaming media is "not behaving the way it normally does" is fundamental.  Analytics that aid in that recognition can be more or less sophisticated, and can be as simple as noticing that the apparent round trip times for media traffic carried over TCP transport on some paths are suddenly and significantly longer than usual.  Passive monitors can detect changes in the elapsed time between the acknowledgements for specific TCP segments from a TCP receiver, since TCP octet sequence numbers and acknowledgements for those sequence numbers are "carried in the clear", even if the TCP payload itself is encrypted.  See tcp- behavior for more information. As transport protocols evolve to encrypt their transport header fields, one side effect of increasing encryption is that the kind of passive monitoring, or even "performance enhancement" (RFC3135) that was possible with the older transport protocols (UDP, described in udp-behavior and TCP, described in tcp-behavior) is no longer possible with newer transport protocols such as QUIC (described in quic-behavior).  The IETF has specified a "latency spin bit" mechanism in Section 17.4 of RFC9000 to allow passive latency monitoring from observation points on the network path throughout the duration of a connection, but currently chartered work in the IETF is focusing on end-point monitoring and reporting, rather than on passive monitoring. One example is the "qlog" mechanism I-D.ietf-quic-qlog-main-schema, a protocol-agnostic mechanism used to provide better visibility for encrypted protocols such as QUIC (I-D.ietf-quic-qlog-quic-events) and for HTTP/3 (I-D.ietf-quic-qlog-h3-events). 2.3. </ins> The bitrate requirements in scaling are per end-user actively consuming a media feed, so in the worst case, the bitrate demands can be multiplied by the number of simultaneous users to find the bandwidth requirements at peering or ingest points can be reduced to as low as a per-feed requirement instead of a per-user requirement.  <ins> 2.4. </ins> When demand for content is relatively predictable, and especially when that content is relatively static, caching content close to at the same spot in the video for all players playing the segment delivered from that cache node.  <ins> 2.5. </ins> Historical data shows that users consume more video and videos at higher bitrates than they did in the past on their connected devices. jump over the years due to increased consumption of entertainment as well as conversational video.  <ins> 2.6. </ins> Although TCP/IP has been used with a number of widely used applications that have symmetric bandwidth requirements (similar watch for increasing numbers of end users uploading significant amounts of content.  <ins> 2.7. </ins> The causes of unpredictable usage described in sec-unpredict were more or less the result of human choices, but we were reminded during RFCs have been written refining TCP decisions about what to send, how much to send, and when to send it, since 1988 Jacobson-Karels the signals available for TCP senders remained unchanged - end-to-end  <ins> acknowledgements for packets that were successfully sent and received, and packet timeouts for packets that were not. </ins> The success of the largely TCP-based Internet is evidence that the mechanisms TCP used to achieve equilibrium quickly, at a point where