A Firmware Update Architecture for Internet of Things Devices <del> draft-ietf-suit-architecture-00 </del>  Abstract may be to fix bugs in software, to add new functionality, or to re- configure the device. <del> The firmware update process has to ensure that </del>  The firmware image is authenticated and attempts to flash a malicious firmware image are prevented. The following entities are used: <del> Author: The author is the entity that creates the firmware image, signs and/or encrypts it.  The author is most likely a developer using a set of tools. </del>  Device: The device is the recipient of the firmware image and the <del> manifest.  The goal is to update the firmware of the device. Untrusted Storage: Firmware images and manifests may be stored on untrusted fileservers or cloud storage infrastructure.  Some deployments may require storage of the firmware images/manifests to be stored on various entities before they reach the device. </del>  3. 3.1. Firmware images can be conveyed to devices in a variety of ways, <del> including USB, UART, WiFi, BLE, low-power WAN technologies, etc. and </del>  use different protocols (e.g., CoAP, HTTP).  The specified mechanism needs to be agnostic to the distribution of the firmware images and manifests. 3.9. <del> A device may have many modules that require updating individually. It may also need to trust several actors in order to authorize an update.  These actors might include the following (this is not a comprehensive list). These actors exert their authority on the device by making claims (as in claims). </del>  <del> For example, a firmware author may not have the authority to install firmware on a device in critical infrastructure without the authorization of a device operator.  In this case, the device may be programmed to reject firmware updates unless they are signed both by the firmware author and by the device operator.  To facilitate complex use-cases such as this, updates require several claims. </del>  <del> Alternatively, a device may trust precisely one authority, which does all permission management and coordination.  Effectively, the authority allows the device to offload complex permissions calculations for the device. </del>  3.10. There are three broad classifications of update operating modes. <del> Self initiated updates take the form of a proactive IoT device that checks for updates.  Third-party initiated updates are triggered by an actor other than the IoT device, be it a server, a peer, or a user.  Hybrid updates are those that require agreement from both the target IoT device and another actor. Third-party initiated updates are important to consider because timing of updates may need to be tightly controlled in some high- reliability environments. An IoT device goes through several steps in the course of an update, each of which can be self-initiated or third-party initiated, or hybrid.  An IoT device may go through the following steps, though this is not a comprehensive list. The notification step consists informing an IoT device that an update is available.  This can be accomplished via polling (self-initiated), push notifications (third-party initiated), or more complex mechanisms. The pre-authorisation step involves verifying the update authority and making a determination that the device is prepared to initiate the fetching and processing of updates.  If the device has all information that is necessary to make this determination, then the pre-authorisation may be self-initiated.  However, the device can wait for instruction to begin (third-party initiated).  Hybrid approaches are possible as well. </del>  A dependency resolution phase is needed when more than one component can be updated or when a differential update is used.  The necessary dependencies must be available prior to installation. The download step is the process of acquiring a local copy of the <del> payload.  When the download is self-initiated, this means that the IoT device chooses when a download occurs and initiates the download process.  When a download is third-party initiated, this means that either the remote service tells the IoT device when to download or that it initiates the transfer directly to the IoT device.  For example, a download from an HTTP server is initiated locally.  A transfer to a LwM2M Firmware Update resource LwM2M is initiated remotely. </del>  Installation is the act of processing the payload into a format that <del> the IoT device can recognise. </del>  <del> Each of these steps may require different permissions expressed in claims and may be implemented in a variety of ways. </del>  4. <del> When a simple set of permissions fails to encapsulate the rules required for a device to make decisions about firmware, claims can be used instead.  Claims represent a form of policy.  Several claims can be used together, when multiple actors should have the rights to set policies. Some example claims are: </del>  <del> Trust the actor identified by the referenced public key. Trust the actor with access to the referenced shared secret (MAC). Three actors are trusted identified by their public keys. Signatures from at least two of these actors are required to trust a manifest. The actor identified by the referenced public key is authorized to create secondary policies The baseline claims for all manifests are described in SUIT-IM.  In summary, they are: </del>  Do not install firmware with earlier metadata than the current metadata. Only install firmware that is before its best-before timestamp. <del> Only install firmware with metadata signed/authenticated by a trusted actor. Only allow an actor to exercise rights on the device via a manifest if that actor has signed the manifest. Only allow a firmware installation if all required rights have been met through signatures/MACs (one or more) or manifest dependencies (one or more). Use the instructions provided by the manifest to install the firmware. Install any and all firmware images that are linked together with manifest dependencies. </del>  Choose the mechanism to install the firmware, based on the type of firmware it is. 5. <del> We start the architectural description with the security model.  It is based on end-to-end security.  In e2e-figure a firmware image is created by an author, sent to the device and subsequently installed. When the author is ready to distribute the firmware image it is conveyed using some communication channel to the device, which will typically involve the use of untrusted storage.  Examples of untrusted storage are FTP servers, Web servers or USB sticks.  End- to-end security mechanisms are used to protect the firmware image. e2e-figure does not show the manifest itself, which provides the meta-data about the firmware image and offers the security protection.  It may bundled with the firmware image or travel as a standalone item. </del>  Whether the firmware image and the manifest is pushed to the device <del> or fetched by the device is outside the scope of this work and existing device management protocols can be used for efficiently distributing this information. </del>  The following assumptions are made to allow the device to verify the received firmware image and manifest before updating software: <del> To accept an update, a device needs to decide whether the author signing the firmware image and the manifest is authorized to make the updates.  We use public key cryptography to accomplish this. The device verifies the signature covering the manifest using a digital signature algorithm OR the device verifies the MAC covering the manifest using a MAC algorithm.  The device is provisioned with a trust anchor that is used to validate the digital signature or MAC produced by the author.  This trust anchor is potentially different from the trust anchor used to validate the digital signature produced for other protocols (such as device management protocols).  This trust anchor may be provisioned to the device during manufacturing or during commissioning. </del>  For confidentiality protection of firmware images the author needs to be in possession of the certificate/public key or a pre-shared <del> key of a device. </del>  There are different types of delivery modes, which are illustrated based on examples below. cryptographic information, such as digital signatures or message authentication codes (MACs). <del> The manifest information model is described in SUIT-IM. </del>  7.  The following example message flow illustrates the interaction for distributing a firmware image to a device starting with an author <del> uploading the new firmware to untrusted storage and creating a manifest.  The firmware and manifest are stored on the same untrusted storage. </del>  <del> 8. </del>  This document does not require any actions by IANA. <del> 9. </del>  Firmware updates fix security vulnerabilities and are considered to be an important building block in securing IoT devices.  Due to the incentives for manufacturers to offer a firmware update mechanism as part of their IoT products. <del> 10. </del>  The discussion list for this document is located at the e-mail address suit@ietf.org [1].  Information on the group and information Archives of the list can be found at: https://www.ietf.org/mail- archive/web/suit/current/index.html [3] <del> 11.  References </del>  <del> 11.1.  URIs </del>  [1] mailto:suit@ietf.org

Added David Brown as co-author Added further terms Updated text about permissions Re-wrote text about operating modes Editorial changes (including formatting) Revised section about claims Added new architecture figure Reworked security description Included device firmware update examples provided by David. Updated example flow (and included bootloader)

A Firmware Update Architecture for Internet of Things Devices  <ins> draft-ietf-suit-architecture-01 </ins> Abstract may be to fix bugs in software, to add new functionality, or to re- configure the device.  <ins> The firmware update process, among other goals, has to ensure that </ins> The firmware image is authenticated and attempts to flash a malicious firmware image are prevented. The following entities are used:  <ins> Author: The author is the entity that creates the firmware image. There may be multiple authors in a system either when a device consists of multiple micro-controllers or when the the final firmware image consists of software components from multiple companies. </ins> Device: The device is the recipient of the firmware image and the  <ins> manifest.  The goal is to update the firmware of the device.  A single device may need to obtain more than one firmware image and manifest to succesfully perform an update. Communicator: The communicator component of the device interacts with the firmware update server.  It receives firmware images and triggers an update, if needed.  The communicator either polls a firmware update server for the most recent manifest/firmware or manifests/firmware images are pushed to it.  Note that the firmware update process may involve multiple stages since one or multiple manifests may need to be downloaded before the communicator can fetch one or multiple firmware images/software components. Status Tracker: The status tracker offers device management functionality that includes keep track of the firmware update process.  This includes fine-grained monitoring of changes at the device, for example, what state of the firmware update cycle the device is currently in. Firmware Server: Entity that stores firmware images and manifests. Some deployments may require storage of the firmware images/ manifests on more than one entities before they reach the device. Device Operator: The actor responsible for the day-to-day operation of a fleet of IoT devices. Network Operator: The actor responsible for the operation of a network to which IoT devices connect. In addition to the entities in the list above there is an orthogonal infrastructure with a Trust Provisioning Authority (TPA) distributing trust anchors and authorization permissions to various entities in the system.  The TPA may also delegate rights to install, update, enhance, or delete trust anchors and authorization permissions to other parties in the system.  This infrastructure overlaps the communication architecture and different deployments may empower certain entities while other deployments may not.  For example, in some cases, the Original Design Manufacturer (ODM), which is a company that designs and manufactures a product, may act as a TPA and may decide to remain in full control over the firmware update process of their products. The terms 'trust anchor' and 'trust anchor store' are defined in RFC6024: "A trust anchor represents an authoritative entity via a public key and associated data.  The public key is used to verify digital signatures, and the associated data is used to constrain the types of information for which the trust anchor is authoritative." "A trust anchor store is a set of one or more trust anchors stored in a device.  A device may have more than one trust anchor store, each of which may be used by one or more applications." Furthermore, the following abbreviations are used in this document: Microcontroller (MCU for microcontroller unit) is a small computer on a single integrated circuit, which is often used for mass volumne IoT devices. System on Chip (SoC) is an integrated circuit that integrates all components of a computer, such as CPU, memory, input/output ports, secondary storage, etc. Homogeneous Storage Architecture (HoSA): A device that stores all firmware components in the same way, for example in a file system or in flash memory. Heterogeneous Storage Architecture (HeSA): A device that stores at least one firmware component differently from the rest, for example a device with an external, updatable radio, or a device with internal and external flash memory. </ins> 3. 3.1. Firmware images can be conveyed to devices in a variety of ways,  <ins> including USB, UART, WiFi, BLE, low-power WAN technologies, etc.  and </ins> use different protocols (e.g., CoAP, HTTP).  The specified mechanism needs to be agnostic to the distribution of the firmware images and manifests. 3.9.  <ins> When a device obtains a monolithic firmware image from a single author without any additional approval steps then the authorization flow is relatively simple.  There are, however, other cases where more complex policy decisions need to be made before updating a device. </ins>  <ins> In this architecture the authorization policy is separated from the underlying communication architecture.  This is accomplished by separating the entities from their permissions.  For example, an author may not have the authority to install a firmware image on a device in critical infrastructure without the authorization of a device operator.  In this case, the device may be programmed to reject firmware updates unless they are signed both by the firmware author and by the device operator. </ins>  <ins> Alternatively, a device may trust precisely one entity, which does all permission management and coordination.  This entity allows the device to offload complex permissions calculations for the device. </ins> 3.10. There are three broad classifications of update operating modes.  <ins> Client-initiated Update Server-initiated Update Hybrid Update Client-initiated updates take the form of a communicator on a device proactively checking for new firmware imagines provided by firmware servers. Server-initiated updates are important to consider because timing of updates may need to be tightly controlled in some high- reliability environments.  In this case the communicator, potentially in coordination with the status tracker, determines what devices qualify for a firmware update.  Once those devices have been selected the firmware server distributes updates to those devices. Note: This assumes that the firmware server is able to reach the device, which may require devices to keep reachability information at the communicator and / or at the firmware server up-to-date.  This may also require keeping state at NATs and stateful packet filtering firewalls alive. Hybrid updates are those that require an interaction between the device and the firmware server / communicator.  The communicator pushes notifications of availability of an update to the device, and the device then downloads the image from the firmware server when it wants. An alternative approach is to consider the steps a device has to go through in the course of an update: Notification Pre-authorisation Dependency resolution Download Installation The notification step consists of the communicator informing the device that an update is available.  This can be accomplished via polling (client-initiated), push notifications (server-initiated), or more complex mechanisms. The pre-authorisation step involves verifying whether the entity signing the manifest is indeed authorized to perform an update.  The device must also determine whether it should fetching and processing of the firmware image (unless it has been attached already to the manifest itself). </ins> A dependency resolution phase is needed when more than one component can be updated or when a differential update is used.  The necessary dependencies must be available prior to installation. The download step is the process of acquiring a local copy of the  <ins> firmware image.  When the download is client-initiated, this means that the device chooses when a download occurs and initiates the download process.  When a download is server-party initiated, this means that either the communicator / firmware server tells the device when to download or that it initiates the transfer directly to the device.  For example, a download from an HTTP-based firmware server is client-initiated.  A transfer to a LwM2M Firmware Update resource LwM2M is server-initiated. If the Device has downloaded a new firmware image and is ready to install it it may need to wait for a trigger from a Communicator to install the firmware update, may trigger the update automatically, or may go through a more complex decision making process to determine the appropriate timing for an update (such as delaying the update process to a later time when end users are less impacted by the update process). </ins> Installation is the act of processing the payload into a format that  <ins> the IoT device can recognise and the bootloader is responsible for then booting from the newly installed firmware image. </ins>  <ins> Each of these steps may require different permissions. </ins> 4.  <ins> Claims in the manifest offer a way to convey instructions to a device that impact the firmware update process.  To have any value the manifest containing those claims must be authenticated and integrity protected.  The credential used to must be directly or indirectly related to the trust anchor installed at the device by the Trust Provisioning Authority. </ins>  <ins> The baseline claims for all manifests are described in I-D.ietf-suit- information-model.  For example, there are: </ins> Do not install firmware with earlier metadata than the current metadata. Only install firmware that is before its best-before timestamp.  <ins> Only allow a firmware installation if dependencies have been met. </ins> Choose the mechanism to install the firmware, based on the type of firmware it is. 5.  <ins> arch-figure shows the communication architecture where a firmware image is created by an author, and uploaded to a firmware server. The firmware image/manifest is distributed to the device either in a push or pull manner using the communicator residing on the device. The device operator keeps track of the process using the status tracker.  This allows the device operator to know and control what devices have received an update and which of them are still pending an update. End-to-end security mechanisms are used to protect the firmware image and the manifest although e2e-figure does not show the manifest itself since it may be distributed independently. </ins> Whether the firmware image and the manifest is pushed to the device  <ins> or fetched by the device is a deployment specific decision. </ins> The following assumptions are made to allow the device to verify the received firmware image and manifest before updating software:  <ins> To accept an update, a device needs to verify the signature covering the manifest.  There may be one or multiple manifests that need to be validated, potentially signed by different parties.  The device needs to be in possession of the trust anchors to verify those signatures.  Installing trust anchors to devices via the Trust Provisioning Authority happens in an out-of- band fashion prior to the firmware update process. Not all entities creating and signing manifests have the same permissions.  A device needs to determine whether the requested action is indeed covered by the permission of the party that signed the manifest.  Informing the device about the permissions of the different parties also happens in an out-of-band fashion and is also a duty of the Trust Provisioning Authority. </ins> For confidentiality protection of firmware images the author needs to be in possession of the certificate/public key or a pre-shared  <ins> key of a device.  The use of confidentiality protection of firmware images is deployment specific. </ins> There are different types of delivery modes, which are illustrated based on examples below. cryptographic information, such as digital signatures or message authentication codes (MACs).  <ins> The manifest information model is described in I-D.ietf-suit- information-model. </ins> 7. <ins> Although these documents attempt to define a firmware update architecture that is applicable to both existing systems, as well as yet-to-be-conceived systems; it is still helpful to consider existing architectures. 7.1. The simplest, and currently most common, architecture consists of a single MCU along with its own peripherals.  These SoCs generally contain some amount of flash memory for code and fixed data, as well as RAM for working storage.  These systems either have a single firmware image, or an immutable bootloader that runs a single image. A notable characteristic of these SoCs is that the primary code is generally execute in place (XIP).  Combined with the non-relocatable nature of the code, firmware updates need to be done in place. 7.2. Another configuration consists of a similar architecture to the previous, with a single CPU.  However, this CPU supports a security partitioning scheme that allows memory (in addition to other things) to be divided into secure and normal mode.  There will generally be two images, one for secure mode, and one for normal mode.  In this configuration, firmware upgrades will generally be done by the CPU in secure mode, which is able to write to both areas of the flash device.  In addition, there are requirements to be able to update either image independently, as well as to update them together atomically, as specified in the associated manifests. 7.3. This configuration has two or more CPUs in a single SoC that share memory (flash and RAM).  Generally, they will be a protection mechanism to prevent one CPU from accessing the other's memory. Upgrades in this case will typically be done by one of the CPUs, and is similar to the single CPU with secure mode. 7.4. This configuration has two or more CPUs, each having their own memory.  There will be a communication channel between them, but it will be used as a peripheral, not via shared memory.  In this case, each CPU will have to be responsible for its own firmware upgrade. It is likely that one of the CPUs will be considered a master, and will direct the other CPU to do the upgrade.  This configuration is commonly used to offload specific work to other CPUs.  Firmware dependencies are similar to the other solutions above, sometimes allowing only one image to be upgraded, other times requiring several to be upgraded atomically.  Because the updates are happening on multiple CPUs, upgrading the two images atomically is challenging. 8. </ins> The following example message flow illustrates the interaction for distributing a firmware image to a device starting with an author  <ins> uploading the new firmware to Firmware Server and creating a manifest.  The firmware and manifest are stored on the same Firmware Server. </ins>  <ins> 9. </ins> This document does not require any actions by IANA.  <ins> 10. </ins> Firmware updates fix security vulnerabilities and are considered to be an important building block in securing IoT devices.  Due to the incentives for manufacturers to offer a firmware update mechanism as part of their IoT products.  <ins> 11. </ins> The discussion list for this document is located at the e-mail address suit@ietf.org [1].  Information on the group and information Archives of the list can be found at: https://www.ietf.org/mail- archive/web/suit/current/index.html [3]  <ins> 12.  References </ins>  <ins> 12.1.  URIs </ins> [1] mailto:suit@ietf.org


<del> Problem Statement of IoT integrated with Edge Computing </del>  draft-hong-t2trg-iot-edge-computing-latest Abstract cost, uninterrupted services, privacy and security, for IoT services originated from the IoT environmental changes.  In order to address those new challenges, the integration of Edge computing and IoT has <del> been emerged as a promising solution.  This document discribes the concept of IoT integrated with Edge computing as well as the state- of-the-art of IoT Edge computing.  It also proposes an architecture of IoT Edge computing.  The direction of Edge computing for IoT should be discussed in the IETF/IRTF. </del>  1. to augment their capabilities in order to fulfill their application tasks.  This is hard and partly impossible to achieve with cloud services for systems such as vehicles, drones, or oil rigs that have <del> intermittent network connectivity. </del>  4.4. 5. <del> As described in section 4, there are new challenges for supporting emerging IoT services and Edge computing is one of the candidates to satisfy these challenges.  The motivation for IoT Edge computing was discussed at an Edge computing discussion in IETF/IRTF meetings as follows: </del>  <del> Delay-sensitive High-volume </del>  <del> Trust-sensitive </del>  <del> (Intermittently) disconnected </del>  <del> Energy-challenged </del>  <del> Costly to transmit </del>  <del> As we described at previous sections, the above motivation for IoT Edge computing could directly be benefits of Edge computing in the IoT environment.  The above motivation for IoT Edge computing is mainly related to IoT data and other motivation for IoT Edge computing can exist as other aspects of networking and communication. </del>  <del> In spite of its benefits, Edge computing in IoT services has challenges such as programmability, naming, data abstraction, service management, privacy and security and optimization metrics. </del>  <del> Edge computing can support IoT services independently of Cloud computing.  However, Edge computing is increasingly connected to Cloud computing in most IoT systems for processing and storaging data.  Thus, the relationship of Edge Computing to Cloud Computing is also another challenge of Edge Computing in IoT ISO_TR. </del>  <del> 5.1. </del>  <del> As an aspect of IoT, Edge computing can provide many capabilities for IoT services because IoT systems are based on sensors and actuator devices in edge area and IoT data generated from sensors and actuator devices are gathered through a gateway ISO_TR.  Besides on IoT data, other functions such as computing, control and network functions are also very remarkable to support IoT services.  In this document, we will first concentrate on IoT data's aspect since the benefit of Edge computing with IoT data is very big in use cases. 5.1.1. As tremendous IoT sensors, IoT actuators, and IoT devices are connected to the Internet, IoT data volume from these things are expected to increase explosively.  And it is expected that much of this high volume of IoT data is produced and/or consumed within edge/ local networks, not to traverse through cloud networks.  Until now, most IoT data generated by IoT things is transferred and accumulated in a remote server and storage of IoT data in a remote server is expensive in transmission and storage.  To mitigate the cost of transmission and storage, it is required to divide IoT data into two types of data; one is stored in edge/local networks and the other is stored in cloud networks.  The effect of Edge computing is revealed with the handling IoT data in edge/local networks. 5.1.2. Until now, most network equipment such as routers, gateways, and switches just forward data delivered from other network devices without reading or modifying the content.  In end-to-end communication, data is acknowledged and proceed at a final corresponding node.  This is a typical usage of cloud computing and a client-server communication.  But, in the IoT environment, some IoT data will be transferred to a cloud network and some will be delivered to an edge node.  The main reason of this separation is to provide real-time processing and security enhancement in IoT. Although there are many new technologies to reduce the delay and transmission time, it is not easy to guarantee real-time processing. The typical use case of this requirement is industrial Internet and smart factory.  Even though there are also several solutions to provide security in IoT, the more basic rule is not to expose the privacy data to public networks.  If we separate IoT data into private and non-private data, and keep private data within an edge/ local network not to expose them in a public network, the security and privacy in IoT cna be addressed by the separation. 5.1.3. If it is possible to separate IoT data in edge/local networks and cloud networks, Edge computing can do more functions with IoT data in edge/local networks.  Because Edge computing has the capabilities to handle IoT data in edge/local networks, it is also possible to analyze IoT data to provide enhanced IoT services such as intelligence.  To analyze IoT data in an edge/local network, it is required to have comparatively processing performance and this requirement is not obstacle to deploy Edge computing due to the development of H/W and S/W. </del>  <del> 5.2. </del>  <del> If we consider new challenges of IoT services, not only the big volume of IoT data but also the massive number of IoT things can be a critical problem.  Even though, we acknowledge this future problem, the Internet architecture originally has the capability of scalability and it will mitigate scalability issue in the IoT environment.  But, we cannot estimate the number of IoT things in the future and we cannot guarantee the Internet architecture still sustain the scalability issue in the IoT environment.  Edge computing will separate the scalability domain into edge/local networks and outside network (e.g., cloud networks) and this separation of scalability domain can provide more efficient way to tackle the massive number of IoT things. Because Edge computing can handle IoT data in an edge area and store the IoT data in an edge node, and proceed IoT data if it is needed, it can also separate the management domain into two parts.  Edge Computing can concentrate on management of IoT things in an edge area and cooperate with the management of other outside networks. </del>  <del> 6. </del>  <del> When we consider the implementation and deployment of Edge computing, it can be mainly referred to an IoT Gateway.  The role of an IoT Gateway is to provide multiple accesses to the heterogeneous IoT devices/sensors, handling IoT data and delivering the IoT data to the final destinations such as cloud networks.  Similar to an IoT Gateway, an Edge computing architecture as an edge computing node provides downside connectivity to IoT sensors and devices (southbound connectivity) and upside connectivity to cloud networks (northbound connectivity).  Also, the architecture provides the function of data storage.  Beside these functions, the Edge computing architecture should provide the computing functions, such as data processing, data analyzing, and additional function of intelligence. It is expected that the Edge computing architecture will play an important role to deploy new IoT services with integration to big data and AI services. </del>  <del> 7. </del>  <del> 7.1. </del>  This section provides an overview of today's IoT Edge Computing field, based on a limited review of standards, research, open-source edge computing devices (or deferred to off-peak hours or using physical shipping); and downlink for control and software updates. <del> 7.2. </del>  <del> 7.2.1. </del>  In traditional construction domain, there are many heavy equipment and machineries and dangerous elements.  Even though human pay By adjusting the resolution/degree of audio and video data, it can reduce transmission cost significantly. <del> 7.2.2. </del>  In future smart cities, Smart grids will be critical in ensuring availability and efficiency for energy saving and control in city- can help monitoring power generation and power demands, and making electrical energy storage decisions in the Smart grid system. <del> 7.2.3. </del>  The Water system is one of the most important aspects for building smart city.  Effective use of water, and cost-effective and management, control the reduction of water losses and improve the city water system through Edge computing. <del> 7.2.4. </del>  T.B.D. <del> 7.2.5. </del>  T.B.D. <del> 7.2.6. </del>  T.B.D. <del> 8. </del>  T.B.D. <del> 9. </del>  The authors would like to thank Joo-Sang Youn and Akbak Rahman for their valuable comments and suggestions on this document.

Main change: Merged 2 sections (IoT Integrated with Edge Computing and architecture section) into a new "model" section. Updated the architecture diagram into a model diagram including the logical functions proposed by Dirk. Heavily reworded/simplified text from the previous 2 sections - I tried to keep relevant info, but removed what was redundant with the existing challenges section. The architectures (gateway vs. distributed) are shortly listed above the figure, but the rest aims to be general. Some renaming for your consideration (feel free to ignore/change): new draft title: IoT Edge Computing Challenges, Model and State-of-Art Review challenges section ("new IoT challenges" could seem a bit general): Challenges for IoT and Impacts of Edge Computing Minor: edits in abstract
Yong-Geun, Jungha, and Dirk agreed via e-mail. Since the new version has been sent to the Datatracker, I will merge.

<ins> IoT Edge Computing Challenges, Model and State-of-Art Review </ins> draft-hong-t2trg-iot-edge-computing-latest Abstract cost, uninterrupted services, privacy and security, for IoT services originated from the IoT environmental changes.  In order to address those new challenges, the integration of Edge computing and IoT has  <ins> emerged as a promising solution.  This document describes the concept of IoT integrated with Edge computing as well as the state-of-the-art of IoT Edge computing.  It also proposes a general model for IoT Edge computing.  The direction of Edge computing for IoT should be discussed in the IETF/IRTF. </ins> 1. to augment their capabilities in order to fulfill their application tasks.  This is hard and partly impossible to achieve with cloud services for systems such as vehicles, drones, or oil rigs that have  <ins> intermittent network connectivity.  Example of related challenges include support for IoT device and edge computing node mobility, as well as software instance migration. </ins> 4.4. 5.  <ins> It is expected edge computing architectures will play an important role to deploy new IoT services integrated with big data and AI services.  In this section we discuss a general model that aims to be applicable to multiple architectures, such as: A single IoT gateway, or a hierarchy of IoT gateways, typically connected to the cloud (e.g., to extend the traditionally cloud- based management of IoT devices and data to the edge).  A common role of an IoT Gateway is to provide access to an heterogeneous set of IoT devices/sensors; handle IoT data; and deliver IoT data to its final destination in a cloud network. A set of distributed computing nodes, e.g. embedded in switches, routers, edge cloud servers or mobile devices.  In the future, some IoT end devices may have enough computing capabilities to participate in such distributed systems. The edge computing domains is interconnected with IoT end devices (southbound connectivity) and possibly with a remote/cloud network (northbound connectivity).  Edge computing nodes provide multiple logical functions such as finding resources, authentication, storage/ processing and management. </ins>  <ins> 5.1. </ins>  <ins> A northbound interface is provided by a gateway component to a remote network, e.g. a cloud, home or enterprise network.  These components may not exist in standalone scenarios, where edge computing is provided locally without a connection to a remote network.  The northbound interface is a data plane interface.  Nevertheless, the remote network may also host an edge cloud manager function. </ins>  <ins> 5.2. </ins>  <ins> Edge computing domain management includes management of resources and functions in the edge computing domain.  Management of IoT end devices and IoT data may be included or may be part of the edge computing logical functions (OPEN QUESTION).  The management function can provides SaaS, PaaS, IaaS service APIs to an edge computing manager.  The edge management role may be taken by an entity in the cloud, but it may also be a local entity, or even be non-existent (during normal operation) in autonomic systems. </ins>  <ins> 5.3. </ins>  <ins> Edge computing nodes host logical functions relative to: </ins>  <ins> Finding resources, such as compute, storage or data resources; </ins>  <ins> Authenticating platforms, end devices, functions, data; </ins>  <ins> Providing compute and storage offloading; </ins>  <ins> Management, e.g. of IoT end devices and data. </ins>  <ins> With regard to the high level challenges listed in sec-challenges, data storage and processing at the edge is a major aspect of IoT edge computing.  Data may therefore need to be classified (e.g. in terms of privacy, importance, validity, etc.).  Data analysis such as performed in AI/ML tasks performed at the edge may benefit from specialized hardware support on computing nodes.  IoT edge computing will face detailed challenges in term of, for example, programmability, naming, data abstraction and service management. Furthermore, while edge computing can support IoT services independently of cloud computing, it is increasingly connected to cloud computing in most IoT systems: thus, the relationship of IoT edge Computing to cloud Computing is another potential challenge ISO_TR. </ins>  <ins> 5.4. </ins>  <ins> IoT end devices can be sensors, actuators, or more generally IoT things.  Not only the big volume of IoT data but also the massive number of IoT end devices are the cause of a massive scalability issue in future IoT environments.  To address this challenge edge computing separates the scalability domain into edge/local networks and remote network. </ins>  <ins> Edge computing nodes communicate between themselves and with end devices over an underlying network.  There is therefore a need for the edge computing domain to directly or indirectly control those network functions. </ins>  <ins> 6. </ins>  <ins> 6.1. </ins> This section provides an overview of today's IoT Edge Computing field, based on a limited review of standards, research, open-source edge computing devices (or deferred to off-peak hours or using physical shipping); and downlink for control and software updates.  <ins> 6.2. </ins>  <ins> 6.2.1. </ins> In traditional construction domain, there are many heavy equipment and machineries and dangerous elements.  Even though human pay By adjusting the resolution/degree of audio and video data, it can reduce transmission cost significantly.  <ins> 6.2.2. </ins> In future smart cities, Smart grids will be critical in ensuring availability and efficiency for energy saving and control in city- can help monitoring power generation and power demands, and making electrical energy storage decisions in the Smart grid system.  <ins> 6.2.3. </ins> The Water system is one of the most important aspects for building smart city.  Effective use of water, and cost-effective and management, control the reduction of water losses and improve the city water system through Edge computing.  <ins> 6.2.4. </ins> T.B.D.  <ins> 6.2.5. </ins> T.B.D.  <ins> 6.2.6. </ins> T.B.D.  <ins> 7. </ins> T.B.D.  <ins> 8. </ins> The authors would like to thank Joo-Sang Youn and Akbak Rahman for their valuable comments and suggestions on this document.


can summarize a general philosophy of Edge computing as "Distribute the required functions close to users and data".  4. As the IoT is maturing, systems are converging, deployments are 6. <del> 6.1. This section provides an overview of today's IoT Edge Computing field, based on a limited review of standards, research, open-source and proprietary products in sec-overview. Common aspects of IoT Edge computing service platforms are summarized here: IoT gateways (sec-iot-gateways, sec-gateway-platforms) represent a common class of IoT Edge computing products, where the gateway is providing a local service on customer premises, and is remotely managed through a cloud service.  IoT communication protocols are typically used between IoT devices and the gateway, including CoAP, MQTT and many specialized IoT protocols, while the gateway communicates with the distant cloud using typically HTTP and WebSocket.  Virtualization platforms enable the deployment of virtual Edge computing functions, including IoT gateway software, on servers in the mobile network infrastructure (at base station and concentration points), in edge datacenters (in central offices) or regional datacenters located near central offices. End devices as computing devices are envisioned in fog architecture and research projects, but are not commonly used as such today. Physical or virtual IoT gateways can host application programs built using an SDK.  Edge cloud system operators host their customers' applications VMs or containers on servers located in or near access networks.  These application have access to edge service APIs.  For example, mobile network services include radio network information, location, bandwidth management.  In a cloud- like service model, service providers consume low-level edge platform APIs and offer high-level APIs to their own customers' applications.  This cloud-like model can be offered as an edge cloud service, or as an hybrid cloud service covering edge and distant cloud. Life cycle management of services and applications on physical IoT gateways is often cloud-based.  Edge cloud management platforms and products (sec-cloud-management-platforms, sec-cloud-platforms) adapt cloud management technologies (e.g. kubernetes) to the edge cloud, i.e. to smaller, distributed computing devices running outside a controlled data center.  Services and application life- cycle is typically using a NFV-like management and orchestration model. The platform typically includes services to advertise or consume APIs, and enables communicating with local and remote endpoints. The service platform is typically extensible by edge applications, since they can advertise an API that other edge applications can consume.  IoT communication services include protocols translation, analytics and transcoding.  Communication between Edge computing devices is enabled in tiered deployments or distributed deployments. An edge cloud platform may enable pass-through without storage, local storage (e.g. on IoT gateways).  Some edge cloud platforms use a distributed form of storage, e.g. an ICN network or a distributed storage platform.  External storage, e.g. on databases in distant or local IT cloud, is typically used for filtered data deemed worthy of long term storage, or in some cases for all data, for example when required for regulatory reasons. Stateful computing is supported on platforms hosting native programs, VMs or containers.  Stateless computing is supported on platforms providing a "serverless computing" service (a.k.a. function-as-a-service), or on systems based on named function networking. Network traffic is typically high volume uplink with throttling by Edge computing devices (or deferred to off-peak hours or using physical shipping); and downlink for control and software updates. 6.2. 6.2.1. In traditional construction domain, there are many heavy equipment and machineries and dangerous elements.  Even though human pay attention to risk elements, it is not easy to avoid them.  If some accidents are happened in a construction site, it causes a loss of lives and property.  Thus, there have been many trials in a construction area to protect lives and property. Measurements of noise, vibration, and gas in a construction area are recorded on a remote server and reported to an inspector.  Today, data produced bu such measurements is collected by a gateway in a construction area and transferred to a remote server.  This incurs transmission cost, e.g. over a LTE connection, and storage cost, e.g. when using Amazon Web Services.  When an inspector wants to investigate some accidents, he checks the information stored in a server. If we deploy Edge computing in a construction area, the sensor data can be processed and analyzed in a gateway located within or near a construction area.  And with the help of a statistical analysis or machine learning technologies, we can predict future accidents in advance and this prediction can be used as an alarm in a construction area and a notification to an inspector. To determine the exact cause of some accident, not only sensor data but also audio and video data are transferred to a remote server or cloud networks.  In this case, the data volume of audio and video is quite big and the cost of transmission can be a problem.  If Edge computing can predict the time of accident, it can reduce the data volume of transmission; in general period, it can transmit the audio and video data with a low resolution/degree and in emergent period, it transmits the audio and video data with a high resolution/degree. By adjusting the resolution/degree of audio and video data, it can reduce transmission cost significantly. 6.2.2. In future smart cities, Smart grids will be critical in ensuring availability and efficiency for energy saving and control in city- wide electricity management.  Edge computing is expected to play a significant role in those systems to improve transmission efficiency of electricity, react and restore for power disturbances, reduce operation cost, reuse renewable energy effectively, save energy of electricity for future usage, and so on.  In addition, Edge computing can help monitoring power generation and power demands, and making electrical energy storage decisions in the Smart grid system. 6.2.3. The Water system is one of the most important aspects for building smart city.  Effective use of water, and cost-effective and environment-friendly treatment of water are critical for water control and management.  This can be facilitated by Edge computing in Smart water systems, to help monitor water consumption, transportation, prediction of future water use, and so on.  For example, water harvesting and ground water monitoring will be supported from Edge computing.  Also, a Smart water system is able to analyze collected information related to water control and management, control the reduction of water losses and improve the city water system through Edge computing. 7. </del> T.B.D. <del> 8. </del>  The authors would like to thank Joo-Sang Youn and Akbak Rahman for their valuable comments and suggestions on this document.

This update is for issue : re-ordered use cases and state-of-the-art sections. Also did some minor editorial updates (on section titles and on use cases text).
Use cases to introduction State of the art before deriving patterns ... Keep appendix with detailed projects definitely while working on the document (might also stay in published document)
Closed after publishing -02

can summarize a general philosophy of Edge computing as "Distribute the required functions close to users and data". <ins> 3.4. 3.4.1. In traditional construction domain, heavy equipment and machineries pause risks to humans and property.  Thus, there have been many attempts to deploy technology to protect lives and property in construction sites.  For example, measurements of noise, vibration, and gas in a construction site can be recorded on a remote server and reported to an inspector.  Today, data produced by such measurements is collected by a gateway in a construction site and transferred to a remote server.  This incurs transmission costs, e.g. over a LTE connection, and storage costs, e.g. when using Amazon Web Services. When an inspector needs to investigate some accidents, he checks the information stored in a server. If we leverage IoT edge computing, sensor data can be processed and analyzed in a gateway located within or near a construction site. And with the help of a statistical analysis or machine learning technologies, we can predict future accidents in advance and this prediction can trigger an alarm in a construction site and a notification to an inspector. To determine the exact cause of an accident, not only sensor data but also audio and video data are transferred to a remote server or cloud networks.  In this case, the data volume of audio and video is quite big and the cost of transmission can be an issue.  If edge computing can be leveraged to predict the time of an accident, this can reduce the data volume of transmission; while during a normal time period audio and video data may be transmitted with a low resolution, during an emergency, this transmission may use a high resolution.  This adjustment can reduce transmission cost significantly. 3.4.2. In future smart city scenarios, the smart grid will be critical in ensuring highly available/efficient energy saving and control in city-wide electricity management.  Edge computing is expected to play a significant role in those systems to improve transmission efficiency of electricity, to react and restore power after a disturbance, reduce operation cost and reuse renewable energy effectively.  In addition, edge computing can help monitoring power generation and power demand, and making electrical energy storage decisions in the Smart grid system. 3.4.3. The water system is one of the most important aspects of a city. Effective use of water, and cost-effective and environment-friendly treatment of water are critical for water control and management. This can be facilitated by edge computing in smart water systems, to help monitor water consumption, transportation and prediction of future water use.  For example, water harvesting and ground water monitoring will be supported through edge computing.  Also, a smart water system is able to analyze collected information related to water control and management, control the reduction of water losses and improve the city water system through edge computing. 3.5. This section provides an overview of today's IoT Edge Computing field, based on a limited review of standards, research, open-source and proprietary products in sec-overview. Common aspects of IoT Edge computing service platforms are summarized here: IoT gateways (sec-iot-gateways, sec-gateway-platforms) represent a common class of IoT Edge computing products, where the gateway is providing a local service on customer premises, and is remotely managed through a cloud service.  IoT communication protocols are typically used between IoT devices and the gateway, including CoAP, MQTT and many specialized IoT protocols, while the gateway communicates with the distant cloud using typically HTTP and WebSocket.  Virtualization platforms enable the deployment of virtual Edge computing functions, including IoT gateway software, on servers in the mobile network infrastructure (at base station and concentration points), in edge datacenters (in central offices) or regional datacenters located near central offices. End devices as computing devices are envisioned in fog architecture and research projects, but are not commonly used as such today. Physical or virtual IoT gateways can host application programs built using an SDK.  Edge cloud system operators host their customers' applications VMs or containers on servers located in or near access networks.  These application have access to edge service APIs.  For example, mobile network services include radio network information, location, bandwidth management.  In a cloud- like service model, service providers consume low-level edge platform APIs and offer high-level APIs to their own customers' applications.  This cloud-like model can be offered as an edge cloud service, or as an hybrid cloud service covering edge and distant cloud. Life cycle management of services and applications on physical IoT gateways is often cloud-based.  Edge cloud management platforms and products (sec-cloud-management-platforms, sec-cloud-platforms) adapt cloud management technologies (e.g. kubernetes) to the edge cloud, i.e. to smaller, distributed computing devices running outside a controlled data center.  Services and application life- cycle is typically using a NFV-like management and orchestration model. The platform typically includes services to advertise or consume APIs, and enables communicating with local and remote endpoints. The service platform is typically extensible by edge applications, since they can advertise an API that other edge applications can consume.  IoT communication services include protocols translation, analytics and transcoding.  Communication between Edge computing devices is enabled in tiered deployments or distributed deployments. An edge cloud platform may enable pass-through without storage, local storage (e.g. on IoT gateways).  Some edge cloud platforms use a distributed form of storage, e.g. an ICN network or a distributed storage platform.  External storage, e.g. on databases in distant or local IT cloud, is typically used for filtered data deemed worthy of long term storage, or in some cases for all data, for example when required for regulatory reasons. Stateful computing is supported on platforms hosting native programs, VMs or containers.  Stateless computing is supported on platforms providing a "serverless computing" service (a.k.a. function-as-a-service), or on systems based on named function networking. Network traffic is typically high volume uplink with throttling by Edge computing devices (or deferred to off-peak hours or using physical shipping); and downlink for control and software updates. </ins> 4. As the IoT is maturing, systems are converging, deployments are 6.  T.B.D.  <ins> 7. </ins> The authors would like to thank Joo-Sang Youn and Akbak Rahman for their valuable comments and suggestions on this document.


logical functions such as finding resources, authentication, storage/ processing and management. <del> 5.1. A northbound interface is provided by a gateway component to a remote network, e.g. a cloud, home or enterprise network.  These components may not exist in standalone scenarios, where Edge computing is provided locally without a connection to a remote network.  The northbound interface is a data plane interface.  Nevertheless, the remote network may also host an edge cloud manager function. 5.2. Edge computing domain management includes management of resources and functions in the Edge computing domain.  Management of IoT end devices and IoT data may be included or may be part of the Edge computing logical functions (OPEN QUESTION).  The management function can provides SaaS, PaaS, IaaS service APIs to an Edge computing manager.  The edge management role may be taken by an entity in the cloud, but it may also be a local entity, or even be non-existent (during normal operation) in autonomic systems. 5.3. Edge computing nodes host logical functions relative to: Finding resources, such as compute, network, storage or data resources; Authenticating platforms, end devices, functions, data; Providing compute and storage offloading; </del>  <del> Management, e.g. of IoT end devices and data. </del>  With regard to the high level challenges listed in sec-challenges, <del> data storage and processing at the edge is a major aspect of IoT Edge </del>  computing.  Data may therefore need to be classified (e.g. in terms of privacy, importance, validity, etc.).  Data analysis such as performed in AI/ML tasks performed at the edge may benefit from <del> specialized hardware support on computing nodes.  IoT Edge computing will face detailed challenges in term of, for example, programmability, naming, data abstraction and service management. Furthermore, while Edge computing can support IoT services independently of Cloud computing, it is increasingly connected to Cloud computing in most IoT systems: thus, the relationship of IoT Edge Computing to Cloud Computing is another potential challenge ISO_TR. 5.4. IoT end devices can be sensors, actuators, or more generally IoT things.  Not only the big volume of IoT data but also the massive number of IoT end devices are the cause of a massive scalability issue in future IoT environments.  To address this challenge Edge computing separates the scalability domain into edge/local networks and remote network. Edge computing nodes communicate between themselves and with end devices over an underlying network.  There is therefore a need for the Edge computing domain to directly or indirectly control those network functions. </del>  <del> 6. </del>  <del> T.B.D. </del>  7.  The authors would like to thank Joo-Sang Youn and Akbak Rahman for their valuable comments and suggestions on this document.

For issue
So far an initial list of functions/components has been writen in a new components section. Another action item from IETF 106 Singapore is to identify whitespots in edge functions, which may be also performed in this section.
Closed after publishing -02

logical functions such as finding resources, authentication, storage/ processing and management.  <ins> 6. </ins>  <ins> This section lists typical architecture components of an edge computing system.  Most components can be centralized or distributed, implemented in the edge network, or through some interworking between the edge network and a remote cloud network. 6.1. Edge computing OAM goes beyond the network-related OAM functions listed in RFC 6291.  Besides infrastructure (network, storage and computing resources), edge computing systems can also include computing environments (for VMs, software containers, functions), IoT end devices, data and code. Operation related functions include performance monitoring for service level agreement measurement; fault management and provisioning for links, nodes, compute and storage resources, platforms and services.  Administration covers network/compute/ storage resources, platforms and services discovery, configuration and planning.  Management covers monitoring and diagnostics of failures, as well as means to minimize their occurrence and take corrective actions.  This may include software updates management, high service availability through redundancy and multipath communication. We further detail a few OAM aspects. 6.1.1. This OAM component is about finding infrastructure resources, such as compute, network and storage, but also other resources such as IoT end devices, sensors, data, code, or services. 6.1.2. Some IoT edge computing systems make use of virtualized (compute, storage and networking) resources, which needs to be allocated and configured. 6.1.3. This can cover authenticating platforms, end devices, data and code units.  Today authentication typically relies on the installation of a secret on IoT end devices and on computing devices (e.g. a device certificate). 6.1.4. IoT end device management includes managing information about the IoT devices, including their sensors, how to communicate with them, etc. Edge computing addresses the scalability challenges from the massive number of IoT end devices and IoT data value by separating the scalability domain into edge/local networks and remote network. 6.2. 6.2.1. An IoT edge cloud may provide a northbound data plane or management plane interface to a remote network, e.g. a cloud, home or enterprise network.  This interface does not exist in standalone (local-only) scenarios.  To support such an interface when it exists, an edge computing component needs to expose an API, deal with authentication and authorization, support secure communication. An IoT edge cloud may provide an API or interface to local users (e.g. to facilitate local management), or to mobile users (e.g. to provide access to services and applications). 6.2.2. A typical function of IoT edge computing is to facilitate communication with IoT end devices: for example, enable clients to register as recipients for data from devices, as well as forwarding/ routing of traffic to or from IoT end devices.  Another aspect of a communication component is dispatching of alerts and notifications to interested consumers both inside and outside of the edge computing domain.  Protocol translation may also be performed when necessary. Communication brokering may be centralized in some systems, e.g. using a hub-and-spoke message broker, or distributed like with message buses.  Distributed systems may leverage direct communication between end devices and communication devices, such as device-to- device links.  Brokers functions can include ensuring communication reliability, traceability, an in some cases transaction management. QoS can be provided in some systems through the combination of network QoS (e.g. traffic engineering or wireless resource scheduling) and compute/storage resource allocations.  In some systems a bandwidth manager service can be exposed to enable allocation of bandwidth to/from an edge computing application instance. 6.2.3. A core function of IoT edge computing is to enable computation offloading, i.e. to perform computation on an edge node on behalf of a device or user.  The support for in-network computation may vary in term of capability, e.g. computing nodes can host a virtual machine able run stateful or stateless code, or a rule engine providing an API to register actions in response to conditions such as IoT device ID, sensor values to check, thresholds, etc.  Computation offloading includes orchestration or application lifecycle related aspects, such as: selecting an appropriate compute device based on available resources, compute node properties, etc.; onboarding code on a platform or compute device; invoking a remote code execution; relocating an instance from one compute node to another. 6.2.4. A purpose of local caching may be to enable local data processing (e.g. pre-processing or analysis), or to enable delayed virtual or physical shipping.  A responsibility of the edge caching component is to manage data persistence, e.g. to schedule removal of data when it is no longer needed.  Another aspect of this component may be to authenticate an encrypt data.  It can for example take the form of a distributed storage system (e.g.  Ceph). 6.2.5. </ins> With regard to the high level challenges listed in sec-challenges,  <ins> data storage and processing at the edge is a major aspect of IoT edge </ins> computing.  Data may therefore need to be classified (e.g. in terms of privacy, importance, validity, etc.).  Data analysis such as performed in AI/ML tasks performed at the edge may benefit from  <ins> specialized hardware support on computing nodes.  IoT edge computing will face challenges in term of, for example, programmability, naming, data abstraction and service management.  Furthermore, while edge computing can support IoT services independently of cloud computing, it is increasingly connected to cloud computing in most IoT systems: thus, the relationship of IoT edge computing to cloud computing is another potential challenge ISO_TR. </ins>  <ins> 6.2.6. </ins>  <ins> Data generated by IoT devices and associated information obtained from the access network may be used to provide high level services such as end device location, radio network information, etc. </ins> 7. <ins> T.B.D. 8. </ins> The authors would like to thank Joo-Sang Youn and Akbak Rahman for their valuable comments and suggestions on this document.


Edge computing nodes host logical functions relative to: <del> Finding resources, such as compute, storage or data resources; </del>  Authenticating platforms, end devices, functions, data;

This is the example change we went through at the meeting at IETF 106.
We agreed at the F2F. Merging.

Edge computing nodes host logical functions relative to:  <ins> Finding resources, such as compute, network, storage or data resources; </ins> Authenticating platforms, end devices, functions, data;


the context of RESTful design for IoT systems.  For terminology of constrained nodes and networks, see RFC7228. <del> The state kept by a client between requests.  This typically includes the "current" resource, the set of active requests, the history of requests, bookmarks (URIs stored for later retrieval) and application-specific state. </del> A local store of response messages and the subsystem that controls storage, retrieval, and deletion of messages in it. A node that sends requests to servers and receives responses.  The practice of determining the "best" representation for a client when examining the current state of a resource.  The most common forms of content negotiation are Proactive Content Negotiation and A component embedded in a representation that identifies a resource for future hypermedia interactions, such as a link or a form.  If the client engages in an interaction with the identified <del> resource, the result may be a change resource state and/or application state. </del>  A method where multiple identical requests with that method lead to the same visible resource state as a single such request.  For non-existing resource. A hypermedia control that enables a client to navigate between <del> resources and thereby change the application state. </del>  A string such as "text/html" or "application/json" that is used to label representations so that it is known how the representation When designing a RESTful system, the state of the distributed application must be assigned to the different components.  Here, it <del> is important to distinguish between "session state" and "resource </del>  state". <del> Session state encompasses the control flow and the interactions </del>  between the components (see sec-terms).  Following the statelessness <del> constraint, the session state must be kept only on clients.  On the </del>  one hand, this makes requests a bit more verbose since every request must contain all the information necessary to process it.  On the other hand, this makes servers efficient, since they do not have to

Addressing issue
Thanks Ari :) I will do a final pass and merge.
Originally, application state defined the state kept on the client, that is, what Web page is currently rendered and which pages were visited before. This does not make sense anymore when resource state is more than static Web pages, but parameters, physical state, etc. "Application state" logically would mean the state of the overall IoT application (state of the client and resoource state).
Related comment by Mike Amundsen:
I would add a comment in the definition, that this is called "Application State" in Fielding's thesis, but has some ambiguity in mordern (IoT) applications, where resource state is very dynamic and reflects part of the state in which the overall application is.

the context of RESTful design for IoT systems.  For terminology of constrained nodes and networks, see RFC7228.  A local store of response messages and the subsystem that controls storage, retrieval, and deletion of messages in it. A node that sends requests to servers and receives responses. <ins> The state kept by a client between requests.  This typically includes the "current" resource, the set of active requests, the history of requests, bookmarks (URIs stored for later retrieval) and application-specific state.  (Note that this is called "Application State" in REST, which has some ambiguity in modern (IoT) systems where the overall state of the distributed application (i.e., application state) is reflected in the union of all Client States and Resource States of all clients and servers involved.) </ins> The practice of determining the "best" representation for a client when examining the current state of a resource.  The most common forms of content negotiation are Proactive Content Negotiation and A component embedded in a representation that identifies a resource for future hypermedia interactions, such as a link or a form.  If the client engages in an interaction with the identified  <ins> resource, the result may be a change to resource state and/or client state. </ins> A method where multiple identical requests with that method lead to the same visible resource state as a single such request.  For non-existing resource. A hypermedia control that enables a client to navigate between  <ins> resources and thereby change the client state. </ins> A string such as "text/html" or "application/json" that is used to label representations so that it is known how the representation When designing a RESTful system, the state of the distributed application must be assigned to the different components.  Here, it  <ins> is important to distinguish between "client state" and "resource </ins> state".  <ins> Client state encompasses the control flow and the interactions </ins> between the components (see sec-terms).  Following the statelessness  <ins> constraint, the client state must be kept only on clients.  On the </ins> one hand, this makes requests a bit more verbose since every request must contain all the information necessary to process it.  On the other hand, this makes servers efficient, since they do not have to


6.2. <del> To modify resource state, clients usually GET a representation from the server, process it locally, and transfer the resulting state back to the server with a PUT.  Sometimes, however, the state can only be modified on the server side, for instance, because representations would be too large to transfer or part of the required information shall not be accessible to clients. In this case, resource state is modified by calling a function.  This is usually modeled with a POST request, as it leaves the behavior semantics completely to the server. </del>  6.2.1.  6.2.2.  6.2.3. GET is cachable, good for static information such as look-up tables

The term "function" might imply idempotence and/or freedom from side effects, depending on who you ask. RPC uses the term "procedure" in order to be more general.
NAME would you suggest s/function/procedure/g ?
Changed to "procedure"

6.2.  <ins> To modify resource state, clients usually use GET to retrieve a representation from the server, modify that locally, and transfer the resulting state back to the server with a PUT (see sec-uniform- interface).  Sometimes, however, the state can only be modified on the server side, for instance, because representations would be too large to transfer or part of the required information shall not be accessible to clients.  In this case, resource state is modified by calling a procedure (or "function").  This is usually modeled with a POST request, as this method leaves the behavior semantics completely to the server.  Procedure calls can be divided into two different classes based on how long they are expected to execute: "instantly" returning and long-running. </ins> 6.2.1. <ins> When the procedure can return within the expected response time of the system, the result can be directly returned in the response.  The result can either be actual content or just a confirmation that the call was successful.  In either case, the response does not contain a representation of the resource, but a so-called action result. Action results can still have hypermedia controls to provide the possible transitions in the application state machine. </ins> 6.2.2. <ins> When the procedure takes longer than the expected response time of the system, or even longer than the response timeout, it is a good pattern to create a new resource to track the "task" execution.  The server would respond instantly with a "Created" status (HTTP code 201 or CoAP 2.01) and indicate the location of the task resource in the corresponding header field (or CoAP option) or as a link in the action result.  The created resource can be used to monitor the progress, to potentially modify queued tasks or cancel tasks, and to eventually retrieve the result. Monitoring information would be modeled as state of the task resource, and hence be retrievable as representation.  The result - when available - can be embedded in the representation or given as link to another sub-resource.  Modifying tasks can be modeled with forms that either update sub-resources via PUT or do a partial write using PATCH or POST.  Canceling a task would be modeled with a form that uses DELETE to remove the task resource. </ins> 6.2.3. GET is cachable, good for static information such as look-up tables


A local store of response messages and the subsystem that controls storage, retrieval, and deletion of messages in it. <del> A node that sends requests to servers and receives responses. </del>  The state kept by a client between requests.  This typically includes the currently processed representation, the set of active behalf of the client.  This may be useful, for example, when the client lacks the capability to make the request itself or to service the response from a cache in order to reduce response <del> time, network bandwidth and energy consumption. </del>  A reverse proxy that provides an interface to a non-RESTful system such as legacy systems or alternative technologies such as Bluetooth ATT/GATT.  See also "Reverse Proxy". <del> A component embedded in a representation that identifies a resource for future hypermedia interactions, such as a link or a form.  If the client engages in an interaction with the identified resource, the result may be a change to resource state and/or client state. </del>  A method where multiple identical requests with that method lead <del> to the same visible resource state as a single such request.  For example, the PUT method replaces the state of a resource with a new state; replacing the state multiple times with the same new state still results in the same state for the resource.  However, the response from the server can be different when the same idempotent method is used multiple times.  For example when DELETE is used twice on an existing resource, the first request would remove the association and return success acknowledgement whereas the second request would likely result in error response due to non-existing resource. </del>  A hypermedia control that enables a client to navigate between resources and thereby change the client state. A content negotiation mechanism where the server selects a representation based on the expressed preference of the client. <del> For example, in an IoT application, a client could send a request </del>  with preferred media type "application/senml+json". A content negotiation mechanism where the client selects a multiple machines. A method that does not result in any state change on the origin <del> server when applied to a resource.  For example, the GET method only returns a representation of the resource state but does not change the resource.  Thus, it is always safe for a client to retrieve a representation without affecting server-side state. </del>  A node that listens for requests, performs the requested operation and sends responses back to the clients. Section 4.3 of RFC7231 defines the set of methods in HTTP; Section 5.8 of RFC7252 defines the set of methods in CoAP.  As part of the Uniform Interface constraint, each method can have certain <del> properties that give guarantees to clients: Safe methods do not cause any state change on the origin server when applied to a resource. </del>  Idempotent methods can be applied multiple times to the same resource while causing the same visible resource state as a single such <del> request.  The following lists the most relevant methods and gives a short explanation of their semantics. </del>  3.5.1.

Note about client role and reference to architecture section for details.
+1 (if the newline in the definitions () still does render correctly...
How do you like your newlines cooked?
The definitions seem to render correctly. I'll merge this.

A local store of response messages and the subsystem that controls storage, retrieval, and deletion of messages in it.  <ins> A node that sends requests to servers and receives responses.  In RESTful IoT systems it's common for nodes to have more than one role (e.g., both server and client; see sec-architecture). </ins> The state kept by a client between requests.  This typically includes the currently processed representation, the set of active behalf of the client.  This may be useful, for example, when the client lacks the capability to make the request itself or to service the response from a cache in order to reduce response  <ins> time, network bandwidth, and energy consumption. </ins> A reverse proxy that provides an interface to a non-RESTful system such as legacy systems or alternative technologies such as Bluetooth ATT/GATT.  See also "Reverse Proxy".  <ins> A component, such as a link or a form, embedded in a representation that identifies a resource for future hypermedia interactions.  If the client engages in an interaction with the identified resource, the result may be a change to resource state and/or client state. </ins> A method where multiple identical requests with that method lead  <ins> to the same visible resource state as a single such request. </ins> A hypermedia control that enables a client to navigate between resources and thereby change the client state. A content negotiation mechanism where the server selects a representation based on the expressed preference of the client.  <ins> For example, an IoT application could send a request to a sensor </ins> with preferred media type "application/senml+json". A content negotiation mechanism where the client selects a multiple machines. A method that does not result in any state change on the origin  <ins> server when applied to a resource. </ins> A node that listens for requests, performs the requested operation and sends responses back to the clients. Section 4.3 of RFC7231 defines the set of methods in HTTP; Section 5.8 of RFC7252 defines the set of methods in CoAP.  As part of the Uniform Interface constraint, each method can have certain  <ins> properties that give guarantees to clients. Safe methods do not cause any state change on the origin server when applied to a resource.  For example, the GET method only returns a representation of the resource state but does not change the resource.  Thus, it is always safe for a client to retrieve a representation without affecting server-side state. </ins> Idempotent methods can be applied multiple times to the same resource while causing the same visible resource state as a single such  <ins> request.  For example, the PUT method replaces the state of a resource with a new state; replacing the state multiple times with the same new state still results in the same state for the resource. However, the response from the server can be different when the same idempotent method is used multiple times.  For example when DELETE is used twice on an existing resource, the first request would remove the association and return success acknowledgement whereas the second request would likely result in error response due to non-existing resource. The following lists the most relevant methods and gives a short explanation of their semantics. </ins> 3.5.1.


entry point for requesting the links hosted by servers with known or discovered addresses RFC6690.  6. Certain kinds of design problems are often recurring in variety of

I just gave this advice and it might be a good rule of thumb: Assuming human-readable representation formats (i.e., text-based not binary) and a client supporting the URI scheme used, a good rule of thumb for a good hypermedia-driven design is the following: A developer should only need an entry point URI to drive the application. All further information how to navigate through the application (links) and how to construct more complex requests (forms) are published by the server(s). There must be no need for additional, out-of-band information (e.g., API specification). For machines, a well-chosen set of information needs to be shared a priori to agree on machine-understandable semantics. (Agreeing on the exact semantics of terms for relation types and data elements will of course also help the developer.)
Yes, except that text-based doesn't mean human-readable (as exemplified by JSON -- in reality you need a formatting tool to look at any instances that aren't entirely trivial). More important is that the format is self-describing (and the tool you need to look at that not too obscure).
Yes, yes, all this is already in the assumptions to get a concise rule of thumb :) Next iteration: Assuming self-describing representation formats (i.e., human-readable with carefully chosen terms or processible by a formatting tool) and a client supporting the URI scheme used, a good rule of thumb for a good hypermedia-driven design is the following: A developer should only need an entry point URI to drive the application. All further information how to navigate through the application (links) and how to construct more complex requests (forms) are published by the server(s). There must be no need for additional, out-of-band information (e.g., API specification). For machines, a well-chosen set of information needs to be shared a priori to agree on machine-understandable semantics. (Agreeing on the exact semantics of terms for relation types and data elements will of course also help the developer.)
At least one little update :)Looks good. There's just a small problem: Nobody is doing it.

entry point for requesting the links hosted by servers with known or discovered addresses RFC6690. <ins> 5.4. Assuming self-describing representation formats (i.e., human-readable with carefully chosen terms or processible by a formatting tool) and a client supporting the URI scheme used, a good rule of thumb for a good hypermedia-driven design is the following: A developer should only need an entry point URI to drive the application.  All further information how to navigate through the application (links) and how to construct more complex requests (forms) are published by the server(s).  There must be no need for additional, out-of-band information (e.g., API specification). For machines, a well-chosen set of information needs to be shared a priori to agree on machine-understandable semantics.  Agreeing on the exact semantics of terms for relation types and data elements will of course also help the developer. </ins> 6. Certain kinds of design problems are often recurring in variety of


identical, these are two different resources, in two distinct namespaces.  The query parameters can be used to parametrize the resource.  For example, a GET request may use query parameters to request the server to send only certain kind data of the resource (i.e., filtering the

Tweak a bit the message of this: non-resource identifiers are not really part of the REST style, but a system usually needs them and the IETF has good means with URIs, which include URNs, and can be used for such non-resource identifiers (e.g., for physical asset, device, etc.).
We added the terms to the terminology, but should explain a bit more at the end of the URI sections.
Solved in

identical, these are two different resources, in two distinct namespaces. <ins> Some schemes are for URIs with main purpose as identifiers and hence are not dereferencable, e.g., the "urn" scheme can be used to construct unique names in registered namespaces.  In particular the "urn:dev" I-D.ietf-core-dev-urn details multiple ways for generating and representing endpoint identifiers of IoT devices. </ins> The query parameters can be used to parametrize the resource.  For example, a GET request may use query parameters to request the server to send only certain kind data of the resource (i.e., filtering the


servers respond requests, while origin servers are the ultimate recipient of requests that intent to modify resource state. <del> This improves simplicity and visibility, as it is clear which component started an interaction.  Furthermore, it improves modifiability through a clear separation of concerns. </del>  4.2. message traces contain all the information to understand the logged interactions.  Furthermore, the Stateless constraint enables caching. <del> The scaling properties of REST become particularly important in massive IoT scenarios where thousands or millions of IoT devices may attempt to interact with a system at the same time. </del>  4.3. requests can be answered even if the origin server is temporarily not available.  4.4. All RESTful APIs use the same, uniform interface independent of the URIs to identify resources <del> representation formats to represent (and retrieve and manipulate) resource state </del>  self-descriptive messages with a standard set of methods (e.g., GET, POST, PUT, DELETE with their guaranteed properties) hypermedia-driven, uniform interfaces allow clients and servers to evolve independently, and hence enable a system to evolve.  4.5. This constraint enforces that a client cannot see beyond the server such as load balancers can be introduced without changing the client side.  The clean separation of concerns helps with simplicity.  4.6. This principle enables origin servers to ship code to clients. format).  It also improves performance, as the server can provide code for local pre-processing before transferring the data.  5. Hypermedia-driven applications take advantage of hypermedia controls,

I extended the constraints sections (4.x) with a paragraph on the role for IoT. This does not introduce the MAY/SHOULD/MUST levels by Mike, but starts a basis for reaching such as conclusion.
From WISHI hackathon breakout: Capture that constrained networks may split data from metadata -- they are still bound together, but not always sent together.
Minimal text under 4.2 Stateless with PR
Was addressed with 41
Sections URL REST constraints URL
Many points were moved to the corresponding issue.
Let's pick the interesting discussion about the REST constraints here: I like the idea of revisiting the constraints. I am less sure of MAY, SHOULD, and MUST is the best taxonomy here. The experience from CoAP shows that it is most practical to have both roles simultaneously on a device. The division into client and server is still very useful because it declares which component had the initiative to do something, which is helpful for the visibility, but also accountability in "digital forensics"(?). When an IoT device has initiative (because there is a user, e.g., pressing a button, or it has some policy/rules/AI), it acts as a client. When a device offers a service, it is in server role. Both can happen on the same device There are not details in the mail why Mike picked SHOULD. One concern is of course efficiency and compact messages. Introducing state can help a lot here. I guess we need a proper evaluation of the desired properties enabled through statelessness and the advantages from breaking the constraint. Maybe helpful: FTP vs HTTP comparison about how filepath is conveyed. Yes, must be enabled whenever possible to protect constrained devices. This is a bit endangered through security, as transport layer security is currently predominant. Object security is quite interesting here. One should analyze how to enable caching for encrypted content that can be accessed by multiple participants. There should be guidance what trade-offs are made with respect to security, since cache-key options may not be encrypted... Caching usually only makes sense when the data is used by multiple/many participants. In the IoT, it might make sense to cache also individual data to protect constrained devices. Here the explanation of the methods would fit, and why one should respect their properties (no side effects with a GET...). CoAP is collecting more methods (FETCH, PATCH). Explain the different philosophy compared to RPC. Not sure how to understand this. Is this related to queries and FETCH? Hmm. This simply results from the uniform interface and context-dependent meaning of methods. Should this be reinterpreted somehow? An interesting point here could be the handling of events, that is state that is pushed from the server to the client and each individual notification is important and must not be lost (which makes them different from Observe notifications). Partly, this is statelessness. The other part is not relying on implicit out-of-band information. That is, proper use of Internet Media Types (and not just labeling everything app/json). IoT often has "dead ends" in the application: a sensor or actuator that does not link any further. For discovery, HATEOAS is still very important and I see a MUST there to enable loose coupling. Might this be related to sleepy devices? Explicit delegation? We have to find examples where this would apply, in particular since code is shipped from the server to the client. For instance, form checking is irrelevant in a Thing-to-Thing context. Preprocessing of data is interesting, but that should be done on the server side, which is the origin of data (e.g., a sensor).
Let's start with having a short discussion paragraph about this under Sections 4.x
waiting to be merged. Carsten/Kovatsch: First, fine session today in the t2trg room. thanks for that. Now, I am not suggesting we NOT adopt any of Fielding's Constraints (there are ten[1] to consider). But we should keep in mind that the ten requirements (think 2119 words) he selected are based on the 16 Architectural Properties of Key Interest [2] Fielding identified in Chapter IoT might not share all those desireable properties and some IoT ones might be missing from Fielding's WWW-inspired list. As a simple exercise, we can tick down the list of ten Constraints and see what 2119 words we want to invoke in our guidance document. here's my straw-man: Applying Fielidng's Constraints to IoT using RFC2119 Words Client-Server : MAY Stateless : SHOULD Cache : MUST Uniform Interface : MUST Identification of Resources : MUST Manipulation of Resources through Representations : MUST Self-Describing Messages : SHOULD Hypermedia as the Engine of App State : MAY Layered System : MUST Code-On-Demand : MAY Now, rather than say "is it RESTful" we can look at this list and start asking about the value of each of these ten constraints when it comes to IoT. Even more important, I suspect that there are constraints missing from this list that would be important to IoT. And we haven't even started on the list of Properties of Key Interest for IoT. BTW - naming is fun. I toss out IoTA for Internet of Things Architecture Cheers. [1] URL [2] URL mamund Mike Amundsen +1.859.757.1449 skype: mca.amundsen URL URL URL URL >
I read through the draft this week and am send along my initial comments for discussion. I like the notion of providing guidance to IoT developers and hope this document can be a cornerstone of that effort. Section 1. Introduction The intro says this document is meant as guidance for designing IoT systems that follow the principles of the REST style but those principles do not appear anywhere in this document nor are the appropriate sections in Fielding's dissertation noted. You come close with the following: "RESTful design facilitates many desirable features for a system, such as good scaling properties." This is where I'd refer to the full set of "Architectural Properties of Key Interest" from the dissertation[0]. I think it is then worthwhile to mention Fielding's requirements[1] (note these are WWW requirements, not IoT requirements) and the constraints [2] he selects to induce the properties. The remainder of the document focuses not on archtectural issues, but on protocol issues and is limited in even this area to focusing on HTTP and CoAP. I'm not sure if you want to provide protocol guidance or design guidance and will leave that up to discussion before making any other comments on this part of the doc. Section 2 Terminology - no comment for now Section 3.,1 - no comment Section 3.2 I suggest using the words "client state" and "resource state" in section of draft-keranen-t2trg-rest-iot. This more clearly identifies who is responsible for which state elements and avoids discussions about "application" and "session" definitions. Flow control MAY be managed from either the client ("please set this valve to position 3") or the resource server ("According to my configuration data, your client certificate only allows you to do one of the following...") Rather than try to define "session state" as flow control, I think it s better the establish lines of responsibility for client state and resource state. Section 3.3 I suggest the following updated wording for the opening: "Important part of designing a system is to model is as a set of resources which represent the "state of the system." Clients MAY be allowed to modify the state of resources by passing representations to the server (using request bodies). Servers manipulate the state of clients by passing representations of resources (using response bodies)." Section 3.4 I think most of the text here can be removed and replaced with a reference to existing RFCs on URI (e.g. 3986). I think it would also be good guidance to reference RFC7320 ("URI Design and Ownership")[3]. Section 3.5 & 3.6 I think a reference to the relevant RFCs is all that is needed here. Listing methods here runs the risk of becoming outdated as the list of valid methods changes over time. Section 4 I think it's a mistake to call out RFCs here since even media types will have security considerations and we can't know which types are in use (including new types as they are created). I suggest something about "Relavent RFCs" or something here. I appreciate the oppty to comment and look forward to any follow up discussions. thanks. [0] URL [1] URL [2] URL [3] URL mamund Mike Amundsen +1.859.757.1449 skype: mca.amundsen URL URL URL URL

servers respond requests, while origin servers are the ultimate recipient of requests that intent to modify resource state.  <ins> This improves simplicity and visibility (also for digital forensics), as it is clear which component started an interaction.  Furthermore, it improves modifiability through a clear separation of concerns. In IoT systems, endpoints often assume both roles of client and (origin) server simultaneously.  When an IoT device has initiative (because there is a user, e.g., pressing a button, or installed rules/policies), it acts as a client.  When a device offers a service, it is in server role. </ins> 4.2. message traces contain all the information to understand the logged interactions.  Furthermore, the Stateless constraint enables caching.  <ins> For IoT, the scaling properties of REST become particularly important.  Note that being self-contained does not necessarily mean that all information has to be inlined.  Constrained IoT devices may choose to externalize metadata and hypermedia controls using Web linking, so that only the dynamic content needs to be sent and the static content such as schemas or controls can be cached. </ins> 4.3. requests can be answered even if the origin server is temporarily not available. <ins> Caching usually only makes sense when the data is used by multiple participants.  In the IoT, however, it might make sense to cache also individual data to protect constrained devices.  Security often hinders the ability to cache responses.  For IoT systems, object security may be preferable over transport layer security, as it enables intermediaries to cache responses while preserving security. </ins> 4.4. All RESTful APIs use the same, uniform interface independent of the URIs to identify resources  <ins> representation formats to represent and manipulate resource state </ins> self-descriptive messages with a standard set of methods (e.g., GET, POST, PUT, DELETE with their guaranteed properties) hypermedia-driven, uniform interfaces allow clients and servers to evolve independently, and hence enable a system to evolve. <ins> For a large number of IoT applications, the hypermedia controls are mainly used for the discovery of resources, as they often serve sensor data.  Such resources are "dead ends", as they usually do not link any further and only have one form of interaction: fetching the sensor value.  For IoT, the critical parts of the Uniform Interface constraint are the descriptions of messages and representation formats used.  Simply using, for instance, "application/json" does not help machine clients to understand the semantics of the representation.  Yet defining very precise media types limits the re- usability and interoperability.  Representation formats such as SenML RFC8428 try to find a good trade-off between precision and re- usability.  Another approach is to combine a generic format such as JSON with syntactic as well as semantic annotations (see I- D.handrews-json-schema-validation and W3C-TD, resp.). </ins> 4.5. This constraint enforces that a client cannot see beyond the server such as load balancers can be introduced without changing the client side.  The clean separation of concerns helps with simplicity. <ins> IoT systems greatly benefit from this constraint, as it allows to effectively shield constrained devices behind intermediaries and is also the basis for gateways, which are used to integrate other (IoT) ecosystems. </ins> 4.6. This principle enables origin servers to ship code to clients. format).  It also improves performance, as the server can provide code for local pre-processing before transferring the data. <ins> As of today, code-on-demand has not been explored much in IoT systems.  Aspects to consider are that either one or both nodes are constrained and might not have the resources to host or dynamically fetch and execute such code.  Moreover, the origin server often has no understanding of the actual application a mashup client realizes. Still, code-on-demand can be useful for small polyfills, e.g., to decode payloads, and potentially other features in the future. </ins> 5. Hypermedia-driven applications take advantage of hypermedia controls,