draft-ietf-opsawg-ntf-09.txt   draft-ietf-opsawg-ntf-10.txt 
OPSAWG H. Song OPSAWG H. Song
Internet-Draft Futurewei Internet-Draft Futurewei
Intended status: Informational F. Qin Intended status: Informational F. Qin
Expires: 16 April 2022 China Mobile Expires: 12 May 2022 China Mobile
P. Martinez-Julia P. Martinez-Julia
NICT NICT
L. Ciavaglia L. Ciavaglia
Nokia Rakuten Mobile
A. Wang A. Wang
China Telecom China Telecom
13 October 2021 8 November 2021
Network Telemetry Framework Network Telemetry Framework
draft-ietf-opsawg-ntf-09 draft-ietf-opsawg-ntf-10
Abstract Abstract
Network telemetry is a technology for gaining network insight and Network telemetry is a technology for gaining network insight and
facilitating efficient and automated network management. It facilitating efficient and automated network management. It
encompasses various techniques for remote data generation, encompasses various techniques for remote data generation,
collection, correlation, and consumption. This document describes an collection, correlation, and consumption. This document describes an
architectural framework for network telemetry, motivated by architectural framework for network telemetry, motivated by
challenges that are encountered as part of the operation of networks challenges that are encountered as part of the operation of networks
and by the requirements that ensue. This document clarifies the and by the requirements that ensue. This document clarifies the
skipping to change at page 1, line 48 skipping to change at page 1, line 48
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/. Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on 16 April 2022. This Internet-Draft will expire on 12 May 2022.
Copyright Notice Copyright Notice
Copyright (c) 2021 IETF Trust and the persons identified as the Copyright (c) 2021 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents (https://trustee.ietf.org/ Provisions Relating to IETF Documents (https://trustee.ietf.org/
license-info) in effect on the date of publication of this document. license-info) in effect on the date of publication of this document.
Please review these documents carefully, as they describe your rights Please review these documents carefully, as they describe your rights
skipping to change at page 2, line 36 skipping to change at page 2, line 36
3.3. Challenges . . . . . . . . . . . . . . . . . . . . . . . 9 3.3. Challenges . . . . . . . . . . . . . . . . . . . . . . . 9
3.4. Network Telemetry . . . . . . . . . . . . . . . . . . . . 10 3.4. Network Telemetry . . . . . . . . . . . . . . . . . . . . 10
3.5. The Necessity of a Network Telemetry Framework . . . . . 13 3.5. The Necessity of a Network Telemetry Framework . . . . . 13
4. Network Telemetry Framework . . . . . . . . . . . . . . . . . 14 4. Network Telemetry Framework . . . . . . . . . . . . . . . . . 14
4.1. Top Level Modules . . . . . . . . . . . . . . . . . . . . 14 4.1. Top Level Modules . . . . . . . . . . . . . . . . . . . . 14
4.1.1. Management Plane Telemetry . . . . . . . . . . . . . 18 4.1.1. Management Plane Telemetry . . . . . . . . . . . . . 18
4.1.2. Control Plane Telemetry . . . . . . . . . . . . . . . 18 4.1.2. Control Plane Telemetry . . . . . . . . . . . . . . . 18
4.1.3. Forwarding Plane Telemetry . . . . . . . . . . . . . 19 4.1.3. Forwarding Plane Telemetry . . . . . . . . . . . . . 19
4.1.4. External Data Telemetry . . . . . . . . . . . . . . . 21 4.1.4. External Data Telemetry . . . . . . . . . . . . . . . 21
4.2. Second Level Function Components . . . . . . . . . . . . 22 4.2. Second Level Function Components . . . . . . . . . . . . 22
4.3. Data Acquisition Mechanism and Type Abstraction . . . . . 23 4.3. Data Acquisition Mechanism and Type Abstraction . . . . . 24
4.4. Mapping Existing Mechanisms into the Framework . . . . . 25 4.4. Mapping Existing Mechanisms into the Framework . . . . . 26
5. Evolution of Network Telemetry Applications . . . . . . . . . 26 5. Evolution of Network Telemetry Applications . . . . . . . . . 27
6. Security Considerations . . . . . . . . . . . . . . . . . . . 26 6. Security Considerations . . . . . . . . . . . . . . . . . . . 27
7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 28 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 29
8. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 28 8. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 29
9. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 28 9. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 30
10. Informative References . . . . . . . . . . . . . . . . . . . 29 10. Informative References . . . . . . . . . . . . . . . . . . . 30
Appendix A. A Survey on Existing Network Telemetry Techniques . 33 Appendix A. A Survey on Existing Network Telemetry Techniques . 35
A.1. Management Plane Telemetry . . . . . . . . . . . . . . . 33 A.1. Management Plane Telemetry . . . . . . . . . . . . . . . 35
A.1.1. Push Extensions for NETCONF . . . . . . . . . . . . . 34 A.1.1. Push Extensions for NETCONF . . . . . . . . . . . . . 36
A.1.2. gRPC Network Management Interface . . . . . . . . . . 34 A.1.2. gRPC Network Management Interface . . . . . . . . . . 36
A.2. Control Plane Telemetry . . . . . . . . . . . . . . . . . 34 A.2. Control Plane Telemetry . . . . . . . . . . . . . . . . . 36
A.2.1. BGP Monitoring Protocol . . . . . . . . . . . . . . . 34 A.2.1. BGP Monitoring Protocol . . . . . . . . . . . . . . . 36
A.3. Data Plane Telemetry . . . . . . . . . . . . . . . . . . 35 A.3. Data Plane Telemetry . . . . . . . . . . . . . . . . . . 37
A.3.1. The Alternate Marking (AM) technology . . . . . . . . 35 A.3.1. The Alternate Marking (AM) technology . . . . . . . . 37
A.3.2. Dynamic Network Probe . . . . . . . . . . . . . . . . 36 A.3.2. Dynamic Network Probe . . . . . . . . . . . . . . . . 38
A.3.3. IP Flow Information Export (IPFIX) protocol . . . . . 37 A.3.3. IP Flow Information Export (IPFIX) Protocol . . . . . 39
A.3.4. In-Situ OAM . . . . . . . . . . . . . . . . . . . . . 37 A.3.4. In-Situ OAM . . . . . . . . . . . . . . . . . . . . . 39
A.3.5. Postcard Based Telemetry . . . . . . . . . . . . . . 37 A.3.5. Postcard Based Telemetry . . . . . . . . . . . . . . 39
A.4. External Data and Event Telemetry . . . . . . . . . . . . 37 A.3.6. Existing OAM for Specific Data Planes . . . . . . . . 39
A.4.1. Sources of External Events . . . . . . . . . . . . . 38 A.4. External Data and Event Telemetry . . . . . . . . . . . . 40
A.4.2. Connectors and Interfaces . . . . . . . . . . . . . . 39 A.4.1. Sources of External Events . . . . . . . . . . . . . 40
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 39 A.4.2. Connectors and Interfaces . . . . . . . . . . . . . . 41
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 41
1. Introduction 1. Introduction
Network visibility is the ability of management tools to see the Network visibility is the ability of management tools to see the
state and behavior of a network, which is essential for successful state and behavior of a network, which is essential for successful
network operation. Network Telemetry revolves around network data network operation. Network Telemetry revolves around network data
that can help provide insights about the current state of the that can help provide insights about the current state of the
network, including network devices, forwarding, control, and network, including network devices, forwarding, control, and
management planes, and that can be generated and obtained through a management planes, and that can be generated and obtained through a
variety of techniques, including but not limited to network variety of techniques, including but not limited to network
skipping to change at page 5, line 10 skipping to change at page 5, line 10
structured data. structured data.
gRPC: gRPC Remote Procedure Call, an open source high performance gRPC: gRPC Remote Procedure Call, an open source high performance
RPC framework that gNMI is based on. See [grpc] for details. RPC framework that gNMI is based on. See [grpc] for details.
IPFIX: IP Flow Information Export Protocol, specified in [RFC7011]. IPFIX: IP Flow Information Export Protocol, specified in [RFC7011].
IOAM: In-situ OAM, a dataplane on-path telemetry technique. IOAM: In-situ OAM, a dataplane on-path telemetry technique.
JSON: An open standard file format and data interchange format that JSON: An open standard file format and data interchange format that
uses human-readable text to store and transmit data objects. uses human-readable text to store and transmit data objects,
specified in [RFC8259].
MIB: Management Information Base, a database used for managing the MIB: Management Information Base, a database used for managing the
entities in a network. entities in a network.
NETCONF: Network Configuration Protocol, specified in [RFC6241]. NETCONF: Network Configuration Protocol, specified in [RFC6241].
NetFlow: A Cisco protocol for flow record collecting, described in NetFlow: A Cisco protocol for flow record collecting, described in
[RFC3594]. [RFC3594].
Network Telemetry: The process and instrumentation for acquiring and Network Telemetry: The process and instrumentation for acquiring and
skipping to change at page 5, line 40 skipping to change at page 5, line 41
OAM: Operations, Administration, and Maintenance. A group of OAM: Operations, Administration, and Maintenance. A group of
network management functions that provide network fault network management functions that provide network fault
indication, fault localization, performance information, and data indication, fault localization, performance information, and data
and diagnosis functions. Most conventional network monitoring and diagnosis functions. Most conventional network monitoring
techniques and protocols belong to network OAM. techniques and protocols belong to network OAM.
PBT: Postcard-Based Telemetry, a dataplane on-path telemetry PBT: Postcard-Based Telemetry, a dataplane on-path telemetry
technique. technique.
RESTCONF: An HTTP-based protocol that provides a programmatic
interface for accessing data defined in YANG, using the datastore
concepts defined in NETCONF, as specified in [RFC8040].
SMIv2 Structure of Management Information Version 2, defining MIB SMIv2 Structure of Management Information Version 2, defining MIB
objects, specified in [RFC2578]. objects, specified in [RFC2578].
SNMP: Simple Network Management Protocol. Version 1 and 2 are SNMP: Simple Network Management Protocol. Version 1 and 2 are
specified in [RFC1157] and [RFC3416], respectively. specified in [RFC1157] and [RFC3416], respectively.
XML; Extensible Markup Language is a markup language for data
encoding that is both human-readable and machine-readable,
specified by W3C [xml].
YANG: YANG is a data modeling language for the definition of data YANG: YANG is a data modeling language for the definition of data
sent over network management protocols such as the NETCONF and sent over network management protocols such as the NETCONF and
RESTCONF. YANG is defined in [RFC6020] and [RFC7950]. RESTCONF. YANG is defined in [RFC6020] and [RFC7950].
YANG ECA A YANG model for Event-Condition-Action policies, defined YANG ECA A YANG model for Event-Condition-Action policies, defined
in [I-D.wwx-netmod-event-yang]. in [I-D.wwx-netmod-event-yang].
YANG-Push: A mechanism that allows subscriber applications to YANG-Push: A mechanism that allows subscriber applications to
request a stream of updates from a YANG datastore on a network request a stream of updates from a YANG datastore on a network
device. Details are specified in [RFC8641] and [RFC8639]. device. Details are specified in [RFC8641] and [RFC8639].
3. Background 3. Background
The term "big data" is used to describe the extremely large volume of The term "big data" is used to describe the extremely large volume of
data sets that can be analyzed computationally to reveal patterns, data sets that can be analyzed computationally to reveal patterns,
trends, and associations. Networks are undoubtedly a source of big trends, and associations. Networks are undoubtedly a source of big
data because of their scale and the volume of network traffic they data because of their scale and the volume of network traffic they
forward. It is easy to see that network operations can benefit from forward. When a network's endpoints do not represent individual
network big data to gather insights into flows without breaching users (e.g. in industrial, datacenter, and infrastructure contexts),
privacy. network operations can often benefit from large-scale data collection
without breaching user privacy.
Today one can access advanced big data analytics capability through a Today one can access advanced big data analytics capability through a
plethora of commercial and open source platforms (e.g., Apache plethora of commercial and open source platforms (e.g., Apache
Hadoop), tools (e.g., Apache Spark), and techniques (e.g., machine Hadoop), tools (e.g., Apache Spark), and techniques (e.g., machine
learning). Thanks to the advance of computing and storage learning). Thanks to the advance of computing and storage
technologies, network big data analytics gives network operators an technologies, network big data analytics gives network operators an
opportunity to gain network insights and move towards network opportunity to gain network insights and move towards network
autonomy. Some operators start to explore the application of autonomy. Some operators start to explore the application of
Artificial Intelligence (AI) to make sense of network data. Software Artificial Intelligence (AI) to make sense of network data. Software
tools can use the network data to detect and react on network faults, tools can use the network data to detect and react on network faults,
skipping to change at page 12, line 35 skipping to change at page 12, line 35
real-time processing is required. real-time processing is required.
* In-band Data Collection: In addition to the passive and active * In-band Data Collection: In addition to the passive and active
data collection approaches, the new hybrid approach allows to data collection approaches, the new hybrid approach allows to
directly collect data for any target flow on its entire forwarding directly collect data for any target flow on its entire forwarding
path [I-D.song-opsawg-ifit-framework]. path [I-D.song-opsawg-ifit-framework].
It is worth noting that a network telemetry system should not be It is worth noting that a network telemetry system should not be
intrusive to normal network operations by avoiding the pitfall of the intrusive to normal network operations by avoiding the pitfall of the
"observer effect". That is, it should not change the network "observer effect". That is, it should not change the network
behavior and affect the forwarding performance. Otherwise, the whole behavior and affect the forwarding performance. Moreover, high-
purpose of network telemetry is compromised. volume telemetry traffic may cause network congestion unless proper
isolation or traffic engineering techniques are in place, or
congestion control mechanisms ensure that telemetry traffic backs off
if it exceeds the network capacity. [RFC8084] and [RFC8085] are
relevant Best Current Practices (BCP) in this space.
Although in many cases a system for network telemetry involves a Although in many cases a system for network telemetry involves a
remote data collecting and consuming entity, it is important to remote data collecting and consuming entity, it is important to
understand that there are no inherent assumptions about how a system understand that there are no inherent assumptions about how a system
should be architected. Telemetry data producers and consumers can should be architected. While a network architecture with centralized
work in distributed or peer-to-peer fashions rather than assuming a controller (e.g., SDN) seems a natural fit for network telemetry,
centralized data consuming entity. In such cases, a network node can network telemetry can work in distributed fashions as well. For
be the direct consumer of telemetry data from other nodes. example, telemetry data producers and consumers can have a peer-to-
peer relationship, in which a network node can be the direct consumer
of telemetry data from other nodes.
3.5. The Necessity of a Network Telemetry Framework 3.5. The Necessity of a Network Telemetry Framework
Network data analytics and machine-learning technologies are applied Network data analytics and machine-learning technologies are applied
for network operation automation, relying on abundant and coherent for network operation automation, relying on abundant and coherent
data from networks. Data acquisition that is limited to a single data from networks. Data acquisition that is limited to a single
source and static in nature will in many cases not be sufficient to source and static in nature will in many cases not be sufficient to
meet an application's telemetry data needs. As a result, multiple meet an application's telemetry data needs. As a result, multiple
data sources, involving a variety of techniques and standards, will data sources, involving a variety of techniques and standards, will
need to be integrated. It is desirable to have a framework that need to be integrated. It is desirable to have a framework that
skipping to change at page 16, line 22 skipping to change at page 16, line 22
capabilities, different choices of data model, encoding, and capabilities, different choices of data model, encoding, and
transport method are made to balance the performance and cost. For transport method are made to balance the performance and cost. For
example, the forwarding chip has high throughput but limited capacity example, the forwarding chip has high throughput but limited capacity
for processing complex data and maintaining states, while the main for processing complex data and maintaining states, while the main
control CPU is capable of complex data and state processing, but has control CPU is capable of complex data and state processing, but has
limited bandwidth for high throughput data. As a result, the limited bandwidth for high throughput data. As a result, the
suitable telemetry protocol for each module can be different. Some suitable telemetry protocol for each module can be different. Some
representative techniques are shown in the corresponding table blocks representative techniques are shown in the corresponding table blocks
to highlight the technical diversity of these modules. Note that the to highlight the technical diversity of these modules. Note that the
selected techniques just reflect the de facto state of the art and selected techniques just reflect the de facto state of the art and
are not exhaustive. The key point is that one cannot expect to use a are by no means exhaustive (e.g., IPFIX can also be implemented over
universal protocol to cover all the network telemetry requirements. TCP and SCTP but that is not recommended for forwarding plane). The
key point is that one cannot expect to use a universal protocol to
cover all the network telemetry requirements.
+---------+--------------+--------------+---------------+-----------+ +-----------+-------------+-------------+--------------+----------+
| Module | Management | Control | Forwarding | External | | Module |Management |Control |Forwarding |External |
| | Plane | Plane | Plane | Data | | |Plane |Plane |Plane |Data |
+---------+--------------+--------------+---------------+-----------+ +-----------+-------------+-------------+--------------+----------+
|Object | config. & | control | flow & packet | terminal, | |Object |config. & |control |flow & packet |terminal, |
| | operation | protocol & | QoS, traffic | social & | | |operation |protocol & |QoS, traffic |social & |
| | state | signaling, | stat., buffer | environ- | | |state |signaling, |stat., buffer |environ- |
| | | RIB | & queue stat.,| mental | | | |RIB |& queue stat.,|mental |
| | | | ACL, FIB | | | | | |ACL, FIB | |
+---------+--------------+--------------+---------------+-----------+ +-----------+-------------+-------------+--------------+----------+
|Export | main control | main control | fwding chip | various | |Export |main control |main control |fwding chip |various |
|Location | CPU | CPU, | or linecard | | |Location |CPU |CPU, |or linecard | |
| | | linecard CPU | CPU; main | | | | |linecard CPU |CPU; main | |
| | | or forwarding| control CPU | | | | |or forwarding|control CPU | |
| | | chip | unlikely | | | | |chip |unlikely | |
+---------+--------------+--------------+---------------+-----------+ +-----------+-------------+-------------+--------------+----------+
|Data | YANG, MIB, | YANG, | template, | YANG, | |Data |YANG, MIB, |YANG, |template, |YANG, |
|Model | syslog | custom | YANG, | custom | |Model |syslog |custom |YANG, |custom |
| | | | custom | | | | | |custom | |
+---------+--------------+--------------+---------------+-----------+ +-----------+-------------+-------------+--------------+----------+
|Data | GPB, JSON, | GPB, JSON, | plain | GPB, JSON | |Data |GPB, JSON, |GPB, JSON, |plain |GPB, JSON |
|Encoding | XML | XML, plain | | XML, plain| |Encoding |XML |XML, plain | |XML, plain|
+---------+--------------+--------------+---------------+-----------+ +-----------+-------------+-------------+--------------+----------+
|Protocol | gRPC,NETCONF,| gRPC,NETCONF,| IPFIX, mirror,| gRPC | |Application|gRPC,NETCONF,|gRPC,NETCONF,|IPFIX, mirror,|gRPC |
| | | IPFIX, mirror| gRPC, NETFLOW | | |Protocol |RESTCONF |IPFIX, mirror|gRPC, NETFLOW | |
+---------+--------------+--------------+---------------+-----------+ +-----------+-------------+-------------+--------------+----------+
|Transport| HTTP, TCP | HTTP, TCP, | UDP | HTTP,TCP | |Data |HTTP, TCP |HTTP, TCP, |UDP |HTTP,TCP |
| | | UDP | | UDP | |Transport | |UDP | |UDP |
+---------+--------------+--------------+---------------+-----------+ +-----------+-------------+-------------+--------------+----------+
Figure 2: Comparison of the Data Object Modules Figure 2: Comparison of the Data Object Modules
Note that the interaction with the applications that consume network Note that the interaction with the applications that consume network
telemetry data can be indirect. Some in-device data transfer is telemetry data can be indirect. Some in-device data transfer is
possible. For example, in the management plane telemetry, the possible. For example, in the management plane telemetry, the
management plane will need to acquire data from the data plane. Some management plane will need to acquire data from the data plane. Some
operational states can only be derived from data plane data sources operational states can only be derived from data plane data sources
such as the interface status and statistics. As another example, such as the interface status and statistics. As another example,
obtaining control plane telemetry data may require the ability to obtaining control plane telemetry data may require the ability to
skipping to change at page 18, line 38 skipping to change at page 18, line 38
and normalize data encoding and transformation. and normalize data encoding and transformation.
* High Speed Data Transport: In order to keep up with the velocity * High Speed Data Transport: In order to keep up with the velocity
of information, a server needs to be able to send large amounts of of information, a server needs to be able to send large amounts of
data at high frequency. Compact encoding formats or data data at high frequency. Compact encoding formats or data
compression schemes are needed to reduce the quantity of data and compression schemes are needed to reduce the quantity of data and
improve the data transport efficiency. The subscription mode, by improve the data transport efficiency. The subscription mode, by
replacing the query mode, reduces the interactions between clients replacing the query mode, reduces the interactions between clients
and servers and helps to improve the server's efficiency. and servers and helps to improve the server's efficiency.
* Network Congestion Avoidance: The application must protect the
network from congestion by congestion control mechanisms or at
least circuit breakers. [RFC8084] and [RFC8085] provide some
solutions in this space.
4.1.2. Control Plane Telemetry 4.1.2. Control Plane Telemetry
The control plane telemetry refers to the health condition monitoring The control plane telemetry refers to the health condition monitoring
of different network control protocols at all layers of the protocol of different network control protocols at all layers of the protocol
stack. Keeping track of the operational status of these protocols is stack. Keeping track of the operational status of these protocols is
beneficial for detecting, localizing, and even predicting various beneficial for detecting, localizing, and even predicting various
network issues, as well as network optimization, in real-time and network issues, as well as network optimization, in real-time and
with fine granularity. Some particular challenges and issues faced with fine granularity. Some particular challenges and issues faced
by the control plane telemetry are as follows: by the control plane telemetry are as follows:
skipping to change at page 19, line 26 skipping to change at page 19, line 32
* An example of the control plane telemetry is the BGP monitoring * An example of the control plane telemetry is the BGP monitoring
protocol (BMP), it is currently used for monitoring the BGP routes protocol (BMP), it is currently used for monitoring the BGP routes
and enables rich applications, such as BGP peer analysis, AS and enables rich applications, such as BGP peer analysis, AS
analysis, prefix analysis, and security analysis. However, the analysis, prefix analysis, and security analysis. However, the
monitoring of other layers, protocols and the cross-layer, cross- monitoring of other layers, protocols and the cross-layer, cross-
protocol KPI correlations are still in their infancy (e.g., IGP protocol KPI correlations are still in their infancy (e.g., IGP
monitoring is not as extensive as BMP), which require further monitoring is not as extensive as BMP), which require further
research. research.
* The requirement and solutions for network congestion avoidance are
also applicable to the control plane telemetry.
4.1.3. Forwarding Plane Telemetry 4.1.3. Forwarding Plane Telemetry
An effective forwarding plane telemetry system relies on the data An effective forwarding plane telemetry system relies on the data
that the network device can expose. The quality, quantity, and that the network device can expose. The quality, quantity, and
timeliness of data must meet some stringent requirements. This timeliness of data must meet some stringent requirements. This
raises some challenges to the network data plane devices where the raises some challenges to the network data plane devices where the
first-hand data originates. first-hand data originates.
* A data plane device's main function is user traffic processing and * A data plane device's main function is user traffic processing and
forwarding. While supporting network visibility is important, the forwarding. While supporting network visibility is important, the
skipping to change at page 20, line 15 skipping to change at page 20, line 26
* The data should be structured and labeled, and easy for * The data should be structured and labeled, and easy for
applications to parse and consume. At the same time, the data applications to parse and consume. At the same time, the data
types needed by applications can vary significantly. The data types needed by applications can vary significantly. The data
plane devices need to provide enough flexibility and plane devices need to provide enough flexibility and
programmability to support the precise data provision for programmability to support the precise data provision for
applications. applications.
* The data plane telemetry should support incremental deployment and * The data plane telemetry should support incremental deployment and
work even though some devices are unaware of the system. work even though some devices are unaware of the system.
* The requirement and solutions for network congestion avoidance are
also applicable to the forwarding plane telemetry.
Although not specific to the forwarding plane, these challenges are Although not specific to the forwarding plane, these challenges are
more difficult to the forwarding plane because of the limited more difficult to the forwarding plane because of the limited
resource and flexibility. Data plane programmability is essential to resource and flexibility. Data plane programmability is essential to
support network telemetry. Newer data plane forwarding chips are support network telemetry. Newer data plane forwarding chips are
equipped with advanced telemetry features and provide flexibility to equipped with advanced telemetry features and provide flexibility to
support customized telemetry functions. support customized telemetry functions.
Technique Taxonomy: concerning about how one instruments the Technique Taxonomy: concerning about how one instruments the
telemetry, there can be multiple possible dimensions to classify the telemetry, there can be multiple possible dimensions to classify the
forwarding plane telemetry techniques. forwarding plane telemetry techniques.
* Active, Passive, and Hybrid: This dimension concerns about the * Active, Passive, and Hybrid: This dimension concerns about the
end-to-end measurement. Active and passive methods (as well as end-to-end measurement. Active and passive methods (as well as
the hybrid types) are well documented in [RFC7799]. Passive the hybrid types) are well documented in [RFC7799]. Passive
methods include TCPDUMP, IPFIX [RFC7011], sflow, and traffic methods include TCPDUMP, IPFIX [RFC7011], sflow, and traffic
mirroring. These methods usually have low data coverage. The mirroring. These methods usually have low data coverage. The
bandwidth cost is very high in order to improve the data coverage. bandwidth cost is very high in order to improve the data coverage.
On the other hand, active methods include Ping, OWAMP [RFC4656], On the other hand, active methods include Ping, OWAMP [RFC4656],
TWAMP [RFC5357], and Cisco's SLA Protocol [RFC6812]. These TWAMP [RFC5357], STAMP [RFC8762], and Cisco's SLA Protocol
methods are intrusive and only provide indirect network [RFC6812]. These methods are intrusive and only provide indirect
measurements. Hybrid methods, including in-situ OAM network measurements. Hybrid methods, including in-situ OAM
[I-D.ietf-ippm-ioam-data], Alternate-Marking (AM) [RFC8321], and [I-D.ietf-ippm-ioam-data], Alternate-Marking (AM) [RFC8321], and
Multipoint Alternate Marking [I-D.ietf-ippm-multipoint-alt-mark], Multipoint Alternate Marking [I-D.ietf-ippm-multipoint-alt-mark],
provide a well-balanced and more flexible approach. However, provide a well-balanced and more flexible approach. However,
these methods are also more complex to implement. these methods are also more complex to implement.
* In-Band and Out-of-Band: Telemetry data carried in user packets * In-Band and Out-of-Band: Telemetry data carried in user packets
before being exported to a data collector is considered in-band before being exported to a data collector is considered in-band
(e.g., in-situ OAM [I-D.ietf-ippm-ioam-data]). Telemetry data (e.g., in-situ OAM [I-D.ietf-ippm-ioam-data]). Telemetry data
that is directly exported to a data collector without modifying that is directly exported to a data collector without modifying
user packets is considered out-of-band (e.g., the postcard-based user packets is considered out-of-band (e.g., the postcard-based
skipping to change at page 22, line 5 skipping to change at page 22, line 29
perform the notifications, their timeliness is assumed. However, perform the notifications, their timeliness is assumed. However,
once messages have been dispatched, they must be quickly collected once messages have been dispatched, they must be quickly collected
and inserted into the control plane with variable priority, which and inserted into the control plane with variable priority, which
is higher for important sources and events and lower for secondary is higher for important sources and events and lower for secondary
ones. ones.
* The schema used by external detectors must be easily adopted by * The schema used by external detectors must be easily adopted by
current and future devices and applications. Therefore, it must current and future devices and applications. Therefore, it must
be easily mapped to current data models, such as in terms of YANG. be easily mapped to current data models, such as in terms of YANG.
* As the communication with external entities outside the boundary
of a provider network may be realized over the Internet, the risk
of congestion is even more relevant in this context and proper
counter-measures must be taken. Solutions such as network
transport circuit breakers are needed as well.
Organizing both internal and external telemetry information together Organizing both internal and external telemetry information together
will be key for the general exploitation of the management will be key for the general exploitation of the management
possibilities of current and future network systems, as reflected in possibilities of current and future network systems, as reflected in
the incorporation of cognitive capabilities to new hardware and the incorporation of cognitive capabilities to new hardware and
software (virtual) elements. software (virtual) elements.
4.2. Second Level Function Components 4.2. Second Level Function Components
The telemetry module at each plane can be further partitioned into The telemetry module at each plane can be further partitioned into
five distinct conceptual components: five distinct conceptual components:
skipping to change at page 25, line 16 skipping to change at page 26, line 16
| Event-triggered Data |<----+ Streaming Data | | Event-triggered Data |<----+ Streaming Data |
+-------+---+----------+ +-----+---+-------+ +-------+---+----------+ +-----+---+-------+
| | | | | | | |
| | | | | | | |
| | +--------------+ | | | | +--------------+ | |
| +-->| Derived Data |<--+ | | +-->| Derived Data |<--+ |
| +------+------ + | | +------+------ + |
| | | | | |
| V | | V |
| +--------------+ | | +--------------+ |
+------>| Simple Data |<------+ +------>| Simple Data |<------+
+--------------+ +--------------+
Figure 4: Data Type Relationship Figure 4: Data Type Relationship
Subscription usually deals with event-triggered data and streaming Subscription usually deals with event-triggered data and streaming
data, and query usually deals with simple data and derived data. But data, and query usually deals with simple data and derived data. But
the other ways are also possible. Advanced network telemetry the other ways are also possible. Advanced network telemetry
techniques are designed mainly for event-triggered or streaming data techniques are designed mainly for event-triggered or streaming data
subscription, and derived data query. subscription, and derived data query.
skipping to change at page 25, line 44 skipping to change at page 26, line 44
Also, some comprehensive protocols and techniques may cover multiple Also, some comprehensive protocols and techniques may cover multiple
aspects or modules of the framework, so a name in a block only aspects or modules of the framework, so a name in a block only
emphasizes one particular characteristic of it. More details about emphasizes one particular characteristic of it. More details about
some listed mechanisms can be found in Appendix A. some listed mechanisms can be found in Appendix A.
+-------------+-----------------+---------------+--------------+ +-------------+-----------------+---------------+--------------+
| | Management | Control | Forwarding | | | Management | Control | Forwarding |
| | Plane | Plane | Plane | | | Plane | Plane | Plane |
+-------------+-----------------+---------------+--------------+ +-------------+-----------------+---------------+--------------+
| data config.| gNMI, NETCONF, | gNMI, NETCONF,| NETCONF, | | data config.| gNMI, NETCONF, | gNMI, NETCONF,| NETCONF, |
| & subscribe | SNMP, YANG-Push | YANG-Push | YANG-Push | | & subscribe | RESTCONF, SNMP, | RESTCONF, | RESTCONF, |
| | YANG-Push | YANG-Push | YANG-Push |
+-------------+-----------------+---------------+--------------+ +-------------+-----------------+---------------+--------------+
| data gen. & | MIB, | YANG | IOAM, PSAMP | | data gen. & | MIB, | YANG | IOAM, PSAMP |
| process | YANG | | PBT, AM, | | process | YANG | | PBT, AM, |
+-------------+-----------------+---------------+--------------+ +-------------+-----------------+---------------+--------------+
| data encode.| gRPC, HTTP, TCP | BMP, TCP | IPFIX, UDP | | data encode.| gRPC, HTTP, TCP | BMP, TCP | IPFIX, UDP |
| & export | | | | | & export | | | |
+-------------+-----------------+---------------+--------------+ +-------------+-----------------+---------------+--------------+
Figure 5: Existing Work Mapping Figure 5: Existing Work Mapping
5. Evolution of Network Telemetry Applications 5. Evolution of Network Telemetry Applications
skipping to change at page 28, line 19 skipping to change at page 29, line 19
Access Control and Event-Condition-Action policies. Also, potential Access Control and Event-Condition-Action policies. Also, potential
conflicts between network telemetry mechanisms must be detected conflicts between network telemetry mechanisms must be detected
accurately and resolved quickly to avoid unnecessary network accurately and resolved quickly to avoid unnecessary network
telemetry traffic propagation escalating into an unintended or telemetry traffic propagation escalating into an unintended or
intended denial of service attack. intended denial of service attack.
Further study of the security issues will be required, and it is Further study of the security issues will be required, and it is
expected that the security mechanisms and protocols are developed and expected that the security mechanisms and protocols are developed and
deployed along with a network telemetry system. deployed along with a network telemetry system.
In addition to security, privacy is also an important issue. Network In addition to security, privacy is also an important issue. Large-
telemetry means to improve the network operation which can ultimately scale network data collection is a major threat to user privacy
benefit end user's quality of experience. The network operators must [RFC7258]. The Network Telemetry Framework is not applicable to
be held accountable and strive for a balance between managing the networks whose endpoints represent individual users, such as general-
network and maintaining the user privacy of that network. purpose access networks. Any collection or retention of data in
those networks must be tightly limited to protect user privacy.
7. IANA Considerations 7. IANA Considerations
This document includes no request to IANA. This document includes no request to IANA.
8. Contributors 8. Contributors
The other contributors of this document are listed as follows. The other contributors of this document are listed as follows.
* Tianran Zhou * Tianran Zhou
skipping to change at page 28, line 49 skipping to change at page 30, line 9
* Daniel King * Daniel King
* Adrian Farrel * Adrian Farrel
* Alexander Clemm * Alexander Clemm
9. Acknowledgments 9. Acknowledgments
We would like to thank Rob Wilton, Greg Mirsky, Randy Presuhn, Joe We would like to thank Rob Wilton, Greg Mirsky, Randy Presuhn, Joe
Clarke, Victor Liu, James Guichard, Uri Blumenthal, Giuseppe Clarke, Victor Liu, James Guichard, Uri Blumenthal, Giuseppe
Fioccola, Yunan Gu, Parviz Yegani, Young Lee, Qin Wu, and many others Fioccola, Yunan Gu, Parviz Yegani, Young Lee, Qin Wu, Gyan Mishra,
who have provided helpful comments and suggestions to improve this Ben Schwartz, Alexey Melnikov, Michael Scharf, and many others who
have provided helpful comments and suggestions to improve this
document. document.
10. Informative References 10. Informative References
[gnmi] "gNMI - gRPC Network Management Interface", [gnmi] "gNMI - gRPC Network Management Interface",
<https://github.com/openconfig/reference/tree/master/rpc/ <https://github.com/openconfig/reference/tree/master/rpc/
gnmi>. gnmi>.
[grpc] "gPPC, A high performance, open-source universal RPC [grpc] "gPPC, A high performance, open-source universal RPC
framework", <https://grpc.io>. framework", <https://grpc.io>.
skipping to change at page 29, line 33 skipping to change at page 30, line 42
Evens, T., Bayraktar, S., Bhardwaj, M., and P. Lucente, Evens, T., Bayraktar, S., Bhardwaj, M., and P. Lucente,
"Support for Local RIB in BGP Monitoring Protocol (BMP)", "Support for Local RIB in BGP Monitoring Protocol (BMP)",
Work in Progress, Internet-Draft, draft-ietf-grow-bmp- Work in Progress, Internet-Draft, draft-ietf-grow-bmp-
local-rib-13, 31 August 2021, local-rib-13, 31 August 2021,
<https://www.ietf.org/archive/id/draft-ietf-grow-bmp- <https://www.ietf.org/archive/id/draft-ietf-grow-bmp-
local-rib-13.txt>. local-rib-13.txt>.
[I-D.ietf-ippm-ioam-data] [I-D.ietf-ippm-ioam-data]
Brockners, F., Bhandari, S., and T. Mizrahi, "Data Fields Brockners, F., Bhandari, S., and T. Mizrahi, "Data Fields
for In-situ OAM", Work in Progress, Internet-Draft, draft- for In-situ OAM", Work in Progress, Internet-Draft, draft-
ietf-ippm-ioam-data-15, 3 October 2021, ietf-ippm-ioam-data-16, 8 November 2021,
<https://www.ietf.org/archive/id/draft-ietf-ippm-ioam- <https://www.ietf.org/archive/id/draft-ietf-ippm-ioam-
data-15.txt>. data-16.txt>.
[I-D.ietf-ippm-multipoint-alt-mark] [I-D.ietf-ippm-multipoint-alt-mark]
Fioccola, G., Cociglio, M., Sapio, A., and R. Sisto, Fioccola, G., Cociglio, M., Sapio, A., and R. Sisto,
"Multipoint Alternate-Marking Method for Passive and "Multipoint Alternate-Marking Method for Passive and
Hybrid Performance Monitoring", Work in Progress, Hybrid Performance Monitoring", Work in Progress,
Internet-Draft, draft-ietf-ippm-multipoint-alt-mark-09, 23 Internet-Draft, draft-ietf-ippm-multipoint-alt-mark-09, 23
March 2020, <https://www.ietf.org/archive/id/draft-ietf- March 2020, <https://www.ietf.org/archive/id/draft-ietf-
ippm-multipoint-alt-mark-09.txt>. ippm-multipoint-alt-mark-09.txt>.
[I-D.ietf-netconf-distributed-notif] [I-D.ietf-netconf-distributed-notif]
Zhou, T., Zheng, G., Voit, E., Graf, T., and P. Francois, Zhou, T., Zheng, G., Voit, E., Graf, T., and P. Francois,
"Subscription to Distributed Notifications", Work in "Subscription to Distributed Notifications", Work in
Progress, Internet-Draft, draft-ietf-netconf-distributed- Progress, Internet-Draft, draft-ietf-netconf-distributed-
notif-02, 6 May 2021, <https://www.ietf.org/archive/id/ notif-02, 6 May 2021, <https://www.ietf.org/archive/id/
draft-ietf-netconf-distributed-notif-02.txt>. draft-ietf-netconf-distributed-notif-02.txt>.
[I-D.ietf-netconf-udp-notif] [I-D.ietf-netconf-udp-notif]
Zheng, G., Zhou, T., Graf, T., Francois, P., and P. Zheng, G., Zhou, T., Graf, T., Francois, P., Feng, A. H.,
Lucente, "UDP-based Transport for Configured and P. Lucente, "UDP-based Transport for Configured
Subscriptions", Work in Progress, Internet-Draft, draft- Subscriptions", Work in Progress, Internet-Draft, draft-
ietf-netconf-udp-notif-03, 12 July 2021, ietf-netconf-udp-notif-04, 21 October 2021,
<https://www.ietf.org/archive/id/draft-ietf-netconf-udp- <https://www.ietf.org/archive/id/draft-ietf-netconf-udp-
notif-03.txt>. notif-04.txt>.
[I-D.irtf-nmrg-ibn-concepts-definitions] [I-D.irtf-nmrg-ibn-concepts-definitions]
Clemm, A., Ciavaglia, L., Granville, L. Z., and J. Clemm, A., Ciavaglia, L., Granville, L. Z., and J.
Tantsura, "Intent-Based Networking - Concepts and Tantsura, "Intent-Based Networking - Concepts and
Definitions", Work in Progress, Internet-Draft, draft- Definitions", Work in Progress, Internet-Draft, draft-
irtf-nmrg-ibn-concepts-definitions-05, 2 September 2021, irtf-nmrg-ibn-concepts-definitions-05, 2 September 2021,
<https://www.ietf.org/archive/id/draft-irtf-nmrg-ibn- <https://www.ietf.org/archive/id/draft-irtf-nmrg-ibn-
concepts-definitions-05.txt>. concepts-definitions-05.txt>.
[I-D.kumar-rtgwg-grpc-protocol] [I-D.kumar-rtgwg-grpc-protocol]
skipping to change at page 31, line 15 skipping to change at page 32, line 24
[I-D.song-opsawg-dnp4iq] [I-D.song-opsawg-dnp4iq]
Song, H. and J. Gong, "Requirements for Interactive Query Song, H. and J. Gong, "Requirements for Interactive Query
with Dynamic Network Probes", Work in Progress, Internet- with Dynamic Network Probes", Work in Progress, Internet-
Draft, draft-song-opsawg-dnp4iq-01, 19 June 2017, Draft, draft-song-opsawg-dnp4iq-01, 19 June 2017,
<https://www.ietf.org/archive/id/draft-song-opsawg-dnp4iq- <https://www.ietf.org/archive/id/draft-song-opsawg-dnp4iq-
01.txt>. 01.txt>.
[I-D.song-opsawg-ifit-framework] [I-D.song-opsawg-ifit-framework]
Song, H., Qin, F., Chen, H., Jin, J., and J. Shin, "In- Song, H., Qin, F., Chen, H., Jin, J., and J. Shin, "In-
situ Flow Information Telemetry", Work in Progress, situ Flow Information Telemetry", Work in Progress,
Internet-Draft, draft-song-opsawg-ifit-framework-15, 28 Internet-Draft, draft-song-opsawg-ifit-framework-16, 21
September 2021, <https://www.ietf.org/archive/id/draft- October 2021, <https://www.ietf.org/archive/id/draft-song-
song-opsawg-ifit-framework-15.txt>. opsawg-ifit-framework-16.txt>.
[I-D.wwx-netmod-event-yang] [I-D.wwx-netmod-event-yang]
Wu, Q., Bryskin, I., Birkholz, H., Liu, X., and B. Claise, Wu, Q., Bryskin, I., Birkholz, H., Liu, X., and B. Claise,
"A YANG Data model for ECA Policy Management", Work in "A YANG Data model for ECA Policy Management", Work in
Progress, Internet-Draft, draft-wwx-netmod-event-yang-10, Progress, Internet-Draft, draft-wwx-netmod-event-yang-10,
1 November 2020, <https://www.ietf.org/archive/id/draft- 1 November 2020, <https://www.ietf.org/archive/id/draft-
wwx-netmod-event-yang-10.txt>. wwx-netmod-event-yang-10.txt>.
[RFC1157] Case, J., Fedor, M., Schoffstall, M., and J. Davin, [RFC1157] Case, J., Fedor, M., Schoffstall, M., and J. Davin,
"Simple Network Management Protocol (SNMP)", RFC 1157, "Simple Network Management Protocol (SNMP)", RFC 1157,
skipping to change at page 32, line 14 skipping to change at page 33, line 24
[RFC3877] Chisholm, S. and D. Romascanu, "Alarm Management [RFC3877] Chisholm, S. and D. Romascanu, "Alarm Management
Information Base (MIB)", RFC 3877, DOI 10.17487/RFC3877, Information Base (MIB)", RFC 3877, DOI 10.17487/RFC3877,
September 2004, <https://www.rfc-editor.org/info/rfc3877>. September 2004, <https://www.rfc-editor.org/info/rfc3877>.
[RFC4656] Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M. [RFC4656] Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M.
Zekauskas, "A One-way Active Measurement Protocol Zekauskas, "A One-way Active Measurement Protocol
(OWAMP)", RFC 4656, DOI 10.17487/RFC4656, September 2006, (OWAMP)", RFC 4656, DOI 10.17487/RFC4656, September 2006,
<https://www.rfc-editor.org/info/rfc4656>. <https://www.rfc-editor.org/info/rfc4656>.
[RFC5085] Nadeau, T., Ed. and C. Pignataro, Ed., "Pseudowire Virtual
Circuit Connectivity Verification (VCCV): A Control
Channel for Pseudowires", RFC 5085, DOI 10.17487/RFC5085,
December 2007, <https://www.rfc-editor.org/info/rfc5085>.
[RFC5357] Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J. [RFC5357] Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J.
Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)", Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)",
RFC 5357, DOI 10.17487/RFC5357, October 2008, RFC 5357, DOI 10.17487/RFC5357, October 2008,
<https://www.rfc-editor.org/info/rfc5357>. <https://www.rfc-editor.org/info/rfc5357>.
[RFC6020] Bjorklund, M., Ed., "YANG - A Data Modeling Language for [RFC6020] Bjorklund, M., Ed., "YANG - A Data Modeling Language for
the Network Configuration Protocol (NETCONF)", RFC 6020, the Network Configuration Protocol (NETCONF)", RFC 6020,
DOI 10.17487/RFC6020, October 2010, DOI 10.17487/RFC6020, October 2010,
<https://www.rfc-editor.org/info/rfc6020>. <https://www.rfc-editor.org/info/rfc6020>.
skipping to change at page 32, line 40 skipping to change at page 34, line 11
S., and E. Yedavalli, "Cisco Service-Level Assurance S., and E. Yedavalli, "Cisco Service-Level Assurance
Protocol", RFC 6812, DOI 10.17487/RFC6812, January 2013, Protocol", RFC 6812, DOI 10.17487/RFC6812, January 2013,
<https://www.rfc-editor.org/info/rfc6812>. <https://www.rfc-editor.org/info/rfc6812>.
[RFC7011] Claise, B., Ed., Trammell, B., Ed., and P. Aitken, [RFC7011] Claise, B., Ed., Trammell, B., Ed., and P. Aitken,
"Specification of the IP Flow Information Export (IPFIX) "Specification of the IP Flow Information Export (IPFIX)
Protocol for the Exchange of Flow Information", STD 77, Protocol for the Exchange of Flow Information", STD 77,
RFC 7011, DOI 10.17487/RFC7011, September 2013, RFC 7011, DOI 10.17487/RFC7011, September 2013,
<https://www.rfc-editor.org/info/rfc7011>. <https://www.rfc-editor.org/info/rfc7011>.
[RFC7258] Farrell, S. and H. Tschofenig, "Pervasive Monitoring Is an
Attack", BCP 188, RFC 7258, DOI 10.17487/RFC7258, May
2014, <https://www.rfc-editor.org/info/rfc7258>.
[RFC7276] Mizrahi, T., Sprecher, N., Bellagamba, E., and Y. [RFC7276] Mizrahi, T., Sprecher, N., Bellagamba, E., and Y.
Weingarten, "An Overview of Operations, Administration, Weingarten, "An Overview of Operations, Administration,
and Maintenance (OAM) Tools", RFC 7276, and Maintenance (OAM) Tools", RFC 7276,
DOI 10.17487/RFC7276, June 2014, DOI 10.17487/RFC7276, June 2014,
<https://www.rfc-editor.org/info/rfc7276>. <https://www.rfc-editor.org/info/rfc7276>.
[RFC7540] Belshe, M., Peon, R., and M. Thomson, Ed., "Hypertext [RFC7540] Belshe, M., Peon, R., and M. Thomson, Ed., "Hypertext
Transfer Protocol Version 2 (HTTP/2)", RFC 7540, Transfer Protocol Version 2 (HTTP/2)", RFC 7540,
DOI 10.17487/RFC7540, May 2015, DOI 10.17487/RFC7540, May 2015,
<https://www.rfc-editor.org/info/rfc7540>. <https://www.rfc-editor.org/info/rfc7540>.
skipping to change at page 33, line 24 skipping to change at page 34, line 45
[RFC7854] Scudder, J., Ed., Fernando, R., and S. Stuart, "BGP [RFC7854] Scudder, J., Ed., Fernando, R., and S. Stuart, "BGP
Monitoring Protocol (BMP)", RFC 7854, Monitoring Protocol (BMP)", RFC 7854,
DOI 10.17487/RFC7854, June 2016, DOI 10.17487/RFC7854, June 2016,
<https://www.rfc-editor.org/info/rfc7854>. <https://www.rfc-editor.org/info/rfc7854>.
[RFC7950] Bjorklund, M., Ed., "The YANG 1.1 Data Modeling Language", [RFC7950] Bjorklund, M., Ed., "The YANG 1.1 Data Modeling Language",
RFC 7950, DOI 10.17487/RFC7950, August 2016, RFC 7950, DOI 10.17487/RFC7950, August 2016,
<https://www.rfc-editor.org/info/rfc7950>. <https://www.rfc-editor.org/info/rfc7950>.
[RFC8040] Bierman, A., Bjorklund, M., and K. Watsen, "RESTCONF
Protocol", RFC 8040, DOI 10.17487/RFC8040, January 2017,
<https://www.rfc-editor.org/info/rfc8040>.
[RFC8084] Fairhurst, G., "Network Transport Circuit Breakers",
BCP 208, RFC 8084, DOI 10.17487/RFC8084, March 2017,
<https://www.rfc-editor.org/info/rfc8084>.
[RFC8085] Eggert, L., Fairhurst, G., and G. Shepherd, "UDP Usage
Guidelines", BCP 145, RFC 8085, DOI 10.17487/RFC8085,
March 2017, <https://www.rfc-editor.org/info/rfc8085>.
[RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data
Interchange Format", STD 90, RFC 8259,
DOI 10.17487/RFC8259, December 2017,
<https://www.rfc-editor.org/info/rfc8259>.
[RFC8321] Fioccola, G., Ed., Capello, A., Cociglio, M., Castaldelli, [RFC8321] Fioccola, G., Ed., Capello, A., Cociglio, M., Castaldelli,
L., Chen, M., Zheng, L., Mirsky, G., and T. Mizrahi, L., Chen, M., Zheng, L., Mirsky, G., and T. Mizrahi,
"Alternate-Marking Method for Passive and Hybrid "Alternate-Marking Method for Passive and Hybrid
Performance Monitoring", RFC 8321, DOI 10.17487/RFC8321, Performance Monitoring", RFC 8321, DOI 10.17487/RFC8321,
January 2018, <https://www.rfc-editor.org/info/rfc8321>. January 2018, <https://www.rfc-editor.org/info/rfc8321>.
[RFC8639] Voit, E., Clemm, A., Gonzalez Prieto, A., Nilsen-Nygaard, [RFC8639] Voit, E., Clemm, A., Gonzalez Prieto, A., Nilsen-Nygaard,
E., and A. Tripathy, "Subscription to YANG Notifications", E., and A. Tripathy, "Subscription to YANG Notifications",
RFC 8639, DOI 10.17487/RFC8639, September 2019, RFC 8639, DOI 10.17487/RFC8639, September 2019,
<https://www.rfc-editor.org/info/rfc8639>. <https://www.rfc-editor.org/info/rfc8639>.
[RFC8641] Clemm, A. and E. Voit, "Subscription to YANG Notifications [RFC8641] Clemm, A. and E. Voit, "Subscription to YANG Notifications
for Datastore Updates", RFC 8641, DOI 10.17487/RFC8641, for Datastore Updates", RFC 8641, DOI 10.17487/RFC8641,
September 2019, <https://www.rfc-editor.org/info/rfc8641>. September 2019, <https://www.rfc-editor.org/info/rfc8641>.
[RFC8762] Mirsky, G., Jun, G., Nydell, H., and R. Foote, "Simple
Two-Way Active Measurement Protocol", RFC 8762,
DOI 10.17487/RFC8762, March 2020,
<https://www.rfc-editor.org/info/rfc8762>.
[RFC8924] Aldrin, S., Pignataro, C., Ed., Kumar, N., Ed., Krishnan,
R., and A. Ghanwani, "Service Function Chaining (SFC)
Operations, Administration, and Maintenance (OAM)
Framework", RFC 8924, DOI 10.17487/RFC8924, October 2020,
<https://www.rfc-editor.org/info/rfc8924>.
[xml] "Extensible Markup Language (XML) 1.0 (Fifth Edition)",
<https://www.w3.org/TR/2008/REC-xml-20081126/>.
Appendix A. A Survey on Existing Network Telemetry Techniques Appendix A. A Survey on Existing Network Telemetry Techniques
In this non-normative appendix, we provide an overview of some In this non-normative appendix, we provide an overview of some
existing techniques and standard proposals for each network telemetry existing techniques and standard proposals for each network telemetry
module. module.
A.1. Management Plane Telemetry A.1. Management Plane Telemetry
A.1.1. Push Extensions for NETCONF A.1.1. Push Extensions for NETCONF
NETCONF [RFC6241] is a popular network management protocol NETCONF [RFC6241] is a popular network management protocol
skipping to change at page 37, line 5 skipping to change at page 39, line 5
generation. The data subscription needs to define the derived data generation. The data subscription needs to define the derived data
which can be composed and derived from the raw data sources. The which can be composed and derived from the raw data sources. The
data generation takes advantage of the moderate in-network computing data generation takes advantage of the moderate in-network computing
to produce the desired data. to produce the desired data.
While DNP can introduce unforeseeable flexibility to the data plane While DNP can introduce unforeseeable flexibility to the data plane
telemetry, it also faces some challenges. It requires a flexible telemetry, it also faces some challenges. It requires a flexible
data plane that can be dynamically reprogrammed at run-time. The data plane that can be dynamically reprogrammed at run-time. The
programming API is yet to be defined. programming API is yet to be defined.
A.3.3. IP Flow Information Export (IPFIX) protocol A.3.3. IP Flow Information Export (IPFIX) Protocol
Traffic on a network can be seen as a set of flows passing through Traffic on a network can be seen as a set of flows passing through
network elements. IP Flow Information Export (IPFIX) [RFC7011] network elements. IP Flow Information Export (IPFIX) [RFC7011]
provides a means of transmitting traffic flow information for provides a means of transmitting traffic flow information for
administrative or other purposes. A typical IPFIX enabled system administrative or other purposes. A typical IPFIX enabled system
includes a pool of Metering Processes that collects data packets at includes a pool of Metering Processes that collects data packets at
one or more Observation Points, optionally filters them and one or more Observation Points, optionally filters them and
aggregates information about these packets. An Exporter then gathers aggregates information about these packets. An Exporter then gathers
each of the Observation Points together into an Observation Domain each of the Observation Points together into an Observation Domain
and sends this information via the IPFIX protocol to a Collector. and sends this information via the IPFIX protocol to a Collector.
skipping to change at page 37, line 44 skipping to change at page 39, line 44
A.3.5. Postcard Based Telemetry A.3.5. Postcard Based Telemetry
PBT [I-D.song-ippm-postcard-based-telemetry] is a proposed PBT [I-D.song-ippm-postcard-based-telemetry] is a proposed
complementary technique to IOAM. PBT directly exports data at each complementary technique to IOAM. PBT directly exports data at each
node through an independent packet. At the cost of higher bandwidth node through an independent packet. At the cost of higher bandwidth
overhead and the need for data correlation, PBT shows several overhead and the need for data correlation, PBT shows several
advantages over IOAM. It can also help to identify packet drop advantages over IOAM. It can also help to identify packet drop
location in case a packet is dropped on its forwarding path. location in case a packet is dropped on its forwarding path.
A.3.6. Existing OAM for Specific Data Planes
Various data planes raises unique OAM requirements. IETF has
published OAM technique and framework documents (e.g., [RFC8924] and
[RFC5085]) targeting different data planes such as MPLS, L2-VPN,
NVO3, VXLAN, BIER, SFC, and DETNET. The aforementioned data plane
telemetry techniques can be used to enhance the OAM capability on
such data planes.
A.4. External Data and Event Telemetry A.4. External Data and Event Telemetry
A.4.1. Sources of External Events A.4.1. Sources of External Events
To ensure that the information provided by external event detectors To ensure that the information provided by external event detectors
and used by the network management solutions is meaningful for and used by the network management solutions is meaningful for
management purposes, the network telemetry framework must ensure that management purposes, the network telemetry framework must ensure that
such detectors (sources) are easily connected to the management such detectors (sources) are easily connected to the management
solutions (sinks). This requires the specification of a list of solutions (sinks). This requires the specification of a list of
potential external data sources that could be of interest in network potential external data sources that could be of interest in network
management and match it to the connectors and/or interfaces required management and match it to the connectors and/or interfaces required
to connect them. to connect them.
skipping to change at page 39, line 39 skipping to change at page 41, line 41
those situations there will be a special connector that provides the those situations there will be a special connector that provides the
typical interfaces found in most other elements connected to the typical interfaces found in most other elements connected to the
management plane. For instance, the interfaces could accomplish this management plane. For instance, the interfaces could accomplish this
with a specific data model (YANG) and specific telemetry protocol, with a specific data model (YANG) and specific telemetry protocol,
such as NETCONF, YANG-Push, or gRPC. such as NETCONF, YANG-Push, or gRPC.
Authors' Addresses Authors' Addresses
Haoyu Song Haoyu Song
Futurewei Futurewei
2330 Central Expressway
Santa Clara,
United States of America United States of America
Email: haoyu.song@futurewei.com Email: haoyu.song@futurewei.com
Fengwei Qin Fengwei Qin
China Mobile China Mobile
No. 32 Xuanwumenxi Ave., Xicheng District
Beijing, 100032
P.R. China P.R. China
Email: qinfengwei@chinamobile.com
Email: qinfengwei@chinamobile.com
Pedro Martinez-Julia Pedro Martinez-Julia
NICT NICT
4-2-1, Nukui-Kitamachi, Tokyo
184-8795
Japan Japan
Email: pedro@nict.go.jp Email: pedro@nict.go.jp
Laurent Ciavaglia Laurent Ciavaglia
Nokia Rakuten Mobile
91460 Villarceaux
France France
Email: laurent.ciavaglia@nokia.com Email: laurent.ciavaglia@rakuten.com
Aijun Wang Aijun Wang
China Telecom China Telecom
Beiqijia Town, Changping District
Beijing, 102209
P.R. China P.R. China
Email: wangaj.bri@chinatelecom.cn Email: wangaj.bri@chinatelecom.cn
 End of changes. 44 change blocks. 
104 lines changed or deleted 184 lines changed or added

This html diff was produced by rfcdiff 1.48. The latest version is available from http://tools.ietf.org/tools/rfcdiff/