draft-ietf-opsawg-ntf-09.txt		draft-ietf-opsawg-ntf-10.txt
	OPSAWG H. Song		OPSAWG H. Song
	Internet-Draft Futurewei		Internet-Draft Futurewei
	Intended status: Informational F. Qin		Intended status: Informational F. Qin
	Expires: 16 April 2022 China Mobile		Expires: 12 May 2022 China Mobile
	P. Martinez-Julia		P. Martinez-Julia
	NICT		NICT
	L. Ciavaglia		L. Ciavaglia
	Nokia		Rakuten Mobile
	A. Wang		A. Wang
	China Telecom		China Telecom
	13 October 2021		8 November 2021
	Network Telemetry Framework		Network Telemetry Framework
	draft-ietf-opsawg-ntf-09		draft-ietf-opsawg-ntf-10
	Abstract		Abstract
	Network telemetry is a technology for gaining network insight and		Network telemetry is a technology for gaining network insight and
	facilitating efficient and automated network management. It		facilitating efficient and automated network management. It
	encompasses various techniques for remote data generation,		encompasses various techniques for remote data generation,
	collection, correlation, and consumption. This document describes an		collection, correlation, and consumption. This document describes an
	architectural framework for network telemetry, motivated by		architectural framework for network telemetry, motivated by
	challenges that are encountered as part of the operation of networks		challenges that are encountered as part of the operation of networks
	and by the requirements that ensue. This document clarifies the		and by the requirements that ensue. This document clarifies the
	skipping to change at page 1, line 48 ¶		skipping to change at page 1, line 48 ¶
	Internet-Drafts are working documents of the Internet Engineering		Internet-Drafts are working documents of the Internet Engineering
	Task Force (IETF). Note that other groups may also distribute		Task Force (IETF). Note that other groups may also distribute
	working documents as Internet-Drafts. The list of current Internet-		working documents as Internet-Drafts. The list of current Internet-
	Drafts is at https://datatracker.ietf.org/drafts/current/.		Drafts is at https://datatracker.ietf.org/drafts/current/.
	Internet-Drafts are draft documents valid for a maximum of six months		Internet-Drafts are draft documents valid for a maximum of six months
	and may be updated, replaced, or obsoleted by other documents at any		and may be updated, replaced, or obsoleted by other documents at any
	time. It is inappropriate to use Internet-Drafts as reference		time. It is inappropriate to use Internet-Drafts as reference
	material or to cite them other than as "work in progress."		material or to cite them other than as "work in progress."
	This Internet-Draft will expire on 16 April 2022.		This Internet-Draft will expire on 12 May 2022.
	Copyright Notice		Copyright Notice
	Copyright (c) 2021 IETF Trust and the persons identified as the		Copyright (c) 2021 IETF Trust and the persons identified as the
	document authors. All rights reserved.		document authors. All rights reserved.
	This document is subject to BCP 78 and the IETF Trust's Legal		This document is subject to BCP 78 and the IETF Trust's Legal
	Provisions Relating to IETF Documents (https://trustee.ietf.org/		Provisions Relating to IETF Documents (https://trustee.ietf.org/
	license-info) in effect on the date of publication of this document.		license-info) in effect on the date of publication of this document.
	Please review these documents carefully, as they describe your rights		Please review these documents carefully, as they describe your rights
	skipping to change at page 2, line 36 ¶		skipping to change at page 2, line 36 ¶
	3.3. Challenges . . . . . . . . . . . . . . . . . . . . . . . 9		3.3. Challenges . . . . . . . . . . . . . . . . . . . . . . . 9
	3.4. Network Telemetry . . . . . . . . . . . . . . . . . . . . 10		3.4. Network Telemetry . . . . . . . . . . . . . . . . . . . . 10
	3.5. The Necessity of a Network Telemetry Framework . . . . . 13		3.5. The Necessity of a Network Telemetry Framework . . . . . 13
	4. Network Telemetry Framework . . . . . . . . . . . . . . . . . 14		4. Network Telemetry Framework . . . . . . . . . . . . . . . . . 14
	4.1. Top Level Modules . . . . . . . . . . . . . . . . . . . . 14		4.1. Top Level Modules . . . . . . . . . . . . . . . . . . . . 14
	4.1.1. Management Plane Telemetry . . . . . . . . . . . . . 18		4.1.1. Management Plane Telemetry . . . . . . . . . . . . . 18
	4.1.2. Control Plane Telemetry . . . . . . . . . . . . . . . 18		4.1.2. Control Plane Telemetry . . . . . . . . . . . . . . . 18
	4.1.3. Forwarding Plane Telemetry . . . . . . . . . . . . . 19		4.1.3. Forwarding Plane Telemetry . . . . . . . . . . . . . 19
	4.1.4. External Data Telemetry . . . . . . . . . . . . . . . 21		4.1.4. External Data Telemetry . . . . . . . . . . . . . . . 21
	4.2. Second Level Function Components . . . . . . . . . . . . 22		4.2. Second Level Function Components . . . . . . . . . . . . 22
	4.3. Data Acquisition Mechanism and Type Abstraction . . . . . 23		4.3. Data Acquisition Mechanism and Type Abstraction . . . . . 24
	4.4. Mapping Existing Mechanisms into the Framework . . . . . 25		4.4. Mapping Existing Mechanisms into the Framework . . . . . 26
	5. Evolution of Network Telemetry Applications . . . . . . . . . 26		5. Evolution of Network Telemetry Applications . . . . . . . . . 27
	6. Security Considerations . . . . . . . . . . . . . . . . . . . 26		6. Security Considerations . . . . . . . . . . . . . . . . . . . 27
	7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 28		7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 29
	8. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 28		8. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 29
	9. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 28		9. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 30
	10. Informative References . . . . . . . . . . . . . . . . . . . 29		10. Informative References . . . . . . . . . . . . . . . . . . . 30
	Appendix A. A Survey on Existing Network Telemetry Techniques . 33		Appendix A. A Survey on Existing Network Telemetry Techniques . 35
	A.1. Management Plane Telemetry . . . . . . . . . . . . . . . 33		A.1. Management Plane Telemetry . . . . . . . . . . . . . . . 35
	A.1.1. Push Extensions for NETCONF . . . . . . . . . . . . . 34		A.1.1. Push Extensions for NETCONF . . . . . . . . . . . . . 36
	A.1.2. gRPC Network Management Interface . . . . . . . . . . 34		A.1.2. gRPC Network Management Interface . . . . . . . . . . 36
	A.2. Control Plane Telemetry . . . . . . . . . . . . . . . . . 34		A.2. Control Plane Telemetry . . . . . . . . . . . . . . . . . 36
	A.2.1. BGP Monitoring Protocol . . . . . . . . . . . . . . . 34		A.2.1. BGP Monitoring Protocol . . . . . . . . . . . . . . . 36
	A.3. Data Plane Telemetry . . . . . . . . . . . . . . . . . . 35		A.3. Data Plane Telemetry . . . . . . . . . . . . . . . . . . 37
	A.3.1. The Alternate Marking (AM) technology . . . . . . . . 35		A.3.1. The Alternate Marking (AM) technology . . . . . . . . 37
	A.3.2. Dynamic Network Probe . . . . . . . . . . . . . . . . 36		A.3.2. Dynamic Network Probe . . . . . . . . . . . . . . . . 38
	A.3.3. IP Flow Information Export (IPFIX) protocol . . . . . 37		A.3.3. IP Flow Information Export (IPFIX) Protocol . . . . . 39
	A.3.4. In-Situ OAM . . . . . . . . . . . . . . . . . . . . . 37		A.3.4. In-Situ OAM . . . . . . . . . . . . . . . . . . . . . 39
	A.3.5. Postcard Based Telemetry . . . . . . . . . . . . . . 37		A.3.5. Postcard Based Telemetry . . . . . . . . . . . . . . 39
	A.4. External Data and Event Telemetry . . . . . . . . . . . . 37		A.3.6. Existing OAM for Specific Data Planes . . . . . . . . 39
	A.4.1. Sources of External Events . . . . . . . . . . . . . 38		A.4. External Data and Event Telemetry . . . . . . . . . . . . 40
	A.4.2. Connectors and Interfaces . . . . . . . . . . . . . . 39		A.4.1. Sources of External Events . . . . . . . . . . . . . 40
	Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 39		A.4.2. Connectors and Interfaces . . . . . . . . . . . . . . 41
			Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 41
	1. Introduction		1. Introduction
	Network visibility is the ability of management tools to see the		Network visibility is the ability of management tools to see the
	state and behavior of a network, which is essential for successful		state and behavior of a network, which is essential for successful
	network operation. Network Telemetry revolves around network data		network operation. Network Telemetry revolves around network data
	that can help provide insights about the current state of the		that can help provide insights about the current state of the
	network, including network devices, forwarding, control, and		network, including network devices, forwarding, control, and
	management planes, and that can be generated and obtained through a		management planes, and that can be generated and obtained through a
	variety of techniques, including but not limited to network		variety of techniques, including but not limited to network
	skipping to change at page 5, line 10 ¶		skipping to change at page 5, line 10 ¶
	structured data.		structured data.
	gRPC: gRPC Remote Procedure Call, an open source high performance		gRPC: gRPC Remote Procedure Call, an open source high performance
	RPC framework that gNMI is based on. See [grpc] for details.		RPC framework that gNMI is based on. See [grpc] for details.
	IPFIX: IP Flow Information Export Protocol, specified in [RFC7011].		IPFIX: IP Flow Information Export Protocol, specified in [RFC7011].
	IOAM: In-situ OAM, a dataplane on-path telemetry technique.		IOAM: In-situ OAM, a dataplane on-path telemetry technique.
	JSON: An open standard file format and data interchange format that		JSON: An open standard file format and data interchange format that
	uses human-readable text to store and transmit data objects.		uses human-readable text to store and transmit data objects,
			specified in [RFC8259].
	MIB: Management Information Base, a database used for managing the		MIB: Management Information Base, a database used for managing the
	entities in a network.		entities in a network.
	NETCONF: Network Configuration Protocol, specified in [RFC6241].		NETCONF: Network Configuration Protocol, specified in [RFC6241].
	NetFlow: A Cisco protocol for flow record collecting, described in		NetFlow: A Cisco protocol for flow record collecting, described in
	[RFC3594].		[RFC3594].
	Network Telemetry: The process and instrumentation for acquiring and		Network Telemetry: The process and instrumentation for acquiring and
	skipping to change at page 5, line 40 ¶		skipping to change at page 5, line 41 ¶
	OAM: Operations, Administration, and Maintenance. A group of		OAM: Operations, Administration, and Maintenance. A group of
	network management functions that provide network fault		network management functions that provide network fault
	indication, fault localization, performance information, and data		indication, fault localization, performance information, and data
	and diagnosis functions. Most conventional network monitoring		and diagnosis functions. Most conventional network monitoring
	techniques and protocols belong to network OAM.		techniques and protocols belong to network OAM.
	PBT: Postcard-Based Telemetry, a dataplane on-path telemetry		PBT: Postcard-Based Telemetry, a dataplane on-path telemetry
	technique.		technique.
			RESTCONF: An HTTP-based protocol that provides a programmatic
			interface for accessing data defined in YANG, using the datastore
			concepts defined in NETCONF, as specified in [RFC8040].
	SMIv2 Structure of Management Information Version 2, defining MIB		SMIv2 Structure of Management Information Version 2, defining MIB
	objects, specified in [RFC2578].		objects, specified in [RFC2578].
	SNMP: Simple Network Management Protocol. Version 1 and 2 are		SNMP: Simple Network Management Protocol. Version 1 and 2 are
	specified in [RFC1157] and [RFC3416], respectively.		specified in [RFC1157] and [RFC3416], respectively.
			XML; Extensible Markup Language is a markup language for data
			encoding that is both human-readable and machine-readable,
			specified by W3C [xml].
	YANG: YANG is a data modeling language for the definition of data		YANG: YANG is a data modeling language for the definition of data
	sent over network management protocols such as the NETCONF and		sent over network management protocols such as the NETCONF and
	RESTCONF. YANG is defined in [RFC6020] and [RFC7950].		RESTCONF. YANG is defined in [RFC6020] and [RFC7950].
	YANG ECA A YANG model for Event-Condition-Action policies, defined		YANG ECA A YANG model for Event-Condition-Action policies, defined
	in [I-D.wwx-netmod-event-yang].		in [I-D.wwx-netmod-event-yang].
	YANG-Push: A mechanism that allows subscriber applications to		YANG-Push: A mechanism that allows subscriber applications to
	request a stream of updates from a YANG datastore on a network		request a stream of updates from a YANG datastore on a network
	device. Details are specified in [RFC8641] and [RFC8639].		device. Details are specified in [RFC8641] and [RFC8639].
	3. Background		3. Background
	The term "big data" is used to describe the extremely large volume of		The term "big data" is used to describe the extremely large volume of
	data sets that can be analyzed computationally to reveal patterns,		data sets that can be analyzed computationally to reveal patterns,
	trends, and associations. Networks are undoubtedly a source of big		trends, and associations. Networks are undoubtedly a source of big
	data because of their scale and the volume of network traffic they		data because of their scale and the volume of network traffic they
	forward. It is easy to see that network operations can benefit from		forward. When a network's endpoints do not represent individual
	network big data to gather insights into flows without breaching		users (e.g. in industrial, datacenter, and infrastructure contexts),
	privacy.		network operations can often benefit from large-scale data collection
			without breaching user privacy.
	Today one can access advanced big data analytics capability through a		Today one can access advanced big data analytics capability through a
	plethora of commercial and open source platforms (e.g., Apache		plethora of commercial and open source platforms (e.g., Apache
	Hadoop), tools (e.g., Apache Spark), and techniques (e.g., machine		Hadoop), tools (e.g., Apache Spark), and techniques (e.g., machine
	learning). Thanks to the advance of computing and storage		learning). Thanks to the advance of computing and storage
	technologies, network big data analytics gives network operators an		technologies, network big data analytics gives network operators an
	opportunity to gain network insights and move towards network		opportunity to gain network insights and move towards network
	autonomy. Some operators start to explore the application of		autonomy. Some operators start to explore the application of
	Artificial Intelligence (AI) to make sense of network data. Software		Artificial Intelligence (AI) to make sense of network data. Software
	tools can use the network data to detect and react on network faults,		tools can use the network data to detect and react on network faults,
	skipping to change at page 12, line 35 ¶		skipping to change at page 12, line 35 ¶
	real-time processing is required.		real-time processing is required.
	* In-band Data Collection: In addition to the passive and active		* In-band Data Collection: In addition to the passive and active
	data collection approaches, the new hybrid approach allows to		data collection approaches, the new hybrid approach allows to
	directly collect data for any target flow on its entire forwarding		directly collect data for any target flow on its entire forwarding
	path [I-D.song-opsawg-ifit-framework].		path [I-D.song-opsawg-ifit-framework].
	It is worth noting that a network telemetry system should not be		It is worth noting that a network telemetry system should not be
	intrusive to normal network operations by avoiding the pitfall of the		intrusive to normal network operations by avoiding the pitfall of the
	"observer effect". That is, it should not change the network		"observer effect". That is, it should not change the network
	behavior and affect the forwarding performance. Otherwise, the whole		behavior and affect the forwarding performance. Moreover, high-
	purpose of network telemetry is compromised.		volume telemetry traffic may cause network congestion unless proper
			isolation or traffic engineering techniques are in place, or
			congestion control mechanisms ensure that telemetry traffic backs off
			if it exceeds the network capacity. [RFC8084] and [RFC8085] are
			relevant Best Current Practices (BCP) in this space.
	Although in many cases a system for network telemetry involves a		Although in many cases a system for network telemetry involves a
	remote data collecting and consuming entity, it is important to		remote data collecting and consuming entity, it is important to
	understand that there are no inherent assumptions about how a system		understand that there are no inherent assumptions about how a system
	should be architected. Telemetry data producers and consumers can		should be architected. While a network architecture with centralized
	work in distributed or peer-to-peer fashions rather than assuming a		controller (e.g., SDN) seems a natural fit for network telemetry,
	centralized data consuming entity. In such cases, a network node can		network telemetry can work in distributed fashions as well. For
	be the direct consumer of telemetry data from other nodes.		example, telemetry data producers and consumers can have a peer-to-
			peer relationship, in which a network node can be the direct consumer
			of telemetry data from other nodes.
	3.5. The Necessity of a Network Telemetry Framework		3.5. The Necessity of a Network Telemetry Framework
	Network data analytics and machine-learning technologies are applied		Network data analytics and machine-learning technologies are applied
	for network operation automation, relying on abundant and coherent		for network operation automation, relying on abundant and coherent
	data from networks. Data acquisition that is limited to a single		data from networks. Data acquisition that is limited to a single
	source and static in nature will in many cases not be sufficient to		source and static in nature will in many cases not be sufficient to
	meet an application's telemetry data needs. As a result, multiple		meet an application's telemetry data needs. As a result, multiple
	data sources, involving a variety of techniques and standards, will		data sources, involving a variety of techniques and standards, will
	need to be integrated. It is desirable to have a framework that		need to be integrated. It is desirable to have a framework that
	skipping to change at page 16, line 22 ¶		skipping to change at page 16, line 22 ¶
	capabilities, different choices of data model, encoding, and		capabilities, different choices of data model, encoding, and
	transport method are made to balance the performance and cost. For		transport method are made to balance the performance and cost. For
	example, the forwarding chip has high throughput but limited capacity		example, the forwarding chip has high throughput but limited capacity
	for processing complex data and maintaining states, while the main		for processing complex data and maintaining states, while the main
	control CPU is capable of complex data and state processing, but has		control CPU is capable of complex data and state processing, but has
	limited bandwidth for high throughput data. As a result, the		limited bandwidth for high throughput data. As a result, the
	suitable telemetry protocol for each module can be different. Some		suitable telemetry protocol for each module can be different. Some
	representative techniques are shown in the corresponding table blocks		representative techniques are shown in the corresponding table blocks
	to highlight the technical diversity of these modules. Note that the		to highlight the technical diversity of these modules. Note that the
	selected techniques just reflect the de facto state of the art and		selected techniques just reflect the de facto state of the art and
	are not exhaustive. The key point is that one cannot expect to use a		are by no means exhaustive (e.g., IPFIX can also be implemented over
	universal protocol to cover all the network telemetry requirements.		TCP and SCTP but that is not recommended for forwarding plane). The
			key point is that one cannot expect to use a universal protocol to
			cover all the network telemetry requirements.
	+---------+--------------+--------------+---------------+-----------+		+-----------+-------------+-------------+--------------+----------+
	| Module | Management | Control | Forwarding | External |		| Module |Management |Control |Forwarding |External |
	| | Plane | Plane | Plane | Data |		| |Plane |Plane |Plane |Data |
	+---------+--------------+--------------+---------------+-----------+		+-----------+-------------+-------------+--------------+----------+
	|Object | config. & | control | flow & packet | terminal, |		|Object |config. & |control |flow & packet |terminal, |
	| | operation | protocol & | QoS, traffic | social & |		| |operation |protocol & |QoS, traffic |social & |
	| | state | signaling, | stat., buffer | environ- |		| |state |signaling, |stat., buffer |environ- |
	| | | RIB | & queue stat.,| mental |		| | |RIB |& queue stat.,|mental |
	| | | | ACL, FIB | |		| | | |ACL, FIB | |
	+---------+--------------+--------------+---------------+-----------+		+-----------+-------------+-------------+--------------+----------+
	|Export | main control | main control | fwding chip | various |		|Export |main control |main control |fwding chip |various |
	|Location | CPU | CPU, | or linecard | |		|Location |CPU |CPU, |or linecard | |
	| | | linecard CPU | CPU; main | |		| | |linecard CPU |CPU; main | |
	| | | or forwarding| control CPU | |		| | |or forwarding|control CPU | |
	| | | chip | unlikely | |		| | |chip |unlikely | |
	+---------+--------------+--------------+---------------+-----------+		+-----------+-------------+-------------+--------------+----------+
	|Data | YANG, MIB, | YANG, | template, | YANG, |		|Data |YANG, MIB, |YANG, |template, |YANG, |
	|Model | syslog | custom | YANG, | custom |		|Model |syslog |custom |YANG, |custom |
	| | | | custom | |		| | | |custom | |
	+---------+--------------+--------------+---------------+-----------+		+-----------+-------------+-------------+--------------+----------+
	|Data | GPB, JSON, | GPB, JSON, | plain | GPB, JSON |		|Data |GPB, JSON, |GPB, JSON, |plain |GPB, JSON |
	|Encoding | XML | XML, plain | | XML, plain|		|Encoding |XML |XML, plain | |XML, plain|
	+---------+--------------+--------------+---------------+-----------+		+-----------+-------------+-------------+--------------+----------+
	|Protocol | gRPC,NETCONF,| gRPC,NETCONF,| IPFIX, mirror,| gRPC |		|Application|gRPC,NETCONF,|gRPC,NETCONF,|IPFIX, mirror,|gRPC |
	| | | IPFIX, mirror| gRPC, NETFLOW | |		|Protocol |RESTCONF |IPFIX, mirror|gRPC, NETFLOW | |
	+---------+--------------+--------------+---------------+-----------+		+-----------+-------------+-------------+--------------+----------+
	|Transport| HTTP, TCP | HTTP, TCP, | UDP | HTTP,TCP |		|Data |HTTP, TCP |HTTP, TCP, |UDP |HTTP,TCP |
	| | | UDP | | UDP |		|Transport | |UDP | |UDP |
	+---------+--------------+--------------+---------------+-----------+		+-----------+-------------+-------------+--------------+----------+
	Figure 2: Comparison of the Data Object Modules		Figure 2: Comparison of the Data Object Modules
	Note that the interaction with the applications that consume network		Note that the interaction with the applications that consume network
	telemetry data can be indirect. Some in-device data transfer is		telemetry data can be indirect. Some in-device data transfer is
	possible. For example, in the management plane telemetry, the		possible. For example, in the management plane telemetry, the
	management plane will need to acquire data from the data plane. Some		management plane will need to acquire data from the data plane. Some
	operational states can only be derived from data plane data sources		operational states can only be derived from data plane data sources
	such as the interface status and statistics. As another example,		such as the interface status and statistics. As another example,
	obtaining control plane telemetry data may require the ability to		obtaining control plane telemetry data may require the ability to
	skipping to change at page 18, line 38 ¶		skipping to change at page 18, line 38 ¶
	and normalize data encoding and transformation.		and normalize data encoding and transformation.
	* High Speed Data Transport: In order to keep up with the velocity		* High Speed Data Transport: In order to keep up with the velocity
	of information, a server needs to be able to send large amounts of		of information, a server needs to be able to send large amounts of
	data at high frequency. Compact encoding formats or data		data at high frequency. Compact encoding formats or data
	compression schemes are needed to reduce the quantity of data and		compression schemes are needed to reduce the quantity of data and
	improve the data transport efficiency. The subscription mode, by		improve the data transport efficiency. The subscription mode, by
	replacing the query mode, reduces the interactions between clients		replacing the query mode, reduces the interactions between clients
	and servers and helps to improve the server's efficiency.		and servers and helps to improve the server's efficiency.
			* Network Congestion Avoidance: The application must protect the
			network from congestion by congestion control mechanisms or at
			least circuit breakers. [RFC8084] and [RFC8085] provide some
			solutions in this space.
	4.1.2. Control Plane Telemetry		4.1.2. Control Plane Telemetry
	The control plane telemetry refers to the health condition monitoring		The control plane telemetry refers to the health condition monitoring
	of different network control protocols at all layers of the protocol		of different network control protocols at all layers of the protocol
	stack. Keeping track of the operational status of these protocols is		stack. Keeping track of the operational status of these protocols is
	beneficial for detecting, localizing, and even predicting various		beneficial for detecting, localizing, and even predicting various
	network issues, as well as network optimization, in real-time and		network issues, as well as network optimization, in real-time and
	with fine granularity. Some particular challenges and issues faced		with fine granularity. Some particular challenges and issues faced
	by the control plane telemetry are as follows:		by the control plane telemetry are as follows:
	skipping to change at page 19, line 26 ¶		skipping to change at page 19, line 32 ¶
	* An example of the control plane telemetry is the BGP monitoring		* An example of the control plane telemetry is the BGP monitoring
	protocol (BMP), it is currently used for monitoring the BGP routes		protocol (BMP), it is currently used for monitoring the BGP routes
	and enables rich applications, such as BGP peer analysis, AS		and enables rich applications, such as BGP peer analysis, AS
	analysis, prefix analysis, and security analysis. However, the		analysis, prefix analysis, and security analysis. However, the
	monitoring of other layers, protocols and the cross-layer, cross-		monitoring of other layers, protocols and the cross-layer, cross-
	protocol KPI correlations are still in their infancy (e.g., IGP		protocol KPI correlations are still in their infancy (e.g., IGP
	monitoring is not as extensive as BMP), which require further		monitoring is not as extensive as BMP), which require further
	research.		research.
			* The requirement and solutions for network congestion avoidance are
			also applicable to the control plane telemetry.
	4.1.3. Forwarding Plane Telemetry		4.1.3. Forwarding Plane Telemetry
	An effective forwarding plane telemetry system relies on the data		An effective forwarding plane telemetry system relies on the data
	that the network device can expose. The quality, quantity, and		that the network device can expose. The quality, quantity, and
	timeliness of data must meet some stringent requirements. This		timeliness of data must meet some stringent requirements. This
	raises some challenges to the network data plane devices where the		raises some challenges to the network data plane devices where the
	first-hand data originates.		first-hand data originates.
	* A data plane device's main function is user traffic processing and		* A data plane device's main function is user traffic processing and
	forwarding. While supporting network visibility is important, the		forwarding. While supporting network visibility is important, the
	skipping to change at page 20, line 15 ¶		skipping to change at page 20, line 26 ¶
	* The data should be structured and labeled, and easy for		* The data should be structured and labeled, and easy for
	applications to parse and consume. At the same time, the data		applications to parse and consume. At the same time, the data
	types needed by applications can vary significantly. The data		types needed by applications can vary significantly. The data
	plane devices need to provide enough flexibility and		plane devices need to provide enough flexibility and
	programmability to support the precise data provision for		programmability to support the precise data provision for
	applications.		applications.
	* The data plane telemetry should support incremental deployment and		* The data plane telemetry should support incremental deployment and
	work even though some devices are unaware of the system.		work even though some devices are unaware of the system.
			* The requirement and solutions for network congestion avoidance are
			also applicable to the forwarding plane telemetry.
	Although not specific to the forwarding plane, these challenges are		Although not specific to the forwarding plane, these challenges are
	more difficult to the forwarding plane because of the limited		more difficult to the forwarding plane because of the limited
	resource and flexibility. Data plane programmability is essential to		resource and flexibility. Data plane programmability is essential to
	support network telemetry. Newer data plane forwarding chips are		support network telemetry. Newer data plane forwarding chips are
	equipped with advanced telemetry features and provide flexibility to		equipped with advanced telemetry features and provide flexibility to
	support customized telemetry functions.		support customized telemetry functions.
	Technique Taxonomy: concerning about how one instruments the		Technique Taxonomy: concerning about how one instruments the
	telemetry, there can be multiple possible dimensions to classify the		telemetry, there can be multiple possible dimensions to classify the
	forwarding plane telemetry techniques.		forwarding plane telemetry techniques.
	* Active, Passive, and Hybrid: This dimension concerns about the		* Active, Passive, and Hybrid: This dimension concerns about the
	end-to-end measurement. Active and passive methods (as well as		end-to-end measurement. Active and passive methods (as well as
	the hybrid types) are well documented in [RFC7799]. Passive		the hybrid types) are well documented in [RFC7799]. Passive
	methods include TCPDUMP, IPFIX [RFC7011], sflow, and traffic		methods include TCPDUMP, IPFIX [RFC7011], sflow, and traffic
	mirroring. These methods usually have low data coverage. The		mirroring. These methods usually have low data coverage. The
	bandwidth cost is very high in order to improve the data coverage.		bandwidth cost is very high in order to improve the data coverage.
	On the other hand, active methods include Ping, OWAMP [RFC4656],		On the other hand, active methods include Ping, OWAMP [RFC4656],
	TWAMP [RFC5357], and Cisco's SLA Protocol [RFC6812]. These		TWAMP [RFC5357], STAMP [RFC8762], and Cisco's SLA Protocol
	methods are intrusive and only provide indirect network		[RFC6812]. These methods are intrusive and only provide indirect
	measurements. Hybrid methods, including in-situ OAM		network measurements. Hybrid methods, including in-situ OAM
	[I-D.ietf-ippm-ioam-data], Alternate-Marking (AM) [RFC8321], and		[I-D.ietf-ippm-ioam-data], Alternate-Marking (AM) [RFC8321], and
	Multipoint Alternate Marking [I-D.ietf-ippm-multipoint-alt-mark],		Multipoint Alternate Marking [I-D.ietf-ippm-multipoint-alt-mark],
	provide a well-balanced and more flexible approach. However,		provide a well-balanced and more flexible approach. However,
	these methods are also more complex to implement.		these methods are also more complex to implement.
	* In-Band and Out-of-Band: Telemetry data carried in user packets		* In-Band and Out-of-Band: Telemetry data carried in user packets
	before being exported to a data collector is considered in-band		before being exported to a data collector is considered in-band
	(e.g., in-situ OAM [I-D.ietf-ippm-ioam-data]). Telemetry data		(e.g., in-situ OAM [I-D.ietf-ippm-ioam-data]). Telemetry data
	that is directly exported to a data collector without modifying		that is directly exported to a data collector without modifying
	user packets is considered out-of-band (e.g., the postcard-based		user packets is considered out-of-band (e.g., the postcard-based
	skipping to change at page 22, line 5 ¶		skipping to change at page 22, line 29 ¶
	perform the notifications, their timeliness is assumed. However,		perform the notifications, their timeliness is assumed. However,
	once messages have been dispatched, they must be quickly collected		once messages have been dispatched, they must be quickly collected
	and inserted into the control plane with variable priority, which		and inserted into the control plane with variable priority, which
	is higher for important sources and events and lower for secondary		is higher for important sources and events and lower for secondary
	ones.		ones.
	* The schema used by external detectors must be easily adopted by		* The schema used by external detectors must be easily adopted by
	current and future devices and applications. Therefore, it must		current and future devices and applications. Therefore, it must
	be easily mapped to current data models, such as in terms of YANG.		be easily mapped to current data models, such as in terms of YANG.
			* As the communication with external entities outside the boundary
			of a provider network may be realized over the Internet, the risk
			of congestion is even more relevant in this context and proper
			counter-measures must be taken. Solutions such as network
			transport circuit breakers are needed as well.
	Organizing both internal and external telemetry information together		Organizing both internal and external telemetry information together
	will be key for the general exploitation of the management		will be key for the general exploitation of the management
	possibilities of current and future network systems, as reflected in		possibilities of current and future network systems, as reflected in
	the incorporation of cognitive capabilities to new hardware and		the incorporation of cognitive capabilities to new hardware and
	software (virtual) elements.		software (virtual) elements.
	4.2. Second Level Function Components		4.2. Second Level Function Components
	The telemetry module at each plane can be further partitioned into		The telemetry module at each plane can be further partitioned into
	five distinct conceptual components:		five distinct conceptual components:
	skipping to change at page 25, line 16 ¶		skipping to change at page 26, line 16 ¶
	| Event-triggered Data |<----+ Streaming Data |		| Event-triggered Data |<----+ Streaming Data |
	+-------+---+----------+ +-----+---+-------+		+-------+---+----------+ +-----+---+-------+
	| | | |		| | | |
	| | | |		| | | |
	| | +--------------+ | |		| | +--------------+ | |
	| +-->| Derived Data |<--+ |		| +-->| Derived Data |<--+ |
	| +------+------ + |		| +------+------ + |
	| | |		| | |
	| V |		| V |
	| +--------------+ |		| +--------------+ |
	+------>| Simple Data |<------+		+------>| Simple Data |<------+
	+--------------+		+--------------+
	Figure 4: Data Type Relationship		Figure 4: Data Type Relationship
	Subscription usually deals with event-triggered data and streaming		Subscription usually deals with event-triggered data and streaming
	data, and query usually deals with simple data and derived data. But		data, and query usually deals with simple data and derived data. But
	the other ways are also possible. Advanced network telemetry		the other ways are also possible. Advanced network telemetry
	techniques are designed mainly for event-triggered or streaming data		techniques are designed mainly for event-triggered or streaming data
	subscription, and derived data query.		subscription, and derived data query.
	skipping to change at page 25, line 44 ¶		skipping to change at page 26, line 44 ¶
	Also, some comprehensive protocols and techniques may cover multiple		Also, some comprehensive protocols and techniques may cover multiple
	aspects or modules of the framework, so a name in a block only		aspects or modules of the framework, so a name in a block only
	emphasizes one particular characteristic of it. More details about		emphasizes one particular characteristic of it. More details about
	some listed mechanisms can be found in Appendix A.		some listed mechanisms can be found in Appendix A.
	+-------------+-----------------+---------------+--------------+		+-------------+-----------------+---------------+--------------+
	| | Management | Control | Forwarding |		| | Management | Control | Forwarding |
	| | Plane | Plane | Plane |		| | Plane | Plane | Plane |
	+-------------+-----------------+---------------+--------------+		+-------------+-----------------+---------------+--------------+
	| data config.| gNMI, NETCONF, | gNMI, NETCONF,| NETCONF, |		| data config.| gNMI, NETCONF, | gNMI, NETCONF,| NETCONF, |
	| & subscribe | SNMP, YANG-Push | YANG-Push | YANG-Push |		| & subscribe | RESTCONF, SNMP, | RESTCONF, | RESTCONF, |
			| | YANG-Push | YANG-Push | YANG-Push |
	+-------------+-----------------+---------------+--------------+		+-------------+-----------------+---------------+--------------+
	| data gen. & | MIB, | YANG | IOAM, PSAMP |		| data gen. & | MIB, | YANG | IOAM, PSAMP |
	| process | YANG | | PBT, AM, |		| process | YANG | | PBT, AM, |
	+-------------+-----------------+---------------+--------------+		+-------------+-----------------+---------------+--------------+
	| data encode.| gRPC, HTTP, TCP | BMP, TCP | IPFIX, UDP |		| data encode.| gRPC, HTTP, TCP | BMP, TCP | IPFIX, UDP |
	| & export | | | |		| & export | | | |
	+-------------+-----------------+---------------+--------------+		+-------------+-----------------+---------------+--------------+
	Figure 5: Existing Work Mapping		Figure 5: Existing Work Mapping
	5. Evolution of Network Telemetry Applications		5. Evolution of Network Telemetry Applications
	skipping to change at page 28, line 19 ¶		skipping to change at page 29, line 19 ¶
	Access Control and Event-Condition-Action policies. Also, potential		Access Control and Event-Condition-Action policies. Also, potential
	conflicts between network telemetry mechanisms must be detected		conflicts between network telemetry mechanisms must be detected
	accurately and resolved quickly to avoid unnecessary network		accurately and resolved quickly to avoid unnecessary network
	telemetry traffic propagation escalating into an unintended or		telemetry traffic propagation escalating into an unintended or
	intended denial of service attack.		intended denial of service attack.
	Further study of the security issues will be required, and it is		Further study of the security issues will be required, and it is
	expected that the security mechanisms and protocols are developed and		expected that the security mechanisms and protocols are developed and
	deployed along with a network telemetry system.		deployed along with a network telemetry system.
	In addition to security, privacy is also an important issue. Network		In addition to security, privacy is also an important issue. Large-
	telemetry means to improve the network operation which can ultimately		scale network data collection is a major threat to user privacy
	benefit end user's quality of experience. The network operators must		[RFC7258]. The Network Telemetry Framework is not applicable to
	be held accountable and strive for a balance between managing the		networks whose endpoints represent individual users, such as general-
	network and maintaining the user privacy of that network.		purpose access networks. Any collection or retention of data in
			those networks must be tightly limited to protect user privacy.
	7. IANA Considerations		7. IANA Considerations
	This document includes no request to IANA.		This document includes no request to IANA.
	8. Contributors		8. Contributors
	The other contributors of this document are listed as follows.		The other contributors of this document are listed as follows.
	* Tianran Zhou		* Tianran Zhou
	skipping to change at page 28, line 49 ¶		skipping to change at page 30, line 9 ¶
	* Daniel King		* Daniel King
	* Adrian Farrel		* Adrian Farrel
	* Alexander Clemm		* Alexander Clemm
	9. Acknowledgments		9. Acknowledgments
	We would like to thank Rob Wilton, Greg Mirsky, Randy Presuhn, Joe		We would like to thank Rob Wilton, Greg Mirsky, Randy Presuhn, Joe
	Clarke, Victor Liu, James Guichard, Uri Blumenthal, Giuseppe		Clarke, Victor Liu, James Guichard, Uri Blumenthal, Giuseppe
	Fioccola, Yunan Gu, Parviz Yegani, Young Lee, Qin Wu, and many others		Fioccola, Yunan Gu, Parviz Yegani, Young Lee, Qin Wu, Gyan Mishra,
	who have provided helpful comments and suggestions to improve this		Ben Schwartz, Alexey Melnikov, Michael Scharf, and many others who
			have provided helpful comments and suggestions to improve this
	document.		document.
	10. Informative References		10. Informative References
	[gnmi] "gNMI - gRPC Network Management Interface",		[gnmi] "gNMI - gRPC Network Management Interface",
	<https://github.com/openconfig/reference/tree/master/rpc/		<https://github.com/openconfig/reference/tree/master/rpc/
	gnmi>.		gnmi>.
	[grpc] "gPPC, A high performance, open-source universal RPC		[grpc] "gPPC, A high performance, open-source universal RPC
	framework", <https://grpc.io>.		framework", <https://grpc.io>.
	skipping to change at page 29, line 33 ¶		skipping to change at page 30, line 42 ¶
	Evens, T., Bayraktar, S., Bhardwaj, M., and P. Lucente,		Evens, T., Bayraktar, S., Bhardwaj, M., and P. Lucente,
	"Support for Local RIB in BGP Monitoring Protocol (BMP)",		"Support for Local RIB in BGP Monitoring Protocol (BMP)",
	Work in Progress, Internet-Draft, draft-ietf-grow-bmp-		Work in Progress, Internet-Draft, draft-ietf-grow-bmp-
	local-rib-13, 31 August 2021,		local-rib-13, 31 August 2021,
	<https://www.ietf.org/archive/id/draft-ietf-grow-bmp-		<https://www.ietf.org/archive/id/draft-ietf-grow-bmp-
	local-rib-13.txt>.		local-rib-13.txt>.
	[I-D.ietf-ippm-ioam-data]		[I-D.ietf-ippm-ioam-data]
	Brockners, F., Bhandari, S., and T. Mizrahi, "Data Fields		Brockners, F., Bhandari, S., and T. Mizrahi, "Data Fields
	for In-situ OAM", Work in Progress, Internet-Draft, draft-		for In-situ OAM", Work in Progress, Internet-Draft, draft-
	ietf-ippm-ioam-data-15, 3 October 2021,		ietf-ippm-ioam-data-16, 8 November 2021,
	<https://www.ietf.org/archive/id/draft-ietf-ippm-ioam-		<https://www.ietf.org/archive/id/draft-ietf-ippm-ioam-
	data-15.txt>.		data-16.txt>.
	[I-D.ietf-ippm-multipoint-alt-mark]		[I-D.ietf-ippm-multipoint-alt-mark]
	Fioccola, G., Cociglio, M., Sapio, A., and R. Sisto,		Fioccola, G., Cociglio, M., Sapio, A., and R. Sisto,
	"Multipoint Alternate-Marking Method for Passive and		"Multipoint Alternate-Marking Method for Passive and
	Hybrid Performance Monitoring", Work in Progress,		Hybrid Performance Monitoring", Work in Progress,
	Internet-Draft, draft-ietf-ippm-multipoint-alt-mark-09, 23		Internet-Draft, draft-ietf-ippm-multipoint-alt-mark-09, 23
	March 2020, <https://www.ietf.org/archive/id/draft-ietf-		March 2020, <https://www.ietf.org/archive/id/draft-ietf-
	ippm-multipoint-alt-mark-09.txt>.		ippm-multipoint-alt-mark-09.txt>.
	[I-D.ietf-netconf-distributed-notif]		[I-D.ietf-netconf-distributed-notif]
	Zhou, T., Zheng, G., Voit, E., Graf, T., and P. Francois,		Zhou, T., Zheng, G., Voit, E., Graf, T., and P. Francois,
	"Subscription to Distributed Notifications", Work in		"Subscription to Distributed Notifications", Work in
	Progress, Internet-Draft, draft-ietf-netconf-distributed-		Progress, Internet-Draft, draft-ietf-netconf-distributed-
	notif-02, 6 May 2021, <https://www.ietf.org/archive/id/		notif-02, 6 May 2021, <https://www.ietf.org/archive/id/
	draft-ietf-netconf-distributed-notif-02.txt>.		draft-ietf-netconf-distributed-notif-02.txt>.
	[I-D.ietf-netconf-udp-notif]		[I-D.ietf-netconf-udp-notif]
	Zheng, G., Zhou, T., Graf, T., Francois, P., and P.		Zheng, G., Zhou, T., Graf, T., Francois, P., Feng, A. H.,
	Lucente, "UDP-based Transport for Configured		and P. Lucente, "UDP-based Transport for Configured
	Subscriptions", Work in Progress, Internet-Draft, draft-		Subscriptions", Work in Progress, Internet-Draft, draft-
	ietf-netconf-udp-notif-03, 12 July 2021,		ietf-netconf-udp-notif-04, 21 October 2021,
	<https://www.ietf.org/archive/id/draft-ietf-netconf-udp-		<https://www.ietf.org/archive/id/draft-ietf-netconf-udp-
	notif-03.txt>.		notif-04.txt>.
	[I-D.irtf-nmrg-ibn-concepts-definitions]		[I-D.irtf-nmrg-ibn-concepts-definitions]
	Clemm, A., Ciavaglia, L., Granville, L. Z., and J.		Clemm, A., Ciavaglia, L., Granville, L. Z., and J.
	Tantsura, "Intent-Based Networking - Concepts and		Tantsura, "Intent-Based Networking - Concepts and
	Definitions", Work in Progress, Internet-Draft, draft-		Definitions", Work in Progress, Internet-Draft, draft-
	irtf-nmrg-ibn-concepts-definitions-05, 2 September 2021,		irtf-nmrg-ibn-concepts-definitions-05, 2 September 2021,
	<https://www.ietf.org/archive/id/draft-irtf-nmrg-ibn-		<https://www.ietf.org/archive/id/draft-irtf-nmrg-ibn-
	concepts-definitions-05.txt>.		concepts-definitions-05.txt>.
	[I-D.kumar-rtgwg-grpc-protocol]		[I-D.kumar-rtgwg-grpc-protocol]
	skipping to change at page 31, line 15 ¶		skipping to change at page 32, line 24 ¶
	[I-D.song-opsawg-dnp4iq]		[I-D.song-opsawg-dnp4iq]
	Song, H. and J. Gong, "Requirements for Interactive Query		Song, H. and J. Gong, "Requirements for Interactive Query
	with Dynamic Network Probes", Work in Progress, Internet-		with Dynamic Network Probes", Work in Progress, Internet-
	Draft, draft-song-opsawg-dnp4iq-01, 19 June 2017,		Draft, draft-song-opsawg-dnp4iq-01, 19 June 2017,
	<https://www.ietf.org/archive/id/draft-song-opsawg-dnp4iq-		<https://www.ietf.org/archive/id/draft-song-opsawg-dnp4iq-
	01.txt>.		01.txt>.
	[I-D.song-opsawg-ifit-framework]		[I-D.song-opsawg-ifit-framework]
	Song, H., Qin, F., Chen, H., Jin, J., and J. Shin, "In-		Song, H., Qin, F., Chen, H., Jin, J., and J. Shin, "In-
	situ Flow Information Telemetry", Work in Progress,		situ Flow Information Telemetry", Work in Progress,
	Internet-Draft, draft-song-opsawg-ifit-framework-15, 28		Internet-Draft, draft-song-opsawg-ifit-framework-16, 21
	September 2021, <https://www.ietf.org/archive/id/draft-		October 2021, <https://www.ietf.org/archive/id/draft-song-
	song-opsawg-ifit-framework-15.txt>.		opsawg-ifit-framework-16.txt>.
	[I-D.wwx-netmod-event-yang]		[I-D.wwx-netmod-event-yang]
	Wu, Q., Bryskin, I., Birkholz, H., Liu, X., and B. Claise,		Wu, Q., Bryskin, I., Birkholz, H., Liu, X., and B. Claise,
	"A YANG Data model for ECA Policy Management", Work in		"A YANG Data model for ECA Policy Management", Work in
	Progress, Internet-Draft, draft-wwx-netmod-event-yang-10,		Progress, Internet-Draft, draft-wwx-netmod-event-yang-10,
	1 November 2020, <https://www.ietf.org/archive/id/draft-		1 November 2020, <https://www.ietf.org/archive/id/draft-
	wwx-netmod-event-yang-10.txt>.		wwx-netmod-event-yang-10.txt>.
	[RFC1157] Case, J., Fedor, M., Schoffstall, M., and J. Davin,		[RFC1157] Case, J., Fedor, M., Schoffstall, M., and J. Davin,
	"Simple Network Management Protocol (SNMP)", RFC 1157,		"Simple Network Management Protocol (SNMP)", RFC 1157,
	skipping to change at page 32, line 14 ¶		skipping to change at page 33, line 24 ¶
	[RFC3877] Chisholm, S. and D. Romascanu, "Alarm Management		[RFC3877] Chisholm, S. and D. Romascanu, "Alarm Management
	Information Base (MIB)", RFC 3877, DOI 10.17487/RFC3877,		Information Base (MIB)", RFC 3877, DOI 10.17487/RFC3877,
	September 2004, <https://www.rfc-editor.org/info/rfc3877>.		September 2004, <https://www.rfc-editor.org/info/rfc3877>.
	[RFC4656] Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M.		[RFC4656] Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M.
	Zekauskas, "A One-way Active Measurement Protocol		Zekauskas, "A One-way Active Measurement Protocol
	(OWAMP)", RFC 4656, DOI 10.17487/RFC4656, September 2006,		(OWAMP)", RFC 4656, DOI 10.17487/RFC4656, September 2006,
	<https://www.rfc-editor.org/info/rfc4656>.		<https://www.rfc-editor.org/info/rfc4656>.
			[RFC5085] Nadeau, T., Ed. and C. Pignataro, Ed., "Pseudowire Virtual
			Circuit Connectivity Verification (VCCV): A Control
			Channel for Pseudowires", RFC 5085, DOI 10.17487/RFC5085,
			December 2007, <https://www.rfc-editor.org/info/rfc5085>.
	[RFC5357] Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J.		[RFC5357] Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J.
	Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)",		Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)",
	RFC 5357, DOI 10.17487/RFC5357, October 2008,		RFC 5357, DOI 10.17487/RFC5357, October 2008,
	<https://www.rfc-editor.org/info/rfc5357>.		<https://www.rfc-editor.org/info/rfc5357>.
	[RFC6020] Bjorklund, M., Ed., "YANG - A Data Modeling Language for		[RFC6020] Bjorklund, M., Ed., "YANG - A Data Modeling Language for
	the Network Configuration Protocol (NETCONF)", RFC 6020,		the Network Configuration Protocol (NETCONF)", RFC 6020,
	DOI 10.17487/RFC6020, October 2010,		DOI 10.17487/RFC6020, October 2010,
	<https://www.rfc-editor.org/info/rfc6020>.		<https://www.rfc-editor.org/info/rfc6020>.
	skipping to change at page 32, line 40 ¶		skipping to change at page 34, line 11 ¶
	S., and E. Yedavalli, "Cisco Service-Level Assurance		S., and E. Yedavalli, "Cisco Service-Level Assurance
	Protocol", RFC 6812, DOI 10.17487/RFC6812, January 2013,		Protocol", RFC 6812, DOI 10.17487/RFC6812, January 2013,
	<https://www.rfc-editor.org/info/rfc6812>.		<https://www.rfc-editor.org/info/rfc6812>.
	[RFC7011] Claise, B., Ed., Trammell, B., Ed., and P. Aitken,		[RFC7011] Claise, B., Ed., Trammell, B., Ed., and P. Aitken,
	"Specification of the IP Flow Information Export (IPFIX)		"Specification of the IP Flow Information Export (IPFIX)
	Protocol for the Exchange of Flow Information", STD 77,		Protocol for the Exchange of Flow Information", STD 77,
	RFC 7011, DOI 10.17487/RFC7011, September 2013,		RFC 7011, DOI 10.17487/RFC7011, September 2013,
	<https://www.rfc-editor.org/info/rfc7011>.		<https://www.rfc-editor.org/info/rfc7011>.
			[RFC7258] Farrell, S. and H. Tschofenig, "Pervasive Monitoring Is an
			Attack", BCP 188, RFC 7258, DOI 10.17487/RFC7258, May
			2014, <https://www.rfc-editor.org/info/rfc7258>.
	[RFC7276] Mizrahi, T., Sprecher, N., Bellagamba, E., and Y.		[RFC7276] Mizrahi, T., Sprecher, N., Bellagamba, E., and Y.
	Weingarten, "An Overview of Operations, Administration,		Weingarten, "An Overview of Operations, Administration,
	and Maintenance (OAM) Tools", RFC 7276,		and Maintenance (OAM) Tools", RFC 7276,
	DOI 10.17487/RFC7276, June 2014,		DOI 10.17487/RFC7276, June 2014,
	<https://www.rfc-editor.org/info/rfc7276>.		<https://www.rfc-editor.org/info/rfc7276>.
	[RFC7540] Belshe, M., Peon, R., and M. Thomson, Ed., "Hypertext		[RFC7540] Belshe, M., Peon, R., and M. Thomson, Ed., "Hypertext
	Transfer Protocol Version 2 (HTTP/2)", RFC 7540,		Transfer Protocol Version 2 (HTTP/2)", RFC 7540,
	DOI 10.17487/RFC7540, May 2015,		DOI 10.17487/RFC7540, May 2015,
	<https://www.rfc-editor.org/info/rfc7540>.		<https://www.rfc-editor.org/info/rfc7540>.
	skipping to change at page 33, line 24 ¶		skipping to change at page 34, line 45 ¶
	[RFC7854] Scudder, J., Ed., Fernando, R., and S. Stuart, "BGP		[RFC7854] Scudder, J., Ed., Fernando, R., and S. Stuart, "BGP
	Monitoring Protocol (BMP)", RFC 7854,		Monitoring Protocol (BMP)", RFC 7854,
	DOI 10.17487/RFC7854, June 2016,		DOI 10.17487/RFC7854, June 2016,
	<https://www.rfc-editor.org/info/rfc7854>.		<https://www.rfc-editor.org/info/rfc7854>.
	[RFC7950] Bjorklund, M., Ed., "The YANG 1.1 Data Modeling Language",		[RFC7950] Bjorklund, M., Ed., "The YANG 1.1 Data Modeling Language",
	RFC 7950, DOI 10.17487/RFC7950, August 2016,		RFC 7950, DOI 10.17487/RFC7950, August 2016,
	<https://www.rfc-editor.org/info/rfc7950>.		<https://www.rfc-editor.org/info/rfc7950>.
			[RFC8040] Bierman, A., Bjorklund, M., and K. Watsen, "RESTCONF
			Protocol", RFC 8040, DOI 10.17487/RFC8040, January 2017,
			<https://www.rfc-editor.org/info/rfc8040>.
			[RFC8084] Fairhurst, G., "Network Transport Circuit Breakers",
			BCP 208, RFC 8084, DOI 10.17487/RFC8084, March 2017,
			<https://www.rfc-editor.org/info/rfc8084>.
			[RFC8085] Eggert, L., Fairhurst, G., and G. Shepherd, "UDP Usage
			Guidelines", BCP 145, RFC 8085, DOI 10.17487/RFC8085,
			March 2017, <https://www.rfc-editor.org/info/rfc8085>.
			[RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data
			Interchange Format", STD 90, RFC 8259,
			DOI 10.17487/RFC8259, December 2017,
			<https://www.rfc-editor.org/info/rfc8259>.
	[RFC8321] Fioccola, G., Ed., Capello, A., Cociglio, M., Castaldelli,		[RFC8321] Fioccola, G., Ed., Capello, A., Cociglio, M., Castaldelli,
	L., Chen, M., Zheng, L., Mirsky, G., and T. Mizrahi,		L., Chen, M., Zheng, L., Mirsky, G., and T. Mizrahi,
	"Alternate-Marking Method for Passive and Hybrid		"Alternate-Marking Method for Passive and Hybrid
	Performance Monitoring", RFC 8321, DOI 10.17487/RFC8321,		Performance Monitoring", RFC 8321, DOI 10.17487/RFC8321,
	January 2018, <https://www.rfc-editor.org/info/rfc8321>.		January 2018, <https://www.rfc-editor.org/info/rfc8321>.
	[RFC8639] Voit, E., Clemm, A., Gonzalez Prieto, A., Nilsen-Nygaard,		[RFC8639] Voit, E., Clemm, A., Gonzalez Prieto, A., Nilsen-Nygaard,
	E., and A. Tripathy, "Subscription to YANG Notifications",		E., and A. Tripathy, "Subscription to YANG Notifications",
	RFC 8639, DOI 10.17487/RFC8639, September 2019,		RFC 8639, DOI 10.17487/RFC8639, September 2019,
	<https://www.rfc-editor.org/info/rfc8639>.		<https://www.rfc-editor.org/info/rfc8639>.
	[RFC8641] Clemm, A. and E. Voit, "Subscription to YANG Notifications		[RFC8641] Clemm, A. and E. Voit, "Subscription to YANG Notifications
	for Datastore Updates", RFC 8641, DOI 10.17487/RFC8641,		for Datastore Updates", RFC 8641, DOI 10.17487/RFC8641,
	September 2019, <https://www.rfc-editor.org/info/rfc8641>.		September 2019, <https://www.rfc-editor.org/info/rfc8641>.
			[RFC8762] Mirsky, G., Jun, G., Nydell, H., and R. Foote, "Simple
			Two-Way Active Measurement Protocol", RFC 8762,
			DOI 10.17487/RFC8762, March 2020,
			<https://www.rfc-editor.org/info/rfc8762>.
			[RFC8924] Aldrin, S., Pignataro, C., Ed., Kumar, N., Ed., Krishnan,
			R., and A. Ghanwani, "Service Function Chaining (SFC)
			Operations, Administration, and Maintenance (OAM)
			Framework", RFC 8924, DOI 10.17487/RFC8924, October 2020,
			<https://www.rfc-editor.org/info/rfc8924>.
			[xml] "Extensible Markup Language (XML) 1.0 (Fifth Edition)",
			<https://www.w3.org/TR/2008/REC-xml-20081126/>.
	Appendix A. A Survey on Existing Network Telemetry Techniques		Appendix A. A Survey on Existing Network Telemetry Techniques
	In this non-normative appendix, we provide an overview of some		In this non-normative appendix, we provide an overview of some
	existing techniques and standard proposals for each network telemetry		existing techniques and standard proposals for each network telemetry
	module.		module.
	A.1. Management Plane Telemetry		A.1. Management Plane Telemetry
	A.1.1. Push Extensions for NETCONF		A.1.1. Push Extensions for NETCONF
	NETCONF [RFC6241] is a popular network management protocol		NETCONF [RFC6241] is a popular network management protocol
	skipping to change at page 37, line 5 ¶		skipping to change at page 39, line 5 ¶
	generation. The data subscription needs to define the derived data		generation. The data subscription needs to define the derived data
	which can be composed and derived from the raw data sources. The		which can be composed and derived from the raw data sources. The
	data generation takes advantage of the moderate in-network computing		data generation takes advantage of the moderate in-network computing
	to produce the desired data.		to produce the desired data.
	While DNP can introduce unforeseeable flexibility to the data plane		While DNP can introduce unforeseeable flexibility to the data plane
	telemetry, it also faces some challenges. It requires a flexible		telemetry, it also faces some challenges. It requires a flexible
	data plane that can be dynamically reprogrammed at run-time. The		data plane that can be dynamically reprogrammed at run-time. The
	programming API is yet to be defined.		programming API is yet to be defined.
	A.3.3. IP Flow Information Export (IPFIX) protocol		A.3.3. IP Flow Information Export (IPFIX) Protocol
	Traffic on a network can be seen as a set of flows passing through		Traffic on a network can be seen as a set of flows passing through
	network elements. IP Flow Information Export (IPFIX) [RFC7011]		network elements. IP Flow Information Export (IPFIX) [RFC7011]
	provides a means of transmitting traffic flow information for		provides a means of transmitting traffic flow information for
	administrative or other purposes. A typical IPFIX enabled system		administrative or other purposes. A typical IPFIX enabled system
	includes a pool of Metering Processes that collects data packets at		includes a pool of Metering Processes that collects data packets at
	one or more Observation Points, optionally filters them and		one or more Observation Points, optionally filters them and
	aggregates information about these packets. An Exporter then gathers		aggregates information about these packets. An Exporter then gathers
	each of the Observation Points together into an Observation Domain		each of the Observation Points together into an Observation Domain
	and sends this information via the IPFIX protocol to a Collector.		and sends this information via the IPFIX protocol to a Collector.
	skipping to change at page 37, line 44 ¶		skipping to change at page 39, line 44 ¶
	A.3.5. Postcard Based Telemetry		A.3.5. Postcard Based Telemetry
	PBT [I-D.song-ippm-postcard-based-telemetry] is a proposed		PBT [I-D.song-ippm-postcard-based-telemetry] is a proposed
	complementary technique to IOAM. PBT directly exports data at each		complementary technique to IOAM. PBT directly exports data at each
	node through an independent packet. At the cost of higher bandwidth		node through an independent packet. At the cost of higher bandwidth
	overhead and the need for data correlation, PBT shows several		overhead and the need for data correlation, PBT shows several
	advantages over IOAM. It can also help to identify packet drop		advantages over IOAM. It can also help to identify packet drop
	location in case a packet is dropped on its forwarding path.		location in case a packet is dropped on its forwarding path.
			A.3.6. Existing OAM for Specific Data Planes
			Various data planes raises unique OAM requirements. IETF has
			published OAM technique and framework documents (e.g., [RFC8924] and
			[RFC5085]) targeting different data planes such as MPLS, L2-VPN,
			NVO3, VXLAN, BIER, SFC, and DETNET. The aforementioned data plane
			telemetry techniques can be used to enhance the OAM capability on
			such data planes.
	A.4. External Data and Event Telemetry		A.4. External Data and Event Telemetry
	A.4.1. Sources of External Events		A.4.1. Sources of External Events
	To ensure that the information provided by external event detectors		To ensure that the information provided by external event detectors
	and used by the network management solutions is meaningful for		and used by the network management solutions is meaningful for
	management purposes, the network telemetry framework must ensure that		management purposes, the network telemetry framework must ensure that
	such detectors (sources) are easily connected to the management		such detectors (sources) are easily connected to the management
	solutions (sinks). This requires the specification of a list of		solutions (sinks). This requires the specification of a list of
	potential external data sources that could be of interest in network		potential external data sources that could be of interest in network
	management and match it to the connectors and/or interfaces required		management and match it to the connectors and/or interfaces required
	to connect them.		to connect them.
	skipping to change at page 39, line 39 ¶		skipping to change at page 41, line 41 ¶
	those situations there will be a special connector that provides the		those situations there will be a special connector that provides the
	typical interfaces found in most other elements connected to the		typical interfaces found in most other elements connected to the
	management plane. For instance, the interfaces could accomplish this		management plane. For instance, the interfaces could accomplish this
	with a specific data model (YANG) and specific telemetry protocol,		with a specific data model (YANG) and specific telemetry protocol,
	such as NETCONF, YANG-Push, or gRPC.		such as NETCONF, YANG-Push, or gRPC.
	Authors' Addresses		Authors' Addresses
	Haoyu Song		Haoyu Song
	Futurewei		Futurewei
	2330 Central Expressway
	Santa Clara,
	United States of America		United States of America
	Email: haoyu.song@futurewei.com		Email: haoyu.song@futurewei.com
	Fengwei Qin		Fengwei Qin
	China Mobile		China Mobile
	No. 32 Xuanwumenxi Ave., Xicheng District
	Beijing, 100032
	P.R. China		P.R. China
	Email: qinfengwei@chinamobile.com
			Email: qinfengwei@chinamobile.com
	Pedro Martinez-Julia		Pedro Martinez-Julia
	NICT		NICT
	4-2-1, Nukui-Kitamachi, Tokyo
	184-8795
	Japan		Japan
	Email: pedro@nict.go.jp		Email: pedro@nict.go.jp
	Laurent Ciavaglia		Laurent Ciavaglia
	Nokia		Rakuten Mobile
	91460 Villarceaux
	France		France
	Email: laurent.ciavaglia@nokia.com		Email: laurent.ciavaglia@rakuten.com
	Aijun Wang		Aijun Wang
	China Telecom		China Telecom
	Beiqijia Town, Changping District
	Beijing, 102209
	P.R. China		P.R. China
	Email: wangaj.bri@chinatelecom.cn		Email: wangaj.bri@chinatelecom.cn
End of changes. 44 change blocks.
	104 lines changed or deleted		184 lines changed or added
This html diff was produced by rfcdiff 1.48. The latest version is available from http://tools.ietf.org/tools/rfcdiff/