draft-ietf-ippm-npmps-04.txt   draft-ietf-ippm-npmps-05.txt 
Network Working Group V. Raisanen
INTERNET-DRAFT Nokia
Expiration Date: July 2001 G. Grotefeld
Motorola
January 2001
Network performance measurement for periodic streams
<draft-ietf-ippm-npmps-04.txt>
1. Status of this Memo
IP Performance Measurement Working Group V.Raisanen
Internet Draft Nokia
Document: <draft-ietf-ippm-npmps-05.txt> G.Grotefeld
Category: Informational Motorola
A.Morton
AT&T Labs
Network performance measurement with periodic streams
Status of this Memo
This document is an Internet-Draft and is in full conformance with This document is an Internet-Draft and is in full conformance with
all provisions of Section 10 of RFC2026. all provisions of Section 10 of RFC2026 [1].
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet- other groups may also distribute working documents as Internet-
Drafts. Drafts. Internet-Drafts are draft documents valid for a maximum of
six months and may be updated, replaced, or made obsolete by other
Internet-Drafts are draft documents valid for a maximum of six months documents at any time. It is inappropriate to use Internet-Drafts as
and may be updated, replaced, or obsoleted by other documents at any reference material or to cite them other than as "work in progress."
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt http://www.ietf.org/ietf/1id-abstracts.txt
The list of Internet-Draft Shadow Directories can be accessed at
The list of Internet-Draft shadow directories can be accessed at http://www.ietf.org/shadow.html.
http://www.ietf.org/shadow.html 1. Abstract
This memo describes a periodic sampling method and relevant metrics
This memo provides information for the Internet community. This for assessing the performance of IP networks. First, the memo
memo does not specify an Internet standard of any motivates periodic sampling and addresses the question of its value
kind. Distribution of this memo is unlimited. as an alternative to Poisson sampling described in RFC 2330. The
benefits include applicability to active and passive measurements,
2. Abstract simulation of constant bit rate (CBR) traffic (typical of multimedia
communication, or nearly CBR, as found with voice activity
This document describes a sample metric suitable for application- detection), and several instances where analysis can be simplified.
level IP network transport measurement for periodic streams, such as The sampling method avoids predictability by mandating random start
VoIP or streaming multimedia over IP. In this document, the reader times and finite length tests. Following descriptions of the
is assumed to be familiar with the terminology of the Framework for sampling method and sample metric parameters, measurement methods
IP Performance Metrics RFC 2330 [1]. This document is parallel to and errors are discussed. Finally, we give additional information on
A One-way Delay Metric for IPPM RFC 2679 [2]. Although this document periodic measurements including security considerations.
is based on the delay metrics, other characteristics can be measured 2. Conventions used in this document
with this approach, too. For example, packet loss rate, reordering /
out-of sequence, and successive delay variation are all additional
metrics which can be built from this baseline set of measurements.
Raisanen, Grotefeld expires July 2001 [Page 1]
3. Introduction
This document discusses concepts relevant to application-level
performance measurements of an IP network. The original driver for
this work is Quality of Service of interactive periodic streams such
as multimedia conference over IP, but the idea of application-level
measurement may have a wider scope. In the following, interactive
multimedia traffic is used as an example to illustrate the concept.
A constant bit-rate (CBR), or nearly CBR, streaming (hereinafter
called periodic) multimedia bit stream may be simulated by
transmitting uniformly sized packets (or mostly uniformly sized
packets) at regular intervals through the network to be evaluated.
The "mostly uniformly sized packets" may be found in applications
that may use smaller packets during a portion of the stream (e.g.
digitally coded voice during silence periods). As noted in the
framework document [1], a sample metric using regularly spaced
singleton tests has some limitations when considered from a
general measurement point of view: only part of the network
performance spectrum is sampled. However, from the point of view of
application-level performance, this is actually good news as
explained below.
IP delivery service measurements have been discussed within the
International Telecommunications Union (ITU). A framework for IP
service level measurements (with references to the framework for IP
performance [1]) that is intended to be suitable for service planning
has been approved as I.380 [3]. The emphasis in the ITU
recommendation is on passive measurements, though not explicitly
forbidding active measurements. The present contribution proposes a
method that is usable both for service planning and end-user testing
purposes, and is based on active measurements.
3.1 Terminology
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [4]. document are to be interpreted as described in RFC 2119 [2].
Although RFC 2119 was written with protocols in mind, the key words Although RFC 2119 was written with protocols in mind, the key words
Raisanen,Grotefeld,Morton Informational exp. May 2002 1 Network performance measurement with periodic streams Nov 2001
are used in this document for similar reasons. They are used to are used in this document for similar reasons. They are used to
ensure the results of measurements from two different implementations ensure the results of measurements from two different
are comparable, and to note instances when an implementation could implementations are comparable, and to note instances when an
perturb the network. implementation could perturb the network.
3. Introduction
Raisanen, Grotefeld expires July 2001 [Page 2] This memo describes a sampling method and performance metrics
relevant to certain applications of IP networks. The original driver
3.2 Considerations related to delay for this work was Quality of Service of interactive periodic streams
such as multimedia conferencing over IP, but the idea of periodic
For interactive multimedia sessions, end-to-end delay is an sampling and measurement has wider applicability. Interactive
important factor. Too large a delay reduces the quality of the multimedia traffic is used as an example below to illustrate the
multimedia session as perceived by the participants. One approach for concept.
managing end-to-end delays on an Internet path involving Transmitting equal size packets (or mostly same-size packets)
heterogeneous link layer technologies is to use per-domain delay through a network at regular intervals simulates a constant bit-rate
quotas (e.g. 50 ms for a particular IP domain). The 50 ms would (CBR), or nearly CBR multimedia bit stream. Hereafter, these packets
then be included into a calculation of an end-to-end delay bound. A are called periodic streams. Cases of "mostly same-size packets" may
practical implementation of such as scheme ought to address issues be found in applications that have multiple coding methods (e.g.
like possibility of asymmetric delays in a route in different digitally coded comfort noise during silence gaps in speech).
directions, and sensitivity of an application to delay variations in In the following sections, a sampling methodology and metrics are
a given domain. There are several alternatives as to which kind of presented for periodic streams. The measurement results may be used
derivative delay metric one ought to use in managing end-to-end QoS. in derivative metrics such as average and maximum delays. The memo
This question, although very interesting, is not within the scope of seeks to formalize periodic stream measurements to achieve
this draft and is not discussed further here. comparable results between independent implementations.
3.1 Motivation
In the following, a methodology and metric are presented for As noted in the IPPM framework RFC 2330 [3], a sample metric using
measuring media stream transport QoS in an IP domain. The regularly spaced singleton tests has some limitations when
measurement results may be used in derivative metrics such as considered from a general measurement point of view: only part of
average and maximum delays. A metric is presented that is a standard the network performance spectrum is sampled. However, some
way for performing a measurement irrespective of the possible QoS applications also sample this limited performance spectrum and their
mechanism utilized in the core network. As an example, for a QoS performance may be of critical interest.
mechanism without hard guarantees, measurements may be used to Periodic sampling is useful for the following reasons:
ascertain that the "best" class gets the service that has been * It is applicable to passive measurement, as well as active
promised for the traffic class in question. Moreover, an operator measurement.
could study the quality of a cheap, low-guarantee service * An active measurement can be configured to match the
implemented using possible slack bandwidth in other classes. Such characteristics of media flows, and simplifies the estimation of
measurements could be made either in studying the feasibility of a application performance.
new service, or on a regular basis. * Measurements of many network impairments (e.g., delay variation,
consecutive loss, reordering) are sensitive to the sampling
The present draft seeks to formalize the measurements in such a way frequency. When the impairments themselves are time-varying (and
that interoperable results are achieved. the variations are somewhat rare, yet important), a constant
sampling frequency simplifies analysis.
3.3 Protocol level issues Raisanen,Grotefeld,Morton Informational exp.May 2002 2 Network performance measurement with periodic streams Nov 2001
* Frequency Domain analysis is simplified when the samples are
The version of the Internet Protocol used in the measurement affects equally spaced.
(at least) packet sizes, and should be reported. Simulation of CBR flows with periodic streams encourages dense
sampling of network performance, since typical multimedia flows have
Fig.1 illustrates measurements on multiple protocol levels that 10 to 100 packets in each second. Dense sampling permits the
are relevant to this draft. The major focus of the present draft characterization of network phenomena with short duration.
is on transport quality evaluation from application point of 4. Periodic Sampling Methodology
view. However, to properly account for quality effects of, e.g., The Framework RFC [3] points out the following potential problems
operating system and codec on packet voice, it is beneficial to be with Periodic Sampling:
able to measure quality at IP level [5]. Link layer monitoring 1. The performance sampled may be synchronized with some other
provides a way of accounting for link layer characteristics such periodic behavior, or the samples may be anticipated and the results
as bit error rates. manipulated. Unpredictable sampling is preferred.
2. Active measurements can cause congestion, and periodic sampling
Raisanen, Grotefeld expires July 2001 [Page 3] might drive congestion-aware senders into a synchronized state,
--------------- producing atypical results.
| application | Poisson sampling produces an unbiased sample for the various IP
--------------- performance metrics, yet there are situations where alternative
| transport | <-- sampling methods are advantageous (as discussed under Motivation).
--------------- We can prescribe periodic sampling methods that address the problems
| network | <-- listed above. Predictability and some forms of synchronization can
--------------- be mitigated through the use of random start times and limited
| link | <-- stream duration over a test interval. The periodic sampling
--------------- parameters produce bias, and judicious selection can produce a known
| physical | bias of interest. The total traffic generated by this or any
--------------- sampling method should be limited to avoid adverse affects on non-
test traffic (packet size, packet rate, and sample duration and
Fig. 1: Different possibilities for performing measurements: a frequency should all be considered).
protocol view. Above, "application" refers to all layers above The configuration parameters of periodic sampling are:
L4 and is not used in the OSI sense. + T, the beginning of a time interval where a periodic sample is
desired.
In general, the results of measurements may be influenced by + dT, the duration of the interval for allowed sample start times.
individual application requirements/responses related to the + T0, a time that MUST be selected at random from the interval [T,
following issues: T+dT] to start generating packets and taking measurements.
+ Tf, a time, greater than T0, for stopping generation of packets
+ Lost packets: Applications may have varying tolerance to lost for a sample (Tf may be relative to T0 if desired).
packets. Another consideration is the distribution of lost + incT, the nominal duration of inter-packet interval, first bit to
packets (i.e. random or bursty). first bit.
+ Long delays: Many applications will consider packets delayed T0 may be drawn from a uniform distribution, or T0 = T + Unif(0,dT).
longer than a certain value to be equivalent to lost packets Other distributions may also be appropriate. Start times in
(i.e. real time applications). successive time intervals MUST use an independent value drawn from
+ Duplicate packets: Some applications may be perturbed if the distribution. In passive measurement, the arrival of user media
duplicate packets are received. Raisanen,Grotefeld,Morton Informational exp.May 2002 3 Network performance measurement with periodic streams Nov 2001
+ Out-of-sequence: Some applications may be perturbed if packets flows may have sufficient randomness, or a randomized start time of
are received out of sequence. This may be in addition to the the measurement during a flow may be needed to meet this
possibility of exceeding the "long" delay threshold as a result requirement.
of being out of sequence. An out-of-sequence packet outcome When a mix of packet sizes is desired, passive measurements usually
occurs when a single IP packet received at a DST measurement possess the sequence and statistics of sizes in actual use, while
point has a sequence number higher than that which is active measurements would need to reproduce the intended
expected, and therefore, the packet is OOS due to re-ordering. distribution of sizes.
+ Corrupt packet header: Most applications will probably treat a 5. Sample metrics for periodic streams
packet with a corrupt header as equivalent to a lost packet.
+ Corrupt packet payload: Some applications (e.g. digital voice
codecs) may accept corrupt packet payload. In some cases, the
packet payload may contain application specific forward error
correction (FEC) that can compensate for some level of
corruption.
+ Spurious packet: Dst may receive spurious packets (i.e. packets
that are not sent by the Src as part of the metric). Many
applications may be perturbed by spurious packets.
Depending, e.g., on the observed protocol level, some issues listed
above may be indistinguishable from others by the application, it
may be important to preserve the distinction for the operators of
Src, Dst, and/or the intermediate network(s).
Raisanen, Grotefeld expires July 2001 [Page 4]
Because of the possible errors listed above, in most cases it is
recommended to use a packet identifier for each packet generated at
Src. Identifiers for the metric sample may be those used by the
underlying transport layer (e.g. RTP sequence number) or the same
identifiers used by an application if the application to be modeled
by the metric uses an identifier. The possibility of identifier
roll-over (reuse if intentional) during a metric collected over
a "long" (application dependent) time should be observed.
If the application does not use an identifier, it may still be
useful to add identifiers to the packets in the metric sample to
help identify possible anomalies such as out of sequence packets.
This would be most useful in the case where the application
expects to receive packets in sequence, but has no capability to
identify the sequence of packets received at Dst.
3.4 Application-level measurement
In what follows, a metric is proposed for application-level network
performance measurement. In effect, the metric is an emulation of
periodic multimedia stream performance. The justification for using
realistic application metrics in the measurement:
+ The results of the measurement are automatically relevant to the
performance as perceived by the application in question.
+ All the packets in the measurement contribute to accuracy of the
estimation of performance variation at timescale that is
important to the multimedia application (packetization
interval).
+ Effects of elastic traffic (TCP) on measurement packets are
different for a sustained stream than for single packets during
overloading situations as discussed in [3].
3.5 Measurement types
Delay measurements can be one-way [2,3], paired one-way, or
round-trip [6]. Accordingly, the measurements may be performed
either with synchronized or unsynchronized Src/Dst host clocks.
Different possibilities are listed below.
The reference measurement setup for all measurement types is
shown in Fig. 2.
Raisanen, Grotefeld expires July 2001 [Page 5]
----------------< IP >--------------------
| | | |
------- ------- -------- --------
| Src | | MP | | MP | | Dst |
------- |(Src)| |(Dst) | --------
------- --------
Fig. 2: Example setup for the metric usage.
An example of the use of the metric is a setup with a source host
(Src), a destination host (Dst), and corresponding measurement
points (MP(Src) and MP(Dst)) as shown in Figure 2. Separate equipment
for measurement points may be used if having Src and/or Dst conduct
the measurement may significantly affect the delay performance to be
measured. MP(Src)should be placed/measured close to the egress point
of packets from Src. MP(Dst) should be placed/measure close to
the ingress point of packets for Dst. "Close" is defined as a
distance sufficiently small so that application-level performance
characteristics measured (such as delay) can be expected to follow
the corresponding performance characteristic between Src and Dst to
an adequate accuracy. Basic principle here is that measurement
results between MP(Src) and MP(Dst) should be the same as for a
measurement between Src and Dst, within the general error margin
target of the measurement (e.g., < 1 ms; number of lost packets is
the same). If this is not possible, the difference between MP-MP
measurement and Src-Dst measurement should preferably be systematic.
The test setup just described fulfills two important criteria:
1) Test is made with realistic stream metrics, emulating - for example -
a full-duplex Voice over IP (VoIP) call.
2) Either one-way or round-trip characteristics may be obtained.
It is also possible to have intermediate measurement points between
MP(Src) and MP(Dst), but that is beyond the scope of this document.
3.5.1 One way measurement
In the interests of specifying metrics that are as generally usable
as possible, application-level measurements based on one-way delays
are used in the example metrics. The implication of application-level
measurement for bi-directional applications such as interactive
multimedia conferencing is discussed below.
Performing a single one-way measurement only yields information on
network behavior in one direction. Moreover, the stream at the
network transport level does not emulate accurately a full-duplex
multimedia connection.
Raisanen, Grotefeld expires July 2001 [Page 6]
3.5.2 Paired one way measurement
Paired one way delay refers to two multimedia streams: Src to Dst
and Dst to Src for the same Src and Dst. By way of example, for
some applications, the delay performance of each one way path is
more important than the round trip delay. This is the case for
delay-limited signals such as VoIP. Possible reasons for the
difference between one-way delays is different routing of streams
from Src to Dst vs. Dst to Src.
For example, a paired one way measurement may show that Src to Dst
has an average delay of 30ms while Dst to Src has an average delay
of 120ms. To a round trip delay measurement, this example would
look like an average of 150ms delay. Without the knowledge of the
asymmetry, we might miss a problem that the application at either
end may have with delays averaging more than 100ms.
Moreover, paired one way delay measurement emulates a full-duplex
VoIP call more accurately than a single one-way measurement only.
3.5.3 Round trip measurement
From the point of view of periodic multimedia streams,
round-trip measurements have two advantages: they avoid the need of
host clock synchronization and they allow for a simulation of
full-duplex connections. The former aspect means that a measurement
is easily performed, since no special equipment or NTP setup is
needed. The latter property means that measurement streams are
transmitted in both directions. Thus, the measurement provides
information on quality of service as experienced by appropriate
application.
The downsides of round-trip measurement are the need for more
bandwidth than an one-way test and more complex accounting of
packet loss. Moreover, the stream that is returning towards the
original sender may be more bursty than the one on the first "leg" of
the round-trip journey. The last issue, however, means in practice
that returning stream experiences worse QoS than the other one, and
the performance estimates thus obtained are pessimistic ones. The
possibility of asymmetric routing and queuing must be taken into
account during analysis of the results.
Please note that with suitable arrangements, round-trip measurements
may be performed using paired one way measurements.
Raisanen, Grotefeld expires July 2001 [Page 7]
4 Sample metric for multimedia stream simulation
The sample metric presented here is similar to the sample metric The sample metric presented here is similar to the sample metric
Type-P-One-way-Delay-Poisson-Stream presented in [2]. "Singletons", as Type-P-One-way-Delay-Poisson-Stream presented in RFC 2679[4].
defined in [1] and [2] are not directly used in this document because Singletons defined in [3] and [4] are applicable here.
certain key results (such as duplicate or out of sequence packets) 5.1 Metric name
cannot be identified in the context of a singleton, but only as part
of a sample.
4.1 Metric name
Type-P-One-way-Delay-Periodic-Stream Type-P-One-way-Delay-Periodic-Stream
5.2 Metric parameters
4.2 Metric parameters 5.2.1 Global metric parameters
These parameters apply in all the sub-sections that follow (5.2.2,
4.2.1 Global metric parameters 5.2.3, and 5.2.4).
Parameters that each Singleton usually includes:
These parameters are applicable to the metrics collected in the
following sections (4.2.2, 4.2.3, and 4.2.4).
+ Src, the IP address of a host + Src, the IP address of a host
+ Dst, the IP address of a host + Dst, the IP address of a host
+ IPV, the IP version (IPv4/IPv6) used in the measurement + IPV, the IP version (IPv4/IPv6) used in the measurement
+ T0, a time, for starting to generate packets and taking + dTloss, a time interval, the maximum waiting time for a packet
measurements for a sample before declaring it lost.
+ Tf, a time, greater than T0, for stopping generation of packets + packet size p(j), the desired number of bytes in the Type-P
for a sample packet, where j is the size index.
+ incT, nominal duration of inter-packet interval Optional parameters:
+ packet size p(j), the number of bytes in each packet of Type-P of + PktType, any additional qualifiers (transport address)
size j + Tcons, a time interval for consolidating parameters collected at
+ dTloss, a time interval, used for determining if a packet should the measurement points.
be considered lost
+ Tcons, a time interval [optional]
While a number of applications will use one packet size (j = 1), While a number of applications will use one packet size (j = 1),
other applications may use packets of different sizes (j > 1). other applications may use packets of different sizes (j > 1).
Especially in cases of congestion, it may be useful to have Especially in cases of congestion, it may be useful to use packets
packets smaller than the maximum or predominant size of packets smaller than the maximum or predominant size of packets in the
in the periodic stream. periodic stream.
A topology where Src and Dst are separate from the measurement
4.2.2 Metrics collected at MP(Src) points is assumed.
5.2.2 Parameters collected at the measurement point MP(Src)
Parameters that each Singleton usually includes:
Raisanen,Grotefeld,Morton Informational exp.May 2002 4 Network performance measurement with periodic streams Nov 2001
+ Tstamp(Src)[i], for each packet [i], the time of the packet as + Tstamp(Src)[i], for each packet [i], the time of the packet as
measured at MP(Src) measured at MP(Src)
+ PktID [i], for each packet [i], an identification number for the Additional parameters:
the packet sent from Src to Dst. + PktID(Src) [i], for each packet [i], a unique identification or
sequence number.
Raisanen, Grotefeld expires July 2001 [Page 8] + PktSi(Src) [i], for each packet [i], the actual packet size.
+ PktSiTy [i], for each packet [i], the packet size and/or type. Some applications may use packets of different sizes, either
Some applications may use packets of different size, either
because of application requirements or in response to IP because of application requirements or in response to IP
performance experienced. performance experienced.
5.2.3 Parameters collected at the measurement point MP(Dst)
4.2.3 Metrics collected at MP (Dst)
+ dTstop, a time interval, used to add to time Tf to determine when to
stop collecting metrics for a sample
+ Tstamp(Dst)[i], for each packet [i], the time of the packet as + Tstamp(Dst)[i], for each packet [i], the time of the packet as
measured at MP(Dst) measured at MP(Dst)
+ PktID [i], for each packet [i], an identification number for the + PktID(Dst) [i], for each packet [i], a unique identification or
the packet received at Dst from Src. sequence number.
+ PktSiTy [i], for each packet [i], the packet size and/or type. + PktSi(Dst) [i], for each packet [i], the actual packet size.
Some applications may use packets of different size, either Optional parameters:
because of application requirements or in response to IP + dTstop, a time interval, used to add to time Tf to determine when
performance experienced. to stop collecting metrics for a sample
+ PktStatus [i], for each packet [i], the status of the packet
received. Possible status includes: OK, packet header corrupt,
packet payload corrupt, spurious, duplicate, out-of-sequence.
4.2.4 Metrics resulting when metrics collected at MP(Src) and MP(Dst)
are merged
These parameters are only available as a complete set when the
parameters from the preceding sections (4.2.1, 4.2.2, and 4.2.3 are
combined.
+ Tstamp(Src)[i], for each packet [i], the time of the packet as
measured at MP(Src). This entry may be blank or noted as N/A
for spurious packets received at MP(Dst)
+ Tstamp(Dst)[i], for each packet [i], the time of the packet as
measured at MP(Dst). This entry may be blank or noted as N/A
for packets not received at MP(Dst), received with corrupt
packet headers, or for duplicate packets received at MP(Dst).
+ PktID [i], for each packet [i], an identification number for the
the packet received. This identification number may be corrupted
for certain packets received at MP (Dst).
+ PktSiTy [i], for each packet [i], the packet size and/or type.
+ PktStatus [i], for each packet [i], the status of the packet + PktStatus [i], for each packet [i], the status of the packet
received. Possible status includes: OK, packet header corrupt, received. Possible status includes OK, packet header corrupt,
packet payload corrupt, spurious, duplicate, out-of-sequence. packet payload corrupt, duplicate, fragment. The criteria to
determine the status MUST be specified, if used.
Raisanen, Grotefeld expires July 2001 [Page 9] 5.2.4 Sample Metrics resulting from combining parameters at MP(Src) and
+ Delay [i], for each packet [i], the time interval Tstamp(Dst)[i] - MP(Dst)
Tstamp(Src)[i]. For the following conditions, it will not be Using the parameters above, a delay singleton would be calculated as
possible to be able to compute delay: follows:
+ Delay [i], for each packet [i], the time interval
Delay[i] = Tstamp(Dst)[i] - Tstamp(Src)[i]
For the following conditions, it will not be possible to be able to
compute delay singletons:
Spurious: There will be no Tstamp(Src)[i] time Spurious: There will be no Tstamp(Src)[i] time
Not received: There will be no Tstamp (Dst) [i] Not received: There will be no Tstamp (Dst) [i]
Corrupt packet header: There will be no Tstamp (Dst) [i] Corrupt packet header: There will be no Tstamp (Dst) [i]
Duplicate: Only the first non-corrupt copy of the packet Duplicate: Only the first non-corrupt copy of the packet
received at Dst should have Delay [i] computed. received at Dst should have Delay [i] computed.
+ SDV[i] [optional] , for each packet [i] except the first one: A sample metric for average delay is as follows
momentary delay variation between successive packets, i.e., the AveDelay = (1/N)Sum(from i=1 to N, Delay[i])
time interval Delay[i] - Delay [i-1]. SDV[i] may be negative, assuming all packets i= 1 though N have valid singletons.
zero, or positive. Delay for both packets i and i+1 must be A delay variation [5] singleton can also be computed:
calculable according to the definition above or SDV[i] is Raisanen,Grotefeld,Morton Informational exp.May 2002 5 Network performance measurement with periodic streams Nov 2001
undefined. + IPDV[i], for each packet [i] except the first one, delay
variation between successive packets would be calculated as
4.3 High level description of the procedure to collect a sample IPDV[I] = Delay[i] - Delay [i-1]
IPDV[i] may be negative, zero, or positive. Delay singletons for
Beginning on or after time T0, Type-P packets are generated packets i and i-1 must be calculable or IPDV[i] is undefined.
by Src and sent to Dst until time Tf is reached with a nominal An example metric for the IPDV sample is the range:
interval between the first bit of successive packets of incT as RangeIPDV = max(IPDV[]) - min(IPDV[])
measured at MP(Src). incT may be nominal due to a number of reasons: 5.3 High level description of the procedure to collect a sample
variation in packet generation at Src, clock issues (see section 4.6), Beginning on or after time T0, Type-P packets are generated by Src
etc. and sent to Dst until time Tf is reached with a nominal interval
between the first bit of successive packets of incT as measured at
MP(Src) records the following information only for packets with MP(Src). incT may be nominal due to a number of reasons: variation
timestamps between and including T0 and Tf: timestamp, in packet generation at Src, clock issues (see section 5.6), etc.
packet identifier, and packet size/type of each packet sent from Src MP(Src) records the parameters above only for packets with
to Dst that is part of the sample. timestamps between and including T0 and Tf having the required Src,
Dst, and any other qualifiers. MP (Dst) also records for packets
MP (Dst) records the following information only for packets with with time stamps between T0 and (Tf + dTstop).
time stamps between T0 and (Tf+ dTstop): timestamp, packet identifier, Optionally at a time Tf + Tcons (but eventually in all cases), the
packet size/type, and received status of each packet received from data from MP(Src) and MP(Dst) are consolidated to derive the sample
Src at Dst that is part of the sample. Optionally, at a time Tf + metric results. To prevent stopping data collection too soon,
Tcons, the data from MP(Src) and MP(Dst) are consolidated to derive dTcons should be greater than or equal to dTstop. Conversely, to
the results of the sample metric. keep data collection reasonably efficient, dTstop should be some
reasonable time interval (seconds/minutes/hours), even if dTloss is
To prevent stopping data collection too soon, dTcons should be greater infinite or extremely long.
than or equal to dTstop. Conversely, to keep data collection 5.4 Discussion
reasonably efficient, dTstop should be some reasonable time interval This sampling methodology is intended to quantify the delays and the
(seconds/minutes/hours), even if dTloss is infinite or extremely long. delay variation as experienced by multimedia streams of an
application. Due to the definitions of these metrics, also packet
Raisanen, Grotefeld expires July 2001 [Page 10] loss status is recorded. The nominal interval between packets
assesses network performance variations on a specific time scale.
4.4 Discussion
The sample metric thus defined is intended to probe the delays and
the delay variation as experienced by multimedia streams of
an application. Due to the definition of the metric, also packet loss
status of packets is recorded. The delay is assumed to be measured at
transport layer level. Since a range of packet sizes and nominal
interval between packets is used, the method probes only a specific
time scale of network QoS variations.
There are a number of factors that should be taken into account when There are a number of factors that should be taken into account when
collecting a sample metric of Type-P-One-way-Delay-Periodic-Stream. collecting a sample metric of Type-P-One-way-Delay-Periodic-Stream.
+ The interval T0 to Tf should be specified to cover a long enough
+ T0 and (Tf + dTloss) should specify a long enough time interval to time interval to represent a reasonable use of the application under
represent a reasonable use of the application under test (e.g. do test, yet not excessively long in the same context(e.g. phone calls
not provide only a 100 ms time interval for a phone call) last longer than 100ms, but less than one week).
+ The nominal interval between packets (incT) and the packet
+ T0 and (Tf + dTloss) should specify a time interval that is not size(s) (p(j)) should not define an equivalent bit rate that exceeds
excessively long compared to the usage of the application under test the capacity of the egress port of Src, the ingress port of Dst,
(e.g. do not provide a one week continuous phone call) Raisanen,Grotefeld,Morton Informational exp.May 2002 6 Network performance measurement with periodic streams Nov 2001
or the capacity of the intervening network(s), if known. There may
+ The nominal interval between packets (incT) and the packet size(s)
(p(j)) should not define an equivalent bit rate that is in excess
of the capacity of the egress port of Src, the ingress port of Dst,
or the carrying capacity of the intervening network(s). There may
be exceptional cases to test the response of the application to be exceptional cases to test the response of the application to
overload conditions in the transport networks, but these cases overload conditions in the transport networks, but these cases
should be strictly controlled. should be strictly controlled.
+ Real delay values will be positive. Therefore, it does not make + Real delay values will be positive. Therefore, it does not make
sense to report a negative value as a real delay. However, an sense to report a negative value as a real delay. However, an
individual zero or negative delay value might be useful as part of individual zero or negative delay value might be useful as part of
a stream when trying to discover a distribution of the delay values a stream when trying to discover a distribution of the delay errors.
of a stream.
+ Depending on measurement topology, delay values may be as low as + Depending on measurement topology, delay values may be as low as
100 usec to 10 msec, whereby it may be important for Src and Dst to 100 usec to 10 msec, whereby it may be important for Src and Dst to
synchronize very closely. GPS systems afford one way to achieve synchronize very closely. GPS systems afford one way to achieve
synchronization to within several 10s of usec. Ordinary application synchronization to within several 10s of usec. Ordinary application
of NTP may allow synchronization to within several msec, but this of NTP may allow synchronization to within several msec, but this
depends on the stability and symmetry of delay properties among those depends on the stability and symmetry of delay properties among the
NTP agents used, and this delay is what we are trying to measure. A NTP agents used, and this delay is what we are trying to measure.
combination of some GPS-based NTP servers and a conservatively
designed and deployed set of other NTP servers should yield good
results, but this is yet to be tested.
+ Reordering of packets is best discussed in terms of the entire
set of measurement packets received, i.e. should be addressed in
Sec. 4.9.1.
Raisanen, Grotefeld expires July 2001 [Page 11]
+ A given methodology will have to include a way to determine + A given methodology will have to include a way to determine
whether packet was lost or whether delay is merely very large (and whether packet was lost or whether delay is merely very large (and
the packet is yet to arrive at Dst). The global metric parameter the packet is yet to arrive at Dst). The global metric parameter
dTloss defines a time interval such that delays larger than dTloss dTloss defines a time interval such that delays larger than dTloss
are interpreted as losses. are interpreted as losses. {Comment: For many applications, the
{Comment: Note that, for many applications of these metrics, the treatment a large delay as infinite/loss will be inconsequential. A
harm in treating a large delay as infinite might be zero or very TCP data packet, for example, that arrives only after several
small. A TCP data packet, for example, that arrives only after multiples of the usual RTT may as well have been lost.}
several multiples of the RTT may as well have been lost.} 5.5 Additional Methodology Aspects
4.5 Additional Methodology Aspects
As with other Type-P-* metrics, the detailed methodology will depend As with other Type-P-* metrics, the detailed methodology will depend
on the Type-P (e.g., protocol number, UDP/TCP port number, size, on the Type-P (e.g., protocol number, UDP/TCP port number, size,
precedence). precedence).
5.6 Errors and uncertainties
4.6 Errors and uncertainties
The description of any specific measurement method should include an The description of any specific measurement method should include an
accounting and analysis of various sources of error or uncertainty. accounting and analysis of various sources of error or uncertainty.
The Framework document [1] provides general guidance on this point, The Framework RFC [3] provides general guidance on this point, but
but we note here the following specifics related to periodic we note here the following specifics related to periodic streams and
streams and delay metrics: delay metrics:
+ Error due to variation of incT. The reasons for this can be
+ Error due to variation of incT. The reasons for this can be e.g.
uneven process scheduling, possibly due to CPU load. uneven process scheduling, possibly due to CPU load.
+ Errors or uncertainties due to uncertainties in the clocks of the + Errors or uncertainties due to uncertainties in the clocks of the
MP(Src) and MP(Dst) measurement points. MP(Src) and MP(Dst) measurement points.
+ Errors or uncertainties due to the difference between 'wire time' + Errors or uncertainties due to the difference between 'wire time'
and 'host time'. and 'host time'.
5.6.1. Errors or uncertainties related to Clocks
4.6.1. Errors or uncertainties related to Clocks Raisanen,Grotefeld,Morton Informational exp.May 2002 7 Network performance measurement with periodic streams Nov 2001
The uncertainty in a measurement of one-way delay is related, in The uncertainty in a measurement of one-way delay is related, in
part, to uncertainties in the clocks of MP(Src) and MP(Dst). In part, to uncertainties in the clocks of MP(Src) and MP(Dst). In the
the following, we refer to the clock used to measure when the packet following, we refer to the clock used to measure when the packet was
was measured at MP(Src) as the MP(Src) clock and we refer to the measured at MP(Src) as the MP(Src) clock and we refer to the clock
clock used to measure when the packet was received at MP(Dst) as the used to measure when the packet was received at MP(Dst) as the
MP(Dst) clock. Alluding to the notions of synchronization, accuracy, MP(Dst) clock. Alluding to the notions of synchronization,
resolution, and skew, we note the following: accuracy, resolution, and skew, we note the following:
+ Any error in the synchronization between the MP(Src) clock and + Any error in the synchronization between the MP(Src) clock and
the MP(Dst) clock will contribute to error in the delay the MP(Dst) clock will contribute to error in the delay measurement.
measurement. We say that the MP(Src) clock and the MP(Dst) We say that the MP(Src) clock and the MP(Dst) clock have a
clock have a synchronization error of Tsynch if the MP(Src) clock synchronization error of Tsynch if the MP(Src) clock is Tsynch ahead
is Tsynch ahead of the MP(Dst) clock. Thus, if we know the of the MP(Dst) clock. Thus, if we know the value of Tsynch exactly,
value of Tsynch exactly, we could correct for clock we could correct for clock synchronization by adding Tsynch to the
synchronization by adding Tsynch to the uncorrected value of uncorrected value of Tstamp(Dst)[i] - Tstamp(Src) [i].
Tstamp(Dst)[i] - Tstamp(Src) [i].
Raisanen, Grotefeld expires July 2001 [Page 12]
+ The accuracy of a clock is important only in identifying the time
at which a given delay was measured. Accuracy, per se, has no
importance to the accuracy of the measurement of delay. When
computing delays, we are interested only in the differences
between clock values, not the values themselves.
+ The resolution of a clock adds to uncertainty about any time + The resolution of a clock adds to uncertainty about any time
measured with it. Thus, if the MP(Src) clock has a resolution of measured with it. Thus, if the MP(Src) clock has a resolution of
10 msec, then this adds 10 msec of uncertainty to any time value 10 msec, then this adds 10 msec of uncertainty to any time value
measured with it. We will denote the resolution of the source measured with it. We will denote the resolution of the source
clock and the MP(Dst) clock as ResMP(Src) and ResMP(Dst), clock and the MP(Dst) clock as ResMP(Src) and ResMP(Dst),
respectively. respectively.
+ The skew of a clock is not so much an additional issue as it is a + The skew of a clock is not so much an additional issue as it is a
realization of the fact that Tsynch is itself a function of time. realization of the fact that Tsynch is itself a function of time.
Thus, if we attempt to measure or to bound Tsynch, this needs to Thus, if we attempt to measure or to bound Tsynch, this needs to
be done periodically. Over some periods of time, this function be done periodically. Over some periods of time, this function can
can be approximated as a linear function plus some higher order be approximated as a linear function plus some higher order terms;
terms; in these cases, one option is to use knowledge of the in these cases, one option is to use knowledge of the linear
linear component to correct the clock. Using this correction, the component to correct the clock. Using this correction, the residual
residual Tsynch is made smaller, but remains a source of Tsynch is made smaller, but remains a source of uncertainty that
uncertainty that must be accounted for. We use the function must be accounted for. We use the function Esynch(t) to denote an
Esynch(t) to denote an upper bound on the uncertainty in upper bound on the uncertainty in synchronization. Thus,
synchronization. Thus, |Tsynch(t)| <= Esynch(t). |Tsynch(t)| <= Esynch(t).
Taking these items together, we note that naive computation Taking these items together, we note that naive computation
Tstamp(Dst)[i] - Tstamp(Src) [i] will be off by Tsynch(t) +/- Tstamp(Dst)[i] - Tstamp(Src) [i] will be off by Tsynch(t) +/-
(ResMP(SRc) + ResMP(Dst)). Using the notion of Esynch(t), we note (ResMP(SRc) + ResMP(Dst)). Using the notion of Esynch(t), we note
that these clock-related problems introduce a total uncertainty of that these clock-related problems introduce a total uncertainty of
Esynch(t)+ Rsource + Rdest. This estimate of total clock-related Esynch(t)+ Rsource + Rdest. This estimate of total clock-related
uncertainty should be included in the error/uncertainty analysis of uncertainty should be included in the error/uncertainty analysis of
any measurement implementation. any measurement implementation.
5.6.2. Errors or uncertainties related to Wire-time vs Host-time
4.6.2. Errors or uncertainties related to Wire-time vs Host-time We would like to measure the time between when a packet is measured
and time-stamped at MP(Src) and when it arrives and is time-stamped
As we have defined one-way periodic delay, we would like to measure at MP(Dst) and we refer to these as "wire times." If timestamps are
the time between when a packet is measured and time-stamped at applied by software on Src and Dst, however, then this software can
MP(Src) and when it arrives and is time-stamped at MP(Dst) and we
refer to these as "wire times." If the timings are themselves
performed by software on Src and Dst, however, then this software can
only directly measure the time between when Src generates the packet only directly measure the time between when Src generates the packet
just prior to sending the test packet and when Dst has started to just prior to sending the test packet and when Dst has started to
process the packet after having received the test packet, and we refer Raisanen,Grotefeld,Morton Informational exp.May 2002 8 Network performance measurement with periodic streams Nov 2001
to these two points as "host times". process the packet after having received the test packet, and we
refer to these two points as "host times".
To the extent that the difference between wire time and host time is To the extent that the difference between wire time and host time is
accurately known, this knowledge can be used to correct for wire time accurately known, this knowledge can be used to correct for wire
measurements and the corrected value more accurately estimates the time measurements and the corrected value more accurately estimates
desired (host time) metric. the desired (host time) metric, and visa-versa.
Raisanen, Grotefeld expires July 2001 [Page 13]
To the extent, however, that the difference between wire time and To the extent, however, that the difference between wire time and
host time is uncertain, this uncertainty must be accounted for in an host time is uncertain, this uncertainty must be accounted for in an
analysis of a given measurement method. We denote by Hsource an analysis of a given measurement method. We denote by Hsource an
upper bound on the uncertainty in the difference between wire time upper bound on the uncertainty in the difference between wire time
of MP(Src) and host time on the Src host, and similarly define Hdest of MP(Src) and host time on the Src host, and similarly define Hdest
for the difference between the host time on the Dst host and the wire for the difference between the host time on the Dst host and the
time of MP(Dst). We then note that these problems introduce a total wire time of MP(Dst). We then note that these problems introduce a
uncertainty of Hsource+Hdest. This estimate of total wire-vs-host total uncertainty of Hsource+Hdest. This estimate of total wire-vs-
uncertainty should be included in the error/uncertainty analysis of host uncertainty should be included in the error/uncertainty
any measurement implementation. analysis of any measurement implementation.
5.6.3. Calibration
4.6.3. Calibration
Generally, the measured values can be decomposed as follows: Generally, the measured values can be decomposed as follows:
measured value = true value + systematic error + random error measured value = true value + systematic error + random error
If the systematic error (the constant bias in measured values) can
If the systematic error (the constant bias in measured values) can be be determined, it can be compensated for in the reported results.
determined, it can be compensated for in the reported results.
reported value = measured value - systematic error reported value = measured value - systematic error
therefore therefore
reported value = true value + random error reported value = true value + random error
The goal of calibration is to determine the systematic and random The goal of calibration is to determine the systematic and random
error generated by the instruments themselves in as much detail as error generated by the instruments themselves in as much detail as
possible. At a minimum, a bound ("e") should be found such that the possible. At a minimum, a bound ("e") should be found such that the
reported value is in the range (true value - e) to (true value + e) reported value is in the range (true value - e) to (true value + e)
at least 95 percent of the time. We call "e" the calibration error at least 95 percent of the time. We call "e" the calibration error
for the measurements. It represents the degree to which the values for the measurements. It represents the degree to which the values
produced by the measurement instrument are repeatable; that is, how produced by the measurement instrument are repeatable; that is, how
closely an actual delay of 30 ms is reported as 30 ms. {Comment: 95 closely an actual delay of 30 ms is reported as 30 ms.
percent was chosen due to reasons discussed in [2], briefly {Comment: 95 percent was chosen due to reasons discussed in [4],
summarized as (1) some confidence level is desirable to be able to briefly summarized as (1) some confidence level is desirable to be
remove outliers, which will be found in measuring any physical able to remove outliers, which will be found in measuring any
property; (2) a particular confidence level should be specified so physical property; (2) a particular confidence level should be
that the results of independent implementations can be compared.} specified so that the results of independent implementations can be
compared.}
From the discussion in the previous two sections, the error in From the discussion in the previous two sections, the error in
measurements could be bounded by determining all the individual measurements could be bounded by determining all the individual
uncertainties, and adding them together to form uncertainties, and adding them together to form
Raisanen,Grotefeld,Morton Informational exp.May 2002 9 Network performance measurement with periodic streams Nov 2001
Esynch(t) + ResMP(Src) + ResMP(Dst) + Hsource + Hdest. Esynch(t) + ResMP(Src) + ResMP(Dst) + Hsource + Hdest.
Raisanen, Grotefeld expires July 2001 [Page 14]
However, reasonable bounds on both the clock-related uncertainty However, reasonable bounds on both the clock-related uncertainty
captured by the first three terms and the host-related uncertainty captured by the first three terms and the host-related uncertainty
captured by the last two terms should be possible by careful design captured by the last two terms should be possible by careful design
techniques and calibrating the instruments using a known, isolated, techniques and calibrating the instruments using a known, isolated,
network in a lab. network in a lab.
For example, the clock-related uncertainties are greatly reduced For example, the clock-related uncertainties are greatly reduced
through the use of a GPS time source. The sum of Esynch(t) + through the use of a GPS time source. The sum of Esynch(t) +
ResMP(Src) + ResMP(Dst) is small, and is also bounded for the ResMP(Src) + ResMP(Dst) is small, and is also bounded for the
duration of the measurement because of the global time source. duration of the measurement because of the global time source.
The host-related uncertainties, Hsource + Hdest, could be bounded by The host-related uncertainties, Hsource + Hdest, could be bounded by
connecting two instruments back-to-back with a high-speed serial link connecting two instruments back-to-back with a high-speed serial
or isolated LAN segment. In this case, repeated measurements are link or isolated LAN segment. In this case, repeated measurements
measuring the same one-way delay. are measuring the same one-way delay.
If the test packets are small, such a network connection has a If the test packets are small, such a network connection has a
minimal delay that may be approximated by zero. The measured delay minimal delay that may be approximated by zero. The measured delay
therefore contains only systematic and random error in the therefore contains only systematic and random error in the
instrumentation. The "average value" of repeated measurements is the instrumentation. The "average value" of repeated measurements is
systematic error, and the variation is the random error. the systematic error, and the variation is the random error.
One way to compute the systematic error, and the random error to a One way to compute the systematic error, and the random error to a
95% confidence is to repeat the experiment many times - at least 95% confidence is to repeat the experiment many times - at least
hundreds of tests. The systematic error would then be the median. hundreds of tests. The systematic error would then be the median.
The random error could then be found by removing the systematic error The random error could then be found by removing the systematic
from the measured values. The 95% confidence interval would be the error from the measured values. The 95% confidence interval would
range from the 2.5th percentile to the 97.5th percentile of these be the range from the 2.5th percentile to the 97.5th percentile of
deviations from the true value. The calibration error "e" could then these deviations from the true value. The calibration error "e"
be taken to be the largest absolute value of these two numbers, plus could then be taken to be the largest absolute value of these two
the clock-related uncertainty. {Comment: as described, this bound is numbers, plus the clock-related uncertainty. {Comment: as
relatively loose since the uncertainties are added, and the absolute described, this bound is relatively loose since the uncertainties
value of the largest deviation is used. As long as the resulting are added, and the absolute value of the largest deviation is used.
value is not a significant fraction of the measured values, it is a As long as the resulting value is not a significant fraction of the
reasonable bound. If the resulting value is a significant fraction measured values, it is a reasonable bound. If the resulting value
of the measured values, then more exact methods will be needed to is a significant fraction of the measured values, then more exact
compute the calibration error.} methods will be needed to compute the calibration error.}
Note that random error is a function of measurement load. For Note that random error is a function of measurement load. For
example, if many paths will be measured by one instrument, this might example, if many paths will be measured by one instrument, this
increase interrupts, process scheduling, and disk I/O (for example, might increase interrupts, process scheduling, and disk I/O (for
recording the measurements), all of which may increase the random example, recording the measurements), all of which may increase the
error in measured singletons. Therefore, in addition to minimal load random error in measured singletons. Therefore, in addition to
measurements to find the systematic error, calibration measurements minimal load measurements to find the systematic error, calibration
should be performed with the same measurement load that the measurements should be performed with the same measurement load that
instruments will see in the field. the instruments will see in the field.
Raisanen, Grotefeld expires July 2001 [Page 15]
We wish to reiterate that this statistical treatment refers to the We wish to reiterate that this statistical treatment refers to the
calibration of the instrument; it is used to "calibrate the meter calibration of the instrument; it is used to "calibrate the meter
stick" and say how well the meter stick reflects reality. stick" and say how well the meter stick reflects reality.
Raisanen,Grotefeld,Morton Informational exp.May 2002 10 Network performance measurement with periodic streams Nov 2001
4.7 Reporting the metric 5.6.4 Errors in incT
The nominal interval between packets, incT, can vary during either
The calibration and context in which the metric is measured MUST be active or passive measurements. In passive measurement, packet
carefully considered, and SHOULD always be reported along with metric headers may include a timestamp applied prior to most of the
results. We now present five items to consider: the Type-P of test protocol stack, and the actual sending time may vary due to
packets, the threshold of delay equivalent to loss, error processor scheduling. For example, H.323 systems are required to
have packets ready for the network stack within 5 ms of their ideal
time. There may be additional variation from the network between the
Src and the MP(Src). Active measurement systems may encounter
similar errors, but to a lesser extent. These errors must be
accounted for in some types of analysis.
5.7 Reporting
The calibration and context in which the method is used MUST be
carefully considered, and SHOULD always be reported along with
metric results. We next present five items to consider: the Type-P
of test packets, the threshold of delay equivalent to loss, error
calibration, the path traversed by the test packets, and background calibration, the path traversed by the test packets, and background
conditions at Src, Dst, and the intervening networks during a sample. conditions at Src, Dst, and the intervening networks during a
This list is not exhaustive; any additional information that could be sample. This list is not exhaustive; any additional information that
useful in interpreting applications of the metrics should also be could be useful in interpreting applications of the metrics should
reported. also be reported.
5.7.1. Type-P
4.7.1. Type-P As noted in the Framework document [3], the value of a metric may
As noted in the Framework document [1], the value of the metric may
depend on the type of IP packets used to make the measurement, or depend on the type of IP packets used to make the measurement, or
"type-P". The value of Type-P-One-way-Periodic-Delay could change "type-P". The value of Type-P-One-way-Periodic-Delay could change
if the protocol (UDP or TCP), port number, size, or arrangement for if the protocol (UDP or TCP), port number, size, or arrangement for
special treatment (e.g., IP precedence or RSVP) changes. The exact special treatment (e.g., IP precedence or RSVP) changes. The exact
Type-P used to make the measurements MUST be accurately reported. Type-P used to make the measurements MUST be reported.
5.7.2. Threshold for delay equivalent to loss
4.7.2. Threshold for delay equivalent to loss
In addition, the threshold for delay equivalent to loss (or In addition, the threshold for delay equivalent to loss (or
methodology to determine this threshold) MUST be reported. methodology to determine this threshold) MUST be reported.
5.7.3. Calibration results
4.7.3. Calibration results
+ If the systematic error can be determined, it SHOULD be removed + If the systematic error can be determined, it SHOULD be removed
from the measured values. from the measured values.
+ You SHOULD also report the calibration error, e, such that the + You SHOULD also report the calibration error, e, such that the
true value is the reported value plus or minus e, with 95% true value is the reported value plus or minus e, with 95%
confidence (see the last section.) confidence (see the last section.)
+ If possible, the conditions under which a test packet with finite + If possible, the conditions under which a test packet with finite
delay is reported as lost due to resource exhaustion on the delay is reported as lost due to resource exhaustion on the
measurement instrument SHOULD be reported. measurement instrument SHOULD be reported.
5.7.4. Path
Raisanen, Grotefeld expires July 2001 [Page 16] Raisanen,Grotefeld,Morton Informational exp.May 2002 11 Network performance measurement with periodic streams Nov 2001
4.7.4. Path
The path traversed by the packets SHOULD be reported, if possible. The path traversed by the packets SHOULD be reported, if possible.
In general it is impractical to know the precise path a given packet In general it is impractical to know the precise path a given packet
takes through the network. The precise path may be known for takes through the network. The precise path may be known for
certain Type-P packets on short or stable paths. If Type-P includes certain Type-P packets on short or stable paths. If Type-P includes
the record route (or loose-source route) option in the IP header, the record route (or loose-source route) option in the IP header,
and the path is short enough, and all routers* on the path support and the path is short enough, and all routers on the path support
record (or loose-source) route, then the path will be precisely record (or loose-source) route, then the path will be precisely
recorded. recorded.
This may be impractical because the route must be short enough, many
This may be impractical because the route must be short enough, routers do not support (or are not configured for) record route, and
many routers do not support (or are not configured for) record route, use of this feature would often artificially worsen the performance
and use of this feature would often artificially worsen the observed by removing the packet from common-case processing.
performance observed by removing the packet from common-case However, partial information is still valuable context. For example,
processing. However, partial information is still valuable context. if a host can choose between two links (and hence two separate
For example, if a host can choose between two links* (and hence two routes from Src to Dst), then the initial link used is valuable
separate routes from Src to Dst), then the initial link used is context. {Comment: For example, with one commercial setup, a Src on
valuable context. {Comment: For example, with Merit's NetNow setup, one NAP can reach a Dst on another NAP by either of several
a Src on one NAP can reach a Dst on another NAP by either of several
different backbone networks.} different backbone networks.}
6. Additional discussion on periodic sampling
4.7.5 Background conditions Fig.1 illustrates measurements on multiple protocol levels that are
relevant to this memo. The user's focus is on transport quality
In many cases, the results of a sample may be influenced by conditions evaluation from application point of view. However, to properly
at Src, Dst, and/or any intervening networks. Some things that may separate the quality contribution of the operating system and codec
affect the results of a sample include: traffic levels and/or bursts on packet voice, for example, it is beneficial to be able to measure
during the sample, link and/or host failures, etc. Information about quality at IP level [6]. Link layer monitoring provides a way of
the background conditions may only be available by non-Internet means accounting for link layer characteristics such as bit error rates.
(e.g. phone calls, television) and may only become available days after ---------------
samples are taken. | application |
---------------
4.8 Single sample vs. a "sample of samples" | transport | <--
---------------
Because this metric represents a periodic stream as one sample, there | network | <--
may be value in running multiple tests using this metric to collect ---------------
a "sample of samples". For example, it may be more appropriate to | link | <--
test 1,000 two-minute VoIP calls rather than a single 2,000 minute ---------------
VoIP call. When considering collection of a sample of samples, issues | physical |
like the interval between samples (e.g. Poisson vs. periodic, time of ---------------
day/day of week), composition of samples (e.g. equal (Tf-T0 duration, Fig. 1: Different possibilities for performing measurements: a
different packet sizes), and network considerations (e.g. run different protocol view. Above, "application" refers to all layers above L4
samples over different intervening link-host combinations) should be and is not used in the OSI sense.
taken into account. For items like the interval between samples, In general, the results of measurements may be influenced by
the pattern of use of the application being measured should be individual application requirements/responses related to the
considered. following issues:
Raisanen,Grotefeld,Morton Informational exp.May 2002 12 Network performance measurement with periodic streams Nov 2001
Raisanen, Grotefeld expires July 2001 [Page 17] + Lost packets: Applications may have varying tolerance to lost
packets. Another consideration is the distribution of lost packets
4.9 Statistics based on Type-P-One-way-Delay-Periodic-Stream (i.e. random or bursty).
+ Long delays: Many applications will consider packets delayed
4.9.1 Statistics calculable from one sample longer than a certain value to be equivalent to lost packets
(i.e. real time applications).
As a metric based on a sample representative of certain + Duplicate packets: Some applications may be perturbed if
applications, some general purpose statistics (e.g. median and duplicate packets are received.
percentile) may be less applicable than ways to characterize the + Reordering: Some applications may be perturbed if packets arrive
range of delay values recorded during the sample metrics. out of sequence. This may be in addition to the possibility of
exceeding the "long" delay threshold as a result of being out of
Example, a sample metric generates 100 packets as measured at MP(Src) sequence.
with the following measurements at MP(Dst) + Corrupt packet header: Most applications will probably treat a
packet with a corrupt header as equivalent to a lost packet. +
Corrupt packet payload: Some applications (e.g. digital voice
codecs) may accept corrupt packet payload. In some cases, the
packet payload may contain application specific forward error
correction (FEC) that can compensate for some level of
corruption.
+ Spurious packet: Dst may receive spurious packets (i.e. packets
that are not sent by the Src as part of the metric). Many
applications may be perturbed by spurious packets.
Depending, e.g., on the observed protocol level, some issues listed
above may be indistinguishable from others by the application, it
may be important to preserve the distinction for the operators of
Src, Dst, and/or the intermediate network(s).
6.1 Measurement applications
This sampling method provides a way to perform measurements
irrespective of the possible QoS mechanisms utilized in the IP
network. As an example, for a QoS mechanism without hard guarantees,
measurements may be used to ascertain that the "best" class gets the
service that has been promised for the traffic class in question.
Moreover, an operator could study the quality of a cheap, low-
guarantee service implemented using possible slack bandwidth in
other classes. Such measurements could be made either in studying
the feasibility of a new service, or on a regular basis.
IP delivery service measurements have been discussed within the
International Telecommunications Union (ITU). A framework for IP
service level measurements (with references to the framework for IP
performance [3]) that is intended to be suitable for service
planning has been approved as I.380 [7]. ITU-T Recommendation I.380
covers abstract definitions of performance metrics. This memo
describes a method that is useful both for service planning and end-
user testing purposes, in both active and passive measurements.
Delay measurements can be one-way [3,4], paired one-way, or round-
trip [8]. Accordingly, the measurements may be performed either with
Raisanen,Grotefeld,Morton Informational exp.May 2002 13 Network performance measurement with periodic streams Nov 2001
synchronized or unsynchronized Src/Dst host clocks. Different
possibilities are listed below.
The reference measurement setup for all measurement types is shown
in Fig. 2.
----------------< IP >--------------------
| | | |
------- ------- -------- --------
| Src | | MP | | MP | | Dst |
------- |(Src)| |(Dst) | --------
------- --------
Fig. 2: Example measurement setup.
An example of the use of the method is a setup with a source host
(Src), a destination host (Dst), and corresponding measurement
points (MP(Src) and MP(Dst)) as shown in Figure 2. Separate
equipment for measurement points may be used if having Src and/or
Dst conduct the measurement may significantly affect the delay
performance to be measured. MP(Src)should be placed/measured close
to the egress point of packets from Src. MP(Dst) should be
placed/measure close to the ingress point of packets for Dst.
"Close" is defined as a distance sufficiently small so that
application-level performance characteristics measured (such as
delay) can be expected to follow the corresponding performance
characteristic between Src and Dst to an adequate accuracy. Basic
principle here is that measurement results between MP(Src) and
MP(Dst) should be the same as for a measurement between Src and Dst,
within the general error margin target of the measurement (e.g., < 1
ms; number of lost packets is the same). If this is not possible,
the difference between MP-MP measurement and Src-Dst measurement
should preferably be systematic.
The test setup just described fulfills two important criteria: 1)
Test is made with realistic stream metrics, emulating - for example
- a full-duplex Voice over IP (VoIP) call. 2) Either one-way or
round-trip characteristics may be obtained.
It is also possible to have intermediate measurement points between
MP(Src) and MP(Dst), but that is beyond the scope of this document.
6.1.1 One way measurement
In the interests of specifying metrics that are as generally usable
as possible, application-level measurements based on one-way delays
are used in the example metrics. The implication of application-
level measurement for bi-directional applications such as
interactive multimedia conferencing is discussed below.
Performing a single one-way measurement only yields information on
network behavior in one direction. Moreover, the stream at the
Raisanen,Grotefeld,Morton Informational exp.May 2002 14 Network performance measurement with periodic streams Nov 2001
network transport level does not emulate accurately a full-duplex
multimedia connection.
6.1.2 Paired one way measurement
Paired one way delay refers to two multimedia streams: Src to Dst
and Dst to Src for the same Src and Dst. By way of example, for some
applications, the delay performance of each one way path is more
important than the round trip delay. This is the case for delay-
limited signals such as VoIP. Possible reasons for the difference
between one-way delays is different routing of streams from Src to
Dst vs. Dst to Src.
For example, a paired one way measurement may show that Src to Dst
has an average delay of 30ms while Dst to Src has an average delay
of 120ms. To a round trip delay measurement, this example would look
like an average of 150ms delay. Without the knowledge of the
asymmetry, we might miss a problem that the application at either
end may have with delays averaging more than 100ms.
Moreover, paired one way delay measurement emulates a full-duplex
VoIP call more accurately than a single one-way measurement only.
6.1.3 Round trip measurement
From the point of view of periodic multimedia streams, round-trip
measurements have two advantages: they avoid the need of host clock
synchronization and they allow for a simulation of full-duplex
communication. The former aspect means that a measurement is easily
performed, since no special equipment or NTP setup is needed. The
latter property means that measurement streams are transmitted in
both directions. Thus, the measurement provides information on
quality of service as experienced by two-way applications.
The downsides of round-trip measurement are the need for more
bandwidth than an one-way test and more complex accounting of packet
loss. Moreover, the stream that is returning towards the original
sender may be more bursty than the one on the first "leg" of the
round-trip journey. The last issue, however, means in practice that
returning stream may experience worse QoS than the out-going one,
and the performance estimates thus obtained are pessimistic ones.
The possibility of asymmetric routing and queuing must be taken into
account during analysis of the results.
Note that with suitable arrangements, round-trip measurements may be
performed using paired one way measurements.
6.2 Statistics calculable from one sample
Raisanen,Grotefeld,Morton Informational exp.May 2002 15 Network performance measurement with periodic streams Nov 2001
Some statistics may be particularly relevant to applications
simulated by periodic streams, such as the range of delay values
recorded during the sample.
For example, a sample metric generates 100 packets at MP(Src) with
the following measurements at MP(Dst):
+ 80 packets received with delay [i] <= 20 ms + 80 packets received with delay [i] <= 20 ms
+ 8 packets received with delay [i] > 20 ms + 8 packets received with delay [i] > 20 ms
+ 5 packets received with corrupt packet headers + 5 packets received with corrupt packet headers
+ 4 packets from MP(Src) with no matching packet recorded + 4 packets from MP(Src) with no matching packet recorded
at MP(Dst) (effectively lost) at MP(Dst) (effectively lost)
+ 3 packets received with corrupt packet payload and + 3 packets received with corrupt packet payload and delay [i] <=
and delay [i] <= 20 ms 20 ms
+ 2 packets that duplicate one of the 80 packets received + 2 packets that duplicate one of the 80 packets received
correctly in the first line correctly as indicated in the first item
For this example, packets are considered acceptable if they are For this example, packets are considered acceptable if they are
received with less than or equal to 20ms delays and without corrupt received with less than or equal to 20ms delays and without corrupt
packet headers or packet payload. In this case, the percentage packet headers or packet payload. In this case, the percentage of
of acceptable packets is 80/100 = 80%. acceptable packets is 80/100 = 80%.
For a different application which will accept packets with corrupt For a different application which will accept packets with corrupt
packet payload and no delay bound (so long as the packet is received), packet payload and no delay bound (so long as the packet is
the percentage of acceptable packets is (80+8+3)/100 = 91%. received), the percentage of acceptable packets is (80+8+3)/100 =
91%.
4.9.2 Statistics calculable from multiple samples 6.3 Statistics calculable from multiple samples
There may be value in running multiple tests using this method to
For computing statistics, a "sample of samples" series of collect a "sample of samples". For example, it may be more
measurements may be performed. As discussed in section 4.8, under appropriate to simulate 1,000 two-minute VoIP calls rather than a
these conditions, general purpose statistics (e.g. median, percentile, single 2,000 minute call. When considering collection of multiple
etc.) may be more relevant as a more statistically significant samples, issues like the interval between samples (e.g. minutes,
number of packets are used. hours), composition of samples (e.g. equal Tf-T0 duration, different
packet sizes), and network considerations (e.g. run different
5. Security Considerations samples over different intervening link-host combinations) should be
taken into account. For items like the interval between samples,
5.1 Denial of Service Attacks the usage pattern for the application of interest should be
considered.
This metric generates a periodic stream of packets from one host (Src) When computing statistics for multiple samples, more general
to another host (Dst) through intervening networks. This metric statistics (e.g. median, percentile, etc.) may have relevance with a
could be abused for denial of service attacks directed at Dst and/or larger number of packets.
the intervening network(s). 6.4 Background conditions
In many cases, the results may be influenced by conditions at Src,
Raisanen, Grotefeld expires July 2001 [Page 18] Dst, and/or any intervening networks. Factors that may affect the
results include: traffic levels and/or bursts during the sample,
link and/or host failures, etc. Information about the background
conditions may only be available by external means (e.g. phone
Raisanen,Grotefeld,Morton Informational exp.May 2002 16 Network performance measurement with periodic streams Nov 2001
calls, television) and may only become available days after samples
are taken.
6.5 Considerations related to delay
For interactive multimedia sessions, end-to-end delay is an
important factor. Too large a delay reduces the quality of the
multimedia session as perceived by the participants. One approach
for managing end-to-end delays on an Internet path involving
heterogeneous link layer technologies is to use per-domain delay
quotas (e.g. 50 ms for a particular IP domain). However, this scheme
has clear inefficiencies, and can over-constrain the problem of
achieving some end-to-end delay objective. A more flexible
implementation ought to address issues like possibility of
asymmetric delays on paths, and sensitivity of an application to
delay variations in a given domain. There are several alternatives
as to the delay statistic one ought to use in managing end-to-end
QoS. This question, although very interesting, is not within the
scope of this memo and is not discussed further here.
7. Security Considerations
7.1 Denial of Service Attacks
This metric generates a periodic stream of packets from one host
(Src) to another host (Dst) through intervening networks. This
method could be abused for denial of service attacks directed at Dst
and/or the intervening network(s).
Administrators of Src, Dst, and the intervening network(s) should Administrators of Src, Dst, and the intervening network(s) should
establish bilateral or multi-lateral agreements regarding the timing, establish bilateral or multi-lateral agreements regarding the
size, and frequency of collection of sample metrics. Use of this timing, size, and frequency of collection of sample metrics. Use of
metric in excess the terms agreed between the participants may BE this method in excess of the terms agreed between the participants
cause for immediate rejection or discard of packets or other may be cause for immediate rejection or discard of packets or other
escalation procedures defined between the affected parties. escalation procedures defined between the affected parties.
7.2 User data confidentiality
5.2 User data confidentiality Active use of this method generates packets for a sample, rather
than taking samples based on user data, and does not threaten user
This metric generates packets for a sample metric, rather than data confidentiality. Passive measurement must restrict attention to
taking samples based on user data. Thus, this metric does not the headers of interest. Since user payloads may be temporarily
threaten user data confidentiality. stored for length analysis, suitable precautions MUST be taken to
keep this information safe and confidential.
5.3 Interference with the metric 7.3 Interference with the metric
It may be possible to identify that a certain packet or stream of It may be possible to identify that a certain packet or stream of
packets are part of a sample metric. With that knowledge at Dst packets is part of a sample. With that knowledge at Dst and/or the
and/or the intervening networks, it is possible to change the intervening networks, it is possible to change the processing of the
processing of the packets (e.g. increasing or decreasing delay) packets (e.g. increasing or decreasing delay) that may distort the
that may distort the measured performance. It may also be measured performance. It may also be possible to generate
possible to generate additional packets that appear to be part of Raisanen,Grotefeld,Morton Informational exp.May 2002 17 Network performance measurement with periodic streams Nov 2001
the sample metric. These additional packets are likely to perturb additional packets that appear to be part of the sample metric.
the results of the sample measurement. These additional packets are likely to perturb the results of the
sample measurement.
To discourage the kind of interference mentioned above, packet To discourage the kind of interference mentioned above, packet
interference checks, such as cryptographic hash, may be used. interference checks, such as cryptographic hash, may be used.
8. IANA Considerations
6. Acknowledgements Since this method and metric do not define a protocol or well-known
values, there are no IANA considerations in this memo.
The authors wish to thank the chairs of the IPPM WG for comments 9. References
that have made the present draft clearer and more focused. Howard 1 Bradner, S., "The Internet Standards Process -- Revision 3", BCP
Stanislevic and Al Morton ahave presented useful comments and 9, RFC 2026, October 1996.
questions. The authors have also built on the substantial 2 Bradner, S., "Key words for use in RFCs to Indicate Requirement
foundations laid by the authors of the framework for IP Levels", RFC 2119, March 1997.
performance [1]. 3 Paxson, V., Almes, G., Mahdavi, J., and Mathis, M., "Framework
for IP Performance Metrics", RFC 2330, May 1998.
Raisanen, Grotefeld expires July 2001 [Page 19] 4 Almes, G., Kalidindi, S., and Zekauskas, M., "A one-way delay
metric for IPPM", RFC 2679, September 1999.
7. References 5 Demichelis, C., and Chimento, P., "IP Packet Delay Variation
Metric for IPPM", work in progress.
[1] V.Paxson, G.Almes, J.Mahdavi, and M.Mathis: Framework for IP 6 ETSI TIPHON document TS-101329-5 (to be published in July).
Performance Metrics, IETF RFC 2330, May 1998. 7 International Telecommunications Union, "Internet protocol data
[2] G.Almes, S.Kalidindi, and M.Zekauskas: A one-way delay metric communication service _ IP packet transfer and availability
for IPPM, IETF RFC 2679, September 1999. performance parameters", Telecommunications Sector Recommendation
[3] International Telecommunications Union recommendation I.380, I.380, February 1999.
February 1999. 8 Almes, G., Kalidindi, S., and Zekauskas, M., "A round-trip delay
[4] S. Bradner: Key words for use in RFCs to Indicate Requirement metric for IPPM", IETF RFC 2681.
Levels, RFC 2119, March 1997. 10. Acknowledgments
[5] ETSI TIPHON document TS-101329-5 (to be published in July). The authors wish to thank the chairs of the IPPM WG (Matt Zekauskas
[6] G.Almes, S.Kalidindi, and M.Zekauskas: A round-trip delay and Merike Kaeo) for comments that have made the present draft
metric for IPPM, IETF RFC 2681. clearer and more focused. Howard Stanislevic and Will Leland have
also presented useful comments and questions. We also acknowledge
8. Authors' contact information Henk Uijterwaal's continued challenge to develop the motivation for
this method. The authors have built on the substantial foundation
Vilho Raisanen <Vilho.Raisanen@nokia.com> laid by the authors of the framework for IP performance [3].
P.O. Box 300 11. Author's Addresses
Vilho Raisanen
Raisanen,Grotefeld,Morton Informational exp.May 2002 18 Network performance measurement with periodic streams Nov 2001
Nokia Networks Nokia Networks
P.O. Box 300
FIN-00045 Nokia Group FIN-00045 Nokia Group
Finland Finland
Phone +358 9 51121 Phone +358 9 4376 1 Fax. +358 9 4376 6852
Fax. +358 9 4376 8924 <Vilho.Raisanen@nokia.com>
Glenn Grotefeld
Glenn Grotefeld <g.grotefeld@motorola.com>
Motorola, Inc. Motorola, Inc.
1303 E. Algonquin Road 1501 W. Shure Drive, MS 2F1
4th Floor Arlington Heights, IL 60004 USA
Schaumburg, IL 60196 Phone +1 847 435-0730 Fax +1 847 632-6800
USA <g.grotefeld@motorola.com>
Phone +1 847 576-5992 Al Morton
Fax +1 847 538-7455 AT&T Labs
Room D3 - 3C06
EXPIRES July 2001 200 Laurel Ave. South
Middletown, NJ 07748 USA
Phone +1 732 420 1571 Fax +1 732 368 1192
<acmorton@att.com>
Full Copyright Statement
"Copyright (C) The Internet Society (date). All Rights Reserved.
This document and translations of it may be copied and furnished to
others, and derivative works that comment on or otherwise explain it
or assist in its implmentation may be prepared, copied, published
and distributed, in whole or in part, without restriction of any
kind, provided that the above copyright notice and this paragraph
are included on all such copies and derivative works. However, this
document itself may not be modified in any way, such as by removing
the copyright notice or references to the Internet Society or other
Internet organizations, except as needed for the purpose of
developing Internet standards in which case the procedures for
copyrights defined in the Internet Standards process must be
followed, or as required to translate it into languages other than
English.
The limited permissions granted above are perpetual and will not be
revoked by the Internet Society or its successors or assigns.
This document and the information contained herein is provided on an
"AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING
BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Raisanen,Grotefeld,Morton Informational exp.May 2002 19
 End of changes. 

This html diff was produced by rfcdiff 1.23, available from http://www.levkowetz.com/ietf/tools/rfcdiff/