INTERNET-DRAFT N. Elkins
Inside Products
R. Hamilton
Chemical Abstracts Service
M. Ackermann
Intended Status: Proposed Standard BCBS Michigan
Expires: March 27, August 10, 2017 February 6, 2017 September 23, 2016
IPv6 Performance and Diagnostic Metrics (PDM) Destination Option
draft-ietf-ippm-6man-pdm-option-06
draft-ietf-ippm-6man-pdm-option-07
Abstract
To assess performance problems, measurements based on optional
sequence numbers and timing may be embedded in each packet. Such
measurements may be interpreted in real-time or after the fact. An
implementation of the existing IPv6 Destination Options extension
header, the Performance and Diagnostic Metrics (PDM) Destination
Options extension header as well as the field limits, calculations,
and usage of the PDM in measurement are included in this document.
Status of this Memo
This Internet-Draft is submitted to IETF in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as
Internet-Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/1id-abstracts.html
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html
Copyright and License Notice
Copyright (c) 2016 2017 IETF Trust and the persons identified as the
document authors. All rights reserved.
IETF Trust Legal Provisions of 28-dec-2009, Section 6.b(i), paragraph
3: This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Table of Contents
1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.1 Terminology . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2 End User Quality of Service (QoS) . . . . . . . . . . . . . 4
1.3 Need for a Packet Sequence Number . . . . . . . . . . . . . 5
1.4 Rationale for defined solution . . . . . . . . . . . . . . . 5
1.5 PDM Works in Collaboration with Other Headers . . . . . . . 6
1.6 IPv6 Transition Technologies . . . . . . . . . . . . . . . . 6
2 Measurement Information Derived from PDM . . . . . . . . . . . . 6
2.1 Round-Trip Delay . . . . . . . . . . . . . . . . . . . . . . 7
2.2 Server Delay . . . . . . . . . . . . . . . . . . . . . . . . 7
3 Performance and Diagnostic Metrics Destination Option Layout . . 7
3.1 Destination Options Header . . . . . . . . . . . . . . . . . 7
3.2 Performance and Diagnostic Metrics Destination Option . . . 7
3.3 Header Placement
3.2.1 PDM Layout . . . . . . . . . . . . . . . . . . . . . . 10
3.4 Header Placement Using IPSec ESP Mode . 7
3.2.2 Base Unit for Time Measurement . . . . . . . . . . 11
3.5 Implementation Considerations . . . 10
3.2.3 Considerations of this time-differential
representation . . . . . . . . . . . . 12
3.6 Dynamic Configuration Options . . . . . . . . . 10
3.2.3.1 Limitations with this encoding method . . . . . . 12
3.6 5-tuple Aging . 11
3.2.3.2 Loss of precision induced by timer value
truncation . . . . . . . . . . . . . . . . . . . . . 11
3.3 Header Placement . 12
4 Considerations of Timing Representation . . . . . . . . . . . . 12
4.1 Encoding the Delta-Time Values . . . . . . . . . 12
3.4 Header Placement Using IPSec ESP Mode . . . . . . . 12
4.2 Timer registers are different on different hardware . . . . 13
4.3 Timer Units on Other Systems
3.4.1 Using ESP Transport Mode . . . . . . . . . . . . . . . . 13
4.4 Time Base
3.4.2 Using ESP Tunnel Mode . . . . . . . . . . . . . . . . . 14
3.5 Implementation Considerations . . . . . . . . . . . . . . . 14
4.5 Timer-value scaling
3.6 Dynamic Configuration Options . . . . . . . . . . . . . . . 15
3.6 5-tuple Aging . . . . . 14
4.6 Limitations with this encoding method . . . . . . . . . . . 15
4.7 Lack of precision induced by timer value truncation . . . . 16
5 . . . 15
4 PDM Flow - Simple Client Server . . . . . . . . . . . . . . . . 17
5.1 15
4.1 Step 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
5.2 16
4.2 Step 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
5.3 17
4.3 Step 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
5.4 17
4.4 Step 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
5.5 18
4.5 Step 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
6 19
5 Other Flows . . . . . . . . . . . . . . . . . . . . . . . . . . 22
6.1 20
5.1 PDM Flow - One Way Traffic . . . . . . . . . . . . . . . . . 22
6.2 20
5.2 PDM Flow - Multiple Send Traffic . . . . . . . . . . . . . . 23
6.3 21
5.3 PDM Flow - Multiple Send with Errors . . . . . . . . . . . . 24
7 22
6 Potential Overhead Considerations . . . . . . . . . . . . . . . 26
8 23
7 Security Considerations . . . . . . . . . . . . . . . . . . . . 27
8.1. 24
7.1. SYN Flood and Resource Consumption Attacks . . . . . . . . 27
8.2 24
7.2 Pervasive monitoring . . . . . . . . . . . . . . . . . . . 27
8.3 25
7.3 PDM as a Covert Channel . . . . . . . . . . . . . . . . . . 28
9 25
7.4 Timing Attacks . . . . . . . . . . . . . . . . . . . . . . . 26
8 IANA Considerations . . . . . . . . . . . . . . . . . . . . . . 28
10 26
9 References . . . . . . . . . . . . . . . . . . . . . . . . . . 29
10.1 . 27
9.1 Normative References . . . . . . . . . . . . . . . . . . . 29
10.2 . 27
9.2 Informative References . . . . . . . . . . . . . . . . . . 29
11 Acknowledgments . 27
Appendix A : Timing Considerations . . . . . . . . . . . . . . . . 28
A.1 Time Differential Calculations . . . . . . . 29
Authors' Addresses . . . . . . . . 28
Acknowledgments . . . . . . . . . . . . . . . . 30
1 Background
To assess performance problems, measurements based on optional
sequence numbers and timing may be embedded in each . . . . . . . . . 29
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 29
1 Background
To assess performance problems, measurements based on optional
sequence numbers and timing may be embedded in each packet. Such
measurements may be interpreted in real-time or after the fact.
As defined in RFC2460 [RFC2460], destination options are carried by
the IPv6 Destination Options extension header. Destination options
include optional information that need be examined only by the IPv6
node given as the destination address in the IPv6 header, not by
routers or other "middle boxes". This document specifies a new
destination option, the Performance and Diagnostic Metrics (PDM)
destination option. This document specifies the layout, field
limits, calculations, and usage of the PDM in measurement.
1.1 Terminology
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [RFC2119].
1.2 End User Quality of Service (QoS)
The timing values in the PDM embedded in the packet will be used to
estimate QoS as experienced by an end user device.
For many applications, the key user performance indicator is response
time. When the end user is an individual, he is generally
indifferent to what is happening along the network; what he really
cares about is how long it takes to get a response back. But this is
not just a matter of individuals' personal convenience. In many
cases, rapid response is critical to the business being conducted.
When the end user is a device (e.g. with the Internet of Things),
what matters is the speed with which requested data can be
transferred -- specifically, whether the requested data can be
transferred in time to accomplish the desired actions. This can be
important when the relevant external conditions are subject to rapid
change.
Low, reliable and acceptable responses response times are not just "nice to
have". On many networks, the impact can be financial hardship or can
endanger human life. In some cities, the emergency police contact
system operates over IP, IP; law enforcement uses TCP/IP networks, enforcement, at all levels, use IP
networks; transactions on our stock exchanges are settled using IP
networks. The critical nature of such activities to our daily lives
and financial well-being demand a simple solution to support response
time measurements.
1.3 Need for a Packet Sequence Number
While performing network diagnostics of an end-to-end connection, it
often becomes necessary to isolate the factors along the network path
responsible for problems. Diagnostic data may be collected at
multiple places along the path (if possible), or at the source and
destination. Then, in post-collection processing, the diagnostic
data corresponding to each packet at different observation points
must be matched for proper measurements. A sequence number in each
packet provides sufficient basis for the matching process. If need
be, the timing fields may be used along with the sequence number to
ensure uniqueness.
This method of data collection along the path is of special use to
determine where packet loss or packet corruption is happening.
The packet sequence number needs to be unique in the context of the
session (5-tuple). See section 2 for a definition of 5-tuple.
1.4 Rationale for defined solution
The current IPv6 specification does not provide timing nor a similar
field in the IPv6 main header or in any extension header. So, we
define the IPv6 Performance and Diagnostic Metrics destination option
(PDM).
Advantages include:
1. Real measure of actual transactions.
2. Independence from transport layer protocols.
3. Ability to span organizational boundaries with consistent
instrumentation
4. No time synchronization needed between session partners
5. Ability to handle all transport protocols (TCP, UDP, SCTP, etc)
in a uniform way
The PDM provides the ability to quickly determine quickly if the (latency)
problem is in the network or in the server (application). More
intermediate measurements may be needed if the host or network
discrimination is not sufficient. At the client, TCP/IP stack time
vs. applications application time may still need to be broken out by client
software.
1.5 PDM Works in Collaboration with Other Headers
The purpose of the PDM is not to supplant all the variables present
in all other headers but to provide data which is not available or
very difficult to get. The way PDM would be used is by a technician
(or tool) looking at a packet capture. Within the packet capture,
they would have available to them the layer 2 header, IP header (v6
or v4), TCP, UCP, ICMP, SCTP or other headers. All information
would be looked at together to make sense of the packet flow. The
technician or processing tool could analyze, report or ignore the
data from PDM, as necessary.
For an example of how PDM can help with TCP retransmit problems,
please look at section 8.
1.6 IPv6 Transition Technologies
In the path to full implementation of IPv6, transition technologies
such as translation or tunneling may be employed. The PDM header is
not expected to work in such scenarios. It is likely that an IPv6
packet containing PDM will be dropped if using IPv6 transition
technologies.
2 Measurement Information Derived from PDM
Each packet contains information about the sender and receiver. In IP
protocol, the identifying information is called a "5-tuple".
The 5-tuple consists of:
SADDR : IP address of the sender
SPORT : Port for sender
DADDR : IP address of the destination
DPORT : Port for destination
PROTC : Protocol for upper layer (ex. TCP, UDP, ICMP, etc.)
The PDM contains the following base fields:
PSNTP : Packet Sequence Number This Packet
PSNLR : Packet Sequence Number Last Received
DELTATLR : Delta Time Last Received
DELTATLS : Delta Time Last Sent
Other fields for scaling and storing time base scaling factors are also in the PDM and
will be described in section 3.
This information, combined with the 5-tuple, allows the measurement
of the following metrics:
1. Round-trip delay
2. Server delay
2.1 Round-Trip Delay
Round-trip *Network* delay is the delay for packet transfer from a
source host to a destination host and then back to the source host.
This measurement has been defined, and the advantages and
disadvantages discussed in "A Round-trip Delay Metric for IPPM"
[RFC2681].
2.2 Server Delay
Server delay is the interval between when a packet is received by a
device and the first corresponding packet is sent back in response.
This may be "Server Processing Time". It may also be a delay caused
by acknowledgements. Server processing time includes the time taken
by the combination of the stack and application to return the
response. The stack delay may be related to network performance. If
this aggregate time is seen as a problem, and there is a need to make
a clear distinction between application processing time and stack
delay, including that caused by the network, then more client based
measurements are needed.
3 Performance and Diagnostic Metrics Destination Option Layout
3.1 Destination Options Header
The IPv6 Destination Options Header is used to carry optional
information that needs to be examined only by a packet's destination
node(s). The Destination Options Header is identified by a Next
Header value of 60 in the immediately preceding header and is defined
in RFC2460 [RFC2460]. The IPv6 Performance and Diagnostic Metrics
Destination Option (PDM) is an implementation of the Destination
Options Header. The PDM does not require time synchronization.
3.2 Performance and Diagnostic Metrics Destination Option
3.2.1 PDM Layout
The IPv6 Performance and Diagnostic Metrics Destination Option (PDM)
contains the following fields:
TIMEBASE : Base timer unit
SCALEDTLR: Scale for Delta Time Last Received
SCALEDTLS: Scale for Delta Time Last Sent
PSNTP : Packet Sequence Number This Packet
PSNLR : Packet Sequence Number Last Received
DELTATLR : Delta Time Last Received
DELTATLS : Delta Time Last Sent
The PDM destination option is encoded in type-length-value (TLV)
format as follows:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Option Type | Option Length |TB |ScaleDTLR | ScaleDTLR | ScaleDTLS |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| PSN This Packet | PSN Last Received |
|-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Delta Time Last Received | Delta Time Last Sent |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Option Type
TBD = 0xXX (TBD) [To be assigned by IANA] [RFC2780]
Option Length
8-bit unsigned integer. Length of the option, in octets, excluding
the Option Type and Option Length fields. This field MUST be set to
16.
Time Base
2-bit binary value. It will indicate the lowest granularity possible
for this device. That is, for a value of 00 in the Time Base field,
a value of 1 in the DELTA fields indicates 1 millisecond.
This field is being included so that a device may choose the
granularity which most suits its timer ticks. That is, so that it
does not have to do more work than needed to convert values required
for the PDM.
The possible values of Time Base are as follows:
00 - milliseconds
01 - microseconds
10 - nanoseconds
11 - picoseconds
Scale Delta Time Last Received (SCALEDTLR)
7-bit signed
8-bit unsigned integer. This is the scaling value for the Delta Time
Last Received (DELTATLR) field. The possible values are from -128 to
+127. 0-255.
See Section 4 for further discussion on Timing Considerations and
formatting of the scaling values.
Scale Delta Time Last Sent (SCALEDTLS)
7-bit
8-bit signed integer. This is the scaling value for the Delta Time
Last Sent (DELTATLS) field. The possible values are from -128 0 to
+127. 255.
Packet Sequence Number This Packet (PSNTP)
16-bit unsigned integer. This field will wrap. It is intended for
use while analyzing packet traces.
Initialized at a random number and incremented monotonically for each
packet of the session flow of the 5-tuple. The 5-tuple consists of
the source and destination IP addresses, the source and destination
ports, and the upper layer protocol (ex. TCP, ICMP, etc). The
random number initialization is intended to make it harder to spoof
and insert such packets.
Operating systems MUST implement a separate packet sequence number
counter per 5-tuple. Operating systems MUST NOT implement a single
counter for all connections.
Packet Sequence Number Last Received (PSNLR)
16-bit unsigned integer. This is the PSNTP of the packet last
received on the 5-tuple.
Delta Time Last Received (DELTATLR)
A 16-bit unsigned integer field. The value is set according to the
scale in SCALEDTLR.
Delta Time Last Received = (Send time packet 2 - Receive time packet
1)
Delta Time Last Sent (DELTATLS)
A 16-bit unsigned integer field. The value is set according to the
scale in SCALEDTLS.
Delta Time Last Sent = (Receive time packet 2 - Send time packet 1)
Option Type
The two highest-order bits of the Option Type field are encoded to
indicate specific processing of the option; for the PDM destination
option, these two bits MUST be set to 00. This indicates the
following processing requirements:
00 - skip over this option and continue processing the header.
RFC2460 [RFC2460] defines other values for the Option Type field.
These MUST NOT be used in the PDM.
In keeping with RFC2460 [RFC2460], the third-highest-order bit of the
Option Type specifies whether or not the Option Data of that option
can change en-route to the packet's final destination.
In the PDM, the value of the third-highest-order bit MUST be 0. The
possible values are as follows:
0 - Option Data does not change en-route
1 - Option Data may change en-route
The three high-order bits described above are to be treated as part
of the Option Type, not independent of the Option Type. That is, a
particular option is identified by a full 8-bit Option Type, not just
the low-order 5 bits of an Option Type.
3.3 Header Placement
The PDM destination option MUST be placed as follows:
- Before
3.2.2 Base Unit for Time Measurement
A time differential is always a whole number in a CPU; it is the upper-layer header or unit
specification -- hours, seconds, nanoseconds -- that determine what
the ESP header. numeric value means. For PDM, we establish the base time unit as
1 attosecond (asec). This follows allows for a common unit and scaling of the order defined in RFC2460 [RFC2460]
IPv6 header
Hop-by-Hop Options header
Destination Options header <--------
Routing header
Fragment header
Authentication header
Encapsulating Security Payload header
Destination Options header <------------
upper-layer header
time differential among all IP stacks and hardware implementations.
Note that there is we are trying to provide the ability to measure both time
differentials that are extremely small, and time differentials in a choice of
DTN-type environment where to place the Destination Options
header. If using ESP mode, please see section 3.4 of delays may be very great. To store a
time differential in just 16 bits that must range in this document
for placement way will
require some scaling of the PDM Destination Options header.
For each IPv6 packet header, the PDM MUST NOT appear more than once.
However, an encapsulated packet MAY contain a separate PDM associated
with each encapsulated IPv6 header.
3.4 Header Placement Using IPSec ESP Mode
IP Encapsulating Security Payload (ESP) time differential value.
One issue is defined the conversion from the native time base in [RFC4303] and
is widely used. Section 3.1.1 of [RFC4303] discusses placement the CPU
hardware of
Destination Options Headers. Below whatever device is the diagram from [RFC4303]
discussing placement. PDM MUST in use to some number of attoseconds.
It might seem this will be placed before an astronomical number, but the ESP header in
order conversion
is straightforward. It involves multiplication by an appropriate
power of 10 to work. If placed before change the ESP header, value into a number of attoseconds. Then,
to scale the PDM header will
flow in value so that it fits into DELTATLR or DELTATLS, the clear over
value is shifted by of a number of bits, retaining the network thus allowing gathering 16 high-order
or most significant bits. The number of
performance and diagnostic data without sacrificing security.
BEFORE APPLYING ESP
---------------------------------------
IPv6 | | ext hdrs | | |
| orig IP hdr |if present| TCP | Data |
---------------------------------------
AFTER APPLYING ESP
---------------------------------------------------------
IPv6 | orig |hop-by-hop,dest*,| |dest| | | ESP | ESP|
|IP hdr|routing,fragment.|ESP|opt*|TCP|Data|Trailer| ICV|
---------------------------------------------------------
|<--- encryption ---->|
|<------ integrity ------>|
* = if present, could be before ESP, after ESP, bits shifted becomes the
scaling factor, stored as SCALEDTLR or both
3.5 Implementation SCALEDTLS, respectively. For a
full description of this process, including examples, please see
Appendix A.
3.2.3 Considerations
The PDM destination options extension header SHOULD be turned on by
each stack on of this time-differential representation
There are a host node. It MAY also few considerations to be turned on only in case taken into account with this
representation of
diagnostics needed for problem resolution.
3.6 Dynamic Configuration Options
If implemented, each operating system MUST have a default
configuration parameter, e.g. diag_header_sys_default_value=yes/no. time differential. The operating system MAY also have a dynamic configuration option to
change the configuration setting as needed.
If the PDM destination options extension header first is used, then it MAY
be turned whether there are
any limitations on for all packets flowing through the host, applied to an
upper-layer protocol (TCP, UDP, SCTP, etc), a local port, maximum or IP
address only. These are at the discretion minimum time differential that can
be expressed using method of a delta value and a scaling factor. The
second is the implementation.
As amount of imprecision introduced by this method.
3.2.3.1 Limitations with all other destination options extension headers, the PDM is
for destination nodes only. As specified above, intermediate devices
MUST neither set nor modify this field.
3.6 5-tuple Aging
Within the operating system, metrics must be kept on a 5-tuple basis. encoding method
The 5-tuple is:
SADDR : IP address DELTATLS and DELTATLR fields store only the 16 most-significant
bits of the sender SPORT : Port for sender DADDR : IP
address of time differential value. Thus the destination DPORT : Port for destination PROTC :
Protocol for upper layer (ex. TCP, UDP, ICMP)
The question comes of when range, excluding the
scaling factor, is from 0 to stop keeping data 65535, or restarting the
numbering for a 5-tuple. For example, maximum of 2**16-1. This
method is further described in the case [TRAM-TCPM].
The actual magnitude of TCP, at some
point, the connection will terminate. Keeping data in control blocks
forever, will have unfortunate consequences for time differential is determined by the operating system.
So,
scaling factor. SCALEDTLR and SCALEDTLS are 8-bit unsigned integers,
so the recommendation is to use a known aging parameter such as Max
Segment Lifetime (MSL) as defined in Transmission Control Protocol
[RFC0793] scaling factor ranges from 0 to reuse or drop the control block. 255. The choice smallest number that
can be represented would have a value of aging
parameter is left up to 1 in the implementation.
4 Considerations delta field and a
value of Timing Representation
4.1 Encoding 0 in the Delta-Time Values associated scale field. This section makes reference to and expands on is the document "Encoding
of Time Intervals representation
for 1 attosecond. Clearly this allows PDM to measure extremely small
time differentials.
On the TCP Timestamp Option" [TRAM-TCPM].
4.2 Timer registers are different on different hardware
One other end of the problems with timestamp recording scale, the maximum delta value is 65535, or
FFFF in hexadecimal. If the variety maximum scale value of
hardware that generates 255 is used, the
time value differential represented is 65535*2**255, which is over 3*10**55
years, essentially, forever. So there appears to be used. Different CPUs
track no real
limitation to the time in registers differential that can be represented.
3.2.3.2 Loss of different sizes, precision induced by timer value truncation
As PDM specifies the DELTATLR and DELTATLS values as 16-bit unsigned
integers, any time the most-
frequently-iterated bit could precision is greater than those 16 bits, there
will be truncation of the first on the left or the first
on trailing bits, with an accompanying loss of
precision in the right. In order to generate some examples here it is necessary
to indicate value.
Any time differential value smaller than 65536 asec can be stored
exactly in DELTATLR or DELTATLS, because the type representation of timer register being used.
As described this
value requires at most 16 bits.
Since the time differential values in PDM are measured in
attoseconds, the "IBM z/Architecture Principles range of Operation"
[IBM-POPS], values that would be truncated to the Time-Of-Day clock in a zSeries CPU same
encoded value is 2**(Scale)-1 asec.
For example, the smallest time differential that would be truncated
to fit into a 104-bit
register, where bit 51 delta field is incremented approximately every
microsecond:
1
0 1 2 3 4 5 6 0
+--------+---------+---------+---------+---------+---------+--+...+
| | | | | |* | |
+--------+---------+---------+---------+---------+---------+--+...+
^ ^ ^
0 51 =
1 usec 103
To represent these values concisely a hexadecimal representation will
be used, where each digit represents 4 binary bits. Thus:
0000 0000 0000 0001 = 1 timer unit (2**-12 usec, or about 244 psec)
0000 0000 0000 1000 = 1 microsecond
0000 0000 003E 65536 asec
This value would be encoded as a delta value of 8000 = 1 millisecond
0000 0000 F424 0000 = (hexadecimal)
with a scaling factor of 1. The value
1 second
0000 0039 3870 0000 = 1 minute 0000 0D69 3A40 0000 = 1 hour 0001 41DD 7600 0000 = 1 day
Note that only the first 64 bits of the register are commonly
represented, 65537 asec
would also be encoded as that represents a count delta value of timer units on this
hardware. Commonly the first 52 bits are all that are displayed, as
that represents 8000 with a count scaling factor
of microseconds.
4.3 Timer Units on Other Systems 1. This encoding method works actually is the largest value that would be truncated to
that same with other hardware clock
formats. The method uses a microsecond as encoded value. When the basic scale value and allows
for large time differentials.
4.4 Time Base
This specification allows for is 1, the fact that different CPU TOD clocks
use different binary points. For some clocks, a value of 1 could
indicate range
is calculated as 2**1 - 1, or 1 microsecond, whereas other clocks could use asec, which you can see is the value 1
difference between these minimum and maximum values.
The scaling factor is defined as the number of low-order bits
truncated to
indicate 1 millisecond. In reduce the former case, size of the binary digits resulting value so it fits into a
16-bit delta field. If, for example, you had to truncate 12 bits, the
right
loss of that binary point measure 2**(-n) microseconds, and in precision would depend on the
latter case, 2**(-n) milliseconds. bits you truncated. The Time Base allows us range
of these values would be
0000 0000 0000 = 0 asec
to ensure we have a common reference, at
1111 1111 1111 = 4095 asec
So the
very least, common knowledge minimum loss of what precision would be 0 asec, where the binary point is for delta
value exactly represents the
transmitted values.
We define a base unit for time differential, and the time. This is a 2-bit binary value
indicating maximum loss
of precision would be 4095 asec. As stated above, the lowest granularity possible for this device. That is,
for a value scaling factor
of 00 in 12 means the Time Base field, a value maximum loss of 1 in precision is 2**12-1 asec, or 4095
asec.
Compare this loss of precision to the DELTA
fields indicates 1 millisecond. actual time differential. The possible
range of actual time differential values that would incur this loss
of Time Base are as follows:
00 - milliseconds
01 - microseconds
10 - nanoseconds
11 - picoseconds
Time base precision is not necessarily equivalent from
1000 0000 0000 0000 0000 0000 0000 = 2**27 asec or 134217728 asec
to length of one timer tick.
That is, on many, if not all, systems,
1111 1111 1111 1111 1111 1111 1111 = 2**28-1 asec or 268435455 asec
Granted, these are small values, but the timer tick point is, any value between
these two values will not
be in complete units of nanoseconds, milliseconds, etc. For example,
on an IBM zSeries machine, one timer tick (or clock unit) is 2 to the
-12th microseconds.
Therefore, some amount of conversion may be needed to approximate
Time Base units.
4.5 Timer-value scaling
As discussed in [TRAM-TCPM] we define storing not an entire time-
interval value, but just the most significant bits of that value,
along with have a scaling factor to indicate the magnitude of the time-
interval value. In our case, we will use the high-order 16 bits. The
scaling value will be the number maximum loss of bits in the timer register to the
right precision of the 16th significant bit. That is, if the timer register
contains this binary value:
1110100011010100101001010001000000000000
<-16 bits -><-24 bits ->
then, the values stored would be 1110 1000 1101 0100 in binary (E8D4
hexadecimal) 4095 asec,
or about 0.00305% for the time value first value, as encoded, and 24 at most
0.001526% for the second. These maximum-loss percentages are
consistent for all scaling value. values.
3.3 Header Placement
The PDM destination option follows the order defined in RFC2460
[RFC2460].
IPv6 header
Hop-by-Hop Options header
Destination Options header <--------
Routing header
Fragment header
Authentication header
Encapsulating Security Payload header
Destination Options header <------------
upper-layer header
Note that the displayed value there is the binary equivalent of 1 second
expressed in picoseconds.
The below table represents a device which has a TimeBase choice of
picoseconds (or 11). The smallest and simplest value where to represent is
1 picosecond; place the time value stored is 1, and Destination Options
header. If using ESP mode, please see section 3.4 of this document
for placement of the scaling value is 0.
Using values from PDM Destination Options header.
For each IPv6 packet header, the table below, we have:
Time value in Encoded Scaling
Delta time picoseconds value decimal
--------------------------------------------------------
1 picosecond 1 1 0
1 nanosecond 3E8 3E8 0
1 microsecond F4240 F424 4
1 millisecond 3B9ACA00 3B9A 16
1 second E8D4A51000 E8D4 24
1 minute 3691D6AFC000 3691 32
1 hour cca2e51310000 CCA2 36
1 day 132f4579c980000 132F 44
365 days 1b5a660ea44b80000 1B5A 52
Sample binary values (high order 16 bits taken)
1 psec 1 0001
1 nsec 3E8 0011 1110 1000
1 usec F4240 1111 0100 0010 0100 0000
1 msec 3B9ACA00 0011 1011 1001 1010 1100 1010 0000 0000
1 sec E8D4A51000 1110 1000 1101 0100 1010 0101 0001 0000 0000 0000
4.6 Limitations PDM MUST NOT appear more than once.
However, an encapsulated packet MAY contain a separate PDM associated
with each encapsulated IPv6 header.
3.4 Header Placement Using IPSec ESP Mode
IPSec Encapsulating Security Payload (ESP) is defined in [RFC4303]
and is widely used. Section 3.1.1 of [RFC4303] discusses placement
of Destination Options Headers.
The placement of PDM is different depending on if ESP is used in
tunnel or transport mode.
3.4.1 Using ESP Transport Mode
Below is the diagram from [RFC4303] discussing placement of headers.
Note that Destination Options MAY be placed before or after ESP or
both. If using PDM in ESP transport mode, PDM MUST be placed after
the ESP header so as not to leak information.
BEFORE APPLYING ESP
---------------------------------------
IPv6 | | ext hdrs | | |
| orig IP hdr |if present| TCP | Data |
---------------------------------------
AFTER APPLYING ESP
---------------------------------------------------------
IPv6 | orig |hop-by-hop,dest*,| |dest| | | ESP | ESP|
|IP hdr|routing,fragment.|ESP|opt*|TCP|Data|Trailer| ICV|
---------------------------------------------------------
|<--- encryption ---->|
|<------ integrity ------>|
* = if present, could be before ESP, after ESP, or both
3.4.2 Using ESP Tunnel Mode
Below is the diagram from [RFC4303] discussing placement of headers.
Note that Destination Options MAY be placed before or after ESP or
both in both the outer set of IP headers and the inner set of IP
headers.
In ESP tunnel mode, PDM MAY be placed before or after the ESP header
or both.
BEFORE APPLYING ESP
---------------------------------------
IPv6 | | ext hdrs | | |
| orig IP hdr |if present| TCP | Data |
---------------------------------------
AFTER APPLYING ESP
------------------------------------------------------------
IPv6 | new* |new ext | | orig*|orig ext | | | ESP | ESP|
|IP hdr| hdrs* |ESP|IP hdr| hdrs * |TCP|Data|Trailer| ICV|
------------------------------------------------------------
|<--------- encryption ---------->|
|<------------ integrity ------------>|
* = if present, construction of outer IP hdr/extensions and
modification of inner IP hdr/extensions is discussed in
the Security Architecture document.
As a completely new IP packet will be made, it means that PDM
information for that packet does not contain any information from the
inner packet, i.e. the PDM information will NOT be based on the
transport layer (TCP, UDP, etc) ports etc in the inner header, but
will be specific to the ESP flow.
If PDM information for the inner packet is desired, the original host
sending the inner packet needs to put PDM header in the tunneled
packet, and then the PDM information will be specific for that
stream.
3.5 Implementation Considerations
The PDM destination options extension header MUST be explicitly
turned on by each stack on a host node by administrative action. The
default value of PDM is off.
PDM MUST NOT be turned on merely if a packet is received with a PDM
header. The received packet could be spoofed by another device.
3.6 Dynamic Configuration Options
If implemented, each operating system MUST have a default
configuration parameter, e.g. diag_header_sys_default_value=yes/no.
The operating system MAY also have a dynamic configuration option to
change the configuration setting as needed.
If the PDM destination options extension header is used, then it MAY
be turned on for all packets flowing through the host, applied to an
upper-layer protocol (TCP, UDP, SCTP, etc), a local port, or IP
address only. These are at the discretion of the implementation.
As with all other destination options extension headers, the PDM is
for destination nodes only. As specified above, intermediate devices
MUST neither set nor modify this encoding method
According field.
3.6 5-tuple Aging
Within the operating system, metrics must be kept on a 5-tuple basis.
The 5-tuple is:
SADDR : IP address of the sender SPORT : Port for sender DADDR : IP
address of the destination DPORT : Port for destination PROTC :
Protocol for upper layer (ex. TCP, UDP, ICMP)
The question comes of when to stop keeping data or restarting the
numbering for a 5-tuple. For example, in the case of TCP, at some
point, the connection will terminate. Keeping data in control blocks
forever, will have unfortunate consequences for the operating system.
So, the recommendation is to use a known aging parameter such as Max
Segment Lifetime (MSL) as defined in Transmission Control Protocol
[RFC0793] to reuse or drop the control block. The choice of aging
parameter is left up to the implementation.
4 PDM Flow - Simple Client Server
Following is a sample simple flow for the PDM with one packet sent
from Host A and one packet received by Host B. The PDM does not
require time synchronization between Host A and Host B. The
calculations to derive meaningful metrics for network diagnostics are
shown below each packet sent or received.
Each packet, in addition to the specification in [TRAM-TCPM] PDM contains information on the size of one such
time-interval field is limited to this 11-bit value
sender and 5-bit
unsigned scale so that they fit into receiver. As discussed before, a 16 bit space. With 5-tuple consists of:
SADDR : IP address of the sender
SPORT : Port for sender
DADDR : IP address of the destination
DPORT : Port for destination
PROTC : Protocol for upper layer (ex. TCP, UDP, ICMP)
It should be understood that
limitation, the maximum value packet identification information is
in each packet. We will not repeat that could be stored in 16 bits is:
11-bit value Scale
============= ======
1111 1111 111 1 1111
or an encoded value of 3FF and a scale value each of 31. This value, in
microseconds, corresponds the following
steps.
4.1 Step 1
Packet 1 is sent from Host A to any Host B. The time differential between:
|<Count of zeroes for Host A is set
initially to 10:00AM.
The time and packet sequence number are saved by the sender
internally. The packet sequence number and delta times are sent in
the packet.
Packet 1
+----------+ +----------+
| | | |
| Host | ----------> | Host |
| A | | B |
| | | |
+----------+ +----------+
PDM Contents:
PSNTP : Packet Sequence Number This Packet: 25
PSNLR : Packet Sequence Number Last Received: -
DELTATLR : Delta Time Last Received: -
SCALEDTLR: Scale value>|
11 1111 1111 1000 0000 0000 0000 0000 0000 0000 0000 (binary)
3 F F 8 0 0 0 0 0 of Delta Time Last Received: 0
DELTATLS : Delta Time Last Sent: -
SCALEDTLS: Scale of Delta Time Last Sent: 0 (hexadecimal)
and
11 1111 1111 1111 1111 1111 1111 1111 1111 1111 1111 (binary)
3 F F F F F F F F F F (hexadecimal)
The latter time value, 3FFFFFFFFFF, converts to 50 days, 21 hours 40
minutes and 46.511103 seconds. A time differential 1 microsecond
longer won't fit into 16 bits using this encoding scheme.
Thus, PDM separates
Internally, within the scaling value from sender, Host A, it must keep:
Packet Sequence Number of the interval value, giving
a full 16 bits for last packet sent: 25
Time the interval (DELTA) values and introducing a
signed scaling value that can range from -128 to +127.
4.7 Lack of precision induced by timer value truncation
As PDM specifies last packet was sent: 10:00:00
Note, the DELTA values as 16-bit unsigned values, any initial PSNTP from Host A starts at a random number. In
this case, 25. The time
the precision in these examples is greater than those 16 bits, there will be truncation
of the trailing bits, with an obvious loss precision shown in seconds for
the value.
Using microseconds as the Time Base, the range sake of values that would
be truncated to the same encoded value simplicity.
4.2 Step 2
Packet 1 is 2**(Scale)-1 microseconds.
The smallest received at Host B. Its time differential that would be truncated is
1 0000 0000 0000 0000 = 65536 usec
The value
1 0000 0000 0000 0001 = 65537 usec
would be truncated set to one hour later
than Host A. In this case, 11:00AM
Internally, within the same encoded value, which receiver, Host B, it must note:
Packet Sequence Number of the last packet received: 25
Time the last packet was received : 11:00:03
Note, this timestamp is 8000 in hex Host B time. It has nothing whatsoever to
do with a Scale value Host A time. The Packet Sequence Number of 1. When the Scale value is 1, the value range
is calculated as 2**1 - 1, or 1 microsecond, last packet
received will become PSNLR which you can see is the
difference between these minimum and maximum values.
With that will be sent out in mind, let's look at that table of DELTA time values
again, where the Precision is the range from packet sent
by Host B in the smallest value
corresponding to this encoded value next step. The time last received will be used to
calculate the largest:
Time DELTALR value to be sent out in Encoded
Delta time microseconds value Scale Precision
1 microsecond 1 1 0 0:00.000000
1 millisecond 38E 38E 0 0:00.000000
1 second F4240 F424 4 0:00.000015
1 minute 3938700 3938 12 0:00.004095
1 hour D693A400 D693 16 0:00.065535
1 day 141DD76000 141D 24 0:16.777215
Maximum value 3FFFFFFFFFF FFFF 36 19:05:19.476735
This maximum DELTA value is nearly 52,125 days. At the greatest
value, you can be assured of accuracy within about 19 hours. More
simply, response times packet sent by Host
B in the range of a few seconds can be encoded
with next step.
4.3 Step 3
Packet 2 is sent by Host B to Host A. Note, the initial packet
sequence number (PSNTP) from Host B starts at a loss of precision no greater than random number. In
this case, 12. Before sending the packet, Host B does a tenth calculation
of a millisecond.
Most importantly, response times shorter than 65.5 milliseconds deltas. Since Host B knows when it is sending the packet, and it
knows when it received the previous packet, it can
be encoded with no loss of precision.
5 PDM Flow do the following
calculation:
Sending time : packet 2 - Simple Client Server
Following is a sample simple flow for receive time : packet 1
We will call the PDM with one result of this calculation: Delta Time Last Received
(DELTATLR)
That is:
Delta Time Last Received = (Sending time: packet sent
from Host A and one 2 - receive time:
packet received by Host B. The PDM does not
require 1)
Note, both sending time synchronization between Host A and receive time are saved internally in Host
B. The
calculations to derive meaningful metrics for network diagnostics are
shown below each packet sent or received.
Each packet, They do not travel in addition to the PDM contains information on packet. Only the
sender and receiver. As discussed before, a 5-tuple consists of:
SADDR : IP address of Delta is in the sender
SPORT : Port for sender
DADDR : IP address
packet.
Assume that within Host B is the following:
Packet Sequence Number of the destination
DPORT : Port for destination
PROTC : Protocol for upper layer (ex. TCP, UDP, ICMP)
It should be understood that last packet received: 25
Time the last packet was received: 11:00:03
Packet Sequence Number of this packet: 12
Time this packet identification information is
in each packet. being sent: 11:00:07
We will not repeat that can now calculate a delta value to be sent out in each of the following
steps.
5.1 Step 1
Packet 1 packet.
DELTATLR becomes:
4 seconds = 11:00:07 - 11:00:03 = 3782DACE9D900000 asec
This is sent from Host A to Host B. the derived metric: Server Delay. The time for Host A is set
initially to 10:00AM.
The and scaling
factor must be converted; in this case, the time differential is
DE0B, and packet sequence number are saved by the sender
internally. The scaling factor is 2E, or 46 in decimal. Then, these
values, along with the packet sequence number and delta times are numbers will be sent in
the packet. to Host A
as follows:
Packet 1 2
+----------+ +----------+
| | | |
| Host | ----------> <---------- | Host |
| A | | B |
| | | |
+----------+ +----------+
PDM Contents:
PSNTP : Packet Sequence Number This Packet: 25 12
PSNLR : Packet Sequence Number Last Received: - 25
DELTATLR : Delta Time Last Received: - DE0B (4 seconds)
SCALEDTLR: Scale of Delta Time Last Received: 0 2E (46 decimal)
DELTATLS : Delta Time Last Sent: -
SCALEDTLS: Scale of Delta Time Last Sent: 0
TIMEBASE : Granularity of Time: 00 (Milliseconds)
Internally, within
The metric left to be calculated is the sender, Round-Trip Delay. This will
be calculated by Host A, A when it receives Packet 2.
4.4 Step 4
Packet 2 is received at Host A. Remember, its time is set to one
hour earlier than Host B. Internally, it must keep: note:
Packet Sequence Number of the last packet sent: 25 received: 12
Time the last packet was sent: 10:00:00 received : 10:00:12
Note, the initial PSNTP from this timestamp is in Host A starts time. It has nothing whatsoever to
do with Host B time.
So, now, Host A can calculate total end-to-end time. That is:
End-to-End Time = Time Last Received - Time Last Sent
For example, packet 25 was sent by Host A at 10:00:00. Packet 12 was
received by Host A at 10:00:12 so:
End-to-End time = 10:00:12 - 10:00:00 or 12 (Server and Network RT
delay combined). This time may also be called total Overall Round-
trip time (which includes Network RTT and Host Response Time).
This derived metric we will call Delta Time Last Sent (DELTATLS)
We can now also calculate round trip delay. The formula is:
Round trip delay = (Delta Time Last Sent - Delta Time Last Received)
Or:
Round trip delay = 12 - 4 or 8
Now, the only problem is that at this point all metrics are in Host A
only and not exposed in a random number. In packet. To do that, we need a third packet.
Note: this case, 25. The time in these examples simple example assumes one send and one receive. That
is shown in seconds done only for
the sake purposes of simplicity.
5.2 Step 2
Packet 1 is received at Host B. Its time is set to one hour later
than Host A. In this case, 11:00AM
Internally, within explaining the receiver, Host B, it must note:
Packet Sequence Number function of the last packet received: 25
Time PDM. In
cases where there are multiple packets returned, one would take the last packet was received : 11:00:03
Note, this timestamp is
time in Host B time. It has nothing whatsoever to
do with Host A time. The Packet Sequence Number of the last packet
received will become PSNLR which will be sent out in the packet sent
by Host B in the next step. sequence. The time last received will be used to
calculate the DELTALR value to be sent out in calculations of such
timings and intelligent processing is the packet sent by Host
B in function of post-processing
of the next step.
5.3 data.
4.5 Step 3 5
Packet 2 3 is sent by Host B to Host A. Note, the initial packet
sequence number (PSNTP) from Host B starts at a random number. In
this case, 12. Before sending the packet, A to Host B does a calculation
of deltas. Since B.
+----------+ +----------+
| | | |
| Host B knows when it is sending the packet, and it
knows when it received the previous packet, it can do the following
calculation:
Sending time | ----------> | Host |
| A | | B |
| | | |
+----------+ +----------+
PDM Contents:
PSNTP : packet 2 - receive time Packet Sequence Number This Packet: 26
PSNLR : packet 1
We will call the result Packet Sequence Number Last Received: 12
DELTATLR : Delta Time Last Received: 0
SCALEDTLS: Scale of this calculation: Delta Time Last Received
(DELTATLR)
That is: 0
DELTATLS : Delta Time Last Received = (Sending time: Sent: A688 (scaled value)
SCALEDTLR: Scale of Delta Time Last Received: 30 (48 decimal)
To calculate Two-Way Delay, any packet 2 - receive time: capture device may look at
these packets and do what is necessary.
5 Other Flows
What we have discussed so far is a simple flow with one packet 1)
Note, both sending time sent
and receive one returned. Let's look at how PDM may be useful in other
types of flows.
5.1 PDM Flow - One Way Traffic
The flow on a particular session may not be a send-receive paradigm.
Let us consider some other situations. In the case of a one-way
flow, one might see the following:
Note: The time are saved internally is expressed in Host
B. They generic units for simplicity. That
is, these values do not travel in the packet. Only represent a number of attoseconds,
microseconds or any other real units of time.
Packet Sender PSN PSN Delta Time Delta Time
This Packet Last Recvd Last Recvd Last Sent
=====================================================================
1 Server 1 0 0 0
2 Server 2 0 0 5
3 Server 3 0 0 12
4 Server 4 0 0 20
What does this mean and how is it useful?
In a one-way flow, only the Delta Time Last Sent will be seen as
used. Recall, Delta Time Last Sent is in the
packet.
Assume that within Host B is difference between the following:
Packet Sequence Number
send of the last one packet received: 25
Time from a device and the last packet was received: 11:00:03
Packet Sequence Number of this packet: 12
Time this packet next. This is being sent: 11:00:07
We can now calculate a delta value to be sent out in measure of
throughput for the packet.
DELTATLR becomes:
4 seconds = 11:00:07 sender - 11:00:03
This according to the sender's point of view.
That is, it is a measure of how fast is the derived metric: Server Delay. The application itself (with
stack time and scaling
factor must be calculated. Then, included) able to send packets.
How might this value, along with be useful? If one is having a performance issue at
the client and sees that packet
sequence numbers will be 2, for example, is sent after 5 time
units from the server but takes 10 times that long to Host A as follows:
Packet 2
+----------+ +----------+
| | | |
| Host | <---------- | Host |
| A | | B |
| | | |
+----------+ +----------+
PDM Contents:
PSNTP : Packet Sequence Number This Packet: 12
PSNLR : Packet Sequence Number Last Received: 25
DELTATLR : Delta Time Last Received: FA0 (4 seconds)
SCALEDTLR: Scale of Delta Time Last Received: 00
DELTATLS : Delta Time Last Sent: -
SCALEDTLS: Scale of Delta Time Last Sent: 0
TIMEBASE : Granularity of Time: 00 (Milliseconds)
The metric left to be calculated is arrive at the
destination, then one may safely conclude that there are delays in
the Round-Trip Delay. This will path other than at the server which may be calculated causing the delivery
issue of that packet. Such delays may include the network links,
middle-boxes, etc.
Now, true one-way traffic is quite rare. What people often mean by Host A when it receives Packet 2.
5.4 Step 4
Packet 2
"one-way" traffic is received at Host A. Remember, its time an application such as FTP where a group of
packets (for example, a TCP window size worth) is set to one
hour earlier than Host B. Internally, it must note:
Packet Sequence Number sent, then the
sender waits for acknowledgment. This type of flow would actually
fall into the last packet received: 12
Time "multiple-send" traffic model.
5.2 PDM Flow - Multiple Send Traffic
Assume that two packets are sent for each ACK from the last packet was received : 10:00:12
Note, this timestamp is in Host A time. It has nothing whatsoever to server. For
example, a TCP flow will do with Host B time.
So, now, Host A can calculate total end-to-end time. That is:
End-to-End this, per RFC1122 [RFC1122] Section-
4.2.3.
Packet Sender PSN PSN Delta Time = Delta Time
This Packet Last Received - Time Recvd Last Recvd Last Sent
For example, packet 25 was sent by Host A at 10:00:00. Packet 12 was
received by Host A at 10:00:12 so:
End-to-End time = 10:00:12 - 10:00:00 or 12 (Server and Network RT
delay combined). This time may also
=====================================================================
1 Server 1 0 0 0
2 Server 2 0 0 5
3 Client 1 2 20 0
4 Server 3 1 10 15
How might this be called total Overall Round-
trip time (which includes Network RTT and Host Response Time).
This derived metric we will call used?
Notice that in packet 3, the client has a value of Delta Time Last Sent (DELTATLS)
We can now also calculate round trip delay. The formula is:
Round trip delay = (Delta
received of 20. Recall that Delta Time Last Sent Received is the Send
time of packet 3 - receive time of packet 2. So, what does one know
now? In this case, Delta Time Last Received)
Or:
Round trip delay = 12 - 4 or 8
Now, Received is the only problem processing time
for the Client to send the next packet.
How to interpret this depends on what is that at actually being sent.
Remember, PDM is not being used in isolation, but to supplement the
fields found in other headers. Let's take some examples:
1. Client is sending a standalone TCP ACK. One would find this point all metrics are by
looking at the payload length in Host A
only the IPv6 header and not exposed the TCP
Acknowledgement field in the TCP header. So, in a packet. To do that, we need a third packet.
Note: this simple example assumes one case, the
client is taking 20 units to send and one receive. That back the ACK. This may or may not
be interesting.
2. Client is done only for purposes of explaining sending data with the packet. Again, one would find
this by looking at the payload length in the IPv6 header and the TCP
Acknowledgement field in the function of TCP header. So, in this case, the PDM. In
cases where
client is taking 20 units to send back data. This may represent
"User Think Time". Again, this may or may not be interesting, in
isolation. But, if there are multiple packets returned, one would take is a performance problem receiving data at
the
time server, then taken in the last conjunction with RTT or other packet in timing
information, this information may be quite interesting.
Of course, one also needs to look at the sequence. The calculations PSN Last Received field to
make sure of such
timings and intelligent processing is the function of post-processing interpretation of the this data.
5.5 Step 5
Packet 3 is sent from Host A That is, to Host B.
+----------+ +----------+
| | | |
| Host | ----------> | Host |
| A | | B |
| | | |
+----------+ +----------+
PDM Contents:
PSNTP : Packet Sequence Number This Packet: 26
PSNLR : Packet Sequence Number Last Received: 12
DELTATLR : Delta Time Last Received: 0
SCALEDTLS: Scale of make
sure that the Delta Time Last Received 0
DELTATLS : Delta Time Last Sent: 2EE0 (12 seconds)
SCALEDTLR: Scale corresponds to the packet of Delta Time Last Received: 00
TIMEBASE : Granularity
interest.
The benefits of Time: 00 (Milliseconds)
To calculate Two-Way Delay, any packet capture device may look at
these packets and do what is necessary.
6 Other Flows
What PDM are that we have discussed so far is such information available in a simple flow
uniform manner for all applications and all protocols without
extensive changes required to applications.
5.3 PDM Flow - Multiple Send with one packet sent
and one returned. Let's Errors
Let us now look at a case of how PDM may be useful able to help in other
types of flows.
6.1 PDM Flow - One Way Traffic
The flow on a particular session may not be a send-receive paradigm.
Let us consider some other situations. In the case of a one-way
flow, one might see
TCP retransmission and add to the following:
Packet information that is sent in the TCP
header.
Assume that three packets are sent with each send from the server.
From the server, this is what is seen.
Pkt Sender PSN PSN Delta Time Delta Time TCP Data
This Packet Last Recvd Last Recvd Last Sent Pkt LastRecvd LastRecvd LastSent SEQ Bytes
=====================================================================
1 Server 1 0 0 0 123 100
2 Server 2 0 0 5 223 100
3 Server 3 0 0 12
4 Server 4 0 0 20
What 5 333 100
The client, however, does this mean and how is it useful?
In a one-way flow, only the Delta Time Last Sent will be seen as
used. Recall, Delta Time Last Sent is the difference between the
send of one packet from a device and the next. This is a measure of
throughput for the sender - according to the sender's point of view.
That is, it is a measure of how fast is not receive all the application itself (with
stack time included) able to send packets.
How might From the
client, this be useful? If one is having a performance issue at
the client and sees that packet 2, for example, what is seen for the packets sent after 5
microseconds from the server but takes server.
Pkt Sender PSN PSN Delta Time Delta Time TCP Data
This Pkt LastRecvd LastRecvd LastSent SEQ Bytes
=====================================================================
1 Server 1 0 0 0 123 100
2 Server 3 minutes to arrive at the
destination, then one may safely conclude 0 0 5 333 100
Let's assume that there are delays in
the path other than at the server which may be causing now retransmits the delivery
issue of that packet. Such delays may include the network links,
middle-boxes, etc.
Now, true one-way traffic is quite rare. What people often mean by
"one-way" traffic is an application such as FTP where
(Obviously, a group of duplicate acknowledgment sequence for fast retransmit
or a retransmit timeout would occur. To illustrate the point, these
packets (for example, are being left out.)
So, then if a TCP window size worth) retransmission is sent, done, then from the
sender waits client, this
is what is seen for acknowledgment. This type of flow would actually
fall into the "multiple-send" traffic model.
6.2 PDM Flow - Multiple Send Traffic
Assume that two packets are sent for each ACK from the server. For
example, a
Pkt Sender PSN PSN Delta Time Delta Time TCP flow will do this, per RFC1122 [RFC1122] Section-
4.2.3. Data
This Pkt LastRecvd LastRecvd LastSent SEQ Bytes
=====================================================================
1 Server 4 0 0 30 223 100
The server has resent the old packet 2 with TCP sequence number of
223. The retransmitted packet now has a PSN This Packet value of 4.
The Delta Last Sent is 30 - the time between sending the packet with
PSN of 3 and this current packet.
Let's say that packet 4 is lost again. Then, after some amount of
time (RTO) then the packet with TCP sequence number of 223 is resent.
From the client, this is what is seen for the packets sent from the
server.
Pkt Sender PSN PSN Delta Time Delta Time TCP Data
This Packet Last Recvd Last Recvd Last Sent Pkt LastRecvd LastRecvd LastSent SEQ Bytes
=====================================================================
1 Server 1 0 0 0
2 Server 2 0 0 5
3 Client 1 2 20 0
4 Server 3 1 10 15
How might 0 60 223 100
If now, this be used?
Notice that in packet 3, arrives at the client destination, one has a value of Delta Time Last
received of 20. Recall very good
idea that Delta Time Last Received packets exist which are being sent from the server as
retransmissions and not arriving at the client. This is because the Send
time of packet 3 - receive time
PSN of the resent packet 2. So, what does one know
now? In from the server is 5 rather than 4. If we
had used TCP sequence number alone, we would never have seen this case, Delta Time Last Received
situation. The TCP sequence number in all situations is 223.
This situation would be experienced by the processing time
for the Client to send user of the next packet.
How to interpret this depends on what is actually being sent.
Remember, PDM is not application
(the human being used in isolation, but actually sitting somewhere) as a "hangs" or long
delay between packets. On large networks, to supplement diagnose problems such
as these where packets are lost somewhere on the
fields found in other headers. Let's network, one has to
take some examples:
1. Client is sending a standalone TCP ACK. One would multiple traces to find this by
looking out exactly where.
The first thing is to start with doing a trace at the payload length in the IPv6 header client and the TCP
Acknowledgement field in the TCP header.
server. So, in this case, we can see if the server sent a particular packet and
the client is taking 20 units to send back received it. If the ACK. This may or may client did not
be interesting.
2. Client is sending data with the packet. Again, one would find
this by looking receive it, then we
start tracking back to trace points at the payload length in router right after the IPv6 header
server and the TCP
Acknowledgement field in router right before the TCP header. So, in this case, client. Did they get these
packets which the
client is taking 20 units to send back data. server has sent? This may represent
"User Think Time". Again, this may or may not be interesting, in
isolation. But, if there is a performance problem receiving data at time consuming
activity.
With PDM, we can speed up the server, then taken in conjunction with RTT or other packet timing
information, this information diagnostic time because we may be quite interesting.
Of course, one also needs able
to look at use only the PSN Last Received field to
make sure of trace taken at the interpretation of this data. That is, client to make
sure that see what the Delta Last Received corresponds server is
sending.
6 Potential Overhead Considerations
Questions have been posed as to the packet of
interest.
The benefits potential overhead of PDM.
First, PDM are that we have such information available in is entirely optional. That is, a
uniform manner for all applications and all protocols without
extensive changes required site may choose to applications.
6.3
implement PDM Flow - Multiple Send or not as they wish. If they are happy with Errors
Let us now look at a case the costs
of how PDM may vs. the benefits, then the choice should be able to help in theirs.
Below is a case of
TCP retransmission and add to table outlining the information that is sent potential overhead in terms of
additional time to deliver the TCP
header.
Assume that three packets are sent with each send from the server.
From response to the server, this is what is seen.
Pkt Sender PSN PSN Delta Time Delta Time TCP Data
This Pkt LastRecvd LastRecvd LastSent SEQ end user for various
assumed RTTs.
Bytes RTT Bytes Bytes New Overhead
in Packet Per Milli in PDM RTT
=====================================================================
1000 1000 milli 1 Server 1 0 0 0 123 100
2 Server 2 0 0 5 223 16 1016.000 16.000 milli
1000 100
3 Server 3 0 0 5 333 milli 10 16 101.600 1.600 milli
1000 10 milli 100 16 10.160 .160 milli
1000 1 milli 1000 16 1.016 .016 milli
Below are some examples of actual RTTs for packets traversing large
enterprise networks. The client, however, does not receive all the packets. From the
client, this is what first example is seen for the packets sent from the server.
Pkt Sender PSN PSN Delta Time Delta Time TCP Data
This Pkt LastRecvd LastRecvd LastSent SEQ going to
multiple business partners.
Bytes
=====================================================================
1 Server 1 0 0 0 123 100
2 Server 3 0 0 5 333 100
Let's assume RTT Bytes Bytes New Overhead
in Packet Per Milli in PDM RTT
=====================================================================
1000 17 milli 58 16 17.360 .360 milli
The second example is for packets at a large enterprise customer
within a data center. Notice that the server scale is now retransmits in microseconds
rather than milliseconds.
Bytes RTT Bytes Bytes New Overhead
in Packet Per Micro in PDM RTT
=====================================================================
1000 20 micro 50 16 20.320 .320 micro
7 Security Considerations
PDM may introduce some new security weaknesses.
7.1. SYN Flood and Resource Consumption Attacks
PDM needs to calculate the packet.
(Obviously, a duplicate acknowledgment sequence deltas for fast retransmit
or a retransmit timeout would occur. To illustrate time and keep track of the point, these
packets are being left out.)
So, then if
sequence numbers. This means that control blocks must be kept at the
end hosts per 5-tuple. Any time a TCP retransmission control block is done, then from kept, an
attacker can try to mis-use the client, this control blocks such that there is what a
compromise of the end host.
PDM is seen for used only at the packets sent from end hosts and the server.
Pkt Sender PSN PSN Delta Time Delta Time TCP Data
This Pkt LastRecvd LastRecvd LastSent SEQ Bytes
=====================================================================
1 Server 4 0 0 30 223 100
The server has resent control blocks are only
kept at the old packet 2 with TCP sequence number end host and not at routers or middle boxes. Remember,
PDM is an implementation of
223. The retransmitted packet now has a PSN This Packet value the Destination Option extension header.
A "SYN flood" type of 4.
The Delta Last Sent attack succeeds because a TCP SYN packet is 30 -
small but it causes the time between sending end host to start creating a place holder for
the packet with
PSN session such that quite a bit of 3 control block and this current packet.
Let's say that packet 4 other storage
is lost again. Then, after some used. This is an asynchronous type of attack in that a small
amount of
time (RTO) then work by the packet with TCP sequence number attacker creates a large amount of 223 work by the
resource attacked.
For PDM, the amount of data to be kept is resent.
From quite small. That is, the client, this
control block is what quite lightweight. Concerns about SYN Flood and
other type of resource consumption attacks (memory, processing power,
etc) can be alleviated by having a limit on the number of control
block entries.
We recommend that implementation of PDM SHOULD have a limit on the
number of control block entries.
7.2 Pervasive monitoring
Since PDM passes in the clear, a concern arises as to whether the
data can be used to fingerprint the system or somehow obtain
information about the contents of the payload.
Let us discuss fingerprinting of the end host first. It is seen for possible
that seeing the packets sent from pattern of deltas or the
server.
Pkt Sender PSN PSN Delta Time Delta Time TCP Data
This Pkt LastRecvd LastRecvd LastSent SEQ Bytes
=====================================================================
1 Server 5 0 0 60 223 100
If now, this packet arrives at absolute values could give
some information as to the destination, one has speed of the end host - that is, if it is
a very good
idea that packets exist which are being sent from fast system or an older, slow device. This may be useful to
the server as
retransmissions and not arriving at attacker. However, if the client. This is because attacker has access to PDM, the
PSN of
attacker also has access to the resent entire packet from and could make such a
deduction based merely on the server time frames elapsed between packets
WITHOUT PDM.
As far as deducing the content of the payload, it appears to us that
PDM is 5 rather than 4. If we
had used TCP sequence number alone, we would never have seen quite unhelpful in this
situation. The TCP sequence number regard.
7.3 PDM as a Covert Channel
PDM provides a set of fields in all situations the packet which could be used to
leak data. But, there is 223.
This situation no real reason to suspect that PDM would
be experienced by the user chosen rather than another part of the application
(the human being actually sitting somewhere) as a "hangs" payload or long
delay between packets. On large networks, to diagnose problems such
as these where packets are lost somewhere on another
Extension Header.
A firewall or another device could sanity check the network, one has to
take multiple traces fields within the
PDM but randomly assigned sequence numbers and delta times might be
expected to find out exactly where. vary widely. The first thing biggest problem though is how an
attacker would get access to start with doing a trace at the client and PDM in the
server. So, we can see if first place to leak data.
The attacker would have to either compromise the server sent a particular packet and end host or have Man
in the client received it. If Middle (MitM). It is possible that either one could change
the client did not receive it, fields. But, then we
start tracking back to trace points at the router right after other end host would get sequence numbers
and deltas that don't make any sense. Presumably, one is using PDM
and doing packet tracing for diagnostic purposes, so the
server changes
would be obvious. It is conceivable that someone could compromise
an end host and make it start sending packets with PDM without the router right before
knowledge of the client. Did they get these
packets which host. But, again, the server has sent? This bigger problem is a time consuming
activity.
With PDM, we can speed up the diagnostic time because we may be able
to use only
compromise of the trace taken at end host. Once that is done, the client attacker
probably has better ways to see what leak data.
Having said that, an implementation SHOULD stop using PDM if it gets
some number of "nonsensical" sequence numbers.
7.4 Timing Attacks
The fact that PDM can help in the server is
sending.
7 Potential Overhead Considerations
Questions have been posed as separation of node processing time
from network latency brings value to the potential overhead performance monitoring. Yet, it
is this very characteristic of PDM.
First, PDM is entirely optional. That is, a site which may choose be misused to
implement PDM or not as they wish. If they are happy with make
certain new type of timing attacks against protocols and
implementations possible.
Depending on the nature of the cryptographic protocol used, it may be
possible to leak the costs long term credentials of the device. For
example, if an attacker is able to create an attack which causes the
enterprise to turn on PDM vs. to diagnose the benefits, attack, then the choice should be theirs.
Below is a table outlining the potential overhead in terms of
additional attacker
might use PDM during that debugging time to deliver launch a timing attack
against the response to long term keying material used by the end user for various
assumed RTTs.
Bytes RTT Bytes Bytes New Overhead
in Packet Per Milli in cryptographic
protocol.
An implementation may want to be sure that PDM RTT
=====================================================================
1000 1000 milli 1 16 1016.000 16.000 milli
1000 100 milli 10 16 101.600 1.600 milli
1000 10 milli 100 16 10.160 .160 milli
1000 1 milli 1000 16 1.016 .016 milli
Below are is enabled only for
certain ip addresses, or only for some examples ports. Additionally, we
recommend that the implementation SHOULD require an explicit restart
of actual RTTs for packets traversing large
enterprise networks. The first monitoring after a certain timeperiod (for example is for packets going 1 hour), to
multiple business partners.
Bytes RTT Bytes Bytes New Overhead
in Packet Per Milli in PDM RTT
=====================================================================
1000 17 milli 58 16 17.360 .360 milli
The second example is for packets at a large enterprise customer
within a data center. Notice
make sure that the scale is now in microseconds
rather than milliseconds.
Bytes RTT Bytes Bytes New Overhead
in Packet Per Micro in PDM RTT
=====================================================================
1000 20 micro 50 16 20.320 .320 micro
8 Security Considerations PDM does is not introduce any new security weakness.
8.1. SYN Flood and Resource Consumption Attacks
PDM needs to calculate accidently left on after debugging has been
done etc.
Even so, if using PDM, we introduce the deltas for time and keep track concept of the
sequence numbers. This means that control blocks must user "Consent to
be kept at the
end hosts per 5-tuple. Any time Measured" as a control block pre-requisite for using PDM. Consent is kept, an
attacker can try to mis-use the control blocks such common in
enterprises and with some subscription services. So, if with PDM, we
recommend that there is a
compromise of the end host.
PDM is used only at user SHOULD consent to its use.
8 IANA Considerations
This draft requests an Option Type assignment in the end hosts Destination
Options and Hop-by-Hop Options sub-registry of Internet Protocol
Version 6 (IPv6) Parameters [ref to RFCs and URL below].
http://www.iana.org/assignments/ipv6-parameters/ipv6-
parameters.xhtml#ipv6-parameters-2
Hex Value Binary Value Description Reference
act chg rest
-------------------------------------------------------------------
TBD TBD Performance and [This draft]
Diagnostic Metrics
(PDM)
9 References
9.1 Normative References
[RFC0793] Postel, J., "Transmission Control Protocol", STD 7, RFC
793, September 1981.
[RFC1122] Braden, R., "Requirements for Internet Hosts --
Communication Layers", RFC 1122, October 1989.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC2460] Deering, S. and R. Hinden, "Internet Protocol, Version 6
(IPv6) Specification", RFC 2460, December 1998.
[RFC2681] Almes, G., Kalidindi, S., and M. Zekauskas, "A Round-trip
Delay Metric for IPPM", RFC 2681, September 1999.
[RFC2780] Bradner, S. and V. Paxson, "IANA Allocation Guidelines For
Values In the control blocks are only
kept at the end host Internet Protocol and not at routers or middle boxes. Remember,
PDM is an implementation Related Headers", BCP 37, RFC
2780, March 2000.
[RFC4303] Kent, S, "IP Encapsulating Security Payload (ESP)", RFC
4303, December 2005.
9.2 Informative References
[TRAM-TCPM] Trammel, B., "Encoding of Time Intervals for the Destination Option extension header. TCP
Timestamp Option-01", Internet Draft, July 2013. [Work in Progress]
Appendix A "SYN flood" type of attack succeeds because : Timing Considerations
A.1 Time Differential Calculations
The time counter in a TCP SYN packet CPU is
small but it causes the end host to start creating a place holder for
the session such that quite a bit of control block and other storage
is used. This is an asynchronous type of attack in that binary whole number, representing a small
amount
number of work by the attacker creates a large amount milliseconds (msec), microseconds (usec) or even
picoseconds (psec). Representing one of work these values as attoseconds
(asec) means multiplying by the
resource attacked.
For PDM, the amount of data 10 raised to be kept is quite small. That is, the
control block is quite lightweight. Concerns about SYN Flood and
other type some exponent. Refer to this
table of resource consumption attacks (memory, processing power,
etc) can be alleviated by having a limit on the number equalities:
Base value = # of control
block entries.
We recommend that implementation sec = # of PDM SHOULD asec 1000s of asec
--------------- ------------- ------------- -------------
1 second 1 sec 10**18 asec 1000**6 asec
1 millisecond 10**-3 sec 10**15 asec 1000**5 asec
1 microsecond 10**-6 sec 10**12 asec 1000**4 asec
1 nanosecond 10**-9 sec 10**9 asec 1000**3 asec
1 picosecond 10**-12 sec 10**6 asec 1000**2 asec
1 femtosecond 10**-15 sec 10**3 asec 1000**1 asec
For example, if you have a limit on the
number of control block entries.
8.2 Pervasive monitoring
Since PDM passes time differential expressed in
microseconds, since each microsecond is 10**12 asec, you would
multiply your time value by 10**12 to obtain the clear, a concern arises as to whether the
data can be used to fingerprint the system or somehow obtain
information about the contents of the payload.
Let us discuss fingerprinting of the end host first. It is possible
that seeing the pattern number of deltas or the absolute values could give
some information as
attoseconds. If you time differential is expressed in nanoseconds,
you would multiply by 10**9 to get the speed number of the end host - that is, if it attoseconds.
The result is a very fast system or an older, slow device. This may be useful to
the attacker. However, if the attacker has access to PDM, the
attacker also has access binary value that will need to the entire packet and could make such be shortened by a
deduction based merely on the time frames elapsed between packets
WITHOUT PDM.
As far as deducing the content
number of the payload, bits so it appears to us that will fit into the 16-bit PDM DELTA field.
The next step is quite unhelpful to divide by 2 until the value is contained in this regard.
8.3 PDM as a Covert Channel
PDM provides a set just
16 significant bits. The exponent of fields the value in the packet which could be used to
leak data. But, there is no real reason to suspect that PDM would
be chosen rather than another part last column of
of the payload or another
Extension Header.
A firewall or another device could sanity check the fields within table is useful here; the
PDM but randomly assigned sequence numbers and delta times might be
expected to vary widely. The biggest problem though initial scaling factor is how an
attacker would get access that
exponent multiplied by 10. This is the minimum number of low-order
bits to PDM in be shifted-out or discarded. It represents dividing the first place time
value by 1024 raised to leak data. that exponent.
The attacker would have resulting value may still be too large to either compromise fit into 16 bits, but
can be normalized by shifting out more bits (dividing by 2) until the end host or have Man
in
value fits into the Middle (MitM). It 16-bit DELTA field. The number of extra bits
shifted out is possible that either one could change
the fields. But, then added to the other end host would get sequence numbers
and deltas that don't make any sense. Presumably, one is using PDM
and doing packet tracing for diagnostic purposes, so scaling factor. The scaling factor,
the changes
would be obvious. It total number of low-order bits dropped, is conceivable that someone could compromise the SCALEDTL value.
For example: say an end host and make it application has these start sending packets with PDM without and finish timer
values (hexadecimal values, in microseconds):
Finish: 27C849234 usec (02:57:58.997556)
-Start: 27C83F696 usec (02:57:58.957718)
========== ========= ===============
Difference 9B9E usec 00.039838 sec or 39838 usec
To convert this differential value to binary attoseconds, multiply
the
knowledge number of microseconds by 10**12. Divide by 1024**4, or simply
discard 40 bits from the host. But, again, right. The result is 36232, or 8D88 in hex,
with a scaling factor or SCALEDTL value of 40.
For another example, presume the bigger problem time differential is larger, say
32.311072 seconds, which is 32311072 usec. Each microsecond is 10**12
asec, so multiply by 10**12, giving the
compromise hexadecimal value
1C067FCCAE8120000. Using the initial scaling factor of 40, drop the end host. Once
last 10 characters (40 bits) from that string, giving 1C067FC. This
will not fit into a DELTA field, as it is done, 25 bits long. Shifting the attacker
probably has better ways
value to leak data.
Having said that, an implementation SHOULD stop using PDM if it gets
some number of "nonsensical" sequence numbers. the right another 9 IANA Considerations
This draft requests an Option Type assignment bits results in a DELTA value of E033,
with a resulting scaling factor of 49.
When the time differential value is a small number, regardless of the
time unit, the exponent trick given above is not useful in
determining the Destination
Options and Hop-by-Hop Options sub-registry of Internet Protocol
Version 6 (IPv6) Parameters [ref to RFCs and URL below].
http://www.iana.org/assignments/ipv6-parameters/ipv6-
parameters.xhtml#ipv6-parameters-2
Hex Value Binary Value Description Reference
act chg rest
-------------------------------------------------------------------
TBD TBD Performance proper scaling value. For example, if the time
differential is 3 seconds and [This draft]
Diagnostic Metrics
(PDM)
10 References
10.1 Normative References
[RFC0793] Postel, J., "Transmission Control Protocol", STD 7, RFC
793, September 1981.
[RFC1122] Braden, R., "Requirements for Internet Hosts --
Communication Layers", RFC 1122, October 1989.
[RFC2119] Bradner, S., "Key words for use in RFCs you want to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC2460] Deering, S. and R. Hinden, "Internet Protocol, Version 6
(IPv6) Specification", RFC 2460, December 1998.
[RFC2681] Almes, G., Kalidindi, S., and M. Zekauskas, "A Round-trip
Delay Metric for IPPM", RFC 2681, September 1999.
[RFC2780] Bradner, S. and V. Paxson, "IANA Allocation Guidelines For
Values In convert that directly, you
would follow this path:
3 seconds = 3*10**18 asec (decimal)
= 29A2241AF62C0000 asec (hexadecimal)
If you just truncate the Internet Protocol last 60 bits, you end up with a delta value
of 2 and Related Headers", BCP 37, RFC
2780, March 2000.
[RFC4303] Kent, S, "IP Encapsulating Security Payload (ESP)", RFC
4303, December 2005.
10.2 Informative References
[TRAM-TCPM] Trammel, B., "Encoding a scaling factor of Time Intervals for the TCP
Timestamp Option-01", Internet Draft, July 2013. [Work 60, when what you really wanted was a
delta value with more significant digits. The most precision with
which you can store this value in Progress]
[IBM-POPS] IBM Corporation, "IBM z/Architecture Principles of
Operation", SA22-7832, 1990-2012
11 16 bits is A688, with a scaling
factor of 46.
Acknowledgments
The authors would like to thank Keven Haining, Al Morton, Brian
Trammel, David Boyes, Bill Jouris, Richard Scheffenegger, and Rick
Troth for their comments and assistance. We would also like to thank
Tero Kivinen for his detailed and perceptive review.
Authors' Addresses
Nalini Elkins
Inside Products, Inc.
36A Upper Circle
Carmel Valley, CA 93924
United States
Phone: +1 831 659 8360
Email: nalini.elkins@insidethestack.com
http://www.insidethestack.com
Robert M. Hamilton
Chemical Abstracts Service
A Division of the American Chemical Society
2540 Olentangy River Road
Columbus, Ohio 43202
United States
Phone: +1 614 447 3600 x2517
Email: rhamilton@cas.org
http://www.cas.org
Michael S. Ackermann
Blue Cross Blue Shield of Michigan
P.O. Box 2888
Detroit, Michigan 48231
United States
Phone: +1 310 460 4080
Email: mackermann@bcbsm.com
http://www.bcbsm.com