Internet Engineering Task Force Yoshihiro Kikuchi - Toshiba Internet Draft Toshiyuki Nomura - NEC Document:
draft-ietf-avt-rtp-mpeg4-es-00.txtdraft-ietf-avt-rtp-mpeg4-es-01.txt Shigeru Fukunaga - Oki Yoshinori Matsui - Matsushita Hideaki Kimata - NTT February 1,May 31, 2000 RTP payload format for MPEG-4 Audio/Visual streams Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026 . Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet- Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Abstract This document describes RTP payload formats for the carriage of MPEG-4 Audio and Visual streams, and an RTCP format for MPEG-4 upstream messages functionalities. In this specification, MPEG-4 Audio/Visual bitstreams are directly mapped into RTP packets without using MPEG-4 Systems.packets. The RTP header fields usage and the fragmentation rule for MPEG-4 Visual and Audio bitstreams are specified. It also specifies an RTCP packet usage to carry the MPEG-4 upstream messages. In addition, MIME type registrations and SDP usages for the MPEG-4 Audio and Visual streams are defined in this document. 1. Introduction 1.1 Why MPEG-4 Audio/Visual RTP format needed? The RTP payload formats described in this Internet-Draft specify the normativea way onof how MPEG-4 Audio/VisualAudio and Visual streams are fragmented and mapped directly onto RTP packets. No extra header field is used for such functionality as error protection or grouping of streams.H.323 terminals could be the case.an example where such RTP payload formats are used. MPEG-4 Audio/Visual streams are not managed by Object DescriptorsDescriptors of MPEG-4 Systems but H.245, andby H.245. The streams are directly mapped intoonto RTP packets without Sync Layer.using the synchronization functionality of MPEG-4 Systems . The semantics of RTP headers in this casesuch cases need to be clearly defineddefined, including the association with the MPEG-4 Audio/Visual data elements. In addition, it would be beneficial to define the fragmentation rule of RTP packets for MPEG-4 Video streams so as to enhance error resiliency by utilizing the error resilience tools provided inside the MPEG-4 Video stream. However, they havethese items are not been studied until now.covered by other RTP payload format proposals. 1.2 Consideration on theMPEG-4 Visual RTP payload format MPEG-4 Visual is a visual coding standard with many new functionalities: high coding efficiency, high error resiliency, multiple arbitrary shaped object based coding, etc. . It covers a wide range of bitrate from several Kbps to many Mbps. It also covers a wide variety of networks ranging from guarantied withto be almost error-free to mobile networks with high error rate bydue to its error resilience functionalities. A normative way offragmentation ofrule for an MPEG-4 visual bitstream into RTP packets is defined in this Internet draft.document. Since MPEG-4 Visual is used for a wide variety of networks, it is desirable not desiredto apply too much restriction onto the fragmentation. A fragmentation rule like a"a single video packet shall always be mapped on a single RTP packet.packet" may be inappropriate. On the other hand, a careless media unaware fragmentation may cause degradation of the error resiliency and the bandwidth efficiency. The fragmentation rule described in this Internet draftdocument is flexible but to define the minimum rules to preventand guidelines for preventing the meaningless fragmentation and to utilizing the error resilience functionality of e.g. splitting a header into packets.MPEG-4 visual. For video coding media such as H.261 or MPEG-1/2, the additional media specific RTP header works effectively for recovering e.g.recovering. e.g., of a picture header corruptcorrupted by packet losses. However, there are error resilience functionalities inside MPEG-4 Visual to recover corrupt headers. These functionalities can commonly be used on RTP/IP network as well as other networks. (H.223/mobile, MPEG-2/TS, etc.) Therefore, no extra RTP header fields are defined in the MPEG-4 Visual RTP payload format. 1.3 Consideration on the MPEG-4 Audio RTP payload format MPEG-4 Audio is a new kind of audio standard that integrates many different types of audio coding tools. It also supports a mechanism representing synthesized sounds. Low-overhead MPEG-4 Audio Transport Multiplex (LATM) manages the sequence of the compressed or the represented audio data by MPEG-4 Audio tools with relatively small overhead. In audio-only applications, the LATM-based MPEG-4 Audio bitstreams, therefore, are desirable to be directly mapped into the RTP packets without using MPEG-4 Systems. Furthermore, if the payload of a packet is a single audio frame, a packet loss does not impair the decodability of adjacent packets. Therefore, a payload specific header for MPEG-4 Audio is not required as same as one for the other audio coders. 1.4 MPEG-4 Audio/Visual upstream messaging on RTCP packets In some cases,Some particular tools of MPEG-4 Audio/Visual hassupport upstream messaging functionalities. These messages are extremely Audio/Visual specific, since coders directly use these messages for controlling coding parameters. From the point of view of controlling parameters, these messages should be transmitted without delay. ThereforeTherefore, these messages are directly mapped onto some kind of low delay RTCP packets. The use of this type of RTCP packets is limited to the case when the MPEG-4 upstream functionalities in some particular profiles are used (e.g. MPEG-4 Visual Advanced Real Time Simple Profile, NEWPRED tool). 2. Conventions used in this document The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC-2119 . 3. RTP Packetization of MPEG-4 Visual bitstream This section specifies the RTP packetization rule for MPEG-4 Visual content. An MPEG-4 Visual bitstream is mapped directly onto the RTP payload without any addition of extra header fields or removal of any Visual syntax elements. The Combined Configuration/Elementary streams mode is used so that the configuration information is carried in the same RTP port as the elementary stream. (see 6.2.1 "Start codes" of ISO/IEC 14496-2 ) When the short video header mode is used, RTP payload format for H.263 specified in the relevant RFCs or other standards MAY be used. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |V=2|P|X| CC |M| PT | sequence number | RTP +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | timestamp | Header +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | synchronization source (SSRC) identifier | +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ | contributing source (CSRC) identifiers | | .... | +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ | | RTP | MPEG-4 Visual stream (byte aligned) | Payload | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | :...OPTIONAL RTP padding | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 1 - An RTP packet for MPEG-4 Visual stream 3.1 RTP header fields usage for MPEG-4 Visual Payload Type (PT): Distinct payload type should be assigned to specify MPEG-4 Visual RTP payload format. If the dynamic payload type assignment is used, it is specified by some out-of-band means (e.g. H.245, SDP, etc.) that the MPEG-4 Visual payload format is used for the corresponding RTP packet. Extension (X) bit: Defined by the RTP profile used. Sequence Number: Increment by one for each RTP data packet sent. It starts with a random initial value for security reasons. Marker (M) bit: The marker bit is set to one to indicate the last RTP packet (or only RTP packet) of a VOP. Timestamp: The timestamp indicates the composition time, or the presentation time in a no-compositor decoder by adding a constant random offset for security reasons. For a video object plane, it is defined by vop_time_increment (in units of 1/vop_time_increment_resolution seconds) plus the cumulative number of whole seconds specified by module_time_base and time_code of Group_of_VideoObjectPlane() if present. In the case of interlaced video, a VOP consists of lines from two fields and the timestamp indicates the composition time of the first field. If the RTP packet contains only configuration information and/or Group_of_VideoObjectPlane(), the composition time of the subsequent VOP in the coding order is used. If the RTP packet contains only visual_object_sequence_end_code, the composition time of the immediately preceding VOP in the coding order is used. Unless specified by an out-of-band means,means (e.g. SDP parameter or MIME parameter as defined in section 6), the resolution of the timestamp is set to its default (90KHz). SSRC, CC and CSRC fields are used as described in RFC 1889 . 3.2 Fragmentation of MPEG-4 Visual bitstream A fragmented MPEG-4 Visual bitstream is mapped directly onto the RTP payload without any addition of extra header fields or removal of any Visual syntax elements. The Combined Configuration/Elementary streams mode is used. The following rules apply for the fragmentation. (1) The configuration information and Group_of_VideoObjectPlane() SHALL be placed at the beginning of the RTP payload (just after the RTP header) or just after the header of the syntactically upper layer function. (2) If one or more headers exist in the RTP payload, the RTP payload SHALL begin with the header of the syntactically highest function. Note: The visual_object_sequence_end_code is regarded as the lowest function. (3) A header SHALL NOT be split into a plurality of RTP packets. (4) Two or more VOPs SHALL be fragmented into different RTP packets so that one RTP packet consists of the data bytes associated with an unique presentation time (that indicated to the timestamp field in the RTP packet header). (5) A single video packet SHOULD NOT be split into a plurality of RTP packets. The size of a video packet SHOULD be adjusted such that the resulting RTP packet is not larger than the path-MTU. Hear,A video packet MAY be split into a plurality of RTP packets when the size of the video packet is large. Here, header means: - Configuration information (Visual Object Sequence Header, Visual Object Header and VisualVideo Object Layer Header) - visual_object_sequence_end_code - The header of the entry point function for an elementary stream (Group_of_VideoObjectPlane() or the header of VideoObjectPlane(), video_plane_with_short_header(), MeshObject() or FaceObject()) - The video packet header (video_packet_header() excluding next_resync_marker()) - The header of gob_layer() See 6.2.1 "Start codes" of ISO/IEC 14496-2 for the definition of the configuration information and the entry point functions. The video packet starts with the VOP header or the video packet header, followed by motion_shape_texture(), and ends with next_resync_marker() or next_start_code). 3.3 Examples of packetized MPEG-4 Visual bitstream Considering that MPEG-4 Visual is used on a wide variety of networks from several Kbps to many Mbps, from guarantiedguaranteed networks withwhich are almost error- freeerror-free to mobile networks with high error rate, it is desirable not desiredto apply too much restriction onto the fragmentation like a single video packet shall always be mapped on a single RTP packet.fragmentation. On the other hand, a careless media unaware fragmentation will cause degradation of the error resiliency and the bandwidth efficiency. The fragmentation criteria described in 3.2 are flexible but to define the minimum rules to prevent themeaningless fragmentation of e.g. splitting a header into packets.fragmentation. For video coding media such as H.261 or MPEG-1/2, the additional media specific RTP header works effectively for recovering e.g.recovering, e.g., of a picture header corruptcorrupted by packet losses. However, there is an error resilience functionality inside MPEG-4 Visual to recover corrupt headers. This functionality can commonly be used on RTP/IP network as well as other networks. (H.223/mobile, MPEG-2/TS, etc.) Therefore, there is no strong reason to define MPEG-4 Visual specific extra RTP header fields. Figure 2 shows examples of RTP packets generated based on the criteria described in 3.2 (a) is an example of the first RTP packet or the random access point of an MPEG-4 visual bitstream. This RTP packet contains the configuration information. According to the criterion (1), the Visual Object Sequence Header(VS header) is placed at the beginning of the RTP payload, and the Visual Object Header and the VisualVideo Object Layer Header(VO header, VOL header) follow it. Since the fragmentation rule defined in 3.2 guarantiesguarantees that the configuration information, starting with visual_object_sequence_start_code, is always placed at the beginning of the RTP payload, RTP receivers can detect the random access point by checking if the first 32-bit field of the RTP payload is visual_object_sequence_start_code. (b) is ananother example of the RTP packet that contains Group_of_VideoObjectPlane(GOV). Followingcontaining the criterion (1),configuration information. The difference from the GOVexample (a) is placed atthat this RTP packet also contains a video packet in the VOP following the configuration information. Since the length of the configuration information is relatively short (typically several ten bytes), an RTP packet containing only the configuration information may increase the overhead. Therefore, the configuration information and the immediately following GOV and/or (a part of) VOP can be packetized into a single RTP packet like this example. (c) is an example the RTP packet that contains Group_of_VideoObjectPlane(GOV). Following the criterion (1), the GOV is placed at the beginning of the RTP payload. It is a waste of RTP/IP header overhead to generate a RTP packet containing only a GOV whose length is 7 bytes. Therefore, (a part of) the following VOP can be placed in the same RTP packet as shown in (b). (c)(c). (d) is an example thatof the case where one video packet is packetized into one RTP packet. When the packet-loss rate of the underlying network is high, this kind of packetization is recommended. It is strongly recommended to set resync_marker_disable to 0 in the VOL header to enable adjustment of the video packet size. Even when the RTP packet containing the VOP header is discarded by a packet loss, the other RTP packets can be decoded by using the HEC(Header Extension Code) information in the video packet header. No extra RTP header field is necessary. (d)(e) is an example thatof the case where more than one video packets are packetized into one RTP packet. This kind of packetization is effective to save the overhead of RTP/IP headers if the bit-rate of the underlying network is low. However, it will decrease the packet-loss resiliency because multiple video packets are discarded by a single RTP packet loss. The adequate number of video packets in a RTP packet and the RTP packet length depend the packet-loss rate and the bit-rate of the underlying network. Figure 3 shows examples of RTP packets prohibited by the criteria of 3.2. Fragmentation of a header into multiple RTP packets, like (a), will not only increase the overhead of RTP/IP headers but also decrease the error resiliency. Therefore, it is prohibited by the criterion (3). When concatenating more than one video packets into aan RTP packet, VOP header or video_packet_header() shall not be placed in the middle of the RTP payload. The packetization like (b) is not allowed by the criterion (2). This is because of the error resiliency. Comparing this example with Figure 2(c), two video packets are mapped onto two RTP packets in both cases. However, there is a difference between the packet-loss resiliency. When the second RTP packet is lost, both video packets 1 and 2 are lost in the case of Figure 3(b) whereas only video packet 2 is lost in the case of Figure 2(c). AAn RTP packet containing more than one VOPs, like (c), is not allowed. +------+------+------+------+ (a) | RTP | VS | VO | VOL | |header|header|header|header| +------+------+------+------+ +------+-----+------------------++------+------+------+------+------------+ (b) | RTP | VS | VO | VOL |Video Packet| |header|header|header|header| | +------+------+------+------+------------+ +------+-----+------------------+ (c) | RTP | GOV |Video Object Plane| |header| | | +------+-----+------------------+ +------+------+------------+ +------+------+------------+ (c)(d) | RTP | VOP |Video Packet| | RTP | VP |Video Packet| |header|header| (1) | |header|header| (2) | +------+------+------------+ +------+------+------------+ +------+------+------------+------+------------+------+------------+ (d)(e) | RTP | VP |Video Packet| VP |Video Packet| VP |Video Packet| |header|header| (1) |header| (2) |header| (3) | +------+------+------------+------+------------+------+------------+ Figure 2 - Examples of RTP packetized MPEG-4 Visual bitstream +------+-------------+ +------+------------+------------+ (a) | RTP |First half of| | RTP |Last half of|Video Packet| |header| VP header | |header| VP header | | +------+-------------+ +------+------------+------------+ +------+------+----------+ +------+---------+------+------------+ (b) | RTP | VOP |First half| | RTP |Last half| VP |Video Packet| |header|header| of VP(1) | |header| of VP(1)|header| (2) | +------+------+----------+ +------+---------+------+------------+ +------+------+------------------+------+------------------+ (c) | RTP | VOP |Video Object Plane| VOP |Video Object Plane| |header|header| (1) |header| (2) | +------+------+------------------+------+------------------+ Figure 3 - Examples of prohibited RTP packetization for MPEG-4 Visual bitstream 4. RTP Packetization of MPEG-4 Audio bitstream When tools defined in MPEG-4 Systems are not used MPEG-4 Audio stream is formatted by LATM (Low-overhead MPEG-4 Audio Transport Multiplex) format, and then mapped onto RTP packets as described the subsequent section. 4.1 RTP Packet Format The LATM consists of the sequence of audioMuxElements that include one or more audio frames. A complete audioMuxElement or the part of audioMuxElements SHALL be mapped directly onto the RTP payload without removal of any audioMuxElement syntax elements as shown in Figure 4. The first byte of each audioMuxElement SHALL be located at the first payload location of an RTP packet. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |V=2|P|X| CC |M| PT | sequence number |RTP +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | timestamp |Header +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | synchronization source (SSRC) identifier | +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ | contributing source (CSRC) identifiers | | .... | +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ | |RTP : audioMuxElement (byte aligned) :Payload | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | :...OPTIONAL RTP padding | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 4 - An RTP packet for MPEG-4 Audio It is required for the audioMuxElement to indicate the following muxConfigPresent information by an out-of-band means. muxConfigPresent: If this information is set to 1, the audioMuxElement SHALL include an indication bit "useSameStreamMux" and MAY include the configuration information for audio compression "StreamMuxConfig". The useSameStreamMux bit indicates whether the StreamMuxConfig element in the previous frame is applied in the current frame. 4.2 RTP Header Fields Usage Payload Type (PT): Distinct payload type should be assigned to specify MPEG-4 Audio RTP payload format. If the dynamic payload type assignment is used, it is specified by some out-of-band means (e.g. H.245, SDP, etc.) that the MPEG-4 Audio payload format is used for the corresponding RTP packet. Marker (M) bit: The marker bit indicates audioMuxElement boundaries. This bit is set to one to mark the RTP packet contains a complete audioMuxElement or the last fragment of an audioMuxElement. Timestamp: The timestamp indicates the composition time, or the presentation time in a no-compositor decoder. Timestamps are recommended to start at a random value for security reasons. Unless specified by an out-of-band means, the resolution of the timestamp is set to its default (90 kHz). Sequence Number: Increment by one for each RTP packet sent. It starts with a random value for security reasons. SSRC, CC and CSRC fields are used as described in RFC 1889 . 4.3 Fragmentation of MPEG-4 Audio bitstream It is desirable to put one audioMuxElement per RTP packet. The size of an audioMuxElement is tried to be adjusted such that the resulting RTP packet is not larger than the path-MTU. If this is not possible, the audioMuxElement MAY be fragmented across several packets based on the following rules. (1) "payloadMux" which consists of payload elements MAY be fragmented into several RTP packets so that one RTP packet consists of one or more payload elements. A payload element SHOULD NOT be fragmented. (2) If the audioMuxElement includes StreamMuxConfig, StreamMuxConfig SHALL be included into the RTP packet containing the first payload element. 5. RTCP Packetization of MPEG-4 upstream messages This section specifies the usage of particular RTCP packets to carry the upstream messages generated using the MPEG-4 Audio/Visual upstream messaging functionalities, e.g. NEWPRED. RTCP packets specifiedfunctionalities. In the current specification, NEWPRED in the MPEG-4 Visual Advance Real Time Simple (ARTS) Profile is only the tool which uses this sectionRTCP payload specification. This particular RTCP packet SHALL ONLY be used when it is indicated by some out of band means that the corresponding MPEG-4 Visual codec is compliant with the ARTS profile and level indicationit is indicated in the configuration information of MPEG-4the codecs have such functionalities. (e.g. Advanced Real Time Simple Visual Profile) TheMPEG-4 upstream messages are transmitted on particular RTCP packets, like H.261 RTCP control packets . In the casevisual bitstream that the RTP session uses a multicast address, the MPEG-4 upstream message packets are not transmittedNEWPRED tool is enabled (newpred_enable is set to 1). 5.1. Abstract of NEWPRED in the ARTS profile NEWPRED in the ARTS profile is an error resilience tool using the normal RTCP destination transport address. Instead, theseupstream message packets are sent directly via unicastmessages from the decoder to the coder. The destination port number of these upstream message packetsencoder. As the inter-frame coding is always same toused in the port number ofMPEG-4 Visual standard, the normal RTCP address. As a consequence, these upstream message packets may onlyimage degradation by packet loss will be used when no RTP mixers or translators intervene in the path frompropagated to the coderafter several frames. In order to prevent the decoder. If such intermediate systems do intervene,temporal error propagation, the addressreference frames of the coder would no longer be present asinter-frame coding are switched according to the network-level source addressupstream messages in packets received bythe decoder, and in fact, it might notNEWPRED. As the correct frames are used as the reference frame, the error propagation is refreshed. As neither the re-transmission nor the intra refresh are used, the coding efficiency can be possible forkept high. And the decoder to send packets directly toNEWPRED can achieve the coder. Some reliable multicast protocols use similar NACK control packetsfaster error recovery than the intra refresh. There are two types of upstream messages; acknowledged message (NP_ACK) and non-acknowledged message (NP_NACK). NP_ACK and/or NP_NACK messages are transmitted overon the normal multicast distribution channel, however they typically use random delays to prevent a NACK implosion problem.particular RTCP packets in the NEWPRED. The goalselecting methods of such protocols is to provide reliable multicast packet delivery atreference frames are dependent on the expensekind of delay, which is appropriate for applications such as a shared whiteboard. On the other hand,used messages. 5.2. Particular RTCP packets keep low delay The real-time Audio/Visual transmission is more sensitive to delay and does not require full reliability. For Audio/Visual applications it is more effective to send the MPEG-4 upstream message packets as soon as possible, i.e. as soon as a loss is detected, without adding any random delays. 5.1. MPEG-4 Visual upstream message packets definition 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |V=2|P| UMT | PT=RTCP_MP4UM | length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | SSRC | +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ | MPEG-4 Upstream Messages Payload (byte aligned) | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | : padding | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ version (V): 2 bits Identifies5.3. Congestion control In the versioncases of RTP, whichthe demand type of intra refresh or the re-transmission, the amount of bits during the congestion is larger than that in the sameerror free terms. Therefore they may cause some another congestion. While in RTCP packetsthe NEWPRED, as in RTP data packets. padding (P): 1 bit Ifthe padding bitintra-frame coding is set, this RTCP packet contains some additional padding octets at the end which arenot partused, the increased amount of bits is much lower than that of the control information.intra refresh or the re- transmission even in the case of packet loss. Therefore NEWPRED causes less additional burden for the congestion. The last octetamount of the paddingupstream messages is a countdependent on the strategy of how many padding octetsthe selecting methods of reference frames of the encoder and that of the sending upstream messages of the decoder. In order to avoid congestion, the amount of upstream message packets should be ignored.small. In the case severalNEWPRED, the decoder can control the amount of them by not sending some upstream messages; For example, in the case that the NP_NACK messages are mapped onto one RTCP packet, padding should only be required onmainly used to select the reference frames in the encoder, the decoder may not send the NP_ACK messages even if it receives downstream data. On the other hand, in the case that the NP_ACK messages are mainly used in the encoder, the decoder may not send the NP_NACK messages. The amount of the last individual message.upstream message type (UMT): 5 bits Identifiesmessages is at most 5% (normally about 1%) of the typevisual downstream data. Especially the amount of NP_ACK messages is decreased in the MPEG-4case of packet loss. Therefore the NP_ACK message has no additional burden for the congestion. On the other hand, NP_NACK messages corresponding to the lost packets are usually sent after the congestion, because the decoder detects the packet loss after the next downstream packet reaches. Therefore the NP_NACK message has less additional burden for the congestion, too. And to reduce the number of particular RTCP packets, multiple upstream messages. 0: forbidden 1: MPEG-4 Visual NEWPRED 2-63: reservedmessages can be concatenated in the payload of one particular RTCP packet. In this internet-draft, only NEWPREDcase, it is desirable to send these concatenated messages as soon as possible. The particular RTCP transmission interval is according to the interval of the decoding the visual downstream data. Both the receiving interval of the visual RTP packet and the decoding time for each packet data have some jitter for themselves. Therefore the particular RTCP transmission interval has some jitter for itself. It is effective for the congestion control, and there is no need to add any random delays. This means that the size of sending jitter is enough to avoid another congestion only in case of the unicast. 5.4. Limiting to Unicast The NEWPRED can work in multicast only in the case that the number of decoders is small. However in order to avoid the additional congestion, the NEWPRED over RTP/RTCP SHALL NOT be used in multicast. 5.5. Relations with SR and RR The particular low delay RTCP packets for the MPEG-4 upstream messages SHALL be treated as the completely different kind of packets from the normal RTCP packets; such as SR, RR and so on. For example, if the particular RTCP packets would be included in the calculation of RTCP sending interval, the RR packets should be generated in the timing of the particular low delay RTCP packets. In this case, the interval of the RR packets would be smaller than 5 seconds, and the number of the normal RTCP packets is much increased. It is bad for the congestion. Therefore all particular RTCP packets SHALL be ignored to analyze the information in the sender and receiver reports (SR and RR), and only normal RTCP packets are used. Multiple particular RTCP packets can be concatenated without any intervening separators to form a compound RTCP packet. The normal compound RTCP packet SHOULD start with SR or RR packets. However in the case of compound particular RTCP packet, other normal RTCP packets SHALL NOT be included, and only particular RTCP packets SHALL be included in one compound particular RTCP packet. 5.6. MPEG-4 Visual upstream message packets definition 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |V=2|P| UMT | PT=RTCP_MP4U | length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | SSRC | +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ | MPEG-4 Upstream Messages Payload (byte aligned) | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | : padding | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ version (V): 2 bits Identifies the version of RTP, which is the same in RTCP packets as in RTP data packets. padding (P): 1 bit If the padding bit is set, this RTCP packet contains some additional padding octets at the end which are not part of the control information. The last octet of the padding is a count of how many padding octets should be ignored. In the case several upstream messages are mapped onto one RTCP packet, padding should only be required on the last individual message. upstream message type (UMT): 5 bits Identifies the type of the MPEG-4 upstream messages. 0: forbidden 1: MPEG-4 Visual NEWPRED in the ARTS Profile 2-63: reserved In this internet-draft, only the NEWPRED in the ARTS profile is assigned as the candidate of the UMT for the moment. Some other MPEG-4 Audio/Visual applications using the upstream messages may be assigned in the future. packet type (PT): 8 bits The value of the packet type (PT) identifier is the constant RTCP_MP4U (TBD). SSRC: 32 bits SSRC is the synchronization source identifier for the sender of this packet. MPEG-4 Upstream Message Payload: variable The syntax and semantics of the MPEG-4 upstream messages are defined in the ISO/IEC 14496-2/3. All messages are byte aligned. Normally one message is mapped onto one RTCP packet, and several messages with same UMT could be continuously mapped onto one RTCP packet. One message SHALL NOT be fragmented into different RTCP packets. 6. MIME type registration for MPEG-4 Audio/Visual streams The following sections describe the MIME type registrations for the MPEG- 4 Audio/Visual streams. MIME type registration and SDP usage for the MPEG-4 Visual stream are described in sections 6.1 and 6.2, respectively. MIME type registration and SDP usage for the MPEG-4 Audio stream are described in sections 6.3 and 6.4, respectively. (In the following sections, the RFC number "XXXX" represents the RFC number, which should be assigned for this Internet Draft.) 6.1 MIME type registration for MPEG-4 Visual MIME media type name: video MIME subtype name: MP4V Required parameters: none Optional parameters: rate: This parameter is used only for RTP transport. It indicates the resolution of the timestamp field in the RTP header. If this parameter is not specified, the default value of 90000 (90KHz) is used. profile-level-id: A decimal representation of MPEG-4 Visual Profile Level indication value (profile_and_level_indication) defined in Table G-1 of ISO/IEC 14496-2 . mpeg4-newpred-upstream-message: A boolean number to indicate the receiver capability of sending the upstream message of NEWPRED in MPEG-4 video. The upstream messages are delivered on the particular RTCP packets which are described in section 5. This optional exist when and only when the "profile-level-id" is 145, 146, 147 or 148 (Advance Real Time Simple Profile/Level 1, 2, 3 or 4). Example usages for these parameters are show bellow: - MPEG-4 Visual Core Profile/Level 2: Content-type: video/mp4v; profile-level-id=34 - MPEG-4 Visual Advanced Real Time Simple Profile/Level 1, upstream message is used: Content-type: video/mp4v; profile-level-id=145; mpeg4-newpred- upstream-message=1 Published specification: The specification of MPEG-4 Visual stream is presented in ISO/IEC 14469-2. The RTP payload format is described in RFCXXXX. Encoding considerations: A video bitstream must be generated according to the MPEG-4 Visual specification (ISO/IEC 14496-2). The video bitstream is binary data, and must be encoded for non-binary transport; the Base64 encoding is suitable for Email. This type is also defined for transfer via RTP. The RTP packets must be packetized according to the MPEG-4 Visual RTP payload format defined in RFCXXXX. Security considerations: See section 9 of RFCXXXX. Interoperability considerations: MPEG-4 Visual provides a large and rich set of tools for the coding of visual objects. In order to allow effective implementations of the standard, subsets of the MPEG-4 Visual tool sets have been identified, that can be used for specific applications. These subsets, called 'Profiles', limit the tool set a decoder has to implement. For each of these Profiles, one or more Levels have been set, restricting the computational complexity. A Profile@Level combination allows: o a codec builder to implement only the subset of the standard he needs, while maintaining interworking with other MPEG-4 devices built to the same combination, and o checking whether MPEG-4 devices comply with the standard ('conformance testing'). The visual stream SHALL be compliant with the MPEG-4 Visual Profile@Level specified by the parameter "profile-level-id". The interoperability between a sender and a receiver may be achieved by specifying the parameter "profile-level-id" in MIME content, or by exchanging this parameter in the capability exchange procedure. Applications which use this media type: Audio and visual streaming and conferencing tools, Internet messaging and e-mail applications. Additional information: none Person & email address to contact for further information: The authors of RFCXXXX. (See section 9) Intended usage: COMMON Author/Change controller: The authors of RFCXXXX. (See section 9) 6.2 SDP usage of MPEG-4 Visual The MIME media type video/MP4V string is mapped to fields in the Session Description Protocol (SDP), RFC 2327, as follows: o The MIME type (video) goes in SDP "m=" as the media name. o The MIME subtype (MP4V) goes in SDP "a=rtpmap" as the encoding name. o The optional parameter "rate" goes in "a=rtpmap" as clock rate. o The optional parameter "profile-level-id" MAY go in "a=fmtp" line. The optional parameter "mpeg4-newpred-upstream-message" MAY go in "a=fmtp" line, when and only when the "profile-level-id" is 145, 146, 147 or 148(Advance Real Time Simple Profile/Level 1, 2, 3 or 4). The format and syntax of these parameters is the MIME media type string as a semicolon separated list of parameter=value pairs. The followings are some examples of the media representation in SDP: Simple Profile/Level 1, rate=90000(90KHz), "profile-level-id" is present in "a=fmtp" line: m=video 49170/2 RTP/AVP 98 a=rtpmap:98 MP4V/90000 a=fmtp:98 profile-level-id=1 Advance Real Time Simple Profile/Level 1, rate=25(25Hz), "profile-level- id" and " newpred- mpeg4- upstream-message" are present in "a=fmtp" line: m=video 49170/2 RTP/AVP 98 a=rtpmap:98 MP4V/25 a=fmtp:98 profile-level-id=145; mpeg4-newpred-upstream-message=1 6.3 MIME type registration of MPEG-4 Audio MIME media type name: audio MIME subtype name: MP4A Required parameters: rate: the rate parameter indicates the RTP time stamp clock rate. The default value is 90000. Other rates CAN be specified only if it would be set to the same value with the audio sampling rate (number of samples per second). Optional parameters: profile-level-id: a decimal representation of MPEG-4 Audio Profile Level indication value defined in ISO/IEC 14496-1 . This parameter indicates the capability of subsets in MPEG-4 Audio tools. object: a decimal representation of MPEG-4 Audio Object Type value defined in ISO/IEC 14496-3 . This parameter specifies the tool to be used by the coder. It CAN be used to limit the capability within the specified "profile-level-id". bitrate: the data rate for the audio bit stream. cpresent: this parameter indicates whether audio payload configuration data is multiplexed into the RTP payload (See section 4.1 in this document). config: a hexadecimal representation of octet string indicating the audio payload configuration data "StreamMuxConfig" defined in ISO/IEC 14496-3 . The configuration data is mapped into the octet string in an MSB-first basis. The first bit of the configuration data shall be located at the MSB of the first octet. In the last octet, zero-padding bits shall follow the configuration data, if necessary. ptime: RECOMMENDED duration of each packet in milliseconds. Published specification: The payload format specification is described in this document. The specification of encoding is provided in ISO/IEC 14496-3 . Encoding considerations: This type is only defined for transfer via RTP [RFC YYYY, draft-ietf- avt-rtp-new]. Security considerations: See section 9 of RFCXXXX. Interoperability considerations: MPEG-4 Audio provides a large and rich set of tools for the coding of visual objects. In order to allow effective implementations of the standard, subsets of the MPEG-4 Audio tool sets have been identified similar to MPEG-4 Audio (See section 6.1). The audio stream SHALL be compliant with the MPEG-4 Audio Profile@Level specified by the parameter "profile-level-id". The interoperability between a sender and a receiver may be achieved by specifying the parameter "profile-level-id" in MIME content, or by exchanging this parameter in the capability exchange procedure. Furthermore, the "object" parameter can be used to limit the capability within the specified Profile@Level in capability exchange. Applications which use this media type: Audio and video streaming and conferencing tools. Additional information: none Personal & email address to contact for further information: See section 9 of RFCXXXX. Intended usage: COMMON Author/Change controller: See section 9 of RFCXXXX. 6.4 SDP usage of MPEG-4 Audio The MIME media type audio/MP4A string is assigned asmapped to fields in the candidate ofSession Description Protocol (SDP), RFC 2327, as follows: o The MIME type (audio) goes in SDP "m=" as the UMT formedia name. o The MIME subtype (MP4A) goes in SDP "a=rtpmap" as the moment. Some other MPEG-4 Audio/Visual applications usingencoding name. o The required parameter "rate" goes in "a=rtpmap" as clock rate. o The optional parameter "ptime" goes in SDP "a=ptime" attribute. o The optional parameter "profile-level-id" goes in "a=fmtp" line to indicate the upstream messages may be assignedcoder capability. The "object" parameter goes in "a=fmtp" attribute. Any payload-format-specific parameters "bitrate", "cpresent" and "config" go in "a=fmtp" line. The format and syntax of these parameters is the future. packetMIME media type (PT): 8 bitsstring as a semicolon separated list of parameter=value pairs. The valuefollowings are some examples of the packet type (PT) identifiermedia representation in SDP: For 6 kb/s CELP bitstream (the audio sampling rate of 8 kHz), m=audio 49230 RTP/AVP 96 a=rtpmap:96 MP4A/8000 a=fmtp:96 profile-level-id=9;object=8;cpresent=0;config=9128B1071070 a=ptime:20 For 64 kb/s AAC LC stereo bitstream (the audio sampling rate is 24 kHz), m=audio 49230 RTP/AVP 96 a=rtpmap:96 MP4A/24000 a=fmtp:96 profile-level-id=1; bitrate=64000; cpresent=0; config=9122620000 In the constant RTCP_MP4UM (TBD). SSRC: 32 bits SSRCabove two examples, the audio configuration data is not multiplexed into the synchronization source identifier forRTP payload and is described only in SDP. Furthermore, the sender of this packet. MPEG-4 Upstream Message Payload: variable"clock rate" is set to the audio sampling rate. If it is set to its default, the audio sampling rate can be obtained by parsing the "config" parameter. The syntax and semantics offollowing example shows that the MPEG-4 upstream messages are definedaudio configuration data appears in the ISO/IEC 14496-2/3. All messages are byte aligned. Normally one messageRTP payload. The value specified in "config" parameter is mapped onto one RTCP packet, and several messages with same UMT could be continuously mapped onto one RTCP packet. One message SHALL NOT be fragmented into different RTCP packets. 6.used as an initial value to setup coding parameters. m=audio 49230 RTP/AVP 96 a=rtpmap:96 MP4A/90000 a=fmtp:96 cpresent=1; config=9128B1071070 7. Security Considerations RTP packets using the payload format defined in this specification are subject to the security considerations discussed in the RTP specification . This implies that confidentiality of the media streams is achieved by encryption. Because the data compression used with this payload format is applied end-to-end, encryption may be performed on the compressed data so there is no conflict between the two operations. This payload type does not exhibit any significant non-uniformity in the receiver side computational complexity for packet processing to cause a potential denial-of-service threat. 7.8. References 1 Bradner, S., "The Internet Standards Process -- Revision 3", BCP 9, RFC 2026, October 1996. 2 ISO/IEC 14496-2:1999, "Information technology - Coding of audio-visual objects - Part2: Visual", December 1999. 3 ISO/IEC 14496-3:1999, "Information technology - Coding of audio-visual objects - Part3: Audio", December 1999. 4 ISO/IEC 14496-2:1999/FDAM1:2000, December 1999. 5 ISO/IEC 14496-3:1999/FDAM1:2000, December 1999. 6 ISO/IEC 14496-1:1999, "Information technology - Coding of audio-visual objects - Part1: Systems", December 1999. 7 Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997 8 H. Schulzrinne, S. Casner, R. Frederick, V. Jacobson "RTP: A Transport Protocol for Real Time Applications", RFC 1889, Internet Engineering Task Force, January 1996. 9 ISO/IEC 14496-2/DCOR1, October 1999. 10 T. Turletti, C. Hitema, "RTP Payload Format for H.261 Video Streams", RFC 2032, Octover 1996. 8.14496-2/COR1, "Information technology - Coding of audio-visual objects - Part2: Visual, Technical corrigendum 1", March 2000. 9. Author's Addresses Yoshihiro Kikuchi Toshiba corporation 1, Komukai Toshiba-cho, Saiwai-ku, Kawasaki, 212-8582, Japan Email: email@example.com@toshiba.co.jp Yoshinori Matsui Matsushita Electric Industrial Co., LTD. 1006, Kadoma, Kadoma-shi, Osaka, Japan Email: firstname.lastname@example.org Toshiyuki Nomura NEC Corporation 4-1-1,Miyazaki,Miyamae-ku,Kawasaki,JAPAN Email: email@example.com Shigeru Fukunaga Oki Electric Industry Co., Ltd. 1-2-27 Shiromi, Chuo-ku, Osaka 540-6025 Japan. Email: firstname.lastname@example.org Hideaki Kimata Nippon Telegraph and Telephone Corporation 1-1, Hikari-no-oka, Yokosuka-shi, Kanagawa, Japan Email: email@example.com Full Copyright Statement "Copyright (C) The Internet Society (date). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implmentationimplementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns.