draft-ietf-ltru-matching-04.txt   draft-ietf-ltru-matching-05.txt 
Network Working Group A. Phillips, Ed. Network Working Group A. Phillips, Ed.
Internet-Draft Quest Software Internet-Draft Quest Software
Expires: March 27, 2006 M. Davis, Ed. Obsoletes: 3066 (if approved) M. Davis, Ed.
IBM Expires: April 10, 2006 IBM
September 23, 2005 October 7, 2005
Matching Tags for the Identification of Languages Matching Tags for the Identification of Languages
draft-ietf-ltru-matching-04 draft-ietf-ltru-matching-05
Status of this Memo Status of this Memo
By submitting this Internet-Draft, each author represents that any By submitting this Internet-Draft, each author represents that any
applicable patent or other IPR claims of which he or she is aware applicable patent or other IPR claims of which he or she is aware
have been or will be disclosed, and any of which he or she becomes have been or will be disclosed, and any of which he or she becomes
aware will be disclosed, in accordance with Section 6 of BCP 79. aware will be disclosed, in accordance with Section 6 of BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF), its areas, and its working groups. Note that
skipping to change at page 1, line 35 skipping to change at page 1, line 35
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt. http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html. http://www.ietf.org/shadow.html.
This Internet-Draft will expire on March 27, 2006. This Internet-Draft will expire on April 10, 2006.
Copyright Notice Copyright Notice
Copyright (C) The Internet Society (2005). Copyright (C) The Internet Society (2005).
Abstract Abstract
This document describes different mechanisms for comparing, matching, This document describes different mechanisms for comparing, matching,
and evaluating language tags. Possible algorithms for language and evaluating language tags. Possible algorithms for language
negotiation and content selection are described. negotiation and content selection are described.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
2. The Language Range . . . . . . . . . . . . . . . . . . . . . . 4 2. The Language Range . . . . . . . . . . . . . . . . . . . . . . 4
2.1 Basic Language Range . . . . . . . . . . . . . . . . . . . 4 2.1. Lists of Language Ranges . . . . . . . . . . . . . . . . . 4
2.1.1 Matching . . . . . . . . . . . . . . . . . . . . . . . 5 2.2. Basic Language Range . . . . . . . . . . . . . . . . . . . 4
2.1.2 Lookup . . . . . . . . . . . . . . . . . . . . . . . . 6 2.2.1. Matching . . . . . . . . . . . . . . . . . . . . . . . 5
2.2 Extended Language Range . . . . . . . . . . . . . . . . . 7 2.2.2. Lookup . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2.1 Extended Range Matching . . . . . . . . . . . . . . . 8 2.3. Extended Language Range . . . . . . . . . . . . . . . . . 7
2.2.2 Extended Range Lookup . . . . . . . . . . . . . . . . 9 2.3.1. Extended Range Matching . . . . . . . . . . . . . . . 9
2.2.3 Distance Metric Scheme . . . . . . . . . . . . . . . . 10 2.3.2. Extended Range Lookup . . . . . . . . . . . . . . . . 10
2.3 Meaning of Language Tags and Ranges . . . . . . . . . . . 12 2.3.3. Distance Metric Scheme . . . . . . . . . . . . . . . . 11
2.4 Choosing Between Alternate Matching Schemes . . . . . . . 13 2.4. Meaning of Language Tags and Ranges . . . . . . . . . . . 13
2.5 Considerations for Private Use Subtags . . . . . . . . . . 14 2.5. Choosing Between Alternate Matching Schemes . . . . . . . 14
2.6 Length Considerations in Matching . . . . . . . . . . . . 14 2.6. Considerations for Private Use Subtags . . . . . . . . . . 15
3. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 17 2.7. Length Considerations in Matching . . . . . . . . . . . . 16
4. Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 3. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 18
5. Security Considerations . . . . . . . . . . . . . . . . . . . 19 4. Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
6. Character Set Considerations . . . . . . . . . . . . . . . . . 20 5. Security Considerations . . . . . . . . . . . . . . . . . . . 20
7. References . . . . . . . . . . . . . . . . . . . . . . . . . . 21 6. Character Set Considerations . . . . . . . . . . . . . . . . . 21
7.1 Normative References . . . . . . . . . . . . . . . . . . . 21 7. References . . . . . . . . . . . . . . . . . . . . . . . . . . 22
7.2 Informative References . . . . . . . . . . . . . . . . . . 22 7.1. Normative References . . . . . . . . . . . . . . . . . . . 22
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . 23 7.2. Informative References . . . . . . . . . . . . . . . . . . 23
A. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 24 Appendix A. Acknowledgements . . . . . . . . . . . . . . . . . . 24
Intellectual Property and Copyright Statements . . . . . . . . 25 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 25
Intellectual Property and Copyright Statements . . . . . . . . . . 26
1. Introduction 1. Introduction
Human beings on our planet have, past and present, used a number of Human beings on our planet have, past and present, used a number of
languages. There are many reasons why one would want to identify the languages. There are many reasons why one would want to identify the
language used when presenting or requesting information. language used when presenting or requesting information.
Information about a user's language preferences commonly needs to be Information about a user's language preferences commonly needs to be
identified so that appropriate processing can be applied. For identified so that appropriate processing can be applied. For
example, the user's language preferences in a browser can be used to example, the user's language preferences in a browser can be used to
skipping to change at page 4, line 21 skipping to change at page 4, line 21
[RFC2616] describes language ranges in its discussion of the Accept- [RFC2616] describes language ranges in its discussion of the Accept-
Language header (Section 14.4), which is used for selecting content Language header (Section 14.4), which is used for selecting content
from servers based on the language of that content. from servers based on the language of that content.
When selecting content according to its language, it is useful to When selecting content according to its language, it is useful to
have a mechanism for identifying sets of language tags that share have a mechanism for identifying sets of language tags that share
specific attributes. This allows users to select or filter content specific attributes. This allows users to select or filter content
based on specific requirements. Such an identifier is called a based on specific requirements. Such an identifier is called a
"Language Range". "Language Range".
2.1 Basic Language Range 2.1. Lists of Language Ranges
A basic language range identifies the set of content whose language When users specify a language preference they often need to specify a
tags begin with the same sequence of subtags. Basic language ranges prioritized list of language ranges in order to best reflect their
are described in [RFC3066] and HTTP 1.1 [RFC2616] (where they are language requirements for the matching operation. This is especially
referred to as simply a "language range"). A basic language range is true for speakers of minority languages. A speaker of Breton in
identified by its 'language-range' tag, by adapting the France, for example, may specify "be" followed by "fr", meaning that
if Breton is available, it is preferred, but otherwise French is the
best alternative. It can get more complex: a speaker may wish to
fallback from Skolt Sami to Northern Sami to Finnish.
A "Language Priority List" consists of a prioritized or weighted list
of language ranges. One well known example of such a list is the
"Accept-Language" header defined in RFC 2616 [RFC2616] (see Section
14.4) and RFC 3282 [RFC3282]. The various matching operations
described in this document include considerations for using a
language priority list.
2.2. Basic Language Range
A "Basic Language Range" identifies the set of content whose language
tags begin with the same sequence of subtags. A basic language range
is identified by its 'language-range' tag, by adapting the
ABNF[RFC2234bis] from HTTP/1.1 [RFC2616] : ABNF[RFC2234bis] from HTTP/1.1 [RFC2616] :
language-range = language-tag / "*" language-range = language-tag / "*"
language-tag = 1*8[alphanum] *["-" 1*8alphanum] language-tag = 1*8[alphanum] *["-" 1*8alphanum]
alphanum = ALPHA / DIGIT alphanum = ALPHA / DIGIT
That is, a language-range has the same syntax as a language-tag or is That is, a language-range has the same syntax as a language-tag or is
the single character "*". Basic Language Ranges imply that there is the single character "*". Basic Language Ranges imply that there is
a semantic relationship between language tags that share the same a semantic relationship between language tags that share the same
prefix. While this is often the case, it is not always true. prefix. While this is often the case, it is not always true and
users should note that the set of language tags that match a specific
language-range may not be mutually intelligible.
In particular, the set of language tags that match a specific Basic language ranges were originally described in [RFC3066] and HTTP
language-range might not all be mutually intelligible. Matching a 1.1 [RFC2616] (where they are referred to as simply a "language
language-range (prefix) to various language tags does not mean that range").
it is always true that if a user understands a language identified by
a certain tag, then this user will also understand all of the Users SHOULD avoid subtags that add no distinguishing value to a
languages for which this tag is a prefix. The use of prefixes (and language range. For example, script subtags SHOULD NOT be used to
thus basic language ranges) simply allows the use of a prefix if this form a language range with language subtags which have a matching
is the case. Suppress-Script field in their registry record. Thus the language
range "en-Latn" is probably inappropriate for most applications
(because the vast majority English documents are written in the Latin
script and thus the 'en' language subtag has a Suppress-Script field
for 'Latn' in the registry).
Language tags and thus language ranges are to be treated as case Language tags and thus language ranges are to be treated as case
insensitive: there exist conventions for the capitalization of some insensitive: there exist conventions for the capitalization of some
of the subtags, but these MUST NOT be taken to carry meaning. of the subtags, but these MUST NOT be taken to carry meaning.
Matching of language tags to language ranges MUST be done in a case Matching of language tags to language ranges MUST be done in a case
insensitive manner. insensitive manner.
When working with tags and ranges users SHOULD also note the When working with tags and ranges, note that extensions and most
following: private use subtags are generally orthogonal to language tag fallback
and users SHOULD avoid using these subtags in language ranges, since
1. Private-use and Extension subtags are normally orthogonal to they will often interfere with the selection of available language
language tag fallback. Implementations SHOULD ignore content. Since these subtags are always at the end of the sequence
unrecognized private-use and extension subtags when performing of subtags, they don't normally interfere with the use of prefixes
language tag fallback. Since these subtags are always at the end for matching in the schemes described below.
of the sequence of subtags, they don't normally interfere with
the use of prefixes for matching in the schemes described below.
2. Implementations that choose not to interpret one or more private-
use or extension subtags SHOULD NOT remove or modify these
extensions in content that they are processing. When a language
tag instance is to be used in a specific, known protocol, and is
not being passed through to other protocols, language tags MAY be
filtered to remove subtags and extensions that are not supported
by that protocol. Such filtering SHOULD be avoided, if possible,
since it removes information that might be relevant if services
on the other end of the protocol would make use of that
information.
3. Some applications of language tags might want or need to consider
extensions and private-use subtags when matching tags. If
extensions and private-use subtags are included in a matching or
filtering process that utilizes the one of the schemes described
in this document, then the implementation SHOULD canonicalize the
language tags and/or ranges before performing the matching. Note
that language tag processors that claim to be "well-formed"
processors as defined in [draft-registry] generally fall into
this category.
There are two matching schemes that are commonly associated with There are two matching schemes that are commonly associated with
basic language ranges: matching and lookup. basic language ranges: matching and lookup.
Note that neither matching nor lookup using basic language ranges Note that neither matching nor lookup using basic language ranges
attempt to process the semantics of the tags or ranges in any way. attempt to process the semantics of the tags or ranges in any way.
The language tag and language range are compared in a case The language tag and language range are compared in a case
insensitive manner using basic string processing. The choice of insensitive manner using basic string processing. The choice of
subtags in both the language tag and language range may affect the subtags in both the language tag and language range may affect the
results produced as a result. results produced as a result.
2.1.1 Matching 2.2.1. Matching
Language tag matching is used to select all content that matches a Language tag matching is used to select all content that matches a
given prefix. In matching, the language range represents the least given prefix. In matching, the language range represents the least
specific tag which is an acceptable match and every piece of content specific tag which is an acceptable match and every piece of content
that matches is returned. that matches is returned. If the language priority list contains
more than one range, the matches returned are typically ordered in
descending level of preference.
For example, if an application is applying a style to all content in For example, if an application is applying a style to all content in
a document in a particular language, it might use language tag a document in a particular language, it might use language tag
matching to select the content to which the style is applied. matching to select the content to which the style is applied.
A language-range matches a language-tag if it exactly equals the tag, A language-range matches a language-tag if it exactly equals the tag,
or if it exactly equals a prefix of the tag such that the first or if it exactly equals a prefix of the tag such that the first
character following the prefix is "-". (That is, the language-range character following the prefix is "-". (That is, the language-range
"de-de" matches the language tag "de-DE-1996", but not the language "de-de" matches the language tag "de-DE-1996", but not the language
tag "de-Deva".) tag "de-Deva".)
The special range "*" matches any tag. A protocol which uses The special range "*" matches any tag. A protocol which uses
language ranges MAY specify additional rules about the semantics of language ranges MAY specify additional rules about the semantics of
"*"; for instance, HTTP/1.1 specifies that the range "*" matches only "*"; for instance, HTTP/1.1 specifies that the range "*" matches only
languages not matched by any other range within an "Accept-Language" languages not matched by any other range within an "Accept-Language"
header. header.
2.1.2 Lookup 2.2.2. Lookup
Content lookup is used to select the single information item that Content lookup is used to select the single information item that
best matches the language range for a given request. In lookup, the best matches the language priority list for a given request. In
language range represents the most specific tag which is an lookup, each language range in the language priority list represents
acceptable match and only the closest matching item is returned. the most specific tag which is an acceptable match; only the closest
matching item according the user's priority is returned.
For example, if an application inserts some dynamic content into a For example, if an application inserts some dynamic content into a
document, returning an empty string if there is no exact match is not document, returning an empty string if there is no exact match is not
an option. Instead, the application "falls back" until it finds a an option. Instead, the application "falls back" until it finds a
suitable piece of content to insert. suitable piece of content to insert.
When performing lookup, the language range is progressively truncated When performing lookup, the language range is progressively truncated
from the end until a matching piece of content is located. For from the end until a matching piece of content is located. For
example, starting with the range "zh-Hant-CN-x-wadegile", the lookup example, starting with the range "zh-Hant-CN-x-wadegile", the lookup
would progressively search for content as shown below: would progressively search for content as shown below:
skipping to change at page 7, line 5 skipping to change at page 7, line 6
4. zh 4. zh
5. (default content or the empty tag) 5. (default content or the empty tag)
Figure 2: Default Fallback Pattern Example Figure 2: Default Fallback Pattern Example
This scheme allows some flexibility in finding content. It also This scheme allows some flexibility in finding content. It also
typically provides better results when data is not available at a typically provides better results when data is not available at a
specific level of tag granularity or is sparsely populated (than if specific level of tag granularity or is sparsely populated (than if
the default language for the system or content were used). the default language for the system or content were used).
2.2 Extended Language Range When performing lookup using a language priority list, the
progressive search MUST proceed to consider each language range
before finding the default content or empty tag. For example, for
the list "fr-FR; zh-Hant" would search for content as follows:
1. fr-FR
2. fr
3. zh-Hant // next language
4. zh
5. (default content or the empty tag)
Figure 3: Lookup Using a Language Priority List
2.3. Extended Language Range
Prefix matching using a Basic Language Range, as described above, is Prefix matching using a Basic Language Range, as described above, is
not always the most appropriate way to access the information not always the most appropriate way to access the information
contained in language tags when selecting or filtering content. Some contained in language tags when selecting or filtering content. Some
applications might wish to define a more granular matching scheme and applications might wish to define a more granular matching scheme and
such a matching scheme requires the ability to specify the various such a matching scheme requires the ability to specify the various
attributes of a language tag in the language range. An extended attributes of a language tag in the language range. An extended
language range can be represented by the following ABNF: language range can be represented by the following ABNF:
extended-language-range = range ; a range extended-language-range = range ; a range
/ privateuse ; private use tag / privateuse ; private use tag
skipping to change at page 8, line 27 skipping to change at page 8, line 41
special subtag "*". For example, the language range "en-*-US" special subtag "*". For example, the language range "en-*-US"
specifies a primary language of 'en', followed by any script subtag, specifies a primary language of 'en', followed by any script subtag,
followed by the region subtag 'US'. followed by the region subtag 'US'.
A field not present in the middle of an extended language range MAY A field not present in the middle of an extended language range MAY
be treated as if the field contained a "*". For example, the range be treated as if the field contained a "*". For example, the range
"en-US" MAY be considered to be equivalent to the range "en-*-US". "en-US" MAY be considered to be equivalent to the range "en-*-US".
This also means that multiple wildcards can be collapsed (so that This also means that multiple wildcards can be collapsed (so that
"en-*-*-US" is equivalent to "en-*-US"). "en-*-*-US" is equivalent to "en-*-US").
When working with tags and ranges users SHOULD note the following:
1. Private-use and Extension subtags are normally orthogonal to
language tag fallback. Implementations SHOULD ignore
unrecognized private-use and extension subtags when performing
language tag fallback. Since these subtags are always at the end
of the sequence of subtags, they don't normally interfere with
the use of prefixes for matching in the schemes described below.
2. Implementations that choose not to interpret one or more private-
use or extension subtags SHOULD NOT remove or modify these
extensions in content that they are processing. When a language
tag instance is to be used in a specific, known protocol, and is
not being passed through to other protocols, language tags MAY be
filtered to remove subtags and extensions that are not supported
by that protocol. Such filtering SHOULD be avoided, if possible,
since it removes information that might be relevant if services
on the other end of the protocol would make use of that
information.
3. Some applications of language tags might want or need to consider
extensions and private-use subtags when matching tags. If
extensions and private-use subtags are included in a matching or
filtering process that utilizes the one of the schemes described
in this document, then the implementation SHOULD canonicalize the
language tags and/or ranges before performing the matching. Note
that language tag processors that claim to be "well-formed"
processors as defined in [draft-registry] generally fall into
this category.
There are several matching algorithms or schemes which can be applied There are several matching algorithms or schemes which can be applied
when matching extended language ranges to language tags. when matching extended language ranges to language tags.
2.2.1 Extended Range Matching 2.3.1. Extended Range Matching
In extended range matching, the subtags in a language tag are In extended range matching, each extended language range in the
compared to the corresponding subtags in the extended language range. language priority list is considered in turn, according to priority.
A subtag is considered to match if it exactly matches the The subtags in each extended language range are compared to the
corresponding subtag in the range or the range contains a subtag with corresponding subtags in the language tag being examined. The subtag
the value "*" (which matches all subtags, including the empty from the range is considered to match if it exactly matches the
subtag). Extended Range Matching is an extension of basic matching corresponding subtag in the tag or the range's subtag has the value
(Section 2.1.1): the language range represents the least specific tag "*" (which matches all subtags, including the empty subtag).
Extended Range Matching is an extension of basic matching
(Section 2.2.1): the language range represents the least specific tag
which is an acceptable match. which is an acceptable match.
By default all extensions and their subtags are ignored for extended
language range matching.
Private use subtags MAY be specified in the language range and MUST Private use subtags MAY be specified in the language range and MUST
NOT be ignored when matching. NOT be ignored when matching.
Subtags not specified, including those at the end of the language Subtags not specified, including those at the end of the language
range, are assigned the value "*". This makes each range into a range, are assigned the value "*". This makes each range into a
prefix much like that used in basic language range matching. For prefix much like that used in basic language range matching. For
example, the extended language range "zh-*-CN" matches all of the example, the extended language range "zh-*-CN" matches all of the
following tags because the unspecified variant field is expanded to following tags because the unspecified variant field is expanded to
"*": "*":
skipping to change at page 9, line 18 skipping to change at page 10, line 14
zh-CN zh-CN
zh-Hans-CN zh-Hans-CN
zh-CN-x-wadegile zh-CN-x-wadegile
zh-Latn-CN-boont zh-Latn-CN-boont
zh-cmn-Hans-CN-x-wadegile zh-cmn-Hans-CN-x-wadegile
2.2.2 Extended Range Lookup 2.3.2. Extended Range Lookup
In extended range lookup, the subtags in a language tag are compared In extended range lookup, each extended language range in the
to the corresponding subtags in the extended language range. The language priority list is considered in turn. The subtags in each
subtag is considered to match if it exactly matches the corresponding extended language range are compared to the corresponding subtags in
subtag in the range or the range contains a subtag with the value "*" the language tag being examined. A subtag is considered to match if
(which matches all subtags, including the empty subtag). Extended it exactly matches the corresponding subtag in the tag or the range's
language range lookup is an extension of basic lookup subtag has the value "*" (which matches all subtags, including the
(Section 2.1.2): the language range represents the most specific tag empty subtag). Extended language range lookup is an extension of
which will form an acceptable match. basic lookup (Section 2.2.2): each language range represents the most
specific tag which will form an acceptable match. If no match is
found, the default content or content with the empty language tag is
usually returned (or the search can be considered to have failed).
Subtags not specified are assigned the value "*" prior to performing Subtags not specified are assigned the value "*" prior to performing
tag matching. Unlike in extended range matching, however, fields at tag matching. Unlike in extended range matching, however, fields at
the end of the range MUST NOT be expanded in this manner. For the end of the range MUST NOT be expanded in this manner. For
example, "en-US" MUST NOT be considered to be the same as the range example, "en-US" MUST NOT be considered to be the same as the range
"en-US-*". This allows ranges to be specific. The "*" wildcard MUST "en-US-*". This allows ranges to be specific. The "*" wildcard MUST
be used at the end of the range to indicate that all tags with the be used at the end of the range to indicate that all tags with the
range as a prefix are allowable matches. That is, the range "zh-*" range as a prefix are allowable matches. That is, the range "zh-*"
matches the tags "zh-Hant" and "zh-Hant-CN", while the range "zh" matches the tags "zh-Hant" and "zh-Hant-CN", while the range "zh"
matches neither of those tags. matches neither of those tags.
The wildcard "*" at the end of a range SHOULD be considered to match The wildcard "*" at the end of a range SHOULD be considered to match
any private use subtag sequences (making extended language range any private use subtag sequences (making extended language range
lookup function exactly like extended range matching Section 2.2.1). lookup function exactly like extended range matching Section 2.3.1).
By default all extensions and their subtags SHOULD be ignored for By default all extensions and their subtags SHOULD be ignored for
extended language range lookup. Private use subtags MAY be specified extended language range lookup. Private use subtags MAY be specified
in the language range and MUST NOT be ignored when performing lookup. in the language range and MUST NOT be ignored when performing lookup.
The wildcard "*" at the end of a range SHOULD be considered to match The wildcard "*" at the end of a range SHOULD be considered to match
any private use subtag sequences in addition to variants. any private use subtag sequences in addition to variants.
For example, the range "*-US" matches all of the following tags: For example, the range "*-US" matches all of the following tags:
en-US en-US
skipping to change at page 10, line 40 skipping to change at page 11, line 37
en-Latn-US-scouse en-Latn-US-scouse
en-US en-US
en-scouse en-scouse
Note that the ability to be specific in extended range lookup can Note that the ability to be specific in extended range lookup can
make this matching scheme a more appropriate replacement for basic make this matching scheme a more appropriate replacement for basic
matching than the extended range matching scheme. matching than the extended range matching scheme.
2.2.3 Distance Metric Scheme 2.3.3. Distance Metric Scheme
Both Basic and Extended Language Ranges produce simple boolean Both Basic and Extended Language Ranges produce simple boolean
matches. Some applications may benefit by providing an array of matches. Some applications may benefit by providing an array of
results with different levels of matching, for example, sorting results with different levels of matching, for example, sorting
results based on the overall "quality" of the match. results based on the overall "quality" of the match.
This type of matching is sometimes called a "distance metric". A This type of matching is sometimes called a "distance metric". A
distance metric assigns a pair of language tags a numeric value distance metric assigns a pair of language tags a numeric value
representing the 'distance' between the two. A distance of zero representing the 'distance' between the two. A distance of zero
means that they are identical, a small distance indicates that they means that they are identical, a small distance indicates that they
skipping to change at page 12, line 32 skipping to change at page 13, line 32
"en-US-x-foo" 4 (variant mismatch: range is the empty string) "en-US-x-foo" 4 (variant mismatch: range is the empty string)
"en-US-r-wadegile" 1 (extension mismatch: range is the empty string) "en-US-r-wadegile" 1 (extension mismatch: range is the empty string)
Implementations may want to use more sophisticated weights that Implementations may want to use more sophisticated weights that
depend on the values of the corresponding elements. For example, depend on the values of the corresponding elements. For example,
depending on the domain, an implemenation might give a small distance depending on the domain, an implemenation might give a small distance
to the difference between the language subtag 'no' and the closely to the difference between the language subtag 'no' and the closely
related language subtags 'nb' or 'nn'; or between the script subtags related language subtags 'nb' or 'nn'; or between the script subtags
'Kata' and 'Hira'; or between the region subtags 'US' and 'UM'. 'Kata' and 'Hira'; or between the region subtags 'US' and 'UM'.
2.3 Meaning of Language Tags and Ranges 2.4. Meaning of Language Tags and Ranges
A language tag defines a language as spoken (or written, signed or A language tag defines a language as spoken (or written, signed or
otherwise signaled) by human beings for communication of information otherwise signaled) by human beings for communication of information
to other human beings. to other human beings.
If a language tag B contains language tag A as a prefix, then B is If a language tag B contains language tag A as a prefix, then B is
typically "narrower" or "more specific" than A. For example, "zh- typically "narrower" or "more specific" than A. For example, "zh-
Hant-TW" is more specific than "zh-Hant". Hant-TW" is more specific than "zh-Hant".
This relationship is not guaranteed in all cases: specifically, This relationship is not guaranteed in all cases: specifically,
skipping to change at page 13, line 44 skipping to change at page 14, line 44
structure (including the whole document itself). For example, one structure (including the whole document itself). For example, one
could write <span lang="FR">C'est la vie.</span> inside a could write <span lang="FR">C'est la vie.</span> inside a
Norwegian document; the Norwegian-speaking user could then access Norwegian document; the Norwegian-speaking user could then access
a French-Norwegian dictionary to find out what the marked section a French-Norwegian dictionary to find out what the marked section
meant. If the user were listening to that document through a meant. If the user were listening to that document through a
speech synthesis interface, this formation could be used to signal speech synthesis interface, this formation could be used to signal
the synthesizer to appropriately apply French text-to-speech the synthesizer to appropriately apply French text-to-speech
pronunciation rules to that span of text, instead of misapplying pronunciation rules to that span of text, instead of misapplying
the Norwegian rules. the Norwegian rules.
2.4 Choosing Between Alternate Matching Schemes 2.5. Choosing Between Alternate Matching Schemes
Implementations can choose to implement different styles of matching Implementers are faced with the decision of what form of matching to
for different kinds of processing. For example, an implementation use in a specific application. An application can choose to
could treat an absent script subtag as a "wildcard" field; thus implement different styles of matching for different kinds of
"az-AZ" would match "az-AZ", "az-Cyrl-AZ", "az-Latn-AZ", etc. but not processing.
"az" (this is an example of extended range lookup). If one item is
to be chosen, the implementation could pick among those matches based
on other information, such as the most likely script used in the
language/region in question or the script used by other content
selected.
Because the primary language subtag cannot be absent in a language The most basic choice is between schemes that produce an open-ended
tag, the 'und' subtag is sometimes be used as a 'wildcard' in basic set of content (a "matching" application) and those that usually
matching. For example, in a query where you want to select all produce a single information item (a "lookup" application). Note
language tags that contain 'Latn' as the script code and 'AZ' as the that lookup applications can produce multiple items, but usually only
region code, you could use the range "und-Latn-AZ". This requires an a single item for any given piece of content, and they can be used to
implementation to examine the actual values of the subtags, though, order content (the later in the overall fallback that the content
and users SHOULD NOT assume that the value 'und' will be treated as a appears to match, the more distant the match).
wildcard.
The matching schemes described elsewhere in this document are Matching applications can produce an ordered or unordered set of
designed such that implementations do not have to examine the values results. For example, applying formatting to a document based on the
of the subtags supplied and, except for scored matching, they do not language of specific pieces of content does not require the content
need access to the Language Subtag Registry nor do they require the to be ordered. It is sufficient to know whether a specific piece of
use of valid subtags in language tags or ranges. This has great content matches or does not match. A search application, on the
benefit for speed and simplicity of implementation. other hand, probably would put the results into a priority order.
If single item is to be chosen, it may sometimes be useful to apply
additional information, such as the most likely script used in the
language or region in question or the script used by other content
selected, in order to make a more "informed" choice.
The matching schemes in this document are designed so that
implementations do not have to examine the values of the subtags
supplied and, except for scored matching, they do not need access to
the Language Subtag Registry nor do they require the use of valid
subtags in language tags or ranges. This has great benefit for speed
and simplicity of implementation.
Implementations might also wish to use semantic information external Implementations might also wish to use semantic information external
to the langauge tags when performing fallback. For example, the to the langauge tags when performing fallback. For example, the
primary language subtags 'nn' (Nynorsk Norwegian) and 'nb' (Bokmal primary language subtags 'nn' (Nynorsk Norwegian) and 'nb' (Bokmal
Norwegian) might both be usefully matched to the more general subtag Norwegian) might both be usefully matched to the more general subtag
'no' (Norwegian). Or an application might infer that content labeled 'no' (Norwegian). Or an application might infer that content labeled
"zh-CN" is morely likely to match the range "zh-Hans" than equivalent "zh-CN" is morely likely to match the range "zh-Hans" than equivalent
content labeled "zh-TW". content labeled "zh-TW".
2.5 Considerations for Private Use Subtags 2.6. Considerations for Private Use Subtags
Private-use subtags require private agreement between the parties Private-use subtags require private agreement between the parties
that intend to use or exchange language tags that use them and great that intend to use or exchange language tags that use them and great
caution SHOULD be used in employing them in content or protocols caution SHOULD be used in employing them in content or protocols
intended for general use. Private-use subtags are simply useless for intended for general use. Private-use subtags are simply useless for
information exchange without prior arrangement. information exchange without prior arrangement.
The value and semantic meaning of private-use tags and of the subtags The value and semantic meaning of private-use tags and of the subtags
used within such a language tag are not defined. Matching private used within such a language tag are not defined. Matching private
use tags using language ranges or extended language ranges can result use tags using language ranges or extended language ranges can result
in unpredictable content being returned. in unpredictable content being returned.
2.6 Length Considerations in Matching 2.7. Length Considerations in Matching
RFC 3066 [RFC3066] did not provide an upper limit on the size of RFC 3066 [RFC3066] did not provide an upper limit on the size of
language tags or ranges. RFC 3066 did define the semantics of language tags or ranges. RFC 3066 did define the semantics of
particular subtags in such a way that most language tags or ranges particular subtags in such a way that most language tags or ranges
consisted of language and region subtags with a combined total length consisted of language and region subtags with a combined total length
of up to six characters. Larger tags and ranges (in terms of both of up to six characters. Larger tags and ranges (in terms of both
subtags and characters) did exist, however. subtags and characters) did exist, however.
[draft-registry] also does not impose a fixed upper limit on the [draft-registry] also does not impose a fixed upper limit on the
number of subtags in a language tag or range (and thus an upper bound number of subtags in a language tag or range (and thus an upper bound
skipping to change at page 15, line 26 skipping to change at page 16, line 34
be clearly documented, and such documentation SHOULD include the be clearly documented, and such documentation SHOULD include the
disposition of any longer tags or ranges (for example, whether an disposition of any longer tags or ranges (for example, whether an
error value is generated or the language tag or range is truncated). error value is generated or the language tag or range is truncated).
If truncation is permitted it MUST NOT permit a subtag to be divided, If truncation is permitted it MUST NOT permit a subtag to be divided,
since this changes the semantics of the subtag being matched and can since this changes the semantics of the subtag being matched and can
result in false positives or negatives. result in false positives or negatives.
Implementations that restrict storage SHOULD consider the impact of Implementations that restrict storage SHOULD consider the impact of
tag or range truncation on the resulting matches. For example, tag or range truncation on the resulting matches. For example,
removing the "*" from the end of an extended language range (see removing the "*" from the end of an extended language range (see
Section 2.2) can greatly modify the set of returned matches. A Section 2.3) can greatly modify the set of returned matches. A
protocol that allows tags or ranges to be truncated at an arbitrary protocol that allows tags or ranges to be truncated at an arbitrary
limit, without giving any indication of what that limit is, has the limit, without giving any indication of what that limit is, has the
potential for causing harm by changing the meaning of values in potential for causing harm by changing the meaning of values in
substantial ways. substantial ways.
In practice, most tags do not require additional subtags or In practice, most tags do not require additional subtags or
substantially more characters. Additional subtags sometimes add substantially more characters. Additional subtags sometimes add
useful distinguishing information, but extraneous subtags interfere useful distinguishing information, but extraneous subtags interfere
with the meaning, understanding, and especially matching of language with the meaning, understanding, and especially matching of language
tags. Since language tags or ranges MAY be truncated by an tags. Since language tags or ranges MAY be truncated by an
skipping to change at page 16, line 24 skipping to change at page 17, line 32
example: example:
Tag to truncate: zh-Hant-CN-variant1-a-extend1-x-wadegile-private1 Tag to truncate: zh-Hant-CN-variant1-a-extend1-x-wadegile-private1
1. zh-Hant-CN-variant1-a-extend1-x-wadegile 1. zh-Hant-CN-variant1-a-extend1-x-wadegile
2. zh-Hant-CN-variant1-a-extend1 2. zh-Hant-CN-variant1-a-extend1
3. zh-Hant-CN-variant1 3. zh-Hant-CN-variant1
4. zh-Hant-CN 4. zh-Hant-CN
5. zh-Hant 5. zh-Hant
6. zh 6. zh
Figure 6: Example of Tag Truncation Figure 7: Example of Tag Truncation
3. IANA Considerations 3. IANA Considerations
This document presents no new or existing considerations for IANA. This document presents no new or existing considerations for IANA.
4. Changes 4. Changes
This is the first version of this document. This is the first version of this document.
The following changes were put into this document since draft-03: The following changes were put into this document since draft-03:
skipping to change at page 21, line 7 skipping to change at page 22, line 7
6. Character Set Considerations 6. Character Set Considerations
The syntax of language tags and language ranges permit only the The syntax of language tags and language ranges permit only the
characters A-Z, a-z, 0-9, and HYPHEN-MINUS (%x2D). These characters characters A-Z, a-z, 0-9, and HYPHEN-MINUS (%x2D). These characters
are present in most character sets, so presentation of language tags are present in most character sets, so presentation of language tags
should not present any character set issues. should not present any character set issues.
7. References 7. References
7.1 Normative References 7.1. Normative References
[ID.ietf-ltru-initial] [ID.ietf-ltru-initial]
Ewell, D., Ed., "Language Tags Initial Registry (work in Ewell, D., Ed., "Language Tags Initial Registry (work in
progress)", August 2005, <http://www.ietf.org/ progress)", August 2005, <http://www.ietf.org/
internet-drafts/draft-ietf-ltru-initial-04.txt>. internet-drafts/draft-ietf-ltru-initial-04.txt>.
[RFC1327] Hardcastle-Kille, S., "Mapping between X.400(1988) / ISO [RFC1327] Hardcastle-Kille, S., "Mapping between X.400(1988) / ISO
10021 and RFC 822", RFC 1327, May 1992. 10021 and RFC 822", RFC 1327, May 1992.
[RFC1521] Borenstein, N. and N. Freed, "MIME (Multipurpose Internet [RFC1521] Borenstein, N. and N. Freed, "MIME (Multipurpose Internet
skipping to change at page 22, line 14 skipping to change at page 23, line 14
[RFC3629] Yergeau, F., "UTF-8, a transformation format of ISO [RFC3629] Yergeau, F., "UTF-8, a transformation format of ISO
10646", STD 63, RFC 3629, November 2003. 10646", STD 63, RFC 3629, November 2003.
[draft-registry] [draft-registry]
Phillips, A., Ed. and M. Davis, Ed., "Tags for the Phillips, A., Ed. and M. Davis, Ed., "Tags for the
Identification of Languages (work in progress)", Identification of Languages (work in progress)",
August 2005, <http://www.ietf.org/internet-drafts/ August 2005, <http://www.ietf.org/internet-drafts/
draft-ietf-ltru-registry-12.txt>. draft-ietf-ltru-registry-12.txt>.
7.2 Informative References 7.2. Informative References
[ISO15924] [ISO15924]
"ISO 15924:2004. Information and documentation -- Codes "ISO 15924:2004. Information and documentation -- Codes
for the representation of names of scripts", January 2004. for the representation of names of scripts", January 2004.
[ISO3166-1] [ISO3166-1]
"ISO 3166-1:1997. Codes for the representation of names of "ISO 3166-1:1997. Codes for the representation of names of
countries and their subdivisions -- Part 1: Country countries and their subdivisions -- Part 1: Country
codes", 1997. codes", 1997.
skipping to change at page 22, line 39 skipping to change at page 23, line 39
[ISO639-2] [ISO639-2]
"ISO 639-2:1998. Codes for the representation of names of "ISO 639-2:1998. Codes for the representation of names of
languages -- Part 2: Alpha-3 code, first edition", 1998. languages -- Part 2: Alpha-3 code, first edition", 1998.
[RFC1766] Alvestrand, H., "Tags for the Identification of [RFC1766] Alvestrand, H., "Tags for the Identification of
Languages", RFC 1766, March 1995. Languages", RFC 1766, March 1995.
[RFC3066] Alvestrand, H., "Tags for the Identification of [RFC3066] Alvestrand, H., "Tags for the Identification of
Languages", BCP 47, RFC 3066, January 2001. Languages", BCP 47, RFC 3066, January 2001.
[RFC3282] Alvestrand, H., "Content Language Headers", RFC 3282,
May 2002.
[RFC3339] Klyne, G. and C. Newman, "Date and Time on the Internet: [RFC3339] Klyne, G. and C. Newman, "Date and Time on the Internet:
Timestamps", RFC 3339, July 2002. Timestamps", RFC 3339, July 2002.
[UN_M.49] Statistics Division, United Nations, "Standard Country or [UN_M.49] Statistics Division, United Nations, "Standard Country or
Area Codes for Statistical Use", UN Standard Country or Area Codes for Statistical Use", UN Standard Country or
Area Codes for Statistical Use, Revision 4 (United Nations Area Codes for Statistical Use, Revision 4 (United Nations
publication, Sales No. 98.XVII.9, June 1999. publication, Sales No. 98.XVII.9, June 1999.
Authors' Addresses
Addison Phillips (editor)
Quest Software
Email: addison dot phillips at quest dot com
Mark Davis (editor)
IBM
Email: mark dot davis at ibm dot com
Appendix A. Acknowledgements Appendix A. Acknowledgements
Any list of contributors is bound to be incomplete; please regard the Any list of contributors is bound to be incomplete; please regard the
following as only a selection from the group of people who have following as only a selection from the group of people who have
contributed to make this document what it is today. contributed to make this document what it is today.
The contributors to [draft-registry], [RFC3066] and [RFC1766], each The contributors to [draft-registry], [RFC3066] and [RFC1766], each
of which is a precursor to this document, made enormous of which is a precursor to this document, made enormous contributions
contributions directly or indirectly to this document and are directly or indirectly to this document and are generally responsible
generally responsible for the success of language tags. for the success of language tags.
The following people (in alphabetical order by family name) The following people (in alphabetical order by family name)
contributed to this document: contributed to this document:
Jeremy Carroll, John Cowan, Frank Ellermann, Doug Ewell, Kent Jeremy Carroll, John Cowan, Frank Ellermann, Doug Ewell, Kent
Karlsson, Ira McDonald, M. Patton, Randy Presuhn and many, many Karlsson, Ira McDonald, M. Patton, Randy Presuhn and many, many
others. others.
Very special thanks must go to Harald Tveit Alvestrand, who Very special thanks must go to Harald Tveit Alvestrand, who
originated RFCs 1766 and 3066, and without whom this document would originated RFCs 1766 and 3066, and without whom this document would
not have been possible. not have been possible.
For this particular document, John Cowan originated the scheme For this particular document, John Cowan originated the scheme
described in Section 2.2.3. Mark Davis originated the scheme described in Section 2.3.3. Mark Davis originated the scheme
described in the Section 2.1.2. described in the Section 2.2.2.
Authors' Addresses
Addison Phillips (editor)
Quest Software
Email: addison dot phillips at quest dot com
Mark Davis (editor)
IBM
Email: mark dot davis at ibm dot com
Intellectual Property Statement Intellectual Property Statement
The IETF takes no position regarding the validity or scope of any The IETF takes no position regarding the validity or scope of any
Intellectual Property Rights or other rights that might be claimed to Intellectual Property Rights or other rights that might be claimed to
pertain to the implementation or use of the technology described in pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights this document or the extent to which any license under such rights
might or might not be available; nor does it represent that it has might or might not be available; nor does it represent that it has
made any independent effort to identify any such rights. Information made any independent effort to identify any such rights. Information
on the procedures with respect to rights in RFC documents can be on the procedures with respect to rights in RFC documents can be
 End of changes. 37 change blocks. 
148 lines changed or deleted 204 lines changed or added

This html diff was produced by rfcdiff 1.27, available from http://www.levkowetz.com/ietf/tools/rfcdiff/