draft-ietf-ltru-matching-03.txt   draft-ietf-ltru-matching-04.txt 
Network Working Group A. Phillips, Ed. Network Working Group A. Phillips, Ed.
Internet-Draft Quest Software Internet-Draft Quest Software
Expires: December 30, 2005 M. Davis, Ed. Expires: March 27, 2006 M. Davis, Ed.
IBM IBM
June 28, 2005 September 23, 2005
Matching Tags for the Identification of Languages Matching Tags for the Identification of Languages
draft-ietf-ltru-matching-03 draft-ietf-ltru-matching-04
Status of this Memo Status of this Memo
By submitting this Internet-Draft, each author represents that any By submitting this Internet-Draft, each author represents that any
applicable patent or other IPR claims of which he or she is aware applicable patent or other IPR claims of which he or she is aware
have been or will be disclosed, and any of which he or she becomes have been or will be disclosed, and any of which he or she becomes
aware will be disclosed, in accordance with Section 6 of BCP 79. aware will be disclosed, in accordance with Section 6 of BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF), its areas, and its working groups. Note that
skipping to change at page 1, line 35 skipping to change at page 1, line 35
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt. http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html. http://www.ietf.org/shadow.html.
This Internet-Draft will expire on December 30, 2005. This Internet-Draft will expire on March 27, 2006.
Copyright Notice Copyright Notice
Copyright (C) The Internet Society (2005). Copyright (C) The Internet Society (2005).
Abstract Abstract
This document describes different mechanisms for comparing, matching, This document describes different mechanisms for comparing, matching,
and evaluating language tags. Possible algorithms for language and evaluating language tags. Possible algorithms for language
negotiation and content selection are described. negotiation and content selection are described.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
2. The Language Range . . . . . . . . . . . . . . . . . . . . . . 4 2. The Language Range . . . . . . . . . . . . . . . . . . . . . . 4
2.1 Basic Language Range . . . . . . . . . . . . . . . . . . . 4 2.1 Basic Language Range . . . . . . . . . . . . . . . . . . . 4
2.1.1 Matching . . . . . . . . . . . . . . . . . . . . . . . 5 2.1.1 Matching . . . . . . . . . . . . . . . . . . . . . . . 5
2.1.2 Lookup . . . . . . . . . . . . . . . . . . . . . . . . 6 2.1.2 Lookup . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2 Extended Language Range . . . . . . . . . . . . . . . . . 6 2.2 Extended Language Range . . . . . . . . . . . . . . . . . 7
2.2.1 Extended Range Matching . . . . . . . . . . . . . . . 7 2.2.1 Extended Range Matching . . . . . . . . . . . . . . . 8
2.2.2 Extended Range Lookup . . . . . . . . . . . . . . . . 8 2.2.2 Extended Range Lookup . . . . . . . . . . . . . . . . 9
2.2.3 Scored Matching . . . . . . . . . . . . . . . . . . . 9 2.2.3 Distance Metric Scheme . . . . . . . . . . . . . . . . 10
2.3 Meaning of Language Tags and Ranges . . . . . . . . . . . 10 2.3 Meaning of Language Tags and Ranges . . . . . . . . . . . 12
2.4 Choosing Between Alternate Matching Schemes . . . . . . . 11 2.4 Choosing Between Alternate Matching Schemes . . . . . . . 13
2.5 Considerations for Private Use Subtags . . . . . . . . . . 12 2.5 Considerations for Private Use Subtags . . . . . . . . . . 14
2.6 Length Considerations in Matching . . . . . . . . . . . . 12 2.6 Length Considerations in Matching . . . . . . . . . . . . 14
3. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 14 3. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 17
4. Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 4. Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
5. Security Considerations . . . . . . . . . . . . . . . . . . . 16 5. Security Considerations . . . . . . . . . . . . . . . . . . . 19
6. Character Set Considerations . . . . . . . . . . . . . . . . . 17 6. Character Set Considerations . . . . . . . . . . . . . . . . . 20
7. References . . . . . . . . . . . . . . . . . . . . . . . . . . 18 7. References . . . . . . . . . . . . . . . . . . . . . . . . . . 21
7.1 Normative References . . . . . . . . . . . . . . . . . . . 18 7.1 Normative References . . . . . . . . . . . . . . . . . . . 21
7.2 Informative References . . . . . . . . . . . . . . . . . . 19 7.2 Informative References . . . . . . . . . . . . . . . . . . 22
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . 19 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . 23
A. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 21 A. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 24
Intellectual Property and Copyright Statements . . . . . . . . 22 Intellectual Property and Copyright Statements . . . . . . . . 25
1. Introduction 1. Introduction
Human beings on our planet have, past and present, used a number of Human beings on our planet have, past and present, used a number of
languages. There are many reasons why one would want to identify the languages. There are many reasons why one would want to identify the
language used when presenting or requesting information. language used when presenting or requesting information.
Information about a user's language preferences commonly needs to be Information about a user's language preferences commonly needs to be
identified so that appropriate processing can be applied. For identified so that appropriate processing can be applied. For
example, the user's language preferences in a browser can be used to example, the user's language preferences in a browser can be used to
select web pages appropriately. A choice of language preference can select web pages appropriately. A choice of language preference can
also be used to select among tools (such as dictionaries) to assist also be used to select among tools (such as dictionaries) to assist
in the processing or understanding of content in different languages. in the processing or understanding of content in different languages.
Given a set of language identifiers, such as those defined in Given a set of language identifiers, such as those defined in [draft-
[ID.ietf-ltru-registry], various mechanisms can be envisioned for registry], various mechanisms can be envisioned for performing
performing language negotiation and tag matching. The suitability of language negotiation and tag matching. The suitability of a
a particular mechanism to a particular application depends on the particular mechanism to a particular application depends on the needs
needs of that application. of that application.
This document defines language ranges and syntax for specifying user This document defines several mechanisms for matching and filtering
preferences in a request for language content. It also specifies natural language content identified using Language Tags [draft-
various schemes and mechanisms that can be used with language ranges registry]. It also defines the syntax (called a "language range")
when matching or filtering content based on language tags. associated with each of these mechanisms for specifying user language
preferences.
The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in [RFC2119]. document are to be interpreted as described in [RFC2119].
2. The Language Range 2. The Language Range
Language Tags are used to identify the language of some information Language Tags [draft-registry] are used to identify the language of
item or content. Applications that use language tags are often faced some information item or content. Applications that use language
with the problem of identifying sets of content that share certain tags are often faced with the problem of identifying sets of content
language attributes. For example, HTTP 1.1 [RFC2616] describes that share certain language attributes. For example, HTTP 1.1
language ranges in its discussion of the Accept-Language header [RFC2616] describes language ranges in its discussion of the Accept-
(Section 14.4), which is used for selecting content from servers Language header (Section 14.4), which is used for selecting content
based on the language of that content. from servers based on the language of that content.
When selecting content according to its language, it is useful to When selecting content according to its language, it is useful to
have a mechanism for identifying sets of language tags that share have a mechanism for identifying sets of language tags that share
specific attributes. This allows users to select or filter content specific attributes. This allows users to select or filter content
based on specific requirements. Such an identifier is called a based on specific requirements. Such an identifier is called a
"Language Range". "Language Range".
2.1 Basic Language Range 2.1 Basic Language Range
A basic language range (such as described in [RFC3066] and HTTP 1.1 A basic language range identifies the set of content whose language
[RFC2616]) is a set of languages whose tags all begin with the same tags begin with the same sequence of subtags. Basic language ranges
sequence of subtags. A basic language range can be represented by a are described in [RFC3066] and HTTP 1.1 [RFC2616] (where they are
'language-range' tag, by using the definition from HTTP/1.1 [RFC2616] referred to as simply a "language range"). A basic language range is
: identified by its 'language-range' tag, by adapting the
ABNF[RFC2234bis] from HTTP/1.1 [RFC2616] :
language-range = language-tag / "*" language-range = language-tag / "*"
language-tag = 1*8[alphanum] *["-" 1*8alphanum]
alphanum = ALPHA / DIGIT
That is, a language-range has the same syntax as a language-tag or is That is, a language-range has the same syntax as a language-tag or is
the single character "*". This definition of language-range implies the single character "*". Basic Language Ranges imply that there is
that there is a semantic relationship between tags that share the a semantic relationship between language tags that share the same
same prefix. prefix. While this is often the case, it is not always true.
In particular, the set of language tags that match a specific In particular, the set of language tags that match a specific
language-range might not all be mutually intelligible. The use of a language-range might not all be mutually intelligible. Matching a
prefix when matching tags to language ranges does not imply that language-range (prefix) to various language tags does not mean that
language tags are assigned to languages in such a way that it is it is always true that if a user understands a language identified by
always true that if a user understands a language with a certain tag, a certain tag, then this user will also understand all of the
then this user will also understand all languages with tags for which languages for which this tag is a prefix. The use of prefixes (and
this tag is a prefix. The prefix rule simply allows the use of thus basic language ranges) simply allows the use of a prefix if this
prefix tags if this is the case. is the case.
When working with tags and ranges you SHOULD also note the following: Language tags and thus language ranges are to be treated as case
insensitive: there exist conventions for the capitalization of some
of the subtags, but these MUST NOT be taken to carry meaning.
Matching of language tags to language ranges MUST be done in a case
insensitive manner.
When working with tags and ranges users SHOULD also note the
following:
1. Private-use and Extension subtags are normally orthogonal to 1. Private-use and Extension subtags are normally orthogonal to
language tag fallback. Implementations SHOULD ignore language tag fallback. Implementations SHOULD ignore
unrecognized private-use and extension subtags when performing unrecognized private-use and extension subtags when performing
language tag fallback. Since these subtags are always at the end language tag fallback. Since these subtags are always at the end
of the sequence of subtags, they don't normally interfere with of the sequence of subtags, they don't normally interfere with
the use of prefixes for matching in the schemes described below. the use of prefixes for matching in the schemes described below.
2. Implementations that choose not to interpret one or more private- 2. Implementations that choose not to interpret one or more private-
use or extension subtags SHOULD NOT remove or modify these use or extension subtags SHOULD NOT remove or modify these
skipping to change at page 5, line 23 skipping to change at page 5, line 33
on the other end of the protocol would make use of that on the other end of the protocol would make use of that
information. information.
3. Some applications of language tags might want or need to consider 3. Some applications of language tags might want or need to consider
extensions and private-use subtags when matching tags. If extensions and private-use subtags when matching tags. If
extensions and private-use subtags are included in a matching or extensions and private-use subtags are included in a matching or
filtering process that utilizes the one of the schemes described filtering process that utilizes the one of the schemes described
in this document, then the implementation SHOULD canonicalize the in this document, then the implementation SHOULD canonicalize the
language tags and/or ranges before performing the matching. Note language tags and/or ranges before performing the matching. Note
that language tag processors that claim to be "well-formed" that language tag processors that claim to be "well-formed"
processors as defined in [ID.ietf-ltru-registry] generally fall processors as defined in [draft-registry] generally fall into
into this category. this category.
There are two matching schemes that are commonly associated with There are two matching schemes that are commonly associated with
basic language ranges: matching and lookup. basic language ranges: matching and lookup.
Note that neither matching nor lookup using basic language ranges
attempt to process the semantics of the tags or ranges in any way.
The language tag and language range are compared in a case
insensitive manner using basic string processing. The choice of
subtags in both the language tag and language range may affect the
results produced as a result.
2.1.1 Matching 2.1.1 Matching
Language tag matching is used to select all content that matches a Language tag matching is used to select all content that matches a
given prefix. In matching, the language range represents the least given prefix. In matching, the language range represents the least
specific tag which is an acceptable match and every piece of content specific tag which is an acceptable match and every piece of content
that matches is returned. that matches is returned.
For example, if an application is applying a style to all content in For example, if an application is applying a style to all content in
a web page in a particular language, it might use language tag a document in a particular language, it might use language tag
matching to select the content to which the style is applied. matching to select the content to which the style is applied.
A language-range matches a language-tag if it exactly equals the tag, A language-range matches a language-tag if it exactly equals the tag,
or if it exactly equals a prefix of the tag such that the first or if it exactly equals a prefix of the tag such that the first
character following the prefix is "-". (That is, the language-range character following the prefix is "-". (That is, the language-range
"de-de" matches the language tag "de-DE-1996", but not the language "de-de" matches the language tag "de-DE-1996", but not the language
tag "de-Deva".) tag "de-Deva".)
The special range "*" matches any tag. A protocol which uses The special range "*" matches any tag. A protocol which uses
language ranges MAY specify additional rules about the semantics of language ranges MAY specify additional rules about the semantics of
"*"; for instance, HTTP/1.1 specifies that the range "*" matches only "*"; for instance, HTTP/1.1 specifies that the range "*" matches only
languages not matched by any other range within an "Accept-Language:" languages not matched by any other range within an "Accept-Language"
header. header.
2.1.2 Lookup 2.1.2 Lookup
Content lookup is used to select the single information item that Content lookup is used to select the single information item that
best matches the language range for a given request. In lookup, the best matches the language range for a given request. In lookup, the
language range represents the most specific tag which is an language range represents the most specific tag which is an
acceptable match and only the closest matching item is returned. acceptable match and only the closest matching item is returned.
For example, if an application inserts some dynamic content into a For example, if an application inserts some dynamic content into a
web page, returning an empty string if there is no exact match is not document, returning an empty string if there is no exact match is not
an option. Instead, the application "falls back". an option. Instead, the application "falls back" until it finds a
suitable piece of content to insert.
When performing lookup, the language range is progressively truncated When performing lookup, the language range is progressively truncated
from the end until a matching piece of content is located. For from the end until a matching piece of content is located. For
example, starting with the range "zh-Hant-CN-x-wadegile", the lookup example, starting with the range "zh-Hant-CN-x-wadegile", the lookup
would progressively search for content as shown below: would progressively search for content as shown below:
Range to match: zh-Hant-CN-x-wadegile Range to match: zh-Hant-CN-x-wadegile
1. zh-Hant-CN-x-wadegile 1. zh-Hant-CN-x-wadegile
2. zh-Hant-CN 2. zh-Hant-CN
3. zh-Hant 3. zh-Hant
skipping to change at page 7, line 4 skipping to change at page 7, line 14
2.2 Extended Language Range 2.2 Extended Language Range
Prefix matching using a Basic Language Range, as described above, is Prefix matching using a Basic Language Range, as described above, is
not always the most appropriate way to access the information not always the most appropriate way to access the information
contained in language tags when selecting or filtering content. Some contained in language tags when selecting or filtering content. Some
applications might wish to define a more granular matching scheme and applications might wish to define a more granular matching scheme and
such a matching scheme requires the ability to specify the various such a matching scheme requires the ability to specify the various
attributes of a language tag in the language range. An extended attributes of a language tag in the language range. An extended
language range can be represented by the following ABNF: language range can be represented by the following ABNF:
extended-language-range = range ; a range
/ privateuse ; private use tag
/ grandfathered ; grandfathered registrations
extended-language-range = grandfathered / privateuse / range range = (language
range = ( lang [ "-" script ] [ "-" region ] *( "-" variant ) ["-" script]
["-" region]
*("-" variant)
*("-" extension)
[ "-" privateuse ] ) [ "-" privateuse ] )
lang = 2*8ALPHA / extlang / "*"
extlang = 2*3ALPHA *2("-" 3ALPHA) ( "-" ( 3ALPHA / "*" ) ) language = (2*3ALPHA [ extlang ]) ; shortest ISO 639 code
script = 4ALPHA / "*" / 4ALPHA ; reserved for future use
region = 2ALPHA / 3DIGIT / "*" / 5*8ALPHA ; registered language subtag
variant = 5*8alphanum / ( DIGIT 3alphanum ) / "*" / "*" ; ... or wildcard
extlang = *2("-" 3ALPHA) ("-" ( 3ALPHA / "*"))
; reserved for future use
; wildcard can only appear
; at the end
script = 4ALPHA ; ISO 15924 code
/ "*" ; or wildcard
region = 2ALPHA ; ISO 3166 code
/ 3DIGIT ; UN M.49 code
/ "*" ; ... or wildcard
variant = 5*8alphanum ; registered variants
/ (DIGIT 3alphanum) ;
/ "*" ; ... or wildcard
extension = singleton *("-" (2*8alphanum)) [ "-*" ]
; extension subtags
; wildcard can only appear
; at the end
singleton = %x41-57 / %x59-5A / %x61-77 / %x79-7A / DIGIT
; "a"-"w" / "y"-"z" / "A"-"W" / "Y"-"Z" / "0"-"9"
; Single letters: x/X is reserved for private use
privateuse = ( "x" / "X" ) 1*( "-" ( 1*8alphanum ) ) privateuse = ( "x" / "X" ) 1*( "-" ( 1*8alphanum ) )
grandfathered = 1*3ALPHA 1*2( "-" ( 2*8alphanum ) ) grandfathered = 1*3ALPHA 1*2( "-" ( 2*8alphanum ) )
alphanum = ( ALPHA / DIGIT ) ; grandfathered registration
; Note: i is the only singleton
; that starts a grandfathered tag
alphanum = (ALPHA / DIGIT) ; letters and numbers
In an extended language range, the identifier takes the form of a In an extended language range, the identifier takes the form of a
series of subtags which must consist of well-formed subtags or the series of subtags which must consist of well-formed subtags or the
special subtag "*". For example, the language range "en-*-US" special subtag "*". For example, the language range "en-*-US"
specifies a primary language of 'en', followed by any script subtag, specifies a primary language of 'en', followed by any script subtag,
followed by the region subtag 'US'. followed by the region subtag 'US'.
A field not present in the middle of an extended language range MAY A field not present in the middle of an extended language range MAY
be treated as if the field contained a "*". For example, the range be treated as if the field contained a "*". For example, the range
"en-US" MAY be considered to be equivalent to the range "en-*-US". "en-US" MAY be considered to be equivalent to the range "en-*-US".
This also means that multiple wildcards can be collapsed (so that
"en-*-*-US" is equivalent to "en-*-US").
There are several matching algorithms or schemes which can be applied There are several matching algorithms or schemes which can be applied
when matching extended language ranges to language tags. when matching extended language ranges to language tags.
2.2.1 Extended Range Matching 2.2.1 Extended Range Matching
In extended range matching, the subtags in a language tag are In extended range matching, the subtags in a language tag are
compared to the corresponding subtags in the extended language range. compared to the corresponding subtags in the extended language range.
A subtag is considered to match if it exactly matches the A subtag is considered to match if it exactly matches the
corresponding subtag in the range or the range contains a subtag with corresponding subtag in the range or the range contains a subtag with
skipping to change at page 8, line 15 skipping to change at page 9, line 16
zh-Hant-CN zh-Hant-CN
zh-CN zh-CN
zh-Hans-CN zh-Hans-CN
zh-CN-x-wadegile zh-CN-x-wadegile
zh-Latn-CN-boont zh-Latn-CN-boont
zh-cmn-Hans-CN-x-wadegile
2.2.2 Extended Range Lookup 2.2.2 Extended Range Lookup
In extended range lookup, the subtags in a language tag are compared In extended range lookup, the subtags in a language tag are compared
to the corresponding subtags in the extended language range. The to the corresponding subtags in the extended language range. The
subtag is considered to match if it exactly matches the corresponding subtag is considered to match if it exactly matches the corresponding
subtag in the range or the range contains a subtag with the value "*" subtag in the range or the range contains a subtag with the value "*"
(which matches all subtags, including the empty subtag). Extended (which matches all subtags, including the empty subtag). Extended
language range lookup is an extension of basic lookup language range lookup is an extension of basic lookup
(Section 2.1.2): the language range represents the most specific tag (Section 2.1.2): the language range represents the most specific tag
which will form an acceptable match. which will form an acceptable match.
skipping to change at page 9, line 35 skipping to change at page 10, line 40
en-Latn-US-scouse en-Latn-US-scouse
en-US en-US
en-scouse en-scouse
Note that the ability to be specific in extended range lookup can Note that the ability to be specific in extended range lookup can
make this matching scheme a more appropriate replacement for basic make this matching scheme a more appropriate replacement for basic
matching than the extended range matching scheme. matching than the extended range matching scheme.
2.2.3 Scored Matching 2.2.3 Distance Metric Scheme
In the "scored matching" scheme, the extended language range and the Both Basic and Extended Language Ranges produce simple boolean
language tags are pre-normalized by mapping grandfathered and matches. Some applications may benefit by providing an array of
obsolete tags into modern equivalents. results with different levels of matching, for example, sorting
results based on the overall "quality" of the match.
The language range and the language tags are normalized into This type of matching is sometimes called a "distance metric". A
quadruples of the form (language, script, country, variant), where distance metric assigns a pair of language tags a numeric value
extended language is considered part of language and x-private-codes representing the 'distance' between the two. A distance of zero
are considered part of the language if they are initial and part of means that they are identical, a small distance indicates that they
the variant if not initial. Missing components are set to "*". An are very similar, and a large distance indicated that they are very
"*" pattern becomes the quadruple ("*", "*", "*", "*"). different. Using a distance metric, implementations can, for
example, allow users to select a threshold distance for a match to be
successful or a filter to be applied.
Each language tag being matched or filtered is assigned a "quality The first step in the process is to normalize the extended language
value" such that higher values indicate better matches and lower range and the language tags to be matched to it by canonicalizing
values indicate worse ones. If the language matches, add 8 to the them, mapping grandfathered and obsolete tags into modern
quality value. If the script matches, add 4 to the quality value. equivalents.
If the region matches, add 2 to the quality value. If the variant The language range and the language tags are then transformed into
matches, add 1 to the quality value. Elements of the quadruples are quintuples of elements of the form (language, script, country,
considered to match if they are the same or if one of them is "*". variant, extension). Any extended language subtags are considered
part of the language element; private use subtag sequences are
considered part of the language element if in the initial position in
the tag and part of the variant element if not. Language subtags
'und', 'mul', and the script subtag 'Zyyy' are converted to "*".
A value of 15 is a perfect match; 0 is no match at all. Different Missing components in the language-tag are set to "*"; thus a "*"
values could be more or less appropriate for different applications pattern becomes the quintuple ("*", "*", "*", "*", "*"). Missing
and implementations SHOULD probably allow users to choose the most components in the extended language-range are handled similarly to
appropriate selection value. extended range lookup: missing internal subtags are expanded to "*".
Missing end subtags are expanded as the empty string. Thus a pattern
"en-US" becomes the quintuple ("en","*","US","","").
Here are some examples of language-tags and their quintuples:
en-US ("en","*","US","*","*")
sr-Latn ("sr,"Latn","*","*","*")
zh-cmn-Hant ("zh-cmn","Hant","*","*","*")
x-foo ("x-foo","*","*","*","*")
en-x-foo ("en","*","*","x-foo","*")
i-default ("i-default","*","*","*","*")
sl-Latn-IT-roazj ("sl","Latn","IT","rozaj","*")
zh-r-wadegile ("zh","*","*","*","r-wadegile") // hypothetical
Each language-range/language-tag pair being matched or filtered is
assigned a distance value, whereby small values indicate better
matches and large values indicate worse ones. The distance between
the pair is the sum of the distances for each of the corresponding
elements of the quintuple. If the elements are identical or one is
'*', then the distance value between them is zero. Otherwise, it is
given by the following table:
256 language mismatch
128 script mismatch
32 region mismatch
4 variant mismatch
1 extension mismatch
A value of 0 is a perfect match; 421 is no match at all. Different
threshold values might be appropriate for different applications and
implementations will probably allow users to choose the most
appropriate selection value, ranking the selections based on score.
Examples of various tag's distances from the range "en-US":
"fr" 256 (language mismatch, region match)
"en-GB" 384 (language, region mismatch)
"en-Latn-US" 0 (all fields match)
"en-Brai" 32 (region mismatch)
"en-US-x-foo" 4 (variant mismatch: range is the empty string)
"en-US-r-wadegile" 1 (extension mismatch: range is the empty string)
Implementations may want to use more sophisticated weights that
depend on the values of the corresponding elements. For example,
depending on the domain, an implemenation might give a small distance
to the difference between the language subtag 'no' and the closely
related language subtags 'nb' or 'nn'; or between the script subtags
'Kata' and 'Hira'; or between the region subtags 'US' and 'UM'.
2.3 Meaning of Language Tags and Ranges 2.3 Meaning of Language Tags and Ranges
A language tag defines a language as spoken (or written, signed or A language tag defines a language as spoken (or written, signed or
otherwise signaled) by human beings for communication of information otherwise signaled) by human beings for communication of information
to other human beings. to other human beings.
If a language tag B contains language tag A as a prefix, then B is If a language tag B contains language tag A as a prefix, then B is
typically "narrower" or "more specific" than A. For example, "zh- typically "narrower" or "more specific" than A. For example, "zh-
Hant-TW" is more specific than "zh-Hant". Hant-TW" is more specific than "zh-Hant".
skipping to change at page 11, line 28 skipping to change at page 13, line 46
Norwegian document; the Norwegian-speaking user could then access Norwegian document; the Norwegian-speaking user could then access
a French-Norwegian dictionary to find out what the marked section a French-Norwegian dictionary to find out what the marked section
meant. If the user were listening to that document through a meant. If the user were listening to that document through a
speech synthesis interface, this formation could be used to signal speech synthesis interface, this formation could be used to signal
the synthesizer to appropriately apply French text-to-speech the synthesizer to appropriately apply French text-to-speech
pronunciation rules to that span of text, instead of misapplying pronunciation rules to that span of text, instead of misapplying
the Norwegian rules. the Norwegian rules.
2.4 Choosing Between Alternate Matching Schemes 2.4 Choosing Between Alternate Matching Schemes
Implementations MAY choose to implement different styles of matching Implementations can choose to implement different styles of matching
for different kinds of processing. For example, an implementation for different kinds of processing. For example, an implementation
could treat an absent script subtag as a "wildcard" field; thus could treat an absent script subtag as a "wildcard" field; thus
"az-AZ" would match "az-AZ", "az-Cyrl-AZ", "az-Latn-AZ", etc. but not "az-AZ" would match "az-AZ", "az-Cyrl-AZ", "az-Latn-AZ", etc. but not
"az" (this is extended range lookup). If one item is to be chosen, "az" (this is an example of extended range lookup). If one item is
the implementation could pick among those matches based on other to be chosen, the implementation could pick among those matches based
information, such as the most likely script used in the language/ on other information, such as the most likely script used in the
region in question or the script used by other content selected. language/region in question or the script used by other content
selected.
Because the primary language subtag cannot be absent in a language Because the primary language subtag cannot be absent in a language
tag, the 'UND' subtag is sometimes be used as a 'wildcard' in basic tag, the 'und' subtag is sometimes be used as a 'wildcard' in basic
matching. For example, in a query where you want to select all matching. For example, in a query where you want to select all
language tags that contain 'Latn' as the script code and 'AZ' as the language tags that contain 'Latn' as the script code and 'AZ' as the
region code, you could use the range "und-Latn-AZ". This requires an region code, you could use the range "und-Latn-AZ". This requires an
implementation to examine the actual values of the subtags, though. implementation to examine the actual values of the subtags, though,
and users SHOULD NOT assume that the value 'und' will be treated as a
wildcard.
The matching schemes described elsewhere in this document are The matching schemes described elsewhere in this document are
designed such that implementations do not have to examine the values designed such that implementations do not have to examine the values
or subtags supplied and, except for scored matching, they do not need of the subtags supplied and, except for scored matching, they do not
access to the Language Subtag Registry nor the use of valid subtags need access to the Language Subtag Registry nor do they require the
in language tags or ranges. This has great benefit for speed and use of valid subtags in language tags or ranges. This has great
simplicity of implementation. benefit for speed and simplicity of implementation.
Implementations might also wish to use semantic information external Implementations might also wish to use semantic information external
to the langauge tags when performing fallback. For example, the to the langauge tags when performing fallback. For example, the
primary language subtags 'nn' (Nynorsk Norwegian) and 'nb' (Bokmal primary language subtags 'nn' (Nynorsk Norwegian) and 'nb' (Bokmal
Norwegian) might both be usefully matched to the more general subtag Norwegian) might both be usefully matched to the more general subtag
'no' (Norwegian). Or an application might infer that content labeled 'no' (Norwegian). Or an application might infer that content labeled
"zh-CN" is morely likely to match the range "zh-Hans" than equivalent "zh-CN" is morely likely to match the range "zh-Hans" than equivalent
content labeled "zh-TW". content labeled "zh-TW".
2.5 Considerations for Private Use Subtags 2.5 Considerations for Private Use Subtags
skipping to change at page 12, line 25 skipping to change at page 14, line 47
intended for general use. Private-use subtags are simply useless for intended for general use. Private-use subtags are simply useless for
information exchange without prior arrangement. information exchange without prior arrangement.
The value and semantic meaning of private-use tags and of the subtags The value and semantic meaning of private-use tags and of the subtags
used within such a language tag are not defined. Matching private used within such a language tag are not defined. Matching private
use tags using language ranges or extended language ranges can result use tags using language ranges or extended language ranges can result
in unpredictable content being returned. in unpredictable content being returned.
2.6 Length Considerations in Matching 2.6 Length Considerations in Matching
[RFC3066] did not provide an upper limit on the size of language tags RFC 3066 [RFC3066] did not provide an upper limit on the size of
or ranges. RFC 3066 did define the semantics of particular subtags language tags or ranges. RFC 3066 did define the semantics of
in such a way that most language tags or ranges consisted of language particular subtags in such a way that most language tags or ranges
and region subtags with a combined total length of up to six consisted of language and region subtags with a combined total length
characters. Larger tags and ranges (in terms of both subtags and of up to six characters. Larger tags and ranges (in terms of both
characters) did exist, however. subtags and characters) did exist, however.
[ID.ietf-ltru-registry] also does not impose a fixed upper limit on [draft-registry] also does not impose a fixed upper limit on the
the number of subtags in a language tag or range (and thus an upper number of subtags in a language tag or range (and thus an upper bound
bound on the size of either). The syntax in that document suggests on the size of either). The syntax in that document suggests that,
that, depending on the specific language or range of languages, more depending on the specific language or range of languages, more
subtags (and thus characters) are sometimes necessary as a result. subtags (and thus characters) are sometimes necessary as a result.
Length considerations and their impact on the selection and Length considerations and their impact on the selection and
processing of tags are described in Section 2.1.1 of that document. processing of tags are described in Section 2.1.1 of that document.
A matching implementation MAY choose to limit the length of the A matching implementation MAY choose to limit the length of the
language tags or ranges used in matching. Any such limitation SHOULD language tags or ranges used in matching. Any such limitation SHOULD
be clearly documented, and such documentation SHOULD include the be clearly documented, and such documentation SHOULD include the
disposition of any longer tags or ranges (for example, whether an disposition of any longer tags or ranges (for example, whether an
error value is generated or the language tag or range is truncated). error value is generated or the language tag or range is truncated).
If truncation is permitted it MUST NOT permit a subtag to be divided, If truncation is permitted it MUST NOT permit a subtag to be divided,
skipping to change at page 13, line 19 skipping to change at page 15, line 41
In practice, most tags do not require additional subtags or In practice, most tags do not require additional subtags or
substantially more characters. Additional subtags sometimes add substantially more characters. Additional subtags sometimes add
useful distinguishing information, but extraneous subtags interfere useful distinguishing information, but extraneous subtags interfere
with the meaning, understanding, and especially matching of language with the meaning, understanding, and especially matching of language
tags. Since language tags or ranges MAY be truncated by an tags. Since language tags or ranges MAY be truncated by an
application or protocol that limits storage, when choosing language application or protocol that limits storage, when choosing language
tags or ranges users and applications SHOULD avoid adding subtags tags or ranges users and applications SHOULD avoid adding subtags
that add no distinguishing value. In particular, users and that add no distinguishing value. In particular, users and
implementations SHOULD follow the 'Prefix' and 'Suppress-Script' implementations SHOULD follow the 'Prefix' and 'Suppress-Script'
fields in the registry (defined in Section 3.6 of [ID.ietf-ltru- fields in the registry (defined in Section 3.6 of [draft-registry]):
registry]): these fields provide guidance on when specific additional these fields provide guidance on when specific additional subtags
subtags SHOULD (and SHOULD NOT) be used. SHOULD (and SHOULD NOT) be used.
Implementations MUST support a limit of at least 33 characters. This Implementations MUST support a limit of at least 33 characters. This
limit includes at least one subtag of each non-extension, non-private limit includes at least one subtag of each non-extension, non-private
use type. When choosing a buffer limit, a length of at least 42 use type. When choosing a buffer limit, a length of at least 42
characters is strongly RECOMMENDED. characters is strongly RECOMMENDED.
The practical limit on tags or ranges derived solely from registered The practical limit on tags or ranges derived solely from registered
values is 42 characters. Implementations MUST be able to handle tags values is 42 characters. Implementations MUST be able to handle tags
and ranges of this length. Support for tags and ranges of at least and ranges of this length. Support for tags and ranges of at least
62 characters in length is RECOMMENDED. Implementations MAY support 62 characters in length is RECOMMENDED. Implementations MAY support
skipping to change at page 13, line 50 skipping to change at page 16, line 24
example: example:
Tag to truncate: zh-Hant-CN-variant1-a-extend1-x-wadegile-private1 Tag to truncate: zh-Hant-CN-variant1-a-extend1-x-wadegile-private1
1. zh-Hant-CN-variant1-a-extend1-x-wadegile 1. zh-Hant-CN-variant1-a-extend1-x-wadegile
2. zh-Hant-CN-variant1-a-extend1 2. zh-Hant-CN-variant1-a-extend1
3. zh-Hant-CN-variant1 3. zh-Hant-CN-variant1
4. zh-Hant-CN 4. zh-Hant-CN
5. zh-Hant 5. zh-Hant
6. zh 6. zh
Figure 4: Example of Tag Truncation Figure 6: Example of Tag Truncation
3. IANA Considerations 3. IANA Considerations
This document presents no new or existing considerations for IANA. This document presents no new or existing considerations for IANA.
4. Changes 4. Changes
This is the first version of this document. This is the first version of this document.
The following changes were put into this document since draft-02: The following changes were put into this document since draft-03:
Turned on symrefs and replaced all reference IDs to make them Modified the ABNF to match changes in [draft-registry]
readable (F.Ellermann) (K.Karlsson)
Removed all external references from the abstract (R.Presuhn) Matched the references and reference formats to [draft-registry]
(K.Karlsson)
Various edits, additions, and emendations to deal with changes in
the Last Call of draft-registry as well as cleaning up the text.
5. Security Considerations 5. Security Considerations
Language ranges used in content negotiation might be used to infer Language ranges used in content negotiation might be used to infer
the nationality of the sender, and thus identify potential targets the nationality of the sender, and thus identify potential targets
for surveillance. In addition, unique or highly unusual language for surveillance. In addition, unique or highly unusual language
ranges or combinations of language ranges might be used to track ranges or combinations of language ranges might be used to track
specific individual's activities. specific individual's activities.
This is a special case of the general problem that anything you send This is a special case of the general problem that anything you send
skipping to change at page 18, line 9 skipping to change at page 21, line 9
The syntax of language tags and language ranges permit only the The syntax of language tags and language ranges permit only the
characters A-Z, a-z, 0-9, and HYPHEN-MINUS (%x2D). These characters characters A-Z, a-z, 0-9, and HYPHEN-MINUS (%x2D). These characters
are present in most character sets, so presentation of language tags are present in most character sets, so presentation of language tags
should not present any character set issues. should not present any character set issues.
7. References 7. References
7.1 Normative References 7.1 Normative References
[ID.ietf-ltru-registry] [ID.ietf-ltru-initial]
Phillips, A., Ed. and M. Davis, Ed., "Tags for the Ewell, D., Ed., "Language Tags Initial Registry (work in
Identification of Languages (Internet-Draft)", June 2005, progress)", August 2005, <http://www.ietf.org/
<http://www.ietf.org/internet-drafts/ internet-drafts/draft-ietf-ltru-initial-04.txt>.
draft-ietf-ltru-registry-07.txt>.
[RFC1327] Hardcastle-Kille, S., "Mapping between X.400(1988) / ISO [RFC1327] Hardcastle-Kille, S., "Mapping between X.400(1988) / ISO
10021 and RFC 822", RFC 1327, May 1992. 10021 and RFC 822", RFC 1327, May 1992.
[RFC1521] Borenstein, N. and N. Freed, "MIME (Multipurpose Internet [RFC1521] Borenstein, N. and N. Freed, "MIME (Multipurpose Internet
Mail Extensions) Part One: Mechanisms for Specifying and Mail Extensions) Part One: Mechanisms for Specifying and
Describing the Format of Internet Message Bodies", Describing the Format of Internet Message Bodies",
RFC 1521, September 1993. RFC 1521, September 1993.
[RFC2028] Hovey, R. and S. Bradner, "The Organizations Involved in [RFC2028] Hovey, R. and S. Bradner, "The Organizations Involved in
the IETF Standards Process", BCP 11, RFC 2028, the IETF Standards Process", BCP 11, RFC 2028,
October 1996. October 1996.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997. Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC2231] Freed, N. and K. Moore, "MIME Parameter Value and Encoded [RFC2231] Freed, N. and K. Moore, "MIME Parameter Value and Encoded
Word Extensions: Character Sets, Languages, and Word Extensions: Character Sets, Languages, and
Continuations", RFC 2231, November 1997. Continuations", RFC 2231, November 1997.
[RFC2234] Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax [RFC2234bis]
Specifications: ABNF", RFC 2234, November 1997. Crocker, D. and P. Overell, "Augmented BNF for Syntax
Specifications: ABNF", draft-crocker-abnf-rfc2234bis-00
(work in progress), March 2005.
[RFC2396] Berners-Lee, T., Fielding, R., and L. Masinter, "Uniform [RFC2396] Berners-Lee, T., Fielding, R., and L. Masinter, "Uniform
Resource Identifiers (URI): Generic Syntax", RFC 2396, Resource Identifiers (URI): Generic Syntax", RFC 2396,
August 1998. August 1998.
[RFC2434] Narten, T. and H. Alvestrand, "Guidelines for Writing an [RFC2434] Narten, T. and H. Alvestrand, "Guidelines for Writing an
IANA Considerations Section in RFCs", BCP 26, RFC 2434, IANA Considerations Section in RFCs", BCP 26, RFC 2434,
October 1998. October 1998.
[RFC2616] Fielding, R., Gettys, J., Mogul, J., Frystyk, H., [RFC2616] Fielding, R., Gettys, J., Mogul, J., Frystyk, H.,
Masinter, L., Leach, P., and T. Berners-Lee, "Hypertext Masinter, L., Leach, P., and T. Berners-Lee, "Hypertext
Transfer Protocol -- HTTP/1.1", RFC 2616, June 1999. Transfer Protocol -- HTTP/1.1", RFC 2616, June 1999.
[RFC2860] Carpenter, B., Baker, F., and M. Roberts, "Memorandum of [RFC2860] Carpenter, B., Baker, F., and M. Roberts, "Memorandum of
Understanding Concerning the Technical Work of the Understanding Concerning the Technical Work of the
Internet Assigned Numbers Authority", RFC 2860, June 2000. Internet Assigned Numbers Authority", RFC 2860, June 2000.
[RFC3629] Yergeau, F., "UTF-8, a transformation format of ISO [RFC3629] Yergeau, F., "UTF-8, a transformation format of ISO
10646", STD 63, RFC 3629, November 2003. 10646", STD 63, RFC 3629, November 2003.
7.2 Informative References [draft-registry]
Phillips, A., Ed. and M. Davis, Ed., "Tags for the
[ISO639-1] Identification of Languages (work in progress)",
International Organization for Standardization, "ISO 639- August 2005, <http://www.ietf.org/internet-drafts/
1:2002, Codes for the representation of names of languages draft-ietf-ltru-registry-12.txt>.
-- Part 1: Alpha-2 code", ISO Standard 639, 2002.
[ISO639-2] 7.2 Informative References
International Organization for Standardization, "ISO 639-
2:1998 - Codes for the representation of names of
languages -- Part 2: Alpha-3 code - edition 1",
August 1988.
[ISO15924] [ISO15924]
ISO TC46/WG3, "ISO 15924:2003 (E/F) - Codes for the "ISO 15924:2004. Information and documentation -- Codes
representation of names of scripts", January 2004. for the representation of names of scripts", January 2004.
[ISO3166] International Organization for Standardization, "Codes for [ISO3166-1]
the representation of names of countries, 3rd edition", "ISO 3166-1:1997. Codes for the representation of names of
ISO Standard 3166, August 1988. countries and their subdivisions -- Part 1: Country
codes", 1997.
[UN_M49] Statistical Division, United Nations, "Standard Country or [ISO639-1]
Area Codes for Statistical Use", UN Standard Country or "ISO 639-1:2002. Codes for the representation of names of
Area Codes for Statistical Use, Revision 4 (United Nations languages -- Part 1: Alpha-2 code", 2002.
publication, Sales No. 98.XVII.9, June 1999.
[ISO639-2]
"ISO 639-2:1998. Codes for the representation of names of
languages -- Part 2: Alpha-3 code, first edition", 1998.
[RFC1766] Alvestrand, H., "Tags for the Identification of [RFC1766] Alvestrand, H., "Tags for the Identification of
Languages", RFC 1766, March 1995. Languages", RFC 1766, March 1995.
[RFC3066] Alvestrand, H., "Tags for the Identification of [RFC3066] Alvestrand, H., "Tags for the Identification of
Languages", BCP 47, RFC 3066, January 2001. Languages", BCP 47, RFC 3066, January 2001.
[RFC3339] Klyne, G. and C. Newman, "Date and Time on the Internet: [RFC3339] Klyne, G. and C. Newman, "Date and Time on the Internet:
Timestamps", RFC 3339, July 2002. Timestamps", RFC 3339, July 2002.
[UN_M.49] Statistics Division, United Nations, "Standard Country or
Area Codes for Statistical Use", UN Standard Country or
Area Codes for Statistical Use, Revision 4 (United Nations
publication, Sales No. 98.XVII.9, June 1999.
Authors' Addresses Authors' Addresses
Addison Phillips (editor) Addison Phillips (editor)
Quest Software Quest Software
Email: addison dot phillips at quest dot com Email: addison dot phillips at quest dot com
Mark Davis (editor) Mark Davis (editor)
IBM IBM
Email: mark dot davis at ibm dot com Email: mark dot davis at ibm dot com
Appendix A. Acknowledgements Appendix A. Acknowledgements
Any list of contributors is bound to be incomplete; please regard the Any list of contributors is bound to be incomplete; please regard the
following as only a selection from the group of people who have following as only a selection from the group of people who have
contributed to make this document what it is today. contributed to make this document what it is today.
skipping to change at page 21, line 11 skipping to change at page 24, line 11
IBM IBM
Email: mark dot davis at ibm dot com Email: mark dot davis at ibm dot com
Appendix A. Acknowledgements Appendix A. Acknowledgements
Any list of contributors is bound to be incomplete; please regard the Any list of contributors is bound to be incomplete; please regard the
following as only a selection from the group of people who have following as only a selection from the group of people who have
contributed to make this document what it is today. contributed to make this document what it is today.
The contributors to [ID.ietf-ltru-registry], [RFC3066] and [RFC1766], The contributors to [draft-registry], [RFC3066] and [RFC1766], each
each of which is a precursor to this document, made enormous of which is a precursor to this document, made enormous
contributions directly or indirectly to this document and are contributions directly or indirectly to this document and are
generally responsible for the success of language tags. generally responsible for the success of language tags.
The following people (in alphabetical order by family name) The following people (in alphabetical order by family name)
contributed to this document: contributed to this document:
Jeremy Carroll, John Cowan, Frank Ellermann, Doug Ewell, Ira Jeremy Carroll, John Cowan, Frank Ellermann, Doug Ewell, Kent
McDonald, M. Patton, Randy Presuhn and many, many others. Karlsson, Ira McDonald, M. Patton, Randy Presuhn and many, many
others.
Very special thanks must go to Harald Tveit Alvestrand, who Very special thanks must go to Harald Tveit Alvestrand, who
originated RFCs 1766 and 3066, and without whom this document would originated RFCs 1766 and 3066, and without whom this document would
not have been possible. not have been possible.
For this particular document, John Cowan originated the scheme For this particular document, John Cowan originated the scheme
described in Section 2.2.3. Mark Davis originated the scheme described in Section 2.2.3. Mark Davis originated the scheme
described in the Section 2.1.2. described in the Section 2.1.2.
Intellectual Property Statement Intellectual Property Statement
 End of changes. 54 change blocks. 
149 lines changed or deleted 287 lines changed or added

This html diff was produced by rfcdiff 1.27, available from http://www.levkowetz.com/ietf/tools/rfcdiff/