draft-ietf-ltru-matching-10.txt   draft-ietf-ltru-matching-11.txt 
Network Working Group A. Phillips, Ed. Network Working Group A. Phillips, Ed.
Internet-Draft Yahoo! Inc Internet-Draft Yahoo! Inc
Obsoletes: 3066 (if approved) M. Davis, Ed. Obsoletes: 3066 (if approved) M. Davis, Ed.
Expires: August 27, 2006 Google Expires: September 5, 2006 Google
February 23, 2006 March 4, 2006
Matching of Language Tags Matching of Language Tags
draft-ietf-ltru-matching-10 draft-ietf-ltru-matching-11
Status of this Memo Status of this Memo
By submitting this Internet-Draft, each author represents that any By submitting this Internet-Draft, each author represents that any
applicable patent or other IPR claims of which he or she is aware applicable patent or other IPR claims of which he or she is aware
have been or will be disclosed, and any of which he or she becomes have been or will be disclosed, and any of which he or she becomes
aware will be disclosed, in accordance with Section 6 of BCP 79. aware will be disclosed, in accordance with Section 6 of BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF), its areas, and its working groups. Note that
skipping to change at page 1, line 35 skipping to change at page 1, line 35
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt. http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html. http://www.ietf.org/shadow.html.
This Internet-Draft will expire on August 27, 2006. This Internet-Draft will expire on September 5, 2006.
Copyright Notice Copyright Notice
Copyright (C) The Internet Society (2006). Copyright (C) The Internet Society (2006).
Abstract Abstract
This document describes different mechanisms for comparing, matching, This document describes different mechanisms for comparing and
and evaluating language tags. Possible algorithms for language matching language tags. Possible algorithms for language negotiation
negotiation or content selection, filtering, and lookup are or content selection, filtering, and lookup are described. This
described. This document, in combination with RFC 3066bis (Ed.: document, in combination with RFC 3066bis (Ed.: replace "3066bis"
replace "3066bis" with the RFC number assigned to with the RFC number assigned to draft-ietf-ltru-registry-14),
draft-ietf-ltru-registry-14), replaces RFC 3066, which replaced RFC replaces RFC 3066, which replaced RFC 1766.
1766.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
2. The Language Range . . . . . . . . . . . . . . . . . . . . . . 4 2. The Language Range . . . . . . . . . . . . . . . . . . . . . . 4
2.1. Basic Language Range . . . . . . . . . . . . . . . . . . . 4 2.1. Basic Language Range . . . . . . . . . . . . . . . . . . . 4
2.2. Extended Language Range . . . . . . . . . . . . . . . . . 5 2.2. Extended Language Range . . . . . . . . . . . . . . . . . 5
2.3. The Language Priority List . . . . . . . . . . . . . . . . 5 2.3. The Language Priority List . . . . . . . . . . . . . . . . 5
3. Types of Matching . . . . . . . . . . . . . . . . . . . . . . 7 3. Types of Matching . . . . . . . . . . . . . . . . . . . . . . 7
3.1. Choosing a Type of Matching . . . . . . . . . . . . . . . 7 3.1. Choosing a Type of Matching . . . . . . . . . . . . . . . 7
3.2. Filtering . . . . . . . . . . . . . . . . . . . . . . . . 8 3.2. Filtering . . . . . . . . . . . . . . . . . . . . . . . . 9
3.2.1. Basic Filtering . . . . . . . . . . . . . . . . . . . 9 3.2.1. Basic Filtering . . . . . . . . . . . . . . . . . . . 9
3.2.2. Extended Filtering . . . . . . . . . . . . . . . . . . 10 3.2.2. Extended Filtering . . . . . . . . . . . . . . . . . . 10
3.3. Lookup . . . . . . . . . . . . . . . . . . . . . . . . . . 10 3.3. Lookup . . . . . . . . . . . . . . . . . . . . . . . . . . 11
4. Other Considerations . . . . . . . . . . . . . . . . . . . . . 14 4. Other Considerations . . . . . . . . . . . . . . . . . . . . . 15
4.1. Choosing Language Ranges . . . . . . . . . . . . . . . . . 14 4.1. Choosing Language Ranges . . . . . . . . . . . . . . . . . 15
4.2. Meaning of Language Tags and Ranges . . . . . . . . . . . 15 4.2. Meaning of Language Tags and Ranges . . . . . . . . . . . 16
4.3. Considerations for Private Use Subtags . . . . . . . . . . 15 4.3. Considerations for Private Use Subtags . . . . . . . . . . 16
4.4. Length Considerations in Matching . . . . . . . . . . . . 15 4.4. Length Considerations for Language Ranges . . . . . . . . 17
5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 17 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 18
6. Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 6. Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
7. Security Considerations . . . . . . . . . . . . . . . . . . . 19 7. Security Considerations . . . . . . . . . . . . . . . . . . . 20
8. Character Set Considerations . . . . . . . . . . . . . . . . . 20 8. Character Set Considerations . . . . . . . . . . . . . . . . . 21
9. References . . . . . . . . . . . . . . . . . . . . . . . . . . 21 9. References . . . . . . . . . . . . . . . . . . . . . . . . . . 22
9.1. Normative References . . . . . . . . . . . . . . . . . . . 21 9.1. Normative References . . . . . . . . . . . . . . . . . . . 22
9.2. Informative References . . . . . . . . . . . . . . . . . . 21 9.2. Informative References . . . . . . . . . . . . . . . . . . 22
Appendix A. Acknowledgements . . . . . . . . . . . . . . . . . . 22 Appendix A. Acknowledgements . . . . . . . . . . . . . . . . . . 23
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 23 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 24
Intellectual Property and Copyright Statements . . . . . . . . . . 24 Intellectual Property and Copyright Statements . . . . . . . . . . 25
1. Introduction 1. Introduction
Human beings on our planet have, past and present, used a number of Human beings on our planet have, past and present, used a number of
languages. There are many reasons why one would want to identify the languages. There are many reasons why one would want to identify the
language used when presenting or requesting information or in some language used when presenting or requesting information or in some
specific set of information items or "content". specific set of information items or "content".
One use for language identifiers, such as those defined in One use for language identifiers, such as those defined in
[RFC3066bis], is to select content by matching the associated [RFC3066bis], is to select content by matching the associated
language tags to a user's language preferences. language tags to a user's language preferences.
This document defines a syntax (called a language range (Section 2)) This document defines a syntax (called a language range (Section 2))
for specifying items in the user's language preferences (called a for specifying items in the user's list of language preferences
language priority list (Section 2.3)), as well as several schemes for (called a language priority list (Section 2.3)), as well as several
selecting or filtering sets of content by comparing the content's schemes for selecting or filtering sets of content by comparing the
language tags to the user's preferences. Applications, protocols, or content's language tags to the user's preferences. Applications,
specifications will have varying needs and requirements that affect protocols, or specifications will have varying needs and requirements
the choice of a suitable matching scheme. Depending on the choice of that affect the choice of a suitable matching scheme. Depending on
scheme, there are various options left to the implementation. the choice of scheme, there are various options left to the
Protocols that implement a matching scheme either need to specify implementation. Protocols that implement a matching scheme either
each particular choice or indicate the options that are left to the need to specify each particular choice or indicate the options that
implementation to decide. are left to the implementation to decide.
This document is divided into three main sections. One describes how This document is divided into three main sections. One describes how
to indicate a user's preferences using language ranges. Then a to indicate a user's preferences using language ranges. Then a
section describes various schemes for matching these ranges to a set section describes various schemes for matching these ranges to a set
of language tags. There is also a section that deals with various of language tags. There is also a section that deals with various
practical considerations that apply to implementing and using these practical considerations that apply to implementing and using these
schemes. schemes.
This document, in combination with [RFC3066bis] (Ed.: replace This document, in combination with [RFC3066bis] (Ed.: replace
"3066bis" globally in this document with the RFC number assigned to "3066bis" globally in this document with the RFC number assigned to
draft-ietf-ltru-registry-14), replaces [RFC3066], which replaced draft-ietf-ltru-registry-14), replaces [RFC3066], which replaced
[RFC1766]. [RFC1766].
The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in [RFC2119]. document are to be interpreted as described in [RFC2119].
2. The Language Range 2. The Language Range
Language Tags [RFC3066bis] are used to identify the language of some Language Tags [RFC3066bis] are used to identify the language of some
information item or content. Applications or protocols that use information item or "content". Applications or protocols that use
language tags are often faced with the problem of identifying sets of language tags are often faced with the problem of identifying sets of
content that share certain language attributes. For example, content that share certain language attributes. For example,
HTTP/1.1 [RFC2616] describes one such mechanism in its discussion of HTTP/1.1 [RFC2616] describes one such mechanism in its discussion of
the Accept-Language header (Section 14.4), which is used when the Accept-Language header (Section 14.4), which is used when
selecting content from servers based on the language of that content. selecting content from servers based on the language of that content.
When selecting content according to its language, it is useful to When selecting content according to its language, it is useful to
have a mechanism for identifying sets of language tags that share have a mechanism for identifying sets of language tags that share
specific attributes. This allows users to select or filter content specific attributes. This allows users to select or filter content
based on specific requirements. Such an identifier is called a based on specific requirements. Such an identifier is called a
"Language Range". "language range".
There are different types of language range, whose specific There are different types of language range, whose specific
attributes vary to match their application. Language ranges are attributes vary according to their application. Language ranges are
similar in content to language tags: they consist of a sequence of similar to language tags: they consist of a sequence of subtags
subtags separated by hyphens. In a language range, each subtag MUST separated by hyphens. In a language range, each subtag MUST either
either be a sequence of ASCII alphanumeric characters or the single be a sequence of ASCII alphanumeric characters or the single
character '*' (%2A, ASTERISK). The character '*' is a "wildcard" character '*' (%2A, ASTERISK). The character '*' is a "wildcard"
that matches any sequence of subtags. Restrictions on the meaning that matches any sequence of subtags. The meaning and uses of
and use of wildcards vary according to the type of language range. wildcards vary according to the type of language range.
Language tags and thus language ranges are to be treated as case- Language tags and thus language ranges are to be treated as case-
insensitive: there exist conventions for the capitalization of some insensitive: there exist conventions for the capitalization of some
of the subtags, but these MUST NOT be taken to carry meaning. of the subtags, but these MUST NOT be taken to carry meaning.
Matching of language tags to language ranges MUST be done in a case- Matching of language tags to language ranges MUST be done in a case-
insensitive manner. insensitive manner.
2.1. Basic Language Range 2.1. Basic Language Range
A "basic language range" identifies the set of language tags that all A "basic language range" describes a user's language preference as a
begin with the same sequence of subtags. Each range consists of a specific, uninterrupted, sequence of subtags. Each range consists of
sequence of alphanumeric subtags separated by hyphens. The basic a sequence of alphanumeric subtags separated by hyphens. The basic
language range is defined by the following ABNF [RFC4234]: language range is defined by the following ABNF [RFC4234]:
language-range = (1*8ALPHA *("-" 1*8alphanum)) / "*" language-range = (1*8ALPHA *("-" 1*8alphanum)) / "*"
alphanum = ALPHA / DIGIT alphanum = ALPHA / DIGIT
Basic language ranges (originally described by HTTP/1.1 [RFC2616] and Basic language ranges (originally described by HTTP/1.1 [RFC2616] and
later [RFC3066]) have the same syntax as an [RFC3066] language tag or later [RFC3066]) have the same syntax as an [RFC3066] language tag or
are the single character "*". They differ from the language tags are the single character "*". They differ from the language tags
defined in [RFC3066bis] only in that there is no requirement that defined in [RFC3066bis] only in that there is no requirement that
they be "well-formed" or be validated against the IANA Language they be "well-formed" or be validated against the IANA Language
skipping to change at page 5, line 15 skipping to change at page 5, line 15
'language-range': this is mentioned in the errata) 'language-range': this is mentioned in the errata)
Use of a basic language range seems to imply that there is a semantic Use of a basic language range seems to imply that there is a semantic
relationship between language tags that share the same prefix. While relationship between language tags that share the same prefix. While
this is often the case, it is not always true and users should note this is often the case, it is not always true and users should note
that the set of language tags that match a specific language range that the set of language tags that match a specific language range
may not represent mutually intelligible languages. may not represent mutually intelligible languages.
2.2. Extended Language Range 2.2. Extended Language Range
Basic language ranges allow users to specify a set of language tags Occasionally users will wish to select a set of language tags based
that share the same initial subtags. Occasionally users will wish to on the presence of specific subtags. An "extended language range"
select a set of language tags based on the presence of specific describes a user's language preference as an ordered sequence of
subtags. For example, a user might wish to select all language tags subtags. For example, a user might wish to select all language tags
that contains the region subtag 'CH'. Extended language ranges are that contain the region subtag 'CH' (Switzerland). Extended language
useful in specifying a particular sequence of subtags that appear in ranges are useful in specifying a particular sequence of subtags that
the set of matching tags without having to specify all of the appear in the set of matching tags without having to specify all of
intervening subtags. the intervening subtags.
An extended language range can be represented by the following ABNF: An extended language range can be represented by the following ABNF:
extended-language-range = (1*8ALPHA / "*") extended-language-range = (1*8ALPHA / "*")
*("-" (1*8alphanum / "*")) *("-" (1*8alphanum / "*"))
Figure 2: Extended Language Range Figure 2: Extended Language Range
The wildcard subtag '*' MAY occur in any position in the extended The wildcard subtag '*' can occur in any position in the extended
language range, where it matches any sequence of subtags that might language range, where it matches any sequence of subtags that might
occur in that position in a language tag. However wildcards outside occur in that position in a language tag. However, wildcards outside
the first position in an extended language range are ignored by most the first position in an extended language range are ignored by most
matching schemes. Use of multiple wildcards SHOULD NOT be taken to matching schemes. Use of one or more wildcards SHOULD NOT be taken
imply that a certain number of subtags will appear in the matching to imply that a certain number of subtags will appear in the matching
set of language tags. set of language tags.
Implementations that specify basic ranges MAY map extended language Implementations that specify basic ranges MAY map extended language
ranges to basic language ranges: if the first subtag is a "*" then ranges to basic language ranges: if the first subtag is a "*" then
the entire range is treated as "*" (which matches the default the entire range is treated as "*", otherwise each wildcard subtag is
content), otherwise each wildcard subtag is removed. For example, if removed. For example, if the language range were "en-*-US", then the
the language range were "en-*-US", then the range would be mapped to range would be mapped to "en-US".
"en-US".
2.3. The Language Priority List 2.3. The Language Priority List
When users specify a language preference they often need to specify a A user's language preferences will often need to specify more than
prioritized list of language ranges in order to best reflect their one language range and thus users often need to specify a prioritized
language preferences. This is especially true for speakers of list of language ranges in order to best reflect their language
minority languages. A speaker of Breton in France, for example, may preferences. This is especially true for speakers of minority
specify "be" followed by "fr", meaning that if Breton is available, languages. A speaker of Breton in France, for example, may specify
it is preferred, but otherwise French is the best alternative. It "be" followed by "fr", meaning that if Breton is available, it is
can get more complex: a speaker may wish to fall back from Skolt Sami preferred, but otherwise French is the best alternative. It can get
to Northern Sami to Finnish. more complex: a user may wish to fall back from Skolt Sami to
Northern Sami to Finnish.
A "Language Priority List" is a prioritized or weighted list of A "language priority list" is a prioritized or weighted list of
language ranges. One well known example of such a list is the language ranges. One well known example of such a list is the
"Accept-Language" header defined in RFC 2616 [RFC2616] (see Section "Accept-Language" header defined in RFC 2616 [RFC2616] (see Section
14.4) and RFC 3282 [RFC3282]. A simple list of ranges, i.e. one that 14.4) and RFC 3282 [RFC3282].
contains no weighting information, is considered to be in descending
order of priority.
The various matching operations described in this document include The various matching operations described in this document include
considerations for using a language priority list. This document considerations for using a language priority list. This document
does not define any syntax for a language priority list; defining does not define the syntax for a language priority list; defining
such a syntax is the responsibility of the protocol, application, or such a syntax is the responsibility of the protocol, application, or
implementation that uses it. When given as examples in this specification that uses it. When given as examples in this document,
document, language priority lists will be shown as a quoted sequence language priority lists will be shown as a quoted sequence of ranges
of ranges separated by semicolons, like this: "en; fr; zh-Hant" separated by commas, like this: "en, fr, zh-Hant" (which would be
(which would be read as "English before French before Chinese as read as "English before French before Chinese as written in the
written in the Traditional script"). Traditional script").
Where a language priority list provides "quality weights" for the A simple list of ranges is considered to be in descending order of
language ranges, such as the use of Q weights in the syntax of the priority. Other language priority lists provide "quality weights"
"Accept-Language" header (defined in [RFC2616], Section 14.4, and for the language ranges in order to specify the relative priority of
[RFC3282]), language ranges without a weight are given values equal the user's language preferences. An example of this would be the use
to the value of the previous language range (processing from first to of "q" values in the syntax of the "Accept-Language" header (defined
last). If the first language range has no weight, it is given a in [RFC2616], Section 14.4, and [RFC3282]).
value of 1.0. Then language ranges with zero weights are removed.
For example, "fr, en;q=0.5, de, it" becomes "fr;q=1.0, en;q=0.5,
de;q=0.5, it;q=0.5". The language priority list is then sorted from
highest priority to lowest, with language ranges that share the same
weights remain in the same order as in the original language priority
list.
3. Types of Matching 3. Types of Matching
Matching language ranges to language tags can be done in a number of Matching language ranges to language tags can be done in a number of
different ways. This section describes several different matching different ways. This section describes several different matching
schemes, as well as the considerations for choosing between them. schemes, as well as the considerations for choosing between them.
Protocols and specifications SHOULD clearly indicate the particular Protocols and specifications SHOULD clearly indicate the particular
mechanism used in selecting or matching language tags. mechanism used in selecting or matching language tags.
There are several types of matching scheme. This document presents There are several types of matching scheme. This document presents
two types: those that produce zero or more information items (called two types: those that produce zero or more information items (called
"filtering") and those that produce a single information item for a "filtering") and those that produce a single information item for a
given request (called "lookup"). given request (called "lookup").
Implementations or protocols MAY use different matching schemes than Implementations or protocols MAY use different matching schemes from
the ones described in this document, as long as those mechanisms are the ones described in this document, as long as those mechanisms are
clearly specified. clearly specified.
3.1. Choosing a Type of Matching 3.1. Choosing a Type of Matching
Applications, protocols, and specifications are faced with the Applications, protocols, and specifications are faced with the
decision of what type of matching to use. Sometimes, different decision of what type of matching to use. Sometimes, different
styles of matching might be suited for different kinds of processing styles of matching are suited to different kinds of processing within
within a particular application or protocol. a particular application or protocol.
Language tag matching is a tool, and does not by itself specify a Language tag matching is a tool, and does not by itself specify a
complete procedure for the use of language tags. Such procedures are complete procedure for the use of language tags. Such procedures are
intimately tied to the application protocol in which they occur. intimately tied to the application protocol in which they occur.
When specifying a protocol operation using matching, the protocol When specifying a protocol operation using matching, the protocol
MUST specify: MUST specify:
o Which type(s) of language tag matching it uses o Which type(s) of language tag matching it uses
o Whether the operation returns a single result (lookup) or a o Whether the operation returns a single result (lookup) or a
skipping to change at page 8, line 7 skipping to change at page 8, line 7
1. Basic Filtering (Section 3.2.1) matches a language priority list 1. Basic Filtering (Section 3.2.1) matches a language priority list
consisting of basic language ranges (Section 2.1) to sets of consisting of basic language ranges (Section 2.1) to sets of
language tags. language tags.
2. Extended Filtering (Section 3.2.2) matches a language priority 2. Extended Filtering (Section 3.2.2) matches a language priority
list consisting of extended language ranges (Section 2.2) to sets list consisting of extended language ranges (Section 2.2) to sets
of language tags. of language tags.
3. Lookup (Section 3.3) matches a language priority list consisting 3. Lookup (Section 3.3) matches a language priority list consisting
of basic language ranges to sets of language tags find the of basic language ranges to sets of language tags to find the one
_exactly_ one language tag that best matches the range. _exact_ language tag that best matches the range.
Both types of filtering can be used to produce a set of results (such Filtering can be used to produce a set of results (such as a
as a collection of documents) by comparing the user's preferences to collection of documents) by comparing the user's preferences to
language tags associated with the set of content. For example, when language tags associated with the set of content. For example, when
performing a search, one might use filtering to limit the results to performing a search, one might use filtering to limit the results to
documents tagged as being written in French. They might also be used items tagged as being in the French language. Filtering can also be
when deciding whether to perform a language-sensitive process on some used when deciding whether to perform a language-sensitive process on
content. For example, a process might cause paragraphs whose some content. For example, a process might cause paragraphs whose
language tag matched the language range "nl" to be displayed in language tag matched the language range "nl" to be displayed in
italics within a document. italics within a document.
Lookup produces the single result that best matches a given set of Lookup produces the single result that best matches the user's
user preferences, so it is useful in cases in which only a single preferences, so it is useful in cases in which only a single item can
item can be returned. For example, if a process were to insert a be returned. For example, if a process were to insert a human
human readable error message into a protocol header, it might select readable error message into a protocol header, it might select the
the text based on the user's language priority list. Since the text based on the user's language priority list. Since the process
process can return only one item, it must choose a single item and it can return only one item, it must choose a single item and it must
must return some item, even if no content's language tag matches the return some item, even if none of the content's language tags match
language priority list supplied by the user. the language priority list supplied by the user.
The types of matching in this document are designed so that The types of matching in this document are designed so that
implementations are not required to validate or understand any of the implementations are not required to validate or understand any of the
semantics of the language tags or ranges or of the subtags in them. semantics of the language tags or ranges or of the subtags in them.
None of them require access to the IANA Language Subtag Registry (see None of them require access to the IANA Language Subtag Registry (see
Section 3 in [RFC3066bis]). This simplifies and speeds the Section 3 in [RFC3066bis]). This simplifies implementation of these
performance of implementations. schemes. An implementation MAY choose to check if either the
language ranges or language tags being matched are "well-formed" or
"valid" (see [RFC3066bis], Section 2.2.9) and MAY choose not to
process invalid ranges.
Regardless of the matching scheme chosen, protocols and Regardless of the matching scheme chosen, protocols and
implementations MAY canonicalize language tags and ranges by mapping implementations MAY canonicalize language tags and ranges by mapping
grandfathered and obsolete tags or subtags into modern equivalents. grandfathered and obsolete tags or subtags into modern equivalents.
If an implementation canonicalizes either ranges or tags, then the If an implementation canonicalizes either ranges or tags, then the
implementation will require the IANA Language Subtag Registry implementation will require the IANA Language Subtag Registry
information for that purpose. Implementations MAY also use semantic information for that purpose. Implementations MAY also use semantic
information external to the registry when matching tags. For information external to the registry when matching tags. For
example, the primary language subtags 'nn' (Nynorsk Norwegian) and example, the primary language subtags 'nn' (Nynorsk Norwegian) and
'nb' (Bokmal Norwegian) might both be usefully matched to the more 'nb' (Bokmal Norwegian) might both be usefully matched to the more
general subtag 'no' (Norwegian). Or an implementation might infer general subtag 'no' (Norwegian). Or an implementation might infer
that content labeled "zh-CN" is more likely to match the range "zh- that content labeled "zh-Hans" (Chinese as written in the Simplified
Hans" than equivalent content labeled "zh-TW". script) is more likely to match the range "zh-CN" (Chinese as used in
China, where the Simplified script is predominant) than equivalent
content labeled "zh-TW" (Chinese as used in Taiwan, where the
Traditional script is predominant).
3.2. Filtering 3.2. Filtering
Filtering is used to select the set of language tags that matches a Filtering is used to select the set of language tags that matches a
given language priority list and return the associated content. It given language priority list and return the associated content. It
is called "filtering" because this set might contain no items at all is called "filtering" because this set might contain no items at all
or it might return an arbitrarily large number of matching items: as or it might return an arbitrarily large number of matching items: as
many items as match the language priority list, thus "filtering out" many items as match the language priority list, thus "filtering out"
the non-matching items. the non-matching items.
In filtering, the language range represents the _least_ specific In filtering, each language range represents the _least_ specific
(that is, the fewest number of subtags) language tag which is an language tag (that is, the language tag with fewest number of
acceptable match. All of the language tags in the matching set of subtags) which is an acceptable match. All of the language tags in
tags will have an equal or greater number of subtags than the the matching set of tags will have an equal or greater number of
language range. Every non-wildcard subtag in the language range will subtags than the language range. Every non-wildcard subtag in the
appear in every one of the matching language tags. For example, if language range will appear in every one of the matching language
the language priority list consists of the range "de-CH", one might tags. For example, if the language priority list consists of the
see tags such as "de-CH-1996" but one will never see a tag such as range "de-CH", one might see tags such as "de-CH-1996" but one will
"de" (because the 'CH' subtag is missing). never see a tag such as "de" (because the 'CH' subtag is missing).
If the language priority list (see Section 2.3) contains more than If the language priority list (see Section 2.3) contains more than
one range, the content returned is typically ordered in descending one range, the content returned is typically ordered in descending
level of preference. level of preference, but it MAY be unordered, according to the needs
of the application or protocol.
Some examples of applications where filtering might be appropriate Some examples of applications where filtering might be appropriate
include: include:
o Applying a style to sections of a document in a particular set of o Applying a style to sections of a document in a particular set of
languages. languages.
o Displaying the set of documents containing a particular set of o Displaying the set of documents containing a particular set of
keywords written in a specific set of languages. keywords written in a specific set of languages.
o Selecting all email items written in a specific set of languages. o Selecting all email items written in a specific set of languages.
The content returned MAY either be ordered or unordered according to o Selecting audio files spoken in a particular language.
the priority in the language priority list (and other criteria),
according to the needs of the application or protocol.
3.2.1. Basic Filtering 3.2.1. Basic Filtering
When filtering using basic language ranges, each basic language range When filtering using basic language ranges, each basic language range
in the language priority list is considered in turn, according to in the language priority list is considered in turn, according to
priority. A particular language tag matches a language range if it priority. A particular language tag matches a language range if, in
exactly equals the tag, or if it exactly equals a prefix of the tag a case-insensitive comparison, it exactly equals the tag, or if it
such that the first character following the prefix is "-". For exactly equals a prefix of the tag such that the first character
example, the language-range "de-de" matches the language tag "de-DE- following the prefix is "-". For example, the language-range "de-de"
1996", but not the language tags "de-Deva" or "de-Latn-DE". matches the language tag "de-DE-1996", but not the language tags "de-
Deva" or "de-Latn-DE".
The special range "*" in a language priority list matches any tag. A The special range "*" in a language priority list matches any tag. A
protocol which uses language ranges MAY specify additional rules protocol which uses language ranges MAY specify additional rules
about the semantics of "*"; for instance, HTTP/1.1 [RFC2616] about the semantics of "*"; for instance, HTTP/1.1 [RFC2616]
specifies that the range "*" matches only languages not matched by specifies that the range "*" matches only languages not matched by
any other range within an "Accept-Language" header. any other range within an "Accept-Language" header.
Basic filtering is identical to the type of matching described in
[RFC3066], Section 2.5 (Language-range).
3.2.2. Extended Filtering 3.2.2. Extended Filtering
When filtering using extended language ranges, each extended language When filtering using extended language ranges, each extended language
range in the language priority list is considered in turn, according range in the language priority list is considered in turn, according
to priority. A particular language range is compared to each to priority. A particular language range is compared to each
language tag using the following process: language tag using the following process:
Compare the first subtag in the extended language tag to the first Compare the first subtag in the extended language tag to the first
subtag in the language tag in a case insensitive manner. If the subtag in the language tag in a case insensitive manner. If the
first subtag in the range is "*", it matches any value. Otherwise first subtag in the range is "*", it matches any value. Otherwise
the two values must match or the overall match fails. the two values must match or the overall match fails.
Take each non-wildcard subtag in the language range and compare it to Take each non-wildcard subtag in the language range and compare it in
the next subtag in the language tag in turn until a matching subtag a case-insensitive manner to the next subtag in the language tag. If
is found or the langauge tag is exhausted. If the end of the the range's subtag exactly matches the tag's subtag, proceed to the
language tag is found first, the match fails. If a match is found, next non-wildcard subtag in the language range (and beginning with
this step is repeated with the next non-wildcard subtag in the the next subtag in the language tag) until the list of subtags in the
language range (and beginning with the next subtag in the language language range is exhausted or the match fails. If the tag's subtag
tag) until the list of subtags in the language range is exhausted or is a "singleton" (a single letter or digit, which, in this case,
the match fails. includes the private-use subtag 'x') and the range's subtag does not
match or if the language tag's list of subtags is exhausted, the
match fails. If the language range's list of subtags is exhausted,
the match succeeds.
Subtags not specified, including those at the end of the language Subtags not specified, including those at the end of the language
range, are thus treated as if assigned the wildcard value "*". range, are thus treated as if assigned the wildcard value "*". Much
Extended filtering works, therefore, much like basic filtering. For like basic filtering, extended filtering selects content with
arbitrarily long tags that share the same initial subtags as the
language range. In addition extended filtering selects content with
any intermediate subtags unspecified in the language range. For
example, the extended language range "de-*-DE" matches all of the example, the extended language range "de-*-DE" matches all of the
following tags: following tags:
de-DE de-DE
de-Latn-DE de-Latn-DE
de-Latf-DE de-Latf-DE
de-de
de-DE-x-goethe de-DE-x-goethe
de-Latn-DE-1996 de-Latn-DE-1996
The same range does not match any of the following tags for the
reasons shown:
de (missing 'DE')
de-x-DE (singleton 'x' occurs before 'DE')
de-Deva ('Deva' not equal to 'DE')
Note: The structure of language tags defined by [RFC3066bis] defines
each type of subtag (language, script, region, and so forth)
according to position, size, and content. This means that subtags in
a language range can only match specific types of subtags in a
language tag. For example, a subtag such as 'Latn' is always a
script subtag (unless it follows a singleton) while a subtag such as
'nedis' can only match the equivalent variant subtag.
3.3. Lookup 3.3. Lookup
Lookup is used to select the single language tag that best matches Lookup is used to select the single language tag that best matches
the language priority list for a given request and return the the language priority list for a given request and return the
associated content. When performing lookup, each language range in associated content. When performing lookup, each language range in
the language priority list is considered in turn, according to the language priority list is considered in turn, according to
priority. By contrast with filtering, each language range represents priority. By contrast with filtering, each language range represents
the _most_ specific tag which is an acceptable match. The first the _most_ specific tag which is an acceptable match. The first
content found with a matching tag, according to the user's priority, content found with a matching tag, according to the user's priority,
is considered the closest match and is the content returned. For is considered the closest match and is the content returned. For
skipping to change at page 11, line 25 skipping to change at page 12, line 13
insert. Examples of lookup might include: insert. Examples of lookup might include:
o Selection of a template containing the text for an automated email o Selection of a template containing the text for an automated email
response. response.
o Selection of a item containing some text for inclusion in a o Selection of a item containing some text for inclusion in a
particular Web page. particular Web page.
o Selection of a string of text for inclusion in an error log. o Selection of a string of text for inclusion in an error log.
o Selection of an audio file to play as a prompt in a phone system.
In the lookup scheme, the language range is progressively truncated In the lookup scheme, the language range is progressively truncated
from the end until a matching piece of content is located. For from the end until a matching piece of content is located. Single
example, starting with the range "zh-Hant-CN-x-private", the lookup letter or digit subtags (including both the letter 'x' which
progressively searches for content as shown below: introduces private-use sequences, and the subtags that introduce
extensions) are removed at the same time as their closest trailing
subtag. For example, starting with the range "zh-Hant-CN-x-private1-
private2", the lookup progressively searches for content as shown
below:
Range to match: zh-Hant-CN-x-private Range to match: zh-Hant-CN-x-private1-private2
1. zh-Hant-CN-x-private 1. zh-Hant-CN-x-private1-private2
2. zh-Hant-CN 2. zh-Hant-CN-x-private1
3. zh-Hant 3. zh-Hant-CN
4. zh 4. zh-Hant
5. (default content) 5. zh
6. (default content)
Figure 3: Example of a Lookup Fallback Pattern Figure 3: Example of a Lookup Fallback Pattern
This scheme allows some flexibility in finding a match. For example, This allows some flexibility in finding a match. For example, lookup
lookup provides better results for cases in which content is not provides better results for cases in which content is not available
available that exactly matches the user request than if the default that exactly matches the user request than if the default language
language for the system or content were returned immediately. Not for the system or content were returned immediately. Not every
every specific level of tag granularity is usually available or specific level of tag granularity is usually available or language
language content may be sparsely populated, so "falling back" through content may be sparsely populated. "Falling back" through the subtag
the subtag sequence provides more opportunity to find a match between sequence provides more opportunity to find a match between available
available language tags and the user's request. language tags and the user's request.
The default behavior when no tag matches the language priority list The default behavior when no tag matches the language priority list
is implementation defined. An implementation might, for example, is implementation defined. An implementation might, for example,
return content with no language tag; might supply content with an return content:
empty language tag value (the built-in attribute xml:lang in [XML10]
permits the empty value); might be a particular language designated o with no language tag
for the bit of content being selected; or it might select the tag
"i-default" (see [RFC2277]). When performing lookup using a language o of a non-linguistic nature, such as an image or sound
priority list, the progressive search MUST proceed to consider each
language range in the list before finding the default content or o with an empty language tag value, in cases where the protocol
empty tag. permits the empty value (see, for example, "xml:lang" in [XML10],
which indicates that the element contains non-linguistic content)
o in a particular language designated for the bit of content being
selected
o labelled with the tag "i-default" (see [RFC2277])
When performing lookup using a language priority list, the
progressive search MUST process each language range in the list
before finding the default content or empty tag.
One common way for an application or implementation to provide for a One common way for an application or implementation to provide for a
default is to allow a specific language range to be set as the default is to allow a specific language range to be set as the
default for a specific type of request. This language range is then default for a specific type of request. This language range is then
treated as if it were appended to the end of the language priority treated as if it were appended to the end of the language priority
list as a whole, rather than after each item in the language priority list as a whole, rather than after each item in the language priority
list. list.
For example, if a particular user's language priority list were For example, if a particular user's language priority list were
"fr-FR; zh-Hant" and the program doing the matching had a default "fr-FR, zh-Hant" and the program doing the matching had a default
language range of "ja-JP", the program would search for content as language range of "ja-JP", the program would search for content as
follows: follows:
1. fr-FR 1. fr-FR
2. fr 2. fr
3. zh-Hant // next language 3. zh-Hant // next language
4. zh 4. zh
5. (search for the default content) 5. (search for the default content)
a. ja-JP a. ja-JP
b. ja b. ja
c. (implementation defined default) c. (implementation defined default)
Figure 4: Lookup Using a Language Priority List Figure 4: Lookup Using a Language Priority List
Implementations SHOULD ignore extensions and unrecognized private-use Implementations SHOULD ignore extensions and unrecognized private-use
subtags when performing lookup, since these subtags are usually subtags when performing lookup, since these subtags are usually
orthogonal to the user's request. orthogonal to the user's request.
The special language range "*" matches any language tag. In the The special language range "*" matches any language tag. In the
lookup scheme, this range does not convey enough information by lookup scheme, this range does not convey enough information by
itself to determine which content is most appropriate, since it itself to determine which content is most appropriate, since it
matches everything. If the language range "*" is the only one in the matches everything. If the language range "*" is followed by other
language priority list, it matches the default content. If the language ranges, it SHOULD be skipped. If the language range "*" is
language range "*" is followed by other language ranges, it should be the only one in the language priority list or if no other language
skipped. range follows, the default content SHOULD be returned.
In some cases, the language priority list might contain one or more In some cases, the language priority list might contain one or more
extended language ranges (as, for example, when the same language extended language ranges (as, for example, when the same language
priority list is used as input for both lookup and filtering priority list is used as input for both lookup and filtering
operations). Wildcard values in an extended language range normally operations). Wildcard values in an extended language range normally
match any value that occurs in that position in a language tag. match any value that occurs in that position in a language tag.
Since only one item can be returned for any given lookup request, Since only one item can be returned for any given lookup request,
wildcards in a language range have to be processed in a consistent wildcards in a language range have to be processed in a consistent
manner or the same request will produce widely varying results. manner or the same request will produce widely varying results.
Implementations that accept extended language ranges MUST define Implementations that accept extended language ranges MUST define
which content is returned when more than one item matches the which content is returned when more than one item matches the
extended language range. extended language range.
For example, an implementation could return the matching tag that is For example, an implementation could return the matching tag that is
first in ASCII-order. If the language range were "*-CH" and the set first in ASCII-order. If the language range were "*-CH" and the set
of tags included "de-CH", "fr-CH", and "it-CH", then the tag "de-CH" of tags included "de-CH", "fr-CH", and "it-CH", then the tag "de-CH"
would be returned. Another example would be for an implementation to would be returned. Another possibility would be for an
map the extended language ranges to basic ranges. implementation to map the extended language ranges to basic ranges.
4. Other Considerations 4. Other Considerations
When working with language ranges and matching schemes, there are When working with language ranges and matching schemes, there are
some additional points that may influence the choice of either. some additional points that may influence the choice of either.
4.1. Choosing Language Ranges 4.1. Choosing Language Ranges
Users indicate their language preferences via the choice of a Users indicate their language preferences via the choice of a
language range or the list of language ranges in a language priority language range or the list of language ranges in a language priority
list. The type of matching affects what the best choice is for a list. The type of matching affects what the best choice is for a
given user. user.
Most matching schemes make no attempt to process the semantic meaning Most matching schemes make no attempt to process the semantic meaning
of the subtags. The language range (or its subtags) is usually of the subtags and the language range is compared, in a case-
compared in a case-insensitive manner to each language tag being insensitive manner, to each language tag being matched, using basic
matched, using basic string processing. string processing. Users SHOULD select language ranges that are
well-formed, valid language tags according to [RFC3066bis]
(substituting wildcards as appropriate in extended language ranges).
Users SHOULD replace tags or subtags which have been deprecated with
the Preferred-Value from the IANA Language Subtag Registry. If the
user is working with content that might use the older form, the user
might include both the new and old forms in a language priority list.
For example, the tag "art-lojban" is deprecated. The subtag 'jbo' is
supposed to be used instead, so the user might use it to form the
language range. Or the user might include both in a language
priority list: "jbo, art-lojban".
Users SHOULD avoid subtags that add no distinguishing value to a Users SHOULD avoid subtags that add no distinguishing value to a
language range. Generally, the fewer subtags that appear in the language range. When filtering, the fewer the number of subtags that
language range, the more content the range will match. appear in the language range, the more content the range will
probably match, while in lookup unnecessary subtags might cause
"better", more-specific content to be skipped in favor of less
specific content. For example, the range "de-Latn-DE" would return
content tagged "de" instead of content tagged "de-DE", even though
the latter is probably a better match.
Most notably, script subtags SHOULD NOT be used to form a language Many languages are written predominantly in a single script. This is
range in combination with language subtags that have a matching usually recorded in the Suppress-Script field in that language
Suppress-Script field in their registry entry. Thus the language subtag's registry entry. For these languages, script subtags SHOULD
range "en-Latn" is probably inappropriate in most cases (because the NOT be used to form a language range. Thus the language range "en-
vast majority of English documents are written in the Latin script Latn" is inappropriate in most cases (because the vast majority of
and thus the 'en' language subtag has a Suppress-Script field for English documents are written in the Latin script and thus the 'en'
'Latn' in the registry). language subtag has a Suppress-Script field for 'Latn' in the
registry).
When working with tags and ranges, note that extensions and most When working with tags and ranges, note that extensions and most
private-use subtags are orthogonal to language tag matching, in that private-use subtags are orthogonal to language tag matching, in that
they specify additional attributes of the text not related to the they specify additional attributes of the text not related to the
goals of most matching schemes. Users SHOULD avoid using these goals of most matching schemes. Users SHOULD avoid using these
subtags in language ranges, since they interfere with the selection subtags in language ranges, since they interfere with the selection
of available content. When used in language tags (as opposed to of available content. When used in language tags (as opposed to
ranges), these subtags normally do not interfere with filtering ranges), these subtags normally do not interfere with filtering
(Section 3), since they appear at the end of the tag and will match (Section 3), since they appear at the end of the tag and will match
all prefixes. all prefixes. Lookup (Section 3.3) implementations often ignore
unrecognized private-use and extension subtags when performing
Private-use and Extension subtags are normally orthogonal to language language tag fallback.
tag fallback. Implementations or specifications that use a lookup
(Section 3.3) matching scheme often ignore unrecognized private-use
and extension subtags when performing language tag fallback. In
addition, since these subtags are always at the end of the sequence
of subtags, their use in language tags normally doesn't interfere
with the use of ranges that omit them in the filtering (Section 3.2)
matching schemes described below. However, they do interfere with
filtering when used in language ranges and SHOULD be avoided in
ranges as a result.
Applications, specifications, or protocols that choose not to Applications, specifications, or protocols that choose not to
interpret one or more private-use or extension subtags SHOULD NOT interpret one or more private-use or extension subtags SHOULD NOT
remove or modify these extensions in content that they are remove or modify these extensions in content that they are
processing. When a language tag instance is to be used in a processing. When a language tag instance is to be used in a
specific, known protocol, and is not being passed through to other specific, known protocol, and is not being passed through to other
protocols, language tags MAY be filtered to remove subtags and protocols, language tags MAY be altered to remove subtags and
extensions that are not supported by that protocol. Such filtering extensions that are not supported by that protocol. Such alterations
SHOULD be avoided, if possible, since it removes information that SHOULD be avoided, if possible, since they remove information that
might be relevant to services on the other end of the protocol that might be relevant elsewhere that would make use of that information.
would make use of that information.
Some applications of language tags might want or need to consider Some applications of language tags might want or need to consider
extensions and private-use subtags when matching tags. If extensions extensions and private-use subtags when matching tags. If extensions
and private-use subtags are included in a matching or filtering and private-use subtags are included in a matching process that
process that utilizes one of the schemes described in this document, utilizes one of the schemes described in this document, then the
then the implementation SHOULD canonicalize the language tags and/or implementation SHOULD canonicalize the language tags and/or ranges
ranges before performing the matching. Note that language tag before performing the matching. Note that language tag processors
processors that claim to be "well-formed" processors as defined in that claim to be "well-formed" processors as defined in [RFC3066bis]
[RFC3066bis] generally fall into this category. generally fall into this category.
4.2. Meaning of Language Tags and Ranges 4.2. Meaning of Language Tags and Ranges
Selecting content using language ranges requires some understanding Selecting content using language ranges requires some understanding
by users of what they are selecting. The meaning of the various by users of what they are selecting. The meaning of the various
subtags in a language range are identical to their meaning in a subtags in a language range are identical to their meaning in a
language tag (see Section 4.2 in [RFC3066bis]), with the addition language tag (see Section 4.2 in [RFC3066bis]), with the addition
that the wildcard "*" represents any matching sequence of values. that the wildcard "*" represents any matching sequence of values.
4.3. Considerations for Private Use Subtags 4.3. Considerations for Private Use Subtags
skipping to change at page 15, line 47 skipping to change at page 17, line 6
that intend to use or exchange language tags that use them and great that intend to use or exchange language tags that use them and great
caution SHOULD be used in employing them in content or protocols caution SHOULD be used in employing them in content or protocols
intended for general use. Private-use subtags are simply useless for intended for general use. Private-use subtags are simply useless for
information exchange without prior arrangement. information exchange without prior arrangement.
The value and semantic meaning of private-use tags and of the subtags The value and semantic meaning of private-use tags and of the subtags
used within such a language tag are not defined. Matching private- used within such a language tag are not defined. Matching private-
use tags using language ranges or extended language ranges can result use tags using language ranges or extended language ranges can result
in unpredictable content being returned. in unpredictable content being returned.
4.4. Length Considerations in Matching 4.4. Length Considerations for Language Ranges
Language ranges are very similar to language tags in terms of content Language ranges are very similar to language tags in terms of content
and usage. The same types of restrictions on length that apply to and usage. The same types of restrictions on length that apply to
language tags could also apply to language ranges. Implementation, language tags can also apply to language ranges. See [RFC3066bis]
protocol, and specificiation authors SHOULD apply the considerations Section 4.3 (Length Considerations).
in [RFC3066bis] Section 4.3 (Length Considerations) where appropriate
to language ranges and language priority lists.
5. IANA Considerations 5. IANA Considerations
This document presents no new or existing considerations for IANA. This document presents no new or existing considerations for IANA.
6. Changes 6. Changes
This is the first version of this document. This is the first version of this document.
7. Security Considerations 7. Security Considerations
skipping to change at page 22, line 28 skipping to change at page 24, line 5
Harald Alvestrand, Jeremy Carroll, John Cowan, Martin Duerst, Frank Harald Alvestrand, Jeremy Carroll, John Cowan, Martin Duerst, Frank
Ellermann, Doug Ewell, Marion Gunn, Kent Karlsson, Ira McDonald, M. Ellermann, Doug Ewell, Marion Gunn, Kent Karlsson, Ira McDonald, M.
Patton, Randy Presuhn, Eric van der Poel, Markus Scherer, and many, Patton, Randy Presuhn, Eric van der Poel, Markus Scherer, and many,
many others. many others.
Very special thanks must go to Harald Tveit Alvestrand, who Very special thanks must go to Harald Tveit Alvestrand, who
originated RFCs 1766 and 3066, and without whom this document would originated RFCs 1766 and 3066, and without whom this document would
not have been possible. not have been possible.
For this particular document, John Cowan originated the scoring
scheme. Mark Davis originated the scheme described in Section 3.3.
Authors' Addresses Authors' Addresses
Addison Phillips (editor) Addison Phillips (editor)
Yahoo! Inc Yahoo! Inc
Email: addison at inter dash locale dot com Email: addison at inter dash locale dot com
Mark Davis (editor) Mark Davis (editor)
Google Google
 End of changes. 60 change blocks. 
221 lines changed or deleted 264 lines changed or added

This html diff was produced by rfcdiff 1.29, available from http://www.levkowetz.com/ietf/tools/rfcdiff/