draft-ietf-ltru-matching-09.txt   draft-ietf-ltru-matching-10.txt 
Network Working Group A. Phillips, Ed. Network Working Group A. Phillips, Ed.
Internet-Draft Yahoo! Inc Internet-Draft Yahoo! Inc
Obsoletes: 3066 (if approved) M. Davis, Ed. Obsoletes: 3066 (if approved) M. Davis, Ed.
Expires: August 10, 2006 Google Expires: August 27, 2006 Google
February 6, 2006 February 23, 2006
Matching of Language Tags Matching of Language Tags
draft-ietf-ltru-matching-09 draft-ietf-ltru-matching-10
Status of this Memo Status of this Memo
By submitting this Internet-Draft, each author represents that any By submitting this Internet-Draft, each author represents that any
applicable patent or other IPR claims of which he or she is aware applicable patent or other IPR claims of which he or she is aware
have been or will be disclosed, and any of which he or she becomes have been or will be disclosed, and any of which he or she becomes
aware will be disclosed, in accordance with Section 6 of BCP 79. aware will be disclosed, in accordance with Section 6 of BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF), its areas, and its working groups. Note that
skipping to change at page 1, line 35 skipping to change at page 1, line 35
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt. http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html. http://www.ietf.org/shadow.html.
This Internet-Draft will expire on August 10, 2006. This Internet-Draft will expire on August 27, 2006.
Copyright Notice Copyright Notice
Copyright (C) The Internet Society (2006). Copyright (C) The Internet Society (2006).
Abstract Abstract
This document describes different mechanisms for comparing, matching, This document describes different mechanisms for comparing, matching,
and evaluating language tags. Possible algorithms for language and evaluating language tags. Possible algorithms for language
negotiation or content selection, filtering, and lookup are negotiation or content selection, filtering, and lookup are
described. This document, in combination with RFC 3066bis (replace described. This document, in combination with RFC 3066bis (Ed.:
"3066bis" with the RFC number assigned to replace "3066bis" with the RFC number assigned to
draft-ietf-ltru-registry-14), replaces RFC 3066, which replaced RFC draft-ietf-ltru-registry-14), replaces RFC 3066, which replaced RFC
1766. 1766.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
2. The Language Range . . . . . . . . . . . . . . . . . . . . . . 4 2. The Language Range . . . . . . . . . . . . . . . . . . . . . . 4
2.1. Basic Language Range . . . . . . . . . . . . . . . . . . . 4 2.1. Basic Language Range . . . . . . . . . . . . . . . . . . . 4
2.2. Extended Language Range . . . . . . . . . . . . . . . . . 5 2.2. Extended Language Range . . . . . . . . . . . . . . . . . 5
2.3. The Language Priority List . . . . . . . . . . . . . . . . 7 2.3. The Language Priority List . . . . . . . . . . . . . . . . 5
3. Types of Matching . . . . . . . . . . . . . . . . . . . . . . 8 3. Types of Matching . . . . . . . . . . . . . . . . . . . . . . 7
3.1. Choosing a Type of Matching . . . . . . . . . . . . . . . 8 3.1. Choosing a Type of Matching . . . . . . . . . . . . . . . 7
3.2. Filtering . . . . . . . . . . . . . . . . . . . . . . . . 9 3.2. Filtering . . . . . . . . . . . . . . . . . . . . . . . . 8
3.2.1. Filtering with Basic Language Ranges . . . . . . . . . 11 3.2.1. Basic Filtering . . . . . . . . . . . . . . . . . . . 9
3.2.2. Filtering with Extended Language Ranges . . . . . . . 11 3.2.2. Extended Filtering . . . . . . . . . . . . . . . . . . 10
3.2.3. Scored Filtering . . . . . . . . . . . . . . . . . . . 11 3.3. Lookup . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.3. Lookup . . . . . . . . . . . . . . . . . . . . . . . . . . 15 4. Other Considerations . . . . . . . . . . . . . . . . . . . . . 14
4. Other Considerations . . . . . . . . . . . . . . . . . . . . . 19 4.1. Choosing Language Ranges . . . . . . . . . . . . . . . . . 14
4.1. Choosing Language Ranges . . . . . . . . . . . . . . . . . 19 4.2. Meaning of Language Tags and Ranges . . . . . . . . . . . 15
4.2. Meaning of Language Tags and Ranges . . . . . . . . . . . 20 4.3. Considerations for Private Use Subtags . . . . . . . . . . 15
4.3. Considerations for Private Use Subtags . . . . . . . . . . 21 4.4. Length Considerations in Matching . . . . . . . . . . . . 15
4.4. Length Considerations in Matching . . . . . . . . . . . . 22 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 17
5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 24 6. Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
6. Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 7. Security Considerations . . . . . . . . . . . . . . . . . . . 19
7. Security Considerations . . . . . . . . . . . . . . . . . . . 26 8. Character Set Considerations . . . . . . . . . . . . . . . . . 20
8. Character Set Considerations . . . . . . . . . . . . . . . . . 27 9. References . . . . . . . . . . . . . . . . . . . . . . . . . . 21
9. References . . . . . . . . . . . . . . . . . . . . . . . . . . 28 9.1. Normative References . . . . . . . . . . . . . . . . . . . 21
9.1. Normative References . . . . . . . . . . . . . . . . . . . 28 9.2. Informative References . . . . . . . . . . . . . . . . . . 21
9.2. Informative References . . . . . . . . . . . . . . . . . . 28 Appendix A. Acknowledgements . . . . . . . . . . . . . . . . . . 22
Appendix A. Acknowledgements . . . . . . . . . . . . . . . . . . 29 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 23
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 30 Intellectual Property and Copyright Statements . . . . . . . . . . 24
Intellectual Property and Copyright Statements . . . . . . . . . . 31
1. Introduction 1. Introduction
Human beings on our planet have, past and present, used a number of Human beings on our planet have, past and present, used a number of
languages. There are many reasons why one would want to identify the languages. There are many reasons why one would want to identify the
language used when presenting or requesting information. language used when presenting or requesting information or in some
specific set of information items or "content".
Information about a user's language preferences commonly needs to be
identified so that appropriate processing can be applied. For
example, the user's language preferences in a browser can be used to
select web pages appropriately. Language preferences can also be
used to select among tools (such as dictionaries) to assist in the
processing or understanding of content in different languages.
Given a set of language identifiers, such as those defined in One use for language identifiers, such as those defined in
[RFC3066bis], various mechanisms can be envisioned for performing [RFC3066bis], is to select content by matching the associated
language negotiation and tag matching. language tags to a user's language preferences.
This document defines a syntax (called a language range (Section 2)) This document defines a syntax (called a language range (Section 2))
for specifying a user's language preferences, as well as several for specifying items in the user's language preferences (called a
schemes for selecting or filtering content by comparing language language priority list (Section 2.3)), as well as several schemes for
ranges to the language tags [RFC3066bis] used to identify the natural selecting or filtering sets of content by comparing the content's
language of that content. Applications, protocols, or specifications language tags to the user's preferences. Applications, protocols, or
will have varying needs and requirements that affect the choice of a specifications will have varying needs and requirements that affect
suitable matching scheme. Depending on the choice of scheme, there the choice of a suitable matching scheme. Depending on the choice of
are various options left to the implementation. Protocols that scheme, there are various options left to the implementation.
implement a matching scheme either need to specify each particular Protocols that implement a matching scheme either need to specify
choice or indicate the options that are left to the implementation to each particular choice or indicate the options that are left to the
decide. implementation to decide.
This document is divided into three main sections. One describes how This document is divided into three main sections. One describes how
to indicate a user's preferences using language ranges. Then a to indicate a user's preferences using language ranges. Then a
section describes various schemes for matching these ranges to a set section describes various schemes for matching these ranges to a set
of language tags in order to select specific content. There is also of language tags. There is also a section that deals with various
a section that deals with various practical considerations that apply practical considerations that apply to implementing and using these
to implementing and using these schemes. schemes.
This document, in combination with [RFC3066bis] (Ed.: replace This document, in combination with [RFC3066bis] (Ed.: replace
"3066bis" globally in this document with the RFC number assigned to "3066bis" globally in this document with the RFC number assigned to
draft-ietf-ltru-registry-14), replaces [RFC3066], which replaced draft-ietf-ltru-registry-14), replaces [RFC3066], which replaced
[RFC1766]. [RFC1766].
The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in [RFC2119]. document are to be interpreted as described in [RFC2119].
skipping to change at page 4, line 21 skipping to change at page 4, line 21
HTTP/1.1 [RFC2616] describes one such mechanism in its discussion of HTTP/1.1 [RFC2616] describes one such mechanism in its discussion of
the Accept-Language header (Section 14.4), which is used when the Accept-Language header (Section 14.4), which is used when
selecting content from servers based on the language of that content. selecting content from servers based on the language of that content.
When selecting content according to its language, it is useful to When selecting content according to its language, it is useful to
have a mechanism for identifying sets of language tags that share have a mechanism for identifying sets of language tags that share
specific attributes. This allows users to select or filter content specific attributes. This allows users to select or filter content
based on specific requirements. Such an identifier is called a based on specific requirements. Such an identifier is called a
"Language Range". "Language Range".
Language ranges are similar in structure and content to language There are different types of language range, whose specific
tags: they consist of alphanumeric "subtags" separated by hyphens, attributes vary to match their application. Language ranges are
plus a special subtag consisting of the character "*" (%2A, similar in content to language tags: they consist of a sequence of
ASTERISK), which is used in ranges as a "wildcard", that is, a value subtags separated by hyphens. In a language range, each subtag MUST
that matches any subtag. either be a sequence of ASCII alphanumeric characters or the single
character '*' (%2A, ASTERISK). The character '*' is a "wildcard"
that matches any sequence of subtags. Restrictions on the meaning
and use of wildcards vary according to the type of language range.
Language tags and thus language ranges are to be treated as case- Language tags and thus language ranges are to be treated as case-
insensitive: there exist conventions for the capitalization of some insensitive: there exist conventions for the capitalization of some
of the subtags, but these MUST NOT be taken to carry meaning. of the subtags, but these MUST NOT be taken to carry meaning.
Matching of language tags to language ranges MUST be done in a case- Matching of language tags to language ranges MUST be done in a case-
insensitive manner as well. insensitive manner.
2.1. Basic Language Range 2.1. Basic Language Range
A "basic language range" identifies the set of content whose language A "basic language range" identifies the set of language tags that all
tags begin with the same sequence of subtags. Each range consists of begin with the same sequence of subtags. Each range consists of a
a sequence of alphanumeric subtags separated by hyphens. The basic sequence of alphanumeric subtags separated by hyphens. The basic
language range is defined by the following ABNF[RFC4234]: language range is defined by the following ABNF[RFC4234]:
language-range = language-tag / "*" language-range = (1*8ALPHA *("-" 1*8alphanum)) / "*"
language-tag = 1*8[alphanum] *["-" 1*8alphanum]
alphanum = ALPHA / DIGIT alphanum = ALPHA / DIGIT
Basic language ranges (originally described by HTTP/1.1 [RFC2616] and Basic language ranges (originally described by HTTP/1.1 [RFC2616] and
later [RFC3066]) have the same syntax as an [RFC3066] language tag or later [RFC3066]) have the same syntax as an [RFC3066] language tag or
are the single character "*". They differ from the language tags are the single character "*". They differ from the language tags
defined in [RFC3066bis] only in that there is no requirement that defined in [RFC3066bis] only in that there is no requirement that
they be "well-formed" or be validated against the IANA Language they be "well-formed" or be validated against the IANA Language
Subtag Registry (although such ill-formed ranges will probably not Subtag Registry (although such ill-formed ranges will probably not
match anything). match anything). (Note that the ABNF [RFC4234] in [RFC2616] is
incorrect, since it disallows the use of digits anywhere in the
'language-range': this is mentioned in the errata)
Use of a basic language range seems to imply that there is a semantic Use of a basic language range seems to imply that there is a semantic
relationship between language tags that share the same prefix. While relationship between language tags that share the same prefix. While
this is often the case, it is not always true and users should note this is often the case, it is not always true and users should note
that the set of language tags that match a specific language-range that the set of language tags that match a specific language range
may not be mutually intelligible. may not represent mutually intelligible languages.
2.2. Extended Language Range 2.2. Extended Language Range
A Basic Language Range does not always provide the most appropriate Basic language ranges allow users to specify a set of language tags
way to specify a user's preferences. Sometimes it is beneficial to that share the same initial subtags. Occasionally users will wish to
use a more fine-grained matching scheme that takes advantage of the select a set of language tags based on the presence of specific
internal structure of language tags. This allows the user to subtags. For example, a user might wish to select all language tags
specify, for example, the value of a specific field in a language tag that contains the region subtag 'CH'. Extended language ranges are
or to indicate which values are of interest in filtering or selecting useful in specifying a particular sequence of subtags that appear in
the content. the set of matching tags without having to specify all of the
intervening subtags.
In an extended language range, the identifier takes the form of a
series of subtags which MUST consist of well-formed subtags or the
special subtag "*". For example, the language range "en-*-US"
specifies a primary language of 'en', followed by any script subtag,
followed by the region subtag 'US'.
An extended language range can be represented by the following ABNF: An extended language range can be represented by the following ABNF:
extended-language-range = range ; a range extended-language-range = (1*8ALPHA / "*")
/ privateuse ; a private-use range *("-" (1*8alphanum / "*"))
/ grandfathered ; a grandfathered registration
range = (language
["-" script]
["-" region]
*("-" variant)
*("-" extension)
["-" privateuse])
language = (2*3ALPHA [ extlang ]) ; shortest ISO 639 code
/ 4ALPHA ; reserved for future use
/ 5*8ALPHA ; registered language subtag
/ "*" ; or wildcard
extlang = *2("-" 3ALPHA) ("-" ( 3ALPHA / "*"))
; reserved for future use
; wildcard can only appear
; at the end
script = 4ALPHA ; ISO 15924 code
/ "*" ; or wildcard
region = 2ALPHA ; ISO 3166 code
/ 3DIGIT ; UN M.49 code
/ "*" ; or wildcard
variant = 5*8alphanum ; registered variants
/ (DIGIT 3alphanum) ;
/ "*" ; or wildcard
extension = singleton *("-" (2*8alphanum)) [ "-*" ]
; extension subtags
; wildcard can only appear
; at the end
singleton = %x41-57 / %x59-5A / %x61-77 / %x79-7A / DIGIT
; single letters (except for "x") or digits
privateuse = "x" 1*("-" (1*8alphanum)) Figure 2: Extended Language Range
grandfathered = 1*3ALPHA 1*2("-" (2*8alphanum)) The wildcard subtag '*' MAY occur in any position in the extended
; grandfathered registration language range, where it matches any sequence of subtags that might
; Note: I is the only singleton occur in that position in a language tag. However wildcards outside
; that starts a grandfathered tag the first position in an extended language range are ignored by most
matching schemes. Use of multiple wildcards SHOULD NOT be taken to
imply that a certain number of subtags will appear in the matching
set of language tags.
alphanum = (ALPHA / DIGIT) ; letters and numbers Implementations that specify basic ranges MAY map extended language
A field not present in the middle of an extended language range is ranges to basic language ranges: if the first subtag is a "*" then
treated as if the field contained a "*". Implementations that the entire range is treated as "*" (which matches the default
normalize extended language ranges SHOULD expand missing fields to be content), otherwise each wildcard subtag is removed. For example, if
"*" so that the semantic meaning of the language range is clear to the language range were "en-*-US", then the range would be mapped to
the user. At the same time, multiple wildcards in a row are "en-US".
redundant and implementations SHOULD collapse these to a single
wildcard when normalizing the range (for brevity). For example, both
the range "sl-nedis" and the range "sl-*-*-nedis" are equivalent to
and should be normalized as "sl-*-nedis".
2.3. The Language Priority List 2.3. The Language Priority List
When users specify a language preference they often need to specify a When users specify a language preference they often need to specify a
prioritized list of language ranges in order to best reflect their prioritized list of language ranges in order to best reflect their
language preferences. This is especially true for speakers of language preferences. This is especially true for speakers of
minority languages. A speaker of Breton in France, for example, may minority languages. A speaker of Breton in France, for example, may
specify "be" followed by "fr", meaning that if Breton is available, specify "be" followed by "fr", meaning that if Breton is available,
it is preferred, but otherwise French is the best alternative. It it is preferred, but otherwise French is the best alternative. It
can get more complex: a speaker may wish to fall back from Skolt Sami can get more complex: a speaker may wish to fall back from Skolt Sami
skipping to change at page 7, line 38 skipping to change at page 6, line 22
14.4) and RFC 3282 [RFC3282]. A simple list of ranges, i.e. one that 14.4) and RFC 3282 [RFC3282]. A simple list of ranges, i.e. one that
contains no weighting information, is considered to be in descending contains no weighting information, is considered to be in descending
order of priority. order of priority.
The various matching operations described in this document include The various matching operations described in this document include
considerations for using a language priority list. This document considerations for using a language priority list. This document
does not define any syntax for a language priority list; defining does not define any syntax for a language priority list; defining
such a syntax is the responsibility of the protocol, application, or such a syntax is the responsibility of the protocol, application, or
implementation that uses it. When given as examples in this implementation that uses it. When given as examples in this
document, language priority lists will be shown as a quoted sequence document, language priority lists will be shown as a quoted sequence
of ranges separated by semi-colons, like this: "en; fr; zh-Hant" of ranges separated by semicolons, like this: "en; fr; zh-Hant"
(which would be read as "English before French before Chinese as (which would be read as "English before French before Chinese as
written in the Traditional script"). written in the Traditional script").
Where a language priority list provides "quality weights" for the
language ranges, such as the use of Q weights in the syntax of the
"Accept-Language" header (defined in [RFC2616], Section 14.4, and
[RFC3282]), language ranges without a weight are given values equal
to the value of the previous language range (processing from first to
last). If the first language range has no weight, it is given a
value of 1.0. Then language ranges with zero weights are removed.
For example, "fr, en;q=0.5, de, it" becomes "fr;q=1.0, en;q=0.5,
de;q=0.5, it;q=0.5". The language priority list is then sorted from
highest priority to lowest, with language ranges that share the same
weights remain in the same order as in the original language priority
list.
3. Types of Matching 3. Types of Matching
Matching language ranges to language tags can be done in a number of Matching language ranges to language tags can be done in a number of
different ways. This section describes several different matching different ways. This section describes several different matching
schemes, as well as the considerations for choosing between them. schemes, as well as the considerations for choosing between them.
Protocols and specifications SHOULD clearly indicate the particular Protocols and specifications SHOULD clearly indicate the particular
mechanism used in selecting or matching language tags. mechanism used in selecting or matching language tags.
There are two basic types of matching scheme: those that produce zero There are several types of matching scheme. This document presents
or more information items (called "filtering") and those that produce two types: those that produce zero or more information items (called
a single information item for a given request (called "lookup"). "filtering") and those that produce a single information item for a
given request (called "lookup").
A key difference between these two types of matching scheme is that Implementations or protocols MAY use different matching schemes than
the language ranges in the language priority list represent the the ones described in this document, as long as those mechanisms are
_least_ specific content one will accept as a match, while for lookup clearly specified.
operations the language ranges represent the _most_ specific content.
3.1. Choosing a Type of Matching 3.1. Choosing a Type of Matching
Applications, protocols, and specifications are faced with the Applications, protocols, and specifications are faced with the
decision of what type of matching to use. Sometimes, different decision of what type of matching to use. Sometimes, different
styles of matching might be suited for different kinds of processing styles of matching might be suited for different kinds of processing
within a particular application or protocol. within a particular application or protocol.
Language tag matching is a tool, and does not by itself specify a Language tag matching is a tool, and does not by itself specify a
complete procedure for the use of language tags. Such procedures are complete procedure for the use of language tags. Such procedures are
intimately tied to the application protocol in which they occur. intimately tied to the application protocol in which they occur.
When specifying a protocol operation using matching, the protocol When specifying a protocol operation using matching, the protocol
MUST specify: MUST specify:
o Which type(s) of language tag matching it uses o Which type(s) of language tag matching it uses
o Whether the operation returns a single result (lookup) or a o Whether the operation returns a single result (lookup) or a
possibly empty set of results (filtering) possibly empty set of results (filtering)
o For lookup, what the result is when no matching tag is found. For o For lookup, what the result is when no matching tag is found. For
instance, a protocol might result in failure of the operation, an instance, a protocol might define the result as failure of the
empty value, returning some protocol defined or implementation operation, an empty value, returning some protocol defined or
defined default, or returning i-default [RFC2277]. implementation defined default, or returning i-default [RFC2277].
Filtering can be used to produce a set of results (such as a This document describes three types of matching:
collection of documents). For example, if using a search engine, one
might use filtering to limit the results to documents written in
French. It can also be used when deciding whether to perform a
language-sensitive process on some content. For example, a process
might cause paragraphs whose language tag matched the language range
"nl" to be displayed in italics within a document.
This document describes four types of matching (three types of 1. Basic Filtering (Section 3.2.1) matches a language priority list
filtering, plus the lookup scheme): consisting of basic language ranges (Section 2.1) to sets of
language tags.
1. Basic Filtering (Section 3.2.1) is used to match content using 2. Extended Filtering (Section 3.2.2) matches a language priority
basic language ranges (Section 2.1). list consisting of extended language ranges (Section 2.2) to sets
of language tags.
2. Extended Range Filtering (Section 3.2.2) is used to match content 3. Lookup (Section 3.3) matches a language priority list consisting
using extended language ranges (Section 2.2). of basic language ranges to sets of language tags find the
_exactly_ one language tag that best matches the range.
3. Scored Filtering (Section 3.2.3) produces an ordered set of Both types of filtering can be used to produce a set of results (such
content using extended language ranges. It SHOULD be used when as a collection of documents) by comparing the user's preferences to
the quality of the match within a specific language range is language tags associated with the set of content. For example, when
important, as when presenting a list of documents resulting from performing a search, one might use filtering to limit the results to
a search. documents tagged as being written in French. They might also be used
when deciding whether to perform a language-sensitive process on some
content. For example, a process might cause paragraphs whose
language tag matched the language range "nl" to be displayed in
italics within a document.
4. Lookup (Section 3.3) is used when each request needs to produce Lookup produces the single result that best matches a given set of
_exactly_ one piece of content. For example, if a process were user preferences, so it is useful in cases in which only a single
to insert a human readable error message into a protocol header, item can be returned. For example, if a process were to insert a
it might select the text based on the user's language preference. human readable error message into a protocol header, it might select
Since it can return only one item, it must choose a single item the text based on the user's language priority list. Since the
and it must return some item, even if no content matches the process can return only one item, it must choose a single item and it
must return some item, even if no content's language tag matches the
language priority list supplied by the user. language priority list supplied by the user.
Most types of matching in this document are designed so that The types of matching in this document are designed so that
implementations are not required to validate or understand any of the implementations are not required to validate or understand any of the
semantics of the subtags supplied and, except for scored filtering, semantics of the language tags or ranges or of the subtags in them.
they do not need access to the IANA Language Subtag Registry (see None of them require access to the IANA Language Subtag Registry (see
Section 3 in [RFC3066bis]). This simplifies and speeds the Section 3 in [RFC3066bis]). This simplifies and speeds the
performance of implementations. performance of implementations.
Regardless of the matching scheme chosen, protocols and Regardless of the matching scheme chosen, protocols and
implementations MAY canonicalize language tags and ranges by mapping implementations MAY canonicalize language tags and ranges by mapping
grandfathered and obsolete tags or subtags into modern equivalents. grandfathered and obsolete tags or subtags into modern equivalents.
If an implementation canonicalizes either ranges or tags, then the If an implementation canonicalizes either ranges or tags, then the
implementation will require the IANA Language Subtag Registry implementation will require the IANA Language Subtag Registry
information for that purpose. Implementations MAY also use semantic information for that purpose. Implementations MAY also use semantic
information external to the registry when matching tags. For information external to the registry when matching tags. For
example, the primary language subtags 'nn' (Nynorsk Norwegian) and example, the primary language subtags 'nn' (Nynorsk Norwegian) and
'nb' (Bokmal Norwegian) might both be usefully matched to the more 'nb' (Bokmal Norwegian) might both be usefully matched to the more
general subtag 'no' (Norwegian). Or an implementation might infer general subtag 'no' (Norwegian). Or an implementation might infer
that content labeled "zh-CN" is more likely to match the range "zh- that content labeled "zh-CN" is more likely to match the range "zh-
Hans" than equivalent content labeled "zh-TW". Hans" than equivalent content labeled "zh-TW".
3.2. Filtering 3.2. Filtering
Filtering is used to select the set of content that matches a given Filtering is used to select the set of language tags that matches a
language priority list. It is called "filtering" because this set of given language priority list and return the associated content. It
content may contain no items at all or it may return an arbitrarily is called "filtering" because this set might contain no items at all
large number of matching items: as many items as match the language or it might return an arbitrarily large number of matching items: as
priority list, thus "filtering out" the non-matching items. many items as match the language priority list, thus "filtering out"
the non-matching items.
In filtering, the language range represents the _least_ specific In filtering, the language range represents the _least_ specific
(that is, the fewest number of subtags) language tag which is an (that is, the fewest number of subtags) language tag which is an
acceptable match. That is, all of the language tags in the set of acceptable match. All of the language tags in the matching set of
filtered content will have an equal or greater number of subtags than tags will have an equal or greater number of subtags than the
the language range. For example, if the language priority list language range. Every non-wildcard subtag in the language range will
consists of the range "de-CH", one might see matching content with appear in every one of the matching language tags. For example, if
the tag "de-CH-1996" but one will never see a match with the tag the language priority list consists of the range "de-CH", one might
"de". see tags such as "de-CH-1996" but one will never see a tag such as
"de" (because the 'CH' subtag is missing).
If the language priority list (see Section 2.3) contains more than If the language priority list (see Section 2.3) contains more than
one range, the content returned is typically ordered in descending one range, the content returned is typically ordered in descending
level of preference. level of preference.
Some examples where filtering might be appropriate include: Some examples of applications where filtering might be appropriate
include:
o Applying a style to sections of a document in a particular set of o Applying a style to sections of a document in a particular set of
languages. languages.
o Displaying the set of documents containing a particular set of o Displaying the set of documents containing a particular set of
keywords written in a specific set of languages. keywords written in a specific set of languages.
o Selecting all email items written in a specific set of languages. o Selecting all email items written in a specific set of languages.
Filtering can produce either an ordered or an unordered set of The content returned MAY either be ordered or unordered according to
results. For example, applying formatting to a document based on the the priority in the language priority list (and other criteria),
language of specific pieces of content does not require the content according to the needs of the application or protocol.
to be ordered. It is sufficient to know whether a specific piece of
content is selected by the language priority list (or not). A search
application, on the other hand, probably would want to order the
results.
If an ordered set is desired, as described above, then the
application or protocol needs to determine the relative "quality" of
the match between different language tags and the language range.
This measurement is called a "distance metric". A distance metric
assigns a numeric value to the comparison of a language tag to a
language range that represents the 'distance' between the two. A
distance of zero means that they are identical, a small distance
indicates that they are very similar, and a large distance indicates
that they are very different. Using a distance metric,
implementations can, for example, allow users to select a threshold
distance for a match to be "successful" while filtering, or they
might use the numeric values to order the results.
3.2.1. Filtering with Basic Language Ranges 3.2.1. Basic Filtering
When filtering using basic language ranges, each basic language range When filtering using basic language ranges, each basic language range
in the language priority list is considered in turn, according to in the language priority list is considered in turn, according to
priority. A particular language tag matches a language range if it priority. A particular language tag matches a language range if it
exactly equals the tag, or if it exactly equals a prefix of the tag exactly equals the tag, or if it exactly equals a prefix of the tag
such that the first character following the prefix is "-". (That is, such that the first character following the prefix is "-". For
the language-range "de-de" matches the language tag "de-DE-1996", but example, the language-range "de-de" matches the language tag "de-DE-
not the language tag "de-Deva".) 1996", but not the language tags "de-Deva" or "de-Latn-DE".
The special range "*" in a language priority list matches any tag. A The special range "*" in a language priority list matches any tag. A
protocol which uses language ranges MAY specify additional rules protocol which uses language ranges MAY specify additional rules
about the semantics of "*"; for instance, HTTP/1.1 [RFC2616] about the semantics of "*"; for instance, HTTP/1.1 [RFC2616]
specifies that the range "*" matches only languages not matched by specifies that the range "*" matches only languages not matched by
any other range within an "Accept-Language" header. any other range within an "Accept-Language" header.
3.2.2. Filtering with Extended Language Ranges 3.2.2. Extended Filtering
When filtering using extended language ranges, each extended language When filtering using extended language ranges, each extended language
range in the language priority list is considered in turn, according range in the language priority list is considered in turn, according
to priority. The subtags in each extended language range are to priority. A particular language range is compared to each
compared to the corresponding subtags in the language tag being language tag using the following process:
examined. The subtag from the range is considered to match if it
exactly matches the corresponding subtag in the tag or the range's Compare the first subtag in the extended language tag to the first
subtag has the value "*" (which matches all subtags, including the subtag in the language tag in a case insensitive manner. If the
empty subtag). first subtag in the range is "*", it matches any value. Otherwise
the two values must match or the overall match fails.
Take each non-wildcard subtag in the language range and compare it to
the next subtag in the language tag in turn until a matching subtag
is found or the langauge tag is exhausted. If the end of the
language tag is found first, the match fails. If a match is found,
this step is repeated with the next non-wildcard subtag in the
language range (and beginning with the next subtag in the language
tag) until the list of subtags in the language range is exhausted or
the match fails.
Subtags not specified, including those at the end of the language Subtags not specified, including those at the end of the language
range, are assigned the wildcard value "*". This makes each range range, are thus treated as if assigned the wildcard value "*".
into a prefix much like that used in basic language range matching. Extended filtering works, therefore, much like basic filtering. For
For example, the extended language range "de-*-DE" matches all of the example, the extended language range "de-*-DE" matches all of the
following tags, in part because the unspecified variant, extension, following tags:
and private-use subtags are expanded to "*":
de-DE de-DE
de-Latn-DE de-Latn-DE
de-Latf-DE de-Latf-DE
de-DE-x-goethe de-DE-x-goethe
de-Latn-DE-1996 de-Latn-DE-1996
3.2.3. Scored Filtering
Both basic and extended language range filtering produce simple
boolean matches between a language range and a language tag.
Sometimes it may be useful to provide an array of results with
different levels of matching, for example, sorting results based on
the overall "quality" of the match. Scored (or "distance metric")
filtering provides a way to generate these quality values.
As with the other forms of filtering, the process considers each
language range in the language priority list in order of priority.
Each extended language range and language tag MUST first be
canonicalized by mapping grandfathered and obsolete tags into modern
equivalents. This requires the information in the IANA Language
Subtag Registry (see Section 3 of [RFC3066bis]).
The language range and each language tag it is to be compared to are
then transformed into a "quintuple" consisting of five "elements" in
the form (language, script, country, variant, extension).
Any extended language subtags are considered part of the language
"element". For example, the language element for the tag "zh-cmn-
Hans" would be "zh-cmn".
Private-use subtag sequences are considered part of the language
"element" if in the initial position in the tag and part of the
variant "element" if not. The different handling of private-use
sequences prevents a range such as "x-twain" from matching all
possible tags, while a range such as "en-US-x-twain" would closely
match nearly all tags for English as used in the United States.
Language subtags 'und', 'mul', and the script subtag 'Zyyy' are
converted to "*": these subtag values represent undetermined,
multiple, or private-use values which are consistent with the use of
the wildcard.
For language tags that have no script subtag but whose language
subtag's record in the IANA Language Subtag Registry contains the
field "Suppress-Script", the script element in the quintuple MUST be
set to the script subtag in the Suppress-Script field. This is
necessary because [RFC3066bis] strongly recommends that users not use
this subtag to form language tags and this document (see Section 4.1)
recommends that users not use them to form ranges. Languages which
have a "Suppress-Script" field in the registry are predominantly
written in that single script, making the subtag redundant in forming
a language tag or range. Thus if the script were not expanded in
this manner, a range such as "de-DE" would produce a more-distant
score for content that happened to be labeled "de-Latn-DE" than users
would expect that it should.
Any remaining missing components in the language tag are set to "*";
thus an empty language tag becomes the quintuple ("*", "*", "*", "*",
"*"). Missing components in the language range are handled similarly
to extended range lookup: missing internal subtags are expanded to
"*". Missing end subtags are expanded as the empty string. Thus a
pattern "en-US" becomes the quintuple ("en","*","US","","").
Here are some examples of language tags, showing their quintuples as
both language tags and language ranges:
en-US
Tag: (en, *, US, *, *)
Range: (en, *, US, "", "")
sr-Latn
Tag: (sr, Latn, *, *, *)
Range: (sr, Latn, "", "", "")
zh-cmn-Hant
Tag: (zh-cmn, Hant, *, *, *)
Range: (zh-cmn, Hant, "", "", "")
x-foo
Tag: (x-foo, *, *, *, *)
Range: (x-foo, "", "", "", "")
en-x-foo
Tag: (en, *, *, x-foo, *)
Range: (en, *, *, x-foo, "")
i-default
Tag: (i-default, *, *, *, *)
Range: (i-default, "", "", "", "")
sl-Latn-IT-rozaj
Tag: (sl, Latn, IT, rozaj, *)
Range: (sl, Latn, IT, rozaj, "")
zh-r-wadegile (hypothetical)
Tag: (zh, *, *, *, r-wadegile)
Range: (zh, *, *, *, r-wadegile)
Figure 3: Examples of Distance Metric Quintuples
Each pair of quintuples being compared is assigned a distance value,
in which small values indicate better matches and large values
indicate worse ones. The distance between the pair is the sum of the
distances for each of the corresponding elements of the quintuple.
If the elements are identical or one is '*', then the distance value
between them is zero. Otherwise, it is given by the following table:
256 language mismatch
128 script mismatch
32 region mismatch
4 variant mismatch
1 extension mismatch
A value of 0 is a perfect match; 421 is no match at all. Different
threshold values might be appropriate for different applications or
protocols. Implementations will usually allow users to choose the
most appropriate selection value, ranking the matched items based on
score.
Examples of various tag's distances from the range "en-US":
"fr-FR" 384 (language & region mismatch)
"fr" 256 (language mismatch, region match)
"en-GB" 32 (region mismatch)
"en-Latn-US" 0 (all fields match)
"en-Brai" 32 (region mismatch)
"en-US-x-foo" 4 (variant mismatch: range is the empty string)
"en-US-r-wadegile" 1 (extension mismatch: range is the empty string)
Where a language priority list follows the syntax of the "Accept-
Language" header defined in [RFC2616] (see Section 14.4) and
[RFC3282], language ranges without a Q value are given values equal
to the value of the previous language range in the list (processing
from first to last). If the first language range has no Q value, it
is given a value of 1.0. Language ranges with Q values of zero are
removed. For example, "fr, en;q=0.5, de, it" becomes
"fr;q=1.0,en;q=0.5,de;q=0.5,it;q=0.5". The distance values given
above are then divided by the Q values. For example, if that
language tag "fr-FR" has a distance of 384 from a language range with
a Q value of 0.8, then the resulting distance is 480 (384 div 0.8).
Implementations or protocols MAY use different weighting systems than
the ones described above, as long as the weightings and weighting
mechanisms are clearly specified. Thus, for example, an
implementation or protocol could give all language tags with missing
Q values a value of 1.0, or give the distance value 1000 to a
language mismatch. They MAY also use more sophisticated weights that
depend on the values of the corresponding elements. For example, an
implementation might give a small distance to the difference closely
related subtags. Some examples of closely related subtags might be:
Language:
no (Norwegian)
nb (Bokmal Norwegian)
nn (Nynorsk Norwegian)
Script:
Kata (katakana)
Hira (hiragana)
Region:
US (United States of America)
UM (United States Minor Outlying Islands)
Figure 6: Examples of Closely Related Subtags
3.3. Lookup 3.3. Lookup
Lookup is used to select the single information item that best Lookup is used to select the single language tag that best matches
matches the language priority list for a given request. When the language priority list for a given request and return the
performing lookup, each language range in the language priority list associated content. When performing lookup, each language range in
is considered in turn, according to priority. By contrast with the language priority list is considered in turn, according to
filtering, each language ranges represents the _most_ specific tag priority. By contrast with filtering, each language range represents
which is an acceptable match. The first information item found with the _most_ specific tag which is an acceptable match. The first
a matching tag, according the user's priority, is considered the content found with a matching tag, according to the user's priority,
closest match and is the item returned. For example, if the language is considered the closest match and is the content returned. For
range is "de-CH", one might expect to receive an information item example, if the language range is "de-ch", a lookup operation might
with the tag "de" but never one with the tag "de-CH-1996". Usually produce content with the tags "de" or "de-CH" but never one with the
if no content matches the request, a "default" item is returned. tag "de-CH-1996". Usually if no content matches the request, the
"default" content is returned.
For example, if an application inserts some dynamic content into a For example, if an application inserts some dynamic content into a
document, returning an empty string if there is no exact match is not document, returning an empty string if there is no exact match is not
an option. Instead, the application "falls back" until it finds a an option. Instead, the application "falls back" until it finds a
suitable piece of content to insert. Other examples of lookup might matching language tag associated with a suitable piece of content to
include: insert. Examples of lookup might include:
o Selection of a template containing the text for an automated email o Selection of a template containing the text for an automated email
response. response.
o Selection of a item containing some text for inclusion in a o Selection of a item containing some text for inclusion in a
particular Web page. particular Web page.
o Selection of a string of text for inclusion in an error log. o Selection of a string of text for inclusion in an error log.
In the lookup scheme, the language range is progressively truncated In the lookup scheme, the language range is progressively truncated
from the end until a matching piece of content is located. For from the end until a matching piece of content is located. For
example, starting with the range "zh-Hant-CN-x-private", the lookup example, starting with the range "zh-Hant-CN-x-private", the lookup
progressively searches for content as shown below: progressively searches for content as shown below:
Range to match: zh-Hant-CN-x-private Range to match: zh-Hant-CN-x-private
1. zh-Hant-CN-x-private 1. zh-Hant-CN-x-private
2. zh-Hant-CN 2. zh-Hant-CN
3. zh-Hant 3. zh-Hant
4. zh 4. zh
5. (default content or the empty tag) 5. (default content)
Figure 7: Example of a Lookup Fallback Pattern Figure 3: Example of a Lookup Fallback Pattern
This scheme allows some flexibility in finding content. For example, This scheme allows some flexibility in finding a match. For example,
it provides better results for cases in which data is not available lookup provides better results for cases in which content is not
that exactly matches the user request than if the default language available that exactly matches the user request than if the default
for the system or content were returned immediately. Not every language for the system or content were returned immediately. Not
specific level of tag granularity is usually available or language every specific level of tag granularity is usually available or
content may be sparsely populated, so "falling back" through the language content may be sparsely populated, so "falling back" through
subtag sequence provides more opportunity to find a match between the subtag sequence provides more opportunity to find a match between
available content and the user's request. available language tags and the user's request.
The default content is implementation defined. It might be content The default behavior when no tag matches the language priority list
with no language tag; might have an empty value (the built-in is implementation defined. An implementation might, for example,
attribute xml:lang in [XML10] permits the empty value); might be a return content with no language tag; might supply content with an
particular language designated for that bit of content; or it might empty language tag value (the built-in attribute xml:lang in [XML10]
be content that is labeled with the tag "i-default" (see [RFC2277]). permits the empty value); might be a particular language designated
When performing lookup using a language priority list, the for the bit of content being selected; or it might select the tag
progressive search MUST proceed to consider each language range in "i-default" (see [RFC2277]). When performing lookup using a language
the list before finding the default content or empty tag. priority list, the progressive search MUST proceed to consider each
language range in the list before finding the default content or
empty tag.
One common way for an application or implementation to provide for One common way for an application or implementation to provide for a
default content is to allow a specific language range to be set as default is to allow a specific language range to be set as the
the default for a specific type of request. This language range is default for a specific type of request. This language range is then
then treated as if it were appended to the end of the language treated as if it were appended to the end of the language priority
priority list as a whole, rather than after each item in the language list as a whole, rather than after each item in the language priority
priority list. list.
For example, if a particular user's language priority list were For example, if a particular user's language priority list were
"fr-FR; zh-Hant" and the program doing the matching had a default "fr-FR; zh-Hant" and the program doing the matching had a default
language range of "ja-JP", the program would search for content as language range of "ja-JP", the program would search for content as
follows: follows:
1. fr-FR 1. fr-FR
2. fr 2. fr
3. zh-Hant // next language 3. zh-Hant // next language
4. zh 4. zh
5. (search for the default content) 5. (search for the default content)
a. ja-JP a. ja-JP
b. ja b. ja
c. (implementation defined default) c. (implementation defined default)
Figure 8: Lookup Using a Language Priority List Figure 4: Lookup Using a Language Priority List
Implementations SHOULD ignore extensions and unrecognized private-use Implementations SHOULD ignore extensions and unrecognized private-use
subtags when performing lookup, since these subtags are usually subtags when performing lookup, since these subtags are usually
orthogonal to the user's request. orthogonal to the user's request.
The special language range "*" matches any language tag. In the The special language range "*" matches any language tag. In the
lookup scheme, this range does not convey enough information by lookup scheme, this range does not convey enough information by
itself to determine which content is most appropriate, since it itself to determine which content is most appropriate, since it
matches everything. If the language range "*" is the only one in the matches everything. If the language range "*" is the only one in the
language priority list, it matches the default content. If the language priority list, it matches the default content. If the
language range "*" is followed by other language ranges, it should be language range "*" is followed by other language ranges, it should be
skipping to change at page 17, line 28 skipping to change at page 13, line 9
priority list is used as input for both lookup and filtering priority list is used as input for both lookup and filtering
operations). Wildcard values in an extended language range normally operations). Wildcard values in an extended language range normally
match any value that occurs in that position in a language tag. match any value that occurs in that position in a language tag.
Since only one item can be returned for any given lookup request, Since only one item can be returned for any given lookup request,
wildcards in a language range have to be processed in a consistent wildcards in a language range have to be processed in a consistent
manner or the same request will produce widely varying results. manner or the same request will produce widely varying results.
Implementations that accept extended language ranges MUST define Implementations that accept extended language ranges MUST define
which content is returned when more than one item matches the which content is returned when more than one item matches the
extended language range. extended language range.
For example, an implementation could return the matching content that For example, an implementation could return the matching tag that is
is first in ASCII-order. For example, if the language range were first in ASCII-order. If the language range were "*-CH" and the set
"*-CH" and the set of content included "de-CH", "fr-CH", and "it-CH", of tags included "de-CH", "fr-CH", and "it-CH", then the tag "de-CH"
then the content labeled "de-CH" would be returned. would be returned. Another example would be for an implementation to
map the extended language ranges to basic ranges.
Implementations MAY also map extended language ranges to basic
language ranges: if the first subtag is a "*" then the entire range
is treated as "*" (which matches the default content), otherwise each
wildcard subtag is removed. For example, if the language range were
"en-*-US", then the range would be mapped to "en-US".
Where a language priority list contains Q values as in the syntax of
the "Accept-Language" header defined in [RFC2616] (see Section 14.4)
and [RFC3282], language tags without a Q value are given values equal
to the value of the previous language tag (processing from first to
last). If the first language tag has no Q value, it is given a value
of 1.0. Then language tags with zero Q values are removed. For
example, "fr, en;q=0.5, de, it" becomes "fr;q=1.0, en;q=0.5,
de;q=0.5, it;q=0.5". The language priority list is then sorted from
highest priority to lowest, whereby any two language tags with the
same Q values are remain in the same order as in the original
language priority list. This list is then traversed as described
above in doing lookup.
Implementations or protocols MAY use different lookup mechanisms
systems than the ones described above, as long as those mechanisms
are clearly specified.
4. Other Considerations 4. Other Considerations
When working with language ranges and matching schemes, there are When working with language ranges and matching schemes, there are
some additional points that may influence the choice of either. some additional points that may influence the choice of either.
4.1. Choosing Language Ranges 4.1. Choosing Language Ranges
Users indicate their language preferences via the choice of a Users indicate their language preferences via the choice of a
language range or the list of language ranges in a language priority language range or the list of language ranges in a language priority
skipping to change at page 19, line 44 skipping to change at page 14, line 44
When working with tags and ranges, note that extensions and most When working with tags and ranges, note that extensions and most
private-use subtags are orthogonal to language tag matching, in that private-use subtags are orthogonal to language tag matching, in that
they specify additional attributes of the text not related to the they specify additional attributes of the text not related to the
goals of most matching schemes. Users SHOULD avoid using these goals of most matching schemes. Users SHOULD avoid using these
subtags in language ranges, since they interfere with the selection subtags in language ranges, since they interfere with the selection
of available content. When used in language tags (as opposed to of available content. When used in language tags (as opposed to
ranges), these subtags normally do not interfere with filtering ranges), these subtags normally do not interfere with filtering
(Section 3), since they appear at the end of the tag and will match (Section 3), since they appear at the end of the tag and will match
all prefixes. all prefixes.
When working with language tags and language ranges note that: Private-use and Extension subtags are normally orthogonal to language
tag fallback. Implementations or specifications that use a lookup
o Private-use and Extension subtags are normally orthogonal to (Section 3.3) matching scheme often ignore unrecognized private-use
language tag fallback. Implementations or specifications that use and extension subtags when performing language tag fallback. In
a lookup (Section 3.3) matching scheme often ignore unrecognized addition, since these subtags are always at the end of the sequence
private-use and extension subtags when performing language tag of subtags, their use in language tags normally doesn't interfere
fallback. In addition, since these subtags are always at the end with the use of ranges that omit them in the filtering (Section 3.2)
of the sequence of subtags, their use in language tags normally matching schemes described below. However, they do interfere with
doesn't interfere with the use of ranges that omit them in the filtering when used in language ranges and SHOULD be avoided in
filtering (Section 3.2) matching schemes described below. ranges as a result.
However, they do interfere with filtering when used in language
ranges and SHOULD be avoided in ranges as a result.
o Applications, specifications, or protocols that choose not to Applications, specifications, or protocols that choose not to
interpret one or more private-use or extension subtags SHOULD NOT interpret one or more private-use or extension subtags SHOULD NOT
remove or modify these extensions in content that they are remove or modify these extensions in content that they are
processing. When a language tag instance is to be used in a processing. When a language tag instance is to be used in a
specific, known protocol, and is not being passed through to other specific, known protocol, and is not being passed through to other
protocols, language tags MAY be filtered to remove subtags and protocols, language tags MAY be filtered to remove subtags and
extensions that are not supported by that protocol. Such extensions that are not supported by that protocol. Such filtering
filtering SHOULD be avoided, if possible, since it removes SHOULD be avoided, if possible, since it removes information that
information that might be relevant to services on the other end of might be relevant to services on the other end of the protocol that
the protocol that would make use of that information. would make use of that information.
o Some applications of language tags might want or need to consider Some applications of language tags might want or need to consider
extensions and private-use subtags when matching tags. If extensions and private-use subtags when matching tags. If extensions
extensions and private-use subtags are included in a matching or and private-use subtags are included in a matching or filtering
filtering process that utilizes one of the schemes described in process that utilizes one of the schemes described in this document,
this document, then the implementation SHOULD canonicalize the then the implementation SHOULD canonicalize the language tags and/or
language tags and/or ranges before performing the matching. Note ranges before performing the matching. Note that language tag
that language tag processors that claim to be "well-formed" processors that claim to be "well-formed" processors as defined in
processors as defined in [RFC3066bis] generally fall into this [RFC3066bis] generally fall into this category.
category.
4.2. Meaning of Language Tags and Ranges 4.2. Meaning of Language Tags and Ranges
Selecting content using language ranges requires some understanding Selecting content using language ranges requires some understanding
by users of what they are selecting. A language tag or range by users of what they are selecting. The meaning of the various
identifies a language as spoken (or written, signed or otherwise subtags in a language range are identical to their meaning in a
signaled) by human beings for communication of information to other language tag (see Section 4.2 in [RFC3066bis]), with the addition
human beings. that the wildcard "*" represents any matching sequence of values.
If a language tag B contains language tag A as a prefix, then B is
typically "narrower" or "more specific" than A. For example, "zh-
Hant-TW" is more specific than "zh-Hant".
This relationship is not guaranteed in all cases: specifically,
languages that begin with the same sequence of subtags are NOT
guaranteed to be mutually intelligible, although they might be.
For example, the tag "az" shares a prefix with both "az-Latn"
(Azerbaijani written using the Latin script) and "az-Arab"
(Azerbaijani written using the Arabic script). A person fluent in
one script might not be able to read the other, even though the text
might be otherwise identical. Content tagged as "az" most probably
is written in just one script and thus might not be intelligible to a
reader familiar with the other script.
Variant subtags in particular seem to represent specific divisions in
mutual understanding, since they often encode dialects or other
idiosyncratic variations within a language. They also seem to
represent relatively low divisions with a high chance of at least
limited understanding, although this depends on the specific variant
in question.
The relationship between the language tag and the information it
relates to is defined by the standard describing the context in which
it appears. Accordingly, this section can only give possible
examples of its usage:
o For a single information object, the associated language tags
might be interpreted as the set of languages that are necessary
for a complete comprehension of the complete object. Example:
Plain text documents.
o For an aggregation of information objects, the associated language
tags could be taken as the set of languages used inside components
of that aggregation. Examples: Document stores and libraries.
o For information objects whose purpose is to provide alternatives,
the associated language tags could be regarded as a hint that the
content is provided in several languages, and that one has to
inspect each of the alternatives in order to find its language or
languages. In this case, the presence of multiple tags might not
mean that one needs to be multi-lingual to get complete
understanding of the document. Example: MIME multipart/
alternative.
o In markup languages, such as HTML and XML, language information
can be added to each part of the document identified by the markup
structure (including the whole document itself). For example, one
could write <span lang="FR">C'est la vie.</span> inside a
Norwegian document; the Norwegian-speaking user could then access
a French-Norwegian dictionary to find out what the marked section
meant. If the user were listening to that document through a
speech synthesis interface, this formation could be used to signal
the synthesizer to appropriately apply French text-to-speech
pronunciation rules to that span of text, instead of misapplying
the Norwegian rules.
4.3. Considerations for Private Use Subtags 4.3. Considerations for Private Use Subtags
Private-use subtags require private agreement between the parties Private-use subtags require private agreement between the parties
that intend to use or exchange language tags that use them and great that intend to use or exchange language tags that use them and great
caution SHOULD be used in employing them in content or protocols caution SHOULD be used in employing them in content or protocols
intended for general use. Private-use subtags are simply useless for intended for general use. Private-use subtags are simply useless for
information exchange without prior arrangement. information exchange without prior arrangement.
The value and semantic meaning of private-use tags and of the subtags The value and semantic meaning of private-use tags and of the subtags
used within such a language tag are not defined. Matching private- used within such a language tag are not defined. Matching private-
use tags using language ranges or extended language ranges can result use tags using language ranges or extended language ranges can result
in unpredictable content being returned. in unpredictable content being returned.
4.4. Length Considerations in Matching 4.4. Length Considerations in Matching
RFC 3066 [RFC3066] did not provide an upper limit on the size of Language ranges are very similar to language tags in terms of content
language tags or ranges. RFC 3066 did define the semantics of and usage. The same types of restrictions on length that apply to
particular subtags in such a way that most language tags or ranges language tags could also apply to language ranges. Implementation,
consisted of language and region subtags with a combined total length protocol, and specificiation authors SHOULD apply the considerations
of up to six characters. Larger tags and ranges (in terms of both in [RFC3066bis] Section 4.3 (Length Considerations) where appropriate
subtags and characters) did exist, however. to language ranges and language priority lists.
[RFC3066bis] also does not impose a fixed upper limit on the number
of subtags in a language tag or range (and thus an upper bound on the
size of either). The syntax in that document suggests that,
depending on the specific language or range of languages, more
subtags (and thus characters) are sometimes necessary as a result.
Length considerations and their impact on the selection and
processing of tags are described in Section 2.1.1 of that document.
An application or protocol MAY choose to limit the length of the
language tags or ranges used in matching. Any such limitation SHOULD
be clearly documented, and such documentation SHOULD include the
disposition of any longer tags or ranges (for example, whether an
error value is generated or the language tag or range is truncated).
If truncation is permitted it MUST NOT permit a subtag to be divided,
since this changes the semantics of the subtag being matched and can
result in false positives or negatives.
Applications or protocols that restrict storage SHOULD consider the
impact of tag or range truncation on the resulting matches. For
example, removing the "*" from the end of an extended language range
(see Section 2.2) can greatly modify the set of returned matches. A
protocol that allows tags or ranges to be truncated at an arbitrary
limit, without giving any indication of what that limit is, has the
potential for causing harm by changing the meaning of values in
substantial ways.
In practice, most tags do not require additional subtags or
substantially more characters. Additional subtags sometimes add
useful distinguishing information, but extraneous subtags interfere
with the meaning, understanding, and especially matching of language
tags. Since language tags or ranges MAY be truncated by an
application or protocol that limits storage, when choosing language
tags or ranges users and applications SHOULD avoid adding subtags
that add no distinguishing value. In particular, users and
implementations SHOULD follow the 'Prefix' and 'Suppress-Script'
fields in the registry (defined in Section 3.6 of [RFC3066bis]):
these fields provide guidance on when specific additional subtags
SHOULD (and SHOULD NOT) be used.
Implementations MUST support a limit of at least 33 characters. This
limit includes at least one subtag of each non-extension, non-private
use type. When choosing a buffer limit, a length of at least 42
characters is strongly RECOMMENDED.
The practical limit on tags or ranges derived solely from registered
values is 42 characters. Implementations MUST be able to handle tags
and ranges of this length. Support for tags and ranges of at least
62 characters in length is RECOMMENDED. Implementations MAY support
longer values, including matching extensive sets of private-use or
extension subtags.
Applications or protocols which have to truncate a tag MUST do so by
progressively removing subtags along with their preceding "-" from
the right side of the language tag until the tag is short enough for
the given buffer. If the resulting tag ends with a single-character
subtag, that subtag and its preceding "-" MUST also be removed. For
example:
Tag to truncate: zh-Latn-CN-variant1-a-extend1-x-wadegile-private1
1. zh-Latn-CN-variant1-a-extend1-x-wadegile
2. zh-Latn-CN-variant1-a-extend1
3. zh-Latn-CN-variant1
4. zh-Latn-CN
5. zh-Latn
6. zh
Figure 9: Example of Tag Truncation
5. IANA Considerations 5. IANA Considerations
This document presents no new or existing considerations for IANA. This document presents no new or existing considerations for IANA.
6. Changes 6. Changes
This is the first version of this document. This is the first version of this document.
The following changes were put into this document since draft-07:
Added a mention of "*" to the Character Set Considerations section
(D.Ewell)
7. Security Considerations 7. Security Considerations
Language ranges used in content negotiation might be used to infer Language ranges used in content negotiation might be used to infer
the nationality of the sender, and thus identify potential targets the nationality of the sender, and thus identify potential targets
for surveillance. In addition, unique or highly unusual language for surveillance. In addition, unique or highly unusual language
ranges or combinations of language ranges might be used to track a ranges or combinations of language ranges might be used to track a
specific individual's activities. specific individual's activities.
This is a special case of the general problem that anything you send This is a special case of the general problem that anything you send
is visible to the receiving party. It is useful to be aware that is visible to the receiving party. It is useful to be aware that
skipping to change at page 28, line 33 skipping to change at page 21, line 33
9.2. Informative References 9.2. Informative References
[RFC1766] Alvestrand, H., "Tags for the Identification of [RFC1766] Alvestrand, H., "Tags for the Identification of
Languages", RFC 1766, March 1995. Languages", RFC 1766, March 1995.
[RFC2616] Fielding, R., Gettys, J., Mogul, J., Frystyk, H., [RFC2616] Fielding, R., Gettys, J., Mogul, J., Frystyk, H.,
Masinter, L., Leach, P., and T. Berners-Lee, "Hypertext Masinter, L., Leach, P., and T. Berners-Lee, "Hypertext
Transfer Protocol -- HTTP/1.1", RFC 2616, June 1999. Transfer Protocol -- HTTP/1.1", RFC 2616, June 1999.
[RFC2616errata]
IETF, "HTTP/1.1 Specification Errata", 10 2004,
<http://purl.org/NET/http-errata>.
[RFC3066] Alvestrand, H., "Tags for the Identification of [RFC3066] Alvestrand, H., "Tags for the Identification of
Languages", BCP 47, RFC 3066, January 2001. Languages", BCP 47, RFC 3066, January 2001.
[RFC3282] Alvestrand, H., "Content Language Headers", RFC 3282, [RFC3282] Alvestrand, H., "Content Language Headers", RFC 3282,
May 2002. May 2002.
[XML10] Bray (et al), T., "Extensible Markup Language (XML) 1.0", [XML10] Bray (et al), T., "Extensible Markup Language (XML) 1.0",
02 2004. 02 2004.
Appendix A. Acknowledgements Appendix A. Acknowledgements
skipping to change at page 29, line 28 skipping to change at page 22, line 28
Harald Alvestrand, Jeremy Carroll, John Cowan, Martin Duerst, Frank Harald Alvestrand, Jeremy Carroll, John Cowan, Martin Duerst, Frank
Ellermann, Doug Ewell, Marion Gunn, Kent Karlsson, Ira McDonald, M. Ellermann, Doug Ewell, Marion Gunn, Kent Karlsson, Ira McDonald, M.
Patton, Randy Presuhn, Eric van der Poel, Markus Scherer, and many, Patton, Randy Presuhn, Eric van der Poel, Markus Scherer, and many,
many others. many others.
Very special thanks must go to Harald Tveit Alvestrand, who Very special thanks must go to Harald Tveit Alvestrand, who
originated RFCs 1766 and 3066, and without whom this document would originated RFCs 1766 and 3066, and without whom this document would
not have been possible. not have been possible.
For this particular document, John Cowan originated the scheme For this particular document, John Cowan originated the scoring
described in Section 3.2.3. Mark Davis originated the scheme scheme. Mark Davis originated the scheme described in Section 3.3.
described in the Section 3.3.
Authors' Addresses Authors' Addresses
Addison Phillips (editor) Addison Phillips (editor)
Yahoo! Inc Yahoo! Inc
Email: addison at inter dash locale dot com Email: addison at inter dash locale dot com
Mark Davis (editor) Mark Davis (editor)
Google Google
 End of changes. 61 change blocks. 
617 lines changed or deleted 270 lines changed or added

This html diff was produced by rfcdiff 1.29, available from http://www.levkowetz.com/ietf/tools/rfcdiff/