draft-ietf-ltru-matching-07.txt   draft-ietf-ltru-matching-08.txt 
Network Working Group A. Phillips, Ed. Network Working Group A. Phillips, Ed.
Internet-Draft Quest Software Internet-Draft Quest Software
Obsoletes: 3066 (if approved) M. Davis, Ed. Obsoletes: 3066 (if approved) M. Davis, Ed.
Expires: May 22, 2006 IBM Expires: June 10, 2006 IBM
November 18, 2005 December 7, 2005
Matching of Language Tags Matching of Language Tags
draft-ietf-ltru-matching-07 draft-ietf-ltru-matching-08
Status of this Memo Status of this Memo
By submitting this Internet-Draft, each author represents that any By submitting this Internet-Draft, each author represents that any
applicable patent or other IPR claims of which he or she is aware applicable patent or other IPR claims of which he or she is aware
have been or will be disclosed, and any of which he or she becomes have been or will be disclosed, and any of which he or she becomes
aware will be disclosed, in accordance with Section 6 of BCP 79. aware will be disclosed, in accordance with Section 6 of BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF), its areas, and its working groups. Note that
skipping to change at page 1, line 35 skipping to change at page 1, line 35
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt. http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html. http://www.ietf.org/shadow.html.
This Internet-Draft will expire on May 22, 2006. This Internet-Draft will expire on June 10, 2006.
Copyright Notice Copyright Notice
Copyright (C) The Internet Society (2005). Copyright (C) The Internet Society (2005).
Abstract Abstract
This document describes different mechanisms for comparing, matching, This document describes different mechanisms for comparing, matching,
and evaluating language tags. Possible algorithms for language and evaluating language tags. Possible algorithms for language
negotiation and content selection are described. This document, in negotiation or content selection, filtering, and lookup are
combination with RFC 3066bis (replace "3066bis" with the RFC number described. This document, in combination with RFC 3066bis (replace
assigned to draft-ietf-ltru-registry-14), replaces RFC 3066, which "3066bis" with the RFC number assigned to
replaced RFC 1766. draft-ietf-ltru-registry-14), replaces RFC 3066, which replaced RFC
1766.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
2. The Language Range . . . . . . . . . . . . . . . . . . . . . . 4 2. The Language Range . . . . . . . . . . . . . . . . . . . . . . 4
2.1. Lists of Language Ranges . . . . . . . . . . . . . . . . . 4 2.1. Basic Language Range . . . . . . . . . . . . . . . . . . . 4
2.2. Basic Language Range . . . . . . . . . . . . . . . . . . . 4 2.2. Extended Language Range . . . . . . . . . . . . . . . . . 5
2.3. Extended Language Range . . . . . . . . . . . . . . . . . 5 2.3. The Language Priority List . . . . . . . . . . . . . . . . 7
2.4. Choosing a Language Range . . . . . . . . . . . . . . . . 6 3. Types of Matching . . . . . . . . . . . . . . . . . . . . . . 8
3. Types of Matching . . . . . . . . . . . . . . . . . . . . . . 9 3.1. Choosing a Type of Matching . . . . . . . . . . . . . . . 8
3.1. Choosing a Type of Matching . . . . . . . . . . . . . . . 9 3.2. Filtering . . . . . . . . . . . . . . . . . . . . . . . . 9
3.2. Filtering . . . . . . . . . . . . . . . . . . . . . . . . 10 3.2.1. Filtering with Basic Language Ranges . . . . . . . . . 10
3.2.1. Filtering with Basic Language Ranges . . . . . . . . . 11
3.2.2. Filtering with Extended Language Ranges . . . . . . . 11 3.2.2. Filtering with Extended Language Ranges . . . . . . . 11
3.2.3. Scored Filtering . . . . . . . . . . . . . . . . . . . 12 3.2.3. Scored Filtering . . . . . . . . . . . . . . . . . . . 11
3.3. Lookup . . . . . . . . . . . . . . . . . . . . . . . . . . 14 3.3. Lookup . . . . . . . . . . . . . . . . . . . . . . . . . . 15
4. Other Considerations . . . . . . . . . . . . . . . . . . . . . 18 4. Other Considerations . . . . . . . . . . . . . . . . . . . . . 18
4.1. Meaning of Language Tags and Ranges . . . . . . . . . . . 18 4.1. Choosing Language Ranges . . . . . . . . . . . . . . . . . 18
4.2. Considerations for Private Use Subtags . . . . . . . . . . 19 4.2. Meaning of Language Tags and Ranges . . . . . . . . . . . 19
4.3. Length Considerations in Matching . . . . . . . . . . . . 19 4.3. Considerations for Private Use Subtags . . . . . . . . . . 20
5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 22 4.4. Length Considerations in Matching . . . . . . . . . . . . 21
6. Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 23
7. Security Considerations . . . . . . . . . . . . . . . . . . . 24 6. Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
8. Character Set Considerations . . . . . . . . . . . . . . . . . 25 7. Security Considerations . . . . . . . . . . . . . . . . . . . 25
9. References . . . . . . . . . . . . . . . . . . . . . . . . . . 26 8. Character Set Considerations . . . . . . . . . . . . . . . . . 26
9.1. Normative References . . . . . . . . . . . . . . . . . . . 26 9. References . . . . . . . . . . . . . . . . . . . . . . . . . . 27
9.2. Informative References . . . . . . . . . . . . . . . . . . 26 9.1. Normative References . . . . . . . . . . . . . . . . . . . 27
Appendix A. Acknowledgements . . . . . . . . . . . . . . . . . . 27 9.2. Informative References . . . . . . . . . . . . . . . . . . 27
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 28 Appendix A. Acknowledgements . . . . . . . . . . . . . . . . . . 28
Intellectual Property and Copyright Statements . . . . . . . . . . 29 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 29
Intellectual Property and Copyright Statements . . . . . . . . . . 30
1. Introduction 1. Introduction
Human beings on our planet have, past and present, used a number of Human beings on our planet have, past and present, used a number of
languages. There are many reasons why one would want to identify the languages. There are many reasons why one would want to identify the
language used when presenting or requesting information. language used when presenting or requesting information.
Information about a user's language preferences commonly need to be Information about a user's language preferences commonly needs to be
identified so that appropriate processing can be applied. For identified so that appropriate processing can be applied. For
example, the user's language preferences in a browser can be used to example, the user's language preferences in a browser can be used to
select web pages appropriately. Language preferences can also be select web pages appropriately. Language preferences can also be
used to select among tools (such as dictionaries) to assist in the used to select among tools (such as dictionaries) to assist in the
processing or understanding of content in different languages. processing or understanding of content in different languages.
Given a set of language identifiers, such as those defined in Given a set of language identifiers, such as those defined in
[RFC3066bis], various mechanisms can be envisioned for performing [RFC3066bis], various mechanisms can be envisioned for performing
language negotiation and tag matching. Applications, protocols, or language negotiation and tag matching.
specifications will have varying needs and requirements that affect
the choice of a suitable mechanism.
This document defines several mechanisms for matching, selecting, or This document defines a syntax (called a language range (Section 2))
filtering content whose natural language is identified using Language for specifying a user's language preferences, as well as several
Tags [RFC3066bis], as well as the syntax (called a "language range") schemes for selecting or filtering content by comparing language
associated with each of these mechanisms for specifying the user's ranges to the language tags [RFC3066bis] used to identify the natural
language preferences. language of that content. Applications, protocols, or specifications
will have varying needs and requirements that affect the choice of a
suitable matching scheme. Depending on the choice of scheme, there
are various options left to the implementation. Protocols that
implement a matching scheme either need to choose a particular option
or indicate that the particular options is left to the specific
implementation to decide.
This document, in combination with [RFC3066bis] (replace "3066bis" This document is divided into three main sections. One describes how
globally in this document with the RFC number assigned to to indicate a user's preferences using language ranges. Then a
section describes various schemes for matching these ranges to a set
of language tags in order to select specific content. There is also
a section that deals with various practical considerations that apply
to implementing and using these schemes.
This document, in combination with [RFC3066bis] (Ed.: replace
"3066bis" globally in this document with the RFC number assigned to
draft-ietf-ltru-registry-14), replaces [RFC3066], which replaced draft-ietf-ltru-registry-14), replaces [RFC3066], which replaced
[RFC1766]. [RFC1766].
The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in [RFC2119]. document are to be interpreted as described in [RFC2119].
2. The Language Range 2. The Language Range
Language Tags [RFC3066bis] are used to identify the language of some Language Tags [RFC3066bis] are used to identify the language of some
information item or content. Applications or protocols that use information item or content. Applications or protocols that use
language tags are often faced with the problem of identifying sets of language tags are often faced with the problem of identifying sets of
content that share certain language attributes. For example, content that share certain language attributes. For example,
HTTP/1.1 [RFC2616] describes language ranges in its discussion of the HTTP/1.1 [RFC2616] describes one such mechanism in its discussion of
Accept-Language header (Section 14.4). These are to be used when the Accept-Language header (Section 14.4), which is used when
selecting content from servers based on the language of that content. selecting content from servers based on the language of that content.
When selecting content according to its language, it is useful to When selecting content according to its language, it is useful to
have a mechanism for identifying sets of language tags that share have a mechanism for identifying sets of language tags that share
specific attributes. This allows users to select or filter content specific attributes. This allows users to select or filter content
based on specific requirements. Such an identifier is called a based on specific requirements. Such an identifier is called a
"Language Range". "Language Range".
Language tags and thus language ranges are to be treated as case Language tags and thus language ranges are to be treated as case-
insensitive: there exist conventions for the capitalization of some insensitive: there exist conventions for the capitalization of some
of the subtags, but these MUST NOT be taken to carry meaning. of the subtags, but these MUST NOT be taken to carry meaning.
Matching of language tags to language ranges MUST be done in a case Matching of language tags to language ranges MUST be done in a case-
insensitive manner as well. insensitive manner as well.
2.1. Lists of Language Ranges 2.1. Basic Language Range
When users specify a language preference they often need to specify a
prioritized list of language ranges in order to best reflect their
language preferences. This is especially true for speakers of
minority languages. A speaker of Breton in France, for example, may
specify "be" followed by "fr", meaning that if Breton is available,
it is preferred, but otherwise French is the best alternative. It
can get more complex: a speaker may wish to fall back from Skolt Sami
to Northern Sami to Finnish.
A "Language Priority List" consists of a prioritized or weighted list
of language ranges. One well known example of such a list is the
"Accept-Language" header defined in RFC 2616 [RFC2616] (see Section
14.4) and RFC 3282 [RFC3282].
The various matching operations described in this document include
considerations for using a language priority list. When given as
examples in this document, language priority lists will be shown as a
quoted sequence of ranges separated by semi-colons, like this: "en;
fr; zh-Hant" (which would be read as "English before French before
Chinese as written in the Traditional script").
2.2. Basic Language Range
A "Basic Language Range" identifies the set of content whose language A "basic language range" identifies the set of content whose language
tags begin with the same sequence of subtags. A basic language range tags begin with the same sequence of subtags. Each range consists of
is identified by its 'language range' tag, by adapting the a sequence of alphanumeric subtags separated by hyphens. The basic
ABNF[RFC4234] from HTTP/1.1 [RFC2616] : language range is defined by the following the ABNF[RFC4234]:
language-range = language-tag / "*" language-range = language-tag / "*"
language-tag = 1*8[alphanum] *["-" 1*8alphanum] language-tag = 1*8[alphanum] *["-" 1*8alphanum]
alphanum = ALPHA / DIGIT alphanum = ALPHA / DIGIT
That is, a language-range has the same syntax as a language-tag or is Basic language ranges (originally described by HTTP/1.1 [RFC2616] and
the single character "*". Basic Language Ranges imply that there is later [RFC3066]) have the same syntax as an [RFC3066] language tag or
a semantic relationship between language tags that share the same are the single character "*". They differ from the language tags
prefix. While this is often the case, it is not always true and defined in [RFC3066bis] only in that there is no requirement that
users should note that the set of language tags that match a specific they be "well-formed" or be validated against the IANA Language
language-range may not be mutually intelligible. Subtag Registry (although such ill-formed ranges will probably not
match anything).
Basic language ranges were originally described in [RFC3066] and Use of a basic language range seems to imply that there is a semantic
HTTP/1.1 [RFC2616] (where they are referred to as simply a "language relationship between language tags that share the same prefix. While
range"). this is often the case, it is not always true and users should note
that the set of language tags that match a specific language-range
may not be mutually intelligible.
2.3. Extended Language Range 2.2. Extended Language Range
A Basic Language Range does not always provide the most appropriate A Basic Language Range does not always provide the most appropriate
way to specify a user's preferences. Sometimes it is beneficial to way to specify a user's preferences. Sometimes it is beneficial to
use a more granular matching scheme that takes advantage of the use a more fine-grained matching scheme that takes advantage of the
internal structure of language tags, by allowing the user to specify, internal structure of language tags. This allows the user to
for example, the value of a specific field in a language tag or to specify, for example, the value of a specific field in a language tag
indicate which values are of interest in filtering or selecting the or to indicate which values are of interest in filtering or selecting
content. the content.
In an extended language range, the identifier takes the form of a In an extended language range, the identifier takes the form of a
series of subtags which MUST consist of well-formed subtags or the series of subtags which MUST consist of well-formed subtags or the
special subtag "*". For example, the language range "en-*-US" special subtag "*". For example, the language range "en-*-US"
specifies a primary language of 'en', followed by any script subtag, specifies a primary language of 'en', followed by any script subtag,
followed by the region subtag 'US'. followed by the region subtag 'US'.
An extended language range can be represented by the following ABNF: An extended language range can be represented by the following ABNF:
extended-language-range = range ; a range extended-language-range = range ; a range
/ privateuse ; private-use tag / privateuse ; private-use tag
/ grandfathered ; grandfathered registrations / grandfathered ; grandfathered registrations
range = (language range = (language
["-" script] ["-" script]
["-" region] ["-" region]
*("-" variant) *("-" variant)
*("-" extension) *("-" extension)
["-" privateuse]) ["-" privateuse])
skipping to change at page 6, line 29 skipping to change at page 6, line 42
variant = 5*8alphanum ; registered variants variant = 5*8alphanum ; registered variants
/ (DIGIT 3alphanum) ; / (DIGIT 3alphanum) ;
/ "*" ; ... or wildcard / "*" ; ... or wildcard
extension = singleton *("-" (2*8alphanum)) [ "-*" ] extension = singleton *("-" (2*8alphanum)) [ "-*" ]
; extension subtags ; extension subtags
; wildcard can only appear ; wildcard can only appear
; at the end ; at the end
singleton = %x41-57 / %x59-5A / %x61-77 / %x79-7A / DIGIT singleton = "a"-"w" / "y"-"z" / "A"-"W" / "Y"-"Z" / "0"-"9"
; "a"-"w" / "y"-"z" / "A"-"W" / "Y"-"Z" / "0"-"9"
; Single letters: x/X is reserved for private use ; Single letters: x/X is reserved for private use
privateuse = ("x"/"X") 1*("-" (1*8alphanum)) privateuse = ("x"/"X") 1*("-" (1*8alphanum))
grandfathered = 1*3ALPHA 1*2("-" (2*8alphanum)) grandfathered = 1*3ALPHA 1*2("-" (2*8alphanum))
; grandfathered registration ; grandfathered registration
; Note: I is the only singleton ; Note: I is the only singleton
; that starts a grandfathered tag ; that starts a grandfathered tag
alphanum = (ALPHA / DIGIT) ; letters and numbers alphanum = (ALPHA / DIGIT) ; letters and numbers
A field not present in the middle of an extended language range is
treated as if the field contained a "*". Implementations that
normalize extended language ranges SHOULD expand missing fields to be
"*" so that the semantic meaning of the language range is clear to
the user. At the same time, multiple wildcards in a row are
redundant and implementations SHOULD collapse these to a single
wildcard when normalizing the range (for brevity). For example, both
the range "sl-nedis" and the range "sl-*-*-nedis" are equivalent to
and should be normalized as "sl-*-nedis".
A field not present in the middle of an extended language range MAY 2.3. The Language Priority List
be treated as if the field contained a "*". For example, the range
"en-US" MAY be considered to be equivalent to the range "en-*-US".
This also means that multiple wildcards can be collapsed (so that
"en-*-*-US" is equivalent to "en-*-US").
2.4. Choosing a Language Range
Users indicate their language preferences via the choice of a
language range or the set of language ranges in the language priority
list. The type of matching will affect what the best choice is for
given user. In addition, user's should be aware that, when working
with language ranges, most matching schemes make no attempt to
process the semantic meaning of the subtags. The language tag and
language range (or their subtags) are usually compared in a case
insensitive manner using basic string processing. Thus the choice of
subtags in both the language tag and language range may affect the
results produced.
Users SHOULD avoid subtags that add no distinguishing value to a
language range. For example, script subtags SHOULD NOT be used to
form a language range with language subtags which have a matching
Suppress-Script field in their registry record. Thus the language
range "en-Latn" is probably inappropriate in most cases (because the
vast majority of English documents are written in the Latin script
and thus the 'en' language subtag has a Suppress-Script field for
'Latn' in the registry).
When working with tags and ranges, note that extensions and most
private-use subtags are orthogonal to language tag fallback and users
SHOULD avoid using these subtags in language ranges, since they will
often interfere with the selection of available language content.
Since these subtags are always at the end of the sequence of subtags,
they don't normally interfere with the use of prefixes for the
filtering schemes described below in Section 3.
When working with tags and ranges users SHOULD note the following:
1. Private-use and Extension subtags are normally orthogonal to When users specify a language preference they often need to specify a
language tag fallback. Implementations or specifications that prioritized list of language ranges in order to best reflect their
use a lookup (Section 3.3) matching scheme SHOULD ignore language preferences. This is especially true for speakers of
unrecognized private-use and extension subtags when performing minority languages. A speaker of Breton in France, for example, may
language tag fallback. Since these subtags are always at the end specify "be" followed by "fr", meaning that if Breton is available,
of the sequence of subtags, they don't normally interfere with it is preferred, but otherwise French is the best alternative. It
the use of prefixes for matching in the schemes described below. can get more complex: a speaker may wish to fall back from Skolt Sami
to Northern Sami to Finnish.
2. Applications, specifications, or protocols that choose not to A "Language Priority List" is a prioritized or weighted list of
interpret one or more private-use or extension subtags SHOULD NOT language ranges. One well known example of such a list is the
remove or modify these extensions in content that they are "Accept-Language" header defined in RFC 2616 [RFC2616] (see Section
processing. When a language tag instance is to be used in a 14.4) and RFC 3282 [RFC3282]. A simple list of ranges, i.e. one that
specific, known protocol, and is not being passed through to contains no weighting information, is considered to be in descending
other protocols, language tags MAY be filtered to remove subtags order of priority.
and extensions that are not supported by that protocol. Such
filtering SHOULD be avoided, if possible, since it removes
information that might be relevant if services on the other end
of the protocol would make use of that information.
3. Some applications of language tags might want or need to consider The various matching operations described in this document include
extensions and private-use subtags when matching tags. If considerations for using a language priority list. This document
extensions and private-use subtags are included in a matching or does not define any syntax for a language priority list; defining
filtering process that utilizes the one of the schemes described such a syntax is the responsibility of the protocol, application, or
in this document, then the implementation SHOULD canonicalize the implementation that uses it. When given as examples in this
language tags and/or ranges before performing the matching. Note document, language priority lists will be shown as a quoted sequence
that language tag processors that claim to be "well-formed" of ranges separated by semi-colons, like this: "en; fr; zh-Hant"
processors as defined in [RFC3066bis] generally fall into this (which would be read as "English before French before Chinese as
category. written in the Traditional script").
3. Types of Matching 3. Types of Matching
Matching language ranges to language tags can be done in a number of Matching language ranges to language tags can be done in a number of
different ways. This section describes the different types of different ways. This section describes several different matching
matching scheme, as well as the considerations for choosing between schemes, as well as the considerations for choosing between them.
them. Protocols and specifications SHOULD clearly indicate the Protocols and specifications SHOULD clearly indicate the particular
particular mechanism used in selecting or matching language tags. mechanism used in selecting or matching language tags.
There are two basic types of matching scheme: those that produce an There are two basic types of matching scheme: those that produce zero
open-ended set of content (called "filtering") and those that produce or more information items (called "filtering") and those that produce
a single information item for a given request (called "lookup"). a single information item for a given request (called "lookup").
A key difference between these two types of matching scheme is that A key difference between these two types of matching scheme is that
the language range for filtering operations is always the _least_ the language ranges in the language priority list represent the
specific tag one will accept as a match, while for lookup operations _least_ specific content one will accept as a match, while for lookup
the language range is always the _most_ specific tag. operations the language ranges represent the _most_ specific content.
3.1. Choosing a Type of Matching 3.1. Choosing a Type of Matching
Applications, protocols, and specifications are faced with the Applications, protocols, and specifications are faced with the
decision of what type of matching to use. Sometimes, different decision of what type of matching to use. Sometimes, different
styles of matching might be suited for different kinds of processing styles of matching might be suited for different kinds of processing
within a particular application or protocol. within a particular application or protocol.
Language tag matching is a tool, and does not by itself specify a
complete procedure for the use of language tags. Such procedures are
intimately tied to the application protocol in which they occur.
When specifying a protocol operation using matching, the protocol
MUST specify:
o Which type(s) of language tag matching it uses
o Whether the operation returns a single result (lookup) or a
possibly empty set of results (filtering)
o For lookup, what the result is when no matching tag is found. For
instance, a protocol might result in failure of the operation, an
empty value, returning some protocol defined or implementation
defined default, or returning i-default [RFC2277].
Filtering can be used to produce a set of results (such as a Filtering can be used to produce a set of results (such as a
collection of documents). For example, if using a search engine, one collection of documents). For example, if using a search engine, one
might use filtering to limit the results to documents written in might use filtering to limit the results to documents written in
French. It can also be used when deciding whether to perform some French. It can also be used when deciding whether to perform a
processing that is language sensitive on some content. For example, language-sensitive process on some content. For example, a process
a process might cause paragraphs whose language tag matched the might cause paragraphs whose language tag matched the language range
language range "nl" to be displayed in italics within a document. "nl" to be displayed in italics within a document.
This document describes three types of filtering: This document describes four types of matching (three types of
filtering, plus the lookup scheme):
1. Basic Filtering (Section 3.2.1) is used to match content using 1. Basic Filtering (Section 3.2.1) is used to match content using
basic language ranges (Section 2.2). It is compatible with basic language ranges (Section 2.1).
implementations that do not produce extended language ranges.
2. Extended Range Filtering (Section 3.2.2) is used to match content 2. Extended Range Filtering (Section 3.2.2) is used to match content
using extended language ranges (Section 2.3). Newer using extended language ranges (Section 2.2).
implementations SHOULD use this form of filtering in preference
to basic filtering.
3. Scored Filtering (Section 3.2.3) produces an ordered set of 3. Scored Filtering (Section 3.2.3) produces an ordered set of
content using either basic or extended language ranges. It content using extended language ranges. It SHOULD be used when
SHOULD be used when the quality of the match within a specific the quality of the match within a specific language range is
language range is important, as when presenting a list of important, as when presenting a list of documents resulting from
documents resulting from a search. a search.
Lookup (Section 3.3) is used when each request MUST produce exactly 4. Lookup (Section 3.3) is used when each request needs to produce
one piece of content. For example, a Web server might use the _exactly_ one piece of content. For example, if process were to
Accept-Language HTTP header to choose which language to return a insert a human readable error message into a protocol header, it
custom 404 page in: since it can return only one page, it must choose might select the text based on the user's language preference.
a single item and it must return some item, even if no content Since it can return only one item, it must choose a single item
matches the language ranges supplied by the user. and it must return some item, even if no content matches the
language priority list supplied by the user.
Most types of matching in this document are designed so that Most types of matching in this document are designed so that
implementations do not have to examine the values of the subtags implementations are not required to validate or understand any of the
supplied and, except for scored filtering, they do not need access to semantics of the subtags supplied and, except for scored filtering,
the Language Subtag Registry nor do they require the use of valid they do not need access to the IANA Language Subtag Registry (see
subtags in either language tags or language ranges. This has great Section 3 in [RFC3066bis]). This simplifies and speeds the
benefit for speed and simplicity of implementation. performance of implementations.
Implementations might also wish to use semantic information external If an implementation canonicalizes either ranges or tags, then the
to the language tags when performing fallback. For example, the implementation will require the IANA Language Subtag Registry
primary language subtags 'nn' (Nynorsk Norwegian) and 'nb' (Bokmal information for that purpose. Implementations MAY use semantic
Norwegian) might both be usefully matched to the more general subtag information external to the registry when matching tags. For
'no' (Norwegian). Or an implementation might infer that content example, the primary language subtags 'nn' (Nynorsk Norwegian) and
labeled "zh-CN" is more likely to match the range "zh-Hans" than 'nb' (Bokmal Norwegian) might both be usefully matched to the more
equivalent content labeled "zh-TW". general subtag 'no' (Norwegian). Or an implementation might infer
that content labeled "zh-CN" is more likely to match the range "zh-
Hans" than equivalent content labeled "zh-TW".
3.2. Filtering 3.2. Filtering
Filtering is used to select the set of content that matches a given Filtering is used to select the set of content that matches a given
prefix. It is called "filtering" because this set of content may language priority list. It is called "filtering" because this set of
contain no items at all or it may return an arbitrary number of content may contain no items at all or it may return an arbitrarily
matching items--as many as match the language range used to specify large number of matching items--as many as match the language range
the items, thus filtering out the non-matching content. used to specify the items, thus filtering out the non-matching
content.
In filtering, the language range represents the _least_ specific tag In filtering, the language range represents the _least_ specific
which is an acceptable match. That is, all of the language tags in (that is, the fewest number of subtags) language tag which is an
the set of filtered content will have an equal or greater number of acceptable match. That is, all of the language tags in the set of
subtags than the language range. For example, if the language range filtered content will have an equal or greater number of subtags than
is "de-CH", one might see matching content with the tag "de-CH-1996" the language range. For example, if the language priority list
but one will never see a match with the tag "de". consists of the range "de-CH", one might see matching content with
the tag "de-CH-1996" but one will never see a match with the tag
"de".
If the language priority list (see Section 2.1) contains more than If the language priority list (see Section 2.3) contains more than
one range, the content returned is typically ordered in descending one range, the content returned is typically ordered in descending
level of preference. level of preference.
Some examples where filtering might be appropriate include: Some examples where filtering might be appropriate include:
o Applying a style to sections of a document in a particular o Applying a style to sections of a document in a particular set of
language range. languages.
o Displaying the set of documents containing a particular set of o Displaying the set of documents containing a particular set of
keywords written in a specific language. keywords written in a specific set of languages.
o Selecting all email items written in specific range of languages. o Selecting all email items written in a specific set of languages.
Filtering can produce either an ordered or an unordered set of Filtering can produce either an ordered or an unordered set of
results. For example, applying formatting to a document based on the results. For example, applying formatting to a document based on the
language of specific pieces of content does not require the content language of specific pieces of content does not require the content
to be ordered. It is sufficient to know whether a specific piece of to be ordered. It is sufficient to know whether a specific piece of
content matches or does not match. A search application, on the content is selected by the language priority list (or not). A search
other hand, probably would put the results into a priority order. application, on the other hand, probably would want to order the
results.
If an ordered set is desired, as described above, then the If an ordered set is desired, as described above, then the
application or protocol needs to determine the relative "quality" of application or protocol needs to determine the relative "quality" of
the match between different language tags and the language range. the match between different language tags and the language range.
This measurement is called a "distance metric". A distance metric This measurement is called a "distance metric". A distance metric
assigns a numeric value to the comparison of each language tag to a assigns a numeric value to the comparison of a language tag to a
language range and represents the 'distance' between the two. A language range that represents the 'distance' between the two. A
distance of zero means that they are identical, a small distance distance of zero means that they are identical, a small distance
indicates that they are very similar, and a large distance indicated indicates that they are very similar, and a large distance indicates
that they are very different. Using a distance metric, that they are very different. Using a distance metric,
implementations can, for example, allow users to select a threshold implementations can, for example, allow users to select a threshold
distance for a match to be "successful" while filtering or it can use distance for a match to be "successful" while filtering, or they
the numeric value to order the results. might use the numeric values to order the results.
3.2.1. Filtering with Basic Language Ranges 3.2.1. Filtering with Basic Language Ranges
When filtering using a basic language range, the language range When filtering using basic language ranges, each basic language range
matches a language tag if it exactly equals the tag, or if it exactly in the language priority list is considered in turn, according to
equals a prefix of the tag such that the first character following priority. A particular language tag matches a language range if it
the prefix is "-". (That is, the language-range "de-de" matches the exactly equals the tag, or if it exactly equals a prefix of the tag
language tag "de-DE-1996", but not the language tag "de-Deva".) such that the first character following the prefix is "-". (That is,
the language-range "de-de" matches the language tag "de-DE-1996", but
not the language tag "de-Deva".)
The special range "*" matches any tag. A protocol which uses The special range "*" in a language priority list matches any tag. A
language ranges MAY specify additional rules about the semantics of protocol which uses language ranges MAY specify additional rules
"*"; for instance, HTTP/1.1 specifies that the range "*" matches only about the semantics of "*"; for instance, HTTP/1.1 [RFC2616]
languages not matched by any other range within an "Accept-Language" specifies that the range "*" matches only languages not matched by
header. any other range within an "Accept-Language" header.
3.2.2. Filtering with Extended Language Ranges 3.2.2. Filtering with Extended Language Ranges
In the Extended Range Matching scheme, each extended language range When filtering using extended language ranges, each extended language
in the language priority list is considered in turn, according to range in the language priority list is considered in turn, according
priority. The subtags in each extended language range are compared to priority. The subtags in each extended language range are
to the corresponding subtags in the language tag being examined. The compared to the corresponding subtags in the language tag being
subtag from the range is considered to match if it exactly matches examined. The subtag from the range is considered to match if it
the corresponding subtag in the tag or the range's subtag has the exactly matches the corresponding subtag in the tag or the range's
value "*" (which matches all subtags, including the empty subtag). subtag has the value "*" (which matches all subtags, including the
Extended Range Matching is an extension of basic matching empty subtag).
(Section 3.2.1): the language range represents the least specific tag
which is an acceptable match.
private-use subtags MAY be specified in the language range and MUST
NOT be ignored when matching.
Subtags not specified, including those at the end of the language Subtags not specified, including those at the end of the language
range, are assigned the value "*". This makes each range into a range, are assigned the wildcard value "*". This makes each range
prefix much like that used in basic language range matching. For into a prefix much like that used in basic language range matching.
example, the extended language range "de-*-DE" matches all of the For example, the extended language range "de-*-DE" matches all of the
following tags because the unspecified variant field is expanded to following tags because the unspecified variant field is expanded to
"*": "*":
de-DE de-DE
de-Latn-DE de-Latn-DE
de-Latf-DE de-Latf-DE
de-DE-x-goethe de-DE-x-goethe
de-Latn-DE-1996 de-Latn-DE-1996
3.2.3. Scored Filtering 3.2.3. Scored Filtering
Both basic and extended language range filtering produce simple Both basic and extended language range filtering produce simple
boolean matches. Sometimes it may be beneficial to provide an array boolean matches between a language range and a language tag.
of results with different levels of matching, for example, sorting Sometimes it may be useful to provide an array of results with
results based on the overall "quality" of the match. Scored (or different levels of matching, for example, sorting results based on
"distance metric") filtering provides a way to generate these quality the overall "quality" of the match. Scored (or "distance metric")
values. filtering provides a way to generate these quality values.
First both the extended language range and the language tags to be As with the other forms of filtering, the process considers each
matched to it must be canonicalized by mapping grandfathered and language range in the language priority list in order of priority.
obsolete tags into modern equivalents.
The language range and the language tags are then transformed into Each extended language range and language tag MUST first be
quintuples of elements of the form (language, script, country, canonicalized by mapping grandfathered and obsolete tags into modern
variant, extension). Any extended language subtags are considered equivalents. This requires the information in the IANA Language
part of the language element; private-use subtag sequences are Subtag Registry (see Section 3 of [RFC3066bis]).
considered part of the language element if in the initial position in
the tag and part of the variant element if not. Language subtags
'und', 'mul', and the script subtag 'Zyyy' are converted to "*".
Missing components in the language-tag are set to "*"; thus a "*" The language range and each language tag it is to be compared to are
pattern becomes the quintuple ("*", "*", "*", "*", "*"). Missing then transformed into a "quintuple" consisting of five "elements" in
components in the extended language-range are handled similarly to the form (language, script, country, variant, extension).
extended range lookup: missing internal subtags are expanded to "*".
Missing end subtags are expanded as the empty string. Thus a pattern Any extended language subtags are considered part of the language
"en-US" becomes the quintuple ("en","*","US","",""). "element". For example, the language element for the tag "zh-cmn-
Hans" would be "zh-cmn".
Private-use subtag sequences are considered part of the language
"element" if in the initial position in the tag and part of the
variant "element" if not. The different handling of private-use
sequences prevents a range such as "x-twain" from matching all
possible tags, while a range such as "en-US-x-twain" would closely
match nearly all tags for English as used in the United States.
Language subtags 'und', 'mul', and the script subtag 'Zyyy' are
converted to "*": these subtag values represent undetermined,
multiple, or private-use values which are consistent with the use of
the wildcard.
For language tags that have no script subtag but whose language
subtag's record in the IANA Language Subtag Registry contains the
field "Suppress-Script", the script element in the quintuple MUST be
set to the script subtag in the Suppress-Script field. This is
necessary because [RFC3066bis] strongly recommends that users not use
this subtag to form language tags and this document recommends that
users not use them to form ranges. For example, if the script were
not expanded in this manner, a range such as "de-DE" would produce a
more-distant score for content that happened to be labeled
"de-Latn-DE" than users would expect that it should. Note that
languages which have a "Suppress-Script" field in the registry are
predominantly written in a single script.
Any remaining missing components in the language tag are set to "*";
thus an empty language tag becomes the quintuple ("*", "*", "*", "*",
"*"). Missing components in the language range are handled similarly
to extended range lookup: missing internal subtags are expanded to
"*". Missing end subtags are expanded as the empty string. Thus a
pattern "en-US" becomes the quintuple ("en","*","US","","").
Here are some examples of language tags, showing their quintuples as Here are some examples of language tags, showing their quintuples as
both language tags and language ranges: both language tags and language ranges:
en-US en-US
Tag: (en, *, US, *, *) Tag: (en, *, US, *, *)
Range: (en, *, US, "", "") Range: (en, *, US, "", "")
sr-Latn sr-Latn
Tag: (sr, Latn, *, *, *) Tag: (sr, Latn, *, *, *)
skipping to change at page 13, line 38 skipping to change at page 13, line 38
i-default i-default
Tag: (i-default, *, *, *, *) Tag: (i-default, *, *, *, *)
Range: (i-default, "", "", "", "") Range: (i-default, "", "", "", "")
sl-Latn-IT-rozaj sl-Latn-IT-rozaj
Tag: (sl, Latn, IT, rozaj, *) Tag: (sl, Latn, IT, rozaj, *)
Range: (sl, Latn, IT, rozaj, "") Range: (sl, Latn, IT, rozaj, "")
zh-r-wadegile (hypothetical) zh-r-wadegile (hypothetical)
Tag: (z., *, *, *, r-wadegile) Tag: (zh, *, *, *, r-wadegile)
Range: (z., *, *, *, r-wadegile) Range: (zh, *, *, *, r-wadegile)
Figure 3: Examples of Distance Metric Quintuples Figure 3: Examples of Distance Metric Quintuples
Each language-range/language-tag pair being compared is assigned a Each pair of quintuples being compared is assigned a distance value,
distance value, whereby small values indicate better matches and in which small values indicate better matches and large values
large values indicate worse ones. The distance between the pair is indicate worse ones. The distance between the pair is the sum of the
the sum of the distances for each of the corresponding elements of distances for each of the corresponding elements of the quintuple.
the quintuple. If the elements are identical or one is '*', then the If the elements are identical or one is '*', then the distance value
distance value between them is zero. Otherwise, it is given by the between them is zero. Otherwise, it is given by the following table:
following table:
256 language mismatch 256 language mismatch
128 script mismatch 128 script mismatch
32 region mismatch 32 region mismatch
4 variant mismatch 4 variant mismatch
1 extension mismatch 1 extension mismatch
A value of 0 is a perfect match; 421 is no match at all. Different A value of 0 is a perfect match; 421 is no match at all. Different
threshold values might be appropriate for different applications or threshold values might be appropriate for different applications or
protocols. Implementations will usually allow users to choose the protocols. Implementations will usually allow users to choose the
skipping to change at page 14, line 27 skipping to change at page 14, line 27
Examples of various tag's distances from the range "en-US": Examples of various tag's distances from the range "en-US":
"fr-FR" 384 (language & region mismatch) "fr-FR" 384 (language & region mismatch)
"fr" 256 (language mismatch, region match) "fr" 256 (language mismatch, region match)
"en-GB" 32 (region mismatch) "en-GB" 32 (region mismatch)
"en-Latn-US" 0 (all fields match) "en-Latn-US" 0 (all fields match)
"en-Brai" 32 (region mismatch) "en-Brai" 32 (region mismatch)
"en-US-x-foo" 4 (variant mismatch: range is the empty string) "en-US-x-foo" 4 (variant mismatch: range is the empty string)
"en-US-r-wadegile" 1 (extension mismatch: range is the empty string) "en-US-r-wadegile" 1 (extension mismatch: range is the empty string)
Implementations or protocols sometimes might wish to use more Note: A variation of this algorithm might vary the scoring used
sophisticated weights that depend on the values of the corresponding overall or for specific values. For example, sometimes it might make
elements. For example, depending on the domain, an implementation sense to use more sophisticated weighting that depends on the values
might give a small distance to the difference closely related of the corresponding elements. Thus, depending on the domain, an
subtags. Some examples of closely related subtags might be: implementation might assign a smaller distance to the difference
between closely related subtags (or treat certain values as equal).
Some examples of closely related subtags might be:
Language: Language:
no (Norwegian) no (Norwegian)
nb (Bokmal Norwegian) nb (Bokmal Norwegian)
nn (Nynorsk Norwegian) nn (Nynorsk Norwegian)
Script: Script:
Kata (katakana) Kata (katakana)
Hira (hiragana) Hira (hiragana)
Region: Region:
US (United States of America) US (United States of America)
UM (United States Minor Outlying Islands UM (United States Minor Outlying Islands)
Figure 6: Examples of Closely Related Subtags Figure 6: Examples of Closely Related Subtags
3.3. Lookup 3.3. Lookup
Lookup is used to select the single information item that best Lookup is used to select the single information item that best
matches the language priority list for a given request. In lookup, matches the language priority list for a given request. When
each language-range in the language priority list represents the performing lookup, each language range in the language priority list
_most_ specific tag which is an acceptable match; only the closest is considered in turn, according to priority. By contrast with
matching item according the user's priority is returned. For filtering, each language ranges represents the _most_ specific tag
example, if the language range is "de-CH", one might expect to which is an acceptable match. The first information item found with
receive an information item with the tag "de" but never one with the a matching tag, according the user's priority, is considered the
tag "de-CH-1996". Usually if no content matches the request, a closest match and is the item returned. For example, if the language
"default" item is returned. range is "de-CH", one might expect to receive an information item
with the tag "de" but never one with the tag "de-CH-1996". Usually
if no content matches the request, a "default" item is returned.
For example, if an application inserts some dynamic content into a For example, if an application inserts some dynamic content into a
document, returning an empty string if there is no exact match is not document, returning an empty string if there is no exact match is not
an option. Instead, the application "falls back" until it finds a an option. Instead, the application "falls back" until it finds a
suitable piece of content to insert. Other examples of lookup might suitable piece of content to insert. Other examples of lookup might
include: include:
o Selection of a template containing the text for an automated email o Selection of a template containing the text for an automated email
response. response.
o Selection of a graphic containing text for inclusion in a o Selection of a item containing some text for inclusion in a
particular Web page. particular Web page.
o Selection of a string of text for inclusion in an error log. o Selection of a string of text for inclusion in an error log.
In the Lookup scheme, the language-range is progressively truncated In the lookup scheme, the language range is progressively truncated
from the end until a matching piece of content is located. For from the end until a matching piece of content is located. For
example, starting with the range "zh-Hant-CN-x-private", the lookup example, starting with the range "zh-Hant-CN-x-private", the lookup
would progressively search for content as shown below: progressively searches for content as shown below:
Range to match: zh-Hant-CN-x-private Range to match: zh-Hant-CN-x-private
1. zh-Hant-CN-x-private 1. zh-Hant-CN-x-private
2. zh-Hant-CN 2. zh-Hant-CN
3. zh-Hant 3. zh-Hant
4. z. 4. zh
5. (default content or the empty tag) 5. (default content or the empty tag)
Figure 7: Example of a Lookup Fallback Pattern Figure 7: Example of a Lookup Fallback Pattern
This scheme allows some flexibility in finding content. It also This scheme allows some flexibility in finding content. For example,
typically provides better results when data is not available at a it provides better results for cases in which data is not available
specific level of tag granularity or is sparsely populated (than if that exactly matches the user request than if the default language
the default language for the system or content were used). for the system or content were returned immediately. Not every
specific level of tag granularity is usually available or language
The language range "*" matches any language tag. In the lookup content may be sparsely populated, so "falling back" through the
scheme, this language range does not convey enough information to subtag sequence provides more opportunity to find a match between
determine which content is most appropriate. If this language range available content and the user's request.
is the only one in the language priority list, it matches the default
content. If this language range is followed by other language
ranges, it should be skipped.
The default content is implementation defined. It might be content
with no language tag; might have an empty value (the built-in
attribute xml:lang in [XML10] permits the empty value); might be a
particular language designated for that bit of content; or it might
be content that is labeled with the tag "i-default" (see [RFC2277]).
When performing lookup using a language priority list, the When performing lookup using a language priority list, the
progressive search MUST proceed to consider each language range progressive search MUST proceed to consider each language range in
before finding the default content or empty tag. The default content the list before finding the default content or empty tag.
might be content with no language tag (or with an empty value, as
with xml:lang in the XML specification), or it might be a particular
language designated for that bit of content.
One common way to provide for default content is to allow a specific One common way for an application or implementation to provide for
language range to be set as the default for a specific type of default content is to allow a specific language range to be set as
request. This language range is then treated as if it were appended the default for a specific type of request. This language range is
to the end of the language priority list, rather than after each item then treated as if it were appended to the end of the language
in the language priority list. priority list as a whole, rather than after each item in the language
priority list.
For example, if a particular user's language priority list were For example, if a particular user's language priority list were
"fr-FR; zh-Hant" and the program doing the matching had a default "fr-FR; zh-Hant" and the program doing the matching had a default
language range of "ja-JP", the program would search for content as language range of "ja-JP", the program would search for content as
follows: follows:
1. fr-FR 1. fr-FR
2. fr 2. fr
3. zh-Hant // next language 3. zh-Hant // next language
4. z. 4. zh
5. (return default content) 5. (search for the default content)
a. ja-JP a. ja-JP
b. ja b. ja
c. (empty tag or other default content) c. (implementation defined default)
Figure 8: Lookup Using a Language Priority List Figure 8: Lookup Using a Language Priority List
Implementations SHOULD ignore extensions and unrecognized private-use
subtags when performing lookup, since these subtags are usually
orthogonal to the user's request.
The special language range "*" matches any language tag. In the
lookup scheme, this range does not convey enough information by
itself to determine which content is most appropriate, since it
matches everything. If the language range "*" is the only one in the
language priority list, it matches the default content. If the
language range "*" is followed by other language ranges, it should be
skipped.
In some cases, the language priority list might contain one or more In some cases, the language priority list might contain one or more
extended language ranges (as, for example, when the same language extended language ranges (as, for example, when the same language
priority list is used as input for both lookup and filtering priority list is used as input for both lookup and filtering
operations). Wildcard values in an extended language range are operations). Wildcard values in an extended language range normally
supposed to match any value that occurs in that position in a match any value that occurs in that position in a language tag.
language tag. Since only one item can be returned for any given Since only one item can be returned for any given lookup request,
lookup request, the wildcards must be processed in a predictable wildcards in a language range have to be processed in a consistent
manner (or the same request might produce widely varying results). manner or the same request will produce widely varying results.
Thus, for each range in the language priority list, the following Implementations that accept extended language ranges MUST define
rules must be applied to produce a basic language range for use in which content is returned when more than one item matches the
the fallback mechanism: extended language range.
1. If the first subtag in the extended language range is a "*" then For example, an implementation could return the matching content that
entire range is converted to "*". is first in ASCII-order. For example, if the language range were
"*-CH" and the set of content included "de-CH", "fr-CH", and "it-CH",
then the content labeled "de-CH" would be returned.
2. For each subsequent subtag, if the value is a "*" then that Another way an implementation could address extended language ranges
subtag and its preceding hyphen are removed. would be to map them to basic language ranges: if the first subtag is
a "*" then the entire range is treated as "*" (which matches the
default content), otherwise the wildcard subtag is removed. For
example, if the language range were "en-*-US", then the range would
be mapped to "en-US".
For example: 4. Other Considerations
*-US becomes * When working with language ranges and matching schemes, there are
en-*-US becomes en-US some additional points that may influence the choice of either.
en-Latn-* becomes en-Latn
Figure 9: Transformation of Extended Language Ranges 4.1. Choosing Language Ranges
For the language priority list "*-US; fr-*-FR; zh-Hant", the fallback Users indicate their language preferences via the choice of a
pattern would be: language range or the list of language ranges in a language priority
1. * (skipped) list. The type of matching affects what the best choice is for a
2. fr-FR given user.
3. fr
4. zh-Hant
5. z.
6. (default content)
Figure 10: Extended Language Range Fallback Example Most matching schemes make no attempt to process the semantic meaning
of the subtags. The language range (or its subtags) is usually
compared in a case-insensitive manner to each language tag being
matched, using basic string processing.
4. Other Considerations Users SHOULD avoid subtags that add no distinguishing value to a
language range. Generally, the fewer subtags that appear in the
language range, the more content the range will match.
When working with language ranges and matching schemes, there are Most notably, script subtags SHOULD NOT be used to form a language
some additional points that may influence the choice of either. range in combination with language subtags that have a matching
Suppress-Script field in their registry entry. Thus the language
range "en-Latn" is probably inappropriate in most cases (because the
vast majority of English documents are written in the Latin script
and thus the 'en' language subtag has a Suppress-Script field for
'Latn' in the registry).
4.1. Meaning of Language Tags and Ranges When working with tags and ranges, note that extensions and most
private-use subtags are orthogonal to language tag matching, in that
they specify additional attributes of the text not related to the
goals of most matching schemes. Users SHOULD avoid using these
subtags in language ranges, since they interfere with the selection
of available content. When used in language tags (as opposed to
ranges), these subtags normally do not interefer with filtering
(Section 3), since they appear at the end of the tag and will match
all prefixes.
When working with language tags and language ranges note that:
o Private-use and Extension subtags are normally orthogonal to
language tag fallback. Implementations or specifications that use
a lookup (Section 3.3) matching scheme often ignore unrecognized
private-use and extension subtags when performing language tag
fallback. In addition, since these subtags are always at the end
of the sequence of subtags, their use in language tags normally
doesn't interfere with the use of ranges that omit them in the
filtering (Section 3.2) matching schemes described below.
However, they do interfere with filtering when used in language
ranges and SHOULD be avoided in ranges as a result.
o Applications, specifications, or protocols that choose not to
interpret one or more private-use or extension subtags SHOULD NOT
remove or modify these extensions in content that they are
processing. When a language tag instance is to be used in a
specific, known protocol, and is not being passed through to other
protocols, language tags MAY be filtered to remove subtags and
extensions that are not supported by that protocol. Such
filtering SHOULD be avoided, if possible, since it removes
information that might be relevant to services on the other end of
the protocol that would make use of that information.
o Some applications of language tags might want or need to consider
extensions and private-use subtags when matching tags. If
extensions and private-use subtags are included in a matching or
filtering process that utilizes one of the schemes described in
this document, then the implementation SHOULD canonicalize the
language tags and/or ranges before performing the matching. Note
that language tag processors that claim to be "well-formed"
processors as defined in [RFC3066bis] generally fall into this
category.
4.2. Meaning of Language Tags and Ranges
Selecting content using language ranges requires some understanding Selecting content using language ranges requires some understanding
by users of what they are selecting. A language tag or range by users of what they are selecting. A language tag or range
identifies a language as spoken (or written, signed or otherwise identifies a language as spoken (or written, signed or otherwise
signaled) by human beings for communication of information to other signaled) by human beings for communication of information to other
human beings. human beings.
If a language tag B contains language tag A as a prefix, then B is If a language tag B contains language tag A as a prefix, then B is
typically "narrower" or "more specific" than A. For example, "zh- typically "narrower" or "more specific" than A. For example, "zh-
Hant-TW" is more specific than "zh-Hant". Hant-TW" is more specific than "zh-Hant".
skipping to change at page 18, line 36 skipping to change at page 20, line 7
For example, the tag "az" shares a prefix with both "az-Latn" For example, the tag "az" shares a prefix with both "az-Latn"
(Azerbaijani written using the Latin script) and "az-Arab" (Azerbaijani written using the Latin script) and "az-Arab"
(Azerbaijani written using the Arabic script). A person fluent in (Azerbaijani written using the Arabic script). A person fluent in
one script might not be able to read the other, even though the text one script might not be able to read the other, even though the text
might be otherwise identical. Content tagged as "az" most probably might be otherwise identical. Content tagged as "az" most probably
is written in just one script and thus might not be intelligible to a is written in just one script and thus might not be intelligible to a
reader familiar with the other script. reader familiar with the other script.
Variant subtags in particular seem to represent specific divisions in Variant subtags in particular seem to represent specific divisions in
mutual understanding, since they often encode dialects or other mutual understanding, since they often encode dialects or other
idiosyncratic variations within a language. idiosyncratic variations within a language. They also seem to
represent relatively low divisions with a high chance of at least
limited understanding, although this depends on the specific variant
in question.
The relationship between the language tag and the information it The relationship between the language tag and the information it
relates to is defined by the standard describing the context in which relates to is defined by the standard describing the context in which
it appears. Accordingly, this section can only give possible it appears. Accordingly, this section can only give possible
examples of its usage: examples of its usage:
o For a single information object, the associated language tags o For a single information object, the associated language tags
might be interpreted as the set of languages that are necessary might be interpreted as the set of languages that are necessary
for a complete comprehension of the complete object. Example: for a complete comprehension of the complete object. Example:
Plain text documents. Plain text documents.
skipping to change at page 19, line 26 skipping to change at page 20, line 47
structure (including the whole document itself). For example, one structure (including the whole document itself). For example, one
could write <span lang="FR">C'est la vie.</span> inside a could write <span lang="FR">C'est la vie.</span> inside a
Norwegian document; the Norwegian-speaking user could then access Norwegian document; the Norwegian-speaking user could then access
a French-Norwegian dictionary to find out what the marked section a French-Norwegian dictionary to find out what the marked section
meant. If the user were listening to that document through a meant. If the user were listening to that document through a
speech synthesis interface, this formation could be used to signal speech synthesis interface, this formation could be used to signal
the synthesizer to appropriately apply French text-to-speech the synthesizer to appropriately apply French text-to-speech
pronunciation rules to that span of text, instead of misapplying pronunciation rules to that span of text, instead of misapplying
the Norwegian rules. the Norwegian rules.
4.2. Considerations for Private Use Subtags 4.3. Considerations for Private Use Subtags
Private-use subtags require private agreement between the parties Private-use subtags require private agreement between the parties
that intend to use or exchange language tags that use them and great that intend to use or exchange language tags that use them and great
caution SHOULD be used in employing them in content or protocols caution SHOULD be used in employing them in content or protocols
intended for general use. Private-use subtags are simply useless for intended for general use. Private-use subtags are simply useless for
information exchange without prior arrangement. information exchange without prior arrangement.
The value and semantic meaning of private-use tags and of the subtags The value and semantic meaning of private-use tags and of the subtags
used within such a language tag are not defined. Matching private- used within such a language tag are not defined. Matching private-
use tags using language ranges or extended language ranges can result use tags using language ranges or extended language ranges can result
in unpredictable content being returned. in unpredictable content being returned.
4.3. Length Considerations in Matching 4.4. Length Considerations in Matching
RFC 3066 [RFC3066] did not provide an upper limit on the size of RFC 3066 [RFC3066] did not provide an upper limit on the size of
language tags or ranges. RFC 3066 did define the semantics of language tags or ranges. RFC 3066 did define the semantics of
particular subtags in such a way that most language tags or ranges particular subtags in such a way that most language tags or ranges
consisted of language and region subtags with a combined total length consisted of language and region subtags with a combined total length
of up to six characters. Larger tags and ranges (in terms of both of up to six characters. Larger tags and ranges (in terms of both
subtags and characters) did exist, however. subtags and characters) did exist, however.
[RFC3066bis] also does not impose a fixed upper limit on the number [RFC3066bis] also does not impose a fixed upper limit on the number
of subtags in a language tag or range (and thus an upper bound on the of subtags in a language tag or range (and thus an upper bound on the
skipping to change at page 20, line 20 skipping to change at page 21, line 40
be clearly documented, and such documentation SHOULD include the be clearly documented, and such documentation SHOULD include the
disposition of any longer tags or ranges (for example, whether an disposition of any longer tags or ranges (for example, whether an
error value is generated or the language tag or range is truncated). error value is generated or the language tag or range is truncated).
If truncation is permitted it MUST NOT permit a subtag to be divided, If truncation is permitted it MUST NOT permit a subtag to be divided,
since this changes the semantics of the subtag being matched and can since this changes the semantics of the subtag being matched and can
result in false positives or negatives. result in false positives or negatives.
Applications or protocols that restrict storage SHOULD consider the Applications or protocols that restrict storage SHOULD consider the
impact of tag or range truncation on the resulting matches. For impact of tag or range truncation on the resulting matches. For
example, removing the "*" from the end of an extended language range example, removing the "*" from the end of an extended language range
(see Section 2.3) can greatly modify the set of returned matches. A (see Section 2.2) can greatly modify the set of returned matches. A
protocol that allows tags or ranges to be truncated at an arbitrary protocol that allows tags or ranges to be truncated at an arbitrary
limit, without giving any indication of what that limit is, has the limit, without giving any indication of what that limit is, has the
potential for causing harm by changing the meaning of values in potential for causing harm by changing the meaning of values in
substantial ways. substantial ways.
In practice, most tags do not require additional subtags or In practice, most tags do not require additional subtags or
substantially more characters. Additional subtags sometimes add substantially more characters. Additional subtags sometimes add
useful distinguishing information, but extraneous subtags interfere useful distinguishing information, but extraneous subtags interfere
with the meaning, understanding, and especially matching of language with the meaning, understanding, and especially matching of language
tags. Since language tags or ranges MAY be truncated by an tags. Since language tags or ranges MAY be truncated by an
skipping to change at page 21, line 15 skipping to change at page 22, line 36
the given buffer. If the resulting tag ends with a single-character the given buffer. If the resulting tag ends with a single-character
subtag, that subtag and its preceding "-" MUST also be removed. For subtag, that subtag and its preceding "-" MUST also be removed. For
example: example:
Tag to truncate: zh-Latn-CN-variant1-a-extend1-x-wadegile-private1 Tag to truncate: zh-Latn-CN-variant1-a-extend1-x-wadegile-private1
1. zh-Latn-CN-variant1-a-extend1-x-wadegile 1. zh-Latn-CN-variant1-a-extend1-x-wadegile
2. zh-Latn-CN-variant1-a-extend1 2. zh-Latn-CN-variant1-a-extend1
3. zh-Latn-CN-variant1 3. zh-Latn-CN-variant1
4. zh-Latn-CN 4. zh-Latn-CN
5. zh-Latn 5. zh-Latn
6. z. 6. zh
Figure 11: Example of Tag Truncation Figure 9: Example of Tag Truncation
5. IANA Considerations 5. IANA Considerations
This document presents no new or existing considerations for IANA. This document presents no new or existing considerations for IANA.
6. Changes 6. Changes
This is the first version of this document. This is the first version of this document.
The following changes were put into this document since draft-06: The following changes were put into this document since draft-07:
Changed the document title from the unwieldy "Matching Tags for
the Identification of Languages" to "Matching Language Tags" (Ed.)
Fixed problems with the distance metric filtering scheme
(Section 3.2.3) examples (in which tags were expanded
incorrectly). (D.Ewell)
Moved the sentence "Protocols and specifications SHOULD clearly
indicate the particular mechanism used in selecting or matching
language tags." from the introduction (where there should not be
any normative language) to the start of Section 3. (A.Phillips)
Created section Section 2.4 and moved text there (A.Phillips)
Modified the examples of closely related subtags in Section 3.2.3
to show what the examples mean (M.Duerst)
Various spelling and grammatical fixes (D.Ewell) Added a mention of "*" to the Character Set Considerations section
(D.Ewell)
7. Security Considerations 7. Security Considerations
Language ranges used in content negotiation might be used to infer Language ranges used in content negotiation might be used to infer
the nationality of the sender, and thus identify potential targets the nationality of the sender, and thus identify potential targets
for surveillance. In addition, unique or highly unusual language for surveillance. In addition, unique or highly unusual language
ranges or combinations of language ranges might be used to track a ranges or combinations of language ranges might be used to track a
specific individual's activities. specific individual's activities.
This is a special case of the general problem that anything you send This is a special case of the general problem that anything you send
is visible to the receiving party. It is useful to be aware that is visible to the receiving party. It is useful to be aware that
such concerns can exist in some cases. such concerns can exist in some cases.
The evaluation of the exact magnitude of the threat, and any possible The evaluation of the exact magnitude of the threat, and any possible
countermeasures, is left to each application or protocol. countermeasures, is left to each application or protocol.
8. Character Set Considerations 8. Character Set Considerations
The syntax of language tags and language ranges permit only the Language tags permit only the characters A-Z, a-z, 0-9, and HYPHEN-
characters A-Z, a-z, 0-9, and HYPHEN-MINUS (%x2D). These characters MINUS (%x2D). Language ranges also use the character ASTERISK
are present in most character sets, so presentation of language tags (%x2A). These characters are present in most character sets, so
should not present any character set issues. presentation or exchange of language tags or ranges should not be
constrained by character set issues.
9. References 9. References
9.1. Normative References 9.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997. Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC2616] Fielding, R., Gettys, J., Mogul, J., Frystyk, H., [RFC2277] Alvestrand, H., "IETF Policy on Character Sets and
Masinter, L., Leach, P., and T. Berners-Lee, "Hypertext Languages", BCP 18, RFC 2277, January 1998.
Transfer Protocol -- HTTP/1.1", RFC 2616, June 1999.
[RFC3066bis] [RFC3066bis]
Phillips, A., Ed. and M. Davis, Ed., "Tags for the Phillips, A., Ed. and M. Davis, Ed., "Tags for the
Identification of Languages", October 2005, <http:// Identification of Languages", October 2005, <http://
www.ietf.org/internet-drafts/ www.ietf.org/internet-drafts/
draft-ietf-ltru-registry-14.txt>. draft-ietf-ltru-registry-14.txt>.
[RFC4234] Crocker, D. and P. Overell, "Augmented BNF for Syntax [RFC4234] Crocker, D. and P. Overell, "Augmented BNF for Syntax
Specifications: ABNF", RFC 4234, October 2005. Specifications: ABNF", RFC 4234, October 2005.
9.2. Informative References 9.2. Informative References
[RFC1766] Alvestrand, H., "Tags for the Identification of [RFC1766] Alvestrand, H., "Tags for the Identification of
Languages", RFC 1766, March 1995. Languages", RFC 1766, March 1995.
[RFC2616] Fielding, R., Gettys, J., Mogul, J., Frystyk, H.,
Masinter, L., Leach, P., and T. Berners-Lee, "Hypertext
Transfer Protocol -- HTTP/1.1", RFC 2616, June 1999.
[RFC3066] Alvestrand, H., "Tags for the Identification of [RFC3066] Alvestrand, H., "Tags for the Identification of
Languages", BCP 47, RFC 3066, January 2001. Languages", BCP 47, RFC 3066, January 2001.
[RFC3282] Alvestrand, H., "Content Language Headers", RFC 3282, [RFC3282] Alvestrand, H., "Content Language Headers", RFC 3282,
May 2002. May 2002.
[XML10] Bray (et al), T., "Extensible Markup Language (XML) 1.0",
02 2004.
Appendix A. Acknowledgements Appendix A. Acknowledgements
Any list of contributors is bound to be incomplete; please regard the Any list of contributors is bound to be incomplete; please regard the
following as only a selection from the group of people who have following as only a selection from the group of people who have
contributed to make this document what it is today. contributed to make this document what it is today.
The contributors to [RFC3066bis], [RFC3066] and [RFC1766], each of The contributors to [RFC3066bis], [RFC3066] and [RFC1766], each of
which is a precursor to this document, made enormous contributions which is a precursor to this document, made enormous contributions
directly or indirectly to this document and are generally responsible directly or indirectly to this document and are generally responsible
for the success of language tags. for the success of language tags.
 End of changes. 96 change blocks. 
377 lines changed or deleted 450 lines changed or added

This html diff was produced by rfcdiff 1.27, available from http://www.levkowetz.com/ietf/tools/rfcdiff/