Network Working Group                                   A. Phillips, Ed.
Internet-Draft                                                Yahoo! Inc
Obsoletes: 3066 (if approved)                              M. Davis, Ed.
Expires: August 27, September 5, 2006                                        Google
                                                       February 23,
                                                           March 4, 2006

                       Matching of Language Tags
                      draft-ietf-ltru-matching-10
                      draft-ietf-ltru-matching-11

Status of this Memo

   By submitting this Internet-Draft, each author represents that any
   applicable patent or other IPR claims of which he or she is aware
   have been or will be disclosed, and any of which he or she becomes
   aware will be disclosed, in accordance with Section 6 of BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as Internet-
   Drafts.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/ietf/1id-abstracts.txt.

   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html.

   This Internet-Draft will expire on August 27, September 5, 2006.

Copyright Notice

   Copyright (C) The Internet Society (2006).

Abstract

   This document describes different mechanisms for comparing, matching, comparing and evaluating
   matching language tags.  Possible algorithms for language negotiation
   or content selection, filtering, and lookup are described.  This
   document, in combination with RFC 3066bis (Ed.: replace "3066bis"
   with the RFC number assigned to draft-ietf-ltru-registry-14),
   replaces RFC 3066, which replaced RFC 1766.

Table of Contents

   1.  Introduction . . . . . . . . . . . . . . . . . . . . . . . . .  3
   2.  The Language Range . . . . . . . . . . . . . . . . . . . . . .  4
     2.1.  Basic Language Range . . . . . . . . . . . . . . . . . . .  4
     2.2.  Extended Language Range  . . . . . . . . . . . . . . . . .  5
     2.3.  The Language Priority List . . . . . . . . . . . . . . . .  5
   3.  Types of Matching  . . . . . . . . . . . . . . . . . . . . . .  7
     3.1.  Choosing a Type of Matching  . . . . . . . . . . . . . . .  7
     3.2.  Filtering  . . . . . . . . . . . . . . . . . . . . . . . .  8  9
       3.2.1.  Basic Filtering  . . . . . . . . . . . . . . . . . . .  9
       3.2.2.  Extended Filtering . . . . . . . . . . . . . . . . . . 10
     3.3.  Lookup . . . . . . . . . . . . . . . . . . . . . . . . . . 10 11
   4.  Other Considerations . . . . . . . . . . . . . . . . . . . . . 14 15
     4.1.  Choosing Language Ranges . . . . . . . . . . . . . . . . . 14 15
     4.2.  Meaning of Language Tags and Ranges  . . . . . . . . . . . 15 16
     4.3.  Considerations for Private Use Subtags . . . . . . . . . . 15 16
     4.4.  Length Considerations in Matching  . . . . for Language Ranges  . . . . . . . . 15 17
   5.  IANA Considerations  . . . . . . . . . . . . . . . . . . . . . 17 18
   6.  Changes  . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 19
   7.  Security Considerations  . . . . . . . . . . . . . . . . . . . 19 20
   8.  Character Set Considerations . . . . . . . . . . . . . . . . . 20 21
   9.  References . . . . . . . . . . . . . . . . . . . . . . . . . . 21 22
     9.1.  Normative References . . . . . . . . . . . . . . . . . . . 21 22
     9.2.  Informative References . . . . . . . . . . . . . . . . . . 21 22
   Appendix A.  Acknowledgements  . . . . . . . . . . . . . . . . . . 22 23
   Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 23 24
   Intellectual Property and Copyright Statements . . . . . . . . . . 24 25

1.  Introduction

   Human beings on our planet have, past and present, used a number of
   languages.  There are many reasons why one would want to identify the
   language used when presenting or requesting information or in some
   specific set of information items or "content".

   One use for language identifiers, such as those defined in
   [RFC3066bis], is to select content by matching the associated
   language tags to a user's language preferences.

   This document defines a syntax (called a language range (Section 2))
   for specifying items in the user's list of language preferences
   (called a language priority list (Section 2.3)), as well as several
   schemes for selecting or filtering sets of content by comparing the
   content's language tags to the user's preferences.  Applications,
   protocols, or specifications will have varying needs and requirements
   that affect the choice of a suitable matching scheme.  Depending on
   the choice of scheme, there are various options left to the
   implementation.  Protocols that implement a matching scheme either
   need to specify each particular choice or indicate the options that
   are left to the implementation to decide.

   This document is divided into three main sections.  One describes how
   to indicate a user's preferences using language ranges.  Then a
   section describes various schemes for matching these ranges to a set
   of language tags.  There is also a section that deals with various
   practical considerations that apply to implementing and using these
   schemes.

   This document, in combination with [RFC3066bis] (Ed.: replace
   "3066bis" globally in this document with the RFC number assigned to
   draft-ietf-ltru-registry-14), replaces [RFC3066], which replaced
   [RFC1766].

   The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in [RFC2119].

2.  The Language Range

   Language Tags [RFC3066bis] are used to identify the language of some
   information item or content. "content".  Applications or protocols that use
   language tags are often faced with the problem of identifying sets of
   content that share certain language attributes.  For example,
   HTTP/1.1 [RFC2616] describes one such mechanism in its discussion of
   the Accept-Language header (Section 14.4), which is used when
   selecting content from servers based on the language of that content.

   When selecting content according to its language, it is useful to
   have a mechanism for identifying sets of language tags that share
   specific attributes.  This allows users to select or filter content
   based on specific requirements.  Such an identifier is called a
   "Language Range".
   "language range".

   There are different types of language range, whose specific
   attributes vary according to match their application.  Language ranges are
   similar in content to language tags: they consist of a sequence of subtags
   separated by hyphens.  In a language range, each subtag MUST either
   be a sequence of ASCII alphanumeric characters or the single
   character '*' (%2A, ASTERISK).  The character '*' is a "wildcard"
   that matches any sequence of subtags.  Restrictions on the  The meaning and use uses of
   wildcards vary according to the type of language range.

   Language tags and thus language ranges are to be treated as case-
   insensitive: there exist conventions for the capitalization of some
   of the subtags, but these MUST NOT be taken to carry meaning.
   Matching of language tags to language ranges MUST be done in a case-
   insensitive manner.

2.1.  Basic Language Range

   A "basic language range" identifies the set of describes a user's language tags that all
   begin with the same preference as a
   specific, uninterrupted, sequence of subtags.  Each range consists of
   a sequence of alphanumeric subtags separated by hyphens.  The basic
   language range is defined by the following ABNF [RFC4234]:

   language-range   = (1*8ALPHA *("-" 1*8alphanum)) / "*"
   alphanum         = ALPHA / DIGIT

   Basic language ranges (originally described by HTTP/1.1 [RFC2616] and
   later [RFC3066]) have the same syntax as an [RFC3066] language tag or
   are the single character "*".  They differ from the language tags
   defined in [RFC3066bis] only in that there is no requirement that
   they be "well-formed" or be validated against the IANA Language
   Subtag Registry (although such ill-formed ranges will probably not
   match anything).  (Note that the ABNF [RFC4234] in [RFC2616] is
   incorrect, since it disallows the use of digits anywhere in the
   'language-range': this is mentioned in the errata)

   Use of a basic language range seems to imply that there is a semantic
   relationship between language tags that share the same prefix.  While
   this is often the case, it is not always true and users should note
   that the set of language tags that match a specific language range
   may not represent mutually intelligible languages.

2.2.  Extended Language Range

   Basic language ranges allow users to specify a set of language tags
   that share the same initial subtags.

   Occasionally users will wish to select a set of language tags based
   on the presence of specific subtags.  An "extended language range"
   describes a user's language preference as an ordered sequence of
   subtags.  For example, a user might wish to select all language tags
   that contains contain the region subtag 'CH'. 'CH' (Switzerland).  Extended language
   ranges are useful in specifying a particular sequence of subtags that
   appear in the set of matching tags without having to specify all of
   the intervening subtags.

   An extended language range can be represented by the following ABNF:

   extended-language-range = (1*8ALPHA / "*")
                             *("-" (1*8alphanum / "*"))

   Figure 2: Extended Language Range

   The wildcard subtag '*' MAY can occur in any position in the extended
   language range, where it matches any sequence of subtags that might
   occur in that position in a language tag.  However  However, wildcards outside
   the first position in an extended language range are ignored by most
   matching schemes.  Use of multiple one or more wildcards SHOULD NOT be taken
   to imply that a certain number of subtags will appear in the matching
   set of language tags.

   Implementations that specify basic ranges MAY map extended language
   ranges to basic language ranges: if the first subtag is a "*" then
   the entire range is treated as "*" (which matches the default
   content), "*", otherwise each wildcard subtag is
   removed.  For example, if the language range were "en-*-US", then the
   range would be mapped to "en-US".

2.3.  The Language Priority List

   When users

   A user's language preferences will often need to specify a more than
   one language preference they range and thus users often need to specify a prioritized
   list of language ranges in order to best reflect their language
   preferences.  This is especially true for speakers of minority
   languages.  A speaker of Breton in France, for example, may specify
   "be" followed by "fr", meaning that if Breton is available, it is
   preferred, but otherwise French is the best alternative.  It can get
   more complex: a speaker user may wish to fall back from Skolt Sami to
   Northern Sami to Finnish.

   A "Language Priority List" "language priority list" is a prioritized or weighted list of
   language ranges.  One well known example of such a list is the
   "Accept-Language" header defined in RFC 2616 [RFC2616] (see Section
   14.4) and RFC 3282 [RFC3282].  A simple list of ranges, i.e. one that
   contains no weighting information, is considered to be in descending
   order of priority.

   The various matching operations described in this document include
   considerations for using a language priority list.  This document
   does not define any the syntax for a language priority list; defining
   such a syntax is the responsibility of the protocol, application, or
   implementation
   specification that uses it.  When given as examples in this document,
   language priority lists will be shown as a quoted sequence of ranges
   separated by semicolons, commas, like this: "en; fr; "en, fr, zh-Hant" (which would be
   read as "English before French before Chinese as written in the
   Traditional script").

   Where a

   A simple list of ranges is considered to be in descending order of
   priority.  Other language priority list provides lists provide "quality weights"
   for the language ranges, such as ranges in order to specify the relative priority of
   the user's language preferences.  An example of this would be the use
   of Q weights "q" values in the syntax of the "Accept-Language" header (defined
   in [RFC2616], Section 14.4, and
   [RFC3282]), language ranges without a weight are given values equal
   to the value [RFC3282]).

3.  Types of the previous Matching

   Matching language range (processing from first ranges to
   last).  If the first language range has no weight, it is given tags can be done in a
   value of 1.0.  Then language ranges with zero weights are removed.
   For example, "fr, en;q=0.5, de, it" becomes "fr;q=1.0, en;q=0.5,
   de;q=0.5, it;q=0.5".  The language priority list is then sorted from
   highest priority to lowest, with language ranges that share the same
   weights remain in the same order as in the original language priority
   list.

3.  Types of Matching

   Matching language ranges to language tags can be done in a number number of
   different ways.  This section describes several different matching
   schemes, as well as the considerations for choosing between them.
   Protocols and specifications SHOULD clearly indicate the particular
   mechanism used in selecting or matching language tags.

   There are several types of matching scheme.  This document presents
   two types: those that produce zero or more information items (called
   "filtering") and those that produce a single information item for a
   given request (called "lookup").

   Implementations or protocols MAY use different matching schemes than from
   the ones described in this document, as long as those mechanisms are
   clearly specified.

3.1.  Choosing a Type of Matching

   Applications, protocols, and specifications are faced with the
   decision of what type of matching to use.  Sometimes, different
   styles of matching might be are suited for to different kinds of processing within
   a particular application or protocol.

   Language tag matching is a tool, and does not by itself specify a
   complete procedure for the use of language tags.  Such procedures are
   intimately tied to the application protocol in which they occur.
   When specifying a protocol operation using matching, the protocol
   MUST specify:

   o  Which type(s) of language tag matching it uses

   o  Whether the operation returns a single result (lookup) or a
      possibly empty set of results (filtering)

   o  For lookup, what the result is when no matching tag is found.  For
      instance, a protocol might define the result as failure of the
      operation, an empty value, returning some protocol defined or
      implementation defined default, or returning i-default [RFC2277].

   This document describes three types of matching:

   1.  Basic Filtering (Section 3.2.1) matches a language priority list
       consisting of basic language ranges (Section 2.1) to sets of
       language tags.

   2.  Extended Filtering (Section 3.2.2) matches a language priority
       list consisting of extended language ranges (Section 2.2) to sets
       of language tags.

   3.  Lookup (Section 3.3) matches a language priority list consisting
       of basic language ranges to sets of language tags to find the
       _exactly_ one
       _exact_ language tag that best matches the range.

   Both types of filtering

   Filtering can be used to produce a set of results (such as a
   collection of documents) by comparing the user's preferences to
   language tags associated with the set of content.  For example, when
   performing a search, one might use filtering to limit the results to
   documents
   items tagged as being written in French.  They might the French language.  Filtering can also be
   used when deciding whether to perform a language-sensitive process on
   some content.  For example, a process might cause paragraphs whose
   language tag matched the language range "nl" to be displayed in
   italics within a document.

   Lookup produces the single result that best matches a given set of
   user the user's
   preferences, so it is useful in cases in which only a single item can
   be returned.  For example, if a process were to insert a human
   readable error message into a protocol header, it might select the
   text based on the user's language priority list.  Since the process
   can return only one item, it must choose a single item and it must
   return some item, even if no none of the content's language tag matches tags match
   the language priority list supplied by the user.

   The types of matching in this document are designed so that
   implementations are not required to validate or understand any of the
   semantics of the language tags or ranges or of the subtags in them.
   None of them require access to the IANA Language Subtag Registry (see
   Section 3 in [RFC3066bis]).  This simplifies and speeds the
   performance implementation of implementations. these
   schemes.  An implementation MAY choose to check if either the
   language ranges or language tags being matched are "well-formed" or
   "valid" (see [RFC3066bis], Section 2.2.9) and MAY choose not to
   process invalid ranges.

   Regardless of the matching scheme chosen, protocols and
   implementations MAY canonicalize language tags and ranges by mapping
   grandfathered and obsolete tags or subtags into modern equivalents.
   If an implementation canonicalizes either ranges or tags, then the
   implementation will require the IANA Language Subtag Registry
   information for that purpose.  Implementations MAY also use semantic
   information external to the registry when matching tags.  For
   example, the primary language subtags 'nn' (Nynorsk Norwegian) and
   'nb' (Bokmal Norwegian) might both be usefully matched to the more
   general subtag 'no' (Norwegian).  Or an implementation might infer
   that content labeled "zh-CN" "zh-Hans" (Chinese as written in the Simplified
   script) is more likely to match the range "zh-
   Hans" "zh-CN" (Chinese as used in
   China, where the Simplified script is predominant) than equivalent
   content labeled "zh-TW". "zh-TW" (Chinese as used in Taiwan, where the
   Traditional script is predominant).

3.2.  Filtering

   Filtering is used to select the set of language tags that matches a
   given language priority list and return the associated content.  It
   is called "filtering" because this set might contain no items at all
   or it might return an arbitrarily large number of matching items: as
   many items as match the language priority list, thus "filtering out"
   the non-matching items.

   In filtering, the each language range represents the _least_ specific
   language tag (that is, the language tag with fewest number of
   subtags) language tag which is an acceptable match.  All of the language tags in
   the matching set of tags will have an equal or greater number of
   subtags than the language range.  Every non-wildcard subtag in the
   language range will appear in every one of the matching language
   tags.  For example, if the language priority list consists of the
   range "de-CH", one might see tags such as "de-CH-1996" but one will
   never see a tag such as "de" (because the 'CH' subtag is missing).

   If the language priority list (see Section 2.3) contains more than
   one range, the content returned is typically ordered in descending
   level of preference. preference, but it MAY be unordered, according to the needs
   of the application or protocol.

   Some examples of applications where filtering might be appropriate
   include:

   o  Applying a style to sections of a document in a particular set of
      languages.

   o  Displaying the set of documents containing a particular set of
      keywords written in a specific set of languages.

   o  Selecting all email items written in a specific set of languages.

   The content returned MAY either be ordered or unordered according to
   the priority

   o  Selecting audio files spoken in the language priority list (and other criteria),
   according to the needs of the application or protocol. a particular language.

3.2.1.  Basic Filtering

   When filtering using basic language ranges, each basic language range
   in the language priority list is considered in turn, according to
   priority.  A particular language tag matches a language range if if, in
   a case-insensitive comparison, it exactly equals the tag, or if it
   exactly equals a prefix of the tag such that the first character
   following the prefix is "-".  For example, the language-range "de-de"
   matches the language tag "de-DE-
   1996", "de-DE-1996", but not the language tags "de-Deva" "de-
   Deva" or "de-Latn-DE".

   The special range "*" in a language priority list matches any tag.  A
   protocol which uses language ranges MAY specify additional rules
   about the semantics of "*"; for instance, HTTP/1.1 [RFC2616]
   specifies that the range "*" matches only languages not matched by
   any other range within an "Accept-Language" header.

   Basic filtering is identical to the type of matching described in
   [RFC3066], Section 2.5 (Language-range).

3.2.2.  Extended Filtering

   When filtering using extended language ranges, each extended language
   range in the language priority list is considered in turn, according
   to priority.  A particular language range is compared to each
   language tag using the following process:

   Compare the first subtag in the extended language tag to the first
   subtag in the language tag in a case insensitive manner.  If the
   first subtag in the range is "*", it matches any value.  Otherwise
   the two values must match or the overall match fails.

   Take each non-wildcard subtag in the language range and compare it in
   a case-insensitive manner to the next subtag in the language tag in turn until a matching subtag
   is found or the langauge tag is exhausted. tag.  If
   the end of the
   language tag is found first, range's subtag exactly matches the match fails.  If a match is found,
   this step is repeated with tag's subtag, proceed to the
   next non-wildcard subtag in the language range (and beginning with
   the next subtag in the language tag) until the list of subtags in the
   language range is exhausted or the match fails.

   Subtags not specified, including those at  If the end of tag's subtag
   is a "singleton" (a single letter or digit, which, in this case,
   includes the language
   range, are thus treated private-use subtag 'x') and the range's subtag does not
   match or if the language tag's list of subtags is exhausted, the
   match fails.  If the language range's list of subtags is exhausted,
   the match succeeds.

   Subtags not specified, including those at the end of the language
   range, are thus treated as if assigned the wildcard value "*".
   Extended filtering works, therefore, much  Much
   like basic filtering. filtering, extended filtering selects content with
   arbitrarily long tags that share the same initial subtags as the
   language range.  In addition extended filtering selects content with
   any intermediate subtags unspecified in the language range.  For
   example, the extended language range "de-*-DE" matches all of the
   following tags:

      de-DE
      de-Latn-DE

      de-Latf-DE

      de-de

      de-DE-x-goethe

      de-Latn-DE-1996

   The same range does not match any of the following tags for the
   reasons shown:

      de (missing 'DE')

      de-x-DE (singleton 'x' occurs before 'DE')

      de-Deva ('Deva' not equal to 'DE')

   Note: The structure of language tags defined by [RFC3066bis] defines
   each type of subtag (language, script, region, and so forth)
   according to position, size, and content.  This means that subtags in
   a language range can only match specific types of subtags in a
   language tag.  For example, a subtag such as 'Latn' is always a
   script subtag (unless it follows a singleton) while a subtag such as
   'nedis' can only match the equivalent variant subtag.

3.3.  Lookup

   Lookup is used to select the single language tag that best matches
   the language priority list for a given request and return the
   associated content.  When performing lookup, each language range in
   the language priority list is considered in turn, according to
   priority.  By contrast with filtering, each language range represents
   the _most_ specific tag which is an acceptable match.  The first
   content found with a matching tag, according to the user's priority,
   is considered the closest match and is the content returned.  For
   example, if the language range is "de-ch", a lookup operation might
   produce content with the tags "de" or "de-CH" but never one with the
   tag "de-CH-1996".  Usually if no content matches the request, the
   "default" content is returned.

   For example, if an application inserts some dynamic content into a
   document, returning an empty string if there is no exact match is not
   an option.  Instead, the application "falls back" until it finds a
   matching language tag associated with a suitable piece of content to
   insert.  Examples of lookup might include:

   o  Selection of a template containing the text for an automated email
      response.

   o  Selection of a item containing some text for inclusion in a
      particular Web page.

   o  Selection of a string of text for inclusion in an error log.

   o  Selection of an audio file to play as a prompt in a phone system.

   In the lookup scheme, the language range is progressively truncated
   from the end until a matching piece of content is located.  Single
   letter or digit subtags (including both the letter 'x' which
   introduces private-use sequences, and the subtags that introduce
   extensions) are removed at the same time as their closest trailing
   subtag.  For example, starting with the range "zh-Hant-CN-x-private", "zh-Hant-CN-x-private1-
   private2", the lookup progressively searches for content as shown
   below:

   Range to match: zh-Hant-CN-x-private zh-Hant-CN-x-private1-private2
   1. zh-Hant-CN-x-private zh-Hant-CN-x-private1-private2
   2. zh-Hant-CN zh-Hant-CN-x-private1
   3. zh-Hant zh-Hant-CN
   4. zh zh-Hant
   5. zh
   6. (default content)

   Figure 3: Example of a Lookup Fallback Pattern

   This scheme allows some flexibility in finding a match.  For example, lookup
   provides better results for cases in which content is not available
   that exactly matches the user request than if the default language
   for the system or content were returned immediately.  Not every
   specific level of tag granularity is usually available or language
   content may be sparsely populated, so "falling populated.  "Falling back" through the subtag
   sequence provides more opportunity to find a match between available
   language tags and the user's request.

   The default behavior when no tag matches the language priority list
   is implementation defined.  An implementation might, for example,
   return content content:

   o  with no language tag; might supply content tag

   o  of a non-linguistic nature, such as an image or sound

   o  with an empty language tag value (the built-in attribute xml:lang value, in [XML10] cases where the protocol
      permits the empty value); might be value (see, for example, "xml:lang" in [XML10],
      which indicates that the element contains non-linguistic content)

   o  in a particular language designated for the bit of content being selected; or it might select
      selected

   o  labelled with the tag "i-default" (see [RFC2277]). [RFC2277])

   When performing lookup using a language priority list, the
   progressive search MUST proceed to consider process each language range in the list
   before finding the default content or empty tag.

   One common way for an application or implementation to provide for a
   default is to allow a specific language range to be set as the
   default for a specific type of request.  This language range is then
   treated as if it were appended to the end of the language priority
   list as a whole, rather than after each item in the language priority
   list.

   For example, if a particular user's language priority list were
   "fr-FR;
   "fr-FR, zh-Hant" and the program doing the matching had a default
   language range of "ja-JP", the program would search for content as
   follows:
   1. fr-FR
   2. fr
   3. zh-Hant // next language
   4. zh
   5. (search for the default content)
      a. ja-JP
      b. ja
      c. (implementation defined default)

   Figure 4: Lookup Using a Language Priority List

   Implementations SHOULD ignore extensions and unrecognized private-use
   subtags when performing lookup, since these subtags are usually
   orthogonal to the user's request.

   The special language range "*" matches any language tag.  In the
   lookup scheme, this range does not convey enough information by
   itself to determine which content is most appropriate, since it
   matches everything.  If the language range "*" is the only one in the followed by other
   language priority list, ranges, it matches the default content. SHOULD be skipped.  If the language range "*" is followed by
   the only one in the language priority list or if no other language ranges, it should
   range follows, the default content SHOULD be
   skipped. returned.

   In some cases, the language priority list might contain one or more
   extended language ranges (as, for example, when the same language
   priority list is used as input for both lookup and filtering
   operations).  Wildcard values in an extended language range normally
   match any value that occurs in that position in a language tag.
   Since only one item can be returned for any given lookup request,
   wildcards in a language range have to be processed in a consistent
   manner or the same request will produce widely varying results.
   Implementations that accept extended language ranges MUST define
   which content is returned when more than one item matches the
   extended language range.

   For example, an implementation could return the matching tag that is
   first in ASCII-order.  If the language range were "*-CH" and the set
   of tags included "de-CH", "fr-CH", and "it-CH", then the tag "de-CH"
   would be returned.  Another example possibility would be for an
   implementation to map the extended language ranges to basic ranges.

4.  Other Considerations

   When working with language ranges and matching schemes, there are
   some additional points that may influence the choice of either.

4.1.  Choosing Language Ranges

   Users indicate their language preferences via the choice of a
   language range or the list of language ranges in a language priority
   list.  The type of matching affects what the best choice is for a
   given user.

   Most matching schemes make no attempt for a
   user.

   Most matching schemes make no attempt to process the semantic meaning
   of the subtags and the language range is compared, in a case-
   insensitive manner, to each language tag being matched, using basic
   string processing.  Users SHOULD select language ranges that are
   well-formed, valid language tags according to [RFC3066bis]
   (substituting wildcards as appropriate in extended language ranges).

   Users SHOULD replace tags or subtags which have been deprecated with
   the Preferred-Value from the IANA Language Subtag Registry.  If the
   user is working with content that might use the older form, the user
   might include both the new and old forms in a language priority list.
   For example, the tag "art-lojban" is deprecated.  The subtag 'jbo' is
   supposed to process be used instead, so the semantic meaning
   of user might use it to form the subtags.  The
   language range (or its subtags) is usually
   compared range.  Or the user might include both in a case-insensitive manner to each language tag being
   matched, using basic string processing.
   priority list: "jbo, art-lojban".

   Users SHOULD avoid subtags that add no distinguishing value to a
   language range.  Generally,  When filtering, the fewer the number of subtags that
   appear in the language range, the more content the range will
   probably match, while in lookup unnecessary subtags might cause
   "better", more-specific content to be skipped in favor of less
   specific content.  For example, the range "de-Latn-DE" would return
   content tagged "de" instead of content tagged "de-DE", even though
   the latter is probably a better match.

   Most notably,

   Many languages are written predominantly in a single script.  This is
   usually recorded in the Suppress-Script field in that language
   subtag's registry entry.  For these languages, script subtags SHOULD
   NOT be used to form a language
   range in combination with language subtags that have a matching
   Suppress-Script field in their registry entry. range.  Thus the language range "en-Latn" "en-
   Latn" is probably inappropriate in most cases (because the vast majority of
   English documents are written in the Latin script and thus the 'en'
   language subtag has a Suppress-Script field for 'Latn' in the
   registry).

   When working with tags and ranges, note that extensions and most
   private-use subtags are orthogonal to language tag matching, in that
   they specify additional attributes of the text not related to the
   goals of most matching schemes.  Users SHOULD avoid using these
   subtags in language ranges, since they interfere with the selection
   of available content.  When used in language tags (as opposed to
   ranges), these subtags normally do not interfere with filtering
   (Section 3), since they appear at the end of the tag and will match
   all prefixes.

   Private-use and Extension subtags are normally orthogonal to language
   tag fallback.  Implementations or specifications that use a lookup  Lookup (Section 3.3) matching scheme implementations often ignore
   unrecognized private-use and extension subtags when performing
   language tag fallback.  In
   addition, since these subtags are always at the end of the sequence
   of subtags, their use in language tags normally doesn't interfere
   with the use of ranges that omit them in the filtering (Section 3.2)
   matching schemes described below.  However, they do interfere with
   filtering when used in language ranges and SHOULD be avoided in
   ranges as a result.

   Applications, specifications, or protocols that choose not to
   interpret one or more private-use or extension subtags SHOULD NOT
   remove or modify these extensions in content that they are
   processing.  When a language tag instance is to be used in a
   specific, known protocol, and is not being passed through to other
   protocols, language tags MAY be filtered altered to remove subtags and
   extensions that are not supported by that protocol.  Such filtering alterations
   SHOULD be avoided, if possible, since it removes they remove information that
   might be relevant to services on the other end of the protocol elsewhere that would make use of that information.

   Some applications of language tags might want or need to consider
   extensions and private-use subtags when matching tags.  If extensions
   and private-use subtags are included in a matching or filtering process that
   utilizes one of the schemes described in this document, then the
   implementation SHOULD canonicalize the language tags and/or ranges
   before performing the matching.  Note that language tag processors
   that claim to be "well-formed" processors as defined in [RFC3066bis]
   generally fall into this category.

4.2.  Meaning of Language Tags and Ranges

   Selecting content using language ranges requires some understanding
   by users of what they are selecting.  The meaning of the various
   subtags in a language range are identical to their meaning in a
   language tag (see Section 4.2 in [RFC3066bis]), with the addition
   that the wildcard "*" represents any matching sequence of values.

4.3.  Considerations for Private Use Subtags

   Private-use subtags require private agreement between the parties
   that intend to use or exchange language tags that use them and great
   caution SHOULD be used in employing them in content or protocols
   intended for general use.  Private-use subtags are simply useless for
   information exchange without prior arrangement.

   The value and semantic meaning of private-use tags and of the subtags
   used within such a language tag are not defined.  Matching private-
   use tags using language ranges or extended language ranges can result
   in unpredictable content being returned.

4.4.  Length Considerations in Matching for Language Ranges

   Language ranges are very similar to language tags in terms of content
   and usage.  The same types of restrictions on length that apply to
   language tags could can also apply to language ranges.  Implementation,
   protocol, and specificiation authors SHOULD apply the considerations
   in  See [RFC3066bis]
   Section 4.3 (Length Considerations) where appropriate
   to language ranges and language priority lists. Considerations).

5.  IANA Considerations

   This document presents no new or existing considerations for IANA.

6.  Changes

   This is the first version of this document.

7.  Security Considerations

   Language ranges used in content negotiation might be used to infer
   the nationality of the sender, and thus identify potential targets
   for surveillance.  In addition, unique or highly unusual language
   ranges or combinations of language ranges might be used to track a
   specific individual's activities.

   This is a special case of the general problem that anything you send
   is visible to the receiving party.  It is useful to be aware that
   such concerns can exist in some cases.

   The evaluation of the exact magnitude of the threat, and any possible
   countermeasures, is left to each application or protocol.

8.  Character Set Considerations

   Language tags permit only the characters A-Z, a-z, 0-9, and HYPHEN-
   MINUS (%x2D).  Language ranges also use the character ASTERISK
   (%x2A).  These characters are present in most character sets, so
   presentation or exchange of language tags or ranges should not be
   constrained by character set issues.

9.  References

9.1.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119, March 1997.

   [RFC2277]  Alvestrand, H., "IETF Policy on Character Sets and
              Languages", BCP 18, RFC 2277, January 1998.

   [RFC3066bis]
              Phillips, A., Ed. and M. Davis, Ed., "Tags for the
              Identification of Languages", October 2005, <http://
              www.ietf.org/internet-drafts/
              draft-ietf-ltru-registry-14.txt>.

   [RFC4234]  Crocker, D. and P. Overell, "Augmented BNF for Syntax
              Specifications: ABNF", RFC 4234, October 2005.

9.2.  Informative References

   [RFC1766]  Alvestrand, H., "Tags for the Identification of
              Languages", RFC 1766, March 1995.

   [RFC2616]  Fielding, R., Gettys, J., Mogul, J., Frystyk, H.,
              Masinter, L., Leach, P., and T. Berners-Lee, "Hypertext
              Transfer Protocol -- HTTP/1.1", RFC 2616, June 1999.

   [RFC2616errata]
              IETF, "HTTP/1.1 Specification Errata", 10 2004,
              <http://purl.org/NET/http-errata>.

   [RFC3066]  Alvestrand, H., "Tags for the Identification of
              Languages", BCP 47, RFC 3066, January 2001.

   [RFC3282]  Alvestrand, H., "Content Language Headers", RFC 3282,
              May 2002.

   [XML10]    Bray (et al), T., "Extensible Markup Language (XML) 1.0",
              02 2004.

Appendix A.  Acknowledgements

   Any list of contributors is bound to be incomplete; please regard the
   following as only a selection from the group of people who have
   contributed to make this document what it is today.

   The contributors to [RFC3066bis], [RFC3066] and [RFC1766], each of
   which is a precursor to this document, made enormous contributions
   directly or indirectly to this document and are generally responsible
   for the success of language tags.

   The following people (in alphabetical order by family name)
   contributed to this document:

   Harald Alvestrand, Jeremy Carroll, John Cowan, Martin Duerst, Frank
   Ellermann, Doug Ewell, Marion Gunn, Kent Karlsson, Ira McDonald, M.
   Patton, Randy Presuhn, Eric van der Poel, Markus Scherer, and many,
   many others.

   Very special thanks must go to Harald Tveit Alvestrand, who
   originated RFCs 1766 and 3066, and without whom this document would
   not have been possible.

   For this particular document, John Cowan originated the scoring
   scheme.  Mark Davis originated the scheme described in Section 3.3.

Authors' Addresses

   Addison Phillips (editor)
   Yahoo! Inc

   Email: addison at inter dash locale dot com

   Mark Davis (editor)
   Google

   Email: mark dot davis at macchiato dot com

Intellectual Property Statement

   The IETF takes no position regarding the validity or scope of any
   Intellectual Property Rights or other rights that might be claimed to
   pertain to the implementation or use of the technology described in
   this document or the extent to which any license under such rights
   might or might not be available; nor does it represent that it has
   made any independent effort to identify any such rights.  Information
   on the procedures with respect to rights in RFC documents can be
   found in BCP 78 and BCP 79.

   Copies of IPR disclosures made to the IETF Secretariat and any
   assurances of licenses to be made available, or the result of an
   attempt made to obtain a general license or permission for the use of
   such proprietary rights by implementers or users of this
   specification can be obtained from the IETF on-line IPR repository at
   http://www.ietf.org/ipr.

   The IETF invites any interested party to bring to its attention any
   copyrights, patents or patent applications, or other proprietary
   rights that may cover technology that may be required to implement
   this standard.  Please address the information to the IETF at
   ietf-ipr@ietf.org.

Disclaimer of Validity

   This document and the information contained herein are provided on an
   "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
   OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET
   ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED,
   INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE
   INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
   WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Copyright Statement

   Copyright (C) The Internet Society (2006).  This document is subject
   to the rights, licenses and restrictions contained in BCP 78, and
   except as set forth therein, the authors retain all their rights.

Acknowledgment

   Funding for the RFC Editor function is currently provided by the
   Internet Society.