Network Working Group A. Phillips, Ed. Internet-Draft Quest Software Obsoletes: 3066 (if approved) M. Davis, Ed. Expires:
May 22,June 10, 2006 IBM November 18,December 7, 2005 Matching of Language Tags draft-ietf-ltru-matching-07draft-ietf-ltru-matching-08 Status of this Memo By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she becomes aware will be disclosed, in accordance with Section 6 of BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on May 22,June 10, 2006. Copyright Notice Copyright (C) The Internet Society (2005). Abstract This document describes different mechanisms for comparing, matching, and evaluating language tags. Possible algorithms for language negotiation andor content selectionselection, filtering, and lookup are described. This document, in combination with RFC 3066bis (replace "3066bis" with the RFC number assigned to draft-ietf-ltru-registry-14), replaces RFC 3066, which replaced RFC 1766. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 2. The Language Range . . . . . . . . . . . . . . . . . . . . . . 4 2.1. Lists of Language Ranges . . . . . . . . . . . . . . . . . 4 2.2.Basic Language Range . . . . . . . . . . . . . . . . . . . 4 220.127.116.11. Extended Language Range . . . . . . . . . . . . . . . . . 5 2.4. Choosing a2.3. The Language RangePriority List . . . . . . . . . . . . . . . . 67 3. Types of Matching . . . . . . . . . . . . . . . . . . . . . . 98 3.1. Choosing a Type of Matching . . . . . . . . . . . . . . . 98 3.2. Filtering . . . . . . . . . . . . . . . . . . . . . . . . 109 3.2.1. Filtering with Basic Language Ranges . . . . . . . . . 1110 3.2.2. Filtering with Extended Language Ranges . . . . . . . 11 3.2.3. Scored Filtering . . . . . . . . . . . . . . . . . . . 1211 3.3. Lookup . . . . . . . . . . . . . . . . . . . . . . . . . . 1415 4. Other Considerations . . . . . . . . . . . . . . . . . . . . . 18 4.1. Choosing Language Ranges . . . . . . . . . . . . . . . . . 18 4.2. Meaning of Language Tags and Ranges . . . . . . . . . . . 18 4.2.19 4.3. Considerations for Private Use Subtags . . . . . . . . . . 19 4.3.20 4.4. Length Considerations in Matching . . . . . . . . . . . . 1921 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 2223 6. Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . 2324 7. Security Considerations . . . . . . . . . . . . . . . . . . . 2425 8. Character Set Considerations . . . . . . . . . . . . . . . . . 2526 9. References . . . . . . . . . . . . . . . . . . . . . . . . . . 2627 9.1. Normative References . . . . . . . . . . . . . . . . . . . 2627 9.2. Informative References . . . . . . . . . . . . . . . . . . 2627 Appendix A. Acknowledgements . . . . . . . . . . . . . . . . . . 2728 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 2829 Intellectual Property and Copyright Statements . . . . . . . . . . 2930 1. Introduction Human beings on our planet have, past and present, used a number of languages. There are many reasons why one would want to identify the language used when presenting or requesting information. Information about a user's language preferences commonly needneeds to be identified so that appropriate processing can be applied. For example, the user's language preferences in a browser can be used to select web pages appropriately. Language preferences can also be used to select among tools (such as dictionaries) to assist in the processing or understanding of content in different languages. Given a set of language identifiers, such as those defined in [RFC3066bis], various mechanisms can be envisioned for performing language negotiation and tag matching. This document defines a syntax (called a language range (Section 2)) for specifying a user's language preferences, as well as several schemes for selecting or filtering content by comparing language ranges to the language tags [RFC3066bis] used to identify the natural language of that content. Applications, protocols, or specifications will have varying needs and requirements that affect the choice of a suitable mechanism.matching scheme. Depending on the choice of scheme, there are various options left to the implementation. Protocols that implement a matching scheme either need to choose a particular option or indicate that the particular options is left to the specific implementation to decide. This document defines several mechanismsis divided into three main sections. One describes how to indicate a user's preferences using language ranges. Then a section describes various schemes for matching, selecting, or filtering content whose naturalmatching these ranges to a set of language tags in order to select specific content. There is identified using Language Tags [RFC3066bis], as well as the syntax (calledalso a "language range") associatedsection that deals with each ofvarious practical considerations that apply to implementing and using these mechanisms for specifying the user's language preferences.schemes. This document, in combination with [RFC3066bis] (replace(Ed.: replace "3066bis" globally in this document with the RFC number assigned to draft-ietf-ltru-registry-14), replaces [RFC3066], which replaced [RFC1766]. The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119]. 2. The Language Range Language Tags [RFC3066bis] are used to identify the language of some information item or content. Applications or protocols that use language tags are often faced with the problem of identifying sets of content that share certain language attributes. For example, HTTP/1.1 [RFC2616] describes language ranges inone such mechanism in its discussion of the Accept-Language header (Section 14.4). These are to be14.4), which is used when selecting content from servers based on the language of that content. When selecting content according to its language, it is useful to have a mechanism for identifying sets of language tags that share specific attributes. This allows users to select or filter content based on specific requirements. Such an identifier is called a "Language Range". Language tags and thus language ranges are to be treated as casecase- insensitive: there exist conventions for the capitalization of some of the subtags, but these MUST NOT be taken to carry meaning. Matching of language tags to language ranges MUST be done in a casecase- insensitive manner as well. 2.1. Lists of Language Ranges When users specify a language preference they often need to specify a prioritized list of language ranges in order to best reflect their language preferences. This is especially true for speakers of minority languages. A speaker of Breton in France, for example, may specify "be" followed by "fr", meaning that if Breton is available, it is preferred, but otherwise French is the best alternative. It can get more complex: a speaker may wish to fall back from Skolt Sami to Northern Sami to Finnish. A "Language Priority List" consists of a prioritized or weighted list of language ranges. One well known example of such a list is the "Accept-Language" header defined in RFC 2616 [RFC2616] (see Section 14.4) and RFC 3282 [RFC3282]. The various matching operations described in this document include considerations for using a language priority list. When given as examples in this document, language priority lists will be shown as a quoted sequence of ranges separated by semi-colons, like this: "en; fr; zh-Hant" (which would be read as "English before French before Chinese as written in the Traditional script"). 2.2.Basic Language Range A "Basic Language Range""basic language range" identifies the set of content whose language tags begin with the same sequence of subtags. AEach range consists of a sequence of alphanumeric subtags separated by hyphens. The basic language range is identified by its 'language range' tag,defined by adaptingthe ABNF[RFC4234] from HTTP/1.1 [RFC2616] :following the ABNF[RFC4234]: language-range = language-tag / "*" language-tag = 1*8[alphanum] *["-" 1*8alphanum] alphanum = ALPHA / DIGIT That is, a language-range hasBasic language ranges (originally described by HTTP/1.1 [RFC2616] and later [RFC3066]) have the same syntax as a language-tagan [RFC3066] language tag or isare the single character "*". Basic Language Ranges implyThey differ from the language tags defined in [RFC3066bis] only in that there is no requirement that they be "well-formed" or be validated against the IANA Language Subtag Registry (although such ill-formed ranges will probably not match anything). Use of a basic language range seems to imply that there is a semantic relationship between language tags that share the same prefix. While this is often the case, it is not always true and users should note that the set of language tags that match a specific language-range may not be mutually intelligible. Basic language ranges were originally described in [RFC3066] and HTTP/1.1 [RFC2616] (where they are referred to as simply a "language range"). 18.104.22.168. Extended Language Range A Basic Language Range does not always provide the most appropriate way to specify a user's preferences. Sometimes it is beneficial to use a more granularfine-grained matching scheme that takes advantage of the internal structure of language tags, by allowingtags. This allows the user to specify, for example, the value of a specific field in a language tag or to indicate which values are of interest in filtering or selecting the content. In an extended language range, the identifier takes the form of a series of subtags which MUST consist of well-formed subtags or the special subtag "*". For example, the language range "en-*-US" specifies a primary language of 'en', followed by any script subtag, followed by the region subtag 'US'. An extended language range can be represented by the following ABNF: extended-language-range = range ; a range / privateuse ; private-use tag / grandfathered ; grandfathered registrations range = (language ["-" script] ["-" region] *("-" variant) *("-" extension) ["-" privateuse]) language = (2*3ALPHA [ extlang ]) ; shortest ISO 639 code / 4ALPHA ; reserved for future use / 5*8ALPHA ; registered language subtag / "*" ; ... or wildcard extlang = *2("-" 3ALPHA) ("-" ( 3ALPHA / "*")) ; reserved for future use ; wildcard can only appear ; at the end script = 4ALPHA ; ISO 15924 code / "*" ; or wildcard region = 2ALPHA ; ISO 3166 code / 3DIGIT ; UN M.49 code / "*" ; ... or wildcard variant = 5*8alphanum ; registered variants / (DIGIT 3alphanum) ; / "*" ; ... or wildcard extension = singleton *("-" (2*8alphanum)) [ "-*" ] ; extension subtags ; wildcard can only appear ; at the end singleton = %x41-57 / %x59-5A / %x61-77 / %x79-7A / DIGIT ;"a"-"w" / "y"-"z" / "A"-"W" / "Y"-"Z" / "0"-"9" ; Single letters: x/X is reserved for private use privateuse = ("x"/"X") 1*("-" (1*8alphanum)) grandfathered = 1*3ALPHA 1*2("-" (2*8alphanum)) ; grandfathered registration ; Note: I is the only singleton ; that starts a grandfathered tag alphanum = (ALPHA / DIGIT) ; letters and numbers A field not present in the middle of an extended language range MAY beis treated as if the field contained a "*". For example, the range "en-US" MAY be considered to be equivalent to the range "en-*-US". This also means that multiple wildcards can be collapsed (soImplementations that "en-*-*-US" is equivalent to "en-*-US"). 2.4. Choosing a Language Range Users indicate their language preferences via the choice of a language range or the set ofnormalize extended language ranges in the language priority list. The type of matching will affect what the best choice is for given user. In addition, user's should be aware that, when working with language ranges, most matching schemes make no attemptSHOULD expand missing fields to processbe "*" so that the semantic meaning of the subtags. The language tag andlanguage range (or their subtags) are usually comparedis clear to the user. At the same time, multiple wildcards in a case insensitive manner using basic string processing. Thusrow are redundant and implementations SHOULD collapse these to a single wildcard when normalizing the choice of subtags inrange (for brevity). For example, both the language tag and languagerange may affect"sl-nedis" and the results produced. Users SHOULD avoid subtags that add no distinguishing valuerange "sl-*-*-nedis" are equivalent to and should be normalized as "sl-*-nedis". 2.3. The Language Priority List When users specify a language range. For example, script subtags SHOULD NOT be usedpreference they often need to formspecify a prioritized list of language range with language subtags which have a matching Suppress-Script fieldranges in order to best reflect their registry record. Thus thelanguage range "en-Latn"preferences. This is probably inappropriate in most cases (because the vast majorityespecially true for speakers of English documents are writtenminority languages. A speaker of Breton in France, for example, may specify "be" followed by "fr", meaning that if Breton is available, it is preferred, but otherwise French is the Latin script and thus the 'en'best alternative. It can get more complex: a speaker may wish to fall back from Skolt Sami to Northern Sami to Finnish. A "Language Priority List" is a prioritized or weighted list of language subtag hasranges. One well known example of such a Suppress-Script field for 'Latn' inlist is the registry). When working with tags"Accept-Language" header defined in RFC 2616 [RFC2616] (see Section 14.4) and RFC 3282 [RFC3282]. A simple list of ranges, notei.e. one that extensions and most private-use subtags are orthogonalcontains no weighting information, is considered to language tag fallback and users SHOULD avoid using these subtagsbe in language ranges, since they will often interfere with the selectiondescending order of availablepriority. The various matching operations described in this document include considerations for using a language content. Since these subtags are always atpriority list. This document does not define any syntax for a language priority list; defining such a syntax is the endresponsibility of the protocol, application, or implementation that uses it. When given as examples in this document, language priority lists will be shown as a quoted sequence of subtags, they don't normally interfere with the use of prefixes for the filtering schemes described below in Section 3. When working with tags andranges users SHOULD note the following: 1. Private-use and Extension subtags are normally orthogonal to language tag fallback. Implementations or specifications that use a lookup (Section 3.3) matching scheme SHOULD ignore unrecognized private-use and extension subtags when performing language tag fallback. Since these subtags are always at the end of the sequence of subtags, they don't normally interfere with the use of prefixes for matching in the schemes described below. 2. Applications, specifications, or protocols that choose not to interpret one or more private-use or extension subtags SHOULD NOT remove or modify these extensions in content that they are processing. When a language tag instance is to be used in a specific, known protocol, and is not being passed through to other protocols, language tags MAY be filtered to remove subtags and extensions that are not supportedseparated by that protocol. Such filtering SHOULD be avoided, if possible, since it removes information that might be relevant if services on the other end of the protocolsemi-colons, like this: "en; fr; zh-Hant" (which would make use of that information. 3. Some applications of language tags might want or need to consider extensions and private-use subtags when matching tags. If extensions and private-use subtags are included in a matching or filtering process that utilizes the one of the schemes described in this document, then the implementation SHOULD canonicalize the language tags and/or ranges before performing the matching. Note that language tag processors that claim tobe "well-formed" processorsread as defined"English before French before Chinese as written in [RFC3066bis] generally fall into this category.the Traditional script"). 3. Types of Matching Matching language ranges to language tags can be done in a number of different ways. This section describes theseveral different types ofmatching scheme,schemes, as well as the considerations for choosing between them. Protocols and specifications SHOULD clearly indicate the particular mechanism used in selecting or matching language tags. There are two basic types of matching scheme: those that produce an open-ended set of contentzero or more information items (called "filtering") and those that produce a single information item for a given request (called "lookup"). A key difference between these two types of matching scheme is that the language range for filtering operations is alwaysranges in the language priority list represent the _least_ specific tagcontent one will accept as a match, while for lookup operations the language range is alwaysranges represent the _most_ specific tag.content. 3.1. Choosing a Type of Matching Applications, protocols, and specifications are faced with the decision of what type of matching to use. Sometimes, different styles of matching might be suited for different kinds of processing within a particular application or protocol. Language tag matching is a tool, and does not by itself specify a complete procedure for the use of language tags. Such procedures are intimately tied to the application protocol in which they occur. When specifying a protocol operation using matching, the protocol MUST specify: o Which type(s) of language tag matching it uses o Whether the operation returns a single result (lookup) or a possibly empty set of results (filtering) o For lookup, what the result is when no matching tag is found. For instance, a protocol might result in failure of the operation, an empty value, returning some protocol defined or implementation defined default, or returning i-default [RFC2277]. Filtering can be used to produce a set of results (such as a collection of documents). For example, if using a search engine, one might use filtering to limit the results to documents written in French. It can also be used when deciding whether to perform some processing that is language sensitivea language-sensitive process on some content. For example, a process might cause paragraphs whose language tag matched the language range "nl" to be displayed in italics within a document. This document describes threefour types of matching (three types of filtering:filtering, plus the lookup scheme): 1. Basic Filtering (Section 3.2.1) is used to match content using basic language ranges (Section 2.2). It is compatible with implementations that do not produce extended language ranges.2.1). 2. Extended Range Filtering (Section 3.2.2) is used to match content using extended language ranges (Section 2.3). Newer implementations SHOULD use this form of filtering in preference to basic filtering.2.2). 3. Scored Filtering (Section 3.2.3) produces an ordered set of content using either basic orextended language ranges. It SHOULD be used when the quality of the match within a specific language range is important, as when presenting a list of documents resulting from a search. 4. Lookup (Section 3.3) is used when each request MUSTneeds to produce exactly_exactly_ one piece of content. For example, if process were to insert a Web serverhuman readable error message into a protocol header, it might useselect the Accept-Language HTTP header to choose whichtext based on the user's language to return a custom 404 page in: sincepreference. Since it can return only one page,item, it must choose a single item and it must return some item, even if no content matches the language rangespriority list supplied by the user. Most types of matching in this document are designed so that implementations doare not haverequired to examinevalidate or understand any of the valuessemantics of the subtags supplied and, except for scored filtering, they do not need access to the IANA Language Subtag Registry nor do they require(see Section 3 in [RFC3066bis]). This simplifies and speeds the useperformance of valid subtags inimplementations. If an implementation canonicalizes either language tagsranges or language ranges. This has great benefittags, then the implementation will require the IANA Language Subtag Registry information for speed and simplicity of implementation.that purpose. Implementations might also wish toMAY use semantic information external to the language tagsregistry when performing fallback.matching tags. For example, the primary language subtags 'nn' (Nynorsk Norwegian) and 'nb' (Bokmal Norwegian) might both be usefully matched to the more general subtag 'no' (Norwegian). Or an implementation might infer that content labeled "zh-CN" is more likely to match the range "zh-Hans""zh- Hans" than equivalent content labeled "zh-TW". 3.2. Filtering Filtering is used to select the set of content that matches a given prefix.language priority list. It is called "filtering" because this set of content may contain no items at all or it may return an arbitraryarbitrarily large number of matching items--as many as match the language range used to specify the items, thus filtering out the non-matching content. In filtering, the language range represents the _least_ specific (that is, the fewest number of subtags) language tag which is an acceptable match. That is, all of the language tags in the set of filtered content will have an equal or greater number of subtags than the language range. For example, if the language priority list consists of the range is"de-CH", one might see matching content with the tag "de-CH-1996" but one will never see a match with the tag "de". If the language priority list (see Section 2.1)2.3) contains more than one range, the content returned is typically ordered in descending level of preference. Some examples where filtering might be appropriate include: o Applying a style to sections of a document in a particular language range.set of languages. o Displaying the set of documents containing a particular set of keywords written in a specific language.set of languages. o Selecting all email items written in a specific rangeset of languages. Filtering can produce either an ordered or an unordered set of results. For example, applying formatting to a document based on the language of specific pieces of content does not require the content to be ordered. It is sufficient to know whether a specific piece of content matches or does not match.is selected by the language priority list (or not). A search application, on the other hand, probably would putwant to order the results into a priority order.results. If an ordered set is desired, as described above, then the application or protocol needs to determine the relative "quality" of the match between different language tags and the language range. This measurement is called a "distance metric". A distance metric assigns a numeric value to the comparison of eacha language tag to a language range andthat represents the 'distance' between the two. A distance of zero means that they are identical, a small distance indicates that they are very similar, and a large distance indicatedindicates that they are very different. Using a distance metric, implementations can, for example, allow users to select a threshold distance for a match to be "successful" while filteringfiltering, or it canthey might use the numeric valuevalues to order the results. 3.2.1. Filtering with Basic Language Ranges When filtering using abasic language range, theranges, each basic language range in the language priority list is considered in turn, according to priority. A particular language tag matches a language tagrange if it exactly equals the tag, or if it exactly equals a prefix of the tag such that the first character following the prefix is "-". (That is, the language-range "de-de" matches the language tag "de-DE-1996", but not the language tag "de-Deva".) The special range "*" in a language priority list matches any tag. A protocol which uses language ranges MAY specify additional rules about the semantics of "*"; for instance, HTTP/1.1 [RFC2616] specifies that the range "*" matches only languages not matched by any other range within an "Accept-Language" header. 3.2.2. Filtering with Extended Language Ranges In the Extended Range Matching scheme,When filtering using extended language ranges, each extended language range in the language priority list is considered in turn, according to priority. The subtags in each extended language range are compared to the corresponding subtags in the language tag being examined. The subtag from the range is considered to match if it exactly matches the corresponding subtag in the tag or the range's subtag has the value "*" (which matches all subtags, including the empty subtag). Extended Range Matching is an extension of basic matching (Section 3.2.1): the language range represents the least specific tag which is an acceptable match. private-use subtags MAY be specified in the language range and MUST NOT be ignored when matching.Subtags not specified, including those at the end of the language range, are assigned the wildcard value "*". This makes each range into a prefix much like that used in basic language range matching. For example, the extended language range "de-*-DE" matches all of the following tags because the unspecified variant field is expanded to "*": de-DE de-Latn-DE de-Latf-DE de-DE-x-goethe de-Latn-DE-1996 3.2.3. Scored Filtering Both basic and extended language range filtering produce simple boolean matches.matches between a language range and a language tag. Sometimes it may be beneficialuseful to provide an array of results with different levels of matching, for example, sorting results based on the overall "quality" of the match. Scored (or "distance metric") filtering provides a way to generate these quality values. First bothAs with the other forms of filtering, the process considers each language range in the language priority list in order of priority. Each extended language range and thelanguage tags to be matched to it musttag MUST first be canonicalized by mapping grandfathered and obsolete tags into modern equivalents. This requires the information in the IANA Language Subtag Registry (see Section 3 of [RFC3066bis]). The language range and theeach language tagstag it is to be compared to are then transformed into quintuples of elementsa "quintuple" consisting of five "elements" in the form (language, script, country, variant, extension). Any extended language subtags are considered part of the language element; private-use"element". For example, the language element for the tag "zh-cmn- Hans" would be "zh-cmn". Private-use subtag sequences are considered part of the language element"element" if in the initial position in the tag and part of the variant element"element" if not. Language subtags 'und', 'mul', and the script subtag 'Zyyy' are converted to "*". MissingThe different handling of private-use sequences prevents a range such as "x-twain" from matching all possible tags, while a range such as "en-US-x-twain" would closely match nearly all tags for English as used in the United States. Language subtags 'und', 'mul', and the script subtag 'Zyyy' are converted to "*": these subtag values represent undetermined, multiple, or private-use values which are consistent with the use of the wildcard. For language tags that have no script subtag but whose language subtag's record in the IANA Language Subtag Registry contains the field "Suppress-Script", the script element in the quintuple MUST be set to the script subtag in the Suppress-Script field. This is necessary because [RFC3066bis] strongly recommends that users not use this subtag to form language tags and this document recommends that users not use them to form ranges. For example, if the script were not expanded in this manner, a range such as "de-DE" would produce a more-distant score for content that happened to be labeled "de-Latn-DE" than users would expect that it should. Note that languages which have a "Suppress-Script" field in the registry are predominantly written in a single script. Any remaining missing components in the language-taglanguage tag are set to "*"; thus a "*" patternan empty language tag becomes the quintuple ("*", "*", "*", "*", "*"). Missing components in the extended language-rangelanguage range are handled similarly to extended range lookup: missing internal subtags are expanded to "*". Missing end subtags are expanded as the empty string. Thus a pattern "en-US" becomes the quintuple ("en","*","US","",""). Here are some examples of language tags, showing their quintuples as both language tags and language ranges: en-US Tag: (en, *, US, *, *) Range: (en, *, US, "", "") sr-Latn Tag: (sr, Latn, *, *, *) Range: (sr, Latn, "", "", "") zh-cmn-Hant Tag: (zh-cmn, Hant, *, *, *) Range: (zh-cmn, Hant, "", "", "") x-foo Tag: (x-foo, *, *, *, *) Range: (x-foo, "", "", "", "") en-x-foo Tag: (en, *, *, x-foo, *) Range: (en, *, *, x-foo, "") i-default Tag: (i-default, *, *, *, *) Range: (i-default, "", "", "", "") sl-Latn-IT-rozaj Tag: (sl, Latn, IT, rozaj, *) Range: (sl, Latn, IT, rozaj, "") zh-r-wadegile (hypothetical) Tag: (z.,(zh, *, *, *, r-wadegile) Range: (z.,(zh, *, *, *, r-wadegile) Figure 3: Examples of Distance Metric Quintuples Each language-range/language-tagpair of quintuples being compared is assigned a distance value, wherebyin which small values indicate better matches and large values indicate worse ones. The distance between the pair is the sum of the distances for each of the corresponding elements of the quintuple. If the elements are identical or one is '*', then the distance value between them is zero. Otherwise, it is given by the following table: 256 language mismatch 128 script mismatch 32 region mismatch 4 variant mismatch 1 extension mismatch A value of 0 is a perfect match; 421 is no match at all. Different threshold values might be appropriate for different applications or protocols. Implementations will usually allow users to choose the most appropriate selection value, ranking the matched items based on score. Examples of various tag's distances from the range "en-US": "fr-FR" 384 (language & region mismatch) "fr" 256 (language mismatch, region match) "en-GB" 32 (region mismatch) "en-Latn-US" 0 (all fields match) "en-Brai" 32 (region mismatch) "en-US-x-foo" 4 (variant mismatch: range is the empty string) "en-US-r-wadegile" 1 (extension mismatch: range is the empty string) ImplementationsNote: A variation of this algorithm might vary the scoring used overall or protocolsfor specific values. For example, sometimes it might wishmake sense to use more sophisticated weightsweighting that dependdepends on the values of the corresponding elements. For example,Thus, depending on the domain, an implementation might giveassign a smallsmaller distance to the difference between closely related subtags.subtags (or treat certain values as equal). Some examples of closely related subtags might be: Language: no (Norwegian) nb (Bokmal Norwegian) nn (Nynorsk Norwegian) Script: Kata (katakana) Hira (hiragana) Region: US (United States of America) UM (United States Minor Outlying IslandsIslands) Figure 6: Examples of Closely Related Subtags 3.3. Lookup Lookup is used to select the single information item that best matches the language priority list for a given request. InWhen performing lookup, each language-rangelanguage range in the language priority list is considered in turn, according to priority. By contrast with filtering, each language ranges represents the _most_ specific tag which is an acceptable match; only the closest matchingmatch. The first information item found with a matching tag, according the user's prioritypriority, is returned. For example, ifconsidered the closest match and is the item returned. For example, if the language range is "de-CH", one might expect to receive an information item with the tag "de" but never one with the tag "de-CH-1996". Usually if no content matches the request, a "default" item is returned. For example, if an application inserts some dynamic content into a document, returning an empty string if there is no exact match is not an option. Instead, the application "falls back" until it finds a suitable piece of content to insert. Other examples of lookup might include: o Selection of a template containing the text for an automated email response. o Selection of a graphicitem containing some text for inclusion in a particular Web page. o Selection of a string of text for inclusion in an error log. In the Lookuplookup scheme, the language-rangelanguage range is progressively truncated from the end until a matching piece of content is located. For example, starting with the range "zh-Hant-CN-x-private", the lookup wouldprogressively searchsearches for content as shown below: Range to match: zh-Hant-CN-x-private 1. zh-Hant-CN-x-private 2. zh-Hant-CN 3. zh-Hant 4. z.zh 5. (default content or the empty tag) Figure 7: Example of a Lookup Fallback Pattern This scheme allows some flexibility in finding content. It also typicallyFor example, it provides better results whenfor cases in which data is not available at a specific level of tag granularity or is sparsely populated (thanthat exactly matches the user request than if the default language for the system or content were used). The language range "*" matches anyreturned immediately. Not every specific level of tag granularity is usually available or language tag. Incontent may be sparsely populated, so "falling back" through the lookup scheme, this language range does not convey enough informationsubtag sequence provides more opportunity to determine whichfind a match between available content and the user's request. The default content is most appropriate. If thisimplementation defined. It might be content with no language range is the only onetag; might have an empty value (the built-in attribute xml:lang in [XML10] permits the empty value); might be a particular language priority list, it matches the default content. If this language range is followed by other language ranges,designated for that bit of content; or it shouldmight be skipped.content that is labeled with the tag "i-default" (see [RFC2277]). When performing lookup using a language priority list, the progressive search MUST proceed to consider each language range in the list before finding the default content or empty tag. The default content might be content with no language tag (or with an empty value, as with xml:lang in the XML specification), or it might be a particular language designated for that bit of content.One common way for an application or implementation to provide for default content is to allow a specific language range to be set as the default for a specific type of request. This language range is then treated as if it were appended to the end of the language priority list,list as a whole, rather than after each item in the language priority list. For example, if a particular user's language priority list were "fr-FR; zh-Hant" and the program doing the matching had a default language range of "ja-JP", the program would search for content as follows: 1. fr-FR 2. fr 3. zh-Hant // next language 4. z.zh 5. (return(search for the default content) a. ja-JP b. ja c. (empty tag or other default content)(implementation defined default) Figure 8: Lookup Using a Language Priority List In some cases, the language priority list might contain one or more extended language ranges (as, for example,Implementations SHOULD ignore extensions and unrecognized private-use subtags when performing lookup, since these subtags are usually orthogonal to the sameuser's request. The special language priorityrange "*" matches any language tag. In the lookup scheme, this range does not convey enough information by itself to determine which content is most appropriate, since it matches everything. If the language range "*" is the only one in the language priority list, it matches the default content. If the language range "*" is followed by other language ranges, it should be skipped. In some cases, the language priority list might contain one or more extended language ranges (as, for example, when the same language priority list is used as input for both lookup and filtering operations). Wildcard values in an extended language range are supposed tonormally match any value that occurs in that position in a language tag. Since only one item can be returned for any given lookup request, thewildcards mustin a language range have to be processed in a predictableconsistent manner (oror the same request mightwill produce widely varying results). Thus, for each rangeresults. Implementations that accept extended language ranges MUST define which content is returned when more than one item matches the extended language range. For example, an implementation could return the matching content that is first in ASCII-order. For example, if the language priority list,range were "*-CH" and the following rules mustset of content included "de-CH", "fr-CH", and "it-CH", then the content labeled "de-CH" would be returned. Another way an implementation could address extended language ranges would be appliedto produce amap them to basic language range for use in the fallback mechanism: 1. Ifranges: if the first subtag in the extended language rangeis a "*" then the entire range is converted to "*". 2. For each subsequent subtag, if the value is atreated as "*" then that(which matches the default content), otherwise the wildcard subtag and its preceding hyphen areis removed. For example: *-US becomes * en-*-US becomes en-US en-Latn-* becomes en-Latn Figure 9: Transformation of Extended Language Ranges Forexample, if the language priority list "*-US; fr-*-FR; zh-Hant",range were "en-*-US", then the fallback patternrange would be: 1. * (skipped) 2. fr-FR 3. fr 4. zh-Hant 5. z. 6. (default content) Figure 10: Extended Language Range Fallback Examplebe mapped to "en-US". 4. Other Considerations When working with language rangesranges and matching schemes, there are some additional points that may influence the choice of either. 4.1. Choosing Language Ranges Users indicate their language preferences via the choice of a language range or the list of language ranges in a language priority list. The type of matching affects what the best choice is for a given user. Most matching schemes make no attempt to process the semantic meaning of the subtags. The language range (or its subtags) is usually compared in a case-insensitive manner to each language tag being matched, using basic string processing. Users SHOULD avoid subtags that add no distinguishing value to a language range. Generally, the fewer subtags that appear in the language range, the more content the range will match. Most notably, script subtags SHOULD NOT be used to form a language range in combination with language subtags that have a matching Suppress-Script field in their registry entry. Thus the language range "en-Latn" is probably inappropriate in most cases (because the vast majority of English documents are written in the Latin script and thus the 'en' language subtag has a Suppress-Script field for 'Latn' in the registry). When working with tags and ranges, note that extensions and most private-use subtags are orthogonal to language tag matching, in that they specify additional attributes of the text not related to the goals of most matching schemes. Users SHOULD avoid using these subtags in language ranges, since they interfere with the selection of available content. When used in language tags (as opposed to ranges), these subtags normally do not interefer with filtering (Section 3), since they appear at the end of the tag and will match all prefixes. When working with language tags and language ranges note that: o Private-use and Extension subtags are normally orthogonal to language tag fallback. Implementations or specifications that use a lookup (Section 3.3) matching scheme often ignore unrecognized private-use and extension subtags when performing language tag fallback. In addition, since these subtags are always at the end of the sequence of subtags, their use in language tags normally doesn't interfere with the use of ranges that omit them in the filtering (Section 3.2) matching schemes described below. However, they do interfere with filtering when used in language ranges and SHOULD be avoided in ranges as a result. o Applications, specifications, or protocols that choose not to interpret one or more private-use or extension subtags SHOULD NOT remove or modify these extensions in content that they are processing. When a language tag instance is to be used in a specific, known protocol, and is not being passed through to other protocols, language tags MAY be filtered to remove subtags and extensions that are not supported by that protocol. Such filtering SHOULD be avoided, if possible, since it removes information that might be relevant to services on the other end of the protocol that would make use of that information. o Some applications of language tags might want or need to consider extensions and private-use subtags when matching schemes, theretags. If extensions and private-use subtags are some additional pointsincluded in a matching or filtering process that may influence the choiceutilizes one of either. 4.1.the schemes described in this document, then the implementation SHOULD canonicalize the language tags and/or ranges before performing the matching. Note that language tag processors that claim to be "well-formed" processors as defined in [RFC3066bis] generally fall into this category. 4.2. Meaning of Language Tags and Ranges Selecting content using language ranges requires some understanding by users of what they are selecting. A language tag or range identifies a language as spoken (or written, signed or otherwise signaled) by human beings for communication of information to other human beings. If a language tag B contains language tag A as a prefix, then B is typically "narrower" or "more specific" than A. For example, "zh- Hant-TW" is more specific than "zh-Hant". This relationship is not guaranteed in all cases: specifically, languages that begin with the same sequence of subtags are NOT guaranteed to be mutually intelligible, although they might be. For example, the tag "az" shares a prefix with both "az-Latn" (Azerbaijani written using the Latin script) and "az-Arab" (Azerbaijani written using the Arabic script). A person fluent in one script might not be able to read the other, even though the text might be otherwise identical. Content tagged as "az" most probably is written in just one script and thus might not be intelligible to a reader familiar with the other script. Variant subtags in particular seem to represent specific divisions in mutual understanding, since they often encode dialects or other idiosyncratic variations within a language. They also seem to represent relatively low divisions with a high chance of at least limited understanding, although this depends on the specific variant in question. The relationship between the language tag and the information it relates to is defined by the standard describing the context in which it appears. Accordingly, this section can only give possible examples of its usage: o For a single information object, the associated language tags might be interpreted as the set of languages that are necessary for a complete comprehension of the complete object. Example: Plain text documents. o For an aggregation of information objects, the associated language tags could be taken as the set of languages used inside components of that aggregation. Examples: Document stores and libraries. o For information objects whose purpose is to provide alternatives, the associated language tags could be regarded as a hint that the content is provided in several languages, and that one has to inspect each of the alternatives in order to find its language or languages. In this case, the presence of multiple tags might not mean that one needs to be multi-lingual to get complete understanding of the document. Example: MIME multipart/ alternative. o In markup languages, such as HTML and XML, language information can be added to each part of the document identified by the markup structure (including the whole document itself). For example, one could write <span lang="FR">C'est la vie.</span> inside a Norwegian document; the Norwegian-speaking user could then access a French-Norwegian dictionary to find out what the marked section meant. If the user were listening to that document through a speech synthesis interface, this formation could be used to signal the synthesizer to appropriately apply French text-to-speech pronunciation rules to that span of text, instead of misapplying the Norwegian rules. 22.214.171.124. Considerations for Private Use Subtags Private-use subtags require private agreement between the parties that intend to use or exchange language tags that use them and great caution SHOULD be used in employing them in content or protocols intended for general use. Private-use subtags are simply useless for information exchange without prior arrangement. The value and semantic meaning of private-use tags and of the subtags used within such a language tag are not defined. Matching private- use tags using language ranges or extended language ranges can result in unpredictable content being returned. 126.96.36.199. Length Considerations in Matching RFC 3066 [RFC3066] did not provide an upper limit on the size of language tags or ranges. RFC 3066 did define the semantics of particular subtags in such a way that most language tags or ranges consisted of language and region subtags with a combined total length of up to six characters. Larger tags and ranges (in terms of both subtags and characters) did exist, however. [RFC3066bis] also does not impose a fixed upper limit on the number of subtags in a language tag or range (and thus an upper bound on the size of either). The syntax in that document suggests that, depending on the specific language or range of languages, more subtags (and thus characters) are sometimes necessary as a result. Length considerations and their impact on the selection and processing of tags are described in Section 2.1.1 of that document. An application or protocol MAY choose to limit the length of the language tags or ranges used in matching. Any such limitation SHOULD be clearly documented, and such documentation SHOULD include the disposition of any longer tags or ranges (for example, whether an error value is generated or the language tag or range is truncated). If truncation is permitted it MUST NOT permit a subtag to be divided, since this changes the semantics of the subtag being matched and can result in false positives or negatives. Applications or protocols that restrict storage SHOULD consider the impact of tag or range truncation on the resulting matches. For example, removing the "*" from the end of an extended language range (see Section 2.3)2.2) can greatly modify the set of returned matches. A protocol that allows tags or ranges to be truncated at an arbitrary limit, without giving any indication of what that limit is, has the potential for causing harm by changing the meaning of values in substantial ways. In practice, most tags do not require additional subtags or substantially more characters. Additional subtags sometimes add useful distinguishing information, but extraneous subtags interfere with the meaning, understanding, and especially matching of language tags. Since language tags or ranges MAY be truncated by an application or protocol that limits storage, when choosing language tags or ranges users and applications SHOULD avoid adding subtags that add no distinguishing value. In particular, users and implementations SHOULD follow the 'Prefix' and 'Suppress-Script' fields in the registry (defined in Section 3.6 of [RFC3066bis]): these fields provide guidance on when specific additional subtags SHOULD (and SHOULD NOT) be used. Implementations MUST support a limit of at least 33 characters. This limit includes at least one subtag of each non-extension, non-private use type. When choosing a buffer limit, a length of at least 42 characters is strongly RECOMMENDED. The practical limit on tags or ranges derived solely from registered values is 42 characters. Implementations MUST be able to handle tags and ranges of this length. Support for tags and ranges of at least 62 characters in length is RECOMMENDED. Implementations MAY support longer values, including matching extensive sets of private-use or extension subtags. Applications or protocols which have to truncate a tag MUST do so by progressively removing subtags along with their preceding "-" from the right side of the language tag until the tag is short enough for the given buffer. If the resulting tag ends with a single-character subtag, that subtag and its preceding "-" MUST also be removed. For example: Tag to truncate: zh-Latn-CN-variant1-a-extend1-x-wadegile-private1 1. zh-Latn-CN-variant1-a-extend1-x-wadegile 2. zh-Latn-CN-variant1-a-extend1 3. zh-Latn-CN-variant1 4. zh-Latn-CN 5. zh-Latn 6. z.zh Figure 11:9: Example of Tag Truncation 5. IANA Considerations This document presents no new or existing considerations for IANA. 6. Changes This is the first version of this document. The following changes were put into this document since draft-06: Changed the document title from the unwieldy "Matching Tags for the Identificationdraft-07: Added a mention of Languages" to "Matching Language Tags" (Ed.) Fixed problems with the distance metric filtering scheme (Section 3.2.3) examples (in which tags were expanded incorrectly). (D.Ewell) Moved the sentence "Protocols and specifications SHOULD clearly indicate the particular mechanism used in selecting or matching language tags." from the introduction (where there should not be any normative language)"*" to the start of Section 3. (A.Phillips) CreatedCharacter Set Considerations section Section 2.4 and moved text there (A.Phillips) Modified the examples of closely related subtags in Section 3.2.3 to show what the examples mean (M.Duerst) Various spelling and grammatical fixes(D.Ewell) 7. Security Considerations Language ranges used in content negotiation might be used to infer the nationality of the sender, and thus identify potential targets for surveillance. In addition, unique or highly unusual language ranges or combinations of language ranges might be used to track a specific individual's activities. This is a special case of the general problem that anything you send is visible to the receiving party. It is useful to be aware that such concerns can exist in some cases. The evaluation of the exact magnitude of the threat, and any possible countermeasures, is left to each application or protocol. 8. Character Set Considerations The syntax of languageLanguage tags and language rangespermit only the characters A-Z, a-z, 0-9, and HYPHEN-MINUSHYPHEN- MINUS (%x2D). Language ranges also use the character ASTERISK (%x2A). These characters are present in most character sets, so presentation or exchange of language tags or ranges should not present anybe constrained by character set issues. 9. References 9.1. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. [RFC2616] Fielding, R., Gettys, J., Mogul, J., Frystyk,[RFC2277] Alvestrand, H., Masinter, L., Leach, P.,"IETF Policy on Character Sets and T. Berners-Lee, "Hypertext Transfer Protocol -- HTTP/1.1",Languages", BCP 18, RFC 2616, June 1999.2277, January 1998. [RFC3066bis] Phillips, A., Ed. and M. Davis, Ed., "Tags for the Identification of Languages", October 2005, <http:// www.ietf.org/internet-drafts/ draft-ietf-ltru-registry-14.txt>. [RFC4234] Crocker, D. and P. Overell, "Augmented BNF for Syntax Specifications: ABNF", RFC 4234, October 2005. 9.2. Informative References [RFC1766] Alvestrand, H., "Tags for the Identification of Languages", RFC 1766, March 1995. [RFC2616] Fielding, R., Gettys, J., Mogul, J., Frystyk, H., Masinter, L., Leach, P., and T. Berners-Lee, "Hypertext Transfer Protocol -- HTTP/1.1", RFC 2616, June 1999. [RFC3066] Alvestrand, H., "Tags for the Identification of Languages", BCP 47, RFC 3066, January 2001. [RFC3282] Alvestrand, H., "Content Language Headers", RFC 3282, May 2002. [XML10] Bray (et al), T., "Extensible Markup Language (XML) 1.0", 02 2004. Appendix A. Acknowledgements Any list of contributors is bound to be incomplete; please regard the following as only a selection from the group of people who have contributed to make this document what it is today. The contributors to [RFC3066bis], [RFC3066] and [RFC1766], each of which is a precursor to this document, made enormous contributions directly or indirectly to this document and are generally responsible for the success of language tags. The following people (in alphabetical order by family name) contributed to this document: Harald Alvestrand, Jeremy Carroll, John Cowan, Martin Duerst, Frank Ellermann, Doug Ewell, Marion Gunn, Kent Karlsson, Ira McDonald, M. Patton, Randy Presuhn, Eric van der Poel, and many, many others. Very special thanks must go to Harald Tveit Alvestrand, who originated RFCs 1766 and 3066, and without whom this document would not have been possible. For this particular document, John Cowan originated the scheme described in Section 3.2.3. Mark Davis originated the scheme described in the Section 3.3. Authors' Addresses Addison Phillips (editor) Quest Software Email: addison dot phillips at quest dot com Mark Davis (editor) IBM Email: mark dot davis at ibm dot com Intellectual Property Statement The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79. Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at firstname.lastname@example.org. Disclaimer of Validity This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Copyright Statement Copyright (C) The Internet Society (2005). This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights. Acknowledgment Funding for the RFC Editor function is currently provided by the Internet Society.