Network Working Group A. Phillips, Ed. Internet-Draft Yahoo!
IncInc. Obsoletes: 3066 (if approved) M. Davis, Ed. Expires: September 5,October 8, 2006 Google March 4,April 6, 2006 Matching of Language Tags draft-ietf-ltru-matching-11draft-ietf-ltru-matching-12 Status of this Memo By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she becomes aware will be disclosed, in accordance with Section 6 of BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on September 5,October 8, 2006. Copyright Notice Copyright (C) The Internet Society (2006). Abstract This document describes different mechanisms for comparing and matching language tags. Possible algorithms for language negotiation or content selection, filtering, and lookup are described. This document, in combination with RFC 3066bis (Ed.: replace "3066bis" with the RFC number assigned to draft-ietf-ltru-registry-14), replaces RFC 3066, which replaced RFC 1766. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 2. The Language Range . . . . . . . . . . . . . . . . . . . . . . 4 2.1. Basic Language Range . . . . . . . . . . . . . . . . . . . 4 2.2. Extended Language Range . . . . . . . . . . . . . . . . . 5 2.3. The Language Priority List . . . . . . . . . . . . . . . . 5 3. Types of Matching . . . . . . . . . . . . . . . . . . . . . . 7 3.1. Choosing a Type of Matching . . . . . . . . . . . . . . . 7 3.2. Implementation Considerations . . . . . . . . . . . . . . 8 3.3. Filtering . . . . . . . . . . . . . . . . . . . . . . . . 9 22.214.171.124.3.1. Basic Filtering . . . . . . . . . . . . . . . . . . . 9 126.96.36.199 3.3.2. Extended Filtering . . . . . . . . . . . . . . . . . . 10 188.8.131.52. Lookup . . . . . . . . . . . . . . . . . . . . . . . . . . 1112 3.4.1. Default Values . . . . . . . . . . . . . . . . . . . . 14 4. Other Considerations . . . . . . . . . . . . . . . . . . . . . 1516 4.1. Choosing Language Ranges . . . . . . . . . . . . . . . . . 1516 4.2. Meaning of Language Tags and Ranges . . . . . . . . . . . 1617 4.3. Considerations for Private Use Subtags . . . . . . . . . . 1617 4.4. Length Considerations for Language Ranges . . . . . . . . 1718 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 18 6. Changes . . . . . . . . . . . . . . . . . . . . . . . . . . .19 7.6. Security Considerations . . . . . . . . . . . . . . . . . . . 20 8.7. Character Set Considerations . . . . . . . . . . . . . . . . . 21 9.8. References . . . . . . . . . . . . . . . . . . . . . . . . . . 22 184.108.40.206. Normative References . . . . . . . . . . . . . . . . . . . 22 220.127.116.11. Informative References . . . . . . . . . . . . . . . . . . 22 Appendix A. Acknowledgements . . . . . . . . . . . . . . . . . . 23 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 24 Intellectual Property and Copyright Statements . . . . . . . . . . 25 1. Introduction Human beings on our planet have, past and present, used a number of languages. There are many reasons why one would want to identify the language used when presenting or requesting information or in some specific set of information itemsitems. Applications, protocols, or "content". Onespecifications that use forlanguage identifiers, such as thosethe language tags defined in [RFC3066bis], issometimes need to select content by matching the associatedmatch language tags to a user's language preferences. This document defines a syntax (called a language range (Section 2)) for specifying items in the user's list of language preferences (called a language priority list (Section 2.3)), as well as several schemes for selecting or filtering sets of contentlanguage tags by comparing the content'slanguage tags to the user's preferences. Applications, protocols, or specifications will have varying needs and requirements that affect the choice of a suitable matching scheme. Depending on the choice of scheme, there are various options left to the implementation. Protocols that implement a matching scheme either need to specify each particular choice or indicate the options that are left to the implementation to decide.This document is divided into three main sections. One describesdescribes: how to indicate a user's preferences using language ranges. Then a section describes variousranges; three schemes for matching these ranges to a set of language tags. There is also a section that deals withtags; and the various practical considerations that apply to implementing and using these schemes. This document, in combination with [RFC3066bis] (Ed.: replace "3066bis" globally in this document with the RFC number assigned to draft-ietf-ltru-registry-14), replaces [RFC3066], which replaced [RFC1766]. The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119]. 2. The Language Range Language Tagstags [RFC3066bis] are used to help identify languages, whether spoken, written, signed, or otherwise signaled, for the languagepurpose of some information item or "content". Applicationscommunication. Applications, protocols, or protocolsspecifications that use language tags are often faced with the problem of identifying sets of content that share certain language attributes. For example, HTTP/1.1 [RFC2616] describes one such mechanism in its discussion of the Accept-Language header (Section 14.4), which is used when selecting content from servers based on the language of that content. When selecting content according to its language, it isIt is, thus, useful to have a mechanism for identifying sets of language tags that share specific attributes. This allows users to select or filter contentthe language tags based on specific requirements. Such an identifier is called a "language range". There are different types of language range, whose specific attributes vary according to their application. Language ranges are similar to language tags: they consist of a sequence of subtags separated by hyphens. In a language range, each subtag MUST either be a sequence of ASCII alphanumeric characters or the single character '*' (%2A, ASTERISK). The character '*' is a "wildcard" that matches any sequence of subtags. The meaning and uses of wildcards vary according to the type of language range. Language tags and thus language ranges are to be treated as case- insensitive: there exist conventions for the capitalization of some of the subtags, but these MUST NOT be taken to carry meaning. Matching of language tags to language ranges MUST be done in a case- insensitive manner. 2.1. Basic Language Range A "basic language range" describes a user's language preference as a specific, uninterrupted, sequence of subtags. Each rangeconsists of a sequence of alphanumeric subtags separated by hyphens. The basic language rangeIt is defined by the following ABNF [RFC4234]: language-range = (1*8ALPHA *("-" 1*8alphanum)) / "*" alphanum = ALPHA / DIGIT Basic language ranges (originally described by HTTP/1.1 [RFC2616] and later [RFC3066]) have the same syntax as an [RFC3066] language tag or are the single character "*". They differ from the language tags defined in [RFC3066bis] only in that there is no requirement that they be "well-formed" or be validated against the IANA Language Subtag Registry (although suchRegistry. Such ill-formed ranges will probably not match anything). (Noteanything. Note that the ABNF [RFC4234] in [RFC2616] is incorrect, since it disallows the use of digits anywhere in the 'language-range': this is mentioned in the errata) Use of a basic language range seems'language-range' (see: [RFC2616errata]). 2.2. Extended Language Range Occasionally users will wish to imply that there isselect a semantic relationship betweenset of language tags that share the same prefix. While this is often the case, it is not always true and users should note that the set of language tags that match a specific language range may not represent mutually intelligible languages. 2.2. Extended Language Range Occasionally users will wish to select a set of language tags based onbased on the presence of specific subtags. An "extended language range" describes a user's language preference as an ordered sequence of subtags. For example, a user might wish to select all language tags that contain the region subtag 'CH' (Switzerland). Extended language ranges are useful in specifying a particular sequence of subtags that appear in the set of matching tags without having to specify all of the intervening subtags. An extended language range can be represented by the following ABNF: extended-language-range = (1*8ALPHA / "*") *("-" (1*8alphanum / "*")) Figure 2: Extended Language Range The wildcard subtag '*' can occur in any position in the extended language range, where it matches any sequence of subtags that might occur in that position in a language tag. However, wildcards outside the first position in an extended language rangeare ignored by most matching schemes. UseExtended Filtering (see Section 3.2.2). The use or absence of one or more wildcards SHOULD NOTcannot be taken to imply that a certain number of subtags will appear in the matching set of language tags. Implementations that specify basic ranges MAY map extended language ranges to basic language ranges: if the first subtag is a "*" then the entire range is treated as "*", otherwise each wildcard subtag is removed. For example, if the language range were "en-*-US", then the range would be mapped to "en-US".2.3. The Language Priority List A user's language preferences will often need to specify more than one language range and thus users often need to specify a prioritized list of language ranges in order to best reflect their language preferences. This is especially true for speakers of minority languages. A speaker of Breton in France, for example, may specify "be""br" followed by "fr", meaning that if Breton is available, it is preferred, but otherwise French is the best alternative. It can get more complex: a user may wish to fall back from Skolt Sami to Northern Sami to Finnish. A "language priority list" is a prioritized or weighted list of language ranges. One well known example of such a list is the "Accept-Language" header defined in RFC 2616 [RFC2616] (see Section 14.4) and RFC 3282 [RFC3282]. The various matching operations described in this document include considerations for using a language priority list. This document does not define the syntax for a language priority list; defining such a syntax is the responsibility of the protocol, application, or specification that uses it. When given as examples in this document, language priority lists will be shown as a quoted sequence of ranges separated by commas, like this: "en, fr, zh-Hant" (which would be read as "English before French before Chinese as written in the Traditional script"). A simple list of ranges is considered to be in descending order of priority. Other language priority lists provide "quality weights" for the language ranges in order to specify the relative priority of the user's language preferences. An example of this would be the use of "q" values in the syntax of the "Accept-Language" header (defined in [RFC2616], Section 14.4, and [RFC3282]). 3. Types of Matching Matching language ranges to language tags can be done in a number ofmany different ways. This section describes several differentthree such matching schemes, as well as the considerations for choosing between them. Protocols and specifications SHOULDrequiring conformance to this specification MUST clearly indicate the particular mechanism used in selecting or matching language tags. There are severaltwo types of matching scheme. This document presents two types: thosescheme in this document. A matching scheme that produceproduces zero or more information items (called "filtering") and thosematching language tags is called "filtering". A matching scheme that produce a single information itemproduces exactly one match for a given request (called "lookup"). Implementations or protocols MAY use different matching schemes from the ones described in this document, as long as those mechanisms are clearly specified.is called "lookup". 3.1. Choosing a Type of Matching Applications, protocols, and specifications are faced with the decision of what type of matching to use. Sometimes, different styles of matching are suited to different kinds of processing within a particular application or protocol. Language tag matching is a tool, and does not by itself specifyThis document describes three types of matching: 1. Basic Filtering (Section 3.3.1) matches a complete procedure for the uselanguage priority list consisting of basic language tags. Such procedures are intimately tiedranges (Section 2.1) to the application protocol in which they occur. When specifying a protocol operation using matching, the protocol MUST specify: o Which type(s) of language tag matching it uses o Whether the operation returns a single result (lookup) or a possibly empty set of results (filtering) o For lookup, what the result is when no matching tag is found. For instance, a protocol might define the result as failure of the operation, an empty value, returning some protocol defined or implementation defined default, or returning i-default [RFC2277]. This document describes three types of matching: 1. Basic Filtering (Section 3.2.1) matches a language priority list consisting of basic language ranges (Section 2.1) to setssets of language tags. 2. Extended Filtering (Section 3.2.2)3.3.2) matches a language priority list consisting of extended language ranges (Section 2.2) to sets of language tags. 3. Lookup (Section 3.3)3.4) matches a language priority list consisting of basic language ranges to sets of language tags to find the one _exact_ language tag that best matches the range. Filtering can be used to produce a set of results (such as a collection of documents) by comparing the user's preferences to language tags associated with thea set of content.language tags. For example, when performing a search, one might use filtering to limit the results to items tagged as being in the French language. Filtering can also be used when deciding whether to perform a language-sensitive process on some content. For example, a process might cause paragraphs whose language tag matched the language range "nl" to be displayed in italics within a document. Lookup produces the single result that best matches the user's preferences,preferences from the list of available tags, so it is useful in cases in which a single item is required (and for which only a single item can be returned.returned). For example, if a process were to insert a human readable error message into a protocol header, it might select the text based on the user's language priority list. Since the process can return only one item, it must choose a single item and it must return some item, even if none of the content's language tags match the language priority list supplied by the user. The types of3.2. Implementation Considerations Language tag matching in this documentis a tool, and does not by itself specify a complete procedure for the use of language tags. Such procedures are designed so that implementationsintimately tied to the application protocol in which they occur. When specifying a protocol operation using matching, the protocol MUST specify: o Which type(s) of language tag matching it uses o Whether the operation returns a single result (lookup) or a possibly empty set of results (filtering) o For lookup, what the default item is (or the sequence of operations or configuration information used to determine the default) when no matching tag is found. For instance, a protocol might define the result as failure of the operation, an empty value, returning some protocol defined or implementation defined default, or returning i-default [RFC2277]. Applications, protocols, and specifications are not required to validate or understand any of the semantics of the language tags or ranges or of the subtags in them. None of themthem, nor do they require access to the IANA Language Subtag Registry (see Section 3 in [RFC3066bis]). This simplifies implementationimplementation. However, designers of these schemes. An implementation MAY choose to check if either the language rangesapplications, protocols, or language tags being matchedspecifications are "well-formed" or "valid" (see [RFC3066bis], Section 2.2.9) and MAY choose notencouraged to process invalid ranges. Regardless ofuse the matching scheme chosen, protocols and implementations MAY canonicalizeinformation from the IANA Language Subtag Registry to support canonicalizing language tags and ranges by mappingin order to map grandfathered and obsolete tags or subtags into modern equivalents. If an implementation canonicalizes either rangesApplications, protocols, or tags, thenspecifications that canonicalize ranges MUST either perform matching operations with both the implementation will requirecanonical and original (unmodified) form of the IANA Language Subtag Registry information for that purpose. Implementations MAYrange or MUST also use semantic information external tocanonicalize each tag for the registry when matching tags.purposes of comparison. Note that canonicalizing language ranges makes certain operations impossible. For example, the primary language subtags 'nn' (Nynorsk Norwegian) and 'nb' (Bokmal Norwegian) might both be usefully matched to the more general subtag 'no' (Norwegian). Oran implementation might inferthat content labeled "zh-Hans" (Chinese as written incanonicalizes the Simplified script) is more likelylanguage range "art-lojban" to matchuse the range "zh-CN" (Chinese asmore modern "jbo" cannot be used in China, whereto select just the Simplified script is predominant) than equivalent content labeled "zh-TW" (Chinese as used in Taiwan, whereitems with the Traditional script is predominant). 3.2. Filtering Filtering is usedolder tag. Applications, protocols, or specifications that use basic ranges might sometimes receive extended language ranges instead. An application, protocol, or specification MUST choose to: a) map extended language ranges to selectbasic ranges using the set ofalgorithm below, b) reject any extended language tags that matches a givenranges in the language priority list and return the associated content. It is called "filtering" because this set might contain no items at allthat are not valid basic language ranges, or itc) treat each extended language range as if it were a basic language range, which will have the same result as ignoring them, since these ranges will won't match any valid language tags. An extended language range is mapped to a basic language range as follows: if the first subtag is a '*' then the entire range is treated as "*", otherwise each wildcard subtag is removed. For example, if the language range were "en-*-US", then the range would be mapped to "en-US". Applications, protocols, or specifications, in addressing their particular requirements, can offer pre-processing or configuration options. For example, an implementation could allow a user to associate or map a particular language range to a different value. Such a user might wish to associate the language range subtags 'nn' (Nynorsk Norwegian) and 'nb' (Bokmal Norwegian) with the more general subtag 'no' (Norwegian). Or perhaps the user could associate the range "zh-Hans" (Chinese as written in the Simplified script) with the language tag "zh-CN" (Chinese as used in China, where the Simplified script is predominant) because content is available with that tag. Documentation on how the ranges or tags are altered, prioritized, or compared in the subsequent match in such an implementation will assist users in making the best configuration choices. 3.3. Filtering Filtering is used to select the set of language tags that matches a given language priority list. It is called "filtering" because this set might contain no items at all or it might return an arbitrarily large number of matching items: as many items as match the language priority list, thus "filtering out" the non-matching items. In filtering, each language range represents the _least_ specific language tag (that is, the language tag with fewest number of subtags) which is an acceptable match. All of the language tags in the matching set of tags will have an equal or greater number of subtags than the language range. Every non-wildcard subtag in the language range will appear in every one of the matching language tags. For example, if the language priority list consists of the range "de-CH", one might see tags such as "de-CH-1996" but one will never see a tag such as "de" (because the 'CH' subtag is missing). If the language priority list (see Section 2.3) contains more than one range, the content returned is typically ordered in descending level of preference, but it MAY be unordered, according to the needs of the application or protocol. Some examples of applications where filtering might be appropriate include: o Applying a style to sections of a document in a particular set of languages. o Displaying the set of documents containing a particular set of keywords written in a specific set of languages. o Selecting all email items written in a specific set of languages. o Selecting audio files spoken in a particular language. 3.2.1. BasicFiltering When filtering using basic language ranges, each basicseems to imply that there is a semantic relationship between language range intags that share the language priority listsame prefix. While this is considered in turn,often the case, it is not always true and users should note that the set of language tags that match a specific language range do not necessarily represent mutually intelligible languages. 3.3.1. Basic Filtering Basic filtering uses basic language ranges. Each basic language range in the language priority list is considered in turn, according to priority. A particularlanguage tagrange matches a particular language rangetag if, in a case-insensitive comparison, it exactly equals the tag, or if it exactly equals a prefix of the tag such that the first character following the prefix is "-". For example, the language-range "de-de" matches the language tag "de-DE-1996", but not the language tags "de- Deva" or "de-Latn-DE". The special range "*" in a language priority list matches any tag. A protocol which uses language ranges MAY specify additional rules about the semantics of "*"; for instance, HTTP/1.1 [RFC2616] specifies that the range "*" matches only languages not matched by any other range within an "Accept-Language" header. Basic filtering is identical to the type of matching described in [RFC3066], Section 2.5 (Language-range). 18.104.22.168.3.2. Extended Filtering WhenExtended filtering usingcompares extended language ranges, eachranges to language tags. Each extended language range in the language priority list is considered in turn, according to priority. A language range matches a particular language tag if their list of subtags match. To determine a match: 1. Split both the extended language range is compared to eachand the language tag usingbeing compared into a list of subtags by dividing on the following process: Comparehyphen (%2D) character. Two subtags match if either they are the first subtag insame when compared case-insensitively or the extendedlanguage tag torange's subtag is the wildcard '*'. 2. Begin with the first subtag in the language tag in a case insensitive manner.each list. If the first subtag in the range is "*", it matches any value. Otherwise the two values mustdoes not match orthe first subtag in the tag, the overall match fails. Take each non-wildcardOtherwise, move to the next subtag in both the languagerange and compare it in a case-insensitive manner tothe next subtagtag. 3. While there are more subtags left in the language tag.range's list: A. If the range'ssubtag exactly matchescurrently being examined in the tag's subtag, proceedrange is the wildcard ('*'), move to the next non-wildcardsubtag in the languagerange (and beginningand continue with the next subtagloop. B. Else, if there are no more subtags in the language tag) untiltag's list, the match fails. C. Else, if the current subtag in the range's list of subtagsmatches the current subtag in the language range is exhausted ortag's list, move to the match fails. Ifnext subtag in both lists and continue with the loop. D. Else, if the language tag's subtag is a "singleton" (a single letter or digit, which, in this case,which includes the private-use subtag 'x') andthe range's subtag does notmatch or iffails. E. Else, move to the next subtag in the language tag's list of subtags is exhausted,and continue with the match fails. Ifloop. 4. When the language range's list of subtags is exhausted,has no more subtags, the match succeeds. Subtags not specified, including those at the end of the language range, are thus treated as if assigned the wildcard value "*".'*'. Much like basic filtering, extended filtering selects content with arbitrarily long tags that share the same initial subtags as the language range. In additionaddition, extended filtering selects content withlanguage tags that contain any intermediate subtags unspecifiednot specified in the language range. For example, the extended language range "de-*-DE" (or its synonym "de-DE") matches all of the following tags: de-DE de-Latn-DE de-Latf-DE de-de de-DE-x-goethe de-Latn-DE-1996 de-Deva-DE The same range does not match any of the following tags for the reasons shown: de (missing 'DE') de-x-DE (singleton 'x' occurs before 'DE') de-Deva ('Deva' not equal to 'DE') Note: The structure of language tags defined by[RFC3066bis] defines each type of subtag (language, script, region, and so forth) according to position, size, and content. This means that subtags in a language range can only match specific types of subtags in a language tag. For example, a subtag such as 'Latn' is always a script subtag (unless it follows a singleton) while a subtag such as 'nedis' can only match the equivalent variant subtag. 3.3.One such difference is that two-letter subtags in initial position have a different type (language) than two-letter subtags in later positions (region). This is the reason why a wildcard in the extended language range is significant in the first position and subsequently ignored. 3.4. Lookup Lookup is used to select the single language tag that best matches the language priority list for a given request and return the associated content.request. When performing lookup, each language range in the language priority list is considered in turn, according to priority. By contrast with filtering, each language range represents the _most_ specific tag which is an acceptable match. The first content found with amatching tag,tag found, according to the user's priority, is considered the closest match and is the contentitem returned. For example, if the language range is "de-ch", a lookup operation mightcan produce content with the tags "de" or "de-CH" but never onecontent with the tag "de-CH-1996". Usually ifIf no contentlanguage tag matches the request, the "default" contentvalue is returned. For example, if an application inserts some dynamic content into a document, returning an empty string if there is no exact match is not an option. Instead, the application "falls back" until it finds a matching language tag associated with a suitable piece of content to insert. Examples of lookup might include: o Selection of a template containing the text for an automated email response. o Selection of a item containing some text for inclusion in a particular Web page. o Selection of a string of text for inclusion in an error log. o Selection of an audio file to play as a prompt in a phone system. In the lookup scheme, the language range is progressively truncated from the end until a matching piece of contentlanguage tag is located. Single letter or digit subtags (including both the letter 'x' which introduces private-use sequences, and the subtags that introduce extensions) are removed at the same time as their closest trailing subtag. For example, starting with the range "zh-Hant-CN-x-private1- private2","zh-Hant-CN-x-private1-private2", the lookup progressively searches for content as shown below: Range to match: zh-Hant-CN-x-private1-private2 1. zh-Hant-CN-x-private1-private2 2. zh-Hant-CN-x-private1 3. zh-Hant-CN 4. zh-Hant 5. zh 6. (default content)(default) Figure 3: Example of a Lookup Fallback Pattern This allows some flexibility in finding a match. For example, lookup provides better results for cases in which content is not available that exactly matches the user request than if the default language for the system or content were returned immediately. NotLanguage material is sometimes sparsely populated, so an item might not be available at every specificlevel of tag granularity is usually available or language content may be sparsely populated.granularity. "Falling back" through the subtag sequence provides more opportunity to find a match between available language tags and the user's request. The default behavior when no tag matches the language priority list is implementation defined. An implementation might, for example, return content: o with no language tag o of a non-linguistic nature, such as an image or sound o with an empty language tag value, in cases where the protocol permits the empty value (see, for example, "xml:lang" in [XML10], which indicates that the element contains non-linguistic content) o inExtensions and unrecognized private-use subtags might be unrelated to a particular language designated forapplication of lookup. Since these subtags come at the bitend of content being selected o labelled withthe tag "i-default" (see [RFC2277]) When performing lookup using a language priority list,subtag sequence, they are removed first during the progressive search MUSTfallback process each language range in the list before finding the default content or empty tag. One common way forand usually pose no barrier to interoperability. However, an application orimplementation MAY remove these from ranges prior to provide for a default is to allow a specific language range to be set asperforming the default for a specific type of request. This language range is then treated as if it were appended tolookup (provided the end ofimplementation also removes them from the language priority list as a whole, rather than after each item intags being compared). Such modification is internal to the language priority list. For example, if a particular user's language priority list were "fr-FR, zh-Hant"implementation and the program doing the matching had a default language range of "ja-JP", the program would search for content as follows: 1. fr-FR 2. fr 3. zh-Hant // next language 4. zh 5. (search for the default content) a. ja-JP b. ja c. (implementation defined default) Figure 4: Lookup Using a Language Priority List Implementationsapplications, protocols, or specifications SHOULD ignore extensions and unrecognized private-use subtags when performing lookup, since theseNOT remove or modify subtags are usually orthogonal to the user's request.in content that they return or forward, because this removes information that might be used elsewhere. The special language range "*" matches any language tag. In the lookup scheme, this range does not convey enough information by itself to determine which contentlanguage tag is most appropriate, since it matches everything. If the language range "*" is followed by other language ranges, it SHOULD beis skipped. If the language range "*" is the only one in the language priority list or if no other language range follows, the default content SHOULD bevalue is computed and returned. In some cases, the language priority list might contain one or more extended language ranges (as, for example, when the same language priority list is used as input for both lookup and filtering operations). Wildcard values in an extended language range normally match any value that occurscan occur in that position in a language tag. Since only one item can be returned for any given lookup request, wildcards in a language range have to be processed in a consistent manner or the same request will produce widely varying results. ImplementationsApplications, protocols, or specifications that accept extended language ranges MUST define which contentitem is returned when more than one item matches the extended language range. For example, an implementation could return the matching tag that is first in ASCII-order. If the language range were "*-CH" and the set of tags included "de-CH", "fr-CH", and "it-CH", then the tag "de-CH" would be returned. Another possibility would be for an implementation to mapan implementation to map the extended language ranges to basic ranges. 3.4.1. Default Values Each application, protocol, or specification MUST define the defaulting behavior when no tag matches the language priority list. What this action consists of strongly depends on how lookup is being applied. Some examples of defaulting behavior might include: o return an item with no language tag or an item of a non-linguistic nature, such as an image or sound o return a null string as the language tag value, in cases where the protocol permits the empty value (see, for example, "xml:lang" in [XML10]) o return a particular language tag designated for the operation o return the language tag "i-default" (see: [RFC2277]) o return an error condition or error message o return a list of available languages for the user to select from When performing lookup using a language priority list, the progressive search MUST process each language range in the list before seeking or calculating the default. The default value MAY be calculated and might include additional searching or matching. Applications, protocols, or specifications can specify different ways in which users can specify or override the defaults. One common way to provide for a default is to allow a specific language range to be set as the default for a specific type of request. If this approach is chosen, this language range MUST be treated as if it were appended to the end of the language priority list as a whole, rather than after each item in the language priority list. The application, protocol, or specification MUST also define the defaulting behavior if that search fails to find a matching tag or item. For example, if a particular user's language priority list were "fr-FR, zh-Hant" and the program doing the matching had a default language range of "ja-JP", the program would search as follows: 1. fr-FR 2. fr 3. zh-Hant // next language 4. zh 5. ja-JP // now searching for the extended language ranges to basic ranges.default content 6. ja 7. (implementation defined default) Figure 4: Lookup Using a Language Priority List 4. Other Considerations When working with language ranges and matching schemes, there are some additional points that may influence the choice of either. 4.1. Choosing Language Ranges Users indicate their language preferences via the choice of a language range or the list of language ranges in a language priority list. The type of matching affects what the best choice is for a user. Most matching schemes make no attempt to process the semantic meaning of the subtags and thesubtags. The language range is compared, in a case- insensitive manner, to each language tag being matched, using basic string processing. Users SHOULD select language ranges that are well-formed, valid language tags according to [RFC3066bis] (substituting wildcards as appropriate in extended language ranges). Users SHOULD replaceApplications are encouraged to canonicalize language tags or subtags which have been deprecated withand ranges by using the Preferred-Value from the IANA Language Subtag Registry.Registry for tags or subtags which have been deprecated. If the user is working with content that might use the older form, the user might want to include both the new and old forms in a language priority list. For example, the tag "art-lojban" is deprecated. The subtag 'jbo' is supposed to be used instead, so the user might use it to form the language range. Or the user might include both in a language priority list: "jbo, art-lojban". Users SHOULD avoid subtags that add no distinguishing value to a language range. When filtering,language range. When filtering, the fewer the number of subtags that appear in the language range, the more content the range will probably match, while in lookup unnecessary subtags might cause "better", more-specific content to be skipped in favor of less specific content. For example, the range "de-Latn-DE" would return content tagged "de" instead of content tagged "de-DE", even though the latter is probably a better match. Whether a subtag adds distinguishing value can depend on the context of the request. For example, a user who reads both Simplified and Traditional Chinese, but who prefers Simplified, might use the range "zh" for filtering (matching all items that user can read) but "zh- Hans" for lookup (making sure that user gets the preferred form if it's available, but the fallback to "zh" will still work). On the other hand, content in this case should be labeled as "zh-Hans" (or "zh-Hant" if that applies) for filtering, but for lookup, if there is either "zh-Hans" content or "zh-Hant" content, then one of them (the one considered 'default') should also be available under a simple "zh". Note that the user can create a language priority list "zh- Hans, zh" that delivers the best possible results for both schemes. If the user cannot be sure which scheme is being used (or if more than one might be applied to a given request), the feweruser SHOULD specify the most specific (largest number of subtags that appear in the language range, the more content thesubtags) range will probably match, while in lookup unnecessary subtags might cause "better", more-specific content to be skippedfirst and then supply shorter prefixes later in favor of less specific content. For example, the range "de-Latn-DE" would return content tagged "de" instead of content tagged "de-DE", even thoughthe latter is probablylist to ensure that filtering returns a better match.complete set of tags. Many languages are written predominantly in a single script. This is usually recorded in the Suppress-Script field in that language subtag's registry entry. For these languages, script subtags SHOULD NOT be used to form a language range. Thus the language range "en- Latn" is inappropriate in most cases (because the vast majority of English documents are written in the Latin script and thus the 'en' language subtag has a Suppress-Script field for 'Latn' in the registry). When working with tags and ranges, note that extensions and most private-use subtags are orthogonal to language tag matching, in that they specify additional attributes of the text not related to the goals of most matching schemes. Users SHOULD avoid using these subtags in language ranges, since they interfere with the selection of available content. When used in language tags (as opposed to ranges), these subtags normally do not interfere with filtering (Section 3), since they appear at the end of the tag and will match all prefixes. Lookup (Section 3.3)3.4) implementations oftenare advised to ignore unrecognized private-use and extension subtags when performing language tag fallback. Applications, specifications, or protocols that choose not to interpret one or more private-use or extension subtags SHOULD NOT remove or modify these extensions in content that they are processing. When a language tag instance is to be used in a specific, known protocol, and is not being passed through to other protocols, language tags MAY be altered to remove subtags and extensions that are not supported by that protocol. Such alterations SHOULD be avoided, if possible, since they remove information that might be relevant elsewhere that would make use of that information. Some applications of language tags might want or need to consider extensions and private-use subtags when matching tags. If extensions and private-use subtags are included in a matching process that utilizes one of the schemes described in this document, then the implementation SHOULD canonicalize the language tags and/or ranges before performing the matching. Note that language tag processors that claim to be "well-formed" processors as defined in [RFC3066bis] generally fall into this category.4.2. Meaning of Language Tags and Ranges Selecting contentlanguage tags using language ranges requires some understanding by users of what they are selecting. The meaning of the various subtags in a language range are identical to their meaning in a language tag (see Section 4.2 in [RFC3066bis]), with the addition that the wildcard "*" represents any matching sequence of values. 4.3. Considerations for Private Use Subtags Private-use subtags require private agreement between the parties that intend to use or exchange language tags that use them and great cautionthem. They SHOULD NOT be used in employing them incontent or protocols intended for general use. Private-use subtags are simply useless for information exchange without prior arrangement. The value and semantic meaning of private-use tags and of the subtags used within such a language tag are not defined. Matching private- use tags using language ranges or extended language ranges can result in unpredictable content being returned. 4.4. Length Considerations for Language Ranges Language ranges are very similar to language tags in terms of content and usage. The same types of restrictions on length that apply to language tags can also apply to language ranges. See [RFC3066bis] Section 4.3 (Length Considerations). 5. IANA Considerations This document presents no new or existing considerations for IANA. 6. Changes This is the first version of this document. 7.Security Considerations Language ranges used in content negotiation might be used to infer the nationality of the sender, and thus identify potential targets for surveillance. In addition, unique or highly unusual language ranges or combinations of language ranges might be used to track a specific individual's activities. This is a special case of the general problem that anything you send is visible to the receiving party. It is useful to be aware that such concerns can exist in some cases. The evaluation of the exact magnitude of the threat, and any possible countermeasures, is left to each application or protocol. 8.7. Character Set Considerations Language tags permit only the characters A-Z, a-z, 0-9, and HYPHEN- MINUS (%x2D). Language ranges also use the character ASTERISK (%x2A). These characters are present in most character sets, so presentation or exchange of language tags or ranges should not be constrained by character set issues. 9.8. References 22.214.171.124. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. [RFC2277] Alvestrand, H., "IETF Policy on Character Sets and Languages", BCP 18, RFC 2277, January 1998. [RFC3066bis] Phillips, A., Ed. and M. Davis, Ed., "Tags for the Identification of Languages", October 2005, <http:// www.ietf.org/internet-drafts/ draft-ietf-ltru-registry-14.txt>. [RFC4234] Crocker, D. and P. Overell, "Augmented BNF for Syntax Specifications: ABNF", RFC 4234, October 2005. 126.96.36.199. Informative References [RFC1766] Alvestrand, H., "Tags for the Identification of Languages", RFC 1766, March 1995. [RFC2616] Fielding, R., Gettys, J., Mogul, J., Frystyk, H., Masinter, L., Leach, P., and T. Berners-Lee, "Hypertext Transfer Protocol -- HTTP/1.1", RFC 2616, June 1999. [RFC2616errata] IETF, "HTTP/1.1 Specification Errata", 10October 2004, <http://purl.org/NET/http-errata>. [RFC3066] Alvestrand, H., "Tags for the Identification of Languages", BCP 47, RFC 3066, January 2001. [RFC3282] Alvestrand, H., "Content Language Headers", RFC 3282, May 2002. [XML10] Bray (et al), T., "Extensible Markup Language (XML) 1.0", 02February 2004. Appendix A. Acknowledgements Any list of contributors is bound to be incomplete; please regard the following as only a selection from the group of people who have contributed to make this document what it is today. The contributors to [RFC3066bis], [RFC3066] and [RFC1766], each of which is a precursor to this document, made enormous contributions directly or indirectly to this document and are generally responsible for the success of language tags. The following people (in alphabetical order by family name) contributed to this document: Harald Alvestrand, Stephane Bortzmeyer, Jeremy Carroll, John Cowan, Martin Duerst, Frank Ellermann, Doug Ewell, Debbie Garside, Marion Gunn, Kent Karlsson, Ira McDonald, M. Patton, Randy Presuhn, Eric van der Poel, Markus Scherer, and many, many others. Very special thanks must go to Harald Tveit Alvestrand, who originated RFCs 1766 and 3066, and without whom this document would not have been possible. Authors' Addresses Addison Phillips (editor) Yahoo! IncInc. Email: addison at inter dash locale dot firstname.lastname@example.org Mark Davis (editor) Google Email: mark dot davis at macchiato dot email@example.com Intellectual Property Statement The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79. Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at firstname.lastname@example.org. Disclaimer of Validity This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Copyright Statement Copyright (C) The Internet Society (2006). This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights. Acknowledgment Funding for the RFC Editor function is currently provided by the Internet Society.