Thai-English Bilingual Sorting

ASCII Sort vs. Dictionary Sequence

Even for English, the simple sequential comparison of ANSI C strcmp() based on ASCII code does not follow traditional dictionary sequence, because the punctuation marks and cases are not treated properly. For example,

ISO/IEC 14651 Sorting Model

In ISO/IEC 14651 International String Ordering, a sorting method for international strings based on ISO/IEC 10646 is provided. The method tries to achieve the dictionary sequence type of sorting, covering multilingual texts.

As an example, for Latin scripts, input string will be decomposed into four levels of comparisons:

Level 1

Input string are rendered case insensitive and diacritical mark insensitive. All special characters are removed.
Ex.

• résumé! -> resume
• August -> august

Level 2

Diacritical marks are extracted out of the input string before being compared.
Ex.

• résumé! -> resumea***a*
• August -> august******

where a is a symbolic value representing acute accent. Other values are g (grave accent), c (circumflex), t (tilde), u (umlaut) and r (ring). [Note that diacritic properties are encoded in reverse order to follow traditional French dictionaries.]

Level 3

Case information is extracted to be compared.
Ex.

• résumé! -> resumea***a*llllll
• August -> august******ulllll

where u and l are symbolic values representing upper case and lower case properties, respectively.

Level 4

Special characters are then considered.
Ex.

• résumé! -> resumea***a*lllllld7!
• August -> august******ulllll

where d is a delimitor. The number preceeding the special character represents the position of the character in the string.

An Extension for Thai Script

Thai string comparisons can be rendered in three levels.

Level 1

Only consonants and vowels are considered. Every leading vowel is swapped with the immediate consequent consonant.
Ex.

• แว่น-แก้ว -> วแนกแว
• แว่นแก้ว- -> วแนกแว

Level 2

Tonal and diacritical marks are then compared.
Ex.

• แว่น-แก้ว -> วแนกแว**อ***ท*
• แว่นแก้ว- -> วแนกแว**อ***ท*

where อ and ท are symbolic values representing Mai Ek and Mai Tho, respectively. Other values are พ (Phinthu), ย (Yamakkan), ฆ (Thantakhat), ค (Mai Tai Khu), ต (Mai Tri), and จ (Mai Chattawa).

Level 3

Punctuation marks are then scored.
Ex.

• แว่น-แก้ว -> วแนกแว**อ***ท*d5-
• แว่นแก้ว- -> วแนกแว**อ***ท*d1-

where d is a delimitor. The number preceeding the special character represents the position of the character in the string. [Note: Here we make a distiction from that of ISO/IEC 14651 by counting back from the string tail (in other words, by keeping the complement of the position instead of the position itself), to conform to the sequences in the Random House and Webster Dictionary.]

These levels correspond to level 1, 2 and 4 of ISO/IEC 14651 model, respectively, except that we do not encode Thai tonal marks in reverse order like when encoding Latin diacritical marks. Therefore, we can add Thai script to ISO/IEC 14651 Common Template. And one of us has created an example of such extension.

Thai Sorting Issues

Paiyan Noi and Mai Yamok

Paiyan Noi (ฯ) is an abbreviation sign, representing omitted text. Its function is like elipsis, although the more correspondent Thai punctuation mark for elipsis is Paiyan Yai (ฯลฯ). Therefore, it makes sense to order it after elipsis category in the Common Template, after Devanagari abbreviation sign <U0970>.

Mai Yamok (ๆ) is a duplication sign, representing a repeat of the preceeding word. There's no appropriate category for it in the Common Templat. So, it deserves a new category within the general punctuation portion.

Nikhahit

According to traditional Thai principle, Nikhahit (-ํ) is a name of a vowel symbol. It, with Pin I (-ิ) compose Sara Ue (-ึ), with Lakkhang (า) compose Sara Am (-ํา).

However, in Thai document, such function is ineffective, because TIS-620 has defined codes for such combined vowels. Instead, Nikhahit is normally used as Pali-Sanskrit-inherited consonant. For example, following Pali, พุทธํ is pronounced "Bhut-Dhang"; and following Sanskrit, ชุํนุํ is pronounced "Chum-Num". Therefore, Nikhahit arguably deserves to be classified as a consonant, ordered after the last Thai consonant (ฮ).

Yamakkan, Pinthu, Thanthakhat, Mai Taikhu and Tonal Marks

Yamakkan (-๎) is an ancient punctuation mark used to mark cluster, such as พ๎ราห๎มณ. Pinthu (-ฺ) has two functions, one is the same as Yamakkan, such as a Sanskrit word พฺราหฺมณ, the other is to mark final consonant in Pali writing system, such as วิชฺชา.

Thanthakhat (-์) is used for killing letter's sounds, such as พราหมณ์, in which the consonant ณ is killed (not pronounced), ลักษมณ์, in which the three consonants ษมณ are killed, ปาฏิหาริย์, in which the two letters -ิย are killed.

The three marks can be considered as having less effect in changing words than tonal marks. For example, the meaning of พราหมณ and พราหมณ์ are not different, while บน and บ่น are. Therefore, they deserved to be ordered before tonal marks.

Pinthu may come between Yamakkan and Thanthakhat, because another function of Thanthakhat is to mark final consonant in ancient writing of Pali words. In summary, the order should be Yamakkan, Pinthu, Thanthakhat, Mai Taikhu, Mai Ek, Mai Tho, Mai Tri, and Mai Chattawa.

Lakkhang Yao and Sara Aa

Lakkhang Yao (ๅ) is always written after Ru (ฤ) or Lu (ฦ), to produce new symbols Rue (ฤๅ) or Lue (ฦๅ), respectively. Traditional Thai principle treats each of these two new symbols as an atomic entity. Therefore, in general, Lakkhang Yao is never written after any letters other than Ru (ฤ) and Lu (ฦ). Meanwhile, the two symbols are never followed by a vowel. Hence, the occurrence of Lakkhang Yao (ๅ) and Sara Aa (า --a vowel) are mutually exclusive, and assigning order between them seems not necessary. However, due to their similar shapes, they are occasionally confused (in a reasonable way). So, they should be treated identical unless all other parts are equal, that is, their weights should be different at the last level of comparison.

Fongman, Angkhankhu, and Khomut

Fongman (๏) is used in ancient books as paragraph, sentence, or poem stanza begin marker. Angkhandeaw (ฯ) is used to end a sentence or a stanza. Angkhankhu (๚) is used to end a chapter or episode. Khomut (๛) ends a story.

These marks can be classified as typographic symbols in the Common Template. Unfortunately, there is no specific code for Angkhandeaw (ฯ) and Paiyan Noi is always used instead.

Sorting for TIS-620

Within TIS-620 character set, sorting is less complicated than that on ISO/IEC 10646. We can assign the four levels of weights to each class of character in this manner:

After this model, Theppitak Karoonboonyanan has written a C++ library named ThColl, which is downloadable.

References

[1]

Trin Tantsetthi, Thai Locale Documentation.

[2]

Theppitak Karoonboonyanan, Thai Sorting Algorithms.

[3]

Pruet Boonma, Thai sorting support for free database server.

[4]

The Royal Institute, The Principle of Punctuation Marks and Other Symbols, The Principle of Spacing, and The Principle of Abbreviation Writing, (5th Publishing), Bangkok, 1990.

[5]

Thai Developer Network Mail Archive.

Created: 1998-11-07 Last Updated: 2003-03-03 Theppitak Karoonboonyanan Samphan Raruenrom Pruet Boonma Back to Theppitak's Homepage