thdltools/www
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
www/htdocs/ACIP_To_Tibetan_Converter.html

1537 lines
54 KiB
HTML
Raw Blame History

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<!-- @author David Chandler -->
<!-- @editor Emacs, baby! -->
<head>
<title>ACIP To Tibetan Converters</title>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<script type="text/javascript" src="http://iris.lib.virginia.edu/tibet/scripts/thdl_scripts.js"></script>
<link rel="stylesheet" type="text/css" href="http://iris.lib.virginia.edu/tibet/style/thdl-styles.css"/>
</head>
<body>
<div id="banner">
<a id="logo" href="http://iris.lib.virginia.edu/tibet/index.html"><img id="test" alt="THDL Logo" src="http://iris.lib.virginia.edu/tibet/images/logo.png"/></a>
<h1>The Tibetan &amp; Himalayan Digital Library</h1>
<div id="menubar">
<script type='text/javascript'>function Go(){return}</script>
<script type='text/javascript' src='http://iris.lib.virginia.edu/tibet/scripts/new/thdl_menu_config.js'></script>
<script type='text/javascript' src='http://iris.lib.virginia.edu/tibet/scripts/new/menu_new.js'></script>
<script type='text/javascript' src='http://iris.lib.virginia.edu/tibet/scripts/new/menu9_com.js'></script>
<noscript><p>Your browser does not support javascript.</p></noscript>
<div id='MenuPos' >Menu Loading... </div>
</div><!--END menubar-->
</div><!--END banner-->
<div id="sub_banner">
<div id="search">
<form method="get" action="http://www.google.com/u/thdl">
<p>
<input type="text" name="q" id="q" size="15" maxlength="255" value="" />
<input type="submit" name="sa" id="sa" value="Search"/>
<input type="hidden" name="hq" id="hq" value="inurl:iris.lib.virginia.edu"/>
</p>
</form>
</div>
<div id="breadcrumbs">
<a href="http://iris.lib.virginia.edu/tibet/index.html">Home</a> &gt; <a href="index.html">Tools</a> &gt; <a href="http://iris.lib.virginia.edu/tibet/tools/allfonts.html">Fonts &amp; Input</a> &gt; <a href="http://iris.lib.virginia.edu/tibet/tools/conv.html">Converters</a> &gt; <a href="TMW_RTF_TO_THDL_WYLIE.html">Converters in Jskad</a> &gt; ACIP To Tibetan Converters
</div>
</div><!--END banner-->
<div id="main">
<h2>ACIP To Tibetan Converters</h2>
<p>
This document describes the ACIP-&gt;Tibetan converters built atop
<a
href="http://iris.lib.virginia.edu/tibet/tools/jskad.html">Jskad</a>.&nbsp;
These converters were initially written by David Chandler, a
volunteer with the <a
href="http://iris.lib.virginia.edu/tibet/index.html">Tibetan and
Himalayan Digital Library</a>, in the latter half of 2003.&nbsp;
They built upon the work of Tony Duff, Edward Garrett, and Than
Garson, and they would not be possible without the assistance of
David Chapman, Robert Chilton, and Andr&eacute;s Montano
Pellegrini.&nbsp; (Please correct, and forgive, any omissions from
these lists.)
</p>
<p>
These converters accept <a href="http://asianclassics.org">Asian
Classics Input Project</a> (ACIP) transliteration of Tibetan (using
ACIP's <a
href="http://asianclassics.org/download/tibetancode/ticode.pdf">Tibetan
Input Code</a>), a Roman transliteration scheme.&nbsp; ACIP has many
Buddhist texts available in ACIP transliteration, which alone makes
ACIP transliteration (or just ACIP for short) important.
</p>
<p>
The converters here accept a text file of ACIP and output either a
Unicode UTF-8-encoded text file or a Rich Text Format (RTF) file of
<a href="http://iris.lib.virginia.edu/tibet/tools/tmw.html">Tibetan
Machine Web</a> (TMW).&nbsp; The latter is ready to use onscreen and
to make beautiful hardcopy today; the former will be understood by
software for a long time to come.
</p>
<p>
The converters are meant to produce perfect results even for
imperfect input.&nbsp; To give you an idea of the thought and care
that went into these converters, consider the following partial list
of features:
</p>
<ul>
<li>
Four tiers of <a href="#diagnostics">warning and error
messages</a> are available.
</li>
<li>
Some transliterations specified by the ACIP standard are not
accepted (i.e., they cause <a href="#diagnostics">errors</a>)
because they are used too often improperly in Release IV texts
(e.g., {\}); some non-standard transliteration is understood
because it is used in ACIP Release IV texts (e.g., {[DD1]}).
</li>
<li>
Non-standard <a href="#escapes">Unicode character escapes</a> are
supported.&nbsp; (In this way, the glyph that the ACIP {\} refers
to according to the standard can in fact be represented, via
{\u0F84}.)
</li>
<li>
<a href="#colors">Color-coding</a> can help find typos in the
input.
</li>
<li>
A <a href="#sub">substitution</a> mechanism allows for correcting
erroneous documents on the fly.
</li>
<li>
The converters can output frequency <a
href="#stats">statistics</a>.
</li>
<li>
The <a href="#lex">&quot;lexical analyzer&quot;</a> and <a
href="#parse">&quot;parser&quot;</a> handle every intricacy of
real ACIP Release IV texts.
</li>
<li>
The knowledge regarding the TMW font has been verified by
independent teams as described <a
href="TMW_or_TM_To_X_Converters.html#vv">here</a>.
</li>
</ul>
<p>
The ACIP-&gt;Unicode and ACIP-&gt;TMW converters are equally
good.&nbsp; There are some differences between the two,
though.&nbsp; The TMW font has only a fixed set of glyphs, whereas
Unicode can encode arbitrary Tibetan glyphs.&nbsp; Thus, the
hypothetical ACIP {GAI}, which parses as {G+AI} due to <a
href="#prefix">prefix rules</a>, will give an error in an
ACIP-&gt;TMW conversion because no glyph exists for this
stack.&nbsp; The ACIP-&gt;Unicode conversion will succeed, having
generated correct Unicode.&nbsp; This is the only difference between
the two conversions.
</p>
<p>
The converters are actively maintained; your <a
href="mailto:thdltools-devel@lists.sourceforge.net">feedback</a> is
valued.
</p>
<p>
Note that there are also <a
href="TMW_or_TM_To_X_Converters.html">TMW-&gt;ACIP</a> converters
available; this document does not cover them.
</p>
<p>
In what follows, you will learn <a href="#using">how to use</a> the
converters, including all the features listed above, and you'll find
a list of <a href="#bugs">known bugs</a> and places where there is
<a href="#room">room for improvement</a>.
</p>
<a name="using"></a><h2>Using the Converters</h2>
<p>
This section briefly describes how the converters are best used.
</p>
<p>
The GUI and command-line interfaces are both sufficient; the GUI
interface is your best bet if you've not used the converters
before.&nbsp; To learn how to invoke these interfaces, read <a
href="TMW_RTF_TO_THDL_WYLIE.html#invok">these instructions</a>.
</p>
<p>
First, review the <a href="#bugs">known bugs</a> and be sure you can
live with them.
</p>
<p>
Now perform a trial conversion of your document with <a
href="#diagnostics">warnings</a> disabled.&nbsp; You will first
ensure that no outright <a href="#diagnostics">errors</a> appear in
the input.&nbsp; If any do, make a copy of the input, edit the
input, and feed it through again.&nbsp; Feel free to try this out as
soon as you're comfortable; the error messages themselves are
sometimes self-explanatory.
</p>
<p>
Once all errors have been corrected, do a conversion with warning
level 'Some'.&nbsp; If any warnings mark real problems, correct
those problems.
</p>
<p>
If you have the patience, now do a conversion with warning level
'Most' and correct further problems.&nbsp; If any warnings mark real
problems, correct those problems.
</p>
<p>
The 'All' warning level is pedantic; you might find it useful if
you're writing software that is to produce ACIP transliteration that
is easily read by machines.&nbsp; If you find any useful warnings at
this level, report it as a bug -- such warnings should be 'Most' or
'Some' level.
</p>
<p>
For best results, produce <a href="#colors">color-coded
output</a>.&nbsp; Scan the output for non-<a
href="#native">native</a> <i>tsheg bar</i>s and ensure that they
match the original document (the one from which the ACIP
transliteration was produced).&nbsp; Color-coding is useful because,
for example, {ZHIGN} is probably a typo for {ZHING}; {ZHIGN} will
appear colored, whereas {ZHING} is not colored.
</p>
<p>
Note that the ACIP {%} gives a warning every time.&nbsp; Use the <a
href="#escapes">Unicode escape</a> {\u0F35} if you want to avoid
this warning, but <i>note well</i> that Unicode escapes are not part
of the ACIP standard.&nbsp; Thus, other tools that work with ACIP
transliteration will likely not understand {\u0F35}.
</p>
<p>
To save time, you may use the <a href="#sub"><i>tsheg-bar</i>
substitution</a> mechanism when appropriate.
</p>
<p>
Even if your desired end result is Unicode output, an ACIP-&gt;TMW
conversion is sometimes useful.&nbsp; One benefit is that errors
will appear for any ACIP <i>tsheg bar</i> that refers to a consonant
stack not included in TMW.&nbsp; These stacks should be scrutinized,
because TMW contains over 500 of the most common consonant stacks.
</p>
<p>
Finally, check a few folios by hand against the original document to
be sure that you're satisfied with the conversion.
</p>
<a name="diagnostics"></a><h2>Diagnostics: Warnings and Errors</h2>
<p>
These converters are designed such that the output is just what you
yourself would create by hand.&nbsp; Whenever there is doubt about
what output is desired, a warning or error is issued.&nbsp; This
means that a helpful warning or error message will appear in the
output, and that you will be told at the end of the conversion that
one or more warnings or errors have indeed occurred.&nbsp; You can
then search your output document for the text <tt>[#ERROR</tt> or
<tt>[#WARNING</tt>.
</p>
<p>
There are four warning levels: 'None', 'Some', 'Most', and
'All'.&nbsp; Choose 'None' if you don't want any warnings to appear
in your output and be brought to your attention at the end of
conversion.&nbsp; Choose 'Some' if you want to see the most
important warnings, 'Most' if you want some real confidence in your
output, and 'All' if you've absolutely got to know that the output
is right.
</p>
<p>
Errors will always appear; you cannot disable them.
</p>
<p>
The following are some (but not all) error and warning messages,
accompanied by further explication:
</p>
<ul>
<li>
<tt>[#ERROR CONVERTING ACIP DOCUMENT: The Unicode escape with
ordinal 3912 does not match up with any TibetanMachineWeb
glyph.]</tt> appears for the input {\u0F48} because there is no
character at the Unicode codepoint U+0F48 (decimal 3912).
</li>
<li>
<tt>[#ERROR The ACIP {G+N+NA} cannot be represented with the
TibetanMachine or TibetanMachineWeb fonts because no such glyph
exists in these fonts.]</tt> appears because the Tibetan Machine
Web font has only a limited number of ready-made, precomposed
glyphs, and {G+N+NA} is not one of them.&nbsp; You'll only see
this error in an ACIP-&gt;TMW conversion, not an ACIP-&gt;Unicode
conversion.
</li>
<li>
<tt>[#ERROR CONVERTING ACIP DOCUMENT: This converter cannot
convert the ACIP {x} to Tibetan because it is unclear what the
result should be.]</tt> appears because the appropriate output for
this likely requires special mark-up.
</li>
<li>
<tt>[#ERROR CONVERTING ACIP DOCUMENT: Lexical error: The ACIP {^}
must precede a tsheg bar.]</tt> appears for
{^&nbsp;&nbsp;GONG&nbsp;SA}, for example, because only
{^GONG&nbsp;SA} and {^&nbsp;GONG&nbsp;SA} are supported in this
implementation.
</li>
<li>
<tt>[#ERROR CONVERTING ACIP DOCUMENT: The tsheg bar ("syllable") :
has these errors: Cannot convert ACIP A: because A: is a "vowel"
without an associated consonant]</tt> appears for the input {:}
because {:} cannot appear alone.&nbsp; (Sloppily, this message
exposes you to the internals of the converter, where {:} is
thought of as {A:} in some contexts.)
</li>
<li>
<tt>[#ERROR CONVERTING ACIP DOCUMENT: Lexical error: The ACIP x
must be glued to the end of a tsheg bar, but this one was
not]</tt> appears because {%}, {o}, and {x} are really only to be
applied to whole <i>tsheg bar</i>s, and should not occur alone.
</li>
<li>
<tt>[#WARNING CONVERTING ACIP DOCUMENT: The ACIP DGYA has been
interpreted as two stacks, not one, but you may wish to confirm
that the original text had two stacks as it would be an easy
mistake to make to see one stack and forget to input it with '+'
characters.]</tt> appears because it helps evince the impact of <a
href="#prefix">prefix rules</a>, a subtle point with regards to
ACIP because they are implied, but not discussed explicitly in
depth, by the ACIP standard.
</li>
<li>
<tt>[#WARNING CONVERTING ACIP DOCUMENT: Warning: We're going with
{B+NA}, but only because our knowledge of prefix rules says that
{B}{NA} is not a legal Tibetan tsheg bar ("syllable")]</tt>
appears for the same reason as above.
</li>
<li>
<tt>[#WARNING CONVERTING ACIP DOCUMENT: Lexical warning: The ACIP
{%} is treated by this converter as U+0F35, but sometimes might
represent U+0F14 in practice. To avoid seeing this warning again,
change the input to use {\u0F35} instead of {%}.]</tt> appears
because some ACIP transliteration out there does use {%} to mean
U+0F14.
</li>
</ul>
<p>
When warning or error messages refer to a 'Lexical error', that is
an error that occurs when <a href="#lex">breaking an input text up
into <i>tsheg bar</i>s</a>.&nbsp; To fully understand all warning
and error messages, a thorough understanding of <a href="#lex">that
process</a> and of the <a href="#parse">interpretation of ACIP
<i>tsheg bar</i>s</a> is required.
</p>
<a name="colors"></a><h2>Coloration</h2>
<p>
For ACIP-&gt;TMW conversions (not ACIP-&gt;Unicode), color-coding of
<i>tsheg bar</i>s is an option.&nbsp; The command-line converters
accept a flag <tt>--colors&nbsp;yes|no</tt>; the conversion GUI in
Jskad has a checkbox for color-coding.
</p>
<p>
Warnings and errors appear in <font color="red">red</font>; <i>tsheg
bar</i>s that would parse differently if other <a
href="#prefix">prefix rules</a> were used appear in <font
color="yellow">yellow</font>; non-<a href="#native">native</a>
<i>tsheg bar</i>s appear in <font color="green">green</font>.
</p>
<a name="stats"></a><h2><i>Tsheg-bar</i> Statistics</h2>
<p>
The ACIP-&gt;Tibetan converters provide a simple-minded accounting
mechanism with which one can determine which <i>tsheg bar</i>s
appear in a conversion or how many times each <i>tsheg bar</i>
appears.&nbsp; This mechanism is for power users only at this point;
its user interface leaves much to be desired.&nbsp; If you wish to
produce frequency information, and if you are not familiar with some
sort of scripting (via Excel macros, Unix shell scripts, etc.), then
the output produced will likely be useless to you.
</p>
<p>
To support the calculation of frequency statistics, that is, how
many times each <i>tsheg bar</i> appears, the converter can output
all <i>tsheg bar</i>s to the Java error console (i.e.,
<tt>System.err</tt>).&nbsp; Each will appear on the console as many
times as it appears in the input.&nbsp; To activate this
functionality, <a href="#sysprops">set the system property</a>
<tt>org.thdl.tib.text.ttt.OutputAllTshegBars</tt> to <tt>true</tt>,
and be prepared for voluminous output.&nbsp; Massaging this output
into a friendly tabular format is quite possible but not described
here; contact <a
href="mailto:thdltools-devel@lists.sourceforge.net">the
developers</a> for help.
</p>
<p>
To support the generation of syllabaries, the converter can output
each <i>tsheg bar</i> encountered to the Java error console (i.e.,
<tt>System.err</tt>).&nbsp; Each will appear on the console only
once, no matter how many times it appears in the input.&nbsp; To
activate this functionality, <a href="#sysprops">set the system
property</a> <tt>org.thdl.tib.text.ttt.OutputUniqueTshegBars</tt> to
<tt>true</tt>, and be prepared for voluminous output.
</p>
<p>
If desired, each <i>tsheg bar</i> output can be prefixed with a
string of your choice by <a href="#sysprops">setting the system
property</a> <tt>org.thdl.tib.text.ttt.PrefixForOutputTshegBars</tt>
to that string.&nbsp; This is useful if the converter is producing
other output on the console and you want to separate that output
from the statistics.
</p>
<!-- DLC LINK TO THE EXCEL SPREADSHEET OF STATS -->
<a name="sub"></a><h2><i>Tsheg-bar</i> Substitution</h2>
<!-- NOTE WELL: The text here is largely the same as the text in the
class comment for org.thdl.tib.text.ttt.MidLexSubstitution. -->
<p>
The ACIP-&gt;Tibetan converters provide a mechanism for
automatically correcting common transliteration typos.&nbsp; For
example, if your document contains 100 occurrences of {KAsh} that
all in fact intend {K+sh}, then you can specify just once the rule
<tt>{KAsh}-&gt;{K+sh}</tt>, and all 100 occurrences will be treated
correctly.&nbsp; This mechanism is not very easy to use, but it is
completely customizable; you can specify any number of rules.&nbsp;
You can only perform such substitutions at the <i>tsheg bar</i>
level, though.&nbsp; This means, for example, that you cannot
specify the rule <tt>{GONG SA}-&gt;{^GONG SA}</tt>; you can only
specify <tt>{GONG}-&gt;{^GONG}</tt>, which would affect {GONG LA}
just as it would affect {GONG SA}.
</p>
<p>
To perform substitutions, <a href="#sysprops">set the system
property</a> <tt>org.thdl.tib.text.ttt.ReplacementMap</tt> to be a
comma-delimited list of <tt>x=&gt;y</tt> pairs.&nbsp; For example,
if you think BLKU, which parses as B+L+KU, should parse as B-L+KU,
and you want KAsh to be parsed as K+sh because the input operators
mistyped it, then set <tt>org.thdl.tib.text.ttt.ReplacementMap</tt>
to <tt>BLKU=&gt;B-L+KU,KAsh=&gt;K+sh</tt>.&nbsp; Note that this will
not cause {B+L+KU} to become {B-L+KU} -- we are doing the
replacement during lexical analysis of the input file, not during
parsing.&nbsp; And it will cause {SBLKU} to become {SB-L+KU}, which
is parsed as {S+B-L+KU}, probably not what you wanted.&nbsp; If you
fear such things, you can see if they happen by setting the system
property <tt>org.thdl.tib.text.ttt.VerboseReplacementMap</tt> to
<tt>true</tt>, which will cause an informational message to be
printed on the Java console every time a replacement is made.
</p>
<p>
Furthermore, you can use the regular expression notations <tt>^</tt>
and <tt>$</tt> to denote the beginning and end of the <i>tsheg
bar</i>, respectively.&nbsp; For example, <tt>^BLKU$=&gt;B-L+KU</tt>
is a useful rule.&nbsp; Note that full regular expressions are not
supported -- the tool just borrows a bit of the notation.&nbsp; The
rule <tt>^BLKU=&gt;B-L+KU</tt> means that {BLKUM} and {BLKU} will
both be replaced, but {SBLKU} and {SBLKUM} will not be.&nbsp; The
caret, <tt>^</tt>, means that we only match if BLKU is at the
beginning.&nbsp; The dollar sign, <tt>$</tt>, means that we only
match if the pattern is at the end.&nbsp; The rule
<tt>BLKU$=&gt;B-L+KU</tt> will cause {SBLKU} to be replaced, but not
{BLKUM}.&nbsp; Note that performance is far better for
<tt>^FOO$</tt> than for <tt>^FOO</tt>, <tt>FOO$</tt>, or
<tt>FOO</tt> alone.
</p>
<p>
Only one substitution is made per <i>tsheg bar</i>.&nbsp;
<tt>^FOO$</tt>-style mappings will be tried first, then
<tt>^FOO</tt>-style, then <tt>FOO$</tt>-style, and finally
<tt>FOO</tt>-style.
</p>
<p>
An example of a useful substitution is <tt>o$=&gt;\u0F35</tt>.&nbsp;
This is useful because the converters interpret the ACIP {o} as
U+0F37 by default, but you might prefer U+0F35 in your output.
</p>
<p>
Note that you cannot literally replace {FOO} with {BAR} using this
mechanism -- because {F} is not an ACIP character, the lex will not
get far enough to use this substitution mechanism.&nbsp; This is not
considered a design flaw -- serious errors require user
intervention.&nbsp; Sophisticated users can use something akin to
perl, sed, or awk scripts to preprocess the input.
</p>
<p>
Note also that you cannot use the rule <tt>ONYA=&gt;O&amp;</tt>,
although it would be nice if you could.&nbsp; Technically, {&amp;}
is considered to be punctuation (i.e., that which divides <i>tsheg
bar</i>s) and is not understood inside a <i>tsheg bar</i>.
</p>
<p>
Note that this mechanism is also useful for fixing problems in the
converter itself rather than in the input.
</p>
<a name="escapes"></a><h2>Unicode Character Escapes</h2>
<p>
The ACIP-&gt;Tibetan converters support some non-standard extensions
to the <a
href="http://asianclassics.org/download/tibetancode/ticode.pdf">ACIP
Tibetan Input Code Standard</a>.&nbsp; One of those is Unicode
character escape sequences.&nbsp; This extension makes it possible
to represent characters that the <a
href="http://asianclassics.org/download/tibetancode/ticode.pdf">ACIP
standard</a> does not address, and to represent one character,
U+0F84, that ACIP does address with the transliteration {\} but that
is misused in practice so often to refer to U+0F3C that the
ACIP-&gt;Tibetan converters always produce an error upon seeing {\}.
</p>
<p>
Outside of comments, {\uKLMN} is interpreted as referring to the
Unicode character with ordinal <i>KLMN</i>, where each of K, L, M,
and N are case-insensitive hexadecimal digits.&nbsp; For example,
the ACIP {KA KHA GA NGA&nbsp;} is exactly equivalent to
{\u0F40\u0f0B\u0F41\u0F0B\u0F42\u0F0B\u0F44\u0f0b}.&nbsp; Unicode
escapes produce the obvious Unicode in an ACIP-&gt;Unicode
conversion, and they produce the correct TMW glyph in an
ACIP-&gt;TMW conversion.&nbsp; There are limits, though, when
converting to TMW; multiple escapes in sequence are not handled
correctly.&nbsp; It would take a Unicode to TMW converter to produce
the correct glyphs for {\u0F42\u0F92\u0FB7\u0F7C}.&nbsp; The escapes
for vowels and other characters that are mapped to multiple TMW
glyphs are also not handled perfectly.&nbsp; Best practice is to use
escapes only when necessary in an ACIP-&gt;TMW conversion.
</p>
<p>
The Unicode character represented need not be a Tibetan one; for
example, {\u0040} produces the at sign, <tt>@</tt>.
</p>
<p>
The latest <a
href="http://iris.lib.virginia.edu/tibet/collections/langling/ewts/">Extended
Wylie Transliteration Scheme</a> standard has assigned private-use
area (PUA) Unicode codepoints to some TMW glyphs.&nbsp; ACIP
documents that have a <a href="#escapes">Unicode escape</a> in the
range U+F021 to U+F0FF, inclusive, are interpreted as intending
these TMW glyphs.&nbsp; ACIP-&gt;Unicode produces an error for such
an escape because it is font-dependent and not standard.&nbsp; Other
tools will likely not understand such Unicode, so the converter will
not produce it.&nbsp; If you want it in the output, it is there in
the error message.
</p>
<p>
Note well the <a href="#bugs">known bug</a> with regard to
whitespace in transliteration that follows a Unicode escape.&nbsp;
In large part, this bug affects characters that can be
transliterated by other, simpler, standard means.
</p>
<p>
If you do want to disable the use of Unicode escapes, <a
href="#sysprops">set the system property</a>
<tt>thdl.tib.text.disallow.unicode.character.escapes.in.acip</tt> to
<tt>true</tt>.
</p>
<a name="lex"></a><h2>Breaking a Text Up Into <i>tsheg bar</i>s</h2>
<p>
The ACIP-&gt;Tibetan converters all take ACIP transliteration as
input.&nbsp; The first step in conversion is to break up the input
into manageable pieces.&nbsp; (This is known as <i>lexical
analysis</i> in the context of programming languages, and you may
see the term in diagnostic messages though a linguist who studies
human language like Tibetan might balk at the term.)&nbsp; The
correct pieces in this case are <i>tsheg bar</i>s (in ACIP, {TSEG
BAR}), punctuation, comments, whitespace, folio markers, formatting
codes, etc.&nbsp; In this section, the intracacies of how the
converter does that will be laid bare.&nbsp; With luck, this will
help you understand why the converter treated one space character
(i.e, '&nbsp;', U+0020) as a <i>tsheg</i> and another as Tibetan
whitespace.
</p>
<p>
The Tibetan term <i>tsheg bar</i> refers to &quot;the stuff between
the dots&quot;.&nbsp; In the ACIP {BKRA SHIS&nbsp;[# Notice that
this comment is embedded in the Tibetan greeting pronounced 'tashi
delay']BDE&nbsp;LEGS,}, there are four <i>tsheg bar</i>s, 'BKRA',
'SHIS', 'BDE', and 'LEGS'.&nbsp; In this case 'BDE' is literally
&quot;between the dots&quot;; i.e., it is sandwiched by two U+0F0B
characters (because comments are in a sense invisible).&nbsp; One of
the &quot;dots&quot; that touches 'LEGS' does not look like a dot --
it is a <i>shad</i>, U+0F0D.&nbsp; The lexical analyzer also finds
one comment, which will appear in a Latin typeface in the output,
and it finds four pieces of punctuation -- three <i>tsheg</i>s and a
<i>shad</i>.
</p>
<p>
The converter will not allow an illegal character into a <i>tsheg
bar</i>.&nbsp; For example, {jA} is an error and causes an error
message to appear in the output.
</p>
<p>
Now that the basic operation is clear from the above example, let's
cover the fine points of how standard ACIP is handled.&nbsp; We'll
also cover some non-standard constructs that appear commonly in
actual ACIP Release IV texts.
</p>
<p>
The first construct that deserves explanation is the line
break.&nbsp; By the ACIP standard, line breaks in the input do not
become line breaks in the output unless there are two line breaks in
the input.&nbsp; For example, the ACIP snippet below has only one
line break in the output although three line breaks appear in the
input:
</p>
<pre>
BKRA SHIS
BDE LEGS,
THUGS RJE CHE ... and so on ...
</pre>
<p>
One fine point is that the converter does not require a space before
a line break. If {SHIS} appears before a line break, the converter
inserts a space so that it's treated just like {SHIS&nbsp;} is
treated.&nbsp; This oddity is needed to convert real ACIP documents.
</p>
<p>
Another fine point is that ACIP's {^} character &quot;eats&quot; a
following space or a newline.&nbsp; This is so that
{^&nbsp;GONG&nbsp;SA&nbsp;} is treated identically to
{^GONG&nbsp;SA&nbsp;}.
</p>
<p>
Comments appear in a Latin typeface always.&nbsp; Comments are not
allowed just anywhere -- a comment cannot occur within a single
<i>tsheg bar</i>, for example, and it cannot appear between a
<i>tsheg bar</i> and the <i>tsheg</i> that follows it.&nbsp; That
is, {BD[#COMMENT]E} is not like {BDE}, and {BDE[#COMMENT]&nbsp;LEGS}
is not like {BDE&nbsp;LEGS} (though {BDE&nbsp;[#COMMENT]LEGS} is).
</p>
<p>
Corrections are interpreted as Tibetan, not English, by default, but
there is a built-in list of corrections that should appear in the
output in a Latin typeface.&nbsp; (Actually, any correction that
starts with a certain string will appear in a Latin typeface.)&nbsp;
The full list is the following:
</p>
<pre>
"LINE" // from KD0001I1.ACT
"DATA" // from KL0009I2.INC
"BLANK" // from KL0009I2.INC
"NOTE" // from R0001F.ACM
"alternate" // from R0018F.ACE
"02101-02150 missing" // from R1003A3.INC
"51501-51550 missing" // from R1003A52.ACT
"BRTAGS ETC" // from S0002N.ACT
"TSAN, ETC" // from S0015N.ACT
"SNYOMS, THROUGHOUT" // from S0016N.ACT
"KYIS ETC" // from S0019N.ACT
"MISSING" // from S0455M.ACT
"this" // from S6850I1B.ALT
"THIS" // from S0057M.ACT
</pre>
<p>
Somewhat related is the converter's treatment of a few oddball
comments.&nbsp; The oddity is that these comments use the syntax
{[COMMENT]} rather than the standard syntax {[#COMMENT]}.&nbsp; The
converter will treat the following as comments:
</p>
<pre>
From S5274I.ACT:
"[FIRST]"
From S5274I.ACT:
"[SECOND]"
From S0216M.ACT:
"[Additional verses added by Khen Rinpoche here are]"
From S0216M.ACT:
"[ADDENDUM: The text of]"
From S0216M.ACT:
"[END OF ADDENDUM]"
From S0216M.ACT:
"[Some of the verses added here by Khen Rinpoche include:]"
From S0216M.ACT (note the typo):
"[Note that, in the second verse, the {YUL LJONG} was orignally {GANG LJONG},
and is now recited this way since the ceremony is not only taking place in Tibet.]"
From S6954E1.ACT:
"[text missing]"
From TD3817I.INC:
"[INCOMPLETE]"
From S0935m.act:
"[MISSING PAGE]"
From S0975I.INC:
"[MISSING FOLIO]"
From S0839D1I.INC:
"[UNCLEAR LINE]"
From SE6260A.INC:
"[THE FOLLOWING TEXT HAS INCOMPLETE SECTIONS, WHICH ARE ON ORDER]"
From SE6260A.INC:
"[@DATA INCOMPLETE HERE]"
From SE6260A.INC:
"[@DATA MISSING HERE]"
From TD4035I.INC:
"[LINE APPARENTLY MISSING THIS PAGE]"
From TD4226I2.INC:
"[DATA INCOMPLETE HERE]"
To be consistent with the above:
"[DATA MISSING HERE]"
From S0018N.ACT:
"[FOLLOWING SECTION WAS NOT AVAILABLE WHEN THIS EDITION WAS
PRINTED, AND IS SUPPLIED FROM ANOTHER, PROBABLY THE ORIGINAL:]"
From S0018N.ACT:
"[THESE PAGE NUMBERS RESERVED IN THIS EDITION FOR PAGES
MISSING FROM ORIGINAL ON WHICH IT WAS BASED]"
From S0018N.ACT:
"[PAGE NUMBERS RESERVED FROM THIS EDITION FOR MISSING
SECTION SUPPLIED BY PRECEDING]"
From S0057M.ACT:
"[SW: OK]"
From S0057M.ACT:
"[m:ok]"
From S0057M.ACT:
"[A FIRST ONE
MISSING HERE?]"
From S0195A1.INC:
"[THE INITIAL PART OF THIS TEXT WAS INPUT BY THE SERA MEY LIBRARY IN
TIBETAN FONT AND NEEDS TO BE REDONE BY DOUBLE INPUT]"
</pre>
<p>
The converter also supports several non-standard folio
markers.&nbsp; A review of ACIP Release IV texts determined that the
following types of folio markers can appear:
</p>
<pre>
@001
@001A
@001B
@01A.3
@012A.3
@[07B]
@00007B
@00007
@B00007
@[00007A]
</pre>
<p>
Similarly, to support real ACIP Release IV texts, the converter
treats {[DD1]}, {[DD2]}, {[ DD ]}, and {[DDD]} just like {[DD]}
(which is specified in the ACIP standard).&nbsp; It treats {[ BP ]}
and {[BLANK PAGE]} just like {[BP]}, also.
</p>
<p>
The lists above were created by a most fallible process of reviewing
a large number of ACIP Release IV texts.&nbsp; Your suggestions for
additions to these lists are highly valued; please contact <a
href="mailto:thdltools-devel@lists.sourceforge.net">the
developers</a>.
</p>
<p>
FIXME: describe when the converter treats a space as a <i>tsheg</i> and when a space is Tibetan whitespace.&nbsp; Describe how a tsheg does not appear after {KA} and {GA} with most vowels, describe the handling of {NGA,} as {NGA&nbsp;,}.&nbsp; Talk about dzongkha vs. tibetan when it comes to a <i>tsheg</i> at the end of a string of <i>tsheg bar</i>s.&nbsp; Describe treatment of final line break or lack thereof.&nbsp; Warn users to watch out for lines that end with {-}.&nbsp; Describe treatment of {.} in certain contexts as U+0F0C.&nbsp; Etc.
<!-- <h1>DLC</h1>
<pre>
DLC warn on BUR'ANG because BUR'ANG and BUR-'ANG both appear once in ACIP files. Tell RC. DLC: subst it!
lex: Spaces
DLC configurable tibetan spaces bhutan vs. tibet
{NGA,} -> {NGA ,}
DLC hyphens and ' end lines but shouldn't, eh?
DLC ACIP {.} is just an error! Have the error message mention {\u0F0C} for those desiring a non-breaking tsheg.
DLC whitespace -- newlines [final newline DLC?], spaces
</pre> -->
</p>
<a name="parse"></a><h2>Parsing <i>tsheg bar</i>s: Greedy Stacking and
Nativeness</h2>
<p>
This section is a technical reference sufficiently detailed so that
you can fully understand the inner workings of the converter as it
decides which Unicode or TMW to use for a given <i>tsheg
bar</i>.&nbsp; The problem of <a href="#lex">breaking up a text into
<i>tsheg bar</i>s</a> is a separate issue; this section describes
what happens to a <i>tsheg bar</i> after it's been chipped away from
the text.
</p>
<a name="native"></a>
<p>
The ACIP-&gt;Tibetan converters have a notion of
<i>nativeness</i>.&nbsp; Each <i>tsheg bar</i> is either native
Tibetan or non-native.&nbsp; For example, in Buddhist texts written
in Tibetan, Sanskrit mantras often appear in Tibetan
characters.&nbsp; This &quot;Tibetanized Sanskrit&quot; is
non-native.&nbsp; The <i>tsheg bar</i>s that make up this mantra
(and here, take &quot;tsheg bar&quot; somewhat literally to mean the
characters delimited by punctuation and whitespace) are some native
and some non-native in the converter's eyes.&nbsp; For example, the
<i>tsheg bar</i> {MA&nbsp;} appears in some mantras, and is thus in
fact non-native.&nbsp; The converter, however, treats {MA&nbsp;} as
native in all contexts.&nbsp; Thus, &quot;native&quot; is a
technical term with a slightly different meaning than usual.
</p>
<p>
The idea of nativeness is important because it affects how the
converter treats a <i>tsheg bar</i>.&nbsp; In ACIP transliteration,
the rule is that consonants stack up until punctuation, whitespace,
or a vowel appears.&nbsp; For example, {RDZYA} is equivalent to
{R+DZ+YA}.&nbsp; ({DZA} always means the letter {DZA} itself, never
{D+ZA}.)&nbsp; But this greedy stacking does not apply to {SOGS},
which is equivalent to {SOG-S}, not {SOG+S}.&nbsp; Why not?&nbsp;
Because {SOGS} is a native <i>tsheg bar</i> where GA is the suffix
and SA is the postsuffix.&nbsp; Similarly, {GNAD} is {G-NAD}, not
{G+NAD}.&nbsp; Why?&nbsp; Because GA is a prefix in this native
Tibetan <i>tsheg bar</i>.
</p>
<p>
In this section, we will illustrate the inner workings of this
aspect of the converter.&nbsp; You will be able to determine which
snippets of transliteration the converter considers to be native
<i>tsheg bar</i>s, where greedy stacking does not apply except for
the root stack, and which snippets are non-native, and thus wholly
subject to greedy stacking.
</p>
<h3>Anatomy of a Native <i>tsheg bar</i></h3>
<p>
First, the <a href="#lex">lexical analyzer</a> ensures that only the
Tibetan and Sanskrit consonants, the vowels {A}, {I}, {U}, {E}, {O},
{OO}, {EE}, {i}, {'A}, {'I}, {'U}, {'E}, {'O}, {'OO}, {'EE}, and
{'i}, and the adornments {m} and {:} are allowed in a <i>tsheg
bar</i>.
</p>
<p>
As far as the converter is concerned, a native <i>tsheg bar</i>
consists of an optional prefix, a native root stack, an optional
suffix, an optional postsuffix (also known as a secondary suffix)
that may only be present if a suffix is present, and zero or more
<i>appendages</i> (my term, created because I don't know what a
grammarian calls such a thing).&nbsp; An appendage is one of the
following stack sequences:
</p>
<ul>
<li>{'E}</li>
<li>{'I}</li>
<li>{'O}</li>
<li>{'U}</li>
<li>{'US}</li>
<li>{'UR}</li>
<li>{'UM}</li>
<li>{'ONG}</li>
<li>{'ONGS}</li>
<li>{'OS}</li>
<li>{'IS}</li>
<li>{'UNG}</li>
<li>{'ANG}</li>
<li>{'AM}</li>
</ul>
<p>
A <i>tsheg bar</i> is non-native if it has a non-native root stack
or if it contains the {:} character.&nbsp; Any vowel is allowed on a
native root stack, even {'EEm}, {i}, or the like.
</p>
<p>
The rule about native root stacks is important, for example, in
determining that {KTYAMS} is {K+T+YAM+SA} instead of {K+T+YAMASA}
(because K+T+YA is not a native stack).&nbsp; Another example is
{GNVA}, which is treated like {G+N+VA}, not {G-N+VA}, even though
{GNA} is treated like {G-NA} because NA can take a GA prefix.&nbsp;
The complete list of native stacks is the following:
</p>
<ul>
<li>KA</li>
<li>KHA</li>
<li>GA</li>
<li>NGA</li>
<li>CA</li>
<li>CHA</li>
<li>JA</li>
<li>NYA</li>
<li>TA</li>
<li>THA</li>
<li>DA</li>
<li>NA</li>
<li>PA</li>
<li>PHA</li>
<li>BA</li>
<li>MA</li>
<li>TZA</li>
<li>TSA</li>
<li>DZA</li>
<li>WA</li>
<li>ZHA</li>
<li>ZA</li>
<li>'A</li>
<li>YA</li>
<li>RA</li>
<li>LA</li>
<li>SHA</li>
<li>SA</li>
<li>HA</li>
<li>AA</li>
<li>R+KA (RKA)</li>
<li>R+GA (RGA)</li>
<li>R+NGA (RNGA)</li>
<li>R+JA (RJA)</li>
<li>R+NYA (RNYA)</li>
<li>R+TA (RTA)</li>
<li>R+DA (RDA)</li>
<li>R+NA (RNA)</li>
<li>R+BA (RBA)</li>
<li>R+MA (RMA)</li>
<li>R+TZA (RTZA)</li>
<li>R+DZA (RDZA)</li>
<li>L+KA (LKA)</li>
<li>L+GA (LGA)</li>
<li>L+NGA (LNGA)</li>
<li>L+CA (LCA)</li>
<li>L+JA (LJA)</li>
<li>L+TA (LTA)</li>
<li>L+DA (LDA)</li>
<li>L+PA (LPA)</li>
<li>L+BA (LBA)</li>
<li>L+HA (LHA)</li>
<li>S+KA (SKA)</li>
<li>S+GA (SGA)</li>
<li>S+NGA (SNGA)</li>
<li>S+NYA (SNYA)</li>
<li>S+TA (STA)</li>
<li>S+DA (SDA)</li>
<li>S+NA (SNA)</li>
<li>S+PA (SPA)</li>
<li>S+BA (SBA)</li>
<li>S+MA (SMA)</li>
<li>S+TZA (STZA)</li>
<li>K+VA (KVA)</li>
<li>KH+VA (KHVA)</li>
<li>G+VA (GVA)</li>
<li>C+VA (CVA)</li>
<li>NY+VA (NYVA)</li>
<li>T+VA (TVA)</li>
<li>D+VA (DVA)</li>
<li>TZ+VA (TZVA)</li>
<li>TS+VA (TSVA)</li>
<li>ZH+VA (ZHVA)</li>
<li>Z+VA (ZVA)</li>
<li>R+VA (RVA)</li>
<li>SH+VA (SHVA)</li>
<li>S+VA (SVA)</li>
<li>H+VA (HVA)</li>
<li>K+YA (KYA)</li>
<li>KH+YA (KHYA)</li>
<li>G+YA (GYA)</li>
<li>P+YA (PYA)</li>
<li>PH+YA (PHYA)</li>
<li>B+YA (BYA)</li>
<li>M+YA (MYA)</li>
<li>K+RA (KRA)</li>
<li>KH+RA (KHRA)</li>
<li>G+RA (GRA)</li>
<li>T+RA (TRA)</li>
<li>TH+RA (THRA)</li>
<li>D+RA (DRA)</li>
<li>P+RA (PRA)</li>
<li>PH+RA (PHRA)</li>
<li>B+RA (BRA)</li>
<li>M+RA (MRA)</li>
<li>SH+RA (SHRA)</li>
<li>S+RA (SRA)</li>
<li>H+RA (HRA)</li>
<li>K+LA (KLA)</li>
<li>G+LA (GLA)</li>
<li>B+LA (BLA)</li>
<li>Z+LA (ZLA)</li>
<li>R+LA (RLA)</li>
<li>S+LA (SLA)</li>
<li>R+K+YA (RKYA)</li>
<li>R+G+YA (RGYA)</li>
<li>R+M+YA (RMYA)</li>
<li>R+G+VA (RGVA)</li>
<li>R+TZ+VA (RTZVA)</li>
<li>S+K+YA (SKYA)</li>
<li>S+G+YA (SGYA)</li>
<li>S+P+YA (SPYA)</li>
<li>S+B+YA (SBYA)</li>
<li>S+M+YA (SMYA)</li>
<li>S+K+RA (SKRA)</li>
<li>S+G+RA (SGRA)</li>
<li>S+N+RA (SNRA)</li>
<li>S+P+RA (SPRA)</li>
<li>S+B+RA (SBRA)</li>
<li>S+M+RA (SMRA)</li>
<li>G+R+VA (GRVA)</li>
<li>D+R+VA (DRVA)</li>
<li>PH+Y+VA (PHYVA)</li>
</ul>
<p>
(Some would argue that LVA is notably absent.&nbsp; It is seen in
ACIP Buddhist texts in {AELVA}, {LVAm}, {LVU}, {LVUN}, {LVAR},
{LVE}, {LVANG}, and {LVA}.&nbsp; Greedy stacking affects none of
these <i>tsheg bar</i>s' parsing, however.)
</p>
<a name="prefix"></a>
<p>
Not all characters can be prefixes and the like.&nbsp; Only the five
prefixes (GA, DA, BA, MA, 'A), ten suffixes (GA, NGA, DA, NA, BA,
MA, 'A, RA, LA, SA), and two postsuffixes (DA, SA) every Tibetan
student knows are allowed, and they cannot appear with vowels.&nbsp;
(In {LE'U}, {'} is not a suffix -- it is part of an
appendage.)&nbsp; In fact, certain prefixes may only appear with
certain root stacks.&nbsp; The reason that these prefix rules matter
is that they govern how <i>tsheg bar</i>s are parsed.&nbsp; For
example, {GNA} is parsed like {G-NA}, because NA takes a GA
prefix.&nbsp; But {GPA} is parsed like {G+PA}, because PA does not
take a GA prefix.
</p>
<p>
Prefix rules are a topic of some controversy; different grammars
give different lists of prefix rules.&nbsp; For a converter, it is
important that the converter's knowledge of prefix rules matches the
knowledge of the person who typed in the ACIP transliteration, not
that the converter agrees with a grammarian.&nbsp; For example, if
the input technician thought that PA could take a GA prefix, then
the converter will produce {G+PA} when {G-PA} was intended.&nbsp;
For this reason, the converter can produce a warning every time a
prefix rule prohibited the treatment of one of the five prefixes as
a prefix.&nbsp; For example, {GPA} produces this warning.&nbsp;
However, {GNA} produces no warning, because the converter assumes
that it is unlikely that an input technician would enter {GNA} upon
seeing {G+NA}.&nbsp; Part of the reason for this assumption is that
the <i>Asian Classics Input Project Entry Operator Transcription
Chart</i> as of Spring, 1993, explicitly enumerates the following
cases for special treatment by input operators:
</p>
<ul>
<li>{BDA'} vs. {B+DA}</li>
<li>{DBANG} vs. {D+BA}</li>
<li>{DGA'} vs. {D+GA}</li>
<li>{DGRA} vs. {D+GRA}</li>
<li>{DGYES} vs. {D+GYA}</li>
<li>{DMAR} vs. {D+MA}</li>
<li>{GDA'} vs. {G+DA}</li>
<li>{GNAD} vs. {G+NA}</li>
<li>{MNA'} vs. {M+NA}</li>
</ul>
<p>
Regardless, for best results, you should ensure that the input
technician's knowledge of prefix rules matches the converter's
knowledge.&nbsp; The following are the legal combinations of prefix
and root stack in the converter's eyes:
</p>
<ul>
<li>
The BA prefix may occur with any of the following stacks:
<ul>
<li>KA</li>
<li>SA</li>
<li>CA</li>
<li>TA</li>
<li>TZA</li>
<li>GA</li>
<li>DA</li>
<li>ZHA</li>
<li>ZA</li>
<li>SHA</li>
<li>K+YA (KYA)</li>
<li>G+YA (GYA)</li>
<li>K+RA (KRA)</li>
<li>G+RA (GRA)</li>
<li>S+RA (SRA)</li>
<li>G+LA (GLA)</li>
<li>K+LA (KLA)</li>
<li>Z+LA (ZLA)</li>
<li>R+LA (RLA)</li>
<li>S+LA (SLA)</li>
<li>S+KA (SKA)</li>
<li>S+GA (SGA)</li>
<li>S+NGA (SNGA)</li>
<li>S+NYA (SNYA)</li>
<li>S+TA (STA)</li>
<li>S+DA (SDA)</li>
<li>S+NA (SNA)</li>
<li>S+TZA (STZA)</li>
<li>R+KA (RKA)</li>
<li>R+GA (RGA)</li>
<li>R+NGA (RNGA)</li>
<li>R+JA (RJA)</li>
<li>R+NYA (RNYA)</li>
<li>R+TA (RTA)</li>
<li>R+DA (RDA)</li>
<li>R+NA (RNA)</li>
<li>R+TZA (RTZA)</li>
<li>R+DZA (RDZA)</li>
<li>L+CA (LCA)</li>
<li>L+TA (LTA)</li>
<li>L+DA (LDA)</li>
<li>R+K+YA (RKYA)</li>
<li>R+G+YA (RGYA)</li>
<li>S+K+YA (SKYA)</li>
<li>S+G+YA (SGYA)</li>
<li>S+K+RA (SKRA)</li>
<li>S+G+RA (SGRA)</li>
</ul>
</li>
<li>
The GA prefix may occur with any of the following stacks:
<ul>
<li>CA</li>
<li>DA</li>
<li>NA</li>
<li>NYA</li>
<li>SA</li>
<li>SHA</li>
<li>TA</li>
<li>TZA</li>
<li>YA</li>
<li>ZA</li>
<li>ZHA</li>
</ul>
</li>
<li>
The 'A prefix may occur with any of the following stacks:
<ul>
<li>GA</li>
<li>JA</li>
<li>DA</li>
<li>BA</li>
<li>DZA</li>
<li>KHA</li>
<li>CHA</li>
<li>THA</li>
<li>PHA</li>
<li>TSA</li>
<li>PH+YA (PHYA)</li>
<li>B+YA (BYA)</li>
<li>KH+YA (KHYA)</li>
<li>G+YA (GYA)</li>
<li>B+RA (BRA)</li>
<li>KH+RA (KHRA)</li>
<li>G+RA (GRA)</li>
<li>D+RA (DRA)</li>
<li>PH+RA (PHRA)</li>
</ul>
</li>
<li>
The MA prefix may occur with any of the following stacks:
<ul>
<li>KHA</li>
<li>GA</li>
<li>CHA</li>
<li>JA</li>
<li>THA</li>
<li>TSA</li>
<li>DA</li>
<li>DZA</li>
<li>NGA</li>
<li>NYA</li>
<li>NA</li>
<li>KH+YA (KHYA)</li>
<li>G+YA (GYA)</li>
<li>KH+RA (KHRA)</li>
<li>G+RA (GRA)</li>
</ul>
</li>
<li>
The DA prefix may occur with any of the following stacks:
<ul>
<li>BA</li>
<li>GA</li>
<li>KA</li>
<li>MA</li>
<li>NGA</li>
<li>PA</li>
<li>B+RA (BRA)</li>
<li>B+YA (BYA)</li>
<li>G+RA (GRA)</li>
<li>G+YA (GYA)</li>
<li>K+RA (KRA)</li>
<li>K+YA (KYA)</li>
<li>M+YA (MYA)</li>
<li>P+RA (PRA)</li>
<li>P+YA (PYA)</li>
</ul>
</li>
</ul>
<p>
In the above list, the presence of wa-zur (ACIP {V}) does not
disallow a prefix-root combination; nor does the presence of any
vowel, even {'EEm}.&nbsp; The presence of {:} does disallow
prefix-root combinations; e.g., {GN'EEm} is {G-N'EEm}, but {GNA:} is
{G+NA:}.&nbsp; ({GNVA} is parsed as {G+N+VA} not because NVA cannot
take a GA prefix, but because NVA is not a native stack.)
</p>
<p>
The converter will allow any suffix to go with any native root or
prefix-root combination; it will allow any postsuffix to follow any
suffix.&nbsp; It will allow any appendage on any native <i>tsheg
bar</i>.
</p>
<p>
For example, {SOGS}, {BSOGS}, {BS'EEmGS}, {LE'U'I'O} and
{BSKYABS-'UR-'UNG-'O} are all native <i>tsheg bar</i>s in the
converter's eyes.&nbsp; Note the need for disambiguation: {PAM-'AM}
is a native <i>tsheg bar</i>, but {PAM'AM}, which parses as the
three stacks {PA}, {M'A}, and {MA}, is not.&nbsp; (In practice,
appendages rarely occur after prefixes.&nbsp; {BUR-'ANG} appears at
least once in ACIP files and {DGA'-'AM} appears at least twice, but
these may be typos.&nbsp; The converter does allow it, though.&nbsp;
It thinks {BIR'U} and {WAN'U} (which also occur, but only very
rarely) are both non-native, though, and thus treats {'} as U+0F71
(subscribed) and not U+0F60 (full form) in each case.)
</p>
<p>
Note a fine point.&nbsp; When turning a <i>tsheg bar</i> into
Tibetan, the ACIP-&gt;Tibetan converters assume that subjoined YA
and RA consonants are not fixed-form -- not U+0FBB and U+0FBC -- but
rather are the usual subjoined forms U+0FB1 and U+0FB2.&nbsp; The
only exceptions are the stacks R+Y, Y+Y, and n+d+Y, which are known
to have fixed-form subjoined YA, and the stacks n+d+R+Y (where RA
but not YA is full-form) and K+sh+R, which are known to have
fixed-form subjoined RA.&nbsp; Wa-zur, U+0FAD, is never confused
with full-form subjoined WA, U+0FBA, though, because ACIP represents
the former with {V} and the latter with {W}.&nbsp; Furthermore, the
converter never generates U+0F6A, the fixed-form RA (<i>rango</i>);
U+0F62 is always produced.&nbsp; (Note that U+0F62 is often
displayed as a fixed-form RA itself, as in {RNYA}.)
</p>
<p>
So far, we have spoken about consonants and vowels.&nbsp; In fact,
it is not trivial to determine when something is a consonant and
when it is a vowel.&nbsp; {A} can represent U+0F68, the Tibetan
letter, or the implicit vowel.&nbsp; {'} can represent U+0F71, the
subscribed a-chung, or U+0F60, the full-sized consonant
a-chung.&nbsp; The converter treats {TAA} as {T+AA}, not {TA-AA},
but treats {TAAA} like {TA-AA}, not {T+AA-A}.&nbsp; It treats
{PA'AM} like {PA-'A-M}, not {P+A'A-M}.&nbsp; In short, it first
tries out treating {'} and {A} like vowels, but will backtrack if
that leads to a clearly invalid <i>tsheg bar</i>.
</p>
<p>
Finally, a string of numbers can be a <i>tsheg bar</i> also.&nbsp;
It is illegal for numbers and consonants to appear together within
one <i>tsheg bar</i>, however.
</p>
<p>
The above is the complete understanding of the converter's
algorithms for parsing <i>tsheg bar</i>s.&nbsp; You the native
Tibetan speaker may know that {BSKYABS-'UR-'UNG-'O} is not allowed
and thus think that {B+S+K+YAB+S-'UR-'UNG-'O} should be the result,
but the converter has no such knowledge, and thinks this is a native
tsheg bar equivalent to {B-S+K+YAB-S-'UR-'UNG-'O}.
</p>
<a name="sysprops"></a><h2>System Properties</h2>
<p>
The <a href="#sub"><i>tsheg-bar</i> substitution</a> mechanism is
customizable via system properties.&nbsp; Java developers likely
know what these are, but few users do.&nbsp; This section will
perhaps get a determined person started, but if you have trouble,
contact <a href="mailto:thdltools-devel@lists.sourceforge.net">the
developers</a> so that we can improve this documentation or create a
better user interface.
</p>
<p>
For the tool to respect the value of a system property, you must
invoke the tool from the command line as follows:
</p>
<p>
<tt>
java
"-Dorg.thdl.tib.text.ttt.ReplacementMap=KAsh=&gt;K+sh,ONYA=&gt;[#ERROR-ONYA-IS-O&amp;]"
-Dorg.thdl.tib.text.ttt.VerboseReplacementMap=true
-jar Jskad.jar
</tt>
</p>
<a name="bugs"></a><h2>Known Bugs</h2>
<p>
This section presents areas where the current tool's behavior is
wrong.&nbsp; Before doing serious work with the converter,
familiarize yourself with this section and develop a plan to work
around the bugs or to ensure that your documents will not trigger
the bugs.&nbsp; At the same time, if any of these bugs affects you,
contact <a href="mailto:thdltools-devel@lists.sourceforge.net">the
developers</a> so that we can fix them.&nbsp; The squeaky wheel
surely gets the grease; these bugs may never be fixed if there are
no complaints.
</p>
<p>
The following are all known bugs:
</p>
<ul>
<li>
When ACIP {MTHARo} is given, the {o} glyph should be centered
under the THA glyph in ACIP-&gt;TMW conversions.&nbsp; At present,
the {o} glyph appears underneath the rightmost stack.&nbsp;
Similarly, {\u0F35} and {\u0F37} are not centered properly.&nbsp;
[<a
href="http://sourceforge.net/tracker/index.php?func=detail&aid=838594&group_id=61934&atid=502515">838594</a>]
</li>
<li>
ACIP-&gt;TMW conversion for {\u0F3E} is not correct.&nbsp; Fear
not; the character U+0F3E is so rare that no ACIP transliteration
exists for it.&nbsp; [<a
href="http://sourceforge.net/tracker/index.php?func=detail&aid=855478&group_id=61934&atid=502515">855478</a>]
</li>
<li>
In a command-line ACIP-&gt;Unicode text file conversion, no
warning or error is given when the input is {KA (KHA)}.&nbsp; (The
output is a text file and does not have a mechanism for indicating
a change in font size.)&nbsp; [<a
href="http://sourceforge.net/tracker/index.php?func=detail&aid=855519&group_id=61934&atid=502515">855519</a>]
</li>
</ul>
<a name="room"></a><h2>Room for Improvement</h2>
<p>
This section presents areas where the current tool could be
improved.&nbsp; None of the current behavior described here is
incontrovertibly flawed (i.e., there are no bugs described here, see
<a href="#bugs">known bugs</a> for that); current behavior is
technically correct.&nbsp; However, the current behavior is not, in
everyone's eyes, perfect.
</p>
<p>
The following are the current areas in which the tool could be
better:
</p>
<ul>
<li>
The glyph TibetanMachineWeb9.61 -- the {O'I} special combination
(i.e., the glyph for the Unicode string U+0F7C,U+0F60,U+0F72) --
is never output by the ACIP-&gt;TMW converter.&nbsp; It is
sometimes more beautiful than the glyphs that are presently output
(three separate glyphs instead of the one).
</li>
<li>
Though the ACIP standard disallows it, you will find in ACIP
documents from the Buddhist Canon things like {/NYA\} where the
standard demands {/NYA/}.&nbsp; Presently, this triggers an error;
it would be better if this were converted like {/NYA/} is, and
triggered only a <tt>Most</tt>-level warning.
</li>
<li>
The hypothetical comment {[# \u0F40 may have been intended...]}
should cause a warning saying that Unicode escapes do not apply
within comments.
</li>
<li>
The whitespace after a <a href="#escapes">Unicode escape</a> is
not interpreted correctly when that Unicode escape represents
something that is part of a <i>tsheg bar</i>.&nbsp; For example,
the space in {KA KHA} is treated as a <i>tsheg</i> (i.e., U+0F0B),
but the space in {\u0F40 KHA} is wrongly treated as Tibetan
whitespace.&nbsp; [<a
href="http://sourceforge.net/tracker/index.php?func=detail&aid=855482&group_id=61934&atid=502515">855482</a>]
</li>
<li>
Though not standard, {:} and {:-} sometimes are intended to
represent U+0F14.&nbsp; The latter causes an error; it should
cause a warning suggested that the <a href="#escapes">Unicode
escape</a> {\u0F14} be used instead.&nbsp; The former is always
treated as U+0F7F; it should cause a warning in some or all
contexts.
</li>
<li>
The <a href="#sub"><i>tsheg-bar</i> substitution</a> mechanism
should be more general.&nbsp; The useful rule
<tt>ONYA=&gt;O&amp;</tt> should be supported.
</li>
<li>
The converters should support a white list of acceptable
non-native <i>tsheg bar</i>s (where the term &quot;tsheg bar&quot;
is to be interpreted somewhat literally here as any characters
between punctuation).&nbsp; Non-native <i>tsheg bar</i>s not on
the list should produce warnings or errors.&nbsp; Similarly, but
perhaps less urgently, a syllabary of native <i>tsheg bar</i>s
should be supported too.&nbsp; (A workaround is to use <a
href="#colors">coloring</a>, have your word processor delete
everything but the colored text, sort the colored <i>tsheg
bar</i>s, and inspect them all by hand.&nbsp; Also, <a
href="#stats"><i>tsheg-bar</i> statistics</a> will help you to
find uncommon <i>tsheg bar</i>s.)
</li>
<li>
ACIP-&gt;Unicode conversions produce Unicode text files at
present.&nbsp; While more compact than Rich Text Format (RTF)
files, a text file does not allow for supporting the two font
sizes in {KA (KA)}.&nbsp; A workaround is to use an ACIP-&gt;TMW
conversion followed by a separate <a
href="TMW_or_TM_To_X_Converters.html">TMW-&gt;Unicode</a>
conversion.
</li>
<li>
The converter should warn for each occurrence of the vowels {'E},
{'O}, {'EE}, or {'OO}.
</li>
</ul>
<h2>License</h2>
<p>Both the ACIP-&gt;Tibetan converters and this document are released
under the <a
href="http://iris.lib.virginia.edu/tibet/tools/thdl_license.txt">THDL
Open Community License Version 1.0</a>.</p>
<p>
Please
<a href="mailto:thdltools-devel@lists.sourceforge.net">
e-mail us</a>
your comments about this page.
</p>
<p>
The
<a href="http://www.sourceforge.net/projects/thdltools">
THDL Tools</a>
project is generously hosted by:
<!--
DO NOT DELETE THE SF.NET LOGO.
We have a choice of colors and sizes for this logo (see
"https://sourceforge.net/docman/display_doc.php?docid=790&group_id=1"),
but we do not have the option of removing it. SourceForge requests
that we put it on each web page for our project, and to give us
incentive to do so, they will not track the number of hits for our
project web pages unless we put this link in. To track hits, see
"http://sourceforge.net/project/stats/index.php?report=months&group_id=61934".
-->
<a href="http://sourceforge.net/">
<img src="http://sourceforge.net/sflogo.php?group_id=61934&amp;type=1"
width="88" height="31" alt="SourceForge Logo" />
</a>
<!-- AGAIN, DO NOT DELETE THE SF.NET LOGO. -->
</p>
</div>
</body>
</html>
Reference in a new issue View git blame Copy permalink