Added support for two more oddballs.
Deprecated the oddball lookup method because it drops up to 30 glyphs in
TibetanMachine. The correct solution is to transform the RTF before Java's
busted RTF readers ever see it. \'97 becomes \u151, e.g.
beginning of the document as they should and as they are documented to.
They now do, and they bracket the bad characters with the TM or TMW for
U+0F3C on the left and the TM or TMW for U+0F3D on the right.
Some cleanup.
the troublesome glyphs are now put at the beginning of the document
AFTER AN ACHEN. This makes a glyph like \tmw7095 visible atop the
achen.
Major fix to the handling of paragraphs in conversion; we were (for
whatever reason) dropping paragraphs before.
faster than TMW->Unicode etc.; this is because many fewer replacements
are made (i.e., more text is replaced each time a replacement is
performed).
I must find a way to still preserve formatting but do many fewer
replacements in TMW->{Unicode,TM} and TM->TMW.
inserting 5 characters at a time and then skipping ahead just one
position. I don't think this affected correctness.
I believe there's still a terrible (exponential?) slowdown as the
input file gets bigger, however. Perhaps not -- but we run through
the first 1000 TMW glyphs in 6 seconds, the 20th thousand takes at
least 60 seconds. Is TMW->Wylie faster than TMW->Unicode? If so,
why?
Thought: don't use a DuffPane within TibetanConverter -- it can only
add overhead, right? My hprof profile said that the conversion was
taking just a couple of percent of the work; the rest was going to
display-related stuff that you should only see if you were displaying
the document. I'm not!
Is the EWTS '_' to be represented as U+0020, or is it a wider space?
Does TMW9.42, Dza, map to U+0F5F,U+0F39?
Does TMW6.60, r+y, map to U+0F62,U+0FBB or to U+0F6A,U+0FBB? (Likewise with r+w, TMW6.61, TMW6.62, etc.)
Is U+0F7E a bindu? What Unicode does TMW7.96 map to, for example? What does TMW7.91 map to?
Should TMW8.97 and TMW8.98 map to swastiskas elsewhere in Unicode? If so, which codepoints? Likewise with TMW9.60, a Chinese character.
Does TMW7.68 map to U+0F39?
Does TMW7.74, the ITHI secret sign, have a Unicode mapping? f68,fa0,f80,f72 comes close, but fa0 would be too large, wouldn't it?
What Unicode does TMW9.61 map to? Is it for sequences like f40,f7c,f60,f72? Or is it for f60,f72,f7c?
and then tried to convert that TMW to Wylie. I swear it's Java's
problem (see the ugly stack trace in the code and decide for
yourself), and I tried replacing rather than
inserting-and-then-removing, but it didn't work. I've left these
things as options.
default, all oddballs to appear once in the resulting document.
This'll help me find the correct glyphs for the oddballs, and it'll
prevent the average user from converting a document with oddballs.
I've fixed that.
I've also added a couple of Unicode mappings to give a flavor for how
multi-codepoint mappings will be represented.
TM->TMW conversion takes about 1 second per thousand glyphs on my
PIII-550.
noticed that formatting is mostly OK but sometimes gets bungled slightly.
I tried everything I could think of, and now I'm passing the buck to Java's
RTF support.
TMW_RTF_TO_THDL_WYLIE (now misnamed) support TMW->TM
conversion (but not TM->TMW). There is an automated test case for a
TMW->TM conversion.
I have full confidence in this conversion. Even the smallest glitch in the core
functionality (not formatting) would surprise me.
Note that the JUnit test TMW_RTF_TO_THDL_WYLIETest sometimes fails
due to one- or two-line diffs between the actual and expected outputs. This
is because Java's RTF support is not deterministic, I'm guessing, and is not
a real failure. I'm too lazy to make a more elaborate sed/diff mechanism
that works on all platforms, and that would complicate the build anyway.
verified this extensively and have full confidence that these mappings
agree with Tony Duff's Tibetan! 5.1 documentation (except as described
below).
To get them, I had to disregard Tony Duff's tables for a few glyphs: the
characters with ordinal 32 and 45 (space and hyphen in Roman ASCII,
space and tsheg in Tibetan). For these glyphs, we must have mappings
from TibetanMachineSkt4.32 to something, etc., and those mappings were
not present. I've normalized the mapping for these glyphs, as it is arbitrary
because the same two glyphs just appear fifteen times each.
brace problem upon opening any RTF document.
The TMW_RTF_TO_THDL_WYLIE test baselines changed because
I fixed (a while ago) some inconsistencies between the EWTS standard and
Jskad.
Conversion of TibetanMachineWeb8.40, @#, to Wylie now works correctly.
Unfortunately, though, typing @# doesn't produce 8.40, it still produces
8.38 and 8.39, two glyphs.
