mirrors/cosmopolitan
1
0
Fork 0
mirror of https://github.com/jart/cosmopolitan.git synced 2025-02-07 23:13:34 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 1
cosmopolitan/third_party/python/Modules/cjkcodecs
History
Justine Tunney 47a53e143b Productionize new APE loader and more 
The APE_NO_MODIFY_SELF loader payload has been moved out of the examples
folder and improved so that it works on BSD systems, and permits general
elf program headers. This brings its quality up enough that it should be
acceptable to use by default for many programs, e.g. Python, Lua, SQLite
and Python. It's the responsibility of the user to define an appropriate
TMPDIR if /tmp is considered an adversarial environment. Mac OS shall be
supported by APE_NO_MODIFY_SELF soon.

Fixes and improvements have been made to program_executable_name as it's
now the one true way to get the absolute path of the executing image.

This change fixes a memory leak in linenoise history loading, introduced
by performance optimizations in 51904e2687
This change fixes a longstanding regression with Mach system calls, that
23ae9dfceb back in February which impacted
our sched_yield() implementation, which is why no one noticed until now.

The Blinkenlights PC emulator has been improved. We now fix rendering on
XNU and BSD by not making the assumption that the kernel terminal driver
understands UTF8 since that seems to break its internal modeling of \r\n
which is now being addressed by using \e[𝑦H instead. The paneling is now
more compact in real mode so you won't need to make your font as tiny if
you're only emulating an 8086 program. The CLMUL ISA is now emulated too

This change also makes improvement to time. CLOCK_MONOTONIC now does the
right thing on Windows NT. The nanosecond time module functions added in
Python 3.7 have been backported.

This change doubles the performance of Argon2 password stretching simply
by not using its copy_block and xor_block helper functions, as they were
trivial to inline thus resulting in us needing to iterate over each 1024
byte block four fewer times.

This change makes code size improvements. _PyUnicode_ToNumeric() was 64k
in size and now it's 10k. The CJK codec lookup tables now use lazy delta
zigzag deflate (δzd) encoding which reduces their size from 600k to 200k
plus the code bloat caused by macro abuse in _decimal.c is now addressed
so our fully-loaded statically-linked hermetically-sealed Python virtual
interpreter container is now 9.4 megs in the default build mode and 5.5m
in MODE=tiny which leaves plenty of room for chibicc.

The pydoc web server now accommodates the use case of people who work by
SSH'ing into a different machine w/ python.com -m pydoc -p8080 -h0.0.0.0

Finally Python Capsulae delenda est and won't be supported in the future
2021-10-02 08:27:03 -07:00
..
clinic Undiamond Python headers 2021-08-12 14:07:40 -07:00
__big5_decmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
__big5_encmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
__big5hkscs_bmp_encmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
__big5hkscs_decmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
__big5hkscs_nonbmp_encmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
__cp932ext_decmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
__cp932ext_encmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
__cp949_encmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
__cp949ext_decmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
__cp950ext_decmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
__cp950ext_encmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
__gb2312_decmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
__gb18030ext_decmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
__gb18030ext_encmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
__gbcommon_encmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
__gbkext_decmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
__jisx0208_decmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
__jisx0212_decmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
__jisx0213_1_bmp_decmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
__jisx0213_1_emp_decmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
__jisx0213_2_bmp_decmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
__jisx0213_2_emp_decmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
__jisx0213_bmp_encmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
__jisx0213_emp_encmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
__jisx0213_pair_decmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
__jisxcommon_encmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
__ksx1001_decmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
_codecs_cn.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
_codecs_hk.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
_codecs_iso2022.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
_codecs_jp.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
_codecs_kr.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
_codecs_tw.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
alg_jisx0201.inc Perform build and magnum tuning 2021-08-10 10:26:13 -07:00
big5_decmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
big5_encmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
big5hkscs_bmp_encmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
big5hkscs_decmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
big5hkscs_nonbmp_encmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
cjkcodecs.h Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
cp932ext_decmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
cp932ext_encmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
cp949_encmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
cp949ext_decmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
cp950ext_decmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
cp950ext_encmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
emu_jisx0213_2000.inc Perform build and magnum tuning 2021-08-10 10:26:13 -07:00
gb2312_decmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
gb18030ext_decmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
gb18030ext_encmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
gbcommon_encmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
gbkext_decmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
jisx0208_decmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
jisx0212_decmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
jisx0213_1_bmp_decmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
jisx0213_1_emp_decmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
jisx0213_2_bmp_decmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
jisx0213_2_emp_decmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
jisx0213_bmp_encmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
jisx0213_emp_encmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
jisx0213_pair_decmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
jisx0213_pair_encmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
jisxcommon_encmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
ksx1001_decmap.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
multibytecodec.c Productionize new APE loader and more 2021-10-02 08:27:03 -07:00
multibytecodec.h Undiamond Python headers 2021-08-12 14:07:40 -07:00
README python-3.6.zip added from Github 2021-08-09 05:39:42 -07:00
somanyencodings.h Productionize new APE loader and more 2021-10-02 08:27:03 -07:00

README 

To generate or modify mapping headers
-------------------------------------
Mapping headers are imported from CJKCodecs as pre-generated form.
If you need to tweak or add something on it, please look at tools/
subdirectory of CJKCodecs' distribution.



Notes on implmentation characteristics of each codecs
-----------------------------------------------------

1) Big5 codec

  The big5 codec maps the following characters as cp950 does rather
  than conforming Unicode.org's that maps to 0xFFFD.

    BIG5        Unicode     Description

    0xA15A      0x2574      SPACING UNDERSCORE
    0xA1C3      0xFFE3      SPACING HEAVY OVERSCORE
    0xA1C5      0x02CD      SPACING HEAVY UNDERSCORE
    0xA1FE      0xFF0F      LT DIAG UP RIGHT TO LOW LEFT
    0xA240      0xFF3C      LT DIAG UP LEFT TO LOW RIGHT
    0xA2CC      0x5341      HANGZHOU NUMERAL TEN
    0xA2CE      0x5345      HANGZHOU NUMERAL THIRTY

  Because unicode 0x5341, 0x5345, 0xFF0F, 0xFF3C is mapped to another
  big5 codes already, a roundtrip compatibility is not guaranteed for
  them.


2) cp932 codec

  To conform to Windows's real mapping, cp932 codec maps the following
  codepoints in addition of the official cp932 mapping.

    CP932     Unicode     Description

    0x80      0x80        UNDEFINED
    0xA0      0xF8F0      UNDEFINED
    0xFD      0xF8F1      UNDEFINED
    0xFE      0xF8F2      UNDEFINED
    0xFF      0xF8F3      UNDEFINED


3) euc-jisx0213 codec

  The euc-jisx0213 codec maps JIS X 0213 Plane 1 code 0x2140 into
  unicode U+FF3C instead of U+005C as on unicode.org's mapping.
  Because euc-jisx0213 has REVERSE SOLIDUS on 0x5c already and A140
  is shown as a full width character, mapping to U+FF3C can make
  more sense.

  The euc-jisx0213 codec is enabled to decode JIS X 0212 codes on
  codeset 2. Because JIS X 0212 and JIS X 0213 Plane 2 don't have
  overlapped by each other, it doesn't bother standard conformations
  (and JIS X 0213 Plane 2 is intended to use so.) On encoding
  sessions, the codec will try to encode kanji characters in this
  order:

    JIS X 0213 Plane 1 -> JIS X 0213 Plane 2 -> JIS X 0212


4) euc-jp codec

  The euc-jp codec is a compatibility instance on these points:
   - U+FF3C FULLWIDTH REVERSE SOLIDUS is mapped to EUC-JP A1C0 (vice versa)
   - U+00A5 YEN SIGN is mapped to EUC-JP 0x5c. (one way)
   - U+203E OVERLINE is mapped to EUC-JP 0x7e. (one way)


5) shift-jis codec

  The shift-jis codec is mapping 0x20-0x7e area to U+20-U+7E directly
  instead of using JIS X 0201 for compatibility. The differences are:
   - U+005C REVERSE SOLIDUS is mapped to SHIFT-JIS 0x5c.
   - U+007E TILDE is mapped to SHIFT-JIS 0x7e.
   - U+FF3C FULL-WIDTH REVERSE SOLIDUS is mapped to SHIFT-JIS 815f.