Commit graph

518 commits

Author SHA1 Message Date
Gautham
7fe9e70117
Backporting METH_FASTCALL from Python 3.7 (#317)
* dict copy speedup

refer to bpo-31179 or python/cpython@boa7a037b8fde

* __build_class__() uses METH_FASTCALL

refer python/cpython@69de71b255
refer python/cpython@773dc6dd06

a single test related to __prepare__ fails.

* type_prepare uses METH_FASTCALL

refer python/cpython@d526cfe546
refer python/cpython@80ab22fa2c

the prepare-related test still fails.  It's just related to the error
message format though.

* separate into ParseStack and ParseStackAndKeywords

refer python/cpython@6518a93cb1
refer python/cpython@3e1fad6913
refer python/cpython@c0083fc47d

* Add _PyArg_NoStackKeywords

refer python/cpython@29d39cc8f5

* _PyStack_UnpackDict now returns int

refer python/cpython@998c20962c

* METH_FASTCALL changes to .inc files

done via python's Argument Clinic tool,
refer python/cpython@259f0e4437

* Added _PyArg_UnpackStack

refer python/cpython@fe54dda08

* Argument Clinic FASTCALL again

refer python/cpython@0c4a828ca

* Argument Clinic for ordered dictionary object

refer python/cpython@b05cbac052

* speed up getargs

refer python/cpython@1741441649

* FASTCALL for sorted, next, and getattr

refer python/cpython@5a60ecaa7a
refer python/cpython@fda6d0acf0
refer python/cpython@84b388bb80

* Optimize methoddescr_call

refer python/cpython@2a1b676d1f
refer python/cpython@c52572319c
refer python/cpython@35ecebe165
refer python/cpython@8128d5a491

* cleanup _PyMethodDef_RawFastCallDict

refer python/cpython@0a2e46835d
refer python/cpython@98ccba8344
refer python/cpython@c89ef828cf
refer python/cpython@250e4b0063

* print now uses METH_FASTCALL

refer python/cpython@c3858bd7c6
refer python/cpython@bd584f169f
refer python/cpython@06d34393c2

* _struct module now uses Argument Clinic

refer python/cpython@3f2d10132d

* make deque methods faster

refer python/cpython@dd407d5006

* recursive calls in PyObject_Call

refer python/cpython@7399a05965

only partially ported, because RawFastCallKeywords hasn't been ported

* add macros

refer python/cpython@68a001dd59

* all tests pass in MODE=dbg

* convert some internal functions to FASTCALL

__import__ might need to be changed later, if it is possible to backport
the METH_FASTCALL | METH_KEYWORDS flag distinction later.

* speed up unpickling

refer python/cpython@bee09aecc2

* added _PyMethodDef_RawFastCallKeywords

refer python/cpython@7399a05965

* PyCFunction_Call performance

refer python/cpython@12c5838dae

* avoid PyMethodObject in slots

main change in python/cpython@516b98161a
test_exceptions changed in python/cpython@331bbe6aaa
type_settattro changed in python/cpython@193f7e094f
_PyObject_CallFunctionVa changed in python/cpython@fe4ff83049

* fix refcount error found in MODE=dbg

all tests now pass in MODE=dbg
2021-11-12 15:26:57 -08:00
Gautham
6f658f058b
Change noinline to dontinline (#312)
We defined `noinline` as an abbreviation for the longer version
`__attribute__((__noinline__))` which caused name clashes since
third party codebases often write it as `__attribute__((noinline))`.
2021-11-12 15:12:18 -08:00
Paul Kulchenko
ca611efc43
Redbean getstatus and more (#308)
* Add GetBody() Lua API to redbean.
   This improves consistency with RFC 7230 terminology and
   should be favored over the old GetPayload function.
* Add GetStatus() API to redbean.
   This is useful to get status after it's changed/set by Redbean,
   for example if 505 or 508 is set when ServeRedirect is called.
* Introduce GetAssetComment() API to redbean.
   This function should be favored over the old name GetComment().
* Introduce IsLoopbackClient() API to redbean
* Limit redbean reason to 128 chars when set instead of reporting an error
2021-11-12 15:00:41 -08:00
Paul Kulchenko
e5d1536256
Redbean doc updates (#307)
* Fix redbean re.NEWLINE documentation
* Add documentation for method and body parameters in redbean Fetch
* Add documentation for redbean ProgramAddr
* Update redbean SetHeader documentation to clarify behavior with Serve* calls
2021-11-01 18:52:02 -07:00
Gautham
d7ff346b52
Add some Python 3.7 backports (#306)
* make.com now uses stack size of 2mb
* optimize _PyCFunction_FastCallKeywords
* backport python@cpython/7fc252adfbedece75f2330bcfdadbf84dee7836f
2021-10-29 22:54:14 -07:00
Justine Tunney
903cc38c37 Revert "Make MODE=tiny not depend on default build"
This reverts commit 30cd28b1f8.
2021-10-26 17:14:38 -07:00
ahgamut
3616f2a432 get make to build 2021-10-26 14:58:05 -07:00
ahgamut
7a50dba812 changes made to enable compilation 2021-10-26 14:58:05 -07:00
ahgamut
19f70a154e add make-4.3.tar.gz 2021-10-26 14:58:05 -07:00
jacereda
0a0997a872
Add WIN32 symbols for OpenGL support (#304) 2021-10-26 14:18:13 -07:00
Justine Tunney
30cd28b1f8 Make MODE=tiny not depend on default build
This issue was spotted by @ahgamut in #292.
2021-10-25 16:26:22 -07:00
Justine Tunney
91d783352a Apply fixup to cthread initialization (#301)
Cosmopolitan Threads are currently Linux-only (with some NetBSD
and Windows support too!). This change ensures we only initialize
the high-level threading runtime when Cosmopolitan Threads are used.
2021-10-25 16:22:10 -07:00
Florian Lemaitre
45a7435788
[WIP] Threading phase 2 (#301)
* Exponential back-off
* Removed "native" specifier
* Abstract away Futex for cthread
* Complete setup for TLS (including main thread)
2021-10-25 16:02:26 -07:00
Paul Kulchenko
660ff56d40
Redbean fix lua panic (#300)
* Fix Lua panic in redbean when calling request/connection functions from .init.lua

The error calls were triggered appropriately, but used the global Lua
state instead of the current Lua state (within protected call), which
triggered Lua panic. This executes the error in the proper context.

* Rename global Lua state object to make its (mis-)usage easier to detect
2021-10-25 14:54:56 -07:00
Paul Kulchenko
c00755f921
Improve error messages for consistency (#297) 2021-10-25 14:44:04 -07:00
Paul Kulchenko
013f03e33f
Redbean fix redirect (#294)
* Fix redbean crash during redirect in debug logging mode

* Add reset for redirect loop check in redbean Route

Without this reset a combination of RoutePath() and Route() calls
could return "508 loop detected", since RoutePath could be called
twice for the same redirected path.

The protection against looping is still there, as it can only
loop inside the Route() call (as it always serves something).

* Update redbean redirect message for clarity
2021-10-25 14:04:57 -07:00
Gautham
49db877fbe
Minimize Python startup imports (#292)
* get_exports_list should return list
* remove unintentional `CC=clang` in makefile
* avoid importing sysconfig during startup

site.py requires only a couple of functions from sysconfig, but needs to
load the entirety of sysconfig to get those functions. This commit
makes it such that sysconfig is imported only when sys.platform is darwin.

* remove redundant constants from stat module

The constants are only there in case the C implementation (ie the _stat
module) is not available. With Cosmopolitan the _stat module is always
available. The entire Lib/stat.py file can be removed if the Windows-based
constants can be moved into the Modules/_stat.c.

* minimal changes to os.py

python checks os-based assumptions at startup, some of  which can be
bypassed since this is Cosmopolitan Python.
2021-10-25 14:04:04 -07:00
Et7f3
253ac31a64
Fix build from docker build with alpine from macOS (#286)
* fix(build): Remove unused variable and fix const warning.
* fix(build): Fix others -Werror=discarded-qualifiers
Co-authored-by: Élie BRAMI <Et7f3@mbp-de-elie.home>
2021-10-25 13:39:05 -07:00
Justine Tunney
67b5200a0b Add MODE=optlinux build mode (#141) 2021-10-14 19:36:49 -07:00
Justine Tunney
226aaf3547 Improve memory safety
This commit makes numerous refinements to cosmopolitan memory handling.

The default stack size has been reduced from 2mb to 128kb. A new macro
is now provided so you can easily reconfigure the stack size to be any
value you want. Work around the breaking change by adding to your main:

    STATIC_STACK_SIZE(0x00200000);  // 2mb stack

If you're not sure how much stack you need, then you can use:

    STATIC_YOINK("stack_usage_logging");

After which you can `sort -nr o/$MODE/stack.log`. Based on the unit test
suite, nothing in the Cosmopolitan repository (except for Python) needs
a stack size greater than 30kb. There are also new macros for detecting
the size and address of the stack at runtime, e.g. GetStackAddr(). We
also now support sigaltstack() so if you want to see nice looking crash
reports whenever a stack overflow happens, you can put this in main():

    ShowCrashReports();

Under `make MODE=dbg` and `make MODE=asan` the unit testing framework
will now automatically print backtraces of memory allocations when
things like memory leaks happen. Bugs are now fixed in ASAN global
variable overrun detection. The memtrack and asan runtimes also handle
edge cases now. The new tools helped to identify a few memory leaks,
which are fixed by this change.

This change should fix an issue reported in #288 with ARG_MAX limits.
Fixing this doubled the performance of MKDEPS.COM and AR.COM yet again.
2021-10-13 17:27:13 -07:00
Florian Lemaitre
a0b39f886c
[WIP] Threading (#282)
* Thread creation
* Proper thread creation and exit
* Join/Detach protocol
* Added semaphore with futex (hopefully fast)
2021-10-13 11:26:05 -07:00
Gautham
d852640a1e
Add Python ftrace contextmanager (#285) 2021-10-13 11:00:25 -07:00
Justine Tunney
425a57080d Fix minor issues 2021-10-08 20:51:11 -07:00
Justine Tunney
bba144246a Improve Das Blinkenlights
It's now possible to scroll panels is the TUI while the display
is blocked on input. INT 16h now translates UTF-8 to CP-437 and
displays unmappable characters using a lambda symbol. Bracketed
paste mode guards will also be filtered out.
2021-10-08 16:44:19 -07:00
Justine Tunney
7061c79c22 Make fixes, improvements, and chibicc python bindings
- python now mixes audio 10x faster
- python octal notation is restored
- chibicc now builds code 3x faster
- chibicc now has help documentation
- chibicc can now generate basic python bindings
- linenoise now supports some paredit-like features

See #141
2021-10-08 08:41:57 -07:00
Justine Tunney
28997f3acb Make mkdeps.com go faster
This program usually runs once at the begininng of each GNU Make
invocation. It generates an o//depend file with 170,000 lines of
Makefile code to define source -> headers relationships.

This change makes that take 650 milliseconds rather than 1,100ms
by improving the performance of strstr(), using longsort(), plus
migrating to the new append library.
2021-10-04 06:46:46 -07:00
Justine Tunney
725f4d79f6 Apply fixes and speedups 2021-10-04 03:23:31 -07:00
Justine Tunney
7521bf9e73 Add stack overflow checking to Python 2021-10-02 10:50:41 -07:00
Justine Tunney
47a53e143b Productionize new APE loader and more
The APE_NO_MODIFY_SELF loader payload has been moved out of the examples
folder and improved so that it works on BSD systems, and permits general
elf program headers. This brings its quality up enough that it should be
acceptable to use by default for many programs, e.g. Python, Lua, SQLite
and Python. It's the responsibility of the user to define an appropriate
TMPDIR if /tmp is considered an adversarial environment. Mac OS shall be
supported by APE_NO_MODIFY_SELF soon.

Fixes and improvements have been made to program_executable_name as it's
now the one true way to get the absolute path of the executing image.

This change fixes a memory leak in linenoise history loading, introduced
by performance optimizations in 51904e2687
This change fixes a longstanding regression with Mach system calls, that
23ae9dfceb back in February which impacted
our sched_yield() implementation, which is why no one noticed until now.

The Blinkenlights PC emulator has been improved. We now fix rendering on
XNU and BSD by not making the assumption that the kernel terminal driver
understands UTF8 since that seems to break its internal modeling of \r\n
which is now being addressed by using \e[𝑦H instead. The paneling is now
more compact in real mode so you won't need to make your font as tiny if
you're only emulating an 8086 program. The CLMUL ISA is now emulated too

This change also makes improvement to time. CLOCK_MONOTONIC now does the
right thing on Windows NT. The nanosecond time module functions added in
Python 3.7 have been backported.

This change doubles the performance of Argon2 password stretching simply
by not using its copy_block and xor_block helper functions, as they were
trivial to inline thus resulting in us needing to iterate over each 1024
byte block four fewer times.

This change makes code size improvements. _PyUnicode_ToNumeric() was 64k
in size and now it's 10k. The CJK codec lookup tables now use lazy delta
zigzag deflate (δzd) encoding which reduces their size from 600k to 200k
plus the code bloat caused by macro abuse in _decimal.c is now addressed
so our fully-loaded statically-linked hermetically-sealed Python virtual
interpreter container is now 9.4 megs in the default build mode and 5.5m
in MODE=tiny which leaves plenty of room for chibicc.

The pydoc web server now accommodates the use case of people who work by
SSH'ing into a different machine w/ python.com -m pydoc -p8080 -h0.0.0.0

Finally Python Capsulae delenda est and won't be supported in the future
2021-10-02 08:27:03 -07:00
Justine Tunney
9cb54218ab Add error checks to Python objectifier (#281)
PYOBJ.COM was failing when statically analyzing _pyio.py in MODE=dbg
because co_consts contained a big number, which dirtied the interpreter
exception state. We now do comprehensive error checking w/ Python API.

The -DSTACK_FRAME_UNLIMITED CPPFLAG has been removed from DES since its
self test function has been fixed to use heap memory rather than making
aggressive use of the stack.

This change also fixes a regression with function tracing (the --ftrace
flag a.k.a. ftrace_install() a.k.a. cosmo.ftrace) in ASAN build modes.
Lastly, the _tracemalloc module should now always be available for use
in MODE=dbg.
2021-10-02 06:17:17 -07:00
Gautham
57f0eed382
Fix Pyston speedups (#281)
We remove (i.e. hide behind a debug ifdef) the recursion checking methods,
and the memory hooks and memory allocator methods. ASAN mode has no
PYMALLOC, so we need a macro. Fix build break with des.c stack allocation.
2021-10-02 01:28:51 -07:00
Gautham
2fe8571010
Add zipimport hook at the end just in case (#280)
In Python, the zipimport path hook is usually the first entry in
sys.path_hooks, so that any zip files in sys.path can be handled
correctly. In the APE, the zipimport hook was removed because it was
relatively slow compared to Cosmopolitan when it came to handling
imports from the APE's internal zip store.

However, some python scripts (for example when pip installs some
packages) modify sys.path to consider a local zip file, and then attempt
to import from it. This change prevents potential "unable to import"
errors in such cases, so that Actually Portable Python can be more of a
drop-in improved replacement.
2021-09-29 01:50:34 -07:00
Paul Kulchenko
9cabdaabfd
Add clock() polyfill on Windows (#279)
We normally use CLOCK_PROCESS_CPUTIME_ID but there's no clock_gettime()
on Windows NT.
2021-09-28 02:14:24 -07:00
Justine Tunney
39bf41f4eb Make numerous improvements
- Python static hello world now 1.8mb
- Python static fully loaded now 10mb
- Python HTTPS client now uses MbedTLS
- Python REPL now completes import stmts
- Increase stack size for Python for now
- Begin synthesizing posixpath and ntpath
- Restore Python \N{UNICODE NAME} support
- Restore Python NFKD symbol normalization
- Add optimized code path for Intel SHA-NI
- Get more Python unit tests passing faster
- Get Python help() pagination working on NT
- Python hashlib now supports MbedTLS PBKDF2
- Make memcpy/memmove/memcmp/bcmp/etc. faster
- Add Mersenne Twister and Vigna to LIBC_RAND
- Provide privileged __printf() for error code
- Fix zipos opendir() so that it reports ENOTDIR
- Add basic chmod() implementation for Windows NT
- Add Cosmo's best functions to Python cosmo module
- Pin function trace indent depth to that of caller
- Show memory diagram on invalid access in MODE=dbg
- Differentiate stack overflow on crash in MODE=dbg
- Add stb_truetype and tools for analyzing font files
- Upgrade to UNICODE 13 and reduce its binary footprint
- COMPILE.COM now logs resource usage of build commands
- Start implementing basic poll() support on bare metal
- Set getauxval(AT_EXECFN) to GetModuleFileName() on NT
- Add descriptions to strerror() in non-TINY build modes
- Add COUNTBRANCH() macro to help with micro-optimizations
- Make error / backtrace / asan / memory code more unbreakable
- Add fast perfect C implementation of μ-Law and a-Law audio codecs
- Make strtol() functions consistent with other libc implementations
- Improve Linenoise implementation (see also github.com/jart/bestline)
- COMPILE.COM now suppresses stdout/stderr of successful build commands
2021-09-28 01:52:34 -07:00
Paul Kulchenko
fa7b4f5bd1 Updated lua-Argon2 to use Argon2_id variant
As recommended in RFC9106:
https://www.rfc-editor.org/rfc/rfc9106.html#name-recommendations
2021-09-27 21:42:40 -07:00
Paul Kulchenko
b883473a6c Remove optimized blamka rounds from Argon2
Optimized blamka rounds occasionally SIGSERV for me,
so the reference implementation is used instead.
2021-09-27 21:42:40 -07:00
Paul Kulchenko
af1153e057 Reorganize handling of added SSE intrinsics to simplify. 2021-09-27 21:42:40 -07:00
Paul Kulchenko
2e4b40042a Add missing Intel SSE2 intrinsics needed for Argon2 2021-09-27 21:42:40 -07:00
Paul Kulchenko
42aafc847e Update lua-argon2 for cosmopolitan/redbean 2021-09-27 21:42:40 -07:00
Paul Kulchenko
b51db9f923 Add lua-argon2 module for redbean/lua 2021-09-27 21:42:40 -07:00
Paul Kulchenko
6fba9601ff Update Argon2 for cosmopolitan libc 2021-09-27 21:42:40 -07:00
Paul Kulchenko
9e7773a076 Add Argon2 implementation 2021-09-27 21:42:40 -07:00
jacereda
48a2668239
Fix bad dependency (#276) 2021-09-27 20:39:38 -07:00
Paul Kulchenko
5ffeaa8e33
Fix SSE scalar ops (#271) 2021-09-17 16:49:53 -07:00
Florian Lemaitre
18ccfeb919
Embed APE loader inside APE (#267)
The `ape-no-modify-self.o` bootloader now has an mmap-based
payload that helps read-only APE binaries be load faster since it
doesn't need to copy any files.
2021-09-17 16:49:10 -07:00
Justine Tunney
b5f743cdc3 Begin incorporating Python unit tests into build
We now build a separate APE binary for each test so they can run in
parallel. We've got 148 tests running fast and stable so far.
2021-09-12 21:04:44 -07:00
Justine Tunney
51904e2687 Improve Python and Linenoise
This change reinvents all the GNU Readline features I discovered that I
couldn't live without, e.g. UTF-8, CTRL-R search and CTRL-Y yanking. It
now feels just as good in terms of user interface from the subconscious
workflow perspective. It's real nice to finally have an embeddable line
reader that's actually good with a 30 kb footprint and a bsd-2 license.

This change adds a directory to the examples folder, explaining how the
new Python compiler may be used.  Some of the bugs with Python binaries
have been addressed but overall it's still a work in progress.
2021-09-11 22:30:37 -07:00
Justine Tunney
ad52387b74 Fix race condition in quickjs build config (#251)
qjsc.com now has a -n do nothing flag so the makefile can create a
localized binary. See also #267 where we have an exciting new change
aiming to address this particular APE gotcha.
2021-09-11 22:16:17 -07:00
agreppin
6ad0602392
quickjs: add repl + qjscalc (#251)
- Compile repl.js & qjscalc.js and include them in qjs.com zip
- Change quickjs-libc.c / js_os_poll to handle Windows console

This last change is needed because poll and select on Windows forbid
non-socket handles and Cosmopolitan Libc hasn't polyfilled that yet.
2021-09-11 16:52:49 -07:00
Paul Kulchenko
1b93066883
Add GetCookie to redbean Lua (#269) 2021-09-11 16:46:21 -07:00