This change makes some miracle modifications to the System Five system
call support, which lets us have safe, correct, and atomic handling of
thread cancellations. It all turned out to be cheaper than anticipated
because it wasn't necessary to modify the system call veneers. We were
able to encode the cancellability of each system call into the magnums
found in libc/sysv/syscalls.sh. Since cancellations are so waq, we are
also supporting a lovely Musl Libc mask feature for raising ECANCELED.
All tests pass now under WSL2. They should pass under WSL1 too, but only
WSL2 is integrated into the test fleet right now. This change also fills
in some gaps in the error numbers.
Fixes#665
- ASAN memory morgue is now lockless
- Make C11 atomics header more portable
- Rewrote pthread keys support to be lockless
- Simplify Python's unicode table unpacking code
- Make crash report write(2) closer to being atomic
- Make it possible to strace/ftrace a single thread
- ASAN now checks nul-terminated strings fast and properly
- Windows fork() now restores TLS memory of calling thread
- Invent iso8601us() for faster timestamps
- Improve --strace descriptions of sigset_t
- Rebuild the Landlock Make bootstrap binary
- Introduce MODE=sysv for non-Windows builds
- Permit OFD fcntl() locks under pledge(flock)
- redbean can now protect your kernel from ddos
- Have vfork() fallback to sys_fork() not fork()
- Change kmalloc() to not die when out of memory
- Improve documentation for some termios functions
- Rewrite putenv() and friends to conform to POSIX
- Fix linenoise + strace verbosity issue on Windows
- Fix regressions in our ability to show backtraces
- Change redbean SetHeader() to no-op if value is nil
- Improve fcntl() so SQLite locks work in non-WAL mode
- Remove some unnecessary work during fork() on Windows
- Create redbean-based SSL reverse proxy for IPv4 TurfWar
- Fix ape/apeinstall.sh warning when using non-bash shells
- Add ProgramTrustedIp(), and IsTrustedIp() APIs to redbean
- Support $PWD, $UID, $GID, and $EUID in command interpreter
- Introduce experimental JTqFpD APE prefix for non-Windows builds
- Invent blackhole daemon for firewalling IP addresses via UNIX named socket
- Add ProgramTokenBucket(), AcquireToken(), and CountTokens() APIs to redbean
If threads are being used, then fork() will now acquire and release and
runtime locks so that fork() may be safely used from threads. This also
makes vfork() thread safe, because pthread mutexes will do nothing when
the process is a child of vfork(). More torture tests have been written
to confirm this all works like a charm. Additionally:
- Invent hexpcpy() api
- Rename nsync_malloc_() to kmalloc()
- Complete posix named semaphore implementation
- Make pthread_create() asynchronous signal safe
- Add rm, rmdir, and touch to command interpreter builtins
- Invent sigisprecious() and modify sigset functions to use it
- Add unit tests for posix_spawn() attributes and fix its bugs
One unresolved problem is the reclaiming of *NSYNC waiter memory in the
forked child processes, within apps which have threads waiting on locks
- SQLite file locking now works on Windows
- SQLite will now use fdatasync() on non-Apple platforms
- Fix Ctrl-C handler on Windows to not crash with TLS
- Signals now work in multithreaded apps on Windows
- fcntl() will now accurately report EINVAL errors
- fcntl() now has excellent --strace logging
- Token bucket replenish now go 100x faster
- *NSYNC cancellations now work on Windows
- Support closefrom() on NetBSD
This change addresses various open source compatibility issues, so that
we pass 313/411 of the tests in https://github.com/jart/libc-test where
earlier today we were passing about 30/411 of them, due to header toil.
Please note that Glibc only passes 341/411 so 313 today is pretty good!
- Make the conformance of libc/isystem/ headers nearly perfect
- Import more of the remaining math library routines from Musl
- Fix inconsistencies with type signatures of calls like umask
- Write tests for getpriority/setpriority which work great now
- conform to `struct sockaddr *` on remaining socket functions
- Import a bunch of uninteresting stdlib functions e.g. rand48
- Introduce readdir_r, scandir, pthread_kill, sigsetjmp, etc..
Follow the instructions in our `tool/scripts/cosmocc` toolchain to run
these tests yourself. You use `make CC=cosmocc` on the test repository
You can now do things like implement mutexes using futexes in your
redbean lua code. This provides the fastest possible inter-process
communication for your production systems when SQLite alone as ipc
or things like pipes aren't sufficient.
We need to make sure no existing mappings exist between the
MAP_GROWSDOWN page and the guard page, since otherwise it's
not going to be able to grow down thus causing difficult to
troubleshoot failures.
This change also found a few POSIX compliance bugs with errnos. Another
bug was discovered where, on Windows, pread() and pwrite() could modify
the file position in cases where ReadFile() returned an error e.g. when
seeking past the end of file. We also have more tests!
This change improves copy_file_range(), sendfile(), splice(), openpty(),
closefrom(), close_range(), fadvise() and posix_fadvise() in addition to
writing tests that confirm things like errno and seeking behavior across
platforms. We now less aggressively polyfill behavior with some of these
functions when the platform support isn't available. Please see:
https://justine.lol/cosmopolitan/functions.html
Doing this makes binaries tinier, since we don't need to have all the
extra code for supporting a 32-bit address space. It also benefits us
because we're able to use WIN32 futexes, which makes locking simpler.
b69f3d2488 is what officially ended our
Windows 7 support. This change is merely a formalization. You can use
old versions of Cosmo now and forevermore if you need Windows 7 since
our repository is hermetic and vendors all its dependencies.
Won't fix#617
- Fix preadv() and pwritev() for old distros
- Introduce _npassert() and _unassert() macros
- Prove that file locks work properly on Windows
- Support fcntl(F_DUPFD_CLOEXEC) on more systems
640 bytes for old kDos2Errno table
182 bytes for new kDos2Errno under hello2.com (MODE=fastbuild)
122 bytes for new kDos2Errno under hello2.com (MODE=tiny)
This makes breaking changes to add underscores to many non-standard
function names provided by the c library. MODE=tiny is now tinier and
we now use smaller locks that are better for tiny apps in this mode.
Some headers have been renamed to be in the same folder as the build
package, so it'll be easier to know which build dependency is needed.
Certain old misguided interfaces have been removed. Intel intrinsics
headers are now listed in libc/isystem (but not in the amalgamation)
to help further improve open source compatibility. Header complexity
has also been reduced. Lastly, more shell scripts are now available.
The organization of the source files is now much more rational.
Old experiments that didn't work out are now deleted. Naming of
things like files is now more intuitive.
This change tunes the default stack size for the outside world to 8mb
while at the same time, reducing Cosmopolitan's default stack size to
64kb. You can override the stack size using STATIC_STACK_SIZE(). Your
build scripts should point to o//ape/public/ape.lds
This change also fixes the definition of SOMAXCONN and removes AF_RDS
since it's not polyfilled and Python 3.11 complained.
It now works most excellently across all supported operating
sytsems (earlier it didn't work on NT and XNU). Demo code is
available in examples/clock.c and this change also adds some
of the newer ANSI C time functions like timespec_get(), plus
timespec_getres() which hasn't even come out yet as it's C23
This change fixes a nasty bug where SIG_IGN and SIG_DFL weren't working
as advertised on BSDs. This change also fixes the tkill() definition on
MacOS so it maps to __pthread_kill().
This change introduces the nointernet() function which may be called to
prevent a process and its descendants from communicating with publicly
routable Internet addresses. GNU Make has been modified to always call
this function. In the future Landlock Make will have a way to whitelist
subnets to override this behavior, or disable it entirely. Support is
available for Linux only. Our firewall does not require root access.
Calling nointernet() will return control to the caller inside a new
process that has a SECCOMP BPF filter installed, which traps network
related system calls. Your original process then becomes a permanent
ptrace() supervisor that monitors all processes and threads descending
from the returned child. Whenever a networking system call happens the
kernel will stop the process and wakes up the monitor, which then peeks
into the child memory to read the sockaddr_in to determine if it's ok.
The downside to doing this is that there can be only one supervisor at a
time using ptrace() on a process. So this firewall won't be enabled if
you run make under strace or inside gdb. It also makes testing tricky.
- 10.5% reduction of o//depend dependency graph
- 8.8% reduction in latency of make command
- Fix issue with temporary file cleanup
There's a new -w option in compile.com that turns off the recent
Landlock output path workaround for "good commands" which do not
unlink() the output file like GNU tooling does.
Our new GNU Make unveil sandboxing appears to have zero overhead
in the grand scheme of things. Full builds are pretty fast since
the only thing that's actually slowed us down is probably libcxx
make -j16 MODE=rel
RL: took 85,732,063µs wall time
RL: ballooned to 323,612kb in size
RL: needed 828,560,521µs cpu (11% kernel)
RL: caused 39,080,670 page faults (99% memcpy)
RL: 350,073 context switches (72% consensual)
RL: performed 0 reads and 11,494,960 write i/o operations
pledge() and unveil() no longer consider ENOSYS to be an error.
These functions have also been added to Python's cosmo module.
This change also removes some WIN32 APIs and System Five magnums
which we're not using and it's doubtful anyone else would be too
- We now kill the program on violations like OpenBSD
- We now print a message explaining which promise is needed
- This change also fixes a linkage bug with thread local storage
- Your sigaction() handlers should now be more thread safe
A new `__pledge_mode` global has been introduced to make pledge() more
customizable on Linux. For example:
__attribute__((__constructor__)) static void init(void) {
__pledge_mode = SECCOMP_RET_ERRNO | EPERM;
}
Can be used to restore our old permissive pledge() behavior.
- Make memmem() faster
- Make readdir() thread safe
- Remove 64kb limit from mkdeps.com
- Add old crypt() function from Musl
- Improve new fix-third-party.py tool
- Improve libc/isystem/ headers and fix bugs
The whole repository is now buildable with GNU Make Landlock sandboxing.
This proves that no Makefile targets exist which touch files other than
their declared prerequisites. In order to do this, we had to:
1. Stop code morphing GCC output in package.com and instead run a
newly introduced FIXUPOBJ.COM command after GCC invocations.
2. Disable all the crumby Python unit tests that do things like create
files in the current directory, or rename() files between folders.
This ended up being a lot of tests, but most of them are still ok.
3. Introduce an .UNSANDBOXED variable to GNU Make to disable Landlock.
We currently only do this for things like `make tags`.
4. This change deletes some GNU Make code that was preventing the
execve() optimization from working. This means it should no longer
be necessary in most cases for command invocations to be indirected
through the cocmd interpreter.
5. Missing dependencies had to be declared in certain places, in cases
where they couldn't be automatically determined by MKDEPS.COM
6. The libcxx header situation has finally been tamed. One of the
things that makes this difficult is MKDEPS.COM only wants to
consider the first 64kb of a file, in order to go fast. But libcxx
likes to have #include lines buried after huge documentation.
7. An .UNVEIL variable has been introduced to GNU Make just in case
we ever wish to explicitly specify additional things that need to
be whitelisted which aren't strictly prerequisites. This works in
a manner similar to the recently introduced .EXTRA_PREREQS feature.
There's now a new build/bootstrap/make.com prebuilt binary available. It
should no longer be possible to write invalid Makefile code.
This change also fixes a bug with gettid() being incorrect after fork().
We now implement the ENOENT behavior for getauxval(). The getuid() etc.
system calls are now faster too. Plus issetugid() will work on BSDs.
- Fix getpriority()
- Add AT_MINSIGSTKSZ
- Fix bugs in BPF code
- Show more stuff in printargs.com
- Write manual test for pledge.com
- pledge() now generates tinier BPF code
- Have pledge("exec") only enable execve()
- Fix pledge.com chroot setuid functionality
- Improve pledge.com unveiling of ape loader
This change fixes bugs, adds more system calls, and improves
compatibility with OpenBSD. Going forward, versions on the web will be
pinned to a permanent version. There were many other changes over the
last week which also improved this new release.
- Introduce path module to redbean
- Fix glitch with linenoise printing extra line on eof
- Introduce closefrom() and close_range() system calls
- Make file descriptor closing more secure in pledge.com
This change reconciles our pledge() implementation with the OpenBSD
kernel source code. We now a polyfill that's much closer to OpenBSD's
behavior. For example, it was discovered that "stdio" permits threads.
There were a bunch of Linux system calls that needed to be added, like
sched_yield(). The exec / execnative category division is now dropped.
We're instead using OpenBSD's "prot_exec" promise for launching APE
binaries and dynamic shared objects. We also now filter clone() flags.
The pledge.com command has been greatly improved. It now does unveiling
by default when Landlock is available. It's now smart enough to unveil a
superset of paths that OpenBSD automatically unveils with pledge(), such
as /etc/localtime. pledge.com also now checks if the executable being
launched is a dynamic shared object, in which case it unveils libraries.
These changes now make it possible to pledge curl on ubuntu 20.04 glibc:
pledge.com -p 'stdio rpath prot_exec inet dns tty sendfd recvfd' \
curl -s https://justine.lol/hello.txt
Here's what pledging curl on Alpine 3.16 with Musl Libc looks like:
pledge.com -p 'stdio rpath prot_exec dns inet' \
curl -s https://justine.lol/hello.txt
Here's what pledging curl.com w/ ape loader looks like:
pledge.com -p 'stdio rpath prot_exec dns inet' \
o//examples/curl.com https://justine.lol/hello.txt
The most secure sandbox, is curl.com converted to static ELF:
o//tool/build/assimilate.com o//examples/curl.com
pledge.com -p 'stdio rpath dns inet' \
o//examples/curl.com https://justine.lol/hello.txt
A weird corner case needed to be handled when resolving symbolic links
during the unveiling process, that's arguably a Landlock bug. It's not
surprising since Musl and Glibc are also inconsistent here too.
We had previously not enabled TLS in MODE=tiny in order to keep the
smallest example programs (e.g. life.com) just 16kb in size. But it
was error prone doing that, so now we just always enable it because
this change uses hacks to ensure it won't increase life.com's size.
This change also fixes a bug on NetBSD, where signal handlers would
break thread local storage if SA_SIGINFO was being used. This looks
like it might be a bug in NetBSD, but it's got a simple workaround.
The pledge.com command now supports the new [WIP] unveil() support. For
example, to strongly sandbox our command for listing directories.
o//tool/build/assimilate.com o//examples/ls.com
pledge.com -v /etc -p 'stdio rpath' o//examples/ls.com /etc
This file system sandboxing is going to be perfect for us, because APE
binaries are self-contained static executables that really don't use the
filesystem that much. On the other hand, with non-static executables,
sandboxing is going to be more difficult. For example, here's how to
sandbox the `ls` command on the latest Alpine:
pledge.com -v rx:/lib -v /usr/lib -v /etc -p 'stdio rpath exec' ls /etc
This change fixes the `execpromises` API with pledge().
This change also adds unix.unveil() to redbean.
Fixes#494
This change simplifies the thread-local storage support code. On Windows
and Mac OS X the startup latency of __enable_tls() has been reduced from
30ms to 1ms. On Windows, TLS memory accesses will now go much faster due
to better self-modifying code that prevents a function call and acquires
our thread information block pointer in a single instruction.
We now rewrite the binary image at runtime on Windows and XNU to change
mov %fs:0,%reg instructions to use %gs instead. There's also simpler
threading API introduced by this change and it's called _spawn() and
_join(), which has replaced most clone() usage.
It's never worked very well having nesemu1.com and printvideo.com
spawning an ffmpeg or sox subprocess and streaming audio samples via
pipes. Since these programs don't work very well for that purpose, and
if you're SSH'ing into the cloud, the speaker could be very far away.
This change is part of an experiment to instead patch desktop terminals
such as PuTTY, KiTTY, gnome-terminal, etc. to support receiving inband
audio samples as ANSI code, and then playing them on the speakers of the
local machine that's being used. This way we can use printf() as a cross
platform audio playback library.
- Wrap clock_getres()
- Wrap sched_setscheduler()
- Make sleep() api conformant
- Polyfill sleep() using select()
- Improve clock_gettime() polyfill
- Make nanosleep() POSIX conformant
- Slightly improve some DNS functions
- Further strengthen pledge() sandboxing
- Improve rounding of timeval / timespec
- Allow layering of pledge() calls on Linux
- Polyfill sched_yield() using select() on XNU
- Delete more system constants we probably don't need
This will help make it easier to troubleshoot ABI breakages with on
operating systems that, unlike Linux don't have ironclad guarantees
to not break userspace.
We were using the Mach system call swtch() earlier. It's possible Apple
removed this system call in their recent 12.4 upgrade. We're better off
using x86 PAUSE here, since Mach is less public than the UNIX syscalls.
See #426
- Fix Makefile flaking due to ZIPOBJ_FLAGS generation
- Make printf() floating point and gdtoa thread safe
- Polish up the runit / runitd programs some more
- Prune some more makefile dependencies
- Finish cleaning up the stdio unlocked APIs
- Make __cxa_finalize() properly thread safe
- Don't log locks if threads aren't being used
- Add some more mutex guards to places using _mmi
- Specific lock names now appear in the --ftrace logs
- Fix mkdeps.com generating invalid Makefiles sometimes
- Simplify and fix bugs in the test runner infrastructure
- Fix issue where sometimes some functions wouldn't be logged
- Write tests for cthreads
- Fix bugs in pe2.com tool
- Fix ASAN issue with GetDosEnviron()
- Consolidate the cthread header files
- Some code size optimizations for MODE=
- Attempted to squash a tls linker warning
- Attempted to get futexes working on FreeBSD
- Document redbean's argon2 module
- Fix regressions in cthreads library
- Make testlib work better with threads
- Give the cthreads library lots of love
- Remove some of the stdio assembly code
- Implement getloadavg() across platforms
- Code size optimizations for errnos, etc.
- Only check for signals in main thread on Windows
- Make errnos for dup2 / dup3 consistent with posix
This change also fixes a bug in the argon2 module, where the NUL
terminator was being included in the hash encoded ascii string. This
shouldn't require any database migrations to folks who found this module
and productionized it, since the argon2 library treats it as a c string.
- Implement openpty()
- Add `--assimilate` flag to APE bootloader
- Restore Linux vDSO clock_gettime() support
- Use `$(APE_NO_MODIFY_SELF)` on more programs
- Add FreeBSD-specific mmap() flags
- Reduce size of the APE loader from 8kb to 4kb
- Work towards fixing the Makefile build on WSL
- Automate testing of APE no-modify-self behaviors
- Make the ape.S shell script code cleaner and tinier
- Improve the APE sanity check to test behavior better
- Fixed issue with ShowCrashReports() sigaltstack() on BSDs
- Delete symbols for S_MODE magnums which wasted compile time
If you checked out yesterday's APE commit, please run:
rm -f /usr/bin/ape o/tmp/ape /tmp/ape "${TMPDIR:-/tmp}/ape"
Because this change fixes certain aspects of the new ABI. We don't have
automated migrations for APE loader versions yet. Thanks! You can also
download prebuilt binaries here:
- https://justine.lol/ape.elf (Linux/FreeBSD/NetBSD/OpenBSD)
- https://justine.lol/ape.macho (Apple)
Install the appropriate one as `/usr/bin/ape`.
This change fixes a nasty regression caused by
80b211e314 which deadlocked.
This change also causes MbedTLS to prefer the ChaCha ciphersuite on
older CPUs that don't have AES hardware instructions.
- add vdso dump utility
- tests now log stack usage
- rename g_ftrace to __ftrace
- make internal spinlocks go faster
- add conformant c11 atomics library
- function tracing now logs stack usage
- make function call tracing thread safe
- add -X unsecure (no ssl) mode to redbean
- munmap() has more consistent behavior now
- pacify fsync() calls on python unit tests
- make --strace flag work better in redbean
- start minimizing and documenting compiler flags
Windows support for this example is still a work in progress. It's
encountering some unusual crashes. Thank you Chris Wellons for the cool
synchronization code too!
This change introduces a `-W /dev/pts/1` flag to redbean. What it does
is use the mincore() system call to create a dual-screen terminal
display that lets you troubleshoot the virtual address space. This is
useful since page faults are an important thing to consider when using a
forking web server. Now we have a colorful visualization of which pages
are going to fault and which ones are resident in memory.
The memory monitor, if enabled, spawns as a thread that just outputs
ANSI codes to the second terminal in a loop. In order to make this
happen using the new clone() polyfill, stdio is now thread safe.
This change also introduces some new demo pages to redbean. It also
polishes the demos we already have, to look a bit nicer and more
presentable for the upcoming release, with better explanations too.
- Get threads working on NetBSD
- Get threads working on OpenBSD
- Fix Emacs config for Emacs v28
- Improve --strace logging of sigset_t
- Improve --strace logging of struct stat
- Improve memory safety of DescribeThing functions
- Refactor auto stack allocation into LIBC_RUNTIME
- Introduce shell.com example which works on Windows
- Refactor __strace_thing into DescribeThing functions
- Document the CHECK macros and improve them in NDEBUG mode
- Rewrite MAP_STACK so it uses FreeBSD behavior across platforms
- Deprecate and discourage the use of MAP_GROWSDOWN (it's weird)
- Add rusage to redbean Lua API
- Add more redbean documentation
- Add pledge() to redbean Lua API
- Polyfill OpenBSD pledge() for Linux
- Increase PATH_MAX limit to 1024 characters
- Untrack sibling processes after fork() on Windows
- Add hierarchical auto-completion to redbean's repl
- Fetch latest localtime() and strftime() from Eggert
- Shave a few milliseconds off redbean start latency
- Fix redbean repl with multi-line statements
- Make the Lua unix module code more elegant
- Harden Lua data structure serialization
- Add GetCpuCount() API to redbean
- Add unix.gmtime() API to redbean
- Add unix.readlink() API to redbean
- Add unix.localtime() API to redbean
- Perfect the new redbean UNIX module APIs
- Integrate with Linux clock_gettime() vDSO
- Run Lua garbage collector when malloc() fails
- Fix another regression quirk with linenoise repl
- Fix GetProgramExecutableName() for systemwide installs
- Fix a build flake with test/libc/mem/test.mk SRCS list
- Improve serialization
- Add Benchmark() API to redbean
- Refactor UNIX API to be assert() friendly
- Make the redbean Lua REPL print data structures
- Fix recent regressions in linenoise reverse search
- Add -i flag so redbean can be a language interpreter
- Expand redbean UNIX module
- Expand redbean documentation
- Ensure Lua copyright is embedded in binary
- Increase the PATH_MAX limit especially on NT
- Use column major sorting for linenoise completions
- Fix some suboptimalities in redbean's new UNIX API
- Figured out right flags for Multics newline in raw mode
- Get clone() working on FreeBSD
- Increase some Python build quotas
- Add more atomic builtins to chibicc
- Fix ASAN poisoning of alloca() memory
- Make MODE= mandatory link path tinier
- Improve the examples folder a little bit
- Start working on some more resource limits
- Make the linenoise auto-complete UI as good as GNU readline
- Update compile.com, avoiding AVX codegen on non-AVX systems
- Make sure empty path to syscalls like opendir raises ENOENT
- Correctly polyfill ENOENT vs. ENOTDIR on the New Technology
- Port bestline's paredit features to //third_party/linenoise
- Remove workarounds for RHEL 5.0 bugs that were fixed in 5.1
It's now possible to pass the `-S` or `-SS` flags to sandbox redbean
worker proecsses after they've been forked. The first `-S` flag is
intended to be a permissive builtin policy that limits system calls to
only that which the various parts of redbean serving need. The second
`-SS` flag is intended to be more restrictive, preventing things like
the Lua extensions you download off the web from using the HTTP client
or sockets APIs. In upcoming changes you'll be able to implement your
own Berkeley Packet Filter sandbox programs and load them via Lua.
- Fix a regression with the previous change that broke redbean
- Add chroot(), resource limit, seccomp, and other stuff to redbean
- Write lots and lots of documentation
- Iron out more system call issues
This change makes further effort towards improving our poll()
implementation on the New Technology. The stdin worker didn't work out
so well for Python so it's not being used for now. System call tracing
with the --strace flag should now be less noisy now on Windows unless
you modify the strace.internal.h defines to turn on some optional ones
that are most useful for debugging the system call wrappers.
- Fix bugs in kDos2Errno definition
- malloc() should now be thread safe
- Fix bug in rollup.com header generator
- Fix open(O_APPEND) on the New Technology
- Fix select() on the New Technology and test it
- Work towards refactoring i/o for thread safety
- Socket reads and writes on NT now poll for signals
- Work towards i/o completion ports on the New Technology
- Make read() and write() intermittently check for signals
- Blinkenlights keyboard i/o so much better on NT w/ poll()
- You can now poll() files and sockets at the same time on NT
- Fix bug in appendr() that manifests with dlmalloc footers off
You can now call functions like fork() from Lua and it'll work across
all supported platforms, including Windows. This gives you a level of
control of the system that Lua traditionally hasn't been able to have
due to its focus on old portable stdio rather modern POSIX APIs. Demo
code has been added to redbean-demo.com to show how it works.
This change also modifies Lua so that integer literals with a leading
zero will be interpreted as octal. That should help avoid shooting in
the foot with POSIX APIs that frequently use octal mode bits.
This change fixes a bug in opendir(".") on New Technology.
Lastly, redbean will now serve crash reports to private network IPs.
This is consistent with other frameworks. However that isn't served
to public IPs unless the -E flag is passed to redbean at startup.
- Document sigaction()
- Simplify New Technology fork() code
- Testing and many bug fixes for mprotect()
- Distribute Intel Xed ILD in the amalgamation
- Turn Xed enums into defines to avoid DWARF bloat
- Improve polyfilling of SA_SIGINFO on BSDs and fix bugs
- setpgid(getpid(), getpid()) on Windows will ignore CTRL-C
- Work around issues relating to NT mappings being executable
- Permit automatic executable stack override via `ape_stack_pf`
- Introduce fast spinlock API
- Double rand64() perf w/ spinlock
- Improve raise() on New Technology
- Support gettid() across platforms
- Implement SA_NODEFER on New Technology
- Move the lock intrinsics into LIBC_INTRIN
- Make SIGTRAP recoverable on New Technology
- Block SIGCHLD in wait4() on New Technology
- Add threading prototypes for XNU and FreeBSD
- Rewrite abort() fixing its minor bugs on XNU/NT
- Shave down a lot of the content in libc/bits/bits.h
- Let signal handlers modify CPU registers on New Technology
Some of these are from userspace APIs and therefore don't need to be in
consts.sh. Others are poorly supported, rarely used, niche hardware not
in scope for support any time soon. Ideally, we should only have listed
the stuff that's supported really well. This is a conservative deletion
- Improve i/o perf on New Technology
- Code cleanup on read() for New Technology
- Fix bad bug with dup() of socket on New Technology
- Clean up some more strace errors on New Technology
- Fix sigsuspend() on XNU
- Fix strsignal() on non-Linux
- Add unit tests for strsignal()
- Add unit tests for setitimer()
- Add unit tests for sigsuspend()
- Rewrite setitimer() for New Technology
- Rewrite nanosleep() for New Technology
- Polyfill SIGALRM on the New Technology
- select(0,0,0,0) on NT now calls pause()
- Remove some NTDLL calls that aren't needed
- Polyfill SA_NOCLDWAIT on the New Technology
- Polyfill SA_RESETHAND on the New Technology
- Polyfill sigprocmask() on the New Technology
- Polyfill SIGCHLD+SIG_IGN on the New Technology
- Polyfill SA_RESTART masking on the New Technology
- Deliver console signals from main thread on New Technology
- Document SA_RESTART behavior w/ @sarestartable / @norestart
- System call trace in MODE=dbg now prints inherited FDs and signal mask
This change fixes minor bugs and adds a feature, which lets us store the
ELF symbol table, inside the ZIP directory. We use the path /zip/.symtab
which can be safely removed using a zip editing tool, to make the binary
smaller after compilation. This supplements the existing method of using
a separate .com.dbg file, which is still supported. The intent is people
don't always know that it's a good idea to download the debug file. It's
not great having someone's first experience be a crash report, that only
has numbers rather than symbols. This will help fix that!
Now that we have understandable system call tracing on Windows, this
change rewrites many of the polyfill internals for that platform, to
help things get closer to tip top shape. Support for complex forking
scenarios had been in a regressed state for quite some time. Now, it
works! Subsequent changes should be able to address the performance.
This is similar to the --ftrace (c function call trace) flag, except
it's less noisy since it only logs system calls to stderr. Having this
flag is valuable because (1) system call tracing tells us a lot about
the behavior of complex programs and (2) it's usually very hard to get
system call tracing on various operating systems, e.g. strace, ktrace,
dtruss, truss, nttrace, etc. Especially on Apple platforms where even
with the special boot trick, debuggers still aren't guaranteed to work.
make -j8 o//examples
o//examples/hello.com --strace
This is enabled by default in MODE=, MODE=opt, and MODE=dbg. In MODE=dbg
extra information will be printed.
make -j8 MODE=dbg o/dbg/examples
o/dbg/examples/hello.com --strace |& less
This change also changes:
- Rename IsText() → _istext()
- Rename IsUtf8() → _isutf8()
- Fix madvise() on Windows NT
- Fix empty string case of inet_ntop()
- vfork() wrapper now saves and restores errno
- Update xsigaction() to yoink syscall support
- Fix build flakes
- Polyfill SIGWINCH on Windows
- Fix an execve issue on Windows
- Make strerror show more information
- Improve cmd.exe setup/teardown on Windows
- Support bracketed paste mode in Blinkenlights
- Show keyboard shortcuts in Blinkenlights status bar
- Fixed copy_file_range() and copyfile() w/ zip filesystem
- Size optimize GetDosArgv() to keep life.com 12kb in size
- Improve Blinkenlights ability to load weird ELF executables
- Fix program_executable_name and add GetInterpreterExecutableName
- Make Python in tiny mode fail better if docstrings are requested
- Update Python test exclusions in tiny* modes such as tinylinux
- Add bulletproof unbreakable kprintf() troubleshooting function
- Remove "oldskool" keyword from ape.S for virus scanners
- Fix issue that caused backtraces to not print sometimes
- Improve Blinkenlights serial uart character i/o
- Make clock_gettime() not clobber errno on xnu
- Improve sha256 cpuid check for old computers
- Integrate some bestline linenoise fixes
- Show runit process names better in htop
- Remove SIGPIPE from ShowCrashReports()
- Make realpath() not clobber errno
- Avoid attaching GDB on non-Linux
- Improve img.com example
This commit makes numerous refinements to cosmopolitan memory handling.
The default stack size has been reduced from 2mb to 128kb. A new macro
is now provided so you can easily reconfigure the stack size to be any
value you want. Work around the breaking change by adding to your main:
STATIC_STACK_SIZE(0x00200000); // 2mb stack
If you're not sure how much stack you need, then you can use:
STATIC_YOINK("stack_usage_logging");
After which you can `sort -nr o/$MODE/stack.log`. Based on the unit test
suite, nothing in the Cosmopolitan repository (except for Python) needs
a stack size greater than 30kb. There are also new macros for detecting
the size and address of the stack at runtime, e.g. GetStackAddr(). We
also now support sigaltstack() so if you want to see nice looking crash
reports whenever a stack overflow happens, you can put this in main():
ShowCrashReports();
Under `make MODE=dbg` and `make MODE=asan` the unit testing framework
will now automatically print backtraces of memory allocations when
things like memory leaks happen. Bugs are now fixed in ASAN global
variable overrun detection. The memtrack and asan runtimes also handle
edge cases now. The new tools helped to identify a few memory leaks,
which are fixed by this change.
This change should fix an issue reported in #288 with ARG_MAX limits.
Fixing this doubled the performance of MKDEPS.COM and AR.COM yet again.
The APE_NO_MODIFY_SELF loader payload has been moved out of the examples
folder and improved so that it works on BSD systems, and permits general
elf program headers. This brings its quality up enough that it should be
acceptable to use by default for many programs, e.g. Python, Lua, SQLite
and Python. It's the responsibility of the user to define an appropriate
TMPDIR if /tmp is considered an adversarial environment. Mac OS shall be
supported by APE_NO_MODIFY_SELF soon.
Fixes and improvements have been made to program_executable_name as it's
now the one true way to get the absolute path of the executing image.
This change fixes a memory leak in linenoise history loading, introduced
by performance optimizations in 51904e2687
This change fixes a longstanding regression with Mach system calls, that
23ae9dfceb back in February which impacted
our sched_yield() implementation, which is why no one noticed until now.
The Blinkenlights PC emulator has been improved. We now fix rendering on
XNU and BSD by not making the assumption that the kernel terminal driver
understands UTF8 since that seems to break its internal modeling of \r\n
which is now being addressed by using \e[𝑦H instead. The paneling is now
more compact in real mode so you won't need to make your font as tiny if
you're only emulating an 8086 program. The CLMUL ISA is now emulated too
This change also makes improvement to time. CLOCK_MONOTONIC now does the
right thing on Windows NT. The nanosecond time module functions added in
Python 3.7 have been backported.
This change doubles the performance of Argon2 password stretching simply
by not using its copy_block and xor_block helper functions, as they were
trivial to inline thus resulting in us needing to iterate over each 1024
byte block four fewer times.
This change makes code size improvements. _PyUnicode_ToNumeric() was 64k
in size and now it's 10k. The CJK codec lookup tables now use lazy delta
zigzag deflate (δzd) encoding which reduces their size from 600k to 200k
plus the code bloat caused by macro abuse in _decimal.c is now addressed
so our fully-loaded statically-linked hermetically-sealed Python virtual
interpreter container is now 9.4 megs in the default build mode and 5.5m
in MODE=tiny which leaves plenty of room for chibicc.
The pydoc web server now accommodates the use case of people who work by
SSH'ing into a different machine w/ python.com -m pydoc -p8080 -h0.0.0.0
Finally Python Capsulae delenda est and won't be supported in the future
- Python static hello world now 1.8mb
- Python static fully loaded now 10mb
- Python HTTPS client now uses MbedTLS
- Python REPL now completes import stmts
- Increase stack size for Python for now
- Begin synthesizing posixpath and ntpath
- Restore Python \N{UNICODE NAME} support
- Restore Python NFKD symbol normalization
- Add optimized code path for Intel SHA-NI
- Get more Python unit tests passing faster
- Get Python help() pagination working on NT
- Python hashlib now supports MbedTLS PBKDF2
- Make memcpy/memmove/memcmp/bcmp/etc. faster
- Add Mersenne Twister and Vigna to LIBC_RAND
- Provide privileged __printf() for error code
- Fix zipos opendir() so that it reports ENOTDIR
- Add basic chmod() implementation for Windows NT
- Add Cosmo's best functions to Python cosmo module
- Pin function trace indent depth to that of caller
- Show memory diagram on invalid access in MODE=dbg
- Differentiate stack overflow on crash in MODE=dbg
- Add stb_truetype and tools for analyzing font files
- Upgrade to UNICODE 13 and reduce its binary footprint
- COMPILE.COM now logs resource usage of build commands
- Start implementing basic poll() support on bare metal
- Set getauxval(AT_EXECFN) to GetModuleFileName() on NT
- Add descriptions to strerror() in non-TINY build modes
- Add COUNTBRANCH() macro to help with micro-optimizations
- Make error / backtrace / asan / memory code more unbreakable
- Add fast perfect C implementation of μ-Law and a-Law audio codecs
- Make strtol() functions consistent with other libc implementations
- Improve Linenoise implementation (see also github.com/jart/bestline)
- COMPILE.COM now suppresses stdout/stderr of successful build commands
