Our makefile generator now accepts badly formatted include lines. It's
now more hermetic with better error checking in the cosmo repo, and it
can be configured to not be hermetic at all.
Every program built using Cosmopolitan is statically-linked. However
there are some cases, e.g. GUIs and video drivers, where linking the
host platform libraries is desirable. So what we do in such cases is
launch a stub executable using the host platform's libc, and longjmp
back into this executable. The stub executable passes back to us the
platform-specific dlopen() implementation, which we shall then wrap.
Here's the list of platforms that are supported so far:
- x86-64 Linux w/ Glibc
- x86-64 Linux w/ Musl Libc
- x86-64 FreeBSD
- x86-64 Windows
- aarch64 Linux w/ Glibc
- aarch64 MacOS
What this means is your Cosmo programs can call foreign functions on
your host operating system. However, it's important to note that any
foreign library you link won't have the ability to call functions in
your Cosmopolitan program. For example it's now technically possible
that Lua can load a module, however that almost certainly won't work
since the Lua module won't have access to Cosmo's Lua API.
Kudos to @jacereda for figuring out how to do this.
- We now serialize the file descriptor table when spawning / executing
processes on Windows. This means you can now inherit more stuff than
just standard i/o. It's needed by bash, which duplicates the console
to file descriptor #255. We also now do a better job serializing the
environment variables, so you're less likely to encounter E2BIG when
using your bash shell. We also no longer coerce environ to uppercase
- execve() on Windows now remotely controls its parent process to make
them spawn a replacement for itself. Then it'll be able to terminate
immediately once the spawn succeeds, without having to linger around
for the lifetime as a shell process for proxying the exit code. When
process worker thread running in the parent sees the child die, it's
given a handle to the new child, to replace it in the process table.
- execve() and posix_spawn() on Windows will now provide CreateProcess
an explicit handle list. This allows us to remove handle locks which
enables better fork/spawn concurrency, with seriously correct thread
safety. Other codebases like Go use the same technique. On the other
hand fork() still favors the conventional WIN32 inheritence approach
which can be a little bit messy, but is *controlled* by guaranteeing
perfectly clean slates at both the spawning and execution boundaries
- sigset_t is now 64 bits. Having it be 128 bits was a mistake because
there's no reason to use that and it's only supported by FreeBSD. By
using the system word size, signal mask manipulation on Windows goes
very fast. Furthermore @asyncsignalsafe funcs have been rewritten on
Windows to take advantage of signal masking, now that it's much more
pleasant to use.
- All the overlapped i/o code on Windows has been rewritten for pretty
good signal and cancelation safety. We're now able to ensure overlap
data structures are cleaned up so long as you don't longjmp() out of
out of a signal handler that interrupted an i/o operation. Latencies
are also improved thanks to the removal of lots of "busy wait" code.
Waits should be optimal for everything except poll(), which shall be
the last and final demon we slay in the win32 i/o horror show.
- getrusage() on Windows is now able to report RUSAGE_CHILDREN as well
as RUSAGE_SELF, thanks to aggregation in the process manager thread.
- Every unit test now passes on Apple Silicon. The final piece of this
puzzle was porting our POSIX threads cancelation support, since that
works differently on ARM64 XNU vs. AMD64. Our semaphore support on
Apple Silicon is also superior now compared to AMD64, thanks to the
grand central dispatch library which lets *NSYNC locks go faster.
- The Cosmopolitan runtime is now more stable, particularly on Windows.
To do this, thread local storage is mandatory at all runtime levels,
and the innermost packages of the C library is no longer being built
using ASAN. TLS is being bootstrapped with a 128-byte TIB during the
process startup phase, and then later on the runtime re-allocates it
either statically or dynamically to support code using _Thread_local.
fork() and execve() now do a better job cooperating with threads. We
can now check how much stack memory is left in the process or thread
when functions like kprintf() / execve() etc. call alloca(), so that
ENOMEM can be raised, reduce a buffer size, or just print a warning.
- POSIX signal emulation is now implemented the same way kernels do it
with pthread_kill() and raise(). Any thread can interrupt any other
thread, regardless of what it's doing. If it's blocked on read/write
then the killer thread will cancel its i/o operation so that EINTR can
be returned in the mark thread immediately. If it's doing a tight CPU
bound operation, then that's also interrupted by the signal delivery.
Signal delivery works now by suspending a thread and pushing context
data structures onto its stack, and redirecting its execution to a
trampoline function, which calls SetThreadContext(GetCurrentThread())
when it's done.
- We're now doing a better job managing locks and handles. On NetBSD we
now close semaphore file descriptors in forked children. Semaphores on
Windows can now be canceled immediately, which means mutexes/condition
variables will now go faster. Apple Silicon semaphores can be canceled
too. We're now using Apple's pthread_yield() funciton. Apple _nocancel
syscalls are now used on XNU when appropriate to ensure pthread_cancel
requests aren't lost. The MbedTLS library has been updated to support
POSIX thread cancelations. See tool/build/runitd.c for an example of
how it can be used for production multi-threaded tls servers. Handles
on Windows now leak less often across processes. All i/o operations on
Windows are now overlapped, which means file pointers can no longer be
inherited across dup() and fork() for the time being.
- We now spawn a thread on Windows to deliver SIGCHLD and wakeup wait4()
which means, for example, that posix_spawn() now goes 3x faster. POSIX
spawn is also now more correct. Like Musl, it's now able to report the
failure code of execve() via a pipe although our approach favors using
shared memory to do that on systems that have a true vfork() function.
- We now spawn a thread to deliver SIGALRM to threads when setitimer()
is used. This enables the most precise wakeups the OS makes possible.
- The Cosmopolitan runtime now uses less memory. On NetBSD for example,
it turned out the kernel would actually commit the PT_GNU_STACK size
which caused RSS to be 6mb for every process. Now it's down to ~4kb.
On Apple Silicon, we reduce the mandatory upstream thread size to the
smallest possible size to reduce the memory overhead of Cosmo threads.
The examples directory has a program called greenbean which can spawn
a web server on Linux with 10,000 worker threads and have the memory
usage of the process be ~77mb. The 1024 byte overhead of POSIX-style
thread-local storage is now optional; it won't be allocated until the
pthread_setspecific/getspecific functions are called. On Windows, the
threads that get spawned which are internal to the libc implementation
use reserve rather than commit memory, which shaves a few hundred kb.
- sigaltstack() is now supported on Windows, however it's currently not
able to be used to handle stack overflows, since crash signals are
still generated by WIN32. However the crash handler will still switch
to the alt stack, which is helpful in environments with tiny threads.
- Test binaries are now smaller. Many of the mandatory dependencies of
the test runner have been removed. This ensures many programs can do a
better job only linking the the thing they're testing. This caused the
test binaries for LIBC_FMT for example, to decrease from 200kb to 50kb
- long double is no longer used in the implementation details of libc,
except in the APIs that define it. The old code that used long double
for time (instead of struct timespec) has now been thoroughly removed.
- ShowCrashReports() is now much tinier in MODE=tiny. Instead of doing
backtraces itself, it'll just print a command you can run on the shell
using our new `cosmoaddr2line` program to view the backtrace.
- Crash report signal handling now works in a much better way. Instead
of terminating the process, it now relies on SA_RESETHAND so that the
default SIG_IGN behavior can terminate the process if necessary.
- Our pledge() functionality has now been fully ported to AARCH64 Linux.
This way complex runtime features (e.g. ftrace, symbol tables) can
always yoink zipos support. This is important now that apelink.com
automates embedding symbol tables for multiple cpus.
This change has the insight that dwExitCode isn't an exit code but
rather should be used to pass the wait status. This lets us report
killing as a termination status, similar to UNIX. This change also
fixes the fact that exit(259) on Windows will break the parent due
way WIN32 is designed. We now work around that.
It turns out that NetBSD and OpenBSD, will let you have exit codes
beyond 255. This change will let you use them when it's possible.
- Fix unused local variable errors
- Remove yoinks from sigaction() header
- Add nox87 and aarch64 to github actions
- Fix cosmocc -fportcosmo in linking mode
- It's now possible to build `make m=llvm o/llvm/libc`
The *NSYNC linked list API is good enough that it deserves to be part of
the C libray, so this change writes an improved version of it which uses
that offsetof() trick from the Linux Kernel. We vendor all of the *NSYNC
tests in third_party which helped confirm the needed refactoring is safe
This change also deletes more old code that didn't pan out. My goal here
is to work towards a vision where the Cosmopolitan core libraries become
less experimental and more focused on curation. This better reflects the
current level of quality we've managed to achieve.
- Found some bugs in LLVM compiler-rt library
- The useless LIBC_STUBS package is now deleted
- Improve the overflow checking story even further
- Get chibicc tests working in MODE=dbg mode again
- The libc/isystem/ headers now have correctly named guards
This change fixes an issue with the tcflow() magic numbers that was
causing bash to freeze up on Linux. While auditing termios polyfills,
several other issues were identified with XNU/BSD compatibility.
Out of an abundance of caution this change undefines as much surface
area from libc/calls/struct/termios.h as possible, so that autoconf
scripts are less likely to detect non-POSIX teletypewriter APIs that
haven't been polyfilled by Cosmopolitan.
This is a *breaking change* for your static archives in /opt/cosmos if
you use the cosmocc toolchain. That's because this change disables the
ioctl() undiamonding trick for code outside the monorepo, specifically
because it'll lead to brittle ABI breakages like this. If you're using
the cosmocc toolchain, you'll need to rebuild libraries like ncurses,
readline, etc. Yes diamonds cause bloat. To work around that, consider
using tcgetwinsize() instead of ioctl(TIOCGWINSZ) since it'll help you
avoid pulling every single ioctl-related polyfill into the linkage.
The cosmocc script was specifying -DNDEBUG for some reason. It's fixed.
This change takes an entirely new approach to the incremental linking of
pkzip executables. The assets created by zipobj.com are now treated like
debug data. After a .com.dbg is compiled, fixupobj.com should be run, so
it can apply fixups to the offsets and move the zip directory to the end
of the file. Since debug data doesn't get objcopy'd, a new tool has been
introduced called zipcopy.com which should be run after objcopy whenever
a .com file is created. This is all automated by the `cosmocc` toolchain
which is rapidly becoming the new recommended approach.
This change also introduces the new C23 checked arithmetic macros.
This change integrates e58abc1110b335a3341e8ad5821ad8e3880d9bb2 from
https://github.com/ahgamut/musl-cross-make/ which fixes the issues we
were having with our C language extension for symbolic constants. This
change also performs some code cleanup and bug fixes to getaddrinfo().
It's now possible to compile projects like ncurses, readline and python
without needing to patch anything upstream, except maybe a line or two.
Pretty soon it should be possible to build a Linux distro on Cosmo.
This change implements a new approach to function call logging, that's
based on the GCC flag: -fpatchable-function-entry. Read the commentary
in build/config.mk to learn how it works.
This change progresses our AARCH64 support:
- The AARCH64 build and tests are now passing
- Add 128-bit floating-point support to printf()
- Fix clone() so it initializes cosmo's x28 TLS register
- Fix TLS memory layout issue with aarch64 _Alignas vars
- Revamp microbenchmarking tools so they work on aarch64
- Make some subtle improvements to aarch64 crash reporting
- Make kisdangerous() memory checks more accurate on aarch64
- Remove sys_open() since it's not available on Linux AARCH64
This change makes general improvements to Cosmo and Redbean:
- Introduce GetHostIsa() function in Redbean
- You can now feature check using pledge(0, 0)
- You can now feature check using unveil("",0)
- Refactor some more x86-specific asm comments
- Refactor and write docs for some libm functions
- Make the mmap() API behave more similar to Linux
- Fix WIFSIGNALED() which wrongly returned true for zero
- Rename some obscure cosmo keywords from noFOO to dontFOO
- Introduce epoll_pwait()
- Rewrite -ftrapv and ffs() libraries in C code
- Use more FreeBSD code in math function library
- Get significantly more tests passing on qemu-aarch64
- Fix many Musl long double functions that were broken on AARCH64
We can once again create 2mb statically-linked Python binaries:
$ make -j8 m=tiny o/tiny/examples/pyapp/pyapp.com
$ ls -hal o/tiny/examples/pyapp/pyapp.com
-rwxr-xr-x 1 jart jart 2.1M May 1 14:04 o/tiny/examples/pyapp/pyapp.com
$ o/tiny/examples/pyapp/pyapp.com
cosmopolitan is cool!
The regression was caused by Python thread support in b15f9eb58
Python threads are now generally working, however some parts of Python's
regression tests for threads are flaky. This is possibly due to needing
more locking primitives in Cosmo's IO system call wrappers, e.g. close.
make o//third_party/python/Lib/test/test_threading.py.runs
See #747
- enable WITH_THREAD and _POSIX_THREADS
- add headers everywhere
- breaks only two tests (faulthandler and signal)
- disabled terminal completion because it causes segfaults for some
reason (probably could not get the current thread)
This change restores the .symtab symbol table files in our flagship
programs (e.g. redbean.com, python.com) needed to show backtraces. This
also rolls back earlier changes to zip.com w.r.t. temp directories since
the right way to do it turned out to be the -b DIR flag.
This change also improves the performance of zip.com. It turned out
mmap() wasn't being used, because zip.com was assuming a 4096-byte
granularity, but cosmo requires 65536. There was also a chance to speed
up stdio scanning using the unlocked functions.
Landlock Make will no longer sandbox prerequisites that end with a
trailing slash. This means you can use use directory prerequisites
for detecting deleted files when using using globbing, without the
effect of unveiling the entire directory. When you do want make to
unveil directories, you can omit the trailing slash.
- Polyfill pselect() on Windows
- Add -O NOFILE flag to pledge.com
- Polyfill ppoll() on NetBSD, XNU, and Windows
- Support negative numbers and errno in sizetol()
- Add .RSS, .NOFILE, and .MAXCORE to Landlock Make
- Fix issue with .PLEDGE preventing touching of output files
- Add __watch() function (like ftrace) for logging memory changes
This change also removes the futimens() call on the Landlock Make output
file workaround, since it caused problems with commands like fixupobj
which modify-in-place. It turns out if a file is opened for writing and
then no writes actually occur, then the modified time doesn't change.
- 10.5% reduction of o//depend dependency graph
- 8.8% reduction in latency of make command
- Fix issue with temporary file cleanup
There's a new -w option in compile.com that turns off the recent
Landlock output path workaround for "good commands" which do not
unlink() the output file like GNU tooling does.
Our new GNU Make unveil sandboxing appears to have zero overhead
in the grand scheme of things. Full builds are pretty fast since
the only thing that's actually slowed us down is probably libcxx
make -j16 MODE=rel
RL: took 85,732,063µs wall time
RL: ballooned to 323,612kb in size
RL: needed 828,560,521µs cpu (11% kernel)
RL: caused 39,080,670 page faults (99% memcpy)
RL: 350,073 context switches (72% consensual)
RL: performed 0 reads and 11,494,960 write i/o operations
pledge() and unveil() no longer consider ENOSYS to be an error.
These functions have also been added to Python's cosmo module.
This change also removes some WIN32 APIs and System Five magnums
which we're not using and it's doubtful anyone else would be too
This change fixes Landlock Make so that only the output target file is
unveiled, rather than unveiling the directory that contains it. This
gives us a much stronger sandbox. It also helped identify problematic
build code in our repo that should have been using o/tmp instead.
Landlock isn't able to let us unveil files that don't exist. Even if
they do, then once a file is deleted, the sandboxing for it goes away.
This caused problems for Landlock Make because tools like GNU LD will
repeatedly delete and recreate the output file. This change uses the
compile.com wrapper to ensure on changes happen to the output inode.
New binary available on https://justine.lol/make/Fixes#528
The whole repository is now buildable with GNU Make Landlock sandboxing.
This proves that no Makefile targets exist which touch files other than
their declared prerequisites. In order to do this, we had to:
1. Stop code morphing GCC output in package.com and instead run a
newly introduced FIXUPOBJ.COM command after GCC invocations.
2. Disable all the crumby Python unit tests that do things like create
files in the current directory, or rename() files between folders.
This ended up being a lot of tests, but most of them are still ok.
3. Introduce an .UNSANDBOXED variable to GNU Make to disable Landlock.
We currently only do this for things like `make tags`.
4. This change deletes some GNU Make code that was preventing the
execve() optimization from working. This means it should no longer
be necessary in most cases for command invocations to be indirected
through the cocmd interpreter.
5. Missing dependencies had to be declared in certain places, in cases
where they couldn't be automatically determined by MKDEPS.COM
6. The libcxx header situation has finally been tamed. One of the
things that makes this difficult is MKDEPS.COM only wants to
consider the first 64kb of a file, in order to go fast. But libcxx
likes to have #include lines buried after huge documentation.
7. An .UNVEIL variable has been introduced to GNU Make just in case
we ever wish to explicitly specify additional things that need to
be whitelisted which aren't strictly prerequisites. This works in
a manner similar to the recently introduced .EXTRA_PREREQS feature.
There's now a new build/bootstrap/make.com prebuilt binary available. It
should no longer be possible to write invalid Makefile code.
- Fix Makefile flaking due to ZIPOBJ_FLAGS generation
- Make printf() floating point and gdtoa thread safe
- Polish up the runit / runitd programs some more
- Prune some more makefile dependencies
This change turns symbol table compression back on using Puff, which
noticeably reduces the size of programs like redbean and Python. The
redbean web server receives some minor API additions for controlling
things like SSL in addition to filling gaps in the documentation.
- Fix some minor issues in ar.com
- Have execve() look for `ape` command
- Rewrite NT paths using /c/ rather /??/c:/
- Replace broken GCC symlinks with .sym files
- Rewrite $PATH environment variables on startup
- Make $(APE_NO_MODIFY_SELF) the default bootloader
- Add all build command dependencies to build/bootstrap
- Get the repository mostly building from source on non-Linux
