Embedding Blink builds in Cosmo executables was a failed experiment. It
turned out to be easier than expected to let the mono repo have support
for multiple architectures. Blink still works great; it's supported and
recommended; just please use it as a separate program. For example, you
can use Blink to run Cosmo binaries on architectures like i486 / s390x.
If you install qemu-user from apt then glibc links a lot of address
space bloat that causes pthread_create() to ENOMEM (a.k.a. EAGAIN).
Boosting the virtual memory quota from 512m to 2048m will hopefully
future proof the build for the future, as Linux distros get fatter.
Please note this only applies to MODE=aarch64 on x86_64 builds when
you're using QEMU from Debian/Ubuntu rather than installing the one
cosmo provides in third_party/qemu/qemu-aarch64.gz. This change may
also be useful to people who are using the host compiler toolchain.
- `__cxa_*` runtime functions are expected to be in the `abi` namespace,
which is currently an alias for `__cxxabiv1`.
- Rely on the header provided by `libcxxabi` for functions that we do
not implement ourselves anymore.
This will help C++ code that uses exceptions to be tinier. For example,
this change shaves away 1000 lines of assembly code from LLVM's libcxx,
which is 0.7% of all assembly instructions in the entire library.
We now store values in jmp_buf where the compiler wants them to be. This
fixes code that calls __builtin_setjmp() and __builtin_longjmp() such as
libunwind. All libcxxabi tests are now passing on ARM64.
See #1076
Renaming gc() to _gc() was a mistake since the better thing to do is put
it behind the _COSMO_SOURCE macro. We need this change because I haven't
wanted to use my amazing garbage collector ever since we renamed it. You
now need to define _COSMO_SOURCE yourself when using amalgamation header
and cosmocc users need to pass the -mcosmo flag to get the gc() function
Some other issues relating to cancelation have been fixed along the way.
We're also now putting cosmocc in a folder named `.cosmocc` so it can be
more safely excluded by grep --exclude-dir=.cosmocc --exclude-dir=o etc.
* third_party: Add libcxxabi
Added libcxxabi from LLVM 17.0.6
The library implements the Itanium C++ exception handling ABI.
* third_party/libcxxabi: Enable __cxa_thread_atexit
Enable `__cxa_thread_atexit` from libcxxabi.
`__cxa_thread_atexit_impl` is still implemented by the cosmo libc.
The original `__cxa_thread_atexit` has been removed.
* third_party/libcxx: Build with exceptions
Build libcxx with exceptions enabled.
- Removed `_LIBCPP_NO_EXCEPTIONS` from `__config`.
- Switched the exception implementation to `libcxxabi`. These two files
are taken from the same `libcxx` version as mentioned in `README.cosmo`.
- Removed `new_handler_fallback` in favor of `libcxxabi` implementation.
- Enable `-fexceptions` and `-frtti` for `libcxx`.
- Removed `THIRD_PARTY_LIBCXX` dependency from `libcxxabi` and
`libunwind`. These libraries do not use any runtime `libcxx` functions,
just headers.
* libc: Remove remaining redundant cxa functions
- `__cxa_pure_virtual` in `libcxxabi` is also a stub similar to the
existing one.
- `__cxa_guard_*` from `libcxxabi` is used instead of the ones from
Android.
Now there should be no more duplicate implementations.
`__cxa_thread_atexit_impl`, `__cxa_atexit`, and related supporting
functions, are still left to other libraries as in `libcxxabi`.
`libcxxabi` is also now added to `cosmopolitan.a` to make up for the
removed functions.
Affected in-tree libraries (`third_party/double-conversion`) have been
updated.
The AMD HIP SDK for Linux ships prebuilt DSOs with an RWX PT_GNU_STACK
since old versions of GCC made it nearly impossible to build artifacts
where that wasn't the case, however modern glibc systems will flat out
refuse to link RWX DSOs from an execuatble that uses PT_GNU_STACK = RW
The cosmo_dlsym() function now returns the raw function address. You
need to call cosmo_dltramp() on the result, to make it safe to call.
This change is important, because cosmo_dltramp() magic can't always
work; for some tricky functions, you need to translate ABIs by hand.
Now we do them for assimilated binaries (except on OpenBSD or XNU
non-Silicon), for XnuSilicon, and for binaries with the preserve-
argv[0] auxv flag set. We check whether to pass the argv[0] value
at the test site rather than the Child site. We move a lot of the
test initialization into Child in the non-child case, in order to
get at the pre-init value of `__program_executable_name`. Finally,
we print out info about what we are skipping.
If cosmo_dlopen() is linked on AMD64 then the runtime will switch to
using %gs for thread-local storage. This eliminates the need for the
imported symbol trampoline. It's now safer to pass function pointers
back and forth with imported libraries. Your program gets recompiled
at runtime to make it happen and the overhead is a few milliseconds.
The toolchain will now be downloaded going forward from multiple pinned
URLs which have shasums. Either wget or curl must be installed.
This change unblocks #1053
Our dynamic linking implementation is now able to support functions with
dozens of parameters. In addition to having extra integral arguments you
can now pass vector registers using intrinsic types. Lastly, you can now
return multiple values, which is useful for functions returning structs.
* Reorder Launch arguments, pass aarch64 os
Third and fourth arguments are now identical between cosmo and Launch.
By passing sp as argument 4, we save a bit of register juggling.
Fourth argument (os) is now always passed by the loader on aarch64. It
is not yet processed by cosmo. Pushing this change separately, as the
cosmo side turns out to be somewhat more involved.
* cosmo2 receives os from loader
FreeBSD aarch64 now traps early rather than pretending to be Linux.
o/aarch64/examples/env.com still works on Linux and Xnu.
Now that our socket system call polyfills are good enough to support
Musl's DNS library we should be using that rather than the barebones
domain name system implementation we rolled on our own. There's many
benefits to making this change. So many, that I myself wouldn't feel
qualified to enumerate them all. The Musl DNS code had to be changed
in order to support Windows of course, which looks very solid so far
Missed this when changing the code back to be like the old version.
com is now a parameter.
The only plausible way to trigger this would be to pass a loader
pathname close to MAX_PATH characters long, and then remove that
path prior to the first sys_faccessat.
This implements proposals 1 and 2a from this gist:
https://gist.github.com/mrdomino/2222cab61715fd527e82e036ba4156b1
The only reason to use realpath from the loader was to try to prevent a
TOCTOU between the loader and the binary. But this is only a real issue
in set-id contexts, and in those cases there is already a canonical way
to do it: `/dev/fd`, passed by the kernel to the loader, so all we have
to do is pass that along to the binary.
Aside from realpath, there is no reason to absolutize the path we supply
to the binary, since it can call `getcwd` as well as we can, and on non-
M1 the binary is in a much better position to make that call.
Since we no longer absolutize the path, the binary does need to do this,
so we make its argv-parsing code generic and apply that to the different
possible places the path could come from. This means that `_` is finally
usable as a relative path, as a nice side benefit.
The M1 realpath code had a significant bug - it uses the wrong offset to
truncate the `.ape` in the `$prog.ape` case.
This PR also fixes a regression in `ape $progname` out of `$PATH` on the
two BSDs (Free and Net) that did not implement `RealPath`.
Fixes a regression in GetProgramExecutableName on Linux against old
loaders. In the loader case, /proc/self/exe gives the loader's path.
We tried to detect this by checking for `/usr/bin/ape`. But that is
only one of the possible places the loader could be.
Somehow or another, I previously had missed `BUILD.mk` files.
In the process I found a few straggler cases where the modeline was
different from the file, including one very involved manual fix where a
file had been treated like it was ts=2 and ts=8 on separate occasions.
The commit history in the PR shows the gory details; the BUILD.mk was
automated, everything else was mostly manual.
The ape loader now passes the program executable name directly as a
register. `x2` is used on aarch64, `%rdx` on x86_64. This is passed
as the third argument to `cosmo()` (M1) or `Launch` (non-M1) and is
assigned to the global `__program_executable_name`.
`GetProgramExecutableName` now returns this global's value, setting
it if it is initially null. `InitProgramExecutableName` first tries
exotic, secure methods: `KERN_PROC_PATHNAME` on FreeBSD/NetBSD, and
`/proc` on Linux. If those produce a reasonable response (i.e., not
`"/usr/bin/ape"`, which happens with the loader before this change),
that is used. Otherwise, if `issetugid()`, the empty string is used.
Otherwise, the old argv/envp parsing code is run.
The value returned from the loader is always the full absolute path
of the binary to be executed, having passed through `realpath`. For
the non-M1 loader, this necessitated writing `RealPath`, which uses
`readlinkat` of `"/proc/self/fd/[progfd]"` on Linux, `F_GETPATH` on
Xnu, and the `__realpath` syscall on OpenBSD. On FreeBSD/NetBSD, it
punts to `GetProgramExecutableName`, which is secure on those OSes.
With the loader, all platforms now have a secure program executable
name. With no loader or an old loader, everything still works as it
did, but setuid/setgid is not supported if the insecure pathfinding
code would have been needed.
Fixes#991.
We have received multiple reports of GCC breaking builds when compiler
flags like `-std=c11` were being passed. The workaround until the next
release is to simply not define `__STRICT_ANSI__` which is a bad idea.
At least in neovim, `│vi:` is not recognized as a modeline because it
has no preceding whitespace. After fixing this, opening a file yields
an error because `net` is not an option. (`noet`, however, is.)
* Introduce env.com
Handy tool for debugging environment issues.
* Inject path as COSMOPOLITAN_PROGRAM_EXECUTABLE
`argv[0]` was previously being used as a communication channel between
the loader and the binary, giving the binary its full path for use e.g.
in `GetProgramExecutableName`. But `argv[0]` is not a good channel for
this; much of what made 2a3813c6 so gross is due to that.
This change fixes the issue by preserving `argv[0]` and establishing a
new communication channel: `COSMOPOLITAN_PROGRAM_EXECUTABLE`.
The M1 loader will always set this as the first variable. Linux should
soon follow. On the other side, `GetProgramExecutableName` checks that
variable first. If it sees it, it trusts it as-is.
A lot of the churn in `ape/ape-m1.c` in this change is actually backing
out hacks introduced in 2a3813c6; the best comparison is:
git diff 2a3813c6^..
* Better refcounting
Cribbed from [Rust Arc][1] and the [Boost docs][2]:
"""
Increasing the reference counter can always be done with
memory_order_relaxed: New references to an object can only be formed
from an existing reference, and passing an existing reference from one
thread to another must already provide any required synchronization.
It is important to enforce any possible access to the object in one
thread (through an existing reference) to happen before deleting the
object in a different thread. This is achieved by a "release" operation
after dropping a reference (any access to the object through this
reference must obviously happened before), and an "acquire" operation
before deleting the object.
It would be possible to use memory_order_acq_rel for the fetch_sub
operation, but this results in unneeded "acquire" operations when the
reference counter does not yet reach zero and may impose a performance
penalty.
"""
[1] https://moshg.github.io/rust-std-ja/src/alloc/arc.rs.html
[2] https://www.boost.org/doc/libs/1_55_0/doc/html/atomic/usage_examples.html
* Make ZiposHandle's pos atomic
Implements a somewhat stronger guarantee than POSIX specifies: reads and
seeks are atomic. They may be arbitrarily reordered between threads, but
each one happens all the way and leaves the fd in a consistent state.
This is achieved by "locking" pos in __zipos_read by storing SIZE_MAX to
pos during the operation, so only one can be in-flight at a time. Seeks,
on the other hand, just update pos in one go, and rerun if it changed in
the meantime.
I used `LIKELY` / `UNLIKELY` to pessimize the concurrent case; hopefully
that buys back some performance.
* Implement __zipos_dup
Makes ZiposHandle reference-counted by an `rc` field in a union with its
freelist `next` pointer. The functions `__zipos_free` and `__zipos_keep`
function as incref/decref for it. Adds `__zipos_postdup` to fix metadata
on file descriptors after dup-like operations, and adds zipos support to
`sys_dup_nt` + `sys_close_nt`.
* Remove noop __zipos_postdup
rc is never a zipos file because it is always a previously unused file
descriptor. fd is never a zipos file because that case has been handled
above by __zipos_fcntl.
On UNIX if dup2(newfd) was a ZipOS file descriptor, then its resources
weren't being released, and the newly created file descriptor would be
mistaken for ZipOS due to its memory not being cleared. On Windows, an
issue also existed relating to newfd resources not being released.
Munging of paths passed inside the system() interpreter command is no
longer supported. You have to pass your paths to posix_spawn() or the
execve() family of functions if you want them to be munged. The first
three characters must match `^/[a-z]/` in which case, it'll be turned
into a DOS-style drive path with backslashes.
- Introduce MAP_JIT which is zero on other platforms
- Invent __jit_begin() and __jit_end() which wrap Apple's APIs
- Runtime dispatch to sys_icache_invalidate() in __clear_cache()
- Use good ELF technique in cosmo_dlopen()
- Make strerror() conform more to other libc impls
- Introduce __clear_cache() and use it in cosmo_dlopen()
- Remove libc/fmt/fmt.h header (trying to kill off LIBC_FMT)
After hearing horror stories from a trusted colleague, I don't think
this is the kind of API we want to be supporting. Also SQLite wisdom
regarding fdatasync() has been added to the documentation.
Imported functions are now aspected with a trampoline that blocks
signals and changes the thread-local storage register. This means
bigger more complicated libraries can now be imported even though
the whole technique remains fundamentally unsafe.
* [metal] Ensure DF is clear when calling C from exception handler
* [metal] Mark some internal routines and declarations as `@internal`
* [metal] Fix crash under UEFI when command line string is NULL
* [metal] Fix argc & argv[] setting, & VM page freeing, for UEFI
Part of the memory occupied by the argv[] contents was
erroneously used for page tables & then later erroneously
freed. The symptom was that argv[0] would show up as an
empty string ("").
We now have an `#include <cxxabi.h>` header which defines all the APIs
Cosmopolitan's implemented so far. The `cosmocc` README.md file is now
greatly expanded with documentation.
Our makefile generator now accepts badly formatted include lines. It's
now more hermetic with better error checking in the cosmo repo, and it
can be configured to not be hermetic at all.
Using `cosmocc -std=c11` was causing `ucontext_t` to become misaligned.
This change also adds the GNU constants on x86_64 for accessing general
registers, so you will not need `#ifdef`s to support both Cosmo and GNU
wait4() is now solid enough to run `make -j100` on Windows. You can now
use MSG_DONTWAIT on Windows. There was a handle leak in accept() that's
been fixed. Our WIN32 overlapped i/o code has been simplified. Priority
class now inherits into subprocesses, so the verynice command will work
and the signal mask will now be inherited by execve() and posix_spawn()
Every program built using Cosmopolitan is statically-linked. However
there are some cases, e.g. GUIs and video drivers, where linking the
host platform libraries is desirable. So what we do in such cases is
launch a stub executable using the host platform's libc, and longjmp
back into this executable. The stub executable passes back to us the
platform-specific dlopen() implementation, which we shall then wrap.
Here's the list of platforms that are supported so far:
- x86-64 Linux w/ Glibc
- x86-64 Linux w/ Musl Libc
- x86-64 FreeBSD
- x86-64 Windows
- aarch64 Linux w/ Glibc
- aarch64 MacOS
What this means is your Cosmo programs can call foreign functions on
your host operating system. However, it's important to note that any
foreign library you link won't have the ability to call functions in
your Cosmopolitan program. For example it's now technically possible
that Lua can load a module, however that almost certainly won't work
since the Lua module won't have access to Cosmo's Lua API.
Kudos to @jacereda for figuring out how to do this.
This change addresses a bug that was reported in #923 where bash on
Windows behaved strangely. It turned out that our weak linking of
malloc() caused bash's configure script to favor its own getcwd()
function, which is implemented in the most astonishing way, using
opendir() and readdir() to recursively construct the current path.
This change moves getcwd() into LIBC_STDIO so it can strongly link
malloc(). A new __getcwd() function is now introduced, so all the
low-level runtime services can still use the actual system call. It
provides the Linux Kernel API convention across platforms, and is
overall a higher-quality implementation than what we had before.
In the future, we should probably take a closer look into why bash's
getcwd() polyfill wasn't working as intended on Windows, since there
might be a potential opportunity there to improve our readdir() too.
This function was invented by the BSDs (it's not in POSIX.1). It
provides a high-level interface into ioctl(SIOCGIFCONF) which is
comparatively clumsy to use. We already made the ioctls portable
across our entire support vector back in 2021, so this interface
is portable too. See o//tool/viz/getifaddrs.com for our demo app
