- ARM Neon headers are now exported in libc/isystem/
- stat() and access() now do a better job reporting which files are
executable which ones aren't. They do this by reading the first two
bytes in a file to see if it's `MZ` or `#!`.
Unlike CMD.EXE, CreateProcess() doesn't care if an executable name ends
with .COM or .EXE. We now have the unbourne shell and bash working well
on Windows, so we don't need DOS anymore. Making this change will grant
us better performance, particularly for builds, because commandv() will
need to make fewer system calls. Path mangling magic still happens with
WinMain() and ntspawn() in order to do things like turn \ into / so the
interop works well at the borders. But all the code in libraries, which
did that, has been removed. It's not possible for libraries to abstract
the differences between paths.
- Improved async signal safety of read() particularly for longjmp()
- Started adding cancel cleanup handlers for locks / etc on Windows
- Make /dev/tty work better particularly for uses like `foo | less`
- Eagerly read console input into a linked list, so poll can signal
- Fix some libc definitional bugs, which configure scripts detected
This change removes our use of ENABLE_VIRTUAL_TERMINAL_INPUT (which
isn't very good) in favor of having read() translate Windows Console
input events to ANSI/XTERM sequences by hand. This makes it possible to
capture important keystrokes (e.g. ctrl-space) that weren't possible
before. Most importantly this change also removes the stdin/sigwinch
worker threads, which never really worked that well. Interactive TTY
sessions will now work reliably when a Cosmo process spawns or forks
another Cosmo process, e.g. unbourne.com launching emacs.com.
- Every unit test now passes on Apple Silicon. The final piece of this
puzzle was porting our POSIX threads cancelation support, since that
works differently on ARM64 XNU vs. AMD64. Our semaphore support on
Apple Silicon is also superior now compared to AMD64, thanks to the
grand central dispatch library which lets *NSYNC locks go faster.
- The Cosmopolitan runtime is now more stable, particularly on Windows.
To do this, thread local storage is mandatory at all runtime levels,
and the innermost packages of the C library is no longer being built
using ASAN. TLS is being bootstrapped with a 128-byte TIB during the
process startup phase, and then later on the runtime re-allocates it
either statically or dynamically to support code using _Thread_local.
fork() and execve() now do a better job cooperating with threads. We
can now check how much stack memory is left in the process or thread
when functions like kprintf() / execve() etc. call alloca(), so that
ENOMEM can be raised, reduce a buffer size, or just print a warning.
- POSIX signal emulation is now implemented the same way kernels do it
with pthread_kill() and raise(). Any thread can interrupt any other
thread, regardless of what it's doing. If it's blocked on read/write
then the killer thread will cancel its i/o operation so that EINTR can
be returned in the mark thread immediately. If it's doing a tight CPU
bound operation, then that's also interrupted by the signal delivery.
Signal delivery works now by suspending a thread and pushing context
data structures onto its stack, and redirecting its execution to a
trampoline function, which calls SetThreadContext(GetCurrentThread())
when it's done.
- We're now doing a better job managing locks and handles. On NetBSD we
now close semaphore file descriptors in forked children. Semaphores on
Windows can now be canceled immediately, which means mutexes/condition
variables will now go faster. Apple Silicon semaphores can be canceled
too. We're now using Apple's pthread_yield() funciton. Apple _nocancel
syscalls are now used on XNU when appropriate to ensure pthread_cancel
requests aren't lost. The MbedTLS library has been updated to support
POSIX thread cancelations. See tool/build/runitd.c for an example of
how it can be used for production multi-threaded tls servers. Handles
on Windows now leak less often across processes. All i/o operations on
Windows are now overlapped, which means file pointers can no longer be
inherited across dup() and fork() for the time being.
- We now spawn a thread on Windows to deliver SIGCHLD and wakeup wait4()
which means, for example, that posix_spawn() now goes 3x faster. POSIX
spawn is also now more correct. Like Musl, it's now able to report the
failure code of execve() via a pipe although our approach favors using
shared memory to do that on systems that have a true vfork() function.
- We now spawn a thread to deliver SIGALRM to threads when setitimer()
is used. This enables the most precise wakeups the OS makes possible.
- The Cosmopolitan runtime now uses less memory. On NetBSD for example,
it turned out the kernel would actually commit the PT_GNU_STACK size
which caused RSS to be 6mb for every process. Now it's down to ~4kb.
On Apple Silicon, we reduce the mandatory upstream thread size to the
smallest possible size to reduce the memory overhead of Cosmo threads.
The examples directory has a program called greenbean which can spawn
a web server on Linux with 10,000 worker threads and have the memory
usage of the process be ~77mb. The 1024 byte overhead of POSIX-style
thread-local storage is now optional; it won't be allocated until the
pthread_setspecific/getspecific functions are called. On Windows, the
threads that get spawned which are internal to the libc implementation
use reserve rather than commit memory, which shaves a few hundred kb.
- sigaltstack() is now supported on Windows, however it's currently not
able to be used to handle stack overflows, since crash signals are
still generated by WIN32. However the crash handler will still switch
to the alt stack, which is helpful in environments with tiny threads.
- Test binaries are now smaller. Many of the mandatory dependencies of
the test runner have been removed. This ensures many programs can do a
better job only linking the the thing they're testing. This caused the
test binaries for LIBC_FMT for example, to decrease from 200kb to 50kb
- long double is no longer used in the implementation details of libc,
except in the APIs that define it. The old code that used long double
for time (instead of struct timespec) has now been thoroughly removed.
- ShowCrashReports() is now much tinier in MODE=tiny. Instead of doing
backtraces itself, it'll just print a command you can run on the shell
using our new `cosmoaddr2line` program to view the backtrace.
- Crash report signal handling now works in a much better way. Instead
of terminating the process, it now relies on SA_RESETHAND so that the
default SIG_IGN behavior can terminate the process if necessary.
- Our pledge() functionality has now been fully ported to AARCH64 Linux.
This changes *NSYNC to allocate waiters on the stack so our locks don't
need to depend on dynamic memory. This make our runtiem simpler, and it
also fixes bugs with thread cancellation support.
This change fixes bugs in the APE loader. The execve() unit tests are
now enabled for MODE=aarch64. See the README for how you need to have
binfmt_misc configured with Qemu to run them. Apple Silicon bugs have
been fixed too, e.g. tkill() now works.
The new asynchronous signal delivery technique is now also being used
for tkill(), raise(), etc. Many subtle issues have been addresesd. We
now signal handling on Windows that's remarkably similar to the POSIX
behaviors. However that's just across threads. We're lacking a way to
have the signal semantics work well, across multiple WIN32 processes.
The stdio reader thread now appears to be working recursively along
cosmopolitan subprocesses. For example, it's now possible to launch
vim.com from the unbourne.com bestline repl, thanks to hacks plus a
bug fix to select() timeouts.
- Remove misguided __assert_disabled variable
- Change EPROCLIM to be EAGAIN on BSD distros
- Improve quality of greenbean with cancellations
- Fix thread race condition crash with file descriptors
- Polyfill readlink("foo/") dir check on Windows
- Support asynchronous signal delivery on Windows
- Restore Windows Console from execve() daisy chain
- Work around bug in AARCH64 Optimized Routines memcmp()
- Disable unbourne.com shell completion on Windows for now
- Don't always set virtual terminal input state on console
- Remove Musl Libc's unusual preservation of realpath("//")
- Make realpath() strongly link malloc() to pass configure test
- Delete cosh.com shell, now that unbourne.com works on Windows!
- Invent openatemp() API
- Invent O_UNLINK open flag
- Introduce getenv_secure() API
- Remove `git pull` from cosmocc
- Fix utimes() when path is NULL
- Fix mktemp() to never return NULL
- Fix utimensat() UTIME_OMIT on XNU
- Improve utimensat() code for RHEL5
- Turn `argv[0]` C:/ to /C/ on Windows
- Introduce tmpnam() and tmpnam_r() APIs
- Fix more const issues with internal APIs
- Permit utimes() on WIN32 in O_RDONLY mode
- Fix fdopendir() to check fd is a directory
- Fix recent crash regression in landlock make
- Fix futimens(AT_FDCWD, NULL) to return EBADF
- Use workaround so `make -j` doesn't fork bomb
- Rename dontdiscard to __wur (just like glibc)
- Fix st_size for WIN32 symlinks containing UTF-8
- Introduce stdio ext APIs needed by GNU coreutils
- Fix lstat() on WIN32 for symlinks to directories
- Move some constants from normalize.inc to limits.h
- Fix segv with memchr() and memcmp() overlapping page
- Implement POSIX fflush() behavior for reader streams
- Implement AT_SYMLINK_NOFOLLOW for utimensat() on WIN32
- Don't change read-only status of existing files on WIN32
- Correctly handle `0x[^[:xdigit:]]` case in strtol() functions
This change fixes Cosmopolitan so it has fewer opinions about compiler
warnings. The whole repository had to be cleaned up to be buildable in
-Werror -Wall mode. This lets us benefit from things like strict const
checking. Some actual bugs might have been caught too.
- Fix stdio fmemopen() buffer behaviors
- Fix scanf() to return EOF when appropriate
- Prefer fseek/ftell names over fseeko/ftello
- Ensure locale field is always set in the TIB
- Fix recent regression in vfprintf() return count
- Make %n directive in scanf() have standard behavior
- Introduce ualarm() function
- Make rename() report EISEMPTY on Windows
- Always raise EINVAL upon open(O_RDONLY|O_TRUNC)
- Add macro so ./configure will detect SOCK_CLOEXEC
- Fix O_TRUNC without O_CREAT not working on Windows
- Let fcntl(F_SETFL) change O_APPEND status on Windows
- Make sure pwrite() / pread() report ESPIPE on sockets
- Raise ESPIPE on Windows when pwrite() is used on pipe
- Properly compute O_APPEND CreateFile() flags on Windows
- Don't require O_DIRECTORY to open directories on Windows
- Fix more instances of Windows reporting EISDIR and ENOTDIR
- Normalize EFTYPE and EMLINK to ELOOP on NetBSD and FreeBSD
- Make unlink() / rmdir() work on read-only files on Windows
- Validate UTF-8 on Windows paths to fix bug with overlong NUL
- Always print signal name to stderr when crashing due to SIG_DFL
- Fix Windows bug where denormalized paths >260 chars didn't work
- Block signals on BSDs when thread exits before trashing its own stack
- Get SIGWINCH working again on the New Technology
- Correctly handle O_NOFOLLOW in open() on Windows
- Implement synthetic umask() functionality on Windows
- Do a better job managing file execute access on Windows
- Fill in `st_uid` and `st_gid` with username hash on Windows
- Munge UNICODE control pictures into control codes on Windows
- Do a better job ensuring Windows console settings are restored
- Introduce KPRINTF_LOG environment variable to log kprintf to a file
This change brings the /zip/... read-only filesystem into performance
parity with the native Linux filesystem which doesn't use compression
therefore, imagine how much faster this could be with bloom filtering
rather than simple binary search, and if we used zstd instead of zlib
This change reduces the memory requirements of your APE Loader by 10x,
in terms of virtual memory size, thanks to the help of alloca(). We're
also now creating argument blocks with the same layout across systems.
- Fix mkdeps.com out of memory error
- Remove static memory from __get_cpu_count()
- Add support for passing hyphen to cat in cocmd
- Change more ZipOS errors from ENOTSUP to EROFS
- Specify mem_unit in sysinfo() output on BSD OSes
readdir() will now always yield an inode that's consistent with stat()
on ZipOS and Windows in general. More APIs have been updated to return
the appropriate error code when inappropriately trying to do ops, like
sockets, with a zip file descriptor. The path normalization algorithms
are now fully fleshed out. Some socket APIs have been fixed so they'll
raise EBADF vs. ENOTSOCK appropriately. Lastly seekdir() will now work
properly on NetBSD and FreeBSD (not sure why anyone would even use it)
This change improves the dirstream library in a lot of respects,
especially for /zip/... files. Also turn off MAP_STACK on Aarch64
because Qemu seems to implement it differently than Linux and it's
probably responsible for a lot of mysterious crashes.
You can now play Super Mario Bros in CMD.EXE using Cosmopolitan! This is
thanks to a new worker thread that's spawned on Windows whenever any one
of poll(), select(), or ioctl(FIONREAD) is linked.
make all test CC=fatcosmocc AR='fatcosmoar rcu'
This change introduces a program named mktemper.com which provides more
reliable and secure temporary file name generation for scripts. It also
makes our ar.com program more permissive in what commands it'll accept.
The cosmocc command is improved by this change too.
This way complex runtime features (e.g. ftrace, symbol tables) can
always yoink zipos support. This is important now that apelink.com
automates embedding symbol tables for multiple cpus.
This new script is an alternative to the `cosmocc` command. It's still a
work in progress. It abstracts all the gory details of building separate
copies of your executable and then running the apelink.com program.
If you build a static ELF executable in `ld -q` mode (which leaves rela
sections inside the binary) then you can run it through the elf2pe.com
program afterwards, which will turn it into a PE executable. We have a
new trick for defining WIN32 DLL imports in C without any assembly code.
This also achieves the optimally tiny and perfect PE binary structure.
We need this because it isn't possible to have a GNU ld linker script
generate a PE file where the virtual pointer and the file pointer can
drift apart. This post-linker can do that. One cool benefit is we can
now use a smaller 512-byte alignment in the file, and an even bigger
64kb alignment for the segment virtual addresses, and the executable
ends up being smaller.
Another program introduced by this change is pecheck.com which can do
extensive linting of PE static executables to help explain why Windows
won't load it.
- Blocking read operations on the Windows Console can now EINTR
- Blocking read operations on Windows pipes now EINTR more reliably
- setitimer() will no longer be inherited across fork() on Windows
- It's now possible to use ECHO when the console is in raw mode
- The ECHOCTL flag now works correctly on the Windows Console
- The ICRNL flag now works correctly on the Windows Console
- pread() and pwrite() will now raise ESPIPE on Windows
- Opening /dev/tty on Windows is improved (untested)
- Overlapped I/O is now implemented in a better way
This change has the insight that dwExitCode isn't an exit code but
rather should be used to pass the wait status. This lets us report
killing as a termination status, similar to UNIX. This change also
fixes the fact that exit(259) on Windows will break the parent due
way WIN32 is designed. We now work around that.
It turns out that NetBSD and OpenBSD, will let you have exit codes
beyond 255. This change will let you use them when it's possible.
This change makes posix_spawn_test no longer flaky on Windows, by (1)
fixing a race condition in wait(), and (2) removing a misguided vfork
implementation which was letting Windows bypass pthread_atfork().
- Remove PAGESIZE constant
- Fix realloc() documentation
- Fix ttyname_r() error reporting
- Make forking more reliable on Windows
- Make execvp() a few microseconds faster
- Make system() a few microseconds faster
- Tighten up the socket-related magic numbers
- Loosen restrictions on mmap() offset alignment
- Improve GetProgramExecutableName() with getenv("_")
- Use mkstemp() as basis for mktemp(), tmpfile(), tmpfd()
- Fix flakes in pthread_cancel_test, unix_test, fork_test
- Fix recently introduced futex stack overflow regression
- Let sockets be passed as stdio to subprocesses on Windows
- Improve security of bind() on Windows w/ SO_EXCLUSIVEADDRUSE
This change addresses everything from stack smashing to %SYSTEMROOT%
breaking socket(). Issues relating to compile.com not reporting text
printed to stderr has been resolved for Windows builds.
This change ports APE Loader to Linux AARCH64, so that Raspberry Pi
users can run programs like redbean, without the executable needing
to modify itself. Progress has also slipped into this change on the
issue of making progress better conforming to user expectations and
industry standards regarding which symbols we're allowed to declare
This change also incorporates more bug fixes and improvements to a wide
variety of small things. For example this fixes#860 so Windows console
doesn't get corrupted after exit. An system stack memory map issue with
aarch64 has been fixed. We no longer use O_NONBLOCK on AF_UNIX sockets.
Crash reports on Arm64 will now demangle C++ symbols, even when c++filt
isn't available. Most importantly the Apple M1 version of APE Loader is
brought up to date by this change. A prebuilt unsigned binary for it is
being included in build/bootstrap/. One more thing: retrieving the term
dimensions under --strace was causing the stack to become corrupted and
now that's been solved too. PSS: We're now including an ELF PT_NOTE for
APE in the binaries we build, that has the APE Loader version.
This version has better error messages and safety checks. It supports
loading static position-independent executables. It correctly handles
more kinds of weird ELF program header layouts. A force flag has been
added to avoid system execve(). Finally the longstanding misalignment
with our ELF PT_NOTE section has been addressed.
- The functions that return a sockaddr now do so the same way the Linux
Kernel does across platforms, e.g. getpeername(), accept4()
- Socket system calls on Windows will now only check for interrupts when
a blocking operation needs to be performed.
- Write tests for recvfrom() system call
lz4toasm should now more easily accept LZ4 files output by
compressor programs that do not support the extracted-size
field, such as Stephan Brumme's smallz4.
This patch also proposes to add a new lz4len() function to
the libc: it parses an LZ4 compressed block to compute the
unpacked content size, without really unpacking the block.
Co-authored-by: tkchia <tkchia-cosmo@gmx.com>
- tcgetpgrp(STDIN_FILENO) should be equal to getpgrp() on Windows also,
found while reading wget source code which uses this check to decide
whether to print to stderr or to a file
- IN6_ADDR_ARE_EQUAL is a comparison macro used when IPV6 is allowed,
found while reading CPython3.11 source code
- the changes in signal.h and addition of ucontext.h are because
CPython3.11 source code expect sigaltstack to be available
- the sqlite3.mk change is because CPython3.11 requires sqlite3 to be
built with -DOMIT_SHARED_CACHE
- unistd.h has getopt.h now, because some libraries like it there
This change introduces new tests for `O_NONBLOCK` and `SOCK_NONBLOCK` to
confirm that non-blocking i/o is now working on all supported platforms,
including Windows. For example, you can now say on Windows, MacOS, etc.:
socket(AF_INET, SOCK_STREAM | SOCK_NONBLOCK, IPPROTO_TCP);
To create a non-blocking IPv4 TCP socket. Or you can enable non-blocking
i/o on an existing socket / pipe / etc. file descriptor by calling fcntl
fcntl(fd, F_SETFL, fcntl(fd, F_GETFL) | O_NONBLOCK);
This functionality is polyfilled on older Linux kernels too, e.g. RHEL5.
Now that fcntl() support is much better the FIOCLEX / FIONCLEX polyfills
for ioctl() have been removed since they're ugly non-POSIX diameond APIs
This change fixes a weakness in kprintf() that was causing Windows trace
tools to frequently crash.
- This commit mints a new release of APE Loader v1.2 which supports
loading ELF programs with a non-contiguous virtual address layout
even though we've never been able to take advantage of it, due to
how `objcopy -SO binary` fills any holes left by PT_LOAD. This'll
change soon, since we'll have a new way of creating APE binaries.
- The undiamonding trick with our ioctl() implementation is removed
since POSIX has been killing ioctl() for years and they've done a
much better job. One problem it resolves, is that ioctl(FIONREAD)
wasn't working earlier and that caused issues when building Emacs
- Fix unused local variable errors
- Remove yoinks from sigaction() header
- Add nox87 and aarch64 to github actions
- Fix cosmocc -fportcosmo in linking mode
- It's now possible to build `make m=llvm o/llvm/libc`
- compile.com now polyfills -march=native which gcc/clang removed
- Guarantee zero Windows code is linked into non-Windows binaries
- MODE=tinylinux binaries are now back to being as tiny as ~4kb
- Improve the runtime's stack allocation / alignment hack
- GitHub Actions now tests Linux modes for assurance
- Fix handling of precision in hex float formatting
- Enhance the cocmd interpreter for system() and popen()
- Manually ran the Lua unit tests, which are now passing
- Let stdio i/o operations happen when file is in error state
- We're now saving and restoring xmm in ftrace out of paranoia
It turned out that Landlock Make hasn't been applying sandboxing for a
while, due to a mistyped if statement for `$(USE_SYSTEM_TOOLCHAIN)` it
should have had the opposite meaning. Regressions in the build configs
have been fixed. The rmrf() function works better now. The rm.com tool
works according to POSIX with the exception of supporting prompts.
This change figures out some of the build configuration issues we've
been having with libcxx. The c++ span header is added. Per a Discord
discussion we're now turning off `-g` for the default build mode, so
consider using `make MODE=dbg` or `make MODE=zero` for GDB debugging
which works much better than `MODE=` ever has. Note that the default
build mode has always had very good function call / system call logs
plus you can still use ShowCrashReports() for backtrace. Making this
change ensures cosmocc will better conform to FOSS norms. Lastly the
LoadZipArgs() API has been added to cosmopolitan.a and <cosmo.h>.
This change fixes stderr to be unbuffered. Added hardware AES on ARM64
to help safeguard against timing attacks. The curl.com command will be
somewhat more pleasant to use.
The *NSYNC linked list API is good enough that it deserves to be part of
the C libray, so this change writes an improved version of it which uses
that offsetof() trick from the Linux Kernel. We vendor all of the *NSYNC
tests in third_party which helped confirm the needed refactoring is safe
This change also deletes more old code that didn't pan out. My goal here
is to work towards a vision where the Cosmopolitan core libraries become
less experimental and more focused on curation. This better reflects the
current level of quality we've managed to achieve.
After going through the MODE=dbg and MODE=zero build modes, a bunch of
little issues were identified, which have been addressed. Fixing those
issues created even more troubles for the project, because it improved
our ability to detect latent problems which are getting fixed so fast.
The build/bootstrap/ar.com program is now tinier. This change reduces
its size from 140kb to 53kb. Nothing was traded away. Cosmopolitan Ar
performance is now 2x better than llvm-ar largely thanks to using the
copy_file_range() system call. This change homebrews a new allocation
API that addresses the shortcomings of the C standard library design.
Using these new balloc() and reballoc() functions I managed to reduce
memory consumption so much that Cosmpolitan Ar should now use roughly
100x fewer bytes of peak resident memory compared to llvm-ar. Correct
behavior with better compatibility has been assured. Binary output is
now pretty much bit-identical to llvm-ar, as of this change. This can
and should be the living proof we need to show that a better world is
possible for software.
This change introduces support for Linux-style uc_context manipulation
that's fast and works well on all supported OSes and architectures. It
also integrates with the Cosmpolitan runtime which can show backtraces
comprised of multiple stacks and fibers. See the test and example code
for further details. This will be used by Mold once it's been vendored
This change (1) upgrades to OpenBSD's newer kernel ABIs, and (2)
modifies APE to have a read-only data segment. Doing this required
creating APE Loader v1.1, which is backwards and forwards compatible
with the previous version.
If you've run the following commands in the past to install your APE
Loader systemwide, then you need to run them again. Ad-hoc installations
shouldn't be impacted. It's also recommended that APE binaries be remade
after upgrading, since they embed old versions of the APE Loader.
ape/apeuninstall.sh
ape/apeinstall.sh
This change does more than just fix OpenBSD. The new loader is smarter
and more reliable. We're now able create much tinier ELF and Mach-O data
structures than we could before. Both APE Loader and execvpe() will now
normalize ambiguous argv[0] resolution the same way as the UNIX shell.
Badness with TLS linkage has been solved.
Fixes#826
The recent change to crt.S that aggressively aligns the system-provided
stack has been rolled back on non-Linux until we can find a better way,
since it can cause a segfault early in execution on several platforms.
This change fixes a regression in tcgetattr() and tcsetattr() on OpenBSD
and NetBSD caused by 4778cd4d27.
This change has been tested across the runitd test fleet which is green.
The intent with pledge("anet") has been to prevent outbound connections.
However we were only doing that for TCP sockets, and outbound UDP could
still get through, by using socket() plus sendto(). This change fixed
that by preventing UDP sockets from being created.
Credit goes to chc4 on Hacker News for finding this.
- Found some bugs in LLVM compiler-rt library
- The useless LIBC_STUBS package is now deleted
- Improve the overflow checking story even further
- Get chibicc tests working in MODE=dbg mode again
- The libc/isystem/ headers now have correctly named guards
This change upgrades to the latest portcosmo gcc patch
6728fe1a25185560603ca312a8d4352af2a4e515 which lets us avoid needing to
define __tmpcosmo_FOO constants. We're now using an appropriate binutils
version for GCC 11. The older binutils sometimes wasn't able to print
backtraces, due to not being able to find a .debug_ranges section.
This is breaking change for /opt/cosmos libraries :'( due to this weird
"error: need linked-to section for --gc-sections" that pops up.
Please run `make clean` in the monorepo before rebuilding.
Since 8ff48201ca we no longer need the
hack where, when running .com.dbg files, we scanned for the embedded
.com file offset, and then computed zip offsets realtive to that. It
wasn't very reliable in the first place, and was causing issues with
running our new .com.dbg executables, which are true zip files.
- More timspec_*() and timeval_*() APIs have been introduced.
- The copyfd() function is now simplified thanks to POSIX rules.
- More Cosmo-specific APIs have been moved behind the COSMO define.
- The setitimer() polyfill for Windows NT is now much higher quality.
- Fixed build error for MODE=aarch64 due to -mstringop-strategy=loop.
- This change introduces `make MODE=nox87 toolchain` which makes it
possible to build programs using your cosmocc toolchain that don't
have legacy fpu instructions. This is useful, for example, if you
want to have a ~22kb tinier blink virtual machine.
This change fixes an issue with the tcflow() magic numbers that was
causing bash to freeze up on Linux. While auditing termios polyfills,
several other issues were identified with XNU/BSD compatibility.
Out of an abundance of caution this change undefines as much surface
area from libc/calls/struct/termios.h as possible, so that autoconf
scripts are less likely to detect non-POSIX teletypewriter APIs that
haven't been polyfilled by Cosmopolitan.
This is a *breaking change* for your static archives in /opt/cosmos if
you use the cosmocc toolchain. That's because this change disables the
ioctl() undiamonding trick for code outside the monorepo, specifically
because it'll lead to brittle ABI breakages like this. If you're using
the cosmocc toolchain, you'll need to rebuild libraries like ncurses,
readline, etc. Yes diamonds cause bloat. To work around that, consider
using tcgetwinsize() instead of ioctl(TIOCGWINSZ) since it'll help you
avoid pulling every single ioctl-related polyfill into the linkage.
The cosmocc script was specifying -DNDEBUG for some reason. It's fixed.
As of now, the syscall function is implemented as alike to how the
linux kernel sycall ABI works, returning -errno upon errors without
setting the value of errno. However, this does not conform to the
expectations of most software, which expect it to return -1 and set
errno, which is how it works on other libcs, which document it
accordingly:
> The return value is defined by the system call being invoked. In
> general, a 0 return value indicates success. A -1 return value
> indicates an error, and an error number is stored in errno.
- Linux man-pages, syscall(2)
> The return value is the return value from the system call, unless
> the system call failed. In that case, ‘syscall’ returns ‘-1’ and
> sets ‘errno’ to an error code that the system call returned.
- glibc manual, (libc)System Calls
> When the C-bit is set, syscall returns -1 and sets the external
> variable errno (see intro(2)).
- 4BSD manual, syscall(2)
> A -1 return value indicates an error, and an error code is stored in
> errno.
- 4.4BSD, FreeBSD, OpenBSD and NetBSD manuals (same quote is found in
all of them), syscall(2)
This patch corrects the syscall function to work in the same way as in
other libcs.
This change takes an entirely new approach to the incremental linking of
pkzip executables. The assets created by zipobj.com are now treated like
debug data. After a .com.dbg is compiled, fixupobj.com should be run, so
it can apply fixups to the offsets and move the zip directory to the end
of the file. Since debug data doesn't get objcopy'd, a new tool has been
introduced called zipcopy.com which should be run after objcopy whenever
a .com file is created. This is all automated by the `cosmocc` toolchain
which is rapidly becoming the new recommended approach.
This change also introduces the new C23 checked arithmetic macros.
This change improves the way internal APIs are being hidden behind the
`COSMO` define. The cosmo.h header will take care of defining that, so
that a separate define statement isn't needed. This change also does a
lot more to define which APIs are standard, and which belong to Cosmo.
This change integrates e58abc1110b335a3341e8ad5821ad8e3880d9bb2 from
https://github.com/ahgamut/musl-cross-make/ which fixes the issues we
were having with our C language extension for symbolic constants. This
change also performs some code cleanup and bug fixes to getaddrinfo().
It's now possible to compile projects like ncurses, readline and python
without needing to patch anything upstream, except maybe a line or two.
Pretty soon it should be possible to build a Linux distro on Cosmo.
In order to improve our chances of success building other open source
projects we shouldn't define APIs that'll lead any ./configure script
astray. For example:
- brk() and sbrk() can break mac/windows support
- syscall() is a superb way to break portability
- arch_prctl() is the greatest of all horror shows
- Work towards improving non-optimized build support
- Introduce MODE=zero which is -O0 without ASAN/UBSAN
- Use system GCC when ~/.cosmo.mk has USE_SYSTEM_TOOLCHAIN=1
- Have package.com check .privileged code doesn't call non-privileged
Garbage collection will now happen on arm64 when a function returns,
rather than kicking the can down the road to when the process exits.
This change also does some code cleanup and incorporates suggestions
- Get mprotect_test working on aarch64
- Get completion working on python.com repl again
- Improve quality of printvideo.com and printimage.com
- Fix bug in openpty() so examples/script.c works again
This change greatly reduces the number of modules that need to be
compiled. The only issue right now is that sometimes when viewing
symbol table entries, the aliased symbol is chosen.
Apparently IANA has abolished the WHOIS protocol and no longer lists it
as a service. Therefore distros which naively create /etc/services from
IANA's braindead recommendation will inadvertently break any tools that
rely on /etc/services to determine this well-known Internet port.
This change implements a new approach to function call logging, that's
based on the GCC flag: -fpatchable-function-entry. Read the commentary
in build/config.mk to learn how it works.
- Now using 10x better GCD semaphores
- We now generate Linux-like thread ids
- We now use fast system clock / sleep libraries
- The APE M1 loader now generates Linux-like stacks
llama.com can now load weights that use the new file format which was
introduced a few weeks ago. Note that, unlike llama.cpp, we will keep
support for old file formats in our tool so you don't need to convert
your weights when the upstream project makes breaking changes. Please
note that using ggjt v3 does make avx2 inference go 5% faster for me.
This change progresses our AARCH64 support:
- The AARCH64 build and tests are now passing
- Add 128-bit floating-point support to printf()
- Fix clone() so it initializes cosmo's x28 TLS register
- Fix TLS memory layout issue with aarch64 _Alignas vars
- Revamp microbenchmarking tools so they work on aarch64
- Make some subtle improvements to aarch64 crash reporting
- Make kisdangerous() memory checks more accurate on aarch64
- Remove sys_open() since it's not available on Linux AARCH64
This change makes general improvements to Cosmo and Redbean:
- Introduce GetHostIsa() function in Redbean
- You can now feature check using pledge(0, 0)
- You can now feature check using unveil("",0)
- Refactor some more x86-specific asm comments
- Refactor and write docs for some libm functions
- Make the mmap() API behave more similar to Linux
- Fix WIFSIGNALED() which wrongly returned true for zero
- Rename some obscure cosmo keywords from noFOO to dontFOO
* Add `anet` pledge for `inet` without connect
This is useful for configurations where it's desirable to start redbean
under these restrictions, but not to allow `connect` socket calls.
* Update message on protected/unpledged syscalls for clarity
* Update redbean to add reporting for unpledged sigaction
Previously it would abort without indicating what signal it failed to
install when sigaction is not pledged (although it fails all of them).
* Move GetHostIps before processing command line options
This allows using unix.pledge as part of the options without affecting
retrieving host IP addresses (which requires `connect`). It may still
fail under external `pledge` command as expected; in this case IPs
would need to be passed manually.
* Update tests for pledge anet promise
There's a new program named ape/ape-m1.c which will be used to build an
embeddable binary that can load ape and elf executables. The support is
mostly working so far, but still chasing down ABI issues.
- Fix UX issues with llama.com
- Do housekeeping on libm code
- Add more vectorization to GGML
- Get GGJT quantizer programs working well
- Have the quantizer keep the output layer as f16c
- Prefetching improves performance 15% if you use fewer threads
- Perform some housekeeping on scalar math function code
- Import ARM's Optimized Routines for SIMD string processing
- Upgrade to latest Chromium zlib and enable more SIMD optimizations
- Introduce epoll_pwait()
- Rewrite -ftrapv and ffs() libraries in C code
- Use more FreeBSD code in math function library
- Get significantly more tests passing on qemu-aarch64
- Fix many Musl long double functions that were broken on AARCH64
- Utilities like pledge.com now build
- kprintf() will no longer balk at 48-bit addresses
- There's a new aarch64-dbg build mode that should work
- gc() and defer() are mostly pacified; avoid using them on aarch64
- THIRD_PART_STB now has Arm Neon intrinsics for fast image handling
It's now possible to run commands like:
make -j8 m=aarch64 o/aarch64/test/libc/str
Which will cross-compile and run the test suites in a qemu-aarch64
binary that's vendored in the third_party/qemu/ folder within your
x86_64 build environment.
The ShowCrashReports() feature for aarch64 should work even better than
the x86 crash reports. Thanks to the benefit of hindsight these reports
should be rock solid reliable and beautiful to read.
This change also improves the syscall polyfills for aarch64. Some of the
sys_foo() functions have been removed, usually because they're legacy or
downright footguns not worth building.
The new stack size is 256kb in order to compromise with llama.cpp's
aggressive use of stack memory, which can't be easily patched. This
change disables the dynamic alloca() and VLA warnings for now, plus
frame sizes for individual functions may be <=50% of the stack size
This only applies to code in the cosmo monorepo. Open source builds
should already be using an 8mb stack by default, like everyone else
Right now, cosmopolitan uses Linux Landlock ABI version 2 on Linux,
meaning that the polyfill for unveil() cannot restrict operations such
as truncate() (a limitation of Landlock's ABI from then). This means
that to restrict truncation operations Cosmopolitan instead has to ban
the syscall through a SECCOMP BPF filter, meaning that completely
legitimate truncate() calls are blocked
However, the newest version of the Landlock ABI (version 3) introduced
in Linux 6.2, released in February 2023, implements support for controlling truncation
operations. As such, the previous SECCOMP BPF truncate() filtering is
no longer needed when the new ABI is available
This patch implements unveil truncate support for Linux Landlock ABI
version 3
The rpath pledge as currently implemented in cosmopolitan does not
allow for usage of the old getdents syscall (0x4e), which is different
from the newer getdents syscall (0xd9) solely in that it does not
support 64-bit filesystems.
This means that, for example, old statically linked binaries cannot
use `readdir` and other such functions which use this syscall instead
of the more modern one, even though there is no threat in allowing
that syscall alongside the more modern one (except that the binary may
have issues with 64-bit filesystems, but that's a separate problem).
This patch fixes this.
The C standard states that, in the context of an x conversion
specifier given to scanf:
> Matches an optionally signed hexadecimal integer, whose format is
> the same as expected for the subject sequence of the strtoul
> function with the value 16 for the base argument.
- C standard, 7.23.6.2.11. The fscanf function
Cosmopolitan fails to do this, as 0 should be parsed as a 0 by such an
invocation of strtoul. Instead, cosmopolitan errors out as though such
input is invalid, which is wrong.
This means that a program such as this:
#include <stdio.h>
#undef NDEBUG
#include <assert.h>
int main()
{
int v = 0;
assert(sscanf("0", "%x", &v) == 1);
}
will not run correctly on cosmpolitan, instead failing the assertion.
This patch fixes this, along with the associated GitHub issue,
https://github.com/jart/cosmopolitan/issues/778
The C standard, when defining field width and precision, never gives
any limit on the values used for them (except, I believe, that they
fit within an int). In other words, if the user gives a field width of
32145 and a precision of 9218, the implementation has to handle these
values correctly. However, when such kinds of high numbers are used
with integer conversions, cosmopolitan is limited by an internal
buffer size of 144, which means precisions and field widths have to
fit within this, which violates the standard.
This means that for example, the following program:
#include <stdio.h>
#include <string.h>
int main()
{
char buf2[512] = {};
int i = snprintf(buf2, sizeof(buf2), "%.9999u", 10);
printf("%d %zu\n", i, strlen(buf2));
}
would, instead of printing "9999 511" (the correct output), instead
print "144 144" under cosmopolitan.
This patch fixes this.
The C standard states:
> The fprintf function returns the number of characters transmitted,
> or a negative value if an output or encoding error occurred or if
> the implementation does not support a specified width length
> modifier.
- C Standard, 7.23.6.1.15. The fprintf function
However, cosmopolitan fails to return a negative value in the case of
an output error, meaning that a program such as:
#include <stdio.h>
int main()
{
FILE *fp = fopen("/dev/full", "w");
setbuf(fp, NULL);
printf("fprintf: %d\n", fprintf(fp, "test\n"));
printf("fflush: %d\n", fflush(fp));
}
will, under cosmopolitan, print that no error occured in either of the
calls to fprintf and fflush.
This patch fixes this, along with the associated GitHub issue,
https://github.com/jart/cosmopolitan/issues/784
_PFLINK is supposed to automatically pull in required functions for
specific conversion specifiers. However, it fails to do so for the F,
G and E conversion specifiers.
This means that, for example, the following program:
#include <stdio.h>
int main()
{
printf("%F %G %E\n", .0, .0, .0);
}
fails to run correctly, printing "? ? ?" instead of
"0.000000 0 0.000000E+00".
This patch fixes this.
The C standard states:
> Unless explicitly stated otherwise, the functions described in this
> subclause order two wide characters the same way as two integers of
> the underlying integer type designated by wchar_t.
>
> [...]
>
> The wcscmp function returns an integer greater than, equal to, or
> less than zero, accordingly as the wide string pointed to by s1 is
> greater than, equal to, or less than the wide string pointed to by
> s2.
>
> [...]
>
> The wcsncmp function returns an integer greater than, equal to, or
> less than zero, accordingly as the possibly null-terminated array
> pointed to by s1 is greater than, equal to, or less than the
> possibly null-terminated array pointed to by s2.
- C Standard, 7.31.4.4. Wide string comparison functions
Cosmopolitan fails to obey this in cases where the difference between
two wide characters is larger than WCHAR_MAX.
This means that, for example, the following program:
#include <stdio.h>
#include <wchar.h>
#include <limits.h>
int main()
{
wchar_t str1[] = { WCHAR_MIN, L'\0' };
wchar_t str2[] = { WCHAR_MAX, L'\0' };
printf("%d\n", wcscmp(str1, str2));
printf("%d\n", wcsncmp(str1, str2, 2));
}
will print `1` twice, instead of the negative numbers mandated by the
standard (as WCHAR_MIN is less than WCHAR_MAX)
This patch fixes this, along with the associated Github issue,
https://github.com/jart/cosmopolitan/issues/783
The C standard states that, within the context of a printf-family
function, when specifying the precision of a conversion specification:
> A negative precision argument is taken as if the precision were
> omitted.
- Quoth the C Standard, 7.23.6.1. The fprintf function
Cosmopolitan instead treated negative precision arguments as
though they had a value of 0, which was non-conforming. This
change fixes that. Another issue we found relates to:
> For o conversion, it increases the precision, if and only if
> necessary, to force the first digit of the result to be a zero (if
> the value and precision are both 0, a single 0 is printed).
- Quoth the C standard, 7.23.6.1.6. The fprintf function
When printing numbers in their alternative form, with a precision and
with a conversion specifier of o (octal), Cosmopolitan wasn't following
the standard in two ways:
1. When printing a value with a precision that results in 0-padding,
cosmopolitan would still add an extra 0 even though this should be
done "if and only if necessary"
2. When printing a value of 0 with a precision of 0, nothing is
printed, even though the standard specifically states that a single
0 is printed in this case
This change fixes those issues too. Furthermore, regression tests have
been introduced to ensure Cosmopolitan continues to be conformant
going forward.
Fixes#774Fixes#782Fixes#789
Cosmopolitan now conforms to the C Standard 7.8.1 specification
of the PRI and SCN macros, because this change fixes a bug where
the FAST16 ones were incorrectly using the %hd specifier.
The standard states that, when the # flag is used:
> The result is converted to an "alternative form". [...] For x (or X)
conversion, a nonzero result has 0x (or 0X) prefixed to it.
- C standard, 7.23.6.1. The fprintf function
cosmopolitan fails to use the correct alternative form (0X) when the X
conversion specifier is used, instead using 0x, which is not
capitalized.
This patch fixes this, along with the several tests that test for the
wrong behavior.
We were checking for anonymous mappings earlier on Windows by seeing if
the file descriptor argument to mmap() was supplied as -1. This was not
correct. The proper thing to do is check `flags & MAP_ANONYMOUS`.
The C standard states, for conversions using the d, i, b, B, o, u, x or X conversion specifiers:
> The precision specifies the minimum number of digits to appear; if
> the value being converted can be represented in fewer digits, it is
> expanded with leading zeros.
- C standard, 7.23.6.1. The fprintf function
However, cosmopolitan currently suppresses the addition of leading
zeros when the minus flag is set. This is not reflected by anything
within the C standard, meaning that behavior is incorrect.
This patch fixes this.
* Implement S conversion specifier for printf-related functions
POSIX specifies that a conversion specifier of S must be interpreted
the same way as %ls. This patch implements this.
* clang-format
---------
Co-authored-by: Gavin Hayes <gavin@computoid.com>
Python triggered the undefined behavior previously since it appears to
be posting to a semaphore owned by a different process that wasn't set
to process shared mode. The performance loss to process shared futexes
is so low and semaphores are generally used for this purpose, so it'll
be much simpler to simply not impose undefined behavior here.
This fixes a bug where the caller's timeval will be clobbered on Linux.
The Kernel ABI *always* modifies the timeout argument but POSIX says it
should be a const parameter. The wrapper now handles the difference and
sys_select() may be used if obtaining the remainder on Linux is needed.
The system call wrapper was wrongfully reinterpreting kernel data. The
examples/sysinfo.c program is now updated to show how to correctly use
what's returned.
libc/sock/if.h is removed because:
- it contained only IFNAMSIZ, which was also defined in
libc/sock/struct/ifreq.h
- it was not included in any file EXCEPT libc/isystem/net/if.h, which
also included the libc/sock/struct/ifreq.h
hence we remove this file, and move the IFNAMSIZ definition to
libc/sock/struct/ifreq.h which is included in isystem.
IFNAMSIZ is defined in:
- libc/sock/if.h
- libc/sock/struct/ifreq.h
we add a check in the latter to avoid a complaint when using the
amalgamated header.
* [metal] Copy program pages to extended memory at startup
* [metal] Reclaim base memory pages for later app use
* [metal] Load program pages beyond 1st 440 KiB to extended memory
o//examples/hellolua.com now runs correctly under QEMU (in
legacy BIOS mode).
* [metal] Place GDT in read/write segment
The CPU absolutely needs to alter the GDT when loading the
task register (via ltr). To account for this, I move the
GDT into a read/write data section. There is still a "rump"
read-only GDT in the text section that is used by the real
mode bootloader.
We also delay the loading of the task register (ltr) until
after the IDT and TSS are finally set up.
* [metal] Get examples/vga2.c serial output working for UEFI boot
* [metal] Get examples/vga2.c VGA output working for UEFI boot
* [metal] Allow munmap() to reclaim dynamically allocated pages
* Place TLS sections right after .text, not after embedded zip file
Co-authored-by: tkchia <tkchia-cosmo@gmx.com>
- Improve compatibility with Blink virtual machine
- Add non-POSIX APIs for joining threads and signal masks
- Never ever use anything except 32-bit integers for atomics
- Add some `#undef` statements to workaround `ctags` problems
This change adds a double linked list of threads, so that pthread_exit()
will know when it should call exit() from an orphaned child. This change
also improves ftrace and strace logging.
- clock_nanosleep() is now much faster on OpenBSD and NetBSD
- Thread joining is now much faster on NetBSD
- FreeBSD timestamps are now more accurate
- Thread spawning now goes faster on XNU
- Clean up the clone() code
- Clean up sigaction() code
- Add a port scanner example
- Introduce a ParseCidr() API
- Clean up our futex abstraction code
- Fix a harmless integer overflow in ParseIp()
- Use kernel semaphores on NetBSD to make threads much faster
- Exhaustively document cancellation points
- Rename SIGCANCEL to SIGTHR just like BSDs
- Further improve POSIX thread cancellations
- Ensure asynchronous cancellations work correctly
- Elevate the quality of getrandom() and getentropy()
- Make futexes cancel correctly on OpenBSD 6.x and 7.x
- Add reboot.com and shutdown.com to examples directory
- Remove underscore prefix from awesome timespec_*() APIs
- Create assertions that help verify our cancellation points
- Remove bad timespec APIs (cmp generalizes eq/ne/gt/gte/lt/lte)
This change makes some miracle modifications to the System Five system
call support, which lets us have safe, correct, and atomic handling of
thread cancellations. It all turned out to be cheaper than anticipated
because it wasn't necessary to modify the system call veneers. We were
able to encode the cancellability of each system call into the magnums
found in libc/sysv/syscalls.sh. Since cancellations are so waq, we are
also supporting a lovely Musl Libc mask feature for raising ECANCELED.
This change includes a fix to Fetch() where an out of bounds memory read
could happen, when the reverse proxied endpoint omits the content-length
header. This caused a bunch of NUL chars to appear on TurfWar's /statusz
since it wouldn't actually overrun the buffer, and if it did it would've
been caught by MODE=asan builds.
All tests pass now under WSL2. They should pass under WSL1 too, but only
WSL2 is integrated into the test fleet right now. This change also fills
in some gaps in the error numbers.
Fixes#665
- ASAN memory morgue is now lockless
- Make C11 atomics header more portable
- Rewrote pthread keys support to be lockless
- Simplify Python's unicode table unpacking code
- Make crash report write(2) closer to being atomic
- Make it possible to strace/ftrace a single thread
- ASAN now checks nul-terminated strings fast and properly
- Windows fork() now restores TLS memory of calling thread
- Invent iso8601us() for faster timestamps
- Improve --strace descriptions of sigset_t
- Rebuild the Landlock Make bootstrap binary
- Introduce MODE=sysv for non-Windows builds
- Permit OFD fcntl() locks under pledge(flock)
- redbean can now protect your kernel from ddos
- Have vfork() fallback to sys_fork() not fork()
- Change kmalloc() to not die when out of memory
- Improve documentation for some termios functions
- Rewrite putenv() and friends to conform to POSIX
- Fix linenoise + strace verbosity issue on Windows
- Fix regressions in our ability to show backtraces
- Change redbean SetHeader() to no-op if value is nil
- Improve fcntl() so SQLite locks work in non-WAL mode
- Remove some unnecessary work during fork() on Windows
- Create redbean-based SSL reverse proxy for IPv4 TurfWar
- Fix ape/apeinstall.sh warning when using non-bash shells
- Add ProgramTrustedIp(), and IsTrustedIp() APIs to redbean
- Support $PWD, $UID, $GID, and $EUID in command interpreter
- Introduce experimental JTqFpD APE prefix for non-Windows builds
- Invent blackhole daemon for firewalling IP addresses via UNIX named socket
- Add ProgramTokenBucket(), AcquireToken(), and CountTokens() APIs to redbean
