Cosmopolitan currently doesn't support threads and it doesn't do
anything fancy in longjmp/setjmp so this change was simple to do
- localeconv
- _setjmp (same as setjmp)
- _longjmp (same as longjmp)
- strcoll (same as strcmp)
- flockfile (does nothing)
- funlockfile (does nothing)
- ftrylockfile (does nothing)
See #61
The ucontext_t data structure XNU passes us doesn't appear to be part of
known memory. So we can't use ASAN during the trampoline, which converts
it to a Linux ucontext_t data structure. Please note that this change
doesn't impact the signal handler itself, only the trampoline.
Here's why we got those `Killed: 11` failures on MacOS after modifying
the contentns of the redbean.com executable. If you were inserting a
small file, such as a HelloWorld.html file, then InfoZIP might have
decreased the size of the executable to less than what the Mach-O
section had been expecting.
That's because when zipobj.com put things like time zone data in the
executable, it aligned each zip file entry on a 64-byte boundary, simply
for the sake of readability in binary dumps. But when InfoZIP edited the
file it would rewrite every entry using ZIP's usual 2-byte alignment.
Thus causing shrinkage.
The solution was to reconfigure the linker script so that zip file bits
that get put into the executable at link-time, such as timezone data,
aren't officially part of the executable image, i.e. we don't want the
operating system to load that part.
The original decision to put the linked zip files into the .data section
was mostly made so that when the executable was run in its .com.dbg form
it would still have the zip entries be accessible, even though there was
tons of GNU debug data following the central directory. We're not going
to be able to do that. The .com executable should be the canonical
executable. We have really good tools for automatically attaching and
configuring GDB correctly with debug symbols even when the .com is run.
We'll have to rely on those in cases where zip embedding is used.
See #53
See #54
See #68
Right now we can't call poll() on file and fifos on Windows. So we work
around that by simply blocking on keyboard input. This is OK because on
Windows we can't use /proc/*/mem for realtime monitoring, so there's no
reason not to block.
See #65
- Polyfill ucontext_t on FreeBSD/OpenBSD/NetBSD
- Add tests confirming signals can edit CPU state
- Work towards supporting ZIP filesystem on bare metal
- Add more tinymath unit tests for POSIX conformance
- Add X87 and SSE status flags to crash report
- Fix some bugs in blinkenlights
- Fix llvm build breakage
Your Actually Portable Executables now contains a simple virtual memory
that works similarly to the Linux Kernel in the sense that it maps your
physical memory to negative addresses. This is needed to support mmap()
and malloc(). This functionality has zero code size impact. For example
the MODE=tiny LIFE.COM executable is still only 12KB in size.
The APE bootloader code has also been simplified to improve readibility
and further elevate the elegance by which we're able to support so many
platforms thereby enhancing verifiability so that we may engender trust
in this bootloading process.
We're now scrubbing environment variables in compile.com since gnu make
was not behaving as expected. It also appears there was a regression in
recent revisions that caused ASAN to be turned off for most binaries in
dbg mode, which has now been fixed. Cosmopolitan is fully ASAN hardened
down to the lowest level libraries and it doesn't need any interceptors
- Reduce full build latency from ~20s to ~18s
- Bring back silent mode if `make V=0` is passed
- Demodernize utimes() polyfill so it works RHEL5
- Delete some old shell scripts that are no longer needed
- Truncate long lines when outputting builds to Emacs buffers
You can now build Cosmopolitan with Clang:
make -j8 MODE=llvm
o/llvm/examples/hello.com
The assembler and linker code is now friendly to LLVM too.
So it's not needed to configure Clang to use binutils under
the hood. If you love LLVM then you can now use pure LLVM.
It turns out adding OpenBSD msyscall() origin verification broke the
--ftrace flag. The executable needs to issue raw syscalls while it's
rewriting itself. So they need to be in the same section, and that's
just plain simpler too.
Compilers like GCC require comments on lines like `#endif rdmsr`. Since
the rdmsr macro was only being used in arch_prctl(), I've localized the
macro, and I'm considering deleting arch_prctl() too, since there isn't
any way to have mem segments unfortunately across operating systems ;_;
The remaining changed lines are due to clang-format which runs on auto.
You can now use cosmopolitan.h with an ANSI C89 compiler like MSVC. The
Cosmopolitan codebase itself won't support being compiled that way. But
you can build objects that link against Cosmopolitan using any compiler
and you can furthermore use tools like IntelliSense that can't even GNU
See also #40
- Get ASAN working on Windows.
- Deleting directories and then recreating them with the same name in a
short period of time appears to be a no-no on Windows.
- There's no reason to call FlushFileBuffers on close() for pipes, and
it's harmful since it might block indefinitely for no good reason.
- Support deterministic stacks on OpenBSD
- Support OpenBSD system call origin verification
- Fix overrun by one in chibicc string token allocator
- Get all chibicc tests passing under Address Sanitizer
We now have an integration test that runs the amalgamated sources
through a C++ compiler, to prevent the recurrence of such issues.
Fixes#38
Thanks @gbbnfhb for the report!
This change enables Address Sanitizer systemically w/ `make MODE=dbg`.
Our version of Rust's `unsafe` keyword is named `noasan` which is used
for two functions that do aligned memory chunking, like `strcpy.c` and
we need to fix the tiny DEFLATE code, but that's it everything else is
fabulous you can have all the fischer price security blankets you need
Best of all is we're now able to use the ASAN data in Blinkenlights to
colorize the memory dumps. See the screenshot below of a test program:
https://justine.lol/blinkenlights/asan.png
Which is operating on float arrays stored on the stack, with red areas
indicating poisoned memory, and the green areas indicate valid memory.
The libm code from musl wasn't being used since most of these functions
are implemented using x87 which goes faster than a library intended for
risc machines.
We always favor calling functions like openat(), fstatat(), etc. because
Linux, XNU, FreeBSD, and OpenBSD all elected to support them, while some
systems like Android love them so much, that they stopped supporting the
old interfaces.
This change ensures that when dirfd is actually a dirfd and not AT_FDCWD
we'll do the right thing on Windows NT. We use an API that's been around
since Vista to accomplish that.
This change also adds exponential backoff to chdir() on Windows since it
seems almost as flaky on Windows 7 as the rmdir() function.
Justine says nay for the time being. Only XNU implements this. It's not
clear what ABI XNU is using but it's obviously not the one in the POSIX
documentation link below. Since all platforms implement vfork, it might
be better to empirically gauge the intersection of consensus which will
have better performance than this interface.
https://pubs.opengroup.org/onlinepubs/009696699/xrat/xsh_chap03.html
I rebooted Windows 7 and noticed that KERNEL32.DLL got relocated.
Microsoft needs to change and embrace the dream that was MULTICS.
Until then we'll need a better way to work around their dominion.
For the first time ever, all tests in this codebase now pass, when
run automatically on macos, freebsd, openbsd, rhel5, rhel7, alpine
and windows via the network using the runit and runitd build tools
- Fix vfork exec path etc.
- Add XNU opendir() support
- Add OpenBSD opendir() support
- Add Linux history to syscalls.sh
- Use copy_file_range on FreeBSD 13+
- Fix system calls with 7+ arguments
- Fix Windows with greater than 16 FDs
- Fix RUNIT.COM and RUNITD.COM flakiness
- Fix OpenBSD munmap() when files are mapped
- Fix long double so it's actually long on Windows
- Fix OpenBSD truncate() and ftruncate() thunk typo
- Let Windows fcntl() be used on socket files descriptors
- Fix Windows fstat() which had an accidental printf statement
- Fix RHEL5 CLOCK_MONOTONIC by not aliasing to CLOCK_MONOTONIC_RAW
This is wonderful. I never could have dreamed it would be possible
to get it working so well on so many platforms with tiny binaries.
Fixes#31Fixes#25Fixes#14
This change unbreaks test/libc/nexgen32e/memeqmask_test.c which appears
to currently be the only reference to this api. It was originally meant
to be used for speeding up terminal video rendering. We might delete it
Fixes#26
This bug impacts folks who purchased Intel chips made in 2011-2012.
We're now using `vxorps` instead of `vpxor` which is great since it
means we do not need to change `X86_HAVE(AVX)` to `X86_HAVE(AVX2)`,
because AVX2 is only available on Haswell and later.
Fixes#16
Windows 7 support is now confirmed to be working. However it comes at
the cost of needing to workaround this tragically oppressive behavior
which is imposed for reasons unknown.
Fixes#19
- Remove XD bit in page tables
- Fix cylinder+head+sector arithmetic
- Implement fstat() for serial file descriptors on metal
Here's how to boot an Actually Portable Executable in QEMU:
make -j12 o//tool/viz/deathstar.com
qemu-system-x86_64 -serial stdio -fda o//tool/viz/deathstar.com
Here's a screenshot of DEATHSTAR.COM booted in QEMU:
https://justine.lol/cosmopolitan/cosmo-metal-qemu.png
Thus metal support is in much better shape now, but still incomplete.
Only a few system calls have been polyfilled. To figure out which ones
your program needs, simply boot it in the blinkenlights emulator with a
breakpoint, and press CTRL-C to continue to the system call breakpoint.
If it doesn't break then you should be good. (Note: to emulate normally
you can press 'c' and use CTRL-T and ALT-T to tune the speed.)
m=tiny
make -j12 SILENT=0 MODE=$m \
o/$m/tool/build/blinkenlights.com \
o/$m/tool/viz/deathstar.com
o/$m/tool/build/blinkenlights.com \
-r -t -b systemfive.linux \
o/$m/tool/viz/deathstar.com
Thank @Theldus for the bug report that made this change possible.
Fixes#20 which explains this change further.
It turned out that the linker was doing the wrong with the amalgamation
library concerning weak stubs. A regression test has been added and new
binaries have been uploaded to https://justine.lol/cosmopolitan/
Ideally this should be fixed by building a tool that turns multiple .a
files into a single .a file with deduplication. As a workaround for now
the cosmopolitan.a build is restructured to not include LIBC_STUBS which
meant technical debt needed to be paid off where non-stub interfaces
were moved to LIBC_INTRIN and LIBC_NEXGEN32E.
Thank @PerfectProductions in #31 for the report!
This change pays off technical debt with the function -> DLL mappings in
libc/nt/master.sh, which was originally defined based on binary analysis
on Windows 10. It's now been updated so the kernel32/kernelbase/advapi32
imports should be exactly as they are written, on the MSDN documentation
and that wouldn't have been easy without Geoff Chappell's work thank him
https://www.geoffchappell.com/studies/windows/win32/index.htm
This program popped up on Hacker News recently. It's the only modern
compiler I've ever seen that doesn't have dependencies and is easily
modified. So I added all of the missing GNU extensions I like to use
which means it might be possible soon to build on non-Linux and have
third party not vendor gcc binaries.
A new rollup tool now exists for flattening out the headers in a way
that works better for our purposes than cpp. A lot of the API clutter
has been removed. APIs that aren't a sure thing in terms of general
recommendation are now marked internal.
There's now a smoke test for the amalgamation archive and gigantic
header file. So we can now guarantee you can use this project on the
easiest difficulty setting without the gigantic repository.
A website is being created, which is currently a work in progress:
https://justine.storage.googleapis.com/cosmopolitan/index.html
This is done without using Microsoft's internal APIs. MAP_PRIVATE
mappings are copied to the subprocess via a pipe, since Microsoft
doesn't want us to have proper COW pages. MAP_SHARED mappings are
remapped without needing to do any copying. Global variables need
copying along with the stack and the whole heap of anonymous mem.
This actually improves the reliability of the redbean http server
although one shouldn't expect 10k+ connections on a home computer
that isn't running software built to serve like Linux or FreeBSD.
blinkenlights now does a pretty good job emulating what happens when
binaries boot from BIOS into long mode. So it's been much easier to
debug the bare metal process and wrinkle out many issues.
This change includes many bug fixes, for the NT polyfills, strings,
memory, boot, and math libraries which were discovered by adding more
tools for recreational programming, such as PC emulation. Lemon has also
been vendored because it works so well at parsing languages.
- Emulator can now test the αcτµαlly pδrταblε εxεcµταblε bootloader
- Whipped up a webserver named redbean. It services 150k requests per
second on a single core. Bundling assets inside zip enables extremely
fast serving for two reasons. The first is that zip central directory
lookups go faster than stat() system calls. The second is that both
zip and gzip content-encoding use DEFLATE, therefore, compressed
responses can be served via the sendfile() system call which does an
in-kernel copy directly from the zip executable structure. Also note
that red bean zip executables can be deployed easily to all platforms,
since these native executables work on Linux, Mac, BSD, and Windows.
- Address sanitizer now works very well
I wanted a tiny scriptable meltdown proof way to run userspace programs
and visualize how program execution impacts memory. It helps to explain
how things like Actually Portable Executable works. It can show you how
the GCC generated code is going about manipulating matrices and more. I
didn't feel fully comfortable with Qemu and Bochs because I'm not smart
enough to understand them. I wanted something like gVisor but with much
stronger levels of assurances. I wanted a single binary that'll run, on
all major operating systems with an embedded GPL barrier ZIP filesystem
that is tiny enough to transpile to JavaScript and run in browsers too.
https://justine.storage.googleapis.com/emulator625.mp4
One of the benefits of implementing system call support from scratch is
that we're able to have embedded zip filesystem support which trickles
into libraries such as stdio, without unportable symbolic interposition.
It's also be great if we could say open("gs://bucket/object", O_RDONLY)
for seamless GCS, similar to Java NIO, but abstracted by the C library.
The binary system interfaces designed at Bell Labs were what helped our
friends like Linus Torvalds become successful. It's why Torvalds always
respected syscall abi and made every effort to not break the userspace.
Sadly OpenBSD developer Theo de Raadt thinks respecting Bell interfaces
is a risk for security and conjured up the Return Oriented Programming
bogeyman to justify his policies, per https://lwn.net/Articles/806776/
See libc/nexgen32e/gc.S where we use ROP concepts for garbage collection
in C due to our belief that powerful programming techniques can be good.
See https://opensource.googleblog.com/2017/03/operation-rosehub.html for
an example of something similar to rop but potentially more of a concern
