It turned out that Landlock Make hasn't been applying sandboxing for a
while, due to a mistyped if statement for `$(USE_SYSTEM_TOOLCHAIN)` it
should have had the opposite meaning. Regressions in the build configs
have been fixed. The rmrf() function works better now. The rm.com tool
works according to POSIX with the exception of supporting prompts.
This change (1) upgrades to OpenBSD's newer kernel ABIs, and (2)
modifies APE to have a read-only data segment. Doing this required
creating APE Loader v1.1, which is backwards and forwards compatible
with the previous version.
If you've run the following commands in the past to install your APE
Loader systemwide, then you need to run them again. Ad-hoc installations
shouldn't be impacted. It's also recommended that APE binaries be remade
after upgrading, since they embed old versions of the APE Loader.
ape/apeuninstall.sh
ape/apeinstall.sh
This change does more than just fix OpenBSD. The new loader is smarter
and more reliable. We're now able create much tinier ELF and Mach-O data
structures than we could before. Both APE Loader and execvpe() will now
normalize ambiguous argv[0] resolution the same way as the UNIX shell.
Badness with TLS linkage has been solved.
Fixes#826
This change fixes an issue with the tcflow() magic numbers that was
causing bash to freeze up on Linux. While auditing termios polyfills,
several other issues were identified with XNU/BSD compatibility.
Out of an abundance of caution this change undefines as much surface
area from libc/calls/struct/termios.h as possible, so that autoconf
scripts are less likely to detect non-POSIX teletypewriter APIs that
haven't been polyfilled by Cosmopolitan.
This is a *breaking change* for your static archives in /opt/cosmos if
you use the cosmocc toolchain. That's because this change disables the
ioctl() undiamonding trick for code outside the monorepo, specifically
because it'll lead to brittle ABI breakages like this. If you're using
the cosmocc toolchain, you'll need to rebuild libraries like ncurses,
readline, etc. Yes diamonds cause bloat. To work around that, consider
using tcgetwinsize() instead of ioctl(TIOCGWINSZ) since it'll help you
avoid pulling every single ioctl-related polyfill into the linkage.
The cosmocc script was specifying -DNDEBUG for some reason. It's fixed.
This change takes an entirely new approach to the incremental linking of
pkzip executables. The assets created by zipobj.com are now treated like
debug data. After a .com.dbg is compiled, fixupobj.com should be run, so
it can apply fixups to the offsets and move the zip directory to the end
of the file. Since debug data doesn't get objcopy'd, a new tool has been
introduced called zipcopy.com which should be run after objcopy whenever
a .com file is created. This is all automated by the `cosmocc` toolchain
which is rapidly becoming the new recommended approach.
This change also introduces the new C23 checked arithmetic macros.
This change improves the way internal APIs are being hidden behind the
`COSMO` define. The cosmo.h header will take care of defining that, so
that a separate define statement isn't needed. This change also does a
lot more to define which APIs are standard, and which belong to Cosmo.
This change integrates e58abc1110b335a3341e8ad5821ad8e3880d9bb2 from
https://github.com/ahgamut/musl-cross-make/ which fixes the issues we
were having with our C language extension for symbolic constants. This
change also performs some code cleanup and bug fixes to getaddrinfo().
It's now possible to compile projects like ncurses, readline and python
without needing to patch anything upstream, except maybe a line or two.
Pretty soon it should be possible to build a Linux distro on Cosmo.
In order to improve our chances of success building other open source
projects we shouldn't define APIs that'll lead any ./configure script
astray. For example:
- brk() and sbrk() can break mac/windows support
- syscall() is a superb way to break portability
- arch_prctl() is the greatest of all horror shows
- Work towards improving non-optimized build support
- Introduce MODE=zero which is -O0 without ASAN/UBSAN
- Use system GCC when ~/.cosmo.mk has USE_SYSTEM_TOOLCHAIN=1
- Have package.com check .privileged code doesn't call non-privileged
This change implements a new approach to function call logging, that's
based on the GCC flag: -fpatchable-function-entry. Read the commentary
in build/config.mk to learn how it works.
- Now using 10x better GCD semaphores
- We now generate Linux-like thread ids
- We now use fast system clock / sleep libraries
- The APE M1 loader now generates Linux-like stacks
This change progresses our AARCH64 support:
- The AARCH64 build and tests are now passing
- Add 128-bit floating-point support to printf()
- Fix clone() so it initializes cosmo's x28 TLS register
- Fix TLS memory layout issue with aarch64 _Alignas vars
- Revamp microbenchmarking tools so they work on aarch64
- Make some subtle improvements to aarch64 crash reporting
- Make kisdangerous() memory checks more accurate on aarch64
- Remove sys_open() since it's not available on Linux AARCH64
This change makes general improvements to Cosmo and Redbean:
- Introduce GetHostIsa() function in Redbean
- You can now feature check using pledge(0, 0)
- You can now feature check using unveil("",0)
- Refactor some more x86-specific asm comments
- Refactor and write docs for some libm functions
- Make the mmap() API behave more similar to Linux
- Fix WIFSIGNALED() which wrongly returned true for zero
- Rename some obscure cosmo keywords from noFOO to dontFOO
There's a new program named ape/ape-m1.c which will be used to build an
embeddable binary that can load ape and elf executables. The support is
mostly working so far, but still chasing down ABI issues.
This is an official tagged version. The 1.1 version used earlier was
informal. Formal versions will always have major.minor.patch going
forward. See https://github.com/jart/blink/tags 1.0.0 coming soon.
Your APE binaries will now run on eight hardware architectures. The
tradeoff is this change adds about 800kb of bloat to the executable
which to be fair is still more than half as small as Go hello world
Your APE shell script extracts an embedded build of Blink, based on
`$(uname -m)`. The Blink binaries are statically-linked (manylinux)
which means, they'll run on every distro's stock install, including
Apple M1's that haven't enabled Rosetta. Blink's emulation has been
designed to be nearly as fast as Rosetta, using a JIT technique for
faster startup time. Blink has also been tested more thoroughly, w/
emulating Cosmopolitan software, since we're the ones who wrote it.
* [metal] Copy program pages to extended memory at startup
* [metal] Reclaim base memory pages for later app use
* [metal] Load program pages beyond 1st 440 KiB to extended memory
o//examples/hellolua.com now runs correctly under QEMU (in
legacy BIOS mode).
* [metal] Place GDT in read/write segment
The CPU absolutely needs to alter the GDT when loading the
task register (via ltr). To account for this, I move the
GDT into a read/write data section. There is still a "rump"
read-only GDT in the text section that is used by the real
mode bootloader.
We also delay the loading of the task register (ltr) until
after the IDT and TSS are finally set up.
* [metal] Get examples/vga2.c serial output working for UEFI boot
* [metal] Get examples/vga2.c VGA output working for UEFI boot
* [metal] Allow munmap() to reclaim dynamically allocated pages
* Place TLS sections right after .text, not after embedded zip file
Co-authored-by: tkchia <tkchia-cosmo@gmx.com>
- Improve compatibility with Blink virtual machine
- Add non-POSIX APIs for joining threads and signal masks
- Never ever use anything except 32-bit integers for atomics
- Add some `#undef` statements to workaround `ctags` problems
Files beginning with "MZqFpD" are Actually Portable Executables,
which have a printf statement in the first 4096 bytes with octal
codes that specify the ELF header. APE also specifies `jartsr='`
as an alternative prefix, intended for binaries that do not want
to be identified as Windows executables. Like the \177ELF magic,
all these prefixes decode as x86 jump instructions that could be
used for 16-bit bootloaders or 32-bit / 64-bit flat executables.
Most importantly they provide a fallback path for Thompson shell
copmatible command interpreters, which do not require a shebang,
e.g. bash, zsh, fish, bourne, almquist, etc. Please note that in
order to meet the requirements of POSIX.1, the single quote must
be followed by a newline character, before any null bytes occur.
See also: https://www.austingroupbugs.net/view.php?id=1250
- Invent iso8601us() for faster timestamps
- Improve --strace descriptions of sigset_t
- Rebuild the Landlock Make bootstrap binary
- Introduce MODE=sysv for non-Windows builds
- Permit OFD fcntl() locks under pledge(flock)
- redbean can now protect your kernel from ddos
- Have vfork() fallback to sys_fork() not fork()
- Change kmalloc() to not die when out of memory
- Improve documentation for some termios functions
- Rewrite putenv() and friends to conform to POSIX
- Fix linenoise + strace verbosity issue on Windows
- Fix regressions in our ability to show backtraces
- Change redbean SetHeader() to no-op if value is nil
- Improve fcntl() so SQLite locks work in non-WAL mode
- Remove some unnecessary work during fork() on Windows
- Create redbean-based SSL reverse proxy for IPv4 TurfWar
- Fix ape/apeinstall.sh warning when using non-bash shells
- Add ProgramTrustedIp(), and IsTrustedIp() APIs to redbean
- Support $PWD, $UID, $GID, and $EUID in command interpreter
- Introduce experimental JTqFpD APE prefix for non-Windows builds
- Invent blackhole daemon for firewalling IP addresses via UNIX named socket
- Add ProgramTokenBucket(), AcquireToken(), and CountTokens() APIs to redbean
- use PAGE_RSRV bit (originally only for blinkenlights),
rather than PAGE_V bit, to indicate that a virtual address
page has been reserved — this should allow a program to
create & reserve inaccessible "guard pages"
- mark page table entries for non-code pages with PAGE_XD bit,
which should be supported on (circa) post-2004 x86-64 CPUs
libc/intrin/interrupts.S should not be linked in unless
an IDT (or TSS) is explicitly requested somewhere (i.e.
it should probably not be a mandatory module).
The metal boot sector code was wrapped in a single-quoted
string, like so:
MZqFpD='
...MZ HEADER...
...E_LFANEW...
...BIOS BOOT SECTOR...'
This might break the shell code loader if the boot sector code
contains a 0x27 (single quote) byte. This patch wraps the
boot sector code in a here-document instead:
MZqFpD='
...MZ HEADER...' <<'@'
...E_LFANEW...
...BIOS BOOT SECTOR...
@
This is harder to break — when interpreted as a shell script,
the code can only accidentally terminate the here-document
if there is a "\n@\n" sequence inside the BIOS boot portion.
This change reduces the .bss memory requirement for all executables by
O(64kb). The brk system calls are now fully tested and figured out and
might be useful for tiny programs that only target System Five.
Doing this makes binaries tinier, since we don't need to have all the
extra code for supporting a 32-bit address space. It also benefits us
because we're able to use WIN32 futexes, which makes locking simpler.
b69f3d2488 is what officially ended our
Windows 7 support. This change is merely a formalization. You can use
old versions of Cosmo now and forevermore if you need Windows 7 since
our repository is hermetic and vendors all its dependencies.
Won't fix#617
You can now run bare metal on bare metal!
* Fix handling of int 0x15 eax = 0xe820 memory map
* Fix some issues in initial page table creation
* hello4.com now works outside emulators
* Ensure area for identity page tables are zeroed first
* Simplify logic for creating page table entries, this partly
reverts 577c0f6226
* Add degenerate MBR partition entry, to ease testing
Co-authored-by: tkchia <tkchia-cosmo@gmx.com>
This makes breaking changes to add underscores to many non-standard
function names provided by the c library. MODE=tiny is now tinier and
we now use smaller locks that are better for tiny apps in this mode.
Some headers have been renamed to be in the same folder as the build
package, so it'll be easier to know which build dependency is needed.
Certain old misguided interfaces have been removed. Intel intrinsics
headers are now listed in libc/isystem (but not in the amalgamation)
to help further improve open source compatibility. Header complexity
has also been reduced. Lastly, more shell scripts are now available.
This change tunes the default stack size for the outside world to 8mb
while at the same time, reducing Cosmopolitan's default stack size to
64kb. You can override the stack size using STATIC_STACK_SIZE(). Your
build scripts should point to o//ape/public/ape.lds
This change also fixes the definition of SOMAXCONN and removes AF_RDS
since it's not polyfilled and Python 3.11 complained.
This allows e.g. `qemu-system-x86_64 -s o/examples/hello.com
-serial stdio` to work without having to add extra padding to
the end of the `hello.com` "disk image".
(The sector count computation is divided among two instructions
in the assembly code. This is done on purpose, to prevent an
ASCII 0x27 (single quote) byte from appearing in the bare
metal loader code, which will break the shell script loader.
There is probably a better way.)
Co-authored-by: tkchia <tkchia-cosmo@gmx.com>
- We now kill the program on violations like OpenBSD
- We now print a message explaining which promise is needed
- This change also fixes a linkage bug with thread local storage
- Your sigaction() handlers should now be more thread safe
A new `__pledge_mode` global has been introduced to make pledge() more
customizable on Linux. For example:
__attribute__((__constructor__)) static void init(void) {
__pledge_mode = SECCOMP_RET_ERRNO | EPERM;
}
Can be used to restore our old permissive pledge() behavior.
The whole repository is now buildable with GNU Make Landlock sandboxing.
This proves that no Makefile targets exist which touch files other than
their declared prerequisites. In order to do this, we had to:
1. Stop code morphing GCC output in package.com and instead run a
newly introduced FIXUPOBJ.COM command after GCC invocations.
2. Disable all the crumby Python unit tests that do things like create
files in the current directory, or rename() files between folders.
This ended up being a lot of tests, but most of them are still ok.
3. Introduce an .UNSANDBOXED variable to GNU Make to disable Landlock.
We currently only do this for things like `make tags`.
4. This change deletes some GNU Make code that was preventing the
execve() optimization from working. This means it should no longer
be necessary in most cases for command invocations to be indirected
through the cocmd interpreter.
5. Missing dependencies had to be declared in certain places, in cases
where they couldn't be automatically determined by MKDEPS.COM
6. The libcxx header situation has finally been tamed. One of the
things that makes this difficult is MKDEPS.COM only wants to
consider the first 64kb of a file, in order to go fast. But libcxx
likes to have #include lines buried after huge documentation.
7. An .UNVEIL variable has been introduced to GNU Make just in case
we ever wish to explicitly specify additional things that need to
be whitelisted which aren't strictly prerequisites. This works in
a manner similar to the recently introduced .EXTRA_PREREQS feature.
There's now a new build/bootstrap/make.com prebuilt binary available. It
should no longer be possible to write invalid Makefile code.
Rosetta doesn't correctly respect the startup registers as defined in LC_UNIXTHREAD
which makes platform detection go awry. But at least Rosetta appears to consistently
set rbx to 0x00000000ffffffff and rdx to 0x0000000000000001 at startup for every
x64 executable I could get my hands on. So we use that to detect Rosetta's presence
and set up the correct registers for XNU.
This should strike a better compromise that keeps people happy about the
security of APE when it extracts the loader. This way systems with users
who aren't trusted (e.g. CPanel) won't be at any risk of compromise when
there isn't an `ape` loader on the system `$PATH`.
This change also bumps redbean up to 2.0.3
Calls to lock/unlock functions are now NOPs by default. The first time
clone() is called, they get turned into CALL instructions. Doing this
caused funcctions like fputc() to shrink from 85 bytes to 45+4 bytes.
Since the ANSI solution of `(__threaded && lock())` inlines os much
superfluous binary content into functions all over the place.
- Finish cleaning up the stdio unlocked APIs
- Make __cxa_finalize() properly thread safe
- Don't log locks if threads aren't being used
- Add some more mutex guards to places using _mmi
- Specific lock names now appear in the --ftrace logs
- Fix mkdeps.com generating invalid Makefiles sometimes
- Simplify and fix bugs in the test runner infrastructure
- Fix issue where sometimes some functions wouldn't be logged
Microsoft's version of Linux requires that the ELF program headers say
that the executable has 4096 byte alignment. However, it doesn't force
us to pad the binary with NOPs to a page-aligned size.
- Write tests for cthreads
- Fix bugs in pe2.com tool
- Fix ASAN issue with GetDosEnviron()
- Consolidate the cthread header files
- Some code size optimizations for MODE=
- Attempted to squash a tls linker warning
- Attempted to get futexes working on FreeBSD
- Document redbean's argon2 module
- Fix regressions in cthreads library
- Make testlib work better with threads
- Give the cthreads library lots of love
- Remove some of the stdio assembly code
- Implement getloadavg() across platforms
- Code size optimizations for errnos, etc.
- Only check for signals in main thread on Windows
- Make errnos for dup2 / dup3 consistent with posix
This change also fixes a bug in the argon2 module, where the NUL
terminator was being included in the hash encoded ascii string. This
shouldn't require any database migrations to folks who found this module
and productionized it, since the argon2 library treats it as a c string.
The longjmp relocation in the master boot record coincidentally had a
quote character in it, which caused the master boot record to be used
when interpreter by the shell. The solution, is to move the grub stub
below the long mode loader so the real mode loader relocation is near
the master boot record. This change includes a regression test.
- Fix some minor issues in ar.com
- Have execve() look for `ape` command
- Rewrite NT paths using /c/ rather /??/c:/
- Replace broken GCC symlinks with .sym files
- Rewrite $PATH environment variables on startup
- Make $(APE_NO_MODIFY_SELF) the default bootloader
- Add all build command dependencies to build/bootstrap
- Get the repository mostly building from source on non-Linux
- Implement openpty()
- Add `--assimilate` flag to APE bootloader
- Restore Linux vDSO clock_gettime() support
- Use `$(APE_NO_MODIFY_SELF)` on more programs
- Add FreeBSD-specific mmap() flags
- Reduce size of the APE loader from 8kb to 4kb
- Work towards fixing the Makefile build on WSL
- Automate testing of APE no-modify-self behaviors
- Make the ape.S shell script code cleaner and tinier
- Improve the APE sanity check to test behavior better
- Fixed issue with ShowCrashReports() sigaltstack() on BSDs
- Delete symbols for S_MODE magnums which wasted compile time
If you checked out yesterday's APE commit, please run:
rm -f /usr/bin/ape o/tmp/ape /tmp/ape "${TMPDIR:-/tmp}/ape"
Because this change fixes certain aspects of the new ABI. We don't have
automated migrations for APE loader versions yet. Thanks! You can also
download prebuilt binaries here:
- https://justine.lol/ape.elf (Linux/FreeBSD/NetBSD/OpenBSD)
- https://justine.lol/ape.macho (Apple)
Install the appropriate one as `/usr/bin/ape`.
The "no modify self" variant of Actually Portable Executable is now
supported on all platforms. If you use `$(APE_NO_MODIFY_SELF)` then
ld.bfd will embed a 4096 byte ELF binary and a 4096 byte Macho file
which are installed on the fly to ${TMPDIR:-/tmp}, which enables us
launch the executable, without needing to copy the whole executable
To prevent it from copying a tiny executable to your temp directory
you need to install the `ape` command (renamed from ape-loader), to
a system path. For example:
# FreeBSD / NetBSD / OpenBSD
make -j8 o//ape/ape
cp o//ape/ape /usr/bin/ape
# Mac OS
# make -j8 o//ape/ape.macho
curl https://justine.lol/ape.macho >/usr/bin/ape
chmod +x /usr/bin/ape
On Linux you can get even more performance with the new binfmt_misc
support which makes launching non-modifying APE binaries as fast as
launching ELF executables. Running the following command:
# Linux
ape/apeinstall.sh
Will copy APE loader to /usr/bin/ape and register with binfmt_misc
Lastly, this change also fixes a really interesting race condition
with OpenBSD thread joining.
- Document more compiler flags
- Expose new __print_maps() api
- Better overflow checking in mmap()
- Improve the shell example somewhat
- Fix minor runtime bugs regarding stacks
- Make kill() on fork()+execve()'d children work
- Support CLONE_CHILD_CLEARTID for proper joining
- Fix recent possible deadlock regression with --ftrace
- Document sigaction()
- Simplify New Technology fork() code
- Testing and many bug fixes for mprotect()
- Distribute Intel Xed ILD in the amalgamation
- Turn Xed enums into defines to avoid DWARF bloat
- Improve polyfilling of SA_SIGINFO on BSDs and fix bugs
- setpgid(getpid(), getpid()) on Windows will ignore CTRL-C
- Work around issues relating to NT mappings being executable
- Permit automatic executable stack override via `ape_stack_pf`
Now that we have understandable system call tracing on Windows, this
change rewrites many of the polyfill internals for that platform, to
help things get closer to tip top shape. Support for complex forking
scenarios had been in a regressed state for quite some time. Now, it
works! Subsequent changes should be able to address the performance.
- Fix build flakes
- Polyfill SIGWINCH on Windows
- Fix an execve issue on Windows
- Make strerror show more information
- Improve cmd.exe setup/teardown on Windows
- Support bracketed paste mode in Blinkenlights
- Show keyboard shortcuts in Blinkenlights status bar
- Fixed copy_file_range() and copyfile() w/ zip filesystem
- Size optimize GetDosArgv() to keep life.com 12kb in size
- Improve Blinkenlights ability to load weird ELF executables
- Fix program_executable_name and add GetInterpreterExecutableName
- Make Python in tiny mode fail better if docstrings are requested
- Update Python test exclusions in tiny* modes such as tinylinux
- Add bulletproof unbreakable kprintf() troubleshooting function
- Remove "oldskool" keyword from ape.S for virus scanners
- Fix issue that caused backtraces to not print sometimes
- Improve Blinkenlights serial uart character i/o
- Make clock_gettime() not clobber errno on xnu
- Improve sha256 cpuid check for old computers
- Integrate some bestline linenoise fixes
- Show runit process names better in htop
- Remove SIGPIPE from ShowCrashReports()
- Make realpath() not clobber errno
- Avoid attaching GDB on non-Linux
- Improve img.com example
This commit makes numerous refinements to cosmopolitan memory handling.
The default stack size has been reduced from 2mb to 128kb. A new macro
is now provided so you can easily reconfigure the stack size to be any
value you want. Work around the breaking change by adding to your main:
STATIC_STACK_SIZE(0x00200000); // 2mb stack
If you're not sure how much stack you need, then you can use:
STATIC_YOINK("stack_usage_logging");
After which you can `sort -nr o/$MODE/stack.log`. Based on the unit test
suite, nothing in the Cosmopolitan repository (except for Python) needs
a stack size greater than 30kb. There are also new macros for detecting
the size and address of the stack at runtime, e.g. GetStackAddr(). We
also now support sigaltstack() so if you want to see nice looking crash
reports whenever a stack overflow happens, you can put this in main():
ShowCrashReports();
Under `make MODE=dbg` and `make MODE=asan` the unit testing framework
will now automatically print backtraces of memory allocations when
things like memory leaks happen. Bugs are now fixed in ASAN global
variable overrun detection. The memtrack and asan runtimes also handle
edge cases now. The new tools helped to identify a few memory leaks,
which are fixed by this change.
This change should fix an issue reported in #288 with ARG_MAX limits.
Fixing this doubled the performance of MKDEPS.COM and AR.COM yet again.
