We now guarantee TMPDIR will be defined on a per build rule basis. It'll
be an absolute path. It'll be secure and unique. It'll be rm -rf'd after
the last shell script line in your build rule is executed. If $TMPDIR is
already defined, then it'll be created as a subdirectory of your $TMPDIR
and then replace the variable with the new definition. The Landlock Make
repository will be updated with examples shortly after this change which
shall be known as Landlock Make 1.1.1.
See #530
It turned out that specifying all SRCS and INCS as dependencies on the
pattern rules for all headers, caused `make` memory usage to skyrocket
from 40mb ot 160mb. This change also reduces the build graph another 4%.
This change introduces the nointernet() function which may be called to
prevent a process and its descendants from communicating with publicly
routable Internet addresses. GNU Make has been modified to always call
this function. In the future Landlock Make will have a way to whitelist
subnets to override this behavior, or disable it entirely. Support is
available for Linux only. Our firewall does not require root access.
Calling nointernet() will return control to the caller inside a new
process that has a SECCOMP BPF filter installed, which traps network
related system calls. Your original process then becomes a permanent
ptrace() supervisor that monitors all processes and threads descending
from the returned child. Whenever a networking system call happens the
kernel will stop the process and wakes up the monitor, which then peeks
into the child memory to read the sockaddr_in to determine if it's ok.
The downside to doing this is that there can be only one supervisor at a
time using ptrace() on a process. So this firewall won't be enabled if
you run make under strace or inside gdb. It also makes testing tricky.
This change also removes the futimens() call on the Landlock Make output
file workaround, since it caused problems with commands like fixupobj
which modify-in-place. It turns out if a file is opened for writing and
then no writes actually occur, then the modified time doesn't change.
- 10.5% reduction of o//depend dependency graph
- 8.8% reduction in latency of make command
- Fix issue with temporary file cleanup
There's a new -w option in compile.com that turns off the recent
Landlock output path workaround for "good commands" which do not
unlink() the output file like GNU tooling does.
Our new GNU Make unveil sandboxing appears to have zero overhead
in the grand scheme of things. Full builds are pretty fast since
the only thing that's actually slowed us down is probably libcxx
make -j16 MODE=rel
RL: took 85,732,063µs wall time
RL: ballooned to 323,612kb in size
RL: needed 828,560,521µs cpu (11% kernel)
RL: caused 39,080,670 page faults (99% memcpy)
RL: 350,073 context switches (72% consensual)
RL: performed 0 reads and 11,494,960 write i/o operations
pledge() and unveil() no longer consider ENOSYS to be an error.
These functions have also been added to Python's cosmo module.
This change also removes some WIN32 APIs and System Five magnums
which we're not using and it's doubtful anyone else would be too
This change fixes Landlock Make so that only the output target file is
unveiled, rather than unveiling the directory that contains it. This
gives us a much stronger sandbox. It also helped identify problematic
build code in our repo that should have been using o/tmp instead.
Landlock isn't able to let us unveil files that don't exist. Even if
they do, then once a file is deleted, the sandboxing for it goes away.
This caused problems for Landlock Make because tools like GNU LD will
repeatedly delete and recreate the output file. This change uses the
compile.com wrapper to ensure on changes happen to the output inode.
New binary available on https://justine.lol/make/Fixes#528
The whole repository is now buildable with GNU Make Landlock sandboxing.
This proves that no Makefile targets exist which touch files other than
their declared prerequisites. In order to do this, we had to:
1. Stop code morphing GCC output in package.com and instead run a
newly introduced FIXUPOBJ.COM command after GCC invocations.
2. Disable all the crumby Python unit tests that do things like create
files in the current directory, or rename() files between folders.
This ended up being a lot of tests, but most of them are still ok.
3. Introduce an .UNSANDBOXED variable to GNU Make to disable Landlock.
We currently only do this for things like `make tags`.
4. This change deletes some GNU Make code that was preventing the
execve() optimization from working. This means it should no longer
be necessary in most cases for command invocations to be indirected
through the cocmd interpreter.
5. Missing dependencies had to be declared in certain places, in cases
where they couldn't be automatically determined by MKDEPS.COM
6. The libcxx header situation has finally been tamed. One of the
things that makes this difficult is MKDEPS.COM only wants to
consider the first 64kb of a file, in order to go fast. But libcxx
likes to have #include lines buried after huge documentation.
7. An .UNVEIL variable has been introduced to GNU Make just in case
we ever wish to explicitly specify additional things that need to
be whitelisted which aren't strictly prerequisites. This works in
a manner similar to the recently introduced .EXTRA_PREREQS feature.
There's now a new build/bootstrap/make.com prebuilt binary available. It
should no longer be possible to write invalid Makefile code.
- Introduce __assert_disable global
- Improve strsignal() thread safety
- Make system call tracing thread safe
- Fix SO_RCVTIMEO / SO_SNDTIMEO on Windows
- Refactor DescribeFoo() functions into one place
- Fix fork() on Windows when TLS and MAP_STACK exist
- Round upwards in setsockopt(SO_RCVTIMEO) on Windows
- Disable futexes on OpenBSD which seem extremely broken
- Implement a better kludge for monotonic time on Windows
This change hardens the code for opening /zip/ files using the system
call interface. Thread safety and signal safety has been improved for
file descriptors in general. We now document fixed addresses that are
needed for low level allocations.
This change turns symbol table compression back on using Puff, which
noticeably reduces the size of programs like redbean and Python. The
redbean web server receives some minor API additions for controlling
things like SSL in addition to filling gaps in the documentation.
- Document redbean's argon2 module
- Fix regressions in cthreads library
- Make testlib work better with threads
- Give the cthreads library lots of love
- Remove some of the stdio assembly code
- Implement getloadavg() across platforms
- Code size optimizations for errnos, etc.
- Only check for signals in main thread on Windows
- Make errnos for dup2 / dup3 consistent with posix
This change also fixes a bug in the argon2 module, where the NUL
terminator was being included in the hash encoded ascii string. This
shouldn't require any database migrations to folks who found this module
and productionized it, since the argon2 library treats it as a c string.
The "no modify self" variant of Actually Portable Executable is now
supported on all platforms. If you use `$(APE_NO_MODIFY_SELF)` then
ld.bfd will embed a 4096 byte ELF binary and a 4096 byte Macho file
which are installed on the fly to ${TMPDIR:-/tmp}, which enables us
launch the executable, without needing to copy the whole executable
To prevent it from copying a tiny executable to your temp directory
you need to install the `ape` command (renamed from ape-loader), to
a system path. For example:
# FreeBSD / NetBSD / OpenBSD
make -j8 o//ape/ape
cp o//ape/ape /usr/bin/ape
# Mac OS
# make -j8 o//ape/ape.macho
curl https://justine.lol/ape.macho >/usr/bin/ape
chmod +x /usr/bin/ape
On Linux you can get even more performance with the new binfmt_misc
support which makes launching non-modifying APE binaries as fast as
launching ELF executables. Running the following command:
# Linux
ape/apeinstall.sh
Will copy APE loader to /usr/bin/ape and register with binfmt_misc
Lastly, this change also fixes a really interesting race condition
with OpenBSD thread joining.
The `HDRS` and/or `INCS` variables need to be defined in order for
`mkdeps.com` to compute build dependencies for `o//depend`. We're using
INCS here since makeint.h is the evil kind of header that isn't actually
defining a library interface but rather is a hodgepodge of common code.
- Get threads working on NetBSD
- Get threads working on OpenBSD
- Fix Emacs config for Emacs v28
- Improve --strace logging of sigset_t
- Improve --strace logging of struct stat
- Improve memory safety of DescribeThing functions
- Refactor auto stack allocation into LIBC_RUNTIME
- Introduce shell.com example which works on Windows
- Refactor __strace_thing into DescribeThing functions
- Document the CHECK macros and improve them in NDEBUG mode
- Rewrite MAP_STACK so it uses FreeBSD behavior across platforms
- Deprecate and discourage the use of MAP_GROWSDOWN (it's weird)
- Add hierarchical auto-completion to redbean's repl
- Fetch latest localtime() and strftime() from Eggert
- Shave a few milliseconds off redbean start latency
- Fix redbean repl with multi-line statements
- Make the Lua unix module code more elegant
- Harden Lua data structure serialization
This change makes further effort towards improving our poll()
implementation on the New Technology. The stdin worker didn't work out
so well for Python so it's not being used for now. System call tracing
with the --strace flag should now be less noisy now on Windows unless
you modify the strace.internal.h defines to turn on some optional ones
that are most useful for debugging the system call wrappers.
You can now call functions like fork() from Lua and it'll work across
all supported platforms, including Windows. This gives you a level of
control of the system that Lua traditionally hasn't been able to have
due to its focus on old portable stdio rather modern POSIX APIs. Demo
code has been added to redbean-demo.com to show how it works.
This change also modifies Lua so that integer literals with a leading
zero will be interpreted as octal. That should help avoid shooting in
the foot with POSIX APIs that frequently use octal mode bits.
This change fixes a bug in opendir(".") on New Technology.
Lastly, redbean will now serve crash reports to private network IPs.
This is consistent with other frameworks. However that isn't served
to public IPs unless the -E flag is passed to redbean at startup.
- Introduce fast spinlock API
- Double rand64() perf w/ spinlock
- Improve raise() on New Technology
- Support gettid() across platforms
- Implement SA_NODEFER on New Technology
- Move the lock intrinsics into LIBC_INTRIN
- Make SIGTRAP recoverable on New Technology
- Block SIGCHLD in wait4() on New Technology
- Add threading prototypes for XNU and FreeBSD
- Rewrite abort() fixing its minor bugs on XNU/NT
- Shave down a lot of the content in libc/bits/bits.h
- Let signal handlers modify CPU registers on New Technology
This change fixes minor bugs and adds a feature, which lets us store the
ELF symbol table, inside the ZIP directory. We use the path /zip/.symtab
which can be safely removed using a zip editing tool, to make the binary
smaller after compilation. This supplements the existing method of using
a separate .com.dbg file, which is still supported. The intent is people
don't always know that it's a good idea to download the debug file. It's
not great having someone's first experience be a crash report, that only
has numbers rather than symbols. This will help fix that!
Continuous Integration (via runit and runitd) is now re-enabled on win7
and win10. The `make test` command, which runs the tests on all systems
is now the fastest and most stable it's been since the project started.
UBSAN is now enabled in MODE=dbg in addition to ASAN. Many instances of
undefined behavior have been removed. Mostly things like passing a NULL
argument to memcpy(), which works fine with Cosmopolitan Libc, but that
doesn't prevents the compiler from being unhappy. There was an issue w/
GNU make where static analysis claims a sprintf() call can overflow. We
also now have nicer looking crash reports on Windows since uname should
now be supported and msys64 addr2line works reliably.
- Simulate SIGPIPE on Windows NT
- Fix commandv() regression on Windows NT
- Fix sigprocmask() strace bug on OpenBSD
- Add many more system calls to --strace logging
- Make errno state more pristine in redbean strace
