This change reconciles our pledge() implementation with the OpenBSD
kernel source code. We now a polyfill that's much closer to OpenBSD's
behavior. For example, it was discovered that "stdio" permits threads.
There were a bunch of Linux system calls that needed to be added, like
sched_yield(). The exec / execnative category division is now dropped.
We're instead using OpenBSD's "prot_exec" promise for launching APE
binaries and dynamic shared objects. We also now filter clone() flags.
The pledge.com command has been greatly improved. It now does unveiling
by default when Landlock is available. It's now smart enough to unveil a
superset of paths that OpenBSD automatically unveils with pledge(), such
as /etc/localtime. pledge.com also now checks if the executable being
launched is a dynamic shared object, in which case it unveils libraries.
These changes now make it possible to pledge curl on ubuntu 20.04 glibc:
pledge.com -p 'stdio rpath prot_exec inet dns tty sendfd recvfd' \
curl -s https://justine.lol/hello.txt
Here's what pledging curl on Alpine 3.16 with Musl Libc looks like:
pledge.com -p 'stdio rpath prot_exec dns inet' \
curl -s https://justine.lol/hello.txt
Here's what pledging curl.com w/ ape loader looks like:
pledge.com -p 'stdio rpath prot_exec dns inet' \
o//examples/curl.com https://justine.lol/hello.txt
The most secure sandbox, is curl.com converted to static ELF:
o//tool/build/assimilate.com o//examples/curl.com
pledge.com -p 'stdio rpath dns inet' \
o//examples/curl.com https://justine.lol/hello.txt
A weird corner case needed to be handled when resolving symbolic links
during the unveiling process, that's arguably a Landlock bug. It's not
surprising since Musl and Glibc are also inconsistent here too.
We now rewrite the binary image at runtime on Windows and XNU to change
mov %fs:0,%reg instructions to use %gs instead. There's also simpler
threading API introduced by this change and it's called _spawn() and
_join(), which has replaced most clone() usage.
- Introduce __assert_disable global
- Improve strsignal() thread safety
- Make system call tracing thread safe
- Fix SO_RCVTIMEO / SO_SNDTIMEO on Windows
- Refactor DescribeFoo() functions into one place
- Fix fork() on Windows when TLS and MAP_STACK exist
- Round upwards in setsockopt(SO_RCVTIMEO) on Windows
- Disable futexes on OpenBSD which seem extremely broken
- Implement a better kludge for monotonic time on Windows
This change fixes a regression in unix.connect() caused by the recent
addition of UNIX domain sockets. The BSD finger command has been added
to third_party for fun and profit. A new demo has been added to redbean
showing how a protocol as simple as finger can be implemented.
- Finish cleaning up the stdio unlocked APIs
- Make __cxa_finalize() properly thread safe
- Don't log locks if threads aren't being used
- Add some more mutex guards to places using _mmi
- Specific lock names now appear in the --ftrace logs
- Fix mkdeps.com generating invalid Makefiles sometimes
- Simplify and fix bugs in the test runner infrastructure
- Fix issue where sometimes some functions wouldn't be logged
- Write tests for cthreads
- Fix bugs in pe2.com tool
- Fix ASAN issue with GetDosEnviron()
- Consolidate the cthread header files
- Some code size optimizations for MODE=
- Attempted to squash a tls linker warning
- Attempted to get futexes working on FreeBSD
- Document more compiler flags
- Expose new __print_maps() api
- Better overflow checking in mmap()
- Improve the shell example somewhat
- Fix minor runtime bugs regarding stacks
- Make kill() on fork()+execve()'d children work
- Support CLONE_CHILD_CLEARTID for proper joining
- Fix recent possible deadlock regression with --ftrace
- Add hierarchical auto-completion to redbean's repl
- Fetch latest localtime() and strftime() from Eggert
- Shave a few milliseconds off redbean start latency
- Fix redbean repl with multi-line statements
- Make the Lua unix module code more elegant
- Harden Lua data structure serialization
- Improve serialization
- Add Benchmark() API to redbean
- Refactor UNIX API to be assert() friendly
- Make the redbean Lua REPL print data structures
- Fix recent regressions in linenoise reverse search
- Add -i flag so redbean can be a language interpreter
- Expand redbean UNIX module
- Expand redbean documentation
- Ensure Lua copyright is embedded in binary
- Increase the PATH_MAX limit especially on NT
- Use column major sorting for linenoise completions
- Fix some suboptimalities in redbean's new UNIX API
- Figured out right flags for Multics newline in raw mode
- Get clone() working on FreeBSD
- Increase some Python build quotas
- Add more atomic builtins to chibicc
- Fix ASAN poisoning of alloca() memory
- Make MODE= mandatory link path tinier
- Improve the examples folder a little bit
- Start working on some more resource limits
- Make the linenoise auto-complete UI as good as GNU readline
- Update compile.com, avoiding AVX codegen on non-AVX systems
- Make sure empty path to syscalls like opendir raises ENOENT
- Correctly polyfill ENOENT vs. ENOTDIR on the New Technology
- Port bestline's paredit features to //third_party/linenoise
- Remove workarounds for RHEL 5.0 bugs that were fixed in 5.1
You can now interact with the global web server state on the command
line, which the web server is running. This supports Emacs shortcuts
with history, readline parity, <tab> completions, plus hints. Enjoy!
You can now call functions like fork() from Lua and it'll work across
all supported platforms, including Windows. This gives you a level of
control of the system that Lua traditionally hasn't been able to have
due to its focus on old portable stdio rather modern POSIX APIs. Demo
code has been added to redbean-demo.com to show how it works.
This change also modifies Lua so that integer literals with a leading
zero will be interpreted as octal. That should help avoid shooting in
the foot with POSIX APIs that frequently use octal mode bits.
This change fixes a bug in opendir(".") on New Technology.
Lastly, redbean will now serve crash reports to private network IPs.
This is consistent with other frameworks. However that isn't served
to public IPs unless the -E flag is passed to redbean at startup.
- Fix sigsuspend() on XNU
- Fix strsignal() on non-Linux
- Add unit tests for strsignal()
- Add unit tests for setitimer()
- Add unit tests for sigsuspend()
- Rewrite setitimer() for New Technology
- Rewrite nanosleep() for New Technology
- Polyfill SIGALRM on the New Technology
- select(0,0,0,0) on NT now calls pause()
- Remove some NTDLL calls that aren't needed
- Polyfill SA_NOCLDWAIT on the New Technology
- Polyfill SA_RESETHAND on the New Technology
- Polyfill sigprocmask() on the New Technology
- Polyfill SIGCHLD+SIG_IGN on the New Technology
- Polyfill SA_RESTART masking on the New Technology
- Deliver console signals from main thread on New Technology
- Document SA_RESTART behavior w/ @sarestartable / @norestart
- System call trace in MODE=dbg now prints inherited FDs and signal mask
This change fixes minor bugs and adds a feature, which lets us store the
ELF symbol table, inside the ZIP directory. We use the path /zip/.symtab
which can be safely removed using a zip editing tool, to make the binary
smaller after compilation. This supplements the existing method of using
a separate .com.dbg file, which is still supported. The intent is people
don't always know that it's a good idea to download the debug file. It's
not great having someone's first experience be a crash report, that only
has numbers rather than symbols. This will help fix that!
You can now use the hardest fastest and most dangerous language there is
with Cosmopolitan. So far about 75% of LLVM libcxx has been added. A few
breaking changes needed to be made to help this go smoothly.
- Rename nothrow to dontthrow
- Rename nodiscard to dontdiscard
- Add some libm functions, e.g. lgamma, nan, etc.
- Change intmax_t from int128 to int64 like everything else
- Introduce %jjd formatting directive for int128_t
- Introduce strtoi128(), strtou128(), etc.
- Rename bsrmax() to bsr128()
Some of the templates that should be working currently are std::vector,
std::string, std::map, std::set, std::deque, etc.
Continuous Integration (via runit and runitd) is now re-enabled on win7
and win10. The `make test` command, which runs the tests on all systems
is now the fastest and most stable it's been since the project started.
UBSAN is now enabled in MODE=dbg in addition to ASAN. Many instances of
undefined behavior have been removed. Mostly things like passing a NULL
argument to memcpy(), which works fine with Cosmopolitan Libc, but that
doesn't prevents the compiler from being unhappy. There was an issue w/
GNU make where static analysis claims a sprintf() call can overflow. We
also now have nicer looking crash reports on Windows since uname should
now be supported and msys64 addr2line works reliably.
Now that we have understandable system call tracing on Windows, this
change rewrites many of the polyfill internals for that platform, to
help things get closer to tip top shape. Support for complex forking
scenarios had been in a regressed state for quite some time. Now, it
works! Subsequent changes should be able to address the performance.
This is similar to the --ftrace (c function call trace) flag, except
it's less noisy since it only logs system calls to stderr. Having this
flag is valuable because (1) system call tracing tells us a lot about
the behavior of complex programs and (2) it's usually very hard to get
system call tracing on various operating systems, e.g. strace, ktrace,
dtruss, truss, nttrace, etc. Especially on Apple platforms where even
with the special boot trick, debuggers still aren't guaranteed to work.
make -j8 o//examples
o//examples/hello.com --strace
This is enabled by default in MODE=, MODE=opt, and MODE=dbg. In MODE=dbg
extra information will be printed.
make -j8 MODE=dbg o/dbg/examples
o/dbg/examples/hello.com --strace |& less
This change also changes:
- Rename IsText() → _istext()
- Rename IsUtf8() → _isutf8()
- Fix madvise() on Windows NT
- Fix empty string case of inet_ntop()
- vfork() wrapper now saves and restores errno
- Update xsigaction() to yoink syscall support
- Add Lua backtraces to redbean!
- Wipe serving keys after redbean forks
- Audit redbean to remove free via exit
- Log SSL client ciphersuite preferences
- Increase ASAN malloc() backtrace depth
- Make GetSslRoots() behave as a singleton
- Move leaks.c from LIBC_TESTLIB to LIBC_LOG
- Add undocumented %n to printf() for newlines
- Fix redbean memory leak reindexing inode change
- Fix redbean memory leak with Fetch() DNS object
- Restore original environ after __cxa_finalize()
- Make backtrace always work after __cxa_finalize()
- Introduce COUNTEXPR() diagnostic / benchmark tool
- Fix a few more instances of errno being clobbered
- Consolidate the ANSI color disabling internal APIs
- Fix build flakes
- Polyfill SIGWINCH on Windows
- Fix an execve issue on Windows
- Make strerror show more information
- Improve cmd.exe setup/teardown on Windows
- Support bracketed paste mode in Blinkenlights
- Show keyboard shortcuts in Blinkenlights status bar
- Fixed copy_file_range() and copyfile() w/ zip filesystem
- Size optimize GetDosArgv() to keep life.com 12kb in size
- Improve Blinkenlights ability to load weird ELF executables
- Fix program_executable_name and add GetInterpreterExecutableName
- Make Python in tiny mode fail better if docstrings are requested
- Update Python test exclusions in tiny* modes such as tinylinux
- Add bulletproof unbreakable kprintf() troubleshooting function
- Remove "oldskool" keyword from ape.S for virus scanners
- Fix issue that caused backtraces to not print sometimes
- Improve Blinkenlights serial uart character i/o
- Make clock_gettime() not clobber errno on xnu
- Improve sha256 cpuid check for old computers
- Integrate some bestline linenoise fixes
- Show runit process names better in htop
- Remove SIGPIPE from ShowCrashReports()
- Make realpath() not clobber errno
- Avoid attaching GDB on non-Linux
- Improve img.com example
We defined `noinline` as an abbreviation for the longer version
`__attribute__((__noinline__))` which caused name clashes since
third party codebases often write it as `__attribute__((noinline))`.
This commit makes numerous refinements to cosmopolitan memory handling.
The default stack size has been reduced from 2mb to 128kb. A new macro
is now provided so you can easily reconfigure the stack size to be any
value you want. Work around the breaking change by adding to your main:
STATIC_STACK_SIZE(0x00200000); // 2mb stack
If you're not sure how much stack you need, then you can use:
STATIC_YOINK("stack_usage_logging");
After which you can `sort -nr o/$MODE/stack.log`. Based on the unit test
suite, nothing in the Cosmopolitan repository (except for Python) needs
a stack size greater than 30kb. There are also new macros for detecting
the size and address of the stack at runtime, e.g. GetStackAddr(). We
also now support sigaltstack() so if you want to see nice looking crash
reports whenever a stack overflow happens, you can put this in main():
ShowCrashReports();
Under `make MODE=dbg` and `make MODE=asan` the unit testing framework
will now automatically print backtraces of memory allocations when
things like memory leaks happen. Bugs are now fixed in ASAN global
variable overrun detection. The memtrack and asan runtimes also handle
edge cases now. The new tools helped to identify a few memory leaks,
which are fixed by this change.
This change should fix an issue reported in #288 with ARG_MAX limits.
Fixing this doubled the performance of MKDEPS.COM and AR.COM yet again.
- python now mixes audio 10x faster
- python octal notation is restored
- chibicc now builds code 3x faster
- chibicc now has help documentation
- chibicc can now generate basic python bindings
- linenoise now supports some paredit-like features
See #141
This program usually runs once at the begininng of each GNU Make
invocation. It generates an o//depend file with 170,000 lines of
Makefile code to define source -> headers relationships.
This change makes that take 650 milliseconds rather than 1,100ms
by improving the performance of strstr(), using longsort(), plus
migrating to the new append library.
The APE_NO_MODIFY_SELF loader payload has been moved out of the examples
folder and improved so that it works on BSD systems, and permits general
elf program headers. This brings its quality up enough that it should be
acceptable to use by default for many programs, e.g. Python, Lua, SQLite
and Python. It's the responsibility of the user to define an appropriate
TMPDIR if /tmp is considered an adversarial environment. Mac OS shall be
supported by APE_NO_MODIFY_SELF soon.
Fixes and improvements have been made to program_executable_name as it's
now the one true way to get the absolute path of the executing image.
This change fixes a memory leak in linenoise history loading, introduced
by performance optimizations in 51904e2687
This change fixes a longstanding regression with Mach system calls, that
23ae9dfceb back in February which impacted
our sched_yield() implementation, which is why no one noticed until now.
The Blinkenlights PC emulator has been improved. We now fix rendering on
XNU and BSD by not making the assumption that the kernel terminal driver
understands UTF8 since that seems to break its internal modeling of \r\n
which is now being addressed by using \e[𝑦H instead. The paneling is now
more compact in real mode so you won't need to make your font as tiny if
you're only emulating an 8086 program. The CLMUL ISA is now emulated too
This change also makes improvement to time. CLOCK_MONOTONIC now does the
right thing on Windows NT. The nanosecond time module functions added in
Python 3.7 have been backported.
This change doubles the performance of Argon2 password stretching simply
by not using its copy_block and xor_block helper functions, as they were
trivial to inline thus resulting in us needing to iterate over each 1024
byte block four fewer times.
This change makes code size improvements. _PyUnicode_ToNumeric() was 64k
in size and now it's 10k. The CJK codec lookup tables now use lazy delta
zigzag deflate (δzd) encoding which reduces their size from 600k to 200k
plus the code bloat caused by macro abuse in _decimal.c is now addressed
so our fully-loaded statically-linked hermetically-sealed Python virtual
interpreter container is now 9.4 megs in the default build mode and 5.5m
in MODE=tiny which leaves plenty of room for chibicc.
The pydoc web server now accommodates the use case of people who work by
SSH'ing into a different machine w/ python.com -m pydoc -p8080 -h0.0.0.0
Finally Python Capsulae delenda est and won't be supported in the future
- Python static hello world now 1.8mb
- Python static fully loaded now 10mb
- Python HTTPS client now uses MbedTLS
- Python REPL now completes import stmts
- Increase stack size for Python for now
- Begin synthesizing posixpath and ntpath
- Restore Python \N{UNICODE NAME} support
- Restore Python NFKD symbol normalization
- Add optimized code path for Intel SHA-NI
- Get more Python unit tests passing faster
- Get Python help() pagination working on NT
- Python hashlib now supports MbedTLS PBKDF2
- Make memcpy/memmove/memcmp/bcmp/etc. faster
- Add Mersenne Twister and Vigna to LIBC_RAND
- Provide privileged __printf() for error code
- Fix zipos opendir() so that it reports ENOTDIR
- Add basic chmod() implementation for Windows NT
- Add Cosmo's best functions to Python cosmo module
- Pin function trace indent depth to that of caller
- Show memory diagram on invalid access in MODE=dbg
- Differentiate stack overflow on crash in MODE=dbg
- Add stb_truetype and tools for analyzing font files
- Upgrade to UNICODE 13 and reduce its binary footprint
- COMPILE.COM now logs resource usage of build commands
- Start implementing basic poll() support on bare metal
- Set getauxval(AT_EXECFN) to GetModuleFileName() on NT
- Add descriptions to strerror() in non-TINY build modes
- Add COUNTBRANCH() macro to help with micro-optimizations
- Make error / backtrace / asan / memory code more unbreakable
- Add fast perfect C implementation of μ-Law and a-Law audio codecs
- Make strtol() functions consistent with other libc implementations
- Improve Linenoise implementation (see also github.com/jart/bestline)
- COMPILE.COM now suppresses stdout/stderr of successful build commands
This change reinvents all the GNU Readline features I discovered that I
couldn't live without, e.g. UTF-8, CTRL-R search and CTRL-Y yanking. It
now feels just as good in terms of user interface from the subconscious
workflow perspective. It's real nice to finally have an embeddable line
reader that's actually good with a 30 kb footprint and a bsd-2 license.
This change adds a directory to the examples folder, explaining how the
new Python compiler may be used. Some of the bugs with Python binaries
have been addressed but overall it's still a work in progress.
We can now link even smaller Python binaries. For example, the hello.com
program in the Python build directory is a compiled linked executable of
hello.py which just prints hello world. Using decentralized sections, we
can make that binary 1.9mb in size (noting that python.com is 6.3 megs!)
This works for nontrivial programs too. For example, say we want an APE
binary that's equivalent to python.com -m http.server. Our makefile now
builds such a binary using the new launcher and it's only 3.2mb in size
since Python sources get turned into ELF objects, which tell our linker
that we need things like native hashing algorithm code.
The ZIP filesystem has a breaking change. You now need to use /zip/ to
open() / opendir() / etc. assets within the ZIP structure of your APE
binary, instead of the previous convention of using zip: or zip! URIs.
This is needed because Python likes to use absolute paths, and having
ZIP paths encoded like URIs simply broke too many things.
Many more system calls have been updated to be able to operate on ZIP
files and file descriptors. In particular fcntl() and ioctl() since
Python would do things like ask if a ZIP file is a terminal and get
confused when the old implementation mistakenly said yes, because the
fastest way to guarantee native file descriptors is to dup(2). This
change also improves the async signal safety of zipos and ensures it
doesn't maintain any open file descriptors beyond that which the user
has opened.
This change makes a lot of progress towards adding magic numbers that
are specific to platforms other than Linux. The philosophy here is that,
if you use an operating system like FreeBSD, then you should be able to
take advantage of FreeBSD exclusive features, even if we don't polyfill
them on other platforms. For example, you can now open() a file with the
O_VERIFY flag. If your program runs on other platforms, then Cosmo will
automatically set O_VERIFY to zero. This lets you safely use it without
the need for #ifdef or ifstatements which detract from readability.
One of the blindspots of the ASAN memory hardening we use to offer Rust
like assurances has always been that memory passed to the kernel via
system calls (e.g. writev) can't be checked automatically since the
kernel wasn't built with MODE=asan. This change makes more progress
ensuring that each system call will verify the soundness of memory
before it's passed to the kernel. The code for doing these checks is
fast, particularly for buffers, where it can verify 64 bytes a cycle.
- Correct O_LOOP definition on NT
- Introduce program_executable_name
- Add ASAN guards to more system calls
- Improve termios compatibility with BSDs
- Fix bug in Windows auxiliary value encoding
- Add BSD and XNU specific errnos and open flags
- Add check to ensure build doesn't talk to internet
Actually Portable Python is now outperforming the Python binaries
that come bundled with Linux distros, at things like HTTP serving.
You can now have a fully featured Python install in just one .com
file that runs on six operating systems and is about 10mb in size.
With tuning, the tiniest is ~1mb. We've got most of the libraries
working, including pysqlite, and the repl now feels very pleasant.
The things you can't do quite yet are: threads and shared objects
but that can happen in the future, if the community falls in love
with this project and wants to see it developed further. Changes:
- Add siginterrupt()
- Add sqlite3 to Python
- Add issymlink() helper
- Make GetZipCdir() faster
- Add tgamma() and finite()
- Add legacy function lutimes()
- Add readlink() and realpath()
- Use heap allocations when appropriate
- Reorganize Python into two-stage build
- Save Lua / Python shell history to dotfile
- Integrate Python Lib embedding into linkage
- Make isregularfile() and isdirectory() go faster
- Make Python shell auto-completion work perfectly
- Make crash reports work better if changed directory
- Fix Python+NT open() / access() flag overflow error
- Disable Python tests relating to \N{LONG NAME} syntax
- Have Python REPL copyright() show all notice embeddings
The biggest technical challenge at the moment is working around
when Python tries to be too clever about filenames.
This change gets the Python codebase into a state where it conforms to
the conventions of this codebase. It's now possible to include headers
from Python, without worrying about ordering. Python has traditionally
solved that problem by "diamonding" everything in Python.h, but that's
problematic since it means any change to any Python header invalidates
all the build artifacts. Lastly it makes tooling not work. Since it is
hard to explain to Emacs when I press C-c C-h to add an import line it
shouldn't add the header that actually defines the symbol, and instead
do follow the nonstandard Python convention.
Progress has been made on letting Python load source code from the zip
executable structure via the standard C library APIs. System calss now
recognizes zip!FILENAME alternative URIs as equivalent to zip:FILENAME
since Python uses colon as its delimiter.
Some progress has been made on embedding the notice license terms into
the Python object code. This is easier said than done since Python has
an extremely complicated ownership story.
- Some termios APIs have been added
- Implement rewinddir() dirstream API
- GetCpuCount() API added to Cosmopolitan Libc
- More bugs in Cosmopolitan Libc have been fixed
- zipobj.com now has flags for mangling the path
- Fixed bug a priori with sendfile() on certain BSDs
- Polyfill F_DUPFD and F_DUPFD_CLOEXEC across platforms
- FIOCLEX / FIONCLEX now polyfilled for fast O_CLOEXEC changes
- APE now supports a hybrid solution to no-self-modify for builds
- Many BSD-only magnums added, e.g. O_SEARCH, O_SHLOCK, SF_NODISKIO
This change enables SSL compression. It significantly reduces the
network load of the testing infrastructure, for free, since this
revision didn't need to change any runit protocol code. However we
turn it off by default in redbean since no browsers support it.
It turns out that some TLSv1.0 clients (e.g. curl command on RHEL5) will
send an SSLv2-style ClientHello. These types of clients are usually ten+
years old and were designed to interop with servers ten years older than
them. Your redbean is now able to interop with these clients even though
redbean doesn't actually support SSLv2 or SSLv3. Please note that the -B
flag may be passed to disable this along with TLSv1.0, TLSv1.1, 3DES, &c
The following Lua APIs have been added to redbean:
- ProgramSslCompression(bool)
- ProgramSslCiphersuite(name:str)
- ProgramSslPresharedKey(key:str,identity:str)
Lastly the DHE ciphersuites have been enabled. IANA recommends DHE and
with old clients like RHEL5 it's the only perfect forward secrecy they
implement.
This change makes SSL virtual hosting possible. You can now load
multiple certificates for multiple domains and redbean will just
figure out which one to use, even if you only have 1 ip address.
You can also use a jumbo certificate that lists all your domains
in the the subject alternative names.
This change also makes performance improvements to MbedTLS. Here
are some benchmarks vs. cc1920749e
BEFORE AFTER (microsecs)
suite_ssl.com 2512881 191738 13.11x faster
suite_pkparse.com 36291 3295 11.01x faster
suite_x509parse.com 854669 120293 7.10x faster
suite_pkwrite.com 6549 1265 5.18x faster
suite_ecdsa.com 53347 18778 2.84x faster
suite_pk.com 49051 18717 2.62x faster
suite_ecdh.com 19535 9502 2.06x faster
suite_shax.com 15848 7965 1.99x faster
suite_rsa.com 353257 184828 1.91x faster
suite_x509write.com 162646 85733 1.90x faster
suite_ecp.com 20503 11050 1.86x faster
suite_hmac_drbg.no_reseed.com 19528 11417 1.71x faster
suite_hmac_drbg.nopr.com 12460 8010 1.56x faster
suite_mpi.com 687124 442661 1.55x faster
suite_hmac_drbg.pr.com 11890 7752 1.53x faster
There aren't any special tricks to the performance imporvements.
It's mostly due to code cleanup, assembly and intel instructions
like mulx, adox, and adcx.
This change boosts SSL handshake performance from 2,627 to ~10,000 per
second which is the same level of performance as NGINX at establishing
secure connections. That's impressive if we consider that redbean is a
forking frontend application server. This was accomplished by:
1. Enabling either SSL session caching or SSL tickets. We choose to
use tickets since they reduce network round trips too and that's
a more important metric than wrk'ing localhost.
2. Fixing mbedtls_mpi_sub_abs() which is the most frequently called
function. It's called about 12,000 times during an SSL handshake
since it's the basis of most arithmetic operations like addition
and for some strange reason it was designed to make two needless
copies in addition to calling malloc and free. That's now fixed.
3. Improving TLS output buffering during the SSL handshake only, so
that only a single is write and read system call is needed until
blocking on the ping pong.
redbean will now do a better job wiping sensitive memory from a child
process as soon as it's not needed. The nice thing about fork is it's
much faster than reverse proxying so the goal is to use the different
address spaces along with setuid() to minimize the risk that a server
key will be compromised in the event that application code is hacked.
The following Lua APIs have been added:
- IsDaemon() → bool
- ProgramPidPath(str)
The following Lua hooks have been added:
- OnClientConnection(ip:int,port:int,serverip:int,serverport:int) → bool
- OnProcessCreate(pid:int,ip:int,port:int,serverip:int,serverport:int)
- OnProcessDestroy(pid:int)
- OnServerStart()
- OnServerStop()
- OnWorkerStart()
- OnWorkerStop()
redbean now does a better job at applying gzip on the fly from the local
filesystem, using a streaming chunked api with constant memory, which is
useful for doing things like serving a 4gb text file off NFS, and having
it start transmitting in milliseconds. redbean will also compute entropy
on the beginnings of files to determine if compression is profitable.
This change pays off technical debts relating to memory, such as relying
on exit() to free() allocations. That's now mostly fixed so it should be
easier now to spot memory leaks in malloc traces.
This change also fixes bugs and makes improvements to our SSL support.
Uniprocess mode failed handshakes are no longer an issue. Token Alpn is
offered so curl -v looks less weird. Hybrid SSL certificate loading is
now smarter about naming conflicts. Self-signed CA root anchors will no
longer be delivered to the client during the handshake.
Your redbean can now interoperate with clients that require TLS crypto.
This is accomplished using a protocol polyglot that lets us distinguish
between HTTP and HTTPS regardless of the port number. Certificates will
be generated automatically, if none are supplied by the user. Footprint
increases by only a few hundred kb so redbean in MODY=tiny is now 1.0mb
- Add lseek() polyfills for ZIP executable
- Automatically polyfill /tmp/FOO paths on NT
- Fix readdir() / ftw() / nftw() bugs on Windows
- Introduce -B flag for slower SSL that's stronger
- Remove mbedtls features Cosmopolitan doesn't need
- Have base64 decoder support the uri-safe alternative
- Remove Truncated HMAC because it's forbidden by the IETF
- Add all the mbedtls test suites and make them go 3x faster
- Support opendir() / readdir() / closedir() on ZIP executable
- Use Everest for ECDHE-ECDSA because it's so good it's so good
- Add tinier implementation of sha1 since it's not worth the rom
- Add chi-square monte-carlo mean correlation tests for getrandom()
- Source entropy on Windows from the proper interface everyone uses
We're continuing to outperform NGINX and other servers on raw message
throughput. Using SSL means that instead of 1,000,000 qps you can get
around 300,000 qps. However redbean isn't as fast as NGINX yet at SSL
handshakes, since redbean can do 2,627 per second and NGINX does 4.3k
Right now, the SSL UX story works best if you give your redbean a key
signing key since that can be easily generated by openssl using a one
liner then redbean will do all the things that are impossibly hard to
do like signing ecdsa and rsa certificates that'll work in chrome. We
should integrate the let's encrypt acme protocol in the future.
Live Demo: https://redbean.justine.lol/
Root Cert: https://redbean.justine.lol/redbean1.crt
redbean lua handlers that perform sql queries can do 400k qps.
We now use a separate compile-time options for SQLite, when building the
SQLite shell versus building the production web serving code. It doesn't
seem appropriate for something like redbean to include backups, progress
callbacks, query completion, profiling, EXPLAIN, ALTER, ANALYZE, VACUUM,
etc. since those tasks are better left to the sqlite3.com shell program.
Lua SQLite pointer APIs have been removed since we're not using threads.
The Lua APIs for installing update / commit / rollback hooks are removed
due to a general sense of disagreement and an overall lack of comfort.
Full-Text Search and R*Tree are as large as the rest of SQLite combined.
Turning those off keeps redbean under 1mb when built for MODE=tiny which
is nice for marketing purposes.
If you need something that was removed, file an issue, and we'll add it.
- Better UBSAN error messages
- POSIX Advisory Locks polyfills
- Move redbean manual to /.help.txt
- System call memory safety in ASAN mode
- Character classification now does UNICODE
This change brings page faults for precompressed static asset serving
down from 27 to 20 (or fewer) after fork. This is more of an art than
science. Hopefully Blinkenlights can visualize page faults soon.
- Fix regression with `%lu`
- Added some more headers witnessed in the wild
- Added `-M INT` option to redbean to tune max payload size
- Work around InfoZIP 256 character limit on comment line size
redbean improvements:
- Explicitly disable corking
- Simulate Python regex API for Lua
- Send warmup requests in main process on startup
- Add Class-A granular IPv4 network classification
- Add /statusz page so you can monitor your redbean's health
- Fix regressions on OpenBSD/NetBSD caused by recent changes
- Plug Authorization header into Lua GetUser and GetPass APIs
- Recognize X-Forwarded-{For,Host} from local reverse proxies
- Add many additional functions to redbean Lua server page API
- Report resource usage of child processes on `/` listing page
- Introduce `-a` flag for logging child process resource usage
- Introduce `-t MILLIS` flag and `ProgramTimeout(ms)` init API
- Introduce `-H "Header: value"` flag and `ProgramHeader(k,v)` API
Cosmopolitan Libc improvements:
- Make strerror() simpler
- Make inet_pton() not depend on sscanf()
- Fix OpenExecutable() which broke .data section earlier
- Fix stdio in cases where it overflows kernel tty buffer
- Fix bugs in crash reporting w/o .com.dbg binary present
- Add polyfills for SO_LINGER, SO_RCVTIMEO, and SO_SNDTIMEO
- Polyfill TCP_CORK on BSD and XNU using TCP_NOPUSH magnums
New netcat clone in examples/nc.c:
While testing some of the failure conditions for redbean, I noticed that
BusyBox's `nc` command is pretty busted, if you use it as an interactive
tool, rather than having it be part of a pipeline. Unfortunately this'll
only work on UNIX since Windows doesn't let us poll on stdio and sockets
at the same time because I don't think they want tools like this running
on their platform. So if you want forbidden fruit, it's here so enjoy it
The most exciting improvement is dynamic pages will soon be able to use
the executable itself as an object store. it required a heroic technique
for overcoming ETXTBSY restrictions which lets us open the executable in
read/write mode, which means (1) wa can restore the APE header, and (2)
we can potentially containerize redbean extension code so that modules
you download for your redbean online will only impact your redbean.
Here's a list of breaking changes to redbean:
- Remove /tool/net/ prefix from magic ZIP paths
- GetHeader() now returns NIL if header is absent
Here's a list of fixes and enhancements to redbean:
- Support 64-bit ZIP archives
- Record User-Agent header in logs
- Add twelve error handlers to accept()
- Display octal st_mode on listing page
- Show ZIP file comments on listing page
- Restore APE MZ header on redbean startup
- Track request count on redbean index page
- Report server uptime on redbean index page
- Don't bind server socket using SO_REUSEPORT
- Fix#151 where Lua LoadAsset() could free twice
- Report rusage accounting when workers exit w/ -vv
- Use ZIP iattr field as text/plain vs. binary hint
- Add ParseUrl() API for parsing things like a.href
- Add ParseParams() API for parsing HTTP POST bodies
- Add IsAcceptablePath() API for checking dots, etc.
- Add IsValidHttpToken() API for validating sane ASCII
- Add IsAcceptableHostPort() for validating HOST[:PORT]
- Send 400 response to HTTP/1.1 requests without a Host
- Send 403 response if ZIP or file isn't other readable
- Add virtual hosting that tries prepending Host to path
- Route requests based on Host in Request-URI if present
- Host routing will attempt to remove or add the www. prefix
- Sign-extend UNIX timestamps and don't adjust FileTime zone
Here's some of the improvements made to Cosmopolitan Libc:
- Fix ape.S indentation
- Improve consts.sh magnums
- Write pretty good URL parser
- Improve rusage accounting apis
- Bring mremap() closer to working
- Added ZIP APIs which will change
- Check for overflow in reallocarray()
- Remove overly fancy linkage in strerror()
- Fix GDB attach on crash w/ OpenBSD msyscall()
- Make sigqueue() portable to most UNIX distros
- Make integer serialization macros more elegant
- Bring back 34x tprecode8to16() performance boost
- Make malloc() more resilient to absurdly large sizes
Buffering now has optimal performance, bugs have been fixed, and some
missing apis have been introduced. This implementation is also now more
production worthy since it's less brittle now in terms of system errors.
That's going to help redbean since lua i/o is all based on stdio.
See #97
Cosmopolitan currently doesn't support threads and it doesn't do
anything fancy in longjmp/setjmp so this change was simple to do
- localeconv
- _setjmp (same as setjmp)
- _longjmp (same as longjmp)
- strcoll (same as strcmp)
- flockfile (does nothing)
- funlockfile (does nothing)
- ftrylockfile (does nothing)
See #61
You can now build Cosmopolitan with Clang:
make -j8 MODE=llvm
o/llvm/examples/hello.com
The assembler and linker code is now friendly to LLVM too.
So it's not needed to configure Clang to use binutils under
the hood. If you love LLVM then you can now use pure LLVM.
You can now use cosmopolitan.h with an ANSI C89 compiler like MSVC. The
Cosmopolitan codebase itself won't support being compiled that way. But
you can build objects that link against Cosmopolitan using any compiler
and you can furthermore use tools like IntelliSense that can't even GNU
See also #40
- Support deterministic stacks on OpenBSD
- Support OpenBSD system call origin verification
- Fix overrun by one in chibicc string token allocator
- Get all chibicc tests passing under Address Sanitizer
This change enables Address Sanitizer systemically w/ `make MODE=dbg`.
Our version of Rust's `unsafe` keyword is named `noasan` which is used
for two functions that do aligned memory chunking, like `strcpy.c` and
we need to fix the tiny DEFLATE code, but that's it everything else is
fabulous you can have all the fischer price security blankets you need
Best of all is we're now able to use the ASAN data in Blinkenlights to
colorize the memory dumps. See the screenshot below of a test program:
https://justine.lol/blinkenlights/asan.png
Which is operating on float arrays stored on the stack, with red areas
indicating poisoned memory, and the green areas indicate valid memory.
We always favor calling functions like openat(), fstatat(), etc. because
Linux, XNU, FreeBSD, and OpenBSD all elected to support them, while some
systems like Android love them so much, that they stopped supporting the
old interfaces.
This change ensures that when dirfd is actually a dirfd and not AT_FDCWD
we'll do the right thing on Windows NT. We use an API that's been around
since Vista to accomplish that.
This change also adds exponential backoff to chdir() on Windows since it
seems almost as flaky on Windows 7 as the rmdir() function.
For the first time ever, all tests in this codebase now pass, when
run automatically on macos, freebsd, openbsd, rhel5, rhel7, alpine
and windows via the network using the runit and runitd build tools
- Fix vfork exec path etc.
- Add XNU opendir() support
- Add OpenBSD opendir() support
- Add Linux history to syscalls.sh
- Use copy_file_range on FreeBSD 13+
- Fix system calls with 7+ arguments
- Fix Windows with greater than 16 FDs
- Fix RUNIT.COM and RUNITD.COM flakiness
- Fix OpenBSD munmap() when files are mapped
- Fix long double so it's actually long on Windows
- Fix OpenBSD truncate() and ftruncate() thunk typo
- Let Windows fcntl() be used on socket files descriptors
- Fix Windows fstat() which had an accidental printf statement
- Fix RHEL5 CLOCK_MONOTONIC by not aliasing to CLOCK_MONOTONIC_RAW
This is wonderful. I never could have dreamed it would be possible
to get it working so well on so many platforms with tiny binaries.
Fixes#31Fixes#25Fixes#14
It turned out that the linker was doing the wrong with the amalgamation
library concerning weak stubs. A regression test has been added and new
binaries have been uploaded to https://justine.lol/cosmopolitan/
Ideally this should be fixed by building a tool that turns multiple .a
files into a single .a file with deduplication. As a workaround for now
the cosmopolitan.a build is restructured to not include LIBC_STUBS which
meant technical debt needed to be paid off where non-stub interfaces
were moved to LIBC_INTRIN and LIBC_NEXGEN32E.
Thank @PerfectProductions in #31 for the report!
This program popped up on Hacker News recently. It's the only modern
compiler I've ever seen that doesn't have dependencies and is easily
modified. So I added all of the missing GNU extensions I like to use
which means it might be possible soon to build on non-Linux and have
third party not vendor gcc binaries.
A new rollup tool now exists for flattening out the headers in a way
that works better for our purposes than cpp. A lot of the API clutter
has been removed. APIs that aren't a sure thing in terms of general
recommendation are now marked internal.
There's now a smoke test for the amalgamation archive and gigantic
header file. So we can now guarantee you can use this project on the
easiest difficulty setting without the gigantic repository.
A website is being created, which is currently a work in progress:
https://justine.storage.googleapis.com/cosmopolitan/index.html
This change includes many bug fixes, for the NT polyfills, strings,
memory, boot, and math libraries which were discovered by adding more
tools for recreational programming, such as PC emulation. Lemon has also
been vendored because it works so well at parsing languages.
- Emulator can now test the αcτµαlly pδrταblε εxεcµταblε bootloader
- Whipped up a webserver named redbean. It services 150k requests per
second on a single core. Bundling assets inside zip enables extremely
fast serving for two reasons. The first is that zip central directory
lookups go faster than stat() system calls. The second is that both
zip and gzip content-encoding use DEFLATE, therefore, compressed
responses can be served via the sendfile() system call which does an
in-kernel copy directly from the zip executable structure. Also note
that red bean zip executables can be deployed easily to all platforms,
since these native executables work on Linux, Mac, BSD, and Windows.
- Address sanitizer now works very well
I wanted a tiny scriptable meltdown proof way to run userspace programs
and visualize how program execution impacts memory. It helps to explain
how things like Actually Portable Executable works. It can show you how
the GCC generated code is going about manipulating matrices and more. I
didn't feel fully comfortable with Qemu and Bochs because I'm not smart
enough to understand them. I wanted something like gVisor but with much
stronger levels of assurances. I wanted a single binary that'll run, on
all major operating systems with an embedded GPL barrier ZIP filesystem
that is tiny enough to transpile to JavaScript and run in browsers too.
https://justine.storage.googleapis.com/emulator625.mp4
