- Python static hello world now 1.8mb
- Python static fully loaded now 10mb
- Python HTTPS client now uses MbedTLS
- Python REPL now completes import stmts
- Increase stack size for Python for now
- Begin synthesizing posixpath and ntpath
- Restore Python \N{UNICODE NAME} support
- Restore Python NFKD symbol normalization
- Add optimized code path for Intel SHA-NI
- Get more Python unit tests passing faster
- Get Python help() pagination working on NT
- Python hashlib now supports MbedTLS PBKDF2
- Make memcpy/memmove/memcmp/bcmp/etc. faster
- Add Mersenne Twister and Vigna to LIBC_RAND
- Provide privileged __printf() for error code
- Fix zipos opendir() so that it reports ENOTDIR
- Add basic chmod() implementation for Windows NT
- Add Cosmo's best functions to Python cosmo module
- Pin function trace indent depth to that of caller
- Show memory diagram on invalid access in MODE=dbg
- Differentiate stack overflow on crash in MODE=dbg
- Add stb_truetype and tools for analyzing font files
- Upgrade to UNICODE 13 and reduce its binary footprint
- COMPILE.COM now logs resource usage of build commands
- Start implementing basic poll() support on bare metal
- Set getauxval(AT_EXECFN) to GetModuleFileName() on NT
- Add descriptions to strerror() in non-TINY build modes
- Add COUNTBRANCH() macro to help with micro-optimizations
- Make error / backtrace / asan / memory code more unbreakable
- Add fast perfect C implementation of μ-Law and a-Law audio codecs
- Make strtol() functions consistent with other libc implementations
- Improve Linenoise implementation (see also github.com/jart/bestline)
- COMPILE.COM now suppresses stdout/stderr of successful build commands
This change reinvents all the GNU Readline features I discovered that I
couldn't live without, e.g. UTF-8, CTRL-R search and CTRL-Y yanking. It
now feels just as good in terms of user interface from the subconscious
workflow perspective. It's real nice to finally have an embeddable line
reader that's actually good with a 30 kb footprint and a bsd-2 license.
This change adds a directory to the examples folder, explaining how the
new Python compiler may be used. Some of the bugs with Python binaries
have been addressed but overall it's still a work in progress.
The ZIP filesystem has a breaking change. You now need to use /zip/ to
open() / opendir() / etc. assets within the ZIP structure of your APE
binary, instead of the previous convention of using zip: or zip! URIs.
This is needed because Python likes to use absolute paths, and having
ZIP paths encoded like URIs simply broke too many things.
Many more system calls have been updated to be able to operate on ZIP
files and file descriptors. In particular fcntl() and ioctl() since
Python would do things like ask if a ZIP file is a terminal and get
confused when the old implementation mistakenly said yes, because the
fastest way to guarantee native file descriptors is to dup(2). This
change also improves the async signal safety of zipos and ensures it
doesn't maintain any open file descriptors beyond that which the user
has opened.
This change makes a lot of progress towards adding magic numbers that
are specific to platforms other than Linux. The philosophy here is that,
if you use an operating system like FreeBSD, then you should be able to
take advantage of FreeBSD exclusive features, even if we don't polyfill
them on other platforms. For example, you can now open() a file with the
O_VERIFY flag. If your program runs on other platforms, then Cosmo will
automatically set O_VERIFY to zero. This lets you safely use it without
the need for #ifdef or ifstatements which detract from readability.
One of the blindspots of the ASAN memory hardening we use to offer Rust
like assurances has always been that memory passed to the kernel via
system calls (e.g. writev) can't be checked automatically since the
kernel wasn't built with MODE=asan. This change makes more progress
ensuring that each system call will verify the soundness of memory
before it's passed to the kernel. The code for doing these checks is
fast, particularly for buffers, where it can verify 64 bytes a cycle.
- Correct O_LOOP definition on NT
- Introduce program_executable_name
- Add ASAN guards to more system calls
- Improve termios compatibility with BSDs
- Fix bug in Windows auxiliary value encoding
- Add BSD and XNU specific errnos and open flags
- Add check to ensure build doesn't talk to internet
Actually Portable Python is now outperforming the Python binaries
that come bundled with Linux distros, at things like HTTP serving.
You can now have a fully featured Python install in just one .com
file that runs on six operating systems and is about 10mb in size.
With tuning, the tiniest is ~1mb. We've got most of the libraries
working, including pysqlite, and the repl now feels very pleasant.
The things you can't do quite yet are: threads and shared objects
but that can happen in the future, if the community falls in love
with this project and wants to see it developed further. Changes:
- Add siginterrupt()
- Add sqlite3 to Python
- Add issymlink() helper
- Make GetZipCdir() faster
- Add tgamma() and finite()
- Add legacy function lutimes()
- Add readlink() and realpath()
- Use heap allocations when appropriate
- Reorganize Python into two-stage build
- Save Lua / Python shell history to dotfile
- Integrate Python Lib embedding into linkage
- Make isregularfile() and isdirectory() go faster
- Make Python shell auto-completion work perfectly
- Make crash reports work better if changed directory
- Fix Python+NT open() / access() flag overflow error
- Disable Python tests relating to \N{LONG NAME} syntax
- Have Python REPL copyright() show all notice embeddings
The biggest technical challenge at the moment is working around
when Python tries to be too clever about filenames.
Thanks to all the refactorings we now have the ability to enforce
reasonable limitations on the amount of resources any individual
compile or test can consume. Those limits are currently:
- `-C 8` seconds of 3.1ghz CPU time
- `-M 256mebibytes` of virtual memory
- `-F 100megabyte` limit on file size
Only one file currently needs to exceed these limits:
o/$(MODE)/third_party/python/Objects/unicodeobject.o: \
QUOTA += -C16 # overrides cpu limit to 16 seconds
This change introduces a new sizetol() function to LIBC_FMT for parsing
byte or bit size strings with Si unit suffixes. Functions like atoi()
have been rewritten too.
- Use nullness checks when calling weakly linked functions.
- Avoid typedef for reasons described in Linux Kernel style guide.
- Avoid enum in in Windows headers. Earlier in Cosmo's history all one
hundred files in libc/nt/enum/ used to be enums and it resulted in
gigabytes of DWARF data almost as large as everything else in the
codebase combined.
- Bitfields aren't our friends. They have frequent ABI breakages,
inconsistent arithmetic across compilers, and different endianness
between cpus. Compiler authors also haven't invested much roi into
making bit fields go fast so they produce poor assembly.
- Use memccpy() instead of strncpy() or snprintf() for length-bounded
copying of C strings. strncpy() is a misunderstood function and
snprintf() is awesome but memccpy() deserves more love.
- Better UBSAN error messages
- POSIX Advisory Locks polyfills
- Move redbean manual to /.help.txt
- System call memory safety in ASAN mode
- Character classification now does UNICODE
The most exciting improvement is dynamic pages will soon be able to use
the executable itself as an object store. it required a heroic technique
for overcoming ETXTBSY restrictions which lets us open the executable in
read/write mode, which means (1) wa can restore the APE header, and (2)
we can potentially containerize redbean extension code so that modules
you download for your redbean online will only impact your redbean.
Here's a list of breaking changes to redbean:
- Remove /tool/net/ prefix from magic ZIP paths
- GetHeader() now returns NIL if header is absent
Here's a list of fixes and enhancements to redbean:
- Support 64-bit ZIP archives
- Record User-Agent header in logs
- Add twelve error handlers to accept()
- Display octal st_mode on listing page
- Show ZIP file comments on listing page
- Restore APE MZ header on redbean startup
- Track request count on redbean index page
- Report server uptime on redbean index page
- Don't bind server socket using SO_REUSEPORT
- Fix#151 where Lua LoadAsset() could free twice
- Report rusage accounting when workers exit w/ -vv
- Use ZIP iattr field as text/plain vs. binary hint
- Add ParseUrl() API for parsing things like a.href
- Add ParseParams() API for parsing HTTP POST bodies
- Add IsAcceptablePath() API for checking dots, etc.
- Add IsValidHttpToken() API for validating sane ASCII
- Add IsAcceptableHostPort() for validating HOST[:PORT]
- Send 400 response to HTTP/1.1 requests without a Host
- Send 403 response if ZIP or file isn't other readable
- Add virtual hosting that tries prepending Host to path
- Route requests based on Host in Request-URI if present
- Host routing will attempt to remove or add the www. prefix
- Sign-extend UNIX timestamps and don't adjust FileTime zone
Here's some of the improvements made to Cosmopolitan Libc:
- Fix ape.S indentation
- Improve consts.sh magnums
- Write pretty good URL parser
- Improve rusage accounting apis
- Bring mremap() closer to working
- Added ZIP APIs which will change
- Check for overflow in reallocarray()
- Remove overly fancy linkage in strerror()
- Fix GDB attach on crash w/ OpenBSD msyscall()
- Make sigqueue() portable to most UNIX distros
- Make integer serialization macros more elegant
- Bring back 34x tprecode8to16() performance boost
- Make malloc() more resilient to absurdly large sizes
We can put this back the moment someone requests it. Pain-free garbage
collection for the C language is pretty cool. All it does is overwrite
the return address with a trampoline that calls free(). It's not clear
what it should be named if it's made a public API.
- Polyfill ucontext_t on FreeBSD/OpenBSD/NetBSD
- Add tests confirming signals can edit CPU state
- Work towards supporting ZIP filesystem on bare metal
- Add more tinymath unit tests for POSIX conformance
- Add X87 and SSE status flags to crash report
- Fix some bugs in blinkenlights
- Fix llvm build breakage
You can now build Cosmopolitan with Clang:
make -j8 MODE=llvm
o/llvm/examples/hello.com
The assembler and linker code is now friendly to LLVM too.
So it's not needed to configure Clang to use binutils under
the hood. If you love LLVM then you can now use pure LLVM.
- Get ASAN working on Windows.
- Deleting directories and then recreating them with the same name in a
short period of time appears to be a no-no on Windows.
- There's no reason to call FlushFileBuffers on close() for pipes, and
it's harmful since it might block indefinitely for no good reason.
We always favor calling functions like openat(), fstatat(), etc. because
Linux, XNU, FreeBSD, and OpenBSD all elected to support them, while some
systems like Android love them so much, that they stopped supporting the
old interfaces.
This change ensures that when dirfd is actually a dirfd and not AT_FDCWD
we'll do the right thing on Windows NT. We use an API that's been around
since Vista to accomplish that.
This change also adds exponential backoff to chdir() on Windows since it
seems almost as flaky on Windows 7 as the rmdir() function.
For the first time ever, all tests in this codebase now pass, when
run automatically on macos, freebsd, openbsd, rhel5, rhel7, alpine
and windows via the network using the runit and runitd build tools
- Fix vfork exec path etc.
- Add XNU opendir() support
- Add OpenBSD opendir() support
- Add Linux history to syscalls.sh
- Use copy_file_range on FreeBSD 13+
- Fix system calls with 7+ arguments
- Fix Windows with greater than 16 FDs
- Fix RUNIT.COM and RUNITD.COM flakiness
- Fix OpenBSD munmap() when files are mapped
- Fix long double so it's actually long on Windows
- Fix OpenBSD truncate() and ftruncate() thunk typo
- Let Windows fcntl() be used on socket files descriptors
- Fix Windows fstat() which had an accidental printf statement
- Fix RHEL5 CLOCK_MONOTONIC by not aliasing to CLOCK_MONOTONIC_RAW
This is wonderful. I never could have dreamed it would be possible
to get it working so well on so many platforms with tiny binaries.
Fixes#31Fixes#25Fixes#14
Multiple users have reported that this test fails, for reasons that
are currently unknown. It's possible that, some type of environment
variable configuration, e.g. PATH, CC, etc. is causing the failure.
Once the root cause is identified and addressed we'll re-enable it.
Anyone who is able to help us do that, please comment on the issue.
It turned out that the linker was doing the wrong with the amalgamation
library concerning weak stubs. A regression test has been added and new
binaries have been uploaded to https://justine.lol/cosmopolitan/
Ideally this should be fixed by building a tool that turns multiple .a
files into a single .a file with deduplication. As a workaround for now
the cosmopolitan.a build is restructured to not include LIBC_STUBS which
meant technical debt needed to be paid off where non-stub interfaces
were moved to LIBC_INTRIN and LIBC_NEXGEN32E.
Thank @PerfectProductions in #31 for the report!
A new rollup tool now exists for flattening out the headers in a way
that works better for our purposes than cpp. A lot of the API clutter
has been removed. APIs that aren't a sure thing in terms of general
recommendation are now marked internal.
There's now a smoke test for the amalgamation archive and gigantic
header file. So we can now guarantee you can use this project on the
easiest difficulty setting without the gigantic repository.
A website is being created, which is currently a work in progress:
https://justine.storage.googleapis.com/cosmopolitan/index.html
This change includes many bug fixes, for the NT polyfills, strings,
memory, boot, and math libraries which were discovered by adding more
tools for recreational programming, such as PC emulation. Lemon has also
been vendored because it works so well at parsing languages.
I wanted a tiny scriptable meltdown proof way to run userspace programs
and visualize how program execution impacts memory. It helps to explain
how things like Actually Portable Executable works. It can show you how
the GCC generated code is going about manipulating matrices and more. I
didn't feel fully comfortable with Qemu and Bochs because I'm not smart
enough to understand them. I wanted something like gVisor but with much
stronger levels of assurances. I wanted a single binary that'll run, on
all major operating systems with an embedded GPL barrier ZIP filesystem
that is tiny enough to transpile to JavaScript and run in browsers too.
https://justine.storage.googleapis.com/emulator625.mp4
