The ZIP filesystem has a breaking change. You now need to use /zip/ to
open() / opendir() / etc. assets within the ZIP structure of your APE
binary, instead of the previous convention of using zip: or zip! URIs.
This is needed because Python likes to use absolute paths, and having
ZIP paths encoded like URIs simply broke too many things.
Many more system calls have been updated to be able to operate on ZIP
files and file descriptors. In particular fcntl() and ioctl() since
Python would do things like ask if a ZIP file is a terminal and get
confused when the old implementation mistakenly said yes, because the
fastest way to guarantee native file descriptors is to dup(2). This
change also improves the async signal safety of zipos and ensures it
doesn't maintain any open file descriptors beyond that which the user
has opened.
This change makes a lot of progress towards adding magic numbers that
are specific to platforms other than Linux. The philosophy here is that,
if you use an operating system like FreeBSD, then you should be able to
take advantage of FreeBSD exclusive features, even if we don't polyfill
them on other platforms. For example, you can now open() a file with the
O_VERIFY flag. If your program runs on other platforms, then Cosmo will
automatically set O_VERIFY to zero. This lets you safely use it without
the need for #ifdef or ifstatements which detract from readability.
One of the blindspots of the ASAN memory hardening we use to offer Rust
like assurances has always been that memory passed to the kernel via
system calls (e.g. writev) can't be checked automatically since the
kernel wasn't built with MODE=asan. This change makes more progress
ensuring that each system call will verify the soundness of memory
before it's passed to the kernel. The code for doing these checks is
fast, particularly for buffers, where it can verify 64 bytes a cycle.
- Correct O_LOOP definition on NT
- Introduce program_executable_name
- Add ASAN guards to more system calls
- Improve termios compatibility with BSDs
- Fix bug in Windows auxiliary value encoding
- Add BSD and XNU specific errnos and open flags
- Add check to ensure build doesn't talk to internet
Status lines for Emacs and Vim have been added to Python sources so
they'll be easier to edit using Python's preferred coding style.
Some DNS helper functions have been broken up into multiple files. It's
nice to have one function per file whenever possible, since that way we
don't need -ffunction-sections. Another reason it's good to have small
source files, is because the build will be enforcing resource limits on
compilation and testing soon.
The implementations of the getproto* functions follow from the getserv*
functions: same static name allocation, same type of internal function
that opens a file to search, aliases are not written to the struct, same
type of error handling/returns.
This changes also fixes a getaddrinfo AI_PASSIVE memory error. When
getaddrinfo is passed name = NULL and AI_PASSIVE in hints->ai_flags, it was
setting the s_addr value to INADDR_ANY but *not* returning the addrinfo
pointer via *res = ai. This caused a free(NULL) memory error when the caller
tried to free res, because the caller expects res to be a valid pointer to a
struct addrinfo.
Our non-standard API parseport() has been updated to use strtoimax.
strtoimax has an extra parameter endptr to store where the parsing was
terminated. endptr is used in parseport to check if the provided string
was valid.
For every call of getservbyname/getservbyport the lookup of
/etc/services is done by opening the file and parsing each line
one-by-one. This is slow, but the implementation is simple.
This change also adds fixes for the gethostbyname function.
gethostbyname, gethostbyaddr follow simple implementations: they
internally call getaddrinfo and getnameinfo respectively, and fill out
the minimum details. remaining functions are stubs.
- Use nullness checks when calling weakly linked functions.
- Avoid typedef for reasons described in Linux Kernel style guide.
- Avoid enum in in Windows headers. Earlier in Cosmo's history all one
hundred files in libc/nt/enum/ used to be enums and it resulted in
gigabytes of DWARF data almost as large as everything else in the
codebase combined.
- Bitfields aren't our friends. They have frequent ABI breakages,
inconsistent arithmetic across compilers, and different endianness
between cpus. Compiler authors also haven't invested much roi into
making bit fields go fast so they produce poor assembly.
- Use memccpy() instead of strncpy() or snprintf() for length-bounded
copying of C strings. strncpy() is a misunderstood function and
snprintf() is awesome but memccpy() deserves more love.
Your redbean can now interoperate with clients that require TLS crypto.
This is accomplished using a protocol polyglot that lets us distinguish
between HTTP and HTTPS regardless of the port number. Certificates will
be generated automatically, if none are supplied by the user. Footprint
increases by only a few hundred kb so redbean in MODY=tiny is now 1.0mb
- Add lseek() polyfills for ZIP executable
- Automatically polyfill /tmp/FOO paths on NT
- Fix readdir() / ftw() / nftw() bugs on Windows
- Introduce -B flag for slower SSL that's stronger
- Remove mbedtls features Cosmopolitan doesn't need
- Have base64 decoder support the uri-safe alternative
- Remove Truncated HMAC because it's forbidden by the IETF
- Add all the mbedtls test suites and make them go 3x faster
- Support opendir() / readdir() / closedir() on ZIP executable
- Use Everest for ECDHE-ECDSA because it's so good it's so good
- Add tinier implementation of sha1 since it's not worth the rom
- Add chi-square monte-carlo mean correlation tests for getrandom()
- Source entropy on Windows from the proper interface everyone uses
We're continuing to outperform NGINX and other servers on raw message
throughput. Using SSL means that instead of 1,000,000 qps you can get
around 300,000 qps. However redbean isn't as fast as NGINX yet at SSL
handshakes, since redbean can do 2,627 per second and NGINX does 4.3k
Right now, the SSL UX story works best if you give your redbean a key
signing key since that can be easily generated by openssl using a one
liner then redbean will do all the things that are impossibly hard to
do like signing ecdsa and rsa certificates that'll work in chrome. We
should integrate the let's encrypt acme protocol in the future.
Live Demo: https://redbean.justine.lol/
Root Cert: https://redbean.justine.lol/redbean1.crt
The getnameinfo implementation requires an address -> name lookup on the
hosts file (ie struct HostsTxt) and the previous implementation used
flags to check whether HostsTxt was sorted according to address or name,
and then re-sorted it if necessary. Now getnameinfo lookup does not
require sorting, it does a simple linear lookup, and so the related code
was simplified
See #172 for discussion.
Added necessary constants (DNS_TYPE_PTR, NI_NUMERICHOST etc.).
Implementation of getnameinfo is similar to getaddrinfo, with internal
functions:
* ResolveDnsReverse: performs rDNS query and parses the PTR record
* ResolveHostsReverse: reads /etc/hosts to map hostname to
address
Earlier, the HOSTS.txt would only need to be sorted at loading time,
because the only kind of lookup was name -> address. Now since address
-> name lookups are also possible, so the HostsTxt struct, the sorting
method (and the related tests) was changed to reflect this.
- Better UBSAN error messages
- POSIX Advisory Locks polyfills
- Move redbean manual to /.help.txt
- System call memory safety in ASAN mode
- Character classification now does UNICODE
Buffering now has optimal performance, bugs have been fixed, and some
missing apis have been introduced. This implementation is also now more
production worthy since it's less brittle now in terms of system errors.
That's going to help redbean since lua i/o is all based on stdio.
See #97
You can now build Cosmopolitan with Clang:
make -j8 MODE=llvm
o/llvm/examples/hello.com
The assembler and linker code is now friendly to LLVM too.
So it's not needed to configure Clang to use binutils under
the hood. If you love LLVM then you can now use pure LLVM.
It turned out that the linker was doing the wrong with the amalgamation
library concerning weak stubs. A regression test has been added and new
binaries have been uploaded to https://justine.lol/cosmopolitan/
Ideally this should be fixed by building a tool that turns multiple .a
files into a single .a file with deduplication. As a workaround for now
the cosmopolitan.a build is restructured to not include LIBC_STUBS which
meant technical debt needed to be paid off where non-stub interfaces
were moved to LIBC_INTRIN and LIBC_NEXGEN32E.
Thank @PerfectProductions in #31 for the report!
This change pays off technical debt with the function -> DLL mappings in
libc/nt/master.sh, which was originally defined based on binary analysis
on Windows 10. It's now been updated so the kernel32/kernelbase/advapi32
imports should be exactly as they are written, on the MSDN documentation
and that wouldn't have been easy without Geoff Chappell's work thank him
https://www.geoffchappell.com/studies/windows/win32/index.htm
A new rollup tool now exists for flattening out the headers in a way
that works better for our purposes than cpp. A lot of the API clutter
has been removed. APIs that aren't a sure thing in terms of general
recommendation are now marked internal.
There's now a smoke test for the amalgamation archive and gigantic
header file. So we can now guarantee you can use this project on the
easiest difficulty setting without the gigantic repository.
A website is being created, which is currently a work in progress:
https://justine.storage.googleapis.com/cosmopolitan/index.html
I wanted a tiny scriptable meltdown proof way to run userspace programs
and visualize how program execution impacts memory. It helps to explain
how things like Actually Portable Executable works. It can show you how
the GCC generated code is going about manipulating matrices and more. I
didn't feel fully comfortable with Qemu and Bochs because I'm not smart
enough to understand them. I wanted something like gVisor but with much
stronger levels of assurances. I wanted a single binary that'll run, on
all major operating systems with an embedded GPL barrier ZIP filesystem
that is tiny enough to transpile to JavaScript and run in browsers too.
https://justine.storage.googleapis.com/emulator625.mp4
