Thanks to all the refactorings we now have the ability to enforce
reasonable limitations on the amount of resources any individual
compile or test can consume. Those limits are currently:
- `-C 8` seconds of 3.1ghz CPU time
- `-M 256mebibytes` of virtual memory
- `-F 100megabyte` limit on file size
Only one file currently needs to exceed these limits:
o/$(MODE)/third_party/python/Objects/unicodeobject.o: \
QUOTA += -C16 # overrides cpu limit to 16 seconds
This change introduces a new sizetol() function to LIBC_FMT for parsing
byte or bit size strings with Si unit suffixes. Functions like atoi()
have been rewritten too.
- Update redbean Lua to check there is enough stack to use
- Add explicit Lua gc pass after each message is processed
- Add Lua memory reporting to redbean
This change gets the Python codebase into a state where it conforms to
the conventions of this codebase. It's now possible to include headers
from Python, without worrying about ordering. Python has traditionally
solved that problem by "diamonding" everything in Python.h, but that's
problematic since it means any change to any Python header invalidates
all the build artifacts. Lastly it makes tooling not work. Since it is
hard to explain to Emacs when I press C-c C-h to add an import line it
shouldn't add the header that actually defines the symbol, and instead
do follow the nonstandard Python convention.
Progress has been made on letting Python load source code from the zip
executable structure via the standard C library APIs. System calss now
recognizes zip!FILENAME alternative URIs as equivalent to zip:FILENAME
since Python uses colon as its delimiter.
Some progress has been made on embedding the notice license terms into
the Python object code. This is easier said than done since Python has
an extremely complicated ownership story.
- Some termios APIs have been added
- Implement rewinddir() dirstream API
- GetCpuCount() API added to Cosmopolitan Libc
- More bugs in Cosmopolitan Libc have been fixed
- zipobj.com now has flags for mangling the path
- Fixed bug a priori with sendfile() on certain BSDs
- Polyfill F_DUPFD and F_DUPFD_CLOEXEC across platforms
- FIOCLEX / FIONCLEX now polyfilled for fast O_CLOEXEC changes
- APE now supports a hybrid solution to no-self-modify for builds
- Many BSD-only magnums added, e.g. O_SEARCH, O_SHLOCK, SF_NODISKIO
- Update redbean to include stack trace in Lua errors
- Extend Lua in redbean to include stack trace in all logged errors
- Update default error page in redbean with error details (when allowed)
- Prepend `@` to Lua paths in redbean to recognize them as paths in error messages
- Replace GetClientAddr with GetRemoteAddr to avoid backtrace leak in proxy scenarios
- Fix typo in GetRemoteAddr documentation
Many of the API functions provided by redbean are only appropriate to
call in certain contexts, such as request handling or .init.lua, etc.
For example, Fetch can't be called from the global scope of .init.lua
because SSL hasn't been configured yet. Earlier if this happened then
redbean would crash, which was confusing. What we'll do now is show a
friendly error message. See #97
This change also undocuments redbean ssl compression support since it
seems to be causing a flake in the testing infrastructure.
Redbean Lua Server Pages may now be stored in the zip as byte code. This
can improve performance, since redbean currently doesn't cache byte code
but it might be more useful for anyone wanting to create a closed source
redbean. The .lua extension should be used for byte code files. Lua will
tell them apart based on a magic number at the start of the file.
This change also improves some Lua error reporting conditions. See #97
This change enables SSL compression. It significantly reduces the
network load of the testing infrastructure, for free, since this
revision didn't need to change any runit protocol code. However we
turn it off by default in redbean since no browsers support it.
It turns out that some TLSv1.0 clients (e.g. curl command on RHEL5) will
send an SSLv2-style ClientHello. These types of clients are usually ten+
years old and were designed to interop with servers ten years older than
them. Your redbean is now able to interop with these clients even though
redbean doesn't actually support SSLv2 or SSLv3. Please note that the -B
flag may be passed to disable this along with TLSv1.0, TLSv1.1, 3DES, &c
The following Lua APIs have been added to redbean:
- ProgramSslCompression(bool)
- ProgramSslCiphersuite(name:str)
- ProgramSslPresharedKey(key:str,identity:str)
Lastly the DHE ciphersuites have been enabled. IANA recommends DHE and
with old clients like RHEL5 it's the only perfect forward secrecy they
implement.
AWS Lambda uses a nonstandard kernel behavior that causes
OpenExecutable() to SIGSEGV after redbean restores its APE
header. This change fixes a bug so that Lambda users may
work around the issue by removing the `.ape` file inside the
zip structure of redbean.com.
Fixes#233
- Abort if .init.lua fails
- Refactor redbean to use new append library
- Use first certificate if SNI routing fails
- Use function/data sections when building Lua
- Don't use self-signed auto-generated cert for client
- Add -D staging dirs to redbean lua module default path
redbean will now set Referer-Policy to no-referrer-when-downgrade on
text/html responses by default. There's better explanations on the bits
of security redbean is offering. In short, it's 128+ for modern clients
and 112+ for legacy. If the -B flag is used then it's 192+ for modern
and 150+ for non-EC.
- Add GetRedbeanVersion() to redbean (#97)
- Rename GetVersion to GetHttpVersion() for consistency (#97)
- Update Content-Length format to use a proper size modifier (#97, #218)
This change makes SSL virtual hosting possible. You can now load
multiple certificates for multiple domains and redbean will just
figure out which one to use, even if you only have 1 ip address.
You can also use a jumbo certificate that lists all your domains
in the the subject alternative names.
This change also makes performance improvements to MbedTLS. Here
are some benchmarks vs. cc1920749e
BEFORE AFTER (microsecs)
suite_ssl.com 2512881 191738 13.11x faster
suite_pkparse.com 36291 3295 11.01x faster
suite_x509parse.com 854669 120293 7.10x faster
suite_pkwrite.com 6549 1265 5.18x faster
suite_ecdsa.com 53347 18778 2.84x faster
suite_pk.com 49051 18717 2.62x faster
suite_ecdh.com 19535 9502 2.06x faster
suite_shax.com 15848 7965 1.99x faster
suite_rsa.com 353257 184828 1.91x faster
suite_x509write.com 162646 85733 1.90x faster
suite_ecp.com 20503 11050 1.86x faster
suite_hmac_drbg.no_reseed.com 19528 11417 1.71x faster
suite_hmac_drbg.nopr.com 12460 8010 1.56x faster
suite_mpi.com 687124 442661 1.55x faster
suite_hmac_drbg.pr.com 11890 7752 1.53x faster
There aren't any special tricks to the performance imporvements.
It's mostly due to code cleanup, assembly and intel instructions
like mulx, adox, and adcx.
This change boosts SSL handshake performance from 2,627 to ~10,000 per
second which is the same level of performance as NGINX at establishing
secure connections. That's impressive if we consider that redbean is a
forking frontend application server. This was accomplished by:
1. Enabling either SSL session caching or SSL tickets. We choose to
use tickets since they reduce network round trips too and that's
a more important metric than wrk'ing localhost.
2. Fixing mbedtls_mpi_sub_abs() which is the most frequently called
function. It's called about 12,000 times during an SSL handshake
since it's the basis of most arithmetic operations like addition
and for some strange reason it was designed to make two needless
copies in addition to calling malloc and free. That's now fixed.
3. Improving TLS output buffering during the SSL handshake only, so
that only a single is write and read system call is needed until
blocking on the ping pong.
redbean will now do a better job wiping sensitive memory from a child
process as soon as it's not needed. The nice thing about fork is it's
much faster than reverse proxying so the goal is to use the different
address spaces along with setuid() to minimize the risk that a server
key will be compromised in the event that application code is hacked.
The following Lua APIs have been added:
- IsDaemon() → bool
- ProgramPidPath(str)
The following Lua hooks have been added:
- OnClientConnection(ip:int,port:int,serverip:int,serverport:int) → bool
- OnProcessCreate(pid:int,ip:int,port:int,serverip:int,serverport:int)
- OnProcessDestroy(pid:int)
- OnServerStart()
- OnServerStop()
- OnWorkerStart()
- OnWorkerStop()
redbean now does a better job at applying gzip on the fly from the local
filesystem, using a streaming chunked api with constant memory, which is
useful for doing things like serving a 4gb text file off NFS, and having
it start transmitting in milliseconds. redbean will also compute entropy
on the beginnings of files to determine if compression is profitable.
This change pays off technical debts relating to memory, such as relying
on exit() to free() allocations. That's now mostly fixed so it should be
easier now to spot memory leaks in malloc traces.
This change also fixes bugs and makes improvements to our SSL support.
Uniprocess mode failed handshakes are no longer an issue. Token Alpn is
offered so curl -v looks less weird. Hybrid SSL certificate loading is
now smarter about naming conflicts. Self-signed CA root anchors will no
longer be delivered to the client during the handshake.
- Fix Content-Type inference when file extension has number
- Remove shoddy Class A granular IP classiifcation
- Have setuid() and setgid() take effect w/o daemonization
- Make GetParams() return empty table instead of nil
- Change SetLogLevel(int) to only apply to one message
- Make SetLogLevel(int) good enough to be access_log off
- Introduce ProgramUid(int) which is same as -U INT
- Introduce ProgramGid(int) which is same as -G INT
- Introduce ProgramLogPath(str) which is same as -L PATH
- Introduce ProgramDirectory(str) which is same as -D PATH
- Introduce ProgramLogBodies(bool) which is same as -b
- Introduce ProgramLogMessages(bool) which is same as -m
You can now say the following in your redbean Lua code:
status,headers,payload = Fetch("https://foo.example")
The following Lua APIs have been introduced:
- Fetch(str) → str,{str:str},str
- GetHttpReason(int) → str
- GetHttpReason(int) → str
- ProgramSslFetchVerify(bool)
- ProgramSslClientVerify(bool)
The following flags have been introduced:
- `-j` enables client SSL verification
- `-k` disables Fetch() SSL verification
- `-t INT` may now be passed a negative value for keepalive
Lua exceptions now invoke Cosmopolitan's garbage collector when
unwinding the stack. So it's now safe to use _gc() w/ Lua 𝔱𝔥𝔯𝔬𝔴
See #97
Under MODE=tiny or MODE=tinylinux we now go back to using my homebrew
version of DEFLATE decompression which is 10x smaller but 10x slower
than Chromium Zlib. In tiny mode we also disable compressed responses
howewver redbean will still serve precompressed responses. This change
also removes a few other dependencies like strftime() and getaddrinfo()
which means you can't say `-l localhost` in tiny mode, you have to say
something like `-l 127.0.0.1`.
m=tinylinux
make -j8 MODE=$m o/$m/tool/net/redbean-original.com
ls -hal o/$m/tool/net/redbean-original.com
This change reduces the above size from 191.4 to 150.9 kb.
- Use nullness checks when calling weakly linked functions.
- Avoid typedef for reasons described in Linux Kernel style guide.
- Avoid enum in in Windows headers. Earlier in Cosmo's history all one
hundred files in libc/nt/enum/ used to be enums and it resulted in
gigabytes of DWARF data almost as large as everything else in the
codebase combined.
- Bitfields aren't our friends. They have frequent ABI breakages,
inconsistent arithmetic across compilers, and different endianness
between cpus. Compiler authors also haven't invested much roi into
making bit fields go fast so they produce poor assembly.
- Use memccpy() instead of strncpy() or snprintf() for length-bounded
copying of C strings. strncpy() is a misunderstood function and
snprintf() is awesome but memccpy() deserves more love.
Your redbean can now interoperate with clients that require TLS crypto.
This is accomplished using a protocol polyglot that lets us distinguish
between HTTP and HTTPS regardless of the port number. Certificates will
be generated automatically, if none are supplied by the user. Footprint
increases by only a few hundred kb so redbean in MODY=tiny is now 1.0mb
- Add lseek() polyfills for ZIP executable
- Automatically polyfill /tmp/FOO paths on NT
- Fix readdir() / ftw() / nftw() bugs on Windows
- Introduce -B flag for slower SSL that's stronger
- Remove mbedtls features Cosmopolitan doesn't need
- Have base64 decoder support the uri-safe alternative
- Remove Truncated HMAC because it's forbidden by the IETF
- Add all the mbedtls test suites and make them go 3x faster
- Support opendir() / readdir() / closedir() on ZIP executable
- Use Everest for ECDHE-ECDSA because it's so good it's so good
- Add tinier implementation of sha1 since it's not worth the rom
- Add chi-square monte-carlo mean correlation tests for getrandom()
- Source entropy on Windows from the proper interface everyone uses
We're continuing to outperform NGINX and other servers on raw message
throughput. Using SSL means that instead of 1,000,000 qps you can get
around 300,000 qps. However redbean isn't as fast as NGINX yet at SSL
handshakes, since redbean can do 2,627 per second and NGINX does 4.3k
Right now, the SSL UX story works best if you give your redbean a key
signing key since that can be easily generated by openssl using a one
liner then redbean will do all the things that are impossibly hard to
do like signing ecdsa and rsa certificates that'll work in chrome. We
should integrate the let's encrypt acme protocol in the future.
Live Demo: https://redbean.justine.lol/
Root Cert: https://redbean.justine.lol/redbean1.crt
redbean lua handlers that perform sql queries can do 400k qps.
We now use a separate compile-time options for SQLite, when building the
SQLite shell versus building the production web serving code. It doesn't
seem appropriate for something like redbean to include backups, progress
callbacks, query completion, profiling, EXPLAIN, ALTER, ANALYZE, VACUUM,
etc. since those tasks are better left to the sqlite3.com shell program.
Lua SQLite pointer APIs have been removed since we're not using threads.
The Lua APIs for installing update / commit / rollback hooks are removed
due to a general sense of disagreement and an overall lack of comfort.
Full-Text Search and R*Tree are as large as the rest of SQLite combined.
Turning those off keeps redbean under 1mb when built for MODE=tiny which
is nice for marketing purposes.
If you need something that was removed, file an issue, and we'll add it.
