Modules/Setup and Modules/Setup.local contain the build recipes for
various extensions, wrote a custom script to translate them for
python.mk. Modules/config.c needs to be changed if any extensions are
removed or added.
Most of the source modifications are for missing headers or compile time
build vars like ABIFLAGS.
Created separate mk files for the C extensions and the Python stdlib.
Can use find for adding the python files to the APE ZIP store, but right
now necessary files are just hardcoded.
python.com loads but some build configs are still missing (showing 1 Jan
1970 as time of compilation).
These are the commits from
https://github.com/ahgamut/cpython/tree/cosmo_py36 squashed for
simplicity.
Also included is the pyconfig.h used for compilation. The pyconfig.h has
to be changed manually in case Cosmopolitan gets new features.
AWS Lambda uses a nonstandard kernel behavior that causes
OpenExecutable() to SIGSEGV after redbean restores its APE
header. This change fixes a bug so that Lambda users may
work around the issue by removing the `.ape` file inside the
zip structure of redbean.com.
Fixes#233
- Abort if .init.lua fails
- Refactor redbean to use new append library
- Use first certificate if SNI routing fails
- Use function/data sections when building Lua
- Don't use self-signed auto-generated cert for client
- Add -D staging dirs to redbean lua module default path
redbean will now set Referer-Policy to no-referrer-when-downgrade on
text/html responses by default. There's better explanations on the bits
of security redbean is offering. In short, it's 128+ for modern clients
and 112+ for legacy. If the -B flag is used then it's 192+ for modern
and 150+ for non-EC.
- Add GetRedbeanVersion() to redbean (#97)
- Rename GetVersion to GetHttpVersion() for consistency (#97)
- Update Content-Length format to use a proper size modifier (#97, #218)
One of the disadvantages of x25519 and ℘256 is it only provides 126 bits
of security, so that seems like a weak link in the chain, if we're using
ECDHE-ECDSA-AES256-GCM-SHA384. The U.S. government wants classified data
to be encrypted using a curve at least as strong as ℘384, which provides
192 bits of security, but if you read the consensus of stack exchange it
would give you the impression that ℘384 is three times slower.
This change (as well as the previous one) makes ℘384 three times as fast
by tuning its modulus and multiplication subroutines with new tests that
should convincingly show: the optimized code behaves the same way as the
old code. Some of the diff noise from the previous change is now removed
too, so that our vendored fork can be more easily compared with upstream
sources. So you can now have stronger cryptography without compromises.
℘384 modulus Justine l: 28𝑐 9𝑛𝑠
℘384 modulus MbedTLS NIST l: 127𝑐 41𝑛𝑠
℘384 modulus MbedTLS MPI l: 1,850𝑐 597𝑛𝑠
The benchmarks above show the improvements made by secp384r1() which is
an important function since it needs to be called 13,000 times whenever
someone establishes a connection to your web server. The same's true of
Mul6x6Adx() which is able to multiply 384-bit numbers in 73 cycles, but
only if your CPU was purchased after 2014 when Broadwell was introduced
This change makes SSL virtual hosting possible. You can now load
multiple certificates for multiple domains and redbean will just
figure out which one to use, even if you only have 1 ip address.
You can also use a jumbo certificate that lists all your domains
in the the subject alternative names.
This change also makes performance improvements to MbedTLS. Here
are some benchmarks vs. cc1920749e
BEFORE AFTER (microsecs)
suite_ssl.com 2512881 191738 13.11x faster
suite_pkparse.com 36291 3295 11.01x faster
suite_x509parse.com 854669 120293 7.10x faster
suite_pkwrite.com 6549 1265 5.18x faster
suite_ecdsa.com 53347 18778 2.84x faster
suite_pk.com 49051 18717 2.62x faster
suite_ecdh.com 19535 9502 2.06x faster
suite_shax.com 15848 7965 1.99x faster
suite_rsa.com 353257 184828 1.91x faster
suite_x509write.com 162646 85733 1.90x faster
suite_ecp.com 20503 11050 1.86x faster
suite_hmac_drbg.no_reseed.com 19528 11417 1.71x faster
suite_hmac_drbg.nopr.com 12460 8010 1.56x faster
suite_mpi.com 687124 442661 1.55x faster
suite_hmac_drbg.pr.com 11890 7752 1.53x faster
There aren't any special tricks to the performance imporvements.
It's mostly due to code cleanup, assembly and intel instructions
like mulx, adox, and adcx.
This change boosts SSL handshake performance from 2,627 to ~10,000 per
second which is the same level of performance as NGINX at establishing
secure connections. That's impressive if we consider that redbean is a
forking frontend application server. This was accomplished by:
1. Enabling either SSL session caching or SSL tickets. We choose to
use tickets since they reduce network round trips too and that's
a more important metric than wrk'ing localhost.
2. Fixing mbedtls_mpi_sub_abs() which is the most frequently called
function. It's called about 12,000 times during an SSL handshake
since it's the basis of most arithmetic operations like addition
and for some strange reason it was designed to make two needless
copies in addition to calling malloc and free. That's now fixed.
3. Improving TLS output buffering during the SSL handshake only, so
that only a single is write and read system call is needed until
blocking on the ping pong.
redbean will now do a better job wiping sensitive memory from a child
process as soon as it's not needed. The nice thing about fork is it's
much faster than reverse proxying so the goal is to use the different
address spaces along with setuid() to minimize the risk that a server
key will be compromised in the event that application code is hacked.
The following Lua APIs have been added:
- IsDaemon() → bool
- ProgramPidPath(str)
The following Lua hooks have been added:
- OnClientConnection(ip:int,port:int,serverip:int,serverport:int) → bool
- OnProcessCreate(pid:int,ip:int,port:int,serverip:int,serverport:int)
- OnProcessDestroy(pid:int)
- OnServerStart()
- OnServerStop()
- OnWorkerStart()
- OnWorkerStop()
redbean now does a better job at applying gzip on the fly from the local
filesystem, using a streaming chunked api with constant memory, which is
useful for doing things like serving a 4gb text file off NFS, and having
it start transmitting in milliseconds. redbean will also compute entropy
on the beginnings of files to determine if compression is profitable.
This change pays off technical debts relating to memory, such as relying
on exit() to free() allocations. That's now mostly fixed so it should be
easier now to spot memory leaks in malloc traces.
This change also fixes bugs and makes improvements to our SSL support.
Uniprocess mode failed handshakes are no longer an issue. Token Alpn is
offered so curl -v looks less weird. Hybrid SSL certificate loading is
now smarter about naming conflicts. Self-signed CA root anchors will no
longer be delivered to the client during the handshake.
The implementations of the getproto* functions follow from the getserv*
functions: same static name allocation, same type of internal function
that opens a file to search, aliases are not written to the struct, same
type of error handling/returns.
This changes also fixes a getaddrinfo AI_PASSIVE memory error. When
getaddrinfo is passed name = NULL and AI_PASSIVE in hints->ai_flags, it was
setting the s_addr value to INADDR_ANY but *not* returning the addrinfo
pointer via *res = ai. This caused a free(NULL) memory error when the caller
tried to free res, because the caller expects res to be a valid pointer to a
struct addrinfo.
Our non-standard API parseport() has been updated to use strtoimax.
strtoimax has an extra parameter endptr to store where the parsing was
terminated. endptr is used in parseport to check if the provided string
was valid.
- Fix Content-Type inference when file extension has number
- Remove shoddy Class A granular IP classiifcation
- Have setuid() and setgid() take effect w/o daemonization
- Make GetParams() return empty table instead of nil
- Change SetLogLevel(int) to only apply to one message
- Make SetLogLevel(int) good enough to be access_log off
- Introduce ProgramUid(int) which is same as -U INT
- Introduce ProgramGid(int) which is same as -G INT
- Introduce ProgramLogPath(str) which is same as -L PATH
- Introduce ProgramDirectory(str) which is same as -D PATH
- Introduce ProgramLogBodies(bool) which is same as -b
- Introduce ProgramLogMessages(bool) which is same as -m
You can now say the following in your redbean Lua code:
status,headers,payload = Fetch("https://foo.example")
The following Lua APIs have been introduced:
- Fetch(str) → str,{str:str},str
- GetHttpReason(int) → str
- GetHttpReason(int) → str
- ProgramSslFetchVerify(bool)
- ProgramSslClientVerify(bool)
The following flags have been introduced:
- `-j` enables client SSL verification
- `-k` disables Fetch() SSL verification
- `-t INT` may now be passed a negative value for keepalive
Lua exceptions now invoke Cosmopolitan's garbage collector when
unwinding the stack. So it's now safe to use _gc() w/ Lua 𝔱𝔥𝔯𝔬𝔴
See #97
Things are a little better. The LLD that comes with Linux seems to work.
Old versions like LLVM 8 haven't been supported since Cosmopolitan v0.2.
Running Clang on Windows with --target=x86_64-pc-linux-gnu doesn't seem
to work. It has something to do with the recently added .zip section in
the linker script. But even if that's removed, LLD on Windows thinks it
is building an EFI application for some reason. Linker scripts are such
a brittle house of cards, even for just ld.bfd alone..
We should just find a way to run our one true musl-cross-make linux gcc
toolchain under Blinkenlights on non-Linux because GCC and Clang are so
nondeterministic, inconsistent, and unreproducible when built for other
operating systems. We need an actually portable compiler/linker that'll
always behave the same way no matter what.
See #180
For every call of getservbyname/getservbyport the lookup of
/etc/services is done by opening the file and parsing each line
one-by-one. This is slow, but the implementation is simple.
This change also adds fixes for the gethostbyname function.
