Commit graph

178 commits

Author SHA1 Message Date
Paul Kulchenko
c00755f921
Improve error messages for consistency (#297) 2021-10-25 14:44:04 -07:00
Paul Kulchenko
013f03e33f
Redbean fix redirect (#294)
* Fix redbean crash during redirect in debug logging mode

* Add reset for redirect loop check in redbean Route

Without this reset a combination of RoutePath() and Route() calls
could return "508 loop detected", since RoutePath could be called
twice for the same redirected path.

The protection against looping is still there, as it can only
loop inside the Route() call (as it always serves something).

* Update redbean redirect message for clarity
2021-10-25 14:04:57 -07:00
Justine Tunney
67b5200a0b Add MODE=optlinux build mode (#141) 2021-10-14 19:36:49 -07:00
Justine Tunney
226aaf3547 Improve memory safety
This commit makes numerous refinements to cosmopolitan memory handling.

The default stack size has been reduced from 2mb to 128kb. A new macro
is now provided so you can easily reconfigure the stack size to be any
value you want. Work around the breaking change by adding to your main:

    STATIC_STACK_SIZE(0x00200000);  // 2mb stack

If you're not sure how much stack you need, then you can use:

    STATIC_YOINK("stack_usage_logging");

After which you can `sort -nr o/$MODE/stack.log`. Based on the unit test
suite, nothing in the Cosmopolitan repository (except for Python) needs
a stack size greater than 30kb. There are also new macros for detecting
the size and address of the stack at runtime, e.g. GetStackAddr(). We
also now support sigaltstack() so if you want to see nice looking crash
reports whenever a stack overflow happens, you can put this in main():

    ShowCrashReports();

Under `make MODE=dbg` and `make MODE=asan` the unit testing framework
will now automatically print backtraces of memory allocations when
things like memory leaks happen. Bugs are now fixed in ASAN global
variable overrun detection. The memtrack and asan runtimes also handle
edge cases now. The new tools helped to identify a few memory leaks,
which are fixed by this change.

This change should fix an issue reported in #288 with ARG_MAX limits.
Fixing this doubled the performance of MKDEPS.COM and AR.COM yet again.
2021-10-13 17:27:13 -07:00
Justine Tunney
39bf41f4eb Make numerous improvements
- Python static hello world now 1.8mb
- Python static fully loaded now 10mb
- Python HTTPS client now uses MbedTLS
- Python REPL now completes import stmts
- Increase stack size for Python for now
- Begin synthesizing posixpath and ntpath
- Restore Python \N{UNICODE NAME} support
- Restore Python NFKD symbol normalization
- Add optimized code path for Intel SHA-NI
- Get more Python unit tests passing faster
- Get Python help() pagination working on NT
- Python hashlib now supports MbedTLS PBKDF2
- Make memcpy/memmove/memcmp/bcmp/etc. faster
- Add Mersenne Twister and Vigna to LIBC_RAND
- Provide privileged __printf() for error code
- Fix zipos opendir() so that it reports ENOTDIR
- Add basic chmod() implementation for Windows NT
- Add Cosmo's best functions to Python cosmo module
- Pin function trace indent depth to that of caller
- Show memory diagram on invalid access in MODE=dbg
- Differentiate stack overflow on crash in MODE=dbg
- Add stb_truetype and tools for analyzing font files
- Upgrade to UNICODE 13 and reduce its binary footprint
- COMPILE.COM now logs resource usage of build commands
- Start implementing basic poll() support on bare metal
- Set getauxval(AT_EXECFN) to GetModuleFileName() on NT
- Add descriptions to strerror() in non-TINY build modes
- Add COUNTBRANCH() macro to help with micro-optimizations
- Make error / backtrace / asan / memory code more unbreakable
- Add fast perfect C implementation of μ-Law and a-Law audio codecs
- Make strtol() functions consistent with other libc implementations
- Improve Linenoise implementation (see also github.com/jart/bestline)
- COMPILE.COM now suppresses stdout/stderr of successful build commands
2021-09-28 01:52:34 -07:00
Paul Kulchenko
42aafc847e Update lua-argon2 for cosmopolitan/redbean 2021-09-27 21:42:40 -07:00
Paul Kulchenko
1b93066883
Add GetCookie to redbean Lua (#269) 2021-09-11 16:46:21 -07:00
Justine Tunney
559b024e1d Decentralize Python native module linkage
We can now link even smaller Python binaries. For example, the hello.com
program in the Python build directory is a compiled linked executable of
hello.py which just prints hello world. Using decentralized sections, we
can make that binary 1.9mb in size (noting that python.com is 6.3 megs!)

This works for nontrivial programs too. For example, say we want an APE
binary that's equivalent to python.com -m http.server. Our makefile now
builds such a binary using the new launcher and it's only 3.2mb in size
since Python sources get turned into ELF objects, which tell our linker
that we need things like native hashing algorithm code.
2021-09-07 11:40:11 -07:00
Paul Kulchenko
1eed7d47bd
Add error reporting to redbean Slurp (#268)
This should allow `content = assert(Slurp(filename))` to work as
expected and report an error if file doesn't exist or can't be read.
2021-09-06 08:21:37 -07:00
Paul Kulchenko
31dd714081
Add SetCookie method to redbean Lua (#265) 2021-09-04 02:12:12 -07:00
Paul Kulchenko
0584684a82
Improve logger API (#262)
This breaking change improves naming consistency.

- Rename LOGF to INFOF
- Rename recently introduced ANYF to LOGF
- Remove V* log calls, as they are not being used
2021-09-03 21:14:26 -07:00
Paul Kulchenko
0b7737b20a
Fix redbean StoreAsset w/ Deflate (#258) 2021-08-25 05:43:26 -07:00
Paul Kulchenko
818926dbae
Fix redbean latency and resource usage logging (#257) 2021-08-22 20:16:10 -07:00
Paul Kulchenko
41b9eb6873
Redbean log improvements (#256)
* Update log level for redbean messages for better log content
* Add categories to redbean log messages
* Reorganize Lua error logging for consistency
* Replace perror logging with FATALF
* Update server failure reporting to better identify response code
* Replace `flogf` with DIEF logging for consistency
2021-08-22 15:01:52 -07:00
Justine Tunney
00611e9b06 Improve ZIP filesystem and change its prefix
The ZIP filesystem has a breaking change. You now need to use /zip/ to
open() / opendir() / etc. assets within the ZIP structure of your APE
binary, instead of the previous convention of using zip: or zip! URIs.
This is needed because Python likes to use absolute paths, and having
ZIP paths encoded like URIs simply broke too many things.

Many more system calls have been updated to be able to operate on ZIP
files and file descriptors. In particular fcntl() and ioctl() since
Python would do things like ask if a ZIP file is a terminal and get
confused when the old implementation mistakenly said yes, because the
fastest way to guarantee native file descriptors is to dup(2). This
change also improves the async signal safety of zipos and ensures it
doesn't maintain any open file descriptors beyond that which the user
has opened.

This change makes a lot of progress towards adding magic numbers that
are specific to platforms other than Linux. The philosophy here is that,
if you use an operating system like FreeBSD, then you should be able to
take advantage of FreeBSD exclusive features, even if we don't polyfill
them on other platforms. For example, you can now open() a file with the
O_VERIFY flag. If your program runs on other platforms, then Cosmo will
automatically set O_VERIFY to zero. This lets you safely use it without
the need for #ifdef or ifstatements which detract from readability.

One of the blindspots of the ASAN memory hardening we use to offer Rust
like assurances has always been that memory passed to the kernel via
system calls (e.g. writev) can't be checked automatically since the
kernel wasn't built with MODE=asan. This change makes more progress
ensuring that each system call will verify the soundness of memory
before it's passed to the kernel. The code for doing these checks is
fast, particularly for buffers, where it can verify 64 bytes a cycle.

- Correct O_LOOP definition on NT
- Introduce program_executable_name
- Add ASAN guards to more system calls
- Improve termios compatibility with BSDs
- Fix bug in Windows auxiliary value encoding
- Add BSD and XNU specific errnos and open flags
- Add check to ensure build doesn't talk to internet
2021-08-22 01:11:53 -07:00
Paul Kulchenko
2730c66f4a
Add GetTime to redbean Lua (#255) 2021-08-22 00:59:47 -07:00
Paul Kulchenko
7341336b1a
Improve redbean docs (#249)
- Add Location to ProgramHeader exclusions
- Add ProgramHeader documentation
- Update ProgramRedirect to check location validity
2021-08-19 09:34:50 -07:00
Paul Kulchenko
a63b147a93
Add file/line reporting to redbean LuaLog (#250) 2021-08-19 09:32:33 -07:00
Paul Kulchenko
4486ad5c9e
Add ServeRedirect function to redbean Lua (#246) 2021-08-17 14:26:33 -07:00
Paul Kulchenko
916f19eea1
Add GetRandomBytes to redbean Lua (#244) 2021-08-16 12:12:29 -07:00
Justine Tunney
579b597ded Refactor out some duplicated code 2021-08-14 06:17:56 -07:00
Justine Tunney
e963d9c8e3 Add cpu / mem / fsz limits to build system
Thanks to all the refactorings we now have the ability to enforce
reasonable limitations on the amount of resources any individual
compile or test can consume. Those limits are currently:

- `-C 8` seconds of 3.1ghz CPU time
- `-M 256mebibytes` of virtual memory
- `-F 100megabyte` limit on file size

Only one file currently needs to exceed these limits:

    o/$(MODE)/third_party/python/Objects/unicodeobject.o: \
        QUOTA += -C16  # overrides cpu limit to 16 seconds

This change introduces a new sizetol() function to LIBC_FMT for parsing
byte or bit size strings with Si unit suffixes. Functions like atoi()
have been rewritten too.
2021-08-13 23:40:53 -07:00
Justine Tunney
71273bc5c9 Revert explicit Lua GC pass (#241) 2021-08-13 03:06:23 -07:00
Paul Kulchenko
a22f0736d8
Improve Redbean Lua memory (#241)
- Update redbean Lua to check there is enough stack to use
- Add explicit Lua gc pass after each message is processed
- Add Lua memory reporting to redbean
2021-08-13 02:11:49 -07:00
Justine Tunney
b420ed8248 Undiamond Python headers
This change gets the Python codebase into a state where it conforms to
the conventions of this codebase. It's now possible to include headers
from Python, without worrying about ordering. Python has traditionally
solved that problem by "diamonding" everything in Python.h, but that's
problematic since it means any change to any Python header invalidates
all the build artifacts. Lastly it makes tooling not work. Since it is
hard to explain to Emacs when I press C-c C-h to add an import line it
shouldn't add the header that actually defines the symbol, and instead
do follow the nonstandard Python convention.

Progress has been made on letting Python load source code from the zip
executable structure via the standard C library APIs. System calss now
recognizes zip!FILENAME alternative URIs as equivalent to zip:FILENAME
since Python uses colon as its delimiter.

Some progress has been made on embedding the notice license terms into
the Python object code. This is easier said than done since Python has
an extremely complicated ownership story.

- Some termios APIs have been added
- Implement rewinddir() dirstream API
- GetCpuCount() API added to Cosmopolitan Libc
- More bugs in Cosmopolitan Libc have been fixed
- zipobj.com now has flags for mangling the path
- Fixed bug a priori with sendfile() on certain BSDs
- Polyfill F_DUPFD and F_DUPFD_CLOEXEC across platforms
- FIOCLEX / FIONCLEX now polyfilled for fast O_CLOEXEC changes
- APE now supports a hybrid solution to no-self-modify for builds
- Many BSD-only magnums added, e.g. O_SEARCH, O_SHLOCK, SF_NODISKIO
2021-08-12 14:07:40 -07:00
Paul Kulchenko
20bb8db9f8
Fix type signature of new Lua hooks (#239) 2021-08-12 00:15:39 -07:00
Paul Kulchenko
a2e443edd7
Update redbean to show Lua stack traces (#237)
- Update redbean to include stack trace in Lua errors
- Extend Lua in redbean to include stack trace in all logged errors
- Update default error page in redbean with error details (when allowed)
- Prepend `@` to Lua paths in redbean to recognize them as paths in error messages
- Replace GetClientAddr with GetRemoteAddr to avoid backtrace leak in proxy scenarios
- Fix typo in GetRemoteAddr documentation
2021-08-11 23:27:39 -07:00
Justine Tunney
79ba5233db Fix regression caused by state assertions change 2021-08-10 11:36:17 -07:00
Justine Tunney
ee7e296339 Add state assertions to redbean Lua APIs
Many of the API functions provided by redbean are only appropriate to
call in certain contexts, such as request handling or .init.lua, etc.
For example, Fetch can't be called from the global scope of .init.lua
because SSL hasn't been configured yet. Earlier if this happened then
redbean would crash, which was confusing. What we'll do now is show a
friendly error message. See #97

This change also undocuments redbean ssl compression support since it
seems to be causing a flake in the testing infrastructure.
2021-08-09 15:35:38 -07:00
Justine Tunney
3bfb7580c5 Add Lua compiler
Redbean Lua Server Pages may now be stored in the zip as byte code. This
can improve performance, since redbean currently doesn't cache byte code
but it might be more useful for anyone wanting to create a closed source
redbean. The .lua extension should be used for byte code files. Lua will
tell them apart based on a magic number at the start of the file.

This change also improves some Lua error reporting conditions. See #97
2021-08-09 13:09:14 -07:00
Justine Tunney
53b9f83e1c Make redbean SSL more tunable
This change enables SSL compression. It significantly reduces the
network load of the testing infrastructure, for free, since this
revision didn't need to change any runit protocol code. However we
turn it off by default in redbean since no browsers support it.

It turns out that some TLSv1.0 clients (e.g. curl command on RHEL5) will
send an SSLv2-style ClientHello. These types of clients are usually ten+
years old and were designed to interop with servers ten years older than
them. Your redbean is now able to interop with these clients even though
redbean doesn't actually support SSLv2 or SSLv3. Please note that the -B
flag may be passed to disable this along with TLSv1.0, TLSv1.1, 3DES, &c

The following Lua APIs have been added to redbean:

  - ProgramSslCompression(bool)
  - ProgramSslCiphersuite(name:str)
  - ProgramSslPresharedKey(key:str,identity:str)

Lastly the DHE ciphersuites have been enabled. IANA recommends DHE and
with old clients like RHEL5 it's the only perfect forward secrecy they
implement.
2021-08-09 07:38:57 -07:00
Robert Hencke
75fc601ff5
Only re-open executable as O_RDWR if .ape present (#234)
AWS Lambda uses a nonstandard kernel behavior that causes
OpenExecutable() to SIGSEGV after redbean restores its APE
header. This change fixes a bug so that Lambda users may
work around the issue by removing the `.ape` file inside the
zip structure of redbean.com.

Fixes #233
2021-08-09 00:52:54 -07:00
w13b3
d8faadf44e Fix redbean Log() in global scope (#230) 2021-08-07 17:30:49 -07:00
Paul Kulchenko
6bbb44c165
Add GetHostOs to redbean (#228) 2021-08-07 14:25:55 -07:00
Justine Tunney
aeeb851422 Fix bugs and make improvements to redbean
- Abort if .init.lua fails
- Refactor redbean to use new append library
- Use first certificate if SNI routing fails
- Use function/data sections when building Lua
- Don't use self-signed auto-generated cert for client
- Add -D staging dirs to redbean lua module default path
2021-08-06 14:18:34 -07:00
Paul Kulchenko
b142ea7176
Add following redirects to redbean Fetch (#226) 2021-08-06 04:48:46 -07:00
Paul Kulchenko
fd76fa0016
Fix OnServerStart/Stop hooks (#225)
Fixes #224
2021-08-06 04:27:50 -07:00
Justine Tunney
533f3d1ef1 Reduce build latency and fix old cpu bugs 2021-08-05 14:43:53 -07:00
Justine Tunney
df8ab0aa0c Restore Referer-Policy and wrap up MbedTLS changes
redbean will now set Referer-Policy to no-referrer-when-downgrade on
text/html responses by default. There's better explanations on the bits
of security redbean is offering. In short, it's 128+ for modern clients
and 112+ for legacy. If the -B flag is used then it's 192+ for modern
and 150+ for non-EC.
2021-08-04 01:05:49 -07:00
Paul Kulchenko
344d2dc356
Redbean doc update (#221)
- Update redbean documentation for consistency and fix typo (#97)
- Update redbean constants for consistency
- Add Fetch documentation to redbean (#97)
2021-08-03 17:57:15 -07:00
Paul Kulchenko
f7b4804251
Add Sleep API to redbean (#220) 2021-08-02 14:53:24 -07:00
Paul Kulchenko
64d87d4e5a
Add GetRedbeanVersion & rename GetVersion (#219)
- Add GetRedbeanVersion() to redbean (#97)
- Rename GetVersion to GetHttpVersion() for consistency (#97)
- Update Content-Length format to use a proper size modifier (#97, #218)
2021-08-02 14:46:43 -07:00
Paul Kulchenko
0b317523a0
Accept table arg in redbean Fetch() API (#218) 2021-07-28 09:33:10 -07:00
Justine Tunney
398f0c16fb Add SNI support to redbean and improve SSL perf
This change makes SSL virtual hosting possible. You can now load
multiple certificates for multiple domains and redbean will just
figure out which one to use, even if you only have 1 ip address.
You can also use a jumbo certificate that lists all your domains
in the the subject alternative names.

This change also makes performance improvements to MbedTLS. Here
are some benchmarks vs. cc1920749e

                                   BEFORE    AFTER   (microsecs)
suite_ssl.com                     2512881   191738 13.11x faster
suite_pkparse.com                   36291     3295 11.01x faster
suite_x509parse.com                854669   120293  7.10x faster
suite_pkwrite.com                    6549     1265  5.18x faster
suite_ecdsa.com                     53347    18778  2.84x faster
suite_pk.com                        49051    18717  2.62x faster
suite_ecdh.com                      19535     9502  2.06x faster
suite_shax.com                      15848     7965  1.99x faster
suite_rsa.com                      353257   184828  1.91x faster
suite_x509write.com                162646    85733  1.90x faster
suite_ecp.com                       20503    11050  1.86x faster
suite_hmac_drbg.no_reseed.com       19528    11417  1.71x faster
suite_hmac_drbg.nopr.com            12460     8010  1.56x faster
suite_mpi.com                      687124   442661  1.55x faster
suite_hmac_drbg.pr.com              11890     7752  1.53x faster

There aren't any special tricks to the performance imporvements.
It's mostly due to code cleanup, assembly and intel instructions
like mulx, adox, and adcx.
2021-07-23 13:56:13 -07:00
Justine Tunney
f3e28aa192 Make SSL handshakes much faster
This change boosts SSL handshake performance from 2,627 to ~10,000 per
second which is the same level of performance as NGINX at establishing
secure connections. That's impressive if we consider that redbean is a
forking frontend application server. This was accomplished by:

  1. Enabling either SSL session caching or SSL tickets. We choose to
     use tickets since they reduce network round trips too and that's
     a more important metric than wrk'ing localhost.

  2. Fixing mbedtls_mpi_sub_abs() which is the most frequently called
     function. It's called about 12,000 times during an SSL handshake
     since it's the basis of most arithmetic operations like addition
     and for some strange reason it was designed to make two needless
     copies in addition to calling malloc and free. That's now fixed.

  3. Improving TLS output buffering during the SSL handshake only, so
     that only a single is write and read system call is needed until
     blocking on the ping pong.

redbean will now do a better job wiping sensitive memory from a child
process as soon as it's not needed. The nice thing about fork is it's
much faster than reverse proxying so the goal is to use the different
address spaces along with setuid() to minimize the risk that a server
key will be compromised in the event that application code is hacked.
2021-07-11 23:17:47 -07:00
Justine Tunney
8c4cce043c Make improvements to redbean
The following Lua APIs have been added:

  - IsDaemon() → bool
  - ProgramPidPath(str)

The following Lua hooks have been added:

  - OnClientConnection(ip:int,port:int,serverip:int,serverport:int) → bool
  - OnProcessCreate(pid:int,ip:int,port:int,serverip:int,serverport:int)
  - OnProcessDestroy(pid:int)
  - OnServerStart()
  - OnServerStop()
  - OnWorkerStart()
  - OnWorkerStop()

redbean now does a better job at applying gzip on the fly from the local
filesystem, using a streaming chunked api with constant memory, which is
useful for doing things like serving a 4gb text file off NFS, and having
it start transmitting in milliseconds. redbean will also compute entropy
on the beginnings of files to determine if compression is profitable.

This change pays off technical debts relating to memory, such as relying
on exit() to free() allocations. That's now mostly fixed so it should be
easier now to spot memory leaks in malloc traces.

This change also fixes bugs and makes improvements to our SSL support.
Uniprocess mode failed handshakes are no longer an issue. Token Alpn is
offered so curl -v looks less weird. Hybrid SSL certificate loading is
now smarter about naming conflicts. Self-signed CA root anchors will no
longer be delivered to the client during the handshake.
2021-07-10 15:19:37 -07:00
Paul Kulchenko
98c674d915
Update Fetch() to POST if body is provided (#211) 2021-07-10 12:39:02 -07:00
Justine Tunney
c002e4ba76 Support hybrid ECDSA / RSA certificate loading 2021-07-09 19:21:00 -07:00
Justine Tunney
fe881982b5 Make slight SSL performance improvements 2021-07-08 21:54:21 -07:00
Justine Tunney
4178896aa0 Fix bug with redbean help flag 2021-07-08 18:28:11 -07:00
Justine Tunney
45d72920ba Fix chained certificate loading 2021-07-08 17:29:32 -07:00
Justine Tunney
feb0f9fb3a Make improvements to redbean
- Fix Content-Type inference when file extension has number
- Remove shoddy Class A granular IP classiifcation
- Have setuid() and setgid() take effect w/o daemonization
- Make GetParams() return empty table instead of nil
- Change SetLogLevel(int) to only apply to one message
- Make SetLogLevel(int) good enough to be access_log off
- Introduce ProgramUid(int) which is same as -U INT
- Introduce ProgramGid(int) which is same as -G INT
- Introduce ProgramLogPath(str) which is same as -L PATH
- Introduce ProgramDirectory(str) which is same as -D PATH
- Introduce ProgramLogBodies(bool) which is same as -b
- Introduce ProgramLogMessages(bool) which is same as -m
2021-07-08 15:56:23 -07:00
Justine Tunney
cc9366b200 Add file Slurp() API to redbean 2021-07-08 13:21:16 -07:00
Justine Tunney
c89bc56f6a Add HTTP/HTTPS Fetch() API to redbean
You can now say the following in your redbean Lua code:

    status,headers,payload = Fetch("https://foo.example")

The following Lua APIs have been introduced:

  - Fetch(str) → str,{str:str},str
  - GetHttpReason(int) → str
  - GetHttpReason(int) → str
  - ProgramSslFetchVerify(bool)
  - ProgramSslClientVerify(bool)

The following flags have been introduced:

  - `-j` enables client SSL verification
  - `-k` disables Fetch() SSL verification
  - `-t INT` may now be passed a negative value for keepalive

Lua exceptions now invoke Cosmopolitan's garbage collector when
unwinding the stack. So it's now safe to use _gc() w/ Lua 𝔱𝔥𝔯𝔬𝔴

See #97
2021-07-07 21:44:27 -07:00
Justine Tunney
36b2710e1a Update curl example to support ssl / https
Now that we know our SSL client works, and that it's able to verify
certificates, the next step will be adding it as an API to redbean.

See #97
2021-07-06 13:39:18 -07:00
Justine Tunney
0ecd71f697 Make chacha20 go faster 2021-07-05 14:03:50 -07:00
Justine Tunney
58fb2fb3d3 Add chunked transfer decoding to redbean 2021-07-05 01:05:10 -07:00
Justine Tunney
8d5f60a9cd Add more hashing apis to redbean 2021-07-05 01:05:10 -07:00
Justine Tunney
a7bd4ed9ea Make redbean tinier
Under MODE=tiny or MODE=tinylinux we now go back to using my homebrew
version of DEFLATE decompression which is 10x smaller but 10x slower
than Chromium Zlib. In tiny mode we also disable compressed responses
howewver redbean will still serve precompressed responses. This change
also removes a few other dependencies like strftime() and getaddrinfo()
which means you can't say `-l localhost` in tiny mode, you have to say
something like `-l 127.0.0.1`.

    m=tinylinux
    make -j8 MODE=$m o/$m/tool/net/redbean-original.com
    ls -hal o/$m/tool/net/redbean-original.com

This change reduces the above size from 191.4 to 150.9 kb.
2021-07-04 12:26:54 -07:00
Justine Tunney
300876ee50 Fix typo 2021-07-03 10:16:38 -07:00
Justine Tunney
e6b22f1f65 Remove remaining build references to zip command 2021-07-03 08:37:15 -07:00
Justine Tunney
d6873f637f Release redbean 1.4 2021-07-03 05:58:47 -07:00
Justine Tunney
74200a0ea0 Make redbean ssl handshake go a little faster 2021-07-03 05:51:04 -07:00
Justine Tunney
a68cc690ff Merge HTTP request / response parsing code
This change also fixes a bug so that DNS lookups work correctly when the
first answer is a CNAME record.
2021-06-27 17:04:32 -07:00
Justine Tunney
5144c22189 Add test for ioctl(SIOCGIFCONF) and polyfill on BSDs
- Use nullness checks when calling weakly linked functions.

- Avoid typedef for reasons described in Linux Kernel style guide.

- Avoid enum in in Windows headers. Earlier in Cosmo's history all one
  hundred files in libc/nt/enum/ used to be enums and it resulted in
  gigabytes of DWARF data almost as large as everything else in the
  codebase combined.

- Bitfields aren't our friends. They have frequent ABI breakages,
  inconsistent arithmetic across compilers, and different endianness
  between cpus. Compiler authors also haven't invested much roi into
  making bit fields go fast so they produce poor assembly.

- Use memccpy() instead of strncpy() or snprintf() for length-bounded
  copying of C strings. strncpy() is a misunderstood function and
  snprintf() is awesome but memccpy() deserves more love.
2021-06-25 18:44:04 -07:00
Justine Tunney
86ab24ce56 Correct minor issues with recent SSL change 2021-06-24 15:22:24 -07:00
Justine Tunney
cc1920749e Add SSL to redbean
Your redbean can now interoperate with clients that require TLS crypto.
This is accomplished using a protocol polyglot that lets us distinguish
between HTTP and HTTPS regardless of the port number. Certificates will
be generated automatically, if none are supplied by the user. Footprint
increases by only a few hundred kb so redbean in MODY=tiny is now 1.0mb

- Add lseek() polyfills for ZIP executable
- Automatically polyfill /tmp/FOO paths on NT
- Fix readdir() / ftw() / nftw() bugs on Windows
- Introduce -B flag for slower SSL that's stronger
- Remove mbedtls features Cosmopolitan doesn't need
- Have base64 decoder support the uri-safe alternative
- Remove Truncated HMAC because it's forbidden by the IETF
- Add all the mbedtls test suites and make them go 3x faster
- Support opendir() / readdir() / closedir() on ZIP executable
- Use Everest for ECDHE-ECDSA because it's so good it's so good
- Add tinier implementation of sha1 since it's not worth the rom
- Add chi-square monte-carlo mean correlation tests for getrandom()
- Source entropy on Windows from the proper interface everyone uses

We're continuing to outperform NGINX and other servers on raw message
throughput. Using SSL means that instead of 1,000,000 qps you can get
around 300,000 qps. However redbean isn't as fast as NGINX yet at SSL
handshakes, since redbean can do 2,627 per second and NGINX does 4.3k

Right now, the SSL UX story works best if you give your redbean a key
signing key since that can be easily generated by openssl using a one
liner then redbean will do all the things that are impossibly hard to
do like signing ecdsa and rsa certificates that'll work in chrome. We
should integrate the let's encrypt acme protocol in the future.

Live Demo: https://redbean.justine.lol/
Root Cert: https://redbean.justine.lol/redbean1.crt
2021-06-24 13:20:50 -07:00
Justine Tunney
a3ccc5af1f Fix redbean hidden path bug 2021-06-10 08:55:50 -07:00
Justine Tunney
f271ffcf90 Release redbean 1.2 2021-06-10 08:14:45 -07:00
Justine Tunney
88806b79b1 Tune SQLite build for redbean (#97)
redbean lua handlers that perform sql queries can do 400k qps.

We now use a separate compile-time options for SQLite, when building the
SQLite shell versus building the production web serving code. It doesn't
seem appropriate for something like redbean to include backups, progress
callbacks, query completion, profiling, EXPLAIN, ALTER, ANALYZE, VACUUM,
etc. since those tasks are better left to the sqlite3.com shell program.

Lua SQLite pointer APIs have been removed since we're not using threads.
The Lua APIs for installing update / commit / rollback hooks are removed
due to a general sense of disagreement and an overall lack of comfort.

Full-Text Search and R*Tree are as large as the rest of SQLite combined.
Turning those off keeps redbean under 1mb when built for MODE=tiny which
is nice for marketing purposes.

If you need something that was removed, file an issue, and we'll add it.
2021-06-10 08:00:08 -07:00
Justine Tunney
eb08b9fbeb Fix redbean inappropriately linking ASAN
Fixes #188
2021-06-10 07:59:50 -07:00
Paul Kulchenko
b3a08d9286 Add sqlite3 support to Lua scripts in redbean. 2021-06-05 18:38:38 -07:00
Justine Tunney
7cbc2bc083 Cosmopolitan 1.0 2021-05-16 20:35:10 -07:00
Justine Tunney
6a8c21269f Make trivial code size enhancements 2021-05-16 18:22:39 -07:00
Justine Tunney
4864565198 Make minor improvements 2021-05-15 21:53:26 -07:00
Justine Tunney
690be544da Make redbean StoreAsset() work better
- Better UBSAN error messages
- POSIX Advisory Locks polyfills
- Move redbean manual to /.help.txt
- System call memory safety in ASAN mode
- Character classification now does UNICODE
2021-05-14 05:44:37 -07:00
Justine Tunney
1b5a5719c3 Improve some unicode functions 2021-05-05 07:25:39 -07:00
Justine Tunney
b9187061a7 Let redbean use its own ZIP EXE as NoSQL database 2021-05-03 13:07:58 -07:00
Justine Tunney
e56a9d0e23 Mold the redbean binary to minimize page faults
This change brings page faults for precompressed static asset serving
down from 27 to 20 (or fewer) after fork. This is more of an art than
science. Hopefully Blinkenlights can visualize page faults soon.
2021-05-03 12:14:13 -07:00
Justine Tunney
2d34819779 Escape LaunchBrowser path
This also fix a bug where CTRL-C'ing redbean on Linux would kill the
browser process. It furthermore fixes a regression with the APE self
repair process that happened in a recent change.

See #158
2021-05-03 11:52:24 -07:00
Rowan Easter-Robinson
6215c91e90
Added Parameter to LaunchBrowser to set path (#158) 2021-05-03 10:55:21 -07:00
Justine Tunney
01e6b3ad8d Reduce number of disk seeks in redbean 2021-05-03 01:21:50 -07:00
Justine Tunney
daa32d27d4 Add live reindexing to redbean when zip changes 2021-05-02 11:50:43 -07:00
Justine Tunney
84001a246c Fix redbean date header in daemonize mode 2021-05-02 11:11:26 -07:00
Justine Tunney
cef08d47b6 Fix typo in redbean.c 2021-05-01 05:42:02 -07:00
Justine Tunney
41801c9802 Change default redbean log level to info
A new -s flag has been added to increase silence. This change is
intended to make it a little less weird opening the redbean binary
directly from the browser and seeing an empty terminal window.
2021-05-01 05:28:15 -07:00
Justine Tunney
1966369e8e Fix bugs and make improvements
- Fix regression with `%lu`
- Added some more headers witnessed in the wild
- Added `-M INT` option to redbean to tune max payload size
- Work around InfoZIP 256 character limit on comment line size
2021-05-01 05:11:35 -07:00
Justine Tunney
c029e83dd8 Set up http://redbean.justine.lol/ demo server 2021-04-24 19:49:49 -07:00
Justine Tunney
472b95fea3 Support OnHttpRequest Lua callback
If your redbean `/.init.lua` file defines a global callable named
`OnHttpRequest` then redbean will delegate all serving control to
your function. You may then restore the default serving paths, by
calling the new `Route()`, `RouteHost()`, and `RoutePath()` APIs.

Closes #150
2021-04-24 17:27:12 -07:00
Justine Tunney
dc6d11a031 Improve performance of printf functions 2021-04-24 13:58:50 -07:00
Justine Tunney
b107d2709f Add /statusz page to redbean plus other enhancements
redbean improvements:

- Explicitly disable corking
- Simulate Python regex API for Lua
- Send warmup requests in main process on startup
- Add Class-A granular IPv4 network classification
- Add /statusz page so you can monitor your redbean's health
- Fix regressions on OpenBSD/NetBSD caused by recent changes
- Plug Authorization header into Lua GetUser and GetPass APIs
- Recognize X-Forwarded-{For,Host} from local reverse proxies
- Add many additional functions to redbean Lua server page API
- Report resource usage of child processes on `/` listing page
- Introduce `-a` flag for logging child process resource usage
- Introduce `-t MILLIS` flag and `ProgramTimeout(ms)` init API
- Introduce `-H "Header: value"` flag and `ProgramHeader(k,v)` API

Cosmopolitan Libc improvements:

- Make strerror() simpler
- Make inet_pton() not depend on sscanf()
- Fix OpenExecutable() which broke .data section earlier
- Fix stdio in cases where it overflows kernel tty buffer
- Fix bugs in crash reporting w/o .com.dbg binary present
- Add polyfills for SO_LINGER, SO_RCVTIMEO, and SO_SNDTIMEO
- Polyfill TCP_CORK on BSD and XNU using TCP_NOPUSH magnums

New netcat clone in examples/nc.c:

While testing some of the failure conditions for redbean, I noticed that
BusyBox's `nc` command is pretty busted, if you use it as an interactive
tool, rather than having it be part of a pipeline. Unfortunately this'll
only work on UNIX since Windows doesn't let us poll on stdio and sockets
at the same time because I don't think they want tools like this running
on their platform. So if you want forbidden fruit, it's here so enjoy it
2021-04-23 18:53:57 -07:00
Justine Tunney
4effa23528 Make more major improvements to redbean
- POSIX regular expressions for Lua
- Improved protocol parsing and encoding
- Additional APIs for ZIP storage retrieval
- Fix st_mode issue on NT for regular files
- Generalized APIs for URL and Host handling
- Worked out the kinks in resource resolution
- Allow for custom error pages like /404.html
2021-04-20 19:14:21 -07:00
Justine Tunney
bf03b2e64c Make major improvements to redbean and libraries
The most exciting improvement is dynamic pages will soon be able to use
the executable itself as an object store. it required a heroic technique
for overcoming ETXTBSY restrictions which lets us open the executable in
read/write mode, which means (1) wa can restore the APE header, and (2)
we can potentially containerize redbean extension code so that modules
you download for your redbean online will only impact your redbean.

Here's a list of breaking changes to redbean:

- Remove /tool/net/ prefix from magic ZIP paths
- GetHeader() now returns NIL if header is absent

Here's a list of fixes and enhancements to redbean:

- Support 64-bit ZIP archives
- Record User-Agent header in logs
- Add twelve error handlers to accept()
- Display octal st_mode on listing page
- Show ZIP file comments on listing page
- Restore APE MZ header on redbean startup
- Track request count on redbean index page
- Report server uptime on redbean index page
- Don't bind server socket using SO_REUSEPORT
- Fix #151 where Lua LoadAsset() could free twice
- Report rusage accounting when workers exit w/ -vv
- Use ZIP iattr field as text/plain vs. binary hint
- Add ParseUrl() API for parsing things like a.href
- Add ParseParams() API for parsing HTTP POST bodies
- Add IsAcceptablePath() API for checking dots, etc.
- Add IsValidHttpToken() API for validating sane ASCII
- Add IsAcceptableHostPort() for validating HOST[:PORT]
- Send 400 response to HTTP/1.1 requests without a Host
- Send 403 response if ZIP or file isn't other readable
- Add virtual hosting that tries prepending Host to path
- Route requests based on Host in Request-URI if present
- Host routing will attempt to remove or add the www. prefix
- Sign-extend UNIX timestamps and don't adjust FileTime zone

Here's some of the improvements made to Cosmopolitan Libc:

- Fix ape.S indentation
- Improve consts.sh magnums
- Write pretty good URL parser
- Improve rusage accounting apis
- Bring mremap() closer to working
- Added ZIP APIs which will change
- Check for overflow in reallocarray()
- Remove overly fancy linkage in strerror()
- Fix GDB attach on crash w/ OpenBSD msyscall()
- Make sigqueue() portable to most UNIX distros
- Make integer serialization macros more elegant
- Bring back 34x tprecode8to16() performance boost
- Make malloc() more resilient to absurdly large sizes
2021-04-18 12:34:15 -07:00
Rowan Easter-Robinson
69c508729e
Added LaunchBrowser API (#153)
You can now add LaunchBrowser() to .init.lua which will ask
redbean to open your website in the local desktop browser
when the server is launched.
2021-04-18 07:24:36 -07:00
Justine Tunney
59575f7e80 Perform some minor code cleanup 2021-04-06 12:46:52 -07:00
Justine Tunney
6c16f208b5 Remove unintended log statements 2021-04-01 20:01:26 -07:00
Justine Tunney
83abd68029 Parse Content-Range with missing right hand side
Fixes #144
2021-04-01 18:52:54 -07:00
Justine Tunney
7abca1531f Cosmopolitan 0.3 2021-03-29 03:20:03 -07:00
Justine Tunney
7dc8973567 Enable serving zip directory entries 2021-03-29 03:10:11 -07:00
Justine Tunney
1753b669cf Have redbean show zip listing as default / handler
If an "index.lua" or "index.html" doesn't exist in zip file or the
filesystem, and no redirects have been defined for it either, then
redbean will render a listing of the zip central directory content
only if the request uri points to the root path.
2021-03-29 01:22:49 -07:00