Commit graph

667 commits

Author SHA1 Message Date
Justine Tunney
5005f2e446
Rewrite brk() and sbrk()
This change reduces the .bss memory requirement for all executables by
O(64kb). The brk system calls are now fully tested and figured out and
might be useful for tiny programs that only target System Five.
2022-10-01 23:11:56 -07:00
Justine Tunney
bfb85fe6d0
Remove a bunch of stuff that shouldn't be in docs 2022-09-20 02:57:46 -07:00
Justine Tunney
2cc1d5ac4c
Improve redbean method=get parameter handling 2022-09-19 20:04:39 -07:00
Justine Tunney
0060940d84
Fix MODE=opt build 2022-09-19 15:43:52 -07:00
Justine Tunney
c7a8cd21e9
Improve system call wrappers
This change improves copy_file_range(), sendfile(), splice(), openpty(),
closefrom(), close_range(), fadvise() and posix_fadvise() in addition to
writing tests that confirm things like errno and seeking behavior across
platforms. We now less aggressively polyfill behavior with some of these
functions when the platform support isn't available. Please see:

https://justine.lol/cosmopolitan/functions.html
2022-09-19 15:06:25 -07:00
Justine Tunney
224c12f54d
Make zlib go faster 2022-09-18 06:30:45 -07:00
Justine Tunney
e14ae1642e
Fix MbedTLS Cloudflare interop in MODE=tiny 2022-09-18 03:52:09 -07:00
Justine Tunney
97bb5a3a80
Fix MODE=dbg build 2022-09-18 03:16:14 -07:00
Justine Tunney
bc3d84341d
Fix redbean unix.nanosleep() documentation 2022-09-18 03:02:26 -07:00
Gavin Hayes
4c40c500b8
Add getgroups and setgroups (#619) 2022-09-18 02:48:53 -07:00
Justine Tunney
775944a2d0
Fix important bugs in redbean
This change upgrades to the latest Chromium Zlib, fixes bugs in redbean,
and introduces better support for reverse proxies like Cloudflare. This
change improves the security of redbean and it's recommended that users
upgrade to the release that'll follow. This change also updates the docs
to clarify how to use the security tools redbean provides e.g. pledge(),
unveil(), and the MODE=asan builds which improve memory safety.
2022-09-17 02:18:06 -07:00
Justine Tunney
134ffee519
Change support vector to Windows 8+
Doing this makes binaries tinier, since we don't need to have all the
extra code for supporting a 32-bit address space. It also benefits us
because we're able to use WIN32 futexes, which makes locking simpler.

b69f3d2488 is what officially ended our
Windows 7 support. This change is merely a formalization. You can use
old versions of Cosmo now and forevermore if you need Windows 7 since
our repository is hermetic and vendors all its dependencies.

Won't fix #617
2022-09-15 03:55:05 -07:00
Gavin Hayes
14731d61fa
fix MODE=dbg python build (#620) 2022-09-14 22:53:44 -07:00
Justine Tunney
3f49889841
Make important improvements
- Fix preadv() and pwritev() for old distros
- Introduce _npassert() and _unassert() macros
- Prove that file locks work properly on Windows
- Support fcntl(F_DUPFD_CLOEXEC) on more systems
2022-09-14 22:39:08 -07:00
Justine Tunney
654ceaba7d
Clean up threading code some more 2022-09-13 20:17:34 -07:00
Justine Tunney
aab4ee4072
Add sys_ prefix to unwrapped system calls
This change also implements getlogin() and getlogin_r().
2022-09-13 11:20:35 -07:00
Justine Tunney
06f9a5b627
Get repository to build with GCC 11
See #594
2022-09-13 04:14:55 -07:00
Justine Tunney
22b63d0b98
Add *NSYNC to libc/isystem 2022-09-13 01:47:12 -07:00
Justine Tunney
6f7d0cb1c3
Pay off more technical debt
This makes breaking changes to add underscores to many non-standard
function names provided by the c library. MODE=tiny is now tinier and
we now use smaller locks that are better for tiny apps in this mode.
Some headers have been renamed to be in the same folder as the build
package, so it'll be easier to know which build dependency is needed.
Certain old misguided interfaces have been removed. Intel intrinsics
headers are now listed in libc/isystem (but not in the amalgamation)
to help further improve open source compatibility. Header complexity
has also been reduced. Lastly, more shell scripts are now available.
2022-09-12 23:36:56 -07:00
Justine Tunney
b69f3d2488
Optimize memory layout 2022-09-12 04:26:52 -07:00
Justine Tunney
9b89a7c51d
Add *NSYNC mu_test 2022-09-11 18:56:29 -07:00
Justine Tunney
b5cb71ab84
Use *NSYNC for POSIX threads locking APIs
Condition variables, barriers, and r/w locks now work very well.
2022-09-11 11:04:50 -07:00
Justine Tunney
5dff63a31c
Import nsync 2022-09-10 16:12:31 -07:00
Justine Tunney
cdb2284f0d
Remove stdio lock macros from amalgamation 2022-09-10 12:03:36 -07:00
Justine Tunney
155b378a39
Tidy up the threading implementation
The organization of the source files is now much more rational.
Old experiments that didn't work out are now deleted. Naming of
things like files is now more intuitive.
2022-09-10 02:56:25 -07:00
Justine Tunney
e97f1a99cf
Fix pthread stacks with larger guard size 2022-09-09 06:41:22 -07:00
Paul Kulchenko
f68151c010
Add sigpending to redbean unix.* module (#605) 2022-09-09 04:20:13 -07:00
Justine Tunney
b73e35c6fa
Improve open source compatibility
This change tunes the default stack size for the outside world to 8mb
while at the same time, reducing Cosmopolitan's default stack size to
64kb. You can override the stack size using STATIC_STACK_SIZE(). Your
build scripts should point to o//ape/public/ape.lds

This change also fixes the definition of SOMAXCONN and removes AF_RDS
since it's not polyfilled and Python 3.11 complained.
2022-09-08 03:19:35 -07:00
Justine Tunney
0e2b1bfeed
Make garbage collection thread safe
- You can now use _gc(malloc()) in multithreaded programs
- This change fixes a bug where fork() on NT disabled TLS
- Fixed TLS code morphing on XNU/NT, for R8-R15 registers
2022-09-08 02:33:01 -07:00
Justine Tunney
8bd9ad8342
Simplify redbean serialization code 2022-09-06 20:07:29 -07:00
Justine Tunney
dbf12c30b0
Make more compatibility improvements 2022-09-06 12:36:34 -07:00
Justine Tunney
12d9e1e128
Improve quality of our ANSI C clock() function
It now works most excellently across all supported operating
sytsems (earlier it didn't work on NT and XNU). Demo code is
available in examples/clock.c and this change also adds some
of the newer ANSI C time functions like timespec_get(), plus
timespec_getres() which hasn't even come out yet as it's C23
2022-09-05 23:03:49 -07:00
Justine Tunney
7ff0ea8c13
Make pthread mutexes more scalable
pthread_mutex_lock() now uses a better algorithm which goes much faster
in multithreaded environments that have lock contention. This comes at
the cost of adding some fixed-cost overhead to mutex invocations. That
doesn't matter for Cosmopolitan because our core libraries all encode
locking operations as NOP instructions when in single-threaded mode.
Overhead only applies starting the moment you first call clone().
2022-09-05 15:57:51 -07:00
Gautham
4b4ea046c4
Ensure libcxx.a is built when providing folder (#591) 2022-09-05 09:17:16 -07:00
Justine Tunney
9be364d40a
Implement POSIX threads API 2022-09-05 08:27:15 -07:00
Justine Tunney
8dd4ec68d0
Add more missing C / C++ headers 2022-09-04 04:53:52 -07:00
Gavin Hayes
263711965f
Change sigaction_f to match sysv signature (#585) 2022-09-02 05:08:35 -07:00
Justine Tunney
8cab92a069 Release awk-2022-08-22.com 2022-08-22 21:28:19 -07:00
Justine Tunney
5a632cf72d Add lchown, lchmod, statvfs, fstatvfs 2022-08-22 20:50:16 -07:00
Gautham
7b74f6e20e
Tidy up pydoc help (#567) 2022-08-22 20:40:00 -07:00
Justine Tunney
253cc1754b Shave another 2% off GNU Make latency 2022-08-22 17:42:32 -07:00
Justine Tunney
8722006c46 Shave another 10% off GNU Make latency
We avoid memcpy() by creating a likely branch with a constant size.
2022-08-22 17:04:54 -07:00
Justine Tunney
58d22c7c3d Handle missing stdio handles on Windows 2022-08-22 17:03:11 -07:00
Justine Tunney
2227cefa5d Run fix-third-party.py and man2txt on sed 2022-08-21 22:26:48 -07:00
Justine Tunney
f0bf9f3389 Import NetBSD sed 2022-08-21 22:26:48 -07:00
Justine Tunney
bf757c4646 Fix _Thread_local with fork() on Windows 2022-08-21 15:51:44 -07:00
Justine Tunney
99a92048b4 Get awk to build and make it hackable 2022-08-21 13:39:20 -07:00
Justine Tunney
2f1679e5cf Import The One True Awk
https://github.com/onetrueawk/awk
2022-08-21 13:38:12 -07:00
Justine Tunney
73845be1f0 Restore zip.com and .symtab files
This change restores the .symtab symbol table files in our flagship
programs (e.g. redbean.com, python.com) needed to show backtraces. This
also rolls back earlier changes to zip.com w.r.t. temp directories since
the right way to do it turned out to be the -b DIR flag.

This change also improves the performance of zip.com. It turned out
mmap() wasn't being used, because zip.com was assuming a 4096-byte
granularity, but cosmo requires 65536. There was also a chance to speed
up stdio scanning using the unlocked functions.
2022-08-21 00:17:20 -07:00
Justine Tunney
35203c0551 Do some string library work 2022-08-20 22:17:14 -07:00
Justine Tunney
83d41e4588 Clean up some code 2022-08-20 12:32:51 -07:00
Justine Tunney
d76dfadc7a Reduce GNU Make latency 17% for cosmo
fgets() is now 4x faster which makes Make 2% faster. Landlock Make now
has a builtin $(uniq ...) function that uses critbit trees rather than
functional programming. Since uniq is the most important function this
optimization makes our cold start latency 15% faster.
2022-08-19 15:29:24 -07:00
Justine Tunney
8835b82a7c Avoid sandboxing directory prerequisites
Landlock Make will no longer sandbox prerequisites that end with a
trailing slash. This means you can use use directory prerequisites
for detecting deleted files when using using globbing, without the
effect of unveiling the entire directory. When you do want make to
unveil directories, you can omit the trailing slash.
2022-08-19 13:02:25 -07:00
Justine Tunney
283fd9e590 Pledge tty on Python ioctl test 2022-08-18 18:02:49 -07:00
Justine Tunney
30e1c5bca9 Add statfs() and fstatfs() to redbean unix module 2022-08-17 23:27:17 -07:00
Justine Tunney
8e176fb026 Reduce build graph by another 14 percent
That's a 37% reduction total from what it was on 2022-05-23.
2022-08-17 20:01:21 -07:00
Justine Tunney
f7ee9d7d99 Polyfill statfs() and fstatfs() on BSD distros 2022-08-17 14:54:03 -07:00
Justine Tunney
a1aaf23dc1 Add more apis to redbean unix module
- Document unix.fcntl()
- Add POSIX Advisory Locks
- Add mask parameter to unix.poll()
- Add lowest parameter to unix.dup()
2022-08-16 23:23:34 -07:00
Justine Tunney
ce588dd56b Release pledge.com 1.7 and landlockmake.com 1.3
- pledge("chown") now supported
- pledge("stdio") now allows killing self
- Write tests for pselect() and ppoll()
2022-08-15 19:52:00 -07:00
Justine Tunney
f0701d2a24 Make improvements
- Polyfill pselect() on Windows
- Add -O NOFILE flag to pledge.com
- Polyfill ppoll() on NetBSD, XNU, and Windows
- Support negative numbers and errno in sizetol()
- Add .RSS, .NOFILE, and .MAXCORE to Landlock Make
- Fix issue with .PLEDGE preventing touching of output files
- Add __watch() function (like ftrace) for logging memory changes
2022-08-15 15:20:36 -07:00
Justine Tunney
0cf9716039 Improve build latency of repository 2022-08-14 22:19:46 -07:00
Justine Tunney
7ab15e0b23 Add .PLEDGE/.CPU/.MEMORY/etc. to Landlock Make 1.2 2022-08-14 20:16:44 -07:00
Justine Tunney
6c0bbfac4a Fix some bugs 2022-08-14 13:28:07 -07:00
Justine Tunney
d36d0634db Add automatic TMPDIR setup/teardown to GNU Make
We now guarantee TMPDIR will be defined on a per build rule basis. It'll
be an absolute path. It'll be secure and unique. It'll be rm -rf'd after
the last shell script line in your build rule is executed. If $TMPDIR is
already defined, then it'll be created as a subdirectory of your $TMPDIR
and then replace the variable with the new definition. The Landlock Make
repository will be updated with examples shortly after this change which
shall be known as Landlock Make 1.1.1.

See #530
2022-08-14 02:03:41 -07:00
Justine Tunney
ead3fc2b31 Fix Landlock Make so it can read pattern rule vars
It turned out that specifying all SRCS and INCS as dependencies on the
pattern rules for all headers, caused `make` memory usage to skyrocket
from 40mb ot 160mb. This change also reduces the build graph another 4%.
2022-08-13 17:23:05 -07:00
Justine Tunney
ad775a75b8 Restore strict header checking 2022-08-13 16:02:01 -07:00
Justine Tunney
b212f49ef4 Fix build rule for pushing cosmo nightlies 2022-08-13 15:16:06 -07:00
Justine Tunney
0ea0d33a77 Reduce build graph by another eight percent 2022-08-13 13:11:56 -07:00
Justine Tunney
367d06d9e4 Fold LIBC_UNICODE into LIBC_STR 2022-08-13 08:42:32 -07:00
Justine Tunney
17aea99bb3 Fold LIBC_ALG into LIBC_MEM 2022-08-13 08:32:34 -07:00
Justine Tunney
7cf66bc161 Prevent Make from talking to public Internet
This change introduces the nointernet() function which may be called to
prevent a process and its descendants from communicating with publicly
routable Internet addresses. GNU Make has been modified to always call
this function. In the future Landlock Make will have a way to whitelist
subnets to override this behavior, or disable it entirely. Support is
available for Linux only. Our firewall does not require root access.

Calling nointernet() will return control to the caller inside a new
process that has a SECCOMP BPF filter installed, which traps network
related system calls. Your original process then becomes a permanent
ptrace() supervisor that monitors all processes and threads descending
from the returned child. Whenever a networking system call happens the
kernel will stop the process and wakes up the monitor, which then peeks
into the child memory to read the sockaddr_in to determine if it's ok.

The downside to doing this is that there can be only one supervisor at a
time using ptrace() on a process. So this firewall won't be enabled if
you run make under strace or inside gdb. It also makes testing tricky.
2022-08-12 21:51:39 -07:00
Justine Tunney
8a0a2c0c36 Fold LIBC_RAND into LIBC_STDIO/TINYMATH/INTRIN 2022-08-11 12:32:00 -07:00
Justine Tunney
05b8f82371 Fold LIBC_BITS into LIBC_INTRIN 2022-08-11 12:13:18 -07:00
Justine Tunney
625aa365f1 Refactor pledge() to be more configurable
The earlier iterations did too much guesswork when it came to things
like stderr logging and syscall origin verification. This change will
make things more conformant to existing practices. The __pledge_mode
extension now can be configured in a better way.

There's also a new `-q` flag added to pledge.com, e.g.

    o//tool/build/pledge.com -qv. ls

Is a good way to disable warnings about `tty` access attempts.
2022-08-11 11:35:30 -07:00
Justine Tunney
10fd8bdb70 Unbloat the build
This change resurrects ae5d06dc53
2022-08-11 00:15:29 -07:00
Justine Tunney
2d64b9994b Avoid creating temporary output files
This change also removes the futimens() call on the Landlock Make output
file workaround, since it caused problems with commands like fixupobj
which modify-in-place. It turns out if a file is opened for writing and
then no writes actually occur, then the modified time doesn't change.
2022-08-10 22:31:16 -07:00
Justine Tunney
4c3ab6d11e Make it easier to configure Landlock Make 2022-08-10 21:24:31 -07:00
Justine Tunney
7b993d561c Use private keyword on makefile target variables 2022-08-10 18:36:11 -07:00
Justine Tunney
e015defa70 Fix Make regression with .UNSANDBOXED 2022-08-10 17:06:59 -07:00
Justine Tunney
37ebd8f8d5 Don't generate output file for .PHONY targets 2022-08-10 15:52:06 -07:00
Justine Tunney
7e1c78b4f3 Cherry-pick bug fixes and Landlock Make 1.0.2 2022-08-10 12:56:45 -07:00
Justine Tunney
c1d99676c4 Revert "Unbloat build config"
This reverts commit ae5d06dc53.
2022-08-10 12:44:56 -07:00
Justine Tunney
95f54eeb40 Release Landlock Make v1.0.1 2022-08-10 12:17:06 -07:00
Justine Tunney
5c5cf0e01d Revert "Release Landlock Make 1.0"
This reverts commit 6746d768c6.
2022-08-10 12:13:47 -07:00
Danny Robinson
de48bae30e
Improve cosmopolitan libc documentation (#516) 2022-08-10 10:21:35 -07:00
Justine Tunney
6746d768c6 Release Landlock Make 1.0 2022-08-10 09:27:04 -07:00
Justine Tunney
ae5d06dc53 Unbloat build config
- 10.5% reduction of o//depend dependency graph
- 8.8% reduction in latency of make command
- Fix issue with temporary file cleanup

There's a new -w option in compile.com that turns off the recent
Landlock output path workaround for "good commands" which do not
unlink() the output file like GNU tooling does.

Our new GNU Make unveil sandboxing appears to have zero overhead
in the grand scheme of things. Full builds are pretty fast since
the only thing that's actually slowed us down is probably libcxx

    make -j16 MODE=rel
    RL: took 85,732,063µs wall time
    RL: ballooned to 323,612kb in size
    RL: needed 828,560,521µs cpu (11% kernel)
    RL: caused 39,080,670 page faults (99% memcpy)
    RL: 350,073 context switches (72% consensual)
    RL: performed 0 reads and 11,494,960 write i/o operations

pledge() and unveil() no longer consider ENOSYS to be an error.
These functions have also been added to Python's cosmo module.

This change also removes some WIN32 APIs and System Five magnums
which we're not using and it's doubtful anyone else would be too
2022-08-10 04:43:09 -07:00
Justine Tunney
133c693650 Work around Landlock output inode in compile.com
This change fixes Landlock Make so that only the output target file is
unveiled, rather than unveiling the directory that contains it. This
gives us a much stronger sandbox. It also helped identify problematic
build code in our repo that should have been using o/tmp instead.

Landlock isn't able to let us unveil files that don't exist. Even if
they do, then once a file is deleted, the sandboxing for it goes away.
This caused problems for Landlock Make because tools like GNU LD will
repeatedly delete and recreate the output file. This change uses the
compile.com wrapper to ensure on changes happen to the output inode.

New binary available on https://justine.lol/make/

Fixes #528
2022-08-09 07:55:44 -07:00
Justine Tunney
c464f45692 Clean up some variables 2022-08-09 02:55:24 -07:00
Justine Tunney
0277d7d6e9 Rewrite Linux pledge() code so it can be a payload
It's now possible to build our pledge() polyfill as a dynamic shared
object that can be injected into a glibc executable using LD_PRELOAD
2022-08-08 11:41:08 -07:00
Justine Tunney
b77cae2d57 Fix some regressions with execution 2022-08-07 22:10:18 -07:00
Justine Tunney
5546559034 Improve pledge() usability and consistency
- We now kill the program on violations like OpenBSD
- We now print a message explaining which promise is needed
- This change also fixes a linkage bug with thread local storage
- Your sigaction() handlers should now be more thread safe

A new `__pledge_mode` global has been introduced to make pledge() more
customizable on Linux. For example:

    __attribute__((__constructor__)) static void init(void) {
      __pledge_mode = SECCOMP_RET_ERRNO | EPERM;
    }

Can be used to restore our old permissive pledge() behavior.
2022-08-07 16:18:33 -07:00
Justine Tunney
13c1c45075 Make some last minute improvements to make.com 2022-08-07 05:59:53 -07:00
Justine Tunney
6a5717a48f Make more libc improvements
- Make memmem() faster
- Make readdir() thread safe
- Remove 64kb limit from mkdeps.com
- Add old crypt() function from Musl
- Improve new fix-third-party.py tool
- Improve libc/isystem/ headers and fix bugs
2022-08-06 17:18:40 -07:00
Justine Tunney
a8cf0f7e89 Add more missing libc functionality 2022-08-06 10:50:51 -07:00
Justine Tunney
cf93ecbbb2 Prove that Makefile is fully defined
The whole repository is now buildable with GNU Make Landlock sandboxing.
This proves that no Makefile targets exist which touch files other than
their declared prerequisites. In order to do this, we had to:

  1. Stop code morphing GCC output in package.com and instead run a
     newly introduced FIXUPOBJ.COM command after GCC invocations.

  2. Disable all the crumby Python unit tests that do things like create
     files in the current directory, or rename() files between folders.
     This ended up being a lot of tests, but most of them are still ok.

  3. Introduce an .UNSANDBOXED variable to GNU Make to disable Landlock.
     We currently only do this for things like `make tags`.

  4. This change deletes some GNU Make code that was preventing the
     execve() optimization from working. This means it should no longer
     be necessary in most cases for command invocations to be indirected
     through the cocmd interpreter.

  5. Missing dependencies had to be declared in certain places, in cases
     where they couldn't be automatically determined by MKDEPS.COM

  6. The libcxx header situation has finally been tamed. One of the
     things that makes this difficult is MKDEPS.COM only wants to
     consider the first 64kb of a file, in order to go fast. But libcxx
     likes to have #include lines buried after huge documentation.

  7. An .UNVEIL variable has been introduced to GNU Make just in case
     we ever wish to explicitly specify additional things that need to
     be whitelisted which aren't strictly prerequisites. This works in
     a manner similar to the recently introduced .EXTRA_PREREQS feature.

There's now a new build/bootstrap/make.com prebuilt binary available. It
should no longer be possible to write invalid Makefile code.
2022-08-06 04:05:08 -07:00
Justine Tunney
af3df0893b Refactor gettimeofday() 2022-08-05 02:13:41 -07:00
Justine Tunney
ce5cb8a2f8 Have redbean Lua repl show utf-8 when it's valid
Otherwise EncodeLua() will assume strings are binary and display them
using hex sequences.
2022-07-23 07:22:19 -07:00
Justine Tunney
31e4b0867b Pledge and unveil redbean's unit tests 2022-07-23 07:22:19 -07:00
Justine Tunney
76d2f68c91 Release pledge.com v1.1
This change fixes bugs, adds more system calls, and improves
compatibility with OpenBSD. Going forward, versions on the web will be
pinned to a permanent version. There were many other changes over the
last week which also improved this new release.
2022-07-22 13:44:00 -07:00
Justine Tunney
516b68606f Add pretty printing to redbean serializers 2022-07-22 10:10:33 -07:00
Justine Tunney
31e746c937 Add more missing libc/libc++ functions
- Add sched_rr_get_interval()
- Add `unbuffer` command example
- Add more locale function stubs
- Vendor most of remaining libcxx content
2022-07-22 07:20:21 -07:00
Justine Tunney
84caee23ba Make sorted serialization faster
Redbean Lua and JSON serialization now goes faster because we're now
inserting object entries into tree data structure rather than making
an array and sorting it at the end. For example, when serializing an
object with 10,000 entries this goes twice as fast. However it still
goes slower than saying EncodeJson(x, {sorted=false}).
2022-07-22 04:19:01 -07:00
Jared Miller
9de3d8f1e6
Revert whitespace fixes to third_party (#501) 2022-07-21 21:46:07 -07:00
Justine Tunney
8b469389f6 Remove plenty of makefile misconfigurations 2022-07-21 09:20:59 -07:00
Justine Tunney
9172fd42a0 Add Landlock sandboxing to GNU Make 2022-07-21 09:16:27 -07:00
Justine Tunney
1837dc2e85 Make improvements
- Introduce path module to redbean
- Fix glitch with linenoise printing extra line on eof
- Introduce closefrom() and close_range() system calls
- Make file descriptor closing more secure in pledge.com
2022-07-21 03:36:42 -07:00
Jared Miller
7e2eae5c15
Remove trailing whitespace from all files (#497) 2022-07-20 20:31:16 -07:00
jared
ed205e98a1
WIP: Correct all typos (#498) 2022-07-20 14:01:15 -07:00
Justine Tunney
69f4152f38 Always initialize thread local storage
We had previously not enabled TLS in MODE=tiny in order to keep the
smallest example programs (e.g. life.com) just 16kb in size. But it
was error prone doing that, so now we just always enable it because
this change uses hacks to ensure it won't increase life.com's size.

This change also fixes a bug on NetBSD, where signal handlers would
break thread local storage if SA_SIGINFO was being used. This looks
like it might be a bug in NetBSD, but it's got a simple workaround.
2022-07-19 00:21:46 -07:00
Justine Tunney
e81edf7b04 Improve pledge() and unveil()
The pledge.com command now supports the new [WIP] unveil() support. For
example, to strongly sandbox our command for listing directories.

    o//tool/build/assimilate.com o//examples/ls.com
    pledge.com -v /etc -p 'stdio rpath' o//examples/ls.com /etc

This file system sandboxing is going to be perfect for us, because APE
binaries are self-contained static executables that really don't use the
filesystem that much. On the other hand, with non-static executables,
sandboxing is going to be more difficult. For example, here's how to
sandbox the `ls` command on the latest Alpine:

    pledge.com -v rx:/lib -v /usr/lib -v /etc -p 'stdio rpath exec' ls /etc

This change fixes the `execpromises` API with pledge().

This change also adds unix.unveil() to redbean.

Fixes #494
2022-07-18 07:58:20 -07:00
Justine Tunney
4f4889ddf7 Use futexes on OpenBSD and improve threading 2022-07-17 19:59:49 -07:00
Justine Tunney
5b11033d4d Add redbean -I flag for launching browser
It's now possible with any redbean (including redbean-original) to
launch the system web browser without having to use the Lua API. For
example, you can create an args file:

    echo -I/ >.args
    zip redbean-original.com .args

That will white-label redbean so it launches a specific page when you
double-click on the executable.

See https://github.com/jart/cosmopolitan/discussions/472
2022-07-17 06:12:57 -07:00
Justine Tunney
4d25f8c3c9 Add tcp syn packet fingerprinting to redbean
This change also fixes bugs in enoprotoopt reporting with setsockopt and
getsockopt error returns.
2022-07-17 02:43:49 -07:00
Justine Tunney
2e3958c6dc Make some touchups 2022-07-15 18:07:34 -07:00
Justine Tunney
28b9d9f781 Fix C stack remaining check in MODE=tiny 2022-07-14 07:23:15 -07:00
Justine Tunney
b707fca77a Make JSON parser perfectly conformant 2022-07-13 23:02:19 -07:00
Justine Tunney
60164a7266 Add assimilate.com command for APE binaries 2022-07-13 20:56:24 -07:00
Justine Tunney
e3cd476a9b Improve Lua and JSON serialization 2022-07-12 23:35:11 -07:00
Justine Tunney
3f3e7e92d7 Audit every single JSON test 2022-07-12 12:40:38 -07:00
Justine Tunney
6ee18986e4 Further improve JSON serialization 2022-07-11 23:06:49 -07:00
Justine Tunney
3f015b1e51 Make some minor fixups to bug reporting, etc. 2022-07-11 05:58:24 -07:00
ProducerMatt
84764ce7b8
lre_test.lua: Benchmark() fails on some systems (#474) 2022-07-10 23:05:50 -07:00
Justine Tunney
5f4f6b0e69 Make _Thread_local work across platforms
We now rewrite the binary image at runtime on Windows and XNU to change
mov %fs:0,%reg instructions to use %gs instead. There's also simpler
threading API introduced by this change and it's called _spawn() and
_join(), which has replaced most clone() usage.
2022-07-10 04:01:17 -07:00
Justine Tunney
ee82cee432 Further improve json / lua serialization 2022-07-09 16:27:26 -07:00
Justine Tunney
2189877856 Implement new JSON parser for redbean 2022-07-09 11:44:19 -07:00
Justine Tunney
d37536bd4b Fix oops w/ array serialization ordering 2022-07-09 10:29:26 -07:00
Justine Tunney
727d9cbf56 Bump redbean to 2.0.9 2022-07-09 05:50:05 -07:00
Justine Tunney
c9e68b0ebc Make redbean serialization deterministic 2022-07-09 04:09:51 -07:00
Justine Tunney
1c83670229 Write more redbean unit tests
- Fix DescribeSigset()
- Introduce new unix.rmrf() API
- Fix redbean sigaction() doc example code
- Fix unix.sigaction() w/ more than two args
- Improve redbean re module API (non-breaking)
- Enhance Lua with Python string multiplication
- Make third parameter of unix.socket() default to 0
2022-07-08 23:10:02 -07:00
Justine Tunney
50a6df89b8 Fix Lua type of zero
This change fixes a regression from 281a0f27 which resulted in the
integer literal `0` being interpreted as a floating point number. This
should also fix a reported issue with Fennel integration.
2022-07-08 10:12:19 -07:00
Justine Tunney
853b6c3864 Improve system calls
- Wrap clock_getres()
- Wrap sched_setscheduler()
- Make sleep() api conformant
- Polyfill sleep() using select()
- Improve clock_gettime() polyfill
- Make nanosleep() POSIX conformant
- Slightly improve some DNS functions
- Further strengthen pledge() sandboxing
- Improve rounding of timeval / timespec
- Allow layering of pledge() calls on Linux
- Polyfill sched_yield() using select() on XNU
- Delete more system constants we probably don't need
2022-07-08 06:42:03 -07:00
Justine Tunney
3c92adfd6e Strengthen the pledge() polyfill 2022-06-27 13:02:17 -07:00
Gautham
b535937fca
Add sys.meta_path entry for APE zip store (#425) 2022-06-26 05:51:00 -07:00
Justine Tunney
5cd0e876bd Scrub double-conversion and add quick wrapper 2022-06-26 05:03:25 -07:00
Gautham
b1de862e16
Add double-conversion to third_party (#447) 2022-06-26 03:41:55 -07:00
Justine Tunney
2f515ac5e3 Fix code completion bug in Lua REPL 2022-06-26 03:11:56 -07:00
Justine Tunney
61257d48d4 Make some quick fixes and cleanup 2022-06-26 02:58:36 -07:00
Justine Tunney
fbc053e018 Make fixes and improvements
- Introduce __assert_disable global
- Improve strsignal() thread safety
- Make system call tracing thread safe
- Fix SO_RCVTIMEO / SO_SNDTIMEO on Windows
- Refactor DescribeFoo() functions into one place
- Fix fork() on Windows when TLS and MAP_STACK exist
- Round upwards in setsockopt(SO_RCVTIMEO) on Windows
- Disable futexes on OpenBSD which seem extremely broken
- Implement a better kludge for monotonic time on Windows
2022-06-25 21:09:09 -07:00
Justine Tunney
4b9ee980a1 Release redbean 2.0.6 2022-06-23 17:37:55 -07:00
Justine Tunney
a4601a24d3 Perform some code cleanup 2022-06-23 10:21:07 -07:00
Theta Nil
0dd9629562
Fix GitHub Actions build (#442) 2022-06-23 07:04:59 -07:00
Theta Nil
2b54f1bcf6
Migrate from Travis to GitHub Actions (#441) 2022-06-23 04:39:00 -07:00
Justine Tunney
206f073181 Add stat mode macros to redbean unix api 2022-06-23 04:05:51 -07:00
Justine Tunney
17cbe73411 Add finger demo to redbean and fix regression
This change fixes a regression in unix.connect() caused by the recent
addition of UNIX domain sockets. The BSD finger command has been added
to third_party for fun and profit. A new demo has been added to redbean
showing how a protocol as simple as finger can be implemented.
2022-06-23 03:42:05 -07:00
Justine Tunney
fc097ac275 Add unix domain socket support to redbean 2022-06-22 03:04:25 -07:00
Justine Tunney
1216b6e7c7 Add xadd to chibicc 2022-06-21 04:55:27 -07:00
Justine Tunney
f317a47cd8 Fixup and polish chibicc some more
Fixes #433
2022-06-20 04:32:25 -07:00
Justine Tunney
a988896048 Add atomics to chibicc
This change also fixes #434 and makes the chibicc assembler better.
2022-06-20 03:08:00 -07:00
Justine Tunney
d5312b60f7 Make improvements to locking
This change makes pthread_mutex_lock() as fast as _spinlock() by
default. Thread instability issues on NetBSD have been resolved.
Improvements made to gdtoa thread code. Crash reporting will now
synchronize between threads in a slightly better way.
2022-06-19 01:30:12 -07:00
Justine Tunney
25041b8026 Remove unsafe emacs config from tidy 2022-06-19 00:59:58 -07:00
Justine Tunney
c1cfca8ae1 Make fixes and improvements
- Polyfill UTIME_OMIT on XNU
- Refactor Lua build code so it's better
- Add unix module to lua.com (Discord request)
- Add unix.utimensat() and unix.futimens() to redbean
- Avoid creating double slash path in linenoise (#428)
- Remove double slashes in NT paths automatically (#428)
- Make strerror() smarter about showing NT errors (#428)

Fixes #428
2022-06-18 01:46:12 -07:00
Justine Tunney
34e39ad027 Enable sqlite zipfile module in redbean
This change also breaks out a bunch of extension files that the SQLite
authors inlined into a shell.c amalgamation.
2022-06-17 02:49:04 -07:00
Justine Tunney
e466dd0553 Add torture test for zipos file descriptors
This change hardens the code for opening /zip/ files using the system
call interface. Thread safety and signal safety has been improved for
file descriptors in general. We now document fixed addresses that are
needed for low level allocations.
2022-06-15 16:29:49 -07:00
ProducerMatt
bba7f5e296
Extend quotas for various slow tests (#420) 2022-06-14 17:21:57 -07:00
Justine Tunney
1c387727fe Add long double printf formatting support
You can now easily print numbers that have 80 bits of precision. This
change also imports many of the test vectors from the gdtoa codebase.
2022-06-14 02:18:03 -07:00
Justine Tunney
a3865ecc3c Make more fixes and improvements
- Fix Makefile flaking due to ZIPOBJ_FLAGS generation
- Make printf() floating point and gdtoa thread safe
- Polish up the runit / runitd programs some more
- Prune some more makefile dependencies
2022-06-13 11:02:13 -07:00
Justine Tunney
8cdec62f5b Apply even more fixups
- Finish cleaning up the stdio unlocked APIs
- Make __cxa_finalize() properly thread safe
- Don't log locks if threads aren't being used
- Add some more mutex guards to places using _mmi
- Specific lock names now appear in the --ftrace logs
- Fix mkdeps.com generating invalid Makefiles sometimes
- Simplify and fix bugs in the test runner infrastructure
- Fix issue where sometimes some functions wouldn't be logged
2022-06-12 11:57:00 -07:00
Justine Tunney
29af890efa Disable malloc mutex when not using threads 2022-06-11 19:18:09 -07:00
Justine Tunney
41c86fe86b Fix build error in MODE=dbg 2022-06-10 20:25:29 -07:00
Justine Tunney
8a967ae6fe Fix header dependencies in third_party/make 2022-06-10 13:07:11 -07:00
Justine Tunney
af4687cc3f Add Info-ZIP unzip.com 2022-06-10 05:09:11 -07:00
Justine Tunney
1531f1a4a7 Add CTRL-U shortcut to QuickJS REPL 2022-06-10 03:39:05 -07:00
Justine Tunney
a41669dec6 Make dlmalloc a little faster
This change also documents the libc arena allocator.
2022-06-09 21:25:21 -07:00
Justine Tunney
fa1e8a3e65 Make emacs not croak when editing dlmalloc 2022-06-09 17:45:30 -07:00
Justine Tunney
3c7ae0fc72 Add w3c html tidy 2022-06-09 06:51:08 -07:00
Justine Tunney
adac64a52b Reduce makefile dependencies by 10%
The includes in libc/calls/calls.h have now been refactored so that
functions with struct parameters are declared in libc/calls/struct/
2022-06-08 20:01:28 -07:00
Justine Tunney
4e7ce1538c Fix linenoise flickering on CTRL+L 2022-06-08 18:40:51 -07:00
Justine Tunney
13ee75150c Improve redbean plus code size optimizations
This change turns symbol table compression back on using Puff, which
noticeably reduces the size of programs like redbean and Python. The
redbean web server receives some minor API additions for controlling
things like SSL in addition to filling gaps in the documentation.
2022-05-29 08:21:19 -07:00
Justine Tunney
425ff5dff0 Refactor some code
- Write tests for cthreads
- Fix bugs in pe2.com tool
- Fix ASAN issue with GetDosEnviron()
- Consolidate the cthread header files
- Some code size optimizations for MODE=
- Attempted to squash a tls linker warning
- Attempted to get futexes working on FreeBSD
2022-05-28 14:09:21 -07:00
Gautham
909e54510d
Fix Python test_repl in MODE=dbg (#412) 2022-05-28 05:50:42 -07:00
Justine Tunney
de5de19004 Make improvements
- Document redbean's argon2 module
- Fix regressions in cthreads library
- Make testlib work better with threads
- Give the cthreads library lots of love
- Remove some of the stdio assembly code
- Implement getloadavg() across platforms
- Code size optimizations for errnos, etc.
- Only check for signals in main thread on Windows
- Make errnos for dup2 / dup3 consistent with posix

This change also fixes a bug in the argon2 module, where the NUL
terminator was being included in the hash encoded ascii string. This
shouldn't require any database migrations to folks who found this module
and productionized it, since the argon2 library treats it as a c string.
2022-05-28 00:28:09 -07:00
Gautham
7e9fb0a9f1
Move importlib functions to within C (#408)
This offers a 10% speedup in Python startup time. It also
makes debugging using cosmopolitan tooling easier.
2022-05-26 23:20:59 -07:00
Justine Tunney
10b97ca630 Add test for sendfile() and reduce branches 2022-05-25 22:29:10 -07:00
Justine Tunney
8f12cd980d Fix execution bug w/o binfmt_misc
The longjmp relocation in the master boot record coincidentally had a
quote character in it, which caused the master boot record to be used
when interpreter by the shell. The solution, is to move the grub stub
below the long mode loader so the real mode loader relocation is near
the master boot record. This change includes a regression test.
2022-05-25 17:24:02 -07:00
Justine Tunney
1a29424c52 Fix ARG_MAX with chibicc documentation generator 2022-05-25 14:06:11 -07:00
Justine Tunney
d230a01222 Make build hermetic without shell scripts
- Fix some minor issues in ar.com
- Have execve() look for `ape` command
- Rewrite NT paths using /c/ rather /??/c:/
- Replace broken GCC symlinks with .sym files
- Rewrite $PATH environment variables on startup
- Make $(APE_NO_MODIFY_SELF) the default bootloader
- Add all build command dependencies to build/bootstrap
- Get the repository mostly building from source on non-Linux
2022-05-25 13:55:57 -07:00
Justine Tunney
d44ff6ce1f Make improvements
- Implement openpty()
- Add `--assimilate` flag to APE bootloader
- Restore Linux vDSO clock_gettime() support
- Use `$(APE_NO_MODIFY_SELF)` on more programs
2022-05-24 10:58:48 -07:00
Justine Tunney
99e67c348b Reduce Makefile dependencies by 4% 2022-05-23 15:07:01 -07:00
Justine Tunney
312ed5c67c Fix some issues and do some code cleanup 2022-05-23 10:15:53 -07:00
Justine Tunney
1f229e4efc Use re-entrant locks on stdio 2022-05-22 08:28:33 -07:00
ProducerMatt
056dc5f554
Extend quotas for various slow tests (#411) 2022-05-21 20:13:01 -07:00
Justine Tunney
db0d8dd806 Support Linux binfmt_misc and APE loading on Apple
The "no modify self" variant of Actually Portable Executable is now
supported on all platforms. If you use `$(APE_NO_MODIFY_SELF)` then
ld.bfd will embed a 4096 byte ELF binary and a 4096 byte Macho file
which are installed on the fly to ${TMPDIR:-/tmp}, which enables us
launch the executable, without needing to copy the whole executable

To prevent it from copying a tiny executable to your temp directory
you need to install the `ape` command (renamed from ape-loader), to
a system path. For example:

    # FreeBSD / NetBSD / OpenBSD
    make -j8 o//ape/ape
    cp o//ape/ape /usr/bin/ape

    # Mac OS
    # make -j8 o//ape/ape.macho
    curl https://justine.lol/ape.macho >/usr/bin/ape
    chmod +x /usr/bin/ape

On Linux you can get even more performance with the new binfmt_misc
support which makes launching non-modifying APE binaries as fast as
launching ELF executables. Running the following command:

    # Linux
    ape/apeinstall.sh

Will copy APE loader to /usr/bin/ape and register with binfmt_misc
Lastly, this change also fixes a really interesting race condition
with OpenBSD thread joining.
2022-05-21 09:28:25 -07:00
Justine Tunney
96781d0679 Fix mistake with previous commit 2022-05-20 15:08:02 -07:00
Justine Tunney
f4fc16754a Fix build invalidation issue with GNU Make
The `HDRS` and/or `INCS` variables need to be defined in order for
`mkdeps.com` to compute build dependencies for `o//depend`. We're using
INCS here since makeint.h is the evil kind of header that isn't actually
defining a library interface but rather is a hodgepodge of common code.
2022-05-20 14:20:54 -07:00
Gautham
fb4382e9a4
os.realname variable for correct USER_SITE value (#410)
In site.py, Python uses os.name to decide where the USER_SITE (ie the
folder containing the user's locally installed packages) is located.
With cosmo we have set os.name as "posix" always, so we use a new
os.realname to decide the USER_SITE location.
2022-05-20 08:17:10 -07:00
Justine Tunney
ec2cb88058 Make fixes and improvements
- Document more compiler flags
- Expose new __print_maps() api
- Better overflow checking in mmap()
- Improve the shell example somewhat
- Fix minor runtime bugs regarding stacks
- Make kill() on fork()+execve()'d children work
- Support CLONE_CHILD_CLEARTID for proper joining
- Fix recent possible deadlock regression with --ftrace
2022-05-19 16:57:49 -07:00
Justine Tunney
6e52cba37a Fix stdio regression
This change fixes a nasty regression caused by
80b211e314 which deadlocked.

This change also causes MbedTLS to prefer the ChaCha ciphersuite on
older CPUs that don't have AES hardware instructions.
2022-05-19 00:51:15 -07:00
Justine Tunney
9208c83f7a Make some systemic improvements
- add vdso dump utility
- tests now log stack usage
- rename g_ftrace to __ftrace
- make internal spinlocks go faster
- add conformant c11 atomics library
- function tracing now logs stack usage
- make function call tracing thread safe
- add -X unsecure (no ssl) mode to redbean
- munmap() has more consistent behavior now
- pacify fsync() calls on python unit tests
- make --strace flag work better in redbean
- start minimizing and documenting compiler flags
2022-05-18 16:52:36 -07:00
Justine Tunney
b2c6ec6eab Make linenoise properly recover from backgrounding 2022-05-17 12:10:22 -07:00
Justine Tunney
2743f3d012 Clean up the clone() code a bit 2022-05-17 07:40:00 -07:00
Justine Tunney
ce71677156 Improve threading support further 2022-05-17 04:14:28 -07:00
Justine Tunney
55de4ca6b5 Support thread local storage 2022-05-16 13:20:08 -07:00
Justine Tunney
54e6f564c1 Let ctrl-c interrupt lua server pages in repl mode 2022-05-14 11:47:16 -07:00
Justine Tunney
80b211e314 Add raw memory visualization tool to redbean
This change introduces a `-W /dev/pts/1` flag to redbean. What it does
is use the mincore() system call to create a dual-screen terminal
display that lets you troubleshoot the virtual address space. This is
useful since page faults are an important thing to consider when using a
forking web server. Now we have a colorful visualization of which pages
are going to fault and which ones are resident in memory.

The memory monitor, if enabled, spawns as a thread that just outputs
ANSI codes to the second terminal in a loop. In order to make this
happen using the new clone() polyfill, stdio is now thread safe.

This change also introduces some new demo pages to redbean. It also
polishes the demos we already have, to look a bit nicer and more
presentable for the upcoming release, with better explanations too.
2022-05-14 04:33:58 -07:00
Justine Tunney
578cb21591 Add syscalls to Blinkenlights and fix bugs 2022-05-13 13:31:21 -07:00
Gautham
f6df29cc3d
Add Python JSON tests (#407) 2022-05-13 13:02:15 -07:00
Justine Tunney
d25a67f4eb Fix cross-device link in Python tests 2022-05-13 06:50:55 -07:00
Justine Tunney
77b70ba193 Add fixes to previous commit 2022-05-13 05:05:27 -07:00
Gautham
83b743cf96
Python 3.7 METH_FASTCALL backport (#406) 2022-05-13 05:05:12 -07:00
Justine Tunney
4e62cefa6e Add zlib gzip functions
These will now be included in the cosmopolitan.a releases. It took a bit
of time because, these functions depend on heavyweight parts of the libc
that wouldn't be appropriate for the core zlib library to depend upon.

Fixes #345
2022-05-12 12:17:40 -07:00
Justine Tunney
4499f98e76 Add /.args feature to Redbean/Lua/SQLite/Python/QuickJS
You now have some ability to truly make an executable yours, by adding a
`.args` file to the root of the zip structure. If this is specified,
then you'll be overriding the default CLI args.

This will be a great feature for folks who want to distribute their own
apps, using the interpreter executable, but have the executable appears
to be just your app rather than being the interpreter.
2022-05-12 11:04:47 -07:00
Justine Tunney
2ea1dc405c Revert "Backport METH_FASTCALL from Python 3.7 (#328)"
This reverts commit cf73bbd678.
2022-05-12 06:49:54 -07:00
Justine Tunney
e7611a8476 Make improvements
- Get threads working on NetBSD
- Get threads working on OpenBSD
- Fix Emacs config for Emacs v28
- Improve --strace logging of sigset_t
- Improve --strace logging of struct stat
- Improve memory safety of DescribeThing functions
- Refactor auto stack allocation into LIBC_RUNTIME
- Introduce shell.com example which works on Windows
- Refactor __strace_thing into DescribeThing functions
- Document the CHECK macros and improve them in NDEBUG mode
- Rewrite MAP_STACK so it uses FreeBSD behavior across platforms
- Deprecate and discourage the use of MAP_GROWSDOWN (it's weird)
2022-05-12 06:45:36 -07:00
Gautham
cf73bbd678
Backport METH_FASTCALL from Python 3.7 (#328) 2022-05-12 02:27:16 -07:00
Gautham
363d2ec436
Load Python C extensions of external packages correctly (#391) 2022-05-08 21:49:50 -07:00
Justine Tunney
15c59e716f Fix build break in MODE=asan 2022-05-05 11:12:48 -07:00
Justine Tunney
c9f966d73c Fix build suboptimality
Non-default build modes were having artifacts invalidated needlessly.
2022-05-04 22:51:51 -07:00
Justine Tunney
e4b559c76a Simplify the redbean JSON encoder 2022-04-29 06:35:27 -07:00
Justine Tunney
2d1731b995 Polish redbean serialization 2022-04-29 06:10:10 -07:00
Justine Tunney
7aafa64ab3 Make improvements
- Bump redbean up to 2.0
- Trim down the MODE=tiny build a bit
- Add Indian Standard Time to zoneinfo
2022-04-29 00:42:47 -07:00
Justine Tunney
c9a981fdbe Fix some more reported issues 2022-04-28 20:36:33 -07:00
Justine Tunney
47b3274665 Make improvements
- Add rusage to redbean Lua API
- Add more redbean documentation
- Add pledge() to redbean Lua API
- Polyfill OpenBSD pledge() for Linux
- Increase PATH_MAX limit to 1024 characters
- Untrack sibling processes after fork() on Windows
2022-04-28 09:57:07 -07:00
Justine Tunney
cc0d1ec076 Fix some bugs
- addr2line backtrace should continue on eintr
- lua crashes if we try to iterate a non-table
2022-04-27 20:18:34 -07:00
Justine Tunney
6a145a9262 Make improvements
- Add hierarchical auto-completion to redbean's repl
- Fetch latest localtime() and strftime() from Eggert
- Shave a few milliseconds off redbean start latency
- Fix redbean repl with multi-line statements
- Make the Lua unix module code more elegant
- Harden Lua data structure serialization
2022-04-27 05:39:39 -07:00
Justine Tunney
d57b81aac7 Make improvements
- Add GetCpuCount() API to redbean
- Add unix.gmtime() API to redbean
- Add unix.readlink() API to redbean
- Add unix.localtime() API to redbean
- Perfect the new redbean UNIX module APIs
- Integrate with Linux clock_gettime() vDSO
- Run Lua garbage collector when malloc() fails
- Fix another regression quirk with linenoise repl
- Fix GetProgramExecutableName() for systemwide installs
- Fix a build flake with test/libc/mem/test.mk SRCS list
2022-04-26 16:46:15 -07:00
Paul Kulchenko
e6fab847a0
Update redbean to stringify error objects (#393) 2022-04-25 21:14:36 -07:00
Justine Tunney
72e9be5c20 Update redbean lua example code 2022-04-25 09:31:28 -07:00
Justine Tunney
451e3f73d9 Improve redbean
- Improve serialization
- Add Benchmark() API to redbean
- Refactor UNIX API to be assert() friendly
- Make the redbean Lua REPL print data structures
- Fix recent regressions in linenoise reverse search
- Add -i flag so redbean can be a language interpreter
2022-04-25 08:30:14 -07:00
Justine Tunney
2046c0d2ae Make improvements
- Expand redbean UNIX module
- Expand redbean documentation
- Ensure Lua copyright is embedded in binary
- Increase the PATH_MAX limit especially on NT
- Use column major sorting for linenoise completions
- Fix some suboptimalities in redbean's new UNIX API
- Figured out right flags for Multics newline in raw mode
2022-04-24 10:06:05 -07:00
Justine Tunney
2f56ebfe78 Do code cleanup use duff device linenoise i/o 2022-04-22 18:56:52 -07:00
Justine Tunney
6ff46ca373 Make it possible to compile redbean with chibicc
This cuts build latency down from 5 seconds to 500 milliseconds.
2022-04-22 15:25:04 -07:00
Justine Tunney
c39d6111f2 Improve redbean ctrl-c handling 2022-04-21 15:24:44 -07:00
Justine Tunney
1599b818d9 Make more fixes and improvements 2022-04-21 13:44:59 -07:00
Justine Tunney
9d61e23c80 Make more fixes and improvements
This change attempts to fix some report build issues. It also builds
upon development work described in previous changes.
2022-04-21 09:18:45 -07:00
Justine Tunney
87396f43bc Flatten InfoZIP directory and fix build issues 2022-04-20 22:40:33 -07:00
Justine Tunney
ae638c0850 Fix bugs and make improvements
- Get clone() working on FreeBSD
- Increase some Python build quotas
- Add more atomic builtins to chibicc
- Fix ASAN poisoning of alloca() memory
- Make MODE= mandatory link path tinier
- Improve the examples folder a little bit
- Start working on some more resource limits
- Make the linenoise auto-complete UI as good as GNU readline
- Update compile.com, avoiding AVX codegen on non-AVX systems
- Make sure empty path to syscalls like opendir raises ENOENT
- Correctly polyfill ENOENT vs. ENOTDIR on the New Technology
- Port bestline's paredit features to //third_party/linenoise
- Remove workarounds for RHEL 5.0 bugs that were fixed in 5.1
2022-04-20 10:05:34 -07:00
Justine Tunney
5a132f9652 Add seccomp bpf sandboxing to redbean
It's now possible to pass the `-S` or `-SS` flags to sandbox redbean
worker proecsses after they've been forked. The first `-S` flag is
intended to be a permissive builtin policy that limits system calls to
only that which the various parts of redbean serving need. The second
`-SS` flag is intended to be more restrictive, preventing things like
the Lua extensions you download off the web from using the HTTP client
or sockets APIs. In upcoming changes you'll be able to implement your
own Berkeley Packet Filter sandbox programs and load them via Lua.
2022-04-18 08:54:42 -07:00
Justine Tunney
7166679620 Fix bugs and add security features to redbean
- Fix a regression with the previous change that broke redbean
- Add chroot(), resource limit, seccomp, and other stuff to redbean
- Write lots and lots of documentation
- Iron out more system call issues
2022-04-18 00:01:26 -07:00
Justine Tunney
f1dfa4bdfa Add some more builtins to chibicc
https://justine.lol/cosmopolitan/documentation.html should now contain a
lot of functions that had been missing previously due to not having them
2022-04-17 13:07:07 -07:00
Justine Tunney
a6b02ce5a6 Add lua repl interface to redbean
You can now interact with the global web server state on the command
line, which the web server is running. This supports Emacs shortcuts
with history, readline parity, <tab> completions, plus hints. Enjoy!
2022-04-16 20:31:16 -07:00
Justine Tunney
f6b6204b9e Add readline-style completions to Lua/Python/Almquist 2022-04-16 15:10:05 -07:00
Justine Tunney
933f33bcc1 Include variables in redbean lua traceback 2022-04-16 12:49:34 -07:00
Justine Tunney
dc0ea6640e Fix bugs with recent change
This change makes further effort towards improving our poll()
implementation on the New Technology. The stdin worker didn't work out
so well for Python so it's not being used for now. System call tracing
with the --strace flag should now be less noisy now on Windows unless
you modify the strace.internal.h defines to turn on some optional ones
that are most useful for debugging the system call wrappers.
2022-04-16 10:40:23 -07:00
Justine Tunney
933411ba99 Improve synchronization
- Fix bugs in kDos2Errno definition
- malloc() should now be thread safe
- Fix bug in rollup.com header generator
- Fix open(O_APPEND) on the New Technology
- Fix select() on the New Technology and test it
- Work towards refactoring i/o for thread safety
- Socket reads and writes on NT now poll for signals
- Work towards i/o completion ports on the New Technology
- Make read() and write() intermittently check for signals
- Blinkenlights keyboard i/o so much better on NT w/ poll()
- You can now poll() files and sockets at the same time on NT
- Fix bug in appendr() that manifests with dlmalloc footers off
2022-04-15 15:31:55 -07:00
Paul Kulchenko
233144b19d
Fix memory deallocation while yielding in redbean. (#384)
The yielded coroutine was removed from the stack too early,
leaving it not being anchored, which led to memory freed prematurely.
2022-04-14 18:13:53 -07:00
Justine Tunney
fb7e8ef1e6 Add more raw system calls to redbean
We now have execve, setitimer, sigaction, sigsuspend, and sigprocmask.
2022-04-13 14:43:42 -07:00
Justine Tunney
281a0f2730 Implement raw system call for redbean lua code
You can now call functions like fork() from Lua and it'll work across
all supported platforms, including Windows. This gives you a level of
control of the system that Lua traditionally hasn't been able to have
due to its focus on old portable stdio rather modern POSIX APIs. Demo
code has been added to redbean-demo.com to show how it works.

This change also modifies Lua so that integer literals with a leading
zero will be interpreted as octal. That should help avoid shooting in
the foot with POSIX APIs that frequently use octal mode bits.

This change fixes a bug in opendir(".") on New Technology.

Lastly, redbean will now serve crash reports to private network IPs.
This is consistent with other frameworks. However that isn't served
to public IPs unless the -E flag is passed to redbean at startup.
2022-04-13 08:53:24 -07:00
Justine Tunney
f684e348d4 Improve signals and memory protection
- Document sigaction()
- Simplify New Technology fork() code
- Testing and many bug fixes for mprotect()
- Distribute Intel Xed ILD in the amalgamation
- Turn Xed enums into defines to avoid DWARF bloat
- Improve polyfilling of SA_SIGINFO on BSDs and fix bugs
- setpgid(getpid(), getpid()) on Windows will ignore CTRL-C
- Work around issues relating to NT mappings being executable
- Permit automatic executable stack override via `ape_stack_pf`
2022-04-12 22:11:00 -07:00
Justine Tunney
046c7ebd4a Improve locks and signals
- Introduce fast spinlock API
- Double rand64() perf w/ spinlock
- Improve raise() on New Technology
- Support gettid() across platforms
- Implement SA_NODEFER on New Technology
- Move the lock intrinsics into LIBC_INTRIN
- Make SIGTRAP recoverable on New Technology
- Block SIGCHLD in wait4() on New Technology
- Add threading prototypes for XNU and FreeBSD
- Rewrite abort() fixing its minor bugs on XNU/NT
- Shave down a lot of the content in libc/bits/bits.h
- Let signal handlers modify CPU registers on New Technology
2022-04-12 05:20:17 -07:00
Justine Tunney
4f98ad1054 Improve system call support on NT
- Improve i/o perf on New Technology
- Code cleanup on read() for New Technology
- Fix bad bug with dup() of socket on New Technology
- Clean up some more strace errors on New Technology
2022-04-07 20:30:04 -07:00
Justine Tunney
29bf8b1a30 Make improvements
- Make rand64() thread safe
- Introduce lemur64 lcg prng
- Improve strace on New Technology
- Improve msync() on New Technology
2022-04-07 00:15:35 -07:00
Justine Tunney
f59b5dbd7c Get strace.com working beautifully 2022-04-06 09:43:11 -07:00
Justine Tunney
82f2e758c7 Change the default command prompts 2022-04-06 06:56:11 -07:00
Justine Tunney
ce9bdbb0bf Delete some more unneeded Make code 2022-04-06 06:55:08 -07:00
Justine Tunney
58d74ede4c Flatten GNU Make directory structure 2022-04-06 00:22:54 -07:00
Justine Tunney
04d39d47f1 Fix basename() and dirname() 2022-04-06 00:13:44 -07:00
Justine Tunney
38cb6e71ca Improve alloca() memory safety
Now that all the bugs have been wormed out of the ASAN memory module we
can successfully check for underruns on large stack allocations.
2022-04-01 22:44:43 -07:00
Paul Kulchenko
2fe386ff8c
Fix Redbean EncodeUrl params w/o values (#376) 2022-03-27 22:36:43 -07:00
Justine Tunney
98909b1391 Fix more things
- Update a couple unicode data files
- Disable strace during logger calls
- SQLite now uses pread() / pwrite()
- pread() past EOF on NT now returns 0
- Make the NT mmap() and fork() code elegant
- Give NT a big performance boost with memory
- Add many more mmap() tests to prove it works
2022-03-24 00:12:29 -07:00