Commit graph

177 commits

Author SHA1 Message Date
Justine Tunney
91ee2b19d4 Polish greenbean example a bit more
Windows support for this example is still a work in progress. It's
encountering some unusual crashes. Thank you Chris Wellons for the cool
synchronization code too!
2022-05-15 09:19:49 -07:00
Justine Tunney
80b211e314 Add raw memory visualization tool to redbean
This change introduces a `-W /dev/pts/1` flag to redbean. What it does
is use the mincore() system call to create a dual-screen terminal
display that lets you troubleshoot the virtual address space. This is
useful since page faults are an important thing to consider when using a
forking web server. Now we have a colorful visualization of which pages
are going to fault and which ones are resident in memory.

The memory monitor, if enabled, spawns as a thread that just outputs
ANSI codes to the second terminal in a loop. In order to make this
happen using the new clone() polyfill, stdio is now thread safe.

This change also introduces some new demo pages to redbean. It also
polishes the demos we already have, to look a bit nicer and more
presentable for the upcoming release, with better explanations too.
2022-05-14 04:33:58 -07:00
Justine Tunney
fec396037a Get threads working on all platforms
We now have a high-quality clone() implementation for creating
lightweight threads on Linux/Windows/FreeBSD/NetBSD/OpenBSD.
2022-05-13 03:59:09 -07:00
Justine Tunney
ec6e0fa5f4 Wrap sched_getaffinity
Fixes #385
2022-05-12 07:22:57 -07:00
Justine Tunney
335d158ab6 Decrease strace verbosity 2022-05-12 06:58:51 -07:00
Justine Tunney
e7611a8476 Make improvements
- Get threads working on NetBSD
- Get threads working on OpenBSD
- Fix Emacs config for Emacs v28
- Improve --strace logging of sigset_t
- Improve --strace logging of struct stat
- Improve memory safety of DescribeThing functions
- Refactor auto stack allocation into LIBC_RUNTIME
- Introduce shell.com example which works on Windows
- Refactor __strace_thing into DescribeThing functions
- Document the CHECK macros and improve them in NDEBUG mode
- Rewrite MAP_STACK so it uses FreeBSD behavior across platforms
- Deprecate and discourage the use of MAP_GROWSDOWN (it's weird)
2022-05-12 06:45:36 -07:00
Justine Tunney
2aebda7718 Fix some bugs 2022-05-11 02:50:30 -07:00
Justine Tunney
15c59e716f Fix build break in MODE=asan 2022-05-05 11:12:48 -07:00
Justine Tunney
80c4533303 Unbreak MODE=tiny build
This change also fixes a minor issue with pledge() polyfill introduced
the last change setting errno appropriately.

Fixes #396
2022-04-28 10:43:05 -07:00
Justine Tunney
47b3274665 Make improvements
- Add rusage to redbean Lua API
- Add more redbean documentation
- Add pledge() to redbean Lua API
- Polyfill OpenBSD pledge() for Linux
- Increase PATH_MAX limit to 1024 characters
- Untrack sibling processes after fork() on Windows
2022-04-28 09:57:07 -07:00
Justine Tunney
6a145a9262 Make improvements
- Add hierarchical auto-completion to redbean's repl
- Fetch latest localtime() and strftime() from Eggert
- Shave a few milliseconds off redbean start latency
- Fix redbean repl with multi-line statements
- Make the Lua unix module code more elegant
- Harden Lua data structure serialization
2022-04-27 05:39:39 -07:00
Justine Tunney
d57b81aac7 Make improvements
- Add GetCpuCount() API to redbean
- Add unix.gmtime() API to redbean
- Add unix.readlink() API to redbean
- Add unix.localtime() API to redbean
- Perfect the new redbean UNIX module APIs
- Integrate with Linux clock_gettime() vDSO
- Run Lua garbage collector when malloc() fails
- Fix another regression quirk with linenoise repl
- Fix GetProgramExecutableName() for systemwide installs
- Fix a build flake with test/libc/mem/test.mk SRCS list
2022-04-26 16:46:15 -07:00
Justine Tunney
451e3f73d9 Improve redbean
- Improve serialization
- Add Benchmark() API to redbean
- Refactor UNIX API to be assert() friendly
- Make the redbean Lua REPL print data structures
- Fix recent regressions in linenoise reverse search
- Add -i flag so redbean can be a language interpreter
2022-04-25 08:30:14 -07:00
Justine Tunney
2046c0d2ae Make improvements
- Expand redbean UNIX module
- Expand redbean documentation
- Ensure Lua copyright is embedded in binary
- Increase the PATH_MAX limit especially on NT
- Use column major sorting for linenoise completions
- Fix some suboptimalities in redbean's new UNIX API
- Figured out right flags for Multics newline in raw mode
2022-04-24 10:06:05 -07:00
Justine Tunney
2f56ebfe78 Do code cleanup use duff device linenoise i/o 2022-04-22 18:56:52 -07:00
Justine Tunney
a259e43d84 Add execve() test 2022-04-21 19:56:21 -07:00
Justine Tunney
38728cef79 Add poll() embedded webserver demo to redbean 2022-04-21 19:13:19 -07:00
Justine Tunney
1599b818d9 Make more fixes and improvements 2022-04-21 13:44:59 -07:00
Justine Tunney
9d61e23c80 Make more fixes and improvements
This change attempts to fix some report build issues. It also builds
upon development work described in previous changes.
2022-04-21 09:18:45 -07:00
Justine Tunney
9bfa6ec06e Add more documentation to redbean
This change also improves the unix module, adding a reboot() system call
for fun and profit, fixing the execve() api, and a printimage release.
2022-04-21 04:01:42 -07:00
Justine Tunney
87396f43bc Flatten InfoZIP directory and fix build issues 2022-04-20 22:40:33 -07:00
Justine Tunney
ae638c0850 Fix bugs and make improvements
- Get clone() working on FreeBSD
- Increase some Python build quotas
- Add more atomic builtins to chibicc
- Fix ASAN poisoning of alloca() memory
- Make MODE= mandatory link path tinier
- Improve the examples folder a little bit
- Start working on some more resource limits
- Make the linenoise auto-complete UI as good as GNU readline
- Update compile.com, avoiding AVX codegen on non-AVX systems
- Make sure empty path to syscalls like opendir raises ENOENT
- Correctly polyfill ENOENT vs. ENOTDIR on the New Technology
- Port bestline's paredit features to //third_party/linenoise
- Remove workarounds for RHEL 5.0 bugs that were fixed in 5.1
2022-04-20 10:05:34 -07:00
Justine Tunney
5a132f9652 Add seccomp bpf sandboxing to redbean
It's now possible to pass the `-S` or `-SS` flags to sandbox redbean
worker proecsses after they've been forked. The first `-S` flag is
intended to be a permissive builtin policy that limits system calls to
only that which the various parts of redbean serving need. The second
`-SS` flag is intended to be more restrictive, preventing things like
the Lua extensions you download off the web from using the HTTP client
or sockets APIs. In upcoming changes you'll be able to implement your
own Berkeley Packet Filter sandbox programs and load them via Lua.
2022-04-18 08:54:42 -07:00
Justine Tunney
7166679620 Fix bugs and add security features to redbean
- Fix a regression with the previous change that broke redbean
- Add chroot(), resource limit, seccomp, and other stuff to redbean
- Write lots and lots of documentation
- Iron out more system call issues
2022-04-18 00:01:26 -07:00
Justine Tunney
f1dfa4bdfa Add some more builtins to chibicc
https://justine.lol/cosmopolitan/documentation.html should now contain a
lot of functions that had been missing previously due to not having them
2022-04-17 13:07:07 -07:00
Justine Tunney
a6b02ce5a6 Add lua repl interface to redbean
You can now interact with the global web server state on the command
line, which the web server is running. This supports Emacs shortcuts
with history, readline parity, <tab> completions, plus hints. Enjoy!
2022-04-16 20:31:16 -07:00
Justine Tunney
be7c5e1071 Fix debug mode build 2022-04-16 12:25:53 -07:00
Justine Tunney
dc0ea6640e Fix bugs with recent change
This change makes further effort towards improving our poll()
implementation on the New Technology. The stdin worker didn't work out
so well for Python so it's not being used for now. System call tracing
with the --strace flag should now be less noisy now on Windows unless
you modify the strace.internal.h defines to turn on some optional ones
that are most useful for debugging the system call wrappers.
2022-04-16 10:40:23 -07:00
Justine Tunney
933411ba99 Improve synchronization
- Fix bugs in kDos2Errno definition
- malloc() should now be thread safe
- Fix bug in rollup.com header generator
- Fix open(O_APPEND) on the New Technology
- Fix select() on the New Technology and test it
- Work towards refactoring i/o for thread safety
- Socket reads and writes on NT now poll for signals
- Work towards i/o completion ports on the New Technology
- Make read() and write() intermittently check for signals
- Blinkenlights keyboard i/o so much better on NT w/ poll()
- You can now poll() files and sockets at the same time on NT
- Fix bug in appendr() that manifests with dlmalloc footers off
2022-04-15 15:31:55 -07:00
Justine Tunney
fb7e8ef1e6 Add more raw system calls to redbean
We now have execve, setitimer, sigaction, sigsuspend, and sigprocmask.
2022-04-13 14:43:42 -07:00
Justine Tunney
281a0f2730 Implement raw system call for redbean lua code
You can now call functions like fork() from Lua and it'll work across
all supported platforms, including Windows. This gives you a level of
control of the system that Lua traditionally hasn't been able to have
due to its focus on old portable stdio rather modern POSIX APIs. Demo
code has been added to redbean-demo.com to show how it works.

This change also modifies Lua so that integer literals with a leading
zero will be interpreted as octal. That should help avoid shooting in
the foot with POSIX APIs that frequently use octal mode bits.

This change fixes a bug in opendir(".") on New Technology.

Lastly, redbean will now serve crash reports to private network IPs.
This is consistent with other frameworks. However that isn't served
to public IPs unless the -E flag is passed to redbean at startup.
2022-04-13 08:53:24 -07:00
Justine Tunney
f684e348d4 Improve signals and memory protection
- Document sigaction()
- Simplify New Technology fork() code
- Testing and many bug fixes for mprotect()
- Distribute Intel Xed ILD in the amalgamation
- Turn Xed enums into defines to avoid DWARF bloat
- Improve polyfilling of SA_SIGINFO on BSDs and fix bugs
- setpgid(getpid(), getpid()) on Windows will ignore CTRL-C
- Work around issues relating to NT mappings being executable
- Permit automatic executable stack override via `ape_stack_pf`
2022-04-12 22:11:00 -07:00
Justine Tunney
c95c9d9508 Trim down MODE= linkage slightly 2022-04-12 08:05:22 -07:00
Justine Tunney
046c7ebd4a Improve locks and signals
- Introduce fast spinlock API
- Double rand64() perf w/ spinlock
- Improve raise() on New Technology
- Support gettid() across platforms
- Implement SA_NODEFER on New Technology
- Move the lock intrinsics into LIBC_INTRIN
- Make SIGTRAP recoverable on New Technology
- Block SIGCHLD in wait4() on New Technology
- Add threading prototypes for XNU and FreeBSD
- Rewrite abort() fixing its minor bugs on XNU/NT
- Shave down a lot of the content in libc/bits/bits.h
- Let signal handlers modify CPU registers on New Technology
2022-04-12 05:20:17 -07:00
Justine Tunney
f68f1789bd Add XNU magnums for preadv() and writev()
Apple introduced support sometime between 2018 and 2020.
2022-04-12 01:10:26 -07:00
Justine Tunney
4f98ad1054 Improve system call support on NT
- Improve i/o perf on New Technology
- Code cleanup on read() for New Technology
- Fix bad bug with dup() of socket on New Technology
- Clean up some more strace errors on New Technology
2022-04-07 20:30:04 -07:00
Justine Tunney
29bf8b1a30 Make improvements
- Make rand64() thread safe
- Introduce lemur64 lcg prng
- Improve strace on New Technology
- Improve msync() on New Technology
2022-04-07 00:15:35 -07:00
Justine Tunney
f59b5dbd7c Get strace.com working beautifully 2022-04-06 09:43:11 -07:00
Justine Tunney
04d39d47f1 Fix basename() and dirname() 2022-04-06 00:13:44 -07:00
Justine Tunney
1ff1854107 Fix rusage memory reporting on NT
This issue was caused by c23b6ecc3 a few days ago, where we started
polling that information for the first time.
2022-03-31 17:07:30 -07:00
Justine Tunney
072e1d2910 Make signal handling work well across platforms
- Fix sigsuspend() on XNU
- Fix strsignal() on non-Linux
- Add unit tests for strsignal()
- Add unit tests for setitimer()
- Add unit tests for sigsuspend()
- Rewrite setitimer() for New Technology
- Rewrite nanosleep() for New Technology
- Polyfill SIGALRM on the New Technology
- select(0,0,0,0) on NT now calls pause()
- Remove some NTDLL calls that aren't needed
- Polyfill SA_NOCLDWAIT on the New Technology
- Polyfill SA_RESETHAND on the New Technology
- Polyfill sigprocmask() on the New Technology
- Polyfill SIGCHLD+SIG_IGN on the New Technology
- Polyfill SA_RESTART masking on the New Technology
- Deliver console signals from main thread on New Technology
- Document SA_RESTART behavior w/ @sarestartable / @norestart
- System call trace in MODE=dbg now prints inherited FDs and signal mask
2022-03-25 07:28:57 -07:00
Justine Tunney
3b9e66ecba Add Linux ptrace() tutorial 2022-03-24 08:00:36 -07:00
Justine Tunney
98909b1391 Fix more things
- Update a couple unicode data files
- Disable strace during logger calls
- SQLite now uses pread() / pwrite()
- pread() past EOF on NT now returns 0
- Make the NT mmap() and fork() code elegant
- Give NT a big performance boost with memory
- Add many more mmap() tests to prove it works
2022-03-24 00:12:29 -07:00
Justine Tunney
b90fa996b4 Get life.com in MODE=tiny back down to 12kb 2022-03-23 08:09:01 -07:00
Justine Tunney
23b72eb617 Add support for symbol table in .com files
This change fixes minor bugs and adds a feature, which lets us store the
ELF symbol table, inside the ZIP directory. We use the path /zip/.symtab
which can be safely removed using a zip editing tool, to make the binary
smaller after compilation. This supplements the existing method of using
a separate .com.dbg file, which is still supported. The intent is people
don't always know that it's a good idea to download the debug file. It's
not great having someone's first experience be a crash report, that only
has numbers rather than symbols. This will help fix that!
2022-03-23 06:34:46 -07:00
Justine Tunney
393ca4be40 Polyfill SIGCHLD on the New Technology
redbean will now cleanup child processes properly. New accounting
information is available too, such as page faults and memory usage. The
way it works is Cosmopolitan Libc samples the process collection on
entry into read() and poll() to see if SIGCHLD needs to be raised.

This change also fixes an issue with chibicc /tmp cleanup. There was
also a regression in MODE=dbg because STL needed ASAN runtime support.
2022-03-22 21:31:12 -07:00
Justine Tunney
c23b6ecc31 Fix some win32 definitions
You can now use psapi.dll and pdh.dll. Some TODOs for Windows have been
cleared out. We might have a working load average for the platform that
should help GNU Make work well.
2022-03-22 19:54:36 -07:00
Justine Tunney
868af3f950 Import C++ Standard Template Library
You can now use the hardest fastest and most dangerous language there is
with Cosmopolitan. So far about 75% of LLVM libcxx has been added. A few
breaking changes needed to be made to help this go smoothly.

- Rename nothrow to dontthrow
- Rename nodiscard to dontdiscard
- Add some libm functions, e.g. lgamma, nan, etc.
- Change intmax_t from int128 to int64 like everything else
- Introduce %jjd formatting directive for int128_t
- Introduce strtoi128(), strtou128(), etc.
- Rename bsrmax() to bsr128()

Some of the templates that should be working currently are std::vector,
std::string, std::map, std::set, std::deque, etc.
2022-03-22 06:41:54 -07:00
Justine Tunney
5e8ae2d5bc Restart CI for New Technology and UBSAN hunting
Continuous Integration (via runit and runitd) is now re-enabled on win7
and win10. The `make test` command, which runs the tests on all systems
is now the fastest and most stable it's been since the project started.

UBSAN is now enabled in MODE=dbg in addition to ASAN. Many instances of
undefined behavior have been removed. Mostly things like passing a NULL
argument to memcpy(), which works fine with Cosmopolitan Libc, but that
doesn't prevents the compiler from being unhappy. There was an issue w/
GNU make where static analysis claims a sprintf() call can overflow. We
also now have nicer looking crash reports on Windows since uname should
now be supported and msys64 addr2line works reliably.
2022-03-21 04:32:57 -07:00
Justine Tunney
0cb6b6ff4b Get Redbean fork() working on the New Technology
Now that we have understandable system call tracing on Windows, this
change rewrites many of the polyfill internals for that platform, to
help things get closer to tip top shape. Support for complex forking
scenarios had been in a regressed state for quite some time. Now, it
works! Subsequent changes should be able to address the performance.
2022-03-20 08:01:14 -07:00