Commit graph

105 commits

Author SHA1 Message Date
Justine Tunney
425ff5dff0 Refactor some code
- Write tests for cthreads
- Fix bugs in pe2.com tool
- Fix ASAN issue with GetDosEnviron()
- Consolidate the cthread header files
- Some code size optimizations for MODE=
- Attempted to squash a tls linker warning
- Attempted to get futexes working on FreeBSD
2022-05-28 14:09:21 -07:00
Justine Tunney
de5de19004 Make improvements
- Document redbean's argon2 module
- Fix regressions in cthreads library
- Make testlib work better with threads
- Give the cthreads library lots of love
- Remove some of the stdio assembly code
- Implement getloadavg() across platforms
- Code size optimizations for errnos, etc.
- Only check for signals in main thread on Windows
- Make errnos for dup2 / dup3 consistent with posix

This change also fixes a bug in the argon2 module, where the NUL
terminator was being included in the hash encoded ascii string. This
shouldn't require any database migrations to folks who found this module
and productionized it, since the argon2 library treats it as a c string.
2022-05-28 00:28:09 -07:00
Justine Tunney
cb67223051 Add malloc logging tool
STATIC_YOINK("enable_memory_log");
2022-05-26 23:19:22 -07:00
Justine Tunney
10b97ca630 Add test for sendfile() and reduce branches 2022-05-25 22:29:10 -07:00
Justine Tunney
99e67c348b Reduce Makefile dependencies by 4% 2022-05-23 15:07:01 -07:00
Justine Tunney
e8632cfc30 Revert recent change to ASAN module 2022-05-23 13:30:16 -07:00
Justine Tunney
312ed5c67c Fix some issues and do some code cleanup 2022-05-23 10:15:53 -07:00
Justine Tunney
1f229e4efc Use re-entrant locks on stdio 2022-05-22 08:28:33 -07:00
Justine Tunney
4e9662cbc7 Write tests for new APE loader and fix bugs
- Add FreeBSD-specific mmap() flags
- Reduce size of the APE loader from 8kb to 4kb
- Work towards fixing the Makefile build on WSL
- Automate testing of APE no-modify-self behaviors
- Make the ape.S shell script code cleaner and tinier
- Improve the APE sanity check to test behavior better
- Fixed issue with ShowCrashReports() sigaltstack() on BSDs
- Delete symbols for S_MODE magnums which wasted compile time

If you checked out yesterday's APE commit, please run:

    rm -f /usr/bin/ape o/tmp/ape /tmp/ape "${TMPDIR:-/tmp}/ape"

Because this change fixes certain aspects of the new ABI. We don't have
automated migrations for APE loader versions yet. Thanks! You can also
download prebuilt binaries here:

- https://justine.lol/ape.elf    (Linux/FreeBSD/NetBSD/OpenBSD)
- https://justine.lol/ape.macho  (Apple)

Install the appropriate one as `/usr/bin/ape`.
2022-05-22 05:45:38 -07:00
Justine Tunney
7838edae88 Fix many thread and file descriptor issues on Windows
The greenbean web server now works nearly perfectly on Windows with over
1000 threads. But some synchronization issues still remain which prevent
us from going over nine thousand.
2022-05-20 20:57:19 -07:00
Justine Tunney
c8a2f04058 Reduce ftrace overhead to 280ns 2022-05-20 04:46:42 -07:00
Justine Tunney
ec2cb88058 Make fixes and improvements
- Document more compiler flags
- Expose new __print_maps() api
- Better overflow checking in mmap()
- Improve the shell example somewhat
- Fix minor runtime bugs regarding stacks
- Make kill() on fork()+execve()'d children work
- Support CLONE_CHILD_CLEARTID for proper joining
- Fix recent possible deadlock regression with --ftrace
2022-05-19 16:57:49 -07:00
Justine Tunney
9208c83f7a Make some systemic improvements
- add vdso dump utility
- tests now log stack usage
- rename g_ftrace to __ftrace
- make internal spinlocks go faster
- add conformant c11 atomics library
- function tracing now logs stack usage
- make function call tracing thread safe
- add -X unsecure (no ssl) mode to redbean
- munmap() has more consistent behavior now
- pacify fsync() calls on python unit tests
- make --strace flag work better in redbean
- start minimizing and documenting compiler flags
2022-05-18 16:52:36 -07:00
Justine Tunney
2743f3d012 Clean up the clone() code a bit 2022-05-17 07:40:00 -07:00
Justine Tunney
ce71677156 Improve threading support further 2022-05-17 04:14:28 -07:00
Justine Tunney
55de4ca6b5 Support thread local storage 2022-05-16 13:20:08 -07:00
Justine Tunney
80b211e314 Add raw memory visualization tool to redbean
This change introduces a `-W /dev/pts/1` flag to redbean. What it does
is use the mincore() system call to create a dual-screen terminal
display that lets you troubleshoot the virtual address space. This is
useful since page faults are an important thing to consider when using a
forking web server. Now we have a colorful visualization of which pages
are going to fault and which ones are resident in memory.

The memory monitor, if enabled, spawns as a thread that just outputs
ANSI codes to the second terminal in a loop. In order to make this
happen using the new clone() polyfill, stdio is now thread safe.

This change also introduces some new demo pages to redbean. It also
polishes the demos we already have, to look a bit nicer and more
presentable for the upcoming release, with better explanations too.
2022-05-14 04:33:58 -07:00
Justine Tunney
fec396037a Get threads working on all platforms
We now have a high-quality clone() implementation for creating
lightweight threads on Linux/Windows/FreeBSD/NetBSD/OpenBSD.
2022-05-13 03:59:09 -07:00
Justine Tunney
e7611a8476 Make improvements
- Get threads working on NetBSD
- Get threads working on OpenBSD
- Fix Emacs config for Emacs v28
- Improve --strace logging of sigset_t
- Improve --strace logging of struct stat
- Improve memory safety of DescribeThing functions
- Refactor auto stack allocation into LIBC_RUNTIME
- Introduce shell.com example which works on Windows
- Refactor __strace_thing into DescribeThing functions
- Document the CHECK macros and improve them in NDEBUG mode
- Rewrite MAP_STACK so it uses FreeBSD behavior across platforms
- Deprecate and discourage the use of MAP_GROWSDOWN (it's weird)
2022-05-12 06:45:36 -07:00
Justine Tunney
93906f8f28 Comment out psrldq_fuzz test
There's a mysterious error on Intel's low-power CPUs. It appears to be
triggered by the C code. Here's the vectors in CP-437:

    n=1
    u"x2ΩΦ▄█b∙¥¼U╞╣“┘$"
    u"2ΩΦ▄█b∙¥¼U╞╣“┘$ "
    u"2ΩΦ▄█b¥¼¼U╞╣“┘$ "
2022-05-04 21:43:38 -07:00
Justine Tunney
47b3274665 Make improvements
- Add rusage to redbean Lua API
- Add more redbean documentation
- Add pledge() to redbean Lua API
- Polyfill OpenBSD pledge() for Linux
- Increase PATH_MAX limit to 1024 characters
- Untrack sibling processes after fork() on Windows
2022-04-28 09:57:07 -07:00
Justine Tunney
451e3f73d9 Improve redbean
- Improve serialization
- Add Benchmark() API to redbean
- Refactor UNIX API to be assert() friendly
- Make the redbean Lua REPL print data structures
- Fix recent regressions in linenoise reverse search
- Add -i flag so redbean can be a language interpreter
2022-04-25 08:30:14 -07:00
Justine Tunney
2046c0d2ae Make improvements
- Expand redbean UNIX module
- Expand redbean documentation
- Ensure Lua copyright is embedded in binary
- Increase the PATH_MAX limit especially on NT
- Use column major sorting for linenoise completions
- Fix some suboptimalities in redbean's new UNIX API
- Figured out right flags for Multics newline in raw mode
2022-04-24 10:06:05 -07:00
Justine Tunney
2f56ebfe78 Do code cleanup use duff device linenoise i/o 2022-04-22 18:56:52 -07:00
Justine Tunney
552525cbdd Fix a few reported issues 2022-04-21 22:07:21 -07:00
Justine Tunney
a259e43d84 Add execve() test 2022-04-21 19:56:21 -07:00
Justine Tunney
ae638c0850 Fix bugs and make improvements
- Get clone() working on FreeBSD
- Increase some Python build quotas
- Add more atomic builtins to chibicc
- Fix ASAN poisoning of alloca() memory
- Make MODE= mandatory link path tinier
- Improve the examples folder a little bit
- Start working on some more resource limits
- Make the linenoise auto-complete UI as good as GNU readline
- Update compile.com, avoiding AVX codegen on non-AVX systems
- Make sure empty path to syscalls like opendir raises ENOENT
- Correctly polyfill ENOENT vs. ENOTDIR on the New Technology
- Port bestline's paredit features to //third_party/linenoise
- Remove workarounds for RHEL 5.0 bugs that were fixed in 5.1
2022-04-20 10:05:34 -07:00
Justine Tunney
5a132f9652 Add seccomp bpf sandboxing to redbean
It's now possible to pass the `-S` or `-SS` flags to sandbox redbean
worker proecsses after they've been forked. The first `-S` flag is
intended to be a permissive builtin policy that limits system calls to
only that which the various parts of redbean serving need. The second
`-SS` flag is intended to be more restrictive, preventing things like
the Lua extensions you download off the web from using the HTTP client
or sockets APIs. In upcoming changes you'll be able to implement your
own Berkeley Packet Filter sandbox programs and load them via Lua.
2022-04-18 08:54:42 -07:00
Justine Tunney
7166679620 Fix bugs and add security features to redbean
- Fix a regression with the previous change that broke redbean
- Add chroot(), resource limit, seccomp, and other stuff to redbean
- Write lots and lots of documentation
- Iron out more system call issues
2022-04-18 00:01:26 -07:00
Justine Tunney
f1dfa4bdfa Add some more builtins to chibicc
https://justine.lol/cosmopolitan/documentation.html should now contain a
lot of functions that had been missing previously due to not having them
2022-04-17 13:07:07 -07:00
Justine Tunney
ab38f0823d Make small fixes 2022-04-17 10:40:32 -07:00
Justine Tunney
a6b02ce5a6 Add lua repl interface to redbean
You can now interact with the global web server state on the command
line, which the web server is running. This supports Emacs shortcuts
with history, readline parity, <tab> completions, plus hints. Enjoy!
2022-04-16 20:31:16 -07:00
Justine Tunney
be7c5e1071 Fix debug mode build 2022-04-16 12:25:53 -07:00
Justine Tunney
dc0ea6640e Fix bugs with recent change
This change makes further effort towards improving our poll()
implementation on the New Technology. The stdin worker didn't work out
so well for Python so it's not being used for now. System call tracing
with the --strace flag should now be less noisy now on Windows unless
you modify the strace.internal.h defines to turn on some optional ones
that are most useful for debugging the system call wrappers.
2022-04-16 10:40:23 -07:00
Justine Tunney
933411ba99 Improve synchronization
- Fix bugs in kDos2Errno definition
- malloc() should now be thread safe
- Fix bug in rollup.com header generator
- Fix open(O_APPEND) on the New Technology
- Fix select() on the New Technology and test it
- Work towards refactoring i/o for thread safety
- Socket reads and writes on NT now poll for signals
- Work towards i/o completion ports on the New Technology
- Make read() and write() intermittently check for signals
- Blinkenlights keyboard i/o so much better on NT w/ poll()
- You can now poll() files and sockets at the same time on NT
- Fix bug in appendr() that manifests with dlmalloc footers off
2022-04-15 15:31:55 -07:00
Justine Tunney
281a0f2730 Implement raw system call for redbean lua code
You can now call functions like fork() from Lua and it'll work across
all supported platforms, including Windows. This gives you a level of
control of the system that Lua traditionally hasn't been able to have
due to its focus on old portable stdio rather modern POSIX APIs. Demo
code has been added to redbean-demo.com to show how it works.

This change also modifies Lua so that integer literals with a leading
zero will be interpreted as octal. That should help avoid shooting in
the foot with POSIX APIs that frequently use octal mode bits.

This change fixes a bug in opendir(".") on New Technology.

Lastly, redbean will now serve crash reports to private network IPs.
This is consistent with other frameworks. However that isn't served
to public IPs unless the -E flag is passed to redbean at startup.
2022-04-13 08:53:24 -07:00
Justine Tunney
f684e348d4 Improve signals and memory protection
- Document sigaction()
- Simplify New Technology fork() code
- Testing and many bug fixes for mprotect()
- Distribute Intel Xed ILD in the amalgamation
- Turn Xed enums into defines to avoid DWARF bloat
- Improve polyfilling of SA_SIGINFO on BSDs and fix bugs
- setpgid(getpid(), getpid()) on Windows will ignore CTRL-C
- Work around issues relating to NT mappings being executable
- Permit automatic executable stack override via `ape_stack_pf`
2022-04-12 22:11:00 -07:00
Justine Tunney
c95c9d9508 Trim down MODE= linkage slightly 2022-04-12 08:05:22 -07:00
Justine Tunney
046c7ebd4a Improve locks and signals
- Introduce fast spinlock API
- Double rand64() perf w/ spinlock
- Improve raise() on New Technology
- Support gettid() across platforms
- Implement SA_NODEFER on New Technology
- Move the lock intrinsics into LIBC_INTRIN
- Make SIGTRAP recoverable on New Technology
- Block SIGCHLD in wait4() on New Technology
- Add threading prototypes for XNU and FreeBSD
- Rewrite abort() fixing its minor bugs on XNU/NT
- Shave down a lot of the content in libc/bits/bits.h
- Let signal handlers modify CPU registers on New Technology
2022-04-12 05:20:17 -07:00
Justine Tunney
a157940ba6 Delete most undocumented New Technology APIs
This change removes LIBC_KERNELBASE which is legacy code from the
initial import which was generated off a script that resolved the
delegated references, on Windows 10. All the important stuff here
should have long since been filed under kernel32.dll for windows7

Many FooA functions that were never assigned an arity are removed
because we almost never use the ASCII versions of WIN32 functions
therefore it's not worth having them slowing down the build. Some
other functions that overlap uncomfortably with libc are gone too

If something you need was removed, file an issue we'll restore it
2022-04-11 23:41:12 -07:00
Justine Tunney
4f98ad1054 Improve system call support on NT
- Improve i/o perf on New Technology
- Code cleanup on read() for New Technology
- Fix bad bug with dup() of socket on New Technology
- Clean up some more strace errors on New Technology
2022-04-07 20:30:04 -07:00
Justine Tunney
29bf8b1a30 Make improvements
- Make rand64() thread safe
- Introduce lemur64 lcg prng
- Improve strace on New Technology
- Improve msync() on New Technology
2022-04-07 00:15:35 -07:00
Justine Tunney
f59b5dbd7c Get strace.com working beautifully 2022-04-06 09:43:11 -07:00
Justine Tunney
04d39d47f1 Fix basename() and dirname() 2022-04-06 00:13:44 -07:00
Justine Tunney
9b11206ae3 Improve alloca() safety even more 2022-04-01 22:50:15 -07:00
Justine Tunney
38cb6e71ca Improve alloca() memory safety
Now that all the bugs have been wormed out of the ASAN memory module we
can successfully check for underruns on large stack allocations.
2022-04-01 22:44:43 -07:00
Justine Tunney
072e1d2910 Make signal handling work well across platforms
- Fix sigsuspend() on XNU
- Fix strsignal() on non-Linux
- Add unit tests for strsignal()
- Add unit tests for setitimer()
- Add unit tests for sigsuspend()
- Rewrite setitimer() for New Technology
- Rewrite nanosleep() for New Technology
- Polyfill SIGALRM on the New Technology
- select(0,0,0,0) on NT now calls pause()
- Remove some NTDLL calls that aren't needed
- Polyfill SA_NOCLDWAIT on the New Technology
- Polyfill SA_RESETHAND on the New Technology
- Polyfill sigprocmask() on the New Technology
- Polyfill SIGCHLD+SIG_IGN on the New Technology
- Polyfill SA_RESTART masking on the New Technology
- Deliver console signals from main thread on New Technology
- Document SA_RESTART behavior w/ @sarestartable / @norestart
- System call trace in MODE=dbg now prints inherited FDs and signal mask
2022-03-25 07:28:57 -07:00
Justine Tunney
3b9e66ecba Add Linux ptrace() tutorial 2022-03-24 08:00:36 -07:00
Justine Tunney
98909b1391 Fix more things
- Update a couple unicode data files
- Disable strace during logger calls
- SQLite now uses pread() / pwrite()
- pread() past EOF on NT now returns 0
- Make the NT mmap() and fork() code elegant
- Give NT a big performance boost with memory
- Add many more mmap() tests to prove it works
2022-03-24 00:12:29 -07:00
Justine Tunney
b90fa996b4 Get life.com in MODE=tiny back down to 12kb 2022-03-23 08:09:01 -07:00