Commit graph

76 commits

Author SHA1 Message Date
Justine Tunney
dd04aeba1c
Increase stack size to 128k and guard size to 16k
This improves our compatibility with Apple M1.
2022-12-18 22:58:29 -08:00
Justine Tunney
f2af97711b
Make improvements
- Improve compatibility with Blink virtual machine
- Add non-POSIX APIs for joining threads and signal masks
- Never ever use anything except 32-bit integers for atomics
- Add some `#undef` statements to workaround `ctags` problems
2022-11-10 21:52:47 -08:00
Justine Tunney
cee6871710
Make detached threads work better
This change adds a double linked list of threads, so that pthread_exit()
will know when it should call exit() from an orphaned child. This change
also improves ftrace and strace logging.
2022-11-09 03:58:57 -08:00
Justine Tunney
0d7c265392
Fix backtraces on cancellation points 2022-11-04 20:22:13 -07:00
Justine Tunney
e522aa3a07
Make more threading improvements
- ASAN memory morgue is now lockless
- Make C11 atomics header more portable
- Rewrote pthread keys support to be lockless
- Simplify Python's unicode table unpacking code
- Make crash report write(2) closer to being atomic
- Make it possible to strace/ftrace a single thread
- ASAN now checks nul-terminated strings fast and properly
- Windows fork() now restores TLS memory of calling thread
2022-11-01 23:28:26 -07:00
Justine Tunney
3f49889841
Make important improvements
- Fix preadv() and pwritev() for old distros
- Introduce _npassert() and _unassert() macros
- Prove that file locks work properly on Windows
- Support fcntl(F_DUPFD_CLOEXEC) on more systems
2022-09-14 22:39:08 -07:00
Justine Tunney
6f7d0cb1c3
Pay off more technical debt
This makes breaking changes to add underscores to many non-standard
function names provided by the c library. MODE=tiny is now tinier and
we now use smaller locks that are better for tiny apps in this mode.
Some headers have been renamed to be in the same folder as the build
package, so it'll be easier to know which build dependency is needed.
Certain old misguided interfaces have been removed. Intel intrinsics
headers are now listed in libc/isystem (but not in the amalgamation)
to help further improve open source compatibility. Header complexity
has also been reduced. Lastly, more shell scripts are now available.
2022-09-12 23:36:56 -07:00
Justine Tunney
b5cb71ab84
Use *NSYNC for POSIX threads locking APIs
Condition variables, barriers, and r/w locks now work very well.
2022-09-11 11:04:50 -07:00
Justine Tunney
155b378a39
Tidy up the threading implementation
The organization of the source files is now much more rational.
Old experiments that didn't work out are now deleted. Naming of
things like files is now more intuitive.
2022-09-10 02:56:25 -07:00
Justine Tunney
9f963dc597
Clean up some of the threading code 2022-09-08 12:31:56 -07:00
Justine Tunney
0e2b1bfeed
Make garbage collection thread safe
- You can now use _gc(malloc()) in multithreaded programs
- This change fixes a bug where fork() on NT disabled TLS
- Fixed TLS code morphing on XNU/NT, for R8-R15 registers
2022-09-08 02:33:01 -07:00
Justine Tunney
8dd4ec68d0
Add more missing C / C++ headers 2022-09-04 04:53:52 -07:00
Justine Tunney
8e176fb026 Reduce build graph by another 14 percent
That's a 37% reduction total from what it was on 2022-05-23.
2022-08-17 20:01:21 -07:00
Justine Tunney
17aea99bb3 Fold LIBC_ALG into LIBC_MEM 2022-08-13 08:32:34 -07:00
Justine Tunney
05b8f82371 Fold LIBC_BITS into LIBC_INTRIN 2022-08-11 12:13:18 -07:00
Justine Tunney
10fd8bdb70 Unbloat the build
This change resurrects ae5d06dc53
2022-08-11 00:15:29 -07:00
Justine Tunney
c1d99676c4 Revert "Unbloat build config"
This reverts commit ae5d06dc53.
2022-08-10 12:44:56 -07:00
Justine Tunney
ae5d06dc53 Unbloat build config
- 10.5% reduction of o//depend dependency graph
- 8.8% reduction in latency of make command
- Fix issue with temporary file cleanup

There's a new -w option in compile.com that turns off the recent
Landlock output path workaround for "good commands" which do not
unlink() the output file like GNU tooling does.

Our new GNU Make unveil sandboxing appears to have zero overhead
in the grand scheme of things. Full builds are pretty fast since
the only thing that's actually slowed us down is probably libcxx

    make -j16 MODE=rel
    RL: took 85,732,063µs wall time
    RL: ballooned to 323,612kb in size
    RL: needed 828,560,521µs cpu (11% kernel)
    RL: caused 39,080,670 page faults (99% memcpy)
    RL: 350,073 context switches (72% consensual)
    RL: performed 0 reads and 11,494,960 write i/o operations

pledge() and unveil() no longer consider ENOSYS to be an error.
These functions have also been added to Python's cosmo module.

This change also removes some WIN32 APIs and System Five magnums
which we're not using and it's doubtful anyone else would be too
2022-08-10 04:43:09 -07:00
Justine Tunney
3f015b1e51 Make some minor fixups to bug reporting, etc. 2022-07-11 05:58:24 -07:00
Justine Tunney
5cd0e876bd Scrub double-conversion and add quick wrapper 2022-06-26 05:03:25 -07:00
Justine Tunney
61257d48d4 Make some quick fixes and cleanup 2022-06-26 02:58:36 -07:00
Justine Tunney
fbc053e018 Make fixes and improvements
- Introduce __assert_disable global
- Improve strsignal() thread safety
- Make system call tracing thread safe
- Fix SO_RCVTIMEO / SO_SNDTIMEO on Windows
- Refactor DescribeFoo() functions into one place
- Fix fork() on Windows when TLS and MAP_STACK exist
- Round upwards in setsockopt(SO_RCVTIMEO) on Windows
- Disable futexes on OpenBSD which seem extremely broken
- Implement a better kludge for monotonic time on Windows
2022-06-25 21:09:09 -07:00
Justine Tunney
e466dd0553 Add torture test for zipos file descriptors
This change hardens the code for opening /zip/ files using the system
call interface. Thread safety and signal safety has been improved for
file descriptors in general. We now document fixed addresses that are
needed for low level allocations.
2022-06-15 16:29:49 -07:00
Justine Tunney
8cdec62f5b Apply even more fixups
- Finish cleaning up the stdio unlocked APIs
- Make __cxa_finalize() properly thread safe
- Don't log locks if threads aren't being used
- Add some more mutex guards to places using _mmi
- Specific lock names now appear in the --ftrace logs
- Fix mkdeps.com generating invalid Makefiles sometimes
- Simplify and fix bugs in the test runner infrastructure
- Fix issue where sometimes some functions wouldn't be logged
2022-06-12 11:57:00 -07:00
Justine Tunney
6070a53e89 Make a few fixups 2022-06-12 07:32:14 -07:00
Justine Tunney
c260345e06 Make locks more reliable
This change switches most of the core locks to be re-entrant, in order
to reduce the chance of deadlocking code that does, clever things with
asynchronous signal handlers. This change implements it it in pthreads
so we're one step closer to having a standardized threading primitives
2022-06-11 02:07:20 -07:00
Justine Tunney
425ff5dff0 Refactor some code
- Write tests for cthreads
- Fix bugs in pe2.com tool
- Fix ASAN issue with GetDosEnviron()
- Consolidate the cthread header files
- Some code size optimizations for MODE=
- Attempted to squash a tls linker warning
- Attempted to get futexes working on FreeBSD
2022-05-28 14:09:21 -07:00
Justine Tunney
99e67c348b Reduce Makefile dependencies by 4% 2022-05-23 15:07:01 -07:00
Justine Tunney
e8632cfc30 Revert recent change to ASAN module 2022-05-23 13:30:16 -07:00
Justine Tunney
312ed5c67c Fix some issues and do some code cleanup 2022-05-23 10:15:53 -07:00
Justine Tunney
ec2cb88058 Make fixes and improvements
- Document more compiler flags
- Expose new __print_maps() api
- Better overflow checking in mmap()
- Improve the shell example somewhat
- Fix minor runtime bugs regarding stacks
- Make kill() on fork()+execve()'d children work
- Support CLONE_CHILD_CLEARTID for proper joining
- Fix recent possible deadlock regression with --ftrace
2022-05-19 16:57:49 -07:00
Justine Tunney
9208c83f7a Make some systemic improvements
- add vdso dump utility
- tests now log stack usage
- rename g_ftrace to __ftrace
- make internal spinlocks go faster
- add conformant c11 atomics library
- function tracing now logs stack usage
- make function call tracing thread safe
- add -X unsecure (no ssl) mode to redbean
- munmap() has more consistent behavior now
- pacify fsync() calls on python unit tests
- make --strace flag work better in redbean
- start minimizing and documenting compiler flags
2022-05-18 16:52:36 -07:00
Justine Tunney
ce71677156 Improve threading support further 2022-05-17 04:14:28 -07:00
Justine Tunney
e7611a8476 Make improvements
- Get threads working on NetBSD
- Get threads working on OpenBSD
- Fix Emacs config for Emacs v28
- Improve --strace logging of sigset_t
- Improve --strace logging of struct stat
- Improve memory safety of DescribeThing functions
- Refactor auto stack allocation into LIBC_RUNTIME
- Introduce shell.com example which works on Windows
- Refactor __strace_thing into DescribeThing functions
- Document the CHECK macros and improve them in NDEBUG mode
- Rewrite MAP_STACK so it uses FreeBSD behavior across platforms
- Deprecate and discourage the use of MAP_GROWSDOWN (it's weird)
2022-05-12 06:45:36 -07:00
Justine Tunney
2046c0d2ae Make improvements
- Expand redbean UNIX module
- Expand redbean documentation
- Ensure Lua copyright is embedded in binary
- Increase the PATH_MAX limit especially on NT
- Use column major sorting for linenoise completions
- Fix some suboptimalities in redbean's new UNIX API
- Figured out right flags for Multics newline in raw mode
2022-04-24 10:06:05 -07:00
Justine Tunney
2f56ebfe78 Do code cleanup use duff device linenoise i/o 2022-04-22 18:56:52 -07:00
Justine Tunney
ae638c0850 Fix bugs and make improvements
- Get clone() working on FreeBSD
- Increase some Python build quotas
- Add more atomic builtins to chibicc
- Fix ASAN poisoning of alloca() memory
- Make MODE= mandatory link path tinier
- Improve the examples folder a little bit
- Start working on some more resource limits
- Make the linenoise auto-complete UI as good as GNU readline
- Update compile.com, avoiding AVX codegen on non-AVX systems
- Make sure empty path to syscalls like opendir raises ENOENT
- Correctly polyfill ENOENT vs. ENOTDIR on the New Technology
- Port bestline's paredit features to //third_party/linenoise
- Remove workarounds for RHEL 5.0 bugs that were fixed in 5.1
2022-04-20 10:05:34 -07:00
Justine Tunney
a6b02ce5a6 Add lua repl interface to redbean
You can now interact with the global web server state on the command
line, which the web server is running. This supports Emacs shortcuts
with history, readline parity, <tab> completions, plus hints. Enjoy!
2022-04-16 20:31:16 -07:00
Justine Tunney
933411ba99 Improve synchronization
- Fix bugs in kDos2Errno definition
- malloc() should now be thread safe
- Fix bug in rollup.com header generator
- Fix open(O_APPEND) on the New Technology
- Fix select() on the New Technology and test it
- Work towards refactoring i/o for thread safety
- Socket reads and writes on NT now poll for signals
- Work towards i/o completion ports on the New Technology
- Make read() and write() intermittently check for signals
- Blinkenlights keyboard i/o so much better on NT w/ poll()
- You can now poll() files and sockets at the same time on NT
- Fix bug in appendr() that manifests with dlmalloc footers off
2022-04-15 15:31:55 -07:00
Justine Tunney
f684e348d4 Improve signals and memory protection
- Document sigaction()
- Simplify New Technology fork() code
- Testing and many bug fixes for mprotect()
- Distribute Intel Xed ILD in the amalgamation
- Turn Xed enums into defines to avoid DWARF bloat
- Improve polyfilling of SA_SIGINFO on BSDs and fix bugs
- setpgid(getpid(), getpid()) on Windows will ignore CTRL-C
- Work around issues relating to NT mappings being executable
- Permit automatic executable stack override via `ape_stack_pf`
2022-04-12 22:11:00 -07:00
Justine Tunney
046c7ebd4a Improve locks and signals
- Introduce fast spinlock API
- Double rand64() perf w/ spinlock
- Improve raise() on New Technology
- Support gettid() across platforms
- Implement SA_NODEFER on New Technology
- Move the lock intrinsics into LIBC_INTRIN
- Make SIGTRAP recoverable on New Technology
- Block SIGCHLD in wait4() on New Technology
- Add threading prototypes for XNU and FreeBSD
- Rewrite abort() fixing its minor bugs on XNU/NT
- Shave down a lot of the content in libc/bits/bits.h
- Let signal handlers modify CPU registers on New Technology
2022-04-12 05:20:17 -07:00
Justine Tunney
9b11206ae3 Improve alloca() safety even more 2022-04-01 22:50:15 -07:00
Justine Tunney
38cb6e71ca Improve alloca() memory safety
Now that all the bugs have been wormed out of the ASAN memory module we
can successfully check for underruns on large stack allocations.
2022-04-01 22:44:43 -07:00
Justine Tunney
23b72eb617 Add support for symbol table in .com files
This change fixes minor bugs and adds a feature, which lets us store the
ELF symbol table, inside the ZIP directory. We use the path /zip/.symtab
which can be safely removed using a zip editing tool, to make the binary
smaller after compilation. This supplements the existing method of using
a separate .com.dbg file, which is still supported. The intent is people
don't always know that it's a good idea to download the debug file. It's
not great having someone's first experience be a crash report, that only
has numbers rather than symbols. This will help fix that!
2022-03-23 06:34:46 -07:00
Justine Tunney
393ca4be40 Polyfill SIGCHLD on the New Technology
redbean will now cleanup child processes properly. New accounting
information is available too, such as page faults and memory usage. The
way it works is Cosmopolitan Libc samples the process collection on
entry into read() and poll() to see if SIGCHLD needs to be raised.

This change also fixes an issue with chibicc /tmp cleanup. There was
also a regression in MODE=dbg because STL needed ASAN runtime support.
2022-03-22 21:31:12 -07:00
Justine Tunney
868af3f950 Import C++ Standard Template Library
You can now use the hardest fastest and most dangerous language there is
with Cosmopolitan. So far about 75% of LLVM libcxx has been added. A few
breaking changes needed to be made to help this go smoothly.

- Rename nothrow to dontthrow
- Rename nodiscard to dontdiscard
- Add some libm functions, e.g. lgamma, nan, etc.
- Change intmax_t from int128 to int64 like everything else
- Introduce %jjd formatting directive for int128_t
- Introduce strtoi128(), strtou128(), etc.
- Rename bsrmax() to bsr128()

Some of the templates that should be working currently are std::vector,
std::string, std::map, std::set, std::deque, etc.
2022-03-22 06:41:54 -07:00
Justine Tunney
0cb6b6ff4b Get Redbean fork() working on the New Technology
Now that we have understandable system call tracing on Windows, this
change rewrites many of the polyfill internals for that platform, to
help things get closer to tip top shape. Support for complex forking
scenarios had been in a regressed state for quite some time. Now, it
works! Subsequent changes should be able to address the performance.
2022-03-20 08:01:14 -07:00
Justine Tunney
14e192e5ba Introduce --strace flag for system call tracing
This is similar to the --ftrace (c function call trace) flag, except
it's less noisy since it only logs system calls to stderr. Having this
flag is valuable because (1) system call tracing tells us a lot about
the behavior of complex programs and (2) it's usually very hard to get
system call tracing on various operating systems, e.g. strace, ktrace,
dtruss, truss, nttrace, etc. Especially on Apple platforms where even
with the special boot trick, debuggers still aren't guaranteed to work.

    make -j8 o//examples
    o//examples/hello.com --strace

This is enabled by default in MODE=, MODE=opt, and MODE=dbg. In MODE=dbg
extra information will be printed.

    make -j8 MODE=dbg o/dbg/examples
    o/dbg/examples/hello.com --strace |& less

This change also changes:

- Rename IsText() → _istext()
- Rename IsUtf8() → _isutf8()
- Fix madvise() on Windows NT
- Fix empty string case of inet_ntop()
- vfork() wrapper now saves and restores errno
- Update xsigaction() to yoink syscall support
2022-03-18 18:07:28 -07:00
Justine Tunney
af645fcbec Make exciting improvements
- Add Lua backtraces to redbean!
- Wipe serving keys after redbean forks
- Audit redbean to remove free via exit
- Log SSL client ciphersuite preferences
- Increase ASAN malloc() backtrace depth
- Make GetSslRoots() behave as a singleton
- Move leaks.c from LIBC_TESTLIB to LIBC_LOG
- Add undocumented %n to printf() for newlines
- Fix redbean memory leak reindexing inode change
- Fix redbean memory leak with Fetch() DNS object
- Restore original environ after __cxa_finalize()
- Make backtrace always work after __cxa_finalize()
- Introduce COUNTEXPR() diagnostic / benchmark tool
- Fix a few more instances of errno being clobbered
- Consolidate the ANSI color disabling internal APIs
2022-03-18 03:02:00 -07:00
Justine Tunney
46a3b88594 Have ASAN errors show origin of memory 2022-03-17 00:53:45 -07:00