Commit graph

209 commits

Author SHA1 Message Date
Justine Tunney
369f9740de
Run clang-format on most sources 2023-04-27 05:44:32 -07:00
Justine Tunney
006c44ff5d
Update tests and CPU detection for Blink 2023-01-18 00:56:09 -08:00
Justine Tunney
dd04aeba1c
Increase stack size to 128k and guard size to 16k
This improves our compatibility with Apple M1.
2022-12-18 22:58:29 -08:00
Justine Tunney
3da887c58f
Introduce CPUID detection for Blink 2022-12-17 00:38:07 -08:00
Justine Tunney
f2af97711b
Make improvements
- Improve compatibility with Blink virtual machine
- Add non-POSIX APIs for joining threads and signal masks
- Never ever use anything except 32-bit integers for atomics
- Add some `#undef` statements to workaround `ctags` problems
2022-11-10 21:52:47 -08:00
Justine Tunney
cee6871710
Make detached threads work better
This change adds a double linked list of threads, so that pthread_exit()
will know when it should call exit() from an orphaned child. This change
also improves ftrace and strace logging.
2022-11-09 03:58:57 -08:00
Justine Tunney
bf7843833f
Rename hidden keyword to _Hide 2022-11-08 12:55:28 -08:00
Justine Tunney
3f0bcdc3ef
Improve cancellations, randomness, and time
- Exhaustively document cancellation points
- Rename SIGCANCEL to SIGTHR just like BSDs
- Further improve POSIX thread cancellations
- Ensure asynchronous cancellations work correctly
- Elevate the quality of getrandom() and getentropy()
- Make futexes cancel correctly on OpenBSD 6.x and 7.x
- Add reboot.com and shutdown.com to examples directory
- Remove underscore prefix from awesome timespec_*() APIs
- Create assertions that help verify our cancellation points
- Remove bad timespec APIs (cmp generalizes eq/ne/gt/gte/lt/lte)
2022-11-05 23:45:32 -07:00
Justine Tunney
022536cab6
Make futexes cancellable by pthreads 2022-11-04 18:36:34 -07:00
Justine Tunney
14d036b68d
Add WSL to test fleet
All tests pass now under WSL2. They should pass under WSL1 too, but only
WSL2 is integrated into the test fleet right now. This change also fills
in some gaps in the error numbers.

Fixes #665
2022-11-02 06:49:42 -07:00
Justine Tunney
e522aa3a07
Make more threading improvements
- ASAN memory morgue is now lockless
- Make C11 atomics header more portable
- Rewrote pthread keys support to be lockless
- Simplify Python's unicode table unpacking code
- Make crash report write(2) closer to being atomic
- Make it possible to strace/ftrace a single thread
- ASAN now checks nul-terminated strings fast and properly
- Windows fork() now restores TLS memory of calling thread
2022-11-01 23:28:26 -07:00
Justine Tunney
f7ff77d865
Make fixes and improvements
- Invent iso8601us() for faster timestamps
- Improve --strace descriptions of sigset_t
- Rebuild the Landlock Make bootstrap binary
- Introduce MODE=sysv for non-Windows builds
- Permit OFD fcntl() locks under pledge(flock)
- redbean can now protect your kernel from ddos
- Have vfork() fallback to sys_fork() not fork()
- Change kmalloc() to not die when out of memory
- Improve documentation for some termios functions
- Rewrite putenv() and friends to conform to POSIX
- Fix linenoise + strace verbosity issue on Windows
- Fix regressions in our ability to show backtraces
- Change redbean SetHeader() to no-op if value is nil
- Improve fcntl() so SQLite locks work in non-WAL mode
- Remove some unnecessary work during fork() on Windows
- Create redbean-based SSL reverse proxy for IPv4 TurfWar
- Fix ape/apeinstall.sh warning when using non-bash shells
- Add ProgramTrustedIp(), and IsTrustedIp() APIs to redbean
- Support $PWD, $UID, $GID, and $EUID in command interpreter
- Introduce experimental JTqFpD APE prefix for non-Windows builds
- Invent blackhole daemon for firewalling IP addresses via UNIX named socket
- Add ProgramTokenBucket(), AcquireToken(), and CountTokens() APIs to redbean
2022-10-19 07:19:19 -07:00
Justine Tunney
60cb435cb4
Implement pthread_atfork()
If threads are being used, then fork() will now acquire and release and
runtime locks so that fork() may be safely used from threads. This also
makes vfork() thread safe, because pthread mutexes will do nothing when
the process is a child of vfork(). More torture tests have been written
to confirm this all works like a charm. Additionally:

- Invent hexpcpy() api
- Rename nsync_malloc_() to kmalloc()
- Complete posix named semaphore implementation
- Make pthread_create() asynchronous signal safe
- Add rm, rmdir, and touch to command interpreter builtins
- Invent sigisprecious() and modify sigset functions to use it
- Add unit tests for posix_spawn() attributes and fix its bugs

One unresolved problem is the reclaiming of *NSYNC waiter memory in the
forked child processes, within apps which have threads waiting on locks
2022-10-16 12:25:13 -07:00
Justine Tunney
3d2d04b25e
Make cli commands yoinkable into cosmo shell
This lets our system() and popen() commands function sort of like
BusyBox and ToyBox. By default the Cosmopolitan Shell is lightweight.
But if you use STATIC_YOINK then you can pull the individual commands
you want into the linkage, and they'll be included in a single binary.
For example the demo binary embeds `tr` and `sed` and ends up ~140kb.
2022-10-14 16:37:29 -07:00
Justine Tunney
997ce29ddc
Elevate Windows production worthiness
- SQLite file locking now works on Windows
- SQLite will now use fdatasync() on non-Apple platforms
- Fix Ctrl-C handler on Windows to not crash with TLS
- Signals now work in multithreaded apps on Windows
- fcntl() will now accurately report EINVAL errors
- fcntl() now has excellent --strace logging
- Token bucket replenish now go 100x faster
- *NSYNC cancellations now work on Windows
- Support closefrom() on NetBSD
2022-10-13 13:44:41 -07:00
Justine Tunney
0cee831da3
Revamp posix_spawn() implementation 2022-10-12 05:26:58 -07:00
Justine Tunney
e557058ac8
Improve cosmo's conformance to libc-test
This change addresses various open source compatibility issues, so that
we pass 313/411 of the tests in https://github.com/jart/libc-test where
earlier today we were passing about 30/411 of them, due to header toil.
Please note that Glibc only passes 341/411 so 313 today is pretty good!

- Make the conformance of libc/isystem/ headers nearly perfect
- Import more of the remaining math library routines from Musl
- Fix inconsistencies with type signatures of calls like umask
- Write tests for getpriority/setpriority which work great now
- conform to `struct sockaddr *` on remaining socket functions
- Import a bunch of uninteresting stdlib functions e.g. rand48
- Introduce readdir_r, scandir, pthread_kill, sigsetjmp, etc..

Follow the instructions in our `tool/scripts/cosmocc` toolchain to run
these tests yourself. You use `make CC=cosmocc` on the test repository
2022-10-10 17:52:41 -07:00
Justine Tunney
467a332e38
Introduce sigtimedwait() and sigwaitinfo()
This change also invents sigcountset() and strsignal_r() and improves
the quality of siginfo_t handling.
2022-10-10 07:39:44 -07:00
Justine Tunney
d5910e2673
Fix bugs and make code tinier
- Fixed bug where stdio eof wasn't being sticky
- Fixed bug where fseeko() wasn't clearing eof state
- Removed assert() usage from libc favoring _unassert() / _npassert()
2022-10-09 23:21:34 -07:00
Justine Tunney
4a6fd3d910
Make more improvements to threading support
- fix rare thread exit race condition on openbsd
- pthread_getattr_np() now supplies detached status
- child threads may now pthread_join() the main thread
- introduce sigandset(), sigorset(), and sigisemptyset()
- introduce pthread_cleanup_push() and pthread_cleanup_pop()
2022-10-09 00:08:47 -07:00
tkchia
bc8532688b
Enable CPU exception handling w/ IDT & TSS (#640) 2022-09-29 14:43:08 -07:00
Justine Tunney
1ad2f530f9
Add script.com and whois.com courtesy of FreeBSD 2022-09-13 20:26:13 -07:00
Justine Tunney
6f7d0cb1c3
Pay off more technical debt
This makes breaking changes to add underscores to many non-standard
function names provided by the c library. MODE=tiny is now tinier and
we now use smaller locks that are better for tiny apps in this mode.
Some headers have been renamed to be in the same folder as the build
package, so it'll be easier to know which build dependency is needed.
Certain old misguided interfaces have been removed. Intel intrinsics
headers are now listed in libc/isystem (but not in the amalgamation)
to help further improve open source compatibility. Header complexity
has also been reduced. Lastly, more shell scripts are now available.
2022-09-12 23:36:56 -07:00
Justine Tunney
b69f3d2488
Optimize memory layout 2022-09-12 04:26:52 -07:00
Justine Tunney
b5cb71ab84
Use *NSYNC for POSIX threads locking APIs
Condition variables, barriers, and r/w locks now work very well.
2022-09-11 11:04:50 -07:00
Justine Tunney
cdb2284f0d
Remove stdio lock macros from amalgamation 2022-09-10 12:03:36 -07:00
Justine Tunney
155b378a39
Tidy up the threading implementation
The organization of the source files is now much more rational.
Old experiments that didn't work out are now deleted. Naming of
things like files is now more intuitive.
2022-09-10 02:56:25 -07:00
Justine Tunney
e9272f03fb
Make some minor touchups for nightlies 2022-09-09 19:12:40 -07:00
Justine Tunney
3265324e00
Don't relocate file descriptor memory
This change fixes #496 where ASAN spotted a race condition that could
happen in multithreaded programs, with more than OPEN_MAX descriptors
when using ZipOS or Windows NT, which require tracking open file info
and this change fixes that table so it never relocates, thus allowing
us to continue to enjoy the benefits of avoiding locks while reading.
2022-09-09 16:54:28 -07:00
Justine Tunney
9f963dc597
Clean up some of the threading code 2022-09-08 12:31:56 -07:00
Justine Tunney
0e2b1bfeed
Make garbage collection thread safe
- You can now use _gc(malloc()) in multithreaded programs
- This change fixes a bug where fork() on NT disabled TLS
- Fixed TLS code morphing on XNU/NT, for R8-R15 registers
2022-09-08 02:33:01 -07:00
Justine Tunney
a4522fba8d Fix build 2022-09-03 21:10:37 -07:00
Gavin Hayes
263711965f
Change sigaction_f to match sysv signature (#585) 2022-09-02 05:08:35 -07:00
Justine Tunney
83d41e4588 Clean up some code 2022-08-20 12:32:51 -07:00
Justine Tunney
8e176fb026 Reduce build graph by another 14 percent
That's a 37% reduction total from what it was on 2022-05-23.
2022-08-17 20:01:21 -07:00
Justine Tunney
f0701d2a24 Make improvements
- Polyfill pselect() on Windows
- Add -O NOFILE flag to pledge.com
- Polyfill ppoll() on NetBSD, XNU, and Windows
- Support negative numbers and errno in sizetol()
- Add .RSS, .NOFILE, and .MAXCORE to Landlock Make
- Fix issue with .PLEDGE preventing touching of output files
- Add __watch() function (like ftrace) for logging memory changes
2022-08-15 15:20:36 -07:00
Justine Tunney
ead3fc2b31 Fix Landlock Make so it can read pattern rule vars
It turned out that specifying all SRCS and INCS as dependencies on the
pattern rules for all headers, caused `make` memory usage to skyrocket
from 40mb ot 160mb. This change also reduces the build graph another 4%.
2022-08-13 17:23:05 -07:00
Justine Tunney
0ea0d33a77 Reduce build graph by another eight percent 2022-08-13 13:11:56 -07:00
Justine Tunney
367d06d9e4 Fold LIBC_UNICODE into LIBC_STR 2022-08-13 08:42:32 -07:00
Justine Tunney
17aea99bb3 Fold LIBC_ALG into LIBC_MEM 2022-08-13 08:32:34 -07:00
Justine Tunney
8a0a2c0c36 Fold LIBC_RAND into LIBC_STDIO/TINYMATH/INTRIN 2022-08-11 12:32:00 -07:00
Justine Tunney
05b8f82371 Fold LIBC_BITS into LIBC_INTRIN 2022-08-11 12:13:18 -07:00
Justine Tunney
10fd8bdb70 Unbloat the build
This change resurrects ae5d06dc53
2022-08-11 00:15:29 -07:00
Justine Tunney
7b993d561c Use private keyword on makefile target variables 2022-08-10 18:36:11 -07:00
Justine Tunney
c1d99676c4 Revert "Unbloat build config"
This reverts commit ae5d06dc53.
2022-08-10 12:44:56 -07:00
Justine Tunney
ae5d06dc53 Unbloat build config
- 10.5% reduction of o//depend dependency graph
- 8.8% reduction in latency of make command
- Fix issue with temporary file cleanup

There's a new -w option in compile.com that turns off the recent
Landlock output path workaround for "good commands" which do not
unlink() the output file like GNU tooling does.

Our new GNU Make unveil sandboxing appears to have zero overhead
in the grand scheme of things. Full builds are pretty fast since
the only thing that's actually slowed us down is probably libcxx

    make -j16 MODE=rel
    RL: took 85,732,063µs wall time
    RL: ballooned to 323,612kb in size
    RL: needed 828,560,521µs cpu (11% kernel)
    RL: caused 39,080,670 page faults (99% memcpy)
    RL: 350,073 context switches (72% consensual)
    RL: performed 0 reads and 11,494,960 write i/o operations

pledge() and unveil() no longer consider ENOSYS to be an error.
These functions have also been added to Python's cosmo module.

This change also removes some WIN32 APIs and System Five magnums
which we're not using and it's doubtful anyone else would be too
2022-08-10 04:43:09 -07:00
Justine Tunney
5546559034 Improve pledge() usability and consistency
- We now kill the program on violations like OpenBSD
- We now print a message explaining which promise is needed
- This change also fixes a linkage bug with thread local storage
- Your sigaction() handlers should now be more thread safe

A new `__pledge_mode` global has been introduced to make pledge() more
customizable on Linux. For example:

    __attribute__((__constructor__)) static void init(void) {
      __pledge_mode = SECCOMP_RET_ERRNO | EPERM;
    }

Can be used to restore our old permissive pledge() behavior.
2022-08-07 16:18:33 -07:00
Justine Tunney
6a5717a48f Make more libc improvements
- Make memmem() faster
- Make readdir() thread safe
- Remove 64kb limit from mkdeps.com
- Add old crypt() function from Musl
- Improve new fix-third-party.py tool
- Improve libc/isystem/ headers and fix bugs
2022-08-06 17:18:40 -07:00
Justine Tunney
acdf591833 Avoid long double timestamps in redbean 2022-08-05 19:24:05 -07:00
Justine Tunney
ce5cb8a2f8 Have redbean Lua repl show utf-8 when it's valid
Otherwise EncodeLua() will assume strings are binary and display them
using hex sequences.
2022-07-23 07:22:19 -07:00