Compare commits

...

83 commits

Author SHA1 Message Date
Hugues Morisset
f1e83d5240
Add IPv6 support to getifaddrs() on Linux (#1415)
Some checks failed
build / matrix_on_mode () (push) Has been cancelled
build / matrix_on_mode (optlinux) (push) Has been cancelled
build / matrix_on_mode (rel) (push) Has been cancelled
build / matrix_on_mode (tiny) (push) Has been cancelled
build / matrix_on_mode (tinylinux) (push) Has been cancelled
2025-05-21 01:20:22 -07:00
Steven Dee (Jōshin)
2fe8338f92
Better mtimes for github workflow build cache (#1421)
Saves and restores mtimes to a file, also covering the `o/` directory to
hopefully preserve make dependency information better.
2025-05-20 22:17:55 -07:00
ShalokShalom
4ca513cba2
Add C++ to README (#1407)
Some checks failed
build / matrix_on_mode () (push) Has been cancelled
build / matrix_on_mode (optlinux) (push) Has been cancelled
build / matrix_on_mode (rel) (push) Has been cancelled
build / matrix_on_mode (tiny) (push) Has been cancelled
build / matrix_on_mode (tinylinux) (push) Has been cancelled
2025-04-25 15:47:50 -07:00
Steven Dee (Jōshin)
455910e8f2
Make more shared_ptr fixes (#1401)
Some checks failed
build / matrix_on_mode () (push) Has been cancelled
build / matrix_on_mode (optlinux) (push) Has been cancelled
build / matrix_on_mode (rel) (push) Has been cancelled
build / matrix_on_mode (tiny) (push) Has been cancelled
build / matrix_on_mode (tinylinux) (push) Has been cancelled
* Make refcount reads explicitly atomic
* Consistently put `const` in the same place
* Write the general `operator=` on `weak_ptr`
2025-04-21 05:36:50 -07:00
Steven Dee (Jōshin)
9c68bc19b5
Cache .cosmocc and o for github workflows (#1400)
Some checks failed
build / matrix_on_mode () (push) Has been cancelled
build / matrix_on_mode (optlinux) (push) Has been cancelled
build / matrix_on_mode (rel) (push) Has been cancelled
build / matrix_on_mode (tiny) (push) Has been cancelled
build / matrix_on_mode (tinylinux) (push) Has been cancelled
Uses GitHub’s actions/cache@v4 to store the cosmocc distribution and the
output directory between runs of the build workflow, with the version of
cosmocc as the cache key.

Upgrades to actions/checkout@v4.
2025-04-17 15:55:27 -07:00
Steven Dee (Jōshin)
66d1050af6
Correctly implement weak_ptr assignment/copy/moves (#1399) 2025-04-17 14:01:20 -07:00
Justine Tunney
fbc4fcbb71
Get GDB working
Some checks failed
build / matrix_on_mode () (push) Has been cancelled
build / matrix_on_mode (optlinux) (push) Has been cancelled
build / matrix_on_mode (rel) (push) Has been cancelled
build / matrix_on_mode (tiny) (push) Has been cancelled
build / matrix_on_mode (tinylinux) (push) Has been cancelled
You can now say `gdb hello.com.dbg` and it'll work perfectly.
2025-03-30 15:25:55 -07:00
Steven Dee (Jōshin)
afc986f741
Fix shared_ptr<T>::owner_before (#1390)
Some checks failed
build / matrix_on_mode () (push) Has been cancelled
build / matrix_on_mode (optlinux) (push) Has been cancelled
build / matrix_on_mode (rel) (push) Has been cancelled
build / matrix_on_mode (tiny) (push) Has been cancelled
build / matrix_on_mode (tinylinux) (push) Has been cancelled
`!(a < b)` is not the same as `b < a`.

I think I originally wrote it this way to avoid making weak_ptr a friend
of shared_ptr, but weak_ptr already is a friend.
2025-03-25 01:49:34 -04:00
Derek
5eb7cd6643
Add support for getcpu() system call to pledge() (#1387)
Some checks failed
build / matrix_on_mode () (push) Has been cancelled
build / matrix_on_mode (optlinux) (push) Has been cancelled
build / matrix_on_mode (rel) (push) Has been cancelled
build / matrix_on_mode (tiny) (push) Has been cancelled
build / matrix_on_mode (tinylinux) (push) Has been cancelled
This fixes redbean Lua tests which were failing with SIGSYS on Linux.
2025-03-21 16:08:25 -07:00
Brett Jia
a8ed4fdd09
Add NetBSD evbarm and fix segfault (#1384)
Some checks failed
build / matrix_on_mode () (push) Has been cancelled
build / matrix_on_mode (optlinux) (push) Has been cancelled
build / matrix_on_mode (rel) (push) Has been cancelled
build / matrix_on_mode (tiny) (push) Has been cancelled
build / matrix_on_mode (tinylinux) (push) Has been cancelled
This change fixes a segmentation fault when comparing loaders that don't
have a target kernel set. Additionally, adds evbarm, which is the output
of uname -m on NetBSD on aarch64.
2025-03-12 17:37:46 -07:00
Brett Jia
7b69652854
Add -k OSNAME flag to apelink (#1383)
Let's say you pass the `-M blink-mips.elf` flag to apelink, so that your
ape binary will bundle a compressed build of blink, and the shell script
will extract that binary and launch your program under it, if running on
a MIPS system. However, for any given microprocessor architecture, we'll
need a separate loader for each operating system. The issue is ELF OSABI
isn't very useful. As an example, SerenityOS and Linux both have SYSV in
the OSABI field. So to tell their binaries apart we'd have to delve into
various other conventions, like special sections and PT_NOTE structures.

To make things simple this change introduces the `-k OS` flag to apelink
which generate shell script content that ensures `OS` matches `uname -s`
before attempting to execute a loader. For example, you could say:

    apelink -k Linux -M blink-linux-arm.elf -M blink-linux-mips.elf \
            -k Darwin -M blink-darwin-ppc.elf \
            ...

To introduce support for old 32-bit architectures on multiple OSes, when
building your cosmo binary.
2025-03-12 13:26:51 -07:00
Leal G.
b235492e71
Add usertrust certificate (#1382)
Some checks are pending
build / matrix_on_mode () (push) Waiting to run
build / matrix_on_mode (optlinux) (push) Waiting to run
build / matrix_on_mode (rel) (push) Waiting to run
build / matrix_on_mode (tiny) (push) Waiting to run
build / matrix_on_mode (tinylinux) (push) Waiting to run
Bundle USERTrust CA certificates to /usr/share/ssl/root for TLS verifies
2025-03-11 17:59:34 -07:00
Brett Jia
fc81fd8d16
Support additional architectures in apelink (#1381)
Some checks failed
build / matrix_on_mode () (push) Has been cancelled
build / matrix_on_mode (optlinux) (push) Has been cancelled
build / matrix_on_mode (rel) (push) Has been cancelled
build / matrix_on_mode (tiny) (push) Has been cancelled
build / matrix_on_mode (tinylinux) (push) Has been cancelled
This updates apelink to support machine architectures not in the source
program input list by adding additional loaders, extracting the correct
one that matches the host uname machine. With this change, blink can be
supplied as the additional loader to run the program in x86_64 VMs. The
change has been verified against blink 1.0, powerpc64le and mips64el in
Docker using QEMU.
2025-03-06 10:26:31 -08:00
Gautham
38930de8e0
Make tool for replacing ELF strings (#1344)
Some checks failed
build / matrix_on_mode () (push) Has been cancelled
build / matrix_on_mode (optlinux) (push) Has been cancelled
build / matrix_on_mode (rel) (push) Has been cancelled
build / matrix_on_mode (tiny) (push) Has been cancelled
build / matrix_on_mode (tinylinux) (push) Has been cancelled
2025-02-08 21:17:42 -08:00
Brett Jia
0e557d041d
Check downloaded gcc/clang checksums (#1367)
Check sha256 checksums of the downloaded gcc and clang toolchains. It'll
allow us to extend trust to external toolchains if building from source.
2025-02-08 17:46:09 -08:00
Brett Jia
1d676b36e6
Make cosmoranlib executable (#1366)
Fixes #1325
2025-02-08 17:38:00 -08:00
Brett Jia
10a92cee94
Support building cosmocc on MacOS (#1365)
Some checks are pending
build / matrix_on_mode () (push) Waiting to run
build / matrix_on_mode (optlinux) (push) Waiting to run
build / matrix_on_mode (rel) (push) Waiting to run
build / matrix_on_mode (tiny) (push) Waiting to run
build / matrix_on_mode (tinylinux) (push) Waiting to run
This updates the cosmocc toolchain packaging script to work on MacOS. It
has been tested on GitHub Actions macos-13 (x86_64) and macos-14 (arm64)
runners, and is verified to still work on Ubuntu (GitHub Actions runners
ubuntu-24.04 and ubuntu-24.04-arm). It'll help bring cosmocc to MacPorts
by running the packaging script. We favor `gmake` rather than the `make`
command because it distinguishes GNU Make from BSD Make, and Xcode Make.
Additionally, APE loader from the bootstrapper toolchain is used instead
of a system APE, which may not be available.
2025-02-08 12:45:45 -08:00
A2va
42a9ed0131
Adds some NT functions (#1358) 2025-02-08 08:08:08 -08:00
Björn Buckwalter
12cb0669fb
Clarify unix.mapshared versus file locks (#1355)
Some checks are pending
build / matrix_on_mode () (push) Waiting to run
build / matrix_on_mode (optlinux) (push) Waiting to run
build / matrix_on_mode (rel) (push) Waiting to run
build / matrix_on_mode (tiny) (push) Waiting to run
build / matrix_on_mode (tinylinux) (push) Waiting to run
2025-02-08 00:48:38 -08:00
rufeooo
7f6a7d6fff
Fix sigaction example code (#1363)
Some checks are pending
build / matrix_on_mode () (push) Waiting to run
build / matrix_on_mode (optlinux) (push) Waiting to run
build / matrix_on_mode (rel) (push) Waiting to run
build / matrix_on_mode (tiny) (push) Waiting to run
build / matrix_on_mode (tinylinux) (push) Waiting to run
2025-02-07 11:42:47 -08:00
Steven Dee (Jōshin)
9f6bf6ea71
tool/zsh/mkofs: doas 2025-01-13 16:48:55 -08:00
Steven Dee (Jōshin)
102edf4ea2
tool/zsh/mmake: style 2025-01-05 20:53:53 -08:00
Steven Dee (Jōshin)
21968acf99
Standard make path (#1353)
Modifies download-cosmocc.sh to maintain a .cosmocc/current symlink that
always points to the most recently downloaded version of cosmocc. We can
use this to point at a canonical make for a bootstrapped repository. For
first-time builds, we suggest: https://cosmo.zip/pub/cosmos/bin/make and
have updated the docs in a few places to mention this.

Fixes the other part of #1346.
2025-01-05 20:47:34 -08:00
Justine Tunney
98861b23fc
Make some style fixes to prng code 2025-01-05 20:18:05 -08:00
Steven Dee (Jōshin)
dab6d7a345
Resolve multiple definition of __sig (fixes #1346) (#1352) 2025-01-05 19:54:49 -08:00
Justine Tunney
90119c422c
Fix 404 url
Closes #1347
2025-01-05 17:04:37 -08:00
Justine Tunney
5907304049
Release Cosmopolitan v4.0.2 2025-01-05 14:05:49 -08:00
Justine Tunney
035b0e2a62
Attempt to fix MODE=dbg Windows execve() flake 2025-01-05 14:05:49 -08:00
Justine Tunney
7b67b20dae
Fix Windows MODE=tiny breakage 2025-01-05 14:05:49 -08:00
Himanshu Pal
f0b0f926bf
Enable sqlite3 serialization in redbean (#1349)
This fixes a failing demo page, that requires us to enable serialization
in the lsqlite3 library that's used by the redbean server.
2025-01-05 13:59:10 -08:00
Justine Tunney
29eb7e67bb
Fix fork() regression on Windows
Recent optimizations to fork() introduced a regression, that could cause
the subprocess to fail unexpectedly, when TlsAlloc() returns a different
index. This is because we were burning the indexes into the displacement
of x86 opcodes. So when fork() happened and the executable memory copied
it would use the old index. Right now the way this is being solved is to
not copy the executable on fork() and then re-apply code changes. If you
need to be able to preserve self-modified code on fork, reach out and we
can implement a better solution for you. This gets us unblocked quickly.
2025-01-05 09:25:23 -08:00
Justine Tunney
f71f61cd40
Add some temporary logging statements 2025-01-04 23:37:32 -08:00
Justine Tunney
53c6edfd18
Make correction to last change 2025-01-04 21:38:47 -08:00
Justine Tunney
42a3bb729a
Make execve() linger when it can't spoof parent
It's now possible to use execve() when the parent process isn't built by
cosmo. In such cases, the current process will kill all threads and then
linger around, waiting for the newly created process to die, and then we
propagate its exit code to the parent. This should help bazel and others

Allocating private anonymous memory is now 5x faster on Windows. This is
thanks to VirtualAlloc() which is faster than the file mapping APIs. The
fork() function also now goes 30% faster, since we are able to avoid the
VirtualProtect() calls on mappings in most cases now.

Fixes #1253
2025-01-04 21:13:37 -08:00
Justine Tunney
c97a858470
Remove missing definitions 2025-01-04 00:20:45 -08:00
Justine Tunney
4acd12a514
Release Cosmopolitan v4.0.1 2025-01-03 19:51:34 -08:00
Justine Tunney
b734eec836
Test restricting tests to single cpu 2025-01-03 19:51:09 -08:00
Justine Tunney
fe01642a20
Add missing lock to fork() on Windows 2025-01-03 19:01:58 -08:00
Justine Tunney
e939659b70
Fix ordering of pthread_create(pthread_t *thread)
This change fixes a bug where signal_latency_async_test would flake less
than 1/1000 of the time. What was happening was pthread_kill(sender_thr)
would return EFAULT. This was because pthread_create() was not returning
the thread object pointer until after clone() had been called. So it was
actually possible for the main thread to stall after calling clone() and
during that time the receiver would launch and receive a signal from the
sender thread, and then fail when it tried to send a pong. I thought I'd
use a barrier at first, in the test, to synchronize thread creation, but
I firmly believe that pthread_create() was to blame and now that's fixed
2025-01-03 17:34:29 -08:00
Justine Tunney
ed6d133a27
Use tgkill() on Linux and FreeBSD
This eliminates the chance of rare bugs when thread IDs are recycled.
2025-01-03 17:27:13 -08:00
Justine Tunney
97fc2aab41
Release Cosmopolitan v4.0.0 2025-01-02 22:27:34 -08:00
Justine Tunney
662e7b217f
Remove pthread_setcanceltype() from non-dbg strace 2025-01-02 22:25:29 -08:00
Justine Tunney
27f2777cc6
Fix aarch64 build 2025-01-02 22:19:49 -08:00
Justine Tunney
538ce338f4
Fix fork thread handle leak on windows 2025-01-02 19:33:14 -08:00
Justine Tunney
a15958edc6
Remove some legacy cruft
Function trace logs will report stack usage accurately. It won't include
the argv/environ block. Our clone() polyfill is now simpler and does not
use as much stack memory. Function call tracing on x86 is now faster too
2025-01-02 18:44:07 -08:00
Justine Tunney
8db646f6b2
Fix bug with systemvpe()
See #1253
2025-01-02 09:19:59 -08:00
Justine Tunney
fde03f8487
Remove leaf attribute where appropriate
This change fixes a bug where gcc assumed thread synchronization such as
pthread_cond_wait() wouldn't alter static variables, because the headers
were using __attribute__((__leaf__)) inappropriately.
2025-01-02 08:07:15 -08:00
Justine Tunney
f24c854b28
Write more runtime tests and fix bugs
This change adds tests for the new memory manager code particularly with
its windows support. Function call tracing now works reliably on Silicon
since our function hooker was missing new Apple self-modifying code APIs

Many tests that were disabled a long time ago on aarch64 are reactivated
by this change, now that arm support is on equal terms with x86. There's
been a lot of places where ftrace could cause deadlocks, which have been
hunted down across all platforms thanks to new tests. A bug in Windows's
kill() function has been identified.
2025-01-01 22:25:22 -08:00
Justine Tunney
0b3c81dd4e
Make fork() go 30% faster
This change makes fork() go nearly as fast as sys_fork() on UNIX. As for
Windows this change shaves about 4-5ms off fork() + wait() latency. This
is accomplished by using WriteProcessMemory() from the parent process to
setup the address space of a suspended process; it is better than a pipe
2025-01-01 04:59:38 -08:00
Justine Tunney
98c5847727
Fix fork waiter leak in nsync
This change fixes a bug where nsync waiter objects would leak. It'd mean
that long-running programs like runitd would run out of file descriptors
on NetBSD where waiter objects have ksem file descriptors. On other OSes
this bug is mostly harmless since the worst that can happen with a futex
is to leak a little bit of ram. The bug was caused because tib_nsync was
sneaking back in after the finalization code had cleared it. This change
refactors the thread exiting code to handle nsync teardown appropriately
and in making this change I found another issue, which is that user code
which is buggy, and tries to exit without joining joinable threads which
haven't been detached, would result in a deadlock. That doesn't sound so
bad, except the main thread is a joinable thread. So this deadlock would
be triggered in ways that put libc at fault. So we now auto-join threads
and libc will log a warning to --strace when that happens for any thread
2024-12-31 01:30:13 -08:00
Justine Tunney
fd7da586b5
Introduce example flash card program named rote 2024-12-30 03:03:43 -08:00
Justine Tunney
a51ccc8fb1
Remove old shuffle header 2024-12-30 03:03:32 -08:00
Justine Tunney
c7e3d9f7ff
Make recursive mutexes slightly faster 2024-12-30 01:37:14 -08:00
Justine Tunney
9ba5b227d9
Unblock stalled i/o signals on windows 2024-12-29 00:22:41 -08:00
Justine Tunney
aca4214ff6
Simplify memory manager code 2024-12-28 17:09:28 -08:00
Justine Tunney
379cd77078
Improve memory manager and signal handling
On Windows, mmap() now chooses addresses transactionally. It reduces the
risk of badness when interacting with the WIN32 memory manager. We don't
throw darts anymore. There is also no more retry limit, since we recover
from mystery maps more gracefully. The subroutine for combining adjacent
maps has been rewritten for clarity. The print maps subroutine is better

This change goes to great lengths to perfect the stack overflow code. On
Windows you can now longjmp() out of a crash signal handler. Guard pages
previously weren't being restored properly by the signal handler. That's
fixed, so on Windows you can now handle a stack overflow multiple times.
Great thought has been put into selecting the perfect SIGSTKSZ constants
so you can save sigaltstack() memory. You can now use kprintf() with 512
bytes of stack available. The guard pages beneath the main stack are now
recorded in the memory manager.

This change fixes getcontext() so it works right with the %rax register.
2024-12-27 01:33:00 -08:00
Justine Tunney
36e5861b0c
Reduce stack virtual memory consumption on Linux 2024-12-25 20:58:08 -08:00
Justine Tunney
cc8a9eb93c
Document execve() limitation on Windows
Closes #1253
2024-12-24 12:20:48 -08:00
Justine Tunney
0158579493
Use ape interpreter in flakes program 2024-12-24 12:16:50 -08:00
Justine Tunney
2de3845b25
Build tool for hunting down flakes 2024-12-24 11:36:16 -08:00
Justine Tunney
93e22c581f
Reduce pthread memory usage 2024-12-24 10:30:59 -08:00
Justine Tunney
ec2db4e40e
Avoid pthread_rwlock_wrlock() starvation 2024-12-24 10:30:11 -08:00
Justine Tunney
55b7aa1632
Allow user to override pthread mutex and cond 2024-12-23 21:57:52 -08:00
Justine Tunney
4705705548
Fix bugs in times() function 2024-12-23 20:57:10 -08:00
Justine Tunney
c8e10eef30
Make bulk_free() go faster 2024-12-23 20:31:57 -08:00
Justine Tunney
624573207e
Make threads faster and more reliable
This change doubles the performance of thread spawning. That's thanks to
our new stack manager, which allows us to avoid zeroing stacks. It gives
us 15µs spawns rather than 30µs spawns on Linux. Also, pthread_exit() is
faster now, since it doesn't need to acquire the pthread GIL. On NetBSD,
that helps us avoid allocating too many semaphores. Even if that happens
we're now able to survive semaphores running out and even memory running
out, when allocating *NSYNC waiter objects. I found a lot more rare bugs
in the POSIX threads runtime that could cause things to crash, if you've
got dozens of threads all spawning and joining dozens of threads. I want
cosmo to be world class production worthy for 2025 so happy holidays all
2024-12-21 22:13:00 -08:00
Justine Tunney
906bd06a5a
Fix MODE=tiny build 2024-12-17 01:36:29 -08:00
Justine Tunney
c8c81af0c7
Remove distracting code from dlmalloc 2024-12-16 22:54:30 -08:00
Justine Tunney
af7bd80430
Eliminate cyclic locks in runtime
This change introduces a new deadlock detector for Cosmo's POSIX threads
implementation. Error check mutexes will now track a DAG of nested locks
and report EDEADLK when a deadlock is theoretically possible. These will
occur rarely, but it's important for production hardening your code. You
don't even need to change your mutexes to use the POSIX error check mode
because `cosmocc -mdbg` will enable error checking on mutexes by default
globally. When cycles are found, an error message showing your demangled
symbols describing the strongly connected component are printed and then
the SIGTRAP is raised, which means you'll also get a backtrace if you're
using ShowCrashReports() too. This new error checker is so low-level and
so pure that it's able to verify the relationships of every libc runtime
lock, including those locks upon which the mutex implementation depends.
2024-12-16 22:25:12 -08:00
Justine Tunney
26c051c297
Spoof PID across execve() on Windows
It's now possible with cosmo and redbean, to deliver a signal to a child
process after it has called execve(). However the executed program needs
to be compiled using cosmocc. The cosmo runtime WinMain() implementation
now intercepts a _COSMO_PID environment variable that's set by execve().
It ensures the child process will use the same C:\ProgramData\cosmo\sigs
file, which is where kill() will place the delivered signal. We are able
to do this on Windows even better than NetBSD, which has a bug with this

Fixes #1334
2024-12-14 13:13:08 -08:00
Justine Tunney
9cc1bd04b2
Test rwlock more 2024-12-14 09:40:13 -08:00
Justine Tunney
69402f4d78
Support building ltests.c in MODE=dbg
Fixes #1226
2024-12-13 08:19:42 -08:00
Justine Tunney
838b54f906
Fix C++ math.h include order issue
Fixes #1257
2024-12-13 07:49:59 -08:00
Justine Tunney
2d43d400c6
Support process shared pthread_rwlock
Cosmo now has a non-nsync implementation of POSIX read-write locks. It's
possible to call pthread_rwlockattr_setpshared in PTHREAD_PROCESS_SHARED
mode. Furthermore, if cosmo is built with PTHREAD_USE_NSYNC set to zero,
then Cosmo shouldn't use nsync at all. That's helpful if you want to not
link any Apache 2.0 licensed code.
2024-12-13 03:00:06 -08:00
Justine Tunney
c22b413ac4
Make strcasestr() faster 2024-12-12 22:50:20 -08:00
Justine Tunney
22094ae9ca
Change language in leak detector 2024-12-10 11:04:35 -08:00
Justine Tunney
bda2a4d55e
Fix jtckdint version number 2024-12-07 03:19:11 -08:00
Justine Tunney
b490e23d63
Improve Windows sleep accuracy from 15ms to 15µs 2024-12-06 23:03:57 -08:00
Steven Dee (Jōshin)
b40140e6c5
Improve redbean concurrency (#1332)
In the course of playing with redbean I was confused about how the state
was behaving and then noticed that some stuff is maybe getting edited by
multiple processes. I tried to improve things by changing the definition
of the counter variables to be explicitly atomic. Claude assures me that
most modern Unixes support cross-process atomics, so I just went with it
on that front.

I also added some mutexes to the shared state to try to synchronize some
other things that might get written or read from workers but couldn't be
made atomic, mainly the rusage and time values. I could've probably been
less granular and just had a global shared-state lock, but I opted to be
fairly granular as a starting point.

This also reorders the resetting of the lastmeltdown timespec before the
SIGUSR2 signal is sent; hopefully this is okay.
2024-12-02 14:05:38 -08:00
Steven Dee (Jōshin)
3142758675
Fix atomic_fetch_sub on workers (#1331)
clangd was showing a diagnostic for this line.
2024-11-29 16:57:43 -08:00
Justine Tunney
cf9252f429
Correct redbean unix.commandv() docs
Fixes #1330
2024-11-29 12:15:03 -08:00
Justine Tunney
5fae582e82
Protect privileged demangler from stack overflow 2024-11-24 06:43:17 -08:00
Justine Tunney
ef00a7d0c2
Fix AFL crashes in C++ demangler
American Fuzzy Lop didn't need to try very hard, to crash our privileged
__demangle() implementation. This change helps ensure our barebones impl
will fail rather than crash when given adversarial input data.
2024-11-23 14:25:09 -08:00
471 changed files with 10380 additions and 6487 deletions

View file

@ -1,5 +1,8 @@
name: build
env:
COSMOCC_VERSION: 3.9.2
on:
push:
branches:
@ -19,13 +22,48 @@ jobs:
matrix:
mode: ["", tiny, rel, tinylinux, optlinux]
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
# Full checkout needed for git-restore-mtime-bare.
fetch-depth: 0
# TODO(jart): fork this action.
- uses: chetan/git-restore-mtime-action@v2
- uses: actions/cache/restore@v4
id: cache
with:
path: |
.cosmocc
o
key: ${{ env.COSMOCC_VERSION }}-${{ matrix.mode }}-${{ github.sha }}
restore-keys: |
${{ env.COSMOCC_VERSION }}-${{ matrix.mode }}-
${{ env.COSMOCC_VERSION }}-
- name: Restore mtimes
if: steps.cache.outputs.cache-hit == 'true'
run: |
while read mtime file; do
[ -f "$file" ] && touch -d "@$mtime" "$file"
done < o/.mtimes
- name: support ape bins 1
run: sudo cp build/bootstrap/ape.elf /usr/bin/ape
run: sudo cp -a build/bootstrap/ape.elf /usr/bin/ape
- name: support ape bins 2
run: sudo sh -c "echo ':APE:M::MZqFpD::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register"
- name: make matrix
run: V=0 make -j2 MODE=${{ matrix.mode }}
- name: Save mtimes
run: |
find o -type f -exec stat -c "%Y %n" {} \; > o/.mtimes
- uses: actions/cache/save@v4
with:
path: |
.cosmocc
o
key: ${{ env.COSMOCC_VERSION }}-${{ matrix.mode }}-${{ github.sha }}

1
.gitignore vendored
View file

@ -15,3 +15,4 @@ __pycache__
/tool/emacs/*.elc
/perf.data
/perf.data.old
/qemu*core

View file

@ -77,7 +77,8 @@ COMMA := ,
PWD := $(shell pwd)
# detect wsl2 running cosmopolitan binaries on the host by checking whether:
# - user ran build/bootstrap/make, in which case make's working directory is in wsl
# - user ran .cosmocc/current/bin/make, in which case make's working directory
# is in wsl
# - user ran make, in which case cocmd's working directory is in wsl
ifneq ($(findstring //wsl.localhost/,$(CURDIR) $(PWD)),)
$(warning wsl2 interop is enabled)
@ -89,7 +90,7 @@ UNAME_S := $(shell uname -s)
# apple still distributes a 17 year old version of gnu make
ifeq ($(MAKE_VERSION), 3.81)
$(error please use build/bootstrap/make)
$(error please use https://cosmo.zip/pub/cosmos/bin/make)
endif
LC_ALL = C
@ -135,7 +136,7 @@ ARCH = aarch64
HOSTS ?= pi pi5 studio freebsdarm
else
ARCH = x86_64
HOSTS ?= freebsd rhel7 xnu openbsd netbsd win10
HOSTS ?= freebsd rhel7 xnu openbsd netbsd win10 luna
endif
ZIPOBJ_FLAGS += -a$(ARCH)

View file

@ -3,7 +3,7 @@
[![build](https://github.com/jart/cosmopolitan/actions/workflows/build.yml/badge.svg)](https://github.com/jart/cosmopolitan/actions/workflows/build.yml)
# Cosmopolitan
[Cosmopolitan Libc](https://justine.lol/cosmopolitan/index.html) makes C
[Cosmopolitan Libc](https://justine.lol/cosmopolitan/index.html) makes C/C++
a build-once run-anywhere language, like Java, except it doesn't need an
interpreter or virtual machine. Instead, it reconfigures stock GCC and
Clang to output a POSIX-approved polyglot format that runs natively on
@ -87,15 +87,22 @@ ape/apeinstall.sh
```
You can now build the mono repo with any modern version of GNU Make. To
make life easier, we've included one in the cosmocc toolchain, which is
guaranteed to be compatible and furthermore includes our extensions for
doing build system sandboxing.
bootstrap your build, you can install Cosmopolitan Make from this site:
https://cosmo.zip/pub/cosmos/bin/make
E.g.:
```sh
build/bootstrap/make -j8
curl -LO https://cosmo.zip/pub/cosmos/bin/make
./make -j8
o//examples/hello
```
After you've built the repo once, you can also use the make from your
cosmocc at `.cosmocc/current/bin/make`. You might even prefer to alias
make to `$COSMO/.cosmocc/current/bin/make`.
Since the Cosmopolitan repository is very large, you might only want to
build one particular thing. Here's an example of a target that can be
compiled relatively quickly, which is a simple POSIX test that only
@ -103,7 +110,7 @@ depends on core LIBC packages.
```sh
rm -rf o//libc o//test
build/bootstrap/make o//test/posix/signal_test
.cosmocc/current/bin/make o//test/posix/signal_test
o//test/posix/signal_test
```
@ -112,21 +119,21 @@ list out each individual one. For example if you wanted to build and run
all the unit tests in the `TEST_POSIX` package, you could say:
```sh
build/bootstrap/make o//test/posix
.cosmocc/current/bin/make o//test/posix
```
Cosmopolitan provides a variety of build modes. For example, if you want
really tiny binaries (as small as 12kb in size) then you'd say:
```sh
build/bootstrap/make m=tiny
.cosmocc/current/bin/make m=tiny
```
You can furthermore cut out the bloat of other operating systems, and
have Cosmopolitan become much more similar to Musl Libc.
```sh
build/bootstrap/make m=tinylinux
.cosmocc/current/bin/make m=tinylinux
```
For further details, see [//build/config.mk](build/config.mk).

View file

@ -259,6 +259,9 @@ SECTIONS {
.debug_ranges 0 : { *(.debug_ranges) }
.debug_macro 0 : { *(.debug_macro) }
.debug_addr 0 : { *(.debug_addr) }
.debug_names 0 : { *(.debug_names) }
.debug_loclists 0 : { *(.debug_loclists) }
.debug_str_offsets 0 : { *(.debug_str_offsets) }
.ARM.attributes 0 : { KEEP(*(.ARM.attributes)) KEEP(*(.gnu.attributes)) }
.note.gnu.arm.ident 0 : { KEEP(*(.note.gnu.arm.ident)) }

View file

@ -386,6 +386,13 @@ SECTIONS {
_tbss_end = .;
} :Tls
.eh_frame : {
__eh_frame_start = .;
KEEP(*(.eh_frame))
*(.eh_frame.*)
__eh_frame_end = .;
} :Ram
.data . : {
/*BEGIN: Read/Write Data */
#if SupportsWindows()
@ -426,11 +433,6 @@ SECTIONS {
KEEP(*(.dtors))
__fini_array_end = .;
__eh_frame_start = .;
KEEP(*(.eh_frame))
*(.eh_frame.*)
__eh_frame_end = .;
/*BEGIN: Post-Initialization Read-Only */
. = ALIGN(. != 0 ? __SIZEOF_POINTER__ : 0);
KEEP(*(SORT_BY_NAME(.piro.relo.sort.*)))
@ -439,7 +441,6 @@ SECTIONS {
KEEP(*(.piro.pad.data))
*(.igot.plt)
KEEP(*(.dataepilogue))
. = ALIGN(. != 0 ? CONSTANT(COMMONPAGESIZE) : 0);
/*END: NT FORK COPYING */
_edata = .;
@ -519,6 +520,9 @@ SECTIONS {
.debug_rnglists 0 : { *(.debug_rnglists) }
.debug_macro 0 : { *(.debug_macro) }
.debug_addr 0 : { *(.debug_addr) }
.debug_names 0 : { *(.debug_names) }
.debug_loclists 0 : { *(.debug_loclists) }
.debug_str_offsets 0 : { *(.debug_str_offsets) }
.gnu.attributes 0 : { KEEP(*(.gnu.attributes)) }
.GCC.command.line 0 : { *(.GCC.command.line) }
@ -582,11 +586,11 @@ ape_rom_memsz = ape_rom_filesz;
ape_rom_align = CONSTANT(COMMONPAGESIZE);
ape_rom_rva = RVA(ape_rom_vaddr);
ape_ram_vaddr = ADDR(.data);
ape_ram_vaddr = ADDR(.eh_frame);
ape_ram_offset = ape_ram_vaddr - __executable_start;
ape_ram_paddr = LOADADDR(.data);
ape_ram_filesz = ADDR(.bss) - ADDR(.data);
ape_ram_memsz = _end - ADDR(.data);
ape_ram_paddr = LOADADDR(.eh_frame);
ape_ram_filesz = ADDR(.bss) - ADDR(.eh_frame);
ape_ram_memsz = _end - ADDR(.eh_frame);
ape_ram_align = CONSTANT(COMMONPAGESIZE);
ape_ram_rva = RVA(ape_ram_vaddr);
@ -596,7 +600,7 @@ ape_stack_offset = 0;
ape_stack_vaddr = DEFINED(ape_stack_vaddr) ? ape_stack_vaddr : 0x700000000000;
ape_stack_paddr = ape_ram_paddr + ape_ram_filesz;
ape_stack_filesz = 0;
ape_stack_memsz = DEFINED(ape_stack_memsz) ? ape_stack_memsz : 8 * 1024 * 1024;
ape_stack_memsz = DEFINED(ape_stack_memsz) ? ape_stack_memsz : 4 * 1024 * 1024;
ape_note_offset = ape_cod_offset + (ape_note - ape_cod_vaddr);
ape_note_filesz = ape_note_end - ape_note;

View file

@ -10,8 +10,8 @@ if [ ! -f ape/loader.c ]; then
cd "$COSMO" || exit
fi
if [ -x build/bootstrap/make ]; then
MAKE=build/bootstrap/make
if [ -x .cosmocc/current/bin/make ]; then
MAKE=.cosmocc/current/bin/make
else
MAKE=make
fi

View file

@ -99,3 +99,8 @@ rm -f cosmocc.zip cosmocc.zip.sha256sum
# commit output directory
cd "${OLDPWD}" || die
mv "${OUTPUT_TMP}" "${OUTPUT_DIR}" || die
# update current symlink
BASE=$(basename "${OUTPUT_DIR}")
DIR=$(dirname "${OUTPUT_DIR}")
ln -sfn "$BASE" "$DIR/current"

View file

@ -97,12 +97,12 @@ class shared_ref
size_t use_count() const noexcept
{
return shared + 1;
return __atomic_load_n(&shared, __ATOMIC_RELAXED) + 1;
}
size_t weak_count() const noexcept
{
return weak;
return __atomic_load_n(&weak, __ATOMIC_RELAXED);
}
private:
@ -349,7 +349,7 @@ class shared_ptr
template<typename U>
bool owner_before(const weak_ptr<U>& r) const noexcept
{
return !r.owner_before(*this);
return rc < r.rc;
}
private:
@ -382,6 +382,34 @@ class weak_ptr
rc->keep_weak();
}
weak_ptr(const weak_ptr& r) noexcept : p(r.p), rc(r.rc)
{
if (rc)
rc->keep_weak();
}
template<typename U>
requires __::shared_ptr_compatible<T, U>
weak_ptr(const weak_ptr<U>& r) noexcept : p(r.p), rc(r.rc)
{
if (rc)
rc->keep_weak();
}
weak_ptr(weak_ptr&& r) noexcept : p(r.p), rc(r.rc)
{
r.p = nullptr;
r.rc = nullptr;
}
template<typename U>
requires __::shared_ptr_compatible<T, U>
weak_ptr(weak_ptr<U>&& r) noexcept : p(r.p), rc(r.rc)
{
r.p = nullptr;
r.rc = nullptr;
}
~weak_ptr()
{
if (rc)
@ -410,6 +438,19 @@ class weak_ptr
swap(rc, r.rc);
}
weak_ptr& operator=(weak_ptr r) noexcept
{
swap(r);
return *this;
}
template<typename U>
requires __::shared_ptr_compatible<T, U>
weak_ptr& operator=(weak_ptr<U> r) noexcept
{
weak_ptr<T>(move(r)).swap(*this);
}
shared_ptr<T> lock() const noexcept
{
if (expired())

View file

@ -337,7 +337,7 @@ int main(int argc, char *argv[]) {
sigaddset(&block, SIGQUIT);
pthread_attr_t attr;
unassert(!pthread_attr_init(&attr));
unassert(!pthread_attr_setstacksize(&attr, 65536));
unassert(!pthread_attr_setstacksize(&attr, 65536 - getpagesize()));
unassert(!pthread_attr_setguardsize(&attr, getpagesize()));
unassert(!pthread_attr_setsigmask_np(&attr, &block));
unassert(!pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, 0));

322
examples/rote.c Normal file
View file

@ -0,0 +1,322 @@
#/*────────────────────────────────────────────────────────────────╗
To the extent possible under law, Justine Tunney has waived
all copyright and related or neighboring rights to this file,
as it is written in the following disclaimers:
http://unlicense.org/ │
http://creativecommons.org/publicdomain/zero/1.0/ │
*/
#include <ctype.h>
#include <signal.h>
#include <stdatomic.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <termios.h>
/**
* @fileoverview cosmopolitan flash cards viewer
*/
struct Card {
char* qa[2];
};
atomic_int g_done;
void onsig(int sig) {
g_done = 1;
}
void* xmalloc(int n) {
void* p;
if ((p = malloc(n)))
return p;
perror("malloc");
exit(1);
}
void* xrealloc(void* p, int n) {
if ((p = realloc(p, n)))
return p;
perror("realloc");
exit(1);
}
char* xstrcat(const char* a, const char* b) {
char* p;
size_t n, m;
n = strlen(a);
m = strlen(b);
p = xmalloc(n + m + 1);
memcpy(p, a, n);
memcpy(p + n, b, m + 1);
return p;
}
void shuffle(struct Card* a, int n) {
while (n > 1) {
int i = rand() % n--;
struct Card t = a[i];
a[i] = a[n];
a[n] = t;
}
}
char* trim(char* s) {
int i;
if (s) {
while (isspace(*s))
++s;
for (i = strlen(s); i--;) {
if (isspace(s[i])) {
s[i] = 0;
} else {
break;
}
}
}
return s;
}
char* readline(FILE* f) {
for (;;) {
char* line = trim(fgetln(f, 0));
if (!line)
return 0;
if (*line != '#')
if (*line)
return line;
}
}
char* fill(const char* text, int max_line_width, int* out_line_count) {
int text_len = strlen(text);
char* result = xmalloc(text_len * 2 + 1);
int result_pos = 0;
int line_start = 0;
int line_count = 1;
int i = 0;
while (i < text_len && isspace(text[i]))
i++;
while (i < text_len) {
int word_end = i;
while (word_end < text_len && !isspace(text[word_end]))
word_end++;
int word_length = word_end - i;
if ((result_pos - line_start) + (result_pos > line_start ? 1 : 0) +
word_length >
max_line_width) {
if (result_pos > line_start) {
++line_count;
result[result_pos++] = '\n';
line_start = result_pos;
}
} else if (result_pos > line_start) {
result[result_pos++] = ' ';
}
memcpy(result + result_pos, text + i, word_length);
result_pos += word_length;
i = word_end;
while (i < text_len && isspace(text[i]))
i++;
}
result[result_pos] = '\0';
result = xrealloc(result, result_pos + 1);
if (out_line_count)
*out_line_count = line_count;
return result;
}
void show(const char* text, int i, int n) {
// get pseudoteletypewriter dimensions
struct winsize ws = {80, 25};
tcgetwinsize(1, &ws);
int width = ws.ws_col;
if (width > (int)(ws.ws_col * .9))
width = ws.ws_col * .9;
if (width > 80)
width = 80;
width &= -2;
// clear display
printf("\033[H\033[J");
// display flash card text in middle of display
char buf[32];
int line_count;
char* lines = fill(text, width, &line_count);
sprintf(buf, "%d/%d\r\n\r\n", i + 1, n);
line_count += 2;
char* extra = xstrcat(buf, lines);
free(lines);
char* tokens = extra;
for (int j = 0;; ++j) {
char* line = strtok(tokens, "\n");
tokens = 0;
if (!line)
break;
printf("\033[%d;%dH%s", ws.ws_row / 2 - line_count / 2 + j + 1,
ws.ws_col / 2 - strlen(line) / 2 + 1, line);
}
free(extra);
fflush(stdout);
}
void usage(FILE* f, const char* prog) {
fprintf(f,
"usage: %s FILE\n"
"\n"
"here's an example of what your file should look like:\n"
"\n"
" # cosmopolitan flash cards\n"
" # california dmv drivers test\n"
" \n"
" which of the following point totals could result in "
"your license being suspended by the dmv?\n"
" 4 points in 12 months (middle)\n"
" \n"
" at 55 mph under good conditions a passenger vehicle can stop "
"within\n"
" 300 feet (not 200, not 400, middle)\n"
" \n"
" two sets of solid double yellow lines spaced two or more feet "
"apart indicate\n"
" a BARRIER (do not cross unless there's an opening)\n"
"\n"
"more specifically, empty lines are ignored, lines starting with\n"
"a hash are ignored, then an even number of lines must remain,\n"
"where each two lines is a card, holding question and answer.\n",
prog);
}
int main(int argc, char* argv[]) {
// show help
if (argc != 2) {
usage(stderr, argv[0]);
return 1;
}
if (!strcmp(argv[1], "-?") || //
!strcmp(argv[1], "-h") || //
!strcmp(argv[1], "--help")) {
usage(stdout, argv[0]);
return 0;
}
// teletypewriter is required
if (!isatty(0) || !isatty(1)) {
perror("isatty");
return 2;
}
// load cards
FILE* f = fopen(argv[1], "r");
if (!f) {
perror(argv[1]);
return 3;
}
int count = 0;
struct Card* cards = 0;
for (;;) {
struct Card card;
if (!(card.qa[0] = readline(f)))
break;
card.qa[0] = strdup(card.qa[0]);
if (!(card.qa[1] = readline(f))) {
fprintf(stderr, "%s: flash card file has odd number of lines\n", argv[1]);
exit(1);
}
card.qa[1] = strdup(card.qa[1]);
cards = xrealloc(cards, (count + 1) * sizeof(struct Card));
cards[count++] = card;
}
fclose(f);
// randomize
srand(time(0));
shuffle(cards, count);
// catch ctrl-c
struct sigaction sa;
sa.sa_flags = 0;
sa.sa_handler = onsig;
sigemptyset(&sa.sa_mask);
sigaction(SIGINT, &sa, 0);
// enter raw mode
struct termios ot;
tcgetattr(1, &ot);
struct termios nt = ot;
cfmakeraw(&nt);
nt.c_lflag |= ISIG;
tcsetattr(1, TCSANOW, &nt);
printf("\033[?25l");
// show flash cards
int i = 0;
while (!g_done) {
show(cards[i / 2].qa[i % 2], i / 2, count);
// press any key
char b[8] = {0};
read(0, b, sizeof(b));
// q quits
if (b[0] == 'q')
break;
// b or ctrl-b goes backward
if (b[0] == 'b' || //
b[0] == ('B' ^ 0100)) {
if (--i < 0)
i = count * 2 - 1;
i &= -2;
continue;
}
// p or ctrl-p goes backward
if (b[0] == 'p' || //
b[0] == ('P' ^ 0100)) {
if (--i < 0)
i = count * 2 - 1;
i &= -2;
continue;
}
// up arrow goes backward
if (b[0] == 033 && //
b[1] == '[' && //
b[2] == 'A') {
if (--i < 0)
i = count * 2 - 1;
i &= -2;
continue;
}
// left arrow goes backward
if (b[0] == 033 && //
b[1] == '[' && //
b[2] == 'D') {
if (--i < 0)
i = count * 2 - 1;
i &= -2;
continue;
}
// only advance
if (++i == count * 2)
i = 0;
}
// free memory
for (int i = 0; i < count; ++i)
for (int j = 0; j < 2; ++j)
free(cards[i].qa[j]);
free(cards);
// cleanup terminal and show cursor
tcsetattr(1, TCSANOW, &ot);
printf("\033[?25h");
printf("\n");
}

View file

@ -7,9 +7,13 @@
http://creativecommons.org/publicdomain/zero/1.0/ │
*/
#endif
#include "libc/dce.h"
#include "libc/intrin/maps.h"
#include "libc/mem/alg.h"
#include "libc/mem/mem.h"
#include "libc/runtime/runtime.h"
#include "libc/runtime/stack.h"
#include "libc/runtime/winargs.internal.h"
#include "libc/stdio/stdio.h"
#include "libc/x/xasprintf.h"
@ -67,8 +71,18 @@ int main(int argc, char *argv[]) {
Append((uintptr_t)&__auxv[i + 1],
xasprintf("&auxv[%d] = %#lx", i + 1, __auxv[i + 1]));
}
qsort(things.p, things.n, sizeof(*things.p), Compare);
for (int i = 0; i < things.n; ++i) {
printf("%012lx %s\n", things.p[i].i, things.p[i].s);
if (!IsWindows()) {
struct AddrSize stak = __get_main_stack();
Append((intptr_t)stak.addr + stak.size, "top of stack");
Append((intptr_t)stak.addr, "bottom of stack");
} else {
#ifdef __x86_64__
Append(GetStaticStackAddr(0) + GetStaticStackSize(), "top of stack");
Append(GetStaticStackAddr(0) + GetGuardSize(), "bottom of stack");
Append(GetStaticStackAddr(0), "bottom of guard region");
#endif
}
qsort(things.p, things.n, sizeof(*things.p), Compare);
for (int i = 0; i < things.n; ++i)
printf("%012lx %s\n", things.p[i].i, things.p[i].s);
}

17
examples/thread.c Normal file
View file

@ -0,0 +1,17 @@
#include <pthread.h>
#include <stdio.h>
// how to spawn a thread
void *my_thread(void *arg) {
printf("my_thread(%p) is running\n", arg);
return (void *)0x456L;
}
int main(int argc, char *argv[]) {
void *res;
pthread_t th;
pthread_create(&th, 0, my_thread, (void *)0x123L);
pthread_join(th, &res);
printf("my_thread() returned %p\n", res);
}

View file

@ -16,6 +16,7 @@
TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
PERFORMANCE OF THIS SOFTWARE.
*/
#include "libc/atomic.h"
#include "libc/calls/internal.h"
#include "libc/calls/struct/sigset.internal.h"
#include "libc/calls/struct/timespec.h"
@ -23,26 +24,37 @@
#include "libc/calls/syscall-sysv.internal.h"
#include "libc/errno.h"
#include "libc/intrin/atomic.h"
#include "libc/nt/enum/status.h"
#include "libc/nt/ntdll.h"
#include "libc/stdio/sysparam.h"
#include "libc/sysv/consts/clock.h"
#include "libc/sysv/consts/timer.h"
#include "libc/thread/tls.h"
#ifdef __x86_64__
static atomic_int usingRes;
static atomic_bool changedRes;
static textwindows int sys_clock_nanosleep_nt_impl(int clock,
struct timespec abs,
sigset_t waitmask) {
uint32_t msdelay;
struct timespec now;
for (;;) {
if (sys_clock_gettime_nt(clock, &now))
return -1;
if (timespec_cmp(now, abs) >= 0)
return 0;
msdelay = timespec_tomillis(timespec_sub(abs, now));
msdelay = MIN(msdelay, -1u);
if (_park_norestart(msdelay, waitmask) == -1)
return -1;
}
struct timespec now, wall;
uint32_t minRes, maxRes, oldRes;
sys_clock_gettime_nt(0, &wall);
if (sys_clock_gettime_nt(clock, &now))
return -1;
bool wantRes = clock == CLOCK_REALTIME || //
clock == CLOCK_MONOTONIC || //
clock == CLOCK_BOOTTIME;
if (wantRes && !atomic_fetch_add(&usingRes, 1))
changedRes = NtSuccess(NtQueryTimerResolution(&minRes, &maxRes, &oldRes)) &&
NtSuccess(NtSetTimerResolution(maxRes, true, &oldRes));
if (timespec_cmp(abs, now) > 0)
wall = timespec_add(wall, timespec_sub(abs, now));
int rc = _park_norestart(wall, waitmask);
if (wantRes && atomic_fetch_sub(&usingRes, 1) == 1 && changedRes)
NtSetTimerResolution(0, false, &minRes);
return rc;
}
textwindows int sys_clock_nanosleep_nt(int clock, int flags,

View file

@ -23,9 +23,9 @@
#include "libc/sysv/consts/clock.h"
#include "libc/sysv/errfuns.h"
int sys_clock_nanosleep_openbsd(int clock, int flags,
const struct timespec *req,
struct timespec *rem) {
relegated int sys_clock_nanosleep_openbsd(int clock, int flags,
const struct timespec *req,
struct timespec *rem) {
int res;
struct timespec start, relative, remainder;
if (!flags) {

View file

@ -57,6 +57,7 @@
*
* @param clock may be
* - `CLOCK_REALTIME`
* - `CLOCK_BOOTTIME`
* - `CLOCK_MONOTONIC`
* - `CLOCK_REALTIME_COARSE` but is likely to sleep negative time
* - `CLOCK_MONTONIC_COARSE` but is likely to sleep negative time

View file

@ -51,6 +51,7 @@
#include "libc/sysv/consts/fio.h"
#include "libc/sysv/consts/o.h"
#include "libc/sysv/errfuns.h"
#include "libc/thread/posixthread.internal.h"
#include "libc/thread/thread.h"
struct FileLock {
@ -67,7 +68,9 @@ struct FileLocks {
struct FileLock *free;
};
static struct FileLocks g_locks;
static struct FileLocks g_locks = {
.mu = PTHREAD_MUTEX_INITIALIZER,
};
static textwindows struct FileLock *NewFileLock(void) {
struct FileLock *fl;
@ -110,7 +113,7 @@ static textwindows bool EqualsFileLock(struct FileLock *fl, int64_t off,
textwindows void sys_fcntl_nt_lock_cleanup(int fd) {
struct FileLock *fl, *ft, **flp;
pthread_mutex_lock(&g_locks.mu);
_pthread_mutex_lock(&g_locks.mu);
for (flp = &g_locks.list, fl = *flp; fl;) {
if (fl->fd == fd) {
*flp = fl->next;
@ -122,7 +125,7 @@ textwindows void sys_fcntl_nt_lock_cleanup(int fd) {
fl = *flp;
}
}
pthread_mutex_unlock(&g_locks.mu);
_pthread_mutex_unlock(&g_locks.mu);
}
static textwindows int64_t GetfileSize(int64_t handle) {
@ -353,9 +356,9 @@ textwindows int sys_fcntl_nt(int fd, int cmd, uintptr_t arg) {
} else if (cmd == F_SETLK || cmd == F_SETLKW || cmd == F_GETLK) {
struct Fd *f = g_fds.p + fd;
if (f->cursor) {
pthread_mutex_lock(&g_locks.mu);
_pthread_mutex_lock(&g_locks.mu);
rc = sys_fcntl_nt_lock(f, fd, cmd, arg);
pthread_mutex_unlock(&g_locks.mu);
_pthread_mutex_unlock(&g_locks.mu);
} else {
rc = ebadf();
}

View file

@ -21,24 +21,23 @@
#include "libc/calls/syscall_support-nt.internal.h"
#include "libc/dce.h"
#include "libc/fmt/conv.h"
#include "libc/intrin/cxaatexit.h"
#include "libc/macros.h"
#include "libc/nt/accounting.h"
#include "libc/runtime/runtime.h"
#include "libc/thread/thread.h"
#define CTOR __attribute__((__constructor__(99)))
#define FT(x) (x.dwLowDateTime | (uint64_t)x.dwHighDateTime << 32)
static int cpus;
static double load;
static pthread_spinlock_t lock;
static struct NtFileTime idle1, kern1, user1;
textwindows int sys_getloadavg_nt(double *a, int n) {
int i, rc;
uint64_t elapsed, used;
struct NtFileTime idle, kern, user;
BLOCK_SIGNALS;
pthread_spin_lock(&lock);
__cxa_lock();
if (GetSystemTimes(&idle, &kern, &user)) {
elapsed = (FT(kern) - FT(kern1)) + (FT(user) - FT(user1));
if (elapsed) {
@ -54,12 +53,11 @@ textwindows int sys_getloadavg_nt(double *a, int n) {
} else {
rc = __winerr();
}
pthread_spin_unlock(&lock);
ALLOW_SIGNALS;
__cxa_unlock();
return rc;
}
__attribute__((__constructor__(40))) static textstartup void ntinitload(void) {
CTOR static textstartup void sys_getloadavg_nt_init(void) {
if (IsWindows()) {
load = 1;
cpus = __get_cpu_count() / 2;

View file

@ -28,8 +28,8 @@
// @return rdi is rdi+edx
.text.startup
__getntsyspath:
push %rbp
mov %rsp,%rbp
beg
pro
push %rdx
movpp %rdi,%rcx # call f=%rax(p1=%rcx,p2=%rdx)
sub $40,%rsp
@ -55,6 +55,7 @@ __getntsyspath:
jne 2f
movb $'/',-1(%rdi)
2: .loop 1b
leave
epi
ret
end
.endfn __getntsyspath,globl,hidden

View file

@ -96,9 +96,8 @@ static int OldApeLoader(char *s) {
static int CopyWithCwd(const char *q, char *p, char *e) {
char c;
if (*q != '/') {
if (q[0] == '.' && q[1] == '/') {
if (q[0] == '.' && q[1] == '/')
q += 2;
}
int got = __getcwd(p, e - p - 1 /* '/' */);
if (got != -1) {
p += got - 1;
@ -118,9 +117,10 @@ static int CopyWithCwd(const char *q, char *p, char *e) {
// if q exists then turn it into an absolute path.
static int TryPath(const char *q) {
if (!CopyWithCwd(q, g_prog.u.buf, g_prog.u.buf + sizeof(g_prog.u.buf))) {
if (!q)
return 0;
if (!CopyWithCwd(q, g_prog.u.buf, g_prog.u.buf + sizeof(g_prog.u.buf)))
return 0;
}
return !sys_faccessat(AT_FDCWD, g_prog.u.buf, F_OK, 0);
}
@ -129,9 +129,8 @@ static int TryPath(const char *q) {
void __init_program_executable_name(void) {
if (__program_executable_name && *__program_executable_name != '/' &&
CopyWithCwd(__program_executable_name, g_prog.u.buf,
g_prog.u.buf + sizeof(g_prog.u.buf))) {
g_prog.u.buf + sizeof(g_prog.u.buf)))
__program_executable_name = g_prog.u.buf;
}
}
static inline void InitProgramExecutableNameImpl(void) {
@ -212,14 +211,12 @@ static inline void InitProgramExecutableNameImpl(void) {
}
// don't trust argv or envp if set-id.
if (issetugid()) {
if (issetugid())
goto UseEmpty;
}
// try argv[0], then then $_.
if (TryPath(__argv[0]) || TryPath(__getenv(__envp, "_").s)) {
if (TryPath(__argv[0]) || TryPath(__getenv(__envp, "_").s))
goto UseBuf;
}
// give up and just copy argv[0] into it
if ((q = __argv[0])) {

View file

@ -114,7 +114,8 @@ static ssize_t GetDevUrandom(char *p, size_t n, unsigned f) {
ssize_t __getrandom(void *p, size_t n, unsigned f) {
ssize_t rc;
if (IsWindows()) {
rc = ProcessPrng(p, n) ? n : __winerr();
ProcessPrng(p, n); // never fails
rc = n;
} else if (have_getrandom) {
if (IsXnu() || IsOpenbsd()) {
rc = GetRandomBsd(p, n, GetRandomEntropy);
@ -184,9 +185,7 @@ ssize_t __getrandom(void *p, size_t n, unsigned f) {
* @raise EFAULT if the `n` bytes at `p` aren't valid memory
* @raise EINTR if we needed to block and a signal was delivered instead
* @cancelationpoint
* @asyncsignalsafe
* @restartable
* @vforksafe
*/
ssize_t getrandom(void *p, size_t n, unsigned f) {
ssize_t rc;

View file

@ -21,6 +21,7 @@
#include "libc/calls/syscall-sysv.internal.h"
#include "libc/dce.h"
#include "libc/intrin/describeflags.h"
#include "libc/intrin/rlimit.h"
#include "libc/intrin/strace.h"
#include "libc/runtime/runtime.h"
#include "libc/runtime/stack.h"
@ -47,8 +48,7 @@ int getrlimit(int resource, struct rlimit *rlim) {
} else if (!IsWindows()) {
rc = sys_getrlimit(resource, rlim);
} else if (resource == RLIMIT_STACK) {
rlim->rlim_cur = GetStaticStackSize();
rlim->rlim_max = GetStaticStackSize();
*rlim = __rlimit_stack_get();
rc = 0;
} else if (resource == RLIMIT_AS) {
rlim->rlim_cur = __virtualmax;

View file

@ -3,6 +3,7 @@
#include "libc/atomic.h"
#include "libc/calls/struct/sigset.h"
#include "libc/calls/struct/sigval.h"
#include "libc/calls/struct/timespec.h"
#include "libc/dce.h"
#include "libc/intrin/fds.h"
#include "libc/macros.h"
@ -33,6 +34,7 @@ int64_t GetConsoleOutputHandle(void);
void EchoConsoleNt(const char *, size_t, bool);
int IsWindowsExecutable(int64_t, const char16_t *);
void InterceptTerminalCommands(const char *, size_t);
void sys_read_nt_wipe_keystrokes(void);
forceinline bool __isfdopen(int fd) {
return 0 <= fd && fd < g_fds.n && g_fds.p[fd].kind != kFdEmpty;
@ -46,8 +48,8 @@ int _check_signal(bool);
int _check_cancel(void);
bool _is_canceled(void);
int sys_close_nt(int, int);
int _park_norestart(uint32_t, uint64_t);
int _park_restartable(uint32_t, uint64_t);
int _park_norestart(struct timespec, uint64_t);
int _park_restartable(struct timespec, uint64_t);
int sys_openat_metal(int, const char *, int, unsigned);
#ifdef __x86_64__

View file

@ -67,10 +67,9 @@ textstartup void InitializeMetalFile(void) {
size_t size = ROUNDUP(_ezip - __executable_start, 4096);
// TODO(jart): Restore support for ZIPOS on metal.
void *copied_base;
struct DirectMap dm;
dm = sys_mmap_metal(NULL, size, PROT_READ | PROT_WRITE,
MAP_SHARED_linux | MAP_ANONYMOUS_linux, -1, 0);
copied_base = dm.addr;
void *addr = sys_mmap_metal(NULL, size, PROT_READ | PROT_WRITE,
MAP_SHARED_linux | MAP_ANONYMOUS_linux, -1, 0);
copied_base = addr;
npassert(copied_base != (void *)-1);
memcpy(copied_base, (void *)(BANE + IMAGE_BASE_PHYSICAL), size);
__ape_com_base = copied_base;

View file

@ -17,6 +17,7 @@
PERFORMANCE OF THIS SOFTWARE.
*/
#include "libc/proc/ntspawn.h"
#include "libc/calls/state.internal.h"
#include "libc/calls/struct/sigset.internal.h"
#include "libc/calls/syscall_support-nt.internal.h"
#include "libc/intrin/strace.h"
@ -38,12 +39,15 @@
#include "libc/nt/struct/procthreadattributelist.h"
#include "libc/nt/struct/startupinfo.h"
#include "libc/nt/struct/startupinfoex.h"
#include "libc/nt/thunk/msabi.h"
#include "libc/proc/ntspawn.h"
#include "libc/stdalign.h"
#include "libc/str/str.h"
#include "libc/sysv/errfuns.h"
#ifdef __x86_64__
__msabi extern typeof(CloseHandle) *const __imp_CloseHandle;
struct SpawnBlock {
char16_t path[PATH_MAX];
char16_t cmdline[32767];
@ -63,10 +67,12 @@ static textwindows ssize_t ntspawn_read(intptr_t fh, char *buf, size_t len) {
bool ok;
uint32_t got;
struct NtOverlapped overlap = {.hEvent = CreateEvent(0, 0, 0, 0)};
ok = (ReadFile(fh, buf, len, 0, &overlap) ||
ok = overlap.hEvent &&
(ReadFile(fh, buf, len, 0, &overlap) ||
GetLastError() == kNtErrorIoPending) &&
GetOverlappedResult(fh, &overlap, &got, true);
CloseHandle(overlap.hEvent);
if (overlap.hEvent)
__imp_CloseHandle(overlap.hEvent);
return ok ? got : -1;
}
@ -86,7 +92,7 @@ static textwindows int ntspawn2(struct NtSpawnArgs *a, struct SpawnBlock *sb) {
if (fh == -1)
return -1;
ssize_t got = ntspawn_read(fh, p, pe - p);
CloseHandle(fh);
__imp_CloseHandle(fh);
if (got < 3)
return enoexec();
pe = p + got;

View file

@ -49,11 +49,9 @@ int sys_openat_metal(int dirfd, const char *file, int flags, unsigned mode) {
if ((fd = __reservefd(-1)) == -1)
return -1;
if (!_weaken(calloc) || !_weaken(free)) {
struct DirectMap dm;
dm = sys_mmap_metal(NULL, ROUNDUP(sizeof(struct MetalFile), 4096),
PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1,
0);
state = dm.addr;
state = sys_mmap_metal(NULL, ROUNDUP(sizeof(struct MetalFile), 4096),
PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS,
-1, 0);
if (state == (void *)-1)
return -1;
} else {

View file

@ -19,65 +19,96 @@
#include "libc/calls/internal.h"
#include "libc/calls/sig.internal.h"
#include "libc/calls/struct/sigset.h"
#include "libc/calls/struct/timespec.h"
#include "libc/calls/syscall_support-nt.internal.h"
#include "libc/fmt/wintime.internal.h"
#include "libc/intrin/atomic.h"
#include "libc/intrin/weaken.h"
#include "libc/nt/enum/wait.h"
#include "libc/nt/events.h"
#include "libc/nt/runtime.h"
#include "libc/nt/synchronization.h"
#include "libc/str/str.h"
#include "libc/sysv/consts/sicode.h"
#include "libc/sysv/errfuns.h"
#include "libc/thread/posixthread.internal.h"
#ifdef __x86_64__
// returns 0 on timeout or spurious wakeup
// returns 0 if deadline is reached
// raises EINTR if a signal delivery interrupted wait operation
// raises ECANCELED if this POSIX thread was canceled in masked mode
textwindows static int _park_thread(uint32_t msdelay, sigset_t waitmask,
textwindows static int _park_thread(struct timespec deadline, sigset_t waitmask,
bool restartable) {
struct PosixThread *pt = _pthread_self();
for (;;) {
uint32_t handl = 0;
intptr_t hands[2];
// perform the wait operation
intptr_t sigev;
if (!(sigev = CreateEvent(0, 0, 0, 0)))
return __winerr();
pt->pt_event = sigev;
pt->pt_blkmask = waitmask;
atomic_store_explicit(&pt->pt_blocker, PT_BLOCKER_EVENT,
memory_order_release);
//!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!//
int sig = 0;
uint32_t ws = 0;
if (!_is_canceled() &&
!(_weaken(__sig_get) && (sig = _weaken(__sig_get)(waitmask))))
ws = WaitForSingleObject(sigev, msdelay);
//!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!//
atomic_store_explicit(&pt->pt_blocker, 0, memory_order_release);
CloseHandle(sigev);
// create event object
intptr_t sigev;
if (!(sigev = CreateEvent(0, 0, 0, 0)))
return __winerr();
hands[handl++] = sigev;
// recursion is now safe
if (ws == -1u)
return __winerr();
int handler_was_called = 0;
if (sig)
handler_was_called = _weaken(__sig_relay)(sig, SI_KERNEL, waitmask);
if (_check_cancel())
return -1;
if (handler_was_called & SIG_HANDLED_NO_RESTART)
return eintr();
if (handler_was_called & SIG_HANDLED_SA_RESTART)
if (!restartable)
// create high precision timer if needed
if (memcmp(&deadline, &timespec_max, sizeof(struct timespec))) {
intptr_t hTimer;
if ((hTimer = CreateWaitableTimer(NULL, true, NULL))) {
int64_t due = TimeSpecToWindowsTime(deadline);
if (SetWaitableTimer(hTimer, &due, 0, NULL, NULL, false)) {
hands[handl++] = hTimer;
} else {
CloseHandle(hTimer);
}
}
}
// perform wait operation
struct PosixThread *pt = _pthread_self();
pt->pt_event = sigev;
pt->pt_blkmask = waitmask;
atomic_store_explicit(&pt->pt_blocker, PT_BLOCKER_EVENT,
memory_order_release);
//!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!//
int sig = 0;
uint32_t wi = 0;
if (!_is_canceled() &&
!(_weaken(__sig_get) && (sig = _weaken(__sig_get)(waitmask))))
wi = WaitForMultipleObjects(handl, hands, false, -1u);
//!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!/!//
atomic_store_explicit(&pt->pt_blocker, 0, memory_order_release);
for (int i = 0; i < handl; ++i)
CloseHandle(hands[i]);
// recursion is now safe
if (wi == 1)
return 0;
if (wi == -1u)
return __winerr();
int handler_was_called = 0;
if (!sig) {
if (_check_cancel())
return -1;
if (_weaken(__sig_get))
sig = _weaken(__sig_get)(waitmask);
}
if (sig)
handler_was_called = _weaken(__sig_relay)(sig, SI_KERNEL, waitmask);
if (_check_cancel())
return -1;
if (handler_was_called & SIG_HANDLED_NO_RESTART)
return eintr();
return 0;
if (handler_was_called & SIG_HANDLED_SA_RESTART)
if (!restartable)
return eintr();
}
}
textwindows int _park_norestart(uint32_t msdelay, sigset_t waitmask) {
return _park_thread(msdelay, waitmask, false);
textwindows int _park_norestart(struct timespec deadline, sigset_t waitmask) {
return _park_thread(deadline, waitmask, false);
}
textwindows int _park_restartable(uint32_t msdelay, sigset_t waitmask) {
return _park_thread(msdelay, waitmask, true);
textwindows int _park_restartable(struct timespec deadline, sigset_t waitmask) {
return _park_thread(deadline, waitmask, true);
}
#endif /* __x86_64__ */

View file

@ -18,21 +18,20 @@
*/
#include "libc/calls/internal.h"
#include "libc/calls/struct/sigset.internal.h"
#include "libc/calls/struct/timespec.h"
#include "libc/calls/syscall_support-nt.internal.h"
#ifdef __x86_64__
textwindows int sys_pause_nt(void) {
int rc;
// we don't strictly need to block signals, but it reduces signal
// delivery latency, by preventing other threads from delivering a
// signal asynchronously. it takes about ~5us to deliver a signal
// using SetEvent() whereas it takes ~30us to use SuspendThread(),
// GetThreadContext(), SetThreadContext(), and ResumeThread().
BLOCK_SIGNALS;
while (!(rc = _park_norestart(-1u, 0)))
donothing;
_park_norestart(timespec_max, 0);
ALLOW_SIGNALS;
return rc;
return -1;
}
#endif /* __x86_64__ */

View file

@ -694,6 +694,7 @@ static const uint16_t kPledgeStdio[] = {
__NR_linux_sched_getaffinity, //
__NR_linux_sched_setaffinity, //
__NR_linux_sigtimedwait, //
__NR_linux_getcpu, //
};
static const uint16_t kPledgeFlock[] = {

View file

@ -318,8 +318,8 @@ textwindows static int sys_poll_nt_actual(struct pollfd *fds, uint64_t nfds,
textwindows static int sys_poll_nt_impl(struct pollfd *fds, uint64_t nfds,
struct timespec deadline,
const sigset_t waitmask) {
uint32_t waitms;
int i, n, rc, got = 0;
struct timespec now, next, target;
// we normally don't check for signals until we decide to wait, since
// it's nice to have functions like write() be unlikely to EINTR, but
@ -344,9 +344,16 @@ textwindows static int sys_poll_nt_impl(struct pollfd *fds, uint64_t nfds,
}
if (got)
return got;
if (!(waitms = sys_poll_nt_waitms(deadline)))
now = sys_clock_gettime_monotonic_nt();
if (timespec_cmp(now, deadline) >= 0)
return 0;
if (_park_norestart(waitms, waitmask) == -1)
next = timespec_add(now, timespec_frommillis(POLL_INTERVAL_MS));
if (timespec_cmp(next, deadline) >= 0) {
target = deadline;
} else {
target = next;
}
if (_park_norestart(target, waitmask) == -1)
return -1;
}
}

View file

@ -48,6 +48,7 @@
#include "libc/nt/enum/wait.h"
#include "libc/nt/errors.h"
#include "libc/nt/events.h"
#include "libc/nt/memory.h"
#include "libc/nt/runtime.h"
#include "libc/nt/struct/inputrecord.h"
#include "libc/nt/synchronization.h"
@ -127,33 +128,46 @@ struct Keystrokes {
bool ohno_decckm;
bool bypass_mode;
uint16_t utf16hs;
int16_t freekeys;
size_t free_keys;
int64_t cin, cot;
struct Dll *list;
struct Dll *line;
struct Dll *free;
pthread_mutex_t lock;
struct Keystroke pool[512];
};
static struct Keystrokes __keystroke;
static pthread_mutex_t __keystroke_lock = PTHREAD_MUTEX_INITIALIZER;
textwindows void WipeKeystrokes(void) {
textwindows void sys_read_nt_wipe_keystrokes(void) {
bzero(&__keystroke, sizeof(__keystroke));
_pthread_mutex_wipe_np(&__keystroke_lock);
}
textwindows static void FreeKeystrokeImpl(struct Dll *key) {
dll_make_first(&__keystroke.free, key);
++__keystroke.freekeys;
++__keystroke.free_keys;
}
textwindows static struct Keystroke *AllocKeystroke(void) {
struct Keystroke *k;
if (!(k = HeapAlloc(GetProcessHeap(), 0, sizeof(struct Keystroke))))
return 0;
dll_init(&k->elem);
return k;
}
textwindows static struct Keystroke *NewKeystroke(void) {
struct Dll *e = dll_first(__keystroke.free);
if (!e) // See MIN(freekeys) before ReadConsoleInput()
__builtin_trap();
struct Keystroke *k = KEYSTROKE_CONTAINER(e);
dll_remove(&__keystroke.free, &k->elem);
--__keystroke.freekeys;
struct Dll *e;
struct Keystroke *k;
if ((e = dll_first(__keystroke.free))) {
dll_remove(&__keystroke.free, e);
k = KEYSTROKE_CONTAINER(e);
--__keystroke.free_keys;
} else {
// PopulateKeystrokes() should make this branch impossible
if (!(k = AllocKeystroke()))
return 0;
}
k->buflen = 0;
return k;
}
@ -169,15 +183,22 @@ textwindows static void FreeKeystrokes(struct Dll **list) {
FreeKeystroke(list, key);
}
textwindows static void PopulateKeystrokes(size_t want) {
struct Keystroke *k;
while (__keystroke.free_keys < want) {
if ((k = AllocKeystroke())) {
FreeKeystrokeImpl(&k->elem);
} else {
break;
}
}
}
textwindows static void OpenConsole(void) {
__keystroke.cin = CreateFile(u"CONIN$", kNtGenericRead | kNtGenericWrite,
kNtFileShareRead, 0, kNtOpenExisting, 0, 0);
__keystroke.cot = CreateFile(u"CONOUT$", kNtGenericRead | kNtGenericWrite,
kNtFileShareWrite, 0, kNtOpenExisting, 0, 0);
for (int i = 0; i < ARRAYLEN(__keystroke.pool); ++i) {
dll_init(&__keystroke.pool[i].elem);
FreeKeystrokeImpl(&__keystroke.pool[i].elem);
}
}
textwindows static int AddSignal(int sig) {
@ -191,11 +212,11 @@ textwindows static void InitConsole(void) {
}
textwindows static void LockKeystrokes(void) {
pthread_mutex_lock(&__keystroke.lock);
_pthread_mutex_lock(&__keystroke_lock);
}
textwindows static void UnlockKeystrokes(void) {
pthread_mutex_unlock(&__keystroke.lock);
_pthread_mutex_unlock(&__keystroke_lock);
}
textwindows int64_t GetConsoleInputHandle(void) {
@ -518,14 +539,12 @@ textwindows static void IngestConsoleInputRecord(struct NtInputRecord *r) {
!(__ttyconf.magic & kTtyNoIexten)) { // IEXTEN
if (__keystroke.bypass_mode) {
struct Keystroke *k = NewKeystroke();
if (!k)
return;
memcpy(k->buf, buf, sizeof(k->buf));
k->buflen = len;
dll_make_last(&__keystroke.line, &k->elem);
EchoConsoleNt(buf, len, true);
if (!__keystroke.freekeys) {
dll_make_last(&__keystroke.list, __keystroke.line);
__keystroke.line = 0;
}
__keystroke.bypass_mode = false;
return;
} else if (len == 1 && buf[0] && //
@ -615,6 +634,8 @@ textwindows static void IngestConsoleInputRecord(struct NtInputRecord *r) {
// allocate object to hold keystroke
struct Keystroke *k = NewKeystroke();
if (!k)
return;
memcpy(k->buf, buf, sizeof(k->buf));
k->buflen = len;
@ -628,12 +649,12 @@ textwindows static void IngestConsoleInputRecord(struct NtInputRecord *r) {
} else {
dll_make_last(&__keystroke.line, &k->elem);
// flush canonical mode line if oom or enter
if (!__keystroke.freekeys || (len == 1 && buf[0] &&
((buf[0] & 255) == '\n' || //
(buf[0] & 255) == __ttyconf.veol || //
((buf[0] & 255) == __ttyconf.veol2 &&
!(__ttyconf.magic & kTtyNoIexten))))) {
// flush canonical mode line on enter
if (len == 1 && buf[0] &&
((buf[0] & 255) == '\n' || //
(buf[0] & 255) == __ttyconf.veol || //
((buf[0] & 255) == __ttyconf.veol2 &&
!(__ttyconf.magic & kTtyNoIexten)))) {
dll_make_last(&__keystroke.list, __keystroke.line);
__keystroke.line = 0;
}
@ -644,15 +665,17 @@ textwindows static void IngestConsoleInput(void) {
uint32_t i, n;
struct NtInputRecord records[16];
for (;;) {
if (!__keystroke.freekeys)
return;
if (__keystroke.end_of_file)
return;
if (!GetNumberOfConsoleInputEvents(__keystroke.cin, &n))
goto UnexpectedEof;
if (!n || !__keystroke.freekeys)
if (n > ARRAYLEN(records))
n = ARRAYLEN(records);
PopulateKeystrokes(n + 1);
if (n > __keystroke.free_keys)
n = __keystroke.free_keys;
if (!n)
return;
n = MIN(__keystroke.freekeys, MIN(ARRAYLEN(records), n));
if (!ReadConsoleInput(__keystroke.cin, records, n, &n))
goto UnexpectedEof;
for (i = 0; i < n && !__keystroke.end_of_file; ++i)
@ -974,8 +997,10 @@ textwindows ssize_t ReadBuffer(int fd, void *data, size_t size, int64_t offset,
if (f->kind == kFdDevNull)
return 0;
if (f->kind == kFdDevRandom)
return ProcessPrng(data, size) ? size : __winerr();
if (f->kind == kFdDevRandom) {
ProcessPrng(data, size);
return size;
}
if (f->kind == kFdConsole)
return ReadFromConsole(f, data, size, waitmask);

View file

@ -23,6 +23,7 @@
#include "libc/dce.h"
#include "libc/errno.h"
#include "libc/intrin/describeflags.h"
#include "libc/intrin/rlimit.h"
#include "libc/intrin/strace.h"
#include "libc/macros.h"
#include "libc/runtime/runtime.h"
@ -88,10 +89,12 @@ int setrlimit(int resource, const struct rlimit *rlim) {
} else if (!IsWindows() && !(IsNetbsd() && resource == RLIMIT_AS)) {
rc = sys_setrlimit(resource, rlim);
} else if (resource == RLIMIT_STACK) {
rc = enotsup();
rc = 0;
} else {
rc = einval();
}
if (!rc && resource == RLIMIT_STACK)
__rlimit_stack_set(*rlim); // so __rlimit_stack_get() works on all OSes
if (resource == RLIMIT_AS) {
__virtualmax = rlim->rlim_cur;
errno = olde;

View file

@ -35,9 +35,8 @@ void shm_path_np(const char *name, char buf[hasatleast 78]) {
const char *a;
uint8_t digest[BLAKE2B256_DIGEST_LENGTH];
a = "/tmp/", n = 5;
if (IsLinux() && isdirectory("/dev/shm")) {
if (IsLinux() && isdirectory("/dev/shm"))
a = "/dev/shm/", n = 9;
}
BLAKE2B256(name, strlen(name), digest);
p = mempcpy(buf, a, n);
p = hexpcpy(p, digest, BLAKE2B256_DIGEST_LENGTH);

View file

@ -423,7 +423,7 @@ static int __sigaction(int sig, const struct sigaction *act,
* }
*
* void ContinueOnCrash(void) {
* struct sigaction sa = {.sa_handler = OnSigSegv,
* struct sigaction sa = {.sa_sigaction = OnCrash,
* .sa_flags = SA_SIGINFO | SA_RESETHAND};
* sigaction(SIGSEGV, &sa, 0);
* sigaction(SIGFPE, &sa, 0);

View file

@ -113,7 +113,7 @@ static int sigaltstack_bsd(const struct sigaltstack *neu,
* struct sigaction sa;
* struct sigaltstack ss;
* ss.ss_flags = 0;
* ss.ss_size = sysconf(_SC_MINSIGSTKSZ) + 8192;
* ss.ss_size = sysconf(_SC_SIGSTKSZ);
* ss.ss_sp = malloc(ss.ss_size);
* sigaltstack(&ss, 0);
* sigemptyset(&sa.ss_mask);
@ -121,11 +121,16 @@ static int sigaltstack_bsd(const struct sigaltstack *neu,
* sa.sa_handler = OnStackOverflow;
* sigaction(SIGSEGV, &sa, 0);
*
* Your stack size should be `sysconf(_SC_SIGSTKSZ)` which should be
* somewhere in the ballpark of 32kb to 64kb. You should go no lower
* than `sysconf(_SC_MINSIGSTKSZ) + 2048` which could be 4kb - 34kb.
* Cosmo also defines `SIGSTKSZ` as 32kb, which should also be safe.
*
* @param neu if non-null will install new signal alt stack
* @param old if non-null will receive current signal alt stack
* @return 0 on success, or -1 w/ errno
* @raise EFAULT if bad memory was supplied
* @raise ENOMEM if `neu->ss_size` is less than `MINSIGSTKSZ`
* @raise ENOMEM if `neu->ss_size` is beneath `sysconf(_SC_MINSIGSTKSZ)`
*/
int sigaltstack(const struct sigaltstack *neu, struct sigaltstack *old) {
int rc;

View file

@ -33,6 +33,7 @@
#include "libc/runtime/syslib.internal.h"
#include "libc/str/str.h"
#include "libc/sysv/consts/sa.h"
#include "libc/sysv/consts/sig.h"
/**
* @fileoverview XNU kernel callback normalization.
@ -513,6 +514,7 @@ privileged void __sigenter_xnu(int sig, struct siginfo_xnu *xnuinfo,
flags = __sighandflags[sig];
#ifdef __aarch64__
// xnu silicon claims to support sa_resethand but it does nothing
// this can be tested, since it clears the bit from flags as well
if (flags & SA_RESETHAND) {
@ -521,6 +523,13 @@ privileged void __sigenter_xnu(int sig, struct siginfo_xnu *xnuinfo,
__sighandflags[sig] = 0;
__sighandrvas[sig] = 0;
}
// unlike amd64, the instruction pointer on arm64 isn't advanced
// past the debugger breakpoint instruction automatically. we need
// this so execution can resume after __builtin_trap().
if (xnuctx && sig == SIGTRAP)
xnuctx->uc_mcontext->__ss.__pc += 4;
#endif
if (~flags & SA_SIGINFO) {

View file

@ -21,6 +21,7 @@
#include "libc/calls/sig.internal.h"
#include "libc/calls/struct/sigset.h"
#include "libc/calls/struct/sigset.internal.h"
#include "libc/calls/struct/timespec.h"
#include "libc/dce.h"
#include "libc/errno.h"
#include "libc/intrin/atomic.h"
@ -59,8 +60,7 @@ int sigsuspend(const sigset_t *ignore) {
// using SetEvent() whereas it takes ~30us to use SuspendThread(),
// GetThreadContext(), SetThreadContext(), and ResumeThread().
BLOCK_SIGNALS;
while (!(rc = _park_norestart(-1u, waitmask)))
donothing;
rc = _park_norestart(timespec_max, waitmask);
ALLOW_SIGNALS;
} else {
rc = sys_sigsuspend((uint64_t[2]){waitmask}, 8);

View file

@ -13,7 +13,6 @@ extern unsigned __sighandflags[NSIG + 1];
extern uint64_t __sighandmask[NSIG + 1];
extern const struct NtSecurityAttributes kNtIsInheritable;
void __fds_wipe(void);
void __fds_lock(void);
void __fds_unlock(void);

View file

@ -4,19 +4,21 @@ COSMOPOLITAN_C_START_
typedef uint64_t sigset_t;
int sigaddset(sigset_t *, int) paramsnonnull();
int sigdelset(sigset_t *, int) paramsnonnull();
int sigemptyset(sigset_t *) paramsnonnull();
int sigfillset(sigset_t *) paramsnonnull();
int sigandset(sigset_t *, const sigset_t *, const sigset_t *) paramsnonnull();
int sigorset(sigset_t *, const sigset_t *, const sigset_t *) paramsnonnull();
int sigisemptyset(const sigset_t *) paramsnonnull() nosideeffect;
int sigismember(const sigset_t *, int) paramsnonnull() nosideeffect;
int sigcountset(const sigset_t *) paramsnonnull() nosideeffect;
int sigprocmask(int, const sigset_t *, sigset_t *);
int sigsuspend(const sigset_t *);
int sigpending(sigset_t *);
int pthread_sigmask(int, const sigset_t *, sigset_t *);
/* clang-format off */
int sigaddset(sigset_t *, int) libcesque paramsnonnull();
int sigdelset(sigset_t *, int) libcesque paramsnonnull();
int sigemptyset(sigset_t *) libcesque paramsnonnull();
int sigfillset(sigset_t *) libcesque paramsnonnull();
int sigandset(sigset_t *, const sigset_t *, const sigset_t *) libcesque paramsnonnull();
int sigorset(sigset_t *, const sigset_t *, const sigset_t *) libcesque paramsnonnull();
int sigisemptyset(const sigset_t *) libcesque paramsnonnull() nosideeffect;
int sigismember(const sigset_t *, int) libcesque paramsnonnull() nosideeffect;
int sigcountset(const sigset_t *) libcesque paramsnonnull() nosideeffect;
int sigprocmask(int, const sigset_t *, sigset_t *) dontthrow;
int sigsuspend(const sigset_t *) dontthrow;
int sigpending(sigset_t *) libcesque;
int pthread_sigmask(int, const sigset_t *, sigset_t *) dontthrow;
/* clang-format on */
COSMOPOLITAN_C_END_
#endif /* COSMOPOLITAN_LIBC_CALLS_STRUCT_SIGSET_H_ */

View file

@ -5,27 +5,15 @@
#include "libc/sysv/consts/sig.h"
COSMOPOLITAN_C_START_
#ifndef MODE_DBG
/* block sigs because theoretical edge cases */
#define BLOCK_SIGNALS \
do { \
sigset_t _SigMask; \
_SigMask = __sig_block()
#define ALLOW_SIGNALS \
__sig_unblock(_SigMask); \
} \
while (0)
#else
/* doesn't block signals so we can get a crash
report, when a core runtime library crashes */
#define BLOCK_SIGNALS \
do { \
sigset_t _SigMask; \
sigprocmask(SIG_SETMASK, 0, &_SigMask)
#define ALLOW_SIGNALS \
} \
while (0)
#endif
sigset_t __sig_block(void);
void __sig_unblock(sigset_t);

View file

@ -1,13 +1,14 @@
#ifndef COSMOPOLITAN_LIBC_CALLS_STRUCT_UCONTEXT_INTERNAL_H_
#define COSMOPOLITAN_LIBC_CALLS_STRUCT_UCONTEXT_INTERNAL_H_
#include "libc/calls/ucontext.h"
#include "libc/nt/struct/context.h"
COSMOPOLITAN_C_START_
#ifdef __x86_64__
#define PC rip
#define SP rsp
#define BP rbp
#define RES0 rax
#define RES1 rdx
#define ARG0 rdi
#define ARG1 rsi
#define ARG2 rdx
@ -18,6 +19,8 @@ COSMOPOLITAN_C_START_
#define PC pc
#define SP sp
#define BP regs[29]
#define RES0 regs[0]
#define RES1 regs[1]
#define ARG0 regs[0]
#define ARG1 regs[1]
#define ARG2 regs[2]
@ -28,8 +31,5 @@ COSMOPOLITAN_C_START_
#error "unsupported architecture"
#endif
void _ntcontext2linux(struct ucontext *, const struct NtContext *);
void _ntlinux2context(struct NtContext *, const ucontext_t *);
COSMOPOLITAN_C_END_
#endif /* COSMOPOLITAN_LIBC_CALLS_STRUCT_UCONTEXT_INTERNAL_H_ */

View file

@ -2,6 +2,9 @@
#define COSMOPOLITAN_LIBC_CALLS_SYSCALL_NT_INTERNAL_H_
COSMOPOLITAN_C_START_
extern int sys_getppid_nt_cosmo;
extern int sys_getppid_nt_win32;
bool32 sys_isatty(int);
int sys_chdir_nt(const char *);
int sys_dup_nt(int, int, int, int);
@ -37,6 +40,7 @@ int sys_unlinkat_nt(int, const char *, int);
int64_t sys_lseek_nt(int, int64_t, int);
ssize_t sys_read_nt_impl(int, void *, size_t, int64_t);
ssize_t sys_readlinkat_nt(int, const char *, char *, size_t);
void sys_getppid_nt_wipe(int, int);
COSMOPOLITAN_C_END_
#endif /* COSMOPOLITAN_LIBC_CALLS_SYSCALL_NT_INTERNAL_H_ */

View file

@ -16,10 +16,9 @@
TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
PERFORMANCE OF THIS SOFTWARE.
*/
#include "libc/time.h"
#include "libc/calls/struct/timeval.h"
#include "libc/dce.h"
#include "libc/sysv/errfuns.h"
#include "libc/calls/calls.h"
#include "libc/calls/struct/timespec.h"
#include "libc/sysv/consts/clock.h"
/**
* Returns time as seconds from UNIX epoch.
@ -29,15 +28,11 @@
* @asyncsignalsafe
*/
int64_t time(int64_t *opt_out_ret) {
int64_t secs;
struct timeval tv;
if (gettimeofday(&tv, 0) != -1) {
secs = tv.tv_sec;
if (opt_out_ret) {
*opt_out_ret = secs;
}
} else {
secs = -1;
}
int64_t secs = -1;
struct timespec ts;
if (!clock_gettime(CLOCK_REALTIME, &ts))
secs = ts.tv_sec;
if (opt_out_ret)
*opt_out_ret = secs;
return secs;
}

View file

@ -80,7 +80,8 @@ textwindows int IsWindowsExecutable(int64_t handle, const char16_t *path) {
uint32_t got;
BLOCK_SIGNALS;
struct NtOverlapped overlap = {.hEvent = CreateEvent(0, 0, 0, 0)};
ok = (ReadFile(handle, buf, 2, 0, &overlap) ||
ok = overlap.hEvent &&
(ReadFile(handle, buf, 2, 0, &overlap) ||
GetLastError() == kNtErrorIoPending) &&
GetOverlappedResult(handle, &overlap, &got, true);
CloseHandle(overlap.hEvent);

View file

@ -9,18 +9,33 @@ COSMOPOLITAN_C_START_
#define _COSMO_ATOMIC(x) x
#endif
errno_t cosmo_once(_COSMO_ATOMIC(unsigned) *, void (*)(void)) libcesque;
errno_t cosmo_once(_COSMO_ATOMIC(unsigned) *, void (*)(void));
int systemvpe(const char *, char *const[], char *const[]) libcesque;
char *GetProgramExecutableName(void) libcesque;
void unleaf(void) libcesque;
bool32 IsLinuxModern(void) libcesque;
int __demangle(char *, const char *, size_t) libcesque;
int __is_mangled(const char *) libcesque;
bool32 IsLinuxModern(void) libcesque;
int LoadZipArgs(int *, char ***) libcesque;
int cosmo_args(const char *, char ***) libcesque;
int LoadZipArgs(int *, char ***) libcesque;
int cosmo_futex_wake(_COSMO_ATOMIC(int) *, int, char);
int cosmo_futex_wait(_COSMO_ATOMIC(int) *, int, char, int,
const struct timespec *);
errno_t cosmo_stack_alloc(size_t *, size_t *, void **) libcesque;
errno_t cosmo_stack_free(void *, size_t, size_t) libcesque;
void cosmo_stack_clear(void) libcesque;
void cosmo_stack_setmaxstacks(int) libcesque;
int cosmo_stack_getmaxstacks(void) libcesque;
int __deadlock_check(void *, int) libcesque;
int __deadlock_tracked(void *) libcesque;
void __deadlock_record(void *, int) libcesque;
void __deadlock_track(void *, int) libcesque;
void __deadlock_untrack(void *) libcesque;
COSMOPOLITAN_C_END_
#endif /* COSMOPOLITAN_LIBC_COSMO_H_ */

View file

@ -47,7 +47,14 @@ __oops_win32:
// @note ape.S and ape-loader both set RCX to XNU on Darwin
// @noreturn
_start:
#ifdef __x86_64__
.cfi_startproc
#if defined(__x86_64__)
.cfi_undefined rip
#elif defined(__aarch64__)
.cfi_undefined x30
#endif /* __x86_64__ */
#if defined(__x86_64__)
#if SupportsFreebsd()
// detect free besiyata dishmaya
@ -159,4 +166,5 @@ _start:
#else
#error "architecture unsupported"
#endif /* __x86_64__ */
.cfi_endproc
.endfn _start,weak,hidden

View file

@ -57,6 +57,7 @@
#include "libc/sysv/consts/prot.h"
#include "libc/sysv/errfuns.h"
#include "libc/temp.h"
#include "libc/thread/posixthread.internal.h"
#include "libc/thread/thread.h"
#include "libc/thread/tls.h"
@ -131,6 +132,8 @@ struct {
long __sysv2nt14();
long foreign_tramp();
void __dlopen_lock(void);
void __dlopen_unlock(void);
static _Thread_local char dlerror_buf[128];
@ -251,7 +254,7 @@ static bool elf_slurp(struct Loaded *l, int fd, const char *file) {
return true;
}
static dontinline bool elf_load(struct Loaded *l, const char *file, long pagesz,
dontinline static bool elf_load(struct Loaded *l, const char *file, long pagesz,
char *interp_path, size_t interp_size) {
int fd;
if ((fd = open(file, O_RDONLY | O_CLOEXEC)) == -1)
@ -277,7 +280,7 @@ static long *push_strs(long *sp, char **list, int count) {
return sp;
}
static wontreturn dontinstrument void foreign_helper(void **p) {
wontreturn dontinstrument static void foreign_helper(void **p) {
__foreign.dlopen = p[0];
__foreign.dlsym = p[1];
__foreign.dlclose = p[2];
@ -285,7 +288,7 @@ static wontreturn dontinstrument void foreign_helper(void **p) {
_longjmp(__foreign.jb, 1);
}
static dontinline void elf_exec(const char *file, char **envp) {
dontinline static void elf_exec(const char *file, char **envp) {
// get microprocessor page size
long pagesz = __pagesize;
@ -409,7 +412,7 @@ static char *dlerror_set(const char *str) {
return dlerror_buf;
}
static dontinline char *foreign_alloc_block(void) {
dontinline static char *foreign_alloc_block(void) {
char *p = 0;
size_t sz = 65536;
if (!IsWindows()) {
@ -432,17 +435,16 @@ static dontinline char *foreign_alloc_block(void) {
return p;
}
static dontinline void *foreign_alloc(size_t n) {
dontinline static void *foreign_alloc(size_t n) {
void *res;
static char *block;
static pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER;
pthread_mutex_lock(&lock);
__dlopen_lock();
if (!block || READ32LE(block) + n > 65536)
if (!(block = foreign_alloc_block()))
return 0;
res = block + READ32LE(block);
WRITE32LE(block, READ32LE(block) + n);
pthread_mutex_unlock(&lock);
__dlopen_unlock();
return res;
}
@ -546,7 +548,7 @@ static void *foreign_thunk_nt(void *func) {
return code;
}
static dontinline bool foreign_compile(char exe[hasatleast PATH_MAX]) {
dontinline static bool foreign_compile(char exe[hasatleast PATH_MAX]) {
// construct path
strlcpy(exe, get_tmp_dir(), PATH_MAX);
@ -808,7 +810,7 @@ void *cosmo_dlopen(const char *path, int mode) {
}
ALLOW_CANCELATION;
ALLOW_SIGNALS;
STRACE("dlopen(%#s, %d) → %p% m", path, mode, res);
STRACE("cosmo_dlopen(%#s, %d) → %p% m", path, mode, res);
return res;
}
@ -853,7 +855,7 @@ void *cosmo_dlsym(void *handle, const char *name) {
} else {
func = 0;
}
STRACE("dlsym(%p, %#s) → %p", handle, name, func);
STRACE("cosmo_dlsym(%p, %#s) → %p", handle, name, func);
return func;
}
@ -888,7 +890,7 @@ int cosmo_dlclose(void *handle) {
} else {
res = -1;
}
STRACE("dlclose(%p) → %d", handle, res);
STRACE("cosmo_dlclose(%p) → %d", handle, res);
return res;
}
@ -907,6 +909,6 @@ char *cosmo_dlerror(void) {
} else {
res = dlerror_buf;
}
STRACE("dlerror() → %#s", res);
STRACE("cosmo_dlerror() → %#s", res);
return res;
}

View file

@ -17,6 +17,10 @@
PERFORMANCE OF THIS SOFTWARE.
*/
#include "libc/dlopen/dlfcn.h"
#include "libc/intrin/strace.h"
#define DLOPEN_ERROR \
"dlopen() isn't supported; consider using cosmo_dlopen() and read its docs"
/**
* Opens dynamic shared object using host platform libc.
@ -27,12 +31,13 @@
*
* @return null always
*/
void *dlopen(const char *, int) {
void *dlopen(const char *path, int mode) {
STRACE("dlopen(%#s, %d) → 0 [%s]", path, mode, DLOPEN_ERROR);
return 0;
}
char *dlerror(void) {
return "dlopen() isn't supported by cosmo; try using cosmo_dlopen()";
return DLOPEN_ERROR;
}
void *dlsym(void *, const char *) {

View file

@ -68,6 +68,7 @@
#define EM_NONE 0
#define EM_M32 1
#define EM_386 3
#define EM_MIPS 8
#define EM_PPC64 21
#define EM_S390 22
#define EM_ARM 40

View file

@ -26,11 +26,11 @@ COSMOPOLITAN_C_START_
/* this header is included by 700+ files; therefore we */
/* hand-roll &__get_tls()->tib_errno to avoid #include */
/* cosmopolitan uses x28 as the tls register b/c apple */
#define errno \
(*__extension__({ \
errno_t *__ep; \
__asm__("sub\t%0,x28,#512-0x3c" : "=r"(__ep)); \
__ep; \
#define errno \
(*__extension__({ \
errno_t *__ep; \
__asm__("sub\t%0,x28,#1024-0x3c" : "=r"(__ep)); \
__ep; \
}))
#else
#define errno (*__errno_location())

View file

@ -49,6 +49,6 @@
int __vcscanf(int (*)(void *), int (*)(int, void *), void *, const char *,
va_list);
int __fmt(void *, void *, const char *, va_list, int *);
__msabi char16_t *__itoa16(char16_t[21], uint64_t);
char16_t *__itoa16(char16_t[21], uint64_t) __msabi;
#endif /* COSMOPOLITAN_LIBC_FMT_STRTOL_H_ */

View file

@ -135,7 +135,7 @@ typedef struct {
#define strftimeesque(n) __attribute__((__format__(__strftime__, n, 0)))
#ifndef privileged
#define privileged _Section(".privileged") dontinline dontinstrument dontubsan
#define privileged _Section(".privileged") dontinstrument dontubsan
#endif
#ifndef wontreturn

View file

@ -2,9 +2,9 @@
#undef __COSMOPOLITAN__
#endif
#define __COSMOPOLITAN_MAJOR__ 3
#define __COSMOPOLITAN_MINOR__ 9
#define __COSMOPOLITAN_PATCH__ 7
#define __COSMOPOLITAN_MAJOR__ 4
#define __COSMOPOLITAN_MINOR__ 0
#define __COSMOPOLITAN_PATCH__ 2
#define __COSMOPOLITAN__ \
(100000000 * __COSMOPOLITAN_MAJOR__ + 1000000 * __COSMOPOLITAN_MINOR__ + \
__COSMOPOLITAN_PATCH__)

View file

@ -20,7 +20,7 @@
#include "libc/intrin/getenv.h"
#include "libc/intrin/kprintf.h"
privileged struct Env __getenv(char **p, const char *k) {
privileged optimizesize struct Env __getenv(char **p, const char *k) {
char *t;
int i, j;
for (i = 0; (t = p[i]); ++i) {

View file

@ -16,155 +16,18 @@
TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
PERFORMANCE OF THIS SOFTWARE.
*/
#include "libc/assert.h"
#include "libc/dce.h"
#include "libc/nexgen32e/nexgen32e.h"
#include "libc/nexgen32e/x86feature.h"
#include "libc/str/str.h"
typedef char xmm_t __attribute__((__vector_size__(16), __aligned__(1)));
typedef long long xmm_a __attribute__((__vector_size__(16), __aligned__(16)));
static void bzero128(char *p, size_t n) {
xmm_t v = {0};
if (n <= 32) {
*(xmm_t *)(p + n - 16) = v;
*(xmm_t *)p = v;
} else {
do {
n -= 32;
*(xmm_t *)(p + n) = v;
*(xmm_t *)(p + n + 16) = v;
} while (n > 32);
*(xmm_t *)(p + 16) = v;
*(xmm_t *)p = v;
}
}
#if defined(__x86_64__) && !defined(__chibicc__)
_Microarchitecture("avx") static void bzero_avx(char *p, size_t n) {
xmm_t v = {0};
if (n <= 32) {
*(xmm_t *)(p + n - 16) = v;
*(xmm_t *)p = v;
} else if (n >= 1024 && X86_HAVE(ERMS)) {
asm("rep stosb" : "+D"(p), "+c"(n), "=m"(*(char(*)[n])p) : "a"(0));
} else {
if (n < kHalfCache3 || !kHalfCache3) {
do {
n -= 32;
*(xmm_t *)(p + n) = v;
*(xmm_t *)(p + n + 16) = v;
} while (n > 32);
} else {
while ((uintptr_t)(p + n) & 15) {
p[--n] = 0;
}
do {
n -= 32;
__builtin_ia32_movntdq((xmm_a *)(p + n), (xmm_a)v);
__builtin_ia32_movntdq((xmm_a *)(p + n + 16), (xmm_a)v);
} while (n > 32);
asm("sfence");
}
*(xmm_t *)(p + 16) = v;
*(xmm_t *)p = v;
}
}
#endif
/**
* Sets memory to zero.
*
* bzero n=0 661 picoseconds
* bzero n=1 661 ps/byte 1,476 mb/s
* bzero n=2 330 ps/byte 2,952 mb/s
* bzero n=3 220 ps/byte 4,428 mb/s
* bzero n=4 165 ps/byte 5,904 mb/s
* bzero n=7 94 ps/byte 10,333 mb/s
* bzero n=8 41 ps/byte 23,618 mb/s
* bzero n=15 44 ps/byte 22,142 mb/s
* bzero n=16 20 ps/byte 47,236 mb/s
* bzero n=31 21 ps/byte 45,760 mb/s
* bzero n=32 20 ps/byte 47,236 mb/s
* bzero n=63 10 ps/byte 92,997 mb/s
* bzero n=64 15 ps/byte 62,982 mb/s
* bzero n=127 15 ps/byte 62,490 mb/s
* bzero n=128 10 ps/byte 94,473 mb/s
* bzero n=255 14 ps/byte 68,439 mb/s
* bzero n=256 9 ps/byte 105 gb/s
* bzero n=511 15 ps/byte 62,859 mb/s
* bzero n=512 11 ps/byte 83,976 mb/s
* bzero n=1023 15 ps/byte 61,636 mb/s
* bzero n=1024 10 ps/byte 88,916 mb/s
* bzero n=2047 9 ps/byte 105 gb/s
* bzero n=2048 8 ps/byte 109 gb/s
* bzero n=4095 8 ps/byte 115 gb/s
* bzero n=4096 8 ps/byte 118 gb/s
* bzero n=8191 7 ps/byte 129 gb/s
* bzero n=8192 7 ps/byte 130 gb/s
* bzero n=16383 6 ps/byte 136 gb/s
* bzero n=16384 6 ps/byte 137 gb/s
* bzero n=32767 6 ps/byte 140 gb/s
* bzero n=32768 6 ps/byte 141 gb/s
* bzero n=65535 15 ps/byte 64,257 mb/s
* bzero n=65536 15 ps/byte 64,279 mb/s
* bzero n=131071 15 ps/byte 63,166 mb/s
* bzero n=131072 15 ps/byte 63,115 mb/s
* bzero n=262143 15 ps/byte 62,052 mb/s
* bzero n=262144 15 ps/byte 62,097 mb/s
* bzero n=524287 15 ps/byte 61,699 mb/s
* bzero n=524288 15 ps/byte 61,674 mb/s
* bzero n=1048575 16 ps/byte 60,179 mb/s
* bzero n=1048576 15 ps/byte 61,330 mb/s
* bzero n=2097151 15 ps/byte 61,071 mb/s
* bzero n=2097152 15 ps/byte 61,065 mb/s
* bzero n=4194303 16 ps/byte 60,942 mb/s
* bzero n=4194304 16 ps/byte 60,947 mb/s
* bzero n=8388607 16 ps/byte 60,872 mb/s
* bzero n=8388608 16 ps/byte 60,879 mb/s
*
* @param p is memory address
* @param n is byte length
* @return p
* @asyncsignalsafe
*/
void bzero(void *p, size_t n) {
char *b;
uint64_t x;
b = p;
#ifdef __x86_64__
asm("xorl\t%k0,%k0" : "=r"(x));
#else
if (1) {
memset(p, 0, n);
return;
}
x = 0;
#endif
if (n <= 16) {
if (n >= 8) {
__builtin_memcpy(b, &x, 8);
__builtin_memcpy(b + n - 8, &x, 8);
} else if (n >= 4) {
__builtin_memcpy(b, &x, 4);
__builtin_memcpy(b + n - 4, &x, 4);
} else if (n) {
do {
asm volatile("" ::: "memory");
b[--n] = x;
} while (n);
}
#if defined(__x86_64__) && !defined(__chibicc__)
} else if (IsTiny()) {
asm("rep stosb" : "+D"(b), "+c"(n), "=m"(*(char(*)[n])b) : "a"(0));
return;
} else if (X86_HAVE(AVX)) {
bzero_avx(b, n);
#endif
} else {
bzero128(b, n);
}
memset(p, 0, n);
}
__weak_reference(bzero, explicit_bzero);

View file

@ -59,7 +59,7 @@ textwindows int sys_clock_gettime_nt(int clock, struct timespec *ts) {
// —Quoth MSDN § Windows Time
//
QueryUnbiasedInterruptTimePrecise(&hectons);
*ts = timespec_fromnanos(hectons * 100);
*ts = WindowsDurationToTimeSpec(hectons);
return 0;
case _CLOCK_MONOTONIC_COARSE:
//
@ -83,7 +83,7 @@ textwindows int sys_clock_gettime_nt(int clock, struct timespec *ts) {
// —Quoth MSDN § QueryUnbiasedInterruptTimePrecise
//
QueryUnbiasedInterruptTime(&hectons);
*ts = timespec_fromnanos(hectons * 100);
*ts = WindowsDurationToTimeSpec(hectons);
return 0;
case _CLOCK_BOOTTIME:
//
@ -95,7 +95,7 @@ textwindows int sys_clock_gettime_nt(int clock, struct timespec *ts) {
// —Quoth MSDN § Interrupt Time
//
QueryInterruptTimePrecise(&hectons);
*ts = timespec_fromnanos(hectons * 100);
*ts = WindowsDurationToTimeSpec(hectons);
return 0;
case _CLOCK_PROCESS_CPUTIME_ID:
GetProcessTimes(GetCurrentProcess(), &ftCreation, &ftExit, &ftKernel,

View file

@ -18,6 +18,7 @@
*/
#include "libc/assert.h"
#include "libc/atomic.h"
#include "libc/calls/cp.internal.h"
#include "libc/calls/internal.h"
#include "libc/calls/sig.internal.h"
#include "libc/calls/struct/sigset.h"
@ -232,6 +233,7 @@ static int cosmo_futex_fix_timeout (struct timespec *memory, int clock,
* @raise EAGAIN if `*w` wasn't `expect`
* @raise EINTR if a signal handler was called while waiting
* @raise ECANCELED if calling thread was canceled while waiting
* @cancelationpoint
*/
int cosmo_futex_wait (atomic_int *w, int expect, char pshare,
int clock, const struct timespec *abstime) {
@ -240,6 +242,7 @@ int cosmo_futex_wait (atomic_int *w, int expect, char pshare,
struct PosixThread *pt;
struct timespec tsmem;
struct timespec *timeout = 0;
BEGIN_CANCELATION_POINT;
cosmo_once (&g_cosmo_futex.once, cosmo_futex_init);
@ -351,6 +354,7 @@ Finished:
DescribeTimespec (0, abstime),
DescribeErrno (rc));
END_CANCELATION_POINT;
return rc;
}

View file

@ -21,16 +21,15 @@
.privileged
cosmo_futex_thunk:
beg
pro
#ifdef __x86_64__
push %rbp
mov %rsp,%rbp
mov %rcx,%r10
mov __NR_futex,%eax
clc
syscall
jnc 1f
neg %eax
1: pop %rbp
#elif defined(__aarch64__)
ldr x7,=__hostos
ldr w7,[x7]
@ -46,5 +45,7 @@ cosmo_futex_thunk:
#else
#error "unsupported architecture"
#endif /* __x86_64__ */
1: ret
1: epi
ret
end
.endfn cosmo_futex_thunk,globl,hidden

26
libc/intrin/count.c Normal file
View file

@ -0,0 +1,26 @@
/*-*- mode:c;indent-tabs-mode:nil;c-basic-offset:2;tab-width:8;coding:utf-8 -*-│
vi: set et ft=c ts=2 sts=2 sw=2 fenc=utf-8 :vi
Copyright 2024 Justine Alexandra Roberts Tunney
Permission to use, copy, modify, and/or distribute this software for
any purpose with or without fee is hereby granted, provided that the
above copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL
WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE
AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL
DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR
PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER
TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
PERFORMANCE OF THIS SOFTWARE.
*/
#include "libc/atomic.h"
#include "libc/stdalign.h"
#include "libc/thread/thread.h"
// this counter is important because pthread_exit() needs to know if
// it's an orphan thread, without needing to acquire _pthread_lock()
// which causes contention and a file descriptor explosion on netbsd
alignas(64) atomic_uint _pthread_count = 1;

View file

@ -20,16 +20,13 @@
#include "libc/intrin/atomic.h"
#include "libc/intrin/fds.h"
#include "libc/runtime/runtime.h"
#include "libc/thread/posixthread.internal.h"
struct Cursor *__cursor_new(void) {
struct Cursor *c;
if ((c = _mapanon(sizeof(struct Cursor)))) {
if ((c->shared = _mapshared(sizeof(struct CursorShared)))) {
pthread_mutexattr_t attr;
pthread_mutexattr_init(&attr);
pthread_mutexattr_setpshared(&attr, PTHREAD_PROCESS_SHARED);
pthread_mutex_init(&c->shared->lock, &attr);
pthread_mutexattr_destroy(&attr);
c->shared->lock = (pthread_mutex_t)PTHREAD_SHARED_MUTEX_INITIALIZER_NP;
} else {
munmap(c, sizeof(struct Cursor));
c = 0;
@ -56,9 +53,9 @@ int __cursor_unref(struct Cursor *c) {
}
void __cursor_lock(struct Cursor *c) {
pthread_mutex_lock(&c->shared->lock);
_pthread_mutex_lock(&c->shared->lock);
}
void __cursor_unlock(struct Cursor *c) {
pthread_mutex_unlock(&c->shared->lock);
_pthread_mutex_unlock(&c->shared->lock);
}

View file

@ -18,9 +18,9 @@ struct CxaAtexitBlocks {
extern struct CxaAtexitBlocks __cxa_blocks;
void __cxa_lock(void) libcesque;
void __cxa_unlock(void) libcesque;
void __cxa_thread_finalize(void) libcesque;
void __cxa_lock(void) dontthrow;
void __cxa_unlock(void) dontthrow;
void __cxa_thread_finalize(void) dontthrow;
void __cxa_printexits(FILE *, void *) libcesque;
int __cxa_thread_atexit_impl(void *, void *, void *);

View file

@ -17,22 +17,15 @@
PERFORMANCE OF THIS SOFTWARE.
*/
#include "libc/intrin/cxaatexit.h"
#include "libc/thread/posixthread.internal.h"
#include "libc/thread/thread.h"
static pthread_mutex_t __cxa_lock_obj;
void __cxa_wipe(void) {
pthread_mutex_init(&__cxa_lock_obj, 0);
}
pthread_mutex_t __cxa_lock_obj = PTHREAD_MUTEX_INITIALIZER;
void __cxa_lock(void) {
pthread_mutex_lock(&__cxa_lock_obj);
_pthread_mutex_lock(&__cxa_lock_obj);
}
void __cxa_unlock(void) {
pthread_mutex_unlock(&__cxa_lock_obj);
}
__attribute__((__constructor__(60))) static textstartup void __cxa_init() {
pthread_atfork(__cxa_lock, __cxa_unlock, __cxa_wipe);
_pthread_mutex_unlock(&__cxa_lock_obj);
}

277
libc/intrin/deadlock.c Normal file
View file

@ -0,0 +1,277 @@
/*-*- mode:c;indent-tabs-mode:nil;c-basic-offset:2;tab-width:8;coding:utf-8 -*-│
vi: set et ft=c ts=2 sts=2 sw=2 fenc=utf-8 :vi
Copyright 2024 Justine Alexandra Roberts Tunney
Permission to use, copy, modify, and/or distribute this software for
any purpose with or without fee is hereby granted, provided that the
above copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL
WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE
AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL
DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR
PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER
TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
PERFORMANCE OF THIS SOFTWARE.
*/
#include "ape/sections.internal.h"
#include "libc/assert.h"
#include "libc/atomic.h"
#include "libc/cosmo.h"
#include "libc/dce.h"
#include "libc/errno.h"
#include "libc/intrin/atomic.h"
#include "libc/intrin/kprintf.h"
#include "libc/intrin/maps.h"
#include "libc/macros.h"
#include "libc/str/str.h"
#include "libc/thread/lock.h"
#include "libc/thread/thread.h"
#include "libc/thread/tls.h"
/**
* @fileoverview deadlock detector for statically allocated locks
*
* This module helps you spot multi-threading bugs in your program.
* High-level abstractions like mutexes are much easier to use than
* atomics, but they still carry their own non-obvious dangers. For
* example, nesting locks need to be nested in a consistent way and
* normal mutexes can't be required recursively. Normally this will
* cause your program to deadlock, i.e. hang indefinitely, but this
* module can detect such conditions and return errors instead, and
* better yet print helpful information when using `cosmocc -mdbg`.
*/
#define ABI privileged optimizesize
// building our visitor function using this optimizesize keyword shrinks
// the stack memory requirement from 7168 to 2048 bytes. totally amazing
// although please note this maximum isn't a hard limit. for normal mode
// builds your posix mandated mutex error checking will be less accurate
// but still helpful and reliable, although your cosmocc -mdbg will trap
// and report that you've run into the limit, so you can talk to justine
#define MAX_LOCKS 64
// cosmo's tib reserves space for 64 nested locks before things degrade.
// the cosmopolitan c runtime defines 16 locks, which are all registered
// with pthread_atfork(). it means you get to have 48 mutexes right now,
// and if you register all of them, then calling fork() will cause there
// to be 2080 edges in your lock graph. talk to justine if you need more
// because we're obviously going to need to find a way to make this grow
#define LOCK_EDGES_MAX 2080
// supported lock objects must define `void *_edges`
#define LOCK_EDGES_OFFSET 0
static_assert(offsetof(struct MapLock, edges) == LOCK_EDGES_OFFSET);
static_assert(offsetof(pthread_mutex_t, _edges) == LOCK_EDGES_OFFSET);
struct LockEdge {
struct LockEdge *next;
void *dest;
};
struct VisitedLock {
struct VisitedLock *next;
void *lock;
};
typedef _Atomic(struct LockEdge *) LockEdges;
static struct DeadlockDetector {
atomic_size_t edges_allocated;
struct LockEdge edges_memory[LOCK_EDGES_MAX];
} __deadlock;
forceinline struct CosmoTib *__deadlock_tls(void) {
return __get_tls_privileged();
}
forceinline LockEdges *get_lock_edges(void *lock) {
return (LockEdges *)((char *)lock + LOCK_EDGES_OFFSET);
}
forceinline struct LockEdge *load_lock_edges(LockEdges *edges) {
return atomic_load_explicit(edges, memory_order_relaxed);
}
ABI static int is_static_memory(void *lock) {
return _etext <= (unsigned char *)lock && (unsigned char *)lock < _end;
}
ABI static struct LockEdge *__deadlock_alloc(void) {
size_t edges_allocated =
atomic_load_explicit(&__deadlock.edges_allocated, memory_order_relaxed);
for (;;) {
if (edges_allocated == LOCK_EDGES_MAX) {
if (IsModeDbg()) {
kprintf("error: cosmo LOCK_EDGES_MAX needs to be increased\n");
DebugBreak();
}
return 0;
}
if (atomic_compare_exchange_weak_explicit(
&__deadlock.edges_allocated, &edges_allocated, edges_allocated + 1,
memory_order_relaxed, memory_order_relaxed))
return &__deadlock.edges_memory[edges_allocated];
}
}
ABI static void __deadlock_add_edge(void *from, void *dest) {
LockEdges *edges = get_lock_edges(from);
for (struct LockEdge *e = load_lock_edges(edges); e; e = e->next)
if (e->dest == dest)
return;
struct LockEdge *edge;
if ((edge = __deadlock_alloc())) {
edge->next = load_lock_edges(edges);
edge->dest = dest;
// we tolerate duplicate elements in the interest of performance.
// once an element is inserted, it's never removed. that's why we
// don't need need to worry about the aba problem. the cas itself
// is very important since it ensures inserted edges aren't lost.
for (;;)
if (atomic_compare_exchange_weak_explicit(edges, &edge->next, edge,
memory_order_relaxed,
memory_order_relaxed))
break;
}
}
ABI static bool __deadlock_visit(void *lock, struct VisitedLock *visited,
int notrap, int depth) {
if (++depth == MAX_LOCKS) {
if (IsModeDbg()) {
kprintf("error: too much recursion in deadlock detector\n");
DebugBreak();
}
return false;
}
for (struct VisitedLock *v = visited; v; v = v->next) {
if (v->lock == lock) {
if (IsModeDbg() && !notrap) {
// lock hierarchy violated!
//
// when you lock mutexes in a nested way, your locks must be
// nested in the same order globally. otherwise deadlocks might
// occur. for example, if you say in your first thread
//
// pthread_mutex_lock(&x);
// pthread_mutex_lock(&y);
// pthread_mutex_unlock(&y);
// pthread_mutex_unlock(&x);
//
// then in your second thread you say
//
// pthread_mutex_lock(&y);
// pthread_mutex_lock(&x);
// pthread_mutex_unlock(&x);
// pthread_mutex_unlock(&y);
//
// then a deadlock might happen, because {x→y, y→x} is cyclic!
// they don't happen often, but this is the kind of thing that
// matters if you want to build carrier grade production stuff
kprintf("error: cycle detected in directed graph of nested locks\n");
for (struct VisitedLock *v = visited; v; v = v->next)
kprintf("\t- %t\n", v->lock); // strongly connected component
DebugBreak();
}
return true;
}
}
LockEdges *edges = get_lock_edges(lock);
struct VisitedLock visit = {visited, lock};
for (struct LockEdge *e = load_lock_edges(edges); e; e = e->next)
if (__deadlock_visit(e->dest, &visit, notrap, depth))
return true;
return false;
}
/**
* Returns true if lock is already locked by calling thread.
*
* This function may return false negatives if we run out of TLS memory.
* That suboptimal condition will be reported in debug mode.
*
* @return 1 if lock is certainly owned by calling thread, 0 if lock is
* certainly not owned by calling thread, and -1 if we're uncertain
*/
ABI int __deadlock_tracked(void *lock) {
int full = 1;
int owned = 0;
struct CosmoTib *tib = __deadlock_tls();
for (int i = 0; i < ARRAYLEN(tib->tib_locks); ++i) {
full &= tib->tib_locks[i] != NULL;
owned |= tib->tib_locks[i] == lock;
}
if (full)
return -1;
if (!owned && !is_static_memory(lock))
return -1;
return owned;
}
/**
* Records that lock is held by thread.
* @param notrap can prevent error printing and debug breaking
* @asyncsignalsafe
*/
ABI void __deadlock_track(void *lock, int notrap) {
if (!notrap && !is_static_memory(lock))
return;
struct CosmoTib *tib = __deadlock_tls();
for (int i = 0; i < ARRAYLEN(tib->tib_locks); ++i) {
if (!tib->tib_locks[i]) {
tib->tib_locks[i] = lock;
return;
}
}
if (IsModeDbg()) {
kprintf("error: cosmo tls max lock depth needs to be increased!\n");
DebugBreak();
}
}
/**
* Records relationship for all held locks to `lock`.
* @param notrap can prevent error printing and debug breaking
* @asyncsignalsafe
*/
ABI void __deadlock_record(void *lock, int notrap) {
if (!notrap && !is_static_memory(lock))
return;
struct CosmoTib *tib = __deadlock_tls();
for (int i = 0; i < ARRAYLEN(tib->tib_locks); ++i)
if (tib->tib_locks[i] && tib->tib_locks[i] != lock)
__deadlock_add_edge(tib->tib_locks[i], lock);
}
/**
* Returns EDEADLK if locking `lock` could cause a deadlock.
* @param notrap can prevent error printing and debug breaking
* @asyncsignalsafe
*/
ABI int __deadlock_check(void *lock, int notrap) {
struct CosmoTib *tib = __deadlock_tls();
for (int i = 0; i < ARRAYLEN(tib->tib_locks); ++i) {
if (tib->tib_locks[i] == lock)
return 0;
if (tib->tib_locks[i]) {
struct VisitedLock visit = {0, tib->tib_locks[i]};
if (__deadlock_visit(lock, &visit, notrap, 0))
return EDEADLK;
}
}
return 0;
}
/**
* Records that lock isn't held by thread.
* @asyncsignalsafe
*/
ABI void __deadlock_untrack(void *lock) {
struct CosmoTib *tib = __deadlock_tls();
for (int i = 0; i < ARRAYLEN(tib->tib_locks); ++i)
tib->tib_locks[i] = tib->tib_locks[i] != lock ? tib->tib_locks[i] : 0;
}

View file

@ -91,6 +91,8 @@ Copyright (c) 2024 Justine Tunney <jtunney@gmail.com>");
*
*/
#define ABI privileged optimizesize
#define DEMANGLE_NO_FLOATING_POINT
#define ASSERT(x) (void)0
@ -103,6 +105,7 @@ Copyright (c) 2024 Justine Tunney <jtunney@gmail.com>");
#define ELFTC_SUCCESS 1
#define VECTOR_DEF_CAPACITY 1
#define MAX_DEPTH 20
typedef unsigned short index_t;
@ -188,6 +191,7 @@ struct demangle_data {
enum type_qualifier ref_qualifier_type; /* ref qualifier type */
enum push_qualifier push_qualifier; /* which qualifiers to push */
int func_type;
int depth;
const char *cur; /* current mangled name ptr */
const char *last_sname; /* last source name */
intptr_t jmpbuf[5];
@ -220,16 +224,18 @@ static int demangle_read_sname(struct demangle_data *);
static int demangle_read_subst(struct demangle_data *);
static int demangle_read_type(struct demangle_data *, struct type_delimit *);
static privileged size_t
ABI static size_t
demangle_strlen(const char *s)
{
size_t n = 0;
while (*s++)
while (*s++) {
asm volatile("" ::: "memory");
++n;
}
return n;
}
static privileged char *
ABI static char *
demangle_stpcpy(char *d, const char *s)
{
size_t i = 0;
@ -240,7 +246,7 @@ demangle_stpcpy(char *d, const char *s)
}
}
static privileged void *
ABI static void *
demangle_mempcpy(void *a, const void *b, size_t n)
{
char *d = a;
@ -250,14 +256,14 @@ demangle_mempcpy(void *a, const void *b, size_t n)
return d;
}
static privileged void *
ABI static void *
demangle_memcpy(void *a, const void *b, size_t n)
{
demangle_mempcpy(a, b, n);
return a;
}
static privileged int
ABI static int
demangle_strncmp(const char *a, const char *b, size_t n)
{
size_t i = 0;
@ -268,7 +274,7 @@ demangle_strncmp(const char *a, const char *b, size_t n)
return (a[i] & 0xff) - (b[i] & 0xff);
}
static privileged int
ABI static int
demangle_memcmp(const void *a, const void *b, size_t n)
{
int c;
@ -283,7 +289,7 @@ demangle_memcmp(const void *a, const void *b, size_t n)
return 0;
}
static privileged void
ABI static void
demangle_strlcpy(char *dst, const char *src, size_t dsize)
{
size_t remain;
@ -295,7 +301,7 @@ demangle_strlcpy(char *dst, const char *src, size_t dsize)
*dst = 0;
}
static privileged long
ABI static long
demangle_strtol(const char *s, int base)
{
static const uint8_t demangle_base36[80] = { 1, 2, 3, 4, 5, 6, 7, 8, 9,
@ -312,7 +318,7 @@ demangle_strtol(const char *s, int base)
return x;
}
static privileged char *
ABI static char *
demangle_strstr(const char *haystack, const char *needle)
{
size_t i;
@ -333,7 +339,7 @@ demangle_strstr(const char *haystack, const char *needle)
return 0;
}
static privileged char *
ABI static char *
demangle_utoa(char *p, unsigned long long x)
{
char t;
@ -354,7 +360,7 @@ demangle_utoa(char *p, unsigned long long x)
return p + i;
}
static privileged char *
ABI static char *
demangle_itoa(char *p, long long x)
{
if (x < 0)
@ -362,7 +368,7 @@ demangle_itoa(char *p, long long x)
return demangle_utoa(p, x);
}
static privileged void
ABI static void
demangle_free(struct demangle_data *h, void *ptr)
{
index_t base;
@ -379,14 +385,17 @@ demangle_free(struct demangle_data *h, void *ptr)
}
}
static privileged returnspointerwithnoaliases returnsnonnull void *
demangle_malloc(struct demangle_data *h, int a, int n)
ABI static returnspointerwithnoaliases returnsnonnull void *
demangle_malloc(struct demangle_data *h, long a, long n)
{
int rem;
long rem;
uintptr_t ptr;
index_t next, next2;
index_t *link, *link2;
int b = sizeof(index_t);
long b = sizeof(index_t);
if (n < 0 || n >= 32768)
__builtin_longjmp(h->jmpbuf, 1);
/* Roundup size. */
n += a - 1;
@ -433,7 +442,7 @@ demangle_malloc(struct demangle_data *h, int a, int n)
}
}
static privileged returnspointerwithnoaliases char *
ABI static returnspointerwithnoaliases char *
demangle_strdup(struct demangle_data *h, const char *s)
{
char *d = 0;
@ -445,7 +454,7 @@ demangle_strdup(struct demangle_data *h, const char *s)
return d;
}
static privileged void
ABI static void
demangle_vector_str_dest(struct demangle_data *h, struct vector_str *v)
{
int i;
@ -454,7 +463,7 @@ demangle_vector_str_dest(struct demangle_data *h, struct vector_str *v)
demangle_free(h, v->container);
}
static privileged void
ABI static void
demangle_vector_type_qualifier_dest(struct demangle_data *d,
struct vector_type_qualifier *v)
{
@ -462,7 +471,7 @@ demangle_vector_type_qualifier_dest(struct demangle_data *d,
demangle_vector_str_dest(d, &v->ext_name);
}
static privileged void
ABI static void
demangle_stack_str_init(struct stack_str *ss)
{
ss->str = ss->buf;
@ -471,7 +480,7 @@ demangle_stack_str_init(struct stack_str *ss)
ss->cap = sizeof(ss->buf);
}
static privileged void
ABI static void
demangle_stack_str_append(struct demangle_data *h, struct stack_str *ss,
const char *str, size_t len)
{
@ -494,7 +503,7 @@ demangle_stack_str_append(struct demangle_data *h, struct stack_str *ss,
#define demangle_stack_str_append_str(h, ss, s) \
demangle_stack_str_append(h, ss, s, demangle_strlen(s))
static privileged size_t
ABI static size_t
demangle_get_strlen_sum(struct demangle_data *h, const struct vector_str *v)
{
size_t i, len = 0;
@ -504,7 +513,7 @@ demangle_get_strlen_sum(struct demangle_data *h, const struct vector_str *v)
return len;
}
static privileged int
ABI static int
demangle_demangle_strncmp(const char *a, const char *b, size_t n)
{
size_t i = 0;
@ -522,7 +531,7 @@ demangle_demangle_strncmp(const char *a, const char *b, size_t n)
* @param l Length of the string.
* @return -1 at failed, 0 at not found, 1 at found.
*/
static privileged int
ABI static int
demangle_vector_str_find(struct demangle_data *h, const struct vector_str *v,
const char *o, size_t l)
{
@ -546,7 +555,7 @@ demangle_vector_str_find(struct demangle_data *h, const struct vector_str *v,
* @param l Length of the string.
* @return NULL at failed or NUL terminated new allocated string.
*/
static privileged char *
ABI static char *
demangle_vector_str_get_flat(struct demangle_data *ddata,
const struct vector_str *v, size_t *l)
{
@ -572,7 +581,7 @@ demangle_vector_str_get_flat(struct demangle_data *ddata,
return rtn;
}
static privileged void
ABI static void
demangle_vector_str_grow(struct demangle_data *ddata, struct vector_str *v)
{
size_t i, tmp_cap;
@ -600,7 +609,7 @@ demangle_vector_str_grow(struct demangle_data *ddata, struct vector_str *v)
* @brief Initialize vector_str.
* @return false at failed, true at success.
*/
static privileged void
ABI static void
demangle_vector_str_init(struct demangle_data *ddata, struct vector_str *v)
{
v->size = 0;
@ -616,7 +625,7 @@ demangle_vector_str_init(struct demangle_data *ddata, struct vector_str *v)
* @brief Remove last element in vector_str.
* @return false at failed, true at success.
*/
static privileged bool
ABI static bool
demangle_vector_str_pop(struct vector_str *v)
{
if (!v)
@ -636,7 +645,7 @@ demangle_vector_str_pop(struct vector_str *v)
* @brief Push back string to vector.
* @return false at failed, true at success.
*/
static privileged bool
ABI static bool
demangle_vector_str_push(struct demangle_data *ddata, struct vector_str *v,
const char *str, size_t len)
{
@ -660,7 +669,7 @@ demangle_vector_str_push(struct demangle_data *ddata, struct vector_str *v,
* @brief Push front org vector to det vector.
* @return false at failed, true at success.
*/
static privileged bool
ABI static bool
demangle_vector_str_push_vector_head(struct demangle_data *ddata,
struct vector_str *dst, struct vector_str *org)
{
@ -693,7 +702,7 @@ demangle_vector_str_push_vector_head(struct demangle_data *ddata,
* @brief Push org vector to the tail of det vector.
* @return false at failed, true at success.
*/
static privileged bool
ABI static bool
demangle_vector_str_push_vector(struct demangle_data *ddata,
struct vector_str *dst, struct vector_str *org)
{
@ -731,7 +740,7 @@ demangle_vector_str_push_vector(struct demangle_data *ddata,
* If r_len is not NULL, string length will be returned.
* @return NULL at failed or NUL terminated new allocated string.
*/
static privileged returnspointerwithnoaliases char *
ABI static returnspointerwithnoaliases char *
demangle_vector_str_substr(struct demangle_data *ddata,
const struct vector_str *v, size_t begin, size_t end, size_t *r_len)
{
@ -757,7 +766,7 @@ demangle_vector_str_substr(struct demangle_data *ddata,
return rtn;
}
static privileged int
ABI static int
demangle_vector_read_cmd_pop(struct vector_read_cmd *v)
{
if (!v->size)
@ -770,7 +779,7 @@ demangle_vector_read_cmd_pop(struct vector_read_cmd *v)
return 1;
}
static privileged void
ABI static void
demangle_vector_read_cmd_init(struct demangle_data *ddata,
struct vector_read_cmd *v)
{
@ -781,7 +790,7 @@ demangle_vector_read_cmd_init(struct demangle_data *ddata,
alignof(*v->r_container), sizeof(*v->r_container) * v->capacity);
}
static privileged void
ABI static void
demangle_data_init(struct demangle_data *d, const char *cur)
{
demangle_vector_str_init(d, &d->output);
@ -811,7 +820,7 @@ demangle_data_init(struct demangle_data *d, const char *cur)
d->last_sname = NULL;
}
static privileged int
ABI static int
demangle_push_str(struct demangle_data *ddata, const char *str, size_t len)
{
if (!str || !len)
@ -828,7 +837,7 @@ demangle_push_str(struct demangle_data *ddata, const char *str, size_t len)
}
#ifndef DEMANGLE_NO_FLOATING_POINT
static privileged int
ABI static int
demangle_push_fp(struct demangle_data *ddata,
char *decoder(struct demangle_data *, const char *, size_t))
{
@ -857,13 +866,13 @@ demangle_push_fp(struct demangle_data *ddata,
}
#endif // DEMANGLE_NO_FLOATING_POINT
static privileged int
ABI static int
demangle_pop_str(struct demangle_data *ddata)
{
return demangle_vector_str_pop(ddata->cur_output);
}
static privileged int
ABI static int
demangle_push_subst(struct demangle_data *ddata, const char *str, size_t len)
{
if (!str || !len)
@ -875,7 +884,7 @@ demangle_push_subst(struct demangle_data *ddata, const char *str, size_t len)
return 1;
}
static privileged int
ABI static int
demangle_push_subst_v(struct demangle_data *ddata, struct vector_str *v)
{
int rtn;
@ -895,7 +904,7 @@ demangle_push_subst_v(struct demangle_data *ddata, struct vector_str *v)
return rtn;
}
static privileged int
ABI static int
demangle_push_type_qualifier(struct demangle_data *ddata,
struct vector_type_qualifier *v, const char *type_str)
{
@ -1128,7 +1137,7 @@ demangle_push_type_qualifier(struct demangle_data *ddata,
return 1;
}
static privileged int
ABI static int
demangle_get_subst(struct demangle_data *ddata, size_t idx)
{
size_t len;
@ -1146,7 +1155,7 @@ demangle_get_subst(struct demangle_data *ddata, size_t idx)
return 1;
}
static privileged int
ABI static int
demangle_get_tmpl_param(struct demangle_data *ddata, size_t idx)
{
size_t len;
@ -1163,7 +1172,7 @@ demangle_get_tmpl_param(struct demangle_data *ddata, size_t idx)
return 1;
}
static privileged int
ABI static int
demangle_read_array(struct demangle_data *ddata)
{
size_t i, num_len, exp_len, p_idx, idx;
@ -1235,7 +1244,7 @@ demangle_read_array(struct demangle_data *ddata)
#ifndef DEMANGLE_NO_FLOATING_POINT
/* Simple hex to integer function used by decode_to_* function. */
static privileged int
ABI static int
hex_to_dec(char c)
{
switch (c) {
@ -1283,7 +1292,7 @@ hex_to_dec(char c)
* Todo
* Replace these functions to macro.
*/
static privileged returnspointerwithnoaliases char *
ABI static returnspointerwithnoaliases char *
decode_fp_to_double(struct demangle_data *ddata, const char *p, size_t len)
{
double f;
@ -1327,7 +1336,7 @@ again:
return rtn;
}
static privileged returnspointerwithnoaliases char *
ABI static returnspointerwithnoaliases char *
decode_fp_to_float(struct demangle_data *ddata, const char *p, size_t len)
{
size_t i, rtn_len, limit;
@ -1369,7 +1378,7 @@ again:
return rtn;
}
static privileged returnspointerwithnoaliases char *
ABI static returnspointerwithnoaliases char *
decode_fp_to_long_double(struct demangle_data *ddata, const char *p, size_t len)
{
long double f;
@ -1413,7 +1422,7 @@ again:
return rtn;
}
static privileged returnspointerwithnoaliases char *
ABI static returnspointerwithnoaliases char *
decode_fp_to_float128(struct demangle_data *ddata, const char *p, size_t len)
{
long double f;
@ -1470,7 +1479,7 @@ decode_fp_to_float128(struct demangle_data *ddata, const char *p, size_t len)
}
}
static privileged returnspointerwithnoaliases char *
ABI static returnspointerwithnoaliases char *
decode_fp_to_float80(struct demangle_data *ddata, const char *p, size_t len)
{
long double f;
@ -1533,7 +1542,7 @@ decode_fp_to_float80(struct demangle_data *ddata, const char *p, size_t len)
#endif // DEMANGLE_NO_FLOATING_POINT
static privileged int
ABI static int
demangle_read_expr_primary(struct demangle_data *ddata)
{
const char *num;
@ -1625,7 +1634,7 @@ demangle_read_expr_primary(struct demangle_data *ddata)
* http://gcc.gnu.org/bugzilla/show_bug.cgi?id=31775
* http://gcc.gnu.org/viewcvs?view=rev&revision=124467
*/
static privileged int
ABI static int
demangle_local_source_name(struct demangle_data *ddata)
{
/* L */
@ -1651,7 +1660,7 @@ demangle_local_source_name(struct demangle_data *ddata)
* read unqualified-name, unqualified name are operator-name, ctor-dtor-name,
* source-name
*/
static privileged int
ABI static int
demangle_read_uqname(struct demangle_data *ddata)
{
size_t len;
@ -2080,7 +2089,7 @@ demangle_read_uqname(struct demangle_data *ddata)
* Read template parameter that forms in 'T[number]_'.
* This function much like to read_subst but only for types.
*/
static privileged int
ABI static int
demangle_read_tmpl_param(struct demangle_data *ddata)
{
long nth;
@ -2098,10 +2107,11 @@ demangle_read_tmpl_param(struct demangle_data *ddata)
/* T_ is first */
++nth;
while (*ddata->cur != '_')
while (*ddata->cur && *ddata->cur != '_')
++ddata->cur;
ASSERT(nth > 0);
if (nth <= 0)
return 0;
return demangle_get_tmpl_param(ddata, nth);
}
@ -2110,7 +2120,7 @@ demangle_read_tmpl_param(struct demangle_data *ddata)
return 0;
}
static privileged int
ABI static int
demangle_vector_read_cmd_push(struct demangle_data *ddata,
struct vector_read_cmd *v, enum read_cmd cmd, void *data)
{
@ -2139,7 +2149,7 @@ demangle_vector_read_cmd_push(struct demangle_data *ddata,
return 1;
}
static privileged int
ABI static int
demangle_read_tmpl_arg(struct demangle_data *ddata)
{
if (*ddata->cur == '\0')
@ -2158,7 +2168,7 @@ demangle_read_tmpl_arg(struct demangle_data *ddata)
return demangle_read_type(ddata, NULL);
}
static privileged int
ABI static int
demangle_read_tmpl_args(struct demangle_data *ddata)
{
struct vector_str *v;
@ -2211,7 +2221,7 @@ demangle_read_tmpl_args(struct demangle_data *ddata)
return demangle_vector_read_cmd_pop(&ddata->cmd);
}
static privileged int
ABI static int
demangle_read_expression_trinary(struct demangle_data *ddata, const char *name1,
size_t len1, const char *name2, size_t len2)
{
@ -2230,7 +2240,7 @@ demangle_read_expression_trinary(struct demangle_data *ddata, const char *name1,
return demangle_read_expression(ddata);
}
static privileged int
ABI static int
demangle_read_expression_unary(struct demangle_data *ddata, const char *name,
size_t len)
{
@ -2242,7 +2252,7 @@ demangle_read_expression_unary(struct demangle_data *ddata, const char *name,
return demangle_push_str(ddata, name, len);
}
static privileged int
ABI static int
demangle_read_expression_binary(struct demangle_data *ddata, const char *name,
size_t len)
{
@ -2256,8 +2266,8 @@ demangle_read_expression_binary(struct demangle_data *ddata, const char *name,
return demangle_read_expression(ddata);
}
static privileged int
demangle_read_expression(struct demangle_data *ddata)
ABI static int
demangle_read_expression_impl(struct demangle_data *ddata)
{
if (*ddata->cur == '\0')
return 0;
@ -2538,7 +2548,18 @@ demangle_read_expression(struct demangle_data *ddata)
return 0;
}
static privileged int
ABI static int
demangle_read_expression(struct demangle_data *ddata)
{
if (ddata->depth == MAX_DEPTH)
__builtin_longjmp(ddata->jmpbuf, 1);
++ddata->depth;
int res = demangle_read_expression_impl(ddata);
--ddata->depth;
return res;
}
ABI static int
demangle_read_expression_flat(struct demangle_data *ddata, char **str)
{
struct vector_str *output;
@ -2567,7 +2588,7 @@ demangle_read_expression_flat(struct demangle_data *ddata, char **str)
}
/* size, capacity, ext_name */
static privileged void
ABI static void
demangle_vector_type_qualifier_init(struct demangle_data *ddata,
struct vector_type_qualifier *v)
{
@ -2583,7 +2604,7 @@ demangle_vector_type_qualifier_init(struct demangle_data *ddata,
demangle_vector_str_init(ddata, &v->ext_name);
}
static privileged struct read_cmd_item *
ABI static struct read_cmd_item *
demangle_vector_read_cmd_find(struct vector_read_cmd *v, enum read_cmd dst)
{
int i;
@ -2598,7 +2619,7 @@ demangle_vector_read_cmd_find(struct vector_read_cmd *v, enum read_cmd dst)
return 0;
}
static privileged int
ABI static int
demangle_read_function(struct demangle_data *ddata, int *ext_c,
struct vector_type_qualifier *v)
{
@ -2734,7 +2755,7 @@ demangle_read_function(struct demangle_data *ddata, int *ext_c,
return 1;
}
static privileged int
ABI static int
demangle_read_offset_number(struct demangle_data *ddata)
{
bool negative;
@ -2752,7 +2773,7 @@ demangle_read_offset_number(struct demangle_data *ddata)
start = ddata->cur;
}
while (*ddata->cur != '_')
while (*ddata->cur && *ddata->cur != '_')
++ddata->cur;
if (negative && !DEM_PUSH_STR(ddata, "-"))
@ -2770,7 +2791,7 @@ demangle_read_offset_number(struct demangle_data *ddata)
return 1;
}
static privileged int
ABI static int
demangle_read_nv_offset(struct demangle_data *ddata)
{
if (!DEM_PUSH_STR(ddata, "offset : "))
@ -2779,7 +2800,7 @@ demangle_read_nv_offset(struct demangle_data *ddata)
return demangle_read_offset_number(ddata);
}
static privileged int
ABI static int
demangle_read_v_offset(struct demangle_data *ddata)
{
if (!DEM_PUSH_STR(ddata, "offset : "))
@ -2795,7 +2816,7 @@ demangle_read_v_offset(struct demangle_data *ddata)
}
/* read offset, offset are nv-offset, v-offset */
static privileged int
ABI static int
demangle_read_offset(struct demangle_data *ddata)
{
if (*ddata->cur == 'h') {
@ -2809,7 +2830,7 @@ demangle_read_offset(struct demangle_data *ddata)
return 0;
}
static privileged int
ABI static int
demangle_read_type_flat(struct demangle_data *ddata, char **str)
{
struct vector_str *output;
@ -2841,7 +2862,7 @@ demangle_read_type_flat(struct demangle_data *ddata, char **str)
* read number
* number ::= [n] <decimal>
*/
static privileged int
ABI static int
demangle_read_number(struct demangle_data *ddata, long *rtn)
{
long len, negative_factor;
@ -2859,19 +2880,18 @@ demangle_read_number(struct demangle_data *ddata, long *rtn)
return 0;
len = demangle_strtol(ddata->cur, 10);
if (len < 0)
__builtin_longjmp(ddata->jmpbuf, 1);
while (ELFTC_ISDIGIT(*ddata->cur))
++ddata->cur;
ASSERT(len >= 0);
ASSERT(negative_factor == 1 || negative_factor == -1);
*rtn = len * negative_factor;
return 1;
}
static privileged int
ABI static int
demangle_read_number_as_string(struct demangle_data *ddata, char **str)
{
long n;
@ -2888,9 +2908,8 @@ demangle_read_number_as_string(struct demangle_data *ddata, char **str)
return 1;
}
/* read encoding, encoding are function name, data name, special-name */
static privileged int
demangle_read_encoding(struct demangle_data *ddata)
ABI static int
demangle_read_encoding_impl(struct demangle_data *ddata)
{
char *name, *type, *num_str;
long offset;
@ -3097,7 +3116,19 @@ demangle_read_encoding(struct demangle_data *ddata)
return demangle_read_name(ddata);
}
static privileged int
/* read encoding, encoding are function name, data name, special-name */
ABI static int
demangle_read_encoding(struct demangle_data *ddata)
{
if (ddata->depth == MAX_DEPTH)
__builtin_longjmp(ddata->jmpbuf, 1);
++ddata->depth;
int res = demangle_read_encoding_impl(ddata);
--ddata->depth;
return res;
}
ABI static int
demangle_read_local_name(struct demangle_data *ddata)
{
struct vector_str local_name;
@ -3178,7 +3209,7 @@ demangle_read_local_name(struct demangle_data *ddata)
return 1;
}
static privileged int
ABI static int
demangle_read_nested_name(struct demangle_data *ddata)
{
struct stack_str v;
@ -3266,8 +3297,8 @@ next:
return 1;
}
static privileged int
demangle_read_name(struct demangle_data *ddata)
ABI static int
demangle_read_name_impl(struct demangle_data *ddata)
{
struct stack_str v;
struct vector_str *output;
@ -3328,7 +3359,18 @@ clean:
return rtn;
}
static privileged int
ABI static int
demangle_read_name(struct demangle_data *ddata)
{
if (ddata->depth == MAX_DEPTH)
__builtin_longjmp(ddata->jmpbuf, 1);
++ddata->depth;
int res = demangle_read_name_impl(ddata);
--ddata->depth;
return res;
}
ABI static int
demangle_read_name_flat(struct demangle_data *ddata, char **str)
{
struct vector_str *output;
@ -3356,7 +3398,7 @@ demangle_read_name_flat(struct demangle_data *ddata, char **str)
return 1;
}
static privileged int
ABI static int
demangle_read_pointer_to_member(struct demangle_data *ddata,
struct vector_type_qualifier *v)
{
@ -3416,9 +3458,10 @@ clean1:
}
/* read source-name, source-name is <len> <ID> */
static privileged int
ABI static int
demangle_read_sname(struct demangle_data *ddata)
{
size_t lim;
long len;
int err;
@ -3438,12 +3481,15 @@ demangle_read_sname(struct demangle_data *ddata)
ddata->last_sname = VEC_STR(ddata, ddata->cur_output,
ddata->cur_output->size - 1);
lim = demangle_strlen(ddata->cur);
if (len > lim)
len = lim;
ddata->cur += len;
return 1;
}
static privileged int
ABI static int
demangle_read_subst_stdtmpl(struct demangle_data *ddata, const char *str)
{
struct vector_str *output;
@ -3481,7 +3527,7 @@ demangle_read_subst_stdtmpl(struct demangle_data *ddata, const char *str)
return 1;
}
static privileged int
ABI static int
demangle_read_subst_std(struct demangle_data *ddata)
{
struct vector_str *output, v;
@ -3532,7 +3578,7 @@ demangle_read_subst_std(struct demangle_data *ddata)
return 1;
}
static privileged int
ABI static int
demangle_read_subst(struct demangle_data *ddata)
{
long nth;
@ -3647,10 +3693,11 @@ demangle_read_subst(struct demangle_data *ddata)
/* first was '_', so increase one */
++nth;
while (*ddata->cur != '_')
while (*ddata->cur && *ddata->cur != '_')
++ddata->cur;
ASSERT(nth > 0);
if (nth <= 0)
return 0;
return demangle_get_subst(ddata, nth);
}
@ -3659,7 +3706,7 @@ demangle_read_subst(struct demangle_data *ddata)
return 0;
}
static privileged int
ABI static int
demangle_vector_type_qualifier_push(struct demangle_data *ddata,
struct vector_type_qualifier *v, enum type_qualifier t)
{
@ -3688,8 +3735,8 @@ demangle_vector_type_qualifier_push(struct demangle_data *ddata,
return 1;
}
static privileged int
demangle_read_type(struct demangle_data *ddata, struct type_delimit *td)
ABI static int
demangle_read_type_impl(struct demangle_data *ddata, struct type_delimit *td)
{
struct vector_type_qualifier v;
struct vector_str *output, sv;
@ -3881,7 +3928,7 @@ again:
case 'E':
/* unexpected end (except some things) */
if (ddata->is_guard_variable)
if (td && ddata->is_guard_variable)
td->paren = false;
if (ddata->is_guard_variable ||
(ddata->ref_qualifier && ddata->is_functype)) {
@ -4102,6 +4149,8 @@ again:
if (!demangle_vector_str_push(ddata, &v.ext_name, ddata->cur,
len))
return 0;
if (len > demangle_strlen(ddata->cur))
len = demangle_strlen(ddata->cur);
ddata->cur += len;
if (!demangle_vector_type_qualifier_push(ddata, &v, TYPE_EXT))
return 0;
@ -4209,7 +4258,18 @@ clean:
return 0;
}
static privileged int
ABI static int
demangle_read_type(struct demangle_data *ddata, struct type_delimit *td)
{
if (ddata->depth == MAX_DEPTH)
__builtin_longjmp(ddata->jmpbuf, 1);
++ddata->depth;
int res = demangle_read_type_impl(ddata, td);
--ddata->depth;
return res;
}
ABI static int
demangle_copy_output(struct demangle_data *ddata, char *buf,
const struct vector_str *v, size_t buflen)
{
@ -4232,14 +4292,14 @@ demangle_copy_output(struct demangle_data *ddata, char *buf,
return -1;
}
static privileged int
ABI static int
demangle_failure(char *buf, const char *org, size_t buflen)
{
demangle_strlcpy(buf, org, buflen);
return -1;
}
static privileged int
ABI static int
demangle(struct demangle_data *ddata, char *buf, const char *org, size_t buflen)
{
struct vector_str ret_type;
@ -4391,7 +4451,7 @@ demangle(struct demangle_data *ddata, char *buf, const char *org, size_t buflen)
* @return bytes of output name or -1 upon error or truncation
* @asyncsignalsafe
*/
privileged int
ABI int
__demangle(char *buf, const char *org, size_t buflen)
{
struct demangle_data ddata[1];
@ -4405,7 +4465,7 @@ __demangle(char *buf, const char *org, size_t buflen)
*
* This means it starts with either "_Z" or "_GLOBAL__I_".
*/
privileged int
ABI int
__is_mangled(const char *org)
{
if (!org)

View file

@ -0,0 +1,32 @@
/*-*- mode:c;indent-tabs-mode:nil;c-basic-offset:2;tab-width:8;coding:utf-8 -*-│
vi: set et ft=c ts=2 sts=2 sw=2 fenc=utf-8 :vi
Copyright 2024 Justine Alexandra Roberts Tunney
Permission to use, copy, modify, and/or distribute this software for
any purpose with or without fee is hereby granted, provided that the
above copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL
WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE
AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL
DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR
PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER
TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
PERFORMANCE OF THIS SOFTWARE.
*/
#include "libc/intrin/describeflags.h"
#include "libc/macros.h"
#include "libc/nt/enum/memflags.h"
static const struct DescribeFlags kNtAllocationTypeFlags[] = {
{kNtMemCommit, "Commit"}, //
{kNtMemReserve, "Reserve"}, //
{kNtMemReset, "Reset"}, //
};
const char *_DescribeNtAllocationType(char buf[48], uint32_t x) {
return _DescribeFlags(buf, 48, kNtAllocationTypeFlags,
ARRAYLEN(kNtAllocationTypeFlags), "kNtMem", x);
}

View file

@ -24,13 +24,15 @@
#define N 160
privileged static bool IsDangerous(const void *ptr) {
#define ABI privileged optimizesize
ABI static bool IsDangerous(const void *ptr) {
if (_weaken(kisdangerous))
return _weaken(kisdangerous)(ptr);
return false;
}
privileged static char *FormatHex(char *p, unsigned long x) {
ABI static char *FormatHex(char *p, unsigned long x) {
int k = x ? (__builtin_clzl(x) ^ 63) + 1 : 1;
k = (k + 3) & -4;
while (k > 0)
@ -39,8 +41,7 @@ privileged static char *FormatHex(char *p, unsigned long x) {
return p;
}
privileged dontinstrument const char *_DescribeBacktrace(
char buf[N], const struct StackFrame *fr) {
ABI const char *_DescribeBacktrace(char buf[N], const struct StackFrame *fr) {
char *p = buf;
char *pe = p + N;
bool gotsome = false;

View file

@ -29,6 +29,7 @@ const char *_DescribeMapping(char[8], int, int) libcesque;
const char *_DescribeMremapFlags(char[30], int) libcesque;
const char *_DescribeMsg(char[16], int) libcesque;
const char *_DescribeMsyncFlags(char[48], int) libcesque;
const char *_DescribeNtAllocationType(char[48], uint32_t);
const char *_DescribeNtConsoleInFlags(char[256], uint32_t) libcesque;
const char *_DescribeNtConsoleOutFlags(char[128], uint32_t) libcesque;
const char *_DescribeNtCreationDisposition(uint32_t) libcesque;
@ -87,6 +88,7 @@ const char *_DescribeWhichPrio(char[12], int) libcesque;
#define DescribeMremapFlags(x) _DescribeMremapFlags(alloca(30), x)
#define DescribeMsg(x) _DescribeMsg(alloca(16), x)
#define DescribeMsyncFlags(x) _DescribeMsyncFlags(alloca(48), x)
#define DescribeNtAllocationType(x) _DescribeNtAllocationType(alloca(48), x)
#define DescribeNtConsoleInFlags(x) _DescribeNtConsoleInFlags(alloca(256), x)
#define DescribeNtConsoleOutFlags(x) _DescribeNtConsoleOutFlags(alloca(128), x)
#define DescribeNtFileAccessFlags(x) _DescribeNtFileAccessFlags(alloca(512), x)

View file

@ -16,25 +16,29 @@
TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
PERFORMANCE OF THIS SOFTWARE.
*/
#include "libc/dce.h"
#include "libc/intrin/describeflags.h"
#include "libc/macros.h"
#include "libc/nt/enum/consolemodeflags.h"
#include "libc/sysv/consts/map.h"
#include "libc/sysv/consts/prot.h"
#define MAP_GROWSDOWN_LINUX 0x00000100
const char *_DescribeMapFlags(char buf[64], int x) {
const struct DescribeFlags kMapFlags[] = {
{MAP_PRIVATE, "PRIVATE"}, //
{MAP_ANONYMOUS, "ANONYMOUS"}, //
{MAP_SHARED, "SHARED"}, //
{MAP_FIXED, "FIXED"}, //
{MAP_FIXED_NOREPLACE, "FIXED_NOREPLACE"}, //
{MAP_HUGETLB, "HUGETLB"}, //
{MAP_CONCEAL, "CONCEAL"}, //
{MAP_LOCKED, "LOCKED"}, //
{MAP_NORESERVE, "NORESERVE"}, //
{MAP_NONBLOCK, "NONBLOCK"}, //
{MAP_POPULATE, "POPULATE"}, //
{MAP_PRIVATE, "PRIVATE"}, //
{MAP_ANONYMOUS, "ANONYMOUS"}, //
{MAP_SHARED, "SHARED"}, //
{MAP_FIXED, "FIXED"}, //
{MAP_FIXED_NOREPLACE, "FIXED_NOREPLACE"}, //
{MAP_HUGETLB, "HUGETLB"}, //
{MAP_CONCEAL, "CONCEAL"}, //
{MAP_LOCKED, "LOCKED"}, //
{MAP_NORESERVE, "NORESERVE"}, //
{MAP_NONBLOCK, "NONBLOCK"}, //
{MAP_POPULATE, "POPULATE"}, //
{IsLinux() ? MAP_GROWSDOWN_LINUX : 0, "GROWSDOWN"}, //
};
return _DescribeFlags(buf, 64, kMapFlags, ARRAYLEN(kMapFlags), "MAP_", x);
}

View file

@ -17,6 +17,7 @@
PERFORMANCE OF THIS SOFTWARE.
*/
#include "libc/intrin/describeflags.h"
#include "libc/intrin/maps.h"
#include "libc/runtime/memtrack.internal.h"
#include "libc/sysv/consts/map.h"
#include "libc/sysv/consts/prot.h"
@ -24,12 +25,13 @@
static char DescribeMapType(int flags) {
switch (flags & MAP_TYPE) {
case MAP_FILE:
if (flags & MAP_NOFORK)
return 'i'; // executable image
return '-';
case MAP_PRIVATE:
if (flags & MAP_NOFORK)
return 'P';
else
return 'p';
return 'w'; // windows memory
return 'p';
case MAP_SHARED:
return 's';
default:

View file

@ -21,6 +21,8 @@
#include "libc/sysv/consts/prot.h"
const char *_DescribeProtFlags(char buf[48], int x) {
if (!x)
return "PROT_NONE";
const struct DescribeFlags kProtFlags[] = {
{PROT_READ, "READ"}, //
{PROT_WRITE, "WRITE"}, //

View file

@ -19,7 +19,6 @@
#include "libc/calls/calls.h"
#include "libc/calls/internal.h"
#include "libc/calls/metalfile.internal.h"
#include "libc/intrin/directmap.h"
#include "libc/macros.h"
#include "libc/runtime/pc.internal.h"
#include "libc/str/str.h"
@ -32,19 +31,11 @@
static uint64_t sys_mmap_metal_break;
static struct DirectMap bad_mmap(void) {
struct DirectMap res;
res.addr = (void *)-1;
res.maphandle = -1;
return res;
}
struct DirectMap sys_mmap_metal(void *vaddr, size_t size, int prot, int flags,
int fd, int64_t off) {
void *sys_mmap_metal(void *vaddr, size_t size, int prot, int flags, int fd,
int64_t off) {
/* asan runtime depends on this function */
size_t i;
struct mman *mm;
struct DirectMap res;
uint64_t addr, faddr = 0, page, e, *pte, *fdpte, *pml4t;
mm = __get_mm();
pml4t = __get_pml4t();
@ -54,18 +45,18 @@ struct DirectMap sys_mmap_metal(void *vaddr, size_t size, int prot, int flags,
struct Fd *sfd;
struct MetalFile *file;
if (off < 0 || fd < 0 || fd >= g_fds.n)
return bad_mmap();
return MAP_FAILED;
sfd = &g_fds.p[fd];
if (sfd->kind != kFdFile)
return bad_mmap();
return MAP_FAILED;
file = (struct MetalFile *)sfd->handle;
/* TODO: allow mapping partial page at end of file, if file size not
* multiple of page size */
if (off > file->size || size > file->size - off)
return bad_mmap();
return MAP_FAILED;
faddr = (uint64_t)file->base + off;
if (faddr % 4096 != 0)
return bad_mmap();
return MAP_FAILED;
}
if (!(flags & MAP_FIXED_linux)) {
if (!addr) {
@ -88,7 +79,7 @@ struct DirectMap sys_mmap_metal(void *vaddr, size_t size, int prot, int flags,
if ((flags & MAP_ANONYMOUS_linux)) {
page = __new_page(mm);
if (!page)
return bad_mmap();
return MAP_FAILED;
__clear_page(BANE + page);
e = page | PAGE_RSRV | PAGE_U;
if ((prot & PROT_WRITE))
@ -114,9 +105,7 @@ struct DirectMap sys_mmap_metal(void *vaddr, size_t size, int prot, int flags,
break;
}
}
res.addr = (void *)addr;
res.maphandle = -1;
return res;
return (void *)addr;
}
#endif /* __x86_64__ */

View file

@ -1,122 +0,0 @@
/*-*- mode:c;indent-tabs-mode:nil;c-basic-offset:2;tab-width:8;coding:utf-8 -*-│
vi: set et ft=c ts=2 sts=2 sw=2 fenc=utf-8 :vi
Copyright 2020 Justine Alexandra Roberts Tunney
Permission to use, copy, modify, and/or distribute this software for
any purpose with or without fee is hereby granted, provided that the
above copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL
WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE
AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL
DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR
PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER
TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
PERFORMANCE OF THIS SOFTWARE.
*/
#include "libc/assert.h"
#include "libc/calls/internal.h"
#include "libc/calls/state.internal.h"
#include "libc/errno.h"
#include "libc/intrin/directmap.h"
#include "libc/nt/enum/filemapflags.h"
#include "libc/nt/enum/pageflags.h"
#include "libc/nt/errors.h"
#include "libc/nt/memory.h"
#include "libc/nt/runtime.h"
#include "libc/nt/struct/processmemorycounters.h"
#include "libc/nt/struct/securityattributes.h"
#include "libc/sysv/consts/map.h"
#include "libc/sysv/consts/o.h"
#include "libc/sysv/consts/prot.h"
textwindows struct DirectMap sys_mmap_nt(void *addr, size_t size, int prot,
int flags, int fd, int64_t off) {
int64_t handle;
if (flags & MAP_ANONYMOUS) {
handle = kNtInvalidHandleValue;
} else {
handle = g_fds.p[fd].handle;
}
// mark map handle as inheritable if fork might need it
const struct NtSecurityAttributes *mapsec;
if ((flags & MAP_TYPE) == MAP_SHARED) {
mapsec = &kNtIsInheritable;
} else {
mapsec = 0;
}
// nt will whine under many circumstances if we change the execute bit
// later using mprotect(). the workaround is to always request execute
// and then virtualprotect() it away until we actually need it. please
// note that open-nt.c always requests an kNtGenericExecute accessmask
int iscow = false;
struct ProtectNt fl;
if (handle != -1) {
if ((flags & MAP_TYPE) != MAP_SHARED) {
// windows has cow pages but they can't propagate across fork()
// that means we only get copy-on-write for the root process :(
fl = (struct ProtectNt){kNtPageExecuteWritecopy,
kNtFileMapCopy | kNtFileMapExecute};
iscow = true;
} else {
if ((g_fds.p[fd].flags & O_ACCMODE) == O_RDONLY) {
fl = (struct ProtectNt){kNtPageExecuteRead,
kNtFileMapRead | kNtFileMapExecute};
} else {
fl = (struct ProtectNt){kNtPageExecuteReadwrite,
kNtFileMapWrite | kNtFileMapExecute};
}
}
} else {
unassert(flags & MAP_ANONYMOUS);
fl = (struct ProtectNt){kNtPageExecuteReadwrite,
kNtFileMapWrite | kNtFileMapExecute};
}
int e = errno;
struct DirectMap dm;
TryAgain:
if ((dm.maphandle = CreateFileMapping(handle, mapsec, fl.flags1,
(size + off) >> 32, (size + off), 0))) {
if ((dm.addr = MapViewOfFileEx(dm.maphandle, fl.flags2, off >> 32, off,
size, addr))) {
uint32_t oldprot;
if (VirtualProtect(dm.addr, size, __prot2nt(prot, iscow), &oldprot))
return dm;
UnmapViewOfFile(dm.addr);
}
CloseHandle(dm.maphandle);
} else if (!(prot & PROT_EXEC) && //
(fl.flags2 & kNtFileMapExecute) && //
GetLastError() == kNtErrorAccessDenied) {
// your file needs to have been O_CREAT'd with exec `mode` bits in
// order to be mapped with executable permission. we always try to
// get execute permission if the kernel will give it to us because
// win32 would otherwise forbid mprotect() from elevating later on
fl.flags2 &= ~kNtFileMapExecute;
switch (fl.flags1) {
case kNtPageExecuteWritecopy:
fl.flags1 = kNtPageWritecopy;
break;
case kNtPageExecuteReadwrite:
fl.flags1 = kNtPageReadwrite;
break;
case kNtPageExecuteRead:
fl.flags1 = kNtPageReadonly;
break;
default:
__builtin_unreachable();
}
errno = e;
goto TryAgain;
}
dm.maphandle = kNtInvalidHandleValue;
dm.addr = (void *)(intptr_t)-1;
return dm;
}

View file

@ -2,19 +2,7 @@
#define COSMOPOLITAN_LIBC_INTRIN_DIRECTMAP_H_
COSMOPOLITAN_C_START_
struct ProtectNt {
uint32_t flags1;
uint32_t flags2;
};
struct DirectMap {
void *addr;
int64_t maphandle;
};
struct DirectMap sys_mmap(void *, size_t, int, int, int, int64_t);
struct DirectMap sys_mmap_nt(void *, size_t, int, int, int, int64_t);
struct DirectMap sys_mmap_metal(void *, size_t, int, int, int, int64_t);
void *sys_mmap_metal(void *, size_t, int, int, int, int64_t) libcesque;
int sys_munmap_metal(void *, size_t) libcesque;
int __prot2nt(int, int) libcesque;

34
libc/intrin/dlopen.c Normal file
View file

@ -0,0 +1,34 @@
/*-*- mode:c;indent-tabs-mode:nil;c-basic-offset:2;tab-width:8;coding:utf-8 -*-│
vi: set et ft=c ts=2 sts=2 sw=2 fenc=utf-8 :vi
Copyright 2024 Justine Alexandra Roberts Tunney
Permission to use, copy, modify, and/or distribute this software for
any purpose with or without fee is hereby granted, provided that the
above copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL
WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE
AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL
DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR
PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER
TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
PERFORMANCE OF THIS SOFTWARE.
*/
#include "libc/thread/posixthread.internal.h"
#include "libc/thread/thread.h"
static pthread_mutex_t __dlopen_lock_obj = PTHREAD_MUTEX_INITIALIZER;
void __dlopen_lock(void) {
_pthread_mutex_lock(&__dlopen_lock_obj);
}
void __dlopen_unlock(void) {
_pthread_mutex_unlock(&__dlopen_lock_obj);
}
void __dlopen_wipe(void) {
_pthread_mutex_wipe_np(&__dlopen_lock_obj);
}

View file

@ -44,15 +44,13 @@
#include "libc/sysv/consts/o.h"
#include "libc/sysv/consts/prot.h"
#include "libc/thread/thread.h"
#define OPEN_MAX 16
#include "libc/thread/tls.h"
#ifdef __x86_64__
__static_yoink("_init_fds");
#endif
struct Fds g_fds;
static struct Fd g_fds_static[OPEN_MAX];
static bool TokAtoi(const char **str, long *res) {
int c, d;
@ -86,19 +84,14 @@ static textwindows void SetupWinStd(struct Fds *fds, int i, uint32_t x) {
}
textstartup void __init_fds(int argc, char **argv, char **envp) {
struct Fds *fds;
fds = &g_fds;
fds->n = 4;
atomic_store_explicit(&fds->f, 3, memory_order_relaxed);
if (_weaken(_extend)) {
fds->p = fds->e = (void *)kMemtrackFdsStart;
fds->e =
_weaken(_extend)(fds->p, fds->n * sizeof(*fds->p), fds->e, MAP_PRIVATE,
kMemtrackFdsStart + kMemtrackFdsSize);
} else {
fds->p = g_fds_static;
fds->e = g_fds_static + OPEN_MAX;
}
fds->p = fds->e = (void *)kMemtrackFdsStart;
fds->e = _extend(fds->p, fds->n * sizeof(*fds->p), fds->e, MAP_PRIVATE,
kMemtrackFdsStart + kMemtrackFdsSize);
// inherit standard i/o file descriptors
if (IsMetal()) {
@ -152,8 +145,7 @@ textstartup void __init_fds(int argc, char **argv, char **envp) {
break;
if (!TokAtoi(&fdspec, &protocol))
break;
if (_weaken(__ensurefds_unlocked))
_weaken(__ensurefds_unlocked)(fd);
__ensurefds_unlocked(fd);
struct Fd *f = fds->p + fd;
if (f->handle && f->handle != -1 && f->handle != handle) {
CloseHandle(f->handle);
@ -188,7 +180,9 @@ textstartup void __init_fds(int argc, char **argv, char **envp) {
map->prot = PROT_READ | PROT_WRITE;
map->flags = MAP_SHARED | MAP_ANONYMOUS;
map->hand = shand;
__maps_lock();
__maps_insert(map);
__maps_unlock();
}
}
}

View file

@ -17,12 +17,13 @@
PERFORMANCE OF THIS SOFTWARE.
*/
#include "libc/calls/state.internal.h"
#include "libc/thread/posixthread.internal.h"
#include "libc/thread/thread.h"
void __fds_lock(void) {
pthread_mutex_lock(&__fds_lock_obj);
_pthread_mutex_lock(&__fds_lock_obj);
}
void __fds_unlock(void) {
pthread_mutex_unlock(&__fds_lock_obj);
_pthread_mutex_unlock(&__fds_lock_obj);
}

View file

@ -26,7 +26,9 @@
// @see setcontext()
.ftrace1
getcontext:
beg
.ftrace2
#include "libc/intrin/getcontext.inc"
jmp __getcontextsig
end
.endfn getcontext,globl

View file

@ -17,16 +17,13 @@
PERFORMANCE OF THIS SOFTWARE.
*/
#include "libc/calls/struct/rlimit.h"
#include "libc/calls/struct/rlimit.internal.h"
#include "libc/dce.h"
#include "libc/intrin/getauxval.h"
#include "libc/intrin/kprintf.h"
#include "libc/intrin/maps.h"
#include "libc/intrin/rlimit.h"
#include "libc/macros.h"
#include "libc/runtime/runtime.h"
#include "libc/stdio/sysparam.h"
#include "libc/sysv/consts/auxv.h"
#include "libc/sysv/consts/rlim.h"
#include "libc/sysv/consts/rlimit.h"
// Hack for guessing boundaries of _start()'s stack
//
@ -91,12 +88,9 @@ static uintptr_t __get_main_top(int pagesz) {
}
static size_t __get_stack_size(int pagesz, uintptr_t start, uintptr_t top) {
size_t size, max = 8 * 1024 * 1024;
struct rlimit rlim = {RLIM_INFINITY};
sys_getrlimit(RLIMIT_STACK, &rlim);
if ((size = rlim.rlim_cur) > max)
size = max;
return MAX(ROUNDUP(size, pagesz), ROUNDUP(top - start, pagesz));
size_t stacksz = __rlimit_stack_get().rlim_cur;
stacksz = MIN(stacksz, 1024ul * 1024 * 1024 * 1024);
return MAX(ROUNDDOWN(stacksz, pagesz), ROUNDUP(top - start, pagesz));
}
/**

View file

@ -16,18 +16,47 @@
TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
PERFORMANCE OF THIS SOFTWARE.
*/
#include "libc/calls/struct/siginfo.h"
#include "libc/calls/ucontext.h"
#include "libc/dce.h"
#include "libc/intrin/getauxval.h"
#include "libc/macros.h"
#include "libc/runtime/runtime.h"
#include "libc/sysv/consts/auxv.h"
#include "libc/sysv/consts/ss.h"
long __get_minsigstksz(void) {
struct AuxiliaryValue x;
x = __getauxval(AT_MINSIGSTKSZ);
if (x.isfound) {
return MAX(_MINSIGSTKSZ, x.value);
struct AuxiliaryValue av;
av = __getauxval(AT_MINSIGSTKSZ);
if (av.isfound) {
long res = av.value;
if (!IsLinux())
res += sizeof(struct ucontext) + sizeof(struct siginfo) + 128;
if (res < _MINSIGSTKSZ)
res = _MINSIGSTKSZ;
return res;
} else {
// _MINSIGSTKSZ takes these things into consideration:
//
// 1. The platform definition of MINSIGSTKSZ. This will probably be
// enforced by the kernel when calling sys_sigaltstack(). On ARM
// platforms this might be several kilobytes larger than x86. On
// Linux they really want you to use AT_MINSIGSTKSZ instead. The
// kernel should ideally set this to be the number of bytes that
// get subtracted from the stack pointer when delivering signals
// meaning that if you use this for a stack size your handler is
// called successfully but if it uses the stack then it'll crash
//
// 2. Cosmo sigenter overhead. On non-Linux OSes the kernel calls a
// trampoline in the libc runtime, which translates the platform
// specific signal frame to the Linux memory layout. It means we
// need to push ~1024 extra bytes on the stack to call a handler
//
// 3. Sanity testing. Assume we use sysconf(_SC_MINSIGSTKSZ) + 2048
// as our stack size (see stackoverflow1_test.c). Then we should
// have enough room to use kprintf() from our signal handler. If
// that isn't the case, then this should be increased a bit more
// noting that if 1024 is used then kprintf should print refusal
//
return _MINSIGSTKSZ;
}
}

View file

@ -17,12 +17,11 @@
PERFORMANCE OF THIS SOFTWARE.
*/
#include "ape/sections.internal.h"
#include "libc/intrin/kprintf.h"
#include "libc/runtime/memtrack.internal.h"
#include "libc/runtime/runtime.h"
#include "libc/runtime/stack.h"
#include "libc/thread/posixthread.internal.h"
#include "libc/thread/tls.h"
#include "libc/thread/tls2.internal.h"
/**
* Computes safer buffer size for alloca().
@ -32,18 +31,19 @@
* @return number of bytes to use for your buffer, or negative if the
* allocation would likely cause a stack overflow
*/
privileged long __get_safe_size(long want, long extraspace) {
privileged optimizesize long __get_safe_size(long want, long extraspace) {
if (!__tls_enabled)
return want;
struct PosixThread *pt;
struct CosmoTib *tib = __get_tls_privileged();
long bottom, sp = GetStackPointer();
if ((char *)sp >= tib->tib_sigstack_addr &&
(char *)sp <= tib->tib_sigstack_addr + tib->tib_sigstack_size) {
if (sp >= (long)tib->tib_sigstack_addr &&
sp < (long)tib->tib_sigstack_addr + tib->tib_sigstack_size) {
bottom = (long)tib->tib_sigstack_addr;
} else if ((pt = (struct PosixThread *)tib->tib_pthread) &&
pt->pt_attr.__stacksize) {
bottom = (long)pt->pt_attr.__stackaddr + pt->pt_attr.__guardsize;
sp >= (long)pt->pt_attr.__stackaddr &&
sp < (long)pt->pt_attr.__stackaddr + pt->pt_attr.__stacksize) {
bottom = (long)pt->pt_attr.__stackaddr;
} else {
return want;
}

View file

@ -39,7 +39,7 @@
int gettid(void) {
int tid;
if (VERY_LIKELY(__tls_enabled && !__vforked)) {
tid = atomic_load_explicit(&__get_tls()->tib_tid, memory_order_acquire);
tid = atomic_load_explicit(&__get_tls()->tib_ptid, memory_order_relaxed);
if (VERY_LIKELY(tid > 0))
return tid;
}

43
libc/intrin/itimer.c Normal file
View file

@ -0,0 +1,43 @@
/*-*- mode:c;indent-tabs-mode:nil;c-basic-offset:2;tab-width:8;coding:utf-8 -*-│
vi: set et ft=c ts=2 sts=2 sw=2 fenc=utf-8 :vi
Copyright 2024 Justine Alexandra Roberts Tunney
Permission to use, copy, modify, and/or distribute this software for
any purpose with or without fee is hereby granted, provided that the
above copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL
WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE
AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL
DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR
PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER
TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
PERFORMANCE OF THIS SOFTWARE.
*/
#include "libc/thread/itimer.h"
#include "libc/str/str.h"
#include "libc/thread/posixthread.internal.h"
struct IntervalTimer __itimer = {
.lock = PTHREAD_MUTEX_INITIALIZER,
.cond = PTHREAD_COND_INITIALIZER,
};
textwindows void __itimer_lock(void) {
_pthread_mutex_lock(&__itimer.lock);
}
textwindows void __itimer_unlock(void) {
_pthread_mutex_unlock(&__itimer.lock);
}
textwindows void __itimer_wipe_and_reset(void) {
// timers aren't inherited by forked subprocesses
bzero(&__itimer.it, sizeof(__itimer.it));
_pthread_mutex_wipe_np(&__itimer.lock);
bzero(&__itimer.cond, sizeof(__itimer.cond));
__itimer.thread = 0;
__itimer.once = 0;
}

View file

@ -0,0 +1,38 @@
/*-*- mode:c;indent-tabs-mode:nil;c-basic-offset:2;tab-width:8;coding:utf-8 -*-│
vi: set et ft=c ts=2 sts=2 sw=2 fenc=utf-8 :vi
Copyright 2024 Justine Alexandra Roberts Tunney
Permission to use, copy, modify, and/or distribute this software for
any purpose with or without fee is hereby granted, provided that the
above copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL
WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE
AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL
DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR
PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER
TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
PERFORMANCE OF THIS SOFTWARE.
*/
#include "libc/intrin/kprintf.h"
#include "libc/intrin/maps.h"
#include "libc/runtime/runtime.h"
privileged optimizesize bool32 kisdangerous(const void *addr) {
bool32 res = true;
__maps_lock();
if (__maps.maps) {
struct Map *map;
if ((map = __maps_floor(addr)))
if ((const char *)addr >= map->addr &&
(const char *)addr <
map->addr + ((map->size + __pagesize - 1) & -__pagesize))
res = false;
} else {
res = false;
}
__maps_unlock();
return res;
}

View file

@ -65,10 +65,11 @@
#include "libc/sysv/consts/o.h"
#include "libc/sysv/consts/prot.h"
#include "libc/thread/tls.h"
#include "libc/thread/tls2.internal.h"
#include "libc/vga/vga.internal.h"
#include "libc/wctype.h"
#define ABI privileged optimizesize
#define STACK_ERROR "kprintf error: stack is about to overflow\n"
#define KGETINT(x, va, t, s) \
@ -159,23 +160,7 @@ __funline bool kischarmisaligned(const char *p, signed char t) {
return false;
}
privileged bool32 kisdangerous(const void *addr) {
bool32 res = true;
__maps_lock();
if (__maps.maps) {
struct Map *map;
if ((map = __maps_floor(addr)))
if ((const char *)addr >= map->addr &&
(const char *)addr < map->addr + map->size)
res = false;
} else {
res = false;
}
__maps_unlock();
return res;
}
privileged static void klogclose(long fd) {
ABI static void klogclose(long fd) {
#ifdef __x86_64__
long ax = __NR_close;
asm volatile("syscall"
@ -192,7 +177,7 @@ privileged static void klogclose(long fd) {
#endif
}
privileged static long klogfcntl(long fd, long cmd, long arg) {
ABI static long klogfcntl(long fd, long cmd, long arg) {
#ifdef __x86_64__
char cf;
long ax = __NR_fcntl;
@ -224,7 +209,7 @@ privileged static long klogfcntl(long fd, long cmd, long arg) {
#endif
}
privileged static long klogopen(const char *path) {
ABI static long klogopen(const char *path) {
long dirfd = AT_FDCWD;
long flags = O_WRONLY | O_CREAT | O_APPEND;
long mode = 0600;
@ -263,7 +248,7 @@ privileged static long klogopen(const char *path) {
}
// returns log handle or -1 if logging shouldn't happen
privileged long kloghandle(void) {
ABI long kloghandle(void) {
// kprintf() needs to own a file descriptor in case apps closes stderr
// our close() and dup() implementations will trigger this initializer
// to minimize a chance that the user accidentally closes their logger
@ -342,7 +327,7 @@ privileged long kloghandle(void) {
}
#ifdef __x86_64__
privileged void _klog_serial(const char *b, size_t n) {
ABI void _klog_serial(const char *b, size_t n) {
size_t i;
uint16_t dx;
unsigned char al;
@ -362,14 +347,13 @@ privileged void _klog_serial(const char *b, size_t n) {
}
#endif /* __x86_64__ */
privileged void klog(const char *b, size_t n) {
ABI void klog(const char *b, size_t n) {
#ifdef __x86_64__
long h;
uint32_t wrote;
long rax, rdi, rsi, rdx;
if ((h = kloghandle()) == -1) {
if ((h = kloghandle()) == -1)
return;
}
if (IsWindows()) {
bool32 ok;
intptr_t ev;
@ -420,14 +404,14 @@ privileged void klog(const char *b, size_t n) {
#endif
}
privileged static size_t kformat(char *b, size_t n, const char *fmt,
va_list va) {
ABI static size_t kformat(char *b, size_t n, const char *fmt, va_list va) {
int si;
wint_t t, u;
char *cxxbuf;
const char *abet;
signed char type;
const char *s, *f;
char cxxbuf[3000];
int cxxbufsize = 0;
struct CosmoTib *tib;
unsigned long long x;
unsigned i, j, m, rem, sign, hash, cols, prec;
@ -577,7 +561,7 @@ privileged static size_t kformat(char *b, size_t n, const char *fmt,
tib = __tls_enabled ? __get_tls_privileged() : 0;
if (!(tib && (tib->tib_flags & TIB_FLAG_VFORKED))) {
if (tib) {
x = atomic_load_explicit(&tib->tib_tid, memory_order_relaxed);
x = atomic_load_explicit(&tib->tib_ptid, memory_order_relaxed);
} else {
x = __pid;
}
@ -771,13 +755,25 @@ privileged static size_t kformat(char *b, size_t n, const char *fmt,
x = va_arg(va, intptr_t);
if (_weaken(__symtab) && *_weaken(__symtab) &&
(idx = _weaken(__get_symbol)(0, x)) != -1) {
/* if (p + 1 <= e) */
/* *p++ = '&'; */
s = (*_weaken(__symtab))->name_base +
(*_weaken(__symtab))->names[idx];
if (_weaken(__is_mangled) && _weaken(__is_mangled)(s) &&
_weaken(__demangle)(cxxbuf, s, sizeof(cxxbuf)) != -1)
s = cxxbuf;
#pragma GCC push_options
#pragma GCC diagnostic ignored "-Walloca-larger-than="
// decipher c++ symbols if there's enough stack memory
// stack size requirement assumes max_depth's still 20
if (_weaken(__demangle) && //
_weaken(__is_mangled) && //
_weaken(__is_mangled)(s)) {
if (!cxxbufsize)
if ((cxxbufsize = __get_safe_size(8192, 8192)) >= 512) {
cxxbuf = alloca(cxxbufsize);
CheckLargeStackAllocation(cxxbuf, sizeof(cxxbufsize));
}
if (cxxbufsize >= 512)
if (_weaken(__demangle)(cxxbuf, s, cxxbufsize) != -1)
s = cxxbuf;
}
#pragma GCC pop_options
goto FormatString;
}
base = 4;
@ -1033,7 +1029,7 @@ privileged static size_t kformat(char *b, size_t n, const char *fmt,
* @asyncsignalsafe
* @vforksafe
*/
privileged size_t ksnprintf(char *b, size_t n, const char *fmt, ...) {
ABI size_t ksnprintf(char *b, size_t n, const char *fmt, ...) {
size_t m;
va_list v;
va_start(v, fmt);
@ -1052,7 +1048,7 @@ privileged size_t ksnprintf(char *b, size_t n, const char *fmt, ...) {
* @asyncsignalsafe
* @vforksafe
*/
privileged size_t kvsnprintf(char *b, size_t n, const char *fmt, va_list v) {
ABI size_t kvsnprintf(char *b, size_t n, const char *fmt, va_list v) {
return kformat(b, n, fmt, v);
}
@ -1063,10 +1059,10 @@ privileged size_t kvsnprintf(char *b, size_t n, const char *fmt, va_list v) {
* @asyncsignalsafe
* @vforksafe
*/
privileged void kvprintf(const char *fmt, va_list v) {
ABI void kvprintf(const char *fmt, va_list v) {
#pragma GCC push_options
#pragma GCC diagnostic ignored "-Walloca-larger-than="
long size = __get_safe_size(8000, 8000);
long size = __get_safe_size(8192, 2048);
if (size < 80) {
klog(STACK_ERROR, sizeof(STACK_ERROR) - 1);
return;
@ -1149,7 +1145,7 @@ privileged void kvprintf(const char *fmt, va_list v) {
* @asyncsignalsafe
* @vforksafe
*/
privileged void kprintf(const char *fmt, ...) {
ABI void kprintf(const char *fmt, ...) {
// system call support runtime depends on this function
// function tracing runtime depends on this function
// asan runtime depends on this function

View file

@ -0,0 +1,34 @@
/*-*- mode:c;indent-tabs-mode:nil;c-basic-offset:2;tab-width:8;coding:utf-8 -*-│
vi: set et ft=c ts=2 sts=2 sw=2 fenc=utf-8 :vi
Copyright 2024 Justine Alexandra Roberts Tunney
Permission to use, copy, modify, and/or distribute this software for
any purpose with or without fee is hereby granted, provided that the
above copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL
WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE
AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL
DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR
PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER
TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
PERFORMANCE OF THIS SOFTWARE.
*/
#include "libc/thread/posixthread.internal.h"
#include "third_party/tz/lock.h"
static pthread_mutex_t __localtime_lock_obj = PTHREAD_MUTEX_INITIALIZER;
void __localtime_lock(void) {
_pthread_mutex_lock(&__localtime_lock_obj);
}
void __localtime_unlock(void) {
_pthread_mutex_unlock(&__localtime_lock_obj);
}
void __localtime_wipe(void) {
_pthread_mutex_wipe_np(&__localtime_lock_obj);
}

50
libc/intrin/lockless.h Normal file
View file

@ -0,0 +1,50 @@
#ifndef COSMOPOLITAN_LIBC_INTRIN_LOCKLESS_H_
#define COSMOPOLITAN_LIBC_INTRIN_LOCKLESS_H_
#include "libc/atomic.h"
#include "libc/intrin/atomic.h"
COSMOPOLITAN_C_START_
// lockless memory transactions
//
// - one writer
// - many readers
// - generation is monotonic
// - even numbers mean memory is ready
// - odd numbers mean memory is actively being changed
// - always use acquire semantics inside your read transaction
//
// let's say you want to be able to atomically read and write to 128-bit
// values, but you've only got a 64-bit system. if you expect that it'll
// frequently written, then you should use a mutex. but if you expect it
// to be frequently read and rarely written, then it's possible to do it
// without a mutex; in fact you don't even need the x86 lock instruction
// prefix; all that is required is a series of carefully ordered mov ops
// which are designed to exploit the strong ordering of the architecture
static inline unsigned lockless_write_begin(atomic_uint* genptr) {
unsigned gen = atomic_load_explicit(genptr, memory_order_acquire);
atomic_store_explicit(genptr, gen + 1, memory_order_release);
return gen;
}
static inline void lockless_write_end(atomic_uint* genptr, unsigned gen) {
atomic_store_explicit(genptr, gen + 2, memory_order_release);
}
static inline unsigned lockless_read_begin(atomic_uint* genptr) {
return atomic_load_explicit(genptr, memory_order_acquire);
}
static inline bool lockless_read_end(atomic_uint* genptr, unsigned* want) {
unsigned gen1 = *want;
unsigned gen2 = atomic_load_explicit(genptr, memory_order_acquire);
unsigned is_being_actively_changed = gen1 & 1;
unsigned we_lost_race_with_writers = gen1 ^ gen2;
if (!(is_being_actively_changed | we_lost_race_with_writers))
return true;
*want = gen2;
return false;
}
COSMOPOLITAN_C_END_
#endif /* COSMOPOLITAN_LIBC_INTRIN_LOCKLESS_H_ */

View file

@ -19,21 +19,27 @@
#include "libc/intrin/maps.h"
#include "ape/sections.internal.h"
#include "libc/calls/state.internal.h"
#include "libc/calls/syscall-sysv.internal.h"
#include "libc/cosmo.h"
#include "libc/dce.h"
#include "libc/intrin/describebacktrace.h"
#include "libc/intrin/dll.h"
#include "libc/intrin/kprintf.h"
#include "libc/intrin/maps.h"
#include "libc/macros.h"
#include "libc/nexgen32e/rdtsc.h"
#include "libc/runtime/runtime.h"
#include "libc/runtime/stack.h"
#include "libc/sysv/consts/auxv.h"
#include "libc/sysv/consts/map.h"
#include "libc/sysv/consts/prot.h"
#include "libc/thread/lock.h"
#include "libc/thread/tls.h"
#ifdef __x86_64__
__static_yoink("_init_maps");
#endif
#define ABI privileged optimizespeed
struct Maps __maps;
void __maps_add(struct Map *map) {
@ -51,74 +57,109 @@ void __maps_stack(char *stackaddr, int pagesz, int guardsize, size_t stacksize,
__maps.stack.addr = stackaddr + guardsize;
__maps.stack.size = stacksize - guardsize;
__maps.stack.prot = stackprot;
__maps.stack.hand = -1;
__maps.stack.hand = MAPS_SUBREGION;
__maps.stack.flags = MAP_PRIVATE | MAP_ANONYMOUS;
__maps_adder(&__maps.stack, pagesz);
if (guardsize) {
__maps.guard.addr = stackaddr;
__maps.guard.size = guardsize;
__maps.guard.prot = PROT_NONE;
__maps.guard.prot = PROT_NONE | PROT_GUARD;
__maps.guard.hand = stackhand;
__maps.guard.flags = MAP_PRIVATE | MAP_ANONYMOUS;
__maps_adder(&__maps.guard, pagesz);
} else {
__maps.stack.hand = stackhand;
}
}
void __maps_init(void) {
int pagesz = __pagesize;
// initialize lemur64
__maps.rand = 2131259787901769494;
__maps.rand ^= kStartTsc;
// these static map objects avoid mandatory mmap() in __maps_alloc()
// they aren't actually needed for bootstrapping this memory manager
for (int i = 0; i < ARRAYLEN(__maps.spool); ++i)
__maps_free(&__maps.spool[i]);
// record _start() stack mapping
if (!IsWindows()) {
struct AddrSize stack;
stack = __get_main_stack();
__maps_stack(stack.addr, pagesz, 0, stack.size, (uintptr_t)ape_stack_prot,
0);
// linux v4.12+ reserves 1mb of guard space beneath rlimit_stack
// https://lwn.net/Articles/725832/. if we guess too small, then
// slackmap will create a bunch of zombie stacks in __print_maps
// to coverup the undisclosed memory but no cost if we guess big
size_t guardsize = 1024 * 1024;
guardsize += __pagesize - 1;
guardsize &= -__pagesize;
// track the main stack region that the os gave to start earlier
struct AddrSize stack = __get_main_stack();
__maps_stack(stack.addr - guardsize, pagesz, guardsize,
guardsize + stack.size, (uintptr_t)ape_stack_prot, 0);
}
// record .text and .data mappings
static struct Map text, data;
text.addr = (char *)__executable_start;
text.size = _etext - __executable_start;
text.prot = PROT_READ | PROT_EXEC;
__maps_track((char *)__executable_start, _etext - __executable_start,
PROT_READ | PROT_EXEC, MAP_NOFORK);
uintptr_t ds = ((uintptr_t)_etext + pagesz - 1) & -pagesz;
if (ds < (uintptr_t)_end) {
data.addr = (char *)ds;
data.size = (uintptr_t)_end - ds;
data.prot = PROT_READ | PROT_WRITE;
__maps_adder(&data, pagesz);
}
__maps_adder(&text, pagesz);
if (ds < (uintptr_t)_end)
__maps_track((char *)ds, (uintptr_t)_end - ds, PROT_READ | PROT_WRITE,
MAP_NOFORK);
}
privileged bool __maps_lock(void) {
bool __maps_held(void) {
return !__tls_enabled || (__get_tls()->tib_flags & TIB_FLAG_VFORKED) ||
MUTEX_OWNER(
atomic_load_explicit(&__maps.lock.word, memory_order_relaxed)) ==
atomic_load_explicit(&__get_tls()->tib_ptid, memory_order_relaxed);
}
bool __maps_reentrant(void) {
return __tls_enabled && !(__get_tls()->tib_flags & TIB_FLAG_VFORKED) &&
MUTEX_OWNER(
atomic_load_explicit(&__maps.lock.word, memory_order_relaxed)) ==
atomic_load_explicit(&__get_tls()->tib_ptid, memory_order_relaxed);
}
ABI void __maps_lock(void) {
int me;
uint64_t word, lock;
struct CosmoTib *tib;
if (!__tls_enabled)
return false;
return;
if (!(tib = __get_tls_privileged()))
return false;
return;
if (tib->tib_flags & TIB_FLAG_VFORKED)
return false;
me = atomic_load_explicit(&tib->tib_tid, memory_order_acquire);
if (me <= 0)
return false;
word = atomic_load_explicit(&__maps.lock, memory_order_relaxed);
return;
me = atomic_load_explicit(&tib->tib_ptid, memory_order_relaxed);
word = 0;
lock = MUTEX_LOCK(word);
lock = MUTEX_SET_OWNER(lock, me);
if (atomic_compare_exchange_strong_explicit(&__maps.lock.word, &word, lock,
memory_order_acquire,
memory_order_relaxed))
return;
word = atomic_load_explicit(&__maps.lock.word, memory_order_relaxed);
for (;;) {
if (MUTEX_OWNER(word) == me) {
if (atomic_compare_exchange_weak_explicit(
&__maps.lock, &word, MUTEX_INC_DEPTH(word), memory_order_relaxed,
memory_order_relaxed))
return true;
&__maps.lock.word, &word, MUTEX_INC_DEPTH(word),
memory_order_relaxed, memory_order_relaxed))
return;
continue;
}
word = 0;
lock = MUTEX_LOCK(word);
lock = MUTEX_SET_OWNER(lock, me);
if (atomic_compare_exchange_weak_explicit(&__maps.lock, &word, lock,
if (atomic_compare_exchange_weak_explicit(&__maps.lock.word, &word, lock,
memory_order_acquire,
memory_order_relaxed))
return false;
return;
for (;;) {
word = atomic_load_explicit(&__maps.lock, memory_order_relaxed);
word = atomic_load_explicit(&__maps.lock.word, memory_order_relaxed);
if (MUTEX_OWNER(word) == me)
break;
if (!word)
@ -127,8 +168,7 @@ privileged bool __maps_lock(void) {
}
}
privileged void __maps_unlock(void) {
int me;
ABI void __maps_unlock(void) {
uint64_t word;
struct CosmoTib *tib;
if (!__tls_enabled)
@ -137,19 +177,16 @@ privileged void __maps_unlock(void) {
return;
if (tib->tib_flags & TIB_FLAG_VFORKED)
return;
me = atomic_load_explicit(&tib->tib_tid, memory_order_acquire);
if (me <= 0)
return;
word = atomic_load_explicit(&__maps.lock, memory_order_relaxed);
word = atomic_load_explicit(&__maps.lock.word, memory_order_relaxed);
for (;;) {
if (MUTEX_DEPTH(word)) {
if (MUTEX_DEPTH(word))
if (atomic_compare_exchange_weak_explicit(
&__maps.lock, &word, MUTEX_DEC_DEPTH(word), memory_order_relaxed,
memory_order_relaxed))
&__maps.lock.word, &word, MUTEX_DEC_DEPTH(word),
memory_order_relaxed, memory_order_relaxed))
break;
}
if (atomic_compare_exchange_weak_explicit(
&__maps.lock, &word, 0, memory_order_release, memory_order_relaxed))
if (atomic_compare_exchange_weak_explicit(&__maps.lock.word, &word, 0,
memory_order_release,
memory_order_relaxed))
break;
}
}

View file

@ -3,10 +3,29 @@
#include "libc/intrin/atomic.h"
#include "libc/intrin/tree.h"
#include "libc/runtime/runtime.h"
#include "libc/thread/tls2.internal.h"
COSMOPOLITAN_C_START_
#define MAPS_RETRY ((void *)-1)
/* size of dynamic memory that is used internally by your memory manager */
#define MAPS_SIZE 65536
/* when map->hand is MAPS_RESERVATION it means mmap() is transactionally
reserving address space it is in the process of requesting from win32 */
#define MAPS_RESERVATION -2
/* when map->hand is MAPS_SUBREGION it means that an allocation has been
broken into multiple fragments by mprotect(). the first fragment must
be set to MAPS_VIRTUAL or your CreateFileMapping() handle. your frags
must be perfectly contiguous in memory and should have the same flags */
#define MAPS_SUBREGION -3
/* indicates an allocation was created by VirtualAlloc() and so munmap()
must call VirtualFree() when destroying it. use it on the hand field. */
#define MAPS_VIRTUAL -4
/* if this is used on MAP_PRIVATE memory, then it's assumed to be memory
that win32 allocated, e.g. a CreateThread() stack. if this is used on
MAP_FILE memory, then it's assumed to be part of the executable image */
#define MAP_NOFORK 0x10000000
#define MAP_TREE_CONTAINER(e) TREE_CONTAINER(struct Map, tree, e)
@ -14,8 +33,8 @@ struct Map {
char *addr; /* granule aligned */
size_t size; /* must be nonzero */
int64_t off; /* ignore for anon */
int prot; /* memory protects */
int flags; /* memory map flag */
short prot; /* memory protects */
bool iscow; /* windows nt only */
bool readonlyfile; /* windows nt only */
unsigned visited; /* checks and fork */
@ -26,15 +45,31 @@ struct Map {
};
};
struct MapLock {
void *edges;
_Atomic(uint64_t) word;
};
struct MapSlab {
struct MapSlab *next;
struct Map maps[(MAPS_SIZE - sizeof(struct MapSlab *)) / sizeof(struct Map)];
};
struct Maps {
uint128_t rand;
struct Tree *maps;
_Atomic(uint64_t) lock;
struct MapLock lock;
_Atomic(uintptr_t) freed;
_Atomic(struct MapSlab *) slabs;
size_t count;
size_t pages;
char *pick;
struct Map stack;
struct Map guard;
#ifdef MODE_DBG
struct Map spool[1];
#else
struct Map spool[20];
#endif
};
struct AddrSize {
@ -44,15 +79,18 @@ struct AddrSize {
extern struct Maps __maps;
bool __maps_held(void);
void __maps_init(void);
bool __maps_lock(void);
void __maps_lock(void);
void __maps_check(void);
void __maps_unlock(void);
bool __maps_reentrant(void);
void *__maps_randaddr(void);
void __maps_add(struct Map *);
void __maps_free(struct Map *);
void __maps_insert(struct Map *);
bool __maps_track(char *, size_t);
int __maps_untrack(char *, size_t);
bool __maps_track(char *, size_t, int, int);
struct Map *__maps_alloc(void);
struct Map *__maps_floor(const char *);
void __maps_stack(char *, int, int, size_t, int, intptr_t);
@ -73,6 +111,13 @@ static inline struct Map *__maps_next(struct Map *map) {
return 0;
}
static inline struct Map *__maps_prev(struct Map *map) {
struct Tree *node;
if ((node = tree_prev(&map->tree)))
return MAP_TREE_CONTAINER(node);
return 0;
}
static inline struct Map *__maps_first(void) {
struct Tree *node;
if ((node = tree_first(__maps.maps)))
@ -80,5 +125,16 @@ static inline struct Map *__maps_first(void) {
return 0;
}
static inline struct Map *__maps_last(void) {
struct Tree *node;
if ((node = tree_last(__maps.maps)))
return MAP_TREE_CONTAINER(node);
return 0;
}
static inline bool __maps_isalloc(struct Map *map) {
return map->hand != MAPS_SUBREGION;
}
COSMOPOLITAN_C_END_
#endif /* COSMOPOLITAN_MAPS_H_ */

File diff suppressed because it is too large Load diff

View file

@ -22,7 +22,6 @@
#include "libc/intrin/describeflags.h"
#include "libc/intrin/directmap.h"
#include "libc/intrin/dll.h"
#include "libc/intrin/kprintf.h"
#include "libc/intrin/maps.h"
#include "libc/intrin/strace.h"
#include "libc/intrin/tree.h"
@ -67,17 +66,18 @@ int __mprotect(char *addr, size_t size, int prot) {
// normalize size
size = (size + pagesz - 1) & -pagesz;
// test for signal handler reentry
if (__maps_reentrant())
return edeadlk();
// change mappings
int rc = 0;
bool found = false;
if (__maps_lock()) {
__maps_unlock();
return edeadlk();
}
struct Map *map, *floor;
StartOver:
floor = __maps_floor(addr);
for (map = floor; map && map->addr <= addr + size; map = __maps_next(map)) {
__maps_lock();
struct Map *map;
if (!(map = __maps_floor(addr)))
map = __maps_first();
for (; map && map->addr <= addr + size; map = __maps_next(map)) {
char *map_addr = map->addr;
size_t map_size = map->size;
char *beg = MAX(addr, map_addr);
@ -86,7 +86,7 @@ StartOver:
continue;
found = true;
if (addr <= map_addr && addr + size >= map_addr + PGUP(map_size)) {
// change protection of entire mapping
// change protection status of pages
if (!__mprotect_chunk(map_addr, map_size, prot, map->iscow)) {
map->prot = prot;
} else {
@ -98,8 +98,6 @@ StartOver:
size_t right = map_size - left;
struct Map *leftmap;
if ((leftmap = __maps_alloc())) {
if (leftmap == MAPS_RETRY)
goto StartOver;
if (!__mprotect_chunk(map_addr, left, prot, false)) {
leftmap->addr = map_addr;
leftmap->size = left;
@ -111,7 +109,7 @@ StartOver:
leftmap->hand = map->hand;
map->addr += left;
map->size = right;
map->hand = -1;
map->hand = MAPS_SUBREGION;
if (!(map->flags & MAP_ANONYMOUS))
map->off += left;
tree_insert(&__maps.maps, &leftmap->tree, __maps_compare);
@ -130,8 +128,6 @@ StartOver:
size_t right = map_addr + map_size - addr;
struct Map *leftmap;
if ((leftmap = __maps_alloc())) {
if (leftmap == MAPS_RETRY)
goto StartOver;
if (!__mprotect_chunk(map_addr + left, right, prot, false)) {
leftmap->addr = map_addr;
leftmap->size = left;
@ -144,7 +140,7 @@ StartOver:
map->addr += left;
map->size = right;
map->prot = prot;
map->hand = -1;
map->hand = MAPS_SUBREGION;
if (!(map->flags & MAP_ANONYMOUS))
map->off += left;
tree_insert(&__maps.maps, &leftmap->tree, __maps_compare);
@ -164,14 +160,8 @@ StartOver:
size_t right = map_size - middle - left;
struct Map *leftmap;
if ((leftmap = __maps_alloc())) {
if (leftmap == MAPS_RETRY)
goto StartOver;
struct Map *midlmap;
if ((midlmap = __maps_alloc())) {
if (midlmap == MAPS_RETRY) {
__maps_free(leftmap);
goto StartOver;
}
if (!__mprotect_chunk(map_addr + left, middle, prot, false)) {
leftmap->addr = map_addr;
leftmap->size = left;
@ -186,10 +176,10 @@ StartOver:
midlmap->off = (map->flags & MAP_ANONYMOUS) ? 0 : map->off + left;
midlmap->prot = prot;
midlmap->flags = map->flags;
midlmap->hand = -1;
midlmap->hand = MAPS_SUBREGION;
map->addr += left + middle;
map->size = right;
map->hand = -1;
map->hand = MAPS_SUBREGION;
if (!(map->flags & MAP_ANONYMOUS))
map->off += left + middle;
tree_insert(&__maps.maps, &leftmap->tree, __maps_compare);
@ -222,11 +212,20 @@ StartOver:
/**
* Modifies restrictions on virtual memory address range.
*
* @param addr needs to be 4kb aligned
* @param prot can have PROT_{NONE,READ,WRITE,EXEC}
* POSIX doesn't require mprotect() to be async signal safe. However you
* should be able to call this from a signal handler safely, if you know
* that your signal will never interrupt the cosmopolitan memory manager
* and the only way you can ensure that, is by blocking signals whenever
* you call mmap(), munmap(), mprotect(), etc.
*
* @param addr needs to be page size aligned
* @param size is rounded up to the page size
* @param prot can be PROT_NONE or a combination of PROT_READ,
* PROT_WRITE, and PROT_EXEC
* @return 0 on success, or -1 w/ errno
* @raise EINVAL if `size` is zero
* @raise ENOMEM on tracking memory oom
* @see mmap()
* @raise EDEADLK if called from signal handler interrupting mmap()
*/
int mprotect(void *addr, size_t size, int prot) {
int rc;

View file

@ -19,34 +19,34 @@
#include "libc/calls/syscall-nt.internal.h"
#include "libc/intrin/maps.h"
#include "libc/nt/memory.h"
#include "libc/nt/runtime.h"
#include "libc/runtime/runtime.h"
#include "libc/stdio/sysparam.h"
#include "libc/sysv/consts/auxv.h"
#include "libc/sysv/consts/map.h"
#include "libc/sysv/errfuns.h"
textwindows int sys_msync_nt(char *addr, size_t size, int flags) {
size = (size + __pagesize - 1) & -__pagesize;
int pagesz = __pagesize;
size = (size + pagesz - 1) & -pagesz;
if ((uintptr_t)addr & (pagesz - 1))
if ((uintptr_t)addr & (__pagesize - 1))
return einval();
if (__maps_reentrant())
return edeadlk();
int rc = 0;
if (__maps_lock()) {
rc = edeadlk();
} else {
struct Map *map, *floor;
floor = __maps_floor(addr);
for (map = floor; map && map->addr <= addr + size; map = __maps_next(map)) {
char *beg = MAX(addr, map->addr);
char *end = MIN(addr + size, map->addr + map->size);
if (beg < end)
if (!FlushViewOfFile(beg, end - beg))
rc = -1;
// TODO(jart): FlushFileBuffers too on g_fds handle if MS_SYNC?
}
__maps_lock();
struct Map *map;
if (!(map = __maps_floor(addr)))
map = __maps_first();
for (; map && map->addr <= addr + size; map = __maps_next(map)) {
if (map->flags & MAP_ANONYMOUS)
continue; // msync() is about coherency between file and memory
char *beg = MAX(addr, map->addr);
char *end = MIN(addr + size, map->addr + map->size);
if (beg >= end)
continue; // didn't overlap mapping
if (!FlushViewOfFile(beg, end - beg))
rc = -1;
// TODO(jart): FlushFileBuffers too on g_fds handle if MS_SYNC?
}
__maps_unlock();

View file

@ -38,6 +38,7 @@
* @param flags needs MS_ASYNC or MS_SYNC and can have MS_INVALIDATE
* @return 0 on success or -1 w/ errno
* @raise ECANCELED if thread was cancelled in masked mode
* @raise EDEADLK if called from signal handler interrupting mmap()
* @raise EINTR if we needed to block and a signal was delivered instead
* @raise EINVAL if `MS_SYNC` and `MS_ASYNC` were both specified
* @raise EINVAL if unknown `flags` were passed
@ -67,23 +68,19 @@ int msync(void *addr, size_t size, int flags) {
} else {
sysflags = MS_ASYNC;
}
if (flags & MS_INVALIDATE) {
if (flags & MS_INVALIDATE)
sysflags |= MS_INVALIDATE;
}
// FreeBSD's manual says "The flags argument was both MS_ASYNC and
// MS_INVALIDATE. Only one of these flags is allowed." which makes
// following the POSIX recommendation somewhat difficult.
if (IsFreebsd()) {
if (sysflags == (MS_ASYNC | MS_INVALIDATE)) {
if (IsFreebsd())
if (sysflags == (MS_ASYNC | MS_INVALIDATE))
sysflags = MS_INVALIDATE;
}
}
// FreeBSD specifies MS_SYNC as 0 so we shift the Cosmo constants
if (IsFreebsd()) {
if (IsFreebsd())
sysflags >>= 1;
}
BEGIN_CANCELATION_POINT;
if (!IsWindows()) {

View file

@ -41,8 +41,6 @@ int sys_munmap(void *p, size_t n) {
} else {
rc = __sys_munmap(p, n);
}
if (!rc)
__virtualsize -= n;
KERNTRACE("sys_munmap(%p, %'zu) → %d", p, n, rc);
return rc;
}

View file

@ -1,82 +0,0 @@
/*-*- mode:c;indent-tabs-mode:nil;c-basic-offset:2;tab-width:8;coding:utf-8 -*-│
vi: set et ft=c ts=2 sts=2 sw=2 fenc=utf-8 :vi
Copyright 2020 Justine Alexandra Roberts Tunney
Permission to use, copy, modify, and/or distribute this software for
any purpose with or without fee is hereby granted, provided that the
above copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL
WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE
AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL
DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR
PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER
TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
PERFORMANCE OF THIS SOFTWARE.
*/
#include "libc/calls/ucontext.h"
#include "libc/log/libfatal.internal.h"
#include "libc/nt/struct/context.h"
#include "libc/str/str.h"
#ifdef __x86_64__
textwindows void _ntcontext2linux(ucontext_t *ctx, const struct NtContext *cr) {
if (!cr)
return;
ctx->uc_mcontext.eflags = cr->EFlags;
ctx->uc_mcontext.rax = cr->Rax;
ctx->uc_mcontext.rbx = cr->Rbx;
ctx->uc_mcontext.rcx = cr->Rcx;
ctx->uc_mcontext.rdx = cr->Rdx;
ctx->uc_mcontext.rdi = cr->Rdi;
ctx->uc_mcontext.rsi = cr->Rsi;
ctx->uc_mcontext.rbp = cr->Rbp;
ctx->uc_mcontext.rsp = cr->Rsp;
ctx->uc_mcontext.rip = cr->Rip;
ctx->uc_mcontext.r8 = cr->R8;
ctx->uc_mcontext.r9 = cr->R9;
ctx->uc_mcontext.r10 = cr->R10;
ctx->uc_mcontext.r11 = cr->R11;
ctx->uc_mcontext.r12 = cr->R12;
ctx->uc_mcontext.r13 = cr->R13;
ctx->uc_mcontext.r14 = cr->R14;
ctx->uc_mcontext.r15 = cr->R15;
ctx->uc_mcontext.cs = cr->SegCs;
ctx->uc_mcontext.gs = cr->SegGs;
ctx->uc_mcontext.fs = cr->SegFs;
ctx->uc_mcontext.fpregs = &ctx->__fpustate;
__repmovsb(&ctx->__fpustate, &cr->FltSave, sizeof(ctx->__fpustate));
ctx->__fpustate.mxcsr = cr->MxCsr;
}
textwindows void _ntlinux2context(struct NtContext *cr, const ucontext_t *ctx) {
if (!cr)
return;
cr->EFlags = ctx->uc_mcontext.eflags;
cr->Rax = ctx->uc_mcontext.rax;
cr->Rbx = ctx->uc_mcontext.rbx;
cr->Rcx = ctx->uc_mcontext.rcx;
cr->Rdx = ctx->uc_mcontext.rdx;
cr->Rdi = ctx->uc_mcontext.rdi;
cr->Rsi = ctx->uc_mcontext.rsi;
cr->Rbp = ctx->uc_mcontext.rbp;
cr->Rsp = ctx->uc_mcontext.rsp;
cr->Rip = ctx->uc_mcontext.rip;
cr->R8 = ctx->uc_mcontext.r8;
cr->R9 = ctx->uc_mcontext.r9;
cr->R10 = ctx->uc_mcontext.r10;
cr->R11 = ctx->uc_mcontext.r11;
cr->R12 = ctx->uc_mcontext.r12;
cr->R13 = ctx->uc_mcontext.r13;
cr->R14 = ctx->uc_mcontext.r14;
cr->R15 = ctx->uc_mcontext.r15;
cr->SegCs = ctx->uc_mcontext.cs;
cr->SegGs = ctx->uc_mcontext.gs;
cr->SegFs = ctx->uc_mcontext.fs;
cr->MxCsr = ctx->__fpustate.mxcsr;
__repmovsb(&cr->FltSave, &ctx->__fpustate, sizeof(ctx->__fpustate));
}
#endif /* __x86_64__ */

View file

@ -16,39 +16,92 @@
TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
PERFORMANCE OF THIS SOFTWARE.
*/
#include "libc/dce.h"
#include "libc/fmt/conv.h"
#include "libc/fmt/itoa.h"
#include "libc/intrin/bsr.h"
#include "libc/intrin/describeflags.h"
#include "libc/intrin/kprintf.h"
#include "libc/intrin/maps.h"
#include "libc/macros.h"
#include "libc/runtime/memtrack.internal.h"
#include "libc/runtime/runtime.h"
#include "libc/sysv/consts/auxv.h"
/**
* Prints memory mappings.
*/
void __print_maps(size_t limit) {
char mappingbuf[8], sb[16];
__maps_lock();
// this will usually return 12 since x86 pml4t uses a 47 bit address
// space in userspace, and decent arm machines uses a 48 bit address
// space. however it could go lower on embedded devices. it can also
// rise higher on expensive x86 machines with pml5t, if user uses it
static int get_address_digits(int pagesz) {
int max_bits = 0;
for (struct Tree *e = tree_first(__maps.maps); e; e = tree_next(e)) {
struct Map *map = MAP_TREE_CONTAINER(e);
kprintf("%012lx-%012lx %!s", map->addr, map->addr + map->size,
char *end = map->addr + ((map->size + pagesz - 1) & -pagesz);
int bits = bsrll((uintptr_t)end) + 1;
if (bits > max_bits)
max_bits = bits;
}
return ((max_bits + 3) & -4) / 4;
}
/**
* Prints memory mappings known to cosmo.
*/
void __print_maps(size_t limit) {
__maps_lock();
char sb[16];
char mappingbuf[8];
struct Map *last = 0;
int pagesz = __pagesize;
int gransz = __gransize;
int digs = get_address_digits(pagesz);
for (struct Tree *e = tree_first(__maps.maps); e; e = tree_next(e)) {
struct Map *map = MAP_TREE_CONTAINER(e);
// show gaps between maps
if (last) {
char *beg = last->addr + ((last->size + gransz - 1) & -gransz);
char *end = map->addr;
if (end > beg) {
size_t gap = end - beg;
sizefmt(sb, gap, 1024);
kprintf("%0*lx-%0*lx %sb\n", digs, beg, digs, end, sb);
}
}
last = map;
// show mapping
kprintf("%0*lx-%0*lx %!s", digs, map->addr, digs, map->addr + map->size,
_DescribeMapping(mappingbuf, map->prot, map->flags));
sizefmt(sb, map->size, 1024);
kprintf(" %!sb", sb);
if (map->hand && map->hand != -1)
kprintf(" hand=%ld", map->hand);
if (IsWindows()) {
switch (map->hand) {
case MAPS_RESERVATION:
kprintf(" reservation");
break;
case MAPS_SUBREGION:
break;
case MAPS_VIRTUAL:
kprintf(" virtual");
break;
default:
kprintf(" hand=%ld", map->hand);
break;
}
}
if (map->iscow)
kprintf(" cow");
if (map->readonlyfile)
kprintf(" readonlyfile");
kprintf("\n");
// stay beneath our limit
if (!--limit)
break;
}
kprintf("# %'zu bytes in %'zu mappings\n", __maps.pages * __pagesize,
// print summary
kprintf("# %'zu bytes in %'zu mappings\n", __maps.pages * pagesz,
__maps.count);
__maps_unlock();
}

View file

@ -23,6 +23,7 @@
#include "libc/nt/enum/memflags.h"
#include "libc/nt/memory.h"
#include "libc/runtime/runtime.h"
#include "libc/stdio/sysparam.h"
#include "libc/str/str.h"
static const struct DescribeFlags kNtMemState[] = {
@ -46,20 +47,25 @@ const char *DescribeNtMemType(char buf[64], uint32_t x) {
return _DescribeFlags(buf, 64, kNtMemType, ARRAYLEN(kNtMemType), "kNtMem", x);
}
void __print_maps_win32(void) {
void __print_maps_win32(int64_t hProcess, const char *addr, size_t size) {
char *p, b[5][64];
struct NtMemoryBasicInformation mi;
kprintf("%-12s %-12s %10s %16s %16s %32s %32s\n", "Allocation", "BaseAddress",
"RegionSize", "State", "Type", "AllocationProtect", "Protect");
for (p = 0;; p = (char *)mi.BaseAddress + mi.RegionSize) {
bzero(&mi, sizeof(mi));
if (!VirtualQuery(p, &mi, sizeof(mi)))
if (!VirtualQueryEx(hProcess, p, &mi, sizeof(mi)))
break;
sizefmt(b[0], mi.RegionSize, 1024);
kprintf("%.12lx %.12lx %10s %16s %16s %32s %32s\n", mi.AllocationBase,
kprintf("%.12lx %.12lx %10s %16s %16s %32s %32s%s\n", mi.AllocationBase,
mi.BaseAddress, b[0], DescribeNtMemState(b[1], mi.State),
DescribeNtMemType(b[2], mi.Type),
_DescribeNtPageFlags(b[3], mi.AllocationProtect),
_DescribeNtPageFlags(b[4], mi.Protect));
_DescribeNtPageFlags(b[4], mi.Protect),
(mi.State != kNtMemFree &&
MAX(addr, (const char *)mi.BaseAddress) <
MIN(addr + size, (const char *)mi.BaseAddress + mi.RegionSize))
? " [OVERLAPS]"
: "");
}
}

View file

@ -17,69 +17,89 @@
PERFORMANCE OF THIS SOFTWARE.
*/
#include "libc/calls/blockcancel.internal.h"
#include "libc/calls/calls.h"
#include "libc/calls/state.internal.h"
#include "libc/cosmo.h"
#include "libc/dce.h"
#include "libc/errno.h"
#include "libc/intrin/atomic.h"
#include "libc/intrin/describeflags.h"
#include "libc/intrin/kprintf.h"
#include "libc/intrin/strace.h"
#include "libc/intrin/weaken.h"
#include "libc/macros.h"
#include "libc/runtime/internal.h"
#include "libc/thread/lock.h"
#include "libc/thread/posixthread.internal.h"
#include "libc/thread/thread.h"
#include "libc/thread/tls.h"
#include "third_party/nsync/mu.h"
static errno_t pthread_mutex_lock_normal_success(pthread_mutex_t *mutex,
uint64_t word) {
if (IsModeDbg() || MUTEX_TYPE(word) == PTHREAD_MUTEX_ERRORCHECK) {
__deadlock_track(mutex, MUTEX_TYPE(word) == PTHREAD_MUTEX_ERRORCHECK);
__deadlock_record(mutex, MUTEX_TYPE(word) == PTHREAD_MUTEX_ERRORCHECK);
}
return 0;
}
// see "take 3" algorithm in "futexes are tricky" by ulrich drepper
// slightly improved to attempt acquiring multiple times b4 syscall
static void pthread_mutex_lock_drepper(atomic_int *futex, char pshare) {
int word = 0;
static int pthread_mutex_lock_drepper(pthread_mutex_t *mutex, uint64_t word,
bool is_trylock) {
int val = 0;
if (atomic_compare_exchange_strong_explicit(
futex, &word, 1, memory_order_acquire, memory_order_acquire))
return;
LOCKTRACE("acquiring pthread_mutex_lock_drepper(%t)...", futex);
if (word == 1)
word = atomic_exchange_explicit(futex, 2, memory_order_acquire);
&mutex->_futex, &val, 1, memory_order_acquire, memory_order_acquire))
return pthread_mutex_lock_normal_success(mutex, word);
if (is_trylock)
return EBUSY;
LOCKTRACE("acquiring pthread_mutex_lock_drepper(%t)...", mutex);
if (val == 1)
val = atomic_exchange_explicit(&mutex->_futex, 2, memory_order_acquire);
BLOCK_CANCELATION;
while (word > 0) {
cosmo_futex_wait(futex, 2, pshare, 0, 0);
word = atomic_exchange_explicit(futex, 2, memory_order_acquire);
while (val > 0) {
cosmo_futex_wait(&mutex->_futex, 2, MUTEX_PSHARED(word), 0, 0);
val = atomic_exchange_explicit(&mutex->_futex, 2, memory_order_acquire);
}
ALLOW_CANCELATION;
return pthread_mutex_lock_normal_success(mutex, word);
}
static errno_t pthread_mutex_lock_recursive(pthread_mutex_t *mutex,
uint64_t word) {
uint64_t word, bool is_trylock) {
uint64_t lock;
int backoff = 0;
int me = gettid();
int me = atomic_load_explicit(&__get_tls()->tib_ptid, memory_order_relaxed);
bool once = false;
for (;;) {
if (MUTEX_OWNER(word) == me) {
if (MUTEX_TYPE(word) != PTHREAD_MUTEX_ERRORCHECK) {
if (MUTEX_DEPTH(word) < MUTEX_DEPTH_MAX) {
if (atomic_compare_exchange_weak_explicit(
&mutex->_word, &word, MUTEX_INC_DEPTH(word),
memory_order_relaxed, memory_order_relaxed))
return 0;
continue;
} else {
return EAGAIN;
}
if (MUTEX_DEPTH(word) < MUTEX_DEPTH_MAX) {
if (atomic_compare_exchange_weak_explicit(
&mutex->_word, &word, MUTEX_INC_DEPTH(word),
memory_order_relaxed, memory_order_relaxed))
return 0;
continue;
} else {
return EDEADLK;
return EAGAIN;
}
}
if (IsModeDbg())
__deadlock_check(mutex, 0);
word = MUTEX_UNLOCK(word);
lock = MUTEX_LOCK(word);
lock = MUTEX_SET_OWNER(lock, me);
if (atomic_compare_exchange_weak_explicit(&mutex->_word, &word, lock,
memory_order_acquire,
memory_order_relaxed)) {
if (IsModeDbg()) {
__deadlock_track(mutex, 0);
__deadlock_record(mutex, 0);
}
mutex->_pid = __pid;
return 0;
}
if (is_trylock)
return EBUSY;
if (!once) {
LOCKTRACE("acquiring pthread_mutex_lock_recursive(%t)...", mutex);
once = true;
@ -97,25 +117,33 @@ static errno_t pthread_mutex_lock_recursive(pthread_mutex_t *mutex,
#if PTHREAD_USE_NSYNC
static errno_t pthread_mutex_lock_recursive_nsync(pthread_mutex_t *mutex,
uint64_t word) {
int me = gettid();
uint64_t word,
bool is_trylock) {
int me = atomic_load_explicit(&__get_tls()->tib_ptid, memory_order_relaxed);
for (;;) {
if (MUTEX_OWNER(word) == me) {
if (MUTEX_TYPE(word) != PTHREAD_MUTEX_ERRORCHECK) {
if (MUTEX_DEPTH(word) < MUTEX_DEPTH_MAX) {
if (atomic_compare_exchange_weak_explicit(
&mutex->_word, &word, MUTEX_INC_DEPTH(word),
memory_order_relaxed, memory_order_relaxed))
return 0;
continue;
} else {
return EAGAIN;
}
if (MUTEX_DEPTH(word) < MUTEX_DEPTH_MAX) {
if (atomic_compare_exchange_weak_explicit(
&mutex->_word, &word, MUTEX_INC_DEPTH(word),
memory_order_relaxed, memory_order_relaxed))
return 0;
continue;
} else {
return EDEADLK;
return EAGAIN;
}
}
_weaken(nsync_mu_lock)((nsync_mu *)mutex->_nsyncx);
if (IsModeDbg())
__deadlock_check(mutex, 0);
if (!is_trylock) {
_weaken(nsync_mu_lock)((nsync_mu *)mutex->_nsync);
} else {
if (!_weaken(nsync_mu_trylock)((nsync_mu *)mutex->_nsync))
return EBUSY;
}
if (IsModeDbg()) {
__deadlock_track(mutex, 0);
__deadlock_record(mutex, 0);
}
word = MUTEX_UNLOCK(word);
word = MUTEX_LOCK(word);
word = MUTEX_SET_OWNER(word, me);
@ -126,69 +154,82 @@ static errno_t pthread_mutex_lock_recursive_nsync(pthread_mutex_t *mutex,
}
#endif
static errno_t pthread_mutex_lock_impl(pthread_mutex_t *mutex) {
uint64_t word;
static errno_t pthread_mutex_lock_impl(pthread_mutex_t *mutex,
bool is_trylock) {
uint64_t word = atomic_load_explicit(&mutex->_word, memory_order_relaxed);
// get current state of lock
word = atomic_load_explicit(&mutex->_word, memory_order_relaxed);
// handle recursive mutexes
if (MUTEX_TYPE(word) == PTHREAD_MUTEX_RECURSIVE) {
#if PTHREAD_USE_NSYNC
if (_weaken(nsync_mu_lock) &&
MUTEX_PSHARED(word) == PTHREAD_PROCESS_PRIVATE) {
return pthread_mutex_lock_recursive_nsync(mutex, word, is_trylock);
} else {
return pthread_mutex_lock_recursive(mutex, word, is_trylock);
}
#else
return pthread_mutex_lock_recursive(mutex, word, is_trylock);
#endif
}
// check if normal mutex is already owned by calling thread
if (!is_trylock &&
(MUTEX_TYPE(word) == PTHREAD_MUTEX_ERRORCHECK ||
(IsModeDbg() && MUTEX_TYPE(word) == PTHREAD_MUTEX_DEFAULT))) {
if (__deadlock_tracked(mutex) == 1) {
if (IsModeDbg() && MUTEX_TYPE(word) != PTHREAD_MUTEX_ERRORCHECK) {
kprintf("error: attempted to lock non-recursive mutex that's already "
"held by the calling thread: %t\n",
mutex);
DebugBreak();
}
return EDEADLK;
}
}
// check if locking will create cycle in lock graph
if (IsModeDbg() || MUTEX_TYPE(word) == PTHREAD_MUTEX_ERRORCHECK)
if (__deadlock_check(mutex, MUTEX_TYPE(word) == PTHREAD_MUTEX_ERRORCHECK))
return EDEADLK;
#if PTHREAD_USE_NSYNC
// use superior mutexes if possible
if (MUTEX_TYPE(word) == PTHREAD_MUTEX_NORMAL && //
MUTEX_PSHARED(word) == PTHREAD_PROCESS_PRIVATE && //
if (MUTEX_PSHARED(word) == PTHREAD_PROCESS_PRIVATE &&
_weaken(nsync_mu_lock)) {
// on apple silicon we should just put our faith in ulock
// otherwise *nsync gets struck down by the eye of sauron
if (!IsXnuSilicon()) {
_weaken(nsync_mu_lock)((nsync_mu *)mutex);
return 0;
if (!is_trylock) {
_weaken(nsync_mu_lock)((nsync_mu *)mutex->_nsync);
return pthread_mutex_lock_normal_success(mutex, word);
} else {
if (_weaken(nsync_mu_trylock)((nsync_mu *)mutex->_nsync))
return pthread_mutex_lock_normal_success(mutex, word);
return EBUSY;
}
}
}
#endif
// handle normal mutexes
if (MUTEX_TYPE(word) == PTHREAD_MUTEX_NORMAL) {
pthread_mutex_lock_drepper(&mutex->_futex, MUTEX_PSHARED(word));
return 0;
}
// handle recursive and error checking mutexes
#if PTHREAD_USE_NSYNC
if (_weaken(nsync_mu_lock) &&
MUTEX_PSHARED(word) == PTHREAD_PROCESS_PRIVATE) {
return pthread_mutex_lock_recursive_nsync(mutex, word);
} else {
return pthread_mutex_lock_recursive(mutex, word);
}
#else
return pthread_mutex_lock_recursive(mutex, word);
#endif
// isc licensed non-recursive mutex implementation
return pthread_mutex_lock_drepper(mutex, word, is_trylock);
}
/**
* Locks mutex.
*
* Here's an example of using a normal mutex:
* Locks mutex, e.g.
*
* pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER;
* pthread_mutex_lock(&lock);
* // do work...
* pthread_mutex_unlock(&lock);
* pthread_mutex_destroy(&lock);
*
* Cosmopolitan permits succinct notation for normal mutexes:
*
* pthread_mutex_t lock = {0};
* pthread_mutex_lock(&lock);
* // do work...
* pthread_mutex_unlock(&lock);
*
* Here's an example of the proper way to do recursive mutexes:
* The long way to do that is:
*
* pthread_mutex_t lock;
* pthread_mutexattr_t attr;
* pthread_mutexattr_init(&attr);
* pthread_mutexattr_settype(&attr, PTHREAD_MUTEX_RECURSIVE);
* pthread_mutexattr_settype(&attr, PTHREAD_MUTEX_DEFAULT);
* pthread_mutexattr_setpshared(&attr, PTHREAD_PROCESS_PRIVATE);
* pthread_mutex_init(&lock, &attr);
* pthread_mutexattr_destroy(&attr);
* pthread_mutex_lock(&lock);
@ -196,28 +237,101 @@ static errno_t pthread_mutex_lock_impl(pthread_mutex_t *mutex) {
* pthread_mutex_unlock(&lock);
* pthread_mutex_destroy(&lock);
*
* This function does nothing in vfork() children.
* The following non-POSIX initializers are also provided by cosmo libc:
*
* You can debug locks the acquisition of locks by building your program
* with `cosmocc -mdbg` and passing the `--strace` flag to your program.
* This will cause a line to be logged each time a mutex or spin lock is
* locked or unlocked. When locking, this is printed after the lock gets
* acquired. The entry to the lock operation will be logged too but only
* if the lock couldn't be immediately acquired. Lock logging works best
* when `mutex` refers to a static variable, in which case its name will
* be printed in the log.
* - `PTHREAD_RECURSIVE_MUTEX_INITIALIZER_NP`
* - `PTHREAD_ERRORCHECK_MUTEX_INITIALIZER_NP`
* - `PTHREAD_NORMAL_MUTEX_INITIALIZER_NP`
*
* Locking a mutex that's already locked by the calling thread will make
* the thread hang indefinitely, i.e. it's a deadlock condition. You can
* use `PTHREAD_MUTEX_RECURSIVE` to allow recursive locking, which could
* result in somewhat less performance. An alternative solution is using
* the `PTHREAD_MUTEX_ERRORCHECK` mode, which raises `EDEADLK` for that.
*
* If a thread locks a mutex while other mutexes are already locked then
* you need to observe a consistent global ordering, otherwise deadlocks
* might occur. The Cosmopolitan runtime can detect these cycles quickly
* so you can fix your code before it becomes an issue. With error check
* mode, an EPERM will be returned. If your app is using `cosmocc -mdbg`
* then an error message will be printed including the demangled symbols
* of the mutexes in the strongly connected component that was detected.
* Please note that, even for debug builds mutexes set to explicitly use
* the `PTHREAD_MUTEX_ERRORCHECK` mode will return an error code instead
* which means the cosmo debug mode only influences undefined behaviors.
*
* Cosmopolitan only supports error checking on mutexes stored in static
* memory, i.e. your `mutex` pointer must point inside the .data or .bss
* sections of your executable. When compiling your programs using -mdbg
* all your locks will gain error checking automatically. When deadlocks
* are detected an error message will be printed and a SIGTRAP signal is
* raised, which may be ignored to force EDEADLK and EPERM to be raised.
*
* Using `cosmocc -mdbg` also enhances `--strace` with information about
* mutexes. First, locks and unlocks will be logged. Since the lock line
* only appears after the lock is acquired, that might mean you'll never
* get an indication about a lock that takes a very long time to acquire
* so, whenever a lock can't immediately be acquired, a second line gets
* printed *before* the lock is acquired to let you know that the thread
* is waiting for a particular lock. If your mutex object resides within
* static memory, then its demangled symbol name will be printed. If you
* call ShowCrashReports() at the beginning of your main() function then
* you'll also see a backtrace when a locking violation occurs. When the
* symbols in the violation error messages show up as numbers, and it is
* desirable to see demangled symbols without enabling full crash report
* functionality the GetSymbolTable() function may be called for effect.
*
* If you use `PTHREAD_MUTEX_NORMAL`, instead of `PTHREAD_MUTEX_DEFAULT`
* then deadlocking is actually defined behavior according to POSIX.1 so
* the helpfulness of `cosmocc -mdbg` will be somewhat weakened.
*
* If your `mutex` object resides in `MAP_SHARED` memory, then undefined
* behavior will happen unless you use `PTHREAD_PROCESS_SHARED` mode, if
* the lock is used by multiple processes.
*
* This function does nothing when the process is in vfork() mode.
*
* @return 0 on success, or error number on failure
* @raise EDEADLK if mutex is recursive and locked by another thread
* @raise EDEADLK if mutex is non-recursive and locked by current thread
* @raise EDEADLK if cycle is detected in global nested lock graph
* @raise EAGAIN if maximum recursive locks is exceeded
* @see pthread_spin_lock()
* @vforksafe
*/
errno_t pthread_mutex_lock(pthread_mutex_t *mutex) {
if (!__vforked) {
errno_t err = pthread_mutex_lock_impl(mutex);
errno_t _pthread_mutex_lock(pthread_mutex_t *mutex) {
if (__tls_enabled && !__vforked) {
errno_t err = pthread_mutex_lock_impl(mutex, false);
LOCKTRACE("pthread_mutex_lock(%t) → %s", mutex, DescribeErrno(err));
return err;
} else {
LOCKTRACE("skipping pthread_mutex_lock(%t) due to vfork", mutex);
LOCKTRACE("skipping pthread_mutex_lock(%t) due to runtime state", mutex);
return 0;
}
}
/**
* Attempts acquiring lock.
*
* Unlike pthread_mutex_lock() this function won't block and instead
* returns an error immediately if the lock couldn't be acquired.
*
* @return 0 if lock was acquired, otherwise an errno
* @raise EBUSY if lock is currently held by another thread
* @raise EAGAIN if maximum number of recursive locks is held
* @raise EDEADLK if `mutex` is `PTHREAD_MUTEX_ERRORCHECK` and the
* current thread already holds this mutex
*/
errno_t _pthread_mutex_trylock(pthread_mutex_t *mutex) {
if (__tls_enabled && !__vforked) {
errno_t err = pthread_mutex_lock_impl(mutex, true);
LOCKTRACE("pthread_mutex_trylock(%t) → %s", mutex, DescribeErrno(err));
return err;
} else {
LOCKTRACE("skipping pthread_mutex_trylock(%t) due to runtime state", mutex);
return 0;
}
}
__weak_reference(_pthread_mutex_lock, pthread_mutex_lock);
__weak_reference(_pthread_mutex_trylock, pthread_mutex_trylock);

View file

@ -1,152 +0,0 @@
/*-*- mode:c;indent-tabs-mode:nil;c-basic-offset:2;tab-width:8;coding:utf-8 -*-│
vi: set et ft=c ts=2 sts=2 sw=2 fenc=utf-8 :vi
Copyright 2023 Justine Alexandra Roberts Tunney
Permission to use, copy, modify, and/or distribute this software for
any purpose with or without fee is hereby granted, provided that the
above copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL
WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE
AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL
DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR
PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER
TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
PERFORMANCE OF THIS SOFTWARE.
*/
#include "libc/calls/calls.h"
#include "libc/dce.h"
#include "libc/errno.h"
#include "libc/intrin/atomic.h"
#include "libc/intrin/weaken.h"
#include "libc/runtime/internal.h"
#include "libc/thread/lock.h"
#include "libc/thread/thread.h"
#include "third_party/nsync/mu.h"
static errno_t pthread_mutex_trylock_drepper(atomic_int *futex) {
int word = 0;
if (atomic_compare_exchange_strong_explicit(
futex, &word, 1, memory_order_acquire, memory_order_acquire))
return 0;
return EBUSY;
}
static errno_t pthread_mutex_trylock_recursive(pthread_mutex_t *mutex,
uint64_t word) {
uint64_t lock;
int me = gettid();
for (;;) {
if (MUTEX_OWNER(word) == me) {
if (MUTEX_TYPE(word) != PTHREAD_MUTEX_ERRORCHECK) {
if (MUTEX_DEPTH(word) < MUTEX_DEPTH_MAX) {
if (atomic_compare_exchange_weak_explicit(
&mutex->_word, &word, MUTEX_INC_DEPTH(word),
memory_order_relaxed, memory_order_relaxed))
return 0;
continue;
} else {
return EAGAIN;
}
} else {
return EDEADLK;
}
}
word = MUTEX_UNLOCK(word);
lock = MUTEX_LOCK(word);
lock = MUTEX_SET_OWNER(lock, me);
if (atomic_compare_exchange_weak_explicit(&mutex->_word, &word, lock,
memory_order_acquire,
memory_order_relaxed)) {
mutex->_pid = __pid;
return 0;
}
return EBUSY;
}
}
static errno_t pthread_mutex_trylock_recursive_nsync(pthread_mutex_t *mutex,
uint64_t word) {
int me = gettid();
for (;;) {
if (MUTEX_OWNER(word) == me) {
if (MUTEX_TYPE(word) != PTHREAD_MUTEX_ERRORCHECK) {
if (MUTEX_DEPTH(word) < MUTEX_DEPTH_MAX) {
if (atomic_compare_exchange_weak_explicit(
&mutex->_word, &word, MUTEX_INC_DEPTH(word),
memory_order_relaxed, memory_order_relaxed))
return 0;
continue;
} else {
return EAGAIN;
}
} else {
return EDEADLK;
}
}
if (_weaken(nsync_mu_trylock)((nsync_mu *)mutex->_nsyncx)) {
word = MUTEX_UNLOCK(word);
word = MUTEX_LOCK(word);
word = MUTEX_SET_OWNER(word, me);
mutex->_word = word;
mutex->_pid = __pid;
return 0;
} else {
return EBUSY;
}
}
}
/**
* Attempts acquiring lock.
*
* Unlike pthread_mutex_lock() this function won't block and instead
* returns an error immediately if the lock couldn't be acquired.
*
* @return 0 if lock was acquired, otherwise an errno
* @raise EAGAIN if maximum number of recursive locks is held
* @raise EBUSY if lock is currently held in read or write mode
* @raise EINVAL if `mutex` doesn't refer to an initialized lock
* @raise EDEADLK if `mutex` is `PTHREAD_MUTEX_ERRORCHECK` and the
* current thread already holds this mutex
*/
errno_t pthread_mutex_trylock(pthread_mutex_t *mutex) {
// get current state of lock
uint64_t word = atomic_load_explicit(&mutex->_word, memory_order_relaxed);
#if PTHREAD_USE_NSYNC
// use superior mutexes if possible
if (MUTEX_TYPE(word) == PTHREAD_MUTEX_NORMAL &&
MUTEX_PSHARED(word) == PTHREAD_PROCESS_PRIVATE && //
_weaken(nsync_mu_trylock)) {
// on apple silicon we should just put our faith in ulock
// otherwise *nsync gets struck down by the eye of sauron
if (!IsXnuSilicon()) {
if (_weaken(nsync_mu_trylock)((nsync_mu *)mutex)) {
return 0;
} else {
return EBUSY;
}
}
}
#endif
// handle normal mutexes
if (MUTEX_TYPE(word) == PTHREAD_MUTEX_NORMAL)
return pthread_mutex_trylock_drepper(&mutex->_futex);
// handle recursive and error checking mutexes
#if PTHREAD_USE_NSYNC
if (_weaken(nsync_mu_trylock) &&
MUTEX_PSHARED(word) == PTHREAD_PROCESS_PRIVATE) {
return pthread_mutex_trylock_recursive_nsync(mutex, word);
} else {
return pthread_mutex_trylock_recursive(mutex, word);
}
#else
return pthread_mutex_trylock_recursive(mutex, word);
#endif
}

View file

@ -22,11 +22,15 @@
#include "libc/dce.h"
#include "libc/errno.h"
#include "libc/intrin/atomic.h"
#include "libc/intrin/describeflags.h"
#include "libc/intrin/kprintf.h"
#include "libc/intrin/strace.h"
#include "libc/intrin/weaken.h"
#include "libc/runtime/internal.h"
#include "libc/thread/lock.h"
#include "libc/thread/posixthread.internal.h"
#include "libc/thread/thread.h"
#include "libc/thread/tls.h"
#include "third_party/nsync/mu.h"
// see "take 3" algorithm in "futexes are tricky" by ulrich drepper
@ -40,7 +44,7 @@ static void pthread_mutex_unlock_drepper(atomic_int *futex, char pshare) {
static errno_t pthread_mutex_unlock_recursive(pthread_mutex_t *mutex,
uint64_t word) {
int me = gettid();
int me = atomic_load_explicit(&__get_tls()->tib_ptid, memory_order_relaxed);
for (;;) {
// we allow unlocking an initialized lock that wasn't locked, but we
@ -61,15 +65,18 @@ static errno_t pthread_mutex_unlock_recursive(pthread_mutex_t *mutex,
// actually unlock the mutex
if (atomic_compare_exchange_weak_explicit(
&mutex->_word, &word, MUTEX_UNLOCK(word), memory_order_release,
memory_order_relaxed))
memory_order_relaxed)) {
if (IsModeDbg())
__deadlock_untrack(mutex);
return 0;
}
}
}
#if PTHREAD_USE_NSYNC
static errno_t pthread_mutex_unlock_recursive_nsync(pthread_mutex_t *mutex,
uint64_t word) {
int me = gettid();
int me = atomic_load_explicit(&__get_tls()->tib_ptid, memory_order_relaxed);
for (;;) {
// we allow unlocking an initialized lock that wasn't locked, but we
@ -89,63 +96,87 @@ static errno_t pthread_mutex_unlock_recursive_nsync(pthread_mutex_t *mutex,
// actually unlock the mutex
mutex->_word = MUTEX_UNLOCK(word);
_weaken(nsync_mu_unlock)((nsync_mu *)mutex->_nsyncx);
_weaken(nsync_mu_unlock)((nsync_mu *)mutex->_nsync);
if (IsModeDbg())
__deadlock_untrack(mutex);
return 0;
}
}
#endif
/**
* Releases mutex.
*
* This function does nothing in vfork() children.
*
* @return 0 on success or error number on failure
* @raises EPERM if in error check mode and not owned by caller
* @vforksafe
*/
errno_t pthread_mutex_unlock(pthread_mutex_t *mutex) {
uint64_t word;
static errno_t pthread_mutex_unlock_impl(pthread_mutex_t *mutex) {
uint64_t word = atomic_load_explicit(&mutex->_word, memory_order_relaxed);
if (__vforked) {
LOCKTRACE("skipping pthread_mutex_lock(%t) due to vfork", mutex);
return 0;
// check if mutex isn't held by calling thread
if (MUTEX_TYPE(word) == PTHREAD_MUTEX_ERRORCHECK || IsModeDbg()) {
if (__deadlock_tracked(mutex) == 0) {
if (IsModeDbg() && MUTEX_TYPE(word) != PTHREAD_MUTEX_ERRORCHECK) {
kprintf("error: unlock mutex not owned by calling thread: %t\n", mutex);
DebugBreak();
}
return EPERM;
}
}
LOCKTRACE("pthread_mutex_unlock(%t)", mutex);
// get current state of lock
word = atomic_load_explicit(&mutex->_word, memory_order_relaxed);
// handle recursive mutexes
if (MUTEX_TYPE(word) == PTHREAD_MUTEX_RECURSIVE) {
#if PTHREAD_USE_NSYNC
if (_weaken(nsync_mu_unlock) &&
MUTEX_PSHARED(word) == PTHREAD_PROCESS_PRIVATE) {
return pthread_mutex_unlock_recursive_nsync(mutex, word);
} else {
return pthread_mutex_unlock_recursive(mutex, word);
}
#else
return pthread_mutex_unlock_recursive(mutex, word);
#endif
}
#if PTHREAD_USE_NSYNC
// use superior mutexes if possible
if (MUTEX_TYPE(word) == PTHREAD_MUTEX_NORMAL && //
MUTEX_PSHARED(word) == PTHREAD_PROCESS_PRIVATE && //
if (MUTEX_PSHARED(word) == PTHREAD_PROCESS_PRIVATE && //
_weaken(nsync_mu_unlock)) {
// on apple silicon we should just put our faith in ulock
// otherwise *nsync gets struck down by the eye of sauron
if (!IsXnuSilicon()) {
_weaken(nsync_mu_unlock)((nsync_mu *)mutex);
_weaken(nsync_mu_unlock)((nsync_mu *)mutex->_nsync);
if (MUTEX_TYPE(word) == PTHREAD_MUTEX_ERRORCHECK || IsModeDbg())
__deadlock_untrack(mutex);
return 0;
}
}
#endif
// implement barebones normal mutexes
if (MUTEX_TYPE(word) == PTHREAD_MUTEX_NORMAL) {
pthread_mutex_unlock_drepper(&mutex->_futex, MUTEX_PSHARED(word));
pthread_mutex_unlock_drepper(&mutex->_futex, MUTEX_PSHARED(word));
if (MUTEX_TYPE(word) == PTHREAD_MUTEX_ERRORCHECK || IsModeDbg())
__deadlock_untrack(mutex);
return 0;
}
/**
* Releases mutex.
*
* POSIX.1 says it's undefined behavior to unlock a mutex that wasn't
* locked by the calling thread. Therefore, if `mutex` isn't locked, or
* it is locked and the thing that locked it was a different thread or
* process, then you should expect your program to deadlock or crash.
*
* This function does nothing in vfork() children.
*
* @return 0 on success or error number on failure
* @raises EPERM if mutex ownership isn't acceptable
* @vforksafe
*/
errno_t _pthread_mutex_unlock(pthread_mutex_t *mutex) {
if (__tls_enabled && !__vforked) {
errno_t err = pthread_mutex_unlock_impl(mutex);
LOCKTRACE("pthread_mutex_unlock(%t) → %s", mutex, DescribeErrno(err));
return err;
} else {
LOCKTRACE("skipping pthread_mutex_lock(%t) due to runtime state", mutex);
return 0;
}
// handle recursive and error checking mutexes
#if PTHREAD_USE_NSYNC
if (_weaken(nsync_mu_unlock) &&
MUTEX_PSHARED(word) == PTHREAD_PROCESS_PRIVATE) {
return pthread_mutex_unlock_recursive_nsync(mutex, word);
} else {
return pthread_mutex_unlock_recursive(mutex, word);
}
#else
return pthread_mutex_unlock_recursive(mutex, word);
#endif
}
__weak_reference(_pthread_mutex_unlock, pthread_mutex_unlock);

Some files were not shown because too many files have changed in this diff Show more