Commit graph

731 commits

Author SHA1 Message Date
Justine Tunney
82b1e61443
Fix Landlock Make build config issue 2023-07-23 16:43:22 -07:00
Justine Tunney
0ba3199915
Fix some more socket bugs
- The functions that return a sockaddr now do so the same way the Linux
  Kernel does across platforms, e.g. getpeername(), accept4()

- Socket system calls on Windows will now only check for interrupts when
  a blocking operation needs to be performed.

- Write tests for recvfrom() system call
2023-07-23 16:31:10 -07:00
Justine Tunney
1d4eb08fa1
Support non-blocking i/o across platforms
This change introduces new tests for `O_NONBLOCK` and `SOCK_NONBLOCK` to
confirm that non-blocking i/o is now working on all supported platforms,
including Windows. For example, you can now say on Windows, MacOS, etc.:

    socket(AF_INET, SOCK_STREAM | SOCK_NONBLOCK, IPPROTO_TCP);

To create a non-blocking IPv4 TCP socket. Or you can enable non-blocking
i/o on an existing socket / pipe / etc. file descriptor by calling fcntl

    fcntl(fd, F_SETFL, fcntl(fd, F_GETFL) | O_NONBLOCK);

This functionality is polyfilled on older Linux kernels too, e.g. RHEL5.
Now that fcntl() support is much better the FIOCLEX / FIONCLEX polyfills
for ioctl() have been removed since they're ugly non-POSIX diameond APIs

This change fixes a weakness in kprintf() that was causing Windows trace
tools to frequently crash.
2023-07-23 02:56:47 -07:00
Justine Tunney
1ee2e89326
Make improvements
- This commit mints a new release of APE Loader v1.2 which supports
  loading ELF programs with a non-contiguous virtual address layout
  even though we've never been able to take advantage of it, due to
  how `objcopy -SO binary` fills any holes left by PT_LOAD. This'll
  change soon, since we'll have a new way of creating APE binaries.

- The undiamonding trick with our ioctl() implementation is removed
  since POSIX has been killing ioctl() for years and they've done a
  much better job. One problem it resolves, is that ioctl(FIONREAD)
  wasn't working earlier and that caused issues when building Emacs
2023-07-11 04:41:50 -07:00
Justine Tunney
a2d269dc38
Brush up some more code 2023-07-10 10:17:26 -07:00
Justine Tunney
f7ae50462a
Make improvements
- Fix unused local variable errors
- Remove yoinks from sigaction() header
- Add nox87 and aarch64 to github actions
- Fix cosmocc -fportcosmo in linking mode
- It's now possible to build `make m=llvm o/llvm/libc`
2023-07-10 04:35:14 -07:00
Justine Tunney
3dc86ce154
Fix breakages in Linux-only build modes
- compile.com now polyfills -march=native which gcc/clang removed
- Guarantee zero Windows code is linked into non-Windows binaries
- MODE=tinylinux binaries are now back to being as tiny as ~4kb
- Improve the runtime's stack allocation / alignment hack
- GitHub Actions now tests Linux modes for assurance
2023-07-09 19:51:44 -07:00
Justine Tunney
42ba9901e4
Fix some behavioral issues on Windows 2023-07-09 09:59:22 -07:00
Justine Tunney
41396ff48a
Make fixes and improvements
- Fix handling of precision in hex float formatting
- Enhance the cocmd interpreter for system() and popen()
- Manually ran the Lua unit tests, which are now passing
- Let stdio i/o operations happen when file is in error state
- We're now saving and restoring xmm in ftrace out of paranoia
2023-07-09 05:21:11 -07:00
Justine Tunney
a75175fe94
Make build hermetically sealed again
It turned out that Landlock Make hasn't been applying sandboxing for a
while, due to a mistyped if statement for `$(USE_SYSTEM_TOOLCHAIN)` it
should have had the opposite meaning. Regressions in the build configs
have been fixed. The rmrf() function works better now. The rm.com tool
works according to POSIX with the exception of supporting prompts.
2023-07-08 07:06:25 -07:00
Justine Tunney
a186143f62
Add EncodeHex() and DecodeHex() to Redbean 2023-07-06 15:38:08 -07:00
Justine Tunney
00acd81b2f
Delete more dead code 2023-07-06 09:12:28 -07:00
Justine Tunney
0a24b4fc3c
Clean up more code
The *NSYNC linked list API is good enough that it deserves to be part of
the C libray, so this change writes an improved version of it which uses
that offsetof() trick from the Linux Kernel. We vendor all of the *NSYNC
tests in third_party which helped confirm the needed refactoring is safe

This change also deletes more old code that didn't pan out. My goal here
is to work towards a vision where the Cosmopolitan core libraries become
less experimental and more focused on curation. This better reflects the
current level of quality we've managed to achieve.
2023-07-06 08:03:24 -07:00
Justine Tunney
97b7116953
Hunt down more bugs
After going through the MODE=dbg and MODE=zero build modes, a bunch of
little issues were identified, which have been addressed. Fixing those
issues created even more troubles for the project, because it improved
our ability to detect latent problems which are getting fixed so fast.
2023-07-03 18:43:29 -07:00
Justine Tunney
197aa0d465
Implement swapcontext() and makecontext()
This change introduces support for Linux-style uc_context manipulation
that's fast and works well on all supported OSes and architectures. It
also integrates with the Cosmpolitan runtime which can show backtraces
comprised of multiple stacks and fibers. See the test and example code
for further details. This will be used by Mold once it's been vendored
2023-07-02 09:01:44 -07:00
Justine Tunney
7ec84655b4
Get setcontext() and getcontext() working on Aarch64
This change also adds the missing code for getting and restoring the
thread's signal mask, since that's explicitly listed by the man page
2023-07-01 22:53:23 -07:00
Justine Tunney
40eb3b9d5d
Fully support OpenBSD 7.3
This change (1) upgrades to OpenBSD's newer kernel ABIs, and (2)
modifies APE to have a read-only data segment. Doing this required
creating APE Loader v1.1, which is backwards and forwards compatible
with the previous version.

If you've run the following commands in the past to install your APE
Loader systemwide, then you need to run them again. Ad-hoc installations
shouldn't be impacted. It's also recommended that APE binaries be remade
after upgrading, since they embed old versions of the APE Loader.

    ape/apeuninstall.sh
    ape/apeinstall.sh

This change does more than just fix OpenBSD. The new loader is smarter
and more reliable. We're now able create much tinier ELF and Mach-O data
structures than we could before. Both APE Loader and execvpe() will now
normalize ambiguous argv[0] resolution the same way as the UNIX shell.
Badness with TLS linkage has been solved.

Fixes #826
2023-07-01 18:14:27 -07:00
Justine Tunney
053681cb97
Fix BSD regressions
The recent change to crt.S that aggressively aligns the system-provided
stack has been rolled back on non-Linux until we can find a better way,
since it can cause a segfault early in execution on several platforms.

This change fixes a regression in tcgetattr() and tcsetattr() on OpenBSD
and NetBSD caused by 4778cd4d27.

This change has been tested across the runitd test fleet which is green.
2023-07-01 00:17:33 -07:00
Justine Tunney
48b2afb192
Address weakness with new pledge("anet") promise
The intent with pledge("anet") has been to prevent outbound connections.
However we were only doing that for TCP sockets, and outbound UDP could
still get through, by using socket() plus sendto(). This change fixed
that by preventing UDP sockets from being created.

Credit goes to chc4 on Hacker News for finding this.
2023-06-18 18:06:47 -07:00
Justine Tunney
226375933a
Implement more toolchain fixes 2023-06-18 05:39:31 -07:00
Justine Tunney
d7c79f43ef
Clean up more code
- Found some bugs in LLVM compiler-rt library
- The useless LIBC_STUBS package is now deleted
- Improve the overflow checking story even further
- Get chibicc tests working in MODE=dbg mode again
- The libc/isystem/ headers now have correctly named guards
2023-06-18 01:00:05 -07:00
Justine Tunney
b881c0ec9e
Remove printf() linking hack 2023-06-17 10:13:50 -07:00
Justine Tunney
4eebd6b9dc
Improve new C23 checked arithmetic feature 2023-06-16 15:32:18 -07:00
Justine Tunney
c3440d040c
Make improvements
- More timspec_*() and timeval_*() APIs have been introduced.
- The copyfd() function is now simplified thanks to POSIX rules.
- More Cosmo-specific APIs have been moved behind the COSMO define.
- The setitimer() polyfill for Windows NT is now much higher quality.
- Fixed build error for MODE=aarch64 due to -mstringop-strategy=loop.
- This change introduces `make MODE=nox87 toolchain` which makes it
  possible to build programs using your cosmocc toolchain that don't
  have legacy fpu instructions. This is useful, for example, if you
  want to have a ~22kb tinier blink virtual machine.
2023-06-15 14:50:53 -07:00
Justine Tunney
4778cd4d27
Fix bugs in termios library and cleanup code
This change fixes an issue with the tcflow() magic numbers that was
causing bash to freeze up on Linux. While auditing termios polyfills,
several other issues were identified with XNU/BSD compatibility.

Out of an abundance of caution this change undefines as much surface
area from libc/calls/struct/termios.h as possible, so that autoconf
scripts are less likely to detect non-POSIX teletypewriter APIs that
haven't been polyfilled by Cosmopolitan.

This is a *breaking change* for your static archives in /opt/cosmos if
you use the cosmocc toolchain. That's because this change disables the
ioctl() undiamonding trick for code outside the monorepo, specifically
because it'll lead to brittle ABI breakages like this. If you're using
the cosmocc toolchain, you'll need to rebuild libraries like ncurses,
readline, etc. Yes diamonds cause bloat. To work around that, consider
using tcgetwinsize() instead of ioctl(TIOCGWINSZ) since it'll help you
avoid pulling every single ioctl-related polyfill into the linkage.

The cosmocc script was specifying -DNDEBUG for some reason. It's fixed.
2023-06-14 19:30:52 -07:00
Justine Tunney
06b749ae03
Remove blinkenlights
New home: https://github.com/jart/blink
2023-06-14 19:30:52 -07:00
Justine Tunney
8ff48201ca
Rewrite .zip.o file linker
This change takes an entirely new approach to the incremental linking of
pkzip executables. The assets created by zipobj.com are now treated like
debug data. After a .com.dbg is compiled, fixupobj.com should be run, so
it can apply fixups to the offsets and move the zip directory to the end
of the file. Since debug data doesn't get objcopy'd, a new tool has been
introduced called zipcopy.com which should be run after objcopy whenever
a .com file is created. This is all automated by the `cosmocc` toolchain
which is rapidly becoming the new recommended approach.

This change also introduces the new C23 checked arithmetic macros.
2023-06-10 09:29:44 -07:00
Justine Tunney
4a59210008
Introduce #include <cosmo.h> to toolchain users
This change improves the way internal APIs are being hidden behind the
`COSMO` define. The cosmo.h header will take care of defining that, so
that a separate define statement isn't needed. This change also does a
lot more to define which APIs are standard, and which belong to Cosmo.
2023-06-09 18:03:05 -07:00
Justine Tunney
9b55dbe417
Get GCC to mostly build with Cosmo 2023-06-09 06:41:34 -07:00
Justine Tunney
4b2023ffab
Disable linker map generation and improve tinyness 2023-06-09 03:29:26 -07:00
Justine Tunney
23e235b7a5
Fix bugs in cosmocc toolchain
This change integrates e58abc1110b335a3341e8ad5821ad8e3880d9bb2 from
https://github.com/ahgamut/musl-cross-make/ which fixes the issues we
were having with our C language extension for symbolic constants. This
change also performs some code cleanup and bug fixes to getaddrinfo().
It's now possible to compile projects like ncurses, readline and python
without needing to patch anything upstream, except maybe a line or two.
Pretty soon it should be possible to build a Linux distro on Cosmo.
2023-06-08 23:44:03 -07:00
Justine Tunney
4d629fd424
Fix stack abuse in llama.cc
This change also incorporates improvements for MODE=asan. It's been
confirmed that o/asan/third_party/ggml/llama.com will work.

Fixes #829
2023-06-08 07:12:26 -07:00
Justine Tunney
32682f0ce7
Remove some problematic APIs
In order to improve our chances of success building other open source
projects we shouldn't define APIs that'll lead any ./configure script
astray. For example:

- brk() and sbrk() can break mac/windows support
- syscall() is a superb way to break portability
- arch_prctl() is the greatest of all horror shows
2023-06-08 06:12:26 -07:00
Justine Tunney
daf4454a06
Validate privileged code relationships
- Work towards improving non-optimized build support
- Introduce MODE=zero which is -O0 without ASAN/UBSAN
- Use system GCC when ~/.cosmo.mk has USE_SYSTEM_TOOLCHAIN=1
- Have package.com check .privileged code doesn't call non-privileged
2023-06-08 04:38:06 -07:00
Justine Tunney
01fd655097
Get garbage collector working on aarch64
Garbage collection will now happen on arm64 when a function returns,
rather than kicking the can down the road to when the process exits.
This change also does some code cleanup and incorporates suggestions
2023-06-07 03:34:45 -07:00
Justine Tunney
b6182db813
Simplify ftrace_hook()
We now have a test to prove that its transitive closure doesn't perform
floating point computations.
2023-06-06 11:10:38 -07:00
Justine Tunney
61b9677c05
Make improvements
- Get mprotect_test working on aarch64
- Get completion working on python.com repl again
- Improve quality of printvideo.com and printimage.com
- Fix bug in openpty() so examples/script.c works again
2023-06-06 09:12:30 -07:00
Justine Tunney
b8a6a989c0
Create ELF aliases for identical symbols
This change greatly reduces the number of modules that need to be
compiled. The only issue right now is that sometimes when viewing
symbol table entries, the aliased symbol is chosen.
2023-06-06 03:33:49 -07:00
Justine Tunney
eb40cb371d
Get --ftrace working on aarch64
This change implements a new approach to function call logging, that's
based on the GCC flag: -fpatchable-function-entry. Read the commentary
in build/config.mk to learn how it works.
2023-06-05 23:35:31 -07:00
Justine Tunney
5b908bc756
Fix some build errors 2023-06-05 15:53:44 -07:00
Justine Tunney
7558549d44
Test m=aarch64 on GitHub Actions 2023-06-05 15:23:00 -07:00
Justine Tunney
35dcaca53c
Fix test fleet errors 2023-06-04 10:57:11 -07:00
Justine Tunney
a91a945b88
Fix build breakage 2023-06-04 09:33:48 -07:00
Justine Tunney
4aa1d09b9e
Improve aarch64 native support some more
This change introduces partial support for automating remote testing of
aarch64 binaries on Raspberry Pi and Apple Silicon.
2023-06-04 08:58:47 -07:00
Justine Tunney
8f522cb702
Make improvements
This change progresses our AARCH64 support:

- The AARCH64 build and tests are now passing
- Add 128-bit floating-point support to printf()
- Fix clone() so it initializes cosmo's x28 TLS register
- Fix TLS memory layout issue with aarch64 _Alignas vars
- Revamp microbenchmarking tools so they work on aarch64
- Make some subtle improvements to aarch64 crash reporting
- Make kisdangerous() memory checks more accurate on aarch64
- Remove sys_open() since it's not available on Linux AARCH64

This change makes general improvements to Cosmo and Redbean:

- Introduce GetHostIsa() function in Redbean
- You can now feature check using pledge(0, 0)
- You can now feature check using unveil("",0)
- Refactor some more x86-specific asm comments
- Refactor and write docs for some libm functions
- Make the mmap() API behave more similar to Linux
- Fix WIFSIGNALED() which wrongly returned true for zero
- Rename some obscure cosmo keywords from noFOO to dontFOO
2023-06-03 08:12:22 -07:00
Paul Kulchenko
5655c9a4e7
Extend Pledge with anet (same as inet, but with no connect) (#827)
* Add `anet` pledge for `inet` without connect

This is useful for configurations where it's desirable to start redbean
under these restrictions, but not to allow `connect` socket calls.

* Update message on protected/unpledged syscalls for clarity

* Update redbean to add reporting for unpledged sigaction

Previously it would abort without indicating what signal it failed to
install when sigaction is not pledged (although it fails all of them).

* Move GetHostIps before processing command line options

This allows using unix.pledge as part of the options without affecting
retrieving host IP addresses (which requires `connect`). It may still
fail under external `pledge` command as expected; in this case IPs
would need to be passed manually.

* Update tests for pledge anet promise
2023-06-03 07:50:29 -07:00
Justine Tunney
1422e96b4e
Introduce native support for MacOS ARM64
There's a new program named ape/ape-m1.c which will be used to build an
embeddable binary that can load ape and elf executables. The support is
mostly working so far, but still chasing down ABI issues.
2023-05-20 04:17:03 -07:00
Justine Tunney
e7eb0b3070
Make more ML improvements
- Fix UX issues with llama.com
- Do housekeeping on libm code
- Add more vectorization to GGML
- Get GGJT quantizer programs working well
- Have the quantizer keep the output layer as f16c
- Prefetching improves performance 15% if you use fewer threads
2023-05-16 08:07:23 -07:00
Justine Tunney
210187cf77
Perform some code cleanup 2023-05-15 16:32:10 -07:00
Justine Tunney
cc1732bc42
Make AARCH64 harder, better, faster, stronger
- Perform some housekeeping on scalar math function code
- Import ARM's Optimized Routines for SIMD string processing
- Upgrade to latest Chromium zlib and enable more SIMD optimizations
2023-05-15 02:15:34 -07:00
Justine Tunney
550b52abf6
Port a lot more code to AARCH64
- Introduce epoll_pwait()
- Rewrite -ftrapv and ffs() libraries in C code
- Use more FreeBSD code in math function library
- Get significantly more tests passing on qemu-aarch64
- Fix many Musl long double functions that were broken on AARCH64
2023-05-14 09:37:26 -07:00
Justine Tunney
ba49e86e20
Get TEST_LIBC_CALLS passing on AARCH64 2023-05-13 02:41:41 -07:00
Justine Tunney
802e7eb4ef
Mop up more test regressions 2023-05-13 01:09:44 -07:00
Justine Tunney
fd34ef732d
Make considerably more progress on AARCH64
- Utilities like pledge.com now build
- kprintf() will no longer balk at 48-bit addresses
- There's a new aarch64-dbg build mode that should work
- gc() and defer() are mostly pacified; avoid using them on aarch64
- THIRD_PART_STB now has Arm Neon intrinsics for fast image handling
2023-05-12 22:42:57 -07:00
Justine Tunney
414667b1c9
Get TEST_LIBC_STR passing on AARCH64
It's now possible to run commands like:

    make -j8 m=aarch64 o/aarch64/test/libc/str

Which will cross-compile and run the test suites in a qemu-aarch64
binary that's vendored in the third_party/qemu/ folder within your
x86_64 build environment.
2023-05-12 18:09:23 -07:00
Justine Tunney
5e2f7f7ced
Get LIBC_TESTLIB building on AARCH64 2023-05-11 19:57:09 -07:00
Justine Tunney
1f6f9e6701
Remove division from matrix multiplication
This change reduces llama.com CPU cycles systemically by 2.5% according
to the Linux Kernel `perf stat -Bddd` utility.
2023-05-10 21:19:54 -07:00
Justine Tunney
290a49952e
Fix some more issues with aarch64 and llama.cpp 2023-05-10 07:34:26 -07:00
Justine Tunney
4e81d3277c
Do some basic housekeeping in LIBC_STR 2023-05-10 06:17:20 -07:00
Justine Tunney
f312f706f4
Bring MODE=tiny binary sizes down to 20kb minimum
aarch64 binaries start at 4kb.
2023-05-10 04:20:48 -07:00
Justine Tunney
59766efd3e
Do some more aarch64 fixups 2023-05-10 04:20:47 -07:00
Justine Tunney
a0237a017c
Get llama.com working on aarch64 2023-05-10 04:20:47 -07:00
Justine Tunney
4c093155a3
Get llama.com building as an aarch64 native binary 2023-05-10 04:20:47 -07:00
Justine Tunney
d04430f4ef
Get LIBC_MEM and LIBC_STDIO building with aarch64 2023-05-10 04:20:47 -07:00
Justine Tunney
7e46645193
Get LIBC_FMT to build for aarch64 2023-05-10 04:20:47 -07:00
Justine Tunney
036b9a0002
Make further progress on non-x86 support 2023-05-10 04:20:47 -07:00
Justine Tunney
135080fd3e
Get libc/tinymath/ compiling on aarch64 2023-05-10 04:20:46 -07:00
Justine Tunney
7c9ef924bf
Add more sorting algorithms 2023-04-27 05:44:44 -07:00
Justine Tunney
614229e3f4
Make some sorting functions faster 2023-04-24 23:11:18 -07:00
Justine Tunney
4a6be2154d
Fix hermeticity issue with new test 2023-04-18 06:37:01 -07:00
Justine Tunney
3e702e8f65
Push new rolling release 2023-04-18 05:11:15 -07:00
Gavin Hayes
d484e1dbd4
Improve system() / popen() builtin shell (#753)
* cocmd: stop splitting redirect expressions
* cocmd: allow passing unpaired } for passing Perl test
* cocmd: propagate exit status from semicolon
2023-04-17 16:18:16 -07:00
Gabriel Ravier
d5b8b644c2
unveil: Added truncate support on Linux 6.2+ (#803)
Right now, cosmopolitan uses Linux Landlock ABI version 2 on Linux,
meaning that the polyfill for unveil() cannot restrict operations such
as truncate() (a limitation of Landlock's ABI from then). This means
that to restrict truncation operations Cosmopolitan instead has to ban
the syscall through a SECCOMP BPF filter, meaning that completely
legitimate truncate() calls are blocked

However, the newest version of the Landlock ABI (version 3) introduced
in Linux 6.2, released in February 2023, implements support for controlling truncation
operations. As such, the previous SECCOMP BPF truncate() filtering is
no longer needed when the new ABI is available

This patch implements unveil truncate support for Linux Landlock ABI
version 3
2023-04-17 16:17:02 -07:00
Gavin Hayes
6758c3f400
commandv: fix dir tests on non-NT (#764) 2023-04-15 06:26:13 -07:00
Gabriel Ravier
9a5d69c842
Fix scanf x specifier with string of 0 (#793)
The C standard states that, in the context of an x conversion
specifier given to scanf:
> Matches an optionally signed hexadecimal integer, whose format is
> the same as expected for the subject sequence of the strtoul
> function with the value 16 for the base argument.
- C standard, 7.23.6.2.11. The fscanf function

Cosmopolitan fails to do this, as 0 should be parsed as a 0 by such an
invocation of strtoul. Instead, cosmopolitan errors out as though such
input is invalid, which is wrong.

This means that a program such as this:

 #include <stdio.h>
 #undef NDEBUG
 #include <assert.h>

int main()
{
    int v = 0;
    assert(sscanf("0", "%x", &v) == 1);
}

will not run correctly on cosmpolitan, instead failing the assertion.

This patch fixes this, along with the associated GitHub issue,
https://github.com/jart/cosmopolitan/issues/778
2023-04-15 06:25:35 -07:00
Gabriel Ravier
12e07798df
Fix printf precision/field width being limited by internal buffer size (#799)
The C standard, when defining field width and precision, never gives
any limit on the values used for them (except, I believe, that they
fit within an int). In other words, if the user gives a field width of
32145 and a precision of 9218, the implementation has to handle these
values correctly. However, when such kinds of high numbers are used
with integer conversions, cosmopolitan is limited by an internal
buffer size of 144, which means precisions and field widths have to
fit within this, which violates the standard.

This means that for example, the following program:

 #include <stdio.h>
 #include <string.h>

int main()
{
    char buf2[512] = {};

    int i = snprintf(buf2, sizeof(buf2), "%.9999u", 10);
    printf("%d %zu\n", i, strlen(buf2));
}

would, instead of printing "9999 511" (the correct output), instead
print "144 144" under cosmopolitan.

This patch fixes this.
2023-04-04 14:16:34 -04:00
Gabriel Ravier
9c6d78c26d
Fix vfprintf and derived functions not handling write errors (#798)
The C standard states:
> The fprintf function returns the number of characters transmitted,
> or a negative value if an output or encoding error occurred or if
> the implementation does not support a specified width length
> modifier.
- C Standard, 7.23.6.1.15. The fprintf function

However, cosmopolitan fails to return a negative value in the case of
an output error, meaning that a program such as:

 #include <stdio.h>

int main()
{
    FILE *fp = fopen("/dev/full", "w");

    setbuf(fp, NULL);
    printf("fprintf: %d\n", fprintf(fp, "test\n"));
    printf("fflush: %d\n", fflush(fp));
}

will, under cosmopolitan, print that no error occured in either of the
calls to fprintf and fflush.

This patch fixes this, along with the associated GitHub issue,
https://github.com/jart/cosmopolitan/issues/784
2023-03-31 09:57:29 -04:00
Gabriel Ravier
36f52ea687
Fix PFLINK mechanism for uppercase float conversion specifiers (#796)
_PFLINK is supposed to automatically pull in required functions for
specific conversion specifiers. However, it fails to do so for the F,
G and E conversion specifiers.

This means that, for example, the following program:

 #include <stdio.h>

int main()
{
    printf("%F %G %E\n", .0, .0, .0);
}

fails to run correctly, printing "? ? ?" instead of
"0.000000 0 0.000000E+00".

This patch fixes this.
2023-03-29 22:18:59 -04:00
Gavin Hayes
669b4c5f19
Add memfd fexecve zipos support (#752) 2023-03-29 18:16:46 -07:00
Gabriel Ravier
ba42248575
Fix larger than WCHAR_MAX differences in wcs{,n}cmp (#795)
The C standard states:

> Unless explicitly stated otherwise, the functions described in this
> subclause order two wide characters the same way as two integers of
> the underlying integer type designated by wchar_t.
>
> [...]
>
> The wcscmp function returns an integer greater than, equal to, or
> less than zero, accordingly as the wide string pointed to by s1 is
> greater than, equal to, or less than the wide string pointed to by
> s2.
>
> [...]
>
> The wcsncmp function returns an integer greater than, equal to, or
> less than zero, accordingly as the possibly null-terminated array
> pointed to by s1 is greater than, equal to, or less than the
> possibly null-terminated array pointed to by s2.
- C Standard, 7.31.4.4. Wide string comparison functions

Cosmopolitan fails to obey this in cases where the difference between
two wide characters is larger than WCHAR_MAX.

This means that, for example, the following program:

 #include <stdio.h>
 #include <wchar.h>
 #include <limits.h>

int main()
{
    wchar_t str1[] = { WCHAR_MIN, L'\0' };
    wchar_t str2[] = { WCHAR_MAX, L'\0' };
    printf("%d\n", wcscmp(str1, str2));
    printf("%d\n", wcsncmp(str1, str2, 2));
}

will print `1` twice, instead of the negative numbers mandated by the
standard (as WCHAR_MIN is less than WCHAR_MAX)

This patch fixes this, along with the associated Github issue,
https://github.com/jart/cosmopolitan/issues/783
2023-03-29 20:28:18 -04:00
Justine Tunney
999481ace0
Fix memcpy(size=0) ubsan warning in vsnprintf()
Fixes #785
2023-03-29 01:28:10 -07:00
Gabriel Ravier
7f925e6be9
Fix issues 774, 782 and 789 (printf precision bugs) (#790)
The C standard states that, within the context of a printf-family
function, when specifying the precision of a conversion specification:

> A negative precision argument is taken as if the precision were
> omitted.
- Quoth the C Standard, 7.23.6.1. The fprintf function

Cosmopolitan instead treated negative precision arguments as
though they had a value of 0, which was non-conforming. This
change fixes that. Another issue we found relates to:

> For o conversion, it increases the precision, if and only if
> necessary, to force the first digit of the result to be a zero (if
> the value and precision are both 0, a single 0 is printed).
- Quoth the C standard, 7.23.6.1.6. The fprintf function

When printing numbers in their alternative form, with a precision and
with a conversion specifier of o (octal), Cosmopolitan wasn't following
the standard in two ways:

1. When printing a value with a precision that results in 0-padding,
   cosmopolitan would still add an extra 0 even though this should be
   done "if and only if necessary"
2. When printing a value of 0 with a precision of 0, nothing is
   printed, even though the standard specifically states that a single
   0 is printed in this case

This change fixes those issues too. Furthermore, regression tests have
been introduced to ensure Cosmopolitan continues to be conformant
going forward.

Fixes #774 
Fixes #782 
Fixes #789
2023-03-29 01:11:48 -07:00
Gabriel Ravier
2f4335e081
Fix inttypes.h FAST16 macros to have a correct definition (#791)
Cosmopolitan now conforms to the C Standard 7.8.1 specification
of the PRI and SCN macros, because this change fixes a bug where
the FAST16 ones were incorrectly using the %hd specifier.
2023-03-29 00:19:40 -07:00
Gabriel Ravier
0adefbf152
Fix the X conversion specifier's alternative form (#788)
The standard states that, when the # flag is used:
> The result is converted to an "alternative form". [...] For x (or X)
conversion, a nonzero result has 0x (or 0X) prefixed to it.
- C standard, 7.23.6.1. The fprintf function

cosmopolitan fails to use the correct alternative form (0X) when the X
conversion specifier is used, instead using 0x, which is not
capitalized.

This patch fixes this, along with the several tests that test for the
wrong behavior.
2023-03-29 00:10:53 -07:00
Gabriel Ravier
792b1c84c0
Fix padding+minus flag on numbers for printf-family functions (#787)
The C standard states, for conversions using the d, i, b, B, o, u, x or X conversion specifiers:
> The precision specifies the minimum number of digits to appear; if
> the value being converted can be represented in fewer digits, it is
> expanded with leading zeros.
- C standard, 7.23.6.1. The fprintf function

However, cosmopolitan currently suppresses the addition of leading
zeros when the minus flag is set. This is not reflected by anything
within the C standard, meaning that behavior is incorrect.

This patch fixes this.
2023-03-25 14:39:25 -04:00
Gabriel Ravier
2d6ea2fbc9
Fix issue #771 by implementing S conversion specifier for printf-related functions (#786)
* Implement S conversion specifier for printf-related functions

POSIX specifies that a conversion specifier of S must be interpreted
the same way as %ls. This patch implements this.

* clang-format

---------

Co-authored-by: Gavin Hayes <gavin@computoid.com>
2023-03-25 14:38:21 -04:00
Alexandre Gomes Gaigalas
775f456d4c
Avoid matching directories when searching PATH (#717)
When searching for an executable, performs an additional check to
determine if the path is a file.
2023-03-06 11:15:32 -05:00
Justine Tunney
e16b969829
Make intrin_test go faster 2023-03-05 22:57:07 -08:00
Justine Tunney
2112fb1736
Fix SHUFPD and SHUFPS and add fuzz tests
See: https://github.com/jart/blink/issues/72
See: https://github.com/jart/blink/issues/68
2023-03-05 16:12:28 -08:00
Gavin Hayes
c5de653b98
GetZipCdir: prevent integer underflow. posix_spawn_test.c: stop attempting to load zipos from /usr/bin/ape. (#758) 2023-03-05 14:29:38 -08:00
Gavin Hayes
f5520209f5
execve_test.com: fix run with landlock make. Cleanup execve/fexecve tests (#755) 2023-03-01 22:32:15 -08:00
Gavin Hayes
5923d483a4
Add execve / fexecve support to ZIpOS (#727) 2023-02-24 11:48:24 -08:00
Justine Tunney
298ba74a45
Make POSIX semaphores always process shared
Python triggered the undefined behavior previously since it appears to
be posting to a semaphore owned by a different process that wasn't set
to process shared mode. The performance loss to process shared futexes
is so low and semaphores are generally used for this purpose, so it'll
be much simpler to simply not impose undefined behavior here.
2023-02-23 08:07:54 -08:00
Gavin Hayes
ff9c15f48a
Add APE fexecve() support (#733) 2023-02-22 18:58:23 -08:00
Justine Tunney
9fce2751c8
Remove problematic test 2023-02-05 16:50:04 -08:00
Justine Tunney
49c1dea03b
Add msync_test 2023-02-05 16:49:52 -08:00
Gavin Hayes
fd0da9c0df
Implement zipos mmap (#725) 2023-02-02 02:28:50 -08:00
Justine Tunney
006c44ff5d
Update tests and CPU detection for Blink 2023-01-18 00:56:09 -08:00
Justine Tunney
10d1c6da18
Support argv[0] munging on Windows 2023-01-03 03:22:40 -08:00
Justine Tunney
dd04aeba1c
Increase stack size to 128k and guard size to 16k
This improves our compatibility with Apple M1.
2022-12-18 22:58:29 -08:00
Justine Tunney
3da887c58f
Introduce CPUID detection for Blink 2022-12-17 00:38:07 -08:00
Justine Tunney
ed161b240e
Clean up some code 2022-12-11 14:30:59 -08:00
Justine Tunney
a277170c9a
Make kprintf_test resistant to limbo 2022-11-13 14:26:19 -08:00
Justine Tunney
d55b019284
Fix the build 2022-11-11 11:13:21 -08:00
Justine Tunney
f2af97711b
Make improvements
- Improve compatibility with Blink virtual machine
- Add non-POSIX APIs for joining threads and signal masks
- Never ever use anything except 32-bit integers for atomics
- Add some `#undef` statements to workaround `ctags` problems
2022-11-10 21:52:47 -08:00
Justine Tunney
cee6871710
Make detached threads work better
This change adds a double linked list of threads, so that pthread_exit()
will know when it should call exit() from an orphaned child. This change
also improves ftrace and strace logging.
2022-11-09 03:58:57 -08:00
Justine Tunney
b74d8c1acd
Remove unnecessary STRACE() statement 2022-11-08 15:33:16 -08:00
Justine Tunney
bf7843833f
Rename hidden keyword to _Hide 2022-11-08 12:55:28 -08:00
Justine Tunney
b407327972
Make fixes and improvements
- clock_nanosleep() is now much faster on OpenBSD and NetBSD
- Thread joining is now much faster on NetBSD
- FreeBSD timestamps are now more accurate
- Thread spawning now goes faster on XNU
- Clean up the clone() code
2022-11-08 10:11:46 -08:00
Justine Tunney
aee50b1327
Fix test breakage 2022-11-07 06:31:30 -08:00
Justine Tunney
4de2cf34e6
Allow HTTP range past EOF
Fixes #683
2022-11-07 05:52:24 -08:00
Justine Tunney
c995838e5c
Make improvements
- Clean up sigaction() code
- Add a port scanner example
- Introduce a ParseCidr() API
- Clean up our futex abstraction code
- Fix a harmless integer overflow in ParseIp()
- Use kernel semaphores on NetBSD to make threads much faster
2022-11-07 02:26:06 -08:00
Justine Tunney
3f0bcdc3ef
Improve cancellations, randomness, and time
- Exhaustively document cancellation points
- Rename SIGCANCEL to SIGTHR just like BSDs
- Further improve POSIX thread cancellations
- Ensure asynchronous cancellations work correctly
- Elevate the quality of getrandom() and getentropy()
- Make futexes cancel correctly on OpenBSD 6.x and 7.x
- Add reboot.com and shutdown.com to examples directory
- Remove underscore prefix from awesome timespec_*() APIs
- Create assertions that help verify our cancellation points
- Remove bad timespec APIs (cmp generalizes eq/ne/gt/gte/lt/lte)
2022-11-05 23:45:32 -07:00
Justine Tunney
022536cab6
Make futexes cancellable by pthreads 2022-11-04 18:36:34 -07:00
Justine Tunney
2278327eba
Implement support for POSIX thread cancellations
This change makes some miracle modifications to the System Five system
call support, which lets us have safe, correct, and atomic handling of
thread cancellations. It all turned out to be cheaper than anticipated
because it wasn't necessary to modify the system call veneers. We were
able to encode the cancellability of each system call into the magnums
found in libc/sysv/syscalls.sh. Since cancellations are so waq, we are
also supporting a lovely Musl Libc mask feature for raising ECANCELED.
2022-11-04 01:04:43 -07:00
Justine Tunney
37d40e087f
Ignore SIGSYS on BSD by default 2022-11-03 09:32:12 -07:00
Justine Tunney
179e048bba
Make some small fixes to recent changes 2022-11-03 05:45:33 -07:00
Justine Tunney
14d036b68d
Add WSL to test fleet
All tests pass now under WSL2. They should pass under WSL1 too, but only
WSL2 is integrated into the test fleet right now. This change also fills
in some gaps in the error numbers.

Fixes #665
2022-11-02 06:49:42 -07:00
Justine Tunney
fae0c0286f
Workaround WSL not supporting x87 in ucontext_t 2022-11-02 03:20:39 -07:00
Justine Tunney
f44d88707e
Workaround sendfile() bug in WSL 2022-11-02 02:17:03 -07:00
Justine Tunney
e522aa3a07
Make more threading improvements
- ASAN memory morgue is now lockless
- Make C11 atomics header more portable
- Rewrote pthread keys support to be lockless
- Simplify Python's unicode table unpacking code
- Make crash report write(2) closer to being atomic
- Make it possible to strace/ftrace a single thread
- ASAN now checks nul-terminated strings fast and properly
- Windows fork() now restores TLS memory of calling thread
2022-11-01 23:28:26 -07:00
Justine Tunney
69bee64a59
Make some last minute production changes 2022-10-19 10:00:29 -07:00
Justine Tunney
f7ff77d865
Make fixes and improvements
- Invent iso8601us() for faster timestamps
- Improve --strace descriptions of sigset_t
- Rebuild the Landlock Make bootstrap binary
- Introduce MODE=sysv for non-Windows builds
- Permit OFD fcntl() locks under pledge(flock)
- redbean can now protect your kernel from ddos
- Have vfork() fallback to sys_fork() not fork()
- Change kmalloc() to not die when out of memory
- Improve documentation for some termios functions
- Rewrite putenv() and friends to conform to POSIX
- Fix linenoise + strace verbosity issue on Windows
- Fix regressions in our ability to show backtraces
- Change redbean SetHeader() to no-op if value is nil
- Improve fcntl() so SQLite locks work in non-WAL mode
- Remove some unnecessary work during fork() on Windows
- Create redbean-based SSL reverse proxy for IPv4 TurfWar
- Fix ape/apeinstall.sh warning when using non-bash shells
- Add ProgramTrustedIp(), and IsTrustedIp() APIs to redbean
- Support $PWD, $UID, $GID, and $EUID in command interpreter
- Introduce experimental JTqFpD APE prefix for non-Windows builds
- Invent blackhole daemon for firewalling IP addresses via UNIX named socket
- Add ProgramTokenBucket(), AcquireToken(), and CountTokens() APIs to redbean
2022-10-19 07:19:19 -07:00
Justine Tunney
648bf6555c
Reimport zip into third party 2022-10-16 13:39:41 -07:00
Justine Tunney
60cb435cb4
Implement pthread_atfork()
If threads are being used, then fork() will now acquire and release and
runtime locks so that fork() may be safely used from threads. This also
makes vfork() thread safe, because pthread mutexes will do nothing when
the process is a child of vfork(). More torture tests have been written
to confirm this all works like a charm. Additionally:

- Invent hexpcpy() api
- Rename nsync_malloc_() to kmalloc()
- Complete posix named semaphore implementation
- Make pthread_create() asynchronous signal safe
- Add rm, rmdir, and touch to command interpreter builtins
- Invent sigisprecious() and modify sigset functions to use it
- Add unit tests for posix_spawn() attributes and fix its bugs

One unresolved problem is the reclaiming of *NSYNC waiter memory in the
forked child processes, within apps which have threads waiting on locks
2022-10-16 12:25:13 -07:00
Justine Tunney
3d2d04b25e
Make cli commands yoinkable into cosmo shell
This lets our system() and popen() commands function sort of like
BusyBox and ToyBox. By default the Cosmopolitan Shell is lightweight.
But if you use STATIC_YOINK then you can pull the individual commands
you want into the linkage, and they'll be included in a single binary.
For example the demo binary embeds `tr` and `sed` and ends up ~140kb.
2022-10-14 16:37:29 -07:00
Justine Tunney
5af19b7eed
Make some foss compatibility improvements 2022-10-14 13:59:34 -07:00
Justine Tunney
8111462789
Add posix semaphores support
There's still some bugs to work out on Windows and OpenBSD.
2022-10-14 09:21:02 -07:00
Justine Tunney
f52f65b2e3
Make system() and popen() thread safe 2022-10-13 15:54:05 -07:00
Justine Tunney
997ce29ddc
Elevate Windows production worthiness
- SQLite file locking now works on Windows
- SQLite will now use fdatasync() on non-Apple platforms
- Fix Ctrl-C handler on Windows to not crash with TLS
- Signals now work in multithreaded apps on Windows
- fcntl() will now accurately report EINVAL errors
- fcntl() now has excellent --strace logging
- Token bucket replenish now go 100x faster
- *NSYNC cancellations now work on Windows
- Support closefrom() on NetBSD
2022-10-13 13:44:41 -07:00
Justine Tunney
0f89140882
Further improve cocmd interpreter 2022-10-12 10:44:54 -07:00
Justine Tunney
0cee831da3
Revamp posix_spawn() implementation 2022-10-12 05:26:58 -07:00
Justine Tunney
b41f91c658
Greatly expand system() shell code features
The cosmopolitan command interpreter now has 13 builtin commands,
variable support, support for ; / && / || syntax, asynchronous support,
and plenty of unit tests with bug fixes.

This change fixes a bug in posix_spawn() with null envp arg. strace
logging now uses atomic writes for scatter functions. Breaking change
renaming GetCpuCount() to _getcpucount(). TurfWar is now updated to use
the new token bucket algorithm. WIN32 affinity masks now inherit across
fork() and execve().
2022-10-11 21:30:31 -07:00
Justine Tunney
e7329b7cba
Implement atomic swar token bucket library 2022-10-11 21:22:04 -07:00
tkchia
7a06760e6f
Rename LINUX to _HOSTLINUX etc. to reduce clashes (#655)
Co-authored-by: tkchia <tkchia-cosmo@gmx.com>
2022-10-10 20:31:25 -07:00
Malcolm Seyd
84b9b8ed87
Fix ParseHttpMessage failing to store >2 repeatable headers (#657) 2022-10-10 20:17:17 -07:00
Justine Tunney
e557058ac8
Improve cosmo's conformance to libc-test
This change addresses various open source compatibility issues, so that
we pass 313/411 of the tests in https://github.com/jart/libc-test where
earlier today we were passing about 30/411 of them, due to header toil.
Please note that Glibc only passes 341/411 so 313 today is pretty good!

- Make the conformance of libc/isystem/ headers nearly perfect
- Import more of the remaining math library routines from Musl
- Fix inconsistencies with type signatures of calls like umask
- Write tests for getpriority/setpriority which work great now
- conform to `struct sockaddr *` on remaining socket functions
- Import a bunch of uninteresting stdlib functions e.g. rand48
- Introduce readdir_r, scandir, pthread_kill, sigsetjmp, etc..

Follow the instructions in our `tool/scripts/cosmocc` toolchain to run
these tests yourself. You use `make CC=cosmocc` on the test repository
2022-10-10 17:52:41 -07:00
Justine Tunney
467a332e38
Introduce sigtimedwait() and sigwaitinfo()
This change also invents sigcountset() and strsignal_r() and improves
the quality of siginfo_t handling.
2022-10-10 07:39:44 -07:00
Justine Tunney
7ae556463a
Rename rand64() to _rand64() 2022-10-10 04:12:06 -07:00
Justine Tunney
d5910e2673
Fix bugs and make code tinier
- Fixed bug where stdio eof wasn't being sticky
- Fixed bug where fseeko() wasn't clearing eof state
- Removed assert() usage from libc favoring _unassert() / _npassert()
2022-10-09 23:21:34 -07:00
Justine Tunney
4a6fd3d910
Make more improvements to threading support
- fix rare thread exit race condition on openbsd
- pthread_getattr_np() now supplies detached status
- child threads may now pthread_join() the main thread
- introduce sigandset(), sigorset(), and sigisemptyset()
- introduce pthread_cleanup_push() and pthread_cleanup_pop()
2022-10-09 00:08:47 -07:00
Justine Tunney
8aca94f951
Add shared memory hit counter example to redbean 2022-10-08 08:39:13 -07:00
Justine Tunney
9849b4c7ba
Add *NSYNC unit test suite
This change also fixes the clock_nanosleep() api and polyfills futexes
on Windows, Mac, and NetBSD using exponential backoff.
2022-10-07 21:34:15 -07:00
Justine Tunney
05197afca2
Do some work on TurfWar 2022-10-07 03:11:07 -07:00
Justine Tunney
59ac141e49
Improve the affinity system calls 2022-10-06 15:08:29 -07:00
Justine Tunney
60b68d7152
Fix naming for redbean shared memory atomics 2022-10-06 09:41:22 -07:00
Justine Tunney
7822917fc2
Add shared memory apis to redbean
You can now do things like implement mutexes using futexes in your
redbean lua code. This provides the fastest possible inter-process
communication for your production systems when SQLite alone as ipc
or things like pipes aren't sufficient.
2022-10-06 04:55:26 -07:00
Justine Tunney
81ee11a16e
Fix futimes() regression
Fixes #647
2022-10-05 19:25:07 -07:00
Hugues Morisset
f155205eb0
Add imaxdiv, wcscoll, getdtablesize (#639) 2022-10-05 07:14:58 -07:00
Justine Tunney
fe3216e961
Perform some code cleanup 2022-10-04 23:32:16 -07:00