Commit graph

299 commits

Author SHA1 Message Date
Justine Tunney
77a7873057
Improve AARCH64 execution
This change fixes bugs in the APE loader. The execve() unit tests are
now enabled for MODE=aarch64. See the README for how you need to have
binfmt_misc configured with Qemu to run them. Apple Silicon bugs have
been fixed too, e.g. tkill() now works.
2023-09-11 14:46:46 -07:00
Justine Tunney
032b1f3449
Implement thread cancellation for aarch64 2023-09-07 08:48:38 -07:00
Justine Tunney
f531acc8f9
Make improvements
- Invent openatemp() API
- Invent O_UNLINK open flag
- Introduce getenv_secure() API
- Remove `git pull` from cosmocc
- Fix utimes() when path is NULL
- Fix mktemp() to never return NULL
- Fix utimensat() UTIME_OMIT on XNU
- Improve utimensat() code for RHEL5
- Turn `argv[0]` C:/ to /C/ on Windows
- Introduce tmpnam() and tmpnam_r() APIs
- Fix more const issues with internal APIs
- Permit utimes() on WIN32 in O_RDONLY mode
- Fix fdopendir() to check fd is a directory
- Fix recent crash regression in landlock make
- Fix futimens(AT_FDCWD, NULL) to return EBADF
- Use workaround so `make -j` doesn't fork bomb
- Rename dontdiscard to __wur (just like glibc)
- Fix st_size for WIN32 symlinks containing UTF-8
- Introduce stdio ext APIs needed by GNU coreutils
- Fix lstat() on WIN32 for symlinks to directories
- Move some constants from normalize.inc to limits.h
- Fix segv with memchr() and memcmp() overlapping page
- Implement POSIX fflush() behavior for reader streams
- Implement AT_SYMLINK_NOFOLLOW for utimensat() on WIN32
- Don't change read-only status of existing files on WIN32
- Correctly handle `0x[^[:xdigit:]]` case in strtol() functions
2023-09-06 12:34:59 -07:00
michalbiesek
8596e83cce
Linux add definition for madvise flag (#883) 2023-09-06 12:34:54 -07:00
Justine Tunney
6ef2a471e4
Get GNU MPFR and MPC tests to pass
This change fixes more issues with our scanf() function.
2023-08-21 15:05:10 -07:00
Justine Tunney
755ae64e73
Remove long path tests for Windows 2023-08-21 04:43:53 -07:00
Justine Tunney
fffcd98b0e
Fix msync() flags on FreeBSD 2023-08-21 04:15:05 -07:00
Justine Tunney
ebf784d4f5
Make improvements
- Introduce ualarm() function
- Make rename() report EISEMPTY on Windows
- Always raise EINVAL upon open(O_RDONLY|O_TRUNC)
- Add macro so ./configure will detect SOCK_CLOEXEC
- Fix O_TRUNC without O_CREAT not working on Windows
- Let fcntl(F_SETFL) change O_APPEND status on Windows
- Make sure pwrite() / pread() report ESPIPE on sockets
- Raise ESPIPE on Windows when pwrite() is used on pipe
- Properly compute O_APPEND CreateFile() flags on Windows
- Don't require O_DIRECTORY to open directories on Windows
- Fix more instances of Windows reporting EISDIR and ENOTDIR
- Normalize EFTYPE and EMLINK to ELOOP on NetBSD and FreeBSD
- Make unlink() / rmdir() work on read-only files on Windows
- Validate UTF-8 on Windows paths to fix bug with overlong NUL
- Always print signal name to stderr when crashing due to SIG_DFL
- Fix Windows bug where denormalized paths >260 chars didn't work
- Block signals on BSDs when thread exits before trashing its own stack
2023-08-21 02:34:17 -07:00
michalbiesek
d6f72aa4a6
Extend pr.h header with recent options (#881) 2023-08-20 02:49:58 -07:00
michalbiesek
4f517503e4
Add PTRACE_SECCOMP_GET_METADATA (#882)
see
  linux commit: 26500475ac1b499d8636ff281311d633909f5d20
  ptrace, seccomp: add support for retrieving seccomp metadata
2023-08-20 02:49:22 -07:00
Justine Tunney
965516e313
Make improvements for Actually Portable Emacs
- Get SIGWINCH working again on the New Technology
- Correctly handle O_NOFOLLOW in open() on Windows
- Implement synthetic umask() functionality on Windows
- Do a better job managing file execute access on Windows
- Fill in `st_uid` and `st_gid` with username hash on Windows
- Munge UNICODE control pictures into control codes on Windows
- Do a better job ensuring Windows console settings are restored
- Introduce KPRINTF_LOG environment variable to log kprintf to a file
2023-08-19 06:44:58 -07:00
Justine Tunney
7100b1cf91
Get Fat Emacs working in Windows Console 2023-08-18 05:00:30 -07:00
Justine Tunney
dc6c67256f
Remove old stack code and improve dirstream 2023-08-16 07:54:40 -07:00
Justine Tunney
110559ce6a
Make ZipOS and Qemu work better
This change improves the dirstream library in a lot of respects,
especially for /zip/... files. Also turn off MAP_STACK on Aarch64
because Qemu seems to implement it differently than Linux and it's
probably responsible for a lot of mysterious crashes.
2023-08-15 18:32:50 -07:00
Justine Tunney
c776a32f75
Replace COSMO define with _COSMO_SOURCE
This change might cause ABI breakages for /opt/cosmos. It's needed to
help us better conform to header declaration practices.
2023-08-13 20:55:04 -07:00
Justine Tunney
b40d41085d
Fix the build 2023-08-13 07:17:40 -07:00
Justine Tunney
6942d7b820
Fix bug with temporary files on Windows 2023-08-13 07:10:33 -07:00
Justine Tunney
ab9a284640
Further improve fatcosmocc 2023-08-13 01:51:39 -07:00
Justine Tunney
18bb5888e1
Make more fixes and improvements
- Remove PAGESIZE constant
- Fix realloc() documentation
- Fix ttyname_r() error reporting
- Make forking more reliable on Windows
- Make execvp() a few microseconds faster
- Make system() a few microseconds faster
- Tighten up the socket-related magic numbers
- Loosen restrictions on mmap() offset alignment
- Improve GetProgramExecutableName() with getenv("_")
- Use mkstemp() as basis for mktemp(), tmpfile(), tmpfd()
- Fix flakes in pthread_cancel_test, unix_test, fork_test
- Fix recently introduced futex stack overflow regression
- Let sockets be passed as stdio to subprocesses on Windows
- Improve security of bind() on Windows w/ SO_EXCLUSIVEADDRUSE
2023-07-29 18:44:15 -07:00
Justine Tunney
83341a4269
Remove hints from Windows imports 2023-07-27 14:09:07 -07:00
Justine Tunney
7e0a09feec
Mint APE Loader v1.5
This change ports APE Loader to Linux AARCH64, so that Raspberry Pi
users can run programs like redbean, without the executable needing
to modify itself. Progress has also slipped into this change on the
issue of making progress better conforming to user expectations and
industry standards regarding which symbols we're allowed to declare
2023-07-26 13:54:49 -07:00
Justine Tunney
6843150e0c
Mint APE Loader v1.4
This change also incorporates more bug fixes and improvements to a wide
variety of small things. For example this fixes #860 so Windows console
doesn't get corrupted after exit. An system stack memory map issue with
aarch64 has been fixed. We no longer use O_NONBLOCK on AF_UNIX sockets.
Crash reports on Arm64 will now demangle C++ symbols, even when c++filt
isn't available. Most importantly the Apple M1 version of APE Loader is
brought up to date by this change. A prebuilt unsigned binary for it is
being included in build/bootstrap/. One more thing: retrieving the term
dimensions under --strace was causing the stack to become corrupted and
now that's been solved too. PSS: We're now including an ELF PT_NOTE for
APE in the binaries we build, that has the APE Loader version.
2023-07-25 05:48:08 -07:00
Justine Tunney
0ba3199915
Fix some more socket bugs
- The functions that return a sockaddr now do so the same way the Linux
  Kernel does across platforms, e.g. getpeername(), accept4()

- Socket system calls on Windows will now only check for interrupts when
  a blocking operation needs to be performed.

- Write tests for recvfrom() system call
2023-07-23 16:31:10 -07:00
Justine Tunney
f83eb440f7
Incorporate more small improvements 2023-07-23 10:57:18 -07:00
Justine Tunney
1d4eb08fa1
Support non-blocking i/o across platforms
This change introduces new tests for `O_NONBLOCK` and `SOCK_NONBLOCK` to
confirm that non-blocking i/o is now working on all supported platforms,
including Windows. For example, you can now say on Windows, MacOS, etc.:

    socket(AF_INET, SOCK_STREAM | SOCK_NONBLOCK, IPPROTO_TCP);

To create a non-blocking IPv4 TCP socket. Or you can enable non-blocking
i/o on an existing socket / pipe / etc. file descriptor by calling fcntl

    fcntl(fd, F_SETFL, fcntl(fd, F_GETFL) | O_NONBLOCK);

This functionality is polyfilled on older Linux kernels too, e.g. RHEL5.
Now that fcntl() support is much better the FIOCLEX / FIONCLEX polyfills
for ioctl() have been removed since they're ugly non-POSIX diameond APIs

This change fixes a weakness in kprintf() that was causing Windows trace
tools to frequently crash.
2023-07-23 02:56:47 -07:00
Justine Tunney
1ee2e89326
Make improvements
- This commit mints a new release of APE Loader v1.2 which supports
  loading ELF programs with a non-contiguous virtual address layout
  even though we've never been able to take advantage of it, due to
  how `objcopy -SO binary` fills any holes left by PT_LOAD. This'll
  change soon, since we'll have a new way of creating APE binaries.

- The undiamonding trick with our ioctl() implementation is removed
  since POSIX has been killing ioctl() for years and they've done a
  much better job. One problem it resolves, is that ioctl(FIONREAD)
  wasn't working earlier and that caused issues when building Emacs
2023-07-11 04:41:50 -07:00
Justine Tunney
f7ae50462a
Make improvements
- Fix unused local variable errors
- Remove yoinks from sigaction() header
- Add nox87 and aarch64 to github actions
- Fix cosmocc -fportcosmo in linking mode
- It's now possible to build `make m=llvm o/llvm/libc`
2023-07-10 04:35:14 -07:00
Paul Kulchenko
dc8fbf38ad
Update Windows IP_MULTICAST constants to use Ws2tcpip.h values (#839) (#842) 2023-07-01 19:44:27 -07:00
Justine Tunney
40eb3b9d5d
Fully support OpenBSD 7.3
This change (1) upgrades to OpenBSD's newer kernel ABIs, and (2)
modifies APE to have a read-only data segment. Doing this required
creating APE Loader v1.1, which is backwards and forwards compatible
with the previous version.

If you've run the following commands in the past to install your APE
Loader systemwide, then you need to run them again. Ad-hoc installations
shouldn't be impacted. It's also recommended that APE binaries be remade
after upgrading, since they embed old versions of the APE Loader.

    ape/apeuninstall.sh
    ape/apeinstall.sh

This change does more than just fix OpenBSD. The new loader is smarter
and more reliable. We're now able create much tinier ELF and Mach-O data
structures than we could before. Both APE Loader and execvpe() will now
normalize ambiguous argv[0] resolution the same way as the UNIX shell.
Badness with TLS linkage has been solved.

Fixes #826
2023-07-01 18:14:27 -07:00
Justine Tunney
053681cb97
Fix BSD regressions
The recent change to crt.S that aggressively aligns the system-provided
stack has been rolled back on non-Linux until we can find a better way,
since it can cause a segfault early in execution on several platforms.

This change fixes a regression in tcgetattr() and tcsetattr() on OpenBSD
and NetBSD caused by 4778cd4d27.

This change has been tested across the runitd test fleet which is green.
2023-07-01 00:17:33 -07:00
Justine Tunney
0409096658
Get us closer to building busybox
This change undefines __linux__ and adds APIs like clock_settime(). The
gosh darned getopt_long() API has been reintroduced, thanks to OpenBSD.
2023-06-18 04:13:45 -07:00
Justine Tunney
afc58a8b41
Upgrade cosmopolitan gcc and binutils to 2.35.2
This change upgrades to the latest portcosmo gcc patch
6728fe1a25185560603ca312a8d4352af2a4e515 which lets us avoid needing to
define __tmpcosmo_FOO constants. We're now using an appropriate binutils
version for GCC 11. The older binutils sometimes wasn't able to print
backtraces, due to not being able to find a .debug_ranges section.

This is breaking change for /opt/cosmos libraries :'( due to this weird
"error: need linked-to section for --gc-sections" that pops up.

Please run `make clean` in the monorepo before rebuilding.
2023-06-18 00:54:14 -07:00
Justine Tunney
562a1384cd
Make blink support conditionally linkable into APE 2023-06-17 07:55:35 -07:00
Justine Tunney
c3440d040c
Make improvements
- More timspec_*() and timeval_*() APIs have been introduced.
- The copyfd() function is now simplified thanks to POSIX rules.
- More Cosmo-specific APIs have been moved behind the COSMO define.
- The setitimer() polyfill for Windows NT is now much higher quality.
- Fixed build error for MODE=aarch64 due to -mstringop-strategy=loop.
- This change introduces `make MODE=nox87 toolchain` which makes it
  possible to build programs using your cosmocc toolchain that don't
  have legacy fpu instructions. This is useful, for example, if you
  want to have a ~22kb tinier blink virtual machine.
2023-06-15 14:50:53 -07:00
Justine Tunney
4778cd4d27
Fix bugs in termios library and cleanup code
This change fixes an issue with the tcflow() magic numbers that was
causing bash to freeze up on Linux. While auditing termios polyfills,
several other issues were identified with XNU/BSD compatibility.

Out of an abundance of caution this change undefines as much surface
area from libc/calls/struct/termios.h as possible, so that autoconf
scripts are less likely to detect non-POSIX teletypewriter APIs that
haven't been polyfilled by Cosmopolitan.

This is a *breaking change* for your static archives in /opt/cosmos if
you use the cosmocc toolchain. That's because this change disables the
ioctl() undiamonding trick for code outside the monorepo, specifically
because it'll lead to brittle ABI breakages like this. If you're using
the cosmocc toolchain, you'll need to rebuild libraries like ncurses,
readline, etc. Yes diamonds cause bloat. To work around that, consider
using tcgetwinsize() instead of ioctl(TIOCGWINSZ) since it'll help you
avoid pulling every single ioctl-related polyfill into the linkage.

The cosmocc script was specifying -DNDEBUG for some reason. It's fixed.
2023-06-14 19:30:52 -07:00
Justine Tunney
4a59210008
Introduce #include <cosmo.h> to toolchain users
This change improves the way internal APIs are being hidden behind the
`COSMO` define. The cosmo.h header will take care of defining that, so
that a separate define statement isn't needed. This change also does a
lot more to define which APIs are standard, and which belong to Cosmo.
2023-06-09 18:03:05 -07:00
Justine Tunney
2eb7148474
Create a fake syscall() function due to Python 2023-06-09 05:18:38 -07:00
Justine Tunney
23e235b7a5
Fix bugs in cosmocc toolchain
This change integrates e58abc1110b335a3341e8ad5821ad8e3880d9bb2 from
https://github.com/ahgamut/musl-cross-make/ which fixes the issues we
were having with our C language extension for symbolic constants. This
change also performs some code cleanup and bug fixes to getaddrinfo().
It's now possible to compile projects like ncurses, readline and python
without needing to patch anything upstream, except maybe a line or two.
Pretty soon it should be possible to build a Linux distro on Cosmo.
2023-06-08 23:44:03 -07:00
Justine Tunney
22f81a8d50
Improve cosmocc / cosmoc++ toolchain scripts
- Get out of the red zone
- Generate --ftrace nops unless -Os is passed
- Intercept -o path to generate .com / .com.dbg appropriately
2023-06-08 14:29:22 -07:00
Justine Tunney
daf4454a06
Validate privileged code relationships
- Work towards improving non-optimized build support
- Introduce MODE=zero which is -O0 without ASAN/UBSAN
- Use system GCC when ~/.cosmo.mk has USE_SYSTEM_TOOLCHAIN=1
- Have package.com check .privileged code doesn't call non-privileged
2023-06-08 04:38:06 -07:00
Justine Tunney
eb40cb371d
Get --ftrace working on aarch64
This change implements a new approach to function call logging, that's
based on the GCC flag: -fpatchable-function-entry. Read the commentary
in build/config.mk to learn how it works.
2023-06-05 23:35:31 -07:00
Justine Tunney
7558549d44
Test m=aarch64 on GitHub Actions 2023-06-05 15:23:00 -07:00
Justine Tunney
4aa1d09b9e
Improve aarch64 native support some more
This change introduces partial support for automating remote testing of
aarch64 binaries on Raspberry Pi and Apple Silicon.
2023-06-04 08:58:47 -07:00
Justine Tunney
bcf9af94bf
Get threads working well on MacOS Arm64
- Now using 10x better GCD semaphores
- We now generate Linux-like thread ids
- We now use fast system clock / sleep libraries
- The APE M1 loader now generates Linux-like stacks
2023-06-04 01:57:10 -07:00
Justine Tunney
8f522cb702
Make improvements
This change progresses our AARCH64 support:

- The AARCH64 build and tests are now passing
- Add 128-bit floating-point support to printf()
- Fix clone() so it initializes cosmo's x28 TLS register
- Fix TLS memory layout issue with aarch64 _Alignas vars
- Revamp microbenchmarking tools so they work on aarch64
- Make some subtle improvements to aarch64 crash reporting
- Make kisdangerous() memory checks more accurate on aarch64
- Remove sys_open() since it's not available on Linux AARCH64

This change makes general improvements to Cosmo and Redbean:

- Introduce GetHostIsa() function in Redbean
- You can now feature check using pledge(0, 0)
- You can now feature check using unveil("",0)
- Refactor some more x86-specific asm comments
- Refactor and write docs for some libm functions
- Make the mmap() API behave more similar to Linux
- Fix WIFSIGNALED() which wrongly returned true for zero
- Rename some obscure cosmo keywords from noFOO to dontFOO
2023-06-03 08:12:22 -07:00
Justine Tunney
1422e96b4e
Introduce native support for MacOS ARM64
There's a new program named ape/ape-m1.c which will be used to build an
embeddable binary that can load ape and elf executables. The support is
mostly working so far, but still chasing down ABI issues.
2023-05-20 04:17:03 -07:00
Justine Tunney
cc1732bc42
Make AARCH64 harder, better, faster, stronger
- Perform some housekeeping on scalar math function code
- Import ARM's Optimized Routines for SIMD string processing
- Upgrade to latest Chromium zlib and enable more SIMD optimizations
2023-05-15 02:15:34 -07:00
Justine Tunney
550b52abf6
Port a lot more code to AARCH64
- Introduce epoll_pwait()
- Rewrite -ftrapv and ffs() libraries in C code
- Use more FreeBSD code in math function library
- Get significantly more tests passing on qemu-aarch64
- Fix many Musl long double functions that were broken on AARCH64
2023-05-14 09:37:26 -07:00
Justine Tunney
fd34ef732d
Make considerably more progress on AARCH64
- Utilities like pledge.com now build
- kprintf() will no longer balk at 48-bit addresses
- There's a new aarch64-dbg build mode that should work
- gc() and defer() are mostly pacified; avoid using them on aarch64
- THIRD_PART_STB now has Arm Neon intrinsics for fast image handling
2023-05-12 22:42:57 -07:00
Justine Tunney
414667b1c9
Get TEST_LIBC_STR passing on AARCH64
It's now possible to run commands like:

    make -j8 m=aarch64 o/aarch64/test/libc/str

Which will cross-compile and run the test suites in a qemu-aarch64
binary that's vendored in the third_party/qemu/ folder within your
x86_64 build environment.
2023-05-12 18:09:23 -07:00