Commit graph

285 commits

Author SHA1 Message Date
Justine Tunney
7b284f6bda
Fix bugs and regressions in the pledge command
This change gets the pledge (formerly pledge.com) command back in tip
top shape for a 3.0.1 cosmos release. It now runs on all platforms, even
though it's mostly a no-op on ones that lack the kernel security stuff.
The binary footprint is now smaller, since it no longer needs to link
malloc. It's also now able to be built as a fat binary.
2023-11-01 06:08:58 -07:00
Justine Tunney
a657f3e878
Delete old wcwidth() implementation
This shaves away 144kb of bss memory from every binary linking printf at
the expense of slightly increased binary footprint. Kudos for Byron Lai.
2023-10-14 03:15:05 -07:00
Justine Tunney
2db2f40a98
Rewrite special file handling on Windows
This change gets GNU grep working. What caused it to not work, is it
wouldn't write to an output file descriptor when its dev/ino equaled
/dev/null's. So now we invent special dev/ino values for these files
2023-10-14 02:53:34 -07:00
Justine Tunney
285c565051
Clean up some code 2023-10-11 11:45:31 -07:00
Justine Tunney
791f79fcb3
Make improvements
- We now serialize the file descriptor table when spawning / executing
  processes on Windows. This means you can now inherit more stuff than
  just standard i/o. It's needed by bash, which duplicates the console
  to file descriptor #255. We also now do a better job serializing the
  environment variables, so you're less likely to encounter E2BIG when
  using your bash shell. We also no longer coerce environ to uppercase

- execve() on Windows now remotely controls its parent process to make
  them spawn a replacement for itself. Then it'll be able to terminate
  immediately once the spawn succeeds, without having to linger around
  for the lifetime as a shell process for proxying the exit code. When
  process worker thread running in the parent sees the child die, it's
  given a handle to the new child, to replace it in the process table.

- execve() and posix_spawn() on Windows will now provide CreateProcess
  an explicit handle list. This allows us to remove handle locks which
  enables better fork/spawn concurrency, with seriously correct thread
  safety. Other codebases like Go use the same technique. On the other
  hand fork() still favors the conventional WIN32 inheritence approach
  which can be a little bit messy, but is *controlled* by guaranteeing
  perfectly clean slates at both the spawning and execution boundaries

- sigset_t is now 64 bits. Having it be 128 bits was a mistake because
  there's no reason to use that and it's only supported by FreeBSD. By
  using the system word size, signal mask manipulation on Windows goes
  very fast. Furthermore @asyncsignalsafe funcs have been rewritten on
  Windows to take advantage of signal masking, now that it's much more
  pleasant to use.

- All the overlapped i/o code on Windows has been rewritten for pretty
  good signal and cancelation safety. We're now able to ensure overlap
  data structures are cleaned up so long as you don't longjmp() out of
  out of a signal handler that interrupted an i/o operation. Latencies
  are also improved thanks to the removal of lots of "busy wait" code.
  Waits should be optimal for everything except poll(), which shall be
  the last and final demon we slay in the win32 i/o horror show.

- getrusage() on Windows is now able to report RUSAGE_CHILDREN as well
  as RUSAGE_SELF, thanks to aggregation in the process manager thread.
2023-10-08 08:59:53 -07:00
Justine Tunney
ff77f2a6af
Make improvements
- This change fixes a bug that allowed unbuffered printf() output (to
  streams like stderr) to be truncated. This regression was introduced
  some time between now and the last release.

- POSIX specifies all functions as thread safe by default. This change
  works towards cleaning up our use of the @threadsafe / @threadunsafe
  documentation annotations to reflect that. The goal is (1) to use
  @threadunsafe to document functions which POSIX say needn't be thread
  safe, and (2) use @threadsafe to document functions that we chose to
  implement as thread safe even though POSIX didn't mandate it.

- Tidy up the clock_gettime() implementation. We're now trying out a
  cleaner approach to system call support that aims to maintain the
  Linux errno convention as long as possible. This also fixes bugs that
  existed previously, where the vDSO errno wasn't being translated
  properly. The gettimeofday() system call is now a wrapper for
  clock_gettime(), which reduces bloat in apps that use both.

- The recently-introduced improvements to the execute bit on Windows has
  had bugs fixed. access(X_OK) on a directory on Windows now succeeds.
  fstat() will now perform the MZ/#! ReadFile() operation correctly.

- Windows.h is no longer included in libc/isystem/, because it confused
  PCRE's build system into thinking Cosmopolitan is a WIN32 platform.
  Cosmo's Windows.h polyfill was never even really that good, since it
  only defines a subset of the subset of WIN32 APIs that Cosmo defines.

- The setlongerjmp() / longerjmp() APIs are removed. While they're nice
  APIs that are superior to the standardized setjmp / longjmp functions,
  they weren't superior enough to not be dead code in the monorepo. If
  you use these APIs, please file an issue and they'll be restored.

- The .com appending magic has now been removed from APE Loader.
2023-10-03 06:17:16 -07:00
Justine Tunney
c88f95a892
Remove Windows executable path guessing logic
Unlike CMD.EXE, CreateProcess() doesn't care if an executable name ends
with .COM or .EXE. We now have the unbourne shell and bash working well
on Windows, so we don't need DOS anymore. Making this change will grant
us better performance, particularly for builds, because commandv() will
need to make fewer system calls. Path mangling magic still happens with
WinMain() and ntspawn() in order to do things like turn \ into / so the
interop works well at the borders. But all the code in libraries, which
did that, has been removed. It's not possible for libraries to abstract
the differences between paths.
2023-09-21 08:13:50 -07:00
Justine Tunney
ec480f5aa0
Make improvements
- Every unit test now passes on Apple Silicon. The final piece of this
  puzzle was porting our POSIX threads cancelation support, since that
  works differently on ARM64 XNU vs. AMD64. Our semaphore support on
  Apple Silicon is also superior now compared to AMD64, thanks to the
  grand central dispatch library which lets *NSYNC locks go faster.

- The Cosmopolitan runtime is now more stable, particularly on Windows.
  To do this, thread local storage is mandatory at all runtime levels,
  and the innermost packages of the C library is no longer being built
  using ASAN. TLS is being bootstrapped with a 128-byte TIB during the
  process startup phase, and then later on the runtime re-allocates it
  either statically or dynamically to support code using _Thread_local.
  fork() and execve() now do a better job cooperating with threads. We
  can now check how much stack memory is left in the process or thread
  when functions like kprintf() / execve() etc. call alloca(), so that
  ENOMEM can be raised, reduce a buffer size, or just print a warning.

- POSIX signal emulation is now implemented the same way kernels do it
  with pthread_kill() and raise(). Any thread can interrupt any other
  thread, regardless of what it's doing. If it's blocked on read/write
  then the killer thread will cancel its i/o operation so that EINTR can
  be returned in the mark thread immediately. If it's doing a tight CPU
  bound operation, then that's also interrupted by the signal delivery.
  Signal delivery works now by suspending a thread and pushing context
  data structures onto its stack, and redirecting its execution to a
  trampoline function, which calls SetThreadContext(GetCurrentThread())
  when it's done.

- We're now doing a better job managing locks and handles. On NetBSD we
  now close semaphore file descriptors in forked children. Semaphores on
  Windows can now be canceled immediately, which means mutexes/condition
  variables will now go faster. Apple Silicon semaphores can be canceled
  too. We're now using Apple's pthread_yield() funciton. Apple _nocancel
  syscalls are now used on XNU when appropriate to ensure pthread_cancel
  requests aren't lost. The MbedTLS library has been updated to support
  POSIX thread cancelations. See tool/build/runitd.c for an example of
  how it can be used for production multi-threaded tls servers. Handles
  on Windows now leak less often across processes. All i/o operations on
  Windows are now overlapped, which means file pointers can no longer be
  inherited across dup() and fork() for the time being.

- We now spawn a thread on Windows to deliver SIGCHLD and wakeup wait4()
  which means, for example, that posix_spawn() now goes 3x faster. POSIX
  spawn is also now more correct. Like Musl, it's now able to report the
  failure code of execve() via a pipe although our approach favors using
  shared memory to do that on systems that have a true vfork() function.

- We now spawn a thread to deliver SIGALRM to threads when setitimer()
  is used. This enables the most precise wakeups the OS makes possible.

- The Cosmopolitan runtime now uses less memory. On NetBSD for example,
  it turned out the kernel would actually commit the PT_GNU_STACK size
  which caused RSS to be 6mb for every process. Now it's down to ~4kb.
  On Apple Silicon, we reduce the mandatory upstream thread size to the
  smallest possible size to reduce the memory overhead of Cosmo threads.
  The examples directory has a program called greenbean which can spawn
  a web server on Linux with 10,000 worker threads and have the memory
  usage of the process be ~77mb. The 1024 byte overhead of POSIX-style
  thread-local storage is now optional; it won't be allocated until the
  pthread_setspecific/getspecific functions are called. On Windows, the
  threads that get spawned which are internal to the libc implementation
  use reserve rather than commit memory, which shaves a few hundred kb.

- sigaltstack() is now supported on Windows, however it's currently not
  able to be used to handle stack overflows, since crash signals are
  still generated by WIN32. However the crash handler will still switch
  to the alt stack, which is helpful in environments with tiny threads.

- Test binaries are now smaller. Many of the mandatory dependencies of
  the test runner have been removed. This ensures many programs can do a
  better job only linking the the thing they're testing. This caused the
  test binaries for LIBC_FMT for example, to decrease from 200kb to 50kb

- long double is no longer used in the implementation details of libc,
  except in the APIs that define it. The old code that used long double
  for time (instead of struct timespec) has now been thoroughly removed.

- ShowCrashReports() is now much tinier in MODE=tiny. Instead of doing
  backtraces itself, it'll just print a command you can run on the shell
  using our new `cosmoaddr2line` program to view the backtrace.

- Crash report signal handling now works in a much better way. Instead
  of terminating the process, it now relies on SA_RESETHAND so that the
  default SIG_IGN behavior can terminate the process if necessary.

- Our pledge() functionality has now been fully ported to AARCH64 Linux.
2023-09-18 21:04:47 -07:00
Justine Tunney
26e254fb4d
Overhaul process spawning 2023-09-10 08:17:44 -07:00
Justine Tunney
425c055116
Make improvements
- Polyfill readlink("foo/") dir check on Windows
- Support asynchronous signal delivery on Windows
- Restore Windows Console from execve() daisy chain
- Work around bug in AARCH64 Optimized Routines memcmp()
- Disable unbourne.com shell completion on Windows for now
- Don't always set virtual terminal input state on console
- Remove Musl Libc's unusual preservation of realpath("//")
- Make realpath() strongly link malloc() to pass configure test
- Delete cosh.com shell, now that unbourne.com works on Windows!
2023-09-06 22:48:05 -07:00
Justine Tunney
f531acc8f9
Make improvements
- Invent openatemp() API
- Invent O_UNLINK open flag
- Introduce getenv_secure() API
- Remove `git pull` from cosmocc
- Fix utimes() when path is NULL
- Fix mktemp() to never return NULL
- Fix utimensat() UTIME_OMIT on XNU
- Improve utimensat() code for RHEL5
- Turn `argv[0]` C:/ to /C/ on Windows
- Introduce tmpnam() and tmpnam_r() APIs
- Fix more const issues with internal APIs
- Permit utimes() on WIN32 in O_RDONLY mode
- Fix fdopendir() to check fd is a directory
- Fix recent crash regression in landlock make
- Fix futimens(AT_FDCWD, NULL) to return EBADF
- Use workaround so `make -j` doesn't fork bomb
- Rename dontdiscard to __wur (just like glibc)
- Fix st_size for WIN32 symlinks containing UTF-8
- Introduce stdio ext APIs needed by GNU coreutils
- Fix lstat() on WIN32 for symlinks to directories
- Move some constants from normalize.inc to limits.h
- Fix segv with memchr() and memcmp() overlapping page
- Implement POSIX fflush() behavior for reader streams
- Implement AT_SYMLINK_NOFOLLOW for utimensat() on WIN32
- Don't change read-only status of existing files on WIN32
- Correctly handle `0x[^[:xdigit:]]` case in strtol() functions
2023-09-06 12:34:59 -07:00
Justine Tunney
0d748ad58e
Fix warnings
This change fixes Cosmopolitan so it has fewer opinions about compiler
warnings. The whole repository had to be cleaned up to be buildable in
-Werror -Wall mode. This lets us benefit from things like strict const
checking. Some actual bugs might have been caught too.
2023-09-01 20:50:18 -07:00
Justine Tunney
63a1636e1f
Get GNU GMP test suite fully passing
- Fix stdio fmemopen() buffer behaviors
- Fix scanf() to return EOF when appropriate
- Prefer fseek/ftell names over fseeko/ftello
- Ensure locale field is always set in the TIB
- Fix recent regression in vfprintf() return count
- Make %n directive in scanf() have standard behavior
2023-08-21 10:16:42 -07:00
Justine Tunney
ebf784d4f5
Make improvements
- Introduce ualarm() function
- Make rename() report EISEMPTY on Windows
- Always raise EINVAL upon open(O_RDONLY|O_TRUNC)
- Add macro so ./configure will detect SOCK_CLOEXEC
- Fix O_TRUNC without O_CREAT not working on Windows
- Let fcntl(F_SETFL) change O_APPEND status on Windows
- Make sure pwrite() / pread() report ESPIPE on sockets
- Raise ESPIPE on Windows when pwrite() is used on pipe
- Properly compute O_APPEND CreateFile() flags on Windows
- Don't require O_DIRECTORY to open directories on Windows
- Fix more instances of Windows reporting EISDIR and ENOTDIR
- Normalize EFTYPE and EMLINK to ELOOP on NetBSD and FreeBSD
- Make unlink() / rmdir() work on read-only files on Windows
- Validate UTF-8 on Windows paths to fix bug with overlong NUL
- Always print signal name to stderr when crashing due to SIG_DFL
- Fix Windows bug where denormalized paths >260 chars didn't work
- Block signals on BSDs when thread exits before trashing its own stack
2023-08-21 02:34:17 -07:00
Justine Tunney
c776a32f75
Replace COSMO define with _COSMO_SOURCE
This change might cause ABI breakages for /opt/cosmos. It's needed to
help us better conform to header declaration practices.
2023-08-13 20:55:04 -07:00
Justine Tunney
3f2f0e3a74
Make fatcosmocc good enough to build ncurses 6.4 2023-08-12 22:30:05 -07:00
Justine Tunney
decf216655
Perform inconsequential code cleanup 2023-08-07 20:24:50 -07:00
Justine Tunney
712b1aab3c
Make quick fix for aarch64 build 2023-07-30 15:06:28 -07:00
Justine Tunney
c8aa33e0e2
Improve wait statuses
This change has the insight that dwExitCode isn't an exit code but
rather should be used to pass the wait status. This lets us report
killing as a termination status, similar to UNIX. This change also
fixes the fact that exit(259) on Windows will break the parent due
way WIN32 is designed. We now work around that.

It turns out that NetBSD and OpenBSD, will let you have exit codes
beyond 255. This change will let you use them when it's possible.
2023-07-30 14:51:37 -07:00
Justine Tunney
d9d5f45e2d
Restore missing cosmopolitan documentation on website 2023-07-30 11:07:53 -07:00
Justine Tunney
7e0a09feec
Mint APE Loader v1.5
This change ports APE Loader to Linux AARCH64, so that Raspberry Pi
users can run programs like redbean, without the executable needing
to modify itself. Progress has also slipped into this change on the
issue of making progress better conforming to user expectations and
industry standards regarding which symbols we're allowed to declare
2023-07-26 13:54:49 -07:00
Justine Tunney
e0c2b91b3e
Remove _Hide keyword
It never did anything and isn't worthwhile as documentation.
2023-07-24 08:34:58 -07:00
tkchia
0ffc0dd461
Let lz4toasm accept LZ4 files that lack extracted-size field (#858)
lz4toasm should now more easily accept LZ4 files output by
compressor programs that do not support the extracted-size
field, such as Stephan Brumme's smallz4.

This patch also proposes to add a new lz4len() function to
the libc: it parses an LZ4 compressed block to compute the
unpacked content size, without really unpacking the block.

Co-authored-by: tkchia <tkchia-cosmo@gmx.com>
2023-07-23 11:12:22 -07:00
Justine Tunney
a2d269dc38
Brush up some more code 2023-07-10 10:17:26 -07:00
Justine Tunney
f7ae50462a
Make improvements
- Fix unused local variable errors
- Remove yoinks from sigaction() header
- Add nox87 and aarch64 to github actions
- Fix cosmocc -fportcosmo in linking mode
- It's now possible to build `make m=llvm o/llvm/libc`
2023-07-10 04:35:14 -07:00
Justine Tunney
41396ff48a
Make fixes and improvements
- Fix handling of precision in hex float formatting
- Enhance the cocmd interpreter for system() and popen()
- Manually ran the Lua unit tests, which are now passing
- Let stdio i/o operations happen when file is in error state
- We're now saving and restoring xmm in ftrace out of paranoia
2023-07-09 05:21:11 -07:00
Justine Tunney
0d3c1c8b1a
Do work on curl/mbedtls/zstd
This change fixes stderr to be unbuffered. Added hardware AES on ARM64
to help safeguard against timing attacks. The curl.com command will be
somewhat more pleasant to use.
2023-07-07 10:13:35 -07:00
Justine Tunney
00acd81b2f
Delete more dead code 2023-07-06 09:12:28 -07:00
Justine Tunney
0a24b4fc3c
Clean up more code
The *NSYNC linked list API is good enough that it deserves to be part of
the C libray, so this change writes an improved version of it which uses
that offsetof() trick from the Linux Kernel. We vendor all of the *NSYNC
tests in third_party which helped confirm the needed refactoring is safe

This change also deletes more old code that didn't pan out. My goal here
is to work towards a vision where the Cosmopolitan core libraries become
less experimental and more focused on curation. This better reflects the
current level of quality we've managed to achieve.
2023-07-06 08:03:24 -07:00
Justine Tunney
d7c79f43ef
Clean up more code
- Found some bugs in LLVM compiler-rt library
- The useless LIBC_STUBS package is now deleted
- Improve the overflow checking story even further
- Get chibicc tests working in MODE=dbg mode again
- The libc/isystem/ headers now have correctly named guards
2023-06-18 01:00:05 -07:00
Justine Tunney
12452976bd
Remove getopt_long() from headers
We removed this API a long time ago and it was breaking the GNU make
open source build.
2023-06-17 16:03:16 -07:00
Justine Tunney
b881c0ec9e
Remove printf() linking hack 2023-06-17 10:13:50 -07:00
Justine Tunney
52d28966f7
Remove old zip base skew hack
Since 8ff48201ca we no longer need the
hack where, when running .com.dbg files, we scanned for the embedded
.com file offset, and then computed zip offsets realtive to that. It
wasn't very reliable in the first place, and was causing issues with
running our new .com.dbg executables, which are true zip files.
2023-06-17 04:20:16 -07:00
Justine Tunney
4778cd4d27
Fix bugs in termios library and cleanup code
This change fixes an issue with the tcflow() magic numbers that was
causing bash to freeze up on Linux. While auditing termios polyfills,
several other issues were identified with XNU/BSD compatibility.

Out of an abundance of caution this change undefines as much surface
area from libc/calls/struct/termios.h as possible, so that autoconf
scripts are less likely to detect non-POSIX teletypewriter APIs that
haven't been polyfilled by Cosmopolitan.

This is a *breaking change* for your static archives in /opt/cosmos if
you use the cosmocc toolchain. That's because this change disables the
ioctl() undiamonding trick for code outside the monorepo, specifically
because it'll lead to brittle ABI breakages like this. If you're using
the cosmocc toolchain, you'll need to rebuild libraries like ncurses,
readline, etc. Yes diamonds cause bloat. To work around that, consider
using tcgetwinsize() instead of ioctl(TIOCGWINSZ) since it'll help you
avoid pulling every single ioctl-related polyfill into the linkage.

The cosmocc script was specifying -DNDEBUG for some reason. It's fixed.
2023-06-14 19:30:52 -07:00
Justine Tunney
8ff48201ca
Rewrite .zip.o file linker
This change takes an entirely new approach to the incremental linking of
pkzip executables. The assets created by zipobj.com are now treated like
debug data. After a .com.dbg is compiled, fixupobj.com should be run, so
it can apply fixups to the offsets and move the zip directory to the end
of the file. Since debug data doesn't get objcopy'd, a new tool has been
introduced called zipcopy.com which should be run after objcopy whenever
a .com file is created. This is all automated by the `cosmocc` toolchain
which is rapidly becoming the new recommended approach.

This change also introduces the new C23 checked arithmetic macros.
2023-06-10 09:29:44 -07:00
Justine Tunney
4a59210008
Introduce #include <cosmo.h> to toolchain users
This change improves the way internal APIs are being hidden behind the
`COSMO` define. The cosmo.h header will take care of defining that, so
that a separate define statement isn't needed. This change also does a
lot more to define which APIs are standard, and which belong to Cosmo.
2023-06-09 18:03:05 -07:00
Justine Tunney
9b55dbe417
Get GCC to mostly build with Cosmo 2023-06-09 06:41:34 -07:00
Justine Tunney
4b2023ffab
Disable linker map generation and improve tinyness 2023-06-09 03:29:26 -07:00
Justine Tunney
23e235b7a5
Fix bugs in cosmocc toolchain
This change integrates e58abc1110b335a3341e8ad5821ad8e3880d9bb2 from
https://github.com/ahgamut/musl-cross-make/ which fixes the issues we
were having with our C language extension for symbolic constants. This
change also performs some code cleanup and bug fixes to getaddrinfo().
It's now possible to compile projects like ncurses, readline and python
without needing to patch anything upstream, except maybe a line or two.
Pretty soon it should be possible to build a Linux distro on Cosmo.
2023-06-08 23:44:03 -07:00
Justine Tunney
4d629fd424
Fix stack abuse in llama.cc
This change also incorporates improvements for MODE=asan. It's been
confirmed that o/asan/third_party/ggml/llama.com will work.

Fixes #829
2023-06-08 07:12:26 -07:00
Justine Tunney
32682f0ce7
Remove some problematic APIs
In order to improve our chances of success building other open source
projects we shouldn't define APIs that'll lead any ./configure script
astray. For example:

- brk() and sbrk() can break mac/windows support
- syscall() is a superb way to break portability
- arch_prctl() is the greatest of all horror shows
2023-06-08 06:12:26 -07:00
Justine Tunney
61b9677c05
Make improvements
- Get mprotect_test working on aarch64
- Get completion working on python.com repl again
- Improve quality of printvideo.com and printimage.com
- Fix bug in openpty() so examples/script.c works again
2023-06-06 09:12:30 -07:00
Justine Tunney
b8a6a989c0
Create ELF aliases for identical symbols
This change greatly reduces the number of modules that need to be
compiled. The only issue right now is that sometimes when viewing
symbol table entries, the aliased symbol is chosen.
2023-06-06 03:33:49 -07:00
Justine Tunney
eb40cb371d
Get --ftrace working on aarch64
This change implements a new approach to function call logging, that's
based on the GCC flag: -fpatchable-function-entry. Read the commentary
in build/config.mk to learn how it works.
2023-06-05 23:35:31 -07:00
Justine Tunney
1422e96b4e
Introduce native support for MacOS ARM64
There's a new program named ape/ape-m1.c which will be used to build an
embeddable binary that can load ape and elf executables. The support is
mostly working so far, but still chasing down ABI issues.
2023-05-20 04:17:03 -07:00
Justine Tunney
210187cf77
Perform some code cleanup 2023-05-15 16:32:10 -07:00
Justine Tunney
cc1732bc42
Make AARCH64 harder, better, faster, stronger
- Perform some housekeeping on scalar math function code
- Import ARM's Optimized Routines for SIMD string processing
- Upgrade to latest Chromium zlib and enable more SIMD optimizations
2023-05-15 02:15:34 -07:00
Justine Tunney
fd34ef732d
Make considerably more progress on AARCH64
- Utilities like pledge.com now build
- kprintf() will no longer balk at 48-bit addresses
- There's a new aarch64-dbg build mode that should work
- gc() and defer() are mostly pacified; avoid using them on aarch64
- THIRD_PART_STB now has Arm Neon intrinsics for fast image handling
2023-05-12 22:42:57 -07:00
Justine Tunney
414667b1c9
Get TEST_LIBC_STR passing on AARCH64
It's now possible to run commands like:

    make -j8 m=aarch64 o/aarch64/test/libc/str

Which will cross-compile and run the test suites in a qemu-aarch64
binary that's vendored in the third_party/qemu/ folder within your
x86_64 build environment.
2023-05-12 18:09:23 -07:00
Justine Tunney
5e2f7f7ced
Get LIBC_TESTLIB building on AARCH64 2023-05-11 19:57:09 -07:00
Justine Tunney
4e81d3277c
Do some basic housekeeping in LIBC_STR 2023-05-10 06:17:20 -07:00
Justine Tunney
5a455eaa0b
Work on magic numbers for aarch64 2023-05-10 04:20:48 -07:00
Justine Tunney
d04430f4ef
Get LIBC_MEM and LIBC_STDIO building with aarch64 2023-05-10 04:20:47 -07:00
Justine Tunney
7e46645193
Get LIBC_FMT to build for aarch64 2023-05-10 04:20:47 -07:00
Justine Tunney
036b9a0002
Make further progress on non-x86 support 2023-05-10 04:20:47 -07:00
Justine Tunney
aef9a69a60
Make more progress on aarch64 2023-05-10 04:20:47 -07:00
Justine Tunney
2b73e72d59
Make more code aarch64 friendly 2023-05-10 04:20:46 -07:00
Justine Tunney
ca2860947f
Make progress towards aarch64 build 2023-05-10 04:20:46 -07:00
Justine Tunney
369f9740de
Run clang-format on most sources 2023-04-27 05:44:32 -07:00
Justine Tunney
614229e3f4
Make some sorting functions faster 2023-04-24 23:11:18 -07:00
Gabriel Ravier
ba42248575
Fix larger than WCHAR_MAX differences in wcs{,n}cmp (#795)
The C standard states:

> Unless explicitly stated otherwise, the functions described in this
> subclause order two wide characters the same way as two integers of
> the underlying integer type designated by wchar_t.
>
> [...]
>
> The wcscmp function returns an integer greater than, equal to, or
> less than zero, accordingly as the wide string pointed to by s1 is
> greater than, equal to, or less than the wide string pointed to by
> s2.
>
> [...]
>
> The wcsncmp function returns an integer greater than, equal to, or
> less than zero, accordingly as the possibly null-terminated array
> pointed to by s1 is greater than, equal to, or less than the
> possibly null-terminated array pointed to by s2.
- C Standard, 7.31.4.4. Wide string comparison functions

Cosmopolitan fails to obey this in cases where the difference between
two wide characters is larger than WCHAR_MAX.

This means that, for example, the following program:

 #include <stdio.h>
 #include <wchar.h>
 #include <limits.h>

int main()
{
    wchar_t str1[] = { WCHAR_MIN, L'\0' };
    wchar_t str2[] = { WCHAR_MAX, L'\0' };
    printf("%d\n", wcscmp(str1, str2));
    printf("%d\n", wcsncmp(str1, str2, 2));
}

will print `1` twice, instead of the negative numbers mandated by the
standard (as WCHAR_MIN is less than WCHAR_MAX)

This patch fixes this, along with the associated Github issue,
https://github.com/jart/cosmopolitan/issues/783
2023-03-29 20:28:18 -04:00
Gavin Hayes
c5de653b98
GetZipCdir: prevent integer underflow. posix_spawn_test.c: stop attempting to load zipos from /usr/bin/ape. (#758) 2023-03-05 14:29:38 -08:00
Justine Tunney
1473eafd1a
Fix the build and introduce some APIs
The tcgetwinsize() and tcsetwinsize() APIs are now available. The
printargs.com example also now displays the baud rate.
2023-01-20 09:25:45 -08:00
Justine Tunney
dd04aeba1c
Increase stack size to 128k and guard size to 16k
This improves our compatibility with Apple M1.
2022-12-18 22:58:29 -08:00
Justine Tunney
bf7843833f
Rename hidden keyword to _Hide 2022-11-08 12:55:28 -08:00
tkchia
cb9a0466f1
Work towards zipos / open(argv[0]) on metal (#667) 2022-11-06 00:29:47 -07:00
Justine Tunney
3f0bcdc3ef
Improve cancellations, randomness, and time
- Exhaustively document cancellation points
- Rename SIGCANCEL to SIGTHR just like BSDs
- Further improve POSIX thread cancellations
- Ensure asynchronous cancellations work correctly
- Elevate the quality of getrandom() and getentropy()
- Make futexes cancel correctly on OpenBSD 6.x and 7.x
- Add reboot.com and shutdown.com to examples directory
- Remove underscore prefix from awesome timespec_*() APIs
- Create assertions that help verify our cancellation points
- Remove bad timespec APIs (cmp generalizes eq/ne/gt/gte/lt/lte)
2022-11-05 23:45:32 -07:00
Justine Tunney
f7ff77d865
Make fixes and improvements
- Invent iso8601us() for faster timestamps
- Improve --strace descriptions of sigset_t
- Rebuild the Landlock Make bootstrap binary
- Introduce MODE=sysv for non-Windows builds
- Permit OFD fcntl() locks under pledge(flock)
- redbean can now protect your kernel from ddos
- Have vfork() fallback to sys_fork() not fork()
- Change kmalloc() to not die when out of memory
- Improve documentation for some termios functions
- Rewrite putenv() and friends to conform to POSIX
- Fix linenoise + strace verbosity issue on Windows
- Fix regressions in our ability to show backtraces
- Change redbean SetHeader() to no-op if value is nil
- Improve fcntl() so SQLite locks work in non-WAL mode
- Remove some unnecessary work during fork() on Windows
- Create redbean-based SSL reverse proxy for IPv4 TurfWar
- Fix ape/apeinstall.sh warning when using non-bash shells
- Add ProgramTrustedIp(), and IsTrustedIp() APIs to redbean
- Support $PWD, $UID, $GID, and $EUID in command interpreter
- Introduce experimental JTqFpD APE prefix for non-Windows builds
- Invent blackhole daemon for firewalling IP addresses via UNIX named socket
- Add ProgramTokenBucket(), AcquireToken(), and CountTokens() APIs to redbean
2022-10-19 07:19:19 -07:00
Justine Tunney
60cb435cb4
Implement pthread_atfork()
If threads are being used, then fork() will now acquire and release and
runtime locks so that fork() may be safely used from threads. This also
makes vfork() thread safe, because pthread mutexes will do nothing when
the process is a child of vfork(). More torture tests have been written
to confirm this all works like a charm. Additionally:

- Invent hexpcpy() api
- Rename nsync_malloc_() to kmalloc()
- Complete posix named semaphore implementation
- Make pthread_create() asynchronous signal safe
- Add rm, rmdir, and touch to command interpreter builtins
- Invent sigisprecious() and modify sigset functions to use it
- Add unit tests for posix_spawn() attributes and fix its bugs

One unresolved problem is the reclaiming of *NSYNC waiter memory in the
forked child processes, within apps which have threads waiting on locks
2022-10-16 12:25:13 -07:00
Justine Tunney
b41f91c658
Greatly expand system() shell code features
The cosmopolitan command interpreter now has 13 builtin commands,
variable support, support for ; / && / || syntax, asynchronous support,
and plenty of unit tests with bug fixes.

This change fixes a bug in posix_spawn() with null envp arg. strace
logging now uses atomic writes for scatter functions. Breaking change
renaming GetCpuCount() to _getcpucount(). TurfWar is now updated to use
the new token bucket algorithm. WIN32 affinity masks now inherit across
fork() and execve().
2022-10-11 21:30:31 -07:00
Justine Tunney
e557058ac8
Improve cosmo's conformance to libc-test
This change addresses various open source compatibility issues, so that
we pass 313/411 of the tests in https://github.com/jart/libc-test where
earlier today we were passing about 30/411 of them, due to header toil.
Please note that Glibc only passes 341/411 so 313 today is pretty good!

- Make the conformance of libc/isystem/ headers nearly perfect
- Import more of the remaining math library routines from Musl
- Fix inconsistencies with type signatures of calls like umask
- Write tests for getpriority/setpriority which work great now
- conform to `struct sockaddr *` on remaining socket functions
- Import a bunch of uninteresting stdlib functions e.g. rand48
- Introduce readdir_r, scandir, pthread_kill, sigsetjmp, etc..

Follow the instructions in our `tool/scripts/cosmocc` toolchain to run
these tests yourself. You use `make CC=cosmocc` on the test repository
2022-10-10 17:52:41 -07:00
Justine Tunney
467a332e38
Introduce sigtimedwait() and sigwaitinfo()
This change also invents sigcountset() and strsignal_r() and improves
the quality of siginfo_t handling.
2022-10-10 07:39:44 -07:00
Justine Tunney
d5910e2673
Fix bugs and make code tinier
- Fixed bug where stdio eof wasn't being sticky
- Fixed bug where fseeko() wasn't clearing eof state
- Removed assert() usage from libc favoring _unassert() / _npassert()
2022-10-09 23:21:34 -07:00
Hugues Morisset
f155205eb0
Add imaxdiv, wcscoll, getdtablesize (#639) 2022-10-05 07:14:58 -07:00
Justine Tunney
775944a2d0
Fix important bugs in redbean
This change upgrades to the latest Chromium Zlib, fixes bugs in redbean,
and introduces better support for reverse proxies like Cloudflare. This
change improves the security of redbean and it's recommended that users
upgrade to the release that'll follow. This change also updates the docs
to clarify how to use the security tools redbean provides e.g. pledge(),
unveil(), and the MODE=asan builds which improve memory safety.
2022-09-17 02:18:06 -07:00
Justine Tunney
3f49889841
Make important improvements
- Fix preadv() and pwritev() for old distros
- Introduce _npassert() and _unassert() macros
- Prove that file locks work properly on Windows
- Support fcntl(F_DUPFD_CLOEXEC) on more systems
2022-09-14 22:39:08 -07:00
Justine Tunney
1ad2f530f9
Add script.com and whois.com courtesy of FreeBSD 2022-09-13 20:26:13 -07:00
Justine Tunney
6f7d0cb1c3
Pay off more technical debt
This makes breaking changes to add underscores to many non-standard
function names provided by the c library. MODE=tiny is now tinier and
we now use smaller locks that are better for tiny apps in this mode.
Some headers have been renamed to be in the same folder as the build
package, so it'll be easier to know which build dependency is needed.
Certain old misguided interfaces have been removed. Intel intrinsics
headers are now listed in libc/isystem (but not in the amalgamation)
to help further improve open source compatibility. Header complexity
has also been reduced. Lastly, more shell scripts are now available.
2022-09-12 23:36:56 -07:00
Justine Tunney
155b378a39
Tidy up the threading implementation
The organization of the source files is now much more rational.
Old experiments that didn't work out are now deleted. Naming of
things like files is now more intuitive.
2022-09-10 02:56:25 -07:00
Justine Tunney
9f963dc597
Clean up some of the threading code 2022-09-08 12:31:56 -07:00
Justine Tunney
0547eabcd6
Polyfill IPv6 on non-Linux 2022-09-08 06:06:22 -07:00
Justine Tunney
d861d2787b
Import OpenBSD sorting algorithms
OpenBSD's qsort() function is more secure than the ones used by
FreeBSD, NetBSD and MacOS. The best part is it goes faster too!
This change also imports the OpenBSD mergesort() and heapsort()
2022-09-06 17:52:26 -07:00
Justine Tunney
dbf12c30b0
Make more compatibility improvements 2022-09-06 12:36:34 -07:00
Justine Tunney
11ec60d5fc
Use a better sorting algorithm
This change changes qsort() to use the same code as NetBSD and MacOS
because it goes 6x faster than Musl's SmoothSort function. Smoothsort
can still be used if you need something that's provenly linearithmic.
This change also improves GNU Make performance on whole by 7 percent!

netbsd nearly   l:    70,196c    22,673ns   m:    68,428c    22,102ns
musl nearly     l:    53,844c    17,391ns   m:    58,726c    18,968ns
unixv6 nearly   l:    65,885c    21,280ns   m:    63,082c    20,375ns

netbsd reverse  l:   120,290c    38,853ns   m:   122,619c    39,605ns
musl reverse    l:   801,826c   258,985ns   m:   794,689c   256,680ns
unixv6 reverse  l:    58,977c    19,049ns   m:    59,764c    19,303ns

netbsd random   l:   146,745c    47,398ns   m:   145,782c    47,087ns
musl random     l:   855,804c   276,420ns   m:   850,912c   274,840ns
unixv6 random   l:   214,325c    69,226ns   m:   213,906c    69,090ns

netbsd 2n       l:    77,299c    24,967ns   m:    76,773c    24,797ns
musl 2n         l:   818,012c   264,213ns   m:   818,282c   264,301ns
unixv6 2n       l: 3,967,009c 1,281,322ns   m: 3,941,792c 1,273,177ns

https://justine.lol/dox/sort.pdf
2022-09-06 11:06:10 -07:00
Justine Tunney
55c6297e13
Make more compatibility improvements 2022-09-06 07:04:13 -07:00
Justine Tunney
8dd4ec68d0
Add more missing C / C++ headers 2022-09-04 04:53:52 -07:00
Justine Tunney
35203c0551 Do some string library work 2022-08-20 22:17:14 -07:00
Gavin Hayes
27416e7dd6
Add fake support for locale="" (#546) 2022-08-18 15:51:23 -07:00
Justine Tunney
8e176fb026 Reduce build graph by another 14 percent
That's a 37% reduction total from what it was on 2022-05-23.
2022-08-17 20:01:21 -07:00
Justine Tunney
d36d0634db Add automatic TMPDIR setup/teardown to GNU Make
We now guarantee TMPDIR will be defined on a per build rule basis. It'll
be an absolute path. It'll be secure and unique. It'll be rm -rf'd after
the last shell script line in your build rule is executed. If $TMPDIR is
already defined, then it'll be created as a subdirectory of your $TMPDIR
and then replace the variable with the new definition. The Landlock Make
repository will be updated with examples shortly after this change which
shall be known as Landlock Make 1.1.1.

See #530
2022-08-14 02:03:41 -07:00
Justine Tunney
0ea0d33a77 Reduce build graph by another eight percent 2022-08-13 13:11:56 -07:00
Justine Tunney
367d06d9e4 Fold LIBC_UNICODE into LIBC_STR 2022-08-13 08:42:32 -07:00
Justine Tunney
17aea99bb3 Fold LIBC_ALG into LIBC_MEM 2022-08-13 08:32:34 -07:00
Justine Tunney
05b8f82371 Fold LIBC_BITS into LIBC_INTRIN 2022-08-11 12:13:18 -07:00
Justine Tunney
10fd8bdb70 Unbloat the build
This change resurrects ae5d06dc53
2022-08-11 00:15:29 -07:00
Justine Tunney
7b993d561c Use private keyword on makefile target variables 2022-08-10 18:36:11 -07:00
Justine Tunney
c1d99676c4 Revert "Unbloat build config"
This reverts commit ae5d06dc53.
2022-08-10 12:44:56 -07:00
Justine Tunney
ae5d06dc53 Unbloat build config
- 10.5% reduction of o//depend dependency graph
- 8.8% reduction in latency of make command
- Fix issue with temporary file cleanup

There's a new -w option in compile.com that turns off the recent
Landlock output path workaround for "good commands" which do not
unlink() the output file like GNU tooling does.

Our new GNU Make unveil sandboxing appears to have zero overhead
in the grand scheme of things. Full builds are pretty fast since
the only thing that's actually slowed us down is probably libcxx

    make -j16 MODE=rel
    RL: took 85,732,063µs wall time
    RL: ballooned to 323,612kb in size
    RL: needed 828,560,521µs cpu (11% kernel)
    RL: caused 39,080,670 page faults (99% memcpy)
    RL: 350,073 context switches (72% consensual)
    RL: performed 0 reads and 11,494,960 write i/o operations

pledge() and unveil() no longer consider ENOSYS to be an error.
These functions have also been added to Python's cosmo module.

This change also removes some WIN32 APIs and System Five magnums
which we're not using and it's doubtful anyone else would be too
2022-08-10 04:43:09 -07:00
Justine Tunney
6a5717a48f Make more libc improvements
- Make memmem() faster
- Make readdir() thread safe
- Remove 64kb limit from mkdeps.com
- Add old crypt() function from Musl
- Improve new fix-third-party.py tool
- Improve libc/isystem/ headers and fix bugs
2022-08-06 17:18:40 -07:00
Justine Tunney
a8cf0f7e89 Add more missing libc functionality 2022-08-06 10:50:51 -07:00