Commit graph

36 commits

Author SHA1 Message Date
Justine Tunney
fae2a17d2c Fix build breakage in MODE=dbg 2022-06-15 19:37:39 -07:00
Justine Tunney
425ff5dff0 Refactor some code
- Write tests for cthreads
- Fix bugs in pe2.com tool
- Fix ASAN issue with GetDosEnviron()
- Consolidate the cthread header files
- Some code size optimizations for MODE=
- Attempted to squash a tls linker warning
- Attempted to get futexes working on FreeBSD
2022-05-28 14:09:21 -07:00
Justine Tunney
d230a01222 Make build hermetic without shell scripts
- Fix some minor issues in ar.com
- Have execve() look for `ape` command
- Rewrite NT paths using /c/ rather /??/c:/
- Replace broken GCC symlinks with .sym files
- Rewrite $PATH environment variables on startup
- Make $(APE_NO_MODIFY_SELF) the default bootloader
- Add all build command dependencies to build/bootstrap
- Get the repository mostly building from source on non-Linux
2022-05-25 13:55:57 -07:00
Justine Tunney
8141988e92 Shave 4kb off default mode binaries 2022-05-20 08:18:20 -07:00
Justine Tunney
9208c83f7a Make some systemic improvements
- add vdso dump utility
- tests now log stack usage
- rename g_ftrace to __ftrace
- make internal spinlocks go faster
- add conformant c11 atomics library
- function tracing now logs stack usage
- make function call tracing thread safe
- add -X unsecure (no ssl) mode to redbean
- munmap() has more consistent behavior now
- pacify fsync() calls on python unit tests
- make --strace flag work better in redbean
- start minimizing and documenting compiler flags
2022-05-18 16:52:36 -07:00
Justine Tunney
55de4ca6b5 Support thread local storage 2022-05-16 13:20:08 -07:00
Justine Tunney
e7611a8476 Make improvements
- Get threads working on NetBSD
- Get threads working on OpenBSD
- Fix Emacs config for Emacs v28
- Improve --strace logging of sigset_t
- Improve --strace logging of struct stat
- Improve memory safety of DescribeThing functions
- Refactor auto stack allocation into LIBC_RUNTIME
- Introduce shell.com example which works on Windows
- Refactor __strace_thing into DescribeThing functions
- Document the CHECK macros and improve them in NDEBUG mode
- Rewrite MAP_STACK so it uses FreeBSD behavior across platforms
- Deprecate and discourage the use of MAP_GROWSDOWN (it's weird)
2022-05-12 06:45:36 -07:00
Justine Tunney
15c59e716f Fix build break in MODE=asan 2022-05-05 11:12:48 -07:00
Justine Tunney
80c4533303 Unbreak MODE=tiny build
This change also fixes a minor issue with pledge() polyfill introduced
the last change setting errno appropriately.

Fixes #396
2022-04-28 10:43:05 -07:00
Justine Tunney
47b3274665 Make improvements
- Add rusage to redbean Lua API
- Add more redbean documentation
- Add pledge() to redbean Lua API
- Polyfill OpenBSD pledge() for Linux
- Increase PATH_MAX limit to 1024 characters
- Untrack sibling processes after fork() on Windows
2022-04-28 09:57:07 -07:00
Justine Tunney
d57b81aac7 Make improvements
- Add GetCpuCount() API to redbean
- Add unix.gmtime() API to redbean
- Add unix.readlink() API to redbean
- Add unix.localtime() API to redbean
- Perfect the new redbean UNIX module APIs
- Integrate with Linux clock_gettime() vDSO
- Run Lua garbage collector when malloc() fails
- Fix another regression quirk with linenoise repl
- Fix GetProgramExecutableName() for systemwide installs
- Fix a build flake with test/libc/mem/test.mk SRCS list
2022-04-26 16:46:15 -07:00
Justine Tunney
451e3f73d9 Improve redbean
- Improve serialization
- Add Benchmark() API to redbean
- Refactor UNIX API to be assert() friendly
- Make the redbean Lua REPL print data structures
- Fix recent regressions in linenoise reverse search
- Add -i flag so redbean can be a language interpreter
2022-04-25 08:30:14 -07:00
Justine Tunney
1599b818d9 Make more fixes and improvements 2022-04-21 13:44:59 -07:00
Justine Tunney
ae638c0850 Fix bugs and make improvements
- Get clone() working on FreeBSD
- Increase some Python build quotas
- Add more atomic builtins to chibicc
- Fix ASAN poisoning of alloca() memory
- Make MODE= mandatory link path tinier
- Improve the examples folder a little bit
- Start working on some more resource limits
- Make the linenoise auto-complete UI as good as GNU readline
- Update compile.com, avoiding AVX codegen on non-AVX systems
- Make sure empty path to syscalls like opendir raises ENOENT
- Correctly polyfill ENOENT vs. ENOTDIR on the New Technology
- Port bestline's paredit features to //third_party/linenoise
- Remove workarounds for RHEL 5.0 bugs that were fixed in 5.1
2022-04-20 10:05:34 -07:00
Justine Tunney
7166679620 Fix bugs and add security features to redbean
- Fix a regression with the previous change that broke redbean
- Add chroot(), resource limit, seccomp, and other stuff to redbean
- Write lots and lots of documentation
- Iron out more system call issues
2022-04-18 00:01:26 -07:00
Justine Tunney
933411ba99 Improve synchronization
- Fix bugs in kDos2Errno definition
- malloc() should now be thread safe
- Fix bug in rollup.com header generator
- Fix open(O_APPEND) on the New Technology
- Fix select() on the New Technology and test it
- Work towards refactoring i/o for thread safety
- Socket reads and writes on NT now poll for signals
- Work towards i/o completion ports on the New Technology
- Make read() and write() intermittently check for signals
- Blinkenlights keyboard i/o so much better on NT w/ poll()
- You can now poll() files and sockets at the same time on NT
- Fix bug in appendr() that manifests with dlmalloc footers off
2022-04-15 15:31:55 -07:00
Justine Tunney
c23b6ecc31 Fix some win32 definitions
You can now use psapi.dll and pdh.dll. Some TODOs for Windows have been
cleared out. We might have a working load average for the platform that
should help GNU Make work well.
2022-03-22 19:54:36 -07:00
Justine Tunney
0cb6b6ff4b Get Redbean fork() working on the New Technology
Now that we have understandable system call tracing on Windows, this
change rewrites many of the polyfill internals for that platform, to
help things get closer to tip top shape. Support for complex forking
scenarios had been in a regressed state for quite some time. Now, it
works! Subsequent changes should be able to address the performance.
2022-03-20 08:01:14 -07:00
Justine Tunney
5b60e5a37d Fix termios struct on Linux
The termios::c_cc field turned out to be incorrectly defined on Linux
due to some confusion between the glibc and kernel definitions. We'll
be using the kernel definition, since it has the strongest consensus.

Fields have been have been added to struct stat for BSD compatibility
such as st_birthtim, plus the GLIBC compatibility of isystem/sys/stat
has been improved.
2021-09-03 22:19:41 -07:00
Justine Tunney
00611e9b06 Improve ZIP filesystem and change its prefix
The ZIP filesystem has a breaking change. You now need to use /zip/ to
open() / opendir() / etc. assets within the ZIP structure of your APE
binary, instead of the previous convention of using zip: or zip! URIs.
This is needed because Python likes to use absolute paths, and having
ZIP paths encoded like URIs simply broke too many things.

Many more system calls have been updated to be able to operate on ZIP
files and file descriptors. In particular fcntl() and ioctl() since
Python would do things like ask if a ZIP file is a terminal and get
confused when the old implementation mistakenly said yes, because the
fastest way to guarantee native file descriptors is to dup(2). This
change also improves the async signal safety of zipos and ensures it
doesn't maintain any open file descriptors beyond that which the user
has opened.

This change makes a lot of progress towards adding magic numbers that
are specific to platforms other than Linux. The philosophy here is that,
if you use an operating system like FreeBSD, then you should be able to
take advantage of FreeBSD exclusive features, even if we don't polyfill
them on other platforms. For example, you can now open() a file with the
O_VERIFY flag. If your program runs on other platforms, then Cosmo will
automatically set O_VERIFY to zero. This lets you safely use it without
the need for #ifdef or ifstatements which detract from readability.

One of the blindspots of the ASAN memory hardening we use to offer Rust
like assurances has always been that memory passed to the kernel via
system calls (e.g. writev) can't be checked automatically since the
kernel wasn't built with MODE=asan. This change makes more progress
ensuring that each system call will verify the soundness of memory
before it's passed to the kernel. The code for doing these checks is
fast, particularly for buffers, where it can verify 64 bytes a cycle.

- Correct O_LOOP definition on NT
- Introduce program_executable_name
- Add ASAN guards to more system calls
- Improve termios compatibility with BSDs
- Fix bug in Windows auxiliary value encoding
- Add BSD and XNU specific errnos and open flags
- Add check to ensure build doesn't talk to internet
2021-08-22 01:11:53 -07:00
Justine Tunney
228fb7428b Improve isystem includes and magic numbers 2021-08-14 23:36:36 -07:00
Justine Tunney
5144c22189 Add test for ioctl(SIOCGIFCONF) and polyfill on BSDs
- Use nullness checks when calling weakly linked functions.

- Avoid typedef for reasons described in Linux Kernel style guide.

- Avoid enum in in Windows headers. Earlier in Cosmo's history all one
  hundred files in libc/nt/enum/ used to be enums and it resulted in
  gigabytes of DWARF data almost as large as everything else in the
  codebase combined.

- Bitfields aren't our friends. They have frequent ABI breakages,
  inconsistent arithmetic across compilers, and different endianness
  between cpus. Compiler authors also haven't invested much roi into
  making bit fields go fast so they produce poor assembly.

- Use memccpy() instead of strncpy() or snprintf() for length-bounded
  copying of C strings. strncpy() is a misunderstood function and
  snprintf() is awesome but memccpy() deserves more love.
2021-06-25 18:44:04 -07:00
fabriziobertocci
fd0eefce17
Add ioctl(SIOCGIFxxx) support (#197)
- SIOCGIFCONFIG: reads and enumerate all the network interfaces
- SIOCGIFADDR: reads network address for a given interface
- SIOCGIFFLAGS: reads network flags for a given interface
- SIOCGIFNETMASK: reads network netmask for a given interface
- SIOCGIFBRDADDR: reads network broadcast address for a given interface
- SIOCGIFDSTADDR: reads peer destination address for a given
  interface (not supported for Windows)

This change defines Linux ABI structs for the above interfaces and adds
polyfills to ensure they behave consistently on XNU and Windows.
2021-06-24 10:53:27 -07:00
Justine Tunney
807706a099 Perform minor fixups
One of those fixups is making sure that AF_LOCAL is equal to AF_UNIX on
the New Technology. See #122
2021-03-13 19:40:04 -08:00
Justine Tunney
19d0c15e03 Perform some code cleanup 2021-02-27 10:33:32 -08:00
Justine Tunney
2f3bd90216 Apply some touchups 2021-02-07 07:02:46 -08:00
Justine Tunney
4e56d89dcd Eliminate some flakes
- Get ASAN working on Windows.

- Deleting directories and then recreating them with the same name in a
  short period of time appears to be a no-no on Windows.

- There's no reason to call FlushFileBuffers on close() for pipes, and
  it's harmful since it might block indefinitely for no good reason.
2021-02-03 06:25:27 -08:00
Justine Tunney
23a14b537c Delete LIBC_CALLS_HEFTY
- fork() no longer requires malloc()
- readdir() moved to LIBC_STDIO
- Custom APIs moved to LIBC_X
2021-02-02 22:17:53 -08:00
Justine Tunney
cbfd4ccd1e Make more functions friendly to Address Sanitizer 2021-02-02 03:45:31 -08:00
Justine Tunney
45b72485ad Fix XNU / FreeBSD / OpenBSD / RHEL5 / NT bugs
For the first time ever, all tests in this codebase now pass, when
run automatically on macos, freebsd, openbsd, rhel5, rhel7, alpine
and windows via the network using the runit and runitd build tools

- Fix vfork exec path etc.
- Add XNU opendir() support
- Add OpenBSD opendir() support
- Add Linux history to syscalls.sh
- Use copy_file_range on FreeBSD 13+
- Fix system calls with 7+ arguments
- Fix Windows with greater than 16 FDs
- Fix RUNIT.COM and RUNITD.COM flakiness
- Fix OpenBSD munmap() when files are mapped
- Fix long double so it's actually long on Windows
- Fix OpenBSD truncate() and ftruncate() thunk typo
- Let Windows fcntl() be used on socket files descriptors
- Fix Windows fstat() which had an accidental printf statement
- Fix RHEL5 CLOCK_MONOTONIC by not aliasing to CLOCK_MONOTONIC_RAW

This is wonderful. I never could have dreamed it would be possible
to get it working so well on so many platforms with tiny binaries.

Fixes #31
Fixes #25
Fixes #14
2021-01-25 18:31:17 -08:00
Justine Tunney
9f68d6eee9 Fix link order in cosmopolitan.a
It turned out that the linker was doing the wrong with the amalgamation
library concerning weak stubs. A regression test has been added and new
binaries have been uploaded to https://justine.lol/cosmopolitan/

Ideally this should be fixed by building a tool that turns multiple .a
files into a single .a file with deduplication. As a workaround for now
the cosmopolitan.a build is restructured to not include LIBC_STUBS which
meant technical debt needed to be paid off where non-stub interfaces
were moved to LIBC_INTRIN and LIBC_NEXGEN32E.

Thank @PerfectProductions in #31 for the report!
2021-01-16 12:05:41 -08:00
Justine Tunney
0e85b136ae Fix Windows 7 support (#19)
This change pays off technical debt with the function -> DLL mappings in
libc/nt/master.sh, which was originally defined based on binary analysis
on Windows 10. It's now been updated so the kernel32/kernelbase/advapi32
imports should be exactly as they are written, on the MSDN documentation
and that wouldn't have been easy without Geoff Chappell's work thank him

https://www.geoffchappell.com/studies/windows/win32/index.htm
2020-12-28 13:52:02 -08:00
Justine Tunney
1fc91f3580 Fold conv package into fmt
Both packages had nearly identical dependency requirements, so merging
them should help reduce the complexity of the build graph.
2020-12-09 16:52:00 -08:00
Justine Tunney
dba7552c1e Add Conway's Game of Life 2020-11-18 08:26:03 -08:00
Justine Tunney
aea89fe832 Make minor improvements
- Work towards simplifying ape.S startup process
- Rewrote ar because it took minutes to build cosmopolitan.a
2020-11-09 15:41:11 -08:00
Justine Tunney
c91b3c5006 Initial import 2020-06-15 07:18:57 -07:00