Commit graph

139 commits

Author SHA1 Message Date
Justine Tunney
8ff48201ca
Rewrite .zip.o file linker
This change takes an entirely new approach to the incremental linking of
pkzip executables. The assets created by zipobj.com are now treated like
debug data. After a .com.dbg is compiled, fixupobj.com should be run, so
it can apply fixups to the offsets and move the zip directory to the end
of the file. Since debug data doesn't get objcopy'd, a new tool has been
introduced called zipcopy.com which should be run after objcopy whenever
a .com file is created. This is all automated by the `cosmocc` toolchain
which is rapidly becoming the new recommended approach.

This change also introduces the new C23 checked arithmetic macros.
2023-06-10 09:29:44 -07:00
Justine Tunney
4a59210008
Introduce #include <cosmo.h> to toolchain users
This change improves the way internal APIs are being hidden behind the
`COSMO` define. The cosmo.h header will take care of defining that, so
that a separate define statement isn't needed. This change also does a
lot more to define which APIs are standard, and which belong to Cosmo.
2023-06-09 18:03:05 -07:00
Justine Tunney
23e235b7a5
Fix bugs in cosmocc toolchain
This change integrates e58abc1110b335a3341e8ad5821ad8e3880d9bb2 from
https://github.com/ahgamut/musl-cross-make/ which fixes the issues we
were having with our C language extension for symbolic constants. This
change also performs some code cleanup and bug fixes to getaddrinfo().
It's now possible to compile projects like ncurses, readline and python
without needing to patch anything upstream, except maybe a line or two.
Pretty soon it should be possible to build a Linux distro on Cosmo.
2023-06-08 23:44:03 -07:00
Justine Tunney
eb40cb371d
Get --ftrace working on aarch64
This change implements a new approach to function call logging, that's
based on the GCC flag: -fpatchable-function-entry. Read the commentary
in build/config.mk to learn how it works.
2023-06-05 23:35:31 -07:00
Justine Tunney
8f522cb702
Make improvements
This change progresses our AARCH64 support:

- The AARCH64 build and tests are now passing
- Add 128-bit floating-point support to printf()
- Fix clone() so it initializes cosmo's x28 TLS register
- Fix TLS memory layout issue with aarch64 _Alignas vars
- Revamp microbenchmarking tools so they work on aarch64
- Make some subtle improvements to aarch64 crash reporting
- Make kisdangerous() memory checks more accurate on aarch64
- Remove sys_open() since it's not available on Linux AARCH64

This change makes general improvements to Cosmo and Redbean:

- Introduce GetHostIsa() function in Redbean
- You can now feature check using pledge(0, 0)
- You can now feature check using unveil("",0)
- Refactor some more x86-specific asm comments
- Refactor and write docs for some libm functions
- Make the mmap() API behave more similar to Linux
- Fix WIFSIGNALED() which wrongly returned true for zero
- Rename some obscure cosmo keywords from noFOO to dontFOO
2023-06-03 08:12:22 -07:00
Justine Tunney
fd34ef732d
Make considerably more progress on AARCH64
- Utilities like pledge.com now build
- kprintf() will no longer balk at 48-bit addresses
- There's a new aarch64-dbg build mode that should work
- gc() and defer() are mostly pacified; avoid using them on aarch64
- THIRD_PART_STB now has Arm Neon intrinsics for fast image handling
2023-05-12 22:42:57 -07:00
Justine Tunney
4edbc98811
Get MbedTLS and its unit tests passing AARCH64 2023-05-11 21:53:15 -07:00
Justine Tunney
d04430f4ef
Get LIBC_MEM and LIBC_STDIO building with aarch64 2023-05-10 04:20:47 -07:00
imawaki
da7c8c7b54
Serve .mjs files as text/javascript (#761) 2023-03-05 14:26:18 -08:00
Michael Ford
ca88fc34c3
doc: remove "everything everything" from ParseUrl doc (#704)
Closes: #702.
2023-02-23 09:34:48 -08:00
Justine Tunney
6dcdf91458
Add a turfwar hilbert decoration 2023-01-03 18:28:35 -08:00
Justine Tunney
a6586cafb2
Retry SQLITE_BUSY when stepping in TurfWar 2023-01-03 06:10:39 -08:00
Justine Tunney
9c72fef086
Remove affinity tuning from IPv4 Games 2023-01-03 04:48:01 -08:00
Justine Tunney
dd04aeba1c
Increase stack size to 128k and guard size to 16k
This improves our compatibility with Apple M1.
2022-12-18 22:58:29 -08:00
Justine Tunney
57c0dcdc29
Decrease TurfWar background query intervals
The database is much larger than it was when we started, so these
background reporting queries have become much more expensive than
before. We also had someone make 750k claims in one day, and even
hundreds of thousands per hour, which broke our assumptions about
hourly / daily reports being cheap.
2022-12-18 02:23:12 -08:00
Justine Tunney
f2af97711b
Make improvements
- Improve compatibility with Blink virtual machine
- Add non-POSIX APIs for joining threads and signal masks
- Never ever use anything except 32-bit integers for atomics
- Add some `#undef` statements to workaround `ctags` problems
2022-11-10 21:52:47 -08:00
Justine Tunney
4de2cf34e6
Allow HTTP range past EOF
Fixes #683
2022-11-07 05:52:24 -08:00
Justine Tunney
c995838e5c
Make improvements
- Clean up sigaction() code
- Add a port scanner example
- Introduce a ParseCidr() API
- Clean up our futex abstraction code
- Fix a harmless integer overflow in ParseIp()
- Use kernel semaphores on NetBSD to make threads much faster
2022-11-07 02:26:06 -08:00
Justine Tunney
3f0bcdc3ef
Improve cancellations, randomness, and time
- Exhaustively document cancellation points
- Rename SIGCANCEL to SIGTHR just like BSDs
- Further improve POSIX thread cancellations
- Ensure asynchronous cancellations work correctly
- Elevate the quality of getrandom() and getentropy()
- Make futexes cancel correctly on OpenBSD 6.x and 7.x
- Add reboot.com and shutdown.com to examples directory
- Remove underscore prefix from awesome timespec_*() APIs
- Create assertions that help verify our cancellation points
- Remove bad timespec APIs (cmp generalizes eq/ne/gt/gte/lt/lte)
2022-11-05 23:45:32 -07:00
Justine Tunney
022536cab6
Make futexes cancellable by pthreads 2022-11-04 18:36:34 -07:00
Justine Tunney
e522aa3a07
Make more threading improvements
- ASAN memory morgue is now lockless
- Make C11 atomics header more portable
- Rewrote pthread keys support to be lockless
- Simplify Python's unicode table unpacking code
- Make crash report write(2) closer to being atomic
- Make it possible to strace/ftrace a single thread
- ASAN now checks nul-terminated strings fast and properly
- Windows fork() now restores TLS memory of calling thread
2022-11-01 23:28:26 -07:00
mattx
ef9776755e
blackholed: allow changing iptables chain (#670)
* Make blackholed use its own chain

This makes blackholed interfere less with any existing iptables raw table configurations.

* Allow specifying iptables chain at runtime

blackholed will now insert blocks to the PREROUTING chain by default,
with a runtime switch to change it to another chain
2022-10-20 10:19:27 -07:00
Justine Tunney
da336b3ea8
Make some more fixes to prod 2022-10-19 13:10:24 -07:00
Justine Tunney
69bee64a59
Make some last minute production changes 2022-10-19 10:00:29 -07:00
Justine Tunney
f7ff77d865
Make fixes and improvements
- Invent iso8601us() for faster timestamps
- Improve --strace descriptions of sigset_t
- Rebuild the Landlock Make bootstrap binary
- Introduce MODE=sysv for non-Windows builds
- Permit OFD fcntl() locks under pledge(flock)
- redbean can now protect your kernel from ddos
- Have vfork() fallback to sys_fork() not fork()
- Change kmalloc() to not die when out of memory
- Improve documentation for some termios functions
- Rewrite putenv() and friends to conform to POSIX
- Fix linenoise + strace verbosity issue on Windows
- Fix regressions in our ability to show backtraces
- Change redbean SetHeader() to no-op if value is nil
- Improve fcntl() so SQLite locks work in non-WAL mode
- Remove some unnecessary work during fork() on Windows
- Create redbean-based SSL reverse proxy for IPv4 TurfWar
- Fix ape/apeinstall.sh warning when using non-bash shells
- Add ProgramTrustedIp(), and IsTrustedIp() APIs to redbean
- Support $PWD, $UID, $GID, and $EUID in command interpreter
- Introduce experimental JTqFpD APE prefix for non-Windows builds
- Invent blackhole daemon for firewalling IP addresses via UNIX named socket
- Add ProgramTokenBucket(), AcquireToken(), and CountTokens() APIs to redbean
2022-10-19 07:19:19 -07:00
Justine Tunney
f52f65b2e3
Make system() and popen() thread safe 2022-10-13 15:54:05 -07:00
Justine Tunney
997ce29ddc
Elevate Windows production worthiness
- SQLite file locking now works on Windows
- SQLite will now use fdatasync() on non-Apple platforms
- Fix Ctrl-C handler on Windows to not crash with TLS
- Signals now work in multithreaded apps on Windows
- fcntl() will now accurately report EINVAL errors
- fcntl() now has excellent --strace logging
- Token bucket replenish now go 100x faster
- *NSYNC cancellations now work on Windows
- Support closefrom() on NetBSD
2022-10-13 13:44:41 -07:00
Justine Tunney
b41f91c658
Greatly expand system() shell code features
The cosmopolitan command interpreter now has 13 builtin commands,
variable support, support for ; / && / || syntax, asynchronous support,
and plenty of unit tests with bug fixes.

This change fixes a bug in posix_spawn() with null envp arg. strace
logging now uses atomic writes for scatter functions. Breaking change
renaming GetCpuCount() to _getcpucount(). TurfWar is now updated to use
the new token bucket algorithm. WIN32 affinity masks now inherit across
fork() and execve().
2022-10-11 21:30:31 -07:00
Justine Tunney
e7329b7cba
Implement atomic swar token bucket library 2022-10-11 21:22:04 -07:00
tkchia
7a06760e6f
Rename LINUX to _HOSTLINUX etc. to reduce clashes (#655)
Co-authored-by: tkchia <tkchia-cosmo@gmx.com>
2022-10-10 20:31:25 -07:00
Malcolm Seyd
84b9b8ed87
Fix ParseHttpMessage failing to store >2 repeatable headers (#657) 2022-10-10 20:17:17 -07:00
Justine Tunney
e557058ac8
Improve cosmo's conformance to libc-test
This change addresses various open source compatibility issues, so that
we pass 313/411 of the tests in https://github.com/jart/libc-test where
earlier today we were passing about 30/411 of them, due to header toil.
Please note that Glibc only passes 341/411 so 313 today is pretty good!

- Make the conformance of libc/isystem/ headers nearly perfect
- Import more of the remaining math library routines from Musl
- Fix inconsistencies with type signatures of calls like umask
- Write tests for getpriority/setpriority which work great now
- conform to `struct sockaddr *` on remaining socket functions
- Import a bunch of uninteresting stdlib functions e.g. rand48
- Introduce readdir_r, scandir, pthread_kill, sigsetjmp, etc..

Follow the instructions in our `tool/scripts/cosmocc` toolchain to run
these tests yourself. You use `make CC=cosmocc` on the test repository
2022-10-10 17:52:41 -07:00
Justine Tunney
d5910e2673
Fix bugs and make code tinier
- Fixed bug where stdio eof wasn't being sticky
- Fixed bug where fseeko() wasn't clearing eof state
- Removed assert() usage from libc favoring _unassert() / _npassert()
2022-10-09 23:21:34 -07:00
Justine Tunney
4a6fd3d910
Make more improvements to threading support
- fix rare thread exit race condition on openbsd
- pthread_getattr_np() now supplies detached status
- child threads may now pthread_join() the main thread
- introduce sigandset(), sigorset(), and sigisemptyset()
- introduce pthread_cleanup_push() and pthread_cleanup_pop()
2022-10-09 00:08:47 -07:00
Justine Tunney
38e3ab57a6
Add the TurfWar schema 2022-10-08 03:23:21 -07:00
Justine Tunney
29d695f4c9
Make unbing() not need initialization 2022-10-07 14:56:02 -07:00
Justine Tunney
05197afca2
Do some work on TurfWar 2022-10-07 03:11:07 -07:00
Justine Tunney
a5b483f2d4
Make some quick fixes 2022-10-06 06:36:46 -07:00
Justine Tunney
b75a4654cf
Introduce clock_nanosleep() 2022-10-05 06:37:15 -07:00
Justine Tunney
fe3216e961
Perform some code cleanup 2022-10-04 23:32:16 -07:00
Justine Tunney
174d288e66
Tune claim SQL query 2022-10-04 10:22:08 -07:00
Justine Tunney
9f9b240f3d
Make HTTP message parsing a little faster
ParseHttpMessage() now does a better job avoiding malloc() calls and
headers commonly sent by CloudFlare can now be consulted in constant
time using our hard-coded perfect hash table.  This increases /claim
performance from 321k to 337k qps which is 5% faster.
2022-10-04 06:37:46 -07:00
Justine Tunney
726f04e8aa
Further enhance IPv4 TurfWar server
- We now vary `/claim` responses based on the `Accept` header
- Remove xasprintf() call to boost `/claim` perf to 321k qps!
2022-10-04 05:27:36 -07:00
Justine Tunney
3ee3736a01
Put finishing touches on turfwar http server
- Shutdown process now has optimal cancellation latency
- Fairer techniques for shedding connections under load
- We no longer need to call poll() which is now removed
2022-10-03 21:50:00 -07:00
Justine Tunney
01bd7d1008
Update a few remaining zlib source files 2022-10-03 19:24:02 -07:00
Justine Tunney
32321ab1e9
Further improve ipv4.games server 2022-10-03 16:00:04 -07:00
Justine Tunney
6ba3b448f3
Fix file descriptor leak in turfwar.c 2022-10-03 08:34:04 -07:00
Justine Tunney
91e167caaf
Rewrite IPv4 Turf War backend
It can now handle 240k SQLite write QPS at 3ms 99 percentile latency.
We're still working out the kinks since it's brand new. But we've got
this running in production already!
2022-10-03 08:23:54 -07:00
Justine Tunney
2cc1d5ac4c
Improve redbean method=get parameter handling 2022-09-19 20:04:39 -07:00
Justine Tunney
775944a2d0
Fix important bugs in redbean
This change upgrades to the latest Chromium Zlib, fixes bugs in redbean,
and introduces better support for reverse proxies like Cloudflare. This
change improves the security of redbean and it's recommended that users
upgrade to the release that'll follow. This change also updates the docs
to clarify how to use the security tools redbean provides e.g. pledge(),
unveil(), and the MODE=asan builds which improve memory safety.
2022-09-17 02:18:06 -07:00