Commit graph

285 commits

Author SHA1 Message Date
Justine Tunney
db0d8dd806 Support Linux binfmt_misc and APE loading on Apple
The "no modify self" variant of Actually Portable Executable is now
supported on all platforms. If you use `$(APE_NO_MODIFY_SELF)` then
ld.bfd will embed a 4096 byte ELF binary and a 4096 byte Macho file
which are installed on the fly to ${TMPDIR:-/tmp}, which enables us
launch the executable, without needing to copy the whole executable

To prevent it from copying a tiny executable to your temp directory
you need to install the `ape` command (renamed from ape-loader), to
a system path. For example:

    # FreeBSD / NetBSD / OpenBSD
    make -j8 o//ape/ape
    cp o//ape/ape /usr/bin/ape

    # Mac OS
    # make -j8 o//ape/ape.macho
    curl https://justine.lol/ape.macho >/usr/bin/ape
    chmod +x /usr/bin/ape

On Linux you can get even more performance with the new binfmt_misc
support which makes launching non-modifying APE binaries as fast as
launching ELF executables. Running the following command:

    # Linux
    ape/apeinstall.sh

Will copy APE loader to /usr/bin/ape and register with binfmt_misc
Lastly, this change also fixes a really interesting race condition
with OpenBSD thread joining.
2022-05-21 09:28:25 -07:00
Justine Tunney
96781d0679 Fix mistake with previous commit 2022-05-20 15:08:02 -07:00
Justine Tunney
f4fc16754a Fix build invalidation issue with GNU Make
The `HDRS` and/or `INCS` variables need to be defined in order for
`mkdeps.com` to compute build dependencies for `o//depend`. We're using
INCS here since makeint.h is the evil kind of header that isn't actually
defining a library interface but rather is a hodgepodge of common code.
2022-05-20 14:20:54 -07:00
Gautham
fb4382e9a4
os.realname variable for correct USER_SITE value (#410)
In site.py, Python uses os.name to decide where the USER_SITE (ie the
folder containing the user's locally installed packages) is located.
With cosmo we have set os.name as "posix" always, so we use a new
os.realname to decide the USER_SITE location.
2022-05-20 08:17:10 -07:00
Justine Tunney
ec2cb88058 Make fixes and improvements
- Document more compiler flags
- Expose new __print_maps() api
- Better overflow checking in mmap()
- Improve the shell example somewhat
- Fix minor runtime bugs regarding stacks
- Make kill() on fork()+execve()'d children work
- Support CLONE_CHILD_CLEARTID for proper joining
- Fix recent possible deadlock regression with --ftrace
2022-05-19 16:57:49 -07:00
Justine Tunney
6e52cba37a Fix stdio regression
This change fixes a nasty regression caused by
80b211e314 which deadlocked.

This change also causes MbedTLS to prefer the ChaCha ciphersuite on
older CPUs that don't have AES hardware instructions.
2022-05-19 00:51:15 -07:00
Justine Tunney
9208c83f7a Make some systemic improvements
- add vdso dump utility
- tests now log stack usage
- rename g_ftrace to __ftrace
- make internal spinlocks go faster
- add conformant c11 atomics library
- function tracing now logs stack usage
- make function call tracing thread safe
- add -X unsecure (no ssl) mode to redbean
- munmap() has more consistent behavior now
- pacify fsync() calls on python unit tests
- make --strace flag work better in redbean
- start minimizing and documenting compiler flags
2022-05-18 16:52:36 -07:00
Justine Tunney
b2c6ec6eab Make linenoise properly recover from backgrounding 2022-05-17 12:10:22 -07:00
Justine Tunney
2743f3d012 Clean up the clone() code a bit 2022-05-17 07:40:00 -07:00
Justine Tunney
ce71677156 Improve threading support further 2022-05-17 04:14:28 -07:00
Justine Tunney
55de4ca6b5 Support thread local storage 2022-05-16 13:20:08 -07:00
Justine Tunney
54e6f564c1 Let ctrl-c interrupt lua server pages in repl mode 2022-05-14 11:47:16 -07:00
Justine Tunney
80b211e314 Add raw memory visualization tool to redbean
This change introduces a `-W /dev/pts/1` flag to redbean. What it does
is use the mincore() system call to create a dual-screen terminal
display that lets you troubleshoot the virtual address space. This is
useful since page faults are an important thing to consider when using a
forking web server. Now we have a colorful visualization of which pages
are going to fault and which ones are resident in memory.

The memory monitor, if enabled, spawns as a thread that just outputs
ANSI codes to the second terminal in a loop. In order to make this
happen using the new clone() polyfill, stdio is now thread safe.

This change also introduces some new demo pages to redbean. It also
polishes the demos we already have, to look a bit nicer and more
presentable for the upcoming release, with better explanations too.
2022-05-14 04:33:58 -07:00
Justine Tunney
578cb21591 Add syscalls to Blinkenlights and fix bugs 2022-05-13 13:31:21 -07:00
Gautham
f6df29cc3d
Add Python JSON tests (#407) 2022-05-13 13:02:15 -07:00
Justine Tunney
d25a67f4eb Fix cross-device link in Python tests 2022-05-13 06:50:55 -07:00
Justine Tunney
77b70ba193 Add fixes to previous commit 2022-05-13 05:05:27 -07:00
Gautham
83b743cf96
Python 3.7 METH_FASTCALL backport (#406) 2022-05-13 05:05:12 -07:00
Justine Tunney
4e62cefa6e Add zlib gzip functions
These will now be included in the cosmopolitan.a releases. It took a bit
of time because, these functions depend on heavyweight parts of the libc
that wouldn't be appropriate for the core zlib library to depend upon.

Fixes #345
2022-05-12 12:17:40 -07:00
Justine Tunney
4499f98e76 Add /.args feature to Redbean/Lua/SQLite/Python/QuickJS
You now have some ability to truly make an executable yours, by adding a
`.args` file to the root of the zip structure. If this is specified,
then you'll be overriding the default CLI args.

This will be a great feature for folks who want to distribute their own
apps, using the interpreter executable, but have the executable appears
to be just your app rather than being the interpreter.
2022-05-12 11:04:47 -07:00
Justine Tunney
2ea1dc405c Revert "Backport METH_FASTCALL from Python 3.7 (#328)"
This reverts commit cf73bbd678.
2022-05-12 06:49:54 -07:00
Justine Tunney
e7611a8476 Make improvements
- Get threads working on NetBSD
- Get threads working on OpenBSD
- Fix Emacs config for Emacs v28
- Improve --strace logging of sigset_t
- Improve --strace logging of struct stat
- Improve memory safety of DescribeThing functions
- Refactor auto stack allocation into LIBC_RUNTIME
- Introduce shell.com example which works on Windows
- Refactor __strace_thing into DescribeThing functions
- Document the CHECK macros and improve them in NDEBUG mode
- Rewrite MAP_STACK so it uses FreeBSD behavior across platforms
- Deprecate and discourage the use of MAP_GROWSDOWN (it's weird)
2022-05-12 06:45:36 -07:00
Gautham
cf73bbd678
Backport METH_FASTCALL from Python 3.7 (#328) 2022-05-12 02:27:16 -07:00
Gautham
363d2ec436
Load Python C extensions of external packages correctly (#391) 2022-05-08 21:49:50 -07:00
Justine Tunney
15c59e716f Fix build break in MODE=asan 2022-05-05 11:12:48 -07:00
Justine Tunney
c9f966d73c Fix build suboptimality
Non-default build modes were having artifacts invalidated needlessly.
2022-05-04 22:51:51 -07:00
Justine Tunney
e4b559c76a Simplify the redbean JSON encoder 2022-04-29 06:35:27 -07:00
Justine Tunney
2d1731b995 Polish redbean serialization 2022-04-29 06:10:10 -07:00
Justine Tunney
7aafa64ab3 Make improvements
- Bump redbean up to 2.0
- Trim down the MODE=tiny build a bit
- Add Indian Standard Time to zoneinfo
2022-04-29 00:42:47 -07:00
Justine Tunney
c9a981fdbe Fix some more reported issues 2022-04-28 20:36:33 -07:00
Justine Tunney
47b3274665 Make improvements
- Add rusage to redbean Lua API
- Add more redbean documentation
- Add pledge() to redbean Lua API
- Polyfill OpenBSD pledge() for Linux
- Increase PATH_MAX limit to 1024 characters
- Untrack sibling processes after fork() on Windows
2022-04-28 09:57:07 -07:00
Justine Tunney
cc0d1ec076 Fix some bugs
- addr2line backtrace should continue on eintr
- lua crashes if we try to iterate a non-table
2022-04-27 20:18:34 -07:00
Justine Tunney
6a145a9262 Make improvements
- Add hierarchical auto-completion to redbean's repl
- Fetch latest localtime() and strftime() from Eggert
- Shave a few milliseconds off redbean start latency
- Fix redbean repl with multi-line statements
- Make the Lua unix module code more elegant
- Harden Lua data structure serialization
2022-04-27 05:39:39 -07:00
Justine Tunney
d57b81aac7 Make improvements
- Add GetCpuCount() API to redbean
- Add unix.gmtime() API to redbean
- Add unix.readlink() API to redbean
- Add unix.localtime() API to redbean
- Perfect the new redbean UNIX module APIs
- Integrate with Linux clock_gettime() vDSO
- Run Lua garbage collector when malloc() fails
- Fix another regression quirk with linenoise repl
- Fix GetProgramExecutableName() for systemwide installs
- Fix a build flake with test/libc/mem/test.mk SRCS list
2022-04-26 16:46:15 -07:00
Paul Kulchenko
e6fab847a0
Update redbean to stringify error objects (#393) 2022-04-25 21:14:36 -07:00
Justine Tunney
72e9be5c20 Update redbean lua example code 2022-04-25 09:31:28 -07:00
Justine Tunney
451e3f73d9 Improve redbean
- Improve serialization
- Add Benchmark() API to redbean
- Refactor UNIX API to be assert() friendly
- Make the redbean Lua REPL print data structures
- Fix recent regressions in linenoise reverse search
- Add -i flag so redbean can be a language interpreter
2022-04-25 08:30:14 -07:00
Justine Tunney
2046c0d2ae Make improvements
- Expand redbean UNIX module
- Expand redbean documentation
- Ensure Lua copyright is embedded in binary
- Increase the PATH_MAX limit especially on NT
- Use column major sorting for linenoise completions
- Fix some suboptimalities in redbean's new UNIX API
- Figured out right flags for Multics newline in raw mode
2022-04-24 10:06:05 -07:00
Justine Tunney
2f56ebfe78 Do code cleanup use duff device linenoise i/o 2022-04-22 18:56:52 -07:00
Justine Tunney
6ff46ca373 Make it possible to compile redbean with chibicc
This cuts build latency down from 5 seconds to 500 milliseconds.
2022-04-22 15:25:04 -07:00
Justine Tunney
c39d6111f2 Improve redbean ctrl-c handling 2022-04-21 15:24:44 -07:00
Justine Tunney
1599b818d9 Make more fixes and improvements 2022-04-21 13:44:59 -07:00
Justine Tunney
9d61e23c80 Make more fixes and improvements
This change attempts to fix some report build issues. It also builds
upon development work described in previous changes.
2022-04-21 09:18:45 -07:00
Justine Tunney
87396f43bc Flatten InfoZIP directory and fix build issues 2022-04-20 22:40:33 -07:00
Justine Tunney
ae638c0850 Fix bugs and make improvements
- Get clone() working on FreeBSD
- Increase some Python build quotas
- Add more atomic builtins to chibicc
- Fix ASAN poisoning of alloca() memory
- Make MODE= mandatory link path tinier
- Improve the examples folder a little bit
- Start working on some more resource limits
- Make the linenoise auto-complete UI as good as GNU readline
- Update compile.com, avoiding AVX codegen on non-AVX systems
- Make sure empty path to syscalls like opendir raises ENOENT
- Correctly polyfill ENOENT vs. ENOTDIR on the New Technology
- Port bestline's paredit features to //third_party/linenoise
- Remove workarounds for RHEL 5.0 bugs that were fixed in 5.1
2022-04-20 10:05:34 -07:00
Justine Tunney
5a132f9652 Add seccomp bpf sandboxing to redbean
It's now possible to pass the `-S` or `-SS` flags to sandbox redbean
worker proecsses after they've been forked. The first `-S` flag is
intended to be a permissive builtin policy that limits system calls to
only that which the various parts of redbean serving need. The second
`-SS` flag is intended to be more restrictive, preventing things like
the Lua extensions you download off the web from using the HTTP client
or sockets APIs. In upcoming changes you'll be able to implement your
own Berkeley Packet Filter sandbox programs and load them via Lua.
2022-04-18 08:54:42 -07:00
Justine Tunney
7166679620 Fix bugs and add security features to redbean
- Fix a regression with the previous change that broke redbean
- Add chroot(), resource limit, seccomp, and other stuff to redbean
- Write lots and lots of documentation
- Iron out more system call issues
2022-04-18 00:01:26 -07:00
Justine Tunney
f1dfa4bdfa Add some more builtins to chibicc
https://justine.lol/cosmopolitan/documentation.html should now contain a
lot of functions that had been missing previously due to not having them
2022-04-17 13:07:07 -07:00
Justine Tunney
a6b02ce5a6 Add lua repl interface to redbean
You can now interact with the global web server state on the command
line, which the web server is running. This supports Emacs shortcuts
with history, readline parity, <tab> completions, plus hints. Enjoy!
2022-04-16 20:31:16 -07:00
Justine Tunney
f6b6204b9e Add readline-style completions to Lua/Python/Almquist 2022-04-16 15:10:05 -07:00