Find a file
Paul Kulchenko db16f0129a
Redbean sqlite fix multi close (#749)
* Update redbean SQLite config to handle more options

This requires moving sqlite3_initialize call to open, as configuration
should be allowed before initialization is done. This call is effective
only for the first time and then no-op after that.

* Fix redbean SQLite for closing db with already finalized statements

There is a loop in cleanupdb that finalizes all vms that are associated
with that db when it's being closed. Under some circumstances (detailed
below) that loop may contain references pointing to already collected
objects, thus leading to SIGSEGV when those references are used.

This may happen with the following sequence of events ("VM" is the name
used in lsqlite and describes the same thing as "statement"):
1. A finalized statement is created (for example, by preparing an empty
string or a string with no statement that is still grammatically valid).
2. This statement goes out of scope before the DB object it's associated
with does and is garbage collected.
3. When it's garbage collected, dbvm_gc method is called, which checks
for svm->vm being not NULL.
4. Since the VM is already finalized, cleanupvm method is not called,
so the VM reference is not removed from the table of VMs tracked for
that DB.
5. When the DB is finally closed or garbage collected, all the VMs
associated with it are accessed to be finalized, including the ones that
have been garbage collected and have invalid references (thus leading
to a memory access error).

Here is an example of a stacktrace from the resulting SIGSEGV:

70000003de20 5df71a getgeneric+26
70000003fac0 5dfc7f luaH_get+111
70000003faf0 5e06c8 luaH_set+40
70000003fb20 5c5bd7 aux_rawset+55
70000003fb50 5c70cb lua_rawset+27
70000003fb60 4fa8e7 cleanupvm+71
70000003fb80 4fa988 cleanupdb+88
70000003fbc0 4fe899 db_gc+41

One way to fix this is to use userdata references (which anchor their
targets) instead of lightuserdata references (which do not), but this
would prevent the targets (VMs) from being garbage collected until the
DB itself is garbage collected, so this needs to be combined with
weakening the keys in the DB table. The code in cleanupdb to remove the
VM references is no longer needed, as this is handled by having weak keys.

The patch also switches to using close_v2, as it is intended for use
with garbage collected languages where the order in which destructors
are called is arbitrary, as is the case here.

* Remove GC collection from redbean SQLite session

The behavior of sqlite3session_delete is undefined after the DB
connection is closed, so we need to avoid calling it from gc handler.
2023-02-23 20:10:03 -08:00
.github Tune GitHub Actions to use APE loader 2022-10-02 10:20:08 -07:00
.vscode Remove testonly keyword 2022-09-05 08:41:43 -07:00
ape Upgrade embedded blink to 0.9.2 2023-02-23 12:25:01 -08:00
build Make mkdeps more helpful when files are missing 2023-02-23 08:11:08 -08:00
dsp Get repository to build with GCC 11 2022-09-13 04:14:55 -07:00
examples Fix sysinfo() 2023-02-23 06:54:58 -08:00
libc Add some POSIX feature definitions 2023-02-23 08:09:34 -08:00
net doc: remove "everything everything" from ParseUrl doc (#704) 2023-02-23 09:34:48 -08:00
test Make POSIX semaphores always process shared 2023-02-23 08:07:54 -08:00
third_party Disable Python test_sys 2023-02-23 08:56:31 -08:00
tool Redbean sqlite fix multi close (#749) 2023-02-23 20:10:03 -08:00
usr/share Make improvements 2022-04-29 00:42:47 -07:00
.clang-format Add x86_64-linux-gnu emulator 2020-08-25 04:43:42 -07:00
.gitattributes Perform build and magnum tuning 2021-08-10 10:26:13 -07:00
.gitignore Reduce GNU Make latency 17% for cosmo 2022-08-19 15:29:24 -07:00
CONTRIBUTING.md Add CONTRIBUTING.md 2021-04-11 00:55:53 -07:00
LICENSE Add title to the LICENSE file 2021-02-27 13:25:59 -08:00
Makefile Serialize ZipOS handle IO across threads 2023-02-23 06:55:10 -08:00
README.md Add note about APE loader and noexec mounts (#707) 2022-12-21 23:00:51 -08:00

Cosmopolitan Honeybadger

build

Cosmopolitan

Cosmopolitan Libc makes C a build-once run-anywhere language, like Java, except it doesn't need an interpreter or virtual machine. Instead, it reconfigures stock GCC and Clang to output a POSIX-approved polyglot format that runs natively on Linux + Mac + Windows + FreeBSD + OpenBSD + NetBSD + BIOS with the best possible performance and the tiniest footprint imaginable.

Background

For an introduction to this project, please read the αcτµαlly pδrταblε εxεcµταblε blog post and cosmopolitan libc website. We also have API documentation.

Getting Started

If you're doing your development work on Linux or BSD then you need just five files to get started. Here's what you do on Linux:

wget https://justine.lol/cosmopolitan/cosmopolitan-amalgamation-2.2.zip
unzip cosmopolitan-amalgamation-2.2.zip
printf 'main() { printf("hello world\\n"); }\n' >hello.c
gcc -g -Os -static -nostdlib -nostdinc -fno-pie -no-pie -mno-red-zone \
  -fno-omit-frame-pointer -pg -mnop-mcount -mno-tls-direct-seg-refs -gdwarf-4 \
  -o hello.com.dbg hello.c -fuse-ld=bfd -Wl,-T,ape.lds -Wl,--gc-sections \
  -include cosmopolitan.h crt.o ape-no-modify-self.o cosmopolitan.a
objcopy -S -O binary hello.com.dbg hello.com

You now have a portable program.

./hello.com
bash -c './hello.com'  # zsh/fish workaround (we patched them in 2021)

If ./hello.com executed on Linux throws an error about not finding an interpreter, it should be fixed by running the following command (although note that it may not survive a system restart):

sudo sh -c "echo ':APE:M::MZqFpD::/bin/sh:' >/proc/sys/fs/binfmt_misc/register"

If the same command produces puzzling errors on WSL or WINE when using Redbean 2.x, they may be fixed by disabling binfmt_misc:

sudo sh -c 'echo -1 >/proc/sys/fs/binfmt_misc/status'

Since we used the ape-no-modify-self.o bootloader (rather than ape.o) your executable will not modify itself when it's run. What it'll instead do, is extract a 4kb program (the APE loader) to ${TMPDIR:-${HOME:-.}} that maps your program into memory without needing to copy it. The APE loader must be in an executable location (e.g. not stored on a noexec mount) for it to run. See below for alternatives:

It's possible to install the APE loader systemwide as follows.

# System-Wide APE Install
# for Linux, Darwin, and BSDs
# 1. Copies APE Loader to /usr/bin/ape
# 2. Registers w/ binfmt_misc too if Linux
ape/apeinstall.sh

# System-Wide APE Uninstall
# for Linux, Darwin, and BSDs
ape/apeuninstall.sh

It's also possible to convert APE binaries into the system-local format by using the --assimilate flag. Please note that if binfmt_misc is in play, you'll need to unregister it temporarily before doing this, since the assimilate feature is part of the shell script header.

$ file hello.com
hello.com: DOS/MBR boot sector
./hello.com --assimilate
$ file hello.com
hello.com: ELF 64-bit LSB executable

Now that you're up and running with Cosmopolitan Libc and APE, here's some of the most important troubleshooting tools APE offers that you should know, in case you encounter any issues:

./hello.com --strace   # log system calls to stderr
./hello.com --ftrace   # log function calls to stderr

Do you love tiny binaries? If so, you may not be happy with Cosmo adding heavyweight features like tracing to your binaries by default. In that case, you may want to consider using our build system:

make m=tiny

Which will cause programs such as hello.com and life.com to shrink from 60kb in size to about 16kb. There's also a prebuilt amalgamation online https://justine.lol/cosmopolitan/cosmopolitan-tiny.zip hosted on our download page https://justine.lol/cosmopolitan/download.html.

MacOS

If you're developing on MacOS you can install the GNU compiler collection for x86_64-elf via homebrew:

brew install x86_64-elf-gcc

Then in the above scripts just replace gcc and objcopy with x86_64-elf-gcc and x86_64-elf-objcopy to compile your APE binary.

Windows

If you're developing on Windows then you need to download an x86_64-pc-linux-gnu toolchain beforehand. See the Compiling on Windows tutorial. It's needed because the ELF object format is what makes universal binaries possible.

Cosmopolitan officially only builds on Linux. However, one highly experimental (and currently broken) thing you could try, is building the entire cosmo repository from source using the cross9 toolchain.

mkdir -p o/third_party
rm -rf o/third_party/gcc
wget https://justine.lol/linux-compiler-on-windows/cross9.zip
unzip cross9.zip
mv cross9 o/third_party/gcc
build/bootstrap/make.com

Source Builds

Cosmopolitan can be compiled from source on any Linux distro. First, you need to download or clone the repository.

wget https://justine.lol/cosmopolitan/cosmopolitan.tar.gz
tar xf cosmopolitan.tar.gz  # see releases page
cd cosmopolitan

This will build the entire repository and run all the tests:

build/bootstrap/make.com
o//examples/hello.com
find o -name \*.com | xargs ls -rShal | less

If you get an error running make.com then it's probably because you have WINE installed to binfmt_misc. You can fix that by installing the the APE loader as an interpreter. It'll improve build performance too!

ape/apeinstall.sh

Since the Cosmopolitan repository is very large, you might only want to build a particular thing. Cosmopolitan's build config does a good job at having minimal deterministic builds. For example, if you wanted to build only hello.com then you could do that as follows:

build/bootstrap/make.com o//examples/hello.com

Sometimes it's desirable to build a subset of targets, without having to list out each individual one. You can do that by asking make to build a directory name. For example, if you wanted to build only the targets and subtargets of the chibicc package including its tests, you would say:

build/bootstrap/make.com o//third_party/chibicc
o//third_party/chibicc/chibicc.com --help

Cosmopolitan provides a variety of build modes. For example, if you want really tiny binaries (as small as 12kb in size) then you'd say:

build/bootstrap/make.com m=tiny

Here's some other build modes you can try:

build/bootstrap/make.com m=dbg       # asan + ubsan + debug
build/bootstrap/make.com m=asan      # production memory safety
build/bootstrap/make.com m=opt       # -march=native optimizations
build/bootstrap/make.com m=rel       # traditional release binaries
build/bootstrap/make.com m=optlinux  # optimal linux-only performance
build/bootstrap/make.com m=fastbuild # build 28% faster w/o debugging
build/bootstrap/make.com m=tinylinux # tiniest linux-only 4kb binaries

For further details, see //build/config.mk.

GDB

Here's the recommended ~/.gdbinit config:

set host-charset UTF-8
set target-charset UTF-8
set target-wide-charset UTF-8
set osabi none
set complaints 0
set confirm off
set history save on
set history filename ~/.gdb_history
define asm
  layout asm
  layout reg
end
define src
  layout src
  layout reg
end
src

You normally run the .com.dbg file under gdb. If you need to debug the .com file itself, then you can load the debug symbols independently as

gdb foo.com -ex 'add-symbol-file foo.com.dbg 0x401000'

Support Vector

Platform Min Version Circa
AMD K8 Venus 2005
Intel Core 2006
Linux 2.6.18 2007
Windows 8 [1] 2012
Mac OS X 15.6 2018
OpenBSD 6.4 2018
FreeBSD 13 2020
NetBSD 9.2 2021

[1] See our vista branch for a community supported version of Cosmopolitan that works on Windows Vista and Windows 7.

Special Thanks

Funding for this project is crowdsourced using GitHub Sponsors and Patreon. Your support is what makes this project possible. Thank you! We'd also like to give special thanks to the following groups and individuals:

For publicly sponsoring our work at the highest tier.