This change fixes Cosmopolitan so it has fewer opinions about compiler
warnings. The whole repository had to be cleaned up to be buildable in
-Werror -Wall mode. This lets us benefit from things like strict const
checking. Some actual bugs might have been caught too.
* Update redbean SQLite config to handle more options
This requires moving sqlite3_initialize call to open, as configuration
should be allowed before initialization is done. This call is effective
only for the first time and then no-op after that.
* Fix redbean SQLite for closing db with already finalized statements
There is a loop in cleanupdb that finalizes all vms that are associated
with that db when it's being closed. Under some circumstances (detailed
below) that loop may contain references pointing to already collected
objects, thus leading to SIGSEGV when those references are used.
This may happen with the following sequence of events ("VM" is the name
used in lsqlite and describes the same thing as "statement"):
1. A finalized statement is created (for example, by preparing an empty
string or a string with no statement that is still grammatically valid).
2. This statement goes out of scope before the DB object it's associated
with does and is garbage collected.
3. When it's garbage collected, dbvm_gc method is called, which checks
for svm->vm being not NULL.
4. Since the VM is already finalized, cleanupvm method is not called,
so the VM reference is not removed from the table of VMs tracked for
that DB.
5. When the DB is finally closed or garbage collected, all the VMs
associated with it are accessed to be finalized, including the ones that
have been garbage collected and have invalid references (thus leading
to a memory access error).
Here is an example of a stacktrace from the resulting SIGSEGV:
70000003de20 5df71a getgeneric+26
70000003fac0 5dfc7f luaH_get+111
70000003faf0 5e06c8 luaH_set+40
70000003fb20 5c5bd7 aux_rawset+55
70000003fb50 5c70cb lua_rawset+27
70000003fb60 4fa8e7 cleanupvm+71
70000003fb80 4fa988 cleanupdb+88
70000003fbc0 4fe899 db_gc+41
One way to fix this is to use userdata references (which anchor their
targets) instead of lightuserdata references (which do not), but this
would prevent the targets (VMs) from being garbage collected until the
DB itself is garbage collected, so this needs to be combined with
weakening the keys in the DB table. The code in cleanupdb to remove the
VM references is no longer needed, as this is handled by having weak keys.
The patch also switches to using close_v2, as it is intended for use
with garbage collected languages where the order in which destructors
are called is arbitrary, as is the case here.
* Remove GC collection from redbean SQLite session
The behavior of sqlite3session_delete is undefined after the DB
connection is closed, so we need to avoid calling it from gc handler.
* Proof of concept of sqlite serialization
This is a minimal proof of concept in order to show that it is easily possible to store the sqlite database within the zip file itself not requiring creating an external file first. Changes include compiling the sqlite library with the serialization flag, adding serialize/deserialize to the lua sqlite library and demonstrating the work via the redbean demo.
* Change demo for sqlite serialization
As explained in https://github.com/jart/cosmopolitan/pull/436#issuecomment-1164706893 the original use case is not possible with sqlite serialization, as an in-memory database cannot be shared across multiple processes. Thereby, this use case simply creates a backup of the in-memory database created in '.init.lua' and loads it to do a query.
* Fix sqlite3_deserialize parameters
The call to the sqlite3 library for the deserilization wasn't fully correct. This should fix the size parameters.
- 10.5% reduction of o//depend dependency graph
- 8.8% reduction in latency of make command
- Fix issue with temporary file cleanup
There's a new -w option in compile.com that turns off the recent
Landlock output path workaround for "good commands" which do not
unlink() the output file like GNU tooling does.
Our new GNU Make unveil sandboxing appears to have zero overhead
in the grand scheme of things. Full builds are pretty fast since
the only thing that's actually slowed us down is probably libcxx
make -j16 MODE=rel
RL: took 85,732,063µs wall time
RL: ballooned to 323,612kb in size
RL: needed 828,560,521µs cpu (11% kernel)
RL: caused 39,080,670 page faults (99% memcpy)
RL: 350,073 context switches (72% consensual)
RL: performed 0 reads and 11,494,960 write i/o operations
pledge() and unveil() no longer consider ENOSYS to be an error.
These functions have also been added to Python's cosmo module.
This change also removes some WIN32 APIs and System Five magnums
which we're not using and it's doubtful anyone else would be too
- Improve serialization
- Add Benchmark() API to redbean
- Refactor UNIX API to be assert() friendly
- Make the redbean Lua REPL print data structures
- Fix recent regressions in linenoise reverse search
- Add -i flag so redbean can be a language interpreter
redbean lua handlers that perform sql queries can do 400k qps.
We now use a separate compile-time options for SQLite, when building the
SQLite shell versus building the production web serving code. It doesn't
seem appropriate for something like redbean to include backups, progress
callbacks, query completion, profiling, EXPLAIN, ALTER, ANALYZE, VACUUM,
etc. since those tasks are better left to the sqlite3.com shell program.
Lua SQLite pointer APIs have been removed since we're not using threads.
The Lua APIs for installing update / commit / rollback hooks are removed
due to a general sense of disagreement and an overall lack of comfort.
Full-Text Search and R*Tree are as large as the rest of SQLite combined.
Turning those off keeps redbean under 1mb when built for MODE=tiny which
is nice for marketing purposes.
If you need something that was removed, file an issue, and we'll add it.
