Commit graph

1125 commits

Author SHA1 Message Date
Justine Tunney
58fb2fb3d3 Add chunked transfer decoding to redbean 2021-07-05 01:05:10 -07:00
Justine Tunney
8d5f60a9cd Add more hashing apis to redbean 2021-07-05 01:05:10 -07:00
Justine Tunney
a7bd4ed9ea Make redbean tinier
Under MODE=tiny or MODE=tinylinux we now go back to using my homebrew
version of DEFLATE decompression which is 10x smaller but 10x slower
than Chromium Zlib. In tiny mode we also disable compressed responses
howewver redbean will still serve precompressed responses. This change
also removes a few other dependencies like strftime() and getaddrinfo()
which means you can't say `-l localhost` in tiny mode, you have to say
something like `-l 127.0.0.1`.

    m=tinylinux
    make -j8 MODE=$m o/$m/tool/net/redbean-original.com
    ls -hal o/$m/tool/net/redbean-original.com

This change reduces the above size from 191.4 to 150.9 kb.
2021-07-04 12:26:54 -07:00
Justine Tunney
300876ee50 Fix typo 2021-07-03 10:16:38 -07:00
Justine Tunney
e6b22f1f65 Remove remaining build references to zip command 2021-07-03 08:37:15 -07:00
Justine Tunney
d6873f637f Release redbean 1.4 2021-07-03 05:58:47 -07:00
Justine Tunney
74200a0ea0 Make redbean ssl handshake go a little faster 2021-07-03 05:51:04 -07:00
Justine Tunney
053ee714bd Add Braille Dump tool 2021-07-02 10:24:12 -07:00
Justine Tunney
8e69cf133a Restore basicidea.c from gist 2021-07-02 08:09:56 -07:00
Justine Tunney
014d4bdab2 Make minor revisions to previous change 2021-06-30 10:45:27 -07:00
Justine Tunney
a68cc690ff Merge HTTP request / response parsing code
This change also fixes a bug so that DNS lookups work correctly when the
first answer is a CNAME record.
2021-06-27 17:04:32 -07:00
Justine Tunney
5144c22189 Add test for ioctl(SIOCGIFCONF) and polyfill on BSDs
- Use nullness checks when calling weakly linked functions.

- Avoid typedef for reasons described in Linux Kernel style guide.

- Avoid enum in in Windows headers. Earlier in Cosmo's history all one
  hundred files in libc/nt/enum/ used to be enums and it resulted in
  gigabytes of DWARF data almost as large as everything else in the
  codebase combined.

- Bitfields aren't our friends. They have frequent ABI breakages,
  inconsistent arithmetic across compilers, and different endianness
  between cpus. Compiler authors also haven't invested much roi into
  making bit fields go fast so they produce poor assembly.

- Use memccpy() instead of strncpy() or snprintf() for length-bounded
  copying of C strings. strncpy() is a misunderstood function and
  snprintf() is awesome but memccpy() deserves more love.
2021-06-25 18:44:04 -07:00
Justine Tunney
86ab24ce56 Correct minor issues with recent SSL change 2021-06-24 15:22:24 -07:00
Justine Tunney
cc1920749e Add SSL to redbean
Your redbean can now interoperate with clients that require TLS crypto.
This is accomplished using a protocol polyglot that lets us distinguish
between HTTP and HTTPS regardless of the port number. Certificates will
be generated automatically, if none are supplied by the user. Footprint
increases by only a few hundred kb so redbean in MODY=tiny is now 1.0mb

- Add lseek() polyfills for ZIP executable
- Automatically polyfill /tmp/FOO paths on NT
- Fix readdir() / ftw() / nftw() bugs on Windows
- Introduce -B flag for slower SSL that's stronger
- Remove mbedtls features Cosmopolitan doesn't need
- Have base64 decoder support the uri-safe alternative
- Remove Truncated HMAC because it's forbidden by the IETF
- Add all the mbedtls test suites and make them go 3x faster
- Support opendir() / readdir() / closedir() on ZIP executable
- Use Everest for ECDHE-ECDSA because it's so good it's so good
- Add tinier implementation of sha1 since it's not worth the rom
- Add chi-square monte-carlo mean correlation tests for getrandom()
- Source entropy on Windows from the proper interface everyone uses

We're continuing to outperform NGINX and other servers on raw message
throughput. Using SSL means that instead of 1,000,000 qps you can get
around 300,000 qps. However redbean isn't as fast as NGINX yet at SSL
handshakes, since redbean can do 2,627 per second and NGINX does 4.3k

Right now, the SSL UX story works best if you give your redbean a key
signing key since that can be easily generated by openssl using a one
liner then redbean will do all the things that are impossibly hard to
do like signing ecdsa and rsa certificates that'll work in chrome. We
should integrate the let's encrypt acme protocol in the future.

Live Demo: https://redbean.justine.lol/
Root Cert: https://redbean.justine.lol/redbean1.crt
2021-06-24 13:20:50 -07:00
Justine Tunney
8d7d00af3a Pacify file locks on Windows
This change gets redbean SQLite working in write mode on Windows.
Warnings have been added to the appropriate and responsible places.
Hacking proprietary PC systems into production-worthy servers isn't
terribly high on the list of priorities. Consider BSD or Linux when
building online systems that service requests from multiple people.

Fixes #193
2021-06-12 00:01:55 -07:00
Justine Tunney
a3ccc5af1f Fix redbean hidden path bug 2021-06-10 08:55:50 -07:00
Justine Tunney
f271ffcf90 Release redbean 1.2 2021-06-10 08:14:45 -07:00
Justine Tunney
88806b79b1 Tune SQLite build for redbean (#97)
redbean lua handlers that perform sql queries can do 400k qps.

We now use a separate compile-time options for SQLite, when building the
SQLite shell versus building the production web serving code. It doesn't
seem appropriate for something like redbean to include backups, progress
callbacks, query completion, profiling, EXPLAIN, ALTER, ANALYZE, VACUUM,
etc. since those tasks are better left to the sqlite3.com shell program.

Lua SQLite pointer APIs have been removed since we're not using threads.
The Lua APIs for installing update / commit / rollback hooks are removed
due to a general sense of disagreement and an overall lack of comfort.

Full-Text Search and R*Tree are as large as the rest of SQLite combined.
Turning those off keeps redbean under 1mb when built for MODE=tiny which
is nice for marketing purposes.

If you need something that was removed, file an issue, and we'll add it.
2021-06-10 08:00:08 -07:00
Justine Tunney
eb08b9fbeb Fix redbean inappropriately linking ASAN
Fixes #188
2021-06-10 07:59:50 -07:00
Paul Kulchenko
cd2f984ec2
Fix lsqlite make #187 2021-06-08 14:55:30 -07:00
Paul Kulchenko
b3a08d9286 Add sqlite3 support to Lua scripts in redbean. 2021-06-05 18:38:38 -07:00
Paul Kulchenko
067c14891f Add lsqlite source (v0.9.5). 2021-06-05 18:38:38 -07:00
Justine Tunney
7cbc2bc083 Cosmopolitan 1.0 2021-05-16 20:35:10 -07:00
Justine Tunney
3057315a1b Make default redbean lua module path zip:.lua/...
Fixes #157
2021-05-16 19:05:35 -07:00
Justine Tunney
6a8c21269f Make trivial code size enhancements 2021-05-16 18:22:39 -07:00
Justine Tunney
b3838173ec Remove undefined behaviors 2021-05-16 11:16:28 -07:00
Justine Tunney
4864565198 Make minor improvements 2021-05-15 21:53:26 -07:00
Justine Tunney
690be544da Make redbean StoreAsset() work better
- Better UBSAN error messages
- POSIX Advisory Locks polyfills
- Move redbean manual to /.help.txt
- System call memory safety in ASAN mode
- Character classification now does UNICODE
2021-05-14 05:44:37 -07:00
Justine Tunney
1b5a5719c3 Improve some unicode functions 2021-05-05 07:25:39 -07:00
Justine Tunney
b9187061a7 Let redbean use its own ZIP EXE as NoSQL database 2021-05-03 13:07:58 -07:00
Justine Tunney
e56a9d0e23 Mold the redbean binary to minimize page faults
This change brings page faults for precompressed static asset serving
down from 27 to 20 (or fewer) after fork. This is more of an art than
science. Hopefully Blinkenlights can visualize page faults soon.
2021-05-03 12:14:13 -07:00
Justine Tunney
2d34819779 Escape LaunchBrowser path
This also fix a bug where CTRL-C'ing redbean on Linux would kill the
browser process. It furthermore fixes a regression with the APE self
repair process that happened in a recent change.

See #158
2021-05-03 11:52:24 -07:00
Rowan Easter-Robinson
6215c91e90
Added Parameter to LaunchBrowser to set path (#158) 2021-05-03 10:55:21 -07:00
Justine Tunney
01e6b3ad8d Reduce number of disk seeks in redbean 2021-05-03 01:21:50 -07:00
Justine Tunney
daa32d27d4 Add live reindexing to redbean when zip changes 2021-05-02 11:50:43 -07:00
Justine Tunney
84001a246c Fix redbean date header in daemonize mode 2021-05-02 11:11:26 -07:00
Justine Tunney
1f2288be6e Improve backwards compatibility with GNU Make 2021-05-02 07:48:59 -07:00
Justine Tunney
cef08d47b6 Fix typo in redbean.c 2021-05-01 05:42:02 -07:00
Justine Tunney
41801c9802 Change default redbean log level to info
A new -s flag has been added to increase silence. This change is
intended to make it a little less weird opening the redbean binary
directly from the browser and seeing an empty terminal window.
2021-05-01 05:28:15 -07:00
Justine Tunney
1966369e8e Fix bugs and make improvements
- Fix regression with `%lu`
- Added some more headers witnessed in the wild
- Added `-M INT` option to redbean to tune max payload size
- Work around InfoZIP 256 character limit on comment line size
2021-05-01 05:11:35 -07:00
Justine Tunney
c029e83dd8 Set up http://redbean.justine.lol/ demo server 2021-04-24 19:49:49 -07:00
Justine Tunney
472b95fea3 Support OnHttpRequest Lua callback
If your redbean `/.init.lua` file defines a global callable named
`OnHttpRequest` then redbean will delegate all serving control to
your function. You may then restore the default serving paths, by
calling the new `Route()`, `RouteHost()`, and `RoutePath()` APIs.

Closes #150
2021-04-24 17:27:12 -07:00
Justine Tunney
dc6d11a031 Improve performance of printf functions 2021-04-24 13:58:50 -07:00
Justine Tunney
b107d2709f Add /statusz page to redbean plus other enhancements
redbean improvements:

- Explicitly disable corking
- Simulate Python regex API for Lua
- Send warmup requests in main process on startup
- Add Class-A granular IPv4 network classification
- Add /statusz page so you can monitor your redbean's health
- Fix regressions on OpenBSD/NetBSD caused by recent changes
- Plug Authorization header into Lua GetUser and GetPass APIs
- Recognize X-Forwarded-{For,Host} from local reverse proxies
- Add many additional functions to redbean Lua server page API
- Report resource usage of child processes on `/` listing page
- Introduce `-a` flag for logging child process resource usage
- Introduce `-t MILLIS` flag and `ProgramTimeout(ms)` init API
- Introduce `-H "Header: value"` flag and `ProgramHeader(k,v)` API

Cosmopolitan Libc improvements:

- Make strerror() simpler
- Make inet_pton() not depend on sscanf()
- Fix OpenExecutable() which broke .data section earlier
- Fix stdio in cases where it overflows kernel tty buffer
- Fix bugs in crash reporting w/o .com.dbg binary present
- Add polyfills for SO_LINGER, SO_RCVTIMEO, and SO_SNDTIMEO
- Polyfill TCP_CORK on BSD and XNU using TCP_NOPUSH magnums

New netcat clone in examples/nc.c:

While testing some of the failure conditions for redbean, I noticed that
BusyBox's `nc` command is pretty busted, if you use it as an interactive
tool, rather than having it be part of a pipeline. Unfortunately this'll
only work on UNIX since Windows doesn't let us poll on stdio and sockets
at the same time because I don't think they want tools like this running
on their platform. So if you want forbidden fruit, it's here so enjoy it
2021-04-23 18:53:57 -07:00
Justine Tunney
4effa23528 Make more major improvements to redbean
- POSIX regular expressions for Lua
- Improved protocol parsing and encoding
- Additional APIs for ZIP storage retrieval
- Fix st_mode issue on NT for regular files
- Generalized APIs for URL and Host handling
- Worked out the kinks in resource resolution
- Allow for custom error pages like /404.html
2021-04-20 19:14:21 -07:00
Justine Tunney
bf03b2e64c Make major improvements to redbean and libraries
The most exciting improvement is dynamic pages will soon be able to use
the executable itself as an object store. it required a heroic technique
for overcoming ETXTBSY restrictions which lets us open the executable in
read/write mode, which means (1) wa can restore the APE header, and (2)
we can potentially containerize redbean extension code so that modules
you download for your redbean online will only impact your redbean.

Here's a list of breaking changes to redbean:

- Remove /tool/net/ prefix from magic ZIP paths
- GetHeader() now returns NIL if header is absent

Here's a list of fixes and enhancements to redbean:

- Support 64-bit ZIP archives
- Record User-Agent header in logs
- Add twelve error handlers to accept()
- Display octal st_mode on listing page
- Show ZIP file comments on listing page
- Restore APE MZ header on redbean startup
- Track request count on redbean index page
- Report server uptime on redbean index page
- Don't bind server socket using SO_REUSEPORT
- Fix #151 where Lua LoadAsset() could free twice
- Report rusage accounting when workers exit w/ -vv
- Use ZIP iattr field as text/plain vs. binary hint
- Add ParseUrl() API for parsing things like a.href
- Add ParseParams() API for parsing HTTP POST bodies
- Add IsAcceptablePath() API for checking dots, etc.
- Add IsValidHttpToken() API for validating sane ASCII
- Add IsAcceptableHostPort() for validating HOST[:PORT]
- Send 400 response to HTTP/1.1 requests without a Host
- Send 403 response if ZIP or file isn't other readable
- Add virtual hosting that tries prepending Host to path
- Route requests based on Host in Request-URI if present
- Host routing will attempt to remove or add the www. prefix
- Sign-extend UNIX timestamps and don't adjust FileTime zone

Here's some of the improvements made to Cosmopolitan Libc:

- Fix ape.S indentation
- Improve consts.sh magnums
- Write pretty good URL parser
- Improve rusage accounting apis
- Bring mremap() closer to working
- Added ZIP APIs which will change
- Check for overflow in reallocarray()
- Remove overly fancy linkage in strerror()
- Fix GDB attach on crash w/ OpenBSD msyscall()
- Make sigqueue() portable to most UNIX distros
- Make integer serialization macros more elegant
- Bring back 34x tprecode8to16() performance boost
- Make malloc() more resilient to absurdly large sizes
2021-04-18 12:34:15 -07:00
Rowan Easter-Robinson
69c508729e
Added LaunchBrowser API (#153)
You can now add LaunchBrowser() to .init.lua which will ask
redbean to open your website in the local desktop browser
when the server is launched.
2021-04-18 07:24:36 -07:00
Justine Tunney
8a91518633 Fix issues revealed by ECMAScript test262
Cosmopolitan's QuickJS is now equally conformant and performant, with
the exception of Atomics, which have been disabled since Cosmopolitan
currently doesn't support pthreads.

QuickJS memory usage -- BigNum 2021-03-27 version, 64-bit, malloc limit: -1

NAME                    COUNT     SIZE
memory allocated          937   131764  (140.6 per block)
memory used               938   116103  (8 overhead, 16.7 average slack)
atoms                     513    21408  (41.7 per atom)
objects                   170    12279  (72.2 per object)
  properties              864    15531  (5.1 per object)
  shapes                   58    12995  (224.1 per shape)
bytecode functions         13     1512
  bytecode                 13      867  (66.7 per function)
C functions                99
arrays                      1
  fast arrays               1
  elements                  1       16  (1.0 per fast array)

Result: 35/74740 errors, 1279 excluded, 485 skipped, 19 new, 2 fixed

real    2m40.828s
user    2m29.764s
sys     0m10.939s
2021-04-10 17:15:35 -07:00
Justine Tunney
8f52c0d773 Get Fabrice Bellard's JavaScript engine to build
$ m=tiny
$ make -j12 MODE=$m o/$m/third_party/quickjs/qjs.com
$ o/$m/third_party/quickjs/qjs.com -e 'console.log(2 + 2)'
4
$ ls -hal o/$m/third_party/quickjs/qjs.com
631.5K

See #97
2021-04-09 01:06:57 -07:00
Justine Tunney
59575f7e80 Perform some minor code cleanup 2021-04-06 12:46:52 -07:00
Justine Tunney
6c16f208b5 Remove unintended log statements 2021-04-01 20:01:26 -07:00
Justine Tunney
83abd68029 Parse Content-Range with missing right hand side
Fixes #144
2021-04-01 18:52:54 -07:00
Justine Tunney
7abca1531f Cosmopolitan 0.3 2021-03-29 03:20:03 -07:00
Justine Tunney
7dc8973567 Enable serving zip directory entries 2021-03-29 03:10:11 -07:00
Justine Tunney
1753b669cf Have redbean show zip listing as default / handler
If an "index.lua" or "index.html" doesn't exist in zip file or the
filesystem, and no redirects have been defined for it either, then
redbean will render a listing of the zip central directory content
only if the request uri points to the root path.
2021-03-29 01:22:49 -07:00
Justine Tunney
c6f75bf95a Fix issue with testing service
What was I thinking imposing arbitrary limits. I hate limits.
2021-03-28 18:22:36 -07:00
Justine Tunney
52565e7af3 Make minor performance improvement to HTTP parser
redbean's message parser now takes 300 nanoseconds to parse a standard
request sent by chrome, whereas it previous took 600 nanoseconds.
2021-03-28 18:00:29 -07:00
Justine Tunney
3c19b6e352 Add support for serving directly from filesystem
You can now pass `-D directory` to redbean which will serve assets from
the local filesystem. This is useful for development since it allows us
to skip needing to shut down the server and run InfoZIP when testing an
iteration of a lua server page script.

See #97
2021-03-28 08:02:17 -07:00
Justine Tunney
a1677d605a Transcode ISO-8859-1 in HTTP headers
If we keep making changes like this, redbean might not be a toy anymore.
Additional steps are also being taken now to prevent ANSI control codes
sent by the client from slipping into logs.
2021-03-28 00:10:17 -07:00
Justine Tunney
dcbd2b8668 Add base64 support
Lua Server Pages may now call the EncodeBase64() and DecodeBase64()
functions should they wish to do things like emit embeded data URIs

See #97
2021-03-27 18:17:54 -07:00
Justine Tunney
4d21cd315d Support POST parameters in redbean server pages
See #97
2021-03-27 07:29:55 -07:00
Justine Tunney
da36e7e256 Make major improvements to stdio
Buffering now has optimal performance, bugs have been fixed, and some
missing apis have been introduced. This implementation is also now more
production worthy since it's less brittle now in terms of system errors.
That's going to help redbean since lua i/o is all based on stdio.

See #97
2021-03-26 22:31:41 -07:00
Justine Tunney
09bcfa23d5 Make major improvements to redbean
- lua server pages
- lua http library
- http v0.9 support
- request uri parsing
- fork failure recovery
- accelerated redirects
- http pipelining support
- lenient message framing
- html / uri / js escaping
- fix shutdown signal handling
2021-03-25 02:24:05 -07:00
Justine Tunney
6b90ff60cd Add preliminary lua support to redbean
This change only implements enough Lua support to send a Hello World
response. The redbean executable size increases from ~128kb to 260kb
and the requests per second decreases from 1000k to 600k. That's the
fastest it can go and that's extremely impressive compared to Python

See #97
2021-03-21 17:27:53 -07:00
fabriziobertocci
5af25f687a
Rename eai2str to gai_strerror (#131) 2021-03-20 20:48:40 -07:00
Justine Tunney
807706a099 Perform minor fixups
One of those fixups is making sure that AF_LOCAL is equal to AF_UNIX on
the New Technology. See #122
2021-03-13 19:40:04 -08:00
Justine Tunney
3f35b37b68 Remove nonstandard defines from limits.h (#115) 2021-03-08 17:34:44 -08:00
Justine Tunney
33e8fc8687 Expose public garbage collector API for C language
You can now do epic things like this:

    puts(_gc(xasprintf("%d", 123)));

The _gc() API is shorthand for _defer() which works like Go's keyword:

    const char *s = xasprintf("%d", 123);
    _defer(free, s);
    puts(s);

Be sure to always use -fno-omit-frame-pointer which makes code fast too.

Enjoy! See also #114
2021-03-08 10:59:34 -08:00
Justine Tunney
2bd1e72d5a Remove garbage collector macro from header (#114)
We can put this back the moment someone requests it. Pain-free garbage
collection for the C language is pretty cool. All it does is overwrite
the return address with a trampoline that calls free(). It's not clear
what it should be named if it's made a public API.
2021-03-07 20:23:29 -08:00
Alison Winters
8a6ac6dd63
Set errno when out of range in strtoimax (#111) 2021-03-07 14:38:56 -08:00
Justine Tunney
7583b3accc Show better error on bind() error
See #51
2021-03-07 11:31:44 -08:00
Justine Tunney
4d4aa9e791 Add automatic tests for pull requests
Now when you send a pull request Travis CI will build the entire
repository and run all the tests for you automatically. It takes
approximately two minutes to finish so you can get fast feedback
2021-03-06 21:39:03 -08:00
Justine Tunney
c3ed8d6c7f Make %p consistent with glibc and musl
See #61
2021-03-06 14:36:44 -08:00
Justine Tunney
e26bdbec52 Make examples folder somewhat more focused 2021-03-05 06:09:12 -08:00
Justine Tunney
43b2475aaa Handle more pow cases (#61) 2021-03-04 06:14:07 -08:00
Justine Tunney
1a08594f95 Add honeybadger to ape.S 2021-03-03 15:00:41 -08:00
Justine Tunney
e3ee3c78e9 Refactor some Blinkenlights code 2021-03-03 13:09:06 -08:00
Justine Tunney
8af91bcbe7 Make pow() conform to standard definition 2021-03-03 09:21:27 -08:00
Justine Tunney
754974faaa Remove SOCK_NONBLOCK from Echo Server example
See #102 for further explanation.
2021-03-03 06:17:56 -08:00
Justine Tunney
9367253b4d Add more libm unit tests and fixes
See #61
2021-03-02 13:57:23 -08:00
Justine Tunney
3e19b96ab8 Add more POSIX function stubs
Cosmopolitan currently doesn't support threads and it doesn't do
anything fancy in longjmp/setjmp so this change was simple to do

- localeconv
- _setjmp (same as setjmp)
- _longjmp (same as longjmp)
- strcoll (same as strcmp)
- flockfile (does nothing)
- funlockfile (does nothing)
- ftrylockfile (does nothing)

See #61
2021-03-02 03:27:55 -08:00
Justine Tunney
8a6a1e25b1 Fix redbean daemonization
Fixes #94
2021-03-02 00:43:30 -08:00
Justine Tunney
7e4021bf82 Put Redbean in uniprocess mode on NT temporarily
The Windows fork() polyfill (which we implement using pure WIN32)
appears to be acting strangely. It's possible a regression was
introduced by recent changes that our tests didn't catch.

This change is workaround so we can upload a new working binary release
to the Redbean website until we can fully fix the problem.

See also #90
2021-03-01 23:45:49 -08:00
Justine Tunney
f4298f10c2 Generate ZIP files the same way as Windows 2021-03-01 06:24:11 -08:00
Justine Tunney
d932948fb4 Remove more nonstandard stuff from cosmopolitan.h
Fixes #61
2021-03-01 00:18:23 -08:00
Justine Tunney
c42dbe0e9e Issue new Blinkenlights release 2021-02-27 22:35:50 -08:00
Justine Tunney
94afa982c3 Fix zip executables on MacOS
Here's why we got those `Killed: 11` failures on MacOS after modifying
the contentns of the redbean.com executable. If you were inserting a
small file, such as a HelloWorld.html file, then InfoZIP might have
decreased the size of the executable to less than what the Mach-O
section had been expecting.

That's because when zipobj.com put things like time zone data in the
executable, it aligned each zip file entry on a 64-byte boundary, simply
for the sake of readability in binary dumps. But when InfoZIP edited the
file it would rewrite every entry using ZIP's usual 2-byte alignment.
Thus causing shrinkage.

The solution was to reconfigure the linker script so that zip file bits
that get put into the executable at link-time, such as timezone data,
aren't officially part of the executable image, i.e. we don't want the
operating system to load that part.

The original decision to put the linked zip files into the .data section
was mostly made so that when the executable was run in its .com.dbg form
it would still have the zip entries be accessible, even though there was
tons of GNU debug data following the central directory. We're not going
to be able to do that. The .com executable should be the canonical
executable. We have really good tools for automatically attaching and
configuring GDB correctly with debug symbols even when the .com is run.
We'll have to rely on those in cases where zip embedding is used.

See #53
See #54
See #68
2021-02-27 18:16:59 -08:00
Justine Tunney
cc56038b05 Fix memzoom on Windows
Right now we can't call poll() on file and fifos on Windows. So we work
around that by simply blocking on keyboard input. This is OK because on
Windows we can't use /proc/*/mem for realtime monitoring, so there's no
reason not to block.

See #65
2021-02-27 15:01:00 -08:00
Thomas Bernard
1c9065510f redbean.c: remove duplicate png content-types 2021-02-27 13:37:36 -08:00
Justine Tunney
218ef49147 Fix redbean zip central directory lookup
This regression snuck in at some point. It resulted in the program
sometimes failing to load when the zip content was changed.
2021-02-27 12:08:28 -08:00
Justine Tunney
19d0c15e03 Perform some code cleanup 2021-02-27 10:33:32 -08:00
Justine Tunney
3e17c7b20f Make Reddit users happy 2021-02-26 20:18:10 -08:00
Justine Tunney
91f4167a45 Add root redirect for redbean webserver
We have a webserver demo:

    make -j8 o//tool/net/redbean.com
    o/tool/net/redbean.com -v

It's been a little bit confusing that until now you had to visit the
following URL in order to see the default web page:

    http://127.0.0.1:8080/tool/net/redbean.html

The following URLs will now redirect to the above page, but only if
nothing's been defined for those paths and they would otherwise result
in a 404 response:

    http://127.0.0.1:8080/
    http://127.0.0.1:8080/index.html
2021-02-25 19:14:22 -08:00
Justine Tunney
40291c9db3 Improve signal handling and math
- Polyfill ucontext_t on FreeBSD/OpenBSD/NetBSD
- Add tests confirming signals can edit CPU state
- Work towards supporting ZIP filesystem on bare metal
- Add more tinymath unit tests for POSIX conformance
- Add X87 and SSE status flags to crash report
- Fix some bugs in blinkenlights
- Fix llvm build breakage
2021-02-25 18:33:33 -08:00
Justine Tunney
cdc54ea1fd Use unsigned leb128 for magnums 2021-02-24 04:00:38 -08:00
Justine Tunney
edd9297eba Support malloc() on bare metal
Your Actually Portable Executables now contains a simple virtual memory
that works similarly to the Linux Kernel in the sense that it maps your
physical memory to negative addresses. This is needed to support mmap()
and malloc(). This functionality has zero code size impact. For example
the MODE=tiny LIFE.COM executable is still only 12KB in size.

The APE bootloader code has also been simplified to improve readibility
and further elevate the elegance by which we're able to support so many
platforms thereby enhancing verifiability so that we may engender trust
in this bootloading process.
2021-02-24 00:53:24 -08:00
Justine Tunney
537c21338b Add UEFI support
This is mutually exclusive with Windows support. Documentation for how
to use it has been written in libc/runtime/efimain.c
2021-02-21 21:33:04 -08:00
Justine Tunney
4a5698b5c9 Implement better hypot function 2021-02-21 11:06:18 -08:00
Justine Tunney
e345b42d78 Don't scrub PATH variable since Clang needs it 2021-02-20 12:54:13 -08:00
Justine Tunney
e85aeda4ba Fix some more build issues (#43)
We're now scrubbing environment variables in compile.com since gnu make
was not behaving as expected. It also appears there was a regression in
recent revisions that caused ASAN to be turned off for most binaries in
dbg mode, which has now been fixed. Cosmopolitan is fully ASAN hardened
down to the lowest level libraries and it doesn't need any interceptors
2021-02-20 12:39:39 -08:00
Justine Tunney
3e1fd1d962 Add tool for parsing strace output 2021-02-20 10:45:55 -08:00
Justine Tunney
b740cca642 Improve build system
- Reduce full build latency from ~20s to ~18s
- Bring back silent mode if `make V=0` is passed
- Demodernize utimes() polyfill so it works RHEL5
- Delete some old shell scripts that are no longer needed
- Truncate long lines when outputting builds to Emacs buffers
2021-02-19 23:03:34 -08:00
Justine Tunney
5f1415af3a Add help dialog to Blinkenlights emulator 2021-02-19 17:16:17 -08:00
Justine Tunney
2c15efc249 Add curl example (#42)
make -j8 o//examples/curl.com
    o//examples/curl.com http://justine.lol/ape.html
2021-02-18 19:20:41 -08:00
Justine Tunney
bfa8581537 Trim down executable sizes 2021-02-11 08:37:18 -08:00
Justine Tunney
e75ffde09e Get codebase completely working with LLVM
You can now build Cosmopolitan with Clang:

    make -j8 MODE=llvm
    o/llvm/examples/hello.com

The assembler and linker code is now friendly to LLVM too.
So it's not needed to configure Clang to use binutils under
the hood. If you love LLVM then you can now use pure LLVM.
2021-02-09 02:57:32 -08:00
Justine Tunney
0e36cb3ac4 Improve dead code elimination 2021-02-08 04:04:42 -08:00
Justine Tunney
2f3bd90216 Apply some touchups 2021-02-07 07:02:46 -08:00
Justine Tunney
9f149e1de3 Elevate .text.syscall to .privileged
It turns out adding OpenBSD msyscall() origin verification broke the
--ftrace flag. The executable needs to issue raw syscalls while it's
rewriting itself. So they need to be in the same section, and that's
just plain simpler too.
2021-02-06 04:49:22 -08:00
Justine Tunney
d7733579d3 Fix Clang support
The amalgamated release is now confirmed to be working with Clang,
including its integrated assembler.

Fixes #41
2021-02-06 00:29:09 -08:00
Justine Tunney
a91ba89d85 Fix sigsuspend() and sigprocmask() on OpenBSD
Test reliability is now iron clad.
2021-02-05 12:19:43 -08:00
Justine Tunney
b2cd58a322 Get all tests passing on NetBSD 2021-02-05 09:49:28 -08:00
Justine Tunney
23ae9dfceb Add NetBSD support 2021-02-05 06:19:45 -08:00
Alexander Nicholi
dbd7edba10
sync and try changes 2021-02-04 08:33:22 -05:00
Justine Tunney
c93a4661a3 Add Cosmopolitan Honeybadger to README.md 2021-02-04 03:31:03 -08:00
Justine Tunney
a37960a3af Remove dollars from system call support symbols 2021-02-03 19:35:29 -08:00
Justine Tunney
a8d7195777 Make Cosmopolitan ANSI C89 compatible
You can now use cosmopolitan.h with an ANSI C89 compiler like MSVC. The
Cosmopolitan codebase itself won't support being compiled that way. But
you can build objects that link against Cosmopolitan using any compiler
and you can furthermore use tools like IntelliSense that can't even GNU

See also #40
2021-02-03 17:48:59 -08:00
Justine Tunney
46085797b6 Make ANSI mode closer to being ANSI 2021-02-03 17:14:17 -08:00
Justine Tunney
28135b7a20 Add q keyboard shortcut to printvideo.com (#37) 2021-02-03 15:53:33 -08:00
Alexander Nicholi
ec9bfd8c56
finish intellisense support and sync with upstream 2021-02-03 13:50:08 -05:00
Justine Tunney
4e56d89dcd Eliminate some flakes
- Get ASAN working on Windows.

- Deleting directories and then recreating them with the same name in a
  short period of time appears to be a no-no on Windows.

- There's no reason to call FlushFileBuffers on close() for pipes, and
  it's harmful since it might block indefinitely for no good reason.
2021-02-03 06:25:27 -08:00
Justine Tunney
23a14b537c Delete LIBC_CALLS_HEFTY
- fork() no longer requires malloc()
- readdir() moved to LIBC_STDIO
- Custom APIs moved to LIBC_X
2021-02-02 22:17:53 -08:00
Justine Tunney
c843243322 Implement more security stuff
- Support deterministic stacks on OpenBSD
- Support OpenBSD system call origin verification
- Fix overrun by one in chibicc string token allocator
- Get all chibicc tests passing under Address Sanitizer
2021-02-02 20:21:06 -08:00
Justine Tunney
cbfd4ccd1e Make more functions friendly to Address Sanitizer 2021-02-02 03:45:31 -08:00
Justine Tunney
1ff9ab95ac Make C memory safe like Rust
This change enables Address Sanitizer systemically w/ `make MODE=dbg`.
Our version of Rust's `unsafe` keyword is named `noasan` which is used
for two functions that do aligned memory chunking, like `strcpy.c` and
we need to fix the tiny DEFLATE code, but that's it everything else is
fabulous you can have all the fischer price security blankets you need

Best of all is we're now able to use the ASAN data in Blinkenlights to
colorize the memory dumps. See the screenshot below of a test program:

  https://justine.lol/blinkenlights/asan.png

Which is operating on float arrays stored on the stack, with red areas
indicating poisoned memory, and the green areas indicate valid memory.
2021-02-01 03:58:46 -08:00
Justine Tunney
caee314a50 Make PATH search do the right thing w/ empty path 2021-01-29 01:27:09 -08:00
Justine Tunney
971bc8147f Check for EINTR in poll$nt()
This is a bandaid that lets CTRL-C work in daemons until a better
solution for signals on Windows can be implemented.
2021-01-28 15:53:06 -08:00
Justine Tunney
efc66f8525 Fix our remarkable 184kb x86_64 emulator
Here's how to build it from source and try it out!

    m=tiny
    make -j18 MODE=$m o/$m/tool/build/tinyemu.com
    make -j18 MODE=$m o/$m/tool/viz/deathstar.com
    o/$m/tool/build/tinyemu.com o/$m/tool/viz/deathstar.com
2021-01-27 20:02:23 -08:00
Justine Tunney
d7ac16a9ed Work towards improving signals and processes 2021-01-27 19:34:02 -08:00
Justine Tunney
45b72485ad Fix XNU / FreeBSD / OpenBSD / RHEL5 / NT bugs
For the first time ever, all tests in this codebase now pass, when
run automatically on macos, freebsd, openbsd, rhel5, rhel7, alpine
and windows via the network using the runit and runitd build tools

- Fix vfork exec path etc.
- Add XNU opendir() support
- Add OpenBSD opendir() support
- Add Linux history to syscalls.sh
- Use copy_file_range on FreeBSD 13+
- Fix system calls with 7+ arguments
- Fix Windows with greater than 16 FDs
- Fix RUNIT.COM and RUNITD.COM flakiness
- Fix OpenBSD munmap() when files are mapped
- Fix long double so it's actually long on Windows
- Fix OpenBSD truncate() and ftruncate() thunk typo
- Let Windows fcntl() be used on socket files descriptors
- Fix Windows fstat() which had an accidental printf statement
- Fix RHEL5 CLOCK_MONOTONIC by not aliasing to CLOCK_MONOTONIC_RAW

This is wonderful. I never could have dreamed it would be possible
to get it working so well on so many platforms with tiny binaries.

Fixes #31
Fixes #25
Fixes #14
2021-01-25 18:31:17 -08:00
Justine Tunney
f0600a898c Fix metal bugs so deathstar.com runs in qemu (#20)
- Remove XD bit in page tables
- Fix cylinder+head+sector arithmetic
- Implement fstat() for serial file descriptors on metal

Here's how to boot an Actually Portable Executable in QEMU:

    make -j12 o//tool/viz/deathstar.com
    qemu-system-x86_64 -serial stdio -fda o//tool/viz/deathstar.com

Here's a screenshot of DEATHSTAR.COM booted in QEMU:
https://justine.lol/cosmopolitan/cosmo-metal-qemu.png

Thus metal support is in much better shape now, but still incomplete.
Only a few system calls have been polyfilled. To figure out which ones
your program needs, simply boot it in the blinkenlights emulator with a
breakpoint, and press CTRL-C to continue to the system call breakpoint.
If it doesn't break then you should be good. (Note: to emulate normally
you can press 'c' and use CTRL-T and ALT-T to tune the speed.)

    m=tiny
    make -j12 SILENT=0 MODE=$m          \
      o/$m/tool/build/blinkenlights.com \
      o/$m/tool/viz/deathstar.com
    o/$m/tool/build/blinkenlights.com   \
      -r -t -b systemfive.linux         \
      o/$m/tool/viz/deathstar.com

Thank @Theldus for the bug report that made this change possible.
Fixes #20 which explains this change further.
2021-01-16 17:52:15 -08:00
Justine Tunney
9f68d6eee9 Fix link order in cosmopolitan.a
It turned out that the linker was doing the wrong with the amalgamation
library concerning weak stubs. A regression test has been added and new
binaries have been uploaded to https://justine.lol/cosmopolitan/

Ideally this should be fixed by building a tool that turns multiple .a
files into a single .a file with deduplication. As a workaround for now
the cosmopolitan.a build is restructured to not include LIBC_STUBS which
meant technical debt needed to be paid off where non-stub interfaces
were moved to LIBC_INTRIN and LIBC_NEXGEN32E.

Thank @PerfectProductions in #31 for the report!
2021-01-16 12:05:41 -08:00
Justine Tunney
04f1d89f84 Replace .pushsection directives (#30) 2021-01-10 13:36:31 -08:00
Justine Tunney
0e85b136ae Fix Windows 7 support (#19)
This change pays off technical debt with the function -> DLL mappings in
libc/nt/master.sh, which was originally defined based on binary analysis
on Windows 10. It's now been updated so the kernel32/kernelbase/advapi32
imports should be exactly as they are written, on the MSDN documentation
and that wouldn't have been easy without Geoff Chappell's work thank him

https://www.geoffchappell.com/studies/windows/win32/index.htm
2020-12-28 13:52:02 -08:00
Justine Tunney
37a4c70c36 Change license 2020-12-27 17:18:44 -08:00
Justine Tunney
548dcb9f08 Further refine documentation 2020-12-27 17:05:03 -08:00
Justine Tunney
1bc3a25505 Improve documentation
The Cosmo API documentation page is pretty good now
https://justine.lol/cosmopolitan/documentation.html
2020-12-27 07:02:35 -08:00
Justine Tunney
13437dd19b Auto-generate some documentation 2020-12-26 02:09:07 -08:00
Justine Tunney
117d0111ab Fix space path bug in printvideo.com (#11)
In addition to removing the URI downloading support. It was originally
intended to have this command play YouTube videos too but that support
was never completed. Latest release binaries can be obtained via:

    wget https://justine.lol/printvideo.com      # 284k ape executable
    wget https://justine.lol/printvideo.com.dbg
2020-12-24 16:13:13 -08:00
Justine Tunney
830334d767 Hunt down some small bugs 2020-12-24 16:01:48 -08:00
Justine Tunney
95b142e4e5 Make minor improvements 2020-12-23 23:42:56 -08:00
Justine Tunney
113eeabd85 Fix another build error
It would appear o/depend can't help with generated files.
2020-12-19 13:47:51 -08:00
Justine Tunney
b562d6fdb3 Make minor improvements 2020-12-19 11:21:04 -08:00
Justine Tunney
1fc91f3580 Fold conv package into fmt
Both packages had nearly identical dependency requirements, so merging
them should help reduce the complexity of the build graph.
2020-12-09 16:52:00 -08:00
Justine Tunney
9df2cef4c4 Enhance chibicc 2020-12-09 04:00:48 -08:00
Justine Tunney
8da931a7f6 Add chibicc
This program popped up on Hacker News recently. It's the only modern
compiler I've ever seen that doesn't have dependencies and is easily
modified. So I added all of the missing GNU extensions I like to use
which means it might be possible soon to build on non-Linux and have
third party not vendor gcc binaries.
2020-12-06 16:20:21 -08:00
Justine Tunney
e44a0cf6f8 Make improvements 2020-12-01 03:43:40 -08:00
Justine Tunney
3e4fd4b0ad Add epoll and do more release readiness changes
This change also pays off some of the remaining technical debt with
stdio, file descriptors, and memory managemnt polyfills.
2020-11-28 12:01:51 -08:00
Justine Tunney
ea0b5d9d1c Get Cosmopolitan into releasable state
A new rollup tool now exists for flattening out the headers in a way
that works better for our purposes than cpp. A lot of the API clutter
has been removed. APIs that aren't a sure thing in terms of general
recommendation are now marked internal.

There's now a smoke test for the amalgamation archive and gigantic
header file. So we can now guarantee you can use this project on the
easiest difficulty setting without the gigantic repository.

A website is being created, which is currently a work in progress:
https://justine.storage.googleapis.com/cosmopolitan/index.html
2020-11-25 08:19:00 -08:00
Justine Tunney
dba7552c1e Add Conway's Game of Life 2020-11-18 08:26:03 -08:00
Justine Tunney
db33973e0a Get fork() working on Windows
This is done without using Microsoft's internal APIs. MAP_PRIVATE
mappings are copied to the subprocess via a pipe, since Microsoft
doesn't want us to have proper COW pages. MAP_SHARED mappings are
remapped without needing to do any copying. Global variables need
copying along with the stack and the whole heap of anonymous mem.
This actually improves the reliability of the redbean http server
although one shouldn't expect 10k+ connections on a home computer
that isn't running software built to serve like Linux or FreeBSD.
2020-11-13 03:14:39 -08:00
Justine Tunney
aea89fe832 Make minor improvements
- Work towards simplifying ape.S startup process
- Rewrote ar because it took minutes to build cosmopolitan.a
2020-11-09 15:41:11 -08:00
Justine Tunney
95bc650be8 Fix missing header include 2020-11-07 00:05:25 -08:00
Justine Tunney
9fe95ef12b Add tool for viewing memory
https://justine.storage.googleapis.com/memzoom/index.html
2020-11-06 20:20:10 -08:00
Justine Tunney
2d80bbc802 Get binaries closer to running without an o/s
blinkenlights now does a pretty good job emulating what happens when
binaries boot from BIOS into long mode. So it's been much easier to
debug the bare metal process and wrinkle out many issues.
2020-11-02 19:12:47 -08:00
Justine Tunney
feed0d2b0e Add minor improvements and cleanup 2020-10-27 03:39:46 -07:00
Justine Tunney
9e3e985ae5 Make terminal ui binaries work well everywhere
Here's some screenshots of an emulator tui program that was compiled on
Linux, then scp'd it to Windows, Mac, and FreeBSD.

https://justine.storage.googleapis.com/blinkenlights-cmdexe.png
https://justine.storage.googleapis.com/blinkenlights-imac.png
https://justine.storage.googleapis.com/blinkenlights-freebsd.png
https://justine.storage.googleapis.com/blinkenlights-lisp.png

How is this even possible that we have a nontrivial ui binary that just
works on Mac, Windows, Linux, and BSD? Surely a first ever achievement.

Fixed many bugs. Bootstrapped John McCarthy's metacircular evaluator on
bare metal in half the size of Altair BASIC (about 2.5kb) and ran it in
emulator for fun and profit.
2020-10-19 06:38:31 -07:00
Justine Tunney
680daf1210 Fix build/test breakage caused by last commit 2020-10-06 12:16:17 -07:00
Justine Tunney
c45e46f871 Add fixes performance and static web server 2020-10-05 23:11:49 -07:00
Justine Tunney
b6793d42d5 Add The LISP Challenge
This change introduces a 2.5kb program that's comes pretty close so far
to bootstrapping John McCarthy's metacircular evaluator on bare metal.
2020-10-01 01:20:13 -07:00
Justine Tunney
23d333c090 Make more improvements
This change includes many bug fixes, for the NT polyfills, strings,
memory, boot, and math libraries which were discovered by adding more
tools for recreational programming, such as PC emulation. Lemon has also
been vendored because it works so well at parsing languages.
2020-09-28 01:20:34 -07:00
Justine Tunney
416fd86676 Make improvements
- Emulator can now test the αcτµαlly pδrταblε εxεcµταblε bootloader

- Whipped up a webserver named redbean. It services 150k requests per
  second on a single core. Bundling assets inside zip enables extremely
  fast serving for two reasons. The first is that zip central directory
  lookups go faster than stat() system calls. The second is that both
  zip and gzip content-encoding use DEFLATE, therefore, compressed
  responses can be served via the sendfile() system call which does an
  in-kernel copy directly from the zip executable structure. Also note
  that red bean zip executables can be deployed easily to all platforms,
  since these native executables work on Linux, Mac, BSD, and Windows.

- Address sanitizer now works very well
2020-09-14 00:02:34 -07:00
Justine Tunney
7327c345f9 Get address sanitizer mostly working 2020-09-03 05:44:37 -07:00
Justine Tunney
f213556366 Add binfmt_misc workaround detector to builds
This is a recently introduced Linux Kernel feature that gives people
like Debian package mantainers the power to arbitrarily redefine how
executables are interpreted by the kernel. If your system gets tuned
this way and you're not able to disable it, then you need to restore
default behavior for the APE MZqFpD prefix as follows:

    sudo sh -c "echo ':APE:M::MZqFpD::/bin/sh:' >/proc/sys/fs/binfmt_misc/register"

This prefix will cover all .com executables built with this tooling.
Please don't run the above command unless you're certain you need it.
See #2 for additional context.
2020-08-31 20:09:23 -07:00
Justine Tunney
bd29223891 Fix bugs and have emulator emulate itself 2020-08-31 05:17:31 -07:00
Justine Tunney
5aabacb361 Add pseudoteletypewriter to emulator
https://justine.storage.googleapis.com/emulator628.mp4
2020-08-29 23:51:09 -07:00
Justine Tunney
e86cff8ba0 Fix printvideo regression and minor improvements 2020-08-26 09:41:07 -07:00
Justine Tunney
f4f4caab0e Add x86_64-linux-gnu emulator
I wanted a tiny scriptable meltdown proof way to run userspace programs
and visualize how program execution impacts memory. It helps to explain
how things like Actually Portable Executable works. It can show you how
the GCC generated code is going about manipulating matrices and more. I
didn't feel fully comfortable with Qemu and Bochs because I'm not smart
enough to understand them. I wanted something like gVisor but with much
stronger levels of assurances. I wanted a single binary that'll run, on
all major operating systems with an embedded GPL barrier ZIP filesystem
that is tiny enough to transpile to JavaScript and run in browsers too.

https://justine.storage.googleapis.com/emulator625.mp4
2020-08-25 04:43:42 -07:00
Justine Tunney
72b654cb6c Add NES emulator
It's now possible to play classic video game roms with teletypewriters.
https://justine.storage.googleapis.com/nesemu.png
2020-07-02 13:46:08 -07:00
Justine Tunney
4918121810 Add new calculator app
Cosmopolitan makes it easy to build and maintain programming languages,
since it abstracts system call #ifdef toil, so you can focus on vision.
Here's an example of a language that isn't turing complete, weighing in
at <1,000 lines of modern C, intended to help with testing libc / libm:

    .1 .2 + .3 - abs epsilon < assert
    pi sqrt pi sqrt * pi - abs epsilon < assert
    -.5 rint dup 0 = assert signbit assert
    -.5 nearbyint dup 0 = assert signbit assert
    -.5 ceil dup 0 = assert signbit assert
    -.5 trunc dup 0 = assert signbit assert
    -.5 round -1 = assert
    -.5 floor -1 = assert
    0 signbit ! assert

CALCULATOR.COM pays homage to CALC.EXE recently removed from Windows 10.
Microsoft should bundle this app instead. It too is roughly 100kb, works
just fine w/ command prompt, and portable since it runs on Mac/Linux/BSD
too while bundling even more features than the calculator on Google.com.

It should be possible to run CALCULATOR.COM on Android and iOS too, just
in case anyone needs a backend pipe driven framework, for graphical user
interfaces of calculators. Sadly we haven't tried it since we don't know
how to run software on telephones so the system call support is a priori
2020-06-30 19:58:08 -07:00
Justine Tunney
ac00be1a4e Make small fixes and oops ran clang-format on dtoa 2020-06-30 19:55:47 -07:00
Justine Tunney
d51409ccd9 Add glob and some finer tuning of documentation 2020-06-21 15:23:35 -07:00
Justine Tunney
2e979c00c3 Polish up repository and other revisions 2020-06-16 06:38:43 -07:00
Justine Tunney
b4269930f7 Add scouts honor escape hatch for source embedding 2020-06-15 19:01:28 -07:00
Justine Tunney
c91b3c5006 Initial import 2020-06-15 07:18:57 -07:00