Commit graph

500 commits

Author SHA1 Message Date
Gautham
63b867bd2f
Added _multiprocessing to Python (#259)
Also changed some PYTHON_YOINKs so that http.server would work in
MODE=tiny.
2021-08-25 19:45:59 -07:00
Paul Kulchenko
0b7737b20a
Fix redbean StoreAsset w/ Deflate (#258) 2021-08-25 05:43:26 -07:00
Paul Kulchenko
818926dbae
Fix redbean latency and resource usage logging (#257) 2021-08-22 20:16:10 -07:00
Justine Tunney
7d25fb0090 Import some Lua documentation
I personally find it easier to read the documentation in Emacs
using JavaDoc style comments.
2021-08-22 15:03:04 -07:00
Paul Kulchenko
41b9eb6873
Redbean log improvements (#256)
* Update log level for redbean messages for better log content
* Add categories to redbean log messages
* Reorganize Lua error logging for consistency
* Replace perror logging with FATALF
* Update server failure reporting to better identify response code
* Replace `flogf` with DIEF logging for consistency
2021-08-22 15:01:52 -07:00
Justine Tunney
00611e9b06 Improve ZIP filesystem and change its prefix
The ZIP filesystem has a breaking change. You now need to use /zip/ to
open() / opendir() / etc. assets within the ZIP structure of your APE
binary, instead of the previous convention of using zip: or zip! URIs.
This is needed because Python likes to use absolute paths, and having
ZIP paths encoded like URIs simply broke too many things.

Many more system calls have been updated to be able to operate on ZIP
files and file descriptors. In particular fcntl() and ioctl() since
Python would do things like ask if a ZIP file is a terminal and get
confused when the old implementation mistakenly said yes, because the
fastest way to guarantee native file descriptors is to dup(2). This
change also improves the async signal safety of zipos and ensures it
doesn't maintain any open file descriptors beyond that which the user
has opened.

This change makes a lot of progress towards adding magic numbers that
are specific to platforms other than Linux. The philosophy here is that,
if you use an operating system like FreeBSD, then you should be able to
take advantage of FreeBSD exclusive features, even if we don't polyfill
them on other platforms. For example, you can now open() a file with the
O_VERIFY flag. If your program runs on other platforms, then Cosmo will
automatically set O_VERIFY to zero. This lets you safely use it without
the need for #ifdef or ifstatements which detract from readability.

One of the blindspots of the ASAN memory hardening we use to offer Rust
like assurances has always been that memory passed to the kernel via
system calls (e.g. writev) can't be checked automatically since the
kernel wasn't built with MODE=asan. This change makes more progress
ensuring that each system call will verify the soundness of memory
before it's passed to the kernel. The code for doing these checks is
fast, particularly for buffers, where it can verify 64 bytes a cycle.

- Correct O_LOOP definition on NT
- Introduce program_executable_name
- Add ASAN guards to more system calls
- Improve termios compatibility with BSDs
- Fix bug in Windows auxiliary value encoding
- Add BSD and XNU specific errnos and open flags
- Add check to ensure build doesn't talk to internet
2021-08-22 01:11:53 -07:00
Paul Kulchenko
2730c66f4a
Add GetTime to redbean Lua (#255) 2021-08-22 00:59:47 -07:00
Paul Kulchenko
c506584174
Don't hyphenate delta log lines (#253) 2021-08-19 09:43:01 -07:00
Justine Tunney
ed485391da Reduce memory needed by ASAN morgue 2021-08-19 09:34:24 -07:00
Justine Tunney
d5a37de435 Fix upstream Lua test breakage due to Linenoise 2021-08-19 09:34:24 -07:00
Justine Tunney
65f32fad52 Fix regression in _gclongjmp() 2021-08-19 09:34:24 -07:00
Justine Tunney
da45c7c80b Clean up gdtoa code 2021-08-19 09:34:24 -07:00
Paul Kulchenko
7341336b1a
Improve redbean docs (#249)
- Add Location to ProgramHeader exclusions
- Add ProgramHeader documentation
- Update ProgramRedirect to check location validity
2021-08-19 09:34:50 -07:00
Paul Kulchenko
a63b147a93
Add file/line reporting to redbean LuaLog (#250) 2021-08-19 09:32:33 -07:00
Paul Kulchenko
0ba3617e7f
Add ERRORF macro for kLogError log level (#252) 2021-08-19 04:14:08 -07:00
Justine Tunney
8af197560e Improve Libc by making Python work even better
Actually Portable Python is now outperforming the Python binaries
that come bundled with Linux distros, at things like HTTP serving.
You can now have a fully featured Python install in just one .com
file that runs on six operating systems and is about 10mb in size.
With tuning, the tiniest is ~1mb. We've got most of the libraries
working, including pysqlite, and the repl now feels very pleasant.
The things you can't do quite yet are: threads and shared objects
but that can happen in the future, if the community falls in love
with this project and wants to see it developed further. Changes:

- Add siginterrupt()
- Add sqlite3 to Python
- Add issymlink() helper
- Make GetZipCdir() faster
- Add tgamma() and finite()
- Add legacy function lutimes()
- Add readlink() and realpath()
- Use heap allocations when appropriate
- Reorganize Python into two-stage build
- Save Lua / Python shell history to dotfile
- Integrate Python Lib embedding into linkage
- Make isregularfile() and isdirectory() go faster
- Make Python shell auto-completion work perfectly
- Make crash reports work better if changed directory
- Fix Python+NT open() / access() flag overflow error
- Disable Python tests relating to \N{LONG NAME} syntax
- Have Python REPL copyright() show all notice embeddings

The biggest technical challenge at the moment is working around
when Python tries to be too clever about filenames.
2021-08-18 22:16:23 -07:00
Gautham
98ccbf44b1 Tell _frozen_importlib to consider bytecode first (#248) 2021-08-18 21:58:05 -07:00
Justine Tunney
ebb8c85496 Experiment with making Python go faster
The goal is to put the compiled pyc files in the APE ZIP.
2021-08-18 21:57:11 -07:00
Paul Kulchenko
4486ad5c9e
Add ServeRedirect function to redbean Lua (#246) 2021-08-17 14:26:33 -07:00
Justine Tunney
3d0347e26e Improve deltaify.com program
This program deserves its own project page on http://justine.lol since
it's so good at troubleshooting latency, e.g. `make |& deltaify.com`
2021-08-17 00:06:45 -07:00
Justine Tunney
bc464a8898 Fix a few more Python tests 2021-08-16 23:47:47 -07:00
Justine Tunney
59e1c245d1 Get more Python tests passing (#141) 2021-08-16 15:26:31 -07:00
Paul Kulchenko
916f19eea1
Add GetRandomBytes to redbean Lua (#244) 2021-08-16 12:12:29 -07:00
Justine Tunney
5029e20bef Improve linenoise and get it working on Windows
Some progress has been made on introducing completion but there's been
difficulties using the Python C API to get local shell variables.
2021-08-15 14:34:05 -07:00
Gautham
968474d291
Fix typo bug in strtoll (#245) 2021-08-15 12:07:33 -07:00
Justine Tunney
228fb7428b Improve isystem includes and magic numbers 2021-08-14 23:36:36 -07:00
Justine Tunney
1e5bd4d23e Ues linenoise in Lua, Python, and SQLite 2021-08-14 11:26:23 -07:00
Justine Tunney
fe29710e4e Add linenoise to third party 2021-08-14 11:09:54 -07:00
Justine Tunney
579b597ded Refactor out some duplicated code 2021-08-14 06:17:56 -07:00
Justine Tunney
e963d9c8e3 Add cpu / mem / fsz limits to build system
Thanks to all the refactorings we now have the ability to enforce
reasonable limitations on the amount of resources any individual
compile or test can consume. Those limits are currently:

- `-C 8` seconds of 3.1ghz CPU time
- `-M 256mebibytes` of virtual memory
- `-F 100megabyte` limit on file size

Only one file currently needs to exceed these limits:

    o/$(MODE)/third_party/python/Objects/unicodeobject.o: \
        QUOTA += -C16  # overrides cpu limit to 16 seconds

This change introduces a new sizetol() function to LIBC_FMT for parsing
byte or bit size strings with Si unit suffixes. Functions like atoi()
have been rewritten too.
2021-08-13 23:40:53 -07:00
Justine Tunney
9b29358511 Make whitespace changes
Status lines for Emacs and Vim have been added to Python sources so
they'll be easier to edit using Python's preferred coding style.

Some DNS helper functions have been broken up into multiple files. It's
nice to have one function per file whenever possible, since that way we
don't need -ffunction-sections.  Another reason it's good to have small
source files, is because the build will be enforcing resource limits on
compilation and testing soon.
2021-08-13 03:20:45 -07:00
Justine Tunney
71273bc5c9 Revert explicit Lua GC pass (#241) 2021-08-13 03:06:23 -07:00
Gautham
1aa0df696c
Test changes to Actually Portable Python (#240)
- Add missing `os.pipe` and `os.getuid`
- Commented out _dummy_thread from Lib/threading.py so tests
  don't simulate multi-threading and waste time/error out
- Revert test_hashlib to avoid blake2
2021-08-13 02:24:43 -07:00
Paul Kulchenko
a22f0736d8
Improve Redbean Lua memory (#241)
- Update redbean Lua to check there is enough stack to use
- Add explicit Lua gc pass after each message is processed
- Add Lua memory reporting to redbean
2021-08-13 02:11:49 -07:00
Justine Tunney
b420ed8248 Undiamond Python headers
This change gets the Python codebase into a state where it conforms to
the conventions of this codebase. It's now possible to include headers
from Python, without worrying about ordering. Python has traditionally
solved that problem by "diamonding" everything in Python.h, but that's
problematic since it means any change to any Python header invalidates
all the build artifacts. Lastly it makes tooling not work. Since it is
hard to explain to Emacs when I press C-c C-h to add an import line it
shouldn't add the header that actually defines the symbol, and instead
do follow the nonstandard Python convention.

Progress has been made on letting Python load source code from the zip
executable structure via the standard C library APIs. System calss now
recognizes zip!FILENAME alternative URIs as equivalent to zip:FILENAME
since Python uses colon as its delimiter.

Some progress has been made on embedding the notice license terms into
the Python object code. This is easier said than done since Python has
an extremely complicated ownership story.

- Some termios APIs have been added
- Implement rewinddir() dirstream API
- GetCpuCount() API added to Cosmopolitan Libc
- More bugs in Cosmopolitan Libc have been fixed
- zipobj.com now has flags for mangling the path
- Fixed bug a priori with sendfile() on certain BSDs
- Polyfill F_DUPFD and F_DUPFD_CLOEXEC across platforms
- FIOCLEX / FIONCLEX now polyfilled for fast O_CLOEXEC changes
- APE now supports a hybrid solution to no-self-modify for builds
- Many BSD-only magnums added, e.g. O_SEARCH, O_SHLOCK, SF_NODISKIO
2021-08-12 14:07:40 -07:00
Paul Kulchenko
20bb8db9f8
Fix type signature of new Lua hooks (#239) 2021-08-12 00:15:39 -07:00
Paul Kulchenko
a2e443edd7
Update redbean to show Lua stack traces (#237)
- Update redbean to include stack trace in Lua errors
- Extend Lua in redbean to include stack trace in all logged errors
- Update default error page in redbean with error details (when allowed)
- Prepend `@` to Lua paths in redbean to recognize them as paths in error messages
- Replace GetClientAddr with GetRemoteAddr to avoid backtrace leak in proxy scenarios
- Fix typo in GetRemoteAddr documentation
2021-08-11 23:27:39 -07:00
Gautham
9454788223
Fix sysconfig check for build vars (#238) 2021-08-11 23:21:54 -07:00
Justine Tunney
31a37aeb1a Fix some example code 2021-08-10 13:38:27 -07:00
Justine Tunney
79ba5233db Fix regression caused by state assertions change 2021-08-10 11:36:17 -07:00
Justine Tunney
d26d7ae0e4 Perform build and magnum tuning
Building o//third_party/python now takes 5 seconds on my PC

This change works towards modifying Python to use runtime dispatching
when appropriate. For example, when loading the magnums in the socket
module, it's a good idea to check if the magnum is zero, because that
means the local system platform doesn't support it.
2021-08-10 10:26:13 -07:00
Justine Tunney
ee7e296339 Add state assertions to redbean Lua APIs
Many of the API functions provided by redbean are only appropriate to
call in certain contexts, such as request handling or .init.lua, etc.
For example, Fetch can't be called from the global scope of .init.lua
because SSL hasn't been configured yet. Earlier if this happened then
redbean would crash, which was confusing. What we'll do now is show a
friendly error message. See #97

This change also undocuments redbean ssl compression support since it
seems to be causing a flake in the testing infrastructure.
2021-08-09 15:35:38 -07:00
Justine Tunney
3bfb7580c5 Add Lua compiler
Redbean Lua Server Pages may now be stored in the zip as byte code. This
can improve performance, since redbean currently doesn't cache byte code
but it might be more useful for anyone wanting to create a closed source
redbean. The .lua extension should be used for byte code files. Lua will
tell them apart based on a magic number at the start of the file.

This change also improves some Lua error reporting conditions. See #97
2021-08-09 13:09:14 -07:00
Justine Tunney
b703eee96e Fix obvious Python performance suboptimality 2021-08-09 10:41:06 -07:00
Justine Tunney
5a441ea57f Remove Python gitignore 2021-08-09 09:15:23 -07:00
Justine Tunney
10aade69e3 Remove wildcard from Python build config
It's important for build performance to use := rather than = notation so
that $(wildcard foo/*) isn't a lazily evaluated lambda. In the case of
Python where we need a lot of tuning and excludes, it should help to
spell things out a bit more to just not use wildcard for now.
2021-08-09 08:59:18 -07:00
Justine Tunney
53b9f83e1c Make redbean SSL more tunable
This change enables SSL compression. It significantly reduces the
network load of the testing infrastructure, for free, since this
revision didn't need to change any runit protocol code. However we
turn it off by default in redbean since no browsers support it.

It turns out that some TLSv1.0 clients (e.g. curl command on RHEL5) will
send an SSLv2-style ClientHello. These types of clients are usually ten+
years old and were designed to interop with servers ten years older than
them. Your redbean is now able to interop with these clients even though
redbean doesn't actually support SSLv2 or SSLv3. Please note that the -B
flag may be passed to disable this along with TLSv1.0, TLSv1.1, 3DES, &c

The following Lua APIs have been added to redbean:

  - ProgramSslCompression(bool)
  - ProgramSslCiphersuite(name:str)
  - ProgramSslPresharedKey(key:str,identity:str)

Lastly the DHE ciphersuites have been enabled. IANA recommends DHE and
with old clients like RHEL5 it's the only perfect forward secrecy they
implement.
2021-08-09 07:38:57 -07:00
Justine Tunney
d86027fe90 Integrate Chromium Zlib changes 2021-08-09 06:59:10 -07:00
Justine Tunney
798d542f15 Fix build and delete superfluous files
- Make Python make formatting pristine
- Add missing `#pragma weak` to Python source
- Fix Clang script flake due to missing directory
2021-08-09 06:57:14 -07:00
ahgamut
295b3d6ca5 removed unnecessary libs from zip store 2021-08-09 05:39:42 -07:00