The C standard specifies that, upon handling the a conversion specifier,
the argument is converted to a string in which "there is one hexadecimal
digit (which is nonzero [...]) before the decimal-point character", this
being a requirement which cosmopolitan does not currently always handle,
sometimes printing numbers like "0x0.1p+5", where a correct output would
have been e.g. "0x1.0p+1" (despite both representing the same value, the
first one illegally has a '0' digit before the decimal-point character).
The C standard indicates that when processing the a conversion specifier
"if the precision is zero *and* the # flag is not specified, no decimal-
point character appears.". This means that __fmt needs to ensure that it
prints the decimal-point character not only when the precision is non-0,
but also when the # flag is specified - cosmopolitan currently does not.
This patch fixes this, along with adding a few tests for this behaviour.
The a conversion specifier to printf had some issues w.r.t. rounding, in
particular in edge cases w.r.t. "to nearest, ties to even" rounding (for
instance, "%.1a" with 0x1.78p+4 outputted 0x1.7p+4 instead of 0x1.8p+4).
This patch fixes this and adds several tests w.r.t ties to even rounding
POSIX specifies the <apostrophe> flag character for printf as formatting
decimal conversions with the thousands' grouping characters specified by
the current locale. Given that cosmopolitan currently has no support for
obtaining the locale's grouping character, all that is required (when in
the C/POSIX locale) for supporting this flag is ignoring it, and as it's
already used to indicate quoting (for non-decimal conversions), all that
has to be done is to avoid having it be an alias for the <space> flag so
that decimal conversions don't accidentally behave as though the <space>
flag has also been specified whenever the <apostrophe> flag is utilized.
This patch adds this flag, as described above, along with a test for it.
When reading hexadecimal floats, cosmopolitan would previously sometimes
print a number of warnings relating to undefined behavior on left shift:
third_party/gdtoa/gethex.c:172: ubsan warning: signed left shift changed
sign bit or overflowed 12 'int' 28 'int' is undefined behavior
This is because gdtoa assumes left shifts are safe when overflow happens
even on signed integers - this is false: the C standard considers it UB.
This is easy to fix, by simply casting the shifted value to unsigned, as
doing so does not change the value or the semantics of the left shifting
(except for avoiding the undefined behavior, as the C standard specifies
that unsigned overflow yields wraparound, avoiding undefined behaviour).
This commit does this, and adds a testcase that previously triggered UB.
(this also adds test macros to test for exact float equality, instead of
the existing {EXPECT,ASSERT}_FLOAT_EQ macros which only tests inputs for
being "almost equal" (with a significant epsilon) whereas exact equality
makes more sense for certain things such as reading floats from strings,
and modifies other testcases for sscanf/fscanf of floats to utilize it).
Hexadecimal printing of floating-point numbers in cosmopolitan (that is,
using the the conversion specifier) is improved to have correct rounding
of results in rounding modes other than the default one (ie. FE_NEAREST)
This commit fixes that, and adds tests for the change (note that there's
still some rounding issues with the a conversion specifier in general in
relatively rare cases (that is without non-default rounding modes) where
I've left commented-out tests for anyone interested in improving it more
Currently, in cosmopolitan, there is no handling of the current rounding
mode for long double conversions, such that round-to-nearest gets always
used, regardless of the current rounding mode. %Le also improperly calls
gdtoa with a too small precision (which led to relatively similar bugs).
This patch fixes these issues, in particular by modifying the FPI object
passed to gdtoa such that it is modifiable (so that __fmt can adjust its
rounding field to correspond to FLT_ROUNDS (note that this is not needed
for dtoa, which checks FLT_ROUNDS directly)) and ors STRTOG_Neg into the
kind field in both of the __fmt_dfpbits and __fmt_ldfpbits functions, as
the gdtoa function also depends on it to be able to accurately round any
negative arguments. The change to kind also requires a few other changes
to make sure kind's upper bits (which include STRTOG_Neg) are masked off
when attempting to only examine the lower bits' value. Furthermore, this
patch also makes exactly one change in gdtoa, which appears to be needed
to fix rounding issues with FE_TOWARDZERO (this seems like a gdtoa bug).
The patch also adds a few tests for these issues, along with also taking
the opportunity to clean up some of the previous tests to do the asserts
in the right order (i.e. with the first argument as the expected result,
and the second one being used as the value that it is compared against).
Before this commit, cosmopolitan had some issues with handling arguments
of 0 and signs, such as returning an incorrect sign when the input value
== -0.0, and incorrectly handling ndigit == 0 on fcvt (ndigit determines
the amount of digits *after* the radix character on fcvt, thus the parts
before it still must be outputted before fcvt's job is completely done).
This patch fixes these issues, and adds tests with corresponding inputs.
The C Standard specifies that, when a conversion specification specifies
a conversion specifier of n, the type of the passed pointer is specified
by the length modifier (if any), i.e. that e.g. the argument for %hhn is
of type signed char *, but Cosmopolitan currently does not handle this -
instead always simply assuming that the pointer always points to an int.
This patch implements, and tests, length modifiers with the n conversion
specifier, with the tests testing all of the available length modifiers.
POSIX specifies the C conversion specifier as being "equivalent to %lc",
i.e. printf("%C", arg) is equivalent in behaviour to printf("%lc", arg).
This patch implements this conversion specifier, and adds a test for it,
alongside another test, which ensures that va_arg uses the correct size,
even though we set signbit to 63 in the code (which one might think will
result in the wrong size of argument being va_arg-ed, but having signbit
set to 63 is in fact what __fmt_stoa expects and is a requirement for it
properly formatting the wchar_t argument - this does not result in wrong
usage of va_arg because the implementation of the c conversion specifier
(which the implementation of the C conversion specifier fallsthrough to)
always calls va_arg with an argument type of int, to avoid the very same
bug occuring with %lc, as the l length modifier also sets signbit to 63)
poll() and select() now delegate to ppoll() and pselect() for assurances
that both polyfill implementations are correct and well-tested. Poll now
polyfills XNU and BSD quirks re: the hanndling of POLLNVAL and the other
similar status flags. This change resolves a misunderstanding concerning
how select(exceptfds) is intended to map to POLPRI. We now use E2BIG for
bouncing requests that exceed the 64 handle limit on Windows. With pipes
and consoles on Windows our poll impl will now report POLLHUP correctly.
Issues with Windows path generation have been fixed. For example, it was
problematic on Windows to say: posix_spawn_file_actions_addchdir_np("/")
due to the need to un-UNC paths in some additional places. Calling fstat
on UNC style volume path handles will now work. posix_spawn now supports
simulating the opening of /dev/null and other special paths on Windows.
Cosmopolitan no longer defines epoll(). I think wepoll is a nice project
for using epoll() on Windows socket handles. However we need generalized
file descriptor support to make epoll() for Windows work well enough for
inclusion in a C library. It's also not worth having epoll() if we can't
get it to work on XNU and BSD OSes which provide different abstractions.
Even epoll() on Linux isn't that great of an abstraction since it's full
of footguns. Last time I tried to get it to be useful I had little luck.
Considering how long it took to get poll() and select() to be consistent
across platforms, we really have no business claiming to have epoll too.
While it'd be nice to have fully implemented, the only software that use
epoll() are event i/o libraries used by things like nodejs. Event i/o is
not the best paradigm for handling i/o; threads make so much more sense.
Cosmopolitan's printf-family functions will currently crash if one tries
formatting a floating point number with a larger precision (large enough
that gdtoa attempts to allocate memory to format the number) while under
memory pressure (i.e. when malloc fails) because gdtoa fails to check if
malloc fails.
The added tests (which would previously crash under cosmopolitan without
this patch) show how to reproduce the issue.
This patch fixes this, and adds the aforementioned tests.
Cosmopolitan's printf-family functions currently very poorly handle
being passed a long double infinity.
For instance, a program such as:
```cpp
#include <stdio.h>
int main()
{
printf("%f\n", 1.0 / 0.0);
printf("%Lf\n", 1.0L / 0.0L);
printf("%e\n", 1.0 / 0.0);
printf("%Le\n", 1.0L / 0.0L);
printf("%g\n", 1.0 / 0.0);
printf("%Lg\n", 1.0L / 0.0L);
}
```
will currently output the following:
```
inf
0.000000[followed by 32763 more zeros]
inf
N.aN0000e-32769
inf
N.aNe-32769
```
when the correct expected output would be:
```
inf
inf
inf
inf
inf
inf
```
This patch fixes this, and adds tests for the behavior.
This change introduces accumulate, addressof, advance, all_of, distance,
array, enable_if, allocator_traits, back_inserter, bad_alloc, is_signed,
any_of, copy, exception, fill, fill_n, is_same, is_same_v, out_of_range,
lexicographical_compare, is_integral, uninitialized_fill_n, is_unsigned,
numeric_limits, uninitialized_fill, iterator_traits, move_backward, min,
max, iterator_tag, move_iterator, reverse_iterator, uninitialized_move_n
This change experiments with rewriting the ctl::vector class to make the
CTL design more similar to the STL. So far it has not slowed things down
to have 42 #include lines rather than 2, since it's still almost nothing
compared to LLVM's code. In fact the closer we can flirt with being just
like libcxx, the better chance we might have of discovering exactly what
makes it so slow to compile. It would be an enormous discovery if we can
find one simple trick to solving the issue there instead.
This also fixes a bug in `ctl::string(const string &s)` when `s` is big.
It hasn't been helpful enough to be justify the maintenance burden. What
actually does help is mprotect(), kprintf(), --ftrace and --strace which
can always be counted upon to work correctly. We aren't losing much with
this change. Support for ASAN on AARCH64 was never implemented. Applying
ASAN to the core libc runtimes was disabled many months ago. If there is
some way to have an ASAN runtime for user programs that is less invasive
we can potentially consider reintroducing support. But now is premature.
Cosmopolitan now supports 104 time zones. They're embedded inside any
binary that links the localtime() function. Doing so adds about 100kb
to the binary size. This change also gets time zones working properly
on Windows for the first time. It's not needed to have /etc/localtime
exist on Windows, since we can get this information from WIN32. We're
also now updated to the latest version of Paul Eggert's TZ library.
Signals are extremely difficult to unit test reliably. This is why
functions like sigsuspend() exist. When testing something else and
portably it becomes impossible without access to kernel internals.
OpenMP flakes in QEMU on one of my workstations. I don't think the
support is production worthy, because there's been issues on MacOS
additionally. It works great for every experiment I've used it for
though. However a flaky test is worse than no test at all. So it's
removed until someone takes an interest in productionizing it.
Commit bc6c183 introduced a bunch of discrepancies between what files
look like in the repo and what clang-format says they should look like.
However, there were already a few discrepancies prior to that. Most of
these discrepancies seemed to be unintentional, but a few of them were
load-bearing (e.g., a #include that violated header ordering needing
something to have been #defined by a 'later' #include.)
I opted to take what I hope is a relatively smooth-brained approach: I
reverted the .clang-format change, ran clang-format on the whole repo,
reapplied the .clang-format change, reran clang-format again, and then
reverted the commit that contained the first run. Thus the full effect
of this PR should only be to apply the changed formatting rules to the
repo, and from skimming the results, this seems to be the case.
My work can be checked by applying the short, manual commits, and then
rerunning the command listed in the autogenerated commits (those whose
messages I have prefixed auto:) and seeing if your results agree.
It might be that the other diffs should be fixed at some point but I'm
leaving that aside for now.
fd '\.c(c|pp)?$' --print0| xargs -0 clang-format -i
The WIN32 CreateProcess() function does not require an .exe or .com
suffix in order to spawn an executable. Now that we have Cosmo bash
we're no longer so dependent on the cmd.exe prompt.
* Fix reading the same symbol twice when using `{f,}scanf()`
PR #924 appears to use `unget()` subtly incorrectly when parsing
floating point numbers. The rest of the code only uses `unget()`
immediately followed by `goto Done;` to return back the symbol that
can't possibly belong to the directive we're processing.
With floating-point, however, the ungot characters could very well
be valid for the *next* directive, so we will essentially read them
twice. It can't be seen in `sscanf()` tests because `unget()` is a
no-op there, but the test I added for `fscanf()` fails like this:
...
EXPECT_EQ(0xDEAD, i1)
need 57005 (or 0xdead) =
got 908973 (or 0x000ddead)
...
EXPECT_EQ(0xBEEF, i2)
need 48879 (or 0xbeef) =
got 769775 (or 0x000bbeef)
This means we read 0xDDEAD instead of 0xDEAD and 0xBBEEF instead of
0xBEEF. I checked that both musl and glibc read 0xDEAD/0xBEEF, as
expected.
Fix the failing test by removing the unneeded `unget()` calls.
* Don't read invalid floating-point numbers in `*scanf()`
Currently, we just ignore any errors from `strtod()`. They can
happen either because no valid float can be parsed at all, or
because the state machine recognizes only a prefix of a valid
floating-point number.
Fix this by making sure `strtod()` parses everything we recognized,
provided it's non-empty. This requires to pop the last character
off the FP buffer, which is supposed to be parsed by the next
`*scanf()` directive.
* Make `%c` parsing in `*scanf()` respect the C standard
Currently, `%c`-style directives always succeed even if there
are actually fewer characters in the input than requested.
Before the fix, the added test fails like this:
...
EXPECT_EQ(2, sscanf("ab", "%c %c %c", &c2, &c3, &c4))
need 2 (or 0x02 or '\2' or ENOENT) =
got 3 (or 0x03 or '\3' or ESRCH)
...
EXPECT_EQ(0, sscanf("abcd", "%5c", s2))
need 0 (or 0x0 or '\0') =
got 1 (or 0x01 or '\1' or EPERM)
musl and glibc pass this test.
Renaming gc() to _gc() was a mistake since the better thing to do is put
it behind the _COSMO_SOURCE macro. We need this change because I haven't
wanted to use my amazing garbage collector ever since we renamed it. You
now need to define _COSMO_SOURCE yourself when using amalgamation header
and cosmocc users need to pass the -mcosmo flag to get the gc() function
Some other issues relating to cancelation have been fixed along the way.
We're also now putting cosmocc in a folder named `.cosmocc` so it can be
more safely excluded by grep --exclude-dir=.cosmocc --exclude-dir=o etc.
Somehow or another, I previously had missed `BUILD.mk` files.
In the process I found a few straggler cases where the modeline was
different from the file, including one very involved manual fix where a
file had been treated like it was ts=2 and ts=8 on separate occasions.
The commit history in the PR shows the gory details; the BUILD.mk was
automated, everything else was mostly manual.
At least in neovim, `│vi:` is not recognized as a modeline because it
has no preceding whitespace. After fixing this, opening a file yields
an error because `net` is not an option. (`noet`, however, is.)
- Use good ELF technique in cosmo_dlopen()
- Make strerror() conform more to other libc impls
- Introduce __clear_cache() and use it in cosmo_dlopen()
- Remove libc/fmt/fmt.h header (trying to kill off LIBC_FMT)
Our makefile generator now accepts badly formatted include lines. It's
now more hermetic with better error checking in the cosmo repo, and it
can be configured to not be hermetic at all.
- On Windows connect() can now be interrupted by a signal; connect() w/
O_NONBLOCK will now raise EINPROGRESS; and connect() with SO_SNDTIMEO
will raise ETIMEDOUT after the interval has elapsed.
- We now get the AcceptEx(), ConnectEx(), and TransmitFile() functions
from the WIN32 API the officially blessed way, using WSAIoctl().
- Do nothing on Windows when fsync() is called on a directory handle.
This was raising EACCES earlier becaues GENERIC_WRITE is required on
the handle. It's possible to FlushFileBuffers() a directory handle if
it's opened with write access but MSDN doesn't document what it does.
If you have any idea, please let us know!
- Prefer manual reset event objects for read() and write() on Windows.
- Do some code cleanup on our dlmalloc customizations.
- Fix errno type error in Windows blocking routines.
- Make the futex polyfill simpler and faster.
This change deletes mkfifo() so that GNU Make on Windows will work in
parallel mode using its pipe-based implementation. There's an example
called greenbean2 now, which shows how to build a scalable web server
for Windows with 10k+ threads. The accuracy of clock_nanosleep is now
significantly improved on Linux.
- Every unit test now passes on Apple Silicon. The final piece of this
puzzle was porting our POSIX threads cancelation support, since that
works differently on ARM64 XNU vs. AMD64. Our semaphore support on
Apple Silicon is also superior now compared to AMD64, thanks to the
grand central dispatch library which lets *NSYNC locks go faster.
- The Cosmopolitan runtime is now more stable, particularly on Windows.
To do this, thread local storage is mandatory at all runtime levels,
and the innermost packages of the C library is no longer being built
using ASAN. TLS is being bootstrapped with a 128-byte TIB during the
process startup phase, and then later on the runtime re-allocates it
either statically or dynamically to support code using _Thread_local.
fork() and execve() now do a better job cooperating with threads. We
can now check how much stack memory is left in the process or thread
when functions like kprintf() / execve() etc. call alloca(), so that
ENOMEM can be raised, reduce a buffer size, or just print a warning.
- POSIX signal emulation is now implemented the same way kernels do it
with pthread_kill() and raise(). Any thread can interrupt any other
thread, regardless of what it's doing. If it's blocked on read/write
then the killer thread will cancel its i/o operation so that EINTR can
be returned in the mark thread immediately. If it's doing a tight CPU
bound operation, then that's also interrupted by the signal delivery.
Signal delivery works now by suspending a thread and pushing context
data structures onto its stack, and redirecting its execution to a
trampoline function, which calls SetThreadContext(GetCurrentThread())
when it's done.
- We're now doing a better job managing locks and handles. On NetBSD we
now close semaphore file descriptors in forked children. Semaphores on
Windows can now be canceled immediately, which means mutexes/condition
variables will now go faster. Apple Silicon semaphores can be canceled
too. We're now using Apple's pthread_yield() funciton. Apple _nocancel
syscalls are now used on XNU when appropriate to ensure pthread_cancel
requests aren't lost. The MbedTLS library has been updated to support
POSIX thread cancelations. See tool/build/runitd.c for an example of
how it can be used for production multi-threaded tls servers. Handles
on Windows now leak less often across processes. All i/o operations on
Windows are now overlapped, which means file pointers can no longer be
inherited across dup() and fork() for the time being.
- We now spawn a thread on Windows to deliver SIGCHLD and wakeup wait4()
which means, for example, that posix_spawn() now goes 3x faster. POSIX
spawn is also now more correct. Like Musl, it's now able to report the
failure code of execve() via a pipe although our approach favors using
shared memory to do that on systems that have a true vfork() function.
- We now spawn a thread to deliver SIGALRM to threads when setitimer()
is used. This enables the most precise wakeups the OS makes possible.
- The Cosmopolitan runtime now uses less memory. On NetBSD for example,
it turned out the kernel would actually commit the PT_GNU_STACK size
which caused RSS to be 6mb for every process. Now it's down to ~4kb.
On Apple Silicon, we reduce the mandatory upstream thread size to the
smallest possible size to reduce the memory overhead of Cosmo threads.
The examples directory has a program called greenbean which can spawn
a web server on Linux with 10,000 worker threads and have the memory
usage of the process be ~77mb. The 1024 byte overhead of POSIX-style
thread-local storage is now optional; it won't be allocated until the
pthread_setspecific/getspecific functions are called. On Windows, the
threads that get spawned which are internal to the libc implementation
use reserve rather than commit memory, which shaves a few hundred kb.
- sigaltstack() is now supported on Windows, however it's currently not
able to be used to handle stack overflows, since crash signals are
still generated by WIN32. However the crash handler will still switch
to the alt stack, which is helpful in environments with tiny threads.
- Test binaries are now smaller. Many of the mandatory dependencies of
the test runner have been removed. This ensures many programs can do a
better job only linking the the thing they're testing. This caused the
test binaries for LIBC_FMT for example, to decrease from 200kb to 50kb
- long double is no longer used in the implementation details of libc,
except in the APIs that define it. The old code that used long double
for time (instead of struct timespec) has now been thoroughly removed.
- ShowCrashReports() is now much tinier in MODE=tiny. Instead of doing
backtraces itself, it'll just print a command you can run on the shell
using our new `cosmoaddr2line` program to view the backtrace.
- Crash report signal handling now works in a much better way. Instead
of terminating the process, it now relies on SA_RESETHAND so that the
default SIG_IGN behavior can terminate the process if necessary.
- Our pledge() functionality has now been fully ported to AARCH64 Linux.
