vbatts/llama.cpp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

server : disable tests relying on parallel determinism

Browse source
This commit is contained in:
Georgi Gerganov 2024-05-20 11:17:18 +03:00
parent 204695fd10
commit 457b07f134
No known key found for this signature in database
GPG key ID: BF970631944C16B7
1 changed files with 7 additions and 8 deletions
Download patch file Download diff file Expand all files Collapse all files

15
examples/server/tests/features/results.feature
View file

@ -27,7 +27,8 @@ Feature: Results
Examples: Examples:
| n_slots | | n_slots |
| 1 | | 1 |
| 2 | # FIXME: unified KV cache nondeterminism
# | 2 |
Scenario Outline: different results with different seed Scenario Outline: different results with different seed
Given <n_slots> slots Given <n_slots> slots
@ -73,14 +74,13 @@ Feature: Results
Examples: Examples:
| n_parallel | temp | | n_parallel | temp |
| 1 | 0.0 | | 1 | 0.0 |
| 2 | 0.0 |
| 4 | 0.0 |
| 1 | 1.0 | | 1 | 1.0 |
# FIXME: These tests fail on master. # FIXME: unified KV cache nondeterminism
# Problems: unified KV cache (except for CPU backend with LLAMA_NO_LLAMAFILE=1), SIMD nondeterminism.
# See https://github.com/ggerganov/whisper.cpp/issues/1941#issuecomment-1986923227 # See https://github.com/ggerganov/whisper.cpp/issues/1941#issuecomment-1986923227
# and https://github.com/ggerganov/llama.cpp/pull/6122#discussion_r1531405574 # and https://github.com/ggerganov/llama.cpp/pull/6122#discussion_r1531405574
# and https://github.com/ggerganov/llama.cpp/pull/7347 . # and https://github.com/ggerganov/llama.cpp/pull/7347 .
# | 2 | 0.0 |
# | 4 | 0.0 |
# | 2 | 1.0 | # | 2 | 1.0 |
# | 4 | 1.0 | # | 4 | 1.0 |
@ -108,12 +108,11 @@ Feature: Results
Examples: Examples:
| n_slots | n_kv | n_predict | n_parallel | | n_slots | n_kv | n_predict | n_parallel |
| 4 | 1024 | 1 | 1 | | 4 | 1024 | 1 | 1 |
| 4 | 1024 | 1 | 4 | # FIXME: unified KV cache nondeterminism
# FIXME: These tests fail on master.
# Problems: unified KV cache (except for CPU backend with LLAMA_NO_LLAMAFILE=1), SIMD nondeterminism.
# See https://github.com/ggerganov/whisper.cpp/issues/1941#issuecomment-1986923227 # See https://github.com/ggerganov/whisper.cpp/issues/1941#issuecomment-1986923227
# and https://github.com/ggerganov/llama.cpp/pull/6122#discussion_r1531405574 # and https://github.com/ggerganov/llama.cpp/pull/6122#discussion_r1531405574
# and https://github.com/ggerganov/llama.cpp/pull/7347 . # and https://github.com/ggerganov/llama.cpp/pull/7347 .
# | 4 | 1024 | 1 | 4 |
# | 4 | 1024 | 100 | 1 | # | 4 | 1024 | 100 | 1 |
# This test still fails even the above patches; the first token probabilities are already different. # This test still fails even the above patches; the first token probabilities are already different.
# | 4 | 1024 | 100 | 4 | # | 4 | 1024 | 100 | 4 |