vbatts/llama.cpp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

server : fix logprobs, make it OAI-compatible (#10783)

Browse source 
* server : fix logprobs, make it openai-compatible

* update docs

* add std::log

* return pre-sampling p

* sort before apply softmax

* add comment

* fix test

* set p for sampled token

* update docs

* add --multi-token-probs

* update docs

* add `post_sampling_probs` option

* update docs [no ci]

* remove --multi-token-probs

* "top_probs" with "post_sampling_probs"

* resolve review comments

* rename struct token_prob to prob_info

* correct comment placement

* fix setting prob for sampled token
This commit is contained in:
Xuan Son Nguyen 2024-12-19 15:40:08 +01:00 • committed by GitHub
parent a3c33b1dce
commit 57bb2c40cd
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
6 changed files with 396 additions and 107 deletions
Download patch file Download diff file Expand all files Collapse all files

3
examples/server/tests/unit/test_embedding.py
View file

@ -50,6 +50,8 @@ def test_embedding_multiple():
@pytest.mark.parametrize(
"input,is_multi_prompt",
[
# do not crash on empty input
("", False),
# single prompt
("string", False),
([12, 34, 56], False),
@ -103,6 +105,7 @@ def test_embedding_pooling_none_oai():
# /v1/embeddings does not support pooling type 'none'
assert res.status_code == 400
assert "error" in res.body
def test_embedding_openai_library_single():