vbatts/llama.cpp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

server: metrics: add llamacpp:prompt_seconds_total and llamacpp:tokens_predicted_seconds_total, reset bucket only on /metrics. Fix values cast to int. Add Process-Start-Time-Unix header. (#5937)

Browse source 
Closes #5850
This commit is contained in:
Pierrick Hymbert 2024-03-08 12:25:04 +01:00 • committed by GitHub
parent e457fb3540
commit 76e868821a
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
3 changed files with 46 additions and 13 deletions
Download patch file Download diff file Expand all files Collapse all files

1
examples/server/tests/features/server.feature
View file

@ -29,6 +29,7 @@ Feature: llama.cpp server
And a completion request with no api error
Then <n_predicted> tokens are predicted matching <re_content>
And prometheus metrics are exposed
And metric llamacpp:tokens_predicted is <n_predicted>
Examples: Prompts
| prompt | n_predict | re_content | n_predicted |