vbatts/llama.cpp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

grammars: update performance gotchas w/ repetition advice

Browse source
This commit is contained in:
ochafik 2024-04-20 01:25:58 +01:00
parent 93b754ec5c
commit 2ecc2ae900
1 changed files with 1 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

2
grammars/README.md
View file

@ -102,4 +102,4 @@ Grammars currently have performance gotchas (see https://github.com/ggerganov/ll
A common pattern is to allow repetitions of a pattern `x` up to N times.
While semantically correct, the syntax `x? x? x?.... x?` (with N repetitions) will result in extremely slow inference. Instead, you can write `(x (x (x ... (x)?...)?)?)?` (w/ N-deep nesting)
While semantically correct, the syntax `x? x? x?.... x?` (with N repetitions) may result in extremely slow sampling. Instead, you can write `x{0,N}` (or `(x (x (x ... (x)?...)?)?)?` w/ N-deep nesting in earlier llama.cpp versions).