vbatts/llama.cpp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

grammars: x{min,max} repetition operator (#6640)

Browse source 
* grammars: x{min,max} repetition operator + tweak +/*/? to avoid duplication of original over alternates

* grammars: handle `x{n}` and fix `x{n,n}`

* grammars: document new repetition operators

* grammars: uniform use of int for min & max

* grammars: refactor parser test

* grammar: parsing tests w/ natural pretty print of updated expectations

* grammars: much prettier print of expectations (+ TEST_GRAMMAR_PARSER_PRINT_ALL=1 to force all)

* grammars: improve test pretty print again

* grammars: pretty print rules and chars

* grammars: fix copy rule skipping

* grammars: disallow `a{,}` (not allowed in regexps)

* Update common/grammar-parser.cpp

Co-authored-by: Clint Herron <hanclinto@gmail.com>

* grammars: fix copy rule skipping (again) & display of expectations

* grammars: more test cases

* grammars: update reps parsing to bring ? / * / + closer to before

* json: use new GBNF repetitions{m,n} syntax

* grammars: update performance gotchas w/ repetition advice

* Update examples/json_schema_to_grammar.py

Co-authored-by: Clint Herron <hanclinto@gmail.com>

* Update examples/server/public/json-schema-to-grammar.mjs

Co-authored-by: Clint Herron <hanclinto@gmail.com>

* grammars: comment on rule repetitions

* grammars: ensure unambiguous number alternatives

* grammar: nit typo switched error msgs

* grammar: nit numbering in comment

* json: update numeric rule to be unambiguous

* Apply suggestions from code review

Co-authored-by: Clint Herron <hanclinto@gmail.com>

* Update examples/server/public/json-schema-to-grammar.mjs

Co-authored-by: Clint Herron <hanclinto@gmail.com>

* json: fix integral-part

* grammar: add repetition tests

---------

Co-authored-by: Clint Herron <hanclinto@gmail.com>
This commit is contained in:
Olivier Chafik 2024-06-06 10:07:06 +01:00 • committed by GitHub
parent f5d7b268ec
commit 55b2d0849d
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
9 changed files with 736 additions and 418 deletions
Download patch file Download diff file Expand all files Collapse all files

76
tests/test-grammar-integration.cpp
View file

@ -292,6 +292,82 @@ static void test_quantifiers() {
"catyyy",
}
);
test_grammar(
"simple exact repetition",
// Grammar
R"""(
root ::= [ab]{4}
)""",
// Passing strings
{
"aaaa",
"bbbb",
"abab",
},
// Failing strings
{
"a",
"b",
"aaaaa",
}
);
test_grammar(
"simple min repetition",
// Grammar
R"""(
root ::= [ab]{4,}
)""",
// Passing strings
{
"aaaa",
"aaaaab",
"bbbb",
"ababab",
},
// Failing strings
{
"",
"aba",
}
);
test_grammar(
"simple max repetition",
// Grammar
R"""(
root ::= [ab]{0,4}
)""",
// Passing strings
{
"",
"a",
"aa",
"aaa",
"aaab",
},
// Failing strings
{
"aaaaa",
}
);
test_grammar(
"min / max repetition",
// Grammar
R"""(
root ::= ("0x" [A-F0-9]{2} " "?){3,5}
)""",
// Passing strings
{
"0xFF 0x12 0xAB",
"0xFF 0x12 0xAB 0x00 0x00",
},
// Failing strings
{
"",
"0xFF",
"0xFF 0x12",
"0xFF 0x12 0xAB 0x00 0x00 0x00",
}
);
}
static void test_failure_missing_root() {