llama : move vocab, grammar and sampling into separate files (#8508)

Browse source

* llama : move sampling code into llama-sampling

ggml-ci

* llama : move grammar code into llama-grammar

ggml-ci

* cont

ggml-ci

* cont : pre-fetch rules

* cont

ggml-ci

* llama : deprecate llama_sample_grammar

* llama : move tokenizers into llama-vocab

ggml-ci

* make : update llama.cpp deps [no ci]

* llama : redirect external API to internal APIs

ggml-ci

* llama : suffix the internal APIs with "_impl"

ggml-ci

* llama : clean-up

...

This commit is contained in:

Georgi Gerganov 2024-07-23 13:10:17 +03:00 • committed by GitHub

parent 751fcfc6c3

commit 938943cdbf

No known key found for this signature in database

GPG key ID: B5690EEEBB952194

18 changed files with 3521 additions and 2968 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download patch file Download diff file Expand all files Collapse all files

1721

src/llama-vocab.cpp Normal file

View file

File diff suppressed because it is too large Load diff