d6bd4d46dd
* llama : support StableLM 2 1.6B
* convert : fix Qwen's set_vocab wrongly naming all special tokens [PAD{id}]
* convert : refactor Qwen's set_vocab to use it for StableLM 2 too
* nix : add tiktoken to llama-python-extra
* convert : use presence of tokenizer.json to determine StableLM tokenizer loader
It's a less arbitrary heuristic than the vocab size.
|
||
|---|---|---|
| .. | ||
| nix | ||
| cloud-v-pipeline | ||
| full-cuda.Dockerfile | ||
| full-rocm.Dockerfile | ||
| full.Dockerfile | ||
| llama-cpp-clblast.srpm.spec | ||
| llama-cpp-cublas.srpm.spec | ||
| llama-cpp.srpm.spec | ||
| main-cuda.Dockerfile | ||
| main-rocm.Dockerfile | ||
| main.Dockerfile | ||
| tools.sh | ||