Commit graph

3 commits

Author SHA1 Message Date
Francis Couture-Harpin
698f0b3479 convert-hf : remove unused n_dims in extra_*_tensors 2024-04-30 15:02:34 -04:00
Francis Couture-Harpin
47e02eb7bc convert-hf : begin refactoring write_tensor 2024-04-30 14:07:28 -04:00
Sigbjørn Skjæret
03c0946d73
convert : support models with multiple chat templates (#6588)
* Support converting models with multiple chat templates

Adds the following metadata:
* tokenizer.chat_templates
* tokenizer.chat_template.<name1>
* tokenizer.chat_template.<name2>
* tokenizer.chat_template.<...>

Where `tokenizer.chat_templates` is an array of the template names (except `default`), `default` is added to the regular `tokenizer.chat_template`.

* replace filtered characters with underscore

* New script to add/modify/remove metadata

This scripts creates a copy of a GGUF file and allows you to add/modify/remove metadata in the process.

Most importantly this allows you to update chat templates, either as a string or directly from an updated tokenizer_config.json file.

* Add files via upload

add new script to project/readme

* flake--
2024-04-18 14:49:01 +03:00