diff --git a/common/chaton.hpp b/common/chaton.hpp index a14f3351f..6bd8250de 100644 --- a/common/chaton.hpp +++ b/common/chaton.hpp @@ -56,12 +56,16 @@ * * however in many of these models, the subsequent user messages will have the * BeginOfSentenceTag and or RolePrefixTag. * + * * Some models may require a BoS for a group of messages, independent of BoS (if any) + * wrt individual roles. + * * * ## The Strategy * * The template meta data json file allows the user to specify the above mentioned tags wrt - * each of the Role. Depending on whether a given model uses a given tag or not you either - * specify the required tag or else you specify a empty string. + * each of the Role as well as any global tag for a group of messages. Depending on whether + * a given model uses/needs a given tag or not you either specify the required tag or else + * you specify a empty string. * * A tag could be a single word or multiple words, and may include newline char specified * using \n and so on. The tag is always demarcated using double quotes and thus also allows @@ -80,6 +84,8 @@ * the assistant's suffix or end tag or to the user's begin or prefix tag, depending on what * is generated by the model at the end of its response. * + * Currently flags for trimming wrt user text (be it wrt system or user role) is not added. + * * * ## The JSON File * @@ -149,20 +155,38 @@ * These always adds any role specific begin+prefix and suffix+end around * the passed message. * + * ## other uses be it wrt llama.cpp-as-library or examples/server or ... + * + * This module exposes a c-api which is equivalent to the current hardcoded + * templating logic's llama_chat_apply_template. So any program using llama.cpp's + * chat templating logic can be easily migrated to make use of this generic code + * with text based config file based flow. + * + * If a program doesnt want to bring in json dependency into their project, + * there is also common/simpcfg.hpp, which provides a simple text based config + * file format, along with the corresponding parser for the same. This can be + * modified to work with simpcfg easily, if needed. * * ## Adding support for new model / chat-handshake-template-standard * * 1. Add suitable entries in json for that model/standard - * 2. Try to reuse the generic flow in chaton-tmpl-apply, as much as possible, - * before trying to add a custom logic. + * This in itself should work for most of the models. + * + * 2. If some new model introduces a totally different kind of chat-templating + * tag inter/intra mixing, Try to reuse and update the generic flow in + * chaton-tmpl-apply, as much as possible, before trying to add any custom logic. + * * If you update the generic flow, cross check if existing json files will * need to be updated or not. * * * ## Notes * - * Look at the sample chaton_meta.json in examples folder for how the above may apply - * * llama2, llama3, gemma, chatml, zephyr, deepseek(normal and coder), monarch, mistral + * Look at the sample chaton_meta.json in examples folder for how the above may apply to + * the different llm's out there like + * + * * llama2, llama3, gemma, zephyr, deepseek(normal and coder), monarch, mistral, phi3 + * * chatml, command-r, orion, openchat, vicuna * */