minor : comment + spacing

This commit is contained in:
Georgi Gerganov 2023-10-17 18:08:57 +03:00 committed by GitHub
parent 1c28116de4
commit fc82541b1d
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
2 changed files with 2 additions and 1 deletions

View file

@ -516,6 +516,7 @@ extern "C" {
/// @return Returns the number of tokens on success, no more than n_max_tokens
/// @return Returns a negative number on failure - the number of tokens that would have been returned
/// @param special Allow tokenizing special and/or control tokens which otherwise are not exposed and treated as plaintext.
/// Does not insert a leading space.
LLAMA_API int llama_tokenize(
const struct llama_model * model,
const char * text,