Mathijs Henquet 78203641fe server : Add option to return token pieces in /tokenize endpoint (#9108) ... * server : added with_pieces functionality to /tokenize endpoint * server : Add tokenize with pieces tests to server.feature * Handle case if tokenizer splits along utf8 continuation bytes * Add example of token splitting * Remove trailing ws * Fix trailing ws * Maybe fix ci * maybe this fix windows ci? --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co>		2024-09-12 22:30:11 +02:00
..
bench.yml.disabled	ci : disable bench workflow (#9010)	2024-08-15 10:11:11 +03:00
build.yml	ci : update HIP SDK to 24.Q3 (ROCm 6.1) (#9329)	2024-09-12 14:28:43 +03:00
close-issue.yml	ci : exempt confirmed bugs from being tagged as stale (#7014)	2024-05-01 08:13:59 +03:00
docker.yml	ci : disable rocm image creation (#9340)	2024-09-07 10:48:54 +03:00
editorconfig.yml	ci: exempt master branch workflows from getting cancelled (#6486)	2024-04-04 18:30:53 +02:00
gguf-publish.yml	ci : update checkout, setup-python and upload-artifact to latest (#6456)	2024-04-03 21:01:13 +03:00
labeler.yml	labeler.yml: Use settings from ggerganov/llama.cpp [no ci] (#7363)	2024-05-19 20:51:03 +10:00
nix-ci-aarch64.yml	ci: exempt master branch workflows from getting cancelled (#6486)	2024-04-04 18:30:53 +02:00
nix-ci.yml	ci: exempt master branch workflows from getting cancelled (#6486)	2024-04-04 18:30:53 +02:00
nix-flake-update.yml	ci: nix-flake-update: new token with pr permissions (#4879)	2024-01-11 17:22:34 +00:00
nix-publish-flake.yml	workflows: nix-flakestry: drop tag filters	2023-12-31 13:14:58 -08:00
python-check-requirements.yml	py : fix requirements check '==' -> '~=' (#8982)	2024-08-12 11:02:01 +03:00
python-lint.yml	convert.py : add python logging instead of print() (#6511)	2024-05-03 22:36:41 +03:00
python-type-check.yml	py : type-check all Python scripts with Pyright (#8341)	2024-07-07 15:04:39 -04:00
server.yml	server : Add option to return token pieces in /tokenize endpoint (#9108)	2024-09-12 22:30:11 +02:00