Georgi Gerganov
af05268cdd
unicode : cleanup
2024-04-29 11:20:42 +03:00
Georgi Gerganov
c68d2596ea
tests : add more vocabs and tests
...
ggml-ci
2024-04-29 11:09:17 +03:00
Georgi Gerganov
43708d22c3
tests : refactor vocab tests
...
ggml-ci
2024-04-29 10:46:43 +03:00
Georgi Gerganov
ef4cca9e87
cmake : refactor test targets
2024-04-29 09:53:49 +03:00
Georgi Gerganov
7b1210f6a8
lint : fix
2024-04-28 22:51:13 +03:00
Georgi Gerganov
78081502e9
convert : exercise contractions
...
ggml-ci
2024-04-28 22:18:20 +03:00
Georgi Gerganov
0f9058ceec
convert : add comments
2024-04-28 22:10:04 +03:00
Georgi Gerganov
02fd977fe1
convert : remove unused functions
2024-04-28 22:03:21 +03:00
Georgi Gerganov
e8dd4a1494
lint : fix
2024-04-28 22:02:10 +03:00
Georgi Gerganov
491f2339bb
lint : fix
2024-04-28 21:42:58 +03:00
Georgi Gerganov
1545550ec2
unicode : normalize signatures
2024-04-28 21:40:36 +03:00
Georgi Gerganov
1c888eb4da
convert : add falcon
...
ggml-ci
2024-04-28 21:26:40 +03:00
Georgi Gerganov
4e3e6d8ecc
lint : update
2024-04-28 21:16:50 +03:00
Georgi Gerganov
7642973616
convert : add convert-hf-to-gguf-update.py
...
ggml-ci
2024-04-28 20:52:31 +03:00
Georgi Gerganov
ee6d1b3fb4
unicode : simplify
2024-04-28 18:36:57 +03:00
Georgi Gerganov
e972e6cbf8
unicode : clean-up
2024-04-28 18:30:37 +03:00
Georgi Gerganov
d63cc9068b
Merge branch 'master' into gg/bpe-preprocess
...
ggml-ci
2024-04-28 15:34:45 +03:00
Georgi Gerganov
b97add52a4
unicode : category support via std::regex
2024-04-28 15:15:57 +03:00
github-actions[bot]
6e472f58e4
flake.lock: Update
...
Flake lock file updates:
• Updated input 'nixpkgs':
'github:NixOS/nixpkgs/5c24cf2f0a12ad855f444c30b2421d044120c66f?narHash=sha256-XtTSSIB2DA6tOv%2Bl0FhvfDMiyCmhoRbNB%2B0SeInZkbk%3D' (2024-04-19)
→ 'github:NixOS/nixpkgs/7bb2ccd8cdc44c91edba16c48d2c8f331fb3d856?narHash=sha256-Drmja/f5MRHZCskS6mvzFqxEaZMeciScCTFxWVLqWEY%3D' (2024-04-25)
2024-04-28 11:12:50 +00:00
mgroeber9110
4dba7e8114
Replace "alternative" boolean operator in conditional compilation directive ( #6949 )
2024-04-27 21:02:06 +02:00
Pierrick Hymbert
b7368332e2
ci: server: tests python env on github container ubuntu latest / fix n_predict ( #6935 )
...
* ci: server: fix python env
* ci: server: fix server tests after #6638
* ci: server: fix windows is not building PR branch
2024-04-27 17:50:48 +02:00
Georgi Gerganov
581c4a0239
unicode : try fix windows
2024-04-27 18:36:00 +03:00
Georgi Gerganov
91eaa414bf
unicode : support \p{N}, \p{L} and \p{P} natively
2024-04-27 17:48:38 +03:00
Georgi Gerganov
ce5485aee0
unicode : always use std::wregex
2024-04-27 17:11:34 +03:00
Georgi Gerganov
2affd0b221
unicode : set bomb
2024-04-27 11:56:02 +03:00
Georgi Gerganov
a22645c2a7
unicode : set bomb
2024-04-27 11:48:24 +03:00
Georgi Gerganov
4434c9d6c2
minor
2024-04-27 11:33:16 +03:00
Georgi Gerganov
ad929833cb
llama : adapt punctuation regex + add llama 3 regex
2024-04-27 11:06:08 +03:00
Georgi Gerganov
96965f67e6
models : add llama v3 vocab file
2024-04-27 11:05:12 +03:00
Georgi Gerganov
c160818ec0
wip
2024-04-27 00:28:36 +03:00
Georgi Gerganov
a774d7084e
make : add test-tokenizer-0-llama-v3
2024-04-26 21:25:36 +03:00
Georgi Gerganov
8791e94e3c
lint : fix
2024-04-26 21:12:05 +03:00
agray3
928e0b7013
Reset schedule earlier to allow overlap with ggml graph computation on device ( #6933 )
...
* Reset schedule earlier to allow overlap with graph computation on device
2024-04-26 20:08:30 +02:00
Pierrick Hymbert
0c4d489e29
quantize: add imatrix and dataset metadata in GGUF ( #6658 )
...
* imatrix: save the dataset file used in the output file
* llama: support kv overrides type string string
* common: factorize KV Overrides parsing between common and server
* quantize: add imatrix n entries and dataset KV metadata
quantize: factorize KV Overrides parsing between common
#6656
* llama: remove kv override str_value initialization as it does not compile on some toolchain
* quantize: add imatrix m_last_call as `quantize.imatrix.chunks_count`
* quantize: add imatrix filename in KV
* llama: add llama_model_kv_override_free
* common: add llama_model_kv_override_free
common: free kv override if used after model loading
* llama: finally move the string KV override value to the stack
* llama : minor
* no need to add a NUL to the std::vector, std::string can be initialized from a pair of iterators.
Co-authored-by: slaren <slarengh@gmail.com>
* kv override: ensure string termination
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: slaren <slarengh@gmail.com>
2024-04-26 20:06:33 +02:00
Georgi Gerganov
1b9b79dd14
convert : fix pre-tokenizer type writing
2024-04-26 20:55:14 +03:00
Georgi Gerganov
43e12ce8e5
llama : use new pre-tokenizer type
2024-04-26 20:08:57 +03:00
slaren
017e6999b5
add basic tensor data validation function ( #6884 )
...
* add basic tensor data validation function
* add --check-tensors command line argument
tensor validation is disabled by default and can be enabled by adding
`--check-tensors` to the command line arguments.
quantize always validates tensors.
2024-04-26 18:39:58 +02:00
Georgi Gerganov
9b4d63ae53
convert : add "tokenizer.ggml.pre" GGUF KV (wip)
2024-04-26 19:21:55 +03:00
Georgi Gerganov
e3f6dc7409
Merge branch 'master' into gg/bpe-preprocess
2024-04-26 18:08:40 +03:00
slaren
e2764cd7ca
gguf : fix mismatch between alloc and free functions ( #6929 )
2024-04-26 18:07:42 +03:00
Justine Tunney
4b1c3c98b4
llamafile : use 64-bit integers in sgemm ( #6928 )
2024-04-26 17:05:33 +03:00
Georgi Gerganov
e9891769ff
unicode : first try custom implementations
2024-04-26 15:09:07 +03:00
Georgi Gerganov
e8c206be61
unicode : shot in the dark to fix tests on Windows
2024-04-26 14:57:12 +03:00
Georgi Gerganov
4907e41aa7
llama : towards llama3 tokenization support (wip)
2024-04-26 14:55:37 +03:00
Georgi Gerganov
ed42711b90
gguf-py : reader prints warnings on duplicate keys
2024-04-26 14:32:22 +03:00
Georgi Gerganov
e1b2bf783e
tests : add sample usage
2024-04-26 13:43:54 +03:00
Georgi Gerganov
aeafb43ed7
tests : remove and rename tokenizer test scripts
2024-04-26 13:39:03 +03:00
Georgi Gerganov
d999cf65c5
unicode : remove redundant headers
2024-04-26 13:29:48 +03:00
Pierrick Hymbert
bbe3c6e761
ci: server: fix python installation ( #6925 )
2024-04-26 12:27:25 +02:00
Georgi Gerganov
7a44e44342
tests : add tokenizer tests for numbers
2024-04-26 13:21:28 +03:00
