Georgi Gerganov
1545550ec2
unicode : normalize signatures
2024-04-28 21:40:36 +03:00
Georgi Gerganov
1c888eb4da
convert : add falcon
...
ggml-ci
2024-04-28 21:26:40 +03:00
Georgi Gerganov
4e3e6d8ecc
lint : update
2024-04-28 21:16:50 +03:00
Georgi Gerganov
7642973616
convert : add convert-hf-to-gguf-update.py
...
ggml-ci
2024-04-28 20:52:31 +03:00
Georgi Gerganov
ee6d1b3fb4
unicode : simplify
2024-04-28 18:36:57 +03:00
Georgi Gerganov
e972e6cbf8
unicode : clean-up
2024-04-28 18:30:37 +03:00
Georgi Gerganov
d63cc9068b
Merge branch 'master' into gg/bpe-preprocess
...
ggml-ci
2024-04-28 15:34:45 +03:00
Georgi Gerganov
b97add52a4
unicode : category support via std::regex
2024-04-28 15:15:57 +03:00
github-actions[bot]
6e472f58e4
flake.lock: Update
...
Flake lock file updates:
• Updated input 'nixpkgs':
'github:NixOS/nixpkgs/5c24cf2f0a12ad855f444c30b2421d044120c66f?narHash=sha256-XtTSSIB2DA6tOv%2Bl0FhvfDMiyCmhoRbNB%2B0SeInZkbk%3D' (2024-04-19)
→ 'github:NixOS/nixpkgs/7bb2ccd8cdc44c91edba16c48d2c8f331fb3d856?narHash=sha256-Drmja/f5MRHZCskS6mvzFqxEaZMeciScCTFxWVLqWEY%3D' (2024-04-25)
2024-04-28 11:12:50 +00:00
mgroeber9110
4dba7e8114
Replace "alternative" boolean operator in conditional compilation directive ( #6949 )
2024-04-27 21:02:06 +02:00
Pierrick Hymbert
b7368332e2
ci: server: tests python env on github container ubuntu latest / fix n_predict ( #6935 )
...
* ci: server: fix python env
* ci: server: fix server tests after #6638
* ci: server: fix windows is not building PR branch
2024-04-27 17:50:48 +02:00
Georgi Gerganov
581c4a0239
unicode : try fix windows
2024-04-27 18:36:00 +03:00
Georgi Gerganov
91eaa414bf
unicode : support \p{N}, \p{L} and \p{P} natively
2024-04-27 17:48:38 +03:00
Georgi Gerganov
ce5485aee0
unicode : always use std::wregex
2024-04-27 17:11:34 +03:00
Georgi Gerganov
2affd0b221
unicode : set bomb
2024-04-27 11:56:02 +03:00
Georgi Gerganov
a22645c2a7
unicode : set bomb
2024-04-27 11:48:24 +03:00
Georgi Gerganov
4434c9d6c2
minor
2024-04-27 11:33:16 +03:00
Georgi Gerganov
ad929833cb
llama : adapt punctuation regex + add llama 3 regex
2024-04-27 11:06:08 +03:00
Georgi Gerganov
96965f67e6
models : add llama v3 vocab file
2024-04-27 11:05:12 +03:00
Georgi Gerganov
c160818ec0
wip
2024-04-27 00:28:36 +03:00
Georgi Gerganov
a774d7084e
make : add test-tokenizer-0-llama-v3
2024-04-26 21:25:36 +03:00
Georgi Gerganov
8791e94e3c
lint : fix
2024-04-26 21:12:05 +03:00
agray3
928e0b7013
Reset schedule earlier to allow overlap with ggml graph computation on device ( #6933 )
...
* Reset schedule earlier to allow overlap with graph computation on device
2024-04-26 20:08:30 +02:00
Pierrick Hymbert
0c4d489e29
quantize: add imatrix and dataset metadata in GGUF ( #6658 )
...
* imatrix: save the dataset file used in the output file
* llama: support kv overrides type string string
* common: factorize KV Overrides parsing between common and server
* quantize: add imatrix n entries and dataset KV metadata
quantize: factorize KV Overrides parsing between common
#6656
* llama: remove kv override str_value initialization as it does not compile on some toolchain
* quantize: add imatrix m_last_call as `quantize.imatrix.chunks_count`
* quantize: add imatrix filename in KV
* llama: add llama_model_kv_override_free
* common: add llama_model_kv_override_free
common: free kv override if used after model loading
* llama: finally move the string KV override value to the stack
* llama : minor
* no need to add a NUL to the std::vector, std::string can be initialized from a pair of iterators.
Co-authored-by: slaren <slarengh@gmail.com>
* kv override: ensure string termination
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: slaren <slarengh@gmail.com>
2024-04-26 20:06:33 +02:00
Georgi Gerganov
1b9b79dd14
convert : fix pre-tokenizer type writing
2024-04-26 20:55:14 +03:00
Georgi Gerganov
43e12ce8e5
llama : use new pre-tokenizer type
2024-04-26 20:08:57 +03:00
slaren
017e6999b5
add basic tensor data validation function ( #6884 )
...
* add basic tensor data validation function
* add --check-tensors command line argument
tensor validation is disabled by default and can be enabled by adding
`--check-tensors` to the command line arguments.
quantize always validates tensors.
2024-04-26 18:39:58 +02:00
Georgi Gerganov
9b4d63ae53
convert : add "tokenizer.ggml.pre" GGUF KV (wip)
2024-04-26 19:21:55 +03:00
Georgi Gerganov
e3f6dc7409
Merge branch 'master' into gg/bpe-preprocess
2024-04-26 18:08:40 +03:00
slaren
e2764cd7ca
gguf : fix mismatch between alloc and free functions ( #6929 )
2024-04-26 18:07:42 +03:00
Justine Tunney
4b1c3c98b4
llamafile : use 64-bit integers in sgemm ( #6928 )
2024-04-26 17:05:33 +03:00
Georgi Gerganov
e9891769ff
unicode : first try custom implementations
2024-04-26 15:09:07 +03:00
Georgi Gerganov
e8c206be61
unicode : shot in the dark to fix tests on Windows
2024-04-26 14:57:12 +03:00
Georgi Gerganov
4907e41aa7
llama : towards llama3 tokenization support (wip)
2024-04-26 14:55:37 +03:00
Georgi Gerganov
ed42711b90
gguf-py : reader prints warnings on duplicate keys
2024-04-26 14:32:22 +03:00
Georgi Gerganov
e1b2bf783e
tests : add sample usage
2024-04-26 13:43:54 +03:00
Georgi Gerganov
aeafb43ed7
tests : remove and rename tokenizer test scripts
2024-04-26 13:39:03 +03:00
Georgi Gerganov
d999cf65c5
unicode : remove redundant headers
2024-04-26 13:29:48 +03:00
Pierrick Hymbert
bbe3c6e761
ci: server: fix python installation ( #6925 )
2024-04-26 12:27:25 +02:00
Georgi Gerganov
7a44e44342
tests : add tokenizer tests for numbers
2024-04-26 13:21:28 +03:00
Pierrick Hymbert
7f5ff558ee
server: stop generation at n_ctx_train if n_predict is not set ( #6638 )
...
* server: cap n_predict if not set to n_ctx_train
* server: fix infinite loop
* server: infinite loop, move in process_token
server: infinite loop: set stop limit to true
* minor: spaces
* minor: spaces
* server: include prompt tokens in the EOS limit
2024-04-26 12:15:30 +02:00
Georgi Gerganov
c56e19db4b
lint : fix whitespaces
2024-04-26 12:58:07 +03:00
Georgi Gerganov
06d3e693db
unicode : fix? unicode_wstring_to_utf8
2024-04-26 12:55:11 +03:00
Pierrick Hymbert
9e4e077ec5
ci: server: fix python installation ( #6922 )
2024-04-26 11:11:51 +02:00
Kazim Abrar Mahi
36d983262e
Fixed issue with gpt2 regex custom preprocessor
2024-04-26 11:43:29 +03:00
Kazim Abrar Mahi
753580360b
Fixed issues
2024-04-26 11:43:29 +03:00
Kazim Abrar Mahi
feeaf4f39c
Added needed functionality, testing remains
2024-04-26 11:43:29 +03:00
Kazim Abrar Mahi
7e308ed212
Adding unicode regex function
2024-04-26 11:43:29 +03:00
Kazim Abrar Mahi
a5710a4101
Adding unicode regex mappings
2024-04-26 11:43:29 +03:00
Kazim Abrar Mahi
4c3e882a85
Refactored code
2024-04-26 11:43:29 +03:00
