xaedes
b26bd4c34c
add option to save train-text-from-scratch output every N iterations
2023-08-30 16:26:05 +02:00
xaedes
f3590ad8d9
remove trailing whitespace
2023-08-30 16:01:08 +02:00
xaedes
fc456edda6
train-text-from-scratch can train (full finetune) gguf models
...
just pass the gguf model via `--checkpoint-in FN`.
after this, to continue training, pass the generated checkpoint instead of the original gguf model.
tested with smaller models, bigger models may exceed available memory.
use (LORA) finetune for those.
2023-08-30 15:57:17 +02:00
xaedes
e6b7158123
replace custom data getters and setters by ggml functions
2023-08-30 15:21:27 +02:00
xaedes
d487e0531f
move gradient checkpointing code into ggml, new API function:
...
// build gradient checkpointing backward graph gb for gf using provided checkpoints
// gb_tmp will contain original backward graph with rewritten backward process nodes,
// but without the second forward pass nodes.
GGML_API void ggml_build_backward_gradient_checkpointing(
struct ggml_context * ctx,
struct ggml_cgraph * gf,
struct ggml_cgraph * gb,
struct ggml_cgraph * gb_tmp,
struct ggml_tensor * * checkpoints,
int n_checkpoints);
2023-08-30 15:21:27 +02:00
xaedes
2392b6725b
use tensor->view_src instead of ggml_is_view and get_view_source
2023-08-30 14:46:12 +02:00
xaedes
b1709f2d25
Merge branch 'master' into finetune-lora
2023-08-30 13:28:29 +02:00
slaren
06abf8eeba
ggml : add view_src and view_offs to ggml_tensor for views ( #2874 )
...
* ggml : add view_src and view_offs
* update ggml-alloc to use view_src
* update ggml_diag_mask to work correctly with automatic inplace
* exclude other ops that set an inplace flag from automatic inplace
2023-08-29 23:24:42 +02:00
slaren
c03a243abf
remove outdated references to -eps and -gqa from README ( #2881 )
2023-08-29 23:17:34 +02:00
xaedes
bf70e27cd6
fix check_gradient
...
ggml_build_backward_expand was previously replaced by ggml_build_backward, but the assignment of forward graph to backward graph missing
2023-08-29 23:08:30 +02:00
Kawrakow
fa3582f509
Tell users attmepting to run perplexity with too few tokens to use more ( #2882 )
...
Closes #2858
Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
2023-08-29 23:55:45 +03:00
Kawrakow
e37e69dcc3
10X faster BPE tokenizer ( #2876 )
...
* 10X faster BPE tokenizer
* Remove comment that no longer applies
---------
Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
2023-08-29 23:55:03 +03:00
xaedes
5854f51188
fix error message in ggml_allocr_alloc to display actual max_avail
2023-08-29 22:49:01 +02:00
xaedes
281245a48f
Merge branch 'master' into finetune-lora
2023-08-29 21:47:28 +02:00
xaedes
8a96d4c2aa
add missing argument 'int i0' to ggml_get_i32_nd & ggml_set_i32_nd header declarations
2023-08-29 21:24:37 +02:00
xaedes
dd4e4bca09
remove unused 'inplace' argument from ggml_compute_backward function
...
inplace operations to add gradients are no longer created by ggml_compute_backward
use allocator to automatically make inplace operations
2023-08-29 21:21:10 +02:00
xaedes
a76e66ac8d
fix ggml_acc_or_set to return tensor of correct shape
2023-08-29 21:02:10 +02:00
xaedes
b1aa26f718
add sanity check to ggml_compute_backward, asserting the correct shape of gradients
2023-08-29 21:01:17 +02:00
xaedes
5fcfa7e49e
increase test-grad0 context mem size to accommodate for bigger cgraph
2023-08-29 21:00:19 +02:00
xaedes
82c5247a20
add ggml API functions ggml_unravel_index, ggml_get_i32_nd and its analogs for set and for f32
...
ggml_get_i32_1d, ggml_set_i32_1d, ggml_get_f32_1d, ggml_set_f32_1d now support non-contiguous tensors.
in case of non-contiguous tensor, the 1d index is unraveled into a multi index using ggml_unravel_index to be passed to '_nd' function equivalent.
this fixes a bug in test-grad0 which happens due to ggml_build_backward not building purely contiguous tensors anymore
2023-08-29 20:59:31 +02:00
xaedes
5f0a4e971f
avoid stack overflow of large cgraphs in test-grad0
2023-08-29 19:59:41 +02:00
xaedes
794bb7ea42
implement ggml_compute_forward_repeat_f16
2023-08-29 19:59:14 +02:00
xaedes
e28cf7e9ce
update README.md
2023-08-29 19:38:23 +02:00
xaedes
a6165dafcd
remove trailing whitespace
2023-08-29 19:30:42 +02:00
xaedes
5813ac832f
omit tokenization when training is disabled, only save llama lora adapter
...
training can be disabled by passing '-n 0' to finetune
2023-08-29 19:21:45 +02:00
xaedes
ebff3a14c3
remove code to print data checksums which was used to verify correctness of new gguf code
2023-08-29 18:31:20 +02:00
xaedes
1425968ead
remove old checkpoint save & load code
2023-08-29 18:30:16 +02:00
xaedes
6134ad4de7
add python script to convert old finetune checkpoint files to gguf
2023-08-29 18:24:06 +02:00
xaedes
0564f4ed1f
add load & save lora finetune checkpoints via gguf
2023-08-29 18:20:39 +02:00
maddes8cht
53885d7256
py : fix "usage" messages ( #2873 )
...
convert-to-gguf python scripts
2023-08-29 16:51:02 +03:00
jameswu2014
bcce96ba4d
convert.py : fix baichuan7B support ( #2870 )
...
* [Fix]: convert.py support baichuan7B
* convert.py : fix trailing whitespaces
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2023-08-29 12:48:41 +03:00
Jhen-Jie Hong
74e0caeb82
readme : add react-native binding ( #2869 )
2023-08-29 12:30:10 +03:00
Cebtenzzre
d4b5e16c32
make : fix clang tests build, add missing examples ( #2859 )
...
* make : do not pass headers to the compiler
This fixes building tests with clang.
* make : add missing examples
* make : fix build-info.h dependencies
2023-08-29 11:42:41 +03:00
Georgi Gerganov
3a007648f2
metal : add option to disable debug logs ( close #2764 )
2023-08-29 11:33:46 +03:00
Georgi Gerganov
611363ac79
scripts : add pipefail
2023-08-29 10:50:30 +03:00
Marcus Dunn
95b6e5212f
added struct to llama_dump_timing_info_yaml's llama_context ( #2857 )
...
fixes C compat.
2023-08-29 09:33:27 +03:00
xaedes
ecb1b20c85
add gguf constants and load/save functions from train-text-from-scratch
2023-08-29 01:40:02 +02:00
xaedes
e030f7b2c5
add LLM_KV_TRAINING_TYPE to train-text-from-scratch checkpoints
...
so that they can be differentiated from lora finetune checkpoints
2023-08-29 01:27:28 +02:00
xaedes
ca97583f0b
remove vocab related code as it is unnecessary
2023-08-29 01:19:45 +02:00
xaedes
a3b45298f1
remove unused code
2023-08-29 01:12:51 +02:00
xaedes
1faee64db9
handle rms_norm and rope parameters the same as in train-text-from-scratch
2023-08-29 01:09:35 +02:00
xaedes
007280c82f
make default value of float member a float literal
2023-08-29 01:04:57 +02:00
xaedes
49af7fbe12
add comment explaining why finetune checkpoints are allocated in one block
2023-08-29 00:57:39 +02:00
xaedes
9a28bce29a
reduce large memory overhead in train-text-from-scratch
...
all gradients had to be pinned so that graph_reset works correctly.
this is no longer necessary with the changes to ggml_compute_backward introduced in this PR.
2023-08-29 00:56:44 +02:00
xaedes
271c0300de
remove prediction related code to reduce duplicated code with main
...
use main instead
2023-08-29 00:50:59 +02:00
xaedes
5ce92aed37
finetune bug fixes to compile with merged in code from master
2023-08-29 00:41:19 +02:00
xaedes
daedc6f419
replace llama_n_mult by llama_n_ff
2023-08-29 00:40:53 +02:00
xaedes
aa8016e95d
bug fix: replace GGML_TYPE_SIZE[t] by ggml_type_size(t)
2023-08-29 00:40:30 +02:00
xaedes
aecc3b3890
fix dump_non_result_info_yaml to output multiple lora adapters
2023-08-29 00:39:59 +02:00
xaedes
b04263c1fe
Merge branch 'master' into finetune-lora
...
# Conflicts:
# examples/CMakeLists.txt
# examples/train-text-from-scratch/train-text-from-scratch.cpp
# ggml.c
# llama.cpp
# llama.h
2023-08-29 00:17:04 +02:00
