xaedes
8b5b2f089e
fix backward pass for repeat
...
requires ggml_sum_rows
2023-05-01 14:43:50 +02:00
xaedes
ba62c79bd5
add missing GGML_OP_SUM_ROWS
2023-05-01 14:43:50 +02:00
xaedes
c4539ede53
add operation ggml_sum_rows
...
ggml_sum_rows(shape[a,b,c,d]) -> shape[1,b,c,d]
2023-05-01 14:43:49 +02:00
xaedes
2277053839
add todos for llama backward pass
...
- implementation for ADD1 backward pass should probably use sum instead of mean (but this backward pass is not required)
- repeat is not yet tested and looks like it only works for single element src0 inputs.
2023-05-01 14:43:49 +02:00
xaedes
2ecc690980
successfully test backward pass of rms_norm
...
some tests may fail when gradients are large.
could not find a satisfying configuration to check for abs error and relative error that passes all tests while still actually testing the results with tight enough error bounds.
when looking at the values the "failed" tests look actually ok. for example:
rms_norm: ndims=2, i=0, k=2, x0=0.000153, xm=0.000053, xp=0.000253, f0=0.278594, f1=0.086213, g0=961.905457, g1=966.064941, eps=0.000100, error_abs=4.159485, error_rel=0.004324
it is due to the test logic in check_gradients that they fail.
2023-05-01 14:43:49 +02:00
xaedes
84a4b39917
fix backward pass for rms_norm
...
I would have used formulas from other frameworks, but they differed so I could not decide which is correct.
Instead it was derived here in comment using manual forward-backward automatic differention of rms_norm and simplification.
2023-05-01 14:43:49 +02:00
xaedes
b18b72da00
successfully test backward pass of view_1d, view_2d and view_3d
2023-05-01 14:43:49 +02:00
xaedes
84436383eb
fix view backward pass
...
add nb parameters to add_at like in view.
together with offset they define how to view dst and src0 during the add_at operation.
2023-05-01 14:43:49 +02:00
xaedes
f0302fa71b
successfully test get_rows backward
2023-05-01 14:43:49 +02:00
xaedes
96e773bbde
fix get rows backward pass
2023-05-01 14:43:48 +02:00
xaedes
7281f60572
move dup call into the actual add_at functions
2023-05-01 14:43:48 +02:00
xaedes
3dbd649cf9
fix diag_mask to work with non-inplace input
2023-05-01 14:43:48 +02:00
xaedes
b9920e5c3e
test-grad0 : fix test for div
...
nargs and ndims was swapped, corrupting the stack
2023-05-01 14:43:48 +02:00
xaedes
19f51592b5
successfully test diag_mask_inf and diag_mask_zero backward
2023-05-01 14:43:48 +02:00
xaedes
d42531fa56
fix comments
2023-05-01 14:43:48 +02:00
xaedes
1997152f7f
test-grad0.c add TODO for view_2d and view_3d
...
add_at (required for view backward pass) is a bit tricky for n_dims > 1.
2023-05-01 14:43:48 +02:00
xaedes
c601df973c
successfully test transpose backward and permute for all permutations
...
also test sub, mul and div up to max n_dims
2023-05-01 14:43:47 +02:00
xaedes
3d21f2646e
implement ggml_cont backward pass
2023-05-01 14:43:47 +02:00
xaedes
02d3fd0894
fix sub, mul and div functions to work correctly with transposed tensors
...
uses the same logic as in add
2023-05-01 14:43:47 +02:00
xaedes
b0555fce95
some minor test-grad0 fixes
2023-05-01 14:43:47 +02:00
xaedes
a7a837047c
successfully test permute backward
2023-05-01 14:43:47 +02:00
xaedes
86b44a02e4
test-grad0.c : add print_elements to help with debugging
2023-05-01 14:43:47 +02:00
xaedes
339b2adf48
fix ggml_forward_add1 functions to work correctly with transposed tensors
...
uses the same logic as in ggml_compute_forward_add1_q_f32, but make it consistent across all ggml_compute_forward_add1_... functions.
this also slightly changes the mem access pattern of the different threads to works as in ggml_compute_forward_add1_q_f32.
2023-05-01 14:43:47 +02:00
xaedes
b9416d71f8
fix ggml_forward_add functions to work correctly with transposed tensors
...
uses the same logic as in ggml_compute_forward_add_q_f32, but make it consistent across all ggml_compute_forward_add_... functions.
this also slightly changes the mem access pattern of the different threads to works as in ggml_compute_forward_add_q_f32.
2023-05-01 14:43:46 +02:00
xaedes
410a47a79e
minor code format improvement
2023-05-01 14:43:46 +02:00
xaedes
124fdca973
successfully test view backward
2023-05-01 14:43:46 +02:00
xaedes
cecd6c7665
bug fix for add_at forward
...
required for view backward pass
src0 values must be copied to dst, because during addition we don't touch all dst elements in contrast to the normal add function.
2023-05-01 14:43:46 +02:00
xaedes
83fa6b3bcb
fix ggml_compute_forward_dup_same_cont for when nelements < nthreads
...
when more threads are used than elements exist ie1 was less than ie0, resulting in invalid negative byte count argument in memcpy
2023-05-01 14:43:46 +02:00
xaedes
c1a8893de3
de-duplicate ggml_forward_dup code taking care of contiguous tensors of same type.
...
with this we can duplicate tensor of any typ as long as they are contiguous.
2023-05-01 02:42:27 +02:00
xaedes
38675e537c
add shape annotations for llama
2023-05-01 02:42:13 +02:00
xaedes
93106504fd
align shape annotations
2023-05-01 02:42:05 +02:00
xaedes
fea42be47a
successfully test soft_max backward
2023-05-01 02:41:58 +02:00
xaedes
1a80e9a0fa
correctly implement softmax backward pass using new operation ggml_diag
...
ggml_diag constructs diagonal matrices with entries.
ggml_diag(shape[a,1,c,d]) -> shape[a,a,c,d]
2023-05-01 02:41:30 +02:00
xaedes
54ab300cc4
add test-opt.c
...
this uses ggml_opt to train a,b for minimal e=sum(sqr(c - a*b)) for random initial a,b,c
2023-05-01 02:41:30 +02:00
xaedes
ecf949b10f
successfully test reshape backward
2023-05-01 02:41:30 +02:00
xaedes
c483a7dac5
bug fix for reshape backward pass
2023-05-01 02:41:30 +02:00
xaedes
b2bd8222da
successfully test cpy backward
2023-05-01 02:41:30 +02:00
xaedes
0ea8201c86
bug fix for cpy backward pass
2023-05-01 02:41:30 +02:00
xaedes
7571147242
successfully test rope backward
2023-05-01 02:41:30 +02:00
xaedes
b583136cfa
improve performance of sqr backward pass
...
use scale(x,y) instead of mul(x,repeat(y,x))
2023-05-01 02:41:30 +02:00
xaedes
bfe507213c
improve performance of sum backward pass
...
use add1(x,y) instead of add(x,repeat(y,x))
2023-05-01 02:41:30 +02:00
xaedes
0197bcb0ff
successfully test scale backward
2023-05-01 02:41:29 +02:00
xaedes
a367eb9eda
bug fix for scale backward pass
...
use sum instead of mean for gradient of scalar scale parameter
2023-05-01 02:41:29 +02:00
xaedes
671e5922e2
successfully test silu backward
2023-05-01 02:41:29 +02:00
xaedes
6fb08b4554
bug fixes for silu_back
2023-05-01 02:41:29 +02:00
xaedes
9d6fc28f18
disable graph dot export as it floods console
2023-05-01 02:41:29 +02:00
xaedes
9345f4c3a5
test both gradients of mul_mat
2023-05-01 02:41:29 +02:00
xaedes
20e3c1d2b4
use GGML_PRINT_DEBUG for debug messages which will otherwise flood the console
2023-05-01 02:41:29 +02:00
xaedes
0da26753fd
add test-grad0.c
2023-05-01 02:41:29 +02:00
xaedes
4e1f81d32f
implement backward pass for ggml_get_rows and for new operation ggml_get_rows_back
2023-05-01 02:41:29 +02:00
