reverse order of for loop in ggml_build_backward_expand to save memory when using gradient checkpointing and allocator

with this loop order gradient checkpointing with allocator on 16 layer model saves 13% memory; 2 layer memory it saves 2% memory.

the computation results are the same
This commit is contained in:
xaedes 2023-08-14 22:09:36 +02:00
parent 56228461c8
commit 3b5515bbe0
No known key found for this signature in database
GPG key ID: 30030EDD817EA2B1

2
ggml.c
View file

@ -15809,7 +15809,7 @@ void ggml_build_backward_expand(struct ggml_context * ctx, struct ggml_cgraph *
} }
} }
for (int i = gf->n_nodes - 1; i >= 0; i--) { for (int i = 0; i < gf->n_nodes; i++) {
struct ggml_tensor * node = gf->nodes[i]; struct ggml_tensor * node = gf->nodes[i];
if (node->is_param) { if (node->is_param) {