vbatts/llama.cpp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

change default AdamW weight decay parameter defined in ggml to 0.0, making Adam default instead of AdamW

Browse source 
btw: the default weight decay parameter for torch.optim.AdamW is 0.01
This commit is contained in:
xaedes 2023-06-29 21:36:28 +02:00
parent f175ead6ef
commit 97964a4cc9
No known key found for this signature in database
GPG key ID: 30030EDD817EA2B1
1 changed files with 1 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

2
ggml.c
View file

@ -17916,7 +17916,7 @@ struct ggml_opt_params ggml_opt_default_params(enum ggml_opt_type type) {
.adam = { .adam = {
.n_iter = 10000, .n_iter = 10000,
.sched = 1.000f, .sched = 1.000f,
.decay = 0.001f, .decay = 0.0f,
.alpha = 0.001f, .alpha = 0.001f,
.beta1 = 0.9f, .beta1 = 0.9f,
.beta2 = 0.999f, .beta2 = 0.999f,