vbatts/llama.cpp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Wrong input was being fed to moe layer. This needs to be corrected

Browse source 
Co-authored-by: Megha Agarwal <16129366+megha95@users.noreply.github.com>
This commit is contained in:
Pierrick Hymbert 2024-04-12 21:41:14 +02:00 • committed by GitHub
parent 542585fbea
commit ecbfb1b584
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 1 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

2
llama.cpp
View file

@ -7165,7 +7165,7 @@ struct llm_build_context {
// feed-forward network // feed-forward network
// MoE branch // MoE branch
cur = llm_build_norm(ctx0, cur, hparams, cur = llm_build_norm(ctx0, ffn_inp, hparams,
model.layers[il].attn_out_norm, NULL, model.layers[il].attn_out_norm, NULL,
LLM_NORM, cb, il); LLM_NORM, cb, il);
cb(cur, "attn_out_norm", il); cb(cur, "attn_out_norm", il);