Reset schedule earlier to allow overlap with ggml graph computation on device (#6933)

* Reset schedule earlier to allow overlap with graph computation on device
This commit is contained in:
agray3 2024-04-26 19:08:30 +01:00 committed by GitHub
parent 0c4d489e29
commit 928e0b7013
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 11 additions and 5 deletions

View file

@ -11473,6 +11473,10 @@ static int llama_decode_internal(
}
}
// Reset state for the next token before backend sync, to allow the CPU activities in the reset to
// overlap with device computation.
ggml_backend_sched_reset(lctx.sched);
return 0;
}