gpg prompts me thrice for my passphrase - how to resolve it

[Thomas Glanzmann]

Hello,
I sometimes use a yubikey, there gpg-agent only asks me once for my pin,
however if I have my key on the disk, gpg-agent asks me three times:

        - once for local gpg -d test.gpg
        - once for gpg-agent functioning as ssh-agent
        - once for remote gpg -d test.gpg

Now I wonder, if I can tell gpg-agent to prompt me only once or
prepopulate all three secrets at once during startup?

Cheers,
        Thomas