Selecting SSH Key in gpg-agent ssh-agent mode

[Felix Winterhalter]

Hey there fellow gpg-users,

I've been using gpg-agent for a while with my Yubikey and its working 
fine. Asking me the pin once on each plugin and then silently working in 
the background.

For various reasons I also have on-disk ssh-keys with passphrases that I 
added with ssh-add to the gpg-agents keystore.

However on servers where those keys are present gpg-agent will always 
ask me to unlock these keys first even if the Yubikey is already 
unlocked. On declining pinentry it will then continue to use the 
Yubikey's keys.

Is there any setting to reorder the order in which SSH-Keys are tried 
against a server? Or rather is there also a way to specifiy to first try 
unlocked keys?

Cheers,

Felix