GPG agent returns subset of keys for SSH

[LuKaRo]

Hi,

I want to use gpg-agent to authenticate to an SSH server via key. This 
has previously worked on this machine when I was using a Nitrokey, now I 
imported the key that was on the Nitrokey locally from a backup, and SSH 
authentication no longer works.

ssh -vvvv server lists these interesting messages:

debug3: ssh_get_authentication_socket_path: path 
'/run/user/1000/gnupg/S.gpg-agent.ssh'
debug2: get_agent_identities: ssh_agent_bind_hostkey: agent refused 
operation
debug1: get_agent_identities: ssh_fetch_identitylist: agent contains no 
identities

However, gpg --list-secret-keys shows this:

sec   rsa4096 2020-04-07 [SC]
       94B238AAE6682E5063896F2B7920D03B7AA7CD7B
uid           [ultimate] Lu Ro (New general key) <public at lrose.de>
ssb   rsa4096 2020-04-07 [E]
ssb   rsa4096 2020-04-07 [A]

So the authenticate subkey is indeed present.

I executed ssh-add without arguments, and two keys were added from my 
.ssh directory. Now ssh -vvvv shows this:

debug3: ssh_get_authentication_socket_path: path 
'/run/user/1000/gnupg/S.gpg-agent.ssh'
debug2: get_agent_identities: ssh_agent_bind_hostkey: agent refused 
operation
debug1: get_agent_identities: agent returned 2 keys

So communication with the gpg-agent seems to work as well. Any ideas 
what could be the issue?

Thanks in advance,

lukaro