Yubikey 4 gpg 2.1.14 forget passphrase for RSA key

[Thomas Glanzmann]

Hello,
I have a yubikey 4 where I store a 4K RSA for authentication, ecryption
and signature on. I'm using it for ssh and gpg agent forwarding to a
remote machine. So far I'm unable to tell gpg-agent to forget the
passphrase. The only thing that works is replug the yubikey. When I
goggeled for the same, the following hints came up:

killall -s HUP gpg-agent
echo RELOADAGENT | gpg-connect-agent

Both have no effect for me, any hints?

Cheers,
        Thomas