gpg-agent asks for ssh passphrase, although the private key has no passphras set

[gitter at Safe-mail.net]

I am using gpg-agent to manage my one and only ssh key. I generate my (private) ssh key via openpgp2ssh from my private gpg key. Unfortunately, although my private gpg key is not password protected, gpg-agent asks me for a passphrase (via a nice X dialog) before I ssh to my server. Entering nothing works fine, and I can connect to the server.

~$ eval "$(gpg-agent --enable-ssh-support --daemon)"
~$ gpg2 --export-secret-keys | openpgp2ssh | ssh-add /dev/stdin
Identity added: /dev/stdin (/dev/stdin)
~$ ssh me at myserver.con
...

Is there any way I can disable this needless dialog? I am running GNU/Linux; Debian 6; x86_64 - gpg-agent 2.0.14