python bindings for gpgme

[Justus Winter]

Hi,

Quoting Bernhard Reiter (2016-06-01 14:53:38)
> Hi Justus,
> 
> Am Mittwoch, 1. Juni 2016 12:02:57 schrieb Justus Winter:
> > Quoting Bernhard Reiter (2016-05-31 21:23:48)
> >
> > > In my experience what counts most for python developers is that the
> > > bindings are very pythonic.
> >
> > Agreed.
> 
> so we probably should evaluate this.

I just did, and as Werner noted, we are very well beyond the
evaluation phase.

> > > In my limited experience the SWIG layer makes this harder to do.
> >
> > I don't see why.  SWIG creates a really thin layer, and the resulting
> > interface is a very faithful image of the c api.  Note that the same
> > is true for every FFI solution.
> 
> You can probably implement a speficic API with all four solutions
> that have been mentioned (swig, cffi, cython and manual c-code).
> But, as Ben also noted SWIG is old style and the design decision 
> of SWIG (AFAIR) was to generalize the interface to generate bindings for 
> several languages at once.
> 
> > > Also the additional abstraction really pays out if you use SWIG to
> > > create binding for a number of languages.
> >
> > Not true.  SWIG requires target language-specific code.
> 
> That SWIG requires some language-specific code [...]

SWIG requires *only* language-specific code.  In case you are curious,
it is here:

http://git.gnupg.org/cgi-bin/gitweb.cgi?p=gpgme.git;a=blob;f=lang/python/gpgme.i;hb=HEAD

> does not mean it is necessarily good or elegant to create good
> binding for one language.

We did not create the binding, it was already there.

> The design goal is to abstract and so SWIG aims towards that goal,
> making it harded to use SWIG for python only. Of course if you want to support 
> ruby, python and some more at once, this extra effort may pay out.

There is *no* language-independent code that we wrote.

> > > Otherwise it is just one more abstraction layer to learn when you
> > > want to debug or extend things.
> >
> > I disagree.  It is quite the opposite really, with pyme/SWIG we get
> > most enhancements to GPGME for free, whereas in pygpgme it requires a
> > lot of c code talking to Python's c api.
> 
> My argument is: First I have to understand python's c api with SWIG I have to 
> understand python's c api and SWIG (and its generated code).

I disagree.

With SWIG, you write functions translating c types to python types and
back.  Any function provided by the c library is then automagically
wrapped, with the data being converted by the functions you wrote
*once*.  Add a new function to gpgme, and you likely don't need to do
anything in pyme.

With pygpgme, you write code translating the types forth and back
*explicitly for every function you want to make available to python*.
It is a huge overhead, and likely a source of many errors as that code
is likely copied over and over again, without being properly
understood.

Let's look at the sizes:

% sloccount pygpgme/src
[...]
3037    src             ansic=3037
% sloccount lang/python/helpers.* lang/python/gpgme.i lang/python/pyme
544     top_dir         ansic=544
497     pyme            python=497

Based on that I gather that there must be three to six times as many
bugs in pygpgme.

> > > So far I haven't done a comparison between pygpgme and pyme.
> >
> > Let's compare encryption then:
> >
> > pyme:
> > http://git.gnupg.org/cgi-bin/gitweb.cgi?p=gpgme.git;a=blob;f=lang/python/te
> >sts/t-encrypt.py;h=24869fcd15e19a4cef2e5144570d9a5d37586e58;hb=HEAD pygpgme:
> > http://bazaar.launchpad.net/~jamesh/pygpgme/trunk/view/head:/tests/test_enc
> >rypt_decrypt.py#L99
> >
> > I don't see much of a difference really.
> 
> Comparing the operation and the results, pygpgme combination 
> into one call seems more pythonic to me. Over time this will add up to a lot 
> of more code.

So let's combine these operations in pymes python shim.

> > > From my point of view the most pythonic one should be used.
> >
> > And here is the thing.  While pygpgme is implemented purely in c, and
> > any attempt of making it more pythonic must also be done in c, pyme is
> > using SWIG for the FFI, and layers a shim written in Python on top.
> > This is the place one augments or extends the api to make it more
> > pythonic, e.g.:
> >
> > 
> http://git.gnupg.org/cgi-bin/gitweb.cgi?p=gpgme.git;a=blob;f=lang/python/pyme/core.py;h=cc262c95aba3abc9fb7986af62e9b579b8229723;hb=HEAD#l529
> 
> My main concern is the python API.

I understand that, and I believe that providing a more pythonic api is
easier with pyme, for reasons I repeatedly stated.

> A secondary concern is how it is implemented.

I have some experience with creating bindings for Python, and I'm
familiar with the available FFI solutions.  SWIG may be a more classic
solution, but I don't see any problem with the method.

> You can see that python people move off SWIG to stuff like CFFI (which Ben 
> reported about last May).

And for new bindings, cffi should be chosen over SWIG.

> pygpgme is closer to something like CFFI,

No.  Not at all.

> one layer less indirection (though more c-code).

Six times more c code.

> > > pygpgme is already available for python3 for a few years longer than
> > > pyme, I believe.
> >
> > I must admit I didn't compare pygpgme and pyme before, mostly because
> > I wasn't around when the initial effort started, but if I should
> > choose now, I'd chose pyme as a starting point again.  Having said
> > that, I agree that making it a little more pythonic is important, as
> > Python is the cheese to catch developers these days.  And we will do
> > just that ;)
> 
> What will be the status of the gpgme/lang/python binding when gpgme will be 
> released? experimental? beta?

Why would we release something experimental?

> Once people start relying on it, it will be harder to change the API itself.

People are already relying on it.  pygpgme isn't the only binding that
already existed.  Existing projects using pyme should have no trouble
switching over.

> > Here is what a decrypting filter looks like with the current bindings:
> 
> (This is only a simple example, so its value is limited.

Yeah, if you have a better example maybe it is a good idea to write it
down here so that we can discuss it.  I created that simple example
just to see how simple one can implement decryption with the current
bindings.  That was the point.

> I think that the "op_" part looks ugly.)

Which we can drop, breaking all pyme applications in the process.  We
could, however, add a 'decrypt' operation to the Context class *in
Python* that combines decryption and fetching all relevant results.

That is another thing where pyme is superior to pygpgme.  The latter
implements classes in c, which we cannot subclass in Python.


Cheers,
Justus