Reduce GPGME memory usage

[Justus Winter]

Hello :)

Marcel Behlau <marcel.behlau at elfin.de> writes:

> The old version worked fine, now i have to port the stuff to a new 
> system with fewer RAM and bigger update files. This generates some 
> problems , if the maximum RAM is used, caused by to big update files. In 
> my workflow, i'm compare the signer keys of the updatefiles and the 
> expected keys, but the "gpgme_get_key" functions fails with a "Invalid 
> crypto engine" error. Is the error message correct at this position? The 
> gpgme_op_decrypt_verify, gpgme_op_decrypt_result and 
> gpgme_op_verify_result work before properly.

Did gpgme_get_key work before for you?  Looking at the source of
gpgme_get_key I see that it clones the context, so maybe you are setting
up the context in a special way that makes it fail?  Does the
gpgme_op_keylist_start interface work for you?

If you are able to construct a minimal test case, feel free to open a
bug report.

> The size of the update files will be reduced before the release, so i'm 
> hoping, the system will finaly work. But this is very critical, if 
> someday, the applications, which are running on the system, using to 
> much RAM. So 'm now trying to reduce the memory usage for the complete 
> update process.
>
> My gpgme encrypted updatefile contains a single tar file, the tar file 
> contains all necessary files for the update. I'm using libtar to extract 
> the files to the installation path, libtar is working directly on the 
> gpgme memory buffer. The complete update is stored during the process in 
> the memory. This leads to the problems with to few RAM. Is there a way, 
> to reduce the size of the gpgme memory buffer, maybe by reloading (and 
> redecrypting ) data chunks from the orignal crypted update file? The 
> original update file is loaded via stream. I found the " Callback Based 
> Data Buffers" in the documation. It is possible, to use this buffers, to 
> realize the reloading behavior?

I just tried, and the callbacks can indeed be used to implement a
streaming interface, e.g.:

~~~ snip ~~~
import sys
import pyme

def do_write(data):
    sys.stdout.write(chr(data[0]))
    return 1

data = pyme.Data(cbs=(None, do_write, None, lambda: None, None))
with pyme.Context() as c:
    c.decrypt(sys.stdin, sink=data)
~~~ snap ~~~

Likewise you can feed the source in smaller chunks to the engine.


If that is not enough, you might want to look at TinyGPG, a library
implementing a subset of the OpenPGP protocol, that is written with
embedded systems in mind:

https://git.gnupg.org/cgi-bin/gitweb.cgi?p=tgpg.git;a=summary


Cheers,
Justus
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 472 bytes
Desc: not available
URL: </pipermail/attachments/20160803/6d05d9e6/attachment.sig>