6

I'm using dbus to communicate two programs. One creates a large image and it later sends it other program for further processing. I'm passing the image as ByteArray.

With 2000x2000 images my program works, but with 4000x4000 it crasses with:

process 2283: arguments to dbus_message_iter_append_fixed_array() were       
incorrect,assertion "n_elements <= DBUS_MAXIMUM_ARRAY_LENGTH / _dbus_type_get_alignment  
(element_type)" failed in file dbus-message.c line 2628.

I understand that this means that I'm passing an array larger than allowed. Is there other way of passing large data strucutures in dbus?

This is an excerpt of the code I'm using:

handle = StringIO()
# hdulist contains the large data structure
hdulist.writeto(handle)
hdub = dbus.ByteArray(handle.getvalue())
# this sends the image via dbus
self.dbi.store_image(hdub)

In the other end I have something like

def store_image(self, bindata):
    # Convert binary data back to HDUList
    handle = StringIO.StringIO(bindata)
    hdulist = pyfits.open(handle)
Sergio
  • 677
  • 1
  • 10
  • 20

3 Answers3

8

I don't think Dbus is really the best way to send large amounts of data.

How about writing the data structure out to a file in /tmp, and just passing the filename between the programs via dbus instead?

Ali Lown
  • 2,323
  • 1
  • 18
  • 22
  • +1 DBUS is meant for passing small messages, not streaming massive amounts of data. Temporary files or some sort of shared memory is probably the way to go. – detly Jun 03 '11 at 03:16
  • +1. DBus should send a handle to the large chunk of data (a temp file, a shared memory area) and not the data itself. – 9000 Jun 03 '11 at 17:02
  • 1
    To clarify and emphasise what 9000 says, D-Bus provides a facility for passing file descriptors in D-Bus messages, which is better than passing a filename since you can, for example, pass a read-only file descriptor, or pass a handle to an unnamed pipe. The type for file descriptors is `h`. – Philip Withnall Jul 06 '17 at 10:57
  • You can even pass a handle to shared memory, which can be mmapped on the other end. – enigmaticPhysicist Nov 15 '22 at 08:28
5

D-bus won't let you pass more than 128Mb of data per message, and the limit is usually set to even lower value in /etc/dbus-1/session.conf

A named pipe is what you're looking for. It's like a temporary file except that the data is not written to disk, but streamed from one process to another with very low overhead. It has no limits on data size. See http://en.wikipedia.org/wiki/Named_pipe and Create a temporary FIFO (named pipe) in Python? for details.

Community
  • 1
  • 1
Shnatsel
  • 4,008
  • 1
  • 24
  • 25
  • 3
    To clarify this: D-Bus is explicitly intended as a messaging system for small, low latency control messages (referred to as the ‘control plane’ in networking terminology). It is not intended for high bandwidth transmission of large messages (the ‘data plane’). It explicitly provides a way to transmit file descriptors (D-Bus type `h`) between peers, so D-Bus messaging can be used to set up inter-process pipes for high bandwidth data transmission. – Philip Withnall Jul 06 '17 at 10:59
-1

One simple solution that comes to my mind right now is to separate the data structure.. Separate it, send each piece and join it together on the other program. Of course, some care would be necessary to assure that you would join it correctly.

Mauricio
  • 87
  • 2
  • 2
  • This is a bad idea, since you are ultimately sending more data over the bus this way (the same amount of data as you would have otherwise, plus additional message headers). – Philip Withnall Jul 06 '17 at 10:59