Bulk CRUD with Ditto

Question

we want to handle a large amount of Things in Ditto. So my question: Is there a way to bulk create/read/update/delete? I wrote a test updating 10.000 features of Things in Ditto and this took >2min. I think this mainly is a result of issuing 10.000 HTTP Requests rather than the MongoDB performance - bulk operations would fix this.

Thanks in advance

   Thomas

score 2 · Answer 1 · answered Apr 19 '21 at 06:32

2

Eclipse Ditto does currently not support bulk updates. If this is a requirement for you I'd suggest to open an issue in the Eclipse Ditto repository, where this can be discussed.

If you'd like to get rid of the overhead for 10.000 Handshakes (each http request) you could also switch the protocol and either use websockets or the connectivity API of Eclipse Ditto. This should already provide a performance improvement.

answered Apr 19 '21 at 06:32

Yannic Bürgmann

6,301
5
43
77

1

Thnks a lot, I'll try the websockets first. And share my experience here, before opening the request. – Thomas Landgraf Apr 19 '21 at 08:25
Maybe the ditto java client can be of any help. It uses the websocket protocoll https://github.com/eclipse/ditto-clients/tree/master/java – Yannic Bürgmann Apr 19 '21 at 08:33

score 0 · Answer 2 · answered May 02 '21 at 11:36

here are my results:

REST1: 2000 things created, 0 things rejected, in 67.496 seconds -> 29.6 create/s
REST1: 2000 things' features created, 0 features rejected, in 54.492 seconds -> 36.7 features/s
REST1: 20000 things' features updated, 0 updates rejected, in 523.521 seconds -> 38.2 updates/s
REST1: 993 results of 2000 by search, in 1.788 seconds -> 555.3 results/s
REST1: 2000 things deleted, in 50.861 seconds -> 39.3 deletes/s
REST32: 2000 things created, 0 things rejected, in 22.779 seconds -> 87.8 create/s
REST32: 2000 things' features created, 0 features rejected, in 20.654 seconds -> 96.8 features/s
REST32: 20000 things' features updated, 0 updates rejected, in 187.385 seconds -> 106.7 updates/s
REST32: 968 results of 2000 by search, in 1.918 seconds -> 504.7 results/s
REST32: 2000 things deleted, in 19.617 seconds -> 102.0 deletes/s
WEBSOCKET: 2000 things created w websocket, in 33.082 seconds -> 60.5 create/s
WEBSOCKET: 2000 features created, in 33.119 seconds -> 60.4 creates/s
WEBSOCKET: 20000 features modified, in 263.630 seconds -> 75.9 mods/s
WEBSOCKET: 2000 things deleted w websocket, in 24.360 seconds -> 82.1 deletes/s

the REST1 results are from at ThreadWorker Pool of 1 Thread, the REST32 pool has 32 threads, firing 32 requests in parallel.

best Thomas

Bulk CRUD with Ditto

2 Answers2