Stack Overflow
Stack Overflow

Questions tagged [trident]

Abstraction on top of Apache Storm for doing realtime computation.

Trident is a high-level abstraction for doing realtime computing on top of Apache Storm. It allows you to seamlessly intermix high throughput (millions of messages per second), stateful stream processing with low latency distributed querying. If you're familiar with high level batch processing tools like Pig or Cascading, the concepts of Trident will be very familiar – Trident has joins, aggregations, grouping, functions, and filters. In addition to these, Trident adds primitives for doing stateful, incremental processing on top of any database or persistence store. Trident has consistent, exactly-once semantics, so it is easy to reason about Trident topologies. (source)

122 questions
39
votes
5 answers

Storm vs. Trident: When not to use Trident?

I'm working with Storm and it is fine for a lot of use cases. Recently I had a look at Trident, which is a high-level abstraction of Storm. It supports exactly-once processing and makes stateful processing easier. But now I'm wondering.. Why can't I…
Christian Strempfer
  • 7,291
  • 6
  • 50
  • 75
10
votes
1 answer

How to close a database connection opened by an IBackingMap implementation within a Storm Trident topology?

I'm implementing an IBackingMap for my Trident topology to store tuples to ElasticSearch (I know there are several implementations for Trident/ElasticSearch integration already existing at GitHub however I've decided to implement a custom one which…
bopcat
  • 378
  • 4
  • 13
7
votes
1 answer

How to map tuples with persistent state in Trident?

I'm learning Trident framework. There are several methods on Trident Streams for aggregation tuples within a batch, including this one which allows to preform a stateful mapping of the tuples using Aggregator interface. But unfortunately a built-in…
leventov
  • 14,760
  • 11
  • 69
  • 98
7
votes
2 answers

What is Trident State in Storm?

I am new to Trident in Storm. I am breaking my head over TridentState. As far as my understanding trident maintains the state (i.e metadata) for each batch (whether all the tuples in a batch are completely processed by maintaining a Transaction id…
Ezhil
  • 261
  • 2
  • 10
  • 31
4
votes
1 answer

Apache Storm Trident and Kafka Spout Integration

I am unable to find good documentation for correctly integrating Kafka with Apache Storm Trident. I tried to look into the related previously posted questions here, but no sufficient information. I would like to connect Trident with Kafka as…
phaigeim
  • 729
  • 13
  • 34
4
votes
1 answer

Storm UI improper values and Capillary tool

I am quiet new to Apache Storm and have been trying with trident topology for Kafka i.e. TransactionalTridentKafkaSpout. All works fine except the Storm UI. Even though I've not produced any data to my topic, the Storm UI keeps showing invalid…
DMA
  • 1,033
  • 1
  • 11
  • 22
4
votes
2 answers

Storm Trident Topology with Kafka: Received unexpected tuple error

We have a Trident topology that both consumes from and produces to kafka using the kafka-storm OpaqueTridentKafkaSpout and TridentKafkaState. Everything works fine when running on our production storm cluster, but when run in local mode we very…
merpmerp
  • 86
  • 3
4
votes
2 answers

Creating an Apache Storm spout which emits tuples every X seconds

I have a topology that's receiving data from a MQTT broker, and I want a spout to behave like this: Emit a batch of tuples (or a list of strings in a single tuple) every x seconds. How do I achieve this? I read a bit about Storm Trident but its…
touchaponk
  • 404
  • 4
  • 17
4
votes
1 answer

Using tick tuples with trident in storm

I am able to use standard spout,bolt combination to do streaming aggregation and works very well in happy case, when using tick tuples to persist data at some interval to make use of batching. Right now i am doing some failure management (tracking…
user3479897
  • 73
  • 1
  • 6
3
votes
2 answers

NoClassDefFoundError: kafka/api/OffsetRequest

I am trying to write application for real time processing with apache storm , kafka and trident but in initialization of TridentKafkaConfig i see this error Exception in thread "main" java.lang.NoClassDefFoundError: kafka/api/OffsetRequest at…
Ali Kianinejad
  • 965
  • 1
  • 9
  • 17
3
votes
1 answer

How can I write a tuple in to redis as well as cassandra using trident topology

I am writing a Trident topology to process stream of data from Kafka and feed in to Redis and Cassandra. I am able to write the data in to Cassandra. Now I would like to write the same data in to Redis. Is there a way to duplicate the tuples and…
Karthick Selvaraj
  • 53
  • 4
3
votes
0 answers

Merging trident streams blocks the trident spout whereas the storm spout keeps working

I need some help understanding why merging two streams blocks one of the spouts of class FixedBatchSpout. Short Description: I’m trying to merge two streams s1 and s2, but calling topology.merge(s1, s2) blocks the FixedBatchSpout (a trident spout)…
Colegram
  • 106
  • 3
3
votes
2 answers

Trident or Storm topology that writes on Redis

I have a problem with a topology. I try to explain the workflow... I have a source that emits ~500k tuples every 2 minutes, these tuples must be read by a spout and processed exatly once like a single object (i think a batch in trident). After that,…
Eddyman
  • 33
  • 1
  • 3
3
votes
1 answer

parallelism configuration in trident topology (storm)

After reading this and this I'm having difficulties understanding how to configure my trident topology. Basically my storm application is reading from kafka, doing some data manipulations and finally writing to Cassandra. Here is how I'm currently…
forhas
  • 11,551
  • 21
  • 77
  • 111
2
votes
3 answers

Prometheus and nfs storage

As per prometheus storage.md , the recommendation is not to use nfs storage as persistent volume for prometheus. But solutions like prometheus operator and openshift shows examples which uses nfs as persistent volumes for prometheus. So what am I…
swetad90
  • 784
  • 1
  • 13
  • 34
1
2 3
…
8 9