Questions tagged [alpakka]

Alpakka is the collective name for various Akka Streams connectors, integration patterns, and data transformations.

Alpakka is a community-driven initiative that provides connectors, integration patterns, and data transformations that are built with Akka Streams. This toolkit is meant to be a "modern alternative to Apache Camel" (hence its name, which is a homophone of "alpaca," a relative of the camel, and was first used as a code name for an old akka-camel module).

From an introductory blog post:

Akka Streams already has a lot that are useful for integrations. Defining processing pipelines is what the Akka Streams DSL is all about and that is exactly what you need for operating on streaming data that cannot fit in memory as a whole. It handles backpressure in an efficient non-blocking way that prevents out-of-memory errors, which is a typical problem when using unbounded buffering with producers that are faster than consumers.

Connectors:

  • AMQP
  • Apache Geode
  • AWS DynamoDB
  • AWS Kinesis
  • AWS Lambda
  • AWS S3
  • AWS SNS
  • AWS SQS
  • Azure Storage Queue
  • Cassandra
  • Elasticsearch
  • File
  • FTP
  • Google Cloud Pub/Sub
  • HBase
  • IronMq
  • JMS
  • MongoDB
  • MQTT
  • Server-sent Events (SSE)
  • Slick (JDBC)
  • Spring Web

Integration Patterns:

  • Splitter

Data Transformations:

  • Parsing Lines
  • JSON
  • Compressing/decompressing
  • Comma-Separated Values (CSV)
  • RecordIO Framing
  • Extensible Markup Language (XML)

Additional Links:

217 questions
8
votes
1 answer

Proper way to programmatically stop an Alpakka Kafka stream

We are trying to use Akka Streams with Alpakka Kafka to consume a stream of events in a service. For handling event processing errors we are using Kafka autocommit and more than one queue. For example, if we have the topic user_created, which we…
SergiGP
  • 669
  • 7
  • 17
8
votes
3 answers

Alpakka kafka vs Kafka streams

We are building a high-throughput low-latency stream processing application. We are using Apache Kafka both as a messaging platform and as a database. It seems like the Kafka Streams and Alpakka Kafka frameworks do a lot in common, but Kafka Streams…
roman-roman
  • 2,746
  • 19
  • 27
7
votes
2 answers

Alpakka KinesisSink : Can not push messages to Stream

I am trying to use the alpakka kinesis connector to send messages to a Kinesis Stream but I have no success with it. I tried the code below but nothing in my stream. implicit val sys = ActorSystem() implicit val mat = ActorMaterializer() implicit…
ccheneson
  • 49,072
  • 8
  • 63
  • 68
6
votes
1 answer

Count number of elements in Akka Streams

I'm trying to transform a Source of Scala entities into a Source of ByteString via Alpakka's CsvFormatting and count number of elements in the initial stream. Could you suggest the best way to count the initialSource elements and keep the result as…
noname.404
  • 335
  • 3
  • 11
5
votes
1 answer

Multiple download requests with Alpakka S3 connector

I'm trying to use the Alpakka S3 connector to do the following: Download a number of files from AWS S3 Stream the downloaded files via the Alpakka Zip Archive Flow Upload the Zip stream back to S3 Sink The code I used is something like this: val…
rogueai
  • 131
  • 2
  • 7
5
votes
0 answers

Akka stream stops when AMQP server restarts

I'm having a really weird issue using the Alpakka AMQP connector and Akka Streams. When my RabbitMQ message broker restarts, the source seems to restart fine. However, once it's restarted, the stream never completes, and the message gets lost in a…
Le G
  • 408
  • 3
  • 7
4
votes
0 answers

Download and zip folder on GCP via Alpakka

Moin, I'd like to ask for solution. I want to zip content of folder on GCP bucket. I'm using Alpakka 5.0.0 to do that. I've written following: import akka.actor.typed.ActorSystem import akka.actor.typed.scaladsl.Behaviors import…
Teliatko
  • 1,521
  • 1
  • 14
  • 15
4
votes
1 answer

Alpakka Consumer not consuming messages from Kafka running via Docker compose

I've got Kafka and Zookeeper running via Docker compose. I'm able to send/consume messages to a topic using Kafka terminal and I'm able to monitor everything via Conduktor. But unfortunately, I'm not being able to consume msgs via my Scala app using…
4
votes
1 answer

trying to convert java file to bytestring for using in stream

Im creating an api to send me files and I will upload them to S3, and im using akka-stream-alpakka-s3 library to do it using streams. my issue is that in my controller I can convert the file to Jave file: def uploadToS3() =…
jack miao
  • 1,398
  • 1
  • 16
  • 32
4
votes
3 answers

How to access REST API on a unix domain socket with Akka HTTP or Alpakka?

I would like to access the docker API using the /var/lib/docker.sock unix domain socket. I've seen examples where you can use (modern versions of) curl to call the API as follows: curl --unix-socket /var/run/docker.sock http:/containers/json where…
4
votes
1 answer

Akka Streams with Alpakka indexing in ES: index name is only evaluated on starting execution

I've written some code with Akka Streams and Alpakka that reads from Amazon SQS and indexes the events in Elasticsearch. Everything works smoothly and the performance is awesome, but I have a problem with index names. I have this code: class…
SergiGP
  • 669
  • 7
  • 17
4
votes
2 answers

Manual Acknowledgement of ActiveMQ Messages with Alpakka

I am working on implementing Akka Alpakka for consuming from and producing to ActiveMQ queues, in Java. I can consume from the queue successfully, but I haven't yet been able to implement application-level message acknowledgement. My goal is to…
Scott S
  • 2,696
  • 16
  • 20
3
votes
0 answers

Does Kafka support creating projections from multiple category streams?

We are using EventStore for our event sourced solution. We would like to export events into Kafka and do stream processing using Kafka. In EventStore each type of stream belongs to a category, for example Order, Customer, Payment etc. Under each…
3
votes
1 answer

elastic search multi query with alpakka-elasticsearch client

I am new to scala, using alpakka-elasticsearch client to perform query with elastic search. Single search query is working fine with below two ways which internally calls to _search url of elastic search: val writeCustomIndex = ElasticsearchSource …
RKP
  • 750
  • 2
  • 12
  • 23
3
votes
2 answers

Out of memory with Sqs Akka Stream

Below code throws OOO on EC2 instance within 15 min of run (java config xms 1024 xmx2G) but doesn't throw any error when running on intellij. SqsSource(queueUrl, //parallelism = maxBufferSize / maxBatchSize 20 10 …
Mohammad Adnan
  • 6,527
  • 6
  • 29
  • 47
1
2 3
14 15