Stack Overflow
Stack Overflow

Questions tagged [spring-batch-stream]

30 questions
5
votes
1 answer

Dynamically changing the instanceindex with spring cloud stream kafka

Similar to: Changing spring-cloud-stream instance index/count at runtime I do a poc on the launch of batch in a microserver architecture and I am using Spring batch with Spring Cloud Stream Kafka. I am looking for a way to dynamically create…
katy
  • 61
  • 4
5
votes
1 answer

can we process the multiple files sequentially using spring Batch while multiple threads used to process individual files data..?

I want to process multiple files sequentially and each file needs to be processed with the help of multiple threads so used the spring batch FlatFileItemReader and TaskExecutor and it seems to be working fine for me. As mentioned in the…
chandranshu
  • 53
  • 1
  • 6
3
votes
1 answer

In spring batch how can we move the processed files to another folder, I am using MultiResourceItemReader and chunk processing

In Spring batch application how can we move files from BatchFileDir folder to a destination folder as each file gets processed. There would be a bunch of files in BatchFileDir folder so I am using MultiResourceItemReader. I am also following the…
SamA
  • 43
  • 6
2
votes
1 answer

Spring Batch Job not working, Spring stopping execution

I am very new to the spring batch. Trying to create a simple example for myself to understand some of the basic batch-processing concepts. Currently, my spring application starts but unable to execute a simple job which is as follows Read from the…
Mohasin Kazi
  • 65
  • 5
2
votes
1 answer

Spring Batch- how to pass list of multiple items from input to ItemReader, ItemProcessor and ItemWriter

I have a simple csv file that i am reading in chunk of 1000, inserting to database. Now if i want to check if the row exists in db and is equal to input before insert, if exists and row is equal -ignore, else insert or update, I am using…
tryCatch
  • 23
  • 1
  • 6
2
votes
1 answer

How to resolve ClassCastException in MultiResourceItemReader Spring Batch

I'm reading multiple files from the S3 bucket using MultiResourceItemReader, I'm getting ClassCastException before executing the myReader() method, Something wrong with MultiResourceItemReader not sure what's going wrong here. Please find my code…
Mike Marsh
  • 387
  • 3
  • 15
2
votes
1 answer

spring batch getting stuck in parallel processing where works fine in serial processing

I am quite new to Spring Batch and tried to run Spring batch with single thread. Now I need to add multithreading in step and have below configuration, but parallel processing is getting hang after some time and no trace on console after it…
user2025527
  • 109
  • 2
  • 12
1
vote
0 answers

Spring Batch HibernatePagingItemReader / JpaPagingItemReader Paging OFFSET breaks after few 100K pages

What's the usecase? Our usecase is to take 100Million records from Postgres Materialized view and write it in 10K chunks into Redis Cache. What we tried: We used HibernatePagingItemReader to read the records from Postgres Materialized view in…
ArunSelvam P M
  • 429
  • 5
  • 6
1
vote
1 answer

Copy header tag in xml spring batch application

I am using spring-batch in spring-boot application. The Spring Boot version is 2.3.3.RELEASE. What I intend to achieve I have to read a xml file containing thousands of Transactions with header tag (fileInformation). Do some business logic on…
truekiller
  • 470
  • 6
  • 19
1
vote
0 answers

Spring Batch:Non-skippable exception in recoverer while processing;java.lang.NullPointerException at FaultTolerantChunkProcessor.java:289

I have one step configurations as below stepBuilderFactory.get("step2") .chunk(100) .reader(eventLogReader) .processor(eventLogItemProcessor) …
Chetanti Mehta
  • 21
  • 6
0
votes
0 answers

Spring Batch not making full use of multi-core CPU

I have a spring batch application which reads lines from a flat file, converts each line to an object, processes them, and writes them to a database. The steps are multi-threaded using spring's ThreadPoolTaskExecutor, but the performance is still…
ethereal
  • 13
  • 2
0
votes
1 answer

Spring Batch - How to make multi threaded Reader, Processor and Writer?

What is the requirement? I'm using the Spring batch application for Bulk File Processing. Steps: Reading the file using network calls. Preparting a bulk JSON Payload and calling endpoint. Writing responses to files. What is going wrong? Everything…
Zahid Khan
  • 2,130
  • 2
  • 18
  • 31
0
votes
1 answer

some questions about purpose of both ItemStreamReader and ItemStreamWriter in Spring Batch

I'm studying about Spring Batch. I'm using ItemReader and ItemWriter in my Spring Batch Project. However, my project's the biggest problem is that all data reading logic is in constructor without paging. I thought it's so unusual and improper…
Companion Cube
  • 15
  • 2
0
votes
0 answers

Spring Batch JdbcPagingItemReader not fetching all records from DB

Spring Batch JdbcPagingItemReader not fetching all records from DB. As in Our database we dont have primary key. so it not fetching all records. Without primary we are losing data. So we are trying to sort column with combination of 3 columns to…
Gaurav Khandelwal
  • 360
  • 2
  • 12
0
votes
1 answer

Spring Batch Dynamic Insert and Update with DB to DB

There is scenario where we want to load data from DB to DB . But we want to check if data is already present in target system then update it otherwise insert it into DB. We are using below approach: @Bean ItemWriter onosItemWriter1() { …
Gaurav Khandelwal
  • 360
  • 2
  • 12
1
2