Stack Overflow
Stack Overflow

Questions tagged [high-volume]

14 questions
7
votes
3 answers

What happens if a message is rolled back in MQ?

I receive a message from a WebSPhere MQ queue. I try to process and if i receive some exceptions i want to rollback the message to the MQ Queue. I have no problems in doing the same. What happens to the message? Does it go to the bottom of the…
Manglu
  • 10,744
  • 12
  • 44
  • 57
5
votes
1 answer

What allocating method to use for a high volume logger application?

I'm developing a logger/sniffer using Delphi. During operation I get hugh amounts of data, that can accumulate during stress operations to around 3 GB of data. On certain computers when we get to those levels the application stops functioning and…
Ran
  • 391
  • 1
  • 4
  • 10
5
votes
2 answers

Time Calendar Data Structure

We are looking at updating (rewriting) our system which stores information about when people can reserve rooms etc. during the day. Right now we store the start and time and the date the room is available in one table, and in another we store the…
kemiller2002
  • 113,795
  • 27
  • 197
  • 251
2
votes
0 answers

Antlr4 Performance Issue for high volume files

We are facing issues with antlr performance for parsing Oracle files. The oracle files for conversion has significant large size 17/24/38 mb files. While building the parse-tree, it taking a lot of time and memory. Its even giving core OOM dump. We…
user2905570
  • 21
  • 3
1
vote
0 answers

Oracle - Exchange partitions between two List partitioned tables

Need some help here. I have an ETL process which loads the data into a target table A. We have created another table B which is same as the target table in structure and this table is accessed by reporting team to generate reports. This is done to…
Manish Mishra
  • 51
  • 9
1
vote
1 answer

Database design with millions of entry

Suppose there is a messaging system. This system has millions of entry to be sent and get reported and the count is growing by 100K every hour. 2 service accesses db, one is sender, one is reporter. So what would you suggest in order to get maximum…
whoi
  • 3,281
  • 7
  • 36
  • 44
1
vote
2 answers

Which NoSQL Database for Mostly Writing

I'm working on a system that will generate and store large amounts of data to disk. A previously developed system at the company used ordinary files to store its data but for several reasons it became very hard to manage. I believe NoSQL databases…
Elektito
  • 3,863
  • 8
  • 42
  • 72
0
votes
1 answer

Logging statistics for high-volume Java-based API service requests

I want to be able to log usage statistics for a Java-based API service. The estimated request rate is between 10 and 300 per minute, and will potentially grow to a much higher number. I do not need detailed analytics, just a simple hit counter that…
settinghead
  • 95
  • 1
  • 6
0
votes
3 answers

Handle very large throughput with external database

I'm looking to build a Java11 Spring boot application. The application is to handle very large throughput (will have peaks and low traffic) The happy path of the application looks like this. Conceptually its fairly straight forward. The steps…
Robbo_UK
  • 11,351
  • 25
  • 81
  • 117
0
votes
2 answers

Processing huge volume data file using powershell

I am trying to perform a replace operation on a data file which is 4GB. But I am not even able to read this file due to memory exception. The following command gives a memory error. $edwfile = (Get-Content…
TomG
  • 281
  • 1
  • 2
  • 20
0
votes
3 answers

In NetSuite using SuiteTalk, Is it possible to create a CSV file from saved search

Backgorund: I am a newbie in the NetSuite world. We are trying to integrate NetSuite with our ERP and I am doing some preliminary research to find out what would be the best option moving ahead. The primary objective of the first task is to download…
Anit Shrestha
  • 211
  • 3
  • 11
0
votes
0 answers

Cassandra fail to write 500MB data

I am trying to write 2GB(which is limit of Cassandra single key/value) data into single(or many) column using Datastax driver,CQL3 on one machine windows node.I am hardly able to write data like 100MB(in single column), that too by facing almost all…
Deepak Dabi
  • 1
  • 2
0
votes
1 answer

Starting a large number of async processes with multiprocessing

If I call apply_async 10,000 times, assuming the OOM-killer doesn't interfere, will multiprocessing start them all simultaneously, or will it start them in batches. For example.. Every 100 starts, waiting for 90 to finish starting before starting…
Dustin Oprea
  • 9,673
  • 13
  • 65
  • 105
-1
votes
1 answer

Reporting a huge volume of data

We run a SAAS product. One of our customer is an enterprise client and while others generate 8-10,000 rows of data for a bill report for 6 months. They generate about 300,000 rows of data. Each row has 30 columns. So when they try to goto our…
user1763470
  • 197
  • 1
  • 1
  • 11