Stack Overflow
Stack Overflow

Questions tagged [large-data-volumes]

302 questions
75
votes
8 answers

Designing a web crawler

I have come across an interview question "If you were designing a web crawler, how would you avoid getting into infinite loops? " and I am trying to answer it. How does it all begin from the beginning. Say Google started with some hub pages say…
xyz
  • 8,607
  • 16
  • 66
  • 90
59
votes
12 answers

Using Hibernate's ScrollableResults to slowly read 90 million records

I simply need to read each row in a table in my MySQL database using Hibernate and write a file based on it. But there are 90 million rows and they are pretty big. So it seemed like the following would be appropriate: ScrollableResults results =…
at.
  • 50,922
  • 104
  • 292
  • 461
36
votes
8 answers

Is it possible to change argv or do I need to create an adjusted copy of it?

My application has potentially a huge number of arguments passed in and I want to avoid the memory of hit duplicating the arguments into a filtered list. I would like to filter them in place but I am pretty sure that messing with argv array itself,…
ojblass
  • 21,146
  • 22
  • 83
  • 132
33
votes
8 answers

large amount of data in many text files - how to process?

I have large amounts of data (a few terabytes) and accumulating... They are contained in many tab-delimited flat text files (each about 30MB). Most of the task involves reading the data and aggregating (summing/averaging + additional…
hatmatrix
  • 42,883
  • 45
  • 137
  • 231
29
votes
9 answers

Plotting of very large data sets in R

How can I plot a very large data set in R? I'd like to use a boxplot, or violin plot, or similar. All the data cannot be fit in memory. Can I incrementally read in and calculate the summaries needed to make these plots? If so how?
Daniel Arndt
  • 2,268
  • 2
  • 17
  • 22
24
votes
7 answers

Efficiently storing 7.300.000.000 rows

How would you tackle the following storage and retrieval problem? Roughly 2.000.000 rows will be added each day (365 days/year) with the following information per row: id (unique row identifier) entity_id (takes on values between 1 and 2.000.000…
knorv
  • 49,059
  • 74
  • 210
  • 294
24
votes
2 answers

JDBC Batch Insert OutOfMemoryError

I have written a method insert() in which I am trying to use JDBC Batch for inserting half a million records into a MySQL database: public void insert(int nameListId, String[] names) { String sql = "INSERT INTO name_list_subscribers…
craftsman
  • 15,133
  • 17
  • 70
  • 86
21
votes
2 answers

Docker Data Volume Container - Can I share across swarm

I know how to create and mount a data volume container to multiple other containers using --volumes-from, but I do have a few questions regarding it's usage and limitations: Situation: I am looking to use a data volume container to store user…
deankarn
  • 462
  • 2
  • 6
  • 17
21
votes
4 answers

what changes when your input is giga/terabyte sized?

I just took my first baby step today into real scientific computing today when I was shown a data set where the smallest file is 48000 fields by 1600 rows (haplotypes for several people, for chromosome 22). And this is considered tiny. I write…
Wang
  • 3,247
  • 1
  • 21
  • 33
20
votes
4 answers

How to do page navigation for many, many pages? Logarithmic page navigation

What's the best way of displaying page navigation for many, many pages? (Initially this was posted as a how-to tip with my answer included in the question. I've now split my answer off into the "answers" section below). To be more…
Doin
  • 7,545
  • 4
  • 35
  • 37
20
votes
2 answers

Bad idea to transfer large payload using web services?

I gather that there basically isn't a limit to the amount of data that can be sent when using REST via a POST or GET. While I haven't used REST or web services it seems that most services involve transferring limited amounts of data. If you want…
Marcus Leon
  • 55,199
  • 118
  • 297
  • 429
18
votes
6 answers

How to avoid OOM (Out of memory) error when retrieving all records from huge table?

I am given a task to convert a huge table to custom XML file. I will be using Java for this job. If I simply issue a "SELECT * FROM customer", it may return huge amount of data that eventually causing OOM. I wonder, is there a way i can process the…
janetsmith
  • 8,562
  • 11
  • 58
  • 76
16
votes
5 answers

Transferring large payloads of data (Serialized Objects) using wsHttp in WCF with message security

I have a case where I need to transfer large amounts of serialized object graphs (via NetDataContractSerializer) using WCF using wsHttp. I'm using message security and would like to continue to do so. Using this setup I would like to transfer…
jpierson
  • 16,435
  • 14
  • 105
  • 149
12
votes
11 answers

Fastest way to search a 1 GB+ string of data for the first occurrence of a pattern

There's a 1 gigabyte string of arbitrary data which you can assume to be equivalent to something like: 1_gb_string=os.urandom(1*gigabyte) We will be searching this string, 1_gb_string, for an infinite number of fixed width, 1 kilobyte patterns,…
user213060
  • 1,249
  • 3
  • 19
  • 25
11
votes
7 answers

Fastest way for inserting very large number of records into a Table in SQL

The problem is, we have a huge number of records (more than a million) to be inserted into a single table from a Java application. The records are created by the Java code, it's not a move from another table, so INSERT/SELECT won't help. Currently,…
Iravanchi
  • 5,139
  • 9
  • 40
  • 56
1
2 3
…
20 21