Stack Overflow
Stack Overflow

Questions tagged [chunking]

1. In memory management, chunking refers to strategies for improving performance by aggregating related memory-allocation requests. 2. In HTTP message transmission, it refers to a facility that allows inconveniently large messages to be broken into conveniently-sized smaller "chunks." 3. In parallel computing, it refers to the amount of data to assign to each task.

369 questions
140
votes
20 answers

how to split an iterable in constant-size chunks

I am surprised I could not find a "batch" function that would take as input an iterable and return an iterable of iterables. For example: for i in batch(range(0,10), 1): print i [0] [1] ... [9] or: for i in batch(range(0,10), 3): print…
mathieu
  • 2,954
  • 2
  • 20
  • 31
101
votes
10 answers

What is the best way to chop a string into chunks of a given length in Ruby?

I have been looking for an elegant and efficient way to chunk a string into substrings of a given length in Ruby. So far, the best I could come up with is this: def chunk(string, size) …
MiniQuark
  • 46,633
  • 36
  • 147
  • 183
85
votes
11 answers

Is there an elegant way to process a stream in chunks?

My exact scenario is inserting data to database in batches, so I want to accumulate DOM objects then every 1000, flush them. I implemented it by putting code in the accumulator to detect fullness then flush, but that seems wrong - the flush control…
Bohemian
  • 412,405
  • 93
  • 575
  • 722
41
votes
8 answers

Slice chunking in Go

I have a slice with ~2.1 million log strings in it, and I would like to create a slice of slices with the strings being as evenly distributed as possible. Here is what I have so far: // logs is a slice with ~2.1 million strings in it. var divided =…
Sienna
  • 1,570
  • 3
  • 24
  • 48
41
votes
1 answer

Python fastest way to read a large text file (several GB)

i have a large text file (~7 GB). I am looking if exist the fastest way to read large text file. I have been reading about using several approach as read chunk-by-chunk in order to speed the process. at example effbot suggest # File:…
Gianni Spear
  • 7,033
  • 22
  • 82
  • 131
39
votes
1 answer

What is HTML5 File.slice method actually doing?

I'm working with a custom API to allow a user to upload a file (of, hopefully, arbitrary size). If the file is to large, it will be chunkfied, and handled in multiple requests to the server. I'm writing code that uses File and FileReader (HTML5) as…
Ponml
  • 2,917
  • 5
  • 20
  • 17
33
votes
3 answers

HTTP Chunked Encoding. Need an example of 'Trailer' mentioned in SPEC

I am writing an HTTP parser for a transparent proxy. What is stumping me is the Trailer: mentioned in the specs for Transfer-Encoding: chunked. What does it look like? Normally, a HTTP chunked ends like this. 0\r\n \r\n What I am confused about is…
unixman83
  • 9,421
  • 10
  • 68
  • 102
23
votes
5 answers

Convert a Lazy ByteString to a strict ByteString

I have a function that takes a lazy ByteString, that I wish to have return lists of strict ByteStrings (the laziness should be transferred to the list type of the output). import qualified Data.ByteString as B import qualified Data.ByteString.Lazy…
Matt Joiner
  • 112,946
  • 110
  • 377
  • 526
22
votes
3 answers

Splitting a File into Chunks with Javascript

I'm trying to take a single file object and split it into chunks by a specified chunk size. In my example, trying to split a single file into 1MB chunks. So I figure out how many chunks it would take, then I'm trying to slice the file starting from…
Nico Galluzzo
  • 485
  • 1
  • 3
  • 8
20
votes
4 answers

Large File uploading to asp.net MVC

I need a way to upload large files (600 mb to 4 gb) in an asp.net mvc website. Currently I am using swfupload; it works well enough, but it is a huge hit on the webserver because it sends it in one big upload, plus I have to set it in the…
Solmead
  • 4,158
  • 2
  • 26
  • 30
20
votes
1 answer

How to calculate the optimum chunk size for uploading large files

Is there such a thing as an optimum chunk size for processing large files? I have an upload service (WCF) which is used to accept file uploads ranging from several hundred megabytes. I've experimented with 4KB, 8KB through to 1MB chunk sizes. …
Fixer
  • 5,985
  • 8
  • 40
  • 58
18
votes
4 answers

In Clojure, are lazy seqs always chunked?

I was under the impression that the lazy seqs were always chunked. => (take 1 (map #(do (print \.) %) (range))) (................................0) As expected 32 dots are printed because the lazy seq returned by range is chunked into 32 element…
Geo G
  • 345
  • 3
  • 11
17
votes
6 answers

Can I turn off create-react-app chunking mechanism?

I am setting up my React app project using create-react-app. I was wondering if there is a way to turn-off the chunking mechanism that is built-in into the react scripts. The thing is that I need to fix the name of the bundle created on the build.
Darko
  • 525
  • 1
  • 4
  • 14
16
votes
5 answers

Transferring large payloads of data (Serialized Objects) using wsHttp in WCF with message security

I have a case where I need to transfer large amounts of serialized object graphs (via NetDataContractSerializer) using WCF using wsHttp. I'm using message security and would like to continue to do so. Using this setup I would like to transfer…
jpierson
  • 16,435
  • 14
  • 105
  • 149
15
votes
3 answers

How do I avoid Clojure's chunking behavior for lazy seqs that I want to short circuit?

I have a long, lazy sequence that I want to reduce and test lazily. As soon as two sequential elements are not = (or some other predicate) to each other, I want to stop consuming the list, which is expensive to produce. Yes, this sounds like…
ivar
  • 1,484
  • 12
  • 20
1
2 3
…
24 25