Questions tagged [stream-compaction]
23 questions
33
votes
2 answers
How to compress or compact a string in Python
I'm making a python "script" that sends a string to a webservice (in C#). I NEED to compress or compact this string, because the bandwidth and MBs data is LIMITED (yeah, in capitals because it's very limited).
I was thinking of converting it into a…
Santiago Mendoza Ramirez
- 1,497
- 2
- 13
- 26
15
votes
5 answers
How to test whether log compaction is working or not in Kafka?
I have made the changes in server.properties file in Kafka 0.8.1.1 i.e. added log.cleaner.enable=true and also enabled cleanup.policy=compact while creating the topic.
Now when I am testing it, I pushed the following messages to the topic with…
tanmayghosh2507
- 773
- 3
- 12
- 31
11
votes
3 answers
CUDA stream compaction algorithm
I'm trying to construct a parallel algorithm with CUDA that takes an array of integers and removes all of the 0's with or without keeping the order.
Example:
Global Memory: {0, 0, 0, 0, 14, 0, 0, 17, 0, 0, 0, 0, 13}
Host Memory Result: {17, 13, 14,…
Dane Bouchie
- 421
- 5
- 11
8
votes
2 answers
What does cassandra do during compaction?
I know that cassandra merges sstables, row-keys, remove tombstone and all.
But i am really interested to know how it performs compaction ?
As sstables are immutable does it copy all the relevant data to new file? and while writing to this new file…
samarth
- 3,866
- 7
- 45
- 60
8
votes
1 answer
efficient way to convert scatter indices into gather indices?
I'm trying to write a stream compaction (take an array and get rid of empty elements) with SIMD intrinsics. Each iteration of the loop processes 8 elements at a time (SIMD width).
With SSE intrinsics, I can do this fairly efficiently with…
Yale Zhang
- 1,447
- 12
- 30
5
votes
2 answers
Thrust: Removing duplicates in key-value arrays
I have a pair of arrays of equal size, I will call them keys and values.
For example:
K: V
1: 99
1: 100
1: 100
1: 100
1: 103
2: 103
2: 105
3: 45
3: 67
The keys are sorted and the values associated with each key are
sorted. How do I remove the value…
Ashwin Nanjappa
- 76,204
- 83
- 211
- 292
2
votes
1 answer
Improving the Efficiency of Compact/Scatter in CUDA
Summary:
Any ideas about how to further improve upon the basic scatter operation in CUDA? Especially if one knows it will only be used to compact a larger array into a smaller one? or why the below methods of vectorizing memory ops and shared…
dada_dave
- 493
- 4
- 13
2
votes
1 answer
How to store large arrays of booleans in file with the most light size?
My program produces large arrays filled with booleans.
I need the most compact way to save those in file.
I read here http://www.kirupa.com/forum/showthread.php?339670-How-is-boolean-represented-in-memory that 8 booleans in memory may represented as…
Kosmo零
- 4,001
- 9
- 45
- 88
1
vote
0 answers
max.compaction.lag.ms for topics with retention of 7 days
I want to add compaction to our topics, which configured to keep logs for 7 days max.
I noticed the max.compaction.lag.ms has higher value than 7 days, what is the downside of setting this parameter to a lower value (around 12 hours) and is there…
eilon47
- 41
- 2
1
vote
1 answer
AArch64 SVE/2 - Left pack elements from list
I'm trying to implement a SIMD algorithm with AArch64 SVE (or SVE2) that takes a list of elements and only selects the ones that meet a certain condition. It's often called Left Packing (SSE/AVX/AVX-512), or Stream Compaction (CUDA)?
Is it possible…
him
- 99
- 1
- 6
1
vote
1 answer
HIVE 3.1 - Automatic Major compaction triggered only once per partition
I have an acid enabled, partitioned, bucketed hive table to which I am writing using a streaming client. I see that several delta files are created as the records are written into partitions. I wanted to enable auto-compaction and tried the…
irrelevantUser
- 1,172
- 18
- 35
1
vote
1 answer
OpenCL parallel buffer compaction barrier issue
As a school project, we're 4 working on a parallel raytracer with OpenCL.
It's our first project using OpenCL, so we might have some incomprehensions about it.
We're trying to implement parallel buffer compaction to remove finished rays, or rays…
elXor
- 11
- 2
1
vote
1 answer
Does git have a concept of log compaction?
The git version control system, is a kind of distributed log (with some conceptual similarities to the raft consensus protocol).
Raft and some other systems have a concept of log compaction, so new clients don't need to traverse the whole change…
hawkeye
- 34,745
- 30
- 150
- 304
1
vote
1 answer
JSON log file compaction
Given a file with line delimited JSON records:
{"id": 1, "position": 1234}
{"id": 2, "position": 23}
{"id": 3, "position": 43}
{"id": 1, "position": 223}
I would like to compact such a file, by keeping only the last record for an id, e.g. for the…
miku
- 181,842
- 47
- 306
- 310
1
vote
1 answer
Stream compaction within cuda kernel for maintaining priority queue
I am looking for the optimisation strategy for my cuda program. At each iteration inside the for loop of my kernel, each thread produces a score. I am maintaining a shared priority queue of the scores to maintain top-k of them per block. Please see…
Chirag Jain
- 143
- 2
- 8