Stack Overflow
Stack Overflow

Questions tagged [erasure-code]

Forward error correction (FEC) code for the binary erasure channel, which transforms a message of k symbols into a longer message (code word) with n symbols such that the original message can be recovered from a subset of the n symbols

In information theory, an erasure code is a forward error correction (FEC) code for the binary erasure channel, which transforms a message of k symbols into a longer message (code word) with n symbols such that the original message can be recovered from a subset of the n symbols. The fraction r = k/n is called the code rate, the fraction k’/k, where k’ denotes the number of symbols required for recovery, is called reception efficiency.

19 questions
2
votes
0 answers

How to use Ronomon Reed-Solomon Library to split file in chunks

I am trying to use Ronomon/Reed-Solomon JavaScript Library, I want to chunk files based on the number of data shards and parity shards. Any example demonstrating the use of ReedSolomon.encode will help me. My code is as below: Encode.js var…
N_b26
  • 43
  • 7
1
vote
1 answer

Is there solution for viewing chunks distribution in ceph's pools with erasure code?

According to the picture from this page ceph splits objects on chunks and writes them on osds. For viewing objects distribution with replication algorithm I can use commands like ceph pg dump or ceph osd map. But I can't find commands to view data…
anton_t
  • 13
  • 2
1
vote
1 answer

SeaweedFS EC and cold data

During testing seaweedfs, questions began to appear. Can you explain me, what is hot, warm and cold data in seaweedfs logic? And second question about erasurecoding: does it work only with warm data? Cannot I use it for hot or cold? If I don't have…
krakazyabra
  • 31
  • 1
  • 4
1
vote
1 answer

Hadoop 3 : how to configure / enable erasure coding?

I'm trying to setup an Hadoop 3 cluster. Two questions about the Erasure Coding feature : How I can ensure that erasure coding is enabled ? Do I still need to set the replication factor to 3 ? Please indicate the relevant configuration properties…
Klun
  • 78
  • 2
  • 25
1
vote
1 answer

Hadoop 3.0 erasure coding - determining the number of acceptable node failures?

In hadoop 2.0 the default replication factor is 3. And the number of node failures acceptable was 3-1=2. So on a 100 node cluster if a file was divided in to say 10 parts (blocks), with replication factor of 3 the total storage blocks required are…
samshers
  • 1
  • 6
  • 37
  • 84
1
vote
1 answer

How to configure the erasure coding feature in hadoop3 and is it used for storing cold files only by default?

As per the Hadoop 3.x release notes, they have introduced Erasure coding to overcome the problems with storage. Erasure coding is a method for durably storing data with significant space savings compared to replication. Standard encodings like…
Nitesh Gupta
  • 13
  • 3
1
vote
1 answer

HDFS Erasure Coding File Creation

I am looking for a way to create a file (e.g. using copyFromLocal) in Apache HDFS and set the Erasure Coding policy in the process. According to this page, I can use hdfs ec --setPolicy -path -policy RS-6-3-1024k to set the policy for a…
vaporishwhale
  • 46
  • 7
0
votes
0 answers

Simple encoding and decoding: Fulcrum network coding

I have been trying to carry out the basic implementation of fulcrum network coding, but my issue is how the encode the already precoded GF(28) packets using GF(2). To make my problem clearer, look at the following examples. p1 to p4 are the original…
Patrick Enenche
  • 1
  • 1
0
votes
0 answers

Install functionality in Hadoop Cluster in Docker

what I'm trying to do is to use multiple datanodes on a single machine in order to test the erasure code features implemented by version 3.0. I'm using Docker with Big Data Europe's containers (https://github.com/big-data-europe/docker-hadoop) and a…
Alessandro Muscatello
  • 15
  • 6
0
votes
1 answer

survey about the encoding techniques used in cloud computing

I am trying to perform a survey on the encoding techniques which can be applied to a file before storing it in the cloud. So far I have only encountered Reed Solomon erasure coding mechanism, in literature. I am wondering if someone can point out…
bhavs
  • 2,091
  • 8
  • 36
  • 66
0
votes
0 answers

why ceph CLAY plugin behaves poorly in repairing data?

Test in my virtual machine Ubuntu18, single host equipped with 15 osds Parameter as follows: (n,k)=(14,10), d=13 so if I shut down an osd, the only left one will be chosen to store the repaired data, without extra data migration Comprision with…
ApicTimer
  • 1
0
votes
1 answer

Python erasure coding library that can handle larger strings?

I'm looking for an python erasure coding library that works for larger inputs. So far I've checked out: unireedsolomon: fails for 256-byte inputs, unmaintained reedsolo/reedsolomon: fails for a 300-byte input silently. Reed-Solomon clearly a…
Zachary Vance
  • 752
  • 4
  • 18
0
votes
1 answer

What is erasure – what 4 restrictions does erasure place on generics?

Java - What is erasure – what 4 restrictions does erasure place on generics?
user718083
  • 25
  • 3
0
votes
0 answers

Using ReedSolomon library to erasure code a file

Using [ReedSolomon JavaScript Library][1] to erasure code a local file, it gives me following error : shardSize 5432 bufferoffset 0 bufferSize 5432 paritysize 10864 sources 3 targets 12 /Users/nidhisharma/client-sdk/Node/0chain_sdk/encode.js:56 var…
N_b26
  • 43
  • 7
0
votes
0 answers

How to use io.Pipe() for chunking a Large file using Erasure Coding through ReedSolomon Library

Using https://github.com/klauspost/reedsolomon, erasure coding a large file into smaller chunks, I want to use io.Pipe() to create a stream of pipes which connects each other. For example: one pipe which helps in Chunking the file and other which…
N_b26
  • 43
  • 7
1
2