Stack Overflow
Stack Overflow

Questions tagged [retention]

When data is to be retained, downsampled, or eliminated from a: database, filesystem or other storage device use 'retention'.

203 questions
88
votes
5 answers

changing kafka retention period during runtime

With Kafka 0.8.1.1, how do I change the log retention time while it's running? The documentation says the property is log.retention.hours, but trying to change it using kafka-topics.sh returns this error $ bin/kafka-topics.sh --zookeeper…
Foo L
  • 10,977
  • 8
  • 40
  • 52
50
votes
5 answers

How to see the retention for a particular topic in kafka

I want to see the retention period set for a particular topic. Is there any command? I tried with bin/kafka-topics.sh --zookeeper hostname:2181 --alter --config retention.ms=172800000 --topic for deleting 2 days old data. But I want…
manoj
  • 601
  • 1
  • 5
  • 3
19
votes
3 answers

What's the reason for java.lang.annotation.Retention?

I'm well aware about the meaning of the RetentionPolicy and know what they do and when it seems to make sense to use them. For my own annotation, I exactly know if they're needed at runtime, in class files, or just for the compilation. However, with…
maaartinus
  • 44,714
  • 32
  • 161
  • 320
15
votes
5 answers

How to test whether log compaction is working or not in Kafka?

I have made the changes in server.properties file in Kafka 0.8.1.1 i.e. added log.cleaner.enable=true and also enabled cleanup.policy=compact while creating the topic. Now when I am testing it, I pushed the following messages to the topic with…
tanmayghosh2507
  • 773
  • 3
  • 12
  • 31
14
votes
4 answers

Log4j daily rotation and monthly retention with any file appender

Is it possible with any of the log4j appenders to write the logs with specific rotation time and retention limit. The goal would be: to have a log file for each day; create a new file at midnight for the new logs to keep the log files and delete…
PaulEdison
  • 897
  • 1
  • 15
  • 36
10
votes
3 answers

Delete old Windows Azure Diagnostics data from table storage (performance counters, etc.)

I have several Windows VMs running on Azure that are configured to collect performance counters and event logs. All of this is configured in the "Diagnostic settings..." on the VM resource inside Azure Portal. There's a Windows Azure Diagnostics…
Tom Pažourek
  • 9,582
  • 8
  • 66
  • 107
10
votes
1 answer

Difference between retention configuration offsets.retention.minutes and log.retention.minutes

What is the difference between the following two retention configurations? offsets.retention.minutes log.retention.minutes I don't get how it differs or relate to each other. From my understanding, once the offset is removed, the record in the log…
Nicolas Henneaux
  • 11,507
  • 11
  • 57
  • 82
9
votes
3 answers

Kafka log.segment.bytes vs log.retention.hours

I was following the book "Kafka: The Definitive Guide" First Edition to understand when log segments are deleted by the broker. As per the text I understood, a segment will not become eligible for deletion until it is closed. A segment can be closed…
Mayank Madhav
  • 429
  • 1
  • 7
  • 19
7
votes
3 answers

calculating simple retention in R

For the dataset test, my objective is to find out how many unique users carried over from one period to the next on a period-by-period basis. > test user_id period 1 1 1 2 5 1 3 1 1 4 3 1 5 …
iskandarblue
  • 7,208
  • 15
  • 60
  • 130
7
votes
3 answers

Graylog2- how to config logs retention to 1 week

We are using some Graylog2 servers ( graylog-server version 1.3.4). Because we receive too much of log messages, it requires a lot of memory. I am trying to reduce the logs retention to 1 week, every log messages older than 1 week will be deleted.…
SWdream
  • 215
  • 2
  • 3
  • 10
6
votes
3 answers

why my Loki log retention is not working?

"12h" old logs are not deleting even after specifying "retention_period: 12h" due to this I'm facing storage issue because, log are not deleting. please help me with config loki: enabled: true isDefault: true table_manager: …
Vinayak Pawar
  • 93
  • 1
  • 4
6
votes
1 answer

InfluxDB : How to update duration of an existing database

I have a database created with default retention policy which is infinite. # SHOW RETENTION POLICIES ON "my_database" name duration shardGroupDuration replicaN default ---- -------- ------------------ -------- ------- autogen 0s 168h0m0s…
kmonsoor
  • 7,600
  • 7
  • 41
  • 55
6
votes
2 answers

Generating a retention cohort from a pandas dataframe

I have a pandas dataframe that looks like this: +-----------+------------------+---------------+------------+ | AccountID | RegistrationWeek | Weekly_Visits | Visit_Week | +-----------+------------------+---------------+------------+ | ACC1 |…
grzlybear
  • 61
  • 1
  • 4
5
votes
3 answers

File retention mechanism in a large data storage

recently I faced performance problem with mp4 files retention. I have kind a recorder which saves 1 min long mp4 files from multiple RTSP streams. Those files are stored on external drive in file tree like…
przemoch
  • 311
  • 1
  • 10
5
votes
1 answer

How to delete files from blob container that are older than 30 days?

I am looking for a way to delete stored files in a blob container called reports that are older than 30 days or set any other retention policy that would make sense As far as I see Microsoft only writes about retention policies for Logs and…
Yoda
  • 17,363
  • 67
  • 204
  • 344
1
2 3
…
13 14