Prevent my RAM memory from reaching 100%

Question

I have a very simple python script that reads a CSV file and sorts the rows according to the timestamps. However, the file is large enough (16 GB) that its reading uses ram memory completely. When it reaches 100% (i.e. 64 GB RAM memory), my system completely freezes, and I am forced to restart my computer.

Here is the code:

import pandas as pd
from time import time

filename = 'AKER_OB.csv'

start_ = time()
file_ = pd.read_csv(filename)
end_ = time()
duration = end_ - start_
print("The duration to load that file : {}".format(duration))

file_.to_datetime(df['TimeStamps'], format="%Y-%m-%d %H:%M:%S").sort_values()

Head of AKER_OB.csv :

TimeStamp,Bid1,BidSize1,Bid2,BidSize2,Bid3,BidSize3,Bid4,BidSize4,Bid5,BidSize5,Bid6,BidSize6,Bid7,BidSize7,Bid8,BidSize8,Bid9,BidSize9,Bid10,BidSize10,Bid11,BidSize11,Bid12,BidSize12,Bid13,BidSize13,Bid14,BidSize14,Bid15,BidSize15,Bid16,BidSize16,Bid17,BidSize17,Bid18,BidSize18,Bid19,BidSize19,Bid20,BidSize20,Ask1,AskSize1,Ask2,AskSize2,Ask3,AskSize3,Ask4,AskSize4,Ask5,AskSize5,Ask6,AskSize6,Ask7,AskSize7,Ask8,AskSize8,Ask9,AskSize9,Ask10,AskSize10,Ask11,AskSize11,Ask12,AskSize12,Ask13,AskSize13,Ask14,AskSize14,Ask15,AskSize15,Ask16,AskSize16,Ask17,AskSize17,Ask18,AskSize18,Ask19,AskSize19,Ask20,AskSize20
2016-10-08 00:00:00,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
2016-10-08 00:00:01,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
2016-10-08 00:00:02,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
2016-10-08 00:00:03,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
2016-10-08 00:00:04,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
2016-10-08 00:00:05,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
2016-10-08 00:00:06,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
2016-10-08 00:00:07,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
2016-10-08 00:00:08,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0

What is the correct way to fix this problem? A full answer with code snippet will be appreciated.

https://unix.stackexchange.com/questions/44985/limit-memory-usage-for-a-single-linux-process — Charles Shiller, May 05 '18 at 01:27
The correct way to solve this problem is to stop trying to read the entire file into memory all at once. — Ken White, May 05 '18 at 01:55
@KenWhite Interesting! What do you suggest as a python approach? — Jeremie, May 05 '18 at 02:36
on macos/linux could you do it via the os, rather than python? **sort**, with a little bit of **tail** or **head** wizardry to separate out the header line. — JL Peyret, May 05 '18 at 14:08

score 1 · Answer 1 · answered May 05 '18 at 02:41

1

Essentially, you have to implement your own out-of-memory sorting.

Split your file in two or more pieces with Pandas CSV chunker, sort each piece (one piece at a time!), save it into a separate CSV file, and free the memory with del.
Merge the sorted files by opening all of the saved pre-sorted files with CSV chunkers, combining the rows from the chunks, as needed, and appending the sorted rows to the output file.

answered May 05 '18 at 02:41

DYZ

55,249
10
64
93

Are you up to build a code snippet? I will give you the header of `AKER_OB.csv` – Jeremie May 05 '18 at 02:46
It's not a trivial program to write. As SO is not a code writing service, I'd rather not. – DYZ May 05 '18 at 02:47

jesugmz · Answer 2 · 2018-05-24T23:33:33.147

0

Just split the read of the file by chunks. A similar case.

Also consider to add a swap partition or file to your OS, it will help the issue to be out of RAM for other situations.

edited May 24 '18 at 23:33

answered May 05 '18 at 02:39

jesugmz

2,320
2
18
33

Reading the chunks is the first and the easiest part of the problem. Combining the sorted pieces is harder. – DYZ May 05 '18 at 02:42

Prevent my RAM memory from reaching 100%

2 Answers2