32 gb EC2 with Amazon Linux 2 AMI throws memory error for 9.6 gb Pandas dataframe

Asked Dec 03 '20 at 18:45

Active Dec 04 '20 at 06:59

Viewed 396 times

I am working with a 32 Gb EC2 (r5a.xlarge) with the Amazon Linux 2 AMI 2.0.20201126.0 x86_64 HVM gp2. Currently, my Pandas Data Frame has a shape of (36072, 36072) - which translates to 9.69 GiB with the float64 datatype.

Am confused why the EC2 throws the below Memory Error

MemoryError: Unable to allocate 9.69 GiB for an array with shape (36072, 36072) and data type float64

I faced this issue, first, when i ran a smaller EC2 instance (16 gb RAM / t3a.xlarge) today as well. Quite lost how to get this resolved...

I noticed that quite a few Linux posts talk about overcommit mode. One such SO post is here - https://stackoverflow.com/a/57511555/3228300

However, EC2 says I don't have access to set the overcommit mode.

edited Dec 04 '20 at 06:59

asked Dec 03 '20 at 18:45

vsdaking

[Check This Page Out](https://stackoverflow.com/questions/57507832/unable-to-allocate-array-with-shape-and-data-type) – TheLazyScripter Dec 03 '20 at 18:48
Thanks for the link - had seen it just before I posted this. Even did the steps that @Iguananaut showed. Ec2 gives me a `-bash: /proc/sys/vm/overcommit_memory: Permission denied` – vsdaking Dec 03 '20 at 19:10
Are you `root`? – TheLazyScripter Dec 03 '20 at 19:11
no, am using `ec2-user` as my current user. thought ec2 & aws do not allow `root` user access, right? – vsdaking Dec 03 '20 at 19:16
1

It appears that you can, but it's not recommended. It also appears that you can manage roles for credentials and it may be possible to allow that specifically. Does `ec2-user` have access to `sudo`? – TheLazyScripter Dec 03 '20 at 19:20
aye! i can certainly use `sudo` – vsdaking Dec 03 '20 at 19:37

32 gb EC2 with Amazon Linux 2 AMI throws memory error for 9.6 gb Pandas dataframe

0 Answers0