EC2 instance on Amazon and I am greeted with "No space left on the disk"

Question

I have installed Amazon EC2 fedora instance and copying the files from one location to another. But I am greeted with " No space left on the disk".

I did df -f.

with output:

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              15G   15G     0 100% /
none                  312M     0  312M   0% /dev/shm

I want to increase the space for ec2 instance on amazon. Can someone help me with it?

score 86 · Answer 1 · edited Mar 22 '21 at 05:16

86

Here's an even easier method. (My m2.2xlarge instance was created with RedHat Linux 6.2, I discovered it had a paltry 6gb available of it's 850gb):

Via ssh, check space under root: $df -h

Filesystem            Size  Used Avail Use% Mounted on
/dev/xvde1              6G    6G    0G   100% /
none                   17G     0   17G   0% /dev/shm

From aws console, stop the instance
From aws console, detach the volume (though note the mount point under attachment info, eg /dev/sda1)
From aws console, take a snapshot of the volume
From aws console, create a new volume using the snapshot (using all the remaining space for the instance type, eg 825gb in my m2.2xlarge case)
From aws console, attach the new volume to original mount point /dev/sda1
From aws console, restart the instance and ssh back in to the instance
From ssh, run resize2fs on the root Filesystem (see df -h output in step 1) [potentially not needed]

$resize2fs /dev/xvde1

wait for a few minutes, possibly go and watch your buddy who is stopping all the root services etc like a boss : )
observe the new cavernous mount: $df -h

Filesystem            Size  Used   Avail Use%   Mounted on
/dev/xvde1            813G  3.7G    801G   1%     /
none                   17G     0     17G   0%     /dev/shm

edited Mar 22 '21 at 05:16

chris Frisina

19,086
22
87
167

answered Mar 15 '12 at 20:55

Rian Rizvi

9,989
2
37
33

9

Worked great for me, didn't even need the resize2fs part. Thanks! – Sitnik Feb 13 '15 at 20:53
1

Great solution. Works like a charm. Step 8 is not necessary. – Benedikt Schmeitz Nov 25 '18 at 12:33
I followed this answer now I can't access my EC2 instance. It is saying that Site Can't Be Reach. Does anyone have an idea where I was wrong? – Kevz Aug 09 '19 at 01:41
Even after re-attaching the volume I was still not able to do anything (because there was no space). After doing a ```sudo apt-get autoclean``` I was able to finally run ```sudo growpart /dev/nvme0n1 1``` – Johnny Apr 27 '20 at 14:39
How do you find the mount point? I can't find attachment info either. – Jason Melo Hall Aug 20 '21 at 18:09

Alf Bae · Answer 2 · 2019-04-09T12:47:13.527

38

Modify volume size. From AWS Console, you can modify the size of a volume.

From AWS console, open 'ELASTIC BLOCK STORE/Volume'
Select your volume and Modify volume(from Actions button)
Change size (e.g. 8 to 20gib)
Click Modify.
Reboot from EC2 Dashboard.
check size is changed by df -h

edited Apr 09 '19 at 12:47

answered Jan 24 '19 at 13:07

Alf Bae

546
6
10

There is no `reboot` option in EBS modify menu! – backslashN Apr 08 '19 at 10:24
3

You can reboot it from EC2 Dashboard. Click Actions button, Instance State, and Reboot. Cheers! – Alf Bae Apr 09 '19 at 12:45
4

This seemed the easiest and less risky way of solving a disk full problem. I tried it and it worked! One thing to note is that my EC2 wouldn't reboot because it was so full, I had to instance State > Stop and then instance state > start. – ParoX May 27 '20 at 02:16
I think that in case that you increase the EBS volume , it will recreate the instances – Carlos Andres Jul 23 '20 at 16:21
@ParoX, what do you mean by less risky? Genuine question. Thanks in advance. – arranjdavis Nov 09 '21 at 12:02
1

This should be the one marked as the correct answer. – Fernando Kosh Feb 01 '22 at 16:08

score 24 · Accepted Answer · edited Jan 21 '14 at 14:45

24

I got a solution guys yippeeee

Assuming that you are using a linux AMI, in your case you have an easy method for increasing the size of the file system:

1) Stop the instance

2) Detach the root volume

3) Snapshot the volume

4) Create a new volume from the snapshot using the new size

5) Attach the new volume to the instance on the same place where the original one was

6) Start the instance, stop all services except ssh and set the root filesystem read only

7) Enlarge the filesystem (using for example resize2fs) and or the partition if needed

8) Reboot

As an alternative you can also launch a new instance and map the instance storage or you can create a new ami combining the two previous steps.

edited Jan 21 '14 at 14:45

Josh Crowder

1,041
1
10
30

answered May 28 '11 at 11:37

onkar

1,149
4
12
14

1

Will the IP change? – Ajay May 16 '17 at 11:48
if it is private IP then yes. If public IP then no. – onkar Jun 07 '17 at 11:15
1

I believe stopping the instance changes the public IP. If you need to keep your IP, do your research before following these steps. – Harry Netzer Nov 01 '18 at 20:40
1

Use elastic IP to stop it from changing on reboot. – Anand Singh May 06 '19 at 20:02
For any future searchers, here is the [Amazon guide link](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/add-instance-store-volumes.html). – Tyler Chong Mar 26 '21 at 23:08

score 6 · Answer 4 · edited Feb 25 '19 at 18:33

6

find the biggest files with du -a | sort -n
stop some services, they might block hidden/removed files from being finally deleted.

edited Feb 25 '19 at 18:33

vmarquet

2,384
3
25
39

answered Jun 20 '13 at 14:14

Anona112

3,724
4
21
30

score 2 · Answer 5 · answered Jun 12 '14 at 18:11

2

I skipped all the detach/snapshot/new volume stuff... just did the resize.

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1              5904748   4725152   1119608  81% /
none                  15728640         0  15728640   0% /dev/shm
[root@ip-10-25-6-214 ~]# resize2fs /dev/sda1
resize2fs 1.39 (29-May-2006)
Filesystem at /dev/sda1 is mounted on /; on-line resizing required
Performing an on-line resize of /dev/sda1 to 31457280 (4k) blocks.

answered Jun 12 '14 at 18:11

user3735262

29
1

4

Hi welcome to SO. When answering questions please add more information about your solution so that others can easily replicate your steps. – Deepend Jun 12 '14 at 18:30
1

This answer is in the Low Quality Posts review queue because it's just code with no explanation. Please improve your answer by explaining what your code does and how it answers the question. Please read [this advice on answering programming questions helpfully](http://msmvps.com/blogs/jon_skeet/archive/2009/02/17/answering-technical-questions-helpfully.aspx). – Adi Inbar Jun 12 '14 at 18:45

score 1 · Answer 6 · answered Sep 05 '18 at 06:49

1

I mounted the disk on another EC2 instance where I could successfully use growpart and then resize2fs. After that mounting back to the origin EC2 instance.

answered Sep 05 '18 at 06:49

RogerSik

35
4

On the newer Nitro-based systems, this is what I had to do. I would receive an error `No space left on device` if I tried to run `growpart` – BoomShadow Mar 11 '19 at 22:37

score 1 · Answer 7 · answered Dec 21 '22 at 10:18

Not sure if AWS has added additional steps inorder to extend the added disk space but the below commands worked for me:

Follow the answer till the community wiki Step 6 Post that for Step 7 follow the steps as below:

Check whether the volume has a partition. Use the lsblk command

sudo lsblk

Output:

NAME          MAJ:MIN RM SIZE RO TYPE MOUNTPOINT

xvda       202:0    0  20G  0 disk
└─xvda1    202:1    0   8G  0 part /

Check whether the partition needs to be extended. In the lsblk command output from the previous step, compare the partition size and the volume size. If the partition size is smaller than the volume size, continue to the next step. If the partition size is equal to the volume size, the partition can't be extended.
Extend the partition. Use the growpart command and specify the partition to extend.

sudo growpart /dev/xvda 1

Verify that the partition has been extended. Use the lsblk command. The partition size should now be equal to the volume size

sudo lsblk

Output:

NAME    MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
xvda    202:0    0    20G  0 disk
└─xvda1 202:1    0    20G  0 part /

Reference

score 0 · Answer 8 · answered Jun 03 '11 at 09:34

0

Your case is valid when the EC2 instance was created from "EBS-Store" rather than "Instance-Store". EC2 instance created from "instance-store" will always have a huge space (around 200GB +) allocated for /mnt directory.

Otherwise your solution is valid for those EC2 machine created from "EBS-Store". You can do more with such machines.

Change the termination type
Change the instance type
Create an AMI from the instance
Auto-scale with ease.

answered Jun 03 '11 at 09:34

Rakesh Sankar

9,337
4
41
66

This answer is inaccurate; ephemeral volume sizes vary according to instance size and AMI type, not [only] according to the base (instance- or ebs-store) launch type. Furthermore, the answer does not actually address the problem raised in the question, and should really have been a comment rather than an answer. – Eight-Bit Guru Jun 03 '11 at 14:28

EC2 instance on Amazon and I am greeted with "No space left on the disk"

8 Answers8

Linked

Related