JACK

When a GCE VM Disk Fills Up

Jan 2021 - code

Sometimes you write local logs and it’s just too much. SSH doesn’t even work anymore.

This feels like a common issue and I expected a simple step-by-step guide, but resolving it was mildly aggravating. The documentation provided by Google, and others who have dealt with this before, feel like they skipped over some small details.

Here’s what I did:

  1. Turn off VM

  2. Detach disk from the “Edit” menu

  3. Attach disk to a similar VM (same OS) as an “additional disk”

  4. See disk with

    ls /dev/disk/by-id
lsblk -f

  5. Mount the disk

  6. Grow the partition if we’ve decided to increase the disk size (see lsblk)

    sudo apt -y install cloud-guest-utils
sudo growpart /dev/sdb 1

  7. Delete any unnecessary large files

    du -Sm --max-depth=7 2> >(grep -v 'Permission denied') | sort -n -r | head -n 30

  8. Another option is to move large files to a cloud storage bucket

    gsutil mv <...>

  9. Detach the additional disk, reattach as the boot disk in the original instance

  10. It should now SSH fine!

top