Yrurtj

On my Linux Mint 17.2 VMware guest, df -h was reporting my total disk usage steady at about 10GB total. I'm using this machine for Ruby on Rails development inside a Windows host running Workstation 12.1.1 Pro.

The *.vmdk files kept on steadily growing to approx 100GB. Trying to shrink with vmware-vdiskmanager -k or vmware-toolbox-cmd disk shrinkonly made no difference.

I have Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize. A more complete dumpe2fs output is available.

Running e4defrag / and e2fsck -E discard didn't allow any more space to be reclaimed (I killed the shrink after the first few vmdks (which started at around 6GB) showed no sign of shrinking.)

What finally did the trick was filling all free space with NULbytes:

dd if=/dev/zero of=wipefile bs=1M; sync; rm wipefile

I could now shrink to a total *.vmdk size of 15.6GB with vmware-toolbox-cmd disk shrinkonly.

That's about 85GB space saving on a VM with 10GB actual data.

It seems that ext4 is not re-using previously used blocks when asked for an unused block, often preferring to give out never-before-used blocks.

Questions

1. Having old data hanging around longer seems less secure. Why would ext4 not re-use recently-used blocks as soon as possible?




2. Is there a way to force ext4 to reuse just-used blocks?



3. 
   

   Is there a way to prevent a VMware guest's *.vmdk files from continually growing without needing to 0-fill free space on a regular basis?

   
   
   
   
   
   • How do you safely (eg not entirely filling the filesystem) automate this?
   
   
   
   

1. Yes, from security standpoint it would be better to immedeately erase any unused blocks. The reason that is not done (ignoring chattr secure-delete flag and patches) is performance. The same is a reson for not using any recenty freed block - it would lead to heavy fragmentations which hurts performance.




2. No, not really. You could make your whole image much smaller (formatting it to say 15G and then growing only if/when that becomes necessary.) - then it will never grow bigger than 15Gb.




3. You can try to mount filesystem with discard option - see fstab(5), but I'm not sure if your vmware will heed this

No filesystems zero out blocks of deleted files, they just mark those clusters as available. That's why recovery tools can recover deleted files if they weren't overwritten by other files. If filesystem drivers reuse those blocks ASAP then you'll no longer have the ability to recover accidentally erased files and customers will cry, and if they must overwrite the clusters with zero then performance will suffer severely.

Files that are expected to be stored securely should be encrypted instead of leaving as-is on disk. If needed, use shredder tools like shred on Unix and ccleaner drive wiper, eraser, sdelete... on Windows to delete securely.

About VMDK, you should know that it stores sectors in a sparse format, just like VHD, VDI or any other VM's dynamically-sized virtual disk image format. Therefore zeroing out sectors mark them as not needed anymore and the compactor will leave them out, resulting in a smaller file. Any non-zero sectors have to be stored explicitly because the VM doesn't know whether the sector belongs to a deleted file or not

Using dd if=/dev/zero is a bad way because

• it is slow;


• it makes the disk image (temporarily) grow to its maximal extent;


• it (temporarily) uses all free space on the disk, so other concurrent write actions may fail.

as mentioned in the zerofree manpage. Use a tool specifically for zeroing out disks like zerofree instead.

There's no way to prevent a virtual disk image from expanding, because what would you expect if more data need to be written on the disk? Even in case files are deleted, their data are still on disk and take space in the image file. If you don't want the file to grow, the only way is making it fixed-sized on creation.

Possibly because writing a lot of NULs to the VMDK made it reclaim all the deleted blocks and allocate them to wipefile. However, why e4defrag did not do the same is something I do not understand.