Yrurtj

I have about 2.8TB (yes, terabytes) of data I have recovered, this will be scanned for duplicates, the machine these files reside on is quite old and only has 2GB of memory (works fine for LVM, however), so doing the duplicate scan on it is asking for pain.

My question is this, how can I get Debian to move files into a folder with that filetype, rename automatically where needed without needing to specify a list of filetypes.

I have around 800GB of space free on it, so I can do some testing before letting this run loose on my data.

With a directory that looks like

$ ls   

another.doc  file.txt  file1.mp3  myfile.txt

We can build a list of file extensions with this command:

$ exts=$(ls | sed 's/^.*.//' | sort -u)

We can then loop through these extensions moving files into subdirectories:

$ for ext in $exts

> do

> echo Processing $ext

> mkdir $ext

> mv -v *.$ext $ext/

> done

When this is run we get the following output:

Processing doc

'another.doc' -> 'doc/another.doc'

Processing mp3

'file1.mp3' -> 'mp3/file1.mp3'

Processing txt

'file.txt' -> 'txt/file.txt'

'myfile.txt' -> 'txt/myfile.txt'

The result:

$ ls

doc/  mp3/  txt/



$ ls *

doc:

another.doc



mp3:

file1.mp3



txt:

file.txt  myfile.txt

I wrapped Stephen's code in a script and slightly improved the pipe.

#!/bin/bash 

set -e 

set -u 

set -o pipefail



start=$SECONDS



exts=$(ls -dp *.*| grep -v / | sed 's/^.*.//' | sort -u) # not folders

ignore=""



while getopts ':f::i:h' flag; do

  case "$flag" in

    h)

        echo "This script sorts files from the current dir into folders of the same file type. Specific file types can be specified using -f."

        echo "flags:"

        echo '-f (string file types to sort e.g. -f "pdf csv mp3")'

        echo '-i (string file types to ignore e.g. -i "pdf")'

        exit 1

        ;;

    f)

        exts=$OPTARG;;

    i)

        ignore=$OPTARG;;

    :) 

        echo "Missing option argument for -$OPTARG" >&2; 

        exit 1;;

    ?)

        echo "Invalid option: -$OPTARG" >&2

        exit 1

        ;;

  esac

done



for ext in $exts 

do  

    if [[ " ${ignore} " == *" ${ext} "* ]]; then

        echo "Skiping ${ext}"

        continue

    fi

    echo Processing "$ext"

    mkdir -p "$ext"

    mv -vn *."$ext" "$ext"/

done



duration=$(( SECONDS - start ))

echo "--- Completed in $duration seconds ---"