Yrurtj

I'm writing a bash script that needs to fetch all *_out.csv from a directory, on a remote server. All these files are several directories deep inside of another directory. So for instance, say the directory is called ox_20190404/. I can find all my files by going:

find ox_20190404/assessment/LWR/validation -type f -name "*_out.csv"

This question answers part of my question, but since I don't want to copy the directory in it's entirety I need to figure out how to implement the above code. Suppose I start with this:

$ dir="/projects/ox/git"

$ server="myusername@server"

$ scp $server:$dir/$(ssh $server 'ls -t $dir | head -1') .

How would I grab the files I need from there?

The last part of my question wonders if there is a way to then take all the copied files and place them in the same file path and directory they were in on the remote server.

I've adjusted some of your variable names a bit.

Surely there are better ways to do this than something dangerous like parsing the output of ls, but see whether this works for you:

$ pth="/projects/ox/git"

$ server="myusername@server"

$ dir="$(ssh $server "ls -t "$pth" | head -1")"

$ mkdir -p "$pth/$dir"

$ scp -p $server:"$pth/$dir"/'*_out.csv' "$pth/$dir"/

Once dir has been set to the newest remote directory, mkdir -p is used to ensure that the same directory name exists locally. Then scp the files into a local directory with the same path and name as the remote directory. I was looking for an rsync solution, but couldn't think of one.

This will find the most recently modified (created) directory, assuming that the directory name does not contain a newline (n)

newest=$(

    ssh -qn REMOTE 'find ./* -mindepth 0 -maxdepth 0 -type d -printf "%T@t%fn"' |

    sort -t$'t' -r -nk1,2 |

    head -n1 |

    cut -f2-

)

If you can guarantee that the target contains only directories of interest you can simplify it considerably (again, bearing in mind the newline issue)

newest=$(ssh -qn REMOTE ls -t | head -n1)

You can copy an entire tree of files using scp, but if you want to filter it you'll probably be better off using rsync

rsync -av --include '*/' --include '*_out.csv' --exclude '*' --prune-empty-dirs REMOTE:"$newest" "$newest"

If you're keeping the previous set of files locally and you really just wanted to add the latest set without copying the previous ones, rsync can do that too

rsync -av --include '*/' --include '*_out.csv' --exclude '*' --prune-empty-dirs REMOTE: .