Yrurtj

I have a large amount of files within a single directory, currently 10,804 files. the amount of files can float between 5 and 100,000.

I am looking for a way to combine every 250 separate files into one large file with the remainder in a small file.
for example 1200 files
I want 4 files with 250 and 1 file with 200

I am using bash shell.

Simply:

#!/bin/bash

files_count=`ls -1 ./ | wc -l`

block_size=10

blocks_count=$(($files_count/$block_size))



for i in $(seq 1 1 $blocks_count); do

    files=`find . -type f -exec readlink -f {} ; | head -$block_size`

    for j in $files; do

        if [ -f $j ] && [[ "$j" != outfile* ]] ; then

            cat $j >> outfile$i

    fi

    done

done

# remainder part

for i in *; do

    if [ -f $i ] && [[ "$i" != outfile* ]] ; then

        cat $i >> outfilelast

    fi

done

Note:

Your files merged alphabetically and also script should be placed inside the same directory.

You can write a straightforward loop to do this with an array and ${x:s:l} parameter expansion:

files=(*)

for (( i = 0; i < ${#files[@]}; i += 250 ))

do

    cat -- "${files[@]:$i:250}" > "file$i.combined"

done

Here we collect all the (non-hidden) files in . into an array files (file names sorted lexically), and loop over counting from 0 to however many files there are in 250s. For each 250, we expand out the filenames (0-249, 250-499, etc) as arguments to cat and put the output into file0.combined, file250.combined, and so on.

This is just Bash's version of a traditional C-style for loop. Since you're going to have to loop for each separate cat anyway, there's not much point overcomplicating things.

You'll end up with several .combined files at the end - because the filenames were already expanded, those won't be included in the concatenations again, but if you ran the command a second time they would be. If that's a concern, you could put them somewhere else, delete them afterwards, or if it's going straight to the printer even just pipe to lp.

I tried with below method

for ((i=1;i<=1200;i++)); do j=$(($i + 249 )); sed -n ''$i','$j'p' filename >individual_$i ;i=$j; done

Assuming you are okay with combining them in the order find finds them:

find . -maxdepth 1 -type f -print0 |

xargs -0 -L 250 sh -c 'cat "$@" >/tmp/combined-${1##*/}' sh

For a directory containing files with names file-1 up to file-739 (as an example), this would create files in /tmp called combined-file-1,
combined-file-251, and combined-file-501, where the bit after combined- is the name of the first file in that combined file.

It does this by calling cat to concatenate files in batches of a maximum of 250 files at a time in an in-line shell script executed repeatedly by xargs (the ${1##*/} in that script removes any directory path from the current batch's first file's pathname). The xargs utility gets the filenames as nul-terminated strings from find. The find utility will look in the current directory (only) and output all pathnames therein that corresponds to regular files.

You would then print the /tmp/combined-* files.

To only process files with a particular suffix, like .txt, use -name '*.txt' in the find command, before -print0.

The -print0 action of find and the -0 option of xargs are non-standard by commonly implemented.