Solving “mv: Argument list too long”?
I have a folder with more than a million files that needs sorting, but I can't really do anything because mv
outputs this message all the time
-bash: /bin/mv: Argument list too long
I'm using this command to move extension-less files:
mv -- !(*.jpg|*.png|*.bmp) targetdir/
bash shell arguments mv
add a comment |
I have a folder with more than a million files that needs sorting, but I can't really do anything because mv
outputs this message all the time
-bash: /bin/mv: Argument list too long
I'm using this command to move extension-less files:
mv -- !(*.jpg|*.png|*.bmp) targetdir/
bash shell arguments mv
Related: Does "argument list too long" restriction apply to shell builtins?
– codeforester
Nov 22 '17 at 21:16
add a comment |
I have a folder with more than a million files that needs sorting, but I can't really do anything because mv
outputs this message all the time
-bash: /bin/mv: Argument list too long
I'm using this command to move extension-less files:
mv -- !(*.jpg|*.png|*.bmp) targetdir/
bash shell arguments mv
I have a folder with more than a million files that needs sorting, but I can't really do anything because mv
outputs this message all the time
-bash: /bin/mv: Argument list too long
I'm using this command to move extension-less files:
mv -- !(*.jpg|*.png|*.bmp) targetdir/
bash shell arguments mv
bash shell arguments mv
edited Sep 17 '16 at 15:08
Jeff Schaller
39.9k1054126
39.9k1054126
asked May 8 '14 at 23:31
DominiqueDominique
1,35051821
1,35051821
Related: Does "argument list too long" restriction apply to shell builtins?
– codeforester
Nov 22 '17 at 21:16
add a comment |
Related: Does "argument list too long" restriction apply to shell builtins?
– codeforester
Nov 22 '17 at 21:16
Related: Does "argument list too long" restriction apply to shell builtins?
– codeforester
Nov 22 '17 at 21:16
Related: Does "argument list too long" restriction apply to shell builtins?
– codeforester
Nov 22 '17 at 21:16
add a comment |
8 Answers
8
active
oldest
votes
xargs
is the tool for the job. That, or find
with -exec … {} +
. These tools run a command several times, with as many arguments as can be passed in one go.
Both methods are easier to carry out when the variable argument list is at the end, which isn't the case here: the final argument to mv
is the destination. With GNU utilities (i.e. on non-embedded Linux or Cygwin), the -t
option to mv
is useful, to pass the destination first.
If the file names have no whitespace nor any of "'
, then you can simply provide the file names as input to xargs
(the echo
command is a bash builtin, so it isn't subject to the command line length limit):
echo !(*.jpg|*.png|*.bmp) | xargs mv -t targetdir
You can use the -0
option to xargs
to use null-delimited input instead of the default quoted format.
printf '%s' !(*.jpg|*.png|*.bmp) | xargs -0 mv -t targetdir
Alternatively, you can generate the list of file names with find
. To avoid recursing into subdirectories, use -type d -prune
. Since no action is specified for the listed image files, only the other files are moved.
find . -name . -o -type d -prune -o
-name '*.jpg' -o -name '*.png' -o -name '*.bmp' -o
-exec mv -t targetdir/ {} +
(This includes dot files, unlike the shell wildcard methods.)
If you don't have GNU utilities, you can use an intermediate shell to get the arguments in the right order. This method works on all POSIX systems.
find . -name . -o -type d -prune -o
-name '*.jpg' -o -name '*.png' -o -name '*.bmp' -o
-exec sh -c 'mv "$@" "$0"' targetdir/ {} +
In zsh, you can load the mv
builtin:
setopt extended_glob
zmodload zsh/files
mv -- ^*.(jpg|png|bmp) targetdir/
or if you prefer to let mv
and other names keep referring to the external commands:
setopt extended_glob
zmodload -Fm zsh/files b:zf_*
zf_mv -- ^*.(jpg|png|bmp) targetdir/
or with ksh-style globs:
setopt ksh_glob
zmodload -Fm zsh/files b:zf_*
zf_mv -- !(*.jpg|*.png|*.bmp) targetdir/
Alternatively, using GNU mv
and zargs
:
autoload -U zargs
setopt extended_glob
zargs -- ./^*.(jpg|png|bmp) -- mv -t targetdir/
1
The first two commands returned "-bash: !: event not found" and the next two did not move any files at all. I'm on CentOS 6.5 if you should know
– Dominique
May 9 '14 at 1:44
@Dominique I used the same globbing syntax that you used in your question. You'll needshopt -s extglob
to enable it. I'd missed a step in thefind
commands, I've fixed them.
– Gilles
May 9 '14 at 1:50
I'm getting this with the find command "find: invalid expression; you have used a binary operator '-o' with nothing before it." I will now try the other ones.
– Dominique
May 9 '14 at 1:56
@Dominique Thefind
commands I've posted (now) work. You must have left off a part when copy-pasting.
– Gilles
May 9 '14 at 1:58
Gilles, for the find commands, why not use the "not" operator,!
? It's more explicit and easier to understand than the odd trailing-o
. For example,! -name '*.jpg' -a ! -name '*.png' -a ! -name '*.bmp'
– CivFan
Oct 23 '15 at 21:15
|
show 6 more comments
The operating system's argument passing limit does not apply to expansions which happen within the shell interpreter. So in addition to using xargs
or find
, we can simply use a shell loop to break up the processing into individual mv
commands:
for x in *; do case "$x" in *.jpg|*.png|*.bmp) ;; *) mv -- "$x" target ;; esac ; done
This uses only POSIX Shell Command Language features and utilities. This one-liner is clearer with indentation, with unnecessary semicolons removed:
for x in *; do
case "$x" in
*.jpg|*.png|*.bmp)
;; # nothing
*) # catch-all case
mv -- "$x" target
;;
esac
done
With more than a million files, this will in turn spawn more than a millionmv
processes, instead of just the few needed using the POSIXfind
solution @Gilles posted. In other words, this way results in lots of unnecessary CPU churn.
– CivFan
Oct 23 '15 at 20:48
@CivFan Another problem is convincing yourself that the modified version is equivalent to the original. It's easy to see that thecase
statement on the result of*
expansion to filter out several extensions is equivalent to the original!(*.jpg|*.png|*.bmp)
expression. Thefind
answer is in fact not equivalent; it descends into subdirectories (I don't see a-maxdepth
predicate).
– Kaz
Oct 23 '15 at 20:51
-name . -o -type d -prune -o
protects from descending into sub-directories.-maxdepth
is apparently not POSIX compliant, though that's not mentioned in myfind
man page.
– CivFan
Oct 23 '15 at 21:33
Rolled back to revision 1. The question doesn't say anything about source or destination variables, so this adds unnecessary cruft to the answer.
– Kaz
Oct 24 '15 at 0:29
add a comment |
For a more aggressive solution than those previously offered, pull up your kernel source and edit include/linux/binfmts.h
Increase the size of MAX_ARG_PAGES
to something larger than 32. This increases the amount of memory the kernel will allow for program arguments, thereby allowing you to specify your mv
or rm
command for a million files or whatever you're doing. Recompile, install, reboot.
BEWARE! If you set this too large for your system memory, and then run a command with a lot of arguments BAD THINGS WILL HAPPEN! Be extremely cautious doing this to multi-user systems, it makes it easier for malicious users to use up all your memory!
If you don't know how to recompile and reinstall your kernel manually, it's probably best that you just pretend this answer doesn't exist for now.
add a comment |
A more simple solution using "$origin"/!(*.jpg|*.png|*.bmp)
instead of a catch block:
for file in "$origin"/!(*.jpg|*.png|*.bmp); do mv -- "$file" "$destination" ; done
Thanks to @Score_Under
For a multi-line script you can do the following (notice the ;
before the done
is dropped):
for file in "$origin"/!(*.jpg|*.png|*.bmp); do # don't copy types *.jpg|*.png|*.bmp
mv -- "$file" "$destination"
done
To do a more generalized solution that moves all files, you can do the one-liner:
for file in "$origin"/*; do mv -- "$file" "$destination" ; done
Which looks like this if you do indentation:
for file in "$origin"/*; do
mv -- "$file" "$destination"
done
This takes every file in the origin and moves them one by one to the destination. The quotes around $file
are necessary in case there are spaces or other special characters in the filenames.
Here is an example of this method that worked perfectly
for file in "/Users/william/Pictures/export_folder_111210/"*.jpg; do
mv -- "$file" "/Users/william/Desktop/southland/landingphotos/";
done
You could use something like the original glob in the for-loop to get a closer solution to what's being asked for.
– Score_Under
Jun 20 '15 at 0:34
What do you mean original glob?
– Whitecat
Jun 20 '15 at 0:38
Sorry if that was a little cryptic, I was referring to the glob in the question:!(*.jpg|*.png|*.bmp)
. You could add that to your for-loop by globbing"$origin"/!(*.jpg|*.png|*.bmp)
which would avoid the need for the switch used in Kaz's answer and keep the simple body of the for-loop.
– Score_Under
Jun 21 '15 at 3:52
Awesome Score. I incorporated your comment and updated my answer.
– Whitecat
Jun 27 '15 at 22:55
add a comment |
If working with Linux kernel is enough you can simply do
ulimit -s 100000
that will work because Linux kernel included a patch around 10 years ago that changed argument limit to be based on stack size: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=b6a2fea39318e43fee84fa7b0b90d68bed92d2ba
Update: If you feel brave, you can say
ulimit -s unlimited
and you'll be fine with any shell expansions as long as you have enough RAM.
That's a hack. How would you know what to set the stack limit to? This also affects other processes started in the same session.
– Kusalananda
Nov 1 '17 at 9:38
Yeah, it's a hack. Most of the time this kind of hacks are one-off (how often you manually move huge amount of files anyway?). If you are sure that the process is not going to eat all your RAM, you can setulimit -s unlimited
and it will work for practically unlimited files.
– Mikko Rantalainen
Nov 3 '17 at 13:00
Withulimit -s unlimited
the actual command line limit is 2^31 or 2 GB. (MAX_ARG_STRLEN
in kernel source.)
– Mikko Rantalainen
Jun 8 '18 at 7:01
add a comment |
You can get around that restriction while still using mv
if you don't mind running it a couple times.
You can move portions at a time. Let's say for example you had a long list of alphanumeric file names.
mv ./subdir/a* ./
That works. Then knock out another big chunk. After a couple moves, you can just go back to using mv ./subdir/* ./
add a comment |
Sometimes it's easiest to just write a little script, e.g. in Python:
import glob, shutil
for i in glob.glob('*.jpg'):
shutil.move(i, 'new_dir/' + i)
add a comment |
Here is my two cents, append this into .bash_profile
mv() {
if [[ -d $1 ]]; then #directory mv
/bin/mv $1 $2
elif [[ -f $1 ]]; then #file mv
/bin/mv $1 $2
else
for f in $1
do
source_path=$f
#echo $source_path
source_file=${source_path##*/}
#echo $source_file
destination_path=${2%/} #get rid of trailing forward slash
echo "Moving $f to $destination_path/$source_file"
/bin/mv $f $destination_path/$source_file
done
fi
}
export -f mv
Usage
mv '*.jpg' ./destination/
mv '/path/*' ./destination/
New contributor
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "106"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f128559%2fsolving-mv-argument-list-too-long%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
8 Answers
8
active
oldest
votes
8 Answers
8
active
oldest
votes
active
oldest
votes
active
oldest
votes
xargs
is the tool for the job. That, or find
with -exec … {} +
. These tools run a command several times, with as many arguments as can be passed in one go.
Both methods are easier to carry out when the variable argument list is at the end, which isn't the case here: the final argument to mv
is the destination. With GNU utilities (i.e. on non-embedded Linux or Cygwin), the -t
option to mv
is useful, to pass the destination first.
If the file names have no whitespace nor any of "'
, then you can simply provide the file names as input to xargs
(the echo
command is a bash builtin, so it isn't subject to the command line length limit):
echo !(*.jpg|*.png|*.bmp) | xargs mv -t targetdir
You can use the -0
option to xargs
to use null-delimited input instead of the default quoted format.
printf '%s' !(*.jpg|*.png|*.bmp) | xargs -0 mv -t targetdir
Alternatively, you can generate the list of file names with find
. To avoid recursing into subdirectories, use -type d -prune
. Since no action is specified for the listed image files, only the other files are moved.
find . -name . -o -type d -prune -o
-name '*.jpg' -o -name '*.png' -o -name '*.bmp' -o
-exec mv -t targetdir/ {} +
(This includes dot files, unlike the shell wildcard methods.)
If you don't have GNU utilities, you can use an intermediate shell to get the arguments in the right order. This method works on all POSIX systems.
find . -name . -o -type d -prune -o
-name '*.jpg' -o -name '*.png' -o -name '*.bmp' -o
-exec sh -c 'mv "$@" "$0"' targetdir/ {} +
In zsh, you can load the mv
builtin:
setopt extended_glob
zmodload zsh/files
mv -- ^*.(jpg|png|bmp) targetdir/
or if you prefer to let mv
and other names keep referring to the external commands:
setopt extended_glob
zmodload -Fm zsh/files b:zf_*
zf_mv -- ^*.(jpg|png|bmp) targetdir/
or with ksh-style globs:
setopt ksh_glob
zmodload -Fm zsh/files b:zf_*
zf_mv -- !(*.jpg|*.png|*.bmp) targetdir/
Alternatively, using GNU mv
and zargs
:
autoload -U zargs
setopt extended_glob
zargs -- ./^*.(jpg|png|bmp) -- mv -t targetdir/
1
The first two commands returned "-bash: !: event not found" and the next two did not move any files at all. I'm on CentOS 6.5 if you should know
– Dominique
May 9 '14 at 1:44
@Dominique I used the same globbing syntax that you used in your question. You'll needshopt -s extglob
to enable it. I'd missed a step in thefind
commands, I've fixed them.
– Gilles
May 9 '14 at 1:50
I'm getting this with the find command "find: invalid expression; you have used a binary operator '-o' with nothing before it." I will now try the other ones.
– Dominique
May 9 '14 at 1:56
@Dominique Thefind
commands I've posted (now) work. You must have left off a part when copy-pasting.
– Gilles
May 9 '14 at 1:58
Gilles, for the find commands, why not use the "not" operator,!
? It's more explicit and easier to understand than the odd trailing-o
. For example,! -name '*.jpg' -a ! -name '*.png' -a ! -name '*.bmp'
– CivFan
Oct 23 '15 at 21:15
|
show 6 more comments
xargs
is the tool for the job. That, or find
with -exec … {} +
. These tools run a command several times, with as many arguments as can be passed in one go.
Both methods are easier to carry out when the variable argument list is at the end, which isn't the case here: the final argument to mv
is the destination. With GNU utilities (i.e. on non-embedded Linux or Cygwin), the -t
option to mv
is useful, to pass the destination first.
If the file names have no whitespace nor any of "'
, then you can simply provide the file names as input to xargs
(the echo
command is a bash builtin, so it isn't subject to the command line length limit):
echo !(*.jpg|*.png|*.bmp) | xargs mv -t targetdir
You can use the -0
option to xargs
to use null-delimited input instead of the default quoted format.
printf '%s' !(*.jpg|*.png|*.bmp) | xargs -0 mv -t targetdir
Alternatively, you can generate the list of file names with find
. To avoid recursing into subdirectories, use -type d -prune
. Since no action is specified for the listed image files, only the other files are moved.
find . -name . -o -type d -prune -o
-name '*.jpg' -o -name '*.png' -o -name '*.bmp' -o
-exec mv -t targetdir/ {} +
(This includes dot files, unlike the shell wildcard methods.)
If you don't have GNU utilities, you can use an intermediate shell to get the arguments in the right order. This method works on all POSIX systems.
find . -name . -o -type d -prune -o
-name '*.jpg' -o -name '*.png' -o -name '*.bmp' -o
-exec sh -c 'mv "$@" "$0"' targetdir/ {} +
In zsh, you can load the mv
builtin:
setopt extended_glob
zmodload zsh/files
mv -- ^*.(jpg|png|bmp) targetdir/
or if you prefer to let mv
and other names keep referring to the external commands:
setopt extended_glob
zmodload -Fm zsh/files b:zf_*
zf_mv -- ^*.(jpg|png|bmp) targetdir/
or with ksh-style globs:
setopt ksh_glob
zmodload -Fm zsh/files b:zf_*
zf_mv -- !(*.jpg|*.png|*.bmp) targetdir/
Alternatively, using GNU mv
and zargs
:
autoload -U zargs
setopt extended_glob
zargs -- ./^*.(jpg|png|bmp) -- mv -t targetdir/
1
The first two commands returned "-bash: !: event not found" and the next two did not move any files at all. I'm on CentOS 6.5 if you should know
– Dominique
May 9 '14 at 1:44
@Dominique I used the same globbing syntax that you used in your question. You'll needshopt -s extglob
to enable it. I'd missed a step in thefind
commands, I've fixed them.
– Gilles
May 9 '14 at 1:50
I'm getting this with the find command "find: invalid expression; you have used a binary operator '-o' with nothing before it." I will now try the other ones.
– Dominique
May 9 '14 at 1:56
@Dominique Thefind
commands I've posted (now) work. You must have left off a part when copy-pasting.
– Gilles
May 9 '14 at 1:58
Gilles, for the find commands, why not use the "not" operator,!
? It's more explicit and easier to understand than the odd trailing-o
. For example,! -name '*.jpg' -a ! -name '*.png' -a ! -name '*.bmp'
– CivFan
Oct 23 '15 at 21:15
|
show 6 more comments
xargs
is the tool for the job. That, or find
with -exec … {} +
. These tools run a command several times, with as many arguments as can be passed in one go.
Both methods are easier to carry out when the variable argument list is at the end, which isn't the case here: the final argument to mv
is the destination. With GNU utilities (i.e. on non-embedded Linux or Cygwin), the -t
option to mv
is useful, to pass the destination first.
If the file names have no whitespace nor any of "'
, then you can simply provide the file names as input to xargs
(the echo
command is a bash builtin, so it isn't subject to the command line length limit):
echo !(*.jpg|*.png|*.bmp) | xargs mv -t targetdir
You can use the -0
option to xargs
to use null-delimited input instead of the default quoted format.
printf '%s' !(*.jpg|*.png|*.bmp) | xargs -0 mv -t targetdir
Alternatively, you can generate the list of file names with find
. To avoid recursing into subdirectories, use -type d -prune
. Since no action is specified for the listed image files, only the other files are moved.
find . -name . -o -type d -prune -o
-name '*.jpg' -o -name '*.png' -o -name '*.bmp' -o
-exec mv -t targetdir/ {} +
(This includes dot files, unlike the shell wildcard methods.)
If you don't have GNU utilities, you can use an intermediate shell to get the arguments in the right order. This method works on all POSIX systems.
find . -name . -o -type d -prune -o
-name '*.jpg' -o -name '*.png' -o -name '*.bmp' -o
-exec sh -c 'mv "$@" "$0"' targetdir/ {} +
In zsh, you can load the mv
builtin:
setopt extended_glob
zmodload zsh/files
mv -- ^*.(jpg|png|bmp) targetdir/
or if you prefer to let mv
and other names keep referring to the external commands:
setopt extended_glob
zmodload -Fm zsh/files b:zf_*
zf_mv -- ^*.(jpg|png|bmp) targetdir/
or with ksh-style globs:
setopt ksh_glob
zmodload -Fm zsh/files b:zf_*
zf_mv -- !(*.jpg|*.png|*.bmp) targetdir/
Alternatively, using GNU mv
and zargs
:
autoload -U zargs
setopt extended_glob
zargs -- ./^*.(jpg|png|bmp) -- mv -t targetdir/
xargs
is the tool for the job. That, or find
with -exec … {} +
. These tools run a command several times, with as many arguments as can be passed in one go.
Both methods are easier to carry out when the variable argument list is at the end, which isn't the case here: the final argument to mv
is the destination. With GNU utilities (i.e. on non-embedded Linux or Cygwin), the -t
option to mv
is useful, to pass the destination first.
If the file names have no whitespace nor any of "'
, then you can simply provide the file names as input to xargs
(the echo
command is a bash builtin, so it isn't subject to the command line length limit):
echo !(*.jpg|*.png|*.bmp) | xargs mv -t targetdir
You can use the -0
option to xargs
to use null-delimited input instead of the default quoted format.
printf '%s' !(*.jpg|*.png|*.bmp) | xargs -0 mv -t targetdir
Alternatively, you can generate the list of file names with find
. To avoid recursing into subdirectories, use -type d -prune
. Since no action is specified for the listed image files, only the other files are moved.
find . -name . -o -type d -prune -o
-name '*.jpg' -o -name '*.png' -o -name '*.bmp' -o
-exec mv -t targetdir/ {} +
(This includes dot files, unlike the shell wildcard methods.)
If you don't have GNU utilities, you can use an intermediate shell to get the arguments in the right order. This method works on all POSIX systems.
find . -name . -o -type d -prune -o
-name '*.jpg' -o -name '*.png' -o -name '*.bmp' -o
-exec sh -c 'mv "$@" "$0"' targetdir/ {} +
In zsh, you can load the mv
builtin:
setopt extended_glob
zmodload zsh/files
mv -- ^*.(jpg|png|bmp) targetdir/
or if you prefer to let mv
and other names keep referring to the external commands:
setopt extended_glob
zmodload -Fm zsh/files b:zf_*
zf_mv -- ^*.(jpg|png|bmp) targetdir/
or with ksh-style globs:
setopt ksh_glob
zmodload -Fm zsh/files b:zf_*
zf_mv -- !(*.jpg|*.png|*.bmp) targetdir/
Alternatively, using GNU mv
and zargs
:
autoload -U zargs
setopt extended_glob
zargs -- ./^*.(jpg|png|bmp) -- mv -t targetdir/
edited Sep 17 '14 at 21:41
answered May 8 '14 at 23:48
GillesGilles
533k12810721594
533k12810721594
1
The first two commands returned "-bash: !: event not found" and the next two did not move any files at all. I'm on CentOS 6.5 if you should know
– Dominique
May 9 '14 at 1:44
@Dominique I used the same globbing syntax that you used in your question. You'll needshopt -s extglob
to enable it. I'd missed a step in thefind
commands, I've fixed them.
– Gilles
May 9 '14 at 1:50
I'm getting this with the find command "find: invalid expression; you have used a binary operator '-o' with nothing before it." I will now try the other ones.
– Dominique
May 9 '14 at 1:56
@Dominique Thefind
commands I've posted (now) work. You must have left off a part when copy-pasting.
– Gilles
May 9 '14 at 1:58
Gilles, for the find commands, why not use the "not" operator,!
? It's more explicit and easier to understand than the odd trailing-o
. For example,! -name '*.jpg' -a ! -name '*.png' -a ! -name '*.bmp'
– CivFan
Oct 23 '15 at 21:15
|
show 6 more comments
1
The first two commands returned "-bash: !: event not found" and the next two did not move any files at all. I'm on CentOS 6.5 if you should know
– Dominique
May 9 '14 at 1:44
@Dominique I used the same globbing syntax that you used in your question. You'll needshopt -s extglob
to enable it. I'd missed a step in thefind
commands, I've fixed them.
– Gilles
May 9 '14 at 1:50
I'm getting this with the find command "find: invalid expression; you have used a binary operator '-o' with nothing before it." I will now try the other ones.
– Dominique
May 9 '14 at 1:56
@Dominique Thefind
commands I've posted (now) work. You must have left off a part when copy-pasting.
– Gilles
May 9 '14 at 1:58
Gilles, for the find commands, why not use the "not" operator,!
? It's more explicit and easier to understand than the odd trailing-o
. For example,! -name '*.jpg' -a ! -name '*.png' -a ! -name '*.bmp'
– CivFan
Oct 23 '15 at 21:15
1
1
The first two commands returned "-bash: !: event not found" and the next two did not move any files at all. I'm on CentOS 6.5 if you should know
– Dominique
May 9 '14 at 1:44
The first two commands returned "-bash: !: event not found" and the next two did not move any files at all. I'm on CentOS 6.5 if you should know
– Dominique
May 9 '14 at 1:44
@Dominique I used the same globbing syntax that you used in your question. You'll need
shopt -s extglob
to enable it. I'd missed a step in the find
commands, I've fixed them.– Gilles
May 9 '14 at 1:50
@Dominique I used the same globbing syntax that you used in your question. You'll need
shopt -s extglob
to enable it. I'd missed a step in the find
commands, I've fixed them.– Gilles
May 9 '14 at 1:50
I'm getting this with the find command "find: invalid expression; you have used a binary operator '-o' with nothing before it." I will now try the other ones.
– Dominique
May 9 '14 at 1:56
I'm getting this with the find command "find: invalid expression; you have used a binary operator '-o' with nothing before it." I will now try the other ones.
– Dominique
May 9 '14 at 1:56
@Dominique The
find
commands I've posted (now) work. You must have left off a part when copy-pasting.– Gilles
May 9 '14 at 1:58
@Dominique The
find
commands I've posted (now) work. You must have left off a part when copy-pasting.– Gilles
May 9 '14 at 1:58
Gilles, for the find commands, why not use the "not" operator,
!
? It's more explicit and easier to understand than the odd trailing -o
. For example, ! -name '*.jpg' -a ! -name '*.png' -a ! -name '*.bmp'
– CivFan
Oct 23 '15 at 21:15
Gilles, for the find commands, why not use the "not" operator,
!
? It's more explicit and easier to understand than the odd trailing -o
. For example, ! -name '*.jpg' -a ! -name '*.png' -a ! -name '*.bmp'
– CivFan
Oct 23 '15 at 21:15
|
show 6 more comments
The operating system's argument passing limit does not apply to expansions which happen within the shell interpreter. So in addition to using xargs
or find
, we can simply use a shell loop to break up the processing into individual mv
commands:
for x in *; do case "$x" in *.jpg|*.png|*.bmp) ;; *) mv -- "$x" target ;; esac ; done
This uses only POSIX Shell Command Language features and utilities. This one-liner is clearer with indentation, with unnecessary semicolons removed:
for x in *; do
case "$x" in
*.jpg|*.png|*.bmp)
;; # nothing
*) # catch-all case
mv -- "$x" target
;;
esac
done
With more than a million files, this will in turn spawn more than a millionmv
processes, instead of just the few needed using the POSIXfind
solution @Gilles posted. In other words, this way results in lots of unnecessary CPU churn.
– CivFan
Oct 23 '15 at 20:48
@CivFan Another problem is convincing yourself that the modified version is equivalent to the original. It's easy to see that thecase
statement on the result of*
expansion to filter out several extensions is equivalent to the original!(*.jpg|*.png|*.bmp)
expression. Thefind
answer is in fact not equivalent; it descends into subdirectories (I don't see a-maxdepth
predicate).
– Kaz
Oct 23 '15 at 20:51
-name . -o -type d -prune -o
protects from descending into sub-directories.-maxdepth
is apparently not POSIX compliant, though that's not mentioned in myfind
man page.
– CivFan
Oct 23 '15 at 21:33
Rolled back to revision 1. The question doesn't say anything about source or destination variables, so this adds unnecessary cruft to the answer.
– Kaz
Oct 24 '15 at 0:29
add a comment |
The operating system's argument passing limit does not apply to expansions which happen within the shell interpreter. So in addition to using xargs
or find
, we can simply use a shell loop to break up the processing into individual mv
commands:
for x in *; do case "$x" in *.jpg|*.png|*.bmp) ;; *) mv -- "$x" target ;; esac ; done
This uses only POSIX Shell Command Language features and utilities. This one-liner is clearer with indentation, with unnecessary semicolons removed:
for x in *; do
case "$x" in
*.jpg|*.png|*.bmp)
;; # nothing
*) # catch-all case
mv -- "$x" target
;;
esac
done
With more than a million files, this will in turn spawn more than a millionmv
processes, instead of just the few needed using the POSIXfind
solution @Gilles posted. In other words, this way results in lots of unnecessary CPU churn.
– CivFan
Oct 23 '15 at 20:48
@CivFan Another problem is convincing yourself that the modified version is equivalent to the original. It's easy to see that thecase
statement on the result of*
expansion to filter out several extensions is equivalent to the original!(*.jpg|*.png|*.bmp)
expression. Thefind
answer is in fact not equivalent; it descends into subdirectories (I don't see a-maxdepth
predicate).
– Kaz
Oct 23 '15 at 20:51
-name . -o -type d -prune -o
protects from descending into sub-directories.-maxdepth
is apparently not POSIX compliant, though that's not mentioned in myfind
man page.
– CivFan
Oct 23 '15 at 21:33
Rolled back to revision 1. The question doesn't say anything about source or destination variables, so this adds unnecessary cruft to the answer.
– Kaz
Oct 24 '15 at 0:29
add a comment |
The operating system's argument passing limit does not apply to expansions which happen within the shell interpreter. So in addition to using xargs
or find
, we can simply use a shell loop to break up the processing into individual mv
commands:
for x in *; do case "$x" in *.jpg|*.png|*.bmp) ;; *) mv -- "$x" target ;; esac ; done
This uses only POSIX Shell Command Language features and utilities. This one-liner is clearer with indentation, with unnecessary semicolons removed:
for x in *; do
case "$x" in
*.jpg|*.png|*.bmp)
;; # nothing
*) # catch-all case
mv -- "$x" target
;;
esac
done
The operating system's argument passing limit does not apply to expansions which happen within the shell interpreter. So in addition to using xargs
or find
, we can simply use a shell loop to break up the processing into individual mv
commands:
for x in *; do case "$x" in *.jpg|*.png|*.bmp) ;; *) mv -- "$x" target ;; esac ; done
This uses only POSIX Shell Command Language features and utilities. This one-liner is clearer with indentation, with unnecessary semicolons removed:
for x in *; do
case "$x" in
*.jpg|*.png|*.bmp)
;; # nothing
*) # catch-all case
mv -- "$x" target
;;
esac
done
edited Nov 7 '15 at 6:46
Whitecat
218112
218112
answered Sep 17 '14 at 21:22
KazKaz
4,59311432
4,59311432
With more than a million files, this will in turn spawn more than a millionmv
processes, instead of just the few needed using the POSIXfind
solution @Gilles posted. In other words, this way results in lots of unnecessary CPU churn.
– CivFan
Oct 23 '15 at 20:48
@CivFan Another problem is convincing yourself that the modified version is equivalent to the original. It's easy to see that thecase
statement on the result of*
expansion to filter out several extensions is equivalent to the original!(*.jpg|*.png|*.bmp)
expression. Thefind
answer is in fact not equivalent; it descends into subdirectories (I don't see a-maxdepth
predicate).
– Kaz
Oct 23 '15 at 20:51
-name . -o -type d -prune -o
protects from descending into sub-directories.-maxdepth
is apparently not POSIX compliant, though that's not mentioned in myfind
man page.
– CivFan
Oct 23 '15 at 21:33
Rolled back to revision 1. The question doesn't say anything about source or destination variables, so this adds unnecessary cruft to the answer.
– Kaz
Oct 24 '15 at 0:29
add a comment |
With more than a million files, this will in turn spawn more than a millionmv
processes, instead of just the few needed using the POSIXfind
solution @Gilles posted. In other words, this way results in lots of unnecessary CPU churn.
– CivFan
Oct 23 '15 at 20:48
@CivFan Another problem is convincing yourself that the modified version is equivalent to the original. It's easy to see that thecase
statement on the result of*
expansion to filter out several extensions is equivalent to the original!(*.jpg|*.png|*.bmp)
expression. Thefind
answer is in fact not equivalent; it descends into subdirectories (I don't see a-maxdepth
predicate).
– Kaz
Oct 23 '15 at 20:51
-name . -o -type d -prune -o
protects from descending into sub-directories.-maxdepth
is apparently not POSIX compliant, though that's not mentioned in myfind
man page.
– CivFan
Oct 23 '15 at 21:33
Rolled back to revision 1. The question doesn't say anything about source or destination variables, so this adds unnecessary cruft to the answer.
– Kaz
Oct 24 '15 at 0:29
With more than a million files, this will in turn spawn more than a million
mv
processes, instead of just the few needed using the POSIX find
solution @Gilles posted. In other words, this way results in lots of unnecessary CPU churn.– CivFan
Oct 23 '15 at 20:48
With more than a million files, this will in turn spawn more than a million
mv
processes, instead of just the few needed using the POSIX find
solution @Gilles posted. In other words, this way results in lots of unnecessary CPU churn.– CivFan
Oct 23 '15 at 20:48
@CivFan Another problem is convincing yourself that the modified version is equivalent to the original. It's easy to see that the
case
statement on the result of *
expansion to filter out several extensions is equivalent to the original !(*.jpg|*.png|*.bmp)
expression. The find
answer is in fact not equivalent; it descends into subdirectories (I don't see a -maxdepth
predicate).– Kaz
Oct 23 '15 at 20:51
@CivFan Another problem is convincing yourself that the modified version is equivalent to the original. It's easy to see that the
case
statement on the result of *
expansion to filter out several extensions is equivalent to the original !(*.jpg|*.png|*.bmp)
expression. The find
answer is in fact not equivalent; it descends into subdirectories (I don't see a -maxdepth
predicate).– Kaz
Oct 23 '15 at 20:51
-name . -o -type d -prune -o
protects from descending into sub-directories. -maxdepth
is apparently not POSIX compliant, though that's not mentioned in my find
man page.– CivFan
Oct 23 '15 at 21:33
-name . -o -type d -prune -o
protects from descending into sub-directories. -maxdepth
is apparently not POSIX compliant, though that's not mentioned in my find
man page.– CivFan
Oct 23 '15 at 21:33
Rolled back to revision 1. The question doesn't say anything about source or destination variables, so this adds unnecessary cruft to the answer.
– Kaz
Oct 24 '15 at 0:29
Rolled back to revision 1. The question doesn't say anything about source or destination variables, so this adds unnecessary cruft to the answer.
– Kaz
Oct 24 '15 at 0:29
add a comment |
For a more aggressive solution than those previously offered, pull up your kernel source and edit include/linux/binfmts.h
Increase the size of MAX_ARG_PAGES
to something larger than 32. This increases the amount of memory the kernel will allow for program arguments, thereby allowing you to specify your mv
or rm
command for a million files or whatever you're doing. Recompile, install, reboot.
BEWARE! If you set this too large for your system memory, and then run a command with a lot of arguments BAD THINGS WILL HAPPEN! Be extremely cautious doing this to multi-user systems, it makes it easier for malicious users to use up all your memory!
If you don't know how to recompile and reinstall your kernel manually, it's probably best that you just pretend this answer doesn't exist for now.
add a comment |
For a more aggressive solution than those previously offered, pull up your kernel source and edit include/linux/binfmts.h
Increase the size of MAX_ARG_PAGES
to something larger than 32. This increases the amount of memory the kernel will allow for program arguments, thereby allowing you to specify your mv
or rm
command for a million files or whatever you're doing. Recompile, install, reboot.
BEWARE! If you set this too large for your system memory, and then run a command with a lot of arguments BAD THINGS WILL HAPPEN! Be extremely cautious doing this to multi-user systems, it makes it easier for malicious users to use up all your memory!
If you don't know how to recompile and reinstall your kernel manually, it's probably best that you just pretend this answer doesn't exist for now.
add a comment |
For a more aggressive solution than those previously offered, pull up your kernel source and edit include/linux/binfmts.h
Increase the size of MAX_ARG_PAGES
to something larger than 32. This increases the amount of memory the kernel will allow for program arguments, thereby allowing you to specify your mv
or rm
command for a million files or whatever you're doing. Recompile, install, reboot.
BEWARE! If you set this too large for your system memory, and then run a command with a lot of arguments BAD THINGS WILL HAPPEN! Be extremely cautious doing this to multi-user systems, it makes it easier for malicious users to use up all your memory!
If you don't know how to recompile and reinstall your kernel manually, it's probably best that you just pretend this answer doesn't exist for now.
For a more aggressive solution than those previously offered, pull up your kernel source and edit include/linux/binfmts.h
Increase the size of MAX_ARG_PAGES
to something larger than 32. This increases the amount of memory the kernel will allow for program arguments, thereby allowing you to specify your mv
or rm
command for a million files or whatever you're doing. Recompile, install, reboot.
BEWARE! If you set this too large for your system memory, and then run a command with a lot of arguments BAD THINGS WILL HAPPEN! Be extremely cautious doing this to multi-user systems, it makes it easier for malicious users to use up all your memory!
If you don't know how to recompile and reinstall your kernel manually, it's probably best that you just pretend this answer doesn't exist for now.
edited Jun 19 '15 at 21:33
muru
1
1
answered Jun 19 '15 at 17:26
PerkinsPerkins
25525
25525
add a comment |
add a comment |
A more simple solution using "$origin"/!(*.jpg|*.png|*.bmp)
instead of a catch block:
for file in "$origin"/!(*.jpg|*.png|*.bmp); do mv -- "$file" "$destination" ; done
Thanks to @Score_Under
For a multi-line script you can do the following (notice the ;
before the done
is dropped):
for file in "$origin"/!(*.jpg|*.png|*.bmp); do # don't copy types *.jpg|*.png|*.bmp
mv -- "$file" "$destination"
done
To do a more generalized solution that moves all files, you can do the one-liner:
for file in "$origin"/*; do mv -- "$file" "$destination" ; done
Which looks like this if you do indentation:
for file in "$origin"/*; do
mv -- "$file" "$destination"
done
This takes every file in the origin and moves them one by one to the destination. The quotes around $file
are necessary in case there are spaces or other special characters in the filenames.
Here is an example of this method that worked perfectly
for file in "/Users/william/Pictures/export_folder_111210/"*.jpg; do
mv -- "$file" "/Users/william/Desktop/southland/landingphotos/";
done
You could use something like the original glob in the for-loop to get a closer solution to what's being asked for.
– Score_Under
Jun 20 '15 at 0:34
What do you mean original glob?
– Whitecat
Jun 20 '15 at 0:38
Sorry if that was a little cryptic, I was referring to the glob in the question:!(*.jpg|*.png|*.bmp)
. You could add that to your for-loop by globbing"$origin"/!(*.jpg|*.png|*.bmp)
which would avoid the need for the switch used in Kaz's answer and keep the simple body of the for-loop.
– Score_Under
Jun 21 '15 at 3:52
Awesome Score. I incorporated your comment and updated my answer.
– Whitecat
Jun 27 '15 at 22:55
add a comment |
A more simple solution using "$origin"/!(*.jpg|*.png|*.bmp)
instead of a catch block:
for file in "$origin"/!(*.jpg|*.png|*.bmp); do mv -- "$file" "$destination" ; done
Thanks to @Score_Under
For a multi-line script you can do the following (notice the ;
before the done
is dropped):
for file in "$origin"/!(*.jpg|*.png|*.bmp); do # don't copy types *.jpg|*.png|*.bmp
mv -- "$file" "$destination"
done
To do a more generalized solution that moves all files, you can do the one-liner:
for file in "$origin"/*; do mv -- "$file" "$destination" ; done
Which looks like this if you do indentation:
for file in "$origin"/*; do
mv -- "$file" "$destination"
done
This takes every file in the origin and moves them one by one to the destination. The quotes around $file
are necessary in case there are spaces or other special characters in the filenames.
Here is an example of this method that worked perfectly
for file in "/Users/william/Pictures/export_folder_111210/"*.jpg; do
mv -- "$file" "/Users/william/Desktop/southland/landingphotos/";
done
You could use something like the original glob in the for-loop to get a closer solution to what's being asked for.
– Score_Under
Jun 20 '15 at 0:34
What do you mean original glob?
– Whitecat
Jun 20 '15 at 0:38
Sorry if that was a little cryptic, I was referring to the glob in the question:!(*.jpg|*.png|*.bmp)
. You could add that to your for-loop by globbing"$origin"/!(*.jpg|*.png|*.bmp)
which would avoid the need for the switch used in Kaz's answer and keep the simple body of the for-loop.
– Score_Under
Jun 21 '15 at 3:52
Awesome Score. I incorporated your comment and updated my answer.
– Whitecat
Jun 27 '15 at 22:55
add a comment |
A more simple solution using "$origin"/!(*.jpg|*.png|*.bmp)
instead of a catch block:
for file in "$origin"/!(*.jpg|*.png|*.bmp); do mv -- "$file" "$destination" ; done
Thanks to @Score_Under
For a multi-line script you can do the following (notice the ;
before the done
is dropped):
for file in "$origin"/!(*.jpg|*.png|*.bmp); do # don't copy types *.jpg|*.png|*.bmp
mv -- "$file" "$destination"
done
To do a more generalized solution that moves all files, you can do the one-liner:
for file in "$origin"/*; do mv -- "$file" "$destination" ; done
Which looks like this if you do indentation:
for file in "$origin"/*; do
mv -- "$file" "$destination"
done
This takes every file in the origin and moves them one by one to the destination. The quotes around $file
are necessary in case there are spaces or other special characters in the filenames.
Here is an example of this method that worked perfectly
for file in "/Users/william/Pictures/export_folder_111210/"*.jpg; do
mv -- "$file" "/Users/william/Desktop/southland/landingphotos/";
done
A more simple solution using "$origin"/!(*.jpg|*.png|*.bmp)
instead of a catch block:
for file in "$origin"/!(*.jpg|*.png|*.bmp); do mv -- "$file" "$destination" ; done
Thanks to @Score_Under
For a multi-line script you can do the following (notice the ;
before the done
is dropped):
for file in "$origin"/!(*.jpg|*.png|*.bmp); do # don't copy types *.jpg|*.png|*.bmp
mv -- "$file" "$destination"
done
To do a more generalized solution that moves all files, you can do the one-liner:
for file in "$origin"/*; do mv -- "$file" "$destination" ; done
Which looks like this if you do indentation:
for file in "$origin"/*; do
mv -- "$file" "$destination"
done
This takes every file in the origin and moves them one by one to the destination. The quotes around $file
are necessary in case there are spaces or other special characters in the filenames.
Here is an example of this method that worked perfectly
for file in "/Users/william/Pictures/export_folder_111210/"*.jpg; do
mv -- "$file" "/Users/william/Desktop/southland/landingphotos/";
done
edited Jun 15 '17 at 16:52
answered Jun 19 '15 at 21:07
WhitecatWhitecat
218112
218112
You could use something like the original glob in the for-loop to get a closer solution to what's being asked for.
– Score_Under
Jun 20 '15 at 0:34
What do you mean original glob?
– Whitecat
Jun 20 '15 at 0:38
Sorry if that was a little cryptic, I was referring to the glob in the question:!(*.jpg|*.png|*.bmp)
. You could add that to your for-loop by globbing"$origin"/!(*.jpg|*.png|*.bmp)
which would avoid the need for the switch used in Kaz's answer and keep the simple body of the for-loop.
– Score_Under
Jun 21 '15 at 3:52
Awesome Score. I incorporated your comment and updated my answer.
– Whitecat
Jun 27 '15 at 22:55
add a comment |
You could use something like the original glob in the for-loop to get a closer solution to what's being asked for.
– Score_Under
Jun 20 '15 at 0:34
What do you mean original glob?
– Whitecat
Jun 20 '15 at 0:38
Sorry if that was a little cryptic, I was referring to the glob in the question:!(*.jpg|*.png|*.bmp)
. You could add that to your for-loop by globbing"$origin"/!(*.jpg|*.png|*.bmp)
which would avoid the need for the switch used in Kaz's answer and keep the simple body of the for-loop.
– Score_Under
Jun 21 '15 at 3:52
Awesome Score. I incorporated your comment and updated my answer.
– Whitecat
Jun 27 '15 at 22:55
You could use something like the original glob in the for-loop to get a closer solution to what's being asked for.
– Score_Under
Jun 20 '15 at 0:34
You could use something like the original glob in the for-loop to get a closer solution to what's being asked for.
– Score_Under
Jun 20 '15 at 0:34
What do you mean original glob?
– Whitecat
Jun 20 '15 at 0:38
What do you mean original glob?
– Whitecat
Jun 20 '15 at 0:38
Sorry if that was a little cryptic, I was referring to the glob in the question:
!(*.jpg|*.png|*.bmp)
. You could add that to your for-loop by globbing "$origin"/!(*.jpg|*.png|*.bmp)
which would avoid the need for the switch used in Kaz's answer and keep the simple body of the for-loop.– Score_Under
Jun 21 '15 at 3:52
Sorry if that was a little cryptic, I was referring to the glob in the question:
!(*.jpg|*.png|*.bmp)
. You could add that to your for-loop by globbing "$origin"/!(*.jpg|*.png|*.bmp)
which would avoid the need for the switch used in Kaz's answer and keep the simple body of the for-loop.– Score_Under
Jun 21 '15 at 3:52
Awesome Score. I incorporated your comment and updated my answer.
– Whitecat
Jun 27 '15 at 22:55
Awesome Score. I incorporated your comment and updated my answer.
– Whitecat
Jun 27 '15 at 22:55
add a comment |
If working with Linux kernel is enough you can simply do
ulimit -s 100000
that will work because Linux kernel included a patch around 10 years ago that changed argument limit to be based on stack size: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=b6a2fea39318e43fee84fa7b0b90d68bed92d2ba
Update: If you feel brave, you can say
ulimit -s unlimited
and you'll be fine with any shell expansions as long as you have enough RAM.
That's a hack. How would you know what to set the stack limit to? This also affects other processes started in the same session.
– Kusalananda
Nov 1 '17 at 9:38
Yeah, it's a hack. Most of the time this kind of hacks are one-off (how often you manually move huge amount of files anyway?). If you are sure that the process is not going to eat all your RAM, you can setulimit -s unlimited
and it will work for practically unlimited files.
– Mikko Rantalainen
Nov 3 '17 at 13:00
Withulimit -s unlimited
the actual command line limit is 2^31 or 2 GB. (MAX_ARG_STRLEN
in kernel source.)
– Mikko Rantalainen
Jun 8 '18 at 7:01
add a comment |
If working with Linux kernel is enough you can simply do
ulimit -s 100000
that will work because Linux kernel included a patch around 10 years ago that changed argument limit to be based on stack size: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=b6a2fea39318e43fee84fa7b0b90d68bed92d2ba
Update: If you feel brave, you can say
ulimit -s unlimited
and you'll be fine with any shell expansions as long as you have enough RAM.
That's a hack. How would you know what to set the stack limit to? This also affects other processes started in the same session.
– Kusalananda
Nov 1 '17 at 9:38
Yeah, it's a hack. Most of the time this kind of hacks are one-off (how often you manually move huge amount of files anyway?). If you are sure that the process is not going to eat all your RAM, you can setulimit -s unlimited
and it will work for practically unlimited files.
– Mikko Rantalainen
Nov 3 '17 at 13:00
Withulimit -s unlimited
the actual command line limit is 2^31 or 2 GB. (MAX_ARG_STRLEN
in kernel source.)
– Mikko Rantalainen
Jun 8 '18 at 7:01
add a comment |
If working with Linux kernel is enough you can simply do
ulimit -s 100000
that will work because Linux kernel included a patch around 10 years ago that changed argument limit to be based on stack size: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=b6a2fea39318e43fee84fa7b0b90d68bed92d2ba
Update: If you feel brave, you can say
ulimit -s unlimited
and you'll be fine with any shell expansions as long as you have enough RAM.
If working with Linux kernel is enough you can simply do
ulimit -s 100000
that will work because Linux kernel included a patch around 10 years ago that changed argument limit to be based on stack size: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=b6a2fea39318e43fee84fa7b0b90d68bed92d2ba
Update: If you feel brave, you can say
ulimit -s unlimited
and you'll be fine with any shell expansions as long as you have enough RAM.
edited Nov 3 '17 at 13:04
answered Nov 1 '17 at 9:24
Mikko RantalainenMikko Rantalainen
1,472815
1,472815
That's a hack. How would you know what to set the stack limit to? This also affects other processes started in the same session.
– Kusalananda
Nov 1 '17 at 9:38
Yeah, it's a hack. Most of the time this kind of hacks are one-off (how often you manually move huge amount of files anyway?). If you are sure that the process is not going to eat all your RAM, you can setulimit -s unlimited
and it will work for practically unlimited files.
– Mikko Rantalainen
Nov 3 '17 at 13:00
Withulimit -s unlimited
the actual command line limit is 2^31 or 2 GB. (MAX_ARG_STRLEN
in kernel source.)
– Mikko Rantalainen
Jun 8 '18 at 7:01
add a comment |
That's a hack. How would you know what to set the stack limit to? This also affects other processes started in the same session.
– Kusalananda
Nov 1 '17 at 9:38
Yeah, it's a hack. Most of the time this kind of hacks are one-off (how often you manually move huge amount of files anyway?). If you are sure that the process is not going to eat all your RAM, you can setulimit -s unlimited
and it will work for practically unlimited files.
– Mikko Rantalainen
Nov 3 '17 at 13:00
Withulimit -s unlimited
the actual command line limit is 2^31 or 2 GB. (MAX_ARG_STRLEN
in kernel source.)
– Mikko Rantalainen
Jun 8 '18 at 7:01
That's a hack. How would you know what to set the stack limit to? This also affects other processes started in the same session.
– Kusalananda
Nov 1 '17 at 9:38
That's a hack. How would you know what to set the stack limit to? This also affects other processes started in the same session.
– Kusalananda
Nov 1 '17 at 9:38
Yeah, it's a hack. Most of the time this kind of hacks are one-off (how often you manually move huge amount of files anyway?). If you are sure that the process is not going to eat all your RAM, you can set
ulimit -s unlimited
and it will work for practically unlimited files.– Mikko Rantalainen
Nov 3 '17 at 13:00
Yeah, it's a hack. Most of the time this kind of hacks are one-off (how often you manually move huge amount of files anyway?). If you are sure that the process is not going to eat all your RAM, you can set
ulimit -s unlimited
and it will work for practically unlimited files.– Mikko Rantalainen
Nov 3 '17 at 13:00
With
ulimit -s unlimited
the actual command line limit is 2^31 or 2 GB. (MAX_ARG_STRLEN
in kernel source.)– Mikko Rantalainen
Jun 8 '18 at 7:01
With
ulimit -s unlimited
the actual command line limit is 2^31 or 2 GB. (MAX_ARG_STRLEN
in kernel source.)– Mikko Rantalainen
Jun 8 '18 at 7:01
add a comment |
You can get around that restriction while still using mv
if you don't mind running it a couple times.
You can move portions at a time. Let's say for example you had a long list of alphanumeric file names.
mv ./subdir/a* ./
That works. Then knock out another big chunk. After a couple moves, you can just go back to using mv ./subdir/* ./
add a comment |
You can get around that restriction while still using mv
if you don't mind running it a couple times.
You can move portions at a time. Let's say for example you had a long list of alphanumeric file names.
mv ./subdir/a* ./
That works. Then knock out another big chunk. After a couple moves, you can just go back to using mv ./subdir/* ./
add a comment |
You can get around that restriction while still using mv
if you don't mind running it a couple times.
You can move portions at a time. Let's say for example you had a long list of alphanumeric file names.
mv ./subdir/a* ./
That works. Then knock out another big chunk. After a couple moves, you can just go back to using mv ./subdir/* ./
You can get around that restriction while still using mv
if you don't mind running it a couple times.
You can move portions at a time. Let's say for example you had a long list of alphanumeric file names.
mv ./subdir/a* ./
That works. Then knock out another big chunk. After a couple moves, you can just go back to using mv ./subdir/* ./
answered Nov 12 '15 at 2:46
KristianKristian
1113
1113
add a comment |
add a comment |
Sometimes it's easiest to just write a little script, e.g. in Python:
import glob, shutil
for i in glob.glob('*.jpg'):
shutil.move(i, 'new_dir/' + i)
add a comment |
Sometimes it's easiest to just write a little script, e.g. in Python:
import glob, shutil
for i in glob.glob('*.jpg'):
shutil.move(i, 'new_dir/' + i)
add a comment |
Sometimes it's easiest to just write a little script, e.g. in Python:
import glob, shutil
for i in glob.glob('*.jpg'):
shutil.move(i, 'new_dir/' + i)
Sometimes it's easiest to just write a little script, e.g. in Python:
import glob, shutil
for i in glob.glob('*.jpg'):
shutil.move(i, 'new_dir/' + i)
answered Oct 10 '18 at 22:22
duhaimeduhaime
1435
1435
add a comment |
add a comment |
Here is my two cents, append this into .bash_profile
mv() {
if [[ -d $1 ]]; then #directory mv
/bin/mv $1 $2
elif [[ -f $1 ]]; then #file mv
/bin/mv $1 $2
else
for f in $1
do
source_path=$f
#echo $source_path
source_file=${source_path##*/}
#echo $source_file
destination_path=${2%/} #get rid of trailing forward slash
echo "Moving $f to $destination_path/$source_file"
/bin/mv $f $destination_path/$source_file
done
fi
}
export -f mv
Usage
mv '*.jpg' ./destination/
mv '/path/*' ./destination/
New contributor
add a comment |
Here is my two cents, append this into .bash_profile
mv() {
if [[ -d $1 ]]; then #directory mv
/bin/mv $1 $2
elif [[ -f $1 ]]; then #file mv
/bin/mv $1 $2
else
for f in $1
do
source_path=$f
#echo $source_path
source_file=${source_path##*/}
#echo $source_file
destination_path=${2%/} #get rid of trailing forward slash
echo "Moving $f to $destination_path/$source_file"
/bin/mv $f $destination_path/$source_file
done
fi
}
export -f mv
Usage
mv '*.jpg' ./destination/
mv '/path/*' ./destination/
New contributor
add a comment |
Here is my two cents, append this into .bash_profile
mv() {
if [[ -d $1 ]]; then #directory mv
/bin/mv $1 $2
elif [[ -f $1 ]]; then #file mv
/bin/mv $1 $2
else
for f in $1
do
source_path=$f
#echo $source_path
source_file=${source_path##*/}
#echo $source_file
destination_path=${2%/} #get rid of trailing forward slash
echo "Moving $f to $destination_path/$source_file"
/bin/mv $f $destination_path/$source_file
done
fi
}
export -f mv
Usage
mv '*.jpg' ./destination/
mv '/path/*' ./destination/
New contributor
Here is my two cents, append this into .bash_profile
mv() {
if [[ -d $1 ]]; then #directory mv
/bin/mv $1 $2
elif [[ -f $1 ]]; then #file mv
/bin/mv $1 $2
else
for f in $1
do
source_path=$f
#echo $source_path
source_file=${source_path##*/}
#echo $source_file
destination_path=${2%/} #get rid of trailing forward slash
echo "Moving $f to $destination_path/$source_file"
/bin/mv $f $destination_path/$source_file
done
fi
}
export -f mv
Usage
mv '*.jpg' ./destination/
mv '/path/*' ./destination/
New contributor
New contributor
answered 11 mins ago
AkoAko
1011
1011
New contributor
New contributor
add a comment |
add a comment |
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f128559%2fsolving-mv-argument-list-too-long%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Related: Does "argument list too long" restriction apply to shell builtins?
– codeforester
Nov 22 '17 at 21:16