Aws ec2 - How to rsync files between two remotes?
I'm setting up a crontab server to run several jobs to copy files from prod servers to lower environment servers.
I need the cron server job to copy files from one server to another. Here is what I have.
the ip's have been modified
ssh -v -R localhost:50000:1.0.0.2:22 -i host1key.pem ec2-user@1.0.0.1 'rsync -e "ssh -i /home/ec2-user/host2key.pem -p 50000" -vuar /home/ec2-user/test.txt ec2-user@localhost:/home/ec2-user/test.txt'
I'm using two different pem keys and users. I would think this command would work but I get this error in the debug log. Here is more to it and only show the portion that is erroring. It connects to ec2-user@1.0.0.1 successfully. But errors on the 1.0.0.2.
debug1: connect_next: host 1.0.0.2 ([1.0.0.2]:22) in progress, fd=7
debug1: channel 1: new [127.0.0.1]
debug1: confirm forwarded-tcpip
debug1: channel 1: connected to 1.0.0.2 port 22
Host key verification failed.
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]
debug1: channel 0: free: client-session, nchannels 2
debug1: channel 1: free: 127.0.0.1, nchannels 1
Transferred: sent 5296, received 4736 bytes, in 0.9 seconds
Bytes per second: sent 5901.2, received 5277.2
debug1: Exit status 12
ssh rsync amazon-ec2
edited Aug 21 '18 at 1:38
Rui F Ribeiro
41.1k1479138
asked Jul 5 '17 at 22:55
chdev77chdev77
11
add a comment |
I'm setting up a crontab server to run several jobs to copy files from prod servers to lower environment servers.
I need the cron server job to copy files from one server to another. Here is what I have.
the ip's have been modified
ssh -v -R localhost:50000:1.0.0.2:22 -i host1key.pem ec2-user@1.0.0.1 'rsync -e "ssh -i /home/ec2-user/host2key.pem -p 50000" -vuar /home/ec2-user/test.txt ec2-user@localhost:/home/ec2-user/test.txt'
I'm using two different pem keys and users. I would think this command would work but I get this error in the debug log. Here is more to it and only show the portion that is erroring. It connects to ec2-user@1.0.0.1 successfully. But errors on the 1.0.0.2.
debug1: connect_next: host 1.0.0.2 ([1.0.0.2]:22) in progress, fd=7
debug1: channel 1: new [127.0.0.1]
debug1: confirm forwarded-tcpip
debug1: channel 1: connected to 1.0.0.2 port 22
Host key verification failed.
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]
debug1: channel 0: free: client-session, nchannels 2
debug1: channel 1: free: 127.0.0.1, nchannels 1
Transferred: sent 5296, received 4736 bytes, in 0.9 seconds
Bytes per second: sent 5901.2, received 5277.2
debug1: Exit status 12
ssh rsync amazon-ec2
edited Aug 21 '18 at 1:38
Rui F Ribeiro
41.1k1479138
asked Jul 5 '17 at 22:55
chdev77chdev77
11
Are the two hosts in the same VPC? In the same security group? In the same IP subnet?
– DopeGhoti
Jul 5 '17 at 23:10
@DopeGhoti - No, no and no. The prod server (1.0.0.1) is in one AWS account and, the lower sever (1.0.0.2) in another. Also, the cron server is in the same vpc, sg, sub as the product server.
– chdev77
Jul 6 '17 at 2:59
add a comment |
I'm setting up a crontab server to run several jobs to copy files from prod servers to lower environment servers.
I need the cron server job to copy files from one server to another. Here is what I have.
the ip's have been modified
ssh -v -R localhost:50000:1.0.0.2:22 -i host1key.pem ec2-user@1.0.0.1 'rsync -e "ssh -i /home/ec2-user/host2key.pem -p 50000" -vuar /home/ec2-user/test.txt ec2-user@localhost:/home/ec2-user/test.txt'
I'm using two different pem keys and users. I would think this command would work but I get this error in the debug log. Here is more to it and only show the portion that is erroring. It connects to ec2-user@1.0.0.1 successfully. But errors on the 1.0.0.2.
debug1: connect_next: host 1.0.0.2 ([1.0.0.2]:22) in progress, fd=7
debug1: channel 1: new [127.0.0.1]
debug1: confirm forwarded-tcpip
debug1: channel 1: connected to 1.0.0.2 port 22
Host key verification failed.
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]
debug1: channel 0: free: client-session, nchannels 2
debug1: channel 1: free: 127.0.0.1, nchannels 1
Transferred: sent 5296, received 4736 bytes, in 0.9 seconds
Bytes per second: sent 5901.2, received 5277.2
debug1: Exit status 12
ssh rsync amazon-ec2
edited Aug 21 '18 at 1:38
Rui F Ribeiro
41.1k1479138
asked Jul 5 '17 at 22:55
chdev77chdev77
11
I'm setting up a crontab server to run several jobs to copy files from prod servers to lower environment servers.
I need the cron server job to copy files from one server to another. Here is what I have.
the ip's have been modified
ssh -v -R localhost:50000:1.0.0.2:22 -i host1key.pem ec2-user@1.0.0.1 'rsync -e "ssh -i /home/ec2-user/host2key.pem -p 50000" -vuar /home/ec2-user/test.txt ec2-user@localhost:/home/ec2-user/test.txt'
I'm using two different pem keys and users. I would think this command would work but I get this error in the debug log. Here is more to it and only show the portion that is erroring. It connects to ec2-user@1.0.0.1 successfully. But errors on the 1.0.0.2.
debug1: connect_next: host 1.0.0.2 ([1.0.0.2]:22) in progress, fd=7
debug1: channel 1: new [127.0.0.1]
debug1: confirm forwarded-tcpip
debug1: channel 1: connected to 1.0.0.2 port 22
Host key verification failed.
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]
debug1: channel 0: free: client-session, nchannels 2
debug1: channel 1: free: 127.0.0.1, nchannels 1
Transferred: sent 5296, received 4736 bytes, in 0.9 seconds
Bytes per second: sent 5901.2, received 5277.2
debug1: Exit status 12
ssh rsync amazon-ec2
ssh rsync amazon-ec2
edited Aug 21 '18 at 1:38
Rui F Ribeiro
41.1k1479138
asked Jul 5 '17 at 22:55
chdev77chdev77
11
edited Aug 21 '18 at 1:38
Rui F Ribeiro
41.1k1479138
asked Jul 5 '17 at 22:55
chdev77chdev77
11
edited Aug 21 '18 at 1:38
Rui F Ribeiro
41.1k1479138
edited Aug 21 '18 at 1:38
Rui F Ribeiro
41.1k1479138
edited Aug 21 '18 at 1:38
Rui F Ribeiro
41.1k1479138
41.1k1479138
asked Jul 5 '17 at 22:55
chdev77chdev77
11
asked Jul 5 '17 at 22:55
chdev77chdev77
11
asked Jul 5 '17 at 22:55
chdev77chdev77
11
11
Are the two hosts in the same VPC? In the same security group? In the same IP subnet?
– DopeGhoti
Jul 5 '17 at 23:10
@DopeGhoti - No, no and no. The prod server (1.0.0.1) is in one AWS account and, the lower sever (1.0.0.2) in another. Also, the cron server is in the same vpc, sg, sub as the product server.
– chdev77
Jul 6 '17 at 2:59
add a comment |
Are the two hosts in the same VPC? In the same security group? In the same IP subnet?
– DopeGhoti
Jul 5 '17 at 23:10
@DopeGhoti - No, no and no. The prod server (1.0.0.1) is in one AWS account and, the lower sever (1.0.0.2) in another. Also, the cron server is in the same vpc, sg, sub as the product server.
– chdev77
Jul 6 '17 at 2:59
Are the two hosts in the same VPC? In the same security group? In the same IP subnet?
– DopeGhoti
Jul 5 '17 at 23:10
Are the two hosts in the same VPC? In the same security group? In the same IP subnet?
– DopeGhoti
Jul 5 '17 at 23:10
@DopeGhoti - No, no and no. The prod server (1.0.0.1) is in one AWS account and, the lower sever (1.0.0.2) in another. Also, the cron server is in the same vpc, sg, sub as the product server.
– chdev77
Jul 6 '17 at 2:59
@DopeGhoti - No, no and no. The prod server (1.0.0.1) is in one AWS account and, the lower sever (1.0.0.2) in another. Also, the cron server is in the same vpc, sg, sub as the product server.
– chdev77
Jul 6 '17 at 2:59
add a comment |
2 Answers
2
active
oldest
votes
I saw something on https://www.cloudinsidr.com/content/how-to-rsync-files-and-directories-between-remote-hosts/ I don't know if it will be useful to you.
It solved my own challenge.
answered Jul 11 '17 at 9:03
antzshrekantzshrek
3061212
add a comment |
I have exactly the same issue.
In order for RSYNC to do what you want, server 1.0.0.1 and 1.0.0.2 must have a connection between them. This may not seem obvious, however it is, in fact, true.
Reading a little bit between the lines, I'm going to stick my neck out and that there is no direct connection between .1 and .2 and there is no way to get one.
In that scenario, I have always pulled the data from .1 and then pushed it to .2. That makes it 2 processes and takes twice as long.
Hopefully, this answer will help other folks in the same situation.
I spent months trying to figure it out. Maybe this will save some of you a lot of time.
answered 8 mins ago
Scottie HScottie H
346
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "106"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f375608%2faws-ec2-how-to-rsync-files-between-two-remotes%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
I saw something on https://www.cloudinsidr.com/content/how-to-rsync-files-and-directories-between-remote-hosts/ I don't know if it will be useful to you.
It solved my own challenge.
answered Jul 11 '17 at 9:03
antzshrekantzshrek
3061212
add a comment |
I saw something on https://www.cloudinsidr.com/content/how-to-rsync-files-and-directories-between-remote-hosts/ I don't know if it will be useful to you.
It solved my own challenge.
answered Jul 11 '17 at 9:03
antzshrekantzshrek
3061212
add a comment |
I saw something on https://www.cloudinsidr.com/content/how-to-rsync-files-and-directories-between-remote-hosts/ I don't know if it will be useful to you.
It solved my own challenge.
answered Jul 11 '17 at 9:03
antzshrekantzshrek
3061212
I saw something on https://www.cloudinsidr.com/content/how-to-rsync-files-and-directories-between-remote-hosts/ I don't know if it will be useful to you.
It solved my own challenge.
answered Jul 11 '17 at 9:03
antzshrekantzshrek
3061212
answered Jul 11 '17 at 9:03
antzshrekantzshrek
3061212
answered Jul 11 '17 at 9:03
antzshrekantzshrek
3061212
answered Jul 11 '17 at 9:03
antzshrekantzshrek
3061212
3061212
add a comment |
add a comment |
I have exactly the same issue.
In order for RSYNC to do what you want, server 1.0.0.1 and 1.0.0.2 must have a connection between them. This may not seem obvious, however it is, in fact, true.
Reading a little bit between the lines, I'm going to stick my neck out and that there is no direct connection between .1 and .2 and there is no way to get one.
In that scenario, I have always pulled the data from .1 and then pushed it to .2. That makes it 2 processes and takes twice as long.
Hopefully, this answer will help other folks in the same situation.
I spent months trying to figure it out. Maybe this will save some of you a lot of time.
answered 8 mins ago
Scottie HScottie H
346
add a comment |
I have exactly the same issue.
In order for RSYNC to do what you want, server 1.0.0.1 and 1.0.0.2 must have a connection between them. This may not seem obvious, however it is, in fact, true.
Reading a little bit between the lines, I'm going to stick my neck out and that there is no direct connection between .1 and .2 and there is no way to get one.
In that scenario, I have always pulled the data from .1 and then pushed it to .2. That makes it 2 processes and takes twice as long.
Hopefully, this answer will help other folks in the same situation.
I spent months trying to figure it out. Maybe this will save some of you a lot of time.
answered 8 mins ago
Scottie HScottie H
346
add a comment |
I have exactly the same issue.
In order for RSYNC to do what you want, server 1.0.0.1 and 1.0.0.2 must have a connection between them. This may not seem obvious, however it is, in fact, true.
Reading a little bit between the lines, I'm going to stick my neck out and that there is no direct connection between .1 and .2 and there is no way to get one.
In that scenario, I have always pulled the data from .1 and then pushed it to .2. That makes it 2 processes and takes twice as long.
Hopefully, this answer will help other folks in the same situation.
I spent months trying to figure it out. Maybe this will save some of you a lot of time.
answered 8 mins ago
Scottie HScottie H
346
I have exactly the same issue.
In order for RSYNC to do what you want, server 1.0.0.1 and 1.0.0.2 must have a connection between them. This may not seem obvious, however it is, in fact, true.
Reading a little bit between the lines, I'm going to stick my neck out and that there is no direct connection between .1 and .2 and there is no way to get one.
In that scenario, I have always pulled the data from .1 and then pushed it to .2. That makes it 2 processes and takes twice as long.
Hopefully, this answer will help other folks in the same situation.
I spent months trying to figure it out. Maybe this will save some of you a lot of time.
answered 8 mins ago
Scottie HScottie H
346
answered 8 mins ago
Scottie HScottie H
346
answered 8 mins ago
Scottie HScottie H
346
answered 8 mins ago
Scottie HScottie H
346
346
add a comment |
add a comment |
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f375608%2faws-ec2-how-to-rsync-files-between-two-remotes%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Are the two hosts in the same VPC? In the same security group? In the same IP subnet?
– DopeGhoti
Jul 5 '17 at 23:10
@DopeGhoti - No, no and no. The prod server (1.0.0.1) is in one AWS account and, the lower sever (1.0.0.2) in another. Also, the cron server is in the same vpc, sg, sub as the product server.
– chdev77
Jul 6 '17 at 2:59