How to reverse the content of binary file?
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ margin-bottom:0;
}
7
I was solving a challenge where I found a data file with no extension. file command shows that it is a data file(application/octet-stream). hd command shows GNP. in the last line. So if I reverse this file then I will get the .PNG format file, I searched everywhere but I didn't find solution how to reverse content of binary file.
binary hexdump reverse-engineering
asked Jan 11 '18 at 17:44
Prvt_YadvPrvt_Yadv
3,13631329
add a comment |
7
I was solving a challenge where I found a data file with no extension. file command shows that it is a data file(application/octet-stream). hd command shows GNP. in the last line. So if I reverse this file then I will get the .PNG format file, I searched everywhere but I didn't find solution how to reverse content of binary file.
binary hexdump reverse-engineering
asked Jan 11 '18 at 17:44
Prvt_YadvPrvt_Yadv
3,13631329
add a comment |
7
7
7
1
I was solving a challenge where I found a data file with no extension. file command shows that it is a data file(application/octet-stream). hd command shows GNP. in the last line. So if I reverse this file then I will get the .PNG format file, I searched everywhere but I didn't find solution how to reverse content of binary file.
binary hexdump reverse-engineering
asked Jan 11 '18 at 17:44
Prvt_YadvPrvt_Yadv
3,13631329
I was solving a challenge where I found a data file with no extension. file command shows that it is a data file(application/octet-stream). hd command shows GNP. in the last line. So if I reverse this file then I will get the .PNG format file, I searched everywhere but I didn't find solution how to reverse content of binary file.
binary hexdump reverse-engineering
binary hexdump reverse-engineering
asked Jan 11 '18 at 17:44
Prvt_YadvPrvt_Yadv
3,13631329
asked Jan 11 '18 at 17:44
Prvt_YadvPrvt_Yadv
3,13631329
edited 1 hour ago
Prvt_Yadv
asked Jan 11 '18 at 17:44
Prvt_YadvPrvt_Yadv
3,13631329
asked Jan 11 '18 at 17:44
Prvt_YadvPrvt_Yadv
3,13631329
asked Jan 11 '18 at 17:44
Prvt_YadvPrvt_Yadv
3,13631329
3,13631329
add a comment |
add a comment |
5 Answers
5
active
oldest
votes
5
With xxd (from vim) and tac (from GNU coreutils, also tail -r on some systems):
< file.gnp xxd -p -c1 | tac | xxd -p -r > file.png
answered Jan 11 '18 at 21:41
Stéphane ChazelasStéphane Chazelas
313k57592948
Is there any way for this to be combined with vi.stackexchange.com/a/2237/10649? I tried all kind of combinations with no luck :(
– Iulian Onofrei
May 12 '18 at 17:26
This is not a solution because it will mirror all the file.
– Philippe Delteil
Nov 21 '18 at 3:33
add a comment |
5
With perl:
<file.gnp perl -0777 -F -ape '$_=reverse@F' > file.png
-a:awkmode were records are split into fields
-0777 -p: slurp-mode, file read into one$_record, modified record printed afterwards.
-F: empty field separator, so fields are individual bytes
$_=reverse@F: output record is the concatenation of the list of fields (@F) reversed.
answered Jan 11 '18 at 21:46
Stéphane ChazelasStéphane Chazelas
313k57592948
add a comment |
3
In zsh (the only shell that can internally deal with binary data (unless you want to consider ksh93's base64 encoding approach)):
zmodload zsh/mapfile
(LC_ALL=C; printf %s ${(s::Oa)mapfile[file.gnp]} > file.png)
LC_ALL=C: characters are bytes
$mapfile[file.gnp]: content offile.gnpfile
s::: split the string into its byte constituents
Oa: reverseOrder onarray subscript that array
answered Jan 11 '18 at 21:36
Stéphane ChazelasStéphane Chazelas
313k57592948
zshis not the only shell that can handle binary data.
– fpmurphy
Jan 12 '18 at 14:21
add a comment |
2
Here is one way of reversing a binary file using ksh93. I have left the code "loose" to make it easier to understand.
#!/bin/ksh93
typeset -b byte
redirect 3< image.gpj || exit 1
eof=$(3<#((EOF)))
read -r -u 3 -N 1 byte
printf "%B" byte > image.jpg
3<#((CUR - 1))
while (( $(3<#) > 0 ))
do
read -r -u 3 -N 1 byte
printf "%B" byte >> image.jpg
3<#((CUR - 2))
done
read -r -u 3 -N 1 byte
printf "%B" byte >> image.jpg
redirect 3<&- || echo 'cannot close FD 3'
exit 0
answered Jan 12 '18 at 14:15
fpmurphyfpmurphy
2,456916
nice. That's the only answer so far that doesn't involve storing the whole file in memory. However, it's terribly inefficient in that it makes several system calls for each byte of the file (and conversions to/from base64), so wouldn't be suitable for files that don't fit in memory either. On my machine, it processes files at about 10KB/s
– Stéphane Chazelas
Jan 12 '18 at 15:51
Note that the firstreadabove should read nothing as it's done at the end of the file.
– Stéphane Chazelas
Jan 12 '18 at 15:58
Trying to understand why it was so slow, I tried running it understraceandksh93seems to be behaving very weirdly, where it seeks all over the place within the file and reads large amounts at the time. Maybe a variant of github.com/att/ast/issues/15
– Stéphane Chazelas
Jan 12 '18 at 16:00
@StéphaneChazelas. No mystery as to why it is relatively slow. Within the loop it has to seek backwards each time it reads a byte. This can easily be significantly reduced by a factor of 20 or even more by reading and writing more than one byte at a time. The write side of things can similarly be optimized. Lots of other techniques are available to further speed things up. I will leave that exercise up to you.
– fpmurphy
Jan 13 '18 at 5:49
Trystraceon the script to see what I mean.ksh93reads the files thousands of times over. For instance, before reading the first byte, it seeks 64KiB off the end of the file, reads 64KiB, then seeks before the last byte and reads 1 byte and does something similar for every byte. Note that what you can do with those base64 encoded strings is limited, so if you read more than one byte at a time, it's going to be more difficult to extract the individual bytes of that.
– Stéphane Chazelas
Jan 13 '18 at 9:23
add a comment |
0
I tried the following:
tac -rs '.' input.gnp > output.png
The idea is to force 'tac' using any character as separator.
I tried that on a binary file and it seemed to work but any confirmation would be appreciated.
Main advantage is that it does not load file into memory.
answered Apr 16 '18 at 16:00
BouteilleBouteille
1
Doesn't work for me (here with GNUtac8.28) when the input contains newline characters.printf '1n2' | tac -rs . | od -vAn -tcoutputsn 2 1instead of2 n 1. You'd also needLC_ALL=Cor.could match multi-byte characters.
– Stéphane Chazelas
Nov 15 '18 at 20:23
LC_ALL=C tac -rs $'.\|n'seems to work though.
– Stéphane Chazelas
Nov 15 '18 at 20:25
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "106"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f416401%2fhow-to-reverse-the-content-of-binary-file%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
5 Answers
5
active
oldest
votes
5 Answers
5
active
oldest
votes
active
oldest
votes
active
oldest
votes
5
With xxd (from vim) and tac (from GNU coreutils, also tail -r on some systems):
< file.gnp xxd -p -c1 | tac | xxd -p -r > file.png
answered Jan 11 '18 at 21:41
Stéphane ChazelasStéphane Chazelas
313k57592948
Is there any way for this to be combined with vi.stackexchange.com/a/2237/10649? I tried all kind of combinations with no luck :(
– Iulian Onofrei
May 12 '18 at 17:26
This is not a solution because it will mirror all the file.
– Philippe Delteil
Nov 21 '18 at 3:33
add a comment |
5
With xxd (from vim) and tac (from GNU coreutils, also tail -r on some systems):
< file.gnp xxd -p -c1 | tac | xxd -p -r > file.png
answered Jan 11 '18 at 21:41
Stéphane ChazelasStéphane Chazelas
313k57592948
Is there any way for this to be combined with vi.stackexchange.com/a/2237/10649? I tried all kind of combinations with no luck :(
– Iulian Onofrei
May 12 '18 at 17:26
This is not a solution because it will mirror all the file.
– Philippe Delteil
Nov 21 '18 at 3:33
add a comment |
5
5
5
With xxd (from vim) and tac (from GNU coreutils, also tail -r on some systems):
< file.gnp xxd -p -c1 | tac | xxd -p -r > file.png
answered Jan 11 '18 at 21:41
Stéphane ChazelasStéphane Chazelas
313k57592948
With xxd (from vim) and tac (from GNU coreutils, also tail -r on some systems):
< file.gnp xxd -p -c1 | tac | xxd -p -r > file.png
answered Jan 11 '18 at 21:41
Stéphane ChazelasStéphane Chazelas
313k57592948
edited Jan 12 '18 at 6:53
answered Jan 11 '18 at 21:41
Stéphane ChazelasStéphane Chazelas
313k57592948
answered Jan 11 '18 at 21:41
Stéphane ChazelasStéphane Chazelas
313k57592948
answered Jan 11 '18 at 21:41
Stéphane ChazelasStéphane Chazelas
313k57592948
313k57592948
Is there any way for this to be combined with vi.stackexchange.com/a/2237/10649? I tried all kind of combinations with no luck :(
– Iulian Onofrei
May 12 '18 at 17:26
This is not a solution because it will mirror all the file.
– Philippe Delteil
Nov 21 '18 at 3:33
add a comment |
Is there any way for this to be combined with vi.stackexchange.com/a/2237/10649? I tried all kind of combinations with no luck :(
– Iulian Onofrei
May 12 '18 at 17:26
This is not a solution because it will mirror all the file.
– Philippe Delteil
Nov 21 '18 at 3:33
Is there any way for this to be combined with vi.stackexchange.com/a/2237/10649? I tried all kind of combinations with no luck :(
– Iulian Onofrei
May 12 '18 at 17:26
Is there any way for this to be combined with vi.stackexchange.com/a/2237/10649? I tried all kind of combinations with no luck :(
– Iulian Onofrei
May 12 '18 at 17:26
This is not a solution because it will mirror all the file.
– Philippe Delteil
Nov 21 '18 at 3:33
This is not a solution because it will mirror all the file.
– Philippe Delteil
Nov 21 '18 at 3:33
add a comment |
5
With perl:
<file.gnp perl -0777 -F -ape '$_=reverse@F' > file.png
-a:awkmode were records are split into fields
-0777 -p: slurp-mode, file read into one$_record, modified record printed afterwards.
-F: empty field separator, so fields are individual bytes
$_=reverse@F: output record is the concatenation of the list of fields (@F) reversed.
answered Jan 11 '18 at 21:46
Stéphane ChazelasStéphane Chazelas
313k57592948
add a comment |
5
With perl:
<file.gnp perl -0777 -F -ape '$_=reverse@F' > file.png
-a:awkmode were records are split into fields
-0777 -p: slurp-mode, file read into one$_record, modified record printed afterwards.
-F: empty field separator, so fields are individual bytes
$_=reverse@F: output record is the concatenation of the list of fields (@F) reversed.
answered Jan 11 '18 at 21:46
Stéphane ChazelasStéphane Chazelas
313k57592948
add a comment |
5
5
5
With perl:
<file.gnp perl -0777 -F -ape '$_=reverse@F' > file.png
-a:awkmode were records are split into fields
-0777 -p: slurp-mode, file read into one$_record, modified record printed afterwards.
-F: empty field separator, so fields are individual bytes
$_=reverse@F: output record is the concatenation of the list of fields (@F) reversed.
answered Jan 11 '18 at 21:46
Stéphane ChazelasStéphane Chazelas
313k57592948
With perl:
<file.gnp perl -0777 -F -ape '$_=reverse@F' > file.png
-a:awkmode were records are split into fields
-0777 -p: slurp-mode, file read into one$_record, modified record printed afterwards.
-F: empty field separator, so fields are individual bytes
$_=reverse@F: output record is the concatenation of the list of fields (@F) reversed.
answered Jan 11 '18 at 21:46
Stéphane ChazelasStéphane Chazelas
313k57592948
answered Jan 11 '18 at 21:46
Stéphane ChazelasStéphane Chazelas
313k57592948
answered Jan 11 '18 at 21:46
Stéphane ChazelasStéphane Chazelas
313k57592948
answered Jan 11 '18 at 21:46
Stéphane ChazelasStéphane Chazelas
313k57592948
313k57592948
add a comment |
add a comment |
3
In zsh (the only shell that can internally deal with binary data (unless you want to consider ksh93's base64 encoding approach)):
zmodload zsh/mapfile
(LC_ALL=C; printf %s ${(s::Oa)mapfile[file.gnp]} > file.png)
LC_ALL=C: characters are bytes
$mapfile[file.gnp]: content offile.gnpfile
s::: split the string into its byte constituents
Oa: reverseOrder onarray subscript that array
answered Jan 11 '18 at 21:36
Stéphane ChazelasStéphane Chazelas
313k57592948
zshis not the only shell that can handle binary data.
– fpmurphy
Jan 12 '18 at 14:21
add a comment |
3
In zsh (the only shell that can internally deal with binary data (unless you want to consider ksh93's base64 encoding approach)):
zmodload zsh/mapfile
(LC_ALL=C; printf %s ${(s::Oa)mapfile[file.gnp]} > file.png)
LC_ALL=C: characters are bytes
$mapfile[file.gnp]: content offile.gnpfile
s::: split the string into its byte constituents
Oa: reverseOrder onarray subscript that array
answered Jan 11 '18 at 21:36
Stéphane ChazelasStéphane Chazelas
313k57592948
zshis not the only shell that can handle binary data.
– fpmurphy
Jan 12 '18 at 14:21
add a comment |
3
3
3
In zsh (the only shell that can internally deal with binary data (unless you want to consider ksh93's base64 encoding approach)):
zmodload zsh/mapfile
(LC_ALL=C; printf %s ${(s::Oa)mapfile[file.gnp]} > file.png)
LC_ALL=C: characters are bytes
$mapfile[file.gnp]: content offile.gnpfile
s::: split the string into its byte constituents
Oa: reverseOrder onarray subscript that array
answered Jan 11 '18 at 21:36
Stéphane ChazelasStéphane Chazelas
313k57592948
In zsh (the only shell that can internally deal with binary data (unless you want to consider ksh93's base64 encoding approach)):
zmodload zsh/mapfile
(LC_ALL=C; printf %s ${(s::Oa)mapfile[file.gnp]} > file.png)
LC_ALL=C: characters are bytes
$mapfile[file.gnp]: content offile.gnpfile
s::: split the string into its byte constituents
Oa: reverseOrder onarray subscript that array
answered Jan 11 '18 at 21:36
Stéphane ChazelasStéphane Chazelas
313k57592948
edited Jan 12 '18 at 14:29
answered Jan 11 '18 at 21:36
Stéphane ChazelasStéphane Chazelas
313k57592948
answered Jan 11 '18 at 21:36
Stéphane ChazelasStéphane Chazelas
313k57592948
answered Jan 11 '18 at 21:36
Stéphane ChazelasStéphane Chazelas
313k57592948
313k57592948
zshis not the only shell that can handle binary data.
– fpmurphy
Jan 12 '18 at 14:21
add a comment |
zshis not the only shell that can handle binary data.
– fpmurphy
Jan 12 '18 at 14:21
zsh is not the only shell that can handle binary data.– fpmurphy
Jan 12 '18 at 14:21
zsh is not the only shell that can handle binary data.– fpmurphy
Jan 12 '18 at 14:21
add a comment |
2
Here is one way of reversing a binary file using ksh93. I have left the code "loose" to make it easier to understand.
#!/bin/ksh93
typeset -b byte
redirect 3< image.gpj || exit 1
eof=$(3<#((EOF)))
read -r -u 3 -N 1 byte
printf "%B" byte > image.jpg
3<#((CUR - 1))
while (( $(3<#) > 0 ))
do
read -r -u 3 -N 1 byte
printf "%B" byte >> image.jpg
3<#((CUR - 2))
done
read -r -u 3 -N 1 byte
printf "%B" byte >> image.jpg
redirect 3<&- || echo 'cannot close FD 3'
exit 0
answered Jan 12 '18 at 14:15
fpmurphyfpmurphy
2,456916
nice. That's the only answer so far that doesn't involve storing the whole file in memory. However, it's terribly inefficient in that it makes several system calls for each byte of the file (and conversions to/from base64), so wouldn't be suitable for files that don't fit in memory either. On my machine, it processes files at about 10KB/s
– Stéphane Chazelas
Jan 12 '18 at 15:51
Note that the firstreadabove should read nothing as it's done at the end of the file.
– Stéphane Chazelas
Jan 12 '18 at 15:58
Trying to understand why it was so slow, I tried running it understraceandksh93seems to be behaving very weirdly, where it seeks all over the place within the file and reads large amounts at the time. Maybe a variant of github.com/att/ast/issues/15
– Stéphane Chazelas
Jan 12 '18 at 16:00
@StéphaneChazelas. No mystery as to why it is relatively slow. Within the loop it has to seek backwards each time it reads a byte. This can easily be significantly reduced by a factor of 20 or even more by reading and writing more than one byte at a time. The write side of things can similarly be optimized. Lots of other techniques are available to further speed things up. I will leave that exercise up to you.
– fpmurphy
Jan 13 '18 at 5:49
Trystraceon the script to see what I mean.ksh93reads the files thousands of times over. For instance, before reading the first byte, it seeks 64KiB off the end of the file, reads 64KiB, then seeks before the last byte and reads 1 byte and does something similar for every byte. Note that what you can do with those base64 encoded strings is limited, so if you read more than one byte at a time, it's going to be more difficult to extract the individual bytes of that.
– Stéphane Chazelas
Jan 13 '18 at 9:23
add a comment |
2
Here is one way of reversing a binary file using ksh93. I have left the code "loose" to make it easier to understand.
#!/bin/ksh93
typeset -b byte
redirect 3< image.gpj || exit 1
eof=$(3<#((EOF)))
read -r -u 3 -N 1 byte
printf "%B" byte > image.jpg
3<#((CUR - 1))
while (( $(3<#) > 0 ))
do
read -r -u 3 -N 1 byte
printf "%B" byte >> image.jpg
3<#((CUR - 2))
done
read -r -u 3 -N 1 byte
printf "%B" byte >> image.jpg
redirect 3<&- || echo 'cannot close FD 3'
exit 0
answered Jan 12 '18 at 14:15
fpmurphyfpmurphy
2,456916
nice. That's the only answer so far that doesn't involve storing the whole file in memory. However, it's terribly inefficient in that it makes several system calls for each byte of the file (and conversions to/from base64), so wouldn't be suitable for files that don't fit in memory either. On my machine, it processes files at about 10KB/s
– Stéphane Chazelas
Jan 12 '18 at 15:51
Note that the firstreadabove should read nothing as it's done at the end of the file.
– Stéphane Chazelas
Jan 12 '18 at 15:58
Trying to understand why it was so slow, I tried running it understraceandksh93seems to be behaving very weirdly, where it seeks all over the place within the file and reads large amounts at the time. Maybe a variant of github.com/att/ast/issues/15
– Stéphane Chazelas
Jan 12 '18 at 16:00
@StéphaneChazelas. No mystery as to why it is relatively slow. Within the loop it has to seek backwards each time it reads a byte. This can easily be significantly reduced by a factor of 20 or even more by reading and writing more than one byte at a time. The write side of things can similarly be optimized. Lots of other techniques are available to further speed things up. I will leave that exercise up to you.
– fpmurphy
Jan 13 '18 at 5:49
Trystraceon the script to see what I mean.ksh93reads the files thousands of times over. For instance, before reading the first byte, it seeks 64KiB off the end of the file, reads 64KiB, then seeks before the last byte and reads 1 byte and does something similar for every byte. Note that what you can do with those base64 encoded strings is limited, so if you read more than one byte at a time, it's going to be more difficult to extract the individual bytes of that.
– Stéphane Chazelas
Jan 13 '18 at 9:23
add a comment |
2
2
2
Here is one way of reversing a binary file using ksh93. I have left the code "loose" to make it easier to understand.
#!/bin/ksh93
typeset -b byte
redirect 3< image.gpj || exit 1
eof=$(3<#((EOF)))
read -r -u 3 -N 1 byte
printf "%B" byte > image.jpg
3<#((CUR - 1))
while (( $(3<#) > 0 ))
do
read -r -u 3 -N 1 byte
printf "%B" byte >> image.jpg
3<#((CUR - 2))
done
read -r -u 3 -N 1 byte
printf "%B" byte >> image.jpg
redirect 3<&- || echo 'cannot close FD 3'
exit 0
answered Jan 12 '18 at 14:15
fpmurphyfpmurphy
2,456916
Here is one way of reversing a binary file using ksh93. I have left the code "loose" to make it easier to understand.
#!/bin/ksh93
typeset -b byte
redirect 3< image.gpj || exit 1
eof=$(3<#((EOF)))
read -r -u 3 -N 1 byte
printf "%B" byte > image.jpg
3<#((CUR - 1))
while (( $(3<#) > 0 ))
do
read -r -u 3 -N 1 byte
printf "%B" byte >> image.jpg
3<#((CUR - 2))
done
read -r -u 3 -N 1 byte
printf "%B" byte >> image.jpg
redirect 3<&- || echo 'cannot close FD 3'
exit 0
answered Jan 12 '18 at 14:15
fpmurphyfpmurphy
2,456916
edited Jan 12 '18 at 14:23
answered Jan 12 '18 at 14:15
fpmurphyfpmurphy
2,456916
answered Jan 12 '18 at 14:15
fpmurphyfpmurphy
2,456916
answered Jan 12 '18 at 14:15
fpmurphyfpmurphy
2,456916
2,456916
nice. That's the only answer so far that doesn't involve storing the whole file in memory. However, it's terribly inefficient in that it makes several system calls for each byte of the file (and conversions to/from base64), so wouldn't be suitable for files that don't fit in memory either. On my machine, it processes files at about 10KB/s
– Stéphane Chazelas
Jan 12 '18 at 15:51
Note that the firstreadabove should read nothing as it's done at the end of the file.
– Stéphane Chazelas
Jan 12 '18 at 15:58
Trying to understand why it was so slow, I tried running it understraceandksh93seems to be behaving very weirdly, where it seeks all over the place within the file and reads large amounts at the time. Maybe a variant of github.com/att/ast/issues/15
– Stéphane Chazelas
Jan 12 '18 at 16:00
@StéphaneChazelas. No mystery as to why it is relatively slow. Within the loop it has to seek backwards each time it reads a byte. This can easily be significantly reduced by a factor of 20 or even more by reading and writing more than one byte at a time. The write side of things can similarly be optimized. Lots of other techniques are available to further speed things up. I will leave that exercise up to you.
– fpmurphy
Jan 13 '18 at 5:49
Trystraceon the script to see what I mean.ksh93reads the files thousands of times over. For instance, before reading the first byte, it seeks 64KiB off the end of the file, reads 64KiB, then seeks before the last byte and reads 1 byte and does something similar for every byte. Note that what you can do with those base64 encoded strings is limited, so if you read more than one byte at a time, it's going to be more difficult to extract the individual bytes of that.
– Stéphane Chazelas
Jan 13 '18 at 9:23
add a comment |
nice. That's the only answer so far that doesn't involve storing the whole file in memory. However, it's terribly inefficient in that it makes several system calls for each byte of the file (and conversions to/from base64), so wouldn't be suitable for files that don't fit in memory either. On my machine, it processes files at about 10KB/s
– Stéphane Chazelas
Jan 12 '18 at 15:51
Note that the firstreadabove should read nothing as it's done at the end of the file.
– Stéphane Chazelas
Jan 12 '18 at 15:58
Trying to understand why it was so slow, I tried running it understraceandksh93seems to be behaving very weirdly, where it seeks all over the place within the file and reads large amounts at the time. Maybe a variant of github.com/att/ast/issues/15
– Stéphane Chazelas
Jan 12 '18 at 16:00
@StéphaneChazelas. No mystery as to why it is relatively slow. Within the loop it has to seek backwards each time it reads a byte. This can easily be significantly reduced by a factor of 20 or even more by reading and writing more than one byte at a time. The write side of things can similarly be optimized. Lots of other techniques are available to further speed things up. I will leave that exercise up to you.
– fpmurphy
Jan 13 '18 at 5:49
Trystraceon the script to see what I mean.ksh93reads the files thousands of times over. For instance, before reading the first byte, it seeks 64KiB off the end of the file, reads 64KiB, then seeks before the last byte and reads 1 byte and does something similar for every byte. Note that what you can do with those base64 encoded strings is limited, so if you read more than one byte at a time, it's going to be more difficult to extract the individual bytes of that.
– Stéphane Chazelas
Jan 13 '18 at 9:23
nice. That's the only answer so far that doesn't involve storing the whole file in memory. However, it's terribly inefficient in that it makes several system calls for each byte of the file (and conversions to/from base64), so wouldn't be suitable for files that don't fit in memory either. On my machine, it processes files at about 10KB/s
– Stéphane Chazelas
Jan 12 '18 at 15:51
nice. That's the only answer so far that doesn't involve storing the whole file in memory. However, it's terribly inefficient in that it makes several system calls for each byte of the file (and conversions to/from base64), so wouldn't be suitable for files that don't fit in memory either. On my machine, it processes files at about 10KB/s
– Stéphane Chazelas
Jan 12 '18 at 15:51
Note that the first
read above should read nothing as it's done at the end of the file.– Stéphane Chazelas
Jan 12 '18 at 15:58
Note that the first
read above should read nothing as it's done at the end of the file.– Stéphane Chazelas
Jan 12 '18 at 15:58
Trying to understand why it was so slow, I tried running it under
strace and ksh93 seems to be behaving very weirdly, where it seeks all over the place within the file and reads large amounts at the time. Maybe a variant of github.com/att/ast/issues/15– Stéphane Chazelas
Jan 12 '18 at 16:00
Trying to understand why it was so slow, I tried running it under
strace and ksh93 seems to be behaving very weirdly, where it seeks all over the place within the file and reads large amounts at the time. Maybe a variant of github.com/att/ast/issues/15– Stéphane Chazelas
Jan 12 '18 at 16:00
@StéphaneChazelas. No mystery as to why it is relatively slow. Within the loop it has to seek backwards each time it reads a byte. This can easily be significantly reduced by a factor of 20 or even more by reading and writing more than one byte at a time. The write side of things can similarly be optimized. Lots of other techniques are available to further speed things up. I will leave that exercise up to you.
– fpmurphy
Jan 13 '18 at 5:49
@StéphaneChazelas. No mystery as to why it is relatively slow. Within the loop it has to seek backwards each time it reads a byte. This can easily be significantly reduced by a factor of 20 or even more by reading and writing more than one byte at a time. The write side of things can similarly be optimized. Lots of other techniques are available to further speed things up. I will leave that exercise up to you.
– fpmurphy
Jan 13 '18 at 5:49
Try
strace on the script to see what I mean. ksh93 reads the files thousands of times over. For instance, before reading the first byte, it seeks 64KiB off the end of the file, reads 64KiB, then seeks before the last byte and reads 1 byte and does something similar for every byte. Note that what you can do with those base64 encoded strings is limited, so if you read more than one byte at a time, it's going to be more difficult to extract the individual bytes of that.– Stéphane Chazelas
Jan 13 '18 at 9:23
Try
strace on the script to see what I mean. ksh93 reads the files thousands of times over. For instance, before reading the first byte, it seeks 64KiB off the end of the file, reads 64KiB, then seeks before the last byte and reads 1 byte and does something similar for every byte. Note that what you can do with those base64 encoded strings is limited, so if you read more than one byte at a time, it's going to be more difficult to extract the individual bytes of that.– Stéphane Chazelas
Jan 13 '18 at 9:23
add a comment |
0
I tried the following:
tac -rs '.' input.gnp > output.png
The idea is to force 'tac' using any character as separator.
I tried that on a binary file and it seemed to work but any confirmation would be appreciated.
Main advantage is that it does not load file into memory.
answered Apr 16 '18 at 16:00
BouteilleBouteille
1
Doesn't work for me (here with GNUtac8.28) when the input contains newline characters.printf '1n2' | tac -rs . | od -vAn -tcoutputsn 2 1instead of2 n 1. You'd also needLC_ALL=Cor.could match multi-byte characters.
– Stéphane Chazelas
Nov 15 '18 at 20:23
LC_ALL=C tac -rs $'.\|n'seems to work though.
– Stéphane Chazelas
Nov 15 '18 at 20:25
add a comment |
0
I tried the following:
tac -rs '.' input.gnp > output.png
The idea is to force 'tac' using any character as separator.
I tried that on a binary file and it seemed to work but any confirmation would be appreciated.
Main advantage is that it does not load file into memory.
answered Apr 16 '18 at 16:00
BouteilleBouteille
1
Doesn't work for me (here with GNUtac8.28) when the input contains newline characters.printf '1n2' | tac -rs . | od -vAn -tcoutputsn 2 1instead of2 n 1. You'd also needLC_ALL=Cor.could match multi-byte characters.
– Stéphane Chazelas
Nov 15 '18 at 20:23
LC_ALL=C tac -rs $'.\|n'seems to work though.
– Stéphane Chazelas
Nov 15 '18 at 20:25
add a comment |
0
0
0
I tried the following:
tac -rs '.' input.gnp > output.png
The idea is to force 'tac' using any character as separator.
I tried that on a binary file and it seemed to work but any confirmation would be appreciated.
Main advantage is that it does not load file into memory.
answered Apr 16 '18 at 16:00
BouteilleBouteille
1
I tried the following:
tac -rs '.' input.gnp > output.png
The idea is to force 'tac' using any character as separator.
I tried that on a binary file and it seemed to work but any confirmation would be appreciated.
Main advantage is that it does not load file into memory.
answered Apr 16 '18 at 16:00
BouteilleBouteille
1
answered Apr 16 '18 at 16:00
BouteilleBouteille
1
answered Apr 16 '18 at 16:00
BouteilleBouteille
1
answered Apr 16 '18 at 16:00
BouteilleBouteille
1
1
Doesn't work for me (here with GNUtac8.28) when the input contains newline characters.printf '1n2' | tac -rs . | od -vAn -tcoutputsn 2 1instead of2 n 1. You'd also needLC_ALL=Cor.could match multi-byte characters.
– Stéphane Chazelas
Nov 15 '18 at 20:23
LC_ALL=C tac -rs $'.\|n'seems to work though.
– Stéphane Chazelas
Nov 15 '18 at 20:25
add a comment |
Doesn't work for me (here with GNUtac8.28) when the input contains newline characters.printf '1n2' | tac -rs . | od -vAn -tcoutputsn 2 1instead of2 n 1. You'd also needLC_ALL=Cor.could match multi-byte characters.
– Stéphane Chazelas
Nov 15 '18 at 20:23
LC_ALL=C tac -rs $'.\|n'seems to work though.
– Stéphane Chazelas
Nov 15 '18 at 20:25
Doesn't work for me (here with GNU
tac 8.28) when the input contains newline characters. printf '1n2' | tac -rs . | od -vAn -tc outputs n 2 1 instead of 2 n 1. You'd also need LC_ALL=C or . could match multi-byte characters.– Stéphane Chazelas
Nov 15 '18 at 20:23
Doesn't work for me (here with GNU
tac 8.28) when the input contains newline characters. printf '1n2' | tac -rs . | od -vAn -tc outputs n 2 1 instead of 2 n 1. You'd also need LC_ALL=C or . could match multi-byte characters.– Stéphane Chazelas
Nov 15 '18 at 20:23
LC_ALL=C tac -rs $'.\|n' seems to work though.– Stéphane Chazelas
Nov 15 '18 at 20:25
LC_ALL=C tac -rs $'.\|n' seems to work though.– Stéphane Chazelas
Nov 15 '18 at 20:25
add a comment |
draft saved
draft discarded
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f416401%2fhow-to-reverse-the-content-of-binary-file%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
