I'm trying to know in advance the approximate size of a tar file while it's being streamed to stdout. According to the spec (http://www.gnu.org/software/tar/manual/html_node/Standard.html), the first 500 bytes are the header (ASCII-formatted), and bytes 124 to 136 specify the file size.
But, because it's streaming, those bytes always display 00000000 since I suppose filesize is calculated on-the-fly, or at the end.
tar -cf - myfolder | dd count=12 skip=124 iflag=count_bytes,skip_bytes > filesize
Always results in:
00000000000^#
I'm not using compression, so the tarball is roughly the size of the original data. Can tar somehow provide this information in the header before completion?
Thanks.
The answer is no, it cannot be calculated in advance while streaming.
Related
Considering Google can't even do it, I'm assuming the answer is "No"?
I just went through the basic suggestions from audreyr's "Favicon Cheat Sheet" and created a favicon.ico file consisting of two optimized png files using ImageMagick like so:
$convert favicon-16.png favicon-32.png favicon.ico
My favicon-16.png file was 137 bytes after optimizing with optipng and my favicon-32.png file was 144 bytes after optimization.
So you can understand my surprise when the combined favicon.ico file created by ImageMagick ended up being 5,430 bytes. Coincidentally, that's the exact same size as Google's official favicon.ico file.
Is 5,430 bytes the absolute minimum size for any true image/x-icon file?
That seems a little excessive when realistically every single browser accessing my favicon.ico file will be extracting the 144 byte 32x32 png version.
If the source favicon-16.png and favicon-32.png are truecolor (RGB888 or RGBA8888), ImageMagick will write an uncompressed 5430-byte ICO file. However, if they are indexed-color (i.e., in PNG8 format) or grayscale the ICO may be smaller (I observe 3638-byte ICO files in these cases).
The images are stored within the ICO in BMP format, not PNG (only 256x256 images get stored in PNG format inside the ICO).
Let's suppose I have a chunk of deflated data (as in the compressed data right after the first 2 bytes and before the ADLER32 check in a zlib compressed file)
How can I know the length of that chunk? How can I find where it ends?
You would need to get that information from some metadata external to the zlib block. (The same is true of the uncompressed size.) Or you could decompress and see where you end up.
Compressed blocks in deflate format are self-delimiting, so the decoder will terminate at the correct point (unless the datastream has been corrupted).
Most file storage and data transmission formats which include compressed data make this metadata available. But since it is not necessary for decompression, it is not stored in the compressed stream.
The only way to find where it ends is to decompress it. deflate streams are self-terminating.
I'm using tshark to extract specific TCP streams and write that to an output pcap file using the -w option.
But, the frames in the output pcap do not have any timestamps or delta times (they're all zero while in the original pcap there are timestamps and delta times for the frames).
Is there any way to ensure that the original timestamps (from the original pcap file) are preserved in the output pcap?
I'm using TShark 1.10.5 (SVN Rev 54262 from /trunk-1.10) on MacOS.
Thanks!
the frames in the output pcap do not have any timestamps or delta times (they're all zero while in the original pcap there are timestamps and delta times for the frames).
That is what is technically known as a "bug". Please file it as a bug on the Wireshark Bugzilla; if you can attach your original pcap file for testing purposes, that would be good. (If not, please run the file command on it and show the results, just so we know what file type the input file is - it might, for example, be a pcap-ng file rather than a pcap file).
I use ffmpeg to encode my sample videos following the recommanded bitrates in Technical Note TN2224, then use HLS tools to segment it and create playlists, finally create the variant plist file "all.m3u8"
I used the validation tool to validate my HLS content, it ended up showing except for the 64k audio only bandwidth is low, others are stay in the same bandwidth, I opened "all.m3u8" using text editor and seeing that all other bitrate contents are using the same bandwidth. No matter how I change parameters in the ffmpeg command, I still can't correct them. The following command is the one I used to encode contents:
ffmpeg -i input.m4v -acodec libfaac -vcodec libx264 -s 480x360 -b 350k -r 29.97 -vpre medium output.mp4
The following command is for generating the segments and plists:mediafilesegmenter -b http://www.example.com/stream/ -I -f ~/Documents/sample/ output.mp4
The following command is for generating the all.m3u8:variantplaylistcreator -o all.m3u8 http://www.example.com/stream/110/prog_index.m3u8 ~/Documents/sample/110/prog_index.m3u8 -iframe-url http://www.freeyourteam.com/stream/110/iframe_index.m3u8 http://www.example.com/stream/200/prog_index.m3u8 ~/Documents/sample/200/prog_index.m3u8 -iframe-url http://www.freeyourteam.com/stream/200/iframe_index.m3u8 http://www.example.com/stream/350/prog_index.m3u8 ~/Documents/sample/350/prog_index.m3u8 -iframe-url http://www.freeyourteam.com/stream/350/iframe_index.m3u8 http://www.example.com/stream/550/prog_index.m3u8 ~/Documents/sample/550/prog_index.m3u8 -iframe-url http://www.freeyourteam.com/stream/550/iframe_index.m3u8 http://www.example.com/stream/64/prog_index.m3u8 ~/Documents/sample/64/prog_index.m3u8
and in my "all.m3u8", the bandwidths are all 523894:
Please allow me to ask two more basic questions:
In the tech note, recommanded bitrates are 64 Kbps, 110 Kbps, 200 Kbps, 350 Kbps, 550 Kbps, I wonder if this value includes the audio bitrate or exclude the audio.
How do you insert keyframe to segment? Because in the document it says:"You must include at least one keyframe per segment, preferably more. If you only include one, put it at the beginning of the segment." I don't quite get how you can do it.
Thank you very much for your help and I do appreciate your time.
Jason,
To create all.m3u8 should it not be given multiple m3u8 files each corresponding to a different bitrate?
I am guessing you run ffmpeg say 4 times to create for 4 bitrate files. Then you run the segmenter 4 times to create 4 set of segments and its individual m3u8 files.
Finally you have to tell the variantplaylistcreator where the location of the various m3u8 files per bitrate to create a single master m3u8 file.
Eg.
variantplaylistcreator -o mymedia_all.m3u8 http://mywebserver/mymedia_lo/prog_index.m3u8 mymedia_lo.plist http://mywebserver/mymedia_med/prog_index.m3u8 mymedia_med.plist http://mywebserver/mymedia_hi/prog_index.m3u8 mymedia_hi.plist
I don't see you providing the various filese seperately. I hope you get the picture.
EDIT: To answer your other questions:
Bitrates include audio. What you need to do is ensure you have a fixed key frame interval in your encoding. This will allow the segmenter to segment the files at regular intervals. you don't insert anything anywhere.
Out of curiosity why not directly use ffmpeg to give you the output segmented files? It supports it.
Thanks for everybody's attention and suggestions. I finally figured it out. The reason why the bandwidth stayed the same for different bitrate is that my ffmpeg command missed couple settings. I ended up using the following command:ffmpeg -i inputVideo.m4v -f mpegts -acodec libfaac -ar 44100 -ab 64k -vcodec libx264 -b 350k -s 480x360 -r 29.97 -flags +loop -cmp +chroma -partitions +parti4x4+partp8x8+partb8x8 -subq 5 -trellis 1 -refs 1 -coder 0 -me_range 16 -keyint_min 25 -sc_threshold 40 -i_qfactor 0.71 -bt 200k -maxrate 350k -bufsize 350k -rc_eq 'blurCplx^(1-qComp)' -qcomp 0.6 -qmin 10 -qmax 51 -qdiff 4 -level 30 -aspect 4:3 -g 30 -async 2 output.ts
I put it here so that other people who have the same problem as me will have a reference.
It sounds like you may have uncovered a bug in variantplaylistcreator. I recommend to verify that the sub-streams really are the bitrate you expect, and if it's really putting the wrong value, to report it to apple.
It might have something to do with using multiple -iframe-url. I can't understand why it would be necessary to specify it more than once. Adaptive streaming won't work if the substreams have different I-frame positions -- at least all of the segment boundaries must be aligned.
If you need to fix the playlist up programmatically, I recommend to use ffprobe (from ffmpeg suite) to extract the bitrate of each substream, and replace the bandwidth number with the extracted value.
I have video files, that I want to pad with a random number of extra bytes, in order to create a different md5 checksum. Is there a way to do that, and keep them playable?
It depends on the video file format, but you should be able to just add the extra bytes to the end, and most video players should ignore them. Most video formats contain a lot of metadata about the video data (such as "the total video size is X bytes"), so they're robust against this sort of change.
One simple way to do this is to use the >> shell redirection operator to append data, e.g.:
# Add 32 random bytes to the end of the movie.avi
head -c 32 /dev/urandom >> movie.avi
Metadata would be a good thing to change. If the file has metadata about the time the film was made or the software used for encoding, changes to those values should not have any effect on the final result. You'd need to specify the format.
Yegor,
It depends entirely on the video format. Look it up on wikipedia, some have a end of file flag byte sequence, simply adding bytes after it will achieve your effect, others will not work out so simply.