Indy odd/strang behavior with TLS version

Indy odd/strang behavior with TLS version - delphi

I sometimes get TLSv1 even when I set the version explicit to TLSv1.2.
I get this exception:
First chance exception at $75D917D2. Exception class
EIdOSSLConnectError with message 'Error connecting with SSL. EOF was
observed that violates the protocol'.
My Delphi version:
Embarcadero® RAD Studio 10 Seattle Version 23.0.21418.4207
I am on Windows 10:
Microsoft Windows [Version 10.0.17134.345]
My Indy Version: 10.6.2.5311 - That's what's coming with Seattle
This is a trace when it works:
This is a trace when it does NOT work:
As you can see it is TLSv1. Why?
I had a look at following questions:
Using Indy 10 IdHTTP with TLS 1.2
EIdOSSLConnectError Error connecting with SSL - EOF was observed
Here is my example code:
procedure TOpenWeatherFr.SetIcon(const IconCode: String);
var
HTTPS: TSSLHTTP;
MS : TMemoryStream;
PNG: TPngImage;
URL: string;
begin
HTTPS := TSSLHTTP.Create(nil);
MS := TMemoryStream.Create;
PNG := TPngImage.Create;
URL := 'https://openweathermap.org/img/w/' + IconCode + '.png';
try
(HTTPS.IOHandler as TIdSSLIOHandlerSocketOpenSSL).SSLOptions.SSLVersions := [sslvTLSv1_2];
HTTPS.Get(URL, MS);
MS.Seek(0, soFromBeginning);
PNG.LoadFromStream(MS);
Image1.Picture.Assign(PNG);
finally
HTTPS.Free;
PNG.Free;
MS.Free;
end;
Image1.Visible := true;
end;
For the sake of completeness here is the code of TSSLHTTP. The purpose is an old Indy version without SNI Support.
unit SSLHTTP;
interface
uses
Classes, IdHTTP, IdSSLOpenSSLHeaders, IdCTypes;
type
TSSLHTTP = class(TIdHTTP)
private
procedure OnStatusInfoEx(ASender: TObject; const AsslSocket: PSSL; const AWhere, Aret: TIdC_INT; const AType, AMsg: String);
public
constructor Create(AOwner: TComponent);
end;
implementation
uses
IdSSLOpenSSL;
{ TSSLHTTP }
constructor TSSLHTTP.Create(AOwner: TComponent);
begin
inherited Create(AOwner);
IOHandler := TIdSSLIOHandlerSocketOpenSSL.Create(self);
(IOHandler as TIdSSLIOHandlerSocketOpenSSL).OnStatusInfoEx := OnStatusInfoEx;
end;
procedure TSSLHTTP.OnStatusInfoEx(ASender: TObject; const AsslSocket: PSSL;
const AWhere, Aret: TIdC_INT; const AType, AMsg: String);
begin
SSL_set_tlsext_host_name(AsslSocket, Request.Host);
end;
end.
Where is my mistake? What am I doing wrong? This is the URL of the image I download:
https://openweathermap.org/img/w/04d.png
EDIT: Adding paylod as hex dumb
Working payload - Client Hello:
0000 16 03 01 01 49 01 00 01 45 03 03 8b 3d e5 54 59
0010 f8 59 44 82 f7 14 92 45 50 e1 a3 86 68 3a c2 94
0020 76 be ea 8b 54 98 f3 27 69 50 af 00 00 94 c0 30
0030 c0 2c c0 28 c0 24 c0 14 c0 0a 00 a3 00 9f 00 6b
0040 00 6a 00 39 00 38 c0 32 c0 2e c0 2a c0 26 c0 0f
0050 c0 05 00 9d 00 3d 00 35 00 88 00 87 00 84 c0 2f
0060 c0 2b c0 27 c0 23 c0 13 c0 09 00 a2 00 9e 00 67
0070 00 40 00 33 00 32 c0 31 c0 2d c0 29 c0 25 c0 0e
0080 c0 04 00 9c 00 3c 00 2f 00 9a 00 99 00 45 00 44
0090 00 96 00 41 00 07 c0 11 c0 07 c0 0c c0 02 00 05
00a0 00 04 c0 12 c0 08 00 16 00 13 c0 0d c0 03 00 0a
00b0 00 15 00 12 00 09 00 14 00 11 00 08 00 06 00 03
00c0 00 ff 01 00 00 88 00 00 00 17 00 15 00 00 12 6f
00d0 70 65 6e 77 65 61 74 68 65 72 6d 61 70 2e 6f 72
00e0 67 00 0b 00 04 03 00 01 02 00 0a 00 34 00 32 00
00f0 0e 00 0d 00 19 00 0b 00 0c 00 18 00 09 00 0a 00
0100 16 00 17 00 08 00 06 00 07 00 14 00 15 00 04 00
0110 05 00 12 00 13 00 01 00 02 00 03 00 0f 00 10 00
0120 11 00 23 00 00 00 0d 00 20 00 1e 06 01 06 02 06
0130 03 05 01 05 02 05 03 04 01 04 02 04 03 03 01 03
0140 02 03 03 02 01 02 02 02 03 00 0f 00 01 01
Not working payload - Client hello:
0000 16 03 01 01 49 01 00 01 45 03 03 f6 7b de 81 5d
0010 76 74 a7 31 99 36 8d 17 4c 07 5e 73 5d f7 b8 a1
0020 4f 06 5e 91 e5 f0 4b 37 0d 65 e7 00 00 94 c0 30
0030 c0 2c c0 28 c0 24 c0 14 c0 0a 00 a3 00 9f 00 6b
0040 00 6a 00 39 00 38 c0 32 c0 2e c0 2a c0 26 c0 0f
0050 c0 05 00 9d 00 3d 00 35 00 88 00 87 00 84 c0 2f
0060 c0 2b c0 27 c0 23 c0 13 c0 09 00 a2 00 9e 00 67
0070 00 40 00 33 00 32 c0 31 c0 2d c0 29 c0 25 c0 0e
0080 c0 04 00 9c 00 3c 00 2f 00 9a 00 99 00 45 00 44
0090 00 96 00 41 00 07 c0 11 c0 07 c0 0c c0 02 00 05
00a0 00 04 c0 12 c0 08 00 16 00 13 c0 0d c0 03 00 0a
00b0 00 15 00 12 00 09 00 14 00 11 00 08 00 06 00 03
00c0 00 ff 01 00 00 88 00 00 00 17 00 15 00 00 12 6f
00d0 70 65 6e 77 65 61 74 68 65 72 6d 61 70 2e 6f 72
00e0 67 00 0b 00 04 03 00 01 02 00 0a 00 34 00 32 00
00f0 0e 00 0d 00 19 00 0b 00 0c 00 18 00 09 00 0a 00
0100 16 00 17 00 08 00 06 00 07 00 14 00 15 00 04 00
0110 05 00 12 00 13 00 01 00 02 00 03 00 0f 00 10 00
0120 11 00 23 00 00 00 0d 00 20 00 1e 06 01 06 02 06
0130 03 05 01 05 02 05 03 04 01 04 02 04 03 03 01 03
0140 02 03 03 02 01 02 02 02 03 00 0f 00 01 01

Related

Transferring photos from a Nikon Z50 with Airnef/PTPIP program is slowing down until disconnection

I am using Airnef (and the PTPIP protocol) to download pictures from my Nikon Z50 camera through Wi-Fi.
It works perfectly in some cases, but in other cases the transfer is slowing down until the camera has been disconnected.
I have read the associated documentation (available here https://www.testcams.com/airnef/) and seen that a similar issue has been encountered while the program has been developed, which seemed to be caused by a bug in the Nikon firmware which lead to some memory issues and a possible disconnection. The same issue seems to have been found during gphoto2 development (https://sourceforge.net/p/gphoto/mailman/gphoto-devel/thread/20190107075304.GC4614%40jet.franken.de/)
From what I understand, the parameters "maxgetobjbuffersizekb" and "maxgetobjtransferzekb" have been introduced to fix a such issue by trying to limit the requested sizes on the camera to transfer image, however it does not seems to work in my case.
I have tried to customize the values used by these parameters (for instance, by using a value of 256kb for maxgetobjtransferzekb setting), but with no real luck (the connection seems to abort less often but the transfer is still slowing down).
Here is an extract of the trace I get when I increase the log level in airnef.
Download history file "C:\Users\aerkis\AppData\Local\airnef\appdata\Z 50-SN6026496-downloadhist" loaded - 2 entries
Skipping DCIM - object is not file - MTP_OBJFORMAT_Assocation (0x3001)
Skipping 101NZ_50 - object is not file - MTP_OBJFORMAT_Assocation (0x3001)
>> MTP_OP_GetObject
Downloading "DSC_0490.JPG": 0%DSC_0490.JPG - downloading next piece, offset=0x0, count=0x100000
execMtpOp: MTP_OP_GetPartialObject - CmdReq payload:
0000: 06 00 00 00 01 00 00 00 - 1b 10 0b 00 00 00 ea 41 ........ - .......A
0010: 19 29 00 00 00 00 00 00 - 10 00 .)...... - ..
execMtpOp: MTP_OP_GetPartialObject - DataStart payload [expected data bytes is 0x100000]
0000: 09 00 00 00 0b 00 00 00 - 00 00 10 00 00 00 00 00 ........ - ........
0%execMtpOp: MTP_OP_GetPartialObject - Data payload [ID a] (0x00005b3c bytes):
0000: 0a 00 00 00 0b 00 00 00 - ff d8 ff e1 ff fe 45 78 ........ - ......Ex
0010: 69 66 00 00 49 49 2a 00 - 08 00 00 00 0d 00 0f 01 if..II*. - ........
0020: 02 00 12 00 00 00 ac 00 - 00 00 10 01 02 00 0b 00 ........ - ........
0030: 00 00 c0 00 00 00 12 01 - 03 00 01 00 00 00 01 00 ........ - ........
1%execMtpOp: MTP_OP_GetPartialObject - Data payload [ID a] (0x00005b3c bytes):
0000: 0a 00 00 00 0b 00 00 00 - 4f 00 72 a5 99 4e c4 fb ........ - O.r..N..
0010: f3 a3 26 61 66 1d 98 d7 - d7 98 1a 5d cc 26 ac f3 ..&af... - ...].&..
0020: f3 c4 4e 99 a5 72 00 4f - 5e 30 c2 15 12 f9 94 eb ..N..r.O - ^0......
0030: 03 dc 76 d1 ed ca 68 c7 - e7 c8 6a cd f1 d6 7c e3 ..v...h. - ..j...|.
execMtpOp: MTP_OP_GetPartialObject - Data payload [ID a] (0x00005b3c bytes):
0000: 0a 00 00 00 0b 00 00 00 - 00 00 00 00 00 00 00 00 ........ - ........
0010: 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00 ........ - ........
0020: 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00 ........ - ........
0030: 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00 ........ - ........
2%execMtpOp: MTP_OP_GetPartialObject - Partial payload after error (0x00004329 bytes):
0000: 0a 00 00 00 0b 00 00 00 - 20 20 20 20 20 20 20 20 ........ -
0010: 20 20 20 20 20 20 20 20 - 20 20 20 20 20 20 20 20 -
0020: 20 20 20 20 20 20 20 20 - 20 20 20 20 20 20 20 20 -
0030: 20 20 20 20 20 20 20 20 - 20 20 20 20 20 20 20 20 -
0040: 20 20 20 20 20 20 20 20 - 20 20 20 20 20 20 20 20 -
0050: 20 20 20 20 20 20 20 20 - 20 20 20 20 20 20 20 20 -
0060: 20 20 20 20 20 20 20 20 - 20 20 20 20 20 20 20 20 -
0070: 20 20 20 20 20 20 20 20 - 20 20 20 20 20 20 20 20 -
0080: 20 20 20 20 20 20 20 20 - 20 20 20 20 20 20 20 20 -
0090: 20 20 20 20 20 20 20 20 - 20 20 20 20 20 20 20 20 -
DSC_0490.JPG - error during download, writing 0x0 bytes of buffered data
DSC_0490.JPG - writing partial payload data of 0x154bd bytes
C:\Users\aerkis\Pictures\DSC_0490.JPG writing 0x154bd bytes, closeAfterWriting=0
MTP_OP_GetPartialObject: Socket error, partial data received - 0x4321 of 0x5b34 bytes for specific payload, 0x154bd of 0x100000 of total data bytes expected. Error: timed out
Therefore, I request some help to know if it exists a way to better understand this problem and possibly correct it (maybe by configuring some parameters ?).
Any help would be greatly appreciated !

Convert floating point values to hexadecimal

I'm struggling with a floating point to hex conversion in Lua. My application communicates with an old Akai S2000 sampler. The sampler codes two byte messages into four nibble values. The nibbles are in reverse order, hence the most significant nibble is last. There is one parameter that uses a binary encoding of the fraction part of the value. The two MS nibbles are used to encode the integral part of the value and the LS nibbles are used to encode the binary fraction.
Based on this discussion https://bytes.com/topic/c/answers/219928-how-convert-float-hex I have started to implement a Lua algorithm to generate these nibble values from a given parameter value. As I am not that strong with bit calculations I think I am doing many things the wrong way. There should be an easier way to compute these values and avoid a lot of my silly if/else hacks.
My code (pasted below) works for the positive numbers but it is a lot harder with the negative fractions.
In my tests I have added to tables with expected values. Each table works like this:
key = give value * 100
value = expected outcome from my algorithm
I.e. the first entry in the positiveNumbers table represents an input value of 0.01 and the expected output for that value is a four byte MemoryBlock containing 02 00 00 00 (The first two bytes represent the fraction and the last two the integral part).
Currently my algorithm fails at -0.94 and I can't hack around it without breaking for some other value.
Is there anybody who is string with bit calculations and that easily can see any noob mistake I have made especially with converting the negative values?
Any help or pointers would be appreciated!
Lua Code:
function float2nibbles(value)
local nibbles = MemoryBlock(4, true)
-- Retreive integral and fraction parts of the given value to be converted
local integ, fract = math.modf(math.abs(value))
-- Calculate the values of the integral part (last two nibbles)
local bi = BigInteger(integ)
if value < 0 then
-- This variable is sometimes added in the negative conversion of the MS nibbles
local lsAdd = 1
if integ == 0 then
lsAdd = 0
end
nibbles:setByte(2, bit.band(bit.bnot(bi:getBitRangeAsInt(0,4)) + lsAdd, 0xF))
nibbles:setByte(3, bit.band(bit.bnot(bi:getBitRangeAsInt(4,4)), 0xF))
else
nibbles:setByte(2, bit.band(bi:getBitRangeAsInt(0,4), 0xF))
nibbles:setByte(3, bit.band(bi:getBitRangeAsInt(4,4), 0xF))
end
-- Calculate the values of the fraction (first two nibbles)
local remainder = fract
local prevRemain = 0
for i = 1,2 do
remainder = remainder * 16
-- Integral part of the remainder
local d = math.modf(remainder)
if value < 0 and fract ~= 0 then
local lsAdd = 1
if fract == 0 or i == 1 then
lsAdd = 0
end
console(string.format("lsAdd %d", lsAdd))
nibbles:setByte(2 - i, bit.band(bit.bnot(d) + lsAdd, 0xF))
else
nibbles:setByte(2 - i, bit.band(d, 0xF))
end
console(string.format("fract %d = %d, %.2f", i, d, remainder))
prevRemain = remainder
remainder = remainder - d
end
-- For some reason this increment helps when the LS nibble should increment the value of the second nibble
if nibbles:getByte(0) == 0 and nibbles:getByte(1) ~= 0 and value < 0 then
console(string.format("weird increment { %d %d }", nibbles:getByte(0), nibbles:getByte(1)))
nibbles:setByte(1, nibbles:getByte(1) + 1)
end
-- The precision of this data is one byte but apparently they seem to use a third increment to check for rounding
remainder = remainder * 16
console(string.format("final remainder %.2f", remainder))
if math.abs(remainder - prevRemain) > 0.001 and remainder > 14 then
console(string.format("overflow -> %.2f (%.2f)", remainder, prevRemain))
if value < 0 then
nibbles:setByte(0, nibbles:getByte(0) - 1)
else
nibbles:setByte(0, nibbles:getByte(0) + 1)
end
end
console(string.format("%.2f : integral part %s (%s), fract %.2f", value, bit.tohex(integ, 2), nibbles:toHexString(1), fract))
return nibbles
end
local positiveNumbers = {
"02 00 00 00",
"05 00 00 00",
"07 00 00 00",
"0A 00 00 00",
"0C 00 00 00",
"0F 00 00 00",
"02 01 00 00",
"04 01 00 00",
"07 01 00 00",
"09 01 00 00",
"0C 01 00 00",
"0E 01 00 00",
"01 02 00 00",
"03 02 00 00",
"06 02 00 00",
"09 02 00 00",
"0B 02 00 00",
"0E 02 00 00",
"00 03 00 00",
"03 03 00 00",
"05 03 00 00",
"08 03 00 00",
"0B 03 00 00",
"0D 03 00 00",
"00 04 00 00",
"02 04 00 00",
"05 04 00 00",
"07 04 00 00",
"0A 04 00 00",
"0C 04 00 00",
"0F 04 00 00",
"02 05 00 00",
"04 05 00 00",
"07 05 00 00",
"09 05 00 00",
"0C 05 00 00",
"0E 05 00 00",
"01 06 00 00",
"03 06 00 00",
"06 06 00 00",
"09 06 00 00",
"0B 06 00 00",
"0E 06 00 00",
"00 07 00 00",
"03 07 00 00",
"05 07 00 00",
"08 07 00 00",
"0B 07 00 00",
"0D 07 00 00",
"00 08 00 00",
"02 08 00 00",
"05 08 00 00",
"07 08 00 00",
"0A 08 00 00",
"0C 08 00 00",
"0F 08 00 00",
"02 09 00 00",
"04 09 00 00",
"07 09 00 00",
"09 09 00 00",
"0C 09 00 00",
"0E 09 00 00",
"01 0A 00 00",
"03 0A 00 00",
"06 0A 00 00",
"09 0A 00 00",
"0B 0A 00 00",
"0E 0A 00 00",
"00 0B 00 00",
"03 0B 00 00",
"05 0B 00 00",
"08 0B 00 00",
"0B 0B 00 00",
"0D 0B 00 00",
"00 0C 00 00",
"02 0C 00 00",
"05 0C 00 00",
"07 0C 00 00",
"0A 0C 00 00",
"0C 0C 00 00",
"0F 0C 00 00",
"02 0D 00 00",
"04 0D 00 00",
"07 0D 00 00",
"09 0D 00 00",
"0C 0D 00 00",
"0E 0D 00 00",
"01 0E 00 00",
"03 0E 00 00",
"06 0E 00 00",
"09 0E 00 00",
"0B 0E 00 00",
"0E 0E 00 00",
"00 0F 00 00",
"03 0F 00 00",
"05 0F 00 00",
"08 0F 00 00",
"0B 0F 00 00",
"0D 0F 00 00",
"00 00 01 00"
}
local negativeNumbers = {
"0E 0F 0F 0F",
"0B 0F 0F 0F",
"09 0F 0F 0F",
"06 0F 0F 0F",
"04 0F 0F 0F",
"01 0F 0F 0F",
"0E 0E 0F 0F",
"0C 0E 0F 0F",
"09 0E 0F 0F",
"07 0E 0F 0F",
"04 0E 0F 0F",
"02 0E 0F 0F",
"0F 0D 0F 0F",
"0D 0D 0F 0F",
"0A 0D 0F 0F",
"07 0D 0F 0F",
"05 0D 0F 0F",
"02 0D 0F 0F",
"00 0D 0F 0F",
"0D 0C 0F 0F",
"0B 0C 0F 0F",
"08 0C 0F 0F",
"05 0C 0F 0F",
"03 0C 0F 0F",
"00 0C 0F 0F",
"0E 0B 0F 0F",
"0B 0B 0F 0F",
"09 0B 0F 0F",
"06 0B 0F 0F",
"04 0B 0F 0F",
"01 0B 0F 0F",
"0E 0A 0F 0F",
"0C 0A 0F 0F",
"09 0A 0F 0F",
"07 0A 0F 0F",
"04 0A 0F 0F",
"02 0A 0F 0F",
"0F 09 0F 0F",
"0D 09 0F 0F",
"0A 09 0F 0F",
"07 09 0F 0F",
"05 09 0F 0F",
"02 09 0F 0F",
"00 09 0F 0F",
"0D 08 0F 0F",
"0B 08 0F 0F",
"08 08 0F 0F",
"05 08 0F 0F",
"03 08 0F 0F",
"00 08 0F 0F",
"0E 07 0F 0F",
"0B 07 0F 0F",
"09 07 0F 0F",
"06 07 0F 0F",
"04 07 0F 0F",
"01 07 0F 0F",
"0E 06 0F 0F",
"0C 06 0F 0F",
"09 06 0F 0F",
"07 06 0F 0F",
"04 06 0F 0F",
"02 06 0F 0F",
"0F 05 0F 0F",
"0D 05 0F 0F",
"0A 05 0F 0F",
"07 05 0F 0F",
"05 05 0F 0F",
"02 05 0F 0F",
"00 05 0F 0F",
"0D 04 0F 0F",
"0B 04 0F 0F",
"08 04 0F 0F",
"05 04 0F 0F",
"03 04 0F 0F",
"00 04 0F 0F",
"0E 03 0F 0F",
"0B 03 0F 0F",
"09 03 0F 0F",
"06 03 0F 0F",
"04 03 0F 0F",
"01 03 0F 0F",
"0E 02 0F 0F",
"0C 02 0F 0F",
"09 02 0F 0F",
"07 02 0F 0F",
"04 02 0F 0F",
"02 02 0F 0F",
"0F 01 0F 0F",
"0D 01 0F 0F",
"0A 01 0F 0F",
"07 01 0F 0F",
"05 01 0F 0F",
"02 01 0F 0F",
"00 01 0F 0F",
"0D 00 0F 0F",
"0B 00 0F 0F",
"08 00 0F 0F",
"05 00 0F 0F",
"03 00 0F 0F",
"00 00 0F 0F"
}
function verifyFloat2Nibbles(value, expectedMemBlock)
local temp = string.upper(float2nibbles(value):toHexString(1))
assert(expectedMemBlock == temp,
string.format("Incorrect result for %.2f, expected %s, got %s", value, expectedMemBlock, temp))
end
for k,v in pairs(positiveNumbers) do
verifyFloat2Nibbles(k / 100, v)
end
for k,v in pairs(negativeNumbers) do
verifyFloat2Nibbles((k / 100) * -1, v)
end

function float2nibbles(value)
local nibbles = MemoryBlock(4, true)
local n = math.floor(math.abs(value)*256 + 0.13)
n = value < 0 and 0x10000 - n or n
for pos = 0, 3 do
nibbles:setByte(pos, n%16)
n = math.floor(n/16)
end
return nibbles
end

One problem you have is that when you negate, you are not propagating the carry from one nibble to the next. The code is way too complicated to review and fix. Here's a redo:
n2a = {[0]='0','1','2','3','4','5','6','7','8','9','a','b','c','d','e','f'}
function f2h (f)
-- TODO should range check f
local scaledf = f * 256 -- move binary point 8 bits right
-- TODO optionally you could also round scaledf here
local scaledi = math.modf(scaledf) -- integer part
-- convert to nibbles
local scalediparts = { (0xf & scaledi),
(0xf & (scaledi >> 4)),
(0xf & (scaledi >> 8)),
(0xf & (scaledi >> 12)) }
-- output in hex
print(n2a[scalediparts[1]], n2a[scalediparts[2]], n2a[scalediparts[3]], n2a[scalediparts[4]])
-- return the nibbles
return scalediparts
end
and a console test:
> f2h(-0.01)
e f f f
table: 0x7feaf9d06390
> f2h(-0.94)
0 1 f f
table: 0x7feaf9e009a0
> f2h(0.01)
2 0 0 0
table: 0x7feaf9d06410
>
This is using Lua 5.3 bit operators; conversion to the bit library is straightforward. You'll also need to modify the print statement and return value to match your preference.

Progressive JPEG layer/scan offsets (imagemagick?)

I've been searching high and low for this to no avail. I'd like to be able to extract the byte offset at which each progressive JPEG layer/scan occurs.
For example, let's say a 100 kB image has 5 layers used to render the final image; layer 1 ends at kB 5, layer 2 at kB 20, layer 3 at kB 60, etc (for example).
Is it possible to use Imagemagick for this? The identify tool does not seem to support doing so. If so, how? Otherwise what tool(s) could accomplish this? I'd rather not have to write a custom JPEG parser. Thanks.

You can make a progressive JPEG like this with ImageMagick for testing:
magick -interlace plane -size 400x200 gradient: progressive.jpg
exiftool will tell you quite a lot about it like this:
exiftool -v3 progressive.jpg
Sample Output
ExifToolVersion = 12.00
FileName = progressive.jpg
Directory = .
FileSize = 2709
FileModifyDate = 1620144585
FileAccessDate = 1620144586
FileInodeChangeDate = 1620144585
FilePermissions = 33188
FileType = JPEG
FileTypeExtension = JPG
MIMEType = image/jpeg
JPEG APP0 (14 bytes):
0006: 4a 46 49 46 00 01 01 00 00 01 00 01 00 00 [JFIF..........]
+ [BinaryData directory, 9 bytes]
| JFIFVersion = 1 1
| - Tag 0x0000 (2 bytes, int8u[2]):
| 000b: 01 01 [..]
| ResolutionUnit = 0
| - Tag 0x0002 (1 bytes, int8u[1]):
| 000d: 00 [.]
| XResolution = 1
| - Tag 0x0003 (2 bytes, int16u[1]):
| 000e: 00 01 [..]
| YResolution = 1
| - Tag 0x0005 (2 bytes, int16u[1]):
| 0010: 00 01 [..]
| ThumbnailWidth = 0
| - Tag 0x0007 (1 bytes, int8u[1]):
| 0012: 00 [.]
| ThumbnailHeight = 0
| - Tag 0x0008 (1 bytes, int8u[1]):
| 0013: 00 [.]
JPEG DQT (65 bytes):
0018: 00 03 02 02 02 02 02 03 02 02 02 03 03 03 03 04 [................]
0028: 06 04 04 04 04 04 08 06 06 05 06 09 08 0a 0a 09 [................]
0038: 08 09 09 0a 0c 0f 0c 0a 0b 0e 0b 09 09 0d 11 0d [................]
0048: 0e 0f 10 10 11 10 0a 0c 12 13 12 10 13 0f 10 10 [................]
0058: 10 [.]
JPEG SOF2 (9 bytes):
005d: 08 00 c8 01 90 01 01 11 00 [.........]
ImageWidth = 400
ImageHeight = 200
EncodingProcess = 2
BitsPerSample = 8
ColorComponents = 1
JPEG DHT (20 bytes):
006a: 00 01 01 01 00 00 00 00 00 00 00 00 00 00 00 00 [................]
007a: 00 00 04 08 [....]
JPEG SOS
JPEG DHT (19 bytes):
0139: 10 01 01 00 00 00 00 00 00 00 00 00 00 00 00 00 [................]
0149: 00 00 12 [...]
JPEG SOS
JPEG DHT (18 bytes):
0468: 10 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [................]
0478: 00 a0 [..]
JPEG SOS
JPEG DHT (20 bytes):
048a: 10 01 01 01 00 00 00 00 00 00 00 00 00 00 00 00 [................]
049a: 00 71 00 70 [.q.p]
JPEG SOS
JPEG SOS
JPEG DHT (21 bytes):
0728: 10 00 03 01 00 00 00 00 00 00 00 00 00 00 00 00 [................]
0738: 00 01 71 00 50 [..q.P]
JPEG SOS
JPEG EOI
You can also find the SOS markers like this:
xxd -c16 -g1 -u progressive.jpg | grep --color=always -A4 "FF DA"
00000070: 00 00 00 00 00 00 00 00 00 00 00 00 04 08 FF DA ................
00000080: 00 08 01 01 00 00 00 01 D4 60 00 00 00 00 00 08 .........`......
00000090: 40 00 00 00 00 00 11 00 00 00 00 00 00 21 00 00 #............!..
000000a0: 00 00 00 00 42 00 00 00 00 00 00 88 00 00 00 00 ....B...........
000000b0: 00 01 08 00 00 00 00 00 02 10 00 00 00 00 00 04 ................
--
00000140: 00 00 00 00 00 00 00 00 00 00 00 12 FF DA 00 08 ................
00000150: 01 01 00 01 05 02 B5 AD 6B 5A D6 B5 AD 6B 5A D6 ........kZ...kZ.
00000160: B5 AD 6B 5A D6 B5 AD 6B 5A D6 B5 AD 6B 5A D6 B5 ..kZ...kZ...kZ..
00000170: AD 6B 5A D6 B5 AD 6B 5A D6 B5 AD 6B 5A D6 B5 AD .kZ...kZ...kZ...
00000180: 6B 5A D6 B5 AD 6B 5A D6 B5 AD 6B 5A D6 B5 AD 6B kZ...kZ...kZ...k
It may miss some markers if they fall across a 16-byte line end - I am still thinking about making this fool-proof. Maybe run it twice with a 4-byte offset so they can't fall on the boundary in both listings.
Alternatively, I did another answer here that lets you look for arbitrary binary sequences that you could easily adapt from looking for TIFF headers to looking for FF DA.

All you'd have to do is scan the image stream and look for the SOS (Start of Scan) markers.

There is some php example code in this article by Christoph Erdmann:
https://www.smashingmagazine.com/2019/08/faster-image-loading-embedded-previews/#creating-frontend-javascript-code
$img = "progressive.jpg";
$jpgdata = file_get_contents($img);
$positions = [];
$offset = 0;
while ($pos = strpos($jpgdata, "\xFF\xC4", $offset)) {
$positions[] = $pos+2;
$offset = $pos+2;
}

Parsing AMF3 objects

I'm trying to decode an AMF0 message with AMF3 included .. All is well
except making sense of the U29 format and the way string are encoded
in AMF4
channel=3 size=134 pkttype=0x11 time=1973
00000: 00 02 00 14 73 65 6E 64 55 6E 69 76 65 72 73 61 6C 4D 65 73 ....sendUniversalMes
00020: 73 61 67 65 00 00 00 00 00 00 00 00 00 05 02 00 11 6D 62 5F sage.............mb_
00040: 31 32 32 31 30 5F 37 35 39 32 33 33 38 30 05 00 40 58 C0 00 12210_75923380..#X..
00060: 00 00 00 00 11 0A 0B 01 09 68 62 69 64 04 81 CF 5E 09 73 72 .........hbid...^.sr
00080: 63 65 06 49 63 37 62 39 32 33 65 65 2D 30 61 30 38 2D 34 62 ce.Ic7b923ee-0a08-4b
00100: 61 32 2D 38 65 37 63 2D 63 38 32 61 39 33 64 37 37 31 34 32 a2-8e7c-c82a93d77142
00120: 09 68 62 64 6C 04 00 09 74 65 78 74 01 01 .hbdl...text..
first byte I skip
02 = string
00 14 = length of string ( 20 characters , sendUniversalMessage )
00 00 00 00 00 00 00 00 00 = number = 0
05 = null
02 = string
00 11 = length of string ( 17 characters , mb_12210_75923380 )
05 = null
00 40 58 C0 00 00 00 00 00 = number = 99
11 = AMV+
here is where I have problems
0A = AMF3 object
now I need to do a readU29 which starts with
0B = what does this mean
01 = what does this mean
09 = what does this mean
where is the length of the string 'hbid' ?

in U29, if the first byte is worth less than 128, it fits on the first byte. So you have to read this as 3 different u29s, worth 0B, 01, 09.
0B: details about the type of object. It seems you're not too interested in that byte and it's complicated, so pass.
01: The LSB is the bit that says that this isn't a string reference. Then 01 >> 1 = 0 is the length of the class name. This means empty string, which means this is an anonymous (untyped) object.
09: = 00001001. The LSB is also the bit that says this isn't a string reference. Then 1001 >> 1 = 0100 = 4 is the string length.
hope that helps

How to get raw jpeg data (but no metatags / proprietary markers)

I want to get raw jpeg data - no metadata.
I'm very confused looking at the jpeg "standards".
How correct is my understanding of the marker "tree"?
0xFFD8 - Identifies the file as an image
0xFFE? - EXIF, JFIF, SPIFF, ICC, etc
0x???? - the length of the tag
0xFFD8 - Start of Image
0xFFE0 - should follow SOI as per spec, but often preceded by comments ???
0x???? - Matrices, tags, random data ???
There are never other markers in-between these markers?
Or these include the matrices and such?
0xFFDA - Start of Stream - This is what I want, yes?
0xXXXX - length of stream
0xFFD9 - End of Stream (EOI)
0x???? - Comment tags, extra exif, jfif info???
0xFFD9 - End of Image
0xFF00 - escaped 0xFF, not to be confused with a marker
This has been my reading material:
http://en.wikipedia.org/wiki/JPEG
https://ExifTool.org/TagNames/JPEG.html
http://www.media.mit.edu/pia/Research/deepview/exif.html
http://www.faqs.org/faqs/jpeg-faq/part1/section-15.html

Besides the SOS (0xFFDA), you may also want the following:
0xFFDB DQT Quantization Table
0xFFC4 DHT Define Huffman Table

There's a fairly reasonable "Anatomy of a JPEG" here which shows you how to use grep to extract JPEG markers and sections from JPEG files, e.g. for SOF marker:
xxd -c16 -g1 -u testimg.jpg | grep --color=always -C2 "FF C0"
0000090: 32 32 32 32 32 32 32 32 32 32 32 32 32 32 FF C0 22222222222222..
00000a0: 00 11 08 00 95 00 E3 03 01 22 00 02 11 01 03 11 ........."......
00000b0: 01 FF C4 00 1F 00 00 01 05 01 01 01 01 01 01 00 ................
Explanation
0xFF, 0xC0, // SOF0 segement
0x00, 0x11, // length of segment depends on the number of components
0x08, // bits per pixel
0x00, 0x95, // image height
0x00, 0xE3, // image width
0x03, // number of components (should be 1 or 3)
0x01, 0x22, 0x00, // 0x01=Y component, 0x22=sampling factor, quantization table number
0x02, 0x11, 0x01, // 0x02=Cb component, ...
0x03, 0x11, 0x01 // 0x03=Cr component, ...
And there's a complete, highly detailed and thorough series of videos here on YouTube that tell you more than you could possibly ever want to know about JPEG.
Note also that you can extract and examine the JPEG markers in a file with exiftool, like this:
exiftool -v -v -v image.jpg
Sample Output
ExifToolVersion = 12.00
FileName = iphone.jpg
Directory = .
FileSize = 2219100
FileModifyDate = 1596126733
FileAccessDate = 1610564128
FileInodeChangeDate = 1596126733
FilePermissions = 33188
FileType = JPEG
FileTypeExtension = JPG
MIMEType = image/jpeg
JPEG APP1 (12284 bytes):
0006: 45 78 69 66 00 00 4d 4d 00 2a 00 00 00 08 00 0b [Exif..MM.*......]
0016: 01 0f 00 02 00 00 00 06 00 00 00 92 01 10 00 02 [................]
0026: 00 00 00 09 00 00 00 98 01 12 00 03 00 00 00 01 [................]
0036: 00 01 00 00 01 1a 00 05 00 00 00 01 00 00 00 a2 [................]
0046: 01 1b 00 05 00 00 00 01 00 00 00 aa 01 28 00 03 [.............(..]
0056: 00 00 00 01 00 02 00 00 01 31 00 02 00 00 00 06 [.........1......]
0066: 00 00 00 b2 01 32 00 02 00 00 00 14 00 00 00 b8 [.....2..........]
[snip 12172 bytes]
ExifByteOrder = MM
+ [IFD0 directory with 11 entries]
| 0) Make = Apple
| - Tag 0x010f (6 bytes, string[6]):
| 009e: 41 70 70 6c 65 00 [Apple.]
| 1) Model = iPhone 4
| - Tag 0x0110 (9 bytes, string[9]):
| 00a4: 69 50 68 6f 6e 65 20 34 00 [iPhone 4.]
| 2) Orientation = 1
| - Tag 0x0112 (2 bytes, int16u[1]):
| 0036: 00 01 [..]
| 3) XResolution = 72 (72/1)
| - Tag 0x011a (8 bytes, rational64u[1]):
| 00ae: 00 00 00 48 00 00 00 01 [...H....]
| 4) YResolution = 72 (72/1)
| - Tag 0x011b (8 bytes, rational64u[1]):
| 00b6: 00 00 00 48 00 00 00 01 [...H....]
| 5) ResolutionUnit = 2
| - Tag 0x0128 (2 bytes, int16u[1]):
| 005a: 00 02 [..]
| 6) Software = 6.1.2
| - Tag 0x0131 (6 bytes, string[6]):
| 00be: 36 2e 31 2e 32 00 [6.1.2.]
| 7) ModifyDate = 2013:03:09 08:59:50
| - Tag 0x0132 (20 bytes, string[20]):
| 00c4: 32 30 31 33 3a 30 33 3a 30 39 20 30 38 3a 35 39 [2013:03:09 08:59]
| 00d4: 3a 35 30 00 [:50.]
| 8) YCbCrPositioning = 1
| - Tag 0x0213 (2 bytes, int16u[1]):
| 007e: 00 01 [..]
| 9) ExifOffset (SubDirectory) -->
| - Tag 0x8769 (4 bytes, int32u[1]):
| 008a: 00 00 00 cc [....]
| + [ExifIFD directory with 24 entries]
| | 0) ExposureTime = 0.001094091904 (1/914)
| | - Tag 0x829a (8 bytes, rational64u[1]):
| | 01fe: 00 00 00 01 00 00 03 92 [........]
| | 1) FNumber = 2.8 (14/5)
| | - Tag 0x829d (8 bytes, rational64u[1]):
| | 0206: 00 00 00 0e 00 00 00 05 [........]
| | 2) ExposureProgram = 2
| | - Tag 0x8822 (2 bytes, int16u[1]):
| | 00fa: 00 02 [..]
| | 3) ISO = 80
| | - Tag 0x8827 (2 bytes, int16u[1]):
| | 0106: 00 50 [.P]
| | 4) ExifVersion = 0221
| | - Tag 0x9000 (4 bytes, undef[4]):
| | 0112: 30 32 32 31 [0221]
| | 5) DateTimeOriginal = 2013:03:09 08:59:50
| | - Tag 0x9003 (20 bytes, string[20]):
| | 020e: 32 30 31 33 3a 30 33 3a 30 39 20 30 38 3a 35 39 [2013:03:09 08:59]
| | 021e: 3a 35 30 00 [:50.]
| | 6) CreateDate = 2013:03:09 08:59:50
| | - Tag 0x9004 (20 bytes, string[20]):
| | 0222: 32 30 31 33 3a 30 33 3a 30 39 20 30 38 3a 35 39 [2013:03:09 08:59]
| | 0232: 3a 35 30 00 [:50.]
| | 7) ComponentsConfiguration = 1 2 3 0
| | - Tag 0x9101 (4 bytes, undef[4] read as int8u[4]):
| | 0136: 01 02 03 00 [....]
| | 8) ShutterSpeedValue = 9.83619211 (11469/1166)
| | - Tag 0x9201 (8 bytes, rational64s[1]):
| | 0236: 00 00 2c cd 00 00 04 8e [..,.....]
| | 9) ApertureValue = 2.970853574 (4281/1441)
| | - Tag 0x9202 (8 bytes, rational64u[1]):
| | 023e: 00 00 10 b9 00 00 05 a1 [........]
| | 10) BrightnessValue = 8.880310458 (21739/2448)
| | - Tag 0x9203 (8 bytes, rational64s[1]):
| | 0246: 00 00 54 eb 00 00 09 90 [..T.....]
| | 11) MeteringMode = 5
| | - Tag 0x9207 (2 bytes, int16u[1]):
| | 0166: 00 05 [..]
| | 12) Flash = 0
| | - Tag 0x9209 (2 bytes, int16u[1]):
| | 0172: 00 00 [..]
| | 13) FocalLength = 3.85 (77/20)
| | - Tag 0x920a (8 bytes, rational64u[1]):
| | 024e: 00 00 00 4d 00 00 00 14 [...M....]
| | 14) FlashpixVersion = 0100
| | - Tag 0xa000 (4 bytes, undef[4]):
| | 018a: 30 31 30 30 [0100]
| | 15) ColorSpace = 1
| | - Tag 0xa001 (2 bytes, int16u[1]):
| | 0196: 00 01 [..]
| | 16) ExifImageWidth = 2592
| | - Tag 0xa002 (4 bytes, int32u[1]):
| | 01a2: 00 00 0a 20 [... ]
| | 17) ExifImageHeight = 1936
| | - Tag 0xa003 (4 bytes, int32u[1]):
| | 01ae: 00 00 07 90 [....]
| | 18) SensingMethod = 2
| | - Tag 0xa217 (2 bytes, int16u[1]):
| | 01ba: 00 02 [..]
| | 19) CustomRendered = 2
| | - Tag 0xa401 (2 bytes, int16u[1]):
| | 01c6: 00 02 [..]
| | 20) ExposureMode = 0
| | - Tag 0xa402 (2 bytes, int16u[1]):
| | 01d2: 00 00 [..]
| | 21) WhiteBalance = 0
| | - Tag 0xa403 (2 bytes, int16u[1]):
| | 01de: 00 00 [..]
| | 22) FocalLengthIn35mmFormat = 35
| | - Tag 0xa405 (2 bytes, int16u[1]):
| | 01ea: 00 23 [.#]
| | 23) SceneCaptureType = 0
| | - Tag 0xa406 (2 bytes, int16u[1]):
| | 01f6: 00 00 [..]
| 10) GPSInfo (SubDirectory) -->
| - Tag 0x8825 (4 bytes, int32u[1]):
| 0096: 00 00 02 4a [...J]
| + [GPS directory with 9 entries]
| | 0) GPSLatitudeRef = N
| | - Tag 0x0001 (2 bytes, string[2]):
| | 0260: 4e 00 [N.]
| | 1) GPSLatitude = 20 50.66 0 (20/1 5066/100 0/1)
| | - Tag 0x0002 (24 bytes, rational64u[3]):
| | 02c8: 00 00 00 14 00 00 00 01 00 00 13 ca 00 00 00 64 [...............d]
| | 02d8: 00 00 00 00 00 00 00 01 [........]
| | 2) GPSLongitudeRef = E
| | - Tag 0x0003 (2 bytes, string[2]):
| | 0278: 45 00 [E.]
| | 3) GPSLongitude = 107 5.46 0 (107/1 546/100 0/1)
| | - Tag 0x0004 (24 bytes, rational64u[3]):
| | 02e0: 00 00 00 6b 00 00 00 01 00 00 02 22 00 00 00 64 [...k......."...d]
| | 02f0: 00 00 00 00 00 00 00 01 [........]
| | 4) GPSAltitudeRef = 0
| | - Tag 0x0005 (1 bytes, int8u[1]):
| | 0290: 00 [.]
| | 5) GPSAltitude = 1.12868102 (4561/4041)
| | - Tag 0x0006 (8 bytes, rational64u[1]):
| | 02f8: 00 00 11 d1 00 00 0f c9 [........]
| | 6) GPSTimeStamp = 1 59 48.81 (1/1 59/1 4881/100)
| | - Tag 0x0007 (24 bytes, rational64u[3]):
| | 0300: 00 00 00 01 00 00 00 01 00 00 00 3b 00 00 00 01 [...........;....]
| | 0310: 00 00 13 11 00 00 00 64 [.......d]
| | 7) GPSImgDirectionRef = T
| | - Tag 0x0010 (2 bytes, string[2]):
| | 02b4: 54 00 [T.]
| | 8) GPSImgDirection = 324.4435484 (40231/124)
| | - Tag 0x0011 (8 bytes, rational64u[1]):
| | 0318: 00 00 9d 27 00 00 00 7c [...'...|]
+ [IFD1 directory with 6 entries]
| 0) Compression = 6
| - Tag 0x0103 (2 bytes, int16u[1]):
| 032a: 00 06 [..]
| 1) XResolution = 72 (72/1)
| - Tag 0x011a (8 bytes, rational64u[1]):
| 036e: 00 00 00 48 00 00 00 01 [...H....]
| 2) YResolution = 72 (72/1)
| - Tag 0x011b (8 bytes, rational64u[1]):
| 0376: 00 00 00 48 00 00 00 01 [...H....]
| 3) ResolutionUnit = 2
| - Tag 0x0128 (2 bytes, int16u[1]):
| 034e: 00 02 [..]
| 4) ThumbnailOffset = 882
| - Tag 0x0201 (4 bytes, int32u[1]):
| 035a: 00 00 03 72 [...r]
| 5) ThumbnailLength = 7847
| - Tag 0x0202 (4 bytes, int32u[1]):
| 0366: 00 00 1e a7 [....]
JPEG DQT (130 bytes):
3006: 00 01 01 01 01 01 01 01 01 01 01 01 01 01 01 02 [................]
3016: 04 02 02 02 02 02 04 03 03 02 04 05 05 06 06 05 [................]
3026: 05 05 05 06 07 09 07 06 06 08 06 05 05 08 0a 08 [................]
3036: 08 09 09 0a 0a 0a 06 07 0b 0c 0b 0a 0c 09 0a 0a [................]
3046: 09 01 01 01 01 02 02 02 04 02 02 04 09 06 05 06 [................]
3056: 09 09 09 09 09 09 09 09 09 09 09 09 09 09 09 09 [................]
3066: 09 09 09 09 09 09 09 09 09 09 09 09 09 09 09 09 [................]
[snip 18 bytes]
JPEG SOF0 (15 bytes):
308c: 08 07 90 0a 20 03 01 22 00 02 11 01 03 11 01 [.... ..".......]
ImageWidth = 2592
ImageHeight = 1936
EncodingProcess = 0
BitsPerSample = 8
ColorComponents = 3
YCbCrSubSampling = 2 2
JPEG DHT (416 bytes):
309f: 00 00 01 05 01 01 01 01 01 01 00 00 00 00 00 00 [................]
30af: 00 00 01 02 03 04 05 06 07 08 09 0a 0b 10 00 02 [................]
30bf: 01 03 03 02 04 03 05 05 04 04 00 00 01 7d 01 02 [.............}..]
30cf: 03 00 04 11 05 12 21 31 41 06 13 51 61 07 22 71 [......!1A..Qa."q]
30df: 14 32 81 91 a1 08 23 42 b1 c1 15 52 d1 f0 24 33 [.2....#B...R..$3]
30ef: 62 72 82 09 0a 16 17 18 19 1a 25 26 27 28 29 2a [br........%&'()*]
30ff: 34 35 36 37 38 39 3a 43 44 45 46 47 48 49 4a 53 [456789:CDEFGHIJS]
[snip 304 bytes]
JPEG SOS
JPEG EOI
Keywords: JPEG, marker, anatomy, SOF, DQT, DHT, prime.

Develop Reference

ios ruby-on-rails asp.net-mvc docker delphi jenkins grails google-sheets machine-learning dart

Indy odd/strang behavior with TLS version - delphi

Related

Transferring photos from a Nikon Z50 with Airnef/PTPIP program is slowing down until disconnection

Convert floating point values to hexadecimal

Progressive JPEG layer/scan offsets (imagemagick?)

Parsing AMF3 objects

How to get raw jpeg data (but no metatags / proprietary markers)

Categories

Resources