I'm trying to decode an AMF0 message with AMF3 included .. All is well
except making sense of the U29 format and the way string are encoded
in AMF4
channel=3 size=134 pkttype=0x11 time=1973
00000: 00 02 00 14 73 65 6E 64 55 6E 69 76 65 72 73 61 6C 4D 65 73 ....sendUniversalMes
00020: 73 61 67 65 00 00 00 00 00 00 00 00 00 05 02 00 11 6D 62 5F sage.............mb_
00040: 31 32 32 31 30 5F 37 35 39 32 33 33 38 30 05 00 40 58 C0 00 12210_75923380..#X..
00060: 00 00 00 00 11 0A 0B 01 09 68 62 69 64 04 81 CF 5E 09 73 72 .........hbid...^.sr
00080: 63 65 06 49 63 37 62 39 32 33 65 65 2D 30 61 30 38 2D 34 62 ce.Ic7b923ee-0a08-4b
00100: 61 32 2D 38 65 37 63 2D 63 38 32 61 39 33 64 37 37 31 34 32 a2-8e7c-c82a93d77142
00120: 09 68 62 64 6C 04 00 09 74 65 78 74 01 01 .hbdl...text..
first byte I skip
02 = string
00 14 = length of string ( 20 characters , sendUniversalMessage )
00 00 00 00 00 00 00 00 00 = number = 0
05 = null
02 = string
00 11 = length of string ( 17 characters , mb_12210_75923380 )
05 = null
00 40 58 C0 00 00 00 00 00 = number = 99
11 = AMV+
here is where I have problems
0A = AMF3 object
now I need to do a readU29 which starts with
0B = what does this mean
01 = what does this mean
09 = what does this mean
where is the length of the string 'hbid' ?
in U29, if the first byte is worth less than 128, it fits on the first byte. So you have to read this as 3 different u29s, worth 0B, 01, 09.
0B: details about the type of object. It seems you're not too interested in that byte and it's complicated, so pass.
01: The LSB is the bit that says that this isn't a string reference. Then 01 >> 1 = 0 is the length of the class name. This means empty string, which means this is an anonymous (untyped) object.
09: = 00001001. The LSB is also the bit that says this isn't a string reference. Then 1001 >> 1 = 0100 = 4 is the string length.
hope that helps
Related
I am using Airnef (and the PTPIP protocol) to download pictures from my Nikon Z50 camera through Wi-Fi.
It works perfectly in some cases, but in other cases the transfer is slowing down until the camera has been disconnected.
I have read the associated documentation (available here https://www.testcams.com/airnef/) and seen that a similar issue has been encountered while the program has been developed, which seemed to be caused by a bug in the Nikon firmware which lead to some memory issues and a possible disconnection. The same issue seems to have been found during gphoto2 development (https://sourceforge.net/p/gphoto/mailman/gphoto-devel/thread/20190107075304.GC4614%40jet.franken.de/)
From what I understand, the parameters "maxgetobjbuffersizekb" and "maxgetobjtransferzekb" have been introduced to fix a such issue by trying to limit the requested sizes on the camera to transfer image, however it does not seems to work in my case.
I have tried to customize the values used by these parameters (for instance, by using a value of 256kb for maxgetobjtransferzekb setting), but with no real luck (the connection seems to abort less often but the transfer is still slowing down).
Here is an extract of the trace I get when I increase the log level in airnef.
Download history file "C:\Users\aerkis\AppData\Local\airnef\appdata\Z 50-SN6026496-downloadhist" loaded - 2 entries
Skipping DCIM - object is not file - MTP_OBJFORMAT_Assocation (0x3001)
Skipping 101NZ_50 - object is not file - MTP_OBJFORMAT_Assocation (0x3001)
>> MTP_OP_GetObject
Downloading "DSC_0490.JPG": 0%DSC_0490.JPG - downloading next piece, offset=0x0, count=0x100000
execMtpOp: MTP_OP_GetPartialObject - CmdReq payload:
0000: 06 00 00 00 01 00 00 00 - 1b 10 0b 00 00 00 ea 41 ........ - .......A
0010: 19 29 00 00 00 00 00 00 - 10 00 .)...... - ..
execMtpOp: MTP_OP_GetPartialObject - DataStart payload [expected data bytes is 0x100000]
0000: 09 00 00 00 0b 00 00 00 - 00 00 10 00 00 00 00 00 ........ - ........
0%execMtpOp: MTP_OP_GetPartialObject - Data payload [ID a] (0x00005b3c bytes):
0000: 0a 00 00 00 0b 00 00 00 - ff d8 ff e1 ff fe 45 78 ........ - ......Ex
0010: 69 66 00 00 49 49 2a 00 - 08 00 00 00 0d 00 0f 01 if..II*. - ........
0020: 02 00 12 00 00 00 ac 00 - 00 00 10 01 02 00 0b 00 ........ - ........
0030: 00 00 c0 00 00 00 12 01 - 03 00 01 00 00 00 01 00 ........ - ........
1%execMtpOp: MTP_OP_GetPartialObject - Data payload [ID a] (0x00005b3c bytes):
0000: 0a 00 00 00 0b 00 00 00 - 4f 00 72 a5 99 4e c4 fb ........ - O.r..N..
0010: f3 a3 26 61 66 1d 98 d7 - d7 98 1a 5d cc 26 ac f3 ..&af... - ...].&..
0020: f3 c4 4e 99 a5 72 00 4f - 5e 30 c2 15 12 f9 94 eb ..N..r.O - ^0......
0030: 03 dc 76 d1 ed ca 68 c7 - e7 c8 6a cd f1 d6 7c e3 ..v...h. - ..j...|.
execMtpOp: MTP_OP_GetPartialObject - Data payload [ID a] (0x00005b3c bytes):
0000: 0a 00 00 00 0b 00 00 00 - 00 00 00 00 00 00 00 00 ........ - ........
0010: 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00 ........ - ........
0020: 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00 ........ - ........
0030: 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00 ........ - ........
2%execMtpOp: MTP_OP_GetPartialObject - Partial payload after error (0x00004329 bytes):
0000: 0a 00 00 00 0b 00 00 00 - 20 20 20 20 20 20 20 20 ........ -
0010: 20 20 20 20 20 20 20 20 - 20 20 20 20 20 20 20 20 -
0020: 20 20 20 20 20 20 20 20 - 20 20 20 20 20 20 20 20 -
0030: 20 20 20 20 20 20 20 20 - 20 20 20 20 20 20 20 20 -
0040: 20 20 20 20 20 20 20 20 - 20 20 20 20 20 20 20 20 -
0050: 20 20 20 20 20 20 20 20 - 20 20 20 20 20 20 20 20 -
0060: 20 20 20 20 20 20 20 20 - 20 20 20 20 20 20 20 20 -
0070: 20 20 20 20 20 20 20 20 - 20 20 20 20 20 20 20 20 -
0080: 20 20 20 20 20 20 20 20 - 20 20 20 20 20 20 20 20 -
0090: 20 20 20 20 20 20 20 20 - 20 20 20 20 20 20 20 20 -
DSC_0490.JPG - error during download, writing 0x0 bytes of buffered data
DSC_0490.JPG - writing partial payload data of 0x154bd bytes
C:\Users\aerkis\Pictures\DSC_0490.JPG writing 0x154bd bytes, closeAfterWriting=0
MTP_OP_GetPartialObject: Socket error, partial data received - 0x4321 of 0x5b34 bytes for specific payload, 0x154bd of 0x100000 of total data bytes expected. Error: timed out
Therefore, I request some help to know if it exists a way to better understand this problem and possibly correct it (maybe by configuring some parameters ?).
Any help would be greatly appreciated !
I understand the shape of the bytes behind a PACKED-DECIMAL number when on a Big-Endian machine. What I have not been able to discover is what shape would those bytes take on a Little-Endian machine.
Note: The reason I believe there does exist a separate format for such is because the IBM MQ Encoding field has the following values:-
MQENC_DECIMAL_NORMAL
MQENC_DECIMAL_REVERSED
Of MQENC_DECIMAL_REVERSED it simply says:-
Packed-decimal integers are represented in the same way as MQENC_DECIMAL_NORMAL, but with the bytes arranged in reverse order. The bits within each byte are arranged in the same way as MQENC_DECIMAL_NORMAL.
Can anyone corroborate or refute this description?
For example, the number +123.45 stored in a PIC 9(3)V99 COMP-3 would have the following bytes:-
12 34 5C
The above quote suggests to me that if in reversed, the same packed decimal number would be represented on a Little-Endian machine as:-
5C 34 12
N.B. I have not tagged this question with ibm-mq, as I don't really think the crux of my question has anything to do with IBM MQ. It is simply the reason I am asking.
It looks like packed data is stored in "normal" byte order.
The following program was compiled with OpenCOBOL 1.1.0 on Ubuntu 16.04 on a Lenovo ThinkPad...
ID Division.
Program-ID. testcmp3.
Environment Division.
Input-Output Section.
File-Control.
Select OUTPUT01 Assign To './output01'
Status OUTPUT01-STATUS.
Data Division.
File Section.
FD OUTPUT01
Record 16
Block 0
Recording F.
01 OUTPUT01-REC PIC X(016).
Working-Storage Section.
01 CONSTANTS.
05 MYNAME PIC X(008) VALUE 'testcmp3'.
01 WS-OUTPUT01-REC.
05 WS-OUT-S94COMP PIC S9(004) COMP VALUE +0.
05 FILLER PIC X(001) VALUE HIGH-VALUES.
05 WS-OUT-S95COMP3 PIC S9(005) COMP-3 VALUE +0.
05 FILLER PIC X(008) VALUE SPACES.
01 WORK-AREAS.
05 OUTPUT01-STATUS PIC X(002) VALUE ZEROES.
Procedure Division.
OPEN OUTPUT OUTPUT01
IF OUTPUT01-STATUS NOT = '00'
DISPLAY
MYNAME
' open of OUTPUT01 status = '
OUTPUT01-STATUS
END-IF
PERFORM 100-WRITE 100 TIMES
CLOSE OUTPUT01
STOP RUN.
100-WRITE.
WRITE OUTPUT01-REC FROM WS-OUTPUT01-REC
ADD 1 TO WS-OUT-S94COMP
ADD 1 TO WS-OUT-S95COMP3
.
...with the option binary-byteorder set to native.
The resulting file, displayed with the hexdump program, follows.
00000000 00 00 ff 00 00 0c 20 20 20 20 20 20 20 20 20 20 |...... |
00000010 01 00 ff 00 00 1c 20 20 20 20 20 20 20 20 20 20 |...... |
00000020 02 00 ff 00 00 2c 20 20 20 20 20 20 20 20 20 20 |....., |
00000030 03 00 ff 00 00 3c 20 20 20 20 20 20 20 20 20 20 |.....< |
00000040 04 00 ff 00 00 4c 20 20 20 20 20 20 20 20 20 20 |.....L |
00000050 05 00 ff 00 00 5c 20 20 20 20 20 20 20 20 20 20 |.....\ |
00000060 06 00 ff 00 00 6c 20 20 20 20 20 20 20 20 20 20 |.....l |
00000070 07 00 ff 00 00 7c 20 20 20 20 20 20 20 20 20 20 |.....| |
00000080 08 00 ff 00 00 8c 20 20 20 20 20 20 20 20 20 20 |...... |
00000090 09 00 ff 00 00 9c 20 20 20 20 20 20 20 20 20 20 |...... |
000000a0 0a 00 ff 00 01 0c 20 20 20 20 20 20 20 20 20 20 |...... |
000000b0 0b 00 ff 00 01 1c 20 20 20 20 20 20 20 20 20 20 |...... |
000000c0 0c 00 ff 00 01 2c 20 20 20 20 20 20 20 20 20 20 |....., |
000000d0 0d 00 ff 00 01 3c 20 20 20 20 20 20 20 20 20 20 |.....< |
000000e0 0e 00 ff 00 01 4c 20 20 20 20 20 20 20 20 20 20 |.....L |
000000f0 0f 00 ff 00 01 5c 20 20 20 20 20 20 20 20 20 20 |.....\ |
00000100 10 00 ff 00 01 6c 20 20 20 20 20 20 20 20 20 20 |.....l |
00000110 11 00 ff 00 01 7c 20 20 20 20 20 20 20 20 20 20 |.....| |
00000120 12 00 ff 00 01 8c 20 20 20 20 20 20 20 20 20 20 |...... |
00000130 13 00 ff 00 01 9c 20 20 20 20 20 20 20 20 20 20 |...... |
00000140 14 00 ff 00 02 0c 20 20 20 20 20 20 20 20 20 20 |...... |
00000150 15 00 ff 00 02 1c 20 20 20 20 20 20 20 20 20 20 |...... |
00000160 16 00 ff 00 02 2c 20 20 20 20 20 20 20 20 20 20 |....., |
00000170 17 00 ff 00 02 3c 20 20 20 20 20 20 20 20 20 20 |.....< |
00000180 18 00 ff 00 02 4c 20 20 20 20 20 20 20 20 20 20 |.....L |
00000190 19 00 ff 00 02 5c 20 20 20 20 20 20 20 20 20 20 |.....\ |
000001a0 1a 00 ff 00 02 6c 20 20 20 20 20 20 20 20 20 20 |.....l |
000001b0 1b 00 ff 00 02 7c 20 20 20 20 20 20 20 20 20 20 |.....| |
000001c0 1c 00 ff 00 02 8c 20 20 20 20 20 20 20 20 20 20 |...... |
000001d0 1d 00 ff 00 02 9c 20 20 20 20 20 20 20 20 20 20 |...... |
000001e0 1e 00 ff 00 03 0c 20 20 20 20 20 20 20 20 20 20 |...... |
000001f0 1f 00 ff 00 03 1c 20 20 20 20 20 20 20 20 20 20 |...... |
00000200 20 00 ff 00 03 2c 20 20 20 20 20 20 20 20 20 20 | ...., |
00000210 21 00 ff 00 03 3c 20 20 20 20 20 20 20 20 20 20 |!....< |
00000220 22 00 ff 00 03 4c 20 20 20 20 20 20 20 20 20 20 |"....L |
00000230 23 00 ff 00 03 5c 20 20 20 20 20 20 20 20 20 20 |#....\ |
00000240 24 00 ff 00 03 6c 20 20 20 20 20 20 20 20 20 20 |$....l |
00000250 25 00 ff 00 03 7c 20 20 20 20 20 20 20 20 20 20 |%....| |
00000260 26 00 ff 00 03 8c 20 20 20 20 20 20 20 20 20 20 |&..... |
00000270 27 00 ff 00 03 9c 20 20 20 20 20 20 20 20 20 20 |'..... |
00000280 28 00 ff 00 04 0c 20 20 20 20 20 20 20 20 20 20 |(..... |
00000290 29 00 ff 00 04 1c 20 20 20 20 20 20 20 20 20 20 |)..... |
000002a0 2a 00 ff 00 04 2c 20 20 20 20 20 20 20 20 20 20 |*...., |
000002b0 2b 00 ff 00 04 3c 20 20 20 20 20 20 20 20 20 20 |+....< |
000002c0 2c 00 ff 00 04 4c 20 20 20 20 20 20 20 20 20 20 |,....L |
000002d0 2d 00 ff 00 04 5c 20 20 20 20 20 20 20 20 20 20 |-....\ |
000002e0 2e 00 ff 00 04 6c 20 20 20 20 20 20 20 20 20 20 |.....l |
000002f0 2f 00 ff 00 04 7c 20 20 20 20 20 20 20 20 20 20 |/....| |
00000300 30 00 ff 00 04 8c 20 20 20 20 20 20 20 20 20 20 |0..... |
00000310 31 00 ff 00 04 9c 20 20 20 20 20 20 20 20 20 20 |1..... |
00000320 32 00 ff 00 05 0c 20 20 20 20 20 20 20 20 20 20 |2..... |
00000330 33 00 ff 00 05 1c 20 20 20 20 20 20 20 20 20 20 |3..... |
00000340 34 00 ff 00 05 2c 20 20 20 20 20 20 20 20 20 20 |4...., |
00000350 35 00 ff 00 05 3c 20 20 20 20 20 20 20 20 20 20 |5....< |
00000360 36 00 ff 00 05 4c 20 20 20 20 20 20 20 20 20 20 |6....L |
00000370 37 00 ff 00 05 5c 20 20 20 20 20 20 20 20 20 20 |7....\ |
00000380 38 00 ff 00 05 6c 20 20 20 20 20 20 20 20 20 20 |8....l |
00000390 39 00 ff 00 05 7c 20 20 20 20 20 20 20 20 20 20 |9....| |
000003a0 3a 00 ff 00 05 8c 20 20 20 20 20 20 20 20 20 20 |:..... |
000003b0 3b 00 ff 00 05 9c 20 20 20 20 20 20 20 20 20 20 |;..... |
000003c0 3c 00 ff 00 06 0c 20 20 20 20 20 20 20 20 20 20 |<..... |
000003d0 3d 00 ff 00 06 1c 20 20 20 20 20 20 20 20 20 20 |=..... |
000003e0 3e 00 ff 00 06 2c 20 20 20 20 20 20 20 20 20 20 |>...., |
000003f0 3f 00 ff 00 06 3c 20 20 20 20 20 20 20 20 20 20 |?....< |
00000400 40 00 ff 00 06 4c 20 20 20 20 20 20 20 20 20 20 |#....L |
00000410 41 00 ff 00 06 5c 20 20 20 20 20 20 20 20 20 20 |A....\ |
00000420 42 00 ff 00 06 6c 20 20 20 20 20 20 20 20 20 20 |B....l |
00000430 43 00 ff 00 06 7c 20 20 20 20 20 20 20 20 20 20 |C....| |
00000440 44 00 ff 00 06 8c 20 20 20 20 20 20 20 20 20 20 |D..... |
00000450 45 00 ff 00 06 9c 20 20 20 20 20 20 20 20 20 20 |E..... |
00000460 46 00 ff 00 07 0c 20 20 20 20 20 20 20 20 20 20 |F..... |
00000470 47 00 ff 00 07 1c 20 20 20 20 20 20 20 20 20 20 |G..... |
00000480 48 00 ff 00 07 2c 20 20 20 20 20 20 20 20 20 20 |H...., |
00000490 49 00 ff 00 07 3c 20 20 20 20 20 20 20 20 20 20 |I....< |
000004a0 4a 00 ff 00 07 4c 20 20 20 20 20 20 20 20 20 20 |J....L |
000004b0 4b 00 ff 00 07 5c 20 20 20 20 20 20 20 20 20 20 |K....\ |
000004c0 4c 00 ff 00 07 6c 20 20 20 20 20 20 20 20 20 20 |L....l |
000004d0 4d 00 ff 00 07 7c 20 20 20 20 20 20 20 20 20 20 |M....| |
000004e0 4e 00 ff 00 07 8c 20 20 20 20 20 20 20 20 20 20 |N..... |
000004f0 4f 00 ff 00 07 9c 20 20 20 20 20 20 20 20 20 20 |O..... |
00000500 50 00 ff 00 08 0c 20 20 20 20 20 20 20 20 20 20 |P..... |
00000510 51 00 ff 00 08 1c 20 20 20 20 20 20 20 20 20 20 |Q..... |
00000520 52 00 ff 00 08 2c 20 20 20 20 20 20 20 20 20 20 |R...., |
00000530 53 00 ff 00 08 3c 20 20 20 20 20 20 20 20 20 20 |S....< |
00000540 54 00 ff 00 08 4c 20 20 20 20 20 20 20 20 20 20 |T....L |
00000550 55 00 ff 00 08 5c 20 20 20 20 20 20 20 20 20 20 |U....\ |
00000560 56 00 ff 00 08 6c 20 20 20 20 20 20 20 20 20 20 |V....l |
00000570 57 00 ff 00 08 7c 20 20 20 20 20 20 20 20 20 20 |W....| |
00000580 58 00 ff 00 08 8c 20 20 20 20 20 20 20 20 20 20 |X..... |
00000590 59 00 ff 00 08 9c 20 20 20 20 20 20 20 20 20 20 |Y..... |
000005a0 5a 00 ff 00 09 0c 20 20 20 20 20 20 20 20 20 20 |Z..... |
000005b0 5b 00 ff 00 09 1c 20 20 20 20 20 20 20 20 20 20 |[..... |
000005c0 5c 00 ff 00 09 2c 20 20 20 20 20 20 20 20 20 20 |\...., |
000005d0 5d 00 ff 00 09 3c 20 20 20 20 20 20 20 20 20 20 |]....< |
000005e0 5e 00 ff 00 09 4c 20 20 20 20 20 20 20 20 20 20 |^....L |
000005f0 5f 00 ff 00 09 5c 20 20 20 20 20 20 20 20 20 20 |_....\ |
00000600 60 00 ff 00 09 6c 20 20 20 20 20 20 20 20 20 20 |`....l |
00000610 61 00 ff 00 09 7c 20 20 20 20 20 20 20 20 20 20 |a....| |
00000620 62 00 ff 00 09 8c 20 20 20 20 20 20 20 20 20 20 |b..... |
00000630 63 00 ff 00 09 9c 20 20 20 20 20 20 20 20 20 20 |c..... |
00000640
The draft standard document ISO/IEC CD2.1 1989:202x says of the USAGE PACKED-DECIMAL phrase...
The USAGE PACKED-DECIMAL clause specifies that a radix of 10 is used
to represent a numeric item in the storage of the computer.
Furthermore, this clause specifies that each digit position shall
occupy the minimum possible configuration in computer storage. Each
implementor specifies the precise effect of the USAGE PACKED-DECIMAL
clause upon the alignment and representation of the data item in the
storage of the computer, including the representation of any algebraic
sign. Sufficient computer storage shall be allocated by the
implementor to contain the maximum range of values implied by the
associated decimal picture character-string. If the WITH NO SIGN
phrase is specified the representation of the data item in the storage
of the computer reserves no storage for representing any sign value.
The PICTURE character string of the data item shall not contain the
symbol ‘S’; the data item is always considered to have a zero, or
positive value.
I sometimes get TLSv1 even when I set the version explicit to TLSv1.2.
I get this exception:
First chance exception at $75D917D2. Exception class
EIdOSSLConnectError with message 'Error connecting with SSL. EOF was
observed that violates the protocol'.
My Delphi version:
Embarcadero® RAD Studio 10 Seattle Version 23.0.21418.4207
I am on Windows 10:
Microsoft Windows [Version 10.0.17134.345]
My Indy Version: 10.6.2.5311 - That's what's coming with Seattle
This is a trace when it works:
This is a trace when it does NOT work:
As you can see it is TLSv1. Why?
I had a look at following questions:
Using Indy 10 IdHTTP with TLS 1.2
EIdOSSLConnectError Error connecting with SSL - EOF was observed
Here is my example code:
procedure TOpenWeatherFr.SetIcon(const IconCode: String);
var
HTTPS: TSSLHTTP;
MS : TMemoryStream;
PNG: TPngImage;
URL: string;
begin
HTTPS := TSSLHTTP.Create(nil);
MS := TMemoryStream.Create;
PNG := TPngImage.Create;
URL := 'https://openweathermap.org/img/w/' + IconCode + '.png';
try
(HTTPS.IOHandler as TIdSSLIOHandlerSocketOpenSSL).SSLOptions.SSLVersions := [sslvTLSv1_2];
HTTPS.Get(URL, MS);
MS.Seek(0, soFromBeginning);
PNG.LoadFromStream(MS);
Image1.Picture.Assign(PNG);
finally
HTTPS.Free;
PNG.Free;
MS.Free;
end;
Image1.Visible := true;
end;
For the sake of completeness here is the code of TSSLHTTP. The purpose is an old Indy version without SNI Support.
unit SSLHTTP;
interface
uses
Classes, IdHTTP, IdSSLOpenSSLHeaders, IdCTypes;
type
TSSLHTTP = class(TIdHTTP)
private
procedure OnStatusInfoEx(ASender: TObject; const AsslSocket: PSSL; const AWhere, Aret: TIdC_INT; const AType, AMsg: String);
public
constructor Create(AOwner: TComponent);
end;
implementation
uses
IdSSLOpenSSL;
{ TSSLHTTP }
constructor TSSLHTTP.Create(AOwner: TComponent);
begin
inherited Create(AOwner);
IOHandler := TIdSSLIOHandlerSocketOpenSSL.Create(self);
(IOHandler as TIdSSLIOHandlerSocketOpenSSL).OnStatusInfoEx := OnStatusInfoEx;
end;
procedure TSSLHTTP.OnStatusInfoEx(ASender: TObject; const AsslSocket: PSSL;
const AWhere, Aret: TIdC_INT; const AType, AMsg: String);
begin
SSL_set_tlsext_host_name(AsslSocket, Request.Host);
end;
end.
Where is my mistake? What am I doing wrong? This is the URL of the image I download:
https://openweathermap.org/img/w/04d.png
EDIT: Adding paylod as hex dumb
Working payload - Client Hello:
0000 16 03 01 01 49 01 00 01 45 03 03 8b 3d e5 54 59
0010 f8 59 44 82 f7 14 92 45 50 e1 a3 86 68 3a c2 94
0020 76 be ea 8b 54 98 f3 27 69 50 af 00 00 94 c0 30
0030 c0 2c c0 28 c0 24 c0 14 c0 0a 00 a3 00 9f 00 6b
0040 00 6a 00 39 00 38 c0 32 c0 2e c0 2a c0 26 c0 0f
0050 c0 05 00 9d 00 3d 00 35 00 88 00 87 00 84 c0 2f
0060 c0 2b c0 27 c0 23 c0 13 c0 09 00 a2 00 9e 00 67
0070 00 40 00 33 00 32 c0 31 c0 2d c0 29 c0 25 c0 0e
0080 c0 04 00 9c 00 3c 00 2f 00 9a 00 99 00 45 00 44
0090 00 96 00 41 00 07 c0 11 c0 07 c0 0c c0 02 00 05
00a0 00 04 c0 12 c0 08 00 16 00 13 c0 0d c0 03 00 0a
00b0 00 15 00 12 00 09 00 14 00 11 00 08 00 06 00 03
00c0 00 ff 01 00 00 88 00 00 00 17 00 15 00 00 12 6f
00d0 70 65 6e 77 65 61 74 68 65 72 6d 61 70 2e 6f 72
00e0 67 00 0b 00 04 03 00 01 02 00 0a 00 34 00 32 00
00f0 0e 00 0d 00 19 00 0b 00 0c 00 18 00 09 00 0a 00
0100 16 00 17 00 08 00 06 00 07 00 14 00 15 00 04 00
0110 05 00 12 00 13 00 01 00 02 00 03 00 0f 00 10 00
0120 11 00 23 00 00 00 0d 00 20 00 1e 06 01 06 02 06
0130 03 05 01 05 02 05 03 04 01 04 02 04 03 03 01 03
0140 02 03 03 02 01 02 02 02 03 00 0f 00 01 01
Not working payload - Client hello:
0000 16 03 01 01 49 01 00 01 45 03 03 f6 7b de 81 5d
0010 76 74 a7 31 99 36 8d 17 4c 07 5e 73 5d f7 b8 a1
0020 4f 06 5e 91 e5 f0 4b 37 0d 65 e7 00 00 94 c0 30
0030 c0 2c c0 28 c0 24 c0 14 c0 0a 00 a3 00 9f 00 6b
0040 00 6a 00 39 00 38 c0 32 c0 2e c0 2a c0 26 c0 0f
0050 c0 05 00 9d 00 3d 00 35 00 88 00 87 00 84 c0 2f
0060 c0 2b c0 27 c0 23 c0 13 c0 09 00 a2 00 9e 00 67
0070 00 40 00 33 00 32 c0 31 c0 2d c0 29 c0 25 c0 0e
0080 c0 04 00 9c 00 3c 00 2f 00 9a 00 99 00 45 00 44
0090 00 96 00 41 00 07 c0 11 c0 07 c0 0c c0 02 00 05
00a0 00 04 c0 12 c0 08 00 16 00 13 c0 0d c0 03 00 0a
00b0 00 15 00 12 00 09 00 14 00 11 00 08 00 06 00 03
00c0 00 ff 01 00 00 88 00 00 00 17 00 15 00 00 12 6f
00d0 70 65 6e 77 65 61 74 68 65 72 6d 61 70 2e 6f 72
00e0 67 00 0b 00 04 03 00 01 02 00 0a 00 34 00 32 00
00f0 0e 00 0d 00 19 00 0b 00 0c 00 18 00 09 00 0a 00
0100 16 00 17 00 08 00 06 00 07 00 14 00 15 00 04 00
0110 05 00 12 00 13 00 01 00 02 00 03 00 0f 00 10 00
0120 11 00 23 00 00 00 0d 00 20 00 1e 06 01 06 02 06
0130 03 05 01 05 02 05 03 04 01 04 02 04 03 03 01 03
0140 02 03 03 02 01 02 02 02 03 00 0f 00 01 01
I've been searching high and low for this to no avail. I'd like to be able to extract the byte offset at which each progressive JPEG layer/scan occurs.
For example, let's say a 100 kB image has 5 layers used to render the final image; layer 1 ends at kB 5, layer 2 at kB 20, layer 3 at kB 60, etc (for example).
Is it possible to use Imagemagick for this? The identify tool does not seem to support doing so. If so, how? Otherwise what tool(s) could accomplish this? I'd rather not have to write a custom JPEG parser. Thanks.
You can make a progressive JPEG like this with ImageMagick for testing:
magick -interlace plane -size 400x200 gradient: progressive.jpg
exiftool will tell you quite a lot about it like this:
exiftool -v3 progressive.jpg
Sample Output
ExifToolVersion = 12.00
FileName = progressive.jpg
Directory = .
FileSize = 2709
FileModifyDate = 1620144585
FileAccessDate = 1620144586
FileInodeChangeDate = 1620144585
FilePermissions = 33188
FileType = JPEG
FileTypeExtension = JPG
MIMEType = image/jpeg
JPEG APP0 (14 bytes):
0006: 4a 46 49 46 00 01 01 00 00 01 00 01 00 00 [JFIF..........]
+ [BinaryData directory, 9 bytes]
| JFIFVersion = 1 1
| - Tag 0x0000 (2 bytes, int8u[2]):
| 000b: 01 01 [..]
| ResolutionUnit = 0
| - Tag 0x0002 (1 bytes, int8u[1]):
| 000d: 00 [.]
| XResolution = 1
| - Tag 0x0003 (2 bytes, int16u[1]):
| 000e: 00 01 [..]
| YResolution = 1
| - Tag 0x0005 (2 bytes, int16u[1]):
| 0010: 00 01 [..]
| ThumbnailWidth = 0
| - Tag 0x0007 (1 bytes, int8u[1]):
| 0012: 00 [.]
| ThumbnailHeight = 0
| - Tag 0x0008 (1 bytes, int8u[1]):
| 0013: 00 [.]
JPEG DQT (65 bytes):
0018: 00 03 02 02 02 02 02 03 02 02 02 03 03 03 03 04 [................]
0028: 06 04 04 04 04 04 08 06 06 05 06 09 08 0a 0a 09 [................]
0038: 08 09 09 0a 0c 0f 0c 0a 0b 0e 0b 09 09 0d 11 0d [................]
0048: 0e 0f 10 10 11 10 0a 0c 12 13 12 10 13 0f 10 10 [................]
0058: 10 [.]
JPEG SOF2 (9 bytes):
005d: 08 00 c8 01 90 01 01 11 00 [.........]
ImageWidth = 400
ImageHeight = 200
EncodingProcess = 2
BitsPerSample = 8
ColorComponents = 1
JPEG DHT (20 bytes):
006a: 00 01 01 01 00 00 00 00 00 00 00 00 00 00 00 00 [................]
007a: 00 00 04 08 [....]
JPEG SOS
JPEG DHT (19 bytes):
0139: 10 01 01 00 00 00 00 00 00 00 00 00 00 00 00 00 [................]
0149: 00 00 12 [...]
JPEG SOS
JPEG DHT (18 bytes):
0468: 10 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [................]
0478: 00 a0 [..]
JPEG SOS
JPEG DHT (20 bytes):
048a: 10 01 01 01 00 00 00 00 00 00 00 00 00 00 00 00 [................]
049a: 00 71 00 70 [.q.p]
JPEG SOS
JPEG SOS
JPEG DHT (21 bytes):
0728: 10 00 03 01 00 00 00 00 00 00 00 00 00 00 00 00 [................]
0738: 00 01 71 00 50 [..q.P]
JPEG SOS
JPEG EOI
You can also find the SOS markers like this:
xxd -c16 -g1 -u progressive.jpg | grep --color=always -A4 "FF DA"
00000070: 00 00 00 00 00 00 00 00 00 00 00 00 04 08 FF DA ................
00000080: 00 08 01 01 00 00 00 01 D4 60 00 00 00 00 00 08 .........`......
00000090: 40 00 00 00 00 00 11 00 00 00 00 00 00 21 00 00 #............!..
000000a0: 00 00 00 00 42 00 00 00 00 00 00 88 00 00 00 00 ....B...........
000000b0: 00 01 08 00 00 00 00 00 02 10 00 00 00 00 00 04 ................
--
00000140: 00 00 00 00 00 00 00 00 00 00 00 12 FF DA 00 08 ................
00000150: 01 01 00 01 05 02 B5 AD 6B 5A D6 B5 AD 6B 5A D6 ........kZ...kZ.
00000160: B5 AD 6B 5A D6 B5 AD 6B 5A D6 B5 AD 6B 5A D6 B5 ..kZ...kZ...kZ..
00000170: AD 6B 5A D6 B5 AD 6B 5A D6 B5 AD 6B 5A D6 B5 AD .kZ...kZ...kZ...
00000180: 6B 5A D6 B5 AD 6B 5A D6 B5 AD 6B 5A D6 B5 AD 6B kZ...kZ...kZ...k
It may miss some markers if they fall across a 16-byte line end - I am still thinking about making this fool-proof. Maybe run it twice with a 4-byte offset so they can't fall on the boundary in both listings.
Alternatively, I did another answer here that lets you look for arbitrary binary sequences that you could easily adapt from looking for TIFF headers to looking for FF DA.
All you'd have to do is scan the image stream and look for the SOS (Start of Scan) markers.
There is some php example code in this article by Christoph Erdmann:
https://www.smashingmagazine.com/2019/08/faster-image-loading-embedded-previews/#creating-frontend-javascript-code
$img = "progressive.jpg";
$jpgdata = file_get_contents($img);
$positions = [];
$offset = 0;
while ($pos = strpos($jpgdata, "\xFF\xC4", $offset)) {
$positions[] = $pos+2;
$offset = $pos+2;
}
I am trying to convert a string to a list of integers.
String = "08 02 22 97 38 15 00 40 00 75 04 05 07 78 52 12 50 77 91 08".
But
lists:map(fun(X) -> string:to_integer(X) end, string:tokens(String, " ")).
just gives me...
[{8,[]}, {2,[]}, {22,[]}, {97,[]}, ... , {91,[]}, {8,[]}]
Can someone perhaps tell me what a good/nice way would be to get?
[8,2,22,97...91,8]
(Or do I need a helper function?)
This works:
String = "08 02 22 97 38 15 00 40 00 75 04 05 07 78 52 12 50 77 91 08".
lists:map(fun(X) -> {Int, _} = string:to_integer(X),
Int end,
string:tokens(String, " ")).
> [8,2,22,97,38,15,0,40,0,75,4,5,7,78,52,12,50,77,91,8]
See, string:to_integer returns not a single integer, but a tuple:
to_integer(String) -> {Int,Rest} | {error,Reason}
... so you have to extract the first element from this tuple to get the actual number.
Also, you might use list generator syntax:
[begin {Int,_}=string:to_integer(Token), Int end|| Token<-string:tokens(String," ")].
From shell:
1> String = "08 02 22 97 38 15 00 40 00 75 04 05 07 78 52 12 50 77 91 08".
"08 02 22 97 38 15 00 40 00 75 04 05 07 78 52 12 50 77 91 08"
2>
2> [begin {Int,_}=string:to_integer(Token), Int end|| Token<-string:tokens(String," ")].
[8,2,22,97,38,15,0,40,0,75,4,5,7,78,52,12,50,77,91,8]
[list_to_integer(X)|| X<-string:tokens(String," ")].