I have hex file from memory dump created from reading memory area of 32768 bytes in size.
... ... ....
80001174: aaaa aaaa aaaa aaaa aaaa aaaa 27bd ff80 | ............'...
80001184: afa1 0004 afa2 0008 afa3 000c afa4 0010 | ................
80001194: afa5 0014 afa6 0018 afa7 001c afa8 0020 | ...............
800011a4: afa9 0024 afaa 0028 afab 002c afac 0030 | ...$...(...,...0
800011b4: afad 0034 afae 0038 afaf 003c afb0 0040 | ...4...8...<...#
800011c4: afb1 0044 afb2 0048 afb3 004c afb4 0050 | ...D...H...L...P
800011d4: afb5 0054 afb6 0058 afb7 005c afb8 0060 | ...T...X...\...`
800011e4: afb9 0064 afbc 0070 afbe 0078 afbf 007c | ...d...p...x...|
800011f4: 3c1a 8045 275a 1024 0340 f809 0000 0000 | <..E'Z.$.#......
80001204: 8fa1 0004 8fa2 0008 8fa3 000c 8fa4 0010 | ................
I tried convert hex dump to binary file use command
xxd -r -p test.log file.bin
but created binary file size is 40968 bytes, that exceed the expected size 32768 bytes. What can be wrong in conversion?
It seems that your hex file is not a plain hex file as it has the"offsets" listed in the file as well.
xxd will do the correct conversion if you have a plaing hex file without any offset in it.
If you can generate again your hex file please do it in the plain format, otherwise, you can open this Hex file using notepad++ and delete the entire column with the address.
Related
I want to validate that my ZIP file has a correct CRC-32 checksum.
I read that in a ZIP file the CRC-32 data is in bytes 14 to 17:
Offset Bytes Description[30]
0 4 Local file header signature = 0x04034b50 (read as a little-endian number)
4 2 Version needed to extract (minimum)
6 2 General purpose bit flag
8 2 Compression method
10 2 File last modification time
12 2 File last modification date
14 4 CRC-32 of uncompressed data
18 4 Compressed size
22 4 Uncompressed size
26 2 File name length (n)
28 2 Extra field length (m)
30 n File name
30+n m Extra field
I wanted to validate a CRC-32 checksum of a simple ZIP file I created:
00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
-----------------------------------------------
50 4B 03 04 14 00 00 00 00 00 38 81 1C 51 4C 18 | PK........8..QL.
C7 8C 02 00 00 00 02 00 00 00 07 00 00 00 31 32 | nj............12
33 2E 64 61 74 73 73 50 4B 01 02 14 00 14 00 00 | 3.datssPK.......
00 00 00 38 81 1C 51 4C 18 C7 8C 02 00 00 00 02 | ...8..QL.nj.....
00 00 00 07 00 00 00 00 00 00 00 01 00 20 00 00 | ............. ..
00 00 00 00 00 31 32 33 2E 64 61 74 50 4B 05 06 | .....123.datPK..
00 00 00 00 01 00 01 00 35 00 00 00 27 00 00 00 | ........5...'...
00 00 | ..
The CRC-32 is: 0x4C18C78C
I went to this CRC-32 online calculator and added the following un-compressed row from the file:
50 4B 03 04 14 00 00 00 00 00 38 81 1C 51
This is the result:
Algorithm Result Check Poly Init RefIn RefOut XorOut
CRC-32 0x6A858174 0xCBF43926 0x04C11DB7 0xFFFFFFFF true true 0xFFFFFFFF
CRC-32/BZIP2 0xE3FA1205 0xFC891918 0x04C11DB7 0xFFFFFFFF false false 0xFFFFFFFF
CRC-32C 0xB578110E 0xE3069283 0x1EDC6F41 0xFFFFFFFF true true 0xFFFFFFFF
CRC-32D 0xAFE2EEA4 0x87315576 0xA833982B 0xFFFFFFFF true true 0xFFFFFFFF
CRC-32/MPEG-2 0x1C05EDFA 0x0376E6E7 0x04C11DB7 0xFFFFFFFF false false 0x00000000
CRC-32/POSIX 0xFF9B3071 0x765E7680 0x04C11DB7 0x00000000 false false 0xFFFFFFFF
CRC-32Q 0x79334F11 0x3010BF7F 0x814141AB 0x00000000 false false 0x00000000
CRC-32/JAMCRC 0x957A7E8B 0x340BC6D9 0x04C11DB7 0xFFFFFFFF true true 0x00000000
CRC-32/XFER 0xA7F36A3F 0xBD0BE338 0x000000AF 0x00000000 false false 0x00000000
But none of them equal to: 0x4C18C78C.
What am I doing wrong? The CRC-32 of the ZIP is the calculation of all the bytes (0-13) before, no?
The byte sequence you are running against the online CRC calculator are not uncompressed bytes.
50 4B 03 04 14 00 00 00 00 00 38 81 1C 51
Those bytes are the first few bytes of the zip file. The CRC32 value in a zip is calculated by running the CRC32 algorithm against the complete uncompressed payload. In your case the payload is the two byte sequence "ss".
To work that out, I converted your hex dump back into a zip file, tmp.zip. It contains a single member 123.dat
$ unzip -lv tmp.zip
Archive: tmp.zip
Length Method Size Cmpr Date Time CRC-32 Name
-------- ------ ------- ---- ---------- ----- -------- ----
2 Stored 2 0% 2020-08-28 16:09 8cc7184c 123.dat
-------- ------- --- -------
2 2 0% 1 file
When I extract that member to stdout & pipe though hexdump, we find it contains the two bytes string "ss" (hex 73 73)
$ unzip -p tmp.zip | hexdump -C
00000000 73 73 |ss|
Finally, as already mentioned in another comment, you can check that the CRC value is correct by running unzip -t
$ unzip -t tmp.zip
Archive: tmp.zip
testing: 123.dat OK
No errors detected in compressed data of tmp.zip.
I was able to create a zip file that matches the one in the question. The header shows that the compression type == 0, which means no compression, the uncompressed size == 2, the data == {73 73}. CRC32 uses reflected input and output, and the CRC is stored in little endian format, so the CRC == 0x8CC7184C.
I get a match using CRC32 on data of {73 73} using this online CRC calculator:
http://www.sunshine2k.de/coding/javascript/crc/crc_js.html
I build the WebRequest object and then call GetResponse() on it. The url is "https://services.odata.org/V2/Northwind/Northwind.svc/$metadata".
And it comes up in a browser fine. What's really weird is these are returning a 200 so it should be good. But no data is coming back.
According to Fiddler there are two requests that go out:
Request #1
A SSLv3-compatible ClientHello handshake was found. Fiddler extracted the parameters below.
Version: 3.1 (TLS/1.0)
Random: 5E 5B 1A 26 84 02 5A 4B DF 6B BF 22 DF 2C 03 AD 37 62 DA CE 05 79 7E B4 5C 58 AF BE 8C 26 5D CA
"Time": 4/4/1990 2:38:54 PM
SessionID: empty
Extensions:
server_name services.odata.org
supported_groups x25519 [0x1d], secp256r1 [0x17], secp384r1 [0x18]
ec_point_formats uncompressed [0x0]
SessionTicket empty
extended_master_secret empty
renegotiation_info 00
Ciphers:
[C00A] TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
[C009] TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
[C014] TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
[C013] TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
[0035] TLS_RSA_WITH_AES_256_CBC_SHA
[002F] TLS_RSA_WITH_AES_128_CBC_SHA
[000A] SSL_RSA_WITH_3DES_EDE_SHA
Compression:
[00] NO_COMPRESSION
Response #1
Request #2
A SSLv3-compatible ClientHello handshake was found. Fiddler extracted the parameters below.
Version: 3.1 (TLS/1.0)
Random: 5E 5B 1A 26 0D B0 9F BF 4F D8 FC 5D 27 47 A8 7C CB 83 2C 43 4C 2B C3 8A 0C C0 EB 30 3D 6E AB 51
"Time": 4/4/1990 2:38:54 PM
SessionID: empty
Extensions:
server_name services.odata.org
supported_groups x25519 [0x1d], secp256r1 [0x17], secp384r1 [0x18]
ec_point_formats uncompressed [0x0]
SessionTicket empty
extended_master_secret empty
renegotiation_info 00
Ciphers:
[C00A] TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
[C009] TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
[C014] TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
[C013] TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
[0035] TLS_RSA_WITH_AES_256_CBC_SHA
[002F] TLS_RSA_WITH_AES_128_CBC_SHA
[000A] SSL_RSA_WITH_3DES_EDE_SHA
Compression:
[00] NO_COMPRESSION
Response #2
It was the TLS level. Just add:
ServicePointManager.SecurityProtocol |= SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls;
I'm trying to use Bosun to determine if certain processes are running and then eventually alert on if they are up or down. I'm probably misinterpreting the docs but I can't figure this out.
Bosun is running just fine. I have the scollector running on Ubuntu 14 LTS and using my config file correctly.
Here is what I have in my scollector.toml:
host="blah:8070"
hostname="cass01"
[[Process]]
command = "^.*(java).*(CassandraDaemon)$"
name = "Cassandra"
I would then expect to see in bosun under my host cass01 a metric title "cassandra" somewhere but it's nowhere to be seen. Other metrics are there.
Right now Command is a partial match on the process path to the binary, up to the first space delimiter. The Args parameter is a regex to differentiate between multiple instances of the process. So for a java process you would use something like:
[[Process]]
Command = "java"
Name = "Cassandra"
Args = "CassandraDaemon$"
This would match a command line like:
/usr/bin/java /usr/bin/CassandraDaemon
This assumes the /proc/<pid>/cmdline for that process ends in CassandraDaemon. If it doesn't end in that string you would need to change the Args to just "CassandraDaemon" which would match any java process that contains that string.
Also some processes change the cmdline to something other than a nul delimited string. In those cases the Command argument needs to be used to match as Args is expecting nul delimiters. Example:
cat /proc/80156/cmdline | hexdump -C
00000000 2f 75 73 72 2f 62 69 6e 2f 72 65 64 69 73 2d 73 |/usr/bin/redis-s|
00000010 65 72 76 65 72 20 2a 3a 36 33 37 39 00 00 00 00 |erver *:6379....|
00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000030 00 |.|
00000031
#Example for cmdline without NUL (00) delimiters between args
[[Process]]
Command = "redis-server *:6379"
Name = "redis-core"
Once these are in place with the correct matching values you should see metrics show up under linux.proc.* where the the name tag will match the name used in the TOML file.
I am interested in learning about the deflate compression algorithm, particularly how is it represented in a data-stream, and feel that I would greatly benefit from some extra examples (eg. the compression of a short string of text, or the decompression of a compressed chunk).
I am continuing to study some resources I have found: ref1, ref2, ref3 but these do not have many examples of how the actual compression looks as a data-stream.
If I could get a few examples of how some strings would look before and after being compressed, and an explanation of the relationship between them that would be fantastic.
Also if there are other resources that I could be looking at please add those.
You can compress example data with gzip or zlib and use infgen to disassemble and examine the resulting compressed data. infgen also has an option to see the detail in the dynamic headers.
+1 for infgen, but here's a slightly more detailed answer.
You can take a look at the before- and after- using gzip and any hex editor. For example, xxd is included on most linux distros. I'd included both raw hex output (not that interesting without understanding) and infgen's output.
hello hello hello hello (triggers static huffman coding, like most short strings).
~ $ echo -n "hello hello hello hello" | gzip | xxd
00000000: 1f8b 0800 0000 0000 0003 cb48 cdc9 c957 ...........H...W
00000010: c840 2701 e351 3d8d 1700 0000 .#'..Q=.....
~ $ echo -n "hello hello hello hello" | gzip | ./infgen/a.out -i
! infgen 2.4 output
!
gzip
!
last
fixed
literal 'hello h
match 16 6
end
!
crc
length
\xff\xfe\xfd\xfc\xfb\xfa\xf9\xf8\xf7\xf6\xf5\xf4\xf3\xf2\xf1 (triggers uncompressed mode)
~ $ echo -ne "\xff\xfe\xfd\xfc\xfb\xfa\xf9\xf8\xf7\xf6\xf5\xf4\xf3\xf2\xf1" | gzip | xxd
00000000: 1f8b 0800 0000 0000 0003 010f 00f0 ffff ................
00000010: fefd fcfb faf9 f8f7 f6f5 f4f3 f2f1 c6d3 ................
00000020: 157e 0f00 0000 .~....
~ $ echo -ne "\xff\xfe\xfd\xfc\xfb\xfa\xf9\xf8\xf7\xf6\xf5\xf4\xf3\xf2\xf1" | gzip | ./infgen/a.out -i
! infgen 2.4 output
!
gzip
!
last
stored
data 255 254 253 252 251 250 249 248 247 246 245 244 243 242 241
end
!
crc
length
abaabbbabaababbaababaaaabaaabbbbbaa (triggers dynamic huffman coding)
~ $ echo -n "abaabbbabaababbaababaaaabaaabbbbbaa" | gzip | xxd
00000000: 1f8b 0800 0000 0000 0003 1dc6 4901 0000 ............I...
00000010: 1040 c0ac a37f 883d 3c20 2a97 9d37 5e1d .#.....=< *..7^.
00000020: 0c6e 2934 9423 0000 00 .n)4.#...
~ $ echo -n "abaabbbabaababbaababaaaabaaabbbbbaa" | gzip | ./infgen/a.out -i -d
! infgen 2.4 output
!
gzip
!
last
dynamic
count 260 7 18
code 1 4
code 2 1
code 4 4
code 16 4
code 17 4
code 18 2
zeros 97
lens 1 2
zeros 138
zeros 19
lens 4
repeat 3
lens 2
zeros 3
lens 2 2 2
! litlen 97 1
! litlen 98 2
! litlen 256 4
! litlen 257 4
! litlen 258 4
! litlen 259 4
! dist 0 2
! dist 4 2
! dist 5 2
! dist 6 2
literal 'abaabbba
match 4 7
match 3 9
match 5 6
literal 'aaa
match 5 5
literal 'b
match 4 1
literal 'aa
end
!
crc
length
I found infgen was still not enough detail to fully understand the format. I look through decompressing all three examples here bit-by-bit, by hand, in detail on my blog
For concepts, in addition to RFC 1951 (DEFLATE) which is pretty good, I would recommend Feldspar's conceptual overview of Huffman codes and LZ77 in DEFLATE
I tested my application with FastMM and FullDebugMode turned on, since I had some shutdown problems.
After solving bunch of my own problems FastMM started to complain about calling virtual method on a freed object in TPopupList. I tried to move the menus unit as early as possible in uses so that it would be finalized last, but it didn't help. Is this real problem, a bug in vcl or false alarm from FastMM?
Here's the full report from FastMM:
FastMM has detected an attempt to call a virtual method on a freed object. An access violation will now be raised in order to abort the current operation.
Freed object class: TPopupList
Virtual method: Offset +16
Virtual method address: 4714E4
The allocation number was: 220
The object was allocated by thread 0x1CC0, and the stack trace (return addresses) at the time was:
403216 [sys\system.pas][System][System.#GetMem][2654]
404A4F [sys\system.pas][System][System.TObject.NewInstance][8807]
404E16 [sys\system.pas][System][System.#ClassCreate][9472]
404A84 [sys\system.pas][System][System.TObject.Create][8822]
7F2602 [Menus.pas][Menus][Menus.Menus][4223]
40570F [sys\system.pas][System][System.InitUnits][11397]
405777 [sys\system.pas][System][System.#StartExe][11462]
40844F [SysInit.pas][SysInit][SysInit.#InitExe][663]
7F6368 [PCCSServer.dpr][PCCSServer][PCCSServer.PCCSServer][148]
7C90DCBA [ZwSetInformationThread]
7C817077 [Unknown function at RegisterWaitForInputIdle]
The object was subsequently freed by thread 0x1CC0, and the stack trace (return addresses) at the time was:
403232 [sys\system.pas][System][System.#FreeMem][2699]
404A6D [sys\system.pas][System][System.TObject.FreeInstance][8813]
404E61 [sys\system.pas][System][System.#ClassDestroy][9513]
428D15 [common\Classes.pas][Classes][Classes.TList.Destroy][2914]
404AB3 [sys\system.pas][System][System.TObject.Free][8832]
472091 [Menus.pas][Menus][Menus.Finalization][4228]
4056A7 [sys\system.pas][System][System.FinalizeUnits][11256]
4056BF [sys\system.pas][System][System.FinalizeUnits][11261]
7C9032A8 [RtlConvertUlongToLargeInteger]
7C90327A [RtlConvertUlongToLargeInteger]
7C92AA0F [Unknown function at towlower]
The current thread ID is 0x1CC0, and the stack trace (return addresses) leading to this error is:
4714B8 [Menus.pas][Menus][Menus.TPopupList.MainWndProc][3779]
435BB2 [common\Classes.pas][Classes][Classes.StdWndProc][11583]
7E418734 [Unknown function at GetDC]
7E418816 [Unknown function at GetDC]
7E428EA0 [Unknown function at DefWindowProcW]
7E428EEC [Unknown function at DefWindowProcW]
7C90E473 [KiUserCallbackDispatcher]
7E42B1A8 [DestroyWindow]
47CE31 [Controls.pas][Controls][Controls.TWinControl.DestroyWindowHandle][6857]
493BE4 [Forms.pas][Forms][Forms.TCustomForm.DestroyWindowHandle][4564]
4906D9 [Forms.pas][Forms][Forms.TCustomForm.Destroy][2929]
Current memory dump of 256 bytes starting at pointer address 7FF9CFF0:
2C FE 82 00 80 80 80 80 80 80 80 80 80 80 80 80 80 80 80 80 C4 A3 2D 0C 00 00 00 00 B1 D0 F9 7F
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 C0 00 00 00 16 32 40 00 9D 5B 40 00 C8 5B 40 00
CE 82 40 00 3C 40 91 7C B0 B1 94 7C 0A 77 92 7C 84 77 92 7C 7C F0 96 7C 94 B3 94 7C 84 77 92 7C
C0 1C 00 00 32 32 40 00 12 5B 40 00 EF 69 40 00 BA 20 47 00 A7 56 40 00 BF 56 40 00 A8 32 90 7C
7A 32 90 7C 0F AA 92 7C 0A 77 92 7C 84 77 92 7C C0 1C 00 00 0E 00 00 00 00 00 00 00 C7 35 65 59
2C FE 82 00 80 80 80 80 80 80 80 80 80 80 38 CA 9A A6 80 80 80 80 80 80 00 00 00 00 51 D1 F9 7F
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 C1 00 00 00 16 32 40 00 9D 5B 40 00 C8 5B 40 00
CE 82 40 00 3C 40 91 7C B0 B1 94 7C 0A 77 92 7C 84 77 92 7C 7C F0 96 7C 94 B3 94 7C 84 77 92 7C
, þ ‚ . € € € € € € € € € € € € € € € € Ä £ - . . . . . ± Ð ù
. . . . . . . . . . . . . . . . À . . . . 2 # . [ # . È [ # .
Î ‚ # . < # ‘ | ° ± ” | . w ’ | „ w ’ | | ð – | ” ³ ” | „ w ’ |
À . . . 2 2 # . . [ # . ï i # . º G . § V # . ¿ V # . ¨ 2 |
z 2 | . ª ’ | . w ’ | „ w ’ | À . . . . . . . . . . . Ç 5 e Y
, þ ‚ . € € € € € € € € € € 8 Ê š ¦ € € € € € € . . . . Q Ñ ù
. . . . . . . . . . . . . . . . Á . . . . 2 # . [ # . È [ # .
Î ‚ # . < # ‘ | ° ± ” | . w ’ | „ w ’ | | ð – | ” ³ ” | „ w ’ |
I'm using Delphi 2007 and FastMM 4.97.
Edit1: I think the main problem here is why does Classes.StdWndProc call Menus.TPopupList? Digging the call stack inside debugger shows that System.FinalizeUnit is called three times, then it goes to SysUtils.ShowException, which tries to display MessageBox and after bunch of user32.dll calls we end up to classes.StdWndProc.
Edit2: I had problem with interfaces, fixing that made this problem go away. The object with interface was freed, but the reference was released later on. When the interface was released, occured an exception which I initially somehow ignored. Releasing the interface probably corrupted something which caused all other problems.
That situation can happens when a unit finalize after another unit it indirectly depends on.
For exemple, take the following unit:
unit Unit1;
interface
uses
Contnrs;
var
ItemHolder : TObjectList;
implementation
initialization
ItemHolder := TObjectList.Create(True);
finalization
ItemHolder.Free;
end.
That unit only directly depends on Contnrs. For that reason, delphi will ensure that this unit is finalized before Contnrs is. If the ObjectList contains TForms, Delphi won't ensure that Unit1 is finalized before unit Forms. If there are still some forms left while closing the application, TObjectList (Since it owns the object) will free the items it contains(Call TForm.Free). But since Unit1 doesn't depends on TForm, it's possible that the unit Forms is already finalized and that TForm.Destroy isn't in memory anymore.
This is why you need to be very carefull about what you do in finalization sections.
I'm not sure it's the source of your problem, but I would look that way first.
I've seen such problems with Delphi 2007 before. Sometimes the compiler gets confused and generates incorrect initialization or finalization order. Sadly, I was never able to create a reproducible test case to send to the CodeGear/Embarcadero people.
Whenever that happened, a full rebuild helped.
Make sure that FastMM4 is the FIRST line in your project file's uses clause (project|View source). If its not there, then add it.
It looks like one of your forms is getting destroyed after Menus.pas has been finalized. If your form has a menu on it, it would probably have to have Menus in its uses list in the interface section, which should make this impossible.
The only time I've seen issues like this pop up (no pun intended) is when using packages. Are you perhaps using a DPK with a plugin that adds a popup menu or menu items to your program? Package finalization can do some strange things to your program if you're not careful.
Either way, the solution is probably to dispose of the menu yourself before menus.pas finalizes. When it's time for the program to shut down, call Free on your popup menu and see if that solves the problem.
Update: this is only partial workaround
Workaround:
In main form of your application write
finalization
FreeAndNil(PopupList);
end.
this free PopupList and set to nil, so PopupList.Free in menus.pas will be OK.