I'm trying to acquire a memory trace of a run of a simple program where I could see both address and data being read/written, e.g. W 0x7fffffffd928 4 0xe4d829d0 which would stand for writing a 4-byte value 0xe4d829d0 to the address 0x7fffffffd928.
There's a tool in Valgrind (Lackey tool) which only gives you the address and data length. Since Valgrind tools' documentation is quite poor, I viewed Valgrind sources but there doesn't seem to be any direct access to the data.
Another possibility seems to be Pin from Intel but after reading its documentation I haven't found anything relevant either.
This seems to be a very simple problem, however, I can't figure out or find any solution. Thanks!
The key function for PIN is PIN_SafeCopy which must be surrounded by PIN_GetLock and PIN_ReleaseLock. See the full functional PIN tool at Philippe Teuwen's blog.
Related
I am quite new at embedded development and started with a STM32F429 board to improve myself.
I have just developed a basic Caesar encryption application for my board. It is working well, and defined the secret key as "3". Now I would like to extract this super secret(!) key from my device.
How can I do it? Should I dump the memory or firmware of my device, and how?
May you suggest me any software for this proccess? (Not ST Utility or STM softwares please. Because I would like to try gained experiences on other devices as well.)
Thanks!
I take it the value you're looking for is hardcoded. In that case it resides in the internal flash. So yes, a memory dump will be necessary.
I will go the long way and assume that you know very little about how it works, so if you know some of this stuff, well, good for you. I will try to give a few pointers.
Specifically about STM32:
You have an option to boot the microcontroller from the so-called system memory, which is read-only memory, and it is already preprogrammed from factory with a bootloader. You can talk to that running bootloader via UART (most common way, comes with ST-Link, but any cheapo USB-UART bridge also works). Or it can be some other protocol. You can ask that bootloader to read its flash out to you, among other things. This is covered in AN2606.pdf. It has some useful links in it, such as:
names of documents, where you can find specific bootloader commands for any interface you wish. Of course, you only care about interfaces, that the bootloader of your specific MCU F429 supports, which are found in the same AN2606, page 172 (for bootloader version 0.7, there is also 0.9 for those MCUs, I have no idea how to tell which one you have, so...try? UART configuration seems identical anyway):
So what exactly needs to be done? Flip the state of BOOT0 pin - permanently - of the MCU and reset it (power cycle or reset pin, both ok). You will boot the MCU into bootloader instead of booting program from flash. You can read about it in the Reference manual STM32F429, page 69. It talks about states of BOOT0 and BOOT1 pins on boot. What pins are boot0 - if they're not marked on your board, then you'll have to consult F429 datasheet, page 69 (I swear, it's a coincidence). Depending on your specific IC, it will be one pin or another.
It will activate all MCUs peripherals as per docs above and it and wait on its UART and other pins for commands. Commands listed in the documents I provided above. Let's take a look at AN3155 about USART of bootloader:
And the commands are
are all in that document, the table of contents in pdf really helps to find stuff quickly. Of course, if you need specific details, and you will need specific information about specific commands, it's all in there too. How many bytes in command, how many bytes at a time you read from flash etc. Basically, you can either write your own program that does that (even program another microcontroller to program that microcontroller using victim's bootloader), or use any other software that knows what commands to send to the bootloader. It can be ST utility, it can be any other program. They all implement the very same command set, so it doesn't actually matter much. I couldn't find many programs that do that, the only thing that stood out was stm32flash. Never used it myself. I'm ok with ST stuff, since I know what it does (I think).
Oh yeah back to getting the secret value out. I almost forgot about that. Well, then you open the dump in hex viewer/editor, and scroll it around looking for interesting combination of values. Yeah, that's kinda what it looks like. One can run it via disassembly. Scroll disassembled code around, see if there are any numeric values that stand out. You know, some random number 0xD35B581 or something hardcoded in the middle of pretty program could mean something, like be a serial number or a secret number. Unfortunately, I'm reaching boundaries of my competence here, so won't go any further on what one can do with dump.
I'd like to extract the stacktrace from crashing applications with large memory footprints. Ideally, the user wouldn't need to wait while the entire coredump is written to disk.
My current thinking is to install a coredump hook at /proc/sys/kernel/core_pattern which would parse the incoming coredump via stdin and extract just the stacktrace. But, creating a complete copy of the coredump in memory would be impractical, so a streaming approach would be better.
I'm new to the ELF format (http://en.wikipedia.org/wiki/Executable_and_Linkable_Format) and was wondering if it might support a streaming parser. I haven't written a streaming parser of any kind yet - I'm familiar with the concept but need pointers on how to analyze a format for stream-ability.
As a first attempt, I tried:
cat core | readelf -a
But, it doesn't seem like readelf supports input from stdin.
I also found this python elf parser, but it appears at first glance like it reads the entire elf into memory:
https://github.com/eliben/pyelftools
But, if needed, maybe I could use their implementation as reference for a streaming parser.
Thanks a bunch!
It turns out that Google's coredumper documents the ELF core file format:
https://code.google.com/p/google-coredumper/source/browse/trunk/src/elfcore.c
This code snippet was also helpful:
http://emntech.blogspot.com/2012/08/printing-backtracestack-trace-using.html
It appears that the stacktrace is contained in a single segment of the elf. The solution then is to:
Read the elf headers
Find a note entry of type NT_PRSTATUS
Get the top of stack address from a register within this entry
Go to that address
Read the stacktrace
Ignore the rest of the coredump
I still have some work to do in terms of resolving symbols, etc. But, I'll edit this answer if my approach changes significantly. While whether the format can be 'streamed' was not really the right question to ask, I did find a solution which allowed me to read the stacktrace without writing the entire coredump to disk.
EDIT:
According to the answers to How gdb reconstructs stacktrace for C++?, it seems that reconstructing the stack in all cases is quite complex. I believe then the final answer to this question is, no, it's not possible to extract the stack from ELF Cores and an ELF Core is not "streamable."
I believe though there is a chance the heap could be located in a coredump and removed. This would leave the stack intact, allowing gdb to still reconstruct it.
Thanks to help by David Heffernan I have a program written in Freepascal (but a Delphi solution to my question would suffice) that reads a physical disk sector by sector. It does so using the Windows API CreateFileW function for the disk handle, then FileFile, FileSeek etc to navigate and read. If all the sectors are OK, it works fine. However, if the disk had bad sectors, I need to treat them differently.
My question is, is there and procedures or libraries that can be used, while reading these sectors, to determine if they are bad sectors? If not, how might I go about it? I gather it is the disk controller that knows what sectors are bad and which are not, so I don't think my program can actually access a bad sector, so how can I detect which are the bad ones and act accordingly? Does one need to query SMART and if so, how?
I have searched this site (only found this C post, which relates to a program, not code) and Googled it and no obvious solutions came to my attention.
Generally speaking, you can't access bad sectors at all (they have been remapped already so are out of LBA). What you can access are pending sectors, attempts to read them will always cause a read error. SMART will tell you nothing but the number of bad/pending sectors. So you probably should continue using chosen API interpreting persistent read errors as diagnostics for "bad" sectors, just make sure they aren't caused by access sharing violation.
If you want to obtain a p-list or g-list somehow, it is only possible (for PATA/SATA, not SCSI) in terminal mode, what requires connection to HDD's service port, USB-to-COM adapter and is vendor- and product-specific, if possible at all.
Sectors and their hardware status are not things that normal user-level code needs to deal with so there is no easy copy/paste API available for this purpose.
Also in general the sector concept is abstracted away on multiple levels. For one example see the Wikipedia: logical disk address translation. Physical sector status is very low-level concept. Some hardware vendors even don't expose it through public API at all. Bad (or suspicious) sectors are often detected in the hardware itself and automatically redirected to other places. So in general the bad disk-sector concept does not exist
MSDN Logging Guidelines
...Bad sectors. If a disk driver encounters a bad sector, it may be able to read from or write to the sector after retrying the operation, but the sector will go bad eventually. If the disk driver can proceed, it should log a Warning event; otherwise, it should log an Error event. If a file system driver finds a large number of bad sectors and fixes them, logging Warning events might help an administrator determine that the disk may be about to fail...
If you really need to work with this low-level concepts then first forget about Pascal or Delphi as your requirements.
Learn how to use the Windows API and once you know it bind to the API in your language of choice (you can map any Win32 user-level API function to Free Pascal easily).
For understanding how user-level code sees the disk abstraction start reading documentation at MSDN → Dev Center - Desktop → Device Management Reference → Device Management Functions → DeviceIOControl function
For understanding how the kernel-level code sees the hardware and how does it communicate with user-level code start reading documentation at MSDN → Dev Center - Hardware → Develop → Drivers → Concepts for all driver developers
For example of reading S.M.A.R.T. disk information see WinSim Inc. DISKID32 source code function ReadPhysicalDriveInNTUsingSmart() in diskid32.cpp
In my opinion you are going to swim in a dark & deep waters without flashlight and swim ring and you should think twice about what you (or your users) really need/want and perhaps improve the question to get a reasonably-sized on-topic answer
I want to see is fragmentation the reason of increasing memory usage of my twisted server. I have posted a question here:
How to find the source of increasing memory usage of a twisted server?
Now, what I am going to do is to visualize the heap. I found an article: Memory fragmentation. The figure of heap in that article is something just like what I want. It is not difficult for me to draw such a figure with matplotlib or other tools. The most difficult job is: how to record the memory allocation and deallocation?
I know that I can modify the source code of CPython, add some logging code into omalloc.c and recompile the Python, and use the modified CPython to run my server. But however, I don't want to waste time with that. Then I am looking for some available tools. I know there is a tool valgrind can be used to detect memory leak, but I don't know how to record allocation and deallocation. I see its memcheck can detect something like:
Invalid read
Uninitialised
Invalid free
But that's not what I want, all I need is:
Record every allocations and deallocations of memory with time, address and size
My questions are:
How can I do that with valgrind?
If I can't, should I write a module for that?
Or is there any better tools can achieve this?
Thanks.
Replying to question about alternative tools:
I know there there is "Pin" tool, maybe it's worth to check it out: http://www.pintool.org/
I mostly work on C language for my work. I have faced many issues and spent lot time in debugging issues related to dynamically allocated memory corruption/overwriting. Like malloc(A) A bytes but use write more than A bytes. Towards that i was trying to read few things when i read about :-
1.) An approach wherein one allocates more memory than what is needed. And write some known value/pattern in that extra locations. Then during program execution that pattern should be untouched, else it indicated memory corruption/overwriting. But how does this approach work. Does it mean for every write to that pointer which is allocated using malloc() i should be doing a memory read of the additional sentinel pattern and read for its sanity? That would make my whole program very slow.
And to say that we can remove these checks from the release version of the code, is also not fruitful as memory related issues can happen more in 'real scenario'. So can we handle this?
2.) I heard that there is something called HEAP WALKER, which enables programs to detect memory related issues? How can one enable this.
thank you.
-AD.
If you're working under Linux or OSX, have a look at Valgrind (free, available on OSX via Macports). For Windows, we're using Rational PurifyPlus (needs a license).
You can also have a look at Dmalloc or even at Paul Nettle's memory manager which helps tracking memory allocation related bugs.
If you're on Mac OS X, there's an awesome library called libgmalloc. libgmalloc places each memory allocation on a separate page. Any memory access/write beyond the page will immediately trigger a bus error. Note however that running your program with libgmalloc will likely result in a significant slowdown.
Memory guards can catch some heap corruption. It is slower (especially deallocations) but it's just for debug purposes and your release build would not include this.
Heap walking is platform specific, but not necessarily too useful. The simplest check is simply to wrap your allocations and log them to a file with the LINE and FILE information for your debug mode, and most any leaks will be apparent very quickly when you exit the program and numbers don't tally up.
Search google for LINE and I am sure lots of results will show up.