i have a probably stack overflow in my application (off course, only in release mode...), and would like to add some protection/investigation code to it.
i am looking for a windows API to tell me the current state of a thread stack (i..e, the total size and used size).
anyone ?
thx
Noam
The total size of the stack will be the size of the stack you asked for when you created the thread ( or linked the program if it's the main thread ).
There are some preliminary references to getting the stack size for a thread pool in Windows 7 on MSDN ( QueryThreadpoolStackInformation ).
As an approximation, you can compare the address of a local variable with the address of another local variable further down the stack to get a measure of the amount us. I believe that how a program running in windows chooses to lay its local variables out in the virtual memory space windows allocates to a thread is up to the implementation of that language's runtime, rather than something that Windows really knows about; instead you get an exception when you attempt to access an address just below the memory allocated for the stack.
The other alternative to complicating your code with a check whether the stack has reached a limit is to add an exception handler for EXCEPTION_STACK_OVERFLOW, which will get called by the OS when it checks that the stack has reached its limit. There's an example here.
Related
Is there a way to programmatically change the stack size in Contiki?
I know on Linux systems I'm able to call:
ulimit -s SIZE
But I'm currently using Contiki as a flashed binary, and don't really have access to a traditional terminal. I've tried executing the command from C using system() and popen() calls to no avail.
Perhaps there's a CFLAG or LDFLAG I can leverage? Or modifying something in the makefile?
FYI I'm flashing the binary to a Texas Instruments cc2650, which has a 32 bit processor.
CC2650 does not have a MPU (Memory Protection Unit), which implies that no one checks for the boundaries of the stack region during runtime, which in turn implies that there is no way to "reserve" stack in the same sense that stack memory is reserved on Linux.
Essentially, if you keep allocating new things on stack, the stack will keep to grow even after it reaches other memory regions - usually the .data region, which contains dynamically allocated memory, if any, and static/global variables. The growth of stack will corrupt the memory in those other regions in a way that you might even fail to notice, leading to hard-to-find bugs.
There are a couple of things to do. One is to reserve bigger stack memory during compile time. This will not limit the stack region, but will limit the extent of the data region. To do that, change the CC2650 linker script in cpu/cc26xx-cc13xx/cc26xx.ld:
_Min_Stack_Size = 0x100; /* 256 bytes by default for the stack */
The other thing is to use Contiki-NG recent revisions, which have built-in stackoverflow checks. There is still no way to change the stack region size during runtime, but you will get an error if stack overflow happens.
When using an RTOS (ex FreeRTOS), we have separate stack spaces for each thread. So what about ISR (Interrupt Service Routines), does they have a separate stack in the memory? Or is this configurable?
If they don't have a stack where the local variables declared in ISR get stored?
I have the exact same question and a lot of searching leads me to this conclusion: the answer is dependent on your chip and how the OS you use configures that chip.
So looking at one of my favorite chips ARM Cortex-M3 (for which interrupts are a form of exception), the documentation at various spots reads:
Operating Modes
The Cortex-M3 supports Privileged and User (non-privileged) execution.
Code run as Privileged has full access rights whereas code executed as
User has limited access rights. The limitations include restrictions
on instruction use such as MSR fields, access to memory and
peripherals based on system design, and restrictions imposed by the
MPU configuration.
The processor supports two operation modes, Thread mode and Handler
mode. Thread mode is entered on reset and normally on return from an
exception. When in Thread mode, code can be executed as either
Privileged or Unprivileged.
Handler mode will be entered as a result of an exception. Code in
Handler mode is always executed as Privileged, therefore the core will
automatically switch to Privileged mode when exceptions occur. You can
change between Privileged Thread mode and User Thread mode when
returning from an exception by modifying the EXC_RETURN value in the
link register (R14). You can also change from Privileged Thread to
User Thread mode by clearing CONTROL[0] using an MSR instruction.
However, you cannot directly change to privileged mode from
unprivileged mode without going through an exception, for example an
SVC.
Main and Process Stacks
The Cortex-M3 supports two different stacks, a main stack and a
process stack. To support this the Cortex-M3 has two stack pointers
(R13). One of these is banked out depending on the stack in use. This
means that only one stack pointer at a time is visible as R13.
However, both stack pointers can be accessed using the MRS and MSR
instructions. The main stack is used at reset, and is always used in
Handler mode (when entering an exception handler). The process stack
pointer is only available as the current stack pointer when in Thread
mode. You can select which stack pointer (main or process) is used in
Thread mode in one of two ways, either by using the EXC_RETURN value
when exiting from Handler Mode or while in Thread Mode by writing to
CONTROL[1] using an MSR instruction.
And...
When the processor takes an exception, unless the exception is a
tail-chained or a late-arriving exception, the processor pushes
information onto the current stack. This operation is referred to as
stacking and the structure of eight data words is referred as the
stack frame. ...
Immediately after stacking, the stack pointer indicates the lowest
address in the stack frame
From the book "The Definitive Guide to the ARM Cortex-M3":
The MSP, also called SP_main in ARM documentation, is the default SP
after power-up; it is used by kernel code and exception handlers. The
PSP, or SP_process in ARM documentation, is typically used by thread
processes in system with embedded OS running.
Because exception handlers always use the Main Stack Pointer, the main
stack memory should contain enough space for the largest number of
nesting interrupts.
When an exception takes place, the registers R0–R3, R12, LR, PC,
and Program Status (PSR) are pushed to the stack. If the code that is
running uses the Process Stack Pointer (PSP), the process stack will
be used; if the code that is running uses the Main Stack Pointer
(MSP), the main stack will be used. Afterward, the main stack will
always be used during the handler, so all nested interrupts will use
the main stack.
UPDATE 6/2017:
My previous answer was incorrect, I have analyzed FreeRTOS for cortex processors and rewritten my answer to:
The standard FreeRTOS version for the Cortex-M3 does in fact configure and use both the MSP and PSP. When the very first task runs it modifies MSP to point to the first address specified in the vector table (0x00000000), this tends to be the very last word in SRAM, then it triggers a system call, in the system call exception handler it sets the PSP to the next task stack location, then it modifies the exception LR value such that "return to thread mode and on return use the process stack".
This means that the interrupt service routine (AKA exception handler) stack is grows down from the address specified in the vector table.
You can configure your linker and startup code to locate the exception handler stack wherever you like, make sure your heap or other memory areas do not overlap the exception handler area and make sure the area is large enough.
The answer for other chips and operating systems could be completely different!
To help ensure your application has appropriate space on the ISR stack (MSP),
here's some additional code to check actual ISR stack use. Use in addition to the checking you're already doing on FreeRTOS task stack use:
https://sourceforge.net/p/freertos/discussion/382005/thread/8418dd523e/
Update: I've posted my version of port.c that includes the ISR stack use check on github:
https://github.com/DRNadler/FreeRTOS_helpers
I try to understand the basics of concurrent programming in Go. Almost all articles use the term "address space", for example: "All goroutines share the same address space". What does it mean?
I've tried to understand the following topics from wiki, but it wasn't successful:
http://en.wikipedia.org/wiki/Virtual_memory
http://en.wikipedia.org/wiki/Memory_segmentation
http://en.wikipedia.org/wiki/Page_(computer_memory)
...
However at the moment it's difficult to understand for me, because my knowledges in areas like memory management and concurrent programming are really poor. There are many unknown words like segments, pages, relative/absolute addresses, VAS etc.
Could anybody explain to me the basics of the problem? May be there are some useful articles, that I can't find.
Golang spec:
A "go" statement starts the execution of a function call as an independent concurrent thread of control, or goroutine, within the same address space.
Could anybody explain to me the basics of the problem?
"Address space" is a generic term which can apply to many contexts:
Address spaces are created by combining enough uniquely identified qualifiers to make an address unambiguous (within a particular address space)
Dave Cheney's presentation "Five things that make Go fast" illustrates the main issue addressed by having goroutine within the same process address space: stack management.
Dave's qualifies the "address space", speaking first of thread:
Because a process switch can occur at any point in a process’ execution, the operating system needs to store the contents of all of these registers because it does not know which are currently in use.
This lead to the development of threads, which are conceptually the same as processes, but share the same memory space.
(so this is about memory)
Then Dave illustrates the stack within a process address space (the addresses managed by a process):
Traditionally inside the address space of a process,
the heap is at the bottom of memory, just above the program (text) and grows upwards.
The stack is located at the top of the virtual address space, and grows downwards.
See also "What and where are the stack and heap?".
The issue:
Because the heap and stack overwriting each other would be catastrophic, the operating system usually arranges to place an area of unwritable memory between the stack and the heap to ensure that if they did collide, the program will abort.
With threads, that can lead to restrict the heap size of a process:
as the number of threads in your program increases, the amount of available address space is reduced.
goroutine uses a different approach, while still sharing the same process address space:
what about the stack requirements of those goroutines ?
Instead of using guard pages, the Go compiler inserts a check as part of every function call to check if there is sufficient stack for the function to run. If there is not, the runtime can allocate more stack space.
Because of this check, a goroutines initial stack can be made much smaller, which in turn permits Go programmers to treat goroutines as cheap resources.
Go 1.3 introduces a new way of managing those stacks:
Instead of adding and removing additional stack segments, if the stack of a goroutine is too small, a new, larger, stack will be allocated.
The old stack’s contents are copied to the new stack, then the goroutine continues with its new larger stack.
After the first call to H the stack will be large enough that the check for available stack space will always succeed.
When you application runs on the RAM, addresses in RAM are allocated to your application by the memory manager. This is refered to as address space.
Concept:
the processor (CPU) executes instructions in a Fetch-Decode-Execute
cycle. It executes instructions in an applicaiton by fetching it to
the RAM (Random Acces Memory). This is done because it is very
in-efficient to get it all the way from disk. Some-one needs to keep
track of memory usage, so the operating system implements a memory
manager. Your appication, consists of some program, in your case this
is written in Go programming language. When you execute your script,
the OS executes the instructions in the above mentioned fashion.
Reading your post i can empathize. The terms you mentioned will become familiar to you as program more and more.
I first encountered these terms from the operating systems book, a.k.a the dinosaur book.
Hope this helps you.
On computer memory, say IA32, what is the range of stack in general? I know an address like 0xffff1234 is probably on the stack. Is it possible for the stack to grow to 0x0800abcd, for example? How about the heap? I know the heap address is always lower than the stack address, but what is normally its range? Also what is the area below heap?
The stack - The memory the program uses to actually run the program. This contains local variables, call-back data (for example when you call a function, the stack stores the state and place you were in the code before you entered the new function), and some other little things of that nature. You usually don't control the stack directly, the variables and data are destroyed, created when you move in and out function scopes.
The heap - The "dynamic" memory of the program. Each time you create a new object or variable dynamically, it is stored on the heap. This memory is controlled by the programmer directly, you are supposed to take care of the creation AND deletion of the objects there.
Thanks a lot!
Stack:
You can define the size of your stack in linking time.
As I know, windows app default stack size is 2MB.
You can change the size of stack in your project setting. But when App is built, stack size is fixed.
And OS would set guard page for stack overflow. If any operation try to access the guard page would trigger EXCEPTION.
Heap:
Default heap size i guess also can be changed in project settings.
Because in your App, you can create your own heap, or use CRT heap, Win32 heap. So there should be lots of heaps.
When your try to allocate memory, Heap Manager based on algorithm to allocate memory. If there's not enough memory, Heap Manager would apply for memory from Virtual Memory Manager. Until there's not enough memory in User Address Space, throw exception: Out of Memory.
There's several definitions: HeapNode, HeapSegment, LFH, LEA, BEA.
And you can use Windbg: !heap -s, !heap -a, these commands to check the structure of Windows Heap.
So I understand what a stack overflow is, when memory collides (and the title of this website) but what I do not understand is why new entries to the stack are in a decremental memory address. Why are they not in a random memory address, would it not make more sense so that memory collision is not an issue? I am guessing there is some sort of optimizing reason behind that?
** EDIT **
What I did not realize is a stack is given x amount of address space. Makes sense now but brings me to a follow-up question. Can I explicitly state how much memory I want to allocate to a stack?
"Memory collides" would better suit the term of "buffer overflow", where you write outside of the predestined space, but where it is likely to be within a different allocated memory block.
A stack overflow is not about writing outside of one's memory allocation into another memory allocation. It's just about writing outside of one's stack memory allocation. Most likely outside of the stack there's a guard memory page, that is not allocated for anything and which causes a fault on a read or write attempt.
And assigning a random address for each value pushed on the stack makes it hard to find data on the stack (and it's not a stack anymore). When the compiler or programmer knows that subsequent elements occupy subsequent addresses, then it's easy to compute those addresses just from the base pointer of the stack frame.
The answer to this question is probably complex, but basically stack operations are considered to be very primitive functions that the processor does as part of normal execution of code. (Saving return addresses and other stuff.)
So where do you put the memory management code? Where do you track the allocated addresses or add code to allocate new addresses? There really isn't anywhere to do this as these are basic operations performed by the processor itself.
Similar to the memory that holds the code itself, the stack is assumed to be setup before the code runs (and pointed to by the stack register). There really isn't any place to add complex memory management to stack memory. And so, yes, if not enough memory was provided, the stack will overflow.
Stack overflow is when you have used up all available stack space. The space available for the stack is, in most cases just an arbitrary limit chosen by the system designers. It is possible to alter this, but on modern systems, it's not really an issue - code that needs several megabytes of stack, unless the system is REALLY huge, is probably not correctly designed.
The stack grows towards zero from "custom" - it has to go in a defined direction or it would be very hard to follow what is going on, and lower adddress is just as good as higher address. It used to be that stack and heap grew towards each other, which would allow code that uses a lot of stack and not so much heap to work in the same amount of memory as something that uses a smaller amount of stack and a larger amount of heap. But these days, there is typically enough memory (space) that the heap can be defined to be somewhere completely separate from the stack. Instead the stack overflow is detected by having a region of "reserved" memory just at the top of the stack that is not usable - so the OS gets a "trap" for using memory that isn't available, and the application can be killed.