I am working in software firm where hardware independent coding is done on the Network Chipsets and fully Multigthreading coding implemented and various buffers(CRU Buffer, Linear Buffer) are handled and memory (stack memory) is optimally used. And IPC done via Message queues. And Multiple Locks, Semaphores are used for concurrency mechanisum. Now i will be assigned to new development project, where i have to understand and have to develop new features in next one month. I am feeling like middle of the Amazon Jungle :).
=> I am in beginning level in OS concepts. I feel like intermediate level in C language. So expecting, suggestion for "Materail/Book which could help me to improve/concrete my OS skills"
i saw OS Book by Abraham Silberschatz and Modern Operating Systems by Tanenbaum - 3rd Edition. Both are looking big and covers all corners of operating system. I thought to study that book steadily and slowly for future referencee.
==> Now i am looking for the Network materials/books which explaining the "Main concepts" in the detailed manner. For example i have seen virtual memory concepts in one online material where clearly virtual memory explained.
Example abour virtual memory from that material:
amesmol#aubergine:~/test> objdump -f a.out
a.out: file format elf32-i386 architecture: i386, flags 0x00000112: EXEC_P, HAS_SYMS, D_PAGED start address 0x080482a0
explanation:
Notice the start address of the program is at 0x80482a0.Program thinks like where its starting address is actual physical address. But it is a virtual address space. Its original starting address at physical memory location 0x1000000.
As like this( correct point and example), could you people suggest good materials for the OS concepts ( Process Management, Memory Management, IPC)?
Can you also suggest the ways to improve/concrete this skills? (suggest either what kind of mini homework project i can do, etc..)
Thanks in advance
if you are working on projects you have to go over books you mentioned as soon as possible for theoretical explanations, concepts and terminologies. after that,even along with your reading, i suggest you to go to university websites to get hand on skills for small projects. some suggested links are as follows
http://www.eecg.toronto.edu/~lie/Courses/ECE344/
http://web.stanford.edu/~ouster/cgi-bin/cs140-winter13/pintos/pintos.html#SEC_Contents
http://www3.cs.stonybrook.edu/~porter/courses/cse624/f13/project.html
(JOS implementation. very helping instructors if you send them specific queries)
http://www.brokenthorn.com/Resources/OSDev7.html
http://www.osdever.net/bkerndev/Docs/intro.htm
(above two links are not university link but as a beginner I recommend this to start with )
apart from above, Lion's commentary on Unix code with line number reference must be in your reading to understand the implementation of small scale OS
Related
I'm wondering if there is a good resource for FORTH implementations on recent SOCs.
I'm mostly interested in bare metal versions, something that can sit net to RTOS on an ESP32 or RISC-V for instance (so gforth might not be ideal).
And particularly, I'm looking at least at a version that can do networking (e.g. via WIFI, ideally via a source network stack implementation, which in RTOS might not be too hard)
Sadly, it seems pretty hard to tickle useful info out of Google – it mostly seems to think I can't spell fourth. Many results I do find seem outdated, and or seem very commercial; and it feels wrong to pay licensing fees for a network stack in the age of micropython.
This Raspberry Pi JonesFORTH O/S looks pretty promising.
I wouldn't mind doing a bit of porting.
Even if your question could be interpreted as opinion-based and might be looking for resources only, I am going to provide two links to living and active maintained projects which might fit your requirements.
I'm wondering if there is a good resource for FORTH implementations on recent SOCs. I'm mostly interested in bare metal versions
You may have a look into
AmForth
... for the Atmel AVR8 Atmega micro controller family and some variants of the TI MSP430. The RISC-V CPU (32bit) is currently beeing worked on.
and
Mecrisp, Stellaris
an implementation of a standalone native code Forth for MSP430 microcontrollers
or Quintus
a rewrite of classic Mecrisp-Stellaris with almost the same look-and-feel for RISC-V architecture, RV32I, RV32IM or RV32IMC flavour, and it includes support for MIPS M4K cores. FPGAs are conquered by using FemtoRV32 softcores.
With both projects it is possible to achieve progress quite fast and easy depending what you try to achieve.
... can do networking (e.g. via WIFI, ideally via a source network stack implementation ... I wouldn't mind doing a bit of porting.
For a more detailed answer and more guidance this part would need more information and some clearance.
do you have any experience with SD Erlang project?
There seems to be implemented many interesting concepts regarding the comm mesh optimalizations and I'm just curious if some of you used those in production already or in some real project at least.
SD erlang repo
Thanks!
The project has finished a week ago. The main ideas behind SD Erlang are reducing the number of connections Erlang nodes maintain while keeping transitivity and common namespace for groups of nodes. Benchmarks that we used (Orbit, Ant Colony Optimization (ACO), and Instant Messenger) showed very promising results. Unfortunately, we didn't have enough human resources to refactor Sim-Diasca simulation engine. So, no, SD Erlang hasn't been used yet in a real application.
At the moment we are writing up the last deliverable that will provide an overview of what has been achieved. It will appear here in a few weeks (D6.2). In general we are happy with the results we get using SD Erlang, so there are plans for a follow up project to continue to work on it but currently this is work in progress.
This is not a direct answer but I will use SD-Erlang in a embedded application which needs to scale to hundreds of nodes (small embedded CPUs). From what I have seen its ready to be tried out in a real application. To furtehr evaluate lets consider the alternatives:
You have only a few distributed nodes: then you probably don't need it and can just connect all the nodes and for name registry use either the global module (slow but sturdy) or gprocwith the new locks_leader branch which avoids the quite broken gen_leader which so far prevented using gproc in distributed mode in production.
You need many nodes (how many depends on your hardware and requirements but you start to get into interesting territory with > 70 nodes)
Use SD-Erlang and fix whatever problems you encounter in production, or at least report them. It certainly solves a lot of the problems you get with normal Erlang distribution
Roll your own solution either with playing with different cookie values or with hidden nodes: hint you can set different cookie values for different peer nodes. But then you need to roll your own global name registry and management code: looks like a variant of Greenspuns 10th rule or closer to Erlang Virdings 1st rule : you probably will result in implementing half of SD Erlang yourself.
Don't use Erlang distribution at all. That seems to be the industry standard that for anything involving more nodes or crossing data-centers you shouldn't use Erlang distribution at all but run your own protocols. My personal opinion is to rather fix Erlang Distributions problems than just ditch it. Its much too useful and time saving when it works for a use case to just give up on it. And I see SD-Erlang as being the fix for the "too many nodes" problem, its at least the right starting point.
I am new to Computer Architecture.Can somebody help me in understanding the use of limited registers in processing of several complex applications. My question is there are fixed number of registers(For Example :: 80386 contains a total of sixteen registers) that are of interest to the applications programmer.
What happens if we want more registers( for example: to accommodate increased Stack size), are the addresses and data from registers written back to main memory ?.In multitasking environment, are the registers data and addresses of different applications moved from between main memory and back to register for processing ?
Does operating systems have special registers which does not interfere with application general purpose registers ?
And suggest any good resource for understanding such concepts for starters ?
Registers are the fastest memory in a computer. The instruction set of any particular cpu is written specifically for the register architecture. You are right that data/addresses must be backed to memory as more register space is used.
As far as a multitasking system goes, the scheduler generally has to save the execution context between tasks. This context involves the current state of the registers as well as other status bits (depending on the cpu).
A good first step would be to learn assembly programming. It is so close to the hardware that you will learn all of this stuff thoroughly. Once you have that, pick up an operating systems book to see how it is done at a higher level. Depending on your commitment (and curiosity), you could also read some of the source code for smaller real-time operating systems, such as FreeRTOS. Reading up on 8-bit microcontroller architectures is also nice, since they are simple. For example, AVR or HC08 are pretty straightforward architectures to learn. All of the info is free; you just have to read it.
Enjoy.
I keep seeing "bootstrapping" mentioned in discussions of application development. It seems both widespread and important, but I've yet to come across even a poor explanation of what bootstrapping actually is; rather, it seems as though everyone is just supposed to know what it means. I don't, though. Near as I can figure, it has something to do with initialization tasks required of an application upon launch, but I could be completely wrong about that. Can anyone help me to understand this idea?
"Bootstrapping" comes from the term "pulling yourself up by your own bootstraps." That much you can get from Wikipedia.
In computing, a bootstrap loader is the first piece of code that runs when a machine starts, and is responsible for loading the rest of the operating system. In modern computers it's stored in ROM, but I recall the bootstrap process on the PDP-11, where you would poke bits via the front-panel switches to load a particular disk segment into memory, and then run it. Needless to say, the bootstrap loader is normally pretty small.
"Bootstrapping" is also used as a term for building a system using itself -- or more correctly, a predecessor version. For example, ANTLR version 3 is written using a parser developed in ANTLR version 2.
An example of bootstrapping is in some web frameworks. You call index.php (the bootstrapper), and then it loads the frameworks helpers, models, configuration, and then loads the controller and passes off control to it.
As you can see, it's a simple file that starts a large process.
The term "bootstrapping" usually applies to a situation where a system depends on itself to start, sort of a chicken and egg problem.
For instance:
How do you compile a C compiler written in C?
How do you start an OS initialization process if you don't have the OS running yet?
How do you start a distributed (peer-to-peer) system where the clients depend on their currently known peers to find out about new peers in the system?
In that case, bootstrapping refers to a way of breaking the circular dependency, usually with the help of an external entity, e.g.
You can use another C compiler to compile (bootstrap) your own compiler, and then you can use it to recompile itself
You use a separate piece of code that sets up the initial process without depending on any functions provided by the OS
You use a hard-coded list of initial peers or a hard-coded tracker URL that supplies the peer list
etc.
See on the Wikipedia article on bootstrapping.
There is a section and links explaining what it means in Computing. It has four different uses in the field.
Here are some quotes, but for a more in depth explanation, and alternative meanings, consult the links above.
"...is a technique by which a simple computer program activates a more complicated system of programs."
"A different use of the term bootstrapping is to use a compiler to compile itself, by first writing a small part of a compiler of a new programming language in an existing language to compile more programs of the new compiler written in the new language."
In the context of application development, "bootstrapping" usually comes up when talking about modular and/or auto-updatable software.
Rather than the user downloading the entire app, including features he does not need, and re-downloading and manually updating it whenever there is an update, the user only downloads and starts a small "bootstrap" executable, which in turn downloads and installs those parts of the application that the user needs. Additionally, the bootstrap component is able to look for updates and install them each time it is started.
Alex, it's pretty much what your computer does when it boots up. ('Booting' a computer actually comes from the word bootstrapping)
Initially, the small program in your BIOS runs. That contains enough machine code to load and run a larger, more complex program.
That second program is probably something like NTLDR (in Windows) or LILO (in Linux), which then executes and is able to load, then run, the rest of the operating system.
For completeness, it is also a rather important (and relatively new) method in statistics that uses resampling / simulation to infer population properties from a sample. It has its own lengthy Wikipedia article on bootstrapping (statistics).
Boot strapping the dictionary meaning is to start up with minimum resources. In the Context of an OS the OS should be able to swiftly load once the Power On Self Test (POST) determines that its safe to wake up the CPU. The boot strap code will be run from the BIOS. BIOS is a small sized ROM. Generally it is a jump instruction to the set of instructions which will load the Operating system to the RAM. The destination of the Jump is the Boot sector in the Hard Disk. Once the bios program checks it is a valid Boot sector which contains the starting address of the stored OS, ie whether it is a valid MBR (Master Boot Record) or not. If its a valid MBR the OS will be copied to the memory (RAM)from there on the OS takes care of Memory and Process management.
As the question is answered. For web develoment.
I came so far and found a good explanation about bootsrapping in Laravel doc. Here is the link
In general, we mean registering things, including registering service
container bindings, event listeners, middleware, and even routes.
hope it will help someone who learning web application development.
Bootstrapping has yet another meaning in the context of reinforcement learning that may be useful to know for developers, in addition to its use in software development (most answers here, e.g. by kdgregory) and its use in statistics as discussed by Dirk Eddelbuettel.
From Sutton and Barto:
Widrow, Gupta, and Maitra (1973) modified the Least-Mean-Square (LMS)
algorithm of Widrow and Hoff (1960) to produce a reinforcement
learning rule that could learn from success and failure signals
instead of from training examples. They called this form of learning
“selective bootstrap adaptation” and described it as “learning with a
critic” instead of “learning with a teacher.” They analyzed this rule
and showed how it could learn to play blackjack. This was an isolated
foray into reinforcement learning by Widrow, whose contributions to
supervised learning were much more influential.
The book describes various reinforcement algorithms where the target value is based on a previous approximation as bootstrap methods:
Finally, we note
one last special property of DP [Dynamic Programming] methods. All of them update estimates
of the values of states based on estimates of the values of successor
states. That is, they update estimates on the basis of other
estimates. We call this general idea bootstrapping. Many reinforcement
learning methods perform bootstrapping, even those that do not
require, as DP requires, a complete and accurate model of the
environment.
Note that this differs from bootstrap aggregating and intelligence explosion that is mentioned on the wikipedia page on bootstrapping.
I belong to the generation who flipped switches to enter a boot program. In the early 1980s, I worked on a microcomputer called Micro-78, developed by Electronics Corporation of India Ltd (ECIL). It was a sort of clone of Altair 8800. I distinctly remember what happens when a small boot program was entered using the toggle switches and executed by pressing a button. The program reads a second boot program contained in the 1st track of the floppy disk and overwrites it on itself in such a way that the second boot program starts executing to load a disk operating system. I think the term "bootstrap" refers to this process of the first boot program reading and overwriting the second boot program on itself, in a way "pulling itself up" with the additional functionality of the second boot program. That may be the origin of the original meaning of "the bootstrap program".
IMHO there is not a better explanation than the fact about How was the first compiler written?
These days the Operating System loading is the most common process referred as Bootstrapping
In terms of it in regards to using the popular Twitter Bootstrap I feel like this type of bootstrapping is the action of integrating a modular component into a Web application without the Web application having to even acknowledge the modular component exists until it needs it or references it.
The developer can seamlessly integrate a default copy of the CSS Twitter Bootstrap theme by simply loading (referencing) it into the Web application. Vuola! Then you may need to override some of these changes, but you can do so in such a way that the resource/component is untouched and completely reusable.
This same concept is how Web Devs implement jQuery APIs and so on, but it's not really expressed by Devs as bootstrapping per se. What it does is it improves flexibility and reusability while allowing the isolation of different components/resources of an app to reside freely either on the same server/s or possibly on a CDN.
NOTE: In computing bootstrapping deals with the MBR and in UNIX it requires a special bootloader or manager which is a small program in ROM that loads the OS into RAM. If you think about it the same concept takes places in the action of the bootstrap loader checking the MBR and loading the OS based on this table which occurs without the OS having any idea that this takes place.
Bootstrap file is responsible for loading contents of main file. It is a wrapper around main file. This way we can catch errors if loading of file was unsuccessful for some reason.
As a humble beginner in the world of programming, and flicking through all the answers here after seeing this word used a lot in apparently slightly different ways in different places, I found reading the Wikipedia page on Bootstrapping (duh! I didn't think of it either at first) is very informative to understand differences in use of this word. Could it be......on extremely rare occasions......Wikipedia might even have better explanations of certain terms than....(redacted)? Will they bring in rep points on Wikipedia though?
To me, it seems all the meanings something to do with: start with something as simple as possible Thing1, make something slightly more complex with that Thing2, and now you can use Thing2 to do some kind of tasks more efficiently and quickly than you could originally with Thing1. Then repeat from Thing2 to Thing 3 ad infinitum...
I see it as closely connected to both biological evolution and 'Layers of Abstraction' (newbies like me see, ahem, Wikipedia, cough) - the evolution from 1940's computers with switches, machine code, Assembly, C, Python, AIs you can give all kinds of complex instructions to like "make the %4^% dinner to my default &^$% requirements and clean the floor you %$£"#:~" in drunken slang English or Amazon tribal dialect without them 'raising an exception' (for newbies again...you guessed it) - missed out lot of links there due to simple ignorance.
Then in certain specific software meanings:
Meaning1: Thing1 is used to load latest version of Thing2 (because of course Thing2 will be bigger than Thing1, just as Thing3 will be be bigger than Thing2).
Meaning2: Thing1 is a lower level language (closer to 1001011100....011001 than print("Hello, ", user.name)) used to write a little bit of the higher language of Thing2, then this little bit of Thing2 is used to expand Thing2 itself from baby vocabulary level towards adult vocabulary level (Thing2 starts to be processed, or to use correct technical term 'compiled', by the baby version of itself (it's a clever baby!), whereas the baby version of Thing2 itself could of course only be compiled by Thing1, cause it can't exist before it exists, right duh!), then child version of Thing2 compiles Surly Teenager version of Thing2, at which point programming community decides whether Surly Teenager's 'issues' (software term and metaphor term!) are worth spending enough time resolving to be accepted long term, or to abandon them to (not sure where to take the analogy here).
If yes, then Thing2 has 'Bootstrapped' itself (possibly a few times) from babyhood to adulthood: "the child is the father of the man" (Wordsworth, suggest don't try looking up the quote or the author on Stack Overflow).
I want to go backwards and learn more about how compilers, processors and memory operate on my programs. I am also interested in the physics on which all of this depends. Any good references or books would be appreciated...
For compilers, the ultimate book is the Dragon Book, aka Principles of Compiler Design. It is a bit heavy going at first, but it becomes easier with each pass you make at it. It's a classic and should be read by all serious students of programming/computer science.
My first suggestion was going to be Code which has been suggested already. A better, but harder, book on the subject of processors is Computer Organization & Design by Hennessey & Patterson. You might look for an older edition on Amazon or Half.com. They'll be a lot cheaper and have basically the same information.
These will both teach you the basics of how a processor works, assembly language, etc. This will help you understand how your program will be interpreted and thus, what sort of performance bottlenecks might exist based on your design.
It sounds to me like you want to get a Computer Science degree. :-)
The simplest introduction , and very well written is Code by Charles Petzold
alt text http://ecx.images-amazon.com/images/I/31t68r9K28L._BO2,204,203,200_PIsitb-sticker-arrow-click,TopRight,35,-76_AA240_SH20_OU01_.jpg
I don't normally think of physics and compilers at the same time.
This link might get you thinking.
Write a simple emulator for something like C64. By thinking of how to emulate the CPU, Memory and other Chips you will learn how these simple machines worked. Today's computers pretty much operate in the same manner with a lot more complexity but the basic idea is the same.
Your emulator doesnt have to be super fast - try and get the C64 welcome message up and if you did your code right - you should be able to POKE and write basic programs and have them work :)
10 PRINT "DONT LISTEN TO ME - APPLE //E's RULE"
20 GOTO 10
I would suggest learning enough assembler to be able to write a simple Hello World program in assembler.
This will teach you exactly how the CPU works at the register and memory level.
It will also give you a good introduction into how source code is converted to object files and how the linker brings together all these object files together to create a working program.
Pick up any of the Hennessy & Patterson books.
Computer Architecture - A Quantitative Approach or Computer Organization & Design - The Hardware Interface
Feynman has a nice bit on the Physics of Computation:
which addresses the second part of your question.
Pick up a book on "Computer Organization" or "Computer Architecture" on Amazon. This is what we used when I was in college. It's not too thick, and will give you the basics, from the gate level all of the way up to how memory is organized and programs are written. If, after this, you want to look deeper into the physics, then you'll want to pick up a book on semiconductor physics. (But if I were you I'd just start by looking up "logic gate", "diode", and "transistor" on wikipedia!)
If you really want to get into the physics of processors/ IC's, you need to have a solid electromagnetics/circuit analysis background. It's certainly not easy stuff, and won't really make you a better programmer. If you're really interested, borrow some books from an EE friend!
An abstraction up from that is the gate level, which programmers can usually grok since it's all about logic.
I made a similar resolution after reading Steve Yegge's recommendation to learn about compilers. I've been really enjoying Nisan and Schocken's The Elements of Computing Systems: Building a Modern Computer from First Principles which is first a textbook, second, a one-semester course for freshmen or seniors, and third, a website providing the lectures and cross-platform emulation-simulation software: http://www1.idc.ac.il/tecs/ (TECS)
The topics go from NAND (programming digital logic circuits in HDL), up through flip-flops, ALUs and registers, assemblers, parsing and compilers, operating systems, and GUIs. You, the student, implements every one of these topics on the hardware simulator or software emulator provided on the website. To me, it is a celebration of human ingenuity that this range of ideas can be so deeply covered in 3--4 months by undergrads. One of the authors/professors gave a Google Tech Talk on the subject that's worth checking out if you have an hour to spare away from their course.
I can't recommend an equally-compelling resource for the physics of computation, but I can perfunctorily say that my electrical engineering department's first two core EE courses students could take (simultaneously if they chose) were Circuits 1 and Semiconductors 1. The former dealt with voltage, current, inductors, capacitors, op amps, etc.---while the latter dealt with quantum mechanics, crystals, doping, charge carriers, etc., and most directly spoke to the physics of digital processing and storage devices. It's difficult for me to imagine a hands-on simulation environment like TECS for such a topic, though, making it more appropriate for academic than hobbyist/professional study?