I'm trying to write a OS for my own use, I want to show a blank (black) screen with VGA output but I have some problems(questions):
Under FAT32, I have MBR bootloader to read the first sector of the virtual disk image generated by bximage from Bochs. Where (which sector) should I put the second compiled code that shows the black screen? How to do it with dd utility? My second compiled code file is 9 Bytes only.
Is VBR necessary?
How do I know where the data region (FAT32) starts and ends?
I rewrote the bootloader provided from this link.
My disk file specifications is:
20M,
CHS 40/16/63
In chronological order...
Originally there were no hard disks and (if you weren't using "BASIC in ROM") computers booted from a floppy disk. In this case the first sector of the volume (the floppy disk) contains the operating system's boot loader.
Not long after hard disks got added, and worked using a similar scheme (where the first sector of the volume/hard disk contains the operating system's boot loader).
However, people soon realised that using a whole "large" hard disk for a single volume is silly/inflexible; so a partitioning scheme was invented to split the hard disk into multiple volumes. In this case the first sector of the disk (the MBR) contains a partition table where one is marked as the "active" partition, and some code to "chain load" the first sector of the active partition (the boot loader). This became "extremely standard", then people extended it to support multiple different operating systems, and most boot managers support multiple operating systems using this method.
Note 1: I define "boot manager" as something you use to choose which OS to boot, and "boot loader" as something designed to boot the specific OS that was chosen. Ideally these have nothing to do with each other, the boot manager should have nothing to do with any OS, and the end user should be able to change the boot manager with anything they like without upsetting or effecting any OS or any boot loader. Sadly, (for Windows) Microsoft are hostile towards allowing multiple different operating systems to boot using simple, sane and well supported methods (including allowing multiple instances of the same version of Windows to be installed at the same time, which could be useful - e.g. one OS for your work stuff and a separate OS for your kids both installed on the same computer) and try to smother sanity with their own "boot.ini" idiocy that mostly just makes everything horrid for no benefit (other than giving Microsoft more control over what you do with your computer). Of course when the user is only installing one OS on the computer it's nice for the OS installer to (optionally, if and only if the user wants it - e.g. because they don't already have their own boot manager) provide and install a minimal MBR that doesn't nothing more than chain load the operating system's boot loader.
As time passed more devices got added. The first was network cards and the ability to boot from network. This is nothing like "boot from disk". Instead, the network card's ROM (after some negotiation with a DHCP server) downloads an entire "boot file" (which is not limited to 1 sector and can be 500 KiB if you like) from a server, then provides an API (which became known as the "PXE API") that the boot loader can use to access networking (e.g. send/receive packets, download more files using the TFTP protocol, etc).
The other type of device that got added was CD-ROM. For these, a new specification ("El Torito bootable CD-ROM specification") was created, partly so that you could have a boot catalogue with multiple entries for multiple architectures (e.g. one for "80x86 PC", one for "PowerPC", etc) and let the firmware choose the most appropriate boot loader for the computer being booted. For this there are 3 methods for PCs - emulate a floppy disk, emulate a hard disk, or "no emulation". The emulation options work the same as original "boot from disk" method (and use 512-byte sectors, etc), but are limited and slow and probably shouldn't be used for anything other than compatibility with legacy operating systems. For "no emulation" it's completely different to the original "boot from disk" method, firmware is supposed to load an entire "boot file" (which is not limited to 1 sector and can be 500 KiB if you like), and sectors will be 2048 bytes (and not 512 bytes).
Even later; UEFI got invented. For 80x86 PCs this comes in 2 flavours - 32-bit 80x86 and 64-bit 80x86. In theory you can have a 64-bit UEFI boot loader that switches to protected mode/32-bit and starts a 32-bit OS; and you can have a 32-bit UEFI boot loader that switches to long mode/64-bit and starts a 64-bit OS. However, 32-bit UEFI is very rare (a few old Apple Mac's and almost nothing else) and these computers are likely to also support "BIOS compatible boot"; and isn't worth supporting 32-bit UEFI for that reason. For UEFI in general, it loads and executes an entire file (from whatever the boot device was) and provides an API that the boot loader can use (e.g. to setup a video mode, get a memory map, load other file/s, etc).
Note 2: UEFI tries to make it so that boot works the same regardless of which type of device you're booting from. In practice this doesn't work very well and you'll probably want a different boot loader for CD (that accesses file/s on the CD itself and isn't restricted to a weeny FAT file system image) and a different boot loader for network (even if it's only to allow you to pass IP addresses to the OS and avoid repeating the slow DHCP stuff after the OS boots).
With UEFI a new partitioning scheme was also introduced (GPT or "GUID Partition Table"). This has multiple advantages and (for new operating systems being installed as the only OS on a computer) should probably be considered the default (and the old "MBR partitions" should probably be considered obsolete for compatibility with old operating systems only).
Mostly; for 80x86 you'll probably need 4 or more different boot loaders:
one for BIOS and un-partitioned disk devices (floppy)
one for BIOS and disk devices that were partitioned with "MBR partitions"
one for BIOS and disk devices that were partitioned with "GPT partitions"
one for BIOS and network boot/PXE
one for BIOS and "no emulation" CD boot
one for 64-bit UEFI disk
one for 64-bit UEFI CD-ROM
one for 64-bit UEFI network
Of course all of these cases are "different enough" that it's silly to try to have a generic boot loader that covers multiple different cases (and in cases where there are similarities things like "512-bytes only" restrictions are so limiting that you'll be doomed if you try).
I'd also "strongly recommend" having some kind of abstraction between boot loader and the rest of the OS (e.g. a "boot protocol" defined for the OS that describes how a boot loader sets things up, passes information to the OS and transfers control to the OS); such that none of the code in the entire OS needs to know or care what the firmware was (if it was BIOS or UEFI or something else, like maybe kexec()). This means that anyone can create more boot loaders (to support other cases and other devices); and (as long as everything complies with your abstraction's specification) the entire OS will work with the new boot loader/s without any changes.
Under FAT32, I have MBR bootloader to read the first sector of the virtual disk image generated by bximage from Bochs. Where (which sector) should I put the second compiled code that shows the black screen? How to do it with dd utility? My second compiled code file is 9 Bytes only.
This is mostly wrong. For "BIOS hard disk" you should have an MBR (that has nothing to do with the OS at all) and partitions, and your operating system's boot loader should begin in the first sector of the partition (and should be designed to use DS:SI to find the partition table entry that describes its partition, and dl to determine which device the partition is on).
Is VBR necessary?
For some cases (booting from UEFI, network, CD-ROM) a VBR doesn't make sense. For some cases (booting from BIOS hard disk or BIOS USB flash) it's "theoretically optional" but extremely recommended; because some BIOSes may not recognise it (especially for the USB flash case), and other operating systems will assume that the disk isn't formatted (and will tell their users that the disk needs to be initialised/partitioned, convincing the user that your OS is garbage and leading to the user accidentally or intentionally wiping your OS off the disk).
How do I know where the data region (FAT32) starts and ends?
For FAT; there's fields in the BPB ("BIOS Parameter Block", which is misnamed as it's mostly not used by the BIOS at all) in the first sector of the volume/partition that tell you things like how many reserved sectors there are, how many sectors are in each cluster, etc. Really, if you're going to use one of the world's worst file systems for inappropriate things (e.g. for an operating system's main partition where things like effective permissions/security and fault tolerance are sorely needed) then you'll need to learn everything about FAT32 so that you can write code to allow the OS to support it after boot.
Related
Assume there is an MCU(like a cypress PSOC4 chip which I'm using). It contains a flash memory(to store firmware) and a RAM(probably SRAM) inside the chip. I understand that even these two components need to be memory mapped in order for the processing unit to access them.
However, the flash memory and SRAM should be mapped every time the MPU is powered on, right?.
Then where is the configuration for memory map stored?
Is it somehow hardwired inside the MPU? Or is it stored in a separately hidden small piece of RAM?
I once thought that the memory map info should be located at the front of the firmware, but this doesn't make sense because the firmware is stored in the flash, and the MPU would have no idea where the flash is mapped to. So, I think this is a wrong idea.
By the way, is a memory map even configurable?
Yes hardwired in the mcu on boot, some mcus allow for remapping once up and running, but in order to boot the flash/rom has to be mapped to a known place, a sane design would also have the on chip sram mapped and ready to use on boot at a known location.
Some use straps (pins externally hardwired high or low) to manipulate how the mcu boots, sometimes that includes a different mapping. A single strap could for example choose between mapping a bootloader rom vs the user flash into the boot space of the processor. But that would be documented as with other mapping choices in the chip vendors documentation for the part.
Some mcus allow you to in software after boot move ram into the vector/exception table area so you can manipulate it at run time and not be limited to what was in the flash at boot. Some mcus are going so far as to have a mmu like feature, but I have a hard time calling those mcus as they can run in the hundreds of mhz, have floating point uints, caches, etc. Technically they are a SOC with ram and flash on chip, so classified as an MCU.
Your thinking is sane, the flash and sram mappings are in logic and at reset you can know where things will be. It is in the documentation for that product.
I understand that the computer loads the first sector of memory known as BIOS, which runs diagnostics on hardware and the proceeds to load the OS. I guess my question leans towards the hardware side. How does the computer know which memory to boot from (RAM, ROM, FLASH, etc). I understand the differences between memory and I understand computers boot from the hard drive, but Im attempting to make an 8 bit computer with a z80 microprocessor, which will need to boot from ROM or Flash memory. The only problem is that the processor reads only from whatever memory the address pins are connected to and there are no separate address pins for ram and rom. Its also impractical to run the system on rom or flash due to the much slower read/write time compared to ram. The z80 to the best of my knowledge doesnt have separate commands for reading from rom and ram, and it wouldnt matter even if it did because the ram will be blank upon powering up. How does a computer choose to read from rom only upon booting and then switch to ram once the OS has been loaded. Is it hardwired in using logic gates? And how does a computer choose to write to flash memory or a hard drive instead of ram once the OS has been loaded? Would flash memory be treated as a device? Or is this also hardwired into the motherboard using logic gates? Sorry for giving so much background, I just dont want you to waste your time explaining things Ive already grasped. Ive just researched this to a great extent and thought about it for hours on end and cant seem to figure it out, and everywhere Ive looked doesnt explain how the computer chooses which memory to read from, it just says that it does. Thanks
I'm not sure I'm answering what you are asking, but I'll give it a try.
Some computers (at least, IBM PC-compatible computers), after powering up, usually run this BIOS (Basic Input/Output System) program. For this to happen, to the best of my knowledge, the hardware must make the jump to this code, and this code must be accessible (that is, mapped) from the physical memory, since that's where the CPU will execute code from. So, a physical address space with some read-only areas where this code is hard-wired to would do the trick.
Once the BIOS code is running, it can select how to proceed next. It can copy a sector from a hard disk to memory, (or a bunch of data from a Flash drive) and then jump to it, or whatever. That's up to the BIOS writer.
I will try to explain the Pentium boot up process very briefly.
On the flash ROM mounted on the Motherboard. there is a small program called BIOS (basic input, output system). After pressing the power button the BIOS program is executed.
The BIOS contains low level software that performs the following operations :-
checks how much RAM is installed and if all other PCI and ISA buses peripherals are connected.
it checks if all IO devices are connected.
scans a list of boot devices and selects the boot devices based on BIOS configurations setup earlier by the user.
once the boot devices is selected. the first sector from the boot device is read into memory and executed. it contains a simple program which examines the partition table and selects the Active one (Holding the OS). The secondary bootloader is read from that partition. this loader then reads the OS from the partition into the memory and runs it. After running, the OS asks the BIOS for the configuration info for each device and configure the new devices (those have no stored configurations). after all devices configurations are set. they are delivered to the kernel. Then it initializes tables, background boot up processes and starts login program or GUI.
What happens if one software goes to the address 0x1500 and sets the value 6 there, and then another software goes to the same address and reads it, Will read 6? Will it always be like that? What are the mechanisms at the OS level and at the Processor level that are in charge of that?
Well it depends...
it depends on
Processor (8051/x86/x386...)
Processor mode (real/protected)
Operating System (supports protected mode?)
Process mode (user mode/kernel mode)
Address type (shared/private)
Address access privileges (allowed/denied)
now here are some combinations and their outcomes
----------------------------------------------------------------------
PROCESSOR MODE PROCESS MODE ADDRESS TYPE PRIVILEGE OUTCOME
======================================================================
Real Kernel -- -- r/w
Protected Kernel shared ok r/w
Protected user shared ok r/w
Protected user private -- r/w with diff result
Protected Kernel shared no error
Protected user shared no error
....
======================================================================
NOTE: in real mode you can always talk with each other. In protected mode it depends. some (old) processors dont support protected mode (8051/8086 etc.) some new processor may be in one of the two modes - either real or protected. in protected mode, you can talk if the OS allows. if not, you get error. the protected mode could be paged or non-paged. paged mode supports private addresses (with same value), non-paged mode does not. two diff. process can read/write same location in their private address yet they can get diff. values.
as you can see, it is really complicated.
Also,
Normal software work in protected + usermode + private memory. hence two diff software can both perform r/w at address 0x1500 and yet both will have independent values. what that software wrote is what it will read.
This used to be possible by default in early operating systems, such as Windows 95. This was very problematic.
Chips Challenge was a game for Windows 95 that had a bug in it - if you simultaneously collected a key and died on the same tile/step, its logic would go wrong and it would write to the wrong spot in memory. Since every program shared the same memory space, it could now be writing to memory that another program was using - causing other programs to crash and possibly making the operating system unstable and requiring a reboot.
In modern operating systems, each program now has its own virtual memory space - what each program thinks the memory is laid out as, is actually a fiction generated by the CPU and OS together, such that all memory reads and writes, if they are done out of bounds, are intercepted and prevented, and the real places in memory that the program is writing and reading to are unknown to the program (which also allows memory to be 'paged' out from RAM onto the hard disk, and paged back in in a different place when it is next detected to be needed).
http://en.wikipedia.org/wiki/Paging
I am aware that there are programs out there like lojack for laptops that get installed on the BIOS, but I'm still a little confused. When reading about lojack, it seems to me that they can't fully located the laptop's location until the user logs in and tries to access the internet. So I'm thinking that it's a BIOS application so that it wouldn't matter if the thief reformats the HD.
So my question is, does anyone have any ideas of how an internet enables BIOS application would be written. I'm not looking for full answers -- just ideas or resources to get started. For example, is such a thing written in assembly? Once one such app is written, how does it get transfered to the BIOS.
Does the BIOS program itself recognize that there is an internet connection (when the thief logs on to the OS). Or upon logon, does additional processes get spawned? Are there any resources/websites that anyone can direct me too?
You didn't mention whether you were interested in legacy BIOS or EFI BIOS, but I would mention that with EFI there is the capability of writing EFI applications. See Intel Press:
Harnessing the UEFI Shell
The EFI Application toolkit comes with a complete TCP/IP network stack:
http://www.intel.com/technology/efi/toolkit_overview.htm
More at tianocore.org
Regarding "LoJack"-style solutions, one of the providers of this technology is Absolute Software's Computrace product.
Basically there are 3 components: 1) a software component that runs in the OS; 2) a BIOS component which is baked into the system BIOS (accomplished via Absolute working with the PC vendor); 3) servers at Absolute software that talk to the PC.
For more information on how it works visit:
http://www.absolute.com/en/company/Computrace-Persistence.aspx
(see especially the demo video on this site)
To learn something about BIOS, one good source is coreboot.org. It is an open source BIOS (or firmware) and support some physical machines.
Legacy BIOS is written in assembly language, but new generations, such as UEFI or coreboot, are written mostly in C language. BIOS program is stored in the ROM, and executed by the CPU automatically.
The BIOS program itself does not access the internet or perform any of the advertised functions. The LoJack addition to the BIOS firmware is a file copying/patching utility - at boot up it can check the harddrive for a copy of Windows and proceed to silently install/repair the LoJack service if it has been removed. The service itself includes several measures to lower it's profile and prevent itself from being disabled (similar to how many trojans and malware run several processes that each restore the other if one is disabled or killed).
The LoJack BIOS program can't do anything if a unsupported operating system (like Linux) is installed after the harddrive is wiped.
Is WM operating system protects process memory against one another?
Can one badly written application crash some other application just mistakenly writing over the first one memory?
Windows Mobile, at least in all current incarnations, is build on Windows CE 5.0 and therefore uses CE 5.0's memory model (which is the same as it was in CE 3.0). The OS doesn't actually do a lot to protect process memory, but it does enough to generally keep processes from interfering with one another. It's not hard and fast though.
CE processes run in "slots" of which there are 32. The currently running process gets swapped to slot zero, and it's addresses are re-based to zero (so all memory in the running process effectively has 2 addresses, the slot 0 address and it's non-zero slot address). These addresses are proctected (though there's a simple API call to cross the boundary). This means that pointer corruptions, etc will not step on other apps but if you want to, you still can.
Also CE has the concept of shared memory. All processes have access to this area and it is 100% unprotected. If your app is using shared memory (and the memory manager can give you a shared address without you specifically asking, depending on your allocation and its size). If you have shared memory then yes, any process can access that data, including corrupting it, and you will get no error or warning in either process.
Is WM operating system protects process memory against one another?
Yes.
Can one badly written application crash some other application just mistakenly writing over the first one memory?
No (but it might do other things like use up all the 'disk' space).
Even if you're a device driver, to get permission to write to memory that's owned by a different process there's an API which you must invoke explicitly.
While ChrisW's answer is technically correct, my experience of Windows mobile is that it is much easier to crash the entire device from an application than it is on the desktop. I could guess at a few reasons why this is the case;
The operating sytem is often much more heavily OEMed than Windows desktop, that is the amount of manufacturer specific low level code can be very high, which leads to manufacturer specific bugs at a level that can cause bad crashes. On many devices it is common to see a new firmware revision every month or so, where the revisions are fixes to such bugs.
Resources are scarcer, and an application that exhausts all available resources is liable to cause a crash.
The protection mechanisms and architecture vary quite a bit. The device I'm currently working with is SH4 based, while you mostly see ARM, X86 and the odd MIPs CPU..