I'm doing a hobby OS project and I an trying to get Virtual Memory set up. I had another project in an x86 architecture working with Page Tables but I am now learning ArmV8 now.
Now, I now that the maximum amount of bits used for addressing is 48[1]. The last 12 to 16 bits are used "as-is" to index within the selected region (depending on which granule size is selected[2]).
I just don't understand how we get those intermediate bits. Obviously the documentation is showing that intermediate tables are used[3] but it is quite unclear on how those tables are used.
In the first half of the following image, we see translation of an address with 4k granules and using 38 address bits.
I can't understand this image in the slightest. The "offsets", for example bits 38 to 30 point to an entry in the L1 table. How and where is this table defined ?
What I think is happening is, this a 12+8+8+8 address translation scheme. Starting from the right, 12 bits to find an offset within a 4096 block of memory. Right of that is 8 bits for L3, meaning that L3 indexes 256 blocks of 4096 bytes (1MB). Right of this, L2, has 8 bits also so 256 entries of (256*4096), totalling 256MB per L2 entry. Right of L2 is L1 with also 8 bits, 256 entries of 256MB means the total addressable memory is 64GB of physical RAM.
I don't think this is correct because that would only allow a 1:1 mapping of memory. Each table descriptor needs to carry some access flags and what not. Thus going back to the question of: how are those table defined. Each offset section is 8 bits and that's not enough to contain the address of a translation table.
Anyway, I am completely lost. I would appreciate if someone could give me a "plain english" explanation of how a translation table walk is done ? A graph would be nice but probably too much effort, I'll make one and share if after to help me synthesize the information. Or at least, if someone has one, a link to a good video/guide where the information isn't totally obfuscated ?
Here is the list of materials I have consulted:
https://developer.arm.com/documentation/den0024/a/The-Memory-Management-Unit/Translating-a-Virtual-Address-to-a-Physical-Address
https://forums.raspberrypi.com/viewtopic.php?t=227139
https://armv8-ref.codingbelief.com/en/chapter_d4/d42_4_translation_tables_and_the_translation_proces.html
https://github.com/bztsrc/raspi3-tutorial/blob/master/10_virtualmemory/mmu.c
[1]https://developer.arm.com/documentation/den0024/a/The-Memory-Management-Unit/Translation-tables-in-ARMv8-A
[2]https://developer.arm.com/documentation/den0024/a/The-Memory-Management-Unit/Translation-tables-in-ARMv8-A/Effect-of-granule-sizes-on-translation-tables
[3]https://developer.arm.com/documentation/den0024/a/The-Memory-Management-Unit/Translating-a-Virtual-Address-to-a-Physical-Address
The entire model behind translation tables arises from three values: the size of a translation table entry (TTE), the hardware page size (aka "translation granule"), and the amount of bits used for virtual addressing.
On arm64, TTEs are always 8 bytes. The hardware page size can be one of 4KiB, 16KiB or 64KiB (0x1000, 0x4000 or 0x10000 bytes), depending on both hardware support and runtime configuration. The amount of bits used for virtual addressing similarly depends on hardware support and runtime configuration, but with a lot more complex constraints.
By example
For the sake of simplicity, let's consider address translation under TTBR0_EL1 with no block mappings, no virtualization going on, no pointer authentication, no memory tagging, no "large physical address" support and the "top byte ignore" feature being inactive. And let's pick a hardware page size of 0x1000 bytes and 39-bit virtual addressing.
From here, I find it easiest to start at the result and go backwards in order to understand why we arrived here. So suppose you have a virtual address of 0x123456000 and the hardware maps that to physical address 0x800040000 for you. Because the page size is 0x1000 bytes, that means that for 0 <= n <= 0xfff, all accesses to virtual address 0x123456000+n will go to physical address 0x800040000+n. And because 0x1000 = 2^12, that means the lowest 12 bytes of your virtual address are not used for address translation, but indexing into the resulting page. Though the ARMv8 manual does not use this term, they are commonly called the "page offset".
63 12 11 0
+------------------------------------------------------------+-------------+
| upper bits | page offset |
+------------------------------------------------------------+-------------+
Now the obvious question is: how did we get 0x800040000? And the obvious answer is: we got it from a translation table. A "level 3" translation table, specifically. Let's defer how we found that for just a moment and suppose we know it's at 0x800037000. One thing of note is that translation tables adhere to the hardware page size as well, so we have 0x1000 bytes of translation information there. And because we know that one TTE is 8 bytes, that gives us 0x1000/8 = 0x200, or 512 entries in that table. 512 = 2^9, so we'll need 9 bits from our virtual address to index into this table. Since we already use the lower 12 bits as page offset, we take bits 20:12 here, which for our chosen address yield the value 0x56 ((0x123456000 >> 12) & 0x1ff). Multiply by the TTE size, add to the translation table address, and we know that the TTE that gave us 0x800040000 is written at address 0x8000372b0.
63 21 20 12 11 0
+------------------------------------------------------------+-------------+
| upper bits | L3 index | page offset |
+------------------------------------------------------------+-------------+
Now you repeat the same process over for how you got 0x800037000, which this time came from a TTE in a level 2 translation table. You again take 9 bits off your virtual address to index into that table, this time with an value of 0x11a ((0x123456000 >> 21) & 0x1ff).
63 30 29 21 20 12 11 0
+------------------------------------------------------------+-------------+
| upper bits | L2 index | L3 index | page offset |
+------------------------------------------------------------+-------------+
And once more for a level 1 translation table:
63 40 39 30 29 21 20 12 11 0
+------------------------------------------------------------+-------------+
| upper bits | L1 index | L2 index | L3 index | page offset |
+------------------------------------------------------------+-------------+
At this point, you used all 39 bits of your virtual address, so you're done. If you had 40-bit addressing, then there'd be another L0 table to go through. If you had 38-bit addressing, then we would've taken the L1 table all the same, but it would only span 0x800 bytes instead of 0x1000.
But where did the L1 translation table come from? Well, from TTBR0_EL1. Its physical address is just in there, serving as the root for address translation.
Now, to perform the actual translation, you have to do this whole process in reverse. You start with a translation table from TTBR0_EL1, but you don't know ad-hoc whether it's L0, L1, etc. To figure that out, you have to look at the translation granule and the number of bits used for virtual addressing. With 4KiB pages there's a 12-bit page offset and 9 bits for each level of translation tables, so with 39 bits you're looking at an L1 table. Then you take bits 39:30 of the virtual address to index into it, giving you the address of the L2 table. Rinse and repeat with bits 29:21 for L2 and 20:12 for L3, and you've arrived at the physical address of the target page.
I have 2 txt files
The 1) txt file is like this :
sequence_id description
Solyc01g005420.2.1 No description available
Solyc01g006950.3.1 "31.4 cell.vesicle transport Encodes a syntaxin localized at the plasma membrane (SYR1 Syntaxin Related Protein 1 also known as SYP121 PENETRATION1/PEN1). SYR1/PEN1 is a member of the SNARE superfamily proteins. SNARE proteins are involved in cell signaling vesicle traffic growth and development. SYR1/PEN1 functions in positioning anchoring of the KAT1 K+ channel protein at the plasma membrane. Transcription is upregulated by abscisic acid suggesting a role in ABA signaling. Also functions in non-host resistance against barley powdery mildew Blumeria graminis sp. hordei. SYR1/PEN1 is a nonessential component of the preinvasive resistance against Colletotrichum fungus. Required for mlo resistance. syntaxin of plants 121 (SYP121)"
Solyc01g007770.2.1 No description available
Solyc01g008560.3.1 No description available
Solyc01g068490.3.1 20.1 stress.biotic Encodes a protein containing a U-box and an ARM domain. senescence-associated E3 ubiquitin ligase 1 (SAUL1)
..
.
the 2nd txt file has the gene ids:
Solyc02g080050.2.1
Solyc09g083200.3.1
Solyc05g050380.3.1
Solyc09g011490.3.1
Solyc04g051490.3.1
Solyc08g006470.3.1
Solyc01g107810.3.1
Solyc03g095770.3.1
Solyc12g006370.2.1
Solyc03g033840.3.1
Solyc02g069250.3.1
Solyc02g077040.3.1
Solyc03g093890.3.1
..
.
.
Each txt has a lot more lines than the ones i show. I just wanted to know what grep command should i use that i only get the genes that are on the 2nd txt file, deduct from the 1st with the description next to it.
thanks
My requirement is to receive/send more than 8 bytes from 29 bit CAN id. I'm using TP layer for this. Is there any library functions for receiving the can data which is from 29 bit id? Ex: 0x1CDA00FE is tester and
0x1CDAFE00 is server.
non CAN-FD protocol on Data Link Layer does not support payload greater than 8 bytes. Payload in TP greater than 8 bytes is handled as consecutive frames, and usually are incorporated in a packaging protocol, like UDS.
If this is a diagnostic message ID, and you have diag layer (UDS/KWP) configured on this message, there is a complete diag library in CAPL to send/receive diag requests on the configured Diag Layer.
But to do this "by hand" (I do not know what protocol uses your message IDs to send consecutive frames, like in UDS) :
Receiving
message 0x1CDA00FE msgContainer; /*Create/ define an arbitrary instance of CAN message element*/
Here is how you capture the first iteration of the consecutive frames sent (all of them will be 8 bytes, but you will wait for them as individual 8 byte messages until the intended TP layer payload is completely sent to you). So you basically repeat this snippet 6 times (if your TP payload is 6*8=48bytes long), and do with the msgContainer (message with your payload) whatever you want, until the next one comes.
testwaitformessage(0x1CDA00FE,1000); //wait for the message ID to arrive in 1s
testGetWaitEventMsgData(msgContainer); /*Capture the message content from tail to toe in the msgContainer*/
write("The Byte(0) selector of the captured message for example selects the first byte of the 8 byte payload: 0x%X",msgContainer.byte(0)); // see message selectors for more options
Sending:
byte qword QPayload=0x11223344x55667788; /*You can do with byte array also, I don't fancy to write fors now*/
message 0x1CDAFE00 messagetoSend;
messagetoSend.qword(0)=QPayload;
output(messagetoSend);
/*Prepare another payload to send as the next consecutive frame*/
...
I was looking at the data files storage in Neo4j and I came up with interrogations.
I followed those two articles :
http://digitalstain.blogspot.jp/2010/10/neo4j-internals-file-storage.html
http://digitalstain.blogspot.jp/2011/11/rooting-out-redundancy-new-neo4j.html
I'm working with Neo4j 2.0.0
Here are my questions :
The .id files have a 8 bytes id while the other files just need 4 bytes.
Why is the size different?
.
In the neostore.nodestore.db, a node is represented by 14 bytes. We have :
1 : in-use flag
2 to 5 : id of the first relation
6 to 9 : id of the first property
10 to 13 : id of the first label
The 14th byte is always 00 for me, what does it mean?
.
About how you make the difference between
a short string (which can be stocked like the primitives properties)
and a long string (which has to be stocked in a separate file).
In the article, it is said that the limit is 3 blocks and a half.
There may be no direct relation to neo4j but I've tried several strings
the same length strings don't produce the same result (one goes with other properties, the other goes to the separate file).
Can someone explain why?
.
When a long string is stocked in the neostore.relationshiptypestore.db.names file,
each record takes 128 bytes. I don't fully understand the structure :
1 to 3 : ???
4 : length of the string
5 to 8 : ??? (always ff ff ff ff for me, which makes me think about a link but...)
9 to 128 : content of the string
One last question :
In the file neostore.propertystore.db.index, the 2 to 5 bytes are always 00 to me.
In the article, it is said that it represents the property count.
Any additionnal information?
--
Sorry for the long post, didn't know if I had to make one or several posts...
Hope my questions are understandable.
Thanks!
For my program delphi-code-coverage-wizard, I need to verify that a (detailed) mapping file .map matches a Delphi application .exe
Of course, this verification should be realized with Delphi.
Is there a way to check it ? Maybe by verifying some information from the EXE ?
I think a quite simple heuristic would be to check that the various sections in the PE file start and finish at the same place:
For example, here's the top of a map file.
Start Length Name Class
0001:00401000 000A4938H .text CODE
0002:004A6000 00000C9CH .itext ICODE
0003:004A7000 000022B8H .data DATA
0004:004AA000 000052ACH .bss BSS
0005:00000000 0000003CH .tls TLS
I also looked at what dumpbin /headers had to say about these sections:
SECTION HEADER #1
.text name
A4938 virtual size
1000 virtual address (00401000 to 004A5937)
A4A00 size of raw data
400 file pointer to raw data (00000400 to 000A4DFF)
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
60000020 flags
Code
Execute Read
SECTION HEADER #2
.itext name
C9C virtual size
A6000 virtual address (004A6000 to 004A6C9B)
E00 size of raw data
A4E00 file pointer to raw data (000A4E00 to 000A5BFF)
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
60000020 flags
Code
Execute Read
...truncated
Look at the .text section. According to dumpbin it starts at 00401000 and finishes at 004A5937 which is a length of 000A4938, exactly as in the .map file. Naturally you'd read the PE file directly rather than running dumpbin, but this illustrates the point.
I'd expect a vanishingly small number of false positives with this approach.