I'm reading about dart compile, and it has a few option: Executable, AOT snapshot, JIT snapshot and Kernel snapshot and JavaScript.
What is the difference between an executable and a snapshot? Is it purely the fact executables contain the Dart runtime/ VM, whereas a snapshot doesn't. Why is it called a snapshot?
2 highly related question (which I found after posting this question) are:
What is the difference between Dart's snapshots and Java bytecode?
What is the snapshot concept in dart?
This question is different to Dart: Snapshots vs AOT, since is asking the difference between a Snapshot and AOT, but actually AOT files are snapshots. It also primarily asked about the differences between Snapshot options (AOT, Kernel, JIT).
An executable (created by dart compile exe) is a combination of an AOT snapshot, and the Dart runtime. The Dart runtime is needed to run any Dart code, as it performs critical tasks like managing memory (including garbage collection) and performing runtime type checks.
The three kinds of snapshots (AOT, kernel, and JIT) all contain just the compiled source code. They all need a runtime to be run (typically you just use dart run <snapshot>).
The snapshots should perhaps have been named differently. Would 'module' have been easier to understand?
When Dart code is compiled, it is compiled into intermediary byte code rather than native code. This means that in order for the code to be executed in any arbitrary environment, the compiler needs to include the Dart virtual machine inside the executable along with the user code as well as any portions of the core library the program requires. Usually, this is fine because the Dart runtime is actually quite compact, but the obvious downside to this is that the executable will be larger and start-up time will be longer as the runtime needs to be extracted and warmed up before the user code can be run.
If, however, you are compiling code for an environment where you can guarantee that a Dart runtime will be present (such as a server machine or an IoT device), you can omit the runtime from the compiled program by building a snapshot file rather than an executable. This results in a smaller compiled file with a faster start-up time, although it requires a command to execute and as such is less convenient. You can learn more about snapshots and how to build and execute them on the Dart GitHib wiki page on snapshots.
There are three different kinds of snapshots: Kernel snapshots, which contain only AST information rather than compiled byte code making them usable by Dart runtimes on any supported architecture (portable but slow); JIT snapshots, which contain just the parts of the program necessary for startup and leaves the rest to be interpreted at runtime (fastest startup but slower execution); and AOT snapshots, which fully compiles the entire program into byte code (slower startup but fastest execution).
As for why it's called a "snapshot", I couldn't say. If I had to guess, that's because it's a "snapshot" of the program in its compiled state but without the instructions necessary to run it as a standalone executable.
(The above is based on my quick research on the subject and may be missing some key details. If a member of the Dart team happens upon this question, they will probably be able to offer a more detailed and technical explanation.)
Related
I am searching a lot about this great language and got myself on this page https://github.com/dart-lang/sdk/wiki/Snapshots but it didn't answer some questions:
What is the difference between a Snapshot and AOT? Which is faster and why?
Why the AOT generated on a Windows machine doesn't run on a Linux but a snapshot generated through dart --snapshot_kind=kernel does? What is "kernel"? Is it slower?
What is the difference from a Kernel Snapshot and a JIT Snapshot? Which is faster?
With snapshots, nothing of your code (kernel snapshots) or only portions of your code (JIT snapshots) are pre-compiled into binary format. With AOT, all of your code gets pre-compiled to binary (platform specific) format. Pre-compiled binary code (AOT) is faster, because there is no need to compile code at runtime.
As mentioned above, AOT compiled code is transformed to platform specific binary code. Therefor you can not run a program AOT compiled for Windows on a Linux machine. With kernel snapshots, nothing of your code is pre-compiled. So it is portable between platforms, but must be compiled at runtime. So yes, this is slower.
Also as mentioned above, with kernel snapshots nothing of your code gets pre-compiled. With JIT snapshots, the program is executed in a test run and each part of the code that is executed, gets pre-compiled to platform specific binary format. So JIT snapshots is faster then kernel snapshots.
Is there a way to get compiled down version of a dart code for a particular target? Say the following code compiled for Dart VM.
I'm new to Dart and quite often come across high level abstract code and wonder how it is translated in VM.
names.forEach(print);
Dart is not compiled to something like Java-bytecode or DotNet IL.
Dart is compiled to machine code by the VM either ahead of time (AoT) or just in time (JiT)
https://mrale.ph/dartvm/
The name "Dart VM" is historical. Dart VM is a virtual machine in a
sense that it provides an execution environment for a high-level
programming language, however it does not imply that Dart is always
interpreted or JIT-compiled, when executing on Dart VM. For example,
Dart code can be compiled into machine code using Dart VM AOT pipeline
and then executed within a stripped version of the Dart VM, called
precompiled runtime, which does not contain any compiler components
and is incapable of loading Dart source code dynamically.
Dart 2 uses Kernel AST though generated by the common front-end (CFE)
There is some abstraction happening from the Dart language though
https://github.com/dart-lang/sdk/blob/master/pkg/kernel/README.md
Dart Kernel is a small high-level language derived from Dart. It is
designed for use as an intermediate format for whole-program analysis
and transformations, and to be consumed by codegen and execution
backends.
The kernel language has an in-memory representation in Dart and can be
serialized as binary or text.
Both the kernel language and its implementations are unstable and are under development.
See also https://github.com/dart-lang/sdk/blob/master/pkg/kernel/binary.md
I am writing an F# port of a program I wrote in native code in the past. I used BenchmarkDotNet to measure its performance. I also placed a native EXE in the application's output directory.
I set my native program as the baseline benchmark and saw it was 5x faster than my F# program. Just as I expected!
However, the native program is posted on GitHub and distributed as a Win64 binary only. In case somebody using another OS tries to run it, it will crash.
So, how to specify that this benchmark will only run on 64-bit Windows?
In BenchmarkDotNet, there is a concept of Jobs. Jobs define how the benchmark should be executed.
So, you can express your "x64 only" condition as a job. Note that there are several different 64x jit-compilers depends on runtime (LegacyJIT-x64 and RyuJIT-x64 for the full .NET Framework, RyuJIT-x64 for .NET Core, and Mono JIT compiler). You can request not only a specific platform but also a specific JIT-compiler (it can significantly affect performance), e.g.:
[<RyuJitX64Job>]
member this.MyAwesomeBenchmark () = // ...
In this case, a user will be notified that it's impossible to compile the benchmark for required platform.
Unfortunately, there is no way to require a specific OS for now (there is only one option: current OS). So, in your case, it's probably better to check System.Environment.Is64BitOperatingSystem and System.Environment.OSVersion at the start and don't run benchmarks on invalid operation systems.
I've been reading up on Dart snapshots, and they're frequently compared to Smalltalk images. But to me, they sound alot like Java bytecode.
For example:
"A Dart snapshot is just a binary serialization of the token stream, generated from parsing the code. A snapshot is not a "snapshot of a running program", it's generated before the tokens are turned into machine code. So, no program state is captured in a snapshot."
Plus they're cross-platform:
"The snapshot format itself is cross-platform meaning that it works between 32-bit, 64-bit machines and so forth. The format has been made so that it's quick to read into memory with a emphasis on minimizing extra work like pointer fixups."
Am I getting it wrong somewhere?
Sources:
What is the snapshot concept in dart?
http://www.infoq.com/articles/google-dart
Snapshots contain the VM data structures representing the loaded script in a serialized form similar to Smalltalk images. To get a better understanding of what is contained in the snapshot, we should take a look at what the Dart VM creates as it reads the script:
Library objects, referring to all top-level structures such as classes or top-level methods and variables.
Class objects, containing all objects describing all methods and fields.
Script and Tokenstream objects representing all loaded source code.
String objects for all used identifiers and string constants in the source code.
This object graph is serialized into a file when generating a snapshot using a format that is architecture agnostic. This allows the Dart VM to deserialize this snapshot file on 32-bit or 64-bit machines and recreate all of the necessary internal VM data structures much quicker than reading the original scripts from a set of files (see John's answer).
To clarify John's answer a bit. The Dart VM does not parse ALL of the source code when generating the snapshot. It only needs to parse the top level of the sources to be able to extract class, method and field definitions as these are represented in the serialized graph. In particular method bodies are not parsed and as it is customary for a scripting language errors will be only reported once control reaches the particular method.
The purpose of Java bytecodes is entirely different as Ladicek points out. You could create a snapshot of the VM data structures in a JVM once the bytecodes are loaded to get a similar effect.
In short: The snapshot contains an efficient representation of all the data structures allocated on the Dart VM heap which are needed to start executing the script.
A Dart snapshot is just a roll up of all source files that has been parsed ahead of time. A Dart snapshot is not similar to a Java bytecode file. A Java bytecode file consists of JVM machine code and is the product of a compile, link, and assembly (into JVM machine code) phase.
A Dart snapshot is a binary file of a Dart program and it's import/part source file dependencies that has been parsed into an abstract syntax tree and rolled into a single file. Executing a Dart snapshot allows for faster startup times because:
Only 1 file must be loaded from disk or off network. In contrast, a non-snapshot Dart program must be fetched, then any imported files must be fetched, and so on. Before each subsequent source file request can be made the previously fetched source file must be parsed to find out if it's referencing more source files. Imagine if your Dart program imported 10 libraries which consisted of 10 source files each. That means 110 I/O requests and parses that are done one after another.
The parsing has been done ahead of time. It's already known to be syntactically correct and ready to be compiled by the Dart VM.
I will just point out that as of Dart 2+, there are several distinctive concepts when it comes to Snapshots:
Kernel Snapshot
JIT Snapshot
AOT Snapshot
You can read more here.
How can I build and compile my own Lua files on Windows? And make them executable.
I am reading Beginning Lua programming, and I have Windows 7 and MacOS Lion both installed. I am having the hard time to follow the instructions. They do not work for me.
On MacOS I open the terminal and put these in:
export LUA_DIR=/usr/local/lib/lua/5.1
mkdir -p /usr/local/lib/lua/5.1 (it tells me, mkdir: illegal option) and I can not follow from here
SET LUA_DIR=”c:\program files\lua\5.1”
As for Windows I do this according to the book.
This what I see in my shell c:\Users\bd>
mkdir "c:\program files\utility" and it tells me access is denied
I have tried to right click on this folder and check off read only, but it does not work.
Any clues would be appreciated, this part has been really confusing for me.
To package your Lua files into an executable on Windows you have several options. There is srlua, there is wxLuaFreeze from wxLua (available as a binary for Windows), and there are more options in this SO answer.
Essentially, the main two options are: (1) append your Lua code to a precompiled exe file, such that it will be loaded and executed when that exe file is run, and (2) convert your Lua code into real executable by compiling it to bytecode, then to C, and then to your target platform.
As to your MacOS issue, mkdir -p means that mkdir is asked to create intermediate directories (for example, you asked to create /a/b/c, it will also create /a/b if those don't exist). As you don't say which version of MacOS you run, it's difficult to provide more detailed answer.
For now the standard distribution of Lua does not compile a script to native executable code; it execute your scripts by first compiling it to bytecode, then by interpreting the bytecode with a reasonnably fast static interpret (this also means that it is easily portable across native or virtual systems, and very resistant to attacks (that could be targetting bugs in the native compiler itself).
Also Lua still does not feature a runtime JIT compiler like Java and .Net: Lua still does not features a VM to produce a safe sandbox.
There exists Lua packages that convert your bytecode (or directly a source script) to a C source that can be used to convert a Lua library into native mode via the same C compiler used to compile the Lua engine itself (this is how the builtin libraries are produced, though they are slightly optimized manually in some time-critical parts).
However it is possible to compile Lua to a javascript source, and run it with fast performance using Javascript, because today's Javascript interprets do have good performance with their implemented VM featuring a JIT compiler for their own bytecodes.
It is also possible by converting it the Lua bytecode to a .Net or Java source that can then be executed directly from Lua (for that you need a version of Lua that has been ported to .Net or Java or Javascript, something that is not so complicate than developing in C/C++ directly a VM with a JIT compiler (a moderately complex part is the bytecode verifier, but the really complex part is the memory manager its garbage collector and its sandbox so that your Lua script will be fully isolated from the Lua engine itself for itw own memory, but the most complex part if the runtime optimizer and collection of profiling statistics: this has been done in the modern VMs for Java, .Net, Javascript, PHP/Zend, Python, Perl...).
I dont know which other language VM would offer the best performance to port Lua and implement on it a compiler to their own bytecode running at near native speed in their VM. But my own small experience with programs (in a much simpler language) self-generating a bytecode that they can run themselves, has always shown me Java winning in performance over .Net and Javascript. This is most probably because Java features an profiling-based dynamic code optimizer
(On the opposite the .Net optimizer runs only once during program installation, using some profiling data collected during the installation of the .Net VM itself, or at first instanciation of the script, without really knowing any profiling data collected during execution of the compiled program itself, and based on some cheked assumptions about the platform capabilities).
I also don't if would be faster in PHP, Python or Perl; the comparison with newer Javascript engines was never attempted though. Porting/compiling a Lua program to Javascript is relatively easy because it implements closures relatively easy for the resolution of linkages. Then the generated Javascript will compile to native code with the excellent Javascript's JIT compilers we have today (and never cease to improve in performance, so much that I've seen various appliactions running now faster in Javascript than before when they were written in C++ or plain C; as well the memory footprint has largely been reduced, we no longer have memory leaks, and even if there's a garbage collector, today's Javascript VM have a very efficient one, which is even better than the GC implemented in the native Lua).
But Lua remains useful as it is easy to secure and sandbox and offers various security benefits (but there are security issues in Lua as well for some kinds of applications, where Javascript offers some solutions, notably for side-channel attacks based on variation of time of execution; but these side-channel attacks are very hard to solve and can affect any system, any program, any programming language, and this starts becoming a critical issue because they are now more esily exploitable; the reason of that comes from hardware optimizations that we depend more and more today when we want to maximize the performances). And with Lua you may be more immune to these problems that a sandboxing sofware environment cannot solve alone.
Probably later we'll see a true VM implementation of Lua with a JIT and self-generating code and the possibility to instanciate new sandboxed VMs to run their self-generated code. It will take more time to generate an EXE file for distribution; notably because it generally requires adding also an installer and a distribution manager.
So for now we could imagine distributing Lua applications compiled to the bytecode of another JIT-capable VM: this generated bytecode would be faster than the Lua bytecode, and would then be extremely complex to reverse-engineer to the semantics of Lua because it would require two separate reverse engineering first from the bytecode of the other VM to the bytecode of Lua, both bytecodes loosing some easiy inferable rules and options tested and foll, and then again to sme Lua source
For the OSX terminal issue:
This command should work
export LUA_DIR=/usr/local/lib/lua/5.1
This command will probably give you permission problems:
mkdir -p /usr/local/lib/lua/5.1
You may try this to solve that. You will be prompted for your password:
sudo mkdir -p /usr/local/lib/lua/5.1
This command has nothing to do with OSX and will not work. This is a windows command:
SET LUA_DIR=”c:\program files\lua\5.1”
You have a permissions problem with Windows- try creating your cmd or PowerShell in Administrator mode. C:\Program Files is a protected directory that a regular user account doesn't have permission to write to.
As for the OS X issue, check out the mkdir OS X manual page to make sure you have the command correct.
So, if I understood your question correctly, you are trying to build Lua on Windows.
This is of course possible, but not easy for beginners. I would highly recommend you to use a binary distribution, which is much easier to install, unless you have special requirements.
Here are several Windows distributions :
Lua Binaries (Lua 5.1 and 5.2)
LuaForWindows (Lua 5.1)
LuaDist (Lua 5.2)