Background
In AOT compiling Dart for iOS Android (Dart Developer Summit 2016):
iOS restriction: Can't JIT
And also, from reading an article from the dart team: Flutter: Don’t Fear the Garbage Collector, I read:
In debug mode, most of Dart’s plumbing is shipped to the device: the
Dart runtime, the just-in-time compiler/interpreter (JIT for Android
and interpreter for iOS), debugging and profiling services.
Question
I wonder what the differences are between these 2 concepts, for dart specifically. Why can't iOS support JIT compilation but it can support a Dart interpreter?
Non-question
This is not about AOT vs JIT compilation, which is the more common question. You can find out about that here.
I also already know the difference between an Interpreter and a Compiler: Interpreter executes the code step by step in a higher level form, instead of compiling it to machine code, like JIT and AOT execution.
There aren't any technical reasons for iOS not to support the Dart JIT compiler, it's due to a security policy by Apple. Also this limitation is not Dart specific, it affects every JIT compiler out there.
A JIT compiler needs to create new machine code on the fly and write it to executable pages, but on iOS that can only be done by apps with the "dynamic-codesigning" entitlement. This entitlement is granted for example to 1st party apps using JavaScriptCore, but not 3rd party apps.
This means App Store apps cannot write to executable memory, and so a JIT cannot be embedded in them. You can run JIT accelerated JavaScript using a WKWebView (which runs in a separate process with 1st party entitlements) but you cannot run your own JIT compiler on iOS.
An interpreter w/o a JIT on the other hand never generates new machine code so it can run without problems on iOS (though the App Store review process still imposes restrictions on what scripts it is allowed to run.)
Over the years there have been several exploits to get a JIT working, allowing for example to switch pages from +w to +x and back, but never both at the same time, but it seems currently none of those work and using a JIT in a 3rd party app is simply not possible. Even if exploits are still available, the Dart team wouldn't rely on iOS vulnerabilities that could be fixed at any time, or worse, cause apps to be flagged as malicious.
Note you'll find articles online talking about how iOS 14 allows 3rd-party apps to use a JIT compiler, but that seems to be limited to apps sideloaded via developer tools so it's not relevant for Dart / Flutter, which target devs who want to distribute their apps via the App Store.
I know these technologies are all related but could someone please explain what each one is used for and how they fit together?
Oversimplification:
Graal - Java bytecode compiler. Can be used just in time (as part of a JVM) or ahead of time.
SubstrateVM - other things (runtime) needed to actually run ahead-of-time compiled Java bytecode without a JVM. This powers the "native-image" command of GraalVM.
Truffle - framework for implementing languages as AST interpreters which can be just-in-time compiled using graal. Some notable languages implemented are JavaScript, Ruby, R and LLVM bitcode.
GraalVM - most of these technologies packaged together in order to support different use cases, for example: running JVM programs (i.e. anything that compiles to Java bytecode) using Graal as the JIT compiler for better peak performance, ahead-of-time compiling JVM programs for fast startup and low memory footprint, running fast dynamic languages (JS, R, Ruby) that can interoperate without overhead, and so on.
How can I build and compile my own Lua files on Windows? And make them executable.
I am reading Beginning Lua programming, and I have Windows 7 and MacOS Lion both installed. I am having the hard time to follow the instructions. They do not work for me.
On MacOS I open the terminal and put these in:
export LUA_DIR=/usr/local/lib/lua/5.1
mkdir -p /usr/local/lib/lua/5.1 (it tells me, mkdir: illegal option) and I can not follow from here
SET LUA_DIR=”c:\program files\lua\5.1”
As for Windows I do this according to the book.
This what I see in my shell c:\Users\bd>
mkdir "c:\program files\utility" and it tells me access is denied
I have tried to right click on this folder and check off read only, but it does not work.
Any clues would be appreciated, this part has been really confusing for me.
To package your Lua files into an executable on Windows you have several options. There is srlua, there is wxLuaFreeze from wxLua (available as a binary for Windows), and there are more options in this SO answer.
Essentially, the main two options are: (1) append your Lua code to a precompiled exe file, such that it will be loaded and executed when that exe file is run, and (2) convert your Lua code into real executable by compiling it to bytecode, then to C, and then to your target platform.
As to your MacOS issue, mkdir -p means that mkdir is asked to create intermediate directories (for example, you asked to create /a/b/c, it will also create /a/b if those don't exist). As you don't say which version of MacOS you run, it's difficult to provide more detailed answer.
For now the standard distribution of Lua does not compile a script to native executable code; it execute your scripts by first compiling it to bytecode, then by interpreting the bytecode with a reasonnably fast static interpret (this also means that it is easily portable across native or virtual systems, and very resistant to attacks (that could be targetting bugs in the native compiler itself).
Also Lua still does not feature a runtime JIT compiler like Java and .Net: Lua still does not features a VM to produce a safe sandbox.
There exists Lua packages that convert your bytecode (or directly a source script) to a C source that can be used to convert a Lua library into native mode via the same C compiler used to compile the Lua engine itself (this is how the builtin libraries are produced, though they are slightly optimized manually in some time-critical parts).
However it is possible to compile Lua to a javascript source, and run it with fast performance using Javascript, because today's Javascript interprets do have good performance with their implemented VM featuring a JIT compiler for their own bytecodes.
It is also possible by converting it the Lua bytecode to a .Net or Java source that can then be executed directly from Lua (for that you need a version of Lua that has been ported to .Net or Java or Javascript, something that is not so complicate than developing in C/C++ directly a VM with a JIT compiler (a moderately complex part is the bytecode verifier, but the really complex part is the memory manager its garbage collector and its sandbox so that your Lua script will be fully isolated from the Lua engine itself for itw own memory, but the most complex part if the runtime optimizer and collection of profiling statistics: this has been done in the modern VMs for Java, .Net, Javascript, PHP/Zend, Python, Perl...).
I dont know which other language VM would offer the best performance to port Lua and implement on it a compiler to their own bytecode running at near native speed in their VM. But my own small experience with programs (in a much simpler language) self-generating a bytecode that they can run themselves, has always shown me Java winning in performance over .Net and Javascript. This is most probably because Java features an profiling-based dynamic code optimizer
(On the opposite the .Net optimizer runs only once during program installation, using some profiling data collected during the installation of the .Net VM itself, or at first instanciation of the script, without really knowing any profiling data collected during execution of the compiled program itself, and based on some cheked assumptions about the platform capabilities).
I also don't if would be faster in PHP, Python or Perl; the comparison with newer Javascript engines was never attempted though. Porting/compiling a Lua program to Javascript is relatively easy because it implements closures relatively easy for the resolution of linkages. Then the generated Javascript will compile to native code with the excellent Javascript's JIT compilers we have today (and never cease to improve in performance, so much that I've seen various appliactions running now faster in Javascript than before when they were written in C++ or plain C; as well the memory footprint has largely been reduced, we no longer have memory leaks, and even if there's a garbage collector, today's Javascript VM have a very efficient one, which is even better than the GC implemented in the native Lua).
But Lua remains useful as it is easy to secure and sandbox and offers various security benefits (but there are security issues in Lua as well for some kinds of applications, where Javascript offers some solutions, notably for side-channel attacks based on variation of time of execution; but these side-channel attacks are very hard to solve and can affect any system, any program, any programming language, and this starts becoming a critical issue because they are now more esily exploitable; the reason of that comes from hardware optimizations that we depend more and more today when we want to maximize the performances). And with Lua you may be more immune to these problems that a sandboxing sofware environment cannot solve alone.
Probably later we'll see a true VM implementation of Lua with a JIT and self-generating code and the possibility to instanciate new sandboxed VMs to run their self-generated code. It will take more time to generate an EXE file for distribution; notably because it generally requires adding also an installer and a distribution manager.
So for now we could imagine distributing Lua applications compiled to the bytecode of another JIT-capable VM: this generated bytecode would be faster than the Lua bytecode, and would then be extremely complex to reverse-engineer to the semantics of Lua because it would require two separate reverse engineering first from the bytecode of the other VM to the bytecode of Lua, both bytecodes loosing some easiy inferable rules and options tested and foll, and then again to sme Lua source
For the OSX terminal issue:
This command should work
export LUA_DIR=/usr/local/lib/lua/5.1
This command will probably give you permission problems:
mkdir -p /usr/local/lib/lua/5.1
You may try this to solve that. You will be prompted for your password:
sudo mkdir -p /usr/local/lib/lua/5.1
This command has nothing to do with OSX and will not work. This is a windows command:
SET LUA_DIR=”c:\program files\lua\5.1”
You have a permissions problem with Windows- try creating your cmd or PowerShell in Administrator mode. C:\Program Files is a protected directory that a regular user account doesn't have permission to write to.
As for the OS X issue, check out the mkdir OS X manual page to make sure you have the command correct.
So, if I understood your question correctly, you are trying to build Lua on Windows.
This is of course possible, but not easy for beginners. I would highly recommend you to use a binary distribution, which is much easier to install, unless you have special requirements.
Here are several Windows distributions :
Lua Binaries (Lua 5.1 and 5.2)
LuaForWindows (Lua 5.1)
LuaDist (Lua 5.2)
I was wondering if there is any AOT compilation options for the mono for android platform (or anything planned?).
I am asking this because I will port a game using mono for android, and performance is really important, this is why I wanted to know if AOT compilation is available (or any other compilation option that can produce better performances)
I know that Monotouch uses AOT compilation, and that it can optionally use llvm as compiler in order to make some optimisations. So, I am wondering if the same options are available in mono for android.
Thanks in advance
AOT is not available in Mono for Android, I don't think it is on the road-map either.
As far as I understand JIT is not slower than AOT, rather the contrary, because JIT can use the information it knows about the environment it runs on to optimize code to it. An AOT compiler cannot know everything about the same environment.
This thread: Why is Java faster when using a JIT vs. compiling to machine code? has some answers that elaborates on this.
So I really doubt that you would get any performance improvements if there were an AOT compiler.
I came across this quote:
"The .NET Common Language Runtime
(CLR) supplies at least one JIT
compiler for every NET-supported
computer architecture, so the same set
of CIL can be JIT-compiled and run on
different architectures."
I've looked around but can't find a definitive list of the JIT compilers supported by CLR?
Mainly there is one JIT compiler for every processor architecture the CLR runs on. In particular
x86
amd64
ia64
Less sure about the following.
PowerPC (XBox360)
ARM (Mobile devices)