do you have any experience with SD Erlang project?
There seems to be implemented many interesting concepts regarding the comm mesh optimalizations and I'm just curious if some of you used those in production already or in some real project at least.
SD erlang repo
Thanks!
The project has finished a week ago. The main ideas behind SD Erlang are reducing the number of connections Erlang nodes maintain while keeping transitivity and common namespace for groups of nodes. Benchmarks that we used (Orbit, Ant Colony Optimization (ACO), and Instant Messenger) showed very promising results. Unfortunately, we didn't have enough human resources to refactor Sim-Diasca simulation engine. So, no, SD Erlang hasn't been used yet in a real application.
At the moment we are writing up the last deliverable that will provide an overview of what has been achieved. It will appear here in a few weeks (D6.2). In general we are happy with the results we get using SD Erlang, so there are plans for a follow up project to continue to work on it but currently this is work in progress.
This is not a direct answer but I will use SD-Erlang in a embedded application which needs to scale to hundreds of nodes (small embedded CPUs). From what I have seen its ready to be tried out in a real application. To furtehr evaluate lets consider the alternatives:
You have only a few distributed nodes: then you probably don't need it and can just connect all the nodes and for name registry use either the global module (slow but sturdy) or gprocwith the new locks_leader branch which avoids the quite broken gen_leader which so far prevented using gproc in distributed mode in production.
You need many nodes (how many depends on your hardware and requirements but you start to get into interesting territory with > 70 nodes)
Use SD-Erlang and fix whatever problems you encounter in production, or at least report them. It certainly solves a lot of the problems you get with normal Erlang distribution
Roll your own solution either with playing with different cookie values or with hidden nodes: hint you can set different cookie values for different peer nodes. But then you need to roll your own global name registry and management code: looks like a variant of Greenspuns 10th rule or closer to Erlang Virdings 1st rule : you probably will result in implementing half of SD Erlang yourself.
Don't use Erlang distribution at all. That seems to be the industry standard that for anything involving more nodes or crossing data-centers you shouldn't use Erlang distribution at all but run your own protocols. My personal opinion is to rather fix Erlang Distributions problems than just ditch it. Its much too useful and time saving when it works for a use case to just give up on it. And I see SD-Erlang as being the fix for the "too many nodes" problem, its at least the right starting point.
for a study on genetic programming, I would like to implement an evolutionary system on basis of llvm and apply code-mutations (possibly on IR level).
I found llvm-mutate which is quite useful executing point mutations.
As far as I have understood, the instructions get count/numbered, one can then e.g. delete a numbered instruction.
However, introduction of new instructions seems to be possible as one of the availeable statements in the code.
Real mutation however would allow to insert any of the allowed IR instructions, irrespective of it beeing used in the code to be mutated.
In addition, it should be possible to insert library function calls of linked libraries (not used in the current code, but possibly available, because the lib has been linked in clang).
Did I overlook this in the llvm-mutate or is it really not possible so far?
Are there any projects trying to /already have implement(ed) such mutations for llvm?
llvm has lots of code analysis tools which should allow the implementation of the afore mentioned approach. llvm is huge, so I'm a bit disoriented. Any hints which tools could be helpful (e.g. getting a list of available library functions etc.)?
Thanks
Alex
Very interesting question. I have been intrigued by the possibility of doing binary-level genetic programming for a while. With respect to what you ask:
It is apparent from their documentation that LLVM-mutate can't do what you are asking. However, I think it is wise for it not to. My reasoning is that any machine-language genetic program would inevitably face the "Halting Problem", e.g. it would be impossible to know if a randomly generated instruction would completely crash the whole computer (for example, by assigning a value to a OS-reserved pointer), or it might run forever and take all of your CPU cycles. Turing's theorem tells us that it is impossible to know in advance if a given program would do that. Mind you, LLVM-mutate can cause for a perfectly harmless program to still crash or run forever, but I think their approach makes it less likely by only taking existing instructions.
However, such a thing as "impossibility" only deters scientists, not engineers :-)...
What I have been thinking is this: In nature, real mutations work a lot more like LLVM-mutate that like what we do in normal Genetic Programming. In other words, they simply swap letters out of a very limited set (A,T,C,G) and every possible variation comes out of this. We could have a program or set of programs with an initial set of instructions, plus a set of "possible functions" either linked or defined in the program. Most of these functions would not be actually used, but they will be there to provide "raw DNA" for mutations, just like in our DNA. This set of functions would have the complete (or semi-complete) set of possible functions for a problem space. Then, we simply use basic operations like the ones in LLVM-mutate.
Some possible problems though:
Given the amount of possible variability, the only way to have
acceptable execution times would be to have massive amounts of
computing power. Possibly achievable in the Cloud or with GPUs.
You would still have to contend with Mr. Turing's Halting Problem.
However I think this could be resolved by running the solutions in a
"Sandbox" that doesn't take you down if the solution blows up:
Something like a single-use virtual machine or a Docker-like
container, with a time limitation (to get out of infinite loops). A
solution that crashes or times out would get the worst possible
fitness, so that the programs would tend to diverge away from those
paths.
As to why do this at all, I can see a number of interesting applications: Self-healing programs, programs that self-optimize for an specific environment, program "vaccination" against vulnerabilities, mutating viruses, quality assurance, etc.
I think there's a potential open source project here. It would be insane, dangerous and a time-sucking vortex: Just my kind of project. Count me in if someone doing it.
I am working in software firm where hardware independent coding is done on the Network Chipsets and fully Multigthreading coding implemented and various buffers(CRU Buffer, Linear Buffer) are handled and memory (stack memory) is optimally used. And IPC done via Message queues. And Multiple Locks, Semaphores are used for concurrency mechanisum. Now i will be assigned to new development project, where i have to understand and have to develop new features in next one month. I am feeling like middle of the Amazon Jungle :).
=> I am in beginning level in OS concepts. I feel like intermediate level in C language. So expecting, suggestion for "Materail/Book which could help me to improve/concrete my OS skills"
i saw OS Book by Abraham Silberschatz and Modern Operating Systems by Tanenbaum - 3rd Edition. Both are looking big and covers all corners of operating system. I thought to study that book steadily and slowly for future referencee.
==> Now i am looking for the Network materials/books which explaining the "Main concepts" in the detailed manner. For example i have seen virtual memory concepts in one online material where clearly virtual memory explained.
Example abour virtual memory from that material:
amesmol#aubergine:~/test> objdump -f a.out
a.out: file format elf32-i386 architecture: i386, flags 0x00000112: EXEC_P, HAS_SYMS, D_PAGED start address 0x080482a0
explanation:
Notice the start address of the program is at 0x80482a0.Program thinks like where its starting address is actual physical address. But it is a virtual address space. Its original starting address at physical memory location 0x1000000.
As like this( correct point and example), could you people suggest good materials for the OS concepts ( Process Management, Memory Management, IPC)?
Can you also suggest the ways to improve/concrete this skills? (suggest either what kind of mini homework project i can do, etc..)
Thanks in advance
if you are working on projects you have to go over books you mentioned as soon as possible for theoretical explanations, concepts and terminologies. after that,even along with your reading, i suggest you to go to university websites to get hand on skills for small projects. some suggested links are as follows
http://www.eecg.toronto.edu/~lie/Courses/ECE344/
http://web.stanford.edu/~ouster/cgi-bin/cs140-winter13/pintos/pintos.html#SEC_Contents
http://www3.cs.stonybrook.edu/~porter/courses/cse624/f13/project.html
(JOS implementation. very helping instructors if you send them specific queries)
http://www.brokenthorn.com/Resources/OSDev7.html
http://www.osdever.net/bkerndev/Docs/intro.htm
(above two links are not university link but as a beginner I recommend this to start with )
apart from above, Lion's commentary on Unix code with line number reference must be in your reading to understand the implementation of small scale OS
I'm potentially interested in exploring a stack-based language like Forth (or Factor). What I'd like to see is how an application might be built from the ground up, step by step. The tutorials I've found are rudimentary and are not helping me to understand the bigger picture. It's confusing to think of how one might manage the stack when dealing with lots of parts.
I've always thought (maybe wrongly) that a good way to learn a language would be to use it to write a Roguelike game. I'm having trouble trying figure out how one would juggle a lot of things on a stack: a maze, dozens of creatures, treasures, character stats, etc.
In some sense, all languages are equivalent; you write a program by breaking a problem down into smaller pieces, then you code up those pieces and make them all work together. Forth has unusual syntax features, but it's still a programming language.
In fact, Forth places a great deal of power at your fingertips, much as Lisp does. In Lisp, using "macros", you can write your own control structures that are just as good as anything built-in; in Forth, you can do the same.
If you are interested in finding out more about Forth, I suggest you get the classic books Starting Forth and Thinking Forth by Leo Brodie.
Oh! Google just told me that both books are available free online now:
http://www.forth.com/starting-forth/
http://thinking-forth.sourceforge.net/
I'd point you to Factor rather than pure Forth; there are plenty of sample apps, GUIs, web apps, etc. If you're specifically interested in a web framework, look at Furnace.
Ultimately I don't understand the question; what does being stack-based have to do with getting anything done? Back in the old days it was the language of choice for embedded systems; I wrote everything from cereal boxing robots to calculators to... well, mostly everything.
I attempt to directly answer all of your questions at the bottom of this answer. But before that, here are 7 points or aspects of learning stack-based Forth, or generally about stack-based-languages, or ways to proceed with your goals:
1: Here is a key way to think about this: The stack is not the place where you keep your data -- just the way that you pass parameters from one function (or method) to another.
2: There is a "Basic Language" interpreter written in Forth. This would be one way to write an application in Forth -- write your own Basic interpreter (or type in one you find), then write the Roguelike game in the Basic language you just created.
3: If you go through the old "Forth Dimensions" magazine, several different object-oriented extensions to Forth were created and presented (including source code). Type one of those in, and use it to write your rogue-like game in one of those object-oriented ways. If you're still stuck, write it in your favorite C language -- C++, C#, or Java, and here's the key... write it as primitively as you can. Then take your own application apart -- each method, enum, constant, etc. constitutes part of a Domain Specific Language that you created in order to implement the game. Write down those parts, and figure out how to do it in Forth, ideally using only the idioms of Forth. You may have to work in Forth a while to get to that stage.
4: Learn how to program in assembly language, an assembly language for which a Forth exists that has an embedded assembler for the same microcontroller or CPU. This is where Forth really crushed the competition when computers were just becoming more widely available -- bringing a completely new microcontroller up, creating the BIOS and the OS. Once you've done this in pure assembly language, then go into the Forth that has the same assembly language and program the same application, but using the Forth assembler to build the building blocks (the DSL's) and then writing the BIOS and the OS in those DSL's. Forths happen to be both low-level and high-level at the same time, and the best-written applications are written in multiple DSL's, not in stack-based primitives. Once you've written in assembly language, when you try to learn Forth, you will find that it is easy -- the basic mechanisms are simple, but the combination of basic mechanisms is simple, powerful and elegant.
5: Look at either C# or Java, and realize that they are both based on Virtual Machines, which is a specific kind of DSL. But also realize that both Virtual Machines are stack machines, and the language that powers both of them are stack based languages. But if you program in Java, or if you program in C#, you really don't think at all about the stack foundation upon which you are running -- and that's a really good thing! You want to follow the same pattern. A. Realize what basic mechanisms / tools you need, B. Build the tools you need, C. Build your application with the tools you built. You may have to translate your own infix pseudocode to the postfix VM you've created. Do you need Object Orientation?--You can build it with Forth. Do you need multiprocessing?--You can build it. Do you need an Erlang kind of light multithreading?--You can build it. Do you need a Basic, a Pascal, a C#, a Java, a Lua?--You can build it. Build your Roguelike game in your favorite language, then perhaps a different language, then assembly language. Then build it in Forth.
6: At this link, there is a way to make a microprocessor that natively runs CIL (Common Intermediate Language -- the stack based language of both .NET and Mono, which used to be MSIL Microsoft Intermediate Language, before Microsoft Open Sourced the specification). You buy the IP logic block and then program it into an FPGA. Here is the Abstract of a paper that describes the object-oriented stack-based language and implementation of CIL:
Embedded systems and their applications are becoming ubiquitous and
transparent. Nowadays, the designers need to implement both hardware
and software as fast as they can to face the competition. Hence tools
and IPs became an important factor of the equation. In this paper, we
present a synthesizable core similar to the microarchitecture of
Tanenbaum's (2006) JVM processor. The core is an implementation of a
subset of Microsoft's CIL (Common Intermediate Language). We seek to
accelerate the development of embedded software by providing a
platform onto which the whole .NET framework (C#, Visual Basic.NET...)
(along with its object-oriented approach) could execute. We used a
Xilinx Virtex II Pro as the prototyping platform.
7: Go to the current work-place of Charles Moore, the father of Forth, GreenArrays, Inc. and look under Features for "Implements the colorForth instruction set", -- and you will find that he, too, has taken his virtual machine, having optimized Forth his entire life, and reduced it to an insanely fast 144 core microchip with a claimed capability of 100 BIPS. Here, you should be able to look at the "colorForth instruction set" and the description of it -- he has removed any fluff and bloat, and reduced it to an extremely spartan instruction set. We should learn from this genius before it is too late (he is advanced in years, and Covid19 currently threatens many of our patriarchs).
Below, I try to more directly handle each aspect of your question:
How would one code an application in Forth (or Factor)?
Same way as you would with any other application, learn the building blocks, and build the simplest thing that could possibly work, then grow it from there.
I'm potentially interested in exploring a stack-based language like Forth (or Factor).
There are a multitude of stack-based languages around. They seem to be the foundation, not only of most processors, but also of most virtual machines, as in Java and the JVM, and C# and the CIL.
What I'd like to see is how an application might be built from the ground up, step by step.
That's a great idea for a Forth book... Anybody game? I know there are a lot of Forth people out there, and because of Covid19, a lot of you are at home, too. What kink of application would you like to see? Just looking at FPC teaches a lot.
The tutorials I've found are rudimentary and are not helping me to understand the bigger picture.
There are higher level tutorials out there. But widen your search -- include JVM, CIL, and even LUA (because I think it has a VM, too). Write an app in Lisp. Then take the primitives you built in Lisp, and implement them in Forth, or Factor, or CIL, or JVM instructions.
It's confusing to think of how one might manage the stack when dealing with lots of parts.
That's why I said that you should learn an assembly language -- because most stack-based languages are a micro-step up from that. You have to learn the basic levers, springs, latches, and other primitive mechanisms out of which all of our software is ultimately made. But you only think in terms of stacks to make the primitive mechanisms -- once you have those made, in general, don't even touch the stack stuff -- think only in terms of the basic mechanisms, then build progressively higher layers, even progressing to object orientation, as exemplified in the CIL and JVM. The better applications are layers of implicit DSL's, each DSL targeted at a step up from the previous one. The layers are: Assembly-Language, CIL or other VM, BIOS, Operating-System, Library, Business-Language, Application. Most people only think of the Business-Language layer as a DSL, but every single layer is its own DSL, and really, every class, method, object, config-file, and specialized language, like HTML, CSS, JavaScript, etc. -- they are ALL DSL's, either explicitly (formally specified) or implicitly (created by the programmer as things are built.) Understanding gained from book "Code Complete" second edition -- read over and over is my suggestion.
I've always thought (maybe wrongly) that a good way to learn
a language would be to use it to write a Roguelike game.
Sure... but pick anything you like, really. But perhaps try getting an Arm-based Arduino, programming it, disassembling that, then writing your own assembly language -- then get a Forth for that Arm chip, and program it in that. Then implement your Roguelike game in the Arm-Forth.
I'm having trouble trying figure out how one would juggle a lot of
things on a stack: a maze, dozens of creatures, treasures, character
stats, etc.
Build your "Rogue" Domain-Specific-Language, make that into a VM, then implement the VM instructions first in a high level stack language, like CIL, then try to figure out how to do it in one of the low-level stack-languages. There are many flavors of object-orientation implemented in Forth -- spelunk the old Forth Dimensions articles
and some of the old Forth conference presentations to find them. Google, and the "Internet Archive", or Way-Back Machine, are your friends for this.
I can easily launch two or more separate Lua programs (running on LuaJIT2) on the same machine. But is there any way how these programs can communicate?
The best solution I can come up with is that each program can write a text file that orher program can read. But this is ree-aalllyyy slow solution (even if text files are saved on virtual RAM disks).
I know that thera are zeromq and other things that may help with this... The problem is that instructions are just too long and complicated / confusing (for me at least).
Any recommendations? Especially some working code example would be appreciated, no matter how simple. Even how to pass value of one variable from one LuaJIT2 process to another.
(I'm using Windows XP SP3, if that matters...)
Essentially, they keyword you should be looking for is "IPC" (Inter-Process Communication).
Some of the options you might want to explore:
Shared files
Shared memory
Network Sockets
Pipes (on POSIX systems)
Middleware utilizing one of the above
I can't really say that one of them is the best. The choice would depend on the other factors (needed speed, latency, what do you want to communicate), that you didn't provide. I just hope I pointed you in the right direction.
Also, if ZeroMQ is too complicated, i highly recommend you more reading, rather than looking for even simpler code examples. Lua is "do-it-yourself" language.
If you aren't really stick to Lua, take a look at PyRo (Python Remote Objects). Or you might want to implement something like this yourself.