I'm trying to debug a program that is using the Z3 API, and I'm wondering if there's a way, either from within the API or by giving Z3 a command, to print the current logical context, hopefully as if it had been read in an SMT-LIB file.
This question from 7 years ago seemed to indicate that there would be a way to do this, but I couldn't find it in the API docs.
Part of my motivation is that I'm trying to debug whether my program is slow because it's creating an SMT problem that's hard to solve, or whether the slowdown is elsewhere. Being able to view the current context as an SMT-LIB file, and run it in Z3 on the command line, would make this easier.
It's not quite clear what you mean by "logical context." If you mean all the assertions the user has given to the solver, then the command:
(get-assertions)
will return it as an S-expression like list; see Section 4.2.4 of http://smtlib.cs.uiowa.edu/papers/smt-lib-reference-v2.6-r2017-07-18.pdf
But this doesn't sound useful for your purposes; after all it is going to return precisely everything you yourself have asserted.
If you're looking for a dump of all the learned-lemmas, internal assertions the solver created etc; I'm afraid there's no way to do that from SMTLib. You probably can't even do that using the programmatic API either. (Though this needs to be checked.) That would only be possible by actually modifying the source code of z3 itself (which is open-source), and putting in relevant debug traces. But that would require a lot of study of the internals of z3 and would unlikely to help unless you're intimately knowledgeable about z3 code base itself.
I find that running z3 -v:10 can sometimes provide diagnostic info; if you see it repeatedly printing something, it's a good indication that something has gone wrong in that area. But again, what it prints and what it exactly means is guess work unless you study the source code itself.
Related
Is there a way to see what Z3 is doing under the hood? I would like to be able to see the steps it is taking, how long they take, how many steps, etc. I'm checking the equality of floating point addition/multiplication hardware designs with Z3's builtin floating point addition/multiplication. It is taking quite longer than expected, and it would be helpful to see what exactly it's doing in the process.
You can run it with higher verbosity:
z3 -v:10
This will print a lot of diagnostic info. But the output is unlikely to be readable/understandable unless you're really familiar with the source code. Of course, being an open source project, you can always study the code itself: https://github.com/Z3Prover/z3
If you're especially interested in e-matching and quantifiers, you can use the Axiom Profiler (GitHub source, research paper) to analyse a Z3 trace. The profiler shows instantiation graphs, tries to explain instantiations (which term triggered, which equalities were involved), and can even detect and explain matching loops.
For this, run z3 trace=true, and open the generated trace file in the profiler. Note that the tool is a research prototype: I've noticed some bugs, it seems to work more reliable on Windows than on *nix, and it might not always support the latest Z3 versions.
for a study on genetic programming, I would like to implement an evolutionary system on basis of llvm and apply code-mutations (possibly on IR level).
I found llvm-mutate which is quite useful executing point mutations.
As far as I have understood, the instructions get count/numbered, one can then e.g. delete a numbered instruction.
However, introduction of new instructions seems to be possible as one of the availeable statements in the code.
Real mutation however would allow to insert any of the allowed IR instructions, irrespective of it beeing used in the code to be mutated.
In addition, it should be possible to insert library function calls of linked libraries (not used in the current code, but possibly available, because the lib has been linked in clang).
Did I overlook this in the llvm-mutate or is it really not possible so far?
Are there any projects trying to /already have implement(ed) such mutations for llvm?
llvm has lots of code analysis tools which should allow the implementation of the afore mentioned approach. llvm is huge, so I'm a bit disoriented. Any hints which tools could be helpful (e.g. getting a list of available library functions etc.)?
Thanks
Alex
Very interesting question. I have been intrigued by the possibility of doing binary-level genetic programming for a while. With respect to what you ask:
It is apparent from their documentation that LLVM-mutate can't do what you are asking. However, I think it is wise for it not to. My reasoning is that any machine-language genetic program would inevitably face the "Halting Problem", e.g. it would be impossible to know if a randomly generated instruction would completely crash the whole computer (for example, by assigning a value to a OS-reserved pointer), or it might run forever and take all of your CPU cycles. Turing's theorem tells us that it is impossible to know in advance if a given program would do that. Mind you, LLVM-mutate can cause for a perfectly harmless program to still crash or run forever, but I think their approach makes it less likely by only taking existing instructions.
However, such a thing as "impossibility" only deters scientists, not engineers :-)...
What I have been thinking is this: In nature, real mutations work a lot more like LLVM-mutate that like what we do in normal Genetic Programming. In other words, they simply swap letters out of a very limited set (A,T,C,G) and every possible variation comes out of this. We could have a program or set of programs with an initial set of instructions, plus a set of "possible functions" either linked or defined in the program. Most of these functions would not be actually used, but they will be there to provide "raw DNA" for mutations, just like in our DNA. This set of functions would have the complete (or semi-complete) set of possible functions for a problem space. Then, we simply use basic operations like the ones in LLVM-mutate.
Some possible problems though:
Given the amount of possible variability, the only way to have
acceptable execution times would be to have massive amounts of
computing power. Possibly achievable in the Cloud or with GPUs.
You would still have to contend with Mr. Turing's Halting Problem.
However I think this could be resolved by running the solutions in a
"Sandbox" that doesn't take you down if the solution blows up:
Something like a single-use virtual machine or a Docker-like
container, with a time limitation (to get out of infinite loops). A
solution that crashes or times out would get the worst possible
fitness, so that the programs would tend to diverge away from those
paths.
As to why do this at all, I can see a number of interesting applications: Self-healing programs, programs that self-optimize for an specific environment, program "vaccination" against vulnerabilities, mutating viruses, quality assurance, etc.
I think there's a potential open source project here. It would be insane, dangerous and a time-sucking vortex: Just my kind of project. Count me in if someone doing it.
I used fixedpoint of z3, and I found that the running time of the fixedpoint is always different. Have you meet the same problem? why does this happen?
Sounds unexpected if you get large variety for the same code starting from the same state.
If you start from different states (that is, if you have made miscellaneous calls to Z3
whether over the text API or programmatic API, between the rounds). Z3 should otherwise not
exhibit hugely non-deterministic behavior. Non-deterministic behavior may arise from bugs, so
it will be appreciated if you can further and more precisely describe the scenario that is exercising it.
I am currently working on a parser and it seems that I have made a few mistakes druing the
follow set calculation. So I was wondering if someone know a good tool to calculate follow and first sets so I could skip/reevaluate this error prone part of the parser construction.
Take a look at http://hackingoff.com/compilers/predict-first-follow-set
It's an awesome tool to compute first and follow sets in a grammar. also, you can check your answer with this visualization tools:
http://smlweb.cpsc.ucalgary.ca/start.html
I found my mistake by comparing my first/follow-sets with the one generated by this web-app
Most parser generators that I've encountered don't have obvious means to dump this information, let alone dump it in a readable way. (I built one that does, for the reason you are suggesting, but it isn't available by itself and I doubt you want the rest of the baggage).
If your parser definition doesn't work, you mostly don't need to know these things to debug it. Staring at the rules amazingly enough helps; it also helps to build the two smallest grammar instances you can think of, one being something you expect to be accepted, and the other being a slight variant that should be rejected.
In spite of having a parser generator that will dump this information, I rarely resort to using it to debug grammars, and I've built 20-30 pretty big grammars with it.
I am considering to start using Pexpect. On Pexpects homepage I find this:
Q: Why not just use Expect?
A: I love
it. It's great. I has bailed me out of
some real jams, but I wanted something
that would do 90% of what I need from
Expect; be 10% of the size; and allow
me to write my code in Python instead
of TCL. Pexpect is not nearly as big
as Expect, but Pexpect does everything
I have ever used Expect for.
There is a 10% difference between Pexpect and Expect. So my question is what is this 10% difference. What is it that Expect can do that Pexpect can't.
That question ("What is it that Expect can do that Pexpect can't") is a bit misleading. It's not that Pexpect can't do things that Expect can; it's that Expect has a lot of extra support to make this kind of programming easier.
As an example, take the interact command which lets the user interact directly with the spawned process. In Pexpect, that's all interact does. (And that may be sufficient for your needs, as you say.) In contrast, Expect's interact has support for detecting patterns during an interact, hooking together multiple spawned processes, etc. Of course, you can do all this by coding it yourself. But your code will be longer - sometimes a lot longer because you'll essentially have to rewrite your own interact, you'll have to debug it, etc. In fact, you may have encountered these situations already but not realized how much simpler the equivalent Expect code would be.
Of course, the extra support may be more than offset by your preference for Python. :-P