When using Z3 in incremental mode, does the solver discard all lemmas after performing a pop() operation?
In 2012, Leonardo de Moura affirmed that Z3 in fact discards lemmas after a pop() [1].
However my benchmarks using Z3 4.3.1 do not show statistically significant performance slowdown of a check-sat after a pop().
[1] Efficiency of constraint strengthening in SMT solvers
Yes, upon pop() Z3 discards all the lemmas that were derived within the scope(s) of that pop and its corresponding push.
Sorry about the confusion, the old post was not completely clear about that and has since been updated.
Related
While reading Extending Sledgehammer with SMT solvers I read the following:
In the original Sledgehammer architecture, the available lemmas were rewritten
to clause normal form using a naive application of distributive laws before the relevance filter was invoked. To avoid clausifying thousands of lemmas on each invocation, the clauses were kept in a cache. This design was technically incompatible with the (cache-unaware) smt method, and it was already unsatisfactory for ATPs, which include custom polynomial-time clausifiers.
My understanding of SMT so far is as follows: SMTs don't work over clauses. Instead, they try to build a model for the quantifier-free part of a problem. The search is refined by instantiating quantifiers according to some set of active terms. Thus, indeed no clausal form is needed for SMT solvers.
We rewrote the relevance filter so that it operates on arbitrary HOL formulas, trying to simulate the old behavior. To mimic the penalty associated with Skolem functions in the clause-based code, we keep track of polarities and detect quantifiers that give rise to Skolem functions.
What's the penalty associated with Skolem functions? I could understand they are not good for SMTs, but here it seems that they are bad for ATPs too...
First, SMT solvers do work over clauses and there is definitely some (non-naive) normalization internally (e.g., miniscoping). But you do not need to do the normalization before calling the SMT solver (especially, since it will be more naive and generate a larger number of clauses).
Anyway, Section 6.6.7 explains why skolemization was done on the Isabelle side. To summarize: it is not possible to introduce polymorphic constants in a proof in Isabelle; hence it must be done before starting the proof.
It seems likely that, when writing the paper, not changing the filtering lead to worse performance and, hence, the penalty was added. However, I tried to find the relevant code simulating clausification in Sledgehammer, so I don't believe that this happens anymore.
In many programming languages, branching efficiency is dependent on the order in which the clauses are provided. E.g., in Python,
if p or q :
will branch into the if statement as soon as p evaluates to true, so it is generally a good idea to provide computationally light clauses first. I'm wondering if the same is true for satisfiability checking in Z3. In other words, is there any difference between checking And(P, Q) and And(Q, P), provided that one of the formulas is significantly more complex than the other?
Yes, it make can make a difference what order you specify the clauses. However, there is probably not much to be gained by rearranging the clauses when it comes to Z3. Also, it might be difficult (and perhaps not worth your while) to determine a generic "optimal order" beforehand.
With other SMT solvers, I have observed what you are describing in an extreme fashion. With some solvers, it can make or break the computation, in the sense that I have seen certain cases where a solver times out when the clauses are presented in one order, but quickly finds a solution when the same clauses are presented in a different order. However, to me, this is an indication of a poorly designed solver. Z3 and other modern SMT solvers such as CVC4 are much less vulnerable to this kind of problem than older or less robust SMT solvers.
Regarding actually finding the optimal ordering, what constitutes "computationally light" or "more complex" for an SMT solver is different than a traditional computer language. What really matters is whether or not a certain clause in a conjunction is likely to lead to a conflict, or whether a particular clause in a disjunction is likely to lead to a solution. This is akin to trying to find your way out of a maze -- if you make a wrong turn early on, then you might spend a long time following a dead end, before you finally decided to go back to the beginning and take a different branch. Again, modern SMT and SAT solvers have techniques for mitigating this.
Also, When it comes to Z3 specifically, if Z3 could be made more efficient in real-world applications by sorting the clauses in terms of some human-understandable and generic measure of complexity, then that has probably already added as a pre-processing step to Z3. In general, for such complicated optimization questions where there are multiple factors at play (e.g. the particulars of the Z3 implementation) you have to try multiple things and benchmark over a suite of test examples relevant for your application.
As a bit of evidence for some of my claims above, I wrote this little example SMT problem:
(declare-fun p () Int)
(declare-fun q () Int)
(declare-fun n () Int)
(assert (> p 1))
(assert (> q 1))
(assert (and (= n 18679565357) (= n (* p q))))
(check-sat)
(get-value (p q n))
(exit)
Swapping the order of the clauses in the and statement makes very little difference on my system (less than 1%). If I break it up into two separate assertions, then the difference is larger, but to know if this difference holds in general (across different SMT problems of this class) I would have to run a benchmark suite with many such problems.
I'm experimenting with Z3 tactics. I noticed that the solver returned by
context.MkSimpleSolver()
has a different performance profile than the solver
context.MkSolver(context.MkTactic("smt"))
I understand that the "smt" tactic is supposed to perform the most general algorithm that Z3 has. So both these solvers should have identical output. Given this, why do they have different performance profiles?
I am testing this on a formula that is quantified over a bitvector. The quantifier body makes use of integers and bitvectors. The simple solver returns immediately with sat. The other one takes longer than I was willing to wait.
(If it helps here is a little background: I am trying to make Z3 bit-blast all integer terms to bitvectors hoping that this speeds things up. All integers involved are constrained to [0, 3]. I'm trying to get the bit-blast tactic to "take".)
Does z3's SAT solver(s) obtain a complete assignment to the propositional(ized) part of an SMT problem before doing a theory consistency check? In particular, I am curious to know what is done by default for each of the following background theories/combination (if this is theory-dependent): Linear Real Arithmetic (LRA), Linear Integer Real Arithmetic (LIRA), Non-Linear Integer Real Arithmetic (NIRA)? Also, where in the actual code (codeplex stable z3 v4.3.1) is a propositional literal (heuristically) decided by the SAT solver?
No, Z3 does not obtain a complete assignment before doing theory consistency checks.
However, it delays "expensive" checks. "Expensive" checks are performed in a step called final_check that is performed only when a (complete) proprositional assignment is produced. Here the word "expensive" is relative. Linear real arithmetic consistency checks can be quite expensive due to big number arithmetic computations, but they are considered "cheap" in Z3.
Linear real arithmetic checks are done eagerly. Nonlinear and linear integer arithmetic checks are done at the final_check step.
Note that Z3 contains more than one solver. The behavior above is for the one implemented in the directory smt. The nonlinear real arithmetic solver (nlsat directory) works in a completely different way, and it does not use the final_check approach described above.
As far as I understand, Z3, when encountering quantified linear real/rational arithmetic, applies a form of quantifier elimination described in Bjørner, IJCAR 2010 and more recent work by Bjørner and Monniaux (that's what qe_sat_tactic.cpp says, at least).
I was wondering
Whether it still works if the formula is multilinear, in the sense that the "constants" are symbolic. E.g. ∀x, ax≤b ⇒ ax ≤ 0 can be dealt with by separating the cases a<0, a=0 and a>0. This is possible using Weispfenning's virtual substitution approach, but I don't know what ended up being implemented in Z3 (that is, whether it implements the general approach or the one restricted to constant coefficients).
Whether it is possible, in Z3, to output the result of elimination instead of just solving for one model. There might be a Z3 tactic to do so but I don't know how this is supposed to be requested.
Whether it is possible, in Z3, to perform elimination as described above, then use the new nonlinear solver to obtain a model. Again, a succession of tactics might do the trick, but I don't know how this is supposed to be requested.
Thanks.
After long travels (including a travel where I met David at a conference), here is a short summary to answer the questions as they are posed.
There is no specific support for multi-linear forms.
The 'qe' tactic produces results of elimination, but may as a side-effect decide satisfiablity.
This is a very interesting problem to investigate, but it is not supported out of the box.