There are several posts regarding the conversion of bit vectors
to integers (and vice versa) in z3. See for example here,
here and here.
The documentation says that Z3_mk_bv2int is uninterpreted:
"...This function is essentially treated as uninterpreted.
So you cannot expect Z3 to precisely reflect the semantics of this
function when solving constraints with this function..."
However, I could not find a simple example where it does fail
to reflect the expected semantics.
For instance, whenever I use queries like this:
(declare-const s String)
(declare-const someBitVec10 (_ BitVec 10))
(assert (= s "74g\x00!!#2#$$"))
(assert (str.in.re (str.at s (bv2int someBitVec10)) (re.range "1" "3")))
(check-sat)
(get-value (s someBitVec10))
I get a correct answer (index should 7, and it is)
sat
((s "74g\x00!!#2#$$")
(someBitVec10 #b0000000111))
Could anyone please provide a simple example where z3's
bv2int and/or int2bv fail?? thanks!
This issue is now resolved, as it turns out that both int2bv and bv2int are indeed interpreted. The documentation hasn't been updated, and this might have caused the confusion (at least in my case it did). All the details are in this GitHub/z3/issues post.
Related
I noticed that if I create my own array type that stores bitvectors and assert the first array update axiom, simple assertions afterwards fail to find a solution (my example below neither returns sat nor unsat but just keeps running):
(declare-sort MyArray)
; Indices into the array
(declare-sort Id)
; Returns the value in the array located at the specified index
(declare-fun index
(MyArray Id)
(_ BitVec 8))
; Updates the array so that the provided value is stored at the specified index
(declare-fun upd
(MyArray Id (_ BitVec 8))
MyArray)
; First array update axiom
(assert (forall ((a MyArray) (i Id) (v (_ BitVec 8)))
(=
(index (upd a i v) i)
v)))
(declare-const x Int)
(declare-const y Int)
(echo "")
(echo "Sanity check, should be sat:")
(assert (= x y))
(check-sat)
However, if I instead specify that my array stores a custom sort z3 finds a solution very quickly:
(declare-sort MyArray)
; Indices into the array
(declare-sort Id)
; Values stored in the array
(declare-sort Elem)
; Returns the value in the array located at the specified index
(declare-fun index
(MyArray Id)
Elem)
; Updates the array so that the provided value is stored at the specified index
(declare-fun upd
(MyArray Id Elem)
MyArray)
; First array update axiom
(assert (forall ((a MyArray) (i Id) (v Elem))
(=
(index (upd a i v) i)
v)))
(declare-const x Int)
(declare-const y Int)
(echo "")
(echo "Sanity check, should be sat:")
(assert (= x y))
(check-sat)
Does anyone know why this is the case? It's possible that z3 gets caught in some kind of instantiation loop (since the upd function both takes and returns MyArray sort), but I'm surprised that it only seems to get tripped up with bitvectors as the elements. Is this related to Nikolaj's answer that the quantifier elimination tactic is currently fairly simplistic when it comes to bit-vectors?
I'm using bitvectors because my problem ultimately involves some bitvector operations (especially bvxor). Is it better just to define my own operations and essentially recreate part of the theory of bitvectors? Or is there a better way to go about this (than mixing quantifiers, bitvectors, and part of the theory of arrays)? I'm really just interested in operations on bytes so all my bitvectors are of length 8.
I don't think there's a good reason why z3 is not terminating on the first program you gave here. Running with z3 -v:10 suggests it gets into some sort of an unproductive loop, that the second version avoids. I think you should report this at https://github.com/Z3Prover/z3/issues. Even though it's not strictly speaking a "bug," it's surprising behavior and the developers might want to look at it. (Please report back what you find.)
Regarding your second question: Do not reinvent what z3 already has support for! Use the internal arrays; they have a custom decision procedure and it has gone years of tuning. The introduction of quantified axioms will no doubt create more work than is necessary. Does something go wrong if you use internal arrays? Why would you not use them anyhow? Only look at your own axiomatization, if the internal built-in one isn't working well. (Even then, I'd first check with developers to see why.)
I am experimenting with z3. I am surprised that, when setting the logic to LIA, z3 can return real numbers. For example, the following:
from z3 import *
sol = SolverFor("LIA")
vB = Real('vB')
sol.add(vB < 3)
sol.add(vB > 2)
sol.check()
print(sol.model())
returns:
[vB = 5/2]
Can someone please explain how it can be the case?
Strictly speaking, this is a "bug" in z3. Here's the equivalent SMT script:
(set-logic LIA)
(declare-fun vB () Real)
(assert (< vB 3.0))
(assert (> vB 2.0))
(check-sat)
(get-model)
If you run this with cvc4/cvc5, you get:
(error "Parse Error: a.smt2:2.23: Symbol 'Real' not declared as a type
(declare-fun vB () Real)
^
")
And Yices says:
(error "at line 2, column 20: undefined sort: Real")
CVC and Yices are correct here, since you restricted the logic to "LIA", the name "Real" is no longer defined. Z3, however, is playing loose here, and allowing reals by default. (It internally uses logics to pick which algorithms to run, but it's not always consistent in making sure the logic restricts the names available.)
You can report this at the z3 issue tracker here: https://github.com/Z3Prover/z3/issues. I suspect they're already aware of this, but would be good to have it recorded.
I have tried to solve 2^x=4 with Z3, by putting the following on the Z3 website: https://rise4fun.com/z3/tutorial.
(declare-const x Real)
(declare-const y Real)
(declare-const z Real)
(assert (=(^ 2 x) 4))
(check-sat)
(get-model)
Z3 produced
unknown
(model
)
Am I misusing Z3?
Problems involving exponentials are typically beyond the reach of z3, or general-purpose SMT solvers. This doesn't mean that they cannot solve them: Theory of reals is decidable. But they may not have the right "heuristics" to kick in to answer every query involving exponentials as sat/unsat. You can search stack-overflow for keywords like nnf, non-linear, etc., to see a plethora of questions regarding queries that involve such difficult terms.
Having said that, there's a separate line of research called delta-satisfiability that can help with these sorts of problems to a great extent. Note that delta-satisfiability is different than regular satisfiability. It means either the formula is satisfiable, or a delta-perturbation of it is. The most prominent such solver is dReal, and you can read all about it here: http://dreal.github.io/
For your query, dReal says:
[urfa]~/qq>dreal a.smt2
delta-sat with delta = 0.001
(model
(define-fun x () Real [2, 2])
(define-fun y () Real [-0.0005, 0.0005])
(define-fun z () Real [-0.0005, 0.0005])
)
(You didn't actually use y and z in your query, so you can ignore those outputs.)
As you can see dReal determines x must be in the range [2, 2], i.e., it must be 2. But it also says delta-sat with delta = 0.001: This means it has ensured the correctness within that factor. (You can tweak the factor yourself, making it arbitrarily small, but not zero.) When you have problems that arise from physical systems, delta-sat is the right choice in modeling them in the SMT-world.
Using smtlib I would like to make something like modulo using QF_UFNRA. This disables me from using mod, to_int, to_real an such things.
In the end I want to get the fractional part of z in the following code:
(set-logic QF_UFNRA)
(declare-fun z () Real)
(declare-fun z1 () Real)
(define-fun zval_1 ((x Real)) Real
x
)
(declare-fun zval (Real) Real)
(assert (= z 1.5));
(assert (=> (and (<= 0.0 z) (< z 1.0)) (= (zval z) (zval_1 z))))
(assert (=> (>= z 1.0) (= (zval z) (zval (- z 1.0)))))
(assert (= z1 (zval z)))
Of course, as I am asking this question here, implies, that it didn't work out.
Has anybody got an idea how to get the fractional part of z into z1 using logic QF_UFNRA?
This is a great question. Unfortunately, what you want to do is not possible in general if you restrict yourself to QF_UFNRA.
If you could encode such functionality, then you can decide arbitrary Diophantine equations. You would simply cast a given Diophantine equation over reals, compute the "fraction" of the real solution with this alleged method, and assert that the fraction is 0. Since reals are decidable, this would give you a decision procedure for Diophantine equations, accomplishing the impossible. (This is known as Hilbert's 10th problem.)
So, as innocent as the task looks, it is actually not doable. But that doesn't mean you cannot encode this with some extensions, and possibly have the solver successfully decide instances of it.
If you allow quantifiers and recursive functions
If you allow yourself quantifiers and recursive-functions, then you can write:
(set-logic UFNRA)
(define-fun-rec frac ((x Real)) Real (ite (< x 1) x (frac (- x 1))))
(declare-fun res () Real)
(assert (= (frac 1.5) res))
(check-sat)
(get-value (res))
To which z3 responds:
sat
((res (/ 1.0 2.0)))
Note that we used the UFNRA logic allowing quantification, which is required here implicitly due to the use of the define-fun-rec construct. (See the SMTLib manual for details.) This is essentially what you tried to encode in your question, but instead using the recursive-function-definition facilities instead of implicit encoding. There are several caveats in using recursive functions in SMTLib however: In particular, you can write functions that render your system inconsistent rather easily. See Section 4.2.3 of http://smtlib.cs.uiowa.edu/papers/smt-lib-reference-v2.5-draft.pdf for details.
If you can use QF_UFNIRA
If you move to QF_UFNIRA, i.e., allow mixing reals and integers, the encoding is easy:
(set-logic QF_UFNIRA)
(declare-fun z () Real)
(declare-fun zF () Real)
(declare-fun zI () Int)
(assert (= z (+ zF zI)))
(assert (<= 0 zF))
(assert (< zF 1))
(assert (= z 1.5))
(check-sat)
(get-value (zF zI))
z3 responds:
sat
((zF (/ 1.0 2.0))
(zI 1))
(You might have to be careful about the computation of zI when z < 0, but the idea is the same.)
Note that just because the encoding is easy doesn't mean z3 will always be able to answer the query successfully. Due to mixing of Real's and Integer's, the problem remains undecidable as discussed before. If you have other constraints on z, z3 might very well respond unknown to this encoding. In this particular case, it happens to be simple enough so z3 is able to find a model.
If you have sin and pi:
This is more of a thought experiment than a real alternative. If SMTLib allowed for sin and pi, then you can check whether sin (zI * pi) is 0, for a suitably constrained zI. Any satisfying model to this query would ensure that zI is integer. You can then use this value to extract the fractional part by subtracting zI from z.
But this is futile as SMTLib neither allows for sin nor pi. And for good reason: Decidability would be lost. Having said that, maybe some brave soul can design a logic that supported sin, pi, etc., and successfully answered your query correctly, while returning unknown when the problem becomes too hard for the solver. This is already the case for nonlinear arithmetic and the QF_UFNIRA fragment: The solver may give up in general, but the heuristics it employs might solve problems of practical interest.
Restriction to Rationals
As a theoretical aside, it turns out that if you restrict yourself to rationals only (instead of actual reals) then you can indeed write a first-order formula to recognize integers. The encoding is not for the faint of heart, however: http://math.mit.edu/~poonen/papers/ae.pdf. Furthermore, since the encoding involves quantifiers, it's probably quite unlikely that SMT solvers will do well with a formulation based on this idea.
[Incidentally, I should extend thanks to my work colleagues; this question made for a great lunch-time conversation!]
I'm confused and struggling to understand how two different input formats for Z3 fixedpoint engine are related. Short example: suppose I want to prove the existance of negative numbers. I declare a function that returns 1 for non-negative numbers and 0 for negative and then asking solver to fail if there are arguments for which function returns 0. But there is one restriction: I want solver to respond sat when there exists at least one negative number and unsat if all numbers are non-negative.
It is trivially with using declare-rel and query format:
(declare-rel f (Int Int))
(declare-rel fail ())
(declare-var n Int)
(declare-var m Int)
(rule (=> (< n 0) (f n 0)))
(rule (=> (>= n 0) (f n 1)))
(rule (=> (and (f n m) (= m 0)) fail))
(query fail)
But it becomes tricky while using pure SMT-LIB2 format (with forall). For example, straightforward
(set-logic HORN)
(declare-fun f (Int Int) Bool)
(declare-fun fail () Bool)
(assert (forall ((n Int))
(=> (< n 0) (f n 0))))
(assert (forall ((n Int))
(=> (>= n 0) (f n 1))))
(assert (forall ((n Int) (m Int))
(=> (and (f n m) (= m 0)) fail)))
(assert (not fail))
(check-sat)
returns unsat. Unsurprisingly, changing (= m 0) to (= m 1) results the same. We can get sat only implying fail from (= m 2). The problem is that I can't understand, how to ask solver using this format.
How I'm understanding it at the moment, while using forall-form we can ask to find only ∀-solutions, i.e. the answer sat means that solver managed to find interpretation (or invariant) satisfiying all assertions for all values, and unsat means that there are no such functions. In other words, it tries to prove, putting the 'proof' (the invariant) into the model (obviously, when sat).
On the contrary, when querying the solution in the declare-rel format solver searches the solution for some variables, just like the constraints are under the ∃-quantifier. In other words, it gives the counter-example. It can only print the invariant in case of unsat.
I have a couple of questions:
Am I understanding it correct? I feel like I miss some key ideas. For example, a general idea of how to express (query ...) in terms of (assert (forall ...)) will be really helpfull (and will answer question 2 automaticly).
Is there a way to solve such ∃-constraints (outputting sat when counterexample was found) with pure SMT-LIB2 format? If yes then how?
First of all, the format that uses "declare-rel", "declare-var", "rule" and "query" is a custom extension to SMT-LIB2. The "declare-var" feature is convenient for omitting bound variables from multiple rules. It also allows formulating Datalog rules with stratified negation and the semantics of this is what you should expect from stratified negation. By convention it uses "sat" to indicate that a query has a derivation, and "unsat" that no derivation exists for a query.
It turns out that standard SMT-LIB2 can express pretty much what you want for
Horn clauses without negation. Rules become implications and queries are implications of the form: (=> query false), or as you wrote it (not query).
A derivation in the custom format corresponds to a proof of the empty clause (e.g., proof of "query", which then proves "false"). So existence of a derivation means that the SMT-LIB2 assertions are "unsat". Conversely, if there is an interpretation (a model) for the Horn clauses, then such a model establishes that there is no derivation. The clauses are "sat".
In other words:
"sat" for datalog extension <=> "unsat" for SMT-LIB2 formulation
"unsat" for datalog extension <=> "sat" for SMT-LIB2 formulation
The advantage of using the pure SMT-LIB2 format, when it applies, is that
there are no special syntax extensions. These are plain SMT formulas and
others who wish to solve this class of formulas don't have to write special
extensions, they just have to ensure that the solvers that are tuned to
Horn clauses recognize the appropriate class of formulas. (Z3's implementation
of the HORN fragment does allow some flexibility in writing down Horn clauses.
You can have disjunctions in the bodies and you can have Curried implications).
There is one drawback with using the SMT-LIB2 format that the rule-based format helps with: when there is a derivation of the query, then the rule-based format has pragmas for printing elements of a tuple. Note that in general the query relation can take arguments. This feature is useful for finite domain relations.
Your example above uses integers, so the relations are not finite domain, but examples in the online-tutorial contain finite domain instances.
Now a derivation of a query also corresponds to a resolution proof. You can extract a resolution proof from the SMT-LIB2 case, but I have to say it is rather
convoluted and I have not found a way to use it effectively. The "duality" engine for Horn clauses generates derivations in a more accessible format than
the default proof format of Z3. Either way, it is likely that users run into obstacles if they try to work with the proof certificates because they are rarely used. The rule-based format does have another feature that assembles a set of predicates with instances that correspond to a derivation trail. It is easier to eyeball this output.