Add a new delegate to allow API tracing

[daniel-raffler]

This is a preliminary draft for adding API tracing to JavaSMT with the help of a new delegate. The idea is to record all API calls and generate a new Java program from them. By running this program the exact sequence of calls can then be recreated. The main application here is debugging, where the traces allow us to create easy to reproduce examples for solver errors. This is especially useful when the error occurs as part of a larger program where it can be hard to pin down the exact sequence of JavaSMT calls that are needed to trigger the bug.

We use a new delegate to implement this feature. Setting solver.trace to true will enable tracing, and the output will be stored in a file called trace*.java

TODO

• Finish the implementation. Currently we only have (parts of) the ArrayFormulaManager, IntegerFormulaManager, BooleanFormulaManager, UFManager and ProverEnvironment
• Write the trace to a file while it's being created. We'll need this to debug segfaults as the trace is otherwise lost done
• Consider adding an option to skip duplicate calls. (The trace is currently way too long) Fixed, but not committed yet
• Write a simple delta-debugger to shrink the trace down even further3 Maybe later..

We're now using ddSmt, see comment #505 (comment)

Things left to do

• Add support for missing formula managers in the script
  Still missing: floating point, quantifier, strings and separation logic. At least the first two should still be added before merging
• Handle solver options in the script
• Fix undo point in the trace logger
  Done, but we should double check the Rebuilder
• Merge Add support for indexed functions #507
• Add user documentation for debugging with the tracer (see the comment below)
• Update the changelog
• Run some tests in CPAchecker to see if there are still issues in the script
• Add support for quantifiers and interpolation to the Smtlib translation script
• Test with more solvers

[daniel-raffler]

Write a simple delta-debugger to shrink the trace down even further

I started working on the delta-debugger today and wrote a python script to reduce the size of the traces. So far it does little more that some dead-code elimination, but that's already enough to bring down the size of the trace by a factor of ten. I believe that another factor of two should be possible with some aggressive optimization.

The issue now is that I don't quite know where to put such a script in JavaSMT. We could handle this as a separate project, or maybe include it in the JavaSMT source tree, similar to the Example projects. However, neither really seems quite ideal.

@baierd, @kfriedberger: What is your opinion?

Here is the file in question:

#!/usr/bin/env python3
import re
import sys
from collections import defaultdict
from pathlib import Path


# Read a trace file
def readTrace(path):
    with open(path) as file:
        return [line.rstrip() for line in file]


# Build a map with line numbers for all variable definitions
def getLinesForDefinitions(trace):
    lineNumber = 1
    lineDefs = dict()
    for line in trace:
        if line.find('=') >= 0:
            leftSide = line[0:(line.find('=') - 1)]
            name = re.match('var (.*)', leftSide)
            lineDefs[name.group(1)] = lineNumber
        lineNumber = lineNumber + 1
    return lineDefs


# Build a dependency graph for the definitions
# Maps from variables to the places where they are used
def buildDependencies(lineDefs, trace):
    lineNumber = 1
    deps = defaultdict(list)
    for line in trace:
        expr = line[(line.find('=') + 2):] if line.find('=') >= 0 else line
        object = expr[0:expr.find('.')]
        if object[0].islower():
            deps[lineDefs[object]].append(lineNumber)
        # FIXME Parse the expression to get the variables
        for m in re.finditer('(config|logger|notifier|var[0-9]+)', expr):
            deps[lineDefs[m.group()]].append(lineNumber)
        lineNumber += 1
    return deps


# Collect all top-level statements
# Top-level statements are:
#  *.addConstraint(*)
#  *.isUnsat()
#  *.getModel()
#  *.asList()
# FIXME Finish this list
def usedTopLevel(lineDefs, trace):
    tl = set()
    for line in trace:
        m = re.fullmatch(
            'var (var[0-9]+) = (var[0-9]+).(isUnsat\\(\\)|getModel\\(\\)|asList\\(\\)|addConstraint\\((var[0-9]+)\\));',
            line)
        if m != None:
            tl.add(lineDefs[m.group(1)])
    return tl


# Calculate the closure of all used definitions, starting with the top-level statements
def usedClosure(tl, deps):
    cl = set()
    st = set(tl)
    while cl.union(st) != cl:
        cl = cl.union(st)
        st = set()
        for (key, val) in deps.items():
            if set(val).intersection(cl) != set():
                st.add(key)
    return cl


# Keep only statements and definitions that are used
def filterUnused(used, trace):
    lineNumber = 1
    reduced = []
    for line in trace:
        if line.find('=') == -1 or lineNumber in used:
            reduced.append(line)
        lineNumber += 1
    return reduced


# Remove all definitions that are not used (recursively)
def removeDeadCode(trace):
    lineDefs = getLinesForDefinitions(trace)
    deps = buildDependencies(lineDefs, trace)
    tl = usedTopLevel(lineDefs, trace)
    cl = usedClosure(tl, deps)
    return filterUnused(cl, trace)


# We'll use multiple passes to reduce the size of the trace:
# 1. Read the trace
# 2. Remove unused code
# 3. Remove unnecessary toplevel commands
# 4. Loop: Remove aliasing (by duplicating the definitions)
# 5.    Loop: Reduce terms
# 6. Remove unused prover environments
if __name__ == '__main__':
    arg = sys.argv
    if not len(sys.argv) == 2:
        print('Expecting a path to a trace file as argument')
        exit(-1)

    path = Path(sys.argv[1])
    if not (path.is_file()):
        print(f'Could not find file "{path}"')
        exit(-1)

    # TODO Implement steps 3-6
    # TODO Check that the reduced trace still crashes

    trace = readTrace(path)
    for line in removeDeadCode(trace):
        print(line)

The idea is to run JavaSMT with solver.trace=true to collect a trace of the crash, and then use the script to reduce the trace. Since we're "crashing" (posssibly with a segfault) there doesn't seem to be a good way to do this in one go

[daniel-raffler]

I believe this is ready for review now

The approach now uses a two step process: We first run the program with trace=true to generate a JavaSMT trace and then use a script to convert this trace into SMTLIB output. This then allows us to use an existing delta-debugger like ddSmt to reduce the size of the SMTLIB file, as the trace would otherwise grow fairly large

Some examples for how to use the tracing mode can be found in my earlier comments here and here. Notice that traces are now no longer generated automatically for the tests. So the second example requires you to first revert 3068d81 and ce46f96

The last CI run with tracing enabled can be found here. On CVC5, MathSAT and Z3 most of the test are passing just fine. Other solvers still have more issues, mostly due to limitations in the visitor and heavy formula rewriting

Test results with tracing enabled:

• Bitwuzla
• CVC5
• MathSAT5
• OpenSMT
• Princess
• SmtInterpol
• Yices2

The tracing delegate now supports all formula managers in JavaSMT. Trace translation to SMTLIB is still more limited and we don't support enumerations, strings or separation logic at this point. However, these could easily be added. Translation to SMTLIB is also not possible if more than one SolverContext is needed, or if multiple ProverEnvironments are used at the same time (sequential use is fine)

For testing purposes I did run part of svcomp in CPAchecker with tracing enabled. You can find the results here. The largest issue is the lack of support for bvextract while tracing. However, this can be fixed later, once we figured out how to handle indexed functions in JavaSMT. Other than that there were no new exceptions and the generated traces seem to run fine on MathSAT

@baierd, @kfriedberger
Is there anything that is still missing from this PR?