The Diviner

A complement to The Oracle for compilers class. Given some test cases, this program automatically queries the oracle and tests your compiler!

More spefically this program:

  • Runs your *.cshanty test inputs thru the oracle & collects its response to give a “true output”; writes *.truth to test dir.
  • Runs all your test inputs through your compiled compiler & collects outputs to give an “actual output”; writes *.actual to test dir.
  • For each (*.truth, *.actual) pair, essentially runs a more nuanced version of diff to check if your compiler is producing correct output; provides score and details.

Formal Usage:

  • python3 diviner.py <diviner version> <path to cshantyc> <path to dir containing *.cshantyc test cases> <-l, optional>
  • python3 diviner.py --help
  • python3 diviner.py d5 -h

Example usage:

  • python3 diviner d5 cshantyc my_tests/
  • python3 diviner d5 cshantyc my_tests/ --lazy-query
  • python3 diviner d5 cshantyc my_tests/ -l
  • python3 diviner d6 --help
  • python3 diviner d6 -h

…where my_tests/ might contain test1.cshanty, test2.cshanty, etc.

Dependencies:

  • python3 (https://www.python.org/downloads/)

  • bs4 pip3 install bs4

  • tqdm pip3 install tqdm

  • …the rest should be standard.

  • You can also (alternatively) run make install or pip3 install -r requirements.txt for convenience

  • While we are on the topic of make directives, make clean removes all compiled *.pyc files

Notes:

  • For a given *.cshanty, if this program already detects a *.truth it will still requery the oracle for that test. Setting --lazy-query=1 in cmd line args will instead skip it to avoid excess time and requests.
  • Currently, only type analysis version for P5 is developed (diviner version = d5). However, feel free to fork and add new version d[1-8] in versions/. If you inherit from core/diviner.py/DivinerBase it should be very trivial to do other versions.

GitHub

View Github