Exploring expression debugging displays

Computation can be hard. You have something you want to calculate, and you have to tell the computer to do it. Sometimes you have the wrong idea, sometimes you have the right idea but write it down wrongly, and sometimes the computer seems to do something totally unexpected. You ask it to check that price > 0 and it says they aren't because you have a giveaway with a price of exactly zero, which makes you realise you meant price >= 0. The process of working out where the problem lies is known as debugging.

At its core, debugging is about bridging the gap between what you want to happen, and what is actually happening. A way to aid debugging is for the computer to provide useful information on what it did, so you can compare it to your mental model of what you think should be happening.

Visualising evaluation

Many text programming systems have a debugger that is oriented around a line-by-line model. It lets you see the state of the variables and then execute the next line and see the new state. The weaknesses in this model are:

• If you have complex expressions, executing the whole line at once is a large step. But debugger interfaces are rarely designed to show intermediate results from sub-expressions in a long line or step neatly through the different function calls in the expression.
• It is not possible to go backwards. Often you step and step and step and then step once too far and you have to start the whole process again.

Step-gret: the feeling when you spend a while stepping through your code in a debugger, and at a vital point, hit step-over rather than step-into.

— Neil Brown (@neilccbrown) June 24, 2019

Explaining expression evaluation

Since the Columnal software is based around expressions rather than imperative do-this-then-that programming, line-by-line execution doesn't make any sense. We need a display that works well for expressions. Consider the expression minimum([abs(-3), round(0.7) - abs(-3)]). Here's a few ways that we could display the evaluation:

• Linear display inner to outer, substituting results:
  • abs(-3) is 3
  • round(0.7) is 1
  • 1 - 3 is -2
  • minimum([3, -2]) is -2

  Here, the results are substituted into the later expressions. minimum([3, -2]) appears nowhere in the original expression, for example, but is a consequence of substituting in earlier results.

• Linear display outer to inner, no substitution of results:
  • minimum([abs(-3), round(0.7) - abs(-3)]) is -2
  • [abs(-3), round(0.7) - abs(-3)] is [3, -2]
  • abs(-3) is 3
  • round(0.7) - abs(-3) is -2
  • round(0.7) is 1

  Here, the expressions are given in full, with the values implicitly taken from the expressions listed later. Note that in this display and the previous one, repeated sub-expressions (here: abs(-3)) are only listed once because the result is the same every time.

• Tree display, no substitution of results:
  • minimum([abs(-3), round(0.7) - abs(-3)]) is -2
  • [abs(-3), round(0.7) - abs(-3)] is [3, -2]
  • abs(-3) is 3
  • round(0.7) - abs(-3) is -2
  • round(0.7) is 1
  • abs(-3) is 3

  Here the expressions are nested to provide a tree evaluation. This time, repeated expressions are shown every time they are executed.

I'm not sure which I prefer. The tree feels right but can easily get quite deep for slightly longer expressions, and it's a shame to repeat sub-expression results. Without substitution it feels like a lot of mental juggling to track the values, but with substitution you have to do the same juggling to remember which value came from which expression. However, the nice thing about having an interface rather than a static display is that we can help the user to understand by adding interactivity.

Linking

The current design in Columnal has links that highlight all instances of that sub-expression when hovered over, to help the user match a sub-expression to its evaluation:

This still feels like it requires some mental juggling on the user's part to do the substitutions throughout the expression, though, so I want to explore other possibilities.

Substitution

An alternative is to allow the user to manually toggle the substitutions, which I've briefly prototyped:

The jerky transition in the prototype is a pain as it makes it hard to see the changes. If that was fixed, this may be a useful interface.

Source locations

Something that is specific to Columnal's context is that the only external inputs to the expressions are the values from cells in other tables. So it's useful to be able to know which cells were actually involved in a particular result. To explain: when analysing the dependent cells in a spreadsheet, you can get a display of all the cells referred to in the formula – so if you do an array formula A1:A10>0 and the answer is false you'll see all ten cells highlighted as the input. But in Columnal if you do the equivalent calculation for price > 0 then it tells you the cell that caused the result to false:

Related work

The Ocamli system (paper) has an interface similar to the ones presented here, that shows the full expression and underlines the next subexpression to be evaluated:

For longer expressions, repeating the whole expression just to focus on one step feels like overkill, though. DrRacket has a stepping interface that lets you see the same thing but in a side-by-side comparison between each consecutive pairs of steps:

A disadvantage of linearising the evaluation like this (and in some of the Columnal suggestions shown previously) is that although it may be an accurate operational semantics, the user may not be interested in an entire sub-expression, so it would be good to be able to ignore sub-expressions and focus in on where the user thinks the bug lies.

Summary

I'm not totally happy with either the linking interface or the substitution interface for Columnal as they are shown above. It still feels like a bit too much effort on the user's part to keep track of all the different parts when the expression is non-trivial, and much of it may be uninteresting to the user at a given moment.

I suspect that often, the user has in mind which part of the expression is going wrong so it may be better to start with all lines beneath the expression result hidden (folded up), and clicking on each highlighted expression part (like in the linking version) expands it to show more details.

---

Extra note: performance

A Columnal project can have lots of expression calculations. You may wonder about performance and memory use if every evaluation has to store all its intermediate results. The answer is that when expressions are calculated normally, none of this extra information is stored. But we take advantage of the fact that Columnal expressions are pure, which means that if they are evaluated a second time, they will give the same result without affecting or being affected by other values. So to provide the explanation interface shown above, we re-evaluate the expression in a mode that retains all this extra information.