Why Barnaby Never Just Says “Wrong”: Building Clues That Teach

A first grader reads: "Pia has 2 balloons. A friend gives her 3 more balloons. How many balloons does Pia have now?" She types 7.

Most math apps answer the same way: a red mark and "Try again." She tries 6. "Try again." Then the app shows her the answer, 5, and moves on to the next problem.

What did she learn? That 7 and 6 are not 5. Nothing about why, and nothing she can use on the next problem. Her next answer will be another guess.

In SmartyPants, Barnaby the owl answers differently. After her first try he says: "The story doesn't tell us all of Pia's balloons now. We work it out from the bar!" If she misses again: "Start at 3 and count on 2 more. Use your fingers!" Only on a third try does he show the answer, and he gives the reason first: "2 and 3 together make 5."

This article explains how we build those clues, and why we think they are the most important thing in the product.

A wrong answer is information

Children's wrong answers are rarely random. Researchers have catalogued error patterns in children's arithmetic for decades (Radatz, 1979; Ashlock, 2010), and Cognitively Guided Instruction research has mapped how children's thinking about story problems grows (Carpenter et al., 1999). When a first grader misses Pia's problem, the wrong number usually points to the idea that went wrong:

"Try again" treats all four the same way. Each one needs a different conversation. A teacher sitting next to the child would never answer all four with the same words, and an app shouldn't either.

How Barnaby reads the mistake

Every problem in our Grade 1 word-problem lessons is built from its structure: which numbers the story gives, which one is missing, and how they relate. From that structure we generate a message for each common mistake. Every problem gets them, not only the ones someone remembered to write by hand.

• Off by one: "So close! Let's count again, slowly."
• A story number copied into the answer: "That number is already in the story: it tells us the balloons Pia had. The question mark is a new number."
• The wrong operation: "If we take away, we get 1. But here the groups go together, so we add."
• The total in the wrong box: "5 tells us all of Pia's balloons now. Here we need the balloons Pia had."
• The addends in the other order: "3 and 2 make the same total! But the story starts with the balloons Pia had, so we start with 2."

That last one matters to us. Writing 3 + 2 instead of 2 + 3 is good math. Barnaby says so, then ties the number sentence back to the story.

The clue ladder: where, how, why

When an answer is wrong, Barnaby climbs a three-rung ladder. Each rung gives the child something new.

Rung one points attention. Feedback works best when it is about the task and the thinking, not about the child (Hattie & Timperley, 2007). Rung one sends the child back to the right place: to a number in the story ("Which number do we start with?"), or, for the answer box, to the bar, because the answer is never written in the story.

Rung two teaches a strategy. This is the rung most apps skip. A child who understands what the question mark means but can't yet work out 2 + 3 needs a way to get there, not another nudge. Each kind of story gets the counting strategy children naturally develop for it (Carpenter & Moser, 1984): count on from the bigger number to add, count back or count up to take away, count up to the total when a part is missing. The strategy never contains the answer, so the child still does the thinking. Tutoring researchers call this scaffolding: support that does just enough for the child to finish the step on their own (Wood, Bruner & Ross, 1976).

Rung three explains. If the child is still stuck, Barnaby shows the answer. We don't hide it, because an endless loop of "try again" teaches nothing and invites random tapping. But the answer always comes with its reason, since the moment right after a struggle is when a child is most ready to hear it (Shute, 2008).

The answer never appears before the third try. Children who learn that help soon gives the answer away tend to click straight through to it, a pattern well documented in tutoring software (Aleven et al., 2006). Three rungs, each with something new, keep the child working on the problem instead of on the help button.

Directions at every step

Clues are for when something goes wrong. Directions are for before it does. Before every box, Barnaby says what the box is for: "Which number do we start with?" "Which number is added?" "Now the question mark: all of Pia's balloons now. Work it out!"

Our lessons fade support on purpose (Fyfe et al., 2014). The child starts with a Singapore-style bar model that is already filled in (Kho, Yeo & Lim, 2009), then places the story numbers on a bar with labelled parts, then fills an empty bar alone. What fades is the help, never the directions. Even working alone, the child always knows the next step: "Put the story numbers on the bar, then check." Then: "Now write the number sentence."

One word for each thing

A six-year-old has very little working memory to spare (Sweller, Ayres & Kalyuga, 2011). If the walkthrough says "part," the bar says "start" and the hint says "the first number," the child has to translate before they can think. So each kind of story has one set of words, used everywhere: in Barnaby's speech, on the bar and in every clue. Adding stories use start, added and total. Taking-away stories use start, taken away and left. Comparing stories use bigger, smaller and more.

Gentle, every time

None of this works if a wrong answer feels like a punishment. SmartyPants has no red X and no buzzer: the box gives a small wiggle and Barnaby speaks (see our article on gentle validation). Barnaby never says "wrong." He says what to look at next.

Why we build it this way

Marking an answer right or wrong is easy to build. A clue ladder is not. Every kind of story needs its own strategies, every common mistake needs its own message, and every message has to be checked against every wrong answer a child could type. We read each one aloud, the way a six-year-old will hear it.

We think it is worth the work, because it is the difference between grading a child and teaching one. Our goal has always been to show a child where their thinking went wrong, not just that the answer was wrong. The clue ladder is how that goal becomes words on a screen.

What parents and teachers can borrow

You don't need an app to use the ladder. At the kitchen table or in class, when a child's answer is wrong:

1. Ask a "where" question first: "Which number do we start with?"
2. If they are still stuck, give a strategy, not the answer: "Start at 3 and count on 2."
3. If you do give the answer, give the reason with it: "2 and 3 together make 5."
4. Use the same words for the same idea every time.
5. Don't say "wrong." Say what to look at next.

SmartyPants is in active development. The clue ladder described here is the design of our Grade 1 word-problem lessons.

References

1. Aleven, V., McLaren, B., Roll, I., & Koedinger, K. (2006). Toward meta-cognitive tutoring: A model of help seeking with a Cognitive Tutor. International Journal of Artificial Intelligence in Education, 16(2), 101–128.
2. Ashlock, R. B. (2010). Error Patterns in Computation: Using Error Analysis for Improving Instruction (10th ed.). Pearson.
3. Carpenter, T. P., Fennema, E., Franke, M. L., Levi, L., & Empson, S. B. (1999). Children's Mathematics: Cognitively Guided Instruction. Heinemann.
4. Carpenter, T. P., & Moser, J. M. (1984). The acquisition of addition and subtraction concepts in grades one through three. Journal for Research in Mathematics Education, 15(3), 179–202.
5. Fyfe, E. R., McNeil, N. M., Son, J. Y., & Goldstone, R. L. (2014). Concreteness fading in mathematics and science instruction: A systematic review. Educational Psychology Review, 26(1), 9–25.
6. Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112.
7. Kho, T. H., Yeo, S. M., & Lim, J. (2009). The Singapore Model Method for Learning Mathematics. EPB Pan Pacific.
8. Radatz, H. (1979). Error analysis in mathematics education. Journal for Research in Mathematics Education, 10(3), 163–172.
9. Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153–189.
10. Sweller, J., Ayres, P., & Kalyuga, S. (2011). Cognitive Load Theory. Springer.
11. Wood, D., Bruner, J. S., & Ross, G. (1976). The role of tutoring in problem solving. Journal of Child Psychology and Psychiatry, 17(2), 89–100.