Using the Proof-Checking Word Processor Lurch to Teach
Proof-Writing
Nathan C. Carter1 and Kenneth G. Monks2
1
1
Mathematical Sciences Department, Bentley University, Waltham, MA
2
Mathematics Department, University of Scranton, Scranton, PA
Background
Lurch is a free word processor that can check the mathematical reasoning in a document.1 Although it would
be easy to make a word processor that could check simple arithmetic, or even algebra, Lurch aims to do much
more than that. It can check the steps of a mathematical proof, even a proof not written in a formal style.
The current version works best for introduction-to-proof courses, covering topics such as logic, introductory
set theory, algebra, and number theory. As we add features it becomes more able to handle a wider variety
of mathematics courses.
Lurch has been tested at the authors’ two institutions. Bentley University (where the first author teaches)
is a selective business university of about 4,000 undergraduates and 1,500 graduate students in the greater
Boston area, with a liberal arts focus. The University of Scranton (where the second author teaches) is a
Jesuit university in Scranton, PA, of similar size to Bentley and also with a liberal arts focus. While Bentley
has very few theoretical mathematics courses (focusing instead on the mathematics of finance and actuarial
sciences), Scranton has majors in mathematics and various types of mathematics education (secondary,
middle-school) and thus has several upper-division, proof-based mathematics courses.
2
Description and implementation of the method
2.1
Long-term goals
The Lurch project has a two-part mission statement; we state and explain each part here.
Mission Statement - Part 1 Lurch should be as indistinguishable from the ordinary activities of mathematics as possible, except for the additional services it provides. That is, it should only add to your math
experience, not change your math experience.
This is our user interface guideline. Users should be able to write math in whatever style or notation
they prefer, with exposition inserted where they feel it’s helpful. They can already do so on a chalkboard,
on paper, or in LATEX, and especially in a pedagogical context it is important for the software to match the
textbook and lecture notes, because differences would be an obstacle for the student. So mathematics in
Lurch must be as similar as possible to mathematics in the other media just mentioned. Except, of course,
that pencil and paper won’t tell the user whether his or her proof is correct, but Lurch will, in addition to the
other benefits that any word processor provides, such as cut and paste.
1
Lurch was supported from 2008-2012 in part by the National Science Foundation’s Division of Undergraduate Education, in the
Course, Curriculum, and Laboratory Improvement program (grant #0736644). Views expressed in this document are not necessarily
those of the National Science Foundation.
1
Using the Proof-Checking Word Processor Lurch to Teach Proof-Writing
2
Mission Statement - Part 2 Lurch should provide the software infrastructure the mathematical community
needs for validating rigorous mathematics. That is, it should validate mathematical content created by
you—a “spell-checker” for mathematical rigor.
Of course, there are several impressive pieces of software in existence (e.g., Mizar [13] or Coq [14]) for
validating rigorous mathematics, and Lurch is not claiming that it will supercede these offerings. Our goals
are very different from those of proof checkers like Mizar and Coq, and thus Lurch is not trying to compete
with those packages. We list here the important differences between our goals and theirs.
1. More flexibility
Most existing proof checkers require the user to learn a specific language and rules, while Lurch lets
the user (or his or her instructor) define the rules from scratch. Future versions will also allow the user
to define the mathematical language (i.e., notation, syntax) as well.
2. No help with proofs
Proof checkers automate some proof steps, and an ongoing field of research furthers the frontiers of
this capability. Lurch has no such capabilities, nor should it, because if it did proofs for the student,
that would defeat the purpose of teaching the student how to do them. In Lurch, the student types his
or her work while Lurch grades, encourages, and coaches.
3. Familiar user interface
Proof checkers often have interfaces suitable for advanced users, sometimes requiring familiarity with
the command line or shell scripts. Lurch is for the typical student, and its user interface is therefore
a familiar one: It is a word processor that gives feedback visually in the document. We want typical
mathematics students to encounter no learning curve, except what is presented by the mathematics
itself, not Lurch.
We’re targeting students in their first proof-based courses, giving frequent, immediate, clear feedback
on the steps in their work. That’s the part of the mathematical community we’re excited about reaching.
Existing research on educational technology suggests that our goals make sense and are achievable.
Investigations into the effects of automated assessment systems (AiM [10] and MyMathLab [17]) show the
value of computers in giving high-frequency, individualized, and immediate feedback ([15, 19, 18, 20]). The
value of such feedback is well documented in educational research (reviewed in [9]). This research matches
our common sense that more feedback, delivered in immediate response to each of the student’s actions, is
better for learning.
This feedback is also noteworthy in another respect. When we think of students’ experiencing mathematical objects on a computer, we tend to think of interactive graphs with sliders, rotating three-dimensional
surfaces, or other flashy graphics. In Lurch, students experience a far more abstract mathematical object – a
proof. The positive and negative feedback they get while working within a logical system is their experience
of that system, including its rules, axioms, and theorems. In Section 3, quotes from student surveys support
this assertion.
Finally, because Lurch documents are word-processed, they are more legible than typical written assignments. Students who used Lurch were enthusiastic about the fact that it is a free math word processor that
works on any platform and has a significantly lower learing curve than LATEX for typing mathematics. Neat,
organized homework has also been shown to be easier to grade [12]. We have found this to be even more
true of work done in Lurch, because the validity of each individual step in many cases has been verified.
Using the Proof-Checking Word Processor Lurch to Teach Proof-Writing
2.2
3
Current status
The previous section listed our long-term goals for Lurch. But how far we have come (as of this writing, in
July 2013) towards achieving those goals? This section answers that question, and explains briefly our plans
for the future.
A word processor with semantics
First, we have built a word processor that supports mathematical expressions with actual mathematical semantics. Consider the screenshot of our application in Figure 1. The red, yellow, and green icons interspersed
throughout the user’s document are giving the user feedback on the correctness of each step of his or her
work. (This feedback can be disabled to obtain a clean view or printout.) Green thumbs-up icons follow a
correct step of work that is correctly justified or a valid declaration, red thumbs-down icons follow a step
of work that contains an error or is supported by the wrong reason, and yellow lights follow mathematical
expressions that are used as premises to justify later steps of work, but are not themselves justified (i.e., the
hypotheses in a proof). In order to provide this kind of feedback, Lurch needs to know the meaning of the
mathematical expressions in the document, and from the fact that the feedback in Figure 1 is correct, you
can see that it does. We discuss below how we accomplish this.
Figure 1: A Lurch screenshot explained in Section 2.2. Although this screenshot was captured on a Mac,
Lurch is free and open-source for Windows and Unix as well. By default, Lurch uses colored traffic light
icons rather than the thumbs shown here; the thumbs are an option useful for colorblind users and in blackand-white printing.
Notice the important difference here between what Lurch does and what most other mathematical word
processors do. For example, if you type a mathematics paper in Microsoft Word and type your mathematical
expressions in Equation Editor, it will typeset them attractively, but it has no idea what the expressions mean.
Using the Proof-Checking Word Processor Lurch to Teach Proof-Writing
4
Even a simple expression such as a + b can be quite deceptive; does it mean addition of real numbers, or
of elements in a group, or perhaps a vector space? Word cannot know, because it treats each expression
independently, ignoring context.
This same limitation holds for typesetting systems that follow the WYSIWYM paradigm (What You
See Is What You Mean). In LyX or LATEX, for example, you provide information about the structure of your
document, so that the system can typeset of your document structure in any of several visual ways. However,
the meaning in that case is the meaning of your document structure, not the meaning of your mathematical
expressions. Just as Word does not know exactly what a + b means, neither does LATEX; it only knows how
to make it look good on the page.
Lurch does know the meanings of your mathematical expressions, in the sense that the rules defined
elsewhere in the same document (or documents on which your document depends) provide the meanings
for each symbol in your expressions, and Lurch is aware of those rules. It allows you to use those rules to
draw new conclusions from existing mathematical expressions, and it verifies that the steps you take to do
so are in line with the rules you have defined. Those rules can be anything from low-level rules of logic
(permitting you to define new logical systems from scratch in Lurch) up to high-level mathematical axioms
(such as definitions of algebraic structures).
The bubble interface
The second major component of what we have built so far is the user interface through which users tell
Lurch the meaning of the mathematics in their documents. This interface must be as simple as possible so
that the user is not distracted from the mathematics. To this end, we employ a user interface paradigm we
call bubbles.
In order for a section of text to be treated as a mathematical expression, the user must mark it as such
with a single click of a toolbar button (or the corresponding keyboard shortcut) shown in Figure 2. The user
selects a portion of text that he or she wishes Lurch to treat as meaningful mathematics, and then clicks
the “Meaningful expression” button on the toolbar. Lurch then wraps the text in a bubble and reports the
understood meaning in a tag atop the bubble, as in Figure 3.
Common formatting tools
Marks expressions meaningful
Figure 2: One of the Lurch toolbars, shown here on Mac OS X. Some of the buttons are the typical formatting
buttons one expect to see in any word processor, such as bold, italic, and underline. The most important new
one to note here is the red button for marking text as meaningful.
Lurch draws a bubble around a mathematical expression if and only if the user’s cursor is inside it; see
the screenshots in Figure 4. This paradigm has the advantage that when your cursor is not in a mathematical
expression, no extra visual clutter asserts itself. But when your cursor is in one, the software makes this fact
clear in the interface, and clicking a bubble’s tag exposes a menu of actions relevant to the mathematical
expression in the bubble. Both LyX and Word have somewhat similar interfaces for editing mathematics.
The user will create bubbles often, and thus doing so must be intuitive and take near-zero time. As Figure
2 suggests, it is just as easy to mark text as meaningful as it is to make it bold or italic; a single click or
keystroke does it.
Using the Proof-Checking Word Processor Lurch to Teach Proof-Writing
5
After selecting the meaningful section of text,
click the toolbar button
to make it meaningful.
Figure 3: How a sentence within a Lurch document changes when a user applies the button shown in Figure
2 to a section of text. In the top of the figure, the user has selected a section of text. After telling Lurch to
mark it as meaningful, the red bubble shown on the bottom of the figure appears, indicating that Lurch has
read the expression, and knows that it is a universal quantification.
Expression with the cursor inside it, and
thus the bubble shown:
The same expression with the cursor
outside it, and thus the bubble hidden:
Figure 4: In Lurch, bubbles around meaningful expressions appear if and only if the user’s cursor is inside
the expression. In this example, the bubble tag “+ expression” indicates that the expression is a sum.
This interface enables Lurch to identify which portions of the user’s document he or she considers
meaningful mathematics. Therefore, in line with our goals from Section 2.1, the user is stylistically free
to structure their document their own way. Because meaning is determined by what sections of text have
been bubbled, the user can alternate whenever they like between expository text in any natural language and
meaningful mathematics, and arrange their work as informal sentences, formal proof structures, or varying
levels in between.
This interface also brings a pedagogical benefit. Student users, just learning what a proof is, are forced
to indicate what parts of their own proofs are meaningful and which are not. Experience from testing shows
that this is perhaps the clearest lesson students learn from Lurch, compared to our experience in courses
without Lurch. Students come away from a Lurch-integrated course knowing very solidly how a proof is
structured, because they had to point out this structure before it would check their work (not just which
portions of text are meaningful mathematics, but also which cite reasons and premises). In fact, it was not
until after this learning had fully sunk in that students perceived the bubbling process as a burden.
Flexible, powerful logic engine
The reasoning engine underlying Lurch is powerful enough to handle a wide variety of mathematical systems, from simple ones such as propositional logic through complex ones such as introductory set theory,
number theory, and abstract algebra.
Using the Proof-Checking Word Processor Lurch to Teach Proof-Writing
6
One of the most unique aspects of Lurch’s reasoning engine is that it does not come with any built-in
rules of logic; the user is entirely free to specify, from the ground up, the logical system in which they wish to
work. Of course, to prevent users from needing to do this tedious task themselves, Lurch ships with classical
propositional and predicate logic documents that are part of the default set of rules initially available to the
user, but the user is free to edit or replace that logical system by writing a Lurch document that defines their
own.
Given this high level of customizability, it may surprise the reader to know that Lurch comes with a
theorem that proves the correctness of its logic engine. If no particular logic system is required, what can
the theorem be about? The answer is that the theorem is not a soundness theorem about a logical system,
but rather a proof that the algorithms Lurch uses to interpret the given set of rules correctly realize the intent
of those rules, as expressed by the Lurch specification, which in turn is intended to be in line with the user’s
expectations. We state that theorem as follows.
Theorem. Given an abstract Lurch document as defined in [7] (containing no inferred contexts), validating
(as defined in [7]) each meaningful expression in the order they appear in the document will result in a
correctly validated document.
The definition given in [7] of an abstract Lurch document and what it means to validate a meaningful
expression is and is quite detailed. The proof of this theorem is a case analysis that shows that, for every
type of expression that needs validation, the results of that validation will never need to change as a result of
validating an expression later in the document. This proof would be trivial if validating an expression never
had any effect on any aspect of the document preceding it, but that is not the case. We refer the reader to [7]
for details. (See, specifically, the creation of inferred contexts.)
One important fact to note about this theorem is that it is also notation-independent. Although the current
version of Lurch has a particular mathematical notation built into it, which is not customizable by the end
user, we intend future versions to allow users to completely customize the mathematical notation just as they
can currently completely customize the rules of logic (and of various branches of mathematics). Therefore
it is important for no part of Lurch’s implementation to assume or depend upon a specific notation, but to
work in terms of abstract expressions instead.
For this reason, Lurch is built upon an international standard of data structures for mathematical semantics, OpenMath [2]. In that standard, all expressions are stored as trees, with each subtree having an unambiguous mathematical meaning, as specified in published online “content dictionaries.” Although Lurch
does not conform to every aspect of the standard, it uses its data structures and XML file format [1].
What this means from the perspective of an advanced user is that a mathematical expression in Lurch is,
from the software’s point of view, a tree. Lurch provides a way for advanced users to request that individual mathematical expressions be displayed in a notation similar to the S-expressions used in programming
languages like LISP; see Figures 5 and 6.
This form is fully editable and understandable by Lurch, and users can even write whole documents in
it, though that is not desirable from the user’s point of view. The are are two reasons Lurch supports this
LISP-like form of expressions.
1. It is sometimes desirable to be able to ensure that Lurch is interpreting an expression the way the
user expects, and the user can ask Lurch to expand any expression into this form to see its meaning
unambiguously.
2. This form provides a visual representation of the internal structure of the expression, and therefore
can be used when analyzing Lurch’s internal algorithms, such as in a proof of the theorem above.
Using the Proof-Checking Word Processor Lurch to Teach Proof-Writing
7
Figure 5: Three views of the expression ∀x, x + 5 ≥ 5 in Lurch. On top, the way the expression is almost
always written, in Lurch and elsewhere, using ordinary mathematical notation. In the middle, the expanded
view of the expression; this LISP-like notation is rarely used, but gives an unambiguous indication of expression structure. On the bottom, the user has asked Lurch to decorate the expanded view with indications
of how Lurch understands each level of the expression’s hierarchy.
Figure 6: This figure is analogous to Figure 5, but for the expression A ⊆ {1, 2, k}, instead of ∀x, x+5 ≥ x.
Using the Proof-Checking Word Processor Lurch to Teach Proof-Writing
8
Planned improvements
Despite all that has been built so far, there are many plans for enhancing Lurch in the future. The most visible
planned improvement is the introduction of typeset mathematics (fractions, square roots, large operators, and
so on) for both input and output of mathematical expressions. Another, as mentioned earlier, is that users
should be able to define new notation just as easily as they can now define new axioms and rules of logic.
Finally, there are many bells and whistles we wish to add that we will not cover here, and some major
improvements to the application’s efficiency. Our project always welcomes additional developers who wish
to help with these tasks, including undergraduate and graduate students as well as faculty.
2.3
Classroom integration
Lurch has been tested in two semester-long courses, one taught by each of the authors. The first author tested
an early version of the software in a formal logic course at Bentley University in fall 2008, and the second
author tested recent version in an introduction to proofs course at the University of Scranton in spring 2013.
In this section, we will concentrate more on the latter of these two tests, because it is closer to the main
target audience of the software, and because it used the latest version, which is much closer to our goal than
a version from five years earlier.
Since Lurch design is informed by the method of teaching proofs discussed below, it is particularly well
suited for this approach. Indeed the libraries of built-in axioms and rules that currently ship with Lurch reflect this and are effectively just the definitions and notation from the second author’s lecture notes from the
introduction to proof course. But as indicated in our Mission Statement, Lurch is also designed to accommodate almost any choice of rules and definitions, and almost any style of proof that an instructor may want
to use in his or her course. In particular, an instructor can completely ignore the built-in libraries that ship
with Lurch and simply type his or her lecture notes into Lurch and provide them to the students for their use
in the course. Thus the approach to teaching proofs we now present is just one possible method for teaching
proofs that can benefit from Lurch but many other approaches are possible.
What follows refers to the introduction to proof course taught at the University of Scranton by the second
author during the Spring 2013 course using the versions of Lurch that were available at that time (versions
0.76 through 0.7992).
What is a proof?
Any introduction to proof course faces this obvious question. It must be answered both initially in a succinct
way to foreshadow what is to come, and subsequently in a detailed manner as the course unfolds. The
succinct and simple initial answer given in the this class is as follows.
Definition. A proof is an explanation of why a particular mathematical claim must be correct.
Perhaps more important from the students’ perspective at the beginning of the course is to explain what
goals we are trying to accomplish with our proofs, in order to motivate why we want to write proofs in the
first place. In this course we focused on two main goals of a mathematical proof, corresponding to each of
the two bold phrases in the definition above.
Objectivity: We would like our proofs to provide us with objective verification that a given claim
must be correct. For example, a proof that can be verified by a purely mechanical algorithm
would meet this criterion. (This corresponds to the idea that a proof shows that a claim must
be correct.)
Exposition: We would like our proofs to communicate clearly to a human reader the insights,
intuition, and understanding why a given claim must be correct. (A proof must also be an
explanation.)
Using the Proof-Checking Word Processor Lurch to Teach Proof-Writing
9
Naturally these two goals are not always easy to achieve simultaneously. Formal proofs that include
every detail are easy to verify mechanically, analogous to the way computations in arithmetic, algebra, and
calculus can be verified by calculator or computer. Such proofs usually have a very rigid and technical
syntax that makes them verifiable (in principle) by readers with little mathematical sophistication. But they
are almost always too lengthy, detailed, and technical to communicate the main ideas and intuitions involved
in the proof to a human reader.
Expository prose proofs typically found in books and journals are usually far more effective at imparting
an understanding to the reader. Such proofs are usually tailored to a particular target audience, since the
target audience affects things like the choice of natural language used (English vs. Italian, say) and the level
of detail that is provided, skipping details that a reader in the target audience would already know or believe,
and thus better emphasizing the main ideas. But for a student in an introduction to proof course, omitting any
details whatsoever can leave them feeling that the validity of a proof is more an art of getting the instructor
to agree with them than something they can know for themselves is objectively valid.
As at many schools, the introduction to proof course at the University of Scranton is often referred to
as a “bridge” course. In most cases this term refers to a course that provides a topical bridge for students to
cross in order to make the transition from introductory (computation-oriented) mathematics courses such as
calculus to advanced (proof-oriented) courses such as abstract algebra or topology. But in the context of this
particular class the term “bridge” takes on a alternate meaning, namely that it provides a bridge from the
certainty of formal, rigorous proofs, to the more traditional, expository proofs found in books and articles.
Once that bridge is sound, we can meet the goals of objectivity and exposition simultaneously. To make
a objectively verifyable formal proof more expository we can use shortcuts and add explanations to write
the same proof in traditional expository style. More importantly, a student who is writing a more traditional
expository style mathematical proof can try to write the same proof as a formal or semiformal proof (and
actually did do this voluntarily in some cases in the course) in order to know that their proof is objectively
correct.
The students’ journey: from calculations to toy proofs
The students in the Spring 2013 class at Scranton consisted of mostly sophomore math, math education, and
science majors who had completed the first two courses in the calculus sequence but did not have any proofbased courses beyond high school geometry. Their background in mathematics at the start of the course
consisted mainly of computations.
Thus we could begin to build the bridge from what students know coming into the course (calculations)
to proofs by illustrating that a calculation satisfies our criteria for a proof if we can justify each step of
the work with a reason that explains why the step is correct, as in Figure 7. We might call such a justified
calculation a formal calculation as a harbinger of the formal proofs to come.
Ordinary Calculation
Formal Calculation
a·b+a=a·b+a·1
a · b + a = a · b + a · 1 by multiplicative identity
= a · (b + 1)
= a · (b + 1)
by distributivity
Figure 7: On the left, a typical calculation a calculus or algebra student might do as part of a larger problem.
On the right, the same computation as a formal calculation, the difference being a justification for each step.
While this illustrates to the students how something they already know relates to what is to come,
there isn’t much that can be done with formal calculations until they have been told what constitutes a valid
justification or reason that can be used to justify a step of their work. (For example, Figure 7 assumes they’ve
Using the Proof-Checking Word Processor Lurch to Teach Proof-Writing
10
at least heard of things like the multiplicative identity and distributive laws.) Because rules of inference and
syntax can be quite complicated, the course introduces students to some of the basic features of a formal
mathematical reasoning system through some fun, non-mathematical systems before it introduces an actual
mathematical system.
The Toy Proofs software system is developed and maintained as part of the Lurch project. It implements
these fun, formal systems and is available online [8]. It does what we call implicit validation, in which
the software only provides the user with actions that perform valid steps of a proof, and thus validates the
students’ work implicitly by preventing them from making mistakes. (It does not tell them how to create a
correct proof, but it prevents them from making mistakes while trying to do so.)
The Toy Proofs systems introduce several foundational concepts, but in game-like settings: a language
of “statements,” a collection of “rules of inference” by which new statements can be justified, and a “proof,”
consisting of a sequence of statements in the system, each of which is either given or derived from earlier
ones with the rules of inference. The students are introduced to progressively more complicated rule sytems
as they progress through the three systems. The first system is colorful and very game-like, but the final one
is more syntactic and mathematical.
All three Toy Proofs systems emphasize one main concept, that the purely mechanical nature of the
systems provides the objectivity we want in proofs. This point is made clear by the fact that the “statements”
in the Toy Proofs games (e.g., sequences of colors in one game) have no obvious meaning, yet the computer
can verify the “proofs” anyway.
From toy proofs to formal proof
After a few assignments on the Toy Proofs systems the students were then presented with a formal system
for propositional logic. Simultaneously, they graduated from the implicit validation in Toy Proofs to the style
used in Lurch, which we call explicit validation. Since Lurch is a word processor, a student can type anything
he or she likes in a document, including incorrect statements, unjustified statements, and incorrectly justified
statements. Lurch merely informs the user which statements are correctly justified and which are not; it does
not prevent the user from making errors.
The syntax for mathematical statements in Lurch is currently built into the program, but one of the high
priorities for future development is to allow the user to customize the notation and syntax of statements and
expressions. Lurch also comes with a set of rules, which any instructor or student can choose to use, or they
can replace them with rules of their design. Each rule in Lurch can be labeled, and students cite rules by
those labels when specifying reasons for their steps of work.
The style of formal proof used in the beginning of the Scranton bridge course is a variation of Gentzen’s
natural deduction. A formal proof in this system consists of a list of lines containing (a) a line number used
as the label for the line, (b) a statement (at a given level of indentation), (c) a reason justifying that statement
when necessary, and (d) the line numbers of any premises needed for the particular rule specified as the
reason. An example of such a proof is shown in Figure 8.
From formal proof to semiformal proof
After mastering propositional logic, the students in the course are then taught predicate logic with equality,
using the same formal proof style. At this point, having practiced and mastered the concepts of statement,
reason, and premise extensively, the overwhelming level of detail and specificity required in a formal proof
starts to become tedious, and the students are ready to advance to the next level of sophistication.
This general theme permeates the course. Once the students have been exposed to and master a given
concept to the point where it becomes tedious or repetitive, the tedious or repetitive parts are eliminated in
favor of a more advanced proof style. Such parts are eliminated by two mechanisms (both in Lurch and on
the chalkboard) that we call shortcuts and smart rules.
Using the Proof-Checking Word Processor Lurch to Teach Proof-Writing
11
Figure 8: An example of a formal proof in propositional logic validated by Lurch. This uses the particular
style of formal proof used in the second author’s introduction to proof course.
Shortcuts are well defined mechanisms by which portions of a proof may be omitted without losing the
objectivity and certainty of the correctness of the proof. A large and detailed list of these is presented to the
students throughout the semester as they become appropriate.
For example, it is nearly always the case that the statement at line n in a proof is used as a premise on
line n + 1, so one shortcut is to agree that premises don’t have to be cited if they immediately precede the
line being justified. Another shortcut is that most lines in a proof do not need to be numbered or labeled,
excepting those that need to be referred to frequently in the rest of the proof.
A smart rule is a high-level rule of inference that can be used to justify a statement using a more sophisticated algorithm than simple pattern-matching. For example, the only statements that can be proved
in formal propositional logic are tautologies. Thus, once students have mastered propositional logic and
move on to other mathematical topics like set theory or number theory, they can then use a smart rule
simply called “logic” to justify any step of work that could be justified by a proof in propositional logic.
In Lurch, this smart rule simply checks, for premises P1 , P2 , . . . , Pk and conclusion C, if the statement
“P1 and P2 and . . . and Pk ⇒ Q” is a tautology.
Another example is a smart rule for basic algebra that can verify statements like (x + 1)3 = x3 + 3x2 +
3x + 1 using an internal computer algebra system. This rule would be an appropriate smart rule to use, for
example, after students have mastered the properties of basic algebra such as associativity, commutativity,
distributive laws, etc. It is also useful when teaching formal calculations like computing derivatives in calculus, where the focus may be on justifying the steps of differentiation, but not the steps of algebra. The
first of these two example smart rules exists in the current version of Lurch, while the second is queued for
future development.
As shortcuts and smart rules are presented in the course, the style of proof slowly transitions from the
fully formal proofs that provide objectivity and rigor to expository proofs common in mathematics, that
communicate ideas more clearly. This transition passes through an intermediate style of proof that we call
semiformal proof, illustrated in Figure 9.
In a semiformal proof, line numbers and the referencing of most premises are optional, and the spacing,
punctuation and wording of the individual lines in the proof are unrestricted and thus free to be close to what
it might be in an expository proof. The main restrictions that define a semiformal proof in the course are that
every statement must be on a separate line and subproofs must be indented. It should be easy to convert a
semiformal proof into a formal one, while on the other hand, if you combine the individual lines into a few
paragraphs, a semiformal proof becomes very like a traditional proof. Thus, semiformal proofs serve as an
intermediate style of proof that bridges the gap between formal style proof and traditional expository style
proofs.
Using the Proof-Checking Word Processor Lurch to Teach Proof-Writing
12
Figure 9: An example of a semiformal proof in elementary set theory validated by Lurch. A semiformal
proof still uses only one statement per line, but permits omitting line numbers, not specifying immediately
preceding premises, and more natural spacing and punctuation. These features make semiformal proofs a
natural waypoint on the road from formal proofs to expository proofs.
The fact that Lurch can validate all three styles of proof (and others) makes it possible for students to
answer what is arguably the most commonly asked question about their proofs: “How do I know if I’m
right?”.
Using previously-proven theorems
One other way to reduce tediousness and repetitiveness in mathematical proofs is to use previously-proven
theorems as new rules of inference to justify lines in a proof. After logic, the Spring 2013 bridge course
students learned elementary set theory, functions, relations, elementary algebra, and number theory, each
topic presented in a similar way.
First, the instructor presented formal definitions for a topic. Then he introduced theorems that depend
only on one definition (and anything it, in turn, depends on), which we term basics. a shorthand for “immediate consequences of the definition.” For example, the formal definition of subset might be as follows.
Definition. A ⊆ B ⇔ ∀x, x ∈ A ⇒ x ∈ B
So a student could be asked to prove the following “basic” as an immediate consequence of the definition.
Theorem. If A ⊆ B and x ∈ A then x ∈ B.
Thus the typical workflow in this part of the course for students consisted of first learning new definitions, investigating them enough to prove theorems about the basic consequences of each, and then combining more than one definition to prove deeper theorems. This progression is illustrated in Figure 10.
The rules, definitions, and theorems in Lurch are organized in this same sequence, so that students are
asked to prove theorems in one assignment, and then in the next assignment, those theorems are available
to use as reasons in other proofs. The instructor has complete control over which rules, definitions, and
theorems are available for the students to use on a given assignment.
Using the Proof-Checking Word Processor Lurch to Teach Proof-Writing
13
Other Theorems
Proven from multiple definitions,
the “real” business of mathematics
Mathematics
“Basic” Theorems
Proven from a single definition,
often as an exercise in understanding it
Definitions
The bridge between
logic and mathematics
Predicate Logic
Logic
Propositional Logic
Figure 10: After learning the foundation of logic, students are presented with definitions, prove immediate
consequences of the definitions, and then more important theorems that combine more than one definition.
From semiformal proof to expository proof
The final transition to traditional expository style proofs begins with a review of some of the common
typesetting and writing conventions in mathematical writing. A portion of an expository proof (with an
exmaple error) verified by Lurch was shown in Figure 1.
But some of the shortcuts that students need to learn in this final stage of the Lurch does are not support.
In addition, Lurch is designed to be a word processor that can validate proofs, not the best mathematical
typesetting system available. Thus when the students transition to traditional, fully informal proofs, the
instructor switched the software for writing their assignments from Lurch to LATEX. This transition was
facilitated by providing them with a custom LATEX style file which they used with the website writeLATEX
[11].
This produced high-quality typesetting for the students, but LATEX cannot validate their proofs for them.
Thus we were unexpectedly pleased to notice that some students, on their own initiative, wrote semiformal
proofs in Lurch prior to writing an informal version in LATEX for submission. This was excellent confirmation
that students valued Lurch for exactly the purpose we intended.
The journey’s end: combinatorial proofs
The last two weeks of the course covered one last style of proof that is notoriously difficult to formalize,
combinatorial proofs.
Using the Proof-Checking Word Processor Lurch to Teach Proof-Writing
14
Here, a combinatorial proof is one that is based on a counting argument, such as counting the same
collection in two different ways to prove an identity. For example, one might prove that for any natural
number n,
n
n
n
n
+
+
+ ··· +
= 2n
0
1
2
n
using the algebraic formula for binomial coefficients and induction on n, but that would not be a combinatorial proof. A combinatorial proof would argue that the number of subsets of a set with n elements can be
counted first by adding up the number of subsets of each size from zero through n to get the left hand side
of the equation, and then count the same collection again by noting that every subset is determined uniquely
by deciding whether each element is in it or not, so that the number of distinct subsets is 2n .
In the course we restricted the study of combinatorial proofs to proving combinatorial identities such
as the example above (i.e., theorems of the form A = B where A and B are expressions that have a
combinatorial interpretation). The student assignments were done in LATEX, but could not be verified by
Lurch. This was a painful reminder of how valuable it is to the students to have a resource like Lurch available
to check their work. Without it, the only way for the students to get feedback on their combinatorial proofs
was to ask the instructor as they would in any course without Lurch. The students were much less confident
that their submitted proofs were correct, and there was a decrease in the number of questions that they asked
the instructor prior to handing in the assignments as there was no software to tell them that a step of their
work was incorrect (which they were then motivated to resolve before proceding).
2.4
Customizing Lurch for other approaches and courses
What we have discussed above was just one particular method and style of teaching students about mathematical proof. Lurch is designed to be customized beyond just the rules and definitions that ship with the
program to match the lecture notes or textbook used in a particular course.
Figure 11: In this sample Lurch document, an instructor specifies a new definition and illustrates in a proof
snippet how that definition can be used to justify a step of work. Lurch validates that the symbol increasing
is not already defined as something else, certifies that it understands the instructor’s new rule, and validates
the statement g(1) < g(2) (which has been justified by citing the name of the new definition as a reason and
providing the two premises required by the rule). The instructor specifies the rule by bubbling it as shown
in Figure 12.
All of the rules and definitions that are available to students when doing their assignments in Lurch are
written in Lurch documents directly. Users who want to customize the mathematics in Lurch can inspect such
Using the Proof-Checking Word Processor Lurch to Teach Proof-Writing
15
documents, copy them, and alter them to suit their own style, or write new ones from scratch. A document
containing rule definitions can be included in another document as a dependency, making the rules in the
former available in the latter.
Figure 12: To define a new rule or definition the instructor only has mark the rule, its label, (optional)
premises and conclusions. Here we see the semantic structure of the rule defined in Figure 11.
The libraries that ship with Lurch define rules that are based on the natural deduction style of proof used
in the second author’s introduction to proof course. However there is no need to use these rules at all. An
instructor can type his or her definitions, rules, and theorems directly into Lurch to make new dependency
documents that the students can then use when working on their assignments.
For example, Figure 11 shows a snippet of lecture notes that an instructor might type into a Lurch
document to define a new rule. He first declares a new term (increasing) (bubbled as a constant declaration)
which Lurch verifies is not already defined. He then specifies his proposed definition in the form and with
the level of detail and specificity that he desired (for example, in another context he might want to specify
that s, t are in the domain of f as another premise). This is bubbled by the instructor as indicated in Figure
12.
Finally, when a student uses the rule to justify a statement, he does so by supplying the two required
premises, and justifies the expression g(s) < g(t) by citing as a reason the name of the rule (“increasing” in
this case). This is illustrated in Figure 13. Lurch checks that for the appropriate values of f , s, and t in the
rule definition, the required premises are available to justify the desired conclusion.
Figure 13: To use the new rule defined in Figure 11 the user cites the reason with the same name to justify
the meaningful expression that matches the conclusion. In this example the user has also cited another rule
called “arithmetic” to justify the claim that 1 < 2.
The current version of Lurch does not fully support custom notation or render complicated math formulas
involving fractions, large operators, and so forth. But the authors intend that future versions of Lurch will
allow both of these things.
Using the Proof-Checking Word Processor Lurch to Teach Proof-Writing
3
16
Outcomes of using the method
3.1
Early testing, Fall 2008
The first testing of Lurch was in an introduction to logic course at Bentley University in Fall 2008. That
course was not discussed above, because Lurch has changed significantly for the better since 2008. For this
reason (and those mentioned in Section 2.3), this section is brief, in favor of giving more focus to discussion
of results from Spring 2013, the more recent testing. But some of the restuls of that semester of testing are
still noteworthy and relevant.
One of the goals of creating a digital environment embodying the rules of a formal system is that students
will, through that environment, bump up against those rules in much more concrete ways than they would in
paper-and-pencil homework. (That this has value is well-documented, as covered in Section 2.1.) But does it
actually happen when students use Lurch? The following quotes from the student survey from that semester
suggest that it does.
“It helped me learn how to do proofs through trial and error”
“It told me if I was right or wrong”
“...I liked using Lurch because I was able to see what was needed for rules to work.”
Furthermore, the following Likert-scale questions (1 = strongly disagree, 3 = neutral, 5 = strongly agree)
and their average response values suggest the same.
4.6
4.2
I used Lurch for experimentation; I tinkered to learn the results of various actions.
The constant feedback Lurch provides about my work is valuable.
Finally, even the very primitive version tested in 2008 was, according to the students, beneficial.
This early version of Lurch is not yet beneficial to students.
Learning to use Lurch took a lot of time that I could have spent learning logic instead.
2
2.2
Anecdotally, the instructor observed that students who did not choose to use Lurch for their homework
(which was optional) fell into two categories. Either they were doing very well in the class and did not need
any extra help, or they didn’t like being told that they were wrong, and thus avoided the software. This latter
group was small, and quickly began using the software when their first few homeworks received bad grades.
One student reported that receiving back the first homework grade finally explained why she couldn’t get
Lurch to accept her proofs—because they were incorrect!
3.2
3.2.1
Recent testing, Spring 2013
Student survey results
At the conclusion of the Spring 2013 Introduction to Mathematical Proof course at the University of Scranton, the students in the course were surveyed to see how they perceived the use of Lurch in the course,
and in particular, to see if they felt that the goals we have set for the software were being met. The survey
was anonymous and not given to the instructor until after the course grades were already submitted and the
course was over.
The following Likert-scale questions (1 = strongly disagree, 3 = neutral, 5 = strongly agree) and
their average response values suggest that even our current version, which is missing some major planned
features, is meeting those goals from the students’ point of view.
I would recommend that future students (in this course) use Lurch.
Lurch saying my work was right mattered to me.
I spent a lot more time on homework because Lurch showed me my mistakes
The constant feedback Lurch provides about my work is valuable.
Lurch contributed to how much I learned in this course.
4.9
4.7
4.4
4.3
4.1
Using the Proof-Checking Word Processor Lurch to Teach Proof-Writing
17
Even the current incomplete version was, according to the students, not an extra burden to master.
This early version of Lurch is not yet beneficial to students.
Doing homework in Lurch is more difficult than doing it on paper.
Learning to use Lurch took a lot of time that I could have spent learning proofs instead.
1.6
2.0
2.5
One natural worry, given the evidence of student appreciation above, is that perhaps repeated guesswork
alone could create a proof in Lurch. That is, with enough feedback, students lose the need to be creative
and clever, and can just persistently experiment until a proof is done, and have learned nothing. Thus our
surveys also asked for agreement or disagreement with the sentence “It is possible to do a proof in Lurch
by experimental clicking and typing, without thinking.” In the early version of Lurch (which used implicit
validation, and was not at all a word processor) the average response was neutral, 3.2. This latest version of
Lurch, however, has significantly improved this score to a 1.6, between disagree and strongly disagree.
Student comments on the same survey indicated that at some of our goals for the software are being met.
“I liked that we could check our work as we are doing it – it makes the learning
more immediate by providing constant feedback even outside of class.”
“I was intimidated by proofs before taking this class, but Lurch helped boost my
confidence. Using Lurch for formal proofs helped me learn all the pieces you need
for a correct proof and how to go about solving a proof. Also, I am very bad with
computers so I was very nervous about using this program, but it was pretty user
friendly.”
“I like how it broke down proofs into a step by step format. Once I learned how
to use Lurch it made doing proofs easier and clear. It made it easier to identify
statements, reasons, and premises.”
“Lurch told me if my work was correct. If it wasn’t I worked until it was correct.
Doing this also helped me learn definitions and rules more easily. (Plus, I knew I
was right.)”
“It allowed me to guarantee when I got something right, which was helpful early
on when I was unsure about what a proof was.”
“I can’t imagine learning proofs without it!”
3.2.2
In-class exams using Lurch
The first exam in the course was given in class entirely on Lurch. The students brought their personal laptops
to class and were given a Lurch file containing the exam to fill out. They were required to type their proofs
in Lurch, but using Lurch to validate their proofs was optional. The reason for making it optional is that,
in order for Lurch to validate a proof, the student must place bubbles around all mathematical expressions,
as well as citations of statements and reasons, as overviewed in Section 2.2. We call this work “bubbling a
proof,” and we did not want to force this upon them in case they considered it unworthwhile to do so. But
all students chose to bubble their proofs and have Lurch check their work.
One question on the exam consisted of a 17-line formal proof of a DeMorgan’s Law for quantifiers
which the students were asked to provide justification for by providing reasons for each of the 17 statements.
This question was designed to test whether the students understood the concept of a proof without testing
Using the Proof-Checking Word Processor Lurch to Teach Proof-Writing
18
whether or not they can overcome the obvious additional difficulties of crafting one from scratch. If the
correct reasons and premises were cited for each line, the students could ask Lurch to validate their proof.
Incredibly, every student in the class answered this question correctly.
This important understanding about the nature of a proof, even for students who struggle with constructing their own proofs from scratch, seems to have been completely impressed on the students through the use
of Lurch in the initial assignments leading up to the exam. While no data is available for comparison with
previous classes where Lurch was not used, the instructor’s observation is that this was a vast improvement
in student understanding compared with two previous classes who took the course in 2010 and 2011 without
the use of Lurch.
The final exam in the course was given in two parts. The first part was a take-home portion where
the students were asked to write several advanced proofs in expository style using LATEX. While portions
of those proofs could be validated by Lurch, some steps and certain aspects of the notation were beyond
Lurch’s current ability. While it was completely optional, several students did try to work out semiformal
drafts of their proofs before writing them up in expository form in LATEX. We hope that as Lurch becomes
more sophisticated and easier to use, this voluntary use of Lurch to check the validity of a proof will increase.
The second portion of the final exam was again taken in class and consisted of proofs that had to be
done in Lurch on their personal laptops, but whether or not the students wanted to have Lurch validate the
proofs was again left completely up to them. Every student but one chose to bubble their proofs and have
Lurch validate their proofs before handing them in. The one student who chose not to was an A student who
felt confident enough in his proof ability that the extra effort of bubbling wasn’t warrented. As it turned out,
that assumption was incorrect, as that student had the lowest score in the class on that portion of the final
exam. Thus the extra effort and time required for a student to bubble and validate their proofs was not a
hindrance to them scoring well, even on a timed, in-class portion of the final exam. In fact, the evidence
seems to indicate that even the students who did not ultimately receive an A grade in the course were able
to outscore an A student (that student earned an A in the course despite the poor performance on the Lurch
portion of the final exam) who chose not to use Lurch for validation.
3.2.3 Lurch homework assigments
Most of the proofs assigned for homework in the first half of the course were required to be written up
in Lurch. In the second half of the course they switched to LATEX as they made the transition from formal
and semiformal proofs (at which Lurch currently excels) to more notationally sophisticated and abbreviated
expository proofs (with which it is still limited). As with the rest of the course, bubbling their proofs on the
Lurch assigments in order to recieve validation feedback was completely optional, but most students did.
One interesting aspect of doing their assignments in Lurch came about as a result of combining it with
cloud-based folder sharing. The instructor had a shared folder on his computer with each of the students,
and the students would write up their assignments in their shared folder using Lurch. This allowed the
instructor to “look over their shoulder” when they needed help on their assignments, and to easily answer
their questions (typically “Why does Lurch say my proof is wrong at line n?”) even before the assignment
was handed in. To facilitate this, the authors added a “red comment pen” formatting style to Lurch, so that
the instructor could easily annotate a student’s homework document, even while it was still in progress. This
opened up a much more frequent and agile line of communication between the instructor and student than
the typical handing in and grading of an assignment on paper.
3.3
Successes and challenges
Even this early version of Lurch, when combined with the method of instruction explained in Section 2.3
has been quite successful. Student surveys indicate that the students found the software to be both valuable
Using the Proof-Checking Word Processor Lurch to Teach Proof-Writing
19
and unobtrusive in the course. Observations by the instructor on voluntary student use of Lurch to check
their work provides further evidence in support of the project goals.
But there are still hurdles left for our project to overcome in the future. In particular, there are several features that would significantly improve the user experience, which the development team (and any
who wish to join them) need to work on in the near future. These include improving the efficiency and
responsiveness of the application, adding support for typeset mathematical expressions, making typeset expressions editable, and allowing instructors to customize the notation for mathematical expressions to what
their course or textbook prefers. Other enhancements we anticipate are listed in Section 4.1, below.
4
Extending the method
The main challenge with using a new piece of software is getting both the instructor and students over the
initial learning curve in the shortest amount of time. This is so that learning Lurch itself requires the smallest
percentage of the course, leaving nearly 100% to spend on the mathematical course content. We attack this
problem head-on in two important ways, discussed in the following sections.
4.1
Minimal user interface
Each new command a piece of software provides and each new control it presents on screen contributes to the
software’s learning curve. With respect to minimizing the initial learning curve, the fewer such commands
and controls there are (and the simpler each one is) the better. Section 2.2 makese the case that our current
user interface is very good by this metric, but there remain planned advances not yet implemented. We
discuss two here.
First, although the bubble interface described in Section 2.2 seems minimal, it could actually get simpler!
As shown in Figure 3 on page 5, the user must first highlight the section of text they wish to mark as
meaningful before clicking the toolbar button or pressing the keyboard shortcut. To make this even simpler,
the software could be enhanced so that if no text were selected when that same command were invoked,
that Lurch would automatically apply the command to the largest section of text surrounding the cursor
that forms a valid mathematical expression. If such a command were given a shortcut such as Alt-click, the
user could bubble all meaningful expressions in document very quickly, by holding down the Alt key and
clicking on meaningful expressions.
Second, citing a premise used in a step of reasoning is currently a two-part process. Any premises that
immediately precede the conclusion Lurch does not require the user to cite; they are automatically detected
and used. But premises further away must be cited explicitly. Currently, this can be done in any of several
ways, none of which are very difficult, but none of which are trivially easy, either.
We plan to make this simpler in two ways. First, when the user’s cursor is inside a mathematical expression, so that a bubble appears around it, visible arrows should connect that bubble to every other meaningful
expression it uses as a premise, so that users can immediately see the premise-conclusion relationships they
have set up. Second, to add or remove an arrow should require but a single mouse action, such as Controlclick. If your cursor is in a conclusion, Control-click another expression to add or remove it as a premise,
and the arrow to that expression appears or disappears, respectively.
These are but two examples of how we will continue to pursue making the user interface even simpler,
thereby reducing the overhead necessary to using Lurch in a mathematics course.
4.2
Textbooks to ease integration
With the goal of making Lurch easy to integrate into courses at more institutions, one final effort is worth
mentioning. During the summer of 2013, each of the two authors of this article is writing a free, online
Using the Proof-Checking Word Processor Lurch to Teach Proof-Writing
20
textbook that integrates Lurch.
The first author will be taking the open-source textbook forallx written by P.D. Magnus [16] and creating
a version that integrates Lurch throughout. Some topics in the text will also be reordered according to how
the first author has had success using the text in an introductory logic course in the past. That text will be
available by Fall 2013 online [3].
The second author will be writing up a more formal version of the course notes he uses in the bridge
course he created at the University of Scranton. Lurch files will also continue to be updated and revised to
take into account what was learned in teaching the course during the Spring 2013 semester. This should
make the course more accessible and available to instructors who might want to use similar methods in
whole or in part.
5
For more information
The Lurch project welcomes all interested developers, testers, and users. Mathematics faculty with no computer programming capabilities who wish to create new mathematical topics and exercises in the software
are particularly encouraged to download a version, try it out, and share your joys or get answers to your
questions on the mailing list. Faculty and students with development expertise are encouraged to contact us
if you’d like to give time to this young-but-growing open-source software project.
Interested readers are encouraged to visit the Lurch website [4], where you can download the free software, look for the two textbooks mentioned in Section 4.2, as well as introductory videos, future development plans, the project’s email list, and more.
References
[1] Olivier Arsmac and Stephane Dalmas. OpenMath INRIA C/C++ Libraries. Downloaded May 18, 2008.
URL http://www.openmath.org/software/index.html.
[2] S. Buswell, O. Caprotti, D.P. Carlisle, M.C. Dewar, M. Gaëtano, and M. Kohlhase, editors. The OpenMath Standard, version 2.0. The OpenMath Society, June 2004. URL http://www.openmath.
org/standard/.
[3] Nathan Carter and P.D. Magnus. forallx in Lurch. A free and open-source textbook introducing formal logic and integrating Lurch, 2013. Available online. URL http://web.bentley.edu/empl/
c/ncarter/faxil.
[4] Nathan Carter and Kenneth G. Monks. Lurch: A word processor that can check your math. A free
and open-source software project hosted on SourceForge.net, access on April 23, 2013. URL http:
//lurch.sf.net.
[5] Nathan Carter and Kenneth G. Monks. Lurch: A word processor that can grade students’ proofs. In
Joint Proceedings of the 25th OpenMath Workshop, the 8th Workshop on Mathematical User Interfaces
(MathUI), and the Work In Progress Section of the Conference on Intelligent Computer Mathematics,
James Davenport, Johan Jeuring, Christoph Lange, and Paul Libbrecht, editors. CEUR Workshop Proceedings #TBD, Aachen, Bremen, Germany, 2013. ISSN 1613-0073, URL http://ceur-ws.org/
Vol-\#TBD/.
[6] Nathan Carter and Kenneth G. Monks. Lurch: A word processor built on OpenMath that can check
mathematical reasoning. In Joint Proceedings of the 25th OpenMath Workshop, the 8th Workshop on
Using the Proof-Checking Word Processor Lurch to Teach Proof-Writing
21
Mathematical User Interfaces (MathUI), and the Work In Progress Section of the Conference on Intelligent Computer Mathematics, James Davenport, Johan Jeuring, Christoph Lange, and Paul Libbrecht,
editors. CEUR Workshop Proceedings #TBD, Aachen, Bremen, Germany, 2013. ISSN 1613-0073, URL
http://ceur-ws.org/Vol-\#TBD/.
[7] Nathan Carter and Kenneth G. Monks. Introduction to Lurch: Advanced Users Guide URL http:
//lurch.sourceforge.net/AUG-v1.html.
[8] Nathan Carter and Kenneth G. Monks. Toy Proofs. Online web activities for introducing concepts of
formal systems. URL http://lurchmath.org/toyproofs.
[9] A.W. Chickering and Z.F. Gamson. Seven principles for good practice in undergraduate education. American Association for Higher Education Bulletin, pages 3–7, 1987. URL http://
aahebulletin.com/public/archive/sevenprinciples1987.asp.
[10] Gustav Delius and Neil Strickland. AiM—Assessment in Mathematics. A free and open-source software
project hosted on SourceForge.net, accessed on April 22, 2013. URL http://sourceforge.net/
projects/aimmath/.
[11] John Lees-Miller and John Hammersley. WriteLATEX. Online collaborative LaTeX editor with integrated
rapid preview. URL http://www.writelatex.com.
[12] Lisette de Pillis and Michael E. Orrison. Teaching time savers: Is homework grading on your nerves?
FOCUS, 27(1):14–15, January 2007.
[13] Adam Grabowski, Artur Korniłowicz, and Adam Naumowicz. Mizar in a Nutshell. Journal of Formalized Reasoning, 2010. URL http://jfr.unibo.it/article/view/1980/1356.
[14] Yves Bertot and Pierre Castéran. Coq’Art: The Calculus of Inductive Constructions Springer. 2004,
XXV, 472 p. ISBN 978-3-540-20854-9. URL http://coq.inria.fr/
[15] Saliha Klai, Theodore Kolokolnikov, and Norbert Van den Bergh. Using Maple and the Web to grade
mathematics tests. In IWALT 2000 Conference Proceedings, 2000. URL http://allserv.rug.
ac.be/˜nvdbergh/aimrug/docs/iwalt.ps.
[16] P.D. Magnus. forallx. An Introduction to Formal Logic. Published on the World Wide Web under
the Creative Commons Attribution-ShareAlike 3.0 license. URL http://www.fecundity.com/
logic.
[17] Pearson Education. MyMathLab. A series of mathematics courses on the World Wide Web. URL
http://www.mymathlab.com/.
[18] Chris J. Sangwin. Assessing mathematics automatically using computer algebra and the internet.
Teaching Mathematics and its Applications, 23(1):1–14, 2004. URL http://web.mat.bham.ac.
uk/C.J.Sangwin/Publications/tma03.pdf.
[19] Chris J. Sangwin. Assessing higher mathematical skills using computer algebra marking through
AIM. In Proceedings of the Engineering Mathematics and Applications Conference (EMAC03, Sydney, Australia), pages 229–234, 2003. URL http://web.mat.bham.ac.uk/C.J.Sangwin/
Publications/Emac03.pdf.
[20] Michelle D. Speckler. Making the grade: A report on the success of MyMathLab in higher education
math instruction. Pearson Education, Boston, MA, Fall 2005. URL http://www.mymathlab.com.