ILP for Mathematical Discovery
Simon Colton & Stephen Muggleton
Computational Bioinformatics Laboratory
Imperial College
The Automation of Reasoning
Automated
Theorem
Proving

Maths
Automated
Reasoning
Machine
Learning
Bioinformatics
Aims for the talk
– Discuss a new ILP algorithm (ATF)
• and its implementation in the HR system
– Promote maths as a domain for ILP research
From Prediction to Description
Predictive
tasks
Supervised
learning
Know
what you’re
looking for
Don’t know
what you’re
looking for
Descriptive
tasks
Unsupervised
learning
Don’t know
you’re even
looking
A Partial Characterisation
of Learning Tasks
Concept learning
 Outlier/anomaly detection
 Clustering
 Concept formation
 Conjecture making


Theory formation
The HR Program in Overview

Embodies a novel ILP algorithm
– We call this “Automated Theory Formation” (ATF)
– Designed for descriptive tasks (in maths)
• But has had applications to concept learning tasks
– Incrementally builds a theory
• Containing association and classification rules

HR has numerous tools for the user
– To extract information from the theory generated
• Which is relevant to the task at hand
ATF Overview
Invent new concepts
 Derive classification rule from concept
 Induce hypotheses relating the concepts
 Prove/disprove the relationships

– Deductively
• Using state of the art ATP/model generators

Extract association rules
– From the hypotheses
Input to HR

Five inputs to HR
– Objects of interest (graphs, groups, etc.)
– A labelling of the objects
• If the task at hand is predictive…
– Background predicates (Prolog style)
– Axioms relating predicates (ATP style)
– Termination conditions
}
See
Paper
• HR works as an any time algorithm

User can supply
– numerous different combinations of these
Representation of Theory Contents

Three types of frames
– All have a clausal definition slot

Example frame
 Concept frame
–
–
–
–

Slot 1: range-restricted program clause
Slot 2: success set
Slot 3: classification rule afforded by definition
Other slots: measures of value
Hypothesis frame
– Slot 1: clauses (association rules)
– Slot 2: proof/counterexample
– Other slots: details of the concepts related
Cut Down Algorithm Description

Build new concept definition from old
• Using one of 12 generic production rules [PR] (see paper)

Find the success set, S, of new concept
 If S is empty, derive non-existence hypothesis, H
• Extract association rules from H, try to prove/disprove

If S is a repeat, derive equivalence hypothesis, H
• Extract association rules from H, try to prove/disprove

If S is new
– Add new concept to theory
– Derive classification rule
– Derive implication & near-equivalence hypotheses
• Extract association rules, try to prove/disprove

Measure concepts in theory
Concept Space Searched

Space determined by PRs, not language bias
 Clausal definition is:
– range-restricted, fully typed program clauses

Definition: n-connectedness
– Every variable appears in a body predicate with head
variable n, or with a n-connected variable

Example:
– c(X,Y) :- p(X), q(Y), p(Z), r(X,Y), s(Y,Z) is 1-connected
– c(X,Y) :- r(Y), s(X,Z) is not 1-connected

HR’s definitions are all 1-connected
Deriving Classification Rules

Given definition D
– Arity = n, head predicate = p, success set = S
– Classifying function over constants, o, is:

Classification, C, afforded by D:
– Put two objects in the same class if f(o1) = f(o2)

Theorem:
– If a definition D is not 1-connected, then a literal can be
removed without changing the classifiction afforded by D
– So, HR’s search space is non-redundant with respect to C
Illustrative Example

concept17(X,Y) :integer(X), integer(Y), divisor(X,Y), ¬ divisor(Y,2).

S17 = {(1,1), (2,1), (3,1), (3,3), (4,1), (5,1), (5,5), (6,1), (6,3)}

Classifying function:
– f17(1)={(1)} f17(2)={(1)} f17(3)={(1),(3)} f17(4)={(1)}
– f17(5)={(1),(5)} f17(6)={(1),(3)}

Classification afforded by concept 17:
– [ [1,2,4] [3,6] [5] ]
Mathematics Applications

Two applications given here
– Both from external research groups
– Data sets available online

See paper for details
– Of two more applications
Finding
Discriminants

Finding discriminants of residue classes
 Work with Sorge and Meier
 Overall goal: classify algebraic structures
– Bottleneck: showing non-isomorphism

Learning task:
– Given two multiplication tables
– Find a property true of only one
• Which doesn’t refer to individual elements

Data set: 817 pairs of tables (size 5, 6, 10)
Results

HR given 500 steps per task (~22 secs)
– Worked with four production rules

Found discriminants
– For 791 out of 817 pairs (~97%)
– Average of 20 discriminants per pair
– 517 distinct discriminants in total

Example above:
– Idempotent element (a*a=a)
• Appearing once on diagonal
– Only one of two discriminants found for pair
Reformulation of CSPs

Work with Miguel and Walsh
 Constraint satisfaction solving
– Very powerful general purpose technique
– Specifying a problem is still highly skilled

Learning task:
– Given solutions to small problems
• Find concepts to specialise the problem specification
• Find implied constraints to increase efficiency

Data set: QG-quasigroups (5 types)
– Multiplication tables up to size 6
Results

HR ran for an hour for each problem class
– Produced on average 150 association rules
– And 10 specialisation concepts

In each case, a better reformulation was
derived (with human interpretation)
– Up to 10 times speed up in some cases

Nice example: QG3: (a*b)*(b*a)=a
– These are Anti-abelian, i.e., a*b=b*a  a=b
– Symmetry relation: a*b=b  b*a=a
Some Other Applications

Concept learning tasks:
– Extrapolation of integer sequences: ICML’00
– Mutagenesis regression unfriendly

Anomaly detection task:
– Analysis of Bach Chorale melodies (current MSc.)

Conjecture making tasks:
– Generating TPTP library theorems: CADE’02 (& paper)
– Finding links in the Gene Ontology (current MSc.)
– Making Graffiti-style conjectures (current MSc.)

Theory formation task:
– Invention of integer sequences: AAAI’00, JIS’01 (& paper)
Conclusions

Presented the ATF algorithm
– Involves induction & deduction
– Presented for first time in ILP terminology
– Characterised the concept search space

Presented two learning tasks
– In mathematics
– More in paper (and in previous work)

Shown that HR can make discoveries
Future Work

Apply HR to bioinformatics
– Needs more efficient implementation

Look into the conglomeration of
– Creative reasoning techniques

Relate HR to other descriptive programs
– CLAUDIEN and WARMR

Can these programs
– Do better than HR in maths applications?
A Drosophilia for
Descriptive Induction?
“Something from (nearly) nothing”
 Can give HR only 1 concept

– Multiplication in number theory

Invents the concept of refactorable nums
– Number of divisors is itself a divisor
– 1, 2, 8, 9, 12, …

A nice hypothesis it produces is:
– Odd refactorable numbers are square