Solving Jigsaw 3uzzle with 6ymbol 0atrixes
Dai Cao†, Lifang Chen*, Yuan Liu¶
School of Digital Media
Jiangnan University, Wuxi, China
[email protected]†, [email protected]*, [email protected]¶
Abstract— This paper presents a new symbol-matrix-based
jigsaw-puzzle algorithm for image reconstruction. The proposed
algorithm first calculates the compatibility metric using the SSD
(Sum of Squared Distance Scoring) between adjacent pieces.
Then the algorithm constructs a matrix to express the location
relationship of pieces followed by constructing a symbol matrix to
record the number and rotations of pieces. Finally, we use a
greed algorithm to reconstruct the images. The proposed
algorithm does not require any preset conditions and can
reconstruct the images rapidly. The experimental results have
shown that the proposed algorithm can accurately reconstruct
the images with 28% speed-up in execution time. The results also
show that it’s very effective to reconstruct the puzzles with
missing pieces, which is a useful feature for applications such as
artifact reconstruction, biological information reconstruction and
incomplete crime-scene reconstruction.
Keywords—SSD(Sum of Squared Distance Scoring); Jigsaw
puzzle; Symbol matrixes; Missing pieces
I.
INTRODUCTION
Jigsaw puzzle solving is widely used in many fields, such
as image editing, biology, archeology and so on. Though
jigsaw puzzle problem is NP-complete, various solvers were
proposed, some of the algorithms are based on shape matching,
some based on color matching, or a combination of shape and
color matching. When pieces are square, the feature of shape is
missing, only color information can be used, this is the focus of
our work.
When solving jigsaw puzzles, a widely used method is
greedy algorithm, the compatibility between pieces should be
computed first. With the compatibility, a minimal spanning tree
can be built. The process of image reconstruction is also the
step of traversing the spanning tree. In addition to the greedy
algorithm, dynamic programming [4], graph model [5], the
genetic algorithm [6] is also introduced to solve the jigsaw
puzzle problem.
G. Paikin and A. Tal [3] proposed a solver of puzzles with
missing pieces for the first time in 2015, but the algorithm may
produce unsatisfactory results when the images are smooth or
noisy.
This paper proposes a method which outperforms the stateof-the-art, both in accuracy and in efficiency. The algorithm
uses one of the simplest compatibility metric, which we called
sum of squared distance scoring (SSD). With SSD, we can find
out the optimal pairs of each side of the pieces. Combining
these optimal pairs, we can obtain the optimal matrix. The next
step is filtering the optimal matrix, which will reduce errors in
This work is supported by National science and technology support plan
(2015BAH54F01), Natural science foundation of Jiangsu, China
(BK20130161, BK20151131).
978-1-5090-0806-3/16/$31.00 copyright 2016 IEEE
ICIS 2016, June 26-29, 2016, Okayama, Japan
the process of reconstruction. The first piece to reconstruct
should be chosen carefully, because error will cause subsequent
errors, which is also mentioned in [3]. Three symbol matrixes
which we called pieces’ number matrix, rotation number
matrix, pre-number matrix are introduced to reconstruct
images. Data filtering is a key step before reduction, which can
greatly reduce the occurrence of conflict problems in the
process of reconstruction, it improves the efficiency and
accuracy at the same time. In this paper, the algorithm doesn't
need any priori knowledge, it can quickly reconstruct image
accurately, even solve the puzzles with missing pieces.
II.
THE COMPATIBILITY METRICS
There are many kind of compatibility metrics, two of most
famous is SSD [2] [7] and MGC [1], the SSD is one of the
simplest and quickest metric, the Mahalanobis Gradient
Compatibility which we called MGC, is a very precise and
effective metric. There are a lot of measurement is the
improvement on the SSD and MGC, such as the metrics in [3]
and [8].
A. Sum of Squared Distance Scoring (SSD)
SSD is a metric to compute the differences between pieces,
which is mentioned in [7]. It is also called Dissimilarity-based
Compatibility in some papers such as [2], it is the simplest
metric. Its expression is shown in formula 1 or 2.
K
3
D( xi , x j , r ) = ¦¦ ( xi (k , K , d ) − x j (k ,1, d ))2
(1)
k =1 d =1
K
3
D( xi , x j , r) = ¦¦ xi (k, K , d ) − x j (k,1, d )
(2)
k =1 d =1
B. Mahalanobis Gradient Compatibility (MGC)
MGC, which is first proposed in [1], considers not only the
pixel information on the edge, also combined the edge gradient
information, which makes the metric more accurate. This
method improves the accuracy, but it greatly increases the
operation time at the same time.
When computing the compatibility DLR ( xi , x j ) of a jigsaw
piece x j on the right side of piece xi , we find the distribution
of the color gradients near the right edge of piece xi . We
define an array of gradients GiL with 3 columns (one each for
the red, green, and blue color channels), and P rows (where
P is the pixel dimension of the jigsaw piece). GiL describes
the intensity changes along the right size of the jigsaw piece xi
(since it will be on the left of the pair).
the comparison of optimal measure ρ with SSD and MGC, at
the same time with different piece sizes.
Entries in GiL are given as:
From the Fig. 1 we can see MGC’s optimal metric is much
bigger than the SSD’s, especially when the piece size less than
20, this also proves that the MGC is a more effective measures.
It also shows that MGC can express the difference between
pieces with a higher accuracy, but the SSD can still be
effective.
GiL (k, c) = xi (k, K, c) − xi (k, K −1, c)
(3)
The mean distribution of those gradients on the right side
of jigsaw piece xi is found as:
uiL (c) =
1 K
¦GiL (k, c)
K k =1
(4)
For each color channel, uiL is the mean difference
between the final two columns of xi . Similarly, the 3×3
covariance SiL estimated from GiL captures the relationship of
the gradients near the edge of the jigsaw piece between the
color channels.
At this point, we define the compatibility measure from
jigsaw piece xi to x j as follows:
K
DLR ( xi , x j ) = ¦ (GijLR (k ) − μiL )SiL−1 (GijLR (k ) − μiL )T
(5)
k =1
Fig. 1. Accuracy comparison with SSD and MGC
where:
GijLR ( k ) is the gradient from the right side of piece xi to
the left side of piece x j , at row position k. Explicitly,
GijLR (k, c) = x j (k,1, c) − xi (k, K, c)
(6)
The dissimilarity DLR ( xi , x j ) is not symmetric because
the junction between pieces xi and x j is evaluated based on
the distributions estimated from the xi side of the boundary.
Modified Equations, in the spirit of (3) to (6), are used to
define DRL ( x j , xi ) . Finally, the symmetric compatibility
measure MGC ( xi , x j , r ) for placing xi and x j as left-right
neighbors is:
MGC(xi , x j , r) = DLR (xi , xj ) + DRL (x j , xi )
(7)
C. Metric analysis and and choose
Using metric to reconstruct image, general method is to
choose the piece pairs with the highest similarity, and then
connect the two pieces. On the edge of the pieces use the
similarity metric values to determine, the highest similarity
degree measurement (in this paper is the minimum value of
formula 2), called the optimal metric. Every step of
reconstruction need to find the best value, the whole process is
establishing minimum spanning tree. Every piece pairs
corresponds to an optimal metric, the optimal metrics make up
the optimal matrix. The proportion of true location relations
included in the optimal matrix, here called the optimal measure
ρ , is an very important measurement of metric. Fig. 1 shows
Suppose there are N pieces, these pieces were likely to have
N*N*16 positions, the size of measure matrix is N*N*16,
because pieces are rotatable.
Each piece, has four sides, each side has its corresponding
optimal metric, the number of optimal metrics is 4*N in theory.
In fact, the optimal metrics is less than 4*N, because there are
some pieces on the edge of image, there are no corresponding
matching part of the direction of the pieces.
Target image for the complete rectangle, for example,
which has n c o l pieces on the lateral, and n r o w pieces on the
longitudinal, the number of effective the optimal metrics
is 2 * [( n ro w − 1) * n co l + ( n co l − 1) * n ro w ] .To connect all pieces, the
optimal metrics at least should be N – 1 for N pieces, similar to
the tree, no cross. The minimum percentage of optimal metrics
nrow * ncol −1
, a puzzle with 4 rows
2*[(nrow −1)* ncol + (ncol −1)* nrow ]
and 5 columns, p = 0.3065 . The number of pieces is always
is p =
great than 100, with the increasement of pieces’ number, p will
become smaller, in the general case, p ∈ (0.25, 0.3) .
From Fig. 1, we can see that SSD in piece size K ∈ [20, 30] ,
the optimal metric still maintained about 70%, far more than
the lowest proportion p . This suggests that the SSD can
effectively represent the differences between pieces, in order to
improve the computing speed, we using the SSD metric in our
paper.
III.
DATA FILTERING
A. The necessity of data filtering
Data filtering can decrease the amount of calculation, rule
out some wrong position and increase the accuracy of
reconstruction image at the same time. In the process of the
algorithm, the data filtering is very necessary.
In section Ċ .C, showed in Fig. 1, SSD metric between
pieces in most cases also failed to achieve high accuracy. The
relatively small differences between pieces, as shown in Fig. 2,
the relationship between pi , p j and pk , if directly judged by the
SSD metrics, it cannot accurately express position relationship
between them.
Fig. 2. Schematic diagrams of
pi , p j , pk
As seen from Fig. 2, the candidate position matrix
LOC = {( pi , p j ,1,1), ( pi , p j ,1, 4),..., ( pi , p j , 4,1), ( pi , p j , 4, 4)}
shows that the differences between piece pi and p j are really
small, we do not know which sides of the two pieces are really
neighbors. In this case, pk is the interference factor, data
filtering is very effective.
relationship between the two pieces. If the piece pi and
p j have the location left-right, the piece p j and pi have the
location right-left, the two pieces have the relationship of
"Best Buddies", otherwise not.
After filtering, the corresponding position relationship
matrix LOC will still contains some wrong pairs, but these
errors will be resolved in the next section of conflict detection
when the reconstruction.
IV.
IMAGE RECONSTRUCTION AND RESOLVING OF CONFLICT
After filtering, most of the position relationships in LOC are
accurate, they can be used to restore the image, but in the
process of image reconstruction, in order to avoid a small part
of the wrong positional relationship, conflict detection will
filter again to get a more accurate picture.
A. Choose the first piece
In the experiments we found that selecting a good starting
point is crucial in greedy algorithms, as early errors lead to
later failures. The first piece’s choosing in many algorithms
before is randomly selected, which leads to the problems that in
solving the same puzzle the accuracy is greatly different
repeatedly.
When performing image reconstruction algorithm, the
choice of the first piece need to meet the following conditions:
Piece whose 4-neighbors are all satisfy the “Best Buddies”
relationship, if there are multiple pieces to meet the conditions
and adjacent to each other, select the piece in the center of
these pieces as the first piece.
The aim of data filtering is to find the optimal metric,
which shows the real adjacent edges, and remove those
interferences.
The purpose of this selection is to minimize the error of the
initial step, thereby reducing the entire sub-picture reducing
errors and improve accuracy.
B. The process of data filtering
• Select from the compatibility metrics to get the optimal
matrix. In this paper, we call it LOC matrix, whose
length is 4*N.
B. Reconstruct image with symbol matrixes
Specific image reconstruction process is generally as
follows:
• Select from the LOC matrix, choose these have the
relationship of "Best Buddies" [2], delete those are not
satisfied.
• If the same pair of pieces have various locations in
LOC, leaving the best, delete the rest.
Description: LOC is a 4 columns matrix, the first column
represents the piece pi , the second column represents the
piece p j , the third column represents the edge number ri of
piece pi , and the fourth column represents the edge number rj
of piece p j , [ pi , p j , ri , rj ] represents
that piece pi ’s ri edge
is adjacent to piece p j ’s rj edge. In this paper, the default
number of pieces’ edge is up = 1, right = 2, down = 3, left = 4 .
The "Best Buddies" relationship is proposed by Pomeranz
[2], which represents some kind of mutual positional
1) Choose first piece, and initialize M num , M rot , M pre .
2) Select single positional relationship loc from the
optimal matrix LOC one by one, test loc by collision detection.
a) No conflict
If two pieces in loc have been contained in M num ,
nothing happen.
If one of the two pieces in loc is not contained in M num ,
put the piece into the M num ,update M rot and M pre .
b) Confilict
If two pieces in loc have been contained in M num , that is
to say one of the two pieces have two candidate positions.
Then choose the better one, delete the worse one and its pre
information using M pre , update M num and M rot .
If one of the two pieces in loc is not contained in M num ,
but on the target position, there is already another piece. That
is to say there are two candidate pieces on the same position.
Then choose the better piece, delete the worse one and its pre
information using M pre , update M num and M rot .
3) If there are still some pieces not contained in M num ,
global search the matrix to find the best position, update
M num 㸪 M rot 㸪 M pre
4) Reconstruct image with M num and M rot
Note: the matrix M num is pieces’ number matrix, storing
the pieces’ number; M rot is the rotation number matrix, storing
the clockwise rotation number of pieces; M pre is pre-number
matrix, storing the pieces’ pre-numbered pieces, for example,
[ pi , p j , ri , rj ] shows that pi has already in M num , p j not in
TABLE I.
Piece
of
which is saved in M pre .
When the matrixes are initialized, they are all 1x1 size,
respectively stored first piece’s number, number of rotations
with 0, pre-matrix with -1. The matrixes will be automatically
extended in the reconstruction process, if a position is empty,
respectively with 0, NAN, 0.
M num and M rot are very important in the conflict
detection, as shown in Table I. M pre will play a role in the
process of clearing the error information after the conflict.
With the information of M num , M rot and
loc = [ pi , p j , ri , rj ] , we can compute the relative positional
relationship between piece pi and p j , get the rotation number
of p j . These rules have been displayed in Table I.
RULES OF RECONSTRUCTION
p j relative positional relationship
pi
the matrix, if there is no conflict, piece p j ’s pre-piece is pi ,
pi
pj
Piece
rot j : rotation number of p j
(roti + ri )%4 == 0
Left
(rot j + rj )%4 == 2
rot j = (6 − rj )%4
(roti + ri )%4 == 1
Up
(rot j + rj )%4 == 3
rot j = (7 − rj )%4
(roti + ri )%4 == 2
Right
(rot j + rj )%4 == 0
rot j = (4 − rj )%4
(roti + ri )%4 == 3
down
(rot j + rj )%4 == 1
rot j = (5 − rj )%4
TABLE II.
EXAMPLE OF IMAGE RECONSTRUCTION AND CONFLICT DETECTION
M num
Partial matrix of LOC
ª18
«1
«
«5
«
¬3
ª13
« 15
«
«15
«
«15
« 16
«
« 17
« 17
«
«17
«
¬17
25
10
17
19
2
7
9
15
25
3
1
4
3
1
4
1
2
3
2º
1 »»
2»
»
2»
3»
»
4»
4 »»
4»
»
3¼
M rot
0º
15 »»
10 »
»
13 ¼
ª 18 0
« 1 15
«
« 5 10
«
¬3 13
0º
0 »»
0»
»
25¼
ª3
«0
«
«0
«
¬1
ª3
«0
«
«0
«
¬1
Reconstruction
state
Nº
2 »»
0»
»
3¼
Before the start of
blue arrow
N Nº
2 N »»
0 N»
»
3 2 ¼
After the blue
arrow
ª18
«1
«
«5
«
¬3
19 0 0 º
15 17 7 »»
10 9 0 »
»
13 25 0 ¼
ª3
«0
«
«0
«
¬1
1 N Nº
2 2 0 »»
0 1 N»
»
3 2 N¼
After the gray
arrow
ª18
«1
«
« 5
«
¬3
19 25 0 º
15 17 7 »»
10 9 0 »
»
13 25 0 ¼
ª3
«0
«
«0
«
¬1
1 0 Nº
2 2 0 »»
0 1 N»
»
3 2 N¼
Conflict
ª18
«1
«
« 5
«
¬ 3
19 25 0 º
15 17 7 »»
10 9 0 »
»
13 0 0 ¼
ª3
«0
«
«0
«
¬1
1 0 Nº
2 2 0 »»
0 1 N»
»
3 N N¼
After solving of
conflict
Table II shows an example of the reconstruction process
and conflict detection. According to the foregoing image
reconstruction, when the blue arrow points to
loc = [13, 25,3, 2] , piece 25 is not contained in M num , with the
rules in Table I, the number of rotations piece 13 is 3, the third
edge adjacent to piece 25, the piece 25 is on the right side of
the piece 13, and piece 25’s number of rotations is 2, then
there is no conflict, directly update M num , M rot and M pre . The
results updated as shown in table II, the blue arrow pointing to.
When the arrow moved to the gray one, there is no conflict
occurs. When arrow points to the red one, conflict appears,
which will be solved in the next section.
C. Resolve conflicts
1) Choose in the conflicts
Mentioned in Section 4.2, the possibility of a piece on a
position, with the possibility we can select pieces. Here's how
the possibility of representation, is now to be resolved.
Fig. 3. Schematic diagrams of forecast neighbor pieces.
In order to predict whether the two pieces are neighbors,
as well as the possibility of adjacent, we introduced
Differential Equations, as shown in Fig. 3, x(1) and x(2) are
two columns on the edge of one piece, respectively
corresponding to xi (k , K − 1, d ) and xi (k , K , d ) ,where
k ∈ [1, K ], d ∈ {1, 2,3} .
Δx = x(2) − x(1)
(8)
Δx = x(3) − x(2)
(9)
x(3) = 2* x(2) − x(1)
(10)
With formula 8,9,10, we can predict the edges adjacent to
piece xi , as shown in Fig. 3, which is expressed in Equation
11:
x(3) = 2* xi (k , K , d ) − xi (k , K − 1, d )
(11)
Differences between predicted edge x(3) and actual
edge x j (k ,1, d ) smaller, the greater the possibility of two
adjacent pieces, as shown in expression such as Equation 12:
likelihood ( xi , x j , r ) =
large number, in the experiment, we found
Bignum = 10000 can ensure likelihood ( xi , x j , r ) > 0 .
that
2) Multiple candidate pieces on the same position
Description: a position X have piece pi , in the next
process of reconstruction, according with a positional
relationship, piece p j is also on the position X, how to choose
pieces on the X position.
Suppose position X has 4 neighbor pieces, the left
one pL , top one pU , right one pR ,bottom one pD , candidate
pieces are pi and p j .
Using p = likelihood ( pi , pU , r ) , we can get the possibility
of piece pi and pU adjacent to each other. As the same, we can
get the possibility of other pairs. With these comprehensive
information, one method commonly used is that average the
possibility of pi and p j , now we can determine which piece to
choose.
3) The same piece on multiple candidate positions
Description: piece pi is on position X, in the next process
of reconstruction, according with a positional relationship,
piece pi should be on the position Y, how to choose position
for piece pi .
Suppose position X and Y both have 4 neighbor pieces,
respectively pLX , pUX , pRX , pDX of position X, pLY , pUY ,
pRY , pDY of position Y, same definition in section 4.3.2.
Candidate positions are X and Y.
With p X = likelihood ( pi , pUX , r ) , we can get the
possibility of piece pi and pUX adjacent to each other. As the
same, we can get the possibility of other pairs. With these
comprehensive information and average possibility on position
X and Y, determine which position to use. As shown in Table
I, the case of piece 25’s position is an example. When the red
arrow points to the red one, loc = [17, 25,3,3] , the piece 17
and 25 have been in M num , with the rules in Table I, piece 25
should be above the piece 17, the number of rotations should
be 0, while the original pieces 25 at the bottom of piece 9 , the
number of rotations is 2, here conflict happens. Piece 25 has
two candidate positions, then compute the likelihood on the
two positions. Finally, we keep the position above the piece
17. And then, delete the relevant information of the other
position, where M pre will work, in order to remove all the preinformation of piece 25. The results are shown in Table II, as
the red arrows point to.
(12)
Note: If the target position Y has been occupied by
piece pk , it should run the step in section Č.C.(2) after that.
Equation 12 represents the possibility of pieces adjacent
to, its value must be greater than 0, so Bignum here must be a
All of images used in the experiments are collected from
the Internet, piece size is always 28x28 pixels. All of the
experiments are running on the same computer. Comparative
K
3
¦¦ 2* x (k , K , d ) − x (k , K − 1, d ) − x (k ,1, d )
i
1−
i
j
k =1 d =1
Bignum
V.
EXPERIMENTS
experiments are conducted to test the speed and accuracy, the
results show that our algorithm is 28% faster compared to
traditional greedy algorithm. In terms of accuracy, our
algorithm, using the simplest measure SSD, still can
accurately restore the image, especially puzzles with missing
pieces, the algorithm’s performance remained stable.
reconstruction with missing pieces. As seen in Fig. 4, whether
the pieces concentrated region missing, or discrete scattered
missing, our algorithm can correctly restore the image.
TABLE III.
Pieces total
number
432
540
805
1) Speed and accuracy
Since [1]’s algorithm does not solve the problem of
puzzles with missing pieces, the following comparison
experiments were carried out in the case of no missing pieces.
Table III shows the comparison of two algorithms’
running time, it can be seen from the table, our algorithm is
about 28.75% faster than [1].
Table IV shows the accuracy comparison of [1] and ours.
Ours accuracy is slightly better than [1].
2) With missing pieces
Superior to [1]’s algorithm, our algorithm can solve
puzzles with missing pieces. Fig. 4 shows some types of
(a) random
TABLE IV.
Pieces
total
number
432
540
805
(b) circular
COMPARISON OF AVERAGE RUN TIMES
[1]
ours
63.86
100.85
226.75
45.87
71.67
159.30
Speed up
percent
28.17%
28.97%
29.75%
ACCURACY COMPARISON OF RECONSTRUCTION
average
best
worst
[1]
ours
[1]
ours
[1]
ours
0.9309
0.9223
0.8968
0.9378
0.9251
0.9026
0.9632
0.9586
0.9448
0.9669
0.9538
0.9421
0.8179
0.7951
0.7288
0.8361
0.8026
0.7435
(c)hole
(d)mixed
Fig. 4. Reconstruction with missing pieces.
VI.
CONCLUSION
In this paper, using the simplest SSD metric, we calculate
for each piece’s 4 edges to get optimal positional relationship,
which we called optimal matrix in the paper. After date
filtering, we get optimal matrix to reconstruct image. During
reconstruction, symbol matrixes are introduced, they are also
used to solve conflict detection. Though our algorithm only
has a small part of the upgrade in speed and accuracy, it can be
used to resolve puzzles with missing pieces. Our algorithm
does not need any pre-set conditions, it can reconstruct puzzles
with a high accuracy, which greatly increases the value in
practical applications. For example, in the process of
restoration of cultural relics, it is more likely having missing
pieces, they also do not know the overall situation, our
algorithm will play a greater role in this case.
ACKNOWLEDGMENT
We thank Andrew C. Gallagher for his source code to
reproduce the results of his paper [1].
REFERENCES
[1]
[2]
[3]
[4]
[5]
[6]
[7]
[8]
A. Gallagher, Jigsaw puzzles with pieces of unknown orientation. IEEE
International Conference on Computer Vision and Pattern Recognition,
382 – 389,2012.
D. Pomeranz, M. Shemesh, and O. Ben-Shahar. A fully automated
greedy square jigsaw puzzle solver. In Proc.Of CVPR, 9 – 16, 2011.
Paikin, G., Tal, A.Solving multiple square jigsaw puzzles with missing
pieces. Computer Vision and Pattern Recognition (CVPR), 2015 IEEE
Conference, Pages 4832 - 4839, 2015.
N. Alajlan. Solving square jigsaw puzzles using dynamic programming
and the Hungarian procedure. American Journal of Applied Sciences,
6(11), 1941-1947, 2009.
X. Yang, N. Adluru, and L. Latecki. Particle filter with state
permutations for solving image jigsaw puzzles. In Proc. of CVPR, 28732880,2011.
D. Sholomon, O. E. David, and N. S. Netanyahu. A genetic algorithmbased solver for very large jigsaw puzzles. In IEEE Conference on
Computer Vision and Pattern Recognition, pages 1767-1774, 2013.
D. Mondal, Y. Wang, S. Durocher. Robust Solvers for Square Jigsaw
Puzzles. Computer and Robot Vision (CRV), 2013 International
Conference, Pages: 249 – 256.
S. Jin, S. Lee, N. A. Azis. Jigsaw Puzzle Image Retrieval via Pairwise
Compatibility Measurement. Big Data and Smart Computing, 2014
International Conference, Pages: 123–127.