State of the Art in Surface Reconstruction from Point
Clouds
Matthew Berger, Andrea Tagliasacchi, Lee Seversky, Pierre Alliez, Joshua
Levine, Andrei Sharf, Claudio Silva
To cite this version:
Matthew Berger, Andrea Tagliasacchi, Lee Seversky, Pierre Alliez, Joshua Levine, et al.. State
of the Art in Surface Reconstruction from Point Clouds. Eurographics 2014 - State of the Art
Reports, Apr 2014, Strasbourg, France. 1 (1), pp.161-185, 2014, EUROGRAPHICS star report.
<http://diglib.eg.org/EG/DL/conf/EG2014/stars/161-185.pdf>. <10.2312/egst.20141040>.
<hal-01017700>
HAL Id: hal-01017700
https://hal.inria.fr/hal-01017700
Submitted on 3 Jul 2014
HAL is a multi-disciplinary open access
archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.
L’archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d’enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
Author version 2014/
STAR – State of The Art Report
State of the Art in Surface Reconstruction from Point Clouds
Matthew Berger1
Pierre Alliez3
3 Inria
Andrea Tagliasacchi2
Joshua A. Levine4
Lee M. Seversky1
Andrei Sharf5
Claudio T. Silva6
1 Air Force Research Laboratory, Information Directorate
2 École Polytechnique Fédérale de Lausanne (EPFL)
Sophia-Antipolis - Mediterranée 4 Clemson University 5 Ben-Gurion University 6 New York University, School of Engineering
Abstract
The area of surface reconstruction has seen substantial progress in the past two decades. The traditional problem
addressed by surface reconstruction is to recover the digital representation of a physical shape that has been
scanned, where the scanned data contains a wide variety of defects. While much of the earlier work has been
focused on reconstructing a piece-wise smooth representation of the original shape, recent work has taken on more
specialized priors to address significantly challenging data imperfections, where the reconstruction can take on
different representations – not necessarily the explicit geometry. This state-of-the-art report surveys the field of
surface reconstruction, providing a categorization with respect to priors, data imperfections, and reconstruction
output. By considering a holistic view of surface reconstruction, this report provides a detailed characterization of
the field, highlights similarities between diverse reconstruction techniques, and provides directions for future work
in surface reconstruction.
1. Introduction
The modeling, recognition, and analysis of the world around
us is a longstanding goal in the field of Computer Graphics.
Central to these objectives is a means of obtaining a digital
representation of objects in the real world. Surface reconstruction is concerned with recovering such information, where
the basic problem is to capture a 3D point cloud that samples the real world, and reconstruct as much information as
possible concerning the scanned objects.
Surface reconstruction came to importance primarily as
a result of the ability to acquire 3D point clouds and hence
there are very close ties between how the data is acquired and
the method used to reconstruct it. Early on, these techniques
ranged from active methods such as optical laser-based range
scanners, structured light scanners, and LiDAR scanners,
as well as passive methods such as multi-view stereo. A
recent trend has seen the massive proliferation of point clouds
from commodity real-time scanners such as the Microsoft
Kinect. As the diversity, ease of use, and popularity of 3D
acquisition methods continues to increase, so does the need
for the development of new surface reconstruction techniques.
Acquisition methods tend to produce point clouds containing a variety of properties and imperfections that pose signifc Author version
icant challenges for surface reconstruction methods. These
properties, in conjunction with the nature of the scanned
shape, effectively distinguish the class of reconstruction methods that exist today. This diverse set of techniques ranges
from methods that assume a well-sampled point cloud, generalize to arbitrary shapes, and produce a watertight surface
mesh, to methods that make very loose assumptions on the
quality of the point cloud, operate on specific classes of
shapes, and output a non-mesh based shape representation.
It is with this rich space of algorithms in mind that we survey the field of surface reconstruction and provide a detailed
taxonomy of existing methods. This categorization is timely,
as we see the field of surface reconstruction diverging from
its more traditional class of methods in an effort to handle
more challenging data imperfections.
Our survey presents surface reconstruction algorithms
from the perspective of priors: assumptions made by algorithms in order to combat imperfections in the point cloud
and recover as much information about the shape as possible.
Without prior assumptions, the reconstruction problem is illposed; an infinite number of surfaces pass through or near the
data points. Assumptions are usually imposed on the point
cloud itself, such as sampling density, level of noise, and mis-
Berger et al. / State of the Art in Surface Reconstruction from Point Clouds
understanding how to process future types of acquired point
clouds.
Organization. Our STAR is organized as follows. In Section 2 we characterize the problem of surface reconstruction by examining common input and output characteristics,
namely:
• Point Cloud Artifacts: the imperfections of the point
cloud that the method is able to effectively handle.
• Input Requirements: the types of inputs associated with
a point cloud required by the algorithm.
• Shape Class: the class of shapes that the method is capable
of reconstructing.
• Reconstruction Output: the representation and level of
detail of the reconstruction output.
Figure 1: Surface reconstruction has grown in diversity
in recent years, with a wide variety of techniques taking
on specialized priors. ROSA [TZCO09], shown on the left,
uses volumetric smoothness to aid in reconstruction. Nonlocal consolidation [ZSW∗ 10], shown in the middle, uses
global regularity in the form of structural repetition. Part
composition [SFCH12], shown on the right, uses data-driven
techniques to perform reconstruction.
alignment. But just as importantly they are also imposed on
the scanned shape, such as local surface smoothness, volumetric smoothness, absence of boundaries, symmetries, shape
primitives, global regularity, and data-driven assumptions. In
some instances, requirements are made on knowledge of the
acquisition, such as scanner head position, as well as RGB
images of the object. In other cases, the user is involved in
prescribing high-level cues for reconstruction. All of these
factors permit the regularization of the otherwise ill-posed
problem of surface reconstruction, particularly when processing a point cloud containing severe imperfections. Figure 1
depicts several different priors used to reconstruct surfaces
from challenging point clouds.
Historically, priors have evolved according to the types
of point clouds being processed. For instance, local surface
smoothness priors were developed primarily to handle small
objects acquired from desktop scanners. Mobile, real-time
scanners have enabled the dynamic acquisition of more general scenes, rather than single objects, prompting more specialized structural and data-driven priors. Since priors tend
to be coupled with the type of acquisition, we argue that
this perspective of surface reconstruction is beneficial for
We use these factors as a way of examining surface reconstruction methods by prior, starting with traditional surface
smoothness priors in Section 3, and delving into specialized
priors in Sections 4–9. In Table 1.1 we provide a summary
of surface reconstruction methods by prior, characterizing
their input and output, as well as their level of robustness to
various artifacts. We discuss methods for evaluating surface
reconstruction in Section 10, and conclude in Section 11 with
a discussion on future trends in surface reconstruction.
1.1. Survey Scope and Related Works
There are many facets to surface reconstruction. This survey
focuses on those relating to the reconstruction from point
clouds of static objects and scenes acquired through 3D scanners, wherein the point cloud contains a considerable level of
imperfection. Furthermore, we concentrate on methods that
approximate the input point cloud.
Urban reconstruction. Our survey covers a wide variety of
reconstruction methods, with urban reconstruction from point
clouds among them. We note that [MWA∗ 13] surveys urban
reconstruction more broadly: 3D reconstruction from images,
image-based facade reconstruction, as well as reconstruction
from 3D point clouds. Although there exists some overlap
between the surveys, we cover these methods in a different
context, namely the priors that underly the reconstruction
methods and how they address challenges in point cloud
reconstruction.
Surface completion. Given a surface with boundaries, there
exists many methods for inpainting and surface completion
for handling missing data. Though one may use such approaches for reconstruction by first reconstructing a surface
with boundary from a point cloud, this can be quite challenging given other imperfections in the data. These methods are
not covered in this survey and we refer the reader to the recent
survey of [ACK13] on surface completion.
Interpolatory reconstruction. An important field of surface
reconstruction methods are those that interpolate a point
cloud without any additional information, such as normals
c Author version
Berger et al. / State of the Art in Surface Reconstruction from Point Clouds
Surface Smoothness
Tangent Planes [HDD∗ 92]
RBF [CBC∗ 01]
MLS [ABCO∗ 03]
MPU [OBA∗ 03a]
Poisson [KBH06]
Graph Cut [HK06]
Unoriented Indicator [ACSTD07]
LOP [LCOLTE07]
Visibility
VRIP [CL96]
TVL1-VRIP [ZPB07]
Signing the Unsigned [MDGD∗ 10]
Cone Carving [SSZCO10]
Multi-Scale Scan Merge [FG11]
Volumetric smoothness
ROSA [TZCO09]
Arterial Snakes [LLZM10]
VASE [TOZ∗ 11]
l1 Skeleton [HWCO∗ 13]
Geometric Primitives
Primitive Completion [SDK09]
Volume Primitives [XF12]
Point Restructuring [LA13]
CCDT [vKvLV13]
Global Regularity
Symmetry [PMW∗ 08]
Nonlocal Consolidation [ZSW∗ 10]
2D-3D Facades [LZS∗ 11]
Globfit [LWC∗ 11]
Data-driven
Completion by Example [PMG∗ 05]
Semantic Modeling [SXZ∗ 12]
Shape Variability [KMYG12]
Part Composition [SFCH12]
Interactive
Topological Scribble [SLS∗ 07]
Smartboxes [NSZ∗ 10]
O-Snap [ASF∗ 13]
#
#
#
#
#
#
#
#
#
#
#
✓
#
✓
#
#
#
#
#
#
#
#
#
#
#
#
#
✓
✓
#
#
#
#
✓
#
#
#
✓
✓
✓
✓
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
✓
✓
✓
✓
✓
#
#
#
#
✓
#
#
#
#
#
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
#
#
✓
✓
Shape Class
Reconstruction Output
general
general
general
general
general
general
general
general
implicit field
implicit field
point set
implicit field
implicit field
volumetric segmentation
implicit field
point set
general
general
general
general
general
implicit field
implicit field
implicit field
implicit field
implicit field
organic
man-made
general
organic
skeleton curve
skeleton curve
implicit field
skeleton curve
CAD
indoor environment
general
urban environment
volumetric segmentation
interior volume
volumetric segmentation
volumetric segmentation
architectural
architectural
architectural
man-made
point set
point set
point set
primitive relations
general
indoor scene objects
indoor scene objects
man-made
point set
deformed model
deformed model
deformed model parts
general
architectural
architectural
implicit field
primitive shapes
primitive shapes
RGB image
#
#
#
scanner
information
oriented
normals
unoriented
normals
Input Requirements
missing data
misalignment
outliers
noise
Point Cloud Artifacts
nonuniform
sampling
Method
Table 1: A categorization of surface reconstruction in terms of the type of priors used, the ability to handle point cloud artifacts,
input requirements, shape class, and the form of the reconstruction output. Here # indicates that the method is moderately
robust to a particular artifact and indicates that the method is very robust. ✓indicates an input requirement and ✓indicates
optional input.
or scanner information. Delaunay-based methods are quite
common in this area. The basic idea behind these methods
is that the reconstructed triangulated surface is formed by a
subcomplex of the Delaunay triangulation. A comprehensive
survey of these methods is presented in [CG06], as well as
the monograph of [Dey07]. A very attractive aspect of such
methods is that they come with provable guarantees in the
geometric and sometimes topological quality of the reconstruction if a sufficiently dense sampling of the input surface
is provided. These methods place rather strong requirements
on the point cloud and are impractical for scanned real-world
scenes containing significant imperfections. Hence we do
not cover these methods, since we focus on reconstruction
techniques capable of dealing with challenging artifacts.
2. Characterizing Surface Reconstruction
Surface reconstruction methods typically have to handle various types of imperfections, make certain requirements on
c Author version
input associated with the point cloud, contain restrictions
on the class of shapes that they can reconstruct, and may
produce reconstructions of different forms. Here we summarize each of these properties in order to provide a detailed
characterization of surface reconstruction.
2.1. Point Cloud Artifacts
The properties of the input point cloud are an important factor in understanding the behavior of reconstruction methods.
Here we provide a characterization of point clouds according
to properties that have the most impact on reconstruction
algorithms: sampling density, noise, outliers, misalignment,
and missing data. See Figure 2 for a 2D illustration of these
artifacts.
Sampling density. The distribution of the points sampling
the surface is referred to as sampling density. 3D scans typically produce a nonuniform sampling on the surface, which
can be due to the distance from the shape to the scanner posi-
Berger et al. / State of the Art in Surface Reconstruction from Point Clouds
ther explicitly through detection [LCOLTE07], or implicitly
through robust methods [MDGD∗ 10].
(a) Original shape
(b) Nonuniform sampling
(c) Noisy data
(d) Outliers
(e) Misaligned scans
(f) Missing data
Figure 2: Different forms of point cloud artifacts, shown
here in the case of a curve in 2D.
tion, the scanner orientation, as well as the shape’s geometric
features. See Figure 2(b) for an illustration of nonuniform
sampling on a curve. Many surface reconstruction algorithms
must be able to estimate a notion of sampling density at every
point, see e.g. [LCOL06, WSS09], and hence the level of
nonuniformity in the sampling can have a great impact on
estimation accuracy.
Noise. Points that are randomly distributed near the surface
are traditionally considered to be noise – see Figure 2(c). The
specific distribution is commonly a function of scanning artifacts such as sensor noise, depth quantization, and distance
or orientation of the surface in relation to the scanner. For
some popular scanners, noise is introduced along the line
of sight, and can be impacted by surface properties, including scattering characteristics of materials. In the presence
of such noise, the typical goal of surface reconstruction algorithms is to produce a surface that passes near the points
without overfitting to the noise. Robust algorithms that impose smoothness on the output [KBH06], as well as methods
that employ robust statistics [OGG09], are common ways of
handling noise. We note that spatially varying noise poses a
significant challenge [GCSA13], where for many scanners,
the noise level is correlated with the depth measurement –
see [KE12] for such an error study done on the Kinect.
Outliers. Points that are far from the true surface are classified as outliers. Outliers are commonly due to structural
artifacts in the acquisition process. In some instances, outliers are randomly distributed in the volume, where their
density is smaller than the density of the points that sample
the surface. Outliers can also be more structured, however,
where high density clusters of points may exist far from the
surface, see Figure 2(d). This can occur in multi-view stereo
acquition, where view-dependent specularities can result in
false correspondences; see Figure 10. Unlike noise, outliers
are points that should not be used to infer the surface, ei-
Misalignment. The imperfect registration of range scans
results in misalignment. Misalignment tends to occur for
a registration algorithm when the initial configuration of
a set of range scans is far from the optimal alignment –
see [vKZHCO11] for a survey on registration techniques.
Misalignment is a significant challenge for surface reconstruction, as it introduces structured noise via scans that are
slightly offset from the surface; see Figure 5(a). For instance,
it may be inappropriate to simply find the surface that passes
near the scans since this may result in sharp discontinuities
between different scans. Figure 2(e) illustrates such a case,
where there can exist discontinuities when the red and orange
samples stop overlapping.
Missing data. A motivating factor behind many reconstruction methods is dealing with missing data. Missing data is due
to such factors as limited sensor range, high light absorption,
and occlusions in the scanning process where large portions
of the shape are not sampled. Although some of these artifacts may be reduced as scanning technology advances with
higher precision, denser sampling, and lower noise levels,
occlusion remains a persistent problem due to the physical
constraints of the device. We note that missing data differs
from nonuniform sampling, as the sampling density is zero
in such regions – see Figure 2(f).
Many methods deal with missing data by assuming that the
scanned shape is watertight [CBC∗ 01, Kaz05, KBH06, HK06,
ACSTD07]. Within this setting, the goal of some methods is
to handle the aforementioned challenges where data exists,
and infer geometry in parts of the surface that have not been
sampled. Other methods are focused on handling missing
data by trying to infer topological structures in the original
surface at the possible expense of retaining geometric fidelity,
for instance, finding a surface that is homeomorphic to the
original shape [SLS∗ 07].
If the level of missing data is significant, for instance a
single scan, then trying to infer the entire shape can be too
ambiguous. Some methods focus on performing reconstruction only on the available information, effectively preserving
the boundaries from the scan [DGQ∗ 12]. Other approaches
make prior knowledge and assumptions on the missing region,
permitting the reconstruction of higher-level information.
This can range from inferring a skeleton [TZCO09], shape
primitives [SDK09], symmetry relationships [PMW∗ 08], and
canonical regularities [LWC∗ 11].
2.2. Point Cloud Input
Reconstruction methods have different types of input requirements associated with a point cloud. The bare minimum
requirement of all algorithms is a set of 3D points that sample
the surface. Working with the points alone, however, may
fail to sufficiently regularize the problem of reconstruction
c Author version
Berger et al. / State of the Art in Surface Reconstruction from Point Clouds
for certain types of point clouds. Other types of input can be
extremely beneficial in reconstruction from challenging point
clouds. We consider the following basic forms of inputs commonly associated with point clouds: surface normals, scanner
information, and RGB imagery.
2.2.1. Surface Normals
Surface normals are an extremely useful input for reconstruction methods. For smooth surfaces the normal, uniquely
defined at every point, is the direction perpendicular to the
point’s tangent space. The tangent space intuitively represents
a localized surface approximation at a given point. Surface
normals may be oriented, where each normal is consistently
pointing inside or outside of the surface, or may lack such a
direction. Normals that are oriented provide extremely useful
cues for reconstruction algorithms. However, we note that
if certain information associated with the point cloud is not
present, obtaining an orientation can be challenging.
Unoriented normals. Normals that do not possess direction
– the input normal at every point can be expected to be pointing either in the inside or the outside of the surface – are
considered to be unoriented normals. This information can
be used in a number of ways: determining planar regions
in a point cloud [SWK07], the projection of a point onto an
approximation of the surface [ABCO∗ 03], the construction of
an unsigned distance field [AK04], or computing covariance
matrices [ACSTD07]. Unoriented normals are typically computed directly from the point cloud. This is because additional
scanner-specific information can be used to provide a means
to infer normal orientation. A popular and simple method for
computing the normal at a given point p is to perform principal component analysis (PCA) in a local neighborhood of
p, see e.g. [HDD∗ 92]. More specifically, if we denote a local
neighborhood of p by N p , then the basic way to apply PCA
is to compute the spectral decomposition of the covariance
matrix:
Cp =
∑
(p − q)(p − q)⊺
(1)
q∈Np
The eigenvector of C p associated with the smallest eigenvalue
defines the unoriented normal – assuming eigenvalues have
a multiplicity of 1, the eigenvectors are unique up to a sign.
Note that if the smallest eigenvalue is 0, then the region defined by p and N p is planar, since the eigenvectors associated
with the 2 largest eigenvalues capture all of the variance in
the data.
PCA defines a least-squares estimation of a tangent plane
and there are many other methods for computing unoriented normals: using a weighted covariance matrix [PMG04],
higher-order approximations via osculating jets [CP05], or robust methods that employ l1 norm minimization [ASGCO10].
Common to all methods is the need to define a local neighborhood of points, where the neighborhood should be small
enough to accurately represent a point’s tangent space. The
c Author version
scale of the neighborhood should be proportional to the sampling density at the point, where estimation of sampling density is itself a challenging problem, particularly when faced
with nonuniform sampling; see [LCOL06, WSS09]. Furthermore, noise and misalignment may necessitate larger neighborhood sizes in order to combat such imperfections, yet
the size should not be so large as to no longer reflect the
point’s tangent space [GG07]. All of these difficulties often
result in imperfect normal estimation and surface reconstruction algorithms must be robust to inaccuracies in unoriented
normals.
Oriented normals. Normals that have consistent directions,
either pointing in the inside or the outside of the surface are
referred to as being oriented. Knowledge of the exterior and
interior of the surface has proven extremely useful in surface
reconstruction. It can be used to construct a signed distance
field over the ambient space, where up to a sign, the field takes
on positive values in the exterior and negative values in the
interior. The surface is then represented by the zero crossing
of the signed distance field. Other methods generalize this to
implicit fields and indicator functions, but the basic idea of
trying to construct the exterior and interior remains the same,
see [CBC∗ 01, OBA∗ 03a, KBH06] to name a few.
There are numerous ways to compute oriented normals. If
the original 2D range scans are known, then the 2D lattice
structure provides a way of performing consistent orientation
since one always knows how to turn clockwise around a given
vertex. For instance, if we denote the point in a range scan at
pixel (x, y) as px,y , then one can take the normal at px,y simply
as the cross product between (px+1,y − px,y ) and (px,y+1 −
px,y ). If the point cloud is noisy, then this method can produce
rather noisy normals, since it does not use nearby points in
overlapping scans. If the view direction is known, then one
can first estimate unoriented normals as previously discussed
to better handle noise and then use the view direction to
determine the orientation. This can be done by choosing the
orientation that results in the normal vector having the largest
angle with the view direction.
If scanner information is absent altogether, then one must
orient the points exclusively from the unoriented normals.
A very common method for achieving this is to start from a
single point containing an initial orientation and propagate
the orientation to nearby points whose unoriented normals
are facing a similar direction [HDD∗ 92]. While there exist
multiple extensions to this method [HLZ∗ 09, LW10], they
face difficulty in the presence of nonuniform sampling, noise,
and misalignment and as a result can leave some normals
unoriented or pointing in the wrong direction – see Figure 3.
The impact on surface reconstruction largely depends on the
distribution of incorrect orientations: if randomly distributed,
then methods may treat this as spurious noise, but if incorrect
orientations are clustered together over large regions, then
this form of structured noise can be difficult to handle – see
Figure 3 for an illustration.
Berger et al. / State of the Art in Surface Reconstruction from Point Clouds
complete data – for instance, line of sight can be used to infer
that there exists a large concavity in the shape [TOZ∗ 11].
2.2.3. RGB Imagery
Figure 3: The impact of incorrect normal orientation. On the
left we show the result of normal orientation via [HDD∗ 92],
where red splats indicate incorrect orientation. The results
of running Poisson surface reconstruction [KBH06] on this
point cloud are shown in mid-left, where we indicate unwanted surface components due to the clustered normal
flips. Similarly, on the right we show the orientation results
of [LW10], and the corresponding results of [KBH06].
2.2.2. Scanner Information
The scanner from which the point cloud was acquired can
provide useful information for surface reconstruction. As
discussed, the 2D lattice structure of a scan can be used to
determine normal orientation. It can also be used to estimate
the sampling density for a given scan and as a result it can be
used to detect certain forms of outliers in the scan – points
whose scan lattice neighbors are at a far greater distance than
the sampling density are likely outliers. However, caution
must be taken in distinguishing outliers from sharp features.
Scanner information may also be used to define the confidence of a point, which is useful in handling noise. For
instance, the approach of [CL96] demonstrates how to use
confidence measures to suppress noise when integrating range
scans into a reconstructed surface. Certain scanners (e.g. LiDAR) can provide confidence measures in the form of the
reflectivity measured at each point.
One can also derive confidence through line of sight information. Line of sight is defined as a collection of line
segments between each point in the point cloud and the scanner head position from which that point was acquired. In
particular, we can use the grazing angle formed by line of
sight at each point to derive a notion of confidence. The grazing angle is the incident angle between the point’s surface
normal and its line of sight. In active scanning systems such
as optical laser-based scanners, the cause of inaccuracy stems
from the large area formed by the projection (i.e., the laser
stripe) onto the surface at a large grazing angle. Hence we can
assign a confidence measure as being inversely proportional
to the grazing angle.
Note that line of sight also defines a region of space that
may be marked as lying outside of the shape. Combining line
of sight from multiple scans refines the bounding volume in
which the surface lies – this volume is known as the visual
hull. This information is particularly useful when handling in-
Different acquisition modalities that complement depth acquisition can be of great assistance. RGB image acquisition is
a very common modality that accompanies numerous sensors,
such as the Microsoft Kinect. In the case of the Kinect, the
RGB camera is co-located with the IR camera, hence assuming the two are calibrated, it is straightforward to identify
corresponding depth and RGB values at a pixel level. RGB
images are most useful for reconstruction when they are able
to complement depth information that was not measured by
the data. By fusing features present in the depth scan with
image-based features, one can then employ this for inferring
depth from images [LZS∗ 11]. Contours in an image can also
be used by considering their corresponding unsigned distance
field, and how it relates to the point cloud’s unsigned distance
field [SFCH12].
2.3. Shape Class
Surface reconstruction algorithms can be further distinguished by the class of shapes they support. Although earlier
reconstruction methods tended not to focus on specific types
of shapes [HDD∗ 92, CL96], by restricting to a shape class,
specialized assumptions can be made on what the reconstructed surface should be. These shape specific assumptions
greatly help regularize the problem especially in the presence of point cloud artifacts. Quite often, a reconstruction
prior is in part driven by a shape class, so understanding the
characteristics of a shape class is an essential component to
gaining insight into surface reconstruction. Here we cover
the following predominant shape classes: CAD models, manmade shapes, organic shapes, architectural models, urban
environments, and indoor environments.
CAD models. These models are typically composed of a
collection of simpler geometric primitives such as planes,
cylinders, and spheres. The detection of such instances in the
point cloud can be used for denoising and in the presence of
missing data the assembly of all detected primitives can be
used to infer missing regions by extending and intersecting
primitives [SDK09] – see Figure 12.
Man-made (synthetic) shapes. These shapes often contain
certain canonical geometric properties such as coplanar faces,
orthogonal faces, as well as faces that form equal angles that
often repeatedly appear and relate different parts of the shape.
This form of global regularity is often due to aesthetic considerations and a variety of practical constraints, such as cost
considerations, functional requirements, and fabrication constraints. For man-made shapes, this derived form of regularity
can greatly help the problem [LWC∗ 11] – see Figure 14.
Organic shapes. These shapes tend to contain a more freeform structure and are often composed of curvilinear elements. For instance, trees [LYO∗ 10] possess a strong skeletal
c Author version
Berger et al. / State of the Art in Surface Reconstruction from Point Clouds
structure, where imposing certain 1D priors [TZCO09] on
the reconstruction can be used to handle significant missing
data in the scanning process – see Figure 11.
Architectural models. A subset of man-made shapes, these
shapes contain similar global regularities in addition to many
other constraints such as upright orientation and various
functional constraints. By assuming that the shape is an architectural model, the problem can be greatly regularized
by making assumptions on facade structures [ZSW∗ 10],
manhattan-world geometry [VAB12], and structural regularity [PMW∗ 08] – see Figure 16.
Urban environments. Often composed of a limited number
of object types, urban environments are well-suited for datadriven methods for reconstruction. For instance one can make
assumptions on the presence of ground, buildings, vegetation,
and other urban objects to aid in reconstruction [MWA∗ 13].
Indoor environments. The types of shapes within this environment tend to be a mixture of man-made and organic. A distinguishing factor behind indoor environments is that similar
to urban environments, there exist a small number of object
types. For instance, in a typical office setting there exists a
variety of chairs, desks, and tables. Furthermore, each type of
object can often be defined through a low-dimensional shape
space, permitting data-driven methods [NXS12] and methods
that utilize a deformable model for each object [KMYG12] –
see Figure 15.
2.4. Reconstruction Output
It is desirable for a reconstruction algorithm to produce a
faithful and detailed representation of the scanned shape’s
surface. For challenging point clouds containing significant
imperfections, it may be unrealistic to expect such highly detailed information as the output. However for certain methods
it is still possible to obtain a less informative yet valuable
shape representation.
Methods targeting detailed reconstruction generally produce as output either a discrete surface or an implicit function.
The implicit function can either be in the form of a signed
distance field [HDD∗ 92] or an indicator function [Kaz05].
Implicit functions are usually sampled on an underlying grid,
where the reconstructed surface is found via isocontouring for
an appropriate isovalue. For a regular grid, the well-known
Marching Cubes [LC87] is commonly used to extract the
surface [HDD∗ 92, CL96, CBC∗ 01, Kaz05]. Other methods
use grids such as octrees [KBH06, MPS08] or adaptive 3D
triangulations [ACSTD07, MDGD∗ 10] to adapt grid resolution to the point sampling density. Contouring an octree
presents several difficulties in ensuring watertight, manifold
surface meshes – see [JLSW02, KKDH07, MS10] for several approaches. Contouring a triangulation can be done via
marching tetrahedra, but if a mesh with a lower number of
triangles and well-conditioned (i.e. good aspect ratio) trianc Author version
gles is desired, then Delaunay refinement techniques may be
used [BO05].
Some techniques may not produce a surface representation, but rather a resampled point set that addresses imperfections present in the original point cloud [ABCO∗ 03,
FCOS05b, LCOLTE07, HLZ∗ 09]. For severe levels of imperfections, some reconstruction methods may be unable to
provide such detailed representations, but still recover an
informative representation such as a collection of shape primitives [JKS08, SDK09, RKMP13], a curve skeleton [TZCO09,
CTO∗ 10, LLZM10, HWCO∗ 13], a deformed model that best
matches the point cloud [SXZ∗ 12, NXS12, KMYG12], or
segmented parts from multiple models [SFCH12].
3. Surface Smoothness Priors
Early surface reconstruction techniques were developed to
handle broad assumptions on the type of shape being reconstructed. These methods were developed to handle sampling
and noise artifacts, while also supporting small amounts of
missing data. A commonality shared across all of these techniques is the use of a surface smoothness prior to constrain
the output surface, while also ensuring that the reconstructed
surface remains close to the input data. Methods in this category vary based on the smoothness constraints and how they
are prescribed in practice. More specifically, this category
of methods can roughly be divided into local smoothness,
global smoothness, and piecewise smoothness.
Methods that prescribe local smoothness ensure that the
output surface is smooth where the input point cloud exists [ABCO∗ 03, OBA∗ 03a]. In regions away from the point
cloud, however, these methods can behave poorly, failing
to reconstruct the correct global geometry and topology. In
contrast, global smoothness methods prescribe that the entirety of the output surface is smooth [CBC∗ 01, KBH06].
Hence, these methods are better suited for handling missing data, as they are all targeted at producing a watertight
surface. These methods are further distinguished by often
solving an optimization problem that leads to the use of a
global solver of some kind – typically a linear system, an
eigenvalue problem, or a graph cut optimization. Piecewise
smooth methods are focused on explicitly recovering sharp
features or boundary components, while ensuring smoothness away from these features, where smoothness may be
prescribed locally [DHOS07] or globally [ASGCO10].
Notation. We first fix the notation for this section and all
subsequent sections. We assume that we are given a point
cloud P which is a sampling of a shape S. Individual points
in P are indexed as pi ∈ P for the i’th point. Many methods
also require normals associated with the point cloud, where
we define the normal field N as a set of normal vectors such
that for each pi ∈ P there is an accompanying normal ni ∈ N.
The distinction between oriented and unoriented normals is
made explicit for each method.
Berger et al. / State of the Art in Surface Reconstruction from Point Clouds
(a)
(b)
(c)
Figure 4: When sampling density is insufficient to resolve
local curvature (a), the plane fitting operation employed by
moving least squares [ABCO∗ 03] becomes highly unstable
(b). APSS [GG07] addresses this problem by locally fitting
spheres instead of planes. Employing spheres tackles the
aforementioned problem while remaining computationally
inexpensive.
3.1. Local Surface Smoothness Priors
The pioneering method of [HDD∗ 92] was hugely influential
on the class of methods that impose local smoothness priors.
This method approximates a signed distance field f : R3 → R
by assigning, for each point in the ambient space x ∈ R3 , its
signed projection onto the tangent plane of its closest point
to P, denoted pi :
f (q) = (q − pi ) · ni .
(2)
Note that the normal field N must be oriented in order to
obtain an estimate of the signed distance field. The surface is
then defined by the zero level set of f . Although straightforward to implement, this approach suffers from several issues.
The method is very sensitive to the estimated normals – noisy
normals, or worse inverted normal orientations, can give rise
to very inaccurate signed distance estimates. Furthermore,
in the presence of nonuniform sampling, choosing the closest tangent plane to define a signed projection can produce
a rather noisy output – see Figure 6, mid-left. Subsequent
methods based on local surface smoothness have focused on
addressing such issues.
Moving least squares (MLS). These methods approach reconstruction by approximating the surface as a spatiallyvarying low-degree polynomial – see [CWL∗ 08] for a survey
on MLS methods. More specifically, in the traditional MLS
formulation [ABCO∗ 03], points are first locally parameterized by their projection on the local tangent space. Then, in
this parameterized space, a weighted fitting estimates a lowdegree bivariate polynomial approximating the input cloud.
MLS then defines the evaluating point’s projection onto the
reconstructed surface as the closest point to the bivariate polynomial [ABCO∗ 03]. We note that this projection process only
requires unoriented normals and can also be used to define
an unsigned distance function [AK04].
Such a projection process allows for resampling the point
cloud, producing a so called point set surface – the surface is
implicitly defined as the fixed point of the projection operator.
However, it is nontrivial to explicitly construct a continuous
representation, for instance an implicit function or a triangle
mesh, when the user wants to know the precise geometry and
topology of the MLS surface for geometry processing tasks
that require a continuous surface representation. Advancing
front methods [SFS05, SSFS06] produce a triangle mesh by
incrementally laying out triangles to compose fronts, where
vertex positions are determined through the MLS definition.
Care must be taken, however, when fronts merge and split.
If normals are oriented, then one can simplify the MLS definition to arrive at an implicit function representation for the
surface. Namely, the method of [AA04] constructs an implicit
function at a point as the signed distance to a weighted average of nearby tangent planes. The method of [SOS04] uses a
weighted average of distances to nearby tangent planes. Both
methods assume that it is possible to construct a well-defined
tangent plane at each evaluation point, which may not exist
for sparsely sampled data. In this case, a higher-order approximation such as algebraic point set surfaces [GG07], which
uses an MLS definition with spheres for shape approximation,
can be more robust – see Figure 4.
A key property of MLS is the use of a weighting function, used to give larger influence to points near the evaluating point in both estimating the tangent space as well as
constructing the polynomial. This can be used to combat
moderate levels of noise by allowing the weight function to
have a larger spatial influence. For nonuniform sampling, it is
necessary to define a weight function whose spatial support
varies as a function of the sampling density. This may be
done via a user-defined scale proportional to an estimation
of the density [GG07], as well as methods whose support
is derived from a data-dependent error bound on the MLS
approximation [LCOL06]. However, MLS methods are in
general unable to provide a good surface approximation in
the presence of missing data, since it is necessary to use a
rather large spatial support size in such regions for which a
tangent plane (or sphere) may provide a poor fit.
Multi-level partition of unity (MPU). For this set of techniques, the reconstruction problem is approached as a hierarchical fitting problem [OBA∗ 03a]. At a certain scale, a local
shape fit is determined adequate if its error residual is sufficiently small, otherwise the occupied space is refined and
the overall procedure is repeated. Once all shape fits have
been performed, an implicit function over the entire volume
is formed by smoothly blending nearby fits. Note that the
definition of an implicit function requires the shape fits to be
signed fits, hence requiring oriented normals. Compared to
MLS methods, MPU is more robust to nonuniform sampling
since it does not require an estimate of the sampling density: a shape fit is only accepted if it is below a certain error
residual. The level of smoothness and hence robustness to
noise can be adjusted by the error residual tolerance. Missing
c Author version
Berger et al. / State of the Art in Surface Reconstruction from Point Clouds
(a)
Figure 5: The scan on the left contains outliers and scan
misalignment, particularly near its boundaries. The output
of [LCOLTE07], shown on the right, is able to robustly deal
with such challenging data through the use of multivariate
median data fitting, and a point attraction term in the output.
data can be addressed by allowing for the extrapolation and
subsequent blending of spatially adjacent shape fits. However,
such an extrapolation may produce erroneous surface sheets,
depending on the form of missing data. To resolve such poor
fits, a diffusion operator can be defined on the collection of
shape fits, in order to perform smoothing directly on the MPU
representation [NOS09].
Parameterization-free projection. These methods project
the point cloud, without normals, onto the multivariate median, where a point balancing term ensures samples are asuniformly-as-possible distributed in the output [LCOLTE07].
This method is limited to outputting a resampled point cloud,
where the resampled points are restricted to where the input
data lies, retaining boundary components. However, by using
a multivariate median, parameterization-free projection does
not need to estimate a local tangent plane or perform shape
fitting as in MLS and MPU methods, respectively. Hence for
strong noise, outliers, and even misalignment, this type of
approach is quite robust – see Figure 5. This approach was extended by [HLZ∗ 09] to handle highly nonuniform sampling
by incorporating an estimation of sampling density into the
balancing term.
3.2. Global Surface Smoothness Priors
Global smoothness priors typically involve higher order
smoothness, large-scale smoothness, or both. High order
smoothness relates to the variation of differential properties
of the surface: area, tangent plane, curvature, etc. Large-scale
herein relates to the spatial scale where smoothness is enforced.
Radial basis functions (RBFs). RBFs are a well-known
method for scattered data interpolation. Given a set of points
with prescribed function values, RBFs reproduce functions
containing a high degree of smoothness through a linear combination of radially symmetric basis functions. For surface reconstruction, the method of [CBC∗ 01] constructs the surface
c Author version
(b)
(c)
(d)
Figure 6: A point cloud sampling a sphere consisting of 150
points (a) is reconstructed by [HDD∗ 92] resulting in a C0
surface (b). A locally supported RBF [Wen95] reconstruct
a C1 surface, while the global triharmonic RBF (∆3 f = 0,
φ(x) = x3 ) outperforms the previous methods, although incurring a high computational cost.
by finding an implicit function defined via RBFs whose zero
level set represents the surface. More specifically they use
globally-supported basis functions of the form φ(x) = kxk2 .
The implicit function f may then be expressed as:
n
f (x) = p(x) + ∑ λi φ(x − pi ),
(3)
i=1
where p denotes a low-degree polynomial and the basis functions are shifted by the evaluation point x.
The coefficients λi are found by prescribing, as interpolation constraints, a function value of 0 for pi ∈ P. Offsurface constraints are necessary to avoid the trivial solution
of f (x) = 0 for x ∈ R3 . Positively (resp. negative) valued
constraints are set for points displaced at pi along ni in the
positive (resp. negative) direction. The displaced points are
selected such that each one’s closest point in P is pi . The coefficients λi are found via a dense linear system in n, where by
exploiting the structure of φ, fast multipole methods are used
to reduce the complexity from O(n3 ) to O(n log n) [CBC∗ 01].
An advantage to using globally-supported basis functions
for surface reconstruction is that the resulting implicit function is globally smooth; see Figure 6(d) for triharmonic RBFs,
compared to compactly-supported basis functions [Wen95]
shown in Figure 6(c). Hence RBFs can be effective in producing a watertight surface in the presence of nonuniform
sampling and missing data. However, when the input contains moderate noise, determining the proper placement of
off-surface points can become challenging.
Indicator function. These methods approach surface reconstruction by estimating a soft labeling that discriminates the
interior from the exterior of a solid shape. This is accomplished by finding an implicit function χ that best represents
the indicator function, taking on the value of 0 in the interior
of the shape and 1 otherwise. The key observation in this
class of methods is that, assuming a point cloud with oriented
normals, χ can be found by ensuring its gradient is as-closeas-possible to the normal field N, in a least-squares sense,
via k∇χ − Nk22 . If we apply the divergence operator to this
problem, then this amounts to solving the following Poisson
Berger et al. / State of the Art in Surface Reconstruction from Point Clouds
equation:
∇ · ∇χ = ∆χ = ∇ · N.
(4)
Once solved, the surface is found via χ with a suitable isovalue, typically the average or median value of χ evaluated at
all of the input points.
The approach of [Kaz05] solves this by transforming the
problem into the frequency domain, where the Fourier transforms of ∆χ and ∇ · N result in a simple algebraic form for
obtaining the Fourier representation of χ. By operating in the
frequency domain, however, it is necessary to use a regular
grid in order to apply the FFT, hence limiting spatial resolution in the output. In order to scale to larger resolutions,
the method of [KBH06] directly solves for χ in the spatial
domain via a multi-grid approach, hierarchically solving for
χ in a coarse-to-fine resolution manner.
Indicator function methods are an instance of gradientdomain techniques, which are known to be robust methods for such applications as high dynamic range compression [FLW02] and image blending [PGB03]. For surface
reconstruction, such a gradient-domain formulation results in
robustness to nonuniform sampling, noise, outliers, and to a
certain extent missing data. The implicit function’s gradient
being well-constrained at the data points enforces smoothness and a quality fit to the data and since the gradient is
assigned zero away from the point cloud, χ is smooth and
well-behaved in such regions. Furthermore, for small scan
misalignment, normals tend to point in a consistent direction, which yields a well-defined gradient fit for the indicator
function. Several extensions have been made to this original
formulation, addressing limitations related to streaming reconstruction [MPS08], faithfulness to the input [KH13], and
sensitivity to normals [ACSTD07].
The work of [MPS08] solves the Poisson equation in a similar manner to [Kaz05], but rather than using a Fourier basis,
it uses wavelets in such a way that computing the wavelet
transform of χ may be done in a local manner, where higher
resolution wavelet bases contain smaller spatial support. Compact support is particularly advantageous in streaming surface
reconstruction, where the reconstruction is done on a subset
of the data at a time.
A known issue with the approach of [KBH06] is that fitting directly to the gradient of χ can result in over-smoothing
of the data [KH13, Fig. 4(a)]. To address this, the method
of [KH13] directly uses the point cloud as positional constraints into the optimization, resulting in a screened Poisson
problem. Similarly, the method of [CT11] incorporates positional, gradient, as well as Hessian constraints on the implicit function, where the Hessian constraint can improve
surface extrapolation in regions of missing data [KH13,
Fig. 6(a)]. The main difference between the approaches is
that [KH13] solves the problem via a finite-element formulation, whereas [CT11] use finite-differences, due to the complexity in discretizing the Hessian term. In particular, the
Figure 7: From the point cloud on the left, we show a slice
of the implicit function produced in [ACSTD07] on the right,
where only unoriented normals are necessary in producing a
signed field. An isotropic surface mesh of the zero isosurface
of the signed field is also depicted.
screened Poisson formulation [KH13] is up to 2 orders of
magnitude faster than [CT11], see [KH13, Table 1].
All of the above approaches rely on oriented normals,
where although such methods can tolerate sparsely distributed
normal orientation flips, large continuous clusters of improper
normal orientation can significantly impact these methods.
To address this, the method of [ACSTD07] uses covariance
matrices to represent unsigned orientations, rather than using
normals. A covariance matrix is formed at a given point by
taking a union of neighboring Voronoi cells around the point.
The anisotropy of the covariance acts as a notion of normal
confidence. The implicit function is found by maximizing its
gradient’s alignment with the principal component of the covariance matrix at each point, while ensuring smoothness and
a proper signed function by enforcing the function’s biharmonic energy to be small. This amounts to solving a sparse
symmetric generalized eigenproblem – Figure 7 depicts a
slice of the resulting eigenvector for the kitty point cloud.
Volumetric segmentation. These methods perform reconstruction via a hard labeling of a volumetric discretization,
where the goal is to label cells as being either interior or
exterior to the surface. The method of [KSO04] constructs a
graph Laplacian from the Delaunay triangulation of P, where
each node represents a tetrahedron of the triangulation and
each edge measures the likelihood of the surface passing
through the adjacent tetrahedra. The Laplacian eigenvector
with smallest nonzero eigenvalue then smoothly segments
tetrahedra into interior and exterior, as this eigenvector simultaneously seeks a smooth labeling and a partitioning with low
edge weights. This approach has shown to be robust to noise
and outliers without the use of normals, thanks to the robustness of spectral partitioning. Since it produces an explicit
c Author version
Berger et al. / State of the Art in Surface Reconstruction from Point Clouds
volume segmentation, it also ensures a watertight surface.
However, in regions of missing data, the discretization from
the Delaunay triangulation may be too coarse, giving a poor
approximation to the surface [KSO04, Fig. 6].
Methods based on graph cuts are also used in finding a
volumetric segmentation. The method of [HK06] first defines
a small crust on the exterior and interior through a dilation
operation on point-occupied voxels. A graph is imposed on
the crust where edges reflect the local surface confidence –
a function of a smoothed unsigned distance, while nodes on
the boundaries of the interior and exterior crust are connected
to a source and sink node, respectively. To impose global
smoothness, non-terminal edges also include a small regularization constant, which encourages minimal surface area.
This method does not use normals, as it only needs to compute an unsigned distance in order to define edge weights in
the graph cut solve. This results in robustness to nonuniform
sampling, noise, and misalignment. Furthermore, the minimal
surface area regularization allows for the method to handle
missing data, where we note that such a regularization is absent in the spectral segmentation approach of [KSO04]. However, computing the crust such that the interior and exterior
are identifiable can be challenging in certain configurations.
3.3. Piecewise Surface Smoothness Priors
Moving from the smooth, closed case to the piecewise smooth
case (possibly with boundaries) is substantially harder as the
ill-posed nature of the problem applies to each sub-feature of
the inferred shape. The features of a piecewise smooth surface
range from boundary components, sharp creases, corners,
and more specific features such as tips, darts, and cusps.
In addition, the inferred surface may be either a stratified
manifold or a general surface with non-manifold features.
Another difficulty stems from the fact that a feature is a
notion that exists at specific scales, such that reconstruction
and feature approximation cannot be decoupled.
Semi-sharp features. One step toward piecewise smoothness is a class of feature-preserving methods based on implicit representations. Sharp features can be captured through
locally adapted anisotropic basis functions [DTS01]. Adamson and Alexa [AA06] rely on an anisotropic moving least
squares (MLS) approach, where the anisotropy is based on
principal curvatures derived from the points’ positions and
normals. Oztireli et al. [OGG09] extend the MLS through kernel regression to allow for sharper features. However, none of
these techniques reproduce truly sharp features: the features
in a reconstruction contain varying degrees of smoothing.
Moreover, the presence of sharpness in the geometry of a
point set is detected only locally, which often leads to fragmented creases when defects are present [ASGCO10, Fig.
6].
Locally-sharp features. Another way to detect local sharpness consists in performing a local clustering of estimated
c Author version
normals [OBA∗ 03b]: the algorithm fits as many quadrics
as the number of clusters of normals. Improved robustness
is achieved in [FCOS05b] by segmenting neighborhoods
through region growing. Lipman et al. [LCOL07] enriches
the MLS projection framework with sharp edges driven by
the local error of the MLS approximation. However, the
locality of the feature detection can generate fragmented
sharp edges, much like general feature detection approaches
[GWM01, PP09].
Globally-sharp features. To reduce crease fragmentation,
some approaches favor the extraction of long sharp features.
Pauly et al. [PKG03] use a multi-scale approach to detect
feature points and construct a minimum-spanning tree to infer
a feature graph. Daniels et al. [DHOS07] uses a robust projection operator onto sharp creases and grow a set of polylines
through projected points. Jenke et al. [JWS08] extract feature
lines by robustly fitting local surface patches and computing
the intersection of close patches with dissimilar normals.
Sharp and robust. Only few approaches tackle the combined
issue of robustness to defect-laden point clouds and featurepreserving reconstruction [FCOS05a, ASGCO10, HWG∗ 13].
The method of [FCOS05a] uses a least median of squares
regression scheme in its region growing approach to handle
outliers in the data. The approach of [ASGCO10] first estimates normals that preserve sharp features in the shape via
l1 sparse reconstruction, i.e. the vector of all neighboring
normal differences should be sparse, where large nonzero differences reflect sharp features. The positions are then found
as offsets from the recovered sharp normals. The method
of [HWG∗ 13] allows for the preservation of sharp features
in a resampled point cloud by first resampling away from
detected edges in order to reliably compute normals and then
upsampling in a manner that preserves sharp features, determined by the normals. This method employs a similar energy
to [LCOLTE07] away from sharp features, hence they are
able to handle similar types of defects in the point cloud.
4. Visibility Priors
Although methods based on surface smoothness priors can
support a large variety of inputs, such a rather general assumption places restrictions on the extent to which they can
support substantial artifacts in the point cloud. To handle challenging point clouds, it is useful to consider specific priors
for reconstruction. In this section we consider visibility as a
prior and how it can help regularize certain reconstruction
problems.
Visibility has generally been used in three different ways.
The first class of methods considers how to use the visibility
provided by the scanner that produced the point cloud – this
is used primarily to obtain the line of sight associated with
each sample; see Section 4.1. The second class of methods
uses line of sight that is not provided from the scanner, but
rather approximated from the exterior space; see Section 4.2.
Berger et al. / State of the Art in Surface Reconstruction from Point Clouds
The third class of methods uses visibility to approximate
parity – the number of times a ray intersects a surface – in
order to approximate the interior and exterior, as discussed
in Section 4.3. These types of methods tend to make little assumptions on the shape class, and usually produce an implicit
function or a surface mesh in output.
4.1. Scanner Visibility
The most common method for using the visibility information
provided by a scanner is the merging of individual range
scans. This is the approach taken by [CL96], where a signed
distance function is incrementally built up over each range
scan. More specifically, each scan is first triangulated via
its image lattice structure. Then for a given ray originating
at the scanner head position, the signed distance field for
points along the ray are updated based on their distance to
the ray’s intersection point with the scan. Furthermore, as
discussed in Section 2.2.2, a confidence can be assigned to
each point in the range scan via line of sight information,
where [CL96] associates low confidence weights with high
grazing angles. This is particularly useful in combating noise
in the point cloud, since one can easily over smooth or under
smooth if no confidence values are associated with points;
see [CL96, Fig. 4].
Furthermore, one can perform space carving through line
of sight information, via marking regions of space observed
by the scanner as empty. The approach of [CL96] uses this
information to extract geometry between regions marked
empty and regions that are unseen, where the assumption is
that unseen regions are the interior of the shape. This is very
useful in resolving ambiguous topology in regions of missing
data; see [TOZ∗ 11, Fig.9].
For other forms of missing data, the approach of [CL96]
will typically preserve the hole as it does not enforce any
type of smoothness prior. It is possible to incorporate a minimal surface area regularization to encourage smoothness in
regions of missing data, while using line-of-sight as a datafitting term. Existing approaches solve such a formulation via
level-set models [Whi98] and graph cut optimization [LB07].
The method of [LPK09] seeks an interior and exterior labeling of tetrahedra from a Delaunay triangulation of the point
cloud, similar to [KSO04], but formulates it as a graph cut
problem using line of sight information. At each tetrahedron,
the method accumulates evidence for belonging to the exterior through line of sight of all range scans, hence assuming
outliers are randomly distributed, this method is robust to
such defects; see [LPK09, Fig. 13].
For scans that contain a high level of misalignment and
structured outliers, the method of [ZPB07] approaches range
scan merging by using the l1 norm for the data term, and the
minimization of the signed distance gradient magnitude as the
regularization term. This type of regularization, commonly
known as total variation denoising, allows the algorithm
(a)
(b)
(c)
Figure 8: The point cloud “hidden point removal” operator
from [KTB07] applied to an input (a) determines the subset
of visible points as viewed from a given viewpoint (b). Given
this labeling, a view-dependent on-the-fly reconstruction (c)
can be obtained by retaining the topology of well shaped
triangles from the convex hull of the spherical inversion.
to be robust to structured outliers and scan misalignment;
see [ZPB07, Fig. 4].
The method of [FG11] considers the case when range
scans have widely varying scales – the range scans have very
different sampling densities. In such cases, merging multiple
scans of a coarse scale with a single scan at a fine scale can
overly smooth out the fine-grained detail. [FG11] extends
[CL96] by constructing a hierarchical signed distance field.
This permits retaining the high resolution detail of fine-scale
scans, while capturing the more general scene present in
coarse-scale scans.
Scanner visibility was recently used in [UB13] in the reconstruction of thin objects, such as street signs. The method employs a point-based representation for reconstruction, where a
particle system is used to satisfy a data-fitting term, an energy
that encourages the vector formed by neighboring points to
lie on one another’s tangent plane, and a term that penalizes
neighboring points if their normals are in different directions.
Each input point’s target output point is restricted to be along
its line of sight, which helps to constrain point movement in
the particle system and greatly simplifies the optimization.
4.2. Exterior Visibility
It is possible to exploit visibility even in the absence of explicit information from the scanner. Given a chosen camera
position, point set visibility [KTB07] determines the portion
of the point cloud that is not self-occluded. First, a spherical
inversion of the point cloud with respect to the given query
point is computed. Then, visible points are simply identified
as those that lie on the convex hull of this set – see Figure 8.
While [MTSM10] extended this method to handle moderate levels of noise, the input point cloud must respect strict
sampling assumptions to produce satisfactory results.
Occlusion culling. The method of [CCLN10] builds upon
these ideas and reconstructs a watertight surface by carving
c Author version
Berger et al. / State of the Art in Surface Reconstruction from Point Clouds
Figure 9: The approach of [MDGD∗ 10] first computes a
robust unsigned distance function (left), and constructs an
interior/exterior labeling (middle), and associated confidence
(right) of the labeling. Note that low confidence is associated
with regions of missing data, such as the bottom of the scan.
the space occluded by the point cloud when observed by a
sufficiently large and randomly sampled set of directions.
Similarly to [KTB07], the input cloud has to satisfy certain
stringent sampling conditions, limiting its effectiveness with
undersampling and missing data. Conditions on sampling are
relaxed in [CLCL11] where inconsistencies are detected by
observing that if one point’s Voronoi pole [AB99] lies in the
exterior, the other Voronoi pole should be in the interior. If
both are occluded or visible via [KTB07], this indicates an
inconsistency. Unfortunately, since the method uses Voronoi
poles, which cannot always be robustly estimated in the presence of missing data, its applicability remains limited.
Cone carving. The method of [SSZCO10] hypothesizes that
each point in the cloud must have been observed from the
scanner head. It computes high-likelihood visibility cones
originating at each sample and takes the boundary of the
union of all cones as an approximation to the surface. This
method can be used to infer the geometry in large regions
of missing data for challenging scenarios, i.e. two thin, spatially close, and undersampled surface sheets – producing
topologically clean surfaces. The main disadvantage with the
approach is its lack of scalability, since it takes linear time to
construct a cone at each point, resulting in a total quadratic
running time.
4.3. Parity
An alternative way of using visibility is to define a measure
of parity. Assuming a closed surface, the parity for a given
ray (point and direction) is defined as the number of times
the ray intersects the surface – if an odd number of times, this
indicates the point lies in the interior, otherwise the point is
in the exterior. This general idea can be extended to a point
cloud, giving rise to a notion of uncertainty in whether or
not a point belongs in the exterior or interior. The approach
of [MDGD∗ 10] constructs parity on the point cloud through
the use of a robust unsigned distance function. Namely, they
compute a crust, or an offset surface, around the point cloud
via the unsigned distance and evaluate parity at a given point
by randomly shooting rays in different directions, where the
c Author version
Figure 10: The point cloud on the left was captured via
multi-view stereo – note the substantial structured outliers.
The center image is a zoomed-in portion of the point cloud,
centered on the object of interest. The method of [GCSA13]
(right) is highly robust to outliers, while still preserving the
details of the object, due to the use of parity and an unsigned
distance function which adapts to the noise.
crust is used to determine intersections with the surface. A
point will very likely be exterior or interior if the parity estimates are consistent over all directions. This uncertainty
estimate is used in constructing an implicit function, consisting of a data-fitting term and a smoothness term, such that
high smoothness constraints will be assigned to regions that
have high uncertainty (i.e. high disagreement in parity). Figure 9 shows the unsigned distance function for a challenging
point cloud, along with its sign estimate and confidence in
sign. This approach is highly robust to noise and outliers
due to the use of a robust unsigned distance function, which
does not require the estimation of normals. It is also robust to
missing data, and since its regularization is spatially-varying
according to the uncertainty in parity, it will not over smooth
the data where it exists. However, since smoothness is enforced via a Laplacian regularization, this could still result
in poor behavior in regions of missing data, giving the incorrect topology. The method of [SY12] addresses this by
performing space carving, guided by a parity estimate, to only
carve out space where there does not exist highly confident
interior regions. This can better retain topological features
such as tunnels, where smoothness priors may erroneously
over smooth and fill these regions in.
The method of [GCSA13] extends [MDGD∗ 10] by using
a robust unsigned distance function that is adaptive to the
noise level in the point cloud. The method produces a sign
estimate over a random collection of line segments in the
volume. To determine the parity for each line segment, rather
than using a crust as in [MDGD∗ 10], they look at all local
minima in the unsigned distance along the segment, flip the
function according to the local minima, and of all possible
flipped minima choose the one that is smoothest. Figure 10
demonstrates the method’s robustness to strong, structured
outliers.
Berger et al. / State of the Art in Surface Reconstruction from Point Clouds
model image
input point cloud
reconstructed surface
user-edited reconstruction
Figure 11: The high-level geometric regularizer provided by “arterial snakes” [LLZM10] is effective in reconstructing the
delicate geometry of the scanned tubular model affected by noise, outliers, reduced sampling density and missing data. The
sparsity of the model does not only allow the correction of these data imperfections, but also enables intuitive data/user-driven
reconstruction.
5. Volumetric smoothness
In order to handle challenging forms of missing data, a common way to regularize surface reconstruction is to enforce
that the local shape thickness of a surface (i.e. a measurement of its local volume) varies smoothly. For watertight
shapes, local thickness is measured by the radii of maximally
inscribed spheres of its medial axis transform. However, as
the medial axis is an alternative full representation of the
shape, determining the medial axis over which to perform
these measurements is an inherently difficult problem – as
difficult as the reconstruction problem itself.
Skeletal regularizers. The “rosa” method from [TZCO09]
addresses this issue by assuming that the medial axis of a
shape can be approximated by curves instead of surfaces, that
is, by a curve-skeleton. For organic geometry, a reconstruction
of the skeleton can be obtained even in the presence of missing data by exploiting the redundancies of local rotational
symmetry. Given a skeletal structure the geometry can be
reconstructed in regions of missing data by a three-step process: first, the distance from the cloud to the curve-skeleton is
cylindrically parameterized on the curve-skeleton; then, an inpainting of the distance function is performed on this domain;
finally, the inpainted point cloud can be processed by one of
the algorithms in Section 3. It is important to note that a cylindrical parameterization prior constrains the class of shapes
for which a reconstruction is possible to one having a starshaped cross section. While the skeleton extraction method
in [TZCO09] suffered the limitation of requiring oriented normals, subsequent research showed how it is possible to extract
skeletons directly from unstructured point clouds [CTO∗ 10]
and even in the presence of outliers [HWCO∗ 13]. To overcome other limitations, in [CTO∗ 10] a first attempt at removing the necessity for an explicit cylindrical parameterization
was made, while also allowing user-input to locally modify
the regularizer.
Deformable models. Another way to take advantage of volumetric smoothness is to grow a deformable model from within
the volume. The approach of [SLS∗ 06] slowly grows mul-
tiple surface fronts, guided by a signed distance field, from
within the interior volume out to the point cloud. The method
takes care to split and merge fronts in order to progressively
reconstruct the surface in growing levels of details. Noise and
missing data are handled by enforcing a smoothness term on
all fronts as they evolve. This conservative approach leads
to a better control and interpretation of the reconstructed
topology and can be extended to reconstruct a skeletal curve
directly from the point cloud [SLSK07].
To handle higher amounts of missing data, the approach
of [LLZM10] presents a deformable model for reconstructing
skeletal curves of man-made shapes composed of a collection
of tubular components such as metal rods and canes. The
arterial snakes model geometry is obtained by sweeping a
fixed-topology cross section through the input point cloud
starting from regions of high data fidelity. Their approach implicitly encodes volumetric regularization by constraining the
solution to one exhibiting a smoothly varying cross-section,
while also allowing high-level geometric constraints like planarity, contact tangency, and symmetry to further regularize
the reconstruction.
Organic skeletal geometry. Tubular components exhibiting
piecewise smooth radii variations are also suitable to model
organic and natural geometry like trees [RFL∗ 05, NFD07,
LYO∗ 10] or blood vessels [KQ04]. In tree reconstruction,
biological constraints are exploited to simplify the problem
and increase resiliency to data imperfections. Noise is introduced by foliage, as this is typically too small to be finely
resolved by the scanning process and a substantial amount
of self-occlusion is caused by dense branching structures.
Therefore, the focus of the reconstruction is tree branches,
where their approach is to first reconstruct the skeleton of the
captured tree. The skeleton is assumed to be a directed acyclic
graph rooted at the ground plane; limbs are typically piecewise smooth and their thickness almost everywhere smoothly
varying, where a pipe-model [RFL∗ 05] controlling thickness variations can be used at branching locations. While
a pair of orthogonal images have been shown sufficient to
c Author version
Berger et al. / State of the Art in Surface Reconstruction from Point Clouds
hallucinate the 3D information with the help of some user
interaction [NFD07], recent research has attempted to model
the tree structure directly from 3D point clouds [LYO∗ 10].
Generalized volumetric smoothness. Several methods exist
for imposing volumetric smoothness on the medial axis of
the shape, supporting a much broader class of shapes. The
method of [BS12] approaches reconstruction by segmenting
the point cloud into volumetric regions and in each region taking the union of balls to obtain a coarse surface representation.
The union of these regions then serves as an initial surface for
the method of [SLS∗ 06], to recover fine details. Key to their
segmentation is a distance measure defined directly on the
point cloud that robustly measures the likelihood of a medial
ball being generated by any pair of points. This method is robust to noise and missing data, particularly when there exists
nearby surface sheets, but can fail in regions where parts of
the surface corresponding to medial sheets are missing.
Figure 12: (left) CAD models are often obtained by constructive solid geometry as a composition of simple primitives: planes, spheres, cones, etc. (middle) Randomized
search [SWK07] can be used to detect such primitives in the
point cloud data even in the presence of noise, outliers and
missing data. (right) The primitives can then be extrapolated
to obtain a watertight surface from incomplete data [SDK09].
The detection of simpler geometric structures in a point cloud
has shown to be particularly beneficial for surface reconstruction. Knowledge of a surface that can be described as the
composition of geometric primitives can be extremely helpful
for denoising and filling in missing data. Not all shapes adhere to this prior, but typically CAD and architectural models
can be described in this manner.
Primitive consolidation. The work of [JKS08] takes a set
of detected plane primitives and performs reconstruction by
aligning and merging the boundaries of adjacent primitives.
More specifically, the boundaries of the plane primitives are
extracted and an optimal configuration of boundaries is found
by imposing a data-fitting term to the original boundary as
well as a term that favors the snapping of boundary points and
corner points of neighboring planes. Explicitly using corners
prevents the boundary from smoothing itself out. The advancing front method of [SFS05] is used to extract a surface mesh.
This method can reconstruct CAD and architectural models
alike, producing a surface mesh that retains the detected primitive structures. However, the method requires that adjacent
primitive boundaries should be geometrically close to each
other, which may not be satisfied if primitive detection is
noisy, or if missing data is present. The method of [SDK09]
resolves this by explicitly extrapolating shape primitives (of
all kinds) and forming the resulting output as the intersection
of the extrapolated primitives. This extrapolation of primitives is formulated as a graph cut problem, where in addition
to a standard minimal surface area term, a data fitting term
is used that ensures the surface normal at a given point (the
edge in the graph) is aligned with all intersecting primitives
at that point. This does not constrain the primitives in a local
manner: primitives whose boundaries are far away can eventually meet up and intersect with this method, as illustrated
in Figure 12.
Detecting primitives. The method of [SWK07] is an effective method for finding geometric primitives in shapes. It uses
RANSAC to robustly find planes, spheres, cylinders, cones,
and torii, through an efficient means of sampling points for
fitting and evaluating scores, both based on locality sensitive
methods. Importantly, this method produces primitives that
partially match the point cloud – the collection of these shapes
can then be used for reconstruction. We note that although
this method can detect a small set of easily parameterizable
shapes, efficient pose detection methods for arbitrary shapes
can also be used [DUNI10].
Augmenting primitive information. Although the method
of [SDK09] can robustly handle missing data, it can be sensitive to noisy primitives which may fail to define a coherent
model when extrapolated. The work of [CLP10] instead uses
line of sight information to help penalize poorly extrapolated primitives. Namely, this work takes the set of primitives
as well as an additional set of primitives formed near the
boundaries of the input primitives and constructs a cell complex reflecting the extrapolation of the primitives. An energy,
similarly solved via graph cuts is then formed, where the
data-fitting term uses line of sight information to penalize
Among methods employing volumetric regularization
[TOZ∗ 11] is the most general. The surface encoding the
solution is iteratively evolved by level-sets toward the data
considering both visibility and surface smoothness, similarly
to [Whi98], while simultaneously enforcing a volumetric
prior based on the medial axis of the evolving interface. This
technique can prevent the formation of unnecessary holes
in thin surfaces due to under-sampling, as the formation of
a topological feature would correspond to a quickly vanishing medial radii. Furthermore, since the medial axis encodes
local reflectional symmetry, this allows for information to
be effectively propagated throughout the surface permitting
the reconstruction of challenging data like the geometry of
the highly concave areas in a vase – see [TOZ∗ 11, Fig.5].
While highly general, the instability of the medial axis to
surface-perturbations and the complexity of its computation
limit the applicability of the method.
6. Geometric Primitives
c Author version
Berger et al. / State of the Art in Surface Reconstruction from Point Clouds
facets in the complex lying in regions marked empty via space
carving. The method of [RKMP13] uses the point cloud, line
of sight, and edge features in corresponding RGB images to
infer constraints for finding the boundary of each input shape
primitive, consequently producing the reconstructed surface.
Volumetric primitives. In the case of indoor scene reconstruction, an alternative to surface primitives is to employ
volumetric primitives to model the interior space. In [XF12]
the volume is modeled by fitting simple cuboids to the empty
space defined by the boundaries of the scan data. A set of 2D
constructive solid geometry (CSG) vertical slices are built by
incrementally adding and removing candidate rectangles that
best model the interior – a function of line of sight information. A similar process is used to stack up these slices to build
a set of volumetric primitives, producing a 3D CSG model
that composes the interior. The method of [OLA13] starts
from a volume decomposition of the space, and labels volumetric cells as interior and exterior through a parity-driven
cost function solved via graph cuts, where parity is measured
through ray intersections of extracted horizontal and vertical
structures. Such volumetric methods guarantee a watertight
output, and are highly robust to missing data, since only a partial sampling of the volume boundary is needed for a quality
fit.
Hybrid methods. A limitation of primitive-based methods
is that they do not degrade gracefully if certain portions of
the shape are poorly explained by a primitive. The method
of [LA13] resolves this by introducing a hybrid approach
to reconstruction: shape primitives are used to resample the
point cloud and enforce structural constraints in the output,
such as sharp features between adjacent primitives and corners, while a visibility-driven prior is employed in regions
where a primitive fit is not found. A similar approach was
proposed in [vKvLV13], where planar polygons of sufficient
fitting quality are extracted and a conforming, constrained
Delaunay triangulation is constructed on the polygons and
the remaining points so that the polygons are preserved in
the triangulation. A visibility-driven graph cut problem is
then solved, similarly to [LPK09], such that the extracted
polygon primitives are retained, while the rest of the points
rely on line of sight information for reconstruction. These
types of methods greatly generalize the class of shapes in
which primitive-based reconstruction may be performed.
7. Global Regularities
Certain classes of shapes, namely CAD, man-made, urban,
and architectural shapes, possess a certain level of global
regularity. Regularity in a shape can take many forms: a vase
described by a surface of revolution, a building composed of
facade elements, or a mechanical part consisting of recurring
orientation relationships between sub-parts. All of these are
examples of the following more general properties: symmetry,
repetition, and canonical relationships. Commonly associated with high-level shape analysis [MWZ∗ 13], these priors
Figure 13: From the scan on the left, the method
of [PMW∗ 08] is able to discover local similarity transformations which relate individual elements. This can be used to
resample the point cloud, as well as extrapolate the scan into
missing regions, shown on the right.
have also shown to be of great use in handling severe defects
in a point cloud.
7.1. Symmetry
Symmetry is a well-studied problem in shape analysis [MPWC13]. Symmetry detection is focused on finding
either global or local transformations on the shape that maps
the entire shape, or a subset of the shape, onto itself. Finding such transformations can be extremely useful for surface
reconstruction in handling noise and missing data.
Global similarity. The method of [LA11] applies this to the
case of missing data for single-scan completion by finding
small surface patches of the scan that exhibit either bilateral,
rotational, or surface-of-revolution symmetry, and then applying the detected transformation to the rest of the scan to infer
the missing data. This can produce a complete model from
a single view and does not assume a specific type of shape
class. However, it assumes that a shape can be well described
by the application of a single simple transformation, which
does not always hold.
Local similarity. Rather than imposing global relationships,
in [PMW∗ 08] the authors focus on finding repeating elements
(small subsets of the point cloud) that can be mapped onto
one another by local similarity transformations. They show
that the repetition of elements in a point cloud manifests
as a lattice structure in a suitable transformation space. In
particular, partial matches become prominent in this transformation space, hence repeating elements of varying levels of
missing data can be robustly detected and used to reconstruct
incomplete regions, see Figure 13. The method of [LCDF10]
finds symmetries in incomplete point clouds by constructing an affinity matrix that measures how symmetric all pairs
of points are. The key insight made by [LCDF10] is that
this matrix should be block-diagonal for many types of symmetries – i.e. rotational, bilateral, intrinsic. By considering
powers of this matrix, the authors demonstrate how incomplete matches become more pronounced, allowing for a wide
range of detected symmetries in challenging point clouds
containing noise, outliers, and missing data. These simpler
forms of symmetry can be generalized to a notion of subspace
symmetries [BWM∗ 11], where a symmetry group is defined
c Author version
Berger et al. / State of the Art in Surface Reconstruction from Point Clouds
by a set of local transformations as well as a low-dimensional
shape space, in order to handle more general types of shapes.
7.2. Structural Repetition
In certain cases it is difficult to find repeating elements in a
point cloud through symmetry transformations. Instead, directly seeking repeating elements in a transformation-free
manner can provide us with more flexibility in the reconstruction process.
The method of [ZSW∗ 10] utilizes this observation for reconstruction of building facades in noisy terrestrial LiDAR
scans, where occlusions from vegetation or other objects result in significant missing data. For a given type of facade
element, each elements’ planes are detected via [SWK07]
and the individual elements are registered at a per-plane level.
Once registered, denoising is performed across all elements
via the individually registered planes and the consolidated
facade element is projected back onto each instance for reconstruction. The mutual use of information across all elements
allows one to robustly remove noise and fill in missing data.
A drawback to the approach of [ZSW∗ 10] is the strict
requirement of user interaction. This limitation was addressed
in the work of [SHFH11] by adaptively partitioning facades.
The approach of [WS12] takes the consolidated point cloud
of [ZSW∗ 10] and segments it into depth layers and uses a
grammar definition to individually segment each depth layer
into facades via the optimal sequence of grammar derivations.
Facades may be appropriately extruded at each depth layer
to obtain a polygonal representation of the building, at the
possible expense of detail in the geometry due to the lack of
expression in the shape grammar.
The method of [FS12] tackles occluded facade scans by
analyzing vertical scanlines consisting of columns of points
along major facade planes to detect periodic features. The
Discrete Fourier Transform is computed for each scanline
and used to extract the period of repetitive structures. Holes
are filled by extending the periodic features into occluded
regions.
Another means of detecting regularity in incomplete scans
is to find regularity in associated RGB imagery, and propagate this information back to the 3D scan to perform reconstruction. The approach of [LZS∗ 11] achieves this by
decomposing the RGB image into depth layers via the 3D
scan, and upon detecting symmetries with respect to each
layer via [WFP10], consolidates all element instances to robustly denoise and fill in missing data across the instances.
7.3. Canonical Relationships
Another useful prior on global regularities is the canonical
intra-relationship between parts of a scene, or parts of a shape.
Such relationships can be parallel or coplanar parts, recurring
orthogonality between planes, concentric or co-axial parts,
c Author version
input scan
ransac
globfit
Figure 14: From a set of misaligned scans shown on the
left, the primitives extracted via [SWK07] (middle) retain the
misalignment. Globfit [LWC∗ 11] (right) is able to correct
misalignment by enforcing consistent canonical relationships
across primitives.
and regularity in orientation. This often arises in CAD models
due to fabrication restrictions and budget considerations, as
well as urban environments due to functional constraints.
Manhattan constraints. Perhaps the simplest form of a
canonical relationship is the Manhattan-world (MW) assumption: all planar primitives in a scene belong to one
of three mutually orthogonal planes. This can simplify
facade reconstruction, as in the aforementioned methods
of [ZSW∗ 10, SHFH11, WS12, LZS∗ 11]. In [VAB12], MW
is used for building reconstruction by first classifying points
by shape type – wall, edge, convex corner, or concave corner
– and clustering points of a similar type. After constructing
MW-aligned bounding boxes on all clusters, volume regions
are found via parity, analogous to [MDGD∗ 10], where interior regions of consistent parity are considered to belong to
the building’s volume. As edges and corners are detected via
relationships between walls, this method is robust to missing
data, but may be sensitive to noise for adjacent wall configurations.
Consolidating relationships. The method of [LWC∗ 11] reconstructs CAD shapes consisting of a much richer variety of
canonical relationships compared to MW. Namely, starting
from an initial set of detected primitives [SWK07], parallel,
orthogonal, angle-equality, and distance-equality relationships are individually detected and carefully selected so as
to not cause any relationship conflicts. By enforcing these
relationships, structured noise such as scan misalignment can
be effectively handled – see Figure 14.
Canonical building relationships. The work of [LWC∗ 11]
was extended to the case of reconstruction of buildings from
2.5D scans in [ZN12]. The basic observation in this approach
is that there exists three fundamental type of relationships in
buildings: roof-roof relationships that consist of orientation
and placement equalities, roof-roof boundary relationships
that consist of parallelism and orthogonality relationships,
and boundary-boundary relationships that consist of height
and position equality. Upon finding the relationships via clustering (i.e., clustering similar angles, equality, etc..), they are
used to inform the primitive fitting method so that the prim-
Berger et al. / State of the Art in Surface Reconstruction from Point Clouds
Figure 15: From the incomplete and cluttered scan on the left, the approach of [NXS12] first oversegments the point cloud
(mid-left), then iteratively merges segments which agree on class labels (mid-right), and finally deforms a set of detected models
to best resemble their segmented objects (right).
itives simultaneously fit to the data and to the relationships.
Analogous to [LWC∗ 11], this allows for robust reconstruction
from building scans containing strong structured noise.
8. Data-driven priors
The previously discussed priors may not always be appropriate and in practice, certain shapes may simply fail to adhere
to these priors. A more flexible method of specifying a prior
is through a data-driven means: using a collection of known
shapes to help perform reconstruction. This can allow for the
reconstruction of individual objects and more generally, the
reconstruction of scenes composed of a certain type of environment and multiple types of objects. Fine-grained scene
reconstruction is extremely challenging due to the large missing data often associated with scans, such as those derived
from the Microsoft Kinect. Hence, if reconstructing the geometry is infeasible, these methods instead seek the most
similar object in a database, and if necessary, its deformation
in order to fit the input data.
Data driven inpainting. An earlier method of reconstructing individual objects from a database was proposed
in [PMG∗ 05], where an incomplete input point cloud is
matched against a database of complete shapes and the most
relevant shape is then rigidly deformed into the point cloud to
recover missing data. This allows for a watertight reconstruction, but for the algorithm to be effective, strong similarity
between the input data and the best-matching object is necessary. This limitation was addressed by [GSH∗ 07] by using
local shape priors, where a collection of overlapping point
cloud patches are matched against a local shape prior database
and the retrieved priors are used to reconstruct the surface.
A major drawback of these approaches is the assumption
that the point cloud is sufficiently dense, so that matching
the cloud against a set of complete shapes is meaningful.
However, for the acquisition of scenes, in particular indoor
environments, the scanned objects may be too incomplete to
permit this due to occlusion, limited observation directions,
or the geometry is too fine-grained with respect to scanner
resolution. Furthermore, for these methods to be applicable
to scene reconstruction, objects need to be individually segmented.
Reconstruction by rigid retrieval. The method
of [SXZ∗ 12] approaches this problem by first seman-
tically segmenting the point cloud and then finding a
complete model to replace each segment. More specifically,
given a set of scans and RGB images, a conditional random
field class labeling problem is formulated, balancing two
objectives: a data fitting term based on a database of
class-labeled objects and a smoothness term favoring local
consistency in appearance and geometry. A training set
of point clouds is built, where for each object scans are
constructed via a virtual camera over a range of orientations
and distances from the object. This allows the model to be
robust with regard to missing data, as the input should map
to one of these poses. A random forest classification is built
over this training set allowing for the closest complete object
to an incomplete scan to be retrieved. Although retrieving
a rigid template from a database can effectively handle
incomplete data, only rigid transformations with uniform
scaling are considered to best align each matched object.
Reconstruction by non-rigid retrieval. A natural extension
of reconstruction by retrieval is to consider non-rigid transformations of the template geometry to the input data. This
is addressed in [NXS12] where upon finding a certain semantic class for a segmented object in the point cloud, every
model is non-rigidly deformed via localized scale deformations. The best match is identified as the model with the
smallest registration residual. This method approaches classification differently by building a semantically-labeled segmentation through incremental selection of oversegmented
surface patches. A patch is chosen if the resulting merged
object has high confidence in its label. This is particularly effective in noisy, outlier-ridden highly cluttered environments
– see Figure 15. The authors of [KMYG12] extend these ideas
by noting that in indoor environments it is common to have
the same object in multiple poses. Their technique incorporates a deformation model directly into the segmentation and
classification problem, rather than as a post-processing step.
A deformation model is learned over multiple incomplete
scans for a given object, allowing the object to be identified
by incomplete observations of its parts. Given an input scan, it
is first over-segmented and then iteratively merged into parts,
where parts are matched against learned local deformation
modes of a model. Part relationships are then used to verify
the global quality of a match. Compared to [NXS12,SXZ∗ 12],
such deformation models allow one to reconstruct a broader
c Author version
Berger et al. / State of the Art in Surface Reconstruction from Point Clouds
set of objects, and lessens the need for a potentially large
database for recognition.
Reconstruction by part composition. A disadvantage of the
above approaches is that the granularity of the reconstruction
is at the level of a whole model, that is, combining parts from
different models is not possible. The approach of [SFCH12]
overcomes this by combining individual parts from different
objects to best compose the input scan. Namely, starting
from a database of segmented models, 3D data is combined
with RGB imagery to find candidate parts from the database
matching the input. In particular, the use of RGB data can
help find parts that are completely missing in the 3D scan.
The best combination of candidates that closely match the
geometry, while consisting of a small intersection with each
other, composes the final model.
Model-based SFM. The method of [BCLS13] performs reconstruction at the category level (i.e. car, fruit), by learning
a mean shape prior along with corresponding feature points
on the shape from a given set of object instances acquired
through structure from motion (SFM). Then for a given point
cloud and associated imagery, its points are matched to the
learned feature points and used to deform the mean shape to
the given point cloud. Such an approach allows for the reconstruction of coarse details via the learned mean shape, while
preserving fine details present in the point cloud. Such details
are likely to be lost by the previously described approaches.
9. User-Driven Methods
Incorporating the user in the process of surface reconstruction
has shown to be extremely beneficial in dealing with challenging point clouds. The key distinguishing factors between
user-driven methods are the level of intuition and ease of
use an interface provides and the extent at which the user
interaction is coupled with the underlying reconstruction
technique. User-driven techniques were first introduced as a
means for generating required information as inputs to the
reconstruction algorithm, where recent and more encompassing approaches tightly integrate the user interaction with the
core reconstruction algorithm. Below we group user-driven
techniques by the type of information that is solicited from
the user.
Feature classification cues. The methods of [FCOS05b]
and [GG07] showcase how user input can be used to provide valuable information for reconstruction. In both methods
user input is used to augment the point cloud with tags classifying different surface attributes such as regions of surface
smoothness [FCOS05b] and sharp features [GG07]. A simple
brush tool that acts directly on the point cloud enables the
user to identify regions of interest. The tags are incorporated
with moving least squares [ABCO∗ 03] to better handle sharp
feature regions that are difficult to detect and reconstruct automatically. Another example of user interaction is the method
of [LZS∗ 11], which requires the user to correlate 2D photos
c Author version
(a)
(b)
(c)
Figure 16: Given an input point cloud, simple planar primitives identified by RANSAC (a) may result in coarse and
incomplete geometry (b). By exploiting the user’s high-level
knowledge while remaining faithful to the input data (b) a
constrained optimization allows to recover a high-quality
model (c).
and LiDAR scans by identifying corresponding 2D rectangle
regions in both data sources.
Topology cues. The method of [SLS∗ 07] demonstrates how
user information can be more tightly integrated to guide reconstruction in an interactive fashion. Specifically, the approach obtains watertight and topologically correct reconstructions through the automatic detection of topologically
weak regions in a given reconstruction. These low-confidence
regions are then presented to the user to be resolved, via scribbles on a 2D tablet, which translate to interior and exterior
constraints, or potentially no constraints if the user deems
the region valid. The reconstruction is then updated, and the
process repeats through further user edits.
Structural repetition cues. The ability to incorporate user
input in reconstruction allows one to forego traditional watertight and topological requirements. User interactivity and
automatic reconstruction can be tightly coupled in an integrated interactive environment and is especially useful for
the reconstruction of large-scale data. These techniques are
best suited when the sampled objects exhibit high repetition
and can be adequately represented in terms of a collection of
simple geometric primitives. In [NSZ∗ 10] the authors present
a technique to rapidly reconstruct architectural models, such
as those acquired from the scanning of large urban environments. The key idea of the approach is to enable the user to
define and manipulate simple geometric building blocks in
the form of axis-aligned rectangular cuboids named smartboxes. The user sequentially places the smartboxes into the
scene, where contextual regularities and symmetries between
boxes are automatically captured and used to expedite the
fitting process. The final placement of the user manipulated
primitives is determined through an interactive optimization
procedure that automatically adjusts the location, orientation,
and sizes of the box primitives by considering how the cuboid
fits the data and its relationship in context with previously
placed boxes. The method also allows for the grouping and
Berger et al. / State of the Art in Surface Reconstruction from Point Clouds
manipulation of multiple boxes for rapidly copying and fitting
recurring structures.
Primitive relationship cues. This class of methods approaches the reconstruction of models that can be assembled
by simple polygonal primitives. The role of the user is to
provide hints in how to improve the connectivity of the reconstructed model. The approach of [CC08] is focused on
building reconstruction from incomplete scans. They first
estimate planar polygons as well as their boundaries from
the scans. Acknowledging that this estimation will be imperfect under missing data, the authors then allow for the
user to identify boundary lines that bound an absent polygon,
as well as specify multiple polygons that intersect to yield
absent edges or corners. This information is used to infer a
watertight polygonal model of the building. To support more
complicated polygonal relationships the method of [ASF∗ 13]
reconstructs a closed polygonal model by snapping each polygon into alignment with neighboring primitives by solving
a combined optimization problem involving both local and
global spatial constraints – see Figure 16. The interactive
component provides the user a set of modeling operations
comprised of simple edits restricted to an automatically determined plane, hence fixing the view for the user. A polygon
edit mode allows the user to refine existing polygons by editing their boundaries and merging multiple disconnected polygons. A polygon sketching mode allows the user to provide
new polygons for regions where automatic planar detection
failed due to insufficient points or incorrect estimation due
to noise or outliers in the data. For both modes, the user has
to only provide coarse edits, as the automatic snapping optimization is used to align polygon boundaries based on both
local and global relations between primitives.
10. Evaluation of Surface Reconstruction
Given the wide diversity in reconstruction methods, the manner in which one reconstruction is evaluated compared to
another may differ. In this section we look at different evaluation criteria used in the surface reconstruction literature.
Geometric Accuracy. Perhaps the most common method of
evaluation is to directly compare the geometry of the reconstruction output to the ground truth surface from which the
scan was obtained. Hausdorff distance, mean distance, as well
as measuring error in normals are common geometric error
measures in this scenario. However, it is often challenging to
obtain the notion of a ground truth surface from a physical
shape. Hence, computational representations of shapes are
typically used as the ground truth [Kaz05, MPS08], where
synthetic scanning of the digital representation can be used in
place of an actual scanner [BLN∗ 13]. In some cases, a direct
comparison to ground truth data is insufficient when targeting
reconstruction under an error tolerance or comprising several
levels of details (LODs). This suggests evaluating instead the
complexity-distortion tradeoff, or the capability to generate
LODs that are both controllable via intuitive parameters and
meaningful for the targeted applications. Such evaluation criteria consist of the coherence of LODs across the scene, the
ability to incrementally refine the geometry, and the level of
abstraction provided by the LODs, analogous to [MZL∗ 09].
Topological Accuracy. Another important evaluation criteria
is the recovery of higher-level information of the shape and in
particular, its topology. Certain methods are concerned with
reconstructing a shape with the correct genus [SLS∗ 07], while
other methods that focus on recovering a skeletal representation of the shape are more concerned with the topology of the
underlying skeletal structure – recovering important branches
and junctions in the skeleton. Such a topological structure
is of particular importance for structural shape editing applications [LLZM10] and nonrigid registration [ZST∗ 10].
However, we note that most skeleton-based methods are often concerned with qualitative evaluation, hence it can be
difficult to compare different skeleton extraction methods.
Structure Recovery. Beyond geometry and topology it is
also sometimes desirable to recover the structure during reconstruction. Beyond the simple notion of scene decomposition, the term structure has a broad meaning, ranging from the
dimension of geometric entities (manifolds, stratified manifolds, non-manifold configurations) to adjacency relationships through canonical geometric relationships (parallelism,
co-planarity, orthogonality, concentricity, co-axiality) and
regularities (repetitions, symmetries). In addition, controlling
the structure encompasses recovery, preservation, and reinforcement. Structure is especially relevant when dealing with
large-scale scenes, not just individual objects, where scenes
are composed of a collection of objects which may have
structural interrelationships. Structure as well as global regularities are also a means to improve robustness and resilience
to missing data and go beyond reconstruction to consolidation
and abstraction.
Shape Recognition. For data-driven methods that deal
with substantial missing data, recovering precise geometry is often impractical. Instead, recognizing whole shape
classes [SXZ∗ 12] or parts of shapes [SFCH12] from an incomplete scan are common methods of evaluating reconstruction quality. Similar types of recognition methods can be used
in evaluating the detection of geometric primitives.
Ease of Use An important evaluation criterion is the ease of
use of a surface reconstruction algorithm. For methods that
are automatic, this can amount to how sensitive a method
is to its set of parameters across a large range of inputs.
For user-driven methods, this translates to the practicality
of the user interface. This can be evaluated through user
studies [ASF∗ 13] that measure the time it takes to reconstruct
a model. The studies perform these evaluations on both users
who are familiar with computer graphics and 3D modeling as
well as those that are novices.
Reproducibility An important consideration in evaluating
the quality of a reconstruction method is its level of reproducibility. Perhaps the simplest means of determining reproc Author version
Berger et al. / State of the Art in Surface Reconstruction from Point Clouds
ducibility is whether or not certain methods are made publicly
available or have been implemented by a third party, as this
can be an important indicator of implementation complexity
and algorithm robustness. Some surface reconstruction implementations are publicly available, which provides a sense of
methods that have a high level of reproducibility. For instance,
Poisson surface reconstruction [KBH06] is a widely used surface reconstruction method as the code is highly stable and
reliable. The issue of reproducibility and provenance is well
studied in other areas, including visualization and scientific
workflows [SFC07,FKSS08], and it has been shown to be useful for studying 3D model construction [DKP11]. Given the
increasing complexity of reconstruction algorithms, the issue
of reproducibility is likely to be of increasing importance.
11. Conclusions
The area of surface reconstruction has grown from methods
that handle limited defects in point clouds while producing
detailed reconstructions, to methods that handle substantial
artifacts and produce high-level surface representations. Our
survey provides insight into this wide array of methods, highlighting strengths and limitations that currently exist in the
field. In doing so, our survey should also point the way towards future work across all of the different priors – making
potential connections across input assumptions, point cloud
properties, and shape classes that have not been previously
considered.
Hints and solvers. In our survey of recent work we observed
how the surface reconstruction problem is tackled through
either increasingly sophisticated solvers or richer reconstruction hints that make the problem easier to solve. For example,
the availability of oriented normals requires only a linear
solve through the Poisson reconstruction approach [KH13],
while unoriented normals require solving for a generalized
eigenvalue problem [ACSTD07]. Other hints such as generalized parity requires two linear solves: a first solve to
consolidate the local hints as sign guesses, and a second solve
to recover a signed implicit function [GCSA13].
Innovations in acquisition. As 3D acquisition methods continue to increase in variety and popularity, surface reconstruction will continue to be an important component in acquiring
real-world shapes. To provide some historical context, consider the rise in accessibility of the optical laser-based triangulation scanner: since such a device provides line of sight
information, this resulted in a whole category of visibility
priors (Section 4) such as VRIP [CL96]. A more recent example is the Microsoft Kinect: the real-time capture of depth
and RGB imagery has resulted in a new slate of techniques
for reconstructing indoor scenes through data-driven means
(Section 8) such as [SXZ∗ 12, NXS12, KMYG12], since objects in indoor environments tend to satisfy such priors. As
novel acquisition sensors and modalities are developed, it
is likely that surface reconstruction will become even more
specialized in order to handle the nuances of the acquisition
c Author version
type. In this setting our survey should prove useful in the
development of novel priors that need to handle such new
acquisition methods.
Acquisition ubiquity. Beyond the increasing variety of sensors, we are also witnessing a rapid evolution of the acquisition paradigms. The acquisition of our physical world can
now be complemented by exploiting the massive data sets
shared online, referred to as community data. We also predict
a future where geometric data are acquired through disseminated sensors, yielding disseminated data. This evolution
translates into a paradox: despite expectations that technological advances should improve quality, these data are hampered
with high variability and unprecedented amount and variety
of defects. In addition, we are observing a trend brought on by
the speed of technological progress: while many practitioners
use high-end acquisition systems, an increasing number of
them turn to consumer-level acquisition devices, willing to
replace an accurate albeit expensive acquisition by a series
of low-cost acquisitions – see recent work on 3D acquisition from mobile phones [TKM∗ 13]. These new acquisition
paradigms translate into a lower control over the acquisition
process, which must be compensated by an increased robustness of the algorithms and structural or physical a priori
knowledge. Recent works in hair reconstruction [LLR13] and
foliage reconstruction [BNB13] demonstrate the challenges
brought on by acquisition in uncontrolled environments.
Big data and online algorithms. Last, the scale of acquired
data is also quickly growing: we no longer deal exclusively
with individual shapes, but with entire scenes, possibly at the
scale of entire cities with many objects defined as structured
shapes. Recovering the structure of such large scale scenes is
a stimulating scientific challenge. We also envision a future
where the common on-disk paradigm must be replaced by
online algorithms that perform reconstruction during acquisition. Recent works such as Kinect Fusion [NDI∗ 11] and
extensions [CBI13, NZIS13] demonstrate the practicality of
building such online systems. There are applications such
as aero-reconstruction for disaster management where tight
timing restrictions make an online reconstruction approach
indispensable. In particular, we foresee a need to extend the
surveyed priors into the online setting, in order to support
such challenging problems in surface reconstruction.
Acknowledgments. Pierre Alliez is supported by an ERC
Starting Grant “Robust Geometry Processing” (257474).
Claudio Silva was partially supported by the National Science
Foundation grant MRI-1229185, an IBM Faculty Award, and
a grant from the Gordon and Betty Moore and Alfred P. Sloan
Foundations. Joshua Levine is supported by the National Science Foundation award IIS-1314757.
Author Bios
Matthew Berger is a researcher at the Air Force Research
Laboratory, Information Directorate. He received his PhD degree in Computing in 2013 from the University of Utah. His
Berger et al. / State of the Art in Surface Reconstruction from Point Clouds
research interests include various geometry processing topics
such as surface reconstruction, meshing, and shape analysis, as well as computer vision topics such as 3D geometry
acquisition, motion segmentation, and object recognition.
Andrea Tagliasacchi is a postdoctoral scholar in the Computer Graphics and Geometry Laboratory at the Ecole Polytechnique Federale de Lausanne (EPFL). He received his
MSc from Politecnico di Milano and a PhD from Simon
Fraser University (SFU) under the joint supervision of Prof.
Richard (Hao) Zhang and Prof. Daniel Cohen-Or. His research interests include computer graphics, computer vision
and machine learning with a focus in geometry processing.
Lee M. Seversky is a research scientist at the Air Force Research Laboratory, Information Directorate. He received his
PhD degree in Computer Science from Binghamton University where he investigated robust methods for processing and
analyzing incomplete point cloud data. He is currently a principal researcher with the Air Force pursuing new problems
in the areas of large-scale geometry processing, computer
graphics, and computer vision.
Pierre Alliez is Senior Researcher and team leader at Inria Sophia-Antipolis - Mediterranée. He has authored many
scientific publications and several book chapters on topics
commonly referred to as geometry processing: mesh compression, surface reconstruction, mesh generation, surface
remeshing and mesh parameterization. He is an associate editor of the Computational Geometry Algorithms Library, ACM
Transactions on Graphics, Graphical Models and Computer
Aided Geometric Design. He received in 2005 the EUROGRAPHICS young researcher award. He was awarded in
2011 a Starting Grant from the European Research Council
on Robust Digital Geometry Processing.
Joshua A. Levine is an assistant professor in the Visual Computing division of the School of Computing at Clemson University. He received his PhD from The Ohio State University and completed a postdoc at the University of Utah’s
SCI Institute. His research interests include geometric modeling, visualization, mesh generation, topological analysis,
surface reconstruction, vector fields, medical imaging, computer graphics, and computational geometry.
Andrei Sharf is a faculty at the computer sciene department
at Ben-Gurion University, Israel. Previosly, he has been a
Visiting Associate Professor at the Shenzhen Institute of Advanced Technology(SIAT) Chinese Academy of Sciences and
a Postdoctoral researcher at the School of Computer Science in UC-Davis U.S. His research interests are in computer
graphics, including surface reconstruction and interaction
techniques, geometry processing, urban modeling and motion
analysis. He received his PhD on “Surface Reconstruction
Techniques for Imperfect Raw Data” from Tel-Aviv University in 2007 under the supervision of Daniel Cohen-Or and
Ariel Shamir.
Claudio T. Silva is Professor in the School of Engineering,
and Head of Disciplines at the Center for Urban Science
and Progress, New York University. He is also affiliated
with Courant and the Center for Data Science. Claudio
has been active in geometry processing for almost 20
years, with a special emphasis on surface reconstruction
and point-set surfaces. He has served on more than 100
program committees, and he is currently on the editorial
board of the ACM Transactions on Spatial Algorithms and
Systems (TSAS), Computer Graphics Forum, Computing
in Science and Engineering, The Visual Computer, and
Graphical Models. He received four IBM Faculty Awards
and ten best paper awards. He is a Fellow of the IEEE.
References
[AA04] A LEXA M., A DAMSON A.: On normals and projection
operators for surfaces defined by point sets. In Proc. of the EG
conf. on Point-Based Graphics (2004). 8
[AA06] A DAMSON A., A LEXA M.: Anisotropic point set surfaces.
In Proc. AFRIGRAPH (2006). 11
[AB99] A MENTA N., B ERN M.: Surface reconstruction by
voronoi filtering. Discrete & Computational Geometry (1999). 13
[ABCO∗ 03] A LEXA M., B EHR J., C OHEN -O R D., F LEISHMAN
S., L EVIN D., S ILVA C.: Computing and rendering point set
surfaces. Trans. on Visualization and Computer Graphics (2003).
3, 5, 7, 8, 19
[ACK13] ATTENE M., C AMPEN M., KOBBELT L.: Polygon mesh
repairing: An application perspective. ACM Computing Surveys
(CSUR) (2013). 2
[ACSTD07] A LLIEZ P., C OHEN -S TEINER D., T ONG Y., D ES BRUN M.: Voronoi-based variational reconstruction of unoriented
point sets. In Computer Graphics Forum (Proc. of the Symposium
on Geometry Processing) (2007). 3, 4, 5, 7, 10, 21
[AK04] A MENTA N., K IL Y. J.: Defining point-set surfaces. ACM
Trans. on Graphics (2004). 5, 8
[ASF∗ 13] A RIKAN M., S CHWÄRZLER M., F LÖRY S., W IMMER
M., M AIERHOFER S.: O-snap: Optimization-based snapping for
modeling architecture. ACM Trans. Graph. (Proc. SIGGRAPH)
(2013). 3, 20
[ASGCO10] AVRON H., S HARF A., G REIF C., C OHEN -O R D.:
ℓ1 -sparse reconstruction of sharp point set surfaces. ACM Trans.
on Graphics (2010). 5, 7, 11
[BCLS13] BAO S. Y., C HANDRAKER M., L IN Y., S AVARESE
S.: Dense object reconstruction with semantic priors. In CVPR
(2013). 19
[BLN∗ 13] B ERGER M., L EVINE J. A., N ONATO L. G., TAUBIN
G., S ILVA C. T.: A benchmark for surface reconstruction. ACM
Trans. on Graphics (2013). 20
[BNB13] B RADLEY D., N OWROUZEZAHRAI D., B EARDSLEY
P.: Image-based reconstruction and synthesis of dense foliage.
ACM Trans. Graph. (Proc. SIGGRAPH) (2013). 21
[BO05] B OISSONNAT J.-D., O UDOT S.: Provably good sampling
and meshing of surfaces. Graphical Models (2005). 7
[BS12] B ERGER M., S ILVA C. T.: Medial kernels. In Computer
Graphics Forum (Proc. of Eurographics) (2012). 15
[BWM∗ 11] B ERNER A., WAND M., M ITRA N. J., M EWES D.,
S EIDEL H.-P.: Shape analysis with subspace symmetries. Computer Graphics Forum (2011). 16
c Author version
Berger et al. / State of the Art in Surface Reconstruction from Point Clouds
[CBC∗ 01] C ARR J. C., B EATSON R. K., C HERRIE J. B.,
M ITCHELL T. J., F RIGHT W. R., M C C ALLUM B. C., E VANS
T. R.: Reconstruction and representation of 3d objects with radial
basis functions. In Proc. of ACM SIGGRAPH (2001). 3, 4, 5, 7, 9
[CBI13] C HEN J., BAUTEMBACH D., I ZADI S.: Scalable realtime volumetric surface reconstruction. ACM Transactions on
Graphics (TOG) (2013). 21
[CC08] C HEN J., C HEN B.: Architectural modeling from sparsely
scanned range data. International Journal of Computer Vision
(2008). 20
[CCLN10] C HEN Y.-L., C HEN B.-Y., L AI S.-H., N ISHITA T.:
Binary orientation trees for volume and surface reconstruction
from unoriented point clouds. Computer Graphics Forum (2010).
12
[CG06] C AZALS F., G IESEN J.: Delaunay triangulation based
surface reconstruction. In Effective Computational Geometry for
Curves and Surfaces. Springer, 2006. 3
[CL96] C URLESS B., L EVOY M.: A volumetric method for building complex models from range images. In Proc. of ACM SIGGRAPH (1996). 3, 6, 7, 12, 21
[FG11] F UHRMANN S., G OESELE M.: Fusion of depth maps with
multiple scales. In Proc. of ACM SIGGRAPH Asia (2011). 3, 12
[FKSS08] F REIRE J., KOOP D., S ANTOS E., S ILVA C. T.: Provenance for computational tasks: A survey. Comp. in Science &
Engineering (2008). 21
[FLW02] FATTAL R., L ISCHINSKI D., W ERMAN M.: Gradient
domain high dynamic range compression. In ACM Trans. on
Graphics (2002). 10
[FS12] F RIEDMAN S., S TAMOS I.: Online facade reconstruction
from dominant frequencies in structured point clouds. In IEEE
CVPR Workshops (2012). 17
[GCSA13] G IRAUDOT S., C OHEN -S TEINER D., A LLIEZ P.:
Noise-adaptive shape reconstruction from raw point sets. Computer Graphics Forum (Proc. of the Symposium on Geometry
Processing) (2013). 4, 13, 21
[GG07] G UENNEBAUD G., G ROSS M.: Algebraic point set surfaces. ACM Trans. Graph. (Proc. SIGGRAPH) (2007). 5, 8,
19
[CLCL11] C HEN Y.-L., L EE T.-Y., C HEN B.-Y., L AI S.-H.:
Bipartite polar classification for surface reconstruction. Computer
Graphics Forum (2011). 13
[GSH∗ 07] G AL R., S HAMIR A., H ASSNER T., PAULY M.,
C OHEN -O R D.: Surface reconstruction using local shape priors. In Computer Graphics Forum (Proc. of the Symposium on
Geometry Processing) (2007). 18
[CLP10] C HAUVE A.-L., L ABATUT P., P ONS J.-P.: Robust
piecewise-planar 3d reconstruction and completion from largescale unstructured point data. In CVPR (2010). 15
[GWM01] G UMHOLD S., WANG X., M AC L EOD R.: Feature extraction from point clouds. In International Meshing Roundtable
(2001). 11
[CP05] C AZALS F., P OUGET M.: Estimating differential quantities using polynomial fitting of osculating jets. Computer Aided
Geometric Design (2005). 5
[HDD∗ 92] H OPPE H., D E ROSE T., D UCHAMP T., M C D ONALD
J., S TUETZLE W.: Surface reconstruction from unorganized
points. In Computer Graphics (Proc. SIGGRAPH) (1992). 3, 5, 6,
7, 8, 9
[CT11] C ALAKLI F., TAUBIN G.: Ssd: Smooth signed distance
surface reconstruction. Computer Graphics Forum (2011). 10
[CTO∗ 10] C AO J., TAGLIASACCHI A., O LSON M., Z HANG H.,
S U Z.: Point cloud skeletons via laplacian based contraction. In
Proc. of IEEE Shape Modeling International (2010). 7, 14
[CWL∗ 08] C HENG Z.-Q., WANG Y.-Z., L I B., X U K., DANG G.,
J IN S.-Y.: A survey of methods for moving least squares surfaces.
In Proceedings of the Fifth Eurographics/IEEE VGTC conference
on Point-Based Graphics (2008), Eurographics Association. 8
[HK06] H ORNUNG A., KOBBELT L.: Robust reconstruction of
watertight 3d models from non-uniformly sampled point clouds
without normal information. In Computer Graphics Forum (Proc.
of the Symposium on Geometry Processing) (2006). 3, 4, 11
[HLZ∗ 09] H UANG H., L I D., Z HANG H., A SCHER U., C OHEN O R D.: Consolidation of unorganized point clouds for surface
reconstruction. ACM Trans. on Graphics (2009). 5, 7, 9
[Dey07] D EY T. K.: Curve and surface reconstruction: algorithms
with mathematical analysis. Cambridge University Press, 2007. 3
[HWCO∗ 13] H UANG H., W U S., C OHEN -O R D., G ONG M.,
Z HANG H., L I G., C HEN B.: l1 -medial skeleton of point cloud.
ACM Trans. Graph. (Proc. SIGGRAPH) (2013). 3, 7, 14
[DGQ∗ 12] D EY T. K., G E X., Q UE Q., S AFA I., WANG L.,
WANG Y.: Feature-preserving reconstruction of singular surfaces.
In Computer Graphics Forum (2012). 4
[HWG∗ 13] H UANG H., W U S., G ONG M., C OHEN -O R D., A S CHER U., Z HANG H. R.: Edge-aware point set resampling. ACM
Trans. on Graphics (2013). 11
[DHOS07] DANIELS J. I., H A L. K., O CHOTTA T., S ILVA C. T.:
Robust smooth feature extraction from point clouds. In Proc. of
Shape Modeling and Applications (2007). 7, 11
[JKS08] J ENKE P., K RÜCKEBERG B., S TRASSER W.: Surface
reconstruction from fitted shape primitives. In Proc. of Vision
modeling and Visualization (2008). 7, 15
[DKP11] D ENNING J. D., K ERR W. B., P ELLACINI F.: Meshflow:
interactive visualization of mesh construction sequences. In ACM
Trans. on Graphics (2011). 21
[JLSW02] J U T., L OSASSO F., S CHAEFER S., WARREN J.: Dual
contouring of hermite data. In ACM Trans. on Graphics (2002). 7
[DTS01] D INH H. Q., T URK G., S LABAUGH G.: Reconstructing surfaces using anisotropic basis functions. In International
Journal of Computer Vision (2001). 11
[JWS08] J ENKE P., WAND M., S TRASSER W.: Patch-graph reconstruction for piecewise smooth surfaces. Proc. of Vision modeling
and Visualization (2008). 11
[DUNI10] D ROST B., U LRICH M., NAVAB N., I LIC S.: Model
globally, match locally: Efficient and robust 3d object recognition.
In CVPR (2010). 15
[Kaz05] K AZHDAN M.: Reconstruction of solid models from
oriented point sets. In Proc. of the EG/SIGGRAPH Symposium on
Geometry processing (2005). 4, 7, 10, 20
[FCOS05a] F LEISHMAN S., C OHEN -O R D., S ILVA C.: Robust
moving least-squares fitting with sharp features. In ACM Trans.
Graph. (Proc. SIGGRAPH) (2005). 11
[KBH06] K AZHDAN M., B OLITHO M., H OPPE H.: Poisson surface reconstruction. In Proc. of the EG/SIGGRAPH Symposium
on Geometry processing (2006). 3, 4, 5, 6, 7, 10, 21
[FCOS05b] F LEISHMAN S., C OHEN -O R D., S ILVA C. T.: Robust
moving least-squares fitting with sharp features. ACM Trans.
Graph. (Proc. SIGGRAPH) (2005). 7, 11, 19
[KE12] K HOSHELHAM K., E LBERINK S. O.: Accuracy and resolution of kinect depth data for indoor mapping applications. Sensors (2012). 4
c Author version
Berger et al. / State of the Art in Surface Reconstruction from Point Clouds
[KH13] K AZHDAN M., H OPPE H.: Screened poisson surface
reconstruction. ACM Trans. on Graphics (2013). 10, 21
[KKDH07] K AZHDAN M., K LEIN A., DALAL K., H OPPE H.:
Unconstrained isosurface extraction on arbitrary octrees. In Symposium on Geometry Processing (2007). 7
[KMYG12] K IM Y. M., M ITRA N. J., YAN D.-M., G UIBAS L.:
Acquiring 3d indoor environments with variability and repetition.
Proc. of ACM SIGGRAPH Asia (2012). 3, 7, 18, 21
[LZS∗ 11] L I Y., Z HENG Q., S HARF A., C OHEN -O R D., C HEN
B., M ITRA N. J.: 2d-3d fusion for layer decomposition of urban
facades. In ICCV (2011). 3, 6, 17, 19
[MDGD∗ 10] M ULLEN P., D E G OES F., D ESBRUN M., C OHEN S TEINER D., A LLIEZ P.: Signing the unsigned: Robust surface
reconstruction from raw pointsets. Computer Graphics Forum
(Proc. of the Symposium on Geometry Processing) (2010). 3, 4, 7,
13, 17
[KQ04] K IRBAS C., Q UEK F.: A review of vessel extraction
techniques and algorithms. ACM Computing Surveys (2004). 14
[MPS08] M ANSON J., P ETROVA G., S CHAEFER S.: Streaming
surface reconstruction using wavelets. Computer Graphics Forum
(2008). 7, 10, 20
[KSO04] KOLLURI R., S HEWCHUK J. R., O’B RIEN J. F.: Spectral surface reconstruction from noisy point clouds. In Proc. of
the EG/SIGGRAPH Symposium on Geometry processing (2004).
10, 11, 12
[MPWC13] M ITRA N. J., PAULY M., WAND M., C EYLAN D.:
Symmetry in 3d geometry: Extraction and applications. In
Computer Graphics Forum (STAR Proceedings of Eurographics)
(2013). 16
[KTB07] K ATZ S., TAL A., BASRI R.: Direct visibility of point
sets. ACM Trans. Graph. (Proc. SIGGRAPH) (2007). 12, 13
[MS10] M ANSON J., S CHAEFER S.: Isosurfaces over simplicial
partitions of multiresolution grids. In Computer Graphics Forum
(2010). 7
[LA11] L AW A. J., A LIAGA D. G.: Single viewpoint model
completion of symmetric objects for digital inspection. Computer
Vision Image Understanding (2011). 16
[LA13] L AFARGE F., A LLIEZ P.: Surface reconstruction through
point set structuring. Computer Graphics Forum (Proc. of Eurographics) (2013). 3, 16
[MTSM10] M EHRA R., T RIPATHI P., S HEFFER A., M ITRA N. J.:
Visibility of noisy point cloud data. Computers & Graphics (2010).
12
[LB07] L EMPITSKY V., B OYKOV Y.: Global optimization for
shape fitting. In CVPR (2007). 12
[MWA∗ 13] M USIALSKI P., W ONKA P., A LIAGA D. G., W IM MER M., G OOL L., P URGATHOFER W.: A survey of urban
reconstruction. In Computer Graphics Forum (STAR Proceedings
of Eurographics) (2013). 2, 7
[LC87] L ORENSEN W. E., C LINE H. E.: Marching cubes: A high
resolution 3d surface construction algorithm. In Proc. of ACM
SIGGRAPH (1987). 7
[MWZ∗ 13] M ITRA N. J., WAND M., Z HANG H., C OHEN -O R D.,
B OKELOH M.: Structure-aware shape processing. In Computer
Graphics Forum (STAR Proceedings of Eurographics) (2013). 16
[LCDF10] L IPMAN Y., C HEN X., DAUBECHIES I.,
Symmetry factored embedding and
F UNKHOUSER T.:
distance. ACM Trans. on Graphics (2010). 16
[MZL∗ 09] M EHRA R., Z HOU Q., L ONG J., S HEFFER A.,
G OOCH A., M ITRA N. J.: Abstraction of man-made shapes.
ACM Transactions on Graphics (TOG) (2009). 20
[LCOL06] L IPMAN Y., C OHEN -O R D., L EVIN D.: Error bounds
and optimal neighborhoods for mls approximation. In Computer
Graphics Forum (Proc. of the Symposium on Geometry Processing) (2006). 4, 5, 8
[NDI∗ 11] N EWCOMBE R. A., DAVISON A. J., I ZADI S., KOHLI
P., H ILLIGES O., S HOTTON J., M OLYNEAUX D., H ODGES S.,
K IM D., F ITZGIBBON A.: Kinectfusion: Real-time dense surface
mapping and tracking. In IEEE international symposium on Mixed
and augmented reality (ISMAR) (2011), IEEE. 21
[LCOL07] L IPMAN Y., C OHEN -O R D., L EVIN D.: Datadependent MLS for faithful surface approximation. In Computer
Graphics Forum (Proc. of the Symposium on Geometry Processing) (2007). 11
[NFD07] N EUBERT B., F RANKEN T., D EUSSEN O.: Approximate image-based tree-modeling using particle flows. ACM Trans.
Graph. (Proc. SIGGRAPH) (2007). 14, 15
[LCOLTE07] L IPMAN Y., C OHEN -O R D., L EVIN D., TAL -E ZER
H.: Parameterization-free projection for geometry reconstruction.
ACM Trans. Graph. (Proc. SIGGRAPH) (2007). 3, 4, 7, 9, 11
[LLR13] L UO L., L I H., RUSINKIEWICZ S.: Structure-aware hair
capture. ACM Trans. Graph. (Proc. SIGGRAPH) (2013). 21
[LLZM10] L I G., L IU L., Z HENG H., M ITRA N. J.: Analysis,
reconstruction and manipulation using arterial snakes. Proc. of
ACM SIGGRAPH Asia (2010). 3, 7, 14, 20
[LPK09] L ABATUT P., P ONS J.-P., K ERIVEN R.: Robust and efficient surface reconstruction from range data. Computer Graphics
Forum (2009). 12, 16
[LW10] L IU S., WANG C. C.: Orienting unorganized points for
surface reconstruction. Computers & Graphics (2010). 5, 6
[LWC∗ 11]
L I Y., W U X., C HRYSATHOU Y., S HARF A., C OHEN O R D., M ITRA N. J.: Globfit: consistently fitting primitives
by discovering global relations. ACM Trans. Graph. (Proc. SIGGRAPH) (2011). 3, 4, 6, 17, 18
[LYO∗ 10] L IVNY Y., YAN F., O LSON M., C HEN B., Z HANG H.,
E L -S ANA J.: Automatic reconstruction of tree skeletal structures
from point clouds. Proc. of ACM SIGGRAPH Asia (2010). 6, 14,
15
[NOS09] NAGAI Y., O HTAKE Y., S UZUKI H.: Smoothing of
partition of unity implicit surfaces for noise robust surface reconstruction. Computer Graphics Forum (Proc. of the Symposium on
Geometry Processing) (2009). 9
[NSZ∗ 10] NAN L., S HARF A., Z HANG H., C OHEN -O R D.,
C HEN B.: Smartboxes for interactive urban reconstruction. ACM
Trans. Graph. (Proc. SIGGRAPH) (2010). 3, 19
[NXS12] NAN L., X IE K., S HARF A.: A search-classify approach for cluttered indoor scene understanding. Proc. of ACM
SIGGRAPH Asia (2012). 7, 18, 21
[NZIS13]
N IESSNER M., Z OLLHÖFER M., I ZADI S., S TAM M.: Real-time 3d reconstruction at scale using voxel
hashing. ACM Transactions on Graphics (TOG) (2013). 21
MINGER
[OBA∗ 03a] O HTAKE Y., B ELYAEV A., A LEXA M., T URK G.,
S EIDEL H.: Multi-level partition of unity implicits. ACM Trans.
Graph. (Proc. SIGGRAPH) (2003). 3, 5, 7, 8
[OBA∗ 03b] O HTAKE Y., B ELYAEV A., A LEXA M., T URK G.,
S EIDEL H.-P.: Multi-level partition of unity implicits. In ACM
Trans. Graph. (Proc. SIGGRAPH) (2003). 11
[OGG09] O ZTIRELI C., G UENNEBAUD G., G ROSS M.: Feature
preserving point set surfaces based on non-linear kernel regression.
In Computer Graphics Forum (2009). 4, 11
c Author version
Berger et al. / State of the Art in Surface Reconstruction from Point Clouds
[OLA13] O ESAU S., L AFARGE F., A LLIEZ P.: Indoor scene
reconstruction using primitive-driven space partitioning and graphcut. In Proc. of EG workshop on Urban Data Modeling and Vis.
(Girona, Spain, 2013). 16
[SSFS06] S CHREINER J., S CHEIDEGGER C. E., F LEISHMAN S.,
S ILVA C. T.: Direct (re)meshing for efficient surface processing.
Computer Graphics Forum (Proc. of Eurographics) (2006). 8
[PGB03] P ÉREZ P., G ANGNET M., B LAKE A.: Poisson image
editing. ACM Trans. Graph. (Proc. SIGGRAPH) (2003). 10
[SSZCO10] S HALOM S., S HAMIR A., Z HANG H., C OHEN -O R
D.: Cone carving for surface reconstruction. Proc. of ACM
SIGGRAPH Asia (2010). 3, 13
[PKG03] PAULY M., K EISER R., G ROSS M.: Multi-scale feature
extraction on point-sampled surfaces. Computer Graphics Forum
(2003). 11
[SWK07] S CHNABEL R., WAHL R., K LEIN R.: Efficient ransac
for point-cloud shape detection. Computer Graphics Forum
(2007). 5, 15, 17
[PMG04] PAULY M., M ITRA N. J., G UIBAS L.: Uncertainty and
variability in point cloud surface data. In Proc. of the EG conf. on
Point-Based Graphics (2004). 5
[SXZ∗ 12] S HAO T., X U W., Z HOU K., WANG J., L I D., G UO B.:
An interactive approach to semantic modeling of indoor scenes
with an rgbd camera. Proc. of ACM SIGGRAPH Asia (2012). 3,
7, 18, 20, 21
[PMG∗ 05] PAULY M., M ITRA N. J., G IESEN J., G ROSS M. H.,
G UIBAS L. J.: Example-based 3d scan completion. In Proc. of
the EG/SIGGRAPH Symposium on Geometry processing (2005).
3, 18
[PMW∗ 08] PAULY M., M ITRA N. J., WALLNER J., P OTTMANN
H., G UIBAS L. J.: Discovering structural regularity in 3d geometry. ACM Trans. Graph. (Proc. SIGGRAPH) (2008). 3, 4, 7,
16
[PP09] PANG X.-F., PANG M.-Y.: An algorithm for extracting geometric features from point cloud. Int. Conf. on Inf. Management,
Innovation Management and Ind. Eng. (2009). 11
[RFL∗ 05] RUNIONS A., F UHRER M., L ANE B., F EDERL P.,
ROLLAND -L AGAN A.-G., P RUSINKIEWICZ P.: Modeling and
visualization of leaf venation patterns. ACM Trans. Graph. (Proc.
SIGGRAPH) 24, 3 (2005), 702–711. 14
[RKMP13]
R EISNER -KOLLMANN I., M AIERHOFER S., P UR W.: Reconstructing shape boundaries with multimodal constraints. Computers & Graphics (2013). 7, 16
GATHOFER
[SY12] S EVERSKY L. M., Y IN L.: A global parity measure for
incomplete point cloud data. Computer Graphics Forum (2012).
13
[TKM∗ 13] TANSKANEN P., KOLEV K., M EIER L., C AMPOSECO
F., S AURER O., P OLLEFEYS M.: Live metric 3d reconstruction
on mobile phones. In ICCV (2013). 21
[TOZ∗ 11] TAGLIASACCHI A., O LSON M., Z HANG H.,
H AMARNEH G., C OHEN -O R D.: Vase: Volume-aware surface
evolution for surface reconstruction from incomplete point clouds.
Computer Graphics Forum (Proc. of the Symposium on Geometry
Processing) (2011). 3, 6, 12, 15
[TZCO09] TAGLIASACCHI A., Z HANG H., C OHEN -O R D.:
Curve skeleton extraction from incomplete point cloud. ACM
Trans. Graph. (Proc. SIGGRAPH) (2009). 2, 3, 4, 7, 14
[UB13] U MMENHOFER B., B ROX T.: Point-based 3d reconstruction of thin objects. In ICCV (2013). 12
[SDK09] S CHNABEL R., D EGENER P., K LEIN R.: Completion
and reconstruction with primitive shapes. Computer Graphics
Forum (Proc. of Eurographics) (2009). 3, 4, 6, 7, 15
[VAB12] VANEGAS C. A., A LIAGA D. G., B ENES B.: Automatic
extraction of manhattan-world building masses from 3d laser
range scans. Trans. on Visualization and Computer Graphics
(2012). 7, 17
[SFC07] S ILVA C. T., F REIRE J., C ALLAHAN S. P.: Provenance
for visualizations: Reproducibility and beyond. Comp. in Science
& Engineering (2007). 21
[vKvLV13] VAN K REVELD M., VAN L ANKVELD T., V ELTKAMP
R. C.: Watertight scenes from urban lidar and planar surfaces.
Computer Graphics Forum (2013). 3, 16
[SFCH12] S HEN C.-H., F U H., C HEN K., H U S.-M.: Structure
recovery by part assembly. Proc. of ACM SIGGRAPH Asia (2012).
2, 3, 6, 7, 19, 20
[vKZHCO11] VAN K AICK O., Z HANG H., H AMARNEH G.,
C OHEN -O R D.: A survey on shape correspondence. Computer
Graphics Forum (STAR Proceedings of Eurographics) (2011). 4
[SFS05] S CHEIDEGGER C. E., F LEISHMAN S., S ILVA C. T.:
Triangulating point set surfaces with bounded error. In Proc. of
the EG/SIGGRAPH Symposium on Geometry processing (2005).
8, 15
[Wen95] W ENDLAND H.: Piecewise polynomial, positive definite and compactly supported radial functions of minimal degree.
Advances in computational Mathematics (1995). 9
[SHFH11] S HEN C.-H., H UANG S.-S., F U H., H U S.-M.: Adaptive partitioning of urban facades. Proc. of ACM SIGGRAPH Asia
(2011). 17
[SLS∗ 06] S HARF A., L EWINER T., S HAMIR A., KOBBELT L.,
C OHEN -O R D.: Competing fronts for coarse–to–fine surface reconstruction. Computer Graphics Forum (Proc. of Eurographics)
(2006). 14, 15
[SLS∗ 07] S HARF A., L EWINER T., S HKLARSKI G., T OLEDO S.,
C OHEN -O R D.: Interactive topology-aware surface reconstruction. ACM Trans. Graph. (Proc. SIGGRAPH) (2007). 3, 4, 19,
20
[SLSK07] S HARF A., L EWINER T., S HAMIR A., KOBBELT L.:
On-the-fly curve-skeleton computation for 3d shapes. In Computer
Graphics Forum (Proc. of Eurographics) (2007). 14
[SOS04] S HEN C., O’B RIEN J. F., S HEWCHUK J. R.: Interpolating and approximating implicit surfaces from polygon soup. ACM
Trans. on Graphics (2004). 8
c Author version
[WFP10] W U C., F RAHM J.-M., P OLLEFEYS M.: Detecting large
repetitive structures with salient boundaries. In ECCV (2010). 17
[Whi98] W HITAKER R. T.: A level-set approach to 3D reconstruction from range data. International Journal of Computer Vision
29, 3 (1998), 203–231. 12, 15
[WS12] WAN G., S HARF A.: Grammar-based 3d facade segmentation and reconstruction. Computers & Graphics (2012). 17
[WSS09] WANG H., S CHEIDEGGER C. E., S ILVA C. T.: Bandwidth selection and reconstruction quality in point-based surfaces.
Trans. on Visualization and Computer Graphics (2009). 4, 5
[XF12] X IAO J., F URUKAWA Y.: Reconstructing the worlds museums. In ECCV (2012). 3, 16
[ZN12] Z HOU Q.-Y., N EUMANN U.: 2.5 d building modeling by
discovering global regularities. In CVPR (2012). 17
[ZPB07] Z ACH C., P OCK T., B ISCHOF H.: A globally optimal
algorithm for robust tv-l1 range image integration. In ICCV (2007).
3, 12
Berger et al. / State of the Art in Surface Reconstruction from Point Clouds
[ZST∗ 10] Z HENG Q., S HARF A., TAGLIASACCHI A., C HEN B.,
Z HANG H., S HEFFER A., C OHEN -O R D.: Consensus skeleton
for non-rigid space-time registration. Computer Graphics Forum
(Proc. of Eurographics) (2010). 20
[ZSW∗ 10] Z HENG Q., S HARF A., WAN G., L I Y., M ITRA N. J.,
C OHEN -O R D., C HEN B.: Non-local scan consolidation for 3d
urban scenes. ACM Trans. Graph. (Proc. SIGGRAPH) (2010). 2,
3, 7, 17
c Author version