Natural Language Problem Definition
for Computer-Aided Mechanical Design
Hyunmin Cheong
Alex Tessier
Abstract
Dept. of Mech. & Ind. Eng.
Autodesk Research
University of Toronto
210 King St. East
5 King’s College Road
Toronto ON M5A 1J7, Canada
Toronto ON M5S 3G8, Canada
[email protected]
Wei Li
Autodesk Research
Autodesk Research
The Landmark @ One Market,
210 King St. East
Suite 200
Toronto ON M5A 1J7, Canada
San Francisco, CA 94105
[email protected]
[email protected]
This paper presents our research approach and
challenges in designing natural language interaction for
a mechanical computer-aided design (CAD) system. We
intend to use natural language input, ideally speech, as
one method for capturing design problem definition
from designers. Based on the problem definition, a new
type of CAD system could explore a set of geometries
that solves the problem. Our approach involves solving
a natural language understanding problem: translating
problem definition statements into a formal language
that can be used for our exploration procedures.
L. H. Shu
Francesco Iorio
Dept. of Mech. & Ind. Eng.
Author Keywords
Autodesk Research
University of Toronto
210 King St. East
5 King’s College Road
Natural language interaction, natural language
understanding, CAD, design problem definition
Toronto ON M5A 1J7, Canada
Toronto ON M5S 3G8, Canada
[email protected]
[email protected]
Erin Bradner
[email protected]
[email protected]
Copyright is held by the author/owner(s).
ACM Classification Keywords
H.5.2. User Interfaces: Natural language.
Introduction
Computer-aided design (CAD) offers a wide variety of
functions that facilitate a design process. Traditional
CAD systems primarily support modeling and analysis
of conceptual solutions. However, these systems rely
on users to interpret their design problems and create
conceptual solutions on their own. The systems do not
actually explore and provide solutions for users.
Function
Definition: What the
design must do, i.e.,
the purpose of the
design
Example: “The
design must support
the weight of the
shelf.”
Objective
Definition: Measures
used to judge how
well the design solves
the problem.
Example: “The
weight of the design
must be minimized.”
Constraint
Definition: The
limits that the design
should not violate.
Example: “The
bracket cannot be
wider than 3cm.”
Table 1. Definitions and examples of
functional requirements for a new
bracket
We envision a new design workflow that starts with a
user specifying the design problem at hand, and the
system provides a set of geometries that solves the
problem. The system would also have the ability to
provide not just valid, but optimal solutions. To enable
this workflow, the system must translate the design
problem definition specified by the user into a
constrained optimization problem.
Problem definition in design
Designers mostly use natural language to describe
design problems at the early stage of design [3].
Problem definition statements are usually recorded in
product proposal or requirements documents that are
used as reference for subsequent design processes.
The most important aspect of the problem definition
process involves identifying functional requirements
[2]. Functional requirements can include functions,
objectives and constraints of the design [1]. Table 1
shows examples of problem definition statements for
designing a bracket.
Defining functional requirements often requires
describing how the design interacts with other objects
in the environment. For example, a bracket’s function
of supporting a structure is related to the density and
volume of the structure. Hence, we consider
descriptions about environment objects and their
relations to the design as part of the problem definition.
Related work
Most research on natural language input for CAD has
focused on improving existing graphical user interfaces.
Improvements involved using a predefined set of voice
commands to augment other input modes or work in a
virtual environment [4-6]. Recently, Kou et al. applied
natural language processing techniques to handle
variations of similar voice commands [5].
Some design researchers developed formal languages
such as the Functional Basis [7] to support more
consistent and explicit problem definition. Modeling
design problems with the Functional Basis has enabled
automation of conceptual design synthesis [8].
However, functional modeling alone cannot capture
requirements such as objectives and constraints, which
are necessary for formulating optimization problems.
Designers must learn a new modeling language as well.
Instead of asking users to learn and use a formal
language, techniques could be developed to translate
natural language input into formal representations [9].
Specific to mechanical design, Chen et al. worked on
translating descriptions of product requirements into a
formal structure diagram [10]. The authors used
lexical, syntactic, and structural analysis to process
natural language descriptions in complete sentences.
Another approach could be to use a controlled natural
language (CNL). A CNL is a precisely defined subset of
a natural language that restricts the syntax and
lexicons to reduce ambiguities and complexities in the
language. A CNL can be translated automatically into a
formal target language and then be used for automated
reasoning. Early examples of CNL include Cleopatra
[11], an interface introduced for CAD command input.
CNL has also gained much attention as a high-level
interface to knowledge-based systems [12]. In
addition, a CNL has been used to control software
requirement specifications that can be automatically
translated into Unified Modeling Language [13].
“The design must support the
weight of the shelf.”
Lexical analysis
Object (design), Function (support),
MechanicalProperty (weight),
Object (shelf)
Syntactic analysis
 Problem definition type: Function
o Object 1: design
o Object 2: shelf
o Relation: support-weight
Figure 1. Syntactic/lexical analysis
of a problem definition statement
Our research approach
Our approach is to use a CNL to limit the complexity of
natural language input, while applying some natural
language understanding techniques to translate
designer’s problem definition into a formal language.
The formal language should be formulated into
constrained optimization procedures.
We have developed a CNL that can define simple static
mechanics problems. We created a grammar and a set
of lexicons that designers can use to describe their
design problems in different semantic categories: e.g.,
functions, objectives, and constraints of the design.
With a CNL as input, designers only need to learn the
predefined syntax and lexicons, instead of an entirely
new language. Our CNL is designed to achieve a
balance between its expressiveness and complexity.
design domain implies that the weight of one object is
transferred to another, and the two objects are in
contact with each other. Such inferred information must
be captured in our formal language. Creating an
adequate knowledge base for a mechanical design
problem is beyond our current scope and almost a
separate research topic on its own.
After processing user input, we provide immediate
feedback to designers. To visualize relationships
between the design and environment objects, we
create an object-relationship graph diagram (Figure 2).
We also list constraints and properties associated with
each object. Effective feedback is essential in creating
positive user experience in natural language
interaction.
Preliminary user study
To process user input, we first tokenize problem
definition statements based on our lexicons, such as
object/function names and mechanical properties. We
also use part-of-speech tags generated from an off-theshelf tagger to remove ambiguities in the words used.
For example, “support” is identified as either an object
name or a function name based on whether it was used
as a noun or a verb. We then use a parser, designed
based on our grammar, to classify statements into
corresponding problem definition categories. Figure 1
shows an example of input analysis.
We conducted a user study with 18 graduate
mechanical engineering students to evaluate the effect
of using a CNL to create problem definition statements.
We found that using a CNL increases the (humanrated) quality of problem definition statements
compared to statements written in natural language
(Figure 3). However, using a CNL restricted the breadth
of problem definition considered by designers, indicated
by the fact that many natural language statements
could not be translated into CNL statements (Figure 4).
Future work
Figure 2. Visualization of
relationships between objects
We also need some semantic knowledge about the
lexicons of our CNL to translate them into a formal
language. For example, we need to know the meaning
of “support” to interpret the statement, “The design
must support the shelf,” in terms of mechanical and
spatial relationships. “Support” in the mechanical
Our future work will involve expanding our CNL to
increase its expressiveness. Much of our work will focus
on creating a mechanical design knowledge-base. The
knowledge base must formally define various functions,
object shapes, and spatial relations to interpret
problem definition statements. In addition, we need a
database of materials, material properties, and
dimensional properties.
We will also explore applying various natural language
understanding techniques to handle a greater variety of
problem definition statements. For example, syntactic
analysis of statements could be used to interpret
different sentence structures with the same meaning,
e.g., sentences written in active versus passive voice.
Also, anaphora resolution techniques could be used to
allow expression of related statements in a series.
Figure 3. Comparison of average
quality ratings on statements
generated using CNL vs. NL. Error
bars represent standard errors.
Our goal is to demonstrate that the user experience is
in fact enhanced through the use of natural language
input. Therefore, we must investigate how natural
language interaction fits within an overall interface. We
envision that an ideal interface would feature multimodal input, including speech recognition and humancomputer dialogue. Some input types, such as problem
definitions involving spatial relations and geometric
properties, may be better communicated in a graphical
user interface. In addition, speaking a CNL could be
much more challenging than writing in a CNL.
Acknowledgements
We thank Natural Sciences and Engineering Research
Council of Canada for their financial support.
Figure 4. Percentage of NL
statements that could be translated
into CNL statements
References
[1] Ullman, D.G. (1992) The Mechanical Design
Process. McGraw-Hill: New York.
[2] Pahl, G., and Beitz, W. (1996) Engineering Design:
A Systematic Approach, 2nd edition. Springer-Verlag,
London, UK.
[3] McCahan, S. (2013) Introduction to Engineering
Design: Preliminary Edition. Wiley.
[4] Chu, C-C.P., Dani, T.H., and Gadh, R. (1997) Multisensory user interface for a virtual-reality-based
computer-aided design system. Computer-Aided Design
29, 10, 709-725.
[5] Kou, X.Y., Xue, S.K., and Tan, S.T. (2010)
Knowledge-guided inference for voice-enabled CAD.
Computer-Aided Design 42, 545-557.
[6] Golfinopoulos, V., Makris, D., Bardis, G., Miaoulis,
G., and Plemenos, D. (2011) Verbalization of 3d scenes
based on natural language generation techniques.
Intelligent Comp. Graphics, SCI 374, 19–39.
[7] Hirtz, J., Stone, R.B., Szykman, S., McAdams, D.A.,
and Wood, K.L. (2002) A functional basis for
engineering design: reconciling and evolving previous
efforts. NIST Technical Note 1447.
[8] Kurtoglu, T., and Campbell, M.I. (2009) Automated
synthesis of electromechanical design configurations
from empirical analysis of function to form mapping.
Journal of Engineering Design 20, 1, 83-104.
[9] Duan, Y., and Cruz, C. (2011) Formalizing semantic
of natural language through conceptualization from
existence. Int. J. Innovation, Management and
Technology 2, 1, 37-42.
[10] Chen, Z.Y., Yao, S., Lin, J.Q., Zeng, Y., and
Eberlein, A. (2007) Formalisation of product
requirements: from natural language descriptions to
formal specifications. Int. J. Manufacturing Research 2,
3, 362-387.
[11] Samad, T., and Director, S.W. (1985) Towards a
natural language interface for CAD. 22nd Design
Automation Conference, Paper 2.1, pp. 2-8.
[12] Schwitter, R. (2010) Controlled natural language
for knowledge representation. Coling 2010: Poster
Volume, Beijing, August 2010, pp. 1113–1121.
[13] Sarkar, S., Sharma, V.S., and Agarwal, R. (2012)
Creating design from requirements and use cases:
Bridging the gap between requirement and detailed
design. Proc. of ISEC ’12, pp. 3-12.