SafetyAssist
a software framework for automated and
distributed testing and evaluation of ADAS
functions
Nicolas DU LAC – [email protected]
Xavier ROUAH – [email protected]
2014
© 2014 carhs.training gmbh
1
SafetyAssist
Contents
 ADAS validation process
 I-DEEP Architecture
 I-DEEP Ecosystem
 I-DEEP Short Demonstration
 Conclusion
2014
© 2014 carhs.training gmbh
2
SafetyAssist
INTEMPORA - Distributed Evaluation & Execution Platform
ADAS VALIDATION PROCESS
2014
© 2014 carhs.training gmbh
3
SafetyAssist
ADAS Validation Process - Context
 State of the art ADAS functions
 Rely on more and more complex systems like vision systems
 Take vehicle’s control in many situations
 Validation process difficulties




Need hundred of thousands kilometers on road
Difficulty to foresee all possible situations
Fault tolerance assessment: non-detection, false alarm, …
Need to conform with standards like ISO-26262
2014
© 2014 carhs.training gmbh
4
SafetyAssist
ADAS Validation Process
 Needs tools & methodologies to validate ADAS functions
 Reduce needed number of kilometers on road
 Batch evaluation against real or simulated scenarios
 Real or simulated scenarios
 Playback specific scenarios
2014
© 2014 carhs.training gmbh
5
SafetyAssist
INTEMPORA - Distributed Evaluation & Execution Platform
I-DEEP ARCHITECTURE
2014
© 2014 carhs.training gmbh
6
SafetyAssist
I-DEEP Architecture
 I-DEEP: Intempora Distributed Evaluation & Execution Platform
 Enables batch processing of data sets and/or simulated scenarios
 The platform automatically pilots execution tools and simulators according
to defined test cases on a clustered computing environment
 Test cases can be defined statically or dynamically by third party tools
2014
© 2014 carhs.training gmbh
7
SafetyAssist
I-DEEP Architecture
 A web interface enables:
 Upload & manage resources like datasets, simulator scenarios, algorithms, …
 Define & manage test cases
 Consult results
 Resources can be tagged with user-defined and time stamped attributes.
 Open interfaces based on Web services
2014
© 2014 carhs.training gmbh
8
SafetyAssist
I-DEEP Architecture
 Modular approach for control tools, simulators and processing tools
Create test case
Control
Tool
I-DEEP
Retrieve result
Initialize
Load resources
Set parameters
Start / Stop
Check stop conditions
Retrieve results
Simulator
2014
© 2014 carhs.training gmbh
9
Processing
tool
SafetyAssist
INTEMPORA - Distributed Evaluation & Execution Platform
I-DEEP ECOSYSTEM
2014
© 2014 carhs.training gmbh
10
SafetyAssist
I-DEEP Ecosystem
 I-DEEP can interact with variety of tools like
 Control tools
 Generate and upload test cases (eventually dynamically)
 Simulators
 Simulate environments according to test cases
 Provide ground truth information
 Processing & data fusion tools
 Run algorithms and control laws
 Playback recorded datasets
 I-DEEP already works with one tool of each kind…
2014
© 2014 carhs.training gmbh
11
SafetyAssist
I-DEEP Ecosystem
Web clients
MaTeLo
Pro-SiVIC
RTMaps
2014
Sensors
datasets
© 2014 carhs.training gmbh
12
SafetyAssist
I-DEEP Ecosystem : MaTeLo
 Model-Based Testing Tool by All4tec
Model Design
User-oriented
Start
Selection of one
suitable Test
Generation
Strategy (for
instance, Useroriented strategy)
Requirements
public class TestCase1()
public
class
TestCase2()
public
void
setUp()
public
void
setUp()
public
class TestCaseN()
throws Exception
{
throws
Exception
{
public= void setUp()
FirefoxProfile profile
FirefoxProfile
profileException
=
throws
{
new FirefoxProfile();
new
FirefoxProfile();
FirefoxProfile
profile
=
if(activeFireBug)
if(activeFireBug)
new FirefoxProfile();
if(activeFireBug)
Close
OPERATIONAL
COVERAGE
2014
Automatic Test
Generation
Set of Test Scripts
ready to execute
© 2014 carhs.training gmbh
Usage
Test profile
13
SafetyAssist
I-DEEP Ecosystem : MaTeLo
 Key functionalities
 Intuitive : Test Design by using Graphical Models (Usage Models) based on
Markov Chains
 Adapted to multiple types of test :




Functional Test
Structural Test
Performance Test
…
 Suitable Test Generation to the Selected Test Strategy
 Improve Test Automation : Test Cases are generated as Test Scripts
 Integration within I-DEEP
2014
 Tests Scripts generated by MaTeLo are used as inputs by I-DEEP to control the
test environment (asynchronous interaction)
 MaTeLo can adapt the test generation by focusing specific types of test cases
depending on previous test results provided by I-DEEP
© 2014 carhs.training gmbh
14
SafetyAssist
I-DEEP Ecosystem : Pro-SiVIC
 Simulation Platform by Civitec
 Multi-frequency environments
 Multi-technology sensors
Camera
Laser scanner
Inertial, Odometer
…
3D World
Simulation
Environment
as seen by
the sensor
Sensor Output
Time stamped data
Consumed by other
tools
2014
© 2014 carhs.training gmbh
15
SafetyAssist
I-DEEP Ecosystem : Pro-SiVIC
 Key functionalities
 Multiple time management
 Real-Time simulation
 Virtual time simulation
 Offline simulation
 Sensor simulation
 Camera modeling (optical elements, sensor characteristics, digital processing)
 Climatic effects simulation
 Complex lighting
 Integration within I-DEEP
 Simulate dynamic scenarios according to test cases definitions
 Feed processing & data fusion tools with simulated sensors data
2014
© 2014 carhs.training gmbh
16
SafetyAssist
I-DEEP Ecosystem : RTMaps
 Real-Time Multimodal Applications
 Perception
 Localization
 Decision
Simulator data
2014
Sensor datasets
© 2014 carhs.training gmbh
17
SafetyAssist
I-DEEP Ecosystem : RTMaps
 Key functionalities







Supports any kind of sensors and actuators
Asynchronous Data Acquisition
Data Time Stamping
Real Time Data processing & fusion
Record / Playback capabilities
Latencies measurement
Multithread
 Integration within I-DEEP
 Execute ADAS functions against
 Simulated data
 Replayed real datasets
 Provide results to I-DEEP after execution
2014
© 2014 carhs.training gmbh
18
SafetyAssist
I-DEEP Ecosystem : Mathworks Simulink
 Command / Control
 Dynamic systems simulation
2014
© 2014 carhs.training gmbh
19
SafetyAssist
INTEMPORA - Distributed Evaluation & Execution Platform
I-DEEP SHORT DEMONSTRATION
2014
© 2014 carhs.training gmbh
20
SafetyAssist
I-DEEP Short Demonstration
 Use I-DEEP to evaluate a stereo vision algorithm running in RTMaps
 Feed RTMaps with data from highway simulated environment in Pro-SiVIC
 Stereo camera data consumed by the algorithm
 Ground truth used for results assessment
 Evaluation definition:
 Run in various fog densities
 Run 30 seconds for each fog density
 Retrieve percent of correct and wrong detections
2014
© 2014 carhs.training gmbh
21
SafetyAssist
I-DEEP Short Demonstration
 Upload resources
 RTMaps diagram (Stereo Vision algorithm)

 Pro-SiVIC script on highway
2014
© 2014 carhs.training gmbh
22
SafetyAssist
I-DEEP Short Demonstration
Search uploaded resources
 Define an evaluation…
Retrieve available properties in resources
2014
© 2014 carhs.training gmbh
23
SafetyAssist
I-DEEP Short Demonstration
RTMaps
2014
Pro-SiVIC
© 2014 carhs.training gmbh
24
SafetyAssist
INTEMPORA - Distributed Evaluation & Execution Platform
CONCLUSION
2014
© 2014 carhs.training gmbh
25
SafetyAssist
Conclusion
 I-DEEP aims at reducing validation times and efforts for ADAS functions
 Particularly oriented towards vision-based and data-fusion applications
 Reduces real testing needs thanks to simulation tools and sensors datasets
 Relies on a complete software toolset





MaTeLo by ALL4TEC (test cases generation)
Pro-SiVIC by CIVITEC (environment and sensors simulation)
RTMaps by INTEMPORA (multi-sensor data fusion)
Simulink by MATHWORKS (command – control)
Other tools can be integrated easily…
2014
© 2014 carhs.training gmbh
26
SafetyAssist
Thank you
Nicolas DU LAC – [email protected]
Xavier ROUAH – [email protected]
2014
© 2014 carhs.training gmbh
27