Philipp Slusallek
German Research Center for Artificial Intelligence (DFKI)
Saarland University
Excellence Cluster Multimodal Computing and Interaction (MMCI)
European High-Level Expert Group on AI
Wie hilft Künstliche Intelligenz,
die Welt zu verstehen?
Agents & Simulated Reality:AI & Graphics & HPC & Security
Research:
Topics &
Teams
Philipp Slusallek
Distributed & Web-
Based Systems
René Schubotz
Large-Scale Virtual
Environments
Philipp Slusallek
Multimodal
Computing
and
Interaction
High-Performance
Graphics & Computing
Richard Membarth
Distributed Realistic
Graphics
Philipp Slusallek
Computational
3D Imaging
Tim Dahmen
Knowledge- and Technology Transfer
VisCenter
Georg Demme
Strategic
Relations
Hilko Hoffmann
SW-Engineering &
Organization
Georg Demme
Scientific Director
Survivable Systems
and Services
Philipp Slusallek
Intelligent
Information Systems
Matthias Klusch
Multi-Agent
Systems
Klaus Fischer
Behavior, Interaction
& Visualization
Georg Demme
Visual
Computing
Philipp Slusallek
Application
DomainsAutonomous
Driving
Christian Müller
Industrie 4.0
Ingo Zinnikus
Smart
Environments
Hilko Hoffmann
Autonomous
Driving
Christian Müller
Smart System
Security
Andreas Nonnengart
High-Performance
Computing
Richard Membarth
Computational
Sciences
Tim Dahmen
Flexible Production Control
Using Multiagent Systems
Verification and Secure Systems
(BSI-certified Evaluation Center)Physically-Based Image Synthese
Scientific Visualisation GIS and Geo Visualization Reconstruction of Cultural Heritage
Future City Planning and Management Large 3D Models and Environments Large Visualization Systems
Intelligent Human Simulation in Production Web-based 3D Application (XML3D) Distributed Visualization on the Internet
ASR Research Topics
Multi-Agenten-Systeme:
Saarstahl, Völklingen
Flexible Production Control
Using Multiagent Systems
at Saarstahl, Völklingen
Multi-agent technology has been running a steelworks
24/7 for ~10 years now
Physically-Based Image Synthesis
with Real-Time Ray Tracing
Key product offered now by all major HW vendors:
Intel (Embree), Nvidia (OptiX), AMD, Imagination, …
DFKI Agenten und Simulierte Realität
7
Efficient Simulation of Illumination:
Light Propagation and Sensor Models
VCM now part of most commercial renders:
RenderMan, V-Ray, Corona, …
Large Visualization Systems Using Ray-Tracing
Several spin-off companies from our group:
inTrace, Motama, xaitment, Pxio, …
Real-Time Ray Tracing Processor [Siggraph’05]
Real-Time Ray Tracing Hardware is integral
part of every Nvidia GPU starting end of 2018
AnyDSL Compiler Framework
De
ve
lop
er
Computer
Vision
DSL
AnyDSL Compiler Framework (Thorin)
Physics
DSL…
Ray
Tracing
DSL
Various Backends (via LLVM)
Parallel
Runtime
DSL
AnyDSL Unified Program Representation
Layered DSLs
CPUs GPUs FPGAs
Verteilte Visualisierung im
Internet
• Mobile Visualization
• Jens Krüger
Display as a Service (DaaS Pxio):
Distributed Visualization on the Internet
DFKI Agenten und Simulierte Realität 14
Intelligent Human Simulation,
e.g. in Production Environments
What is AI?
• Artificial Intelligence (AI)
“Intelligence” exhibited by machines
Colloquially, the term "artificial intelligence" is applied when a machine mimics "cognitive functions” that humans associate with other human minds “
Study of “Intelligent/Autonomous Agents or Systems"
cf. Russell & Norvig (AI „Bible“), Wikipedia
What is AI?
• Elements of AI Systems: Perception
• … to find out what the world around “looks” like (incl. digital sources)
Machine Learning (Deep Learning)• … to automatically build/adapt models of reality
Knowledge Representation• … to represent, process, and communicate models of the environment
Reasoning• ... to infer and generalize new models based on existing knowledge
Planning• … to plan and simulate in the virtual world, exploring possible
options, evaluating their performance, and making decisions
Acting• … to manipulate and move around in the real/virtual world
Supporting• Validation/Verification, Security, Ethics/Social Aspects, SW/HW-Platform
Applications
DataScience
AI: Structure and Terminology
AI
Learning(Machine Learning)
Reasoning
Deep
LearningSVM …
Technologies
Applied AI Communities
Vision/
GraphicsLanguage Robotics
Rule-Based
Systems
Statistical
Reasoning…
Application Areas
Interaction Ethics …
Automotive Health Industrie4.0 Sciences Public Sector …
What is AI?
• AI Research: Three distinct points of view
AI as a Tool (what we have done successfully for 30+ years)• Enabling intelligent, supportive, and adaptive technology
• Systems aware of their environment and reacting accordingly
• AI should be perceived and handled as a tool!
• Humans decide and are responsible how these tools are used!
AI for Cognitive Sciences (we should do more of this)• Understanding the human brain and cognition
• Helps to better understand how we humans ”work”
AI for Artificial Humans (we should do NONE of this)• A non-goal for our and almost all other AI research
• Main cause of fear in the general public (sci-fi movies, etc.)
State of AI
• Amazing progress in recent years in all of AI Most visible due to Deep Neural Networks (DNNs)
Supported by more data, better HW, improved algorithms
• Success stories Perception: vision, speech, …
Game playing: Chess, Go, simple video games, …
Some complex tasks: translation, autonomous driving, …
• Challenges Where are the limits of AI? How to avoid the hype cycle?
Understanding and explanation of what is learned by AI
Modularity, integration with classical AI techniques
Validation and verification methods – building trust!
Example: Learning to Play Go
From Scratch via Simulations
• AlphaGo Zero [Nature, Oct´17]
Given: Rules + Deep-Learning + Reinforcement-Learning
Learning by simulation (here: playing against itself)
• But: In reality „rules“ are complex AND unknown!
Simultaneously learn rules, simulate, train & validate AIs
Quality of Gameplay
Autonomous Systems:
The Problem
• Our World is extremely complex Geometry, Appearance, Motion, Weather, Environment, …
• Systems must make accurate and reliable decisions Especially in Critical Situations
Increasingly making use of (deep) machine learning
• Learning of critical situations is essentially impossible Often little (good) data even for “normal” situations
Critical situations rarely happen in reality – per definition!
Extremely high-dimensional models
Goal: Scalable Learning from synthetic input data Continuous benchmarking & validation (“Virtual Crash-Test“)
Autonomous Driving:
The Problem
Learning from
real data
Typical situations
with high probability
Critical situations
with low probability
Learning from
synthetic data
Learning for
Long-Tail Distributions
Goal: Validation of correct behavior by
covering high variability of input data
Reality
• Training and Validation in Reality
E.g. driving millions of miles to gather data
Difficult, costly, and non-scalable
Reality
Car
Digital Reality
• Training and Validation in the Digital Reality
Arbitrarily scalable (given the right platform)
But: Where to get the models and the training data from?
Reality
Digital
Reality
CarCar
Teil-ModelleTeil-ModelleTeil-Modelle
Reality
Digital
Reality
Partial
Models
(Rules)
Modeling &
Learning
CarCar
Model L
earn
ing
Digital Reality: Learning
Geometry
Material
Behavior
Motion
Environment
…
SzenarienSzenarien
Teil-ModelleTeil-ModelleTeil-Modelle
Reality
Digital
Reality
Partial
Models
(Rules)
Relevant
Scenarios
Concrete
Instances of
Scenarios
Configuration &
Learning
Modeling &
Learning
Coverage of Variability via
Directed Search
CarCar
Model L
earn
ing
Reasoning
Digital Reality: Reasoning
SzenarienSzenarien
Teil-ModelleTeil-ModelleTeil-Modelle
Reality
Digital
Reality
Partial
Models
(Rules)
Simulation/
Rendering
Relevant
Scenarios
Concrete
Instances of
Scenarios
Configuration &
Learning
Modeling &
Learning
Synthetic Sensor Data,
Labels, …
Adaptation to the
Simulated Environment
(e.g. used sensors)
CarCar
Model L
earn
ing
Sim
ula
tion &
Learn
ing
Reasoning
Digital Reality: Simulation
Coverage of Variability via
Directed Search
SzenarienSzenarien
Teil-ModelleTeil-ModelleTeil-Modelle
Reality
Digital
Reality
Partial
Models
(Rules)
Simulation/
Rendering
Relevant
Scenarios
Concrete
Instances of
Scenarios
Configuration &
Learning
Modeling &
Learning
Synthetic Sensor Data,
Labels, …
Adaptation to the
Simulated Environment
(e.g. used sensors)
Car
Continuous
Validation &
Adaptation
Car
Model L
earn
ing
Reasoning
Validation / Adaptation / Certification
Digital Reality: Validation/Adaptation
Sim
ula
tion &
Learn
ing
Coverage of Variability via
Directed Search
Coverage of Variability via
Directed Search
SzenarienSzenarien
Teil-ModelleTeil-ModelleTeil-Modelle
Reality
Digital
Reality
Partial
Models
(Rules)
Simulation/
Rendering
Relevant
Scenarios
Concrete
Instances of
Scenarios
Configuration &
Learning
Modeling &
Learning
Synthetic Sensor Data,
Labels, …
Adaptation to the
Simulated Environment
(e.g. used sensors)
Car
Continuous
Validation &
Adaptation
Car
Model L
earn
ing
Reasoning
Validation / Adaptation / Certification
Digital Reality: Continuous Learning
Continuous Learning Loop for AI
Not just for Automated Driving:
Works for any AI System
where we can model its interaction with
the environment
Sim
ula
tion &
Learn
ing
Challenges: Better Models of
the World (e.g. Pedestrians)
• Long history in motion research (>40 years)
E.g. Gunnar Johansson's Point Light Walkers (1974)
Significant interdisciplinary research (e.g. psychology)
• Humans can easily discriminate different styles
E.g. gender, age, weight, mood, ...
Based on minimal information
• Can we teach machines the same?
Detect if pedestrian will cross the street
Parameterized motion model & style transfer
Predictive models & physical limits
Challenge: Better Simulation
(e.g. Radar Rendering)
• Key Differences
Longer wavelength: Geometric optics (rays) not sufficient
Need for some wave optics
• Diffraction at rough surfaces and edges
• Need for polarization & resonance
Highly different goals
• Optical: Focus on diffuse effects (+ some highlights, reflections, etc.)
• Radar: Focus on specular transport only (i.e. caustic paths)
• Recent Work on Caustics [Grittmann et al., EGSR’18]
Identifying “useful” specular paths (using VCM)
Guides samples to visible specular effects (e.g. indirect radar echos)
• Combining research on rendering and radar technology
CLAIRE
CONFEDERATION OF LABORATORIES FOR ARTIFICIAL INTELLIGENCE RESEARCH IN EUROPE
Excellence across all of AI. For all of Europe.
With a Human-Centred Focus.
claire-ai.org
Take-Aways
• Digital Reality as a fundamental tool in AI Modeling and simulation even in complex environments Learning and reasoning via feedback loop (e.g. RL) Key element for training & validation of future AI systems
• Continuous Learning Loop using Synthetic and Real Data Loop of model learning, simulation, training, and validation Validation is required to establish trust in AI systems
• Big Challenges Ahead Many promising partial results already – but largely islands Requires closer collaboration of industry & academia CLAIRE: Towards large-scale European initiative
AI: A Central Component of our Future for Many Years