Qualitätssicherung von Software (SWQS)
Prof. Dr. Holger Schlingloff
Humboldt-Universität zu Berlinund
Fraunhofer FOKUS
15.7.2014: Modellbasierter Test (Jaroslav Svacina)
Folie 3H. Schlingloff, Software-Qualitätssicherung
Specification-based Testing
• Constructing the test suite from the specification as opposed to constructing it from the implementation
(= code-based testing)
code-based testing cannot detect missing requirements specification-based testing cannot detect additional
(unspecified) features
specifiedbehaviour
implementedbehaviour
testedbehaviour
Folie 4H. Schlingloff, Software-Qualitätssicherung
Model-based Testing
• SUT is compared to a formal model (requirements)
• Automation of test case design• Early validation of requirements• Traceability• Easy maintenance of test suites for regression
testing
Folie 5H. Schlingloff, Software-Qualitätssicherung
Model-based Design and Model-based Testing
• Often: same syntax, different pragmatics e.g. test cases can be formulated in Java e.g. system spec can be formulated with LTL
Requirements
System Spec Test Spec
System Model
System Impl. Test Cases
Test Model
Folie 6H. Schlingloff, Software-Qualitätssicherung
System models vs. Test models
• Models can help in the development of complex systems
• The more concrete the formalism, the closer it is to an implementation executable code may be generated from state
diagrams We might add additional information such as timing,
communication, variables and such.
• Test model as opposed to implementation model describes properties of the targeted system not aiming at a complete description of the system not aiming at the generation of executable code
Folie 7H. Schlingloff, Software-Qualitätssicherung
Model-based Testing
Folie 8H. Schlingloff, Software-Qualitätssicherung
Model-based Testing
Folie 9H. Schlingloff, Software-Qualitätssicherung
Modelling
• Describe system requirements for test generation
• High abstraction level
Folie 11H. Schlingloff, Software-Qualitätssicherung
Modelling Notations
•Pre/Post (state based) notations State variables, operations
(pre-/postcondition) B, SpecExplorer (c#), UML OCL
•Transition based notations Focus on describing the transitions between
states of the system UML State Machines, Simulink State Flow
•…
Folie 12H. Schlingloff, Software-Qualitätssicherung
Example: A Kitchen Toaster
•A toaster what is the
technical process? what are the states,
events and signals of the (technical) process?
what are the boundaries of the system? which information processing is to be done? what are the interfaces between technical
system and information processing component?
Folie 13H. Schlingloff, Software-Qualitätssicherung
Modeling the Toaster
• User Interfaces: turning knob, side lever, stop button
• Internals: heating element, retainer latch
• Extra: defrost button
• First approach: timing is neglected (timer event)
• Advanced approach: timing depends on various parameters
Folie 14H. Schlingloff, Software-Qualitätssicherung
Toaster – Simple State Machine
Folie 15H. Schlingloff, Software-Qualitätssicherung
Toaster – Hierarchical Design
Folie 16H. Schlingloff, Software-Qualitätssicherung
Toaster – with Variables
Folie 17H. Schlingloff, Software-Qualitätssicherung
Test Generation
Folie 18H. Schlingloff, Software-Qualitätssicherung
Test Generation
•Algorithm Graph traversal yields abstract test cases
- Dijkstra shortest path algorithm
- Depth-first and breadth-first search
Evolutionary algorithm
•Model Transformation
•Static analysis for input parameters Partition Testing Classification tree method Boundary value analysis
Folie 19H. Schlingloff, Software-Qualitätssicherung
All-States with Dijkstra and DFS
Folie 20H. Schlingloff, Software-Qualitätssicherung
Evolutionary Test Generation
Folie 22H. Schlingloff, Software-Qualitätssicherung
Test Selection
Folie 23H. Schlingloff, Software-Qualitätssicherung
Test Generation from State Machines
•Define a test case to be any execution path•How to generate such paths? How many
paths to generate? When to stop testing? Coverage criteria e.g.
all-states all-transitions all-transition-pairs all-n-paths
•Test goal: one particular item to be covered
Folie 25H. Schlingloff, Software-Qualitätssicherung
Tests for State Machines
• For a state machine, a test case is just a finite sequence of external triggers and actions
• A test goal is a particular entity of the state machine (region, pseudostate, transition, n-path, …; for each test and goal it is defined whether the test reaches this goal
• The coverage of a test suite is the percentage of reached test goals the coverage can either be measured during the
execution of a test suite, or statically before execution
Folie 26H. Schlingloff, Software-Qualitätssicherung
Coverage for State Machines
• common coverage criteria for UML state machines all-states all-transitions all configurations: all combinations of parallel substates n-transition-coverage means all reachable transition
sequences of length n are covered (esp.: pairs) All-loop-free-paths All-n-loop-paths decision coverage, condition coverage, MC/DC, ...
• all-requirements
Folie 27H. Schlingloff, Software-Qualitätssicherung
Szenarien als Testselektionskriterium• Schwierige, nicht ausreichende Möglichkeiten zur
Beschreibung der Testselektionskriterien
• Einflussnahme der Domänenexperten auf den Testselektionsprozess
Folie 28H. Schlingloff, Software-Qualitätssicherung
Szenarien als Testselektionskriterium
Folie 29H. Schlingloff, Software-Qualitätssicherung
Automated Test Generation Tools
• More than a dozen commercial and experimental research tools available
• Usually quite costly (>10K€ per license)
• Mostly from UML State Machines
• Comparison e.g. by mutation analysis
Folie 30H. Schlingloff, Software-Qualitätssicherung
Conformiq
Folie 31H. Schlingloff, Software-Qualitätssicherung
ParTeG – Partition Test Generator
• Since we didn’t like the pricing, and wanted to experiment with different technologies, a Ph.D. student built his own… http://parteg.sourceforge.net/
• UML class & state transition diagrams, connected by OCL • Plugin for Eclipse, supports XMI import / export