© Fraunhofer IESE
1
CU! - Continuous Usability mit offline-fähigen Apps
Susanne BraunMatthias Naab
08.03.2016JavaLand 2016
© Fraunhofer IESE
2
Fraunhofer IESE The research institution for software and systems engineering methods
Founded in 1996, headquartered in Kaiserslautern
approx. 240 employees
Our solutions can be scaled flexibly and are suitable for companies of any size
Our most important business areas:
Automotive and Transportation Systems
Automation and Plant Engineering
Health Care
Information Systems
Energy Management
E-Government
© Fraunhofer IESE
3
© Fraunhofer IESE
3
About ACES
The Fraunhofer Approach for Modeling Software and System Architectures
Compiled Best Practices from literature, scaled and tailored for effective architecting in practice
More than 20 years of architecting experiences across domains:Embedded Systems, Information Systems, Smart Ecosystems
ACES –Architecture-Centric Engineering Solutions
© Fraunhofer IESE
4
© Fraunhofer IESE
4
Always Connected
Quelle: http://theoatmeal.com/comics/smartphone
© Fraunhofer IESE
5
© Fraunhofer IESE
6
© Fraunhofer IESE
6
User Experience Expectations of Mobile Users
Quelle: http://theoatmeal.com/comics/smartphone
© Fraunhofer IESE
7
What Mobile Users expect from high quality Apps
Continuous Availability of Mobile Services and Data
Responsive and Polished UIs
Economic Consumption of Battery & Data Plans
Apps need to be offline usable and ship with smart & economic
data synchronization
© Fraunhofer IESE
8
© Fraunhofer IESE
8
Mobile Data Synchronization Classes
One User One Device
One User Multiple Devices
Multiple Users Multiple Devices
Complexity of the Mobile Sync Solution
© Fraunhofer IESE
9
© Fraunhofer IESE
9
Write Conflicts & Lost-Updates
Sync
Sync Read(c) = 100
Write(c) = c+20= 120
Read(c) = 100
Write(c) = c+50= 150
Sync
Sync
Update c += 50 lost
Conflict resolved with Last-Writer-
Wins c = 120
t
Distributed Counter c = 100
c = 120
c = 170
Sync c = 120
Sync c = 150
© Fraunhofer IESE
10
© Fraunhofer IESE
10
ACID & Replication
„Update anywhere-anytime-anyway transactionalreplication has unstable behavior as the workload scales up: a
ten-fold increase in nodes and traffic gives a thousand foldincrease in deadlocks or reconciliations.”
Jim Gray, 1996*
* http://research.microsoft.com/apps/pubs/default.aspx?id=68247
© Fraunhofer IESE
11
© Fraunhofer IESE
11
CAP Theorem*
C onsistency
A vailability PNetworkartition Tolerance
App offering full offline operations
* Eric Brewer
© Fraunhofer IESE
12
© Fraunhofer IESE
12
How to achieve Convergence?
Anti-Entropy
Exchange data versions or update operations between replicas
Reconciliation
Conflict Detection
Conflict Resolution
Highly Application Dependent
© Fraunhofer IESE
13
© Fraunhofer IESE
13
Approaching Mobile Data Synchronization
Mobile Data Synchronization
Data Shipping
Write Set Shipping
…
Operation Shipping
CommutativeOperations Design
Operational Transformation
CQRS Pattern
…
© Fraunhofer IESE
14
Couchbase Mobile
© Fraunhofer IESE
15
© Fraunhofer IESE
15
Couchbase Mobile
Mobile Devicesrunning
Couchbase Lite
Channels
Data Center
Sync Gateway
Couchbase Server / Cluster
Bucket
Bucket
Node Node Node
BucketvBucket
© Fraunhofer IESE
16
© Fraunhofer IESE
16
Couchbase Mobile Channels
User Documents
D1
D2
D3
D4
Channels
© Fraunhofer IESE
17
© Fraunhofer IESE
17
Couchbase Mobile Scalability
Sync Gateway
Couchbase Server / Cluster
Get Changes
Get ChangesChannel 2
Get ChangesChannel 1
[*, 42] D1
[Channel 1, 42] D1
[Channel 2, 42] D1
… …
Channel Changes View
Number ofRequests!
View Growth!
© Fraunhofer IESE
18
© Fraunhofer IESE
18
Couchbase Mobile Conflict Resolution
Automatic and deterministicdetermination of winner version
Without communication overhead
Winner:
Undeleted version with greatest depth in version graph
If more than one: lexigrophic comparison on revision
„Busiest Writer“ wins or arbitrary in worst case
Support for manual conflict resolution
Guarantee that conflicting versions areavailable
No Guarantee that all historic versions areavailabe
AutomaticWinner
ConflictingVersions
© Fraunhofer IESE
19
Zumero
© Fraunhofer IESE
20
© Fraunhofer IESE
20
Zumero
Mobile Devicesrunning SQLite
SQL Conditions
Microsoft Windows / IIS-Plattform
ZumeroServer
MSSQL-Server
App Tables Zumero MetadataTables, Trigger
EntityEntity
© Fraunhofer IESE
21
© Fraunhofer IESE
21
Zumero Conflict Resolution
Support for automatic merge of updateson different entity column values
Database Constraint Violations cannot beautomatically merged
Strategies for conflicting updates on same entity column:
Last-Writer-Wins
First-Writer-Wins
Support for manual conflict resolution
Guarantee that conflicting entities areavailable
Historic entity versions are maintained in metadata tables (unofficial & undocumented)
AutomaticMerge
ConflictResolution Strategy
© Fraunhofer IESE
22
Products & Frameworks
DB Replication
Couchbase Mobile
Zumero
Frameworks
Amazon Cognito
Android Sync Adapter Framework
© Fraunhofer IESE
23
My John Deere Mobile Case Study
© Fraunhofer IESE
24
Important Design Decisions
Consistency Guarantees (What are the desired Consistency Guarantees? How are theyachieved?)*
Update Format (Exchange of data items vs. exchange of update operations?)*
Change Tracking (How to track updates that need to be propagated?)*
Metadata (What metadata is stored and communicated about replicated items?)*
Sync State (What state is maintained at a device for each synchronization partnership?)*
Change Enumeration (How do devices determine which updates still need to bepropagated to which other devices?)*
Communication (What transport protocols are used?)*
Ordering (How to decide on the order in which received upates should be applied?)*
Filtering (How are the contents of partial replicas specified and managed?)*
Conflicts (How are conflicting updates detected and resolved?)*
Synchronization Time (How to decide on the optimal point in time forsynchronization? Impact on battery consumption, data traffic and probability for conflicts.)
* Source: Replicated Data Management for Mobile Computing, Douglas Terry, p. 27
© Fraunhofer IESE
25
© Fraunhofer IESE
25
MJDM Sync Solution Architecture
iOS / Android Devicesrunning
SQLite with ORM
Apache Tomcat
Sync REST
Service
MySQL: Current Entity Data
Flat Objects
Flat Objects
Flat Objects
Spring Context
JPA / Hibernate
Cassandra Java
Driver
Cassandra: Time-Series Data (Historic Entities, Events)
EntityID:YearTimestamp Timestamp Timestamp…
EntityID:YearTimestamp Timestamp Timestamp…
HistoryService
© Fraunhofer IESE
26
© Fraunhofer IESE
26
Change Tracking & Change Enumeration
JPA Entity Base Class• id: UUID• syncState: updated | inserted | deleted | insync• revNr: numeric version
Globally uniqueidentifier
• Track and enumeratechanges on the client
• Mark entities as deletedon client and server
• Version number of theentity
• Managed by Sync Service• Enumerate changes on
the server• Conflict Detection
© Fraunhofer IESE
27
© Fraunhofer IESE
27
Conflict Detection
Client must keep and ship revNr of each entity in the Write Set
Server preprocess Write Set of Client
Server iterates over all entities in Write Set and
Checks if revNr of Client < revNr of Server
Conflict
Server MySQL DB
ID REV …
11
8
3f49f674-….
6d47d45-….
Entity Id: 3f49f674-….revNr: 10
Entity Id: 6d47d456-….revNr: 8
Client Write Set
© Fraunhofer IESE
28
© Fraunhofer IESE
28
Conflict Resolution
Entity Id: 3f49f674-….revNr: 10c
Dirty Client Entity
Entity Id: 3f49f674-….revNr: 10
Entity in version 10
Client Write Set
Cassandra
MySQLEntity
Id: 3f49f674-….revNr: 12c
Merged Entity
Field Changes• …• ….
Field-by-field
comparisonvia
Reflection
Apply to currententity
Entity Id: 3f49f674-….revNr: 11c
Current Entity
• Three-Way-Merge• Last-Writer-Wins (conflicts on the same entity field)
© Fraunhofer IESE
29
© Fraunhofer IESE
29
Two-Phase Protocol
Sync REST
Service
PUT JSON Write Set• Conflict detection• Conflict resolution• Increase sync counter• Update revision number of entities• Update syncState (insync | deleted)• Persist entities• Insert historic entity versions
OK
Sync REST
Service
GET /42• Calculate Write Set • Entities with revision number > 42• Entities either deleted or insync
JSON Write Set
• All entities „insync“• syncState deleted entites are now deleted on client
Atomic
Atomic
© Fraunhofer IESE
30
Caution: Avoid Disadvantageous Data Modelling
Field
Ag Plan
Ag Plans ofField
1
*
Field
Ag Plan
Field of AgPlan
1
*
• Insertion of Ag Plan• = Collection update in Field
• New version of Field• Can conflict with other
Field updates.
• Insertion of Ag Plan• = Insertion of Ag Plan!• -> Cannot conflict with any
other update!
© Fraunhofer IESE
31
Lessons Learned
© Fraunhofer IESE
32
Impact on Data Modelling
© Fraunhofer IESE
33
Impact on Code Complexity
© Fraunhofer IESE
34
The Return of the Fat Clients
in conjunction with Data-CentricArchitectures…
© Fraunhofer IESE
35
Data Validation can become a Show Stopper
© Fraunhofer IESE
36
Write Set Shipping Cries for Generative Approaches
© Fraunhofer IESE
37
Authorization Changes Usually Require a Full Sync
© Fraunhofer IESE
38
Full Sync Usually Requires Special Optimization
© Fraunhofer IESE
39
Historic Data is Valuable
© Fraunhofer IESE
40
Design for Economic ResourceConsumption
© Fraunhofer IESE
41
Design for Great User Experience
is not trivial…
© Fraunhofer IESE
42
“By the end of 2014, the number of mobile-connected devices will exceed the number of
people on earth”1
“Last year’s mobile data traffic was nearly 30 times the s ize of the entire global Internet in 2000”1
Questions?“The Future of Enterprise Applications Is Mobility”2