Architektur von Big Data Lösungen

BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENF HAMBURG KOPENHAGEN LAUSANNE MÜNCHEN STUTTGART WIEN ZÜRICH

Architektur von Big Data LösungenGuido Schmutz ([email protected])

@gschmutz

Guido Schmutz

Working for Trivadis for more than 20 yearsOracle ACE Director for Fusion Middleware and SOACo-Author of different booksConsultant, Trainer, Software Architect for Java, SOA & Big Data / Fast DataMember of Trivadis Architecture BoardTechnology Manager @ Trivadis

More than 30 years of software development experience

Contact: [email protected]: http://guidoschmutz.wordpress.comSlideshare: http://www.slideshare.net/gschmutzTwitter: gschmutz

2 Architektur von Big Data Lösungen

Agenda

1. Introduction2. Big Data Reference Architectures

• Traditional Big Data• Event / Stream-Processing• Lambda Architecture• Kappa Architecture• Unified Architecture• Microservices Architecture

3. Big Data Ecosystem – many choices sorted!


Introduction


Big Data Definition (4 Vs)

+Timetoaction?– BigData+Real-Time=StreamProcessing

CharacteristicsofBigData:ItsVolume,VelocityandVarietyincombination

Reliable Data Ingestion in Big Data/IoT

How to do Big Data? Why is a structuring / architecture important?


Why talk about Big Data Architectures?

Choosing the right architecture is key for any (big data) project

Big Data is still quite a rather young field and therefore a “moving target”

no standard architectures available which have been used for years

In the past years, some architectures and best practices have evolved

Know your use cases before choosing your architecture / technologies

To have a reference architecture in place helps in choosing the right/matching technologies


Important Properties for choosing (Big) Data Architecture

Latency

Keep raw and un-interpreted data “forever” ?

Volume, Velocity, Variety, Veracity

Ad-Hoc Query Capabilities needed ?

Robustness & Fault Tolerance

Scalability

…


Big Data Reference Architectures -Traditional Big Data


“Traditional Architecture” for Big Data

DataIngestion (Analytical)DataProcessing

DataSources

DataConsumer

Reports

Service

AnalyticTools

AlertingTools

Content

RDBMS

Social

ERP

Logfiles

Sensor

Machine

Batchcompute

PushingIngestion ResultStore

QueryEngine

ComputedInformation

RawData(Reservoir)

=DatainMotion =DataatRest

PullingIngestion

Channel


“Traditional Architecture” for Big Data – Hadoop Technology Mapping


DataSources

DataConsumer

Reports

Service

AnalyticTools

AlertingTools

Content

RDBMS

Social

ERP

Logfiles

Sensor

Machine

Batchcompute


QueryEngine

ComputedInformation

RawData(Reservoir)


PullingIngestion

Channel


“Traditional Architecture” for Big Data – Spark Technology Mapping


DataSources

DataConsumer

Reports

Service

AnalyticTools

AlertingTools

Content

RDBMS

Social

ERP

Logfiles

Sensor

Machine

Batchcompute


QueryEngine

ComputedInformation

RawData(Reservoir)


PullingIngestion

Channel


“Traditional Architecture” for Big Data – Feeding in High-Volume Event Streams


DataSources

DataConsumer

Reports

Service

AnalyticTools

AlertingTools

Content

RDBMS

Social

ERP

Logfiles

Sensor

Machine

Batchcompute


QueryEngine

ComputedInformation

RawData(Reservoir)


PullingIngestion

Channel

?

?


Traditional Architecture for Big Data

• Batch Processing - “Data at Rest”

• Not for low latency use cases• Responses are delivered “after the fact”• Maximum value of the identified situation is lost• Decision are made on old and stale data

• Spark Core is a faster alternative to Hadoop Map Reduce, but still Batch Processing

• Spark Ecosystems offers a lot of additional advanced analytic capabilities (machine learning, graph processing, …)


Big Data Reference Architectures –Event/Stream Processing


Event / Stream Processing – “Data in Motion”

“Data in motion”

Events are analyzed and processed in real-time as the arrive

Decisions are timely, contextual and based on fresh data

Decision latency is eliminated


Event / Stream Processing Architecture

DataIngestion

Batchcompute

DataSources

Channel

DataConsumer

Reports

Service

AnalyticTools

AlertingTools

Content

Logfiles

Social

RDBMS

ERP

Sensor

Machine

(Analytical)Real-TimeDataProcessing

Stream/EventProcessing

Messaging

ResultStore

=DatainMotion =DataatRest17 Architektur von Big Data Lösungen

Challenges for Ingesting Data

Multitude of sensors

Real-Time Streaming

Multiple Firmware versions

Bad Data from damaged sensors

Regulatory Constraints

Data Quality


Continuous Data Ingestion

DBSource

BigData

Log

StreamProcessing

IoT Sensor

EventHub

Topic

Topic

REST

Topic

IoT GW

CDCGW

Conn

ectCDC

DBSource

Log CDC Native

IoT Sensor

IoT Sensor

19

DataflowGW

Topic

Topic

Queue

MessageGW

Topic

DataflowGW

Dataflow

TopicRE

ST19FileSourceLog

Log

Log

Social

Native

Topic

Topic


Continuous Data Ingestion

DBSource

BigData

Log

StreamProcessing

IoT Sensor

EventHub

Topic

Topic

REST

Topic

IoT GW

CDCGW

Conn

ectCDC

DBSource

Log CDC Native

IoT Sensor

IoT Sensor

20

DataflowGW

Topic

Topic

Queue

MessageGW

Topic

DataflowGW

Dataflow

TopicRE

ST20FileSourceLog

Log

Log

Social

Native

Topic

Topic


DataIngestion (Analytical)Real-TimeDataProcessing

Event / Stream Processing Architecture – Open Source Technology Mapping

Batchcompute

DataSources

Channel

DataConsumer

Reports

Service

AnalyticTools

AlertingTools

Content

Logfiles

Social

RDBMS

ERP

Sensor

Machine


Messaging

ResultStore


DataIngestion (Analytical)Real-TimeDataProcessing

Event / Stream Processing Architecture – Oracle Technology Mapping

Batchcompute

DataSources

Channel

DataConsumer

Reports

Service

AnalyticTools

AlertingTools

Content

Logfiles

Social

RDBMS

ERP

Sensor

Machine


Messaging

ResultStore


Event / Stream Processing Architecture

The solution for low latency use cases

Process each event separately => low latency

Process events in micro-batches => increases latency but offers better reliability

Previously known as “Complex Event Processing”

Keep the data moving / Data in Motion instead of Data at Rest => raw events were not stored


Event / Stream Processing Architecture - Keep raw event data

DataIngestion

Batchcompute

DataSources

Channel

DataConsumer

Reports

Service

AnalyticTools

AlertingTools

Content

Logfiles

Social

RDBMS

ERP

Sensor

Machine


Stream/EventProcessingMessaging

ResultStore

(Analytical)BatchDataProcessing

RawData(Reservoir)


Big Data Reference Architectures -Lambda Architecture for Big Data


“Lambda Architecture” for Big Data

DataIngestion

(Analytical)BatchDataProcessing

Batchcompute

DataSources

Channel

DataConsumer

Reports

Service

AnalyticTools

AlertingTools

Content

RDBMS

Social

ERP

Logfiles

Sensor

Machine



Batchcompute

Messaging

ResultStore

QueryEngine

ResultStore

ComputedInformation

RawData(Reservoir)


PullingIngestion


Lambda Architecture for Big Data

Combines (Big) Data at Rest with (Fast) Data in Motion

Closes the gap from high-latency batch processing

Keeps the raw information forever

Makes it possible to rerun analytics operations on whole data set if necessary => because the old run had an error or => because we have found a better algorithm we want to apply

Have to implement functionality twice• Once for batch• Once for real-time streaming


Big Data Reference Architectures -„Kappa“ Architecture


“Kappa Architecture” for Big Data

DataIngestion

“RawDataReservoir”

Batchcompute

DataSources

Channel

DataConsumer

Reports

Service

AnalyticTools

AlertingTools

Content

RDBMS

Social

ERP

Logfiles

Sensor

Machine



Messaging

ResultStore

RawData(Reservoir)

ComputedInformation


Queryable State

Organizing NoSQL Data Stores – Different Types KeyValueStore

Wide-columnstore

Documentstore

Graphstore

Key ValueK1 V1K2 V2K3 V3

Document{k1:v1,k2:v2,k3:[v1,v2,v3]}

RowkeyCK1

RK1V1

CK2V2

CK3V3

CK4V4

……

CK1RK2V1

CK4V4

CK6V6

……

…………

CK3V3


Organizing NoSQL Data Stores – and the Products KeyValueStore

Wide-columnstore

Documentstore

Graphstore


Big Data Reference Architectures -„Unified“ Architecture


“Unified Architecture” for Big Data

DataIngestion

(Analytical)BatchDataProcessing(CalculateModelsofincomingdata)

Batchcompute

DataSources

Channel

DataConsumer

Reports

Service

AnalyticTools

AlertingTools

Content

RDBMS

Social

ERP

Logfiles

Sensor

Machine



Batchcompute

Messaging

ResultStore

ResultStore

ComputedInformation

RawData(Reservoir)


PredictionModels


Queryable State

Event Driven (Micro-) Service Architectures


MicroserviceMicroservice

MicroserviceMicroservice

Event-Driven (Micro-) Services Architecture

DataIngestion

“RawDataReservoir”

Batchcompute

DataSources

Channel

DataConsumer

Reports

Service

AnalyticTools

AlertingTools

Content

RDBMS

Social

ERP

Logfiles

Sensor

MachineMicroservice 2

Service

RawData(Reservoir)

ComputedInformation


State

Batchcompute

Microservice 1

Service State

API

ResultStore

Big Data Ecosystem – many choices sorted!


Building Blocks for (Big) Data ProcessingData

Acquisition

FormatFile System

Stream Processing

Batch SQL

Graph DBMS

Document DBMS

Relational DBMS

Visualization

IoT

Messaging

Analytics

OLAP DBMS

Query Federation

Table-Style DBMS

Key Value DBMS

Batch Processing

In-Memory


Big Data Ecosystem – many choices sorted!


Guido SchmutzTechnology Manager

[email protected]


Date post:	13-Apr-2017
Category:	Technology
Upload:	guido-schmutz
View:	61 times
Download:	0 times