Post on 04-Jul-2020
transcript
Vorstellung CERT.atIncident Handling Automation with intelmq
Sebastian Wagner wagner@cert.at
Rollen von CERTs
In Firmen: IT Sicherheitsverantwortlicher (+Team) Product CERTs
In ISPs/IT-Dienstleister: Klassisches abuse-Handling Network / Server Security
Übergreifend: Sektor-Spezifische CERTs (Finanz, ICS, …) MilCERT Government CERT Nationales CERT
Sebastian Wagner wagner@cert.at
CERT.at: Nationales CERT
Projekt von „CERT of last resort“ Informationsdrehscheibe
Innerhalb Österreichs Internationaler Ansprechpartner von Österreich
„Fire Department“, „CDC“, „Public Health“ Verteilung von Hinweisen auf Sicherheitsprobleme Awareness / Warnungen
Incident Response Wem kann man in welchem Ausmaß helfen? Keine Konkurrenz zur Sicherheitsbranche
Sebastian Wagner wagner@cert.at
Nationales CERT
Interessante Position Zuständig für ganz Österreich
Keine Sensoren in den Netzen Keine Meldepflicht an uns
Wir sind keine Behörde Erst recht keine Strafverfolgung Keine formalen Befugnisse
Keine rein technische Rolle Koordination, Vermittlung, … Vertrauen und Kontakte sind unser Hauptkapital
Es geht überraschend viel
Sebastian Wagner wagner@cert.at
Rolle: Government CERT
CERT für Einrichtungen der öffentlichen Verwaltung Ministerien Länder Städte Gemeinden … (die Abgrenzung ist oft nicht einfach)
Beratend Bundeskanzleramt + nic.at (Public / Private Partnership)
CERT.at stellt technischen Teil des GovCERTsSebastian Wagner wagner@cert.at
IntelMQ - a KISS incident handling automationproject (IHAP)
L. Aaron Kaplan kaplan@cert.at Sebastian Wagnerwagner@cert.at Tomas Lima tomas.lima@cert.pt
2015/10/21
L. Aaron Kaplan kaplan@cert.at, Sebastian Wagner wagner@cert.at, Tomas Lima tomas.lima@cert.pt
IntelMQ - a KISS incident handling automation project (IHAP)
Overview
1 Intro to IntelMQ
2 History
3 Background
4 Architecture and data flow
5 Installation
6 Writing a bot
7 Next steps/future
8 Links
L. Aaron Kaplan kaplan@cert.at, Sebastian Wagner wagner@cert.at, Tomas Lima tomas.lima@cert.pt
IntelMQ - a KISS incident handling automation project (IHAP)
Intro to IntelMQ
IHAP = Incident Handling Automation Project. Our overallproject name.
A project of multiple national CERTs (Trusted Introducer)
IntelMQ = Threat Intel feeds + Message Queueing system.A concrete tool.
Idea and architecture inspired by Abusehelper
Data flow oriented toolkit to:
Automatically collect & handle events/incidents
Process and enrich these events
Send them to some output, automatic actions
L. Aaron Kaplan kaplan@cert.at, Sebastian Wagner wagner@cert.at, Tomas Lima tomas.lima@cert.pt
IntelMQ - a KISS incident handling automation project (IHAP)
Intro to IntelMQ (2)
Based on message queues (“MQ“) – redis, RabbitMQ, zmq
Fast
Very easy to extend
GUI interface to create pipelines / modify dataflow(“intelmq-manager”)
L. Aaron Kaplan kaplan@cert.at, Sebastian Wagner wagner@cert.at, Tomas Lima tomas.lima@cert.pt
IntelMQ - a KISS incident handling automation project (IHAP)
History
CERT.at started with Abusehelper (open source)
Our problem with AH: everything is co-routine orientated.That‘s hard to debug. Many CERTs either:
1 give up or
2 if they have the money buy Abuse-SA (commercial, closedsource). For CERT.at it was too expensive so we needed tostay with the standard open source version. But that camewithout support and it is too complex for us.
L. Aaron Kaplan kaplan@cert.at, Sebastian Wagner wagner@cert.at, Tomas Lima tomas.lima@cert.pt
IntelMQ - a KISS incident handling automation project (IHAP)
The Abusehelper Framework
General remarks
Strengths:
nice flow-oriented architecture
lots of existing bots to fetch data
loosely de-coupled: in theory easy to write new “bots” andextend Abusehelper
open source
Issues/Weaknesses:
code complexity. Are you a python guru?
Getting code upstream to maintainer is hard
hard to understand the dataflow
resource-hog => how to improve on this?
no standard way to include into ticket systems likeRTIR/OTRS
L. Aaron Kaplan kaplan@cert.at, Sebastian Wagner wagner@cert.at, Tomas Lima tomas.lima@cert.pt
IntelMQ - a KISS incident handling automation project (IHAP)
The Abusehelper Framework
L. Aaron Kaplan kaplan@cert.at, Sebastian Wagner wagner@cert.at, Tomas Lima tomas.lima@cert.pt
IntelMQ - a KISS incident handling automation project (IHAP)
Alternatives to Abusehelper?
Watch these:
Megatron: open source, Java. Aware of two CERTs using ithttps://github.com/cert-se/megatron-java
n6: CERT.pl http://n6.cert.pl/
CIF: USA http://csirtgadgets.org/
Warden: https://wardenw.cesnet.cz/
overview: https://www.cert.pl/PDF/MP-IST-111-18.pdf
L. Aaron Kaplan kaplan@cert.at, Sebastian Wagner wagner@cert.at, Tomas Lima tomas.lima@cert.pt
IntelMQ - a KISS incident handling automation project (IHAP)
Requirements analysis after the Heraklion meeting 5/2014
Reduce the complexity of system administration
Reduce the complexity of writing new bots for new data feeds
Reduce the probability of events lost in all process withpersistence functionality (even system crash)
Use and improve the existing “Data Harmonization Ontology“(= Abusehelper internal key-value standard)
Use JSON format for all messages
Integration of the existing tools (n6, AbuseHelper, CIF)
Provide easy way to store data into Log Collectors likeElasticSearch, Splunk and DBs
L. Aaron Kaplan kaplan@cert.at, Sebastian Wagner wagner@cert.at, Tomas Lima tomas.lima@cert.pt
IntelMQ - a KISS incident handling automation project (IHAP)
Summer sprint 2014
IntelMQ beta 1 is the result of a sprint July-˜Oct 2014.Persons: Tomas, Mauro, Aaron, Cosmin, . . .
https://github.com/certtools/intelmq
Ideas:
KISS! (Keep it simple stupid)
Very similar architecture as AH, just more modern tools
Message Queues (redis, amq, zmq)
Goal: it takes 15 minutes till 1d to create a new bot (withoutprior knowledge!)
Open Source for ever – no separate commercial version
Python != config language! We want a simple config (GUI!)
Connectable with n6, AH, CIF, syslog, Elastic Search, Splunk,..
L. Aaron Kaplan kaplan@cert.at, Sebastian Wagner wagner@cert.at, Tomas Lima tomas.lima@cert.pt
IntelMQ - a KISS incident handling automation project (IHAP)
IntelMQ @ hack.lu 2014
Very first public presentation and open source version
Test with Fyodor (Taiwan Uni): 15 minutes explanation ofcode + the next morning he had a hpfeeds bot . It is simple.
L. Aaron Kaplan kaplan@cert.at, Sebastian Wagner wagner@cert.at, Tomas Lima tomas.lima@cert.pt
IntelMQ - a KISS incident handling automation project (IHAP)
IntelMQ components
individual and specialized bots
Config files: JSON:
runtime.conf = runtime parameters of bots
startup .conf = which bots to start
BOTS = templates of all bots
Pipeline.conf - describes how bots are connected
Redis, zmq, RabbitMQ or *-MQ as message queue
Lib/{bot.py, pipeline.py, message.py}Web-GUI: IntelMQ-Manager: JS + CSS + AJAX
Outputs: Elastic Search or Postgresql or syslog . . . $foo
L. Aaron Kaplan kaplan@cert.at, Sebastian Wagner wagner@cert.at, Tomas Lima tomas.lima@cert.pt
IntelMQ - a KISS incident handling automation project (IHAP)
IntelMQ dataflow: bots
L. Aaron Kaplan kaplan@cert.at, Sebastian Wagner wagner@cert.at, Tomas Lima tomas.lima@cert.pt
IntelMQ - a KISS incident handling automation project (IHAP)
IntelMQ manager
L. Aaron Kaplan kaplan@cert.at, Sebastian Wagner wagner@cert.at, Tomas Lima tomas.lima@cert.pt
IntelMQ - a KISS incident handling automation project (IHAP)
Examples of expert bots
ASN lookup
abuse contact
whois
deduplication and filtering
geographic data
DNS lookups (A, PTR Records)
L. Aaron Kaplan kaplan@cert.at, Sebastian Wagner wagner@cert.at, Tomas Lima tomas.lima@cert.pt
IntelMQ - a KISS incident handling automation project (IHAP)
Installation
apt−g e t i n s t a l l python3apt−g e t i n s t a l l g i t b u i l d−e s s e n t i a l \
l i b c u r l 4 −g n u t l s−dev l i b f f i −devapt−g e t i n s t a l l python−dev python−p i p python−zmq \
python−p y c u r l python−o p e n s s l python−pyasn1apt−g e t i n s t a l l r e d i s−s e r v e r
L. Aaron Kaplan kaplan@cert.at, Sebastian Wagner wagner@cert.at, Tomas Lima tomas.lima@cert.pt
IntelMQ - a KISS incident handling automation project (IHAP)
Installation (2)
g i t c l o n e h t t p s : / / g i t h u b . com/ c e r t t o o l s / i n t e l m q . g i tcd i n t e l m qp i p 3 i n s t a l l −r REQUIREMENTSpython3 s e t u p . py i n s t a l lu s e r a d d −d / opt / i n t e l m q −U −s / b i n / bash i n t e l m qecho ’ export PATH=”$PATH :$HOME/ b i n ” ’ > \
/ opt / i n t e l m q / . p r o f i l echmod −R 0770 / opt / i n t e l m qchown −R i n t e l m q . i n t e l m q / opt / i n t e l m q
L. Aaron Kaplan kaplan@cert.at, Sebastian Wagner wagner@cert.at, Tomas Lima tomas.lima@cert.pt
IntelMQ - a KISS incident handling automation project (IHAP)
Writing a bot (1)
c l a s s TorExpertBot ( Bot ) :d a t a b a s e = set ( )def i n i t ( s e l f ) :
t ry :w i t h open ( s e l f . p a r a m e t e r s . d a t a b a s e ) as f p :
f o r l i n e i n f p :l i n e = l i n e . s t r i p ( )i f len ( l i n e ) == 0 or l i n e [ 0 ] == ”#” :
continue # sk i p comment and emptyl i n e = l i n e [ l i n e . f i n d ( ” [ ” )+1: l i n e . f i n d ( ” ] ” ) ]i p l i s t = l i n e . s p l i t ( ” , ” )f o r i p i n i p l i s t :
s e l f . d a t a b a s e . add ( i p . s t r i p ( ) )except I O E r r o r : [ . . . ]
L. Aaron Kaplan kaplan@cert.at, Sebastian Wagner wagner@cert.at, Tomas Lima tomas.lima@cert.pt
IntelMQ - a KISS incident handling automation project (IHAP)
Writing a bot (2)
def p r o c e s s ( s e l f ) :e v e n t = s e l f . r e c e i v e m e s s a g e ( )i f e v e n t i s None :
s e l f . acknowledge message ( )return
i f e v e n t . c o n t a i n s ( ’ s o u r c e . i p ’ ) :i f e v e n t . v a l u e ( ’ s o u r c e . i p ’ ) i n s e l f . d a t a b a s e :
e v e n t . add ( ’ s o u r c e . t o r n o d e ’ , True )s e l f . s end mess age ( e v e n t )s e l f . acknowledge message ( )
L. Aaron Kaplan kaplan@cert.at, Sebastian Wagner wagner@cert.at, Tomas Lima tomas.lima@cert.pt
IntelMQ - a KISS incident handling automation project (IHAP)
Next developments and project goals
more feeds/sources
reliability: more unittests, coverage, integration tests
bots working in parallel
more users
world domination
L. Aaron Kaplan kaplan@cert.at, Sebastian Wagner wagner@cert.at, Tomas Lima tomas.lima@cert.pt
IntelMQ - a KISS incident handling automation project (IHAP)
Links
https://github.com/certtools/intelmq
https://github.com/certtools/intelmq-manager
https://www.enisa.europa.eu/activities/cert/
support/incident-handling-automation
Mailing list for developers: https:
//tiss.trusted-introducer.org/mailman/listinfo/ihap
(for TI members)or ask kaplan@cert.at for subscription
L. Aaron Kaplan kaplan@cert.at, Sebastian Wagner wagner@cert.at, Tomas Lima tomas.lima@cert.pt
IntelMQ - a KISS incident handling automation project (IHAP)