Copyright©2016Splunk Inc.
TheTruthinessofWireData:
UsingSplunkStreamforPerformance
Monitoring
DavidJ.Cavuto
PrincipalProductManager,SplunkAppforStream
Disclaimer
Duringthecourseofthispresentation,wemaymakeforwardlookingstatementsregardingfuture
eventsortheexpectedperformanceofthecompany.Wecautionyouthatsuchstatementsreflectour
currentexpectationsandestimatesbasedonfactorscurrentlyknowntousandthatactualeventsor
resultscoulddiffermaterially.Forimportantfactorsthatmaycauseactualresultstodifferfromthose
containedinourforward-lookingstatements,pleasereviewourfilingswiththeSEC.Theforward-looking
statementsmadeinthethispresentationarebeingmadeasofthetimeanddateofitslivepresentation.
Ifreviewedafteritslivepresentation,thispresentationmaynotcontaincurrentoraccurateinformation.
Wedonotassumeanyobligationtoupdateanyforwardlookingstatementswemaymake.Inaddition,
anyinformationaboutourroadmapoutlinesourgeneralproductdirectionandissubjecttochangeat
anytimewithoutnotice.Itisforinformationalpurposesonlyandshallnot,beincorporatedintoany
contractorothercommitment.Splunkundertakesnoobligationeithertodevelopthefeaturesor
functionalitydescribedortoincludeanysuchfeatureorfunctionalityinafuturerelease.
2
CavutoBio
GeneralAwesomeness
3
Adorable
Headshot
PresentationOverview
1.
2.
3.
4.
5.
ProblemStatement
WhatisWireData?WhatisSplunkStream?
UsingWiredatafromStreaminITSI
AutomaticAnomalyDetectionusingWireDatainITSI
Wheretogofromhere
4
ProblemStatement
Applicationsmaynotaccuratelyreporttheirownperformance
accurately,foranumberofreasons
–
–
–
–
–
–
Heavilyloaded
Mistakenmeasurements
Developererror
ItDied!
Noaccesstodata(politically,organizationally)
They’relying!J
Insertquoteabouttrustingsomethingtomonitoritself
5
WhyWireDatawithSplunkStream?
UseWireDataprovidedbySplunkStreamtomonitorapplication
andnetworkperformance
DirectingestintoSplunk(noprops/transforms)makesitsimple
StreamisnotadedicatedAPM/NPMtool,buthasaspectsofboth
Inmostcases,Streamwilldoeverythingyouneedtoisolatefaults
andpinpointthetroublespot(s)
It’sFree!
6
What’sNetwork(Wire)Data?
tcpdump-qns0-A-rblah.pcap
20:57:47.368107IP205.188.159.57.25>67.23.28.65.42385:tcp480
0x0000:[email protected]
0x0010:43171c410019a59150fe18ca9da04681C..A....P.....F.
0x0020:801805a8848f00000101080affd49bb0................
0x0030:2e436bb93232302d726c792d64613033.Ck.220-rly-da03
0x0040:2e6d782e616f6c2e636f6d2045534d54.mx.aol.com.ESMT
0x0050:50206d61696c5f72656c61795f696e2dP.mail_relay_in0x0060:646130332e343b205468752c20303920da03.4;.Thu,.09.
0x0070:4a756c20323030392031363a35373a34Jul.2009.16:57:4
0x0080:37202d303430300d0a3232302d416d657.-0400..220-Ame
0x0090:72696361204f6e6c696e652028414f4crica.Online.(AOL
0x00a0:2920616e642069747320616666696c69).and.its.affili
0x00b0:6174656420636f6d70616e6965732064ated.companies.d
Machinedata
• Poly-structureddata
• Authoritativerecordofreal-time
andhistoricalcommunication
betweenmachinesandapplications
•
Network
TypicalCollectionPoint
EndUsers
7
Servers
SplunkAppforStream(6.6)
MetadataCollection
AggregationMode
– Collectsessentialelementsofthe
applicationconversation
– Eliminatesredundancyofduplicate
packetheaders
– Statisticsgeneratedatendpoint
– Similarto“statssum(x)”inSPL
FilteringatEndpoint
Out-of-BoxContent
LiveInterfaceCollectionOption
– Dashboardsforcommonprotocols
– Collectdirectlyonhosts
– AlsofromataporSPANport
DistributedForwarderMgt
– SimilartoSplunk UFmgt
– Allconfig centrallymanaged
– ForwarderGroups
EstimateMode
– DeployStreamwithoutcollecting
data(oraffectinglicense)
– Testdatavolume
8
HowwillWireDatahelpSolveProblem?
Wiredatarepresentscaptureoftrueconversationsbetween
endpoints
Ithasthe“omniscientview”ofwhatactuallytranspired
Theconversationscontainthedetailsabouteachtransaction,
includingthetimeofoccurrence
Lesschanceofinterference
– Intentional/Malicious
– Loadorresourcebased
9
ProtocolsParsedwithStream6.6
SimpleTransport
TCP
UDP
IP
Infrastructure
SNMP
DHCP
DNS
ICMP
FileTransfer
FTP
HTTP
FileService
NFS
SMB
Authentication
Diameter
LDAP
RADIUS
Email
IMAP
MAPI
POP3
SMTP
Database
MYSQL
Postgres
TDS(Sybase/MS-SQL)
TNS(OracleSQL*Net)
Messaging
AMQP
IRC
SMPP
XMPP
VoIP
SIP
RTP
10
ExamplesofWhat’sAvailableFromtheStreaming
Network/WireData
PerformanceMetrics
ApplicationData
BusinessData
RoundTripTime
POSTContent
ProductID
ClientRequest Time
AJAXData
CustomerID
ServerReplyTime
Section
Shopping Cart ID
ServerSendTime
Sub-Section
CartItems
TotalTimeTaken
PageTitle
Cart Values
BaseHTMLLoadTime
SessionCookie
Discounts
PageContent LoadTime
ProxiedIPAddress
OrderID
TotalPageLoadTime
ErrorMessage
Abandoned?
11
HowtoCollectandMonitorData
Streamhastwodeploymentarchitecturesandtwocollection
methodologies
Deployment:
– Out-of-band(stub)withtaporSPANport
– In-linedirectlyonmonitoredhost
Collection:
– TechnicalAdd-On(TA)withSplunkUniversalForwarder(UF)
– IndependentStreamForwarderusingHTTPEventCollector(HEC)
12
Deployment:DedicatedCollector
Internet
Firewall
TAPorSPAN
EndUsers
Splunk
Indexers
LinuxForwarder
Splunk_TA_Stream
SearchHead
13
Servers
Deployment:RunonServers
Internet
Firewall
EndUsers
PhysicalorVirtualServers
UniversalForwarder
Splunk_TA_stream
PhysicalDatacenter,
PublicorPrivateCloud
Splunk
Indexers
SearchHead
14
StreamForwarderOptions
MakesiteasytoaddStreamanywhereinyourenvironment
1.StreamTA
2.IndependentStreamForwarder
• Streamdeploysasamodularinputon
• Streamdeploysasastand-alonebinary
topofyourSplunkForwarders.
andcommunicatesviaHEC.
• Requires>=Splunk6.3.1
Splunk
Fwdr
HTTP/S
Splunk
Indexers
Splunk
Indexers
AnyLinuxHost
AnyHost
15
300+CommercialApplicationsDetectedJ
•
AdobeFlashPluginUpdateAdobeUpdateManagerAIMexpressAIMTransferAllMusic.com AltirisAmazonAdSystemAmazonCloudDriveAmazonGenericServicesAmazonMP3AmazonVideo
AmazonWebServices/Cloudfront CDNAndroidconnectivityManagerAol AOLInstantMessenger(formerlyOSCAR)AppleAirPlay AppleAirportAppleAirPrint AppleAppStoreAppleFaceTime
AppleGenericServicesAppleHTTPLiveStreamingAppleLocationAppleMapsAppleMusicApplePushNotificationServiceAppleSIRIAppleUpdateASProxy Atlassian BackgroundIntelligent
TransferServiceBaidu PlayerBaidu_wallet Baidu.com Bet365.comBitcoin clientBitTorrent Bittorrent AppsBitTorrent Bleep(akaBitTorrent Chat)BlackBerryLocateBlackBerryMessenger
BlackBerryMessengerAudioBlackBerryMessengerVideoBlackBerry.com BorderGatewayProtocolCARBONITECCProxy ChatON Chatroulette.com ChromeUpdateCiscoDiscoveryProtocolCisco
MeetingPlace CiscoNetflow CommonUnixPrinterSystemCracklecraigslistDataStreamInterfaceDB2Debian/UbuntuUpdateDropbox DownloadDropbox UploadDropbox.com eBay.com Edonkey
Evernote.com EverQuest - EverQuest IIFacebookFacebookMessengerFarmVille FindMyiPhoneFirefoxUpdateFlickrGenericRoutingEncapsulationGitHub GmailBasicGmaildriveGmailMobile
GNUnet GnutellaGoogleAccountsGoogleAnalyticsGoogleAppEngineGoogleCacheGoogleCalendarGoogleChatGoogleCloudMessagingGoogleCloudStorageGoogleDocuments(akaGoogle
Drive)GoogleEarthGoogleGenericGooglegroupsGoogleGStatic GoogleHangouts(formerlyGoogleTalk)GoogleMailGoogleMapsGooglePicasaGooglePlayMusic,Google PlayMusique Google
PlayStoreGooglePlusGoogleSafeBrowsingGoogleTagManagerGoogleToolbarGoogleTranslateGoogle.com GoToDevice RemoteAdministrationGoToMeeting OnlineMeetingGoToMyPC
RemoteAccessGPRSTunnelingProtocolGPRSTunnelingProtocolversion2Half-LifeHi5.comHighEntropyHotStandbyRouterProtocolHPPrinterJobLanguageHulu HyperText TransferProtocol
version2,HTTP/2I2PInvisibleInternetProjectIBMInformixIBMLotusSametime IBMSmartCloud IBMWebsphere MQiCloud (Apple)iHeartRADIO iMessage FileDownloadImgur.com Independant
ComputingArchitecture(Citrix)Instagram InternetGroupManagementProtocolInternetPrintingProtocolInternetSecurityAssociationandKeyManagementProtocolInternetSmallComputer
SystemsInterfaceiOS over-the-air(OTA)updateIPPayloadCompressionProtocolIP-in-IPtunnelingIPsec EncapsulatingSecurityPayloadIRCFileTransferDataiTunesJabberFileTransferJava
UpdateJEDI(Citrix)Kazaa (FastTrack protocol)KIKMessengerKingDigitalEntertainmentLinkedIn.com Livehotmail formobileLivestream.com LogMeIn RescuemagicJack Mail.ru AgentMaktoob
mailMediaGatewayControlProtocolMessageSessionRelayProtocolMicrosoftActiveSyncMicrosoftLync MicrosoftLync OnlineMicrosoftOffice365MicrosoftRemoteProcedureCallMicrosoft
ServiceControlMicrosoftSharePointMicrosoftSharePointAdministrationApplicationMicrosoftSharePointBlogManagementApplicationMicrosoftSharePointCalendarManagementApplication
MicrosoftSharePointDocumentManagementApplicationMultiProtocolLabelSwitchingdata-carryingmechanismNagios RemoteDataProcessorNagios RemotePluginExecutorNameService
ProviderInterfaceNetflix.com NetMeetingILSNetworkTimeProtocolNintendoWi-FiConnectionNortel/SynOptics Netwok ManagementProtocolOkCupid OnlineCertificateStatusProtocolOovoo
OpenShortestPathFirstOperaUpdateOrkut.com OutlookWebAccess(Office365)OutlookWebAppPalTalk Paltalk audiochatPalTalk TransferProtocolPaltalk videoPandoraRadioPastebin
Pastebin_posting PCAnywherePhotobucket.com Pinterest.com Playstation NetworkPlentyOfFishQIKVideoQQQQFileTransferQQGamesQQMailQQWeiBo QQ.com QQDownload QQLive
NetworkPlayerQQMusic QQStream Quakequic QVODPlayerRapidShare.com RealTimeStreamingProtocolRemoteDesktopProtocol(WindowsTerminalServer)RemoteProcedureCall
RetroShare RoutingInformationProtocolV1RoutingInformationProtocolV2RoutingInternetProtocolng1Rovio EntertainmentRSSSalesforce.com SAPSecondLife.com SecureShellSession
TraversalUtilitiesforNATSharePointOnlineSilverlight(MicrosoftSmoothStreaming)SimpleObjectAccessProtocolSkinnyClientControlProtocolSlackerRadioSlingbox Snapchat SOCKet Secure
v5SoMud Bittorrent trackerSoundCloud SourceForge SPDYSpotify SquirrelMail Steampowered.com SymantecNortonAntiVirus UpdatesSyslogSystemsNetworkArchitectureTeamspeak v2
TeamSpeakv3TeamViewer TelnetTeredo protocolTerminalAccessControllerAccess-ControlSystemPlusTIBCORendezVous ProtocolTor2webTumblr TwitchTwitpic TwitterUStream uTorrent
uTP (MicroTransportProtocol)UUSee ProtocolVEVOViber Vimeo.com VineVirtualRouterRedundancyProtocolVMWare vmware_horizon_view Waze SocialGPSMaps&TrafficWebExWhatsApp
MessengerWHOISWiiConnect24Wikipedia.com WindowsAzureCDNWindowsInternetNamingServiceWindowsLiveFileStorageWindowsLiveGroupsWindowsLiveHotmailWindowsLive
HotmailAttachements WindowsLiveSkyDrive WindowsLiveSkyDrive LoginWindowsMarketplaceWindowsUpdateWordPress.com WorldofWarcraft XboxLiveXboxLiveMarketplaceXbox
MusicXboxVideo(MicrosoftMoviesandTv)xHamster.com YahoogroupsYahooMailclassicYahooMailv.2.0YahooMessengerYahooMessengerconferenceserviceYahooMessengerTransfer
ProtocolYahooMessengerVideoYahooSearchYahoowebmailformobileYahooWebmessenger Yahoo.com YellowPageBindYellowPagePasswd YellowPagesServerYoutube.com
16
AnalyzingWireDatausingSplunk
Oncedataiscollected,usenormalSPLtogeneratedashboardsto
analyzedata
Pros:
– FlexibleandDetailedanalytics
– Simpledrill-downstorawdata
Cons:
– SPLishard!
– LongTime-to-Value
17
SplunkITSI
ITServiceIntelligencesolution- Design“Services”comprisedofKPIs
Specifies,Extracts,andMonitorsKeyPerformanceIndicators(KPIs)
inyourSplunkdata
GlasstablesallowsyoutovisualizeKPIsonanintuitivelayoutwith
colorchangesandotherindicators
DeepDivesillustratetime-seriescorrelationsbetweenKPIstoshow
elementsatfaultortemporalcausality
BuiltinAnomalyDetectionwithMachineLearningallows
18
ITSIforApplicationandNetworkMonitoring
Stream+ITSI=APM/NPM??
Notquite,butmanyoftheelements
MonitortheApplication
–
–
–
–
ResponseTime
Connections
ResponseCodes
Simultaneous
MonitortheNetwork
– Bandwidth
– Latency
– ExtraneousTraffic
19
AutomaticAnomalyDetectionwithITSI
Howdoyouknowwhen
somethinghasgonewrong?
Typically,anomalydetection
takesmuchtuning,isdelicate
ITSIautomaticallylearnsthe
patternsinyourdatasoanomaly
detectionhappensautomatically
too!
DynamicMachineLearning– not
simplestaticthresholds
20
Demo
ArtofthePossible
WhatelsecanwedowithStream+ITSI?
AutomaticallyalertwhenanewCommercialApplicationisseenonthe
networkandtrackthesourceuser
Dynamicallyletyouknowwhenanunusualnumberofcustomer
transactions(highorlow)areoccurringonyoureCommerce platform
Alertwhenthedelayaccessingacriticalserverexceedsathreshold,
andthenhelpisolatethesourceofthetrouble(network,host,
storage,etc.)
22
THANKYOU