IXP Manager Workshop
V4 Deep Dive
BarryO’Donovan&NickHilliard
27th Euro-IXForum
Berlin,Germany– October25th 2015
Introduction / House Keeping / Agenda
Whatdoyouwanttocoverhere?
Suggestions/preparedstuff:
– Currentwishlists/plans
– AdeeplookatIXPManagerV4
• Mytoolstack forthoseinterested
– Longtermgoal– funding,sponsorship,etc.
– Somepracticalscripting
CurrentWishLists/
Plans
Current Wish Lists / Plans (and WIP)
HelpdeskIntegration
L2ACLs
AdvancingP2Pfunctionality
Resellerportsviap-tag
Patchpanelmanagement
IntelligentProvisioning
– Portconfigurationtemplatesandautomaticconfiguration
L2 ACLs
Dynamicportsecurityjustdoesnotwork inanIXP
– MostofushavealreadymovedtostaticL2ACLs
NeedthisfunctionalitysupportedinIXPManager:
– Preventdrivererror/PEBKAC
– AllcustomerstoupdateL2ACLs->nomore2AMphonecalls
Usualconstraintsexist:
– Security
– Switchvendoragnosticwithreferenceimplementation(s)
– Someknobsanddials
Advancing P2P Functionality
CurrentlyP2PfunctionalityisviaRRDs
– Thislimitsfeaturedevelopmentforanythingbutsimplyp2pgraphs
EvaluatingOpenTSDB andGraphite/Carbon/Whisper
– WhisperislikeRRDinthatitisafixedsizedatabase
– However,OpenTSDB ismuchmorecomplexandisdistributed
DiskIOisanissue– butnotinsurmountable
Withamoredatabaselikebackend,wecanpresentmore
interestingdetailstoourcustomers:topnpeers,burstiest peers,
presentationofthedatacanbecomealotmoreuserfriendly,…
Reseller Ports via p-tag
IXPManagersupportsthefan-outportmodelforresellersasis
NewequipmentatINEXallowsthep-tagmodel
Thisbreaksalotofthings:
– Port/trafficgraphsviamrtg /snmp
– MACaddresslearningforp2pgraphs
– WehavetwomemberswherewehavenovisibilityL
L2ACLs+advancesinP2P/sflow collectionshouldfixallthisJ
Patch Panel Management
Currentlywemanage~25datacentre patchpanelsviawikiL
HavehopedtoimplementthisinIXPManagerfor~7yearsL
Anumberoffalsestarts.Why?
– We’vebeencompletelyoverthinkingthiswithwiz-bangfeatures
– Wejustneedthewikiequivlent forcryingoutloud!
– Nofancygraphing,nomanaginginternalinterrack panels…
Patch Panel Management
PatchPanelManagementto:
–
–
–
–
–
Recordpatchpanelreferenceandmediatype,numberofports
Onaperportbasis,assigntoacustomer
Recordthedatacentre assignedID
Connection(anddisconnection)date
Optionallyassigntoaswitchport
• Needtomanagecustomerportmovesasaprocessnow!
– Oroptionallyassigntocustomerco-locatedequipment
Nomorebike-sheddingthis!(ourselvesincluded!)
Intelligent Provisioning
Anoftenrequestedfeature– eveninternallyatINEX
Createaprovisioningflowforcommontaskssuchas:
– Joiningtheexchange(westilluseaWorddocument,yuck!L )
– Provisioningaport->verymanual
Intelligent Provisioning
Provisioningaport->verymanual
–
–
–
–
–
–
–
–
–
Assignport
Assignpatchpanelportandsendcrossconnectdetails
Configureport(quarantine)
ConfirmcrossconnectisinplaceandrecordID
Testportandensurespeed/duplexarematched
Quarantineport(checktraffic,checkBGProutesadvertised)
PutportliveinproductionpeeringLAN
Setuproutecollector,routeserverandAS112sessions
Sendannouncement
Intelligent Provisioning
Provisioningaport->verymanual
–
–
–
–
–
–
–
–
–
Assignport
Assignpatchpanelportandsendcrossconnectdetails
Configureport(quarantine)
ConfirmcrossconnectisinplaceandrecordID
Testportandensurespeed/duplexarematched
Quarantineport(checktraffic,checkBGProutesadvertised)
PutportliveinproductionpeeringLAN
Setuproutecollector,routeserverandAS112sessions
Sendannouncement
Intelligent Provisioning
Provisioningaport->verymanual
–
–
–
–
–
–
–
–
Assignportand patchpanelportandsendcrossconnectdetails
Configureport(quarantine)
ConfirmcrossconnectisinplaceandrecordID
Testportandensurespeed/duplexarematched
Quarantineport(checktraffic,checkBGProutesadvertised)
PutportliveinproductionpeeringLAN
Setuproutecollector,routeserverandAS112sessions
Sendannouncement
Intelligent Provisioning
So,howdowedesignaprovisioningsystemforeveryIXP?
Carefully.Intelligently.Dynamically.
Eachstephascommonattributes:
–
–
–
–
–
ActionpendingcustomerorIXP
Testtodetermineifstepiscompletecorrectly/incorrectly
Actiontotakeoncompletion?
Messaging
Advancetonextstep
Createaframeworkusingcontractstobuildupaprocesswithsteps
Intelligent Provisioning
Port Configuration Templates
Standardised portconfigurationsat(obviously)essentialatanIXP
INEXperformedaforkliftupgradefromBrocadetoExtremerecently
UsedIXPManager’sdatabasetoconfigureallportsonnewswitches
– Timeefficient,nooperatorerrors,ensuredstandardconfigs withoutmissing
elements
CouldbeeasilyintegratedintoIXPManagerforcopyandpaste,but:
– Wouldpreferittoactually:shutdownport+wipe+configure+enableagain
– WouldalsolikeittovalidateRANCID/Oxidizedconfigs forerrors/ommissions
Vendoragnostic!
IXPManager
V4
IXP Manager V4
Majorversionchangesusuallymeanmajorchanges
– Thisistruehere->butmainlyonthebackend
IXPManagerisyourtypicalMVCstack
– We’recompletelychangingtwoelementsofthis:
• Thecontroller
• Theview
IXP Manager V4 – Doctrine ORM
Let’stalkaboutthemodelfirst:thedatabase
LayerbetweenMySQL/MariaDB andthePHPapplicationisDoctrine
ORM
V3ofIXPManagerchangedfromDoctrine1.2toDoctrine2
– ActivemodeltoORM(ObjectRelationalMapping)
Anexample:
IXP Manager V4 – Doctrine ORM
<?php
$c = new Entities\Customer;
$c->setName( "Big ISP Ltd" );
$c->setAutsys(64496);
// ...
D2EM::persist( $c );
IXP Manager V4 – Doctrine ORM
$custRepo
= D2EM->getRepository( “Entities\Customer” );
$customers = $custRepo->findAll();
foreach( $customers as $c ) {
// do something
}
IXP Manager V4 – Doctrine ORM
Stableproject– aroundsince2006
It’sactuallyanumberoflibrariesthatstacktogether.Mainly:
– DoctrineDBAL:DatabaseAbstractionLayer
– DoctrineORM:ObjectRelationalMapper/ing
http://www.doctrine-project.org/
IXP Manager V4 – Doctrine ORM
Keyconcepts:
– Entities:representasingledatabaserowfromagiventable
– Proxies:compiled entitiesthatarefullytransparenttoyourcode.
• Allowsforlazyloading,loadingincompletedetails,etc.
– Repositories:handlessetsofentities
• Mostofourcomplexqueriesarehandledbywayofproxies:
IXP Manager V4 – Doctrine ORM
class Customer extends EntityRepository {
/**
* Utility function to provide a array of
* all active and current customers.
*/
public function getCurrentActive(
$asArray =false, $trafficing =false,
$externalOnly = false, $ixp = false )
{
…
}
…
}
IXP Manager V4
HowdidwestarttalkingaboutDoctrine?
IXPManagerisyourtypicalMVCstack
– We’recompletelychangingtwoelementsofthis:
• Thecontroller
• Theview
Let’slookattheviewnext…
IXP Manager V4 – The View
What’saviewcomponent?Whydowehaveit/needit?
– Separateslogicfrompresentation
– Inlargerprojects,UIdesignersdon’tneedtobeabletocodetomanagethe
frontend
– Eradicatesspaghetticode
– Allowsfortemplating withlayouts
– Allowsforskinning
Foryears,theonlygameintowninPHPwasSmarty
– It’sgodawful L Itstinkstohighhell.
– Okay,that’sprettyharsh.It’sofits’ageandhasn’tmovedforward…
IXP Manager V4 – The View
ContenderstoSmarty:
Twig
– modernOOPdesign
– goodextensibility
– wellsupportedandwidelyused
Blade
– thebuilt-indefacto viewforLaravel
Problem:viewsadddeveloperoverhead:moresyntax,libraries,
functions,etc.tolearnL
IXP Manager V4 – The View
PHPPlates
–
–
–
–
NativePHPtemplates– nonewsyntaxtolearn
InspiredbyTwig
Supportslayoutsandinheritance
Easytoextendwithfucntions andextensions
– NB:Platesisafulltemplatesystem,notspagetti codebyanothername
Inreality,IXPManagerv4supportsSmarty,BladeandPlatesoutof
thebox.
IXP Manager V4 – The Controller
ThebiggestpartofMVCisC->thecontroller
Handles:
–
–
–
–
–
Routingofrequests
Middleware
Inputvalidation
Controllers
Responses
OurnewcontrollerisLaravel.Notjustacontroller,aframework.
IXP Manager V4 – The Controller
Whychangeframeworkatall?
–
–
–
–
–
Developerapathywhichleadsto:
Stagnationofthecodebase
Neworprospectivedevelopersareturnedoff
Newfeaturesremainunimplementedbecausetherearebetterways
Staymoderntoleveragenewtechniquesandservices
IXP Manager V4 – The Controller
TheLaravel framework providesnewtechniquesandintegrations:
–
–
–
–
–
–
–
Serviceproviderframework
Events
Queues
Taskscheduling
Testing
Migrations
Packagemanagement
IXP Manager V4 – The Controller
Wecan’tthrowawaytheexistingcodebasethough.
OverthecourseofV4’slifetime,we’llmigratefromZend toLaravel
– i.e.Zend/Smartywillco-existwithLaravel/Plates forquitesometime
– Thismeansnewfeaturescanbeimplementedimmediatelyusingthelatest
technologies
– Wedon’tneedtodisappearforsixmonthstorewritetheentirecodebase
Howwillthisbeachieved?
– An(in)elegent solution!
IXP Manager V4 – The Controller
Laravel isnowthedefaultframeworkandroutesrequests
IfarequesthitsLaravel foraroutethatdoesnotexist/isnot
implementinLaravel:
– Itthrowsa404exception
Inapp/Http/Kernel.php wecatchthat404exception
– AndspinuptheZend Framework
Zend willthenhandleifpossibleorthrowanother404
– (handledasapagenotfoundinZend andpresentedtotheuser)
IXP Manager V4 – The Controller
try {
return $this->sendRequestThroughRouter($request);
} catch( \Symfony\Component\HttpKernel\Exception\NotFoundHttpException $e ) {
require_once 'Zend/Application.php';
$application = new \Zend_Application(
APPLICATION_ENV, APPLICATION_PATH . '/configs/application.ini’
);
$application->bootstrap()->run();
}
IXP Manager V4 – Events and Queues
Eventsprovideasimpleobserverimplementation
– Youcansubscribeandlistenforeventsinpackages/extentions!
– Eventscanbefired whensomethingsignificant happens
– Eventlistenerscanqueuetheeventforofflineprocessing
Laravel queuessupportBeanstalkd,IronMQ,AmazonSQS,Redis and
synchronous(local,immediate)
Whatkindofthingscanwedowiththis..?
IXP Manager V4 – Events and Queues
PhysicalinterfacechangedinIXPManager:firephysIntChangedEvent
– MRTGlistenercancheckforportorportspeedchangeandregenerateMRTG
configurationandreloadthedaemon
– Billingnotificationslistenercancheckforspeedchangeandemailaccounts for
billingpurposes
– Switchconfigurationlistenercanrolloutconfigurationchangetoswitch(firesevent)
• Physicalinterfacestatuslistenercaninspectinterfaceformatchingspeed/
duplex
– Patchpanellistenercouldtakesome actionifaportischangedthathasaconnected
crossconnect
IXP Manager V4 – Events and Queues
VLANinterfacechangedinIXPManager:firevlanIntChangedEvent
–
–
–
–
AS112listenercan(de)configureBGPsessionasnecessary
Routecollectorcan(de) configureBGPsessionasnecessary
Routeserverscan(de)configureBGPsessionasnecessary
Othereventlistenersmayinclude:regeneratingSmokeping &Nagios configurations,
DNSPTRentries.
– EnablingIPv6couldadditionallysendemailwithdetailsorstarttheIPv6enable
processtowalkthecustomerthroughconfigurating sessionstoroutecollector,
servers,as112,etc.
IXP Manager V4 – Service Providers
CentralpieceofLarabel’s applicationbootstrapping
Registers:
– Controllersandroutes
– Eventlisteners
– Middleware
CanbeusedtoextendIXPManagerwithouthackingthemain
codebase
IXP Manager V4 – Contracts
InterfacesthatdefinecoreservicesprovidedbyLaravel
AlsohowwewilldevelopextensionstoIXPManager
– Designacontract
– Developreferenceimplementation(s)tothatcontract
Example:Helpdeskintegration
– Firstabitofhistory….
New Helpdesk - [email protected]
From an INEX Members’ Update
Up to 2008 - Shared IMAP Mailbox
2009 - Cerberus
Served us well but extreme feature creep
2013 - Realisation that we need something new
Helpdesk research => maximum pain
New Helpdesk - [email protected]
From an INEX Members’ Update
New Helpdesk - [email protected]
From an INEX Members’ Update
Up to 2008 - Shared IMAP Mailbox
2009 - Cerberus
Server us well but extreme feature creep
2013 - Realisation that we need something new
Helpdesk research => maximum pain
Pain so great, we stuck with what we had
2015 - Try again, new methodology => pain killerz
New Helpdesk - [email protected]
From an INEX Members’ Update
New Helpdesk - [email protected]
Excellent API (essential as we needed to import old tickets)
Easily configurable triggers and automations
Supports markdown
Nice UI plus iOS / Android apps
Poor reporting (@ Zendesk Regular anyway
From an INEX Members’ Update
Candidates included:
Freshdesk, GrooveHQ, Zendesk, Kayako, Cerb5 and many more…
Contrary to my initial preconceptions, the winner was Zendesk
IXP Manager V4 – Helpdesk Integration
Needtobeableto:
–
–
–
–
–
Createcustomers(organisations)onthehelpdesksystem
Createusersonthehelpdesksystem
Findticketsbyorganisation
Createtickets
Update/closetickets
It’sworkinprogressbutmostoftheintegrationisdone
IXP Manager V4 – Helpdesk Integration
Contract:app/Contracts/Helpdesk.php
–
–
–
–
–
–
–
–
–
ticketsFindAll()
organisationNeedsUpdating($custLocal,$custHelpdesk )
organisationCreate($cust )
organisationUpdate($helpdeskId,$customer)
organisationFind($id)
contactNeedsUpdating(…)
userCreate()
userUpdate()
…
IXP Manager V4 – Helpdesk Integration
Zendesk ReferenceImplementation
– app/Services/Helpdesk/Zendesk.php
ServiceProvider
– app/Providers/HelpdeskServiceProvider.php
– Thisfileneedsupdatingfornewimplementations
Instantiation:
– $helpdesk=App::make('IXP\Contracts\Helpdesk');
Configuration:
– config/helpdesk.php
– EnvironmentconfigurationviaPHPDotEnv
IXP Manager V4 – Helpdesk Integration
Sample.env forZendesk:
HELPDESK_BACKEND=zendesk
HELPDESK_ZENDESK_SUBDOMAIN=ixp
HELPDESK_ZENDESK_TOKEN=yyy
[email protected]
IXP Manager V4 – Statistics Backend
IXPManagercurrentlyonlysupportsMRTG/logforportstats
Fromearlier,wealsowanttosupportportstatsviasflow /p2p
– WouldalsoliketosupportMRTG/rrd
DE-CIXhaveofferedabountyforthiswork
Willbeimplementedinthesamewayasthehelpdesk
– i.e.anybackendcouldbesubstitued onceitisimplementedagainstthe
providedcontract
Threereferenceimplementations:sflow/p2p,MRTG/log,MRTG/rrd
IXP Manager V4 – Installation Changes
ThePHPdevelopmenttoolchainhaschangedsincev3
– Gitsubmodules nolongernecessary
PHPhasapackagemanagementsystemcalledcomposer
– Allthirdpartydependancies nowinstalledviacomposer
– Includes:ZF1,Laravel,Smarty,Plates,Zendesk API,Doctrine,etc.
Frontendassetshandledsimilarlyviabower
– Includesjquery,Bootstrap,etc.
IXP Manager V4 – Vagrant
IXP Manager V4 – Vagrant
IXP Manager V4 – Vagrant
1. InstallVagrant(http://www.vagrantup.com/ )
2. InstallVirtualBox (http://www.virtualbox.org/ )
3. CloneIXPManager,checkoutv4andinstalldependancies:
git clone https://github.com/inex/IXP-Manager.git ixpmanager
cd ixpmanager
git checkout v4
composer update
IXP Manager V4 – Vagrant
4. Start-upVagrant:vagrant up
5. Thiswilltakeawhile– itexecutesbootstrap.sh whichwill:
–
–
–
–
–
apt-getupdate,upgradeandinstallalldependancies forIXPManager’sLAMP
environment
ConfigureMySQLandphpMyAdmin
Installcomposerandbower
ConfigureandpopulatetheIXPManagerdatabasewithsampledata
ConfigureApacheandIXPManager
IXP Manager V4 – Vagrant
6. Onceit’scomplete,youcan:
–
–
–
–
AccessIXPManagerat:http://localhost:8088/
• Adminusernameandpassword:vagrant/vagrant1
SSHintothevirtualmachinewith:vagrant ssh
Yourixpmanager directoryismountedunder/vagrant
MySQLisavailablevia:mysql –uroot–ppassword ixp
• Orhttp://localhost:8088/phpmyadmin
IXP Manager V4 – Vagrant
ManagingyourVM:
–
–
–
–
–
Shutdowncleanlybylogginginand: sudo shutdown –h now
Tosuspend:
vagrant suspend
Toforceshutdown: vagrant halt
Tobringup:
vagrant up
Forstatus:
vagrant status
IXP Manager V4 – Documentation
Remember:v4isabridgingversionfromZF1toLaravel
Assuch,documentationisalsoahalfwayhouse
Existingdocumentationavailableat:
– https://github.com/inex/IXP-Manager/wiki
Newdocumentationwillbe:
– https://ixp-manager.readthedocs.org/en/latest/
– Source:https://github.com/inex/ixp-manager-docs
IXP Manager V4 – My Tool Stack
AppleOSXwithHomebrewfor:
– php,bash,bgpq3,git,joe,mariadb,node,sshfs andmuchmore
Atomasatexteditor
– Withlanguage-php,linter(sameforCSS,JS,etc)andDash
Vagrant(latestUbuntuLTS)
Git,GitHub,TravisCI
Skipper(ORMGUI,http://www.skipper18.com/ )
QuickCodingExample?
Quick Coding Example
Let’smakeanewArtisancommand
– ArtisanistheCLIcomponentofLaravel
./artisan make:console DemoListCustomers
Quick Coding Example
Nowlet’sedittheresultantfile:
– app/Console/Commands/DemoListCustomers.php
Givethecommandanameanddescription
Andlet’sseeifitworks…
Quick Coding Example
Andlet’sseeifitworks…nopeL
Extendingthewrongclass– needtouseandextend:
– useIXP\Console\Commands\CommandasIXPCommand;
– =>classDemoListCustomers extendsIXPCommand {
Nooptions/argumentsrequired
Needtoregisterthecommandinapp/Console/Kernel:
– protected$commands=[…]
Nowitworks!Butdoesnothing...
Quick Coding Example
Let’sgetandlistallcustomers:
Completethefire()method:
$customers =
\D2EM::getRepository( 'Entities\Customer' )->getCurrentActive();
foreach( $customers as $c )
$this->info( $c->getName() );
Thanks for listening!
[email protected]
https://github.com/inex/IXP-Manager
Mailing list:
https://www.inex.ie/mailman/listinfo/ixpmanager