Shared Services with Spotfire
Christian Turri
Lead BI Engineer
Spotfire London User Group
May 2016
Our Spotfire History
Background
•
•
•
Started with v3.3 in 2011, moved to v4.5 in 2013 and to v6.5 on 2014/2015
Initially just Market Risk but now being used by several departments and across multiple
divisions
Some stats
– 8 “active” Production and 8 “passive” Contingency servers (services stopped)
– 25 environments in total (Development, Testing, Production, Contingency)
– 124 CPU Cores and 1TB of RAM installed Prod capacity
– ~200 Scheduled Updates reports
– 1200 Unique Web Player users per month
– ~22k “Open Analysis” success events per month
– ~32k Scheduled Updates refreshed per month
– 450k Information Link executions per month
– 72 Professional Client unique logins to Production per quarter
– ~5000 Production Library Analysis (362 IT controlled, rest user generated)
– LDAP sync with 4 Active Directory domains
2
Operating Model
How we use Spotfire in our organisation?
•
Global User base mainly based in the UK, India and the US
•
Servers running 24x7 with a maintenance window on Sunday morning
•
Shared Services model allows for:
– Core Spotfire Team developers
– Project Team Spotfire developers
– Power Business Users developing their own reports (self-service) and developing
EUDAs (End User Developed Applications) for their business users
•
Some applications embedded Spotfire within their applications (mashups)
•
Core Spotfire Team manages infrastructure, supports and controls the platform
3
Using Spotfire in a large enterprise deployment
Issues we found
•
Large number of changes (avg 20 changes per month) but no versioning control
•
Insufficient logging available
•
No historical reporting
•
Read only access to Administration tools not supported
•
Extremely difficult to monitor and enforce standards and best practices
•
Building new Spotfire environment took 2 man days
4
What we did to address the different platform issues
Solutions we implemented
•
Persisted both Spotfire Server and Web Player logs to the Spotfire metadata database
•
Reverse engineered Spotfire metadata database and created views to expose data in a
useful way
•
Hooked to our enterprise monitoring tool
•
Developed set of “Administration” reports
•
Developed several PowerShell scripts to monitor and automate tasks
•
Developed report to refresh Scheduled Updates manually
•
Automated check-in of Scheduled Updates files into SVN
5
What we did to address the different platform issues
Solutions we implemented
•
Persisted both Spotfire Server and Web Player logs to the Spotfire metadata
database
•
Reverse engineered Spotfire metadata database and created views to expose data in a
useful way
•
Hooked to our enterprise monitoring tool
•
Developed set of “Administration” reports
•
Developed several PowerShell scripts to monitor and automate tasks
•
Developed report to refresh Scheduled Updates manually
•
Automated check-in of Scheduled Updates files into SVN
6
Screen shots
7
Screen shots
8
Screen shots
9
What we did to address the different platform issues
Solutions we implemented
•
Persisted both Spotfire Server and Web Player logs to the Spotfire metadata database
•
Reverse engineered Spotfire metadata database and created views to expose data in
a useful way
•
Hooked to our enterprise monitoring tool
•
Developed set of “Administration” reports
•
Developed several PowerShell scripts to monitor and automate tasks
•
Developed report to refresh Scheduled Updates manually
•
Automated check-in of Scheduled Updates files into SVN
10
Screen shots
11
Screen shots
12
Screen shots
13
What we did to address the different platform issues
Solutions we implemented
•
Persisted both Spotfire Server and Web Player logs to the Spotfire metadata database
•
Reverse engineered Spotfire metadata database and created views to expose data in a
useful way
•
Hooked to our enterprise monitoring tool
•
Developed set of “Administration” reports
•
Developed several PowerShell scripts to monitor and automate tasks
•
Developed report to refresh Scheduled Updates manually
•
Automated check-in of Scheduled Updates files into SVN
14
Screen shots
15
Screen shots
16
What we did to address the different platform issues
Solutions we implemented
•
Persisted both Spotfire Server and Web Player logs to the Spotfire metadata database
•
Reverse engineered Spotfire metadata database and created views to expose data in a
useful way
•
Hooked to our enterprise monitoring tool
•
Developed set of “Administration” reports
•
Developed several PowerShell scripts to monitor and automate tasks
•
Developed report to refresh Scheduled Updates manually
•
Automated check-in of Scheduled Updates files into SVN
17
Screen shots
18
What we did to address the different platform issues
Solutions we implemented
•
Persisted both Spotfire Server and Web Player logs to the Spotfire metadata database
•
Reverse engineered Spotfire metadata database and created views to expose data in a
useful way
•
Hooked to our enterprise monitoring tool
•
Developed set of “Administration” reports
•
Developed several PowerShell scripts to monitor and automate tasks
•
Developed report to refresh Scheduled Updates manually
•
Automated check-in of Scheduled Updates files into SVN
19
Screen shots
20
Screen shots
21
What we did to address the different platform issues
Solutions we implemented
•
Persisted both Spotfire Server and Web Player logs to the Spotfire metadata database
•
Reverse engineered Spotfire metadata database and created views to expose data in a
useful way
•
Hooked to our enterprise monitoring tool
•
Developed set of “Administration” reports
•
Developed several PowerShell scripts to monitor and automate tasks
•
Developed report to refresh Scheduled Updates manually
•
Automated check-in of Scheduled Updates files into SVN
22
Screen shots
23
How we did it?
Some technical details
•
Log4J changes - custom Java loader for Log4J log files
•
Log4Net changes - log4net.Appender.AdoNetAppender for live logging (see Web Player
installation manual 6.12.2.2 Log to Database Example)
•
Oracle PL/SQL package to handle Scheduled Updates logic to enhance data logged
•
Database views on top of Spotfire Metadata objects
•
PowerShell scripts interacting with Spotfire config.bat, SVN command line, Spotfire
Automation Services, Event Logs and Microsoft Log Parser for IIS logs
24
Screen shots
25
Screen shots
26
Future Enhancements planned
•
Scheduled automated SVN check-in of Information Link SQL, IronPython Scripts and other
various configuration files (web.config, log4net, log4j.*, etc)
•
Automated email for users trying to access a report they don’t have access
•
Automated email to the relevant support team when a scheduled update fails
•
Automated way of distributing new config files
•
Look at automating Production releases
27
Q&A
28