Using Shibboleth as an SSO
University at Buffalo
Joel Murphy
[email protected]
overview
•
•
•
•
•
•
•
•
•
•
•
•
Id mgmt @ UB
UB deployment Timeline
Deployment considerations
Certificate management
LDAP
Load Balancer
Web Farm and Service Providers
Webmail proposed configuration
Cosign
Shibboleth Identity Provider
Load Testing
Challenges
Id Mgmt @ UB
• “UBIT Name” branding
– All University affiliated persons (faculty, staff,
students, etc.)
– single username/password, many directories, many
applications
– usernames and passwords and groups synced across
directories
– One user namespace for campus
– People retain same username after returning to Univ
• Unix groups primary mechanism for
authorization (authz) and entitlements
• Applications responsible for authorization
Id Mgmt @ UB
• “Accounts Management” oracle database and
application core to Id Mgmt
– Home grown Perl, C, oracle SQL
• Downstream Directories
– DCE – authentication, authorization, dir svcs
– LDAP – dir svcs, authn via Kerberos, authz
– Kerberos – authn, enables encrypted authenticated
communication channels
– Active Directory
• X.509 Certificates
Password management
• password replication across directories
Long road to SSO
•
•
•
•
•
•
•
•
•
•
•
•
Pre-1997 – Unix passwords/NIS – just Unix services
1997 - DCE – first success at consolidation of Id mgmt and password
database, multiple platforms
1998 – DCE/DFS deployed as enterprise filesystem
2003 – Kerberos/Active Directory/LDAP/Shibboleth/Cosign/Cisco CSS
Summer 2005 – production Oracle RAC database
Fall 2005 – First production Shibboleth Services – Course Management,
Download service
Fall 2005 - LDAP Campus Portal and administrative apps (was DCE)
Summer 2006 – Network Appliance deployment for DFS replacement for
enterprise filesystem (CIFS/NFS)
Summer 2006 – Business Continuity site deployment and machine move
Fall 2006 – Shibbolize authn portions of Campus web server
Fall 2006 – Shibbolize Campus Portal and apps (from LDAP)
Fall 2006 or Spring 2007 – Shibbolize Web Mail (IMP or @mail)
Id Mgmt/SSO Outliers
• Enterprise Active Directory “exception” accounts
(eg. OU administrators) – not in Kerberos/LDAP
• Science and Engineering NIS passwd (migrating
to enterprise Kerberos for auth)
• Various Active Directory Forests outside
Enterprise AD
• Various applications with limited technical
resources or need for Enterprise integration
Id Mgmt challenges
• Authorization not always done or works by “accident”
(user not in /etc/passwd)
• Application owners don’t always know who should be
authorized – typically request a group
• DCE still core for to IdM
– Authoritative for groups and data for deactivated users
– RPCs use heavily, replacing with SOAP/SSL
• CIO level requests
– Better authorization for network access
– Online password reset
• Requests for LDAP access for outsourced apps
• Hard to get resources for not-so-low hanging fruit
Some Shibboleth/SSO and Web
Farm Goals/Considerations
•
•
•
•
Survive network split (multiple campuses)
Don’t preclude logoff
Ability Load balance commodity hardware
Detect and Isolate failures - Transparent failover
of Identity Providers and Service Providers
(multiple machines, same configuration)
• Capacity/Responsiveness – Shibboleth is
Enterprise SSO
• Commitments made to sell Shibboleth solution
How we got there
• Internal certificate signing service
• LDAP environment for our attribute data (eduPerson,
UBEduPerson)
• Cisco CSS (Content Switch) load balancer
• Multi-purpose Oracle RAC database (“Real Application
cluster” - active/active)
– Hitachi SAN
•
•
•
•
Cosign SSO
Shibboleth
Lots of glue
Lots of help and contributions from UB and I2 community
Did we make it?
•
Survive network split (multiple campuses)
– No – multiple load balancers, but can’t migrate IPs across subnets, opted against
DNS delegation
– Business continuity site not on one of the campuses (no people)
•
Don’t preclude logoff
– Supported in Cosign, limited Shib 2.0 support?
•
Detect and Isolate failures - Ability Load balance commodity hardware
–
–
–
–
•
Large farm of Sun 280R Solaris servers for admin applications
Farm of 8 Dell 1750 Linux servers for shibboleth/cosign
4 Sun 280R Solaris servers for LDAP
4 Sun V120 Solaris servers for Kerberos
Transparent failover of Identity Providers and Service Providers
– Yes, except for failure mid-transaction
•
Capacity/Responsiveness – Shibboleth is Enterprise SSO
– Yes, our auth systems are very under capacity
– Adding 2 additional LDAP servers (new/bad applications with unpredictable
performance profiles)
Physical Layout
Shibboleth IDP Setup
•
•
•
•
•
•
•
InCommon member
Apache 1.3/Tomcat connector/Tomcat
Handle Service authenticated by Cosign, trusted by Shib
Still at version 1.2 IDP (origin)
Hindsight: put IDP source tree in CVS
Web Farm setup requires Shib Handle configured as “CryptoHandleGenerator”
instead of shared memory
All Attributes currently resolved via LDAP (from Person objects)
–
•
•
•
•
•
•
•
Non-web apps may have similar needs
Entitlements are currently mapped from group memberships and stored in LDAP
Once configured, IDP changes very infrequently
Attribute release is negotiated when configuring new targets
Meta-data for Federations (sites and trusts) changes periodically
We continue to deploy all configuration in one “war” file.
Log files are somewhat large and unwieldy
No deterministic policies, data custodians authorize attribute release
Application Web Farm/Shibboleth
SP Setup
•
•
•
•
•
•
•
•
Apache 1.3
Caching apache proxy in front of application apache
Shibboleth integrated into application apache
Apache proxy/cache runs SSL
Machines load balanced with Cisco CSS
Currently assume UB IdP for applications
Standard attribute release and acceptance
Multiple services (virtual hosts) on one machine
– Haven’t needed to adjust ARP for Virtual Hosts
• One certificate and ARP shared by all web farm machines
• Modified MySQL cache to be Shibboleth Oracle shar cache
– Sticky connections on CSS makes unnecessary
– Was one source of early stability problems
Application Web Farm/Shibboleth
SP Issues
•
Canonical Service names
– Forcing Canonical host names (UseCanonicalName) makes IDP configuration of
AssertionConsumerServiceURL simple.
– Prevents dreaded error message from Handle Service (unknown ACS URL…)
– Makes it very difficult for developers or sysadmins to poke a specific machine
(use hosts file)
– Alternative – list every possible hostname/short name in meta-data?
•
Web cache
– Cookie based authentication doesn’t prevent caching of data where HTTP
BasicAuth does automatically
– Need to set header in response of authenticated pages: “Cache-Control: private"
•
Web proxy (URL rewrite)
– We needed to set checkAddress=“false” in shibboleth XML
– ShibURLScheme, ServerName and Port needed for apache
• Shibboleth needs to know how to write a “return address” for redirects
• Multiple virtual hosts, odd port mapping schemes allow us to run proxy and server on
same hosts
Shibboleth SP Issues
• Provide a “UB” shibboleth distribution and instructions
– Build out of date, XML config, metadata and instructions mostly
valid
– Where to put certs
– What startup files are needed
– What should be monitored
– What settings to tweak in a web farm environment
• We generally recommend rebuilding apache module for
local apache
– Unnecessary?
• New versions of shib from I2 come as packages and are
significantly simpler than the past
Blackboard/Course Management
• Shibboleth just used for SSO
(authentication)
• Nightly data processing feeds Blackboard
system data from institutional systems
– Creates users object in Blackboard
– Manages course registrations
Download Service
• Shibboleth used as SSO
• Group membership used for authorization
– Authorization based on licensing
– Microsoft software download
Campus Portal
• Shibboleth to be used as SSO
• Current phase is to transition from LDAP
auth (was DCE)
• Nightly processing generates data about
people authorized portal in portal
application
Webmail Proposal
• Cyrus IMAP email
• Would like to link Campus Portal/Webmail/Course mangement
• Multiple possibilities
–
–
–
–
–
Integrate with cosign directly and use Kerberized IMAP session
Shibbolize webmail and use secure fabricated password for auth
Trusted webmail
Pass Kerberos ticket via Shibboleth Attribute???
Upcoming Shibboleth solution for n-tier apps???
• Concerns/Issues
– Desire to abstract out Cosign
– Fabricated password must not be reproducible or usable via normal
IMAP connections
– Fabricated password requires maintaining a password file
– Kerberizing IMAP possibly more Webmail customization than tinkering
with password
Webmail Proposal
• Current working proposal is to use shib
with fabricated passwords
Shibboleth Authorization
•
•
•
•
•
Our group management is inflexible (DCE still authoritative)
Working to remove DCE, would like Grouper (groups of groups?)
Business rules don’t always match single group membership
Needed to match sets of groups
Require membership in active and some staff group:
ShibRequireAll on
require memberof ~ ^…staff$
require memberof ~ ^active$
•
Require membership in active or some staff group:
ShibRequireAll off
require memberof ~ ^…staff$
require memberof ~ ^active$
•
Shib1.3b adds XML access control, nested requirements
–
•
https://authdev.it.ohio-state.edu/twiki/bin/view/Shibboleth/XMLAccessControl
Using mod_setenvif/mod_access does not work with Shibboleth environment
variables
–
Shibboleth module doesn’t have a chance to generate env variables before mod_access and
order is Apache hard coded, not dependant on module order
Cosign Setup
• SSO just for Shibboleth Handle Service
• Cosign supports logoff (hidden)
• Cosign can pass Kerberos tickets and do
certificate based auth
• SSO’s at the time of our deployment did not
support logon server redundancy.
• Cosign writes state information to disk, extended
this to write state to Oracle.
• Oracle transactions through an external server
(cookie_server)
Cosign Issues
• Custom code makes upgrading difficult
– Backported fixes from newer cosigns
• Bugs in our oracle interface
• Oracle RAC configured Active/Active, but updates not
synchronous
– Needed max_commit_propagation_delay=0 on Oracle RAC
instances
• Newer cosign versions have async replication of state
– Could loose some state in logon server failure
• Move state files from Oracle RAC to HA Netapp NFS
share?
Cosign diagram
LDAP Environment
•
•
4 load balanced Sun ONE 5.2 Directory Servers
One giant directory
– ou=People, dc=buffalo, dc=edu
•
•
•
•
•
•
posixAccount
Person
OrganizationalPerson
InetOrgPerson
eduPerson
UBEduPerson
– ou=Groups, dc=buffalo, dc=edu
• posixGroup
• groupOfUniqueNames
•
•
•
Added institutional IDs and UBEPmemberOf to UBEduPerson
memberOf on Person makes shib Attribute release simple and fast
Future deployments should use eduMember schema
– http://middleware.internet2.edu/dir/docs/internet2-mace-dir-ldap-groupmembership-200507.html
LDAP Environment (cont)
•
•
Hourly processing updates LDAP with changes from Id mgmt database
(Perl Net::LDAP)
Oracle layout
– One row per principal
– One column per attribute, multi-valued attributes glued with a separator
•
Goal to co-locate non-institutional data in directory
– Available via LDAP or via Shibboleth
– Allow administrators to manage source oracle table or write changes to LDAP
•
•
•
•
All Access management policy at the root of the directory granting to
groups, some based on attribute value (FERPA)
Avoids confusion by avoiding object acls or deny acls
Multi-master replication – supports application writes with load balancing
No passwords – simple binds proxied to Kerberos with a modified version of
preAuth plugin by Mike Gettes
– Forces simple binds over SSL
LDAP Issues
• Large groups don’t perform well, use memberOf
• Can’t “browse” the directory – administrative limits
– Last modified time
• DS 5.1 didn’t support multi-master replication
• Introduced memory leak in Kerberos pre-auth plugin
when porting to DS 5.2
• Replication of schemas created redundant definitions
(fixed?)
• Replication failures with large directory and lots of
changes
– Fixes for bugs 6242420 and 6283717 included in DS 5.2 patch 4
– Turned multi-master back on to support LDAP applications
needing to write data
Internal Certificate Management
• Manages certs for test servers, shibboleth SP’s
and VPN access
• Openssl on Id mgmt system
• requests via email
• Daily job for monitoring expirations
• No audit trail and little metadata associated with
certs
• Not particularly safe for multiple administrators
• Tightening access/clarifying procedures
• Plan on moving to OpenCA
Load Balancer Setup
• Began with Cisco Local Directors for email system load
balancing (SMTP/IMAP/POP)
• Current - Two Cisco CSS 11503, active/standby
• Load balanced services on private network behind load
balancers, using normal UB address space
• Servers see IP address of Clients
• NAT, health monitoring, sticky connections, load
balancing
• Standard architecture for all UB load balanced services
(LDAP, Shibboleth, Cosign, web farm, etc.)
• Ability to directly connect to a specific machine in a pool
Load Balancer Issues
•
•
•
•
Services see the IP of client connecting
Can easily roll-in/deploy and test changes with no downtime
Network management of private net
Everything on one subnet
– Allows migrating IPs between CSS
– Extended subnet to Bus. Cont. site, but not other campus sites
•
•
Danger of private net connecting to public
Load Balanced services talking to other load balanced services
– Must NAT both source and destination host
•
Sysadmin Management
–
–
–
–
•
Too many services, anyone can take everything down
Shared serial console (HTTP disabled for security/auditing concerns)
Many ports on many machines
Secure web tool in development
Monitoring services, removing from pool
– CSS can monitor HTTP results for testing server health
•
Notification for down servers (hooked into Big Brother)
Load Balancer example
Troubleshooting
• Check our Big Brother console for alerts
• Check Load balancer – what’s active?
• Application Service provider
– Check access_log/error_log
– Check Shibboleth shar.log and shire.log
– Is shar (shibd) running?
– Can we connect to Shib AA with openssl
s_client?
Troubleshooting (cont)
• Identity Provider
– Are apache/cosignd running and accepting
connections?
– Is cookie_server running?
– Check all apache logs
– Check tomcat bootstrap log – did shib start?
– Check shib application logs
Load Testing
•
•
•
•
•
•
Webload (Radview)
JMeter
Perl LWP
SAR (Linux/Solaris)
Infrastructure survives intense load testing
Shibboleth initial logon “costs” lessen with
longer sessions (perhaps less in some
cases?)
Challenges
• Transitioning into production
– Project team not all of support team
• Too much change at one time
– Service maturity takes time
• Lots of layers
–
–
–
–
Lots of machines and services
Troubleshooting complex
Errors don’t go travel upstream well
Multiple support teams
• Help Desk support complex, especially for Fed
apps