Scalla Developer
Summary
xrootd /cmsd
Andrew Hanushevsky
SLAC National Accelerator Laboratory
CERN Workshop
10-November-08
http://xrootd.slac.stanford.edu
Supported Platforms
FreeBSD

New addition (best-effort support only)
Linux

i386_linux24, i386_linux26, x86_64_linux_26
MacOS

ppc_darwin_70, ppc_darwin_80, x86_darwin_90
Solaris

sun4x_58, sun4x_59, sun4x_510, sunx86_510
Windows

XP (client only)
10-November-08
2: http://xrootd.slac.stanford.edu
Supported Compilers
g++
Up to version 4.3.2
 Used for FreeBSD, Linux, and MacOS

icc

Used for Itanium architectures

SGI Linux
Sun CC

Always used for Solaris compilations
10-November-08
3: http://xrootd.slac.stanford.edu
Supported Build Environments
In-House configure.classic
Works on all platforms without problems
 Missing some features


Make install
Autotools make
Supported by Derek Feichtinger
 Has some issues on certain platforms


E.G., Unable to build XrdPosixPreload.so

10-November-08
32/64 bit compilation/linking on 64/32 bit platforms
4: http://xrootd.slac.stanford.edu
General Source Layout I
XProtocol/

cmsd and xrootd protocol data structures
Xrd/

Protocol driver for cmsd and xrootd (networking and scheduling)
XrdAcc/

Default access control (i.e., authorization)
XrdBwm/

Bandwidth manager plug-in
XrdCS2/

Castor plug-in (obsolete)
XrdClient/

Client related code (e.g., TXNetFile, xrdcp, etc)
XrdCms/

Cluster Management Services protocol plug-in
XrdCns/

Cluster Name Space daemon and other components
10-November-08
5: http://xrootd.slac.stanford.edu
General Source Layout II
XrdCrypto/

Cryptography support classes for security plug-ins
XrdMon/

Monitor data collection agent
XrdNet/

All networking related classes (used by everyone)
XrdOdc/

Open Distributed Cluster support (deprecated)
XrdOfs/

Open File System plug-in
XrdOlb/

Open Load Balancing protocol implementation (deprecated)
10-November-08
6: http://xrootd.slac.stanford.edu
General Source Layout III
XrdOss/

Open Storage System plug-in
XrdOuc/

Object Utility Classes (used by everyone)
XrdPosix/

Posix compatibility libraries

Includes preload library
XrdPss/

Proxy plug-in

Used as a storage system plug-in
10-November-08
7: http://xrootd.slac.stanford.edu
General Source Layout IV
XrdSec/

Authentication protocol driver plug-in
XrdSecgsi/

Gsi-based authentication plug-ins
XrdSeckrb4/

Kerberos IV authentication plug-in
XrdSeckrb5/

Kerberos V authentication plug-in
XrdSecpwd/

Password authentication plug-in
XrdSecsss/

Simple Shared Secret authentication plug-in
XrdSecunix/

Unix (i.e., NFS-like) authentication plug-in
10-November-08
8: http://xrootd.slac.stanford.edu
General Source Layout V
XrdSfs/

File System Interface definition

Includes the default implementation
XrdSut/

Security utility classes
XrdSys/

OS-dependent classes (used by everyone)
XrdTokenAuthzOfs/

ALICE ofs plug-in implementing ALICE security
XrdVersion.hh

Holder to identify the version everywhere
XrdXr/

Proxy plug-in (obsolete, to be deleted)
XrdXrootd/

Xrootd protocol plug-in
10-November-08
9: http://xrootd.slac.stanford.edu
So, What’s A Plug-In?
Run-Time Loadable Code

A single class based on an abstract interface


Plus an extern ‘C’ object instantiator
Always packaged as a shared library

Configuration file specifies location of library


Automatically loaded
Objects created as needed
Extends xrootd or cmsd functionality

Avoids massive rebuilds and code branching

The base system uses pre-defined statically linked plug-ins
10-November-08
10: http://xrootd.slac.stanford.edu
Scalla Plugin Architecture
authentication
(gsi, krb5, etc)
Protocol Driver
(Xrd)
Protocol (1 of n)
(xrootd, cmsd)
lfn2pfn
prefix encoding
authorization
(name based)
File System
Storage System
(ofs, sfs, alice, etc)
(Flash
(oss, drm/srm,
Based System)
etc)
Clustering
(cmsd)
10-November-08
11: http://xrootd.slac.stanford.edu
The Plug-Ins
Dynamic plug-in always packaged as a shared library

Authentication plug-in



Authorization plug-in


Available: libXrdProxy.so (provides proxy access)
Protocol plug-in


Available: libXrdOfs.so
Storage System plug-in


Currently, no plug-ins exist (default statically linked-in)
File System plug-in


Always as libXrdSec.so
Others as libXrdSecxxxx.so
PROOF team uses a plug-in to implement PROOF protocol
External Management Interface plug-in (XMI)

Used by Castor to interface cmsd to Castor name space
10-November-08
12: http://xrootd.slac.stanford.edu
Closer Look At Security Plug-Ins
xrootd has no security

FALSE!
By default, security is not enabled

This simplifies setup for most sites
You must configure security to get it

Not difficult but yet another thoughtful step
Once enabled, xrootd is as secure as you want

So, the myth is now busted!
10-November-08
13: http://xrootd.slac.stanford.edu
What Is Security?
Two phases to security

Authentication
Done in the xrootd/cmsd protocol layer
 Multiple simultaneous authentication modes supported


Authorization
Done in the ofs filesystem layer
 Basic user/group/netgroup capability list mode available
 But, authorization is a plug-in


10-November-08
Can substitute any other mode you’d like
14: http://xrootd.slac.stanford.edu
xrootd Authentication
Multi-protocol design
Server provides available protocols
 Client chooses one of the possibilities
 Each protocol name comes with configuration data


Allows client to self-configure for the protocol
Each protocol implemented as a plug-in

Easy to add new protocols
10-November-08
15: http://xrootd.slac.stanford.edu
Currently Available Protocols
gsi

Grid certificate based authentication

Extension gsiGMAPLDAP maps DN to username via LDAP
krb4

Standard Kerberos IV
krb5

Standard Kerberos V
pwd

Hidden password authentication
sss

Simple shared secret
unix

Basic Unix NFS-like [non-]security
10-November-08
16: http://xrootd.slac.stanford.edu
Authentication Protocol Plug-Ins
Each protocol is a shared library plug-in

libXrdSecxxxx.so

The xxxx is the protocol name (e.g., libXrdSeckrb4.so)
The plug-ins are managed by libXrdSec.so

Authentication protocol client/server driver


Finds and loads appropriate shared libraries
Uses abstract security interface for all interactions

Credential generation & authentication
10-November-08
17: http://xrootd.slac.stanford.edu
Abstract Class XrdSec
• C++ framework to manage protocols as plug-ins
• Generic protocol (XrdSecProtocol)
class XrdSecProtocol {
public:
virtual int Authenticate( XrdSecCredentials *cred,
XrdSecParameters **parms,
XrdSecClientName
&client,
XrdOucErrInfo
*einfo=0 )
virtual XrdSecCredentials *getCredentials(
XrdSecParameters
XrdOucErrInfo
//
//
//
//
In
Out
Out
Out
*parms=0, // In
*einfo=0 ) // Out
virtual const char *getParms( int &psize,
const char *host )
};
server
client
// Out
// In
• Protocol implementations inherit from XrdSecProtocol
10-November-08
18: http://xrootd.slac.stanford.edu
Courtesy of
Gerri Ganis
libXrdSec.so
• libXrdSec.so provides the Protocol Manager
- Server: instantiated at start-up from configuration file:
- load protocol plug-ins that server can / wants to run
- binds (subsets of) the list to hosts or host patterns
- controls authentication mode by host
- Client: build-up list loading protocols the first time needed
- library loaded only if authentication is required
• Plug-in implementations provide a public instantiator to create
an instance of the protocol
• Simple negotiation: list of allowed protocols sent to the client,
who chooses the one to try first
Courtesy of
Gerri Ganis
10-November-08
19: http://xrootd.slac.stanford.edu
XrdSec Implementation Is Generic
• Depends only on network and utility modules
• XrdNet, XrdOuc, and XrdSys
• Can be easily used in a non-XROOTD context
• Working example of standalone client and server programs
using XrdSec available at
http://ganis.home.cern.ch/ganis/ROOT/SECURITY/testXrd.tgz
Courtesy of
Gerri Ganis
10-November-08
20: http://xrootd.slac.stanford.edu
Authentication Architecture
1
Get
Credentials
login
Send client-specific security configuration
Get Security
Configuration
2
(Select Protocol)
3
Send credentials
Ask for more
n
Multiple exchanges allowed
get credentials
authenticate
0
Config File
libXrdSec.so
libXrdSecgsi.so
libXrdSeckrb5.so
libXrdSeckrb4.so
libXrdSecpwd.so
Dynamically selected by client
Server specifies availability
Libraries managed by libXrdSec.so
10-November-08
21: http://xrootd.slac.stanford.edu
libXrdSec.so
Authenticated Identity
XrdSecEntity.hh
char
char
char
char
char
char
char
char
void
int
prot[8];
*name;
*host;
*vorg;
*role;
*grps;
*endorsements;
*tident;
*cert
clen;
//
//
//
//
//
//
//
//
//
//
Protocol used
Entity's name
Entity's host name
Entity's virtual organization
Entity's role
Entity’s groups
Protocol specific endorsements
Trace identifier (do not modify)
Pointer to certificate (future)
Length of certificate (future)
Passed to file system layer to be used for authorization
10-November-08
22: http://xrootd.slac.stanford.edu
Why Do It This Way?
Can implement almost any model needed

Without changing any server/client code at all

Simplifies security audit procedures
Can quickly evolve as requirements change

And support different modes for different moods
SLAC Atlas uses sss authentication
 Fermi (a.k.a. GLAST) uses unix authentication
 But generally many experiments don’t want anything


10-November-08
Only for reading but not if they are writing to xrootd
23: http://xrootd.slac.stanford.edu
But Wait!
Where is SSL?

SSL applies to the transport layer
The security framework applies to the protocol layer
 Recall, any protocol can optionally use this framework


A transport protocol does not easily give you any options
And SSH?

You really mean using ssh keys, don’t you?

No one wrote a plug-in for that yet
10-November-08
24: http://xrootd.slac.stanford.edu
Let’s Recap
xrootd/cmsd are all about plug-ins
Expands applicability within the design focus
 Allows you to piggy-back new functionality


For instance, PROOF
Are all points pluggable?

Just the obvious ones

There is always room for improvement here

E.G., MSS plug-in now part of the oss plug-in
So, let’s see talk about writing plug-ins
10-November-08
25: http://xrootd.slac.stanford.edu
Writing Plug-Ins I
Read the documentation in the plug-in “hh”

XrdAccAuthorize.hh


XrdCmsXmi.hh


Protocol abstract interface
XrdSecInterface.hh


Storage System Abstract Interface
XrdProtocol.hh


XMI abstract interface
XrdOss.hh


Authorization abstract interface
Authentication abstract interface
XrdSfsInterface.hh

File system abstract interface
Look at an existing plug-in of the same type
10-November-08
26: http://xrootd.slac.stanford.edu
Writing Plug-Ins II
General things
All plug-ins must be thread-safe
 Avoid high-latency actions


This generally causes pile-up/melt-down


Some interfaces allow background processing


Usually because of thread starvation and timeouts
High latency is not an issue then
Use existing classes in the repository
libCrypto.a, libCryptoLite.a,
 libXrdNet.a, libXrdOuc.a, and libXrdSys.a

10-November-08
27: http://xrootd.slac.stanford.edu
Writing Plug-Ins III
For a plug-in to be included in the CVS repository


Must follow naming conventions
Adhere to the abstract interface


Interface changes are rare and always backward compatible
Be stand-alone

Cannot rely on frameworks and external add-ons




This includes STL
• A sore point but it’s saved us countless hours of debugging
Have a good Makefile and Makefile.am
Compile and run on all supported platforms
Usually have detailed documentation

See existing references (e.g., Security)

http://xrootd.slac.stanford.edu/doc/sec_config/sec_config.htm
Otherwise, we can simply reference your plug-in web page
10-November-08
28: http://xrootd.slac.stanford.edu
Packaging
Official releases are available via web site

http://xrootd.slac.stanford.edu/


We are working on getting a better more generic URL
Currently, Wilko Kroeger cuts official releases

This does not preclude special integrated releases





ALICE
Castor
PROOF
Root
These usually suffer a little drift from the official release
We are working on making the CVS available r/o

The fastest way is an AFS accessible directory

/afs/slac.stanford.edu/public/software/scalla


CVS head checked out for viewing and a gtar file of the same
Planning for a web interface to the repository
10-November-08
29: http://xrootd.slac.stanford.edu
Licensing & Contributions
Currently, Scalla is under a BSD License

http://www.opensource.org/licenses/bsd-license.php
May change to Apache 2.0 license

http://www.opensource.org/licenses/apache2.0.php
We may need to restrict how names are used

cmsd, Scalla, and xrootd

This is to prevent confusion relating to derivative works
Contributions happily accepted


Must conform to licensing requirements
Handled in the Linux tradition
10-November-08
30: http://xrootd.slac.stanford.edu
SLAC Support
Memo of understanding…
The SLAC science program is heavily dependent on xrootd. I can therefore
assure you that xrootd will be maintained by SLAC for at least five years.
As you are aware, xrootd is an open source product and will remain freely
available.
I believe that xrootd is brings valuable and currently unique capabilities
in scalable high-volume data analysis. It is part of SLAC's mission to
encourage wide use of developments like xrootd where they can benefit
national and international science programs.
SLAC staff supporting xrootd will be encouraged to examine, as time permits,
problems and suggestions submitted by users who are not connected with the
SLAC program. Non-trivial work in response to such submissions would
require that work be also beneficial to SLAC's use of xrootd.
I particularly encourage a collaborative approach to maintaining and
developing products like xrootd. This approach promotes wide use, and
creates a situation where effort spent on issues raised by a collaborator
would be considered valuable to SLAC by default.
Richard Mount
10-November-08
31: http://xrootd.slac.stanford.edu
Active Developers
Current Active Software Developers

Andreas Peters (Andreas.Joachim.Peters cern.ch)


Andrew Hanushevsky (abh stanford.edu or Andrew.Bohdan.Hanushevsky cern.ch)


Classic make, Cross-Platform issues, Security and PROOF
Tofigh Azemoon (azemoon slac.stanford.edu)


Client
Gerardo Ganis (Gerardo.Ganis cern.ch)


Autotools
Fabrizio Furano (Fabrizio.Furano cern.ch)


FreeBSD issues, Windows
Derek Feichtinger (Derek.Feichtinger cern.ch)


Server
Bertrand Bellenot (Bertrand.Bellenot cern.ch)


Castor/xrootd
Monitoring
Wilko Kroeger (wilko slac.stanford.edu)

MPS scripts, packaging, and release issues
10-November-08
32: http://xrootd.slac.stanford.edu
Getting Support
Currently available venues

Official web site (always check there first)


General problem mailing list ([email protected])


http://project-arda-dev.web.cern.ch/project-arda-dev/xrootd/site/index.html
Request Tracker (RT) problem system at SLAC


Must be subscribed (see http://xrootd.slac.stanford.edu/xrootdlist.html)
CERN Web Site


http://xrootd.slac.stanford.edu/
This is still experimental
E-Mail the right developer

Actual bugs, contributions, and enhancement requests
User support

An evolving issue for experiments and groups


E.G., OSG provides 1st level support for VDT
Developers generally cannot provide direct user support
10-November-08
33: http://xrootd.slac.stanford.edu
Acknowledgements
Software Contributors




CERN: Derek Feichtinger, Fabrizio Furano, Andreas Peters
Fermi: Tony Johnson (Java)
Root: Gerri Ganis, Bertrand Bellenot
SLAC: Jacek Becla, Tofigh Azemoon, Wilko Kroeger
Operational Collaborators

BNL, INFN, IN2P3
Partial Funding

US Department of Energy

Contract DE-AC02-76SF00515 with Stanford University
10-November-08
34: http://xrootd.slac.stanford.edu