Deploy an Elastic, Resilient,
Load-Balanced Cluster in 5
Minutes with Senlin
Mark Voelker (VMware)
Qiming Teng (IBM)
Xinhui Li (VMware)
1
Agenda
 Requirements for Resource Pool Management
 Senlin Clustering Service Overview
 An All-in-one Sample
2
Managing Resource Pools
3
Generic Requirements
 Manageability
 cluster membership
 Elasticity
 key feature of a cloud, pay-as-you-go
 Load-Balanced
 distribute loads evenly on all members
 Flexibility
 customizable to meet the needs of real-life use cases
 Extensibility
 easy to add resource types and policies
4
What we have on OpenStack
 Heat: The Orchestrator
 AutoScalingGroup +
Scaling Policy + Alarm
 Heat AutoScaling modeled after AWS
AutoScaling
 It may suit some limited use cases, but not
all scenarios considering that OpenStack
aims to build a platform for both private and
public clouds
Heat as a Compiler
YAML
Template
Nova
Heat
•
•
•
•
CREATE
GET
UPDATE
DELETE
Cinder
Neutron
Keystone
 But, Heat's Mission...
 Orchestrate composite cloud apps using a declarative template format
through an OpenStack-native REST API
 It is not about any new service, not auto-scaling, not HA, not load-balance ...
5
Some Specific Requirements
 Cross Availability Zone Placement
 Cross Region Placement
 Anti-Affinity Placement
 Choose a Specific Node to Delete when Scaling-In
 Trigger Autoscaling Using Application-Level Metrics
 Manual Scaling
 Auto Recover Node when Failures Detected
 Migrate Nodes from a Stand-By Cluster for Rapid Provisioning
 Soft Scaling
 ...
6
Senlin Overview
7
What Do We Really Need?
Scalable
- What is missing from
OpenStack?
A Clustering Service
Load-Balanced
Highly-Available
Manageable
......
of any (OpenStack) objects
- Auto-scaling?
Just one of the usage
scenario of a cluster.
- Auto-Healing (HA)?
Just another usage
scenario.
- We can address the
concerns by making
policies orthogonal
8
Senlin Architecture
Policies
Senlin
Client
REST
RPC
Senlin
API
Senlin Engine
Senlin
Database
Profiles
9
Senlin Features
 Profiles: A specification for the objects to be managed
 Policies: Rules to be checked/enforced before/after actions are performed
Policies as Plugins
placement
deletion
Profiles as Plugins
Kubernetes
Cluster/Nodes Managed
Containers
Docker
scaling
Heat
Stacks
Nova
VMs
Senlin
health
load-balance
Ironic
BareMetal
batching
10
Senlin Server Architecture
senlin-api
WSGI
middleware
apiv1
openstacksdk
rpc client
engine
engine
lock
parser
receiver
registry
scheduler
service
actions
cluster
node
MsgQueue
policies
placement
deletion
identity
compute
receiver
webhoook
scaling
network
drivers
dummy
profiles
os.heat.stack
orchestration
openstack
os.nova.server
...
(others)
(others)
extension points for interfacing
with different services or clouds
dbapi
load-balance
extension points
for external
monitoring
services
extension points
facilitating a
smarter cluster
management
health
batching
extension points to talk to different
endpoints for object CRUD operations
11
Senlin Operations (Actions)
 Cluster
•
•
•
•
•
•
•
•
•
•
•
•
CREATE
DELETE
UPDATE
LIST
SHOW
ADD_NODES
DEL_NODES
SCALE_OUT
SCALE_IN
POLICY_ATTACH
POLICY_DETACH
POLICY_UPDATE
 Node
CREATE
DELETE
UPDATE
LIST
SHOW
•
•
•
•
•
 Action
•
•
LIST
SHOW
 Policy
•
•
•
•
•
•
•
•
•
•
•
CREATE
UPDATE
DELETE
LIST
SHOW
 Event
•
 Profile
LIST
SHOW
CREATE
UPDATE
DELETE
LIST
SHOW
 Receiver
•
•
•
•
CREATE
DELETE
LIST
SHOW
12
profile/cluster/node operations
$
$
$
$
$
$
$
$
$
$
$
$
$
$
$
$
$
senlin
senlin
senlin
senlin
senlin
senlin
senlin
senlin
senlin
senlin
senlin
senlin
senlin
senlin
senlin
senlin
senlin
profile-create -s <spec> <name>
profile-list
profile-show <profile>
profile-delete <profile>
profile-update -s <spec> <profile>
cluster-create -p <profile> -n <size> <name>
cluster-delete <cluster>
cluster-list
cluster-show <cluster>
node-create -p <profile> -c <cluster> <name>
node-delete <node>
node-show <node>
node-join -c <cluster> <node>
node-leave <node>
cluster-node-add -n <nodes> <cluster>
cluster-node-del -n <nodes> <cluster>
cluster-node-list <cluster>
13
policies operations
$
$
$
$
$
senlin
senlin
senlin
senlin
senlin
policy-create -s <spec> <name>
policy-list
policy-show <policy>
policy-delete <policy>
policy-update -n <name> ... <policy>
$
$
$
$
$
$
$
senlin
senlin
senlin
senlin
senlin
senlin
senlin
cluster-policy-attach <cluster> <policy>
cluster-policy-detach <cluster> <policy>
cluster-policy-list ... <cluster>
cluster-policy-show <cluster> <policy>
cluster-policy-enable <cluster> <policy>
cluster-policy-disable <cluster> <policy>
cluster-policy-update ... <cluster> <policy>
14
Other Useful Operations
$ senlin cluster-scale-in [-c <count>] <cluster>
$ senlin cluster-scale-out [-c <count>] <cluster>
$ senlin cluster-resize <options> <cluster>
$
$
$
$
senlin
senlin
senlin
senlin
profile-type-list
profile-type-show <type>
policy-type-list
policy-type-show <type>
$ senlin action-list
$ senlin action-show <action-id>
$ senlin event-list
$ senlin event-show <type>
15
senlin cluster-resize
usage: senlin cluster-resize [-c <CAPACITY>] [-a <ADJUSTMENT>] [-p <PERCENTAGE>]
[-t <MIN_STEP>] [-s] [-n MIN] [-m MAX] <CLUSTER>
Positional arguments:
<CLUSTER>
EXACT CAPACITY?
Name or ID of cluster to operate on.
Optional arguments:
-c <CAPACITY>, --capacity <CAPACITY>
The desired number of nodes of the cluster.
-a <ADJUSTMENT>, --adjustment <ADJUSTMENT>
A positive integer meaning the number of nodes to add,
or a negative integer indicating the number of nodes
to remove.
-p <PERCENTAGE>, --percentage <PERCENTAGE>
A value that is interpreted as the percentage of size
adjustment. This value can be positive or negative.
-t <MIN_STEP>, --min-step <MIN_STEP>
An integer specifying the number of nodes for
adjustment when <PERCENTAGE> is specified.
-s, --strict
A boolean specifying whether the resize should be
performed on a best-effort basis when the new capacity
may go beyond size constraints.
-n MIN, --min-size MIN New lower bound of cluster size.
-m MAX, --max-size MAX New upper bound of cluster size. A value of -1
indicates no upper limit on cluster size.
CAPACITY BY PERCENTAGE?
CHANGE IN CAPACITY?
MINIMUM STEP?
STRICT/BEST-EFFORT?
ADJUST MIN_SIZE?
ADJUST MAX_SIZE?
16
ALL-In-ONE:
AS (AutoScaling) + HA (High-Availability) + LB (Load-Balancing)
17
Architecture Overview
Heat Template
Scaling
Policy
LB Policy
LBaaS
Profile
Cluster
Receivers
SCALE IN
SCALE OUT
RECOVER
Alarm
Health
Policy
Compute
18
Profile and Cluster
profile:
type: OS::Senlin::Profile
properties:
type: os.nova.server-1.0
properties:
flavor: {get_param: flavor}
image: {get_param: image}
key_name: {get_param: key_name}
networks:
- network: {get_param: network}
security_groups:
- {get_resource: security_group}
cluster:
type: OS::Senlin::Cluster
properties:
desired_capacity: 2
min_size: 2
profile: {get_resource: profile}
19
LB Policy
lb_policy:
type: OS::Senlin::Policy
properties:
type: senlin.policy.loadbalance-1.0
bindings: [cluster: {get_resource: cluster}]
properties:
pool:
protocol: HTTP
protocol_port: 80
subnet: {get_param: pool_subnet}
vip:
subnet: {get_param: vip_subnet}
protocol: HTTP
protocol_port: 80
health_monitor:
type: HTTP
delay: 10
timeout: 5
max_retries: 4
20
Scaling Policies
scale_in_policy:
type: OS::Senlin::Policy
properties:
type: senlin.policy.scaling-1.0
bindings:
- cluster: {get_resource: cluster}
properties:
event: CLUSTER_SCALE_IN
adjustment:
type: CHANGE_IN_CAPACITY
number: 1
scale_out_policy:
type: OS::Senlin::Policy
properties:
type: senlin.policy.scaling-1.0
bindings:
- cluster: {get_resource: cluster}
properties:
event: CLUSTER_SCALE_OUT
adjustment:
type: CHANGE_IN_CAPACITY
number: 1
21
Health Policy
health_policy:
type: OS::Senlin::Policy
properties:
type: senlin.policy.health-1.0
bindings:
- cluster: {get_resource: cluster}
properties:
detection:
type: NODE_STATUS_POLLING
options:
interval: 60
recovery:
actions:
- RECREATE
22
Receivers and Alarms
receiver_scale_in:
type: OS::Senlin::Receiver
properties:
cluster: {get_resource: cluster}
action: CLUSTER_SCALE_IN
type: webhook
scale_in_alarm:
type: OS::Ceilometer::Alarm
properties:
meter_name:network.services.lb.incoming.bytes.rate
statistic: avg
period: 180
evaluation_periods: 1
threshold: 12000
repeat_actions: True
alarm_actions:
- get_attr:
- receiver_scale_in
- channel
- alarm_url
comparison_operator: le
23
Receivers and Alarms
receiver_scale_out:
type: OS::Senlin::Receiver
properties:
cluster: {get_resource: cluster}
action: CLUSTER_SCALE_OUT
type: webhook
scale_out_alarm:
type: OS::Ceilometer::Alarm
properties:
meter_name: network.services.lb.incoming.bytes.rate
statistic: avg
period: 60
evaluation_periods: 1
threshold: 28000
repeat_actions: True
alarm_actions:
- get_attr:
- receiver_scale_out
- channel
- alarm_url
comparison_operator: ge
24
Short Video, For Free, :-)
25
Questions?
26
Blueprints on Reworking Heat Autoscaling
BP
Priori
ty
Description
autoscaling-apiresources
high
Heat resources invoking AS APIs
as-api-groupresource
high
ScalingGroup resource wrapping AS API's
group functionality
as-api-policyresource
high
ScalingPolicy resource wrapping AS API's policy
functionality
as-apiwebhookresource
high
Webhook resource wrapping AS API's execution
of webhooks
autoscaling-apiclient
high
A python client for Heat to interact with AS API
autoscaling-api
-
A separate service for the implementation of
autoscaling w/ Heat
as-engine
-
A separate engine/service for autoscaling
support AS API
as-engine-db
-
A DB dedicated to autoscaling, using schema
created in as-lib-db
as-lib
-
A separate module to be used by the AS service
as-lib-db
-
A DB for autoscaling bookkeeping
27
Relation To Other Projects
 Senlin provides the array data type for cloud programming
struct person {
// Heat stack containing senlin clusters
int age;
char name[0]; // Senlin cluster of nova servers
}
person team[10];
Ceilometer
// Senlin cluster of Heat stacks
Monasca
Complex Data Types
Primitive Data Types
Sahara
Heat
Nova
Cinder
Tacker
Horizon
Mistral
Magnum
Zaqar
Senlin
Neutron
Swift
Keystone
28