Accelerate Oracle Performance
by Migrating Oracle Objects to
Dorado
This document describes why and how to migrate oracle objects to Dorado Storage System.
HUAWEI OceanStor Dorado2100 and Dorado5100 are SAN storage systems using all solid state disks
designed to eliminate IO bottleneck.
Jarvis WANG
[email protected]
IT Storage Solution & Verification, Enterprise BG
2012-9-28 Version 1.1
1
Why Migrate
As we know, whether in OLTP (online transactional processing) or OLAP (online
analytical processing) systems, Oracle is I/O intensive. Besides, the increasing of
capacity and performance is not in-phase for enterprise hard disk. The capacity of
enterprise hard disk is too big and too slow for Oracle database nowadays. Migrating
oracle files to SSDs (solid state disks, also known as flash disk), user can benefit from
the low latency, high random IOPS, high sequential throughput and low power
consumption features.
OLTP
Bottleneck
The following table shows the top 5 wait events in one oracle OLTP database running on
traditional magnets disks (10K RPM SAS disk). 96.87% of DB time is spent on waits of
“db file sequential read”, and each wait costs 15 milliseconds. The high latency of the
slow magnets disks is definitely the bottleneck of the OLTP system.
OLAP
Bottleneck
The following table shows the top 5 wait events in one oracle OLAP database running on
traditional magnets disks (10K RPM SAS disk). More than 80% of DB time is spent on
User I/O. The User I/O is definitely the bottleneck of the OLAP system.
2
Capacity and
Performance
of Hard Disk
The following table shows the capacity and performance change of enterprise hard disk
in the last 5 years, the capacity is increased year by year, but the performance stays the
same level. When deploying Oracle database on hard disk, users will buy much more
spindles than the capacity needs to suport the high I/O throughput requirements, while
the free capacity can’t be used to store other things because that will make the
performance of database bad. Tiering storage or pure SSD solution are now strongly
recommended in Oracle database, and the solution that migrating hottest objects to
Dorado fit into the performnace and capacity requirements.
OceanStor
Dorado
Year
Capacity
RPM
OLTP Throughput
OLAP Throughput
2008
73 GB
15 K
200 IO/s
30 MB/s
2009
146 GB
15 K
200 IO/s
30 MB/s
2010
300 GB
15 K
200 IO/s
30 MB/s
2011
600 GB
15 K
200 IO/s
30 MB/s
2012
900 GB
15 K
200 IO/s
30MB/s
HUAWEI OceanStor Dorado2100 and Dorado5100 are SAN storage systems using all
solid state disks, they are designed to eliminate IO bottleneck, and accelerate
mission-critical applications by reducing latency. Using Dorado2100 or Dorado5100 to
accelerate oracle performance is the best choice.
The following chart shows the SPC1-LIKE (OLTP workload) benchmark results of
OceanStor Dorado and traditional storage systems. The latency of Dorado is lower than
1ms, but the latency of traditional storage is higher than 10ms. Dorado could
significantly improve the response time and throughput in OLTP systems.
3
Dorado2100
Dorado5100
Traditional Mid-End
Traditional High-End
16.00
14.00
Latency (ms)
12.00
10.00
8.00
6.00
4.00
2.00
0.00
0
100,000
200,000
300,000
400,000
500,000
600,000
IOPS
The following chart shows the improvement of transaction response time and maximum
user load when migrating oracle files from SAS disks to Dorado.
Active Users
Response Time
200.00
60000
48000
199.97
50000
40000
150.00
30000
100.00
20000
12000
50.00
2000
30.35
24.03
0.00
Active Users
250.00
Response Time (ms)
OLTP
Acceleration
10000
0
Base Line - 48 HDDs, Migrate Hot Objects Migrate All Objects
24 used for data
to Dorado
to Dorado
After migrating hot objects to Dorado, the transaction response time (Response
Time) is reduced from 200ms to 30ms, reduced to 15%; the maximum user load
(Active Users) is increased from 2000 to 12000, increased to 600%.
After migrating all schema objects to Dorado, the transaction response time
(Response Time) is reduced from 200ms to 24ms, reduced to 12%; the maximum
4
user load (Active Users) is increased from 2000 to 48000, increased to 2400%.
Users are significantly benefited from the low latency of Dorado.
The following two tables are the “top 5 wait events” before and after migrating all
schema objects to Dorado. The wait time of “db file sequential read” is reduced from
15ms to 1ms, and the DB time percentage is reduced from 96.87% to 35.99%.
The following chart shows the improvement of analytical queries execute time (Queries
Time) and text data load time (Load Time) when migrating oracle OLAP schema objects
from SAS disks to SSDs.
100
90
80
70
60
50
40
30
20
10
0
Load Time
4622
90
1564
1302
29
21
Base Line - 48
HDDs, 28 used
for data, almost
1000G
Migrate Hot
Objects to
Dorado
13
5000
4500
4000
3500
3000
2500
2000
1500
1000
500
0
Queries Time (s)
Queries Time
Data Load Time (s)
OLAP
Acceleration
Migrate All
Migrate External
Objects to Drado Text Data to
Dorado
5
After migrating hot objects to Dorado, the queries time is reduced from 4627
seconds to 1564 seconds, reduced to 34%; the data load time is reduced from 90
seconds to 29 seconds, reduced to 32%.
After migrating all objects to Dorado, the queries time is reduced from 4627
seconds to 1302 seconds, reduced to 28%; the data load time is reduced from 90
seconds to 21 seconds, reduced to 23%.
After migrating the external text data to Dorado, the data load time is reduced
from 90 seconds to 13 seconds, reduced to 14%.
The following two tables are the “top 5 wait events” of an OLAP database before and
after migrating all schema objects to Dorado. The wait time of “direct path read” is
reduced from 37ms to 17ms, and the DB time percentage is reduced from 73.18%
to 35.55%.
6
What to Migrate
In OLTP database, I/O operations on data file are very random and the blocks are
relatively small (2k-8k). Mostly, the performance bottleneck is caused by the latency of
physical random read. The redo write is synchronize I/O operation and very frequent,
and the transaction commit must wait the success of redo log sync, so redo write is
another critical component in OLTP system and is the second files need to be migrated
to Dorado. The undo and temporary tablespaces are the third objects need to be
migrated to Dorado. The archive log and flash back recovery area are sequential and
periodical I/O operations, and are the last to be migrated to Dorado.
In OLAP database, I/O operations on data file are multiple streams sequential read and
the blocks are relatively large (32k-512k). The queries are usually executed in parallel
and there’re multiple streams, causing I/O operations more random and the throughput
decreased. Mostly, the performance bottleneck is caused by the read operation of
temporary and user tablespaces.
The best solution is migrating all oracle objects to Dorado, but when the budget is not
enough, you could follow the steps below to find and migrate the hottest objects to
Dorado.
Hottest
Segments
First of all, you could use Oracle AWR or STACKPACK report to find the hottest segments
and migrate them to Dorado. The following table shows the hottest segments order by
physical reads in an OLTP system which could be migrated to Dorado one by one.
7
Redo Logs
The 2nd step is to migrate redo log files to Dorado, reducing the latency of log file sync
operation.
Undo
Segments
The 3rd step is to migrate undo segments to Dorado, reducing the latency of rollback and
Temporary
Tablespace
The 4th step is to migrate temporary tablespace to Dorado. The temporary tablespace is
transaction processing operation.
a critical component in OLAP systems. Migrating temporary tablespace could
significantly improve the performance of sorting operations.
Archive Log
The 5th step is to set archive log destination to Dorado. Log archiving operation performs
sequential read from redo log and sequential write to archive destination, which has a
performance impact on redo log sync operations. The faster log archives, the smaller
impact on performance. Setting archive log destination to Dorado could improve the
transaction performance.
Flash Back
Recovery
Area
The last step is to set flash back recovery area to Dorado, making the flash backup and
recovery more quickly.
8
How to Migrate
The following steps describe migrating oracle database objects to Dorado. Before
migrating, finding the hottest objects, planning storage size and estimating migration
duration are strongly recommended by HUAWEI. Also, please ensure there’s no user
online when migrating.
Table
Segment
1. Create tablespaces in Dorado storage path
Create tablespaces for the table segment and related index segment in Dorado storage path.
For example, create two AUTOEXTEND BIGFILE tablespace on Dorado ASM disk group, one for the
table segment and the other for the related index segment:
connect / as sysdba
create bigfile tablespace ts_fast_order
datafile '+DRD1/ts_fast_order' size 128M reuse
autoextend on next 128m maxsize unlimited
extent management local
segment space management auto
nologging;
create bigfile tablespace ts_fast_idx_order
datafile '+DRD1/ts_fast_idx_order' size 128M reuse
autoextend on next 128m maxsize unlimited
extent management local
segment space management auto
nologging;
2. Move table to the new tablespace
For example, move table OORDER to tablespace TS_FAST_ORDER:
connect tpcc/tpcc
alter session enable parallel dml;
alter table OORDER
move tablespace ts_fast_order
nologging parallel 32;
3. Rebuild indexes for the table
For example, rebuild index for table OORDER:
connect tpcc/tpcc
alter session enable parallel dml;
alter index K_OORDER
rebuild tablespace ts_fast_idx_order
nologging parallel 32;
4. Compute statistics
9
For example, compute statistics for table OORDER and its index:
connect tpcc/tpcc
analyze table OORDER estimate statistics
for all indexes;
5. Done.
Table
Partition
1. Create tablespaces in Dorado storage path
Create tablespaces for the table partition segment and related index partition segments in Dorado
storage path.
For example, create two AUTOEXTEND BIGFILE tablespace on Dorado ASM disk group, one for the
table partition segment and the other for the related index partition segment:
connect / as sysdba
create bigfile tablespace ts_fast_stock01
datafile '+DRD1/ts_fast_stock01' size 128M reuse
autoextend on next 128m maxsize unlimited
extent management local
segment space management auto
nologging;
create bigfile tablespace ts_fast_idx_stock01
datafile '+DRD1/ts_fast_idx_stock01' size 128M reuse
autoextend on next 128m maxsize unlimited
extent management local
segment space management auto
nologging;
2. Move partition to the new tablespace
For example, move partition STOCK01 to tablespace ts_fast_stock01:
connect tpcc/tpcc
alter session enable parallel dml;
alter table stock
move partition stock01
tablespace ts_fast_stock01 parallel 32;
3. Rebuild indexes for the partition
For example, rebuild index for partition STOCK01:
connect tpcc/tpcc
alter session enable parallel dml;
alter index k_stock
rebuild partition idx_pk_stock01
tablespace ts_fast_idx_stock01
nologging parallel 32;
4. Compute statistics
For example, compute statistics for partition STOCK01 and its index:
10
connect tpcc/tpcc
analyze table STOCK estimate statistics
partition STOCK01
for all indexes;
5. Done.
Index
Segment
1. Create tablespaces in Dorado storage path
For example, create an AUTOEXTEND BIGFILE tablespace on Dorado ASM disk group:
connect / as sysdba
create bigfile tablespace ts_fast_idx_order
datafile '+DRD1/ts_fast_idx_order' size 128M reuse
autoextend on next 128m maxsize unlimited
extent management local
segment space management auto
nologging;
2. Rebuild the index on the new tablespace
connect tpcc/tpcc
alter session enable parallel dml;
alter index K_OORDER
rebuild tablespace ts_fast_idx_order
nologging parallel 32;
3. Compute statistics
connect tpcc/tpcc
analyze index K_OORDER estimate statistics;
4. Done.
Index
Partition
1. Create tablespaces in Dorado storage path
For example, create an AUTOEXTEND BIGFILE tablespace on Dorado ASM disk group:
connect / as sysdba
create bigfile tablespace ts_fast_idx_stock01
datafile '+DRD1/ts_fast_idx_stock01' size 128M reuse
autoextend on next 128m maxsize unlimited
extent management local
segment space management auto
nologging;
2. Rebuild the index partition on the new tablespace
For example, rebuild index IDX_PK_STOCK01:
connect tpcc/tpcc
alter session enable parallel dml;
11
alter index K_STOCK
rebuild partition IDX_PK_STOCK01
tablespace ts_fast_idx_stock01
nologging parallel 32;
3. Compute statistics
connect tpcc/tpcc
analyze index K_STOCK estimate statistics
partition IDX_PK_STOCK01;
4. Done.
Redo Logs
1. Find the current redo log groups and files
SELECT GROUP#,THREAD#,MEMBERS,BYTES/1024/1024 AS SIZE_MB FROM V$LOG;
SELECT GROUP#,MEMBER FROM V$LOGFILE;
2. Add new redo log files in Dorado storage path
ALTER DATABASE ADD LOGFILE THREAD 1 '+DRD0/redo11.log' size 512M;
ALTER DATABASE ADD LOGFILE THREAD 1 '+DRD0/redo12.log' size 512M;
ALTER DATABASE ADD LOGFILE THREAD 1 '+DRD0/redo13.log' size 512M;
ALTER DATABASE ADD LOGFILE THREAD 1 '+DRD0/redo14.log' size 512M;
ALTER DATABASE ADD LOGFILE THREAD 2 '+DRD0/redo21.log' size 512M;
ALTER DATABASE ADD LOGFILE THREAD 2 '+DRD0/redo22.log' size 512M;
ALTER DATABASE ADD LOGFILE THREAD 2 '+DRD0/redo23.log' size 512M;
ALTER DATABASE ADD LOGFILE THREAD 2 '+DRD0/redo24.log' size 512M;
3. Switch current log file to new log
For each thread, execute switch log file statement until current log file is the new created log, and
then execute checkpoint:
alter system switch logfile;
alter system switch logfile;
alter system switch logfile;
alter system switch logfile;
12
alter system checkpoint;
4. Drop the old log groups
alter database drop logfile group 1;
alter database drop logfile group 2;
alter database drop logfile group 3;
alter database drop logfile group 4;
alter database drop logfile group 5;
alter database drop logfile group 6;
alter database drop logfile group 7;
alter database drop logfile group 8;
5. Done.
Undo
Tablespaces
1. Create undo tablespaces for each thread in Dorado storage path
create bigfile undo tablespace ts_undo1
datafile '+DRD0/ts_undo1' size 4g reuse
autoextend on next 128m maxsize unlimited;
create bigfile undo tablespace ts_undo2
datafile '+DRD0/ts_undo2' size 4g reuse
autoextend on next 128m maxsize unlimited;
alter system set undo_tablespace='ts_undo1' sid='hwdb1' scope=spfile;
alter system set undo_tablespace='ts_undo2' sid='hwdb2' scope=spfile;
2. Restart all instances
srvctl stop database -d hwdb
srvctl start database -d hwdb
3. Done.
Temporary
Tablespace
1. Create temporary tablespace in Dorado storage path
create bigfile temporary tablespace ts_temp_tpcc
tempfile '+DRD0/ts_temp_tpcc' size 4g reuse
autoextend on next 128m maxsize unlimited;
2. Set the default tablespace for database users
alter user tpcc temporary tablespace ts_temp_tpcc;
3. Done.
Archive Log
1. Change the archive destination parameter
alter system set log_archive_dest_1="LOCATION=+DRD2" scope=both;
2. Done.
13
Flash Back
Recovery
Area
1. Change the flash back recovery area destination parameter
alter system set db_recovery_file_dest='+DRD3' scope=both;
2. Done.
14
Best Practices for Dorado
This chapter introduces best practices when migrating Oracle objects to Dorado Storage
System.
RAID
RAIDLevel
Level
Dorado provides 4 kinds of RAID level: RAID10, RAID5, RAID0, and RAID1. Level
RAID0 provides maximum random write performance but it’s not safe. RAID10
provides double random write performance than RAID5, but it’s more expensive. When
migrating Oracle objects to Dorado, RAID5 is the best choice. But when the write
performance of RAID5 can’t fit into your database throughput, chose RAID10 or add
more SSDs to build new RAID5 groups for your database.
Write Policy
Dorado provides 3 kinds of write policies for LUN: “write through”, “write back with
cache mirroring”, and “write back without cache mirroring”. “Write through” is the
default policy, in which mode “dirty pages” are directly written to SSDs on Dorado by
DB Writers, which policy can be used for LUNs store user tables and indexes. For
LUNs store redo log files and archive logs, you could change the policy to “write
back with cache mirroring” with the help of HUAWEI technical support engineers.
In “write back” mode, “dirty pages” are written to Dorado Cache Pool, and later synced
to SSDs in the background with LRU-LIKE algorithm. The latency is very low because
blocks are only written to Dorado memory. With “cache mirroring”, each write I/O from
Oracle is first written to the cache of LUN’s owner controller, and at the same time
transferred to another controller through the mirror channel between the two controllers,
making the latency of write I/O higher than “no cache mirroring”.
Linux I/O
Scheduler
Oracle database is widely deployed on Linux operating system. There’re 4 kinds of I/O
scheduler for block devices in Linux kernel 2.6: “noop”, “anticipatory”, “deadline”, and
“cfq”. The default I/O scheduler is “cfq”, which is not suitable for Dorado. For Oracle
database, “deadline” is recommended for OceanStor T series storage systems and
15
“noop” is recommended for OceanStor Dorado storage system.
Using the following command, you can change the I/O scheduler of “/dev/sdb” to “noop” and “/dev/sdc” to “deadline”:
echo noop > /sys/block/sdb/queue/scheduler
echo deadline > /sys/block/sdc/queue/scheduler
SLC or eMLC
Two types of SSD are supported on Dorado, SLC (Single-Layer Chip) and eMLC
(Enterprise Multi-Layer Chip). SLC has much better random write performance and more
number of block erase count, but more expensive than eMLC. SLC and eMLC almost
have the same random read performance.
The write ratio of OLTP workload is typically 20% - 60%, but there’re scenarios lower
than 20%. For write intensive OLTP workload, SLC is a better choice considering
performance and erase count. For read-mostly or read-only OLTP workload, eMLC is a
better choice.
In OLAP database, the data is written once and read many times, and the updated data
is periodically loaded into database. Choosing eMLC is a better idea for OLAP workload.
16
Copyright © Huawei Technologies Co., Ltd. 2012. All rights reserved.
No part of this document may be reproduced or transmitted in any form or by any means without prior written consent of Huawei Technologies Co., Ltd.
Trademark Notice
HUAWEI,
and
are trademarks or registered trademarks of Huawei Technologies Co., Ltd.
Other trademarks, product, service and company names mentioned are the property of their respective owners.
HUAWEI TECHNOLOGIES CO., LTD.
General Disclaimer
The information in this document may contain predictive statements including, without limitation,
Huawei Industrial Base
Bantian Longgang
statements regarding the future financial and operating results, future product portfolio, new
technology, etc. There are a number of factors that could cause actual results and developments
Shenzhen 518129, P.R. China
to differ materially from those expressed or implied in the predictive statements. Therefore, such
Tel: +86-755-28780808
information is provided for reference purpose only and constitutes neither an offer nor an
www.huawei.com
acceptance. Huawei may change the information at any time without notice.
PROVIDED BY HUAWEI STORAGE PERFORMANCE LAB
17