Author: Ahmed
From 20 Hours to 20 minutes – Cloning Ebusiness using Oracle DataGuard
PURPOSE
For customers to clone using Oracle dataguard which should speed up the process and
automating the cloning process for Releases 11i to R12
Client has two offices, head office which is in Bryanston where executive sits and office in
Roodepoort where users (2500users) are located. Oracle Ebusiness is used for Finance, human
resources and payroll. Previously clones should take the client two working days to complete.
Customer should take a full backup of the production environment and then hand over the tapes to
driver to deliver in Roodepoort. Linux Team should restore the tapes on Roodeport development
servers. This process when going smoothly takes 2 days.
The challenge we faced is that between two sites the client had a 100MB network line and each
application running in the environment had a DR server and development server. Therefore the
network was fully utilized which meant the network support would not allow large ftps over the
network. Software is available for automated cloning but due to network restrictions it would not
work in our environment.
Cloning is a process where production environment is duplicated to the development servers for
testing of patches, changes and development of custom code. Due to time that was taken to clone an
environment the client would only clone once a month. The process was frustrating for Linux
admins and oracle apps dbas, if one mistake was made or error in communication the entire
process would have to be redone.
Relational Database Consulting
Page 1
Oracle Data Guard ensures high availability, data protection, and disaster recovery for
Enterprise data. Using Oracle Data Guard we have reduced the cloning from 20 hours to 20 minutes.
Auditors had placed a requirement that functional support team should not have access to
Production system. So there was a requirement to have a daily copy of Production where functional
support team can use to support business.
We had decided to use Oracle data Guard which keeps in sync with Production with every update.
We had built the Data guard environment on the development server in Bryanston. Every
transaction which happens on Production is duplicated on Data guard environment through the
network.
We then copy the data files from Data guard environment to our TEST {TST} environment, daily
copy of production. What we have noticed, that creating this environment had brought huge
benefits











Daily copy of production which functional support team can use
Assistance with OWC with Oracle as scripts and updates can be tested with production data
immediately
We noticed a reduction in TARS as support team could test solutions from metalink
Patches can be tested immediately using Production data
No more P1 SRs
Super users can test changes to the system
Demos can be given to Directors using latest data
Training can be given with data from Production
Disaster recovery environment is available immediately
Changes can be made on TEST system and user acceptance testing can take place easy
Emergency changes can be tested
Relational Database Consulting
Page 2
Assumptions for the following are








Initial clone is done on the source system to create the xml file
Every patch which is applied to Production is applied the next day to TEST
ORACLE HOMES between the servers are at the same patch levels
We do not copy log and out files { $APPLCSF }
All custom forms and reports are copied both to TST and Production environment
If a Data file is added on the dbTier on the Target then you would have to recreate the xml
file on the dbTier on Test environment
Printers which are configured on production must be configured using the same name on
development servers
Not tested on Windows
Software Requirements
Software Component
ZIP
Unzip
Perl
R12.AD.B.DELTA.3
R12.OAM.B.3
Oracle DataGuard
Oracle Ebusiness
Version
2.3
5.53
5
9239089
8919468
11.2.0.1
11.5.10.2  R12.1.3
Before we start on the cloning process, we need to understand the components which build up
Oracle R12
Relational Database Consulting
Page 3
Below is a diagram on the components and how they are refreshed onsite
The following are screenshots on how we perform our daily clone/ copy of our production
environment
Step 1
Relational Database Consulting
Page 4
At Midnight we shutdown the data guard environment { Database Tier }
We Then copy the datafiles from DataGuard to our Test environment, we have written a script
which shuts down the current TST environment, then deletes the data files , copies the datafiles
from dataguard and then runs the clone script
Relational Database Consulting
Page 5
At this stage the Database and listener is started
Step 2 { configure the AppTier}
Relational Database Consulting
Page 6
On the apps Tier , we shutdown the system around 11:30 PM
We then clone the APPStier using the pre-clone command but using “ EOF “ Linux scripting
Clearing the cache as the old cache is not required and uses disk space
Relational Database Consulting
Page 7
Temporary files generated by Concurrent Requests and stored in the $APPLTMP
They are normally cleared by running Purge concurrent request but due to database
been refreshed daily we have to manually clean them out to save on disk space
Clearing the Apache log files will save on disk space as with overwriting the database the system
will not know of previous days log files
Changing the APPS PASSWORD
Relational Database Consulting
Page 8
Once all the steps are complete, you have a fully cloned environment from your production system
without any downtime. The clone is fully automated and no user intervention is required.
We have noticed the following cost saving to business
1. No requirements of tapes
2. No requirements for assistance of Linux admin
3. No requirements for restores
4. No human interaction , only to check logs
5. No functional support or super users on production
6. Time saved from 24 working hours to 20 minutes
7. DR available immediately
8. Data Guard environment is read only which means it can be used for reporting
9. Production Servers are up and running 24 X 7 X 365
10. Clones to DEV, UAT, Training and dba environments takes 1 hour as clones on local servers
11. Users and support team can test anything at anytime
12. Copy of production available 7 days a week.
13. Backup of production can be verified by dataguard.
14. Rman backups can be run from the DataGuard instance.
15. No need to purchase expensive software for cloning.
16. No additional licenses required
The architecture used can be implemented in any organization using Oracle eBusiness together
with dataguard
Relational Database Consulting
Page 9
Oracle Grid Control has a reporting feature which I suggest you use to monitor archives between
your Production environments to your dataguard environment. This way one can confirm if
changes have been applied onto the dataguard environment. DataGuard must be monitored
throughout the day and alerts needs to be setup if changes are not applied on dataguard
environment
Disaster recovery is as simple as 4 steps
1. Shutdown DataGuard
a. Stop the listener {lsnrctl stop dg}
b. Stop DataGuard { recover managed standby database cancel; }
Relational Database Consulting
Page 10
c. Shutdown the database {shutdown immediate; }
d. Copy the datafiles to any instance.
2. On the DB server run post-clone
a. cd $ORACLE_HOME/appsutil/clone/bin { taking into account you know the
locations}
b. perl adcfgclone.pl dbTier { configure your dbTier}
3. On your Apps Server { we normally use our TST environment or have a dedicated disk }
a. cd $COMMON_TOP/clone/bin { taking into account you know the locations}
b. perl adcfgclone.pl appsTier
4. Users can login onto your DR/ DataGuard environment
5. Update your log and out files locations
updateapps.fnd_concurrent_requests
setlogfile_node_name = <New APPS SERVER>,
outfile_node_name = <New APPS SERVER>
where logfile_node_name = <OLD APPS SERVER>
and outfile_node_name = <OLD APPS SERVER>;
House keeping– Be ProActive -----Due to environment being refreshed daily, log files need to be cleaned out as they are
not required and use a lot of disk space, regular housekeeping both on production
and development instances will assist the dba in not running out of disk space and
monitoring the environments for any errors
On the Database server for cloning
1. Clean out the inventory [rm -rf /home/oratst/oraInventory/*]
2. Clean out all the incident files
[rm -rf /tstdb001/11.2/TST/admin/tst_ebusdb/diag/rdbms/tst/tst/incident/*]
3. Clear the listener log file
[>
/tstdb001/11.2/TST/admin/tst_ebusdb/diag/tnslsnr/ebusdb/tst/trace/tst.log]
4. Delete the xml files created by incidents
[rm /tstdb001/11.2/TST/log/diag/tnslsnr/ebusdb/tst/alert/*.xml]
5. Clear alert log and trace files
[ rm –rfrm /tstdb001/11.2/TST/admin/tst_ebusdb/diag/rdbms/tst/tst/trace/* ]
6. Delete all temp files created in utl_file_dir
[rm /admin/utl_file_dir/tst/*tmp]
Visual diagram for database Tier
Relational Database Consulting
Page 11
On the Application server
1. Clean out the inventory [rm -rf /home/appltst/oraInventory/*]
2. Delete all the cache
[rm
/tstapp001/tstapp/inst/apps/tst_ebusapp/logs/ora/10.1.2/reports/cache/*]
3. Clear out all the APPLTMP
[ rm -rf /tstapp001/tstapp/inst/apps/tst_ebusapp/appltmp/*]
4. Delete all your $APPLCSF { I need one days of files}
find /tstapp001/tstapp/inst/apps/tst_ebusapp/logs/appl/conc/log -mtime
+1 -exec rm -f {} \;
find /tstapp001/tstapp/inst/apps/tst_ebusapp/logs/appl/conc/out -mtime
+1 -exec rm -f {} \;
5. Clear out the Apache log files
find /tstapp001/tstapp/inst/apps/tst_ebusapp/logs/ora/10.1.3/Apache mtime +1 -exec rm -f {} \;
Relational Database Consulting
Page 12
Visual diagram for Application Tier
Additional Customer Steps Can be added
1.
2.
3.
4.
5.
Scrambling Salary information, so no salary can be viewed by functional support staff
Updating all open workflows to closed
Change database passwords { system – for patching }
Setting Workflow over ride address, so users don’t get workflows from duplicate instance
Updating site parameters
Example below is SiteName
DECLARE
statboolean;
BEGIN
dbms_output.disable;
dbms_output.enable(100000);
stat := FND_PROFILE.SAVE('SITENAME', '****DAILY CLONE OF
PRODUCTION****TST******DAILY CLONE OF PRODUCTION***Data from
'||to_char(trunc(sysdate)-1,'DD-MON-YYYY
')||'(12:00AM)*****TST**', 'SITE');
IF stat THEN
dbms_output.put_line( 'Stat = TRUE - profile updated' );
Relational Database Consulting
Page 13
ELSE
dbms_output.put_line( 'Stat = FALSE - profile NOT updated' );
END IF;
commit;
END;
/
6. Granting responsibilities to functional support team
7. Resetting functional support user’s passwords.
REFERENCES
1. Cloning Oracle Applications Release 12 with Rapid Clone [ID 406982.1]
ABOUT THE AUTHOR
Ahmed Jassat has over 14 years of experience in working as an Oracle APPS DBA; he has done
numerous implementations in the financial services, mining, manufacturing, information
technology and healthcare sectors. He works as being Pro-active is improving customer experience
in the Oracle Ebus suite. Ahmed is a member of the Oracle customer advisory board and has
presented at the SAOUG 2011& SAOUG 2012, Ahmed has achieved best speaker at SAOUG 2012. He
has mentored and trained clients around South Africa. Ahmed’s white papers have been recognized
globally and he has been interviewed by Oracle Media network, Oracle Customer success and
Oracle Advanced customer services. He is passionate on being PRO-Active and resolving issues
before they occur.
Relational Database Consulting
Page 14