CH A P T E R
5
Managing Databases and the Comma Separated
Value Repository
Introduction
This chapter explains how to use utility scripts to manage the Collection Manager (CM) database and
the comma separated value (CSV) repository.
Note
For instructions on using utility scripts, see the “Using Utility Scripts” section on page 4-2.
•
Common Database Management Tasks, page 5-2
•
Managing the Bundled Sybase Database, page 5-8
•
Managing the External MySQL Database, page 5-10
•
Partitioning the MySQL Database, page 5-11
•
Managing the Oracle Database, page 5-13
•
Aggregating Database Data, page 5-14
•
Managing the CSV Repository, page 5-18
Cisco SCMS Collection Manager User Guide
OL-24186-03
5-1
Chapter 5 Managing Databases and the Comma Separated Value Repository
Common Database Management Tasks
Common Database Management Tasks
The following database management tasks apply to all the supported databases:
•
Generating a list of the database tables
•
Defining and applying the schedule for the periodic deletion of old records
•
Deleting a table
•
Manually deleting old records from a table
The Cisco Service Control Management Suite (SCMS) Collection Manager adds a timestamp to every
raw data record (RDR) that it receives and stores in the database. This timestamp is used when various
maintenance operations are performed on the database tables.
Use the following scripts to configure and maintain the database:
•
~scmscm/scripts/dbtables.sh --rdr
•
~scmscm/scripts/dbperiodic.sh --rdr
•
~scmscm/db_maint/create_periodic_del_procs.sh
•
~scmscm/scripts/droptable.sh --rdr
•
~scmscm/scripts/prunetable.sh --rdr
•
~scmscm/scripts/sybutfsupport.sh
•
~scmscm/scripts/upgradeDB.sh
•
~scmscm/scripts/partitions.sh --rdr
•
~scmscm/scripts/createaggevents.sh --rdr
Listing Database Tables
To display a list of all of the tables in the database, run the dbtables.sh script:
~scmscm/scripts/dbtables.sh --rdr
Where applicable, the script displays the number of lines in the table and the earliest and latest
timestamps.
Use the Cisco Service Control Application (SCA) Reporter to display the actual content of the tables.
For more information, see Cisco Service Control Application Reporter User Guide.
Table 5-1 lists the dbtables.sh script options.
Table 5-1
dbtables.sh Options
Option
Description
-l
Lists the existing table names only (without statistics).
-c
Lists the existing table names with their record counts.
-a
Includes the nonreport tables in the listing.
-f
Enables fast line counting, using the client rather than JDBC.
-t { sec_num }
The maximum waiting time, in seconds, for the response. The default is no
timeout.
-h
Prints this help message and exits.
Cisco SCMS Collection Manager User Guide
5-2
OL-24186-03
Chapter 5 Managing Databases and the Comma Separated Value Repository
Managing the CM:RDR Periodic Deletion of Old Records
The dbtables.sh script generates the following sample output:
>~scmscm/scripts/dbtables.sh --rdr -c
Executing query ...
name| num_lines|
min_time|
max_time|
----------------+--------+------------------------+------------------------+
RPT_SUR| 131000| 2006-10-30 16:46:42.24| 2007-02-15 12:00:32.216|
RPT_LUR| 170000| 2007-04-10 15:25:45.31| 2007-04-11 07:06:05.45|
RPT_VLUR|
4694| 2007-04-11 13:12:39.683| 2007-04-11 13:18:07.396|
RPT_PUR| 116000| 2007-04-09 04:45:55.566| 2007-04-11 07:44:09.186|
RPT_TR|
57766| 2007-04-11 13:12:39.683| 2007-04-11 13:18:07.396|
RPT_MALUR| 109000| 2007-04-09 04:46:35.903| 2007-04-09 13:32:18.42|
RPT_MEDIA| 120000| 2007-04-05 17:14:24.443| 2007-04-11 13:16:29.436|
RPT_TOPS_PERIOD0| 194250| 2007-03-18 20:00:00.01| 2007-04-23 06:00:00.16|
RPT_TOPS_PERIOD1|
46940| 2007-03-19 00:00:00.05|
2007-04-23 00:00:00.1|
Managing the CM:RDR Periodic Deletion of Old Records
To manage the periodic deletion of old records, perform the following steps:
1.
If they were not installed during the Collection Manager installation, install the periodic delete
procedures:
Log in as the scmscm user, start the Collection Manager, wait 1 - 2 minutes for the creation of the
database tables, and then run the script:
~scmscm/db_maint/create_periodic_del_procs.sh. --rdr
Caution
2.
Edit the periodic delete configuration file. See the “Configuring Periodic Delete” section on
page 5-4.
3.
To apply the new configuration, run the dbperiodic.sh utility script. See the “Applying the Periodic
Delete Configuration File” section on page 5-5.
If you enable both aggregation and the periodic delete feature, be sure to increase their respective config
values.
Periodic deletion of a table does not begin while a previous periodic deletion is still running. This
prevents excessive load on the database, which would degrade insertion performance in the adapters.
When two or more tables are scheduled to be reduced at the same time, the tables are processed in the
order in which they are listed in the periodic delete configuration file.
For ease of configuration, you can schedule periodic deletion for all tables consecutively on one
schedule.
All periodic delete activity is recorded in the system log file. For Solaris, the system log file is located
in the /var/adm/messages directory; for Linux, the system log file is located in the /var/log/messages
directory. The following is an example of the system log file at the time when the periodic delete action
occurs.
Feb 17 05:00:07 app-redhat79 logger:
running single del for RPT_LUR to 14
Feb 17 05:00:09 app-redhat79 logger:
Feb 17 05:00:09 app-redhat79 logger:
Note
/opt/scmscm/db_maint/periodic_del.sh for hourly all days
Starting periodic delete for RPT_LUR keeping 14 days
Deleted rows: 0 from the table RPT_LUR
Only MySQL database Version 5, or later, supports periodic delete.
Cisco SCMS Collection Manager User Guide
OL-24186-03
5-3
Chapter 5 Managing Databases and the Comma Separated Value Repository
Managing the CM:RDR Periodic Deletion of Old Records
Caution
Be sure to disable the MySQL partitioning process in Collection Manager when you enable the periodic
delete feature.
•
Configuring Periodic Delete, page 5-4
•
Applying the Periodic Delete Configuration File, page 5-5
Configuring Periodic Delete
By default, the periodic delete configuration file (dbperiodic.conf) is located in the directory
~scmscm/db_maint/. The file has a structure similar to an INI file, where each section describes a
particular data reduction operation for a specific set of tables, to be performed according to a specified
schedule.
Note
The name of each section of the file is not used when the file is parsed. Use the names you wish.
Each section begins with the section name in brackets, and contains the parameters shown in the
following table. (Not all parameters are required in each section of the configuration file.) Separate the
parameters and their values by an equal sign (=). Table 5-2 contains examples of periodic delete
configuration files.
Table 5-2
Parameters in the Periodic Delete Configuration File
Parameter Name
Explanation
active
Default
Example
Specifies whether or not true/false
to use this section of the
configuration file
true
false
tablenames
Names of the tables to
which this section
applies
* (all)
RPT_SUR,RPT_LUR
daystokeep
Number of days to keep Positive integers
records
14
30
minute
The time at which to
perform the deletion in
this section of the
configuration file
0 … 59, *
0*
0
0 … 23, *
(all) *
0,4,8,12,16,20
1 … 31, *
(all) *
1
1 … 12, *
(all)
1,3,5,7,9,11
hour
day
month
Note
Values
Names of tables
separated by commas,
or * for all tables
Values for all parameters except active and daystokeep can be either a single value, a list of values
separated by commas, a range of values (two values separated by a dash), or an asterisk (*) which
signifies all possible values. A range is not possible for tablenames.
Cisco SCMS Collection Manager User Guide
5-4
OL-24186-03
Chapter 5 Managing Databases and the Comma Separated Value Repository
Managing the CM:RDR Periodic Deletion of Old Records
In the following example, all fields are set to their default values.
# This dbperiodic.conf file emulates the legacy style for periodic
# deletion. All tables are processed every hour on the hour, and
# records are kept for 14 days.
[hourly all]
active = true
tablenames = *
daystokeep = 14
minute = 0
hour = *
In this example, all tables are reduced at 4:30 A.M., leaving 10 days of data in each table. In addition,
the real-time tables are reduced every hour, leaving three days of data in each table.
# This dbperiodic.conf file reduces all tables once a day and
# real-time tables once an hour.
[daily all]
active = true
tablenames = *
daystokeep = 10
minute = 30
hour = 4
[hourly real-time]
active = true
tablenames = RPT_SUR,RPT_LUR,RPT_PUR
daystokeep = 3
minute = 0
hour = *
Applying the Periodic Delete Configuration File
To load and apply a new periodic delete configuration file or to view the current file, run the
dbperiodic.sh script:
~scmscm/scripts/dbperiodic.sh --rdr [ --dump ] [ --load |
--loadfile=path_to_dbperiodic.conf ]
When the script is used to load a new configuration file, it parses the file, verifies its validity, and updates
the scmscm user's crontab to reflect the changes. Table 5-3 lists the dbperiodic.sh options.
Table 5-3
dbperiodic.sh Options
Option
Description
--load
Load the periodic delete configuration from
/export/home/scmscm/db_maint/
dbperiodic.conf
--loadfile=path to periodic delete
configuration file
Load the periodic delete configuration file from
the specified directory
--dump
Print the periodic delete configuration
--h
Display these options
The following example shows how to print the current periodic delete configuration:
~scmscm/scripts/dbperiodic.sh --dump
Cisco SCMS Collection Manager User Guide
OL-24186-03
5-5
Chapter 5 Managing Databases and the Comma Separated Value Repository
Deleting a Table
Note
This script prints the loaded periodic delete configuration. If the current periodic delete configuration
file is not yet loaded, the configuration can vary from the script output.
The following example shows how to load the periodic delete configuration file from
~scmscm/db_maint/dbperiodic.conf:
~scmscm/scripts/dbperiodic.sh --rdr --load
The following example shows how to load the periodic delete configuration file from a specified
location:
~scmscm/scripts/dbperiodic.sh --rdr --loadfile=path_to_periodic_delete_configuration_file
Deleting a Table
To delete one table or all current tables from the database, run the droptable.sh script:
~scmscm/scripts/droptable.sh --rdr[-f] tableParameter
Table 5-4 lists the droptable.sh options.
Table 5-4
droptable.sh Options
Option
Description
table_name
Drop table_name from the database
ALLTABLES
Drop all tables from the database
-f
Drop by force (no questions asked or errors
reported)
-h
Display these options
The following example shows how to drop a table named RPT_SUR from the database with no request
for confirmation:
~scmscm/scripts/droptable.sh --rdr –f RPT_SUR
The following example shows how to drop all tables from the database:
~scmscm/scripts/droptable.sh --rdr ALLTABLES
Deleting Old Records
To remove records from a database table based on the timestamps of the records, run the prunetable.sh
script:
~scmscm/scripts/prunetable.sh --rdr [ -f ] num_days table_name
Cisco SCMS Collection Manager User Guide
5-6
OL-24186-03
Chapter 5 Managing Databases and the Comma Separated Value Repository
UNICODE Support for Sybase Database
Table 5-5 lists the prunetable.sh options.
Table 5-5
prunetable.sh Options
Option
Description
num_days
The maximum age (in days) of records not to
deleted.
table_name
The table for which to delete records
-f
Drop by force (no questions asked or errors
reported).
-h
Display these options.
The following example shows how to delete all records that are more than seven days old from a table
named RPT_SUR.
Because the –f flag is not specified, there can be requests for confirmation and errors are reported.
>~scmscm/scripts/prunetable.sh --rdr 7 RPT_SUR
UNICODE Support for Sybase Database
To add the UCS Transformation Format—8-bit (UTF-8) support for the Sybase database:
Step 1
Login to Cisco Service Control Collection Manager as a root user.
Step 2
Run the ~scmscm/scripts/sybutfsupport.sh script.
The script restarts the Sybase after enabling the UTF-8 support in the database.
Step 3
Restart the Cisco Service Control Collection Manager.
The UTF-8 support is enabled on the Cisco Service Control Collection Manager.
Cisco SCMS Collection Manager User Guide
OL-24186-03
5-7
Chapter 5 Managing Databases and the Comma Separated Value Repository
Managing the Bundled Sybase Database
Managing the Bundled Sybase Database
Managing the bundled database includes:
•
Backing up and restoring a database
•
Updating Sybase with a changed IP address
•
Updating the Sybase database statistics
The Cisco Collection Manager adds a timestamp to each RDR it receives and stores in the database. This
timestamp is used when various maintenance operations are performed on the database tables.
Use the following scripts to maintain the bundled Sybase database only:
•
~scmscm/scripts/sybback.sh --rdr
•
~scmscm/scripts/sybrestore.sh --rdr
•
~scmscm/db_maint/update_statistics.sh
Backing up the Database
In Release 3.7.0 and later, by default the backup database files are stored on the Cisco Collection
Manager's home directory under the Sybase directory. An example of the home directory is
/home/scmscm/sybase.
Note
Consult the Sybase Backup Server product documentation for a complete Sybase backup. The bundled
Cisco Collection Manager backup is only designed for backing up and restoring small amounts of data,
for example, transferring small tables between servers.
Cisco SCMS Collection Manager User Guide
5-8
OL-24186-03
Chapter 5 Managing Databases and the Comma Separated Value Repository
Updating Sybase with a Changed IP Address
Updating Sybase with a Changed IP Address
When you change the IP address, you must also update the Sybase server:
Step 1
Edit the hosts.
Step 2
Change the IP address. Run the interface configuration script:
/etc/sysconfig/network-scripts/ifcfg-eth0
Step 3
Run the network restart script:
/etc/rc.d/init.d/network restart
Step 4
Update the Sybase server. As the root user, issue the following command:
~scmscm/setup/syb_interfaces.sh
Step 5
Run the Sybase stop and start script as the root user:
~scmscm/setup/sybase stop/start
Step 6
Restart the Collection Manager, run the following script as the scmscm user:
~scmscm/cm/bin/cm restart
Updating Sybase Database Statistics
Run the update_statistics.sh script once a week at a time of low database load, for example, at night.
~scmscm/db_maint/update_statistics.sh
Cisco SCMS Collection Manager User Guide
OL-24186-03
5-9
Chapter 5 Managing Databases and the Comma Separated Value Repository
Managing the External MySQL Database
Managing the External MySQL Database
Managing the MySQL database includes:
•
Backing up and restoring a database
The Cisco Collection Manager adds a timestamp to each RDR it receives and stores in the database. This
timestamp is used when various maintenance operations are performed on the database tables.
Use the following scripts to maintain the MySQL database:
•
~scmscm/scripts/mysqlback.sh --rdr
•
~scmscm/scripts/mysqlrestore.sh --rdr
Backing up the MySQL Database
To create text file backups of all the tables in the database, use the mysqlback.sh script:
~scmscm/scripts/mysqlback.sh --rdr
The script converts all tables to ASCII files and copies the files to a backup directory. Table 5-6 lists the
mysqlback.sh options.
Table 5-6
mysqlback.sh Options
Option
Description
--rdr
Write backup CM:NF sql files to the specified
directory
--help
Displays these options
Note: The destination location path is defined *.properties file. Modify the path if you want to change
the default location.
Restoring a MySQL Database
To restore a database from the backup file created by the mysqlback.sh script, run the mysqlrestore.sh
script:
~scmscm/scripts/mysqlrestore.sh --rdr
The script converts all tables to ASCII files and copies the files to a backup directory. Table 5-7 lists the
msqlrestore.sh options.
Table 5-7
mysqylrestore.sh Options
Option
Description
--rdr
Restore the CM:RDR database using the sql files
in the specified directory
--help
Displays these options
Cisco SCMS Collection Manager User Guide
5-10
OL-24186-03
Chapter 5 Managing Databases and the Comma Separated Value Repository
Partitioning the MySQL Database
Partitioning the MySQL Database
You can store records in an external MySQL database. When you configure MySQL as the database,
partitioning is supported.
•
Prerequisites, page 5-11
•
Configuring Partitioning, page 5-11
Prerequisites
The partitioning feature is supported only when the MySQL server meets the following conditions:
•
MySQL version 5.1.06 or later
•
open_files_limit value in my.cnf file should be set to 8000 for default partitioning configuration
under the [mysqld] section
•
thread_stack value in my.cnf file should be set to at least 256k under the [mysqld] section
Restart the MySQL server to enable the configuration changes to take effect.
Configuring Partitioning
Caution
If a partition is created on the existing data, all existing data is deleted . Back up existing data before
running the partitioning script.
The partitioning configuration file (partitions.conf) is located in the
~scmscm/cm/config/dbpacks/mysql/4.0.20/partitions/ directory.
The following is a sample of the configuration file:
[partitions_config1]
table_names = RPT_LUR,RPT_TOPS_PERIOD1,RPT_FUR,RPT_GUR
no_of_partition = 30
period = 1
scheduler_time = 23:30
Each section begins with a unique section name starting with partitions_config in brackets and contains
the parameters shown in Table 5-8. Separate the parameters and their values by an equal sign (=).
Table 5-8
Parameters in the Partitioning Configuration File
Parameter Name
Explanation
Values
Default
Example
table_names
Name of the table to
partition.
Start the table name
with the prefix RPT_.
Should not be
duplicated across
sections.
—
RPT_LUR
no_of_partition
The number of
partitions to create in
the database.
1 - 65.
—
30
Cisco SCMS Collection Manager User Guide
OL-24186-03
5-11
Chapter 5 Managing Databases and the Comma Separated Value Repository
Configuring Partitioning
Table 5-8
Parameters in the Partitioning Configuration File (continued)
Parameter Name
Explanation
Values
period
The number of days that 1 - 5.
the records are retained
in each partition.
scheduler_time
The time at which to
trigger the daily
partitioning event
scheduler in the
database.
Hour and minutes.
Format: HH:MM
Default
Example
—
1
—
23:30
To create and view partitions, run the partitions.sh script:
~scmscm/scripts/partitions.sh --rdr [ --create ] [ --non_partitioned ] [ --partitioned ]
[--partitions [table_name]] [events][ --help ]
Table 5-9 lists the partitions.sh options.
Table 5-9
partitions.sh Options
Option
Description
--create
Create partitions for database tables that are not
partitioned
--non_partitioned
Show details for database tables that are not
partitioned
--partitioned
Show details for database tables that are
partitioned
--partitions [table_name]
Show all table partitions details. [table_name]
parameter shows mentioned table partitioned
details.
--events
Show configured partitioning events details
--help
Display these options
Perform the following steps to enable the partitioning feature:
Note
1.
Log in as scmscm user.
2.
Change directory to ~scmscm/scripts/.
3.
Customize the ~scmscm/cm/config/dbpacks/mysql/4.0.20/partitions/partitions.conf file based on
your requirements.
4.
Run ~scmscm/scripts/loadsqlprocedures.sh --rdr --par to load the required support files for
partitioning.
5.
Run ~scmscm/scripts/partitions.sh --rdr --create to create the partitioning.
Ensure that the periodic deletion process is disabled in Collection Manager when you enable the
partitioning feature.
Cisco SCMS Collection Manager User Guide
5-12
OL-24186-03
Chapter 5 Managing Databases and the Comma Separated Value Repository
Managing the Oracle Database
During the creation of the partitions, the partitions.sh script performs the following actions:
•
Enables the event scheduler on the MySQL server.
•
If any of the tables (table_names) listed in the configuration file (partitions.conf) are not available
in the database, partitioning is ignored for that particular table. The remaining tables are processed.
When the MySQL server is restarted after the partitioning, the event scheduler is disabled. To ensure the
event scheduler remains enabled through a server restart, add “event_scheduler = 1” to the [mysqld]
section of the my.cnf file.
Note
If any missing table is created in the database later, run the partitions.sh script to create new partitioning.
You can apply the same event scheduler for the new partitioning in each configuration section. This does
not disturb the existing partitioning.
Note
If any new tables are created in the database, define new configuration sections for these new tables and
then run the partitions.sh script. This does not disturb the existing partitioning.
Although the partitioning event scheduler runs every day, it to creates or deletes a partition based on the
configuration parameters no_of_partition and period. Even if the server is restarted, the partitioning
event scheduler is not disturbed.
Managing the Oracle Database
Oracle Support with Minimum Privileges
The following are the mandatory Privileges that are to be granted to the CM-DB user: Create Session ,
Create Table, Create Procedure, Create Table Space, Unlimited Table Space, Create Index, Execute Any
Procedure
Syntax:
grant CONNECT ,CREATE TABLE ,CREATE TABLESPACE ,CREATE PROCEDURE ,UNLIMITED
TABLESPACE,CREATE ANY INDEX, EXECUTE ANY PROCEDURE to pqb_admin
Note
Connect to the Oracle DB as SYSTEM/SYSDBA, log into the Oracle server with specific schema
(apricot/avocado) and grant the above privileges
Cisco SCMS Collection Manager User Guide
OL-24186-03
5-13
Chapter 5 Managing Databases and the Comma Separated Value Repository
Aggregating Database Data
Aggregating Database Data
You can aggregate database data according to varying time periods and store data in the same database
tables. Both raw and aggregated data are stored in order to maintain compatibility with reporting tools
that pull this data to generate analysis reports. All reports that process raw data for display also can
process the aggregated data because the data structure is the same.
An input date parameter specifies the data records to process. These records are stored per time period.
For xUR tables, hourly aggregation is performed starting from the input date value through the following
fourteen days. Thereafter, aggregation occurs daily. For HTTP, video, and NUR VSA tables, aggregation
occurs daily.
During the aggregation process, table rows are aggregated per time period. These values are placed in a
temporary table. The source records are then removed to allow insertion of the aggregated records.
The aggregation feature is supported in the following tables:
Caution
Note
•
RPT_LUR
•
RPT_PUR
•
RPT_VLUR
•
RPT_MALUR
•
RPT_SUR
•
RPT_SPAM
•
RPT_ZUR
•
RPT_TOP_HTTP_DOMAINS
•
RPT_TOP_VIDEO_DOMAINS
•
RPT_TOP_HTTP_HOSTS
•
RPT_TOP_VIDEO_HOSTS
•
RPT_TOP_APN
•
RPT_TOP_DEVICE_TYPE
•
RPT_TOP_NETWORK_TYPE
•
RPT_TOP_USER_LOCATION
•
RPT_TOP_SGSN
•
RPT_TOP_PCF
•
RPT_TOP_HOME_ AGENT
•
RPT_TOP_MEID
The actual table records are deleted from the table based on the aggregation range values.
If both periodic deletion and the aggregation process are enabled, you can increase the number of days
that records are retained.
Cisco SCMS Collection Manager User Guide
5-14
OL-24186-03
Chapter 5 Managing Databases and the Comma Separated Value Repository
Configuring Aggregation
Configuring Aggregation
The aggregation configuration file ~scmscm/cm/config/aggregation.conf specifies the values used in the
aggregation process.
The following is an example configuration file:
[agg_event1]
procedure_names = P_RPT_MALUR_AGGREGATE, P_RPT_LUR_AGGREGATE
no_of_days = 2
scheduler_time = 23:30
Each section in the aggregation configuration file begins with a unique section name in brackets
([]) starting with agg_event . The file contains the parameters shown in Table 5-10. Separate the
parameters and their values with an equal sign (=).
Table 5-10
aggregation.conf Parameters
Parameter Name
Explanation
procedure_names
Values
Default
—
SQL procedure name to Any aggregation procedure name
aggregate.
present in the database configuration
location.
Example
P_RPT_MALUR_
AGGREGATE
Sybase:
~scmscm/cm/config/dbpacks/
sybase/ase12.5.1/aggregation/
MySQL:
~scmscm/cm/config/dbpacks/
mysql/4.0.20/aggregation/
Oracle:
~scmscm/cm/config/dbpacks/
oracle/9204e/aggregation/
no_of_days
Number of days of raw
data to retain in the
table.
1 to 31
—
2
scheduler_time
The time at which to
trigger the daily
aggregation event
scheduler.
Hour and minutes.
—
23:30
Format: HH:MM
To create a new aggregation event, run the createaggevents.sh script:
~scmscm/scripts/createaggevents.sh --rdr [ --create ] [ --help ]
Table 5-11 lists the createaggevents.sh options.
Table 5-11
createaggevents.sh Options
Option
Description
--create
Creates the corresponding aggregation events
--help
Display these options
Cisco SCMS Collection Manager User Guide
OL-24186-03
5-15
Chapter 5 Managing Databases and the Comma Separated Value Repository
Enabling Aggregation in Sybase
The Sybase, MySQL, and Oracle databases support aggregation. The following sections describe how to
enable the aggregation feature:
Note
•
Enabling Aggregation in Sybase, page 5-16
•
Enabling Aggregation in MySQL, page 5-16
•
Enabling Aggregation in Oracle, page 5-16
Use the loadsqlprocedure.sh --rdr --agg option to update the database configuration changes.
Enabling Aggregation in Sybase
Sybase versions 12 and 15 support data aggregation.
Complete the following steps to enable the aggregation feature:
1.
Login as scmscm user.
2.
Run ~scmscm/scripts/loadsqlprocedures.sh --rdr --agg to load the required support files for
aggregation.
3.
Create a cron job to run
~scmscm/cm/config/dbpacks/sybase/ase12.5.1/aggregation/cron/run_aggregation.sh to
schedule the aggregation process.
Cron job example:
crontab -e
Update the path and expected time to schedule the job:
30 23 * * * /<path>/run_aggregation.sh
4.
Refer to ~scmscm/cm/config/dbpacks/sybase/ase12.5.1/aggregation/cron/
readme_run_aggregation_procedure.txt for more information.
Enabling Aggregation in MySQL
MySQL versions 5.1.06 and later support data aggregation.
Complete the following steps to enable the aggregation feature:
1.
Login as scmscm user.
2.
Run ~scmscm/scripts/loadsqlprocedures.sh --rdr --agg to load the required support files for
aggregation.
Enabling Aggregation in Oracle
Oracle versions 9, 10, and 11 support data aggregation.
Complete the following steps to enable the aggregation feature:
1.
Login as scmscm user.
2.
Run ~scmscm/scripts/loadsqlprocedures.sh --rdr --agg to load the required support files for
aggregation.
Cisco SCMS Collection Manager User Guide
5-16
OL-24186-03
Chapter 5 Managing Databases and the Comma Separated Value Repository
Enabling Aggregation in Oracle
Note
Make sure the Oracle user has sufficient privileges to alter and execute the aggregation events. For
example: grant select ,insert ,delete ,update on sys.job$ to pqb_admin;
Cisco SCMS Collection Manager User Guide
OL-24186-03
5-17
Chapter 5 Managing Databases and the Comma Separated Value Repository
Managing the CSV Repository
Managing the CSV Repository
You can run a utility script to manage the repository of CSV files output by the Collection Manager. the
CSV adapter writes these files to disk for use by a service provider’s operational support system (OSS)
or by a third-party billing system. To prevent disk overflow, monitor the size of the CSV repository.
Note
If the backup parameter is set to true, failure to delete CSV files can result in disk overflow. No CSV
files are ever deleted.
Note
The third-party application is responsible for managing the CSV files and deleting them as necessary.
To successfully invoke this script, the HTTP adaptor of the Collection Manager must be running. If the
adapter is down, an error message is printed.
•
CSV Repository File Structure, page 5-18
•
Configuring the CSV File Repository, page 5-18
•
Configuring the Comma Escape, page 5-19
•
Configuring Escape of Nonprintable Characters, page 5-20
CSV Repository File Structure
CSV files are stored in several subdirectories. Each subdirectory is given the number of a Raw Data
Record (RDR) tag. RDR tags define the type of the RDR. Each RDR is stored in the subdirectory whose
name matches its RDR tag number. For more information on RDR tags, see tCisco Service Control
Application for Broadband Reference Guide.
The CSV files are numbered sequentially (automatically), with separate numbering in each directory.
You can change the location of the parent directory by editing the cm.conf file located in the cm/config
directory.
Configuring the CSV File Repository
Use the ~scmscm/scripts/csvconf.sh script to:
•
List the number of RDRs currently stored in the repository.
•
Configure the maximum number of CSV files and the maximum permissible number of reports
(lines) in each file.
•
Control whether a backup occurs when an old CSV file is about to be overwritten.
•
Control whether each line in a CSV file contains an indication of the IP of the Service Control
Engine (SCE) that sent this RDR. (By default, this option is off.)
Cisco SCMS Collection Manager User Guide
5-18
OL-24186-03
Chapter 5 Managing Databases and the Comma Separated Value Repository
Configuring the Comma Escape
Table 5-12 lists the csvconf.sh options.
Note
Instead of using this script, you can edit the ~scmscm/cm/config/csvadapter.conf file. For changes in
this file to take effect, you must restart the Collection Manager.
Note
The same configuration is applied to all subdirectories in the CSV repository.
Note
Setting these parameters does not change existing CSV files. Parameter value changes only affect files
that are created subsequently.
Table 5-12
csvconf.sh Options
Option
Description
--list
Display the CSV repository contents (the number of RDRs currently
stored in the repository).
--clear
Delete all files from the CSV repository. (This option deletes all
CSV files, but not the directories in which they are contained.)
--maxlines=N
Set the maximum number of RDRs per CSV file to N (an integer
between 1 and 20,000).
--maxfiles=M
Set the maximum number of CSV files in each subdirectory to M (an
integer between 10 and 10,000.)
--backups={true |false}
Enable or disable backup of old CSV files.
--recordsource={true |false}
Enable or disable the inclusion of the record source in CSV files.
The following example shows how to set the maximum number of CSV files per subdirectory to 1000.
>~scmscm/scripts/csvconf.sh --maxfiles=1000
The following example shows how to set the maximum number records per CSV files to 10,000.
>~scmscm/scripts/csvconf.sh --maxlines=10000
The following example shows how to delete all files from the CSV repository:
~scmscm/scripts/csvconf.sh --clear
The following example shows how to disable backing up old CSV files in the repository:
~scmscm/scripts/csvconf.sh –-backups=false
Configuring the Comma Escape
When a comma is contained within a field in a CSV file, an escape sequence indicates that the comma
does not mark the end of the field.
Cisco SCMS Collection Manager User Guide
OL-24186-03
5-19
Chapter 5 Managing Databases and the Comma Separated Value Repository
Configuring Escape of Nonprintable Characters
Three escape methods are supported:
•
Single quotation marks—Single quotation marks surround any field that contains one or more
commas. There is no special treatment of single quotation marks that currently exist in the RDRs.
•
URL—%2C replaces each comma contained within a field. There is no special treatment of such
sequences that currently exist in RDRs.
•
Backslash—A backslash (\) precedes each comma contained within a field. There is no special
treatment of backslashes that currently exist in RDRs.
The first two escape methods are compatible with Microsoft Excel. The backslash method is not
compatible with Excel, but is retained for backward compatibility.
Single quotation marks are used by default. You can change the escape method by modifying the value
of the escapeMethod attribute. This attribute is located in the ~/scmscm/cm/config/csvadapter.conf file.
You must use one of the following values: backslash, quote, or url.
Configuring Escape of Nonprintable Characters
The CSV adapter can escape nonprintable characters. Enabling this option incurs a performance penalty
for the adapter. By default, the option is disabled.
When the option is enabled, a backslash (\) precedes each character that cannot be printed , such as CR
and LF, contained within a field.
Enable this option in the ~/scmscm/cm/config/csvadapter.conf file. For changes to this configuration file
to take effect, you must restart the Collection Manager.
Cisco SCMS Collection Manager User Guide
5-20
OL-24186-03