TECHNICAL NOTES
EMC® NetWorker
Release 7.6 and later
Probe Overview and Troubleshooting
P/N 300-012-625
REV 03
January 13, 2014
This technical note describes how to resolve probe issues while performing scheduled
backups.
This document contains the following sections:
◆
◆
◆
◆
◆
Revision history ........................................................................................................ 2
Product description................................................................................................... 2
Probe process ........................................................................................................... 3
Probe issues ............................................................................................................. 5
Troubleshooting and getting help............................................................................ 11
Revision history
Revision history
This technical note was originally published in May 2011. The following table presents the
revision history of this document:
Date
Description
January 2014
Updated cross references, additional
updates.
May 2011
Various updates.
April 2011
First release of this technical note.
Product description
Note: NetWorker probe-based backups is a separate feature. The EMC NetWorker
Administration Guide provides details on probe-based backups.
When a backup is initiated, a process is initiated on the client to gather information about
the backup. The process that the NetWorker software uses to determine the directories or
the files to back up on each client is called a probe. The NetWorker server uses the output
from the various savefs probes to allocate save sessions across the clients in the backup
group.
Note: It is assumed that the server, device, group, or other parallelism values have not
been reached.
The NetWorker server allocates the save sessions as follows:
1. The server allocates one save session per client in the backup group. The order is
determined by the speed that the probe returns information.
2. If there are still save sessions available, the server allocates one save session per
client.
3. If there are still save sessions available after step 2 has completed, the server
allocates save sessions for each mount points value, up to the limit of maximum
sessions for each client and client parallelism.
2
EMC NetWorker Probe Overview and Troubleshooting
Probe process
Figure 1 on page 3 shows the interactions that occur between the NetWorker server and
client during the probe process.
Figure 1 Probe process
Probe process
The scheduled backup operation begins with the concept of worklist preparation. The
worklist is a concept used to describe the results of queries and probes performed on
both the NetWorker server and the client machines. The worklist defines the work that
must be done on a per-save-set basis. The following discussion assumes that you are
following the worklist preparation for a scheduled backup for the backup group.
The savefs program initiates a probe on the client to determine the list of save sets to be
backed up on the client. The list of save sets is used by the save program to perform the
backup.
The savefs probe uses the following criteria to determine the list of client save sets. One
save set is listed for each local mounted UNIX filesystem or Windows NT volume (with
exceptions, such as CD-ROM, tmpfs, and filesystems or volumes specified by a directive).
The NetWorker server does not attempt to back up NFS-mounted filesystems unless
directed to do so, in which case these become a separate save set.
When the savefs probe returns the list of save sets that are candidates for the backup, the
list is passed to the server. On the server, savegrp queries the NetWorker media database
for records that describe the save time and level of each save set's most recent backup.
This information becomes part of a comparison to determine the level and backup time to
use for a nonfull scheduled backup.
To perform this comparison, the NetWorker server does not use any file or save set
versions that were created during a manual backup. For example, in preparing for an
incremental backup, the server ignores any manual backups that were created since the
last scheduled backup.
EMC NetWorker Probe Overview and Troubleshooting
3
Probe process
During the comparison, the NetWorker server uses only valid save sets. If the server finds
that a save set on the client machine has no corresponding full backup recorded in the
media database, the save set is considered new and is backed up as a level full,
regardless of the scheduled backup level. This is the meaning of the NetWorker message
that reports:
lotus.asme.local:C:\ No full backups of this
save set were found in the media database; performing
a full backup
After the comparison yields a final determination of save times and levels for each save
set identified in the savefs probe, the list is passed to the save program with the indicated
backup level.
The parallelism value returned by the savefs probe depends on the client. The
savegrp/nsrjobd programs try to start as many saves on a client as parallelism allows for
that client. The saves that are actually are started depends on the client's priority and
available parallelisms (client/group/server).
The following command can be used to check the probe for a particular client:
savegrp -vv -p -c <client name> -G <group name>
Figure 2 on page 4 shows sample output of the savegrp command.
Figure 2 Output of savegrp command that checks probe for a client
4
EMC NetWorker Probe Overview and Troubleshooting
Probe issues
Probe issues
Issue: Probe abandoned
The following command output displays the issue:
C:\>savegrp -vv -p -c lotus.asme.local -G Test
90528:savegrp: lotus.asme.local:C:\
level=incr
7236:savegrp: Group will not limit job parallelism
83643:savegrp: lotus.asme.local:savefs
started
savefs -s win2k3.asme.local -c lotus.asme.local -g Test -p- l full -R
-v -F "C:\\"
7224:savegrp: lotus.asme.local:savefs NetWorker server
`win2k3.asme.local' is not in the servers file for the NetWorker
client `lotus.asme.local'
84078:savegrp: command 'savefs -s win2k3.asme.local -c
lotus.asme.local -g Test -p -l full -R -v -F "C:\\"' for client
lotus.asme.local exited with return code 1
77562:savegrp: job (96029) host: lotus.asme.local savepoint:
lotus.asme.local:savefs had SEVERE indication(s) at completion
90491:savegrp: lotus.asme.local:savefs abandoned.
Possible causes
The NetWorker server name is not in the servers file on the NetWorker client.
Check the NetWorker server daemon.raw file for this error:
NetWorker server `xyz' is not in the servers file for the NetWorker
client `abc'
Solution
Add the NetWorker server name to the servers file of the client:
1. Shut down the nsrexecd daemon or service.
2. Open the servers file in a text editor. The default installation location of this file is:
nsr/res/servers (UNIX)
<NetWorker_install_path>\nsr\res\servers (Microsoft Windows)
3. Enter the server name on a separate line.
4. Save the changes, and exit the text editor.
5. Restart the nsrexecd daemon or service.
Note: Deleting the servers file or leaving the file blank can enable any NetWorker server
from another data zone to perform the backup of the client.
EMC NetWorker Probe Overview and Troubleshooting
5
Probe issues
Issue: Probe unexpectedly exited and probe abandoned
Possible causes
◆
The NetWorker client service is not running on the client.
◆
The NetWorker client software is not installed on the client.
The first two bulletted items are usually accompanied with the following error message
in the daemon.raw file:
Program not registered
◆
◆
Name resolution from the server and client is not working.
There is a firewall between the NetWorker client and server that is blocking the
required TCP/IP ports.
◆
The NetWorker client host operating system is shut down.
◆
You are receiving errors related to nsr peer information.
Solution
1. Use the ps -ef command on UNIX systems to confirm that the nsrexecd daemon is
running. On Windows systems, ensure that the NetWorker Remote Exec service is
started through Control Panel > Services.
2. Refer to the EMC NetWorker Installation Guide for details on how to determine if the
client software is installed and for the procedure to install the client software.
3. Confirm that the name resolution between the NetWorker client and server is
configured correctly. Refer to the EMC NetWorker Administration Guide Networking
and connectivity chapter for more details.
4. Limit the TCP/IP port usage on the NetWorker server and client, as described in the
EMC NetWorker Administration Guide Firewall Support appendix, and open the
required port on the firewall.
5. Confirm that the client is operational.
6. Remove the nsr peer information on the server and affected client.
To delete a local host peer resource:
a. Connect to the NetWorker server through the NetWorker Management Console
(NMC).
b. In the Administration window, click Configuration.
c. Open the Local Hosts folder.
d. In the Local Hosts list, click the NetWorker host whose peer resource must be
deleted.
e. Right-click on the peer resource that corresponds to the NetWorker host whose
credentials were changed, and then select Delete.
Note: If this command fails with the Error: 'user <username> on machine <NetWorker
server> is not on administrator list', then the account being used to connect to the NMC
server is not a member of the NetWorker server's administrator list. The EMC NetWorker
6
EMC NetWorker Probe Overview and Troubleshooting
Probe issues
Administration Guide NetWorker Server Management chapter provides details on how to
add an account to the administrator list with the nsraddadmin program on the NetWorker
server.
Issue: Probe unexpectedly exited
The following command output displays the issue:
C:\>savegrp -vv -p -c nmmsql.asme.local -G Test
32451:savegrp:
nmmsql.asme.local:APPLICATIONS:\SqlServerWriter level=incr
7236:savegrp: Group will not limit job parallelism
32494:savegrp: nmmsql.asme.local:probe
started
savefs -s win2k3.asme.local -c nmmsql.asme.local -g Test -p -l full -R
-v -F "APPLICATIONS:\\SqlServerWriter"
92800:savegrp: Unable to determine status for job 96014,
nmmsql.asme.local:savefs: Log file 'C:\Program
Files\Legato\nsr\logs\sg\Test\123' is empty.
7341:savegrp: nmmsql.asme.local:probe unexpectedly exited.
7076:savegrp: --- Probe Summary --nmmsql.asme.local:Probe level=full, dn=-1, mx=0,
vers=pools, p=1
nmmsql.asme.local:Probe level=full, pool=Test,
save as of 4/22/2011 5:55:29 PM
nmmsql.asme.local:APPLICATIONS:\SqlServerWriter level=full, dn=-1,
mx=0, vers=pools, p=1
nmmsql.asme.local:APPLICATIONS:\SqlServerWriter level=full, pool=Test,
save as of 4/22/2011 5:55:29 PM
nmmsql.asme.local:index level=full,
dn=-1, mx=0, vers=pools, p=1
nmmsql.asme.local:index level=full,
pool=Test, save as of 4/22/2011 5:55:29 PM
7241:savegrp: nsrim run recently, skipping
Possible causes
◆
Third-party software is blocking the savefs communication between the server and a
Windows client. Such software is known to create issues with the probe process. You
can find the software by running the Process explorer and checking the stack for a
savefs.exe process. The software modifies the Windows winsock layer incorrectly.
Entries for the third-party programs can be checked by running the command netsh
winsock show catalog. The following is a partial list of known programs that create
issues with the probe process:
• "Tectia
• "Security by Wave Systems
• "Kaseya
• "Infraxs
• "3G AT&T Wireless Card Driver
• "Microsoft tspfix utility
◆
Issues exist with the name resolution from and to the NetWorker server and client.
◆
The NetWorker client or NetWorker server has multiple IP addresses.
EMC NetWorker Probe Overview and Troubleshooting
7
Probe issues
◆
Issues exist with IPv4 or IPv6 name resolution.
Solution
1. Check for and remove any third-party software installed on the client, as listed in the
Possible causes section. Alternatively, run the command netsh winsock reset to reset
the Windows winsock layer to its default settings.
2. Confirm that the name resolution between the NetWorker client and server is
configured correctly. Refer to the EMC NetWorker Administration Guide Networking
and connectivity chapter for more details.
3. If multiple IP addresses exist, create the hosts file entries to resolve a specific IP
address from both the NetWorker server and client side. The IP address that you want
to use should be the first one in the list. Name resolution should work for all the IP
addresses of the host. If you use a hosts file, the IP addresses should be in the same
order in both the NetWorker server and client hosts files.
4. Keep the name resolution in the correct order by using the hosts file.
For example, if you have a client created with FQDN (xyz.abc.com) in the NetWorker
server, create the entry in the hosts file as follows:
1.2.3.4
xyz.abc.com
xyz
5. Resolve any name resolution issues related to IPv4 or IPv6 by using the hosts file or
DNS.
Issue: Probe job had unrecoverable failure
The following command output displays the issue:
Probe job had unrecoverable failure(s), please refer to daemon.raw for
further details
Possible causes
◆
There is a firewall blocking the required ports.
◆
Issues exist as described in the daemon.raw file on the NetWorker server or client.
◆
UAC (User access control) is enabled on Windows 2008 machines.
Solution
1. Limit the TCP/IP port usage on the NetWorker server and client, as described in the
EMC NetWorker Administration Guide Firewall Support appendix, and open the
required port on the firewall.
2. Check the daemon.raw file on the NetWorker server or client for any listed probe
issues with possible solutions.
3. Disable or turn off UAC on Windows 2008 machines. Refer to the Microsoft TechNet
article at: http://technet.microsoft.com/en-us/library/cc709691(WS.10).aspx for
more details.
8
EMC NetWorker Probe Overview and Troubleshooting
Probe issues
Issue: Cannot determine status of probe process
The following command output displays the issue:
C:\>savegrp -vv -p -c nmmsql.asme.local -G Test
32451:savegrp:
nmmsql.asme.local:APPLICATIONS:\SqlServerWriter level=incr
7236:savegrp: Group will not limit job parallelism
32494:savegrp: nmmsql.asme.local:probe
started
savefs -s win2k3.asme.local -c nmmsql.asme.local -g Test -p -l full -R
-v -F "APPLICATIONS:\\SqlServerWriter"
92800:savegrp: Unable to determine status for job 96014,
nmmsql.asme.local:savefs: Log file 'C:\Program
Files\Legato\nsr\logs\sg\Test\123' is empty.
7341:savegrp: nmmsql.asme.local:probe unexpectedly exited.
7076:savegrp: --- Probe Summary --nmmsql.asme.local:Probe level=full, dn=-1, mx=0,
vers=pools, p=1
nmmsql.asme.local:Probe level=full, pool=Test,
save as of 4/22/2011 5:55:29 PM
nmmsql.asme.local:APPLICATIONS:\SqlServerWriter level=full, dn=-1,
mx=0, vers=pools, p=1
nmmsql.asme.local:APPLICATIONS:\SqlServerWriter level=full, pool=Test,
save as of 4/22/2011 5:55:29 PM
nmmsql.asme.local:index level=full,
dn=-1, mx=0, vers=pools, p=1
nmmsql.asme.local:index level=full,
pool=Test, save as of 4/22/2011 5:55:29 PM
7241:savegrp: nsrim run recently, skipping
Possible causes
◆
A local system account or an account specified in the Remote User field does not have
the required privileges.
◆
A known issue exists in NetWorker version 7.5 SP1.
◆
Issues exist with name resolution.
Solution
1. Add the local system account and the account (Domain\Acct) used to run the backup
on the client into the client's local security policy "Replace a process token level", and
restart the NetWorker Remote Exec service.
To assign user rights for the local computer:
a. Open Local Security Policy, click Start, point to Settings, click Control Panel,
double-click Administrative Tools, and double-click Local Security Policy.
b. Go to Security Settings/Local Policies/User Rights Assignments.
c. In the console tree, click User Rights Assignment.
d. In the details pane, double-click the user right that you want to change.
e. In User Rights Properties, click Add User or Group.
f. Add the user or group, and click OK.
EMC NetWorker Probe Overview and Troubleshooting
9
Probe issues
2. Confirm that the name resolution between the NetWorker client and server is
configured correctly. Refer to the EMC NetWorker Administration Guide Networking
and connectivity chapter for more details.
Issue: Probe failed
The following command output displays the issue:
C:\>savegrp -vvp -c lotus.asme.local -G Test
32451:savegrp: lotus.asme.local:C:\Temp
level=incr
7236:savegrp: Group will not limit job parallelism
32494:savegrp: lotus.asme.local:probe
started
savefs -s lotus.asme.local -c lotus.asme.local -g Test -p -l full -R -v
-F "C:\\Temp"
77567:savegrp: Test * lotus.asme.local:Probe See the file C:\Program
Files\Legato\nsr\tmp\sg\Test\sso.000001 for output of save command.
7341:savegrp: lotus.asme.local:probe failed.
7076:savegrp: --- Probe Summary --lotus.asme.local:Probe
vers=pools, p=1
lotus.asme.local:Probe
4/20/2011 9:08:43 PM
lotus.asme.local:C:\Temp
vers=pools, p=1
lotus.asme.local:C:\Temp
4/19/2011 7:47:31 PM
lotus.asme.local:index
p=1
lotus.asme.local:index
4/19/2011 7:45:37 PM
level=full, dn=-1, mx=0,
level=full, pool=Test, save as of
level=incr, dn=-1, mx=0,
level=incr, pool=Test, save as of
level=9, dn=-1, mx=0, vers=pools,
level=9, pool=Test, save as of
Possible causes
◆
◆
Issues exist in the fstab or vfstab file on Linux or UNIX operating systems.
The server network interface entry is incorrect on the Globals (1 of 2) tab of the client
properties in NMC.
Solution
1. Populate the fstab or vfstab file with all the available filesystems or mount points.
2. Correct the server network interface entry on the Globals (1 of 2) tab of the client
properties in NMC.
10
EMC NetWorker Probe Overview and Troubleshooting
Troubleshooting and getting help
Troubleshooting and getting help
EMC support, product, and licensing information can be obtained as follows:
Product information — For documentation, release notes, software updates, or
information about EMC products, go to EMC Online Support at:
https://support.emc.com
Technical support — Go to EMC Online Support and click Service Center. You will see
several options for contacting EMC Technical Support. Note that to open a service request,
you must have a valid support agreement. Contact your EMC sales representative for
details about obtaining a valid support agreement or with questions about your account.
Copyright © 2011-2014 EMC Corporation. All rights reserved. Published in the USA.
Published January 10, 2014
EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without
notice.
The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with
respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a
particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software
license.
EMC2, EMC, and the EMC logo are registered trademarks or trademarks of EMC Corporation in the United States and other countries.
All other trademarks used herein are the property of their respective owners.
For the most up-to-date regulatory document for your product line, go to EMC Online Support (https://support.emc.com).
EMC NetWorker Probe Overview and Troubleshooting
11