IBM® Endpoint Manager Support Open Mic
27 August 2015
IBM BigFix: Troubleshooting <not reported> Actions
Presenter:
Adam McDonald - IBM BigFix L2 Support Engineer
Panelists:
Aram Eblighatian - IBM BigFix Solutions Architect
Daniel Hwang - IBM BigFix Platform Infrastructure Engineer
Brian Green - IBM BigFix Software Engineer
Murtuza Choilawala - Program Director, Product Management & Strategy, Endpoint Manager and Mobility Solutions
Reminder:
You must dial-in to the phone conference to listen to the panelists.
The web cast does not include audio.
●
USA toll-free: 866-803-2141
●
USA toll: 1-203-607-0460
●
Participant passcode: 4558910
Slides and additional dial in numbers:
http://www.ibm.com/support/docview.wss?uid=swg27046281
NOTICE: By participating in this call, you give your irrevocable consent to IBM to
record any statements that you may make during the call, as well as to IBM's use of
such recording in any and all media, including for video postings on YouTube.
If you object, please do not connect to this call.
© 2015 IBM Corporation
Symptom of <not reported> action
© 2015 IBM Corporation
A <not reported> action


An action that appears in the console as not having an action Status or one that has a Status
of <not reported>
An action for which there is no information about its current Status.
© 2015 IBM Corporation
3
Lifecycle of an action
© 2015 IBM Corporation
What must happen for an action to complete successfully
1. Console operator takes action
2. Server puts action in database
3. Database propagator mirrors and increments new site to
wwwrootbes
4. Server notifies top level relays of new site (TCP)
5. Relays notify other relays of new site (TCP)
6. Relays notify clients of new site (UDP)
7. Client requests and gathers new site from relay
8. Client merges new site
9. Client evaluates new site (is it relevant?)
10. If action is relevant, it is executed.
11.Client generates report
12.Client posts report to parent relay
13. Parent relay posts report to its parent relay...etc...
14. Top level relay posts report to Server
15. Server FillDB inserts report into Database
16. Console refreshes displaying the updated action status
© 2015 IBM Corporation
5
Some BigFix Basics
© 2015 IBM Corporation
BigFix Network Traffic
●
Notifications:
–
–
–
–
●
Gathering:
–
–
–
●
Server to Relay: TCP port 52311
Relay to Relay: TCP port 52311
Server to Client: UDP port 52311
Relay to Client: UDP port 52311
Relay gathers from Server: HTTP port 52311
Relay gathers from Relay: HTTP port 52311
Client gathers from Relay: HTTP port 52311
Reporting:
–
–
–
–
Client posts to Relay: HTTP port 52311
Relay posts to Relay: HTTP port 52311
Relay posts to Server: HTTP port 52311
Server (FillDB) inserts into Database: ODBC
port 1433
© 2015 IBM Corporation
7
Understanding Actions & Action Sites
Three types of sites:
§ actionsite:
– All actions taken by a master operator and targeted at endpoints using "Dynamically targeted by
property"
– All client endpoints are mandatorily subscribed to the actionsite
§ opsites:
– All actions taken by a non-master operator and targeted at endpoints using "Dynamically targeted by
property"
– Client endpoints are subscribed to a non-master operator's opsite if the endpoint is managed by the
non-master operator.
§ mailboxsite:
– All actions taken by either a master or non-master operator and targeted at endpoints using either
“Selected devices” or “Enter device names”
– Mailbox files are created on the mailboxsite specific for the endpoints targeted every time an action is
taken using this type of targeting.
© 2015 IBM Corporation
8
Understanding Actions & Action Sites
Note: Mailboxing was introduced in version 9.0 of Endpoint Manager. If some of the endpoints included in the "Select devices"
or "Enter device names" targeting have a client version less than 9.0, the action will not deploy via the mailboxsite; it will deploy
either via the actionsite (if taken by a master operator) or via an opsite (if taken by a non-master operator).
© 2015 IBM Corporation
9
Determining a non-master operator's opsite
§ In the console,
– Navigate to: All Content > Sites > Operator Sites
– In the right side window, right-click on the column headings and activate the URL column
– The name of the operator's opsite is at the end of the URL
© 2015 IBM Corporation
10
Site and Action File Locations
§ Server:
– actionsite and opsites:
●
\BES Server\wwwrootbes\bfsites\
– mailboxsite:
●
\BES Server\wwwrootbes\mailbox\files
§ Relays:
– actionsite and opsites:
●
\BES Relay\wwwrootbes\bfmirror\bfsites\manymirror_#_###
– mailboxsite:
●
\BES Relay\wwwrootbes\mailbox\files
§ Client:
– actionsite, opsites, and mailboxsite
●
\BES Client\__BESData\mailboxsite
●
\BES Client\__BESData\actionsite
●
\BES Client\__BESData\opsite{###}
© 2015 IBM Corporation
11
Checking the Client
© 2015 IBM Corporation
Checking for mailbox actions on the client
§ \BES Client\__BESData\mailboxsite
– The following active actions (1042, 1044, and 1065) are sitting in the mailboxsite folder on the client as
“Action .fxf” files.
– Note: An earlier action (1049) is not there; as the action has already expired, and has been cleaned up
from the endpoint's mailboxsite. If the action were still active, “Action 1049.fxf” would be in this directory.
© 2015 IBM Corporation
13
Checking for actionsite actions on the client
§ \BES Client\__BESData\actionsite
– The following active actions (362, 1054, and 1063) are sitting in the actionsite folder on the client as
“Action .fxf” files.
© 2015 IBM Corporation
14
Checking for opsite actions on the client
§ BES Client\__BESData\opsite{###} (opsite104 in this example)
– The following active actions (1059, 1060, 1061, and 1062) are sitting in operator 104's opsite104 folder
as “Action .fxf” files.
© 2015 IBM Corporation
15
Checking the client log file for the action
Location of client log files on an endpoint:
Windows: \Program Files\BigFix Enterprise\BES Client\__BESData\__Global\Logs
UNIX/Linux: /var/opt/BESClient/__BESData/__Global/Logs
Mac: /Library/Application Support/Bigfix/BES Agent/__BESData/__Global/Logs
* 10 days worth of client logs are present in log directory.
© 2015 IBM Corporation
16
Checking the client log file for the action
Action 1063 taken by a master operator via the actionsite:
© 2015 IBM Corporation
17
Checking the client log file for the action
Action 1064 taken by a non-master operator via opsite103:
© 2015 IBM Corporation
18
Checking the client log file for the action
Action 1065 taken by a non-master operator via the mailboxsite:
© 2015 IBM Corporation
19
Checking the Server
© 2015 IBM Corporation
Checking the server for mailboxed actions
§ Checking that the mailbox action is on the server for a specific computer (for example action 1049):
select * from MAILBOX_FILES where FileID IN (select FileID from MAILBOX_COMPUTER_FILES where
ComputerID = '<COMPUTER_ID>') ORDER BY FileID DESC
© 2015 IBM Corporation
21
Checking the server for actionsite actions
§
C:\Program Files (x86)\BigFix Enterprise\BES Server\Mirror Server\Inbox\bfemapfile.xml
–
<UrlToIdPair ID="1">http://spt1-win2k8r2:52311/cgi-bin/bfgather.exe/actionsite</UrlToIdPair>
§
C:\Program Files (x86)\BigFix Enterprise\BES Server\Mirror Server\Inbox\GatherState.xml
–
<SiteStatus SiteID="1" CommittedLocation="C:\Program Files (x86)\BigFix Enterprise\BES
Server\wwwrootbes\bfsites\actionsite_0_407" CommittedVersion="407" CommittedManyVersion="2700"
Status="GatherComplete" LastRequestedTime="Sun, 23 Aug 2015 04:16:27 +0000"></SiteStatus>
© 2015 IBM Corporation
22
Checking the server for opsite actions
§
C:\Program Files (x86)\BigFix Enterprise\BES Server\Mirror Server\Inbox\bfemapfile.xml
–
<UrlToIdPair ID="68">http://spt1-win2k8r2:52311/cgi-bin/bfgather.exe/opsite104</UrlToIdPair>
§
C:\Program Files (x86)\BigFix Enterprise\BES Server\Mirror Server\Inbox\GatherState.xml
–
<SiteStatus SiteID="68" CommittedLocation="C:\Program Files (x86)\BigFix Enterprise\BES
Server\wwwrootbes\bfsites\opsite104_0_5" CommittedVersion="5" CommittedManyVersion="2698"
Status="GatherComplete" LastRequestedTime="Mon, 24 Aug 2015 10:20:03 +0000"></SiteStatus>
© 2015 IBM Corporation
23
Server/Relay Diagnostics
Failed:
Ready:
In Process:
© 2015 IBM Corporation
24
Checking server/relay(s) for reporting backlogs
§
§
Relay Status Information section on the Relay Diagnostics page on the server/relay(s).
–
File Size Limit or File Count Limit is 100% or very close to 100% there is most likely a backlog.
Actual files moving in and out of bufferdir can be observed:
–
Server: \BES Server\FillDBData\BufferDir\
–
Relay: \BES Relay\FillDBData\BufferDir
© 2015 IBM Corporation
25
Checking the Relay(s)
© 2015 IBM Corporation
Checking a relay for mailboxed actions
§ Verifying the same mailbox file exists on the client's parent relay(s) for a specific computer
§
§
/wwwrootbes/mailbox/Mailboxes.db (is a SQL Lite database)
select * from FILES where FileID = (select FileID from COMPUTER_FILES where ComputerID =
'<COMPUTER ID>')
© 2015 IBM Corporation
27
Checking a relay for actionsite actions
§
C:\Program Files (x86)\BigFix Enterprise\BES Relay\Mirror Server\inbox\bfemapfile.xml
–
<UrlToIdPair ID="2">http://spt1-win2k8r2:52311/cgi-bin/bfgather.exe/actionsite</UrlToIdPair>
§
C:\Program Files (x86)\BigFix Enterprise\BES Relay\Mirror Server\inbox\GatherState.xml
–
<SiteStatus SiteID="2" CommittedLocation="C:\Program Files (x86)\BigFix Enterprise\BES
Relay\wwwrootbes\bfmirror\bfsites\manymirror_2_407_376895136" CommittedVersion="407"
CommittedManyVersion="2700" Status="GatherComplete" LastRequestedTime="Fri, 21 Aug 2015 10:19:47
+0000"></SiteStatus>
© 2015 IBM Corporation
28
Checking a relay for opsite actions
§
C:\Program Files (x86)\BigFix Enterprise\BES Relay\Mirror Server\inbox\bfemapfile.xml
–
<UrlToIdPair ID="20">http://spt1-win2k8r2:52311/cgi-bin/bfgather.exe/opsite104</UrlToIdPair>
§
C:\Program Files (x86)\BigFix Enterprise\BES Relay\Mirror Server\inbox\GatherState.xml
–
<SiteStatus SiteID="20" CommittedLocation="C:\Program Files (x86)\BigFix Enterprise\BES
Relay\wwwrootbes\bfmirror\bfsites\manymirror_20_5_3600388732" CommittedVersion="5"
CommittedManyVersion="2698" Status="GatherComplete" LastRequestedTime="Mon, 24 Aug 2015
10:19:46 +0000"></SiteStatus>
© 2015 IBM Corporation
29
Relay Diagnostics
Failed:
Ready:
In Process:
© 2015 IBM Corporation
30
Notifications
© 2015 IBM Corporation
Verbose logging available for server/relay
§ Verbose logging on the server/relay(s):
–
Client setting: _BESRelay_Log_Verbose
–
Restart BES Root Server or BES Relay service
–
Server messages output to \BES Server\BESRelay.log
–
Relay messages output to \BES Relay\logfile.txt (on Windows) and \BES
Relay\BESRelay.log on (Linux)
§ How to set client settings:
–
http://www.ibm.com/support/docview.wss?uid=swg21506050
© 2015 IBM Corporation
32
Notifications from server/relay to relay
●
●
The server (9.39.151.222) notifying a relay at 9.39.149.187 after a new actionsite action had
been taken: The server's BESRelay.log log (verbose mode);
The relay (9.39.149.187) receiving the server's (9.39.151.222) notification after a new
actionsite action had been taken: The relay's logfile.txt log (verbose mode);
© 2015 IBM Corporation
33
Notifications from relay to client
§ The relay (9.39.149.187) sending a UDP message to the client endpoint (9.39.151.221)
notification after a new actionsite action had been taken: The relay's logfile.txt log (verbose
mode);
§ The client endpoint (9.39.151.221) receiving a UDP message from the relay (9.39.149.187)
notification after a new actionsite action had been taken: The client's log (default mode);
§ If UDP is blocked between relay and client; you can enable “command polling” on the client
which instructs the client to check every X seconds for new content:
§ http://www-01.ibm.com/support/docview.wss?uid=swg21505846 :
–
_BESClient_Comm_CommandPollEnable (set to 1 to enable)
–
_BESClient_Comm_CommandPollIntervalSeconds (set to >= 1800)
© 2015 IBM Corporation
34
Questions for the panel?
Now is your opportunity to ask questions of our panelists.
To ask a question now:
Press
*1 to ask a question over the phone
or
Type your question into the IBM Connections Cloud Meeting chat
To ask a question after this presentation:
You are encouraged to participate in the BigFix forum topic for this event at
https://forum.bigfix.com/t/open-mic-steps-to-take-in-troubleshooting-endpoint-manageractions-with-a-not-reported-status-august-27th-2015/14104
© 2015 IBM Corporation
35
Where do you get more information?
Questions on this or other topics can be directed to the product forum:
https://forum.bigfix.com/
IBM Knowledge Center:
https://www.ibm.com/support/knowledgecenter/
(then expand “Cloud and Smarter Infrastructure” and scroll until you get to
“IBM BigFix”
Useful links: How to Contact IBM Software Support for IBM Security
IBM Support Portal | Sign up for “My Notifications”
Follow us:
© 2015 IBM Corporation
36
Statement of Good Security Practices: IT system security involves protecting systems and information through prevention, detection and response to improper access from within and outside
your enterprise. Improper access can result in information being altered, destroyed, misappropriated or misused or can result in damage to or misuse of your systems, including for use in attacks
on others. No IT system or product should be considered completely secure and no single product, service or security measure can be completely effective in preventing improper use or access.
IBM systems, products and services are designed to be part of a lawful, comprehensive security approach, which will necessarily involve additional operational procedures, and may require other
systems, products or services to be most effective. IBM DOES NOT WARRANT THAT ANY SYSTEMS, PRODUCTS OR SERVICES ARE IMMUNE FROM, OR WILL MAKE YOUR ENTERPRISE
IMMUNE FROM, THE MALICIOUS OR ILLEGAL CONDUCT OF ANY PARTY.
THANK YOU
www.ibm.com/security
© Copyright IBM Corporation 2015. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any
kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor
shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use
of IBM software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and / or
capabilities referenced in these materials may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product
or feature availability in any way. IBM, the IBM logo, and other IBM products and services are trademarks of the International Business Machines Corporation, in the United States, other countries
or both. Other company, product, or service names may be trademarks or service marks of others.