Test Plan For “Dynamic LNET Configuration” Revision History Date Revision November 19, V0.1 2013 Author Amir Shehata High Performance Data Division Table of Contents Introduction (Content should be detailed so that it can be directly used in the Lustre Operations Manual when appropriate) .......................................................................3 I. Feature Setup........................................................................................................3 II. Test cases.............................................................................................................3 Unit Testing .................................................................................................................................................................4 Test cases template .................................................................................. Error! Bookmark not defined. Regression Testing ................................................................................... Error! Bookmark not defined. Test cases template .................................................................................. Error! Bookmark not defined. Scalability Testing .................................................................................... Error! Bookmark not defined. Compatibility/Interoperability Testing .......................................... Error! Bookmark not defined. Performance Testing ............................................................................................................................................ 19 Test cases template (including the current typical test runs) Error! Bookmark not defined. IOR - single-shared file/File per process .......................................... Error! Bookmark not defined. Mdtest – single shared directory/dir per process ......................... Error! Bookmark not defined. SWL..................................................................................................................... Error! Bookmark not defined. Failover Testing ......................................................................................... Error! Bookmark not defined. Test cases template .................................................................................. Error! Bookmark not defined. User Interface Components: (content should be detailed so that it can be directly used in the Lustre Operations Manual if appropriate) ..................................... Error! Bookmark not defined. Procfs/Sysfs entries added or changed: .......................................... Error! Bookmark not defined. Other system settings added or changed: ...................................... Error! Bookmark not defined. LFS commands added or changed: .................................................... Error! Bookmark not defined. LCTL commands added or changed: ................................................. Error! Bookmark not defined. New Scripts added: .................................................................................. Error! Bookmark not defined. Man Pages entries added or changed:........................................................................................................... 28 HPDD High Performance Data Division 2 The intent of this document is to provide a guideline for the engineers developing features to use when creating the test plan for the feature. Sections that do not apply to a particular feature may be deleted or marked as not applicable. Introduction For a detailed description of the Dynamic LNET Configuration refer to the following LUDOC gerrit patch: http://review.whamcloud.com/#/c/8263/ I. Feature Setup For a detailed description of how to setup and use the Dynamic LNET Configuration project refer to the following LU-DOC gerrit patch: http://review.whamcloud.com/#/c/8263/ The system will look as follows: II. Test cases This should include the areas of testing that will need to be completed for this feature. Below are examples: HPDD High Performance Data Division 3 The test cases are divided into 7 components: 1 – Unit Testing 5 – Performance testing System Testing Unit Testing A separate python based unit test tool (Lustre Unit Test Framework – LUTF) was built to unit test the DLC changes. The LUTF is required to test the C API introduced by DLC. It makes direct calls to the C API, thereby enabling the major corner cases to be tested. Test Case Name UTC01 Test missing network parameter behavior Purpose Actors Description Call the lustre_lnet_config_route() function with None as the network parmaeter Environment Settings Trigger Preconditions Post-conditions Special Requirements Assumptions Expected Results Notes and Issues Test Case Name API returns LUSTRE_CFG_RC_MISSING_PARAM UTC02 Test missing gateway parameter behavior Purpose Actors Description Call the lustre_lnet_config_route() function with None as the gateway parmaeter Environment Settings Trigger Preconditions Post-conditions Special Requirements Assumptions HPDD High Performance Data Division 4 Expected Results Notes and Issues API returns LUSTRE_CFG_RC_MISSING_PARAM Test Case Name UTC03 Test missing both gateway & network parameter behavior Purpose Actors Description Call the lustre_lnet_config_route() function with None as the gateway & network parmaeter Environment Settings Trigger Preconditions Post-conditions Special Requirements Assumptions Expected Results Notes and Issues Test Case Name API returns LUSTRE_CFG_RC_MISSING_PARAM UTC04 Test standard route add with no hop or prio Purpose Actors Description Call the lustre_lnet_config_route() function. Show the route afterwards to ensure that the route has been added Environment Settings Trigger Preconditions Post-conditions Special Requirements Assumptions Expected Results API returns LUSTRE_CFG_RC_NO_ERR Shows the route that was just added Notes and Issues Test Case Name Purpose Actors UTC05 Test standard route add with hop but no prio HPDD High Performance Data Division 5 Description Call the lustre_lnet_config_route() function. Show the route filtering on hop afterwards to ensure that the route has been added Environment Settings Trigger Preconditions Post-conditions Special Requirements Assumptions Expected Results API returns LUSTRE_CFG_RC_NO_ERR Shows the route that was just added Notes and Issues Test Case Name UTC06 Test route show with different filters Purpose Actors Description Call the lustre_lnet_show_route() function with the following filters: Gateway Network Hop Network and gateway verify that there exists at least one 'route' entery Environment Settings Trigger Preconditions Post-conditions Special Requirements Assumptions Expected Results API returns LUSTRE_CFG_RC_NO_ERR Shows all routes in the system Notes and Issues Test Case Name Purpose Actors Description UTC07 Test route show with no filter Call the lustre_lnet_show_route() function. HPDD High Performance Data Division 6 verify that there exists at least one 'route' entery Environment Settings Trigger Preconditions Post-conditions Special Requirements Assumptions Expected Results API returns LUSTRE_CFG_RC_NO_ERR Shows all routes in the system Notes and Issues Test Case Name UTC08 Test del route with no net specified Purpose Actors Description Call the lustre_lnet_config_route() function to insert route. delete that route but specify no net expect an error clean the routes Environment Settings Trigger Preconditions Post-conditions Special Requirements Assumptions Expected Results Notes and Issues Test Case Name API returns LUSTRE_CFG_RC_MISSING_PARAM UTC09 Test del route with no gateway specified Purpose Actors Description Call the lustre_lnet_config_route() function to insert route. delete that route but specify no gw expect an error clean the routes Environment Settings Trigger HPDD High Performance Data Division 7 Preconditions Post-conditions Special Requirements Assumptions Expected Results Notes and Issues Test Case Name API returns LUSTRE_CFG_RC_MISSING_PARAM UTC10 Test del route that doesn't exist Purpose Actors Description Call the lustre_lnet_config_route() function to insert route. delete the wrong route expect no errors clean the routes Environment Settings Trigger Preconditions Post-conditions Special Requirements Assumptions Expected Results Notes and Issues Test Case Name API returns LUSTRE_CFG_RC_NO_ERR UTC11 Test Config net with missing 'net' param Purpose Actors Description Call the lustre_lnet_config_net() function to add a net. missing net param expect MISSING Param return code Environment Settings Trigger Preconditions Post-conditions Special Requirements Assumptions Expected Results API returns LUSTRE_CFG_RC_MISSING_PARAM HPDD High Performance Data Division 8 Notes and Issues Test Case Name UTC12 Test Config net with missing 'intf' param Purpose Actors Description Call the lustre_lnet_config_net() function to add a net. missing intf param expect MISSING Param return code Environment Settings Trigger Preconditions Post-conditions Special Requirements Assumptions Expected Results Notes and Issues Test Case Name UTC13 Test Config net with missing 'net' and 'intf' param Purpose Actors Description Call the lustre_lnet_config_net() function to add a net. missing net and intf param expect MISSING Param return code Environment Settings Trigger Preconditions Post-conditions Special Requirements Assumptions Expected Results Notes and Issues Test Case Name Purpose Actors API returns LUSTRE_CFG_RC_MISSING_PARAM API returns LUSTRE_CFG_RC_MISSING_PARAM UTC14 Test Config net with bad 'net' param HPDD High Performance Data Division 9 Description Call the lustre_lnet_config_net() function to add a net. bad net Environment Settings Trigger Preconditions Post-conditions Special Requirements Assumptions Expected Results API Errno 22 – Invalid argument YAML error points to the same Notes and Issues Test Case Name UTC15 Test Config net with bad ‘intf’ param Purpose Actors Description Call the lustre_lnet_config_net() function to add a net. bad interface Environment Settings Trigger Preconditions Post-conditions Special Requirements Assumptions Expected Results API Errno 22 – Invalid argument YAML error points to the same Notes and Issues Test Case Name Purpose Actors Description UTC16 Test Config net properly, verify via show, then delete the new net configured tests: Config Net, Show Net and Delete Net Call the lustre_lnet_config_net() function to add a net. proper parameters expect net to be configured properly call the lustre_lnet_show_net() to verify the net was added proprely call the lustre_lnet_del_net() to remove all networks added. HPDD High Performance Data Division 10 Environment Settings Trigger Preconditions Post-conditions Special Requirements Assumptions Expected Results APIs should return LUSTRE_CFG_RC_NO_ERR YAML show should dump networks in the system Notes and Issues Test Case Name UTC17 Test Config net properly, verify via show, then delete the new net configured tests: Config Net, Show Net and Delete Net Purpose Actors Description Call the lustre_lnet_config_net() function to add a net. define all optional parameters: peer_to, peer_credits, peer_buffer_credits, credits expect net to be configured with parameters specified call the lustre_lnet_show_net() to verify the net was added proprely call the lustre_lnet_del_net() to remove all networks added.success Environment Settings Trigger Preconditions Post-conditions Special Requirements Assumptions Expected Results APIs should return LUSTRE_CFG_RC_NO_ERR YAML show should dump networks in the system Notes and Issues Test Case Name Purpose Actors Description UTC18 Test Config net properly with smp, verify via show, then delete the new net configured tests: Config Net, Show Net and Delete Net Call the lustre_lnet_config_net() function to add a net. HPDD High Performance Data Division 11 define SMP optional parameter expect net to be configured with parameters specified call the lustre_lnet_show_net() to verify the net was added proprely call the lustre_lnet_del_net() to remove all networks added.success Environment Settings Trigger Preconditions Post-conditions Special Requirements Assumptions Expected Results APIs should return LUSTRE_CFG_RC_NO_ERR YAML show should dump networks in the system Notes and Issues Test Case Name UTC19 Test configuring the same netwrk twice. Verify via show that only one net is there, then delete the new net configured Purpose Actors Description Call the lustre_lnet_config_net() function to add a net. proper parameters expect net to be configured with parameters specified second time to be configured, no change should occur call the lustre_lnet_show_net() to verify the net was added proprely call the lustre_lnet_del_net() to remove all networks added.success Environment Settings Trigger Preconditions Post-conditions Special Requirements Assumptions Expected Results APIs should return LUSTRE_CFG_RC_NO_ERR YAML show should dump networks in the system Notes and Issues Test Case Name Purpose UTC20 Test Config buffers with bad parameters HPDD High Performance Data Division 12 Actors Description call lustre_lnet_config_buf with bad parameters to verify error checking Environment Settings Trigger Preconditions Post-conditions Special Requirements Assumptions Expected Results APIs should return LUSTRE_CFG_RC_OUT_OF_RANGE_PARAM YAML error should dump the same error Notes and Issues Test Case Name UTC21.1 Test Config and reconfiguring buffers Purpose Actors Description Enable routing Configure buffers to default Verify values configured Configure buffers to something other than default Verify values configured disable routing Environment Settings Trigger Preconditions Post-conditions Special Requirements Assumptions Expected Results This test assumes an idle system APIs should return LUSTRE_CFG_RC_NO_ERR YAML error should dump the same error Notes and Issues Test Case Name Purpose Actors UTC22 Test Config and reconfiguring buffers HPDD High Performance Data Division 13 Description Enable routing Configure buffers to something other than default Verify values configured disable routing Verify that everything is disabled enable routing Verify values are configured as previously set Verifying ability to reset to default values Environment Settings Trigger Preconditions Post-conditions Special Requirements Assumptions Expected Results This test assumes an idle system APIs should return LUSTRE_CFG_RC_NO_ERR YAML error should dump the same error Notes and Issues Test Case Name UTC23 Test route yaml based API Purpose Actors Description Create a YAML file of multiple routes Call the lustre_yaml_config() - to configure Call the lustre_yaml_show() - to show and verify what has been configured Call the lustre_yaml_del() - to delete what has been configured. Call the lustre_yaml_show() - to show and verify routes are no longer configured Environment Settings Trigger Preconditions Post-conditions Special Requirements Assumptions Expected Results This test assumes an idle system APIs should return LUSTRE_CFG_RC_NO_ERR YAML error should dump the same error Notes and Issues HPDD High Performance Data Division 14 Test Case Name UTC24 Test net yaml based API Purpose Actors Description Create a YAML file of multiple networks Call the lustre_yaml_config() - to configure Call the lustre_yaml_show() - to show and verify what has been configured Call the lustre_yaml_del() - to delete what has been configured. Call the lustre_yaml_show() - to show and verify routes are no longer configured Environment Settings Trigger Preconditions Post-conditions Special Requirements Assumptions Expected Results This test assumes an idle system APIs should return LUSTRE_CFG_RC_NO_ERR YAML error should dump the same error Notes and Issues Test Case Name UTC25 Test buffer yaml based API Purpose Actors Description Create a YAML file of multiple networks Call the lustre_yaml_config() - to configure Call the lustre_yaml_show() - to show and verify what has been configured Call the lustre_yaml_del() - to delete what has been configured. Call the lustre_yaml_show() - to show and verify routes are no longer configured Environment Settings Trigger Preconditions Post-conditions Special Requirements Assumptions Expected Results This test assumes an idle system APIs should return LUSTRE_CFG_RC_NO_ERR HPDD High Performance Data Division 15 YAML error should dump the same error Notes and Issues Test Case Name UTC26 Test mixed configuration yaml based file, including net, route and buffer Purpose Actors Description Create a YAML file of multiple networks Call the lustre_yaml_config() - to configure Call the lustre_yaml_show() - to show and verify what has been configured Call the lustre_yaml_del() - to delete what has been configured. Call the lustre_yaml_show() - to show and verify routes are no longer configured Environment Settings Trigger Preconditions Post-conditions Special Requirements Assumptions Expected Results This test assumes an idle system APIs should return LUSTRE_CFG_RC_NO_ERR YAML error should dump the same error Notes and Issues Test Case Name UTC27 Test the LNET configuration utility basics Purpose Actors Description Configure/show/del route Configure/show/del net Enable/disable routing Adjust router buffer pools Environment Settings Trigger Preconditions Post-conditions Special Requirements Assumptions This test assumes an idle system HPDD High Performance Data Division 16 Expected Results APIs should return LUSTRE_CFG_RC_NO_ERR YAML error should dump the same error Notes and Issues Test Case Name UTC28 Test traffic behavior when dynamically enabling/disabling routing Purpose Actors Description Gets a lustre client to create a number of files thus causing traffic between the client and OST/OSS/MGS/MDT On Router 1 Get the LNET stats and ensure that traffic is not being routed Enable routing wait for 70 seconds Get the LNET stats to ensure that the traffic is being routed Disable routing Get the LNET stats to ensure that traffic is not being routed On Router 2 Get the LNET stats to ensure that the traffic is being routed Environment Settings Trigger Preconditions Post-conditions Special Requirements Assumptions Expected Results This test assumes an idle system APIs should return LUSTRE_CFG_RC_NO_ERR YAML error should dump the same error Notes and Issues Test Case Name Purpose Actors Description UTC29 Test traffic behavior when adding/removing routes On Client Spawn a thread to write into the /mnt/client directory From the other thread check the stats to ensure traffic is being sent Delete a route HPDD High Performance Data Division 17 check that the stats show that traffic is still being sent Delete the second route - now there shouldn't be any routes Check that the stats show that traffic is not being sent Add both the routes back Check that the stats show that traffic is still being sent Environment Settings Trigger Preconditions Post-conditions Special Requirements Assumptions Expected Results This test assumes an idle system APIs should return LUSTRE_CFG_RC_NO_ERR YAML error should dump the same error Notes and Issues Test Case Name UTC30 Test traffic behavior when adding/removing networks Purpose Actors Description Grab the existing routes (assumes 2) because once we delete the net, the routes will be gone too Spawn a thread to write into the /mnt/client directory From the other thread check the stats to ensure traffic is being sent Add a Net check that the stats show that traffic is still being sent Del Net added check that the stats show that traffic is still being sent Del existing Net Traffic stops Environment Settings Trigger Preconditions Post-conditions Special Requirements Assumptions Expected Results This test assumes an idle system APIs should return LUSTRE_CFG_RC_NO_ERR YAML error should dump the same error Notes and Issues HPDD High Performance Data Division 18 Test Case Name UTC31 Test traffic behavior when adding/removing/readding routes Purpose Actors Description Spawn a thread to write into the /mnt/client directory From the other thread check the stats to ensure traffic is being sent Delete a route check that the stats show that traffic is still being sent Delete the second route - now there shouldn't be any routes Check that the stats show that traffic is not being sent Add both the routes back Check that the stats show that traffic is still being sent Environment Settings Trigger Preconditions Post-conditions Special Requirements Assumptions Expected Results This test assumes an idle system APIs should return LUSTRE_CFG_RC_NO_ERR YAML error should dump the same error Notes and Issues Scalability Testing Test Case Name Adding/showing/deleting 10K routes Test the addition of 10 routes Purpose Actors Description Create a YAML configuration file with 10K routes Use the Config yaml API to configure the 10k routes Use the Show YAML API to show the 10k routes Use the del YAML API to remove the 10K routes Environment Settings Trigger Preconditions Postconditions Special Requirements Assumptions Expected Results No routes configured No routes configured Routes configured and showed then deleted properly HPDD High Performance Data Division 19 All APIs should return LUSTRE_CFG_RC_NO_ERR Notes and Issues System Testing 3.3.2 Network Testing Test Case Name dynLNet.system.net_new To verify that new network interfaces can be added Purpose without interrupting the operation of current interfaces. Actors System test setup Add a second net interface to client 2 accessing a file Description system over the first net interface. Environment N/A Settings Trigger 1. System setup as defined in section 2 is running. Preconditions 2. File operations are running in the background. Postconditions Special Requirements Assumptions Expected Results Check for the addition of the net interface New net interface is added successfully. No errors reported in /var/log/messages. File operations are not interrupted. Notes and Issues Test Case Name dynLNet.system.net_existing To verify that trying to add a network interface which Purpose already exists does not fail or affect traffic over that interface. Actors System test setup Description add the existing interface on a client running file HPDD High Performance Data Division 20 operations over it. Environment Settings Trigger Preconditions N/A 1. System setup as defined in section 2 is running. 2. File operations are running in the background. Postconditions Special Requirements Assumptions Expected Results No errors from the “lnet net <net> add” command. No errors reported in /var/log/messages. File operations are not interrupted. Notes and Issues Test Case Name dynLNet.system.net_lnd_params To verify that any LND parameters set when adding a Purpose network interface take effect. Actors System test setup Add a second net interface to a client. Check that the Description LND parameters were properly set. Environment N/A Settings Trigger 1. System setup as defined in section 2 is running. Preconditions 2. File operations are running in the background. Postconditions Special Requirements Assumptions Expected Results Check what LND parameters have been set Parameters set for the LND should match the parameters given. HPDD High Performance Data Division 21 Notes and Issues Test Case Name dynLNet.system.net_smp_params To verify that any SMP affinity parameters set when Purpose adding a network interface take effect. Actors System test setup Add a second net interface to a client. Check that the Description SMP parameters were properly set. Environment N/A Settings Trigger 1. System setup as defined in section 2 is running. Preconditions 2. File operations are running in the background. Postconditions Special Requirements Assumptions Expected Results Notes and Issues Test Case Name Purpose Actors Description Environment Settings Trigger Preconditions Check what LND parameters have been set. Parameters set for SMP Affinity should match the parameters given. dynLNet.system.net_down_exists To verify that an existing network interface can be taken down without affecting other operating network interfaces. System test setup bring down the second interface of the client. N/A 1. System setup as defined in section 2 is running. 2. Second net interface on client is up. Postconditions HPDD High Performance Data Division 22 Special Requirements Assumptions Check for deleted net interface. Expected Results Notes and Issues Test Case Name Purpose Actors Description Environment Settings Trigger Preconditions Net interface should be successfully brought down. No errors reported in /var/log/messages. File operations are not interrupted. dynLNet.system.net_down_not_present To verify that and attempt to take down an interface which does not exist will fail. System test setup bring down a non-existent net interface. N/A 1. System setup as defined in section 2 is running. 2. File operations are running in the background. Postconditions Special Requirements Assumptions Expected Results command should fail. No errors reported in /var/log/messages. File operations are not interrupted. Notes and Issues 3.3.2 Route Testing Test Case Name dynLNet.system.route_new To verify that new routes can be added without Purpose interrupting the operation of current routes. HPDD High Performance Data Division 23 Actors Description Environment Settings Trigger Preconditions Postconditions Special Requirements Assumptions Expected Results System test setup On the client, add a route through router 2 N/A 1. System setup as defined in section 2 is running. 2. File operations are running in the background. Check what routes have been added on success New route should be added. No file operations should be interrupted. Traffic should be split between the two routers. Notes and Issues Test Case Name dynLNet.system.route_existing To verify that trying to add a route which already Purpose exists does not fail or affect any traffic. Actors System test setup On the client, try to add a route through router 1, which already Description exists Environment N/A Settings Trigger 1. System setup as defined in section 2 is running. Preconditions 2. File operations are running in the background. Postconditions Special Requirements Assumptions Expected Results Attempt to add existing route should not fail, but will not change anything. HPDD High Performance Data Division 24 No file operations should be interrupted. Notes and Issues Test Case Name Purpose Actors Description Environment Settings Trigger Preconditions Postconditions Special Requirements Assumptions dynLNet.system.route_del_exists To verify that an existing route can be deleted with affecting other routes or traffic. System test setup Delete the second route N/A 1. Completion of test dynLNet.system.route_new. 2. File operations are running traffic over both routers. Check what routes have been removed on success Expected Results Notes and Issues Test Case Name Purpose Actors Description Environment Settings Second route is successfully removed. File operations should not be interrupted (some packet loss will be experienced). All traffic should be redirected through the one remaining route. dynLNet.system.route_del_not_present To verify that trying to delete a route which does not exist will fail. System test setup Try to delete the second route N/A HPDD High Performance Data Division 25 Trigger Preconditions 1. System setup as defined in section 2 is running. 2. File operations are running in the background. Postconditions Special Requirements Assumptions Expected Results Attempt to delete non-active route should fail. No file operations should be interrupted. Notes and Issues 3.3.3 Route Buffer Testing Test Case dynLNet.system.route_buffer_increase Name To verify that increasing each of the three routing Purpose buffers will take effect without affecting traffic flows. Actors System test setup Description Increase the size of all three router buffers on router. Environment N/A Settings Trigger 1. System setup as defined in section 2 is running. Preconditions 2. File operations are running in the background. Postconditions Special Requirements Assumptions Check what route buffers are set to on router. Expected Results Route buffers on router 1 should match the new values. No file operations should be interrupted. HPDD High Performance Data Division 26 Notes and Issues Test Case Name Purpose Actors Description Environment Settings Trigger Preconditions dynLNet.system.route_buffer_decrease To verify that decreasing each of the three routing buffers will take effect without affecting traffic flows System test setup Decrease the size of all three router buffers on router 1 back to the defaults N/A 1. Completion of test dynLNet.system.route_buffer_increase. 2. File operations are running in the background. Postconditions Special Check what route buffers are set to on router 1. Requirements Assumptions Route buffers on router 1 should become the defaults. Expected Current outstanding buffers will, over time, Results come down to meet the new defaults. No file operations should be interrupted. Notes and Issues 3.3.4 Combination Testing Test Case dynLNet.system.combination_config Name To verify that all four of the ways to configure LNet can Purpose co-exist (module parameters, YAML config file, lnet Configuration Utility command). Actors System test setup HPDD High Performance Data Division 27 Bring up all systems using module parameters, then use YAML config file, lnet utility: Description Environment Settings Trigger Preconditions Postconditions Special Requirements Assumptions Expected Results Notes and Issues Set the module parameters to specific values and then show the configuration to ensure that everything is configured as desired Add net interfaces, routes, and route buffer changes to the YAML config files then run it through the appropriate API Remove net interfaces and routes with the lnet utility. N/A 1. File operations are running in the background. Use the various lnet utility queries to see what changes have taken effect. Add changes made should take effect without trampling unchanged settings. Man Pages entries added or changed: - Test Plan Results: (To be populated by the Test Engineer) - Necessary data that demonstrates the success of the feature HPDD High Performance Data Division 28 - Important data from performance tests (speed, latency, and bandwidth) Include the data defined above in the test plan Additional reports or supporting documentation as defined above in the test plan Detailed configuration of the system used for testing Other names and brands may be claimed as the property of others. HPDD High Performance Data Division 29
© Copyright 2026 Paperzz