Automatically Run Test Case Coverage Analysis
If you want confidence that your regression tests are adequate, your test cases must cover all the scenarios that you believe are being tested.
LIXI has started curating a LIXI2 sample message repository that we use, amongst other things, for testing our schema changes.
A part of our schema change test suite is a coverage test to ensure that every new element and attribute in each release is covered by at least one sample messages. We created this post to share our experience as our methodology might be useful if you are trying to develop your own sample coverage analysis.
Step 1. Prerequisites
Step 2. Create Folder with Schemas and Samples
We will use our LIXILab repositories as the source of schemas and samples. Start this process by creating a folder called 'lixi-coverage-analysis' - I've created this under the 'C:\temp\' directory.
Then in a command or bash window and use the git clone command to clone both the LIXI2 schemas and the LIXI2 samples projects locally from LIXILab. If this is your first time running this command, you will be asked for your credentials for LIXILab.
Step 3. Copy the Schemas and Samples we're interested in
For this example, we'll focus on a single standard ('CAL' - the Credit Application Standard for Australia), so we'll copy across the CAL samples and schemas from the locally cloned repository files. We can use the following Python code saved to 'C:\temp\lixi-coverage-analysis\copyfiles.py'.
Step 4. Perform the Coverage Analysis
Depending on your Python installation, you can now run the following code. Create an empty file called 'analyse-coverage.py' and copy and paste the code from the following code block and save it. You can use an IDE such as Visual Studio Code or execute the python script from a command or bash session. For example, you can run it at a Windows CMD prompt in the 'lixi-coverage-analysis' directory using 'python analyse-coverage.py'.
The script utilizes the LIXI python library to derive all the paths used in the sample messages and all possible schema paths based on the relevant version of the schema and calculates the percentage coverage.
Step 5. Results
Once you run the command, the script prints out the percentage of paths that have been covered by the samples.
Step 6. What Next?
Firstly, you'll replace the LIXI Samples with your own that are targetted at your own use cases and needs. Secondly, you'll most likely be using customised schemas, that have many of the LIXI paths restricted (using Customisation By Restriction) so you'll use your own custom schemas. Like LIXI, your highest priority might be to ensure all NEW paths are covered in the tests, so the script above can be modified to report only the coverage of items that were released between two versions of the standards. Beyond that, it's up to your imagination how you might extend this coverage testing.
Shane Rigby, LIXI Limited CEO
First Published: January 22, 2021