Automatically Run Test Case Coverage Analysis

Automatically Run
Test Case Coverage Analysis

Posted on: January 22, 2021

If you want confidence that your regression tests are adequate, your test cases must cover all the scenarios that you believe are being tested.

LIXI has started curating a LIXI2 sample message repository that we use, amongst other things, for testing our schema changes.

A part of our schema change test suite is a coverage test to ensure that every new element and attribute in each release is covered by at least one sample messages. We created this post to share our experience as our methodology might be useful if you are trying to develop your own sample coverage analysis.

Step 1. Prerequisites

Before starting, you need to have Python 3, have already installed the LIXI Python Package and have access to LIXILab.

Step 2. Create Folder with Schemas and Samples

We will use our LIXILab repositories as the source of schemas and samples. Start this process by creating a folder called 'lixi-coverage-analysis' - I've created this under the 'C:\temp\' directory.

Then in a command or bash window and use the git clone command to clone both the LIXI2 schemas and the LIXI2 samples projects locally from LIXILab. If this is your first time running this command, you will be asked for your credentials for LIXILab.

C:\temp\lixi-coverage-analysis>git clone https://standards.lixi.org.au/lixi2/schemas.git git-schemas
Cloning into 'schemas'...
remote: Enumerating objects: 319, done.
remote: Counting objects: 100% (319/319), done.
remote: Compressing objects: 100% (142/142), done.
Receiving objects: 100% (2850/2850), 10.47 MiB | 1.13 MiB/sving objects:  99% (2849/2850), 10.47 MiB | 1.13 MiB/s
Receiving objects: 100% (2850/2850), 10.73 MiB | 1.19 MiB/s, done.
Resolving deltas: 100% (2667/2667), done.
Updating files: 100% (802/802), done.


C:\temp\lixi-coverage-analysis>git clone https://standards.lixi.org.au/lixi2/samples.git git-samples
Cloning into 'samples'...
remote: Enumerating objects: 2748, done.
remote: Counting objects: 100% (2748/2748), done.
remote: Compressing objects: 100% (689/689), done.
remote: Total 2748 (delta 2089), reused 2705 (delta 2052)
Receiving objects: 100% (2748/2748), 1.15 MiB | 1.30 MiB/s, done.
Resolving deltas: 100% (2089/2089), done.


Step 3. Copy the Schemas and Samples we're interested in

For this example, we'll focus on a single standard ('CAL' - the Credit Application Standard for Australia), so we'll copy across the CAL samples and schemas from the locally cloned repository files. We can use the following Python code saved to 'C:\temp\lixi-coverage-analysis\copyfiles.py'.

import glob, os
from shutil import copyfile

this_path = os.path.realpath(os.path.join(os.getcwd(), os.path.dirname(__file__)))
samples_path = os.path.join(this_path, 'samples')
schemas_path = os.path.join(this_path, 'schemas')

if not os.path.exists(samples_path):    os.makedirs(samples_path)

for file in glob.glob(os.path.join(this_path, 'git-samples', 'Samples', 'CAL', '**/*.xml')):
    copyfile(file, os.path.join(samples_path, os.path.basename(file)))

if not os.path.exists(schemas_path):    os.makedirs(schemas_path)

for file in glob.glob(os.path.join(this_path, 'git-schemas', '**/*LIXI-CAL*.xsd')):
    copyfile(file, os.path.join(schemas_path, os.path.basename(file)))

Step 4. Perform the Coverage Analysis

Depending on your Python installation, you can now run the following code. Create an empty file called 'analyse-coverage.py' and copy and paste the code from the following code block and save it. You can use an IDE such as Visual Studio Code or execute the python script from a command or bash session. For example, you can run it at a Windows CMD prompt in the 'lixi-coverage-analysis' directory using 'python analyse-coverage.py'.

The script utilizes the LIXI python library to derive all the paths used in the sample messages and all possible schema paths based on the relevant version of the schema and calculates the percentage coverage.

import lixi, os, glob, requests, subprocess, json, re
from lxml import etree

# Retrieve this scripts path
this_path = os.path.realpath(os.path.join(os.getcwd(), os.path.dirname(__file__)))

# Derive the path to the cloned schemas
schema_path = os.path.join(this_path, 'schemas')

# Derive the path to the cloned samples
glob_search_string = os.path.join('samples') +'/**/*.xml'

# Set the schema folder path in the lixi library

# Derive the latest and last version of the CAL schema
response = json.loads(requests.get("https://lixi-public.s3-ap-southeast-2.amazonaws.com/versions.json").text)['cal']
CAL_latest_version = response.replace('_','.')

# Derive all schema paths used in messages in message folder.
CAL_message_paths = []

files = glob.glob(glob_search_string, recursive=True)
for message_file in files:
    message = lixi.read_message(message_path = message_file)
    message_paths = message.get_message_paths()
    for path in message_paths:
        if 'List.' not in path and path not in CAL_message_paths:

# Derive all schema paths used in CAL schema.
CAL_schema_paths = lixi.get_schema_paths(lixi_transaction_type='CAL', lixi_version=CAL_latest_version)
temp_paths = []

for path in CAL_schema_paths:
    if 'List.' not in path:
CAL_schema_paths = temp_paths
# Derive subset of paths that are in schema but not yet covered in messages

perc = (len(CAL_message_paths) / len(CAL_schema_paths)) * 100
sub = len(CAL_schema_paths) - len(CAL_message_paths)

# Print out results
print(str(round(perc, 2))+"% of latest ("+CAL_latest_version+") CAL has been covered. "+str(sub)+" paths left.".format(round(perc, 2)))

Step 5. Results

Once you run the command, the script prints out the percentage of paths that have been covered by the samples.

Microsoft Windows [Version 10.0.18363.1316]
(c) 2019 Microsoft Corporation. All rights reserved.

C:\temp\lixi-coverage-analysis>python analyse-coverage.py
7.09% of latest (2.6.34) CAL has been covered. 3459 paths left.

Step 6. What Next?

Firstly, you'll replace the LIXI Samples with your own that are targetted at your own use cases and needs. Secondly, you'll most likely be using customised schemas, that have many of the LIXI paths restricted (using Customisation By Restriction) so you'll use your own custom schemas. Like LIXI, your highest priority might be to ensure all NEW paths are covered in the tests, so the script above can be modified to report only the coverage of items that were released between two versions of the standards. Beyond that, it's up to your imagination how you might extend this coverage testing.

Related Blogs

Written by:
Shane Rigby, LIXI Limited CEO
First Published: January 22, 2021