Rulesrobot

Automatically Build Business Rules using External Data

Automatically Build Business Rules
using External Data

Posted on: February 10, 2021

 

Sometimes using a schema to validate business logic is a case of using the wrong tool for the job. Schemas are generally tightly versioned, and some validation rules that embed any business logic require frequent updating. For example, the complete lists of Australian Business Numbers (ABNs) or Australian Credit Licence (ACL) Numbers are quite dynamic and change frequently. It would be overkill to release a new version of the schema every day to enforce that only valid ABNs or ACLs are provided in the messages.

One of the strengths of using LIXI2 is that there is a library of samples and tools available to get up and running with an impressive amount of automation on the very first day that you are introduced to LIXI.

One piece of automation that we have built for our library of business rules is something that refreshes the rules that validate a number of attributes, including ABNs, BSBs, ACLs and Business Industry Classifications (BIC Codes). You might want to do a similar task for a number of reasons - to refresh product codes or any number of other tasks.

This blog post shares how you can simply write a script to create or update a business rule and test it against a library of sample messages. The example will use the list of Business Industry Code (BIC) list to verify if an institution is using a valid code. This list is publicly available and using this list we will create a business rule that can be validated by the LIXI library against a library of LIXI messages.

This post builds upon concepts already shown in previous posts - make sure to give this post on Automatically Validating Business Rules Against a Library of Messages a read.

Step 1. Prerequisites, Folder creation and Project Clones

As before, you need to have Python 3 and the LIXI Python Package already installed and have access to LIXILab.

As before, create a folder 'lixi-bic-validation-demo' then clone LIXI2 schemas and samples from LIXILab by running the bash/cmd commands below:

C:\temp\lixi-bic-validation-demo>git clone https://standards.lixi.org.au/lixi2/schemas.git
C:\temp\lixi-bic-validation-demo>git clone https://standards.lixi.org.au/lixi2/samples.git

Step 2. Copy the Python Code and Run

Create an empty Python file called validate_bic_standalone_script.py and copy the code from the block at the bottom of this page into the file, hit save and run this Python code.

This script performs the following steps:

  1. Use the beautiful soup library to locate and retrieve an excel file from the https://data.gov.au website that contains the full list of BIC codes.
  2. The retrieved excel file is opened using the openpyxl library and the BIC codes extracted and used to create a Schematron Rule to validate the BIC Code of any Trust Applicant.
  3. The rule is then used to validate every sample in the LIXI Sample Repository and return any validation errors to the console if they occur.

<sch:rule context="/Package/Content/Application/TrustApplicant/Business/@AustralianBIC">
  <sch:assert
    test=".= '01110' or .= '01120' or .= '01130' or .= '01140' or .= '01150' or .= '01210' or .= '01220' or .= '01230' or .= '01310' or .= '01320' or .= '01330' or .= '01340' or .= '01350' or .= '01360' or .= '01370' or .= '01390' or .= '01410' or .= '01420' or .= '01430' or .= '01440' or .= '01450' or .= '01460' or .= '01490' or .= '01510' or .= '01520' or .= '01590' or .= '01600' or .= '01710' or .= '01720' or .= '01800' or .= '01910' or .= '01920' or .= '01930' or .= '01990' or .= '02011' or .= '02019' or .= '02020' or .= '02031' or .= '02039' or .= '03010' or .= '03020' or .= '04111' or .= '04112' or .= '04120' or .= '04130' or .= '04140' or .= '04191' or .= '04192' or .= '04199' or .= '04200' or .= '05100' or .= '05210' or .= '05220' or .= '05290' or .= '06000' or .= '07000' or .= '08010' or .= '08020' or .= '08030' or .= '08040' or .= '08050' or .= '08060' or .= '08070' or .= '08090' or .= '09110' or .= '09190' or .= '09901' or .= '09902' or .= '09903' or .= '09909' or .= '10111' or .= '10112' or .= '10121' or .= '10122' or .= '10900' or .= '11110' or .= '11120' or .= '11130' or .= '11200' or .= '11310' or .= '11320' or .= '11330' or .= '11400' or .= '11500' or .= '11610' or .= '11620' or .= '11710' or .= '11720' or .= '11730' or .= '11740' or .= '11810' or .= '11820' or .= '11910' or .= '11920' or .= '11990' or .= '12110' or .= '12120' or .= '12130' or .= '12140' or .= '12200' or .= '13110' or .= '13120' or .= '13130' or .= '13200' or .= '13310' or .= '13320' or .= '13330' or .= '13340' or .= '13400' or .= '13510' or .= '13520' or .= '14110' or .= '14120' or .= '14130' or .= '14910' or .= '14920' or .= '14930' or .= '14940' or .= '14990' or .= '15100' or .= '15210' or .= '15220' or .= '15230' or .= '15240' or .= '15290' or .= '16110' or .= '16120' or .= '16200' or .= '17010' or .= '17090' or .= '18110' or .= '18120' or .= '18130' or .= '18210' or .= '18290' or .= '18310' or .= '18320' or .= '18410' or .= '18420' or .= '18510' or .= '18520' or .= '18910' or .= '18920' or .= '18990' or .= '19110' or .= '19120' or .= '19130' or .= '19140' or .= '19150' or .= '19160' or .= '19190' or .= '19200' or .= '20100' or .= '20210' or .= '20290' or .= '20310' or .= '20320' or .= '20330' or .= '20340' or .= '20900' or .= '21100' or .= '21210' or .= '21220' or .= '21310' or .= '21320' or .= '21330' or .= '21390' or .= '21410' or .= '21420' or .= '21490' or .= '22100' or .= '22210' or .= '22220' or .= '22230' or .= '22240' or .= '22290' or .= '22310' or .= '22390' or .= '22400' or .= '22910' or .= '22920' or .= '22930' or .= '22990' or .= '23110' or .= '23120' or .= '23130' or .= '23190' or .= '23910' or .= '23920' or .= '23930' or .= '23940' or .= '23990' or .= '24110' or .= '24120' or .= '24190' or .= '24211' or .= '24219' or .= '24220' or .= '24290' or .= '24310' or .= '24320' or .= '24390' or .= '24410' or .= '24490' or .= '24510' or .= '24520' or .= '24610' or .= '24620' or .= '24630' or .= '24690' or .= '24910' or .= '24990' or .= '25110' or .= '25120' or .= '25130' or .= '25190' or .= '25910' or .= '25920' or .= '25990' or .= '26110' or .= '26120' or .= '26190' or .= '26200' or .= '26300' or .= '26400' or .= '27000' or .= '28110' or .= '28120' or .= '29110' or .= '29190' or .= '29210' or .= '29220' or .= '30110' or .= '30190' or .= '30200' or .= '31010' or .= '31091' or .= '31099' or .= '32110' or .= '32120' or .= '32210' or .= '32220' or .= '32230' or .= '32240' or .= '32310' or .= '32320' or .= '32330' or .= '32340' or .= '32390' or .= '32410' or .= '32420' or .= '32430' or .= '32440' or .= '32450' or .= '32910' or .= '32920' or .= '32990' or .= '33110' or .= '33120' or .= '33190' or .= '33210' or .= '33221' or .= '33229' or .= '33230' or .= '33310' or .= '33320' or .= '33390' or .= '34110' or .= '34190' or .= '34910' or .= '34920' or .= '34930' or .= '34940' or .= '34990' or .= '35010' or .= '35020' or .= '35030' or .= '35040' or .= '35050' or .= '36010' or .= '36020' or .= '36030' or .= '36040' or .= '36050' or .= '36061' or .= '36062' or .= '36090' or .= '37110' or .= '37120' or .= '37200' or .= '37310' or .= '37320' or .= '37330' or .= '37340' or .= '37350' or .= '37360' or .= '37390' or .= '38000' or .= '39110' or .= '39120' or .= '39130' or .= '39211' or .= '39219' or .= '39220' or .= '40000' or .= '41100' or .= '41211' or .= '41212' or .= '41213' or .= '41220' or .= '41230' or .= '41290' or .= '42110' or .= '42120' or .= '42130' or .= '42140' or .= '42210' or .= '42220' or .= '42290' or .= '42310' or .= '42320' or .= '42410' or .= '42420' or .= '42430' or .= '42441' or .= '42442' or .= '42450' or .= '42510' or .= '42520' or .= '42530' or .= '42590' or .= '42600' or .= '42711' or .= '42712' or .= '42720' or .= '42731' or .= '42732' or .= '42733' or .= '42740' or .= '42791' or .= '42792' or .= '42793' or .= '42794' or .= '42799' or .= '43101' or .= '43102' or .= '43109' or .= '43201' or .= '43202' or .= '43209' or .= '44000' or .= '45110' or .= '45120' or .= '45130' or .= '45200' or .= '45301' or .= '45302' or .= '46100' or .= '46210' or .= '46220' or .= '46231' or .= '46239' or .= '47100' or .= '47200' or .= '48100' or .= '48200' or .= '49001' or .= '49009' or .= '50100' or .= '50210' or .= '50290' or .= '51010' or .= '51020' or .= '52110' or .= '52120' or .= '52190' or .= '52200' or .= '52910' or .= '52920' or .= '52991' or .= '52999' or .= '53010' or .= '53090' or .= '54110' or .= '54120' or .= '54130' or .= '54140' or .= '54190' or .= '54200' or .= '55110' or .= '55120' or .= '55130' or .= '55140' or .= '55210' or .= '55220' or .= '56100' or .= '56210' or .= '56220' or .= '57000' or .= '58010' or .= '58020' or .= '58090' or .= '59100' or .= '59210' or .= '59220' or .= '60100' or .= '60200' or .= '62100' or .= '62210' or .= '62220' or .= '62230' or .= '62290' or .= '62300' or .= '62400' or .= '63100' or .= '63210' or .= '63220' or .= '63300' or .= '64110' or .= '64190' or .= '64200' or .= '66110' or .= '66190' or .= '66200' or .= '66310' or .= '66320' or .= '66390' or .= '66400' or .= '67110' or .= '67120' or .= '67200' or .= '69100' or .= '69210' or .= '69220' or .= '69230' or .= '69240' or .= '69250' or .= '69310' or .= '69320' or .= '69400' or .= '69500' or .= '69610' or .= '69621' or .= '69622' or .= '69629' or .= '69700' or .= '69910' or .= '69990' or .= '70000' or .= '72110' or .= '72120' or .= '72200' or .= '72910' or .= '72920' or .= '72930' or .= '72940' or .= '72990' or .= '73110' or .= '73120' or .= '73130' or .= '73200' or .= '75100' or .= '75200' or .= '75300' or .= '75400' or .= '75510' or .= '75520' or .= '76000' or .= '77110' or .= '77120' or .= '77130' or .= '77140' or .= '77190' or .= '77200' or .= '80100' or .= '80210' or .= '80220' or .= '80230' or .= '80240' or .= '81011' or .= '81019' or .= '81020' or .= '82110' or .= '82120' or .= '82191' or .= '82199' or .= '82200' or .= '84010' or .= '84020' or .= '85110' or .= '85121' or .= '85122' or .= '85123' or .= '85124' or .= '85125' or .= '85129' or .= '85201' or .= '85202' or .= '85203' or .= '85209' or .= '85310' or .= '85320' or .= '85330' or .= '85340' or .= '85391' or .= '85399' or .= '85910' or .= '85990' or .= '86011' or .= '86012' or .= '86090' or .= '87100' or .= '87900' or .= '89100' or .= '89210' or .= '89220' or .= '90010' or .= '90021' or .= '90029' or .= '90030' or .= '91110' or .= '91121' or .= '91122' or .= '91123' or .= '91124' or .= '91125' or .= '91126' or .= '91129' or .= '91131' or .= '91139' or .= '91140' or .= '91210' or .= '91291' or .= '91292' or .= '91299' or .= '91310' or .= '91390' or .= '92010' or .= '92020' or .= '92091' or .= '92092' or .= '92099' or .= '94110' or .= '94121' or .= '94129' or .= '94191' or .= '94192' or .= '94193' or .= '94199' or .= '94210' or .= '94220' or .= '94290' or .= '94910' or .= '94990' or .= '95110' or .= '95120' or .= '95200' or .= '95310' or .= '95320' or .= '95330' or .= '95340' or .= '95391' or .= '95392' or .= '95393' or .= '95399' or .= '95400' or .= '95510' or .= '95520' or .= '95591' or .= '95599' or .= '96010' or .= '96020' or .= '96030'"
    >The Australian BIC on the Business Detail on a Trust Applicant should use a code from the Australian Taxation Office Business Industry Code spreadsheet.</sch:assert>
</sch:rule>

What Next? Over to you!

This is an extremely simple rule that caters for validating the BIC code for a Trust Applicant in a message. The concept and code can easily be applied to your own business rules. Best of all, this methodology is a great way of building a suite of rules that you can evaluate in a CI/CD suite of automated tests, prior to loading up an external API with untested inputs. You can easily manage different versions of rules for different scenarios. In this way, you can discover errors earlier in your software delivery process while it is easier and chapter to fix them.

Python Code to Save in 'validate_bic_standalone_script.py'

import glob, os, lixi, sys, requests
from bs4 import BeautifulSoup
from io import BytesIO
from openpyxl import Workbook, load_workbook

def get_new_BIC_from_ATO():
    # Retreive updated lists from ATO website
    bic_results = requests.get("https://data.gov.au/data/dataset/ad-hoc-data-requests/resource/72377251-00f3-47a3-b5e3-49eff3e1102a")
    
    # Convert them to a format we can query and obtain url for latest bic excel sheet
    excelURL = BeautifulSoup(bic_results.content, features="lxml").findAll('a', attrs={'class':'resource-url-analytics'})[0].attrs['href']
    
    # Obtain excel sheet with updated BIC codes  
    bic_code_excel = requests.get(excelURL, allow_redirects=True)
    
    # Read Excel data to extract BIC
    wb = load_workbook(filename=BytesIO(bic_code_excel.content))    
    worksheet = wb[wb.sheetnames[0]]
    
    # String with all BIC codes - start 
    bic_string =".= '"+'%05d' % worksheet['A'][1:][0].value
    
    # String with all BIC codes - middle 
    for cell in worksheet['A'][1:][1:]:
        bic_string +="' or .= '"+'%05d' % cell.value
    
    # String with all BIC codes - end 
    bic_string +="'"
    
    # Drafting the final rule for test
    rule = '''
    <sch:schema xmlns:sch="http://purl.oclc.org/dsdl/schematron" xmlns:xs="http://www.w3.org/2001/XMLSchema">
      <sch:pattern id="BIC_test">
        <sch:rule context="/Package/Content/Application/TrustApplicant/Business/@AustralianBIC">
          <sch:assert test="'''+bic_string+'''">The Australian BIC on the Business Detail on a Trust Applicant should use a code from the Australian Taxation Office Business Industry Code spreadsheet..</sch:assert>
        </sch:rule>
      </sch:pattern>
    </sch:schema>'''
    
    return rule

# Retrieve this scripts path
this_path = os.path.realpath(os.path.join(os.getcwd(), os.path.dirname(__file__)))

# Derive the path to the cloned schemas
schema_path = os.path.join(this_path, 'schemas','xsd')

# Retrieve new BIC codes and create business rule for test
bic_rule = get_new_BIC_from_ATO()

# Derive the path to the cloned samples
glob_search_string = os.path.join(this_path, 'samples','Samples') +'/**/*.xml'

# Set the schema folder path in the lixi library
lixi.set_schema_folder(schema_path)

# Initalise a list of valid files
valid_file_list = []

# Initalise a dictionary of invalid files to store the error
invalid_file_dict = {}

# Create a list of files to validate
files_to_check = glob.glob(glob_search_string , recursive=True)

# Print the number of files that are found to be validated
print('Validating ' + str(len(files_to_check)) + ' files.')

# Loop through the files
for filepath in files_to_check:
    try:
        # Validate each message with the LIXI Library
        xml_obj = lixi.read_message(message_path=filepath)
        
        # Validate each message with the specific BIC rule
        valid, message, logs = xml_obj.validate_schematron(schematron_schema_text = bic_rule)
        
        if valid:
            # If successful, add the file path to the list of valid files
            valid_file_list.append(filepath)
        else:
            # If business rule validation fails, add the file path to the dictionary of invalid files with the error(s)
            invalid_file_dict[filepath] = message      

        

    except Exception as e:

        # If validation fails, add the file path to the dictionary of invalid files with the error
        if str(e.__class__.__name__) == 'LIXIValidationError':
            invalid_file_dict[filepath] = e.error_log
        elif  str(e.__class__.__name__) == 'LIXIInvalidSyntax':
            invalid_file_dict[filepath] = e
        elif  str(e.__class__.__name__) == 'LIXIResouceNotFoundError':
            invalid_file_dict[filepath] = e

print('Succesfully Validated ' + str(len(valid_file_list)) + ' files.')
print('Failed to Validate ' + str(len(invalid_file_dict)))

# Loop through the dictionary of invalid file and print the details
for invalid_file, invalid_file_error_log in invalid_file_dict.items():
    print ('Filename : ' + invalid_file)
    if type(invalid_file_error_log).__name__ in('LIXIInvalidSyntax', 'LIXIResouceNotFoundError'):
        print('Error: ' + invalid_file_error_log.args[0].strip())
    else:
        for error_item in invalid_file_error_log: 
            print(error_item.message_location.strip())
            print(error_item.message.strip())
            
print('done')

Related Blogs

restriction

Register for LIXLab

documents

Automate LIXI2 Validation

snake

The LIXI Python Package

Written by:
Shane Rigby, LIXI Limited CEO
First Published: January 22, 2021