Automatically Build Business Rules using External Data
using External Data
Sometimes using a schema to validate business logic is a case of using the wrong tool for the job. Schemas are generally tightly versioned, and some validation rules that embed any business logic require frequent updating. For example, the complete lists of Australian Business Numbers (ABNs) or Australian Credit Licence (ACL) Numbers are quite dynamic and change frequently. It would be overkill to release a new version of the schema every day to enforce that only valid ABNs or ACLs are provided in the messages.
One of the strengths of using LIXI2 is that there is a library of samples and tools available to get up and running with an impressive amount of automation on the very first day that you are introduced to LIXI.
One piece of automation that we have built for our library of business rules is something that refreshes the rules that validate a number of attributes, including ABNs, BSBs, ACLs and Business Industry Classifications (BIC Codes). You might want to do a similar task for a number of reasons - to refresh product codes or any number of other tasks.
This blog post shares how you can simply write a script to create or update a business rule and test it against a library of sample messages. The example will use the list of Business Industry Code (BIC) list to verify if an institution is using a valid code. This list is publicly available and using this list we will create a business rule that can be validated by the LIXI library against a library of LIXI messages.
This post builds upon concepts already shown in previous posts - make sure to give this post on Automatically Validating Business Rules Against a Library of Messages a read.
Step 1. Prerequisites, Folder creation and Project Clones
As before, you need to have Python 3 and the LIXI Python Package already installed and have access to LIXILab.
As before, create a folder 'lixi-bic-validation-demo' then clone LIXI2 schemas and samples from LIXILab by running the bash/cmd commands below:
C:\temp\lixi-bic-validation-demo>git clone https://standards.lixi.org.au/lixi2/schemas.git C:\temp\lixi-bic-validation-demo>git clone https://standards.lixi.org.au/lixi2/samples.git
Step 2. Copy the Python Code and Run
Create an empty Python file called validate_bic_standalone_script.py and copy the code from the block at the bottom of this page into the file, hit save and run this Python code.
This script performs the following steps:
- Use the
beautiful soup
library to locate and retrieve an excel file from thehttps://data.gov.au
website that contains the full list of BIC codes. - The retrieved excel file is opened using the
openpyxl
library and the BIC codes extracted and used to create a Schematron Rule to validate the BIC Code of any Trust Applicant. - The rule is then used to validate every sample in the LIXI Sample Repository and return any validation errors to the console if they occur.
<sch:rule context="/Package/Content/Application/TrustApplicant/Business/@AustralianBIC"> <sch:assert test=".= '01110' or .= '01120' or .= '01130' or .= '01140' or .= '01150' or .= '01210' or .= '01220' or .= '01230' or .= '01310' or .= '01320' or .= '01330' or .= '01340' or .= '01350' or .= '01360' or .= '01370' or .= '01390' or .= '01410' or .= '01420' or .= '01430' or .= '01440' or .= '01450' or .= '01460' or .= '01490' or .= '01510' or .= '01520' or .= '01590' or .= '01600' or .= '01710' or .= '01720' or .= '01800' or .= '01910' or .= '01920' or .= '01930' or .= '01990' or .= '02011' or .= '02019' or .= '02020' or .= '02031' or .= '02039' or .= '03010' or .= '03020' or .= '04111' or .= '04112' or .= '04120' or .= '04130' or .= '04140' or .= '04191' or .= '04192' or .= '04199' or .= '04200' or .= '05100' or .= '05210' or .= '05220' or .= '05290' or .= '06000' or .= '07000' or .= '08010' or .= '08020' or .= '08030' or .= '08040' or .= '08050' or .= '08060' or .= '08070' or .= '08090' or .= '09110' or .= '09190' or .= '09901' or .= '09902' or .= '09903' or .= '09909' or .= '10111' or .= '10112' or .= '10121' or .= '10122' or .= '10900' or .= '11110' or .= '11120' or .= '11130' or .= '11200' or .= '11310' or .= '11320' or .= '11330' or .= '11400' or .= '11500' or .= '11610' or .= '11620' or .= '11710' or .= '11720' or .= '11730' or .= '11740' or .= '11810' or .= '11820' or .= '11910' or .= '11920' or .= '11990' or .= '12110' or .= '12120' or .= '12130' or .= '12140' or .= '12200' or .= '13110' or .= '13120' or .= '13130' or .= '13200' or .= '13310' or .= '13320' or .= '13330' or .= '13340' or .= '13400' or .= '13510' or .= '13520' or .= '14110' or .= '14120' or .= '14130' or .= '14910' or .= '14920' or .= '14930' or .= '14940' or .= '14990' or .= '15100' or .= '15210' or .= '15220' or .= '15230' or .= '15240' or .= '15290' or .= '16110' or .= '16120' or .= '16200' or .= '17010' or .= '17090' or .= '18110' or .= '18120' or .= '18130' or .= '18210' or .= '18290' or .= '18310' or .= '18320' or .= '18410' or .= '18420' or .= '18510' or .= '18520' or .= '18910' or .= '18920' or .= '18990' or .= '19110' or .= '19120' or .= '19130' or .= '19140' or .= '19150' or .= '19160' or .= '19190' or .= '19200' or .= '20100' or .= '20210' or .= '20290' or .= '20310' or .= '20320' or .= '20330' or .= '20340' or .= '20900' or .= '21100' or .= '21210' or .= '21220' or .= '21310' or .= '21320' or .= '21330' or .= '21390' or .= '21410' or .= '21420' or .= '21490' or .= '22100' or .= '22210' or .= '22220' or .= '22230' or .= '22240' or .= '22290' or .= '22310' or .= '22390' or .= '22400' or .= '22910' or .= '22920' or .= '22930' or .= '22990' or .= '23110' or .= '23120' or .= '23130' or .= '23190' or .= '23910' or .= '23920' or .= '23930' or .= '23940' or .= '23990' or .= '24110' or .= '24120' or .= '24190' or .= '24211' or .= '24219' or .= '24220' or .= '24290' or .= '24310' or .= '24320' or .= '24390' or .= '24410' or .= '24490' or .= '24510' or .= '24520' or .= '24610' or .= '24620' or .= '24630' or .= '24690' or .= '24910' or .= '24990' or .= '25110' or .= '25120' or .= '25130' or .= '25190' or .= '25910' or .= '25920' or .= '25990' or .= '26110' or .= '26120' or .= '26190' or .= '26200' or .= '26300' or .= '26400' or .= '27000' or .= '28110' or .= '28120' or .= '29110' or .= '29190' or .= '29210' or .= '29220' or .= '30110' or .= '30190' or .= '30200' or .= '31010' or .= '31091' or .= '31099' or .= '32110' or .= '32120' or .= '32210' or .= '32220' or .= '32230' or .= '32240' or .= '32310' or .= '32320' or .= '32330' or .= '32340' or .= '32390' or .= '32410' or .= '32420' or .= '32430' or .= '32440' or .= '32450' or .= '32910' or .= '32920' or .= '32990' or .= '33110' or .= '33120' or .= '33190' or .= '33210' or .= '33221' or .= '33229' or .= '33230' or .= '33310' or .= '33320' or .= '33390' or .= '34110' or .= '34190' or .= '34910' or .= '34920' or .= '34930' or .= '34940' or .= '34990' or .= '35010' or .= '35020' or .= '35030' or .= '35040' or .= '35050' or .= '36010' or .= '36020' or .= '36030' or .= '36040' or .= '36050' or .= '36061' or .= '36062' or .= '36090' or .= '37110' or .= '37120' or .= '37200' or .= '37310' or .= '37320' or .= '37330' or .= '37340' or .= '37350' or .= '37360' or .= '37390' or .= '38000' or .= '39110' or .= '39120' or .= '39130' or .= '39211' or .= '39219' or .= '39220' or .= '40000' or .= '41100' or .= '41211' or .= '41212' or .= '41213' or .= '41220' or .= '41230' or .= '41290' or .= '42110' or .= '42120' or .= '42130' or .= '42140' or .= '42210' or .= '42220' or .= '42290' or .= '42310' or .= '42320' or .= '42410' or .= '42420' or .= '42430' or .= '42441' or .= '42442' or .= '42450' or .= '42510' or .= '42520' or .= '42530' or .= '42590' or .= '42600' or .= '42711' or .= '42712' or .= '42720' or .= '42731' or .= '42732' or .= '42733' or .= '42740' or .= '42791' or .= '42792' or .= '42793' or .= '42794' or .= '42799' or .= '43101' or .= '43102' or .= '43109' or .= '43201' or .= '43202' or .= '43209' or .= '44000' or .= '45110' or .= '45120' or .= '45130' or .= '45200' or .= '45301' or .= '45302' or .= '46100' or .= '46210' or .= '46220' or .= '46231' or .= '46239' or .= '47100' or .= '47200' or .= '48100' or .= '48200' or .= '49001' or .= '49009' or .= '50100' or .= '50210' or .= '50290' or .= '51010' or .= '51020' or .= '52110' or .= '52120' or .= '52190' or .= '52200' or .= '52910' or .= '52920' or .= '52991' or .= '52999' or .= '53010' or .= '53090' or .= '54110' or .= '54120' or .= '54130' or .= '54140' or .= '54190' or .= '54200' or .= '55110' or .= '55120' or .= '55130' or .= '55140' or .= '55210' or .= '55220' or .= '56100' or .= '56210' or .= '56220' or .= '57000' or .= '58010' or .= '58020' or .= '58090' or .= '59100' or .= '59210' or .= '59220' or .= '60100' or .= '60200' or .= '62100' or .= '62210' or .= '62220' or .= '62230' or .= '62290' or .= '62300' or .= '62400' or .= '63100' or .= '63210' or .= '63220' or .= '63300' or .= '64110' or .= '64190' or .= '64200' or .= '66110' or .= '66190' or .= '66200' or .= '66310' or .= '66320' or .= '66390' or .= '66400' or .= '67110' or .= '67120' or .= '67200' or .= '69100' or .= '69210' or .= '69220' or .= '69230' or .= '69240' or .= '69250' or .= '69310' or .= '69320' or .= '69400' or .= '69500' or .= '69610' or .= '69621' or .= '69622' or .= '69629' or .= '69700' or .= '69910' or .= '69990' or .= '70000' or .= '72110' or .= '72120' or .= '72200' or .= '72910' or .= '72920' or .= '72930' or .= '72940' or .= '72990' or .= '73110' or .= '73120' or .= '73130' or .= '73200' or .= '75100' or .= '75200' or .= '75300' or .= '75400' or .= '75510' or .= '75520' or .= '76000' or .= '77110' or .= '77120' or .= '77130' or .= '77140' or .= '77190' or .= '77200' or .= '80100' or .= '80210' or .= '80220' or .= '80230' or .= '80240' or .= '81011' or .= '81019' or .= '81020' or .= '82110' or .= '82120' or .= '82191' or .= '82199' or .= '82200' or .= '84010' or .= '84020' or .= '85110' or .= '85121' or .= '85122' or .= '85123' or .= '85124' or .= '85125' or .= '85129' or .= '85201' or .= '85202' or .= '85203' or .= '85209' or .= '85310' or .= '85320' or .= '85330' or .= '85340' or .= '85391' or .= '85399' or .= '85910' or .= '85990' or .= '86011' or .= '86012' or .= '86090' or .= '87100' or .= '87900' or .= '89100' or .= '89210' or .= '89220' or .= '90010' or .= '90021' or .= '90029' or .= '90030' or .= '91110' or .= '91121' or .= '91122' or .= '91123' or .= '91124' or .= '91125' or .= '91126' or .= '91129' or .= '91131' or .= '91139' or .= '91140' or .= '91210' or .= '91291' or .= '91292' or .= '91299' or .= '91310' or .= '91390' or .= '92010' or .= '92020' or .= '92091' or .= '92092' or .= '92099' or .= '94110' or .= '94121' or .= '94129' or .= '94191' or .= '94192' or .= '94193' or .= '94199' or .= '94210' or .= '94220' or .= '94290' or .= '94910' or .= '94990' or .= '95110' or .= '95120' or .= '95200' or .= '95310' or .= '95320' or .= '95330' or .= '95340' or .= '95391' or .= '95392' or .= '95393' or .= '95399' or .= '95400' or .= '95510' or .= '95520' or .= '95591' or .= '95599' or .= '96010' or .= '96020' or .= '96030'" >The Australian BIC on the Business Detail on a Trust Applicant should use a code from the Australian Taxation Office Business Industry Code spreadsheet.</sch:assert> </sch:rule>
What Next? Over to you!
This is an extremely simple rule that caters for validating the BIC code for a Trust Applicant in a message. The concept and code can easily be applied to your own business rules. Best of all, this methodology is a great way of building a suite of rules that you can evaluate in a CI/CD suite of automated tests, prior to loading up an external API with untested inputs. You can easily manage different versions of rules for different scenarios. In this way, you can discover errors earlier in your software delivery process while it is easier and chapter to fix them.
Python Code to Save in 'validate_bic_standalone_script.py'
import glob, os, lixi, sys, requests from bs4 import BeautifulSoup from io import BytesIO from openpyxl import Workbook, load_workbook def get_new_BIC_from_ATO(): # Retreive updated lists from ATO website bic_results = requests.get("https://data.gov.au/data/dataset/ad-hoc-data-requests/resource/72377251-00f3-47a3-b5e3-49eff3e1102a") # Convert them to a format we can query and obtain url for latest bic excel sheet excelURL = BeautifulSoup(bic_results.content, features="lxml").findAll('a', attrs={'class':'resource-url-analytics'})[0].attrs['href'] # Obtain excel sheet with updated BIC codes bic_code_excel = requests.get(excelURL, allow_redirects=True) # Read Excel data to extract BIC wb = load_workbook(filename=BytesIO(bic_code_excel.content)) worksheet = wb[wb.sheetnames[0]] # String with all BIC codes - start bic_string =".= '"+'%05d' % worksheet['A'][1:][0].value # String with all BIC codes - middle for cell in worksheet['A'][1:][1:]: bic_string +="' or .= '"+'%05d' % cell.value # String with all BIC codes - end bic_string +="'" # Drafting the final rule for test rule = ''' <sch:schema xmlns:sch="http://purl.oclc.org/dsdl/schematron" xmlns:xs="http://www.w3.org/2001/XMLSchema"> <sch:pattern id="BIC_test"> <sch:rule context="/Package/Content/Application/TrustApplicant/Business/@AustralianBIC"> <sch:assert test="'''+bic_string+'''">The Australian BIC on the Business Detail on a Trust Applicant should use a code from the Australian Taxation Office Business Industry Code spreadsheet..</sch:assert> </sch:rule> </sch:pattern> </sch:schema>''' return rule # Retrieve this scripts path this_path = os.path.realpath(os.path.join(os.getcwd(), os.path.dirname(__file__))) # Derive the path to the cloned schemas schema_path = os.path.join(this_path, 'schemas','xsd') # Retrieve new BIC codes and create business rule for test bic_rule = get_new_BIC_from_ATO() # Derive the path to the cloned samples glob_search_string = os.path.join(this_path, 'samples','Samples') +'/**/*.xml' # Set the schema folder path in the lixi library lixi.set_schema_folder(schema_path) # Initalise a list of valid files valid_file_list = [] # Initalise a dictionary of invalid files to store the error invalid_file_dict = {} # Create a list of files to validate files_to_check = glob.glob(glob_search_string , recursive=True) # Print the number of files that are found to be validated print('Validating ' + str(len(files_to_check)) + ' files.') # Loop through the files for filepath in files_to_check: try: # Validate each message with the LIXI Library xml_obj = lixi.read_message(message_path=filepath) # Validate each message with the specific BIC rule valid, message, logs = xml_obj.validate_schematron(schematron_schema_text = bic_rule) if valid: # If successful, add the file path to the list of valid files valid_file_list.append(filepath) else: # If business rule validation fails, add the file path to the dictionary of invalid files with the error(s) invalid_file_dict[filepath] = message except Exception as e: # If validation fails, add the file path to the dictionary of invalid files with the error if str(e.__class__.__name__) == 'LIXIValidationError': invalid_file_dict[filepath] = e.error_log elif str(e.__class__.__name__) == 'LIXIInvalidSyntax': invalid_file_dict[filepath] = e elif str(e.__class__.__name__) == 'LIXIResouceNotFoundError': invalid_file_dict[filepath] = e print('Succesfully Validated ' + str(len(valid_file_list)) + ' files.') print('Failed to Validate ' + str(len(invalid_file_dict))) # Loop through the dictionary of invalid file and print the details for invalid_file, invalid_file_error_log in invalid_file_dict.items(): print ('Filename : ' + invalid_file) if type(invalid_file_error_log).__name__ in('LIXIInvalidSyntax', 'LIXIResouceNotFoundError'): print('Error: ' + invalid_file_error_log.args[0].strip()) else: for error_item in invalid_file_error_log: print(error_item.message_location.strip()) print(error_item.message.strip()) print('done')
Related Blogs
Shane Rigby, LIXI Limited CEO
First Published: January 22, 2021