LIXILAB3

Exploring Product Reference Data under the Consumer Data Right

Exploring Product Data under the CDR

Posted on: Posted: October 1, 2020 | Updated: October 2, 2020


With the passing of the October 1 deadline, the majority of Australian banks are now required to share their product data via APIs as a result of Consumer Data Right legislation (usually referred to under the umbrella of Open Banking).

As the CEO of LIXI, I'm interested in these APIs since some of the data that is sourced from these and other Open Banking APIs invariably make their way into data integrations based on the LIXI standards.

These product APIs (unlike account APIs) expose no customer data, so they do not require any particular credentials to access them, making them a great candidate for a quick exploration to see how easy they are to work with.

This blog post is a quick exploration of how easy it is to work with the product APIs from a variety of these banks using Python (a completely arbitrary decision of programming language). Spoiler alert - it's VERY easy, but I can see some caveats to how easy it will be to use the data for automatic product evaluation and suggestion. We've added a sample showing Node.js at the bottom of this page.

A word of caution - this code is for exploration and illustration purposes only, and no error handling is incorporated at all.

October 1 Deadline

Due to COVID-19, the original deadline for financial services providers to share product data was delayed to 1 October 2020. This includes non-major ADIs, including non-major banks, building societies, credit unions and non-primary brands under the major banks. The major banks already share this product reference data.

Product reference data includes rates, fees and features of banking products (specified here).

Where to start?

Each bank publishes their API details somewhere, and in a quick search, the Westpac Open Banking Product API is at the top of the list. This page shows the URL of their Get Products API.

Let's start a new python script, import two libraries (json and requests), and create a variable for the url. Setting up your Python environment is beyond the scope of this blog post, I've assumed you can do this already. Alternatively, use one of the many on-line Python interpreters such as the one here (an online tool like this is ok for the product API as it contains publicly available data only).

import requests, json
url = "https://digital-api.westpac.com.au/cds-au/v1/banking/products"

Now since the product API is available to everyone with no restrictions on use, it is a simple as calling a GET request and saving the response into a variable.

response = requests.request("GET", url
                , headers={'Content-Type': 'application/json','x-v': '2','Cookie': ''}, data = {})

We can now load the response data into a json object, iterate through it and interrogate the data for some fields that interest us. The structure of the product details in this json object is available in the CDR Specification here. We might want to print the name, brand, and productId of every product in the list to the console.

productlist = json.loads(response.text.encode('utf8'))['data']['products']
for product in productlist: 
  print(product['name'] + ' offered by ' + product['brand']+ ' (' + product['productId'] +')')

And the console shows:

RAMS Saver  offered by RAMS (RAMSSaver)
RAMS Line Of Credit offered by RAMS (RAMSLineOfCredit)
RAMS Full Feature Home Loan offered by RAMS (RAMSHLVariableFullFeature)
RAMS Essential Home Loan offered by RAMS (RAMSHLVariableEssential)
RAMS Fixed Rate Home Loan offered by RAMS (RAMSHLFixedRate)
RAMS Fixed Rate Classic Home Loan offered by RAMS (RAMSHLFixedRateClassic)
Business Term Deposit offered by St. George Bank (STGTDBusTermDeposit)
Term Deposit offered by St. George Bank (STGTDTermDeposit)
Business Term Deposit offered by BankSA (BSATDBusTermDeposit)
Term Deposit offered by BankSA (BSATDTermDeposit)
Business Term Deposit offered by Bank of Melbourne (BOMTDBusTermDeposit)
Term Deposit offered by Bank of Melbourne (BOMTDTermDeposit)
RAMS Action with Offset offered by RAMS (RAMSActionwithOffset)
RAMS Action  offered by RAMS (RAMSAction)
Concession  offered by Bank of Melbourne (BOMTrConcession)
Retirement Access Plus offered by Bank of Melbourne (BOMTrRetirementAccess)
Complete Freedom  offered by Bank of Melbourne (BOMTrCompleteFreedom)
Business Vantage offered by Bank of Melbourne (BOMCCBusBusinessVantage)
Amplify Business offered by Bank of Melbourne (BOMCCBusAmplifyBusiness)
Vertigo card  offered by Bank of Melbourne (BOMCCVertigocard)
Amplify Signature card (Rewards) offered by Bank of Melbourne (BOMCCAmplifySignaturecardRewards)
Amplify Platinum card (Rewards) offered by Bank of Melbourne (BOMCCAmplifyPlatinumcardRewards)
Vertigo  Platinum offered by Bank of Melbourne (BOMCCVertigoPlatinumcard)
NO ANNUAL FEE offered by Bank of Melbourne (BOMCCNoannualfeecard)
Amplify card (Frequent Flyer) offered by Bank of Melbourne (BOMCCAmplifycardFrequentFlyer)

Well that was easy, in six lines of code, we can obtain product data, but why are there only 25 records in the response? Surely Westpac has more than 25 products?

Pagination

The API is only returning the results one page (of 25 results) at a time, but the API conveniently returns the URL for the next page of responses, so we simply need to loop through until there is no next URL provided. You can read about this in the specification here.

while 'next' in json.loads(response.text.encode('utf8'))['links'] \
    and json.loads(response.text.encode('utf8'))['links']['next'] is not None:
  next_url = json.loads(response.text.encode('utf8'))['links']['next']
  response = requests.request("GET", next_url
                , headers={'Content-Type': 'application/json','x-v': '2','Cookie': ''}, data = {})
  productlist = json.loads(response.text.encode('utf8'))['data']['products']
  for product in productlist: 
    print(product['name'] + ' offered by ' + product['brand']+ ' (' + product['productId'] +')')


More Financial Institutions?

What if I want a selection of products from across a range of financial institutions? It's easy to create a list of a number of APIs and simply loop through each of those URLS. The entire script, with absolutely zero error handling (so buyer beware), now looks like this:

import requests, json

urls = ['https://api.cdr.adelaidebank.com.au/cds-au/v1/banking/products',
'https://api.cdr-api.amp.com.au/cds-au/v1/banking/products',
'https://api.anz/cds-au/v1/banking/products',
'https://api-pub-cdr.suncorpbank.com.au/cds-au/v1/banking/products',
'https://digital-api.westpac.com.au/cds-au/v1/banking/products']

for url in urls:

  response = requests.request("GET", url
                  , headers={'Content-Type': 'application/json','x-v': '2','Cookie': ''}, data = {})

  productlist = json.loads(response.text.encode('utf8'))['data']['products']
  for product in productlist: 
    print(product['name'] + ' offered by ' + product['brand']+ ' (' + product['productId'] +')')

  while 'next' in json.loads(response.text.encode('utf8'))['links'] \
      and json.loads(response.text.encode('utf8'))['links']['next'] is not None:
    next_url = json.loads(response.text.encode('utf8'))['links']['next']
    response = requests.request("GET", next_url
                  , headers={'Content-Type': 'application/json','x-v': '2','Cookie': ''}, data = {})
    productlist = json.loads(response.text.encode('utf8'))['data']['products']
    for product in productlist: 
      print(product['name'] + ' offered by ' + product['brand']+ ' (' + product['productId'] +')')


Conclusion

It's very easy to use the Product List APIs, at least at a superficial level in order to look at hundreds of banking products in just a few minutes of coding and a few seconds of CPU time. I noticed that many of the products have additional whitespace around the name of the product, which does make me wonder about an issue that often arises in working with data standards. Over the years, I've seen any number of ways in which a well-defined standard can leave room for interpretation and variability of implementation. This can lead to unnecessary complexity in how consumers actually use the APIs in real life. Trimming whitespace is trivial, but there are many seemingly minor variations that can have consequences that are far from trivial.

My next step (in a future blog post) will be to look at the quality and consistency of the data as an indicator as to how variable the implementations are, particularly in the more detailed data available in the product detail API.

Another point to note is that all the eligibility data (specified here) that a participant would need to evaluate in order to actually suggest a product to a customer is mostly strings or URLs. It would be very hard to use this data to programmatically analyse the eligibility of a specific product for a specific customer at this stage. This concept of eligibility, particularly when it comes to credit products can become extremely complex. The scale and comprehensiveness of the LIXI data standards is evidence of this, given that the LIXI Data Standards now have been used for 20 years to originate credit products in the Australian lending industry.

Addendum - Node.js Sample

Here is the Node.js you can use to accomplish the same thing, and you can use an online environment such as this one and simply copy the code below into the index.js file and select Run.

const axios = require("axios").default

const options = {
  headers: {
    "Content-Type": "application/json",
    "x-v": "2",
  },
};

function getProduct(url) {
  return response = axios.get(url, options);
}

function desiredOutput(products) {
  const { name, brand, productId } = products;
  return `${name} offered by ${brand} (${productId})`;
}

async function productList(url,option = {"all":false}){
  let hasNextPage = false
  let nextPageUrl = url
  do {
    const response = await getProduct(nextPageUrl);
    for (const pro of response.data["data"]["products"]) {
      console.log(desiredOutput(pro));
    }
    nextPageUrl = response.data["links"]["next"] 
    hasNextPage = !!nextPageUrl;
  }while(hasNextPage && option["all"]=== true);
}

//  To see all the products, pass {"all":true} as the second argument to the productList function
productList("https://digital-api.westpac.com.au/cds-au/v1/banking/products",{"all":true})



Written by:
Shane Rigby, LIXI Limited CEO
First Published: October 1, 2020 | Last Updated: October 2, 2020