Adding Value with AI: A True Story

Organizations everywhere have begun to embrace Artificial Intelligence (AI). The PVSC story indicates the opportunities and challenges that come with the territory.

Kathy Gillis, the CEO of Property Valuation SerIN MARCH 2018, vices Corporation (PVSC), saw an opportunity in the property assessment field to use artificial intelligence (AI). Having little experience in the topic, she and her Vice President of Strategy, Meredith Buchanan, decided to bring their idea to a threeday AI seminar at the Rotman School of Management. The two quickly realized that PVSC was among the first to consider applying AI to the property valuation field globally.

Gillis brought the idea back to PVSC’S Chief Data Scientist, Dr. Ashley Wu, who was keen to pursue an AI approach. Within months, Wu’s team discovered they could use machine learning to predict property values faster, cheaper, and with more accuracy than any other technology. Now, Gillis needed to develop a strategy that would take advantage of this opportunity.

To value properties, assessors collect property data such as lot and building size, number of bedrooms, quality of construction, and recent renovations, often by visiting the site. This data is then combined with real estate market sales information to determine the property’s value. Each year, PVSC reassesses all properties in Nova Scotia, as municipalities use the property values to determine the amount property owners will be taxed. Notices are provided to owners indicating any change in their value. Owners who disagree with the changes may appeal this new value. Although appeals are rare, the process allows

PVSC to gain up -to-date information and may result in revisions to the assessment.

At the outset, Gillis envisioned a tool to “raise the bar for accountability” across the industry, which may require “building a new software or a new CAMA [computer-assisted mass appraisal] system.” Wu felt that the latest AI methods might fit well with PVSC’S “focus on optimization” and that the switch may become “inevitable” for the industry. Buchanan was not convinced. She worried that this might distract from their implementation of multiple regression analysis models for assessment — a project that had been underway for a year and a half, with significant time and resources invested.

Gillis was eager for Wu to start working on this opportunity immediately, but first she wanted to consider the perspectives and potential impacts it could have organizationally and consulted with her executive team. The team identified concerns focused on transparency and how the values predicted with machine learning could be explained and defended to the public, as well as the downside of having no industry standard to support this approach. Given the potential business impact and the need to pause the multiple regression analysis project — which, as Buchanan had indicated, had been a business priority for over a year — Gillis needed to gain the support of the PVSC Board of Directors.

One Board member cautioned Gillis “not to let shiny projects distract her from PVSC’S core services.” In agreement, another longstanding Board member underlined that PVSC would not “be doing AI for AI sake” and that it would be pulled if it was found to be less accurate or more expensive. Despite these concerns, Buchanan recounted a sense that “people felt they needed to support this. The majority of board members were keen to have PVSC research machine learning at the very least, to gather information and recommendations.

Soon after, Gillis gave the green light for Wu to begin research. From July to September 2018, Wu and three data modelers undertook an intensive machine learning course with a statistics expert at Dalhousie University. By the end of that summer, “the team had accomplished more in terms of the modelling, the results, the direction, and the strategy than had been accomplished in the previous year and a half.”

After comparing several approaches, Wu found that two machine learning models — the ‘gradient boosting’ and ‘generalized additive’ methods — best predicted the market value of residential properties in Nova Scotia, and that these models were more accurate than any of the internally or externally tested regression-based models. The statistician at the university reported to Gillis a “14 per cent average mean error, with as many as 80 per cent of properties having a mean error as low as eight to 10 per cent.” Gillis was excited by the accuracy of the results, as well as the speed at which the values had been predicted. This was big news for the industry and she was eager to share the results. Soon after going public, PVSC was asked to conduct a pilot study for another jurisdiction outside of Nova Scotia.

A Board member advised Gillis that machine learning may not be a permissible method for property assessment depending on jurisdictional legislation. In Nova Scotia, the method for conducting property assessment is not prescribed. Regardless of method, the property must be assessed at ‘market value’, defined as ‘the amount which in the opinion of the assessor would be paid if it were sold on a [decided] date in the open market by a willing seller to a willing buyer.’ Gillis turned to her legal counsel and found that the region requesting the pilot study, like Nova Scotia, was not a prescriptive jurisdiction. However, PVSC would still need to prove that machine learning techniques met the International Association for Assessment Officers (IAAO) standard statistical tests for property assessment.

As Gillis shared her results, experts in the property assessment field criticized the approach, remarking that machine learning is a ‘black box’ method that cannot be defended in appeals and is therefore not transparent. To address this issue, Wu and her team developed tools to help assessors defend values and increase transparency with property owners. Wu informed Gillis that the Model Reports tool would generate a report showing the assessor all of the property and market data that contributed to the value, weighing the variables to indicate which had the most impact. For instance, the distance from Halifax would contribute 25 per cent of the predicted value. The Comparable Sales Application is aptly named and provides five or six actual property sales that share attributes with the property being assessed. Wu noted that this allows for an intuitive comparison between the predicted and the market value for the property owner, and also enables errors with the algorithm to be flagged within PVSC.

To solidify the viability of the approach, Gillis asked an experienced assessor to conduct internal audits to ensure the machine learning results were compliant with the IAAO statistical standards and to compare them with PVSC’S traditional approach to valuation. After reviewing the results, the internal auditor advised Gillis that there were “not many issues in meeting the standards”. He acknowledged that any method for prediction would have some errors, and that the machine learning method faced the same challenges with outliers as other approaches — and may even result in fewer errors. Overall, the whole process was much more efficient.

After receiving this support for the approach, Gillis, Wu, and Buchanan pushed for PVSC to start using the machine learning approach in property assessment.

Developing a Machine Learning Strategy

Buchanan and Gillis laid out four viable scenarios for incorporating machine learning into PVSC’S business model:

1. Continue doing assessments without machine learning;

2. Use machine learning to predict assessment values within PVSC;

3. Offer assessment services to other jurisdictions, within the limitations of a not-for-profit organization; and

4. Create a subsidiary of PVSC and a for-profit structure.

Let’s take a closer look at each.

Gillis SCENARIO 1: CONTINUE SERVICES WITHOUT MACHINE LEARNING. knew that focusing on delivering quality property assessments for municipalities without any drastic innovation would be the

safest scenario for PVSC. As a non-profit, PVSC currently had an annual budget of $17 million and faced no competition.

Continuing with the status quo would also avoid the potential for internal conflicts. Gillis could reinstate the multiple regression analysis project and that the 18-month investment would not be lost. If Gillis were to implement machine learning in PVSC’S assessments, the skills required to be an assessor would likely change. Buchanan noted that the new skill set would focus on “data collection, client relationships, value defense, and market expertise.” Gillis believed that many assessors would be able to transition into these roles, but others would find the new skillset difficult or draining, and altogether fewer assessors may be needed. Implementing regression analysis instead of machine learning might be a reasonable stepping-stone towards a more advanced approach for assessors, and it was currently accepted by the industry. However, if they did not pursue machine learning, Gillis feared PVSC may be “subsumed by private sector organizations” who would then develop the technology to offer quick, cheap, and accurate appraisals.

SCENARIO 2: USE MACHINE LEARNING TO PREDICT ASSESSMENT VAL

When considering where to introduce maUES WITHIN PVSC. chine learning in the assessment process, Gillis, Wu, and Buchanan agreed that property value prediction should be the focus. Buchanan commented that officially altering the assessment approach, as opposed to adding peripheral tools such as customer chat bots, would be “the greatest risk but also the greatest opportunity for reward.” Designing a new tool to conduct the assessment may require a skill change amongst assessors and additional data science skills. A machinelearning approach may also cause challenges for assessors defending values during the appeal process and it was not yet recognized as an acceptable methodology by the industry.

SCENARIO 3: OFFER SERVICES TO OTHER JURISDICTIONS, WITHIN THE

Since Nova Scotian LIMITATIONS OF PVSC AS A NOT-FOR-PROFIT. municipalities fund PVSC, the organization could only allocate funds to projects which maintain and improve services for the municipalities. Offering services externally as a not-for-profit would permit projects to be legally undertaken on a cost recovery basis with the objective ‘to improve tools and services at PVSC to better serve Nova Scotian municipalities.’ This strategy would allow PVSC to explore lines-of-service options in external jurisdictions without altering its current not-for-profit status.

Based on the large number of service requests at conferences, Gillis felt that there were three opportunities for lines of service:

a. offering consulting services on how to build and maintain models and how to implement machine learning as a methodology; b. building models and licensing their use; or c. building and maintaining models as a fully outsourced pro

vider.

Gillis considered partnering with the interested Canadian assessment jurisdiction to test PVSC’S model in a more active market. This would prove the approach in a different market and be a valuable learning opportunity; however, she was unsure how much knowledge could be transferred without compromising intellectual property. This project would require PVSC to build and license a model to the jurisdiction (line of service b). However, more knowledge transfer may be required for the assessment jurisdiction to maintain the model. Further involvement, such as intellectual property transfer, may not be in PVSC’S best interest and would go beyond the required improvement mandate.

SCENARIO 4: CREATE A SUBSIDIARY OF PVSC WITH A FOR-PROFIT STRUC

As demand for access to their knowledge increased, Gillis TURE. considered developing services to generate revenue. Seeing the opportunity for the corporation to become a profitable venture, she felt that there was “significant risk in going too slow.” However, Gillis knew that the team lacked the required international law, tax, and finance expertise to advise on a global scale. To pursue this strategy, Gillis would need to attract skills specific to serving international clients in the property assessment field.

As an additional impediment, expanding services and generating a profit would jeopardize PVSC’S not-for-profit status and could also increase the liability to the municipalities that currently funded PVSC. To delineate liabilities and the associated revenue streams, Gillis could create a subsidiary, which the Province indicated would require a change to PVSC’S legislation. Changing legislation takes time, and there are many organizations and competing interests at play that influence which requests for legislation make their way to the legislature. Gillis acknowledged that a potential competitor could emerge between now and when the legislation was eventually passed.

Gillis needed to weigh her options and assess the strategic challenges. Would incorporating AI be the best approach? What

would the potential outcomes be for each option? In the following section, PVSC’S leaders describe what happened next.

The Road Taken: An Update by Kathy Gillis, Hugh Fraser, and Kim Ashizawa

Ultimately, PVSC continued to embrace machine learning. However, we did not choose to follow just one option. Instead, we are now employing two of the strategies detailed by Professor Goldfarb, while planning for a third. Here’s what has happened since 2019:

In 2020, we were PUTTING MACHINE LEARNING STRATEGIES TO WORK. able to move forward with Scenarios 2 and 3, but our corporate goal remains to shift from Scenario 3 to Scenario 4. At this point, the Provincial Government has yet to introduce the legislation that would clear the way for a for-profit subsidiary. While that legislative change is expected in time, PVSC is still able to use machine learning for its core business of providing assessments for Nova Scotia’s 49 municipalities and it is also able to move forward with pilot-scale projects with customers outside of the province.

With the release of our annual asMACHINE LEARNING WITHIN PVSC. sessment roll for Nova Scotia in January of 2020, PVSC became the first Canadian jurisdiction to incorporate machine learning into mass appraisal. We have successfully used machine learning to provide appraisals for 98,000 residential properties and about 3,000 condominiums in the province. In total, there are some 630,000 properties in Nova Scotia and to date, properties appraised with machine learning have received fewer appeals than other properties.

Following on that success, we began preparing our 2021 roll with a goal of ‘more machine learning’. However, about midway through the process, assessors noticed that our model was not providing accurate valuations. We made a decision: For the 2021 assessment roll, we will use our traditional valuation methods for the lion’s share of properties. Machine learning will be used as a quality check for some properties, but until we confirm that our model is robust, it isn’t worth the risk.

Rigorous internal and external reviews made clear that changes were needed to both our model and our organizational structure. The reviews also made clear that machine learning absolutely had to be central to our business model going forward. PVSC changed its org structure so that appraisers and data scientists now work together, not in silos. Months into this change, the cross-pollination has already paid off, creating reliable modelling and strengthening the work of both assessors and data scientists.

Overall, PVSC’S MACHINE LEARNING FOR EXTERNAL PARTNERS/CLIENTS. this experience showed everyone involved the value of adversity in innovation. We were in the midst of working out a services contract with a Dutch mass appraisal organization when we made the decision to pause the machine learning program for the coming assessment roll. The Dutch client appreciated our transparency and underlined how this challenging time will actually help build a more attractive, battle-tested business model for our customers. Knowing that a service provider has faced — and then cleared — the same hurdles that clients might face only strengthened the offering.

For our team, it also became clear that the machine learning tool itself was not our strategy for business development; instead, it was a means to an end and one of the tools that we could offer to clients. Along with the lessons of experience and the importance of change management, this experience demonstrated that the alignment of ‘old school’ approaches with new mindsets serves to building a stronger model — and organization.

In closing

They say luck is what happens when preparation meets opportunity. When the pandemic hit North America in March of 2020, PVSC was able to turn to its machine learning expertise to help make critical decisions in a time of great anxiety and uncertainty. It became clear to its leaders that the unparalleled impacts of COVID-19 and the province’s economic shutdown would likely

COVID-19 has created new opportunities to use machine learning—in property assessment and many other areas.

at PVSC. Kim Ashizawa

The authors thank Leah Morris research assistance.