Machine Learning Not Meeting Your Expectations? Here's Why

In brief:

• The push towards machine learning has resulted in several challenges related to operationalizing and maintaining it long-term.
• How can you ensure lasting adoption and success with machine learning? A comprehensive machine learning operations program.
• Three core principles of an MLOps program will help you make the most of your investment: cross-collaboration, strong measurements and controls, and intentional change management.

Machine learning (ML) is transforming business operations and helping companies maintain a competitive advantage in a rapidly changing world. When implemented well, leaders have access to insights and can more easily identify opportunities, gain a deeper understanding of customers’ needs, improve day-to-day operations, get ahead of market trends, and more.

Despite the benefits businesses can realize with ML, many of them fail to properly take advantage of it. A lot of the time that’s due to unreliable, poorly defined, and quickly crafted processes that do not effectively deliver the valuable insights ML can produce.

Let’s look at an example: A company invests in an ML bot that is designed to enhance customer support. The goal is to use ML to quickly address customer challenges related to billing, product support, and other common needs. The project is off to a good start: The ML team and leaders from across the organization collaborate to develop a model that aligns with and contributes to the overarching business strategy. While the bot is working well in its early days, the company has missed a critical step during implementation: creating a plan for long-term maintenance. In other words, it had not considered that the data pipelines, organizational knowledge, or the bot itself may grow stale over time as the business evolves and real-world conditions change. And leaders lack a clear plan for keeping the bot up to date as these factors shift. While this oversight might not present problems at first, as soon as the company decides to roll out a new line of products, for example, the bot will not register details of the new items and the end-user experience will be compromised. Before you know it, you’ll have customers calling with complaints about dead links or old information.

In this scenario, the organization successfully created a vision of providing faster, easier access and support to customers, but it was not able to achieve value because of technical and organizational challenges. The good news is that those challenges could have been avoided with a rigorous Machine Learning Operations (MLOps) program.

When incorporating ML into your business processes, follow these MLOps program principles to maximize your investment and achieve your vision’s value promise:

1. Cross-collaborate to connect ML to the overarching business strategy

When implementing ML techniques (or any advanced analytics), ensure that there is a close partnership between your data analyst(s) and other key functional or departmental business leaders. Analysts have the technical expertise to build the model, while your functional leader has deep visibility into the business strategy and can ensure the model is designed in a way that helps execute the strategy.

This partnership cannot end once the model has been developed— there must be frequent communication between analysts and leaders as new information becomes available (i.e., a change in a product or policy). These changes need to be fed into the model and the model must be retrained to maintain peak effectiveness.

As a pro tip, you might also consider looping in a software engineer who is well versed in modern IT principles and can ensure software development best practices don’t fall by the wayside in exchange for speed and agility. A software engineer can also help you simplify and improve the reproducibility of ML platforms.

2. Establish strong metrics and stick to best practices

Aging models, like our customer service bot, are frequently a result of poor metrics or lack of metrics altogether. Strong measurements and controls help leaders spot trends or changes—for example, shifts in customer needs or preferences—that would impact the current software and require strategic or technical modifications. With an understanding of what has changed and why the model’s recommendations have evolved, you can retrain and redeploy the bot so it achieves the customer experience you envisioned.

Other approaches to ensuring your software is up to date include:

• Frequently assessing operations feedback loops, which can indicate that something needs to evolve in your model.
• Developing hardened, governed pipelines for the datasets that are used in the predictive algorithm.
• Creating repositories of key pieces of information that feed the algorithm by maturing your feature engineering process. This information might be supportive document metadata, customer information (including purchased products), or a number of historical uses of the bot by this customer, for example.
• Establishing statistics for model quality and actively monitoring those statistics for change, such as average customer satisfaction metrics or time to resolution.
• Formalizing your development processes and automating build pipelines to allow continuous integration and continuous delivery (CICD) in the machine learning discipline.
• Developing automated pipelines for designing and deploying your models—in other words, introducing continuous training (CT) to your existing CICD deployment pipelines so the model is always up to date.

These considerations are a valuable part of maturing MLOps as a technical practice in your organization. This maturity growth is not strictly technical, though. There’s a critical people side to ML adoption.

3. Don’t forget your people in the process

Change management is a vital success factor for long-term adoption of MLOps. Because multiple roles are impacted by it, and it takes a team to pull it off, leaders must:

• Assess current ways of working (WoW) and cultural norms to determine what may need to change to achieve a desired state. Critical to achieving the desired state is a growth mindset, where learning, agility, and continuous improvement are embedded cultural attributes.
• Identify gaps in skills and processes that are required to successfully implement MLOps. Once you’ve identified gaps that need to be filled, establish a plan for reskilling and upskilling to set your employees up for success.
• Conduct a baseline readiness survey to gain quantitative and qualitative insights into how the organization perceives the change. By accounting for your workforce early on in the change process, you can proactively address any resistance that could arise as a result of it. Continue to keep a pulse on how the workforce experiences the change as it unfolds. Deploy periodic surveys to gauge progress of adoption, identify successes, and spot opportunities or needs for additional support.

Developing a comprehensive change management strategy to support new WoW, skills development, and cultural shifts guides leaders and teams through the transformational change, benefits the organization, and enables a greater likelihood of success. In all of these change activities, the organization’s vision must be clear. Leaders must communicate the “why”—why ML contributes to the vision and strategy, why new skills are necessary, why now, etc. Reinforcing these key messages (plus anticipated impacts and expected timelines) keeps teams keenly aware of the direction in which the firm is headed and helps secure workforce buy-in.

Just as development and operations (DevOps) has modernized code development, so too will MLOps modernize machine learning and advanced analytics practices. With embedded IT and change management programs, the benefits of establishing a rigorous MLOps program can be profound, including improved model effective lifespan, increased transparency and trust through effective data governance, and faster development times with more reproducible results.