How to setup a Center of Excellence for AI?

Posted on by Gurjeet Singh 12 mins read

This is a post about specific tactics and recommendations for building a Center of Excellence for AI in large enterprises.

Why build a COE for AI?

The future of all business operations fundamentally relies on utilizing AI and Machine Learning technologies - even conservative estimate puts it in the billions for large enterprises. The big technology companies have already realized this and have been acquiring both unique technologies and the world’s best talent. This is evident with all the new product announcements of late (Siri/Apple HomePod, Google Assistant, Google Duplex, Facebook facial recognition). Many countries, having recognized this trend are already investing (China, Singapore, France6, Japan), or planning (Canada).

Andrew Ng puts it nicely

“In the past, a lot of S&P 500 CEOs wished they had started thinking sooner than they did about their Internet strategy. I think five years from now there will be a number of S&P 500 CEOs that will wish they’d started thinking earlier about their AI strategy.”

All of this is great, but going forward, keep in mind that AI is still a rapidly growing field - there is significant new research that comes out every day. This is much like the internet was in the 2000’s - wildly important, but still being actively developed.

What is AI?

It is important to address what we mean by AI before we go any further. AI is a very broad concept with very fuzzy boundaries and as such remains hard to define. Some use the term to refer to AGI - Artificial General Intelligence, which is the pursuit of creating hardware/software systems such that the same system is at least competitive with human beings in a broad range of tasks. We’re very far from being in this ballpark.

The main difference between regular programs and AI programs is that regular programs are carried out by executing a fixed algorithm. AI programs on the other hand, are parameterized by data. In other words, the behavior of the algorithm can vary vastly based on the data that it is exposed to. For the purposes of this article, I will use the term AI and Machine Learning synonymously.

That said, it is important to think about the problem domains where AI is applied in the enterprise context. There are broadly two problem domains - where the data is based on perception and where it is not:

  1. Perception based problems work with data such as images, audio, video, sound etc. The key fact about perception data is that it tends to be overcomplete for a certain class of inference tasks. In other words, imagine taking a picture with a 12 mega-pixel camera and the same scene with a 1 mega-pixel camera. If you showed the two pictures to a passerby, they will likely make correct, general inferences. Thus, an image with 90% fewer pixels contained enough data to make inferences - i.e. the problem is robust to deleting some data. Many of the challenges of enterprises interfacing with their customers likely fall in this category (e.g. biometrics, chatbots etc.)
  2. Non-perception based problems work with data that is high-dimensional and sparse i.e. data where for every entity we have lots of information. Imagine the rich data that an enterprise has about their customers - as we add sensors to the world, this will keep getting richer. Now, if you look at a table with each row representing a customer, the table will have lots of columns - this is high dimensionality. Unlike perception based data, these problems tend not to be robust to deleting data - you need every last bit of it to make inferences. This is a fundamental difference. Most of the challenges of backend operations in an enterprise fall in this category (e.g. regulatory risk modeling, financial crimes, clinical variation management etc.)

A note about Models

The thinking around practical use of AI is often simplified to mean predictive analytics models - i.e. constructing models that are predictive. This is by far insufficient. Here’s why:

  1. Predictive models need clean, labelled data. The vast majority of data in large enterprises is unlabelled and it turns out that labelling is a very expensive exercise. So what do you do with 99% of your data which is unlabelled?
  2. Models are very difficult to deploy in production. Often large enterprises take models developed in Python/R etc. and re-implement them in production systems in Java/.Net etc.
  3. Models are not applications. I believe that AI is useless without UX. It is really important that front end line of business owners be able to benefit from AI and for that to happen the results of models must be wrapped in UX that makes business sense.
  4. Models don’t evolve. Typically, organizations re-build models manually every so often. The issue with this approach is that the underlying distributions in data can change without the end user realizing - making the model less effective.

Here are 5 basic ideas to look for in any AI platform:

  1. Discover - the capability to learn patterns from large complex data without upfront human intervention. This relies on unsupervised, semi-supervised and generative machine learning techniques and outputs artifacts such as segments and anomalies. Note that this does not require labelled data - it can learn from unlabelled data (and utilize labels if they are available).
  2. Predict - the capability to make predictive models.
  3. Justify - as narrow AIs start getting integrated into the enterprise fabric, these systems will need to build trust with their human operators. Justify relies on student-teacher learning to provide independent justification behind every ‘decision’ proposed by an AI system. This is especially important in regulated industries.
  4. Act - is the ability to integrate AI systems from an engineering point of view. This means the ability for the development system to:
    1. Consume data regularly
    2. Process and push out results to other systems (e.g. push predictive leads into a CRM system)
    3. Provide an interface to build and deploy applications which can be used directly by line of business users.
  5. Learn - is the ability of the AI system to monitor data and suggest to its human operators when the models degrade, or when it finds a new segment in data or a new type of anomaly etc.

Why you need a COE

Every major platform technology wave that gets adopted in the enterprise exhibits the same pattern:

  1. Slow adoption at the fringes where either the need is great or there’s freedom in the enterprise to experiment.
  2. Select customers or enterprises begin dominant use of the technology, both reaping huge benefits as well as training a set of future leaders.
  3. Other enterprise customers, who were left behind, see the broad benefits and start adopting the technology.

This has happened over and over again, from the transition to cloud based software (SAAS), to mobile development. Every large enterprise on the planet today has a COE for cloud and mobile. AI will be no different.

The mission for the COE

Alright, if you are reading this, you likely are bought into the fact that you need a COE. So what does the COE need to accomplish:

  1. A delivery center. While there is scope for experiments, it is really important that the COE deliver results to the business. This is what most COEs get wrong - they setup it up to experiment and research, but not deliver.
  2. Deliver applications. Today, the thinking around AI and ML mostly stops at models or dashboards. This is a good first step, but to have real impact with this technology, it must be expressed within applications that are usable by the end line of business users, who should not need any expertise in AI/ML to use these applications. Think Apps not Models.
  3. Data culture. Data is a pre-requisite to successfully using and deploying AI and in most enterprises, it is quite broken and siloed. An AI COE needs data and a very pleasant by-product is the culture of treating data as an asset.
  4. A program that leverages AI software to help achieve key business objectives, while building strong capabilities in AI.
  5. Self-sufficiency in deploying AI based applications.

With the mission understood, let’s look at some concrete inputs and outputs. Here’s what the organization needs to provide the COE:

  1. Business imperatives
  2. Governance and prioritization from leadership
  3. Data Assets
  4. Program Management discipline
  5. IT requirements
  6. Resources capable to analyzing data and delivering applications (job descriptions and roles in the next section)

Here’s what the organization should expect from the COE:

  1. Operational applications delivering measurable business value
  2. A team of trained resources adept at building ML applications
  3. A knowledge base that drives further development
  4. Industry and organization specific best practices around data management, application build, change management, program management, and reusable code base of recipes/cookbooks
  5. Reduced silos of knowledge and data

Roles and responsibilities

While a lot of the media discussion is about machines replacing people but AI in an enterprise setting requires people - and smart ones at that. This is a very tactical section - the idea being that you should be able to copy-paste the following into a job description/requirement doc. Here are the roles and what to look for when hiring:

  1. Program Manager: Accountable for the overall COE program. Communicate priorities between the leadership and program teams, manage schedule and budget and provide risk management. The ideal candidate will be an excellent leader and will have experience in managing staff of different disciplines to produce results in a timely manner. Requirements are:
    • Over 7 years of experience managing complex, business driven programs at large enterprises.
    • Experienced in commanding a team of personnel (both direct and matrix reporting relationships) and driving to results
    • Expert in leveraging inter-departmental resources towards meeting program goals
    • Skilled at project management principles, Prince2 or PMP a plus
    • A big-4 consulting experience a plus, management degree a plus. Technical degree is a nice to have.
    • Adept at using project management software (Asana, JIRA, Microsoft Project etc)
  2. Business Solution Architect: Understands the nature of the line of business, the driving imperatives and can architect a solution that clearly defines the desired end application state, and the benefits driven by moving from the current state (as-is) solution. Requirements are:
    • Over 5 years of experience building solutions in a multi-faceted environment
    • Able to juggle conflicting requirements and arrive at a clear well documented solution, with buy in from all stakeholders
    • Adept at understanding the value driven by state of art technology, and crafting an end to end solution workflow
    • Skilled at communication, both written and oral, with all types of stakeholders
    • Adept at using Microsoft Office or other equivalent tools to produce flowcharts and requirements documentation.
    • Experience using project management software such as JIRA.
    • A business degree or technical degree is a requirement
  3. Technical Solution Architect: Accountable for understanding and inspecting business requirements, collaborating and architecting a technical solution that clearly defines the desired end application state, and the benefits driven by moving from the current state (as-is) solution. Requirements are:
    • Over 5 years of experience identifying and translating business requirements into functional design, prototyping, process design (including scenario design, flow mapping), testing, training, defining support procedures and supporting implementations.
    • Expert at driving a team to deliver the approved technical solution, following the design and methodology adopted
    • Experienced at performing full system development life cycle implementations.
    • Experienced at creating conceptual and technical designs for multi-tier applications.
    • Expert at developing architecture design documents, including data, interface, architectural and procedural design for large enterprise companies
    • Highly qualified in Python programming. High proficiency in developing using SQL, PL/SQL and a good understanding and exposure to relational databases.
    • Experience in big data technology stack is a plus (Hadoop, Hbase, Sqoop, Hive, Spark etc.,).
    • Skilled at communication, both written and oral, with all types of stakeholders
  4. Developer: Responsible for delivery of robust, high quality applications following an approved design. Requirements are:
    • Over 5 years of experience developing, testing, documenting and delivering complex business solutions and applications
    • Expert at delivery of full system development life cycle implementations, following the technical design for multi-tier applications.
    • Highly qualified in general best practices of programming, expert at Python programming. High proficiency in developing using SQL, PL/SQL and a good understanding and exposure to relational databases.
    • Experience in big data technology stack is a plus (Hadoop, Hbase, Sqoop, Hive, Spark etc.,).
    • Skilled at communication, both written and oral, with all types of stakeholders
  5. Data Scientist: Responsible for driving strong analytical strategy given the requirements of the business, and a strong ability to inspect available data. Requirements are:
    • Over 5 years of experience utilizing Machine Learning in an enterprise setting.
    • Solid understanding of statistics and machine learning algorithms.
    • An MS or PhD in a quantitative field. Note that academic backgrounds that might be considered non-traditional in your area are often a great place to hire out of. Example - as a bank, try hiring bioinformaticists.
    • Expert in coming up with analytic strategy given a business requirement, and data descriptions
    • Excellent at converting analytical ideas using production-quality Python code, building a pilot that holds up to robust data science principles
    • Expertise with data science toolkits such as scikit-learn
    • Skilled at communication, both written and oral, with all types of stakeholders
    • Collaborate closely with IT, business, leadership and other stakeholders to implement applications
  6. IT project manager/SME: Responsible for ensuring the COE has alignment with IT, clears IT hurdles and ensures timeline and quality requirements are met as the ML application moves from a pilot stage through to a production deployment. Requirements are:
    • Over 5 years of experience working within the organization deploying various applications in a multi-faceted organization structure
    • Identify risk and compliance roadblocks and creative at finding solutions to these issues. Drive onboarding of resources, access to data, installation of various packages, and provide clear direction when moving any application to production.
    • Works closely with the COE program manager to ensure COE needs are met.

For most sophisticated organizations – all of these folks are in the building. The challenge for the management team is to get them all in the same room and pursuing the same goals.

Delivery

Services delivered via a COE program:

  1. Value Management: Ensure that the organization invests in valuable projects and derives optimal benefit.
  2. Demand Management: Allocating resources (people, funding) across all projects in flight and monitoring spend is a critical function of COE.
  3. Execution Support: The CoE supports various lines of business during its lifetime. This requires a provision of dedicated technical and subject matter experts, working together with operational resources from the line of business.
  4. Data management: Manage data through stages in its lifetime, from growth to maturity to a managed decline.
  5. Enablement: Provide systematic and iterative education by role, with support at every step in the form of classes and consultation.
  6. Application delivery: Delivery of business specific applications (and associated documentation) that integrate the workflow end to end, from incoming data, to output into downstream applications or dashboards.

Final Thoughts

AI is here. It is real and is already impacting major industries. This is a field where a headstart has massive implications. The best, most proven way to get that headstart, or a least ensure the organization doesn’t fall behind is to put a Center of Excellence in place. A COE provides the structure, the governance, the prioritization capabilities and the measurement mechanism to impart accountability on this, most critical of initiatives.