By Amy Monaghan, Practice Innovations Manager at Perkins Coie LLP

There are numerous options for artificial intelligence (AI) products in the legal technology marketplace and more options are being added on what seems like a weekly basis (2019 has already seen a $1.2 billion explosion in legal tech investment). With the prevalence of options, what do you need to know in order to be an informed customer? The blog post will help you ask questions to identify the problem you are trying to solve and guide you through considerations when evaluating AI solutions.

Identifying the Need

Defining your business problem is the most critical task when determining whether AI is the right solution and, if so, which products meet your needs. Without a clear understanding of your firm’s or legal department’s needs, you cannot make an informed decision and could wind up wasting time and money. Potential business problems include: streamlining large volume transactions, particularly reviewing and analyzing documents; researching and drafting more efficiently; automating routine legal analysis or documents; or analyzing data, including predictive analytics. Talk to your stakeholders to identify what their specific needs are and prioritize accordingly. Once you have done so, gather formal requirements from the ultimate consumers of the solution (this can be the users and the consumers of the output), separating them into two categories: “must have” and “nice to have.” Create a matrix or checklist of these requirements for use when interviewing vendors and evaluating demos of products. Your nice-to-have’s might be on a vendor’s roadmap so including them in your matrix can help inform your product selection.  Be sure to build flexibility into your requirements for future needs—talk to your stakeholders about their business and strategic plans so you have an idea of what solutions they will need in the future.

If your need is to “try out AI,” I urge you to keep digging into that request to uncover the root issue. Similar to the recommendation to not go grocery shopping without a list, do not go AI shopping without a clear need or strategy for how you will use it. That being said, sometimes you do need to test out a technology to know what’s possible before identifying what’s necessary. In this case, your requirements will be more flexible and your exploration will likely inform your ultimate needs.

Going Shopping

With your requirements in hand, it’s time to evaluate your options. You’ve likely done some research already and have an idea of the vendors you want to speak with.  Given the rapid change in the marketplace, do a second pass to see if any newcomers have viable options. Reach out to your peers to ask about their experiences with different solutions. There is a wide range of potential solutions, from open source options to full-service, ready-to-use products.  Be sure you understand what you are evaluating and the flexibility or complexities that come with that approach. (For a wonderful analogy to brownies, see Gwyn McAlpine’s contribution to this ILTACON panel recording at about the 9:30 mark.)

When reviewing the capabilities of a product, ask vendors to tailor their demo to your specific use case. On a panel at the recent Emerging Legal Technology Forum in Toronto, Al Hounsell from Norton Rose Fulbright shared that, when evaluating no-code platforms, his firm gave each vendor the exact same use case and requirements.  As a result, they were able to hold an objective bake-off. Similarly, I participated in an initiative where we gave machine learning contract analysis vendors the same data set and asked them each to train models using the set. We then evaluated the models’ performance on test data. This approach may not be applicable for all scenarios but can be a useful way to objectively evaluate how well as solution meets your requirements.

In addition to planning use cases to direct the demo, educate yourself about the different flavors of AI and which are applicable to your scenarios.  Unsupervised AI largely deals with classification and clustering, whereas supervised AI is used for more targeted tasks like language extraction. Expert systems can use if-then-else logic along with relevancy or inference prioritization, which is ideal for expertise automation or guided interviews. A few vendors are starting to combine machine learning with other forms of AI, like expert systems, to provide multi-tasking solutions, which can be helpful for more complex use cases.

Be sure to evaluate both technical and non-technical considerations related to products and vendors.  Below are some examples of questions to ask vendors to determine if your requirements are met.

Technical Questions

• What type of machine learning is used? Supervised? Unsupervised? Both?
• Does the product come pre-trained with ready to use models? If so, what is the best use of the models? How often are they updated? Who are the trainers? What training data is used? Where sourced? How are the models QA’ed?
• Do you have the ability to self-train models? If yes, do you understand the process for training models and the data and skills that are needed? Depending on the type of models you need to train, you may need very different sizes and types of data sets. Note that most products that offer pre-trained models or pre-configured templates largely rely on publicly available data, which may not be appropriate for your needs. In this case, you will want a product that allows for customization or self-training.
• Does the vendor offer best practices or other guidance on model training?
• Can you collaborate on model training with others outside of your organization? This is a developing request we are seeing in the industry.
• What technical skills are needed to build applications that use logical reasoning for application building or document automation?
• Does the vendor have an open API? This is necessary if you will need to leverage multiple solutions to solve problems.
• Does the product directly integrate with other products? If so, how are product updates handled? Do the vendors coordinate and QA prior to updates?
• Does the vendor offer a user acceptance testing (UAT) environment or other test environment where you can preview new features prior to release?
• For cloud solutions, where does the data reside? Do they offer encryption keys? Who holds the keys?
• What technology resources are needed to assist with implementation?

Company Questions

• Always ask for the roadmap. This will give you an idea of where the company is headed and if it will be a good partnership.
• What is their primary revenue source? Is it their product(s) or professional services? Or a split between the two? If the latter, this could indicate complexities in implementation.
• What is the maturity of the company? Established? Startup? Working with startups might involve greater risk, but the product and company could be a great partnership. For example, I began using Kira Systems (back when they were Diligence Engine!) not too long after they came to market. It’s been a great experience working with them and their product and they have now grown to be a market leader in the machine learning for contract analysis space.
• Do you feel good about the relationship? This will be a partnership so it will be critical to get along and trust the vendor.
• Ask the vendors for references and talk to those references about their experience, including support.

Implementation and Support

• Will the vendor help with implementation and rollout or provide change management guidance specific to their product? Do they have best practices and help resources? If not, preparing your own materials and programs can be very time consuming so plan ahead.
• Does the provider offer a trial period or proof of concept pricing? This is often the best way to determine whether the product is right for your needs.
• If subject matter experts/ultimate consumers of the product were involved in the vetting process, ask them to share their impressions and use cases for your rollout materials.
• Consider asking your early adopters to co-present with you during rollout or provide a testimonial in another form of communication. During our Kira rollout at Perkins Coie, I asked a senior counsel to co-present with me at a firmwide M&A meeting on how he was using the product and provide guidance to his peers. His insight resonated with his colleagues and alleviated their concerns about incorporating AI technologies.
• Will you and your users have regular opportunities to provide feedback to the vendor? What will the format be? How will the vendor use your feedback?

Conclusion

As you can see, there are many considerations when choosing and implementing AI products. Many of the current offerings come with a steep price tag.  By doing your research and knowing your use cases, you can avoid an impulse purchase at the register!

Buying Intelligence: Navigating the AI Supermarket