Name: Ethical and Practical Strategies for AI in eDiscovery
Uploaded: 2026-04-02T18:16:07.204Z
Duration: 1 h 12 s
Description: Ethical and Practical Strategies for AI in eDiscovery

Transcript for "Ethical and Practical Strategies for AI in eDiscovery": Hey, everybody. Alright. Good afternoon. Alright. We'll go I'll give it a couple more seconds here for more people to show up. And then, Cal, I think we're we'll be ready to go. Yeah. Alright. Can't wait. And welcome to everybody who's joining from multiple different time zones wherever you're calling in from. Alright. We have a lot of content here, Cal, that you and I are gonna ask each other a lot of really good questions. So I think it probably makes sense to, let's go ahead and get started. Cal, are you ready? Yeah. Let's do it. Perfect. Well, good afternoon to everybody who is calling in from wherever you are. My name is Daniel Gold, and I am a principal at BDO where I run our ediscovery managed services practice. And I also happen to cohost two different ediscovery podcasts. One of them is the BDO legal tech talk podcast, and the other one is the National Ediscovery Leadership Institute podcast, also called Nelly. By background, I am an attorney, and I've spent to the better part of over twenty years now at the intersection of law and technology and really working with clients, making sure that the technology actually works the way that they need it to. And so today, I'm very excited because this is gonna feel a little bit like a podcast and you all happen to be sitting in in the live audience. There's not gonna be a slide deck. We do have a couple of visuals we're gonna show you. There's not gonna be a product demo. This is just gonna be a real conversation between Cal and myself. We've got some live polling so we can better understand a little bit more about everyone's experience as it relates to AI. And we're also gonna have some space for your questions at the end. So with that being said, I am joined here, as everyone can see, from Cal from Everlaw. We're gonna trade questions back and forth, and the goal is to really pay the outcome we wanna be able to do is to really be able to cut through the noise around generative AI and talk about what's actually defensible and what is actually working. So, Cal, before we dive in, we'll turn over to you to introduce yourself. Yeah. Thanks, Daniel. So hey, everybody. Cali Ewan. I'm on the Evalu AI team here at Everlaw. Our role is to help our clients understand the machine learning and generative AI tools available at Everlaw, how to implement them, and how to really put them into practice for the work that you guys do. Prior to coming to Everlaw, I spent five years as a project attorney at Oracle Analytics using these kinds of tools day in and day out. So I'm bringing a grounded real world experience is maybe the right way to put it. I love it. That's perfect. So, you know, I said that we've got some polls and I wanna be able to do that. I did notice in in the chat before we get started just by way of some housekeeping. There are some questions about whether or not there's any continuing legal education credit or continue professional, credits. There are no credits. This is gonna be we hope you're gonna stay on even though there aren't any CLEs or CPs. This is all about education today, and we hope that you are gonna be able to stay on and benefit from that education. So we've got some help in the in the background here to show some polls here before we jump into the substance. So we I have a first poll question for all of you, which is which best describes your current use of AI in legal work? So are you actively using it? Any discovery matters? Experimenting or running pilots? Are you watching it closely? Not using it yet? Not sure. Look at this. This is very interesting, Cal, watching this come in live. So we'll. give you a few more seconds. Look at this. Does this surprise you, Cal? I I don't think I'm too surprised. I think the biggest trend that I've seen over the past couple of years is if you're looking at these four options and ranking them, you know, one to four just. for organizations like, Used to be a whole lot more force. Right? Used to be a whole lot more people not really sure what these tools are, not wanting to engage until we understood the risks associated with them. Right? Exactly. And then followed by three, then two, then one. And I think that you're seeing that shift. This matches my observational experience where people are really getting into it, getting their hands on these tools now, experimenting with these tools. Some of them have already implemented process. And you still have folks who are maybe hanging back a little bit and seeing where everything settles. So that and and that's really interesting. So, yeah, it's interesting to see that, you know, people are are experimenting or running pilots being very, very close with watching it closely, not yet using. Fascinating. I think that if we were to run this a couple of years ago, you'd see a much, much different result, Cal, like you said, a lot more practical hands on. Let's go ahead, and let's go to that's very interesting. Let's go to question number two. So question number two, if we get that pull up as well, there we go, is where, is AI being used today? So if you are utilizing it, are you using in the discovery platform, legal research and drafting, internal knowledge management, nowhere yet or or maybe it's all areas. So let's see. I love the real time feed here. This is fantastic. So Cal look at you know what's interesting, Cal, is that the majority of votes coming in so far is internal knowledge management. So in other words, an internal LLM, a large language model, perhaps is what we might be seeing here. Yeah. I I think also it's easier to hedge your downside risk experimenting with these tools for the first time where everything's fully internal. Maybe you have another, you know, confirmation or check system. Right? And, you know, there's a level of understanding and comprehension that goes into using some of these tools and some of the other applications. Right? That's true. That's true. And, you know, it's interesting. If I take a look here and I see that only seven votes are on ediscovery platform, which is interesting. I think this the questions that you and I are going to ask each other this afternoon, Cal, I think may perhaps bring about some more confidence levels in being able to utilize it in in any ediscovery platform. Right? And being able to understand that this is interesting. Okay. Let's go ahead and go to the third one. This one, I'm really gonna be interested, Cal, as to what everyone says. And this one is what is the biggest thing that's holding AI adoption back right now? Because we have some questions geared up for one another that's gonna talk about this. Is it defensibility or accuracy? Because that's an obviously a really important one. Is it firmer client policy, training and capability, billing billable hour incentives, or just you're just not convinced that AI is the cat's meow, I suppose. Right? So what do we have? there a yeah. Sure. So interesting that that there is not a single vote for billable hour incentives. That is so interesting. So we're we're definitely gonna talk about that, Cal. Yeah. So. I I maybe one of the in the room. Right? it is an elephant in the room. There's no question. Are are you are you surprised I'm not about the defensibility or accuracy concerns being what appears to be the the majority? So here's here's what I'll say to that. I think that the two highest counts in here, right, defensibility and accuracy slash training capability, Yeah. I think those are going to be inversely correlated at a given organization. The more training and the more folks who are able to run point on being trained, Right. the lower your defensibility and accuracy concerns are going to be because your process is going to account for that and change how QC might operate. Right? Well, I think you're right. So thank you very much. Let's take these polls down. We're gonna come back to we'll we'll talk about these results as we go along, and they're gonna be useful for context for this discussion. So we have five topics for all of you that we're gonna talk about today. And so the first topic that we're gonna talk about is the journey to generative AI. So, Cal, I wanna ask you the question first. I'm very curious about this. We've all been doing this very long time. If you zoom out, k, over the last twenty years of discovery, what actually changed with GenAI and what really didn't? Yeah. It's a great question. So it's important to understand how we got here and what the path was that got us here. Right? So moving from paper to digital, right, starting with just OCR scanning and, you know, we move on to you know, you get email threading, you get TAR one point o, TAR two point o, and eventually generative AI. Right? So there is a path in progression of ever increasing scale of discovery and litigation and the need and the invention coming out of that need to use technology to help facilitate that discovery. Right? And so in one sense, this is the next step in a an established path where we've all as an industry walked down together. The reason why the legal industry was able to quickly pivot and start assessing these tools is we have been using AI and just not calling it AI for, I don't know, a decade. Right? And so stepping back a little bit and making sure we're all talking turkey, you start off with sort of lower implementation of technological tools to be able to just assess, just triage the rate of documents coming in the door. Right? Whether that is just being able to search and keyword search across extracted text or you are threading emails and looking at the most inclusive email chain and instead of looking at the same email chain 17 times. And then eventually getting into the technological basis review that first iteration really of machine learning at in a way that replaces initial determinations or works in conjunction with initial determinations, giving you that zero to one score on a document. And then when you layer the technological thesis review and the analytics, you get right up to the state of practice right right before generative AI hit in 2023. Right? And so now just like you're seeing on the graphic here, you're going from an estimation of relevance in the aggregate layered with a variety of techniques to winnow down the review set to and using only your human judgment and reasoning over that resulting set, right, be it targeted searches, data range restrictions, threading, maybe you're feeling a little frisky. You've got concept clustering in the mix too, right, to really sharing that reasoning with the generative AI, working with the analysis and determination, the who, what, where, when, why that goes along with a yes or no. Yeah. It makes sense. It's almost like a natural technology arc if you think about it. You know, the keyword manual review that that that algorithm started in the nineteen seventies. And then you go to TARN analytics, and that really came out, like you said, you know, around 2010 area ish. And now here, it's sort of as technology evolves, so too does our ability to accelerate outcomes, it sounds like. Yeah. So on that topic, right, from what you're seeing across your client base at BDO, you have this wonderful line of sight on a lot of the market. Does it feel like a true inflection point to you? Is this something entirely different and worlds apart from what we've seen before, or is this another iteration? It is a great question. And so I'll I'll say that this is here to stay, but what we are not seeing is some sort of uniform shift. What do I mean by that? It's fair to say that across the AmWA law firms, the mid market companies, public sector, etcetera, everyone is starting from a different place, Cal. But everyone also has the same goal, and that is what matters, where's the risk. We saw that in some of the surveys just now. And what do I need to know right now? And what is, I think, genuinely different, and this is definitely what we're seeing across clients, is that now we have a different speed of engagement. So when you go back to the keyword search, graphic before. So before, it's very slow. Right? You put in keywords and you expect to get back, you know, just that keyword. If I was looking for the word Alaska in in an entire corpus, I'm only gonna get back those words that have, you know, Alaska. That got a little bit faster. We learned a little bit more. You started talking about conceptual clustering and searching. Also, we're mixing concepts. Right? So now we're getting concepts in. So it got a little bit faster. Now with GenAI, now what's so interesting is is that where clients used to wait days for having the first meaningful insight, now you have plain language question, you have citation backed responses, and you're getting it almost immediately. That changes behavior and importantly, it changes client expectations. And so that's important too because the speed to outcome, the speed to delivery, having more polished answers, having a much faster. Right? It requires a a whole different level of human judgment and analysis, but it also says that, well, if we can get it back cheaper and faster, and I hate to utilize those two words, it's very cliche. But if we are getting back better answers more subsequently or getting to the truth faster, that's really important to know. Because then the client expectations are always gonna be there that, well, wait a second. Why are you doing keyword searching? Why are you billing me for keyword searching, etcetera? Right? And so I think what is also happening is number three. Right? So one, it's not a uniform shift. Number two, speed to outcome is much faster. But three, I think the real shift is what I'm gonna call iterative thinking. K? Not one shot answers. So clients who are very successful in utilizing Gen AI are asking a question. They're evaluating the response. They're challenging it. K? And then they're going again. Right? And so what happens when they do that is that then they are actually learning from it, and they are not just relying upon the answer. They're asking it. They're challenging it. And maybe they're even asking the JIN AI tool to ask it questions. They ask the attorney rather questions. But I think and this is also important. I think that where clients struggle is where they ask once and they stop, is where they copy and they just paste, where they do not apply human judgment, where they are not in the lead, where they are just letting the tool do the work. And here's the thing. And, Cal, I think you obviously see this as being your role in, you know, doing AI at at Everlaw. JN AI is not a magical elixir. It doesn't fix broken processes. In fact, I think it amplifies them. Right? That's right. So if you have a strong discipline, right, it's gonna amplify that. If you do not have a strong discipline, it will amplify that as well. That's right. It is it is 100% a multiplier and amplifier. Right? So if you add it as another tool in your toolbox to think critically and approach problems with a a certain level of rigor, right, then it's gonna be able to multiply in what you can do. If you are instead, you know, maybe just asking those questions and not asking questions of the answers, right, then that's gonna multiply the impact of that decision point. That's right. That's a 100% right. Alright. Why don't we go on and move on to the next topic? Yeah. Sure. So, you know, talking now about Gen AI as a thought partner. Right? We sort of led into this a moment ago. You said that Gen AI fails when people treat it, you know, like Google. Right? So what does treating it like a thought partner actually look like in real life? Okay. Alright. Well, so we're gonna have some fun with this, Cal. Okay? And and by the way, when I say we're gonna have some fun with this, I've seen some really great comments. I'm partially distracted by some of the really good comments. We're gonna really talk through this. So okay. So when people are utilizing Gen AI like it's Google, it's what I call failure mode. Why? Because when people bring Google muscle memory to a tool that requires a different posture, you're not setting yourself up or your clients up for success. That whole, like, one question, one answer, either blind trust or full dismissal, both of those are actually mistakes. And and so what is really required in order to be successful, as I started talking about earlier, is this mindset mindset shift. So with Google, if you were going or any other search engine that you use, you basically go to Google and you say, I know what I'm looking for. Help me find a Google. K? Hard stop. That's what that's what you do. Gen AI does something different. Gen AI should do something like this. I'm still thinking. Help me reason through this. Now that flip actually sounds really, really subtle, but it actually changes everything in which you are how you operate in your operating model. And, you know, one of the things that I've done, Cal, is I created a methodology that I call human as the creno. Bear with me, and I'll explain this for you. So I think a lot of you who are listening right now have heard of that expression, human in the loop. Now, look, I am not a fan of that expression. Why? When you say human in the loop, it suggests that you as the attorney, as the legal professional, are not actually leading the process. You are not leading the tool. And so being a thought leader, not a thought partner, is very important. So what is this human as the krino thing that I've I've created? So krino is actually a Greek word and that Greek word means to judge, discern, and separate the valid from the invalid. That's what we need to do with GenAI Cal. Right? We really need to be able to separate the outputs that are valid from the from the outputs that are invalid. So Crino, that word, I turned it into an acronym. And so, basically, the k in the Crino is what did you have to know what the AI did. Now I wanna be clear about this. I'm not suggesting that everyone here that's listening to this this webinar, know what the black box is. Right? That's not what we're saying. You don't have to know where the wires are are are are crossed. But what you have to know is what you have to know how the AI works and what it did. And then the r in Crino is you have to review the output as a judgment call. And the I means you have to interrogate the methodology. Right? So the iterations that we're talking about before. And then the n is to normalize. So normalize against your subject matter expertise. You are the subject matter expert. It's not the AI tool. And then finally, the most important is the o in the acronym. O means to own. You have to own that determination with your professional license. You are an officer of the court and as such, you have to be able to speak with confidence. You have to own it. If you are going to leverage AI, Gen AI in particular, to help you with your answers, whether it is legal research or it's e discovery or whatever it might be, you have to own that determination. That's how you operationalize professional judgment. And and, Cal, it's worth noting this too. This is really interesting. So I've seen a couple of states now start amending their interpretation of rule 1.1 of the the model rules of professional responsibility and common eight. So we as most of everyone on here knows is that in 2012, it was amended to say that in order for an attorney to be competent, they have to understand the benefits and risks to, technology. But now some, states are actually amending that again to say understanding AI. So it's. not just using it, it's owning it. Yeah. And some something to consider as well is whether or not it's explicitly spelled out in the state's opinions, right, That's right. or not. There is a general ethical obligation to supervise as an attorney, to exercise that judgment as an attorney, and to understand the nature of what you are working with. Right? And so rephrasing and reframing some of what you were just saying from my perspective, what I'm hearing is that you cannot ethically abrogate your judgment. or hand that off to a machine that does not have ethics. Right? So you you need to be that person in the loop, which I know you love that expression. Right? But, like, you you have to I think pairing a couple of thoughts, you have to keep your critical thinking and judgment hat on, right, and exercise. some of those core skills that are quintessential to an attorney. Right? That critical analysis, that judgment and pairing and understanding signal and noise, being that person to understand the tool in your toolbox as opposed to handing off a task. Right? I mean, think you nailed it. So but but stay with the idea for a second, Cal. So how are you seeing this show up specifically inside of Everlaw workflows? Yeah. So you you're talking before about having a partner, right, and. using this as a thought partner. One of the things I think is an important distinction here is you're not doing that to be nice to the tool. Right? Like, that's not why we are acting as a thought partner with these tools. It's because when you instead treat it as an instruction taker or an order taker or a do this, right, you are necessarily limiting the input you're putting into a machine for the output. You are presuming you know what you need it to do. You are presuming perhaps that it has the context of information in your head you have not put into the prompt. And so by asking for a collaborative thought partner perhaps that can follow-up questions, right, you are able to provide greater input to in ensure a greater output. And so what that might look like specifically in Everlaw, we have this great retrieval augmented generation tool, Rank tool, called DeepDive. And what it does among other things is ensures that the tool is only retrieving information that is in your documents and then track the text of your documents and cite specifically to it. Right? So if you ask is why is the sky blue? And there's nothing in your documents about the sky being blue, it will not rely on its training and answer that question. I'll say there's nothing to answer that question. Right? And so that's important because you want to understand the nature of the tool that you are working with and tailor your framework and your approach appropriately. Right? You were referencing before. From the end user, absolutely, you're saying, I know what I want. Go find it for me. Right? But what Google is at its core is an algorithmic ranking tool. Right? That's why there's search engine optimization and everything else. So under the hood, there's a level of work that goes on to return the results in that first page. Right? In the same way, that is step one of several for our retrieval augmented generation tool like DeepDive. Right? And so that allows you to ask semantic questions, the who, what, where, when, and why, and purpose questions as opposed to find me the emails from January 6 that hit on term. Right. Right? And so by by operating in that different way, that semantic way instead of lexical, right, to use some of the technical terms, and by framing your approach in that manner, you are ensuring that you are utilizing the hammer designed as a hammer, the scalpel designed as a scalpel instead of the hammer as a scalpel. Right? Like, you're using the tool in the way it is intended to be used and creatively abstracting away some of the hard technical knowledge necessary to be able to get to those results. Right? By working with this tool, if you understand the meaning behind the ask, then you are able to, you know, avoid the need for a data science background and perhaps be a member of the case team working in the documents. And and and I love what you said just now there. And and, you know, we we're gonna hearken back to a lot of the comments because there is a lot of comments that are being added right now to our conversation. But what you just said was common here that that, it's essential to avoid treating the AI like a human at all. It is a machine system tool. I think that's really important. And so when you're utilizing, like, were talking about with deep dive just now and and you're getting this, you almost get lured into this idea that, like, you are talking to an like, a junior associate. Right? Because if you're treating the AI as a thought partner, right, then you're almost as if you're being lured into think, you're having this two way conversation with this tool. And there's other, you know, commercial grade, consumer based tools, chat, you know, different LLMs, etcetera, that all do this and you have this conversation. You get there's this lure. It's like, well, maybe I'm not really talking to a machine after all. But there's this question about defensibility that comes up. And and and so let's I wanna talk about this and and this is our third topic here for everyone that's tracking along. It's really talking about, like, where defensibility lives, and this is so important. So you've said, Cal, that defensibility lives in the hybrid. So talk to us about what does that mean in the hybrid, and what does that look like more importantly on a real matter? Yeah. So I'll speak in the aggregate and then narrow in on the concrete. Right? Because I I think it's true whether. or not you're talking about specific eDiscovery workflow or just, you know, process in general approaching these tools. In the same way that you can't aggregate your ethical duty to exercise judgment, right, there are certain things these tools are good at and certain things they are not good at, and you have to work using the tool as designed and fitting it into a process that that has a place for it. Right? So what could that look like? The first thing we've touched on this once or twice, is the nature of skeptical judgment and iteration on your prompts. Right? You are seeking knowledge behind this. question, and you are constantly assessing whether or not the tool is functioning as intended to bring you back that information and then working from that greater place of knowledge. Right? And so I give you that framework so we can talk about the concrete. Let's apply it to the discovery workflow. This is something that I did in past practice in Everlaw. Right? So in the same way that you might have a human review or a continuous active learning review where you have initial work that is done that is assessed, there's feedback given, and you have a an improvement loop that happens. Right? Whether that's a human review team getting QC feedback, whether or not that is samples validation sampling overwork or continuous active learning constantly improving coverage of a model by increasing the number of documents that have been reviewed and assessed. Mhmm. What that looks like with Journey to AI is crafting prompts. Think of it as version two point o of a review team memo. Right? And then from there, assessing target documents in a sample small sample to understand whether or not what you have told the machine to do is what it has exercised upon, making sure that all that context in your head is there, making sure that, you know, understanding the nature of your documents perhaps in those initial samples that comes out that there's a Jones litigation that has nothing to do with this case, and you can iterate on your prompt and exclude those documents from assessment. Right? And, So, what let me let me pause you for one second. For everyone that's that's that's listening right now, there this idea of giving it a good prompt, you and I have had some, you know, off stage, so to say, your backstage conversations about about good prompts. I think that there may be some confusion about, or or, you know, an opportunity to grow when understanding what a good prompt looks like, feels like, sounds like, etcetera, in order for the tools to work effectively. In your experience, what what does that look like, a good prompt? Yeah. That that that's a great question. So you need to be able to concretely express what it is you're trying to accomplish. K. Right? I'm looking for privilege. K. Right? I am I am looking for documents related to RFP six. Right? Then you have to be able to define what that means. So, you know, RFP six is perhaps the literal text of RFP six, or maybe it's an amalgam of five different RFPs and you're looking for one issue. You have to be able to describe what constitutes the thing you're looking for. Right? So from there, you need to be able to describe your desired and intended output for judgment. Any document that is directly related to r f e six is a hard yes, right, versus documents which technically fall under the category but are not central to the case are a soft yes, you know, so on and so forth, or you're describing the definition of attorney client privilege, attorney work product. And then from there, you have to make sure it understands the concrete yes value. Right? Like, yeah, that affirmative value, not just assessing the document in front of it, but what turns that from a no into a yes. So in some sense, what you are suggesting is is that just the way we'd wanna give very clear instructions to a junior associate, let's say, you're a partner, you. wanna give those very same explicit instructions to a GenAI tool. Is that correct? That's that's exactly right. And, you know, in the same way that your ethical duty to supervise over that junior associate requires that you review that work product, right, so too do you have to take that critical eye to the output of a generative AI tool. That's right. And and and for everybody else, and so there's actually two different rules that we could think of it off top of my our head is one of them is rule 5.3 b of the rules of professional responsibility. And that's what you're talking about just now, Cal. Right? It's being able to essentially oversee a non attorney's work. And as the comments in our conversation clearly alluded to, the JNDI tool is not a human and is definitely not an attorney. So we wanna make sure that's right. And the other one is also obviously rule 11. I was in law school, that was the number one thing I was always scared about, when I was practicing was was a rule 11 violation. You, an officer of the court, you are attesting to the fact that what you are submitting is true, invalid, and you're utilizing your professional judgment. And I think, Cal, you just honed in on that very exact point just now. Yep. Yeah. And so, you know, something to understand in our framework of where this tool fits into the existing practice. Right? Previously, you would take samples for a predictive coding model or a TAR one point o model to under to help the tool train. You have to give it enough examples of what is and is not relevant so it can start to predict out the ranking score of documents. Right? Mhmm. And then perhaps you have to find poorly covered documents and sample on those to make sure that the model has a rounded understanding of what you're looking for. That's not what the initial samples are for in generative AI. Right? You it trains off of zero documents. It goes entirely based off of your prompts. And what the purpose of those samples is is to make sure you understand that what you have tools the tool is what you have intended to tell the tool and that it is interpreting and running that analysis in a way that's consistent with your understanding of the case. So it naturally bleeds into that next part of the circle, that human judgment and review. Right? And then you iterate and refine upon that. You're working for greater knowledge, and it becomes a virtuous cycle. Right? I like that. Alright. So, you know, from your perspective, Mhmm. right, what does good look like when you have to stand behind the work? That is a loaded question, Cal. And and I think it I think it it you could bring it down or break it down rather to one sentence definition. Good means if a judge or regulator asks why you did this and how you know it's going to be reliable and you could answer that clearly, that's good. K? That is good. And it goes back to what you were talking about before. Right? And both of us, iteration, refinement, human judgment, you understand it. Right? You know it. You've reviewed it. You've interrogated it. You normalize it, and you own it, and you have confidence. So good means if someone asks, you can clearly answer it. Now I do wanna say something else. I think that just the way we have Privlogs. Right? So why do we have Privlogs? Right? We have Privlogs because we wanna make sure that whatever is attorney work client privilege, we're not inadvertently producing it. Right? I think the same can be said for the utilization of GenAI. So what I'm suggesting here is that we have process documentation when we're utilizing GenAI. What does that mean? We can create basically an audit log for ourselves when we utilize GenAI about what was asked, what was used, what was validated, and how are the decisions made. I mean, think about it. It's almost common sense that what we wanna be able to do is have prompt logs. What are the exports? What are the sampling results? It's our it's our rationale, right, as attorneys, right, as legal professionals as to what we decided. And maybe it's even Everlost story builder narratives. Right? Whatever it might be, what we are doing is is we are creating receipts for our human judgment. So if we were asked, well, I would. We can have an offline conversation about the these, logs being attorney work client privilege as well. But what we are doing is for our own benefit, for our own defensibility, right, is we are putting our human judgment in the lead. Right? We, again, are being able to say Gen AI surfaces up to us amazing possibilities, Cal. TAR, technology assistant review, as you were just talking about, prioritizes documents, but the judgment never leaves the human. And now here's it's so timely that we're having this right now. It's supposed be April 2. In February, the Southern District of New York just ruled out a decision, US v Hebner. Won't go into the case, in-depth, but here's basically what I'd love everyone to take a look at in US v Hebner. The courts are not evaluating tools. They're evaluating human decisions, and that court made it very explicit. The court wanted to know what the lawyer's reasoning was, what the lawyer's safeguards were, and what the lawyer's validation was with respect to the tools, not the sophistication of the software. The court doesn't care. The court wants to know if you utilize this tool, tell me about it. Own it. Right? Going back to human as the. And. so good, it summed up as, can you explain this to a judge who doesn't care about your tools? I think that's what it is, Cal. I I think that's great. I I love the the emphasis on process as well because at the end of the day, no matter what your approach is in discovery, the process ensures good work product and consistency, but it also acts as that audit log. Right? It is your trail of effort and outcome and judgment. Right? And so perhaps you're talking about illusion sampling in a predictive coding case or indeed in, you know, using generative AI to rank those documents for that second pass human review. Maybe it is crafting prompts with the intent and understanding that they may be reviewed. Right? And then, you know, adding the initial prompts that led to documents, say, in deep dive to your story builder draft. It could also be classifying which prompts you would not want to reveal because they tend to reveal a thought process account. So all of this is in flux and is being decided in various jurisdictions. Right? That's right. no matter how things land as this gets worked out by precedent, having that forethought, that planning, and that eye on repeatable process that is defensible, Right? Whatever defensible may become as is decided in precedent. Yeah. And and, Cal, what do you think of this idea of creating this audit log for for prompting? I have mixed thoughts. I come from a defense background, and I prefer to give as little as possible. Right. But but but, at least have it as a back pocket, so. that, you know right. Matter what, at the end of the day, motion practice fight, what have you, you need whether it's to the internal partner on the case or to the judge or to the opposing party, you have to be able to say, here is how we tackle the problem. Here is how I can stand behind that work. Here is how I maintained my ethical obligations and duties, right, to practice. And I I think you just nailed it. I mean, I think that no matter what, being able to have something there to say that I have my professional rules, I I I've just doubled down on what you just said. I think that is in fact what no matter what happens as far as an output is concerned is that there's the human in the lead right there. Right. So let let's move on to our next topic. Right? Because I think it's a a pretty good segue here. We're talking about culture, the billable hour, your combi framework. Right? Why is AI adoption fundamentally a behavior problem and not a tools problem? Yeah. So this is really this is gonna be a lot of fun here. And I could and I see a lot of the comments here as well, and I think that this is gonna play into my answer is gonna play into some of the comments here. So from my perspective, AI adoption is not failing because the tools don't work. The tools work. And, frankly, a lot of the software that's coming out, they're all based upon the same large language models that are all out now. They're very popular large language models. The reason why Gen AI adoption may be failing, and and I wanna harken back to the polls that we did in the very beginning, which I thought was very interesting, and it gave us a lot of context about this conversation right now. Behavior only changes when three conditions are present at the same time. And this is what you mentioned before, this this c o m dash b framework. Now this is not my framework. This is an actual, you know, documented, behavioral science framework. But so comb dash b, what or comb, what does this stand for? Behavior changes when you have the capability, you have the opportunity, and you have the motivation. When those three things are present, then change management can occur. And we're talking about GenAI, but we're also not talking about GenAI. It's really talking about any sort of change management and in particular, really, you know, around technology. The idea is very simple. Right? When all of these are in place, you are able to get a total return on investment. So let's walk through this. So what does this really mean? So the c, again, stands for capability. So this really what this means, Cal, is that it's more than just seeing a demo. Because if you've got legal practitioners and legal professionals saying to one another, hey. Have you seen this demo? But that doesn't really help. Right? Yeah. I saw through the demo. That's not capability. What capability really means is do people actually know how to use these tools in a way that's defensible, like we just talked about, repeatable, and professional, like you were talking about a moment ago, Cal. So across our client base of BDO, most lawyers have seen some sort of Gen AI tool before. But very few have been trained upon what we were just talking about a moment ago. How do we ask really good questions? Look. As attorneys, like, that's what we do. Right? We ask really good questions. So how do we ask good questions to the Gen AI tool correctly? How do we validate those outputs, something you were just honing in on? And how do we document decisions? K? And if we don't have that in place, what happens is is that a Gen AI tool feels risky as opposed to empowering. So that's the c capability. So let's move on. Let's talk about the o, which is opportunity. So now we're really talking about culture, Cal. So the environment, the culture, whether it's a law firm or corporate legal department, must be able to allow the use. K? And this is very underrated. So does the environment, right, where I work make it easy and safe to use AI? So in other words, you have approved tools. Are there blessed use cases? Is there clear guidance on when and how AI can or cannot be used? Because if people have to guess whether something is allowed, the default is just gonna be, nope. Not gonna use it. Right? And and and look. That's that's not resistance. Like, that's actual human behavior. That's rational human behavior. If there is not a culture set up by which it feels safe and easy to use and it's documented use cases and policies, etcetera, then you can have capability, but you don't have opportunity. And, therefore, you're not going to have a behavioral change. So alright. So there's capability. It's opportunity. Let's talk about the m motivation. This is this is what was really interesting in the very beginning. So we asked this question in beginning with the polls, Cal, and nobody said that they're not using it because of the billable hour. But I I do wanna talk about this, and this is gonna be a fun area, Cal, I think, for us to talk about. So m stands for motivation and I really do think that that is where the billable hour tension lives. And I think it's also the hardest one. So motivation asks the following question. Do the incentives actually reward the behavior you say you want? Do the incentives actually reward the behavior you say you want? If what does that really mean in practice? Okay. If I'm gonna use Everlaw, right, as an example in my ediscovery, and I'm gonna cut my doc review time down by 40%, and you say to me, Cal, hey, Daniel. Is it gonna be great? I'm gonna cut your time down by 40%. But, Cal, what you just said to me was and you said you worked at a defense firm. Right? So what you just said to me was my utilization's down now. You know, I just lost all these hours. That's what you just told me. So the firm bills by the hour, every firm does, and you just told me that I just lost all these hours because I use this great tool, well, then my motivation is actually going to be offset. Why? Because people learn very quickly what behavior the firm actually rewards, And that's not resistance either. That's also, you know, human rational behavior. But I'm gonna challenge that. Here's my challenge. K? And you you tell me, Cal, what you think. My challenge to everybody on this is this. If you cut back 40% of the time it takes to do a document review, that doesn't mean that there is any shortage of work whatsoever. In fact, I'm gonna counter that by saying, if you get to the truth faster, if you get the information that you need in a timely fashion, the output is better than it was before, and frankly, it's more robust than it was before, and speed to outcome and speed to delivery is much faster than before, then you have higher levels of client satisfaction. And if you have higher levels of client satisfaction, the clients are going to provide you with more work because they're gonna say, gosh, that cow. That cow knows what what he's doing, and I'm gonna give him more work. And then there's gonna be more work on top of that, and then the clients are gonna refer him over to other clients as well because Cal does such a great job because he's utilizing GenAI. Right? So for me, when we put all that together, right, if we if we if we reframe and look at motivation through a different lens, which is motivation being the highest levels of client satisfaction possible using the best technology possible, when a if we do all that together, then we get behavioral change. So, look, when AI adoption stalls, I think that this comb framework gives us a diagnostic. Is it a capability problem? Is there an opportunity problem? Is there a motivation problem? Because, look, in my experience, Cal, most of the time, you've got two of the three that are going to be issues. And and, Cal, I think that is where your experience is so valuable because you've seen what happens firsthand when those three elements actually line up. Correct? Yeah. That that's right. So, I mean, I I lived through this at Affirm, and I would argue that I would not be here talking to you today if it was not for the approach that Org took to these tools. So, you know, zooming out, zooming in, trying to do it a little bit quickly, I think. You look at that that comm framework. Right? That also ties back to our ethical obligations I was talking about at the top. Right? You're looking at the opportunity. The opportunity needs to be created by being able to rely on your CSO and your IT security folks who are able to vet and validate and create their own process and framework for what might be a blessed tool. Right? And then internally, you need to have messaging from the top down. You need to have a culture of innovation, a culture of experimentation where the time spent to figure out whether or not there is value there is not time burned. Right? Because everything comes down to the time framework in the existing law firm paradigm. Right? Having a leaning in culture when it comes to new things has a knock on effect globally throughout the organization. Right? And I am seeing some comments that match some of that to my experience. And so, you know, Orif had time for generative AI experimentation. There was creditable time that you could put on the books to figure out whether or not this tool could be useful to the work that you did day to day, let alone the department or the wider firm. Right? There was opportunity to provide insight and thoughts regarding different tools or regarding what did work or didn't work. We had conversations going on about successful use cases in different kinds of practice. Right? And it wasn't siloed to that practice. And so by having that opportunity and having that culture, it allowed me to redefine what was the motivation because my motivation, being brutally honest, was being able to take lots of fast moving matters and handle them more efficiently in real time. Right? So it was my personal experience that there was no lack of client need to fill my billable time. So? interesting. And so. I I had the the opposite impact where I needed to utilize these tools. The necessity was the mother of invention there to be able to keep up with the pace of things. And, you know, everybody's practice is different. Some folks are working fewer larger cases over a longer amount of time. Mine were fast, low to the ground cases that overlapped on those deadlines. So it was critically important for me to save time by reducing the number of human teams I needed to train, by reducing the amount of QC input time I had on a per case basis and being able to work it in the aggregate. Right? These tools enabled me to do that and enabled me to do more precise work for my clients. And so the reward for pie was more pie, and I got more cases that needed that kind of touch. So I love would. say what I would say is, well, finally, is zeroing down to the individual. Right? Because, like, we're talking about my personal experience. Being able to define not only the process you have for these tools, but how you interact with your clients in the firm and wider practice, that level of purposeful and intentional approach, right, how you balance your higher quality of life that you can have using tools like these. That is so good. And and, you know, I'm gonna double down on something you just said also by quoting one of the comments here, which I think is great, that the tool adoption managed by good leadership makes all the difference. And and I just man, that that couldn't be more accurate. Right? That could not be more accurate. I 100% agree. Look. For those that have seen some of the things I do online, leadership is everything. Right? And you have to have good leadership. And leadership is defined in part by being able to do what you said, Cal, being able to have the freedom of experimentation and be able to, you know, understand what's gonna work and what's not gonna work. And so that that's a great segue actually to our very last topic here, topic number five, and what value looks like next. So what a great segue here. Here's the thing that I'm so curious about. If everybody has the exact same, you know, tool that they can get the exact same answers from, knowledge is no longer the moat, Cal. Right? It's not. Right? Everyone has the same knowledge available to them. So what is the moat now? Yeah. Great question. So historically, there have been two moats for the legal industry. Right? One was knowledge. The terms of art and access to information for case law that required you to be in this profession to be able to meaningfully have access to that information. Right? That's gone, I would argue. However, the real true core of what it is to think and frame and be an attorney, right, that mindset, that is still core to us and cannot be aggregated to technology ethically. Right? And so what am I talking about here? I'm talking about being able to parse signal from noise with all this case law. I'm talking about being able to understand what is missing in a fact pattern to render and exercise that judgment. That is something that you cannot easily replace, and that's something you cannot ethically replace. Right? We talk about human in the loop, and we talk about the different, you know, approaches to what that looks like. But at its core, you as the attorney have multiple ethical obligations and duties that require you to render that judgment and the value you provide to the clients is in your insight and in your ability to understand and pare away the information to what truly matters. Yeah. I love that. I I hope that people will actually rewind to this moment where you said all that because I think that is that is so important. And I think that what a great highlight to to have here. And that is really what value looks like next. Yeah. I think so. And so, you know, we're talking about, like, how does we go from big and we go to more granular in this conversation. Right? So, like, what does that actually look like? I I've been seeing conversations in our section of the industry lately about, you know, the forward deployed lawyer, right, or the forward sort of like a like an attorney version of a forward deployed engineer. I've also seen forward deployed culture agent, which I thought is really interesting. And because that speaks to the the calm framework that we're just discussing. Right? But the fact of the matter is having someone who can marry that ability to have a refined analysis and parse signal and noise to the specific tailored needs of the clients. Right? Whether you are embedding with specific clients or whether you are just understanding the target need and tailoring that knowledge and skill set and tools to that approach, right, as opposed to a one size fits all approach. That, I think, is where some of those are going in the eDiscovery industry. It's certainly where my team is at with the Value AI team, where we are consulting with clients across different customer groups and different target needs for specific implementations. Right? And so that that sort of marrying of legal acumen and consultative knowledge, I think, is where we go as a next step. I love that. The Ford deployed lawyer. K. So, you know, where do you see BDO fitting into that future assuming it's real? Well, I I think that that is a good question, and I think that the the value is really going from knowing to judging. So look. The the the tool is no longer the hard part. Most firms can buy software. We've talked about this. Right? And we talked about the comb framework. But it's it's turning it into outcomes that clients actually care about. That, I think, is gonna be where the most of the value is actually lost. So BDO sits really at this intersection as cliche as it sounds of technology process and people. We're not replacing platforms like Everlaw. We're helping clients operationalize them defensively with human in the lead. Right? So when you're talking about this four deployed professional, right, so legal judgment, technical fluency, change leadership. It's all in one role. And and you're right. I really do feel like that's where the profession is going, and that's where we're going with it. So, look, Everlaw is building the engine. Right? We at BDO, we're just helping clients drive that car responsibly. And I think that that that's how this actually is supposed to work. Everlaw builds incredibly powerful tools, but tools alone don't change culture. They don't change incentives. They don't change habits. So BDO is sitting alongside clients as a partner to help design the workflows, establish the governance, measure success over time. What are the objectives and key results? What are the key performance indicators? How do we make sure there's compliance around them? And how do we adapt as teams and matters evolve? And I think that this is where you start to see the emergence of of that role like you're talking about a forward deployed lawyer. You're you're wearing many hats because you can wear many hats now. And so people who combine, I think, legal judgment and technical fluency and change leadership, I think that's really where we are going. And so when we talk about value in the future, I don't think it's about who has the best model at all, actually. Think it's really about how do we work together as partners to turn insights into action. How do we turn action into outcomes, and how do we turn outcomes into trust? And I think that's where BDO and Everlong clients work really well together and swim together with almost like a Venn diagram. Right? There is that sweet spot, and I think that's where BDO operates in. And I think that's why the partnership between us, the platforms and consultants, I think that's why it it matters more so than ever. Again, like, you guys build the engines. We just help the client drive it responsibly. And I think that's, that's how it's supposed to work in the real world, Cal. Right? Like, I think that's that's really how you define the value. Yeah. I love it. So I wanna reserve the last few minutes to. questions you might have. I know we've got a couple here ready to go. Maybe the right way to do this is I will shoot out the question, and then you give your initial thoughts, and I'll follow-up. Yep. Sounds good. Go go. Alright. So what we're speaking to, this is from Marissa, is a gap that requires cognitive and psychological readiness. Right? And so for interaction with Gen AI. And and I'd like to get your thoughts on that. Yeah. So this this gets very deep. Right? There's a you know, when you're talking about preparedness for metacognition, right, which can be trained, I agree with that. I think that we all tend to operate at this pace of going a thousand miles per hour on the locomotive and and it's hard to stop it. Right? But we have to. We have to slow down. I often like to say that you have to slow down in order to go faster and that requires critical thinking skills. And, if anyone's looking for a really great book to read about this, it's called the AI driven leader by Jeff Woods. And he talks actually about a lot about the the critical thinking that is involved to be able to utilize Gen AI correctly. It is a tool like anything else is a tool. Right? And it is not a human. But in order to get the right output, in order to get the right value, right, it actually does go beyond critical thinking. It does go beyond that, and you have to be able to think about how to be able to utilize the tool correctly in order to be able to get the right outputs. And as sophisticated and incredible as the tool could be, at the end of the day, Cal, it's still garbage in, garbage out. If you're not asking the right inputs, you're gonna get the wrong outputs. Yeah. No. I think that's exactly right. And, you know, this is this is that that nexus point, right, between actually being the trained resource on the ground in a firm or a corporation or wherever it might be. Right? And then having the the framework or the scaffolding to work with that. I love to comment a little bit ago about how that cultural tone being set from the top down needs. to be enabled by that scaffolding or framework. I to my mind, that is all part of the putting the theory into practice when it comes to leadership. Right? And so your your culture exists insofar as it exists, and it's great to say it and speak it out into the world. It's another thing to actually enact it and engage it. Right? And that that. is the of cultural change. You know, it it's interesting. And and and I'll just one little fair comment. So culture cannot be what's on the back of a business card. Culture has to be practiced. Culture has to be, you know, repeated, and it has to be shared amongst everyone from the CEO to a junior associate. Right? It has to be permeating from across everyone. Everyone has to speak the same culture language. And a part of that culture language is embracing change, choosing accountability, and empowerment through knowledge to be able to know that, look, this is a tool that's going to potentially have the impact to be able to not just change how we operate, but bring more value out into the world. But that has to start with leadership, Cal. Yeah. I agree. So alright. So I think we we are just about here out of time. I I knew we had run up against the hour. Cal, I I really appreciated your your candor and all of your comments here. And for everybody that was here, we had hundreds that attended here today. Really, thank you so much. We're Cal and are very, very grateful for all of you who spent time with us today. We appreciate the engagement, obviously, through all of the the chat. That was fantastic. Let's keep this conversation ongoing, and let's keep having with the community. Cal and I are available offline as well, and feel free to reach out to us. But thank you very much for joining us, and we hope to see all of you again very soon. Take care, everybody. Thanks, all.