Name: [Ethics CLE] Ethical Obligations for Legal AI Workflows
Uploaded: 2026-09-29T18:19:24.415Z
Duration: 1 h 5 min 59 s
Description: [Ethics CLE] Ethical Obligations for Legal AI Workflows

Transcript for "[Ethics CLE] Ethical Obligations for Legal AI Workflows": Hey, everybody. Can can y'all hear me? Alright. I'm getting thumbs up. That's always a good sign. Thanks for bearing with us. We're having some slight technical difficulties. Hey, also. We're gonna answer questions for you on the q and a tab. If you have them, just go ahead and ask there, and we will get to them as able one at a time. If you'd like a copy of this slide deck, it is in the docs tab. It's gonna be particularly handy towards the end of this session when we start showing some of these tools live. Yeah. A little bit of housekeeping here. CLE credit eligibility will be determined by your total login time during today's session. Professor professional associations and license numbers were collected when you registered. And if you should receive your CLE certificate via email within thirty days from the date of the program if credit is approved for your state. There's a list of those states below for the one hour of CLE credit. So y'all y'all know me. Y'all know me already, Cal Yeaman, member of the Value AI team and four deployed Everlaw teams here at Everlaw. Come from a practice background in the m one fifty. Lots of practical use case experience. You're gonna be joined here today as well by Leo Park, who's one of our genius product leads, and Erin Patton, one of our solutions architects here at Everlaw. They put together have a vast amount of, knowledge, and I'm looking forward to learning from them as well, which is telling you guys about the recent developments and ethics. Alright. So today, we're gonna be hitting the rulings, opinions, and ethical obligations as they intersect regenerative AI. Some of this is gonna be ADAA comment a and, you know, where we're at with that, recent state decisions. Also, we're gonna be unpacking cases like the Heffner decision and some others. Then we're gonna pivot to how retrieval augmented generation actually works. Leo is gonna run us through what those tools are at a fundamental level, how they operate so that we get a better understanding and grounding of what happens when you use a tool like Deep Dive. And then speaking of, Aaron Patton's gonna come in towards the end here and, show us live how to use some of these tools within that ethical framework we're gonna start off with. So jumping right in, we're gonna start with a poll. We like doing this at the top. It helps us get a read for where the audience is at, what you guys are actually working with and experiencing, and we can sort of talk to that. Right? So, if you could jump in the polls and let us know, these change rapidly over a period of weeks or months, and it's always interesting to see. So I'm I'm not surprised to see legal research and drafting, particularly with, you know, some of the tools outside of Everlaw. And then, ediscovery platform makes a ton of sense, especially for this audience here today. I I I I have noticed that over the past year, year and a half, I'd be curious to hear your thoughts too, Aaron. The number of people going with nowhere yet and shrinking on down, that has been an accelerating trend for me. Same. Same here. We are every conversation I'm having, somebody's using AI somewhere. Even if it's in their personal life, it's it's starting to bleed into work, and and, also, it's starting to create pressure on the work environment to bring new tools in so that so that the the teams can actually kind of be at the same level as the rest of the industry. Yeah. So I think that's, that spread seems pretty consistent the last time I ran a poll like this, right, where we see the most use in legal drafting, really leaning on those cases, those those strengths for AI tools and then following up with the discovery. I wonder how much of that is, courses for courses and how much of that is, familiarity and exposures on those tools. don't know. We're also kind of all over the map here for our next question. What's the biggest thing holding AI adoption back for you? I can certainly understand the security and privacy concerns, making sure that none of the data none of the prompting going in, none of the output coming out is being used for training purposes. It's almost tied with defensibility. Yeah. I what the old adage is garbage in, garbage out. Right? And so I it's been my experience at least with these models that they're they're probabilistic. Right? You can run the same information six different times against six slightly different results. But the more context you put around, the more structure you put into your prompts, the better source of information you provide, the better your output's gonna be on a more reliable basis. Oh, I like that everybody read all the way to the bottom to get to the security and privacy concerns. Yeah. Alright. We're almost done here, guys. But this is really helpful for us to be able to cite in a little bit. How what does using AI look like in your organization? Right? Like, think about that as you're answering this question. Some folks, they they are the resource, and some folks have an entire organization built around vetting tools, testing tools, validating model to model. It's a it's a really widespread Wow. The the. the legal gains only is really low. Yeah. I I'm I'm noting, for me, the tie almost between support for both legal and tech or no support at all. Right? That's that's the spectrum. Yeah. Yeah. I I'd be really curious to understand who, generally speaking, like, position in the industry folks are with no legal tech support or the folks who have the full gamut. I I I think that's an interesting split. Alright. And so, you know, moving on from those polls, and we'll we'll keep that information in mind as we move through. But, our latest innovation survey showed that folks are using generative AI. 42 49% are using it currently, and 42% are planning to use it in the future. It's a massive shift. Right? Over the past, what, even in the past few months, there's been a massive shift with recent development of models. Things are moving really quickly. And when things move quickly, difficult decisions get made. Right? Makes the ethics that are we're grounding all this in more valuable than ever and more foundational to how you approach everyday working life, not just your professional responsibilities. So if we can move on to the next slide. Let me just click here. So diving right in. I think a lot of this starts with the concepts behind ABA model rule 1.1 comment eight. Right? Maintaining not just being aware of the tools, but having a knowledge and a competence regarding the tools, the state, the development. Right? How what does that boil down to in real life, though? I'm not gonna read off the slide to you. If you are not the person who is the expert resource, who understands what tools you were using at your corporation or your firm, you need to have someone who does. That could be many people. Right? It could be, you know, a combination of your IT security folks, a combination of, you know, business of law or practice management folks. Maybe it's a dedicated AI implementation role. We're seeing more and more of those lately across the board. But the fact of the matter is if it's not you, it's someone, and that is part of your ethical duty of practice. Right? Comment a has been adopted in multiple states in various forms, and some states have, explicitly come out and said that this does apply to generative AI. I can't remember if we have those resources on this deck, but I have it on another one. If not, I'd be happy to provide it later. So moving on just a touch. I don't wanna take too long for all this. The ABA formal opinion, right, for five twelve is that lawyers must have a reasonable understanding of the capabilities and limitations of specific GAI technology that a lawyer might use. It's not static. There's fast paced. We've already talked about how quickly things have developed in just past few months. Right? But I think it's really important to hit that this is a reasonableness standard. You don't need to be, you know, Leo Park, for example, knowing how to work on these tools right at their core level, but you do need to know, generally speaking, the difference, for example, between an open public use license of ChatGPT and a private enterprise license and what that means. Right? And, again, you can rely on, the support structure of your organization and other specialists if that is not your field, but you need to have someone. You need to have an action plan there. Hey, Cal. Quick question. for you. So you just mentioned how quickly things are changing and that, you know, offerings are changing, models are changing. Does that mean the level of technical competence gets reset every time things take a step up? I think of it less like a reset and more like another step on the ladder. Right? Or another step on the stairs. You're operating from an ever more competent place of technological learning, either through osmosis in your organization, through dedicated guidance, documents perhaps, or just subject matter expertise. Not everybody has to be a deep AI expert. Right? That's not what I think anybody's saying with these rules. I think what they're saying is consistent with existing, interpretations of our ethical rules and obligations that further extends to generative AI and the changes over the course of generative AI. So what is that reasonable standard that we were just talking about? Right? Is it updating your understanding every four weeks? Maybe. Maybe it's quarterly updates in your organization. Right? Maybe it's just being aware of news and changes and tracking if you don't understand. What it the specific shape it might take may differ for you. Right? Is judgment of ethical standards differ from attorney to attorney. Right? But just making sure we're calling out what some of these are. You know, 1.4, communications, consultation, informed consent. I that is a key starting point here. Right? Like, whether it's the engagement letter with your client noting that you're gonna use the latest technology available to reduce billable costs and be more efficient or, you know, getting informed consent to utilize, for example, ediscovery generative AI tools. Right? You're a duty of charge reasonably for the items within the scope and consent. There's been a lot of discussion about that, but I think where most courts have landed from prior cases is that you and and or, interpretation, opinion letters, that kind of thing. You can't charge for work you didn't do. Right? So, you know, we can we can dive into that, but I I do wanna be mindful of time here. We got a lot to cover. I'm not gonna read off every single one of these rules. You can take the deck home with you. But we I think what would be most useful here is if we actually started shifting through to some of the case law. I do wanna hit first on the duty of confidentiality, though. Right? So we're gonna talk about this when we break down the Hefner case and the Warner case, but the risk here, I think, is clear as day to a lot of folks, is what happens when you utilize these tools in support of brainstorming, legal drafting, any of it. Right? Are you risking work product? Are you risking attorney client privilege? That all stems from 1.6. Right? You have a duty of confidentiality to your client. The work product put into commercial LMs could be used to train AI models. Right? That depends on, for example, an enterprise agreement with zero day data retention versus, you know, a private license where you are entering in information that may still be used to train. Best practices are avoiding putting in client confidential information to unsecured generative AI tools, for example, that public GPT license or public cloud license. Conducts due diligence on vendor security protocols, SOC two certifications, is there a zero day retention policy, consider legal specific tools, purpose built tools that are confidential by design. And so, you know, you can do everything to the best of your ability. Right? Or maybe you haven't had any issues submitting, information to the court, but, you know, you notice there's something wrong with opposing documents or you're handed what appear to be client documents and then you submit to the court and subsequently find out they are hallucinated. Right? You have a duty of candor to the tribunal, and it's it's a spreading and increasing problem where you now have inadvertent production of generative material, right, or generative mod modified material. One thing you all you can really control here, right, are your own actions and inputs. And so making sure that you are precluding output from your AI tool that is, say, not cited to a specific source that you can validate. Right? Provide the link. Making sure that you are independently verifying and validating all those outputs. Courts are issuing standing orders requiring disclosure of AI use and certification of human verification. Right? Like, that is an explicit ask for you to follow rule 3.3 here. Right? Making sure you're reserving that human judgment, being that person in the loop, making sure that you are exercising your discretion to validate the output from the Zach Sousa's tool. So, you know, understanding these tools, you need to know their benefits and their risks. And if you don't know them, you need to have somebody who can help you assess that. Right? Whether it's, internal to your organization, an external consulting expert, whatever it is, you need someone who can speak to that. The proper application of those tools, making sure that you're using, say, private enterprise licenses, for example, and supervision of those you manage directly. Right? Whether it's technology or people. So making sure that the outputs that you're getting from these generators tools are being validated and vetted in the same way you might take a junior or summer associate's work and validate that that work is correct. And so we can move on to the case law now, the fun part, things that have really changed over the past year. So I have paired three pairs of cases to go over concepts of privilege, concepts of protective orders, and of ESI agreements. So starting with Warner versus, Barco. Right? This was an interesting case coming down almost at the exact same time as the Heffner decision. And so this is Eastern District Of Michigan. This is February 2026. We have an employment case where the plaintiff was pro se. Right? They used a public instance of ChatGPT to help them work through their case. The defense found out and moved to compel, Okay. the actual queries and the responses. Right? The argument being that typing your case into a commercial chatbot is a disclosure itself and that the disclosure destroys the work product protection. The ruling was no. The waiver of work product required disclosure to an adversary in an AI program as a tool, not a person. There's a much more nuanced analysis here going into when you store documents in a cloud storage, right, like Google Drive, you know, Box, that kind of thing. And so, you know, we we have links to all these cases in these slides. If you take this deck home with you, you'll be able to reach through those opinions and see them for yourself. But the takeaway here was that using AI itself is not a waiver. The question is who you're disclosing it to and a tool is not a who. Right? So the they liken use of, say, ChatGPT to using cloud storage and the documents being processed or uploading to an online eDiscovery platform or similar. So it it came down to whether or not it was considered a disclosure. In contrast, within a week, right, there is the United States v. Hatcher decision in the Southern District Of New York. That was a criminal defendant who was represented by counsel, and then he utilized a public consumer facing AI, right, not an enterprise account. And he utilized it not at the direction of counsel. So he tried to assess his case and potential defenses for his case, brought that information to his counsel afterwards. He did that of his own accord. The ruling was that neither attorney client privilege nor work product doctrine applied. The tool wasn't in his attorney. Right? There was no privilege at the time he was generating that information. It wasn't at the direction of counsel. The consumer terms permitted training and disclosure on the inputs, and disclosure of inputs. Apologies. So there was no reasonable expectation of confidentiality to protect. And, I will say, though, that Judge Rakoff in that case did leave open whether counsel directed use of the same tool come out differently. So same technology as Warner, kind of the opposite result. But if you're gonna thread the needle there, the difference was whether or not it was at the direction of counsel or tended to reveal thought process to counsel. Right? And so where you had a pro se, plaintiff who was representing themselves and turning it in, it was considered under work product. Right? Where in half or you had an, a defendant who was represented who was acting independent of their counsel and not the direction of counsel. It was not protected. There's an interesting note in the comments, Cal Yeaman, that, Hepner was work product analysis in a criminal context, whereas whereas Hepner was in a civil context. Yes. Yeah. Great I do wonder. how much of the impact there was there. Technically, on a pure reading, I I don't think so, but I think in a practical reading, yes. So moving on to, Jeffrey's. Right? So we're moving on to, protective orders here. AI use was surfaced during the course of discovery, and defendants moved to amend the protective order. Their ask was to bar any discovery material being uploaded into a public or open loop AI tool, whether or not the document was ever marked as confidential. So any previous document. Right? The ruling was granted. Once decision was ingested by the open system, one cannot reliably claw back or delete that material. Right? Because it can be used in the training data. The the the granting of that ruling required five business days advanced notice before the AI touched discovery, plus disclosure of the vendor, the hosting, security controls, and the training policy. So it's a heavy lift to validate prior to utilizing the tool. The important thing to consider here though is that this ruling, regulated the technology, not the confidentiality. Right? It it was independent of the confidentiality basis. If it is received structured materials, right, going into the tool, there needs to be advanced notice. Compare that to the Morgan case, right, where we have a pro state plaintiff utilizing AI to help her prepare filings. The defense wanted the name of the tool, but in addition to the tool, they wanted the prompts and the outputs of the prompts, right, that she had used on confidential material. There was a slightly split ruling. The mental impressions and content stay protected as work product, but the identity of the tool itself is discoverable because the defense has a legitimate need to, assess whether or not confidentiality was compromised. Right? Let them validate for themselves whether or not any of their client materials appear. That is a finer parsing that tends to support the idea that there is a difference here. Erin, you and I have talked about this plenty of times from different sides. Right? You take different positions on this. But whether or not all of your prompts are protected under word products or whether or not there's a differentiation between those prompts that tend to reveal a thought process of counsel and prompts that say reflect verbatim a non confidential document like a request for production from opposing. What do you think about that? It's what what it what it makes me think about actually, can I get a can I get a show with thumbs up? Do I sound better now than before? Hopefully hopefully, that's a yes. Alright. Good. So what this what this makes me think about is, you know, if you if your AI tool is now discoverable, does that also potentially extend to the rest of your AI tool stack? Right? Anytime AI gets any anything that AI is now touching, does that even the name of it just become discoverable, which can be you know, you never know what it's actually going to reveal. You never know about, you know, how thing how people are gonna use that information, but it's it's kind of a natural extension as I think about what I would do with that information. I would definitely dig into what those what that tool works with, what are the consequences of using that tool, what are some of the things that are people are talking about saying, this is what it does well, this is what it doesn't do well, and then trying to leverage that, you know, to the best advantage for my client. Yeah. Absolutely. From from a position perspective, I I come from that defense posture where I see all of this as protected. But I certainly for tracking and processing purposes and maintaining the the project bible or the log of what you did on a case, I did, in past practice, separate out those prompts that tended to reveal thought process counsel from ones that I would consider, you know, perhaps verbatim of RFPs to be able to show easily that we did indeed run the RFPs as they were provided, if that was one of the requirements. Right? This is all a little bit of the wild west early days, and I don't know what will and won't be provided or how it will or won't be viewed two years in the future when things are settled practice. But I kinda figured it doesn't hurt to be as careful as possible and as documented and regimented as possible. You know, with regard to the discoverability of prompts, I'll just throw out that, last year, I was moderating a panel where the way we we're where we came down since there isn't any law on this yet is really not none that I'm aware of was that prompts that have to do with calling data so that you are deciding what you're not looking at or what you're not reviewing would be discoverable, whereas prompts that you're using for fact finding or investigation or depot prep or strategy would not be. And so it was a but it took a while for us to get there on on the day as, when we were talking because initially, everyone was like, no. Everything. And then somebody's like, no. Nothing. And then we we kind of came to a finer point. Yeah. You know, that's actually a pretty good segue as we roll into the ESI protocols. Right? Because that's that's really where the rubber needs to go with this stuff is less to do with what happened in other cases and more to do with what is happening in your case right now. Right? So looking at the Schultz case, this is an antitrust class action case. It's June thirtieth of this year, Northern District Of California. And this and the next case following are interesting because they're both the Northern District Of California within a few days of each other. So this case, antitrust under the existing ESI order, LinkedIn disclosed that it would use Relativity Air, no seed set, no training set, and make a final responsiveness call with queue and review of samples as the QC measure. There are 19 custodians. There are 25 search streams that brought target population down to 204,000 documents and change. Uncalled, two custodians alone had over 800. So just making sure I'm breaking stuff down, the electronically sensitive information order. Right? The SI protocol, the rules that you agree to when you are utilizing eDiscovery on a case linked in in the Schulz versus LinkedIn case disclosed that they would utilize Relativity Air, a generative AI review platform. I believe it was Air for review in this case. They they wouldn't take a validation of statistically defensible or statistically measurable sample that represents the population. They wouldn't, use any, predictive coding. Right? They were use using straight generative AI review. And, in doing so, they had filtered down and predefined a search population where you had a raw count where, you know, you're talking about 800 gigabytes between two custodians, and we had 19 custodians that utilized 25 targeted search sets to create the review set, and you're talking about 204,000 documents. So a major reduction in the review set prior to utilizing the generative AI. Right? And then from there, in response, plaintiffs wanted the following things. And this, Aaron, this goes back to how you and I are different points on the spectrum. I wanna talk about this often, but the plaintiffs wanted the keyword pre cull barred. Right? They wanted error on across everything. They wanted validation metrics produced, including the illusion rate estimate, so how many potentially relevant documents were not discovered by the generative AI, what that error rate was, what the QC overturn rate was on that human review, and the what the reviewer counts were of actually human reviewed documents. From a defense posture, I feel like they're asking for a lot there. It feels like they're asking for the moon. I think from a plaintiff's perspective, and you could validate for me, Aaron, that this seems like a pretty reasonable ask because you're talking about new technology, and you want to know that this other side knows what they're doing and that you're getting what you're being sold. I I think the truth probably lies somewhere in the middle. Right? In this case, though, all three of those asks were denied, and the ESI order required disclosure of the technologically assisted review use. Right? And so when we think of TAR, for those who are familiar, we're probably thinking of machine learning, TAR one point o, two point o, predictive coding, continuous active learning, or Cal. However, technically, technologically, assistive review is a method, not a tool. And so I think here, we're referring to it in the context of generative AI review. Anything past the disclosure of the use of tar was not really covered in the original ESI protocol. And anything past that, the additional asks beyond that seemed to the court to be discovery on discovery, absent a specific production deficiency. Right? So this was, maybe not ripe for the plaintiffs to bring up in the position of the court. And so takeaways there, I think, are that nobody challenged the validity of the tool. What they were figuring out was the process by which they would agree to accept those productions. Right? And this is all sort of a squishy area happening live. What do you think, Aaron Patton, coming from that plaintiff posture? I I I think that the, a lot of the things that the plaintiffs were asking for are reasonable, but they should have been in the ESI protocol. And that was I think that's what leads into the next case, which is, the Sarah Bryce case, where I I they they seem to get at they seem to be opposite findings, but, actually, they're. consistent with each other and that you get. what you agreed to. Quick time check also because I think, Yeah. Zach Sousa. I'm I'm gonna go through this one super fast, Okay. I think. And for folks who are asking, there are links to all of these cases where you can review them in greater detail off of off of this CLE. But the high line here for, the service case is that this was a stipulated ESI protocol. The parties negotiated and the court entered into. Right? That's what we're talking about here. It had AI responsiveness review as its own category of review with its own disclosure obligations attached. The model and the version of the model needed to be disclosed, the prompts, and every change or iteration to those prompts, validation method methodology and an audit trail. There is no, you know, court holding requiring this. There's no federal rule requiring this. This was something both sides agree agreed to and asked the court to enter. Right? So this is one week after Schulte. This is almost the opposite disclosure posture. You're right though, Aaron, that there's a common through line here. Right? That said the the variable is not the generative AI tech. The variable is what the parties agreed to and what binds that discovery. So if you want it, put it in the ESI protocol. Alright. I think from there, you're sort of getting a flavor for these key takeaways. Right? AI is a tool, not an adversary. Consumer AI without counsel does lose that protection. Open loop bans reach nonconfidential material. Strategy is protected by Morgan. The tool's name is not. GenAI review is a technologically assist review, and you disclose per the protocol. And anything that's not in the protocol, right, is not something that you're, if you want something to be considered in this defensible process, you need to put it in that protocol. Right? Maybe we'll have some time at the end to discuss, but I think we can move on now to the actual tech itself and how it works. Thanks, guys. I'm Leo Park, and I will be discussing generative AI and retrieval augmented generation. Generative AI is useful for accelerating many forms of knowledge work, such as drafting memos, searching for relevant text, and summarizing lengthy documents. While they can analyze text in many sophisticated ways, large language models don't currently perform well on tasks requiring judgment or specific domain expertise. When presented with competing or conflicting facts, large language models can perform poorly when the correct answer depends on nuanced factors such as fact recency, fact significance or salience, specialized knowledge, or judgment. Large language models also have a knowledge cutoff, which is the date of the most recent information provided during pre training. Pre training is the unsupervised learning phase where the model consumes vast amounts of text collected from many sources. Due to the effort required to curate and clean the pretraining data and the training time itself, the model's factual knowledge can be more than a year old by the time pretraining is completed. This influences what a large language model predicts as the most likely answer when it is asked questions that require knowledge of recent events. Along with knowledge cutoffs, language models are influenced by the information that is most prevalent during training. For questions that require subject matter expertise to answer or domain specific knowledge, they may provide answers based on the common meaning of words instead of the specialized meaning the author intended. They can also hallucinate when an answer is unexpected based on their training data. In this screenshot shared by a Reddit user, a previous OpenAI model repeated multiple rounds of research and concluded that it was hallucinating when its search results stated that Zoran Mamdani had won the mayoral Democratic Party, primary for the City Of New York. Model performance also degrades significantly before their context windows are full. This graph compares the performance of recent GPT five family models against GPT 4.1 on the open MRCR benchmark. This benchmark measures multi needle retrieval across long contexts full of distracting language patterns. Newer models have greatly improved in context retrieval, which is asking the model to surface information it has already been provided. Still, all models decline in performance as input tokens increases. Achieving the best quality still requires staying far below the model's maximum published context limits. So what is retrieval augmented generation or RAG? RAG is any system that augments the model's context with external information. This is great for answering questions about specific documents. When the user asks a question, relevant text is retrieved from trusted sources, which is added to the model's context along with the original question. With RAG, a model is far more likely to generate faithful answers based upon the retrieved documents. This grounding in trusted sources allows LLMs to provide accurate answers to a wider variety of questions, avoid knowledge cutoff limitations and information frequency bias while reducing hallucinations. RAC has become synonymous with semantic search using vector databases. However, a variety of different information sources can be used for retrieved information, such as web searches or more traditional databases. For example, ChatGPT contains two separate features that utilize rag patterns. Most chat assistants now have a rag feature that allows you to create a project or a folder of shared knowledge. That knowledge can be dynamically retrieved to provide higher quality answers to questions on a specific topic. ChatGPT can also employ web search to retrieve up to date information for answering your questions. This can help to avoid the knowledge cutoff problem. Let's break down rag even further. One of the most common information retrieval systems used in RAG is vector search, also known as semantic search or similarity search. Vector search involves converting your document set into a vector embedding or just vectors. These vectors are numerical representations of the semantic features contained in the original text. Those vectors can then be retrieved based on their similarity to the user's question. In this example, the process begins with the user asking, what do dolphins eat? When the question is submitted, that question is converted into a vector. The question vector is compared against all of the document vectors to identify the ones most similar in meaning. Document vectors are retrieved based on their similarity score. The document vectors are then matched with their original text and metadata and provided to a large language model. The model can now provide an answer grounded in trusted external sources. The similarity score does not always translate into relevance. In this illustration, some retrieved text may discuss dolphins and eating without addressing the user's question. This is where the LLM steps in to generate a more precise answer by selecting retrieved information from the retrieved documents. What are some benefits and limitations of vector search? So document processing is critical to rack performance. Vector search is a... Is popular because it is flexible and performs well over many types of files with minimal preparation. However, chunking schemes and embedding model choices can affect overall rack system performance. Chunking needs to be designed with the input documents and characteristics in mind by choosing the right document length and segmentation logic to create retrievable chunks. The embedding models themselves have different performance characteristics based on the training data. Overall, rack system performance requires careful design and engineering from end to end. The technology stack must be tuned and evaluated to have a system that is secure, maintains data confidentiality, and performs well over the intended documents. Where does semantic search work well? When you want to search with natural language instead of strict keyword matching or Boolean search syntax, natural language querying means that no specialized knowledge is required to start asking questions and getting grounded answers. This also means that you, you get to search by meaning, such as a sentiment search, where you may not know exactly how the words appear in the documents you're searching, but you know that what this... What the speaker intended. Where does semantic search not work well? One area is very domain specific language. Similar to large language models, embedding models are pre trained on large document sets containing many types of language and face the same challenges when working with domain specific text. The result is that specialized terminology may not retain their domain specific meanings within the model's weights, which translates into reduced retrieval performance. In specialized domains like law, be sure that your RAG provider is using a model that is trained for legal retrieval. Both Voyage AI and Google DeepMind have released embedding models that are specifically trained for legal domain tasks. So some of RAG strengths. The benefits of RAG include extracting and summarizing relevant information from large amounts of text. RAG systems can combine the most relevant information from a variety of sources to provide higher quality answers than searching one document at a time, and it is effective over volumes of information that presently cannot be placed within a model's context all at once. Also comparing disparate sources of data automatically. Because RAG combines the retrieval capabilities of vector search and the language capabilities of LLMs, you can find answers from emails, reports, presentations, and other sources of text at the same time using natural language query. And RAG pairs well with traditional search methods for even more powerful results. For example, Everlaw's deep dive is built on top of the existing core discovery platform, allowing you to pair traditional content and metadata search capabilities with advanced reg search. There are risks and limitations. Some questions do not perform well for both retrieval and answer generation. Questions that are multi port or contain an or term, these can be difficult for symmetric retrieval systems because the document needs to match every part of the question to score highly during the retrieval phase. Questions that ask every or all, Rack systems generally do not perform an exhaustive search for every instance of your query. Instead, they focus on the highest quality examples from your document set. Questions requiring an exhaustive answer are better suited for traditional search or linear review. Additional retrieval quality problems can include poor data quality. Retrieval can suffer due to poor data quality such as a loss of meaning due to chunking. While vector embeddings are adaptable to many forms of input text, variations in chunking methods can result in semantically splitting a topic between two chunks, meaning neither chunk contains the full meaning from the original document. Poor retrieval performance can also occur when the embedding model isn't suited for the user's task or the kinds of text information being embedded, such as questions that are common during legal investigations. Retrieval performance is also impacted when a document set contains many duplicate or near duplicates. In a very large document set, it's important to have a deduplication system in place that removes duplicate chunks so that the retrieved document set isn't clogged by repeated examples of the same text. Generation problems include confident but incomplete answers. When provided context, the LLM has no way of knowing how comprehensive or complete the additional context is. Verify answers by asking several related questions or trying variations in wording and see if the generated answers are consistent. Errors and generation, AKA hallucinations. LLMs can make mistakes, and their error rate increases when they are dealing with large amounts of information or out of distribution texts. Ensure that your provider evaluates l o m performance regularly and updates their models, prompts, and contact manage... Context management systems to reduce these types of errors. Newer agentic rag systems can overcome many of these limitations by breaking down the user's question into separate queries, which increases retrieval quality and perform successive rounds of research for more robust answers. Okay. Now it's my turn, and we get to see things happening in action. So everything that Cal Yeaman and Leo just covered, including the info guardrails and the the rag mechanics, so retrieval augmented generation mechanics. It's about to show up in one continuous workflow, and I'm gonna run an like, an actual matter from cold start to a drafted memo using using a rag tool and some generative AI. So some of the key workflows to think about when using Rag are things like early matter exploration. How quickly can you get to insights? How quickly can you identify issues and entities? And also later in the in the case, deposition prep and trial support, even briefing. So think about think about those places when you need to find surgical information when it matters the most. That's where you'll be that's where you'll be reaching for a rag tool in your litigations. So what does this look like in practice? So I'm going to go away from the slides, which I think everybody will be happy about. But also, as you've seen today, this is live. So things get interesting. Now I've I've done a lot of this ahead of time, but I'm gonna take us straight into it so that we can so we can actually get through this. I know that we are at seven minutes until the hour, and if we can get an extra five, then I will make sure I am done by five past the hour. Okay. So I am going to share my screen. Oop. Don't look like I have permissions to share my screen. Let's see. Okay. I had permissions, and now I don't. Maybe I will, stop sharing and see if I can share. There we go. So now let me show my screen. Here we go. So you'll notice that I am starting in Claude, and that's for for a specific reason here. So what I want to be able to do is set up my rag tool. So I set up my rag tool. The comments, by the way, are hilarious. So, I'm gonna set up my rag tool using Claude. And the idea here is to take the complaint as well as a skill I've built for configuring my AI and then creating a document that will not only create the case context but also some of the other pieces like coding suggestions, sample prompts for, the Rag tool as well as writing assistant. So scrolling down here, you can see it's interviewing me, asked me a number of questions about what the case is about. Also, can I get thumbs ups that everyone can hear me okay and this mic is better than the last Star Wars mic? Awesome. Okay. So I answer these questions. I give it the context that it needs. I am the person who is making the decisions about what happens. So now I've got a fully configured case, including a case description, and it puts everything inside of this window. Now I'm gonna scroll to the bottom because that's where I wanna show show you. So I actually have just pop in for one second, Aaron. yeah. So re rephrasing what I just heard you say, groundedness in those ethical frameworks and guidelines. Right? You're making sure you're using a an enterprise license or other protected confidential zero day retention tool. Right? You are controlling what is or is not trained upon. Right? Yep. And you're exercising your discretion and judgment by being the human in the loop so that you're the final arbiter confirming the initial work product. Exactly. Exactly. And I think the other thing is I I made sure that the skill interviews me so that I'm making those choices. So at the end of the interaction, I end up with a nice document here that I can copy and paste from so that I can set the context for my for my rag tool. Here, we're using deep dive since we're gonna be inside Everlaw. Is is it still frozen, or are we still okay? Everybody see the case description? Yeah? Alright. Good. Okay. So now I'm going to take this case description, and I've actually made it even shorter, and I've dropped it in here. So now the case description inside of my tool, has been I've actually made it shorter. One of the things that Leo taught me about the, case description is that if you give it less, it can do more analysis. And so I tried to give it as little as possible so that it has as much, context and analytical power as possible. Okay. So, Cal, I think one other callback here is that I'm not just copying and pasting this. I also had to read it. Right? I have to make sure it's right because things can be confidently wrong. Yep. Yeah. Or, you know, I think there's a people tend to use hallucination as an umbrella term. Right? But they're they're describing a series of different potential concerns using generative AI. For example, maybe everything it said was factually accurate in terms of whether or not it exists, say, in a document's extracted text, but maybe the context and the meaning from those extracted words differs than the interpretation from the model. Right? Right. So now that I've given it the context, I'm actually gonna take one of the, one of the sample prompts that I got from that word document, and I've actually I've already put it in here kind of recognizing that we are gonna be pressed for time at the end. And so you'll notice I can ask questions that are fairly, I can ask questions that are they're fairly informal. I can also quest ask it questions, you know, with more context, like, here's a here's a persona. Look for evidence of legal anxiety, ethical conflict, consequence mitigation, apprehension. What I'm doing is I'm I'm telling it who you are and I'm ask and giving it examples of what to look for. And the result, I'm not gonna run it live because I don't wanna burn any time. But what I get is an answer that's made in context. So what's happening is I've asked my question, and the entire database, like, Leo had suggested, was the entire database has been turned into vector embeddings. My question turns into vector embeddings. Those vectors are now inside the database as points, and the most responsive documents get pulled in. And then my question gets answered against their text with the context that I put into the case description. And now I've got answers. I've got quotations directly to directly to the documents. Like, this is if I need to see what this document actually says, when I click on that, you know, it takes me into that document so that I can see it. Go ahead and close that. So I've now I've made sure that every single one of these claims, now they all have sites being pressured to explore aggressive accounting treatments. So every single thing is transparent, taken all the way back to the document. And so so, Cal Yeaman, I think one of the things that you had mentioned with the duty of candor is that this is a starting point, but it's definitely not a, definitely not the ending point. This is not finished product. That's right. I mean, look. If if you're there there's different ways that you can engage with tools. Right? And when it comes to something like this, keeping that critical thinking cap on, using your your sound legal reasoning and judgment, and assessing the validity of what you've put in and taken out from these tools is part of your ethical duty and obligation. Right? So what does the unethical use case of a workflow like this look like? You load documents into Everlaw, never having looked at them. You ask questions based on the status of your case. You don't look at the whether or not the answer makes sense in context. You don't go to the cited documents and validate that what is synthesizing and threading together in terms of factually accurate extracted text is in context contextually accurate. Right? So that would be the unethical approach as opposed to validating, being human in the loop, exercising judgment, considering what your input is, considering what your output is. Yep. Taking this a step further, now that I've let's let's assume that I've I've met my ethical obligation to review all these documents and make sure that they are what I am looking for. Great. Fantastic. Now I can take all of these documents, and I can put them directly into another generative AI tool inside of Everlaw Gold. It's called story builder. So go ahead and hit copy and move those all over. And so now I've I haven't taken this because I'm the lawyer. I have to make the decisions. I make the conclusions, and so I'm just taking the documents. And now I've put them into this mental state memo, which is what I what I just did. And now those documents live here, and I've got a tool inside of story builder that is it's similar to what you would use with a Harvey or a Agora where there's drafting happening. And so I use this assistant here, and I'll I'll click on this prompt because I got this prompt from the initial configuration document. So I copy and pasted it in, and then I had it run and I ended up with an 11,000 word memo. Now maybe that's too long, but it's giving me something to work with because it's taken all of these documents and all of their context, and it's written it all up for me including citations to those documents. Now granted, this is a first draft. I I imagine when I was a first year associate, I turned in drafts that were 11,000 words probably. So let's take this document as a first draft, and I export it out as a Word document. And now now I'm gonna come back to Claude. So now I've reviewed those documents, and I've made this memo. But now what I'm gonna do is actually start to start to think about what I can do with the information. So what I end up doing is I give it the memo that we just downloaded, but I've also made a skill that looks at the mental state of all of the of the of the people and the participants. And I said, give me a tell actors are, and tell me who the bystanders are, and make me some visualizations that will help me understand really quickly, plus write a memo in our firm's voice. So what I end up with is the conclusion first, and the center of gravity for culpability is the legal function. That's interesting. This is my actor manifest, who they are, what their sentiment and conduct signatures are. These are the bystanders, people who are not replying but are on, you know, cc's but not necessarily, active participants. And then I asked it to make me some actor cards, and you'll notice they're color coded. So I can see the people who are willful or knowing as having red and people who are you know, maybe they're knowing, but they're maybe on the verge of negligent. And I can actually look in here and see some of the documents. I can see the actual base numbers and doc IDs as well as some, some characterizations of who these people are. So we're at 205. So So I'm gonna be quick here because I said 205 was where I was gonna stop. This is the this is what I what the the whole point of putting it into Claude was so we could see people's mental state over time, so that we could see people's culpability over time and so we can see who is pressuring whom. So we can see that Natalie Rhodes is pressuring Alex Carter to refresh his memory to maintain minimal written records. And at the bottom here is a compendium of all of these pressure documents along with an analysis and some high level takeaways at the very beginning. So I'm gonna go back to unsharing. I know everyone's excited to see this. I'm gonna unshare here and come back to the slideshow, because we're at two zero five, and I just wanna do a quick review and also mention a couple of closing items that we, need to read through. So these are the workflows that we just went through. Always remember that we've got humans in the loop, and we have to iterate to make sure that it's right because things can be things can be accurate but incomplete. Also, something important to remember. And always always have human judgment in the loop. So coming to our last slide, Cal, why don't you take us through the last slide and then I'll read the last bit from Zach about the, Sousa and the the ending poll. good. And so as we go through this slide, I just want everybody to keep in mind this this concept and framework. Your ethical rules and obligations as an attorney have not changed. These are a specific flavor of the application of these rules to a new technology. Right? So that same groundwork that you have always operated in still plays today. So, you know, you need to utilize your your judgment as an attorney and as an officer of the court. Right? You need to verify everything, whether that is manually prepared human work product, whether that is AI generated first pass content that you then review, right, or or sourced answers. You need to consider the context of the information that is being pulled together to create information. That's, again, the same as whether or not it was a summer associate pulling together different case law, right, or whether or not you're talking about a synthesized answer from deep dive using a RAG implementation. The confidentiality is, as always, a key consideration. Right? What, the case laws the case law examples that we went over over at the top of the hour, parse and create a finely nuanced line about when something is or is not protected under the traditional work product doctrine as specifically applied to these new tools. Right? And in all of this, as you put these ethical rules into practice, practice, what you're doing is maintaining your judgment and your discretion. Right? Being the human in the loop to make those final decisions. Being the person to assess and validate that the tool is telling you what you think it is telling you. Alright. And, again, guys, if you have questions that we, have not yet addressed, we're gonna look at those and follow-up with you. Alright, Aaron. Yep. Take us a look. So I wanna thank everybody for joining and for bearing with us through the technology and through our, episode with Star Wars sound effects and with mutes and freezes. Live demos are like that, and I we really appreciate everybody hanging with us as long as they could. This is recorded. If you need to see any of this again, we can always set up demonstrations. We can always set up a quick call to walk you through it. Just reach out, let us know, and we can make that happen. Also, all of the skills that I was talking about, totally available if you want them. So just let us know that you want those. And, we've got a short session survey at the very end, that Zach is gonna pop up. So let us know what you thought of the content. No. Maybe the maybe the the a technical difficulties. technology is let's let's see if we can separate the technology and the content hopefully and, and and get some feedback. Because if this is the kind of stuff that you like to see, that's the kind of stuff we will make. Okay. I just. I just wanna say, can you all hear me now? Yes. We hear you now. this working? Okay. Hopefully, the audience can hear me too. I'm so sorry about all the technical difficulties. I'm happy to finally be able to talk to you all and cover. Thanks, Aaron, Cal, and Leo. That was great. As I mentioned, we had this session recorded. You should automatically receive an email within the next twenty four hours reminding you of the link to watch on demand if you wanna come back and watch this again at any point in the future. So please keep an eye out for that. And I did see a comment in the chat, someone asking that they wanted a whole day of CLE on these tools and law. Well, I have something just for you. We have Everlaw Summit coming up. It's our in person summit. So now that the full hour of CLE is done, I'll just plug this really quick. It's available for both existing customers using Everlaw today, or if you're, you know, a potential customer exploring Everlaw, this is a great place to do it. We do have sessions that are accredited for CLE in person in San Francisco. So if you are interested, there's still time to sign up. We also have a 10% off promo code for your ticket order using the code webinar there shown on screen. So take a look, and, we'd love to see you there in about a month. So thanks, everybody. Again, thank you again for bearing with us, and we hope you all enjoy your day. Thanks, y'all. Appreciate your time. Thanks, everyone.