AI & Sloppy Justice: A Conversation w/Civil Rights Corps Senior Attorney Carson White

FULL TRANSCRIPT

This is Due Processing, I'm Avi Singh.

I'm Kate Chatfield.

Hey Kate, what's happening?

Not much, you know, still fighting the good fight. How about you?

All good, all good. Excited for this conversation we're going to be having. We're joined today by supervising attorney at Civil Rights Corps, Carson White.

Carson, welcome.

Thank you, thank you.


AI in Courts

We are talking today about artificial intelligence. We're talking about the ways that it can come up in courts, the ways law enforcement, including district attorneys and actually all attorneys are using it and the ways in which it can cause all sorts of trouble. You know, this can present all sorts of issues for people who are listening to due processing and people who are interested in civil rights and what's happening when kind of humans are moving out of our legal proceedings or kind of taking a back seat.

So Kate, what do you think? How do you approach this question as a starting point?

Well, I think we have to as a starting point accept that it's happening in all kinds of offices, in district attorney's offices, in public defender's offices, in private counsel's offices, and all throughout law enforcement, from police departments to the use of surveillance technology, which transforms into artificial intelligence. It's sort of getting to be the, and I hate to say it, it's sort of like the water that we're swimming in now. And, but we have to put up some guardrails because it really is changing, I think, the practice of law.

And whether that's a good thing or a terrible thing, I think now is the time to start assessing that. So my thoughts, but I'm really excited to talk to Carson because talk about the water that we're swimming in. I know you are swimming in this.

So yeah, talk about what you're seeing around the country.

Yeah, so I can't get into the specifics of it beyond saying that this has come up recently in a case that our office is working on and that a district attorney's office has recently admitted to using AI to draft briefing that resulted in the submission of fabricated legal authority. So non-existent quotations, non-existent case holdings, right? So non-existent laws really in criminal cases.

So I've been reading a lot of, yeah, these opinions from around the country of lawyers getting sanctioned for using AI, which now I think there are literally hundreds. I haven't read all of them, I should say, and articles about it. And it's really, it's like very dystopian, it's very disturbing.

I think my headline for it is, is the more that I learn about it, the clearer it is that AI is really, really bad at drafting legal arguments. And it's bad at a lot of things, but it's, I think, uniquely poorly equipped to do any kind of legal analysis.

AI Legal Flaws

Wow, do you have any sense of why that is?

Yeah, so I also have my prefaces by saying I was a liberal arts major. This is a liberal arts major's understanding.

English major here, yeah.

Translated for us, yeah. Yeah, philosophy. Help us fellow non-tech people.

Eastern European Certificate Studies, AI. I mean, so these programs are, they're language learning models, right? So what that means is it's not, they're not understanding in the way that we would think of text.

They're just predicting what the most likely next word is in a sentence and doing pattern recognition. So the kind of work that it takes to read a legal opinion and to parse out, okay, this is a holding, this is dicta, this because it was decided in this court only apply certain places. Here's how it interacts with another case just isn't something that it's able to do.

I think several studies have come out now too, that generative AI is really bad just in general in any format at like rules, right? At defining rules of a data set that it's given or of being able to tell when rules for a new data set change and adapt to that. It's also, you know, law is so niche, like the things that in these large language models are really good at are spitting back facts that are kind of universally known, really widely reported on, and readily available from a lot of different sources in its training data.

That's not true of most laws, right? There's not a ton of articles on the Internet discussing, you know, especially state court opinions. It's just not something that's readily available.

So, there's some data that shows that it hallucinates, first of all, just rampantly when answering legal questions, like sometimes up to 82%, or even the Westlaw and Lexis specific models, at least the ones that were rolled out a year or two ago, hallucinate something like 33% of the time. But they hallucinate much more about state court cases than they do federal court cases, and even within those, they hallucinate much more about like district level or court of appeal cases, and they do United States Supreme Court or state Supreme Court cases. And that's just because of how frequently those cases are mentioned.

And then there's also this growing consensus that like as a feature, not a bug that AI will necessarily hallucinate an answer if the answer to your query isn't within the data set. So it's this like really dangerous thing where actually the time that it's most likely to hallucinate, or especially likely to hallucinate, is if the legal principle you're asking about isn't supported at all. So it's much more likely to give you something that doesn't exist than it is to give you the wrong case for a legal principle that actually does exist just from a different authority.

So is that that it's giving you authority sort of, and this is real, like to please you in a way or to predict the answer that you want and then giving you that authority, which is the exact opposite of what we should be doing when we're researching. We need to look for and understand the bad cases, the cases that don't support our proposition. Is that a fair summary?

Yeah.

One thing that might kind of compound the problem is that it looks so good, right? You know, you put stuff in and then it spits out this thing and it looks just like a pleading and it looks very legally, legalish. And the way that it's revealed is often appears to be, from cases around the country, pretty subtle.

So for example, it is a real case, however, the proposition for which it stands is wrong, or it's not a real case, but it's citing a common proposition kind of in the vicinity where that common proposition should be going. Does that seem to, is that something you're seeing in your survey of cases, not the case you're working on right now?

A lot of times, too, it's interesting. It's case names that don't exist. So something that's really common are these kind of Frankensteined case citations where the case name will be real, or the reporter citation will be made up, or will go to a completely different or relevant case.

And that's what a lot of courts are seizing on, because I think it's not, unfortunately, I don't think it's that uncommon for a lawyer to just completely misread a case or really fudge what the holding of it is, but it is very uncommon to butcher the reporter citation that badly.

And the reporter citation is the thing that the person who's reviewing the thing would have to type in and get to the case. So it's kind of the first tell that you're not talking about the thing you appeared to be talking about.

Part of what's so hard about it is that it's, unless you're really familiar with the cases or you're jeopardizing all the citations, it can be really hard to spot. So the other tell obviously is if you're really very familiar with the case law that's essentially being talked about and you can tell, wait, that's not the whole thing, or this is the wrong here for that case, something like that.

I think we just kind of are taking for granted that it's a good thing for people who are lawyers to arguments to be citing things that are real. We assume that that's the case, and we assume that the analysis that's included is what it purports to be, right, for us being attorneys and doing legal stuff. What are the specific, I can imagine a bunch, but what are the specific concerns in the criminal legal context?

Criminal Justice Risks

These are the most important cases. It's people being torn apart from their families. It's people being put in cages.

It impacts their livelihood. And it's also the place that's least equipped to deal with these kinds of errors. We're talking, you know, that the system is incredibly overburdened just because of the sheer number, right, of people that are getting conveyor belted through.

You know, the overwhelming majority of decisions in state court, they're not, it's not like federal court. They're not delivered via written opinion, where the judge makes clear what cases they're relying on, or what authorities, even what arguments that they found persuasive. It's often just, you know, delivered orally, granted, denied, whatever the motion is.

So there's, it's very hard to go back and check, you know, has a judge relied on something that wasn't accurate? And of course, it's difficult to find in the first instance. You know, most public defenders offices just do not have the resources to be jeopardize every single citation and every single brief that gets filed.

And the judges certainly don't have the resources to do that, right? It's not like federal courts, where even the magistrate judges often have law clerks or multiple law clerks who are doing that kind of deep dive into the party's briefings. Unfortunately, if you get a judge that reads the briefing in the first place, like that's pretty good, right?

And isn't just going off oral arguments. So it's, yeah, it's a place of the stakes of the highest, that the hallucinations are least likely to get caught. And because of, you know, what we were talking about earlier, where AI knows the least about state court decisions, it's also a place that it's kind of uniquely likely to hallucinate, and of course is uniquely likely to hallucinate when the things that you're saying aren't supported in law at all.

So it's this like really horrible combination of conditions with just horrific potential consequences, which is that judges will be jailing people, denying them their basic constitutional rights on the basis of laws that don't exist.

And I'll just add about, you know, kind of, I don't want the lens, and I think Kate made this pretty clear, to be, you know, what prosecutors are doing badly with AI, because this is an equal, or, you know, anyone could be using AI in a way that causes harm. And that's equally true of, you know, a defense attorney who uses AI and cites an incorrect case, right? They're going to lose all their credibility in a particular argument where somebody's rights are involved.

I question how effective you can be as an advocate when you're not doing stuff, right? I think the particular dangers are where you're telling people what the law purports to be and you're wrong about it. You know, how that shapes your entire ability to advocate for somebody.

But I think there's a particular temptation in public defense. I mean, as I'm listening to this conversation, this is really intersecting with a public defender crisis. You know, the defenders are, you know, overworked, overwhelmed.

In some counties, you know, they're taking way too many cases. So the temptation to rely on, you know, AI even knowing that it's bad is just really too great, you know, and that's something that, you know, we can talk about absolutely shouldn't do it, you, you know, but what are you doing when you're in an office? It has no paralegal, no support staff, you know, you are it, you know, for far too many clients.

That's part of like a larger conversation. But it does seem to me that in terms of like, fault, maybe, DA's offices are much better resourced across this state, at least I would imagine nationally, than public defender or indigent defender offices are. I just, I'm looking at like a gradation of, you know, wrongdoing maybe.

And, but obviously I'm not a disinterested party in this. But anyway, that's, that's sort of an interesting thing. So it's interesting, Carson, that in a case, that it's a DA's office, it'd be interesting to look in that county, like how much better resourced that DA's office that's using AI is than the public defender's office.

And how was this even caught? You know, that, that's the other thing. Like you said, public defenders don't have, you know, the ability to jeopardize, you know, every single case, nor do judges, you know.

You might get one clerk for an entire superior court caseload in a county. So how was this caught?

Yeah, I think that's exactly right. So specifically in this case, the PD's office receives less than half the funding that the DA's office does. And it's sort of, this point is really interesting.

They're not commenting about what's happening in that county specifically, but where that difference in resources can even boil down to like what kind of AI you have access to, right? So there are now these tools for Westlaw and LexisNexis that are kind of specifically legal focused and that do hallucinate much less often than like ChatGPT and these other publicly available ones, but those obviously cost a lot of money. So it's, you know, even within that, I think there's discrepancies.

Overwhelmingly, nationwide, the mistakes are getting caught by the judges themselves. And again, these cases are, I'm not aware of, of course, there wouldn't be a published decision, but I'm not aware of a written sanctions order or anything out of any California, like, Superior Court, trial court level. They're all coming out of courts of appeal or federal courts, which I think makes sense, because it's not happening rampantly in the trial court, but I think because it's not getting caught, or because it's not getting litigated.

Right.

Carson, when you see in your review of cases, when someone is caught, or when the concern about AI has been raised, what I imagine what should happen is, you're right, we used AI, we thought it was appropriate, we reviewed the briefs, we didn't catch the error, you caught the error, and we regret our error, and are embarrassed by it, or something like that. And we'll try to do better in the future. Is that how people are reacting, or what's the AI?


Attorney AI Excuses

Yeah, that's a great question. What's the first phase of AI regret for an attorney? So I think there are some lawyers who respond that way, and that's obviously, if you're ever in the unfortunate position of having submitted fabricated authority to the court, the best move is to own up yourself, right?

Ideally, before the court finds it, before opposing counsel finds it. And then to be really honest, very apologetic, I think, you know, with an explicit understanding of the threat that this is to the integrity of the court, et cetera. A lot of people are not doing that.

People are panicking, I think. I'm looking from the outside looking in. So yeah, I actually, I made a little list of some excuses that lawyers who appear to have submitted AI fabricated authority have given.

A lot of lawyers eventually end up saying that the brief was written by a junior employee and not reviewed, sometimes including people who aren't lawyers, like people who either haven't passed the bar or people who haven't even gone to law school. This happened, a judge wrote an opinion with AI fabricated authority and ended up blaming it on their clerk. A lot of people say that it's copy and paste errors.

Some people say they have too many tabs open on their computer and like mix up the reporter citation or case names. Lots of people saying these are just typos. There's a school of thought, which is to claim that these are in fact real cases.

So some people have said these are just the double down. Oh, yeah, that's what we call it in the biz. The double down.

These are real lower court opinions or opinions with the same case name, but except reporter citation. There was one federal case, which was unfortunately a federal public defender, who was like, well, this case doesn't exist, but it was a criminal case. So the case name was People v.

Really Common Last Name. There was like, but that probably is a real case, People v. Common Last Name.

And the judge was like, are you talking about a specific case? And he was like, well, you know. And that went really poorly.

The soft double down. There could be.

Yeah, exactly. Yeah, it was exactly something like that. Some people had claimed that it was a real case, that they really did read, and then later couldn't find.

And then later changed to, well, I only read a portion of it, but then later couldn't find the excerpt.

Yeah. You're reminding me of the most famous one at the early stage. It might be the most famous because it was the first very public one, which was when the court said give us the citations.

You cited all this stuff, show us the citations, show us your authorities. And then he printed the authorities and filed them and said, here are the authorities, and the authorities had been made up themselves. That's like the one that, you know, that's the...

And he said, just give me the cases. And the thing said, okay, here are your cases. And he just kept going, you know, but that's the...

That might be the worst one, and it happened to be the first one, right? And then, and not in the sense of he was a prosecutor or a defense attorney or whatever, but just in terms of the, you know, double AI.

Yeah, and then there's, I mean, there's explanations for fake citations that are, you know, somebody said that they intended to italicize the language rather than put it in quotation marks with a pin site. You know, a couple other people have said that that language really does exist, but it's in another opinion. And then, well, it's in another opinion with the same case name, but, you know, a different docket number.

Yeah, all of these turn out much, much worse for the attorneys than the folks who just own up in the beginning to have a used AI. And the one that you're citing is across the country, right? They keep coming up, and where lawyers double down or try and claim they're a typos, those are used as the benchmark for how bad a lawyer's behavior is in the individual case.

I think what's hard about this, too, is that you can find out whether or not AI was used to draft something is a noble question, but you'd need forensic evidence from the person's computer, because it can be really tough to tell AI hallucinations from just really bad voiering. If you put infinite monkeys at infinite typewriters, one of them will eventually come up with just a botched citation to these cases.

Or something gets in a template. Attorneys use templates all the time. We use forms all the time.

I think what Kate had mentioned at the very beginning is like there's volume. This is a reality that is happening. And we're in an environment where it isn't really guided in terms of the bar association or professional guidance to say, I don't think the conclusion is never ever use AI.

That's going to sound somewhat strange in 10 or 20 years. But reading the thing that you submit, checking that it's accurate, like if an intern wrote a brief for you, I don't think one is to just take that brief and file it. I think you have to read it for many reasons.

So I wonder, do you think there should be some specific propositions that would prevent the worst outcomes from happening? I mean, I obviously reviewed the stuff and check it.

Yeah.

I mean, it's hard to say. I think to your point, it's just, what would allow the errors to be filed is just such an abdication of really basic ethical obligations that are well settled, which is read everything that you file, read all the cases that you're citing, both as a duty of candidate to the court and to your client. And these cases have been so publicized.

It's hard to think that this is a rule on the attorney side issue, where if there was just a little bit more guidance, I think things would be better.

I think one solution, I think, is almost something that you said earlier. I was thinking, like, especially at the superior court level, if a judge relies on something in a brief for their order, they need to have to at least orally state, I'm relying on this, you know, portion of the district attorney's case citing this. So at least you have something to hang your hat on for it or for an appeal, you know, when then that subsequent attorney can figure out that that was hallucinated.

That reminds me of a, there's a court of, there's a appellate principle, right, that comes up is the court's rulings have to be sufficiently clear to permit appellate review. And in this environment that we're in, right, if there's some concern, right, maybe, you know, for review, the court should be saying, that I'm relying on this legal principle, right, from this site. And that could be something that we ask for in particular cases.

Yeah.

Police AI Reports

Yeah, there's a whole way in which AI is also coming, you know, we're talking about, you know, attorneys and courts, but it's also coming to kind of large data operations and law enforcement, right? So we have body camera footage that has multiplied and kind of created huge amounts of volume in criminal cases for many good reasons. But the industry that produces the cameras is also selling a product that uses AI to turn videos, body camera videos, into police reports.

And there's so much judgment that can go into turning a video into a police report. For example, if someone is being aggressive or someone is approaching with an elevated voice or a bladed stance or whatever, the stuff that we see in police reports, that can absolutely be replicated if the instrument is being trained on police reports, right? It can take the stuff that's in the existing kind of bias world and then put it into their language model.

And then separately, it's like, I don't know what the need is personally to, I'm going to automate a police report from my body camera footage and then I'm going to edit that police report versus just writing that you walked up to somebody and you had an encounter and you discovered something, right? So, it's the value proposition ad versus the risks of made up police reports that aren't caught seems pretty high.

Yeah, I mean, I think the value ad is that it gives them more time to be doing more police work, right? Like, it allows them to funnel more folks through the system.

I don't know that the budget savings are true though, right? Because I think that is one of the ways they say, well, if you spend 20 hours a week on police report writing, this can make it five hours, but you still have to review everything and affirm that it's all true. You know, I saw one story where an officer said, you know, I have writer's block, and this helps me get through writer's block as I'm developing my police report.

And it just strikes me that, like, having writer's block over a police report doesn't make much sense to me. It's not like this creative endeavor, right? You're at a certain street, it was a certain time.

Yeah, I think, unfortunately, we're just not doing that in practice. I would be shocked if they are really going through and rewatching the video and confirming that it's accurate in practice. Like, there was, we were just talking about this before we started recording, but there was an opinion that came out of the Chicago ICE case today, where they were documenting just how many times ICE officers, including like the director of ICE, had lied to the court about their operations in Chicago.

And one of the things that the judge said was that there had been, you know, just a slew of incident reports that did not match the body cam footage. But one of the things that the judge saw on the body cam footage was officers using ChatGPT to write their incident reports. So it's just, yeah, I'm sure this is happening at all levels of law enforcement.

Right. And when you talk about the AI program wanting to please the user, you know, so the police officer wants to prove that somebody was aggressive or whatever the words used by that ChatGPT will fulfill that desire. Yeah, I was, we were also talking.

I mean, the CPDA, the California Public Defender Association sponsored this bill, SB 524, states that the first draft of the police report that's generated by artificial intelligence needs to be preserved, that there needs to be an audit trail. And at least on the police reports, the officer has to ignore, like every page of the police report has to acknowledge that it was created or assisted by artificial intelligence. Now, there, you know, we'll see if, if that's done, you know, the proof will be in the pudding, as it were.

But at least that's like something demanding transparency. And of course, the, you know, the body worn camera footage still exists. So to be compared.

But to your point, I mean, the AI, what I saw in the, you know, an example that we had, we were shown this draft one. The AI is definitely making editorial decisions in their use of words, you know, where, and again, this was a product where we saw a witness and they were using one solo witness using very, let's say, white standard English coming up to the cop and saying, this is what happened. And, you know, and that this person was maybe, you know, interacting with this family.

And I believe that the word that was used was, this person was harassing this family. And I thought that now that's an editorial choice, you know, that I didn't hear in the body camera footage.

Yeah, and then how do we impeach that, right? In terms of normal adversarial process, create some safeguards, right? There should be some safeguards in terms of your police officer.

The normal safeguard to an accurate police report is that one day you may have to answer for that police report in a court. That's the only, you know, protection, really, is that you might think. I mean, of course, there's like, you know, human protections and like concern, you know, for other human beings and dignity, obviously.

But the real structural, like, systemic protection in our system is that people have a right to confrontation, right, and they have a right to cross-examination. And if the report has editorial decisions that aren't accurate or hallucinations or other things that, you know, are kind of giving cover, the cross-examination is, oh, yeah, AI. Oh, yeah, and everyone says, oh, yeah, duh, we just use AI.

That's just what we do. AI must have gotten it wrong, you know, but I think it's everything's okay anyway, which is, you know, pretty concerning. I don't think saying, oh, well, the audit trail says you clicked a box saying it was accurate, right?

I don't know that it's going to land the same way. Then I wrote, you know, the person was struggling with me, so I put my hands in their pockets or something, which, you know, is then contradicted by the video. But we'll have to see how it unfolds.

And I think people just have to be super vigilant in all respects. Well, any closing thoughts, Carson, Kate?


Need for Vigilance

I mean, it's hard to say. Unfortunately, I think a lot of this is on the courts to be really vigilant about and to take seriously instances where people have submitted false statements because of AIUs. I think the vigilance point is just, again, I keep coming back to Kate's point about resources and just, you know, the idea that, you know, part of what is, you know, has always, of course, has always been, you know, the best representation.

You would be jeopardizing every single one of your cases, but it just, it takes so much longer to jeopardize an AI brief. Like it takes, it takes significantly more time to try and track down cases that don't exist or find unsupported principles in cases that don't exist. I think it really needs to be on the courts to be paying closer attention and taking seriously when people submit AI briefing.

All right. Well, that's all we'll talk about for today. That's all the time we have for today.

Thanks for joining us at Due Processing. Carson, thanks so much for joining us.

Thanks so much for having me.

Thanks.

Talk to y'all later. Bye.