Building the AI in ADR Newsletter
A New AI-Generated but Human-Curated Publication from the AAA-ICDR Institute™ and AAAiLab™
The AAA-ICDR Institute, through its AAAiLab, has launched a weekly publication that combines human editorial judgment with a generative AI–powered “staff writer” that produces summaries of recent news items, offering readers a view on developments in AI relevant to alternative dispute resolution (ADR). It’s one window onto how the American Arbitration Association-International Centre for Dispute Resolution® (AAA®) is thinking about this topic.
A snapshot of ADR-relevant AI news, with an AI assist
AI is important to the future of ADR. The AAA believes that. It has potential to improve efficiency, lower costs, and expand access to justice, among other things. And so we try our best to keep up with the always-on firehose of new developments in the field. We want to demo the latest tools, understand their technical underpinnings, and make sure we are employing them to good effect.
The AAA has a collaborative culture, and that shows in how enthusiastically we share AI news among ourselves: through working groups, all-hands emails, webinars, and more targeted dialog among subject matter experts. This newsletter began in that spirit, as an internal touchpoint for AAA staffers focused on AI. They saw benefit in having a 20,000-foot view on recent AI news. Now the AAAiLab is sharing the wealth with a like-minded external audience, whether they be arbitrators, mediators, practitioners, those working in court systems, policymakers, academics, or other ADR administrators.
The value here is in the human curation: Each week, the human editor (me) reviews hundreds of articles, press releases, and other sources identified using social listening platforms, RSS feeds, and other news aggregation tools. The aim is to surface content that might inform our own work. Focusing too strictly on ADR alone seemed limiting, though, so the scope encompasses legal services more broadly, regulatory developments, big-picture AI news (like new frontier models and major corporate events), and how those in other fields are thinking about the technology.
It was also important for the newsletter to be scannable, while providing more than a list of hyperlinked headlines. Including a 50- to 100-word summary for each item strikes a balance, highlighting what might be worth a deeper dive without overwhelming readers. That is where the AI staff writer comes in.
Given the subject matter and the AAA’s fluency in using AI, this was an opportunity to leverage the technology in a transparent and ethical fashion. Generative AI tools are adept at summarization (the AAA is already applying them that way in some of our own processes), and the task of summarization does not call for a tremendous degree of human creativity or expertise. So, why not employ an AI staff writer to craft summaries after reviewing the full text of the articles selected each week by the human editor? We are explicit about that, and we keep a human in the loop to bolster the accuracy, quality, and relevance of each edition.
You can subscribe to the newsletter and other (human-authored) posts from the AAA-ICDR Institute here:
The rest of this article details the prompting strategies that make the newsletter possible, as well as the ethics of AI authorship.
The workflow
I first explored using a Custom GPT to summarize and categorize the news items and generate a well-formatted newsletter—all in one prompt—but ran into issues with incomplete, inconsistent, and at times inaccurate outputs. I refactored my workflow as a Python script (which an LLM helped me write) that instead performs this work in small batches via OpenAI’s API, applies consistent stylistic rules to the outputs, and populates an HTML template. Before publishing, the human editor reviews, edits, and reorders the AI’s “draft.” Then the newsletter is published via Substack.
The summarization and categorization prompt
This is not a particularly complicated prompt. If delegating this task to a human, one might offer very similar guidance:
You are an AI that reads and summarizes full-text articles with originality and conceptual clarity. Follow these steps:
1. Read and deeply understand the article.
2. Mentally reframe the content as if you are explaining it to someone unfamiliar with the topic.
3. Write a single labeled line:
Summary: Provide a concise 60- to 80-word summary in your own words. Focus on the key takeaways and essential meaning, not the language used in the article. Do not echo or paraphrase original sentences or phrases — instead, rephrase the content conceptually as if writing from scratch. Avoid meta-commentary and do not refer to "the article" or "the story."
4. After summarization, categorize the article. Write 'Category:' followed by exactly one of:
- AI in ADR and Legal Services
- Generative AI and LLM Developments
- AI Regulation and Policymaking
- AI News from Other Fields
5. If multiple categories fit, choose the best match. Return the output in a JSON object with 'summary' and 'category' keys.
No extra JSON formatting, and no additional lines.Good prompting practices reflected here include: keeping the prompt succinct, breaking the task into steps, providing logic for handling edge cases, setting constraints (word count), and generating a well-formed output (in this case using JSON, a common format for hierarchically labeling and exchanging structured data) for further processing using more conventional (non-AI) object-oriented programming techniques.
It’s also worth noting that effective prompts need not be built from scratch. I use a specialized Custom GPT with access to a knowledge base of prompt engineering best practices to draft and then refine prompts for all manner of use cases, though similar results can be had by simply prompting a model via the generic LLM chat interface to fix specific issues, build out the logic, constrain how the AI responds, and provide few-shot examples and templates.
The linguistic style transfer prompt
Just as users can customize their browser-based AI interfaces with instructions to be included with every prompt, the OpenAI API allows you to set a “system” prompt programmatically. This is where I am specifying the desired writing style. I masked the numerical values—which were developed by asking the AI to analyze a collection of my own writing—but here are the criteria:
Apply the following style guide to every output you generate. Where exact compliance cannot be guaranteed, approximate as closely as possible.
Style Guide:
1) Maintain an average sentence length of ## words.
2) Keep an average parse tree depth of ##.
3) Achieve ## clauses per sentence.
4) Use a rich vocabulary (Herdan’s C = ##).
5) Keep a ## ratio of verbs to nouns.
6) Maintain part-of-speech diversity for an entropy of ##.
7) Strive for high semantic density (avg cosine similarity = ##).
8) Maintain high cohesion (avg adjacent sentence similarity = ##).
9) Aim for moderate topic variability (variance score = ##).
10) Keep balanced sentence length variability (coefficient = ##).
11) Insert narrative events (## per 1000 words).
12) Use frequent connectives (ratio = ##).
13) Resolve anaphora accurately (score = ##).
14) Use ## words per noun phrase on avg.
15) Use ## words per verb phrase on avg.
16) Maintain info density (Shannon entropy = ## per sentence).
17) Maintain avg sentiment = ##.
18) Vary syntax with a diversity ratio of ##.
19) Include temporal expressions at ## per sentence.
20) Maintain referential clarity (score = ##).
21) Maintain engagement (readability + sentiment variability = ##).Yes, an LLM can write convincingly like Hemingway because his style is so distinctive and the model probably consumed so much of his work during training. But it may have a harder time capturing the writing style of someone who is not so extraordinarily prolific. You can upload example text and ask the LLM to mimic that style but, from my testing, that approach can overfit to the subject matter and turns of phrase in the examples provided. A more generalized linguistic “fingerprint” that is independent of substance and specific wordings would be ideal. It took some prodding for an LLM to come up with this list of metrics, which includes both simple arithmetic (verb/noun ratio) and more complex evaluations (sentiment analysis). This approach can be computationally intensive for older LLMs—even prohibitively so. But, in my experience, GPT-4o, o3, and the like have shown themselves adept at applying it.
Thoughts on authenticity
Read enough AI news, and you see how academics, journalists, creatives, lawyers, and many others are wrestling with AI involvement in human authorship.
The way AI in ADR navigates that is through human oversight of the process and being forthright about how AI is being used. A real person scrutinizes the AI-generated content and revises it where appropriate. This is arguably no different from the role copy editors and fact-checkers play at a fully human-generated publication. And we’re acknowledging the extent of AI involvement at the outset of each edition.
There are certainly other defensible viewpoints on this issue. A news outlet might forbid the use of AI to generate content. A scholarly journal might place similar restrictions on their authors. A movie based on an AI-generated screenplay or using other AI production techniques might have those details taken into account by Golden Globes or Oscars voters.
But the goal of this newsletter isn’t to publish writing that hinges on originality or to win awards. And the human element remains in those stages of the workflow where it really moves the needle. Without AI, this would likely be at best a monthly newsletter, which would not be so helpful for such a fast-moving topic. And leveraging an AI staff writer frees the human editor to devote more effort to identifying interesting and relevant content. At first blush, readers might sense some strangeness in this approach, but we all may very well grow more comfortable over time with the application of AI where it matches or exceeds our own abilities while we focus human judgment and expertise—and time—where it matters most.