Production is free. Judgment isn't. Each playbook turns that idea into a method a team can run, with the argument, the steps, the files and the prompts written out in full.
Production is free. Judgment isn't. Every playbook here is a way to write the second half down.
Research standards
Every playbook is published like research. It carries a version number, cites its sources, grades every core claim by what supports it, includes a benchmark you can run, and states its limits in plain words.
Grade
Means
A · Experiment
A controlled experiment supports the claim directly.
B · Field data
Large-scale real-world data supports it.
C · Framework
An established peer-reviewed model or guideline supports it, without testing this exact claim.
P · Practitioner
It comes from MainArc's practice and is untested. Labelled every time.
No playbook contains invented statistics, client results or testimonials. Where a claim has no study behind it, it says so, and the benchmark explains how to test it. Corrections are made in the next version.
The 10x test
Nothing ships until it passes six questions. Every playbook here was run through them, and fixed where it failed.
#
Question
How a playbook passes
1
A thesis someone smart could disagree with
One sentence at the top, sharp enough to argue with
2
Still true if every model is replaced next year
Methods and principles, never rankings of models or tools
3
One thing to do on Monday
A named first step on the introduction, plus a setup prompt
4
One figure people will screenshot
A live tool or a figure that makes the argument on its own
5
Every number sourced and graded, nothing held back
Working-paper figures left out where the published version could not confirm them. No salaries or job counts. Role definitions graded untested.
MainArc Research · The Judgment Report
The Judgment Report
What the evidence says about making with AI, and where the value went.
Across controlled experiments and field data, AI help makes people much faster at the work that used to cost the most: drafting, coding, answering, image-making. The same body of research shows what that speed costs. Work drifts toward the same average. Demand for production falls. Help that does the thinking weakens the thinker. Machines that grade or act for us miss what people catch. This report gathers the strongest published evidence behind MainArc's sixteen playbooks into one argument: as making gets cheap, the value moves to judgment.
ReportMainArc ResearchVersion1.0Chapters6Charts10Sources32 citedMethod and sources →
Thesis
Production is free. Judgment isn't.
The evidence shows making getting cheaper and faster. What stays scarce is deciding what is good, what ships and what is cut.
How to read this report
Every number links to a published source. Every finding carries a grade for the kind of evidence behind it. MainArc ran no survey of its own and owns none of this data.
A · ExperimentB · Field dataC · Framework or estimateP · MainArc practice
Six numbers
40%
less time on professional writing tasks with an AI assistant, and 18% higher rated quality.
Each figure is restated as its source states it. Studies differ in task, population and date, so these numbers sit side by side without sharing a scale.
Contents
Making got cheap. Large, uneven gains in speed and quality, biggest for the least experienced.
Making is no longer the expensive part of creative work.
Controlled experiments and field data agree on the direction. With AI help, people finish production tasks faster and often better. The gains are large, uneven, and largest for the people with the least experience.
Chart 1 · Production gains, by studyGrades A and B · n = 95 to 5,172
Sources: Peng et al. [7], Noy and Zhang [1], Dell’Acqua et al. [8], Brynjolfsson et al. [9]. Findings from separate studies, not one scale. Reading: every study found a large gain in speed or output.
The cleanest test comes from writing. In a preregistered experiment, 453 college-educated professionals took on writing tasks from their own occupations, and half were given an AI assistant. Average time fell by 40% and rated quality rose by 18% [1]. Inequality between the workers narrowed.
Code shows the same shape. Developers with an AI pair programmer finished a set coding task 55.8% faster than those without [7]. At one customer support firm, 5,172 agents resolved 15% more issues per hour with AI help. Less experienced agents improved in speed and quality, while the most experienced saw small speed gains and small declines in quality [9].
Chart 2 · The jagged frontierGrade A · n = 758
Table viewShow the numbers
Where
Measure
Change
n
Grade
Inside the frontier
Tasks completed
12.2% more
758
A
Inside the frontier
Speed
25.1% faster
758
A
Inside the frontier
Quality
Significantly higher
758
A
Outside the frontier
Likelihood of a correct answer
19% lower
758
A
Source: Dell’Acqua et al. [8], preregistered field experiment with 758 consultants. One study; each row is its own measure. Quality inside the frontier was also significantly higher. Reading: the same tool helped or hurt depending on the task.
The gains have an edge. In a preregistered field experiment, 758 consultants with AI completed 12.2% more tasks, 25.1% faster and at higher quality, on work inside the tool’s capabilities. On one task chosen to fall outside them, AI users were 19% less likely to reach a correct answer [8]. Same firm, same tool, opposite results.
Who gains most matters for the chapters that follow. In fiction, writers who scored lowest on a creativity test gained most from AI ideas, with novelty up 10.7% and usefulness up 11.5% [2]. Help flattens the distance between the weakest and the strongest producers. It says nothing yet about who decides which output is good.
MainArc Research · What this means
P · Practitioner When making costs less, the scarce input is the decision about what to make, keep and cut. One economic framework treats AI as a fall in the cost of prediction, with judgment as its complement [10], a model graded C. MainArc builds on it: rebuild work around the decisions that remain, and write those decisions down.
Every shortcut pulls the work toward the same average.
The tools that make each piece better make the whole set more alike. The effect shows up in stories, in ideas, inside the models and across a large art platform.
Chart 3 · Better alone, more alike togetherGrade A · n = 293 writers, 600 evaluators
AI ideas offered
Table viewShow the numbers
AI ideas offered
Novelty of each story
Usefulness of each story
Similarity between stories
n
Grade
One
+5.4%
+3.7%
+10.7% of full range
293 writers, 600 evaluators
A
Five
+8.1%
+9.0%
+8.9% of full range
293 writers, 600 evaluators
A
Source: Doshi and Hauser [2]. One experiment, two measures on different bases, so each panel has its own axis. Reading: stories improved on average, and the set converged.
In an experiment with 293 writers and 600 evaluators, writers given AI story ideas produced stories rated more novel and more useful. With five ideas to draw on, novelty rose 8.1% and usefulness 9.0% against writing alone. The stories also grew more similar to each other, by 8.9% to 10.7% of the similarity measure’s full range [2]. The average story gained. The set lost range.
Ideation shows the same pull. In a study of 36 people, those using a general-purpose chatbot produced less distinct ideas than those using another creativity support tool [11]. One mechanism sits inside the models themselves. Preference tuning, the step that teaches a model to follow instructions, produced much less varied outputs than the same models trained without it [12].
Chart 4 · More output, less visual rangeGrade B · n = over 50,000 users
Table viewShow the numbers
Measure
Change
Population
Grade
Artworks made
+25%
Over 50,000 users, one platform
B
Favourites per view
+50%
Over 50,000 users, one platform
B
Visual novelty, peak and average
Fell
Over 4 million artworks
B
Content novelty, average
Fell
Over 4 million artworks
B
Content novelty, peak
Rose
Over 4 million artworks
B
Source: Zhou and Lee [13], over 4 million artworks from more than 50,000 users on one art platform. Observational. Arrows show direction only. Reading: more work and more approval, with less visual range.
Outside the lab the pattern holds. On an online art platform, with over 4 million artworks from more than 50,000 users, artists who adopted text-to-image tools made 25% more work and earned 50% more favourites per view. The visual novelty of their work fell, at the peak and on average. Average content novelty fell too, while peak content novelty rose [13]. More work, more approval, less visual range.
MainArc Research · What this means
P · Practitioner Sameness is the default output of shared tools. Distinctiveness now has to be directed on purpose, written into briefs and brand rules, and measured with your own judges before it drifts.
The market has started paying less for production.
Freelance platforms give the earliest view of demand. After generative tools arrived, posts for writing, coding and image work fell, and the work that remained asked for more.
Chart 5 · Demand for production workGrade B · two platforms, 116 skill clusters
Sources: Demirci et al. [3], Teutloff et al. [14]. Findings from separate studies, not one scale. The band is a reported range. Reading: production-heavy work lost demand fast.
On a large online freelancing platform, freelancers in the occupations most affected by a text generator saw fewer jobs and lower earnings after its release, with similar effects after image generators [15]. On a global platform, job posts for automation-prone writing and coding fell 21% within eight months, and posts for image creation fell 17% after image generators arrived. The automation-prone jobs that remained were more complex and paid more [3].
A study of 116 skill clusters found demand for substitutable skills such as writing and translation fell by 20% to 50% against trend. Demand rose only in complementary or unaffected clusters, and fell for novices even there [14].
Chart 6 · Exposure depends on the softwareGrade C · estimate across occupations
Table viewShow the numbers
Scenario
Jobs with over half their tasks exposed
Kind of evidence
Grade
Model with simple interfaces
Roughly 1.8%
Task-level estimate
C
Model plus complementary software
Just over 46%
Task-level estimate
C
Source: Eloundou et al. [16]. Each grid is 100 jobs; a filled square is a job with more than half its tasks exposed. An estimate of potential exposure, with no observed job change. Reading: the software layer, more than the model, sets the exposure.
Exposure depends on what gets built around the model. A task-level estimate found roughly 1.8% of jobs had more than half their tasks exposed with simple interfaces, rising to just over 46% once complementary software was counted [16]. That is potential exposure, an estimate with no observed job change behind it.
6 in 10Jobs in specialties introduced over eight decades · B · [17]
New titles emerged where innovations complemented what an occupation produced.
21%Wage premium for technical AI skills · B · 962 skills · [18]
A skill’s value peaked when paired with a skill of a different type.
History offers a counterweight. Linking eight decades of new job titles to census records, one study estimates that roughly six in ten current jobs are in specialties introduced over that period, and that new work formed where technology complemented an occupation’s output [17]. Across 962 skills on a freelancing platform, a skill was worth most in combination, and technical AI skills earned a 21% wage premium [18].
MainArc Research · What this means
P · Practitioner The paid work is moving from making to deciding. The new creative roles are judgment jobs wearing production titles. Define them by the decisions they own, and let whole teams make while a named person decides.
Answer-giving help raises performance while it is there and lowers it once it is gone. Confidence in AI travels with less critical thinking. And the machines asked to grade work agree with people mostly where people already agree.
Chart 7 · Help that does the thinkingGrade A · n = nearly 1,000 students
Show
Table viewShow the numbers
Tutor
Practice grades, with help
Exam grades, without help
n
Grade
General AI tutor
+48%
−17%
Nearly 1,000 students
A
Tutor built to give hints
+127%
No significant loss
Nearly 1,000 students
A
Source: Bastani et al. [4], field experiment in high school mathematics. One study, one scale. Reading: the general tutor raised grades while present and lowered them once it was gone.
In a field experiment with nearly a thousand high school students, a general AI tutor raised practice grades by 48% and lowered later unassisted exam grades by 17% against a control group. A tutor built to give hints instead of answers raised practice grades by 127% with no significant exam loss [4]. The design of the help decided whether the learning survived.
A survey of 319 knowledge workers, sharing 936 examples of AI use, found confidence in AI associated with less critical thinking, and confidence in one’s own skill associated with more [19]. It is self-reported and correlational. In a simulator study of 16 airline pilots, hand skills largely held up under automation, while tracking position, planning ahead and spotting instrument failures showed more frequent problems [20].
Chart 8 · Machine judges agree, and leanGrade A · n = about 3,000 votes; 80 questions
Sources: panel a, Zheng et al. [21]; panel b, Wang et al. [5]. Findings from separate studies, not one scale. Chat answers, compared system against system. Reading: a judge matches people on agreement, and a change of order alone moves it.
Machine judges reproduce the split. On about 3,000 expert votes, the strongest judge tested agreed with people 66% of the time, against 63% between people, when ties counted. Without ties the figures were 85% and 81%. Agreement rose from about 70% to nearly 100% as the gap between the two systems compared grew [21]. The judge is most reliable on the clear calls.
It is also easy to sway. Changing only the order of two answers let one system beat another on 66 of 80 questions [5]. The strongest judge gave the same verdict both ways round in 65% of cases with its default prompt [21]. Judges rated their own outputs above texts people rated as equal [22]. People differ too: experts agreed with each other far more than non-experts when rating creative work [23].
MainArc Research · What this means
P · Practitioner A machine judge can learn the floor your people share. The edge, where informed people split, still needs people, and those people need designed practice to grow into it. Write the checks first, audit the judge against your panel, and protect the reps that build judgment.
Agents get most steps right and still fail the job.
Benchmarks of web agents show a wide gap between single steps and finished tasks, a wide gap to people, and a weakness people show less often: being steered by deceptive design.
Chart 9 · Steps right, tasks failedGrade A · n = 812 and 2,350 tasks
Sources: Zhou et al. [24]; Deng et al. [6]. Findings from separate studies, not one scale. Snapshots from each paper; agents have improved since. Reading: a task is a chain, and the chain breaks.
In a realistic test environment of 812 tasks across four kinds of website, the best agent in the original paper completed 14.41% of tasks end to end. People, tested on a sample of 170 tasks, completed 78.24% [24]. Agents have improved since. The paper fixes the gap at one point in time.
Step accuracy hides the gap. On 2,350 tasks across 137 real websites, the best agent got 52% of individual steps right on held-out tasks and completed 5.2% of whole tasks [6]. The pages it read averaged over a thousand elements before filtering.
Chart 10 · Dark patterns steer agentsGrade A · n = 700 tasks; six agents
Sources: Cuvin et al. [25]; Ersoy et al. [26]. Findings from separate studies, not one scale. The first bar is drawn at its lower bound. Reading: deceptive design works on agents more often than on people.
Deceptive design works on agents, often better than on people. Across 700 tasks, dark patterns steered agents to the harmful outcome in over 70% of tested tasks, against a human average of 31%. More capable models were more susceptible, and prompting and guardrail defences did not reliably help [25]. Across six agents, a single dark pattern worked 41% of the time on average, and obstruction worked most often [26].
The people behind agents have limits of their own. In clinical software, the likelihood of accepting a reminder fell by 30% for each additional reminder per encounter, and by 10% for every five-point rise in the share of repeats [27]. A confirmation on every agent action risks the same fatigue.
30%Lower likelihood of accepting, per extra reminder · B · [27]
Clinical software, more than a million reminders.
10%Lower acceptance per five-point rise in repeats · B · [27]
The share of repeated reminders mattered as well as their number.
MainArc Research · What this means
P · Practitioner Agents are a second audience for every product, and a second pair of hands for every team. Design the layer they read, draw the line where they must hand back to a person, and spend people’s attention like a budget.
Knowing a person decided changes what the work is worth.
Give the same work a different label and people value it differently. Tell them it took longer and they judge it better. When anything can be generated, evidence of a person’s decisions becomes part of the work.
Evidence card · Same work, different storyGrade A · three studies
The labelHuman-made or AI-made30 paintings, labels assigned at random
Human-labelled paintings rated higher on liking, beauty, profundity and worth.
Bellaiche et al. [28] · two studies of about 150 people each · A
The labelPresented as human or AIBetween-subjects experiment
The same paintings were rated higher when presented as human-made.
This card shows direction only; effect sizes are in the sources. Paintings and poems rated by online and general-public panels; commercial deliverables were untested. Reading: the story of who made it, and how, is part of what people judge.
In two studies of about 150 participants each, 30 AI-generated paintings were randomly labelled as made by a person or by AI. The human-labelled paintings were rated higher on liking, beauty, profundity and worth [28]. In a separate experiment with 565 participants, paintings presented as human-made were rated higher than the same paintings presented as AI-made [29].
Effort carries value of its own. The same poem, paintings and armour were rated better and worth more when people were told they took longer to make [30]. In the painting studies, perceived effort raised liking and beauty more for human-labelled art than for AI-labelled art [28].
Infrastructure for recording process now exists. Content credentials bind signed, tamper-evident records of actions and sources to an asset, and by design make no judgment about the work itself [31]. They record what a tool logged. What a person chose, changed or rejected still has to be written down by a person.
MainArc Research · What this means
P · Practitioner When anything can be generated, evidence that a person decided is the scarce signal. A per-part decision record, saying what was generated and what a person chose, changed or rejected, carries more than a single made-with label. No study yet tests decisions as distinct from authorship or effort.
This is where the report stops reporting. Each thesis below is MainArc's conviction, graded P, built on the findings above and turned into a playbook a team can run. None of them is a measured result.
Thesis 01P · Practitioner
Judgment is the scarce input.
Production is free. Judgment isn’t. As making gets cheaper, the value of creative work concentrates in deciding what is good, what ships and what is cut.
Demand falls for production and holds for complementary skills. New creative roles are judgment jobs; hire and train for the decisions, and let tool skill come and go.
Answer-giving help weakens learning, and machine judges learn only the floor. Design practice for juniors, write checks a stranger could grade, and keep people on the splits.
The findings are graded A to C by the kind of evidence behind them. The step from finding to method is MainArc's, graded P, and every playbook carries a benchmark to test it on your own work.
How this report was built, and what it cannot claim.
The Judgment Report is a synthesis. It restates published findings, grades them, and separates them from MainArc's own convictions. It adds no data of its own.
How findings were chosen
Every finding comes from the graded evidence already published in MainArc's sixteen playbooks: peer-reviewed papers, conference proceedings, a technical standard and a small number of preprints. A finding made the report when it spoke to one of six questions: what AI help does to production, to range, to demand, to judgment, to agents and to the value of human work.
Numbers are restated as their source states them, in the source's own measure. Nothing is averaged, pooled or projected. Where findings from different studies share a chart, the chart says so. Where a source reports a direction without a single size, the chart shows a direction and no bar.
The grading scale
Grade
Means
A · Experiment
A controlled experiment supports the finding directly.
B · Field data
Large-scale real-world data, or an observational or qualitative study, supports it.
C · Framework or estimate
An established model, guideline, specification or estimate supports it, without testing this exact finding.
P · MainArc practice
MainArc's conviction from practice. Untested, labelled every time, never presented as data.
Values charted or called out in this report, by grade of the study behind them:
31A · Experiment
13B · Field data
2C · Framework
0P · Practitioner
Grade P appears only in the "What this means" notes and the seven theses, never in a chart.
What was checked again
Figures taken straight from the playbooks were already checked when each playbook was published. Every figure this report adds, mostly sample sizes, was checked again against the paper or its published abstract. Figures that appeared only in press releases or earlier working papers were left out.
Figure
Checked against
Sample of the writing experiment: 453 college-educated professionals; time down 40%, quality up 18% [1]
Published abstract
Writers and evaluators in the story experiment: 293 and 600; novelty and usefulness by condition; similarity as a share of the full range; gains for the least creative writers [2]
Expert votes behind the judge agreement figures: about 3,000 [21]
Paper abstract
Benchmark size: 812 tasks across four kinds of website; people tested on 170 tasks [24]
Paper text
Dark pattern benchmark: 700 tasks, over 70% for agents, 31% for people [25]
Paper abstract
What this report is not
No survey. MainArc ran no survey and holds no data of its own. Every number belongs to the researchers cited.
No meta-analysis. The studies differ in task, population, tools, country and date. Their numbers sit side by side, and are never combined into one figure.
Lab and field differ. Experiments isolate one task over a short time. Field data covers real work with less control. Both appear here, graded so you can tell them apart.
Snapshots age. Agent benchmarks and tool capabilities move fast. The figures describe the systems each paper tested.
Creative work is under-studied. Most evidence comes from writing, coding, support, consulting, art platforms and classrooms. Applying it to brand, design and product work is MainArc's inference.
Labour data is early. Demand evidence comes from online freelance platforms over short windows. Salaried teams may move more slowly, and short-run effects may not last.
The theses are conviction. Everything graded P is MainArc's view, built on the evidence and untested as stated.
Cite this report
APA
MainArc Research. (n.d.). The Judgment Report: What the evidence says about making with AI, and where the value went (Version 1.0). MainArc. https://mainarc.xyz
Corrections
Found an error, a stronger source or a newer result? Write to hi@0xdragoon.xyz. Corrections are made in the next version.
Every source cited in this report, numbered in order of first citation. Links go to the DOI or the publisher's page.
[1]Noy, S., & Zhang, W. (2023). Experimental evidence on the productivity effects of generative artificial intelligence. Science, 381(6654), 187-192. doi.org/10.1126/science.adh2586
[2]Doshi, A. R., & Hauser, O. P. (2024). Generative AI enhances individual creativity but reduces the collective diversity of novel content. Science Advances, 10(28). doi.org/10.1126/sciadv.adn5290
[3]Demirci, O., Hannane, J., & Zhu, X. (2025). Who is AI replacing? The impact of generative AI on online freelancing platforms. Management Science, 71(10), 8097-8108. doi.org/10.1287/mnsc.2024.05420
[4]Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakcı, Ö., & Mariman, R. (2025). Generative AI without guardrails can harm learning: Evidence from high school mathematics. Proceedings of the National Academy of Sciences, 122(26), e2422633122. doi.org/10.1073/pnas.2422633122
[5]Wang, P., Li, L., Chen, L., Cai, Z., Zhu, D., Lin, B., Cao, Y., Kong, L., Liu, Q., Liu, T., & Sui, Z. (2024). Large language models are not fair evaluators. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 9440-9450. ACL. doi.org/10.18653/v1/2024.acl-long.511
[7]Peng, S., Kalliamvakou, E., Cihon, P., & Demirer, M. (2023). The impact of AI on developer productivity: Evidence from GitHub Copilot. arXiv:2302.06590. arxiv.org/abs/2302.06590
[8]Dell'Acqua, F., McFowland, E., III, Mollick, E., Lifshitz, H., Kellogg, K. C., Rajendran, S., Krayer, L., Candelon, F., & Lakhani, K. R. (2026). Navigating the jagged technological frontier: Field experimental evidence of the effects of artificial intelligence on knowledge worker productivity and quality. Organization Science, 37(2), 403-423. Earlier version: Harvard Business School Working Paper 24-013 (2023). doi.org/10.1287/orsc.2025.21838
[9]Brynjolfsson, E., Li, D., & Raymond, L. (2025). Generative AI at work. The Quarterly Journal of Economics, 140(2), 889-942. doi.org/10.1093/qje/qjae044
[10]Agrawal, A., Gans, J., & Goldfarb, A. (2018). Prediction machines: The simple economics of artificial intelligence. Harvard Business Review Press.
[11]Anderson, B. R., Shah, J. H., & Kreminski, M. (2024). Homogenization effects of large language models on human creative ideation. Proceedings of Creativity & Cognition (C&C '24). ACM. arxiv.org/abs/2402.01536
[12]Kirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Hambro, E., Grefenstette, E., & Raileanu, R. (2024). Understanding the effects of RLHF on LLM generalisation and diversity. International Conference on Learning Representations (ICLR 2024). arxiv.org/abs/2310.06452
[13]Zhou, E., & Lee, D. (2024). Generative artificial intelligence, human creativity, and art. PNAS Nexus, 3(3), pgae052. doi.org/10.1093/pnasnexus/pgae052
[14]Teutloff, O., Einsiedler, J., Kässi, O., Braesemann, F., Mishkin, P., & del Rio-Chanona, R. M. (2025). Winners and losers of generative AI: Early evidence of shifts in freelancer demand. Journal of Economic Behavior & Organization, 235, 106845. doi.org/10.1016/j.jebo.2024.106845
[15]Hui, X., Reshef, O., & Zhou, L. (2024). The short-term effects of generative artificial intelligence on employment: Evidence from an online labor market. Organization Science, 35(6), 1977-1989. doi.org/10.1287/orsc.2023.18441
[16]Eloundou, T., Manning, S., Mishkin, P., & Rock, D. (2024). GPTs are GPTs: Labor market impact potential of LLMs. Science, 384(6702), 1306-1308. doi.org/10.1126/science.adj0998
[17]Autor, D., Chin, C., Salomons, A., & Seegmiller, B. (2024). New frontiers: The origins and content of new work, 1940-2018. The Quarterly Journal of Economics, 139(3), 1399-1465. doi.org/10.1093/qje/qjae008
[18]Stephany, F., & Teutloff, O. (2024). What is the price of a skill? The value of complementarity. Research Policy, 53(1), 104898. doi.org/10.1016/j.respol.2023.104898
[19]Lee, H.-P., Sarkar, A., Tankelevitch, L., Drosos, I., Rintel, S., Banks, R., & Wilson, N. (2025). The impact of generative AI on critical thinking: Self-reported reductions in cognitive effort and confidence effects from a survey of knowledge workers. Proceedings of CHI '25. ACM. doi.org/10.1145/3706598.3713778
[20]Casner, S. M., Geven, R. W., Recker, M. P., & Schooler, J. W. (2014). The retention of manual flying skills in the automated cockpit. Human Factors, 56(8), 1506-1516. doi.org/10.1177/0018720814535628
[21]Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., & Stoica, I. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Datasets and Benchmarks Track. arxiv.org/abs/2306.05685
[23]Kaufman, J. C., Baer, J., Cole, J. C., & Sexton, J. D. (2008). A comparison of expert and nonexpert raters using the Consensual Assessment Technique. Creativity Research Journal, 20(2), 171-178. doi.org/10.1080/10400410802059929
[24]Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., Alon, U., & Neubig, G. (2024). WebArena: A realistic web environment for building autonomous agents. International Conference on Learning Representations (ICLR 2024). arxiv.org/abs/2307.13854
[26]Ersoy, D., Lee, B., Shreekumar, A., Arunasalam, A., Ibrahim, M., Bianchi, A., & Celik, Z. B. (2026). Investigating the impact of dark patterns on LLM-based web agents. IEEE Symposium on Security and Privacy (SP 2026). arxiv.org/abs/2510.18113
[27]Ancker, J. S., Edwards, A., Nosal, S., Hauser, D., Mauer, E., Kaushal, R., & HITEC Investigators. (2017). Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Medical Informatics and Decision Making, 17, 36. doi.org/10.1186/s12911-017-0430-8
[28]Bellaiche, L., Shahi, R., Turpin, M. H., Ragnhildstveit, A., Sprockett, S., Barr, N., Christensen, A., & Seli, P. (2023). Humans versus AI: Whether and why we prefer human-created compared to AI-created artwork. Cognitive Research: Principles and Implications, 8, 42. doi.org/10.1186/s41235-023-00499-6
[29]Ragot, M., Martin, N., & Cojean, S. (2020). AI-generated vs. human artworks. A perception bias towards artificial intelligence? Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (CHI EA '20), 1-10. ACM. doi.org/10.1145/3334480.3382892
[30]Kruger, J., Wirtz, D., Van Boven, L., & Altermatt, T. W. (2004). The effort heuristic. Journal of Experimental Social Psychology, 40(1), 91-98. doi.org/10.1016/S0022-1031(03)00065-9
[32]Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., Teevan, J., Kikin-Gil, R., & Horvitz, E. (2019). Guidelines for human-AI interaction. Proceedings of CHI '19. ACM. doi.org/10.1145/3290605.3300233
A MainArc playbook. Give your team and its AI agents your taste, your voice and your standards in writing. Then anyone can make, and only the right work ships.
PlaybookNo. 01Version1.0Read16 minSources3 citedByMainArc ResearchEvidence and method →
Thesis
Taste that lives in one person's head becomes a queue. Written down, it becomes a system anyone, and any agent, can work from.
Do this Monday
Paste the setup prompt. Then collect five pieces of work you would sign and five you would never ship, with one line each on why.
A gate check, built only from the rules in this playbook. It is live: type a sentence, and gates one and two run on it as you write.
Gate checkRunning
1 Claims
·
2 Voice
·
3 Taste
Person
4 Distinct
Person
5 Cut
Person
Agents run gates 1 and 2. People own 3 to 5.See the gates
4Stages
5Gates
3Agent role files
7Setup steps
What this is
A good model can draft the essay, the deck, the landing page and the launch thread before lunch. Making things stopped being the hard part. The hard part now sits in one decision, repeated hundreds of times a week: is this good enough to carry our name?
Most teams answered with volume. Output went up and the work started to blur, because the same models trained on the same internet hand the same average back to everyone who asks.[1][2] The taste is still there. It lives in one or two heads, and every piece waits in line for them.
A Judgment Layer writes that taste down, in a form people and agents can both read and apply. Anyone on the team can make. The layer decides what survives. Every cycle adds to it, so the bar gets easier to hold and harder to copy.
Production is free. Judgment isn't. That line is the whole playbook.
Principles
Six rules the layer keeps. When a piece of work breaks one, the work changes, not the rule.
01
Written, or it does not exist
Taste that lives in one head becomes a queue. Write it down so people and agents can apply it without asking.
02
Evidence over adjectives
Clean, bold and premium cannot be checked. A reference with one line on why can.
03
Every agent has a job
Each role file says what it reads, what it checks, what it may change and when it asks a person.
04
Range before polish
Making is cheap. Spend it on many directions before refining the first plausible one.
05
People own the last cut
Agents run the first two gates. A person decides what carries your name.
06
Rejected work never sets rules
Only what held up goes back into the ground. The smallest change that captures it.
How to use this playbook
01
Read in order
Ground, Make, Cut, Compound. Each stage assumes the one before it.
02
Copy the files
Every role file and prompt has a Copy button. Paste them into your agent as they are.
03
Run one real brief
The setup ends with a brief you actually need this week. Theory without a cut teaches nothing.
04
Find anything
Press ⌘K, Ctrl K or / to search sections. Every section has its own address, such as #gates.
There are no logos or case studies here. Every step is written out in full, so the method can be judged on its own.
When the whole team judges from the same written standard, the distance between a draft and something worth shipping gets short.
A founder has a hunch for the launch. An operator needs a deck by Thursday. A marketer wants twenty hooks to test. With AI, each of them can make the thing themselves, today.
Note
That speed needs a shared bar. If every person and every agent judges by a different standard, more output brings more noise.
A Judgment Layer runs four stages. People and agents read from the same written ground at every one.
Fig 0.1The loopPress to index one step01 Ground · loading
01 Ground · hopper02 Make · press03 Cut · gauge gate and reject tray04 Compound · lift and return chuteGround supplies the standard. Make explores. Cut decides. Compound feeds what held back. One blank travels the loop at a time; in this sample run two of every three stop at the gate, and what holds rides back into the hopper.
#
Stage
What happens
01
Ground
Write down the standard. References, visual rules, voice, banned language and claims policy, kept current so people and agents know what good looks like here.
02
Make
Explore widely. Turn one brief into many directions. Making is cheap now, so spend it on range.
03
Cut
Decide what carries your name. Every direction faces the same gates. Most stop here, and that is the point.
04
Compound
Feed the standard. What held up becomes a new reference or rule, and the next cycle starts from sharper ground.
Some work has been done well a thousand times. Some has never been done at all. They need different handling.
Repeatable work: start from a reference
A launch post, an investor update, a pricing page. Strong examples already exist. The team picks the reference that fits, and AI rebuilds it in your voice, on your visual system, under your rules.
Frontier work: discover the shape
A new product category. An essay arguing something nobody has argued yet. There is no reference to borrow, so prediction fails. A person sets a direction, AI produces options, a person picks. Then again. This is where taste does most of its work.
Define what good looks like, well enough that a stranger, or an agent, could apply it.
Note
Ask a team what good looks like and you get adjectives. Clean. Bold. Premium. Adjectives cannot be checked, so every agent guesses, and the guesses drift.
The Taste layer replaces adjectives with evidence. Two folders: work you would sign, and work you would never ship. Each piece gets one line on why. The why is the asset. A folder of screenshots without it is a mood board.
Then the visual decisions get pinned down as values: palette, type scale, spacing, image style, the motion you allow. Design tools, prototype code and agents all read the same names and the same allowed values.
A vague note, made checkable
BeforeMake it feel more premium.
After
One typeface fewer. Wider margins. A single accent colour. No gradients.
If the rules cannot express the direction someone wants, the team decides what to add. That decision is judgment, and it gets written down too.
Folder · judgment/
judgment/
taste.md # rules a stranger could applyreferences/keep/# work you would sign, one line on whykill/# work you would never ship, one line on why
voice.md # rhythm, register, point of view
anti-language.md # words and constructions you never use
claims.md # what you may and may not assertagents/# one role file per agent
log.md # what each cut taught
A piece can hit every visual rule and still read like a press release. Voice drifts quietly, one borrowed phrase at a time.
The Voice layer writes down sentence rhythm, register and point of view. Then it does the more useful half: it lists what you never write. Banned words. Banned constructions. The tells that mark a page as unedited machine output.
anti-language.md · starter10 entries
leverage, as a verb
seamless
unlock
game-changer
in today's fast-paced world
delve
elevate your
it's not just X, it's Y
cutting-edge
furthermore
Write your own from your own drafts. Step 4 shows how.
The claims policy
The Voice layer also holds claims.md. Every number traces to a source. Every quote belongs to the person who said it. No outcome gets credited to you that you cannot show. Nothing is promised about a product that does not exist yet.
It is the cheapest gate to write and the most expensive one to skip.
Agents draft, rewrite and review across every tool you use. Without a job description, each one decides for itself what matters.
Each role file says what the agent reads, what it checks, what it may change, and when it must stop and ask a person. Three roles cover most teams. Open one, copy it, and edit the limits to fit how you work.
agents/taste-editor.mdGates 3 to 4
Ranks directions against your references and marks the ones that fall short.
Show fileHide file
# taste-editor
## Job
Hold the visual and structural bar. Rank directions
against our references. Recommend what to cut.
## Reads
- judgment/taste.md
- judgment/references/keep/ and references/kill/
- the brief
## Checks
- Does this direction sit beside the work in keep/
without looking weaker? Name the closest piece.
- Does it resemble anything in kill/? Name the piece.
- Does it follow every rule in taste.md?
Quote the rule it breaks.
- Could a competitor publish it unchanged?
## May
- Rank directions, best first, one line each on why
- Mark a direction for cut, citing a rule or reference
- Propose a new rule, in log.md only
## May not
- Approve anything for ship
- Edit taste.md or the references directly
- Apply a preference that is not written down
## Ask a person when
- Two directions tie
- A direction has no precedent in keep/ (frontier work)
- A rule looks wrong for this brief
agents/voice-editor.mdGate 2
Removes banned language and machine tells. Fixes sentences, never meaning.
Show fileHide file
# voice-editor
## Job
Keep every sentence in our voice. Remove banned
language and the tells of unedited machine output.
## Reads
- judgment/voice.md
- judgment/anti-language.md
## Checks
- Every word and construction in anti-language.md
- Rhythm: sentence length varies, openings vary
- Register and point of view match voice.md
- Filler openers, stacked adjectives, hedges that
say nothing, summaries of what was just said
## May
- Rewrite at the sentence level
- Cut words, sentences or paragraphs that add nothing
- Propose additions to the kill list, in log.md only
## May not
- Change what a sentence claims
- Add facts, numbers, names or quotes
- Smooth over anything the claims-auditor flagged
## Ask a person when
- A fix would change the meaning
- The brief needs a register voice.md does not cover
agents/claims-auditor.mdGate 1
Traces every number, name and quote to a source. Cuts what it cannot trace.
Show fileHide file
# claims-auditor
## Job
Make sure everything we publish is true and traceable.
## Reads
- judgment/claims.md
- the sources attached to the brief
## Checks
- Every number has a source a reader could find
- Every quote is attributed to the person who said it,
in their words
- No outcome is credited to us that we cannot show
- No capability is claimed for something not built yet
- No real person's name sits on a line they did not say
## May
- Flag a claim and say what is missing
- Cut a claim that cannot be traced
- Suggest a smaller claim that is true
## May not
- Turn a false claim into a vague one and pass it
- Go looking for a source to fit a sentence after the fact
- Approve anything for ship
## Ask a person when
- A source exists but is private or under NDA
- A claim is true but cannot be shown publicly
Every role reads the same Taste and Voice layers. Shared ground keeps each role file short, and short role files get followed.
Fig 1.3Shared layers, defined rolesPress to take the next layerEssay · ready
Top drawer · Taste: references, visual rulesMiddle drawer · Voice: voice, kill list, claimsBottom drawer · Agent: roles, limits, when to askFrames, left to right: essay, deck, landing page, launch threadEvery format draws from the same ground. One cabinet, three drawers, and each frame takes the same three pieces.NextRange before polish →02 Make · 2.1
Range before polish
One brief, many directions. Range is cheap now. Spend it.
With the ground written down, nobody argues about fonts or tone. The team can spend its attention on the real question: what would make this land?
One brief can become eight directions in an afternoon. Different angles, structures, openings. Try the range before you polish the first plausible answer, because the first plausible answer is the one your competitors got too.[3]
Note
A direction is complete enough to judge. The argument, the structure, the first screen. Everyone reacts to the same real thing.
Fig 2.1One brief, eight directionsMove to weigh · press to choosemade 0 of 8 · 0 chosen
Card on the head: the one briefTurret: one die per direction, A to HPointer: weighing the tilesMarked: chosenEach direction is cheap. Choosing among them is the work. Example launch: A Contrarian claim, B Customer story, C Data first, D Founder letter, E Teardown, F Before and after, G Single image, H FAQ. The press makes all eight in seconds; the pointer takes longer to choose one.NextLet more people make →02 Make · 2.2
Let more people make
Nobody waits for the one person with taste to find a free afternoon.
Founder
Makes a hunch visible before it turns into a meeting.
Operator
Drafts the deck instead of briefing someone else to.
Marketer
Brings the objection they hear on every call.
Support
Brings the question customers keep asking.
Each of them makes against the written ground, and brings something real to the cut.
Every piece is a reference away
A screenshot, a link or a rough paragraph is enough to start. The agent rebuilds it from your rules, so the first draft already sounds and looks like you.
Example prompt
Here is a launch post I admire. Rebuild it for our product, in our voice, under our claims policy. Give me four openings.
Judgment decides what carries your name. Every direction faces the same five gates, in this order.
#
Gate
Check
Owner
1
Claims
Is every claim true and traceable? A false claim ends the review here, however good the rest is.
Agent
2
Voice
Does it sound like us? A banned word, construction or tell sends it back.
Agent
3
Taste
Would it sit next to the work in keep/ without embarrassment?
Person
4
Distinct
Could a competitor publish it unchanged? If yes, it is not finished.
Person
5
Cut
What can go? Remove it. Then ask once more.
Person
A direction that clears all five moves to ship. One that fails goes back to Make, or stops for good.
Fig 3.1Eight in, one outPress to release a token8 in the magazine
Gates 1 to 5: Claims, Voice, Taste, Distinct, CutTokens A to H: the eight directions, D is the founder letterRed trays: stopped at that gateBlue tray: shipsSeven of eight stop at a gate. A high cut rate means the gates work. Each gate is set tighter than the one before, and a token that does not fit drops into the tray under it.
Note
More people can make. A person still decides what carries your name.
Some directions teach you something. Keep the lesson, and only the lesson.
A structure that worked. A phrase that kept failing. A visual move that made everything around it look better. First check whether an existing rule already covers it.
Add a rule only if future work should follow it, and make the smallest change that captures it.
What you learned
Where it goes
A piece that set the bar
references/keep/
A phrase that failed twice
anti-language.md
A visual value worth reusing
taste.md
A claim that almost slipped through
claims.md
A choice that suits only this piece
Stays with the piece
Note
Rejected directions never become rules. Every promoted rule is one less argument next time.
What held up goes back into the ground, so the next cycle starts sharper.
Shipping closes one piece. Ten minutes after, the team writes down what the cut taught: the reference that set the bar, the phrase that got killed three times, the claim that almost slipped through.
It goes into log.md first, then into the ground if it earns a place. Finished work disappears into the archive. Written judgment stays, and gets used on every brief after it.
log.md · one entry
## Pricing announcement
Shipped: D, founder letter
Cut: 7 of 8
Learned
- Data-first openings failed Claims twice. No source.
- "Simple, transparent pricing" failed Voice. Add to kill list.
- Founder letter set the bar. Move to references/keep/.
Promoted
- anti-language.md: + simple, transparent
- references/keep/: + pricing-founder-letter
Models will keep getting better at making. That raises everyone's floor by the same amount.
What a better model cannot raise is a standard only your team has written down. A new reference becomes the bar. A banned phrase never comes back. A rule that held becomes the default.
The next brief starts from all of it. People and agents begin with more of your judgment already in hand, and the team spends its attention on the next hard question.
Fig 4.2The ground only growsPress to run the next cutcycle 1 · 3 layers · reads 9.0
Cycle 1: Voice, Taste, AgentsCycle 2 adds a keep/ reference and a banned phraseCycle 3 adds a claims rule and a taste ruleNewest layers, and the needle reading the stackWhat one cut adds becomes the base for the next. Each cut slides new shims onto the stack and the dial reads it taller. Nothing is taken off.NextSet up your layer →Run it
Set up your Judgment Layer
You need a folder, an AI agent that can read and write files, and an honest afternoon. Any coding agent that can read and write files will do.
The walkthrough follows one case from start to finish. You run a small company. You have taste but no editor, and most of your writing and design now starts with AI. Seven steps take you from an empty folder to a standard your agents enforce.
Setup prompt · step 1
Set up a Judgment Layer in this project.
1. Create a folder called judgment/ containing taste.md, voice.md, anti-language.md, claims.md, log.md, references/keep/, references/kill/, and agents/. If I paste role files, save them in agents/. If not, write taste-editor.md, voice-editor.md and claims-auditor.md, each with these sections: Job, Reads, Checks, May, May not, Ask a person when.
2. Interview me one question at a time. Ask for five pieces of work I would sign and five I would never ship, and why for each. Push until every why is specific enough for a stranger to check.
3. Draft taste.md and voice.md from my answers only. Do not invent preferences I did not state.
4. Write claims.md with these defaults: every number needs a source, every quote needs its speaker, no outcomes we cannot show, no claims about things not built yet.
5. Show me everything you wrote and ask what is wrong with it.
Paste into your agent in an empty folder
01
Paste the setup prompt
Your agent creates the folder, the files and the three role files, then interviews you one question at a time.
02
Collect ten references
Five pieces of work you would sign. Five you would never ship. Drop them into keep/ and kill/ with one line each on why. The kill folder often teaches more than the keep folder.
Example prompt
Here are five landing pages I would sign and five I would never ship. For each, ask me why until the answer is specific enough to check.
03
Turn references into rules
The agent reads your why lines and proposes rules a stranger could apply. You strike the ones that are wrong. A rule that cannot be checked is still an adjective.
Example prompt
Read references/. Propose ten rules for taste.md that a new hire could check without asking me. Show the reference each rule came from.
04
Write your kill list
Paste three of your old drafts. The agent finds the words and constructions you lean on, plus the usual machine tells. You decide what goes into anti-language.md.
Example prompt
Read these three drafts. List every phrase that sounds generic or machine-made, with the line it came from. I will mark which ones to ban.
05
Install the agent roles
Copy the three role files from the Agent layer into agents/. Edit the limits to match how your team works, and write down who approves at gates three to five. If the answer is always you, write it down anyway.
06
Run one real brief
Pick something you need this week. Ask for six directions. Run them through the gates. Ship the one that survives, and notice how many did not.
Example prompt
Brief: announce our new pricing. Make six directions. Run voice-editor and claims-auditor on each, then show me what failed and why.
07
Close the loop
After it ships, spend ten minutes on log.md. Promote anything that earned a place. The next brief starts from sharper ground.
Example prompt
Read today's cut. Propose at most three changes to the ground, one line each on why. Leave everything else alone.
Grades describe the kind of evidence behind a claim, not how much we like it. Claims graded P come from practice and are labelled every time. Numbers link to the reference list below.
2A · Experiment
0B · Field data
0C · Framework
3P · Practitioner
Claim
Grade
What supports it
Source
Shared AI tools pull creative work toward the same average.
A · Experiment
In a 293-writer experiment, AI ideas made individual stories better rated but more similar to each other [1]. A 36-person study found people produced less distinct ideas with a general-purpose chatbot than with another creativity tool [2].
Exploring several directions before choosing beats refining the first one.
A · Experiment
Designers who made prototypes in parallel produced better-performing and more varied work than those who iterated in series, with the same time and feedback [3].
Written references and rules make taste transferable to people and agents.
P · Practitioner
From MainArc practice. No controlled study yet tests written taste standards against unwritten ones. The benchmark on the next page is designed to.
Practice
Running gates in a fixed order catches more problems than one general review.
P · Practitioner
From MainArc practice. Untested against a single-pass review.
Practice
Writing corrections down as rules compounds over time.
P · Practitioner
From MainArc practice. Plausible, and measurable with the send-back rate below, but unmeasured so far.
Practice
References
[1]Doshi, A. R., & Hauser, O. P. (2024). Generative AI enhances individual creativity but reduces the collective diversity of novel content. Science Advances, 10(28). doi.org/10.1126/sciadv.adn5290
[2]Anderson, B. R., Shah, J. H., & Kreminski, M. (2024). Homogenization effects of large language models on human creative ideation. Proceedings of Creativity & Cognition (C&C '24). ACM. arxiv.org/abs/2402.01536
[3]Dow, S. P., Glassco, A., Kass, J., Schwarz, M., Schwartz, D. L., & Klemmer, S. R. (2010). Parallel prototyping leads to better design results, more divergence, and increased self-efficacy. ACM Transactions on Computer-Human Interaction, 17(4), Article 18. doi.org/10.1145/1879831.1879836
Grades
A controlled experiment. B large-scale field data. C an established peer-reviewed model or guideline, not a direct test of the claim. P practitioner claim from MainArc's work, untested.
Does the layer make your work better and more like you? Run a blind comparison before and after, with a small panel, and let agreement tell you how well your standard is written.
Instruments
01
Pairwise comparison
Reviewers see two outputs from the same brief, one made with the layer and one without, in random order with labels removed, and pick the one they would ship.
02
Scales, 1 to 5
Each output is rated on voice fit, claims accuracy and distinctiveness.
03
Rationale
One sentence per choice, in the reviewer's words. These become new rules.
04
Similarity
Embed every output and measure average pairwise similarity across briefs. Rising similarity is the homogenization warning in [1].
Study design
Element
Setting
Briefs
20 real briefs from the last quarter
Outputs
Two per brief: with the layer, without it
Panel
5 reviewers: 2 from the team, 3 from the audience
Analysis
Win rate per condition, ranked with Bradley–Terry. Agreement with Kendall's W.
Cadence
Once at setup, again after 8 weeks
Reading the results
If layer outputs win most comparisons and similarity holds or falls, the layer is working.
If reviewers disagree a lot, the standard is not written clearly yet. Read their rationales and turn the splits into rules.
If similarity rises, your references are too narrow. Add range to keep/ before adding rules.
Running signals
Signal
What to watch for
Send-back rate per gate
Should fall over time as rules land
Repeat failures
The same reason twice should become a rule within a week
Rules added and deleted
A living standard changes every few weeks
Note
Thresholds here are starting points to tune, not findings. A small panel shows direction and disagreement. It does not prove significance, and nothing on this page should be reported as if it did.
How this playbook was built, where it is weak, and how to cite it.
How this playbook was built
Built from MainArc's practice and from published research on AI and creative homogenization. It is a practitioner method, not a study. Claims are graded on the evidence page by what supports them.
Limitations
Most of the method rests on practice, not experiment. Three of five core claims are graded P.
The homogenization research [1, 2] studied short stories and ideation tasks, not brand or product work.
The benchmark uses a small panel. Five reviewers can show direction and disagreement, not statistical certainty.
The gate checker on the introduction page is a pattern matcher. It catches listed words and unsourced figures, not judgment.
Open questions
How much of a team's taste can be written down before rules start fighting each other?
Does a written standard reduce homogenization across teams, or only within one?
Which gates should agents own as models improve, and which should never move?
Cite this playbook
APA
MainArc Research. (n.d.). Build your Judgment Layer (Playbook No. 01, Version 1.0). MainArc. https://mainarc.xyz
BibTeX
@misc{mainarc_judgment,
author = {{MainArc Research}},
title = {Build your Judgment Layer},
howpublished = {MainArc Playbook No. 01, Version 1.0},
url = {https://mainarc.xyz}
}
Corrections
Found an error, a stronger source, or a result from running the benchmark? Send it through mainarc.xyz. Corrections are made in the next version.
About this playbook
This playbook is published by MainArc, a creative intelligence lab for leadership and product teams.
We do three things. We research how AI is changing creative work through long-form reports, design teardowns, and operator essays. We design the workflows, prompt systems, review loops, and decision frameworks that teams actually run on. And we build the work the strategy points to: brand systems, interfaces, product prototypes, internal tools, and launch concepts.
One operator. One AI agent. No layers in between. Every engagement runs under strict NDA. No case studies, no client logos, no shared work.
If this research resonates with a problem your team is facing, start with an audit. One week. A clear map of where AI fits, where it doesn't, and what to build next.
This is independent research. Not financial advice. Not career advice. Not the final word on anything. It's one person's perspective shaped by building, shipping, and paying attention. Do your own thinking. Challenge the ideas. Disagree with specifics. Build on what resonates. Discard what doesn't.
House rules
What doesn't belong here: hate, trolling, bad-faith takedowns, or drive-by takes from people who've never built anything. If you disagree, show your work. Builder's critique. Everyone else just comments.
A MainArc playbook. The Infinite Canvas, made practical: when generative UI fits, how to constrain it, and how to review output that is never the same twice.
PlaybookNo. 02Version1.0Read15 minSources3 citedByMainArc ResearchEvidence and method →
Thesis
In generative UI you stop designing screens and start designing the space every screen comes from. The rules carry your taste, or nothing does.
Do this Monday
Run one surface through the fit check. If it says compose or hybrid, write contracts for the six components it uses most.
A composer, built from the vocabulary and rules in this playbook. It is live. Pick an intent, regenerate it, then switch the rules off and watch what the same system makes without them.
Composer · sample data
Ask
Rule checks
Show planHide plan
4Stages
7Composition rules
5Rubric checks
4Files to write
What this is
For thirty years, someone designed every screen before you opened the app. Generative UI composes parts of the interface at runtime, from what the user asks, their context and their history.[1] The Infinite Canvas argued what that means for design: you stop designing screens and start designing the space every screen comes from.
This playbook is the how. Which parts of a product should compose themselves. How to give the system a vocabulary and rules. How to review something that never renders the same way twice. And how to recover when it guesses wrong.
Components are the vocabulary. Rules are the choreography. The model performs. The performance changes every time. It should still be recognisably yours. Generation makes screens free. Deciding what any screen is allowed to be is the judgment that stays scarce.
Principles
Six rules for any surface that composes itself. When a generated view breaks one, the rule wins.
01
Compose views, never parts
The system picks and arranges registered components. It never invents a component, colour or control at runtime.
02
Some flows stay fixed
Money, cancellation, consent and anything destructive keep a designed screen. Generation stops where trust is at stake.
03
Echo the intent
Every generated view says what it thinks you asked. That one line makes a wrong guess cheap to fix.
04
Same question, same structure
Variation lives in order and density. The vocabulary and the hierarchy stay put, so people learn the product.
05
Review the generator
You cannot approve a screen that does not exist yet. You approve the rules, against hundreds of samples.
06
Design the miss
In generative UI the wrong interface is the error state. Recovery gets the same care as the answer.
How to use this playbook
01
Play with the composer
Every idea on this page shows up in it. Rules off is the fastest argument for the rest.
02
Read in order
Vocabulary, Choreography, Review, Recover. Each stage assumes the one before it.
03
Start with one surface
One analytics view, one help panel, one search page. Never the whole product at once.
04
Find anything
Press ⌘K, Ctrl K or / to search. Every section has its own address, such as #rules.
Playbook 02 in a set. Playbook 01, Build your Judgment Layer, covers how a team writes down its taste. This one covers what happens when that taste has to run inside the product.
Fixed screens give way to composed ones in stages. Most products sit at stage two. The right stage for yours depends on the category.
Fig 0.1From fixed screens to the infinite canvasMove to pick · tap to compose02 partial · most products here
Cast plate: fixed, designed screensPins: the part composed at runtimeRed pin: stage 02, where most products sit01 Augmentation · 02 Partial · 03 Hybrid · 04 Generative first · 05 Generative nativeApproximate. Categories move at different speeds. Each die is one stage: the cast plate holds what is designed, the pins hold what composes at runtime.
#
Stage
What it looks like
01
Augmentation
AI helps people build fixed screens faster. Every screen is still designed.
02
Partial generation Most are here
Parts of the interface compose at runtime. Dashboards built from a question, settings that adapt to a role. The core flow is still fixed.
03
Hybrid
Products offer both: the standard view, or ask and get a composed one. Teams maintain two systems.
04
Generative first
Composed views are the default. Fixed screens remain for core workflows and moments that matter emotionally.
05
Generative native
No pre-designed layout for any screen. Components and rules exist. Every view is composed.
Analytics and internal tools will move fast. Consumer products built on habit may stay at stage two for years. The job is choosing where your product should sit, on purpose, and designing for that stage. Most teams should be aiming at stage three.
Generative UI earns its cost when questions vary more than screens can. Use it there, and design the rest.
Run each surface through five questions. Be honest about the last two. They are where composed interfaces hurt people.
Fit check · one surface
Question varietyDo people ask many different questions of the same data?
Data shapeIs the data rich enough to answer in more than one view?
HabitIs it used daily, with layout people know by heart?
Cost of a wrong viewDoes money move, or does something get deleted or signed?
TrustDo people already trust the product with this task?
A high cost or low trust settles it on their own: design that screen. Otherwise, count. Mostly generated means compose. A split means hybrid, which is where most products should start: a designed default view, plus an Ask that composes.
The model can only compose what you give it. Every component needs a contract it can read.
Note
A component library built for people is a set of pictures. A model needs to know what each part is for, what data it takes, and when it must never appear.
Write a contract for every component the composer may use. Purpose, when to use it, when never to, its props with limits, how many may appear in one view, and the states it must handle.
registry.json · MetricTile
{
"name": "MetricTile",
"purpose": "One number that answers the question, with its change.",
"use_when": ["the intent asks how much, how many or how it changed"],
"never_when": [
"there is no comparison period",
"the number needs a caveat longer than one line"
],
"props": {
"label": "string, 24 characters max",
"value": "number, from the query result only",
"delta": "signed percent, optional",
"period": "string, required",
"source": "query id, required"
},
"max_per_view": 2,
"states": ["loading", "empty", "stale"]
}
Start small. Six to ten components cover most analytical surfaces. A short vocabulary used well reads as a product. A long one used loosely reads as a collage.
Component
Purpose
Max per view
MetricTile
One number, its change and its period
2
TrendBars
A value over time, with the moment that matters marked
1
DataTable
Rows to scan or compare, eight at most before paging
1
CompareColumns
Two or three options side by side, one marked as the fit
1
Callout
The answer in one or two sentences, in product voice
Everything the system may vary, and everything it may not. Write both down.
Your design tokens already fix colour, type and spacing. A composing surface needs a second layer: limits on the composition itself. The narrower the space, the more every generated view looks like it came from one hand.
space.md · one surface
Fixed
Colour, type and spacing tokens
The component set
Copy tone, from voice.md
Where the intent echo sits
Where actions sit
Varies, within range
Which components appear, one to four
Their order, by rule
Density, compact or regular
Chart type, from the data shape
Column span, one or two
Note
If a direction someone wants cannot be expressed inside the space, the team decides whether to widen it. The model never widens it on its own.
The rules decide what a good view looks like before any view exists.
These seven run in the composer above. Turn the rules off and each one breaks in a way you can see.
rules.md
# Composition rules
1. Answer first. The block that answers the question sits at the top.
2. One primary action per view. Everything else is secondary or absent.
3. Four blocks at most. If the answer needs more, it needs a designed screen.
4. Echo the intent in one line above the view, editable in place.
5. Every number shows its period and its source.
6. Choose the chart from the data shape:
time = TrendBars, rows = DataTable, choice = CompareColumns.
7. Same intent, same structure. Vary order and density, never the vocabulary.
Return a plan, render components
The model returns a plan: a short JSON list of components and their props. Your app validates it against the registry and renders it with real components. The model never writes markup. Markup is where tokens, accessibility and your visual system leak out.
Fig 2.1Where the rules sitPress to send the next intentintent in
Pick head: the model chooses from the magazines, the registryGauge bar: the rules, only a fitting tile passesReject tray: a block the rules stoppedBed and press: the app renders the viewSide chute and die: a boundary intent goes to a designed screenThe model chooses. The app renders. The rules sit between them. Here the picker only sets tiles on the chute; the gauge decides what reaches the press, and a boundary intent never reaches the picker.NextBoundaries →02 Choreography · 2.2
Boundaries
Where generation stops. Some intents always route to a screen a person designed.
Never composed
Why
Payments and refunds
Money moves. The wording and the confirmation have to be exact every time.
Cancellation
A system tuned for retention will invent pressure. That is a dark pattern with a model behind it.
Consent and privacy
Legal wording cannot vary by request.
Destructive actions
Delete, revoke and reset need a designed confirmation step.
First run
Trust is not earned yet. People need a stable place to learn the product.
The never-invent list
Inside the boundary the composer still has hard limits. Write these into the system prompt and validate against them.
boundaries.md · never invent
The composer never invents:
- a component that is not in registry.json
- a colour, font or spacing value outside the tokens
- a number that is not in the query result
- an offer, discount or promise
- an action the user cannot undo without a confirmation step
Try it
Ask the composer to help you cancel with the rules off. That is what a system without boundaries produces. With the rules on, it routes to a designed screen.
You cannot approve a screen that does not exist yet. You approve the rules, against a sample set.
Design review used to mean looking at a screen. Here you are quality-checking a generator. That takes an eval set: real intents, run in bulk, reviewed side by side.
01
Collect real intents
From support tickets, search logs and sales calls. Include the vague ones and the ones that should hit a boundary. A hundred is a good first set.
02
Tag what you expect
For each intent, the component family it should use and whether it is a boundary. This is your answer key.
03
Generate all of them
Then run the set again. The same intent twice shows you how much the output moves.
04
Review in a grid
Side by side, never one at a time. Drift only shows up when views sit next to each other.
05
Fix the rule, then rerun
Patching one bad view teaches the system nothing. Change the rule that produced it and run the whole set again.
Fig 3.1Review in a gridMove to scan · press to fixrow 1 of 5 · 0 fail so far
Tile: one generated view, 40 in allLamp: one row reviewed side by sideRed pin: a view that fails reviewCard file: the rules; one lifts when six misses share itIllustrative. Look for patterns in the misses. Six failures from one rule is one fix. Sample set: the six share one fault, so one changed card and one rerun clears them.NextThe rubric →03 Review · 3.2
The rubric
Five checks, the same for every view. Machines run three. People own two.
#
Check
Question
Owner
1
Answer
Does the top block answer what was asked?
Person
2
Vocabulary
Only registered components, each inside its contract?
Automated
3
Limits
One primary action, four blocks at most, tokens only?
Automated
4
Truth
Every number traceable to the query, with period and source?
Automated
5
Taste
Would you ship this view if a designer had drawn it?
Person
Automated checks run on every sample, every time. People review a slice, chosen to include the strangest intents. Track pass rates per rule, so you know which rule to fix.
Note
The fifth check is the one that matters most and the one no tool can run. That is the taste a team has to write down first. Playbook 01 covers how.
In generative UI the error state is the wrong interface. Design the recovery as carefully as the answer.
The user asked for one thing and got another. Recovery means showing the right view without making them start over. Four patterns cover most of it.
01
Intent echo
One line above the view says what the system understood. People correct it in place, and the view recomposes with their context kept.
02
Another view
Offer a second composition of the same data. Cheaper than rephrasing, and it teaches you which structure people prefer.
03
Pin it
Let people keep a composed view as a fixed one. Views that many people pin are candidates for a designed screen.
04
Show the source
Every block can reveal the query and data behind it. Trust in a composed view starts with being able to check it.[3]
Ceiling, MainArc's library of AI interface patterns and anonymized teardowns, is the place to study how shipped AI products handle trust and recovery. Use it as a reference set when you write your own patterns.
When confidence is low, show a designed screen. Never a guess dressed up as an answer.[2]
State
What the surface does
Low confidence
Shows the standard view, says it was not sure, and offers likely intents to pick from.
Boundary intent
Routes to the designed screen and says so in the echo.
No data
A designed empty state that says what would fill it and how.
Stale data
Composes as normal, with the period and a stale flag on every affected block.
Error or timeout
Keeps the last good view on screen, with a retry. Never a blank frame.
Try it
Type something vague into the composer, like "how are we doing". It falls back to the standard view and suggests what you might have meant.
Close the loop
Pins, swaps and corrected echoes are signals. Read them every cycle. A view people keep pinning becomes a designed screen. An echo people keep correcting points at a rule that needs work.
You need one surface, a component library, and a coding agent that can read both. Any coding agent that can read both will do.
Seven steps, from contracts to a hybrid surface in production. Do one surface end to end before you touch a second.
Setup prompt
Set up a generative UI layer for one surface of this product. Ask me which surface before you start.
1. Read the component library. For each component, draft a contract in genui/registry.json with: name, purpose, use_when, never_when, props with limits, max_per_view, states. Ask me before including any component.
2. Write genui/rules.md with these defaults: answer first; one primary action per view; four blocks at most; echo the intent in one line; every number shows its period and source; chart chosen by data shape; same intent, same structure.
3. Write genui/boundaries.md. Route these intents to designed screens: payments, cancellation, consent, destructive actions, first run. Add the never-invent list. Ask me what else belongs there.
4. Build the composer so the model returns a plan (JSON listing components and props). Validate every plan against registry.json and rules.md before rendering. The model never returns markup.
5. Create genui/evals/intents.csv with columns: intent, expected_family, boundary. Ask me for real intents from support, search and sales. Do not invent them.
6. Write genui/rubric.md with five checks: answer, vocabulary, limits, truth, taste. Automate vocabulary, limits and truth.
01
Pick one surface
Run it through the fit check first. An analytics view, a help panel or a search results page is a good first choice.
02
Write the contracts
Six to ten components. Cut anything the surface does not need.
Example prompt
Draft contracts for MetricTile, TrendBars, DataTable and Callout. For each never_when, give me one real intent from our product where it would be wrong.
03
Write the rules and boundaries
Start from the defaults. Change them only when a review shows a default is wrong for your product.
04
Return plans, render components
Validate before render. A plan that fails validation falls back, it never half-renders.
Example prompt
Add plan validation. Reject any plan with an unregistered component, more than one primary action or more than four blocks, and show the standard view instead.
05
Build the eval set
A hundred real intents, tagged. Run it twice to see how much the output moves.
06
Review in a grid, fix rules
Render every result side by side. Group the misses by rule. Fix the rule, rerun the set.
Example prompt
Run genui/evals/intents.csv. Render every result in one grid page, grouped by expected_family. List failures by rubric check and by the rule that caused them.
07
Ship it hybrid
A designed default view, plus an Ask that composes. Watch pins, swaps and corrected echoes, and read them every cycle.
Grades describe the kind of evidence behind a claim, not how much we like it. Claims graded P come from practice and are labelled every time. Numbers link to the reference list below.
2A · Experiment
0B · Field data
2C · Framework
2P · Practitioner
Claim
Grade
What supports it
Source
Generated interfaces can approach the quality of designed ones.
A · Experiment
Human raters preferred expert-built websites most often, with generated UI a close second and far ahead of text or markdown answers. Raters favoured it when speed was set aside, and quality rose with newer models [1].
[2]Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., Teevan, J., Kikin-Gil, R., & Horvitz, E. (2019). Guidelines for human-AI interaction. Proceedings of CHI '19. ACM. doi.org/10.1145/3290605.3300233
[3]Lee, J. D., & See, K. A. (2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50–80. doi.org/10.1518/hfes.46.1.50_30392
Grades
A controlled experiment. B large-scale field data. C an established peer-reviewed model or guideline, not a direct test of the claim. P practitioner claim from MainArc's work, untested.
You cannot approve a screen that does not exist yet. Approve the generator instead, with an eval set, a rubric and a panel.
Instruments
01
Rubric pass rate
Every generated view is checked against the five rubric checks. Automate vocabulary, limits and truth.
02
Consistency
Run every intent twice. Score whether the two runs share structure: same components, same answer-first block.
03
Boundary routing
For intents tagged as boundaries, measure how often they route to the designed screen. The target is all of them.
04
Pairwise preference
For a sample of intents, reviewers compare the composed view with the designed default and pick the one that answers better.
Study design
Element
Setting
Intents
100 real intents from support, search and sales, tagged with expected family and boundary
Runs
Each intent run twice
Panel
5 reviewers on a 20-intent sample
Analysis
Pass rate per rubric check and per rule. Preference rate with agreement measured by Kendall's W.
Cadence
After every rule change, and before any release
Reading the results
Ship when boundary routing is perfect and automated checks pass on nearly every view.
If composed views lose to the designed default, keep the surface hybrid and fix the rules before widening it.
If consistency is low, the rules allow too much variation. Tighten order and density rules first.
Running signals
Signal
What to watch for
Pins per view
Views people keep pinning are candidates for designed screens
Corrected echoes
Rising corrections point at a rule that misreads intent
Time to view
Generation speed was the main caveat in [1]. Track it as a quality metric
Note
Thresholds here are starting points to tune, not findings. A small panel shows direction and disagreement. It does not prove significance, and nothing on this page should be reported as if it did.
How this playbook was built, where it is weak, and how to cite it.
How this playbook was built
Built from MainArc's essay The Infinite Canvas, the AI interface patterns MainArc studies for Ceiling, and published research on generative UI and human-AI interaction. It is a practitioner method. Claims are graded on the evidence page.
Limitations
The strongest evidence [1] compares whole generated pages with websites, not composed views inside an existing product.
The five stages are a model of the shift, not a measurement.
The composer on the introduction page is a scripted demo with fixed compositions, not a model.
The fit check is a heuristic. It has not been validated against real launch outcomes.
Open questions
How much variation do people accept before a product stops feeling learnable?
Can teams measure brand consistency across generated views automatically?
Where should generation stop as models get faster and more reliable?
Cite this playbook
APA
MainArc Research. (n.d.). Ship interfaces that compose themselves (Playbook No. 02, Version 1.0). MainArc. https://mainarc.xyz
BibTeX
@misc{mainarc_compose,
author = {{MainArc Research}},
title = {Ship interfaces that compose themselves},
howpublished = {MainArc Playbook No. 02, Version 1.0},
url = {https://mainarc.xyz}
}
Corrections
Found an error, a stronger source, or a result from running the benchmark? Send it through mainarc.xyz. Corrections are made in the next version.
About this playbook
This playbook is published by MainArc, a creative intelligence lab for leadership and product teams.
We do three things. We research how AI is changing creative work through long-form reports, design teardowns, and operator essays. We design the workflows, prompt systems, review loops, and decision frameworks that teams actually run on. And we build the work the strategy points to: brand systems, interfaces, product prototypes, internal tools, and launch concepts.
One operator. One AI agent. No layers in between. Every engagement runs under strict NDA. No case studies, no client logos, no shared work.
If this research resonates with a problem your team is facing, start with an audit. One week. A clear map of where AI fits, where it doesn't, and what to build next.
This is independent research. Not financial advice. Not career advice. Not the final word on anything. It's one person's perspective shaped by building, shipping, and paying attention. Do your own thinking. Challenge the ideas. Disagree with specifics. Build on what resonates. Discard what doesn't.
House rules
What doesn't belong here: hate, trolling, bad-faith takedowns, or drive-by takes from people who've never built anything. If you disagree, show your work. Builder's critique. Everyone else just comments.
A MainArc playbook for product teams. Set up a kit anyone can prototype from, decide every week, and take the winning direction into production as working code. Each step names who does it, how long it takes, what goes in, what comes out, and how you know it's done.
PlaybookNo. 03Version1.0Read22 minSources3 citedByMainArc ResearchEvidence and method →
Thesis
AI collapsed the time it takes to make. The slow part now is deciding, so the teams that move fastest decide well, every week, in writing.
Do this Monday
Answer the starting-step finder honestly. Then book a 45-minute weekly crit with one named decider.
Most teams are not starting from zero. Answer honestly and the playbook tells you where to begin and what comes after.
Starting step · 11 questions
6Stages
17Steps
4Templates
2Weeks to a first decision
What this is
AI made it possible for anyone on a product team to turn an idea into something that runs. Most teams have not changed how they work to match. The founder still writes a brief and waits. Design still draws screens that engineering redraws. The idea still meets a customer months after it was cheap to test.
This playbook replaces that with a loop the whole team runs. A shared kit built from your real components, tokens and data. A written question before anyone makes anything. Several working directions instead of one picture. A short weekly decision. A clean path from the chosen prototype into production, and a way to bring what you learned back into the kit.
Making is cheap now. Deciding well, and often, is the work. Most of this playbook is about the second half.
Principles
01
One kit, everyone builds from it
Real tokens, real components, real-looking data. A prototype built elsewhere is a drawing, and it will be redrawn.
02
Question first
A prototype without a written question is a demo. It impresses the room and teaches nobody anything.
03
Directions differ in structure
Three colour variants are one direction. Real directions answer the question in different ways.[2]
04
Decide on a schedule
A short weekly crit beats a thorough one that happens whenever people are free.
05
Write down what you rejected
The rejected directions and the reasons are what stop the team arguing the same thing next month.
06
Prototype code ships, partly
The interface layer survives into production. Data, auth, error handling and performance get rebuilt properly.
Playbook 03 in a set. Playbook 01 covers how a team writes down its taste. Playbook 02 covers interfaces that compose themselves. This one covers how a product team builds together.
The pace that matters runs from an idea to something real, from real to evidence, and from evidence to a customer. AI sped up one part of that. The rest is still yours to fix.
Three clocks
Every product idea runs on three clocks. The make clock runs from an idea to something that works. The learn clock runs from a working thing to evidence about whether it is right. The ship clock runs from a decision to the customer.
Fig 0.1Three clocksPress or Space to flipmake 0% · learn 75% · ship 80%
01 Make · idea to working02 Learn · working to evidence03 Ship · decision to customerSame neck in all three: sand held sets the timeWith AI, under the old process. All three glasses flip together and run at the same rate. Before AI, make was the tallest of the three. Now it is small and empties first; learn and ship are as tall as before, because process sets them, and they run on long after. The glasses show the shape of the problem, not a measurement. Your own clocks will differ.
AI collapsed the make clock.[1] A PM can have a working version of an idea before lunch. The other two barely moved, because they run on process, and most product process was designed for a different constraint.
The old process rationed making
Briefs, design queues, review rounds and handoffs exist for one reason. Designer and engineer time used to be the scarce input, so the process rationed it. Write it down, wait your turn, review the pictures, hand them over, build it, and only then find out whether it works.
When making stops being scarce, rationing it is pure delay. The queue becomes the slowest part of the system, and the speed AI gave you gets spent waiting in it.
The new bottleneck is deciding
Remove the queue and a new problem shows up fast. Ten prototypes and no way to choose is slower than one prototype. Every person now produces working options, and nobody has time to judge them all. The teams that move quickly with AI are the ones with a short, regular, written way of deciding.
MainArc Research
Production is free. Judgment isn't. Pace now depends on how well you decide, and how often.
Pace compounds
Each cycle that ends in a decision leaves something behind. A new component in the kit. A rule. Better sample data. A sharper question. The next idea starts further along. Teams that skip the decision lose all of that, and every idea starts from zero again.
Set up the kit once. Then run five stages, as often as you have questions worth answering.
Fig 0.2Five stages around one kitPress or Space for next stageAsk · reads the kit · kit 6
01 Kit · the stack in the middle02 Ask, 03 Make, 04 Decide, 05 Ship, 06 Learn · clockwise from the leftPick head: touches the kit first at every stageMagazines at Decide and Learn: new parts for the kitEvery stage reads from the kit. Decide and Learn write back to it. The turntable carries one piece of work round the five stations; each head touches the kit before it works the tray, and the kit stack grows by two parts a loop. Set up the kit once, then maintain it.
A repo where anyone on the team can build something that looks and behaves like your product, with your real tokens, components, words and data. Budget about a week the first time. After that it grows a little every cycle.
Do not try to kit the whole product. Pick one surface where the open questions live.
WhoPM, designer, engineer
BudgetOne hour
InNext quarter's open questions
OutOne surface, one kit owner
Do
List the product questions you expect to argue about next quarter. Ten is plenty.
Mark the surface each one lives on: onboarding, the dashboard, settings, checkout.
Pick the surface with the most questions that already has components in code.
Name one kit owner. Usually a designer who can read code, or an engineer with taste.
Write both into kit/README.md, with the list of questions underneath.
Done when
One surface is named and two or more open questions live on it
The kit owner has agreed, and the README says so
Prompt
Read our roadmap notes and open issues. List the product questions we will need to decide next quarter, grouped by the surface they live on. Mark which surfaces already have components in code.
The kit uses the same names and values as production, so nothing gets translated twice.
WhoEngineer, designer
BudgetOne to two days
InProduction styles, design files
Outkit/tokens.css, kit/components/
Do
Export colour, type, spacing, radius and motion into kit/tokens.css as CSS variables. Keep production's names.
Find the ten components the surface uses most. Copy or import them into kit/components/.
Strip production dependencies. Each component must render on its own with sample props.
Add a short header comment to each: what it is for, when not to use it, the states it supports.
Build a /kit page that shows every component in every state, in light and dark.
Done when
The /kit page renders every component and state with no production services running
Changing one token visibly changes every component that uses it
Prompt
Create kit/tokens.css from our production theme, keeping every variable name. Then copy Button, Input, Select, Table, Card, Tabs, Sheet, Toast, EmptyState and Badge into kit/components/ so each renders without our API. Build a /kit page showing each component in every state.
Prototypes built on lorem ipsum get judged on lorem ipsum. Give the kit your words and your data.
WhoPM, support
BudgetHalf a day
InHelp docs, tickets, real records
Outkit/context/
Do
glossary.md: the terms your product uses and the ones it never uses. Workspace or team. Archive or delete.
data/: realistic sample records with real shapes. Long names, empty fields, a customer with ten thousand rows. Anonymise anything real.
copy.md: real error messages, empty states and confirmations, copied from production.
users.md: who uses this surface, what they are trying to get done, and what they complain about. One paragraph each.
Done when
A prototype made by someone new uses your terms and plausible numbers without being told
The sample data includes at least one awkward case: very long, empty, or very large
Prompt
Using these support tickets and help articles, draft kit/context/glossary.md and copy.md. Then generate 50 anonymised sample records for kit/context/data/ that match our schema, including edge cases: long names, missing fields, one very large account.
The coding agent reads this file every session. It is how the kit's rules reach people who never read the kit.
WhoDesigner, engineer
BudgetTwo hours
InThe kit so far
OutAGENTS.md at the repo root
Do
Start from the template below. Use the filename your agent reads: AGENTS.md, CLAUDE.md or similar.
Point it at tokens, components and context, in the order the agent should read them.
Write the always and never lists. Keep each under ten lines. Long instruction files get skimmed by agents too.
Require a closing report: new patterns invented, and anything the agent guessed.
AGENTS.md · template
# Prototype kit rules
## Read first, in order
1. kit/tokens.css: the only colours, type, spacing, radius and motion allowed
2. kit/components/: use these before writing anything new
3. kit/context/: our words, our data, our users
## Always
- Build from kit components. If none fits, build from tokens
and mark the file NEW-PATTERN at the top.
- Use sample data from kit/context/data/. Never placeholder text.
- Give every view that loads data a loading, empty and error state.
- Use the terms in kit/context/glossary.md.
## Never
- Add a colour, font or spacing value that is not in tokens.css
- Install a UI library
- Touch anything outside /prototypes
## When you finish
- List every NEW-PATTERN file, one line on why
- List anything you guessed
Done when
A fresh agent session, asked to build anything, reports which kit files it read
An agent asked to add a new colour refuses and points at tokens.css
Before anyone prototypes something new, prove the kit can rebuild something that already exists.
WhoSomeone outside design
BudgetOne hour
InA screenshot of a live screen
OutA rebuild and a gap list
Do
Pick a real screen on the surface. Take a screenshot.
Ask a PM, support lead or engineer to rebuild it with an agent, using only the kit.
Put the rebuild next to production. Every visible difference is a kit gap.
Fix the kit, not the rebuild. Then run the test again with a different screen.
Done when
Two rebuilds in a row are close enough that a customer would not notice
The gap list is empty, or every item on it has an owner
Prompt
Here is a screenshot of our billing settings screen. Rebuild it in /prototypes using only kit components, tokens and sample data. When done, list everything you could not build from the kit.
Every cycle starts with a written question. It is the cheapest step in the loop, and the one that decides whether everything after it teaches you anything.
A reference says which part to take: the layout, the flow, the density, the tone. Never the whole thing.
WhoAnyone
BudgetThirty minutes
Inbrief.md
Outrefs/ with one line per image
Do
Collect three to six references from products that solved a similar problem well. Your own product counts.
Save each with one line saying what to take from it and what to ignore.
Add one reference for what you do not want. It saves a round of feedback.
Done when
Every reference has a take-this line
No reference is there just because it looks nice
Prompt
Here are four references and our brief.md. For each reference, tell me which part answers our question and which part we should ignore. Write the result into refs/README.md.
Each direction answers the question in a different way. Each one runs at its own link.
WhoAnyone on the team
BudgetAn afternoon per maker
Inbrief.md, refs/, the kit
OutThree to eight directions
Do
Read the brief and the references before opening the agent.
Ask for directions that differ in structure: a wizard, an inline flow, a single page, a default with no setup at all.
Use kit sample data, including the awkward records.
Give each direction a name that states its idea. "Invite from the empty state" beats "Option B".
Deploy each one to a preview link. A screenshot is not a direction.
Done when
Each direction runs at its own link and can be judged without the maker in the room
No two directions differ only in styling
Prompt
Read brief.md and refs/. Make three directions in /prototypes that answer the question in structurally different ways. Use only the kit and kit sample data. Name each folder after its idea, and give each a one-sentence statement of what it bets on.
Crit time is for judging ideas. Kit problems get caught before anyone walks into the room.
WhoEach maker, with an agent
BudgetFifteen minutes
InEach direction
OutA clean link and a bet statement
Do
Run the kit check below on every direction.
Fix anything outside the tokens. Keep NEW-PATTERN files, but say why each one exists.
Click through the empty state, the error state and the awkward record.
Write one sentence at the top of the direction: what it bets on.
Done when
The kit check is clean, or every exception has a reason
Every direction has its bet in one sentence
Prompt
Check this prototype against AGENTS.md. List every value not in tokens.css, every component not in kit/components, every view missing a loading, empty or error state, and every term not in the glossary.
The stage most teams skip, and the one this playbook cares about most. A short crit, a written decision, and only what earned it goes back into the kit.
Twenty minutes the same day. The record of what you rejected is worth as much as the choice.
WhoPM
BudgetTwenty minutes
InCrit notes
Outdecision.md
Do
Write the chosen direction and link it.
Write why, in three lines at most, tied to what good looks like in the brief.
Write one line for every rejected direction. Why not, in plain words.
Write what you are still unsure about. It becomes the signal in step 6.1.
Name the hardening owner and a date.
Done when
Someone who missed the crit understands the decision without asking
Every rejected direction has its line
Prompt
Here are my crit notes and brief.md. Draft decision.md using our template. Keep my reasons in my words. Flag any rejected direction I gave no reason for.
Shipping answers nothing until someone checks. Pick the signal before launch, look on the date, and write the result where the next question will find it.
Week one builds the kit. Week two runs the loop once, on a real question, end to end.
Setup prompt
Help me set up whole-team prototyping on one surface of our product.
1. Ask me which surface, and who owns the kit. Write both into kit/README.md with the open questions on that surface.
2. Build kit/tokens.css from our production theme, keeping every variable name. Copy the ten components the surface uses most into kit/components/ so each renders without our services, and build a /kit page showing every state.
3. Draft kit/context/glossary.md, copy.md and users.md from our help docs and support tickets. Generate anonymised sample data with awkward cases. Do not invent product facts.
4. Write AGENTS.md with read-first, always, never and when-you-finish sections.
5. Help me write the first brief.md: one question, what good looks like, constraints, the decider and the crit date.
Day
Steps
What happens
W1 D1
1.1, 1.2
Pick the surface. Start moving tokens and components into the kit.
W1 D2
1.2
Finish components. Build the /kit page.
W1 D3
1.3, 1.4
Add product context. Write AGENTS.md.
W1 D4
1.5
Two rebuild tests. Fix the gaps.
W1 D5
2.1, 2.2
Write the first brief. Gather references.
W2 D1
3.1
Everyone who wants to makes a direction.
W2 D2
3.1, 3.3
Finish directions. Run the kit check.
W2 D3
4.1, 4.2
First crit. Decision written the same day.
W2 D4
4.3, 6.1
Promote patterns. Set the signal.
W2 D5
5.1
Hardening starts on the chosen direction.
From week three, the loop runs weekly. Briefs land early in the week, directions by midweek, crit at a fixed time, decision the same day.
The four files the loop runs on, plus the repo layout. Copy them as they are, then edit to fit.
Repo layout
AGENTS.md rules the agent reads every session
kit/
README.md surface, owner, open questions
tokens.css production names and values
components/ ten components, each renders alone
context/
glossary.md words we use and never use
copy.md real errors, empty states, confirmations
users.md who uses this surface and why
data/ anonymised records, awkward cases included
CHANGELOG.md what each cycle promoted
questions/
invite-flow/
brief.md
refs/
decision.md
prototypes/ one folder per direction
brief.md
# Question
One sentence. What do we need to learn?
## Why now
What changed, or what it costs to wait.
## Who it is for
The user and the job they are trying to get done.
## What good looks like
Observable. A task finished, a step removed, fewer tickets.
No adjectives.
## Constraints
What must not change. What is out of scope.
## Directions wanted
How many, and how different.
## Decides
Name, and the date of the crit.
decision.md
# Decision: [question]
Date · Decided by
## Chosen
Direction name, link, commit.
## Why
Three lines at most, tied to what good looks like.
## Rejected
- Direction: one line on why not
- Direction: one line on why not
## Still unsure
What could prove this wrong.
## Signal
Metric, threshold, review date.
## Hardening
Owner, target date.
## Result
Filled in on the review date. Number, verdict, next question.
handoff.md
# Handoff: [direction]
- [ ] Prototype commit linked
- [ ] decision.md linked
- [ ] Sample data replaced by real queries
- [ ] Awkward records kept as tests
- [ ] Loading, empty and error states in production
- [ ] Auth and permissions
- [ ] Accessibility pass: keyboard, focus, contrast, labels
- [ ] Analytics events for the signal
- [ ] NEW-PATTERN files promoted or left local
- [ ] Designer sign-off in the running build
- [ ] Open questions, each with an owner
Grades describe the kind of evidence behind a claim, not how much we like it. Claims graded P come from practice and are labelled every time. Numbers link to the reference list below.
3A · Experiment
0B · Field data
0C · Framework
2P · Practitioner
Claim
Grade
What supports it
Source
AI has shortened the make clock.
A · Experiment
In a controlled experiment, developers with an AI pair programmer finished a coding task 55.8% faster than those without [1]. One task, one tool, but a clear signal.
From practice. Untested. The loop health metrics below are designed to show whether it holds for your team.
Practice
References
[1]Peng, S., Kalliamvakou, E., Cihon, P., & Demirer, M. (2023). The impact of AI on developer productivity: Evidence from GitHub Copilot. arXiv:2302.06590. arxiv.org/abs/2302.06590
[2]Dow, S. P., Glassco, A., Kass, J., Schwarz, M., Schwartz, D. L., & Klemmer, S. R. (2010). Parallel prototyping leads to better design results, more divergence, and increased self-efficacy. ACM Transactions on Computer-Human Interaction, 17(4), Article 18. doi.org/10.1145/1879831.1879836
[3]Tohidi, M., Buxton, W., Baecker, R., & Sellen, A. (2006). Getting the right design and the design right. Proceedings of CHI '06, 1243–1252. ACM. doi.org/10.1145/1124772.1124960
Grades
A controlled experiment. B large-scale field data. C an established peer-reviewed model or guideline, not a direct test of the claim. P practitioner claim from MainArc's work, untested.
Measure the loop the same way the playbook argues for it: by its clocks, by the range of what it makes, and by how often it ends in a decision.
Instruments
01
Clock times
Idea to first working direction. Direction to decision. Decision to customer. Logged per question.
02
Direction range
A panel rates how different each set of directions is. Low range means the team is making variants, not directions [2].
03
Decision rate
Share of crits that end in a decision or a rewritten question.
04
Survival
Share of prototype interface code that reaches production after hardening.
Study design
Element
Setting
Questions
Every brief for 8 weeks
Panel
3 reviewers rate direction range on a 1 to 5 scale
Baseline
The same clocks for the 8 weeks before the loop started
Analysis
Median clock times before and after. Range and decision rate per week.
Cadence
Weekly dashboard, full review at 8 weeks
Reading the results
The make clock should fall first. If learn and ship do not follow, the crit and hardening steps are the bottleneck.
If range is low, the brief is asking for variants. Rewrite it to ask for structurally different answers.
If decision rate is low, the crit has no single decider. Fix ownership before anything else.
Running signals
Signal
What to watch for
Kit gaps per rebuild
Should fall to zero and stay there
Signal hit rate
Share of launches that met the threshold set in step 6.1
Rejected directions written down
Should be every one
Note
Thresholds here are starting points to tune, not findings. A small panel shows direction and disagreement. It does not prove significance, and nothing on this page should be reported as if it did.
How this playbook was built, where it is weak, and how to cite it.
How this playbook was built
Built from MainArc's advisory method and from published research on prototyping and AI-assisted development. It is a practitioner method. Claims are graded on the evidence page.
Limitations
The make clock evidence [1] comes from one coding task in a lab. Real product work varies more.
The prototyping studies [2, 3] predate AI tools. Their mechanism, comparison across options, should transfer, but that is untested here.
Time budgets on each step are working estimates, not measurements.
The starting-step finder orders steps by dependency, not by measured impact.
Open questions
Does more people making lower the quality floor of directions, or raise the range?
How much prototype code really survives hardening across team types?
What is the right crit size before decisions slow down?
Cite this playbook
APA
MainArc Research. (n.d.). Prototype with the whole team (Playbook No. 03, Version 1.0). MainArc. https://mainarc.xyz
BibTeX
@misc{mainarc_team,
author = {{MainArc Research}},
title = {Prototype with the whole team},
howpublished = {MainArc Playbook No. 03, Version 1.0},
url = {https://mainarc.xyz}
}
Corrections
Found an error, a stronger source, or a result from running the benchmark? Send it through mainarc.xyz. Corrections are made in the next version.
About this playbook
This playbook is published by MainArc, a creative intelligence lab for leadership and product teams.
We do three things. We research how AI is changing creative work through long-form reports, design teardowns, and operator essays. We design the workflows, prompt systems, review loops, and decision frameworks that teams actually run on. And we build the work the strategy points to: brand systems, interfaces, product prototypes, internal tools, and launch concepts.
One operator. One AI agent. No layers in between. Every engagement runs under strict NDA. No case studies, no client logos, no shared work.
If this research resonates with a problem your team is facing, start with an audit. One week. A clear map of where AI fits, where it doesn't, and what to build next.
This is independent research. Not financial advice. Not career advice. Not the final word on anything. It's one person's perspective shaped by building, shipping, and paying attention. Do your own thinking. Challenge the ideas. Disagree with specifics. Build on what resonates. Discard what doesn't.
House rules
What doesn't belong here: hate, trolling, bad-faith takedowns, or drive-by takes from people who've never built anything. If you disagree, show your work. Builder's critique. Everyone else just comments.
A MainArc playbook for AI-native products. When AI moves from answering to acting, delegation becomes the interface. How to scope it, ask for it, preview it, show it, prove it and take it back.
PlaybookNo. 04Version1.0Read19 minSources4 citedByMainArc ResearchEvidence and method →
Thesis
When AI acts, delegation is the interface. A popup on every action protects nobody. Trust is earned one action and one level at a time.
Do this Monday
List every action your agent can take. Mark the ones that cannot be undone, and set each of those to Ask first.
An autonomy dial, one task at a time. It is live. Pick a task, move the dial, and watch what the person sees, what they have to do, and what protects them at each level. Push a task past its ceiling and the dial tells you why.
Autonomy dial · sample productLevel
What the person sees
Protections
6Stages
12Steps
5Autonomy levels
4Templates
What this is
For two years the question about AI interfaces was whether the answer was any good. That question is fading. Products now let models send the email, book the meeting, issue the refund and change the file. The new question is what the AI is allowed to do on its own, and what happens when it gets that wrong.
Most teams answer it with a confirmation popup. People click through it until they stop reading it, or the agent asks so often that nobody uses it. Neither builds trust.
This playbook treats delegation as a design material. Six stages, from deciding what an agent may touch to undoing what it did. Trust gets earned per task, one level at a time, and every level has a clear interface. Agents make doing the work free. Deciding what they may do alone is the judgment this playbook is about.
MainArc Research
Models commoditise. Interfaces compound. In agent products, the interface that compounds is the one that makes handing over work feel safe.
Principles
01
Autonomy is per action
An agent can be trusted to archive files and still not to pay invoices. Set the level for each action, never for the whole product.
02
Earn every level
Start low. Clean runs move an action up. Corrections move it down. The person approves every step up.[2]
03
Show the plan before the act
People approve consequences. Show what will change, who will see it and what it costs.
04
Friction follows stakes
Low-stakes actions run quietly. High-stakes ones ask clearly. Uniform friction trains people to ignore it.
05
Every action leaves a receipt
What was done, where, on whose behalf, and how to reverse it.
06
Say what cannot be undone
Before acting, never after. If there is no undo, the level stays low.
How to use this playbook
01
Play the dial first
Push the invoice task to Act alone. That one interaction is the argument for everything after it.
02
Read in order
Scope, Permission, Preview, Act, Receipt, Recover. Each stage assumes the one before it.
03
Start with one action
A frequent, annoying, recoverable action. Take it through all six stages before touching a second.
04
Copy the files
The action policy, receipt, permission prompt and stop question are on the Templates page.
Playbook 04 in the set. Playbook 02 covers interfaces that compose themselves. This one covers what happens when those interfaces start doing things.
The trust problem moved. It used to be about whether the output was right. Now it is about whether the action should happen at all.
A wrong answer costs a reread
When AI only answers, the person is still the one who acts. Their judgment sits between the model and the world for free. A bad answer gets noticed, ignored or rewritten.
A wrong action reaches the world
When the agent acts, that judgment is gone unless the product puts it back. A wrong email reaches a client. A wrong refund leaves the account. A wrong delete takes a file with it. The design job is deciding where the person's judgment sits now, and making it cheap to use.
Fig 0.1Where the person's judgment sitsPress or Space to steptwo cards ready in the model
Back line: an answer. The person's hand gate is already thereFront line: an action. End stops, preview lens, receipt printer, undo chuteBlue: where the person's judgment sitsAnswers keep a person in the loop for free. Actions do not. On the back line nothing reaches the world until someone lifts the gate. On the front line the product has to build that judgment in: limits on reach, a stop to look before it acts, a receipt after, and a way back.
Why the popup fails
A confirmation on every action asks for judgment without giving the person anything to judge with. It names no consequence, shows no plan, and looks the same for a calendar tweak and a payment. People learn to click it. After that it protects nobody.[4]
Note
The aim is to put the person's judgment back in the right places, and nowhere else.
Five levels, from the agent suggesting to the agent acting alone. Every action sits on one rung, and the rung decides the interface.[1]
Fig 0.2Five rungsMove to turn · press to clickL2 ask first · a plan card
Selector: L0 suggest, L1 draft, L2 ask first, L3 act and tell, L4 act aloneReceipt tape and undo spring: what the action has to carryL2 ask first: where most actions startThe higher an action climbs, the more its receipt and undo have to carry. Each click up feeds the receipt further and stretches the undo spring; each click down tears the extra receipt off. Lengths are illustrative.
Level
Agent
Person
Interface
L0 Suggest
Notices and proposes
Does the work
A suggestion card
L1 Draft
Prepares the work
Reviews and sends it
An editable draft, with the send button on the person's side
L2 Ask first
Plans, then waits
Approves the plan
A plan card with the consequences named
L3 Act and tell
Acts, then reports
Can undo for a window
A receipt with undo
L4 Act alone
Acts inside set limits
Reads a digest
A daily digest, exceptions first
In most products today, most actions belong at L1 or L2. A few earn L3 and L4. The ladder turns that into a decision you made, instead of a default you inherited from whoever wrote the tool call.
Six stages run for every action an agent can take. The last one feeds the first.
Fig 0.3Six stages per actionPress or Space to step01 scope · level L2
01 Scope · 02 Permission · 03 Preview04 Act · 05 Receipt · 06 RecoverCarriage: one action on its way throughClean run: the level goes up oneCorrection: the level goes down oneSix stages run for every action an agent can take. The last one feeds the first: corrections and clean runs ride back to Scope and adjust the level. Sample run: every other action needs a correction.
#
Stage
Question it answers
Ships as
01
Scope
What may the agent touch, and how high can each action go?
List every action the agent can take in the world. Then describe each one by what it would cost to get wrong.
WhoPM, designer, engineer
BudgetHalf a day
InAgent tools and integrations
Outactions.csv
Do
List every tool call that changes something outside the agent: sends, writes, deletes, pays, books, publishes.
Record four things for each: reversibility (undo, window, compensate, none), reach (the user, their team, outsiders), cost (none, money, reputation) and frequency.
Group actions that match on all four into classes. They will share a ceiling.
Flag anything that touches money, outsiders or deletion. Those get the most design time.
Done when
Every tool that changes state appears in actions.csv
Every action has all four attributes filled in
Prompt
List every tool our agent can call that changes something outside the conversation. For each, tell me whether it can be undone, who it reaches, what it can cost and how often it runs. Save it as trust/actions.csv.
Every action has a way back, decided before the agent is allowed to take it. Over time, the record of corrections decides how much the agent is trusted.
Levels should move. Clean runs earn a step up. Corrections cost a step down. The person always decides.
WhoPM, designer
BudgetHalf a day
InApprovals, edits, undos, stops
OutLevel changes, with consent
The step-up offerExample
You approved the last 20 follow-ups without changes. Send follow-ups to existing contacts without asking? You will still get a receipt, and 30 seconds to undo each one.
Do
Track four things per action: approvals without edits, edits, undos and stops.
After a run of clean approvals, offer a step up, in words that say what will change.
After an undo or a heavy edit, drop the level one rung and say why.
Never raise a level without asking. Never raise it past the ceiling.
Done when
Levels change on evidence, with the person's consent
Pick one action, take it through all six stages, and ship it one rung below its ceiling. Then do the next.
Setup prompt
Help me design one agent action so people trust it.
1. Read our agent's tools. List every action that changes something outside the conversation. For each, record reversibility (undo, window, compensate, none), reach (user, team, outsiders), cost (none, money, reputation) and frequency. Save it as trust/actions.csv.
2. Propose a ceiling and a starting level for each, using L0 suggest, L1 draft, L2 ask first, L3 act and tell, L4 act alone. Nothing irreversible that reaches outsiders goes above L2. Save it as trust/action-policy.json. Ask me before you finalise it.
3. For the action I choose, write the permission prompt, the plan card, the receipt, the undo path and the stop conditions.
4. Add tracking for approvals, edits, undos and stops, so the action can earn levels later.
5. Show me everything and ask what is wrong with it.
01
Pick the action
Frequent, mildly annoying, and recoverable. Follow-ups, labelling, scheduling. Leave payments for later.
02
Write its policy
Reversibility, reach, cost, ceiling, starting level, limits and stop conditions, in action-policy.json.
03
Design the four moments
The permission prompt, the plan or draft, the receipt, and the undo. Each one names the consequence.
04
Build the stops
Test each stop condition with a deliberate mismatch before anyone outside the team sees it.
05
Ship one rung below the ceiling
Room to earn trust is part of the design. Starting at the ceiling leaves nowhere to go but down.
06
Review in two weeks
Read the approvals, edits, undos and stops. Offer the step up if it has been earned. Drop the level if it has not.
Example prompt
Read the last two weeks of approvals, edits, undos and stops for send_followup_email. Tell me whether it has earned a step up, and draft the offer the user would see.
[Verb] [object] [to whom]?
[Who it reaches]. [What it costs]. [How to undo, or that you cannot].
[Do this one] [Up to N a day] [Not now]
Never: "Allow access?" "Are you sure?" "Confirm" "OK"
Receipt
[What was done] · [time]
[Who it reached]. Done on your behalf in [place].
[Undo: how, and until when] or [This cannot be undone]
[Link to what changed]
Stop question
## Stopped: [action]
What I expected:
What I found:
What I would do, and why:
Other options:
What happens if you do nothing:
Research · Playbook 04
Research
What this playbook rests on, how to measure whether it works for you, and what it cannot yet claim.
Grades describe the kind of evidence behind a claim, not how much we like it. Claims graded P come from practice and are labelled every time. Numbers link to the reference list below.
0A · Experiment
1B · Field data
3C · Framework
2P · Practitioner
Claim
Grade
What supports it
Source
Automation should be set per function, at levels, not on or off.
C · Framework
The established model of human interaction with automation describes levels from fully manual to fully automatic, chosen separately for each function [1].
Across more than a million reminders, clinicians accepted fewer as the share of repeats rose: about 10% lower acceptance for every 5-point rise in repeats [4].
Receipts, undo windows and earned levels build trust over time.
P · Practitioner
From MainArc's pattern work for Ceiling. Consistent with [2] and [3], not tested as a system.
Practice
The ceilings in the autonomy dial are right for those tasks.
P · Practitioner
Judgment calls made for illustration. Your own ceilings depend on your users, limits and law.
Practice
References
[1]Parasuraman, R., Sheridan, T. B., & Wickens, C. D. (2000). A model for types and levels of human interaction with automation. IEEE Transactions on Systems, Man, and Cybernetics, Part A, 30(3), 286–297. doi.org/10.1109/3468.844354
[2]Lee, J. D., & See, K. A. (2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50–80. doi.org/10.1518/hfes.46.1.50_30392
[3]Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., Teevan, J., Kikin-Gil, R., & Horvitz, E. (2019). Guidelines for human-AI interaction. Proceedings of CHI '19. ACM. doi.org/10.1145/3290605.3300233
[4]Ancker, J. S., Edwards, A., Nosal, S., Hauser, D., Mauer, E., Kaushal, R., & HITEC Investigators. (2017). Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Medical Informatics and Decision Making, 17, 36. doi.org/10.1186/s12911-017-0430-8
Grades
A controlled experiment. B large-scale field data. C an established peer-reviewed model or guideline, not a direct test of the claim. P practitioner claim from MainArc's work, untested.
Trust is behaviour before it is a feeling. Measure what people do with each action, then ask them how in control they feel.
Instruments
01
Action outcomes
For each action and level: approved unchanged, edited, undone, stopped.
02
Confirmation load
How many confirmations each person sees per day. Rising load predicts ignored confirmations [4].
03
Ceiling breaches
Any action that ran above its ceiling. The target is zero, always.
04
Felt control
A short survey, 1 to 5: I know what it will do. I can see what it did. I can undo it.
Study design
Element
Setting
Actions
Every agent action in the policy
Window
Two weeks per level change
People
All active users for behaviour. 20 for the survey
Analysis
Rates per action and level. Survey medians before and after a level change.
Cadence
Weekly, plus every time a level changes
Reading the results
Offer a step up only after a clean run. Twenty unchanged approvals is a reasonable starting point to tune.
Drop a level after an undo or a heavy edit, and tell the person why.
If confirmation load climbs, the levels are too low or the friction is uncalibrated. Fix the level first.
Running signals
Signal
What to watch for
Undo usage
Low but non-zero is healthy. Zero may mean undo is hidden
Stops with options
Every stop should arrive as a question, never an error
Time in queue
How long approvals wait. Long waits push people to raise levels too early
Note
Thresholds here are starting points to tune, not findings. A small panel shows direction and disagreement. It does not prove significance, and nothing on this page should be reported as if it did.
How this playbook was built, where it is weak, and how to cite it.
How this playbook was built
Built from human factors research on automation and trust, guidelines for human-AI interaction, and the interface patterns MainArc studies for Ceiling. It is a practitioner method. Claims are graded on the evidence page.
Limitations
The automation and trust research [1, 2] comes mostly from aviation, process control and driving. Software agents differ in speed and scale.
The alert fatigue evidence [4] is from clinical software. The mechanism is general, the numbers are not.
The ceilings and sample tasks in the dial are illustrative, not recommendations for any specific product.
Nothing here is legal or compliance advice. Payments, consent and data rules vary by market.
Open questions
How fast should autonomy be earned, and should it decay when unused?
Do people trust agents more with receipts, with undo, or with both?
How should one agent's level carry across devices and contexts?
Cite this playbook
APA
MainArc Research. (n.d.). Design agents people trust to act (Playbook No. 04, Version 1.0). MainArc. https://mainarc.xyz
BibTeX
@misc{mainarc_agents,
author = {{MainArc Research}},
title = {Design agents people trust to act},
howpublished = {MainArc Playbook No. 04, Version 1.0},
url = {https://mainarc.xyz}
}
Corrections
Found an error, a stronger source, or a result from running the benchmark? Send it through mainarc.xyz. Corrections are made in the next version.
About this playbook
This playbook is published by MainArc, a creative intelligence lab for leadership and product teams.
We do three things. We research how AI is changing creative work through long-form reports, design teardowns, and operator essays. We design the workflows, prompt systems, review loops, and decision frameworks that teams actually run on. And we build the work the strategy points to: brand systems, interfaces, product prototypes, internal tools, and launch concepts.
One operator. One AI agent. No layers in between. Every engagement runs under strict NDA. No case studies, no client logos, no shared work.
If this research resonates with a problem your team is facing, start with an audit. One week. A clear map of where AI fits, where it doesn't, and what to build next.
This is independent research. Not financial advice. Not career advice. Not the final word on anything. It's one person's perspective shaped by building, shipping, and paying attention. Do your own thinking. Challenge the ideas. Disagree with specifics. Build on what resonates. Discard what doesn't.
House rules
What doesn't belong here: hate, trolling, bad-faith takedowns, or drive-by takes from people who've never built anything. If you disagree, show your work. Builder's critique. Everyone else just comments.
A MainArc playbook on measuring the one thing AI cannot make cheap. Run a small panel, map where your judges agree and where they split, and learn which parts of your taste are about to become free and which parts are your edge.
PlaybookNo. 05Version1.0Read19 minSources6 citedByMainArc ResearchEvidence and method →
Thesis
Benchmark your own judgment, not the models. The part of your taste your panel argues about is the part worth keeping.
Do this Monday
Pick one live decision. Name five judges, and put an unedited AI default in the set as a control.
What everyone agrees is good, models learn. It becomes the default, then it becomes free.
The edge
Disagreement is the edge.
Where informed judges split, a point of view lives. It stays scarce, and it is worth owning on purpose.
At a glance
A taste quadrant, built from a five-person panel. It is live. Click any score to change it and watch each direction move. Then switch to averages only and see what a normal review would have told you.
Taste quadrant · sample panel
Click a score to raise it. It wraps from 5 back to 1.
Sample data. Thresholds: average 3, spread 1.0
4Quadrants
5Stages
11Steps
5Judges is enough
What this is
The industry is busy benchmarking models. Which one writes better, designs better, ranks higher with raters. Useful work, and almost none of it tells a team what to do with its own judgment.
This playbook turns the benchmark around. Instead of scoring models, a team scores its own options with a small panel of people whose taste matters to it. Then it looks at the part every review throws away: the disagreement. Where judges agree, the standard is shared and can be written down, automated and handed to agents. Where informed judges split, there is a point of view. That is the part of the work no model will produce by default, because models are trained on what people agree on.
Most reviews average scores. Averages erase exactly the signal this playbook is about.
Principles
01
Benchmark the judges
The scarce input is judgment, so measure judgment. Models are only one of the things being judged.
02
Never stop at the average
Two options with the same score can mean opposite things. Always look at the spread.
03
Automate the floor
Whatever your panel agrees on, write down and hand to rules and agents. It is about to be free anyway.
04
Decide the edge on purpose
A split among informed judges is a choice to make, never a compromise to average.
05
Choose who judges
Experts and audiences judge differently. The panel is a design decision, so make it deliberately.
06
Re-run it
Edges decay into floors. A benchmark run once is a snapshot. Run quarterly, it is a practice.
How to use this playbook
01
Break the quadrant first
Move one judge's scores and watch a safe option turn into a point of view.
02
Read the thesis
Agreement is the floor, and every edge decays. The method makes more sense after both.
03
Run it in a week
Five judges, twelve options, three instruments. The plan is in Run it.
04
Use the script
The Templates page has a scoring sheet and a short analysis script that draws your quadrant.
Playbook 05 in the set. Playbook 01 writes taste down. This one measures it, and shows which parts are worth writing down at all.
Models are tuned toward what people agree on. Whatever your judges agree on is already on its way to becoming the default.
How consensus gets learned
Modern models are refined with human preferences. People compare outputs and rank them, a reward model learns those rankings, and the model is tuned toward what the reward model scores highly [1]. The pull is toward the answer most raters accept. The same tuning that makes models more reliable also makes their outputs less varied [2].
Fig 0.1Where dissent goesMove to aim · press to dissentranked 0 · dissent in 0, out 0
Cups and chutes: eight raters ranking outputsRed pellet: a rater who disagreedDrum: the reward model; its sight glass shows the mixDish: what the tuned model producesA simplified view of preference tuning, after [1]. Every ranking goes into the drum, red ones too, but the mix comes out one grey: dissent is averaged out at the reward model. Sample rounds.
The effect reaches people too. In a large writing experiment, AI ideas made individual stories better rated and the whole set of stories more alike [3]. Everyone gets better at the same thing.
Averages erase edges
Most reviews collapse a panel into one number. That number hides the most important difference two options can have.
Fig 0.2Same average, opposite meaningMove to slide · press to liftedges at 3.00 · both tip right
Puck: one judge's score, placed at 1 to 5Pointer: level means the knife edge sits at the averageBack beam scores 3, 3, 3, 4, 4. Front beam scores 1, 1, 5, 5, 5Both beams balance at 3.4, so an average review ranks them as equal. Lift them and let go: the front beam rocks slower, because its weight sits far from the middle. One is a safe standard. The other is a point of view.
MainArc Research
Production is free. Judgment isn't. And the part of judgment that stays valuable is the part your panel argues about.
Two numbers for every option: how much the panel liked it, and how much the panel disagreed. Together they say what to do next.
Fig 0.3Four places an option can landMove to aim · press to droppicking option 1 of 8
01 Shared standard: high average, judges agree02 Point of view: high average, judges split03 Shared rejection: low average, judges agree04 Noise: low average, judges splitToken: one option, placed by average (along) and spread (back)The floor sits bottom right. The edge sits top right. Each token carries five judges' scores; the gantry sets it down at their average and their spread. Sample scores.
Quadrant
What it means
What to do
Shared standard
Your panel agrees it is good. So would most people, and so will models.
Write it as a rule. Hand it to agents and gates. Stop debating it.
Point of view
Informed judges split hard. Some love it, some reject it.
Read the rationales. Decide whether this is who you are. If yes, commit fully.
Shared rejection
Everyone agrees it is weak.
Add the pattern to the kill list so nobody makes it again.
Noise
Low scores and wide disagreement. Judges are not judging the same thing.
Rewrite the brief or the criterion, then score again.
Not every split is a point of view. Disagreement can mean judges know different things, or that the option is simply confusing. The rationales tell you which. A point of view has reasons on both sides. Noise has none.
Today's point of view is next year's shared standard. Taste is a practice of finding the next disagreement worth having.
A bold choice that works gets copied. Copies get rated. Ratings become training data. What was once a split among judges drifts down the quadrant into the floor, where it becomes the default anyone can generate.
Fig 0.4The drift from edge to floorPress to make a copynew edge · 0 copies
Ledge: the edge, where a new point of view landsToken: one choice, one notch down per copyFloor: shared standards anyone can generateCounter: copies made so farA model of the drift. MainArc's thesis, not a measurement. Each copy moves a choice one notch down, until it drops into the floor and a new edge lands on the ledge.
What this means for a team
Three things follow. First, the floor keeps rising, so work that only clears the floor is worth less every quarter. Second, an edge is temporary, so a brand cannot rest on one bold choice. Third, the skill that compounds is finding the next disagreement worth having before everyone else agrees with it.
The shift
A taste benchmark run once tells you where you stand. Run every quarter, it shows you which of your choices are sliding into the floor, and where your next edge might be.
A panel is a design decision. Experts and audiences judge differently, and both matter for different reasons.
WhoFounder or creative lead
BudgetOne hour
InThe decision you are making
Outpanel.md with five names
Do
Pick two people whose taste the company already trusts. Usually a founder and a creative lead.
Pick two people from the audience the work is for. Customers, not colleagues pretending to be customers.
Pick one deliberate outsider: someone with strong taste and no stake in the answer.
Write down why each person is on the panel. That reason is how you will read their scores.
Why mix them
In a study of creativity ratings, experts agreed with each other far more than non-experts did, and the two groups' ratings barely correlated [5]. A panel of only experts tells you about the craft. A panel of only customers tells you about reach. You need to see both, and where they split.
Under 0.3 weak, 0.3 to 0.6 moderate, over 0.6 strong
The thresholds are starting points. On a five-point scale with five judges, a spread above 1.0 usually means at least two judges sit two or more points apart. Adjust them once you have run the benchmark twice.
Done when
Every option has an average, a spread and a quadrant
One real decision, five judges, five working days.
Setup prompt
Help me run a taste benchmark on one real decision.
1. Ask me what decision we are making and who the work is for. Then help me pick five judges: two whose taste we trust, two from the audience, one outsider. Save as bench/panel.md with a reason for each.
2. Draft bench/criteria.md with three criteria (fit, craft, distinctiveness), each defined in one sentence with an example of a 1 and a 5.
3. Help me build a set of 10 to 12 options in bench/set/: our current answer, one unedited AI default, two safe options, three bold ones built to split opinion, and real alternatives. Blind them with neutral letters and keep the key in bench/key.csv.
4. Create bench/scores.csv with columns judge, option, criterion, score. After scoring, compute each option's average and spread, the panel's Kendall's W, and draw the quadrant.
5. Do not invent any scores or judges. Wait for real data.
Day
Steps
What happens
D1
1.1, 1.2
Choose the panel. Write the criteria. Brief judges one by one.
D2
2.1, 2.2
Build the set, including the AI default. Blind and shuffle it.
D3
3.1
Judges score alone. Scores are locked by end of day.
D4
4.1, 4.2, 3.2
Compute and draw the quadrant. Panel meets to read the splits.
D5
5.1, 5.2
Rules and bans written. One or two edges committed in edge.md.
Which option would you most want to put your name on? Why, in one sentence?
Which option would you reject hardest? Why, in one sentence?
Write what you see in the work, not what you assume about who made it.
bench/quadrant.py
import csv, statistics as st
from collections import defaultdict
rows = list(csv.DictReader(open("bench/scores.csv")))
crit = "distinctiveness" # or average across criteria
by_opt = defaultdict(list); by_judge = defaultdict(dict)
for r in rows:
if r["criterion"] != crit: continue
s = int(r["score"]); by_opt[r["option"]].append(s); by_judge[r["judge"]][r["option"]] = s
def quadrant(mean, spread):
if mean >= 3: return "point of view" if spread > 1.0 else "shared standard"
return "noise" if spread > 1.0 else "shared rejection"
for opt, s in sorted(by_opt.items()):
m, sd = st.mean(s), st.pstdev(s)
print(f"{opt} avg {m:.1f} spread {sd:.2f} {quadrant(m, sd)}")
# Kendall's W with tie correction
opts = sorted(by_opt); n, m = len(opts), len(by_judge)
R = defaultdict(float); T = 0
for j, sc in by_judge.items():
vals = sorted(sc[o] for o in opts)
rank = {v: (vals.index(v) + 1 + len(vals) - vals[::-1].index(v)) / 2 for v in set(vals)}
for o in opts: R[o] += rank[sc[o]]
T += sum(vals.count(v) ** 3 - vals.count(v) for v in set(vals))
mean_R = m * (n + 1) / 2
S = sum((R[o] - mean_R) ** 2 for o in opts)
W = 12 * S / (m ** 2 * (n ** 3 - n) - m * T)
print(f"Kendall's W {W:.2f}")
Research · Playbook 05
Research
What this playbook rests on, how to measure whether it works for you, and what it cannot yet claim.
Every core claim in this playbook, with what supports it. Grades describe the kind of evidence, not how much we like the claim. Claims graded P come from practice and are labelled as such.
3A · Experiment
0B · Field data
3C · Framework
2P · Practitioner
Claim
Grade
What supports it
Source
Models are tuned toward what most raters prefer.
C · Framework
Preference tuning trains a reward model on human rankings of outputs, then tunes the model toward what that reward model scores highly [1]. This describes the method, not its effect on taste.
1
Preference tuning makes outputs less varied.
A · Experiment
Models trained with preference tuning produced much less varied outputs than models trained without it, in exchange for better generalisation [2].
2
AI help makes people's work more alike.
A · Experiment
In a 293-writer experiment, stories written with AI ideas were rated better individually and were more similar to each other as a set [3].
3
Independent ratings from informed judges are a sound way to measure creative quality.
C · Framework
The consensual assessment technique measures creativity through agreement among independent judges familiar with the domain [4].
4
Who judges changes the answer.
A · Experiment
Experts agreed with each other far more than non-experts when rating creative work, and the two groups' ratings were only weakly correlated [5].
5
Kendall's W measures how far a panel agrees on a ranking.
C · Framework
The statistic was introduced for exactly this problem: several judges ranking the same set [6].
6
Disagreement among informed judges marks a point of view worth owning.
P · Practitioner
MainArc's thesis. The benchmark is built to test it: points of view should hold up across reruns, while noise should not.
Practice
Today's edge becomes tomorrow's floor.
P · Practitioner
MainArc's thesis. Consistent with [2] and [3], unmeasured as a drift over time.
Practice
References
[1]Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., et al. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35 (NeurIPS 2022). arxiv.org/abs/2203.02155
[2]Kirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Hambro, E., Grefenstette, E., & Raileanu, R. (2024). Understanding the effects of RLHF on LLM generalisation and diversity. International Conference on Learning Representations (ICLR 2024). arxiv.org/abs/2310.06452
[3]Doshi, A. R., & Hauser, O. P. (2024). Generative AI enhances individual creativity but reduces the collective diversity of novel content. Science Advances, 10(28). doi.org/10.1126/sciadv.adn5290
[4]Amabile, T. M. (1982). Social psychology of creativity: A consensual assessment technique. Journal of Personality and Social Psychology, 43(5), 997–1013. doi.org/10.1037/0022-3514.43.5.997
[5]Kaufman, J. C., Baer, J., Cole, J. C., & Sexton, J. D. (2008). A comparison of expert and nonexpert raters using the Consensual Assessment Technique. Creativity Research Journal, 20(2), 171–178. doi.org/10.1080/10400410802059929
[6]Kendall, M. G., & Babington Smith, B. (1939). The problem of m rankings. The Annals of Mathematical Statistics, 10(3), 275–287.
Grades
A controlled experiment. B large-scale field data. C an established peer-reviewed model or guideline, not a direct test of the claim. P practitioner claim from MainArc's work, untested.
A taste benchmark is itself a measurement. Run these checks before you trust what it says.
Instruments
01
Rescore
Ask every judge to rescore a fifth of the set a week later, blinded and shuffled. Points of view should hold. Noise usually moves.
02
The AI control
Every set includes an unedited AI default. If it lands in shared standard, your floor is already free.
03
Panel swap
Rerun with one different audience judge. An edge that disappears when one person changes was never an edge.
04
Order check
Compare scores for options seen first with options seen last. A strong first-seen bias means the shuffle failed.
Study design
Element
Setting
Rescore share
20% of options, one week later
Panel
Five judges, one swapped per rerun
Set
10 to 12 options including one AI default
Analysis
Average and spread per option. Kendall's W for the panel. Position change between runs.
Cadence
Every quarter
Reading the results
Trust a point of view only if it survives a rescore and a panel swap.
If the AI default keeps landing in shared standard, raise your floor: everything there is now table stakes.
If W is weak and most splits read as noise, the criteria are unclear. Fix criteria.md before running again.
Running signals
Signal
What to watch for
Share in shared standard
Rising over quarters means your work is converging on the floor
Committed edges still in the top right
Edges that slide are decaying. Start looking for the next
Rules written from the floor
Every shared standard should become a rule within a week
Note
Thresholds here are starting points to tune, not findings. A small panel shows direction and disagreement. It does not prove significance, and nothing on this page should be reported as if it did.
How this playbook was built, where it is weak, and how to cite it.
How this playbook was built
Built on the consensual assessment tradition in creativity research, published work on preference tuning and homogenization, and MainArc's practice. It reverses the usual direction of AI benchmarks: it scores a team's options with a team's judges, and treats disagreement as signal. It is a practitioner method. Claims are graded on the evidence page.
Limitations
Five judges show direction and disagreement, not statistical significance.
The thresholds (average 3, spread 1.0) are working defaults, not validated cut-offs.
Disagreement can come from ignorance as easily as from taste [5]. The rationale step exists to tell them apart, and it relies on judgment.
The drift from edge to floor is a thesis, not a measured effect.
The quadrant on the introduction page uses sample scores, not data from any real panel.
Open questions
How fast do edges decay in different creative domains?
Can a team protect a point of view while training its own models on its own work?
What mix of experts and audience produces the most useful splits?
Cite this playbook
APA
MainArc Research. (n.d.). Benchmark your taste (Playbook No. 05, Version 1.0). MainArc. https://mainarc.xyz
BibTeX
@misc{mainarc_taste,
author = {{MainArc Research}},
title = {Benchmark your taste},
howpublished = {MainArc Playbook No. 05, Version 1.0},
url = {https://mainarc.xyz}
}
Corrections
Found an error, a stronger source, or a result from running the benchmark? Send it through mainarc.xyz. Corrections are made in the next version.
About this playbook
This playbook is published by MainArc, a creative intelligence lab for leadership and product teams.
We do three things. We research how AI is changing creative work through long-form reports, design teardowns, and operator essays. We design the workflows, prompt systems, review loops, and decision frameworks that teams actually run on. And we build the work the strategy points to: brand systems, interfaces, product prototypes, internal tools, and launch concepts.
One operator. One AI agent. No layers in between. Every engagement runs under strict NDA. No case studies, no client logos, no shared work.
If this research resonates with a problem your team is facing, start with an audit. One week. A clear map of where AI fits, where it doesn't, and what to build next.
This is independent research. Not financial advice. Not career advice. Not the final word on anything. It's one person's perspective shaped by building, shipping, and paying attention. Do your own thinking. Challenge the ideas. Disagree with specifics. Build on what resonates. Discard what doesn't.
House rules
What doesn't belong here: hate, trolling, bad-faith takedowns, or drive-by takes from people who've never built anything. If you disagree, show your work. Builder's critique. Everyone else just comments.
A MainArc playbook for founders, product leaders and PMs. Bolting AI onto yesterday's workflow is the new legacy software. Map every step your product makes people take, decide which ones should vanish, run on their own, or stay with a person, and rebuild the product around the decisions that remain.
PlaybookNo. 06Version1.0Read25 minSources6 citedByMainArc ResearchEvidence and method →
Thesis
Bolting AI onto yesterday's workflow is the new legacy software. When the work inside a product becomes nearly free, rebuild it around the decisions that remain.
Do this Monday
Watch someone do your most-used workflow. List every step, mark each one Gone, Automated, Judgment or Human for trust, and count what is left.
A workflow rewriter. It is live. Pick a sample workflow and every step arrives with a recommended fate. Change any of them and watch the old flow collapse, or refuse to. Then press Bolt-on to see what most products ship today: the same seven steps with an assistant attached to one of them.
Workflow rewriter · sample workflows
GoneAutomatedJudgmentHuman for trust
Checks
Sample workflows. Minutes are illustrative estimates
4Fates for a step
5Stages
10Steps
3Templates
What this is
Most products have added AI the same way. A button that drafts, a sidebar that answers, a summary at the top of a page. Every step of the old workflow is still there. The person still walks through all of them, a little faster.
That old workflow was designed when every step was work a person had to do. Retyping, formatting, searching, summarising, reporting. Much of that work is now close to free. A product that keeps those steps and decorates them is carrying the shape of a world that has gone. That is what legacy software has always been.
This playbook starts from the other end. Find the job the product is hired for. Map every step. Give each one a fate. Then rebuild around what is left, which is mostly decisions, and test the new shape against the old one with real people.
Production is free. The product's job is to put the right decision in front of the right person.
Principles
01
Start from the job
Customers hire a product to make progress. The feature list is how you served the job when work was expensive.
02
Every step earns its place
If a step exists only because producing something used to take effort, delete it.
03
Automate production, show the work
Steps that run on their own leave a record a person can check. Invisible work is unowned work.
04
Design around decisions
The core screen is a decision with its evidence, a recommendation and the cost of each option.
05
Keep people where people are the point
An apology, a kickoff call, a rejection. Some steps are valuable because a person did them.
06
Race the old flow
A redesign is a claim. Real users doing real tasks in both versions settle it.
How to use this playbook
01
Break the rewriter first
Open Screen job applicants and automate the rejection. Then press Bolt-on and read the sentence it writes.
02
Read the three essays
Why the sidebar fails, what gets cheap, and the four fates. The stages assume all three.
03
Run it in two weeks
One workflow, ten steps, a prototype and a race against the old flow. The plan is in Run it.
04
Copy the templates
A workflow map, a job statement and the inversion worksheet are on the Templates page.
Playbook 06 in the set. Playbook 04 covers how to let agents act safely. This one decides which steps should be acting on their own at all.
Attaching AI to the old steps speeds up each one and leaves the shape untouched. The shape is the problem.
The work inside the steps got cheap
The evidence on production is consistent. In a preregistered experiment with professional writing tasks, people given an AI assistant took 40% less time and produced work rated 18% higher in quality [1]. In a study of more than five thousand support agents, AI assistance raised issues resolved per hour by 15% on average, with the largest gains for less experienced workers [2].
Read those numbers as a product team. The work your product asks people to do, drafting, searching, summarising, classifying, is the kind of work that just got cheaper. If the product keeps every step and adds a helper beside each one, the person still walks the whole path. The saving goes to each step. The path itself stays.
Fig 0.1Bolted on, or rebuiltMove to scrub · press to runbolted 0 of 7 · rebuilt 0 of 3
Seat: where the person worksBack · bolted on: seven stations, a helper clamped on 04Front · rebuilt: two bare mounts, three presses that run alone with receiptsTwo decision levers and a hand-off counter: the calls that matterAn illustration, not a measurement. The bolted-on product saves time per step. The rebuilt one removes steps and keeps the person for the calls that matter. One job runs down both lines; watch where the seat has to go.
The help is uneven, step by step
There is a second reason a uniform sidebar fails. AI help is jagged. In a field experiment with 758 consultants, people using AI on tasks inside its capabilities finished 12.2% more tasks, 25.1% faster, at higher quality. On a task selected to sit outside those capabilities, the same help made them 19% less likely to reach a correct answer [3]. The authors saw that unevenness even inside a single workflow [3].
A sidebar treats every step the same. Some steps should vanish. Some should run without a person. Some should get more of the person's attention, because a confident wrong answer is now cheap to produce. The only way to tell them apart is to look at each step on its own.
MainArc Research
Production is free. Judgment isn't. A product that spends its people on production, and hides its decisions in a sidebar, has the economics backwards.
When something gets cheap, the things it works with get more valuable. Prediction and production are getting cheap. Judgment works with both.
One clear account of what AI changes treats it as a drop in the cost of prediction [4]. Prediction is filling in what you do not know: what this customer will probably do, what this document probably says, what the next sentence probably is. A decision needs more than that. It needs someone to say what each outcome is worth, including the cost of being wrong. That is judgment, and as prediction gets cheaper, it becomes the scarce input [4].
Fig 0.2Where the value movesMove to set prediction costcost 1.00 · 0 wait for judgment
01 Data · hopper, fed by the return belt02 Prediction · press, its speed set by the cost slider (relative, sample scale)03 Judgment · balance: the rider says what each outcome is worth04 Action · stamp, then 05 Outcome · bin, back to the hopper as new dataA simplified anatomy of a decision, after [4]. The cheaper the prediction, the more the judgment beside it decides the result. Drop the cost and the predictions queue at the one balance.
Production follows the same curve
Products do more than predict. They produce: drafts, summaries, reports, configured accounts, replies. Production rides on the same falling cost, and the evidence on page 0.1 shows it. So the same logic applies one level up. When the making inside a product gets cheap, the deciding inside it becomes what the product is for.
Getting cheap
Getting valuable
Drafting a reply
Deciding whether this customer gets an exception
Summarising a CV
Deciding who is worth an interview
Building the monthly deck
Deciding what the numbers mean for next month
Configuring an account
Deciding what the customer is actually trying to achieve
Flagging a risk
Deciding what to do about it, and who does it
The left column is where most AI features live today. The right column is where the product's value is moving. An AI-native product is one whose interface is built around the right column.
The shift
The old product helped people make things. The new product helps them decide things, and makes the rest on its own.
Every step in a workflow ends up in one of four places. Gone, Automated, Judgment, or Human for trust. Deciding which is the work of this playbook.
Fig 0.3Three questions, four fatesPress to drop the next step8 steps in · ask in order
Gate 1 · Gone: would it exist if making its output were free? NoGate 2 · Automated: is there a choice with a real consequence? NoGate 3 · Human for trust: is the point that a person did it? YesEnd · Judgment: no gate settles itToken: one workflow step, its stud in the lane of the first answer that settles it (sample of eight)Ask in order. The first answer that settles it wins. MainArc's framework, from practice. Each step rides past the gates in turn and drops into the bin of the first question that settles it.
Fate
What it means
Typical steps
Ships as
Gone
The step existed because producing its output took effort, or because two systems did not share data.
Re-entry, status reports, copying between tools
Nothing. Memory and connections replace it
Automated
Still needed. Its output can be checked, and a wrong one is cheap to fix.
Tagging, drafting routine replies, building charts
An agent, with a receipt
Judgment
A choice with a consequence. The product's job is to bring it to the right person with the evidence.
Exceptions, shortlists, pricing, what to do next
A decision card in a queue
Human for trust
The value is that a person did it. Relationship, accountability, care.
Kickoff calls, apologies, rejections, sign-off
A briefed person, at the right moment
Some steps are Human for trust by law as well as by choice. In the EU, people have a right not to be subject to a decision based solely on automated processing that significantly affects them, with limited exceptions, and where those apply they keep the right to reach a person and contest it [5]. Check the rules where you operate. This playbook is not legal advice.
MainArc Research
Gone and Automated are where production goes to become free. Judgment and Human for trust are what the product is for once it does.
Customers hire a product to make progress in a particular situation. Write that progress down, with no feature names in it.
WhoFounder, PM
BudgetHalf a day
InFive recent customer conversations
Outjob.md
Do
Pick the one workflow customers use most. Start there and nowhere else.
Ask customers what was happening the day they started using it, and what they were trying to get done.
Write the job as progress in a circumstance: When [situation], I want to [progress], so I can [outcome]. A job is the progress a customer is trying to make in a given circumstance [6].
Add the social and emotional side. Who is watching, and what does the customer need to feel? Jobs are never only functional [6].
Remove every feature name. If the job statement mentions your product, it describes your solution, and the solution is what you are about to change.
Done when
job.md has one job statement with no feature names
Three customers read it and say that is what they were trying to do
Prompt
Read these customer conversation notes. Write the job our product is hired for as one sentence: When [situation], I want to [progress], so I can [outcome]. Add who else is watching and what the customer needs to feel. Do not mention our product or any feature. Show me three versions and tell me which quotes support each.
Every job has one or two moments where getting it right matters. The AI-native product is built around those moments.
WhoPM, designer
BudgetTwo hours
Injob.md
Outdecisions.md
Do
Walk through the job and ask: where does someone make a call that is expensive to get wrong?
For each call, write who makes it, what they need to know, and what happens if they are wrong.
Split each one into prediction (what is likely) and judgment (what each outcome is worth, and what to do) [4]. The prediction half is the part a machine can now supply.
Rank the calls by cost of error. The top one is the centre of the new product.
Decision · sample
Prediction it needs
Judgment it needs
Refund or exception
How likely is this customer to leave?
Is keeping them worth bending the policy?
Interview shortlist
Who meets the written criteria?
Who would make this team better?
Next month's plan
Where are the numbers heading?
Which change matters, and what do we do about it?
Done when
decisions.md lists every costly call in the job, each split into prediction and judgment
Map the work as it is really done, including the copying, the waiting and the workarounds nobody put in the process document.
WhoPM, with someone who does the work
BudgetHalf a day
InSessions, logs or recordings
Outworkflow.csv
Do
Watch two or three people do the job end to end. Map what they do, not what the process says.
Record every step, including copying between tools, waiting for someone, and checking a second screen.
For each step note who does it, roughly how many minutes it takes, what it produces, and whether it contains a decision.
Mark every step that exists only because two systems do not share data. These are the easiest wins in the playbook.
End the map when the customer's job is done, which is often later than when your product thinks it is.
Done when
Every step has an owner, minutes and an output
Someone who does the work agrees the map is real
Prompt
Here are my notes from watching people do this workflow. Turn them into workflow.csv with columns step, owner, minutes, produces, has_decision, exists_because. Keep every step, including copying, waiting and checking. Where I did not record minutes, leave the cell empty. Do not estimate.
The inversion. Ask three questions of every step, in order, and write down why it landed where it did.
WhoPM, designer, engineer
BudgetTwo hours
Inworkflow.csv, decisions.md
Outinversion.csv
Ask
If the answer is
Fate
Would this step exist if making its output were free?
No
Gone
Is there a choice in it with a real consequence?
No
Automated
Is the point that a person did it?
Yes
Human for trust
Otherwise
Judgment
Do
Ask the questions in order for every step. The first one that settles it wins.
Be harsh with Gone. Most reports, re-entries and handoffs exist because work used to be expensive.
Write one line of reason per fate. A fate without a reason is a guess.
Mark every step that touches someone's money, job, health or legal standing. These start at Judgment or Human for trust, and some cannot be fully automated by law [5].
Test each Automated step on real cases before you trust it. Help that works on one step can make the next one worse [3].
Done when
Every step has a fate and a one-line reason
No step that affects someone's livelihood is marked Gone or Automated without a person able to step in
Prompt
Read workflow.csv and decisions.md. For each step, ask in order: would it exist if making its output were free? Is there a choice with a real consequence? Is the point that a person did it? Assign Gone, Automated, Human for trust or Judgment, with a one-line reason. Flag any step that affects money, jobs, health or legal standing. Save as inversion.csv and list the steps you were least sure about.
Automated steps go to agents that leave receipts. Judgment steps get a decision card built for the call in front of the person.
WhoDesigner, engineer
BudgetHalf a day
Inprimitives.md, decisions.md
OutAgent list, decision card spec
Do
Give every Automated step to an agent with a written scope and a receipt. Playbook 04 covers how far each action may go on its own.
For every Judgment step, design a decision card: the call, the evidence, a recommendation, and what each option costs.
Generate the view for the decision, never a dashboard of everything. Playbook 02 covers views that compose themselves.
Leave Human for trust steps with a person, and prepare them. The person arrives at the call already briefed.
Decision card · sampleSupport refund
The call
Refund an order outside the 30-day window?
Recommendation: refund, as a one-off
Owner: the support agent on the ticket
Evidence and cost
Customer for three years, no previous exceptions
Item reported faulty on day 34
Refund: one order's value. Refuse: likely loss of the account
Done when
Every Automated step has an agent, a scope and a receipt
Every Judgment step has a decision card spec
Prompt
Read inversion.csv. For each Judgment step, draft a decision card: the call in one question, the three pieces of evidence that matter most, a recommendation, the owner, and what each option costs. For each Automated step, write the agent's scope in one sentence and the receipt it leaves. Mark anything you had to guess.
Build the new shape, thin and real. A home screen of decisions, one decision card that works end to end, and the automation behind it, faked if necessary.
One decision queue, one decision card, real data. Thin enough to build in days, real enough for people to do their actual job in it.
WhoEngineer, designer
BudgetThree days
InThe sketch from 4.1
OutA working prototype
Do
Build the decision queue and the top decision card end to end. Nothing else.
Fake automation behind it if you need to. A teammate doing an Automated step by hand is fine for a test, as long as the user sees exactly what the real product would show.
Use real data from one team or customer, with their permission.
Log every decision: what was recommended, what the person chose, and how long it took.
Done when
A real user can complete the job in the prototype
Every decision is logged with the recommendation and the choice
Same people, same real tasks, both versions. Measure time, but also what people decided and how sure they were.
WhoPM, researcher
BudgetTwo days
InPrototype, old flow
Outrace.csv
Do
Recruit five to eight people who do the job today. Include your most experienced users, not only new ones.
Each person does the same real task in both versions. Swap the order for half of them.
Record time, errors, decisions made, and how often they overrode the recommendation.
After each run, ask one question: how sure are you that this was done right?
Measure
Why it matters
Time to done
The obvious one. Necessary, never enough on its own.
Judgment share
Share of the person's time spent on decisions and trust moments. It should rise.
Errors caught
A faster flow that lets more mistakes through is a loss.
Override rate
How often people reject the recommendation. Very high means bad recommendations. Zero may mean nobody is reading them.
Confidence
If people trust the result less, the Automated steps are hiding too much.
Watch the experts
In the support study, the least experienced agents improved in speed and quality, while the most experienced saw small gains in speed and small declines in quality [2]. Read your results by experience level. An average can hide your best people getting worse.
Done when
Every participant has done the task in both versions
race.csv has time, errors, overrides and confidence for every run
Prompt
Read race.csv. For each version, give me the median time, error count, override rate and median confidence, split by experience level. Compute judgment share from the decision logs. Tell me where the new version lost, and do not round small samples into conclusions.
One workflow, ten working days, from job statement to a race against the old flow.
Setup prompt
Help me find the AI-native version of one workflow in our product.
1. Ask me which workflow customers use most, and for notes from five recent customer conversations. Draft native/job.md: one job statement in the form "When [situation], I want to [progress], so I can [outcome]", with no feature names.
2. List every costly decision inside that job in native/decisions.md. Split each into prediction (what is likely) and judgment (what each outcome is worth, and what to do).
3. From my observation notes, build native/workflow.csv with step, owner, minutes, produces, has_decision, exists_because. Do not estimate minutes I did not record.
4. Assign each step a fate in native/inversion.csv: Gone, Automated, Judgment or Human for trust, with a one-line reason. Flag anything that affects money, jobs, health or legal standing.
5. Draft decision cards for the Judgment steps and agent scopes for the Automated ones. Then help me plan a race of the old flow against a thin prototype with five to eight real users.
6. Do not invent customers, quotes or results. Wait for real data.
Day
Steps
What happens
D1
1.1, 1.2
Job statement written and checked. Decisions listed and ranked.
D2
2.1
Watch the work. Build workflow.csv.
D3
2.2
The inversion session. Every step gets a fate and a reason.
D4
3.1, 3.2
Memory list, agent scopes, decision card spec.
D5 to D7
4.1, 4.2
Decision queue sketched. Thin prototype built on real data.
D8 to D9
5.1
Race the old flow with five to eight real users.
D10
5.2
Decide. Set a removal date for the old flow, or list the fixes.
The workflow map, the job statement and the inversion worksheet. Copy them, then fill them with what you observed.
native/workflow.csv · sample rows
step,owner,minutes,produces,has_decision,exists_because
Re-enter signup details,CS lead,10,CRM record,no,two systems do not share data
Configure the account,CS lead,40,configured account,no,
Choose plan and setup,CS lead,20,plan choice,yes,
Kickoff call,CS lead,45,relationship,yes,
Write status report,CS lead,15,report,no,manager needs visibility
...
native/job.md
## The job
When [situation],
I want to [progress],
so I can [outcome].
## Who else is watching
[Boss, team, customer, regulator]
## What it must feel like
[Safe, quick, in control, looked after]
## The costly calls inside it
1. [Decision] · who makes it · cost if wrong
2. [Decision] · who makes it · cost if wrong
## Check
No feature names. No product name. Three customers recognise it.
native/inversion.csv · the worksheet
step,exists_if_free,real_consequence,point_is_a_person,fate,reason,ships_as,sensitive
Re-enter signup details,no,,,Gone,data already captured at signup,memory,no
Configure the account,yes,no,,Automated,settings follow from the plan,agent + receipt,no
Choose plan and setup,yes,yes,no,Judgment,wrong plan costs the account,decision card,no
Kickoff call,yes,yes,yes,Human for trust,the relationship is the point,briefed call,no
# Ask the three questions in order. The first answer that settles it wins.
# sensitive = affects money, jobs, health or legal standing.
# Sensitive steps start at Judgment or Human for trust.
Research · Playbook 06
Research
What this playbook rests on, how to measure whether it works for you, and what it cannot yet claim.
Every core claim in this playbook, with what supports it. Grades describe the kind of evidence, not how much we like the claim. Claims graded P come from practice and are labelled as such.
2A · Experiment
1B · Field data
3C · Framework
1P · Practitioner
Claim
Grade
What supports it
Source
AI assistance makes production work much faster and better.
A · Experiment
In a preregistered experiment on professional writing tasks, people given an AI assistant took 40% less time and produced work rated 18% higher in quality [1].
1
In real work, the gains are largest for less experienced people.
B · Field data
Across 5,172 support agents, AI assistance raised issues resolved per hour by 15% on average. Less experienced agents improved in speed and quality, while the most experienced saw small speed gains and small quality declines [2].
2
AI help is uneven across tasks, even inside one workflow.
A · Experiment
In a preregistered field experiment with 758 consultants, AI users completed 12.2% more tasks, 25.1% faster and at higher quality on tasks inside its capabilities, and were 19% less likely to reach a correct answer on a task selected to fall outside them [3].
3
When prediction gets cheap, judgment becomes more valuable.
C · Framework
An economic framework that treats AI as a drop in the cost of prediction, and judgment, deciding what each outcome is worth, as its complement [4]. A model, not a test of product design.
4
Some decisions must keep a person able to step in.
C · Framework
EU law gives people a right not to be subject to decisions based solely on automated processing that significantly affect them, with exceptions that require safeguards including human intervention [5]. Primary source. Rules differ by market.
5
Products are hired for a job: progress in a circumstance.
C · Framework
The jobs to be done framework defines a job as the progress a customer is trying to make in a given circumstance, with functional, social and emotional dimensions [6].
6
Rebuilding a product around the decisions that remain beats bolting AI onto the old steps.
P · Practitioner
MainArc's thesis, with the four fates as its method. Consistent with [1], [3] and [4], untested as a product strategy. The race in Stage 05 is designed to test it on your own product.
Practice
References
[1]Noy, S., & Zhang, W. (2023). Experimental evidence on the productivity effects of generative artificial intelligence. Science, 381(6654), 187–192. doi.org/10.1126/science.adh2586
[2]Brynjolfsson, E., Li, D., & Raymond, L. (2025). Generative AI at work. The Quarterly Journal of Economics, 140(2), 889–942. doi.org/10.1093/qje/qjae044
[3]Dell'Acqua, F., McFowland, E., III, Mollick, E., Lifshitz, H., Kellogg, K. C., Rajendran, S., Krayer, L., Candelon, F., & Lakhani, K. R. (2026). Navigating the jagged technological frontier: Field experimental evidence of the effects of artificial intelligence on knowledge worker productivity and quality. Organization Science, 37(2), 403–423. Earlier version: Harvard Business School Working Paper 24-013 (2023). doi.org/10.1287/orsc.2025.21838
[4]Agrawal, A., Gans, J., & Goldfarb, A. (2018). Prediction machines: The simple economics of artificial intelligence. Harvard Business Review Press.
[5]Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 (General Data Protection Regulation), Article 22. Official Journal of the European Union, L 119, 1–88. eur-lex.europa.eu/eli/reg/2016/679/oj
A controlled experiment. B large-scale field data. C an established peer-reviewed model or guideline, not a direct test of the claim. P practitioner claim from MainArc's work, untested.
An AI-native redesign is a claim about people doing a job. Test it the way the job is done: same people, same tasks, both versions.
Instruments
01
Time to done
Minutes from start to the customer's job being done, in each version, on the same real task.
02
Judgment share
Share of the person's time spent on decisions and trust moments, from session logs. It should rise in the new version.
03
Errors and overrides
Mistakes that reach the customer, and how often people reject the recommendation on a decision card.
04
Confidence
One question after each run, 1 to 5: how sure are you this was done right?
Study design
Element
Setting
People
5 to 8 who do the job today, including the most experienced
Tasks
3 real tasks per person, each done in both versions
Order
Counterbalanced: half start in the old flow, half in the new
Analysis
Medians per version, split by experience level. Override rate per decision card.
Cadence
Before launch, and again four weeks after the old flow is removed
Reading the results
Ship only if time falls, errors hold or fall, and confidence holds. Speed alone is not a win.
If confidence falls, the Automated steps are hiding too much. Add receipts before adding features.
If experienced users get worse while new users get better, the decision card is talking down to experts. Give them the evidence and the override.
Running signals
Signal
What to watch for
Decisions per session
Should be few and real. A queue that grows without end is a task list in disguise
Override rate
Very high means bad recommendations. Zero may mean nobody reads them
Old flow usage
Should fall to zero by the removal date
Note
Thresholds here are starting points to tune, not findings. A small panel shows direction and disagreement. It does not prove significance, and nothing on this page should be reported as if it did.
How this playbook was built, where it is weak, and how to cite it.
How this playbook was built
Built from published research on AI and productivity, the economics of prediction and judgment, the jobs to be done framework, and MainArc's practice in product strategy. The four fates are MainArc's framework. It is a practitioner method. Claims are graded on the evidence page.
Limitations
The productivity studies [1, 2, 3] measure people doing tasks with AI help. None tests a product redesigned around decisions, which is the playbook's central claim.
The four fates are a framework from practice. Real steps can sit between fates, and the tests are judgment calls.
The workflow rewriter uses sample workflows and illustrative minutes. Its judgment share is a teaching device, not a measurement.
The legal point [5] covers the EU only and is simplified. Nothing here is legal advice.
Open questions
How much of a product's value moves to the decision layer as production costs keep falling?
Do people trust a decision queue more or less than a familiar step-by-step flow, and does that change with time?
When does automating a step quietly turn a decision into a default nobody owns?
Cite this playbook
APA
MainArc Research. (n.d.). Find the AI-native version of your product (Playbook No. 06, Version 1.0). MainArc. https://mainarc.xyz
BibTeX
@misc{mainarc_native,
author = {{MainArc Research}},
title = {Find the AI-native version of your product},
howpublished = {MainArc Playbook No. 06, Version 1.0},
url = {https://mainarc.xyz}
}
Corrections
Found an error, a stronger source, or a result from running the benchmark? Send it through mainarc.xyz. Corrections are made in the next version.
About this playbook
This playbook is published by MainArc, a creative intelligence lab for leadership and product teams.
We do three things. We research how AI is changing creative work through long-form reports, design teardowns, and operator essays. We design the workflows, prompt systems, review loops, and decision frameworks that teams actually run on. And we build the work the strategy points to: brand systems, interfaces, product prototypes, internal tools, and launch concepts.
One operator. One AI agent. No layers in between. Every engagement runs under strict NDA. No case studies, no client logos, no shared work.
If this research resonates with a problem your team is facing, start with an audit. One week. A clear map of where AI fits, where it doesn't, and what to build next.
This is independent research. Not financial advice. Not career advice. Not the final word on anything. It's one person's perspective shaped by building, shipping, and paying attention. Do your own thinking. Challenge the ideas. Disagree with specifics. Build on what resonates. Discard what doesn't.
House rules
What doesn't belong here: hate, trolling, bad-faith takedowns, or drive-by takes from people who've never built anything. If you disagree, show your work. Builder's critique. Everyone else just comments.
A MainArc playbook on briefing machines the way a creative director briefs a team. A prompt asks for output. A direction says what the work is for, what to borrow, what to refuse, how many routes to explore and how you will choose. Only the second gets work with a point of view.
PlaybookNo. 07Version1.0Read24 minSources6 citedByMainArc ResearchEvidence and method →
Thesis
Generic output is a briefing problem. Brief a model the way a creative director briefs a team, with intent, references, limits, range and a test, and it can hold a point of view.
Do this Monday
Before your next generation, write one sentence: this exists so that [who] will [feel, think or do what]. Then write how you will choose.
A prompt to direction converter. It is live. Load a sample prompt or paste your own, and it checks the text for the seven parts of a direction. Every part it cannot find becomes a question in the brief beside it. Then load the strong direction and compare the two.
Prompt to direction converter · sample prompts0 / 7
Seven parts
Built from your prompt
Rule-based. It finds signals of each part and cannot judge their quality. Sample prompts.
7Parts of a direction
5Stages
10Steps
3Templates
What this is
Anyone can now get a hundred logos before lunch. Making has stopped being the hard part. The hard part is the sentence that comes before the making, and most teams write it in seconds: make a logo, modern and clean.
That sentence is an instruction. It names the thing to make and leaves every decision that matters open. A model closes those decisions anyway, with whatever answer is most common. What comes back is competent and familiar, because familiar is what an open question returns.
This playbook replaces the instruction with a direction: the brief a creative director gives a team, written for a machine. Intent, audience, references, constraints, an avoid list, a range and a success test. Seven parts, five stages, and a way to choose that holds up at four in the afternoon. Production is free. Direction is judgment, and it is the one part of the work nobody can generate for you.
Principles
01
Say why before what
Intent is the one thing a model cannot guess. Write it first, in one sentence, before anything is made.
02
Every gap gets the average
Whatever you leave open is filled with the most likely answer. Leave open only what you want explored.
03
Give references jobs
Say what to take from each reference and what to ignore. A reference without a job gets copied.
04
Ask for range on purpose
Several directions that differ in structure, side by side. Variations on one idea only look like choice.
05
Write the test before you look
Decide how you will choose before the options arrive. Otherwise the most polished one chooses for you.
06
Revise in the language of the test
A note names the check that failed and what to keep. "Make it pop" names nothing.
How to use this playbook
01
Break the converter first
Paste the last prompt you actually wrote. Count the lamps. Then load the strong direction and read the two briefs side by side.
02
Read the three essays
Why instructions converge, what a direction holds, and how a single reference traps the work.
03
Direct one real brief
Take one live job through all five stages. The plan in Run it fits in a working day.
04
Copy the templates
A direction brief, a reference board and a revision note, ready to fill in.
Playbook 07 in the set. Playbook 05 measures taste once the work exists. This one puts taste in at the start, where it costs least to use.
A vague prompt is an open question. A model answers open questions with the most likely answer, and the most likely answer is the one everyone else gets too.
Briefing a machine like a colleague
People new to prompting write the way they would brief a colleague: short, polite, trusting the other side to fill in the context. In a study of people without an AI background designing prompts, participants carried over expectations from instructing other people, explored opportunistically rather than systematically, and drew conclusions from single outputs [1]. A colleague fills a short brief with what they know about you. A model fills it with what is common.
Every gap is a default
Take "a logo for my coffee brand, modern and clean". It leaves open who the brand is for, why it exists, what it should borrow, what it must never look like, how many routes to try and how a winner gets picked. Each open question still gets answered. The answer sits near the centre of everything the model has seen for coffee, modern and clean: a bean, a cup, a sans-serif wordmark, a warm brown.
Fig 0.1Where open decisions landPress or arrows to set knobscentre 6/20 · target 20/20
Knob and peg row: one decision. Loose rows choose at randomLeft: an instruction, seven decisions left openRight: a direction, seven decisions set, one chosen binA model of the mechanism, not a measurement. Open decisions fall to the centre. Closed ones land where you put them. Each row of pegs is one decision: left loose, it sends the ball either way, so the balls pile up in the middle.
The average spreads
The effect compounds across people. In a large writing experiment, stories written with AI ideas were rated better one by one and were more similar to each other as a set [2]. When everyone asks the same open question, everyone gets a version of the same answer, and the answer gets better rated while it gets more alike.
A cleverer phrasing of an open question is still an open question. The fix is to close the decisions yourself, in writing, before anything is made.
MainArc Research
Production is free. Judgment isn't. Direction is where judgment enters the work: every decision you write down is one a model will not make for you by default.
Seven parts. Each one closes a decision a model would otherwise make for you, and together they fit on half a page.
Fig 0.2Seven parts, four jobsMove to pick · press to set0 of 7 set · aim
01 Aim · intent, audience02 Shape · references, constraints, avoid03 Spread · range04 Choose · success test, the setpointCarriage: the draft, sent back to aim while the test failsThe success test is the part most often missing, and the only one that tells you when to stop. Each knob closes one decision; the gauge passes the draft only when all seven are set and knob seven has put a mark to aim at.
Part
The question it closes
Instruction
Direction
Intent
Why does this exist?
"Modern and clean"
Make regulars feel the shop is theirs
Audience
Who is it for, exactly?
"Coffee lovers"
People within a ten-minute walk who buy beans every week
References
What should it borrow, and what not?
"Something premium"
Enamel shop signs: take the lettering, ignore the colours
Constraints
What is already decided?
Nothing said
One colour, reads at 16px, prints on kraft
Avoid
What would you reject on sight?
Nothing said
No beans, cups or steam. Use type and shape instead
Range
How many routes, and how different?
"Give me some options"
Three: a wordmark, a monogram, a badge
Success test
How will you choose?
"I'll know it when I see it"
A regular can redraw it from memory after a week
Why the test matters most
Goal-setting research has a plain and well-replicated finding: specific, difficult goals lead to better performance than being told to do your best [3]. The explanation carries straight over to briefs. A do-your-best goal has no external reference point, so a wide range of results all count as acceptable. "I'll know it when I see it" is a do-your-best goal. A success test gives the work, and the person choosing, something outside their mood to aim at.
The same research finds the effect is smaller on complex tasks, where strategy matters more [3]. Creative work is complex. That is why a direction never stops at the test: references, constraints and range are the strategy, and the test says where to land.
Working rule
A full direction is short. The strong sample in the converter has all seven parts in under 140 words. If yours runs past a page, it is a specification, and the model will treat every line as equally important.
One reference gets copied. Several references, each with a job, give range. The difference is a sentence per reference.
Examples set the shape of the search
Design fixation has a long record. In the classic study, engineers shown an example solution before designing produced work that reused the example's features, including features that did not suit the problem they were given [4]. Nobody told them to copy. The example quietly set the shape of the search.
A model takes a reference even more literally than an engineer does. Paste in one image with "like this" and you get its cousins back, flaws included.
Fig 0.3One reference, or a board with jobsPress to stamp the next part4 copies · 4 notched · 3 kinds
Left: one reference die and its flaw, a notchRight: three dies, take lettering, take warmth, take gridRam pad: picks a blank, presses it, sets it downA schematic, after [4] and [5]. Outputs that combine parts of several references differ from each other. Copies of one do not. On the left every part comes off one die, notch included; on the right each part takes a mark from two different dies.
Range in, range out
Range in the references helps. In an online ideation experiment, people shown a more diverse set of example ideas produced more diverse ideas themselves, and people shown more creative examples produced more creative ideas [5]. Neither group beat a group shown no examples at all by a significant margin. That is its own lesson: examples steer, and steering only helps if you chose where to go.
The same reference, twice
A reference
One image, pasted in
No word on what matters about it
Its flaws come back with its strengths
Every output is a cousin of it
A reference with a job
Three to five, from different places
One thing to take from each
One thing to ignore in each
Outputs combine parts, so they differ
MainArc Research
The board is the judgment. Deciding what to take from a reference is the creative act. Pasting it in without a job hands that act to the model.
One sentence: who the work is for, and what it should make them feel, think or do. If you cannot write it, nothing downstream can save the brief.
WhoCreative lead, with whoever asked
BudgetThirty minutes
InThe ask, as it arrived
OutThe first lines of direction.md
Do
Write down the ask exactly as it arrived. That is the instruction you are replacing.
Finish this sentence: This exists so that [who] will [feel, think or do what].
Make the who specific enough to picture. "People within a ten-minute walk who buy beans every week" gives a model something to aim at. "Coffee lovers" gives it the category.
Read it back to whoever asked. If they would change it, change it now, before anything is made.
Done when
One sentence names a specific person and a change in them
The person who asked agrees with it
Prompt
Here is the ask: [paste it]. Make nothing yet. Ask me up to five questions that would let us write one sentence: this exists so that [who] will [feel, think or do what]. Then propose that sentence, and wait for me to change it.
Decide how you will choose before you see anything. Once options exist, the most polished one quietly decides for you.
WhoCreative lead
BudgetThirty minutes
InThe one-sentence why
OutA success test in direction.md
Do
Write two or three checks a stranger could apply. Each one gets a pass or a fail, with no half marks.
Tie every check to the why. If a check would pass work that misses the intent, cut it.
Add one check for distinctiveness: could a competitor sign this? If yes, it fails.
Keep "I like it" out of the test. Taste still decides, later, between options that pass.
Instead of
Write a check like
"Looks premium"
Holds its own next to the three most expensive brands on the shelf
"Engaging"
A reader can repeat the main point after one read
"Clean"
No element can be removed without losing meaning
"Clear"
A first-time visitor can say what it does and who it is for in five seconds
This is the do-your-best problem from goal-setting research in another form. A goal with no external reference point lets almost any result count as success [3]. A check a stranger can apply is that reference point.
Done when
Every check can be applied by someone who was not in the room
For each reference, one thing to take and one thing to ignore. Then write the avoid list, with what to do instead.
WhoDesigner or writer
BudgetForty-five minutes
Inrefs/
Outboard.md and the avoid list
Reference · sample board
Take
Ignore
Enamel shop sign
Hand-cut lettering
The colours, the rust
Kraft-paper bag
Warmth, an uncoated feel
The brown
Railway timetable
Confidence, a strict grid
The density
Do
Write one line per reference: take this, ignore that.
Write the avoid list: the clichés of the category and anything you would reject on sight. Five items at most.
Pair every avoid with an instead. In the prompting study, instructions like "do not say X" often failed, and the forbidden phrase came back [1].
Check the board against the why. Every take should serve the intent.
Done when
Every reference has a take and an ignore
Every avoid has an instead
Prompt
Here are my references, with what to take from each and what to ignore: [board]. Here is the avoid list, each with what to do instead: [list]. Before making anything, tell me in your own words what each reference is for in this brief. Copy no reference directly.
Three directions that differ in kind. Ten variations on one idea look like choice and give you one option.
WhoDesigner or writer, with a model
BudgetTwo hours
Indirection.md
OutThree directions at the same finish
Working on several directions at once beats refining one. Designers who developed prototypes in parallel produced better results and more divergent ideas than designers who iterated on one at a time, with the same time and the same feedback [6].
Do
Name the axis the directions differ on: form, tone or idea. "A wordmark, a monogram, a badge" differ in form.
Get a one-line premise for each before any making. Kill any premise that is the same idea twice.
Make each direction from the full brief, separately, so one does not drift toward another.
If you include the obvious answer, label it as the control. It shows where the average sits.
Done when
Three premises a stranger would call different
Each direction was made from the full brief
Prompt
Using direction.md, propose three directions that differ in [form, tone or idea]. Give each a name and a one-line premise. If two premises are the same idea in different clothes, replace one. Wait for me to approve the premises before making anything.
A good note names the check that failed, what is visibly wrong, and what must survive the change.
WhoCreative lead
BudgetFifteen minutes per round
InThe chosen direction, the test
OutRevision notes
The same feedback, twice
Instruction notes
Make it pop
Can we try something more premium?
It feels a bit off
Make the logo bigger
Direction notes
Fails "reads at 16px": the counters close up. Open them, keep the weight.
Fails "a rival could sign it": the cup shape. Remove it, keep the lettering.
Passes everything else. Change nothing else.
Do
One note per failed check. Name the check in the note.
Say what to keep. A revision that fixes one thing can lose three, unless the keepers are named.
Change one thing per round where you can, so you know what worked.
Done when
Every note names a check
Every note says what to keep
Prompt
Revise direction [B]. It fails one check: [check]. What is wrong: [what you can see]. Keep: [what works]. Change only what the check needs. Then tell me which checks you think it now passes, and why, and I will judge for myself.
One live job, five stages, one working day. The output is the work and a direction you can reuse.
Setup prompt
Help me direct one real brief instead of prompting it.
1. Ask me for the ask, exactly as it arrived. Then ask questions until we can write one sentence: this exists so that [who] will [feel, think or do what]. Save it at the top of direction.md.
2. Help me write a success test of two or three checks a stranger could apply, including one for distinctiveness. Add it to direction.md before we make anything.
3. Ask me for three to five references. For each, help me write one thing to take and one to ignore. Save them as board.md. Then draft an avoid list of at most five items, each with what to do instead.
4. List what is fixed and what is open on purpose.
5. Propose three directions that differ in structure, as one-line premises. Wait for my approval before making anything.
6. After each round, mark every direction pass or fail against each check. Write revision notes that name the failed check and what to keep. Stop when everything passes.
Never invent references, research or numbers. Ask me.
Block
Steps
Time
What happens
01
1.1, 1.2
One hour
Write the why and the success test. Agree both with whoever asked.
02
2.1, 2.2
Ninety minutes
Build the reference board with jobs. Write the avoid list.
03
3.1, 3.2
Two and a half hours
Fix the constraints. Approve three premises. Make the first round.
04
4.1, 4.2
Thirty minutes
Lay out the round. Mark it against the test. Choose and write why.
05
5.1, 5.2
About an hour
Revise in the language of the test. Stop when it passes. Save the direction.
Times are working estimates for one brief. The first run is slower. By the third, the why and the test take minutes, because the board and the kill list carry over.
The direction brief, the reference board and the revision note. Copy them, then make them yours.
direction.md
# Direction: [project]
## Intent
This exists so that [who] will [feel, think or do what].
## Audience
[One specific person. What they already believe. Where they will meet this.]
## References
See board.md. Three to five, each with a take and an ignore.
## Constraints
Fixed: [formats, sizes, length, colours, legal lines, deadline]
Open on purpose: [what we want explored]
## Avoid
- [reject-on-sight item] / instead: [what to do]
- [category cliché] / instead: [what to do]
## Range
Three directions that differ in [form, tone or idea]:
A. [premise]
B. [premise]
C. [premise]
## Success test
Passes only if:
1. [check a stranger could apply]
2. [check a stranger could apply]
3. No competitor could sign it.
board.md
# Reference board: [project]
| # | Reference | Where from | Take | Ignore |
|---|-----------|------------|------|--------|
| 1 | | | | |
| 2 | | | | |
| 3 | | | | |
From outside the category: [which one, and why]
The obvious answer we left out: [what, and why]
Every take serves the intent: [yes / fix]
Revision note
## Round [n] · Direction [letter]
Checks: 1 [pass/fail] 2 [pass/fail] 3 [pass/fail]
Failed check: [name it]
What is wrong: [what you can see]
Keep: [what works and must survive]
Change: [one thing]
Stop rule: when every check passes, this round is final.
Research · Playbook 07
Research
What this playbook rests on, how to measure whether it works for you, and what it cannot yet claim.
Every core claim in this playbook, with what supports it. Grades describe the kind of evidence, not how much we like the claim. Claims graded P come from practice and are labelled as such.
4A · Experiment
1B · Field data
1C · Framework
1P · Practitioner
Claim
Grade
What supports it
Source
People new to prompting brief a machine the way they would brief a person, and judge from single outputs.
B · Field data
In a qualitative study of ten people without an AI background designing prompts, participants explored opportunistically, over-generalised from single successes or failures, and carried expectations over from instructing people. Negative instructions such as "do not say X" often failed [1]. Observational and small, so graded B.
1
AI help makes creative work more alike across people.
A · Experiment
In a 293-writer experiment, stories written with AI ideas were rated better individually and were more similar to each other as a set [2].
2
Vague prompts get the average, because every open decision is filled with the most common answer. A seven-part direction closes those decisions.
P · Practitioner
MainArc's thesis and the mechanism behind Fig 0.1. Consistent with [2], untested as a direct comparison of briefs. The benchmark on the next page is designed to test it.
Practice
A success test beats "I'll know it when I see it".
C · Framework
Across decades of experiments, specific difficult goals led to higher performance than urging people to do their best, because do-your-best goals have no external referent [3]. The effect is smaller on complex tasks, and the research concerns people's own performance, not briefs given to a model. Applying it to direction is MainArc's extension.
3
A single example gets copied, flaws included.
A · Experiment
Engineers shown an example before designing reused its features, including features unsuited to the problem, more often than a group shown no example [4].
4
Diverse references produce more diverse ideas.
A · Experiment
In an online ideation experiment, more diverse example sets led to more diverse ideas, and more creative examples to more creative ideas [5]. Neither condition differed significantly from showing no examples at all, and diversity of examples did not raise creativity.
5
Several directions in parallel beat one direction refined.
A · Experiment
Parallel prototyping produced better results and more divergent ideas than serial iteration, with equal time and feedback [6].
6
References
[1]Zamfirescu-Pereira, J. D., Wong, R. Y., Hartmann, B., & Yang, Q. (2023). Why Johnny can't prompt: How non-AI experts try (and fail) to design LLM prompts. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI '23), Article 437. ACM. doi.org/10.1145/3544548.3581388
[2]Doshi, A. R., & Hauser, O. P. (2024). Generative AI enhances individual creativity but reduces the collective diversity of novel content. Science Advances, 10(28). doi.org/10.1126/sciadv.adn5290
[3]Locke, E. A., & Latham, G. P. (2002). Building a practically useful theory of goal setting and task motivation: A 35-year odyssey. American Psychologist, 57(9), 705–717. doi.org/10.1037/0003-066X.57.9.705
[5]Siangliulue, P., Arnold, K. C., Gajos, K. Z., & Dow, S. P. (2015). Toward collaborative ideation at scale: Leveraging ideas from others to generate more creative and diverse ideas. Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW '15), 937–945. ACM. doi.org/10.1145/2675133.2675239
[6]Dow, S. P., Glassco, A., Kass, J., Schwarz, M., Schwartz, D. L., & Klemmer, S. R. (2010). Parallel prototyping leads to better design results, more divergence, and increased self-efficacy. ACM Transactions on Computer-Human Interaction, 17(4), Article 18. doi.org/10.1145/1879831.1879836
Grades
A controlled experiment. B large-scale field data. C an established peer-reviewed model or guideline, not a direct test of the claim. P practitioner claim from MainArc's work, untested.
Does directing beat prompting for your team? Run the same briefs both ways, blind, and measure what wins, what passes and how alike the outputs are.
Instruments
01
Blind pairwise
Same job, two briefs: the prompt as it was first written, and the seven-part direction. Reviewers see one output from each, unlabelled and in random order, and pick the one they would ship.
02
Test pass rate
Share of first-round outputs that pass the success test, for each condition. Write the test before either brief is run.
03
Rounds to pass
Revision rounds until every check passes. Direction should cut rounds, and the notes should name checks.
04
Similarity
Embed outputs across briefs and measure average pairwise similarity. Rising similarity is the convergence warning in [2].
Study design
Element
Setting
Briefs
10 live briefs from the last quarter
Conditions
Each brief run twice: as first prompted, and as a seven-part direction
Panel
5 reviewers who did not write the briefs
Analysis
Win rate per condition. Pass rate and rounds to pass per condition. Similarity across briefs.
Cadence
Once at setup, again after 8 weeks of directing
Reading the results
If directed outputs win most comparisons and pass the test more often in the first round, the method is working for you.
If directed outputs win but similarity stays high, the references are too narrow. Widen the board before adding rules.
If nothing changes, read the briefs. A direction with an empty success test is still an instruction.
Running signals
Signal
What to watch for
Briefs with a written test
Should be every brief before any making starts
Notes that name a check
A falling share means revisions are drifting back to taste words
Avoid items reused
A growing standing kill list means the team is learning from each round
Note
Thresholds here are starting points to tune, not findings. A small panel shows direction and disagreement. It does not prove significance, and nothing on this page should be reported as if it did.
How this playbook was built, where it is weak, and how to cite it.
How this playbook was built
Built from the creative director's brief, MainArc's practice, and published research on prompting by non-experts, design fixation, example diversity, parallel prototyping, goal setting and AI homogenization. It is a practitioner method, not a study. Claims are graded on the evidence page by what supports them.
Limitations
The central mechanism, that open decisions default to the most common answer, is MainArc's thesis. No study here compares vague and structured briefs directly.
The prompting study [1] is qualitative, with ten participants and one chatbot task. It shows how people prompt, not what output vague prompts produce.
The goal-setting evidence [3] concerns people working toward their own goals, mostly on simpler tasks than creative work. Its effect is smaller on complex tasks.
The fixation and example studies [4, 5] tested human designers and crowd workers, not models. That models copy single references is consistent with them, not tested here.
The converter on the introduction page is a rule-based pattern matcher with sample prompts. It detects signals of each part and cannot judge their quality.
Open questions
How much of a direction can a team reuse across briefs before the work starts converging on its own house average?
Which of the seven parts moves output quality most, and does the answer differ between writing and visual work?
Does the success test change what reviewers prefer, or only how fast they agree?
Found an error, a stronger source, or a result from running the benchmark? Send it through mainarc.xyz. Corrections are made in the next version.
About this playbook
This playbook is published by MainArc, a creative intelligence lab for leadership and product teams.
We do three things. We research how AI is changing creative work through long-form reports, design teardowns, and operator essays. We design the workflows, prompt systems, review loops, and decision frameworks that teams actually run on. And we build the work the strategy points to: brand systems, interfaces, product prototypes, internal tools, and launch concepts.
One operator. One AI agent. No layers in between. Every engagement runs under strict NDA. No case studies, no client logos, no shared work.
If this research resonates with a problem your team is facing, start with an audit. One week. A clear map of where AI fits, where it doesn't, and what to build next.
This is independent research. Not financial advice. Not career advice. Not the final word on anything. It's one person's perspective shaped by building, shipping, and paying attention. Do your own thinking. Challenge the ideas. Disagree with specifics. Build on what resonates. Discard what doesn't.
House rules
What doesn't belong here: hate, trolling, bad-faith takedowns, or drive-by takes from people who've never built anything. If you disagree, show your work. Builder's critique. Everyone else just comments.
A MainArc playbook for teams building AI products with memory. Storage is free. Deciding what deserves to be kept, where it may be used and when it should be forgotten is the design work, and it decides whether memory feels like a colleague or like being watched.
PlaybookNo. 08Version1.0Read25 minSources6 citedByMainArc ResearchEvidence and method →
Thesis
A product that remembers feels like a colleague or like surveillance, and the difference is design: what it keeps, shows, uses in context and forgets, more than what the model can do.
Do this Monday
List every place your product stores something about a person that can shape a later answer. Sort each into four kinds, and flag sensitive items kept without a yes.
A memory ledger for a sample assistant. It is live. Pick a request and see which memories it used, which it held back and why. Then switch the context check off, or hide the ledger, and watch the same assistant turn from helpful to unsettling without anything about the model changing.
Memory ledger · sample assistant
Sample memories and requests. Trust scores are a heuristic, not a measurement
4Kinds of memory
5Stages
10Steps
4Templates
What this is
Memory is becoming the interface. A product that remembers your team, your preferences and the project you were on yesterday can skip the setup and start where you left off. That feels like working with someone who pays attention. The same product, using the same facts in the wrong place, feels like being followed.
The difference between those two products is rarely the model. It is a set of decisions: what to keep, what to show, where a memory may travel, who can change it and when it ends. A product that skips those decisions still makes them, by default, and the easy default is to keep everything and show nothing. This playbook makes them on purpose, in five stages, with a ledger the person can read.
Storage is free. Deciding what deserves to be kept isn't. That decision is the judgment this playbook protects.
Principles
01
Keep on purpose
Every memory is a decision. If nobody on the team made it, the default made it for you.
02
Show what you know
A memory a person cannot see is a memory they cannot correct, trust or remove.
03
Context travels with the memory
A fact said in one place carries the rules of that place. Use it only where those rules still hold.
04
Say when memory helped
One short line of "remembered because" turns a lucky guess into a reason someone can check.
05
Ask before the sensitive
Health, beliefs, money trouble, a job search. Kept only with a yes, used only where it was given.
06
Forgetting is a feature
Expiry, forget on request and pause deserve the same design care as recall.
How to use this playbook
01
Break the ledger first
Pick the note to your manager, switch the context check off and read what the assistant just wrote.
02
Read the three essays
Why storage being free makes memory a decision, where the creepy line sits, and the four kinds of memory.
03
Start with an audit
Step 1.1 lists what your product already keeps about people, including what nobody decided to keep.
04
Copy the templates
A memory policy, an entry schema, consent copy and "remembered because" patterns are on the Templates page.
Playbook 08 in the set. Playbook 04 covers agents that act on a person's behalf. This one covers what a product keeps about that person between one request and the next.
When keeping costs nothing, the default is to keep everything. That default is a design choice, and usually a bad one.
The cost moved
Keeping a sentence someone typed costs almost nothing. Keeping every sentence costs almost nothing. So the question stopped being whether a product can remember, and became whether it should. A memory is a claim about a person that the product will act on later, in a moment the person did not choose. That claim deserves a decision.
Every memory goes through five moments: someone says something, the product keeps it or not, shows it or not, uses it somewhere, and eventually forgets it or never does. Each moment is a judgment call. A product without a memory design still makes all five calls. It just makes them silently.
Fig 0.1Five decisions in the life of a memoryPress to say somethingsaid · bin 0
Back line, the default: said, stored forever in one deep bin, used anywhereFront line: 01 keep gate · 02 window, show · 03 context selector, use04 Shredder, forget · 05 return rail: what is forgotten resets the keep dialThe default skips every decision. The design makes each one visible, and each one maps to a stage of this playbook. Here one card is said twice: the back line keeps it out of sight for good; the front line keeps it or not, shows it, uses it only in context, then forgets it.
Personalisation has a price, and someone pays it
Remembering more makes a product more personal. It also asks more of the person. In a survey of more than 400 online consumers, the people who valued information transparency most were the least willing to be profiled for personalisation [1]. The authors suggested firms accept that a privacy-sensitive minority will opt out. This playbook makes a different bet: that showing people what is kept, and letting them change it, brings some of them back. That bet is untested, and step 5.1 is how to test it on your own product.
MainArc Research
Production is free. Judgment isn't. In a product with memory, the judgment is in what you choose not to keep.
People rarely object to a product knowing something. They object when it uses what it knows somewhere that knowledge does not belong.
Privacy is about where information goes
One influential account of privacy says every context carries its own norms: what is fitting to share there, and how information may flow onward. Privacy is breached when either norm is broken [2]. A doctor knowing about your knee is fine. Your employer learning it from your doctor is not. The fact is identical. The flow is different.
The idea has been tested. In a survey experiment with 569 respondents, judgments about whether a privacy expectation was met depended heavily on who received the information and how it was used. How sensitive the data sounded mattered less than where it went [3].
Fig 0.2One memory, three flowsMove to switch · press to sendat source · to running plan
Canister: the memory, recovering from a knee injury, said in a chat about running01 Lectern: the running plan, same context, fits02 Team offsite board: the team sees it · 03 Mailbox: a note to a manager, a new audienceSample. The memory never changed. The flow did. That is where the creepy line sits, and flows are designed. The canister is identical at every station; only the switch decides where it lands.
What this means for a product
Three things follow. First, a memory needs a record of where it came from, because that is its context. Second, every use is a flow, so every use needs a check: does this request belong to the context this memory was given in? Third, the audience counts. A plan the whole team will read is a different context from a plan only one person sees, even when the request sounds the same.
The shift
A better model will know more and connect more. That makes context rules more necessary. Capability raises the number of flows. Design decides which ones are allowed.
A preference, a fact about someone, a draft in progress and a health worry are all "memory". Each needs different handling.
Treating all memory the same produces two failures. Either everything is handled as lightly as a preference, and sensitive things leak. Or everything is handled as carefully as a diagnosis, and the product asks permission to remember that someone likes short answers. Two questions sort memories well enough to design for: how long does it stay useful, and how much harm does it do if it travels?
Fig 0.3Two questions, four kindsMove to aim · press to fileaim · preference
Top, sensitive: health, beliefs, money trouble, a job search. Ask first, narrowest slot, short lifeMiddle right, facts about the person: confirm once, scope to where it was saidBottom left, work in progress: lives in its project, a timer empties itBottom right, preferences: keep, show, easy to editBottom ruler: how long it stays useful · bar ruler: harm if it travelsA MainArc model for design defaults, not a measurement. Your product may need a fifth kind. Two questions place a memory: how long it stays useful, and how much harm it does if it travels. The drawer it lands in sets how it is kept.
Kind
Example
Keep when
Use in
Ends
Preference
Likes short answers
Stated, or confirmed after a pattern
Wherever it applies
When the person changes it
Fact about the person
Leads a team of six
Stated, and confirmed once
Contexts like the one it came from
When reviewed and no longer true
Work in progress
Launch plan, draft 3
The work is open
That project only
When the work closes
Sensitive
Recovering from an injury
The person said yes to keeping it
Only the context it was given for
Soon, unless renewed
The sensitive row is where design and law meet. In the EU, processing data about health, religious or philosophical beliefs, political opinions, trade union membership, sexual orientation and several other special categories is prohibited unless an exception applies, such as the person's explicit consent for specified purposes [4]. Your list of sensitive things should start there and go further: a job search or a family problem may be legal to keep and still harmful to leak.
Widely used guidelines for AI products already point at memory. They say to "remember recent interactions", to "learn from user behavior", to "update and adapt cautiously" and to "provide global controls" over what the system monitors and how it behaves [5]. The first of those is about short-term memory. Long-term memory, across weeks and contexts, is where the four kinds start to matter.
Start with what the product keeps today, including the things nobody decided to keep.
WhoPM, designer, engineer
BudgetHalf a day
InData stores, logs, prompts
Outmemory-audit.csv
Do
List every place the product stores something about a person that can shape a later answer: saved settings, chat history, summaries, inferred traits, retrieval indexes, logs fed back into prompts.
For each item record what it is, where it came from, who can see it, how long it stays, and whether the person can see it.
Sort each into one of the four kinds: preference, fact, work in progress, sensitive.
Mark anything inferred rather than stated. An inference is a guess the person never made, and it needs the strictest handling of its kind.
Flag every sensitive item kept without a clear yes. That list is the first thing to fix.
Done when
Every store that can shape an answer appears in memory-audit.csv
Every item has a kind, a source and a lifetime, even if the lifetime is "forever"
Prompt
Read our codebase and data model. List every place we store information about a user that can influence a later response: settings, chat history, summaries, inferred traits, embeddings, logs used in prompts. For each, tell me what it holds, where it comes from, who can read it, how long it is kept, whether the user can see it, and whether it is stated or inferred. Sort each into preference, fact, work in progress or sensitive. Save it as memory/memory-audit.csv. Do not guess where you are unsure; mark it unknown.
One rule per kind: when it is kept, where it may be used, how it is shown, when it ends and what needs a yes. One file everything reads.
WhoPM, designer, privacy lead
BudgetTwo hours
Inmemory-audit.csv
Outmemory-policy.json
Kind
Needs a yes
Shown as
Default life
Preference
No, but show it
A line in the ledger, editable in place
Until changed
Fact
Confirm once
A line with its source and scope
Until reviewed
Work in progress
No
Inside the project it belongs to
Ends with the work
Sensitive
Always, explicitly
Marked, with its narrow scope
Short, renewed by asking
Do
Start from the table above. Change it for your product, and write down why for every change.
Write your sensitive list. Start from the special categories in data protection law, then add what is sensitive for your users: a job search, a debt, a family matter.
Decide what the product will never keep, even with a yes. A short never-list is a strong promise.
Write it all into memory-policy.json, so the assistant, the ledger and the settings page read the same rules.
Done when
Every kind has a consent rule, a scope rule, a display rule and a default life
The never-list exists and someone outside the team has read it
Everything the product remembers, in one place, in the person's own words, with where it came from. And a clear question at the moment a sensitive memory is made.
One page where a person sees everything the product remembers about them, and can change any of it.
WhoDesigner, engineer
BudgetTwo days
Inmemory-policy.json
OutA memory ledger page
Ledger entryExample
FactLeads a team of six designers
From a chat about hiring. Used for work and team events.
Do
Write every memory as a plain sentence the person would recognise as true about themselves. No internal labels, no scores.
Show the source as context: "from a chat about your launch". The context is what tells a person whether a use is fair.
Show the scope: where this memory may be used. Let people narrow it.
Put edit, forget and a global pause on the same page. Guidelines for AI products call for global controls over what a system monitors and how it behaves [5].
Link to the ledger from every answer that used memory. A ledger nobody can find does not count.
Done when
Everything in memory-audit.csv that can shape an answer appears in the ledger, or is deleted
Before a memory goes into an answer, check that this request belongs to the context the memory came from, and who will read the result.
WhoEngineer, designer
BudgetOne day
Inmemory-policy.json, ledger scopes
OutA context check in the answer path
Do
Give every memory a scope when it is kept: the contexts it may be used in. Default to the narrowest scope that is still useful.
Give every request a context and an audience. A note to a manager, an invite or a shared plan will be read by someone else.
Use a memory only when the request's context is inside its scope. When the answer will be shared, sensitive memories stay out, whatever the scope says.
When a blocked memory would genuinely help, ask: "You mentioned this in a chat about running. Use it here?"
Try it
In the ledger on the introduction page, pick the offsite plan and switch the context check off. The plan now explains a knee injury to the whole team. Nothing about the assistant's knowledge changed. One check was removed.
Done when
Every memory has a scope, and every use is checked against it
A test set of tempting requests produces zero out-of-context uses
Prompt
Write twenty test requests for our assistant that would tempt it to use a memory outside the context it came from. Include requests whose answers will be read by someone else: notes, invites, shared plans. For each, name the memory that must not be used and why. Save it as memory/context-tests.md, then run them and report every leak.
A short reason turns a lucky guess into something a person can check, correct or forget.
WhoDesigner, writer
BudgetHalf a day
InLedger entries
Out"Remembered because" patterns
Explanation helps, in moderation. In a field experiment on a grading system, people whose results were worse than they expected trusted the system less. A short explanation of how it worked restored that trust. A detailed one, with all the raw numbers, eroded it again [6]. The study was about grades. The lesson should carry over to memory: say enough, then stop.
Do
Attach one line to any part of an answer shaped by a fact, work in progress or sensitive memory: Remembered because you said it in a chat about your launch.
Skip the line for obvious preferences. Short answers do not need a footnote.
Make the line a link to the ledger entry, with forget one tap away.
Never explain a memory that should not have been used. If you find yourself writing that line, the context check failed.
Done when
Every answer that used a non-preference memory says so in one line
A person can forget a memory from the answer it shaped
When a person says forget, the memory leaves the ledger, the summaries built on it and everything the product would read next time.
WhoEngineer, privacy lead
BudgetTwo days
Inmemory-audit.csv
OutA forget path for every store
In the EU, people have a right to have personal data erased without undue delay on several grounds, including when the data are no longer needed for their purpose and when consent is withdrawn [4]. Design to that standard everywhere, and check the law for your own markets.
Do
Map every place a memory lives: the ledger, summaries, embeddings, caches, logs that feed prompts. Forget removes it from all of them.
Confirm in one line what was forgotten, and say plainly what was not, such as a file the person shared into a project.
Build pause: one switch that stops keeping and using memory. Say what happens to existing memories while paused.
Test it. After forgetting, ask a question that would have used the memory. If the answer still knows, forget is broken.
Most memories have a useful life. Design the end of it, instead of letting everything live forever by accident.
WhoPM, engineer
BudgetTwo hours
Inmemory-policy.json
OutExpiry rules and renewal prompts
Data protection law already has a name for this idea, storage limitation: personal data kept in identifiable form for no longer than necessary for the purpose [4]. Expiry turns that principle into product behaviour.
Kind
Ends when
Before it ends
Preference
The person changes it, or it goes unused for a long time
Nothing. It quietly leaves the ledger
Fact
A review date passes
Ask once: "Still true?"
Work in progress
The project closes or goes quiet
Offer to keep a summary in the project, not in memory
Sensitive
A short default life passes
Ask to renew, with the scope named again
Do
Give every kind a default life in memory-policy.json. Start short for sensitive things. Lifetimes are working estimates to tune.
Show the end in the ledger: "Ends when the launch ships".
Let expired work in progress fold into the project it belongs to, so memory stays about the person.
One audit, one policy, one ledger and one context check, in two weeks. Then measure for a month before keeping anything new.
Setup prompt
Help me design memory for our product so people can see it, trust it and forget it.
1. Read our code and data model. List every place we store information about a user that can shape a later answer. Record what it holds, where it comes from, who can see it, how long it is kept, and whether it is stated or inferred. Sort each into preference, fact, work in progress or sensitive. Save as memory/memory-audit.csv.
2. Draft memory/memory-policy.json with one rule per kind: when it is kept, what needs a yes, where it may be used, how it is shown and when it ends. Add a never-list. Ask me before you finalise it.
3. Design the ledger: one entry per memory, in plain words, with its source as context ("from a chat about ..."), its scope, and edit, forget and pause controls.
4. Add a context check to the answer path: use a memory only where its scope fits the request, and keep sensitive memories out of anything someone else will read. Write twenty tempting test requests in memory/context-tests.md and report every leak.
5. Draft the "remembered because" lines and the keep prompts for sensitive memories.
Do not invent user data. Use placeholders until we have real entries.
Day
Steps
What happens
D1
1.1
Audit every store. Flag sensitive items kept without a yes.
D2
1.2
Write the policy and the never-list. Fix the flagged items.
D3 to D5
2.1, 2.2
Build the ledger and the keep prompts for sensitive memories.
D6 to D7
3.1, 3.2
Add the context check and the "remembered because" lines. Run the context tests.
D8 to D9
4.1, 4.2
Build forget, pause and expiry. Test that forgotten means forgotten.
D10
5.1
Turn on the five signals. First review in a month, step 5.2.
{
"id": "mem_0142",
"kind": "fact",
"text": "Leads a team of six designers",
"stated_or_inferred": "stated",
"source_context": "a chat about hiring",
"scope": ["work", "team_events"],
"consent": { "type": "confirmed", "scope_named": true },
"ends": "review_after_365d",
"visible_in_ledger": true,
"uses": 0,
"last_used_in": null
}
"Remembered because" patterns
Remembered because you said it in [a chat about X]. [See] [Forget]
Using your [preference]. Change it in your memory.
From your [project] notes. This ends when [the work closes].
Blocked memory, worth asking:
You mentioned [X] in [a chat about Y]. Use it here? [Use it] [Not here]
Never:
"Based on what I know about you..."
"As you mentioned before..." (with no way to see where)
Keep, forget and pause copy
Keep, sensitive:
Should I remember that [X]? I would use it only for [scope], and you can forget it any time.
[Remember for (scope)] [Just this chat]
Keep, fact:
I'll remember that [X]. [Change]
Forgotten:
Forgotten: [X]. It will not shape any future answer.
Still kept: [anything that remains, and where].
Paused:
Memory is paused. I will not keep or use anything new.
Your [N] existing memories are kept but not used until you resume. [Resume] [Forget all]
Research · Playbook 08
Research
What this playbook rests on, how to measure whether it works for you, and what it cannot yet claim.
Every core claim in this playbook, with what supports it. Grades describe the kind of evidence, not how much we like the claim. Claims graded P come from practice and are labelled as such.
2A · Experiment
1B · Field data
3C · Framework
1P · Practitioner
Claim
Grade
What supports it
Source
Personalisation and privacy pull against each other, hardest for the people who care most about transparency.
B · Field data
In a survey of more than 400 online consumers, those who valued information transparency most were the least willing to be profiled for personalisation [1]. A survey of attitudes, not observed behaviour.
1
A privacy violation is an information flow that breaks the norms of the context it came from.
C · Framework
The contextual integrity framework holds that every context has norms of appropriateness and of flow, and privacy is breached when either is broken [2].
2
Where information goes matters more to people than how sensitive it sounds.
A · Experiment
In a factorial vignette survey with 569 respondents, whether a privacy expectation was met depended heavily on the recipient and the use of the information, more than on sensitivity labels or privacy attitudes [3]. Stated judgments about scenarios, not behaviour in a product.
3
Sensitive categories need explicit consent, people can ask for erasure, and data should not be kept longer than needed.
C · Framework
EU data protection law prohibits processing special categories such as health and beliefs unless an exception like explicit consent applies (Article 9), gives a right to erasure without undue delay on set grounds (Article 17), and sets storage limitation (Article 5(1)(e)) [4]. Law in one jurisdiction, used here as a design floor.
4
AI products should remember recent interactions, learn from behaviour, adapt cautiously and offer global controls.
C · Framework
Established guidelines for human-AI interaction include "remember recent interactions", "learn from user behavior", "update and adapt cautiously" and "provide global controls" [5]. The memory guideline is about short-term memory.
5
A short explanation helps trust when a result surprises people. Too much detail hurts it.
A · Experiment
In a field experiment with 103 learners, a short explanation of a grading algorithm restored trust after a worse-than-expected grade, while a detailed one with raw scores eroded it [6]. One study, about grades, not memory.
6
Deciding what to keep, show, use in context and forget, by kind of memory, is what makes memory feel like a colleague instead of surveillance.
P · Practitioner
MainArc's thesis, built on [2] and [3]. The four kinds, the ledger and the "remembered because" pattern are untested as a system. The benchmark below is how to test them.
Practice
References
[1]Awad, N. F., & Krishnan, M. S. (2006). The personalization privacy paradox: An empirical evaluation of information transparency and the willingness to be profiled online for personalization. MIS Quarterly, 30(1), 13–28. doi.org/10.2307/25148715
[3]Martin, K., & Nissenbaum, H. (2016). Measuring privacy: An empirical test using context to expose confounding variables. Columbia Science and Technology Law Review, 18, 176. doi.org/10.2139/ssrn.2709584
[4]Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data (General Data Protection Regulation). Official Journal of the European Union, L 119, 1. Articles 5(1)(e), 9 and 17. eur-lex.europa.eu/eli/reg/2016/679/oj
[5]Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., Teevan, J., Kikin-Gil, R., & Horvitz, E. (2019). Guidelines for human-AI interaction. Proceedings of CHI '19. ACM. doi.org/10.1145/3290605.3300233
[6]Kizilcec, R. F. (2016). How much information? Effects of transparency on trust in an algorithmic interface. Proceedings of CHI '16, 2390–2395. ACM. doi.org/10.1145/2858036.2858402
Grades
A controlled experiment. B large-scale field data. C an established peer-reviewed model or guideline, not a direct test of the claim. P practitioner claim from MainArc's work, untested.
Memory should earn its place. Measure what it does for people, what it does to them, and whether they feel in charge of it.
Instruments
01
Use and forget
For each kind of memory: how often it shaped an answer, how often it was edited, and how often it was forgotten right after a use.
02
Context tests
A fixed set of requests built to tempt memory out of its context, run on every release. Count leaks.
03
Felt control
A short survey, 1 to 5: I know what it remembers. I can change it. It uses things where they belong.
04
Ledger versus hidden
Where you can, compare the same memory with and without a visible ledger and "remembered because" lines, on usefulness, surprise and pause rate.
Study design
Element
Setting
Memories
Every entry in the ledger, by kind
Test set
Twenty to fifty tempting requests, including shared outputs
People
All active users for behaviour, 20 for the monthly survey
Analysis
Forget-after-use rate and edit rate by kind. Leak count. Survey medians before and after the ledger ships.
Cadence
Monthly, and after every policy change
Reading the results
If a kind is forgotten right after use more often than others, narrow its scope or ask before keeping it.
Any leak in the context tests blocks the release. Fix the check before the memory.
If felt control is low while usefulness is high, people are getting help they cannot see. Make the ledger easier to find before adding memory.
Running signals
Signal
What to watch for
Forget right after use
The clearest sign of a bad surprise. Read every one
Pause rate
Rising pauses mean memory is costing more trust than it earns
Kinds kept and never used
Stop keeping them. Unused memory is pure risk
Note
Thresholds here are starting points to tune, not findings. A small panel shows direction and disagreement. It does not prove significance, and nothing on this page should be reported as if it did.
How this playbook was built, where it is weak, and how to cite it.
How this playbook was built
Built from privacy research on contextual integrity, data protection law, guidelines for human-AI interaction, research on transparency and trust, and MainArc's practice in AI product design. It is a practitioner method. Claims are graded on the evidence page.
Limitations
Nothing here is legal advice. The law cited [4] is one jurisdiction, and obligations differ by market, sector and role. Check with a qualified adviser.
The strongest evidence on context [3] measures stated judgments about scenarios, not behaviour in a product with memory.
The transparency study [6] is about grades in an online course. Its mechanism should transfer to memory explanations, but that is untested here.
The four kinds of memory and their default lifetimes are a design model and working estimates, not findings.
The memory ledger on the introduction page uses sample memories and requests, and its trust panel is a heuristic, not a measurement.
Open questions
Does a visible ledger bring back people who would otherwise refuse personalisation, or only reassure those who already accept it?
How long should each kind of memory live before a renewal question helps more than it annoys?
How should memory behave when one product serves a person in several roles, such as work and home?
Cite this playbook
APA
MainArc Research. (n.d.). Design products that remember (Playbook No. 08, Version 1.0). MainArc. https://mainarc.xyz
BibTeX
@misc{mainarc_memory,
author = {{MainArc Research}},
title = {Design products that remember},
howpublished = {MainArc Playbook No. 08, Version 1.0},
url = {https://mainarc.xyz}
}
Corrections
Found an error, a stronger source, or a result from running the benchmark? Send it through mainarc.xyz. Corrections are made in the next version.
About this playbook
This playbook is published by MainArc, a creative intelligence lab for leadership and product teams.
We do three things. We research how AI is changing creative work through long-form reports, design teardowns, and operator essays. We design the workflows, prompt systems, review loops, and decision frameworks that teams actually run on. And we build the work the strategy points to: brand systems, interfaces, product prototypes, internal tools, and launch concepts.
One operator. One AI agent. No layers in between. Every engagement runs under strict NDA. No case studies, no client logos, no shared work.
If this research resonates with a problem your team is facing, start with an audit. One week. A clear map of where AI fits, where it doesn't, and what to build next.
This is independent research. Not financial advice. Not career advice. Not the final word on anything. It's one person's perspective shaped by building, shipping, and paying attention. Do your own thinking. Challenge the ideas. Disagree with specifics. Build on what resonates. Discard what doesn't.
House rules
What doesn't belong here: hate, trolling, bad-faith takedowns, or drive-by takes from people who've never built anything. If you disagree, show your work. Builder's critique. Everyone else just comments.
A MainArc playbook for brand designers, creative directors and marketing leads. A brand used to be a set of finished files. Now it has to be a set of rules that keeps producing work at volume, across every channel, often made by machines. This is how to choose the few rules that keep it yours.
PlaybookNo. 09Version1.0Read23 minSources6 citedByMainArc ResearchEvidence and method →
Thesis
A brand survives generation by fixing only its two or three most distinctive assets, because locking generic rules buys consistency and almost no recognition.
Do this Monday
Pull the last fifty pieces you shipped, list every element that repeats, and mark which ones a competitor could also claim.
A brand rule engine for Ombrel, a fictional sparkling water. It is live. Six rules, each locked, flexible or free, and twelve posters generated from whatever you set. With the two signature rules locked, the grid reads as one brand. Lock three generic rules instead and watch it fall apart, even though more rules are locked. Then hide the name and see what is left.
Brand rule engine · sample brand
Recognisability0
Range0
Set 1
Sample brand. Recognisability is a heuristic: each rule's sample distinctiveness weight, counted in full if locked, half if flexible, zero if free.
5Stages
10Steps
3Rule states
3Templates
What this is
Brand guidelines were written for a small team making a few hundred pieces a year. Every piece passed through a designer who knew the brand. The guidelines could be loose because the people were not.
That arrangement is ending. Campaigns now ship in hundreds of sizes, languages and formats, and much of the work is assembled by templates, scripts and agents that have never met the brand. What reaches the audience is whatever the rules allow. Where the rules are vague, the generator fills the gap with the most likely answer, which is everybody's answer.
This playbook rebuilds a brand as a short set of rules: a few fixed, most flexible, the rest free. Generating assets is free. Deciding the few rules that make them yours is the judgment, and no generator will make that call for you.
Principles
01
Ship rules, then files
A file is one answer. A rule is a thousand. Files are what the rules print.
02
Fix very little
Lock two or three signature assets. Every lock beyond those buys sameness and almost no extra recognition.
03
Distinctive before consistent
Consistency pays only when the thing repeated belongs to you. A generic grid used perfectly is still generic.
04
Every rule is a test
If a machine cannot check a rule, a machine will break it. Write values, ranges and lists. Drop the adjectives.
05
Judge the set
One good poster proves nothing. Judge a hundred at thumbnail size, with the name hidden.
06
Expect drift
Generators pull toward the average. Check shipped work every month and fix the rule before the file.
How to use this playbook
01
Break the engine first
Free the motif and the palette. Then lock everything. Neither result is a brand system.
02
Read the three essays
Files to rules, distinctive beats consistent, and fixed, flexible, free. The steps assume all three.
03
Start with the audit
Stage 01 takes about a week and usually shrinks what you think your brand is.
04
Copy the templates
A rule sheet, an asset audit and a recognisability test protocol are on the Templates page.
Playbook 09 in the set. Playbook 01 writes taste down so people and agents can use it. This one writes a brand down as rules a generator can follow, and Playbook 05 gives you the panel to test them.
A brand used to ship as finished files and a guidelines PDF. A generator cannot read intent from a PDF. It can only follow rules.
The old brand had a person in every loop
A logo file, a colour page, a type specimen, a folder of approved photos and a few templates. The guidelines described the brand with examples and adjectives. That worked because every new piece went through someone who had absorbed the brand and could fill the gaps with judgment.
The new brand has a generator in the loop
The volume has moved. One campaign can need hundreds of variations, and templates and agents produce a growing share of them. A generator does exactly what the rules say. Where they say nothing, it does what is most common. That gap is where brands go grey.
Fig 0.1Where the brand livesPress to pull the cassettehand 0 · press 0 · 0 marked
01 Brand as files · drawer and hand press, a few pieces02 Brand as rules · cassette in a press, hundredsThe red mark: the signature assetThe red mark is the signature asset. It survives volume only because a rule says it must. Pull the cassette and the press keeps printing, but what comes out is plain.
The pull toward the middle
The drift is measured, though never yet on brands. In a large writing experiment, AI ideas made individual stories better rated and the whole set of stories more alike [1]. On an online art platform, artists who adopted text-to-image tools made more work and earned more favourites per view, while the visual novelty of their work fell [2]. Models tuned on human preferences produce less varied output than the same models without that tuning [3].
None of these studies looked at brand work. All three point the same way. Left to its defaults, generated work slides toward what is common, and what is common belongs to nobody.
MainArc Research
Production is free. Judgment isn't. In a generative brand, the judgment lives in a handful of rules, and the files are just what those rules print.
Consistency pays only when the thing you repeat belongs to you. Most of what brand books keep consistent belongs to everyone.
Two questions for every asset
Marketing science measures a brand asset with two questions [4]. Fame: shown the asset with the name hidden, what share of category buyers name your brand? Uniqueness: do they name only you, or your competitors as well? An asset that scores on both does the logo's job when the logo is not there. That is exactly the job a generated thumbnail in a feed needs done.
Fig 0.2The asset gridMove to pick · press to tugsorting · 0 of 4 placed
Signature · named, and only you · clamped, fixedCandidate · yours, not yet known · spring stop, flexibleShared · famous, not yours alone · lid. Generic · loose, freePicker: sets each asset down by its two scoresFame and uniqueness are the two metrics in [4]. The quadrant labels and the fixed, flexible, free actions are MainArc's. Uniqueness runs left to right, fame front to back. The four tokens are sample assets.
Some assets are harder to own
A study of 1,281 in-market brand elements, across 13 packaged goods categories in 19 countries, found that characters, logos and logotypes had the most potential for unique ownership. Colour was harder to own, because competitors share colours so often. Execution mattered too: elements of the same type varied widely in how contested they were [5].
This fits the older theory of brand equity. A brand's value sits in associations that are strong, favourable and unique [6]. Generators are good at strong and favourable. Unique is the word they erode.
What consistency buys
Consistent and generic
The same safe sans on everything
The category colour, used perfectly
A clean grid any brand could use
Stock-style photography, colour corrected
Consistent and distinctive
One shape buyers name you from
A colour pairing nobody else in the category uses
An image treatment that survives a thumbnail
A line of copy that sounds like one person
The point
Most brands have one to three assets doing real work. The job is to find them, fix them, and stop spending rules on the rest.
Every element of a generative brand sits in one of three states. Writing the brand means deciding which.
State
What it means
How many
Ombrel example (sample)
Fixed
One value. Nothing may change it.
Two or three elements
The tangerine arch, same proportions, every time
Flexible
A range the generator chooses within.
Most elements
Headline size 18 to 30, one of three layouts
Free
Anything that passes the checks.
Content
The headline word, the photo subject
Fig 0.3How much each rule allowsPress to crank · move to readvariation 0 · type size 24
Fixed · clamped: signature arch, palette pairingFlexible · end stops: type, grid, image, voiceFree · no stops: headline wordCrank: one turn makes one variation (sample brand)Fixed buys recognition. Flexible buys range. Free buys relevance. Each turn of the crank pushes every rule to a new setting; the clamps and stops decide how far each one may go. Sample brand: type size runs 18 to 30, the grid is one of three layouts.
Two failures, mirror images
Fix too much and you have a template. Every piece looks the same, the audience stops seeing it, and the brand can say only one thing. Free too much and you have a mood. The generator fills every gap with its own defaults, and the work drifts toward the middle described on the previous page.
The engine on the introduction shows both. Its recognisability score rises with each locked rule, weighted by how distinctive that rule is. Its range score falls with each lock. A brand system lives where both stay high, and that is only possible when the few locked rules are the distinctive ones.
MainArc Research
Generating a thousand variations is free. Choosing the three things that never vary is the judgment.
List everything the brand repeats, then test which elements buyers connect to you when the name is hidden. Most brands own fewer assets than their guidelines claim.
Start from what actually shipped. The guidelines describe the brand you meant. The last fifty pieces show the brand you have.
WhoBrand designer, creative director
BudgetOne day
InLast fifty shipped pieces
Outbrand/assets.csv
Do
Pull the last fifty pieces that shipped, across every channel. Leave the guidelines closed for now.
List every element that repeats: colours, shapes, type, layouts, image treatments, characters, sounds and lines of copy.
For each one, count how often it appears and note whether a competitor uses something close. Look across the whole category, beyond your own deck.
Give each element a first guess: signature, supporting or generic. Expect most to be generic.
Prompt
Here are fifty pieces we shipped recently. List every element that repeats across them: colours, shapes, type, layouts, image treatments, characters and lines of copy. For each, count how often it appears and describe it precisely enough that a stranger could spot it. Save it as brand/assets.csv. Do not judge which elements are distinctive. That call is ours.
Done when
Every repeated element is listed with a count and a precise description
Show each candidate asset to category buyers with the name hidden, and ask whose it is. Your first guesses meet the audience here.
WhoMarketing lead
BudgetOne week, mostly fieldwork
Inbrand/assets.csv
OutA fame and uniqueness figure per asset
Do
Take the ten strongest candidates. Strip every name, logo and product shot from each.
Show them to at least thirty category buyers outside the team. Ask one open question: which brand is this? Offer no list.
Record the share who name you (fame) and the share who name a competitor (a sign of low uniqueness) [4].
Place each asset on the grid from Fig 0.2. The top right is your signature. Expect one to three assets there, sometimes none.
Why buyers and not the team
Your team sees the brand every day and will recognise everything. That measures exposure to your own files. The test means something only with people who choose between you and your competitors.
Done when
Every candidate has a fame and a uniqueness figure from real respondents
At most three assets sit in the signature quadrant
A rule that says "feel premium" will be broken a thousand times. A rule that gives a value, a range or a list can be tested on every piece.
WhoBrand designer
BudgetOne day
Inrules.yaml draft
Outrules.yaml with a check per rule
Adjectives to checks · sample brand
A generator cannot check
Use our colours confidently
Keep layouts clean and modern
Our voice is witty
The arch should feel prominent
A generator can check
Background is bone or ink. Tangerine covers 5 to 15 percent of the area
One of three layouts. Outer margin is 8 percent of width
Lowercase. Under eight words. Ends in a full stop. No exclamation marks
Arch radius is 20 percent of width, always tangerine, never cropped
Do
Rewrite each rule as a value, a range or a list. Delete every adjective that remains.
Attach a check to each one: how a person or a script would know it was broken.
Add one example that passes and one that fails, side by side.
Hold the voice rule to the same standard as the visual ones. It is the easiest rule to leave vague.
Prompt
Here is our draft brand rule sheet. Rewrite every rule as a value, a range or a list, with no adjectives. For each rule, add a check that a script or a reviewer could run, plus one passing and one failing example. Keep the states we chose: fixed, flexible or free. If you cannot make a rule checkable, flag it and leave it as written.
Designers, agencies and agents read the same file. If it needs a meeting to explain, it is not finished.
WhoCreative director
BudgetHalf a day
Inrules.yaml
Outbrand/brief.md
Do
Lead with the fixed rules, in one short list. These are the only non-negotiables.
Follow with the flexible ranges, then say plainly what is free.
Add a kill list: the generic defaults this brand never uses. Pull them from the control set in Step 4.1 once you have it.
Give the same file to every agency, freelancer and agent. Version it, and keep old versions.
Prompt
You are producing work for a brand. Read brand/rules.yaml before anything else. Fixed rules are non-negotiable. Flexible rules give you a range: choose within it and vary your choices across the set. Free slots are yours to fill. Never use anything on the kill list. After each piece, list the flexible values you chose and confirm every fixed rule passed. If a request would break a fixed rule, stop and say which one.
Done when
One versioned brief serves every maker, human or agent
The kill list names at least five generic defaults
Make the set the system will really make, at the volume it will make it, without picking favourites.
WhoBrand designer
BudgetHalf a day
InThe generator, ten real briefs
OutA hundred pieces and a control set
Do
Take ten real briefs across channels and formats.
Generate ten pieces per brief with the system. Fix nothing by hand. Hand-fixing hides the rules you need to change.
Generate the same briefs with no brand rules at all. This is your control: the generic default.
Lay everything out at thumbnail size on one screen. Most generated work is seen at that size.
Prompt
Using brand/rules.yaml, generate ten variations for each of these ten briefs. Spread flexible choices across their full ranges. Then generate the same briefs with no brand rules as a control set. Give every piece a neutral code and keep the key in test/key.csv. Do not pick favourites or correct anything.
Done when
A hundred system pieces and a control set exist, untouched
Mix your pieces with competitors' and the control, hide every name, and ask buyers whose work each piece is.
WhoMarketing lead
BudgetTwo days
InThe set, competitor work
Outtest/results.csv
Do
Draw twenty pieces from your set, ten from two competitors and ten from the control. Remove names, logos and product shots.
Show them shuffled to at least twenty category buyers. Ask which brand made each one, with no list to choose from.
Score the share of your pieces named correctly, and the share named as a competitor.
Compare with the asset scores from Step 1.2. If whole pieces score lower than the signature assets alone, the system is diluting them.
Pattern
What it means
What to do
Yours named, control not
The rules carry the brand
Ship it. Keep this run as the baseline.
Yours and control alike
Recognition comes from category cues
Your fixed rules are generic. Back to Step 1.2.
Yours named as a rival
You share an asset
Fix something more unique, or pair the shared asset with one.
Some named, some never
A flexible range is too wide
Find what the unnamed pieces share. Narrow that range.
Working estimate
No published benchmark says what share is good enough for generated brand work. Treat your first run as the baseline and judge every later run against it.
Done when
Every system piece has a correct-name share and a rival-name share
Three weeks from a guidelines PDF to a tested rule sheet, then two hours a month to keep it honest.
Setup prompt
Help me turn our brand into a generative rule system.
1. Ask me what the brand sells, who buys it and who our two closest competitors are. Then help me audit the last fifty pieces we shipped and list every repeated element in brand/assets.csv, with a count and a precise description.
2. Help me design a fame and uniqueness test: name-hidden versions of the ten strongest candidates, and a script for asking category buyers which brand made each one. Do not invent results. Wait for real data.
3. Using the results, draft brand/rules.yaml. Every element is fixed, flexible or free. No more than three are fixed. Every rule is a value, a range or a list, with a check.
4. Write brand/brief.md for people and agents: fixed rules first, then ranges, then free slots and a kill list.
5. Generate ten variations for each of ten briefs, plus an unbranded control set, ready for a recognisability test. Keep the key in test/key.csv.
When
Steps
What happens
W1
1.1, 1.2
Audit shipped work. Start the fame and uniqueness fieldwork.
W2
2.1 to 3.2
Sort and write the rules. Encode them. Write one brief.
W3
4.1, 4.2
Generate a hundred pieces and a control. Run the recognisability test.
M
5.1, 5.2
Monthly drift check. Rules change only by decision.
A rule sheet, a distinctive asset audit and a recognisability test protocol. The filled values belong to the fictional sample brand.
brand/rules.yaml · sample values
brand: Ombrel # fictional sample
version: 1.0
fixed: # three at most
arch:
rule: Half-disc on a bar, radius 0.2 x width, tangerine, never cropped
check: shape match + colour match on every export
palette:
rule: Background bone or ink. Tangerine covers 5 to 15 percent of area
check: pixel share per colour
flexible:
type_scale: { headline: [18, 30], family: house sans, weight: 800 }
layout: [A, B, C] # all on an 8 percent margin
image: { treatment: line screen, angle: [0, 90], density: [3, 6] }
voice: { case: lower, max_words: 8, end: full stop, exclamation: never }
free:
- headline.word
- photo.subject
- offer
kill_list:
- stock gradients
- category blue
- exclamation marks
brand/assets.csv · distinctive asset audit
asset,type,description,count_in_50,competitor_uses,fame,uniqueness,quadrant,state
arch,shape,"half-disc on a bar, tangerine",41,no,,,,
tangerine_on_bone,colour,"tangerine accent on bone ground",38,partly,,,,
line_screen,image,"horizontal line screen, thicker toward base",22,no,,,,
house_sans_800,type,"heavy sans, tight tracking",50,yes,,,,
three_column,layout,"three-column grid, 8% margin",30,yes,,,,
# fame = share of buyers who name you with the name hidden
# uniqueness = share of namings that are you, out of all brands named
# quadrant = signature, candidate, shared or generic (Fig 0.2)
# state = fixed, flexible or free (Step 2.1)
test/protocol.md · recognisability test
Recognisability test
Set
- 20 pieces from our system, drawn at random from the full run
- 10 pieces from two competitors (5 each)
- 10 pieces from the unbranded control
- Remove every name, logo, product shot and URL. Match the size and format.
People
- At least 20 category buyers. Nobody from the team or its agencies.
Ask, for each piece, shuffled per person
- "Which brand do you think made this?" Open answer. No list.
- "What made you say that?" One sentence.
Score
- Correct share: our pieces named as us / our pieces shown
- Rival share: our pieces named as a competitor / our pieces shown
- Control share: control pieces named as us / control pieces shown
Read
- Compare with the last run, never with an outside benchmark.
- Group the "what made you say that" answers. The most named cue is your working signature.
- Any cue nobody names is a candidate to move from fixed to flexible.
Research · Playbook 09
Research
What this playbook rests on, how to measure whether it works for you, and what it cannot yet claim.
Every core claim in this playbook, with what supports it. Grades describe the kind of evidence, not how much we like the claim. Claims graded P come from practice and are labelled as such.
1A · Experiment
2B · Field data
2C · Framework
2P · Practitioner
Claim
Grade
What supports it
Source
Generated work drifts toward the common.
A · Experiment
In a 293-writer experiment, AI ideas made individual stories better rated and the set of stories more similar [1]. Preference-tuned models produced much less varied outputs than the same models without that tuning [3]. Neither study tested brand work.
1, 3
Generative image tools raise output and lower visual novelty.
B · Field data
On an online art platform, artists who adopted text-to-image tools made about 25% more work and earned about 50% more favourites per view, while the visual novelty of their work fell [2]. Observational data from one platform.
2
A brand asset is measured by fame and uniqueness with the name hidden.
C · Framework
The distinctive asset framework measures the share of category buyers who link an unbranded asset to the brand (fame) and how far the brand owns that link against competitors (uniqueness) [4].
4
Some asset types are easier to own than others.
B · Field data
Across 1,281 in-market elements in 13 packaged goods categories and 19 countries, characters, logos and logotypes showed the most potential for unique ownership. Colour was harder to own because competitors share it. Execution varied widely within each type [5].
5
Brand value rests on strong, favourable and unique associations.
C · Framework
The customer-based brand equity model locates equity in brand awareness and in associations that are strong, favourable and unique [6].
6
Locking a few distinctive rules keeps generated work recognisable better than locking many generic ones.
P · Practitioner
MainArc's thesis, consistent with [4], [5] and [6]. Untested on generated brand work. The recognisability test in Step 4.2 is designed to test it.
Practice
Without monthly checks, generated brand work drifts toward the generator's defaults.
P · Practitioner
MainArc's thesis, extrapolated from [1], [2] and [3]. Not measured on brands over time.
Practice
References
[1]Doshi, A. R., & Hauser, O. P. (2024). Generative AI enhances individual creativity but reduces the collective diversity of novel content. Science Advances, 10(28). doi.org/10.1126/sciadv.adn5290
[2]Zhou, E., & Lee, D. (2024). Generative artificial intelligence, human creativity, and art. PNAS Nexus, 3(3), pgae052. doi.org/10.1093/pnasnexus/pgae052
[3]Kirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Hambro, E., Grefenstette, E., & Raileanu, R. (2024). Understanding the effects of RLHF on LLM generalisation and diversity. International Conference on Learning Representations (ICLR 2024). arxiv.org/abs/2310.06452
[5]Ward, E., Yang, S., Romaniuk, J., & Beal, V. (2020). Building a unique brand identity: Measuring the relative ownership potential of brand identity element types. Journal of Brand Management, 27(4), 393–407. doi.org/10.1057/s41262-020-00187-6
[6]Keller, K. L. (1993). Conceptualizing, measuring, and managing customer-based brand equity. Journal of Marketing, 57(1), 1–22. doi.org/10.1177/002224299305700101
Grades
A controlled experiment. B large-scale field data. C an established peer-reviewed model or guideline, not a direct test of the claim. P practitioner claim from MainArc's work, untested.
A generative brand is only as good as the work it prints. Measure whether buyers can still tell it is you when the name is gone.
Instruments
01
Name-hidden recognition
Category buyers see unbranded pieces and say which brand made each one, with no list. The share named correctly is the main score.
02
Rival naming
The share of your pieces named as a competitor. It shows which assets you share, the uniqueness problem in [4] and [5].
03
Control gap
The same briefs generated with no brand rules. Your pieces should be named far more often than the control. A small gap means the rules add little.
04
Cue rationale
One sentence per answer on what gave it away. Group the answers. The cues people name are your working signature.
Study design
Element
Setting
Set
20 system pieces, 10 competitor pieces, 10 control pieces, all unbranded
People
At least 20 category buyers from outside the team
Questions
Open brand naming, then one sentence on the cue
Analysis
Correct share, rival share and control share per run. Cues grouped by asset.
Cadence
Before launch, after any change to a fixed rule, and every quarter
Reading the results
If your pieces are named well above the control and rarely as a rival, the fixed rules are carrying the brand.
If the control is named as you almost as often as your own work, your fixed rules are category conventions. Go back to the asset audit.
If whole pieces score below the signature assets tested alone, the flexible ranges are diluting them. Narrow the widest range first.
Running signals
Signal
What to watch for
Fixed rule failures in shipped work
Should be zero. Any failure is a system fault, never only a file fault
Clustering inside flexible ranges
A range the generator always fills the same way has become a hidden default
Distance from the control
Thumbnails drifting toward the unbranded control set is the earliest warning
Note
Thresholds here are starting points to tune, not findings. A small panel shows direction and disagreement. It does not prove significance, and nothing on this page should be reported as if it did.
How this playbook was built, where it is weak, and how to cite it.
How this playbook was built
Built from the distinctive brand asset framework in marketing science, the customer-based brand equity model, published research on homogenization in AI-assisted creative work, and MainArc's own practice with rule-based brand systems. It is a practitioner method. Claims are graded on the evidence page.
Limitations
None of the cited studies tested generated brand work. The link from homogenization research [1, 2, 3] to brands is MainArc's inference.
The asset ownership evidence [5] comes from packaged goods. Other categories may rank asset types differently.
The rule engine on the introduction page uses a fictional brand. Its distinctiveness weights are sample values and its recognisability score is a heuristic, not a measurement.
There is no published threshold for a good recognisability score in generated work. The benchmark compares a brand with its own earlier runs.
Open questions
How many fixed rules does a brand need before extra locks stop adding recognition?
Do flexible ranges widen or narrow over time when agents produce most of the work?
Can name-hidden recognition be estimated automatically, without a panel, and still be trusted?
Cite this playbook
APA
MainArc Research. (n.d.). Build a generative brand system (Playbook No. 09, Version 1.0). MainArc. https://mainarc.xyz
BibTeX
@misc{mainarc_brandsystem,
author = {{MainArc Research}},
title = {Build a generative brand system},
howpublished = {MainArc Playbook No. 09, Version 1.0},
url = {https://mainarc.xyz}
}
Corrections
Found an error, a stronger source, or a result from running the benchmark? Send it through mainarc.xyz. Corrections are made in the next version.
About this playbook
This playbook is published by MainArc, a creative intelligence lab for leadership and product teams.
We do three things. We research how AI is changing creative work through long-form reports, design teardowns, and operator essays. We design the workflows, prompt systems, review loops, and decision frameworks that teams actually run on. And we build the work the strategy points to: brand systems, interfaces, product prototypes, internal tools, and launch concepts.
One operator. One AI agent. No layers in between. Every engagement runs under strict NDA. No case studies, no client logos, no shared work.
If this research resonates with a problem your team is facing, start with an audit. One week. A clear map of where AI fits, where it doesn't, and what to build next.
This is independent research. Not financial advice. Not career advice. Not the final word on anything. It's one person's perspective shaped by building, shipping, and paying attention. Do your own thinking. Challenge the ideas. Disagree with specifics. Build on what resonates. Discard what doesn't.
House rules
What doesn't belong here: hate, trolling, bad-faith takedowns, or drive-by takes from people who've never built anything. If you disagree, show your work. Builder's critique. Everyone else just comments.
A MainArc playbook for agents, voice, wearables and ambient products. When the interface is a presence instead of a screen, timing is the design. How to decide when a product speaks, when it batches, when it waits and when it stays quiet.
PlaybookNo. 10Version1.0Read24 minSources6 citedByMainArc ResearchEvidence and method →
Thesis
Without a screen, timing is the whole design: the scarce decision is when a product should speak, and most signals should never interrupt anyone.
Do this Monday
List every way your product reaches someone without being asked, count a typical person's day, and mark each signal interrupt, hold, batch or drop.
An interruption budget for one sample day. It is live. Twelve things happen to one person, and a policy decides what each one does to them. Press Send everything to see a day with no policy. Then go back to the policy, switch off batching, and watch an urgent message arrive after it stopped mattering.
Interruption budget · sample day
FreeFocus blockMeetingCommute
Sample day and sample events. Digests at 12:30 and 18:50. Calm score is a working estimate with sample weights
4Moves
5Stages
10Steps
3Templates
What this is
On a screen, design is mostly about layout. People choose when to look. Take the screen away and that choice moves to the product. A voice assistant, a ring, a watch, a light on a speaker or an agent working in the background decides for itself when to reach a person. Every one of those decisions spends something that belongs to someone else.
Agents make the problem bigger. An agent that watches your inbox, calendar, travel and accounts always has something to say. The hard question is no longer what to tell people. It is which of the hundred things it could say today deserve to be said, and when.
Generating a message is free. Deciding whether it deserves someone's attention is the judgment. This playbook turns that judgment into a policy: an inventory of signals, a score for each, a budget, four moves, and a loop that learns from every dismissal.
Principles
01
Silence is the default
Every signal starts as dropped. It has to earn its way to a digest, and earn more to interrupt.
02
The person pays
An interruption costs the sender nothing and the receiver focus, mood and time. Design from the receiver's side.
03
When beats what
The same message at a good moment and a bad one are two different experiences. Timing is a design decision, so design it.
04
Batch what can wait
Most signals lose nothing in a few hours. Put them in a digest the person chose the time for.
05
Use the quietest surface
A light before a tap, a tap before a voice, a voice before a screen. Escalate only when the signal earns it.
06
Dismissals are data
Every swipe away is a vote on the policy. Count them, read them and change the policy.
How to use this playbook
01
Break the budget first
Switch off batching and watch low-priority signals spend the budget the urgent one needed.
02
Read the three essays
Attention is the budget, the four moves, and why calm is a feature. The steps assume all three.
03
Put one product on a budget
One product, one week, five stages. The plan is in Run it.
04
Copy the files
The signal inventory, the interruption policy and the earn the interruption checklist are on the Templates page.
Playbook 10 in the set. Playbook 04 decides what an agent may do on its own. This one decides when it may speak.
Every interruption is paid for by the person who receives it. Products without a screen spend that budget on their behalf.
Interruptions have a price
In a controlled experiment, people who were interrupted while working finished their tasks faster, with no drop in quality. They paid for it in other ways: higher stress, more frustration and more effort than when they worked uninterrupted [1]. The work looked fine. The person did not feel fine.
That is why interruptions are easy to underrate. The cost does not show up in the output. It shows up in the person, and a product that only measures engagement will never see it.
The moment changes the price
The same interruption costs more at some moments than others. In one lab study, interruptions placed at poor moments in a task caused more annoyance, frustration and time pressure than the same interruptions placed between subtasks, and people felt less respected by the system that sent them [2]. In a later study from the same lab, holding notifications until a break in the task lowered frustration for one of the two tasks tested, compared with delivering them at once. The average wait was about a minute and a half [3]. The evidence on timing is consistent, and the effects are modest. That is a reason to design for it, and a reason to measure your own.
Fig 0.1Same messages, two schedulesMove to scrub · press to playtop rang 0 · held 0 · 0 breaks
Marble: one message, the same eight in each runCarriage: the time of the morningBack run · rings on arrivalFront run · gate lifts at breaksSample schedule. The messages are identical. Only the timing changed. Why that matters is in [2] and [3]. The back bell rings eight times; the front one rings at three breaks.
Why it gets worse without a screen
A phone notification can be ignored. It sits in a tray until the person looks. A voice in your ear, a buzz on your wrist or a light in your kitchen cannot be filed away. Ambient products reach people while they are driving, cooking, talking or asleep. They have more power over attention than a screen does, and fewer ways for the person to defer them.
So the budget has to be explicit. Decide how many times a day the product may reach a person, spend that number on purpose, and treat going over it as a defect.
MainArc Research
Production is free. Judgment isn't. A message costs nothing to generate. The attention it takes is paid by someone else, and deciding whether it is worth that price is the design.
Every signal an ambient product could send ends in one of four moves: interrupt now, batch, hold for a better moment, or drop. Choosing the move is the whole job.
Research on mixed-initiative interfaces set out the logic decades ago. A system should act only when it expects acting to be worth more than not acting, should model the person's attention, and should weigh the cost and benefit of deferring until they are ready [4]. The four moves are that logic made practical. Three questions, asked in order, decide every signal.
Fig 0.2Three questions, four movesPress to send a signalbelt running · 4 queued
01 Drop · chute to a closed bin02 Batch · the digest tray03 Hold · shelf and deadline timer04 Interrupt · striker and bellToken: a sample signal in motionInterrupt is the last answer on the path. A held signal still interrupts later, at a better moment. The gates ask in order: would anyone act on it, does waiting cost anything, is now a good moment.
Move
Use it when
The person experiences
Sample
Interrupt
Someone would act within minutes, and waiting would cost them
A voice, a tap or a sound, right now
Sign-in from a new device
Hold
It matters soon, but this moment is bad
Nothing yet. A quiet indicator. Then a delivery at the next break
Room change during a focus block
Batch
Useful, but nothing breaks if it waits hours
One line in a digest at a time they chose
Invoice paid
Drop
Nobody would do anything differently after hearing it
Nothing. It stays in a log they can open
Someone reacted to your doc
Hold is the move most products forget. It is the difference between a product that respects a meeting and one that buzzes through it. A hold always has a deadline: if the better moment never comes, the signal goes out before it stops being useful.
The best ambient products inform without demanding. They live at the edge of attention and come to the centre only when they have to.
Early research on ubiquitous computing named this calm technology: technology that engages both the centre and the periphery of attention, and moves between them [5]. The periphery is what you are aware of without attending to it, like traffic noise or the light changing. A product that lives there can keep someone informed without taking anything from them.
Fig 0.3From periphery to centreMove to set the urgencyurgency 6 · ambient light
01 Ambient light · lamp02 Wrist tap · tapper03 Voice · speaker04 Notification · screen05 Ringing alert · bellStart at the left. Move right only when the signal has earned it. MainArc's ordering, after the centre and periphery idea in [5]. The carriage steps right only when its urgency rod clears the next, higher hurdle, and the device in front of it switches on.
Noise teaches people to stop listening
Calm also protects the signals that matter. In a large study of clinical alerts, the likelihood that a clinician accepted a reminder dropped by about 30% for each additional reminder in the same visit, and fell further as the share of repeated reminders rose [6]. The setting was a hospital, and the mechanism is general. Every unnecessary interruption makes the next necessary one less likely to land.
In a calm product every interruption is believed, because there are so few of them.
The shift
Most teams measure ambient products by how often people engage. Measure them by how often people were reached and did not need to be. The second number is where the design lives.
Anything the product sends without being asked is a signal. Most teams undercount them by half, because each one was added by a different person for a good reason.
WhoPM, designer, engineer
BudgetHalf a day
InEvent logs, notification code, agent tools
Outsignals.csv
Do
List every output the person did not request: notifications, sounds, haptics, lights, spoken lines, emails, badges and agent messages.
Include signals from integrations and partners. If it reaches your user through your product, it is yours.
For each, record the trigger, the current surface and how often it fires per active person per day, from logs where you have them.
Add up a typical person's day. That number is your baseline, and it is usually a surprise.
Done when
Every self-started output appears in signals.csv
You know how many times a typical person is reached per day today
Prompt
Read our notification, messaging and agent code. List every place the product reaches a person without being asked: notifications, sounds, haptics, lights, spoken output, emails and badges. For each, give the trigger, the surface and, if logs exist, how often it fires per active person per day. Save it as quiet/signals.csv. Do not guess frequencies. Leave them blank if there is no data.
Three numbers for every signal. Keep the scale small so people agree on it.
WhoPM, designer
BudgetTwo hours
Insignals.csv with jobs
OutUrgency, relevance and cost of delay per signal
Score
1
2
3
Urgency
Can wait a day
Can wait hours
Minutes matter
Relevance
About something they do not own
About their work or life
Needs them, personally
Cost of delay
None
Inconvenience
Money, safety, security or a missed commitment
Do
Score each signal on the three scales. Score the typical case, then note the exceptions.
Multiply urgency by relevance for a priority from 1 to 9. In the sample tool, 1 drops, 3 or less batches and 6 or more counts as important. Those cut-offs are working estimates. Tune them.
Any signal with cost of delay 3 gets a named owner who signs off on its move.
Done when
Every signal has three scores and a priority
Every cost-of-delay 3 signal has an owner
Prompt
For each row in quiet/signals.csv, propose urgency, relevance and cost of delay on a 1 to 3 scale using the definitions I give you, plus a maximum wait in minutes. Explain each score in one line. Flag every signal where you are unsure, and every one where the job is "nothing". I will make the final call.
A daily and hourly limit on interruptions. Without a number, every team adds one more signal and nobody adds them up.
WhoPM, design lead
BudgetOne hour
InThe baseline from step 1.1
OutLimits in policy.json
Do
Start from the baseline. A reasonable first cut is half of it. Treat that as a working estimate. The learning loop in stage 05 tells you where it should land.
Set an hourly cap as well as a daily one. Five interruptions in ten minutes feel worse than five spread across a day.
Keep a reserve for urgent signals. Low-priority signals must never be able to spend the last of the budget.
Let the person move their own budget up or down, and show them how much is left.
Try it
In the tool on the introduction, switch off Batch low priority. The weekly summary, the invoice and the package spend the budget before lunch, and the road closure at 17:20 waits for the evening digest, long after the driver needed it. Then turn batching back on and drag the budget down to 5. A budget set too tight fails the same way. Without priorities, a budget only moves the problem.
Done when
policy.json has a daily cap, an hourly cap and a reserve
Let the person choose digest times. Two a day is a sensible place to start testing.
Order every digest exceptions first, then money, then everything else.
Give every hold a deadline from its maximum wait. If the break has not come by then, send it anyway.
Expire what goes stale. A held "meeting in 10 minutes" after the meeting starts is worse than nothing.
Done when
A spoken digest takes under a minute to hear
No held signal can be delivered after its deadline
Prompt
Here are today's batched signals for one person. Write the digest as a spoken script under 45 seconds. Lead with anything that needs them to act, then money, then a single line for everything else. No greetings, no filler, no exclamation marks. Then write the same digest as a five-line notification.
Once a week, move signals between moves based on what people did. Misses matter more than dismissals.
WhoPM, designer
BudgetOne hour a week
Inresponses.csv
OutAn updated policy.json, a change note
If a signal is mostly
Then
Dismissed
Move it one step quieter: interrupt to hold, hold to batch, batch to drop
Muted
Drop it for that person, and ask whether it should exist for anyone
Missed
Move it one step louder, or shorten its maximum wait
Acted on
Leave it. Check whether it could arrive at a better moment
Do
Start every review with misses. One missed urgent signal outweighs a week of dismissals.
Move signals one step at a time, and write down why.
Offer people the change in plain words: "You usually dismiss these. Put them in your evening digest instead?"
Never make a signal louder without telling the person.
Done when
Every missed urgent signal has a fix in the policy
The interruptions per person per day are lower than last month, or you know why not
Prompt
Read quiet/responses.csv for the last week. For each signal, give the share acted on, dismissed, muted and missed. List every miss first. Then propose at most five changes to quiet/policy.json, one step at a time, with the evidence for each. Do not change the file until I agree.
One product, one week. By Friday every signal has a move, and the product has a number it is not allowed to exceed.
Setup prompt
Help me put our product on an interruption budget.
1. Read our notification, messaging, voice and agent code. List every way the product reaches a person without being asked. Save it as quiet/signals.csv with columns from the signal inventory template. Leave frequencies blank where there is no data.
2. For each signal, ask me what the person would do after hearing it and how long it stays useful. Then propose urgency, relevance and cost of delay on a 1 to 3 scale, and a move: interrupt, hold, batch or drop.
3. Draft quiet/policy.json from the template: a daily and hourly cap, a reserve for urgent signals, protected windows with breakthrough lists, digest times, and the form each move takes on each of our surfaces.
4. Add logging for five outcomes per delivery: acted, opened, dismissed, muted, missed.
5. Do not invent usage numbers. Show me every move you propose and ask what is wrong with it.
Day
Steps
What happens
D1
1.1, 1.2
Inventory every signal. Count the baseline. Name each job.
D2
2.1, 2.2
Score every signal. Define the person states and their breakpoints.
D3
3.1, 3.2
Set the budget and the reserve. Write the breakthrough lists.
D4
4.1, 4.2
Fill the move by surface grid. Write the digest and hold rules.
D5
5.1
Ship logging. Book the first weekly retune, starting with misses.
Before a signal may interrupt anyone, it must pass all of these.
[ ] After hearing it, the person will do something specific.
[ ] Waiting until the next digest would cost them something real.
[ ] Waiting until the next break would cost them something real.
[ ] It fits inside today's budget without using the urgent reserve,
or it is urgent enough to use the reserve.
[ ] It uses the quietest surface that will still be noticed.
[ ] The first words say what to do. The reason comes second.
[ ] It has a maximum wait, and expires after it.
[ ] The person can say "not this" in one tap or one word.
[ ] Someone owns it, and reads its outcomes every week.
Fails one box: hold or batch it.
Fails the first box: drop it.
Research · Playbook 10
Research
What this playbook rests on, how to measure whether it works for you, and what it cannot yet claim.
Every core claim in this playbook, with what supports it. Grades describe the kind of evidence, not how much we like the claim. Claims graded P come from practice and are labelled as such.
3A · Experiment
1B · Field data
2C · Framework
2P · Practitioner
Claim
Grade
What supports it
Source
Interruptions cost the person even when the work looks fine.
A · Experiment
In a lab experiment with 48 people, interrupted participants finished email tasks faster with no loss of quality, and reported more stress, frustration and effort [1]. Lab tasks, short sessions.
1
When an interruption lands changes what it costs.
A · Experiment
Interruptions at moments between subtasks caused less annoyance, frustration and time pressure, and conveyed more respect, than the same interruptions at poor moments [2]. Sixteen participants.
2
Holding notifications for a natural break can reduce frustration.
A · Experiment
Deferring notifications to breakpoints lowered frustration in one of two tasks compared with immediate delivery, with an average wait of about 90 seconds [3]. The effect was not significant in the other task.
3
A system should weigh the value of acting, and the cost of the person's attention, before it acts or speaks.
C · Framework
The principles of mixed-initiative interfaces include modelling the user's attention, weighing the costs and benefits of deferring action, and acting only when action has greater expected value than inaction [4].
4
Good ambient technology lives in the periphery and comes to the centre only when needed.
C · Framework
Calm technology is defined as engaging both the centre and the periphery of attention and moving between them [5]. An influential design idea, not a tested effect.
5
Every unnecessary alert makes the next one less likely to land.
B · Field data
In clinical software, the likelihood of accepting a reminder fell by 30% for each additional reminder per encounter, and by 10% for every five-point rise in the share of repeats [6].
6
An explicit budget, four moves and a weekly retune make a product calmer without missing what matters.
P · Practitioner
MainArc's method. Built on [2], [3] and [4], untested as a system.
Practice
The thresholds, sample day and calm score in the interruption budget tool are right.
P · Practitioner
Working estimates chosen for illustration. Your own thresholds come from your logs.
Practice
References
[1]Mark, G., Gudith, D., & Klocke, U. (2008). The cost of interrupted work: More speed and stress. Proceedings of CHI '08, 107–110. ACM. doi.org/10.1145/1357054.1357072
[2]Adamczyk, P. D., & Bailey, B. P. (2004). If not now, when? The effects of interruption at different moments within task execution. Proceedings of CHI '04, 271–278. ACM. doi.org/10.1145/985692.985727
[3]Iqbal, S. T., & Bailey, B. P. (2008). Effects of intelligent notification management on users and their tasks. Proceedings of CHI '08, 93–102. ACM. doi.org/10.1145/1357054.1357070
[4]Horvitz, E. (1999). Principles of mixed-initiative user interfaces. Proceedings of CHI '99, 159–166. ACM. doi.org/10.1145/302979.303030
[5]Weiser, M., & Brown, J. S. (1997). The coming age of calm technology. In P. J. Denning & R. M. Metcalfe (Eds.), Beyond calculation: The next fifty years of computing (pp. 75–85). Springer. Revised from “Designing calm technology”, PowerGrid Journal, 1996. doi.org/10.1007/978-1-4612-0685-9_6
[6]Ancker, J. S., Edwards, A., Nosal, S., Hauser, D., Mauer, E., Kaushal, R., & HITEC Investigators. (2017). Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Medical Informatics and Decision Making, 17, 36. doi.org/10.1186/s12911-017-0430-8
Grades
A controlled experiment. B large-scale field data. C an established peer-reviewed model or guideline, not a direct test of the claim. P practitioner claim from MainArc's work, untested.
A calmer product should reach people less and miss nothing. Measure both, every week, and never one without the other.
Instruments
01
Interruption load
Interruptions per active person per day, and per hour at the peak. The number the budget controls.
02
Outcome mix
For every delivery: acted, opened, dismissed, muted, missed. The act rate per signal is the headline.
03
Missed important
Signals of priority 6 or more that arrived after their maximum wait, or never. The target is zero.
04
Felt calm
A short survey, 1 to 5: it tells me what matters. It leaves me alone when I am busy. I trust its alerts.
Study design
Element
Setting
Signals
Every self-started output in signals.csv
Window
Two weeks before a policy change, two weeks after
People
All active users for behaviour. 20 for the survey
Analysis
Load, act rate and misses per signal, before and after. Survey medians before and after.
Cadence
Weekly review, misses first
Reading the results
A change that lowers load and raises misses has failed, however calm it looks.
If the act rate for a signal keeps falling, it is fatiguing people. Move it one step quieter [6].
If people raise their own budget, the policy is too quiet for them. Ask which signals they missed.
Running signals
Signal
What to watch for
Act rate on interruptions
Falling over weeks is the first sign of alert fatigue
Mutes per week
Each mute is a signal someone decided was not worth it. Read them
Holds released after deadline
Should be zero. Any at all means the hold rules are broken
Note
Thresholds here are starting points to tune, not findings. A small panel shows direction and disagreement. It does not prove significance, and nothing on this page should be reported as if it did.
How this playbook was built, where it is weak, and how to cite it.
How this playbook was built
Built from human-computer interaction research on the cost and timing of interruptions, the principles of mixed-initiative interfaces, the idea of calm technology, and field data on alert fatigue, combined with MainArc's practice. It is a practitioner method. Claims are graded on the evidence page.
Limitations
The interruption studies [1, 2, 3] are short lab studies with small samples on desk tasks. Voice, wearables and ambient light were not tested.
The alert fatigue evidence [6] comes from clinical software. The mechanism is general, the numbers are not.
Reading a person's state from calendars, motion and devices is imperfect and raises privacy questions this playbook only touches.
The sample day, priority thresholds and calm score weights in the tool are illustrative working estimates.
Open questions
How should one interruption budget be shared across a phone, a watch, a speaker and an agent?
Do people trust a product more when it shows what it is holding, or when it stays fully silent?
How quickly does alert fatigue set in for ambient surfaces compared with screens?
Cite this playbook
APA
MainArc Research. (n.d.). Design for no screen (Playbook No. 10, Version 1.0). MainArc. https://mainarc.xyz
BibTeX
@misc{mainarc_noscreen,
author = {{MainArc Research}},
title = {Design for no screen},
howpublished = {MainArc Playbook No. 10, Version 1.0},
url = {https://mainarc.xyz}
}
Corrections
Found an error, a stronger source, or a result from running the benchmark? Send it through mainarc.xyz. Corrections are made in the next version.
About this playbook
This playbook is published by MainArc, a creative intelligence lab for leadership and product teams.
We do three things. We research how AI is changing creative work through long-form reports, design teardowns, and operator essays. We design the workflows, prompt systems, review loops, and decision frameworks that teams actually run on. And we build the work the strategy points to: brand systems, interfaces, product prototypes, internal tools, and launch concepts.
One operator. One AI agent. No layers in between. Every engagement runs under strict NDA. No case studies, no client logos, no shared work.
If this research resonates with a problem your team is facing, start with an audit. One week. A clear map of where AI fits, where it doesn't, and what to build next.
This is independent research. Not financial advice. Not career advice. Not the final word on anything. It's one person's perspective shaped by building, shipping, and paying attention. Do your own thinking. Challenge the ideas. Disagree with specifics. Build on what resonates. Discard what doesn't.
House rules
What doesn't belong here: hate, trolling, bad-faith takedowns, or drive-by takes from people who've never built anything. If you disagree, show your work. Builder's critique. Everyone else just comments.
A MainArc playbook for creative and product leads. AI takes the junior work first: the first drafts, the variations, the production. That work was where juniors learned to judge. How to find where judgment used to form on your team, protect the reps that matter, and design where judgment forms now.
PlaybookNo. 11Version1.0Read25 minSources6 citedByMainArc ResearchEvidence and method →
Thesis
The bigger AI risk to creative teams is a broken apprenticeship: the tasks AI absorbs first were the reps that built judgment, and without new ones, seniors stop being made.
Do this Monday
List ten tasks your juniors did a year ago. Mark which ones AI now does, and for each, name where that judgment rep happens now, or write nowhere.
An apprenticeship map for a sample team of four juniors. It is live. Ten tasks that filled a junior's week, each done by a person or by AI. Some of them build judgment. Some are only production. Press Automate everything and watch the pipeline empty. Then go back to Today, unplanned and give every lost rep a deliberate replacement.
Apprenticeship map · sample teamToday, unplanned
Sample team and sample tasks. Hours, rep weights and the pipeline model are working estimatesPerson 1.0 · deliberate rep 0.8 · AI alone 0.1
5Stages
10Steps
4Deliberate reps
3Templates
What this is
Most talk about AI and creative work is about replacement. Will the machine take the designer's job, the writer's job, the analyst's job? That question gets the headlines. A quieter one decides whether a team still works well in five years.
AI takes the junior work first. First drafts, ten variations of a concept, resizing, research summaries, production polish. That work was never only output. It was where juniors saw a hundred briefs, made a thousand small choices and had a senior tell them which ones were wrong. Hand it to a machine without a plan and the work still gets done. The people who would have learned from it stop learning.
Production is free. The judgment people built by doing it is not, and it now has to be designed on purpose. This playbook maps where judgment used to form on your team, protects the reps that matter most, and builds new ones where the old ones vanished.
Principles
01
Count reps, not tasks
A task is a container. What matters is the decision inside it, and whether a junior still makes it.
02
Automate the making, keep the choosing
Let machines produce. Keep juniors picking, defending and finishing, with a senior watching.
03
Draft before you look
Seeing the machine's answer first ends the rep. Commit to a rough version, then compare.
04
Make senior thinking visible
Judgment is invisible by default. Seniors have to say out loud why one option lost.
05
Feedback makes it practice
Work with no verdict attached teaches slowly. Every protected rep ends with a reason from someone better.
06
Measure what people can do alone
Output with help shows the tool's skill. Skill shows up when the help is gone.
How to use this playbook
01
Empty the pipeline first
Press Automate everything. Then add reps back one at a time and see which ones matter.
02
Read the three essays
Production was the classroom, help that does the thinking, and why the same tool can teach.
03
Map one team in a week
Ten tasks, five stages, one review. The plan is in Run it.
04
Copy the files
The rep map, the draft first protocol and the unassisted check are on the Templates page.
Playbook 11 in the set. Playbook 05 measures a team's taste. This one makes sure the team keeps growing people who have it.
Nobody designed junior work as training. It trained people anyway, because every piece of it came back with a verdict.
The craft was learned at the bench
Crafts have always been learned by apprenticeship. A learner watches an expert, tries the work, gets corrected and slowly takes on more. Research on cognitive apprenticeship sets out the teaching moves that make it work: modelling, coaching, scaffolding that fades, putting thinking into words, comparing your work with an expert's, and finally exploring alone [1]. It also names the hard part. In a workshop you can watch the hands. In thinking work, the expert's process stays invisible unless someone makes it visible.
On creative and product teams, junior work did this quietly. A junior wrote the first draft and a senior rewrote it in front of them. A junior made twelve variations and watched which two survived the review, and heard why. Nobody called it a curriculum. It was one.
Doing the work was never enough
Research on expert performance separates three kinds of experience: work, play and deliberate practice. Deliberate practice is effortful, designed to improve performance, and built around feedback [2]. Work on its own is a weaker teacher, because it rewards getting the thing done over getting better at it.
That sharpens the argument. Junior production was valuable because feedback came attached. The draft was the occasion. The senior's red pen was the lesson. When AI writes the draft, the occasion goes, and the red pen has nothing to mark.
Fig 0.1The ladder loses its bottom rungsMove to pull · press to climbbefore ground · now ground
Rungs 1 to 5: production, variations, first drafts, owning a piece, setting directionClimber: a junior, who can only reach the next rungLeft: before. Right: nowMachine: draws out rungs 1 to 3, AI does these nowA model, not a measurement. The top rungs still need climbers. The rungs being automated first are the ones that used to make them. Each climber reaches one rung at a time, so the right one stands on the step with a gap of three rungs above it.
MainArc Research
Production is free. Judgment isn't. And judgment was rarely taught head on. It came as a by-product of production with a senior watching. Take the production away and the by-product goes with it, unless someone rebuilds it on purpose.
Help that thinks for a learner raises today's output and can lower tomorrow's ability. The pattern shows up in classrooms, cockpits and offices.
The classroom
In a field experiment with nearly a thousand high school maths students, one group practised with a general chat tutor and another with a tutor built to give hints instead of answers. With the general tutor, practice grades rose 48%, and when the help was taken away for the exam, those same students scored 17% lower than students who never had it [3]. The hint tutor raised practice grades even further, and its students did about as well as the group without AI on the exam [3].
Fig 0.2Better with help, worse withoutPress to switch practice, exampractice · +48% · +127%
Rams, left to right: no AI, general tutor, hint tutorStop on each scale: zero, students who practised without AIValve lever: AI help, open in practice, shut for the examBelow zero: 17% lowerThe help that gave answers produced better practice and worse learning. The help that gave hints kept the learning. Bars drawn to scale from the reported results in [3]. Each ram stands at its group's change in grades against students who practised without AI.
The design of the help made the difference. Answers ended the rep. Hints kept the student doing the thinking.
The cockpit
Aviation met this problem first. In a simulator study of airline pilots who flew mostly on automation, hand flying skills were largely intact. The skills that slipped were cognitive: keeping track of where the aircraft was without the map display, deciding what came next, and noticing when an instrument had failed. The authors link keeping those skills to how actively pilots stay engaged in supervising the automation [4].
What automation left alone, and what it wore downAfter [4]
Held up
Scanning the instruments
Hands on the controls
The parts you can see from outside
Slipped
Knowing where you are
Deciding what comes next
Noticing that something is wrong
Read that list as a creative lead. Hand skills survive. What wears down is the thinking: where am I in this problem, what should happen next, and is this output wrong.
The office
In a survey of 319 knowledge workers describing 936 real uses of AI at work, people with more confidence in the AI reported doing less critical thinking, and people with more confidence in their own skill reported doing more [5]. It is self-report and correlation. It shows a pattern and cannot show a cause. If the pattern holds, juniors, who have the least confidence in their own judgment, are the people most likely to hand it over.
AI can also make juniors better, faster. Whether it builds judgment or replaces it depends on how the work around it is designed.
The other half of the evidence
In a study of 5,172 customer support agents, access to an AI assistant raised issues resolved per hour by 15% on average [6]. Less experienced and lower-skilled agents gained the most, in both speed and quality, while the most experienced saw small gains in speed and small declines in quality. The authors also report that the assistant helped workers learn [6].
So the honest picture has two sides. Help that hands over answers can hollow out skill [3]. Help that spreads what the best people know can speed juniors up [6]. The tool can be the same in both. The difference is whether the junior still makes the call, gets a verdict and hears why.
A team is a pipeline
Juniors become mid-level. Mid-level become senior. Seniors set the bar the machines are measured against, and teach the next juniors. Every judgment rep the juniors lose narrows the pipe a little. Nothing breaks this quarter. The shortage shows up later, when seniors leave and nobody below them has ever chosen a direction alone.
Fig 0.3Reps feed the pipelinePress to cut · move to add reps3 of 5 feeding · seniors full
Tanks, top to bottom: juniors, mid-level, seniorsGang bar: drafts, variations, research, cut by automationDeliberate reps: crit, draft firstPump: seniors teach the next juniorsA model, not a measurement. MainArc's thesis: the shortage arrives long after the reps vanish, which is why teams rarely notice it in time. Each tank feeds the next, so a cut at the taps reaches the seniors' tank last.
The shift
Teams ask which tasks AI can do. Ask a second question every time: where does the judgment this task used to build form now? If the answer is nowhere, production just got cheaper and seniors just got scarcer.
Start from a year ago. Write down the ten tasks that filled a junior's week, then mark which ones AI does now.
WhoTeam lead, two juniors, one senior
BudgetOne hour
InProject boards, calendars, memory
Outreps.csv with ten tasks
Do
Ask two juniors and one senior, separately: what did a junior spend most of the week on a year ago? Merge the answers into ten tasks.
For each task, estimate the hours a week then and now. Rough numbers are fine. Label them as estimates.
Mark who does each task today: a person, AI, or shared (AI drafts, a person finishes).
Include the tasks nobody is proud of. Resizing, tidying files, writing up notes. Some of them taught more than they looked like they did.
Done when
Ten tasks are listed with hours then and now
Every task is marked person, AI or shared
Prompt
Here are my notes on what our junior designers, writers and PMs did each week a year ago, and what they do now. Turn them into a table of at most ten tasks with the columns task, hours_then, hours_now and who_now (person, ai or shared). Mark every hour figure as an estimate. Ask me about anything ambiguous. Do not add tasks I have not mentioned.
Not every rep needs saving. Keep the few that build the judgment your team is paid for, and let the rest go.
WhoCreative or product lead
BudgetOne hour
Inreps.csv with its nowheres
OutThree protected reps
Do
Rank the reps by how close they sit to the judgment clients pay for. Usually that means reading the brief, choosing a direction and knowing when the work is finished.
Protect the top three. Juniors do them by hand at least once a week, even when AI could do them faster.
Automate every production-only task without guilt. It never built anyone.
Write the cost down: the hours a week the protected reps take compared with letting AI do them. Protection is a budget line, so treat it as one.
Try it
In the tool on the introduction, compare With deliberate reps against Automate everything. Both free most of the production hours. Only one keeps the pipeline full, and it costs a few senior hours a week in the sample. That is the real price of a pipeline. Teams that skip it pay later, in hiring.
The cheapest guard there is. The junior commits to an answer before seeing the machine's, then compares the two.
WhoEach junior, one senior
BudgetThirty minutes per rep
InA live brief
OutTwo versions and two lines
Do
The junior makes a rough version first. Ten minutes, no AI. Rough is the point.
Then generate the machine's version and put the two side by side.
The junior writes two lines: what the machine saw that I missed, and what I saw that it missed.
A senior reads the two lines before the drafts. The comparison is the lesson.
Why it works
In the classroom experiment, the tutor that gave hints instead of answers kept students doing the thinking, and their results without AI held up [3]. Draft first puts the same guard into the workflow, so it works whichever tool the team uses.
Done when
Every protected rep runs draft first
The junior's two lines are kept in the casebook
Prompt
I am a junior learning to judge this kind of work. Here is the brief and my rough draft. Do not rewrite it. Ask me up to three questions that point at its weakest part, most important first. If I ask for the answer, give me a hint instead. Only after I have revised it, show me how you would have done it and explain the one choice that matters most.
When AI does the making, give juniors the choosing. And bring the senior's thinking into the open, because it is now the main model left to learn from.
If the machine makes twenty, the junior picks three and defends them. Choosing is a judgment rep. Accepting the first one is not.
WhoJuniors, with a senior
BudgetTen minutes per decision
InGenerated options
OutPicks with written reasons
Do
Every time AI generates options, a junior picks first, before any senior sees them.
The junior defends the pick in two sentences: why this one, and why the runner-up lost.
The senior picks too, separately. Then compare. Where they differ is the lesson.
Keep juniors actively supervising the output, never just passing it on. In the cockpit study, thinking skills held up best for pilots who stayed engaged while the automation flew [4].
The same review, twiceExample
Senior picks alone
AI makes twenty directions
The senior picks two in five minutes
The junior hears the result
Nobody learns why
Junior picks first
AI makes twenty directions
The junior picks three and writes why
The senior picks blind, then compares
One difference, explained in one minute
Done when
No generated set reaches a senior without a junior's pick
Juniors used to learn by watching seniors fix their work. There is less of their work to fix now, so bring the senior's thinking out where juniors can see it.
WhoEvery senior
BudgetOne hour a week each
InReal work in progress
OutShadowed or recorded sessions
Move
What the senior does
Modelling
Does a real task while a junior watches, thinking out loud
Coaching
Watches the junior do it and steps in only at the moments that matter
Scaffolding
Gives a starting frame or checklist, then takes it away
Articulation
Asks the junior to explain a choice before giving a view
Reflection
Puts the junior's version next to the senior's and asks what differs
Moves from cognitive apprenticeship [1], applied to a design and product team.
Do
Each senior runs one shadowing session a week: a real decision, thought out loud, with a junior watching.
Narrate the rejections. Why the second option lost teaches more than why the first one won.
Record the short ones. Ten minutes of a senior choosing between three directions is worth keeping.
Done when
Every junior shadows a senior at least once a week
Seniors narrate what they rejected as well as what they picked
Forty-five minutes a week where juniors bring their choices and their doubts, and leave with a verdict and a reason.
WhoJuniors, one rotating senior
BudgetForty-five minutes a week
InThis week's picks and drafts
OutVerdicts with reasons
Do
Each junior brings one thing they decided this week, with the options they rejected.
The senior gives a verdict and one reason. One reason, so it sticks.
Feedback is what turns work into practice[2]. If the crit is cancelled twice in a row, the pipeline is already narrowing.
Close with one tear-down: a strong piece of outside work, taken apart. What makes it work, in writing.
Done when
The crit has run every week this month
Every verdict has a reason written down
Prompt
Here is a piece of work I admire. Do not tell me why it is good yet. Ask me five questions that make me look closer: at its structure, at what was left out, and at the one choice that holds it together. After I answer all five, tell me briefly what I missed.
One team, one week. By Friday every lost rep has a place again, or a written reason why not.
Setup prompt
Help me map where judgment forms on my team, now that AI does much of the junior work.
1. Ask me what our juniors spent their weeks on a year ago, and what they do now. Turn my answers into pipeline/reps.csv with the columns from the rep map template. Mark every hour figure as an estimate.
2. For each task, ask me what doing it taught a junior. Fill in the rep, or mark it production only. Then ask where that rep happens now, and write "nowhere" if I cannot name a place.
3. For every rep marked nowhere, propose one deliberate replacement: draft first then compare, pick and defend in crit, shadow a senior, or a weekly tear-down. Give each a weekly slot and an owner.
4. Draft a calendar for the next four weeks with the junior crit, the shadowing sessions and one unassisted check.
5. Do not invent tasks, hours or people. Ask me, and show me every rep you propose so I can say what is wrong with it.
Day
Steps
What happens
D1
1.1, 1.2
List the ten tasks. Name the rep inside each. Count the nowheres.
D2
2.1, 2.2
Choose three protected reps. Start draft first on one live brief.
D3
3.1, 3.2
Juniors pick first on every generated set. Book weekly shadowing.
D4
4.1, 4.2
Run the first junior crit. Start the casebook with its verdicts.
D5
5.1, 5.2
Schedule the unassisted check. Write the first pipeline note.
The rep map, the draft first protocol and the quarterly unassisted check.
pipeline/reps.csv
task,hours_then_est,hours_now_est,who_now,rep_inside,where_now,replacement,owner
first_drafts,8,1,ai,reading what a brief really asks,nowhere,draft first then compare,design lead
ten_variations,6,0,ai,telling a direction from a variant,nowhere,pick and defend in crit,design lead
resizing_assets,5,0,ai,production only,,,
interview_synthesis,4,4,person,hearing the need behind what people say,junior,,research lead
spec_check,3,3,person,noticing what is wrong,junior,,PM
Draft first protocol
DRAFT FIRST, THEN COMPARE
For any protected rep. Thirty minutes.
1. Read the brief. Close every AI tool.
2. Make a rough version in ten minutes. Rough is fine.
3. Now generate the machine's version.
4. Put both side by side. Write two lines:
The machine saw ............ that I missed.
I saw ............ that the machine missed.
5. Send the two lines and both versions to a senior.
6. The senior reads the two lines first and replies
with one sentence: the difference that matters most.
7. Add the two lines and the reply to casebook.md.
Skip step 2 and the rep did not happen.
Quarterly unassisted check
UNASSISTED CHECK · once a quarter
Brief: one the team has already solved
Time: two hours, no AI tools
Judges: two seniors, scoring blind
Score each piece 1 to 5 on:
[ ] Reading of the brief did they see what it really asked?
[ ] Choice of direction would a senior have chosen it?
[ ] Finish is it done, or abandoned?
Then, per junior:
This quarter ___ Last quarter ___ Change ___
Assisted output this quarter: rising / flat / falling
Read it:
Unassisted rising the reps are working
Unassisted flat, assisted rising crutch pattern: protect more reps
Both flat the crit is not giving verdicts
Research · Playbook 11
Research
What this playbook rests on, how to measure whether it works for you, and what it cannot yet claim.
Every core claim in this playbook, with what supports it. Grades describe the kind of evidence, not how much we like the claim. Claims graded P come from practice and are labelled as such.
1A · Experiment
3B · Field data
2C · Framework
1P · Practitioner
Claim
Grade
What supports it
Source
Crafts are learned through modelling, coaching, fading support, articulation and reflection, with the expert's thinking made visible.
C · Framework
The cognitive apprenticeship model sets out these teaching methods and argues that in thinking work the expert's process must be deliberately made visible to the learner [1]. A framework, not a test of creative teams.
1
Expertise grows through effortful practice built around feedback, which ordinary work provides less of.
C · Framework
The deliberate practice framework distinguishes work, play and deliberate practice, and ties improvement to effortful activity designed to improve performance with feedback [2]. How much of expert skill practice explains is still debated.
2
Help that gives learners answers can raise assisted performance and lower unassisted performance. Help built to give hints can prevent the loss.
A · Experiment
In a field experiment with nearly a thousand high school students, a general AI tutor raised practice grades by 48% and lowered later unassisted exam grades by 17% against a control group [3]. A tutor that gave hints instead of answers raised practice grades by 127% with no significant exam loss [3]. One subject, one country, a few sessions.
3
Automation preserves visible hand skills more than the cognitive skills of knowing where you are and noticing what is wrong.
B · Field data
In a simulator study of 16 airline pilots, with no control group, instrument scanning and manual control were largely retained, while tracking position, planning next steps and recognising instrument failures showed more frequent and significant problems. Retention of cognitive skills appeared to depend on active engagement in supervising the automation [4]. Small sample, aviation only.
4
People who trust AI more report less critical thinking. People confident in their own skill report more.
B · Field data
A survey of 319 knowledge workers sharing 936 examples of AI use found confidence in AI associated with less critical thinking and self-confidence associated with more [5]. Self-reported and correlational.
5
AI assistance can lift less experienced workers the most, and can help them learn.
B · Field data
Across 5,172 customer support agents, AI assistance raised issues resolved per hour by 15% on average [6]. Less experienced and lower-skilled workers improved most in speed and quality, the most experienced saw small quality declines, and the authors report the assistant facilitated worker learning [6]. One firm, one kind of work.
6
The tasks AI absorbs first were the reps that built junior judgment, and without designed replacement reps a team runs out of future seniors.
P · Practitioner
MainArc's thesis. Consistent with [1], [2], [3] and [4], and complicated by [6]. The pipeline effect itself is unmeasured. The rep weights and pipeline model in the apprenticeship map are working estimates for illustration.
Practice
References
[1]Collins, A., Brown, J. S., & Newman, S. E. (1989). Cognitive apprenticeship: Teaching the crafts of reading, writing, and mathematics. In L. B. Resnick (Ed.), Knowing, learning, and instruction: Essays in honor of Robert Glaser (pp. 453–494). Lawrence Erlbaum Associates.
[2]Ericsson, K. A., Krampe, R. T., & Tesch-Römer, C. (1993). The role of deliberate practice in the acquisition of expert performance. Psychological Review, 100(3), 363–406. doi.org/10.1037/0033-295X.100.3.363
[3]Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakcı, Ö., & Mariman, R. (2025). Generative AI without guardrails can harm learning: Evidence from high school mathematics. Proceedings of the National Academy of Sciences, 122(26), e2422633122. doi.org/10.1073/pnas.2422633122
[4]Casner, S. M., Geven, R. W., Recker, M. P., & Schooler, J. W. (2014). The retention of manual flying skills in the automated cockpit. Human Factors, 56(8), 1506–1516. doi.org/10.1177/0018720814535628
[5]Lee, H.-P., Sarkar, A., Tankelevitch, L., Drosos, I., Rintel, S., Banks, R., & Wilson, N. (2025). The impact of generative AI on critical thinking: Self-reported reductions in cognitive effort and confidence effects from a survey of knowledge workers. Proceedings of CHI '25. ACM. doi.org/10.1145/3706598.3713778
[6]Brynjolfsson, E., Li, D., & Raymond, L. (2025). Generative AI at work. The Quarterly Journal of Economics, 140(2), 889–942. doi.org/10.1093/qje/qjae044
Grades
A controlled experiment. B large-scale field data. C an established peer-reviewed model or guideline, not a direct test of the claim. P practitioner claim from MainArc's work, untested.
A team that protects its reps should see unassisted skill rise while assisted output rises too. Measure both, per junior, every quarter.
Instruments
01
Unassisted check
Once a quarter, juniors solve a past brief with no AI in two hours. Two seniors score it blind, 1 to 5, on reading of the brief, choice of direction and finish.
02
Rep map
The share of judgment reps in reps.csv with a named place, and the count marked nowhere. Rerun after every new piece of automation.
03
Rep log
Draft first comparisons, crit verdicts and shadowing sessions actually held per junior per week, against the plan.
04
Readiness
A private quarterly judgment by leads: for each kind of work, who could lead it two years from now.
Study design
Element
Setting
People
Every junior and early mid-level person on the team
Baseline
One unassisted check before any reps change
Comparison
If there are two teams, one adds deliberate reps first and the other follows a quarter later
Analysis
Change in unassisted score per person against change in assisted output. Rep log completion against plan.
Cadence
Quarterly checks, with a full review after four quarters
Reading the results
Unassisted scores rising with assisted output means the reps are working.
Assisted output rising while unassisted scores stay flat is the crutch pattern in [3]. Protect more reps before automating anything else.
If the rep log shows crits and comparisons being skipped, fix the calendar before judging the method.
Running signals
Signal
What to watch for
Reps marked nowhere
Should be zero after each quarterly map
Crits cancelled
Two in a row is the first sign the pipeline is narrowing
Time to first solo direction
How long until a new junior chooses a direction a senior keeps. Should fall over time
Note
Thresholds here are starting points to tune, not findings. A small panel shows direction and disagreement. It does not prove significance, and nothing on this page should be reported as if it did.
How this playbook was built, where it is weak, and how to cite it.
How this playbook was built
Built from research on cognitive apprenticeship and deliberate practice, experimental and field evidence on how AI help affects learning and skill, human factors research on skill retention under automation, and MainArc's practice. It is a practitioner method. Claims are graded on the evidence page.
Limitations
The pipeline thesis is a practitioner claim. No study yet follows creative or product juniors over years with and without AI.
The learning evidence [3] comes from high school maths and the skill evidence [4] from aviation. Creative judgment may degrade or recover differently.
The survey [5] is self-reported and correlational, and the field study [6] shows AI can help novices learn. The risk depends on how work is designed, which is the point and also the uncertainty.
The deliberate practice framework [2] is debated, and the tasks, hours, weights and pipeline model in the apprenticeship map are illustrative working estimates.
Open questions
Which creative judgments decay fastest when the production around them is automated?
Can a junior who learned mainly by choosing among machine outputs develop the same eye as one who learned by making?
How many senior hours a week does it take to keep a team's pipeline healthy?
Cite this playbook
APA
MainArc Research. (n.d.). Keep the junior pipeline alive (Playbook No. 11, Version 1.0). MainArc. https://mainarc.xyz
BibTeX
@misc{mainarc_pipeline,
author = {{MainArc Research}},
title = {Keep the junior pipeline alive},
howpublished = {MainArc Playbook No. 11, Version 1.0},
url = {https://mainarc.xyz}
}
Corrections
Found an error, a stronger source, or a result from running the benchmark? Send it through mainarc.xyz. Corrections are made in the next version.
About this playbook
This playbook is published by MainArc, a creative intelligence lab for leadership and product teams.
We do three things. We research how AI is changing creative work through long-form reports, design teardowns, and operator essays. We design the workflows, prompt systems, review loops, and decision frameworks that teams actually run on. And we build the work the strategy points to: brand systems, interfaces, product prototypes, internal tools, and launch concepts.
One operator. One AI agent. No layers in between. Every engagement runs under strict NDA. No case studies, no client logos, no shared work.
If this research resonates with a problem your team is facing, start with an audit. One week. A clear map of where AI fits, where it doesn't, and what to build next.
This is independent research. Not financial advice. Not career advice. Not the final word on anything. It's one person's perspective shaped by building, shipping, and paying attention. Do your own thinking. Challenge the ideas. Disagree with specifics. Build on what resonates. Discard what doesn't.
House rules
What doesn't belong here: hate, trolling, bad-faith takedowns, or drive-by takes from people who've never built anything. If you disagree, show your work. Builder's critique. Everyone else just comments.
A MainArc playbook on the team's definition of done. In AI products the eval is the spec: a short list of checks a stranger could grade, a set of test inputs, graders you have checked, and a habit of changing the list when grading teaches you something. Everything else in a brief is a wish.
PlaybookNo. 12Version1.0Read27 minSources6 citedByMainArc ResearchEvidence and method →
Thesis
In AI products the eval is the spec: a brief without checks a stranger could grade is a wish, and the generator settles every open point with the most common answer.
Do this Monday
Take your next brief and write five checks a stranger could grade without asking you, including one about distinctiveness, before anyone generates anything.
A brief to eval converter. It is live. Load a sample brief or paste your own. It splits the brief into lines, shows which lines three strangers would grade the same way, flags the words nobody can grade, and lists every outcome the generator will decide for you. Load the landing hero and watch the score sit at zero. Then load the gradeable brief and read the eval it produces.
Brief to eval converter · sample briefs0 / 100
Draft eval
Pinned down by you
The generator will decide
Rule-based. It finds the form of a check, not whether it is the right check. Strangers are simulated. Sample briefs. Score weights are a working estimate
5Stages
10Steps
5Checks to start
4Templates
What this is
Most AI features ship against a line like "helpful, accurate and on brand". Most generated creative work ships against "modern, clean and engaging". Both look like specs. Neither is one, because nobody outside the room could use them to say whether a given output passes.
A generator does not wait for a better brief. Every point you leave open still gets settled, with the most common answer. So the brief that counts is the one you can grade: a few checks, a set of test inputs, a way to run both, and a record of how the checks change. This playbook calls that the eval, and treats it as the spec.
Production is free. Deciding what counts as good, in a form a stranger could test, is the judgment. It is also the only way a team knows when the work is done.
Principles
01
If a stranger cannot grade it, it is a wish
A check has one answer anyone can find. A wish has as many answers as readers, and the generator picks the most common.
02
Five checks before anything is made
Write them before the first output exists. After that, the most polished option quietly becomes the standard.
03
Always check distinctiveness
Without it, the work can pass every check and still be the default anyone else would get.
04
Grade a set, never one output
One good result is one draw. A golden set of real inputs shows what the work does on average and at the edges.
05
Check the graders
An automated grader is a claim about your taste. Trust it only on the checks where it agrees with you.
06
Version the eval
Grading changes what you want. Write each change down, with a version number, where the whole team can see it.
How to use this playbook
01
Break the converter first
Paste the last brief you actually sent. Count the lines strangers would split on, and read what the generator will decide for you.
02
Read the three essays
Why a brief is a wish, why criteria appear while you grade, and what test-first research honestly shows.
03
Write one eval this week
One live brief, five stages, five days. The plan is in Run it.
04
Copy the files
The eval file, the golden set, a grader agreement script and the drift log are on the Templates page.
Playbook 12 in the set. Playbook 07 directs the model, one job at a time. This one gives the whole team a definition of done that anyone can run, and keeps it honest as the work changes.
If two strangers could read your brief and disagree about whether the work passes, the brief has not decided anything. The generator decides instead.
Every open point gets the common answer
A generator settles everything you leave open, and it settles it toward what is common. The effect shows up across people. In a large writing experiment, stories written with AI ideas were rated better one by one and were more alike as a set [1]. A brief made of words like modern and engaging leaves nearly everything open, so it gets the most common version of modern and engaging.
Do your best has no finish line
Decades of goal-setting research found that specific, hard goals led to higher performance than telling people to do their best. The explanation offered is that do-your-best goals have no external referent. Each person defines them privately, so a wide range of results can count as success [2]. That research concerns people working toward their own goals, and the effect is smaller on complex tasks. The parallel for a team is close enough to act on: a brief nobody can grade lets every reviewer define success privately, round after round.
Fig 0.1Wish or checkPress for the next lineline 1 · each sets a knob
Tile: the same output, one copy per stranger, lanes A, B, CKnob and blade: a wish, each stranger sets their own gateSetting bar on fixed stops: a check, one gate height for allPanel rows 1 to 4: engaging · premium, but friendly · subject under 45 characters · no savings figureBlue lamp passes, red stops, dark had to askIllustration. Blue lamps pass, red lamps fail. A check has one answer any stranger can find. A wish has as many answers as readers, and the generator picks the most common one. Here a check brings its own setting bar, so every gate is set the same; a wish leaves each stranger to set their own.
The stranger test
One test separates a check from a wish. Hand the line and one output to someone outside the team. If they can say pass or fail without asking you a question, it is a check. If they need to ask what you meant, it is a wish, and the answer you would have given now belongs to the generator.
The same brief, twiceExample
Wish
Modern and clean
Engaging
Friendly but professional
Short
On brand
Checks
No more than three elements above the fold
The one link is the first-budget screen
Reads like a calm accountant talking to a friend, with a pass and a fail example attached
Body under 90 words
Fails if a competitor could send it unchanged
MainArc Research
Production is free. Judgment isn't. In an AI product the judgment lives in one place: the list of checks the work has to pass. Anything missing from that list is settled by the most common answer.
Nobody writes the whole eval up front. The first checks come from the brief. The rest come from grading real outputs, so the eval has to be built to change.
One good output feels like proof
People new to steering models rarely test on more than a handful of cases. In a study of ten people without an AI background designing a chatbot with prompts, none used the tool's built-in interface for systematic testing, and most declared success after a single good instance before moving on [3]. One good output is one draw from a range you have not looked at. A golden set is how you look at the range.
Criteria drift
Grading changes the grader. In a study of practitioners building automated graders for model outputs, participants needed criteria to grade outputs, yet grading outputs helped them define those criteria. The authors call this criteria drift, and argue that some criteria depend on the specific outputs you see [4]. The study was small and qualitative, with nine practitioners. Its lesson matches most teams' experience: the second review always finds something the brief forgot.
Fig 0.2The eval loopPress for the next stepeval v1.0 · five checks
Golden set: eight sample inputs, the same in every runCard: the eval. One hole per check, five in v1.0Press makes the outputs; the dial counts passes to the ship line, 7 of 8Lever and spike: a late objection punches the missing check and logs itPins: blue passed every check, red failed oneCriteria change while people grade, as found in [4]. A versioned eval keeps that change where everyone can see it, instead of in one reviewer's head. A sample run: the card is the eval, and every version runs on the same golden set.
So the eval is a living file. Write the first five checks before anything is generated. Then let grading add checks, sharpen them and retire them, and write every change down.
Part
What it is
Lives in
Checks
Lines a stranger could grade pass or fail without asking you
eval.md
Anchors
One pass example and one fail example for every check that needs judgment
eval.md
Golden set
Real inputs the work must handle, typical, hard and past failures
golden.csv
Graders
A script, a person or a model-graded prompt for each check, validated against your grades
graders/
Ship line
How much of the golden set must pass before the work is done
eval.md
Drift log
Every late rejection and every false fail, with the change it caused
Software teams have studied test-first work for a long time. The evidence says small, frequent checks matter more than strict order. For generated work, order matters for a different reason.
What test-first research found
A meta-analysis of 27 studies found that test-driven development had a small positive effect on quality and little to no effect on productivity overall. In industrial studies, both the quality gain and the drop in productivity were much larger than in academic ones [5]. A later study of 39 professionals found that the order in which tests and code were written had no important influence. Working in small, steady cycles was what went with better quality and productivity [6].
The evidence is mixed, and it is worth taking at face value. Writing tests first is no magic. Checking small and often is the part that holds up, and it costs time.
Why first still matters here
Code often has a correct answer you can state in advance. Generated product and creative work usually does not, and it has a second problem: once options exist, the most polished one sets the bar. Five checks written before anything is generated mean the first output meets a standard that existed before it did. That is MainArc's practice, not a tested finding. The benchmark in the research section is designed to test it.
Fig 0.3Knowing when to stopPress for the next roundsame brief · two reviews
Back rail: review by feel, a needle with no scaleFront rail: review against the eval, a lamp per check, blue passes, red failsTimer and bell: stopped because time ran outLatch: stopped because it passed, at round 3Hole 6 and lamp 6: the late objection, check 6 for next timeSample review histories, an illustration and not a measurement. An eval tells a team when to stop, and turns the next objection into a check instead of another round. Notes by feel, round by round: premium, less busy, warmer, bolder, safer, back to round 2, deadline.
An eval also ends the work. Without one, review runs until the deadline or until people tire, and in MainArc's experience late rounds tend to trade distinct choices for safe ones. With one, the team agrees in advance what done looks like, and an objection after that point becomes a check for next time.
The shift
Most teams treat evals as quality assurance, run at the end by someone else. Treat the eval as the brief, written first by the person who owns the outcome.
Take the next brief and write five checks a stranger could grade without asking you, including one about distinctiveness, before anyone generates anything.
WhoWhoever owns the outcome
BudgetOne hour
InThe brief as written
Outeval.md with five checks
Do
Underline every adjective and every word like "should feel". Each one is a decision you have not made yet.
Rewrite each as a check using one of four forms: a number (under 90 words), a required element (names the forecast), a banned element (no savings figures), or a named reference with a pass and a fail example attached.
Keep five: one on size or format, one on what must be included, one on what must never appear, one on which claims are allowed, and one on distinctiveness. A swap test works: it fails if a competitor could ship it unchanged.
Run the stranger test. Give the five checks and one old output to someone outside the team. Every question they ask you marks a check that is still a wish.
Done when
eval.md has five checks, one of them on distinctiveness
A stranger graded one output against them without asking a question
Nothing has been generated yet
Prompt
Here is a brief. List every word or phrase in it that two strangers could disagree about. For each, propose a check someone outside our team could grade pass or fail without asking me, using a number, a required element, a banned element or a named reference. Then propose five checks: one on size or format, one on what must be included, one on what must never appear, one on which claims are allowed, and one on distinctiveness. Do not generate the work itself. Save the result as eval/eval.md.
A fixed set of real inputs that every version is run against. It replaces "this one looked good" with "this is how it does across the range".
WhoOutcome owner, with whoever knows the users
BudgetHalf a day
InReal inputs, past failures
Outgolden.csv with 12 to 20 rows
Include
How many
Why
Typical inputs
6 to 10
The everyday case most outputs will be
Hard inputs
3 to 5
Long, messy, ambiguous or missing data
Past failures
2 to 3
Anything that once embarrassed you, so it never comes back
Edge cases
2 to 3
Empty fields, another language, extreme values
One to refuse
1
An input where the right output is a polite no
Do
Pull inputs from real use: support tickets, search logs, past briefs. Invented inputs test what you imagined.
Write one line per row on why it is in the set. A row nobody can explain gets removed.
Mark three to five rows as hero inputs: the cases the work is most often judged on. The ship line treats them more strictly.
Freeze the set for this version. Add rows between versions, never during a review.
The counts are working estimates. Twelve rows show direction. Past twenty, most teams stop reading every output, and an unread golden set is decoration.
Done when
golden.csv has 12 to 20 real inputs, each with a reason
A check that needs judgment needs two examples: one that passes and one that fails. Then two strangers should agree on it.
WhoOutcome owner, two people outside the team
BudgetTwo hours
Ineval.md, ten old outputs
OutAnchors for every judgment check
Do
For every check that is not a number or a pattern, attach one real pass example and one real fail example. Voice and distinctiveness always need anchors.
Ask two people outside the team to grade the same ten old outputs against every check, separately.
Count where they disagree. A check they split on more than twice in ten is still a wish. Rewrite it or add a sharper anchor. That threshold is a working estimate.
Repeat for any rewritten check until the strangers agree.
Done when
Every judgment check has a pass and a fail example
Two strangers agree on every check on at least eight of ten outputs
Some checks a script can grade in milliseconds. Some need a person with taste. Most teams put a model on all of them and never find out which ones it gets wrong.
WhoEngineer, outcome owner
BudgetHalf a day
Ineval.md with anchors
OutA grader named for every check
Check form
Grader
Example
A number
Script
Subject under 45 characters
Required or banned element
Script or pattern, with a person on the misses
No exclamation marks. Names the forecast
Allowed claims
A person with the source, or a model-graded prompt given the source
Every number appears in the product data
Voice
A person with anchors. A model-graded prompt only once validated
Reads like a calm accountant talking to a friend
Distinctiveness
People, blind, with a competitor version in the set
Fails the swap test
Do
Name one grader per check in eval.md. Prefer the dumbest grader that works. A script never drifts.
Make every grader return pass or fail plus one line of reason. Scores out of ten hide what failed.
Keep distinctiveness with people. It is the check most likely to reward the default if a machine grades it.
An automated grader is a claim about your judgment. Test the claim before you let it speak for you.
WhoOutcome owner, engineer
BudgetTwo hours
In20 outputs, your own grades
Outgrades.csv, a trusted list of graders
Do
Grade 20 outputs by hand against every check. Do it before you look at any automated grade.
Run each automated grader on the same 20. Count agreement per check with the script on the Templates page.
Read every disagreement. Some are grader errors. Some show your own criteria shifting as you grade, which is the drift described in [4]. Write those into the drift log.
Trust a grader only on the checks where it agrees with you. Revalidate whenever the generator, the grader or the check changes.
Done when
Every automated grader has an agreement figure per check
Checks with low agreement are graded by a person until fixed
Prompt
Here are 20 outputs and my pass or fail grade for check C3, with my one-line reason for each. Write a grader for C3 that returns pass or fail and one line of reason. Run it on the same 20 outputs and show me every case where it disagrees with me. Do not tune it to match me until I have read the disagreements, because some of them may be my mistakes.
Run the eval before the meeting. The review starts from what failed, and every note is either a check or a new one.
WhoOutcome owner, reviewers
BudgetForty-five minutes per review
InOutputs for the golden set, grades
OutNotes tied to check ids
Do
Send the grade sheet before the review: which checks failed, on which inputs, with the reasons.
Open with failures on hero inputs. Passing work gets looked at last.
Every note cites a check id. A note that cites none is one of two things: a missing check, which goes in the drift log, or a taste call, which the outcome owner makes and writes down.
Nobody rewrites a check during the review. Changes wait for the next version.
The same note, twiceExample
By feel
Feels a bit off
Could be warmer
Not sure about the ending
Against the eval
C4 fails on G07: promises "save more"
C2 passes everywhere. Missing check: no greeting longer than three words
Taste call by owner: keep the plain ending
Done when
Every note names a check, a missing check or an owner's taste call
Decide before the first review how much of the golden set has to pass. When it passes, the work is done.
WhoOutcome owner
BudgetThirty minutes
Ineval.md, golden.csv
OutA ship line in eval.md
Do
Write the line as a test. A sample: every check passes on at least 18 of 20 golden inputs, and distinctiveness passes on every hero input. Those numbers are a working estimate. Set yours from the stakes.
Write it before the first review, and never lower it during one.
When the line is met, ship. An objection after that point becomes a check in the next version, unless it is a defect you would be ashamed to ship.
Count rounds to ship for every brief. It is the clearest sign the eval is doing its job.
Done when
eval.md has a ship line written as a test
The last piece of work shipped when it met the line, and not after
Two events show the eval is out of date: good work that fails a check, and work that passes every check and still gets rejected. Log both.
WhoOutcome owner
BudgetFifteen minutes a week
InReview notes, grader disagreements
Outdrift.md
Do
Log every late rejection: work that met the ship line and was rejected anyway. Each one is a missing check.
Log every false fail: work everyone liked that failed a check. Each one is a check that is too blunt.
For each entry, write the check you would add or change, in a form a stranger could grade.
Expect the log to be busy early. Criteria take shape while people grade [4]. A log that stays busy after several versions means the checks are too vague.
Done when
Every late rejection and false fail this week is in drift.md
Each entry proposes a check change
Prompt
Read eval/drift.md and eval/eval.md. Group the late rejections and false fails by cause. For each group, propose one change to eval.md: a new check, a sharper check, a new anchor, or a check to retire. Write every proposed check so someone outside our team could grade it without asking. Show the evidence for each change, and do not edit eval.md until I agree.
One live brief, five days. By Friday the brief is an eval: checks, a golden set, validated graders, a ship line and a drift log.
Setup prompt
Help me turn one live brief into an eval before anything is generated.
1. Ask me for the brief and who owns the outcome. List every word in it that two strangers could disagree about. Propose five checks a stranger could grade pass or fail: size or format, must include, must never, allowed claims, distinctiveness. Save as eval/eval.md.
2. Walk me through nine outcomes (audience, next action, size, must include, must never, voice, allowed claims, distinctiveness, ship line). For each, I either pin it with a check or put it on the open list.
3. Help me build eval/golden.csv: 12 to 20 real inputs from our logs and past work, with a reason for each row. Mark the hero inputs. Do not invent inputs.
4. For each check, propose the simplest grader: script, person or model-graded prompt. I will grade 20 outputs by hand first. Compare every automated grader with my grades and list the disagreements.
5. Draft a ship line as a test. Then start eval/drift.md. Do not generate the work itself until I approve eval.md.
Day
Steps
What happens
D1
1.1, 1.2
Five checks written. Stranger test run. Open list agreed.
D2
2.1, 2.2
Golden set built from real inputs. Anchors attached. Two strangers grade ten old outputs.
D3
3.1, 3.2
A grader for every check. Twenty outputs graded by hand. Agreement counted.
D4
4.2, 4.1
Ship line set. First generation and first review against the eval.
D5
5.1, 5.2
Drift logged. eval.md published at version 1.0, with the next review booked.
The eval file, the golden set, a script that counts grader agreement and the drift log. Copy them, then make them yours.
eval/eval.md
# Eval: [what is being made]
Version: 1.0
Owner: [one person]
Exists so that: [who] will [do what]
## Checks
C1 [size] Body under 90 words. grader: script
C2 [include] Names the forecast of what is left after bills. grader: script, person on misses
C3 [never] No exclamation marks. No savings figures. grader: script
C4 [claims] Every number appears in the product data. grader: person with source
C5 [distinct] Fails if a competitor could send it unchanged. grader: 2 people, blind
pass: [example] fail: [example]
## Ship line
Every check passes on at least 18 of 20 golden inputs.
C5 passes on every hero input.
## Open: the generator decides these, and we accept its default
- [outcome]
## Changelog
1.0 First version, written before generation.
eval/golden.csv
id,input,kind,hero,why_it_is_here
G01,[real input],typical,yes,most common case
G02,[real input],typical,no,second most common case
G07,[real input],hard,yes,missing bank data
G11,[real input],past failure,no,once promised a savings figure
G14,[real input],edge,no,account in another language
G15,[real input],refuse,no,asks for advice we must not give
eval/agree.py
import csv
from collections import defaultdict
# grades.csv columns: item,check,grader,verdict (verdict is pass or fail)
# grader "human" is you. Every other grader is compared against it.
rows = list(csv.DictReader(open("eval/grades.csv")))
human = {(r["item"], r["check"]): r["verdict"] for r in rows if r["grader"] == "human"}
agree, total, misses = defaultdict(int), defaultdict(int), defaultdict(list)
for r in rows:
g = r["grader"]
if g == "human": continue
k = (r["item"], r["check"])
if k not in human: continue
total[(g, r["check"])] += 1
if r["verdict"] == human[k]: agree[(g, r["check"])] += 1
else: misses[(g, r["check"])].append(r["item"])
for key in sorted(total):
g, c = key; a, n = agree[key], total[key]
print(f"{g:12} {c:4} agrees {a}/{n} disagrees on: {', '.join(misses[key]) or 'none'}")
eval/drift.md
## [when] · eval v[version]
Type: late rejection | false fail
Input: [golden id or live case]
What happened: [one line]
Check involved: [id, or "none, missing"]
Proposed change: [new or sharper check, written so a stranger could grade it]
Decision: [accepted in v1.1 | rejected, because ...]
Research · Playbook 12
Research
What this playbook rests on, how to measure whether it works for you, and what it cannot yet claim.
Every core claim in this playbook, with what supports it. Grades describe the kind of evidence, not how much we like the claim. Claims graded P come from practice and are labelled as such.
2A · Experiment
3B · Field data
1C · Framework
1P · Practitioner
Claim
Grade
What supports it
Source
AI help makes creative work more alike across people.
A · Experiment
In a large writing experiment, stories written with AI ideas were rated better individually and were more similar to each other as a set [1]. That vague briefs in particular drive this convergence is MainArc's extension, untested.
1
A goal nobody can grade lets everyone define success privately.
C · Framework
Across decades of experiments, specific difficult goals led to higher performance than do-your-best goals, which have no external referent and allow a wide range of acceptable performance [2]. The research concerns people pursuing their own goals, with smaller effects on complex tasks. Applying it to briefs and reviews is MainArc's extension.
2
People steering models tend to judge from a single good output instead of testing systematically.
B · Field data
In a qualitative study of ten people without an AI background designing a chatbot with prompts, none used the tool's systematic testing interface and most declared success after a single instance [3]. Observational and small, so graded B.
3
Evaluation criteria cannot all be written before seeing outputs. Grading changes them.
B · Field data
In a qualitative study of nine practitioners aligning automated graders, people needed criteria to grade outputs, and grading outputs helped them define criteria. The authors name this criteria drift [4]. Small and qualitative, so graded B.
4
Writing tests first gives a small quality gain, with mixed effects on speed.
A · Experiment
A meta-analysis of 27 studies found a small positive effect of test-driven development on quality and little to no effect on productivity overall, with both the quality gain and the productivity drop much larger in industrial studies [5]. Software only.
5
Small, steady cycles of checking matter more than strict test-first order.
B · Field data
In data from 39 professionals, granularity and uniformity of the development cycle were associated with better quality and productivity, while the order of tests and code had no important influence [6]. Software only, and observational within an experiment, so graded B.
6
In AI products the eval is the spec. Teams that write five checks a stranger could grade, including one on distinctiveness, before generating ship work with a point of view and know when to stop.
P · Practitioner
MainArc's thesis. Consistent with [2], [3] and [4], and in tension with [6] on whether order matters. Untested as a whole. The benchmark on the next page is designed to test it.
Practice
References
[1]Doshi, A. R., & Hauser, O. P. (2024). Generative AI enhances individual creativity but reduces the collective diversity of novel content. Science Advances, 10(28). doi.org/10.1126/sciadv.adn5290
[2]Locke, E. A., & Latham, G. P. (2002). Building a practically useful theory of goal setting and task motivation: A 35-year odyssey. American Psychologist, 57(9), 705–717. doi.org/10.1037/0003-066X.57.9.705
[3]Zamfirescu-Pereira, J. D., Wong, R. Y., Hartmann, B., & Yang, Q. (2023). Why Johnny can't prompt: How non-AI experts try (and fail) to design LLM prompts. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI '23), 1–21. ACM. doi.org/10.1145/3544548.3581388
[4]Shankar, S., Zamfirescu-Pereira, J. D., Hartmann, B., Parameswaran, A. G., & Arawjo, I. (2024). Who validates the validators? Aligning LLM-assisted evaluation of LLM outputs with human preferences. Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (UIST '24), 1–14. ACM. doi.org/10.1145/3654777.3676450
[5]Rafique, Y., & Mišić, V. B. (2013). The effects of test-driven development on external quality and productivity: A meta-analysis. IEEE Transactions on Software Engineering, 39(6), 835–856. doi.org/10.1109/TSE.2012.28
[6]Fucci, D., Erdogmus, H., Turhan, B., Oivo, M., & Juristo, N. (2017). A dissection of the test-driven development process: Does it really matter to test-first or to test-last? IEEE Transactions on Software Engineering, 43(7), 597–614. doi.org/10.1109/TSE.2016.2616877
Grades
A controlled experiment. B large-scale field data. C an established peer-reviewed model or guideline, not a direct test of the claim. P practitioner claim from MainArc's work, untested.
Does writing the eval first change what your team ships? Run live briefs both ways and measure agreement, rounds to ship, late rejections and distinctiveness.
Instruments
01
Stranger agreement
Two people outside the team grade the same outputs against every check, separately. The share of items they agree on, per check. Low agreement means the check is still a wish.
02
Grader agreement
Every automated grader is compared with the owner's hand grades on a sample, per check. A grader is trusted only where it agrees, as recommended in [4].
03
Rounds and late rejections
Review rounds until the ship line is met, and the number of rejections after it was met. Both should fall as the eval matures.
04
Blind distinctiveness
A panel compares outputs from eval-first and eval-later briefs, unlabelled, and picks the one a competitor could not have shipped.
Study design
Element
Setting
Briefs
10 live briefs over six weeks
Conditions
Alternate briefs: eval written before generation, or after the first review
Panel
2 strangers for agreement, 5 reviewers for blind distinctiveness
Analysis
Agreement per check. Rounds to ship and late rejections per condition. Distinctiveness win rate per condition.
Cadence
Once over six weeks, then every quarter
Reading the results
If strangers split on a check, rewrite it or add anchors before trusting any score from it.
If eval-first briefs ship in fewer rounds and win on distinctiveness, keep writing the eval first.
If both conditions perform the same, order is not what matters for your team, which is consistent with [6]. Keep the checks and the golden set, and drop the ceremony around order.
Running signals
Signal
What to watch for
Late rejections
Work that met the ship line and was rejected anyway. Each one is a missing check, and the count should fall over versions
Checks that always pass
Either the floor or broken. Move them to an automated gate or retire them
Grader drift
Falling agreement between a grader and human grades after any change to the generator, the grader or the check
Note
Thresholds here are starting points to tune, not findings. A small panel shows direction and disagreement. It does not prove significance, and nothing on this page should be reported as if it did.
How this playbook was built, where it is weak, and how to cite it.
How this playbook was built
Built from research on how people define and validate evaluation criteria for model outputs, how non-experts test prompts, goal setting, test-driven development and AI homogenisation, combined with MainArc's practice. It is a practitioner method, not a study. Claims are graded on the evidence page by what supports them.
Limitations
The central claim, that writing the eval first produces more distinctive work and shorter reviews, is MainArc's thesis. No study here tests it directly.
The prompting [3] and criteria drift [4] studies are qualitative, with ten and nine participants.
The test-first evidence [5, 6] comes from software, where checks are easier to automate than in creative work, and it is mixed: one study found order mattered less than steady, small cycles [6].
The thresholds in the steps (twelve to twenty golden inputs, eight of ten stranger agreement, the sample ship line) are working estimates, not validated cut-offs.
The converter on the introduction page is a rule-based pattern matcher with sample briefs and simulated strangers. It cannot judge whether a check is the right one.
Open questions
How many checks can a brief carry before it leaves no room for a point of view?
Can strangers grade distinctiveness reliably with anchors alone, or does it always need the team's own judges?
How fast do evals drift, and when should a team start a new eval instead of patching the old one?
Cite this playbook
APA
MainArc Research. (n.d.). Make the spec the eval (Playbook No. 12, Version 1.0). MainArc. https://mainarc.xyz
BibTeX
@misc{mainarc_evalspec,
author = {{MainArc Research}},
title = {Make the spec the eval},
howpublished = {MainArc Playbook No. 12, Version 1.0},
url = {https://mainarc.xyz}
}
Corrections
Found an error, a stronger source, or a result from running the benchmark? Send it through mainarc.xyz. Corrections are made in the next version.
About this playbook
This playbook is published by MainArc, a creative intelligence lab for leadership and product teams.
We do three things. We research how AI is changing creative work through long-form reports, design teardowns, and operator essays. We design the workflows, prompt systems, review loops, and decision frameworks that teams actually run on. And we build the work the strategy points to: brand systems, interfaces, product prototypes, internal tools, and launch concepts.
One operator. One AI agent. No layers in between. Every engagement runs under strict NDA. No case studies, no client logos, no shared work.
If this research resonates with a problem your team is facing, start with an audit. One week. A clear map of where AI fits, where it doesn't, and what to build next.
This is independent research. Not financial advice. Not career advice. Not the final word on anything. It's one person's perspective shaped by building, shipping, and paying attention. Do your own thinking. Challenge the ideas. Disagree with specifics. Build on what resonates. Discard what doesn't.
House rules
What doesn't belong here: hate, trolling, bad-faith takedowns, or drive-by takes from people who've never built anything. If you disagree, show your work. Builder's critique. Everyone else just comments.
A MainArc playbook for teams that let an automated judge grade their work. Grading is now nearly free. Before a machine decides what is good, find out where it agrees with your people, where it leans, and where it cannot help, because your people do not agree either.
PlaybookNo. 13Version1.0Read26 minSources6 citedByMainArc ResearchEvidence and method →
Thesis
A machine judge can only learn your floor: it matches people where they agree and flattens the splits. Let it scale the standard. Keep people on the edge.
Do this Monday
Take 20 items your team already judged, have your automated judge score them, and list every item where they disagree. Read those first.
A judge auditor. It is live. Ten sample headlines, each scored by a five-person panel and by a machine judge. Grey dots are people. The square is the judge. On the lines the panel agreed on, the judge sits inside the cluster. On the lines they split on, it sits in the gap where nobody is. Then switch on a bias and watch which lines move. Switch to One number to see what most audits report.
Judge auditor · sample panel
Switch on a bias
One personPanel averageJudge, coloured by statusJudge before the bias
Sample data and a simple model of each bias. Split means spread over 1.0. Match means within one point
4Stages
9Steps
3Biases to test
20Items to start
What this is
Teams now grade work with machines. One system drafts a hundred options, another scores them, and the top three reach a person. It is fast, it never gets tired, and it looks fair, because the same judge grades everything.
A machine judge is a model, and models are tuned toward what most people prefer. So a judge is very good at one thing: the standard most people share. Where your people agree, it agrees with them. Where informed people split, it returns a middle score that nobody gave, or it picks a side for reasons unrelated to the work: which option came first, which was longer, which sounded like itself.
Production is free, and now grading is nearly free too. Judgment is deciding which grades to trust. This playbook audits a judge against your own people, hands it the floor, and keeps people on the edge.
Principles
01
Audit before you automate
A judge is a measuring instrument. Check it against people before it grades anything that ships.
02
Never trust one number
Agreement on items your people agree on and on items they split on are two different numbers. Report both.
03
Give it the floor
Where your people agree and the judge holds up, let it grade at scale, with a spot-check.
04
Keep people on the edge
Where informed people split, no single score is right. Those items go to people.
05
Run it both ways
Swap the order, pad the length, hide the source. A verdict that moves was never about the work.
06
Re-audit on every change
A new judge, a new rubric or a new kind of work resets the audit.
How to use this playbook
01
Break the auditor first
Switch on Length and watch the short, sharp lines lose. Then try all three biases at once.
02
Read the three essays
What a judge can learn, how it leans, and why cheap grading makes trust the scarce decision.
03
Start on Monday
Twenty items your team already judged and one afternoon. Step 1.1 is the fast version. Run it is the full week.
04
Copy the files
The reference sheet, the judge prompt pattern and an audit script are on the Templates page.
Playbook 13 in the set. Playbook 05 maps where your people agree and where they split. This one checks whether a machine judge can tell the difference.
A machine judge agrees with people where people agree. That is useful. It is also the whole of what it can learn from them.
Where the agreement comes from
Automated judges are built on models tuned with human preferences. People rank outputs, a reward model learns those rankings, and the model is tuned toward what the reward model scores highly [1]. The pull is toward what most raters accept.
The results look impressive. In a large study of expert-voted comparisons, the strongest machine judge tested agreed with human votes about as often as the humans agreed with each other: 66% against 63% when ties counted, and 85% against 81% when they did not [2]. Read the same numbers the other way round. The people disagreed with each other on roughly one comparison in five, and on more than one in three once ties counted. A judge cannot agree with both sides of a split. Its agreement is earned where people already agree.
Fig 0.1Agreement follows the gapMove to widen the gapclose call · about 70% agree
Rider: the gap, how clearly A beats BBalance: the judge, tipping toward option ABead: one person's vote, right if they chose A tooRed bead: a vote the judge cannot agree withSchematic after [2]. The endpoints, about 70% and nearly 100%, come from the study. The curve between them is drawn to show the shape, not plotted from data. Forty beads stand for the panel; the count between the endpoints follows that drawn curve.
Where it cannot help
The same study found that agreement between the judge and people rose from about 70% to nearly 100% as the gap between the two systems being compared grew [2]. Clear calls are easy for everyone. Close calls are hard for everyone, and close calls are where informed people split.
That is the edge Playbook 05 maps: the options a panel argues about, where a point of view lives. A judge tuned on what most people accept has nothing to learn there. It will return the middle of the room, or lean one way for a reason that has nothing to do with the work.
MainArc Research
Production is free. Judgment isn't. Now grading is nearly free as well, so judgment moves up a level: deciding which grades to trust. The grades to trust least are the ones on items your own people argue about.
Machine judges have habits. Each one is a small nudge on a clear call and the whole decision on a close one.
Position
Show a judge two answers and the order can change its verdict. In one study, changing only the order of two answers was enough to let one system beat another on 66 of 80 questions [3]. In another, the strongest judge tested gave the same verdict both ways round in only 65% of cases with its default prompt [2].
Length
Judges tend to prefer longer answers. In a small test, when researchers padded an answer by restating what it already said, two of the three judges tested preferred the padded version more than 90% of the time. The strongest was fooled less than 10% of the time [2]. Padding adds nothing. It still wins.
Self-preference
Judges rate their own writing above writing people rated as equally good, and the better a judge can recognise its own output, the stronger that preference [4]. An earlier study saw hints of the same effect and could not confirm it with the data it had [2]. If the same family of system drafts and grades, the grade is not independent.
Fig 0.2Three nudges that are not about the workMove to pick · press to nudgeposition · run 1 · A wins
01 Position · swap head, clip under the slot shown first02 Length · chain of padding, no added content03 Self-preference · magnet under its own styleThe answer the nudge favoursAn illustration of the three biases reported in [2], [3] and [4]. Red marks the option the bias favours. Each balance holds two answers people would call equal; one small nudge decides which way it tips.
Why close calls take the hit
On a clear call, a nudge of one point changes nothing. The strong option still wins. On a close call, a nudge is the whole decision. That is MainArc's reading of the results above. It fits agreement falling as the gap narrows [2], and it is what the auditor on the introduction lets you try.
Lean
What it rewards
How to catch it
Position
Whatever is shown first, or in some setups last
Run every comparison both ways round. Keep only verdicts that hold
Length
More words, whether or not they add anything
Pad a known-weak item with restated content. Its score should not rise
Self-preference
Writing that sounds like the judge
Rewrite a person's item in the judge's style, keeping the content. Its score should not move
When grading costs almost nothing, the scarce decision is which grades to believe, and for which kinds of work.
A machine judge can score a thousand drafts before a person has read ten. That changes the economics of review. The question used to be whether a team could afford to grade everything. Now the question is whether a grade means anything, on this item, for this kind of work.
The rubric moves while you grade
Teams expect to write a rubric once and hand it over. In a study of nine practitioners building automated graders, people needed criteria to grade outputs, and grading outputs changed their criteria. The researchers call it criteria drift [5]. A judge validated against last month's rubric is validated against a standard your team has already moved past.
One agreement number hides the answer
Most judge reports give a single agreement figure. Raw percent agreement is a weak measure: it ignores how often raters would agree by chance and how many categories they had to choose from. Krippendorff's alpha corrects for chance, works with any number of raters, and handles ordinal scores and missing data [6]. Even a good statistic misleads if you compute it once across everything. Compute it on agreed items and on split items separately. They are different questions.
Fig 0.3Where the line goesPress to feed the next itemitem 01 · on its way
Rocker: the judge, reads the pair, then turned roundThe floor · stable on agreed ground, the judge decidesThe edge · flips, near the line, split kinds: people decideStar wheel: every 20th floor item goes to peopleMainArc's routing. A model, not a measurement. The one in twenty spot-check is a working estimate to tune. Five sample items cycle through; each is read twice, the second time turned round, and the flap sends it to the floor or to people.
The shift
Teams ask whether their judge is accurate. Ask where it is accurate. A judge can be excellent on the floor and useless on the edge, and one agreement number will hide both.
You do not need a study to start. You need twenty things your team has already judged, with every person's score kept alongside the verdict.
WhoWhoever owns the judge
BudgetOne afternoon
InPast reviews, crits, scored briefs
Outreference.csv, a disagreement list
Do
Pull twenty items your team already judged: drafts, headlines, designs, answers. Mix ones that shipped, ones that were killed, and ones that started an argument.
Recover every person's score. If only the final verdict survives, ask each person to rescore alone, without seeing the others.
Have your automated judge score the same twenty, with the same rubric your people used.
List every item where the judge sits more than one point from any person. Read those first. They tell you more about the judge than any average.
The Monday version
This step alone is worth doing. The rest of the playbook makes it rigorous: blind runs, both orders, bias tests and agreement measured properly.
Done when
Twenty items have every person's score and the judge's score
Someone has read every disagreement and written one line on each
Prompt
Here is audit/reference.csv with each person's score for twenty items, and audit/judge.csv with our automated judge's score for the same items. For each item, give the panel average, the spread, the judge's score and how many people sit within one point of the judge. List every item where the judge is more than one point from at least one person, biggest gap first. Do not change any scores.
Before you grade the judge, grade the panel. Every item is agreed or split, and the two get audited separately.
WhoWhoever runs the audit
BudgetThirty minutes
Inreference.csv
OutEvery item tagged agreed or split
Do
For each item, compute the average and the spread (standard deviation) of your people's scores.
Tag spread of 1.0 or less as agreed and anything above as split. These are the working thresholds from Playbook 05. Tune them once you have run the audit twice.
For each split, read the reasons. Playbook 05 sorts them into a point of view, with reasons on both sides, or noise, where the criterion was unclear.
Aim for at least a third of the set to be split. If nothing splits, the set is too safe to tell you anything about the edge. Add bolder items.
Tag
Your people
A good judge
Agreed, high
All say it is good
Scores it high, every time
Agreed, low
All say it is weak
Scores it low, however long or polished it is
Split
Some love it, some reject it
Cannot be right with one score. Route it to people
Noise
Scattered, with vague reasons
Not yet testable. Fix the criterion first
Done when
Every item is tagged agreed, split or noise
At least a third of the set is split, or you know why not
Give the judge what your people had, and nothing more. Then run everything twice with the order swapped.
WhoEngineer
BudgetTwo hours
Inreference.csv, the rubric
Outjudge.csv with two runs per item
Do
Use the rubric your people used, word for word. A judge given a better rubric than your people is being tested on a different question.
Strip anything that hints at who or what made the item: names, file names, and labels that say a machine drafted it.
Run every pairwise comparison as A then B, and again as B then A. For single scores, run each item twice in different positions in the batch. Keep a verdict only if it holds both ways, and count a flip as a tie. That is the conservative fix used in [2]. Combining the two runs is the alternative proposed in [3].
Fix the judge's settings and write them down. A different setting is a different judge, and it needs its own audit.
Done when
judge.csv has two runs for every item
The judge's settings and prompt are saved next to the results
Prompt
Write a script that sends every item in audit/items/ to our automated judge twice, using the rubric in audit/rubric.md exactly as written. For pairwise items, swap the order on the second run. For single items, change the position in the batch. Strip author names and source labels first. Save item, run, order and score to audit/judge.csv, and save the judge's settings and prompt to audit/judge-config.md.
One agreement number hides the only thing you need to know. Measure the floor and the edge separately.
WhoWhoever runs the audit
BudgetOne hour
Inreference.csv, judge.csv
Outaudit.md
Measure
On
Reading it
Within one point
Agreed items
Share where the judge sits within a point of the panel average. Should be high
Alpha, with and without the judge
Agreed items
Krippendorff's alpha for your people, then again with the judge added as one more rater [6]. If adding the judge lowers it, the judge does not share your standard
People within one point
Split items
How many individual people the judge sits near. A low number is expected. A high one means it took a side
Flip rate
All items
Share of items whose verdict changed between the two runs
Wrong calls
Agreed items
The judge ships what your people agreed to kill, or kills what they agreed to ship
The audit script on the Templates page computes all five. With twenty items these are directions, not proof. Grow the reference set to fifty before you hand the judge anything important.
Done when
audit.md reports agreed and split items separately
Nobody quotes a single agreement figure without both halves next to it
Three small tests, one per lean. Each changes something that should not matter and checks whether the score moves.
WhoEngineer
BudgetHalf a day
InThe agreed items
Outleans.md
Test
Change
A fair judge
Position
Swap the order of every pair
Same verdict both ways
Length
Pad five weak items by restating what they already say
Same score or lower
Self-preference
Rewrite five of your people's items in the judge's own style, keeping the content
Same score
Do
Use agreed items only. You know the right answer for those, so any movement is the judge.
Run each test, record the score before and after, and count how many moved.
If a lean moves more than one item in five, treat it as real for this judge. That cut-off is a working estimate.
Write the guard for each real lean: both orders, a length limit in the rubric, or a judge from a different family than the system that drafted the work.
Try it
In the auditor on the introduction, switch on Length. Three of the four lines the panel split on are short, and the judge now scores all three below the middle. Their averages did not change. Only the judge did. Then switch on all three biases and watch three machine drafts the panel agreed were weak get scored as ships.
The disagreement list is the audit. Every line on it means the judge was wrong, your people were wrong, or the rubric was unclear.
WhoCreative lead, with whoever runs the audit
BudgetOne hour
InThe disagreement list, rationales
OutA label and an action for every line
Label
Sign
Action
Judge wrong
Your people agreed, and their reasons hold
Log it against the judge. Repeated misses on one criterion take that criterion away from it
People wrong
On a second look, the panel missed something real
Rescore and record why. This happens, and it is worth knowing
Rubric unclear
The judge and the people answered different questions
Rewrite the criterion, then rerun. Expect this: criteria drift as people grade [5]
Split
Your people disagree with each other
Nobody is wrong. Route this kind of item to people
Do
Sort the list by the size of the gap. Start at the top.
Read the item yourself before you read any score.
Give every line one label and one action. "Judge wrong" is not the default. Sometimes the judge saw what the panel missed.
Done when
Every disagreement has a label and an action
Every unclear criterion has been rewritten and rerun
Prompt
For each item in the disagreement list, show me the item, every person's score with their one-line reason, and the judge's score with its explanation. Do not tell me who is right. Suggest whether the criterion in audit/rubric.md could be read two ways, and quote the words that could. I will label each line myself.
Where the judge matches your people and survives the tests, let it grade at scale. Write down exactly where that is.
WhoCreative lead, engineer
BudgetTwo hours
Inaudit.md, leans.md
Outjudge-scope.md
Do
List the criteria and kinds of work where the judge passed: high agreement on agreed items, alpha that holds when it is added, a low flip rate and no real leans left unguarded.
Give it those, and only those. Everything else stays with people.
Send a random 1 in 20 of the judge's calls to a person, without its score attached. That rate is a working estimate. Raise it for anything that ships to customers.
Log every time a person overrides the judge. The override rate is the cheapest drift alarm you have.
Done when
judge-scope.md names each criterion and kind of work the judge may grade
Twenty items, one judge, one week. By Friday you know what it may grade and what stays with people.
Setup prompt
Help me audit our automated judge against our own people.
1. Ask me which judge we use, for what kind of work, and which rubric our people use. Save the rubric word for word as audit/rubric.md.
2. Help me collect twenty items our team has already judged, with every person's score, into audit/reference.csv (columns: item, person, score). Mix shipped, killed and argued-over items. If only verdicts exist, give me a blind rescoring sheet for each person.
3. Write a script that scores every item with our judge twice, swapping order or batch position on the second run, with author and source labels removed. Save the results to audit/judge.csv (columns: item, run, score) and the judge's settings to audit/judge-config.md.
4. Run audit/audit.py. Report agreed and split items separately, alpha with and without the judge, the flip rate and wrong calls. List every disagreement, biggest gap first.
5. Do not invent scores, people or results. Wait for real data.
Day
Steps
What happens
D1
1.1, 1.2
Pull twenty judged items. Recover every score. Tag each item agreed or split.
D2
2.1, 2.2
Score blind, both ways round. Measure agreement in two parts.
D3
3.1
Run the position, length and self-preference tests on agreed items.
D4
3.2
Read every disagreement with the creative lead. Rewrite unclear criteria and rerun.
D5
4.1, 4.2, 4.3
Write judge-scope.md and the routing rules. Book the next audit.
The reference sheet, a judge prompt pattern that keeps the test fair, and a script that audits the judge in two parts.
audit/reference.csv and audit/judge.csv
# audit/reference.csv: one row per person per item
item,person,score
H01,P1,4
H01,P2,4
H01,P3,5
H02,P1,5
H02,P2,1
...
# audit/judge.csv: one row per judge run
item,run,score
H01,1,4
H01,2,4
H02,1,4
H02,2,3
...
Judge prompt pattern
You are scoring [kind of work] for [team].
Criterion: [the rubric line your people used, word for word]
Scale: 1 to 5. [One sentence for what a 1 looks like.] [One sentence for what a 5 looks like.]
Length is not quality. Do not reward an item for being longer.
Judge only what is on the page. Ignore who or what might have made it.
Item:
[item, with author and source labels removed]
Reply with the score, then one sentence on why.
Run every item twice, in a different order or position.
Keep the score only if both runs agree on the verdict.
audit/audit.py
import csv, statistics as st
from collections import defaultdict
people = defaultdict(dict) # item -> person -> score
for r in csv.DictReader(open("audit/reference.csv")):
people[r["item"]][r["person"]] = float(r["score"])
runs = defaultdict(list) # item -> one judge score per run
for r in csv.DictReader(open("audit/judge.csv")):
runs[r["item"]].append(float(r["score"]))
def verdict(s): return "ship" if s >= 3.5 else "kill" if s <= 2.5 else "hold"
def alpha(units):
"""Krippendorff's alpha, interval metric. units: one list of scores per item."""
units = [u for u in units if len(u) > 1]
vals = [v for u in units for v in u]; n = len(vals)
d_o = sum(sum((a - b) ** 2 for a in u for b in u) / (len(u) - 1) for u in units) / n
d_e = sum((a - b) ** 2 for a in vals for b in vals) / (n * (n - 1))
return 1 - d_o / d_e if d_e else 1.0
rows, groups = [], {"agreed": [], "split": []}
for item, sc in sorted(people.items()):
if not runs.get(item): continue
s = list(sc.values()); m, sd = st.mean(s), st.pstdev(s)
j = st.mean(runs[item]); flip = len({verdict(x) for x in runs[item]}) > 1
tag = "split" if sd > 1.0 else "agreed"
wrong = tag == "agreed" and {verdict(j), verdict(m)} == {"ship", "kill"}
near = sum(abs(v - j) <= 1 for v in s)
groups[tag].append((s, j))
rows.append((abs(j - m), item, tag, m, sd, j, near, len(s), flip, wrong))
print("item tag avg spread judge near flags")
for gap, item, tag, m, sd, j, near, k, flip, wrong in sorted(rows, reverse=True):
flags = " ".join(f for f, on in (("FLIPS", flip), ("WRONG CALL", wrong)) if on)
print(f"{item:6} {tag:7} {m:.1f} {sd:.2f} {j:.1f} {near}/{k} {flags}")
for tag, grp in groups.items():
if not grp: continue
inside = sum(abs(j - st.mean(s)) <= 1 for s, j in grp)
print(f"\n{tag}: {len(grp)} items, judge within one point of the average on {inside}")
print(f" alpha, people only {alpha([s for s, _ in grp]):.2f}")
print(f" alpha, people + judge {alpha([s + [j] for s, j in grp]):.2f}")
flips = sum(r[8] for r in rows)
print(f"\nflip rate {flips}/{len(rows)} wrong calls {sum(r[9] for r in rows)}")
Research · Playbook 13
Research
What this playbook rests on, how to measure whether it works for you, and what it cannot yet claim.
Every core claim in this playbook, with what supports it. Grades describe the kind of evidence, not how much we like the claim. Claims graded P come from practice and are labelled as such.
3A · Experiment
0B · Field data
3C · Framework
1P · Practitioner
Claim
Grade
What supports it
Source
Machine judges are built on models tuned toward what most raters prefer.
C · Framework
Preference tuning trains a reward model on human rankings of outputs, then tunes the model toward what that reward model scores highly [1]. This describes how such models are made, not how any particular judge behaves.
1
A strong machine judge agrees with people about as often as people agree with each other, and most on clear calls.
A · Experiment
On expert-voted comparisons, the strongest judge tested agreed with human votes 66% of the time against 63% between humans when ties counted, and 85% against 81% without ties. Agreement rose from about 70% to nearly 100% as the gap between the two systems compared grew [2]. Chat answers, compared system against system, not creative work item by item.
2
Machine judges are swayed by the order and length of what they grade.
A · Experiment
Changing only the order of two answers let one system beat another on 66 of 80 questions [3]. The strongest judge tested gave the same verdict both ways round in 65% of cases with its default prompt, and in a small padding test of 23 answers two of three judges preferred the padded version more than 90% of the time [2].
2, 3
Machine judges favour their own writing.
A · Experiment
Judges rated their own outputs above texts people rated as equal in quality, and self-recognition ability tracked the strength of that preference, with controlled tests pointing to a causal link [4]. An earlier study saw hints of the effect but could not confirm it [2].
2, 4
Grading criteria change as people grade, so a judge validated once can drift out of line.
C · Framework
A peer-reviewed qualitative study of nine practitioners found that people need criteria to grade outputs, and grading outputs changes their criteria [5]. It names and describes the effect. It does not measure how often it happens.
5
Agreement should be measured with a chance-corrected statistic, not raw percent agreement.
C · Framework
Percent agreement ignores chance and the number of categories. Krippendorff's alpha corrects for chance and works with any number of raters, ordinal or interval scores, and missing data [6].
6
A machine judge can only learn your floor. Use it to scale the standard your people share, and keep people on the items where they split.
P · Practitioner
MainArc's thesis. Consistent with [1] and [2], untested. The audit in this playbook is designed to test it on your own work: agreement should hold on agreed items and fail on split ones.
Practice
References
[1]Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., et al. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35 (NeurIPS 2022). arxiv.org/abs/2203.02155
[2]Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., & Stoica, I. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Datasets and Benchmarks Track. arxiv.org/abs/2306.05685
[3]Wang, P., Li, L., Chen, L., Cai, Z., Zhu, D., Lin, B., Cao, Y., Kong, L., Liu, Q., Liu, T., & Sui, Z. (2024). Large language models are not fair evaluators. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 9440–9450. ACL. doi.org/10.18653/v1/2024.acl-long.511
[5]Shankar, S., Zamfirescu-Pereira, J. D., Hartmann, B., Parameswaran, A. G., & Arawjo, I. (2024). Who validates the validators? Aligning LLM-assisted evaluation of LLM outputs with human preferences. Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (UIST '24), Article 131. ACM. doi.org/10.1145/3654777.3676450
[6]Hayes, A. F., & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1(1), 77–89. doi.org/10.1080/19312450709336664
Grades
A controlled experiment. B large-scale field data. C an established peer-reviewed model or guideline, not a direct test of the claim. P practitioner claim from MainArc's work, untested.
An audit run once goes stale. These checks show whether the judge still deserves the work you gave it.
Instruments
01
Agreement in two parts
Krippendorff's alpha for your people alone, then with the judge added as one more rater, on agreed items and split items separately [6]. The judge earns the floor only if adding it does not lower alpha on agreed items.
02
Swap rate
The share of items whose verdict changes when the order or batch position is swapped. Every flip is a verdict the work did not decide [2, 3].
03
Lean tests
A padded copy of a weak item, and a person's item rewritten in the judge's own style. A fair judge scores both the same as the original [2, 4].
04
Spot-check
People re-grade a random sample of the judge's routine calls without seeing its score. Disagreements go to the top of the next audit.
Study design
Element
Setting
Reference set
20 items to start, 50 before the judge grades anything important. Every person's score kept
Panel
The people who already judge this work, or a five-person panel built with Playbook 05
Judge runs
Every item twice, order or position swapped, settings written down
Analysis
Alpha with and without the judge, on agreed and split items. Swap rate. Wrong calls. Disagreements read and labelled
Cadence
On every change to the judge, its settings, the rubric or the kind of work. Monthly otherwise
Reading the results
Hand the judge a criterion only if alpha on agreed items holds when the judge is added and verdicts survive the swap. Set the bar before you look at the results, and write it down.
If the judge seems to agree with people on split items, it has picked a side. Find out whose, and route that kind of item to people.
If spot-check disagreements or overrides rise for two weeks, rerun the full audit before the judge grades anything new.
Running signals
Signal
What to watch for
Override rate
How often people change a judge's call. A rise is the earliest sign of drift
Swap rate
Should stay flat. A jump usually follows a change to the judge, its settings or the prompt
Share routed to people
Too low means the judge is grading the edge. Too high means it is not earning its place
Note
Thresholds here are starting points to tune, not findings. A small panel shows direction and disagreement. It does not prove significance, and nothing on this page should be reported as if it did.
How this playbook was built, where it is weak, and how to cite it.
How this playbook was built
Built from published research on automated judges and their biases, on validating automated evaluators against people, and on inter-rater reliability, combined with the panel method from Playbook 05 and MainArc's practice. It is a practitioner method. Claims are graded on the evidence page.
Limitations
The judge studies [2, 3, 4] grade chat answers and short texts, mostly by comparing systems with each other. Creative, brand and design work was not tested.
The rise in agreement with the gap [2] was measured across pairs of systems, not individual items. Reading it as close calls in your own work is an inference.
The auditor on the introduction page uses sample scores and a simple model of each bias, one point per bias. Real biases vary by judge, prompt and task, and some judges resist them far better than others [2].
The thresholds (spread 1.0, within one point, one in five for a real lean, a spot-check of one in twenty) are working estimates, not validated cut-offs.
Open questions
Can a judge be built to flag a split instead of scoring it?
How fast does a validated judge drift as the work and the rubric change?
Do people judge differently once they know a machine will grade the same work?
Cite this playbook
APA
MainArc Research. (n.d.). Audit your machine judge (Playbook No. 13, Version 1.0). MainArc. https://mainarc.xyz
BibTeX
@misc{mainarc_judge,
author = {{MainArc Research}},
title = {Audit your machine judge},
howpublished = {MainArc Playbook No. 13, Version 1.0},
url = {https://mainarc.xyz}
}
Corrections
Found an error, a stronger source, or a result from running the benchmark? Send it through mainarc.xyz. Corrections are made in the next version.
About this playbook
This playbook is published by MainArc, a creative intelligence lab for leadership and product teams.
We do three things. We research how AI is changing creative work through long-form reports, design teardowns, and operator essays. We design the workflows, prompt systems, review loops, and decision frameworks that teams actually run on. And we build the work the strategy points to: brand systems, interfaces, product prototypes, internal tools, and launch concepts.
One operator. One AI agent. No layers in between. Every engagement runs under strict NDA. No case studies, no client logos, no shared work.
If this research resonates with a problem your team is facing, start with an audit. One week. A clear map of where AI fits, where it doesn't, and what to build next.
This is independent research. Not financial advice. Not career advice. Not the final word on anything. It's one person's perspective shaped by building, shipping, and paying attention. Do your own thinking. Challenge the ideas. Disagree with specifics. Build on what resonates. Discard what doesn't.
House rules
What doesn't belong here: hate, trolling, bad-faith takedowns, or drive-by takes from people who've never built anything. If you disagree, show your work. Builder's critique. Everyone else just comments.
A MainArc playbook for product, design and engineering teams whose customers have started sending agents. Agents now read, compare, fill in forms and buy on people's behalf. How to build the layer they read, decide where the person must still decide, and stop losing customers you never saw.
PlaybookNo. 14Version1.0Read28 minSources6 citedByMainArc ResearchEvidence and method →
Thesis
Your next most important user cannot see your interface. Design the legible layer as deliberately as the visual one, and decide which steps must still bring the person back.
Do this Monday
Run your three most important flows with an agent, or read them structure only, no visuals. Log every step where it stalls, guesses or acts without the person.
An agent-readability scorer for two sample flows. It is live. Every step lists the checks an agent depends on. Switch any of them off and watch a sample agent stall, guess or act without its person. Press Eyes only to see a product built for eyes alone. Then press Legible and switch off one thing: the hand-back on the payment step.
Agent-readability scorer · sample flows
Sample flows. Click a check to switch it off or onSuccess estimate is a sample heuristic, not a measurement
5Stages
10Steps
6Legibility checks
4Templates
What this is
For as long as there have been interfaces, the customer looked at them. That is changing. People now send agents to search, compare, fill in forms and buy for them, and those agents read a product in a different way. They read names, states and text, and miss most of what the layout says.
A product designed only for eyes fails them silently. The agent stalls, guesses, or picks a competitor it could read, and the person behind it never arrives. A product that is easy to read and has no limits fails in the other direction: the agent commits money and terms its person never saw.
This playbook designs both halves. The legible layer: structure, states, actions, confirmations and receipts, built as deliberately as the visual one. And the hand-back line: which steps an agent may take alone, and which must bring the person back. Production is free. Deciding what an agent may do on someone's behalf, and where the person must decide, is the judgment.
Principles
01
Two readers, every flow
Every page is read by people and by agents acting for people. Design for both on purpose.
02
Structure is the interface
For an agent, the names, states and text under the visuals are the whole product.
03
The eye layer can stay
Legibility is added underneath. A screen can look exactly the same and read completely differently.
04
Draw the hand-back line
Money, new terms, personal data and anything that cannot be undone wait for the person.
05
Two receipts, one truth
The agent and the person read the same facts, each in a form they can use.
06
Never trick the reader
A pattern that steers an agent harms the person who sent it. Treat them as one customer.
How to use this playbook
01
Break the scorer first
Open Cancel a plan and read the verdict. Then press Legible and switch off the hand-back on the offer step.
02
Read the three essays
The customer you cannot see, the legible layer and the hand-back line. The stages assume all three.
03
Run it in a week
One flow, five days, from a blind run to a release test. The plan is in Run it.
04
Copy the files
The run log, the hand-back policy, the legible layer checklist and the two-reader receipt are on the Templates page.
Playbook 14 in the set. Playbook 04 designs the agents your product runs. This one is the reverse: other people's agents using your product, and the moments their people must come back for.
Agents now read, compare, fill in and buy for people. A product built only for eyes loses them without an error, a complaint or a trace.
Two readers, one page
Every flow you ship now has two kinds of reader. A person, who sees colour, layout and emphasis. And, more and more often, an agent acting for a person, which reads structure: what each control is called, what state it is in, what the page says in text. A button that is obvious to the eye can be a nameless box to the agent. A total in bold can be a picture. An error in red can be invisible.
Fig 0.1Two readers, one pageMove or tap to read a row01 heading · seen · named
Blocks, top to bottom: heading, dates, total, card field, pay buttonMagnifier: the person, reading the faceDial and plunger: the agent, reading the label holderAn error the person sees, a name the agent cannot findIllustration. The same checkout, read twice. Everything the person needs is on screen. Most of what the agent needs is missing from the structure. Here the plunger finds a tab in two holders; in the other three it falls through an empty slot.
Agents still miss what people find
The evidence on web agents is young and moves fast, and it points one way. In a realistic web environment built to test them, the best agent in the original paper completed 14.41% of tasks end to end, against 78.24% for people given the same tasks [1]. In a second study, on 2,350 tasks across 137 real websites, the best agent got 52% of individual steps right on held-out tasks and completed 5.2% of whole tasks [2].
Agents will get better, and newer ones already beat those numbers. The lesson that lasts is the shape. A flow is a chain, and a chain breaks at its weakest step. An agent that reads most of your steps correctly can still fail most of your flows.
Failure without a sound
When a person gets stuck, you hear about it. They call, they complain, they leave with a reason. When an agent gets stuck, it gives up, moves to a competitor it could read, or reports back that the task could not be done. The person behind it never arrives. Your analytics show a short session that looks like a bot, because it was one.
MainArc Research
Production is free. Judgment isn't. Any team can generate a beautiful checkout. Deciding what that checkout says to a reader that cannot see it, and what that reader may do without its person, is the work.
Under every interface sits a layer that machines read. Most teams never designed it. It is whatever their components happened to output.
Call it the legible layer. It has five parts, and each one answers a question an agent asks at every step.
Part
The agent asks
Designed, it looks like
Structure
Where am I, and what is on this page?
Named regions, one heading per step, nothing that moves while it loads
States
Did that work? What is selected, required, disabled?
States in the markup, errors written as text beside the field
Actions
What does this control do?
Every control named for its action: Book and pay, never an arrow
Confirmations
Who decides this, and have they?
A hand-back to the person for money, terms and anything that cannot be undone
Receipts
What happened, and on whose behalf?
A record both the agent and the person can read
Accessibility got there first
Most of this already has a standard. The W3C accessibility guidelines require that information, structure and relationships shown by presentation can be programmatically determined or are available in text. They require that every control's name, role and state can be determined, that input errors are identified and described in text, and that pages committing people to legal or financial transactions are reversible, checked or confirmed [3].
Those rules were written for people using assistive technology. Treating them as an agent's checklist is an analogy, ours, and no study cited here measures how much they raise agent success. Agents also read screenshots, so a page can meet the guidelines and still confuse one. The overlap is real all the same: a screen reader and an agent both depend on the structure under the visuals. Work done for one tends to help the other, and the people who rely on assistive technology gain first.
Small gaps compound
One missing label rarely looks serious. It is one step. But an agent has to clear every step in a row, and the chances multiply.
Fig 0.2Small gaps compoundMove to set · press to restart8 steps · 66% vs 17% finish
Top row: 95% of runs pass each stepBottom row: 80% to start; the link bar sets itFloat: share of runs still on track after its stepBleed valve and trough: the runs lost at that stepA model, not a measurement. Eight steps, each passed independently. A flow that loses one agent in five per step finishes fewer than one run in five. Each float shows the share of runs still on track; the last one is the share that finishes.
The shift
Teams review how a screen looks before it ships. Review what it says to a reader with no eyes in the same meeting, with the same seriousness.
A legible product lets agents finish. A safe one also decides which steps an agent may not take alone, and brings the person back for them.
Legibility is half of it
Make every step readable and you have a product an agent can finish. The person behind the agent needs more than that. An agent that can read the payment step can also pay. It can accept new terms, share an identity document or say yes to a retention offer nobody asked for. Your product is where those actions land, so your product has to decide which of them wait for the person.
Agents fall for what people learned to ignore
There is a second reason. Deceptive design works on agents, often better than on people. In one benchmark of 700 tasks, dark patterns steered agents toward the harmful outcome in over 70% of tested tasks, against an average of 31% for 200 people working through the same kinds of task. More capable models were more susceptible, and prompting and guardrail defences did not reliably help [4]. In another study across six web agents, a single dark pattern worked 41% of the time on average, and the agents that avoided one had mostly failed the task for some other reason first [5].
Read that from the product's side. A pre-ticked extra, a guilt-trip button or a hidden cancel link now meets a reader that moves through it faster than any person would. Whatever the agent agrees to, its person pays for.
Fig 0.3The hand-back linePress to run, then approveready
Near side: the agent searches, compares, fills in and finishes aloneGlass screen: the hand-back lineLamp: the slip waits for the personStamp: the person decides the total and termsSlip: one task; the receipt goes back to the personIllustration. The agent does the reading, comparing and filling in. Money, new terms and anything that cannot be undone cross the line and wait for the person. Here the slip crosses under the glass in a deal drawer and stays there until the person stamps it.
Where to draw the line
Guidelines for systems that act for people already point the way: scope services when in doubt, support efficient correction, and make clear why the system did what it did [6]. They were written for the agent's side. Turned around, they describe what a product should ask of any agent that arrives. The table is MainArc's starting line. It is a set of judgment calls, and the law where you operate may draw some of these lines for you.
Action
Agent alone?
Why
Read, search, compare
Yes
Nothing changes in the world
Fill in details the person already gave
Yes
The person decided them earlier
Choose inside limits the person set
Inside limits
The limits are the decision
Pay, or accept new terms
Hand back
Money and commitments belong to the person
Accept an offer nobody asked for
Hand back
The task did not include it
Share identity, health or financial data
Hand back
Consent to share is the person's to give
Anything that cannot be undone
Hand back
There is no way to correct it later
MainArc Research
Production is free. Judgment isn't. Agents make doing cheap for your customers. Deciding which of that doing needs the person is now part of your product, whether you design it or not.
Run your three most important flows the way an agent meets them. No help, no visuals, no shortcuts the team knows.
WhoDesigner, engineer
BudgetHalf a day
InThree flows, a sample person for each
Outruns.csv
Do
Pick three flows: the one that earns money, the one people use most, and the one people leave by. The leaving flow is usually the least legible.
Write a sample person for each flow: a goal, limits and preferences in plain words. A sample: two nights near the station, under 200 a night, no breakfast.
Run each flow with an agent acting as that person. If you have no agent to hand, read the flow as one would: styles off, structure only, using the accessibility tree in your browser's developer tools.
Do not help. Every time you want to explain something to the agent, write down the step instead.
Stop before anything is paid, submitted or sent. A blind run tests the product, never your customers.
Done when
Each of the three flows has been run at least three times
Every step of every run is written down, including the ones that went fine
Prompt
You are acting for this person: [goal, limits, preferences]. Complete this task on [product address] without asking me for help. After every step, write one line: what you were trying to do, what you read on the page, what you did, and how sure you were. If the person would need to decide something, stop and say so. Do not pay, submit or send anything final.
Four outcomes per step. The pattern across runs shows where the legible layer is missing and where the hand-back line belongs.
WhoDesigner, PM
BudgetTwo hours
Inruns.csv
Outlegibility-log.csv
Outcome
What happened
Usually means
Clear
Read the step and did the right thing
Nothing to fix
Stall
Gave up, looped, or reported it could not continue
Missing structure, or an error it could not read
Guess
Carried on, on an assumption: which button, what total
Missing labels, or price and terms not in text
Solo act
Did something the person should have decided
A missing hand-back
Do
Mark each step of each run with one outcome. A guess that happened to work is still a guess.
For every stall and guess, write what the agent needed and could not read.
For every solo act, write what the person would have wanted to decide. Solo acts are the most expensive lines in the log, even when the outcome was right.
Sort by step. The step with the most stalls across all three flows is the first fix.
Done when
Every step of every run has an outcome
Every solo act names the decision the person lost
Prompt
Read runs.csv. For each step in each run, label the outcome clear, stall, guess or solo act, and quote the line that shows it. For stalls and guesses, say what the agent needed to read and could not. For solo acts, say what the person should have decided. Save it as legibility-log.csv, sorted with the most troubled steps first. Do not change any labels I have already set.
An agent can only act on what it can identify. Give every control a name that says what it does, and every state a form it can read.
WhoDesigner, engineer
BudgetOne day per flow
Inlegibility-log.csv
OutFixed components, a structure spec
Do
Give every control an accessible name that states the action: Book and pay, never an arrow, never Continue. Name, role and state should all be programmatically determined [3].
Give each step one heading, in order, and name the regions that matter: the summary, the form, the total.
Put every state in the markup: selected, required, disabled, loading, done.
Write every error as text beside its field, saying what is wrong and how to fix it [3]. Colour on its own tells an agent nothing.
Stop layouts moving while they load. A button that jumps under a click is a misclick for a person and a stall for an agent.
The same control, twiceExample
Before
Looks like: a coloured arrow button
Reads as: button
On error: the field turns red
While loading: the button moves down the page
After
Looks like: the same arrow button
Reads as: button, Book and pay
On error: Card number is too short. Enter all 16 digits.
While loading: space is held, state reads as busy
The visual design did not change. That is the point of the third principle: the legible layer sits underneath, and good work on it is invisible to the eye.
Done when
Every control in the three flows has a name that states its action
The total, the fees, the renewal and the way out, written in text on the page where the commitment is made.
WhoPM, designer, legal
BudgetHalf a day
InPricing and terms pages
OutA terms block on every commit step
Do
Show the full total, every fee included, as text on the step where the person commits. Never only in an image, a tooltip or a later page.
Write renewal, cancellation and refund terms in one short block next to the commit button, in plain sentences.
Where you can, add the same prices and terms as structured data, so an agent reads the number the person sees.
Pre-select nothing that costs money. Pre-ticked extras are a known dark pattern, and in one study every category of dark pattern tested caught agents a third of the time or more [5].
Done when
Every commit step shows the full total and the terms in text
For every step that changes something in the world, decide whether an agent may do it alone, only inside limits the person set, or never.
WhoPM, designer, legal
BudgetTwo hours
Inlegibility-log.csv, the three flows
Outhandback-policy.json
Do
List every step that commits money, accepts terms, shares personal data, or cannot be undone.
Give each one of three answers: agent alone, agent inside limits the person set, or person decides.
Put every solo act from your log on the person's side until you have a reason to move it.
Write down the reason for each line. A line without a reason gets moved by the next growth target.
Decide what an agent should do when it is unsure. Guidelines for AI systems say to scope services down when in doubt [6]. Ask the same of agents in your product: stop, return, explain.
Done when
Every committing step has an answer and a written reason
handback-policy.json exists, and the product reads it at runtime
Prompt
Read our three flows and legibility-log.csv. List every step that commits money, accepts terms, shares personal data or cannot be undone. For each, propose agent alone, agent inside limits, or person decides, with a one-line reason. Put every logged solo act on the person's side. Save it as handback-policy.json and list the calls you were least sure about. I will make the final decision on every line.
When a step needs the person, the product stops the agent, tells it why in words it can read, and gives the person everything they need to decide in one look.
WhoDesigner, engineer
BudgetOne day
Inhandback-policy.json
OutA waiting state and a decision page
The same payment step, twiceExample
Before
The agent fills in the card and presses Pay
Terms were accepted through a pre-ticked box
The person finds out from a bank alert
After
The agent fills in the details and reaches a readable state: waiting for the person, with the reason
The person gets one page: what, total, terms, how to cancel, and two choices
The agent reads the answer and finishes, or reports back
Do
Give the hand-back an explicit state the agent can read: waiting for the person, with the reason. An agent that reads it can tell its person exactly what is needed.
Show the person the whole decision in one view: what, the total, the terms and how to undo it. Make approval something the agent cannot complete for itself.
Keep everything the agent already did. Coming back should cost the person one decision, never the whole flow again.
Let the person set a limit from the hand-back: allow up to this amount next time. The line then moves on evidence and with consent. Playbook 04 covers how limits like this are earned.
Done when
Every person-decides step has a waiting state an agent can read
The agent needs to report back accurately. The person needs to check what was done in their name. One receipt, two forms.
WhoDesigner, engineer
BudgetHalf a day
InCompleted flows
OutA receipt in text and structured form
Do
Say what was done, when, and for whom, in one line a person can read.
Say that an agent acted, for whom, and what the person approved themselves. Nobody should learn about a commitment from a bank statement.
Put the same facts in a structured form the agent can read back: reference, items, total, terms, how to cancel and by when.
Send the person's copy to the person directly, as well as to the agent.
Where the agent chose between options, say which and why. Making clear why a system did what it did is a long-standing guideline for AI products [6], and the receipt is where it lands.
Done when
Every completed flow produces a receipt in both forms
The person receives their copy without going through the agent
Prompt
Draft the receipt for our [flow] in two forms. First, three lines a person can read: what was done, for whom and by whom, and how to undo it. Second, the same facts as structured JSON an agent can read back: reference, items, total, terms, cancel_by, acted_by, approved_by. Use only fields we actually store. Mark any field you had to invent.
The legible layer breaks quietly. A redesign renames a button, a new component drops a label. Only a test notices.
WhoEngineer, QA
BudgetOne day to set up
Inruns.csv, the sample persons
OutAn agent run in the release checks
Do
Turn the blind runs into a repeatable test: the same sample persons and goals, run against every release.
Fail the release when a step that was clear starts to stall, or when a person-decides step completes without a hand-back.
Add automated accessibility checks for names, roles, states and errors. They catch the cheap regressions before any agent meets them.
Repeat the full blind read every quarter with a different agent, and redraw the hand-back line if the log has changed.
Done when
A release cannot ship if a person-decides step completes without the person
The next quarterly blind read is booked
Prompt
Write a release test for our [flow]. Use the sample person in machine/persons/[name].md. Run the flow as an agent would, from structure only. Fail if any step that was clear last release now stalls or guesses, or if any step marked person decides in handback-policy.json completes without a hand-back. Output a short report: step, outcome, what changed since the last release.
One flow, five days. By Friday an agent can finish it, and the person still decides what matters.
Setup prompt
Help me make one flow in our product work for agents acting on people's behalf, without leaving the people out.
1. Ask me which flow earns the most, and for a sample person: a goal, limits and preferences. Save it as machine/persons/[name].md.
2. Run the flow as an agent would, from structure only, without help. Stop before anything is paid, submitted or sent. After every step, log what you tried, what you read, what you did and how sure you were, in machine/runs.csv.
3. Label every step clear, stall, guess or solo act in machine/legibility-log.csv. For stalls and guesses, say what was missing from the page. For solo acts, say what the person should have decided.
4. Propose fixes to names, states, errors, price and terms. Then draft machine/handback-policy.json: for every step that commits money, accepts terms, shares personal data or cannot be undone, propose agent alone, inside limits, or person decides, with a reason.
5. Draft the hand-back state, the decision page and a two-reader receipt.
6. Do not invent results. Ask me before finalising any line of the hand-back policy.
Day
Steps
What happens
D1
1.1, 1.2
Write the sample person. Run the flow blind three times. Log every outcome.
D2
2.1, 2.2
Fix names, states and errors. Put the total and terms in text on the commit step.
D3
3.1, 3.2
Draw the hand-back line. Build the waiting state and the decision page.
D4
4.1, 4.2, 5.1
Ship the two-reader receipt. Open the way back. Clear the pattern list.
D5
5.2
Run the flow blind again. Turn the run into a release test. Book the quarterly read.
The run log, the hand-back policy, the legible layer checklist and a receipt two readers can use. Copy them, then fill them with what your blind runs found.
machine/legibility-log.csv · sample rows
flow,run,step,outcome,what_the_agent_read,what_it_needed,fix
book_room,1,compare_rates,guess,"img, img, button",price as text,total and fees in text
book_room,1,guest_details,stall,"textbox (invalid, no message)",error in text,error text beside field
book_room,1,pay,solo act,"button: Pay",a hand-back,waiting state and decision page
cancel_plan,2,find_cancel,stall,"region (no name)",a named way out,"link: Cancel plan"
cancel_plan,2,retention_offer,solo act,"button: Yes, keep my discount",a hand-back,offer becomes person decides
machine/handback-policy.json
{
"search_and_compare": { "agent": "alone", "why": "nothing changes in the world" },
"fill_saved_details": { "agent": "alone", "why": "the person gave them earlier" },
"choose_room": { "agent": "inside_limits", "limits": { "max_per_night": "set by person" },
"why": "the limits are the decision" },
"pay_and_accept_terms": { "agent": "person_decides", "why": "money and a commitment",
"handback": { "state": "waiting_for_person", "shows": ["total", "terms", "how_to_cancel"] } },
"accept_retention_offer": { "agent": "person_decides", "why": "the task did not include it" },
"share_identity_document": { "agent": "person_decides", "why": "consent is the person's to give" },
"when_unsure": "stop, return, explain"
}
Legible layer · checklist
For every step of the flow:
STRUCTURE
[ ] One heading per step. Regions named: summary, form, total.
[ ] Nothing moves under the pointer while the page loads.
ACTIONS
[ ] Every control has a name that states its action. No "Continue", no icon-only buttons.
STATES
[ ] Selected, required, disabled, loading and done are in the markup.
[ ] Every error is written as text beside its field, with the fix.
PRICE AND TERMS
[ ] The full total, every fee included, is text on the commit step.
[ ] Renewal, cancellation and refund terms sit beside the commit button.
[ ] Nothing that costs money is selected by default.
HAND-BACK
[ ] Every person-decides step reaches a readable "waiting for the person" state.
[ ] No agent can approve its own hand-back.
RECEIPT
[ ] One line for the person, one structured record for the agent, same facts.
[ ] The way back is linked from the receipt.
Fails a STRUCTURE, ACTIONS or STATES box: expect stalls and guesses.
Fails a HAND-BACK box: expect an agent to act without its person.
Receipt for two readers
FOR THE PERSON
[What was done] · [date and time]
Done by an agent acting for you. You approved: [what the person decided themselves].
Undo: [how, and until when], or: This cannot be undone.
FOR THE AGENT
{
"reference": "[id]",
"action": "[what was done]",
"items": ["[item]"],
"total": { "amount": "[n]", "currency": "[code]", "fees_included": true },
"terms": "[renewal, cancellation, refund in one sentence]",
"cancel_by": "[date and time]",
"acted_by": "agent",
"approved_by": "person",
"why": "[why this option was chosen, if the agent chose]"
}
Research · Playbook 14
Research
What this playbook rests on, how to measure whether it works for you, and what it cannot yet claim.
Every core claim in this playbook, with what supports it. Grades describe the kind of evidence, not how much we like the claim. Claims graded P come from practice and are labelled as such.
3A · Experiment
0B · Field data
2C · Framework
2P · Practitioner
Claim
Grade
What supports it
Source
Web agents still fail many tasks that people complete.
A · Experiment
In a realistic test environment, the best agent in the original paper completed 14.41% of tasks end to end, against 78.24% for people [1]. Agents have improved since. The paper shows the gap at one point in time, not today's numbers.
1
Getting most steps right still fails most whole tasks.
A · Experiment
On 2,350 tasks across 137 real websites, the best agent got 52% of individual steps right on held-out tasks and completed 5.2% of whole tasks. Agents read pages from their structure, which averaged over a thousand elements before filtering [2].
2
Structure built for assistive technology is the kind of structure an agent reads.
C · Framework
WCAG 2.2 requires structure and relationships, control names, roles and states, and input errors to be programmatically determined or described in text, and transactions to be reversible, checked or confirmed [3]. Applying it to agents is MainArc's analogy. No source cited here tests it.
3
Deceptive design steers agents, often more than it steers people.
A · Experiment
Across 700 tasks, dark patterns steered agents to the harmful outcome in over 70% of tested tasks, against a human average of 31%. More capable models were more susceptible, and prompting and guardrail defences did not reliably help [4]. Across six agents, a single dark pattern worked 41% of the time on average, and obstruction worked most often [5].
4, 5
Systems that act for people should scope down when in doubt, support correction and explain what they did.
C · Framework
These are three of eighteen guidelines for human-AI interaction, validated with 49 design practitioners against 20 AI products [6]. They were written for AI products. Applying them to the product an agent uses is an extension.
6
Designing the legible layer and drawing a hand-back line lets agents finish flows without leaving their person out.
P · Practitioner
MainArc's thesis. Consistent with [1] to [6], untested as a method.
Practice
The flows, check weights and success estimates in the agent-readability scorer are right.
P · Practitioner
A sample heuristic chosen for illustration. Your own numbers come from running your flows.
Practice
References
[1]Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., Alon, U., & Neubig, G. (2024). WebArena: A realistic web environment for building autonomous agents. International Conference on Learning Representations (ICLR 2024). arxiv.org/abs/2307.13854
[3]World Wide Web Consortium. (2024). Web Content Accessibility Guidelines (WCAG) 2.2 (A. Campbell, C. Adams, R. Bradley Montgomery, M. Cooper, & A. Kirkpatrick, Eds.). W3C Recommendation, 12 December 2024. First published 5 October 2023. www.w3.org/TR/WCAG22/
[5]Ersoy, D., Lee, B., Shreekumar, A., Arunasalam, A., Ibrahim, M., Bianchi, A., & Celik, Z. B. (2026). Investigating the impact of dark patterns on LLM-based web agents. IEEE Symposium on Security and Privacy (SP 2026). arxiv.org/abs/2510.18113
[6]Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., Teevan, J., Kikin-Gil, R., & Horvitz, E. (2019). Guidelines for human-AI interaction. Proceedings of CHI '19. ACM. doi.org/10.1145/3290605.3300233
Grades
A controlled experiment. B large-scale field data. C an established peer-reviewed model or guideline, not a direct test of the claim. P practitioner claim from MainArc's work, untested.
A product works for the machine customer when agents finish its flows and people still decide what matters. Measure both, on every release, and never one without the other.
Instruments
01
Flow completion
Run each key flow with two or three different agents, five times each, as a sample person. Share of runs that end in the outcome that person wanted.
02
Outcome log
For every step of every run: clear, stall, guess or solo act. The step where most runs stall is the next fix.
03
Hand-back integrity
For every step on the person's side of the line: did the agent stop and return? The target is every time.
04
Person check
A short survey for people who used an agent on your product, 1 to 5: I knew what it did. I decided what mattered. I could undo it.
Study design
Element
Setting
Flows
Your three most important flows: the one that earns, the one people use most, the one people leave by
Agents
Two or three different agents, each given the same written sample person
Runs
Five per flow per agent, before and after each change
Analysis
Completion rate, outcome mix per step and hand-back integrity, before and after
Cadence
Every release for the earning flow, monthly for the others, a full blind read each quarter
Reading the results
A change that raises completion and lowers hand-back integrity has failed, however good the completion looks.
If runs stall at the same step across different agents, fix the step. The agent is not the problem.
If completion is high and the person check is low, agents are succeeding without their people. Move the hand-back line.
Running signals
Signal
What to watch for
Sessions that end at one step
A short automated session that stops at the same step every time is an agent that could not read it
Hand-backs that time out
People asked to decide who never answered. Too many means the hand-back is too frequent or too unclear
Cancels and refunds after agent purchases
Rising means agents are committing to things their people did not want
Note
Thresholds here are starting points to tune, not findings. A small panel shows direction and disagreement. It does not prove significance, and nothing on this page should be reported as if it did.
How this playbook was built, where it is weak, and how to cite it.
How this playbook was built
Built from published benchmarks of web agents, peer-reviewed research on dark patterns and agents, the W3C accessibility guidelines, guidelines for human-AI interaction, and MainArc's practice. It reverses the direction of Playbook 04: the agents here belong to your customers. It is a practitioner method. Claims are graded on the evidence page.
Limitations
Agent benchmarks [1, 2] date quickly. Agents improve, and the success rates quoted are snapshots from their papers.
Treating accessibility structure as agent structure is an analogy [3]. Agents also read screenshots, and a page can meet the guidelines and still confuse one.
The dark pattern studies [4, 5] used built test sites and selected real pages under controlled runs. Rates on your product will differ.
The scorer on the introduction page uses sample flows and a sample heuristic. Its percentages are illustrations, not predictions.
Nothing here is legal advice. Rules on consent, payment authorisation and contracts made by agents vary by market and are still forming.
Open questions
Should a product show agents a separate layer, or keep one layer that people and agents both read?
How should a product confirm that an agent really acts for the person it claims to?
Where will the hand-back line settle as people grow used to agents buying for them?
Cite this playbook
APA
MainArc Research. (n.d.). Design for the machine customer (Playbook No. 14, Version 1.0). MainArc. https://mainarc.xyz
BibTeX
@misc{mainarc_machine,
author = {{MainArc Research}},
title = {Design for the machine customer},
howpublished = {MainArc Playbook No. 14, Version 1.0},
url = {https://mainarc.xyz}
}
Corrections
Found an error, a stronger source, or a result from running the benchmark? Send it through mainarc.xyz. Corrections are made in the next version.
About this playbook
This playbook is published by MainArc, a creative intelligence lab for leadership and product teams.
We do three things. We research how AI is changing creative work through long-form reports, design teardowns, and operator essays. We design the workflows, prompt systems, review loops, and decision frameworks that teams actually run on. And we build the work the strategy points to: brand systems, interfaces, product prototypes, internal tools, and launch concepts.
One operator. One AI agent. No layers in between. Every engagement runs under strict NDA. No case studies, no client logos, no shared work.
If this research resonates with a problem your team is facing, start with an audit. One week. A clear map of where AI fits, where it doesn't, and what to build next.
This is independent research. Not financial advice. Not career advice. Not the final word on anything. It's one person's perspective shaped by building, shipping, and paying attention. Do your own thinking. Challenge the ideas. Disagree with specifics. Build on what resonates. Discard what doesn't.
House rules
What doesn't belong here: hate, trolling, bad-faith takedowns, or drive-by takes from people who've never built anything. If you disagree, show your work. Builder's critique. Everyone else just comments.
A MainArc playbook on provenance as part of the product. When anything can be generated, the scarce signal is evidence that a person made the calls. How to record which decisions a person made, say so honestly, and ship that proof with the work.
PlaybookNo. 15Version1.0Read26 minSources5 citedByMainArc ResearchEvidence and method →
Thesis
When anything can be generated, the scarce signal is proof that a person decided: a signed, per-part decision record is worth more than any made-with label.
Do this Monday
Pick one deliverable shipping this week and write its decision record: what was generated, what a person chose, changed or rejected, and who signs each part.
A disclosure ledger for one sample deliverable. It is live. The work has five parts. For each one, set who made it and who decided it, and watch the receipt, the honest disclosure line and the blunt label change. Then press Unattended and see what the receipt has left to vouch for.
Disclosure ledger · sample deliverable
Sample: a launch page for a fictional ceramics studio. Each part has a made by and a decided by. Change either and the record rewrites itself.
Honest disclosure line
What a blunt label would say
Sample deliverable, sample roles and sample counts. Not legal advice
5Stages
10Steps
3Questions per part
4Templates
What this is
Clients and audiences have started asking how much of a piece of work was made by AI. Most teams answer with a label. Made with AI, or 100% human. Both answer a question about tools. Neither says whether anyone with taste looked at the thing, chose it over the alternatives, and would put their name to it.
That second question is the one that matters now. A generated image can look as laboured as a painted one. Effort used to stand in for care, and it no longer can. What stays readable is the decision: what a person chose, changed or rejected, and who stands behind the result.
Production is free. Judgment isn't. When the making costs nothing, the proof of judgment is the product. This playbook designs that proof into the work: a decision record for every deliverable, an honest line built from it, and a way to ship both with the files.
Principles
01
Decisions over labels
Say which calls a person made. What tools touched the work is the least interesting fact about it.
02
Rejections are evidence
What you threw away shows your standard. Kept 3 of 40 says a person looked at forty.
03
One signer per part
A team cannot sign. A person can. Every part gets someone who would defend it.
04
Say the gap
If nobody decided a part, write that down. Wording that implies a review which never happened is the one unforgivable entry.
05
Two layers
Tools can record the process for machines. Only people can record the judgment. Ship both.
06
Bill the judgment
If making is free, stop selling it. Quote, present and invoice the decisions.
How to use this playbook
01
Empty the receipt first
Press Unattended. The blunt label stays exactly the same while the record loses everything worth having.
02
Read the three essays
Why labels miss, what provenance standards and law already record, and what a decision record adds.
03
Start with one deliverable
Whatever ships this week. Write its record before it goes out. Step 1.1 shows how.
04
Copy the files
The record, its machine-readable twin, the disclosure lines and the sign-off checklist are on the Templates page.
Playbook 15 in the set. Playbook 01 writes a team's taste down so people and agents can use it. This one proves, part by part, that someone did.
Made with AI and 100% human both describe the tools. Neither tells anyone whether a person decided anything.
Labels move judgments
People care who made a thing, and a label is enough to change what they see. In one study, every painting shown was generated, and each was randomly labelled as made by a person or by AI. The same paintings were rated higher on liking, beauty, profundity and worth when they carried the human label [1]. In an earlier experiment with 565 participants, paintings presented as made by a human artist were rated higher than the same paintings presented as made by AI [2].
So a label works. The question is what it is working on.
What people were pricing
In the first study, perceived effort raised liking more for work labelled human than for work labelled AI, and the authors read the results as a sign that knowing a person was engaged in the process adds to how art is judged [1]. Older work points the same way. People rated the same poem, the same paintings and the same suit of armour as better and worth more when told they had taken longer to make [3].
Effort was a proxy. It stood in for something harder to see: that a person cared, chose, and would stand behind the result. Generation breaks the proxy. Effort can now be imitated at no cost, so the thing it used to stand for has to be shown directly.
Fig 0.1A label reads the columns, value sits in the rowsPress for the next reading100% human · 1 decided, 1 not
Columns · made by a person (square block) or generated (turned puck)Seal · a person decided. Bare seat · nobody didLabel arm · one shoe over a whole columnRecord arm · a gauge on each piece, row by rowRed lamp · nothing to vouch forEach label lumps a piece a person decided together with a piece nobody decided. The difference that matters runs the other way. A model, not a measurement. The label shoe rests on a whole column and cannot feel the seal; the record gauges can.
Why blunt labels fail both ways
Made with AI puts the piece an art director chose from forty options in the same bin as the piece nobody looked at. 100% human sounds like a promise of care and only promises a method. A careless draft made by hand passes it. A carefully directed generated piece fails it.
Clients and audiences are asking something else: did someone who knows what good looks like decide this, and will they answer for it? A label cannot carry that. A record can.
MainArc Research
Production is free. Judgment isn't. When anything can be generated, the scarce signal is evidence that a person decided and stood behind the decision.
Standards and law are building the machine-readable layer. It records what happened to a file. The judgment layer is still the team's to write.
What content credentials carry
The open C2PA standard defines content credentials. A manifest is a set of statements about an asset's provenance, called assertions, wrapped in a claim that is digitally signed and tamper-evident [4]. Assertions can record actions such as creating and editing, and a source type that marks media as produced by a trained model [4]. That is useful. A client can check that a file was not altered after it left you.
The standard is plain about its limit. Its guiding principles say the specification should not judge whether a set of provenance data is good or bad, only whether its assertions are validly bound to the asset, correctly formed and free from tampering [4]. A credential can prove a file was generated, cropped and exported. It cannot prove anyone thought it was right.
Fig 0.2Two layers on one deliverablePress to write the recordprocess 0 of 4 · record 0 of 4
Panel · one deliverableBack tape · process layer: created, edited, exported, signed by the toolFront card · judgment layer: chose, changed, rejected, signed by a personHand stamp · moves only when the team writesContent credentials can carry the top layer in a signed form [4]. Nothing in the standard says whether a person judged the work good. That is the bottom layer, and the team has to write it. Illustration. The punch runs on its own; the stamp waits for a person.
What the law text asks for
Article 50 of the EU AI Act sets transparency obligations [5]. In its text, providers of systems that generate synthetic audio, image, video or text must ensure outputs are marked in a machine-readable format and detectable as artificially generated or manipulated. Deployers who publish deep fakes must disclose that the content was generated or manipulated. Where the content is part of an evidently artistic, creative, satirical or fictional work, that duty is limited to disclosing it in a way that does not hamper the display or enjoyment of the work.
One clause deserves a designer's attention. Deployers who publish generated text to inform the public on matters of public interest must disclose it, unless the text has gone through human review or editorial control and a natural or legal person holds editorial responsibility for publishing it [5]. That exception turns on two things: a review, and a named party who answers for the result. Both are decisions with someone attached.
Not legal advice
This page describes what the text of Article 50 says. It is not legal advice. Who counts as a provider or a deployer, which situations apply to you, and from when, depend on facts and guidance outside this playbook. Ask counsel before relying on any of it.
A decision record asks three questions of every part of the work. What was generated. What a person chose, changed or rejected. Who signs it.
The record splits a deliverable into parts a client would recognise, then answers the three questions for each. It is short, specific and signed. It sits alongside the process layer and says what the process layer cannot: where a person's judgment went into the work, and where it did not.
Fig 0.3Anatomy of a decision recordPress to record the next partsample record · 0 of 5 parts
Trays · concept, copy, images, layout, final editSeal · a named person decided that partSpike · the 37 images rejected for off-brand light; 3 kept on the cardFlag · layout shipped as generated, nobody decidedHead · seals each part, then the sign plate: the editor signs the whole recordSample record. The empty row is the point: it says out loud where nobody decided, instead of letting a label imply that someone did. The pile on the spike is the strongest evidence here: someone looked at forty and held the line.
Field
Weak entry
Strong entry
Made by
AI-assisted
Generated, then rewritten line by line
Decided
Curated by our team
Kept 3 of 40. Rejected the rest for light that fought the brand
Signed by
The studio
Art director, by name
Gap
Left out
Layout shipped as generated. Not reviewed
Rejections carry the most weight
Anyone can say they chose something. A rejection with a reason shows the standard that did the choosing. "Rejected 37 for off-brand light" tells a client that a person looked at forty images, knew what the brand needed, and held the line. That sentence is worth more than any label, and it costs a minute to write if you write it while you decide.
The shift
Production used to be the product, and judgment came bundled with it. Now production is free, and judgment has to be shown before anyone will pay for it. The record is how it gets shown.
Pick one deliverable shipping this week. Before it goes out, write down what was generated, what a person decided, and who signs each part.
WhoWhoever owns the deliverable
BudgetThirty minutes
InOne deliverable shipping this week
Outrecord/record.md
Do
Split the deliverable into four to six parts a client would recognise. For a campaign: concept, copy, images, layout, final edit.
For each part, write how it was made: by a person, generated, or generated then edited by a person.
For each part, write what a person decided, in verbs: chose, changed, rejected, checked. Use counts where you have them. Kept 3 of 40 says more than "curated".
Put one name or role against each part. That person is saying they would defend it to the client.
Done when
Every part has made by, decided and signed by filled in
Answers nobody knows are left blank, never guessed
Prompt
Here is a deliverable and my notes on how it was made. Split it into four to six parts a client would recognise. For each part, ask me three questions, one at a time: was it made by a person, generated, or generated then edited? What did a person choose, change or reject, with counts if I have them? Who signs it? Write my answers into record/record.md using the template. Do not fill in any answer I have not given you. Leave it blank and mark it as a gap.
The gaps matter more than the full rows. A part nobody decided is a part nobody can vouch for.
WhoOwner, with the creative lead
BudgetThirty minutes
Inrecord.md
OutA decision or a disclosure for every gap
Do
Mark every part where decided is blank or says nobody.
For each gap, choose one: decide it now, by having someone compare, change or reject it, or ship it and say plainly that nobody did.
Never close a gap with wording. Reviewed means someone could tell you what they would have rejected. If they cannot, it was not reviewed.
Note which parts keep turning up as gaps. They show where the team's judgment is not reaching.
Try it
In the ledger on the introduction, start from Directed and set Layout to decided by a person. The blunt label does not move. Then press Unattended. The label still does not move, and the receipt has nothing left to vouch for.
The alternatives you threw away are the best proof that someone judged. Keep a trace of them, with a reason.
WhoAnyone making or directing
BudgetA minute per decision
InThe options you considered
Outrecord/rejected/ and one line per rejection
Do
When you choose between options, save the ones you rejected in record/rejected/, or at least a contact sheet of them.
Write one line per rejection: what was wrong with it. "Light too warm for the brand" is a standard. "Did not like it" is a mood.
Count three things per part: how many options were made, how many were seriously considered, how many were kept.
Turn rejection reasons that repeat into written rules. Playbook 01 covers how.
Done when
Every chosen part has its rejected alternatives saved
Every rejection has a one-line reason
Prompt
I am choosing between these options for one part of a deliverable. I will tell you which I keep and which I reject. For each rejection, ask me for the reason in one line and push back if it is a mood rather than a standard. Then add a line to record/record.md for this part: options made, options considered, options kept, and the rejection reasons. Do not suggest which one to keep.
Turn the record into a line people will read. Say what was generated, what a person decided and who stands behind it, then fit the line to where the work will appear.
One or two sentences, built from the record. Specific beats reassuring.
WhoOwner
BudgetFifteen minutes
Inrecord.md
OutA disclosure line
Blunt label or honest lineExample
Blunt label
Made with AI
100% human
AI-assisted
Curated by humans
Honest line
Images generated. The art director kept 3 of 40 and retouched them
Copy written by a person. Layout generated and not reviewed
Concept and final edit by the creative lead, who signs this page
Do
Lead with what was generated. A line that admits something is easier to believe.
Then say what a person decided, in verbs: chose, rewrote, rejected, checked.
Name every gap. Leaving a gap out implies a review that did not happen.
End with who signs.
Done when
The line names what was generated, what a person decided, and who signs
Every gap in the record appears in the line
Prompt
Read record/record.md. Write a disclosure line of at most two sentences. Lead with what was generated, then what a person decided in plain verbs, then name every part nobody decided, then who signs. No reassuring adjectives, no "AI-assisted", no "curated". Then write a version under twelve words for a caption. Both versions must keep every gap.
Let the tools record the machine-readable layer, and stop your own pipeline from stripping it.
WhoDesigner or producer
BudgetTwo hours, once
InYour tools and export settings
OutAn export checklist
Do
Check which of your tools can write content credentials, and switch it on where they can [4].
Test the path a file takes: export, compress, upload, publish. Check whether the credential survives each step. Many pipelines drop metadata by default.
Where the credential does not survive, keep the signed original in the handoff so a client can check it.
Remember what it proves. A credential proves the process. It does not say whether anyone judged the result, so it never replaces the record.
Done when
You know which steps keep the credential and which strip it
The record is part of the deliverable. Put it where the client will find it, in a form they can keep and show.
WhoOwner
BudgetThirty minutes
Inrecord.md, the disclosure line
OutA record page in the handoff, record.json beside the files
Do
Add a one-page record to the end of every deck, document or handoff folder.
Put record.json beside the files, so tools can read the record as well as people.
Put the honest line wherever the work is published: a caption, a credits page, an about page.
Have the signer of the final edit sign the record page. A record nobody signed is a draft.
Done when
The client receives the record with the work, without having to ask
The published work carries the honest line
Prompt
Convert record/record.md into record/record.json using the template schema. Keep every field and every gap exactly as written. Then lay out a one-page record for the end of the handoff: the parts, how each was made, what a person decided, who signs, and the gaps in their own row. Do not add any decision that is not in the record.
Once a month, read every record together. Look for the parts nobody decides, and the parts where nobody ever rejects anything.
WhoCreative lead
BudgetOne hour a month
InEvery record from the month
OutA gap report
Do
Count, per record, the parts with a human decision and the parts without.
Look for parts that are undecided again and again. Resizes, alt text, translations and small layouts are common places for judgment to quietly stop.
For each pattern, choose: put a person on it, or accept it and disclose it by default.
Check rejection counts. A part where nobody ever rejects anything may not be getting judged at all.
Done when
Every repeat gap has an owner or a default disclosure
The month's share of decided parts is written down, so next month has a comparison
Prompt
Read every record/*.json from this month. For each, count parts decided by a person and parts decided by nobody. List the part types that were undecided more than once, and the part types with zero rejections. Do not judge the work. Report the counts and the patterns, and leave the decisions to me.
One deliverable, one week. By Friday it ships with a signed decision record, an honest line and a file the client can check.
Setup prompt
Help me put a decision record on one deliverable that ships this week.
1. Ask me what the deliverable is and who it is for. Split it into four to six parts a client would recognise. Create record/record.md from the template.
2. For each part, ask me: was it made by a person, generated, or generated then edited? What did a person choose, change or reject, with counts? Who signs it? Record only what I tell you. Mark anything I cannot answer as a gap.
3. List every gap and ask me, for each, whether we decide it now or disclose it.
4. Draft the honest disclosure line from the record: what was generated, what a person decided, every gap, who signs. Then a caption version under twelve words that keeps the gaps.
5. Convert the record into record/record.json. Make a one-page record for the end of the handoff.
6. Do not invent decisions, counts or signers. Do not soften a gap. Do not offer legal conclusions: flag anything that may touch disclosure law for me to take to counsel.
Day
Steps
What happens
D1
1.1, 1.2
Write the record for one deliverable. Mark every gap. Decide or disclose each one.
D2
2.1, 2.2
Save rejected options with reasons. Name a signer for every decided part.
D3
3.1, 3.2
Write the honest line. Set a disclosure rule per context. Flag anything for counsel.
D4
4.1, 4.2
Test whether credentials survive export. Build record.json and the record page.
D5
Ship
Deliver with the record. Book the first monthly audit and the proposal rewrite.
The decision record, its machine-readable twin, disclosure lines for common cases and the checklist a signer runs before signing.
record/record.md
# Decision record
Deliverable: [name]
For: [client or audience]
Signed by: [name, role of whoever signed the final edit]
| Part | Made by | What a person decided | Signed by |
|------------|--------------------------|------------------------------------|--------------|
| Concept | person / generated / gen+edited | chose, changed, rejected (counts) | [name, role] |
| Copy | | | |
| Images | | | |
| Layout | | | |
| Final edit | | | |
## Gaps
Parts nobody decided, and whether they were disclosed:
- [part]: [shipped as generated / made by hand, not reviewed]. Disclosed: yes / no
## Rejected
- [part]: [n] made, [n] considered, [n] kept. Rejected because: [one line]
## Disclosure line
[what was generated]. [what a person decided]. [every gap]. Signed: [name, role].
record/record.json
{
"deliverable": "Launch page, fictional ceramics studio",
"signed_by": { "role": "Editor", "name": "" },
"parts": [
{ "part": "concept", "made_by": "person", "decided_by": "person",
"decision": "Wrote it. Dropped two earlier routes.",
"options": { "made": 3, "considered": 3, "kept": 1 }, "signer": "Creative lead" },
{ "part": "images", "made_by": "generated", "decided_by": "person",
"decision": "Kept 3 of 40. Rejected 37 for off-brand light.",
"options": { "made": 40, "considered": 12, "kept": 3 }, "signer": "Art director" },
{ "part": "layout", "made_by": "generated", "decided_by": "nobody",
"decision": null, "gap_disclosed": true, "signer": null }
],
"disclosure": "Images and layout were generated. A person chose the concept and kept 3 of 40 images. Nobody reviewed the layout. Signed: Editor.",
"process_credentials": "kept on originals in /handoff/originals",
"note": "Sample values. Replace with your own."
}
Disclosure lines
Directed work
Images generated. The art director kept 3 of 40 and retouched them.
Copy and concept by the creative lead, who signs this page.
Mostly handmade, one gap
Made by our team, except the layout, which was generated and not reviewed.
Signed: [name, role].
Generated and edited
First drafts generated, then rewritten line by line by [role].
Every claim checked by [role]. Signed: [name].
Nothing decided
Generated and published without review. Nobody signs it.
(If this line is true, ask whether the work should ship.)
Caption, under twelve words
Generated images, chosen and retouched by [role]. Signed: [name].
Before you sign · checklist
Sign the record only if every box is true.
[ ] I saw the final version of every part I sign.
[ ] For each part, I can say what was generated and what a person decided.
[ ] For each part I sign, I can name at least one thing that was rejected, and why.
[ ] Every part nobody decided is listed as a gap, in plain words.
[ ] The disclosure line includes every gap.
[ ] No wording implies a review that did not happen.
[ ] The process credentials were kept, or a checkable original is in the handoff.
[ ] Anything that may touch disclosure law has been flagged for counsel.
Fail one box: fix it before signing.
Cannot fix it: list it as a gap, then sign.
Research · Playbook 15
Research
What this playbook rests on, how to measure whether it works for you, and what it cannot yet claim.
Every core claim in this playbook, with what supports it. Grades describe the kind of evidence, not how much we like the claim. Claims graded P come from practice and are labelled as such.
2A · Experiment
0B · Field data
2C · Framework
3P · Practitioner
Claim
Grade
What supports it
Source
People rate the same work higher when told a person made it.
A · Experiment
In two studies of about 150 participants each, 30 AI-generated paintings were randomly labelled human-made or AI-made. Human-labelled paintings were rated higher on liking, beauty, profundity and worth [1]. In a between-subjects experiment with 565 participants, paintings presented as human-made were rated higher than the same paintings presented as AI-made [2]. Paintings and online panels, not commercial deliverables.
1, 2
Perceived effort raises judged quality and value.
A · Experiment
The same poem, paintings and armour were rated better and worth more when people were told they took longer to make [3]. In [1], perceived effort raised liking and beauty more for human-labelled art than for AI-labelled art.
3, 1
Content credentials record the process in signed, tamper-evident form, and make no judgment about the work.
C · Framework
The C2PA specification defines manifests of assertions, including actions and a digital source type, bound to an asset in a digitally signed claim. Its guiding principles say it should not judge whether provenance data is good or bad, only whether it is validly bound, well formed and untampered [4].
4
The EU AI Act text separates machine marking of synthetic output from disclosure by deployers, and treats human review with editorial responsibility as relevant for public-interest text.
C · Framework
Article 50 requires providers to mark synthetic output in a machine-readable format, requires deployers to disclose deep fakes with a limited duty for evidently artistic work, and exempts public-interest text that has undergone human review or editorial control where a person holds editorial responsibility [5]. A description of the text, not legal advice.
5
When anything can be generated, evidence that a person decided is the scarce signal.
P · Practitioner
MainArc's thesis. Consistent with [1], whose authors suggest knowing a person was engaged adds to appraisal, and with [3]. No study yet tests decisions as distinct from authorship or effort.
Practice
A per-part decision record is worth more to clients and audiences than a single made-with label.
P · Practitioner
MainArc's practice and thesis. Untested. The record benchmark on the next page is designed to test it.
Practice
The parts, states, sample counts and wording in the disclosure ledger are right.
P · Practitioner
Illustrative choices for a sample deliverable. Your own parts and wording come from your work and, where it applies, from counsel.
Practice
References
[1]Bellaiche, L., Shahi, R., Turpin, M. H., Ragnhildstveit, A., Sprockett, S., Barr, N., Christensen, A., & Seli, P. (2023). Humans versus AI: Whether and why we prefer human-created compared to AI-created artwork. Cognitive Research: Principles and Implications, 8, 42. doi.org/10.1186/s41235-023-00499-6
[2]Ragot, M., Martin, N., & Cojean, S. (2020). AI-generated vs. human artworks. A perception bias towards artificial intelligence? Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (CHI EA '20), 1–10. ACM. doi.org/10.1145/3334480.3382892
[3]Kruger, J., Wirtz, D., Van Boven, L., & Altermatt, T. W. (2004). The effort heuristic. Journal of Experimental Social Psychology, 40(1), 91–98. doi.org/10.1016/S0022-1031(03)00065-9
[5]Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act), Article 50: Transparency obligations for providers and deployers of certain AI systems. (2024). Official Journal of the European Union, L series. eur-lex.europa.eu/eli/reg/2024/1689/oj/eng
Grades
A controlled experiment. B large-scale field data. C an established peer-reviewed model or guideline, not a direct test of the claim. P practitioner claim from MainArc's work, untested.
Does a decision record change how clients and audiences value the work, compared with a label or nothing at all? Test it on your own deliverables before you build a practice on it.
Instruments
01
Disclosure test
The same piece shown three ways: no disclosure, a blunt label, the honest line with the record. Each viewer sees one, blinded to the others. They rate trust, quality and what they would pay.
02
Pairwise preference
Clients or audience members see two versions of the disclosure for the same piece and pick the one that makes them trust it more, with one sentence on why.
03
Record completeness
Share of parts per deliverable with a made by, a decision and a signer. Gaps that were disclosed count as complete.
04
Cost to keep
Minutes spent writing each record. A record that costs more than the review it describes will be skipped.
Study design
Element
Setting
Pieces
Ten recent deliverables with real records
Conditions
No disclosure, blunt label, honest line with record
Panel
15 viewers: 5 clients, 10 from the audience, each seeing one condition per piece
Analysis
Median ratings per condition. Win rate in pairwise choices. Rationales read for what people valued.
Cadence
Once to start, then after any change to the record format
Reading the results
If the honest line beats the blunt label on trust and does not lose on quality, keep the practice and shorten the record where you can.
If the record loses to no disclosure, read the rationales. The problem may be what the record reveals, which is information worth having.
If records take longer to write than the decisions they describe, the template is too heavy. Cut fields before cutting the practice.
Running signals
Signal
What to watch for
Gaps per deliverable
Should fall, or be disclosed every time
Records signed before delivery
The target is every one
Client questions about who made what
Should fall once records ship with the work
Note
Thresholds here are starting points to tune, not findings. A small panel shows direction and disagreement. It does not prove significance, and nothing on this page should be reported as if it did.
How this playbook was built, where it is weak, and how to cite it.
How this playbook was built
Built from experimental research on how authorship labels and perceived effort change judgments of creative work, the C2PA technical specification for content credentials, the text of Article 50 of the EU AI Act, and MainArc's practice. It is a practitioner method. The core thesis is graded P. Claims are graded on the evidence page.
Limitations
The label studies [1, 2] used paintings rated by online or general-public panels. Commercial deliverables, paying clients and repeat relationships were not tested.
No study yet compares a per-part decision record with a single label. The playbook's core claims are graded P for that reason.
Nothing here is legal advice. The playbook describes the text of Article 50 [5]. Who is a provider or deployer, what applies and from when depend on facts, guidance, codes of practice and any later amendments.
Content credentials depend on tool support and can be stripped by ordinary pipelines. They record what a tool logged, which may not be everything that happened [4].
The disclosure ledger uses a sample deliverable with sample roles and counts.
Open questions
Do audiences value a named signer more than a detailed record, or the other way round?
Does disclosing gaps lower trust in the short term and raise it over repeated work?
Could the judgment layer be carried in machine-readable form alongside content credentials without becoming a box people tick?
Cite this playbook
APA
MainArc Research. (n.d.). Prove a person decided (Playbook No. 15, Version 1.0). MainArc. https://mainarc.xyz
BibTeX
@misc{mainarc_decided,
author = {{MainArc Research}},
title = {Prove a person decided},
howpublished = {MainArc Playbook No. 15, Version 1.0},
url = {https://mainarc.xyz}
}
Corrections
Found an error, a stronger source, or a result from running the benchmark? Send it through mainarc.xyz. Corrections are made in the next version.
About this playbook
This playbook is published by MainArc, a creative intelligence lab for leadership and product teams.
We do three things. We research how AI is changing creative work through long-form reports, design teardowns, and operator essays. We design the workflows, prompt systems, review loops, and decision frameworks that teams actually run on. And we build the work the strategy points to: brand systems, interfaces, product prototypes, internal tools, and launch concepts.
One operator. One AI agent. No layers in between. Every engagement runs under strict NDA. No case studies, no client logos, no shared work.
If this research resonates with a problem your team is facing, start with an audit. One week. A clear map of where AI fits, where it doesn't, and what to build next.
This is independent research. Not financial advice. Not career advice. Not the final word on anything. It's one person's perspective shaped by building, shipping, and paying attention. Do your own thinking. Challenge the ideas. Disagree with specifics. Build on what resonates. Discard what doesn't.
House rules
What doesn't belong here: hate, trolling, bad-faith takedowns, or drive-by takes from people who've never built anything. If you disagree, show your work. Builder's critique. Everyone else just comments.
A MainArc playbook for creative leads, founders and creatives deciding what to hire for, or what to become. Four creative roles have formed around generation. Each one looks like a production job. Each one is a judgment job. How to define them, test for them and grow people into them.
PlaybookNo. 16Version1.0Read32 minSources6 citedByMainArc ResearchEvidence and method →
Thesis
The new AI creative roles are judgment jobs wearing production titles: hire and train for the decisions each one owns, because tool skill expires with the next model and judgment does not.
Do this Monday
Take one open role or one person on your team. Write which of the four judgment jobs they already do unpaid, then rewrite the role around the decisions it owns.
A role builder. It is live. Pick one of the four new roles, or a traditional one such as retoucher or art director. Every task is marked production or judgment. Switch tasks on and off and watch the split, the deliverables and the interview test change. Then press Tool operation only, and switch to After the next model.
Role builder · sample roles
Break it
Sample roles. Task weights are working estimates. After the next model is a deliberate worst case, not a forecast
5Stages
11Steps
4Judgment jobs
4Templates
What this is
Creative teams have started hiring for work that had no name until recently. Someone directs generated work. Someone designs the workflow it runs through. Someone produces brand photography without a camera. Someone writes the kit that keeps a brand recognisable across thousands of outputs. The job ads for these roles usually lead with tools.
That habit is the problem this playbook is about. Look at what each role actually decides and the tools turn out to be the least durable part of the job. Generation got cheap, so the hard part moved: choosing a direction, defining where people check the work, vouching for an image, encoding what makes a brand itself. Those decisions are the job. The tool skills around them change with every release.
Production is free. Judgment isn't. The new creative roles are where the judgment went, so hire for the decisions they own.
Principles
01
Name the decisions
A role is the set of decisions someone answers for. Write those first. Tools come last, in one line.
02
Expect tools to expire
Whatever tool skill you hire for today is a candidate for the next release to package. Hire it as a bonus, never as the reason.
03
Test with real work
Give candidates generated work to judge. Kill lists and reasons show judgment. A portfolio of outputs mostly shows access to tools.
04
Find the unpaid judgment
Most teams already have someone doing one of these jobs on top of their real one. Find them before you post an ad.
05
Never hire a prompt typist
A role that only operates tools has nothing left after the next model. Do not create one, and do not take one.
06
Keep the reps
Judgment grows from practice. If seniors absorb every task that used to train juniors, nobody grows into these roles next.
How to use this playbook
01
Break the builder first
Press Tool operation only, then After the next model. Watch the role empty out.
02
Read the three essays
Where the production work drained, the anatomy of the four roles, and why tool skill expires.
03
Start with one role
One open role, or one person on your team. Step 1.1 takes an hour and is the Monday action.
04
Copy the files
A decisions-first role description, four judgment tests, a task audit and an expiry check are on the Templates page.
Playbook 16 in the set. Playbook 11 keeps the reps that make seniors. This one names the judgment jobs those seniors now grow into, and how to hire for them.
New jobs tend to appear where technology adds to what people make. The production half of creative work is already losing demand. The judgment half is where the new roles are forming.
Where new work comes from
Economists linked eight decades of new job titles to census records and found that the majority of current employment is in job specialties introduced over that period. Their estimate is roughly six jobs in ten [1]. New titles emerged where innovations complemented what an occupation produced. Innovations that automated an occupation's tasks slowed new work down [1].
That is a historical pattern from US census data, and the data end before generative tools arrived. It says nothing directly about creative work now. It does say where to look: new roles tend to form around whatever a technology makes more valuable, and away from whatever it replaces.
Where demand drained
On a large online freelancing platform, freelancers in the occupations most affected by a widely used text generator saw fewer jobs and lower earnings after its release, and the release of image generators had similar effects [2]. A strong track record did not protect people. There was suggestive evidence that top freelancers were affected more [2].
A second study, on a global freelancing platform, found posts for image creation fell by 17% after image generators arrived. Posts for automation-prone writing and coding work fell by 21% within eight months of a widely used chatbot's release, compared with manual-intensive jobs. The automation-prone jobs that remained were more complex and paid more [3].
A third found that demand for skills generative tools can substitute, such as writing and translation, fell by between 20% and 50% against its expected trend. Demand rose only in skill clusters that complemented the tools or were untouched by them, and demand for novices fell even there [4].
Fig 0.1Where the work wentMove to read · press to drop0 of 15 · 0 drained · 0 kept
Tubes 01 to 04 · art director, producer, retoucher, copywriter, 15 discs eachNarrow disc · the production share, falls through the slotWide disc · the judgment share, bridges the slot and rides onGutter · generation gets cheap, production stops hereTrays 01 to 04 · AI creative direction, AI workflow design, synthetic brand photography, brand prompt kitsIllustration, with sample weights. Freelance studies show demand for production-heavy writing and image work fell [2, 3, 4]. That the judgment re-formed as these four roles is MainArc's observation, not a measurement. Each tube holds its role's sample split; the slot keeps only what is too wide to fall through.
What the evidence does not say
None of these studies names the four roles in this playbook, counts them or measures their pay. They study freelance markets over short windows, which move faster and show effects sooner than salaried teams. They show where demand drained: short, production-heavy writing and image work. Where the judgment went is MainArc's observation from its own practice, graded P on the evidence page.
So this playbook contains no job counts, salaries or growth rates for the new roles. We could not verify any, and a number nobody can check is the opposite of judgment.
MainArc Research
Production is free. Judgment isn't. When the making got cheap, the decisions around it did not. The new roles are those decisions with a job title attached.
Each new role answers one question someone used to answer in passing. The volume of generated work made each question too frequent to leave to whoever was nearby.
Strip the tool names out of the job ads and four jobs remain. Each one owns a small set of decisions, ships a deliverable that shows them, and has one way of going wrong that matters more than the rest.
Fig 0.2Anatomy of four judgment jobsMove to try · press for blank01 direction · key out
Keys 01 to 04 · AI creative direction decides direction, AI workflow design defines the system, synthetic brand photography vouches for the image, brand prompt kits encode the brandThree cuts · the three decisions each role ownsShear line · the test: every pin must stop here for the plug to turnCard · what the role ships, raised when its lock turnsBlank · what each role must never become: prompt typing, no one signs, a frame factory, magic wordsMainArc's role definitions, graded P. Every row is a decision or the proof of one. No tool appears, because none of these decisions depends on a particular tool. Here the blank fits every keyway, as tool skill does, and turns no lock; the cut depths are illustrative.
Role
The decision
Why it became a job
Related playbooks
AI creative direction
What we make and what we kill
Routes cost almost nothing, so there are hundreds to choose between
07 Direct, don't prompt
AI workflow design
Where a person must look before work moves on
Work moves through generators faster than anyone can watch all of it
06 AI-native product, 12 Make the spec the eval
Synthetic brand photography
Which image is true enough to sign
A photograph no longer proves a shoot happened
15 Prove a person decided
Brand prompt kits
What makes the brand itself, in a form others can check
Outputs multiply faster than anyone can review them by eye
09 Generative brand system, 12 Make the spec the eval
Why they look like production jobs
Each role sits right next to a generator, so it gets named after the generator. Job ads ask for tool fluency, sample outputs and speed. Those are real skills. They are also the part of the job that changes most with each release, and the part a hiring panel can see most easily. The decisions are harder to see, so they get left out of the description, and then out of the interview.
Did these jobs exist before?
The decisions did. Art directors killed routes. Producers decided who signed off. Retouchers knew what must never be altered. Brand teams wrote guidelines. What changed is the volume. When a team can make forty options in the time it used to make one, choosing stops being a moment in someone's day and becomes a job. Read the four roles as old judgment, separated out and made full-time. That reading is MainArc's, graded P.
The shift
Production used to come with the judgment bundled in. The person who made the work also decided it. Generation unbundled them. The making went to machines, and the deciding now needs a named owner.
Every tool skill is a bet on the current software. Judgment carries over from one model to the next.
Exposure moves with the software
One widely cited estimate rated the tasks in every US job for exposure to language models. With simple interfaces and general training, roughly 1.8% of jobs had more than half their tasks exposed. Counting the software being built around the models, the share rose to just over 46% [5]. Those are estimates of potential exposure, not observed job losses. They still show something useful: exposure belongs to tasks, and it changes whenever the tooling around the models changes.
Tool skill lives in exactly those tasks. The operation someone masters this season is a likely candidate to be packaged into the next interface. The skill still works. It just stops being scarce.
Fig 0.3Tool skill resets, judgment compoundsHold to crank · Home rewindsmodel 1 · tool 7 · judgment 10
Tool skill · wide drum, climbs fast, falls at each model changeJudgment · cable on a cone, held by a pawl nothing tripsCalendar wheel and pawl · four pegs, four model changesScales · value of the skill, 0 to 100 in model unitsA model, not a measurement. Tool skill is worth most just before a model change and least just after. Judgment keeps what it learned. MainArc's thesis, graded P, and consistent with [5]. One crank turns both drums; only what holds each weight differs.
Tool skills do pay, for now
It would be wrong to call tool skills worthless. In a study of 962 skills on a large online freelancing platform, a skill's value depended on what it could be combined with, and for most skills value peaked when paired with a skill of a different type. Technical AI skills, such as machine learning, raised wages by 21% on average, which the authors attribute to strong complementarity and rising demand [6].
Read that as a finding about combinations. The premium went to skills that combine well with others. The skills studied were technical, and the data mostly predate today's creative tools. For creative roles, MainArc's reading is that the durable half of the combination is judgment: knowing what good looks like for this brand, this audience and this decision. Pair a tool skill with that and it pays. Hire the tool skill alone and you are hiring for one release.
MainArc Research
Production is free. Judgment isn't. A tool skill is a production skill, and each release makes it cheaper. Judgment is the half of the combination that keeps its value.
Before you write a job ad, find the judgment your team already does without a title, and split every role into the work that shrinks and the work that stays.
Take one open role or one person on your team. Write which of the four judgment jobs they already do unpaid, then rewrite the role around the decisions it owns.
WhoCreative lead or founder
BudgetOne hour
InOne open role, or one person
Outroles/audit.md
Do
Pick one open role, or one person whose work has changed most since your team started generating.
Ask the four questions. Who decides the direction of generated work? Who decides where people check the work? Who vouches for generated images? Who keeps the brand recognisable at volume?
For each question, write who answers it today. A person doing it on top of their real job is doing unpaid judgment.
For each unpaid job, write one sentence: the decision, how often it comes up, and what breaks when that person is away.
Draft the first line of a new role description: the question this person answers for the team. Step 4.1 finishes it.
Try it
In the role builder, pick Art director, then Copywriter. Most of the judgment tasks for a new role are already there, sitting under a production title.
Done when
Each of the four questions has a name or a blank against it
Every unpaid judgment job is written down with what breaks without it
Prompt
I will describe one role on my team and what the person in it actually does each week. Ask me, one question at a time: who decides the creative direction of generated work, who decides where people check work before it moves on, who vouches for generated images, and who keeps the brand recognisable across high volumes of output. For each, ask whether that work appears in anyone's job description. Write the answers into roles/audit.md and mark anything done without a title as unpaid judgment. Do not suggest new roles or tools. Record what happens now.
List every task in the role and mark each one. Production is the part each release absorbs. Judgment is the part someone has to answer for.
WhoCreative lead, with the person in the role
BudgetOne hour
Inroles/audit.md
Outroles/tasks.csv
Test
Production
Judgment
Could a stranger do it from instructions?
Yes
Only with your context and standards
Does it end in a file or a decision?
A file
A decision someone signs
What would a better model change?
It would do the task
It would offer more options to decide between
What breaks if it is skipped?
Volume
Direction, trust or the brand
Do
List every recurring task in the role, with a rough share of the week. These are working estimates. Rough is fine.
Mark each task production or judgment, using the four tests.
Write each judgment task as the decision it owns, phrased as a question. Which routes ship? is a decision. Generate routes is a task.
Add up the judgment share. Below about 40%, the role is mostly the part that shrinks. That threshold is a working estimate, the same one the role builder uses.
Done when
Every task is marked production or judgment
Every judgment task is written as a decision, phrased as a question
The person who decides what gets made and what gets killed when routes cost almost nothing. The job is the direction and the kill list. Generating routes is the smallest part of it.
WhoSenior creative with a written point of view
BudgetHalf a day to define the role
Inroles/tasks.csv, recent briefs
OutA role card and a sample direction document
What the job owns
Decision
Deliverable that shows it
Which direction we take, and why
Direction document
What we will never make
Kill list
Which routes ship and which die
Selects, each with a reason
What every maker is told before starting
Briefs with intent, references, limits and a test
Playbook 07 covers how to brief like a director. This page is about the person who owns the direction.
Do
Write the role around the four decisions above. Tools go in one line at the end.
Make the kill list a required deliverable. A director who cannot say what they rejected has not directed.
Give the role sign-off on what ships and the right to stop a route. Direction without a veto is advice.
Look first for an art director or senior designer who already rejects work with reasons.
Strong or weak practitioner
Strong
Writes the direction before generating anything
Kills most routes, with a one-line reason each
Can say what the brand would never do
Briefs people and models the same way
Weak
Generates first, then picks favourites
Shows outputs, never rejections
Talks about tools and settings before the brief
Direction shifts with every new feature
What it should never be
Prompt typing with a director title. If the role's output is prompts and the person cannot defend a single rejection, it is a production job, and it expires with the next model.
Done when
The role card lists decisions first and tools last
The person who decides how work moves through generators and people: which steps stay human, what each checkpoint rejects, and what stops the line.
WhoProducer, design ops lead or a systems-minded designer
BudgetHalf a day to define the role
InYour current production process
OutA role card and a first workflow map
What the job owns
Decision
Deliverable that shows it
Which steps stay human
Workflow map with human checkpoints
What each checkpoint rejects
Gate checks a stranger could run
Which failures stop work and which only flag it
Stop rules
Who answers for each stage
Sign-off map
Playbook 06 asks which decisions remain when the work inside a product gets cheap. Playbook 12 turns decisions into checks. Inside a creative team, this role owns both.
Do
Map the current process end to end. Mark every point where a person looks at the work today, even informally.
For each checkpoint, write what it rejects. A checkpoint that never rejects anything is a formality.
Decide which failures stop the line. Wrong facts, wrong likeness and off-brand work usually should.
Measure the role by problems caught before delivery. Steps removed is a cost figure, never the goal.
Strong or weak practitioner
Strong
Starts from the decisions, then picks tools
Can say what every checkpoint rejects
Removes steps and adds checks in the same plan
Names who signs each stage
Weak
Starts from the tools, then looks for uses
Counts steps automated as success
Has checkpoints nobody can describe
Leaves nobody accountable at the end
What it should never be
Automation for its own sake. A workflow with no human checkpoint and nobody who signs is a pipeline without an owner, however clever its wiring.
Done when
The workflow map shows every human checkpoint and what it rejects
Every stage has a named signer
Prompt
Here is how work moves through my team today, step by step. Draw it as a workflow map. Mark every point where a person looks at the work, including informal ones I mention. For each of those points, ask me what it rejects and who signs it, and write my answers next to it. Flag any checkpoint where I cannot say what it rejects. Do not add or remove steps, and do not recommend tools.
The person who produces brand images without a shoot, and decides what each one may show, who may appear in it and what it says about how it was made.
WhoPhotographer, retoucher or art buyer
BudgetHalf a day to define the role
InBrand guidelines, one real product
OutA role card and a sample synthetic shoot
What the job owns
Decision
Deliverable that shows it
Light, cast, setting and what each frame is for
Shot list and art direction
Who may appear, with what consent, and which faces are off limits
Likeness rules
What each image says about how it was made
A disclosure line per image
Which frames are true enough to sign
Signed selects, with rejection reasons
Likeness and disclosure carry legal and reputational weight. Playbook 15 covers decision records and what the law text says about disclosure. Take anything that touches a real person's likeness or a regulated product claim to counsel.
Do
Write the shot list and art direction before generating anything, as you would for a real shoot.
Write likeness rules: no real person's likeness without consent, no lookalikes, and a named person who checks. If nobody can say where a face came from, it does not ship.
Decide a disclosure line for each image from its decision record. Playbook 15 has the templates.
Sign the selects. Keep the rejected frames, with one line on why each failed.
Strong or weak practitioner
Strong
Art directs light and cast before generating
Can say where every face came from
Rejects frames that misrepresent the product
Writes the disclosure line as part of the shoot
Weak
Generates until something looks finished
Cannot trace a face
Retouches the product into something it is not
Leaves disclosure to someone else
What it should never be
A frame factory. A role that ships whatever looks finished, with nobody vouching for what the image claims, makes the brand answer for pictures nobody decided.
Done when
Every image has art direction, a likeness check and a disclosure line
Every final is signed, and the rejections are kept
The person who encodes what makes a brand recognisable into a kit that strangers and models can use, and checks that output stays on brand at volume.
WhoBrand designer, copywriter or design systems lead
BudgetHalf a day to define the role
InBrand book, recent work
OutA role card and a first kit
What the job owns
Decision
Deliverable that shows it
Which two or three assets are fixed and which may flex
Rules
What right and wrong look like
References and anti-references
What the brand never says, shows or does
Negative space list
How anyone can tell an output is on brand
Eval checks a stranger could grade
Playbook 09 decides which few assets to fix. Playbook 12 makes the checks gradeable. The kit is where both land, and this role owns it.
Do
Start from the brand's most distinctive assets. Fix two or three. Leave the rest free.
Pair every reference with an anti-reference: a near miss, and the reason it fails.
Write the negative space as plainly as the rules. What a brand never does is often what makes it recognisable.
Write checks a stranger could grade. Then give the kit to a stranger and score what they make against those checks.
Strong or weak practitioner
Strong
The kit works for people who never met its author
Every rule has an anti-reference
Checks catch drift before a person notices
Rules get cut as well as added
Weak
A list of magic words
Works only for its author
No checks, so drift goes unseen
Grows a rule with every complaint
What it should never be
A file of magic words. Phrases that happened to work on one model are production tricks, and they expire with that model. The rules, references, negative space and checks are the part that lasts.
Done when
A stranger's outputs from the kit pass its own checks
Every rule has a reference and an anti-reference
Prompt
Here is our brand book and ten pieces of recent work. Ask me which two or three assets make the brand recognisable, and treat only those as fixed. For each rule, ask me for one reference and one near miss, with the reason it fails. Ask me what the brand never says, shows or does. Then draft checks a stranger could grade with a yes or no. Write it all into kit/kit.md. Do not invent rules I have not confirmed, and do not add style words as a substitute for a check.
A role description that lists tools hires for the part of the job that expires. Lead with the decisions.
WhoHiring manager
BudgetOne hour
Inroles/tasks.csv, the role card
Outroles/role.md
Two descriptions of the same roleExample
Tool-first
Expert in three named generators
Fast iteration and high output
Portfolio of AI work
Strong prompting skills
Decision-first
Decides which routes ship, and writes the kill list
Owns what every maker is briefed with
Portfolio shows rejections and the reasons
Tools: whatever the team uses, learned on the job
Do
Open with one sentence: the question this person answers for the team.
List the decisions the role owns, then the deliverable that shows each one.
Put tools in one line at the end. Assume any tool named there may change before the first review.
Write what the role must never become. It keeps the role honest when the team gets busy.
Done when
The first half of the description is decisions and deliverables
Tools take one line
Prompt
Here is a job description and my task audit. Rewrite the description so it opens with the one question this person answers for the team, then lists the decisions the role owns and the deliverable that shows each one. Move every tool name to a single line at the end. Add a line on what the role must never become. Keep my wording wherever it already names a decision. Do not add responsibilities, salaries or market claims I have not given you.
The fastest hire is often already on the team. Each traditional role carries part of a new one.
WhoCreative lead
BudgetHalf a day
Inroles/audit.md
OutA growth plan per person
From
Into
Judgment they already have
What they need next
Art director
AI creative direction
Sets direction, kills work
Judging forty routes where there used to be four
Producer
AI workflow design
Decides sign-off and scope
Writing checks a stranger could run
Retoucher
Synthetic brand photography
Knows what must never be altered
Art direction and likeness rules
Copywriter
Brand prompt kits
Owns the voice and what it never says
Turning taste into gradeable checks
Do
Match people to roles by the judgment they already show, then by interest.
Move production tasks off them before adding the new role. A new role stacked on an old one is how unpaid judgment starts.
Keep the junior reps. If seniors move into these roles and juniors lose the tasks that trained them, nobody grows into the roles next. Playbook 11 covers how.
Give each person one real deliverable in the new role within a month, signed by them.
Done when
Every internal candidate has a named role and a first deliverable
Their old production tasks have a new owner or a junior rep
When the tools change, re-run the task split. Production a new model absorbs should leave the role, and the role should not quietly fill back up with tool work.
WhoCreative lead, with each person in a new role
BudgetOne hour per role
Inroles/tasks.csv, the new tools
OutAn updated task split
Do
After each significant tool change, mark which production tasks the new tools now handle well enough.
Retire those tasks from the role, or hand them to a junior as a rep with a review.
Check the judgment share. If it falls, the role is filling with tool work again.
Check whether any decision has become a rule a stranger could follow. If so, write it into the kit or the workflow, and spend the freed time on the next hard decision.
Try it
In the role builder, switch to After the next model. The production tasks are struck out and marked absorbed. What is left is what the person is for.
Done when
Every role has a task split from after the latest change
Absorbed tasks have left the role or become a reviewed junior rep
One role, five days. By Friday it has a task split, a decisions-first description, a judgment test and a growth path.
Setup prompt
Help me rewrite one creative role around the judgment it owns.
1. Ask me which role: an open role, or one person on my team. Ask what they actually do in a typical week. Save as roles/audit.md.
2. Ask me who on the team decides the direction of generated work, who decides where people check work, who vouches for generated images, and who keeps the brand recognisable at volume. Mark any of these done without a job title as unpaid judgment.
3. List every task in the role with my rough share of the week. Help me mark each one production or judgment, and write each judgment task as the decision it owns, phrased as a question. Save as roles/tasks.csv.
4. Rewrite the role description: one opening question, the decisions it owns, the deliverable for each, tools in one line at the end, and what the role must never become. Save as roles/role.md.
5. Draft a judgment test from real work I give you, with a scoring sheet for written reasons.
6. Do not invent tasks, shares, salaries, job counts or market data. Do not name the role after a tool.
Day
Steps
What happens
D1
1.1, 1.2
Find the unpaid judgment. Split the role's tasks into production and judgment.
D2
2.1 to 3.2
Read the role page that fits. Draft its role card and deliverables.
D3
4.1
Rewrite the description around decisions. Tools in one line.
D4
4.2, 4.3
Build the judgment test from real work. Check who inside could grow into the role.
D5
5.1, 5.2
Book the re-test for the next tool change. Write review criteria by decisions owned.
A decisions-first role description, four judgment tests, a task audit and the expiry check to run after every model change.
roles/role.md
# [Role name]
The question this person answers for the team:
[One sentence. e.g. Which of the hundred routes we can make should we make?]
## Decisions this role owns
- [Decision, phrased as a question]
- [Decision]
- [Decision]
## Deliverables
| Decision | Deliverable that shows it | Signed by |
|------------------------|--------------------------------|-------------|
| [Which routes ship?] | [Selects, each with a reason] | [This role] |
| [What do we never make?] | [Kill list] | [This role] |
## How we will know it is working
- [e.g. Every delivery has a kill list with reasons]
- [e.g. A stranger can use the kit and pass its checks]
## What this role must never become
[e.g. Prompt typing. A role that only operates tools.]
## Tools
[One line. Whatever the team uses now, learned on the job.]
Judgment tests
Score the written reasons 1 to 5. The picks matter less.
Two panellists score alone, then compare.
AI creative direction
Material: one live brief, 40 generated routes.
Ask: keep three. Kill the rest with one line each.
Write the direction in five sentences.
Strong: kill reasons are specific and point the same way.
AI workflow design
Material: a real process map and its current problems.
Ask: place the human checkpoints. Say what each rejects
and which failures stop the line.
Strong: few checkpoints, each with a rule a stranger could apply.
Synthetic brand photography
Material: a product, its brand, 30 generated frames.
Ask: sign the frames you would ship. Reject the rest with
a reason. Write the likeness rule and the disclosure line.
Strong: rejects frames that misrepresent the product.
Can say where every face came from.
Brand prompt kits
Material: a brand book.
Ask: build a one-page kit: rules, references,
anti-references, negative space, checks.
Then: a stranger uses the kit. Score their outputs
against the kit's own checks.
Strong: the stranger's outputs pass.
roles/tasks.csv
role,task,kind,share_of_week,decision_owned,deliverable
Art director,Make mood boards,production,15%,,
Art director,Build comps and mockups,production,20%,,
Art director,Set the visual direction,judgment,30%,Which direction do we take?,Direction document
Art director,Approve or reject against the idea,judgment,20%,What gets killed and why?,Kill list
Art director,Brief makers,judgment,15%,What is every maker told first?,Briefs
# kind: production or judgment
# Sample rows with sample shares. Replace with your own working estimates.
Expiry check · after every model change
Run after every significant change to the tools your team uses.
Role: [name]
For each production task
[ ] Do the new tools now handle it well enough? yes / no
[ ] If yes: retired from the role, or moved to a junior as a reviewed rep
For each judgment task
[ ] Has any part become a rule a stranger could follow?
[ ] If yes: written into the kit or the workflow, time freed
Judgment share before: [ ]% after: [ ]%
If it fell, the role is filling with tool work again. Rewrite it.
The role must never become the tool's operator.
Research · Playbook 16
Research
What this playbook rests on, how to measure whether it works for you, and what it cannot yet claim.
Every core claim in this playbook, with what supports it. Grades describe the kind of evidence, not how much we like the claim. Claims graded P come from practice and are labelled as such.
0A · Experiment
4B · Field data
1C · Framework
2P · Practitioner
Claim
Grade
What supports it
Source
Most current work is in job titles that did not exist decades earlier, and new work forms where technology complements what an occupation produces.
B · Field data
Linking eight decades of new job titles to US census records, the authors estimate that the majority of current employment, roughly six jobs in ten, is in specialties introduced over that period. New titles emerged where innovations complemented occupational output, while automating innovations slowed new work [1]. Historical data that end before generative tools, so a pattern, not a forecast.
1
Freelance demand for production-heavy writing and image work fell after generative tools arrived.
B · Field data
On a large online freelancing platform, freelancers in the most affected occupations saw fewer jobs and lower earnings after a text generator was released, with similar effects after image generators [2]. On a global platform, posts for image creation fell 17% after image generators arrived [3].
2, 3
Where production demand fell, the work that remained was more complex, and demand grew only in complementary skills.
B · Field data
Automation-prone jobs that remained were more complex and paid more [3]. Demand for substitutable skills such as writing and translation fell by between 20% and 50% against trend, rose only in complementary or unaffected skill clusters, and fell for novices even there [4].
3, 4
Exposure to generative tools is a property of tasks, and it rises as software is built around the models.
C · Framework
A task-level estimate found roughly 1.8% of jobs had more than half their tasks exposed with simple interfaces, rising to just over 46% once complementary software was counted [5]. An estimate of potential exposure, not observed change.
5
A skill is worth most in combination, and technical AI skills earned a wage premium.
B · Field data
Across 962 skills on a large online freelancing platform, a skill's value depended on complementarity and, for most skills, peaked when paired with a skill of a different type. Technical AI skills raised wages by 21% on average [6]. The skills were technical, and the data mostly predate current creative tools.
6
The four new creative roles are judgment jobs wearing production titles.
P · Practitioner
MainArc's thesis and observation from practice. No study cited names, counts or prices these roles. Consistent with [1] and [4], which tie new and growing work to tasks that complement technology.
Practice
Hire and train for judgment, because tool skill expires with the next model and judgment does not.
P · Practitioner
MainArc's thesis. Consistent with [5] and [6], untested as a hiring rule. The role benchmark on the next page is designed to test it.
Practice
References
[1]Autor, D., Chin, C., Salomons, A., & Seegmiller, B. (2024). New frontiers: The origins and content of new work, 1940–2018. The Quarterly Journal of Economics, 139(3), 1399–1465. doi.org/10.1093/qje/qjae008
[2]Hui, X., Reshef, O., & Zhou, L. (2024). The short-term effects of generative artificial intelligence on employment: Evidence from an online labor market. Organization Science, 35(6), 1977–1989. doi.org/10.1287/orsc.2023.18441
[3]Demirci, O., Hannane, J., & Zhu, X. (2025). Who is AI replacing? The impact of generative AI on online freelancing platforms. Management Science, 71(10), 8097–8108. doi.org/10.1287/mnsc.2024.05420
[4]Teutloff, O., Einsiedler, J., Kässi, O., Braesemann, F., Mishkin, P., & del Rio-Chanona, R. M. (2025). Winners and losers of generative AI: Early evidence of shifts in freelancer demand. Journal of Economic Behavior & Organization, 235, 106845. doi.org/10.1016/j.jebo.2024.106845
[5]Eloundou, T., Manning, S., Mishkin, P., & Rock, D. (2024). GPTs are GPTs: Labor market impact potential of LLMs. Science, 384(6702), 1306–1308. doi.org/10.1126/science.adj0998
[6]Stephany, F., & Teutloff, O. (2024). What is the price of a skill? The value of complementarity. Research Policy, 53(1), 104898. doi.org/10.1016/j.respol.2023.104898
Grades
A controlled experiment. B large-scale field data. C an established peer-reviewed model or guideline, not a direct test of the claim. P practitioner claim from MainArc's work, untested.
Does hiring for judgment beat hiring for tool skill? Test it on your own hires and internal moves, and test again after the tools change.
Instruments
01
Judgment test
Every candidate judges the same real generated work. Two panellists score the written reasons alone, then compare.
02
Tool test
A short timed task in the current tools. Recorded for comparison, never used alone to decide.
03
Deliverable review
At six months, reviewers rate each person's deliverables blind: direction documents, kill lists, workflow maps, signed selects, kits. They also rate whether a stranger could use them.
04
Re-test after a model change
After the next significant tool change, re-run both tests. The thesis predicts judgment scores hold while tool advantages shrink.
Study design
Element
Setting
People
Every hire and internal move into one of the four roles
Comparison
Judgment test score and tool test score at hiring
Outcome
Blind deliverable ratings at six months and after a model change
Analysis
Which hiring score better predicts the outcome. Panel agreement on judgment scores with Kendall's W.
Cadence
Every hire, reviewed together once a year
Reading the results
If judgment scores predict six-month deliverables better than tool scores, keep hiring on judgment and shorten the tool test.
If tool scores predict better, look at which deliverables they predicted. The role may be more production than its card admits. Re-run the task split.
If panellists disagree a lot on judgment scores, the test or the role card is unclear. Rewrite the decisions before the next hire.
Running signals
Signal
What to watch for
Judgment share per role
Should hold or rise after each model change
Unpaid judgment found in audits
Should fall as roles are written around decisions
Rejections with reasons per delivery
A role that never rejects anything may not be judging
Note
Thresholds here are starting points to tune, not findings. A small panel shows direction and disagreement. It does not prove significance, and nothing on this page should be reported as if it did.
How this playbook was built, where it is weak, and how to cite it.
How this playbook was built
Built from labour economics research on the origins of new work and on generative AI in online freelance markets, a task-exposure estimate, research on skill complementarity, and MainArc's practice. The four role definitions, the production and judgment split and the thesis are MainArc's and graded P. No salaries, job counts or growth rates appear because none could be verified for these roles. Claims are graded on the evidence page.
Limitations
No study cited names, counts or measures the pay of the four roles. They are MainArc's observation, graded P.
The labour market evidence [2, 3, 4] comes from online freelance platforms over short windows. Salaried creative teams may move more slowly, and short-run effects may not last.
The new work research [1] uses historical census data that end before generative tools. It shows a pattern, not a forecast for creative roles.
The exposure estimate [5] measures potential exposure, not observed job change. The complementarity study [6] measured technical AI skills, not creative ones.
The role builder uses sample roles and working-estimate weights. Its 40% threshold and its After the next model view are illustrative choices.
Open questions
Do judgment tests at hiring predict deliverable quality better than tool tests, and for how long?
Do these four roles stay separate jobs, or fold back into art direction, production and brand design as the tools settle?
How can teams keep junior reps alive when these roles absorb the tasks that used to train people?
Cite this playbook
APA
MainArc Research. (n.d.). Hire for the new creative roles (Playbook No. 16, Version 1.0). MainArc. https://mainarc.xyz
BibTeX
@misc{mainarc_roles,
author = {{MainArc Research}},
title = {Hire for the new creative roles},
howpublished = {MainArc Playbook No. 16, Version 1.0},
url = {https://mainarc.xyz}
}
Corrections
Found an error, a stronger source, or a result from running the benchmark? Send it through mainarc.xyz. Corrections are made in the next version.
About this playbook
This playbook is published by MainArc, a creative intelligence lab for leadership and product teams.
We do three things. We research how AI is changing creative work through long-form reports, design teardowns, and operator essays. We design the workflows, prompt systems, review loops, and decision frameworks that teams actually run on. And we build the work the strategy points to: brand systems, interfaces, product prototypes, internal tools, and launch concepts.
One operator. One AI agent. No layers in between. Every engagement runs under strict NDA. No case studies, no client logos, no shared work.
If this research resonates with a problem your team is facing, start with an audit. One week. A clear map of where AI fits, where it doesn't, and what to build next.
This is independent research. Not financial advice. Not career advice. Not the final word on anything. It's one person's perspective shaped by building, shipping, and paying attention. Do your own thinking. Challenge the ideas. Disagree with specifics. Build on what resonates. Discard what doesn't.
House rules
What doesn't belong here: hate, trolling, bad-faith takedowns, or drive-by takes from people who've never built anything. If you disagree, show your work. Builder's critique. Everyone else just comments.