What To Do When the Model Degrades — 4 AIs Review Each Other, a 5th Rules on It

If you use Claude Code without knowing how to code, you hit a seriously infuriating problem. It keeps telling you — “I’ve got this done.”

You can’t read code, so you go run it and find an error. You throw it back, it tells you it’s fixed, you run it, still an error. Three rounds, four rounds — genuinely maddening.

Deep down you know something’s off — you have a vague sense that Claude Code can’t solve this anymore, and there’s nothing you can do about it. The project is stuck right there. It gets especially obvious once the project is fairly complex.

So what do you do? My solution is called the LLM council.

Asking a few more AIs isn’t enough

Hearing the name “LLM council,” you might think — isn’t that just asking a few more models? Ask GPT, Gemini, and Grok the same question, right?

Sounds right. But their answers — they basically all talk past each other. And when you gather those answers up and hand them to Claude Code, it still doesn’t solve the problem.

So I designed a structure that gets every one of them genuinely complete information about the problem at hand, and then makes them actually argue it out.

4 roles, 4 vendors, 4 personalities

The proposer — Google’s Gemini Its job is constructive solutions and executable paths for the situation at hand. It’s in charge of proposing.

The critic — GPT-5 On top of a proposal, its job is finding the edge cases — see, you didn’t think of this one.

I put a bit of pressure-tactic language into GPT’s prompt, because some reports online show that GPT models perform considerably better when they’re being leaned on.

The contrarian — Grok 4.2 multi-agent I ask it to find the blind spots in these proposals in the most contrarian way possible. A council has to have one role forced to push back — otherwise you get several big models patting each other on the back.

The user — Claude Code itself (Opus 4.7) This one isn’t a separately sourced model. Why?

Because the other three are outside consultants. Only Claude Code carries the memory of our earlier work — it knows the traps you’ve hit, what your system is, what your budget is. However smart those other models are, they don’t know you.

Plus Claude Code runs inside the subscription, no extra API spend.

A 5th AI to rule on it

Once the 4 roles are set, you still need a judge. I use Anthropic’s strongest right now — Claude Opus 4.7.

The nastiest trick here — before it summarizes, I pre-process the debate output coming back from the other vendors:

• Strip the speaking style down to no personality markers
• Blur each vendor into Member A / Member B / Member C / Member D

The final adjudicating model judges blind — it consolidates without knowing which member is whom. That way there’s no favoritism.

Packing the context + web research

Before the council convenes, you have to pack a complete context for every speaker:

• What kind of system the program runs on, what stack, how long it’s been running, how many users
• The actual relevant code
• Which approaches have already been tried, which possibilities are ruled out
• Constraints — it can’t just say “buy a GPU” when I have no budget for a GPU, right?

Then it also runs a round of pre-debate research through the Perplexity API. Because model training data has a cutoff date — its knowledge only goes up to some month in 2025, so it’s missing the latest exploration, knowledge, and best practices. This step is extremely important.

How the council runs

Step one — independent reviews All 4 AIs get the material at the same time, unable to see each other. Each gives a first round of feedback in role, and is required to submit a structured summary table — no writing your own little essay, or there’s no way to align them.

Step two — genuinely anonymized debate Each vendor gets to see the other three’s cards, without knowing which model is which. Because some models will notice “oh this is GPT,” and then “GPT’s right about this…” — you can’t let them fall into that. So it’s fully anonymous, they only see “what Member B said.”

There’s also a forced rule — no agreeing for the sake of agreeing. Especially GPT’s flattery mode — “you’re really a genius,” “nobody has ever come up with a solution like this…” You have to pin that down with forced anti-sycophancy prompting.

Step three — synthesis and ruling After reading everyone’s views, Opus 4.7 gives me two documents:

• The first for me — how many points all 4 roles agree on, how many are 3-to-1 majorities, how many are 2-to-2 splits, and finally a recommended plan
• The second for the record — who raised each point first, who later changed position, whether anyone just went along with the room

When to use it

• Architecture selection
• Comparing approaches
• Design review / audit
• Blind-spot checks on key decisions
• Bug review — any time Claude Code can’t solve a problem itself, I run one of these

Expanding to 7

I later found GPT-5.5 alone might not be enough as the critic. So I added a switch — two models play critic at once, two play contrarian at once, broadening the whole thinking surface.

Debate rounds can also go from two to three or four. Though beyond a point extra rounds stop meaning anything.

Letting the council audit itself — it found 5 things I hadn’t noticed

Here’s the really fun part.

Once the skill was finished, I had the council audit itself. It found roughly five things worth optimizing and iterating on — and then it went ahead and reviewed and evolved itself.

Genuinely interesting. I used a tool I wrote myself to audit the tool I wrote myself, and it told me where I was wrong.

Open source

I’ve open-sourced this skill — the complete code and all the prompts are on GitHub (repo link in the download card above).

A run on the default config costs $1 to $2.5 and takes 3 to 5 minutes. If you’re like me — can’t read code but want to get things done with AI — I think this one is worth installing.

Next time your Claude Code tells you “fixed” — don’t just believe it. Convene a council, let four vendors’ AIs take a look, and you’ll actually know where you stand.

When you’re making a big decision, don’t trust just one AI. Have four vendors’ AIs review it at once and a fifth rule on it — getting different brains onto the problem matters even more than the debate itself.

Source: EP0018_audio.mp3 · ASR model gemini-2.5-pro (chunked parallel) · full text of the 8:30 original recording

[00:00] I am so tired — but you can’t skip a day, right?

If you can’t write code and you’re using Claude Code, you’ll run into one really maddening problem. It keeps telling you — this thing, I’ve already finished it. You can’t read code, so you go run it, and you find an error. You throw it back, and it tells you it’s fixed. You run it again, still an error. Three, four rounds of this back and forth, and it’s genuinely maddening.

[00:20] In that situation, you know the thing is wrong, right. And you also have a vague sense that Claude Code can’t solve it anymore, but there’s nothing you can do. Your development is stuck dead right there. It shows up especially once a project gets fairly complex.

[00:33] So what do you do about it? Today I’m going to walk you through it. My solution is called the LLM council.

[00:38] How did this council come about? When you hear the name LLM council, you might think — isn’t that just asking a few more models? I ask GPT, Gemini, and Grok the same question. Sounds right, but the answers they give basically all talk past each other. And especially when you take all their answers, bundle them up, and hand them to Claude Code, it still doesn’t solve the problem.

[01:00] So what then? The only option is to design a setup where they genuinely get enough information about the problem at hand, and then have a real debate. Let’s look at how I designed that structure.

The Council’s 4 Roles

[01:14] So first, 4 roles, 4 vendors, 4 personalities. What are the four?

[01:18] The first one I call the Advocate. I use Google’s Gemini for this. Its whole job is to propose constructive solutions for the current situation, executable paths forward. It’s the model responsible for making proposals.

[01:32] The second one I call the Critic. On top of that proposal, using the GPT-5 model, it goes looking specifically for the edge cases — the things nobody has thought of yet. See, you didn’t think of this one, right? And that direction has some possibilities too. That’s all it does.

[01:50] And with this GPT model, I put a browbeating element into the prompt. Because as you all know — some reports online show that GPT models perform a lot better when they’re being pushed around.

[02:02] The next role down is the Nitpicker. For the Nitpicker I use Musk’s Grok 4.2 multi-agent setup. It’s told to find the blind spots in these proposals in the most contrarian way possible. It’s there to keep these models from patting each other on the back — you have to have one role in there that’s forced to disagree.

[02:23] The last role is the User. That’s not another model I go out and find — it’s Claude Code itself, the Opus 4.7 model it calls.

[02:32] Why have that model in the room? Because the other three are all outside consultants. Only this one has the memory of our earlier work on the project, right. However smart those other models are, they don’t know which potholes you’ve already hit, right. Including what your system is, what your budget is. There’s no way to get all of that over to the other models.

[02:51] And this one runs right inside the subscription — so there’s no extra API money to pay for it either.

A 5th AI Rules on It

[02:56] Once my four models are ready, I also use a Claude Opus 4.7 model as the final judge. That judge is the last summarizer.

[03:07] Here’s the ruthless part. Before it summarizes, I run a pre-processing pass over the debate results coming back from the other vendors. It strips their speaking style down to something with no personality of its own, and it blurs each vendor’s model into Member A, Member B, Member C, Member D.

[03:25] So the model making the final ruling is doing a blind review — it does its summary and roll-up without knowing which member is who. That way there’s no favoritism.

Packing the Context + Live Web Research

[03:37] Across this whole process, we pack up a complete context. Things like what kind of system our program is running on, what tech stack it uses, how long it’s been running, how many users there are. And, say, what the actual relevant code looks like — I send that over too. And which approaches I’ve already tried, which possibilities I’ve ruled out — that goes over as well. Including my constraints — you can’t have a model come back with “buy a GPU” when I don’t have the budget for a GPU, right. The constraints have to go over to it.

[04:05] And then at the end you tell it what the specific problem is that we’re solving right now. All of that gets packed into the context and handed to the council.

[04:12] On top of that, I also use the Perplexity API to do a round of pre-debate research. Because we know models themselves have a training data cutoff. The data inside the model only goes up to some month in 2025. It doesn’t have the new stuff — the new exploration, the new knowledge, the new best practices people have found for this problem.

[04:35] So before the debate opens, I use the Perplexity API to run a round of parallel research, and inject that into the context too. This part is very important.

[04:45] As for how to use the Perplexity API — go watch one of my earlier videos, the one about “using AI to learn AI.” We can have Claude Code walk us through getting a Perplexity API key.

How the Council Runs

[04:58] Moving on — once step one is done, step two is the independent review. The 4 AIs get the materials at the same time and can’t see each other. Each one gives its first round of feedback according to its own role.

[05:09] And I require them to submit a structured summary table, mandatory. Meaning they can’t just freestyle an essay in their own style — it’s more like I hand them a table and have them fill it in. Otherwise what comes back can’t be aligned, can’t be evaluated on the same terms.

[05:31] Moving on again — in the genuinely anonymized debate stage, each one gets to see the other three’s cards, but doesn’t know which model is which, right. Because you know how some models will notice “oh, this is GPT,” and then “GPT’s right about this”… you can’t let them fall into that. So it’s fully anonymous, they can only see what Member B said, and they can only judge by the content.

[05:56] And I also added a hard rule — no piling on to agree with others. Because we know, GPT especially has that suck-up mode — it’ll say “you’re a genius, nobody has ever come up with a solution like this”… it usually slides into flattery mode. So you absolutely have to pin it down with a hard anti-sycophancy prompt.

[06:18] And then at the end there’s a combined ruling. After Opus 4.7 reads everyone’s input, it gives me two things.

[06:28] The first is for me to read — how many points all 4 roles agreed on, then after the debate, the ones that went by majority rule, like how many were 3 to 1, how many were 2 to 2, and finally a recommended plan.

[06:42] And then there’s an on-the-record version — like, who raised each point first, who changed position later, right, whether anybody piled on. All of that gets filed. I don’t necessarily read it every time, but if I want to, I can go see the whole debate.

When to Use It

[06:59] So when do we use an LLM council like this?

First, we can use it for architecture choices. Then when we’re comparing approaches, when we’re doing design reviews and audits, and when we’re doing blind-spot checks on key decisions. Including bug reviews — I do it there too. Basically, any time Claude Code can’t solve a problem on its own, I run a round of this.

Scaling to 7 Members

[07:23] Later, actually, once I was running it, I found that sometimes I felt GPT 5.5 as the sole Critic wasn’t enough. So I put a switch in here that expands the council — say, two models working the Critic role at the same time, right; two models on the Nitpicker role too. That broadens the richness of the whole thinking process.

[07:47] And the number of debate rounds can go from two to three, four, even five. Although too many rounds is pointless, obviously, right.

Letting the Council Audit Itself

[07:54] One more really fun thing to tell you about — after I finished building this skill, I had the council audit the contents of its own skill. And it found roughly five places worth optimizing and iterating on. So it went and completed a review of itself and an evolution of itself. Genuinely fun.

Wrapping Up

[08:15] I’ve already open-sourced this skill, the complete code and all the prompts are up on GitHub. And you can go to the site to download this episode’s related web page and my full transcript.

[08:26] Ugh, I really have to go to sleep. Bye-bye, see you next episode tomorrow, see you tomorrow.