Decision Assistance from an A.I. Council

A couple of weeks ago, for the umpteenth time, I found myself looking for a way to evaluate a decision from a couple of different perspectives. The thought pattern led me to the idea of building an A.I. Council, where each council member would be given a different perspective from which to evaluate a question, after which each would get a chance to provide their opinion / influence the other councillors, and they would vote on an outcome (if the decision was of the kind you could vote for).

A week or so later Andrej Karpathy posted his LLM Council repo to GitHub which was an alternative idea but offered a useful foundational approach. Thanks Andrej!

You can grab a copy of my upgraded version of the LLM Council here: github.com/ncmalan/llm-council-with-research

The Foundation

The version Andrej had built was focused on getting the perspective from different LLMs on a specific topic and then having all the LLMs blindly evaluate the other LLM outputs, after which the chairman consolidates and writes a final answer.

Infographic of the LLM Council Workflow from Andrej Karpathy

When deploying this myself, I upgraded the OpenAI Model to GPT 5.2, and the Anthropic model to Opus 4.5 but left everything else as Andrej had built it.

As a side-note: What was interesting, was that in the majority of cases I ran, the other models voted overwhelmingly in favour of GPT 5.2. This is an interesting observation that might require some more evaluation in the future:

1. Was GPT 5.2 trained on LLM Outputs from those other models and therefore they find its output "familiar"?
2. Is GPT 5.2 particularly convincing to other LLMs? And how does this play out in the future?
3. How will the balance switch as other models get released?

Upgrade 1 - Deep Research

From my personal perspective, answers need to be based in fact or at least have a reference. As such, I opted to start by adding a Deep Research step to the flow.

LLM Council upgraded to add Deep Research

In Stage 0, the Chairman asks each of the models to provide a list of questions they want researched for them to be able to better answer the question posed.

The Chairman then aggregates these questions, and sends it to a Google Deep Research Agent to go and complete a Deep Research job. This can take quite some time to complete.

The output from the Deep Research report is then provided to each of the Council Members for consideration and to incorporate into their answer.

The rest of the flow then remains the same.

Results

So far, the results have been pretty impressive. I like the clarity and detail of the answers, and it removes my manual multi-model research Q&A I've done in the past.

Next Steps

I'll be adding Telegram integration with voice note support so that I can drop questions to the council while I'm out and about.

I want to also expand the councillors more towards my initial requirement. That instead of them being given the same system prompt, I want each LLM to be prompted to evaluate the question from a different perspective. For example: I want to ask a legal question, but each council member needs to evaluate the question from a different perspective: Tax Law, Competition Law, Commercial Law, etc.

I'll keep you posted...