On this tutorial, we construct an uncertainty-aware massive language mannequin system that not solely generates solutions but in addition estimates the boldness in these solutions. We implement a three-stage reasoning pipeline during which the mannequin first produces a solution together with a self-reported confidence rating and a justification. We then introduce a self-evaluation step that permits the mannequin to critique and refine its personal response, simulating a meta-cognitive verify. If the mannequin determines that its confidence is low, we routinely set off an online analysis section that retrieves related data from dwell sources and synthesizes a extra dependable reply. By combining confidence estimation, self-reflection, and automatic analysis, we create a sensible framework for constructing extra reliable and clear AI programs that may acknowledge uncertainty and actively search higher data.
import os, json, re, textwrap, getpass, sys, warnings
from dataclasses import dataclass, subject
from typing import Elective
from openai import OpenAI
from ddgs import DDGS
from wealthy.console import Console
from wealthy.desk import Desk
from wealthy.panel import Panel
from wealthy import field
warnings.filterwarnings("ignore", class=DeprecationWarning)
def _get_api_key() -> str:
key = os.environ.get("OPENAI_API_KEY", "").strip()
if key:
return key
attempt:
from google.colab import userdata
key = userdata.get("OPENAI_API_KEY") or ""
if key.strip():
return key.strip()
besides Exception:
move
console = Console()
console.print(
"n[bold cyan]OpenAI API Key required[/bold cyan]n"
"[dim]Your key is not going to be echoed and isn't saved to disk.n"
"To skip this immediate in future runs, set the setting variable:n"
" export OPENAI_API_KEY=sk-...[/dim]n"
)
key = getpass.getpass(" Enter your OpenAI API key: ").strip()
if not key:
Console().print("[bold red]No API key supplied — exiting.[/bold red]")
sys.exit(1)
return key
OPENAI_API_KEY = _get_api_key()
MODEL = "gpt-4o-mini"
CONFIDENCE_LOW = 0.55
CONFIDENCE_MED = 0.80
shopper = OpenAI(api_key=OPENAI_API_KEY)
console = Console()
@dataclass
class LLMResponse:
query: str
reply: str
confidence: float
reasoning: str
sources: checklist[str] = subject(default_factory=checklist)
researched: bool = False
raw_json: dict = subject(default_factory=dict)We import all required libraries and configure the runtime setting for the uncertainty-aware LLM pipeline. We securely retrieve the OpenAI API key utilizing setting variables, Colab secrets and techniques, or a hidden terminal immediate. We additionally outline the LLMResponse information construction that shops the query, reply, confidence rating, reasoning, and analysis metadata used all through the system.
SYSTEM_UNCERTAINTY = """
You might be an professional AI assistant that's HONEST about what it is aware of and would not know.
For each query you MUST reply with legitimate JSON solely (no markdown, no prose outdoors JSON):
{
"reply": "<your finest reply — thorough, factual>",
"confidence": <float 0.0-1.0>,
"reasoning": "<clarify WHY you're or aren't assured; point out particular data gaps>"
}
Confidence scale:
0.90-1.00 → very excessive: well-established reality, you're sure
0.75-0.89 → excessive: sturdy data, minor uncertainty
0.55-0.74 → medium: believable however it's possible you'll be improper, could possibly be outdated
0.30-0.54 → low: vital uncertainty, reply is a finest guess
0.00-0.29 → very low: largely guessing, minimal dependable data
Be CALIBRATED — don't all the time give excessive confidence. Genuinely mirror uncertainty
about current occasions (after your data cutoff), area of interest subjects, numerical claims,
and something that adjustments over time.
""".strip()
SYSTEM_SYNTHESIS = """
You're a analysis synthesizer. Given a query, a preliminary reply,
and web-search snippets, produce an improved ultimate reply grounded within the proof.
Reply in JSON solely:
{
"reply": "<improved, evidence-grounded reply>",
"confidence": <float 0.0-1.0>,
"reasoning": "<clarify how the search proof modified or confirmed the reply>"
}
""".strip()
def query_llm_with_confidence(query: str) -> LLMResponse:
completion = shopper.chat.completions.create(
mannequin=MODEL,
temperature=0.2,
response_format={"sort": "json_object"},
messages=[
{"role": "system", "content": SYSTEM_UNCERTAINTY},
{"role": "user", "content": question},
],
)
uncooked = json.masses(completion.selections[0].message.content material)
return LLMResponse(
query=query,
reply=uncooked.get("reply", ""),
confidence=float(uncooked.get("confidence", 0.5)),
reasoning=uncooked.get("reasoning", ""),
raw_json=uncooked,
)We outline the system prompts that instruct the mannequin to report solutions together with calibrated confidence and reasoning. We then implement the query_llm_with_confidence perform that performs the primary stage of the pipeline. This stage generates the mannequin’s reply whereas forcing the output to be structured JSON containing the reply, confidence rating, and clarification.
def self_evaluate(response: LLMResponse) -> LLMResponse:
critique_prompt = f"""
Evaluate this reply and its acknowledged confidence. Test for:
1. Logical consistency
2. Whether or not the boldness matches the precise high quality of the reply
3. Any factual errors you possibly can spot
Query: {response.query}
Proposed reply: {response.reply}
Said confidence: {response.confidence}
Said reasoning: {response.reasoning}
Reply in JSON:
{{
"revised_confidence": <float — modify if the self-check adjustments your view>,
"critique": "<temporary critique of the reply high quality>",
"revised_answer": "<improved reply, or repeat authentic if high quality>"
}}
""".strip()
completion = shopper.chat.completions.create(
mannequin=MODEL,
temperature=0.1,
response_format={"sort": "json_object"},
messages=[
{"role": "system", "content": "You are a rigorous self-critic. Respond in JSON only."},
{"role": "user", "content": critique_prompt},
],
)
ev = json.masses(completion.selections[0].message.content material)
response.confidence = float(ev.get("revised_confidence", response.confidence))
response.reply = ev.get("revised_answer", response.reply)
response.reasoning += f"nn[Self-Eval Critique]: {ev.get('critique', '')}"
return response
def web_search(question: str, max_results: int = 5) -> checklist[dict]:
outcomes = DDGS().textual content(question, max_results=max_results)
return checklist(outcomes) if outcomes else []
def research_and_synthesize(response: LLMResponse) -> LLMResponse:
console.print(f" [yellow]🔍 Confidence {response.confidence:.0%} is low — triggering auto-research...[/yellow]")
snippets = web_search(response.query)
if not snippets:
console.print(" [red]No search outcomes discovered.[/red]")
return response
formatted = "nn".be a part of(
f"[{i+1}] {s.get('title','')}n{s.get('physique','')}nURL: {s.get('href','')}"
for i, s in enumerate(snippets)
)
synthesis_prompt = f"""
Query: {response.query}
Preliminary reply (low confidence): {response.reply}
Net search snippets:
{formatted}
Synthesize an improved reply utilizing the proof above.
""".strip()
completion = shopper.chat.completions.create(
mannequin=MODEL,
temperature=0.2,
response_format={"sort": "json_object"},
messages=[
{"role": "system", "content": SYSTEM_SYNTHESIS},
{"role": "user", "content": synthesis_prompt},
],
)
syn = json.masses(completion.selections[0].message.content material)
response.reply = syn.get("reply", response.reply)
response.confidence = float(syn.get("confidence", response.confidence))
response.reasoning += f"nn[Post-Research]: {syn.get('reasoning', '')}"
response.sources = [s.get("href", "") for s in snippets if s.get("href")]
response.researched = True
return responseWe implement a self-evaluation stage during which the mannequin critiques its personal reply and revises its confidence as wanted. We additionally introduce the online search functionality that retrieves dwell data utilizing DuckDuckGo. If the mannequin’s confidence is low, we synthesize the search outcomes with the preliminary reply to supply an improved response grounded in exterior proof.
def self_evaluate(response: LLMResponse) -> LLMResponse:
critique_prompt = f"""
Evaluate this reply and its acknowledged confidence. Test for:
1. Logical consistency
2. Whether or not the boldness matches the precise high quality of the reply
3. Any factual errors you possibly can spot
Query: {response.query}
Proposed reply: {response.reply}
Said confidence: {response.confidence}
Said reasoning: {response.reasoning}
Reply in JSON:
{{
"revised_confidence": <float — modify if the self-check adjustments your view>,
"critique": "<temporary critique of the reply high quality>",
"revised_answer": "<improved reply, or repeat authentic if high quality>"
}}
""".strip()
completion = shopper.chat.completions.create(
mannequin=MODEL,
temperature=0.1,
response_format={"sort": "json_object"},
messages=[
{"role": "system", "content": "You are a rigorous self-critic. Respond in JSON only."},
{"role": "user", "content": critique_prompt},
],
)
ev = json.masses(completion.selections[0].message.content material)
response.confidence = float(ev.get("revised_confidence", response.confidence))
response.reply = ev.get("revised_answer", response.reply)
response.reasoning += f"nn[Self-Eval Critique]: {ev.get('critique', '')}"
return response
def web_search(question: str, max_results: int = 5) -> checklist[dict]:
outcomes = DDGS().textual content(question, max_results=max_results)
return checklist(outcomes) if outcomes else []
def research_and_synthesize(response: LLMResponse) -> LLMResponse:
console.print(f" [yellow]🔍 Confidence {response.confidence:.0%} is low — triggering auto-research...[/yellow]")
snippets = web_search(response.query)
if not snippets:
console.print(" [red]No search outcomes discovered.[/red]")
return response
formatted = "nn".be a part of(
f"[{i+1}] {s.get('title','')}n{s.get('physique','')}nURL: {s.get('href','')}"
for i, s in enumerate(snippets)
)
synthesis_prompt = f"""
Query: {response.query}
Preliminary reply (low confidence): {response.reply}
Net search snippets:
{formatted}
Synthesize an improved reply utilizing the proof above.
""".strip()
completion = shopper.chat.completions.create(
mannequin=MODEL,
temperature=0.2,
response_format={"sort": "json_object"},
messages=[
{"role": "system", "content": SYSTEM_SYNTHESIS},
{"role": "user", "content": synthesis_prompt},
],
)
syn = json.masses(completion.selections[0].message.content material)
response.reply = syn.get("reply", response.reply)
response.confidence = float(syn.get("confidence", response.confidence))
response.reasoning += f"nn[Post-Research]: {syn.get('reasoning', '')}"
response.sources = [s.get("href", "") for s in snippets if s.get("href")]
response.researched = True
return responseWe assemble the primary reasoning pipeline that orchestrates reply technology, self-evaluation, and elective analysis. We compute visible confidence indicators and implement helper features to label their confidence ranges. We additionally constructed a formatted show system that presents the ultimate reply, reasoning, confidence meter, and sources in a clear console interface.
DEMO_QUESTIONS = [
"What is the speed of light in a vacuum?",
"What were the main causes of the 2008 global financial crisis?",
"What is the latest version of Python released in 2025?",
"What is the current population of Tokyo as of 2025?",
]
def run_comparison_table(questions: checklist[str]) -> None:
console.rule("[bold cyan]UNCERTAINTY-AWARE LLM — BATCH RUN[/bold cyan]")
outcomes = []
for i, q in enumerate(questions, 1):
console.print(f"n[bold]Query {i}/{len(questions)}:[/bold] {q}")
r = uncertainty_aware_query(q)
display_response(r)
outcomes.append(r)
console.rule("[bold cyan]SUMMARY TABLE[/bold cyan]")
tbl = Desk(field=field.ROUNDED, show_lines=True, spotlight=True)
tbl.add_column("#", fashion="dim", width=3)
tbl.add_column("Query", max_width=40)
tbl.add_column("Confidence", justify="middle", width=12)
tbl.add_column("Degree", justify="middle", width=10)
tbl.add_column("Researched", justify="middle", width=10)
for i, r in enumerate(outcomes, 1):
emoji, label = confidence_label(r.confidence)
col = "inexperienced" if r.confidence >= 0.75 else "yellow" if r.confidence >= 0.55 else "purple"
tbl.add_row(
str(i),
textwrap.shorten(r.query, 55),
f"[{col}]{r.confidence:.0%}[/{col}]",
f"{emoji} {label}",
"✅ Sure" if r.researched else "—",
)
console.print(tbl)
def interactive_mode() -> None:
console.rule("[bold cyan]INTERACTIVE MODE[/bold cyan]")
console.print(" Sort any query. Sort [bold]stop[/bold] to exit.n")
whereas True:
q = console.enter("[bold cyan]You ▶[/bold cyan] ").strip()
if q.decrease() in ("stop", "exit", "q"):
console.print("Goodbye!")
break
if not q:
proceed
resp = uncertainty_aware_query(q)
display_response(resp)
if __name__ == "__main__":
console.print(Panel(
"[bold white]Uncertainty-Conscious LLM Tutorial[/bold white]n"
"[dim]Confidence Estimation · Self-Analysis · Auto-Analysis[/dim]",
border_style="cyan",
develop=False,
))
run_comparison_table(DEMO_QUESTIONS)
console.print("n")
interactive_mode()We outline demonstration questions and implement a batch pipeline that evaluates the uncertainty-aware system throughout a number of queries. We generate a abstract desk that compares confidence ranges and whether or not analysis was triggered. Lastly, we implement an interactive mode that repeatedly accepts person questions and runs the complete uncertainty-aware reasoning workflow.
In conclusion, we designed and applied an entire uncertainty-aware reasoning pipeline for giant language fashions utilizing Python and the OpenAI API. We demonstrated how fashions can verbalize confidence, carry out inner self-evaluation, and routinely conduct analysis when uncertainty is detected. This strategy improves reliability by enabling the system to acknowledge data gaps and increase its solutions with exterior proof when wanted. By integrating these elements right into a unified workflow, we confirmed how builders can construct AI programs which are clever, calibrated, clear, and adaptive, making them much more appropriate for real-world decision-support functions.
Try the FULL Pocket book Right here. Additionally, be at liberty to comply with us on Twitter and don’t overlook to affix our 120k+ ML SubReddit and Subscribe to our E-newsletter. Wait! are you on telegram? now you possibly can be a part of us on telegram as effectively.
Jean-marc is a profitable AI enterprise govt .He leads and accelerates progress for AI powered options and began a pc imaginative and prescient firm in 2006. He’s a acknowledged speaker at AI conferences and has an MBA from Stanford.
