Suggestions

Learning suggestions are proposed graph, search, selector, and planning improvements derived from scrubbed execution traces.

They are not production truth by default. A suggestion starts as observed evidence, stays collection-scoped, and affects ranking only after promotion.

When To Use

Use suggestions when a run teaches the system something reusable:

• the same query family repeatedly selects the same target
• a retry succeeds after the first target failed
• a plan path reliably satisfies the selected target
• a runtime trace proves a data-flow edge between two tools
• an operator confirms a field mapping, context default, or enum mapping

Do not use suggestions to patch one-off failures, store raw payloads, or encode product-specific shortcuts inside the engine.

Public API

from graph_tool_call.learning import (
    apply_learning_suggestions,
    build_trace_learning_record,
    derive_learning_suggestions,
    learning_signal_map,
    merge_learning_suggestions,
    summarize_learning_state,
)

The helpers are storage-neutral. They return dictionaries that can be persisted inside a collection artifact, JSONB row, file, or any product-specific store.

Suggestion Types

Type	Generated By Core Helper	Meaning	Primary Consumer
target_preference	yes	This query family selected a target successfully	retrieval and target selector
plan_path	yes	This ordered tool path completed successfully	plan synthesis
data_flow_edge	yes	Runtime trace observed useful data flow between tools	graph expansion and planner
field_mapping	adapter/operator	A source field maps to a target field	request binding
context_default_candidate	adapter/operator	A stable context value may remove user prompts	adapter settings
enum_mapping_candidate	adapter/operator	A value-label enum mapping may be reused	enum binding UI

The first three are generated directly from derive_learning_suggestions(). The other types are part of the stable vocabulary so adapters can store reviewed signals without inventing a parallel schema.

Build A Learning Record

Create a record after a search, plan, or execute attempt. Successful records are used to derive suggestions. Failed records are still useful as promotion guards.

from graph_tool_call.learning import build_trace_learning_record

record = build_trace_learning_record(
    query="find refund-ready orders",
    collection_id="bo-dev",
    attempt_id="attempt-001",
    session_id="session-raw-value",
    selected_target="getRefundableOrders",
    llm_target="getOrderDetail",
    plan_tools=["searchOrders", "getRefundableOrders"],
    success=True,
    latency_ms=1430,
    target_selector={
        "selected_target": "getRefundableOrders",
        "overrode_llm": True,
        "reason_codes": ["shape_match"],
    },
    trace_edges=[
        {
            "source": "searchOrders",
            "target": "getRefundableOrders",
            "data_flow": {"to_field": "orderNo"},
        }
    ],
)

The returned record includes:

Field	Notes
query_family	normalized query grouping key
query_fingerprint	stable hash used to match future queries
session_id_hash	hash only, never the raw session id
target_selector	scrubbed selector diagnostics
trace_edges	scrubbed run-observed edge evidence
failure_reason	retained for failed attempts and promotion gates

Derive Suggestions

from graph_tool_call.learning import derive_learning_suggestions

new_suggestions = derive_learning_suggestions(
    record,
    history=previous_attempts,
    existing_suggestions=current_suggestions,
    promotion_policy={
        "min_success_observations": 2,
        "max_recent_failure_ratio": 0.5,
    },
)

derive_learning_suggestions() returns only suggestions created or updated by the input record. Use merge_learning_suggestions() when the adapter maintains the full collection suggestion list.

from graph_tool_call.learning import merge_learning_suggestions

collection_suggestions = merge_learning_suggestions(
    existing=current_suggestions,
    incoming=new_suggestions,
    history=[*previous_attempts, record],
)

Suggestion Shape

A target preference suggestion should look like this after persistence:

{
  "id": "target_preference:3f9d6dd00a0e50c1",
  "type": "target_preference",
  "collection_id": "bo-dev",
  "query_family": "find refund ready orders",
  "query_fingerprint": "84cc9f99591e3f3d",
  "target": "getRefundableOrders",
  "status": "suggested",
  "observations": 1,
  "prior_failure_count": 0,
  "evidence_sources": ["run"],
  "evidence": {
    "selected_target": "getRefundableOrders",
    "llm_target": "getOrderDetail",
    "target_selector": {
      "overrode_llm": true,
      "reason_codes": ["shape_match"]
    }
  }
}

Unknown fields are additive. Consumers should preserve them when forwarding or rendering suggestion records.

Status Lifecycle

Status	Ranking Impact	Meaning
suggested	none	observed, but not ready for ranking
promotable	none by default	repeated success or policy gate says it may be promoted
promoted	low-weight boost	operator or policy allowed ranking/selector impact
rejected	none	operator or policy decided not to use it

The default policy requires at least two matching successes. A recent failure ratio above 0.5 should keep a suggestion out of promotion.

Apply Learning Signals

from graph_tool_call.learning import apply_learning_suggestions

result = apply_learning_suggestions(
    query="find refund-ready orders",
    candidates=[
        {"name": "getOrderDetail", "score": 0.72},
        {"name": "getRefundableOrders", "score": 0.71},
    ],
    suggestions=collection_suggestions,
    mode="promoted",
)

Example output:

{
  "applied_count": 1,
  "mode": "promoted",
  "signals": [
    {
      "source": "learning",
      "target": "getRefundableOrders",
      "suggestion_type": "target_preference",
      "status": "promoted",
      "observations": 3,
      "score": 0.045
    }
  ]
}

Learning boost is deliberately small. It should help a close candidate move up, not override strong semantic, contract, or exact-match evidence.

Shadow Comparison

Run shadow comparison before promotion:

shadow = apply_learning_suggestions(
    query=query,
    candidates=current_candidates,
    suggestions=collection_suggestions,
    mode="shadow",
)

Compare:

Metric	Purpose
current_rank	rank without learning
shadow_rank	rank with suggested/promotable learning
rank_delta	whether the expected target moves up
selected_target_changed	whether learning would alter behavior
allowed_failure_reason	whether the original failure was expected

Only promote when the shadow result improves the desired target without causing ambiguous or unsafe behavior.

Product UI Guidance

Show operators enough information to approve or reject with confidence.

UI Field	Why It Matters
query family	shows the scope of reuse
suggestion type	explains what would change
target or plan path	shows the affected behavior
observations	proves this is not a single lucky run
prior failures	warns about unstable evidence
status	tells whether it affects ranking
evidence source	separates run, manual, and policy evidence
shadow rank delta	shows expected quality impact
promote/reject controls	keeps humans in the loop for risky collections

Safety Checklist

Before promotion:

1. The suggestion is collection-scoped.
2. Raw request and response bodies are absent.
3. Tokens, cookies, API keys, user ids, emails, and phone-like values are scrubbed.
4. The same query family has repeated success or Quality Lab approval.
5. The target or plan path is stable.
6. Recent failure ratio is acceptable.
7. The boost remains low-weight and traceable.

Troubleshooting

Symptom	Check	Likely Fix
no suggestions are created	record success, selected_target, and plan_tools	create records after successful plan/execute runs
duplicate suggestions grow quickly	suggestion id and merge policy	use merge_learning_suggestions()
promoted boost does not apply	query fingerprint and status	confirm query normalization and status=promoted
shadow improves but production does not	active mode	switch only approved collections to promoted mode
sensitive value appears	scrub test and evidence payload	reject suggestion and repair scrubbing

Validation

poetry run pytest tests/test_trace_learning.py -q
poetry run pytest tests/ -q -k "learning or target_selector"

For documentation-only changes:

cd website
npm run typecheck
npm run build

Related Pages

• Trace Learning Loop
• Scrubbing
• Shadow And Promotion
• Target Selection
• Evidence Output