Selectors¶

A ModelSelector picks the best model combination for an agent against a dataset: instantiate the agent with each candidate dict, evaluate, rank by eval_fn. All selectors share one constructor surface and one entry point — select_best() — differing only in their search algorithm.

from agentopt import ModelSelector

selector = ModelSelector(
    agent=MyAgent,
    models={"planner": ["gpt-4o", "gpt-4o-mini"], "solver": ["gpt-4o-mini"]},
    eval_fn=lambda expected, actual: float(actual == expected),
    dataset=[(inp, expected), ...],
    method="auto",                # arm_elimination — strong + cheap
)
results = selector.select_best(parallel=True, max_concurrent=20)
results.print_summary()

Common parameters¶

Parameter	Type	Description
`agent`	`type`	Agent class with `__init__(self, models)` and `run(self, input_data)`. Duck-typed — no base class required.
`models`	`Dict[str, List]`	Maps node names to candidate model lists (e.g. `{"planner": ["gpt-4o", "gpt-4o-mini"]}`).
`eval_fn`	`Callable`	`(expected, actual) -> float` score (higher is better).
`dataset`	`Sequence[Tuple]`	`[(input_data, expected_answer), ...]`.
`model_prices`	`Dict`, optional	Custom pricing overrides: `{"model": {"input_price": x, "output_price": y}}` in $/MTok. Required for cost terms when `lambda_cost > 0`.
`lambda_cost`	`float`, optional	Weight on normalized per-sample cost in the combined objective. Default `0.0` (disabled). See Combined objective below.
`lambda_latency`	`float`, optional	Weight on normalized per-sample latency in the combined objective. Default `0.0` (disabled).
`tracker`	`LLMTracker`, optional	Bring your own. Defaults to a fresh `LLMTracker()` started in the constructor. Pass one in to share a cache across runs, route via a daemon (`AGENTOPT_GATEWAY_URL`), or post-process records after `select_best()` returns.

The selector calls tracker.start() in the constructor and tracker.stop() when select_best() returns or raises. Record queries on the tracker remain valid after stop(), so post-run analysis works:

tracker = LLMTracker(cache_dir="./shared_cache")
selector = ModelSelector(..., tracker=tracker)
selector.select_best()
print(tracker.get_usage())          # tracker.stop() already called; records still here

See tracker.md for the full tracker surface.

Combined objective (optional cost/latency weights)¶

By default, selectors optimize eval_fn score only (typically accuracy) and break ties with latency, then price. To trade accuracy against cost and latency in one scalar reward, pass optional weights on the constructor (or via ModelSelector(..., **kwargs)):

Parameter	Default	Effect
`lambda_cost`	`0.0`	Penalizes normalized per-sample token cost (USD from the tracker, or `model_prices`).
`lambda_latency`	`0.0`	Penalizes normalized per-sample wall-clock latency (seconds).

Omit both parameters (or leave them at 0.0) for the original accuracy-centric behavior. Set one or both when you want multi-metric selection.

Formula¶

For each datapoint, after observations are recorded:

combined = score
         - lambda_cost    * norm(cost)
         - lambda_latency * norm(latency)

score — return value of eval_fn (higher is better).
norm(·) — min–max scale to [0, 1] using running min/max over all samples seen during that selector run (updated as more combos are evaluated).
Per combination — mean of per-datapoint combined values → ModelResult.combined_objective (see results.md).

This is a linear scalarization, not Pareto exploration. Larger lambda_* penalize cost/latency more strongly relative to score.

Example¶

selector = ModelSelector(
    agent=MyAgent,
    models=models,
    eval_fn=eval_fn,
    dataset=dataset,
    method="matrix_ucb",
    lambda_cost=0.3,      # optional — omit for accuracy-only
    lambda_latency=0.2,
    model_prices={        # recommended when lambda_cost > 0
        "gpt-4o": {"input_price": 2.5, "output_price": 10.0},
        "gpt-4o-mini": {"input_price": 0.15, "output_price": 0.6},
    },
)
results = selector.select_best(parallel=True)
results.print_summary()   # ranks by combined_objective when lambdas are set

How each method uses the weights¶

Methods	During search	Final `is_best`
`matrix_ucb`, `matrix_ucb_lrf`	UCB rewards use per-cell combined objective	`_find_best` on `combined_objective`
`arm_elimination`	Elimination stats on combined per-sample objectives	same
`bayesian`	Surrogate target uses combined objective	same
`brute_force`	Does not steer which combos to try	same

After select_best(), a final pass recomputes every result’s combined_objective against the full-run normalizer so rankings are comparable.

`select_best()`¶

results = selector.select_best(
    parallel=False,        # If True, evaluate combos concurrently with asyncio
    max_concurrent=20,     # Total concurrent API-call budget across all combos
)

Returns a SelectionResults. parallel=True requires agent.run to be either async or threadsafe; the selector splits max_concurrent between outer (combos) and inner (datapoints) loops based on dataset size.

Automatic cleanup

select_best() calls tracker.stop() on return or exception — caches flush to disk, masters tear down. The tracker remains queryable; only close() (which select_best does not call) drops the remote backend's HTTP client.

Choosing a method¶

`method`	Algorithm	When to use
`"auto"` (default)	Arm elimination	Strong best-arm identification at lower search cost than brute force. Same impl as `"arm_elimination"`.
`"brute_force"`	Evaluate every combo on the full dataset	Small search space; ground-truth comparison.
`"arm_elimination"`	Successive elimination	Best-arm identification with PAC-style guarantees.
`"matrix_ucb"` / `"matrix_ucb_lrf"`	UCB exploiting cross-combo structure	Large model x datapoint matrices; `lrf` adds low-rank factorization.
`"bayesian"`	Bayesian optimization	Optional extra: `pip install "agentopt-py[bayesian]"`.

Selector Classes¶

`agentopt.model_selection.brute_force.BruteForceModelSelector` ¶

Selects the best model combination by evaluating all combinations.

Supports sequential and async-parallel evaluation via select_best().

`agentopt.model_selection.arm_elimination.ArmEliminationModelSelector` ¶

Select models via successive arm elimination.

`agentopt.model_selection.matrix_ucb.MatrixUCBModelSelector` ¶

UCB on the full combination × datapoint matrix (row means + exploration bonus).

Selection always proceeds in batches of matrix cells; only select_best(..., max_concurrent=...) matters (parallel is ignored).

observation_budget_fraction or, equivalently, sample_fraction (same meaning as in Bayesian optimization: fraction of the search budget — here, fraction of matrix cells to observe) caps evaluations. 1.0 fills the full grid; 0.1 stops after about 10% of cells. If both are passed, sample_fraction wins.

`agentopt.model_selection.matrix_ucb.MatrixUCBLRFModelSelector` ¶

Matrix UCB with low-rank factorization uncertainty (ensemble ALS), per banditeval.

Warmup and main-phase cell batches use select_best(..., max_concurrent=...). Cell budget: observation_budget_fraction or sample_fraction (see :class:MatrixUCBModelSelector). Warmup threshold: warmup_percentage or warmup_fraction — random probes until this fraction of the full grid is observed, then LRF+UCB (banditeval-style).

`agentopt.model_selection.bayesian_optimization.BayesianOptimizationModelSelector` ¶

Select models via Bayesian optimization.