negmas.tournaments

Tournament generation and management.

class negmas.tournaments.ConstructedNegInfo(run_id: int | str | None, mechanism: Mechanism, failures: dict, scenario: Scenario, real_scenario_name: str | None, config: dict[str, Any] = NOTHING)

Bases: object

Information about a negotiation after mechanism construction.

Passed to after_construction_callback in cartesian_tournament to allow inspection and modification of the mechanism before execution.

run_id

Unique identifier for this negotiation run

Type:

int | str | None

mechanism

The constructed mechanism instance

Type:

negmas.mechanisms.Mechanism

failures

Dictionary of any failures during construction

Type:

dict

scenario

The scenario being negotiated

Type:

negmas.inout.Scenario

real_scenario_name

The actual scenario name (may differ from masked name)

Type:

str | None

config

Tournament configuration dictionary (same as saved to config.yaml)

Type:

dict[str, Any]

config: dict[str, Any]

failures: dict

mechanism: Mechanism

real_scenario_name: str | None

run_id: int | str | None

scenario: Scenario

class negmas.tournaments.RunInfo(s: ~negmas.inout.Scenario, run_id: int | str, partners: tuple[type[~negmas.negotiators.negotiator.Negotiator], ...], partner_names: tuple[str] | None = None, partner_params: tuple[dict[str, ~typing.Any]] | None = None, rep: int = 0, path: ~pathlib.Path | None = None, mechanism_type: type[~negmas.mechanisms.Mechanism] = <class 'negmas.sao.mechanism.SAOMechanism'>, mechanism_params: dict[str, ~typing.Any] | None = None, full_names: bool = True, verbosity: int = 0, plot: bool = False, plot_params: dict[str, ~typing.Any] | None = None, stats: ~negmas.preferences.ops.ScenarioStats | None = None, annotation: dict[str, ~typing.Any] | None = None, private_infos: tuple[dict[str, ~typing.Any] | None] | None = None, id_reveals_type: bool = False, name_reveals_type: bool = True, mask_scenario_name: bool = True, ignore_exceptions: bool = False, scored_indices: list[int] | None = None, n_repetitions: int = 1, scenario_index: int = 0, config: dict[str, ~typing.Any] = NOTHING)

Bases: object

Information about a negotiation run before it starts.

Passed to before_start_callback in cartesian_tournament to allow inspection and logging of negotiation parameters before execution.

s

The scenario containing outcome space and utility functions

Type:

negmas.inout.Scenario

partners

Tuple of negotiator classes that will participate

Type:

tuple[type[negmas.negotiators.negotiator.Negotiator], …]

mechanism_type

The mechanism class that will be instantiated

Type:

type[negmas.mechanisms.Mechanism]

mechanism_params

Parameters that will be passed to the mechanism

Type:

dict[str, Any] | None

n_repetitions

Total number of repetitions for this scenario/partner combination

Type:

int

rep

Current repetition index (0-based)

Type:

int

scenario_index

Index of this scenario in the tournament

Type:

int

negotiation_id

Unique identifier for this negotiation run

scored_indices

Indices of negotiators whose scores will be recorded (None means all)

Type:

list[int] | None

config

Tournament configuration dictionary (same as saved to config.yaml)

Type:

dict[str, Any]

annotation: dict[str, Any] | None

config: dict[str, Any]

full_names: bool

id_reveals_type: bool

ignore_exceptions: bool

mask_scenario_name: bool

mechanism_params: dict[str, Any] | None

mechanism_type: type[Mechanism]

n_repetitions: int

name_reveals_type: bool

partner_names: tuple[str] | None

partner_params: tuple[dict[str, Any]] | None

partners: tuple[type[Negotiator], ...]

path: Path | None

plot: bool

plot_params: dict[str, Any] | None

private_infos: tuple[dict[str, Any] | None] | None

rep: int

run_id: int | str

s: Scenario

scenario_index: int

scored_indices: list[int] | None

stats: ScenarioStats | None

verbosity: int

class negmas.tournaments.SimpleTournamentResults(config: dict[str, Any] | None = None, scores: DataFrame | None = None, details: DataFrame | None = None, scores_summary: DataFrame | None = None, final_scores: DataFrame | None = None, path: Path | None = None, memory_optimization: Literal['speed', 'time', 'none', 'balanced', 'space', 'max'] = 'balanced', storage_optimization: Literal['speed', 'time', 'none', 'balanced', 'space', 'max'] = 'space', storage_format: Literal['csv', 'gzip', 'parquet'] | None = None, final_score_stat: tuple[str, str] = ('advantage', 'mean'))

Bases: object

Tournament results with optional lazy loading and memory optimization.

This class stores tournament results and supports different memory optimization levels that control whether data is kept in memory or loaded from disk on demand.

scores

All scores per negotiator (may be loaded lazily)

details

All negotiation results (may be loaded lazily)

scores_summary

All score statistics summarized per type (always in memory)

final_scores

Final rankings sorted from highest to lowest score (always in memory)

path

Location at which the logs are stored

memory_optimization

Memory optimization level (“speed”, “balanced”, “space”)

storage_optimization

Storage optimization level (“speed”, “balanced”, “space”)

storage_format

Storage format for data files (“csv”, “gzip”, “parquet”)

classmethod combine(paths: Path | Iterable[Path], recursive: bool = True, recalc_details: bool = True, recalc_scores: bool = False, must_have_details: bool = False, verbosity: int = 1, final_score_stat: tuple[str, str] = ('advantage', 'mean'), add_tournament_column: bool = True, complete_only: bool = True) → tuple[SimpleTournamentResults, list[Path]]

Combines the results of multiple tournaments stored on disk.

This method can combine tournaments saved with different storage formats (csv, gzip, parquet) and different optimization levels. It auto-detects the format of each tournament’s files.

Parameters:

• paths – Paths to look for results within

• recursive – Check children of given paths recursively

• recalc_details – Recalculate detailed results from the results/ folder

• recalc_scores – Recalculate scores from detailed negotiation results

• must_have_details – Raise an exception if detailed negotiation results cannot be found

• verbosity – Verbosity level (0=silent, 1=basic, 2+=detailed)

• final_score_stat – Tuple of (measure, statistic) for final score calculation. See cartesian_tournament for details.

• add_tournament_column – Add a column called “tournament” with tournament name in details and scores DataFrames.

• complete_only – If True, only include tournaments that completed successfully (have all required files). Incomplete tournaments are ignored.

Returns:

• SimpleTournamentResults with the combined results of all tournaments

• List of Paths that were successfully loaded

Return type:

A tuple of

Raises:

FileNotFoundError – If no valid tournament paths are found or if a needed file is missing (when must_have_details=True)

Notes

File Format Handling:

• Tournaments with different storage formats can be combined

• Each tournament’s format is auto-detected independently

• The combined result uses the default format (csv)

Required Files (when complete_only=True):

• details.csv/.csv.gz/.parquet (any format)

• all_scores.csv/.csv.gz/.parquet (any format)

• scores.csv (final scores, always CSV)

Data Reconstruction:

• If recalc_details=True, details are rebuilt from results/ folder

• If recalc_scores=True, scores are rebuilt from details

• This allows combining tournaments with different storage_optimization levels

property config: dict[str, Any] | None

The complete configuration of the tournament.

property details: DataFrame

All negotiation results (may be loaded from disk).

property final_scores: DataFrame

Final rankings sorted from highest to lowest score.

classmethod from_records(config: dict[str, Any] | None = None, scores: list[dict[str, Any]] | DataFrame | None = None, results: list[dict[str, Any]] | DataFrame | None = None, type_scores: DataFrame | None = None, final_scores: DataFrame | None = None, final_score_stat: tuple[str, str] = ('advantage', 'mean'), path: Path | None = None, stats_aggregated_metrics: dict[str, Callable[[dict[tuple[str, str], float]], float]] | None = None) → SimpleTournamentResults

Creates SimpleTournamentResults from records of results

Parameters:

• scores – The scores of negotiators in all negotiations (If not given, results can be used to calculate it).

• results – Results of all negotiations (If not given, the resulting SimpleTournamentResults object will lack details)

• type_scores – Optionally, type-scores. If not given, it will be calculated from scores

• final_scores – Optionally, final scores. If not given, final_scoer_stat will be used to calculate them

• final_score_stat – A tuple of the measure used and the statistic applied to it for calculating final score. See cartesian_tournament for more details

• path – The path in which the data for this tournament is stored.

• stats_aggregated_metrics – Optional dict mapping new metric names to callables that receive a dict of (metric, stat) tuples to values and return a combined score.

Raises:

ValueError – If no scores or results are given

Returns:

A new SimpleTournamentResults with the given data

classmethod from_result_records(path: Path, verbosity: int = 1, final_score_stat: tuple[str, str] = ('advantage', 'mean')) → SimpleTournamentResults

From result records.

Parameters:

• path – Path.

• verbosity – Verbosity.

• final_score_stat – Final score stat.

Returns:

The result.

Return type:

‘SimpleTournamentResults’

classmethod load(path: Path, must_have_details: bool = False, memory_optimization: Literal['speed', 'time', 'none', 'balanced', 'space', 'max'] = 'balanced', storage_optimization: Literal['speed', 'time', 'none', 'balanced', 'space', 'max'] = 'space', storage_format: Literal['csv', 'gzip', 'parquet'] | None = None) → SimpleTournamentResults

Loads tournament results from the given path.

This method auto-detects the storage format and reconstructs scores if needed. It supports loading from tournaments saved with any storage_optimization level.

Parameters:

• path – Path to load results from

• must_have_details – If True, raise an error if details are not found

• memory_optimization – Memory optimization level for the loaded results: - “speed”: Keep all data in memory - “balanced”: Keep details in memory, compute scores on demand - “space”: Load data from disk on demand

• storage_optimization – Storage optimization level (for metadata only, does not affect loading behavior)

• storage_format – Storage format hint (auto-detected if None from existing files)

Returns:

SimpleTournamentResults loaded from disk

Raises:

FileNotFoundError – If required files are not found

Notes

File Format Detection Priority (for details and all_scores):

1. Parquet (.parquet) - Best compression, preserves types

2. Gzip (.csv.gz) - Good compression

3. CSV (.csv) - Plain text

Data Reconstruction Priority (if scores/details not found):

1. Load from data files (details.parquet/csv.gz/csv, all_scores.parquet/csv.gz/csv)

2. Reconstruct from results/ folder JSON files

3. Reconstruct scores from details DataFrame

Small files (scores.csv, type_scores.csv) are always stored as plain CSV.

property memory_optimization: Literal['speed', 'time', 'none', 'balanced', 'space', 'max']

Memory optimization level.

property path: Path | None

Location at which the logs are stored.

save(path: Path | None = None, exist_ok: bool = True, storage_optimization: Literal['speed', 'time', 'none', 'balanced', 'space', 'max'] | None = None, storage_format: Literal['csv', 'gzip', 'parquet'] | None = None) → None

Save all results to the given path.

Parameters:

• path – Path to save results to. If None, uses self.path

• exist_ok – If True, don’t raise an error if the directory exists

• storage_optimization – Override the instance’s storage_optimization setting. - “speed”: Save all files (scores, details, scores_summary, final_scores) - “balanced”: Same as speed (cleanup happens in cartesian_tournament) - “space”: Skip saving all_scores (can be reconstructed from details)

• storage_format – Override the instance’s storage_format setting. - “csv”: Plain CSV files - “gzip”: Gzip-compressed CSV files - “parquet”: Parquet binary format

property scores: DataFrame

All scores per negotiator (may be computed or loaded from disk).

property scores_summary: DataFrame

All score statistics summarized per type.

property storage_format: Literal['csv', 'gzip', 'parquet']

Storage format for data files.

property storage_optimization: Literal['speed', 'time', 'none', 'balanced', 'space', 'max']

Storage optimization level.

class negmas.tournaments.TournamentResults(scores: DataFrame, total_scores: DataFrame, winners: list[str], winners_scores: ndarray, ttest: DataFrame | None = None, kstest: DataFrame | None = None, stats: DataFrame | None = None, agg_stats: DataFrame | None = None, score_stats: DataFrame | None = None, path: str | Path | None = None, world_stats: DataFrame | None = None, type_stats: DataFrame | None = None, agent_stats: DataFrame | None = None, params: dict[str, Any] | None = None, extra_scores: dict[str, DataFrame] | None = None)

Bases: object

TournamentResults implementation.

agent_stats: DataFrame | None = None

Some statistics about each agent

agg_stats: DataFrame | None = None

Aggregated stats per world

extra_scores: dict[str, DataFrame] | None = None

Extra scores returned from the scoring function. This can be used to have multi-dimensional scoring

kstest: DataFrame | None = None

Results of the nonparametric kstest

params: dict[str, Any] | None = None

Parameters of the tournament

path: str | Path | None = None

Path at which tournament results are stored

score_stats: DataFrame | None = None

Score statistics for different competitor types

scores: DataFrame

Scores of individual agent instantiations

stats: DataFrame | None = None

Stats of all worlds

total_scores: DataFrame

Total scores collected by competitor types

ttest: DataFrame | None = None

Results of ttest analysis of the scores

type_stats: DataFrame | None = None

Some statistics about each type

winners: list[str]

Winner type name(s) which may be a list

winners_scores: ndarray

Winner score (accumulated)

world_stats: DataFrame | None = None

Some statistics about each world run

class negmas.tournaments.WorldGenerator(*args, **kwargs)

Bases: Protocol

A callback-protocol specifying the signature of a world generator function that can be passed to tournament

Parameters:

kwargs – key-value pairs of arguments.

See also

tournament

class negmas.tournaments.WorldRunResults(world_names: list[str], log_file_names: list[str])

Bases: object

Results of a world run

extra_scores: dict[str, list[dict[str, Any]]]

The extra-scores (i.e. extra evaluation metrics). Each is a list of records

ids: list[str]

Agent IDs

log_file_names: list[str]

Log file names

names: list[str]

Agent names

scores: list[float]

Agent scores

types: list[str]

Agent type names

world_names: list[str]

World names (there can be multiple worlds for each scoring call)

negmas.tournaments.cartesian_tournament(competitors: list[type[~negmas.negotiators.negotiator.Negotiator] | str] | tuple[type[~negmas.negotiators.negotiator.Negotiator] | str, ...], scenarios: list[~negmas.inout.Scenario] | tuple[~negmas.inout.Scenario, ...], opponents: list[type[~negmas.negotiators.negotiator.Negotiator] | str] | tuple[type[~negmas.negotiators.negotiator.Negotiator] | str, ...] | None = (), opponent_params: list[dict | None] | None = None, opponent_names: list[str] | None = None, private_infos: list[None | tuple[dict, ...]] | None = None, competitor_params: list[dict | None] | None = None, competitor_names: list[str] | None = None, rotate_ufuns: bool = True, rotate_private_infos: bool = True, n_repetitions: int = 1, path: ~pathlib.Path | None = None, path_exists: ~typing.Literal['continue', 'overwrite', 'fail'] = 'continue', njobs: int = 0, mechanism_type: type[~negmas.mechanisms.Mechanism] = <class 'negmas.sao.mechanism.SAOMechanism'>, mechanism_params: dict[str, ~typing.Any] | None = None, n_steps: int | tuple[int, int] | None = 100, time_limit: float | tuple[float, float] | None = None, pend: float | tuple[float, float] = 0.0, pend_per_second: float | tuple[float, float] = 0.0, step_time_limit: float | tuple[float, float] | None = None, negotiator_time_limit: float | tuple[float, float] | None = None, hidden_time_limit: float | tuple[float, float] | None = None, external_timeout: int | None = None, plot_fraction: float = 0.0, plot_params: dict[str, ~typing.Any] | None = None, verbosity: int = 1, self_play: bool = True, randomize_runs: bool = True, sort_runs: bool = False, save_every: int = 0, save_stats: bool = True, save_scenario_figs: bool = True, recalculate_stats: bool = False, image_format: str = 'webp', opponent_modeling_metrics: tuple[~typing.Literal['kendall', 'kendal_optimality', 'ndcg', 'euclidean'] | ~typing.Callable[[~numpy.ndarray[tuple[~typing.Any, ...], ~numpy.dtype[~numpy.floating[~typing.Any]]], ~numpy.ndarray[tuple[~typing.Any, ...], ~numpy.dtype[~numpy.floating[~typing.Any]]]], float], ...] = (), distribute_opponent_modeling_scores: bool = True, raw_aggregated_metrics: dict[str, ~typing.Callable[[dict[str, float]], float]] | None = None, stats_aggregated_metrics: dict[str, ~typing.Callable[[dict[tuple[str, str], float]], float]] | None = None, final_score: tuple[str, str] = ('advantage', 'mean'), id_reveals_type: bool = False, name_reveals_type: bool = True, shorten_names: bool | None = None, raise_exceptions: bool = True, mask_scenario_names: bool = True, only_failures_on_self_play: bool = False, ignore_discount: bool = False, ignore_reserved: bool = False, normalize_ufuns: bool = True, reserved_value_eps: float = 0.0, storage_optimization: ~typing.Literal['speed', 'time', 'none', 'balanced', 'space', 'max'] = 'space', memory_optimization: ~typing.Literal['speed', 'time', 'none', 'balanced', 'space', 'max'] = 'balanced', storage_format: ~typing.Literal['csv', 'gzip', 'parquet'] | None = None, save_negotiations_as_folders: bool = False, python_class_identifier='__python_class__', before_start_callback: ~typing.Callable[[~negmas.tournaments.neg.simple.cartesian.RunInfo], None] | None = None, progress_callback: ~typing.Callable[[str, int, int], None] | ~typing.Callable[[str, int, int, dict[str, ~typing.Any] | None], None] | None = None, neg_start_callback: ~typing.Callable[[str | int, ~negmas.sao.common.SAOState], None] | None = None, after_construction_callback: ~typing.Callable[[~negmas.tournaments.neg.simple.cartesian.ConstructedNegInfo], None] | None = None, neg_progress_callback: ~typing.Callable[[str | int, ~negmas.sao.common.SAOState], None] | None = None, neg_end_callback: ~typing.Callable[[str | int, ~negmas.sao.common.SAOState], None] | None = None, after_end_callback: ~typing.Callable[[dict[str, ~typing.Any]], None] | ~typing.Callable[[dict[str, ~typing.Any], dict[str, ~typing.Any]], None] | None = None, metadata: dict[str, ~typing.Any] | None = None) → SimpleTournamentResults

Run a Cartesian tournament where negotiators compete across multiple scenarios.

This function runs negotiations between all combinations of competitors across all scenarios, optionally with rotated utility functions. When opponents are provided, competitors only play against opponents (not each other) and only competitor scores are recorded.

Parameters:

• competitors – Negotiator types or class names to compete in the tournament.

• scenarios – Negotiation scenarios, each with an outcome space and utility functions.

• opponents – Optional negotiator types to use as opponents. If provided, competitors only play against opponents (not each other) and only competitors are scored. Competitors will be tested in both first and last positions to evaluate different roles (e.g., buyer vs seller).

• opponent_params – Parameters for initializing opponents (one dict per opponent type).

• opponent_names – Optional list of custom names for opponents. If provided, must have the same length as opponents and contain unique names. If not provided, names are generated from class names using shortest_unique_names().

• private_infos – Private information passed to negotiators via their private_info attribute. Must be a list of tuples, one tuple per scenario.

• competitor_params – Parameters for initializing competitors (one dict per competitor type).

• competitor_names – Optional list of custom names for competitors. If provided, must have the same length as competitors and contain unique names. If not provided, names are generated from class names using shortest_unique_names().

• rotate_ufuns – If True, utility functions are rotated across negotiator positions. For bilateral negotiations, this creates scenarios with reversed preferences. Not recommended when using explicit opponents as roles become ambiguous.

• rotate_private_infos – If True and rotate_ufuns is True, rotate private information with ufuns.

• n_repetitions – Number of times to repeat each scenario/partner combination.

• path – Directory path to save tournament results. If None, results are not saved to disk.

• path_exists – Controls behavior when path already exists (default: “continue”): - “continue”: Resume incomplete tournament by skipping completed negotiations - “overwrite”: Delete existing tournament and start fresh - “fail”: Raise FileExistsError if tournament directory exists

• njobs – Parallelization level. -1 for serial execution (good for debugging), 0 for all available cores, positive integer for specific number of processes.

• mechanism_type – The negotiation protocol/mechanism class to use (default: SAOMechanism).

• mechanism_params – Additional parameters passed to the mechanism constructor.

• n_steps – Maximum rounds per negotiation. Can be int, (min, max) tuple for random sampling, or None for unlimited.

• time_limit – Maximum seconds per negotiation. Can be float, (min, max) tuple, or None.

• pend – Probability of ending negotiation each step. Can be float, (min, max) tuple, or 0.0.

• pend_per_second – Probability of ending negotiation each second. Can be float, (min, max) tuple, or 0.0.

• step_time_limit – Maximum seconds per negotiation step. Can be float, (min, max) tuple, or None.

• negotiator_time_limit – Maximum total seconds for all actions by each negotiator.

• hidden_time_limit – Time limit not revealed to negotiators.

• external_timeout – Timeout in seconds for receiving results from parallel negotiations.

• plot_fraction – Fraction of negotiations to plot (0.0 to 1.0). Only used if path is provided.

• plot_params – Parameters passed to plotting functions.

• verbosity – Logging level (0 for silent, higher for more verbose).

• self_play – If True, allow negotiations where all parties are the same type.

• only_failures_on_self_play – If True, only record self-play negotiations that fail to reach agreement.

• randomize_runs – If True, run negotiations in random order instead of sequentially.

• sort_runs – If True, sort runs by scenario size before execution.

• save_every – Save results to disk after this many negotiations (0 to disable periodic saving).

• save_stats – If True, calculate optimality statistics (Pareto, Nash, Kalai-Smorodinsky, etc.).

• save_scenario_figs – If True, save visualizations of scenarios in utility space.

• recalculate_stats – If True, always recalculate stats even if loaded from disk (old behavior). If False (default), load stats/info/plots from disk when available and use Scenario.rotate_ufuns() to create rotated versions efficiently. This dramatically speeds up tournaments by avoiding redundant stat calculations. When rotate_ufuns=False, stats are never recalculated if available in the scenario folder.

• image_format – Format for saving figures. Supported formats: ‘webp’, ‘png’, ‘jpg’, ‘jpeg’, ‘svg’, ‘pdf’. Default is ‘webp’. Applies to both scenario figures and run plots.

• opponent_modeling_metrics –

  Tuple of utility function comparison methods for evaluating how well each negotiator models their opponent’s preferences. For each metric specified, compares the negotiator’s opponent_ufun attribute (if available) with the actual opponent utility function. Results are added as opp_ columns in the scores DataFrame.

  Valid values include: - ‘kendall_optimality’: Kendall tau correlation (-1 to 1) - ‘ordinal_optimality’: Ordinal ranking similarity (0 to 1) - ‘cardinal_optimality’: Cardinal value similarity (0 to 1) - ‘utility_optimality’: Direct utility comparison (0 to 1) - ‘pareto_optimality’: Pareto efficiency measure (0 to 1) - ‘nash_optimality’: Nash bargaining optimality (0 to 1) - ‘kalai_optimality’: Kalai-Smorodinsky optimality (0 to 1) - ‘max_welfare_optimality’: Maximum welfare optimality (0 to 1)

  These can be used as final scores: final_score=(‘opp_kendall_optimality’, ‘mean’)

  Example:

  results = cartesian_tournament(
      competitors=[MyNegotiator, RandomNegotiator],
      scenarios=scenarios,
      opponent_modeling_metrics=(
          "kendall_optimality",
          "euclidean_optimality",
      ),
  )
  # Access opponent modeling scores
  print(
      results.scores[
          [
              "strategy",
              "opp_kendall_optimality",
              "opp_euclidean_optimality",
          ]
      ]
  )
  

• raw_aggregated_metrics –

  Optional dict mapping custom metric names to aggregation functions. Each function receives a dict of {metric_name: value} containing all per-negotiation metrics for a single negotiator (e.g., ‘advantage’, ‘utility’, ‘nash_optimality’, etc.) and returns a combined score. Results are added as new columns in the scores DataFrame.

  Useful for creating weighted combinations of existing metrics.

  Example:

  results = cartesian_tournament(
      competitors=[...],
      scenarios=[...],
      raw_aggregated_metrics={
          "combined": lambda d: d.get("advantage", 0)
          * 0.5
          + d.get("utility", 0) * 0.5,
          "risk_adjusted": lambda d: d.get("utility", 0)
          - 0.1 * d.get("partner_welfare", 0),
      },
  )
  # Use in final score
  # final_score=('combined', 'mean')
  

• stats_aggregated_metrics –

  Optional dict mapping custom metric names to aggregation functions that operate on summary statistics across all negotiations. Each function receives a dict of {(metric_name, stat_name): value} where stat_name can be ‘mean’, ‘std’, ‘min’, ‘max’, ‘25%’, ‘50%’, ‘75%’, ‘count’. Results are added as (metric_name, ‘value’) columns in scores_summary.

  This is particularly useful for creating custom final scores that combine multiple statistics in ways not possible with standard aggregations.

  Example:

  results = cartesian_tournament(
      competitors=[...],
      scenarios=[...],
      stats_aggregated_metrics={
          'risk_adjusted_score': lambda d: (
              d.get(('advantage', 'mean'), 0) - 0.5 * d.get(('advantage', 'std'), 0)
          ),
          'weighted_final': lambda d: (
              d.get(('advantage', 'mean'), 0) * 0.7 +
              d.get(('utility', 'mean'), 0) * 0.3
          ),
      },
      # Use custom aggregation as final score
      final_score=('weighted_final', 'value'),
  )
  

• final_score – Tuple of (metric, statistic) for ranking. Metric can be ‘advantage’, ‘utility’, ‘partner_welfare’, ‘welfare’, or any calculated statistic. Statistic can be ‘mean’, ‘median’, ‘min’, ‘max’, or ‘std’. Default: (‘advantage’, ‘mean’).

• id_reveals_type – If True, negotiator IDs reveal their type (for analysis).

• name_reveals_type – If True, negotiator names reveal their type.

• shorten_names – Deprecated. Use competitor_names and opponent_names instead. This parameter is ignored and will be removed in a future version.

• raise_exceptions – If True, exceptions from negotiators/mechanisms stop the tournament. If False, exceptions are logged but tournament continues.

• mask_scenario_names – If True, mask scenario names from negotiators.

• only_failures_on_self_play – If True, only record self-play runs that fail to reach agreement.

• ignore_discount – If True, ignore discounting in utility functions (use base ufun).

• ignore_reserved – If True, ignore reserved values in utility functions.

• reserved_value_eps – Epsilon value used to correct problematic reserved values (default: 0.0). When a utility function has None, inf, -inf, or NaN reserved value, it will be corrected to ufun.min() - reserved_value_eps. A warning is emitted for each corrected utility function.

• normalize_ufuns – If True (default), all utility functions are normalized to [0, 1] range before negotiation. Normalization is applied independently to each ufun, guaranteeing that the best outcome has utility 1.0 and the worst has utility 0.0. This normalization happens BEFORE scenarios are saved to disk, so all saved scenarios, statistics, and figures reflect the normalized utilities.

• storage_optimization –

  Controls disk space usage for tournament results (default: “space”): - “speed”/”time”/”none”: Keep all files (results/, all_scores.csv, details.csv, etc.) - “balanced”: Remove results/ folder after details.csv is created - “space”/”max”: Remove both results/ folder AND all_scores.csv (default)

  (scores can be reconstructed from details.csv)

• memory_optimization –

  Controls RAM usage for returned SimpleTournamentResults (default: “balanced”): - “speed”/”time”/”none”: Keep all DataFrames in memory - “balanced”: Keep details + final_scores + scores_summary in memory,

  compute scores on demand then cache (default)

  • ”space”/”max”: Keep only final_scores + scores_summary in memory,

    load details/scores from disk when needed

  Note: If path is None, memory_optimization is ignored (everything kept in memory)

• storage_format – Storage format for large data files (all_scores, details): - “csv”: Plain CSV files (human-readable, larger size) - “gzip”: Gzip-compressed CSV files (good compression, human-readable when decompressed) - “parquet”: Parquet binary format (best compression, preserves types, fastest) - None: Auto-select based on storage_optimization (default; csv for speed, gzip for balanced, parquet for space) Note: Small files (scores.csv, type_scores.csv) are always CSV regardless of this setting.

• save_negotiations_as_folders – If True, save each negotiation as a folder containing trace, agreement_stats (optimality measures), config, and metadata files. If False (default), save as single trace files for compactness.

• python_class_identifier – Function to convert classes to string identifiers.

• before_start_callback – Optional callback invoked before each negotiation starts. Receives a RunInfo object with all negotiation parameters. Useful for logging, monitoring, or custom setup. Exceptions are caught and logged (if verbosity > 0) but don’t stop the tournament.

• after_construction_callback – Optional callback invoked after mechanism construction but before negotiation starts. Receives a ConstructedNegInfo object with the constructed mechanism and scenario details. Useful for inspecting or modifying the mechanism before negotiation. Exceptions are caught and logged (if verbosity > 0) but don’t stop the tournament.

• after_end_callback – Optional callback invoked after each negotiation completes. Receives the complete negotiation record dictionary with agreement, utilities, and all result metadata. Useful for custom analysis or logging. Exceptions are caught and logged (if verbosity > 0) but don’t stop the tournament.

• progress_callback – Optional callback invoked during tournament setup to report progress. Receives (message: str, current: int, total: int) where message describes the current phase, current is the progress index, and total is the expected count. Useful for showing setup progress in UIs before negotiations start. Called during competitor validation, scenario processing, and run configuration building.

• neg_start_callback –

  Optional callback invoked at the start of each negotiation. Receives (run_id: int | str, state: SAOState) where run_id uniquely identifies the negotiation and state is the initial mechanism state. Useful for monitoring negotiation progress. Exceptions are caught and logged but don’t stop the tournament.

  Parallel Execution Note: Callbacks do NOT need to be defined at module level. You can use local functions, lambdas, or closures. However, when running in parallel mode (njobs > 0), callbacks are serialized using cloudpickle and executed in separate worker processes. This means:

  • Callbacks can capture local variables from enclosing scopes (closures work)

  • IMPORTANT: Modifications to captured variables (lists, dicts, etc.) will NOT be visible in the parent process. Each worker gets a copy of the closure.

  • To collect results from parallel callbacks, use side effects that persist across processes (e.g., write to files, database, use multiprocessing.Manager)

  • Callbacks must be picklable (avoid unpicklable objects like file handles, locks)

• neg_progress_callback –

  Optional callback invoked after each step of each negotiation. Receives (run_id: int | str, state: SAOState) where state contains current step number, offers, and agreement status. Useful for real-time monitoring of negotiation progress. Exceptions are caught and logged but don’t stop the tournament.

  See neg_start_callback documentation for parallel execution requirements.

• neg_end_callback –

  Optional callback invoked at the completion of each negotiation. Receives (run_id: int | str, state: SAOState) where state contains the final agreement, step count, and termination reason. Useful for analyzing negotiation outcomes in real-time. Exceptions are caught and logged but don’t stop the tournament.

  See neg_start_callback documentation for parallel execution requirements.

• metadata – Optional dictionary of metadata to include in tournament results.

Returns:

SimpleTournamentResults containing scores, detailed results, score summaries, and final rankings.

Notes

• In explicit opponent mode (opponents provided), competitors appear at the first position. Use rotate_ufuns=True to test performance in different roles.

• Use njobs=-1 for debugging to run serially and see full tracebacks.

Examples

Normal tournament between two negotiators: ```python results = cartesian_tournament(

competitors=[MyNegotiator, TheirNegotiator], scenarios=[scenario1, scenario2], n_steps=100, path=Path(“results/”),

)

Testing a negotiator against fixed opponents: ```python results = cartesian_tournament(

competitors=[MyNegotiator], opponents=[RandomNegotiator, AspirationNegotiator], scenarios=[scenario1], rotate_ufuns=False, # Keep roles fixed

)

Using callbacks for monitoring: ```python def log_start(info: RunInfo):

print(f”Starting negotiation {info.rep} with {info.partners}”)

def log_end(record: dict):

print(f”Ended with agreement: {record[‘agreement’]}”)

results = cartesian_tournament(

competitors=[MyNegotiator, TheirNegotiator], scenarios=[scenario1], before_start_callback=log_start, after_end_callback=log_end,

)

Using per-negotiation callbacks (local closures work in both serial and parallel): ```python # Serial mode - can modify local variables (closure copy) start_times = []

def track_start(run_id, state):

print(f”Negotiation {run_id} started at step {state.step}”) start_times.append(run_id) # Won’t work in parallel mode!

results = cartesian_tournament(

competitors=[MyNegotiator, TheirNegotiator], scenarios=[scenario1], neg_start_callback=track_start, njobs=-1, # Serial mode

)

# Parallel mode - use files or database for side effects from pathlib import Path

log_dir = Path(“negotiation_logs”) log_dir.mkdir(exist_ok=True)

def track_start_parallel(run_id, state):

# Write to file - works across processes msg = f”Started at step {state.step}” (log_dir / f”start_{run_id}.log”).write_text(msg)

results = cartesian_tournament(

competitors=[MyNegotiator, TheirNegotiator], scenarios=[scenario1], neg_start_callback=track_start_parallel, njobs=4, # Parallel mode - callbacks still work!

)

Resuming an interrupted tournament: ```python # Start a tournament results = cartesian_tournament(

competitors=[MyNegotiator, TheirNegotiator], scenarios=scenarios, n_repetitions=100, path=Path(“results/”), path_exists=”continue”, # Resume if interrupted (default)

)

# If interrupted and restarted, only remaining negotiations will run # Use path_exists=”overwrite” to delete and restart from scratch # Use path_exists=”fail” to raise error if directory exists ```

negmas.tournaments.combine_tournament_results(sources: Iterable[str | Path], dest: str | Path | None = None, verbose=False, max_sources: int | None = None) → DataFrame

Combines results of several tournament runs in the destination path.

negmas.tournaments.combine_tournament_stats(sources: Iterable[str | Path], dest: str | Path | None = None, verbose=False, max_sources: int | None = None) → DataFrame

Combines statistical results of several tournament runs in the destination path.

negmas.tournaments.combine_tournaments(sources: Iterable[str | Path], dest: str | Path, verbose=False) → tuple[int, int]

Combines contents of several tournament runs in the destination path allowing for continuation of the tournament

Parameters:

• sources – The sources of tournaments in the filesystem

• dest – where to store the combined tournament.

Returns:

Tuple[int, int] The number of base configs and assigned configs combined

negmas.tournaments.continue_cartesian_tournament(path: Path | str, verbosity: int | None = None, njobs: int | None = None, before_start_callback: Callable[[RunInfo], None] | None = None, progress_callback: Callable[[str, int, int], None] | Callable[[str, int, int, dict[str, Any] | None], None] | None = None, neg_start_callback: Callable[[str | int, SAOState], None] | None = None, after_construction_callback: Callable[[ConstructedNegInfo], None] | None = None, neg_progress_callback: Callable[[str | int, SAOState], None] | None = None, neg_end_callback: Callable[[str | int, SAOState], None] | None = None, after_end_callback: Callable[[dict[str, Any]], None] | Callable[[dict[str, Any], dict[str, Any]], None] | None = None) → SimpleTournamentResults | None

Continue or load a cartesian tournament from a saved path.

This is a convenience function that: 1. Checks if the path contains a valid tournament (config.yaml and scenarios/) 2. If incomplete, continues the tournament by running remaining negotiations 3. If complete, loads and returns the existing results 4. If invalid, returns None

Parameters:

• path – Directory path containing the tournament (must have config.yaml and scenarios/)

• verbosity – Optional verbosity level to override the one in config.yaml

• njobs – Optional parallelization level to override the one in config.yaml

• before_start_callback – Called before each negotiation run starts with RunInfo.

• progress_callback – Called periodically during tournament execution with progress info.

• neg_start_callback – Called when a negotiation starts with (run_id, initial_state).

• after_construction_callback – Called after negotiation is constructed with ConstructedNegInfo.

• neg_progress_callback – Called during negotiation with (run_id, current_state).

• neg_end_callback – Called when a negotiation ends with (run_id, final_state).

• after_end_callback – Called after each negotiation run completes with (RunInfo, Mechanism, results_dict).

Returns:

SimpleTournamentResults if tournament is valid, None otherwise

Examples

```python # Start a tournament results = cartesian_tournament(

competitors=[MyNegotiator, TheirNegotiator], scenarios=scenarios, n_repetitions=100, path=Path(“my_tournament/”),

)

# Later, continue or load it results = continue_cartesian_tournament(Path(“my_tournament/”)) if results is None:

print(“Invalid tournament path”)

```

```python # Continue with different verbosity/parallelization results = continue_cartesian_tournament(

Path(“my_tournament/”), verbosity=2, njobs=-1, # Serial execution for debugging

)

```python # Continue with callbacks def on_neg_end(run_id, state):

print(f”Negotiation {run_id} ended: agreement={state.agreement}”)

results = continue_cartesian_tournament(

Path(“my_tournament/”), neg_end_callback=on_neg_end

)

negmas.tournaments.create_tournament(competitors: Sequence[str | type[Agent]], config_generator: ConfigGenerator, config_assigner: ConfigAssigner, world_generator: WorldGenerator, score_calculator: Callable[[list[World], dict[str, Any], bool], WorldRunResults], competitor_params: Sequence[dict[str, Any]] | None = None, n_competitors_per_world: int | None = None, round_robin: bool = True, agent_names_reveal_type=False, n_agents_per_competitor=1, n_configs: int = 10, max_worlds_per_config: int | None = 100, n_runs_per_world: int = 5, max_n_configs: int | None = None, n_runs_per_config: int | None = None, base_tournament_path: Path | str | None = None, total_timeout: int | None = None, parallelism='parallel', scheduler_ip: str | None = None, scheduler_port: str | None = None, non_competitors: tuple[str | Any, ...] | None = None, non_competitor_params: tuple[dict[str, Any], ...] | None = None, dynamic_non_competitors: tuple[str | Any, ...] | None = None, dynamic_non_competitor_params: tuple[dict[str, Any], ...] | None = None, exclude_competitors_from_reassignment: bool = True, name: str | None = None, verbose: bool = False, compact: bool = False, save_video_fraction: float = 0.0, forced_logs_fraction: float = 0.0, video_params=None, video_saver=None, **kwargs) → Path

Creates a tournament

Parameters:

• name – Tournament name

• config_generator – Used to generate unique configs that will be used to evaluate competitors

• config_assigner – Used to generate assignments of competitors to the configs created by the config_generator

• world_generator – A functions to generate worlds for the tournament that follows the assignments made by the config_assigner

• score_calculator – A function for calculating the score of all agents in a world After it finishes running. The second parameter is a dict describing any scoring context that may have been added by the world config generator or assigneer. The third parameter is a boolean specifying whether this is a dry_run. For dry runs, scores are not expected but names and types should exist in the returned WorldRunResults.

• competitors – A list of class names for the competitors

• competitor_params – A list of competitor parameters (used to initialize the competitors).

• n_competitors_per_world –

  The number of competitors allowed in every world. It must be >= 1 and <= len(competitors) or None.

  • If None or len(competitors), then all competitors will exist in every world.

  • If 1, then each world will have one competitor

• round_robin – Only effective if 1 < n_competitors_per_world < len(competitors). if True, all combinations will be tried otherwise n_competitors_per_world must divide len(competitors) and every competitor appears only in one set.

• agent_names_reveal_type – If true then the type of an agent should be readable in its name (most likely at its beginning).

• n_configs – The number of different world configs (up to competitor assignment) to be generated.

• max_worlds_per_config – The maximum number of worlds to run per config. If None, then all possible assignments of competitors within each config will be tried (all permutations).

• n_runs_per_world – Number of runs per world. All of these world runs will have identical competitor assignment and identical world configuration.

• n_agents_per_competitor – The number of agents of each competing type to be instantiated in the world.

• max_n_configs – [Depricated] The number of configs to use (it is replaced by separately setting n_config and max_worlds_per_config )

• n_runs_per_config – [Depricated] The number of runs (simulation) for every config. It is replaced by n_runs_per_world

• total_timeout – Total timeout for the complete process

• base_tournament_path – Path at which to store all results. A new folder with the name of the tournament will be created at this path. A scores.csv file will keep the scores and logs folder will keep detailed logs

• parallelism – Type of parallelism. Can be ‘serial’ for serial, ‘parallel’ for parallel and ‘distributed’ for distributed! For parallel, you can add the fraction of CPUs to use after a colon (e.g. parallel:0.5 to use half of the CPU in the machine). By defaults parallel uses all CPUs in the machine

• scheduler_port – Port of the dask scheduler if parallelism is dask, dist, or distributed

• scheduler_ip – IP Address of the dask scheduler if parallelism is dask, dist, or distributed

• non_competitors – A list of agent types that will not be competing but will still exist in the world.

• non_competitor_params – paramters of non competitor agents

• dynamic_non_competitors – A list of non-competing agents that are assigned to the simulation dynamically during the creation of the final assignment instead when the configuration is created

• dynamic_non_competitor_params – paramters of dynamic non competitor agents

• exclude_competitors_from_reassignment – If true, copmetitors are not included in the reassignment even if they exist in dynamic_non_competitors

• verbose – Verbosity

• compact – If true, compact logs will be created and effort will be made to reduce the memory footprint

• save_video_fraction – The fraction of simulations for which to save videos

• forced_logs_fraction – The fraction of simulations for which to always save logs. Notice that this has no effect except if no logs were to be saved otherwise (i.e. no_logs is passed as True)

• video_params – The parameters to pass to the video saving function

• video_saver – The parameters to pass to the video saving function after the world

• kwargs – Arguments to pass to the config_generator function

Returns:

The path at which tournament configs are stored

negmas.tournaments.evaluate_tournament(tournament_path: str | Path | None, scores: DataFrame | None = None, stats: DataFrame | None = None, world_stats: DataFrame | None = None, type_stats: DataFrame | None = None, agent_stats: DataFrame | None = None, metric: str | Callable[[DataFrame], float] = 'mean', verbose: bool = False, recursive: bool = True, extra_scores_to_use: str | None = None, compile: bool = True) → TournamentResults

Evaluates the results of a tournament

Parameters:

• tournament_path – Path to save the results to. If scores is not given, it is also used as the source of scores. Pass None to avoid saving the results to disk.

• scores – Optionally the scores of all agents in all world runs. If not given they will be read from the file scores.csv in tournament_path

• stats – Optionally the stats of all world runs. If not given they will be read from the file stats.csv in tournament_path

• world_stats – Optionally the aggregate stats collected in WorldSetRunStats for each world set

• type_stats – Optionally the aggregate stats collected in AgentStats for each agent type

• agent_stats – Optionally the aggregate stats collected in AgentStats for each agent instance

• metric – The metric used for evaluation. Possibilities are: mean, median, std, var, sum, truncated_mean or a callable that receives a pandas data-frame and returns a float.

• verbose – If true, the winners will be printed

• recursive – If true, ALL scores.csv files in all subdirectories of the given tournament_path will be combined

• extra_scores_to_use – The type of extra-scores to use. If None normal scores will be used. Only effective if scores is None.

• compile – Takes effect only if tournament_path is not None. If true, the results will be recompiled from individual world results. This is accurate but slow. If false, it will be assumed that all results are already compiled.

• independent_test (#) – True if you want an independent t-test

Returns:

negmas.tournaments.process_world_run(run_id: str, results: WorldRunResults | None, tournament_name: str) → tuple[list[dict[str, Any]], dict[str, list[dict[str, Any]]]]

Generates a data-frame with the results of this world run

Parameters:

• run_id – The ID of this run (should be unique per tournament)

• results – Results of the world run

• tournament_name – tournament name

Returns:

• A list of records containing scores

• A dict mapping extra-score types to lists of records for this type.

Return type:

A tuple of two items

Remarks:

The score calculator returns a WorldRunResults object which must contain a scores element used for evaluating the agents. It can also return extra_scores that can be used to save additional information about agent performance. These are optional and the second output of this function will be the processed version of these extra scores if any.

negmas.tournaments.run_negotiation(s: ~negmas.inout.Scenario, partners: tuple[type[~negmas.negotiators.negotiator.Negotiator], ...], run_id: int | str, partner_names: tuple[str] | None = None, partner_params: tuple[dict[str, ~typing.Any]] | None = None, rep: int = 0, path: ~pathlib.Path | None = None, mechanism_type: type[~negmas.mechanisms.Mechanism] = <class 'negmas.sao.mechanism.SAOMechanism'>, mechanism_params: dict[str, ~typing.Any] | None = None, full_names: bool = True, verbosity: int = 0, plot=False, plot_params: dict[str, ~typing.Any] | None = None, stats: ~negmas.preferences.ops.ScenarioStats | None = None, annotation: dict[str, ~typing.Any] | None = None, private_infos: tuple[dict[str, ~typing.Any] | None] | None = None, id_reveals_type: bool = False, name_reveals_type: bool = True, mask_scenario_name: bool = True, ignore_exceptions: bool = False, scored_indices: list[int] | None = None, n_repetitions: int = 1, scenario_index: int = 0, before_start_callback: ~typing.Callable[[~negmas.tournaments.neg.simple.cartesian.RunInfo], None] | None = None, after_construction_callback: ~typing.Callable[[~negmas.tournaments.neg.simple.cartesian.ConstructedNegInfo], None] | None = None, after_end_callback: ~typing.Callable[[dict[str, ~typing.Any]], None] | ~typing.Callable[[dict[str, ~typing.Any], dict[str, ~typing.Any]], None] | None = None, neg_start_callback: ~typing.Callable[[str | int, ~negmas.sao.common.SAOState], None] | None = None, neg_end_callback: ~typing.Callable[[str | int, ~negmas.sao.common.SAOState], None] | None = None, neg_progress_callback: ~typing.Callable[[str | int, ~negmas.sao.common.SAOState], None] | None = None, config: dict[str, ~typing.Any] | None = None, image_format: str = 'webp', storage_optimization: ~typing.Literal['speed', 'time', 'none', 'balanced', 'space', 'max'] = 'space', storage_format: ~typing.Literal['csv', 'gzip', 'parquet'] | None = None, save_negotiations_as_folders: bool = False, opponent_modeling_metrics: tuple[~typing.Literal['kendall', 'kendal_optimality', 'ndcg', 'euclidean'] | ~typing.Callable[[~numpy.ndarray[tuple[~typing.Any, ...], ~numpy.dtype[~numpy.floating[~typing.Any]]], ~numpy.ndarray[tuple[~typing.Any, ...], ~numpy.dtype[~numpy.floating[~typing.Any]]]], float], ...] = (), distribute_opponent_modeling_scores: bool = True) → dict[str, Any]

Run a single negotiation session and return comprehensive results.

Creates negotiator instances, runs them through a negotiation mechanism, and returns detailed results including agreement, utilities, timing, and error information.

Parameters:

• s – Scenario containing the outcome space and utility functions for all parties.

• partners – Negotiator types/classes to instantiate for this negotiation, in order.

• partner_names – Display names for negotiators. If None, generated from class names.

• partner_params – Initialization parameters for each negotiator. If None, use defaults.

• rep – Repetition number for this negotiation (for tracking in tournament context).

• path – Directory to save logs and plots. If None, nothing is saved to disk.

• mechanism_type – Negotiation protocol/mechanism class (default: SAOMechanism).

• mechanism_params – Parameters passed to mechanism constructor.

• full_names – If True and partner_names is None, use full class names instead of shortened.

• verbosity – Logging verbosity level (0 for silent).

• plot – If True and path is provided, save a plot of the negotiation.

• plot_params – Parameters passed to plotting function.

• run_id – Unique identifier for this run. If None, generated from timestamp.

• stats – Pre-calculated scenario statistics. If None, calculated if needed.

• annotation – Dictionary stored in mechanism.nmi.annotation (accessible to negotiators via self.nmi.annotation).

• private_infos – Tuple of private info dicts, one per negotiator (accessible via self.private_info).

• id_reveals_type – If True, negotiator IDs reveal their type.

• name_reveals_type – If True, negotiator names reveal their type.

• mask_scenario_name – If True, scenario name is masked from negotiators.

• ignore_exceptions – If True, catch and log exceptions instead of propagating.

• scored_indices – Positions of negotiators to score. None means score all (used internally by tournament).

• n_repetitions – Total number of repetitions for this scenario/partner combo (for RunInfo).

• scenario_index – Index of this scenario in the tournament (for RunInfo).

• before_start_callback – Optional callback invoked before negotiation starts. Receives RunInfo object.

• after_construction_callback – Optional callback invoked after mechanism construction. Receives ConstructedNegInfo object.

• after_end_callback – Optional callback invoked after negotiation ends. Receives the record dictionary and optionally the config dictionary. Supports both (record) and (record, config) signatures.

• config – Tournament configuration dictionary (same as saved to config.yaml). Passed to callbacks.

• save_negotiations_as_folders – If True, save each negotiation as a folder containing trace, agreement_stats, config, and metadata. If False (default), save as single trace files.

Returns:

• agreement: Final agreed outcome or None

• utilities: Utility of agreement for each negotiator

• reserved_values: Reservation values for each negotiator

• max_utils: Maximum possible utility for each negotiator

• partners: Negotiator class names

• negotiator_ids: Unique IDs of negotiator instances

• negotiator_times: Time spent by each negotiator

• scenario: Scenario name

• timedout/broken/has_error: Status flags

• step/time/relative_time: Negotiation progress metrics

• Plus optimality statistics if stats is provided

Return type:

Dictionary containing complete negotiation results

Examples

Basic usage: ```python record = run_negotiation(

s=scenario, partners=(AspirationNegotiator, RandomNegotiator), mechanism_params=dict(n_steps=100),

) print(f”Agreement: {record[‘agreement’]}”) print(f”Utilities: {record[‘utilities’]}”) ```

negmas.tournaments.run_tournament(tournament_path: str | Path, world_generator: WorldGenerator | None = None, score_calculator: Callable[[list[World], dict[str, Any], bool], WorldRunResults] | None = None, total_timeout: int | None = None, parallelism='parallel', scheduler_ip: str | None = None, scheduler_port: str | None = None, tournament_progress_callback: Callable[[WorldRunResults | None, int, int], None] | None = None, world_progress_callback: Callable[[World | None], None] | None = None, verbose: bool = False, compact: bool | None = None, print_exceptions: bool = True, override_ran_worlds: bool = False, max_attempts: int = 9223372036854775807) → None

Runs a tournament

Parameters:

• tournament_path – Path at which configs of this tournament are stored

• world_generator – A functions to generate worlds for the tournament that follows the assignments made by the config_assigner

• score_calculator – A function for calculating the score of all agents in a world After it finishes running. The second parameter is a dict describing any scoring context that may have been added by the world config generator or assigner. The third parameter is a boolean specifying whether this is a dry_run. For dry runs, scores are not expected but names and types should exist in the returned WorldRunResults.

• total_timeout – Total timeout for the complete process

• parallelism – Type of parallelism. Can be ‘serial’ for serial, ‘parallel’ for parallel and ‘distributed’ for distributed! For parallel, you can add the fraction of CPUs to use after a colon (e.g. parallel:0.5 to use half of the CPU in the machine). By defaults parallel uses all CPUs in the machine

• scheduler_port – Port of the dask scheduler if parallelism is dask, dist, or distributed

• scheduler_ip – IP Address of the dask scheduler if parallelism is dask, dist, or distributed

• world_progress_callback – A function to be called after every step of every world run (only allowed for serial and parallel evaluation and should be used with cautious).

• tournament_progress_callback – A function to be called with WorldRunResults after each world finished processing

• verbose – Verbosity

• compact – If true, compact logs will be created and effort will be made to reduce the memory footprint

• print_exceptions – If true, exceptions encountered during world simulation will be printed to stdout

• override_ran_worlds – If true worlds that are already ran will be ran again

• max_attempts – The maximum number of attempts to run each simulation. Default is infinite

negmas.tournaments.run_world(world_params: dict, dry_run: bool = False, save_world_stats: bool = True, attempts_path=None, max_attempts=inf, verbose=False) → tuple[str, list[str], WorldRunResults | None, WorldSetRunStats | None, AgentStats | None, AgentStats | None]

Runs a world and returns stats. This function is designed to be used with distributed systems like dask.

Parameters:

• world_params – World info dict. See remarks for its parameters

• dry_run – If true, the world will not be run. Only configs will be saved

• save_world_stats – If true, saves individual world stats

• attempts_path – The folder containing attempts information

• max_attempts – The maximum number of trials to run a world simulation

Remarks:

The world_params dict should have the following members:

• name: world name [Defaults to random]

• log_file_name: file name to store the world log [Defaults to random]

• __dir_name: directory to store the world stats [Defaults to random]

• __world_generator: full name of the world generator function (including its module) [Required]

• __score_calculator: full name of the score calculator function [Required]

• __tournament_name: name of the tournament [Defaults to random]

• others: values of all other keys are passed to the world generator as kwargs

negmas.tournaments.tournament(competitors: list[str | Agent] | tuple[str | Agent, ...] | Sequence[str | Agent], config_generator: ConfigGenerator, config_assigner: ConfigAssigner, world_generator: WorldGenerator, score_calculator: Callable[[list[World], dict[str, Any], bool], WorldRunResults], competitor_params: Sequence[dict[str, Any]] | None = None, n_competitors_per_world: int | None = None, round_robin: bool = False, stage_winners_fraction: float = 0.0, agent_names_reveal_type=False, n_agents_per_competitor=1, n_configs: int = 10, max_worlds_per_config: int = 100, n_runs_per_world: int = 5, max_n_configs: int | None = None, n_runs_per_config: int | None = None, tournament_path: str | Path | None = None, total_timeout: int | None = None, parallelism='parallel', scheduler_ip: str | None = None, scheduler_port: str | None = None, tournament_progress_callback: Callable[[WorldRunResults | None, int, int], None] | None = None, world_progress_callback: Callable[[World | None], None] | None = None, non_competitors: tuple[str | Any] | None = None, non_competitor_params: tuple[dict[str, Any]] | None = None, dynamic_non_competitors: tuple[str | Any] | None = None, dynamic_non_competitor_params: tuple[dict[str, Any]] | None = None, exclude_competitors_from_reassignment: bool = True, name: str | None = None, verbose: bool = False, configs_only: bool = False, compact: bool = False, print_exceptions: bool = True, metric='median', save_video_fraction: float = 0.0, forced_logs_fraction: float = 0.0, video_params=None, video_saver=None, max_attempts: int = 9223372036854775807, extra_scores_to_use: str | None = None, **kwargs) → TournamentResults | Path

Runs a tournament

Parameters:

• name – Tournament name

• config_generator – Used to generate unique configs that will be used to evaluate competitors

• config_assigner – Used to generate assignments of competitors to the configs created by the config_generator

• world_generator – A functions to generate worlds for the tournament that follows the assignments made by the config_assigner

• score_calculator – A function for calculating the score of all agents in a world After it finishes running. The second parameter is a dict describing any scoring context that may have been added by the world config generator or assigneer. The third parameter is a boolean specifying whether this is a dry_run. For dry runs, scores are not expected but names and types should exist in the returned WorldRunResults.

• competitors – A list of class names for the competitors

• competitor_params – A list of competitor parameters (used to initialize the competitors).

• n_competitors_per_world –

  The number of competitors allowed in every world. It must be >= 1 and <= len(competitors) or None.

  • If None or len(competitors), then all competitors will exist in every world.

  • If 1, then each world will have one competitor

• round_robin – Only effective if 1 < n_competitors_per_world < len(competitors). if True, all combinations will be tried otherwise n_competitors_per_world must divide len(competitors) and every competitor appears only in one set.

• stage_winners_fraction – in [0, 1). Fraction of agents to to go to the next stage at every stage. If zero, and round_robin, it becomes a single stage competition.

• agent_names_reveal_type – If true then the type of an agent should be readable in its name (most likely at its beginning).

• n_configs – The number of different world configs (up to competitor assignment) to be generated.

• max_worlds_per_config – The maximum number of worlds to run per config. If None, then all possible assignments of competitors within each config will be tried (all permutations).

• n_runs_per_world – Number of runs per world. All of these world runs will have identical competitor assignment and identical world configuration.

• n_agents_per_competitor – The number of agents of each competing type to be instantiated in the world.

• max_n_configs – [Depricated] The number of configs to use (it is replaced by separately setting n_config and max_worlds_per_config )

• n_runs_per_config – [Depricated] The number of runs (simulation) for every config. It is replaced by n_runs_per_world

• total_timeout – Total timeout for the complete process

• tournament_path – Path at which to store all results. A new folder with the name of the tournament will be created at this path. A scores.csv file will keep the scores and logs folder will keep detailed logs

• parallelism – Type of parallelism. Can be ‘serial’ for serial, ‘parallel’ for parallel and ‘distributed’ for distributed! For parallel, you can add the fraction of CPUs to use after a colon (e.g. parallel:0.5 to use half of the CPU in the machine). By defaults parallel uses all CPUs in the machine

• scheduler_port – Port of the dask scheduler if parallelism is dask, dist, or distributed

• scheduler_ip – IP Address of the dask scheduler if parallelism is dask, dist, or distributed

• world_progress_callback – A function to be called after every step of every world run (only allowed for serial and parallel evaluation and should be used with cautious).

• tournament_progress_callback – A function to be called with WorldRunResults after each world finished processing

• non_competitors – A list of agent types that will not be competing but will still exist in the world.

• non_competitor_params – paramters of non competitor agents

• dynamic_non_competitors – A list of non-competing agents that are assigned to the simulation dynamically during the creation of the final assignment instead when the configuration is created

• dynamic_non_competitor_params – paramters of dynamic non competitor agents

• exclude_competitors_from_reassignment – If true, competitors are excluded from the dyanamic non-competitors

• verbose – Verbosity

• configs_only – If true, a config file for each

• compact – If true, compact logs will be created and effort will be made to reduce the memory footprint

• print_exceptions – If true, print all exceptions to screen

• metric – The metric to use for evaluation

• save_video_fraction – The fraction of simulations for which to save videos

• forced_logs_fraction – The fraction of simulations for which to always save logs. Notice that this has no effect except if no logs were to be saved otherwise (i.e. no_logs is passed as True)

• video_params – The parameters to pass to the video saving function

• video_saver – The parameters to pass to the video saving function after the world

• max_attempts – The maximum number of times to retry running simulations

• extra_scores_to_use – The type of extra-scores to use. If None normal scores will be used. Only effective if scores is None.

• kwargs – Arguments to pass to the config_generator function

Returns:

TournamentResults The results of the tournament or a Path giving the location where configs were saved