negmas.tournaments

Tournament generation and management.

class negmas.tournaments.RunInfo(s: ~negmas.inout.Scenario, partners: tuple[type[~negmas.negotiators.negotiator.Negotiator]], partner_names: tuple[str] | None = None, partner_params: tuple[dict[str, ~typing.Any]] | None = None, rep: int = 0, path: ~pathlib.Path | None = None, mechanism_type: type[~negmas.mechanisms.Mechanism] = <class 'negmas.sao.mechanism.SAOMechanism'>, mechanism_params: dict[str, ~typing.Any] | None = None, full_names: bool = True, verbosity: int = 0, plot: bool = False, plot_params: dict[str, ~typing.Any] | None = None, run_id: int | str | None = None, stats: ~negmas.preferences.ops.ScenarioStats | None = None, annotation: dict[str, ~typing.Any] | None = None, private_infos: tuple[dict[str, ~typing.Any] | None] | None = None, id_reveals_type: bool = False, name_reveals_type: bool = True, mask_scenario_name: bool = True, ignore_exceptions: bool = False, scored_indices: list[int] | None = None, n_repetitions: int = 1, scenario_index: int = 0)

Bases: object

Information about a negotiation run before it starts.

Passed to before_start_callback in cartesian_tournament to allow inspection and logging of negotiation parameters before execution.

s

The scenario containing outcome space and utility functions

Type:

negmas.inout.Scenario

partners

Tuple of negotiator classes that will participate

Type:

tuple[type[negmas.negotiators.negotiator.Negotiator]]

mechanism_type

The mechanism class that will be instantiated

Type:

type[negmas.mechanisms.Mechanism]

mechanism_params

Parameters that will be passed to the mechanism

Type:

dict[str, Any] | None

n_repetitions

Total number of repetitions for this scenario/partner combination

Type:

int

rep

Current repetition index (0-based)

Type:

int

scenario_index

Index of this scenario in the tournament

Type:

int

negotiation_id

Unique identifier for this negotiation run

scored_indices

Indices of negotiators whose scores will be recorded (None means all)

Type:

list[int] | None

annotation: dict[str, Any] | None

full_names: bool

id_reveals_type: bool

ignore_exceptions: bool

mask_scenario_name: bool

mechanism_params: dict[str, Any] | None

mechanism_type: type[Mechanism]

n_repetitions: int

name_reveals_type: bool

partner_names: tuple[str] | None

partner_params: tuple[dict[str, Any]] | None

partners: tuple[type[Negotiator]]

path: Path | None

plot: bool

plot_params: dict[str, Any] | None

private_infos: tuple[dict[str, Any] | None] | None

rep: int

run_id: int | str | None

s: Scenario

scenario_index: int

scored_indices: list[int] | None

stats: ScenarioStats | None

verbosity: int

class negmas.tournaments.SimpleTournamentResults(scores: DataFrame | None = None, details: DataFrame | None = None, scores_summary: DataFrame | None = None, final_scores: DataFrame | None = None, path: Path | None = None, memory_optimization: Literal['speed', 'time', 'none', 'balanced', 'space', 'max'] = 'balanced', storage_optimization: Literal['speed', 'time', 'none', 'balanced', 'space', 'max'] = 'space', storage_format: Literal['csv', 'gzip', 'parquet'] | None = None, final_score_stat: tuple[str, str] = ('advantage', 'mean'))

Bases: object

Tournament results with optional lazy loading and memory optimization.

This class stores tournament results and supports different memory optimization levels that control whether data is kept in memory or loaded from disk on demand.

scores

All scores per negotiator (may be loaded lazily)

details

All negotiation results (may be loaded lazily)

scores_summary

All score statistics summarized per type (always in memory)

final_scores

Final rankings sorted from highest to lowest score (always in memory)

path

Location at which the logs are stored

memory_optimization

Memory optimization level (“speed”, “balanced”, “space”)

storage_optimization

Storage optimization level (“speed”, “balanced”, “space”)

storage_format

Storage format for data files (“csv”, “gzip”, “parquet”)

classmethod combine(paths: Path | Iterable[Path], recursive: bool = True, recalc_details: bool = True, recalc_scores: bool = False, must_have_details: bool = False, verbosity: int = 1, final_score_stat: tuple[str, str] = ('advantage', 'mean'), add_tournament_column: bool = True, complete_only: bool = True) → tuple[SimpleTournamentResults, list[Path]]

Combines the results of multiple tournaments stored on disk.

This method can combine tournaments saved with different storage formats (csv, gzip, parquet) and different optimization levels. It auto-detects the format of each tournament’s files.

Parameters:

• paths – Paths to look for results within

• recursive – Check children of given paths recursively

• recalc_details – Recalculate detailed results from the results/ folder

• recalc_scores – Recalculate scores from detailed negotiation results

• must_have_details – Raise an exception if detailed negotiation results cannot be found

• verbosity – Verbosity level (0=silent, 1=basic, 2+=detailed)

• final_score_stat – Tuple of (measure, statistic) for final score calculation. See cartesian_tournament for details.

• add_tournament_column – Add a column called “tournament” with tournament name in details and scores DataFrames.

• complete_only – If True, only include tournaments that completed successfully (have all required files). Incomplete tournaments are ignored.

Returns:

• SimpleTournamentResults with the combined results of all tournaments

• List of Paths that were successfully loaded

Return type:

A tuple of

Raises:

FileNotFoundError – If no valid tournament paths are found or if a needed file is missing (when must_have_details=True)

Notes

File Format Handling:

• Tournaments with different storage formats can be combined

• Each tournament’s format is auto-detected independently

• The combined result uses the default format (csv)

Required Files (when complete_only=True):

• details.csv/.csv.gz/.parquet (any format)

• all_scores.csv/.csv.gz/.parquet (any format)

• scores.csv (final scores, always CSV)

Data Reconstruction:

• If recalc_details=True, details are rebuilt from results/ folder

• If recalc_scores=True, scores are rebuilt from details

• This allows combining tournaments with different storage_optimization levels

property details: DataFrame

All negotiation results (may be loaded from disk).

property final_scores: DataFrame

Final rankings sorted from highest to lowest score.

classmethod from_records(scores: list[dict[str, Any]] | DataFrame | None = None, results: list[dict[str, Any]] | DataFrame | None = None, type_scores: DataFrame | None = None, final_scores: DataFrame | None = None, final_score_stat: tuple[str, str] = ('advantage', 'mean'), path: Path | None = None) → SimpleTournamentResults

Creates SimpleTournamentResults from records of results

Parameters:

• scores – The scores of negotiators in all negotiations (If not given, results can be used to calculate it).

• results – Results of all negotiations (If not given, the resulting SimpleTournamentResults object will lack details)

• type_scores – Optionally, type-scores. If not given, it will be calculated from scores

• final_scores – Optionally, final scores. If not given, final_scoer_stat will be used to calculate them

• final_score_stat – A tuple of the measure used and the statistic applied to it for calculating final score. See cartesian_tournament for more details

• path – The path in which the data for this tournament is stored.

Raises:

ValueError – If no scores or results are given

Returns:

A new SimpleTournamentResults with the given data

classmethod from_result_records(path: Path, verbosity: int = 1, final_score_stat: tuple[str, str] = ('advantage', 'mean')) → SimpleTournamentResults

From result records.

Parameters:

• path – Path.

• verbosity – Verbosity.

• final_score_stat – Final score stat.

Returns:

The result.

Return type:

‘SimpleTournamentResults’

classmethod load(path: Path, must_have_details: bool = False, memory_optimization: Literal['speed', 'time', 'none', 'balanced', 'space', 'max'] = 'balanced', storage_optimization: Literal['speed', 'time', 'none', 'balanced', 'space', 'max'] = 'space', storage_format: Literal['csv', 'gzip', 'parquet'] | None = None) → SimpleTournamentResults

Loads tournament results from the given path.

This method auto-detects the storage format and reconstructs scores if needed. It supports loading from tournaments saved with any storage_optimization level.

Parameters:

• path – Path to load results from

• must_have_details – If True, raise an error if details are not found

• memory_optimization – Memory optimization level for the loaded results: - “speed”: Keep all data in memory - “balanced”: Keep details in memory, compute scores on demand - “space”: Load data from disk on demand

• storage_optimization – Storage optimization level (for metadata only, does not affect loading behavior)

• storage_format – Storage format hint (auto-detected if None from existing files)

Returns:

SimpleTournamentResults loaded from disk

Raises:

FileNotFoundError – If required files are not found

Notes

File Format Detection Priority (for details and all_scores):

1. Parquet (.parquet) - Best compression, preserves types

2. Gzip (.csv.gz) - Good compression

3. CSV (.csv) - Plain text

Data Reconstruction Priority (if scores/details not found):

1. Load from data files (details.parquet/csv.gz/csv, all_scores.parquet/csv.gz/csv)

2. Reconstruct from results/ folder JSON files

3. Reconstruct scores from details DataFrame

Small files (scores.csv, type_scores.csv) are always stored as plain CSV.

property memory_optimization: Literal['speed', 'time', 'none', 'balanced', 'space', 'max']

Memory optimization level.

property path: Path | None

Location at which the logs are stored.

save(path: Path | None = None, exist_ok: bool = True, storage_optimization: Literal['speed', 'time', 'none', 'balanced', 'space', 'max'] | None = None, storage_format: Literal['csv', 'gzip', 'parquet'] | None = None) → None

Save all results to the given path.

Parameters:

• path – Path to save results to. If None, uses self.path

• exist_ok – If True, don’t raise an error if the directory exists

• storage_optimization – Override the instance’s storage_optimization setting. - “speed”: Save all files (scores, details, scores_summary, final_scores) - “balanced”: Same as speed (cleanup happens in cartesian_tournament) - “space”: Skip saving all_scores (can be reconstructed from details)

• storage_format – Override the instance’s storage_format setting. - “csv”: Plain CSV files - “gzip”: Gzip-compressed CSV files - “parquet”: Parquet binary format

property scores: DataFrame

All scores per negotiator (may be computed or loaded from disk).

property scores_summary: DataFrame

All score statistics summarized per type.

property storage_format: Literal['csv', 'gzip', 'parquet']

Storage format for data files.

property storage_optimization: Literal['speed', 'time', 'none', 'balanced', 'space', 'max']

Storage optimization level.

class negmas.tournaments.TournamentResults(scores: DataFrame, total_scores: DataFrame, winners: list[str], winners_scores: ndarray, ttest: DataFrame | None = None, kstest: DataFrame | None = None, stats: DataFrame | None = None, agg_stats: DataFrame | None = None, score_stats: DataFrame | None = None, path: str | Path | None = None, world_stats: DataFrame | None = None, type_stats: DataFrame | None = None, agent_stats: DataFrame | None = None, params: dict[str, Any] | None = None, extra_scores: dict[str, DataFrame] | None = None)

Bases: object

TournamentResults implementation.

agent_stats: DataFrame | None = None

Some statistics about each agent

agg_stats: DataFrame | None = None

Aggregated stats per world

extra_scores: dict[str, DataFrame] | None = None

Extra scores returned from the scoring function. This can be used to have multi-dimensional scoring

kstest: DataFrame | None = None

Results of the nonparametric kstest

params: dict[str, Any] | None = None

Parameters of the tournament

path: str | Path | None = None

Path at which tournament results are stored

score_stats: DataFrame | None = None

Score statistics for different competitor types

scores: DataFrame

Scores of individual agent instantiations

stats: DataFrame | None = None

Stats of all worlds

total_scores: DataFrame

Total scores collected by competitor types

ttest: DataFrame | None = None

Results of ttest analysis of the scores

type_stats: DataFrame | None = None

Some statistics about each type

winners: list[str]

Winner type name(s) which may be a list

winners_scores: ndarray

Winner score (accumulated)

world_stats: DataFrame | None = None

Some statistics about each world run

class negmas.tournaments.WorldGenerator(*args, **kwargs)

Bases: Protocol

A callback-protocol specifying the signature of a world generator function that can be passed to tournament

Parameters:

kwargs – key-value pairs of arguments.

See also

tournament

class negmas.tournaments.WorldRunResults(world_names: list[str], log_file_names: list[str])

Bases: object

Results of a world run

extra_scores: dict[str, list[dict[str, Any]]]

The extra-scores (i.e. extra evaluation metrics). Each is a list of records

ids: list[str]

Agent IDs

log_file_names: list[str]

Log file names

names: list[str]

Agent names

scores: list[float]

Agent scores

types: list[str]

Agent type names

world_names: list[str]

World names (there can be multiple worlds for each scoring call)

negmas.tournaments.cartesian_tournament(competitors: list[type[~negmas.negotiators.negotiator.Negotiator] | str] | tuple[type[~negmas.negotiators.negotiator.Negotiator] | str, ...], scenarios: list[~negmas.inout.Scenario] | tuple[~negmas.inout.Scenario, ...], opponents: list[type[~negmas.negotiators.negotiator.Negotiator] | str] | tuple[type[~negmas.negotiators.negotiator.Negotiator] | str, ...] = (), opponent_params: ~typing.Sequence[dict | None] | None = None, private_infos: list[None | tuple[dict, ...]] | None = None, competitor_params: ~typing.Sequence[dict | None] | None = None, rotate_ufuns: bool = True, rotate_private_infos: bool = True, n_repetitions: int = 1, path: ~pathlib.Path | None = None, njobs: int = 0, mechanism_type: type[~negmas.mechanisms.Mechanism] = <class 'negmas.sao.mechanism.SAOMechanism'>, mechanism_params: dict[str, ~typing.Any] | None = None, n_steps: int | tuple[int, int] | None = 100, time_limit: float | tuple[float, float] | None = None, pend: float | tuple[float, float] = 0.0, pend_per_second: float | tuple[float, float] = 0.0, step_time_limit: float | tuple[float, float] | None = None, negotiator_time_limit: float | tuple[float, float] | None = None, hidden_time_limit: float | tuple[float, float] | None = None, external_timeout: int | None = None, plot_fraction: float = 0.0, plot_params: dict[str, ~typing.Any] | None = None, verbosity: int = 1, self_play: bool = True, randomize_runs: bool = True, sort_runs: bool = False, save_every: int = 0, save_stats: bool = True, save_scenario_figs: bool = True, final_score: tuple[str, str] = ('advantage', 'mean'), id_reveals_type: bool = False, name_reveals_type: bool = True, shorten_names: bool = True, raise_exceptions: bool = True, mask_scenario_names: bool = True, only_failures_on_self_play: bool = False, ignore_discount: bool = False, ignore_reserved: bool = False, storage_optimization: ~typing.Literal['speed', 'time', 'none', 'balanced', 'space', 'max'] = 'space', memory_optimization: ~typing.Literal['speed', 'time', 'none', 'balanced', 'space', 'max'] = 'balanced', storage_format: ~typing.Literal['csv', 'gzip', 'parquet'] | None = None, python_class_identifier='__python_class__', before_start_callback: ~typing.Callable[[~negmas.tournaments.neg.simple.cartesian.RunInfo], None] | None = None, after_construction_callback: ~typing.Callable[[~negmas.tournaments.neg.simple.cartesian.ConstructedNegInfo], None] | None = None, after_end_callback: ~typing.Callable[[dict[str, ~typing.Any]], None] | None = None, progress_callback: ~typing.Callable[[str, int, int], None] | None = None) → SimpleTournamentResults

Run a Cartesian tournament where negotiators compete across multiple scenarios.

This function runs negotiations between all combinations of competitors across all scenarios, optionally with rotated utility functions. When opponents are provided, competitors only play against opponents (not each other) and only competitor scores are recorded.

Parameters:

• competitors – Negotiator types or class names to compete in the tournament.

• scenarios – Negotiation scenarios, each with an outcome space and utility functions.

• opponents – Optional negotiator types to use as opponents. If provided, competitors only play against opponents (not each other) and only competitors are scored. Competitors will be tested in both first and last positions to evaluate different roles (e.g., buyer vs seller).

• opponent_params – Parameters for initializing opponents (one dict per opponent type).

• private_infos – Private information passed to negotiators via their private_info attribute. Must be a list of tuples, one tuple per scenario.

• competitor_params – Parameters for initializing competitors (one dict per competitor type).

• rotate_ufuns – If True, utility functions are rotated across negotiator positions. For bilateral negotiations, this creates scenarios with reversed preferences. Not recommended when using explicit opponents as roles become ambiguous.

• rotate_private_infos – If True and rotate_ufuns is True, rotate private information with ufuns.

• n_repetitions – Number of times to repeat each scenario/partner combination.

• path – Directory path to save tournament results. If None, results are not saved to disk.

• njobs – Parallelization level. -1 for serial execution (good for debugging), 0 for all available cores, positive integer for specific number of processes.

• mechanism_type – The negotiation protocol/mechanism class to use (default: SAOMechanism).

• mechanism_params – Additional parameters passed to the mechanism constructor.

• n_steps – Maximum rounds per negotiation. Can be int, (min, max) tuple for random sampling, or None for unlimited.

• time_limit – Maximum seconds per negotiation. Can be float, (min, max) tuple, or None.

• pend – Probability of ending negotiation each step. Can be float, (min, max) tuple, or 0.0.

• pend_per_second – Probability of ending negotiation each second. Can be float, (min, max) tuple, or 0.0.

• step_time_limit – Maximum seconds per negotiation step. Can be float, (min, max) tuple, or None.

• negotiator_time_limit – Maximum total seconds for all actions by each negotiator.

• hidden_time_limit – Time limit not revealed to negotiators.

• external_timeout – Timeout in seconds for receiving results from parallel negotiations.

• plot_fraction – Fraction of negotiations to plot (0.0 to 1.0). Only used if path is provided.

• plot_params – Parameters passed to plotting functions.

• verbosity – Logging level (0 for silent, higher for more verbose).

• self_play – If True, allow negotiations where all parties are the same type.

• only_failures_on_self_play – If True, only record self-play negotiations that fail to reach agreement.

• randomize_runs – If True, run negotiations in random order instead of sequentially.

• sort_runs – If True, sort runs by scenario size before execution.

• save_every – Save results to disk after this many negotiations (0 to disable periodic saving).

• save_stats – If True, calculate optimality statistics (Pareto, Nash, Kalai-Smorodinsky, etc.).

• save_scenario_figs – If True, save PNG visualizations of scenarios in utility space.

• final_score – Tuple of (metric, statistic) for ranking. Metric can be ‘advantage’, ‘utility’, ‘partner_welfare’, ‘welfare’, or any calculated statistic. Statistic can be ‘mean’, ‘median’, ‘min’, ‘max’, or ‘std’. Default: (‘advantage’, ‘mean’).

• id_reveals_type – If True, negotiator IDs reveal their type (for analysis).

• name_reveals_type – If True, negotiator names reveal their type.

• shorten_names – If True, use shortened class names in results.

• raise_exceptions – If True, exceptions from negotiators/mechanisms stop the tournament. If False, exceptions are logged but tournament continues.

• mask_scenario_names – If True, mask scenario names from negotiators.

• only_failures_on_self_play – If True, only record self-play runs that fail to reach agreement.

• ignore_discount – If True, ignore discounting in utility functions (use base ufun).

• ignore_reserved – If True, ignore reserved values in utility functions.

• storage_optimization –

  Controls disk space usage for tournament results (default: “space”): - “speed”/”time”/”none”: Keep all files (results/, all_scores.csv, details.csv, etc.) - “balanced”: Remove results/ folder after details.csv is created - “space”/”max”: Remove both results/ folder AND all_scores.csv (default)

  (scores can be reconstructed from details.csv)

• memory_optimization –

  Controls RAM usage for returned SimpleTournamentResults (default: “balanced”): - “speed”/”time”/”none”: Keep all DataFrames in memory - “balanced”: Keep details + final_scores + scores_summary in memory,

  compute scores on demand then cache (default)

  • ”space”/”max”: Keep only final_scores + scores_summary in memory,

    load details/scores from disk when needed

  Note: If path is None, memory_optimization is ignored (everything kept in memory)

• storage_format – Storage format for large data files (all_scores, details): - “csv”: Plain CSV files (human-readable, larger size) - “gzip”: Gzip-compressed CSV files (good compression, human-readable when decompressed) - “parquet”: Parquet binary format (best compression, preserves types, fastest) - None: Auto-select based on storage_optimization (default; csv for speed, gzip for balanced, parquet for space) Note: Small files (scores.csv, type_scores.csv) are always CSV regardless of this setting.

• python_class_identifier – Function to convert classes to string identifiers.

• before_start_callback – Optional callback invoked before each negotiation starts. Receives a RunInfo object with all negotiation parameters. Useful for logging, monitoring, or custom setup. Exceptions are caught and logged (if verbosity > 0) but don’t stop the tournament.

• after_construction_callback – Optional callback invoked after mechanism construction but before negotiation starts. Receives a ConstructedNegInfo object with the constructed mechanism and scenario details. Useful for inspecting or modifying the mechanism before negotiation. Exceptions are caught and logged (if verbosity > 0) but don’t stop the tournament.

• after_end_callback – Optional callback invoked after each negotiation completes. Receives the complete negotiation record dictionary with agreement, utilities, and all result metadata. Useful for custom analysis or logging. Exceptions are caught and logged (if verbosity > 0) but don’t stop the tournament.

• progress_callback – Optional callback invoked during tournament setup to report progress. Receives (message: str, current: int, total: int) where message describes the current phase, current is the progress index, and total is the expected count. Useful for showing setup progress in UIs before negotiations start. Called during competitor validation, scenario processing, and run configuration building.

Returns:

SimpleTournamentResults containing scores, detailed results, score summaries, and final rankings.

Notes

• In explicit opponent mode (opponents provided), competitors appear at the first position. Use rotate_ufuns=True to test performance in different roles.

• Use njobs=-1 for debugging to run serially and see full tracebacks.

Examples

Normal tournament between two negotiators: ```python results = cartesian_tournament(

competitors=[MyNegotiator, TheirNegotiator], scenarios=[scenario1, scenario2], n_steps=100, path=Path(“results/”),

)

Testing a negotiator against fixed opponents: ```python results = cartesian_tournament(

competitors=[MyNegotiator], opponents=[RandomNegotiator, AspirationNegotiator], scenarios=[scenario1], rotate_ufuns=False, # Keep roles fixed

)

Using callbacks for monitoring: ```python def log_start(info: RunInfo):

print(f”Starting negotiation {info.rep} with {info.partners}”)

def log_end(record: dict):

print(f”Ended with agreement: {record[‘agreement’]}”)

results = cartesian_tournament(

competitors=[MyNegotiator, TheirNegotiator], scenarios=[scenario1], before_start_callback=log_start, after_end_callback=log_end,

)

negmas.tournaments.combine_tournament_results(sources: Iterable[str | Path], dest: str | Path | None = None, verbose=False, max_sources: int | None = None) → DataFrame

Combines results of several tournament runs in the destination path.

negmas.tournaments.combine_tournament_stats(sources: Iterable[str | Path], dest: str | Path | None = None, verbose=False, max_sources: int | None = None) → DataFrame

Combines statistical results of several tournament runs in the destination path.

negmas.tournaments.combine_tournaments(sources: Iterable[str | Path], dest: str | Path, verbose=False) → tuple[int, int]

Combines contents of several tournament runs in the destination path allowing for continuation of the tournament

Parameters:

• sources – The sources of tournaments in the filesystem

• dest – where to store the combined tournament.

Returns:

Tuple[int, int] The number of base configs and assigned configs combined

negmas.tournaments.create_tournament(competitors: Sequence[str | type[Agent]], config_generator: ConfigGenerator, config_assigner: ConfigAssigner, world_generator: WorldGenerator, score_calculator: Callable[[list[World], dict[str, Any], bool], WorldRunResults], competitor_params: Sequence[dict[str, Any]] | None = None, n_competitors_per_world: int | None = None, round_robin: bool = True, agent_names_reveal_type=False, n_agents_per_competitor=1, n_configs: int = 10, max_worlds_per_config: int | None = 100, n_runs_per_world: int = 5, max_n_configs: int | None = None, n_runs_per_config: int | None = None, base_tournament_path: Path | str | None = None, total_timeout: int | None = None, parallelism='parallel', scheduler_ip: str | None = None, scheduler_port: str | None = None, non_competitors: tuple[str | Any, ...] | None = None, non_competitor_params: tuple[dict[str, Any], ...] | None = None, dynamic_non_competitors: tuple[str | Any, ...] | None = None, dynamic_non_competitor_params: tuple[dict[str, Any], ...] | None = None, exclude_competitors_from_reassignment: bool = True, name: str | None = None, verbose: bool = False, compact: bool = False, save_video_fraction: float = 0.0, forced_logs_fraction: float = 0.0, video_params=None, video_saver=None, **kwargs) → Path

Creates a tournament

Parameters:

• name – Tournament name

• config_generator – Used to generate unique configs that will be used to evaluate competitors

• config_assigner – Used to generate assignments of competitors to the configs created by the config_generator

• world_generator – A functions to generate worlds for the tournament that follows the assignments made by the config_assigner

• score_calculator – A function for calculating the score of all agents in a world After it finishes running. The second parameter is a dict describing any scoring context that may have been added by the world config generator or assigneer. The third parameter is a boolean specifying whether this is a dry_run. For dry runs, scores are not expected but names and types should exist in the returned WorldRunResults.

• competitors – A list of class names for the competitors

• competitor_params – A list of competitor parameters (used to initialize the competitors).

• n_competitors_per_world –

  The number of competitors allowed in every world. It must be >= 1 and <= len(competitors) or None.

  • If None or len(competitors), then all competitors will exist in every world.

  • If 1, then each world will have one competitor

• round_robin – Only effective if 1 < n_competitors_per_world < len(competitors). if True, all combinations will be tried otherwise n_competitors_per_world must divide len(competitors) and every competitor appears only in one set.

• agent_names_reveal_type – If true then the type of an agent should be readable in its name (most likely at its beginning).

• n_configs – The number of different world configs (up to competitor assignment) to be generated.

• max_worlds_per_config – The maximum number of worlds to run per config. If None, then all possible assignments of competitors within each config will be tried (all permutations).

• n_runs_per_world – Number of runs per world. All of these world runs will have identical competitor assignment and identical world configuration.

• n_agents_per_competitor – The number of agents of each competing type to be instantiated in the world.

• max_n_configs – [Depricated] The number of configs to use (it is replaced by separately setting n_config and max_worlds_per_config )

• n_runs_per_config – [Depricated] The number of runs (simulation) for every config. It is replaced by n_runs_per_world

• total_timeout – Total timeout for the complete process

• base_tournament_path – Path at which to store all results. A new folder with the name of the tournament will be created at this path. A scores.csv file will keep the scores and logs folder will keep detailed logs

• parallelism – Type of parallelism. Can be ‘serial’ for serial, ‘parallel’ for parallel and ‘distributed’ for distributed! For parallel, you can add the fraction of CPUs to use after a colon (e.g. parallel:0.5 to use half of the CPU in the machine). By defaults parallel uses all CPUs in the machine

• scheduler_port – Port of the dask scheduler if parallelism is dask, dist, or distributed

• scheduler_ip – IP Address of the dask scheduler if parallelism is dask, dist, or distributed

• non_competitors – A list of agent types that will not be competing but will still exist in the world.

• non_competitor_params – paramters of non competitor agents

• dynamic_non_competitors – A list of non-competing agents that are assigned to the simulation dynamically during the creation of the final assignment instead when the configuration is created

• dynamic_non_competitor_params – paramters of dynamic non competitor agents

• exclude_competitors_from_reassignment – If true, copmetitors are not included in the reassignment even if they exist in dynamic_non_competitors

• verbose – Verbosity

• compact – If true, compact logs will be created and effort will be made to reduce the memory footprint

• save_video_fraction – The fraction of simulations for which to save videos

• forced_logs_fraction – The fraction of simulations for which to always save logs. Notice that this has no effect except if no logs were to be saved otherwise (i.e. no_logs is passed as True)

• video_params – The parameters to pass to the video saving function

• video_saver – The parameters to pass to the video saving function after the world

• kwargs – Arguments to pass to the config_generator function

Returns:

The path at which tournament configs are stored

negmas.tournaments.evaluate_tournament(tournament_path: str | Path | None, scores: DataFrame | None = None, stats: DataFrame | None = None, world_stats: DataFrame | None = None, type_stats: DataFrame | None = None, agent_stats: DataFrame | None = None, metric: str | Callable[[DataFrame], float] = 'mean', verbose: bool = False, recursive: bool = True, extra_scores_to_use: str | None = None, compile: bool = True) → TournamentResults

Evaluates the results of a tournament

Parameters:

• tournament_path – Path to save the results to. If scores is not given, it is also used as the source of scores. Pass None to avoid saving the results to disk.

• scores – Optionally the scores of all agents in all world runs. If not given they will be read from the file scores.csv in tournament_path

• stats – Optionally the stats of all world runs. If not given they will be read from the file stats.csv in tournament_path

• world_stats – Optionally the aggregate stats collected in WorldSetRunStats for each world set

• type_stats – Optionally the aggregate stats collected in AgentStats for each agent type

• agent_stats – Optionally the aggregate stats collected in AgentStats for each agent instance

• metric – The metric used for evaluation. Possibilities are: mean, median, std, var, sum, truncated_mean or a callable that receives a pandas data-frame and returns a float.

• verbose – If true, the winners will be printed

• recursive – If true, ALL scores.csv files in all subdirectories of the given tournament_path will be combined

• extra_scores_to_use – The type of extra-scores to use. If None normal scores will be used. Only effective if scores is None.

• compile – Takes effect only if tournament_path is not None. If true, the results will be recompiled from individual world results. This is accurate but slow. If false, it will be assumed that all results are already compiled.

• independent_test (#) – True if you want an independent t-test

Returns:

negmas.tournaments.process_world_run(run_id: str, results: WorldRunResults | None, tournament_name: str) → tuple[list[dict[str, Any]], dict[str, list[dict[str, Any]]]]

Generates a data-frame with the results of this world run

Parameters:

• run_id – The ID of this run (should be unique per tournament)

• results – Results of the world run

• tournament_name – tournament name

Returns:

• A list of records containing scores

• A dict mapping extra-score types to lists of records for this type.

Return type:

A tuple of two items

Remarks:

The score calculator returns a WorldRunResults object which must contain a scores element used for evaluating the agents. It can also return extra_scores that can be used to save additional information about agent performance. These are optional and the second output of this function will be the processed version of these extra scores if any.

negmas.tournaments.run_negotiation(s: ~negmas.inout.Scenario, partners: tuple[type[~negmas.negotiators.negotiator.Negotiator]], partner_names: tuple[str] | None = None, partner_params: tuple[dict[str, ~typing.Any]] | None = None, rep: int = 0, path: ~pathlib.Path | None = None, mechanism_type: type[~negmas.mechanisms.Mechanism] = <class 'negmas.sao.mechanism.SAOMechanism'>, mechanism_params: dict[str, ~typing.Any] | None = None, full_names: bool = True, verbosity: int = 0, plot=False, plot_params: dict[str, ~typing.Any] | None = None, run_id: int | str | None = None, stats: ~negmas.preferences.ops.ScenarioStats | None = None, annotation: dict[str, ~typing.Any] | None = None, private_infos: tuple[dict[str, ~typing.Any] | None] | None = None, id_reveals_type: bool = False, name_reveals_type: bool = True, mask_scenario_name: bool = True, ignore_exceptions: bool = False, scored_indices: list[int] | None = None, n_repetitions: int = 1, scenario_index: int = 0, before_start_callback: ~typing.Callable[[~negmas.tournaments.neg.simple.cartesian.RunInfo], None] | None = None, after_construction_callback: ~typing.Callable[[~negmas.tournaments.neg.simple.cartesian.ConstructedNegInfo], None] | None = None, after_end_callback: ~typing.Callable[[dict[str, ~typing.Any]], None] | None = None) → dict[str, Any]

Run a single negotiation session and return comprehensive results.

Creates negotiator instances, runs them through a negotiation mechanism, and returns detailed results including agreement, utilities, timing, and error information.

Parameters:

• s – Scenario containing the outcome space and utility functions for all parties.

• partners – Negotiator types/classes to instantiate for this negotiation, in order.

• partner_names – Display names for negotiators. If None, generated from class names.

• partner_params – Initialization parameters for each negotiator. If None, use defaults.

• rep – Repetition number for this negotiation (for tracking in tournament context).

• path – Directory to save logs and plots. If None, nothing is saved to disk.

• mechanism_type – Negotiation protocol/mechanism class (default: SAOMechanism).

• mechanism_params – Parameters passed to mechanism constructor.

• full_names – If True and partner_names is None, use full class names instead of shortened.

• verbosity – Logging verbosity level (0 for silent).

• plot – If True and path is provided, save a plot of the negotiation.

• plot_params – Parameters passed to plotting function.

• run_id – Unique identifier for this run. If None, generated from timestamp.

• stats – Pre-calculated scenario statistics. If None, calculated if needed.

• annotation – Dictionary stored in mechanism.nmi.annotation (accessible to negotiators via self.nmi.annotation).

• private_infos – Tuple of private info dicts, one per negotiator (accessible via self.private_info).

• id_reveals_type – If True, negotiator IDs reveal their type.

• name_reveals_type – If True, negotiator names reveal their type.

• mask_scenario_name – If True, scenario name is masked from negotiators.

• ignore_exceptions – If True, catch and log exceptions instead of propagating.

• scored_indices – Positions of negotiators to score. None means score all (used internally by tournament).

• n_repetitions – Total number of repetitions for this scenario/partner combo (for RunInfo).

• scenario_index – Index of this scenario in the tournament (for RunInfo).

• before_start_callback – Optional callback invoked before negotiation starts. Receives RunInfo object.

• after_construction_callback – Optional callback invoked after mechanism construction. Receives ConstructedNegInfo object.

• after_end_callback – Optional callback invoked after negotiation ends. Receives the record dictionary.

Returns:

• agreement: Final agreed outcome or None

• utilities: Utility of agreement for each negotiator

• reserved_values: Reservation values for each negotiator

• max_utils: Maximum possible utility for each negotiator

• partners: Negotiator class names

• negotiator_ids: Unique IDs of negotiator instances

• negotiator_times: Time spent by each negotiator

• scenario: Scenario name

• timedout/broken/has_error: Status flags

• step/time/relative_time: Negotiation progress metrics

• Plus optimality statistics if stats is provided

Return type:

Dictionary containing complete negotiation results

Examples

Basic usage: ```python record = run_negotiation(

s=scenario, partners=(AspirationNegotiator, RandomNegotiator), mechanism_params=dict(n_steps=100),

) print(f”Agreement: {record[‘agreement’]}”) print(f”Utilities: {record[‘utilities’]}”) ```

negmas.tournaments.run_tournament(tournament_path: str | Path, world_generator: WorldGenerator | None = None, score_calculator: Callable[[list[World], dict[str, Any], bool], WorldRunResults] | None = None, total_timeout: int | None = None, parallelism='parallel', scheduler_ip: str | None = None, scheduler_port: str | None = None, tournament_progress_callback: Callable[[WorldRunResults | None, int, int], None] | None = None, world_progress_callback: Callable[[World | None], None] | None = None, verbose: bool = False, compact: bool | None = None, print_exceptions: bool = True, override_ran_worlds: bool = False, max_attempts: int = 9223372036854775807) → None

Runs a tournament

Parameters:

• tournament_path – Path at which configs of this tournament are stored

• world_generator – A functions to generate worlds for the tournament that follows the assignments made by the config_assigner

• score_calculator – A function for calculating the score of all agents in a world After it finishes running. The second parameter is a dict describing any scoring context that may have been added by the world config generator or assigner. The third parameter is a boolean specifying whether this is a dry_run. For dry runs, scores are not expected but names and types should exist in the returned WorldRunResults.

• total_timeout – Total timeout for the complete process

• parallelism – Type of parallelism. Can be ‘serial’ for serial, ‘parallel’ for parallel and ‘distributed’ for distributed! For parallel, you can add the fraction of CPUs to use after a colon (e.g. parallel:0.5 to use half of the CPU in the machine). By defaults parallel uses all CPUs in the machine

• scheduler_port – Port of the dask scheduler if parallelism is dask, dist, or distributed

• scheduler_ip – IP Address of the dask scheduler if parallelism is dask, dist, or distributed

• world_progress_callback – A function to be called after every step of every world run (only allowed for serial and parallel evaluation and should be used with cautious).

• tournament_progress_callback – A function to be called with WorldRunResults after each world finished processing

• verbose – Verbosity

• compact – If true, compact logs will be created and effort will be made to reduce the memory footprint

• print_exceptions – If true, exceptions encountered during world simulation will be printed to stdout

• override_ran_worlds – If true worlds that are already ran will be ran again

• max_attempts – The maximum number of attempts to run each simulation. Default is infinite

negmas.tournaments.run_world(world_params: dict, dry_run: bool = False, save_world_stats: bool = True, attempts_path=None, max_attempts=inf, verbose=False) → tuple[str, list[str], WorldRunResults | None, WorldSetRunStats | None, AgentStats | None, AgentStats | None]

Runs a world and returns stats. This function is designed to be used with distributed systems like dask.

Parameters:

• world_params – World info dict. See remarks for its parameters

• dry_run – If true, the world will not be run. Only configs will be saved

• save_world_stats – If true, saves individual world stats

• attempts_path – The folder containing attempts information

• max_attempts – The maximum number of trials to run a world simulation

Remarks:

The world_params dict should have the following members:

• name: world name [Defaults to random]

• log_file_name: file name to store the world log [Defaults to random]

• __dir_name: directory to store the world stats [Defaults to random]

• __world_generator: full name of the world generator function (including its module) [Required]

• __score_calculator: full name of the score calculator function [Required]

• __tournament_name: name of the tournament [Defaults to random]

• others: values of all other keys are passed to the world generator as kwargs

negmas.tournaments.tournament(competitors: list[str | Agent] | tuple[str | Agent, ...] | Sequence[str | Agent], config_generator: ConfigGenerator, config_assigner: ConfigAssigner, world_generator: WorldGenerator, score_calculator: Callable[[list[World], dict[str, Any], bool], WorldRunResults], competitor_params: Sequence[dict[str, Any]] | None = None, n_competitors_per_world: int | None = None, round_robin: bool = False, stage_winners_fraction: float = 0.0, agent_names_reveal_type=False, n_agents_per_competitor=1, n_configs: int = 10, max_worlds_per_config: int = 100, n_runs_per_world: int = 5, max_n_configs: int | None = None, n_runs_per_config: int | None = None, tournament_path: str | Path | None = None, total_timeout: int | None = None, parallelism='parallel', scheduler_ip: str | None = None, scheduler_port: str | None = None, tournament_progress_callback: Callable[[WorldRunResults | None, int, int], None] | None = None, world_progress_callback: Callable[[World | None], None] | None = None, non_competitors: tuple[str | Any] | None = None, non_competitor_params: tuple[dict[str, Any]] | None = None, dynamic_non_competitors: tuple[str | Any] | None = None, dynamic_non_competitor_params: tuple[dict[str, Any]] | None = None, exclude_competitors_from_reassignment: bool = True, name: str | None = None, verbose: bool = False, configs_only: bool = False, compact: bool = False, print_exceptions: bool = True, metric='median', save_video_fraction: float = 0.0, forced_logs_fraction: float = 0.0, video_params=None, video_saver=None, max_attempts: int = 9223372036854775807, extra_scores_to_use: str | None = None, **kwargs) → TournamentResults | Path

Runs a tournament

Parameters:

• name – Tournament name

• config_generator – Used to generate unique configs that will be used to evaluate competitors

• config_assigner – Used to generate assignments of competitors to the configs created by the config_generator

• world_generator – A functions to generate worlds for the tournament that follows the assignments made by the config_assigner

• score_calculator – A function for calculating the score of all agents in a world After it finishes running. The second parameter is a dict describing any scoring context that may have been added by the world config generator or assigneer. The third parameter is a boolean specifying whether this is a dry_run. For dry runs, scores are not expected but names and types should exist in the returned WorldRunResults.

• competitors – A list of class names for the competitors

• competitor_params – A list of competitor parameters (used to initialize the competitors).

• n_competitors_per_world –

  The number of competitors allowed in every world. It must be >= 1 and <= len(competitors) or None.

  • If None or len(competitors), then all competitors will exist in every world.

  • If 1, then each world will have one competitor

• round_robin – Only effective if 1 < n_competitors_per_world < len(competitors). if True, all combinations will be tried otherwise n_competitors_per_world must divide len(competitors) and every competitor appears only in one set.

• stage_winners_fraction – in [0, 1). Fraction of agents to to go to the next stage at every stage. If zero, and round_robin, it becomes a single stage competition.

• agent_names_reveal_type – If true then the type of an agent should be readable in its name (most likely at its beginning).

• n_configs – The number of different world configs (up to competitor assignment) to be generated.

• max_worlds_per_config – The maximum number of worlds to run per config. If None, then all possible assignments of competitors within each config will be tried (all permutations).

• n_runs_per_world – Number of runs per world. All of these world runs will have identical competitor assignment and identical world configuration.

• n_agents_per_competitor – The number of agents of each competing type to be instantiated in the world.

• max_n_configs – [Depricated] The number of configs to use (it is replaced by separately setting n_config and max_worlds_per_config )

• n_runs_per_config – [Depricated] The number of runs (simulation) for every config. It is replaced by n_runs_per_world

• total_timeout – Total timeout for the complete process

• tournament_path – Path at which to store all results. A new folder with the name of the tournament will be created at this path. A scores.csv file will keep the scores and logs folder will keep detailed logs

• parallelism – Type of parallelism. Can be ‘serial’ for serial, ‘parallel’ for parallel and ‘distributed’ for distributed! For parallel, you can add the fraction of CPUs to use after a colon (e.g. parallel:0.5 to use half of the CPU in the machine). By defaults parallel uses all CPUs in the machine

• scheduler_port – Port of the dask scheduler if parallelism is dask, dist, or distributed

• scheduler_ip – IP Address of the dask scheduler if parallelism is dask, dist, or distributed

• world_progress_callback – A function to be called after every step of every world run (only allowed for serial and parallel evaluation and should be used with cautious).

• tournament_progress_callback – A function to be called with WorldRunResults after each world finished processing

• non_competitors – A list of agent types that will not be competing but will still exist in the world.

• non_competitor_params – paramters of non competitor agents

• dynamic_non_competitors – A list of non-competing agents that are assigned to the simulation dynamically during the creation of the final assignment instead when the configuration is created

• dynamic_non_competitor_params – paramters of dynamic non competitor agents

• exclude_competitors_from_reassignment – If true, competitors are excluded from the dyanamic non-competitors

• verbose – Verbosity

• configs_only – If true, a config file for each

• compact – If true, compact logs will be created and effort will be made to reduce the memory footprint

• print_exceptions – If true, print all exceptions to screen

• metric – The metric to use for evaluation

• save_video_fraction – The fraction of simulations for which to save videos

• forced_logs_fraction – The fraction of simulations for which to always save logs. Notice that this has no effect except if no logs were to be saved otherwise (i.e. no_logs is passed as True)

• video_params – The parameters to pass to the video saving function

• video_saver – The parameters to pass to the video saving function after the world

• max_attempts – The maximum number of times to retry running simulations

• extra_scores_to_use – The type of extra-scores to use. If None normal scores will be used. Only effective if scores is None.

• kwargs – Arguments to pass to the config_generator function

Returns:

TournamentResults The results of the tournament or a Path giving the location where configs were saved