negmas.tournaments

Tournament generation and management.

class negmas.tournaments.SimpleTournamentResults(scores: DataFrame, details: DataFrame, scores_summary: DataFrame, final_scores: DataFrame, path: Path | None = None)

Bases: object

SimpleTournamentResults implementation.

classmethod combine(paths: Path | Iterable[Path], recursive: bool = True, recalc_details: bool = True, recalc_scores: bool = False, must_have_details: bool = False, verbosity: int = 1, final_score_stat: tuple[str, str] = ('advantage', 'mean'), add_tournament_column: bool = True, complete_only: bool = True) → tuple[SimpleTournamentResults, list[Path]]

Combines the results of multiple tournaments stored on disk

Parameters:

• paths – Paths to look for results within

• recursive – Check children of given paths recursively

• recalc_details – Recalculate detailed results from the negotiations folder

• recalc_scores – Recalculate scores from detailed negotiation results

• must_have_details – Raise an exception if detailed negotiation results cannot be found

• verbosity – Verbosity level

• final_score_stat – Used to calculate the final scores. See cartesian_tournament for details.

• add_tournament_column – Add a column called tournament with tournament name in detailed and scores.

• complete_only – If given, only a completed tournament will be used in the combination. The rest are ignored.

Raises:

FileNotFoundError – If a needed file is not found

Returns:

A newly constructed SimpleTournamentResults with the combined results of all tournaments

details: DataFrame

All negotiation results

final_scores: DataFrame

A list of negotiators and their final scores sorted from highest (winner) to lowest score

classmethod from_records(scores: list[dict[str, Any]] | DataFrame | None = None, results: list[dict[str, Any]] | DataFrame | None = None, type_scores: DataFrame | None = None, final_scores: DataFrame | None = None, final_score_stat: tuple[str, str] = ('advantage', 'mean'), path: Path | None = None) → SimpleTournamentResults

Creates SimpleTournamentResults from records of results

Parameters:

• scores – The scores of negotiators in all negotiations (If not given, results can be used to calculate it).

• results – Results of all negotiations (If not given, the resulting SimpleTournamentResults object will lack details)

• type_scores – Optionally, type-scores. If not given, it will be calculated from scores

• final_scores – Optionally, final scores. If not given, final_scoer_stat will be used to calculate them

• final_score_stat – A tuple of the measure used and the statistic applied to it for calculating final score. See cartesian_tournament for more details

• path – The path in which the data for this tournament is stored.

Raises:

ValueError – If no scores or results are given

Returns:

A new SimpleTournamentResults with the given data

classmethod from_result_records(path: Path, verbosity: int = 1, final_score_stat: tuple[str, str] = ('advantage', 'mean')) → SimpleTournamentResults

From result records.

Parameters:

• path – Path.

• verbosity – Verbosity.

• final_score_stat – Final score stat.

Returns:

The result.

Return type:

‘SimpleTournamentResults’

classmethod load(path: Path, must_have_details: bool = False) → SimpleTournamentResults

Loads results from the given path

path: Path | None

Location at which the logs are stored

save(path: Path | None, exist_ok: bool = True) → None

Save all results to the given path

scores: DataFrame

All scores per negotiator

scores_summary: DataFrame

All score statistics summarized per type

class negmas.tournaments.TournamentResults(scores: DataFrame, total_scores: DataFrame, winners: list[str], winners_scores: ndarray, ttest: DataFrame | None = None, kstest: DataFrame | None = None, stats: DataFrame | None = None, agg_stats: DataFrame | None = None, score_stats: DataFrame | None = None, path: str | Path | None = None, world_stats: DataFrame | None = None, type_stats: DataFrame | None = None, agent_stats: DataFrame | None = None, params: dict[str, Any] | None = None, extra_scores: dict[str, DataFrame] | None = None)

Bases: object

TournamentResults implementation.

agent_stats: DataFrame | None = None

Some statistics about each agent

agg_stats: DataFrame | None = None

Aggregated stats per world

extra_scores: dict[str, DataFrame] | None = None

Extra scores returned from the scoring function. This can be used to have multi-dimensional scoring

kstest: DataFrame | None = None

Results of the nonparametric kstest

params: dict[str, Any] | None = None

Parameters of the tournament

path: str | Path | None = None

Path at which tournament results are stored

score_stats: DataFrame | None = None

Score statistics for different competitor types

scores: DataFrame

Scores of individual agent instantiations

stats: DataFrame | None = None

Stats of all worlds

total_scores: DataFrame

Total scores collected by competitor types

ttest: DataFrame | None = None

Results of ttest analysis of the scores

type_stats: DataFrame | None = None

Some statistics about each type

winners: list[str]

Winner type name(s) which may be a list

winners_scores: ndarray

Winner score (accumulated)

world_stats: DataFrame | None = None

Some statistics about each world run

class negmas.tournaments.WorldGenerator(*args, **kwargs)

Bases: Protocol

A callback-protocol specifying the signature of a world generator function that can be passed to tournament

Parameters:

kwargs – key-value pairs of arguments.

See also

tournament

class negmas.tournaments.WorldRunResults(world_names: list[str], log_file_names: list[str])

Bases: object

Results of a world run

extra_scores: dict[str, list[dict[str, Any]]]

The extra-scores (i.e. extra evaluation metrics). Each is a list of records

ids: list[str]

Agent IDs

log_file_names: list[str]

Log file names

names: list[str]

Agent names

scores: list[float]

Agent scores

types: list[str]

Agent type names

world_names: list[str]

World names (there can be multiple worlds for each scoring call)

negmas.tournaments.cartesian_tournament(competitors: list[type[~negmas.negotiators.negotiator.Negotiator] | str] | tuple[type[~negmas.negotiators.negotiator.Negotiator] | str, ...], scenarios: list[~negmas.inout.Scenario] | tuple[~negmas.inout.Scenario, ...], private_infos: list[None | tuple[dict, ...]] | None = None, competitor_params: ~typing.Sequence[dict | None] | None = None, rotate_ufuns: bool = True, rotate_private_infos: bool = True, n_repetitions: int = 1, path: ~pathlib.Path | None = None, njobs: int = 0, mechanism_type: type[~negmas.mechanisms.Mechanism] = <class 'negmas.sao.mechanism.SAOMechanism'>, mechanism_params: dict[str, ~typing.Any] | None = None, n_steps: int | tuple[int, int] | None = 100, time_limit: float | tuple[float, float] | None = None, pend: float | tuple[float, float] = 0.0, pend_per_second: float | tuple[float, float] = 0.0, step_time_limit: float | tuple[float, float] | None = None, negotiator_time_limit: float | tuple[float, float] | None = None, hidden_time_limit: float | tuple[float, float] | None = None, external_timeout: int | None = None, plot_fraction: float = 0.0, plot_params: dict[str, ~typing.Any] | None = None, verbosity: int = 1, self_play: bool = True, randomize_runs: bool = True, sort_runs: bool = False, save_every: int = 0, save_stats: bool = True, save_scenario_figs: bool = True, final_score: tuple[str, str] = ('advantage', 'mean'), id_reveals_type: bool = False, name_reveals_type: bool = True, shorten_names: bool = True, raise_exceptions: bool = True, mask_scenario_names: bool = True, only_failures_on_self_play: bool = False, python_class_identifier='__python_class__') → SimpleTournamentResults

A simplified version of Cartesian tournaments not using the internal machinay of NegMAS tournaments

Parameters:

• competitors – A tuple of the competing negotiator types.

• scenarios – A tuple of base scenarios to use for the tournament.

• competitor_params – Either None for no-parameters or a tuple of dictionaries with parameters to initialize the competitors (in order).

• private_infos – If given, a list of the same length as scenarios. Each item is a tuple giving the private information to be passed to every negotiator in every scenario.

• rotate_ufuns – If True, the ufuns will be rotated over negotiator positions (for bilateral negotiation this leads to two scenarios for each input scenario with reversed ufun order).

• rotate_private_infos – If True and rotate_ufuns is also True, private information will be rotated with the utility functions.

• n_repetitions – Number of times to repeat each scenario/partner combination

• path – Path on disk to save the results and details of this tournament. Pass None to disable logging

• n_jobs – Number of parallel jobs to run. -1 means running serially (useful for debugging) and 0 means using all cores.

• mechanism_type – The mechanism (protocol) used for all negotiations.

• n_steps – Number of steps/rounds allowed for the each negotiation (None for no-limit and a 2-valued tuple for sampling from a range)

• time_limit – Number of seconds allowed for the each negotiation (None for no-limit and a 2-valued tuple for sampling from a range)

• pend – Probability of ending the negotiation every step/round (None for no-limit and a 2-valued tuple for sampling from a range)

• pend_per_second – Probability of ending the negotiation every second (None for no-limit and a 2-valued tuple for sampling from a range)

• step_time_limit – Time limit for every negotiation step (None for no-limit and a 2-valued tuple for sampling from a range)

• negotiator_time_limit – Time limit for all actions of every negotiator (None for no-limit and a 2-valued tuple for sampling from a range)

• hidden_time_limit – Time limit for negotiations that is not known to the negotiators

• external_timeout – A timeout applied directly to reception of results from negotiations in parallel runs only.

• mechanism_params – Parameters of the mechanism (protocol). Usually you need to pass one or more of the following: time_limit (in seconds), n_steps (in rounds), p_ending (probability of ending the negotiation every step).

• plot_fraction – fraction of negotiations for which plots are to be saved (only if path is not None)

• plot_params – Parameters to pass to the plotting function

• verbosity – Verbosity level (minimum is 0)

• self_play – Allow negotiations in which all partners are of the same type

• only_failures_on_self_play – If given, self-play runs will only be recorded if they fail to reach agreement. This is useful if you want to keep self-play but still penalize strategies for failing to reach agreements in self-play

• randomize_runs – If True negotiations will be run in random order, otherwise each scenario/partner combination will be finished before starting on the next

• save_every – Number of negotiations after which we dump details and scores

• save_stats – Whether to calculate and save extra statistics like pareto_optimality, nash_optimality, kalai-smorodinsky optimality (ks_optimality), kalai_optimality, etc

• save_scenario_figs – Whether to save a png of the scenario represented in the utility domain for every scenario.

• final_score – A tuple of two strings giving the metric used for ordering the negotiators for the final score: First string can be one of the following (advantage, utility, partner_welfare, welfare) or any statistic from the set calculated if save_stats is True. The second string can be mean, median, min, max, or std. The default is (‘advantage’, ‘mean’)

• id_reveals_type – Each negotiator ID will reveal its type.

• name_reveals_type – Each negotiator name will reveal its type.

• shorten_names – If True, shorter versions of names will be used for results

• raise_exceptions – When given, negotiators and mechanisms are allowed to raise exceptions stopping the tournament

• mask_scenario_names – If given, scenario names will be masked so that the negotiators do not know the original scenario name

Returns:

A pandas DataFrame with all negotiation results.

negmas.tournaments.combine_tournament_results(sources: Iterable[str | Path], dest: str | Path | None = None, verbose=False, max_sources: int | None = None) → DataFrame

Combines results of several tournament runs in the destination path.

negmas.tournaments.combine_tournament_stats(sources: Iterable[str | Path], dest: str | Path | None = None, verbose=False, max_sources: int | None = None) → DataFrame

Combines statistical results of several tournament runs in the destination path.

negmas.tournaments.combine_tournaments(sources: Iterable[str | Path], dest: str | Path, verbose=False) → tuple[int, int]

Combines contents of several tournament runs in the destination path allowing for continuation of the tournament

Parameters:

• sources – The sources of tournaments in the filesystem

• dest – where to store the combined tournament.

Returns:

Tuple[int, int] The number of base configs and assigned configs combined

negmas.tournaments.create_tournament(competitors: Sequence[str | type[Agent]], config_generator: ConfigGenerator, config_assigner: ConfigAssigner, world_generator: WorldGenerator, score_calculator: Callable[[list[World], dict[str, Any], bool], WorldRunResults], competitor_params: Sequence[dict[str, Any]] | None = None, n_competitors_per_world: int | None = None, round_robin: bool = True, agent_names_reveal_type=False, n_agents_per_competitor=1, n_configs: int = 10, max_worlds_per_config: int | None = 100, n_runs_per_world: int = 5, max_n_configs: int | None = None, n_runs_per_config: int | None = None, base_tournament_path: Path | str | None = None, total_timeout: int | None = None, parallelism='parallel', scheduler_ip: str | None = None, scheduler_port: str | None = None, non_competitors: tuple[str | Any, ...] | None = None, non_competitor_params: tuple[dict[str, Any], ...] | None = None, dynamic_non_competitors: tuple[str | Any, ...] | None = None, dynamic_non_competitor_params: tuple[dict[str, Any], ...] | None = None, exclude_competitors_from_reassignment: bool = True, name: str | None = None, verbose: bool = False, compact: bool = False, save_video_fraction: float = 0.0, forced_logs_fraction: float = 0.0, video_params=None, video_saver=None, **kwargs) → Path

Creates a tournament

Parameters:

• name – Tournament name

• config_generator – Used to generate unique configs that will be used to evaluate competitors

• config_assigner – Used to generate assignments of competitors to the configs created by the config_generator

• world_generator – A functions to generate worlds for the tournament that follows the assignments made by the config_assigner

• score_calculator – A function for calculating the score of all agents in a world After it finishes running. The second parameter is a dict describing any scoring context that may have been added by the world config generator or assigneer. The third parameter is a boolean specifying whether this is a dry_run. For dry runs, scores are not expected but names and types should exist in the returned WorldRunResults.

• competitors – A list of class names for the competitors

• competitor_params – A list of competitor parameters (used to initialize the competitors).

• n_competitors_per_world –

  The number of competitors allowed in every world. It must be >= 1 and <= len(competitors) or None.

  • If None or len(competitors), then all competitors will exist in every world.

  • If 1, then each world will have one competitor

• round_robin – Only effective if 1 < n_competitors_per_world < len(competitors). if True, all combinations will be tried otherwise n_competitors_per_world must divide len(competitors) and every competitor appears only in one set.

• agent_names_reveal_type – If true then the type of an agent should be readable in its name (most likely at its beginning).

• n_configs – The number of different world configs (up to competitor assignment) to be generated.

• max_worlds_per_config – The maximum number of worlds to run per config. If None, then all possible assignments of competitors within each config will be tried (all permutations).

• n_runs_per_world – Number of runs per world. All of these world runs will have identical competitor assignment and identical world configuration.

• n_agents_per_competitor – The number of agents of each competing type to be instantiated in the world.

• max_n_configs – [Depricated] The number of configs to use (it is replaced by separately setting n_config and max_worlds_per_config )

• n_runs_per_config – [Depricated] The number of runs (simulation) for every config. It is replaced by n_runs_per_world

• total_timeout – Total timeout for the complete process

• base_tournament_path – Path at which to store all results. A new folder with the name of the tournament will be created at this path. A scores.csv file will keep the scores and logs folder will keep detailed logs

• parallelism – Type of parallelism. Can be ‘serial’ for serial, ‘parallel’ for parallel and ‘distributed’ for distributed! For parallel, you can add the fraction of CPUs to use after a colon (e.g. parallel:0.5 to use half of the CPU in the machine). By defaults parallel uses all CPUs in the machine

• scheduler_port – Port of the dask scheduler if parallelism is dask, dist, or distributed

• scheduler_ip – IP Address of the dask scheduler if parallelism is dask, dist, or distributed

• non_competitors – A list of agent types that will not be competing but will still exist in the world.

• non_competitor_params – paramters of non competitor agents

• dynamic_non_competitors – A list of non-competing agents that are assigned to the simulation dynamically during the creation of the final assignment instead when the configuration is created

• dynamic_non_competitor_params – paramters of dynamic non competitor agents

• exclude_competitors_from_reassignment – If true, copmetitors are not included in the reassignment even if they exist in dynamic_non_competitors

• verbose – Verbosity

• compact – If true, compact logs will be created and effort will be made to reduce the memory footprint

• save_video_fraction – The fraction of simulations for which to save videos

• forced_logs_fraction – The fraction of simulations for which to always save logs. Notice that this has no effect except if no logs were to be saved otherwise (i.e. no_logs is passed as True)

• video_params – The parameters to pass to the video saving function

• video_saver – The parameters to pass to the video saving function after the world

• kwargs – Arguments to pass to the config_generator function

Returns:

The path at which tournament configs are stored

negmas.tournaments.evaluate_tournament(tournament_path: str | Path | None, scores: DataFrame | None = None, stats: DataFrame | None = None, world_stats: DataFrame | None = None, type_stats: DataFrame | None = None, agent_stats: DataFrame | None = None, metric: str | Callable[[DataFrame], float] = 'mean', verbose: bool = False, recursive: bool = True, extra_scores_to_use: str | None = None, compile: bool = True) → TournamentResults

Evaluates the results of a tournament

Parameters:

• tournament_path – Path to save the results to. If scores is not given, it is also used as the source of scores. Pass None to avoid saving the results to disk.

• scores – Optionally the scores of all agents in all world runs. If not given they will be read from the file scores.csv in tournament_path

• stats – Optionally the stats of all world runs. If not given they will be read from the file stats.csv in tournament_path

• world_stats – Optionally the aggregate stats collected in WorldSetRunStats for each world set

• type_stats – Optionally the aggregate stats collected in AgentStats for each agent type

• agent_stats – Optionally the aggregate stats collected in AgentStats for each agent instance

• metric – The metric used for evaluation. Possibilities are: mean, median, std, var, sum, truncated_mean or a callable that receives a pandas data-frame and returns a float.

• verbose – If true, the winners will be printed

• recursive – If true, ALL scores.csv files in all subdirectories of the given tournament_path will be combined

• extra_scores_to_use – The type of extra-scores to use. If None normal scores will be used. Only effective if scores is None.

• compile – Takes effect only if tournament_path is not None. If true, the results will be recompiled from individual world results. This is accurate but slow. If false, it will be assumed that all results are already compiled.

• independent_test (#) – True if you want an independent t-test

Returns:

negmas.tournaments.process_world_run(run_id: str, results: WorldRunResults | None, tournament_name: str) → tuple[list[dict[str, Any]], dict[str, list[dict[str, Any]]]]

Generates a data-frame with the results of this world run

Parameters:

• run_id – The ID of this run (should be unique per tournament)

• results – Results of the world run

• tournament_name – tournament name

Returns:

• A list of records containing scores

• A dict mapping extra-score types to lists of records for this type.

Return type:

A tuple of two items

Remarks:

The score calculator returns a WorldRunResults object which must contain a scores element used for evaluating the agents. It can also return extra_scores that can be used to save additional information about agent performance. These are optional and the second output of this function will be the processed version of these extra scores if any.

negmas.tournaments.run_negotiation(s: ~negmas.inout.Scenario, partners: tuple[type[~negmas.negotiators.negotiator.Negotiator]], partner_names: tuple[str] | None = None, partner_params: tuple[dict[str, ~typing.Any]] | None = None, rep: int = 0, path: ~pathlib.Path | None = None, mechanism_type: type[~negmas.mechanisms.Mechanism] = <class 'negmas.sao.mechanism.SAOMechanism'>, mechanism_params: dict[str, ~typing.Any] | None = None, full_names: bool = True, verbosity: int = 0, plot=False, plot_params: dict[str, ~typing.Any] | None = None, run_id: int | str | None = None, stats: ~negmas.preferences.ops.ScenarioStats | None = None, annotation: dict[str, ~typing.Any] | None = None, private_infos: tuple[dict[str, ~typing.Any] | None] | None = None, id_reveals_type: bool = False, name_reveals_type: bool = True, mask_scenario_name: bool = True, ignore_exceptions: bool = False) → dict[str, Any]

Run a single negotiation with fully specified parameters

Parameters:

• s – The Scenario representing the negotiation (outcome space and preferences).

• partners – The partners running the negotiation in order of addition to the mechanism.

• real_scenario_name – The real name of the scenario (used when saving logs).

• partner_names – Names of partners. Either None for defaults or a tuple of the same length as partners

• partner_params – Parameters used to create the partners. Either None for defaults or a tuple of the same length as partners

• rep – The repetition number for this run of the negotiation

• path – A folder to save the logs into. If not given, no logs will be saved.

• mechanism_type – the type of the Mechanism to use for this negotiation

• mechanism_params – The parameters used to create the Mechanism or None for defaults

• full_names – Use full names for partner names (only used if partner_names is None)

• verbosity – Verbosity level as an integer

• plot – If true, save a plot of the negotiation (only if path is given)

• plot_params – Parameters to pass to the plotting function

• run_id – A unique ID for this run. If not given one is generated based on date and time

• stats – statistics of the scenario. If not given or path is None, statistics are not saved

• annotation – Common information saved in the mechanism’s annotation (accessible by negotiators using self.nmi.annotation). None for nothing

• private_infos – Private information saved in the negotiator’s private_info attribute (accessible by negotiators as self.private_info). None for nothing

• id_reveals_type – Each negotiator ID will reveal its type.

• name_reveals_type – Each negotiator name will reveal its type.

Returns:

A dictionary of negotiation results that contains the final state of the negotiation alongside other information

negmas.tournaments.run_tournament(tournament_path: str | Path, world_generator: WorldGenerator | None = None, score_calculator: Callable[[list[World], dict[str, Any], bool], WorldRunResults] | None = None, total_timeout: int | None = None, parallelism='parallel', scheduler_ip: str | None = None, scheduler_port: str | None = None, tournament_progress_callback: Callable[[WorldRunResults | None, int, int], None] | None = None, world_progress_callback: Callable[[World | None], None] | None = None, verbose: bool = False, compact: bool | None = None, print_exceptions: bool = True, override_ran_worlds: bool = False, max_attempts: int = 9223372036854775807) → None

Runs a tournament

Parameters:

• tournament_path – Path at which configs of this tournament are stored

• world_generator – A functions to generate worlds for the tournament that follows the assignments made by the config_assigner

• score_calculator – A function for calculating the score of all agents in a world After it finishes running. The second parameter is a dict describing any scoring context that may have been added by the world config generator or assigner. The third parameter is a boolean specifying whether this is a dry_run. For dry runs, scores are not expected but names and types should exist in the returned WorldRunResults.

• total_timeout – Total timeout for the complete process

• parallelism – Type of parallelism. Can be ‘serial’ for serial, ‘parallel’ for parallel and ‘distributed’ for distributed! For parallel, you can add the fraction of CPUs to use after a colon (e.g. parallel:0.5 to use half of the CPU in the machine). By defaults parallel uses all CPUs in the machine

• scheduler_port – Port of the dask scheduler if parallelism is dask, dist, or distributed

• scheduler_ip – IP Address of the dask scheduler if parallelism is dask, dist, or distributed

• world_progress_callback – A function to be called after every step of every world run (only allowed for serial and parallel evaluation and should be used with cautious).

• tournament_progress_callback – A function to be called with WorldRunResults after each world finished processing

• verbose – Verbosity

• compact – If true, compact logs will be created and effort will be made to reduce the memory footprint

• print_exceptions – If true, exceptions encountered during world simulation will be printed to stdout

• override_ran_worlds – If true worlds that are already ran will be ran again

• max_attempts – The maximum number of attempts to run each simulation. Default is infinite

negmas.tournaments.run_world(world_params: dict, dry_run: bool = False, save_world_stats: bool = True, attempts_path=None, max_attempts=inf, verbose=False) → tuple[str, list[str], WorldRunResults | None, WorldSetRunStats | None, AgentStats | None, AgentStats | None]

Runs a world and returns stats. This function is designed to be used with distributed systems like dask.

Parameters:

• world_params – World info dict. See remarks for its parameters

• dry_run – If true, the world will not be run. Only configs will be saved

• save_world_stats – If true, saves individual world stats

• attempts_path – The folder containing attempts information

• max_attempts – The maximum number of trials to run a world simulation

Remarks:

The world_params dict should have the following members:

• name: world name [Defaults to random]

• log_file_name: file name to store the world log [Defaults to random]

• __dir_name: directory to store the world stats [Defaults to random]

• __world_generator: full name of the world generator function (including its module) [Required]

• __score_calculator: full name of the score calculator function [Required]

• __tournament_name: name of the tournament [Defaults to random]

• others: values of all other keys are passed to the world generator as kwargs

negmas.tournaments.tournament(competitors: list[str | Agent] | tuple[str | Agent, ...] | Sequence[str | Agent], config_generator: ConfigGenerator, config_assigner: ConfigAssigner, world_generator: WorldGenerator, score_calculator: Callable[[list[World], dict[str, Any], bool], WorldRunResults], competitor_params: Sequence[dict[str, Any]] | None = None, n_competitors_per_world: int | None = None, round_robin: bool = False, stage_winners_fraction: float = 0.0, agent_names_reveal_type=False, n_agents_per_competitor=1, n_configs: int = 10, max_worlds_per_config: int = 100, n_runs_per_world: int = 5, max_n_configs: int | None = None, n_runs_per_config: int | None = None, tournament_path: str | Path | None = None, total_timeout: int | None = None, parallelism='parallel', scheduler_ip: str | None = None, scheduler_port: str | None = None, tournament_progress_callback: Callable[[WorldRunResults | None, int, int], None] | None = None, world_progress_callback: Callable[[World | None], None] | None = None, non_competitors: tuple[str | Any] | None = None, non_competitor_params: tuple[dict[str, Any]] | None = None, dynamic_non_competitors: tuple[str | Any] | None = None, dynamic_non_competitor_params: tuple[dict[str, Any]] | None = None, exclude_competitors_from_reassignment: bool = True, name: str | None = None, verbose: bool = False, configs_only: bool = False, compact: bool = False, print_exceptions: bool = True, metric='median', save_video_fraction: float = 0.0, forced_logs_fraction: float = 0.0, video_params=None, video_saver=None, max_attempts: int = 9223372036854775807, extra_scores_to_use: str | None = None, **kwargs) → TournamentResults | Path

Runs a tournament

Parameters:

• name – Tournament name

• config_generator – Used to generate unique configs that will be used to evaluate competitors

• config_assigner – Used to generate assignments of competitors to the configs created by the config_generator

• world_generator – A functions to generate worlds for the tournament that follows the assignments made by the config_assigner

• score_calculator – A function for calculating the score of all agents in a world After it finishes running. The second parameter is a dict describing any scoring context that may have been added by the world config generator or assigneer. The third parameter is a boolean specifying whether this is a dry_run. For dry runs, scores are not expected but names and types should exist in the returned WorldRunResults.

• competitors – A list of class names for the competitors

• competitor_params – A list of competitor parameters (used to initialize the competitors).

• n_competitors_per_world –

  The number of competitors allowed in every world. It must be >= 1 and <= len(competitors) or None.

  • If None or len(competitors), then all competitors will exist in every world.

  • If 1, then each world will have one competitor

• round_robin – Only effective if 1 < n_competitors_per_world < len(competitors). if True, all combinations will be tried otherwise n_competitors_per_world must divide len(competitors) and every competitor appears only in one set.

• stage_winners_fraction – in [0, 1). Fraction of agents to to go to the next stage at every stage. If zero, and round_robin, it becomes a single stage competition.

• agent_names_reveal_type – If true then the type of an agent should be readable in its name (most likely at its beginning).

• n_configs – The number of different world configs (up to competitor assignment) to be generated.

• max_worlds_per_config – The maximum number of worlds to run per config. If None, then all possible assignments of competitors within each config will be tried (all permutations).

• n_runs_per_world – Number of runs per world. All of these world runs will have identical competitor assignment and identical world configuration.

• n_agents_per_competitor – The number of agents of each competing type to be instantiated in the world.

• max_n_configs – [Depricated] The number of configs to use (it is replaced by separately setting n_config and max_worlds_per_config )

• n_runs_per_config – [Depricated] The number of runs (simulation) for every config. It is replaced by n_runs_per_world

• total_timeout – Total timeout for the complete process

• tournament_path – Path at which to store all results. A new folder with the name of the tournament will be created at this path. A scores.csv file will keep the scores and logs folder will keep detailed logs

• parallelism – Type of parallelism. Can be ‘serial’ for serial, ‘parallel’ for parallel and ‘distributed’ for distributed! For parallel, you can add the fraction of CPUs to use after a colon (e.g. parallel:0.5 to use half of the CPU in the machine). By defaults parallel uses all CPUs in the machine

• scheduler_port – Port of the dask scheduler if parallelism is dask, dist, or distributed

• scheduler_ip – IP Address of the dask scheduler if parallelism is dask, dist, or distributed

• world_progress_callback – A function to be called after every step of every world run (only allowed for serial and parallel evaluation and should be used with cautious).

• tournament_progress_callback – A function to be called with WorldRunResults after each world finished processing

• non_competitors – A list of agent types that will not be competing but will still exist in the world.

• non_competitor_params – paramters of non competitor agents

• dynamic_non_competitors – A list of non-competing agents that are assigned to the simulation dynamically during the creation of the final assignment instead when the configuration is created

• dynamic_non_competitor_params – paramters of dynamic non competitor agents

• exclude_competitors_from_reassignment – If true, competitors are excluded from the dyanamic non-competitors

• verbose – Verbosity

• configs_only – If true, a config file for each

• compact – If true, compact logs will be created and effort will be made to reduce the memory footprint

• print_exceptions – If true, print all exceptions to screen

• metric – The metric to use for evaluation

• save_video_fraction – The fraction of simulations for which to save videos

• forced_logs_fraction – The fraction of simulations for which to always save logs. Notice that this has no effect except if no logs were to be saved otherwise (i.e. no_logs is passed as True)

• video_params – The parameters to pass to the video saving function

• video_saver – The parameters to pass to the video saving function after the world

• max_attempts – The maximum number of times to retry running simulations

• extra_scores_to_use – The type of extra-scores to use. If None normal scores will be used. Only effective if scores is None.

• kwargs – Arguments to pass to the config_generator function

Returns:

TournamentResults The results of the tournament or a Path giving the location where configs were saved