NegMAS CLI
When installing NegMAS through the pip command, you get two command line tool that can be used to aid your development and testing.
This section describes the negmas
CLI which provides a unified interface to all negmas commands.
The set of supported commands are:
Command |
Meaning |
---|---|
genius-setup |
Downloads the genius bridge and updates your settings. |
genius |
Run a Genius Bridge. This bridge allows you to use GeniusNegotiator agents. Please notice that this command by-default runs in the foreground preventing further input to the terminal. |
tournament |
Runs a tournament |
version |
Prints NegMAS version |
The commands genius-steup
have no parameters and will download genius for later use by the genius
command.
Genius Bridge (negmas genius)
The command genius
can be used to start a JVM running the Genius platform allowing GeniusNegotiator
objects
to interact with existing GENIUS agents (Thanks for Tim Baarslag Lead Developer of GENIUS for allowing us
to ship it within NegMAS).
You can get help on this tool by running:
$ negmas genius --help
This tool supports the following optional arguments:
Argument |
Meaning |
---|---|
-p/–path TEXT |
Path to genius-8.0.4.jar with embedded NegLoader [OPTIONAL] |
-r/–port INTEGER |
Port to run the NegLoader on. Pass 0 for the default value [OPTIONAL] |
–force/–no-force |
Force trial even if an earlier instance exists [OPTIONAL] |
–help |
Show help message and exit. |
Tournament Command (negmas tournament)
The Tournament command (tournament
) allows you to run a tournament between different agents in some world and
compare their relative performance. The tool is general enough to support several world types.
You can get help on this tool by running:
$ negmas tournament --help
The tournament
command has a set of sub-commands for creating, running, and combining tournament results as follows:
Command |
Action |
---|---|
combine |
Finds winners of an arbitrary set of tournaments |
create |
Creates a tournament |
run |
Runs/continues a tournament |
winners |
Finds winners of a tournament or a set of tournaments sharing |
Creating a tournament
These are the optional arguments of this tool:
Argument |
Meaning |
---|---|
-n, –name TEXT |
The name of the tournament. The special value “random” will result in a random name [default: random] |
-s, –steps INTEGER |
Number of steps. If passed then –steps-min and –steps-max are ignored |
–steps-min INTEGER |
Minimum number of steps (only used if –steps was not passed [default: 50] |
–steps-max INTEGER |
Maximum number of steps (only used if –steps was not passed [default: 100] |
-t, –timeout INTEGER |
Timeout the whole tournament after the given number of seconds (0 for infinite) [default: 0] |
–configs INTEGER |
Number of unique configurations to generate. [default: 5] |
–runs INTEGER |
Number of runs for each configuration [default: 2] |
–max-runs INTEGER |
Maximum total number of runs. Zero or negative numbers mean no limit [default: -1] |
–agents INTEGER |
Number of agents per competitor (not used for anac2019std in which this is preset to 1). [default: 3] |
–factories INTEGER |
Minimum numbers of factories to have per level. [default: 5] |
–competitors TEXT |
A semicolon (;) separated list of agent types to use for the competition. |
–jcompetitors, –java-competitors TEXT |
A semicolon (;) separated list of agent types to use for the competition. |
–non-competitors TEXT |
A semicolon (;) separated list of agent types to exist in the worlds as non- competitors (their scores will not be calculated). |
-l, –log DIRECTORY |
Default location to save logs (A folder will be created under it) [default: ~/negmas/logs/tournaments] |
–world-config FILE |
A file to load extra configuration parameters for world simulations from. |
–verbosity INTEGER |
verbosity level (from 0 == silent to 1 == world progress) [default: 1] |
–reveal-names / –hidden-names |
Reveal agent names (should be used only for debugging) [default: True] |
–log-ufuns / –no-ufun-logs |
Log ufuns into their own CSV file. Only effective if –debug is given [default: False] |
–log-negs / –no-neg-logs |
Log all negotiations. Only effective if –debug is given [default: False] |
–compact / –debug |
If True, effort is exerted to reduce the memory footprint whichincludes reducing logs dramatically. [default: True] |
–raise-exceptions / –ignore-exceptions |
Whether to ignore agent exceptions [default: True] |
–path TEXT |
A path to be added to PYTHONPATH in which all competitors are stored. You can path a : separated list of paths on linux/mac and a ; separated list in windows |
–java-interop-class TEXT |
The full name of a class that is used to represent Java agents to the python envirnment. It is only used if jcompetitors was passed |
–config-generator TEXT |
The full path to a configuration generator function that is used to generate all configs for the tournament. MUST be specified |
–world-generator TEXT |
The full path to a world generator function that is used to generate all worlds (given the assigned configs for the tournament. MUST be specified |
–assigner TEXT |
The full path to an assigner function that assigns competitors to different configurations |
–scorer TEXT |
The full path to a scoring function |
–cw INTEGER |
Number of competitors to run at every world simulation. It must either be left at default or be a number > 1 and < the number of competitors passed using –competitors |
–config FILE |
Read configuration from FILE. |
Running a tournament
After creating a tournament using the tournament create
command, it can be run using the tournament run
command.
The parameters for this command are:
Argument |
Meaning |
---|---|
-n, –name TEXT |
The name of the tournament. When invoked after create, there is no need to pass it |
-l, –log DIRECTORY |
Default location to save logs [default: ~/negmas/logs/tournaments] |
–verbosity INTEGER |
verbosity level (from 0 == silent to 1 == world progress) [default: 1] |
–parallel / –serial |
Run a parallel/serial tournament on a single machine [default: True] |
–distributed / –single-machine |
Run a distributed tournament using dask [default: False] |
–ip TEXT |
The IP address for a dask scheduler to run the distributed tournament. Effective only if –distributed [default: 127.0.0.1] |
–port INTEGER |
The IP port number a dask scheduler to run the distributed tournament. Effective only if –distributed [default: 8786] |
–compact / –debug |
If True, effort is exerted to reduce the memory footprint whichincludes reducing logs dramatically. [default: True] |
–path TEXT |
A path to be added to PYTHONPATH in which all competitors are stored. You can path a : separated list of paths on linux/mac and a ; separated list in windows |
–metric TEXT |
The statistical metric used for choosing the winners. Possibilities are mean, median, std, var, sum [default: mean] |
–config FILE |
Read configuration from FILE. |
Upon completion, a complete log and several statistics are saved in a new folder under the log folder
location
specified by the --log
argument (default is negmas/logs/tournaments under the HOME directory). To avoid over-writing
earlier results, a new folder will be created for each run named by the current date and time. The
folder will contain the following files:
File/Folder Name |
Format |
Content |
---|---|---|
configs |
FOLDER |
Contains one json file for each world
run tried during the tournament. You can
re-run this world using |
params.json |
JSON |
The parameters used to create this tournament |
base_configs.json |
JSON |
The base configurations used in the tournament (without agent/factory assignments. |
assigned_configs.json |
JSON |
The configurations used after assigning factories to managers |
scores.csv |
CSV |
Scores of every agent in every world |
total_scores.csv |
CSV |
Scores of every agent type averaged over all runs |
winners.csv |
CSV |
Winner types and their average scores |
ttest.csv |
CSV |
Results of a factorial TTEST comparing the performance of all agent types |
Other than these files, a folder with the same number as the corresponding config file in the configs folder, keeps full
statistics/log of every world but only if –debug is specified (see the World Runner
section for the contents of
this folder.
Combining tournament results
Can be used to combine the results of multiple tournaments runs using tournament combine
.
The parameters of this command are:
Argument |
Meaning |
---|---|
-d, –dest DIRECTORY |
The location to save the results |
–metric TEXT |
The statistical metric used for choosing the winners. Possibilities are mean, median, std, var, sum [default: median] |
–config FILE |
Read configuration from FILE. |
Finding the winners of a tournament
To report the winners of a tournament, you can use tournament winners
. The parameters of this command are:
Argument |
Meaning |
---|---|
-n, –name TEXT |
The name of the tournament. When invoked after create, there is no need to pass it |
-l, –log DIRECTORY |
Default location to save logs [default: ~/negmas/logs/tournaments] |
–recursive / –no-recursive |
Whether to recursively look for tournament results. –name should not be given if –recursive [default: True] |
–metric TEXT |
The statistical metric used for choosing the winners. Possibilities are mean, median, std, var, sum [default: median] |
–config FILE |
Read configuration from FILE. |
Negotiate CLI
When installing NegMAS through the pip command, you get two command line tool that can be used to aid your development and testing.
This section describes the negotiate
CLI which provides a simple way for running negotiations,
plotting them, saving their statistics, etc.
You can find out about all the available options by running
$ negotiate --help