Takuma Yoneda

PowWow: A Dataset and Study on Collaborative Communication in Pommerman

Takuma Yoneda🤖,  Matthew R. Walter🤖,  Jason Naradowsky🦄
🤖: Toyota Technological Institute at Chicago   🦄: Preferred Networks, Inc.
[PDF] [Dataset] [Data collection tools (Github)]

Abstract

In multi-agent learning, agents must coordinate with each other in order to succeed. For humans, this coordination is typically accomplished through the use of language. In this work we perform a controlled study of human language use in a competitive team-based game, and search for useful lessons for structuring communication protocol between autonomous agents.

We construct Pow-Wow, a new dataset for studying situated goal-directed human communication. Using the Pommerman game environment, we enlisted teams of humans to play against teams of AI agents, recording their observations, actions, and communications. We analyze the types of communications which result in effective game strategies, annotate them accordingly, and present corpus-level statistical analysis of how trends in communications affect game outcomes. Based on this analysis, we design a communication policy for learning agents, and show that agents which utilize communication achieve higher win-rates against baseline systems than those which do not.

Dataset

You can download the dataset from HERE.
The dataset includes game logs and annotations.

Brief Statistics

# Games # Messages # Participants
80 2,513 59

Game logs

Game logs are stored in valid_games_anonymized/ as json files. The directory structure is the following:

valid_games_anonymized/
├── pomlog_20191104-150039
│   ├── 000.json
│   ├── 001.json
│   ├── ...
│   ├── 078.json
│   ├── game_result.txt
│   └── envinfo.json
~
└── pomlog_20191203-214947
    ├── 000.json
    ├── 001.json
    ├── ...
    ├── 095.json
    ├── game_result.txt
    └── envinfo.json

A directory pomlog_20xxxxxx-xxxxxx contains the information in a game, and XXX.json in each directory contains recorded dialogues and game states up to the timestep XXX. The result of the game and meta-information of the game environment are stored in game_result.txt and envinfo.json.

Each json file has the following key & value pairs:

For some other details, it can be helpful to visit the official document of Pommerman

Annotations

Annotations are stored in annotations_anonymized.csv.