✉ Contact address: genea-leaderboard@googlegroups.com ✉

THIS WEBSITE IS UNDER CONSTRUCTION!

Summary: To improve benchmarking of speech-driven gesture generation, we are developing the online GENEA Leaderboard based on large-scale human evaluation. This is a cross between the previous GENEA Challenges in gesture generation and recent leaderboards in NLP and computer vision, such as Chatbot Arena and HEIM.

Our goals

Establish a continuously updated **definitive ranking** of state-of-the-art models **on the most common** speech-gesture **motion capture datasets**, based on human evaluation.
Raise the standards for **reproducibility** by releasing all collected model outputs, human ratings, and scripts for motion visualisation, conducting user-studies, and more.
Use the collected human ratings to develop **better objective metrics** that are aligned with human perception of motion;
**Unify gesture-generation researchers** from computer vision, computer graphics, machine learning, NLP, HCI, and robotics.
**Evolve with the community** as new datasets, evaluations, and methodologies become available.

Outcomes

Once the Leaderboard is operational, you will be able to:

**Submit** your gesture-generation model's outputs, and **receive human evaluation results in 2-4 weeks for free**, managed by our expert team;

**Compare** to any state-of-the-art method on the Leaderboard using our comprehensive collection of rendered video outputs, **without having to reproduce baselines**;

**Visualise** your generated motion and **conduct our user studies** on your own **using our easy-to-use open-source tools**;

...and much more!

Leaderboard setup and timeline

(For comprehensive details regarding our current evaluation, please visit our Details page.)

As a living benchmark, the GENEA Leaderboard will be developed in an iterative manner over time.

Our current plans for the very first public release are the following:

The Leaderboard will be based on the body movements of the BEAT-2 dataset in the SMPL-X format. Facial expressions will not be considered for now.
We will accept model submissions, in the form of synthetic SMPL-X motion data generated from the test inputs of the above dataset, until the end of 2025 January.
The human evaluation methodology will initially only measure motion quality and motion-speech appropriateness, as defined in the GENEA Challenge 2023.
The first set of evaluation results will be released in 2025 March.

After the first results are published, we will keep the Leaderboard open for submissions throughout the year, and we will continuously update the ranking and publish new resources (e.g., votes and rendered videos).

Who are we?

Our international team includes leading experts on the evaluation of gesture-generation models, as well as experienced model developers and research engineers with experience in deploying gesture-generation models. We are building this leaderboard to advance the scientific field of gesture generation, equally serving the perspectives of both academic and industrial researchers, as well as senior scientists and upcoming PhD students.

To learn more about the leaderboard team, please visit this page.

✉ Contact address: genea-leaderboard@googlegroups.com ✉

✉ Contact address: genea-leaderboard@googlegroups.com ✉

Our goals

Outcomes

Once the Leaderboard is operational, you will be able to:

Leaderboard setup and timeline

Who are we?

Sponsors

✉ Contact address: genea-leaderboard@googlegroups.com ✉