Many labs around the world conduct research that relies heavily on perception experiments with video. Standards for operating procedures, such as the ITU recommendations, exist for these kinds of experiments. However, the current tools for data collection for subjective testing are outdated or insufficient, and efforts to improve collection methods are too sparse and fractured to be widely effective.
To address this need, we have developed Tally, a web- based data collection tool for subjective video experiments. Our tool has numerous advantages, most notably the decoupled voting and viewing interface, flexibility and robustness, and the ability to share and collaborate on projects. We believe our web-based design is not only superior to previous attempts at such a tool, but is also the proper way of addressing the data collection problem.
Releasing the software as open source encourages growth and a well-supported user community. It is our hope that Tally will be widely adopted. The more people who use it, the better and more effective it will become. We designed Tally on the principles of openness, transparency, and collaboration, and we hope that our tool promotes these values among the research community through its use.
We designed Tally as a web-based system. Since subject scoring is done over a network, the voting control can be decoupled from the media player. The media is displayed to a TV or monitor while voting is done through an web-enabled device such as a smart-phone, tablet, laptop, or desktop.
The web-based nature of our tool offers other new benefits as well. With this design collected data is available from anywhere, and sharing data is simple. Since each person has their own personal account on the website, many people can use the same system with their individual history and data saved.
For any inquiries email firstname.lastname@example.org and we’ll get back to you as soon as we can.
This content is released under the MIT License. For any documented use of our software in reports, publications, etc., we ask that you cite our paper . If you want to cite the Tally website, please use .
 A.K. Jain, C. Bal, and T.Q. Nguyen, "Tally: A Web-Based Subjective Testing Tool," Fifth Int'l Workshop on Quality of Multimedia Experience (QoMEX) 2013, Jul. 2013. [ Link ]