Enhancing visual on-screen accuracy with computer vision techniques
In the ever-evolving landscape of the creative industries, a new tool has emerged to tackle the long-standing issue of diversity: computer vision. This innovative approach, spearheaded by research from Nesta, aims to provide a more objective and scalable means of assessing on-screen representation and character portrayal.
The research, titled "She said more", was presented by Raphael Leung at the IJCAI 2021 AI for Social Good workshop and builds upon a presentation given by Leung and Bartolomeo Meletti at the International Federation of Television Archives (FIAT/IFTA) Conference 2020. The project, commissioned by the Creative Industries Council, was published by Creative PEC, an organisation that offers independent research and policy recommendations for the UK's creative industries.
Computer vision offers a unique solution to the diversity problem in the UK screen industry, which has been notoriously difficult to solve due to persistent structural inequalities. By automating the detection, classification, and quantification of demographic representation in visual media, this technology allows for empirical measurement of how diverse and representative on-screen character portrayals are.
The technique identifies characters visually across frames using facial recognition and tracking algorithms, mapping their appearance and prominence in scenes. It classifies demographic attributes such as ethnicity, gender, and age group through trained models, facilitating automated demographic breakdowns. Furthermore, it quantifies screen time and interactions by detecting characters’ presence and participation in dialogue or actions, helping to evaluate the extent and nature of representation.
Moreover, the exploration goes beyond on-screen presence, considering prominence and portrayal as well. An illustrative example of measuring character prominence is provided using a short video clip. The report, which details these findings, was published as a report and forms part of a series of blogs that explore whether computer vision might provide a new method for measuring on-screen representation.
The use of computer vision in this context aligns with broader artificial intelligence ethics and fairness discussions, as it helps surface implicit bias or underrepresentation in media content, which historically has been challenging to quantify objectively.
However, the application of computer vision is not without its challenges. The ethics of applying this technology to study on-screen characters are discussed via a conceptual framework. The project team, consisting of Raphael Leung, Bartolomeo Meletti, Dr Cath Sleeman, Gabriel A. Hernández, and Gil Toffell, is actively engaged in addressing these concerns.
The exploration is an output of a pilot project conducted by Nesta in partnership with Learning on Screen. The UK's departure from the EU has altered the way British firms trade and collaborate with European counterparts in the Creative Industries, but the potential benefits of this innovative approach could reach far beyond national borders.
As the creative industries continue to evolve, the role of computer vision in promoting diversity and inclusivity is set to become increasingly significant. This research series emphasises how AI, including computer vision, can be harnessed to analyse creative outputs like films and shows in a scalable way, thereby supporting goals like greater inclusivity and balanced representation in media.
- The research titled "She said more" presented by Raphael Leung at IJCAI 2021 AI for Social Good workshop utilizes computer vision, a novel approach for tackling diversity issues in the creative industries.
- Commissioned by the Creative Industries Council and published by Creative PEC, the project employs artificial intelligence to detect, classify, and quantify demographic representation in visual media.
- Computer vision allows for more objective and scalable measurement of on-screen character portrayals, identifying characters visually using facial recognition and tracking algorithms.
- The technology classifies demographic attributes such as ethnicity, gender, and age group, and quantifies screen time and interactions to evaluate character representation.
- Beyond on-screen presence, the exploration examines character prominence and portrayal, with the goal of providing a new method for measuring on-screen representation.
- This application of computer vision aligns with broader artificial intelligence ethics and fairness discussions, helping to surface and quantify implicit bias or underrepresentation in media content.
- This research series highlights the potential for AI, including computer vision, to support goals like greater inclusivity and balanced representation in media across various industries and global markets.
