Navigating Beyond Lifelike Limits: The Emerging Era of Sophisticated Virtual Characters
Transforming Dreams into Reality: Navigating the Unsettling Terrain of 3D Avatars
The quest for lifelike digital companions in 3D media has long captivated creators. Such avatars, resembling real people, foster emotional bonds, making interactions smoother, more engaging, and downright enjoyable.
But this journey isn't a walk in the park. Technological hurdles lurk at every turn: How does one capture an individual's essence without resorting to expensive Hollywood-caliber 3D scanning equipment? And what about the psychological obstacles? The more realistic these digital doppelgangers, the tougher it is to make them feel pleasant.
In 1970, robotics professor Masahiro Mori coined the term uncanny valley: Artificially crafted humanoids that look like us but fall just short, causing fear, unease, and discomfort to a great number of observers. This chilling sensation intensifies with motion: A stationary doll doesn't seem as eerie as a lumbering cybernetic menace.
Decades on, the uncanny valley remains a go-to term among video game developers, used to craft hair-raising scenarios by deploying human-like NPCs with off-kilter appearances or movements. It's mentioned any time 3D artists attempt to create a hyperrealistic or intentionally cartoonish human figure. A quick search on X.com reveals plethora of examples, such as these, these, and these.
Insights from the Other Side of the Valley
Is the uncanny valley effect inborn or learnt? As a pioneer in the realm of realistic 3D models, we exhaustively dug into research on the topic, uncovering a few intriguing facts:
• Cartoonish avatars can similarly creep out viewers, as Zoll et al.'s study found. Anomalies in the level of detail in different parts of the model or between textures and meshes can create a sense of discomfort.
• Avatars that slightly resemble real individuals but aren't realistic don't necessarily trigger the uncanny valley, according to a case study by Henriette C. Van Vugt et al.
• Interestingly, the uncanny valley effect lessens when a human-like figure mirrors the subject's movements, as research by Elena Kokkinara and Rachel McDonnell demonstrates. This might be due to a sense of control over the entity.
• Highly realistic models evoke greater expectations regarding their movements. A study by Jeremy Bailenson et al. revealed that the incongruity between the realistic appearance of agents in VR lowers the sense of shared presence, the level of perception of an agent as a social entity. Human-like agents should behave realistically; toys, less so; and abstract shapes require unrealistic, random movements to create a strong sense of presence.
• The uncanny valley effect varies among age groups, cultures, and demographics. For instance, it's notably weaker for gamers, as shown by Katja Zibrek et al.'s findings. Regular exposure to digital humanoids may desensitize us to imperfections.
Stepping Beyond the Uncanny Valley
The technology for 3D avatar creation is becoming increasingly consumer-friendly. Our research, for instance, has discovered that a single selfie is sufficient to create a realistic 3D model of a person's face, one that real-life acquaintances can recognize.
However, animating the model using technologies like blend shapes reveals the illusion: The similarities disappear, and it becomes apparent this isn't the genuine article.
Several promising avenues may bypass this limitation. They leverage neural networks to analyze input video streams and generate realistic facial models, such as pixel codec avatars, as presented in research by Shugao Ma et al. These solutions address mesh and texture inconsistencies but face challenges integrating them into 3D scenes and rendering engines for realistic animation.
Other tactics focus on creating conventional 3D models ready for engines like Unity and Unreal Engine. For example, Meta is developing pixel codec avatars, while our company recently launched Avatar SDK Leap, which transforms iPhone face camera footage into 3D animations for import into Unity or Unreal Engine. By synthesizing mesh and textures on each frame, we aim for a level of realism never seen before. At this time, Leap is the sole commercially available neural-based animation solution, but more companies are jumping on the bandwagon.
I believe these developments will deliver convincing avatars beyond the uncanny valley, but there's still a long way until we achieve lifelike virtual humans for a VR world. We must find solutions for hair modeling, for example, and realistic humanoid movement remains a struggle even for AAA studios.
With rapid advancements in deep learning and a growing sea of data for training, it's possible we can conquer these challenges within a few years. If so, realistic 3D avatars will soon be in our grasp, leading to better AR/VR applications and more immersive remote collaboration experiences that transcend audio and video calls.
Although the technology isn't yet booming, with Apple allegedly shutting down its smart glasses project and Meta investing heavily in AI, the dream of an always-on, eye-mounted computer screen remains a tantalizing prospect.
We eagerly await news that will be our A New Hope.
Forbes Technology Council is an exclusive gathering of world-class CIOs, CTOs, and technology executives. Ready to join the club?
- Victor Erukhimov, a pioneer in the realm of realistic 3D models, aims to synthesize mesh and textures on each frame to create 3D animations for import into Unity or Unreal Engine, bypassing the limitations of traditional 3D avatar creation methods.
- Despite the advances in 3D avatar technology, generate a sense of unease or discomfort referred to as the uncanny valley is still prevalent, especially when the avatars exhibit anomalies in the level of detail in different parts of the model or between textures and meshes.
- In the race to create convincing avatars beyond the uncanny valley, neural networks are being leveraged to analyze input video streams and generate realistic facial models, such as pixel codec avatars, with the goal of integrating them into 3D scenes and rendering engines for realistic animation.
