Researchers at the Max Planck Institute for Intelligent Systems and Carnegie Mellon University have unveiled a breakthrough technology that allows for highly accurate 3D motion capture using only standard video cameras, eliminating the need for expensive, specialized hardware and the iconic skin-tight motion capture suits covered in reflective markers. They named their system MAMMA – Markerless Accurate Multi-person Motion Acquisition. This system is capable of accurately capturing human motion without the need for markers, making it easy and cheap for everyone to create high-end animations and virtual avatars.
Markerless Accurate Multi-person Motion Acquisition, or Mamma, takes multi-camera video, estimates virtual markers in all camera views, and then fits a human body model (SMPL-X) to these markers. Image credit: MPI-IS
For decades, the gold standard for capturing human movement and transferring this data into the digital world has required actors to wear suits with physical markers that cameras track in a specialized studio. While accurate, this capturing process is time-consuming and expensive. Markers need to be placed manually, they can fall off, and if two people get too close – such as during a hug or a dance – the system fails to track all markers as one person blocks the other from the camera’s view and accuracy drops significantly. Extensive post-processing is needed to recover the motion in such challenging sequences. The creators of the Hollywood blockbusters “Avatar” or “The Lord of the Rings” will know what we mean.
Commercial marker-less systems do exist however they remain expensive and are generally considered to be less accurate than marker-based systems. This problem triggered researchers at the Max Planck Institute for Intelligent Systems (MPI-IS) and at Carnegie Mellon University to develop a computer vision technology that achieves an accuracy comparable to commercial marker-based systems without the tedious manual clean up, and – going one step further – solving the challenging problem of close interaction between people.
An actress and an actor are wearing suits equipped with physical markers that track their movements in a special room at the Max Planck Institute for Intelligent Systems—a model that is being phased out. Image credit: MPI-IS
Markerless Accurate Multi-person Motion Acquisition or MAMMA, takes multi-camera video, estimates virtual markers in all camera views, and then fits a human body model (SMPL-X) to these markers. In this way, the network can “see” 512 virtual tracking points directly on people’s skin from regular video frames. Additionally, it can predict poses it has never seen during training. So unlike previous attempts at marker-less capture, which often fail to accurately recover all body shapes and are not publicly available, MAMMA is specifically designed to work perfectly using standard smartphone cameras, and handle complex social interactions and physical contact. Martial art avatars fighting or two digital people dancing Salsa are just as realistic as the alive-and-kicking sportsmen or dancers.
“The difference in pose accuracy between our technology and suit-and-marker based systems was less than one millimeter, yet MAMMA can process the data nearly three times faster than the manual workflow required for traditional systems. It can even work with consumer-grade equipment, like a set of iPhones,” says Hanz Cuevas Velasquez. He is a research scientist in the Perceiving Systems Department at MPI-IS and first author of the MAMMA paper.
Transforming Industries
The researchers anticipate that this technology will have a profound impact across industries such as medicine and biomechanics, virtual reality and the metaverse, sport science, the film or gaming industry, to name a few.
Anastasios Yiannakidis is a research engineer in the same department, and continues: “The system can be used to track patient movement for physical therapy or biomechanical research. Because it requires no markers, it is much less intrusive for patients and can be used in more natural environments rather than a clinical lab.”
By accurately reconstructing both the shape and pose of a person’s body and hands, MAMMA can create lifelike digital avatars for social VR settings, making virtual interactions feel more human and expressive.
Sports coaches could use this technology to analyze the form and technique of multiple athletes simultaneously during practice sessions without disrupting the athletes’ movements with bulky gear.
“MAMMA is a game changer in many ways. It allows game developers to capture realistic, contact-rich animations like wrestling or dancing without the overhead of a massive motion capture studio. Our technology will allow smaller, independent studios and research teams to create character movements that look as polished as those in big-budget blockbuster movies,” says Data Team Lead Tsvetelina Alexiadis, another one of the paper authors.
By releasing their data and tools to the academic community, the team of scientists hope to spark a new wave of innovation in how the motion of several people is recorded, to better replicate and hence understand human movement.
Source: MPG
