David Rose | The emerging AI shift in sports

David Rose, futurist at global innovation design firm EPAM Continuum, and a prominent lecturer, author, and expert on digital product innovation, explores the important role that artificial intelligence will play in the future of coaching in sports.

What does Lululemon see when it looks into (the) Mirror? A future where artificial intelligence coaching changes the game. Lululemon recently paid $500m (€423m) to acquire Mirror, the maker of a $1,500 vertical home mirror with an embedded training service to guide you through your workouts. The incredible valuation of this startup fires a starting gun for the race to create personal remote coaches, powered by computer vision, for everyone who needs a little motivation and guidance at home. And who doesn’t? 

“Home fitness products like Peloton and streaming classes have boomed during the [Covid-19] pandemic,” writes the New York Times about Lululemon’s acquisition of Mirror. “With gyms and fitness studios are largely closed, and many customers are nervous to return to those that have reopened. Lululemon said that while the pandemic was not the ‘trigger’ for the purchase, it bolstered the case.”

It was a project with a professional sports team that first got our team at EPAM Continuum excited about using computer vision and AI for coaching. We processed soccer game videos to track players and label specific moves such as a shot on goal, sliding tackle, and corner kick to compile a personal post-game coaching video for each player. 

As we interviewed players and coaches, watched a lot of sports, and made the algorithms watch a lot more sports, we discovered a key insight: showing side-by-side comparisons between players was the most efficient way for them to learn and level-up their skills. 

We used this approach when we created our own AI coaching app for yoga and more. There are a million yoga videos online. But none of them offer feedback because they don’t see you. 

For our first experiment in coaching guidance in the home, we showed a yoga expert moving through a workout, and next to the instructor you see yourself as you would in a mirror. A computer vision pose classifier measures key body angles for each yoga pose, to focus your attention on the differences between your technique and theirs. 

Our next experiment explored the limits of AI pose-classification technology. What sports would it not be useful for, either because people wear bulky clothing (skiing and snowboarding), the action is just too fast (fencing, ping-pong, and badminton), the body spins too much (ice-skating and diving), or the sport doesn’t require feedback on technique (weightlifting and rowing)? 

Here’s what we learned: 

  1. Pose classifiers are pretty robust even with bulky clothing
  2. When the action is very fast, we can add another neural network to our video-processing pipeline to insert frames and effectively generate super-slow-mo 240 fps action
  3. For spinning sports, a 3D pose classifier is most effective. It’s computationally expensive, so it’s harder to provide real-time feedback
  4. Every sport needs a coach. Weightlifters and rowers rail at the idea that their sports don’t require serious amounts of technique. I interviewed Dan Boyne, Harvard’s crew coach and author of Essential Sculling, who was adamant that “winning a crew race is equal parts technique and fitness”.

At least four companies now offer a home-gym “Zoom experience” embedded in a large home mirror with special 3D-imaging cameras to analyze your form. Personal trainers guide you through high-intensity interval training workouts, yoga, pilates, boxing, and weightlifting. Prices range from $1,000 to $4,000 for the hardware, with monthly subscription packages for sessions. AI provides counting and feedback and some correct your posture so you don’t injure your knees, shoulder, or back.

But how could we get the benefits of a personal trainer, someone to scrutinize our specific form and offer guidance for less than the cost of a $3,000 mirror? 

Our first prototype offers form feedback one move at a time. Upload a short video of your downward dog, tennis serve, golf swing, or tai chi move, to see how your technique compares to the masters’. 

To do this, we created a cloud-based computer vision pipeline that ingests pairs of instructor and novice videos, crops to the action, slows down very fast moves, infers body pose in 2D and 3D for each frame, and compares your form to that of an instructor (or a superstar athlete) doing the same move. 

To align parallel action, we added an algorithm called dynamic time warping (DTW) to the pipeline. This makes side-by-side comparisons easier because the player and model are both in the same phase of the stroke or action at the same time, even if one player moves faster than the other.

To better understand complex motions like a baseball pitch or golf swing we added stroboscopic effects, like skater and snowboarding magazines, to freeze multiple poses in action. 

Last, coaches often mark up videos – this is called ‘telestration’ in professional sports – to make sense of actions between players. We automated these augmented marks over the video with a technique we call “ribbons.” Inspired by those Olympic gymnasts with streamers, and stunt planes that emit colorful smoke, we offer the option to leave a persistent visual trail from any joint or part of the body. 

By talking with coaches and athletes, studying other offerings, and building our own app, we’ve developed a point of view about the huge opportunity in AI coaching:

  1. Feedback should come fast and slow. There should be a mechanism for lightweight, non-overwhelming, real-time feedback (audio often works best for this), as well as a longer feedback-loop experience that’s more considered and requires more attention, time, and study
  2. Use adjacent social comparisons to motivate and learn. Perhaps the most persuasive form of gamification is the social comparison. We are all interested in seeing how those just above our skill level perform to motivate our next steps. An app should use this “relative to” insight with a leaderboard’s “just catch the next guy” feature, and form comparisons not only with superstars, but also with athletes and friends just above our level
  3. Coaches should have a strong presence, personality, and style. Instead of generic homogeneous coaches, an app should offer a stable of strongly differentiated coaches in demographics, ability, and style. People will gravitate to those they prefer
  4. Celebrate and share summit moments. Make it easy for people to share achievements and performances. These personal records and proud moments are part of the culture of sports. A digital coach should help people memorialize and share them
  5. Take an ecosystem approach to exercise data. People want to share their wins with their network on Strava, save data in Apple Health, get reminders from Alexa, Siri, and Cortana, and eventually share data with their doctor, physical therapist, and carbon-based coaches. We need to make this data easily understandable and shareable with a wide audience of stakeholders.

Where and when exactly should you work out remotely with your AI coach? Let’s integrate exercise moments into the remote-work rhythms of our day: a micro-workout just before or just after a conference call. Or midway through like a seventh inning Zoom stretch. 

Even short, three-minute workouts help maintain cardiovascular health and benefit psychological well-being. And when exercise is done with others, like a short stretching activity, it increases rapport and can bolster our weakened remote relationships.

Ultimately this is a great opportunity for sports brands like Adidas, Nike, or Under Armor to offer this Coach.AI service. These aspirational brands and their athletes exemplify personal fitness, innovation, and great design. And since you have a video conferencing set-up at home and are probably wearing sweatpants and sneakers anyway, we stand ready. Just as stand-up desks have become mainstream, remote workouts and the AI coaches that guide them will become the next normal.