Yves Jacquier: Generative AI and the Future of Gaming

Additional Links

• • Learn more about Learned Motion Matching here.
  • Learn more about Ubisoft Teammates: An Experimental AI Gaming Project.

CHARLIE MELCHER:

Hi, I’m Charlie Melcher. Welcome to the FoST Podcast. A few weeks ago, I had the opportunity to see the future of storytelling up close in Montreal, Canada. I was there with the FoST Explorers Club, visiting several companies working at the forefront of technology and storytelling. One of them was Ubisoft. Ubisoft is the gaming company behind massively successful open world franchises like Assassin’s Creed and Far Cry. Assassin’s Creed alone has sold more than 230 million games worldwide, making it one of the best-selling video game series of all time. While I was there, I had the opportunity to meet with Yves Jacquier, Executive Director of Ubisoft’s La Forge. Yves founded La Forge in 2016 as the first lab in the gaming industry dedicated to applied academic research, and he has been leading it ever since. La Forge strategically brings together game creators with academic researchers to envision and prototype emerging technologies for more interactive, responsive, and transformative gaming experiences.

What makes Yves and La Forge so special is their willingness to experiment, collaborate, and imagine entirely new ways for games to be enjoyed. In this conversation, we have the pleasure of talking about the impact of generative AI on the player experience, the limitation of large language models, and the importance of human-centric design. Please enjoy my conversation with Yves Jacquier.

Yves, bienvenue to the Future of StoryTelling Podcast.

YVES JACQUIER:

Thank you, Charlie.

CHARLIE MELCHER:

It’s such a pleasure to have you with us today. So let me start by asking you a little bit of the experience that you’ve had with La Forge and how that came to be and why it’s important for Ubisoft.

YVES JACQUIER:

So La Forge has been created 10 years ago, and the idea is fairly simple. When you have game production teams who are focusing to deliver a game, trying to start from something from scratch in terms of disruptive technologies, for example, such as AI, it’s difficult to mix both in one production. So we decided to create La Forge to do two things. First, identify those disruptive topics. What are the elements that have the potential to totally change the way we create our video games? And the second is to make that practical. So working with universities, trying to find those concepts and very quickly focus on creating technical prototypes so that we can see the benefits, the potential, but also the limits of those technologies.

CHARLIE MELCHER:

I’ve heard La Forge described as a space to merge academic research with technological prototyping, a place where those two disciplines can come together. Why is the academic lens important for a gaming company?

YVES JACQUIER:

It’s a way for us to keep up with concepts that are still emerging. So for us, working with universities was a way to partner. They would bring their own savoir-faire on that, their know-how, their background, their network as well, because good researchers are connected with different labs and we would bring them kind of a sandbox to play with our technologies, our data, our engines, our expertise, our facilities as well, so that we would focus together on a prototype. We would have the potential to create new knowledge, which is interesting for the researchers, but also to help us create greater games and try to literally think outside of the box.

CHARLIE MELCHER:

So what is an example of an out of the box innovation that’s come out of La Forge?

YVES JACQUIER:

One example of that was, I think it was eight years ago already. It was about animations. Traditional animation in video games, you got animation tree. What does that mean? It means that you create many different small clips of animations and depending on what’s happening in a game, you play one or the other. That’s why it’s like a decision tree, but playing different animations. The problem with that kind of approach is that it does not scale well because when you say you have different situations, different weapons, different type of character, you need to create other trees as well and with all the complexity. So we came up with two approaches, the one called motion matching. So Motion Matching is instead of having to decide at each point what is the right clip to be played, you have the entire database to pick your animation from. That being said, it comes with issues in terms of performance, with issues in terms of memory on your computer, and also in terms of pure computing to find the right animation.

So we came up with a novel way that’s called Learned Motion Matching. Basically, we use this animation to train an AI that has a very smaller footprint in terms of memory and computing power. You have the best of both worlds.

CHARLIE MELCHER:

Let’s talk a little bit about another of the prototyping technologies that you’re working on, which is AI, generative AI. And I guess I would love to come at it from the perspective of the player. How are you thinking about generative AI in terms of really radically transforming or impacting the experience of players in video games and any examples or prototypes you’ve been working on?

YVES JACQUIER:

Basically in the video game industry in general, there are two ways to use generative AI. One which is to augment our game creators. That’s interesting, but there’s also another way to use generative AI, which is trying to not define in advance all the content of your game and let the technology generate whatever needs to be generated as you leave your experience as a player. For example, we did a couple of experiments lately. We have recently created Teammates, which was an experiment where as a player you would have your main character, it was a first person shooter, and you would have two teammates with different personalities and they would help you in your mission. So exchanging with them, you would talk to them instead of trying to find a couple of sentences that would have been pre-baked in the gameplay. And by doing that, you are able to ask them what they see.

You can give them orders, define a strategy with them so that you are able to foster the experience to the way you want to play really. However, it’s difficult to make that really meaningful, does not derail, for example, in terms of interaction. That’s something that’s extremely difficult to do, which is why we’re crafting those experiments and we gather feedback to understand what makes sense, what does not make sense, what’s relevant, what’s not, and et cetera.

CHARLIE MELCHER:

I mean, it’s always felt to me that this is the holy grail of gaming, which is to have it be so responsive that the player is truly co-creating the adventure. So I’m curious of any of the learnings. I mean, I imagine there was a lot of challenge to be able to create these teammates in such a way that they can both be responsive to the player, but also stay, as you were saying, in their own character and stay consistent in the game world.

YVES JACQUIER:

One of the big learning is that when you create such an approach or such a model, no matter what, players will try to break it.

CHARLIE MELCHER:

Oh, that’s just players.

YVES JACQUIER:

Exactly. And so that part is really important because what it means for us is that while we are developing such approaches and technologies, we also need to develop in parallel everything to keep the experience safe. With great technologies, you need to have great countermeasures as well. The other thing is that today that kind of approach is difficult to scale. You need also to make sure that you’re able to balance those costs with the ROI for the player to make sure that it’s absolutely meaningful. So there’s a technological challenge to make it scale beyond the experiment itself and also a design, a pure design challenge to make sure that we make things interesting in a way we will not be able to do that with the, I would say, traditional technologies and traditional design patterns.

CHARLIE MELCHER:

So just so I understand, because I’m so personally fascinated with how we create responsive, personalized, agentic storytelling at scale. And I’ve always thought that AI, generative AI, was going to be our breakthrough, but you’re saying at least at this point it’s too expensive. Is that’s the limiting factor? Is it the cost, the compute power to be able to have it be that responsive in real time for extended periods?

YVES JACQUIER:

That’s one part of the element because if you want to have something that’s very meaningful, you need to understand what the player is saying. You might need to understand the intention of the player. For the example, if the player goes right, do you want to follow him or her or not? You need to have voice synthesis, animation synthesis, if you really want to make sure that the characters are fully believable in the way they interact with you. That can become extremely expensive. So one way to do that is to try to use the player’s computer to run that kind of technology. By doing that, it’s less expensive. It’s more responsive for the player because it runs locally instead of having to hit some servers elsewhere. And also it’s way more ecological. But still that’s difficult because at the moment the player PCs or consoles might be a bit underpowered, I would say, when you want to have extensive experiments.

So that’s why there’s still a lot of work to do, both in terms of design, in terms of technology, and then trying to make that fit as local as possible.

CHARLIE MELCHER:

That’s fascinating, first of all, and makes so much sense as a path forward, just the latency of having to process that much in the cloud and then bring it back in real-time action. You can’t wait five seconds to decide to fire or not. I think when we spoke recently, you said to me that LLMs are dead. I think you’re explaining to me now why. They’re just too unwieldy for the kind of responsiveness and customization needed for this type of storytelling or gameplay.

YVES JACQUIER:

And there are ways to make great use of large language models, but we need to take them for what they’re good at. Large language model is something that’s extremely useful when you need to predict the next word, the next frame, the next image, but it does not understand the concepts of what you’re trying to do. So for example, if we are talking in terms of storytelling, those language models, whether large or small, they don’t understand the context of the discussion. They don’t understand the context of the story. Now there’s a new concept that’s been emerging for, I would say one or two years now. It’s still very academic. It’s called world models. So for example, if you’re using a large language model approach for video, if I’m having whatever object and I’m letting the object fall, the LLM will predict the next frame and the next frame and the next frame so that you will see in the video the object falling on the floor, but the LLM does not understand what gravity is.

If you put the same situation, but let’s say on the moon, the LLM will do the exact same thing because it’s seen so many example of something falling on earth, so that does not understand that gravity is a concept that is different on the moon and on earth. So that’s where enters world models. You train them so that they understand the concept and then you are able to play with those concepts. So in terms of storytelling, it could go way further because you can imagine that you have a world model that could be your environment so that the player would be totally immersed in a world with its own logic and could play with its own logic.

CHARLIE MELCHER:

All good world builders have always talked in that way, that to really build a world, you need to understand its currency and its agriculture and its civilization’s evolution, and you need to understand a lot of the details of its past. But here it sounds like you could train an LLM to understand those bigger concepts that define that world, and it sounds like it could create much more elaborately and even more creatively once the rules are set by the game designers.

YVES JACQUIER:

Exactly. It changes everything in the way we design systems in this kind of situation because then it’s really the holy grail of emerging gameplays. We have been working on different open world games, for example, in Far Cry, where we set up different gameplays and the player can combine them to create things that were not planned in the world and that are surprising for the player so that they can design their own experience. This approach is called the anecdote factory. The idea is that if we play together the same game, so we have a common ground, the setup, the characters, the overarching story, for example, but within the game, I might come up the day after and telling you of an anecdote of something that I did very fun in the game, and I can share that with my friends. So that’s why we are calling that the anecdote factory, giving the player the ability to craft some elements of the experience with combining different type of gameplays.

CHARLIE MELCHER:

Two things going through my mind. One is how radically different it would be to design for this than trees of options and paths of navigation and all of that. And the second is also how completely different for the player. So how do you think this is going to change the nature of what it means to play these kinds of games? How will the games change the player?

YVES JACQUIER:

It’s a way to not only asking the player to master the game or master the gameplay, but it’s also a way to include in the experience all the creativity of the player. And maybe your creativity, your way of solving problems is different from mine. And more importantly, while maybe learning something, it’s really something that can change drastically the way games and players interact.

CHARLIE MELCHER:

Yeah. I am a big believer that these kinds of responsive experiences or stories do foster your creativity. They demand your response and they reward it. And in doing that, they will train new muscles. I think that so much of our media for so long has trained us to be passive consumers as opposed to co-creators. And I’m excited for the opportunity to encourage people’s creativity. So now let’s also talk about the role of the changing nature of the game designer or the teams that make this. I mean, I was reading on your site a story from somebody who I think is a narrative designer, and she was saying she’s used to building a character’s backstory, their hopes and dreams, the experiences that shape their personality, and that she’s done that for so long. And now when she’s doing this using generative AI, she’s actually getting to have a conversation with a character that she’s started to build and that conversation leads in a way to its continued evolution, its development.

YVES JACQUIER:

Absolutely. And what’s very interesting is that you would think that some writers would take that as a threat to their skills. Actually, it’s the opposite because you need people skilled to create characters that are really interesting with a real personality. The analogy that I have is I’ve been an actor for a couple of years in theater. The way we interacted with the director of the play, to me, it’s like the way we are crafting the characters today in our experiments because it’s a discussion with the actor. It’s a way to have interesting things that emerge from the actor. There’s also the director’s vision, and you need to have this conversation with the actor as long as it fits in the director vision. So how can a writer direct all the narrative, having a strong vision and use this generative AI material like you would do somehow with an actor?

And that’s really, to me, what’s changing in the way we approach that. The creator has many different tools amongst them, AI. If you see the trick or if you see the AI, you fail, right? It has to be invisible.

CHARLIE MELCHER:

I like the analogy. I hadn’t thought of that before of a director to an actor, like a prompt writer to an AI, sort of asking the interesting question to try to elicit a different response or the right kind of performance. Do you ever think about at Ubisoft merging this kind of interactivity and the elements of play that come in gaming to wearables into the real world? AR, for example, glasses?

YVES JACQUIER:

I don’t know if we want wearables. It’s so intrusive. I’m predicting that we will need to get our lives back, which means going more into real in-person connections and having a clear distinction probably between what’s happening in our, I would say, digital life that includes entertainment and the real connections we’re making with people. And I think the wearables might be something that is more of an obstacle that keep us away from other people.

CHARLIE MELCHER:

I have a slightly different take on that. First of all, you and I are speaking to each other, both wearing glasses, and I don’t think that’s hurting our conversation. I think it’s helping it. I see you more clearly. I think there’s a huge opportunity for personalized responsive social gameplay that we haven’t really begun to get into with our wearables. But any case, I don’t think we necessarily want that all the time either. I think there’ll be times when we’ll be happy to turn it off, and I think we’re still years and years away from all the breakthroughs necessary to make that a value and encouraging of a human experience as opposed to a clunky thing that hurts our neck or doesn’t really do anything better than our phone.

YVES JACQUIER:

I agree. I totally agree with that. I think that the main thing that we need to resolve is we have glasses, but today with the technology of our glasses, you know that I’m paying attention to you because I don’t have any AR glasses. I don’t have a parallel experience or an experience that is parallel to our discussion. The limit of the wearables, I think it’s not ourself, it’s more the other people. How do we respect our relationship with people and our connection with people while using this type of technology? So to me, it’s more a question of design way more than a question of technology because I agree, I mean the cumbersome aspect or everything that will be solved, but how do we respect the connection of people when we bring those wearable in the physical world, basically?

CHARLIE MELCHER:

Ultimately, what I hear us both saying is that we just need to make sure that these tools evolve in a way that’s very human-centric, that is really in service of what it means to be human.

YVES JACQUIER:

Yeah, absolutely. Absolutely.

CHARLIE MELCHER:

So what else do you see as this next wave of generative AI and gaming? What are some of the things you could see happening?

YVES JACQUIER:

I’m observing what’s happening inside of Ubisoft, inside of the industry, but also outside of industry and everything. So what I’m seeing is really the ability to provide a unique experience to each players with the challenge to keep something that brings us together, something that we can share meaningfully. I’m seeing a future where we will have a lot of that, where we will be able to have kind of a sandbox and we share the same logic you and I in this world, but we will approach the problems, the situations, the characters very differently, and still we’ll be able to share on that. And maybe I will be able to learn from your experience. I will be able to learn from you how you approach that as well.

CHARLIE MELCHER:

When you say sharing, do you mean in real time or do you mean after the fact? Let me tell you the story of what I did.

YVES JACQUIER:

My hope is that we will have those discussions in person. Like today we might exchange on the different shows that we’re seeing on TV, and you might give me your appreciation of this episode or else, and we might have interesting discussions on that. Why not having the same discussions on something that is happening within a game, some things that have happened in the game and the way you approached that and the difficulty you had and the things you learned, and maybe I can find very interesting and maybe I can use your experience and tell you, okay, I will try that the next time in the game as well. Today, it already exists within the gamer’s communities, but we can take that way further with those newer technologies. And there’s been some experiment lately where someone tried with artificial intelligence to recreate a level of a game called Call of Duty. The person asks the AI to understand from all the videos that you can find on the internet what Call of Duty is and to recreate the gameplay and the visuals and everything with one prompt.

Is it the best game I ever saw? The answer is no, absolutely not. But it’s a great hint on where the industry could go because if as a gamer I’m able to create from scratch my own experience of these settings, build my own experience, and why not share my own experience, I think that is really something where the industry should be heading.

CHARLIE MELCHER:

Yeah, I love that. And certainly we’ve seen with success of other things from Minecraft to Roblox that everybody loves the idea of creating their own games and now to be able to do it at a level of sophistication. And I also love that that would allow for people to be able to pull from characters and plot lines and worlds, not just from within gaming, but from many different genres of storytelling and media to form new genres or new media.

YVES JACQUIER:

I’m not sure if it’s going to be a convergence, but definitely I’m expecting new type of media emerging from that, new type of us interacting with narrative, not only being passive, but I can imagine, for example, that if you are teaching history, you might ask your students to create a one hour experience of that particular period of time. And my bet is that the students would learn way more than just by reading books or seeing movies or videos, which is perfectly fine. And that’s the best way for us today to learn all of that. But doing that interactively, it really is the next level.

CHARLIE MELCHER:

Well, Yves, I’m so excited that you are helping to bring such thoughtful science and really human direction to the future of Ubisoft and the future of gaming. I thank you for spending the time. I appreciate the important work that you do merging those two worlds of serious scientific research and the academic breakthroughs, but finding a way for those to make it into the real world and enhancing to players and to people who like stories. So thank you for all of that.

YVES JACQUIER:

Thank you very much, Charlie. It was a very interesting discussion and I really liked our agreements and our disagreements. I feel enriched.

CHARLIE MELCHER:

Once again, I’m Charlie Melcher and this has been the FoST Podcast. My thanks to Yves Jacquier for our conversation and for hosting all the FoST Explorers Club members at the Ubisoft Studios. For more opportunities to learn directly from creative studio leaders who are shaping the future of storytelling, check out the FoST Explorers Club. We have an upcoming trip from September 25th through 27th in London, where we’ll be offering a curated itinerary with special access to the creators behind innovative studios and cutting edge immersive works. Check out the show notes for a link to register or visit our website at fost.org. The FoST Podcast is produced by Melcher Media in collaboration with our talented production partners, Charts & Leisure. I hope to see you again soon for another deep dive into the world of storytelling. Until then, please be safe, stay strong and story on.