Luke Grimes Says Not Having Kevin Costner on 'Yellowstone' Set 'Felt Different' (Exclusive)

For Luke Grimes, it's the gravity of Kevin Costner's departure from his starring role in Yellowstone was not lost when the final episodes of the series were filmed without him.

The 40-year-old actor and musician tells PEOPLE that Costner's decision not to return after nearly eight years together gave him “a finality.”

“When he didn't come back, he felt like, 'Okay, we really need to land this plane now,'” Grimes shares. “I think the departure of the patriarch would always be a part of the story. That was always the question: What are these children doing? What does this family do when their stone is gone?”

Although he acknowledges the “drama” that came before it Yellowstone's return to filming – including scheduling conflicts between Costner Horizon: An American Saga And Yellowstone That led to the Oscar winner stepping away from his role in the series in June – Grimes adds that he felt “it was always going to be this way.”

“Whatever the circumstances, it definitely felt like, 'Okay, we're about to land,'” he says of Costner's absence. “It definitely felt different.”

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

“He’s got a big presence, man, the guy is an icon,” the actor added. “So it felt a little different not having him on set. But I mean, as always, we're one big family now. It was just as special and hopefully the fans can relate to the story just as much.” .. I think they can do it.”

On the subject of family, Grimes points out that “while Kayce as a character feels a bit like an outsider in his own family… I think the thing that got him between a rock and a hard place was so deep.” He felt that Loyalty crushed him even though he knew it might not be what was best for him.

He notes that over the years, “the dynamics have been kind of fun to play with.”

For more on Yellowstone, pick up the latest issue of PEOPLE.

With Costner's departure, fans are wondering what will happen next for America's favorite Montana rancher. Grimes says, “There are always rumblings.”

“I think when something is so successful, there's pressure to keep it going because they know they have an audience and that's what they're looking for,” he says. “But it all depends on the vision of Taylor (Sheridan). He has to see them to write them, and I don't think he'll write anything he doesn't want to. “Basically, we’re just doing it to see if Taylor has an idea that excites him, and that’s going to make all the difference here.”

Yellowstone returns Sunday, November 10th at 8pm ET/PT on Paramount Network. All five seasons are streaming on Peacock and Paramount+.