The last episodes of Yellowstone were easier to film after Kevin Costner’s exit, according to one cast member.

Luke Grimes, who plays son Kayce Dutton to Costner’s John Dutton III in the Paramount drama series, recently spoke with him esquire about the temperature on set after Costner’s character was killed off after five seasons of the hit series.

“Hopefully everyone can see that it was time,” Grimes, 40, told the outlet in an interview published on Tuesday, December 10.

“To be honest, there was some of Kevin leaving which meant some of the conflict was gone. Obviously it wasn’t much fun to be here. I don’t want to point fingers, but it was actually the easiest season we’ve ever filmed,” he said of the second half of the show’s fifth season. Costner left the show after the first half of season five.

Much of the conflict Grimes was referring to had to do with Costner, 69, and series creator Taylor Sheridan not seeing eye to eye while the actor was working on his own Western epic. Horizon: An American Sagawhere there were overlaps in filming time.

“I don’t want to end up in the gutter with this Yellowstone thing, but what I’m telling you is clear. I’ve been beaten by these damn guys and I often know where that comes from. I just decided not to get involved in it,” Costner previously told Deadline.

“I made Yellowstone “To assume anything else would be wrong,” he said in the May interview.

Despite the drama, Grimes added that Yellowstone had of course run its course.

“It feels like good timing,” he said of the series ending. “Seven years is a long time to do something, and I had never played a character for more than a few months.”

The 50 shades of gray Alum added that there were still good times: “I loved everything.”

“But there is a part of it that – if you read the final episodes – realizes that there is a finality. There are always conversations about spinoffs,” Grimes said, noting he would be open to it.

“I would do it (but) I just don’t understand how it would work when the story is over. Kayce wants to be a cowboy and be happy with his family. He doesn’t want to kill people anymore. He doesn’t want the weight of a huge mega ranch, which isn’t sustainable in this day and age. He wants his little piece of heaven. It’s that simple,” he said of his character’s fate.

Grimes’ comments too esquire about Costner’s exit echoed statements he previously told PEOPLE.

“The fact that he didn’t come back felt like, ‘Okay, we really need to land this plane now,'” Grimes tells PEOPLE of the plot.

Never miss a story again — sign up for PEOPLE’s free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from celebrity news to compelling human interest stories.

“I think the departure of the patriarch would always be a part of the story. That was always the question: What are these children doing? What does this family do when their stone is gone?”

He also admitted that it “felt a little different” not having Costner there, calling the cast “one big family now.”

All five seasons of Yellowstone are streaming on Peacock and Paramount+. The series finale will air on Saturday, December 15th.