Rufus Sewell says he ‘struggled’ to be cast as the villain, but jokes it ‘suited the shape of my face’ (Exclusive)

The ‘Diplomat’ star tells PEOPLE that for “a long period of time” in his career, I would get “many, if not evil, stoic and silent characters”

• The diplomat star Rufus Sewell has played the antagonist in films like The legend of Zorro And Helen of Troyplayed the role of prince andrew in netflix’s scoop and portrayed a no-strings-attached love interest in The holiday

• Sewell tells PEOPLE that he has spent his entire career trying to “make the best of the same category of parts.”

• On the way to The diplomatIn the second season of his career, Sewell says he feels “safe” for the “only time” in his career

Rufus Sewel doesn’t think he has the swag of many of the villains he’s played.

“I’m not naturally sure about it,” Sewell, 57, tells PEOPLE. “I actually have a frenetic, nervous, people-friendly character. I’m not a cool guy. I’m feeling quite warm.”

Still, the British actor found himself playing a lot of not-so-angelic characters, including his latest roles as Prince Andreas enters Spoon and Hal Wyler enterThe diplomat.

“When I first started working, the roles I usually played were very, very, very far away from me,” says Sewell. ‘When I left drama school, I always played old men and a hunchback. That’s how it left my mark. There was a long period where I was sent many, if not bad, stoic and silent characters.

Related: Keri Russell Jokes She’s Been With Matthew Rhys So Long She ‘Don’t Remember Other Relationships’ (Exclusive)

However, Sewell did not think these roles played to his strengths.

“That category of parts didn’t really fit with what I was best at,” says Sewell. “But it fit the shape of my face, so I kept my mouth shut. It’s just annoying when people who should know better in the industry send me scripts based on something I really struggled with.”

The Emmy nominee feels like he started to see a change after starring in the lead role in 1994 Middle march“where I played someone my age who wasn’t hunchbacked, who wasn’t like a pervert or a lunatic or a psychopath,” says Sewell, who has also been the antagonist in Bless the Child, Helen of Troy, The Illusionist, A knight’s story And The legend of Zorro. “Then people started seeing me a certain way, and I went the other way for a long time.”

Sewell says that when he took on villain roles, he felt like he could “do something different” with them. “It felt like I was trying to make the best of the same category of parts for a long time,” he continues.

The father of two says he has found play Prince Andreas “Easier in a way” than previous roles, because “there wasn’t any expectation that this was my kind of thing.”

“I think the opposite is often the case with parts I cast because of other parts I’ve narrated,” Sewell says. “People may think things are just the way I am, when in reality it takes a lot of work.”

Sewell admits he’s nowhere near the kind of roles he really wants to take on. “You could say I love it when people think they’re misjudging me,” he says. “I really enjoyed doing theater and comedy, and Hal has a little bit of both. That’s why it’s so fun, because it has a little bit of everything.”

The PEOPLE Puzzler Crossword is here! How quickly can you solve it? Play now!

Therefore, prior to the release of The diplomatIn his second season, Sewell says he feels safe for the first time in his career.

“This is actually the only time I feel safe,” he adds. “It’s only the third show I’ve ever done that had the opportunity to go on, the third long-term thing I’ve ever done. There’s a fear that you have to keep an eye on the writing because it can change, the voices can change, so it creates periods where you don’t sleep the same way. Now I look forward to getting scripts because I don’t care what happens story-wise because I completely trust what the show is.

Despite working in Hollywood for more than thirty years, Sewell says he doesn’t always get recognized along the way, and he considers himself “very lucky.”

“I often get recognized in the few months after something I’m in has been out, and then it just kind of disappears, which is really nice,” he says. “You get people coming up to you and saying nice things, but it doesn’t really hinder my progress in the city. What happens more often, and what I always try to be nice about, is when people come up to me and ask me where they know me from.”

Related: Why The diplomat Will have a shorter season 2 of six episodes

Sewell can usually guess. “I often notice whether people recognize me as JasperThe holiday or that they will recognize me The man in the high castlewhere I play an American Nazi,” he says. “But I usually don’t find out where they know me from and I send them on their way as kindly as possible. But I don’t care at all.”

Being unrecognizable to many, Sewell can easily enjoy his favorite day off: having a cup of coffee and diving into a good book.

“I always have a stack of books that I read one after the other. When I have free time, I like to find little cafes where I can just sit in the corner,” says Sewell, who is currently filming. The diplomat season 3 in New York City, says. “That’s why it’s quite nice that most people don’t recognize me. That’s absolutely fine.”

Never miss a story – sign up PEOPLE’s free daily newsletter to stay up to date on the best of what PEOPLE has to offer, from celebrity news to compelling human interest stories.

The diplomat season 2 premieres Thursday, October 31 on Netflix.