For the last 20 years or so, the most likely place to find the publicity-shy novelist-playwright-screenwriter Cormac McCarthy would be at the Santa Fe Institute in New Mexico. Co-founded in 1984 by Murray Gell-Mann, the 1969 recipient of the Nobel prize in physics, SFI is a thinktank for maverick brainiacs, a flexible category that, in the judgment of the late polymath Gell-Mann, perfectly described McCarthy, his friend and MacArthur Foundation “genius” award winner.

Until recently on many weekdays, the writer could be heard at SFI, clattering away on his portable typewriter from behind his office door. An affable member of this elite community, with no specific tasks to perform, he would regularly emerge for afternoon tea or attend talks by SFI scholars and visiting academics on topics that interested him, such as complex systems theory or quantum computing.

I have known McCarthy since 1992 when I wrote a profile of him for the New York Times Magazine. I profiled him again in 2005 for Vanity Fair. We’ve spoken on the phone many times and, being a car racing fan, he joined me on assignment at the Monaco Grand Prix and at the Indianapolis 500. He came to my wedding. I don’t remember him ever saying that something was off the record. His desire for privacy seemed to suggest that I should know.

Lying on his desk when I visited SFI in 2006, while writing about a Russian linguist there, was a copy of Frank Ramsey’s The Foundations of Mathematics and Other Logical Essays. How deeply McCarthy, a University of Tennessee dropout, was able to drink from Ramsey’s slim but forbiddingly dry 1931 volume – he was perhaps Wittgenstein’s most brilliant acolyte at Cambridge – is hard to say.

Heady reading matter is nonetheless what McCarthy has long preferred. A scan of his shelves at SFI would reveal, instead of novels by Booker prize winners, manuscripts by his many scientist friends, such as Lisa Randall, a theoretical physicist at Harvard, or Lawrence Krauss, a cosmologist from Arizona State University.

During the half dozen times I have seen McCarthy at SFI, I’ve often wondered what he was doing with all this arcane material he was greedily absorbing. Rarefied science hasn’t figured in any of his writings, not even in work completed after he came to SFI, such as No Country for Old Men (2005), The Road (2006) or his screenplay for Ridley Scott’s The Counsellor (2013).

The answer seems to be that he has been saving up what he’s learned and has deposited a fair amount of it into The Passenger and Stella Maris, a pair of novels he is publishing this autumn, 15 years after his last novel The Road won the Pulitzer prize. (They can be read together or separately, and in any order.)

Stella Maris, out in December and more explicitly a product of his years at SFI, focuses on a former mathematics prodigy, Alicia Western, who has lost her place in academia as well as her tenuous grasp on reality.

A topologist once adept enough to have corresponded with the pre-eminent algebraic geometrician of the 20th century, Alexander Grothendieck, Western is now a psychological shipwreck, beset with hallucinations and thoughts of suicide.The book is like a two-person play told in seven interviews between the recuperating Western and a psychiatrist, Dr Cohen. He practises at Stella Maris, a Catholic sanatorium in Wisconsin where she is being treated as a paranoid schizophrenic. It’s her third stay.

The Q&A format gives McCarthy the chance to discourse, through Western, on any topic he chooses. As she retraces the trajectory of her life, for example, she tells Cohen that she has perfect pitch. She discovered music as a child – even before mathematics – and she has theories about both. “Music is not a language,” she declares. “It has no reference to anything but itself.” Her languid manner is a disguise for a stubborn resolve. More than one doctor has labeled her autistic and, being smarter than they are, she seems content to frustrate their attempts to figure her out.

She grew up at Los Alamos, about 30 miles from Santa Fe, where her father was a physicist who worked on the Manhattan Project and afterwards felt lifelong guilt for helping to build the first atomic weapons. She learned to solve equations with him at an early age. “All I can tell you is that I like numbers,” she says. “I like their shapes and their colors and their smells and the way they taste. And I don’t like to take people’s word for things.”

Alicia grew up at Los Alamos, about 30 miles from Santa Fe, where her father was a physicist who worked on the Manhattan Project and afterwards felt lifelong guilt for helping to build the first atomic weapons. She learned to solve equations with him at an early age. “All I can tell you is that I like numbers,” she says.

Her sole abiding relationship is with her brother Bobby, another haunted spirit, for whom she openly expresses incestuous feelings, perhaps the main source of her mental anguish – and his.

She is the first female intellectual in a McCarthy book and his first female protagonist since Rinthy in Outer Dark, his 1968 second novel. (That harrowing tale also revolved around incest: Rinthy searches the hills of Appalachia for her abandoned baby, born as a result of sex with her brother, Culla.)

Alicia is also a major character in The Passenger, the autumn’s other McCarthy novel, published next week. The first scene sees her holed up in a skeevy Chicago rooming house, where she trades insults with her most aggressive hallucination, who functions chiefly to debase Alicia in crude and offensive terms. “Jesus, he said. This place really sucks. Did you see what just crossed the floor? What, are we completely out of Zyklon B?”

The novel has some of the chase and crime elements found in No Country for Old Men. A private plane has gone down near the city of Pass Christian, Louisiana. Bobby Western, who works as an itinerant salvage diver, is hired to recover the pilot’s flight case, the black box, and a passenger. Ten people were booked on the flight but only nine bodies were recovered. While probing the waters of the Gulf of Mexico, Western discovers things that agents with ill intent and unknown allegiances (CIA? Hired guns for a syndicate?) want to take from him. As he flees across the country, they pursue.

The book is in some ways a departure for McCarthy, as Alicia and her hallucinations are woven in and out of the action. In other ways, though, The Passenger marks a return to the pre-SFI, pre-MacArthur winner McCarthy. It is set primarily in New Orleans, which was McCarthy’s home in the 1960s. Much like McCarthy, Bobby Western is a taciturn autodidact. Through his father’s tutoring, he is as informed about particle physics as his sister. Instead of befriending academics, though, he keeps company in the neighbourhood bars with a raffish crowd of opinionated drunks and ne’er-do-wells. He’s the well-liked sounding board for their theories, homespun or cosmic. “You can’t get a decent cheeseburger in a clean restaurant,” claims his friend Oiler. “Once they start sweeping the floor and cleaning the dishes it’s pretty much over.”

These gatherings recall scenes from McCarthy’s novel Suttree (1979). Much of his early fiction was set in or around Knoxville, Tennessee, McCarthy’s home after a stint in the US air force and before he moved to the west in the 1980s.

He speaks fondly of his Knoxville years and once told me that all his friends “were carried off by drugs and drink, of course, but some of them very bright, quite well educated”.

Bobby Western’s checkered biography includes time spent in Knoxville. The city is about 30 miles from Oak Ridge, another of the sites where the atomic bomb was developed. In one scene he visits his grandmother’s house near there. “Western fully understood that he owed his existence to Adolf Hitler.”

McCarthy often seems to have worked on several projects at the same time. After he completed No Country for Old Men in 2005, he told his then-editor Gary Fisketjon that he was nearing the end of a novel about salvage divers in New Orleans. (Asked which one did he want first, Fisketjon replied that “it would be foolish to express a preference”.)

McCarthy is not a psychological writer in the traditional sense. His characters aren’t usually given elaborate back stories explaining motives for their present behaviour. They act and react. He describes with precise economy, as well as sometimes thunderous, witty or laconic oratory, what they do and say, the physical world they inhabit, the risks they face, and the often dire consequences of being alive.

The Passenger, by providing long family histories for Alicia and Bobby, is an exception. But the illicit passion that sister feels for brother in the two books is treated without moralising. You either accept it as an aspect of her character or not.

McCarthy is not an optimist about his own luck with male-female relationships (he’s been married three times) or about social progress. I’ve never asked him if he thinks the arc of the moral universe bends toward justice, because I think I know the answer.

“Evil has no alternative plan,” says one of Bobby’s Knoxville barroom pals in The Passenger. “It is simply incapable of accepting failure.”

McCarthy has cheered himself up over the last 20 years by “being around some of the smartest people on the planet” at SFI. In 2009, in support of their mission and in gratitude for their welcoming him into their midst, he donated the proceeds from an auction at Christie’s of his Olivetti Lettera 32 manual typewriter: $254,500.

Winning the MacArthur Foundation fellowship in 1981, on the recommendation of Saul Bellow and others, changed McCarthy’s fortunes. He has called it “the most profound experience of my life”. Its annual reunions introduced him to scientists from a range of fields he had until then only read about. Several, including whale biologist Roger Payne and Gell-Mann, became lifelong friends.A story collector as well as a storyteller, McCarthy never tires of hearing about the habits and mental gymnastics of great minds, especially 20th-century physicists such as Robert Oppenheimer and Richard Feynman, both of them at Los Alamos.

Both Alicia and Bobby Western allude to Feynman, a colleague and rival of Gell-Mann’s at the California Institute of Technology in the 1950s and 60s. McCarthy delighted in telling me at breakfast one day of the terror they would instil in other physicists. “They would sit on either side of the front row in the lecture hall, both often arriving late. Murray would read a newspaper. Fifteen minutes into a talk Feynman would call across in a stage whisper, ‘Hey, Murray, is this guy smart?’ And Murray would either nod yes or no.”

McCarthy’s years at SFI have inspired reflections on what might have been. “There were a lot of things I could have done,” he once told me. “I certainly had a pretty good grasp of architecture. But truthfully I’m not a scientist. I don’t really think like that. I could have been a physicist but not a world-class physicist. Whatever I was going to be I wanted to be really good.”

McCarthy is 89 and these two novels will likely be his last. Death is not a subject he has ever shied away from, in his fiction or conversation. Indeed, he has measured other writers by how seriously they address it. Both books offer evidence that he has spent time thinking about his own eventual end.

“I don’t think there is some way to prepare for death,” says Alicia in Stella Maris. “There’s no evolutionary advantage to being good at dying. Who would you leave it to? The thing you are dealing with – time – is immalleable. Except that the more you harbor it the less of it you have. The liquor of being is leaking out onto the ground. You need to hurry. But the haste itself is consuming what you wish to preserve. You can’t deal with what you’ve been sent to deal with.”

Gell-Mann’s death in 2019 was a severe loss, though it may also have given McCarthy the impetus to finish books that he had left half-done for too long. Both these novels are chilling and bleak – his readers expect no less – and he seems to take perverse delight in imagining the worst that can befall his characters and describing their failings in his own sprightly prose music. The future extinction of the human race will in any case make individual deaths like his and Gell-Mann’s irrelevant.

“When all trace of our existence is gone,” asks Alicia, “for whom then will this be a tragedy?”