In mid-May, CNN reported that the U.S. intelligence community was about to begin a sweeping review of the way it does business. What prompted the senior officials to action? The answer is simple enough: alarmingly inaccurate predictions as to the durability of the U.S. supported government of Afghanistan, which led to a decidedly ignominious withdrawal of our forces there, as well as overly pessimistic projections of Ukraine’s ability to stave off a major assault by the Russian army.

In view of the gravity of those mistakes, this seems a necessary and laudable undertaking. But… don’t expect the review and inevitable list of recommendations to improve the complicated process of gathering, analyzing, and consuming intelligence products by much. So say two of the leading scholars of intelligence in the English speaking world, Richard Betts and the late Robert Jervis, both of Columbia University’s political science department. After decades of studying the question, these men have concluded that invariably the recommendations of commissions designed to improve the caliber of the intelligence process after American wars tend to produce a new set of problems. As Betts put it in a widely quoted essay on this topic:

Strategic intelligence, which Betts nicely defines with admirable economy as “the acquisition, analysis, and appreciation for relevant data,” is an extremely tricky business. It’s a unique amalgam of science and art, for it invariably involves political and psychological factors that are unique to a given conflict, and subject to abrupt change. And it must not be forgotten that senior intelligence officials have to sell their product well if it is to carry real weight with consumers, and that’s an entirely separate skill than producing good analysis.

Of course, serious students of recent American military history already have a basic understanding of what went wrong in assessments of the final phase of the Afghan tragedy, and in the first phase of the Russia-Ukraine War. Broadly speaking, the American intelligence community—the 18 agencies involved in its collection , along with the chief consumers, the White House and the National Security Council—have become overly dependent on quantitative analysis derived primarily from technical and electronic sources (signal intelligence), at the expense of both human intelligence (agents and sources on site in the arena of conflict) and expertise about the political dynamics and cultural histories of foreign societies.

What Clausewitz called moral, or spiritual, factors in his masterwork, On War—the will to fight among the soldiers of an army, the level of popular support for the government, the creativity and intuition of the political leaders of the adversaries—these are things that Clausewitz says “cannot be classified or counted. These have to be seen and felt.”

On paper, the American-trained Afghan Army of more than 300,000 troops, armed with far more sophisticated weapons than the Taliban, including drones and jet fighters, should have been able to hold off the final offensive Taliban onslaught well into 2022. That didn’t happen, because except for some 30,000 Afghan Special Forces, the rest of the “Army” had no interest in defending a government they and their families perceived to be corrupt, ineffectual, and in the pocket of the West. The majority of the Afghan Army units did not put up any resistance to the Taliban. They negotiated their own surrender or offered no resistance whatsoever.

As for the CIA projections that the Russians would break the back of Ukrainian resistance in a matter of days, it’s clear that analysts relied too much on their quantitatively-based assessment of Russian units and weapons systems, while their grasp of Clausewitz’s “moral factors” on both sides was shaky, at best.

One of the most significant failures in U.S. intelligence since Vietnam was the community’s inability to get a grip on the swirling political and military developments surrounding the Iranian Revolution of 1979. In February of that year, a bizarre collection of liberal reformers, leftists, and Muslim fundamentalist clerics overthrew the Shah of Iran, at the time the United States’ most powerful ally in the Middle East and a bulwark against Soviet expansionism. Led by a glowering, mysteriously charismatic cleric, Ayatollah Khomeini, the clerics deftly outmaneuvered and marginalized their revolutionary allies, and established the world’s first modern Islamic Republic.

Anti-Americanism had been the glue that kept together the disparate factions of resistance to the Shah’s rule. All the revolutionaries believed that the Shah, whose regime had become increasingly oppressive and corrupt, was in the pocket of Washington. Washington completely misread the dynamics of Iranian politics. Less than a year before the Shah was ousted, President Jimmy Carter had praised the Shah lavishly, calling his regime “an island of stability in a turbulent corner of the world.” The turbulence and rising tide of anti-Americanism in Iran had been in plain sight for several years, but the American intelligence community had developed no contacts among the myriad opposition groups and depended heavily on the Shah’s intelligence agencies. They told the Americans not what was really going on, but what the Shah wanted the Americans to know.

The Carter administration’s responses to fast-moving developments in Iran before and after that event, including the infamous hostage crisis of 444 days, was halting, contradictory, and in the opinion of every serious historian of U.S. relations of whom I’m aware, depressingly inept. Among the U.S. intelligence community, opines the noted military historian Lawrence Freedman in his history of U.S policy in the Middle East, A Choice of Enemies, “there was little grasp of the internal power struggles that were soon underway in Tehran. The diplomats and intelligence specialists sent to try to pick up the pieces of U.S.-Iranian relations lacked any expertise in the ideological wellsprings of the Islamic movement… Because clerics were not generally known for their lust for power or their appetite for government, the comforting assumption was that their role would soon be circumscribed by proper politicians.”

Professors Betts and Jervis join a wide consensus of scholars in believing that the most egregious intelligence failures in recent American history lie more with the top-level consumers of intelligence than with the CIA or the other myriad organizations involved in its collection and analysis. Here, the chief villains, writes Betts, are “wishful thinking, disregard of professional analysis, and the preconceptions of consumers.” There was nothing impulsive about the series of decisions that committed the United States to fighting a major war in Vietnam, and then prolonged America’s commitment to winning that conflict, even as signs of failure began to accumulate like buzzards around a corpse.

Between 1950 and the summer of 1965, three U.S. presidents opted to expand America’s involvement in Vietnam, despite that ancient Asian country’s seeming irrelevance to American vital interests, and the extraordinary level of dysfunction and corruption among America’s Vietnamese allies. Had President Johnson heeded the CIA’s pessimistic reports about American prospects in Vietnam, he never would have committed the country to a major ground war.

While the Johnson administration’s “best and the brightest” justified America’s growing military presence in Southeast Asia as a proper response to “wars of national liberation” sponsored by the Kremlin and Beijing, the CIA consistently pointed out that this was simply not the case. Hanoi ran its own show, deftly playing off one communist superpower against the other, and frequently decided to go its own way in the prosecution of the war effort against the Americans. The Agency’s doubts about the trajectory of American policy in the war were especially pronounced during late 1964 and early 1965, when the Johnson administration crossed the Rubicon by deploying American combat units to take the fight to the enemy in the South in March 1965. In effect, Johnson took over management of the war from the South Vietnamese and put it in the hands of his own generals.

Here is a brilliantly prescient assessment by CIA analyst Harold P. Ford, written in April 1965, just as LBJ was committing American Marines to offensive operations for the first time:

Johnson ignored Mr. Ford’s sage advice. Within weeks of receiving this report, he approved General Westmoreland’s three-phase plan to win the war by 1968 through a strategy of attrition. Using as many as half a million U.S. troops, he would destroy the enemy’s main forces with massive “search and destroy” sweeps, using American mobility and firepower to vanquish an enemy without any air power whatsoever, and little motorized transport. Westmoreland would pay lip service to the CIA’s belief that the war had to be fought and won in the villages, but he’d fight and win in the traditional American way: conventional warfare, emphasizing air power and artillery, even though American military operations inflicted massive destruction on the people America had come to South Vietnam to “save.”

Why did America’s policy makers dismiss the astute counsel of the CIA’s wise men? The short answer is that they couldn’t break free of the domino theory—the false notion that if one state fell to communism, a string of others was sure to follow, and that this would lead to an irreversible loss of credibility and prestige for the United States… and for Lyndon Johnson and his senior advisers.

One of the most subtle and perceptive of the CIA analysts, George W. Allen, puts it well in his book, None So Blind: “America failed in Vietnam not because intelligence was lacking, or wrong, but because it was not in accord with what its consumers [i.e., Ike, JFK, LBJ, and their chief advisers] wanted to believe, and because its relevance was outweighed by other factors in the minds of those who made national security policy decisions.”

The disastrous decision by the Bush administration to invade Iraq grew out of a refusal to listen to good intelligence analysis as well. From the spring of 2002 forward, Bush joined with Cheney, Rumsfeld, and several other influential hawks in marginalizing a very substantial body of intelligence and analysis from within and outside the government indicating that an invasion of Iraq might well create more problems for the United States, Iraq, and the entire Middle East than it would solve.

This, at least, was the considered impression of no less a figure than Richard Dearlove, the head of Britain’s equivalent of the CIA, MI6, who engaged in top-secret discussions with the American president and his principal advisers in early July 2002.

A summary of Dearlove’s testimony about those meetings was recorded in a top-secret Downing Street memo: “There was a perceptible shift in attitude. Military action was now seen as inevitable. Bush wanted to remove Saddam, through military action, justified by the conjunction of terrorism and WMD. But the intelligence and facts were being fixed around the policy. The NSC had no patience with the UN route [of diplomatic pressure] . . . There was little discussion in Washington of the aftermath of military action.”

Indeed, the most reliable and objective accounts we have of the administration’s deliberations agree entirely with Dearlove’s assertions that the intelligence and facts were being manipulated to fit the administration’s policy inclinations, and that there was precious little discussion of the likely aftermath of cutting the head off the snake in Iraq.

In its secret discussions during the planning phase and in its public defense of the project, the administration aggressively “worst-cased” the threat posed by Saddam, and “best-cased” the results of removing him from power.

A four-star general who worked on the war plan for months told military writer Tom Ricks that he felt the president was shielded from the advice of those in the upper ranks of the military who thought the United States was heading into a quagmire both before and after the invasion commenced. That advice, he said, was “blown off by the president’s key advisers… the people around the president were so, frankly, intellectually arrogant. They knew that postwar Iraq would be easy and would be a catalyst for change in the Middle East. They were making simplistic assumptions and refused to put them to the test.”

The CIA and State Department analysts were far, far less sanguine about what might happen as a result of the invasion than Rumsfeld, Cheney, and the other hawks. According to Paul Pillar, the top CIA coordinator for intelligence on Iraq from 2001 to 2005, the professional intelligence community presented a picture of a political culture in Iraq that would not provide fertile ground for democracy and foretold a long, difficult, turbulent transition.

It projected that a Marshall Plan-type effort would be required to restore the Iraqi economy, despite Iraq’s abundant oil resources.

It forecast that in a deeply divided Iraqi society, with Sunnis resentful over their loss of their dominant position and Shiites seeking power commensurate with their majority status, there was a significant chance that the groups would engage in violent conflict unless and occupying power prevented it.

And it anticipated that a foreign occupying force would itself be the target of resentment and attacks—including by guerrilla warfare—unless it established security and put Iraq on the road to prosperity in the first few weeks or months after the fall of Saddam… War and occupation would boost political Islam and increase sympathy for terrorists’ objectives—and Iraq would become a magnet for extremists from elsewhere in the Middle East.

The policy implications of “the entire body of official intelligence analysis,” said Pillar, was to avoid war, or “if war was going to be launched, to prepare for a messy aftermath.”

Vietnam and Iraq, of course, were fundamentally irregular, or asymmetric conflicts. Far more than conventional conflicts, irregular wars are shaped more by politics and political organization among the people than by military operations. Since Vietnam, America’s senior foreign policy decision makers have a very unfortunate habit of forgetting this fundamental truth. They have been overly enamored by the power of the U.S. military machine, but obtuse in failing to recognize the limits of military power alone to shape politics in foreign societies.

This tendency goes far in explaining why the United States keeps losing wars.