Matthew Rothman is used to working with people who pride themselves on their rationality. He's a "quant," after all, one of a legion of Ph.D.s on Wall Street who use the emotionless rules of mathematics to pick trading positions. But this week, he caught a whiff of panic.

The trouble started Aug. 3, when stocks started moving not only in ways that commonly used models didn't predict, but in precisely the opposite direction from what was expected. Equally troubling, the moves were far more volatile than models based on decades of testing assumed were likely. Those relatively minor anomalies escalated quickly this week, exploding into a global rout for quantitative funds by Wednesday.

"Wednesday is the type of day people will remember in quant-land for a very long time," said Mr. Rothman, a University of Chicago Ph.D. who ran a quantitative fund before joining Lehman Brothers. "Events that models only predicted would happen once in 10,000 years happened every day for three days."