Get SF Weekly Newsletters
Pin It

Rewarding Failure 

Streetcars that are too wide, long, and heavy (136 @ $3.5 million per). Bus engines that don't fit and are scrapped (20 @ $116,000 a pop). Manhole covers that take 10 engineers and three months to design (cost: $243,000). Muni's management has produced su

Wednesday, Dec 9 1998
In August, Muni's subway had the transit equivalent of a brain hemorrhage. Muni's computerized controls went haywire, and its trains lurched about the Market Street tunnel like stroke victims. Muni shuddered to infuriating halts for weeks.

Just two months after the subway meltdown, though, Muni General Manager Emilio Cruz coolly shared a wooden dais with the five mayoral appointees who serve as the Public Transportation Commission. Exuding the confidence seen usually on the faces of babies, astronauts, and highly paid publicists, Cruz bragged that 33 trains an hour were cruising through the Muni underground. The commissioners smiled and nodded. More trains were moving through the tunnel; the crisis was over.

Suddenly, an outrageous act of rebellion was perpetrated. Ray Antonio, president of the Muni drivers union, stood to deliver a fierce correction. "But those are only one-car trains," Antonio almost shouted. "They used to be four-car trains, but now they are only one-car trains." The implication was clear: More people were not moving through the Market Street tunnel, and even Muni's drivers could only bear so much commuter hell.

Cruz and his commissioners stared at Antonio as if he had spent the last week under a wet rock. "Most riders would rather have a shorter trip standing, than a longer trip sitting," Cruz explained in soothing tones. Antonio deflated and sat. Newspaper reporters yawned. The commissioners watched Cruz for cues.

Ready to move on, Commission President H. Welton Flynn asked, "Do you feel good about Breda?" Cruz nodded sagely. Flynn sighed, apparently in relief, and the commission voted to sign over another $167 million to Breda Costruzioni Ferroviarie S.p.A. of Pistoia, Italy. Without a whisper of dissent, the San Francisco Municipal Railway had spent another huge sum of money to buy 59 more Breda streetcars -- even though the Bredas long ago proved themselves to be a very bad fit for the Muni system.

Muni's subway suffered a stroke this summer because two of its main elements -- the Alcatel computer software meant to control trains in the tunnel, and the new Breda cars supposedly designed to work with that control system -- did not work very well, singly or together. In fact, in the wake of the Muni Metro disaster, Mayor Willie Brown threatened to sue Alcatel, and since then he has publicly savaged Muni's purchase of Breda streetcars.

The mayor's criticisms were understated by many orders of magnitude.
Breda's "light" rail vehicles are, public records show, too heavy, too long, too wide, and too noisy to work acceptably on Muni tracks. Also, they are far more expensive than comparable streetcars.

Alcatel's train control system -- according to contract amendments on file with Muni -- was unproven in service when it was acquired by the transit agency and outdated before it was unpacked. It has already cost at least $17 million more than its original budget had forecast.

Both Alcatel and Breda were shepherded into San Francisco by a McLean, Va., consulting firm, Booz Allen Hamilton. Both contracts were awarded and extended under unusual circumstances. What's more, under its contract with Muni, Booz Allen is paid for supervising Breda and Alcatel in a way that might have fit well in the novel Catch-22: The longer the Breda and Alcatel projects take, the more costly they become, the more money the consultant who recommended the Breda and Alcatel deals will make.

The Breda and Alcatel projects are not anomalies, and Booz Allen Hamilton is not necessarily a lone bad guy. Public documents show that Muni's long-range management decisions have produced incredibly questionable results for some time, whether Booz Allen was or was not involved. For every Booz Allen-recommended, glitch-happy train control system it acquires, Muni creates several other boondoggles entirely on its own.

Muni's top managers and the Public Transportation Commission hire consultants and make final decisions for the transit agency; they, therefore, bear responsibility for Muni's track record in long-term management. That record is full of overpriced, unworkable purchases and expensive, complicated plans that never seem to work, and are subsequently replaced by other complicated plans that do not work. Muni's long-term management record is so bad, in fact, that any reasonable analysis winds up asking the question: Is this incompetence or something worse?

As the 1990s dawned, Muni planners were eager to get rid of the streetcars then in use, built by Boeing-Vertol, the helicopter division of the Seattle-based Boeing Co. The Boeing streetcars had a history of frequent breakdowns and were themselves widely recognized as a Muni boondoggle. Excited planners envisioned acquiring a fleet of smart streetcars guided by the invisible hand of a computer. The new cars would be a special boon to the Market Street subway, where all of Muni's streetcar lines merge: Under central computer control, a train a minute would glide through the Muni underground, transforming the 100,000-passenger crush of morning and evening rush hours into a pleasant commuting experience.

The streetcars Muni eventually bought, at $3.5 million apiece, were made by the Italian manufacturing firm of Breda Costruzioni Ferroviarie S.p.A., but neither the cars nor the decisions to buy them were particularly smart.

Breda's streetcars are too big to comfortably fit Muni's tunnels. Yes, it's true: The streetcars, supposedly custom designed for San Francisco's system, are too wide (some platforms had to be shaved); too heavy (the extra weight, and a lopsided distribution of that heft, is destroying Muni's 70-year-old rails); and too long.

The length problem carries its own special irony: The Breda rail cars were bought, purportedly, to mesh with a computer control system that would move more trains and more passengers through the subway at peak hours. Because the rail cars are two feet too long, four-car Breda trains will not fit in front of several Muni station platforms. Therefore, even if a central control system could move more trains through the Market Street tunnel, the trains would each be one car shorter -- and fewer passengers would move.

The Breda streetcars have other problems. Government records say the vehicles' sliding doors break down constantly. Also, because the doors are on the outside of the car, there is a dangerous 5-inch gap between the door step and Muni platforms. The emergency brakes are so suspect that safety inspectors have reduced the allowable train speed from 50 miles per hour to 30 miles per hour. Excessive weight and faulty engineering cause high- and low-frequency noises that irritate, even enrage, entire districts of the city.

Why would Muni buy these pigs in the poke, and keep on buying them, long after the agency knew they were wrong for its system?

When it comes to managing billions of dollars of capital projects, Muni bureaucrats report directly to Booz Allen Hamilton Inc. A globally connected consulting firm, Booz Allen is a major player in public works management from Washington, D.C., to Kuala Lumpur; the firm is charging Muni about $30 million for consulting on the Breda streetcars and the Alcatel guidance system.

Under its contract with Muni, Booz Allen handles capital procurements from start to finish. The firm writes specifications and bid proposals, and influences the selection of contractors. It manages the nitty-gritty work on a daily basis; in large part, Muni officials sit back and sign off on Booz Allen invoices. And the more time a project takes, the more Booz Allen bills -- at $200 an hour.

There are a dozen major streetcar manufacturing companies in the world. In 1991, Muni advertised for a company to manufacture 35 customized streetcars. Booz Allen Hamilton created the streetcar design specifications, and guided Muni through the bidding process.

It was a very strange process.
A German, a Japanese, and an Italian firm all sought the contract. The Japanese company, C. Itoh & Co. Inc., put in the lowest bid -- about $150,000 per car less than Breda, the next lowest bidder. Price, of course, is not the only factor evaluated in deciding whether to purchase technology as complicated as that used in rail cars. Muni's bid specifications deemed a proven track record in making streetcars and the ability to meet certain technical specifications to be as important as price.

Public records show that Breda's bid was higher than C. Itoh's, and that Breda had less streetcar manufacturing experience than the Duewag Corp., the German firm in the running for the contract.

C. Itoh's low bid was disqualified on a minor technicality. (To this day, city officials are at a loss to explain exactly why the low bidder lost.) With C. Itoh out of the way, city bureaucrats wrote that the remaining two bids, by Breda and the German Duewag Corp., were so close in terms of price and technical promises that either could be chosen. Only after a series of meetings between the two remaining bidders and Booz Allen Hamilton did a victor emerge. In December 1991, Muni awarded the streetcar contract to Breda -- paying $2.5 million for each of the 35 custom-designed streetcars.

This was the beginning of a very good deal for Breda. Muni soon ordered another 42 cars at premium prices -- up to an additional $355,000 for each car. Eventually, Muni ordered 101 additional streetcars (for a total of 136) at an average of $3.47 million for each car -- or 30 percent more per car than was originally bid. At this price, the Muni Bredas may well be the most expensive streetcars on the planet.

After the first contract was let, the subsequent streetcar purchases were never put out to competitive bidding. In government-speak, the 101 additional streetcars were "sole-sourced," or simply added onto Breda's original contract. The lack of competition showed: Each batch of Bredas cost more, per car, than the last.

While San Francisco ended up paying nearly $3.5 million apiece for its streetcars, San Diego was buying light rail cars from Siemens-Duewag for a bit over $1.5 million each. (The federal government's National Transit Database shows that the average cost of a new light rail car in America is currently $2.381 million, or $1 million less than San Francisco pays.)

There were other odd fiscal facets of the Breda contract, including Muni's subsidization of a Breda factory at Pier 80 in San Francisco (see sidebar). But the real question is not why Muni is paying so much to Breda, but why it keeps buying Breda streetcars at all.

According to public records, the "custom" design of the Breda streetcars contained large and small errors that did not become apparent until the cars were test run on the rails in San Francisco. Some flaws were fixed, after expensive modifications. To address the weight problem that generates neighborhood-disturbing rumbles and eats Muni's track, for example, Breda is retrofitting its 38-ton cars with "softer-riding" suspensions.

Breda officials declined to be interviewed for this article. In an interview last week, Cruz said he was unaware that the Breda cars are too long, but that the Bredas' weight problems were being addressed. Attempts to obtain more detailed comment this week from him were unsuccessful.

But if Muni keeps using the Italian rail cars, San Francisco will have to learn to live with some Breda problems. Muni can do little to further reduce the soul-jarring whine that emanates from the heart of the Breda motor.

And there is no way to amputate two feet of length from a Breda streetcar. And so there is no way that $472 million of streetcars will ever fix the subway problem they were supposedly designed to solve.

Ten years ago, Muni planners dreamed of transit utopia. Sleek carbon-steel street-cars would speed through the underground in multicar trains that would come and go automatically, guided by omniscient central control.

Today, however, lone or paired streetcars straggle, very occasionally, into packed subway stations, and waiting thousands crush one another in the rush to load the steel cans.

What went wrong?
Simply put, Muni tried to swallow more than it could digest. Along with building an entirely new fleet of streetcars, Muni set about revolutionizing its signal system by acquiring Alcatel's automatic train control software. If it worked, the control system would be a wonder. In practice, it has been a disaster.

In the Alcatel system, a cable is laid down the middle of Muni's underground tracks. Each streetcar is equipped with a transponder (basically a radio that sends and receives signals from the cable). Using these signals, software running on a simple personal computer keeps track of and -- in theory -- drives trains in the subway from afar. The driver becomes peripheral, even as computer peripherals become the driver.

This supermodern technology was meant to replace the 20-year-old electromechanical relay signaling system -- a system that controlled tunnel traffic lights, which were obeyed by train operators. The relay system had worked just fine in the Muni Metro, and could have been upgraded relatively cheaply. Instead, Muni chose to graft Alcatel's expensive wares onto Muni's humancentric signal system -- which turned out to be a very bad idea, indeed.

There are a number of technical reasons why the Alcatel signal system has never worked properly. Public records show that the system contained design flaws, and that there were manufacturing flaws in some of the equipment used. But it is also now clear that the Alcatel system installed in San Francisco had never been proven in practice, anywhere in the world. To this day, the system remains an experimental product.

So why did Muni buy an experimental guidance system -- especially when the agency's own proposal explicitly states that the system had to have been operated successfully elsewhere?

Booz Allen Hamilton is responsible for "selling" Muni on Alcatel. After designing Muni's bid requirements for a train control system, Booz Allen insisted that only Alcatel Canada Inc. could do the job. This insistence, at times, bore at least the appearance of bias.

Alcatel's competitors were pre-emptorily disqualified because they would not promise there would be no disruption when the new signal system came online. AEG Westinghouse's bid, for example, was thrown out because the company would not swear that its software could simultaneously control the Boeing and Breda cars during the changeover from old to new signals.

Alcatel, on the other hand, solemnly vowed that the transition would be painless. But Alcatel would not guarantee that its software could keep track of 40 trains at one time, another bid requirement. At this point Alcatel could have been disqualified, too. But rather than toss Alcatel, Booz Allen wrote a report -- after bids were opened -- that recommended an easing of the 40-trains-at-once standard.

Another requirement for getting the job involved past performance: The software had to have been successfully proved in service for two years. Booz Allen Hamilton insisted that a train control system Alcatel was installing in London, England, for the Docklands Development Corp.'s new light railway met this requirement.

Muni and Booz Allen failed to mention to federal officials one unusual fact: The Docklands Railway was almost a toy railroad, when compared to the Muni Metro. At the time, the Docklands system used a driverless train moving a maximum of only 5,000 passengers an hour. At rush hour, Muni has to move 20 times that many riders.

On the basis of Booz Allen's lobbying for Alcatel, the Federal Transit Administration allowed Muni to disqualify AEG Westinghouse's bid and to give the contract to Alcatel on a "sole-source" basis -- that is, on the contention that no other responsible company could provide the system.

Shortly after Muni hired Alcatel, Alcatel's Docklands system crashed. Docklands hired Booz Allen to help fix the system, but to this day, the Docklands train control software is rife with bugs.

And so, of course, is Muni's.
Booz Allen was able to direct the contract toward Alcatel because -- as with the Breda streetcars -- Booz Allen designed the specifications and micromanaged the selection process for the contract.

With the Breda bids, one of the two major contenders was disqualified on a minor technicality. With the Alcatel bid, the winner was allowed to promise more than it was capable of delivering.

Years later, the Alcatel system failed its most crucial test -- guiding a mixed fleet of Boeing and Breda streetcars through the Market Street tunnel while the transition between signal systems was made. When Emilio Cruz was interviewed by Rescue Muni, a nonprofit watchdog group, about why Muni hired Alcatel, he said, "We were sold a bill of goods."

Alcatel representatives did not respond to phone calls seeking comment for this article.

A spokesperson for Booz Allen Hamilton confirmed that the Breda streetcars are too long for the Muni Metro and are overweight. The spokesperson confirmed that Booz Allen drew up the specifications for the Breda cars according to Muni's parameters and oversaw the specifications for the Alcatel system, which still does not work properly. When asked why there were problems with measurements for the streetcars, the spokesperson replied, "Those reasons are lost in the dawn of time." The spokesperson also said, however, that the streetcar contract could have been put out to bid after problems cropped up with the Breda vehicles, but "Muni asked us not to."

The Alcatel goods came at a very high price.
Alcatel was awarded a $34 million contract for a project that Muni had initially estimated to cost $25 million. With options and cost overruns, the job is now costing $80 million. (Muni chose to pass millions in cost overruns from the Alcatel project directly on to local taxpayers, rather than seeking more federal funding, which would subject the failing project to closer federal scrutiny.)

Muni has handed a series of nice breaks to the North American subsidiary of Alcatel Alsthom Cie General Electricite, a $30 billion multinational corporation located in Paris, France. Instead of suing Alcatel -- as the mayor threatened -- Muni has twice rewritten the terms of the original contract to suit Alcatel's technical and scheduling needs. Muni is withholding $5 million in Alcatel payments -- but that's small restitution for trading a reliable signal system for one that may never work.

Consultants have been feeding on San Francisco's transit system for decades. Shortly after the turn of the century, San Francisco Mayor Eugene Schmitz and most of the members of the Board of Supervisors went to prison for accepting bribes from a consultant to a private streetcar company. An early Muni consultant, Bion J. Arnold, built tracks into empty dunes that had been bought up by his land-speculating friends. After World War II, a pair of military-industrial consultants, Col. Marmion "Insider" Mills and Col. Sidney Bingham, engineered Muni's switch from environmentally friendly electric streetcars to smoke-spewing diesel buses sold by General Motors.

But Muni does not need to hire consultants to mess up capital projects. Muni can do that on its own.

Muni's most recent internal financial reports show that of Muni's 82 ongoing capital project line items, 46 are over budget -- to the tune of $91 million. Muni can't perform simple addition and subtraction, say its overseers, who note, among other things, $100 million in accounting errors at the agency. And, outraged auditors sputter, Muni has a disturbing habit of throwing away original budgets and schedules in childish attempts to hide cost and time overruns.

Incompetence or something worse? Cascades of large and small "mistakes," chronicled in government records, keep raising the question:

* Two years ago, Muni's engineering division needed to quickly spend some federal money that was earmarked for rehabilitation. It wasn't enough money to rehab a whole building, but it wasn't an insignificant sum, either. As it turned out, there were a series of manholes along the city's cable car lines that were constructed of a "light-weight metal material that warps." The hatch covers did not need to be redesigned, really; all that was needed was a heavier metal cover. Ten engineers were budgeted to spend up to three months deciding what metal to use for the new covers. The cost: $243,342.

* Last summer, Muni paid a consultant to design a computer center at Muni headquarters on Presidio Avenue. After the design was almost complete, Muni executives decided to move the computer center to a temporary office at 875 Stevenson St. So a new design became necessary -- for an office that Muni hopes to leave by the end of 1999.

* Not long ago, Muni paid $2.3 million for 20 massive diesel bus engines, but did not add them to its property inventory. By the time an auditor stumbled across the $116,000 engines, they had been left to rust, because they were the wrong size for Muni's buses. The auditor suggested the engines be sold for scrap value.

* City Budget Analyst Harvey Rose was appalled in 1996 when he found that Muni was "jeopardizing multimillion dollar investments" by completely replacing its fleet of streetcars, buses, and trolleys before finding anywhere to store the new vehicles. And attempts to provide more storage and maintenance space have gone almost humorously awry. Building expansions and improvements have repeatedly been torn apart because architectural drawings specified sizes that turned out, later, not to fit the equipment to be housed.

* Muni engineers have been less than clever about specifying designs for new electric trolley buses. The first round of buys -- a small fleet of trolleys built by New Flyer Industries Limited of Winnipeg, Canada -- was terminated after Muni discovered "inherent design and performance difficulties with the structure and rear axle assemblies." So Muni went to the Czech Republic and bought buses from the Skoda manufacturing firm. But Muni planners forgot to match the new concrete passenger platforms on Market Street (58 inches tall) to the 55-inch-tall step measurements they gave to Skoda. This seemingly small mistake forced major changes to the trolley frame -- and major expenditures by Muni.

It is not necessary to read through file cabinets full of audits, bid proposals, and consultant invoices to know -- and to know absolutely -- that the Municipal Railway does not work. Muni's inability to keep to schedule has made the agency's name a verb. If one is late in San Francisco, it is often acceptable simply to say, "I was Munied."

The reasons for Muni's continuing ineptitude cannot be called simple, because they have developed over decades, but they are hardly as complicated as Muni's managerial and political leaders claim.

There is a core, day-to-day reason why Muni's buses and rail cars run late 45 percent of the time: The agency's union contracts allow employees to stay away from work in droves. A third of the work force on any given day is absent, and when hundreds of drivers and mechanics don't show up, Muni's trains and buses cannot possibly run on time. Paradoxically, this chronic absenteeism has the effect of generating more than $20 million in yearly overtime payments to drivers and mechanics who fill in for absent co-workers.

As daily operations are currently managed, there is no incentive for Muni to run on time, because Muni workers profit when Muni runs late.

Muni's long-term management problems boil down to a question of accountability. The public record reveals incredibly questionable decisions that have led to amazingly costly errors on a regular basis. Yet, the managers and consultants responsible for those mistakes have not been fired. Rather, their science is arcane enough that their explanations are often simply accepted at face value. Bad decisions and mistakes are continued long after they have become known within the transit agency as boondoggles.

Entire stacks of audits and reports show that Muni's current problems are recycled versions of the same old problems. Mayor Willie Brown and Muni General Manager Emilio Cruz have, by and large, suggested the same old solutions to those problems: more money and more employees.

But Muni already plans on spending nearly $4 billion on capital purchases over the next 20 years, and the agency has some $2 billion in the federal funding pipeline. Muni's operating budget has grown dramatically over the last several years, as has its number of authorized employees. Lack of money is not, actually, at the heart of Muni's long-standing failure.

There are ways to fix Muni. It is possible to negotiate reasonable union work rules that reduce Muni's crippling rate of absenteeism. Muni management can be professionalized, and managers who regularly make illogical, costly decisions can be fired. It is possible to buy quality transit equipment at a reasonable price.

But when employees don't show up to work, and suffer no real consequences, when managers waste millions -- even tens of millions -- of dollars and are not disciplined, when major purchases appear to have been steered to undeserving firms, at some point motivation becomes irrelevant. It doesn't matter, really, whether boondoggles are born of incompetence or something worse. Muni fails because it fails to punish, and even rewards, failure. And Muni will not run well until those who make it run badly begin to be held accountable.

For the San Francisco Municipal Railway to start rewarding success rather than failure, choices would have to be made. Union agreements would need to be changed in ways almost certain to anger union members and their leaders. Top Muni managers -- even and especially those who are members-in-good-standing of San Francisco's Democratic machine -- would have to be judged on performance, rather than politics, and be fired when necessary. Consultants who play major roles in creating boondoggles would have to be let go (rather than recommended for more work). The bad news is that these changes would be politically difficult to make in a city run by a Democratic establishment that has no real political competition and walks arm in arm with admittedly self-interested labor unions.

The good news? The changes wouldn't cost taxpayers a dime.

About The Author

Peter Byrne


Subscribe to this thread:

Add a comment

Popular Stories

  1. Most Popular Stories
  2. Stories You Missed


  • clipping at Brava Theater Sept. 11
    Sub Pop recording artists 'clipping.' brought their brand of noise-driven experimental hip hop to the closing night of 2016's San Francisco Electronic Music Fest this past Sunday. The packed Brava Theater hosted an initially seated crowd that ended the night jumping and dancing against the front of the stage. The trio performed a set focused on their recently released Sci-Fi Horror concept album, 'Splendor & Misery', then delved into their dancier and more aggressive back catalogue, and recent single 'Wriggle'. Opening performances included local experimental electronic duo 'Tujurikkuja' and computer music artist 'Madalyn Merkey.'"