The good, bad and ugly of a VoIP implementation

Facing frequent outages in the DSL lines serving 50-plus corporate-owned Gold's gyms, Bobby Badugu knew it was time for a significant network upgrade. After conducting due diligence in April into the various options -- including satellite, frame relay and a different DSL provider -- he opted for a carrier-provided voice-over-IP (VoIP) service. That's when the trouble began.

In the months that followed, the vice president of IT for Gold's Gym, based in Dallas, learned some valuable lessons about not only VoIP, but any major network project that involves upgrades to numerous, far-flung sites. Badugu shared his insights in a presentation at the recent Network World IT Roadmap Conference in Dallas. He said the lessons include using a phased implementation process; establishing clear service-level agreements (SLA), including penalties; and conducting a thorough technology assessment to identify potential problems.

When the rollout began in June, it was scheduled to take eight weeks. As of mid-November, the data portion of the rollout is complete, but only about 20% of the voice lines have been ported to the VoIP network. Badugu remains bullish on VoIP, expects benefits including a 35% to 40% savings in voice and data service costs, and already is saving $6,000 to $10,000 on his conference calling. But if he had it to do over, he'd do more than a few things differently.

Requirements

Gold's Gym started in 1965 with a single facility in Venice, Calif. Through a series of licensing deals, mergers and acquisitions, the company grew to 660 gyms in 26 countries. Of the 550 U.S. Gold's gyms, most are franchises; about 50 are corporate-owned. Badugu is responsible for providing network services to those corporate-owned gyms, which are in California, Hawaii, Maryland, Missouri, Texas and the Washington, D.C., area.

Network services are particularly crucial to the gyms because they rely on a central server for everything from member data to the retail point-of-sale system.

Badugu targeted satellite when he embarked on the upgrade project in February 2006, but he quickly learned that it wouldn't provide enough bandwidth. He also wasn't impressed with the available frame relay offerings, which he considered older and less flexible, and didn't feel comfortable with a DSL service from a different provider.

Then Badugu recalled a Qwest MPLS project from his consulting days, so he called in Qwest to inquire about VPN service.

"That's where we started," he said. "From there the proposal evolved to address a lot of our phone needs. That's when the VoIP solution was proposed."

Once voice was on the table, Badugu brought AT&T and Sprint into the discussion to see what they had to offer. But their proposals were too voice-centric and didn't have a strong strategy for combining voice and data, so Qwest won the day.

The big driver for upgrading the voice network was reducing costs for local, long-distance and conference calling. "As we add gyms and continue to grow our business, those costs became astronomical," Badugu said.

At the same time, with its lean IT staff, the company struggled to manage the lines and ensure each was properly used. "We didn't have the staff to monitor it every month and make sure we're doing the right thing for the company," he said. He also didn't have the staff or the desire to build his own VoIP network.

The plan was to install a T1 line to each gym and implement voice and data service when the circuits were installed. Badugu figured scheduling the T1s from various local exchange carriers (LEC) would be the most complicated issue.

"That was pretty simple. They gave us a date, and when it occurred, they were out there and put the T1 in place," he said. But from there, Qwest subcontractors complicated matters.

At each gym, one contractor would order, configure and install a router. Another configured and installed the firewall and hooked up the VPN tunnel. "As we did each location, we found more and more complexities in that process," Badugu said. "We had six or seven parties involved in putting the phone and data lines in. I really felt like some were pretty much learning the technology or learning the equipment."

Badugu also admitted to shortcomings on his company's part. "One of the things we didn't do correctly was we didn't know our environment," he said. That included the types of phone systems installed in each location and whether they could connect to the T1 line. Even though he had been installing digital, IP-ready phone systems in all gyms, some didn't have the required T1 card, which meant an additional expense, and others were still analog. Locating all the wiring closets was another challenge.

For these reasons, Badugu decided to focus on bringing up the data side first and then add voice later. As a result, the original eight-week implementation window passed and the company didn't have all its T1s installed.

Voice challenges

The decision to delay the voice implementation proved prescient, as that side of the equation presented numerous challenges. Chief among them was porting existing phone numbers to the Qwest VoIP service.

Gold's Gym gave Qwest all its working telephone numbers and billing numbers to begin the porting process from various incumbent LECs (ILEC). "Of the first five numbers we submitted, four failed," he said, because the billing telephone numbers that Gold's Gym had were different from those the ILEC had.

"The whole porting process became a real nightmare," Badugu said. In some instances, he tried to forward calls from the old phone system to the new one. But that often was confusing for callers who heard the phone ring once and then heard a short delay before it rang again. If two calls came in at the same time, the second would reach a busy signal.

Now he is porting the main number for each gym, then assigning new numbers for each extension, which typically aren't called directly. Badugu has a schedule from his ILECs for when each number is supposed to be ported, and then he coordinates with phone technicians to ensure the new system is up and running properly. He is hoping to have all the numbers ported by year-end.

Despite the challenging implementation, Badugu began realizing a savings of 20% on long distance once he got the T1s installed and switched all long-distance service to Qwest. Additionally, conference calls now are carried over the Qwest network, obviating the need for a third-party provider and saving at least $6,000 per month. As each gym is brought onto the VoIP network, he gets closer to his projected savings of 25% to 30% on local calling costs.

If he had to do it over, however, he would take a more phased approach. "We tried the big-bang theory, doing everything at once," he said. That decision was driven by business needs, because the DSL network was so unreliable and because Qwest was urging Gold's to sign a contract that included every gym. "Even still, I think we should've done two or three gyms first and made sure that was OK before we did the rest."

He also advised mapping out details about the steps required to install any new technology and who is responsible for each step, along with contacts and escalation procedures, should things go awry: "So if something fails, you have somewhere to go to keep the process moving."

Another must is technology assessment to identify items like phone systems that were analog or lacked T1 cards. "A site survey would've been good, with a detailed analysis of the phone systems, the phone numbers, how the hunt groups work, how the phone systems are configured, the number of workstations," Badugu said. "For each location, if we had that upfront, it would've made life easier for everybody involved."

Desmond is events editor for Network World and president of PDEdit, an IT publishing company in Southborough, Mass. He can be reached at paul@pdedit.com.

This story, "The good, bad and ugly of a VoIP implementation" was originally published by Network World.

Related:

Copyright © 2006 IDG Communications, Inc.

  
Shop Tech Products at Amazon