We here at Fox Group Consulting have learned some valuable lessons about project planning, supplier management and proper equipment installation recently; all as a result of our recent move of our telecommunications technology to our new Mount Albert facilities.
We’d probably be a lot less bitter about our education if the troubles we encountered hadn’t hit so close to home; and especially affected our short-term ability to serve and support our various enterprise and industry customers. Where does the blame reside – sadly, on both sides of the desk.
So, what went wrong? It should have been straightforward…… We merely wanted to move our voice and data infrastructure that we had been very happy with from our previous office in Markham, Ont., with its existing hardware, software and applications, to our temporary location.
For over one and one-half years we had been successfully using an IP-PBX with application servers and numerous telecom and contact centre applications at our Markham-based location.
Our long term plan for Fox Hollow was to create a true work-life separation from our various business activities. We also had planned for a second supplier’s system to be installed for our residence building. Our plan calls for us to integrate the two separate systems in late summer to test the multi-vendor compatibility for next generation SMB environments as part of growing the SMB ‘living test lab’ facility that we operate – something that we still intend to do.
Given our client commitments and time constraints at the time, we called on our original manufacturer to move the IP-PBX and the various application servers and install them at our temporary location in the heritage farmhouse while the new office building was being finished.
While the vendor offers full project management services that generally include an initial requirements kickoff meeting, then the creation of a formal statement of work, system and network design and dedicated project management including client acceptance criteria.
We all thought it would be only necessary to purchase technician-level services on a time and materials basis to assist with the move to Mount Albert from Markham , particularly since this had been a functioning stable system prior to the move.
During the initial supplier discussions, it was also suggested that we could use the planned downtime as an opportunity to upgrade the IP-PBX to the most recent software release. They also suggested that we should perform the upgrade to ensure that we had the latest and greatest applications, call processing and call centre capabilities and also added necessary redundancy.
After discussions with our own IT and telecom consultants, we decided to accept their recommendations. Very soon after the IP-PBX was installed at our new location in mid December, we started to encounter some service and reliability issues.
First Issue – Incomplete Installation
With construction still in progress on our new office facility, the technician sent to program our phone extension numbers started the job with the system in the home residence.
Without a detailed statement of work (other than ‘move the system’), we had to call the technician back numerous times to finish configuring the system for various sets, voice mail boxes and system programming functions that we had prior to the move.
Without proper project management and vendor co-ordination defined and agreed, the multiple technicians sent to our location started net new each time. They were also not aware of the expectations, requirements and configurations required which meant we had to repeatedly explain our requirements each time to each new person as they arrived on site.
Second Issue – Dialing Problems
Due to the complex GTA 416/905 calling plans, the previous IP-PBX settings couldn’t match which locations that use the “905” area code were local to Mount Albert and which were long-distance. As a result, the system could not connect us with the people we wanted to reach In site of our distributed, virtual network set up.
As well, we not only have conventional Local Link business grade phone lines connecting us to the world and IP phone sets on our desks, we also have Bell Canada’s new Digital Voice service that we were in the process of testing.
This is a new VoIP hosted service that allows our customers and contacts in the “416” area – Toronto to call locally to us through the IP-PBX to our IP sets operating on our in-house LAN to access us while coming through our high speed Internet without incurring long-distance charges. (Try to picture this diagram….we have it if anyone is interested!).
We also attached the Bell Digital Voice service to the system and programmed the automatic route selection to help keep our communication links integrated for both our customers and our consultants.
But the voice system got all mixed up: is this number the user’s trying to reach local to Toronto or local to Mount Albert? Is it local to both? Or neither?
This problem, alongside the various other configuration troubles suggested that perhaps we should have started the move with a more simple network configuration, and also not include any upgrades to hardware or software, at least until we had validated the performance of the basic infrastructure’s underpinnings.
As we worked through troubleshooting the various parts of the voice, server and applications required, we decided to take the Bell Digital Voice circuit out of the equation. We also simplified our multi-level auto attendant IVR tree, hoping that this would solve the various dialing problems. It didn’t.
In fact, we’re not sure exactly what went wrong, and neither is our supplier, although they are working with us to diagnose and analyze the issues.
Third Issue – Phone system losing digits when dialing internal extensions
The VoIP system started to lose digits, maybe as a result of whatever caused the second issue, particularly when it came to long-distance calls. If one of our telecom consultants here at the office dialed “9,” then “1,” the area code and finally the phone number he was trying to reach, he’d receive a message saying he should have dialed “1” before the number. It became an issue for outside callers into the office too. On inbound calls, the system only heard the “0” of our four-digit extensions, causing callers to “zero out” and reach the operator – when they didn’t mean to.
Our supplier tracked the lost-digit issue down to the digital signal processor (DSP) and initial auto route selection (ARS) issues. The vendor sent a technician to install the new DSP. Unfortunately the new DSP proved to be just as addled as the first one, and the system continued to have technical difficulties with a number of total system failures being experienced that required a re-boot of the main process. The supplier continued to work with us to try to isolate and fix these problems.
Meanwhile, we had to move on with trying to run our business, hence the need to quickly install a replacement solution. Since the integration of a separate residential installation was planned for Phase 2, and to add as part of our next ‘living test lab’ test program, we decided to move up the installation date of the second system, and run our home and office on this system until we were able to resolve the outstanding issues on the original platform.
Next week’a articles takes a hard look at reflecting on the problems, the lessons learnt and possible causes.
As always Roberta Fox welcomes your thoughts and feedback on this and other emerging technology topics. Contact [email protected].
