[Image] Case Study: Migration Migraines Preface Microsoft is making a concerted effort to have its Windows NT product line — both the Server and Workstation versions — adopted by everyone as the universal panacea for all that ails the enterprise. To this end, it has put enormous resources behind the development, marketing, sales and support of the product to prove it is a worthwhile successor to Unix servers, mini-computers and mainframe systems, and has been selling it as a loss-leader to gain market share. Windows NT has been around long enough for some Fortune 100 companies which decided to implement NT as the corporate standard to be in production mode. The following is a case study of one such company, documented over several months through interviews with those responsible for the implementation. This is the first of several such case studies to be published which are based on Aberdeen’s on-going research with executives of North American and European businesses and organizations. As a matter of full disclosure, the sources within the Company which is reported on here initially approached Aberdeen seeking assistance with their migration problems. Aberdeen itself received help in understanding the Company’s issues from several of Microsoft’s competitors. Nonetheless, this case study is instructive since all the facts described here were presented by the company’s employees and not by the competitive suppliers. Background Since its inception in the early 1970s, the Company has been growing at a very fast rate, and has effectively used technology to help foster this growth. The Company’s administrative side has over 35,000 desktop systems, most of which use a version of a Microsoft Windows operating system to run Office applications. There are an additional 5,000 OS/2-based desktops used by developers. The focus of this case study is the IS group responsible for different aspects of the Company’s extensive call center operations. The call centers — both in-bound customer service and out-bound telemarketing — are located throughout the country. All are interconnected with state-of-the-art networking technology. The call center applications were first developed in the early 1980s on a then popular mini-computer. In 1991, the senior IS staff made a decision to migrate both the call center development and operations to IBM OS/2 servers and workstations. Eventually, about 12,000 clients were installed, using several OS/2 versions as the operating system. For the servers, OS/2 Warp 3 was the predominant choice for the call center operations and OS/2 Warp 4 was being used by the development group. OS/2 Connect seems to be the favorite workstation version at this time in the call center, and NT or OS/2 Warp 3 in the development group. The various platforms and applications were very stable, and few problems with performance or management were reported. Microsoft’s Entry In 1995 the Company entered into a business relationship with Microsoft, and as part of the agreement Line of Business (LOB) executives — not IS — decreed that the Company would become a total Microsoft shop. In late 1995, the IS groups responsible for the call centers were told to freeze all OS/2-based application development work, and any OS/2 upgrades and enhancements. The decision was to go with Windows 95 for all mobile users, Windows NT Workstation for the desktops used by both developers and call center operators, and Windows NT Server for all mid-tier networked servers. The Company still makes extensive use of mainframes at its enterprise top-tier. As part of a separate decision, the call center application code was to be moved from the client to the server because there were too many problems managing the applications on thousands of clients. As of this writing, the transition of the application code has taken place with few problems, and all call center code is now maintained on the server. Migration Plans The Company’s call center development group began the transition to NT Workstation in the Spring of 1996, and to NT Server in early 1997, with the full migration to be completed by mid-1997. The project missed its deadline. Although some 70% of the migration is completed, the last 30% is now considered to be so difficult that the IS management is no longer willing to predict when the project will be complete. The call center operations group started the transition at the same time, with the additional implementation of NT SQL Server as well as NT Workstation and NT Server throughout the operation. While other companies have implemented NT Server at the workgroup and departmental level as application or database servers, few have implemented NT as the network operating system of choice across multiple domains. However, NT was being introduced at the Company as the sole operating system for the desktop, LAN server, application server, database server and communications server. Truly, Microsoft’s dream of running everything with a total Microsoft solution. Migration Results The impact of the migration completed to date on the call center’s IS budget and management has been profound — a one-third decrease in performance accompanied by a fifty-percent increase in management costs. This does not include production time lost because of workstation freezes, transmission drops, or the constant re-booting needed to keep the server operational. From a very stable environment, the IS staff now report chaotic conditions which have only abated modestly over the months since the migration started. Production Problems No matter how fast the Aberdeen OnSite Analyst wrote, the litany of problems as described by the Company’s IS staff were too numerous to detail. However, describing a few low-lights gives the essence of the myriad problems encountered: * The most stable of the NT Workstations now require "only" two or three re-boots per day. Irrespective of the problems encountered, whenever a workstation freezes, the Microsoft technical support group recommends a re-boot. The call center operators call this "Microsoft’s [expletive deleted] three-fingered salute — control/alt/delete." * Since transmission of transactions is of paramount importance to the Company, the intermittent slowdowns, random packet drops and inexplicable time-outs of routers, increased congestion due to garbage packets, etc., is of serious concern. To date, neither the Company’s nor Microsoft’s technical experts can identify — let alone correct — the problems. * The actual number of end-users is declining, but the number of "domain engines" (NT Servers, concentrators, routers, etc.) has increased due to NT Server’s inability to service more than modest numbers of end-users. * Trouble tickets are up dramatically and new staff have had to be hired to respond, but facts are scarce since the help desk has stopped issuing its monthly customer service level reports because of "embarrassment." * The problems with the SQL Servers were so numerous that they filled four pages of edited notes. A few are summarized in Table 1. IS’ inability to recommend corrective measures and make changes has caused enormous frustration, and has led to an exodus of skilled engineers. Given the tight market for skilled technical staff, it has been necessary to replace lost personnel with junior level engineers — furthering the NT migration problems. This gradual degradation of skill levels was described as a potential "death-spiral." Table 1: SQL Server Problems Documented by the Call Center IS Staff Issues with database carashes, resulting in database rebuilds requiring hours or days to recover. Database corruption is much more frequent than desired or expected, and recovery is time consuming and results in downtime and loss of productivity. Responsiveness to known product bugs/limitations has been poor. Issues which have been known for some time still do not have a fix released over the past three micro-revisions. Much of the product is viewed as inadequate or immature — including cache management, referential integrity, replication, messaging, log file management, native SQL code debugging tools, error messages, and proactive notification of critical system problems. Technical support often recommends rebooting either SQL Server or NT to resolve issues. Concern over the knowledge, responsiveness and effectiveness of Microsoft technical support. Advance communications from technical support on issues such as known problems are missing. Concerns expressed by technical staff concerning the scalability and performance of the product, especially when compared with other DBMS products on the market. Concern with single platform support (the Company is a multi-platform DCE). Concern with Year 2000 compliance. Source: The Company, August 1997 Aberdeen Conclusions The concept of a smooth migration of the scope and size instituted by the Company is illusionary. Problems always emerge, schedules are delayed, and staff comfortable doing things "their way" rebel at the change. Thus, IS’ first complaints about the move to Windows NT were taken with a large grain of salt. But, at considerable risk to their jobs, the IS staff Aberdeen has been talking with have fully documented their woes, and by any objective measure the migration pain they are suffering goes way beyond the pale. While this report captures the "plaintive wail" of frustration emitted by the IS staff who have been planning and implementing this migration over the past two years, it doesn’t capture the level of pain they feel. Based on hundreds of in-person interviews with senior executives in major businesses and organizations, Aberdeen has come to understand how effective use of technology can vastly improve operations. One critical issue often ignored is the need for a delicate balance between the roles of the business executives and IS executives in the development of an IT infrastructure for the enterprise. One simplistic finding is that technology implementations go well when the business managers determine what is needed to improve operations, while the IS managers determine how IT gets implemented to support those business needs. For the Company, the senior business executives decreed both the what and the how. They made a business decision to go with Microsoft irrespective of its various products’ strengths and weaknesses, assuming all were "good enough" to meet the business’ needs. The decree, in effect, tied IS’ hands and prevented it from finding the best solutions available — and now IS finds it politically incorrect to make their bosses aware of the massive problems the decree has caused. It is quite possible that this 1995 purely business decision as currently being implemented will have a material impact on the Company’s operations and bottom line in 1998. For a Company that has been studied by scores of business school students as a successful example of David beating Goliath with advanced technology, the breaking of the decision-making balance between its technical and business groups could lead to a sadder chapter. [Image] [Image][Image][Image][Image] [Image][Image][Image][Image] [Image] AberdeenGroup, Inc. One Boston Place Contact Information: Boston, For further information on AberdeenGroup's Massachusetts products and services please contact us at 02108 USA info@aberdeen.com Tel: 617.723.7890 Fax: 617.723.7897 [Image] This Document is for Electronic Distribution Only -- REPRODUCTION PROHIBITED -- Copyright 1997 Aberdeen Group, Inc., Boston, Massachusetts The trademarks and registered trademarks of the corporations mentioned in this publication are the property of their respective holders. Unless otherwise noted, the entire contents of this publication are copyrighted by Aberdeen Group, Inc. and may not be reproduced, stored in a retrieval system, or transmitted in any form or by any means without prior written consent of the publisher.