Industrial Case Study: Moving to Commercial Off-the-Shelf and Open-Source Software Usage in Telecommunications

  • 12/22/2011
In this software change management case study, you are the lead software engineer in the Engineering division responsible for developing a new switching system. This chapter from Software Change Management: Case Studies and Practical Advice offers insights into how to spot, quantify, and deal with controversial issues related to off-the-shelf and open-source software usage in telecommunications.

Setting the stage

This next case study occurs in a large telecommunications firm. The firm wants to move from a custom architecture to an open architecture for its Switching Systems division’s product offerings. This division has resisted past attempts to make a move to a new platform and architecture because it had millions of dollars invested in specialized software, which its sales and management leadership viewed as a discriminator in the marketplace. Because of new sales opportunities, the firm initiated the development of a new switch that embraces many innovative concepts. By bringing this switch to market, the firm hopes to retain its market share and position in the future. Everyone in the firm’s organizational chain, depicted in Figure 4-1, agrees that it is time to make the changeover and do it right.

The key change being proposed is a move to a new architecture and an open system platform. As shown in Figure 4-2, the applications software will run on top of a POSIX platform riding on top of multiple processors, which execute in parallel to provide growth paths in case more lines need to be added by telephone operating companies (the customers). POSIX will be configured to run using existing facilities to provide platform-designated services on an on-demand basis. Such services include, but are not limited to, configuration and initialization, relational database management, dispatching, distribution, querying, scheduling, and security. Services will run to completion to avoid interrupts that could cause execution to stall, stop, or be rescheduled.

Figure 4-1

Figure 4-1 Telecommunications firm organizational structure.

Figure 4-2

Figure 4-2 Top-level switching system architecture.

In addition to using normal switching applications, the new system will provide users with a novel, knowledge-based, self-regulating load-balancing dispatcher and an innovative self-diagnosis and repair system that will act as the marketplace discriminator for sales. The front end of the switch will provide a wide range of network-based and Internet-accessible communications capabilities. It will provide users with easy access to features and query-on-example ability. Context-sensitive help will be provided along with many improved user-interface features to make the system intuitive, easy to understand, and fun to use.

The two key innovative technical enablers that make it feasible to build such a system now are the following: new dispatching algorithms that the Research and Development (R&D) Laboratories invented that facilitate the optimum scheduling of application threads running on parallel processors, and new middleware that allows the system to bind components together using rule-based, load-balancing techniques. Components that are scheduled are fragments of applications packaged by the middleware to execute in parallel on different processors (parallel threads) and share results (self-combinations). Application fragments can include commercial off-the-shelf (COTS) components, open-source or custom routines, modules, or programs, as long as each scheduled and combined entity adheres to the packaging rules, runs under POSIX, and uses the system’s data model.

Organization

For this case study, assume that you are the lead software engineer in the Engineering division responsible for developing the new switching system. The initial target of opportunity for sales of the system is a telephone operating company that is your largest customer. This company worked with your people on the architectural specification for the system and helped generate the related functional and performance requirements for it. It wants to buy 100 of these switches, assuming that your organization can deliver them within three years. It is ready to help during the development in any manner possible. The company suggests that it perform the independent verification task, where it provides feedback during the development on the products as they incrementally roll off the drawing board.

Most of the work that the Switching Systems division currently performs is aimed at supporting systems in the field. Major developments like the new system come around once a decade. As such, this represents the means to update the organization’s processes, practices, methods, tools, skills, and experience. Management, however, recognizes that by doing too much too quickly your company could fail. In response, they want to attack the development conservatively and use only proven technology. They form a task team to devise a project plan, and you are asked to be a member. You are thrilled and ready to start contributing to the effort.

Project

The project being planned involves the design, development, and validation of a test article that will be used as the model for product development. The development is targeted for three years. Getting manufacturing facilities ready for production will take about year. However, this can be accomplished easily in parallel with the product development because the production facilities are ready for use. There are no budget details yet. Because the future of the division revolves around the success of this project, you believe management will allocate whatever resources are necessary to pull it off. Even though their funds seem limitless, management wants you to justify every penny.

The planning team is made up of the following six people: you, the team lead, a financial person, the chief engineer, a process person, and a customer representative. Besides several other responsibilities, you have been asked to handle all planning activities associated with COTS and open-source software. Everyone on the team is excited and wants to do a good job.

Your team assessed the current situation and found that both systems requirements and architecture specifications for the new system have been completed by the startup team. These specifications were reviewed as a first order of business and judged to be well done. The team also found that a feasibility study was completed that identified 26 candidate COTS and open-source application software packages for potential use on the project.1 Several of these candidates have been analyzed on a try-before-you-buy basis, and the results were documented. During development, you know you will still have to select packages, negotiate licenses, and integrate these packages as part of the switching system. You also know that your work with COTS and open-source software will not stop here. There will be annual updates and licensing costs to worry about after the system is operational. Licensing concerns you because there might be run-time license costs associated with some of the packages that have not been accounted for.

Based on this completed work, the team feels much better because their planning efforts would not have to start at square one. In addition, the startup team has completed a high-level budget of $770 million over the three-year development schedule and determined details of the first year’s operation in the field, which appear in Table 4-1. As part of your tasking, you are asked to review COTS and open-source forecasts to determine whether or not they are realistic for the job at hand.

Table 4-1 Top-level budget for new telecommunications system development and maintenance.

Task

Subtask

Forecasted Budget by Year (in millions)

Year 1

Year 2

Year 3

Year 4

Systems engineering

System engineering plan and trade studies

$10

$10

$—

$—

Integration product team operations

5

10

5

Project management

Project management

20

20

20

Measurement and analysis

2

2

2

Product support

Configuration management

4

4

4

Quality assurance

4

4

4

Supplier management and licensing

2

2

2

Security and network protection

3

3

3

Hardware engineering

Hardware acquisition and readiness

10

10

10

Interface development and test (both hardware and software)

10

10

10

Software engineering

Requirements analysis

$10

$5

$—

$—

Software development

25

50

50

COTS and open-source package acquisition and readiness

5

10

10

Software integration and test

10

10

10

Licenses

2

2

2

System integration and test

Test planning and readiness

10

Hardware and software integration and testing

25

25

System test and evaluation

10

Manufacturing

Specification

8

3

3

Test article fabrication, assembly, and production

20

25

25

Production article fabrication, assembly, and production

25

25

Systems test

Test article testing

10

10

Acceptance test and evaluation

10

Deployment

Staging and delivery

25

Dual operations and cutover

25

Operations and maintenance

Planned product improvements (both hardware and software updates and optimizations)

75

Licenses

5

TOTALS

$160

$240

$290

$80

Process

Your next step in the planning process is to determine what work needs to be accomplished to get the product out, determine who will do it, when it will be done, and at what cost. The team lead suggests that the team use a divide-and-conquer strategy to develop the work plan, where each member of the team develops a task list for different parts of the effort. Of course, you are given the COTS and open-source package work as part of your assignment. Your job is to determine whether the line item totals under “Software engineering” titled “COTS and open-source package acquisition and readiness” and “Licenses” are adequate to cover the work required to be completed in these areas.

You first identify the tasks required to employ COTS and open-source packages2 in the development. Completion of these tasks assumes that the middleware performs as specified and that each package can be cleanly integrated into the system without any rework other than tailoring. The process model used to describe the activities being performed to put COTS and open-source software to work throughout the life cycle is shown in Figure 4-3.

Figure 4-3

Figure 4-3 COTS and open-source package software process model.

For those unfamiliar with terms used in the process model in Figure 4-3, a brief explanation may be in order. After you select a package, you typically configure and tailor it to satisfy your requirements using built-in features. Then you test and accept the package prior to taking delivery. Once you’ve accepted it, you bind the COTS package via your middleware to your system, integrate it, and make it work with the system at large. Next the vendor provides periodic updates that you evaluate and incorporate into your system, if appropriate. You continue in this mode until you decide it is time to either replace the package with an alternative or decommission it.

The wildcard in the case is the unique data model your people have devised. You must determine whether or not the package can be tailored to accommodate it prior to making your purchase decision. Luckily, this was one of the tasks the startup team performed during the trial licensing period. They assessed the candidate packages to make sure that they worked as advertised and were compatible with the architecture’s data-model specifications. You feel relieved when you discover these facts in the notes they provided to you.

Product

You complete your analysis based on the process model shown in Figure 4-3. Based on the selection of the 12 packages listed in Table 4-2, you clearly show that there is more work that needs to be performed than the budget allocated for the packages being considered. The major reason for the cost differential seems to be the run-time licenses for packages for system software (such as the database manager) and tools (such as compilers and debuggers) that operating companies want delivered in case they have to make patches in the field. In addition, as shown in the itemized list in Table 4-2, the budget for COTS and open-source software package licenses is low during software maintenance because many of the development tools and other support licenses were not included in the estimate. The people developing these forecasts overlooked these costs because the costs were not in their frame of reference (software development vs. maintenance).

Table 4-2 Itemized costs for COTS and open-source packages.

No.

Package Type

Development License Cost

Maintenance Cost (plus run-time licenses)

1

System software (POSIX toolset)

$250,000

$800,000

2

Database management toolset

$250,000

$2,200,000

3

Requirements management and traceability toolset

$100,000

$100,000

4

Software design toolset

$50,000

$50,000

5

Software language toolset (compiler, and so forth)

$500,000

$1,500,000

6

Software test toolset (coverage, and so forth)

$250,000

$250,000

7

Hardware CAD tools

$500,000

$500,000

8

Software configuration management tools

$250,000

$350,000

9

Test management toolset

$250,000

$250,000

10

Computer performance evaluation toolset

$250,000

$250,000

11

License management toolset

$50,000

$50,000

12

Documentation toolset

$200,000

$200,000

TOTAL

$2,900,000

$6,500,000

People

You deliver your findings, and they get incorporated into the project plan. After conducting a peer review, the team lead asks you to prepare some backup materials on COTS and open-source packages because most of the team believes that upper management is clueless when it comes to the issues associated with their use. In response, you develop Table 4-3 to summarize the reasons your firm should buy rather than develop custom software packages, and Table 4-4 to highlight the typical risks associated with the purchase option and related strategies that have been used to successfully mitigate them.3 These backup charts are received well by the team as the effort to get the plan out concludes.

Table 4-3 Developing software in-house vs. licensing it.

Develop Software In-House

License Software

You pay the total development and maintenance cost.

You pay only a fraction of the development and maintenance costs.

Custom software takes years to develop.

Software is available immediately.

The product is mature and relatively bug-free.

It takes considerable time to mature the product.

It’s developed to satisfy your customer’s requirements.

It’s primarily developed to satisfy marketplace requirements.

It’s easy to change because you are in charge of the migration path.

It’s harder to change because market forces drive the migration path.

It’s hard to determine whether or not the software, when built, will serve the customer’s needs.

It’s easy to assess what capabilities exist and prototype with existing packages.

Your team must provide user support, training, and documentation for the software.

The package comes with support, training, and user documentation.

Table 4-4 Top 10 risks and related mitigation actions taken with licensing software.

No.

Risk

Mitigation Action

1

Hidden license costs.

Understand licenses, and negotiate favorable terms prior to signing the licensing agreement.

2

Package capabilities are not as advertised.

Assess package capabilities via a trial license prior to licensing.

3

Architectural feature mismatches might be present (different data models).

Make sure that you fully assess the package candidates before signing a license agreement.

4

The software architecture puts a premium on performance rather than adaptability.

Modularize the architecture around foreseeable sources of change and the use of COTS.

5

No control over the migration path.

Establish a relationship with the vendor to influence the migration path.

6

Poor customer service.

Pay for on-site support or premium service contract.

7

Most times, all that you get is the executable, not the source code.

If it’s really critical, negotiate for a source-code license.

8

The vendor might go out of business.

Establish a market watch to identify replacements, and get the source code put in escrow in case of vendor default.

9

Software upgrades are not in synch with your update cycles.

Architect products using COTS and open-source packages to accommodate such updates.

10

Better alternatives appear on the market, but you are locked into this vendor.

Maintain flexibility in licensing, and keep a market watch.