CSQA Prep - Chapter 3 - Quality Baselines

(3-13)The Software Engineering Institute’s Capability Maturity Model Integration (CMMI®1) is an
example of a staged model, although it now has a continuous representation.

An assessment is a process used to measure the current level of performance or service. The
objective is to establish a baseline of performance. You can measure against an internationally
accepted model or an organization’s own defined criteria.

(3-12)A model is an idealized concept to be accomplished. Models are usually developed under the
auspice of national or international standards organizations and may be customized or ‘tailored’ to
meet new or changing needs.

The CMMI® influences an organization’s culture by instilling discipline and continuous process
improvement into the workplace. It is a framework that enables an organization to “build the right
products right.” In the staged representation, maturity levels organize the process areas and provide
a recommended order for approaching process improvement in stages.

A maturity level is a well-defined evolutionary plateau for achieving a mature software process.
Each level contains a set of goals that, when satisfied, stabilizes specific aspects of the software
development process. Achieving each level of the maturity model institutionalizes a different
component, resulting in an overall increase in the process capability of the organization.

The U.S. Department of Commerce is responsible for the MBNQA and Program.

The National Institute of Standards (NIST), an agency of the Department’s Technology Administration manages the Baldrige Program.

(3-23) The scoring responses to Award Criteria and Award applicant feedback are based on two
evaluation dimensions: Process and Results. Criteria users need to furnish information relating to
these dimensions.
Process
Process refers to the methods your organization uses and improves to address the Item
requirements in Categories 1-6. The four factors used to evaluate process are Approach,
Deployment, Learning, and Integration (A-D-L-I).
“Approach” refers to:
• The methods used to accomplish the process
• The appropriateness of the methods to the Item requirements
• The effectiveness of your use of the methods
• The degree to which the approach is repeatable and based on reliable data and information
(i.e., systematic)
“Deployment” refers to the extent to which:
• Your approach is applied in addressing Item requirements relevant and important to
your organization
• Your approach is applied consistently
• Your approach is used by all appropriate work units
“Learning” refers to:
• Refining your approach through cycles of evaluation and improvement
• Encouraging breakthrough change to your approach through innovation
• Sharing of refinements and innovation with other relevant work units and processes
in your organization
“Integration” refers to the extent to which:
• Your approach is aligned with your organizational needs identified in other Criteria
Item requirements
• Your measures, information, and improvement systems are complementary across
processes and work units
• Your plans, processes, results, analyses, learning, and actions are harmonized across
processes and work units to support organization-wide goals
“Results” refers to your organization’s outputs and outcomes in achieving the requirements in
Category 7. The four factors used to evaluate results are:
• Your current level of performance
Quality Baselines
Version 6.2 3-25
• Rate (i.e., slope of trend data) and breadth (i.e., how widely deployed and shared) of
your performance improvements
• Your performance relative to appropriate comparisons and/or benchmarks
• Linkage of your results measures (often through segmentation) to important customer,
product and service, market, process, and action plan performance requirements
identified in your Organizational Profile and in Process Items.

(3-27) The following major clauses comprise the ISO 9001 standard. Sub-clauses are also noted in the
description.
Management Responsibility
Management Responsibility includes these sub-clauses: Management Commitment, Customer
Focus, Quality Policy, Planning, Administration, and Management Review. Customer Focus is an
integral process within the quality management system.
Top management shall demonstrate that customer needs and expectations have been determined
and translated into applicable customer requirements. Top management shall also demonstrate its
commitment to meeting customer requirements for their product or service by:
• Creating an environment for awareness and fulfillment of customer requirements
• Establishing the quality policy and quality objectives
• Establishing a quality management system
• Performing management reviews
• Ensuring the availability of necessary resources
Resource Management
Resource Management includes these sub-clauses: Provision of Resources, Human Resources,
Facilities, and Work Environment.
The organization shall determine and provide the resources needed to establish and improve the
quality management system. This includes, but is not limited to:
• Human resources, such as assignment of personnel, training, qualification, and
competence
• Other resources, such as information, infrastructure, and work environment
Product Realization
Product Realization includes these sub-clauses: Planning of Realization Process, Customer-related
Processes, Design, Development, Purchasing, Production & Service Operations, and Control of
Measuring & Monitoring Devices. It is this Clause into which the chosen domain model should be
inserted.
The organization shall determine which processes are required to satisfy customer requirements.
The sequence and interaction of these processes shall be determined, planned and controlled to
ensure they operate efficiently.
The organization shall ensure that these processes are operated under controlled conditions and
produce outputs that are consistent with the organization’s policy and objectives. For all of these
processes, the organization shall:
• Determine how each process influences the ability to meet product or service
requirements
• Establish methods and practices relevant to process activities, to the extent necessary
to achieve consistent operation of the process
• Verify processes can be operated to achieve product or service conformity
• Determine and implement the criteria and methods to control processes related to
the achievement of product or service conformity
• Determine and implement arrangements for measurement, monitoring, and followup
actions to ensure processes operate effectively
• Ensure the availability of process documentation that provides operating criteria
and information to support the operation and monitoring of processes
• Provide the necessary resources for the effective operation of the processes
Quality Baselines
Version 6.2 3-29
With regards to the customer-related processes, the organization shall establish a process for
identifying customer requirements. The process shall consider:
• Identification of customer requirements
• Review of customer requirements
• Review of the ability to meet defined requirements
• Customer communication
• Customer property (physical property, intellectual property, etc.)
Measurement, Analysis and Improvement
Measurement, Analysis and Improvement includes these sub-clauses: Planning, Measuring &
Monitoring, Control of Nonconformity, Analysis of Data, and Improvement.
The organization shall define and implement measurement, analysis, and improvement processes
as a way of demonstrating that the product or service conforms to specified requirements. This
includes:
• Measurement of system performance
• Measurement of customer satisfaction
• Internal auditing
• Measurement of processes and product conformity, internal improvements
• Measurement of product and/or service
• Control of measuring, inspection and test equipment
• Analysis of data to view operation trends, and to evaluate the effectiveness of the
quality management system, customer satisfaction, and conformance to customer
requirements
• Improvement by implementing corrective and preventive action

(3-29) ISO/IEC 12207, which was published in 1995, is the international standard that covers the software
life cycle from concept through retirement. It contains a framework for managing, controlling, and
improving the software life cycle activities. The standard describes the major processes of the
software life cycle, how these processes interface with each other, and the high- level relations that
govern their interactions. It has subsequently been amended twice; Amendment 1 defines a
Software Engineering Process Reference Model for use with ISO/IEC 15504 Process Assessment.

(3-42) Post-implementation audits are a quality assurance activity. The post-implementation audit is used
to assess the ability of the IT organization to perform effective and efficient work. The results of the
audit will be used to both improve the software system and to improve the process that builds
software systems.

(3-18) The capability of Level 2 organizations is summarized as disciplined, because the ability to
successfully repeat planning and tracking of earlier projects results in stability.

(3-32)
• Acquirers and suppliers see a contract view, which includes the acquisition and supply
processes and begins with a contractual relationship to supply software. Depending on the contract, the supply process may use the development process to create new software, the operation process to provide software operation services, or the maintenance process to
repair or improve the software.
• Operators see an operating view, which includes the operations process. The operations
process may use the maintenance process.
• Developers and maintainers see the engineering view, which includes the maintenance and
development processes. The maintenance process may also use the development process.
• The employer of supporting processes sees the supporting view, which includes the eight
supporting processes.
• Managers see the corporate view, which includes the four organizational processes.

(3-7) Skill Category 1 provides two definitions of quality. These were “meets requirements” and “fit for
use.” Customer surveys use the “fit for use” definition of quality. Customer surveys are subjective
baselines. Customer surveys measure attitude and satisfaction of customers with the products and
services they receive. Because they are subjective it is important that customer surveys are properly
constructed.

(3-42) A post-implementation audit is conducted to determine any, or all, of the following:
• The system objectives were met
• The system specifications were implemented as specified
• The developmental standards were followed
• The IT quality objectives (e.g., maintainability) were achieved

(3-5) Baselines should be quantitative even if it is a subjective measure, but quantitatively subjective. For
example, because quality conformance and nonconformance must be defined by people, we are
looking for ways to put this information into a quantifiable format. This does not convey a lot of
information, but it is indicative of a problem. However, if we develop a five-point scale for
unresponsiveness, and ask your dissatisfied customer to complete that scale, we now have
conveyed a lot more information. If our scale rates “1” as very poor service, and “5” as very good
service, there is a great deal of difference between a “1” rating and a “3” rating for dissatisfaction.

An audit is an independent appraisal activity. While assessments may be done by the individuals
involved in conducting the work, an audit is performed by someone independent of the activity
under audit.
There are two general categories of auditors. These are auditors external to the organization
(external auditors), and auditors internal to the organization (internal auditors). The external
auditors are normally engaged by the organization’s Board of Directors to perform an independent
audit of the organization. Internal auditors are employees of the organization that they audit.

As shown in Figure 3-1, the CMMI® framework is a method for organizing these steps into five
levels of maturity. Each maturity level defines an evolutionary plateau of process improvement and
stabilizes an important part of the organization’s processes.
The five maturity levels are:
Level 1: Initial
At the initial level, an organization does not provide a stable environment for developing and
maintaining software. When an organization lacks sound management practices, the benefits of
good software engineering practices are undermined by reaction-driven commitments. In a
crisis, projects typically abandon any planned procedures and revert to a code and fix
methodology. Success depends on having exceptional people.
The process capability at Level 1 is considered ad hoc because the software development
process constantly changes as the work progresses. Schedules, budgets, functionality, and
product quality are generally unpredictable.
Level 2: Managed
Level 2 organizations have installed basic management controls. Policies for managing a
software project and procedures to implement those policies are established. Planning and
managing projects is based on experience with similar projects. Realistic project commitments
are based upon the results observed on previous projects and on the requirements of the current
project. Project managers track software costs, schedules, and functionality. Problems in
meeting commitments are identified when they arise. Software requirements and the work
products developed to satisfy them are baselined and their integrity is controlled.
The capability of Level 2 organizations is summarized as disciplined, because the ability to
successfully repeat planning and tracking of earlier projects results in stability. To be certified
at Level 2, organizations must document (define), practice, enforce, train, measure, and
improve the following six key process areas:
• Requirements Management
• Software Project Planning
• Software Project Tracking
• Software Subcontract Management
• Software Quality Assurance
• Software Configuration Management
Level 3: Defined
The standard engineering and management processes for developing and maintaining software
across an organization are documented, and these processes are integrated as a whole. There is
a group responsible for the organization’s software process activities, e.g., a standards
development group. An organization-wide training program is implemented to ensure that the
staff and the managers have the knowledge and skills required to fulfill their assigned roles.
Quality Baselines
Version 6.2 3-19
The capability of Level 3 organizations is summarized as standard and consistent because
engineering and management activities are stable and repeatable. Product lines, cost, schedule,
and functionality are under control and quality is tracked. Process definition and deployment
focus on the following key process areas:
• Organization Process Focus
• Organization Process Definition
• Training
• Integrated Software Management
• Software Product Engineering
• Inter-group Coordination
• Peer Reviews
Level 4: Quantitatively Managed
A Level 4 organization sets quantitative quality goals for both software products and processes.
Productivity and quality are measured and included in an organization-wide database. Projects
achieve control over their products and processes by narrowing the variation in their process
performance to fall within acceptable quantitative boundaries.
The capability of Level 4 organizations is summarized as predictable because the process is
measured and operates within measurable limits. Quantitative Process Management and
Software Quality Management are the two key process areas of Level 4.
Level 5: Optimizing
At Level 5 the entire organization is focused on continuous process improvement. The
organization has the means to identify weaknesses and strengthen the process proactively, with
the goal of preventing the occurrence of defects. Software project teams analyze defects to
determine their root causes, and lessons learned are disseminated to other projects.
The capability of Level 5 organizations is characterized as continuously improving, because
projects strive to improve the process capability and process performance. Key process areas of
Level 5 are:
• Defect Prevention
• Technology Change Management
• Process Change Management

(3-5)Subjective baseline studies will be the most commonly conducted studies in measuring quality and
productivity. Subjective means that judgment is applied in making the measure. We noted in the
discussion on objective measures that when the individual involved in recording time has the
option of applying judgment, then the measure becomes subjective.

(3-13)Staged models
Staged models are composed of a number of distinct levels of maturity. Each level of maturity is
further decomposed into a number of processes that are fixed to that level of maturity. The
processes themselves are staged and serve as foundations for the next process. Likewise, each level
of maturity is the foundation for the next maturity level.
Continuous models
In the continuous model, processes are individually improved along a capability scale independent
of each other.

(3-2) Baseline studies can be conducted in one of the following two manners:
• Evaluate entire population
This means all of the parties or products involved will be surveyed. This method is normally
most effective when the information to be analyzed is automated. For example, when looking
at factors such as schedule and budget.
• Sample survey
Using this method, only a part of the population of people/products are surveyed. This
approach is normally most effective when there is a large population to be surveyed and the
data is not automated. While sampling should be done statistically, it is not essential in these
studies for valid statistical samples to be drawn. The reason is that quality is attempting to
eradicate all defects, and even if the defect is only perceived by a part of the population, it is
still a defect and warrants attention.

(3-22)The award criteria are built upon a set of interrelated core values and concepts. These values and
beliefs are embedded behaviors found in high-performing organizations. The core values and
concepts are shown in Figure 3-3 and embodied in the following seven categories.
1 – Leadership
These criteria address the approach that the organization’s senior leaders take towards values,
directions, and performance expectations, as well as how they focus on customers and other
stakeholders, empowerment, innovation, and learning. Also examined is how the organization
addresses its responsibilities to the public and supports its communities.
2 – Strategic Planning
These criteria address how the organization develops strategic objectives and action plans.
Also examined are how the objectives and plans are deployed and how progress is measured.
3 – Customer and Market Focus
These criteria address how the organization determines requirements, expectations, and
preferences of customers and markets. Also examined is how the organization builds
relationships with customers, and determines the key factors that lead to customer acquisition,
satisfaction, and retention and to business expansion.
Quality Baselines
Version 6.2 3-23
4 – Information and Analysis
These criteria address the organization’s information management and performance
measurement systems and how the organization analyzes performance data and information.
5 – Human Resources
These criteria address how the organization motivates and enables its employees to develop
and utilize their full potential in alignment with the organization’s overall objectives and action
plans. Also examined are the organization’s efforts to build and maintain a work environment
and an employee support climate conducive to performance excellence and to personal and
organizational growth.
6 – Process Management
These criteria address the key aspects of the organization’s process management, including
customer-focused design, product and service delivery, primary business and support
processes. It encompasses all key processes and all work units.
7 – Business Results
These criteria address the organization’s performance and improvement in key business areas,
customer satisfaction, product and service performance, human resource results, financial and
marketplace performance, and operational performance. Performance levels relative to those of
competitors are also examined.

(3-7) We can divide customer surveys into report cards and surveys. Report cards ask the customer what
they think about something, for example, were they satisfied with the user manuals provided by IT.
The report card will ask them on a scale of 1-to-5, one meaning very pleased and five meaning very
displeased, on what they think of the user manual. Because the report card is not controlled, it is
sometimes difficult to know how the user calculated a particular rating.

(3-25) Other countries and regions have also created quality awards. These use similar categories and
variations of the scoring methods used by the MBNQA. The Deming Prize is awarded in Japan;
there are also the European Quality Awards and the Australian Quality Award in their respective
regions.

(3-8)There are many different steps that organizations follow in benchmarking. However, most
baselining processes have these four steps:
1. Develop a clearly defined baseline in your organization.
This means that all of the attributes involved in your baseline are defined. In our
example of defects per lines of code, clearly defining what is meant by defect and a line
of code would meet the objective of this step.
2. Identify the organizations you desire to baseline against.
Many factors come into this decision, such as do you want to benchmark within your
industry, do you want to benchmark what you believe are leading organizations, do you
want to benchmark an organization that uses the same tools that are used in your
organization, and do you want to benchmark against organizations with a similar
culture.
3. Compare baseline calculations.
Compare how your baseline is calculated versus the baseline calculation in the
company you want to benchmark against. Benchmarking is only effective when you
benchmark against an organization who has calculated their baseline using
approximately the same approach that your organization used to calculate the baseline.
4. Identify the cause of baseline variance in the organization you benchmarked against.
When you find a variance between the baseline calculation in your company and the
baseline calculation in the organization you are benchmarking against, you need to
identify the cause of variance. For example, if your organization was producing 20
defects per thousand lines of code, and you benchmarked against an organization that
only had 10 defects per thousand lines of code you would want to identify the cause of
the difference. If you cannot identify the cause of difference, there is little value in
benchmarking. Let us assume that the company you benchmarked against had a
different process for requirement definition than your organization. For example,
assume they use JAD (joint application development) and you did not. Learning this,
you may choose to adopt JAD in your organization as a means for reducing your
developmental defect rates.

(A-1) Benchmark An industry or best-of-class norm.

(3-2)Evaluate entire population
This means all of the parties or products involved
most effective when the information to be
at factors such as schedule and budget.

(3-14) In the continuous model, processes are individually improved along a capability scale independent
of each other. For example, the project planning process could be at a much higher capability level
than the quality assurance process.

(3-10) The organizational climate is the workers’ attitude toward their organization. It is a composite of
human behaviors, perception of events, responses of employees to one another, expectations,
interpersonal conflicts, and the opportunities for growth in the organization. The climate is crucial
in creating and maintaining an effective organization, and, therefore, should be periodically
evaluated to discover whether the satisfaction level of the employees is positive or negative.

(3-5) Examples of products/services that can be measured subjectively for developing a baseline include:
• Customer satisfaction
• Effectiveness of standards/manuals
• Helpfulness of methodologies to solve problems
• Areas/activities causing the greatest impediments to quality/productivity
• Causes for missed schedules/over-budget conditions
• Understandability of training materials
• Value of tools
• Importance of activities/standards/methods/tools to individual activity

(3-32) The target audience for ISO/IEC 12207 is:
• Organizations acquiring a system that contains software or a stand-alone software
product
• Organizations that supply software products
• Organizations involved in software operation or software maintenance
The standard is especially applicable for acquisitions, as it recognizes the distinct roles of acquirer
and supplier. Its intended use is for the two parties of an agreement or contract that defines the
development, maintenance, or operation of software. It does not apply to the purchase of
commercial off-the-shelf software products. It is also intended for use with trained, experienced
developers, managers, and acquirers of software.

(3-14)An organization should consider these four criteria when selecting a model:
• Applicability of the model to the organization’s goals and objectives.
Management commitment
Need for baseline assessments
Need for measurable goals and objectives

(3-29) ISO/IEC 12207, which was published in 1995, is the international standard that covers the software
life cycle from concept through retirement. It contains a framework for managing, controlling, and
improving the software life cycle activities. The standard describes the major processes of the
software life cycle, how these processes interface with each other, and the high- level relations that
Guide to the CSQA CBOK
3-30 Version 6.2
govern their interactions. It has subsequently been amended twice; Amendment 1 defines a
Software Engineering Process Reference Model for use with ISO/IEC 15504 Process Assessment.
For each process, the standard also describes the activities and tasks involved, defining specific
responsibilities and identifying outputs of activities and tasks. Since it is a high-level standard, it
does not detail how to perform the activities and tasks. The standard does not assume a specific life
cycle model, nor does it specify names, format, or content of documentation. As a result,
organizations applying ISO/IEC 12207 should use other standards and procedures to specify these
types of details.
The 17 processes covered in this standard are grouped into three categories, as follows:
Primary Processes
• Acquisition Process
• Supply Process
• Development Process
• Operation Process
• Maintenance Process
Supporting Processes
• Documentation Process
• Configuration Management Process
• Quality Assurance Process
• Verification Process
• Validation Process
• Joint Review Process
• Audit Process
• Problem Resolution Process
Organization Processes
• Management Process
• Infrastructure Process
• Improvement Process
• Training Process

(3-6)Planning
To determine where detailed investigation/survey should be undertaken.
Internal analysis
To identify problems/areas for quality improvement. Once the problem/area has been
identified, then no additional effort need be undertaken to formalize the results.
Benchmarking
Comparison against external organizations.

(3-6) Baselines can be conducted for any one of the following three purposes:
Planning
To determine where detailed investigation/survey should be undertaken.
Internal analysis
To identify problems/areas for quality improvement. Once the problem/area has been
identified, then no additional effort need be undertaken to formalize the results.
Benchmarking
Comparison against external organizations.

(3-25) In 1997, ISO conducted a large global survey to better understand their needs. This survey revealed
the need to substantially re-engineer the model itself, with an emphasis on four needs:
• A common structure based upon a Process Model
• Promote use of continuous improvement and prevention of non-conformity
• Simplicity of use, ease of understanding, and use of clear language and terminology
• Consistency with the Plan-Do-Check-Act improvement cycle

(3-10) An evaluator can use the six steps below to assess the climate of a specific organization, group,
committee, task force, etc.
1. Look at the organization (group, committee, task force, etc.)
Assess the mission and goals of the organization, what it is supposed to produce, and
the overriding principles by which it operates.
2. Examine the jobs
Examine each job in the organization. Ask whether the job is necessary, whether it
makes full use of the employee’s capabilities, and whether it is important in
accomplishing the mission and goals of the organization.
3. Assess employees’ performance
Evaluate each employee’s performance in relation to the organization’s mission and
goals. For each job being performed, ask if the employee is doing what should be done,
is using his or her skills effectively, likes his or her job, and has enthusiasm and interest
in performing the job.
4. Evaluate how employees feel about their manager or leader
Good organizational climate requires good leadership. Determine whether each
employee within the group likes his or her manager, whether they follow or ignore the
requests of their manager, and whether they attempt to protect their manager (i.e., make
their manager look good).
5. Create a dialog with the members of the group
Interact with each employee asking a series of hypothetical questions to identify the
employee's true feelings toward the organization. Questions such as, “do you feel the
organization supports your suggestions”, can help draw out the true feelings of each
employee.
6. Rate organizational climate
Based on the responses to steps 1-5, evaluate the climate based on the 5-point scale on p. 3-10.

(3-21) The United States national quality award originated from the Malcolm Baldrige National Quality
Improvement Act (Public Law 100-107), signed by President Ronald Reagan on August 20, 1987.
That act, named after a former Secretary of Commerce, called for the creation of a national quality
award and the development of guidelines and criteria that organizations could use to evaluate their
quality improvement efforts. Awards are given in the five categories below, with no more than two
awards being presented per category per year.
• Manufacturing
• Service
• Small Business
• Educational Organizations
• Health Care Organizations

(3-7) To help understand the type of baselines that are used in IT, QAI has
categorized four most commonly used baselines as listed below.
• Customer surveys
• Benchmarking
• Management established criteria
• Industry models

(3-9)A less formal method for benchmarking is to visit other organizations. This will provide the quality
professionals with these benefits:
• The cost and effort to develop new and innovative quality approaches within IT is
cost-prohibitive for most companies. Learn from others and don’t “reinvent the
wheel”.
• Comparing quality programs to those in other companies can identify gaps in current
processes and lead to obtaining more effective quality practices.
• Interfacing periodically with other quality individuals is good for professional
development. Those colleagues will not exist internally unless the company is
large.
Visiting another company is a five-step process, as follows:
1. Identify discussion areas
As it is important for the visit to be mutually advantageous, get management agreement
on what oral and written information can be shared with the company being visited.
Determine the objective of the visit.
• Identify a specific area such as conducting software reviews or independent
testing.
• If the objective is to gather general information, convert it into a visit objective,
such as identifying the three most effective quality practices used by the
other company.
2. Identify target companies
Visits can be within divisions or subsidiaries of the corporation, or to other
organizations or corporations. Identify constraints that need to be considered as part of
the selection process. Consider items such as whether information can be exchanged
with competitors, the availability of travel funds, whether the size of the target
company should be similar, and whether the maturity of the company's quality function
lends itself to the topics selected (starting a quality function requires a company that
has gone through the process). Select three target companies, prioritizing them by the
desirability, and obtain management's approval to schedule a visit with them.
3. Schedule the visit
Contact your peers at the targeted companies. Identify yourself and your company.
State the purpose for the visit, what you can share, the information you would like to
receive in exchange and how the visit would be mutually advantageous. Offer to
provide a letter requesting the visit if your colleague needs it to get the visit approved.
Set a date and time for the visit. One-half to two days is recommended.
4. Conduct the visit
Visits typically begin with introductions, a restatement of the objectives of the visit and
a tour of the other company’s facilities to put the size and purpose of the IT function in
perspective. Agenda items are then discussed in detail. Written documentation is
exchanged and agreement for usage (such as copying, reworking, distributing, etc.) is
given. The visit concludes by thanking management of the other company, and, if
possible, leaving some memorabilia or some of your company’s products as a token of
remembrance.
5. Put new practice into use
Select the one best idea from the visit for implementation. Demonstrating a positive
outcome from these visits will increase the chance of management approving other
visits. Do not try to implement too many things at once

(3-10) Management can develop a baseline for anything they feel needs to be measured and/or improved.
These are baselines that are organizational dependent. There are no industry models or standards
against which a baseline can be determined.

(3-2 - 3-3)
Quality assurance groups
If one has been established, they should conduct baseline studies in areas of quality concern.
• Quality task forces
A special task force or committee established to study quality/productivity in an information
services function. In many instances these study groups precede the establishment of a quality
assurance function. In fact, many of the individuals who chair the study group later become the
quality assurance manager.
• IT management
Specific managers that have a concern over quality may wish to perform a baseline study. Note
that in many instances the study is actually conducted by a subordinate to that manager.

(3-5) Objective measures are those, which can be accomplished by counting. Examples of objective
baseline measures that can be used for baselines include:
• Project completed on schedule
• Lines of code
• Number of programs
• Number of people assigned to a project
• Number of abnormal terminations
Again, the exactness of the counting will actually determine whether the above measures are
objective or subjective. It is important to recognize that there are very few objective measures, and
thus we are forced to use subjective measures in measuring quality and productivity.

Process
Process refers to the methods your organization uses and improves to address the Item
requirements in Categories 1-6. The four factors used to evaluate process are Approach,
Deployment, Learning, and Integration (A-D-L-I).
“Approach” refers to:
• The methods used to accomplish the process
• The appropriateness of the methods to the Item requirements
• The effectiveness of your use of the methods
• The degree to which the approach is repeatable and based on reliable data and information
(i.e., systematic)
“Deployment” refers to the extent to which:
• Your approach is applied in addressing Item requirements relevant and important to
your organization
• Your approach is applied consistently
• Your approach is used by all appropriate work units
“Learning” refers to:
• Refining your approach through cycles of evaluation and improvement
• Encouraging breakthrough change to your approach through innovation
• Sharing of refinements and innovation with other relevant work units and processes
in your organization
“Integration” refers to the extent to which:
• Your approach is aligned with your organizational needs identified in other Criteria
Item requirements
• Your measures, information, and improvement systems are complementary across
processes and work units
• Your plans, processes, results, analyses, learning, and actions are harmonized across
processes and work units to support organization-wide goals
“Results” refers to your organization’s outputs and outcomes in achieving the requirements in
Category 7. The four factors used to evaluate results are:
• Your current level of performance
Quality Baselines
Version 6.2 3-25
• Rate (i.e., slope of trend data) and breadth (i.e., how widely deployed and shared) of
your performance improvements
• Your performance relative to appropriate comparisons and/or benchmarks
• Linkage of your results measures (often through segmentation) to important customer,
product and service, market, process, and action plan performance requirements
identified in your Organizational Profile and in Process Items.

(3-26) The ISO 9001:2000 standards are based on the following eight quality management principles.
1. Customer Focus
2. Leadership
3. Involvement of People
4. Process Approach
5. System Approach to Management
6. Continuous Improvement
7. Factual Approach to Decision-Making
8. Mutually Beneficial Supplier Relationships

(3-1) Developing a baseline is performing an analysis/study to determine the current level of performance in a specific activity.

(3-18) To be certified
at Level 2, organizations must document (define), practice, enforce, train, measure, and
improve the following six key process areas:
• Requirements Management
• Software Project Planning
• Software Project Tracking
• Software Subcontract Management
• Software Quality Assurance
• Software Configuration Management

(3-4)Objective baseline studies are ones which are viewed as factual and non-argumentative. There are
very few objective studies that can be conducted within information services.

(3-32) The target audience for ISO/IEC 12207 is:
• Organizations acquiring a system that contains software or a stand-alone software
product
• Organizations that supply software products
• Organizations involved in software operation or software maintenance
The standard is especially applicable for acquisitions, as it recognizes the distinct roles of acquirer
and supplier. Its intended use is for the two parties of an agreement or contract that defines the
development, maintenance, or operation of software. It does not apply to the purchase of
commercial off-the-shelf software products. It is also intended for use with trained, experienced
developers, managers, and acquirers of software.

(3-3) The following are typical steps needed to perform a baseline study.
1. Identify products/services to be surveyed
This is the requirements phase of the baseline study. Studies should be directed at
specific products or services. For example, computer programs as a product, and
customer/user interaction as a service.
2. Define conformance and nonconformance
The individuals developing the survey instrument must have defined (at least on a
preliminary basis) the expected conformance and nonconformance. Note that in many
instances the survey will be used to help establish nonconformance, but the general
areas of nonconformance will need to be identified in order to gather nonconformance
data.
3. Identify survey population
The population of data/people to be surveyed needs to be identified. This is a critical
step because the definition of nonconformance will vary significantly depending upon
who is considered the population. For example, programming defects would look
significantly different to the programmer than the end user of the program results. The
programmer may only consider variance from specs a defect, but to the end user not
meeting needs is a defect.
4. Identify size of population to be surveyed
This step is one that involves economics. It is always cheaper to look at fewer than
more. The question that needs to be decided is how few can be examined to give valid
results. Statistically, we rarely try to go below a sample size of twenty, but in surveying
people we may be able to drop below this limit and still get valid results.
5. Develop survey instrument
A specific survey instrument must be developed to meet the survey objectives. Surveys
need to be customized to the specific needs and vocabulary of the population being
surveyed.
6. Conduct survey
The survey instruments should be completed by the surveyed population. From a
quality perspective; it is helpful to train the population on how to complete the survey
questionnaire. This can be done through written instructions, but it is normally better to
do it verbally. If the population group is small, they can be called together for a
meeting, or have the survey instruments hand delivered and explained. Generally, the
validity of the results will increase when extra time is spent to explain the intent and
purpose of the survey.
7. Follow up on incomplete surveys
The survey should have a time limit for response. Normally this should not exceed one
week. If the surveys have not been returned by that time, then the surveying group
should follow up and attempt to get as many surveys completed as possible. Note that it
is generally not realistic to expect every survey to be returned.
8. Accumulate and present survey results
The survey information should be accumulated, consolidated, and put into presentation
format. Suggestions on how to do this follow in a later part of this quality initiative.
9. Take action and notify participants of that action
All surveys should result in some specific action. Even if that action is to do nothing, a
decision should be made based on the survey results. That decision should be given to
the survey participants. Note that whenever a survey is conducted, the participants
expect some action.