Combining qualitative and quantitative software process evaluation sylvie trudel

15-‐10-‐12

1

COMBINING QUALITATIVE AND QUANTITATIVE
SOFTWARE PROCESS EVALUATION:
A PROPOSED APPROACH
Sylvie Trudel
Dept. of Computer Science
UQAM
Alex Turcotte
CEFTI
Université de Sherbrooke
2© TRUDEL & TURCOTTE 2015 IWSM MENSURA 2015 - CRACOW
²  So)ware
process
evalua4on
(SPE)

²  Quan4fying
the
so)ware
process
eﬃciency

²  Combining
qualita4ve/quan4ta4ve
SPE

²  Field
trial:
ﬁnancial
trading
domain

²  Discussion

²  Future
work

Content

15-‐10-‐12

2

SOFTWARE PROCESS EVALUATION
(SPE)
INTRODUCTION: WHY, WHO, HOW & HOW MUCH
© TRUDEL & TURCOTTE 2015 IWSM MENSURA 2015 - CRACOW
4
•  Why: competition, market demands, costs
–  Symptoms: budget & schedule overruns (especially on
larger projects), poor quality, unsatisfied customers/
users
•  How: evaluate against best practice models (e.g. CMMI)
•  Who:
–  Large organizations vs Small organizations:
same methods?
Understanding
what
needs
to
be
improved

15-‐10-‐12

3

CMMI
Maturity
Levels

1. Initial
Unpredictable and Poorly Controlled
4. Quantitatively
managed
Controlled and Measured Process
Predictable Process Process and
Product Quality
5. Optimizing
Emphasis on Process
Improvement
Continuous Improvement
Process
Change
Management
2. Managed
Reactive Project Defined
Process
Project
Management
Structured
Process
3. Defined
Proactive Organizational
Defined Process
Standard and Consistent
Process
Integrated
Engineering Process
Heroes1. Initial
Level Emphasis
5. Optimizing
Quantitative
Management
Basic Project
Management
Process Areas
Decision Analysis and Resolution
Risk Management
Integrated Project Management for IPPD
Organizational Training
Organizational Process Definition
Organizational Process Focus
Validation
Verification
Product Integration
Technical Solution
Requirements Development
Configuration Management
Process and Product Quality Assurance
Measurement and Analysis
Supplier Agreement Management
Project Monitoring and Control
Project Planning
Requirements Management
Risks and
Rework
Organizational Process Performance
Quantitative Project Management
Organizational Innovation and Deployment
Causal Analysis and Resolution
Continuously Improving
Processes
Processes
Standardization
Productivity
and quality
Results
2. Managed

3. Defined
4. Quantitatively
Managed
CMMI-‐Dev
Overview
«Staged»

15-‐10-‐12

4

Process
Evalua9on
Methods
Method
Targeted

organiza9ons

Cer9ﬁed
lead

appraiser

Ra9ng
Rela9ve
cost

SCAMPI
‘A’
Large
þ
þ
$$$$$

SCAMPI
‘B’
Large
þ
-‐
$$$

SCAMPI
‘C’
Large
&
Medium
-‐
-‐
$$

ISO/IEC
29110-‐3
Small
-‐
-‐
$

PEM
Small
&
Medium
-‐
-‐
$

Original
PEM
Method
(1
of
3)
From ISO 14598-5
1- Analysis of Evaluation
Requirements
2- Specification of the
Evaluation
3- Design of the Evaluation
•  Evaluation request
•  Context: projects requirements,
evaluation requirements
•  Statement of work template
•  SOW (draft)
•  CMMI Models
•  Process Area Selection Guide
•  Context
•  Statement of work (draft):
ü  Evaluation objectives
ü  Confidentiality agreement
ü  Assumptions and constraints
•  SOW (final) Scope:
ü  Selected Projects
ü  Selected Model
ü  Selected Process Areas
•  Client contract/agreement
•  SOW
•  Evaluation Plan Template
•  Evaluation Method
•  List of typical questions
•  Evaluation Plan
•  List of Selected Questions

15-‐10-‐12

5

Original
PEM
Method
(2
of
3)
From ISO 14598-5 (continued)
4- Interviewing Project
Participants and Reviewing
Project Documentation
5- Reviewing and Reporting
Observations
6- Conclusion of the
Evaluation
Execution of the Evaluation
•  Interviews and Document
reviews observations
•  Findings (draft)
•  Evaluator’s Checklist (started)
•  List of documents to evaluate
•  Evaluator’s Checklist (new)
•  Interview Guideline
•  Wording of Findings Guideline
•  Finding Selection Guide
•  Evaluation Report Template
•  Evaluator’s Checklist (updated)
•  Evaluation Report (draft)
•  Findings (final and complete)
•  Requesters Comments
•  Evaluation Report (final)
•  Evaluator’s Checklist (final)
reviews observations (destroyed)
Original
PEM
Method
(3
of
3)
Value Added Step
7- Planning of Improvement
Actions (optional)
•  Action plan, including:
ü  Related findings
ü  Activities, deliverables, tools
ü  Estimates and schedule
ü  Stakeholders involvement
ü  Return on investment
•  Evaluation Report (final)
•  Action plan template

15-‐10-‐12

6

QUANTIFYING THE SOFTWARE
PROCESS EFFICIENCY
HOW TO MEASURE ASPECTS OF THE SOFTWARE
PROCESS ?
So)ware
to
measure

Boundary

COSMIC : Overview
Func4onal

process
1

Persistent

storage

Write
(W)

Read
(R)

Func4onal

process
2

Func4onal

process
n

…

Human
users

or

IO

Hardware

Entry
(E)

Exit
(X)

or

Func4onal

Users

or

Other

systems

Data

‘Interfaces’
‘Infrastructures’


15-‐10-‐12

7

Applying
COSMIC:
3
phases

FUR
Chapter 2
Measurement
Strategy
Generic Software Model
Chapter 3
Mapping
Phase
FUR in the form of the
Generic Software Model
Chapter 4
Measurement
Phase
Functional size of
the software in
units of CFP
Definition of each piece of
software to be measured and
of the required measurement
Input from measurement sponsor
Software Context Model
FUR
Poten4al
measurement

purpose:
Quan4fy
the
SW

process
produc4vity
rate

Possible
side
eﬀect:
Iden4fy

defects
in
Func4onal

Requirements!

COMBINING QUALITATIVE AND
QUANTITATIVE SPE
WHY AND HOW: A PROPOSED APPROACH…

15-‐10-‐12

8

²  Mo4va4ons:

Ø  Bring
a
broader
insight
on
the
SW
Process

Ø  Overlooked
issues
related
to
requirements
engineering

Ø  Provide
a
SW
process
produc4vity
rate

²  Hypothesis:
Mutual
inﬂuences
between
measurement

results
and
qualita4ve
ﬁndings

Ø  Requires
to
be
combined
during
execu4on
of
the
evalua4on!

Combining
Qualita9ve
and
Quan9ta9ve
SPE

Proposed
approach
4a- Interviewing Project
Participants and Reviewing
Project Documentation
5- Reviewing and Reporting
Observations
Execution of the Evaluation
•  Evaluator’s Checklist (started)
•  List of documents to evaluate
•  Evaluator’s Checklist (new)
•  Interview Guideline
•  Wording of Findings Guideline
•  Finding Selection Guide
•  Evaluation Report Template
•  Evaluator’s Checklist (updated)
•  Findings (final and complete)
4b- Measuring software
functional size and process
efficiency
•  Selected FSM Method
•  FUR from selected projects
•  Effort from selected projects
•  Quality rating guidelines
•  Functional Size
•  Efficiency data (analysed)
•  Requirements defects
(identified)
•  Findings (updated)

15-‐10-‐12

9

FIELD TRIAL:
FINANCIAL TRADING DOMAIN
WHAT WERE THE RESULTS?
1.  Analysis:
100
staff
members
(most
projects
<
10)
à
trading
SW

for
deriva4ve
markets,
documented
process,
percep4on
of
high

costs
from
clients
à
Budget
overruns
on
larger
projects

2.  Specifica4on:
1
Business
domain
(2
major
clients),
CMMI
2
&
3

except
SAM-‐OT-‐DAR
à
6
projects
selected
(3
large
[500+
staff-‐
days],
3
regular)

3.  Design:
Plan
to
interview
21
par4cipants
(at
least
2
per
role)
à

ques4onnaire
developed,
2-‐page
email
instead
of
kick-‐off

Organiza9on
&
Evalua9on
Characteris9cs
(1/2)

15-‐10-‐12

10

4.  a.
Interviews:
18
out
21
par4cipants
interviewed,
confiden4ality

assured
à
effort
=
50
staff-‐hours

4.

b.
Measurement:
Done
while
reviewing
requirements
&
projects

documenta4on
à
ambigui4es
led
to
examining
SW
code;
results

verified
by
cer4fied
measurer,
then
analysis
done

5.  Review
&
Report:
Valida4on
of
findings
by
par4cipants

6.  Conclusion:
Final
report
combines
results
à
RE
inconsistencies

raised,
recommenda4ons
proposed
for
improvement

Organiza9on
&
Evalua9on
Characteris9cs
(2/2)

Ini4al
produc4vity
model
with
func4onal
size
and
effort
from
all
six
projects

15-‐10-‐12

11

Comparison
of
func4onal
size
and
eﬀort
for
the
remaining
four
projects

DISCUSSION AND FUTURE WORK
WHAT DID WE LEARN AND WHAT’S NEXT?

15-‐10-‐12

12

²  Adapta4on
of
PEM
to
combine
qualita4ve
(CMMI)
and

quan4ta4ve
(COSMIC)
evalua4on
of
the
SW
Process
and
the
SW

Requirements

²  Several
benefits
from
the
field
trial:

1.  Measurement
ßà
Review
of
requirements
quality
(implicit)

2.  Rela4on
between
measurement
results
and
qualita4ve
findings

3.  Preliminary
es4ma4on
model
obtained

4.  Acceptable
level
of
effort
for
a
small
organiza4on

Discussion

²  Adapt
PEM
from
ISO/IEC
14598-‐5
to
its
updated
SQuaRE

version
(i.e.
ISO/IEC
25000)

²  Verify
compliance
of
PEM
with
ISO/IEC
15504

²  Include
a
customer
sa4sfac4on
survey
to
provide
a
360˚

view
of
the
SW
process
being
evaluated

Future
Work

15-‐10-‐12

13

25
26

15-‐10-‐12

14

Thank you !
27

Combining qualitative and quantitative software process evaluation sylvie trudel

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

Similar to Combining qualitative and quantitative software process evaluation sylvie trudel

Similar to Combining qualitative and quantitative software process evaluation sylvie trudel (20)

More from IWSM Mensura

More from IWSM Mensura (20)

Recently uploaded

Recently uploaded (20)

Combining qualitative and quantitative software process evaluation sylvie trudel