Software Testing

What is Software Testing?

According to D.Galin, software testing is a formal process carried out by a specialized testing team in which a software unit, several integrated software units, or an entire software package are examined by running the programs on a computer; all the associated tests are performed according to approved test procedures on approved test cases.

Basic Testing Definitions

Checkout Failure fault and error.

Software testing

Exercise the software with test cases to gain (or reduce) confidence in the system (execution based on test cases)

• Expectation $\to$ reveal faults with failures incidences

Test cases

Set of inputs and a list of expected outputs (sometimes left out)

Test stub

Partial implementation of a component on which the tested component depends

Test driver

Partial implementation of a component that exercises and depends on the tested component.

Test stubs and drivers enable components to be isolated from the rest of the system for testing.

Not all tests will trigger failure! Not all failure can be associated to fault

Three conditions must be present for a failure to be observed

• The location or locations in the program that contain the fault must be reached (Reachability)
• After executing the location, the state of the program must be incorrect (Infection)
• The infected state must propagate to cause some output of the program to be incorrect (Propagation)

Objectives of Testing

• Main goal: demonstrate that the software can be depended upon
• To bring the tested software, after correction of the identified defects and retesting, to an acceptable level of quality
• To perform the required tests efficiently and effectively, within budgetary and scheduling limitation
• To compile a record of software errors for use in error prevention (by corrective and preventive actions)
• Catch faults before they become errors “in the field”

Only a very small portion of the inputs are encoded into the test cases.

Testing Approach

What is testing?

Testing

• Execute a program with a sample of the input data

Dynamic technique

• Program must be executed

Test cases

• Test method is a repeatable way to generate test cases

Optimistic approximation

• The program under test is exercised with a (very small) subset of all the possible input data
• We assume that the behaviour with any other input is consistent with the behaviour shown for the selected subset of input data

Validation vs. Verification

Validation: The process of evaluating software at the end of software development to ensure compliance with intended usage ⇒ Do the right things.

Verification: The process of determining whether the products of a given phase of the software development process fulfill the requirements established during the previous phase ⇒ Do the things right.

Testing Techniques

There are a number of techniques. Each has different processes, artifacts, or approaches.

There are no perfect techniques or there is no best techniques.

Levels of Software Testing

Unit Testing

• Individual units (e.g., a function) are tested in isolation.
• Determine whether each unit functions as designed.

Integration Testing

• Test a group of related units together (e.g., testing database access).
• Find interface issues between units.

Integration of well-tested components may lead to failure due to:

• Bad use of interfaces (bad interface specifications / implementation)
• Wrong hypothesis on the behaviour / state of related modules (bad functional specification / implementation), e.g., wrong assumption about return value
• Use of poor drivers/stubs: a module may behave correctly with (simple) drivers/stubs, but result in failures when integrated with actual (complex) modules

Well tested modules may still FAIL integration tests...

System Testing

• Test the complete software system.
• Evaluate the system’s compliance with the specified requirements.

Approach to Identify Test Cases

Black-Box Testing

• Based on software description (specification)
• Covers as much specified behaviour as possible
• Cannot reveal errors due to implementation details

White-Box Testing

• Based on the code
• Covers as much coded behaviour as possible
• Cannot reveal errors due to missing paths

Testing Maturity Levels

Beizer’s testing levels on test process maturity.

There are 4 levels of maturity:

• Level 4
  • Testing is a mental discipline that helps all IT professionals develop higher quality software
  • Mental discipline that increases quality
  • Testers become technical leaders of projects
  • Improve quality software
• Level 3
  • Purpose of testing is not to prove anything specific but to reduce the risk of using the software
• Level 2
  • Purpose of testing is to show that software doesn’t work
• Level 1
  • Purpose of testing is to show that software works+
• Level 0
  • No difference between testing and debugging
  • Basically what students like me do. If it compiles it’s good…