Verification vs. Validation

• Verification: did we build the system right? Does it work well?
• Validation: did we build the right system? Does the system fit the client's requirements? Are the features the ones that match the specs?

Proving vs. Testing

• Proving: using mathematics and logic to prove that the program is logically sound and has no bugs
  • In general this is very difficult to do and very expensive
  • Statically analyze the program with logic
• Testing: using real-life input and check-expects to see that the program does what it should
  • Can only show the presence of bugs, but not absence
  • Dynamically execute the program with input

Types of Testing

• White-box testing: the tester can see implementation of the system
• Black-box testing: the tester cannot see the implementation, can only interact with the user interface of the system
• Unit testing: testing the smallest unit of a program - functions in isolation
• Regression testing: testing against a set of past failures
• Integration testing: individual units are combined and tested as a group
• show that major subsystems work well together
• System testing: test large parts of the system, using different configurations in different environments
• Recovery testing - forcing the software to fail and verify that it can recover as expected (power failure, OS failure etc)
• Security testing - verify that software is resistant towards malicious intent
• Stress testing - put the system under stress in various ways and test how it fares
  • quantity of input
  • speed of input
  • extreme CPU/memory conditions
• Acceptance testing: test with respect to requirements to ensure the software satisfies the acceptance criteria

Continuous Integration (CI)

Developers continuously commit code to a pipeline that automatically builds and tests the program.

• This can include performance tests, regression tests, unit tests
• Report is usually automatically generated and stored for those who are responsible to look after CI

Testing Methodology

• Harness: a short program that calls the function to be tested and verifies that the results are correct.
• Test coverage: try to cover each branch of the function. There should be at least one test for each of the branches
• Boundary cases: legitimate inputs that are on the boundary of illegal input.
• Test suite: a collection of test cases
• Cycling: a phenomenon sometimes happens when bugs cycle back due to bandaid fixes that didn't fix the root issue.
• Equivalence Parititioning: because the input domain of methods are usually too large to test exhaustively, split the input domain into categories that will be treated similarly, test a few inputs from each category to ensure different types of inputs are tests, but not too many test cases are needed
• Boundary Value Analysis:  take special care when testing boundary cases, tests should include tests on the boundary and on both sides of the boundary to ensure the decisions the program makes for those particular values are correct

Class Testing

• Classes are different from testing individual functions because classes are typically stateful systems.
• Classes can store states with member variables; the methods evoked may have side-effects not observed by the return statement
• Convert the class into a finite state machine (FSM), and test the graph that is produced from this FSM
  • Each state may have a set of expected behaviour (nodes of the graph)
  • Each state may have triggers that transition it to a different state (edges of the graph)
  • The combination of state behaviour and state transitions may create loops or paths that have a set of expected behaviour (paths of the graph)