Testing heuristics when it comes to mocks and fakes and contract verification

04-09-2025

• my heuristics

  • for all tests
    • you want corrrectness and speed
  • for routers
    • ensure the signature is valid (ie the underlying methods and attributes are called correctly)
  • for clients (external APIs)
    • use mocks and autospec
    • ensure the signature is valid (ie the underlying methods and attributes are called correctly)
  • for clients (databases)
    • use sqlite in the session manager, then setup and teardown relevent test cases
  • for business logic that use clients
    • use (verified) fake clients that follow a protocol
      • enforce contract verification tests between the fake objects and the real objects
    • ensure the behaviour is valid (ie the inputs and outputs match expectations)

• https://martinfowler.com/articles/mocksArentStubs.html

  • dummy objects (passed around but never used)
  • fake objects (working implementation with some shortcut ie in-memory database)
  • stubs (provide canned answers to specific calls)
  • spies (stubs that record usage information)
  • mocks (objects pre-programmed with expectations)

• https://pythonspeed.com/articles/verified-fakes/

  • testing external services; slow, or fake and uncertain
  • use test doubles
    • this ensures you match the signaure but not the behaviour
    • example
      • MessageService → TwitterClient
      • Tests → MessageService → TwitterClient
      • Tests → MessageService → FakeTwitterClient
  • use verified fakes (when the API is slow or you want to test frequently)
    • two tests, once against the real client and once against the fake client (contract verification tests)
    • example
      • TwitterContractTests → TwitterClient
      • TwitterContractTests → FakeTwitterClient

• https://www.b-list.org/weblog/2023/dec/08/mock-python-httpx/

  • make testing easier
    • dependency injection helps with this

https://hynek.me/articles/what-to-mock-in-5-mins/

• don't mock what you don't own

• avoid the need to use multiple layers of mocks and patches, or worry about the orderings and references, brittle when module paths are refactored

• ensure determinism in results using stubs

  • https://github.com/alex/pretend

• you may have business logic tests, but spend lots of time with boilerplate trying to mimic a client whose API may change ay any time, are you even testing what you want to test

• https://nedbatchelder.com/blog/202202/why_your_mock_still_doesnt_work.html

  • mocks are dangerous
    • you can call any method or access any attribute and they will return another mock
    • this can silence errors if patch ordering is incorrect or if methods and attributes don't exist
  • use spec=True and autospec=True (which recursively applies to any new mocks from the original mock)

from unittest.mock import patch
 
@patch("httpx.Client")
def test_httpx(mock_client):
    mock_client.fake_method_that_doesnt_exist()  # passes
    mock_client.another_fake_thing()  # passes
    mock_client.xyz.abc()  # passes
    print("All fake methods accepted! 😱")