minitest-meaningful

Minitest::Meaningful

This is a proof-of-concept Minitest plugin to help avoid writing completely meaningless tests. A meaningless test is one that always passes because of incorrect test setup. This plugin helps ensure tests aren't meaningless, by trying to make them fail.

For example, in a Rails app we might have a test that looks like this:

test "#visible_comments excludes hidden comments" do
  post = create(:post)
  comment = create(:comment, status: "hidden")

  refute_includes post.visible_comments, comment
end

The comment record is completely unrelated to the post record. It doesn't matter what the status is, nor does it really matter what the #visible_comments implementation looks like, there's no reason this comment would ever be returned. It always passes, it's a false positive test.

This plugin adds an important! annotation to wrap the most important test setup variable. The test runner can then run the test both with and without evaluating that block. If the test passes when that important! block is not evaluated, then we know it is not meaningful.

test "#visible_comments excludes hidden comments" do
  post = create(:post)
  comment = create(:comment)
  important! { comment.update!(status: "hidden") }

  refute_includes post.visible_comments, comment
end

Installation

Install the gem and add to the application's Gemfile by executing:

$ bundle add minitest-meaningful

If bundler is not being used to manage dependencies, install the gem by executing:

$ gem install minitest-meaningful

Usage

Include the Minitest::Meaningful module in your test:

class ExampleTest > Minitest::Test
  include Minitest::Meaningful
end

Annotate the most important test setup variable:

def test_visible_comments_excludes_hidden_comments
  post = create(:post)
  comment = create(:comment)
  important! { comment.update!(status: "hidden") }

  refute_includes post.visible_comments, comment
end

Assert the test should fail:

assert_meaningful :test_visible_comments_excludes_hidden_comments

Run the test with the --meaningful flag:

$ ruby example_test.rb --meaningful

Contributing

This is definitely a hacky proof-of-concept. If you can think of a better way to integrate this into Minitest test suites, or a way to more robustly identify false-positive tests, please let me know! Also, I hate the name—please suggest something better!