A property-based testing tool for Ruby with experimental features that allow you to run test cases in parallel.
PBT stands for Property-Based Testing.
As for the results of the parallelization experiment, please refer the talk at RubyKaigi 2024: Unlocking Potential of Property Based Testing with Ractor.
Property-Based Testing is a testing methodology that focuses on the properties a system should always satisfy, rather than checking individual examples. Instead of writing tests for predefined inputs and outputs, PBT allows you to specify the general characteristics that your code should adhere to and then automatically generates a wide range of inputs to verify these properties.
The key benefits of property-based testing include the ability to cover more edge cases and the potential to discover bugs that traditional example-based tests might miss. It's particularly useful for identifying unexpected behaviors in your code by testing it against a vast set of inputs, including those you might not have considered.
For a more in-depth understanding of Property-Based Testing, please refer to external resources.
Add this line to your application's Gemfile and run bundle install.
gem 'pbt'
Off course you can install with gem intstall pbt.
# Let's say you have your own sort method.
def sort(array)
return array if array.size <= 2 # Here's a bug! It should be 1.
pivot, *rest = array
left, right = rest.partition { |n| n <= pivot }
sort(left) + [pivot] + sort(right)
end
Pbt.assert do
# The given block is executed 100 times with different arrays with random numbers.
# Besides, the block runs in parallel by Ractor.
Pbt.property(Pbt.array(Pbt.integer)) do |numbers|
result = sort(numbers)
result.each_cons(2) do |x, y|
raise "Sort algorithm is wrong." unless x <= y
end
end
end
# If the method has a bug, the test fails and it reports a minimum counterexample.
# For example, the sort method doesn't work for [0, -1].
#
# Pbt::PropertyFailure:
# Property failed after 23 test(s)
# seed: 43738985293126714007411539287084402325
# counterexample: [0, -1]
# Shrunk 40 time(s)
# Got RuntimeError: Sort algorithm is wrong.
The above snippet is very simple but contains the basic components.
Pbt.assert is the runner. The runner interprets and executes the given property. Pbt.assert takes a property and runs it multiple times. If the property fails, it tries to shrink the input that caused the failure.
The snippet above declared a property by calling Pbt.property. The property describes the following:
predicate) enclosed by do endArbitraryThe predicate block is a function that directly asserts, taking values generated by Arbitrary as input.
Arbitrary generates random values. It is also responsible for shrinking those values if asked to shrink a failed value as input.
Here, we used only one type of arbitrary, Pbt.integer. There are many other built-in arbitraries, and you can create a variety of inputs by combining existing ones.
In PBT, If a test fails, it attempts to shrink the case that caused the failure into a form that is easier for humans to understand. In other words, instead of stopping the test itself the first time it fails and reporting the failed value, it tries to find the minimal value that causes the error.
When there is a test that fails when given an even number, a counterexample of [0, -1] is simpler and easier to understand than any complex example like [-897860, -930517, 577817, -16302, 310864, 856411, -304517, 86613, -78231].
There are many built-in arbitraries in Pbt. You can use them to generate random values for your tests. Here are some representative arbitraries.
rng = Random.new
Pbt.integer.generate(rng) # => 42
Pbt.integer(min: -1, max: 8).generate(rng) # => Integer between -1 and 8
Pbt.symbol.generate(rng) # => :atq
Pbt.ascii_char.generate(rng) # => "a"
Pbt.ascii_string.generate(rng) # => "aagjZfao"
Pbt.boolean.generate(rng) # => true or false
Pbt.constant(42).generate(rng) # => 42 always
rng = Random.new
Pbt.array(Pbt.integer).generate(rng) # => [121, -13141, 9825]
Pbt.array(Pbt.integer, max: 1, empty: true).generate(rng) # => [] or [42] etc.
Pbt.tuple(Pbt.symbol, Pbt.integer).generate(rng) # => [:atq, 42]
Pbt.fixed_hash(x: Pbt.symbol, y: Pbt.integer).generate(rng) # => {x: :atq, y: 42}
Pbt.hash(Pbt.symbol, Pbt.integer).generate(rng) # => {atq: 121, ygab: -1142}
Pbt.one_of(:a, 1, 0.1).generate(rng) # => :a or 1 or 0.1
See ArbitraryMethods module for more details.
Once a test fails it's time to debug. Pbt provides some features to help you debug.
When a test fails, you'll see a message like below.
Pbt::PropertyFailure:
Property failed after 23 test(s)
seed: 43738985293126714007411539287084402325
counterexample: [0, -1]
Shrunk 40 time(s)
Got RuntimeError: Sort algorithm is wrong.
# and backtraces
You can reproduce the failure by passing the seed to Pbt.assert.
Pbt.assert(seed: 43738985293126714007411539287084402325) do
Pbt.property(Pbt.array(Pbt.integer)) do |number|
# your test
end
end
You may want to know which values pass and which values fail. You can enable verbose mode by passing verbose: true to Pbt.assert.
Pbt.assert(verbose: true) do
Pbt.property(Pbt.array(Pbt.integer)) do |numbers|
# your failed test
end
end
The verbose mode prints the results of each tested values.
Encountered failures were:
- [-897860, -930517, 577817, -16302, 310864, 856411, -304517, 86613, -78231]
- [310864, 856411, -304517, 86613, -78231]
- [-304517, 86613, -78231]
(snipped for README)
- [0, -3]
- [0, -2]
- [0, -1]
Execution summary:
. × [-897860, -930517, 577817, -16302, 310864, 856411, -304517, 86613, -78231]
. . √ [-897860, -930517, 577817, -16302, 310864]
. . √ [-930517, 577817, -16302, 310864, 856411]
. . √ [577817, -16302, 310864, 856411, -304517]
. . √ [-16302, 310864, 856411, -304517, 86613]
. . × [310864, 856411, -304517, 86613, -78231]
(snipped for README)
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . √ [-2]
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . √ []
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . × [0, -1]
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . √ [0]
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . √ [-1]
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . √ []
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . √ [0, 0]
You can configure Pbt by calling Pbt.configure before running tests.
Pbt.configure do |config|
# Whether to print verbose output. Default is `false`.
config.verbose = false
# The concurrency method to use. `:ractor`, `:thread`, `:process` and `:none` are supported. Default is `:none`.
config.worker = :none
# The number of runs to perform. Default is `100`.
config.num_runs = 100
# The seed to use for random number generation.
# It's useful to reproduce failed test with the seed you'd pick up from failure messages. Default is a random seed.
config.seed = 42
# Whether to report exceptions in threads.
# It's useful to suppress error logs on Ractor that reports many errors. Default is `false`.
config.thread_report_on_exception = false
# Whether to allow RSpec expectation and matchers in Ractor. It's quite experimental! Default is `false`.
config.experimental_ractor_rspec_integration = false
end
Or, you can pass the configuration to Pbt.assert as an argument.
Pbt.assert(num_runs: 100, seed: 42) do
# ...
end
One of the key features of Pbt is its ability to rapidly execute test cases in parallel or concurrently, using a large number of values (by default, 100) generated by Arbitrary.
For concurrent processing, you can specify any of the three workers—:ractor, :process, or :thread—using the worker option. Alternatively, choose :none for serial execution.
Pbt supports 3 concurrency methods and 1 sequential one. You can choose one of them by setting the worker option.
Be aware that the performance of each method depends on the test subject. For example, if the test subject is CPU-bound, :ractor may be the best choice. Otherwise, :none shall be the best choice for most cases. See benchmarks.
:ractor worker is useful for test cases that are CPU-bound. But it's experimental and has some limitations as described below. If you encounter any issues due to those limitations, consider using :process as workers whose benchmark is the most similar to :ractor.
Pbt.assert(worker: :ractor) do
Pbt.property(Pbt.integer) do |n|
# ...
end
end
Please note that Ractor support is an experimental feature of this gem. Due to Ractor's limitations, you may encounter some issues when using it.
For example, you cannot access anything out of block.
a = 1
Pbt.assert(worker: :ractor) do
Pbt.property(Pbt.integer) do |n|
# You cannot access `a` here because this block is executed in a Ractor and it doesn't allow implicit sharing of objects.
a + n # => Ractor::RemoteError (can not share object between ractors)
end
end
You cannot use any methods provided by test frameworks like expect or assert because they are not available in a Ractor.
it do
Pbt.assert(worker: :ractor) do
Pbt.property(Pbt.integer) do |n|
# This is not possible because `self` if a Ractor here.
expect(n).to be_an(Integer) # => Ractor::RemoteError (cause by NoMethodError for `expect` or `be_an`)
end
end
end
If you're a challenger, you can enable the experimental feature to allow using RSpec expectations and matchers in Ractor. It works but it's quite experimental and could cause unexpected behaviors.
Please note that this feature depends on prism gem. If you use Ruby 3.2 or prior, you need to install the gem by yourself.
it do
Pbt.assert(worker: :ractor, experimental_ractor_rspec_integration: true) do
Pbt.property(Pbt.integer) do |n|
# Some RSpec expectations and matchers are available in Ractor by hack.
# Other features like `let`, `subject`, `before`, `after` that access out of block are still not available.
expect(n).to be_an(Integer)
end
end
end
If you'd like to run test cases that are CPU-bound and :ractor is not available, :process becomes a good choice.
Pbt.assert(worker: :process) do
Pbt.property(Pbt.integer) do |n|
# ...
end
end
If you want to use :process, you need to install the parallel gem.
You may not need to run test cases with multi-threads.
Pbt.assert(worker: :thread) do
Pbt.property(Pbt.integer) do |n|
# ...
end
end
If you want to use :thread, you need to install the parallel gem.
For most cases, :none is the best choice. It runs tests sequentially (without parallelism) but most test cases finishes within a reasonable time.
Pbt.assert(worker: :none) do
Pbt.property(Pbt.integer) do |n|
# ...
end
end
Once this project finishes the following, we will release v1.0.0.
expect, assert to a Ractor.bin/setup
bundle exec rake # Run tests and lint at once
bundle exec rspec
bundle exec rake standard:fix
Bug reports and pull requests are welcome on GitHub at https://github.com/ohbarye/pbt. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the code of conduct.
The gem is available as open source under the terms of the MIT License.
This project draws a lot of inspiration from other testing tools, namely
Everyone interacting in the Pbt project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the code of conduct.
FAQs
Unknown package
We found that pbt demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
PyPI confirms no security flaws were exploited in the Ultralytics supply chain attack and highlights improvements for safer package publishing.
Security News
Research
The Socket Research Team breaks down a malicious wrapper package that uses obfuscation to harvest credentials and exfiltrate sensitive data.
Research
Security News
Attackers used a malicious npm package typosquatting a popular ESLint plugin to steal sensitive data, execute commands, and exploit developer systems.