guess-testing

Guess Testing

Welcome to Guess Testing!

Ain't nobody got the time to write unit tests! But... everybody wants a 100% coverage! If you're not a QA person (or even if you do), this here solution is just PERFECT for you!

Test your code with Guess Testing! No more hard work searching for edge cases, possible exceptions, or immunity to weird inputs. Save your strength and let the CPU do the hard work.

Getting Started

First off clone this project, it is written for Python 3.8, but due to its basic requirements can work on many other versions.

Prerequisites

No additional packages are required for using Guess Testing, but for a pretty progress bar, rich is required. For the CLI, click is required.

pip install click rich

Installing

You can install Guess Testing from PyPI using pip!

pip install guess-testing

That's it!

Usage

Guess Testing is package that can be imported and used for many reasons, such as:

1. Finding the smallest set of parameters for getting a full coverage of a scope.
2. Finding the possible exceptions the code can throw and from where, and which arguments cause these behaviors.
3. Finding all the possible return values of a function, and which arguments cause them.
4. Any form of stress testing, analysing an unknown code, and many, many more cases.
5. Supports coverage checking for instructions as well, and not only line numbers, meaning 1 if False else 0 will be shown as not covered.
6. Generating random values to match specific typing.
7. Generate random typing.

As a python package, it is importable, like so:

import guess_testing

It also features a CLI, which can be used like so:

$ guess-testing generate-from-string 'List[int]' -n 2
[49533, 8829, 6404, 14376, 1608, 29969, -22302, 55508, 13601, 61492, 10087, -2318, -33220, -56423, 46858]
[-32488, 25782]
$ gstst generate-type -n 5  
None
str
List[str]
Union[Tuple[Dict[range, Tuple[None, ...]], bool, List[Set[complex]]], Union[Iterable[complex], None, Optional[Optional[str]], Union[Dict[str, None], Union[range]], bytes, complex, float, str]]
Tuple[bool, range, Tuple[Tuple[bool, ...], ...], Tuple[Tuple[bool, ...], Set[range], str, Iterable[Set[None]], Tuple[range, ...], Union[Optional[None], int], Tuple[Tuple[bytes, ...], Union[bool, bytes, int, range], List[complex], bytes, Iterable[None], float, None, Union[bool, bytes, complex, float, int, str]]], range]

All the options are listed under gstst --help.

Guess Testing offers two main features that can be used separately, guessing and generators:

• Guessing - The ability to guess values for a function until specific requirements are met.
• Generators - The ability to generate values using explicitly constructed generators (many generators are available), or using the factory and retrieving a generators that correlates to the type annotation specified, or a function, supports typing module type specifications.

Test Run

Guessing

Let's see an example of how Guess Testing can be of benefit:

import typing

from guess_testing.guess import Guesser, StopConditions


def e(a: typing.List[int]) -> str:
    if len(a) == 0:
        return 'no enough'
    if len(a) == 1:
        return 'still not enough'
    if a[0] == a[1]:
        return 'wow!'
    if a[0] % a[1] == 0:
        return 'great!!'
    if a[0] % a[1] == 1:
        return 'amazing!!!'
    return 'boo...'


guesser = Guesser(e)
guesser.guess(stop_conditions=StopConditions.FULL_COVERAGE, suppress_exceptions=ZeroDivisionError, pretty=True)
print(guesser.coverage)
print(guesser.exceptions)
print(guesser.return_values)

Now all that's left is running the code, let's see it in action!

The code in this example run can be found in Example E.

More examples are available here.

Now how about checking a code that cannot be really covered (from Example H)?

from guess_testing.guess import Guesser


def h(a: int) -> str:
    return 'a' if a == 666 else 'b'


guesser = Guesser(h, trace_opcodes=False)
guesser.guess(call_limit=10000)
print(f'Attempts: {guesser.attempts_number}, coverage: {guesser.coverage["coverage"]}.')

guesser = Guesser(h, trace_opcodes=True)
guesser.guess(call_limit=10000)
print(f'Attempts: {guesser.attempts_number}, coverage: {guesser.coverage["coverage"]}.')

And here is the output:

Attempts: 1, coverage: 100.0.
Attempts: 10000, coverage: 77.77777777777777.

Generators

Let's review two more examples using the generators' ability (taken from Example F):

import typing

from guess_testing.typing_generators_factory import TypingGeneratorFactory


def f(a: typing.List[int]) -> str:
    pass


generators = TypingGeneratorFactory.get_generators(f)
print(generators)

for _ in range(10):
    print(generators.keyword['a']())

Now the result:

ParametersGenerators(positional=[], var_positional=Iterable[None], keyword={'a': List[int]}, var_keyword=Dict[None, None])
[12405, -21511, -27079, 53831, 30705, 63145, -29037, -13356, 43919, -37399, -17513, -9125, 52108, -22998]
[-45673, -25582, 14730, -29403, 29326, 51133, 15940, -43782, -28558, 62275]
[15058, 24083, 49427, -60482, 34556, 28768, 47541, 34125, 18023, 63406, 55071, -5622, -50858, 64449, -56508, 29186]
[-1099, -62520, 32445, 11092, 40268, -16754, -9336, -60278]
[]
[18620, -11646, 8687, -51744, 2788, -64639]
[-20770, -8160, -43784, -12910, 32457]
[8820, 11288, 41938, -30188, -9662, -7796, 51550, 51373, -13318, -20437, 17889]
[-6548, -55926, 28237, -913, 7181, -10664, 49494, 42848, -55882, 65131]
[28287, -14622, 63845, 34029, 4292, -45424, -55058, -50316, 21225]

Here's another one, a bit more extreme (Example G):

from guess_testing.generators import AnyGenerator

a = AnyGenerator()
for _ in range(5):
    print(a())

for _ in range(5):
    print(a.generate_generator())

The output:

[<generator object IterableGenerator.__call__.<locals>.<genexpr> at 0x7fa0f005dcf0>, <generator object IterableGenerator.__call__.<locals>.<genexpr> at 0x7fa0e0008350>, <generator object IterableGenerator.__call__.<locals>.<genexpr> at 0x7fa0e00083c0>, <generator object IterableGenerator.__call__.<locals>.<genexpr> at 0x7fa0e0008510>, <generator object IterableGenerator.__call__.<locals>.<genexpr> at 0x7fa0e00089e0>, <generator object IterableGenerator.__call__.<locals>.<genexpr> at 0x7fa0e0008ac0>, <generator object IterableGenerator.__call__.<locals>.<genexpr> at 0x7fa0e0008f90>, <generator object IterableGenerator.__call__.<locals>.<genexpr> at 0x7fa0f00ca040>, <generator object IterableGenerator.__call__.<locals>.<genexpr> at 0x7fa0f00ca0b0>, <generator object IterableGenerator.__call__.<locals>.<genexpr> at 0x7fa0f00caeb0>, <generator object IterableGenerator.__call__.<locals>.<genexpr> at 0x7fa0e00fa890>, <generator object IterableGenerator.__call__.<locals>.<genexpr> at 0x7fa0e00fa900>, <generator object IterableGenerator.__call__.<locals>.<genexpr> at 0x7fa0e00fa970>, <generator object IterableGenerator.__call__.<locals>.<genexpr> at 0x7fa0e00fa9e0>]
[{None, 3407.2062877800927, (52160, range(210, -164, -8), None, (-42239.34660829524+31542.572884277804j), (48542.738550590744-11834.802271042106j), range(-127, -51, 4), (28303.036853390033+25426.586453325246j), False, 20651), (20846, range(140, 211, 9), None, (-9787.667116909128-36108.15226304931j), (-28326.935324770995-1654.824857357773j), range(-253, 99), (22136.88909872755+7094.531064717114j), False, -56200), 17108, 'JBygBHyd{-W8xu', 3356, -35619}, {b'#5qOb', (-20434, range(253, 178, -11), None, (-42678.645049940504-12401.42343568847j), (-31018.26686958586+9369.415989918038j), range(-69, 14, 15), (1549.5475654124602-64961.93558722615j), False, -2484), b'@9iL2Xx s@Zp[lk7M5ICsjbb,R', 37288, 'xSq6[W', (4166, range(15, -126, -11), None, (33654.207414087155+19426.691454051557j), (-38140.187482436755+62265.014593785716j), range(223, 127, -11), (-37088.19804402004+18390.61935634943j), False, 40409), 'jyg|m9Z]^&5zp&', b'S1[z_>\t]K$J;T.ZkB', "1'S", 13335.448226791937, 'S+it\\J</P', 'd1P#m&\tK7<ZF}(6k', '\tfyUX* )f\n5,fT|H2[Umm', b'W9ILoWpH#S&YJLH`+NiMEs,', (-20967, range(218, 157, -13), None, (-14123.759361549688+48866.764766850814j), (-5873.291129242614-60443.13133479848j), range(17, -88, -7), (-6320.727481127862+8443.489785797035j), False, 45913)}, set(), {b'', 'g&eB"i4F)k', 'RFZa(oS@Lg%Unke7#A>ljTk8A;vcK', 'h =t9S@d~^\t=v', b"z$'Jn #600P~ayK}>sUie\nTl7%o", 59824.09415612169, 39696, 50548.759432853345, '1[)NADERs9Kb6K*s#s6%|I$', "801v' )vyX~N2x|^=iZPsLxN3~$", 22429}, {'C2q9 *u*2_Fz;p3(2TfDitBmigD%', 49894.63728402753, 'Xh]', '.`F\\bT{Se^hA2[%1bP~17cD', -9587, b'EeTN&5VC~ 9pgxkb{/ewKZ\nY5si$z;a', 27761.176826549286, (42903, range(21, -37, -11), None, (48897.252595155674+59614.132939017785j), (44349.09291976501-10453.958254245692j), range(-29, 147, 6), (-58789.841937925914+34663.60135881053j), True, 42288), 'nb7o6W\t(JoRs8[pYjM\nfL8', 54041.97531317464, "\n'xI|d N&y997"}, {(-42538, range(175, 139, -7), None, (21153.650879510664+50674.202700177295j), (-15148.512103779402+26898.971946537786j), range(164, 238, 11), (12366.180111977039-46811.58799118672j), False, 12015), b'.^jH@MXQ{f0qOe5js)x/US2<=sXi(', 9382, '7n~rJB$W\n\nc<`GN>h+Z2E8nka5&_-!', 18.04357508927933, b'|Fs"x_G]CapGlO', b"E_.cZtfDWU}'lO{", -41195, (28699, range(251, 247, -3), None, (-16220.412941527-12576.39407433664j), (14566.187056888506-19117.171633524136j), range(87, -131, -4), (12192.628992346828-56600.26172259373j), False, -47525), 'Fs/(t7 ', '1i30E!bm2t`fx@$U%i\\|u#@)5=/aWh8'}, {15918.58817113802, (16532, range(253, 160, -11), None, (17493.227899860824-24740.113832680814j), (54029.24665775082-10901.170816058322j), range(70, 83, 10), (-56810.66409780257-18242.67334362249j), True, 42243), '76KQ0Ip', -61383, 52602, (41399, range(-114, -182, -1), None, (-12364.54894227213-60470.97724074805j), (-1585.9368784613325-54957.96565532785j), range(3, -95, -14), (-1261.6325920820673-42063.85639454314j), True, 31531), 23195, 'Kj8q%(R=.S1_8F^hFB0\\IKuCv*sb"'}]
(None, None, None, None)
{9537, -26806, -16213, -42408, -53, 42032, 25104, -64651, -47850, 24983, -42632, -56519, 19482, 18838, 52188, 23421}
-28099
Tuple[None, ...]
Tuple[Tuple[Dict[Optional[range], Union[bool, bytes]], ...], bool, complex, Tuple[Iterable[Tuple[None, ...]], ...], Tuple[Set[bytes], None, List[bool], int, range, complex, None, range], complex, List[List[bool]]]
str
int
None

Technologies and Capabilities

• Guess Testing is written in Python 3.8, but due to no dependencies and using only builtin capabilities should work on other versions just as well.
• Does not require any additional packages.
• Features a pretty progress bar for enhanced satisfaction.
• Very lightweight.
• Flexible guessing stop conditions, like full coverage (by lines and instructions), an exception is thrown, certain time has passed, call count limit is reached...
• Allows getting information by coverage, return values, and exceptions.
• Easily extendable.
• Contains a string representation for the generators that fits typing and the builtin python types.
• Features a CLI to enable actions straight from the terminal.
• What more can I say? It's small, standalone, and can actually be of use.

Documentation

The code is documented using Google Python Style Guide format.

I hope that it can answer whichever questions that may arise.

Contributing

Feel free to contact me if you have any comments, questions, ideas, and anything else you think I should be aware of. I would also love know to which edge cases Guess Testing helped you find, and what's the weirdest/most complicated type you generated.

This project is licensed under the terms of the MIT license.

Authors

• Uriya Harpeness

Acknowledgments

• I would like to thank my wife - Tohar Harpeness, my son - Amittai Harpeness, my parents, my computer, and my free time, for enabling me to work on this small project, it has been fun.

• I thank Typing for their simple and versatile usage, and for being easily parsed.

• I thank my previous experiences trying to debug a code that isn't mine and getting to all of its cases to better understand it, and not being able to do so easily. Which gave me the idea for this nice package to help others like me.