python-testdata

python-testdata

A simple package that generates data for tests.

testdata provides the basic Factory and DictFactory classes that generate content. it also provides many more specialized factories that provide extended functionality. every Factory instance knows how many elements its going to generate, this enables us to generate statistical results.

The DictFactory is especially useful if you want to generate data that you will later input to your NoSQL, Document based database

In addition, using the DictFactory and the DependentField factories allows us to create factorys that depend on the results of other factories. (see Examples for more information).

testdata isn't bound to a specifc database, but does include database specfic modules inside, like - extra.mongodb.py) but it will always be clean of database related dependencies.

Installation

pip install python-testdata

Examples

We integrate the awsome fake-factory package to generate data using FakeDataFactory, this allows us to generate all sorts of content like: * Names (First, last, full names) * companies * addresses * emails * urls * and much much more

lets create a very simple factory that generates Users:

import testdata

class Users(testdata.DictFactory):
    id = testdata.CountingFactory(10)
    firstname = testdata.FakeDataFactory('firstName')
    lastname = testdata.FakeDataFactory('lastName')
    address = testdata.FakeDataFactory('address')
    age = testdata.RandomInteger(10, 30) 
    gender = testdata.RandomSelection(['female', 'male'])

for user in Users().generate(10): # let say we only want 10 users
    print user
    # {'firstname': 'Toni', 'lastname': 'Schaden', 'gender': 'female', 'age': 18, 'address': '0641 Homenick Hills\nSouth Branson, RI 70388', 'id': 10}
    # {'firstname': 'Gene', 'lastname': 'Greenfelder', 'gender': 'male', 'age': 17, 'address': '292 Loy Lights Suite 328\nFritzfort, IN 73914', 'id': 11}
    # or more likely you'd want to insert them into your favorite database (MongoDB, ElasticSearch, ..)

When creating our own subclasses for DictFactory, we can make some fields dependent on other fields. for example:

class ExampleFactory(DictFactory):
    a = CountingFactory(10)
    b = ClonedField("a") # b will have the same value as field 'a'

for e in ExampleFactory().generate(100):
    print e

# {'a': 10, 'b': 10}
# {'a': 11, 'b': 11}
# ...

Lets say we want to generate something like events data, we want events to have a start time, and an end time that will be 20 minutes in the future. In addition, we want the event's start_time will be 12 minutes apart.

import testdata

EVENT_TYPES = ["USER_DISCONNECT", "USER_CONNECTED", "USER_LOGIN", "USER_LOGOUT"]
class EventsFactory(testdata.DictFactory):
    start_time = testdata.DateIntervalFactory(datetime.datetime.now(), datetime.timedelta(minutes=12))
    end_time = testdata.RelativeToDatetimeField("start_time", datetime.timedelta(minutes=20))
    event_code = testdata.RandomSelection(EVENT_TYPES)

for event in EventFactory().generate(100):
    print event
    # {'start_time': datetime.datetime(2013, 12, 23, 13, 37, 1, 591878), 'end_time': datetime.datetime(2013, 12, 23, 13, 57, 1, 591878), 'event_code': 'USER_CONNECTED'}
    # {'start_time': datetime.datetime(2013, 12, 23, 13, 49, 1, 591878), 'end_time': datetime.datetime(2013, 12, 23, 14, 9, 1, 591878), 'event_code': 'USER_LOGIN'}
    # {'start_time': datetime.datetime(2013, 12, 23, 14, 1, 1, 591878), 'end_time': datetime.datetime(2013, 12, 23, 14, 21, 1, 591878), 'event_code': 'USER_DISCONNECT'}

We also have factories that allow us to generate different data distributed by different percentage, for example, lets say we want to create a 'Job', that will have an assigned user field, a state field and a description field. We want the state to be 'pending' in 90% of dictionaries and 'error' in the rest of them. In addition, we want that if the 'state' field is 'error' the assigned user will be 'support', or else it should be 'admin'.

class Job(testdata.DictFactory):
    state = testdata.StatisticalValuesFactory([('pending', 90), ('error', 10)])
    assigned_user = testdata.ConditionalValueField('state', {'error': 'support'}, 'admin')
    description = testdata.RandomLengthStringFactory()

for i in Job().generate(10):
    print i
    # {'state': 'error', 'assigned_user': 'support', 'description': 'jUlyFByPxPdFlBPBfPaGaTPPuajFSHXKkyewzrQ'}
    # {'state': 'pending', 'assigned_user': 'admin', 'description': 'tOzkgmBBnxQZhSYEjVduyXGdLrtqeTZqRxmHNXbaJBfpdNxuLKWyTDxkCZgiZTLHeiKEswvIyDzAnuuOLtXmVWhjvazaOYuu'}
    # {'state': 'pending', 'assigned_user': 'admin', 'description': 'TIDVuvZRUBLLTtG'}
    # {'state': 'pending', 'assigned_user': 'admin', 'description': 'RgcSaFzmMrhwCAZjLofikmXJhtqkVOTsWHnqTXjgrxgzTKH'}
    # {'state': 'pending', 'assigned_user': 'admin', 'description': 'tLkSEkCbYDvlcDBDWUBGMmidEdOxeiLDBADDKnqGqWLnxUBqzOXFXnBxkiGTymuGNbUnmxyawzLGsiummCiwxNSw'}
    # {'state': 'pending', 'assigned_user': 'admin', 'description': 'tUyYLofuZpceaWYKkiRvksQLqFHGOiwACuPIvRxMIuftJPsObSqCBcrQnOkOhqAukfMwrY'}
    # {'state': 'pending', 'assigned_user': 'admin', 'description': 'JbFrUxrERMObfwhEtCQGcxEbimvoTFwJriSfRFLFkBpyemqEfqUCGKmVlgSlVoZrrnetEnLCgbfobFbTMQOZ'}
    # {'state': 'pending', 'assigned_user': 'admin', 'description': 'lqatAwdcQuMMOPiYdVMRyyQgEIzOlcoozijjdCfXsVoZnnTtQjPSGBFZQGSkPblJrTIYLAotiZoyYRFrlncevwuNcqfOmeXeCPD'}
    # {'state': 'pending', 'assigned_user': 'admin', 'description': 'VYxnhydWtIUFiOEPszVQHuxYBIUGDyAefZiPIgkWHCMmophiueXbixXtdwKQkuvWImuErMOOOcwevQHGApXkolhjAq'}
    # {'state': 'pending', 'assigned_user': 'admin', 'description': 'RcawgTkQggchdHppSyQxnbDdNxqkGqbQWnQMSlorqnAQLdAqyWnKtGpXaZuVdxcGQBImzVPQsYAbIFUIpqvDzwTDdRpleBrc'}

In version 1.0.5 we extended the DictFactory to support passing additional factories, or overriding existing factories, for a specific instance. Lets take for example our 'User' example from the begining.

import testdata
class Users(testdata.DictFactory):
    id = testdata.CountingFactory(10)
    firstname = testdata.FakeDataFactory('firstName')
    lastname = testdata.FakeDataFactory('lastName')
    address = testdata.FakeDataFactory('address')
    age = testdata.RandomInteger(10, 30) 
    gender = testdata.RandomSelection(['female', 'male'])

But lets override it so the 'firstname' always returns John, and make the age be a random integer between 40 and 60 and add an 'email' field.

for user in Users(firstname=testdata.Constant('John'), age=testdata.RandomInteger(40, 60), email=testdata.FakeDataFactory('email')).generate(10): # let say we only want 10 users
    print user
    #{'firstname': 'John', 'gender': 'male', 'age': 54, 'email': '[e-mail]', 'lastname': 'Willms', 'address': '245 Pfeffer Light Apt. 309\nEast Audieside, IN 11931', 'id': 10}
    #{'firstname': 'John', 'gender': 'male', 'age': 47, 'email': '[e-mail]', 'lastname': 'Ratke', 'address': '98710 Freddy Gateway\nDelilahborough, GU 50849', 'id': 11}
    #{'firstname': 'John', 'gender': 'male', 'age': 55, 'email': '[e-mail]', 'lastname': 'Cormier', 'address': '432 Block Locks Apt. 547\nNew Estel, NJ 54026', 'id': 12}
    # or more likely you'd want to insert them into your favorite database (MongoDB, ElasticSearch, ..)

Factories

See the Factorie's Docstrings for more examples and doctests.

Bases

Factory Class	Description
Factory	The base class of all the factories.
DictFactory	A very powerful base class. allows sub classing to create factories that generate dicts with a specific schema (see [Examples][#Examples]).
ListFactory	A factory that returns on each iteration a list of elements_per_list items returned from calls to the given factory.
Callable	Gets a callable object as an argument and returns the result of calling the object on every iteration
DependentCallable	Gets a callable object as an argument and returns the result of calling the object passing the defined fields as arguments on every iteration
ClonedField	A factory that copies the value of another factory.

Dates

Factory Class	Description
RandomDateFactory	Generates random dates (python's datetime) between 2 dates
DateIntervalFactory	Generates datetime objects starting from base while adding delta to it each iteration.
RelativeToDatetimeField	Generates datetime object relative to another datetime field, like if you have start_time which is a RandomDateFactory field, and want an end_time field that is always 15 minutes later.

And MUCH MUCH more..

Todos

• Add usage documentation for each factory (using doctest maybe?)
• Add more tests
• Add GeoLocationFactories to generates Location and distance related data (for example, random points near a central point).
• Add MORE Statistical Factories
• more ideas welcome!