pythagora

Pythagora is on a mission to make automated tests
🤖 fully autonomous 🤖

Just run one command and watch the tests being created with GPT-4

The following details are for generating unit tests. To view the docs on how to generate integration tests, click here.

🏃💨️ Quickstart

To install Pythagora run:

npm i pythagora --save-dev

Then, add your API key and you're ready to get tests generated. After that, just run the following command from the root directory of your repo:

npx pythagora --unit-tests --func <FUNCTION_NAME>

Where <FUNCTION_NAME> is the name of the function you want to generate unit tests for. Just make sure that your function is exported from a file. You can see other options like generating tests for multiple files or folders below in the Options section.

If you wish to expand your current test suite with more tests to get better code coverage you can run:

npx pythagora --expand-unit-tests --path <PATH_TO_YOUR_TEST_SUITE>

for more details on expanding existing tests see below in the Expanding existing tests section.

NOTE: on Windows make sure to run all commands using Git Bash and not Power Shell or anything similiar

🎞 Demo

Here are some demo videos that can help you get started.

Pythagora Unit Tests Demo (2 min)

🔎 Examples

Here are examples of open sourced repositories that we forked and created tests with Pythagora so you can easily see it in action.

• Lodash

  • 📝 1604 tests generated
  • 🐞 11 bugs found (1 edge case and 10 bugs)
  • ⏳️ 4 hour run time

  

• node-fs-extra

  • 📝 98 tests generated
  • 🐞 2 bugs found
  • ⏳️ 30 minutes run time

  

🔬 How does it work?

When Pythagora generates unit tests, it uses the following approach:

1. Find the function you want to test
2. Find all the functions that are called from within that function

• This is done with AST (Abstract Syntax Tree) parsing

1. Send the function you want to test and all the related functions to the Pythagora server which then generates the unit tests with GPT-4

• the Pythagora server is open sourced as well here
• You can find the prompts in this folder on the Pythagora server

📈 Expand existing tests

If you already have generated tests for your codebase but you just want to increase your code coverage or cover more edge cases, simply run:

npx pythagora --expand-unit-tests --path <PATH_TO_YOUR_TEST_SUITE>

When running command PATH_TO_YOUR_TEST_SUITE can be path to a single test file or to a folder and all test files inside of that folder will be processed and expanded.

That's all, enjoy your new code coverage!

📖 Options

• To generate unit tests for one single function, run:

  npx pythagora --unit-tests --func <FUNCTION_NAME>
  

• To generate unit tests for one single function in a specific file, run:

  npx pythagora --unit-tests --func <FUNCTION_NAME> --path ./path/to/file.js
  

• To generate unit tests for all functions in a file, run:

  npx pythagora --unit-tests --path ./path/to/file.js
  

• To generate unit tests for all functions in all files in a folder, run:

  npx pythagora --unit-tests --path ./path/to/folder/
  

⚙️ Config

Pythagora uses GPT-4 to generate tests so you either need to have OpenAI API Key or Pythagora API Key. You can get your Pythagora API Key here or OpenAI API Key here. Once you have it, add it to Pythagora with:

npx pythagora --config --pythagora-api-key <API_KEY>

or

npx pythagora --config --openai-api-key <API_KEY>

▶️ How to run unit tests

To run the generated tests, you can simply run

npx jest ./pythagora_tests/

or to run tests from a specific file or a folder, run npx jest <PATH_TO_FILE_OR_FOLDER>. Currently, Pythagora supports only generating Jest tests but if you would like it to generate tests in other frameworks, let us know at [e-mail].

📌️ Notes

• The best unit tests that Pythagora generates are the ones that are standalone functions (eg. helpers). Basically, the parts of the code that actually can be unit tested. For example, take a look at this Pythagora file - it contains helper functions that are a perfect candidate for unit tests. When we ran npx pythagora --unit-tests --path ./src/utils/common.js - it generated 145 tests from which only 17 failed. What is amazing is that only 6 tests failed because they were incorrectly written and the other 11 tests caught bugs in the code itself. You can view these tests here.

• We don't store any of your code on our servers. However, the code is being sent to GPT and hence OpenAI. Here is their privacy policy.

• a function you want to generate tests for needs to be exported from the file. For example, if you have a file like this:

  function mongoObjToJson(originalObj) {
      ...
  }
  
  module.exports = {
      mongoObjToJson
  };
  

  Then, to generate unit tests for the mongoObjToJson function, you can run:

  npx pythagora --unit-tests --func mongoObjToJson
  

🤔️ FAQ

• How accurate are these tests?
  • The best unit tests that Pythagora generates are the ones that are standalone functions. Basically, the parts of the code that actually can be unit tested. For example, take a look at this Pythagora file - it contains helper functions that are a perfect candidate for unit tests. When we ran npx pythagora --unit-tests --path ./src/utils/common.js - it generated 145 tests from which only 17 failed. What is amazing is that only 6 tests failed because they were incorrectly written and the other 11 tests caught bugs in the code itself. You can view these tests here.
  • Here are a couple of observations we've made while testing Pythagora:
    1. It does a great job at testing edge cases. For many repos we created tests for, the tests found bugs right away by testing edge cases.
    2. It works best for testing standalone helper functions. For example, we tried generating tests for the Lodash repo and it create 1000 tests from which only 40 needed additional review. For other, non standalone functions, we're planning to combine recordings from integration tests to generate proper mocks so that should expand Pythagora's test palette.
    3. It's definitely not perfect but the tests it created I wanted to keep and commit them. So, I encourage you to try it out and see how it works for you. If you do that, please let us know via email or Discord. We're super excited to hear how it went for you.
       
       
• Should I review generated tests?
  • Absolutely. As mentioned above, some tests might be incorrectly written so it's best for you to review all tests before committing them. Nevertheless, I think this will save you a lot of time and will help you think about your code in a different way.
    
    
• Tests help me think about my code - I don't want to generate them automatically
  • That's the best thing about Pythagora - it actually does help you think about the code. Just, you don't need to spend time writing tests. This happened to us, who created Pythagora - we coded it as fast as possible but when we added unit test generation, we realized that it cannot create tests for some functions. So, we refactored the code and made it more modular so that unit tests can be generated for it.

🏁 Alpha version

This is an alpha version of Pythagora. To get an update about the beta release or to give a suggestion on tech (framework / database) you want Pythagora to support you can 👉 add your email / comment here 👈 .

🔗 Connect with us

💬 Join the discussion on our Discord server.

📨 Get updates on new features and beta release by adding your email here.

🌟 As an open source tool, it would mean the world to us if you starred the Pythagora repo 🌟