@ghuser/github-contribs

github-contribs

List all GitHub repos a user has contributed to since the beginning of time:

• not just the last few months,
• not just the repos owned by the user or their organisations,
• simply all repos a user has ever pushed to.

$ github-contribs AurelienLourot
✔ Fetched first day at GitHub: 2015-04-04.
⚠ Be patient. The whole process might take up to an hour... Consider using --since and/or --until
✔ Fetched all commits and PRs. Consider using --issues to fetch issues as well.
35 repo(s) found:
AurelienLourot/lsankidb
reframejs/reframe
dracula/gitk
...

⇒ Advanced usage

Installation

$ sudo npm install -g @ghuser/github-contribs

Contributing

To the code

To run your local changes:

$ yarn install
$ ./cli.js --help

Contributors

Thanks goes to these wonderful people (emoji key):

Aurelien Lourot 💬 💻 📖	John Vandenberg 🐛 🤔	Jeaye Wilkerson 🐛	Hagar Shilo 🤔	Romuald Brillout 🤔

This project follows the all-contributors specification. Contributions of any kind welcome!

NOTE: if you should be on the list of contributors but we forgot you, don't be shy and let us know!

FAQ

How does it work?

Normally in order to retrieve all repositories a user has interacted with, one should query the GitHub Events API. Unfortunately it returns only the last 90 days, so we don't use it.

Instead we noticed that the "Contribution Activity" on the profile pages queries such URLs in the background:

• https://github.com/users/AurelienLourot/created_repositories?from=2018-05-17&to=2018-05-17
• https://github.com/users/AurelienLourot/created_pull_request_reviews?from=2018-05-17&to=2018-05-17
• https://github.com/users/AurelienLourot/created_issues?from=2018-07-10&to=2018-07-10

So we're doing the same :)

NOTES:

• It seems like created_issues URLs don't deliver "hot issues" (issues which received more comments than others):

  $ curl -s "https://github.com/users/AurelienLourot/created_issues?from=2015-09-23&to=2015-09-23"
  <div class="profile-rollup-content">
  </div>
  

  To get these, we also query the profile itself:

  $ curl -s "https://github.com/AurelienLourot?from=2015-09-23" | grep issues/
          <a class="text-gray-dark" href="/jfrog/build-info/issues/60">Publish properties aren&#39;t used by build-info-extractor-gradle?</a>
  

• In the past we used to get the pull requests from a created_pull_requests URL but this got removed. We now get the pull requests from the profile itself as well:

  $ curl -s "https://github.com/AurelienLourot?from=2017-08-28" | grep pull/
              <a href="/tt-gf/ant-ivy/pull/2" class="content-title no-underline">
  

• Same now with created_commits:

  $ curl -s "https://github.com/AurelienLourot?from=2017-08-27" | grep commits?
        <a href="/AurelienLourot/mybeir.ut/commits?author=AurelienLourot&amp;since=2017-08-27&amp;until=2017-08-28" class="f6 muted-link ml-1">
  

Why is it so slow?

We hit a rate limit. And since it's not an official API, we can't use a token to raise the limit.

NOTE: the rate limit seems to be 40 requests / minute / endpoint / IP. Thus even if crawling a single user takes about 3 hours on a single machine, crawling many users in parallel on that same machine should still take about 3 hours.

Isn't it likely to break?

Yes, it is since that interface isn't public. We're monitoring it¹ and will react as fast as we can when it breaks.

¹ ghuser.io runs this tool every day.

github-contribs missed some of my commits. Why?

github-contribs can only discover commits considered as GitHub contributions, i.e. commits that would also appear in the activity section of your GitHub profile. For example it doesn't discover commits in forks.

Changelog

2.2.3 (2018-10-20):

• ghuser-io/ghuser.io#172 Fix after GitHub's interface has changed. The created_commits "endpoint" is gone.

2.2.2 (2018-10-13):

• ghuser-io/ghuser.io#172 Fix after GitHub's interface has changed. The created_pull_requests "endpoint" is gone.

2.2.1 (2018-09-15):

• Documentation improvements.

2.2.0 (2018-08-09):

• #1 - Added --issues flag.

2.1.0 (2018-06-25):

• Exported helper function prevDay().

2.0.0 (2018-06-25):

• Exported helper functions stringToDate() and dateToString().

1.0.0 (2018-06-11):

• Support for passing a GitHub API key.

0.0.2 (2018-05-29):

• Cosmetic improvements to the npm page.

0.0.1 (2018-05-29):

• Initial version.

Similar/Related projects

• GitHub-contributions: uses a different technique. It fetches all the user's pull requests from the official API. This is clever but will miss the repos to which the user has pushed directly without pull request.
• gharchive.org: records all GitHub events for all users. In theory it should be possible to replace our implementation by queries to this huge database.