Skip to content

Benchmarking! #387

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
theogf opened this issue Oct 19, 2021 · 1 comment
Open

Benchmarking! #387

theogf opened this issue Oct 19, 2021 · 1 comment

Comments

@theogf
Copy link
Member

theogf commented Oct 19, 2021

Related to #386, I would like to open the discussion for evaluating the performance of basic functions.
Here are some facts/ideas.

Existing tools

  • PkgBenchmark.jl: A nice tool to create a suite of benchmarks and has functions to create nice reports on variations of performance. You can even create a markdown report which could be posted on the PR
  • NanoSoldier.jl: The tool used by Julia to evaluate the performance of the language. I don't believe this can be adapted easily to our setup, but I did not checked the details
  • Github action benchmark: Given a benchmark output will also create a report and has the possibility to directly create comments

Potential issues

  • Benchmark is highly dependent on the machine used, if we use the Github clusters, we might get a large variance in the results depending on the time of the day etc...
  • We cannot benchmark everything, which means we need to restrict ourselves to maybe the most used functions/kernels etc
  • Adding benchmarks can be a lot of work, can we find a framework where adding new tests is smooth

Other ideas

  • Not all PRs are performance-critical, we should be able to call whatever tool we use only when needed? Maybe every time for master and at will for some PRs.
  • What do we want to benchmark? Only pairwise, kernelmatrix or also the performance of the gradients on them?
@theogf
Copy link
Member Author

theogf commented Oct 19, 2021

I additionally found https://github.com/tkf/BenchmarkCI.jl which seems to do exactly what we want!

This was referenced Nov 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant