Running a full test suite for a Ruby app on every change works at a small scale, but eventually becomes slow and expensive. Many existing tools that try to run only the tests affected by a change either skip too much and then miss test failures or add so much overhead that they are unusable in CI. To solve this I built a test impact analysis engine for the open-source Datadog test optimization gem. This tool now runs in production and saves engineering teams 150 000+ hours of CI time per year.

In this talk I’ll show how that work became a deep dive into the Ruby internals with some surprising findings about what coverage means in Ruby: turns out it is far from being only about the lines of code. In this talk I will tell you about a journey of building a high-performance library that tracks what is really covered by your tests and share the tricks I learned along the way.