Performance bugs are really annoying. Our tests are still passing! It works on our machine and in staging! It even works for most customers in production! And then someone reports the app hangs, and there's timeouts...

How would we know something's not correct? This is where observability tools come in: tools for monitoring, performance and debugging. But have you ever wondered how they work? These tools often require access to Ruby internals that regular Ruby libraries or applications don't have (or need).

To support these use-cases, the CRuby VM has a number of low-level APIs that can be used to know more about such things as the state of threads, the global VM lock, object creation, garbage collection, etc.

In this talk, I'll explore Ruby VM's low-level observability APIs: what they provide, how they can be used, lessons learned from using them, and how you can get started in building your own custom tools for performance bug investigations.