torch compiler bisect tool

[eellison]

🚀 The feature, motivation and pitch

Tool to bisect various parts of the torch.compile stack. As I imagining it, the tool can iteratively go through systems and subsystems. as in:

dynamo
aot_eager
aot_eager_with_decomp: decompositions
inductor: pattern matchings, lowerings, fusions, etc.

There would be a way to interface through CLI, and to do an in-process bisect with a callable. You might also want to bisect to find particular offending graph, not sure how that fits in. Happy to hear input or other suggestions.

cc @ezyang @msaroufim @bdhirsh @anijain2305 @chauhang

[ezyang]

My biggest problem with something that does this in process, automatically, is that I have zero confidence I could reliably ask a user with an fblearner flow to turn on this knob and get something useful. Like, we typically ask people to do an ablation when they're like "oh no, NE explosion". What exactly is your in process bisecter going to do?!

[eellison]

For limited iteration speed, probably best way is to bisect offending graph and then save inputs. It should also work out of process.

What is your current MO for NE explosion ?

[ezyang]

Pain 🤣

I don't really have more than "do the ablation" and then hand it to the owner of the appropriate area. If it's in Inductor, you can also try ablating optimizations to see if the NE explosion goes away.