A reader wrote:
I read your EPV paper, and thought it's a really cool idea. But I think they way you guys approached it leaves a rather fundamental problem unresolved. Say, player A is amazingly good. Giving him the ball is really valuable. Who gets the credit for passing to him? If your model realizes that player A is very good, then the credit goes to the person who gave him the pass. If yours model doesn't know how good A is, then the credit goes to A (because after receiving the pass, he will of course do something very impressive, like score or whatever). This means that EPV, as defined, isn't a very robust metric of the contribution of individual players.
You can reformulate the question to make EPV a measure of the decision quality of each player (so explicitly ignore things like movement, shots, etc. - only focusing on decision making). Then I think you could build a more consistent model, since the credit will only be allocated for instantaneous decisions. However, this is a quite different approach to the one you took.
The other concern I had (due to my inability to read complex math) is how you define the "benchmark" for the decision. Specifically, your EPV, theoretically, should allow you to identify the "best" decision in each case - but clearly you're not using this as the benchmark (otherwise, any decision by the players would change EPV only by 0 or less). Instead, I assume, you model imperfect player decision process, and compare whatever "average" decision you think a typical player would make to the actual decision made. I'm not sure how exactly you defined the "typical" decision, but it seems a very fragile part of the model.
In general, we appreciate the reader's critical eye, especially the questions about how robust or fragile our metrics are to particular choices. Long story short, we don't think these represent fragility, so much as they represent different questions that you might try to answer about basketball. We made some very specific choices about questions of exactly this type while designing the methodology, so this email gives us a great opportunity to talk about some of the deeper thought process that went into the EPV paper.