NextGen Scouting Evaluation in Baseball

Being an obsessive, I follow a lot of baseball news to catch news about the Giants, and thus am interested in all parts of a baseball operations.  I've been participating in, basically, a crowdsourcing initiative, and their method of evaluating predictions I think is something that could be used in evaluating scouting, so I thought I would discuss my thoughts.

ogc big picture thoughts

The project I've been assisting (by being a participant) is a prediction platform that allows people to make predictions about what is going to happen.  Who will win an election?  How much ice in a certain part of the North Pole regions?  How many earthquakes of 5+ magnitude in a month?  Will North Korea launch a missile?  How many barrels of oil will an OPEC country produce?  What is the price of a commodity at a certain date?  How many cases of a disease will happen in a country during a period of time?  Will this leader still be in his position in N months?

And participants make their prediction, and then modify their prediction over time, and there is a methodology of measuring how good a prediction they were making, based on their predictions, and relative to what the final correct answer is.  They allot points (+/-) based on whether you were correct, and how correct you were over time, and then people can be compared and ranked.  The methodology used is Brier Score.

NextGen Scouting Evaluation

Sounds exactly like scouting reports, right?  Experts making a prediction about a prospect (HOF, All-Star, solid regular, utility, borderline, career minor leaguer), adjusting their prediction over time, and eventually, there's an answer, though that could take up to 10 years to figure out (see Joe Panik).  Once a result is reached, then each scout's report (prediction) history can be evaluated as to how correct he/she was, and a score could be given. And it's not just your own team prospects, but prospects who were scouted in the amateur ranks, and who ended up with another team.

Of course, it could then take over 10 years to fully evaluate any scout using this method, just because you would need to take that long to determine the final value of all the prospects.  But given that the vast majority of prospects scouted end up either never drafted or out of professional baseball in 3-5 years, preliminary scoreboards could be set up based on how quickly they identified the ones who ended up not major leaguers, and for the ones who have a wide range early on (like Joe Panik), scenarios of potential value could be assigned (perhaps with a best guess estimate of the probabilities of each possibility), which would allow preliminary rating and ranking.

The usefulness of this tool would be to identify those scouts who are better at quickly identifying what type of player a prospect will be.  It's one thing to see the tools that the player has, but it's another to be able to identify who will be a major leaguer, and who will not, and how quickly.  The best scouts would then be that much more obvious to management, given more data points, and measurement of how well they did in forecasting the bad as well as the good.  

And the data for this type of assessment of prediction accuracy is locked up on file cabinets all over baseball, the data is on all the scouting sheets filed away on each prospect scouted, or, if modernized, already digitized for ease of access.  Once digitized, then it's a matter of cleaning up the data for this type of analysis, perhaps requiring additional research to find the missing pieces of data that is necessary for the assessment.

I don't think a team would want to base their whole assessment of a scout based on just this one parameter, but it would be another piece of the puzzle in figuring which scouts have the "magic touch" and which don't have a clue (in which case, would require a more extensive examination of the scout's performance history).  This could be another way baseball analytics can evolve baseball operations with new analytical tools.