Theme 3 (Predictors) Comments
Comments that apply to theme 3. Be sure to read the comments for “all themes” as well.
There was no equivalent to this theme last year. You might want to look at theme 2 for other ideas since it has some similarities.
There is a lot of research on visualization tools for working with classifiers/regressors. You don’t have to look very hard to find something. Finding something good might be trickier. For examples from our group’s work see the Boxer system.
Specific number comments given to at least one student (these would appear as 3.X in your list). Again, the list given in the feedback may not be exhaustive (i.e., some of these might apply to you even if they weren’t listed in what was sent to you).
It is acceptable (but optional) to consider validation loss - it tells you something different than resubstitution (it’s a more realistic test of prediction, whereas the entire data-based model is more of a summary of the data set).
The idea of trying to see what additional variables (e.g. demographics) can be used to improve model performance is a good thing to try. Consider how to make sure this is a visualization / data presentation (or at least a human understanding of model) project rather than just a plain ML experiment.
Allowing the user to experiment with specific predictions (e.g., for a specific, hypothetical person) is a nice idea. But probably has limited visualization. and gets away from the spirit of the assignment (assessment over the testing set)… Specific exemplars may be good for showing how a model works.
Trying to assess the predictability of different variables is a good idea: it will require both an ML part (to figure out how to compare the variables) and also some way to present the results. Simply giving a “predictability score” per variable is not a rich enough visualization.
There may be many good ways to show a single, specific prediction (including one you mention) - but that isn’t the hard problem. The harder problem is to show sets of predictions (and to compare them to known truth, if available).
Interactively re-training the model may not be feasible (it might be time consuming given the data size, and it would require a pretty beefy backend). Focusing a set of pre-trained models is likely to be sufficiently challenging.
Helping the user understand a prediction using various approaches (I like the exemplar finding) is a good thing to do (even if is does conflict with #3 above)
Comparing between models trained on different subsets of the data is a good idea (but it might be hard to do well).
It seems like your proposal is very different than your partners (it’s rare that they are so different). Make sure you are on the same page with your goals.
Building and analyzing two models in parallel is more like two projects stapled together. Be sure there is common development of the visualizations.
Experimenting with using different sets of featurs is good. Any variable can serve as either input or output.
Simply showing standard metrics of classifier performance (even in a visual way) is unlikely to lead to a satisfying performance. On the other hand, standard metrics are good summaries of performance, so can be used to augment deeper summaries (e.g., give RMS error or F1 score in addition to more detailed visualizations).