Ceiling Analysis is a way to systematically find the weakest component of your system, and therefore optimising that weakest component would best serve your time to bring the greatest improvement to the overall system.
Why it is important in case of deep learning?
Ceiling analysis is the process of manually overriding each component in your system to provide 100% accurate predictions with that component. Thereafter, you can observe the overall improvement of your deep learning system component by component.
For photo OCR example, this is what we might have:
|Text Detection (TD)||89%||Make TD 100% accurate|
|Character Segmentation (CS)||90%||Make TD and CS 100% accurate|
|Character Recognition (CR)||100%||Make TD, CS and CR 100% accurate|
By applying ceiling analysis, we can easily tell that TD improves the most (17% improvement), while CS improves the least (1% improvement). So text detection (TD) is where we should improve first.
So we can break our pipeline of task and visualize literally one components dominance in overall accuracy of the system .