Ceiling Analysis in Machine Learning?

Ceiling Analysis is a way to systematically find the weakest component of your system, and therefore optimising that weakest component would best serve your time to bring the greatest improvement to the overall system.

Image result for ml pipeline image
UC Berkeley AMPLab

Why it is important in case of deep learning?

Ceiling analysis is the process of manually overriding each component in your system to provide 100% accurate predictions with that component. Thereafter, you can observe the overall improvement of your deep learning system component by component.

For photo OCR example, this is what we might have:

ComponentAccuracyNote
Overall system72%
Text Detection (TD)89%Make TD 100% accurate
Character Segmentation (CS)90%Make TD and CS 100% accurate
Character Recognition (CR)100%Make TD, CS and CR 100% accurate

By applying ceiling analysis, we can easily tell that TD improves the most (17% improvement), while CS improves the least (1% improvement). So text detection (TD) is where we should improve first.

So we can break our pipeline of task and visualize literally one components dominance in overall accuracy of the system .

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s