classification - Machine Learning Manual Labeling Quality -

i have multi-label classification task.

there set of labels, when evaluate performance see in general labels can divided 2 groups, labels performance , labels bad performance , gap between them significant.

i looking approach how evaluate quality of manual labeling. know it's not trivial, sure can investigation. example, in labels see there set of attributes high weight characterize these labels , labels bad performance not see features.

what else can done in order see differences between labels , bad labels?

it hard give concrete advice without more details setup.

one method commonly used crowd-sourced data ask multiple people labels. if labels categorical in nature, labels selected several labelers used. if labels continuous averaged. need contemplate possibility labelers either maliciously adding noise or don't understand task.

you need careful, though. if labeling reasonable result of experiment telling attributes have not @ estimating label. so, may have description problem more problem quality of labels. these description problems common in nlp , computer vision, example describing objects of interest difficult.

if add more data , want accomplish , results of specific experiments add more specific advice.