User:Mok Yee Man/sandbox

My Report {Topic}

Evaluation Methods (Machine Learning)

Introduction The evaluation methods in Machine Learning are to determine the usefulness of the learning hypotheses or the learning algorithms on various collections of data sets (Japkowicz, n.d.). To evaluate a machine learning model, it is commonly to hold out a sample of data that has been labeled with the target (ground truth) from the training datasource.

Objective The evaluation methods is used for the measurement and comparison on machine learning algorithm. There are different types of evaluation methods can be used;

The first one is Classification Accuracy.

Classification accuracy is the number of correct predictions made as a ratio of all predictions made.

It works well only if there are equal number of samples belonging to each class.

1*yRa2inzTnyASJOre93ep3g.gif

For example, consider that there are 98% samples of class A and 2% samples of class B in our training set. Then our model can easily get 98% training accuracy by simply predicting every training sample belonging to class A.

Here shows an example of calculating classification accuracy.

-RsuAgwKH9bzbJOTldMPgEbNiCT6LfpFKWuRTogHg9nuRTAFtalFkQklpb-C4WpUnA_uAB_tTwlp8-nFkUgNSmYiJliSi-lhXEEuK5VAtriF-A_BaMquXXh-Ger0V0puS29K16rR

The ratio is reported. This can be converted into a percentage by multiplying the value by 100, giving an accuracy score of approximately 77% accurate, which the accuracy is equal to 0.770 (Brownlee, 2016).

The second one is Logarithmic Loss.

Logarithmic Loss or Log Loss, works by penalising the false classifications. It works well for multi-class classification. When working with Log Loss, the classifier must assign probability to each class for all the samples. Suppose, there are N samples belonging to M classes, then the Log Loss is calculated as below :

1*dtpzlB_dNrmxDq7fGnJ4cg.gif

where,

y_ij, indicates whether sample i belongs to class j or not

p_ij, indicates the probability of sample i belonging to class j

Log Loss has no upper bound and it exists on the range [0, ∞). Log Loss nearer to 0 indicates higher accuracy, whereas if the Log Loss is away from 0 then it indicates lower accuracy.

The last one is Area Under Curve.

Area Under Curve(AUC) is used for binary classification problem. AUC of a classifier is equal to the probability that the classifier will rank a randomly chosen positive example higher than a randomly chosen negative example.

True Positive Rate (Sensitivity) : True Positive Rate is defined as TP/ (FN+TP). True Positive Rate corresponds to the proportion of positive data points that are correctly considered as positive, with respect to all positive data points.

1*yw4Y3D7nGNVza2EC2WrOfg.gif

False Positive Rate (Specificity) : False Positive Rate is defined as FP / (FP+TN). False Positive Rate corresponds to the proportion of negative data points that are mistakenly considered as positive, with respect to all negative data points.

1*eIIZS8FNvpxiEzHpFI3QZA.gifpEkorJjQqTfTqZDwVkqP07Ays9YqM2E62PSSvjAAOtoz_PsE7hWyeCYzttZj5UwLGcPfCOBsbjKUmtSfj47Q_3-wYxYyJ9czvi9KJurXayvaNmmzpWiVIrgCEQKODWoQCsiyUB9C

IZM6t7X-lp1MxzrjZG8ZEqFCFg2vn8JmVxg84ryT_i72HUFBV5-WdpvTy_ewBS6NZCtKLNwiGmGjeFRcIZUFjBkp498gnCE22Zz6gLaj-YNjPVE0rUEp5XQ9BLWhVgrSz78B-m0g

As evident, AUC has a range of [0, 1]. The greater the value, the better is the performance of our model (Mishra, n.d.).

Reference Mishra, A. (n.d.). Metrics to Evaluate your Machine Learning Algorithm. Retrieved from

https://towardsdatascience.com/metrics-to-evaluate-your-machine-learning-algorithm-f10ba6e38234

Brownlee, J. (2016, May 25). Metrics To Evaluate Machine Learning Algorithms in Python. Retrieved from

https://machinelearningmastery.com/metrics-evaluate-machine-learning-algorithms-python

Japkowicz, N. (n.d.). Why Question Machine Learning Evaluation Methods? Retrieved from

https://pdfs.semanticscholar.org/46ec/d8f4708764a9c5efb4916395f5d013b970cb.pdf