{"id":68,"date":"2016-09-30T05:15:29","date_gmt":"2016-09-29T22:15:29","guid":{"rendered":"http:\/\/arikuncoro.xyz\/wordpress\/?p=68"},"modified":"2020-12-10T06:53:50","modified_gmt":"2020-12-09T23:53:50","slug":"comparing-the-machine-learning-methods-with-roc-curve","status":"publish","type":"post","link":"https:\/\/arikuncoro.xyz\/blog\/data-science\/data-visualization\/comparing-the-machine-learning-methods-with-roc-curve\/","title":{"rendered":"Comparing The Machine Learning Methods with ROC Curve"},"content":{"rendered":"\n<p>Yesterday I continued a <a href=\"http:\/\/www.datacamp.com\">Datacamp<\/a> online course named <strong>Introduction to Machine Learning<\/strong>. Frankly, this course is very useful to strengthen my understanding in machine learning! Plus, I am a big fan of R! The more you repeat the course, the more you understand the meaning of it. Well, the topic was about &#8220;comparing the methods&#8221;. It is part of chapter 3 &#8211; Classification topic, precisely at the end of the chapter. It says that the powerful tool to compare the machine learning methods, especially classification, is <strong>ROC Curve<\/strong>. FYI, out of the record, this ROC curve analysis was also requested by the one of the client. \ud83d\ude09<\/p>\n\n\n\n<div class=\"wp-block-group\"><div class=\"wp-block-group__inner-container is-layout-flow wp-block-group-is-layout-flow\">\n<p><strong>What is ROC?<\/strong><\/p>\n\n\n\n<p>ROC stands for Receiver Operating Characteristic. In statistics, it is a graphical plot that illustrates the performance of a binary classifier system as its discrimination threshold is varied.&nbsp; Electrical engineers and radar engineers during World War II firstly developed the ROC curve for detecting objects of enemy, then soon used by psychologist to account for perceptual detection of stimuli. At this point, ROC analysis has been used in medicine, radiology, biometrics, machine learning, and data mining research.&nbsp;(Source: <a href=\"https:\/\/en.wikipedia.org\/wiki\/Receiver_operating_characteristic\">here<\/a>).<\/p>\n\n\n\n<p>The sample of ROC curve is illustrated in the Figure 1. The horizontal axis represents the false-positive rate (FPR), while vertical axis represents the true-positive rate (TPR).&nbsp;The true-positive rate is also known as sensitivity, recall or <i>probability of detection<span style=\"font-size: 12px; line-height: 0px;\">&nbsp;<\/span><\/i>in&nbsp;machine learning. The false-positive rate is also known as the fall-out or <i>probability of false alarm<span style=\"font-size: 12px; line-height: 0px;\">&nbsp;<\/span><\/i>and can be calculated as (1 \u2212specificity).<\/p>\n\n\n\n<p class=\"has-text-align-center\"><img decoding=\"async\" src=\"https:\/\/upload.wikimedia.org\/wikipedia\/commons\/6\/6b\/Roccurves.png\" alt=\"ROC Curve - Source: Wikipedia\"><br>Figure 1. ROC Curve &#8211; Source: <a href=\"https:\/\/upload.wikimedia.org\/wikipedia\/commons\/6\/6b\/Roccurves.png\">Wikipedia<\/a>.<\/p>\n\n\n\n<p><strong>How to create this curve in R?<\/strong><\/p>\n\n\n\n<p>You need:<\/p>\n\n\n\n<ul><li>Classifier that&nbsp;outputs probabilities<\/li><li>ROCR Package installed<\/li><\/ul>\n\n\n\n<p>Suppose that you have a data set called <strong>adult<\/strong> that can be downloaded <a href=\"https:\/\/archive.ics.uci.edu\/ml\/machine-learning-databases\/adult\/\">here<\/a> from UCIMLR. It&nbsp;is a medium sized dataset about the income of people given a set of features like education, race, sex, and so on. Each observation is labeled with 1 or 0: 1 means the observation has annual income equal or above $50,000, 0 means the observation has an annual income lower than $50,000. This label information is stored in the income variable. Then data split into train and test. Upon splitting, you can train the data using a method e.g. decision tree (rpart), predict the test data with &#8220;predict&#8221; function and argument type=&#8221;prob&#8221;, and aha&#8230; see the complete R code below.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted lang:r decode:true\">set.seed(1)\n\n# Build a model using decision tree: tree\ntree &lt;- rpart(income ~ ., train, method = \"class\")\n\n# Predict probability values using the model: all_probs\nall_probs &lt;- predict(tree,test,type=\"prob\")\n\n# Print out all_probs\nall_probs\n\n# Select second column of all_probs: probs\nprobs &lt;- all_probs[,2]\n\n# Load the ROCR library\nlibrary(ROCR)\n\n# Make a prediction object: pred\npred &lt;- prediction(probs,test$income)\n\n# Make a performance object: perf\nperf &lt;- performance(pred,\"tpr\",\"fpr\")\n\n# Plot this curve\nplot(perf)<\/pre>\n\n\n\n<p>The plot result is as follow:<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter\"><a href=\"http:\/\/arikuncoro.xyz\/blog\/wp-content\/uploads\/2016\/09\/roc-fig1.png\"><img decoding=\"async\" src=\"http:\/\/arikuncoro.xyz\/blog\/wp-content\/uploads\/2016\/09\/roc-fig1-300x244.png\" alt=\"ROC result of DT\" class=\"wp-image-71\"\/><\/a><figcaption>Figure 2. ROC result of Decision Tree<\/figcaption><\/figure><\/div>\n\n\n\n<p><strong>How to interpret the result of ROC?<\/strong><\/p>\n\n\n\n<p>Basically, the closer the curve to the upper left corner, the better the classifier. In other words, the &#8220;area under curve&#8221; should be closed to maximum value, which is 1. We can do comparison of performance based on ROC curve of two methods which are Decision Tree (DT) and K-Nearest Neighbor K-NN as seen in Figure 3. It shows that the DT method that represented by red line outperforms K-NN that represented by green line.<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter\"><a href=\"http:\/\/arikuncoro.xyz\/blog\/wp-content\/uploads\/2016\/09\/roc-fig2.png\"><img decoding=\"async\" src=\"http:\/\/arikuncoro.xyz\/blog\/wp-content\/uploads\/2016\/09\/roc-fig2-300x244.png\" alt=\"ROC Result of DT and KNN\" class=\"wp-image-72\"\/><\/a><figcaption>Figure 3. ROC Result of Decision Tree and K-Nearest Neighbor<\/figcaption><\/figure><\/div>\n\n\n\n<p>The R Code to draw Figure 3 is represented by the following code:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted lang:r decode:true\"># Load the ROCR library\nlibrary(ROCR)\n\n# Make the prediction objects for both models: pred_t, pred_k\n# probs_t is the result of positive prediction of Decision Tree Model \n# probs_k is the result of positive prediction of K-Nearest Neighbor\npred_t &lt;- prediction(probs_t, test$spam)\npred_k &lt;- prediction(probs_k, test$spam)\n\n# Make the performance objects for both models: perf_t, perf_k\nperf_t &lt;- performance(pred_t,\"tpr\",\"fpr\")\nperf_k &lt;- performance(pred_k,\"tpr\",\"fpr\")\n\n# Draw the ROC lines using draw_roc_lines()\ndraw_roc_lines(perf_t,perf_k)<\/pre>\n\n\n\n<p>Area under curve (AUC) parameter can also be calculated by running this command below. It shows that the AUC of DT is 5% greater than K-NN.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted lang:r decode:true\"># Make the performance object of pred_t and pred_k: auc_t and auc_k\nauc_t &lt;- performance(pred_t,\"auc\")\nauc_k &lt;- performance(pred_k,\"auc\")\n\n# Print the AUC value\nauc_t@y.values[[1]] # AUC result is 0.9504336\nauc_k@y.values[[1]] # AUC result is 0.9076067\n<\/pre>\n\n\n\n<p><strong>Summary<\/strong><\/p>\n\n\n\n<ul><li>ROC (Receiver Operator Characteristic) Curve is a very powerful performance measure.<\/li><li>It is used for binomial classification.<\/li><li>ROCR is a great package to be used in R for drawing ROC curve<\/li><li>The closer the curve to the upper left of area, the better the classifier.<\/li><li>The good classifier has big area under curve.<\/li><\/ul>\n<\/div><\/div>\n","protected":false},"excerpt":{"rendered":"<p>Yesterday I continued a Datacamp online course named Introduction to Machine Learning. Frankly, this course is very useful to strengthen &#8230;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[85,83],"tags":[86,87,89,99,102,103,110,111],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/arikuncoro.xyz\/blog\/wp-json\/wp\/v2\/posts\/68"}],"collection":[{"href":"https:\/\/arikuncoro.xyz\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/arikuncoro.xyz\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/arikuncoro.xyz\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/arikuncoro.xyz\/blog\/wp-json\/wp\/v2\/comments?post=68"}],"version-history":[{"count":3,"href":"https:\/\/arikuncoro.xyz\/blog\/wp-json\/wp\/v2\/posts\/68\/revisions"}],"predecessor-version":[{"id":937,"href":"https:\/\/arikuncoro.xyz\/blog\/wp-json\/wp\/v2\/posts\/68\/revisions\/937"}],"wp:attachment":[{"href":"https:\/\/arikuncoro.xyz\/blog\/wp-json\/wp\/v2\/media?parent=68"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/arikuncoro.xyz\/blog\/wp-json\/wp\/v2\/categories?post=68"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/arikuncoro.xyz\/blog\/wp-json\/wp\/v2\/tags?post=68"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}