Machine Learning Basics: Naive Bayes Algorithm -Part I
Many of us would have come across this term Bayes theorem in probability. The Naive Bayes algorithm is a supervised learning algorithm and is solely dependent on Bayes theorem. We are going to recap about what is Bayes theorem, how is it used in Naive Bayes algorithm, some common areas where this algorithm is used, its advantages and disadvantages.
What is Bayes Theorem?
Bayes theorem is based on conditional probability. Let’s consider a person plays 8 out of 10 days. So the probability he plays is 8/10, and the probability he doesn’t play is 2/10. But, if we want to know what is the probability that he plays when it rains, then it becomes a conditional probability P(play|rain). Bayes theorem gives a solution on finding the conditional probability.
If c is 'play' and X is 'when it rains'
P(play|when it rains)=P(when it rains|play)*P(play)/P(when it rains)Naive Bayes with 1 conditional variable
Let us consider the above table. If we want to predict if a person would play golf when it is raining or not, then we need to compute both the probabilities, P(Play Golf=Yes|Outlook=Rainy) and P(Play Golf=No|Outlook=Rainy). Then compare which of the two is greater and decide. When we plug the details into the Bayes theorem formula
P(Outlook=Rainy)=5/14
P(Play Golf = Yes)= 9/14
P(Play Golf = No)= 5/14
P(Outlook=Rainy|Play Golf=Yes)=2/9
P(Outlook=Rainy|Play Golf=No)=3/5P(Play Golf=Yes|Outlook=Rainy)= P(Outlook=Rainy|Play Golf=Yes)*P(Play Golf = Yes)/P(Outlook=Rainy)=2/9*9/14*14/5=2/5=0.4P(Play Golf=No|Outlook=Rainy)= P(Outlook=Rainy|Play Golf=No)*P(Play Golf = No)/P(Outlook=Rainy)=3/5*5/14*14/5=3/5=0.6Since 0.6>0.4, we would say that the person would not play golf when it is raining
Naive Bayes with 2 conditional variables
Let us consider the same table above. In the previous example we just considered one conditional variable ‘Outlook’. Now, let us consider 2 conditional variables- Outlook and Humidity. Now, our Bayes theorem would look like this
P(A|B,C)=P(B,C|A)*P(A)/P(B,C)
=P(B|A)*P(C|A)*P(A)/P(B)*P(C)We consider all the conditional variables to be conditionally independent
So, to predict if a person will play golf when it Overcast is Rainy and Humidity is Normal, we would have to compute some extra probabilities
P(Outlook=Rainy)=5/14
P(Humidity=Normal)=7/14
P(Play Golf = Yes)= 9/14
P(Play Golf = No)= 5/14
P(Outlook=Rainy|Play Golf=Yes)=2/9
P(Outlook=Rainy|Play Golf=No)=3/5
P(Humidity=Normal|Play Golf=Yes)=6/9
P(Humidity=Normal|Play Golf=No)=1/5P(Play Golf=Yes|Outlook=Rainy,Humidity=Normal)= P(Outlook=Rainy|Play Golf=Yes)*P(Humidity=Normal|Play Golf=Yes)*P(play Golf=Yes)/P(Outlook=Rainy)*P(Humidity=Normal)
=2/9*6/9*9/14*14/5*14/7 = 56/105P(Play Golf=No|Outlook=Rainy,Humidity=Normal)= P(Outlook=Rainy|Play Golf=No)*P(Humidity=Normal|Play Golf=No)*P(play Golf=No)/P(Outlook=Rainy)*P(Humidity=Normal)
=3/5*1/5*5/14*14/5*14/7 = 42/105Since 56/105>42/105, we would say that the person would play golf when it is raining and humidity is normal
In my next blog, I will be writing about using Naive Bayes algorithm for text data and going into some detail about Bernoulli Naive Bayes theorem.
