This document provides an overview of point estimation methods, including maximum likelihood estimation and the method of moments. It begins with an introduction to statistical inference and the theory of estimation. Point estimation is defined as using sample data to calculate a single value as the best estimate of an unknown population parameter. Maximum likelihood estimation maximizes the likelihood function to find the parameter values that make the observed sample data most probable. The method of moments equates sample moments to theoretical moments to derive parameter estimates. Examples are provided to illustrate how to apply each method to obtain point estimators.
2. TOPICS TO BE COVERED
1. Introduction to statistical inference
2. Theory of estimation
3. Methods of estimation
3.1 Method of maximum likelihood estimation
3.2 Method of moments
3. 1. Introduction to statistical inference
Statistics
Inferential Statistics
Estimation Testing of Hypothesis
Descriptive statistics
Descriptive analysis Graphical Presentation
4. What do we mean by Statistical Inference?
Drawing conclusion or making decision about population based
on information collected from the sample.
Population Sample
Representative
Making Conclusions
5. โ Statistical inference is further divided into two parts
Testing of hypothesis &
Theory of Estimation
Testing of hypothesis โ
โข The theory of testing of hypothesis is initiated by J. Neyman and
E. S. Pearson.
โข It provides the rule which makes one to decide about the
acceptance or rejection of the hypothesis under study.
Theory of estimation โ
โข The theory of estimation was founded by Prof. R. A. Fisher.
โข It discuss the ways of assigning the value to a population
parameter based on values of corresponding statistics (function
of sample observations).
6. 2. Theory of estimation
โข The theory of estimation was founded by R. A. Fisher.
Inferential
Statistics
Estimation
Point
Estimation
Interval
Estimation
Testing of
hypothesis
7. What do we mean by Estimation
It discuss the ways of assigning the values to a population parameter
based on the values of the corresponding statistics(function of the
sample observations).
The statistics used to estimate population parameter is called
estimator.
The value of the estimator is called estimate.
9. Point Estimation
It involves the use of sample data to calculate a single value(known
as a Point estimate) which is to serve as a best guess or best estimate
of an unknown population parameter. More formally, it is the
application of a point estimator to the data to obtain a point
estimate.
10. Interval estimation
It is the use of sample data to calculate an interval of possible values
of an unknown population parameter; this is in contrast to point
estimation, which gives a single value
Is an interval which is formed by two quantities based on sample
data within which the parameter will lie with very high probability.
11. 3. Methods of Estimation
โ Following are some of the important methods for obtaining good
estimators :
โข Method of maximum likelihood estimation
โข Method of moments
12. 3.1 Method of maximum likelihood
estimation
โ It is initially formulated by C. F. Gauss.
โ In statistics, maximum likelihood estimation (MLE) is a method
of estimating the parameters of a probability distribution by
maximizing a likelihood function, so that under the
assumed statistical model the observed data is most probable.
The point in the parameter space that maximizes the likelihood
function is called the maximum likelihood estimate.
13. Likelihood function
It is formed from the joint density function of the sample.
i.e.,
๐ฟ = ๐ฟ ๐ = ๐ ๐ฅ1, ๐ โฆ โฆ โฆ . ๐ ๐ฅ๐, ๐ = เท
๐=1
๐
๐ ๐ฅ๐, ๐
Where ๐ฅ1, ๐ฅ2, ๐ฅ3,โฆโฆ. ๐ฅ๐ be a random sample of size n from a
population with density function ๐ ๐ฅ, ๐ .
14. Steps to perform in MLE
1. Define the likelihood, ensuring youโre using the correct
distribution for your classification problem.
2. Take the natural log and reduce the product function to a sum
function.
3. Then compute the parameter by considering the case
๐
๐๐
๐๐๐๐ฟ = 0 &
๐2
๐๐2 ๐๐๐๐ฟ < 0
This equations are usually referred to as the Likelihood Equation for
estimating the parameters.
15. Example
Suppose we have a random sample ๐1, ๐2, โฆ โฆ โฆ , ๐๐ where :
๐๐ = 0 ; ๐๐ ๐ ๐๐๐๐๐๐๐๐ฆ ๐ ๐๐๐๐๐ก๐๐ ๐ ๐ก๐ข๐๐๐๐ก ๐๐๐๐ ๐๐๐ก ๐๐ค๐ ๐ ๐๐๐, ๐๐๐
๐๐ = 1 ; ๐๐ ๐ ๐๐๐๐๐๐๐๐ฆ ๐ ๐๐๐๐๐ก๐๐ ๐ ๐ก๐ข๐๐๐๐ก ๐๐๐๐ ๐๐ค๐ ๐ ๐๐๐.
Assuming that the ๐๐ are independent Bernoulli random variables with
unknown parameter p, find the maximum likelihood estimator of p, the
proportion of students who own a sports car.
16. If the ๐๐ are independent Bernoulli random variables with unknown
parameter p, then the probability mass function of each ๐๐ is :
๐ ๐ฅ; ๐ = ๐๐ฅ 1 โ ๐ 1โ๐ฅ
For ๐๐ = 0๐๐ 1 ๐๐๐ 0 < ๐ < 1.
Therefore, the likelihood function L(p) is, by definition:
Answer
๐ฟ ๐ = ฯ๐=1
๐
๐ ๐ฅ; ๐ = ๐๐ฅ1 1 โ ๐ 1โ๐ฅ1 ร ๐๐ฅ2 1 โ ๐ 1โ๐ฅ2 ร โฏ โฆ โฆ โฆ ร ๐๐ฅ๐แบ
แป
1 โ
๐ 1โ๐ฅ๐
For 0 < p < 1.
Simplifying, by summing up the exponents we get:
๐ฟ ๐ = ๐ฯ๐=1
๐
๐ฅ๐ 1 โ ๐ ๐ โ ฯ๐=1
๐
๐ฅ๐ โฆโฆโฆโฆโฆโฆโฆโฆโฆ..(1)
17. Now, in order to implement the method of maximum likelihood, we
need to find the value of unknown parameter p that maximizes the
likelihood L(p) given in equation (1).
So to maximize the function, we are need to differentiate the likelihood
function with respect to p.
And to make the differentiation easy we are going to use the logarithm
of likelihood function as it is an increasing function of x.
That is, if ๐ฅ1 < ๐ฅ2 , then๐แบ๐ฅ1แป < ๐แบ๐ฅ2แป. That means the value of p that
maximizes the natural logarithm of the likelihood function log L(p) is also
the value of p that maximizes the likelihood function L(p).
18. So we take the derivative of log L(p) with respect to p instead of taking the
derivative of L(p).
In this case, the log likelihood function is :
๐๐๐๐ฟ ๐ = ฯ๐=1
๐
๐ฅ๐ log ๐ + ๐ โ ฯ๐=1
๐
๐ฅ๐ log 1 โ ๐ โฆโฆโฆโฆโฆ.. (2)
Taking the derivative of log L(p) with respect to p and equate it with 0 we get :
๐ log ๐ฟ ๐
๐๐
= 0
19. =>
ฯ ๐ฅ๐
๐
โ
๐ โ ฯ ๐ฅ๐
1 โ ๐
= 0
Now by simplifying this for p we get;
Here (โ^โ) is used to represent the estimate of parameter p.
Though we find the estimate of parameter p, technically to verify that it is
maximum. For that the second derivative of the logL(p) with respect to p should
negative i.e.,
๐2 log ๐ฟ ๐
๐๐2
< 0 => โ๐ < 0 โฆ โฆ โฆ โฆ . แบ๐๐ฆ 3แป
Thus, ฦธ
๐ =
ฯ ๐ฅ๐
๐
is maximum likelihood estimator of p.
20. 3.2 Method of moments
โ This method was discovered and studied in detail by Karl Pearson.
โ The basic idea behind this form of the method is to:
1. Equate the first sample moment about the origin
๐1 =
1
๐
ฯ๐=1
๐
๐๐ = าง
๐ฅ
to the first theoretical moment E(X).
2. Equate the second sample moment about the origin
๐2=
1
๐
ฯ๐=1
๐
๐๐
2
to the second theoretical moment E(๐2
).
21. 3. Continue equating sample moments about the origin, ๐๐, with
the corresponding theoretical moments E(๐๐),k=3,4,โฆ until you
have as many equations as you have parameters.
4. Solve this equation for the parameters.
โ The resulting values are called method of moments estimators. It
seems reasonable that this method would provide good estimates,
since the empirical distribution converges in some sense to the
probability distribution. Therefore, the corresponding moments
should be about equal.
22. Another Form of the Method
โ The basic idea behind this form of the method is to:
1. Equate the first sample moment about the origin ๐1 =
1
๐
ฯ๐=1
๐
๐๐ = าง
๐ฅ to the first theoretical moment E(X).
2. Equate the second sample moment about the mean ๐1 =
1
๐
ฯ๐=1
๐
แบ๐ฅ๐ โ าง
๐ฅแป2 to the second theoretical moment about the
mean E[แบ๐ โ ๐แป2].
3. Continue equating sample moments about the mean Mkโ with the
corresponding theoretical moments about the
mean E[แบ๐ โ ๐แป๐], k=3,4,โฆ until you have as many equations as
you have parameters.
4. Solve for the parameters.
โ Again, the resulting values are called method of moments
estimators.
23. Example
Let ๐1, ๐2, โฆ โฆ โฆ , ๐๐ be normal random variate with mean ๐ and
variance ๐2. What are the method of moment estimators of the
mean ๐ and variance ๐2 ?
24. The first and second theoretical moments about the origin are:
๐ธ ๐๐ = ๐ ๐๐๐ ๐ธ ๐๐
2
= ๐2 + ๐2
Here we have two parameters for which we are trying to derive
method of momentโs estimators.
Answer
25. Therefore, we need two equations here. Equating the first theoretical
moment about the origin with the corresponding sample moment, we get:
๐ธ ๐๐ = ๐ =
1
๐
เท
๐=1
๐
๐๐ โฆ โฆ โฆ โฆ โฆ โฆ โฆ . แบ1แป
And equating the second theoretical moment about the origin with the
corresponding sample moment, we get:
๐ธ ๐๐
2
= ๐2 + ๐2 =
1
๐
เท
๐=1
๐
๐๐
2
โฆ โฆ โฆ โฆ โฆ โฆ โฆ . 2
26. Now from equation (1) we say that the method of moments estimator
for the mean ๐ is the sample mean:
ฦธ
๐๐๐ =
1
๐
เท
๐=1
๐
๐๐ = เดค
๐
And by substituting the sample mean as the estimator of ๐ in the
second equation and solving for ๐2, we get the method of moments
estimator for the variance ๐2
is;
เทข
๐2
๐๐ =
1
๐
เท
๐=1
๐
๐๐
2
โ ๐2 =
1
๐
เท
๐=1
๐
๐๐
2
โ เดค
๐2 =
1
๐
เท
๐=1
๐
แบ๐๐ โ เดฅ
๐ แป2
For this example, if cross check, then method of moments estimators
are the same as the maximum likelihood estimators.