1. The document introduces parameters like mean, variance, and covariance to represent the relationship between two random variables X and Y. 2. Covariance is defined as a measure of how X and Y vary together, while correlation is the standardized covariance between X and Y. 3. Two random variables are uncorrelated if their covariance is 0, while independence implies uncorrelatedness but not vice versa, as shown by an example of dependent but uncorrelated random variables Z and W.

1. 1
10. Joint Moments and Joint Characteristic
Functions
Following section 6, in this section we shall introduce
various parameters to compactly represent the information
contained in the joint p.d.f of two r.vs. Given two r.vs X and
Y and a function define the r.v
Using (6-2), we can define the mean of Z to be
(10-1)
)
,
( Y
X
g
Z 
),
,
( y
x
g
.
)
(
)
( 





 dz
z
f
z
Z
E Z
Z
 (10-2)
PILLAI

2. 2
However, the situation here is similar to that in (6-13), and
it is possible to express the mean of in terms
of without computing To see this, recall from
(5-26) and (7-10) that
where is the region in xy plane satisfying the above
inequality. From (10-3), we get
As covers the entire z axis, the corresponding regions
are nonoverlapping, and they cover the entire xy plane.
   
 
















z
D
y
x
XY
Z
y
x
y
x
f
z
z
Y
X
g
z
P
z
z
f
z
z
Z
z
P
)
,
(
)
,
(
)
,
(
)
(
(10-3)
z
D
)
,
( Y
X
g
Z 
)
,
( y
x
fXY ).
(z
fZ
( , )
( ) ( , ) ( , ) .
z
Z XY
x y D
z f z z g x y f x y x y


   
  (10-4)
z
D
z

PILLAI

3. 3
By integrating (10-4), we obtain the useful formula
or
If X and Y are discrete-type r.vs, then
Since expectation is a linear operator, we also get
( ) ( ) ( , ) ( , ) .
Z XY
E Z z f z dz g x y f x y dxdy
  
  
 
   (10-5)
(10-6)
[ ( , )] ( , ) ( , ) .
XY
E g X Y g x y f x y dxdy
 
 
  
[ ( , )] ( , ) ( , ).
i j i j
i j
E g X Y g x y P X x Y y
  
 (10-7)
( , ) [ ( , )].
k k k k
k k
E a g X Y a E g X Y
 

 
 
  (10-8)
PILLAI

4. 4
If X and Y are independent r.vs, it is easy to see that
and are always independent of each other. In that
case using (10-7), we get the interesting result
However (10-9) is in general not true (if X and Y are not
independent).
In the case of one random variable (see (10- 6)), we defined
the parameters mean and variance to represent its average
behavior. How does one parametrically represent similar
cross-behavior between two random variables? Towards
this, we can generalize the variance definition given in
(6-16) as shown below:
)
(X
g
Z 
)].
(
[
)]
(
[
)
(
)
(
)
(
)
(
)
(
)
(
)
(
)
(
)]
(
)
(
[
Y
h
E
X
g
E
dy
y
f
y
h
dx
x
f
x
g
dxdy
y
f
x
f
y
h
x
g
Y
h
X
g
E
Y
X
Y
X





 
















(10-9)
)
(Y
h
W 
PILLAI

5. 5
Covariance: Given any two r.vs X and Y, define
By expanding and simplifying the right side of (10-10), we
also get
It is easy to see that
To see (10-12), let so that
.
)
(
)
(
)
(
)
(
)
,
(
__
__
____
Y
X
XY
Y
E
X
E
XY
E
XY
E
Y
X
Cov Y
X





 

(10-10)
(10-12)
 .
)
(
)
(
)
,
( Y
X Y
X
E
Y
X
Cov 
 


(10-11)
,
Y
aX
U 

.
)
(
)
(
)
,
( Y
Var
X
Var
Y
X
Cov 
 
 
.
0
)
(
)
,
(
2
)
(
)
(
)
(
)
(
2
2








Y
Var
Y
X
Cov
a
X
Var
a
Y
X
a
E
U
Var Y
X 

(10-13)
PILLAI

6. 6
The right side of (10-13) represents a quadratic in the
variable a that has no distinct real roots (Fig. 10.1). Thus the
roots are imaginary (or double) and hence the discriminant
must be non-positive, and that gives (10-12). Using (10-12),
we may define the normalized parameter
or
and it represents the correlation
coefficient between X and Y.
(10-14)
  )
(
)
(
)
,
(
2
Y
Var
X
Var
Y
X
Cov 
,
1
1
,
)
,
(
)
(
)
(
)
,
(




 XY
Y
X
XY
Y
X
Cov
Y
Var
X
Var
Y
X
Cov




a
)
(U
Var
Fig. 10.1
Y
X
XY
Y
X
Cov 



)
,
( (10-15)
PILLAI

7. 7
Uncorrelated r.vs: If then X and Y are said to be
uncorrelated r.vs. From (11), if X and Y are uncorrelated,
then
Orthogonality: X and Y are said to be orthogonal if
From (10-16) - (10-17), if either X or Y has zero mean, then
orthogonality implies uncorrelatedness also and vice-versa.
Suppose X and Y are independent r.vs. Then from (10-9)
with we get
and together with (10-16), we conclude that the random
variables are uncorrelated, thus justifying the original
definition in (10-10). Thus independence implies
uncorrelatedness.
(10-16)
,
0

XY

,
)
(
,
)
( Y
Y
h
X
X
g 

).
(
)
(
)
( Y
E
X
E
XY
E 
.
0
)
( 
XY
E
(10-17)
),
(
)
(
)
( Y
E
X
E
XY
E 
PILLAI

8. 8
Naturally, if two random variables are statistically
independent, then there cannot be any correlation between
them However, the converse is in general not
true. As the next example shows, random variables can be
uncorrelated without being independent.
Example 10.1: Let   Suppose X and Y
are independent. Define Z = X + Y, W = X - Y . Show that Z
and W are dependent, but uncorrelated r.vs.
Solution: gives the only solution set to be
Moreover
and
),
1
,
0
(
U
X ).
1
,
0
(
U
Y
|
|
,
2
,
2
,
1
1
,
2
0 w
z
w
z
w
z
w
z 









.
2
/
1
|
)
,
(
| 
w
z
J
.
2
,
2
w
z
y
w
z
x




y
x
w
y
x
z 


 ,
).
0
( 
XY

PILLAI

9. 9


 










,
otherwise
,
0
,
|
|
,
2
,
2
,
1
1
,
2
0
,
2
/
1
)
,
(
z
w
w
z
w
z
w
z
w
z
fZW
(10-18)
Thus (see the shaded region in Fig. 10.2)
and hence
or by direct computation ( Z = X + Y )
Fig. 10.2
1

1
w
z
2




















,
2
1
,
2
2
1
,
1
0
,
2
1
)
,
(
)
(
2
2
z
z
dw
z
z
dw
dw
w
z
f
z
f
-z
z-
z
z
ZW
Z
PILLAI

10. 10
and
Clearly Thus Z and W are not
independent. However
and
and hence
implying that Z and W are uncorrelated random variables.













,
otherwise
,
0
,
2
1
,
2
,
1
0
,
)
(
)
(
)
( z
z
z
z
z
f
z
f
z
f Y
X
Z
(10-19)
(10-20)
).
(
)
(
)
,
( w
f
z
f
w
z
f W
Z
ZW 
(10-21)
  ,
0
)
(
)
(
)
)(
(
)
( 2
2





 Y
E
X
E
Y
X
Y
X
E
ZW
E
0
)
(
)
(
)
(
)
,
( 

 W
E
Z
E
ZW
E
W
Z
Cov (10-22)


 





 


.
otherwise
,
0
,
1
1
|,
|
1
2
1
)
,
(
)
(
|
|
2 w
w
dz
dz
w
z
f
w
f
w
|w|
ZW
W
,
0
)
(
)
( 

 Y
X
E
W
E
PILLAI

11. 11
Example 10.2: Let Determine the variance of Z
in terms of and
Solution:
and using (10-15)
In particular if X and Y are independent, then and
(10-23) reduces to
Thus the variance of the sum of independent r.vs is the sum
of their variances
(10-23)
,
0

XY

.
bY
aX
Z 

Y
X 
 , .
XY

( ) ( )
Z X Y
E Z E aX bY a b
  
    
 
 
2
2 2
2 2 2 2
2 2 2 2
( ) ( ) ( ) ( )
( ) 2 ( )( ) ( )
2 .
Z Z X Y
X X Y Y
X XY X Y Y
Var Z E Z E a X b Y
a E X abE X Y b E Y
a ab b
   
   
    
 
 
      
   
      
  
.
2
2
2
2
2
Y
X
Z b
a 

 
 (10-24)
).
1
( 
 b
a PILLAI

12. 12
Moments:
represents the joint moment of order (k,m) for X and Y.
Following the one random variable case, we can define the
joint characteristic function between two random variables
which will turn out to be useful for moment calculations.
Joint characteristic functions:
The joint characteristic function between X and Y is defined
as
Note that
,
)
,
(
]
[  








 dy
dx
y
x
f
y
x
Y
X
E XY
m
k
m
k
(10-25)
 
( ) ( )
( , ) ( , ) .
j Xu Yv j Xu Yv
XY XY
u v E e e f x y dxdy
 
 
 
     (10-26)
.
1
)
0
,
0
(
)
,
( 


 XY
XY v
u
PILLAI

13. 13
It is easy to show that
If X and Y are independent r.vs, then from (10-26), we obtain
Also
More on Gaussian r.vs :
From Lecture 7, X and Y are said to be jointly Gaussian
as if their joint p.d.f has the form in (7-
23). In that case, by direct substitution and simplification,
we obtain the joint characteristic function of two jointly
Gaussian r.vs to be
.
)
,
(
1
)
(
0
,
0
2
2







v
u
XY
v
u
v
u
j
XY
E (10-27)
).
(
)
(
)
(
)
(
)
,
( v
u
e
E
e
E
v
u Y
X
jvY
juX
XY 



 (10-28)
( ) ( ,0), ( ) (0, ).
X XY Y XY
u u v v
      (10-29)
),
,
,
,
,
( 2
2




 Y
X
Y
X
N
PILLAI

14. 14
.
)
(
)
,
(
)
2
(
2
1
)
(
)
(
2
2
2
2
v
uv
u
v
u
j
Yv
Xu
j
XY
Y
Y
X
X
Y
X
e
e
E
v
u





 






 (10-30)
Equation (10-14) can be used to make various conclusions.
Letting in (10-30), we get
and it agrees with (6-47).
From (7-23) by direct computation using (10-11), it is easy
to show that for two jointly Gaussian random variables
Hence from (10-14), in represents
the actual correlation coefficient of the two jointly Gaussian
r.vs in (7-23). Notice that implies
0

v
,
)
0
,
(
)
(
2
2
2
1
u
u
j
XY
X
X
X
e
u
u

 



 (10-31)
)
,
,
,
,
( 2
2




 Y
X
Y
X
N

.
)
,
( Y
X
Y
X
Cov 



0


PILLAI

15. 15
Thus if X and Y are jointly Gaussian, uncorrelatedness does
imply independence between the two random variables.
Gaussian case is the only exception where the two concepts
imply each other.
Example 10.3: Let X and Y be jointly Gaussian r.vs with
parameters Define
Determine
Solution: In this case we can make use of characteristic
function to solve this problem.
).
,
,
,
,
( 2
2




 Y
X
Y
X
N .
bY
aX
Z 

).
(z
fZ
).
,
(
)
(
)
(
)
(
)
( )
(
bu
au
e
E
e
E
e
E
u
XY
jbuY
jauX
u
bY
aX
j
jZu
Z





 

(10-32)
).
(
)
(
)
,
( y
f
x
f
Y
X
f Y
X
XY 
PILLAI

16. 16
From (10-30) with u and v replaced by au and bu
respectively we get
where
Notice that (10-33) has the same form as (10-31), and hence
we conclude that is also Gaussian with mean and
variance as in (10-34) - (10-35), which also agrees with (10-
23).
From the previous example, we conclude that any linear
combination of jointly Gaussian r.vs generate a Gaussian r.v.
,
)
(
2
2
2
2
2
2
2
2
1
)
2
(
2
1
)
( u
u
j
u
b
ab
a
u
b
a
j
Z
Z
Z
Y
Y
X
X
Y
X
e
e
u








 







(10-33)
(10-34)
(10-35)
bY
aX
Z 

.
2
,
2
2
2
2
2
Y
Y
X
X
Z
Y
X
Z
b
ab
a
b
a
















PILLAI

17. 17
In other words, linearity preserves Gaussianity. We can use
the characteristic function relation to conclude an even more
general result.
Example 10.4: Suppose X and Y are jointly Gaussian r.vs as
in the previous example. Define two linear combinations
what can we say about their joint distribution?
Solution: The characteristic function of Z and W is given by
As before substituting (10-30) into (10-37) with u and v
replaced by au + cv and bu + dv respectively, we get
.
, dY
cX
W
bY
aX
Z 


 (10-36)
).
,
(
)
(
)
(
)
(
)
,
(
)
(
)
(
)
(
)
(
)
(
dv
bu
cv
au
e
E
e
E
e
E
v
u
XY
dv
bu
jY
cv
au
jX
v
dY
cX
j
u
bY
aX
j
Wv
Zu
j
ZW















(10-37)
PILLAI

18. 18
,
)
,
(
)
2
(
2
1
)
( 2
2
2
2
v
uv
u
v
u
j
ZW
W
Y
X
ZW
Z
W
Z
e
v
u






 




 (10-38)
where
and
From (10-38), we conclude that Z and W are also jointly
distributed Gaussian r.vs with means, variances and
correlation coefficient as in (10-39) - (10-43).
,
2
,
2
,
,
2
2
2
2
2
2
2
2
2
2
Y
Y
X
X
W
Y
Y
X
X
Z
Y
X
W
Y
X
Z
d
cd
c
b
ab
a
d
c
b
a

























 (10-39)
(10-40)
(10-41)
(10-42)
.
)
( 2
2
W
Z
Y
Y
X
X
ZW
bd
bc
ad
ac










 (10-43)
PILLAI

19. 19
To summarize, any two linear combinations of jointly
Gaussian random variables (independent or dependent) are
also jointly Gaussian r.vs.
Of course, we could have reached the same conclusion by
deriving the joint p.d.f using the technique
developed in section 9 (refer (7-29)).
Gaussian random variables are also interesting because of
the following result:
Central Limit Theorem: Suppose are a set of
zero mean independent, identically distributed (i.i.d) random
Linear
operator
Gaussian input Gaussian output
)
,
( w
z
fZW
n
X
X
X ,
,
, 2
1 
Fig. 10.3
PILLAI

20. 20
variables with some common distribution. Consider their
scaled sum
Then asymptotically (as )
Proof: Although the theorem is true under even more
general conditions, we shall prove it here under the
independence assumption. Let represent their common
variance. Since
we have
.
2
1
n
X
X
X
Y n





(10-44)


n
).
,
0
( 2

N
Y 
2

,
0
)
( 
i
X
E
.
)
(
)
( 2
2


 i
i X
E
X
Var
(10-45)
(10-46)
(10-47)
PILLAI

21. 21
Consider
where we have made use of the independence of the
r.vs But
where we have made use of (10-46) - (10-47). Substituting
(10-49) into (10-48), we obtain
and as
 













n
i
X
n
i
n
u
jX
n
u
X
X
X
j
jYu
Y
n
u
e
E
e
E
e
E
u
i
i
n
1
1
/
/
)
(
)
/
(
)
(
)
(
)
( 2
1 
(10-48)
.
,
,
, 2
1 n
X
X
X 
,
1
2
1
!
3
!
2
1
)
( 2
/
3
2
2
2
/
3
3
3
3
2
2
2
/






















n
o
n
u
n
u
X
j
n
u
X
j
n
u
jX
E
e
E i
i
i
n
u
jXi

 (10-49)
,
1
2
1
)
( 2
/
3
2
2 n
Y
n
o
n
u
u 















 (10-50)
2 2
/ 2
lim ( ) ,
u
Y
n
u e 


  (10-51)
PILLAI

22. 22
.
1
lim x
n
n
e
n
x 









 (10-52)
[Note that terms in (10-50) decay faster than
But (10-51) represents the characteristic function of a zero
mean normal r.v with variance and (10-45) follows.
The central limit theorem states that a large sum of
independent random variables each with finite variance
tends to behave like a normal random variable. Thus the
individual p.d.fs become unimportant to analyze the
collective sum behavior. If we model the noise phenomenon
as the sum of a large number of independent random
variables (eg: electron motion in resistor components), then
this theorem allows us to conclude that noise behaves like a
Gaussian r.v.
3/ 2
(1/ )
o n 3/ 2
1/ ].
n
2

since
PILLAI

23. 23
It may be remarked that the finite variance assumption is
necessary for the theorem to hold good. To prove its
importance, consider the r.vs to be Cauchy distributed, and
let
where each  Then since
substituting this into (10-48), we get
which shows that Y is still Cauchy with parameter
In other words, central limit theorem doesn’t hold good for
a set of Cauchy r.vs as their variances are undefined.
.
2
1
n
X
X
X
Y n




 (10-53)
).
(
C
Xi
,
)
( |
|u
X e
u
i



 (10-54)
 
n
| |/
1
( ) ( / ) ~ ( ),
n
u n
Y X
i
u u n e C n




   
 (10-55)
.
n

PILLAI

24. 24
Joint characteristic functions are useful in determining the
p.d.f of linear combinations of r.vs. For example, with X and
Y as independent Poisson r.vs with parameters and
respectively, let
Then
But from (6-33)
so that

i.e., sum of independent Poisson r.vs is also a Poisson
random variable.
.
Y
X
Z 
 (10-56)
).
(
)
(
)
( u
u
u Y
X
Z 


 (10-57)
)
1
(
)
1
( 2
1
)
(
,
)
( 





ju
ju
e
Y
e
X e
u
e
u 

(10-58)
)
(
)
( 2
1
)
1
)(
( 2
1






 

P
e
u
ju
e
Z
(10-59)
1
 2

PILLAI