hi.. i have been stuying regression analysis... the three red circles really har
ID: 3256541 • Letter: H
Question
hi..
i have been stuying regression analysis...
the three red circles really hard to me.
please help me
Thank you
1. 00) Consider a multiple Tinear regression mode y" XB-te E Nn (0, Io3). 15 of SSR/(p -1)s2, where s2 ssE/in-p). (50) Express sa in terms of s2 and the internally Studentized residual 3 50) Show that th SE of Bis independent of the residu ector 4 50) Derive a 100(1 a)% CI Bi Bi, 0 si s p-1. or j (50) Show that the lower bound of the ith leverage is 1/n in the simple linear regression case. 50) Derive a Durbin-Watson test under AR(1) model. (50) Define the Kullback-Leibler distance, and show that it is always non- negative. (50) Let Y,Y,'s be iid (0,1), A. (ay) and B (bis) be matrices with aii 1, aij 0.5, i j, b 1, Vi, j. Find the distribution ofExplanation / Answer
The Kullback–Leibler divergence from Q to P is often denoted DKL(PQ).
In the context of machine learning, DKL(PQ) is often called the information gain achieved if P is used instead of Q. By analogy with information theory, it is also called the relative entropy of P with respect to Q. In the context of coding theory, DKL(PQ) can be construed as measuring the expected number of extra bits required to code samples from P using a code optimized for Q rather than the code optimized for P.
Expressed in the language of Bayesian inference, DKL(PQ) is a measure of the information gained when one revises one's beliefs from the prior probability distribution Q to the posterior probability distribution P. In other words, it is the amount of information lost when Q is used to approximate P.[4] In applications, P typically represents the "true" distribution of data, observations, or a precisely calculated theoretical distribution, while Q typically represents a theory, model, description, or approximation of P.
The Kullback–Leibler divergence is a special case of a broader class of divergences called f-divergences as well as the class of Bregman divergences. It is the only such divergence over probabilities that is a member of both classes. Although it is often intuited as a way of measuring the distance between probability distributions, the Kullback–Leibler divergence is not a true metric. It does not obey the triangle inequality, and in general DKL(PQ) does not equal DKL(QP). However, its infinitesimal form, specifically its Hessian, gives a metric tensor known as the Fisher information metric.
Definition[edit]
For discrete probability distributions P and Q, the Kullback–Leibler divergence from Q to P is defined[5] to be
{displaystyle D_{mathrm {KL} }(P|Q)=sum _{i}P(i),log { rac {P(i)}{Q(i)}}.} D_{mathrm {KL} }(P|Q)=sum _{i}P(i),log { rac {P(i)}{Q(i)}}.
In other words, it is the expectation of the logarithmic difference between the probabilities P and Q, where the expectation is taken using the probabilities P. The Kullback–Leibler divergence is defined only if Q(i)=0 implies P(i)=0, for all i (absolute continuity). Whenever P(i) is zero the contribution of the i-th term is interpreted as zero because {displaystyle lim _{x o 0}xlog(x)=0} lim _{x o 0}xlog(x)=0.
For distributions P and Q of a continuous random variable, the Kullback–Leibler divergence is defined to be the integral:[6]
{displaystyle D_{mathrm {KL} }(P|Q)=int _{-infty }^{infty }p(x),log { rac {p(x)}{q(x)}},{ m {d}}x,!} D_{mathrm {KL} }(P|Q)=int _{-infty }^{infty }p(x),log { rac {p(x)}{q(x)}},{ m {d}}x,!
where p and q denote the densities of P and Q.
More generally, if P and Q are probability measures over a set X, and P is absolutely continuous with respect to Q, then the Kullback–Leibler divergence from Q to P is defined as
{displaystyle D_{mathrm {KL} }(P|Q)=int _{X}log { rac {{ m {d}}P}{{ m {d}}Q}},{ m {d}}P,!} D_{mathrm {KL} }(P|Q)=int _{X}log { rac {{ m {d}}P}{{ m {d}}Q}},{ m {d}}P,!
where {displaystyle { rac {{ m {d}}P}{{ m {d}}Q}}} { rac {{ m {d}}P}{{ m {d}}Q}} is the Radon–Nikodym derivative of P with respect to Q, and provided the expression on the right-hand side exists. Equivalently, this can be written as
{displaystyle D_{mathrm {KL} }(P|Q)=int _{X}log !left({ rac {{ m {d}}P}{{ m {d}}Q}} ight){ rac {{ m {d}}P}{{ m {d}}Q}},{ m {d}}Q,} D_{mathrm {KL} }(P|Q)=int _{X}log !left({ rac {{ m {d}}P}{{ m {d}}Q}} ight){ rac {{ m {d}}P}{{ m {d}}Q}},{ m {d}}Q,
which we recognize as the entropy of P relative to Q. Continuing in this case, if {displaystyle mu } mu is any measure on X for which {displaystyle p={ rac {{ m {d}}P}{{ m {d}}mu }}} p={ rac {{ m {d}}P}{{ m {d}}mu }} and {displaystyle q={ rac {{ m {d}}Q}{{ m {d}}mu }}} q={ rac {{ m {d}}Q}{{ m {d}}mu }} exist (meaning that p and q are absolutely continuous with respect to {displaystyle mu } mu ), then the Kullback–Leibler divergence from Q to P is given as
{displaystyle D_{mathrm {KL} }(P|Q)=int _{X}p,log { rac {p}{q}},{ m {d}}mu .!} D_{mathrm {KL} }(P|Q)=int _{X}p,log { rac {p}{q}},{ m {d}}mu .!
The logarithms in these formulae are taken to base 2 if information is measured in units of bits, or to base e if information is measured in nats. Most formulas involving the Kullback–Leibler divergence hold regardless of the base of the logarithm.
Various conventions exist for referring to DKL(PQ) in words. Often it is referred to as the divergence between P and Q; however this fails to convey the fundamental asymmetry in the relation. Sometimes, as in this article, it may be found described as the divergence of P from, or with respect to Q. This reflects the asymmetry in Bayesian inference, which starts from a prior Q and updates to the posterior P.
Related Questions
Navigate
Integrity-first tutoring: explanations and feedback only — we do not complete graded work. Learn more.