![\"\"](\"https:\/\/learning.oreilly.com\/api\/v2\/epubs\/urn:orm:book:9781482238129\/files\/bgf2.png\")
<\/p>\n\n\n\nWe will denote by an ensemble X the collection of events (represented by<\/p>\n\n\n\n
a random variable) x occurring with probability p(x):<\/p>\n\n\n\n
X \u2261 {x, p(x)}. (11.4)<\/p>\n\n\n\n
De\ufb01nition 11.1. The entropy function for an ensemble X is given by<\/p>\n\n\n\n
H(X) = \u2212k<\/p>\n\n\n\n
X<\/p>\n\n\n\n
x<\/p>\n\n\n\n
p(x) log p(x). (11.5)<\/p>\n\n\n\n
The number k is a constant which depends on the units in which H is mea-<\/p>\n\n\n\n
sured.<\/p>\n\n\n\n
The function H(X) satis\ufb01es the following properties.<\/p>\n\n\n\n
1. H(X) is always positive, and is continuous as a function of p(x) that is<\/p>\n\n\n\n
symmetric under exchange of any two events x<\/p>\n\n\n\n
i<\/p>\n\n\n\n
and x<\/p>\n\n\n\n
j<\/p>\n\n\n\n
.<\/p>\n\n\n\n
1<\/p>\n\n\n\n
See footnote 1 on page 111.<\/p>\n\n\n\n
<\/p>\n\n\n\n
Characterization of Quantum Information 217<\/p>\n\n\n\n
2. It has a minimum value of 0, when only one event occurs with probability<\/p>\n\n\n\n
1 and all the rest have probability 0. This is obvious to see since H(p)<\/p>\n\n\n\n
is a positive function and its minimum has to be zero.<\/p>\n\n\n\n
3. It has a maximum value of k log n when each x occurs with equal prob-<\/p>\n\n\n\n
ability 1\/n. Here is a simple proof of this fact:<\/p>\n\n\n\n
H(X) \u2212 k log n = k<\/p>\n\n\n\n
X<\/p>\n\n\n\n
x<\/p>\n\n\n\n
p(x) log<\/p>\n\n\n\n
1<\/p>\n\n\n\n
p(x)<\/p>\n\n\n\n
\u2212 k<\/p>\n\n\n\n
X<\/p>\n\n\n\n
x<\/p>\n\n\n\n
p(x) log n<\/p>\n\n\n\n
= k<\/p>\n\n\n\n
X<\/p>\n\n\n\n
p(x) log<\/p>\n\n\n\n
1<\/p>\n\n\n\n
np(x)<\/p>\n\n\n\n
\u2264 k<\/p>\n\n\n\n
X<\/p>\n\n\n\n
p(x)<\/p>\n\n\n\n
\ue012<\/p>\n\n\n\n
1<\/p>\n\n\n\n
np(x)<\/p>\n\n\n\n
\u2212 1<\/p>\n\n\n\n
\ue013<\/p>\n\n\n\n
.<\/p>\n\n\n\n
This is because log x \u2264 x \u2212 1, with equality only if x = 1 (see Figure 11.2),<\/p>\n\n\n\n
an important result often used in information theoretic proofs.<\/p>\n\n\n\n
FIGURE 11.2: Graph of y = x \u2212 1 compared with y = ln x.<\/p>\n\n\n\n
So we have<\/p>\n\n\n\n
H(X) \u2212 k log n \u2264 k<\/p>\n\n\n\n
\ue012<\/p>\n\n\n\n
X<\/p>\n\n\n\n
1<\/p>\n\n\n\n
n<\/p>\n\n\n\n
\u2212<\/p>\n\n\n\n
X<\/p>\n\n\n\n
p(x)<\/p>\n\n\n\n
\ue013<\/p>\n\n\n\n
= 0,<\/p>\n\n\n\n
\u2234 H(X) \u2264 k log n. (11.6)<\/p>\n\n\n\n
Box 11.1: Binary Entropy<\/p>\n\n\n\n
A very useful concept is the entropy function of a probability distribution<\/p>\n\n\n\n
![\"\"](\"https:\/\/learning.oreilly.com\/api\/v2\/epubs\/urn:orm:book:9781482238129\/files\/bgf3.png\")
<\/p>\n\n\n\n
218 Introduction to Quantum Physics and Information Processing<\/p>\n\n\n\n
of a binary random variable, such as the result of the toss of a coin, not<\/p>\n\n\n\n
necessarily unbiased. Here, one value occurs with probability p and the other<\/p>\n\n\n\n
with 1 \u2212 p. We then have<\/p>\n\n\n\n
H<\/p>\n\n\n\n
bin<\/p>\n\n\n\n
(p) = \u2212p log p \u2212 (1 \u2212 p) log(1 \u2212 p). (11.7)<\/p>\n\n\n\n
In this simple case all the listed properties of the mathematical entropy func-<\/p>\n\n\n\n
tion are obvious.<\/p>\n\n\n\n
1. Positive: H<\/p>\n\n\n\n
bin<\/p>\n\n\n\n
(p) > 0 always;<\/p>\n\n\n\n
2. Symmetric: H<\/p>\n\n\n\n
bin<\/p>\n\n\n\n
(p) = H<\/p>\n\n\n\n
bin<\/p>\n\n\n\n
(1 \u2212 p);<\/p>\n\n\n\n
3. H<\/p>\n\n\n\n
bin<\/p>\n\n\n\n
(p) has a maximum of 1 when p = 1\/2 as in a fair coin;<\/p>\n\n\n\n
4. H<\/p>\n\n\n\n
bin<\/p>\n\n\n\n
(p) has a minimum of 0 when p = 1 as in a two-headed coin.<\/p>\n\n\n\n
FIGURE 11.3: The binary entropy function.<\/p>\n\n\n\n
This function is a useful tool in deriving properties of entropy, especially when<\/p>\n\n\n\n
di\ufb00erent probability distributions are mixed together. An important property<\/p>\n\n\n\n
of the entropy function is made evident in this simple case: that of concavity.<\/p>\n\n\n\n
This property is used very often in concluding various results in classical as<\/p>\n\n\n\n
well as quantum information theory. The graph in Figure 11.3 shows that the<\/p>\n\n\n\n
function is literally concave. A mathematical statement of this property is<\/p>\n\n\n\n
that the function lies above any line cutting the graph. Algebraically, for two<\/p>\n\n\n\n
points x<\/p>\n\n\n\n
1<\/p>\n\n\n\n
, x<\/p>\n\n\n\n
2<\/p>\n\n\n\n
< 1, we have<\/p>\n\n\n\n
H<\/p>\n\n\n\n
bin<\/p>\n\n\n\n
\ue000<\/p>\n\n\n\n
px<\/p>\n\n\n\n
1<\/p>\n\n\n\n
+ (1 \u2212 p)x<\/p>\n\n\n\n
2<\/p>\n\n\n\n
\ue001<\/p>\n\n\n\n
\u2265 pH<\/p>\n\n\n\n
bin<\/p>\n\n\n\n
(x<\/p>\n\n\n\n
1<\/p>\n\n\n\n
) + (1 \u2212 p)H<\/p>\n\n\n\n
bin<\/p>\n\n\n\n
(x<\/p>\n\n\n\n
2<\/p>\n\n\n\n
). (11.8)<\/p>\n\n\n\n
<\/p>\n\n\n\n
Characterization of Quantum Information 219<\/p>\n\n\n\n
11.1.3 Relations between entropies of two sets of events<\/p>\n\n\n\n
From the way it is de\ufb01ned, Shannon entropy is closely related to probability<\/p>\n\n\n\n
theory. In this book, we do not expect a thorough background in probability<\/p>\n\n\n\n
theory, so I will simply draw your attention to some important results, so<\/p>\n\n\n\n
that you may be piqued enough to look them up on your own. Consider two<\/p>\n\n\n\n
ensembles X = {x, p(x)} and Y = {y, p(y)}. We will de\ufb01ne various measures<\/p>\n\n\n\n
to compare the probability distributions {p(x)} and {p(y)}.<\/p>\n\n\n\n
1. Relative entropy of X and Y measures the di\ufb00erence between the two<\/p>\n\n\n\n
probability distributions {p(x)} and {p(y)}:<\/p>\n\n\n\n
H(X k Y ) = \u2212<\/p>\n\n\n\n
X<\/p>\n\n\n\n
x,y<\/p>\n\n\n\n
p(x) log p(y) \u2212 H(X)<\/p>\n\n\n\n
=<\/p>\n\n\n\n
X<\/p>\n\n\n\n
x,y<\/p>\n\n\n\n
p(x) log<\/p>\n\n\n\n
p(y)<\/p>\n\n\n\n
p(x)<\/p>\n\n\n\n
. (11.9)<\/p>\n\n\n\n
Here again we use the convention that<\/p>\n\n\n\n
\u22120 log 0 \u2261 0, \u2212p(x) log 0 \u2261 \u221e, p(x) > 0. (11.10)<\/p>\n\n\n\n
An important property of the relative entropy is that it is positive. The<\/p>\n\n\n\n
relative entropy is also called the Kullback\u2013Leibler distance. However,<\/p>\n\n\n\n
it is not symmetric, and so is not a true distance measure, but it gives<\/p>\n\n\n\n
us, for example, the error in assuming that a certain random variable<\/p>\n\n\n\n
has probability distribution {p(y)} when the true distribution is {p(x)}.<\/p>\n\n\n\n
Thus this de\ufb01nition is more useful when we have a set of events X with<\/p>\n\n\n\n
two di\ufb00erent probability distributions {p(x)} and {q(x)},<\/p>\n\n\n\n
H(p k q) =<\/p>\n\n\n\n
X<\/p>\n\n\n\n
x<\/p>\n\n\n\n
p(x) log<\/p>\n\n\n\n
p(x)<\/p>\n\n\n\n
q(x)<\/p>\n\n\n\n
. (11.11)<\/p>\n\n\n\n
2. Joint entropy of X and Y measures the combined information pre-<\/p>\n\n\n\n
sented by both distributions. Classically, the joint probability of X and<\/p>\n\n\n\n
Y , denoted by {p(x, y)}, is de\ufb01ned over a set X \u2297Y . The joint entropy<\/p>\n\n\n\n
is then<\/p>\n\n\n\n
H(X, Y ) = \u2212<\/p>\n\n\n\n
X<\/p>\n\n\n\n
x,y<\/p>\n\n\n\n
p(x, y) log p(x, y). (11.12)<\/p>\n\n\n\n
If X and Y are independent events, then<\/p>\n\n\n\n
H(X, Y ) = H(X) + H(Y ). (11.13)<\/p>\n\n\n\n
3. Conditional entropy measures the information gained by the occur-<\/p>\n\n\n\n
rence of X if Y has already occurred and we know the outcome. The<\/p>\n\n\n\n
220 Introduction to Quantum Physics and Information Processing<\/p>\n\n\n\n FIGURE 11.4: Relationship between entropic quantities.<\/p>\n\n\n\n classical conditional probability of an event x given y is de\ufb01ned as<\/p>\n\n\n\n p(x|y) = p(x, y)\/p(y), and we have<\/p>\n\n\n\n H(X|Y ) = \u2212<\/p>\n\n\n\n X<\/p>\n\n\n\n x,y<\/p>\n\n\n\n p(x|y) log p(x|y) (11.14)<\/p>\n\n\n\n = H(X, Y ) \u2212 H(Y ). (11.15)<\/p>\n\n\n\n The second equation is an important relation: a chain rule for entropies:<\/p>\n\n\n\n H(X, Y ) = H(X) + H(Y |X). (11.16)<\/p>\n\n\n\n 4. Mutual information measures the correlation between the distribu-<\/p>\n\n\n\n tions of X and Y . This is the di\ufb00erence between the information gained<\/p>\n\n\n\n by the occurrence of X, and the information gained by occurrence of X<\/p>\n\n\n\n if Y has already occurred. The mutual information is symmetric, so we<\/p>\n\n\n\n have<\/p>\n\n\n\n I(X; Y ) = H(X) \u2212 H(X|Y ) = H(Y ) \u2212 H(Y |X) (11.17)<\/p>\n\n\n\n = H(X) + H(Y ) \u2212 H(X, Y ). (11.18)<\/p>\n\n\n\n The mutual information is a measure of how much the uncertainty about<\/p>\n\n\n\n X is reduced by a knowledge of Y . You can also see that it is the relative<\/p>\n\n\n\n entropy of the joint distribution p(x, y) and the product distribution<\/p>\n\n\n\n p(x)p(y):<\/p>\n\n\n\n I(X; Y ) =<\/p>\n\n\n\n X<\/p>\n\n\n\n x,y<\/p>\n\n\n\n p(x, y) log<\/p>\n\n\n\n p(x, y)<\/p>\n\n\n\n p(x)p(y)<\/p>\n\n\n\n . (11.19)<\/p>\n\n\n\n One way to picture the interrelationships between these entropic quantities<\/p>\n\n\n\n is the Venn diagram of Figure 11.4. Given these de\ufb01nitions, the Shannon<\/p>\n\n\n\n entropies satisfy the following properties that are easily proved<\/p>\n","protected":false},"excerpt":{"rendered":" We will denote by an ensemble X the collection of events (represented by a random variable) x occurring with probability p(x): X \u2261 {x, p(x)}. (11.4) De\ufb01nition 11.1. The entropy function for an ensemble X is given by H(X) = \u2212k X x p(x) log p(x). (11.5) The number k is a constant which depends […]<\/p>\n","protected":false},"author":1,"featured_media":4045,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[500],"tags":[],"class_list":["post-4138","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-characterization-of-quantum-information"],"jetpack_featured_media_url":"https:\/\/workhouse.sweetdishy.com\/wp-content\/uploads\/2024\/09\/quantum-computing-2.png","_links":{"self":[{"href":"https:\/\/workhouse.sweetdishy.com\/index.php\/wp-json\/wp\/v2\/posts\/4138","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/workhouse.sweetdishy.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/workhouse.sweetdishy.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/workhouse.sweetdishy.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/workhouse.sweetdishy.com\/index.php\/wp-json\/wp\/v2\/comments?post=4138"}],"version-history":[{"count":4,"href":"https:\/\/workhouse.sweetdishy.com\/index.php\/wp-json\/wp\/v2\/posts\/4138\/revisions"}],"predecessor-version":[{"id":4577,"href":"https:\/\/workhouse.sweetdishy.com\/index.php\/wp-json\/wp\/v2\/posts\/4138\/revisions\/4577"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/workhouse.sweetdishy.com\/index.php\/wp-json\/wp\/v2\/media\/4045"}],"wp:attachment":[{"href":"https:\/\/workhouse.sweetdishy.com\/index.php\/wp-json\/wp\/v2\/media?parent=4138"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/workhouse.sweetdishy.com\/index.php\/wp-json\/wp\/v2\/categories?post=4138"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/workhouse.sweetdishy.com\/index.php\/wp-json\/wp\/v2\/tags?post=4138"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}![\"\"](\"https:\/\/learning.oreilly.com\/api\/v2\/epubs\/urn:orm:book:9781482238129\/files\/bgf5.png\")
<\/sup><\/mark><\/code><\/p>\n\n\n\n