You get a bonus - 1 coin for daily activity. Now you have 1 coin

18.4. Conditional entropy. Association of dependent systems

Lecture



Let there be two systems   18.4.  Conditional entropy.  Association of dependent systems and   18.4.  Conditional entropy.  Association of dependent systems generally dependent. Suppose the system   18.4.  Conditional entropy.  Association of dependent systems took the state   18.4.  Conditional entropy.  Association of dependent systems . Denote   18.4.  Conditional entropy.  Association of dependent systems conditional probability that the system   18.4.  Conditional entropy.  Association of dependent systems will take the state   18.4.  Conditional entropy.  Association of dependent systems provided that the system   18.4.  Conditional entropy.  Association of dependent systems is able to   18.4.  Conditional entropy.  Association of dependent systems :

  18.4.  Conditional entropy.  Association of dependent systems . (18.4.1)

We now define the conditional entropy of the system   18.4.  Conditional entropy.  Association of dependent systems provided that the system   18.4.  Conditional entropy.  Association of dependent systems is able to   18.4.  Conditional entropy.  Association of dependent systems . Denote it   18.4.  Conditional entropy.  Association of dependent systems . By general definition, we have:

  18.4.  Conditional entropy.  Association of dependent systems (18.4.2)

or

  18.4.  Conditional entropy.  Association of dependent systems . (18.4.2 ')

The formula (18.4.2) can also be written in the form of a mathematical expectation:

  18.4.  Conditional entropy.  Association of dependent systems , (18.4.3)

where is familiar   18.4.  Conditional entropy.  Association of dependent systems denotes the conditional expectation of the value in brackets, provided   18.4.  Conditional entropy.  Association of dependent systems .

Conditional entropy depends on which state   18.4.  Conditional entropy.  Association of dependent systems adopted the system   18.4.  Conditional entropy.  Association of dependent systems ; for some states, it will be more, for others - less. Determine the average, or total, entropy of the system   18.4.  Conditional entropy.  Association of dependent systems taking into account the fact that the system can take different states. To do this, each conditional entropy (18.4.2) must be multiplied by the probability of the corresponding state   18.4.  Conditional entropy.  Association of dependent systems and all such works add up. Denote the full conditional entropy   18.4.  Conditional entropy.  Association of dependent systems :

  18.4.  Conditional entropy.  Association of dependent systems (18.4.4)

or, using the formula (18.4.2),

  18.4.  Conditional entropy.  Association of dependent systems .

Bringing in   18.4.  Conditional entropy.  Association of dependent systems under the sign of the second sum, we get:

  18.4.  Conditional entropy.  Association of dependent systems (18.4.5)

or

  18.4.  Conditional entropy.  Association of dependent systems . (18.4.5 ')

But by the probability multiplication theorem   18.4.  Conditional entropy.  Association of dependent systems , Consequently,

  18.4.  Conditional entropy.  Association of dependent systems . (18.4.6)

The expression (18.4.6) can also be given the form of a mathematical expectation:

  18.4.  Conditional entropy.  Association of dependent systems . (18.4.7)

Magnitude   18.4.  Conditional entropy.  Association of dependent systems characterizes the degree of system uncertainty   18.4.  Conditional entropy.  Association of dependent systems remaining after system state   18.4.  Conditional entropy.  Association of dependent systems fully defined. We will call it the complete conditional entropy of the system.   18.4.  Conditional entropy.  Association of dependent systems regarding   18.4.  Conditional entropy.  Association of dependent systems .

Example 1. There are two systems.   18.4.  Conditional entropy.  Association of dependent systems and   18.4.  Conditional entropy.  Association of dependent systems merged into one   18.4.  Conditional entropy.  Association of dependent systems ; probabilities of system states   18.4.  Conditional entropy.  Association of dependent systems set by table

  18.4.  Conditional entropy.  Association of dependent systems

  18.4.  Conditional entropy.  Association of dependent systems

  18.4.  Conditional entropy.  Association of dependent systems

  18.4.  Conditional entropy.  Association of dependent systems

  18.4.  Conditional entropy.  Association of dependent systems

  18.4.  Conditional entropy.  Association of dependent systems

  18.4.  Conditional entropy.  Association of dependent systems

0.1

0.2

0

0.3

  18.4.  Conditional entropy.  Association of dependent systems

0

0.3

0

0.3

  18.4.  Conditional entropy.  Association of dependent systems

0

0.2

0.2

0.4

  18.4.  Conditional entropy.  Association of dependent systems

0.1

0.7

0.2

Determine the total conditional entropy   18.4.  Conditional entropy.  Association of dependent systems and   18.4.  Conditional entropy.  Association of dependent systems .

Decision. Adding probabilities   18.4.  Conditional entropy.  Association of dependent systems by columns, we get probabilities   18.4.  Conditional entropy.  Association of dependent systems :

  18.4.  Conditional entropy.  Association of dependent systems ;   18.4.  Conditional entropy.  Association of dependent systems ;   18.4.  Conditional entropy.  Association of dependent systems .

Write them in the bottom, extra row of the table. Similarly, folding   18.4.  Conditional entropy.  Association of dependent systems in rows, we find:

  18.4.  Conditional entropy.  Association of dependent systems ;   18.4.  Conditional entropy.  Association of dependent systems ;   18.4.  Conditional entropy.  Association of dependent systems   18.4.  Conditional entropy.  Association of dependent systems

and write to the right an additional column. Sharing   18.4.  Conditional entropy.  Association of dependent systems on   18.4.  Conditional entropy.  Association of dependent systems , we will receive the table of conditional probabilities   18.4.  Conditional entropy.  Association of dependent systems :

  18.4.  Conditional entropy.  Association of dependent systems

  18.4.  Conditional entropy.  Association of dependent systems

  18.4.  Conditional entropy.  Association of dependent systems

  18.4.  Conditional entropy.  Association of dependent systems

  18.4.  Conditional entropy.  Association of dependent systems

  18.4.  Conditional entropy.  Association of dependent systems

  18.4.  Conditional entropy.  Association of dependent systems

  18.4.  Conditional entropy.  Association of dependent systems

  18.4.  Conditional entropy.  Association of dependent systems

  18.4.  Conditional entropy.  Association of dependent systems

  18.4.  Conditional entropy.  Association of dependent systems

  18.4.  Conditional entropy.  Association of dependent systems

  18.4.  Conditional entropy.  Association of dependent systems

  18.4.  Conditional entropy.  Association of dependent systems

  18.4.  Conditional entropy.  Association of dependent systems

  18.4.  Conditional entropy.  Association of dependent systems

  18.4.  Conditional entropy.  Association of dependent systems

By the formula (18.4.5 ') we find   18.4.  Conditional entropy.  Association of dependent systems . Since the conditional entropy at   18.4.  Conditional entropy.  Association of dependent systems and   18.4.  Conditional entropy.  Association of dependent systems equal to zero then

  18.4.  Conditional entropy.  Association of dependent systems .

Using table 7 of the application, we find

  18.4.  Conditional entropy.  Association of dependent systems (two units).

Similarly, we define   18.4.  Conditional entropy.  Association of dependent systems . From the formula (18.4.5 '), changing places   18.4.  Conditional entropy.  Association of dependent systems and   18.4.  Conditional entropy.  Association of dependent systems , we get:

  18.4.  Conditional entropy.  Association of dependent systems .

Make a table of conditional probabilities   18.4.  Conditional entropy.  Association of dependent systems . Sharing   18.4.  Conditional entropy.  Association of dependent systems on   18.4.  Conditional entropy.  Association of dependent systems we will receive:

  18.4.  Conditional entropy.  Association of dependent systems

  18.4.  Conditional entropy.  Association of dependent systems

  18.4.  Conditional entropy.  Association of dependent systems

  18.4.  Conditional entropy.  Association of dependent systems

  18.4.  Conditional entropy.  Association of dependent systems

  18.4.  Conditional entropy.  Association of dependent systems

  18.4.  Conditional entropy.  Association of dependent systems

  18.4.  Conditional entropy.  Association of dependent systems

  18.4.  Conditional entropy.  Association of dependent systems

  18.4.  Conditional entropy.  Association of dependent systems

  18.4.  Conditional entropy.  Association of dependent systems

  18.4.  Conditional entropy.  Association of dependent systems

  18.4.  Conditional entropy.  Association of dependent systems

  18.4.  Conditional entropy.  Association of dependent systems

  18.4.  Conditional entropy.  Association of dependent systems

  18.4.  Conditional entropy.  Association of dependent systems

  18.4.  Conditional entropy.  Association of dependent systems

From here

  18.4.  Conditional entropy.  Association of dependent systems (two words).

Using the concept of conditional entropy, one can determine the entropy of the combined system through the entropy of its constituent parts.

We prove the following theorem:

If two systems   18.4.  Conditional entropy.  Association of dependent systems and   18.4.  Conditional entropy.  Association of dependent systems combined into one, the entropy of the combined system is equal to the entropy of one of its components plus the conditional entropy of the second part relative to the first:

  18.4.  Conditional entropy.  Association of dependent systems . (18.4.8)

For proof, we write   18.4.  Conditional entropy.  Association of dependent systems in the form of mathematical expectation (18.3.3):

  18.4.  Conditional entropy.  Association of dependent systems .

By the probability multiplication theorem

  18.4.  Conditional entropy.  Association of dependent systems ,

Consequently,

  18.4.  Conditional entropy.  Association of dependent systems ,

from where

  18.4.  Conditional entropy.  Association of dependent systems

or, according to the formulas (18.2.11), (18.3.3)

  18.4.  Conditional entropy.  Association of dependent systems ,

Q.E.D.

In the particular case of systems   18.4.  Conditional entropy.  Association of dependent systems and   18.4.  Conditional entropy.  Association of dependent systems independent   18.4.  Conditional entropy.  Association of dependent systems and we get the one already proven in the previous one   18.4.  Conditional entropy.  Association of dependent systems entropy addition theorem:

  18.4.  Conditional entropy.  Association of dependent systems .

In general

  18.4.  Conditional entropy.  Association of dependent systems . (18.4.9)

The ratio (18.4.9) follows from the fact that the total conditional entropy   18.4.  Conditional entropy.  Association of dependent systems can not surpass the unconditional:

  18.4.  Conditional entropy.  Association of dependent systems . (18.4.10)

Inequality (18.4.10) will be proved in   18.4.  Conditional entropy.  Association of dependent systems 18.6. Intuitively, it seems pretty obvious: it is clear that the degree of uncertainty of the system cannot increase because the state of some other system has become known.

It follows from relation (18.4.9) that the entropy of a complex system reaches a maximum in the extreme case when its components are independent.

Consider another extreme case where the state of one of the systems (for example   18.4.  Conditional entropy.  Association of dependent systems ) completely determines the state of another (   18.4.  Conditional entropy.  Association of dependent systems ). In this case   18.4.  Conditional entropy.  Association of dependent systems and the formula (18.4.7) gives

  18.4.  Conditional entropy.  Association of dependent systems .

If the state of each system   18.4.  Conditional entropy.  Association of dependent systems uniquely identifies the state of another (or, as they say, systems   18.4.  Conditional entropy.  Association of dependent systems and   18.4.  Conditional entropy.  Association of dependent systems are equivalent)

  18.4.  Conditional entropy.  Association of dependent systems .

The entropy theorem of a complex system can be easily extended to any number of systems to be merged:

  18.4.  Conditional entropy.  Association of dependent systems , (18.4.11)

where the entropy of each subsequent system is calculated under the condition that the state of all the previous ones is known.


Comments


To leave a comment
If you have any suggestion, idea, thanks or comment, feel free to write. We really value feedback and are glad to hear your opinion.
To reply

Information and Coding Theory

Terms: Information and Coding Theory