# Entropy and Shannon’s Theorem CS 573: Algorithms, Fall 2013 Lecture 28 ```CS 573: Algorithms, Fall 2013
Entropy and Shannon’s
Theorem
Lecture 28
December 10, 2013
Sariel (UIUC)
CS573
1
Fall 2013
1 / 24
Part I
.
Entropy
.
Sariel (UIUC)
CS573
2
Fall 2013
2 / 24
Part II
.
Extracting randomness
.
Sariel (UIUC)
CS573
3
Fall 2013
3 / 24
Storing all strings of length n and j bits on
. Sn,j : set of all strings of length n with j ones in them.
2. T
n,j : preﬁx tree storing all Tn,j .
1
T0,0
T1,1
Sariel (UIUC)
T1,0
CS573
4
Fall 2013
4 / 24
Preﬁx tree ∀ binary strings of length n with j ones
Tn,j
0
Tn−1,j
1
Tn−1,j−1
Tn,n
Tn,0
0
1
Tn−1,0
Sariel (UIUC)
Tn−1,n−1
CS573
5
Fall 2013
5 / 24
Preﬁx tree ∀ binary strings of length n with j ones
Tn,j
0
1
Tn−1,j
# of leafs:
|Tn,j | = |Tn−1,j | + |Tn−1,j−1 |
Tn−1,j−1
Tn,n
Tn,0
0
1
Tn−1,0
Sariel (UIUC)
Tn−1,n−1
CS573
5
Fall 2013
5 / 24
Preﬁx tree ∀ binary strings of length n with j ones
Tn,j
0
1
# of leafs:
|Tn,j | = |Tn−1,j | + |Tn−1,j−1 |
( )
n
j
Tn−1,j
=
(
)
n−1
j
+
(
)
n−1
j−1
Tn−1,j−1
Tn,n
Tn,0
0
1
Tn−1,0
Sariel (UIUC)
Tn−1,n−1
CS573
5
Fall 2013
5 / 24
Preﬁx tree ∀ binary strings of length n with j ones
Tn,j
0
# of leafs:
|Tn,j | = |Tn−1,j | + |Tn−1,j−1 |
1
Tn−1,j
Tn−1,j−1
( )
n
j
=
(
+
=⇒ |Tn,j | =
(
)
n−1
j−1
( )
n
j
.
Tn,n
Tn,0
0
1
Tn−1,0
Sariel (UIUC)
)
n−1
j
Tn−1,n−1
CS573
5
Fall 2013
5 / 24
Encoding a string in Sn,j
.
2.
3.
4.
5.
1
Tn,j leafs corresponds to strings of Sn,j .
Order all strings of Sn,j order in lexicographical ordering
≡ ordering leafs of Tn,j from left to right.
Input: s ∈ Sn,j : compute index of s in sorted set Sn,j .
EncodeBinomCoeﬀ(s) denote this polytime procedure.
Sariel (UIUC)
CS573
6
Fall 2013
6 / 24
Encoding a string in Sn,j
.
2.
3.
4.
5.
1
Tn,j leafs corresponds to strings of Sn,j .
Order all strings of Sn,j order in lexicographical ordering
≡ ordering leafs of Tn,j from left to right.
Input: s ∈ Sn,j : compute index of s in sorted set Sn,j .
EncodeBinomCoeﬀ(s) denote this polytime procedure.
Sariel (UIUC)
CS573
6
Fall 2013
6 / 24
Encoding a string in Sn,j
. Tn,j leafs corresponds to strings of Sn,j .
2. Order all strings of S
n,j order in lexicographical ordering
3. ≡ ordering leafs of T
n,j from left to right.
Tn,j
1
0
Tn−1,j
1
Tn−1,j−1
. Input: s ∈ Sn,j : compute index of s in sorted set Sn,j .
5. EncodeBinomCoeﬀ(s) denote this polytime procedure.
4
Sariel (UIUC)
CS573
6
Fall 2013
6 / 24
Encoding a string in Sn,j
. Tn,j leafs corresponds to strings of Sn,j .
2. Order all strings of S
n,j order in lexicographical ordering
3. ≡ ordering leafs of T
n,j from left to right.
Tn,j
1
0
Tn−1,j
1
Tn−1,j−1
. Input: s ∈ Sn,j : compute index of s in sorted set Sn,j .
5. EncodeBinomCoeﬀ(s) denote this polytime procedure.
4
Sariel (UIUC)
CS573
6
Fall 2013
6 / 24
Encoding a string in Sn,j
. Tn,j leafs corresponds to strings of Sn,j .
2. Order all strings of S
n,j order in lexicographical ordering
3. ≡ ordering leafs of T
n,j from left to right.
Tn,j
1
0
Tn−1,j
1
Tn−1,j−1
. Input: s ∈ Sn,j : compute index of s in sorted set Sn,j .
5. EncodeBinomCoeﬀ(s) denote this polytime procedure.
4
Sariel (UIUC)
CS573
6
Fall 2013
6 / 24
Decoding a string in Sn,j
. Tn,j leafs corresponds to strings of Sn,j .
2. Order all strings of S
n,j order in lexicographical ordering
3. ≡ ordering leafs of T
n,j from left to right.
{
( )}
n
4. x ∈
1, . . . , j : compute xth string in Sn,j in polytime.
5. DecodeBinomCoeﬀ (x) denote this procedure.
1
Sariel (UIUC)
CS573
7
Fall 2013
7 / 24
Decoding a string in Sn,j
. Tn,j leafs corresponds to strings of Sn,j .
2. Order all strings of S
n,j order in lexicographical ordering
3. ≡ ordering leafs of T
n,j from left to right.
{
( )}
n
4. x ∈
1, . . . , j : compute xth string in Sn,j in polytime.
5. DecodeBinomCoeﬀ (x) denote this procedure.
1
Sariel (UIUC)
CS573
7
Fall 2013
7 / 24
Decoding a string in Sn,j
. Tn,j leafs corresponds to strings of Sn,j .
2. Order all strings of S
n,j order in lexicographical ordering
3. ≡ ordering leafs of T
n,j from left to right.
Tn,j
1
0
Tn−1,j
{
1
Tn−1,j−1
( )}
. x ∈ 1, . . . , n : compute xth string in Sn,j in polytime.
j
5. DecodeBinomCoeﬀ (x) denote this procedure.
4
Sariel (UIUC)
CS573
7
Fall 2013
7 / 24
Decoding a string in Sn,j
. Tn,j leafs corresponds to strings of Sn,j .
2. Order all strings of S
n,j order in lexicographical ordering
3. ≡ ordering leafs of T
n,j from left to right.
Tn,j
1
0
Tn−1,j
{
1
Tn−1,j−1
( )}
. x ∈ 1, . . . , n : compute xth string in Sn,j in polytime.
j
5. DecodeBinomCoeﬀ (x) denote this procedure.
4
Sariel (UIUC)
CS573
7
Fall 2013
7 / 24
Decoding a string in Sn,j
. Tn,j leafs corresponds to strings of Sn,j .
2. Order all strings of S
n,j order in lexicographical ordering
3. ≡ ordering leafs of T
n,j from left to right.
Tn,j
1
0
Tn−1,j
{
1
Tn−1,j−1
( )}
. x ∈ 1, . . . , n : compute xth string in Sn,j in polytime.
j
5. DecodeBinomCoeﬀ (x) denote this procedure.
4
Sariel (UIUC)
CS573
7
Fall 2013
7 / 24
Encoding/decoding strings of Sn,j
.
Lemma
.
Sn,j : Set of binary strings of length n with j ones, sorted
lexicographically.
1. EncodeBinomCoeﬀ(α): Input is string α ∈ S
n,j , compute
index x of α in Sn,j in polynomial time in n.
{
( )}
2. DecodeBinomCoeﬀ(x): Input index x ∈
1, . . . , n .
j
.
Output xth string α in Sn,j , in time O(polylog n + n).
Sariel (UIUC)
CS573
8
Fall 2013
8 / 24
Extracting randomness
.
Theorem
.
Consider a coin that comes up heads with probability p > 1/2. For
any constant δ > 0 and for n suﬃciently large:
(A) One can extract, from an input of a sequence of n ﬂips, an
output sequence of (1 − δ)nH(p) (unbiased) independent
random bits.
(B) One can not extract more than nH(p) bits from such a
sequence.
.
Sariel (UIUC)
CS573
9
Fall 2013
9 / 24
Proof...
( )
. There are n input strings with exactly j heads.
j
2. each has probability p j (1 − p)n−j .
3. map string s like that to index number in the set
{
( )}
Sj = 1, . . . , nj .
4. Given that input string s has j ones (out of n bits) deﬁnes a
uniform distribution on Sn,j .
5. x ← EncodeBinomCoeﬀ(s)
( )
n
6. x uniform distributed in {1, . . . , N }, N =
.
1
j
7. Seen in previous lecture...
8. ... extract in expectation, ⌊lg N ⌋ − 1 bits from uniform
random variable in the range 1, . . . , N .
.9 Extract bits using ExtractRandomness(x, N ):.
Sariel (UIUC)
CS573
10
Fall 2013
10 / 24
Proof...
( )
. There are n input strings with exactly j heads.
j
2. each has probability p j (1 − p)n−j .
3. map string s like that to index number in the set
{
( )}
Sj = 1, . . . , nj .
4. Given that input string s has j ones (out of n bits) deﬁnes a
uniform distribution on Sn,j .
5. x ← EncodeBinomCoeﬀ(s)
( )
n
6. x uniform distributed in {1, . . . , N }, N =
.
1
j
7. Seen in previous lecture...
8. ... extract in expectation, ⌊lg N ⌋ − 1 bits from uniform
random variable in the range 1, . . . , N .
.9 Extract bits using ExtractRandomness(x, N ):.
Sariel (UIUC)
CS573
10
Fall 2013
10 / 24
Proof...
( )
. There are n input strings with exactly j heads.
j
2. each has probability p j (1 − p)n−j .
3. map string s like that to index number in the set
{
( )}
Sj = 1, . . . , nj .
4. Given that input string s has j ones (out of n bits) deﬁnes a
uniform distribution on Sn,j .
5. x ← EncodeBinomCoeﬀ(s)
( )
n
6. x uniform distributed in {1, . . . , N }, N =
.
1
j
7. Seen in previous lecture...
8. ... extract in expectation, ⌊lg N ⌋ − 1 bits from uniform
random variable in the range 1, . . . , N .
.9 Extract bits using ExtractRandomness(x, N ):.
Sariel (UIUC)
CS573
10
Fall 2013
10 / 24
Proof...
( )
. There are n input strings with exactly j heads.
j
2. each has probability p j (1 − p)n−j .
3. map string s like that to index number in the set
{
( )}
Sj = 1, . . . , nj .
4. Given that input string s has j ones (out of n bits) deﬁnes a
uniform distribution on Sn,j .
5. x ← EncodeBinomCoeﬀ(s)
( )
n
6. x uniform distributed in {1, . . . , N }, N =
.
1
j
7. Seen in previous lecture...
8. ... extract in expectation, ⌊lg N ⌋ − 1 bits from uniform
random variable in the range 1, . . . , N .
.9 Extract bits using ExtractRandomness(x, N ):.
Sariel (UIUC)
CS573
10
Fall 2013
10 / 24
Proof...
( )
. There are n input strings with exactly j heads.
j
2. each has probability p j (1 − p)n−j .
3. map string s like that to index number in the set
{
( )}
Sj = 1, . . . , nj .
4. Given that input string s has j ones (out of n bits) deﬁnes a
uniform distribution on Sn,j .
5. x ← EncodeBinomCoeﬀ(s)
( )
n
6. x uniform distributed in {1, . . . , N }, N =
.
1
j
7. Seen in previous lecture...
8. ... extract in expectation, ⌊lg N ⌋ − 1 bits from uniform
random variable in the range 1, . . . , N .
.9 Extract bits using ExtractRandomness(x, N ):.
Sariel (UIUC)
CS573
10
Fall 2013
10 / 24
Proof...
( )
. There are n input strings with exactly j heads.
j
2. each has probability p j (1 − p)n−j .
3. map string s like that to index number in the set
{
( )}
Sj = 1, . . . , nj .
4. Given that input string s has j ones (out of n bits) deﬁnes a
uniform distribution on Sn,j .
5. x ← EncodeBinomCoeﬀ(s)
( )
n
6. x uniform distributed in {1, . . . , N }, N =
.
1
j
7. Seen in previous lecture...
8. ... extract in expectation, ⌊lg N ⌋ − 1 bits from uniform
random variable in the range 1, . . . , N .
.9 Extract bits using ExtractRandomness(x, N ):.
Sariel (UIUC)
CS573
10
Fall 2013
10 / 24
Proof...
( )
. There are n input strings with exactly j heads.
j
2. each has probability p j (1 − p)n−j .
3. map string s like that to index number in the set
{
( )}
Sj = 1, . . . , nj .
4. Given that input string s has j ones (out of n bits) deﬁnes a
uniform distribution on Sn,j .
5. x ← EncodeBinomCoeﬀ(s)
( )
n
6. x uniform distributed in {1, . . . , N }, N =
.
1
j
7. Seen in previous lecture...
8. ... extract in expectation, ⌊lg N ⌋ − 1 bits from uniform
random variable in the range 1, . . . , N .
.9 Extract bits using ExtractRandomness(x, N ):.
Sariel (UIUC)
CS573
10
Fall 2013
10 / 24
Proof...
( )
. There are n input strings with exactly j heads.
j
2. each has probability p j (1 − p)n−j .
3. map string s like that to index number in the set
{
( )}
Sj = 1, . . . , nj .
4. Given that input string s has j ones (out of n bits) deﬁnes a
uniform distribution on Sn,j .
5. x ← EncodeBinomCoeﬀ(s)
( )
n
6. x uniform distributed in {1, . . . , N }, N =
.
1
j
7. Seen in previous lecture...
8. ... extract in expectation, ⌊lg N ⌋ − 1 bits from uniform
random variable in the range 1, . . . , N .
.9 Extract bits using ExtractRandomness(x, N ):.
Sariel (UIUC)
CS573
10
Fall 2013
10 / 24
Proof...
( )
. There are n input strings with exactly j heads.
j
2. each has probability p j (1 − p)n−j .
3. map string s like that to index number in the set
{
( )}
Sj = 1, . . . , nj .
4. Given that input string s has j ones (out of n bits) deﬁnes a
uniform distribution on Sn,j .
5. x ← EncodeBinomCoeﬀ(s)
( )
n
6. x uniform distributed in {1, . . . , N }, N =
.
1
j
7. Seen in previous lecture...
8. ... extract in expectation, ⌊lg N ⌋ − 1 bits from uniform
random variable in the range 1, . . . , N .
.9 Extract bits using ExtractRandomness(x, N ):.
Sariel (UIUC)
CS573
10
Fall 2013
10 / 24
Exciting proof continued...
. Z : random variable: number of heads in input string s.
2. B: number of random bits extracted.
1
[
]
EB =
[
n
∑
]
Pr[Z = k] E B Z = k ,
k=0
⌊
( )⌋
[ ]
n
3. Know:
B
Z
=
k
≥ lg
− 1.
E
k
4. ε < p − 1/2: suﬃciently small constant.
5. n(p − ε) ≤ k ≤ n(p + ε):
( )
n
k
(
≥
)
n
⌊n(p + ε)⌋
≥
2nH(p+ε)
n+1
. ... since 2nH(p) is a good approximation to
previous lecture.
6
Sariel (UIUC)
CS573
11
(
n
np
)
,
as proved in
Fall 2013
11 / 24
Exciting proof continued...
. Z : random variable: number of heads in input string s.
2. B: number of random bits extracted.
1
[
]
EB =
[
n
∑
]
Pr[Z = k] E B Z = k ,
k=0
⌊
( )⌋
[ ]
n
3. Know:
B
Z
=
k
≥ lg
− 1.
E
k
4. ε < p − 1/2: suﬃciently small constant.
5. n(p − ε) ≤ k ≤ n(p + ε):
( )
n
k
(
≥
)
n
⌊n(p + ε)⌋
≥
2nH(p+ε)
n+1
. ... since 2nH(p) is a good approximation to
previous lecture.
6
Sariel (UIUC)
CS573
11
(
n
np
)
,
as proved in
Fall 2013
11 / 24
Exciting proof continued...
. Z : random variable: number of heads in input string s.
2. B: number of random bits extracted.
1
[
]
EB =
[
n
∑
]
Pr[Z = k] E B Z = k ,
k=0
⌊
( )⌋
[ ]
n
3. Know:
B
Z
=
k
≥ lg
− 1.
E
k
4. ε < p − 1/2: suﬃciently small constant.
5. n(p − ε) ≤ k ≤ n(p + ε):
( )
n
k
(
≥
)
n
⌊n(p + ε)⌋
≥
2nH(p+ε)
n+1
. ... since 2nH(p) is a good approximation to
previous lecture.
6
Sariel (UIUC)
CS573
11
(
n
np
)
,
as proved in
Fall 2013
11 / 24
Exciting proof continued...
. Z : random variable: number of heads in input string s.
2. B: number of random bits extracted.
1
[
]
EB =
[
n
∑
]
Pr[Z = k] E B Z = k ,
k=0
⌊
( )⌋
[ ]
n
3. Know:
B
Z
=
k
≥ lg
− 1.
E
k
4. ε < p − 1/2: suﬃciently small constant.
5. n(p − ε) ≤ k ≤ n(p + ε):
( )
n
k
(
≥
)
n
⌊n(p + ε)⌋
≥
2nH(p+ε)
n+1
. ... since 2nH(p) is a good approximation to
previous lecture.
6
Sariel (UIUC)
CS573
11
(
n
np
)
,
as proved in
Fall 2013
11 / 24
Exciting proof continued...
. Z : random variable: number of heads in input string s.
2. B: number of random bits extracted.
1
[
]
EB =
[
n
∑
]
Pr[Z = k] E B Z = k ,
k=0
⌊
( )⌋
[ ]
n
3. Know:
B
Z
=
k
≥ lg
− 1.
E
k
4. ε < p − 1/2: suﬃciently small constant.
5. n(p − ε) ≤ k ≤ n(p + ε):
( )
n
k
(
≥
)
n
⌊n(p + ε)⌋
≥
2nH(p+ε)
n+1
. ... since 2nH(p) is a good approximation to
previous lecture.
6
Sariel (UIUC)
CS573
11
(
n
np
)
,
as proved in
Fall 2013
11 / 24
Exciting proof continued...
. Z : random variable: number of heads in input string s.
2. B: number of random bits extracted.
1
[
]
EB =
[
n
∑
]
Pr[Z = k] E B Z = k ,
k=0
⌊
( )⌋
[ ]
n
3. Know:
B
Z
=
k
≥ lg
− 1.
E
k
4. ε < p − 1/2: suﬃciently small constant.
5. n(p − ε) ≤ k ≤ n(p + ε):
( )
n
k
(
≥
)
n
⌊n(p + ε)⌋
≥
2nH(p+ε)
n+1
. ... since 2nH(p) is a good approximation to
previous lecture.
6
Sariel (UIUC)
CS573
11
(
n
np
)
,
as proved in
Fall 2013
11 / 24
Exciting proof continued...
. Z : random variable: number of heads in input string s.
2. B: number of random bits extracted.
1
[
]
EB =
[
n
∑
]
Pr[Z = k] E B Z = k ,
k=0
⌊
( )⌋
[ ]
n
3. Know:
B
Z
=
k
≥ lg
− 1.
E
k
4. ε < p − 1/2: suﬃciently small constant.
5. n(p − ε) ≤ k ≤ n(p + ε):
( )
n
k
(
≥
)
n
⌊n(p + ε)⌋
≥
2nH(p+ε)
n+1
. ... since 2nH(p) is a good approximation to
previous lecture.
6
Sariel (UIUC)
CS573
11
(
n
np
)
,
as proved in
Fall 2013
11 / 24
Super exciting proof continued...
[
]
∑n
[
]
∑⌈n(p+ε)⌉
EB =
EB ≥
≥
]
[
[
k=⌊n(p−ε)⌋ Pr Z = k E B Z = k
⌈n(p+ε)⌉
∑
≥
]
[
k=0 Pr[Z = k] E B Z = k .
[
Pr Z = k
k=⌊n(p−ε)⌋
⌈n(p+ε)⌉
∑
[
Pr Z = k
k=⌊n(p−ε)⌋
(
]
]
(⌊
( )⌋
n
lg
(
lg
k
]
)
−1
)
2nH(p+ε)
−2
n+1
)
= nH(p + ε) − lg(n + 1) − 2 Pr[|Z − np| ≤ εn]
(
(
))
(
)
nε2
≥ nH(p + ε) − lg(n + 1) − 2 1 − 2 exp −
,
4p
since µ = E[Z ] = np and
(
)
[
]
Pr |Z − np| ≥ pε pn ≤ 2 exp − np
4
( )2
ε
p
(
2
)
, by
= 2 exp − nε
4p
the Chernoﬀ inequality.
Sariel (UIUC)
CS573
12
Fall 2013
12 / 24
Super exciting proof continued...
[
]
∑n
[
]
∑⌈n(p+ε)⌉
EB =
EB ≥
≥
]
[
[
k=⌊n(p−ε)⌋ Pr Z = k E B Z = k
⌈n(p+ε)⌉
∑
≥
]
[
k=0 Pr[Z = k] E B Z = k .
[
Pr Z = k
k=⌊n(p−ε)⌋
⌈n(p+ε)⌉
∑
[
Pr Z = k
k=⌊n(p−ε)⌋
(
]
]
(⌊
( )⌋
n
lg
(
lg
k
]
)
−1
)
2nH(p+ε)
−2
n+1
)
= nH(p + ε) − lg(n + 1) − 2 Pr[|Z − np| ≤ εn]
(
(
))
(
)
nε2
≥ nH(p + ε) − lg(n + 1) − 2 1 − 2 exp −
,
4p
since µ = E[Z ] = np and
(
)
[
]
Pr |Z − np| ≥ pε pn ≤ 2 exp − np
4
( )2
ε
p
(
2
)
, by
= 2 exp − nε
4p
the Chernoﬀ inequality.
Sariel (UIUC)
CS573
12
Fall 2013
12 / 24
Super exciting proof continued...
[
]
∑n
[
]
∑⌈n(p+ε)⌉
EB =
EB ≥
≥
]
[
[
k=⌊n(p−ε)⌋ Pr Z = k E B Z = k
⌈n(p+ε)⌉
∑
≥
]
[
k=0 Pr[Z = k] E B Z = k .
[
Pr Z = k
k=⌊n(p−ε)⌋
⌈n(p+ε)⌉
∑
[
Pr Z = k
k=⌊n(p−ε)⌋
(
]
]
(⌊
( )⌋
n
lg
(
lg
k
]
)
−1
)
2nH(p+ε)
−2
n+1
)
= nH(p + ε) − lg(n + 1) − 2 Pr[|Z − np| ≤ εn]
(
(
))
(
)
nε2
≥ nH(p + ε) − lg(n + 1) − 2 1 − 2 exp −
,
4p
since µ = E[Z ] = np and
(
)
[
]
Pr |Z − np| ≥ pε pn ≤ 2 exp − np
4
( )2
ε
p
(
2
)
, by
= 2 exp − nε
4p
the Chernoﬀ inequality.
Sariel (UIUC)
CS573
12
Fall 2013
12 / 24
Super exciting proof continued...
[
]
∑n
[
]
∑⌈n(p+ε)⌉
EB =
EB ≥
≥
]
[
[
k=⌊n(p−ε)⌋ Pr Z = k E B Z = k
⌈n(p+ε)⌉
∑
≥
]
[
k=0 Pr[Z = k] E B Z = k .
[
Pr Z = k
k=⌊n(p−ε)⌋
⌈n(p+ε)⌉
∑
[
Pr Z = k
k=⌊n(p−ε)⌋
(
]
]
(⌊
( )⌋
n
lg
(
lg
k
]
)
−1
)
2nH(p+ε)
−2
n+1
)
= nH(p + ε) − lg(n + 1) − 2 Pr[|Z − np| ≤ εn]
(
(
))
(
)
nε2
≥ nH(p + ε) − lg(n + 1) − 2 1 − 2 exp −
,
4p
since µ = E[Z ] = np and
(
)
[
]
Pr |Z − np| ≥ pε pn ≤ 2 exp − np
4
( )2
ε
p
(
2
)
, by
= 2 exp − nε
4p
the Chernoﬀ inequality.
Sariel (UIUC)
CS573
12
Fall 2013
12 / 24
Super exciting proof continued...
[
]
∑n
[
]
∑⌈n(p+ε)⌉
EB =
EB ≥
≥
]
[
[
k=⌊n(p−ε)⌋ Pr Z = k E B Z = k
⌈n(p+ε)⌉
∑
≥
]
[
k=0 Pr[Z = k] E B Z = k .
[
Pr Z = k
k=⌊n(p−ε)⌋
⌈n(p+ε)⌉
∑
[
Pr Z = k
k=⌊n(p−ε)⌋
(
]
]
(⌊
( )⌋
n
lg
(
lg
k
]
)
−1
)
2nH(p+ε)
−2
n+1
)
= nH(p + ε) − lg(n + 1) − 2 Pr[|Z − np| ≤ εn]
(
(
))
(
)
nε2
≥ nH(p + ε) − lg(n + 1) − 2 1 − 2 exp −
,
4p
since µ = E[Z ] = np and
(
)
[
]
Pr |Z − np| ≥ pε pn ≤ 2 exp − np
4
( )2
ε
p
(
2
)
, by
= 2 exp − nε
4p
the Chernoﬀ inequality.
Sariel (UIUC)
CS573
12
Fall 2013
12 / 24
Super exciting proof continued...
[
]
∑n
[
]
∑⌈n(p+ε)⌉
EB =
EB ≥
≥
]
[
[
k=⌊n(p−ε)⌋ Pr Z = k E B Z = k
⌈n(p+ε)⌉
∑
≥
]
[
k=0 Pr[Z = k] E B Z = k .
[
Pr Z = k
k=⌊n(p−ε)⌋
⌈n(p+ε)⌉
∑
[
Pr Z = k
k=⌊n(p−ε)⌋
(
]
]
(⌊
( )⌋
n
lg
(
lg
k
]
)
−1
)
2nH(p+ε)
−2
n+1
)
= nH(p + ε) − lg(n + 1) − 2 Pr[|Z − np| ≤ εn]
(
(
))
(
)
nε2
≥ nH(p + ε) − lg(n + 1) − 2 1 − 2 exp −
,
4p
since µ = E[Z ] = np and
(
)
[
]
Pr |Z − np| ≥ pε pn ≤ 2 exp − np
4
( )2
ε
p
(
2
)
, by
= 2 exp − nε
4p
the Chernoﬀ inequality.
Sariel (UIUC)
CS573
12
Fall 2013
12 / 24
Super exciting proof continued...
[
]
∑n
[
]
∑⌈n(p+ε)⌉
EB =
EB ≥
≥
]
[
[
k=⌊n(p−ε)⌋ Pr Z = k E B Z = k
⌈n(p+ε)⌉
∑
≥
]
[
k=0 Pr[Z = k] E B Z = k .
[
Pr Z = k
k=⌊n(p−ε)⌋
⌈n(p+ε)⌉
∑
[
Pr Z = k
k=⌊n(p−ε)⌋
(
]
]
(⌊
( )⌋
n
lg
(
lg
k
]
)
−1
)
2nH(p+ε)
−2
n+1
)
= nH(p + ε) − lg(n + 1) − 2 Pr[|Z − np| ≤ εn]
(
(
))
(
)
nε2
≥ nH(p + ε) − lg(n + 1) − 2 1 − 2 exp −
,
4p
since µ = E[Z ] = np and
(
)
[
]
Pr |Z − np| ≥ pε pn ≤ 2 exp − np
4
( )2
ε
p
(
2
)
, by
= 2 exp − nε
4p
the Chernoﬀ inequality.
Sariel (UIUC)
CS573
12
Fall 2013
12 / 24
Hyper super exciting proof continued...
. Fix ε > 0, such that H(p + ε) > (1 − δ/4)H(p), p is ﬁxed.
2.
=⇒ nH(p) = Ω(n),
δ
3. For n suﬃciently large: − lg(n + 1) ≥ −
nH(p).
10
(
)
2
nε
δ
4. ... also 2 exp −
≤ .
1
4p
10
. For n large enough;
5
(
E[B] ≥ 1 −
δ
δ
)
(
δ
)
−
nH(p) 1 −
4
10
10
≥ (1 − δ) nH(p) ,
Sariel (UIUC)
CS573
13
Fall 2013
13 / 24
Hyper super exciting proof continued...
. Fix ε > 0, such that H(p + ε) > (1 − δ/4)H(p), p is ﬁxed.
2.
=⇒ nH(p) = Ω(n),
δ
3. For n suﬃciently large: − lg(n + 1) ≥ −
nH(p).
10
(
)
2
nε
δ
4. ... also 2 exp −
≤ .
1
4p
10
. For n large enough;
5
(
E[B] ≥ 1 −
δ
δ
)
(
δ
)
−
nH(p) 1 −
4
10
10
≥ (1 − δ) nH(p) ,
Sariel (UIUC)
CS573
13
Fall 2013
13 / 24
Hyper super exciting proof continued...
. Fix ε > 0, such that H(p + ε) > (1 − δ/4)H(p), p is ﬁxed.
2.
=⇒ nH(p) = Ω(n),
δ
3. For n suﬃciently large: − lg(n + 1) ≥ −
nH(p).
10
(
)
2
nε
δ
4. ... also 2 exp −
≤ .
1
4p
10
. For n large enough;
5
(
E[B] ≥ 1 −
δ
δ
)
(
δ
)
−
nH(p) 1 −
4
10
10
≥ (1 − δ) nH(p) ,
Sariel (UIUC)
CS573
13
Fall 2013
13 / 24
Hyper super exciting proof continued...
. Fix ε > 0, such that H(p + ε) > (1 − δ/4)H(p), p is ﬁxed.
2.
=⇒ nH(p) = Ω(n),
δ
3. For n suﬃciently large: − lg(n + 1) ≥ −
nH(p).
10
(
)
2
nε
δ
4. ... also 2 exp −
≤ .
1
4p
10
. For n large enough;
5
(
E[B] ≥ 1 −
δ
δ
)
(
δ
)
−
nH(p) 1 −
4
10
10
≥ (1 − δ) nH(p) ,
Sariel (UIUC)
CS573
13
Fall 2013
13 / 24
Hyper super exciting proof continued...
. Fix ε > 0, such that H(p + ε) > (1 − δ/4)H(p), p is ﬁxed.
2.
=⇒ nH(p) = Ω(n),
δ
3. For n suﬃciently large: − lg(n + 1) ≥ −
nH(p).
10
(
)
2
nε
δ
4. ... also 2 exp −
≤ .
1
4p
10
. For n large enough;
5
(
E[B] ≥ 1 −
δ
δ
)
(
δ
)
−
nH(p) 1 −
4
10
10
≥ (1 − δ) nH(p) ,
Sariel (UIUC)
CS573
13
Fall 2013
13 / 24
Hyper super duper exciting proof continued...
. Need to prove upper bound.
2. If input sequence x has probability Pr[X = x], then
y = Ext(x) has probability to be generated ≥ Pr[X = x].
.3 All sequences of length |y| have equal probability to be
generated (by deﬁnition).
.4 2|Ext(x)| Pr[X = x] ≤ 2|Ext(x)| Pr[y = Ext(x)] ≤ 1.
5.
=⇒ |Ext(x)| ≤ lg(1/ Pr[X = x])
[ ]
[
]
.6 E B = ∑x Pr X = x |Ext(x)|
1
≤
∑
[
]
x Pr X = x lg
Sariel (UIUC)
1
Pr[X=x]
CS573
= H(X) .
14
Fall 2013
14 / 24
Hyper super duper exciting proof continued...
. Need to prove upper bound.
2. If input sequence x has probability Pr[X = x], then
y = Ext(x) has probability to be generated ≥ Pr[X = x].
.3 All sequences of length |y| have equal probability to be
generated (by deﬁnition).
.4 2|Ext(x)| Pr[X = x] ≤ 2|Ext(x)| Pr[y = Ext(x)] ≤ 1.
5.
=⇒ |Ext(x)| ≤ lg(1/ Pr[X = x])
[ ]
[
]
.6 E B = ∑x Pr X = x |Ext(x)|
1
≤
∑
[
]
x Pr X = x lg
Sariel (UIUC)
1
Pr[X=x]
CS573
= H(X) .
14
Fall 2013
14 / 24
Hyper super duper exciting proof continued...
. Need to prove upper bound.
2. If input sequence x has probability Pr[X = x], then
y = Ext(x) has probability to be generated ≥ Pr[X = x].
.3 All sequences of length |y| have equal probability to be
generated (by deﬁnition).
.4 2|Ext(x)| Pr[X = x] ≤ 2|Ext(x)| Pr[y = Ext(x)] ≤ 1.
5.
=⇒ |Ext(x)| ≤ lg(1/ Pr[X = x])
[ ]
[
]
.6 E B = ∑x Pr X = x |Ext(x)|
1
≤
∑
[
]
x Pr X = x lg
Sariel (UIUC)
1
Pr[X=x]
CS573
= H(X) .
14
Fall 2013
14 / 24
Hyper super duper exciting proof continued...
. Need to prove upper bound.
2. If input sequence x has probability Pr[X = x], then
y = Ext(x) has probability to be generated ≥ Pr[X = x].
.3 All sequences of length |y| have equal probability to be
generated (by deﬁnition).
.4 2|Ext(x)| Pr[X = x] ≤ 2|Ext(x)| Pr[y = Ext(x)] ≤ 1.
5.
=⇒ |Ext(x)| ≤ lg(1/ Pr[X = x])
[ ]
[
]
.6 E B = ∑x Pr X = x |Ext(x)|
1
≤
∑
[
]
x Pr X = x lg
Sariel (UIUC)
1
Pr[X=x]
CS573
= H(X) .
14
Fall 2013
14 / 24
Hyper super duper exciting proof continued...
. Need to prove upper bound.
2. If input sequence x has probability Pr[X = x], then
y = Ext(x) has probability to be generated ≥ Pr[X = x].
.3 All sequences of length |y| have equal probability to be
generated (by deﬁnition).
.4 2|Ext(x)| Pr[X = x] ≤ 2|Ext(x)| Pr[y = Ext(x)] ≤ 1.
5.
=⇒ |Ext(x)| ≤ lg(1/ Pr[X = x])
[ ]
[
]
.6 E B = ∑x Pr X = x |Ext(x)|
1
≤
∑
[
]
x Pr X = x lg
Sariel (UIUC)
1
Pr[X=x]
CS573
= H(X) .
14
Fall 2013
14 / 24
Hyper super duper exciting proof continued...
. Need to prove upper bound.
2. If input sequence x has probability Pr[X = x], then
y = Ext(x) has probability to be generated ≥ Pr[X = x].
.3 All sequences of length |y| have equal probability to be
generated (by deﬁnition).
.4 2|Ext(x)| Pr[X = x] ≤ 2|Ext(x)| Pr[y = Ext(x)] ≤ 1.
5.
=⇒ |Ext(x)| ≤ lg(1/ Pr[X = x])
[ ]
[
]
.6 E B = ∑x Pr X = x |Ext(x)|
1
≤
∑
[
]
x Pr X = x lg
Sariel (UIUC)
1
Pr[X=x]
CS573
= H(X) .
14
Fall 2013
14 / 24
Hyper super duper exciting proof continued...
. Need to prove upper bound.
2. If input sequence x has probability Pr[X = x], then
y = Ext(x) has probability to be generated ≥ Pr[X = x].
.3 All sequences of length |y| have equal probability to be
generated (by deﬁnition).
.4 2|Ext(x)| Pr[X = x] ≤ 2|Ext(x)| Pr[y = Ext(x)] ≤ 1.
5.
=⇒ |Ext(x)| ≤ lg(1/ Pr[X = x])
[ ]
[
]
.6 E B = ∑x Pr X = x |Ext(x)|
1
≤
∑
[
]
x Pr X = x lg
Sariel (UIUC)
1
Pr[X=x]
CS573
= H(X) .
14
Fall 2013
14 / 24
Hyper super duper exciting proof continued...
. Need to prove upper bound.
2. If input sequence x has probability Pr[X = x], then
y = Ext(x) has probability to be generated ≥ Pr[X = x].
.3 All sequences of length |y| have equal probability to be
generated (by deﬁnition).
.4 2|Ext(x)| Pr[X = x] ≤ 2|Ext(x)| Pr[y = Ext(x)] ≤ 1.
5.
=⇒ |Ext(x)| ≤ lg(1/ Pr[X = x])
[ ]
[
]
.6 E B = ∑x Pr X = x |Ext(x)|
1
≤
∑
[
]
x Pr X = x lg
Sariel (UIUC)
1
Pr[X=x]
CS573
= H(X) .
14
Fall 2013
14 / 24
Hyper super duper exciting proof continued...
. Need to prove upper bound.
2. If input sequence x has probability Pr[X = x], then
y = Ext(x) has probability to be generated ≥ Pr[X = x].
.3 All sequences of length |y| have equal probability to be
generated (by deﬁnition).
.4 2|Ext(x)| Pr[X = x] ≤ 2|Ext(x)| Pr[y = Ext(x)] ≤ 1.
5.
=⇒ |Ext(x)| ≤ lg(1/ Pr[X = x])
[ ]
[
]
.6 E B = ∑x Pr X = x |Ext(x)|
1
≤
∑
[
]
x Pr X = x lg
Sariel (UIUC)
1
Pr[X=x]
CS573
= H(X) .
14
Fall 2013
14 / 24
Part III
.
Coding: Shannon’s Theorem
.
Sariel (UIUC)
CS573
15
Fall 2013
15 / 24
Shannon’s Theorem
.
Deﬁnition
.
The input to a binary symmetric channel with parameter p is a
sequence of bits x1 , x2 , . . . , and the output is a sequence of bits
y1 , y2 , . . . , such that Pr[xi = yi ] = 1 − p independently for each
i.
.
Sariel (UIUC)
CS573
16
Fall 2013
16 / 24
Encoding/decoding with noise
.
Deﬁnition
.
A (k, n) encoding function Enc : {0, 1}k → {0, 1}n takes as
input a sequence of k bits and outputs a sequence of n bits. A
(k, n) decoding function Dec : {0, 1}n → {0, 1}k takes as input
.a sequence of n bits and outputs a sequence of k bits.
Sariel (UIUC)
CS573
17
Fall 2013
17 / 24
Claude Elwood Shannon
Claude Elwood Shannon (April 30, 1916 - February 24, 2001), an
American electrical engineer and mathematician, has been called “the
father of information theory”.
His master thesis was how to building boolean circuits for any
boolean function.
Sariel (UIUC)
CS573
18
Fall 2013
18 / 24
Shannon’s theorem (1948)
.
Theorem (Shannon’s theorem)
.
For a binary symmetric channel with parameter p < 1/2 and for any
constants δ, γ > 0, where n is suﬃciently large, the following holds:
(i) For an k ≤ n(1 − H(p) − δ) there exists (k, n) encoding
and decoding functions such that the probability the receiver
fails to obtain the correct message is at most γ for every
possible k-bit input messages.
(ii) There are no (k, n) encoding and decoding functions with
k ≥ n(1 − H(p) + δ) such that the probability of decoding
correctly is at least γ for a k-bit input message chosen
uniformly at random.
.
Sariel (UIUC)
CS573
19
Fall 2013
19 / 24
When the sender sends a string...
S = s1 s2 . . . sn
S
Sariel (UIUC)
CS573
20
Fall 2013
20 / 24
When the sender sends a string...
S = s1 s2 . . . sn
S
Sariel (UIUC)
CS573
np
20
Fall 2013
20 / 24
When the sender sends a string...
S = s1 s2 . . . sn
S
Sariel (UIUC)
CS573
np
20
Fall 2013
20 / 24
When the sender sends a string...
S = s1 s2 . . . sn
S
np
(1 − δ)np (1 + δ)np
Sariel (UIUC)
CS573
20
Fall 2013
20 / 24
When the sender sends a string...
S = s1 s2 . . . sn
S
np
(1 − δ)np (1 + δ)np
One ring to rule them all!
Sariel (UIUC)
CS573
20
Fall 2013
20 / 24
Some intuition...
.
2.
3.
4.
5.
6.
senders sent string S = s1 s2 . . . sn .
receiver got string T = t1 t2 . . . tn .
p = Pr[ti ̸= si ], for all i.
]
∑[
U : Hamming distance between S and T : U = i si ̸= ti .
By assumption: E[U ] = pn,[ and U is a binomial variable.
]
By Chernoﬀ inequality: U ∈ (1 − δ)np, (1 + δ)np with
high probability, where δ is tiny constant.
7. T is in a ring R centered at S, with inner radius (1 − δ)np
and outer radius (1 + δ)np.
8. This ring has
1
(1+δ)np
∑
i=(1−δ)np
( )
n
i
(
≤2
)
n
(1 + δ)np
≤ α = 2 · 2nH((1+δ)p) .
strings in it.
Sariel (UIUC)
CS573
21
Fall 2013
21 / 24
Some intuition...
.
2.
3.
4.
5.
6.
senders sent string S = s1 s2 . . . sn .
receiver got string T = t1 t2 . . . tn .
p = Pr[ti ̸= si ], for all i.
]
∑[
U : Hamming distance between S and T : U = i si ̸= ti .
By assumption: E[U ] = pn,[ and U is a binomial variable.
]
By Chernoﬀ inequality: U ∈ (1 − δ)np, (1 + δ)np with
high probability, where δ is tiny constant.
7. T is in a ring R centered at S, with inner radius (1 − δ)np
and outer radius (1 + δ)np.
8. This ring has
1
(1+δ)np
∑
i=(1−δ)np
( )
n
i
(
≤2
)
n
(1 + δ)np
≤ α = 2 · 2nH((1+δ)p) .
strings in it.
Sariel (UIUC)
CS573
21
Fall 2013
21 / 24
Some intuition...
.
2.
3.
4.
5.
6.
senders sent string S = s1 s2 . . . sn .
receiver got string T = t1 t2 . . . tn .
p = Pr[ti ̸= si ], for all i.
]
∑[
U : Hamming distance between S and T : U = i si ̸= ti .
By assumption: E[U ] = pn,[ and U is a binomial variable.
]
By Chernoﬀ inequality: U ∈ (1 − δ)np, (1 + δ)np with
high probability, where δ is tiny constant.
7. T is in a ring R centered at S, with inner radius (1 − δ)np
and outer radius (1 + δ)np.
8. This ring has
1
(1+δ)np
∑
i=(1−δ)np
( )
n
i
(
≤2
)
n
(1 + δ)np
≤ α = 2 · 2nH((1+δ)p) .
strings in it.
Sariel (UIUC)
CS573
21
Fall 2013
21 / 24
Some intuition...
.
2.
3.
4.
5.
6.
senders sent string S = s1 s2 . . . sn .
receiver got string T = t1 t2 . . . tn .
p = Pr[ti ̸= si ], for all i.
]
∑[
U : Hamming distance between S and T : U = i si ̸= ti .
By assumption: E[U ] = pn,[ and U is a binomial variable.
]
By Chernoﬀ inequality: U ∈ (1 − δ)np, (1 + δ)np with
high probability, where δ is tiny constant.
7. T is in a ring R centered at S, with inner radius (1 − δ)np
and outer radius (1 + δ)np.
8. This ring has
1
(1+δ)np
∑
i=(1−δ)np
( )
n
i
(
≤2
)
n
(1 + δ)np
≤ α = 2 · 2nH((1+δ)p) .
strings in it.
Sariel (UIUC)
CS573
21
Fall 2013
21 / 24
Some intuition...
.
2.
3.
4.
5.
6.
senders sent string S = s1 s2 . . . sn .
receiver got string T = t1 t2 . . . tn .
p = Pr[ti ̸= si ], for all i.
]
∑[
U : Hamming distance between S and T : U = i si ̸= ti .
By assumption: E[U ] = pn,[ and U is a binomial variable.
]
By Chernoﬀ inequality: U ∈ (1 − δ)np, (1 + δ)np with
high probability, where δ is tiny constant.
7. T is in a ring R centered at S, with inner radius (1 − δ)np
and outer radius (1 + δ)np.
8. This ring has
1
(1+δ)np
∑
i=(1−δ)np
( )
n
i
(
≤2
)
n
(1 + δ)np
≤ α = 2 · 2nH((1+δ)p) .
strings in it.
Sariel (UIUC)
CS573
21
Fall 2013
21 / 24
Some intuition...
.
2.
3.
4.
5.
6.
senders sent string S = s1 s2 . . . sn .
receiver got string T = t1 t2 . . . tn .
p = Pr[ti ̸= si ], for all i.
]
∑[
U : Hamming distance between S and T : U = i si ̸= ti .
By assumption: E[U ] = pn,[ and U is a binomial variable.
]
By Chernoﬀ inequality: U ∈ (1 − δ)np, (1 + δ)np with
high probability, where δ is tiny constant.
7. T is in a ring R centered at S, with inner radius (1 − δ)np
and outer radius (1 + δ)np.
8. This ring has
1
(1+δ)np
∑
i=(1−δ)np
( )
n
i
(
≤2
)
n
(1 + δ)np
≤ α = 2 · 2nH((1+δ)p) .
strings in it.
Sariel (UIUC)
CS573
21
Fall 2013
21 / 24
Some intuition...
.
2.
3.
4.
5.
6.
senders sent string S = s1 s2 . . . sn .
receiver got string T = t1 t2 . . . tn .
p = Pr[ti ̸= si ], for all i.
]
∑[
U : Hamming distance between S and T : U = i si ̸= ti .
By assumption: E[U ] = pn,[ and U is a binomial variable.
]
By Chernoﬀ inequality: U ∈ (1 − δ)np, (1 + δ)np with
high probability, where δ is tiny constant.
7. T is in a ring R centered at S, with inner radius (1 − δ)np
and outer radius (1 + δ)np.
8. This ring has
1
(1+δ)np
∑
i=(1−δ)np
( )
n
i
(
≤2
)
n
(1 + δ)np
≤ α = 2 · 2nH((1+δ)p) .
strings in it.
Sariel (UIUC)
CS573
21
Fall 2013
21 / 24
Some intuition...
.
2.
3.
4.
5.
6.
senders sent string S = s1 s2 . . . sn .
receiver got string T = t1 t2 . . . tn .
p = Pr[ti ̸= si ], for all i.
]
∑[
U : Hamming distance between S and T : U = i si ̸= ti .
By assumption: E[U ] = pn,[ and U is a binomial variable.
]
By Chernoﬀ inequality: U ∈ (1 − δ)np, (1 + δ)np with
high probability, where δ is tiny constant.
7. T is in a ring R centered at S, with inner radius (1 − δ)np
and outer radius (1 + δ)np.
8. This ring has
1
(1+δ)np
∑
i=(1−δ)np
( )
n
i
(
≤2
)
n
(1 + δ)np
≤ α = 2 · 2nH((1+δ)p) .
strings in it.
Sariel (UIUC)
CS573
21
Fall 2013
21 / 24
Many rings for many codewords...
Sariel (UIUC)
CS573
22
Fall 2013
22 / 24
Some more intuition...
. Pick as many disjoint rings as possible: R1 , . . . , Rκ .
2. If every word in the hypercube would be covered...
3. ... use 2n codewords =⇒ κ ≥
2n
2n
κ≥
≥
≈ 2n(1−H((1+δ)p)) .
|R|
2 · 2nH((1+δ)p)
1
.
5.
6.
7.
Consider all possible strings of length k such that 2k ≤ κ.
Map ith string in {0, 1}k to the center Ci of the ith ring Ri .
If send Ci =⇒ receiver gets a string in Ri .
Decoding is easy - ﬁnd the ring Ri containing the received
string, take its center string Ci , and output the original string it
was mapped to.
8. How many bits?
(
(
))
k = ⌊log κ⌋ = n 1 − H (1 + δ)p ≈ n(1 − H (p)),
4
Sariel (UIUC)
CS573
23
Fall 2013
23 / 24
Some more intuition...
. Pick as many disjoint rings as possible: R1 , . . . , Rκ .
2. If every word in the hypercube would be covered...
3. ... use 2n codewords =⇒ κ ≥
2n
2n
κ≥
≥
≈ 2n(1−H((1+δ)p)) .
|R|
2 · 2nH((1+δ)p)
1
.
5.
6.
7.
Consider all possible strings of length k such that 2k ≤ κ.
Map ith string in {0, 1}k to the center Ci of the ith ring Ri .
If send Ci =⇒ receiver gets a string in Ri .
Decoding is easy - ﬁnd the ring Ri containing the received
string, take its center string Ci , and output the original string it
was mapped to.
8. How many bits?
(
(
))
k = ⌊log κ⌋ = n 1 − H (1 + δ)p ≈ n(1 − H (p)),
4
Sariel (UIUC)
CS573
23
Fall 2013
23 / 24
Some more intuition...
. Pick as many disjoint rings as possible: R1 , . . . , Rκ .
2. If every word in the hypercube would be covered...
3. ... use 2n codewords =⇒ κ ≥
2n
2n
κ≥
≥
≈ 2n(1−H((1+δ)p)) .
|R|
2 · 2nH((1+δ)p)
1
.
5.
6.
7.
Consider all possible strings of length k such that 2k ≤ κ.
Map ith string in {0, 1}k to the center Ci of the ith ring Ri .
If send Ci =⇒ receiver gets a string in Ri .
Decoding is easy - ﬁnd the ring Ri containing the received
string, take its center string Ci , and output the original string it
was mapped to.
8. How many bits?
(
(
))
k = ⌊log κ⌋ = n 1 − H (1 + δ)p ≈ n(1 − H (p)),
4
Sariel (UIUC)
CS573
23
Fall 2013
23 / 24
Some more intuition...
. Pick as many disjoint rings as possible: R1 , . . . , Rκ .
2. If every word in the hypercube would be covered...
3. ... use 2n codewords =⇒ κ ≥
2n
2n
κ≥
≥
≈ 2n(1−H((1+δ)p)) .
|R|
2 · 2nH((1+δ)p)
1
.
5.
6.
7.
Consider all possible strings of length k such that 2k ≤ κ.
Map ith string in {0, 1}k to the center Ci of the ith ring Ri .
If send Ci =⇒ receiver gets a string in Ri .
Decoding is easy - ﬁnd the ring Ri containing the received
string, take its center string Ci , and output the original string it
was mapped to.
8. How many bits?
(
(
))
k = ⌊log κ⌋ = n 1 − H (1 + δ)p ≈ n(1 − H (p)),
4
Sariel (UIUC)
CS573
23
Fall 2013
23 / 24
Some more intuition...
. Pick as many disjoint rings as possible: R1 , . . . , Rκ .
2. If every word in the hypercube would be covered...
3. ... use 2n codewords =⇒ κ ≥
2n
2n
κ≥
≥
≈ 2n(1−H((1+δ)p)) .
|R|
2 · 2nH((1+δ)p)
1
.
5.
6.
7.
Consider all possible strings of length k such that 2k ≤ κ.
Map ith string in {0, 1}k to the center Ci of the ith ring Ri .
If send Ci =⇒ receiver gets a string in Ri .
Decoding is easy - ﬁnd the ring Ri containing the received
string, take its center string Ci , and output the original string it
was mapped to.
8. How many bits?
(
(
))
k = ⌊log κ⌋ = n 1 − H (1 + δ)p ≈ n(1 − H (p)),
4
Sariel (UIUC)
CS573
23
Fall 2013
23 / 24
Some more intuition...
. Pick as many disjoint rings as possible: R1 , . . . , Rκ .
2. If every word in the hypercube would be covered...
3. ... use 2n codewords =⇒ κ ≥
2n
2n
κ≥
≥
≈ 2n(1−H((1+δ)p)) .
|R|
2 · 2nH((1+δ)p)
1
.
5.
6.
7.
Consider all possible strings of length k such that 2k ≤ κ.
Map ith string in {0, 1}k to the center Ci of the ith ring Ri .
If send Ci =⇒ receiver gets a string in Ri .
Decoding is easy - ﬁnd the ring Ri containing the received
string, take its center string Ci , and output the original string it
was mapped to.
8. How many bits?
(
(
))
k = ⌊log κ⌋ = n 1 − H (1 + δ)p ≈ n(1 − H (p)),
4
Sariel (UIUC)
CS573
23
Fall 2013
23 / 24
Some more intuition...
. Pick as many disjoint rings as possible: R1 , . . . , Rκ .
2. If every word in the hypercube would be covered...
3. ... use 2n codewords =⇒ κ ≥
2n
2n
κ≥
≥
≈ 2n(1−H((1+δ)p)) .
|R|
2 · 2nH((1+δ)p)
1
.
5.
6.
7.
Consider all possible strings of length k such that 2k ≤ κ.
Map ith string in {0, 1}k to the center Ci of the ith ring Ri .
If send Ci =⇒ receiver gets a string in Ri .
Decoding is easy - ﬁnd the ring Ri containing the received
string, take its center string Ci , and output the original string it
was mapped to.
8. How many bits?
(
(
))
k = ⌊log κ⌋ = n 1 − H (1 + δ)p ≈ n(1 − H (p)),
4
Sariel (UIUC)
CS573
23
Fall 2013
23 / 24
Some more intuition...
. Pick as many disjoint rings as possible: R1 , . . . , Rκ .
2. If every word in the hypercube would be covered...
3. ... use 2n codewords =⇒ κ ≥
2n
2n
κ≥
≥
≈ 2n(1−H((1+δ)p)) .
|R|
2 · 2nH((1+δ)p)
1
.
5.
6.
7.
Consider all possible strings of length k such that 2k ≤ κ.
Map ith string in {0, 1}k to the center Ci of the ith ring Ri .
If send Ci =⇒ receiver gets a string in Ri .
Decoding is easy - ﬁnd the ring Ri containing the received
string, take its center string Ci , and output the original string it
was mapped to.
8. How many bits?
(
(
))
k = ⌊log κ⌋ = n 1 − H (1 + δ)p ≈ n(1 − H (p)),
4
Sariel (UIUC)
CS573
23
Fall 2013
23 / 24
Some more intuition...
. Pick as many disjoint rings as possible: R1 , . . . , Rκ .
2. If every word in the hypercube would be covered...
3. ... use 2n codewords =⇒ κ ≥
2n
2n
κ≥
≥
≈ 2n(1−H((1+δ)p)) .
|R|
2 · 2nH((1+δ)p)
1
.
5.
6.
7.
Consider all possible strings of length k such that 2k ≤ κ.
Map ith string in {0, 1}k to the center Ci of the ith ring Ri .
If send Ci =⇒ receiver gets a string in Ri .
Decoding is easy - ﬁnd the ring Ri containing the received
string, take its center string Ci , and output the original string it
was mapped to.
8. How many bits?
(
(
))
k = ⌊log κ⌋ = n 1 − H (1 + δ)p ≈ n(1 − H (p)),
4
Sariel (UIUC)
CS573
23
Fall 2013
23 / 24
What is wrong with the above?
. Can not ﬁnd such a large set of disjoint rings.
2. Reason is that when you pack rings (or balls) you are going to
have wasted spaces around.
.3 Overcome this: allow rings to overlap somewhat.
4. Makes things considerably more involved.
5. Details in class notes.
1
Sariel (UIUC)
CS573
24
Fall 2013
24 / 24
What is wrong with the above?
. Can not ﬁnd such a large set of disjoint rings.
2. Reason is that when you pack rings (or balls) you are going to
have wasted spaces around.
.3 Overcome this: allow rings to overlap somewhat.
4. Makes things considerably more involved.
5. Details in class notes.
1
Sariel (UIUC)
CS573
24
Fall 2013
24 / 24
What is wrong with the above?
. Can not ﬁnd such a large set of disjoint rings.
2. Reason is that when you pack rings (or balls) you are going to
have wasted spaces around.
.3 Overcome this: allow rings to overlap somewhat.
4. Makes things considerably more involved.
5. Details in class notes.
1
Sariel (UIUC)
CS573
24
Fall 2013
24 / 24
What is wrong with the above?
. Can not ﬁnd such a large set of disjoint rings.
2. Reason is that when you pack rings (or balls) you are going to
have wasted spaces around.
.3 Overcome this: allow rings to overlap somewhat.
4. Makes things considerably more involved.
5. Details in class notes.
1
Sariel (UIUC)
CS573
24
Fall 2013
24 / 24
What is wrong with the above?
. Can not ﬁnd such a large set of disjoint rings.
2. Reason is that when you pack rings (or balls) you are going to
have wasted spaces around.
.3 Overcome this: allow rings to overlap somewhat.
4. Makes things considerably more involved.
5. Details in class notes.
1
Sariel (UIUC)
CS573
24
Fall 2013
24 / 24
Notes
Sariel (UIUC)
CS573
25
Fall 2013
25 / 24
Notes
Sariel (UIUC)
CS573
26
Fall 2013
26 / 24
Notes
Sariel (UIUC)
CS573
27
Fall 2013
27 / 24
Notes
Sariel (UIUC)
CS573
28
Fall 2013
28 / 24
```