CS 573: Algorithms, Fall 2013 Entropy and Shannon’s Theorem Lecture 28 December 10, 2013 Sariel (UIUC) CS573 1 Fall 2013 1 / 24 Part I . Entropy . Sariel (UIUC) CS573 2 Fall 2013 2 / 24 Part II . Extracting randomness . Sariel (UIUC) CS573 3 Fall 2013 3 / 24 Storing all strings of length n and j bits on . Sn,j : set of all strings of length n with j ones in them. 2. T n,j : preﬁx tree storing all Tn,j . 1 T0,0 T1,1 Sariel (UIUC) T1,0 CS573 4 Fall 2013 4 / 24 Preﬁx tree ∀ binary strings of length n with j ones Tn,j 0 Tn−1,j 1 Tn−1,j−1 Tn,n Tn,0 0 1 Tn−1,0 Sariel (UIUC) Tn−1,n−1 CS573 5 Fall 2013 5 / 24 Preﬁx tree ∀ binary strings of length n with j ones Tn,j 0 1 Tn−1,j # of leafs: |Tn,j | = |Tn−1,j | + |Tn−1,j−1 | Tn−1,j−1 Tn,n Tn,0 0 1 Tn−1,0 Sariel (UIUC) Tn−1,n−1 CS573 5 Fall 2013 5 / 24 Preﬁx tree ∀ binary strings of length n with j ones Tn,j 0 1 # of leafs: |Tn,j | = |Tn−1,j | + |Tn−1,j−1 | ( ) n j Tn−1,j = ( ) n−1 j + ( ) n−1 j−1 Tn−1,j−1 Tn,n Tn,0 0 1 Tn−1,0 Sariel (UIUC) Tn−1,n−1 CS573 5 Fall 2013 5 / 24 Preﬁx tree ∀ binary strings of length n with j ones Tn,j 0 # of leafs: |Tn,j | = |Tn−1,j | + |Tn−1,j−1 | 1 Tn−1,j Tn−1,j−1 ( ) n j = ( + =⇒ |Tn,j | = ( ) n−1 j−1 ( ) n j . Tn,n Tn,0 0 1 Tn−1,0 Sariel (UIUC) ) n−1 j Tn−1,n−1 CS573 5 Fall 2013 5 / 24 Encoding a string in Sn,j . 2. 3. 4. 5. 1 Tn,j leafs corresponds to strings of Sn,j . Order all strings of Sn,j order in lexicographical ordering ≡ ordering leafs of Tn,j from left to right. Input: s ∈ Sn,j : compute index of s in sorted set Sn,j . EncodeBinomCoeﬀ(s) denote this polytime procedure. Sariel (UIUC) CS573 6 Fall 2013 6 / 24 Encoding a string in Sn,j . 2. 3. 4. 5. 1 Tn,j leafs corresponds to strings of Sn,j . Order all strings of Sn,j order in lexicographical ordering ≡ ordering leafs of Tn,j from left to right. Input: s ∈ Sn,j : compute index of s in sorted set Sn,j . EncodeBinomCoeﬀ(s) denote this polytime procedure. Sariel (UIUC) CS573 6 Fall 2013 6 / 24 Encoding a string in Sn,j . Tn,j leafs corresponds to strings of Sn,j . 2. Order all strings of S n,j order in lexicographical ordering 3. ≡ ordering leafs of T n,j from left to right. Tn,j 1 0 Tn−1,j 1 Tn−1,j−1 . Input: s ∈ Sn,j : compute index of s in sorted set Sn,j . 5. EncodeBinomCoeﬀ(s) denote this polytime procedure. 4 Sariel (UIUC) CS573 6 Fall 2013 6 / 24 Encoding a string in Sn,j . Tn,j leafs corresponds to strings of Sn,j . 2. Order all strings of S n,j order in lexicographical ordering 3. ≡ ordering leafs of T n,j from left to right. Tn,j 1 0 Tn−1,j 1 Tn−1,j−1 . Input: s ∈ Sn,j : compute index of s in sorted set Sn,j . 5. EncodeBinomCoeﬀ(s) denote this polytime procedure. 4 Sariel (UIUC) CS573 6 Fall 2013 6 / 24 Encoding a string in Sn,j . Tn,j leafs corresponds to strings of Sn,j . 2. Order all strings of S n,j order in lexicographical ordering 3. ≡ ordering leafs of T n,j from left to right. Tn,j 1 0 Tn−1,j 1 Tn−1,j−1 . Input: s ∈ Sn,j : compute index of s in sorted set Sn,j . 5. EncodeBinomCoeﬀ(s) denote this polytime procedure. 4 Sariel (UIUC) CS573 6 Fall 2013 6 / 24 Decoding a string in Sn,j . Tn,j leafs corresponds to strings of Sn,j . 2. Order all strings of S n,j order in lexicographical ordering 3. ≡ ordering leafs of T n,j from left to right. { ( )} n 4. x ∈ 1, . . . , j : compute xth string in Sn,j in polytime. 5. DecodeBinomCoeﬀ (x) denote this procedure. 1 Sariel (UIUC) CS573 7 Fall 2013 7 / 24 Decoding a string in Sn,j . Tn,j leafs corresponds to strings of Sn,j . 2. Order all strings of S n,j order in lexicographical ordering 3. ≡ ordering leafs of T n,j from left to right. { ( )} n 4. x ∈ 1, . . . , j : compute xth string in Sn,j in polytime. 5. DecodeBinomCoeﬀ (x) denote this procedure. 1 Sariel (UIUC) CS573 7 Fall 2013 7 / 24 Decoding a string in Sn,j . Tn,j leafs corresponds to strings of Sn,j . 2. Order all strings of S n,j order in lexicographical ordering 3. ≡ ordering leafs of T n,j from left to right. Tn,j 1 0 Tn−1,j { 1 Tn−1,j−1 ( )} . x ∈ 1, . . . , n : compute xth string in Sn,j in polytime. j 5. DecodeBinomCoeﬀ (x) denote this procedure. 4 Sariel (UIUC) CS573 7 Fall 2013 7 / 24 Decoding a string in Sn,j . Tn,j leafs corresponds to strings of Sn,j . 2. Order all strings of S n,j order in lexicographical ordering 3. ≡ ordering leafs of T n,j from left to right. Tn,j 1 0 Tn−1,j { 1 Tn−1,j−1 ( )} . x ∈ 1, . . . , n : compute xth string in Sn,j in polytime. j 5. DecodeBinomCoeﬀ (x) denote this procedure. 4 Sariel (UIUC) CS573 7 Fall 2013 7 / 24 Decoding a string in Sn,j . Tn,j leafs corresponds to strings of Sn,j . 2. Order all strings of S n,j order in lexicographical ordering 3. ≡ ordering leafs of T n,j from left to right. Tn,j 1 0 Tn−1,j { 1 Tn−1,j−1 ( )} . x ∈ 1, . . . , n : compute xth string in Sn,j in polytime. j 5. DecodeBinomCoeﬀ (x) denote this procedure. 4 Sariel (UIUC) CS573 7 Fall 2013 7 / 24 Encoding/decoding strings of Sn,j . Lemma . Sn,j : Set of binary strings of length n with j ones, sorted lexicographically. 1. EncodeBinomCoeﬀ(α): Input is string α ∈ S n,j , compute index x of α in Sn,j in polynomial time in n. { ( )} 2. DecodeBinomCoeﬀ(x): Input index x ∈ 1, . . . , n . j . Output xth string α in Sn,j , in time O(polylog n + n). Sariel (UIUC) CS573 8 Fall 2013 8 / 24 Extracting randomness . Theorem . Consider a coin that comes up heads with probability p > 1/2. For any constant δ > 0 and for n suﬃciently large: (A) One can extract, from an input of a sequence of n ﬂips, an output sequence of (1 − δ)nH(p) (unbiased) independent random bits. (B) One can not extract more than nH(p) bits from such a sequence. . Sariel (UIUC) CS573 9 Fall 2013 9 / 24 Proof... ( ) . There are n input strings with exactly j heads. j 2. each has probability p j (1 − p)n−j . 3. map string s like that to index number in the set { ( )} Sj = 1, . . . , nj . 4. Given that input string s has j ones (out of n bits) deﬁnes a uniform distribution on Sn,j . 5. x ← EncodeBinomCoeﬀ(s) ( ) n 6. x uniform distributed in {1, . . . , N }, N = . 1 j 7. Seen in previous lecture... 8. ... extract in expectation, ⌊lg N ⌋ − 1 bits from uniform random variable in the range 1, . . . , N . .9 Extract bits using ExtractRandomness(x, N ):. Sariel (UIUC) CS573 10 Fall 2013 10 / 24 Proof... ( ) . There are n input strings with exactly j heads. j 2. each has probability p j (1 − p)n−j . 3. map string s like that to index number in the set { ( )} Sj = 1, . . . , nj . 4. Given that input string s has j ones (out of n bits) deﬁnes a uniform distribution on Sn,j . 5. x ← EncodeBinomCoeﬀ(s) ( ) n 6. x uniform distributed in {1, . . . , N }, N = . 1 j 7. Seen in previous lecture... 8. ... extract in expectation, ⌊lg N ⌋ − 1 bits from uniform random variable in the range 1, . . . , N . .9 Extract bits using ExtractRandomness(x, N ):. Sariel (UIUC) CS573 10 Fall 2013 10 / 24 Proof... ( ) . There are n input strings with exactly j heads. j 2. each has probability p j (1 − p)n−j . 3. map string s like that to index number in the set { ( )} Sj = 1, . . . , nj . 4. Given that input string s has j ones (out of n bits) deﬁnes a uniform distribution on Sn,j . 5. x ← EncodeBinomCoeﬀ(s) ( ) n 6. x uniform distributed in {1, . . . , N }, N = . 1 j 7. Seen in previous lecture... 8. ... extract in expectation, ⌊lg N ⌋ − 1 bits from uniform random variable in the range 1, . . . , N . .9 Extract bits using ExtractRandomness(x, N ):. Sariel (UIUC) CS573 10 Fall 2013 10 / 24 Proof... ( ) . There are n input strings with exactly j heads. j 2. each has probability p j (1 − p)n−j . 3. map string s like that to index number in the set { ( )} Sj = 1, . . . , nj . 4. Given that input string s has j ones (out of n bits) deﬁnes a uniform distribution on Sn,j . 5. x ← EncodeBinomCoeﬀ(s) ( ) n 6. x uniform distributed in {1, . . . , N }, N = . 1 j 7. Seen in previous lecture... 8. ... extract in expectation, ⌊lg N ⌋ − 1 bits from uniform random variable in the range 1, . . . , N . .9 Extract bits using ExtractRandomness(x, N ):. Sariel (UIUC) CS573 10 Fall 2013 10 / 24 Proof... ( ) . There are n input strings with exactly j heads. j 2. each has probability p j (1 − p)n−j . 3. map string s like that to index number in the set { ( )} Sj = 1, . . . , nj . 4. Given that input string s has j ones (out of n bits) deﬁnes a uniform distribution on Sn,j . 5. x ← EncodeBinomCoeﬀ(s) ( ) n 6. x uniform distributed in {1, . . . , N }, N = . 1 j 7. Seen in previous lecture... 8. ... extract in expectation, ⌊lg N ⌋ − 1 bits from uniform random variable in the range 1, . . . , N . .9 Extract bits using ExtractRandomness(x, N ):. Sariel (UIUC) CS573 10 Fall 2013 10 / 24 Proof... ( ) . There are n input strings with exactly j heads. j 2. each has probability p j (1 − p)n−j . 3. map string s like that to index number in the set { ( )} Sj = 1, . . . , nj . 4. Given that input string s has j ones (out of n bits) deﬁnes a uniform distribution on Sn,j . 5. x ← EncodeBinomCoeﬀ(s) ( ) n 6. x uniform distributed in {1, . . . , N }, N = . 1 j 7. Seen in previous lecture... 8. ... extract in expectation, ⌊lg N ⌋ − 1 bits from uniform random variable in the range 1, . . . , N . .9 Extract bits using ExtractRandomness(x, N ):. Sariel (UIUC) CS573 10 Fall 2013 10 / 24 Proof... ( ) . There are n input strings with exactly j heads. j 2. each has probability p j (1 − p)n−j . 3. map string s like that to index number in the set { ( )} Sj = 1, . . . , nj . 4. Given that input string s has j ones (out of n bits) deﬁnes a uniform distribution on Sn,j . 5. x ← EncodeBinomCoeﬀ(s) ( ) n 6. x uniform distributed in {1, . . . , N }, N = . 1 j 7. Seen in previous lecture... 8. ... extract in expectation, ⌊lg N ⌋ − 1 bits from uniform random variable in the range 1, . . . , N . .9 Extract bits using ExtractRandomness(x, N ):. Sariel (UIUC) CS573 10 Fall 2013 10 / 24 Proof... ( ) . There are n input strings with exactly j heads. j 2. each has probability p j (1 − p)n−j . 3. map string s like that to index number in the set { ( )} Sj = 1, . . . , nj . 4. Given that input string s has j ones (out of n bits) deﬁnes a uniform distribution on Sn,j . 5. x ← EncodeBinomCoeﬀ(s) ( ) n 6. x uniform distributed in {1, . . . , N }, N = . 1 j 7. Seen in previous lecture... 8. ... extract in expectation, ⌊lg N ⌋ − 1 bits from uniform random variable in the range 1, . . . , N . .9 Extract bits using ExtractRandomness(x, N ):. Sariel (UIUC) CS573 10 Fall 2013 10 / 24 Proof... ( ) . There are n input strings with exactly j heads. j 2. each has probability p j (1 − p)n−j . 3. map string s like that to index number in the set { ( )} Sj = 1, . . . , nj . 4. Given that input string s has j ones (out of n bits) deﬁnes a uniform distribution on Sn,j . 5. x ← EncodeBinomCoeﬀ(s) ( ) n 6. x uniform distributed in {1, . . . , N }, N = . 1 j 7. Seen in previous lecture... 8. ... extract in expectation, ⌊lg N ⌋ − 1 bits from uniform random variable in the range 1, . . . , N . .9 Extract bits using ExtractRandomness(x, N ):. Sariel (UIUC) CS573 10 Fall 2013 10 / 24 Exciting proof continued... . Z : random variable: number of heads in input string s. 2. B: number of random bits extracted. 1 [ ] EB = [ n ∑ ] Pr[Z = k] E B Z = k , k=0 ⌊ ( )⌋ [ ] n 3. Know: B Z = k ≥ lg − 1. E k 4. ε < p − 1/2: suﬃciently small constant. 5. n(p − ε) ≤ k ≤ n(p + ε): ( ) n k ( ≥ ) n ⌊n(p + ε)⌋ ≥ 2nH(p+ε) n+1 . ... since 2nH(p) is a good approximation to previous lecture. 6 Sariel (UIUC) CS573 11 ( n np ) , as proved in Fall 2013 11 / 24 Exciting proof continued... . Z : random variable: number of heads in input string s. 2. B: number of random bits extracted. 1 [ ] EB = [ n ∑ ] Pr[Z = k] E B Z = k , k=0 ⌊ ( )⌋ [ ] n 3. Know: B Z = k ≥ lg − 1. E k 4. ε < p − 1/2: suﬃciently small constant. 5. n(p − ε) ≤ k ≤ n(p + ε): ( ) n k ( ≥ ) n ⌊n(p + ε)⌋ ≥ 2nH(p+ε) n+1 . ... since 2nH(p) is a good approximation to previous lecture. 6 Sariel (UIUC) CS573 11 ( n np ) , as proved in Fall 2013 11 / 24 Exciting proof continued... . Z : random variable: number of heads in input string s. 2. B: number of random bits extracted. 1 [ ] EB = [ n ∑ ] Pr[Z = k] E B Z = k , k=0 ⌊ ( )⌋ [ ] n 3. Know: B Z = k ≥ lg − 1. E k 4. ε < p − 1/2: suﬃciently small constant. 5. n(p − ε) ≤ k ≤ n(p + ε): ( ) n k ( ≥ ) n ⌊n(p + ε)⌋ ≥ 2nH(p+ε) n+1 . ... since 2nH(p) is a good approximation to previous lecture. 6 Sariel (UIUC) CS573 11 ( n np ) , as proved in Fall 2013 11 / 24 Exciting proof continued... . Z : random variable: number of heads in input string s. 2. B: number of random bits extracted. 1 [ ] EB = [ n ∑ ] Pr[Z = k] E B Z = k , k=0 ⌊ ( )⌋ [ ] n 3. Know: B Z = k ≥ lg − 1. E k 4. ε < p − 1/2: suﬃciently small constant. 5. n(p − ε) ≤ k ≤ n(p + ε): ( ) n k ( ≥ ) n ⌊n(p + ε)⌋ ≥ 2nH(p+ε) n+1 . ... since 2nH(p) is a good approximation to previous lecture. 6 Sariel (UIUC) CS573 11 ( n np ) , as proved in Fall 2013 11 / 24 Exciting proof continued... . Z : random variable: number of heads in input string s. 2. B: number of random bits extracted. 1 [ ] EB = [ n ∑ ] Pr[Z = k] E B Z = k , k=0 ⌊ ( )⌋ [ ] n 3. Know: B Z = k ≥ lg − 1. E k 4. ε < p − 1/2: suﬃciently small constant. 5. n(p − ε) ≤ k ≤ n(p + ε): ( ) n k ( ≥ ) n ⌊n(p + ε)⌋ ≥ 2nH(p+ε) n+1 . ... since 2nH(p) is a good approximation to previous lecture. 6 Sariel (UIUC) CS573 11 ( n np ) , as proved in Fall 2013 11 / 24 Exciting proof continued... . Z : random variable: number of heads in input string s. 2. B: number of random bits extracted. 1 [ ] EB = [ n ∑ ] Pr[Z = k] E B Z = k , k=0 ⌊ ( )⌋ [ ] n 3. Know: B Z = k ≥ lg − 1. E k 4. ε < p − 1/2: suﬃciently small constant. 5. n(p − ε) ≤ k ≤ n(p + ε): ( ) n k ( ≥ ) n ⌊n(p + ε)⌋ ≥ 2nH(p+ε) n+1 . ... since 2nH(p) is a good approximation to previous lecture. 6 Sariel (UIUC) CS573 11 ( n np ) , as proved in Fall 2013 11 / 24 Exciting proof continued... . Z : random variable: number of heads in input string s. 2. B: number of random bits extracted. 1 [ ] EB = [ n ∑ ] Pr[Z = k] E B Z = k , k=0 ⌊ ( )⌋ [ ] n 3. Know: B Z = k ≥ lg − 1. E k 4. ε < p − 1/2: suﬃciently small constant. 5. n(p − ε) ≤ k ≤ n(p + ε): ( ) n k ( ≥ ) n ⌊n(p + ε)⌋ ≥ 2nH(p+ε) n+1 . ... since 2nH(p) is a good approximation to previous lecture. 6 Sariel (UIUC) CS573 11 ( n np ) , as proved in Fall 2013 11 / 24 Super exciting proof continued... [ ] ∑n [ ] ∑⌈n(p+ε)⌉ EB = EB ≥ ≥ ] [ [ k=⌊n(p−ε)⌋ Pr Z = k E B Z = k ⌈n(p+ε)⌉ ∑ ≥ ] [ k=0 Pr[Z = k] E B Z = k . [ Pr Z = k k=⌊n(p−ε)⌋ ⌈n(p+ε)⌉ ∑ [ Pr Z = k k=⌊n(p−ε)⌋ ( ] ] (⌊ ( )⌋ n lg ( lg k ] ) −1 ) 2nH(p+ε) −2 n+1 ) = nH(p + ε) − lg(n + 1) − 2 Pr[|Z − np| ≤ εn] ( ( )) ( ) nε2 ≥ nH(p + ε) − lg(n + 1) − 2 1 − 2 exp − , 4p since µ = E[Z ] = np and ( ) [ ] Pr |Z − np| ≥ pε pn ≤ 2 exp − np 4 ( )2 ε p ( 2 ) , by = 2 exp − nε 4p the Chernoﬀ inequality. Sariel (UIUC) CS573 12 Fall 2013 12 / 24 Super exciting proof continued... [ ] ∑n [ ] ∑⌈n(p+ε)⌉ EB = EB ≥ ≥ ] [ [ k=⌊n(p−ε)⌋ Pr Z = k E B Z = k ⌈n(p+ε)⌉ ∑ ≥ ] [ k=0 Pr[Z = k] E B Z = k . [ Pr Z = k k=⌊n(p−ε)⌋ ⌈n(p+ε)⌉ ∑ [ Pr Z = k k=⌊n(p−ε)⌋ ( ] ] (⌊ ( )⌋ n lg ( lg k ] ) −1 ) 2nH(p+ε) −2 n+1 ) = nH(p + ε) − lg(n + 1) − 2 Pr[|Z − np| ≤ εn] ( ( )) ( ) nε2 ≥ nH(p + ε) − lg(n + 1) − 2 1 − 2 exp − , 4p since µ = E[Z ] = np and ( ) [ ] Pr |Z − np| ≥ pε pn ≤ 2 exp − np 4 ( )2 ε p ( 2 ) , by = 2 exp − nε 4p the Chernoﬀ inequality. Sariel (UIUC) CS573 12 Fall 2013 12 / 24 Super exciting proof continued... [ ] ∑n [ ] ∑⌈n(p+ε)⌉ EB = EB ≥ ≥ ] [ [ k=⌊n(p−ε)⌋ Pr Z = k E B Z = k ⌈n(p+ε)⌉ ∑ ≥ ] [ k=0 Pr[Z = k] E B Z = k . [ Pr Z = k k=⌊n(p−ε)⌋ ⌈n(p+ε)⌉ ∑ [ Pr Z = k k=⌊n(p−ε)⌋ ( ] ] (⌊ ( )⌋ n lg ( lg k ] ) −1 ) 2nH(p+ε) −2 n+1 ) = nH(p + ε) − lg(n + 1) − 2 Pr[|Z − np| ≤ εn] ( ( )) ( ) nε2 ≥ nH(p + ε) − lg(n + 1) − 2 1 − 2 exp − , 4p since µ = E[Z ] = np and ( ) [ ] Pr |Z − np| ≥ pε pn ≤ 2 exp − np 4 ( )2 ε p ( 2 ) , by = 2 exp − nε 4p the Chernoﬀ inequality. Sariel (UIUC) CS573 12 Fall 2013 12 / 24 Super exciting proof continued... [ ] ∑n [ ] ∑⌈n(p+ε)⌉ EB = EB ≥ ≥ ] [ [ k=⌊n(p−ε)⌋ Pr Z = k E B Z = k ⌈n(p+ε)⌉ ∑ ≥ ] [ k=0 Pr[Z = k] E B Z = k . [ Pr Z = k k=⌊n(p−ε)⌋ ⌈n(p+ε)⌉ ∑ [ Pr Z = k k=⌊n(p−ε)⌋ ( ] ] (⌊ ( )⌋ n lg ( lg k ] ) −1 ) 2nH(p+ε) −2 n+1 ) = nH(p + ε) − lg(n + 1) − 2 Pr[|Z − np| ≤ εn] ( ( )) ( ) nε2 ≥ nH(p + ε) − lg(n + 1) − 2 1 − 2 exp − , 4p since µ = E[Z ] = np and ( ) [ ] Pr |Z − np| ≥ pε pn ≤ 2 exp − np 4 ( )2 ε p ( 2 ) , by = 2 exp − nε 4p the Chernoﬀ inequality. Sariel (UIUC) CS573 12 Fall 2013 12 / 24 Super exciting proof continued... [ ] ∑n [ ] ∑⌈n(p+ε)⌉ EB = EB ≥ ≥ ] [ [ k=⌊n(p−ε)⌋ Pr Z = k E B Z = k ⌈n(p+ε)⌉ ∑ ≥ ] [ k=0 Pr[Z = k] E B Z = k . [ Pr Z = k k=⌊n(p−ε)⌋ ⌈n(p+ε)⌉ ∑ [ Pr Z = k k=⌊n(p−ε)⌋ ( ] ] (⌊ ( )⌋ n lg ( lg k ] ) −1 ) 2nH(p+ε) −2 n+1 ) = nH(p + ε) − lg(n + 1) − 2 Pr[|Z − np| ≤ εn] ( ( )) ( ) nε2 ≥ nH(p + ε) − lg(n + 1) − 2 1 − 2 exp − , 4p since µ = E[Z ] = np and ( ) [ ] Pr |Z − np| ≥ pε pn ≤ 2 exp − np 4 ( )2 ε p ( 2 ) , by = 2 exp − nε 4p the Chernoﬀ inequality. Sariel (UIUC) CS573 12 Fall 2013 12 / 24 Super exciting proof continued... [ ] ∑n [ ] ∑⌈n(p+ε)⌉ EB = EB ≥ ≥ ] [ [ k=⌊n(p−ε)⌋ Pr Z = k E B Z = k ⌈n(p+ε)⌉ ∑ ≥ ] [ k=0 Pr[Z = k] E B Z = k . [ Pr Z = k k=⌊n(p−ε)⌋ ⌈n(p+ε)⌉ ∑ [ Pr Z = k k=⌊n(p−ε)⌋ ( ] ] (⌊ ( )⌋ n lg ( lg k ] ) −1 ) 2nH(p+ε) −2 n+1 ) = nH(p + ε) − lg(n + 1) − 2 Pr[|Z − np| ≤ εn] ( ( )) ( ) nε2 ≥ nH(p + ε) − lg(n + 1) − 2 1 − 2 exp − , 4p since µ = E[Z ] = np and ( ) [ ] Pr |Z − np| ≥ pε pn ≤ 2 exp − np 4 ( )2 ε p ( 2 ) , by = 2 exp − nε 4p the Chernoﬀ inequality. Sariel (UIUC) CS573 12 Fall 2013 12 / 24 Super exciting proof continued... [ ] ∑n [ ] ∑⌈n(p+ε)⌉ EB = EB ≥ ≥ ] [ [ k=⌊n(p−ε)⌋ Pr Z = k E B Z = k ⌈n(p+ε)⌉ ∑ ≥ ] [ k=0 Pr[Z = k] E B Z = k . [ Pr Z = k k=⌊n(p−ε)⌋ ⌈n(p+ε)⌉ ∑ [ Pr Z = k k=⌊n(p−ε)⌋ ( ] ] (⌊ ( )⌋ n lg ( lg k ] ) −1 ) 2nH(p+ε) −2 n+1 ) = nH(p + ε) − lg(n + 1) − 2 Pr[|Z − np| ≤ εn] ( ( )) ( ) nε2 ≥ nH(p + ε) − lg(n + 1) − 2 1 − 2 exp − , 4p since µ = E[Z ] = np and ( ) [ ] Pr |Z − np| ≥ pε pn ≤ 2 exp − np 4 ( )2 ε p ( 2 ) , by = 2 exp − nε 4p the Chernoﬀ inequality. Sariel (UIUC) CS573 12 Fall 2013 12 / 24 Hyper super exciting proof continued... . Fix ε > 0, such that H(p + ε) > (1 − δ/4)H(p), p is ﬁxed. 2. =⇒ nH(p) = Ω(n), δ 3. For n suﬃciently large: − lg(n + 1) ≥ − nH(p). 10 ( ) 2 nε δ 4. ... also 2 exp − ≤ . 1 4p 10 . For n large enough; 5 ( E[B] ≥ 1 − δ δ ) ( δ ) − nH(p) 1 − 4 10 10 ≥ (1 − δ) nH(p) , Sariel (UIUC) CS573 13 Fall 2013 13 / 24 Hyper super exciting proof continued... . Fix ε > 0, such that H(p + ε) > (1 − δ/4)H(p), p is ﬁxed. 2. =⇒ nH(p) = Ω(n), δ 3. For n suﬃciently large: − lg(n + 1) ≥ − nH(p). 10 ( ) 2 nε δ 4. ... also 2 exp − ≤ . 1 4p 10 . For n large enough; 5 ( E[B] ≥ 1 − δ δ ) ( δ ) − nH(p) 1 − 4 10 10 ≥ (1 − δ) nH(p) , Sariel (UIUC) CS573 13 Fall 2013 13 / 24 Hyper super exciting proof continued... . Fix ε > 0, such that H(p + ε) > (1 − δ/4)H(p), p is ﬁxed. 2. =⇒ nH(p) = Ω(n), δ 3. For n suﬃciently large: − lg(n + 1) ≥ − nH(p). 10 ( ) 2 nε δ 4. ... also 2 exp − ≤ . 1 4p 10 . For n large enough; 5 ( E[B] ≥ 1 − δ δ ) ( δ ) − nH(p) 1 − 4 10 10 ≥ (1 − δ) nH(p) , Sariel (UIUC) CS573 13 Fall 2013 13 / 24 Hyper super exciting proof continued... . Fix ε > 0, such that H(p + ε) > (1 − δ/4)H(p), p is ﬁxed. 2. =⇒ nH(p) = Ω(n), δ 3. For n suﬃciently large: − lg(n + 1) ≥ − nH(p). 10 ( ) 2 nε δ 4. ... also 2 exp − ≤ . 1 4p 10 . For n large enough; 5 ( E[B] ≥ 1 − δ δ ) ( δ ) − nH(p) 1 − 4 10 10 ≥ (1 − δ) nH(p) , Sariel (UIUC) CS573 13 Fall 2013 13 / 24 Hyper super exciting proof continued... . Fix ε > 0, such that H(p + ε) > (1 − δ/4)H(p), p is ﬁxed. 2. =⇒ nH(p) = Ω(n), δ 3. For n suﬃciently large: − lg(n + 1) ≥ − nH(p). 10 ( ) 2 nε δ 4. ... also 2 exp − ≤ . 1 4p 10 . For n large enough; 5 ( E[B] ≥ 1 − δ δ ) ( δ ) − nH(p) 1 − 4 10 10 ≥ (1 − δ) nH(p) , Sariel (UIUC) CS573 13 Fall 2013 13 / 24 Hyper super duper exciting proof continued... . Need to prove upper bound. 2. If input sequence x has probability Pr[X = x], then y = Ext(x) has probability to be generated ≥ Pr[X = x]. .3 All sequences of length |y| have equal probability to be generated (by deﬁnition). .4 2|Ext(x)| Pr[X = x] ≤ 2|Ext(x)| Pr[y = Ext(x)] ≤ 1. 5. =⇒ |Ext(x)| ≤ lg(1/ Pr[X = x]) [ ] [ ] .6 E B = ∑x Pr X = x |Ext(x)| 1 ≤ ∑ [ ] x Pr X = x lg Sariel (UIUC) 1 Pr[X=x] CS573 = H(X) . 14 Fall 2013 14 / 24 Hyper super duper exciting proof continued... . Need to prove upper bound. 2. If input sequence x has probability Pr[X = x], then y = Ext(x) has probability to be generated ≥ Pr[X = x]. .3 All sequences of length |y| have equal probability to be generated (by deﬁnition). .4 2|Ext(x)| Pr[X = x] ≤ 2|Ext(x)| Pr[y = Ext(x)] ≤ 1. 5. =⇒ |Ext(x)| ≤ lg(1/ Pr[X = x]) [ ] [ ] .6 E B = ∑x Pr X = x |Ext(x)| 1 ≤ ∑ [ ] x Pr X = x lg Sariel (UIUC) 1 Pr[X=x] CS573 = H(X) . 14 Fall 2013 14 / 24 Hyper super duper exciting proof continued... . Need to prove upper bound. 2. If input sequence x has probability Pr[X = x], then y = Ext(x) has probability to be generated ≥ Pr[X = x]. .3 All sequences of length |y| have equal probability to be generated (by deﬁnition). .4 2|Ext(x)| Pr[X = x] ≤ 2|Ext(x)| Pr[y = Ext(x)] ≤ 1. 5. =⇒ |Ext(x)| ≤ lg(1/ Pr[X = x]) [ ] [ ] .6 E B = ∑x Pr X = x |Ext(x)| 1 ≤ ∑ [ ] x Pr X = x lg Sariel (UIUC) 1 Pr[X=x] CS573 = H(X) . 14 Fall 2013 14 / 24 Hyper super duper exciting proof continued... . Need to prove upper bound. 2. If input sequence x has probability Pr[X = x], then y = Ext(x) has probability to be generated ≥ Pr[X = x]. .3 All sequences of length |y| have equal probability to be generated (by deﬁnition). .4 2|Ext(x)| Pr[X = x] ≤ 2|Ext(x)| Pr[y = Ext(x)] ≤ 1. 5. =⇒ |Ext(x)| ≤ lg(1/ Pr[X = x]) [ ] [ ] .6 E B = ∑x Pr X = x |Ext(x)| 1 ≤ ∑ [ ] x Pr X = x lg Sariel (UIUC) 1 Pr[X=x] CS573 = H(X) . 14 Fall 2013 14 / 24 Hyper super duper exciting proof continued... . Need to prove upper bound. 2. If input sequence x has probability Pr[X = x], then y = Ext(x) has probability to be generated ≥ Pr[X = x]. .3 All sequences of length |y| have equal probability to be generated (by deﬁnition). .4 2|Ext(x)| Pr[X = x] ≤ 2|Ext(x)| Pr[y = Ext(x)] ≤ 1. 5. =⇒ |Ext(x)| ≤ lg(1/ Pr[X = x]) [ ] [ ] .6 E B = ∑x Pr X = x |Ext(x)| 1 ≤ ∑ [ ] x Pr X = x lg Sariel (UIUC) 1 Pr[X=x] CS573 = H(X) . 14 Fall 2013 14 / 24 Hyper super duper exciting proof continued... . Need to prove upper bound. 2. If input sequence x has probability Pr[X = x], then y = Ext(x) has probability to be generated ≥ Pr[X = x]. .3 All sequences of length |y| have equal probability to be generated (by deﬁnition). .4 2|Ext(x)| Pr[X = x] ≤ 2|Ext(x)| Pr[y = Ext(x)] ≤ 1. 5. =⇒ |Ext(x)| ≤ lg(1/ Pr[X = x]) [ ] [ ] .6 E B = ∑x Pr X = x |Ext(x)| 1 ≤ ∑ [ ] x Pr X = x lg Sariel (UIUC) 1 Pr[X=x] CS573 = H(X) . 14 Fall 2013 14 / 24 Hyper super duper exciting proof continued... . Need to prove upper bound. 2. If input sequence x has probability Pr[X = x], then y = Ext(x) has probability to be generated ≥ Pr[X = x]. .3 All sequences of length |y| have equal probability to be generated (by deﬁnition). .4 2|Ext(x)| Pr[X = x] ≤ 2|Ext(x)| Pr[y = Ext(x)] ≤ 1. 5. =⇒ |Ext(x)| ≤ lg(1/ Pr[X = x]) [ ] [ ] .6 E B = ∑x Pr X = x |Ext(x)| 1 ≤ ∑ [ ] x Pr X = x lg Sariel (UIUC) 1 Pr[X=x] CS573 = H(X) . 14 Fall 2013 14 / 24 Hyper super duper exciting proof continued... . Need to prove upper bound. 2. If input sequence x has probability Pr[X = x], then y = Ext(x) has probability to be generated ≥ Pr[X = x]. .3 All sequences of length |y| have equal probability to be generated (by deﬁnition). .4 2|Ext(x)| Pr[X = x] ≤ 2|Ext(x)| Pr[y = Ext(x)] ≤ 1. 5. =⇒ |Ext(x)| ≤ lg(1/ Pr[X = x]) [ ] [ ] .6 E B = ∑x Pr X = x |Ext(x)| 1 ≤ ∑ [ ] x Pr X = x lg Sariel (UIUC) 1 Pr[X=x] CS573 = H(X) . 14 Fall 2013 14 / 24 Hyper super duper exciting proof continued... . Need to prove upper bound. 2. If input sequence x has probability Pr[X = x], then y = Ext(x) has probability to be generated ≥ Pr[X = x]. .3 All sequences of length |y| have equal probability to be generated (by deﬁnition). .4 2|Ext(x)| Pr[X = x] ≤ 2|Ext(x)| Pr[y = Ext(x)] ≤ 1. 5. =⇒ |Ext(x)| ≤ lg(1/ Pr[X = x]) [ ] [ ] .6 E B = ∑x Pr X = x |Ext(x)| 1 ≤ ∑ [ ] x Pr X = x lg Sariel (UIUC) 1 Pr[X=x] CS573 = H(X) . 14 Fall 2013 14 / 24 Part III . Coding: Shannon’s Theorem . Sariel (UIUC) CS573 15 Fall 2013 15 / 24 Shannon’s Theorem . Deﬁnition . The input to a binary symmetric channel with parameter p is a sequence of bits x1 , x2 , . . . , and the output is a sequence of bits y1 , y2 , . . . , such that Pr[xi = yi ] = 1 − p independently for each i. . Sariel (UIUC) CS573 16 Fall 2013 16 / 24 Encoding/decoding with noise . Deﬁnition . A (k, n) encoding function Enc : {0, 1}k → {0, 1}n takes as input a sequence of k bits and outputs a sequence of n bits. A (k, n) decoding function Dec : {0, 1}n → {0, 1}k takes as input .a sequence of n bits and outputs a sequence of k bits. Sariel (UIUC) CS573 17 Fall 2013 17 / 24 Claude Elwood Shannon Claude Elwood Shannon (April 30, 1916 - February 24, 2001), an American electrical engineer and mathematician, has been called “the father of information theory”. His master thesis was how to building boolean circuits for any boolean function. Sariel (UIUC) CS573 18 Fall 2013 18 / 24 Shannon’s theorem (1948) . Theorem (Shannon’s theorem) . For a binary symmetric channel with parameter p < 1/2 and for any constants δ, γ > 0, where n is suﬃciently large, the following holds: (i) For an k ≤ n(1 − H(p) − δ) there exists (k, n) encoding and decoding functions such that the probability the receiver fails to obtain the correct message is at most γ for every possible k-bit input messages. (ii) There are no (k, n) encoding and decoding functions with k ≥ n(1 − H(p) + δ) such that the probability of decoding correctly is at least γ for a k-bit input message chosen uniformly at random. . Sariel (UIUC) CS573 19 Fall 2013 19 / 24 When the sender sends a string... S = s1 s2 . . . sn S Sariel (UIUC) CS573 20 Fall 2013 20 / 24 When the sender sends a string... S = s1 s2 . . . sn S Sariel (UIUC) CS573 np 20 Fall 2013 20 / 24 When the sender sends a string... S = s1 s2 . . . sn S Sariel (UIUC) CS573 np 20 Fall 2013 20 / 24 When the sender sends a string... S = s1 s2 . . . sn S np (1 − δ)np (1 + δ)np Sariel (UIUC) CS573 20 Fall 2013 20 / 24 When the sender sends a string... S = s1 s2 . . . sn S np (1 − δ)np (1 + δ)np One ring to rule them all! Sariel (UIUC) CS573 20 Fall 2013 20 / 24 Some intuition... . 2. 3. 4. 5. 6. senders sent string S = s1 s2 . . . sn . receiver got string T = t1 t2 . . . tn . p = Pr[ti ̸= si ], for all i. ] ∑[ U : Hamming distance between S and T : U = i si ̸= ti . By assumption: E[U ] = pn,[ and U is a binomial variable. ] By Chernoﬀ inequality: U ∈ (1 − δ)np, (1 + δ)np with high probability, where δ is tiny constant. 7. T is in a ring R centered at S, with inner radius (1 − δ)np and outer radius (1 + δ)np. 8. This ring has 1 (1+δ)np ∑ i=(1−δ)np ( ) n i ( ≤2 ) n (1 + δ)np ≤ α = 2 · 2nH((1+δ)p) . strings in it. Sariel (UIUC) CS573 21 Fall 2013 21 / 24 Some intuition... . 2. 3. 4. 5. 6. senders sent string S = s1 s2 . . . sn . receiver got string T = t1 t2 . . . tn . p = Pr[ti ̸= si ], for all i. ] ∑[ U : Hamming distance between S and T : U = i si ̸= ti . By assumption: E[U ] = pn,[ and U is a binomial variable. ] By Chernoﬀ inequality: U ∈ (1 − δ)np, (1 + δ)np with high probability, where δ is tiny constant. 7. T is in a ring R centered at S, with inner radius (1 − δ)np and outer radius (1 + δ)np. 8. This ring has 1 (1+δ)np ∑ i=(1−δ)np ( ) n i ( ≤2 ) n (1 + δ)np ≤ α = 2 · 2nH((1+δ)p) . strings in it. Sariel (UIUC) CS573 21 Fall 2013 21 / 24 Some intuition... . 2. 3. 4. 5. 6. senders sent string S = s1 s2 . . . sn . receiver got string T = t1 t2 . . . tn . p = Pr[ti ̸= si ], for all i. ] ∑[ U : Hamming distance between S and T : U = i si ̸= ti . By assumption: E[U ] = pn,[ and U is a binomial variable. ] By Chernoﬀ inequality: U ∈ (1 − δ)np, (1 + δ)np with high probability, where δ is tiny constant. 7. T is in a ring R centered at S, with inner radius (1 − δ)np and outer radius (1 + δ)np. 8. This ring has 1 (1+δ)np ∑ i=(1−δ)np ( ) n i ( ≤2 ) n (1 + δ)np ≤ α = 2 · 2nH((1+δ)p) . strings in it. Sariel (UIUC) CS573 21 Fall 2013 21 / 24 Some intuition... . 2. 3. 4. 5. 6. senders sent string S = s1 s2 . . . sn . receiver got string T = t1 t2 . . . tn . p = Pr[ti ̸= si ], for all i. ] ∑[ U : Hamming distance between S and T : U = i si ̸= ti . By assumption: E[U ] = pn,[ and U is a binomial variable. ] By Chernoﬀ inequality: U ∈ (1 − δ)np, (1 + δ)np with high probability, where δ is tiny constant. 7. T is in a ring R centered at S, with inner radius (1 − δ)np and outer radius (1 + δ)np. 8. This ring has 1 (1+δ)np ∑ i=(1−δ)np ( ) n i ( ≤2 ) n (1 + δ)np ≤ α = 2 · 2nH((1+δ)p) . strings in it. Sariel (UIUC) CS573 21 Fall 2013 21 / 24 Some intuition... . 2. 3. 4. 5. 6. senders sent string S = s1 s2 . . . sn . receiver got string T = t1 t2 . . . tn . p = Pr[ti ̸= si ], for all i. ] ∑[ U : Hamming distance between S and T : U = i si ̸= ti . By assumption: E[U ] = pn,[ and U is a binomial variable. ] By Chernoﬀ inequality: U ∈ (1 − δ)np, (1 + δ)np with high probability, where δ is tiny constant. 7. T is in a ring R centered at S, with inner radius (1 − δ)np and outer radius (1 + δ)np. 8. This ring has 1 (1+δ)np ∑ i=(1−δ)np ( ) n i ( ≤2 ) n (1 + δ)np ≤ α = 2 · 2nH((1+δ)p) . strings in it. Sariel (UIUC) CS573 21 Fall 2013 21 / 24 Some intuition... . 2. 3. 4. 5. 6. senders sent string S = s1 s2 . . . sn . receiver got string T = t1 t2 . . . tn . p = Pr[ti ̸= si ], for all i. ] ∑[ U : Hamming distance between S and T : U = i si ̸= ti . By assumption: E[U ] = pn,[ and U is a binomial variable. ] By Chernoﬀ inequality: U ∈ (1 − δ)np, (1 + δ)np with high probability, where δ is tiny constant. 7. T is in a ring R centered at S, with inner radius (1 − δ)np and outer radius (1 + δ)np. 8. This ring has 1 (1+δ)np ∑ i=(1−δ)np ( ) n i ( ≤2 ) n (1 + δ)np ≤ α = 2 · 2nH((1+δ)p) . strings in it. Sariel (UIUC) CS573 21 Fall 2013 21 / 24 Some intuition... . 2. 3. 4. 5. 6. senders sent string S = s1 s2 . . . sn . receiver got string T = t1 t2 . . . tn . p = Pr[ti ̸= si ], for all i. ] ∑[ U : Hamming distance between S and T : U = i si ̸= ti . By assumption: E[U ] = pn,[ and U is a binomial variable. ] By Chernoﬀ inequality: U ∈ (1 − δ)np, (1 + δ)np with high probability, where δ is tiny constant. 7. T is in a ring R centered at S, with inner radius (1 − δ)np and outer radius (1 + δ)np. 8. This ring has 1 (1+δ)np ∑ i=(1−δ)np ( ) n i ( ≤2 ) n (1 + δ)np ≤ α = 2 · 2nH((1+δ)p) . strings in it. Sariel (UIUC) CS573 21 Fall 2013 21 / 24 Some intuition... . 2. 3. 4. 5. 6. senders sent string S = s1 s2 . . . sn . receiver got string T = t1 t2 . . . tn . p = Pr[ti ̸= si ], for all i. ] ∑[ U : Hamming distance between S and T : U = i si ̸= ti . By assumption: E[U ] = pn,[ and U is a binomial variable. ] By Chernoﬀ inequality: U ∈ (1 − δ)np, (1 + δ)np with high probability, where δ is tiny constant. 7. T is in a ring R centered at S, with inner radius (1 − δ)np and outer radius (1 + δ)np. 8. This ring has 1 (1+δ)np ∑ i=(1−δ)np ( ) n i ( ≤2 ) n (1 + δ)np ≤ α = 2 · 2nH((1+δ)p) . strings in it. Sariel (UIUC) CS573 21 Fall 2013 21 / 24 Many rings for many codewords... Sariel (UIUC) CS573 22 Fall 2013 22 / 24 Some more intuition... . Pick as many disjoint rings as possible: R1 , . . . , Rκ . 2. If every word in the hypercube would be covered... 3. ... use 2n codewords =⇒ κ ≥ 2n 2n κ≥ ≥ ≈ 2n(1−H((1+δ)p)) . |R| 2 · 2nH((1+δ)p) 1 . 5. 6. 7. Consider all possible strings of length k such that 2k ≤ κ. Map ith string in {0, 1}k to the center Ci of the ith ring Ri . If send Ci =⇒ receiver gets a string in Ri . Decoding is easy - ﬁnd the ring Ri containing the received string, take its center string Ci , and output the original string it was mapped to. 8. How many bits? ( ( )) k = ⌊log κ⌋ = n 1 − H (1 + δ)p ≈ n(1 − H (p)), 4 Sariel (UIUC) CS573 23 Fall 2013 23 / 24 Some more intuition... . Pick as many disjoint rings as possible: R1 , . . . , Rκ . 2. If every word in the hypercube would be covered... 3. ... use 2n codewords =⇒ κ ≥ 2n 2n κ≥ ≥ ≈ 2n(1−H((1+δ)p)) . |R| 2 · 2nH((1+δ)p) 1 . 5. 6. 7. Consider all possible strings of length k such that 2k ≤ κ. Map ith string in {0, 1}k to the center Ci of the ith ring Ri . If send Ci =⇒ receiver gets a string in Ri . Decoding is easy - ﬁnd the ring Ri containing the received string, take its center string Ci , and output the original string it was mapped to. 8. How many bits? ( ( )) k = ⌊log κ⌋ = n 1 − H (1 + δ)p ≈ n(1 − H (p)), 4 Sariel (UIUC) CS573 23 Fall 2013 23 / 24 Some more intuition... . Pick as many disjoint rings as possible: R1 , . . . , Rκ . 2. If every word in the hypercube would be covered... 3. ... use 2n codewords =⇒ κ ≥ 2n 2n κ≥ ≥ ≈ 2n(1−H((1+δ)p)) . |R| 2 · 2nH((1+δ)p) 1 . 5. 6. 7. Consider all possible strings of length k such that 2k ≤ κ. Map ith string in {0, 1}k to the center Ci of the ith ring Ri . If send Ci =⇒ receiver gets a string in Ri . Decoding is easy - ﬁnd the ring Ri containing the received string, take its center string Ci , and output the original string it was mapped to. 8. How many bits? ( ( )) k = ⌊log κ⌋ = n 1 − H (1 + δ)p ≈ n(1 − H (p)), 4 Sariel (UIUC) CS573 23 Fall 2013 23 / 24 Some more intuition... . Pick as many disjoint rings as possible: R1 , . . . , Rκ . 2. If every word in the hypercube would be covered... 3. ... use 2n codewords =⇒ κ ≥ 2n 2n κ≥ ≥ ≈ 2n(1−H((1+δ)p)) . |R| 2 · 2nH((1+δ)p) 1 . 5. 6. 7. Consider all possible strings of length k such that 2k ≤ κ. Map ith string in {0, 1}k to the center Ci of the ith ring Ri . If send Ci =⇒ receiver gets a string in Ri . Decoding is easy - ﬁnd the ring Ri containing the received string, take its center string Ci , and output the original string it was mapped to. 8. How many bits? ( ( )) k = ⌊log κ⌋ = n 1 − H (1 + δ)p ≈ n(1 − H (p)), 4 Sariel (UIUC) CS573 23 Fall 2013 23 / 24 Some more intuition... . Pick as many disjoint rings as possible: R1 , . . . , Rκ . 2. If every word in the hypercube would be covered... 3. ... use 2n codewords =⇒ κ ≥ 2n 2n κ≥ ≥ ≈ 2n(1−H((1+δ)p)) . |R| 2 · 2nH((1+δ)p) 1 . 5. 6. 7. Consider all possible strings of length k such that 2k ≤ κ. Map ith string in {0, 1}k to the center Ci of the ith ring Ri . If send Ci =⇒ receiver gets a string in Ri . Decoding is easy - ﬁnd the ring Ri containing the received string, take its center string Ci , and output the original string it was mapped to. 8. How many bits? ( ( )) k = ⌊log κ⌋ = n 1 − H (1 + δ)p ≈ n(1 − H (p)), 4 Sariel (UIUC) CS573 23 Fall 2013 23 / 24 Some more intuition... . Pick as many disjoint rings as possible: R1 , . . . , Rκ . 2. If every word in the hypercube would be covered... 3. ... use 2n codewords =⇒ κ ≥ 2n 2n κ≥ ≥ ≈ 2n(1−H((1+δ)p)) . |R| 2 · 2nH((1+δ)p) 1 . 5. 6. 7. Consider all possible strings of length k such that 2k ≤ κ. Map ith string in {0, 1}k to the center Ci of the ith ring Ri . If send Ci =⇒ receiver gets a string in Ri . Decoding is easy - ﬁnd the ring Ri containing the received string, take its center string Ci , and output the original string it was mapped to. 8. How many bits? ( ( )) k = ⌊log κ⌋ = n 1 − H (1 + δ)p ≈ n(1 − H (p)), 4 Sariel (UIUC) CS573 23 Fall 2013 23 / 24 Some more intuition... . Pick as many disjoint rings as possible: R1 , . . . , Rκ . 2. If every word in the hypercube would be covered... 3. ... use 2n codewords =⇒ κ ≥ 2n 2n κ≥ ≥ ≈ 2n(1−H((1+δ)p)) . |R| 2 · 2nH((1+δ)p) 1 . 5. 6. 7. Consider all possible strings of length k such that 2k ≤ κ. Map ith string in {0, 1}k to the center Ci of the ith ring Ri . If send Ci =⇒ receiver gets a string in Ri . Decoding is easy - ﬁnd the ring Ri containing the received string, take its center string Ci , and output the original string it was mapped to. 8. How many bits? ( ( )) k = ⌊log κ⌋ = n 1 − H (1 + δ)p ≈ n(1 − H (p)), 4 Sariel (UIUC) CS573 23 Fall 2013 23 / 24 Some more intuition... . Pick as many disjoint rings as possible: R1 , . . . , Rκ . 2. If every word in the hypercube would be covered... 3. ... use 2n codewords =⇒ κ ≥ 2n 2n κ≥ ≥ ≈ 2n(1−H((1+δ)p)) . |R| 2 · 2nH((1+δ)p) 1 . 5. 6. 7. Consider all possible strings of length k such that 2k ≤ κ. Map ith string in {0, 1}k to the center Ci of the ith ring Ri . If send Ci =⇒ receiver gets a string in Ri . Decoding is easy - ﬁnd the ring Ri containing the received string, take its center string Ci , and output the original string it was mapped to. 8. How many bits? ( ( )) k = ⌊log κ⌋ = n 1 − H (1 + δ)p ≈ n(1 − H (p)), 4 Sariel (UIUC) CS573 23 Fall 2013 23 / 24 Some more intuition... . Pick as many disjoint rings as possible: R1 , . . . , Rκ . 2. If every word in the hypercube would be covered... 3. ... use 2n codewords =⇒ κ ≥ 2n 2n κ≥ ≥ ≈ 2n(1−H((1+δ)p)) . |R| 2 · 2nH((1+δ)p) 1 . 5. 6. 7. Consider all possible strings of length k such that 2k ≤ κ. Map ith string in {0, 1}k to the center Ci of the ith ring Ri . If send Ci =⇒ receiver gets a string in Ri . Decoding is easy - ﬁnd the ring Ri containing the received string, take its center string Ci , and output the original string it was mapped to. 8. How many bits? ( ( )) k = ⌊log κ⌋ = n 1 − H (1 + δ)p ≈ n(1 − H (p)), 4 Sariel (UIUC) CS573 23 Fall 2013 23 / 24 What is wrong with the above? . Can not ﬁnd such a large set of disjoint rings. 2. Reason is that when you pack rings (or balls) you are going to have wasted spaces around. .3 Overcome this: allow rings to overlap somewhat. 4. Makes things considerably more involved. 5. Details in class notes. 1 Sariel (UIUC) CS573 24 Fall 2013 24 / 24 What is wrong with the above? . Can not ﬁnd such a large set of disjoint rings. 2. Reason is that when you pack rings (or balls) you are going to have wasted spaces around. .3 Overcome this: allow rings to overlap somewhat. 4. Makes things considerably more involved. 5. Details in class notes. 1 Sariel (UIUC) CS573 24 Fall 2013 24 / 24 What is wrong with the above? . Can not ﬁnd such a large set of disjoint rings. 2. Reason is that when you pack rings (or balls) you are going to have wasted spaces around. .3 Overcome this: allow rings to overlap somewhat. 4. Makes things considerably more involved. 5. Details in class notes. 1 Sariel (UIUC) CS573 24 Fall 2013 24 / 24 What is wrong with the above? . Can not ﬁnd such a large set of disjoint rings. 2. Reason is that when you pack rings (or balls) you are going to have wasted spaces around. .3 Overcome this: allow rings to overlap somewhat. 4. Makes things considerably more involved. 5. Details in class notes. 1 Sariel (UIUC) CS573 24 Fall 2013 24 / 24 What is wrong with the above? . Can not ﬁnd such a large set of disjoint rings. 2. Reason is that when you pack rings (or balls) you are going to have wasted spaces around. .3 Overcome this: allow rings to overlap somewhat. 4. Makes things considerably more involved. 5. Details in class notes. 1 Sariel (UIUC) CS573 24 Fall 2013 24 / 24 Notes Sariel (UIUC) CS573 25 Fall 2013 25 / 24 Notes Sariel (UIUC) CS573 26 Fall 2013 26 / 24 Notes Sariel (UIUC) CS573 27 Fall 2013 27 / 24 Notes Sariel (UIUC) CS573 28 Fall 2013 28 / 24

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF

advertising