Let $X$ be a finite set and $p = (p_x)_{x \in X}$ a probability distribution on $X$. We assume that $p$ charges all points of $X$: $p_x > 0$ for all $x \in X$. We call entropy of $p$ the quantity $$H(p) = -\sum_{x \in X} p_x \ln(p_x)$$ We consider the set $Q_X = \{\boldsymbol{q} = (q_x)_{x \in X} \in \mathbb{R}^X \mid \forall x \in X, q_x \geq 0\}$. For all $\boldsymbol{q}, \boldsymbol{q}' \in Q_X$ such that $q_x' > 0$ for all $x \in X$, we define: $$\mathrm{KL}(\boldsymbol{q}, \boldsymbol{q}') = \sum_{x \in X} \varphi(q_x / q_x') q_x'$$ with $\varphi : \mathbb{R}_+ \rightarrow \mathbb{R}$ defined by $\varphi(x) = x \log(x) - x + 1$ for $x > 0$ and extended to 0 by continuity. (a) Specify $\varphi(0)$. (b) Verify that $\varphi$ is continuous, strictly convex, positive and that $\varphi(x) = 0$ if and only if $x = 1$. (c) Show that $Q_X$ is convex and that $\boldsymbol{q} \mapsto \mathrm{KL}(\boldsymbol{q}, \boldsymbol{q}')$ is strictly convex, positive and vanishes if and only if $q = q'$.
Let $X$ be a finite set and $p = (p_x)_{x \in X}$ a probability distribution on $X$. We assume that $p$ charges all points of $X$: $p_x > 0$ for all $x \in X$. We call entropy of $p$ the quantity
$$H(p) = -\sum_{x \in X} p_x \ln(p_x)$$
We consider the set $Q_X = \{\boldsymbol{q} = (q_x)_{x \in X} \in \mathbb{R}^X \mid \forall x \in X, q_x \geq 0\}$. For all $\boldsymbol{q}, \boldsymbol{q}' \in Q_X$ such that $q_x' > 0$ for all $x \in X$, we define:
$$\mathrm{KL}(\boldsymbol{q}, \boldsymbol{q}') = \sum_{x \in X} \varphi(q_x / q_x') q_x'$$
with $\varphi : \mathbb{R}_+ \rightarrow \mathbb{R}$ defined by $\varphi(x) = x \log(x) - x + 1$ for $x > 0$ and extended to 0 by continuity.\\
(a) Specify $\varphi(0)$.\\
(b) Verify that $\varphi$ is continuous, strictly convex, positive and that $\varphi(x) = 0$ if and only if $x = 1$.\\
(c) Show that $Q_X$ is convex and that $\boldsymbol{q} \mapsto \mathrm{KL}(\boldsymbol{q}, \boldsymbol{q}')$ is strictly convex, positive and vanishes if and only if $q = q'$.