{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "443188b1-bb71-452d-97ee-9239881f0dfa"
    }
   },
   "source": [
    "# Entropy and information\n",
    "\n",
    "Import numpy and pyplot. Make sure plots are displayed inline. Define some useful quantities.   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true,
    "nbpresent": {
     "id": "b4a55703-d499-48b6-a7c1-cd8d08f83e5b"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Entropy $S$ given by $S = k \\ln ⁡W$ where $W$ is number of microstates. Assume that probability of each microstate is the same. Have $W = \\exp⁡ \\left( \\frac{S}{k}\\right)$. Here, $k$ is the Boltzmann constant. Hence, probability of each microstate state is $p = \\frac{1}{W} = \\frac{1}{\\exp⁡ \\left( \\frac{S}{k} \\right) }$ or $p = \\exp⁡ \\left( −\\frac{S}{k} \\right)$. Statement that entropy increases is therefore equivalent to saying that the number of microstates increases.\n",
    "\n",
    "Typical simple example is 1 gas molecule, initially in a container with volme $V$. If the microstate is defined by the molecule being in a sub-volume of size $\\delta V$, we have $W_i = \\frac{V}{\\delta V}$. The initial entropy is $S_i = k \\ln⁡ \\left( \\frac{V}{\\delta V} \\right)$. If the volume of the container is then doubled (e.g. by removing a partitioning wall) the number of microstates becomes $W_f = \\frac{2V}{\\delta V} =2 W_i$. The entropy is then $S_f = k \\ln \\left(⁡ \\frac{2V}{\\delta V} \\right) = S_i \\ln ⁡2$. The probability of each microstate has decreased by 2, and the entropy has increased by $\\ln ⁡2$.\n",
    "\n",
    "Compare this with the information content in the system. Initially, locating the molecule requires that a maximum of $Q$ questions be answered (perform a binary search, is it in left half, is it in the left half of the chosen half…), where $2^Q = W$. That is, the number of bits of information is $Q = \\log_2 ⁡W = \\frac{\\ln W}{\\ln 2}$. Hence $Q = \\frac {S}{k \\ln 2} $. The amount of information needed to describe a system is therefore proportional to its entropy (or to the log of the reciprocal of the probability that it is in a given microstate, $Q = \\frac {\\ln \\left( \\frac{1}{p} \\right)}{\\ln 2} = \\frac {- \\ln p}{\\ln 2}$). Stating this the other way round, the entropy is proportional to the amount of information in a system $S = Q k \\ln 2$.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-1-1792202bfe94>, line 1)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-1-1792202bfe94>\"\u001b[0;36m, line \u001b[0;32m1\u001b[0m\n\u001b[0;31m    ipython profile create [testprofile]\u001b[0m\n\u001b[0m                  ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "ipython profile create [testprofile] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'get_config' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-10-8b941c722a8c>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mc\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mget_config\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m: name 'get_config' is not defined"
     ]
    }
   ],
   "source": [
    "c = get_config()"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  },
  "nbpresent": {
   "slides": {
    "bfdfca28-0624-407b-a4a5-4e1b5abf121e": {
     "id": "bfdfca28-0624-407b-a4a5-4e1b5abf121e",
     "prev": null,
     "regions": {
      "6d2422b7-c2d6-4357-919b-125342d8b8a9": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "id": "6d2422b7-c2d6-4357-919b-125342d8b8a9"
      }
     }
    }
   },
   "themes": {}
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}