Skalierbare_ Datenanalyse_ mit_ IPython_I.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<center>\n",
    "<H1>Skalierbare Datenanalyse mit IPython</H1>\n",
    "<BR><BR>\n",
    "<H3>Data2Day 2016</H3>\n",
    "<BR><BR><BR>\n",
    "<H3><i>Janis Keuper</i></H3>\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<H2>Inhalt</H2>\n",
    "<BR>\n",
    "<BR>\n",
    "<H3>Teil I: Die IPython Umgebung für Datenanalysen</H3><BR><BR>\n",
    "<H3>Teil II: Skalierbare Lösungen mit IPython und Dask</H3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<H2>Die IPython Umgebung für Datenanalyse</H2>\n",
    "<img src=\"fig/PythonEco.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<H3>IPython Beiträge auf der Data2Day16</H3>\n",
    "<BR>\n",
    "* Do, 14:20 <i>Alexander Hendorf</i> <b>Einführung in Datenanalyse mit Pandas</b>\n",
    "* Do, 16:20 <i>Oliver Zeigermann</i> <b>Machine Learning im Browser mit Deep Neural Networks</b>\n",
    "* Do, 16:20 <i>Wilfried Hoge, Stephan Reimann</i> <b>Tools für den Data Scientist/Analyst</b>\n",
    "* Fr, 9:00 <i>Antje Relitz, Daniel Kirsch</i> <b>Workshop: Einführung in Data Science</b>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "<H2>Jupyter Notebooks</H2>\n",
    "* Interactive web frontend for pogramming and data analyses \n",
    "    * supports not only Python code (also Julia, R, Java, C++, ...)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<H2>Interactive code execution in \"Cells\"</H2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print \"hello world\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "A=[1, 2, 4] #python list\n",
    "#support auto completion\n",
    "A"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    " #and API documentatio\n",
    "?A.sort()   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<H2>Markup cells</H2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<H3>Simple Markup</H3>\n",
    "# This is a Header "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<H3>HTML</H3> \n",
    "<table border=2 width=400>\n",
    "<tr>\n",
    "<td>A</td><td>B</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td>4</td><td>6</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<td>5</td><td>1</td>\n",
    "</tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<H3>Images</H3>\n",
    "<center>\n",
    "<img src=\"fig/jupyter.png\" width=50%>\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<H3>Videos</H3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "from IPython.display import YouTubeVideo\n",
    "YouTubeVideo('9k8XnxKgnFg',720,567)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<H3>LaTeX</H3> \n",
    "\n",
    "Ans so we need to compute $a=\\sum_{i=0}^n x_i^2$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<H3>Jupyter Notebooks and Version Control</H3>\n",
    "* Notebooks are pure text files\n",
    "```\n",
    "{\n",
    " \"cells\": [\n",
    "  {\n",
    "   \"cell_type\": \"markdown\",\n",
    "   \"metadata\": {\n",
    "    \"slideshow\": {\n",
    "     \"slide_type\": \"slide\"\n",
    "    }\n",
    "   },\n",
    "   \"source\": [\n",
    "    \"<center>\\n\",\n",
    "    \"<H1>Skalierbare Datenanalyse mit IPython</H1>\\n\",\n",
    "    \"\n",
    "   ]\n",
    "  },\n",
    "...\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<H2>Jupyter Notebooks on GitHub</H2>\n",
    "\n",
    "* find this prensentation Github:\n",
    "https://github.com/keuperj/Data2Day16\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<H2>Jupyter Presentations - RISE</H2>\n",
    "<H3>Live presentations with Reveal.js"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<H2>Exporting Jupyter Notebooks</H2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<H3>Executing Jupyter Notebooks</H3>\n",
    "* local server \n",
    "* remote server (multi user frameworks)\n",
    "* or in the cloud"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<H3>Jupyter TmpNB Server in the Cloud</H3>\n",
    "<img src=\"fig/tmpnb.png\" width=75%>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<H2>Connect to: http://bit.ly/????? (link no longer available)</H2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<H2>NumPy and SciPy - the Mathematical Core</H2>\n",
    "* used in almost all numerical computations using Python\n",
    "* high-performance vector, matrix and higher-dimensional data structures\n",
    "* implemented in C and Fortran\n",
    "    * very high performance\n",
    "    * parallel opperations with Intel MKL\n",
    "        * https://software.intel.com/en-us/intel-distribution-for-python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np #this is a strong community convetion!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "A=np.random.rand(5,5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "np.shape(A)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "#show: print, indexing, opperations\n",
    "A"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<h3>Numpy Notes:</H3>\n",
    "* Numpy array do have strong typing!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "A.dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* Numpy generates references by default!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "A=np.random.rand(5,5)\n",
    "B=A # reference\n",
    "#B=np.copy(A) #explicit copy\n",
    "B[0,0]=0\n",
    "A"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<H2>Pandas - the Library for Structured Data</H2>\n",
    "* library providing high-performance, easy-to-use data structures and data analysis tools\n",
    "* build on top of NumPy\n",
    "* main functions\n",
    "    * importing data (from CSV, Excel, Text, ...)\n",
    "    * mangling and cleaning of data (missing values, NaNs, re-sampling, ...)\n",
    "    * new data structure:DataFrame (powerfull table representation)\n",
    "    * satistic tools\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd #also as strong convention"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "#example data import\n",
    "u_cols = ['user_id', 'age', 'sex', 'occupation', 'zip_code']\n",
    "users = pd.read_csv('movielens100k/u.user', sep='|', names=u_cols)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "#plot first 10 users\n",
    "users[0:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<H3>Queries on Pandas DataFrames</H3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "#compute the mean age of all users\n",
    "np.mean(users.age)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "#What occupations are represented in the dataset?\n",
    "np.unique(users.occupation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "#count the number of femals under 30\n",
    "youngfemales = users[(users.sex=='F') & (users.age < 30)]\n",
    "print len(youngfemales)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<H2>Visualization with Matplotlib</H2>\n",
    "* see http://matplotlib.org/gallery.html "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import matplotlib.pylab as plt #again, a name convention\n",
    "import matplotlib.mlab as mlab\n",
    "#ipython magic command to enable matplotlib within jupyter\n",
    "%matplotlib inline "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(11,11))\n",
    "mu, sigma = np.mean(users.age), np.std(users.age)\n",
    "n, bins, patches = plt.hist(users.age, 50, normed=1, facecolor='green', alpha=0.75)\n",
    "y = mlab.normpdf( bins, mu, sigma);l = plt.plot(bins, y, 'r--', linewidth=3)\n",
    "plt.xlabel('Age');plt.ylabel('Probability');plt.grid(True);plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<H2>Machine Learning with Scikit-Learn</H2>\n",
    "<img src=\"fig/scikit-learn.png\">\n",
    "* Machine learning library written in __Python__\n",
    "* __Simple and efficient__, for both experts and non-experts\n",
    "* Classical, __well-established machine learning algorithms__\n",
    "* Shipped with <a href=\"http://scikit-learn.org/dev/documentation.html\">documentation</a> and <a href=\"http://scikit-learn.org/dev/auto_examples/index.html\">examples</a>\n",
    "* __BSD 3 license__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Generate data\n",
    "%matplotlib inline\n",
    "from matplotlib import pyplot as plt\n",
    "from sklearn.datasets import make_blobs\n",
    "X, y = make_blobs(n_samples=1000, centers=20, random_state=123)\n",
    "labels = [\"b\", \"r\"]\n",
    "y = np.take(labels, (y < 10))\n",
    "\n",
    "# Plot\n",
    "plt.figure()\n",
    "for label in labels:\n",
    "    mask = (y == label)\n",
    "    plt.scatter(X[mask, 0], X[mask, 1], c=label)\n",
    "plt.xlim(-10, 10)\n",
    "plt.ylim(-10, 10)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.svm import SVC\n",
    "clf = SVC(kernel=\"rbf\", probability=True)  # try kernel=\"rbf\" instead\n",
    "clf.fit(X, y)\n",
    "\n",
    "#plotting the result\n",
    "xlim=(-10, 10); ylim=(-10, 10); n_steps=250\n",
    "xx, yy = np.meshgrid(np.linspace(xlim[0], xlim[1], n_steps), \n",
    "                         np.linspace(ylim[0], ylim[1], n_steps))\n",
    "z = clf.predict_proba(np.c_[xx.ravel(), yy.ravel()])[:, 1]\n",
    "z = z.reshape(xx.shape)\n",
    "plt.contourf(xx, yy, z, alpha=0.8, cmap=plt.cm.RdBu_r)\n",
    "plt.scatter(X[:, 0], X[:, 1], c=y)\n",
    "plt.xlim(*xlim)\n",
    "plt.ylim(*ylim)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<H2>Data Science in Action: Building a Recommender System</H2>\n",
    "<H3>Motivation</H3>\n",
    "<img src=\"fig/amazon.png\" width=75%>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<H3>Definition</H3>\n",
    "\n",
    ">The goal of a recommender system is to make product or service recommendations to people. Of course, these recommendations should be for products or services they’re more likely to want to want buy or consume. In a word, recommenders want to identify items that are more relevant. [Gartner]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<H4>Types of Recommender Systems</H4>\n",
    "* <b>Content-based</b> filtering: “If you liked this item, you might also like …”\n",
    "* <b>Collaborative</b> filtering\n",
    "    * <b>Item-Item</b> filtering: “Customers who liked this item also liked …”\n",
    "    * <b>User-Item</b> filtering: “Customers who are similar to you also liked …”"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<H2>Target Application: Movie recommendation</H2>\n",
    "* Data: Database with movie ratings from http://www.movielens.org/\n",
    "* 943 users, 1682 movies and 100000 ratings\n",
    "<img src=\"fig/movielens.png\" width=50%>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<H3>Get the Data</H3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "#only needed if you do not have the data already\n",
    "import urllib2\n",
    "f = urllib2.urlopen(\"http://files.grouplens.org/datasets/movielens/ml-100k.zip\")\n",
    "with open('ml-100k.zip','wb') as output:\n",
    "  output.write(f.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<H2>Data Structure</H2>\n",
    "* Relational Database with several tables [user, ratings, movies]\n",
    "<img src=\"fig/mtable.png\" width=75%>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "#read data to DataFrames\n",
    "u_cols = ['user_id', 'age', 'sex', 'occupation', 'zip_code']\n",
    "users = pd.read_csv('movielens100k/u.user', sep='|', names=u_cols)\n",
    "\n",
    "r_cols = ['user_id', 'movie_id', 'rating', 'timestamp']\n",
    "ratings = pd.read_csv('movielens100k/u.data', sep='\\t', names=r_cols)\n",
    "\n",
    "m_cols=['movie_id', 'title', 'release date', 'video release date', 'IMDb_URL', 'unknown', 'Action', 'Adventure', 'Animation', 'Childrens', 'Comedy', 'Crime', 'Documentary', 'Drama', 'Fantasy', 'Film-Noir', 'Horror', 'Musical', 'Mystery', 'Romance', 'Sci-Fi', 'Thriller', 'War', 'Western']\n",
    "movies = pd.read_csv('movielens100k/u.item', sep='|', names=m_cols  )\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<H2>Let's build a Collaborative Item-Item Recommender</H2>\n",
    "* Aswering the question: “Customers who liked this item also liked …”\n",
    "<BR><BR>\n",
    "<H4>Input: single Movie (ID)</H4><BR>\n",
    "<H4>Output: Top 5 recommended Movies (ID)</H4>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<H3>Step 1: get ID from movie name</H3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "def getIDbyName(name):\n",
    "    return int(movies.movie_id[movies.title.str.contains(name)][:1]), str(movies.title[movies.title.str.contains(name)][:1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "getIDbyName(\"Star\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<H3>Step 2: get all users who rated that movie and gave a rating of 5</H3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "movieID, movieNAme =getIDbyName(\"Star\")\n",
    "userIDs=ratings.user_id[(ratings.movie_id==movieID)&(ratings.rating==5)].unique()\n",
    "userIDs.sort()\n",
    "userIDs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<H3>Step 3: get all ratings >=4 of these users</H3> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "selectedRatings=ratings[(ratings.user_id.isin(userIDs)) & (ratings.rating >=4)]\n",
    "selectedRatings.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<H3>Step 4: group by movie id</H3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "grouped=selectedRatings.groupby('movie_id', as_index=False).sum()\n",
    "\n",
    "#drop input movie if included\n",
    "grouped = grouped.drop(grouped[grouped.movie_id==movieID].index)\n",
    "#sort and get top 5\n",
    "top5=grouped.sort_values('rating',0,False)\n",
    "top5.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<H3>Step 5: get movie names of the Top 5 </H3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "top5ID=top5.movie_id.unique()-1\n",
    "\n",
    "def getNameByID(IDs):\n",
    "    res=movies.ix[IDs]\n",
    "    return res.title\n",
    "getNameByID(top5ID[0:5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "#putting it all together\n",
    "def getIDbyName(name, movies):\n",
    "    if np.size(movies.movie_id[movies.title.str.contains(name)])<1:\n",
    "        raise ValueError(\"no movie found\") \n",
    "    return int(movies.movie_id[movies.title.str.contains(name)][:1]), str(movies.title[movies.title.str.contains(name)][:1])\n",
    "\n",
    "def getNameByID(IDs, movies):\n",
    "    res=movies.ix[IDs]\n",
    "    return res.title\n",
    "\n",
    "def ItemItemRecommend(title, movies, ratings, top=5):\n",
    "    try:\n",
    "        movieID, movieName =getIDbyName(title, movies)\n",
    "        print \"Rocommendations for \", movieName\n",
    "        userIDs=ratings.user_id[(ratings.movie_id==movieID)&(ratings.rating==5)].unique()\n",
    "        userIDs.sort()\n",
    "        selectedRatings=ratings[(ratings.user_id.isin(userIDs)) & (ratings.rating >=3)]\n",
    "        selectedRatings=selectedRatings[selectedRatings.movie_id!=movieID]\n",
    "        grouped=selectedRatings.groupby('movie_id', as_index=False).sum()\n",
    "        top5=grouped.sort_values('rating',0,False)\n",
    "        top5ID=top5.movie_id.unique()-1\n",
    "        print \"top5ID\",top5ID[0:top]\n",
    "        return getNameByID(top5ID[0:top], movies)\n",
    "    except ValueError as err:\n",
    "        print(err.args)\n",
    "        raise\n",
    "    return getNameByID(top5ID, movies)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "#let's play!\n",
    "ItemItemRecommend(\"Paris\", movies, ratings, 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<H3>What would be next ...?</H3>\n",
    "* change input to n movies (would make a nice exercise)\n",
    "    * expecting much better performance\n",
    "* scale to distributed solution with NoSQL / Hadoop / Dask\n",
    "* look at user-item filters\n",
    "    * need introduction to machine learning first"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<H2>So, want to try this at home?</H2>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<H3>A quick starting guide</H3>\n",
    "1. get and install Anaconda: https://www.continuum.io/downloads\n",
    "2. go to you shell type <i>jupyter notebook</i>\n",
    "3. your browser will open a window and ... HAVE FUN!\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<H2>Questions so far ... ???</H2>"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}