ColombFinal Report (Filling in Template).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### CS/ECE/ISyE 524 &mdash; Introduction to Optimization &mdash; Fall 2018 ###\n",
    "\n",
    "# Financial Factor Analysis #\n",
    "\n",
    "### Tyler Behle\n",
    "\n",
    "   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Email: tbehle@wisc.edu  \n",
    "   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Student ID Number: 9072599682  \n",
    "### Mitchel Berg\n",
    "   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Email: mberg9@wisc.edu  \n",
    "   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Student ID Number: 9072092852  \n",
    "### Christian Colomb\n",
    "   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Email: ccolumb@wisc.edu  \n",
    "   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Student ID Number: 9074912750  \n",
    "### Mike Osmian\n",
    "   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Email: osmian@wisc.edu  \n",
    "   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Student ID Number: 9073922677  \n",
    "\n",
    "### Table of Contents\n",
    "\n",
    "1. [Introduction](#1.-Introduction)\n",
    "1. [Mathematical Model](#2.-Mathematical-model)\n",
    "1. [Solution](#3.-Solution)\n",
    "1. [Results and Discussion](#4.-Results-and-discussion)\n",
    "1. [Optional Subsection](#4.A.-Feel-free-to-add-subsections)\n",
    "1. [Conclusion](#5.-Conclusion)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Introduction ##\n",
    "\n",
    "Our project aims to determine what financial factors are most likely to indicate the success of a portfolio.  Similar to the classic portfolio optimization problem, our project goes one step above determining which companies an individual should invest in and instead decides which factors are most important when selecting a company to put into a portfolio.  Doing so would allow an individual to set constraints on these factors that have been deemed \"most important\" and cross reference it with a list of companies to determine which would be best to invest in.\n",
    "\n",
    "To accomplish this goal, we created synthetically generated data of factor values over time to establish our model, which we then tested against another subset of similarly generated data.  The generated data we used for our factor values followed a normal distribution centered at zero.  The mathematical model we created uses infinite regularization, lasso regression, and ridge regression to determine which factors correlate most strongly with high returns.  \n",
    "\n",
    "\n",
    "\n",
    "### Steve's Comments\n",
    "\n",
    "The first few sentences should give a quick overview of the entire project. Then, elaborate with a description of the problem that will be solved, a brief history (with [citations](https://en.wikipedia.org/wiki/Citation)) of how the problem came about, why it's important/interesting, and any other interesting facts you'd like to talk about. You should address and explain where the problem data is coming from (research? the internet? synthetically generated?) Also give an outline of the rest of the report.\n",
    "\n",
    "This section should be 300-600 words long, and **should be accessible to a general audience** (don't assume your reader has taken the class!). Feel free to include images if you think it'll be helpful:\n",
    "![fixit flowchart][flow]\n",
    "\n",
    "For more help on using Markdown, see [this reference](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet).\n",
    "\n",
    "[flow]: https://s-media-cache-ak0.pinimg.com/736x/f5/75/c5/f575c53b93724808c6f0211890a54900.jpg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Mathematical model ##\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "### Steve's Notes\n",
    "\n",
    "A discussion of the modeling assumptions made in the problem (e.g. is it from physics? economics? something else?). Explain the decision variables, the constraints, and the objective function. Finally, show the optimization problem written in standard form. Discuss the model type (LP, QP, MIP, etc.). Equations should be formatted in $\\\\LaTeX$ within the IJulia notebook. For this section you may **assume the reader is familiar with the material covered in class**.\n",
    "\n",
    "Here is an example of an equation:\n",
    "\n",
    "$$\\begin{bmatrix}\n",
    "      1 & 2 \\\\\n",
    "       3 & 4\n",
    "    \\end{bmatrix}\n",
    "    \\begin{bmatrix} x \\\\ y \\end{bmatrix} =\n",
    "    \\begin{bmatrix} 5 \\\\ 6 \\end{bmatrix}$$\n",
    "\n",
    "And here is an example of an optimization problem in standard form:\n",
    "$$\\begin{aligned}\n",
    "  \\underset{x \\in \\mathbb{R^n}}{\\text{maximize}}\\qquad& f_0(x) \\\\\n",
    "    \\text{subject to:}\\qquad& f_i(x) \\le 0 && i=1,\\dots,m\\\\\n",
    "    & h_j(x) = 0 && j=1,\\dots,r\n",
    "    \\end{aligned}$$\n",
    "\n",
    "For some quick tips on using $\\LaTeX$, see [this cheat sheet](http://users.dickinson.edu/~richesod/latex/latexcheatsheet.pdf)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Solution ##\n",
    "\n",
    "\n",
    "### Steve's Notes\n",
    "Here, you should code up your model in Julia + JuMP and solve it. Your code should be clean, easy to read, well annotated and commented, and it should compile! You are not allowed to use other programming languages or DCP packages such as `convex.jl`. **I will be running your code**. I suggest having multiple code blocks separated by text blocks that explain the various parts of your solution. You may also solve several versions of your problem with different models/assumptions.\n",
    "\n",
    "It's fine to call external packages such as `Gurobi`, but try to minimize the use of exotic libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[1m\u001b[36mINFO: \u001b[39m\u001b[22m\u001b[36mPrecompiling module Clp.\n",
      "\u001b[39m"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The total number of horses is: 10.0\n",
      "The total number of donkeys is: 0.0\n",
      "The total number of goats is: 0.0\n"
     ]
    }
   ],
   "source": [
    "using JuMP, Clp\n",
    "\n",
    "m = Model(solver = ClpSolver())\n",
    "\n",
    "things = [:horses, :donkeys, :goats]  # these are the things\n",
    "@variable(m, x[things] >= 0)          # the quantities of each of the things (can't be negative)\n",
    "@constraint(m, sum(x) <= 10)          # we can't have any more than 10 things total\n",
    "@objective(m, Max, x[:horses])        # we want to maximize the number of horses\n",
    "solve(m)\n",
    "\n",
    "for i in things\n",
    "    println(\"The total number of \", i, \" is: \", getvalue(x[i]))     # print result\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Academic license - for non-commercial use only\n",
      "33.97731137497124\t7\n",
      "5.766601280303161\t9\n",
      "4.939021111330837\t10\n",
      "3.268006844609805\t22\n",
      "93.106144610311\t29\n",
      "20.11602039854394\t40\n",
      "39.963734450929906\t48\n",
      "4.198201468312401\t61\n",
      "8.118749765484605\t62\n",
      "45.74950756202005\t64\n",
      "52.569678690011244\t72\n",
      "54.24378924251289\t74\n",
      "65.96509616010478\t75\n",
      "19.77952925778222\t77\n",
      "124.81741892147932\t78\n",
      "68.67471325124713\t79\n",
      "67.98420256786969\t80\n",
      "9.041109925112698\t85\n",
      "65.56608226613841\t88\n",
      "25.29991772685001\t90\n",
      "7.55020183525953\t92\n",
      "22.39805858801727\t95\n",
      "15.010843265386955\t98\n",
      "34.35239164705637\t99\n",
      "24\n",
      "Academic license - for non-commercial use only\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "32.939235110665194\t\n",
      "7.367228692905606\t\n",
      "34.10717313663386\t\n",
      "26.245330839433198\t\n",
      "66.12036297117199\t\n",
      "2.801779390681785\t\n",
      "61.75964995245457\t\n",
      "21.54772988801555\t\n",
      "20.78356538192992\t\n",
      "42.975216834843366\t\n",
      "58.27667849900985\t\n",
      "43.35414443188283\t\n",
      "36.01166942681837\t\n",
      "34.65441724421927\t\n",
      "142.16243083009704\t\n",
      "44.645003702299604\t\n",
      "39.28941505574677\t\n",
      "48.0560123615054\t\n",
      "101.26983265981849\t\n",
      "46.709368287611454\t\n",
      "26.394411821493133\t\n",
      "30.864112301690362\t\n",
      "44.371352907358585\t\n",
      "44.31315532547808\t\n",
      "24"
     ]
    }
   ],
   "source": [
    "using JuMP, Gurobi\n",
    "\n",
    "lambda = .1\n",
    "mu = 5000\n",
    "\n",
    "rets = readcsv(\"./returns.csv\")\n",
    "rets = rets[2:length(rets[:,1]), 2]\n",
    "\n",
    "data = readcsv(\"./percent_changes.csv\")\n",
    "dates = data[2:length(data[:,1]), 1] #extract date column\n",
    "data = data[:, 2:length(data[1,:])]\n",
    "cols = data[1,:] #extract column titles\n",
    "data = data[2:length(data[:,1]),:] #remove column titles\n",
    "\n",
    "data = convert(Array{Float64},data)\n",
    "rets = convert(Array{Float64},rets)\n",
    "\n",
    "num_factors = length(data[1,:])\n",
    "num_dates = length(dates)\n",
    "\n",
    "m = Model(solver = GurobiSolver(OutputFlag = 0))\n",
    "\n",
    "@variable(m, b[1:num_factors])\n",
    "@variable(m, yt[1:num_dates])\n",
    "@variable(m, t[1:num_factors])\n",
    "@variable(m, maxt)\n",
    "@constraint(m, [i in 1:num_dates], yt[i] == dot(b, data[i, :]))\n",
    "@constraint(m, t .>= b)\n",
    "@constraint(m, t .>= -b)\n",
    "@constraint(m, [i in 1:num_factors], maxt >= t[i])\n",
    "@objective(m, Min, sum( (rets - yt).^2 ) + lambda * maxt + mu * sum(t)) #LASSO with L-Infinite Regularization\n",
    "# @objective(m, Min, sum( (rets - yt).^2 ) + mu * maxt) #L-Infinite Regularization\n",
    "# @objective(m, Min, sum( (rets - yt).^2 ) + mu * sum(t)) #LASSO\n",
    "# @objective(m, Min, sum( (rets - yt).^2 ) + lambda * sum(b.^2))\n",
    "# @objective(m, Min, sum( (rets - yt).^2 ))\n",
    "\n",
    "solve(m)\n",
    "\n",
    "selection = []\n",
    "\n",
    "bs = getvalue(b)\n",
    "# bsort = sort(bs)\n",
    "# for i in 1:length(bsort)\n",
    "#     for j in 1:length(bs)\n",
    "#         if bs[j] == bsort[i]\n",
    "#             print(j, \"\\t\", bsort[i],\"\\n\")\n",
    "#         end\n",
    "#     end\n",
    "# end\n",
    "# print(count,\"\\n\")\n",
    "count = 0\n",
    "for i in 1:length(bs)\n",
    "    if bs[i] > 0.0000001\n",
    "        print(bs[i], \"\\t\", cols[i]+1,\"\\n\")\n",
    "        count += 1\n",
    "        push!(selection, i)\n",
    "    end\n",
    "end\n",
    "print(count,\"\\n\")\n",
    "\n",
    "sel_data = data[:, selection]\n",
    "num_factors = length(sel_data[1,:])\n",
    "\n",
    "m2 = Model(solver = GurobiSolver(OutputFlag = 0))\n",
    "\n",
    "@variable(m2, b2[1:num_factors])\n",
    "@variable(m2, yt2[1:num_dates])\n",
    "@constraint(m2, [i in 1:num_dates], yt2[i] == dot(b2, sel_data[i, :]))\n",
    "@objective(m2, Min, sum( (rets - yt2).^2 ) + lambda * sum(b2.^2))\n",
    "\n",
    "solve(m2)\n",
    "\n",
    "print(\"\\n\\n\\n\\n\")\n",
    "bs2 = getvalue(b2)\n",
    "count = 0\n",
    "for i in 1:length(bs2)\n",
    "    if bs2[i] > 0.0000001\n",
    "        print(bs2[i], \"\\t\\n\")\n",
    "        count += 1\n",
    "    end\n",
    "end\n",
    "print(count)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1000×100 Array{Float64,2}:\n",
       "  0.00204736  -0.135141    -0.107811     …  -0.0209075   -0.000109472\n",
       " -0.157469    -0.00915567  -0.0457541       -0.0340217   -0.0456843  \n",
       " -0.136031     0.0542648   -0.000262748      0.0276062    0.00842596 \n",
       " -0.0155871    0.062749    -0.00808564      -0.00299174  -0.0198909  \n",
       " -0.122152    -0.173265     0.0730716        0.0736405    0.0946953  \n",
       "  0.0563802    0.00203149  -0.0232352    …  -0.00519397  -0.00964657 \n",
       " -0.0135168    0.217249    -0.0537929        0.00579626  -0.00440167 \n",
       "  0.0114186   -0.108382    -0.0513474       -0.0868291   -0.0694285  \n",
       " -0.0128393   -0.152965    -0.0638119       -0.1873      -0.186537   \n",
       " -0.233902     0.0716506   -0.0845752       -0.0494206   -0.0651204  \n",
       "  0.0580849    0.0181498   -0.0119948    …   0.0104717   -0.00358676 \n",
       "  0.0990594    0.238115     0.0539379        0.169785     0.154893   \n",
       " -0.0723073   -0.0608562   -0.0309915       -0.0190185   -0.0234224  \n",
       "  ⋮                                      ⋱                           \n",
       "  0.0272768    0.0324902   -0.0182919       -0.0431369   -0.047469   \n",
       " -0.0823332    0.171802     0.00587419       0.0978867    0.0713905  \n",
       " -0.119564     0.0158183    0.00479289   …   0.0516382    0.0381268  \n",
       " -0.164833    -0.0257867   -0.105941        -0.0140812   -0.0217406  \n",
       " -0.0669654   -0.0365968    0.0978927        0.00168367  -0.00211548 \n",
       " -0.017101    -0.163193     0.0184294       -0.0181693    0.00990416 \n",
       " -0.101731    -0.134995     0.0453552        0.0386861    0.058312   \n",
       "  0.0334631    0.187754    -0.0569455    …  -0.014523    -0.0178258  \n",
       "  0.136992     0.0222245    0.0684247        0.0513142    0.0533902  \n",
       "  0.135896     0.0186555    0.0294959        0.00557108   0.0139576  \n",
       " -0.103361    -0.0343203    0.0516322        0.0492652    0.0354978  \n",
       " -0.0985519   -0.109535     0.0613913        0.0184752    0.0180782  "
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.001175924781466682\tdebt\n",
      "0.001176379876447029\tdebtusd\n",
      "0.0011763428882229532\tebitdausd\n",
      "0.0011760615672569486\tepsusd\n",
      "0.00117567451945085\tequityusd\n",
      "0.0011763651023670068\tevebit\n",
      "0.0011763909143901509\tgrossmargin\n",
      "0.00047536625290461675\tinvestmentsc\n",
      "0.0011762452646425403\tncfcommon\n",
      "0.0011755777254313419\tncfi\n",
      "0.00038442906607985056\tncfinv\n",
      "0.0009771962461347118\tnetincdis\n",
      "0.0011761911436200718\tnetincnci\n",
      "0.0003095008901513142\tpayables\n",
      "0.0011763990948827534\tpe\n",
      "0.00018843827921669152\tprefdivis\n",
      "0.0011675827525512285\trevenueusd\n",
      "4.163285033462269e-7\tsharefactor\n",
      "0.0005278349385322946\tsps\n",
      "0.0011754952358400995\ttaxexp\n",
      "20"
     ]
    }
   ],
   "source": [
    "bs = getvalue(b)\n",
    "count = 0\n",
    "for i in 1:length(bs)\n",
    "    if bs[i] > 0.0000001\n",
    "        print(bs[i], \"\\t\", cols[i],\"\\n\")\n",
    "        count += 1\n",
    "    end\n",
    "end\n",
    "print(count)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7×20 Array{Float64,2}:\n",
       " -0.00103938   4.06405    0.156267    …  -1.14286  -4.03807    2.05337  \n",
       "  0.30437      6.46051    0.00650794      0.0       0.245311   0.498002 \n",
       " -0.634001    -0.279962  -3.18678         0.0      -1.1883    -2.03745  \n",
       " -2.86162     -0.494445  -1.13603         0.0       2.1999     0.89586  \n",
       " -0.676859    -2.1649    -1.48907         0.0       1.08407   -0.244727 \n",
       "  3.36258      1.34137   -0.886255    …   0.0      -0.399574  -1.8382   \n",
       "  0.0807311   -0.166374  47.4322          0.0      -2.67365    0.0910096"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sel_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num_factors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.2888055051188396e-5"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "getobjectivevalue(m)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7×3 Array{Float64,2}:\n",
       "  0.0434216   0.106106   -0.716398\n",
       " -0.673924   -0.2201      4.23729 \n",
       " -2.1544     -0.597726   -0.476165\n",
       "  1.75225    -1.36557     0.574577\n",
       " -0.70383     6.28081    -4.20583 \n",
       "  0.141441   -0.0405702  -0.926255\n",
       " -0.386225    0.556266    0.91608 "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[:,[1,20,37]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Results and discussion ##\n",
    "\n",
    "Here, you display and discuss the results. Show figures, plots, images, trade-off curves, or whatever else you can think of to best illustrate your results. The discussion should explain what the results mean, and how to interpret them. You should also explain the limitations of your approach/model and how sensitive your results are to the assumptions you made.\n",
    "\n",
    " Use plots (see `PyPlot` examples from class), or you can display results in a table like this:\n",
    "\n",
    "| Tables        | Are          | Cool  |\n",
    "| ------------- |:-------------| -----:|\n",
    "| col 3 is      |right-aligned |\\$1600 |\n",
    "|  colons       | align columns|  \\$12 |\n",
    "| zebra stripes |    are neat  |   \\$1 |\n",
    "\n",
    "### 4.A. Feel free to add subsections\n",
    "\n",
    "#### 4.A.a. or subsubsections"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Conclusion ##\n",
    "\n",
    "Summarize your findings and your results, and talk about at least one possible future direction; something that might be interesting to pursue as a follow-up to your project."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Julia 0.6.4",
   "language": "julia",
   "name": "julia-0.6"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "0.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}