2019-06-02-climate-with-keras.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Climate classification with Keras\n",
    "\n",
    "One of my favorite hack projects was trying to create a [climate classification](2018-06-11-climate-classification-with-neural-nets.ipynb) by clustering learned embeddings of weather stations.\n",
    "\n",
    "The original model was written in TensorFlow. Since then, I've started to experiment with [Keras](https://keras.io). Because the climate classifier is pretty simple neural network, I rewrote the model using Keras and saved many lines of code.\n",
    "\n",
    "For the problem description and data preparation, see the [original post](2018-06-11-climate-classification-with-neural-nets.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from keras.models import Model\n",
    "from keras.layers import Dense, Flatten, Embedding, Input, Reshape, Concatenate\n",
    "from sklearn.cluster import KMeans"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the data\n",
    "\n",
    "I'm using the same files that I created for the [original post](2018-06-11-climate-classification-with-neural-nets.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "DATA_PATH = 'data/weather'\n",
    "\n",
    "matrix_file = os.path.join(DATA_PATH, 'data/gsn-2000-TMAX-TMIN-PRCP.npz')\n",
    "\n",
    "# column labels\n",
    "STATION_ID_COL = 0\n",
    "MONTH_COL = 1\n",
    "DAY_COL = 2\n",
    "VAL_COLS_START = 3\n",
    "TMAX_COL = 3\n",
    "TMIN_COL = 4\n",
    "PRCP_COL = 5\n",
    "\n",
    "with np.load(matrix_file) as npz_data:\n",
    "    weather_data = npz_data['data'].astype(np.int32)\n",
    "\n",
    "# I decided to switch over to using the day of the year instead of two \n",
    "# eh, this isn't perfect (it assumes all months have 31 days), but it helps differentiate \n",
    "# the first of the month vs the last. \n",
    "weather_data_day_of_year_data = 31 * (weather_data[:, MONTH_COL] - 1) + (weather_data[:, DAY_COL] - 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "station_id_data = weather_data[:, STATION_ID_COL].reshape(-1, 1)\n",
    "weather_data_day_of_year_data = weather_data_day_of_year_data.reshape(-1, 1)\n",
    "weather_prediction = weather_data[:, VAL_COLS_START:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define the network\n",
    "\n",
    "I'll use the same network as [before](2018-06-11-climate-classification-with-neural-nets.ipynb).\n",
    "\n",
    "![](images/2018-06-11-nn.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# network parameters\n",
    "BATCH_SIZE = 50\n",
    "EMBEDDING_SIZE = 20\n",
    "HIDDEN_UNITS = 40\n",
    "\n",
    "# and classification parameters. How many climates I want.\n",
    "CLUSTER_NUMBER = 6\n",
    "\n",
    "# count how many stations there are\n",
    "NUM_STATIONS = np.max(weather_data[:, STATION_ID_COL]) + 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /Users/jessica/miniconda3/envs/blog/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Colocations handled automatically by placer.\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "station_id_input (InputLayer)   (None, 1)            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedded_stations (Embedding)   (None, 1, 20)        24360       station_id_input[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "embedded_station_reshape (Resha (None, 20)           0           embedded_stations[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "month_day_input (InputLayer)    (None, 1)            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "station_and_day (Concatenate)   (None, 21)           0           embedded_station_reshape[0][0]   \n",
      "                                                                 month_day_input[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "hidden (Dense)                  (None, 40)           880         station_and_day[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "prediction (Dense)              (None, 3)            123         hidden[0][0]                     \n",
      "==================================================================================================\n",
      "Total params: 25,363\n",
      "Trainable params: 25,363\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "station_id_input = Input(shape=(1,), name='station_id_input')\n",
    "month_day_input = Input(shape=(1,), name='month_day_input')\n",
    "\n",
    "# Feed the station id through the embedding. This embeddings variable\n",
    "# is the whole point of this network!\n",
    "embedded_stations = Embedding(\n",
    "    output_dim=EMBEDDING_SIZE, \n",
    "    input_dim=NUM_STATIONS,\n",
    "    name='embedded_stations'\n",
    ")(station_id_input)\n",
    "\n",
    "embedded_station_reshape = Reshape((EMBEDDING_SIZE,), name='embedded_station_reshape')(embedded_stations)\n",
    "\n",
    "station_and_day = Concatenate(name='station_and_day')([embedded_station_reshape, month_day_input])\n",
    "\n",
    "# Now build a little network that can learn to predict the weather\n",
    "hidden = Dense(HIDDEN_UNITS, activation='relu', name='hidden')(station_and_day)\n",
    "\n",
    "prediction = Dense(\n",
    "    3,  # Output for each of the attributes of the weather prediction (max, min, precipitation)\n",
    "    activation=None,  # don't use an activation on predictions\n",
    "    name='prediction'\n",
    ")(hidden)\n",
    "\n",
    "\n",
    "model = Model(inputs=[station_id_input, month_day_input], outputs=prediction)\n",
    "\n",
    "model.compile(optimizer='adam',\n",
    "              loss='mean_squared_error')\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    __________________________________________________________________________________________________\n",
    "    Layer (type)                    Output Shape         Param #     Connected to                     \n",
    "    ==================================================================================================\n",
    "    station_id_input (InputLayer)   (None, 1)            0                                            \n",
    "    __________________________________________________________________________________________________\n",
    "    embedded_stations (Embedding)   (None, 1, 20)        24360       station_id_input[0][0]           \n",
    "    __________________________________________________________________________________________________\n",
    "    embedded_station_reshape (Resha (None, 20)           0           embedded_stations[0][0]          \n",
    "    __________________________________________________________________________________________________\n",
    "    month_day_input (InputLayer)    (None, 1)            0                                            \n",
    "    __________________________________________________________________________________________________\n",
    "    station_and_day (Concatenate)   (None, 21)           0           embedded_station_reshape[0][0]   \n",
    "                                                                     month_day_input[0][0]            \n",
    "    __________________________________________________________________________________________________\n",
    "    hidden (Dense)                  (None, 40)           880         station_and_day[0][0]            \n",
    "    __________________________________________________________________________________________________\n",
    "    prediction (Dense)              (None, 3)            123         hidden[0][0]                     \n",
    "    ==================================================================================================\n",
    "    Total params: 25,363\n",
    "    Trainable params: 25,363\n",
    "    Non-trainable params: 0\n",
    "    __________________________________________________________________________________________________"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /Users/jessica/miniconda3/envs/blog/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n",
      "Epoch 1/1\n",
      "2417066/2417066 [==============================] - 178s 74us/step - loss: 4390.3217\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x10666c4e0>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(\n",
    "    [\n",
    "        station_id_data,\n",
    "        weather_data_day_of_year_data,\n",
    "    ], \n",
    "    weather_prediction.reshape(-1, 3), \n",
    "    epochs=1,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    Epoch 1/1\n",
    "    2417066/2417066 [==============================] - 178s 74us/step - loss: 4390.3217\n",
    "\n",
    "\n",
    "There is less boilerplate in the Keras code compared to my TensorFlow implementation in the [original post](2018-06-11-climate-classification-with-neural-nets.ipynb). I think it's cool that most of the code is doing work describing the network."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classify the embeddings\n",
    "\n",
    "Finally, I run KMeans on the trained embeddings to assign a \"climate\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "trained_embeddings = model.get_layer('embedded_stations').get_weights()[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(os.path.join(DATA_PATH, 'data/stations') )as f:\n",
    "    list_of_stations = [line.strip() for line in f.readlines()]\n",
    "\n",
    "kmeans = KMeans(n_clusters=CLUSTER_NUMBER, random_state=0).fit(trained_embeddings)\n",
    "\n",
    "# I can export the classification here:\n",
    "for station, label in zip(list_of_stations, kmeans.labels_):\n",
    "    #print('{}\\t{}'.format(station, label))\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](images/2019-06-02-new-map.png)\n",
    "\n",
    "I get similar climates as before. Since there's randomness in the neural network initializations and batches and in the KMeans, I wouldn't expect to get exactly the same. For example, the latitude boundaries on the East coast have shifted compared to before.\n",
    "\n",
    "For this post, I also switched to [Cartopy](https://github.com/SciTools/cartopy) from [Basemap](https://matplotlib.org/basemap/users/intro.html#cartopy-new-management-and-eol-announcement)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "jessica": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}