d_traineval.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1> 2d. Distributed training and monitoring </h1>\n",
    "\n",
    "In this notebook, we refactor to call ```train_and_evaluate``` instead of hand-coding our ML pipeline. This allows us to carry out evaluation as part of our training loop instead of as a separate step. It also adds in failure-handling that is necessary for distributed training capabilities.\n",
    "\n",
    "We also use TensorBoard to monitor the training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/envs/py3env/lib/python3.5/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.8.0\n"
     ]
    }
   ],
   "source": [
    "import datalab.bigquery as bq\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import shutil\n",
    "import tensorflow as tf\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2> Input </h2>\n",
    "\n",
    "Read data created in Lab1a, but this time make it more general, so that we are reading in batches.  Instead of using Pandas, we will use Datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "CSV_COLUMNS = ['fare_amount', 'pickuplon','pickuplat','dropofflon','dropofflat','passengers', 'key']\n",
    "LABEL_COLUMN = 'fare_amount'\n",
    "DEFAULTS = [[0.0], [-74.0], [40.0], [-74.0], [40.7], [1.0], ['nokey']]\n",
    "\n",
    "def read_dataset(filename, mode, batch_size = 512):\n",
    "  def _input_fn():\n",
    "    def decode_csv(value_column):\n",
    "      columns = tf.decode_csv(value_column, record_defaults = DEFAULTS)\n",
    "      features = dict(zip(CSV_COLUMNS, columns))\n",
    "      label = features.pop(LABEL_COLUMN)\n",
    "      return features, label\n",
    "    \n",
    "    # Create list of files that match pattern\n",
    "    file_list = tf.gfile.Glob(filename)\n",
    "\n",
    "    # Create dataset from file list\n",
    "    dataset = tf.data.TextLineDataset(file_list).map(decode_csv)\n",
    "    \n",
    "    if mode == tf.estimator.ModeKeys.TRAIN:\n",
    "        num_epochs = None # indefinitely\n",
    "        dataset = dataset.shuffle(buffer_size = 10 * batch_size)\n",
    "    else:\n",
    "        num_epochs = 1 # end-of-input after this\n",
    " \n",
    "    dataset = dataset.repeat(num_epochs).batch(batch_size)\n",
    "    return dataset.make_one_shot_iterator().get_next()\n",
    "  return _input_fn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2> Create features out of input data </h2>\n",
    "\n",
    "For now, pass these through.  (same as previous lab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "INPUT_COLUMNS = [\n",
    "    tf.feature_column.numeric_column('pickuplon'),\n",
    "    tf.feature_column.numeric_column('pickuplat'),\n",
    "    tf.feature_column.numeric_column('dropofflat'),\n",
    "    tf.feature_column.numeric_column('dropofflon'),\n",
    "    tf.feature_column.numeric_column('passengers'),\n",
    "]\n",
    "\n",
    "def add_more_features(feats):\n",
    "  # Nothing to add (yet!)\n",
    "  return feats\n",
    "\n",
    "feature_cols = add_more_features(INPUT_COLUMNS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2> train_and_evaluate </h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def serving_input_fn():\n",
    "  feature_placeholders = {\n",
    "    'pickuplon' : tf.placeholder(tf.float32, [None]),\n",
    "    'pickuplat' : tf.placeholder(tf.float32, [None]),\n",
    "    'dropofflat' : tf.placeholder(tf.float32, [None]),\n",
    "    'dropofflon' : tf.placeholder(tf.float32, [None]),\n",
    "    'passengers' : tf.placeholder(tf.float32, [None]),\n",
    "  }\n",
    "  features = {\n",
    "      key: tf.expand_dims(tensor, -1)\n",
    "      for key, tensor in feature_placeholders.items()\n",
    "  }\n",
    "  return tf.estimator.export.ServingInputReceiver(features, feature_placeholders)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_and_evaluate(output_dir, num_train_steps):\n",
    "  estimator = tf.estimator.LinearRegressor(\n",
    "                       model_dir = output_dir,\n",
    "                       feature_columns = feature_cols)\n",
    "  train_spec=tf.estimator.TrainSpec(\n",
    "                       input_fn = read_dataset('./taxi-train.csv', mode = tf.estimator.ModeKeys.TRAIN),\n",
    "                       max_steps = num_train_steps)\n",
    "  exporter = tf.estimator.LatestExporter('exporter', serving_input_fn)\n",
    "  eval_spec=tf.estimator.EvalSpec(\n",
    "                       input_fn = read_dataset('./taxi-valid.csv', mode = tf.estimator.ModeKeys.EVAL),\n",
    "                       steps = None,\n",
    "                       start_delay_secs = 1, # start evaluating after N seconds\n",
    "                       throttle_secs = 10,  # evaluate every N seconds\n",
    "                       exporters = exporter)\n",
    "  tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using default config.\n",
      "INFO:tensorflow:Using config: {'_keep_checkpoint_every_n_hours': 10000, '_service': None, '_task_type': 'worker', '_log_step_count_steps': 100, '_num_ps_replicas': 0, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f422b3e8908>, '_save_checkpoints_secs': 600, '_global_id_in_cluster': 0, '_tf_random_seed': None, '_evaluation_master': '', '_task_id': 0, '_session_config': None, '_is_chief': True, '_num_worker_replicas': 1, '_save_summary_steps': 100, '_train_distribute': None, '_master': '', '_keep_checkpoint_max': 5, '_save_checkpoints_steps': None, '_model_dir': 'taxi_trained'}\n",
      "INFO:tensorflow:Running training and evaluation locally (non-distributed).\n",
      "INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after 10 secs (eval_spec.throttle_secs) or training is finished.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 1 into taxi_trained/model.ckpt.\n",
      "INFO:tensorflow:loss = 118062.05, step = 1\n",
      "INFO:tensorflow:global_step/sec: 35.024\n",
      "INFO:tensorflow:loss = 41458.06, step = 101 (2.860 sec)\n",
      "INFO:tensorflow:global_step/sec: 36.3148\n",
      "INFO:tensorflow:loss = 38027.477, step = 201 (2.753 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 273 into taxi_trained/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 50944.36.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2019-01-08-04:10:22\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-273\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Finished evaluation at 2019-01-08-04:10:22\n",
      "INFO:tensorflow:Saving dict for global step 273: average_loss = 94.742455, global_step = 273, loss = 37210.098\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Regress: None\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict']\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Classify: None\n",
      "INFO:tensorflow:Signatures EXCLUDED from export because they cannot be be served via TensorFlow Serving APIs:\n",
      "INFO:tensorflow:'regression' : Regression input must be a single string Tensor; got {'dropofflat': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=float32>, 'pickuplon': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=float32>, 'passengers': <tf.Tensor 'Placeholder_4:0' shape=(?,) dtype=float32>, 'dropofflon': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'pickuplat': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>}\n",
      "INFO:tensorflow:'serving_default' : Regression input must be a single string Tensor; got {'dropofflat': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=float32>, 'pickuplon': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=float32>, 'passengers': <tf.Tensor 'Placeholder_4:0' shape=(?,) dtype=float32>, 'dropofflon': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'pickuplat': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>}\n",
      "WARNING:tensorflow:Export includes no default signature!\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-273\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: b\"taxi_trained/export/exporter/temp-b'1546920622'/saved_model.pb\"\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-273\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 274 into taxi_trained/model.ckpt.\n",
      "INFO:tensorflow:loss = 51777.96, step = 274\n",
      "INFO:tensorflow:global_step/sec: 34.291\n",
      "INFO:tensorflow:loss = 40958.387, step = 374 (2.921 sec)\n",
      "INFO:tensorflow:global_step/sec: 35.7787\n",
      "INFO:tensorflow:loss = 51001.227, step = 474 (2.797 sec)\n",
      "INFO:tensorflow:global_step/sec: 36.4638\n",
      "INFO:tensorflow:loss = 61493.582, step = 574 (2.745 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 590 into taxi_trained/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 47573.914.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2019-01-08-04:10:33\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-590\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Finished evaluation at 2019-01-08-04:10:33\n",
      "INFO:tensorflow:Saving dict for global step 590: average_loss = 94.548805, global_step = 590, loss = 37134.043\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Regress: None\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict']\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Classify: None\n",
      "INFO:tensorflow:Signatures EXCLUDED from export because they cannot be be served via TensorFlow Serving APIs:\n",
      "INFO:tensorflow:'regression' : Regression input must be a single string Tensor; got {'dropofflat': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=float32>, 'pickuplon': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=float32>, 'passengers': <tf.Tensor 'Placeholder_4:0' shape=(?,) dtype=float32>, 'dropofflon': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'pickuplat': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>}\n",
      "INFO:tensorflow:'serving_default' : Regression input must be a single string Tensor; got {'dropofflat': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=float32>, 'pickuplon': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=float32>, 'passengers': <tf.Tensor 'Placeholder_4:0' shape=(?,) dtype=float32>, 'dropofflon': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'pickuplat': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>}\n",
      "WARNING:tensorflow:Export includes no default signature!\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-590\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: b\"taxi_trained/export/exporter/temp-b'1546920633'/saved_model.pb\"\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-590\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 591 into taxi_trained/model.ckpt.\n",
      "INFO:tensorflow:loss = 39345.22, step = 591\n",
      "INFO:tensorflow:global_step/sec: 35.3302\n",
      "INFO:tensorflow:loss = 38189.87, step = 691 (2.835 sec)\n",
      "INFO:tensorflow:global_step/sec: 36.0285\n",
      "INFO:tensorflow:loss = 74322.5, step = 791 (2.776 sec)\n",
      "INFO:tensorflow:global_step/sec: 36.0848\n",
      "INFO:tensorflow:loss = 54572.7, step = 891 (2.775 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 920 into taxi_trained/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 55533.773.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2019-01-08-04:10:44\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-920\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Finished evaluation at 2019-01-08-04:10:45\n",
      "INFO:tensorflow:Saving dict for global step 920: average_loss = 94.60244, global_step = 920, loss = 37155.11\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Regress: None\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict']\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Classify: None\n",
      "INFO:tensorflow:Signatures EXCLUDED from export because they cannot be be served via TensorFlow Serving APIs:\n",
      "INFO:tensorflow:'regression' : Regression input must be a single string Tensor; got {'dropofflat': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=float32>, 'pickuplon': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=float32>, 'passengers': <tf.Tensor 'Placeholder_4:0' shape=(?,) dtype=float32>, 'dropofflon': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'pickuplat': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:'serving_default' : Regression input must be a single string Tensor; got {'dropofflat': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=float32>, 'pickuplon': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=float32>, 'passengers': <tf.Tensor 'Placeholder_4:0' shape=(?,) dtype=float32>, 'dropofflon': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'pickuplat': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>}\n",
      "WARNING:tensorflow:Export includes no default signature!\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-920\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: b\"taxi_trained/export/exporter/temp-b'1546920645'/saved_model.pb\"\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-920\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 921 into taxi_trained/model.ckpt.\n",
      "INFO:tensorflow:loss = 39814.83, step = 921\n",
      "INFO:tensorflow:global_step/sec: 34.8254\n",
      "INFO:tensorflow:loss = 27247.46, step = 1021 (2.877 sec)\n",
      "INFO:tensorflow:global_step/sec: 36.2849\n",
      "INFO:tensorflow:loss = 57483.61, step = 1121 (2.755 sec)\n",
      "INFO:tensorflow:global_step/sec: 35.8293\n",
      "INFO:tensorflow:loss = 55565.492, step = 1221 (2.791 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 1250 into taxi_trained/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 46272.098.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2019-01-08-04:10:56\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-1250\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Finished evaluation at 2019-01-08-04:10:56\n",
      "INFO:tensorflow:Saving dict for global step 1250: average_loss = 94.5443, global_step = 1250, loss = 37132.273\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Regress: None\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict']\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Classify: None\n",
      "INFO:tensorflow:Signatures EXCLUDED from export because they cannot be be served via TensorFlow Serving APIs:\n",
      "INFO:tensorflow:'regression' : Regression input must be a single string Tensor; got {'dropofflat': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=float32>, 'pickuplon': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=float32>, 'passengers': <tf.Tensor 'Placeholder_4:0' shape=(?,) dtype=float32>, 'dropofflon': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'pickuplat': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>}\n",
      "INFO:tensorflow:'serving_default' : Regression input must be a single string Tensor; got {'dropofflat': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=float32>, 'pickuplon': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=float32>, 'passengers': <tf.Tensor 'Placeholder_4:0' shape=(?,) dtype=float32>, 'dropofflon': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'pickuplat': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>}\n",
      "WARNING:tensorflow:Export includes no default signature!\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-1250\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: b\"taxi_trained/export/exporter/temp-b'1546920656'/saved_model.pb\"\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-1250\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 1251 into taxi_trained/model.ckpt.\n",
      "INFO:tensorflow:loss = 35447.914, step = 1251\n",
      "INFO:tensorflow:global_step/sec: 35.2241\n",
      "INFO:tensorflow:loss = 34234.715, step = 1351 (2.844 sec)\n",
      "INFO:tensorflow:global_step/sec: 36.3339\n",
      "INFO:tensorflow:loss = 57541.92, step = 1451 (2.752 sec)\n",
      "INFO:tensorflow:global_step/sec: 36.1873\n",
      "INFO:tensorflow:loss = 40227.434, step = 1551 (2.766 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 1580 into taxi_trained/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 72595.734.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2019-01-08-04:11:07\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-1580\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Finished evaluation at 2019-01-08-04:11:08\n",
      "INFO:tensorflow:Saving dict for global step 1580: average_loss = 94.57043, global_step = 1580, loss = 37142.535\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Regress: None\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict']\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Classify: None\n",
      "INFO:tensorflow:Signatures EXCLUDED from export because they cannot be be served via TensorFlow Serving APIs:\n",
      "INFO:tensorflow:'regression' : Regression input must be a single string Tensor; got {'dropofflat': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=float32>, 'pickuplon': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=float32>, 'passengers': <tf.Tensor 'Placeholder_4:0' shape=(?,) dtype=float32>, 'dropofflon': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'pickuplat': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>}\n",
      "INFO:tensorflow:'serving_default' : Regression input must be a single string Tensor; got {'dropofflat': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=float32>, 'pickuplon': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=float32>, 'passengers': <tf.Tensor 'Placeholder_4:0' shape=(?,) dtype=float32>, 'dropofflon': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'pickuplat': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>}\n",
      "WARNING:tensorflow:Export includes no default signature!\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-1580\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: b\"taxi_trained/export/exporter/temp-b'1546920668'/saved_model.pb\"\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-1580\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 1581 into taxi_trained/model.ckpt.\n",
      "INFO:tensorflow:loss = 45066.44, step = 1581\n",
      "INFO:tensorflow:global_step/sec: 34.8678\n",
      "INFO:tensorflow:loss = 44026.734, step = 1681 (2.872 sec)\n",
      "INFO:tensorflow:global_step/sec: 36.4351\n",
      "INFO:tensorflow:loss = 44231.305, step = 1781 (2.745 sec)\n",
      "INFO:tensorflow:global_step/sec: 36.2409\n",
      "INFO:tensorflow:loss = 43892.566, step = 1881 (2.759 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 1910 into taxi_trained/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 54812.4.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2019-01-08-04:11:19\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-1910\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Finished evaluation at 2019-01-08-04:11:19\n",
      "INFO:tensorflow:Saving dict for global step 1910: average_loss = 94.70001, global_step = 1910, loss = 37193.43\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Regress: None\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict']\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Classify: None\n",
      "INFO:tensorflow:Signatures EXCLUDED from export because they cannot be be served via TensorFlow Serving APIs:\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:'regression' : Regression input must be a single string Tensor; got {'dropofflat': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=float32>, 'pickuplon': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=float32>, 'passengers': <tf.Tensor 'Placeholder_4:0' shape=(?,) dtype=float32>, 'dropofflon': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'pickuplat': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>}\n",
      "INFO:tensorflow:'serving_default' : Regression input must be a single string Tensor; got {'dropofflat': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=float32>, 'pickuplon': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=float32>, 'passengers': <tf.Tensor 'Placeholder_4:0' shape=(?,) dtype=float32>, 'dropofflon': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'pickuplat': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>}\n",
      "WARNING:tensorflow:Export includes no default signature!\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-1910\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: b\"taxi_trained/export/exporter/temp-b'1546920679'/saved_model.pb\"\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-1910\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 1911 into taxi_trained/model.ckpt.\n",
      "INFO:tensorflow:loss = 81386.625, step = 1911\n",
      "INFO:tensorflow:global_step/sec: 34.9335\n",
      "INFO:tensorflow:loss = 77801.09, step = 2011 (2.867 sec)\n",
      "INFO:tensorflow:global_step/sec: 35.2591\n",
      "INFO:tensorflow:loss = 49465.086, step = 2111 (2.836 sec)\n",
      "INFO:tensorflow:global_step/sec: 35.7902\n",
      "INFO:tensorflow:loss = 56917.89, step = 2211 (2.794 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 2226 into taxi_trained/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 61903.26.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2019-01-08-04:11:30\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-2226\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Finished evaluation at 2019-01-08-04:11:30\n",
      "INFO:tensorflow:Saving dict for global step 2226: average_loss = 94.47426, global_step = 2226, loss = 37104.766\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Regress: None\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict']\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Classify: None\n",
      "INFO:tensorflow:Signatures EXCLUDED from export because they cannot be be served via TensorFlow Serving APIs:\n",
      "INFO:tensorflow:'regression' : Regression input must be a single string Tensor; got {'dropofflat': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=float32>, 'pickuplon': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=float32>, 'passengers': <tf.Tensor 'Placeholder_4:0' shape=(?,) dtype=float32>, 'dropofflon': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'pickuplat': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>}\n",
      "INFO:tensorflow:'serving_default' : Regression input must be a single string Tensor; got {'dropofflat': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=float32>, 'pickuplon': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=float32>, 'passengers': <tf.Tensor 'Placeholder_4:0' shape=(?,) dtype=float32>, 'dropofflon': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'pickuplat': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>}\n",
      "WARNING:tensorflow:Export includes no default signature!\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-2226\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: b\"taxi_trained/export/exporter/temp-b'1546920690'/saved_model.pb\"\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-2226\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 2227 into taxi_trained/model.ckpt.\n",
      "INFO:tensorflow:loss = 39088.01, step = 2227\n",
      "INFO:tensorflow:global_step/sec: 34.6053\n",
      "INFO:tensorflow:loss = 41637.062, step = 2327 (2.894 sec)\n",
      "INFO:tensorflow:global_step/sec: 35.827\n",
      "INFO:tensorflow:loss = 42650.297, step = 2427 (2.792 sec)\n",
      "INFO:tensorflow:global_step/sec: 35.5079\n",
      "INFO:tensorflow:loss = 59747.953, step = 2527 (2.818 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 2556 into taxi_trained/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 75370.66.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2019-01-08-04:11:42\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-2556\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Finished evaluation at 2019-01-08-04:11:42\n",
      "INFO:tensorflow:Saving dict for global step 2556: average_loss = 94.45512, global_step = 2556, loss = 37097.246\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Regress: None\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict']\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Classify: None\n",
      "INFO:tensorflow:Signatures EXCLUDED from export because they cannot be be served via TensorFlow Serving APIs:\n",
      "INFO:tensorflow:'regression' : Regression input must be a single string Tensor; got {'dropofflat': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=float32>, 'pickuplon': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=float32>, 'passengers': <tf.Tensor 'Placeholder_4:0' shape=(?,) dtype=float32>, 'dropofflon': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'pickuplat': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>}\n",
      "INFO:tensorflow:'serving_default' : Regression input must be a single string Tensor; got {'dropofflat': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=float32>, 'pickuplon': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=float32>, 'passengers': <tf.Tensor 'Placeholder_4:0' shape=(?,) dtype=float32>, 'dropofflon': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'pickuplat': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>}\n",
      "WARNING:tensorflow:Export includes no default signature!\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-2556\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: b\"taxi_trained/export/exporter/temp-b'1546920702'/saved_model.pb\"\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-2556\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 2557 into taxi_trained/model.ckpt.\n",
      "INFO:tensorflow:loss = 38640.64, step = 2557\n",
      "INFO:tensorflow:global_step/sec: 33.7709\n",
      "INFO:tensorflow:loss = 47080.1, step = 2657 (2.969 sec)\n",
      "INFO:tensorflow:global_step/sec: 35.2419\n",
      "INFO:tensorflow:loss = 40148.402, step = 2757 (2.834 sec)\n",
      "INFO:tensorflow:global_step/sec: 35.2003\n",
      "INFO:tensorflow:loss = 44429.83, step = 2857 (2.843 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 2872 into taxi_trained/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 32967.934.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2019-01-08-04:11:53\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-2872\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Finished evaluation at 2019-01-08-04:11:53\n",
      "INFO:tensorflow:Saving dict for global step 2872: average_loss = 95.21404, global_step = 2872, loss = 37395.316\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Regress: None\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict']\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Classify: None\n",
      "INFO:tensorflow:Signatures EXCLUDED from export because they cannot be be served via TensorFlow Serving APIs:\n",
      "INFO:tensorflow:'regression' : Regression input must be a single string Tensor; got {'dropofflat': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=float32>, 'pickuplon': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=float32>, 'passengers': <tf.Tensor 'Placeholder_4:0' shape=(?,) dtype=float32>, 'dropofflon': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'pickuplat': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>}\n",
      "INFO:tensorflow:'serving_default' : Regression input must be a single string Tensor; got {'dropofflat': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=float32>, 'pickuplon': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=float32>, 'passengers': <tf.Tensor 'Placeholder_4:0' shape=(?,) dtype=float32>, 'dropofflon': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'pickuplat': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>}\n",
      "WARNING:tensorflow:Export includes no default signature!\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-2872\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: b\"taxi_trained/export/exporter/temp-b'1546920714'/saved_model.pb\"\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-2872\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 2873 into taxi_trained/model.ckpt.\n",
      "INFO:tensorflow:loss = 74828.484, step = 2873\n",
      "INFO:tensorflow:global_step/sec: 20.1246\n",
      "INFO:tensorflow:loss = 22836.594, step = 2973 (4.981 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 3034 into taxi_trained/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 61770.625.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2019-01-08-04:12:06\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-3034\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Finished evaluation at 2019-01-08-04:12:07\n",
      "INFO:tensorflow:Saving dict for global step 3034: average_loss = 94.84001, global_step = 3034, loss = 37248.414\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Regress: None\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict']\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Classify: None\n",
      "INFO:tensorflow:Signatures EXCLUDED from export because they cannot be be served via TensorFlow Serving APIs:\n",
      "INFO:tensorflow:'regression' : Regression input must be a single string Tensor; got {'dropofflat': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=float32>, 'pickuplon': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=float32>, 'passengers': <tf.Tensor 'Placeholder_4:0' shape=(?,) dtype=float32>, 'dropofflon': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'pickuplat': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>}\n",
      "INFO:tensorflow:'serving_default' : Regression input must be a single string Tensor; got {'dropofflat': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=float32>, 'pickuplon': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=float32>, 'passengers': <tf.Tensor 'Placeholder_4:0' shape=(?,) dtype=float32>, 'dropofflon': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'pickuplat': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>}\n",
      "WARNING:tensorflow:Export includes no default signature!\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-3034\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: b\"taxi_trained/export/exporter/temp-b'1546920728'/saved_model.pb\"\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-3034\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 3035 into taxi_trained/model.ckpt.\n",
      "INFO:tensorflow:loss = 36281.89, step = 3035\n",
      "INFO:tensorflow:Saving checkpoints for 3106 into taxi_trained/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 56211.465.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2019-01-08-04:12:23\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-3106\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Finished evaluation at 2019-01-08-04:12:24\n",
      "INFO:tensorflow:Saving dict for global step 3106: average_loss = 94.48807, global_step = 3106, loss = 37110.188\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Regress: None\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict']\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Classify: None\n",
      "INFO:tensorflow:Signatures EXCLUDED from export because they cannot be be served via TensorFlow Serving APIs:\n",
      "INFO:tensorflow:'regression' : Regression input must be a single string Tensor; got {'dropofflat': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=float32>, 'pickuplon': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=float32>, 'passengers': <tf.Tensor 'Placeholder_4:0' shape=(?,) dtype=float32>, 'dropofflon': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'pickuplat': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>}\n",
      "INFO:tensorflow:'serving_default' : Regression input must be a single string Tensor; got {'dropofflat': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=float32>, 'pickuplon': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=float32>, 'passengers': <tf.Tensor 'Placeholder_4:0' shape=(?,) dtype=float32>, 'dropofflon': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'pickuplat': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>}\n",
      "WARNING:tensorflow:Export includes no default signature!\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-3106\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: b\"taxi_trained/export/exporter/temp-b'1546920745'/saved_model.pb\"\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-3106\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 3107 into taxi_trained/model.ckpt.\n",
      "INFO:tensorflow:loss = 49790.785, step = 3107\n",
      "INFO:tensorflow:Saving checkpoints for 3164 into taxi_trained/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 37721.75.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2019-01-08-04:12:39\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-3164\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Finished evaluation at 2019-01-08-04:12:41\n",
      "INFO:tensorflow:Saving dict for global step 3164: average_loss = 94.538925, global_step = 3164, loss = 37130.164\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Regress: None\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict']\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Classify: None\n",
      "INFO:tensorflow:Signatures EXCLUDED from export because they cannot be be served via TensorFlow Serving APIs:\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:'regression' : Regression input must be a single string Tensor; got {'dropofflat': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=float32>, 'pickuplon': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=float32>, 'passengers': <tf.Tensor 'Placeholder_4:0' shape=(?,) dtype=float32>, 'dropofflon': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'pickuplat': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>}\n",
      "INFO:tensorflow:'serving_default' : Regression input must be a single string Tensor; got {'dropofflat': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=float32>, 'pickuplon': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=float32>, 'passengers': <tf.Tensor 'Placeholder_4:0' shape=(?,) dtype=float32>, 'dropofflon': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'pickuplat': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>}\n",
      "WARNING:tensorflow:Export includes no default signature!\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-3164\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: b\"taxi_trained/export/exporter/temp-b'1546920761'/saved_model.pb\"\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-3164\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 3165 into taxi_trained/model.ckpt.\n",
      "INFO:tensorflow:loss = 48004.164, step = 3165\n",
      "INFO:tensorflow:Saving checkpoints for 3226 into taxi_trained/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 53979.41.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2019-01-08-04:12:56\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-3226\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Finished evaluation at 2019-01-08-04:12:58\n",
      "INFO:tensorflow:Saving dict for global step 3226: average_loss = 94.558174, global_step = 3226, loss = 37137.723\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Regress: None\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict']\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Classify: None\n",
      "INFO:tensorflow:Signatures EXCLUDED from export because they cannot be be served via TensorFlow Serving APIs:\n",
      "INFO:tensorflow:'regression' : Regression input must be a single string Tensor; got {'dropofflat': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=float32>, 'pickuplon': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=float32>, 'passengers': <tf.Tensor 'Placeholder_4:0' shape=(?,) dtype=float32>, 'dropofflon': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'pickuplat': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>}\n",
      "INFO:tensorflow:'serving_default' : Regression input must be a single string Tensor; got {'dropofflat': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=float32>, 'pickuplon': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=float32>, 'passengers': <tf.Tensor 'Placeholder_4:0' shape=(?,) dtype=float32>, 'dropofflon': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'pickuplat': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>}\n",
      "WARNING:tensorflow:Export includes no default signature!\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-3226\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: b\"taxi_trained/export/exporter/temp-b'1546920778'/saved_model.pb\"\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-3226\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 3227 into taxi_trained/model.ckpt.\n",
      "INFO:tensorflow:loss = 47802.875, step = 3227\n",
      "INFO:tensorflow:global_step/sec: 34.8479\n",
      "INFO:tensorflow:loss = 45175.83, step = 3327 (2.874 sec)\n",
      "INFO:tensorflow:global_step/sec: 34.7773\n",
      "INFO:tensorflow:loss = 47914.375, step = 3427 (2.876 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 3484 into taxi_trained/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 55747.9.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2019-01-08-04:13:11\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-3484\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Finished evaluation at 2019-01-08-04:13:11\n",
      "INFO:tensorflow:Saving dict for global step 3484: average_loss = 94.85251, global_step = 3484, loss = 37253.324\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Regress: None\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict']\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Classify: None\n",
      "INFO:tensorflow:Signatures EXCLUDED from export because they cannot be be served via TensorFlow Serving APIs:\n",
      "INFO:tensorflow:'regression' : Regression input must be a single string Tensor; got {'dropofflat': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=float32>, 'pickuplon': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=float32>, 'passengers': <tf.Tensor 'Placeholder_4:0' shape=(?,) dtype=float32>, 'dropofflon': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'pickuplat': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>}\n",
      "INFO:tensorflow:'serving_default' : Regression input must be a single string Tensor; got {'dropofflat': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=float32>, 'pickuplon': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=float32>, 'passengers': <tf.Tensor 'Placeholder_4:0' shape=(?,) dtype=float32>, 'dropofflon': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'pickuplat': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>}\n",
      "WARNING:tensorflow:Export includes no default signature!\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-3484\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: b\"taxi_trained/export/exporter/temp-b'1546920792'/saved_model.pb\"\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-3484\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 3485 into taxi_trained/model.ckpt.\n",
      "INFO:tensorflow:loss = 68401.77, step = 3485\n",
      "INFO:tensorflow:global_step/sec: 35.0625\n",
      "INFO:tensorflow:loss = 45127.324, step = 3585 (2.857 sec)\n",
      "INFO:tensorflow:global_step/sec: 35.8158\n",
      "INFO:tensorflow:loss = 50135.82, step = 3685 (2.807 sec)\n",
      "INFO:tensorflow:global_step/sec: 35.9312\n",
      "INFO:tensorflow:loss = 55876.44, step = 3785 (2.773 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 3814 into taxi_trained/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 48553.266.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2019-01-08-04:13:23\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-3814\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Finished evaluation at 2019-01-08-04:13:23\n",
      "INFO:tensorflow:Saving dict for global step 3814: average_loss = 94.48899, global_step = 3814, loss = 37110.55\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Regress: None\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict']\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Classify: None\n",
      "INFO:tensorflow:Signatures EXCLUDED from export because they cannot be be served via TensorFlow Serving APIs:\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:'regression' : Regression input must be a single string Tensor; got {'dropofflat': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=float32>, 'pickuplon': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=float32>, 'passengers': <tf.Tensor 'Placeholder_4:0' shape=(?,) dtype=float32>, 'dropofflon': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'pickuplat': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>}\n",
      "INFO:tensorflow:'serving_default' : Regression input must be a single string Tensor; got {'dropofflat': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=float32>, 'pickuplon': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=float32>, 'passengers': <tf.Tensor 'Placeholder_4:0' shape=(?,) dtype=float32>, 'dropofflon': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'pickuplat': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>}\n",
      "WARNING:tensorflow:Export includes no default signature!\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-3814\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: b\"taxi_trained/export/exporter/temp-b'1546920803'/saved_model.pb\"\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-3814\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 3815 into taxi_trained/model.ckpt.\n",
      "INFO:tensorflow:loss = 41634.03, step = 3815\n",
      "INFO:tensorflow:global_step/sec: 34.2813\n",
      "INFO:tensorflow:loss = 48131.97, step = 3915 (2.922 sec)\n",
      "INFO:tensorflow:global_step/sec: 36.0057\n",
      "INFO:tensorflow:loss = 49404.1, step = 4015 (2.778 sec)\n",
      "INFO:tensorflow:global_step/sec: 36.2463\n",
      "INFO:tensorflow:loss = 74862.84, step = 4115 (2.759 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 4144 into taxi_trained/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 36406.016.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2019-01-08-04:13:34\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-4144\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Finished evaluation at 2019-01-08-04:13:35\n",
      "INFO:tensorflow:Saving dict for global step 4144: average_loss = 94.644806, global_step = 4144, loss = 37171.746\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Regress: None\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict']\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Classify: None\n",
      "INFO:tensorflow:Signatures EXCLUDED from export because they cannot be be served via TensorFlow Serving APIs:\n",
      "INFO:tensorflow:'regression' : Regression input must be a single string Tensor; got {'dropofflat': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=float32>, 'pickuplon': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=float32>, 'passengers': <tf.Tensor 'Placeholder_4:0' shape=(?,) dtype=float32>, 'dropofflon': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'pickuplat': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>}\n",
      "INFO:tensorflow:'serving_default' : Regression input must be a single string Tensor; got {'dropofflat': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=float32>, 'pickuplon': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=float32>, 'passengers': <tf.Tensor 'Placeholder_4:0' shape=(?,) dtype=float32>, 'dropofflon': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'pickuplat': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>}\n",
      "WARNING:tensorflow:Export includes no default signature!\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-4144\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: b\"taxi_trained/export/exporter/temp-b'1546920815'/saved_model.pb\"\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-4144\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 4145 into taxi_trained/model.ckpt.\n",
      "INFO:tensorflow:loss = 47214.43, step = 4145\n",
      "INFO:tensorflow:global_step/sec: 34.8804\n",
      "INFO:tensorflow:loss = 55455.812, step = 4245 (2.874 sec)\n",
      "INFO:tensorflow:global_step/sec: 35.8477\n",
      "INFO:tensorflow:loss = 46781.277, step = 4345 (2.793 sec)\n",
      "INFO:tensorflow:global_step/sec: 35.6528\n",
      "INFO:tensorflow:loss = 34688.54, step = 4445 (2.801 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 4474 into taxi_trained/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 41530.484.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2019-01-08-04:13:46\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-4474\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Finished evaluation at 2019-01-08-04:13:46\n",
      "INFO:tensorflow:Saving dict for global step 4474: average_loss = 94.62814, global_step = 4474, loss = 37165.203\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Regress: None\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict']\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Classify: None\n",
      "INFO:tensorflow:Signatures EXCLUDED from export because they cannot be be served via TensorFlow Serving APIs:\n",
      "INFO:tensorflow:'regression' : Regression input must be a single string Tensor; got {'dropofflat': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=float32>, 'pickuplon': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=float32>, 'passengers': <tf.Tensor 'Placeholder_4:0' shape=(?,) dtype=float32>, 'dropofflon': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'pickuplat': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>}\n",
      "INFO:tensorflow:'serving_default' : Regression input must be a single string Tensor; got {'dropofflat': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=float32>, 'pickuplon': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=float32>, 'passengers': <tf.Tensor 'Placeholder_4:0' shape=(?,) dtype=float32>, 'dropofflon': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'pickuplat': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>}\n",
      "WARNING:tensorflow:Export includes no default signature!\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-4474\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: b\"taxi_trained/export/exporter/temp-b'1546920826'/saved_model.pb\"\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-4474\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 4475 into taxi_trained/model.ckpt.\n",
      "INFO:tensorflow:loss = 35053.906, step = 4475\n",
      "INFO:tensorflow:global_step/sec: 34.7802\n",
      "INFO:tensorflow:loss = 50300.176, step = 4575 (2.882 sec)\n",
      "INFO:tensorflow:global_step/sec: 35.8527\n",
      "INFO:tensorflow:loss = 53345.86, step = 4675 (2.787 sec)\n",
      "INFO:tensorflow:global_step/sec: 36.0268\n",
      "INFO:tensorflow:loss = 57187.496, step = 4775 (2.776 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 4791 into taxi_trained/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 45450.9.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2019-01-08-04:13:57\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-4791\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Finished evaluation at 2019-01-08-04:13:57\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Saving dict for global step 4791: average_loss = 94.45192, global_step = 4791, loss = 37095.992\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Regress: None\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict']\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Classify: None\n",
      "INFO:tensorflow:Signatures EXCLUDED from export because they cannot be be served via TensorFlow Serving APIs:\n",
      "INFO:tensorflow:'regression' : Regression input must be a single string Tensor; got {'dropofflat': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=float32>, 'pickuplon': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=float32>, 'passengers': <tf.Tensor 'Placeholder_4:0' shape=(?,) dtype=float32>, 'dropofflon': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'pickuplat': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>}\n",
      "INFO:tensorflow:'serving_default' : Regression input must be a single string Tensor; got {'dropofflat': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=float32>, 'pickuplon': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=float32>, 'passengers': <tf.Tensor 'Placeholder_4:0' shape=(?,) dtype=float32>, 'dropofflon': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'pickuplat': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>}\n",
      "WARNING:tensorflow:Export includes no default signature!\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-4791\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: b\"taxi_trained/export/exporter/temp-b'1546920838'/saved_model.pb\"\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-4791\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 4792 into taxi_trained/model.ckpt.\n",
      "INFO:tensorflow:loss = 40297.68, step = 4792\n",
      "INFO:tensorflow:global_step/sec: 34.661\n",
      "INFO:tensorflow:loss = 45967.65, step = 4892 (2.890 sec)\n",
      "INFO:tensorflow:global_step/sec: 36.1129\n",
      "INFO:tensorflow:loss = 48047.04, step = 4992 (2.772 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 5000 into taxi_trained/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 43516.594.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2019-01-08-04:14:05\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-5000\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Finished evaluation at 2019-01-08-04:14:06\n",
      "INFO:tensorflow:Saving dict for global step 5000: average_loss = 94.46468, global_step = 5000, loss = 37101.004\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Regress: None\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Predict: ['predict']\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Classify: None\n",
      "INFO:tensorflow:Signatures EXCLUDED from export because they cannot be be served via TensorFlow Serving APIs:\n",
      "INFO:tensorflow:'regression' : Regression input must be a single string Tensor; got {'dropofflat': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=float32>, 'pickuplon': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=float32>, 'passengers': <tf.Tensor 'Placeholder_4:0' shape=(?,) dtype=float32>, 'dropofflon': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'pickuplat': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>}\n",
      "INFO:tensorflow:'serving_default' : Regression input must be a single string Tensor; got {'dropofflat': <tf.Tensor 'Placeholder_2:0' shape=(?,) dtype=float32>, 'pickuplon': <tf.Tensor 'Placeholder:0' shape=(?,) dtype=float32>, 'passengers': <tf.Tensor 'Placeholder_4:0' shape=(?,) dtype=float32>, 'dropofflon': <tf.Tensor 'Placeholder_3:0' shape=(?,) dtype=float32>, 'pickuplat': <tf.Tensor 'Placeholder_1:0' shape=(?,) dtype=float32>}\n",
      "WARNING:tensorflow:Export includes no default signature!\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-5000\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: b\"taxi_trained/export/exporter/temp-b'1546920846'/saved_model.pb\"\n"
     ]
    }
   ],
   "source": [
    "# Run training    \n",
    "OUTDIR = 'taxi_trained'\n",
    "shutil.rmtree(OUTDIR, ignore_errors = True) # start fresh each time\n",
    "train_and_evaluate(OUTDIR, num_train_steps = 5000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2> Monitoring with TensorBoard </h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<p>TensorBoard was started successfully with pid 6986. Click <a href=\"/_proxy/52747/\" target=\"_blank\">here</a> to access it.</p>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>logdir</th>\n",
       "      <th>pid</th>\n",
       "      <th>port</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>./taxi_trained</td>\n",
       "      <td>6986</td>\n",
       "      <td>52747</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           logdir   pid   port\n",
       "0  ./taxi_trained  6986  52747"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from google.datalab.ml import TensorBoard\n",
    "TensorBoard().start('./taxi_trained')\n",
    "TensorBoard().list()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Stopped TensorBoard with pid 6986\n"
     ]
    }
   ],
   "source": [
    "# to stop TensorBoard\n",
    "for pid in TensorBoard.list()['pid']:\n",
    "    TensorBoard().stop(pid)\n",
    "    print('Stopped TensorBoard with pid {}'.format(pid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright 2017 Google Inc. Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}