tensrflow-notes-by-Moustafa-Shomer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Loading/Preprocessing Image Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "## for image processing:\n",
    "* 1- your data directory should be as follows:\n",
    "    - there is train folder for training data , as well as testing folder for the test\n",
    "    - inside every branch of these, put as many folders as your classes, with each class image in it's folder\n",
    "    - then at the function's location point at the main brach of either train folder or test folder\n",
    "    - the function will then sort every image in the classes folder and give it a proper label\n",
    "    - it will also name the main classes with their proper labels\n",
    "    - as it will create label_1 --> label_n, and inside each label it's own data (label_1 0 ->m , label_n 0 -> m)\n",
    "----------\n",
    "* 2- importing the used function `ImageDataGenerator` :\n",
    "\n",
    "            from tensorflow.keras.preprocessing.image import ImageDataGenerator\n",
    "--------------          \n",
    "* 3- create the train_gen/vali_gen object  :\n",
    "    - assign the preprocessing you need inside the image generator, for exampel scaling\n",
    "    \n",
    "             train_gen= ImageDataGenerator( rescale=1.0/255 , rotation_range=40, width_shift_range=0.3,\n",
    "                                  height_shift_range=0.3, shear_range=0.2, zoom_range=0.2,\n",
    "                                  fill_mode='nearest', horizontal_flip=True , validation_split=0.2 )\n",
    "    \n",
    "      *  `featurewise_center`::\tBoolean. Set input mean to 0 over the dataset, feature-wise.\n",
    "      *  `samplewise_center`::\tBoolean. Set each sample mean to 0.\n",
    "      *  `featurewise_std_normalization`::\tBoolean. Divide inputs by std of the dataset, feature-wise.\n",
    "      *  `samplewise_std_normalization`::\tBoolean. Divide each input by its std.\n",
    "      *  `zca_epsilon`::\tepsilon for ZCA whitening. Default is 1e-6.\n",
    "      *  `zca_whitening`::\tBoolean. Apply ZCA whitening.\n",
    "      *  `rotation_range`::\tInt. Degree range for random rotations.\n",
    "      *  `width_shift_range`::\tFloat, 1-D array-like or int\n",
    "      \n",
    "          *  float: fraction of total width, if < 1, or pixels if >= 1.\n",
    "          *  1-D array-like: random elements from the array.\n",
    "          *  int: integer number of pixels from interval (-width_shift_range, +width_shift_range)\n",
    "          *  With width_shift_range=2 possible values are integers [-1, 0, +1], same as with width_shift_range=[-1, 0, +1], while with width_shift_range=1.0 possible values are floats in the interval [-1.0, +1.0).\n",
    "          \n",
    "      *  `height_shift_range`::\tFloat, 1-D array-like or int _\n",
    "      \n",
    "          *  float: fraction of total height, if < 1, or pixels if >= 1.\n",
    "          *  1-D array-like: random elements from the array.\n",
    "          *  int: integer number of pixels from interval (-height_shift_range, +height_shift_range)\n",
    "          *  With height_shift_range=2 possible values are integers [-1, 0, +1], same as with height_shift_range=[-1, 0, +1], while with height_shift_range=1.0 possible values are floats in the interval [-1.0, +1.0).\n",
    "          \n",
    "      *  `brightness_range`::\tTuple or list of two floats. Range for picking a brightness shift value from.\n",
    "      *  `shear_range`::\tFloat. Shear Intensity (Shear angle in counter-clockwise direction in degrees)\n",
    "      *  `zoom_range`::\tFloat or [lower, upper]. Range for random zoom. If a float, [lower, upper] = [1-zoom_range, 1+zoom_range].\n",
    "      *  `channel_shift_range`::\tFloat. Range for random channel shifts.\n",
    "      *  `fill_mode`::\tOne of {\"constant\", \"nearest\", \"reflect\" or \"wrap\"}. Default is 'nearest'. Points outside the boundaries of the input are filled according to the given mode:\n",
    "      \n",
    "          -  'constant': kkkkkkkk|abcd|kkkkkkkk (cval=k)\n",
    "          -  'nearest': aaaaaaaa|abcd|dddddddd\n",
    "          -  'reflect': abcddcba|abcd|dcbaabcd\n",
    "          -  'wrap': abcdabcd|abcd|abcdabcd\n",
    "                          \n",
    "      *  `cval`::\tFloat or Int. Value used for points outside the boundaries when fill_mode = \"constant\".\n",
    "      *  `horizontal_flip`::\tBoolean. Randomly flip inputs horizontally.\n",
    "      *  `vertical_flip`::\tBoolean. Randomly flip inputs vertically.\n",
    "      *  `rescale`::\trescaling factor. Defaults to None. If None or 0, no rescaling is applied, otherwise we multiply the data by the value provided (after applying all other transformations).\n",
    "      *  `preprocessing_function`::\tfunction that will be applied on each input. \n",
    "          * The function will run after the image is resized and augmented. The function should take one argument: one image (Numpy tensor with rank 3), and should output a Numpy tensor with the same shape.\n",
    "      *  `data_format`::\tImage data format, either \"channels_first\" or \"channels_last\".\n",
    "          * \"channels_last\" mode means that the images should have shape (samples, height, width, channels),\n",
    "          * \"channels_first\" mode means that the images should have shape (samples, channels, height, width). \n",
    "          * It defaults to the image_data_format value found in your Keras config file at ~/.keras/keras.json. If you never set it, then it will be \"channels_last\".\n",
    "      *  `validation_split`::\tFloat. Fraction of images reserved for validation (strictly between 0 and 1).\n",
    "      *  `dtype`::\tDtype to use for the generated arrays.\n",
    "---------------          \n",
    "* 4- create the variable to load the data into:\n",
    "    - assign the parameters you need inside\n",
    "    - `target_size` resizes the data inside the variable, NOT THE REAL DATA\n",
    "    - `batch_size` batches for uploading and tweaking the images\n",
    "    - 'class_mode` depends on the classes type, is the data for binary classification or others\n",
    "    - use `flow_from_directory` to Take the path to a directory & generate batches of augmented data.\n",
    "            train_set= train_gen.flow_from_directory(\n",
    "                                                        directory, target_size=(256, 256), color_mode='rgb', classes=None,\n",
    "                                                        class_mode='categorical', batch_size=32, shuffle=True, seed=None,\n",
    "                                                        save_to_dir=None, save_prefix='', save_format='png', follow_links=False,\n",
    "                                                        subset=None, interpolation='nearest' )\n",
    "\n",
    "    \n",
    "* 5- do the same steps for the test_set or dev_set if there is a folder for it          \n",
    "--------------\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Processing and Building Sequence models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## for sequences processing:\n",
    "\n",
    "* 1- load the data as a tensorflow dataset object or as a NUMPY series DO NOT FORGET TO CONVERT THE VARIABLES\n",
    "\n",
    "        dataset = tf.data.Dataset\n",
    "----------------\n",
    "* 2- for plotting the data:\n",
    "\n",
    "        def plot_series(time, series, format=\"-\", start=0, end=None):\n",
    "            plt.plot(time[start:end], series[start:end], format)\n",
    "            plt.xlabel(\"Time\")\n",
    "            plt.ylabel(\"Value\")\n",
    "            plt.grid(True)\n",
    "            \n",
    "        plt.figure(figsize=(10, 6))\n",
    "        plot_series(time_valid, x_valid)\n",
    "        plot_series(time_valid, results)\n",
    "-----\n",
    "### notice the coming codes are atributes of the dataset object AKA affecting the variable\n",
    "\n",
    "* 3- create the window you will use to iterate over the data:\n",
    "\n",
    "            dataset = dataset.window(5, shift=1, drop_remainder=True)\n",
    "    - where 5 is the number of elements in one window\n",
    "    - shift is the value with which the window will move each time - consider it as a stride -\n",
    "    - drop_remainder is used to make all the windows of size 5, so when reaching last 4 elements then 3 then 2 untill the end, it will get rid of those and only keep windows of 5 values\n",
    "    - flatten the window to map it , with the same number of elements in window, ' 5 ' \n",
    "\n",
    "            dataset = dataset.flat_map(lambda window: window.batch(5))\n",
    "    \n",
    "    - mapping/slicing each window inside the dataset variable into x , y , where x is input value and y is the wanted value    \n",
    "\n",
    "            dataset = dataset.map(lambda window: (window[:-1], window[-1:]))\n",
    "\n",
    "    - shuffling the windows:\n",
    "\n",
    "            dataset = dataset.shuffle(buffer_size=10)\n",
    "    - the buffer_size is the total number of elements we have ' in this case from 0 -> 9 = 10 elements '\n",
    "\n",
    "    - batching the data as 2 inputs/outputs at a time\n",
    "\n",
    "            dataset = dataset.batch(2).prefetch(1)\n",
    "\n",
    "    - and now you can access the x , y values from the dataset variable as:\n",
    "        \n",
    "            for x , y in dataset:\n",
    "     - we use a for loop because we are batching the data , so it will come out as batches of 2,  x = [ i1 , i2 ] , y = [ o1 , o2 ]\n",
    "---------\n",
    "\n",
    "* you can simply use this function to return a windowed dataset of a series data:\n",
    "\n",
    "        split_time = 1000\n",
    "        x_train = series_data[:split_time] # the data (training) before the split_time\n",
    "        x_valid = series_data[split_time:] # the data (test/valid) after the split_time\n",
    "        time_train = time[:split_time] # the time interval for the train\n",
    "        time_valid = time[split_time:] # the time interval for the test \n",
    "\n",
    "        def windowed_dataset(series, window_size, batch_size, shuffle_buffer):\n",
    "            series = tf.expand_dims(series, axis=-1)\n",
    "            ds = tf.data.Dataset.from_tensor_slices(series)\n",
    "            ds = ds.window(window_size + 1, shift=1, drop_remainder=True)\n",
    "            ds = ds.flat_map(lambda w: w.batch(window_size + 1))\n",
    "            ds = ds.shuffle(shuffle_buffer)\n",
    "            ds = ds.map(lambda w: (w[:-1], w[1:]))\n",
    "            return ds.batch(batch_size).prefetch(1)\n",
    "          \n",
    "         dataset = windowed_dataset(x_train, window_size, batch_size, shuffle_buffer_size) \n",
    "         # NOTICE that only the x_train goes into this function no further processing needed for any of the other params\n",
    "         \n",
    "  - notice the windows_size +1 is for easy use inside the layers, as the input layer will be of shape window_size \n",
    "\n",
    "\n",
    "        \n",
    "---------\n",
    "* 4- pass the data into the network of choice, make sure the input layer shape is the same as window_size\n",
    "    \n",
    "        layer.Dense(1 , input_shape=[window_size] )\n",
    "    - if you are using rnns or conv1d then you will need to reshape the input, and u can do that easily by using Lambda layer:\n",
    "    \n",
    "        `tf.keras.layers.Lambda( lambda x: tf.expand_dims(x, axis=-1), input_shape=[None])`\n",
    "        - you can also add the lamda function inside the preprocessing wondow function at the first line, to expand the last dim of the series data/x_train\n",
    "    - The last layer must be Dense(1) with no activation\n",
    "-------\n",
    "\n",
    "* 5- Use learning rate scheduler to choose the right lr\n",
    "        \n",
    "            lr_schedule = tf.keras.callbacks.LearningRateScheduler( lambda epoch: lr * 10**(epoch / 20) ) # where lr is the lr in the optimizer\n",
    "        \n",
    "     - plot the lr vs loss to spot the best lr :\n",
    "     \n",
    "            lrs = lr * (10 ** (np.arange(100) / 20)) # you can change the range\n",
    "            plt.semilogx(history.history[\"lr\"], history.history[\"loss\"])\n",
    "            plt.axis([ lr , 1e-1 , 0 , 150 ]) # you can change the values of the first/sec for the start and end points , last axis for hight\n",
    "-----\n",
    "\n",
    "* 6- compile and fit the network as usual, make sure the loss is a linear one, 'mse' or 'rmse' or 'mae' or for high outliers use 'huber'\n",
    "    - if you are training on conv or rnns make sure to multiply the output by factor of 100+ to enhance the performance\n",
    "        `tf.keras.layers.Lambda( lambda x: x*100.0 )`\n",
    "        \n",
    "-------------\n",
    "\n",
    "* 7- forecast the predictions:\n",
    "\n",
    "        forecast=[]\n",
    "        for time in range( len( series_data ) - window_size ): # iterating for the length of the whole data minus the windows_size as it is our step_size\n",
    "            # appending the predictions of the slices of the series_data from Tn to Tn+step_size and organizing it's shape for the neural network\n",
    "            # notice that the slices are on the whole series_data not just the test one\n",
    "            forecast.append( model.predict( series_data[ time : time+window_size ] [np.newaxis] ) )\n",
    "        forecast = forecast[ split_time - window_size : ] # where the forecast is the prediction of the whole data series\n",
    "        #showing the output of the test data only, the unseen data output, from the split_time-window_size until the end of the data\n",
    "        results = np.array(forecast)[:, 0, 0] # putting the forecast data in shape for plotting\n",
    "        \n",
    "        #plot the results\n",
    "        plt.figure(figsize=(10, 6))\n",
    "        plot_series(time_valid, x_valid)\n",
    "        plot_series(time_valid, results)\n",
    "-------------------\n",
    "\n",
    "### To read the next row in a csv file 'in case the firstrow had titles', you use `next( )` and put the reader in it, then you can iterate through that reader variable without having the pain of having the first row useless as:\n",
    "        with open('Sunspots.csv') as csvfile :\n",
    "            ds=csv.reader(csvfile, delimiter=',')\n",
    "            next(ds)\n",
    "            for i in ds: #second row\n",
    "            \n",
    "-------------------\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Building Models in `tf.keras.models`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "* 1: first you have to make sure it's tf.keras because it difers\n",
    "--------------------------\n",
    "### make sure to clear out variables before rerunning the model again using `tf.keras.backend.clear_session()` \n",
    "------------\n",
    "\n",
    "* 2: assign Sequential to a variable 'model'\n",
    "--------------------------\n",
    "* 3: you can add layers by using `model.add( )` which adds 1 layer at a time after initializing the model\n",
    "\n",
    "    - but we will build it for easier reading in a list inside Sequential\n",
    "--------------------------\n",
    "* 4: inside Sequential we add the layers we want in a list:\n",
    "\n",
    "    - take a good care that every layer is from the same library, they are all from tf.keras not keras, you can `import tf.keras as keras` but keep it in mind that they are not the same in calling or initializing variables.\n",
    "    \n",
    "            model = tf.keras.models.Sequential([\n",
    "                                                tf.keras.layers.Flatten(),\n",
    "                                                tf. keras.layers.Dense(512, tf.nn.relu),\n",
    "                                                tf.keras.layers.Dense(128 , tf.nn.tanh),\n",
    "                                                tf.keras.layers.Dense(10 , tf.nn.softmax)\n",
    "                                                ])\n",
    "                                        \n",
    "\n",
    "--------------------------\n",
    "* 5: after initializing the model layers we compile the model it self:\n",
    "\n",
    "    - the optimizer,loss,matrics all can be defined outside of the compile function to tweak their params, and gets passed in as variables\n",
    "\n",
    "            model.compile(\n",
    "                            optimizer='rmsprop', loss=None, metrics=None, loss_weights=None,\n",
    "                            sample_weight_mode=None, weighted_metrics=None, **kwargs )\n",
    "\n",
    "--------------------------\n",
    "* 6: fitting the model by passing in different params:\n",
    "\n",
    "    - the training set, training labels, epochs, batch_size, etc..\n",
    "    \n",
    "            history = model.fit(\n",
    "                            x=None, y=None, batch_size=None, epochs=1, verbose=1, callbacks=None,\n",
    "                            validation_split=0.0, validation_data=None, shuffle=True, class_weight=None,\n",
    "                            sample_weight=None, initial_epoch=0, steps_per_epoch=None,\n",
    "                            validation_steps=None, validation_batch_size=None, validation_freq=1,\n",
    "                            max_queue_size=10, workers=1, use_multiprocessing=False )\n",
    "--------------------------                           \n",
    "* 7: evaluating the model:\n",
    "    - using evaluate returns the loss value & metrics values for the model in test mode  \n",
    "    \n",
    "            model.evaluate(\n",
    "                     x=None, y=None, batch_size=None, verbose=1, sample_weight=None, steps=None,\n",
    "                     callbacks=None, max_queue_size=10, workers=1, use_multiprocessing=False,\n",
    "                     return_dict=False )\n",
    "--------------------------\n",
    "* 8: to retrieve the output of only one layer:\n",
    "    - Retrieves a layer based on either its name (unique) or index.\n",
    "    \n",
    "            model.get_layer( name=None, index=None )\n",
    "\n",
    "--------------------------       \n",
    "* 9: to make single prediction:\n",
    "    - Generates output predictions for the input samples.\n",
    "    \n",
    "            model.predict(\n",
    "                x, batch_size=None, verbose=0, steps=None, callbacks=None, max_queue_size=10,\n",
    "                workers=1, use_multiprocessing=False )\n",
    "--------------------------\n",
    "* 10: to predict a batch of input:\n",
    "\n",
    "           model.predict_on_batch( x )\n",
    "--------------------------\n",
    "* 11: to pritn a summary of the model archeticture:\n",
    "\n",
    "            model.summary(\n",
    "                line_length=None, positions=None, print_fn=None )\n",
    "\n",
    "--------------------------\n",
    "* 12: to save the model:\n",
    "    - Saves the model to Tensorflow SavedModel or a single HDF5 file.\n",
    "\n",
    "           model. save(\n",
    "                filepath, overwrite=True, include_optimizer=True, save_format=None,\n",
    "                signatures=None, options=None )\n",
    "        - The savefile includes:\n",
    "                1 The model architecture, allowing to re-instantiate the model.\n",
    "                2 The model weights.\n",
    "                3 The state of the optimizer, allowing to resume training exactly where you left off.\n",
    "--------------------------\n",
    "* 13: to load the whole model:\n",
    "    - Saved models can be reinstantiated by load_model is a compiled model ready to be used (unless the saved model was never compiled in the first place).\n",
    "\n",
    "    - Models built with the Sequential and Functional API can be saved to both the HDF5 and SavedModel formats. Subclassed models can only be saved with the SavedModel format.\n",
    "    \n",
    "            model_var = keras.models.load_model( )\n",
    "--------------------------\n",
    "* 14: to Save weights of the model layers:\n",
    "    - Saves all layer weights.\n",
    "      Either saves in HDF5 or in TensorFlow format based on the save_format argument.\n",
    "\n",
    "            model.save_weights( filepath, overwrite=True, save_format=None )\n",
    "         - When saving in HDF5 format, the weight file has:\n",
    "\n",
    "            1 layer_names (attribute), a list of strings (ordered names of model layers).\n",
    "            2 For every layer, a group named layer.name\n",
    "            3 For every such layer group, a group attribute weight_names, a list of strings (ordered names of weights tensor of the layer).\n",
    "            4 For every weight in the layer, a dataset storing the weight value, named after the weight tensor.\n",
    "--------------------------            \n",
    "* 15: to Load weights into the model layers:\n",
    "    - Loads all layer weights, either from a TensorFlow or an HDF5 weight file.\n",
    "    \n",
    "            model.load_weights( filepath, by_name=False, skip_mismatch=False )\n",
    "            \n",
    "        - If by_name is False weights are loaded based on the network's topology. This means the architecture should be the same as when the weights were saved. Note that layers that don't have weights are not taken into account in the topological ordering, so adding or removing layers is fine as long as they don't have weights.\n",
    "\n",
    "        - If by_name is True, weights are loaded into layers only if they share the same name. This is useful for fine-tuning or transfer-learning models where some of the layers have changed.\n",
    "--------------------------\n",
    "* 16: to save and load in json format:\n",
    "    - Returns a JSON string containing the network configuration.\n",
    "    \n",
    "            model.to_json( **kwargs )\n",
    "\n",
    "    - To load a network from a JSON save file, use \n",
    "    \n",
    "            model_var= keras.models.model_from_json(json_string, custom_objects={})\n",
    "--------------------------            \n",
    "* 17: to save and load in yml format:\n",
    "    - Returns a yaml string containing the network configuration\n",
    "    \n",
    "            to_yaml( **kwargs )\n",
    "\n",
    "    - To load a network from a yaml save file, use \n",
    "    \n",
    "            keras.models.model_from_yaml(yaml_string, custom_objects={})\n",
    "\n",
    "        - custom_objects should be a dictionary mapping the names of custom losses / layers / etc to the corresponding functions / classes.\n",
    "--------------------------     \n",
    "* 18: you can use the model variable you assigned the model to ' history' to retrieve more infos about the model:\n",
    "    - like using `history.history['acc']` , `history.history['val_acc']`, `history.history['loss']` , etc.\n",
    "--------------------\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Layers in `tf.keras.layers`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "* `Conv2D()`\n",
    "    - don't forget to reshape the images from 3D(m,w,h) to 4D (m,w,h,c) c for channel , by using np.expand_dims\n",
    "    - and dont forget to set the `input_shape(w,h,c)` as a parameter of the conv2d layer\n",
    "    - the parameter `padding='_'` if the value is 'same' then the output from the conv will be of same size as the input\n",
    "    - if `padding='valid'` it means the image will shrunk by the formula ( (x-f)/s )+1 where x is the size of image, f for kernel , s for stride\n",
    "---------------------------------------------------\n",
    "\n",
    "* `MaxPool2D()` \n",
    "\n",
    "    - dont forget that it shrinks the size depending on the stride, same as the formula for cnv2d\n",
    "---------------------------------------------------\n",
    "* `Flatten()` \n",
    "\n",
    "    - Flattens the input. Does not affect the batch size.\n",
    "-----------------------------------------------------\n",
    "* `Dense()`\n",
    "\n",
    "    - acts as a fully connected Nuraon layer\n",
    "---------------------------------------------\n",
    "* `Dropout()`\n",
    "    - The Dropout layer randomly sets input units to 0 with a frequency of rate at each step during training time, which helps prevent overfitting\n",
    "--------------------\n",
    "* `BatchNormalization()`\n",
    "    - Normalize the activations of the previous layer at each batch, i.e. applies a transformation that maintains the mean activation close to 0 and the activation standard deviation close to 1.\n",
    "----------------------\n",
    "* `Embedding()`\n",
    "    - Turns positive integers (indexes) into dense vectors of fixed size.\n",
    "-------------------\n",
    "* `RNN()`\n",
    "    - Base class for recurrent layers.\n",
    "-------------\n",
    "* `LSTM()`\n",
    "    - Long Short-Term Memory layer - Hochreiter 1997.\n",
    "--------------\n",
    "* `GRU()`\n",
    "    - Gated Recurrent Unit - Cho et al. 2014.\n",
    "-------------\n",
    "* `Bidirectional( )`\n",
    "    - Bidirectional wrapper for RNNs, you put inside it the layers you want to be duplicated in reverse direction\n",
    "------------------\n",
    "* `Lambda()`   \n",
    "\n",
    "    - used to perform the function at the first layer of the model, like changing the shape of the input etc. where the variable in the function referes to the input to that layer\n",
    "    - takes the function as a lambda and the input_shape as desired\n",
    "    \n",
    "            Lambda( lambda x: tf.expand_dims(x, axis=-1), input_shape=[None])\n",
    "       - notice the none value makes it take any kind of shape\n",
    "-----------\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Optimizers in `tf.keras.optimizers`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "* `Adam()`: Optimizer that implements the Adam algorithm.\n",
    "\n",
    "        tf.keras.optimizers.Adam( learning_rate=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-07, amsgrad=False )\n",
    "---------------\n",
    "* `RMSprop()`: Optimizer that implements the RMSprop algorithm.\n",
    "\n",
    "        tf.keras.optimizers.RMSprop( learning_rate=0.001, rho=0.9, momentum=0.0, epsilon=1e-07, centered=False )\n",
    "---------------\n",
    "* `SGD()` : Stochastic gradient descent and momentum optimizer.\n",
    "\n",
    "        tf.keras.optimizers.SGD( learning_rate=0.01, momentum=0.0, nesterov=False )\n",
    "----------------\n",
    "* `Adamax()`: Optimizer that implements the Adamax algorithm.\n",
    "\n",
    "        tf.keras.optimizers.Adamax( learning_rate=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-07 )\n",
    "-----------------\n",
    "* `Nadam()`: Optimizer that implements the NAdam algorithm.\n",
    "\n",
    "        tf.keras.optimizers.Nadam( learning_rate=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-07 )\n",
    "-------------------        \n",
    "* `Adagrad()`: Optimizer that implements the Adagrad algorithm.\n",
    "\n",
    "        tf.keras.optimizers.Adagrad( learning_rate=0.001, initial_accumulator_value=0.1, epsilon=1e-07 )\n",
    "--------------------        \n",
    "* `Adadelta()`: Optimizer that implements the Adadelta algorithm.\n",
    "\n",
    "        tf.keras.optimizers.Adadelta( learning_rate=0.001, rho=0.95, epsilon=1e-07 )\n",
    "---------------------\n",
    "* `Ftrl()`: Optimizer that implements the FTRL algorithm.\n",
    "\n",
    "        tf.keras.optimizers.Ftrl(\n",
    "            learning_rate=0.001, learning_rate_power=-0.5, initial_accumulator_value=0.1,\n",
    "            l1_regularization_strength=0.0, l2_regularization_strength=0.0, name='Ftrl',\n",
    "            l2_shrinkage_regularization_strength=0.0, **kwargs )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Metrics in `tf.keras.metrics`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### You can add metrics in a list and pass it to the mertrics variable in compile\n",
    "---------------\n",
    "* `Accuracy()` : Calculates how often predictions equals labels.\n",
    "-----------------------\n",
    "* `BinaryAccuracy()` : Calculates how often predictions matches binary labels.\n",
    "-----------------------\n",
    "* `BinaryCrossentropy()`: Computes the crossentropy metric between the labels and predictions.\n",
    "-----------------------\n",
    "* `CategoricalAccuracy()`: Calculates how often predictions matches one-hot labels.\n",
    "-----------------------\n",
    "* `CategoricalCrossentropy()` : Computes the crossentropy metric between the labels and predictions.\n",
    "-----------------------\n",
    "* `CosineSimilarity()` : Computes the cosine similarity between the labels and predictions.\n",
    "-----------------------\n",
    "* `FalseNegatives()` : Calculates the number of false negatives.\n",
    "-----------------------\n",
    "* `FalsePositives()` : Calculates the number of false positives.\n",
    "-----------------------\n",
    "* `TrueNegatives()` : Calculates the number of true negatives.\n",
    "-----------------------\n",
    "* `TruePositives()` : Calculates the number of true positives.\n",
    "-----------------------\n",
    "* `KLDivergence()` : Computes Kullback-Leibler divergence metric between y_true and y_pred.\n",
    "-----------------------\n",
    "* `LogCoshError()` : Computes the logarithm of the hyperbolic cosine of the prediction error.\n",
    "-----------------------\n",
    "* `MeanAbsoluteError()` : Computes the mean absolute error between the labels and predictions.\n",
    "-----------------------\n",
    "* `MeanAbsolutePercentageError()` : Computes the mean absolute percentage error between y_true and y_pred.\n",
    "-----------------------\n",
    "* `MeanIoU()` : Computes the mean Intersection-Over-Union metric.\n",
    "-----------------------\n",
    "* `MeanRelativeError()` : Computes the mean relative error by normalizing with the given values.\n",
    "-----------------------\n",
    "* `MeanSquaredError()` : Computes the mean squared error between y_true and y_pred.\n",
    "-----------------------\n",
    "* `MeanSquaredLogarithmicError()` : Computes the mean squared logarithmic error between y_true and y_pred.\n",
    "-----------------------\n",
    "* `MeanTensor()` : Computes the element-wise (weighted) mean of the given tensors.\n",
    "-----------------------\n",
    "* `Metric()` : Encapsulates metric logic and state.\n",
    "-----------------------\n",
    "* `Poisson()` : Computes the Poisson metric between y_true and y_pred.\n",
    "-----------------------\n",
    "* `Precision()` : Computes the precision of the predictions with respect to the labels.\n",
    "-----------------------\n",
    "* `PrecisionAtRecall()` : Computes the precision at a given recall.\n",
    "-----------------------\n",
    "* `Recall()` : Computes the recall of the predictions with respect to the labels.\n",
    "-----------------------\n",
    "* `RecallAtPrecision()` : Computes the maximally achievable recall at a required precision.\n",
    "-----------------------\n",
    "* `RootMeanSquaredError()` : Computes root mean squared error metric between y_true and y_pred.\n",
    "-----------------------\n",
    "* `SensitivityAtSpecificity()` : Computes the sensitivity at a given specificity.* `\n",
    "-----------------------\n",
    "* `SparseCategoricalAccuracy()` : Calculates how often predictions matches integer labels.\n",
    "-----------------------\n",
    "* `SparseCategoricalCrossentropy()` : Computes the crossentropy metric between the labels and predictions.\n",
    "-----------------------\n",
    "* `SparseTopKCategoricalAccuracy()` : Computes how often integer targets are in the top K predictions.\n",
    "-----------------------\n",
    "* `SpecificityAtSensitivity()` : Computes the specificity at a given sensitivity.\n",
    "-----------------------\n",
    "* `SquaredHinge()` : Computes the squared hinge metric between y_true and y_pred.\n",
    "-----------------------\n",
    "* `TopKCategoricalAccuracy()` : Computes how often targets are in the top K predictions.\n",
    "-----------------------\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Losses in `tf.keras.losses`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "* `BinaryCrossentropy()` : Computes the cross-entropy loss between true labels and predicted labels.\n",
    "--------------------\n",
    "* `CategoricalCrossentropy()` : Computes the crossentropy loss between the labels and predictions.\n",
    "--------------------\n",
    "* `CategoricalHinge()` : Computes the categorical hinge loss between y_true and y_pred.\n",
    "--------------------\n",
    "* `CosineSimilarity()` : Computes the cosine similarity between y_true and y_pred.\n",
    "--------------------\n",
    "* `Hinge()` : Computes the hinge loss between y_true and y_pred.\n",
    "--------------------\n",
    "* `Huber()` : Computes the Huber loss between y_true and y_pred. Works well with OUTLIERS\n",
    "--------------------\n",
    "* `KLDivergence()` : Computes Kullback-Leibler divergence loss between y_true and y_pred.\n",
    "--------------------\n",
    "* `LogCosh()` : Computes the logarithm of the hyperbolic cosine of the prediction error.\n",
    "--------------------\n",
    "* `Loss()` : Loss base class.\n",
    "--------------------\n",
    "* `MeanAbsoluteError()` : Computes the mean of absolute difference between labels and predictions.\n",
    "--------------------\n",
    "* `MeanAbsolutePercentageError()` : Computes the mean absolute percentage error between y_true and y_pred.\n",
    "--------------------\n",
    "* `MeanSquaredError()` : Computes the mean of squares of errors between labels and predictions.\n",
    "--------------------\n",
    "* `MeanSquaredLogarithmicError()` : Computes the mean squared logarithmic error between y_true and y_pred.\n",
    "--------------------\n",
    "* `Poisson()` : Computes the Poisson loss between y_true and y_pred.\n",
    "--------------------\n",
    "* `Reduction()` : Types of loss reduction.\n",
    "--------------------\n",
    "* `SparseCategoricalCrossentropy()` : Computes the crossentropy loss between the labels and predictions.\n",
    "--------------------\n",
    "* `SquaredHinge()` : Computes the squared hinge loss between y_true and y_pred.\n",
    "--------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Initializers `in tf.keras.initializers`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "* `Constant()` : Initializer that generates tensors with constant values.\n",
    "---------------\n",
    "* `GlorotNormal()` : The Glorot normal initializer, also called Xavier normal initializer.\n",
    "---------------\n",
    "* `GlorotUniform()` : The Glorot uniform initializer, also called Xavier uniform initializer.\n",
    "---------------\n",
    "* `Identity()` : Initializer that generates the identity matrix.\n",
    "---------------\n",
    "* `Initializer()` : Initializer base class: all initializers inherit from this class.\n",
    "---------------\n",
    "* `Ones()` : Initializer that generates tensors initialized to 1.\n",
    "---------------\n",
    "* `Orthogonal()` : Initializer that generates an orthogonal matrix.\n",
    "---------------\n",
    "* `RandomNormal()` : Initializer that generates tensors with a normal distribution.\n",
    "---------------\n",
    "* `RandomUniform()` : Initializer that generates tensors with a uniform distribution.\n",
    "---------------\n",
    "* `TruncatedNormal()` : Initializer that generates a truncated normal distribution.\n",
    "---------------\n",
    "* `VarianceScaling()` : Initializer capable of adapting its scale to the shape of weights tensors.\n",
    "---------------\n",
    "* `Zeros()` : Initializer that generates tensors initialized to 0.\n",
    "---------------\n",
    "* `constant()` : Initializer that generates tensors with constant values.\n",
    "---------------\n",
    "* `glorot_normal()` : The Glorot normal initializer, also called Xavier normal initializer.\n",
    "---------------\n",
    "* `glorot_uniform()` : The Glorot uniform initializer, also called Xavier uniform initializer.\n",
    "---------------\n",
    "* `identity()` : Initializer that generates the identity matrix.\n",
    "---------------\n",
    "* `ones()` : Initializer that generates tensors initialized to 1.\n",
    "---------------\n",
    "* `orthogonal()` : Initializer that generates an orthogonal matrix.\n",
    "---------------\n",
    "* `zeros()` : Initializer that generates tensors initialized to 0.\n",
    "---------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Callbacks in `tf.keras.callbacks`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "* `BaseLogger()` : Callback that accumulates epoch averages of metrics.\n",
    "\n",
    "        tf.keras.callbacks.BaseLogger( stateful_metrics=None )\n",
    "---------------------\n",
    "* `CSVLogger()` : Callback that streams epoch results to a csv file.\n",
    "\n",
    "        tf.keras.callbacks.CSVLogger( filename, separator=',', append=False )\n",
    "---------------------\n",
    "* `Callback()` : Abstract base class used to build new callbacks.\n",
    "\n",
    "        tf.keras.callbacks.Callback()\n",
    "\n",
    "    - The logs dictionary that callback methods take as argument will contain keys for quantities relevant to the current batch or epoch.\n",
    "\n",
    "    - Currently, the .fit() method of the Model class will include the following quantities in the logs that it passes to its callbacks:\n",
    "\n",
    "        - on_epoch_end: logs include `acc` and `loss`, and\n",
    "            optionally include `val_loss`\n",
    "            (if validation is enabled in `fit`), and `val_acc`\n",
    "            (if validation and accuracy monitoring are enabled).\n",
    "        - on_batch_begin: logs include `size`,\n",
    "            the number of samples in the current batch.\n",
    "        - on_batch_end: logs include `loss`, and optionally `acc`\n",
    "            (if accuracy monitoring is enabled).\n",
    "    \n",
    "---------------------\n",
    "* `EarlyStopping()` : Stop training when a monitored metric has stopped improving.\n",
    "\n",
    "        tf.keras.callbacks.EarlyStopping( monitor='val_loss', min_delta=0, patience=0, \n",
    "                                          verbose=0, mode='auto', baseline=None, restore_best_weights=False )\n",
    "\n",
    "---------------------\n",
    "* `LambdaCallback()` : Callback for creating simple, custom callbacks on-the-fly.\n",
    "        \n",
    "        tf.keras.callbacks.LambdaCallback( on_epoch_begin=None, on_epoch_end=None, \n",
    "                                            on_batch_begin=None, on_batch_end=None, \n",
    "                                            on_train_begin=None, on_train_end=None, **kwargs )\n",
    "---------------------\n",
    "* `LearningRateScheduler()` : Learning rate scheduler.\n",
    "        \n",
    "        tf.keras.callbacks.LearningRateScheduler( schedule, verbose=0 )\n",
    "\n",
    "\n",
    "---------------------\n",
    "* `ModelCheckpoint()` : Callback to save the Keras model or model weights at some frequency.\n",
    "\n",
    "        tf.keras.callbacks.ModelCheckpoint(\n",
    "                                            filepath, monitor='val_loss', verbose=0, save_best_only=False,\n",
    "                                            save_weights_only=False, mode='auto', save_freq='epoch', **kwargs )\n",
    "---------------------\n",
    "* `ProgbarLogger()` : Callback that prints metrics to stdout.\n",
    "            \n",
    "        tf.keras.callbacks.ProgbarLogger( count_mode='samples', stateful_metrics=None )\n",
    "---------------------\n",
    "* `ReduceLROnPlateau()` : Reduce learning rate when a metric has stopped improving.\n",
    "\n",
    "        tf.keras.callbacks.ReduceLROnPlateau(   monitor='val_loss', factor=0.1, patience=10, verbose=0, mode='auto',\n",
    "                                                min_delta=0.0001, cooldown=0, min_lr=0, **kwargs )\n",
    "---------------------\n",
    "* `RemoteMonitor()` : Callback used to stream events to a server.\n",
    "\n",
    "        tf.keras.callbacks.RemoteMonitor(   root='http://localhost:9000', path='/publish/epoch/end/', \n",
    "                                            field='data', headers=None, send_as_json=False )\n",
    "---------------------\n",
    "* `TensorBoard()` : Enable visualizations for TensorBoard.\n",
    "\n",
    "        tf.keras.callbacks.TensorBoard( log_dir='logs', histogram_freq=0, write_graph=True, write_images=False,\n",
    "                                        update_freq='epoch', profile_batch=2, embeddings_freq=0,\n",
    "                                        embeddings_metadata=None, **kwargs )\n",
    "---------------------\n",
    "* `TerminateOnNaN()` : Callback that terminates training when a NaN loss is encountered.\n",
    "        \n",
    "        tf.keras.callbacks.TerminateOnNaN()\n",
    "---------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Activations in `tf.keras.activations`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "* `deserialize(...)` : Returns activation function denoted by input string.\n",
    "----------------------\n",
    "\n",
    "* `elu(...)` : Exponential linear unit.\n",
    "----------------------\n",
    "\n",
    "* `exponential(...)` : Exponential activation function.\n",
    "----------------------\n",
    "\n",
    "* `get(...)` : Returns function.\n",
    "----------------------\n",
    "\n",
    "* `hard_sigmoid(...)` : Hard sigmoid activation function.\n",
    "----------------------\n",
    "\n",
    "* `linear(...)` : Linear activation function.\n",
    "----------------------\n",
    "\n",
    "* `relu(...)` : Applies the rectified linear unit activation function.\n",
    "----------------------\n",
    "\n",
    "* `selu(...)` : Scaled Exponential Linear Unit (SELU).\n",
    "----------------------\n",
    "\n",
    "* `serialize(...)` : Returns name attribute (__name__) of function.\n",
    "----------------------\n",
    "\n",
    "* `sigmoid(...)` : Sigmoid activation function.\n",
    "----------------------\n",
    "\n",
    "* `softmax(...)` : Softmax converts a real vector to a vector of categorical probabilities.\n",
    "----------------------\n",
    "\n",
    "* `softplus(...)` : Softplus activation function.\n",
    "----------------------\n",
    "\n",
    "* `softsign(...)` : Softsign activation function.\n",
    "----------------------\n",
    "\n",
    "* `swish(...)` : Swish activation function.\n",
    "----------------------\n",
    "\n",
    "* `tanh(...)` : Hyperbolic tangent activation function.\n",
    "----------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Transfere Learning\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "* 1- import the model you want:\n",
    "\n",
    "        from tensorflow.keras import Model\n",
    "\n",
    "        from tensorflow.keras.applications.inception_v3 import InceptionV3\n",
    "\n",
    "        local_weights_file = '/tmp/inception_v3_weights_tf_dim_ordering_tf_kernels_notop.h5' # if you have the weights you want downloaded already\n",
    "\n",
    "        pre_trained_model = InceptionV3(input_shape = (150, 150, 3),include_top = False, pooling=None , weights = None )\n",
    "-----------       \n",
    "* 2- decide what to do with the weights:\n",
    "    - `weights`:\tone of None (random initialization), 'imagenet' (pre-training on ImageNet), or the path to the weights file to be loaded.\n",
    "    - if you choose to train the model from scratch then use \n",
    "        `classes:\toptional number of classes to classify images into, only to be specified if include_top is True, and if no weights argument is specified.\n",
    "--------------\n",
    "* 3- if you chose to add your own layers:\n",
    "\n",
    "        #freeze the unwanted layers\n",
    "        for layer in pre_trained_model.layers:\n",
    "          layer.trainable = False \n",
    "        \n",
    "        # print the summary to see the layers and their dimentions and names\n",
    "        pre_trained_model.summary()\n",
    "\n",
    "        last_layer = pre_trained_model.get_layer('mixed7')        # here we named the layer to cut at\n",
    "        last_layer= pre_trained_model.git_layer(index=-1)         # or you can use this to refere to the last layer\n",
    "        print('last layer output shape: ', last_layer.output_shape) # to print the shape -not essential-\n",
    "        last_output = last_layer.output\n",
    "    - you can choose any layer you want 'from the summary' and put it as the last_output, you can do that to reduce the params or even cut the model where you want to start attaching your new layers and get rid of the unwanted layers/features after the last_output\n",
    " \n",
    "-----------------\n",
    "* 4- Building the new layers of our own but using tensorflow way, not `tf.keras.model.Sequential`:\n",
    "\n",
    "        # Flatten the output layer to 1 dimension\n",
    "        x = layers.Flatten()(last_output) # using the output of the pretrained model as the input to our new layers\n",
    "        x = layers.Dense(1024, activation='relu')(x)\n",
    "        x = layers.Dropout(0.2)(x)                  \n",
    "        x = layers.Dense  (1, activation='sigmoid')(x)           \n",
    "        \n",
    "        # specifing the input of the new_model 'pretrained model input' and the output 'x that we created'\n",
    "        new_model = Model( pre_trained_model.input, x) \n",
    "----------\n",
    "* 5- then compile and fit as usual.\n",
    "-----------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Helpful functions in `tf.nn`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Normalization:\n",
    "- 1 `batch_norm_with_global_normalization(...)`: Batch normalization.\n",
    "- 2 `batch_normalization(...)`: Batch normalization.\n",
    "------------\n",
    "### Loss:\n",
    "- 1 `l2_loss(...)`: L2 Loss.\n",
    "- 2 `l2_normalize(...)`: Normalizes along dimension axis using an L2 norm.\n",
    "- 3 `softmax_cross_entropy_with_logits(...)`: Computes softmax cross entropy between logits and labels.\n",
    "- 4 `sparse_softmax_cross_entropy_with_logits(...)`: Computes sparse softmax cross entropy between logits and labels.\n",
    "- 5 `weighted_cross_entropy_with_logits(...)`: Computes a weighted cross entropy.\n",
    "-----------------\n",
    "### Activations:\n",
    "- 1 `leaky_relu(...)`: Compute the Leaky ReLU activation function.\n",
    "- 2 `log_softmax(...)`: Computes log softmax activations.\n",
    "- 3 `relu(...)`: Computes rectified linear: max(features, 0).\n",
    "- 4 `sigmoid(...)`: Computes sigmoid of x element-wise.\n",
    "- 5 `softmax(...)`: Computes softmax activations.\n",
    "- 6 `tanh(...)`: Computes hyperbolic tangent of x element-wise.\n",
    "---------------\n",
    "### Sequences stuff:\n",
    "- 1 `embedding_lookup(...)`: Looks up ids in a list of embedding tensors.\n",
    "- 2 `embedding_lookup_sparse(...)`: Computes embeddings for the given ids and weights.\n",
    "- 3 `ctc_beam_search_decoder(...)`: Performs beam search decoding on the logits given in input.\n",
    "- 4 `ctc_greedy_decoder(...)`: Performs greedy decoding on the logits given in input (best path).\n",
    "- 5 `ctc_loss(...)`: Computes CTC (Connectionist Temporal Classification) loss.\n",
    "- 6 `ctc_unique_labels(...)`: Get unique labels and indices for batched labels for tf.nn.ctc_loss.\n",
    "--------------------\n",
    "### Others\n",
    "- 1 `top_k(...)`: Finds values and indices of the k largest entries for the last dimension.\n",
    "- 2 `in_top_k(...)`: Says whether the targets are in the top K predictions.\n",
    "- 3 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Customized callback in Keras\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-23T13:43:10.859870Z",
     "start_time": "2020-06-23T13:43:10.851873Z"
    },
    "hidden": true
   },
   "source": [
    "\n",
    "* 1: define a class with an input of keras.callbacks.Callback :\n",
    "\n",
    "        class myCallback(tf.keras.callbacks.Callback):\n",
    "-------\n",
    "* 2: define the function that should run when the epoch ends `on_epoch_end`  :\n",
    "    - DO NOT EVER CHANGE THE NAME OF THE FUNCTION, u can change the name of the class BUT NOT THIS FUNCTION. ty, maybe there is on_epoch_start func?\n",
    "    - dont forget to give it the first input as `self` , and the epochs for epoch number, and the logs dictionary\n",
    "    - the logs dict contains the loss and accuracy\n",
    "    \n",
    "            def on_epoch_end(self,epochs,logs={}):\n",
    "---------\n",
    "* 3: define the calculations you want inside that function:\n",
    "     - it can be a simple print command or conditions or other commands liek save weights etc..\n",
    "     \n",
    "                if logs.get('acc') >=99:\n",
    "                    print('Reached 99% accuracy so cancelling training!')\n",
    "     - to stop the training you can simply assign the value `True` to `model.stop_training` \n",
    "     \n",
    "                    model.stop_training=True\n",
    "---------\n",
    "* 4: assign the value for the class to a variable, that you will insert into callbacks in the fit method\n",
    "        `cb=myCallback()`\n",
    "        `model.fit( x,y,epoch=1, callbacks=[cb] )`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Prediction on images"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "* 1- importing the image function\n",
    "\n",
    "        from tf.keras.preprocessing import image\n",
    "* 2- setting the path of the image\n",
    "\n",
    "        img_path='path'\n",
    "* 3- uploading the image into a variable\n",
    "\n",
    "        img= image.load_img( path , target_size=( , ) )\n",
    "  - don't forget the target size the model is expecting\n",
    "* 4- processing the image variable to suit the model\n",
    "\n",
    "        x= image.img_to_array( img )\n",
    "        x= np.expand_dims( x , axis=0 )\n",
    "        images= np.vstack( [x] )\n",
    "    - then feed the variable images into the predict fun."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Plot Loss and Accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    import matplotlib.pyplot as plt\n",
    "    acc = history.history['acc']\n",
    "    val_acc = history.history['val_acc']\n",
    "    loss = history.history['loss']\n",
    "    val_loss = history.history['val_loss']\n",
    "\n",
    "    epochs = range(len(acc))\n",
    "\n",
    "    plt.plot(epochs, acc, 'r', label='Training accuracy')\n",
    "    plt.plot(epochs, val_acc, 'b', label='Validation accuracy')\n",
    "    plt.title('Training and validation accuracy')\n",
    "\n",
    "    plt.figure()\n",
    "\n",
    "    plt.plot(epochs, loss, 'r', label='Training Loss')\n",
    "    plt.plot(epochs, val_loss, 'b', label='Validation Loss')\n",
    "    plt.title('Training and validation loss')\n",
    "    plt.legend()\n",
    "\n",
    "    plt.show()\n",
    "        \n",
    "\n",
    "===========================================================================================================================================================\n",
    "* Another Example:\n",
    "\n",
    "            %matplotlib inline\n",
    "            import matplotlib.image  as mpimg\n",
    "            import matplotlib.pyplot as plt`\n",
    "\n",
    "-----------------------------------------------------------\n",
    "* Retrieve a list of list results on training and test data\n",
    "* sets for each training epoch\n",
    "-----------------------------------------------------------\n",
    "\n",
    "            acc=history.history['acc']\n",
    "            val_acc=history.history['val_acc']\n",
    "            loss=history.history['loss']\n",
    "            val_loss=history.history['val_loss']\n",
    "\n",
    "            epochs=range(len(acc)) # Get number of epochs\n",
    "\n",
    "------------------------------------------------\n",
    "* Plot training and validation accuracy per epoch\n",
    "------------------------------------------------\n",
    "\n",
    "            plt.plot(epochs, acc, 'r', \"Training Accuracy\")\n",
    "            plt.plot(epochs, val_acc, 'b', \"Validation Accuracy\")\n",
    "            plt.title('Training and validation accuracy')\n",
    "            plt.figure()\n",
    "\n",
    "------------------------------------------------\n",
    "* Plot training and validation loss per epoch\n",
    "------------------------------------------------\n",
    "\n",
    "            plt.plot(epochs, loss, 'r', \"Training Loss\")\n",
    "            plt.plot(epochs, val_loss, 'b', \"Validation Loss\")\n",
    "            plt.title('Training and validation loss')\n",
    "---------------------------------------------------------------------\n",
    "* Desired output. Charts with training and validation metrics. No crash :)\n",
    "===========================================================================================================================================================\n",
    "* Another Plotting method to zoom into a desired area in the plot:\n",
    "\n",
    "        import matplotlib.image  as mpimg\n",
    "        import matplotlib.pyplot as plt\n",
    "\n",
    "#-----------------------------------------------------------\n",
    "### Retrieve a list of list results on training and test data\n",
    "### sets for each training epoch\n",
    "#-----------------------------------------------------------\n",
    "\n",
    "        loss=history.history['loss']\n",
    "\n",
    "        epochs=range(len(loss)) # Get number of epochs\n",
    "\n",
    "\n",
    "#------------------------------------------------\n",
    "### Plot training and validation loss per epoch\n",
    "#------------------------------------------------\n",
    "\n",
    "        plt.plot(epochs, loss, 'r')\n",
    "        plt.title('Training loss')\n",
    "        plt.xlabel(\"Epochs\")\n",
    "        plt.ylabel(\"Loss\")\n",
    "        plt.legend([\"Loss\"])\n",
    "\n",
    "        plt.figure()\n",
    "\n",
    "\n",
    "\n",
    "        zoomed_loss = loss[200:]\n",
    "        zoomed_epochs = range(200,500)\n",
    "\n",
    "\n",
    "#------------------------------------------------\n",
    "### Plot training and validation loss per epoch\n",
    "#------------------------------------------------\n",
    "\n",
    "        plt.plot(zoomed_epochs, zoomed_loss, 'r')\n",
    "        plt.title('Training loss')\n",
    "        plt.xlabel(\"Epochs\")\n",
    "        plt.ylabel(\"Loss\")\n",
    "        plt.legend([\"Loss\"])\n",
    "\n",
    "        plt.figure()\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Clearing Data using python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "* 1- To Filters the images by the size, if the image has 0 size it will not be moved to the destination:\n",
    "\n",
    "        def clear_data(data_path, destination_path ):\n",
    "\n",
    "            files= os.listdir(data_path) \n",
    "            valid=[]\n",
    "            for i in files:\n",
    "                path= data_path +'/'+ i\n",
    "\n",
    "                if os.path.getsize(path): #checks if the size of the image has a value and not NaN\n",
    "                    valid.append(i)\n",
    "\n",
    "            n_valid = len(valid)\n",
    "\n",
    "            for t in valid:\n",
    "                copyfile( data_path +'/'+t , destination_path +'/'+ t) \n",
    "\n",
    "        #data_path \n",
    "        CAT_SOURCE_DIR = r'M:\\Courses\\Coursera-2020\\TensorFlow-in-Practice-Specialization\\2 convolutional-neural-networks-tensorflow\\Codes\\CatsvsDogs\\Cat' \n",
    "        DOG_SOURCE_DIR = r'M:\\Courses\\Coursera-2020\\TensorFlow-in-Practice-Specialization\\2 convolutional-neural-networks-tensorflow\\Codes\\CatsvsDogs\\Dog' \n",
    "\n",
    "        #destination_path : MAKE SURE YOU HAVE CREATED THAT USING os.mkdir( path_you_want )\n",
    "        cat_dest= r'M:\\Courses\\Coursera-2020\\TensorFlow-in-Practice-Specialization\\2 convolutional-neural-networks-tensorflow\\Codes\\CatsvsDogs\\Catc'\n",
    "        dog_dest= r'M:\\Courses\\Coursera-2020\\TensorFlow-in-Practice-Specialization\\2 convolutional-neural-networks-tensorflow\\Codes\\CatsvsDogs\\Dogc'\n",
    "\n",
    "        clear_data(CAT_SOURCE_DIR, cat_dest )\n",
    "        clear_data(DOG_SOURCE_DIR, dog_dest )\n",
    "---------------------\n",
    "* 2- To split the data on a directory into training and testing :\n",
    "\n",
    "        def split_data(SOURCE, TRAINING, TESTING, SPLIT_SIZE):\n",
    "\n",
    "            files= os.listdir(SOURCE) \n",
    "            valid=[]\n",
    "            for i in files:\n",
    "                path= SOURCE + i\n",
    "\n",
    "                if os.path.getsize(path) > 0:\n",
    "                    valid.append(i)\n",
    "\n",
    "            n_valid = len(valid)\n",
    "\n",
    "\n",
    "            split = int(n_valid * SPLIT_SIZE)\n",
    "\n",
    "            shuffled = random.sample(valid , n_valid)\n",
    "\n",
    "            train_set = shuffled[:split]\n",
    "            test_set = shuffled[split:]\n",
    "\n",
    "            for t in train_set:\n",
    "                copyfile(SOURCE + t , TRAINING + t)\n",
    "\n",
    "            for s in test_set:\n",
    "                copyfile(SOURCE + s, TESTING + s)\n",
    "\n",
    "        CAT_SOURCE_DIR = \"/tmp/PetImages/Cat/\"\n",
    "        TRAINING_CATS_DIR = \"/tmp/cats-v-dogs/training/cats/\"\n",
    "        TESTING_CATS_DIR = \"/tmp/cats-v-dogs/testing/cats/\"\n",
    "        split_size = .9\n",
    "        split_data(CAT_SOURCE_DIR, TRAINING_CATS_DIR, TESTING_CATS_DIR, split_size)\n",
    "------------\n",
    "\n",
    "* 3- To iterate through json text data:\n",
    "\n",
    "        import json\n",
    "\n",
    "        with open(\"/tmp/sarcasm.json\", 'r') as f:\n",
    "            datastore = json.load(f)\n",
    "\n",
    "        # assuming the data in this file is a dictionary having 3 keys ( 'headline', 'is_sarcastic' , 'article_link' )\n",
    "        sentences = [] \n",
    "        labels = []\n",
    "        urls = []\n",
    "        for item in datastore:\n",
    "            sentences.append(item['headline'])\n",
    "            labels.append(item['is_sarcastic'])\n",
    "            urls.append(item['article_link'])\n",
    "\n",
    "------------\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "-------\n",
    "* to reverse a dictionary:\n",
    "\n",
    "        reverse= dict( [(value,key) for (key,value) in dict] ) notice the ' , 'between the tubles we are putting each into list\n",
    "        #or\n",
    "        variable= {u:i for i, u in enumerate(vocab)} #notice the ' : ' between the key and value\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Directory/Path tricks\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## Notice: if the model is not learning and accuracy is not increasing, then the problem will be from the dataset, something wrong with the path or folders in it, make sure you check the '/' of the path carefully\n",
    "---------------------------------\n",
    "## Make sure your directory is correctly written\n",
    "### it should be like:\n",
    "`dir=r'M:\\Courses\\Coursera-2020\\TensorFlow-in-Practice-Specialization\\2 convolutional-neural-networks-tensorflow\\Codes\\CatsvsDogs'+'/'`\n",
    "### we add the r to make sure the backslashes are correct for the path\n",
    "--------------------------------\n",
    "\n",
    "    # Define our example directories path -- the variable points at the folder not it's inside\n",
    "    train_dir = '/tmp/training'\n",
    "    validation_dir = '/tmp/validation'\n",
    "    \n",
    "    # Define the inner directories path -- the variable points at the folder not it's inside\n",
    "    train_horses_dir = '/tmp/training/horses'\n",
    "    train_humans_dir = '/tmp/training/humans'\n",
    "    validation_horses_dir = '/tmp/validation/horses'\n",
    "    validation_humans_dir = '/tmp/validation/humans'\n",
    "    \n",
    "    # to access the names of files inside the folders you need to provide extra '/' to look inside\n",
    "    train_horses_fnames = os.listdir(train_horses_dir + '/' ) \n",
    "    train_humans_fnames = os.listdir(train_humans_dir + '/' ) \n",
    "    validation_horses_fnames = os.listdir( validation_horses_dir + '/') \n",
    "    validation_humans_fnames = os.listdir(validation_humans_dir + '/' ) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# NLP related\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### Steps:\n",
    "#### Dont forget to create variables for the most important params in the tokenizer or embedding layer like:\n",
    "            words = 10000 # Maximum number of words to be tokenized, and picks the most common ‘n’ words\n",
    "            e_dim = 16 # Number of dimensions for the vector representing the word encoding\n",
    "            max_length = 100\n",
    "            trunc_type='post'\n",
    "            padding_type='post'\n",
    "            oov_tok = \"<OOV>\"\n",
    "            training_size = 20000\n",
    "* 1- get the text data from the source or input\n",
    "\n",
    "* 2- create the tokenizer variable with the params needed \n",
    "\n",
    "* 3- fit the text to the tokenizer \n",
    "\n",
    "* 4- create the variable word_to_idx dictionary from the tokenizer\n",
    "\n",
    "#### for the LABELS data and TESTING data you will repeat the next two points\n",
    "\n",
    "* 5- generate the sequences from the tokenizer using the text/labels and assign them to the variable sequences\n",
    "\n",
    "* 6- pad the sequences to the desired length and then convert them into np.array\n",
    "\n",
    "* 7- build the model and pass the padded seq. as an input to the embedded layer as the first layer in the model\n",
    "\n",
    "* 8-in case you will use regular dense layers after the embedding layer use flatten layer or `GlobalAveragePooling1D()` to flatten the embeddings and feed them to the dense layers\n",
    "\n",
    "* 9- in case of using lstm you don't need to flatten the output of the embedding layer, make sure to return sequences from the lstm to the next one if you are going to stack them, with the last lstm NOT returning sequences\n",
    "\n",
    "* 10-  in case you want to build a model to predict a word check the week 4 of tensorflow in practice codes \n",
    "\n",
    "-----------\n",
    "* Tokenizer\n",
    "\n",
    "        tf.keras.preprocessing.text.Tokenizer( num_words=None, oov_token=None , lower=True, split=' ', char_level=False, document_count=0 )\n",
    "        \n",
    "    - split: str. Separator for word splitting.\n",
    "    - char_level: if True, every character will be treated as a token.\n",
    "    - oov_token: if given, it will be added to word_index and used to\n",
    "            replace out-of-vocabulary words during text_to_sequence calls     \n",
    "\n",
    "        * Updates internal vocabulary based on a list of sequences. # Put in mind that sequences are arrays of indices representing the words\n",
    "            `fit_on_sequences( sequences )`\n",
    "        * Updates internal vocabulary based on a list of texts.\n",
    "            `fit_on_texts( texts )`\n",
    "        * Converts a list of sequences into a Numpy matrix.\n",
    "            `sequences_to_matrix( sequences, mode='binary' )\n",
    "        * Transforms each sequence into a list of text.\n",
    "            `sequences_to_texts( sequences )\n",
    "        * Transforms each text in texts to a sequence of integers.\n",
    "            ` texts_to_sequences( texts )`\n",
    "        * Convert a list of texts to a Numpy matrix.\n",
    "            `texts_to_matrix( texts, mode='binary' )`\n",
    "     - Retuns a Dictionary with keys as the words and values are the index\n",
    "            `tokenizer.word_index`\n",
    "     - \n",
    "     \n",
    "--------\n",
    "* Padding:\n",
    "\n",
    "            tf.keras.preprocessing.sequence.pad_sequences( sequences, maxlen=None, dtype='int32', padding='post', truncating='pre', value=0.0 )\n",
    "\n",
    "   - padds the sequences with zeros to make them all of the same `maxlen` or if not specified it will padd them to the  length of the longest individual sequence.\n",
    "   \n",
    "----------\n",
    "* Embedding:\n",
    "\n",
    "            tf.keras.layers.Embedding(  input_dim= -vocab_size- , output_dim= -embedding_dim-  ,input_length= -max_length- , \n",
    "                                        embeddings_initializer='uniform' , activity_regularizer=None,\n",
    "                                        embeddings_constraint=None, mask_zero=False, embeddings_regularizer=None )\n",
    "    - to visualize the embedding layer:\n",
    "       \n",
    "            e = model.layers[0]\n",
    "            weights = e.get_weights()[0]\n",
    "            print(weights.shape) # shape: (-vocab_size-, -embedding_dim-)\n",
    "            import io\n",
    "            out_v = io.open('vecs.tsv', 'w', encoding='utf-8')\n",
    "            out_m = io.open('meta.tsv', 'w', encoding='utf-8')\n",
    "            for word_num in range(1, -vocab_size- ):\n",
    "              word = reverse_word_index[word_num]\n",
    "              embeddings = weights[word_num]\n",
    "              out_m.write(word + \"\\n\")\n",
    "              out_v.write('\\t'.join([str(x) for x in embeddings]) + \"\\n\")\n",
    "            out_v.close()\n",
    "            out_m.close()\n",
    "            #try:\n",
    "            #  from google.colab import files\n",
    "            #except ImportError:\n",
    "            #  pass\n",
    "            #else:\n",
    "            #  files.download('vecs.tsv')\n",
    "            #  files.download('meta.tsv')\n",
    "\n",
    "     - this code will create the two files. To now render the results, go to the TensorFlow Embedding Projector on projector.tensorflow.org\n",
    "     \n",
    "--------     \n",
    "* LSTM:\n",
    "            \n",
    "            tf.keras.layers.LSTM(\n",
    "                                    units, activation='tanh', recurrent_activation='sigmoid', use_bias=True,\n",
    "                                    kernel_initializer='glorot_uniform', recurrent_initializer='orthogonal',\n",
    "                                    bias_initializer='zeros', unit_forget_bias=True, kernel_regularizer=None,\n",
    "                                    recurrent_regularizer=None, bias_regularizer=None, activity_regularizer=None,\n",
    "                                    dropout=0.0, recurrent_dropout=0.0, implementation=2, return_sequences=False,\n",
    "                                    return_state=False, go_backwards=False, stateful=False, time_major=False,\n",
    "                                    unroll=False )\n",
    "--------\n",
    "* Bidirectional:\n",
    "\n",
    "            tf.keras.layers.Bidirectional( layer, merge_mode='concat', weights=None, backward_layer=None )\n",
    "            \n",
    "--------\n",
    "* Conv1D:\n",
    "            \n",
    "            tf.keras.layers.Conv1D( filters, kernel_size, strides=1, padding='valid', activation=None )\n",
    "            \n",
    "-----------------\n",
    "\n",
    "* MaxPooling1D:\n",
    "            \n",
    "            tf.keras.layers.MaxPooling1D(pool_size=4)\n",
    "-------------\n",
    "* GRU:\n",
    "            \n",
    "            tf.keras.layers.GRU(\n",
    "                                   units, activation='tanh', recurrent_activation='sigmoid', use_bias=True,\n",
    "                                   kernel_initializer='glorot_uniform', recurrent_initializer='orthogonal',\n",
    "                                   bias_initializer='zeros', kernel_regularizer=None, recurrent_regularizer=None,\n",
    "                                   bias_regularizer=None, activity_regularizer=None, kernel_constraint=None,\n",
    "                                   recurrent_constraint=None, bias_constraint=None, dropout=0.0,\n",
    "                                   recurrent_dropout=0.0, implementation=2, return_sequences=False,\n",
    "                                   return_state=False, go_backwards=False, stateful=False, unroll=False,\n",
    "                                   time_major=False, reset_after=True )\n",
    "                                   \n",
    "-----------\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}