Skip to content

Commit 86d99ec

Browse files
author
Saurabh Tandale
committedJan 5, 2019
check backpropagation work properly or not
1 parent 1bcc0d7 commit 86d99ec

File tree

6 files changed

+730
-0
lines changed

6 files changed

+730
-0
lines changed
 

‎Gradient checking/.ipynb_checkpoints/Gradient Checking-checkpoint.ipynb

Lines changed: 331 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,331 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {},
6+
"source": [
7+
"# Gradient Checking"
8+
]
9+
},
10+
{
11+
"cell_type": "code",
12+
"execution_count": 10,
13+
"metadata": {
14+
"collapsed": true
15+
},
16+
"outputs": [],
17+
"source": [
18+
"# Packages\n",
19+
"import numpy as np\n",
20+
"from test import *\n",
21+
"from Supporting_func import sigmoid, relu, dictionary_to_vector, vector_to_dictionary, gradients_to_vector"
22+
]
23+
},
24+
{
25+
"cell_type": "markdown",
26+
"metadata": {},
27+
"source": [
28+
"## N-dimensional gradient checking"
29+
]
30+
},
31+
{
32+
"cell_type": "markdown",
33+
"metadata": {
34+
"collapsed": true
35+
},
36+
"source": [
37+
"The following figure describes the forward and backward propagation of your fraud detection model.\n",
38+
"\n",
39+
"<img src=\"images/NDgrad_kiank.png\" style=\"width:600px;height:400px;\">\n",
40+
"<caption><center> <u> **Figure** </u><br>*LINEAR -> RELU -> LINEAR -> RELU -> LINEAR -> SIGMOID*</center></caption>"
41+
]
42+
},
43+
{
44+
"cell_type": "code",
45+
"execution_count": 17,
46+
"metadata": {
47+
"collapsed": true
48+
},
49+
"outputs": [],
50+
"source": [
51+
"def forward_propagation_n(X, Y, parameters):\n",
52+
" \"\"\"\n",
53+
" Implements the forward propagation (and computes the cost) presented in above Figure.\n",
54+
" \n",
55+
" Arguments:\n",
56+
" X -- training set for m examples\n",
57+
" Y -- labels for m examples \n",
58+
" parameters -- python dictionary containing your parameters \"W1\", \"b1\", \"W2\", \"b2\", \"W3\", \"b3\":\n",
59+
" W1 -- weight matrix of shape (5, 4)\n",
60+
" b1 -- bias vector of shape (5, 1)\n",
61+
" W2 -- weight matrix of shape (3, 5)\n",
62+
" b2 -- bias vector of shape (3, 1)\n",
63+
" W3 -- weight matrix of shape (1, 3)\n",
64+
" b3 -- bias vector of shape (1, 1)\n",
65+
" \n",
66+
" Returns:\n",
67+
" cost -- the cost function (logistic cost for one example)\n",
68+
" \"\"\"\n",
69+
" \n",
70+
" # retrieve parameters\n",
71+
" m = X.shape[1]\n",
72+
" W1 = parameters[\"W1\"]\n",
73+
" b1 = parameters[\"b1\"]\n",
74+
" W2 = parameters[\"W2\"]\n",
75+
" b2 = parameters[\"b2\"]\n",
76+
" W3 = parameters[\"W3\"]\n",
77+
" b3 = parameters[\"b3\"]\n",
78+
"\n",
79+
" # LINEAR -> RELU -> LINEAR -> RELU -> LINEAR -> SIGMOID\n",
80+
" Z1 = np.dot(W1, X) + b1\n",
81+
" A1 = relu(Z1)\n",
82+
" Z2 = np.dot(W2, A1) + b2\n",
83+
" A2 = relu(Z2)\n",
84+
" Z3 = np.dot(W3, A2) + b3\n",
85+
" A3 = sigmoid(Z3)\n",
86+
"\n",
87+
" # Cost\n",
88+
" logprobs = np.multiply(-np.log(A3),Y) + np.multiply(-np.log(1 - A3), 1 - Y)\n",
89+
" cost = 1./m * np.sum(logprobs)\n",
90+
" \n",
91+
" cache = (Z1, A1, W1, b1, Z2, A2, W2, b2, Z3, A3, W3, b3)\n",
92+
" \n",
93+
" return cost, cache"
94+
]
95+
},
96+
{
97+
"cell_type": "code",
98+
"execution_count": 30,
99+
"metadata": {
100+
"collapsed": true
101+
},
102+
"outputs": [],
103+
"source": [
104+
"def backward_propagation_n(X, Y, cache):\n",
105+
" \"\"\"\n",
106+
" Implement the backward propagation presented in above figure.\n",
107+
" \n",
108+
" Arguments:\n",
109+
" X -- input datapoint, of shape (input size, 1)\n",
110+
" Y -- true \"label\"\n",
111+
" cache -- cache output from forward_propagation_n()\n",
112+
" \n",
113+
" Returns:\n",
114+
" gradients -- A dictionary with the gradients of the cost with respect to each parameter, activation and pre-activation variables.\n",
115+
" \"\"\"\n",
116+
" \n",
117+
" m = X.shape[1]\n",
118+
" (Z1, A1, W1, b1, Z2, A2, W2, b2, Z3, A3, W3, b3) = cache\n",
119+
" \n",
120+
" dZ3 = A3 - Y\n",
121+
" dW3 = 1./m * np.dot(dZ3, A2.T)\n",
122+
" db3 = 1./m * np.sum(dZ3, axis=1, keepdims = True)\n",
123+
" \n",
124+
" dA2 = np.dot(W3.T, dZ3)\n",
125+
" dZ2 = np.multiply(dA2, np.int64(A2 > 0))\n",
126+
" dW2 = 1./m * np.dot(dZ2, A1.T)*2\n",
127+
" db2 = 1./m * np.sum(dZ2, axis=1, keepdims = True)\n",
128+
" \n",
129+
" dA1 = np.dot(W2.T, dZ2)\n",
130+
" dZ1 = np.multiply(dA1, np.int64(A1 > 0))\n",
131+
" dW1 = 1./m * np.dot(dZ1, X.T)\n",
132+
" db1 = 4./m * np.sum(dZ1, axis=1, keepdims = True)\n",
133+
" \n",
134+
" gradients = {\"dZ3\": dZ3, \"dW3\": dW3, \"db3\": db3,\n",
135+
" \"dA2\": dA2, \"dZ2\": dZ2, \"dW2\": dW2, \"db2\": db2,\n",
136+
" \"dA1\": dA1, \"dZ1\": dZ1, \"dW1\": dW1, \"db1\": db1}\n",
137+
" \n",
138+
" return gradients"
139+
]
140+
},
141+
{
142+
"cell_type": "markdown",
143+
"metadata": {},
144+
"source": [
145+
"**How does gradient checking work?**.\n",
146+
"\n",
147+
"As you want to compare \"gradapprox\" to the gradient computed by backpropagation. The formula is still:\n",
148+
"\n",
149+
"$$ \\frac{\\partial J}{\\partial \\theta} = \\lim_{\\varepsilon \\to 0} \\frac{J(\\theta + \\varepsilon) - J(\\theta - \\varepsilon)}{2 \\varepsilon} \\tag{1}$$\n",
150+
"\n",
151+
"However, $\\theta$ is not a scalar anymore. It is a dictionary called \"parameters\". We implemented a function \"`dictionary_to_vector()`\" for you. It converts the \"parameters\" dictionary into a vector called \"values\", obtained by reshaping all parameters (W1, b1, W2, b2, W3, b3) into vectors and concatenating them.\n",
152+
"\n",
153+
"The inverse function is \"`vector_to_dictionary`\" which outputs back the \"parameters\" dictionary.\n",
154+
"\n",
155+
"<img src=\"images/dictionary_to_vector.png\" style=\"width:600px;height:400px;\">\n",
156+
"<caption><center> <u> **Figure 2** </u>: **dictionary_to_vector() and vector_to_dictionary()**<br> You will need these functions in gradient_check_n()</center></caption>\n",
157+
"\n",
158+
"I have also converted the \"gradients\" dictionary into a vector \"grad\" using gradients_to_vector().\n",
159+
"\n",
160+
"Here is pseudo-code that will help to implement the gradient check.\n",
161+
"\n",
162+
"For each i in num_parameters:\n",
163+
"- To compute `J_plus[i]`:\n",
164+
" 1. Set $\\theta^{+}$ to `np.copy(parameters_values)`\n",
165+
" 2. Set $\\theta^{+}_i$ to $\\theta^{+}_i + \\varepsilon$\n",
166+
" 3. Calculate $J^{+}_i$ using to `forward_propagation_n(x, y, vector_to_dictionary(`$\\theta^{+}$ `))`. \n",
167+
"- To compute `J_minus[i]`: do the same thing with $\\theta^{-}$\n",
168+
"- Compute $gradapprox[i] = \\frac{J^{+}_i - J^{-}_i}{2 \\varepsilon}$\n",
169+
"\n",
170+
"Thus, you get a vector gradapprox, where gradapprox[i] is an approximation of the gradient with respect to `parameter_values[i]`. You can now compare this gradapprox vector to the gradients vector from backpropagation. Just like for the 1D case (Steps 1', 2', 3'), compute: \n",
171+
"$$ difference = \\frac {\\| grad - gradapprox \\|_2}{\\| grad \\|_2 + \\| gradapprox \\|_2 } \\tag{3}$$"
172+
]
173+
},
174+
{
175+
"cell_type": "code",
176+
"execution_count": 31,
177+
"metadata": {
178+
"collapsed": true
179+
},
180+
"outputs": [],
181+
"source": [
182+
"# GRADED FUNCTION: gradient_check_n\n",
183+
"\n",
184+
"def gradient_check_n(parameters, gradients, X, Y, epsilon = 1e-7):\n",
185+
" \"\"\"\n",
186+
" Checks if backward_propagation_n computes correctly the gradient of the cost output by forward_propagation_n\n",
187+
" \n",
188+
" Arguments:\n",
189+
" parameters -- python dictionary containing your parameters \"W1\", \"b1\", \"W2\", \"b2\", \"W3\", \"b3\":\n",
190+
" grad -- output of backward_propagation_n, contains gradients of the cost with respect to the parameters. \n",
191+
" x -- input datapoint, of shape (input size, 1)\n",
192+
" y -- true \"label\"\n",
193+
" epsilon -- tiny shift to the input to compute approximated gradient with formula(1)\n",
194+
" \n",
195+
" Returns:\n",
196+
" difference -- difference (2) between the approximated gradient and the backward propagation gradient\n",
197+
" \"\"\"\n",
198+
" \n",
199+
" # Set-up variables\n",
200+
" parameters_values, _ = dictionary_to_vector(parameters)\n",
201+
" grad = gradients_to_vector(gradients)\n",
202+
" num_parameters = parameters_values.shape[0]\n",
203+
" J_plus = np.zeros((num_parameters, 1))\n",
204+
" J_minus = np.zeros((num_parameters, 1))\n",
205+
" gradapprox = np.zeros((num_parameters, 1))\n",
206+
" \n",
207+
" # Compute gradapprox\n",
208+
" for i in range(num_parameters):\n",
209+
" \n",
210+
" # Compute J_plus[i]. Inputs: \"parameters_values, epsilon\". Output = \"J_plus[i]\".\n",
211+
" # \"_\" is used because the function you have to outputs two parameters but we only care about the first one\n",
212+
" \n",
213+
" thetaplus = np.copy(parameters_values) # Step 1\n",
214+
" thetaplus[i][0] += epsilon # Step 2\n",
215+
" J_plus[i], _ = forward_propagation_n(X, Y, vector_to_dictionary(thetaplus)) # Step 3\n",
216+
" \n",
217+
" \n",
218+
" # Compute J_minus[i]. Inputs: \"parameters_values, epsilon\". Output = \"J_minus[i]\".\n",
219+
" \n",
220+
" thetaminus = np.copy(parameters_values) # Step 1\n",
221+
" thetaminus[i][0] -= epsilon # Step 2 \n",
222+
" J_minus[i], _ = forward_propagation_n(X, Y, vector_to_dictionary(thetaminus)) # Step 3\n",
223+
" \n",
224+
" \n",
225+
" # Compute gradapprox[i]\n",
226+
" \n",
227+
" gradapprox[i] = (J_plus[i]-J_minus[i])/(2*epsilon)\n",
228+
" \n",
229+
" \n",
230+
" # Compare gradapprox to backward propagation gradients by computing difference.\n",
231+
" \n",
232+
" numerator = np.linalg.norm(grad-gradapprox) # Step 1'\n",
233+
" denominator = np.linalg.norm(grad) + np.linalg.norm(gradapprox) # Step 2'\n",
234+
" difference = numerator/denominator # Step 3'\n",
235+
" \n",
236+
"\n",
237+
" if difference > 2e-7:\n",
238+
" print (\"\\033[93m\" + \"There is a mistake in the backward propagation! difference = \" + str(difference) + \"\\033[0m\")\n",
239+
" else:\n",
240+
" print (\"\\033[92m\" + \"Your backward propagation works perfectly fine! difference = \" + str(difference) + \"\\033[0m\")\n",
241+
" \n",
242+
" return difference"
243+
]
244+
},
245+
{
246+
"cell_type": "code",
247+
"execution_count": 32,
248+
"metadata": {
249+
"scrolled": false
250+
},
251+
"outputs": [
252+
{
253+
"name": "stdout",
254+
"output_type": "stream",
255+
"text": [
256+
"\u001b[93mThere is a mistake in the backward propagation! difference = 0.285093156781\u001b[0m\n"
257+
]
258+
}
259+
],
260+
"source": [
261+
"X, Y, parameters = gradient_check_n_test_case()\n",
262+
"\n",
263+
"cost, cache = forward_propagation_n(X, Y, parameters)\n",
264+
"gradients = backward_propagation_n(X, Y, cache)\n",
265+
"difference = gradient_check_n(parameters, gradients, X, Y)"
266+
]
267+
},
268+
{
269+
"cell_type": "markdown",
270+
"metadata": {},
271+
"source": [
272+
"**Expected output**:\n",
273+
"\n",
274+
"<table>\n",
275+
" <tr>\n",
276+
" <td> ** There is a mistake in the backward propagation!** </td>\n",
277+
" <td> difference = 0.285093156781 </td>\n",
278+
" </tr>\n",
279+
"</table>"
280+
]
281+
},
282+
{
283+
"cell_type": "markdown",
284+
"metadata": {},
285+
"source": [
286+
"It seems that there were errors in the `backward_propagation_n` code we gave you! Good that you've implemented the gradient check. Go back to `backward_propagation` and try to find/correct the errors *(Hint: check dW2 and db1)*. Rerun the gradient check when you think you've fixed it. Remember you'll need to re-execute the cell defining `backward_propagation_n()` if you modify the code. \n",
287+
"\n",
288+
"Can you get gradient check to declare your derivative computation correct? Even though this part of the assignment isn't graded, we strongly urge you to try to find the bug and re-run gradient check until you're convinced backprop is now correctly implemented. \n",
289+
"\n",
290+
"**Note** \n",
291+
"- Gradient Checking is slow! Approximating the gradient with $\\frac{\\partial J}{\\partial \\theta} \\approx \\frac{J(\\theta + \\varepsilon) - J(\\theta - \\varepsilon)}{2 \\varepsilon}$ is computationally costly. For this reason, we don't run gradient checking at every iteration during training. Just a few times to check if the gradient is correct. \n",
292+
"- Gradient Checking, at least as we've presented it, doesn't work with dropout. You would usually run the gradient check algorithm without dropout to make sure your backprop is correct, then add dropout. "
293+
]
294+
},
295+
{
296+
"cell_type": "code",
297+
"execution_count": null,
298+
"metadata": {
299+
"collapsed": true
300+
},
301+
"outputs": [],
302+
"source": []
303+
}
304+
],
305+
"metadata": {
306+
"coursera": {
307+
"course_slug": "deep-neural-network",
308+
"graded_item_id": "n6NBD",
309+
"launcher_item_id": "yfOsE"
310+
},
311+
"kernelspec": {
312+
"display_name": "Python 3",
313+
"language": "python",
314+
"name": "python3"
315+
},
316+
"language_info": {
317+
"codemirror_mode": {
318+
"name": "ipython",
319+
"version": 3
320+
},
321+
"file_extension": ".py",
322+
"mimetype": "text/x-python",
323+
"name": "python",
324+
"nbconvert_exporter": "python",
325+
"pygments_lexer": "ipython3",
326+
"version": "3.7.1"
327+
}
328+
},
329+
"nbformat": 4,
330+
"nbformat_minor": 1
331+
}

‎Gradient checking/Gradient Checking.ipynb

Lines changed: 298 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,298 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {},
6+
"source": [
7+
"# Gradient Checking"
8+
]
9+
},
10+
{
11+
"cell_type": "code",
12+
"execution_count": 10,
13+
"metadata": {
14+
"collapsed": true
15+
},
16+
"outputs": [],
17+
"source": [
18+
"# Packages\n",
19+
"import numpy as np\n",
20+
"from test import *\n",
21+
"from Supporting_func import sigmoid, relu, dictionary_to_vector, vector_to_dictionary, gradients_to_vector"
22+
]
23+
},
24+
{
25+
"cell_type": "markdown",
26+
"metadata": {},
27+
"source": [
28+
"## N-dimensional gradient checking"
29+
]
30+
},
31+
{
32+
"cell_type": "markdown",
33+
"metadata": {
34+
"collapsed": true
35+
},
36+
"source": [
37+
"The following figure describes the forward and backward propagation of your fraud detection model.\n",
38+
"\n",
39+
"<img src=\"images/NDgrad_kiank.png\" style=\"width:600px;height:400px;\">\n",
40+
"<caption><center> <u> **Figure** </u><br>*LINEAR -> RELU -> LINEAR -> RELU -> LINEAR -> SIGMOID*</center></caption>"
41+
]
42+
},
43+
{
44+
"cell_type": "code",
45+
"execution_count": 17,
46+
"metadata": {
47+
"collapsed": true
48+
},
49+
"outputs": [],
50+
"source": [
51+
"def forward_propagation_n(X, Y, parameters):\n",
52+
" \"\"\"\n",
53+
" Implements the forward propagation (and computes the cost) presented in above Figure.\n",
54+
" \n",
55+
" Arguments:\n",
56+
" X -- training set for m examples\n",
57+
" Y -- labels for m examples \n",
58+
" parameters -- python dictionary containing your parameters \"W1\", \"b1\", \"W2\", \"b2\", \"W3\", \"b3\":\n",
59+
" W1 -- weight matrix of shape (5, 4)\n",
60+
" b1 -- bias vector of shape (5, 1)\n",
61+
" W2 -- weight matrix of shape (3, 5)\n",
62+
" b2 -- bias vector of shape (3, 1)\n",
63+
" W3 -- weight matrix of shape (1, 3)\n",
64+
" b3 -- bias vector of shape (1, 1)\n",
65+
" \n",
66+
" Returns:\n",
67+
" cost -- the cost function (logistic cost for one example)\n",
68+
" \"\"\"\n",
69+
" \n",
70+
" # retrieve parameters\n",
71+
" m = X.shape[1]\n",
72+
" W1 = parameters[\"W1\"]\n",
73+
" b1 = parameters[\"b1\"]\n",
74+
" W2 = parameters[\"W2\"]\n",
75+
" b2 = parameters[\"b2\"]\n",
76+
" W3 = parameters[\"W3\"]\n",
77+
" b3 = parameters[\"b3\"]\n",
78+
"\n",
79+
" # LINEAR -> RELU -> LINEAR -> RELU -> LINEAR -> SIGMOID\n",
80+
" Z1 = np.dot(W1, X) + b1\n",
81+
" A1 = relu(Z1)\n",
82+
" Z2 = np.dot(W2, A1) + b2\n",
83+
" A2 = relu(Z2)\n",
84+
" Z3 = np.dot(W3, A2) + b3\n",
85+
" A3 = sigmoid(Z3)\n",
86+
"\n",
87+
" # Cost\n",
88+
" logprobs = np.multiply(-np.log(A3),Y) + np.multiply(-np.log(1 - A3), 1 - Y)\n",
89+
" cost = 1./m * np.sum(logprobs)\n",
90+
" \n",
91+
" cache = (Z1, A1, W1, b1, Z2, A2, W2, b2, Z3, A3, W3, b3)\n",
92+
" \n",
93+
" return cost, cache"
94+
]
95+
},
96+
{
97+
"cell_type": "code",
98+
"execution_count": 30,
99+
"metadata": {
100+
"collapsed": true
101+
},
102+
"outputs": [],
103+
"source": [
104+
"def backward_propagation_n(X, Y, cache):\n",
105+
" \"\"\"\n",
106+
" Implement the backward propagation presented in above figure.\n",
107+
" \n",
108+
" Arguments:\n",
109+
" X -- input datapoint, of shape (input size, 1)\n",
110+
" Y -- true \"label\"\n",
111+
" cache -- cache output from forward_propagation_n()\n",
112+
" \n",
113+
" Returns:\n",
114+
" gradients -- A dictionary with the gradients of the cost with respect to each parameter, activation and pre-activation variables.\n",
115+
" \"\"\"\n",
116+
" \n",
117+
" m = X.shape[1]\n",
118+
" (Z1, A1, W1, b1, Z2, A2, W2, b2, Z3, A3, W3, b3) = cache\n",
119+
" \n",
120+
" dZ3 = A3 - Y\n",
121+
" dW3 = 1./m * np.dot(dZ3, A2.T)\n",
122+
" db3 = 1./m * np.sum(dZ3, axis=1, keepdims = True)\n",
123+
" \n",
124+
" dA2 = np.dot(W3.T, dZ3)\n",
125+
" dZ2 = np.multiply(dA2, np.int64(A2 > 0))\n",
126+
" dW2 = 1./m * np.dot(dZ2, A1.T)*2\n",
127+
" db2 = 1./m * np.sum(dZ2, axis=1, keepdims = True)\n",
128+
" \n",
129+
" dA1 = np.dot(W2.T, dZ2)\n",
130+
" dZ1 = np.multiply(dA1, np.int64(A1 > 0))\n",
131+
" dW1 = 1./m * np.dot(dZ1, X.T)\n",
132+
" db1 = 4./m * np.sum(dZ1, axis=1, keepdims = True)\n",
133+
" \n",
134+
" gradients = {\"dZ3\": dZ3, \"dW3\": dW3, \"db3\": db3,\n",
135+
" \"dA2\": dA2, \"dZ2\": dZ2, \"dW2\": dW2, \"db2\": db2,\n",
136+
" \"dA1\": dA1, \"dZ1\": dZ1, \"dW1\": dW1, \"db1\": db1}\n",
137+
" \n",
138+
" return gradients"
139+
]
140+
},
141+
{
142+
"cell_type": "markdown",
143+
"metadata": {},
144+
"source": [
145+
"**How does gradient checking work?**.\n",
146+
"\n",
147+
"As you want to compare \"gradapprox\" to the gradient computed by backpropagation. The formula is still:\n",
148+
"\n",
149+
"$$ \\frac{\\partial J}{\\partial \\theta} = \\lim_{\\varepsilon \\to 0} \\frac{J(\\theta + \\varepsilon) - J(\\theta - \\varepsilon)}{2 \\varepsilon} \\tag{1}$$\n",
150+
"\n",
151+
"However, $\\theta$ is not a scalar anymore. It is a dictionary called \"parameters\".\n",
152+
"I have also converted the \"gradients\" dictionary into a vector \"grad\" using gradients_to_vector().\n",
153+
"\n",
154+
"Here is pseudo-code that will help to implement the gradient check.\n",
155+
"\n",
156+
"For each i in num_parameters:\n",
157+
"- To compute `J_plus[i]`:\n",
158+
" 1. Set $\\theta^{+}$ to `np.copy(parameters_values)`\n",
159+
" 2. Set $\\theta^{+}_i$ to $\\theta^{+}_i + \\varepsilon$\n",
160+
" 3. Calculate $J^{+}_i$ using to `forward_propagation_n(x, y, vector_to_dictionary(`$\\theta^{+}$ `))`. \n",
161+
"- To compute `J_minus[i]`: do the same thing with $\\theta^{-}$\n",
162+
"- Compute $gradapprox[i] = \\frac{J^{+}_i - J^{-}_i}{2 \\varepsilon}$\n",
163+
"\n",
164+
"Thus, you get a vector gradapprox, where gradapprox[i] is an approximation of the gradient with respect to `parameter_values[i]`. You can now compare this gradapprox vector to the gradients vector from backpropagation. Just like for the 1D case (Steps 1', 2', 3'), compute: \n",
165+
"$$ difference = \\frac {\\| grad - gradapprox \\|_2}{\\| grad \\|_2 + \\| gradapprox \\|_2 } \\tag{3}$$"
166+
]
167+
},
168+
{
169+
"cell_type": "code",
170+
"execution_count": 31,
171+
"metadata": {
172+
"collapsed": true
173+
},
174+
"outputs": [],
175+
"source": [
176+
"def gradient_check_n(parameters, gradients, X, Y, epsilon = 1e-7):\n",
177+
" \"\"\"\n",
178+
" Checks if backward_propagation_n computes correctly the gradient of the cost output by forward_propagation_n\n",
179+
" \n",
180+
" Arguments:\n",
181+
" parameters -- python dictionary containing your parameters \"W1\", \"b1\", \"W2\", \"b2\", \"W3\", \"b3\":\n",
182+
" grad -- output of backward_propagation_n, contains gradients of the cost with respect to the parameters. \n",
183+
" x -- input datapoint, of shape (input size, 1)\n",
184+
" y -- true \"label\"\n",
185+
" epsilon -- tiny shift to the input to compute approximated gradient with formula(1)\n",
186+
" \n",
187+
" Returns:\n",
188+
" difference -- difference (2) between the approximated gradient and the backward propagation gradient\n",
189+
" \"\"\"\n",
190+
" \n",
191+
" # Set-up variables\n",
192+
" parameters_values, _ = dictionary_to_vector(parameters)\n",
193+
" grad = gradients_to_vector(gradients)\n",
194+
" num_parameters = parameters_values.shape[0]\n",
195+
" J_plus = np.zeros((num_parameters, 1))\n",
196+
" J_minus = np.zeros((num_parameters, 1))\n",
197+
" gradapprox = np.zeros((num_parameters, 1))\n",
198+
" \n",
199+
" # Compute gradapprox\n",
200+
" for i in range(num_parameters):\n",
201+
" \n",
202+
" # Compute J_plus[i]. Inputs: \"parameters_values, epsilon\". Output = \"J_plus[i]\".\n",
203+
" # \"_\" is used because the function you have to outputs two parameters but we only care about the first one\n",
204+
" \n",
205+
" thetaplus = np.copy(parameters_values) # Step 1\n",
206+
" thetaplus[i][0] += epsilon # Step 2\n",
207+
" J_plus[i], _ = forward_propagation_n(X, Y, vector_to_dictionary(thetaplus)) # Step 3\n",
208+
" \n",
209+
" \n",
210+
" # Compute J_minus[i]. Inputs: \"parameters_values, epsilon\". Output = \"J_minus[i]\".\n",
211+
" \n",
212+
" thetaminus = np.copy(parameters_values) # Step 1\n",
213+
" thetaminus[i][0] -= epsilon # Step 2 \n",
214+
" J_minus[i], _ = forward_propagation_n(X, Y, vector_to_dictionary(thetaminus)) # Step 3\n",
215+
" \n",
216+
" \n",
217+
" # Compute gradapprox[i]\n",
218+
" \n",
219+
" gradapprox[i] = (J_plus[i]-J_minus[i])/(2*epsilon)\n",
220+
" \n",
221+
" \n",
222+
" # Compare gradapprox to backward propagation gradients by computing difference.\n",
223+
" \n",
224+
" numerator = np.linalg.norm(grad-gradapprox) # Step 1'\n",
225+
" denominator = np.linalg.norm(grad) + np.linalg.norm(gradapprox) # Step 2'\n",
226+
" difference = numerator/denominator # Step 3'\n",
227+
" \n",
228+
"\n",
229+
" if difference > 2e-7:\n",
230+
" print (\"\\033[93m\" + \"There is a mistake in the backward propagation! difference = \" + str(difference) + \"\\033[0m\")\n",
231+
" else:\n",
232+
" print (\"\\033[92m\" + \"Your backward propagation works perfectly fine! difference = \" + str(difference) + \"\\033[0m\")\n",
233+
" \n",
234+
" return difference"
235+
]
236+
},
237+
{
238+
"cell_type": "code",
239+
"execution_count": 32,
240+
"metadata": {
241+
"scrolled": false
242+
},
243+
"outputs": [
244+
{
245+
"name": "stdout",
246+
"output_type": "stream",
247+
"text": [
248+
"\u001b[93mThere is a mistake in the backward propagation! difference = 0.285093156781\u001b[0m\n"
249+
]
250+
}
251+
],
252+
"source": [
253+
"X, Y, parameters = gradient_check_n_test_case()\n",
254+
"\n",
255+
"cost, cache = forward_propagation_n(X, Y, parameters)\n",
256+
"gradients = backward_propagation_n(X, Y, cache)\n",
257+
"difference = gradient_check_n(parameters, gradients, X, Y)"
258+
]
259+
},
260+
{
261+
"cell_type": "markdown",
262+
"metadata": {},
263+
"source": [
264+
"It seems that there were errors in the `backward_propagation_n` code we gave you! Good that you've implemented the gradient check. Go back to `backward_propagation` and try to find/correct the errors *(Hint: check dW2 and db1)*. Rerun the gradient check when you think you've fixed it. Remember you'll need to re-execute the cell defining `backward_propagation_n()` if you modify the code. \n",
265+
"\n",
266+
"**Note** \n",
267+
"- Gradient Checking is slow! Approximating the gradient with $\\frac{\\partial J}{\\partial \\theta} \\approx \\frac{J(\\theta + \\varepsilon) - J(\\theta - \\varepsilon)}{2 \\varepsilon}$ is computationally costly. For this reason, we don't run gradient checking at every iteration during training. Just a few times to check if the gradient is correct. \n",
268+
"- Gradient Checking, at least as we've presented it, doesn't work with dropout. You would usually run the gradient check algorithm without dropout to make sure your backprop is correct, then add dropout. "
269+
]
270+
}
271+
],
272+
"metadata": {
273+
"coursera": {
274+
"course_slug": "deep-neural-network",
275+
"graded_item_id": "n6NBD",
276+
"launcher_item_id": "yfOsE"
277+
},
278+
"kernelspec": {
279+
"display_name": "Python 3",
280+
"language": "python",
281+
"name": "python3"
282+
},
283+
"language_info": {
284+
"codemirror_mode": {
285+
"name": "ipython",
286+
"version": 3
287+
},
288+
"file_extension": ".py",
289+
"mimetype": "text/x-python",
290+
"name": "python",
291+
"nbconvert_exporter": "python",
292+
"pygments_lexer": "ipython3",
293+
"version": "3.7.1"
294+
}
295+
},
296+
"nbformat": 4,
297+
"nbformat_minor": 1
298+
}

‎Gradient checking/Supporting_func.py

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
import numpy as np
2+
3+
def sigmoid(x):
4+
"""
5+
Compute the sigmoid of x
6+
7+
Arguments:
8+
x -- A scalar or numpy array of any size.
9+
10+
Return:
11+
s -- sigmoid(x)
12+
"""
13+
s = 1/(1+np.exp(-x))
14+
return s
15+
16+
def relu(x):
17+
"""
18+
Compute the relu of x
19+
20+
Arguments:
21+
x -- A scalar or numpy array of any size.
22+
23+
Return:
24+
s -- relu(x)
25+
"""
26+
s = np.maximum(0,x)
27+
28+
return s
29+
30+
def dictionary_to_vector(parameters):
31+
"""
32+
Roll all our parameters dictionary into a single vector satisfying our specific required shape.
33+
"""
34+
keys = []
35+
count = 0
36+
for key in ["W1", "b1", "W2", "b2", "W3", "b3"]:
37+
38+
# flatten parameter
39+
new_vector = np.reshape(parameters[key], (-1,1))
40+
keys = keys + [key]*new_vector.shape[0]
41+
42+
if count == 0:
43+
theta = new_vector
44+
else:
45+
theta = np.concatenate((theta, new_vector), axis=0)
46+
count = count + 1
47+
48+
return theta, keys
49+
50+
def vector_to_dictionary(theta):
51+
"""
52+
Unroll all our parameters dictionary from a single vector satisfying our specific required shape.
53+
"""
54+
parameters = {}
55+
parameters["W1"] = theta[:20].reshape((5,4))
56+
parameters["b1"] = theta[20:25].reshape((5,1))
57+
parameters["W2"] = theta[25:40].reshape((3,5))
58+
parameters["b2"] = theta[40:43].reshape((3,1))
59+
parameters["W3"] = theta[43:46].reshape((1,3))
60+
parameters["b3"] = theta[46:47].reshape((1,1))
61+
62+
return parameters
63+
64+
def gradients_to_vector(gradients):
65+
"""
66+
Roll all our gradients dictionary into a single vector satisfying our specific required shape.
67+
"""
68+
69+
count = 0
70+
for key in ["dW1", "db1", "dW2", "db2", "dW3", "db3"]:
71+
# flatten parameter
72+
new_vector = np.reshape(gradients[key], (-1,1))
73+
74+
if count == 0:
75+
theta = new_vector
76+
else:
77+
theta = np.concatenate((theta, new_vector), axis=0)
78+
count = count + 1
79+
80+
return theta

‎Gradient checking/images/NDgrad_kiank.png

176 KB
Loading

‎Gradient checking/images/dictionary_to_vector.png

122 KB
Loading

‎Gradient checking/test.py

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
import numpy as np
2+
3+
def gradient_check_n_test_case():
4+
np.random.seed(1)
5+
x = np.random.randn(4,3)
6+
y = np.array([1, 1, 0])
7+
W1 = np.random.randn(5,4)
8+
b1 = np.random.randn(5,1)
9+
W2 = np.random.randn(3,5)
10+
b2 = np.random.randn(3,1)
11+
W3 = np.random.randn(1,3)
12+
b3 = np.random.randn(1,1)
13+
parameters = {"W1": W1,
14+
"b1": b1,
15+
"W2": W2,
16+
"b2": b2,
17+
"W3": W3,
18+
"b3": b3}
19+
20+
21+
return x, y, parameters

0 commit comments

Comments
 (0)
Please sign in to comment.