How To Draw Contour Plot With Errror

Linear Regression often is the introductory affiliate of Car Leaning and Gradient Descent probably is the first optimization technique anyone learns. Most of the time, the instructor uses a Contour Plot in order to explain the path of the Gradient Descent optimization algorithm. I used to wonder how to create those Contour plot. Today I will try to show how to visualize Gradient Descent using Contour plot in Python.

Contour Plot:

Contour Plot is similar a 3D surface plot, where the 3rd dimension (Z) gets plotted as constant slices (contour) on a 2 Dimensional surface. The left plot at the picture below shows a 3D plot and the correct one is the Profile plot of the same 3D plot. You can see how the 3rd dimension (Y here) has been converted to contours of colors ( and lines ). The important part is, the value of Y is always same across the contour line for all the values of X1 & X2.

Contour Plot using Python:

Before jumping into gradient descent, lets understand how to actually plot Contour plot using Python. Here we volition be using Python's most pop data visualization library matplotlib.

Data Grooming:

I volition create two vectors ( numpy assortment ) using np.linspace function. I will spread 100 points between -100 and +100 evenly.

import numpy equally np

import matplotlib . pyplot as plt

x1 = np . linspace (-x.0 , 10.0 , 100 )

x2 = np . linspace (-ten.0 , 10.0 , 100 )

If we simply make a scatter plot using x1 and x2, it volition look like following:

plt . scatter ( x1 , x2 )

plt . show ( )

Now, in order to create a contour plot, nosotros will use np.meshgrid to catechumen x1 and x2 from ( 1 X 100 ) vector to ( 100 Ten 100 ) matrix.

np.meshgrid():

Lets looks at what np.meshgrid() actually does. It takes 2 parameters, in this case will pass 2 vectors. So lets create a 1X3 vector and invoke the np.meshgrid() function. By the fashion, it returns 2 matrix back and non just 1.

a=np . array ( ( 1 , 2 , 3 ) )

a1 , a2=np . meshgrid ( a , a )

If you expect at a1 and a2, you will see now they both are 3X3 matrix and a1 has repeated rows and a2 has repeated cols. The np.meshgrid() function, create a filigree of values where each intersection is a combination of ii values.

In social club to understand this visually, if yous look at the 3D plot in the showtime picture, we have now created the bottom aeroplane of that 3D plot, a mesh/grid.

Out [ 11 ] :

array ( [ [ one , 2 , 3 ] ,

[ 1 , ii , 3 ] ,

[ 1 , two , 3 ] ] )

Out [ 12 ] :

array ( [ [ 1 , 1 , 1 ] ,

[ 2 , 2 , 2 ] ,

[ three , iii , 3 ] ] )

One time the mesh/grid values have been created, we can now create the data for the tertiary (virtual) dimension. Hither I am simply using an eclipse part. Y volition also exist a 100 X 100 matrix.

\(
y=x1^2 + x2^2
\)

X1 , X2 = np . meshgrid ( x1 , x2 )

Y = np . sqrt ( np . square ( X1 ) + np . foursquare ( X2 ) )

Earlier even creating a proper profile plot, if we just plot the values of X1 & X2 and choose the color calibration according to the values of Y, we can easily visualize the graph as following:

cm = plt . cm . get_cmap ( 'viridis' )

plt . besprinkle ( X1 , X2 , c=Y , cmap=cm )

plt . show ( )

plt.contour() and plt.contourf():

We will use matplotlib'southward contour() and contourf() office to create the contour plot. We just need to call the function by passing 3 matrix.

cp = plt . contour ( X1 , X2 , Y )

plt . clabel ( cp , inline=1 , fontsize=10 )

plt . xlabel ( 'X1' )

plt . ylabel ( 'X2' )

plt . bear witness ( )

You can run across the besprinkle plot and contour plots looks kind of same. However, nosotros become much more than control which creating the Profile plot over the besprinkle plot.

Fill Profile Plot:

The contourf() function tin can be used to fill up the contour plot. We can likewise change the line style and width. Please refer the matplotlib'due south developer documentation for other available options.

cp = plt . contour ( X1 , X2 , Y , colors='black' , linestyles='dashed' , linewidths=1 )

plt . clabel ( cp , inline=1 , fontsize=10 )

cp = plt . contourf ( X1 , X2 , Y , )

plt . xlabel ( 'X1' )

plt . ylabel ( 'X2' )

plt . show ( )

Cull custom levels:

We volition look at one more important feature of the plotting library. We can ascertain the levels where nosotros want to depict the profile lines using the level or 4th parameter of the both contour() and contourf() office. The below lawmaking sets abiding levels at unlike Y values.

levels = [ 0.0 , 1.0 , two.0 , iv.0 , 8.0 , 12.0 , 14.0 ]

cp = plt . contour ( X1 , X2 , Y , levels , colors='black' , linestyles='dashed' , linewidths=ane )

plt . clabel ( cp , inline=1 , fontsize=ten )

cp = plt . contourf ( X1 , X2 , Y , levels )

plt . xlabel ( 'X1' )

plt . ylabel ( 'X2' )

plt . bear witness ( )

Gradient Descent Algorithm:

Nosotros will be using the Advertising data for our demo here.
We volition load the data first using pandas library
The sales will be the response/target variable
Idiot box and radio volition be the predictors.
Using StandardScaler to normalize the data ( \( \mu = 0 \) and \( \sigma = ane \) )

import pandas as pd

information = pd . read_csv ( 'http://world wide web-bcf.usc.edu/~gareth/ISL/Advertising.csv' )

y = data [ 'sales' ]

X = np . column_stack ( ( information [ 'Television' ] , data [ 'radio' ] ) )

from sklearn . preprocessing import StandardScaler

scaler = StandardScaler ( )

X_scaled = scaler . fit_transform ( X )

Calculate Gradient and MSE:

Using the following function to calculate the mse and derivate w.r.t west

def gradient_descent ( Due west , x , y ) :

y_hat = x . dot ( Due west ) . flatten ( )

mistake = ( y - y_hat )

mse = ( one.0 / len ( x ) ) * np . sum ( np . square ( error ) )

slope = -( one.0 / len ( x ) ) * mistake . dot ( ten )

return gradient , mse

Next, choosing a starting point for w, setting the learning rate hyper-parameter to 0.1 and convergence tolerance to 1e-three

Too, creating two more arrays, 1 for storing all the intermediate west and mse.

w = np . assortment ( (-twoscore , -twoscore ) )

alpha = . ane

tolerance = 1e-3

old_w = [ ]

errors = [ ]

Gradient Descent Loop:

Beneath is the loop for Gradient Descent where we update due west based on the learning rate. Nosotros are also capturing the w and mse values at every 10 iterations.

fifteen

xvi

# Perform Gradient Descent

iterations = i

for i in range ( 200 ) :

gradient , fault = gradient_descent ( w , X_scaled , y )

new_w = w - alpha * slope

# Impress fault every 10 iterations

if iterations % 10 == 0 :

print ( "Iteration: %d - Fault: %.4f" % ( iterations , mistake ) )

old_w . append ( new_w )

errors . append ( fault )

# Stopping Status

if np . sum ( abs ( new_w - westward ) ) < tolerance :

print ( 'Gradient Descent has converged' )

suspension

iterations += 1

due west = new_w

print ( 'w =' , due west )

That's all, you can see that due west is converging at the following values.

Out [ nineteen ] : array ( [ three.91359776 , ii.77964408 ] )

Annotation: Yous can refer my other tutorial on gradient descent, where I have explained the math and program pace by footstep.

Univariate Linear Regression using Octave – Machine Learning Footstep by Pace

Before nosotros start writing the lawmaking for the Contour plot, we need to have care of few things. Convert the listing (old_w) to a numpy array.

Then I am adding 5 additional levels manually merely to make the Contour plot look ameliorate. You lot tin skip them.

Finally, converting the errors list to numpy array, sorting it and saving it as the levels variable. We need to sort the level values from pocket-sized to larger since that the way the profile() role expects.

all_ws = np . array ( old_w )

# Merely for visualization

errors . append ( 600 )

errors . suspend ( 500 )

errors . append ( 400 )

errors . append ( 300 )

errors . suspend ( 225 )

levels = np . sort ( np . array ( errors ) )

Draw the Profile plot:

Its always helpful to run into first before going through the code. Here is the plot of our gradient descent algorithm we will exist creating next.

Set Axis (w0, w1)

As we have done earlier, we demand to create the w0 and w1 (X1 and X2) vector ( 1 X 100). Last time we used the np.linspace() function and randomly choose some values. Here we will use the converged values of due west to create a space around it.

Our w0 array will exist every bit spaced 100 values between -due west[0] * 5 and +w[0] * 5. Same for the w1.

The mse_vals variable is but a placeholder.

w0 = np . linspace (-w [ 0 ] * v , w [ 0 ] * 5 , 100 )

w1 = np . linspace (-w [ 1 ] * v , westward [ one ] * 5 , 100 )

mse_vals = np . zeros ( shape=( w0 . size , w1 . size ) )

Last fourth dimension employ have used the eclipse formula to create the 3rd dimension, however hither demand to manually calculate the mse for each combination of w0 and w1.

Note: There is shortcut available for the beneath lawmaking, however wanted to go along information technology like this way since its easy to see whats going on.

Prepare the tertiary Dimension :

Nosotros will loop through each values of w0 and w1, then summate the msg for each combination. This mode volition be populating our 100 X 100 mse_vals matrix.

This fourth dimension we are non using the meshgrid, however the concept is the aforementioned.

for i , value1 in enumerate ( w0 ) :

for j , value2 in enumerate ( w1 ) :

w_temp = np . array ( ( value1 , value2 ) )

mse_vals [ i , j ] = gradient_descent ( w_temp , X_scaled , y ) [ 1 ]

Final Plot:

Nosotros take w0, w1 and mse_vals (the 3rd dimension), now its pretty easy to create the contour plot similar we saw earlier.

Use the contourf() function first. Pass the levels we created before.
Plot two axis line at w0=0 and w1=1
Call the plt.annotate() part in loops to create the arrow which shows the convergence path of the gradient descent. We will apply the stored w values for this. The mse for those westward values have already been calculated.
Invoke the contour() office for the contour line plot.

plt . contourf ( w0 , w1 , mse_vals , levels , alpha=. 7 )

plt . axhline ( 0 , color='black' , alpha=. 5 , dashes=[ 2 , 4 ] , linewidth=ane )

plt . axvline ( 0 , color='blackness' , alpha=0.5 , dashes=[ two , four ] , linewidth=one )

for i in range ( len ( old_w ) - 1 ) :

plt . comment ( '' , xy=all_ws [ i + 1 , : ] , xytext=all_ws [ i , : ] ,

arrowprops={ 'arrowstyle' : '->' , 'color' : 'r' , 'lw' : 1 } ,

va='center' , ha='centre' )

CS = plt . contour ( w0 , w1 , mse_vals , levels , linewidths=1 , colors='black' )

plt . clabel ( CS , inline=1 , fontsize=viii )

plt . title ( "Contour Plot of Gradient Descent" )

plt . xlabel ( "w0" )

plt . ylabel ( "w1" )

plt . testify ( )

Conclusion:

Observe the mse values are getting reduced from 732 -> 256 -> 205 -> ... etc. Slope Descent has converged easily here.

Contour plot is very useful to visualize complex structure in an easy manner. Afterward we volition employ this same methodology for Ridge and Lasso regression.

I hope this How to visualize Gradient Descent using Profile plot in Python tutorial will assist y'all build much more complex visualization.