As previously stated, in linear regression, we try to find a linear equation that minimizes the distance between data points and the modeled line.

This relationship can be represented by this canonical linear function:

The model function takes the form:

Here, ss0  or  bias is the intercept, the function value for x is zero, and ss1 is the slope of the modeled line. The variable x is normally called the independent variable, and y the dependent one, but they can also be called the regressor and response variables respectively.

In order to calculate the least squares error for a function, we look for a measure of how close the points are to the modeling line, in a general sense. So we define a function that measures for each tuple x_n, y_n how far it is from the modeled line's corresponding value.

For 2D regression, we have a list of number tuples (X₀,Y₀),(X₁,Y₁)...(X_n,Y_n), and the values to find are of β₀ and β₁, by minimizing the following function: 

In simple terms, the summation represents the sum of Euclidean distances between predicted and actual values.

The reason for the operations is that the summation of the squared errors gives us a unique and simple global number, the difference between expected and real number gives us the proper distance, and the square power gives us a positive number, which penalizes distances in a more-than-linear fashion.

The kind of problem we are trying to solve (least squares) can be presented in matrix form:

Here, J is the cost function and has the following solution:

In this chapter, we will use an iterative method gradient descent, which will be useful in the following chapters in a more generalized fashion.

The gradient descent is by its own nature an iterative method and the most generally used optimization algorithm in the machine learning field. It combines a simple approach with a good convergence rate, considering the complexity of parameter combinations that it can be optimized with it.

A 2D linear regression starts with a function with randomly defined weights or multipliers for the linear coefficient. After the first values are defined, the second step is to apply a recurrent function in the following form:

In this equation, we can easily derive the mechanism of the method. We start with the initial set of coefficients and then move in the opposite direction of maximum change in the function. The α variable is named the step and will affect how far we will move in the direction of the gradient searching for minimal.

The final step is to optionally test the changes between iteration and see whether the changes are greater than an epsilon or to check whether the iteration number is reached.

If the function is not convex, it is suggested to run gradient descent multiple times with random values and then select coefficients for which the cost value is the lowest. In the case of non-convex functions, the gradient descent ends up in a minimum, which can be local. Therefore, as for non-convex functions, the result depends on initial values, it is suggested to randomly set them multiple times and among all the solutions, pick the one with lowest cost.

The training or parameter optimization stage is a vital part of the machine learning workflow.

For this matter, TensorFlow has a tf.train module, which is a helper set of objects dedicated to implementing a range of different optimizing strategies that the data scientist will need. The main object provided by this module is called Optimizer.

tf.train.Optimizer.minimize(loss, global_step=None, var_list=None, gate_gradients=1, aggregation_method=None, colocate_gradients_with_ops=False, name=None) 
 

For this example, we will be generating a synthetic dataset consisting of a linear function with added noise:

import TensorFlow as tf
import numpy as np
trX = np.linspace(-1, 1, 101)
trY = 2 * trX + np.random.randn(*trX.shape) * 0.4 + 0.2 # create a y value which is approximately linear but with some random noise

With these lines, we can represent the lines as a scatter plot and the ideal line function.

import matplotlib.pyplot as plt 
plt.scatter(trX,trY) 
plt.plot (trX, .2 + 2 * trX)  

Generated samples and original linear function without noise

In the dashboard, you can see the image of the loss function we have been recollecting. In the graph section, when you zoom into the Model, you can see the sum and multiplication operation, the parameter variables b0 and b1, and the gradient operation applied over the Model, as shown next:

The resulting plot is as follows; we can see how the initial line rapidly converges into a more plausible result:

With the CostFunction scope zoomed in, we can see the power and subtraction operations and also the written summary, as shown in the following figure:

Now let's check the parameter results, printing the run output of the w and b variables:

printsess.run(w) # Should be around 2  
printsess.run(b) #Should be around 0.2 
2.09422 
0.256044 

It's time to graphically review the data again and the suggested final line.

plt.scatter(trX,trY) 
plt.plot (trX, testb + trX * testw) 

Now let's review the data we saved in TensorBoard.

In order to start TensorBoard, you can go to the logs directory and execute the following line:

$ tensorboard --logdir=. 

TensorBoard will load the event and graph files and will be listening on the 6006 port. You can then go from your browser to localhost:6000 and see the TensorBoard dashboard as shown in the following figure:

The following is the complete source code:

import matplotlib.pyplot as plt # import matplotlib 
import numpy as np # import numpy 
import tensorflow as tf 
import numpy as np 
 
trX = np.linspace(-1, 1, 101) #Create a linear space of 101 points between 1 and 1 
trY = 2 * trX + np.random.randn(*trX.shape) * 0.4 + 0.2 #Create The y function based on the x axis 
plt.figure() # Create a new figure 
plt.scatter(trX,trY) #Plot a scatter draw of the random datapoints 
plt.plot (trX, .2 + 2 * trX) # Draw one line with the line function 
 
get_ipython().magic(u'matplotlib inline') 
 
import matplotlib.pyplot as plt 
import tensorflow as tf 
import numpy as np 
 
trX = np.linspace(-1, 1, 101) 
trY = 2 * trX + np.random.randn(*trX.shape) * 0.4 + 0.2 # create a y value which is approximately linear but with some random noise 
 
plt.scatter(trX,trY) 
plt.plot (trX, .2 + 2 * trX) 
 
X = tf.placeholder("float", name="X") # create symbolic variables 
Y = tf.placeholder("float", name = "Y") 
 
withtf.name_scope("Model"): 
 
    def model(X, w, b): 
        returntf.mul(X, w) + b # We just define the line as X*w + b0   
 
    w = tf.Variable(-1.0, name="b0") # create a shared variable 
    b = tf.Variable(-2.0, name="b1") # create a shared variable 
    y_model = model(X, w, b) 
 
 
withtf.name_scope("CostFunction"): 
    cost = (tf.pow(Y-y_model, 2)) # use sqr error for cost function 
 
train_op = tf.train.GradientDescentOptimizer(0.05).minimize(cost) 
 
 
sess = tf.Session() 
init = tf.initialize_all_variables() 
tf.train.write_graph(sess.graph, '/home/ubuntu/linear','graph.pbtxt') 
cost_op = tf.scalar_summary("loss", cost) 
merged = tf.merge_all_summaries() 
sess.run(init) 
writer = tf.train.SummaryWriter('/home/ubuntu/linear', sess.graph) 
 
fori in range(100): 
for (x, y) in zip(trX, trY): 
sess.run(train_op, feed_dict={X: x, Y: y})     
summary_str = sess.run(cost_op, feed_dict={X: x, Y: y}) 
writer.add_summary(summary_str, i)        
    b0temp=b.eval(session=sess) 
    b1temp=w.eval(session=sess) 
plt.plot (trX, b0temp + b1temp * trX ) 
 
 
printsess.run(w) # Should be around 2  
printsess.run(b) #Should be around 0.2 
 
 
plt.scatter(trX,trY) 
plt.plot (trX, sess.run(b) + trX * sess.run(w)) 
 

This section contains a list of useful libraries that we will be using in this example and in some parts of the rest of the book, outside TensorFlow, to assist the solving of different problems we will be working on.

The dataset is represented in a CSV file, and we will open it using the Pandas library.

The dataset includes the following variables:

Here, we have a simple program that will read the dataset and create a detailed account of the data:

import tensorflow.contrib.learn as skflow 
fromsklearn import datasets, metrics, preprocessing 
import numpy as np 
import pandas as pd 
 
df = pd.read_csv("data/boston.csv", header=0) 
printdf.describe() 

This will output a statistical summary of the dataset's variable. The first six results are as follows:

 
CRIM         ZN       INDUS         CHAS         NOX          RM  \ 
count  506.000000  506.000000  506.000000  506.000000  506.000000  506.000000    
mean     3.613524   11.363636   11.136779    0.069170    0.554695    6.284634    
std      8.601545   23.322453    6.860353    0.253994    0.115878    0.702617    
min      0.006320    0.000000    0.460000    0.000000    0.385000    3.561000    
25%      0.082045    0.000000    5.190000    0.000000    0.449000    5.885500   
50%      0.256510    0.000000    9.690000    0.000000    0.538000    6.208500    
75%      3.677082   12.500000   18.100000    0.000000    0.624000    6.623500    
max     88.976200  100.000000   27.740000    1.000000    0.871000    8.780000    
 

The model we will employ in this example is simple but has almost all the elements that we will need to tackle a more complex one.

In the following diagram, we see the different actors of the whole setup: a model, the CostFunction, and the gradients. A really useful feature of TensorFlow is the ability to automatically differentiate between the Model and functions.

Here, we can find the definition of the variables represented in the preceding section: w, b, and the model linear equation.

X = tf.placeholder("float", name="X") # create symbolic variables 
Y = tf.placeholder("float", name = "Y") 
 
withtf.name_scope("Model"): 
    w = tf.Variable(tf.random_normal([2], stddev=0.01), name="b0") # create a shared variable 
    b = tf.Variable(tf.random_normal([2], stddev=0.01), name="b1") # create a shared variable 
def model(X, w, b): 
returntf.mul(X, w) + b # We just define the line as X*w + b0   
y_model = model(X, w, b)

In this example, we will use the commonly employed mean squared error, but this time with a multivariable; so we apply reduce_mean to collect error values across the different dimensions:

withtf.name_scope("CostFunction"): 
    cost = tf.reduce_mean(tf.pow(Y-y_model, 2)) # use sqr error for cost function 
train_op = tf.train.AdamOptimizer(0.1).minimize(cost)

 for a in range (1,10): 
    cost1=0.0 
fori, j in zip(xvalues, yvalues):    
sess.run(train_op, feed_dict={X: i, Y: j})  
        cost1+=sess.run(cost, feed_dict={X: i, Y: i})/506.00 
        #writer.add_summary(summary_str, i)  
xvalues, yvalues = shuffle (xvalues, yvalues) 
 

The stop condition will simply consist of training the parameters with all data samples for the number of epochs determined in the outer loop.

The following is the result:

1580.53295174 
[ 2.25225258  1.30112672] 
[ 0.80297691  0.22137061] 
1512.3965525 
[ 4.62365675  2.90244412] 
[ 1.16225874  0.28009811] 
1495.47174799 
[ 6.52791834  4.29297304] 
[ 0.824792270.17988272] 
... 
1684.6247849 
[ 29.71323776  29.96078873] 
[-0.68271929 -0.13493828] 
1688.25864746 
[ 29.78564262  30.09841156] 
[-0.58272243 -0.08323665] 
1684.27538102 
[ 29.75390816  30.13044167] 
[-0.59861398 -0.11895057] 
 

From the results, we see that in the final stage of the training, the modeling lines settle on the following coefficients simultaneously:

price = 0.6 x Industry + 29.75

price = 0.1 x Age + 30.13

The following is the complete source code:

import matplotlib.pyplot as plt 
import tensorflow as tf 
import tensorflow.contrib.learn as skflow 
from sklearn.utils import shuffle 
import numpy as np 
import pandas as pd 
 
df = pd.read_csv("data/boston.csv", header=0) 
printdf.describe() 
 
f, ax1 = plt.subplots() 
plt.figure() # Create a new figure 
 
y = df['MEDV'] 
 
for i in range (1,8): 
    number = 420 + i 
    ax1.locator_params(nbins=3) 
    ax1 = plt.subplot(number) 
    plt.title(list(df)[i]) 
    ax1.scatter(df[df.columns[i]],y) #Plot a scatter draw of the datapoints 
plt.tight_layout(pad=0.4, w_pad=0.5, h_pad=1.0) 
 
 
X = tf.placeholder("float", name="X") # create symbolic variables 
Y = tf.placeholder("float", name = "Y") 
 
 
with tf.name_scope("Model"): 
 
    w = tf.Variable(tf.random_normal([2], stddev=0.01), name="b0") # create a shared variable 
    b = tf.Variable(tf.random_normal([2], stddev=0.01), name="b1") # create a shared variable 
 
    def model(X, w, b): 
        return tf.mul(X, w) + b # We just define the line as X*w + b0   
 
    y_model = model(X, w, b) 
 
with tf.name_scope("CostFunction"): 
    cost = tf.reduce_mean(tf.pow(Y-y_model, 2)) # use sqr error for cost function 
 
train_op = tf.train.AdamOptimizer(0.001).minimize(cost) 
 
 
sess = tf.Session() 
init = tf.initialize_all_variables() 
tf.train.write_graph(sess.graph, '/home/bonnin/linear2','graph.pbtxt') 
cost_op = tf.scalar_summary("loss", cost) 
merged = tf.merge_all_summaries() 
sess.run(init) 
writer = tf.train.SummaryWriter('/home/bonnin/linear2', sess.graph) 
 
xvalues = df[[df.columns[2], df.columns[4]]].values.astype(float) 
yvalues = df[df.columns[12]].values.astype(float) 
b0temp=b.eval(session=sess) 
b1temp=w.eval(session=sess) 
 
 
for a in range (1,10): 
    cost1=0.0 
for i, j in zip(xvalues, yvalues):    
sess.run(train_op, feed_dict={X: i, Y: j})  
        cost1+=sess.run(cost, feed_dict={X: i, Y: i})/506.00 
        #writer.add_summary(summary_str, i)  
xvalues, yvalues = shuffle (xvalues, yvalues) 
print (cost1) 
b0temp=b.eval(session=sess) 
b1temp=w.eval(session=sess) 
print (b0temp) 
print (b1temp) 
#plt.plot (trX, b0temp + b1temp * trX )