• Preliminary concepts of neural networks
• Neural network projects on non-linear synthetic function regression
• Projects on predicting car fuel efficiency with nonlinear regression
• Learning to classify wines andmulticlass classification

To build a simple framework into the neural network components and architectures, we will give a simple and straightforward build of the original concepts which paved the way to the current,complexand variedNeural Network landscape.

In this case, we will be using a generated dataset, which will be very similar to the one in Chapter 3, Linear Regression.

We will generate a quadratic function using the common Numpy methods, and then we will add random noise, which will help us to see how the linear regression can generalize.

The core sample creation routines are as follows:

import numpy as np 
trainsamples = 200 
testsamples = 60 
dsX = np.linspace(-1, 1, trainsamples + testsamples).transpose() 
dsY = 0.4* pow(dsX,2) +2 * dsX + np.random.randn(*dsX.shape) * 0.22 + 0.8  

This dataset doesn't need preprocessing as it is being generated, and has good properties such asbeing centered and having a -1, 1 x sample distribution.

The loss for this setup will simply be represented by the same as the mean squares error, with the line:

cost = tf.pow(py_x-Y, 2)/(2)  

In this particular case we will be using the Gradient Descent cost optimizer, which we can invoke with the line:

train_op = tf.train.AdamOptimizer(0.5).minimize(cost)  

predict_op = tf.argmax(py_x, 1)

cost1 += sess.run(cost, feed_dict={X: [[x1]], Y: y1}) / testsamples 

This is a copy of the different epoch results.Note that as this is a very simple function, even the first iteration has very good results:

Average cost for epoch 1:[[ 0.00753353]]
Average cost for epoch 2:[[ 0.00381996]]
Average cost for epoch 3:[[ 0.00134867]]
Average cost for epoch 4:[[ 0.01020064]]
Average cost for epoch 5:[[ 0.00240157]]
Average cost for epoch 6:[[ 0.01248318]]
Average cost for epoch 7:[[ 0.05143405]]
Average cost for epoch 8:[[ 0.00621457]]
Average cost for epoch 9:[[ 0.0007379]]

For this problem, we will be analyzing a very well-known, standard,well-formeddataset, which will allow us to analyze a multi-variable problem: guessing the mpg an automobile will have based on some related variables, discrete and continuous.

This could be considered a toy and somewhat dated example, but it will pave the way to more complex problems, and has the advantage of being already analyzed by numerous bibliographies.

Attribute Information

This dataset has the following data columns:

We won't be doing a detailed analysis of the data, but we can informally infer that all of the continuous variables have a correlation with increasing or decreasing the goal variable:

For this function, we will be using the previously-describedscaler objects, from sklearn:

What we are about to build is a feedforward neural network, with a multivariate input, and a simple output:

score = metrics.mean_squared_error(regressor.predict(scaler.transform(X_test)), y_test)
print('MSE: {0:f}'.format(score))

Step #99, avg. train loss: 182.33624
Step #199, avg. train loss: 25.09151
Step #300, epoch #1, avg. train loss: 11.92343
Step #400, epoch #1, avg. train loss: 11.20414
Step #500, epoch #1, avg. train loss: 5.14056
Total Mean Squared Error: 15.0792258911

%matplotlib inline  
import matplotlib.pyplot as plt 
import pandas as pd 
 
from sklearn import datasets, cross_validation, metrics 
from sklearn import preprocessing 
from tensorflow.contrib import skflow 
 
# Read the original dataset 
df = pd.read_csv("data/mpg.csv", header=0) 
# Convert the displacement column as float 
df['displacement']=df['displacement'].astype(float) 
# We get data columns from the dataset 
# First and last (mpg and car names) are ignored for X 
X = df[df.columns[1:8]] 
y = df['mpg'] 
 
plt.figure() # Create a new figure 
 
for i in range (1,8): 
    number = 420 + i 
    ax1.locator_params(nbins=3) 
    ax1 = plt.subplot(number) 
    plt.title(list(df)[i]) 
    ax1.scatter(df[df.columns[i]],y) #Plot a scatter draw of the  datapoints 
plt.tight_layout(pad=0.4, w_pad=0.5, h_pad=1.0) 
# Split the datasets 
 
X_train, X_test, y_train, y_test = cross_validation.train_test_split(X, y, 
test_size=0.25) 
 
# Scale the data for convergency optimization 
scaler = preprocessing.StandardScaler() 
 
# Set the transform parameters 
X_train = scaler.fit_transform(X_train) 
 
# Build a 2 layer fully connected DNN with 10 and 5 units respectively 
regressor = skflow.TensorFlowDNNRegressor(hidden_units=[10, 5], 
steps=500, learning_rate=0.051, batch_size=1) 
 
# Fit the regressor 
regressor.fit(X_train, y_train) 
 
# Get some metrics based on the X and Y test data 
score = metrics.mean_squared_error(regressor.predict(scaler.transform(X_test)), y_test) 
 
print(" Total Mean Squared Error: " + str(score)) 

This data contains the results of a chemical analysis of wines grown in the same region in Italy but derived from three different cultivars. The analysis determined the quantities of 13 constituents found in each of the three types of wines.

Data variables:

To read the dataset, we will simply use the provided CSV file and pandas:

df = pd.read_csv("./wine.csv", header=0)

As the values on the csv begin at 1, we will normalize the values resting the bias:

y = df['Wine'].values-1 

For the results, we will have to represent the options as a one hot list of arrays:

Y = tf.one_hot(indices = y, depth=3, on_value = 1., off_value = 0., axis = 1 , name = "a").eval() 

We will also shuffle the values beforehand:

X, Y = shuffle (X, Y) 
scaler = preprocessing.StandardScaler() 
X = scaler.fit_transform(X) 

This particular model will consist of a single layer, fully-connected, neural network:

We will use the cross entropy function to measure loss:

y_ = tf.placeholder(tf.float32, [None, 3]) 
cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y), reduction_indices=[1])) 

Again the Gradient Descent method will be used to reduce the loss function:

train_step = tf.train.GradientDescentOptimizer(0.1).minimize(cross_entropy) 

In the convergence test, we will cast every good regression to 1, and every false one to 0, and then get the mean of the values to measure the accuracy of the model:

correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1)) 
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32)) 
print(accuracy.eval({x: Xt, y_: Yt})) 

As we see, we have a variating accuracy as the epochs progress, but it is always superior to a 90% accuracy, with a 30% random base (if we'd generated a random number between 0 and 3 to guess the result).

0.973684
0.921053
0.921053
0.947368
0.921053

Lets have a look at the complete source code:

sess = tf.InteractiveSession() 
import pandas as pd 
# Import data 
from tensorflow.examples.tlutorials.mnist import input_data 
from sklearn.utils import shuffle 
import tensorflow as tf 
 
from sklearn import preprocessing 
 
flags = tf.app.flags 
FLAGS = flags.FLAGS 
 
df = pd.read_csv("./wine.csv", header=0) 
print (df.describe()) 
#df['displacement']=df['displacement'].astype(float) 
X = df[df.columns[1:13]].values 
y = df['Wine'].values-1 
Y = tf.one_hot(indices = y, depth=3, on_value = 1., off_value = 0., axis = 1 , name = "a").eval() 
X, Y = shuffle (X, Y) 
 
scaler = preprocessing.StandardScaler() 
X = scaler.fit_transform(X) 
 
# Create the model 
x = tf.placeholder(tf.float32, [None, 12]) 
W = tf.Variable(tf.zeros([12, 3])) 
b = tf.Variable(tf.zeros([3])) 
y = tf.nn.softmax(tf.matmul(x, W) + b) 
 
# Define loss and optimizer 
y_ = tf.placeholder(tf.float32, [None, 3]) 
cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y), reduction_indices=[1])) 
train_step = tf.train.GradientDescentOptimizer(0.1).minimize(cross_entropy) 
# Train 
tf.initialize_all_variables().run() 
for i in range(100): 
X,Y =shuffle (X, Y, random_state=1) 
 
Xtr=X[0:140,:] 
Ytr=Y[0:140,:] 
 
Xt=X[140:178,:] 
Yt=Y[140:178,:] 
Xtr, Ytr = shuffle (Xtr, Ytr, random_state=0) 
#batch_xs, batch_ys = mnist.train.next_batch(100) 
batch_xs, batch_ys = Xtr , Ytr 
train_step.run({x: batch_xs, y_: batch_ys}) 
cost = sess.run (cross_entropy, feed_dict={x: batch_xs, y_: batch_ys}) 
# Test trained model 
correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1)) 
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32)) 
print(accuracy.eval({x: Xt, y_: Yt}))