x <- rnorm(1000)
y <- -x
data <- data.frame(scale(cbind(x,y)))
library(neuralnet)
model <- neuralnet(y ~ x, hidden=0, data=data)
plot(model)10 Artificial neural networks
Overview
Now we’re going to learn how to predict continuous data using artificial neural networks. These are perhaps the most over-hyped family of models I’m going to teach you, but they do have their uses and they are impressive to talk about with your friends. Try to remember, when using them, that there really is no definitive way to understand how these models work, and thus there is no definitive way (other than by testing their predictive power) to tell whether a particular model specification is the best choice. In my explanation of these models, I will focus on univariate regression; extending these principles to multiple (categorical) response variables is reasonably trivial. I will also explain, without using mathematics, the principles behind how basic artificial neural networks are fit to data, but these principles are not precisely the same as those used in the software you’ll be using in R. The basic principles are, however, the same, and so I wouldn’t worry too much if I were you.
10.0.1 Come with me if you want to live: a gentle introduction to artificial neural networks
Artificial neural networks are so-named because they are intended to mirror the structure of brains. They consist of an input layer where each node (often called neurons, but I prefer to use the term node) represents an explanatory variable, an outer layer of a single layer that represents predictions of the explanatory variables (we will only be working with a single response variable), and, most importantly, one or more hidden layers inbetween that form the basis of the model. If you would like to see a diagram of that, then hold tight: we’ll be plotting them out in the next section.
These hidden layers contain nodes, each of which takes inputs from all of the nodes in previous layer: the input feeding into the first hidden layer, the first hidden layer feeding into a subsequent hidden layer if there is one, and then finally the output layer. Each connection from a node to another node has a weight, which determines how strongly the signal from a node passes to the other node. Each node also has an activation function, which determines how it handles inputs from other nodes.
These activation functions are extremely important, and they’re the only reason that artificial neural networks are in any sense interesting. If the activation function were perfectly linear (i.e., an input of strength 0.5 causes an input to the next node of 0.5) then an artificial neural network is mathematically identical to a standard regression. The weights are also important because they modulate the magnitude and direction (positive/negative) of each node’s effect on each other node.
Fitting artificial neural networks to data is, frankly, a total and complete nightmare. The standard fitting approach is backpropagation: the output of the model is compared with the data (in a so-called feedforward step), and the error is propagated (spread) back through the network, with each node/neuron being altered depending on how much of the error is attributable to it. There’s no magic to how it determines what error is attributable to each neuron: the network knows what the value should have been for each node, so simply finds the difference at each node. The weights are then altered—often not as much as they ‘should be’, since slowing the learning rate in this way yields more stable results—and the whole process is repeated. This is often called a gradient descent approach because, mathematically, it’s like we’re in a valley defined by our current set of coefficients and we’re trying to slide down to the bottom where the best coefficients are.1
Finally, I have literally never known an artificial neural network to work unless all of the input data are normalized, including the response variable. This is because the fitting process requires error to evenly, but if the variables are not directly comparable this is essentially impossible and the whole thing falls apart.
10.0.2 Skynet begins to learn at a geometric rate; your models, not so much
Let’s start by fitting an artificial neural network so simple that it doesn’t even have any hidden layers. In fact, don’t try adding them in, because it could crash R!
What you now see is an artificial neural network—a learning computer2. This one is so simple that it doesn’t have any hidden layers: all you can see is the data going in (\(x\)), the weight that node/neuron has on the output (\(y\)), and the prediction flinging off to the right (the final arrow at the far right). You can also see a blue “1” flying in from the top: this is the function normalizing everything, which it has to do when you’re trying to estimate an intercept and slope. It’s literally just a nuisance term in the mathematics, so don’t give it any thought.
Now let’s simulate some more complicated data, and see how a more reasonably artificial neural network can deals with that.
explanatory <- data.frame(replicate(10, rnorm(400)))
names(explanatory) <- letters[1:10]
response <- with(explanatory, a*2 -0.5*b - i*j + exp(abs(c)))
data <- data.frame(scale(cbind(explanatory,response)))Hopefully, you will agree with me that this is a pretty complicated problem. We’ve got two standard linear terms, an interaction, and a weird exponent of a non-negative number. So how is it going to perform?
training <- sample(nrow(data), nrow(data)/2)
model <- neuralnet(response~a+b+c+d+e+f+g+h+i+j,
dat=data[training,], hidden=5, algorithm="rprop-", stepmax = 1e6)
cor.test(predict(model, data[-training,1:10])[,1],
data$response[-training])
#>
#> Pearson's product-moment correlation
#>
#> data: predict(model, data[-training, 1:10])[, 1] and data$response[-training]
#> t = 16.285, df = 198, p-value < 2.2e-16
#> alternative hypothesis: true correlation is not equal to 0
#> 95 percent confidence interval:
#> 0.6903893 0.8103277
#> sample estimates:
#> cor
#> 0.756654
plot(model)Very well! We’ve got a wonderful \(r^2\), and the output looks very… well, confusing. Yes, you can see the hidden layer in there now, and you can see lots of things feeding through it, but what we can’t tell is what’s actually going on. You see, everything looks important in an artificial neural network, and that’s sort of the problem. Just in the same way that we can’t really tell why someone thinks something just by staring at their brain (yet), we can’t necessarily tell what’s going on just by staring at the coefficients in this model. Indeed, we’ve actually fitted so many coefficients (in the form of weights) that it’s not even necessarily clear whether this is a very parsimonious model (which some people would say is that same thing as a good model). Help is at hand, however, in the form of the “Garson test”, which is capable of figuring out what the most important explanatory variables are in a particular model.
library(NeuralNetTools)
garson(model, bar_plot=FALSE)
#> rel_imp
#> a 0.07204879
#> b 0.10165840
#> c 0.35134950
#> d 0.03804852
#> e 0.08645376
#> f 0.01197474
#> g 0.06871687
#> h 0.08071047
#> i 0.11530163
#> j 0.07373733
garson(model)
#> Warning: `label` cannot be a <ggplot2::element_blank> object.
Two things strike us: firstly, this package was written by someone who loves ggplot2 more than easy-to-use code, and so it’s impossible to get both the plot and the data underlying the plot in the same function call. Secondly, by summing up the absolute magnitude of all the weights that connect explanatory variables to the (single) response variable, we can get some measure of relative importance of variables. This method only works for certain kinds of neural networks (e.g., the ones I’ve taught you), and is obviously somewhat unhelpful in that it doesn’t show you whether something is positively or negatively associated, or what the functional form of the association between a variable and the response is. If you want that, then you’ll simply have to play around with dummy datasets, manipulating some variables while holding others constant, to see what’s going on. This is also a good way to test model fit but, ultimately, demonstrates the most important thing to remember about artificial neural networks: they’re really powerful, but we don’t always know what they’re doing.
10.1 Deep Learning and TensorFlow
This is the third of four sessions in your options series, and is itself first of two related sessions that form an introduction to modern deep learning using artificial neural networks. TensorFlow has emerged as the de facto platform of choice for this sort of work for a number of good reasons. The first is its ease of use and portability: TensorFlow started as Python library but has ‘bindings’ (sort-of life cross-language APIs—formal ways for different programming languages to share code) for languages as diverse as JavaScript and Ruby. The second reasons is its flexibility: TensorFlow can do almost anything you could want from a set of artificial neural networks, and can be extended to have whatever kind of nodes or activation functions you want with remarkably little code overhead. Third, and perhaps the most important part, it’s developed by Google, and that name has a lot of cultural and engineering capital. Google often releases the last generation of their internal code as open source packages, and TensorFlow’s blistering speed and scalability is a direct result of that. Deep learning is a field filled with buzz-words that is developing very fast indeed: you will find very few all-encompassing books to it, and most articles online are essentially shallow tutorials because the concepts themselves can be very difficult and are hidden in the primary literature. My aim with these two sessions is to give you a guide tour of what can be done with TensorFlow, focusing on fundamentals (this section) and image analysis (the next session). I am deliberately not covering the specifics of model fitting because, frankly, by the time you read this Google have likely totally changed everything to make it all even faster. This is only a good thing.
10.1.1 Tensors and their ‘flow’
TensorFlow is fast because it views the world as a series of tensors. Thus I think a good place for us to begin is defining a tensor, and then using that definition to understand, fundamentally, how TensorFlow operates.
You may have heard of scalars and vectors. A scalar is what you might think of as a natural number: it’s something that changes the overall magnitude of other numbers3. For example, the length of a rule is a scalar (it’s the ‘scale’ of the object), and the speed of a ruler as it flies through the air is a scalar. A vector has both a scale component and a single direction. Thus, for example, a ruler’s velocity is a vector because it contains both a speed (speed alone would be a scalar) and a direction (this is the part that makes it a vector). Having a magnitude alone makes something a scalar, having a single direction makes something a vector. A tensor is a generalization of scalars and vectors to deal with cases where you have any number of directions. Thus a tensor of rank 0 is the same thing as a scalar: it’s a magnitude with no direction. A tensor of rank 1 is a magnitude and a single direction. A tensor of rank 2 is a magnitude and two directions.
Tensors become extremely important in university-level physics and engineering, because they allow us to formally examine concepts of stress and strain on a system. A tensor is essentially a way of describing how a system will be transformed, because a tensor will take any kind of input data and move them according to its magnitude and direction. For example, imagine you have a single row of information about two columns of data \((x,y)\)—\((1,1)\), for example—to which a tensor is applied. That tensor is going to transform that data by moving it as far as that tensor’s magnitude in whatever direction(s) the tensor specifies. Indeed, this may lead you to realize that such data was a vector all along: \((1,3)\) is the same thing as a vector that says “go up one unit along the x-axis and three along the y-axis”. Just as how, once humans developed a formal definition of addition and multiplication, we were able to do much more complicated and useful things to numbers, now that we have a more formal definition of how transformations of data can take place, we can do more cool things. We can, for example, combine multiple transformations (tensors), see if there is redundant information in those transformations (reduce the rank of a tensor, called contraction), and model very complicated things.
TensorFlow is so-called because it views the entire world as a series of tensors. Tensors are written out in mathematics as tables4 that specify the magnitude and direction of the tensor(s). To TensorFlow, there is no fundamental difference between data and calculation: everything is just a table5. This allows TensorFlow to do clever things like feed the results of computation back into data, distribute computation and data across multiple processors (if you enable GPU acceleration on your laptop, hundreds, but TensorFlow is used routinely across millions of processors), and rapidly calculate the output of artificial neural networks by ‘simply’ multiplying all the tensors together. Thus TensorFlow is absolutely obsessed about asking about the dimensionality of everything you load into it, because it has to create an appropriately-dimensioned set of tensors to handle all of that information. It means that TensorFlow can sometimes take a little while to initially compile (build) its models, but once it does they often can make predictions and be fit blisteringly fast.
10.1.2 Regression reloaded
The easiest way to see how TensorFlow works is to try it out. Let’s start out by simulating essentially the same data from our first introductory class in artificial neural networks. Note that, because TensorFlow is much faster than what we were using before, I’m now adding some noise to the response variable to make things a little trickier, and I’m keeping my data in matrix format and not creating a data.frame.
# Simulate (this time adding noise to the response variable)
exp <- replicate(10, rnorm(400))
resp <- exp[,1]*2 -0.5*exp[,2] - exp[,7]*exp[,8] + exp(abs(exp[,3])) + rnorm(nrow(exp))
# Scale data and making training subset
exp <- as.matrix(scale(exp)); resp <- as.numeric(scale(resp))
training <- sample(nrow(exp), nrow(exp)/2)Great. To use TensorFlow, we’re going to make use of a high-level wrapper6 called Keras. Keras is the ggplot2 of the artificial neural network world: it interfaces with a number of different packages, and is quite a bit easier to use than working with TensorFlow directly via R. To be honest with you, very few people either use Keras for things other than TensorFlow, or access TensorFlow through R using anything else7, and essentially all the lower-level features of TensorFlow are available through Keras. Let’s now build our model.
# Get Keras ready
library(keras3)
# Specific model
model <- keras_model_sequential(input_shape = 10) %>%
layer_dense(units = 15, activation = 'relu') %>%
layer_dense(units = 15, activation = 'relu') %>%
layer_dense(units = 1)
#> Warning: Some Python package requirements declared via `py_require()` are not installed in the selected Python environment: (/home/will/Dropbox/Documents/intuitive-statistics/datasets/.venv/bin/python)
#> pydot scipy pandas ipython tensorflow-cpuThe code above sets up an artificial neural network with two hidden layers, each with 15 nodes, and an output layer of a single node. Artificial neural networks of the kind we have covered so far are all sequential: data comes in, flows through the network in a straight-line without repeating back through any nodes, and then generates a prediction. Not all networks are like this, and so we have to tell Keras we want such a network in the first line above. Each layer in the node is a single line of code (generated using the layer_dense function), and we can build the network up piece-by-piece using R‘s pipe operator (%>%). The pipe operator basically means “add the thing on the right on to whatever you’re building on the left”, and so you can see that we’re modifying model (which is our network) by appending all these layers onto it. We specify how deep (how many nodes) each layer has with the units argument, specify the activation function with the activation argument, and the shape of its input with input_shape. Remember I said TensorFlow is obsessed with the dimensionality of our data because it’s based around tensors? Our network has a shape of \(10\) because there are ten explanatory variables in our input data. That’s where this number comes from: each ’pulse’ of data coming through our network is ten pieces of data, and the resulting output need to be \(1\) thing (we have a single response variable). This is our data’s format. Now we’re ready to fit our model.
# Compile model
model %>% compile(
loss = 'mean_squared_error',
optimizer = optimizer_rmsprop(),
metrics = c('mean_squared_error')
)
# Train model with data
model %>% fit(exp[training,], resp[training], epochs=500)
#> Epoch 1/500
#> 7/7 - 0s - 36ms/step - loss: 1.4437 - mean_squared_error: 1.4437
#> Epoch 2/500
#> 7/7 - 0s - 3ms/step - loss: 1.3617 - mean_squared_error: 1.3617
#> Epoch 3/500
#> 7/7 - 0s - 3ms/step - loss: 1.3153 - mean_squared_error: 1.3153
#> Epoch 4/500
#> 7/7 - 0s - 3ms/step - loss: 1.2755 - mean_squared_error: 1.2755
#> Epoch 5/500
#> 7/7 - 0s - 3ms/step - loss: 1.2456 - mean_squared_error: 1.2456
#> Epoch 6/500
#> 7/7 - 0s - 3ms/step - loss: 1.2179 - mean_squared_error: 1.2179
#> Epoch 7/500
#> 7/7 - 0s - 3ms/step - loss: 1.1953 - mean_squared_error: 1.1953
#> Epoch 8/500
#> 7/7 - 0s - 4ms/step - loss: 1.1724 - mean_squared_error: 1.1724
#> Epoch 9/500
#> 7/7 - 0s - 3ms/step - loss: 1.1538 - mean_squared_error: 1.1538
#> Epoch 10/500
#> 7/7 - 0s - 3ms/step - loss: 1.1363 - mean_squared_error: 1.1363
#> Epoch 11/500
#> 7/7 - 0s - 3ms/step - loss: 1.1202 - mean_squared_error: 1.1202
#> Epoch 12/500
#> 7/7 - 0s - 3ms/step - loss: 1.1028 - mean_squared_error: 1.1028
#> Epoch 13/500
#> 7/7 - 0s - 3ms/step - loss: 1.0880 - mean_squared_error: 1.0880
#> Epoch 14/500
#> 7/7 - 0s - 3ms/step - loss: 1.0744 - mean_squared_error: 1.0744
#> Epoch 15/500
#> 7/7 - 0s - 3ms/step - loss: 1.0611 - mean_squared_error: 1.0611
#> Epoch 16/500
#> 7/7 - 0s - 3ms/step - loss: 1.0478 - mean_squared_error: 1.0478
#> Epoch 17/500
#> 7/7 - 0s - 3ms/step - loss: 1.0357 - mean_squared_error: 1.0357
#> Epoch 18/500
#> 7/7 - 0s - 3ms/step - loss: 1.0213 - mean_squared_error: 1.0213
#> Epoch 19/500
#> 7/7 - 0s - 3ms/step - loss: 1.0090 - mean_squared_error: 1.0090
#> Epoch 20/500
#> 7/7 - 0s - 3ms/step - loss: 0.9973 - mean_squared_error: 0.9973
#> Epoch 21/500
#> 7/7 - 0s - 3ms/step - loss: 0.9852 - mean_squared_error: 0.9852
#> Epoch 22/500
#> 7/7 - 0s - 3ms/step - loss: 0.9749 - mean_squared_error: 0.9749
#> Epoch 23/500
#> 7/7 - 0s - 3ms/step - loss: 0.9655 - mean_squared_error: 0.9655
#> Epoch 24/500
#> 7/7 - 0s - 3ms/step - loss: 0.9576 - mean_squared_error: 0.9576
#> Epoch 25/500
#> 7/7 - 0s - 3ms/step - loss: 0.9460 - mean_squared_error: 0.9460
#> Epoch 26/500
#> 7/7 - 0s - 3ms/step - loss: 0.9400 - mean_squared_error: 0.9400
#> Epoch 27/500
#> 7/7 - 0s - 3ms/step - loss: 0.9296 - mean_squared_error: 0.9296
#> Epoch 28/500
#> 7/7 - 0s - 3ms/step - loss: 0.9218 - mean_squared_error: 0.9218
#> Epoch 29/500
#> 7/7 - 0s - 3ms/step - loss: 0.9133 - mean_squared_error: 0.9133
#> Epoch 30/500
#> 7/7 - 0s - 3ms/step - loss: 0.9023 - mean_squared_error: 0.9023
#> Epoch 31/500
#> 7/7 - 0s - 3ms/step - loss: 0.8935 - mean_squared_error: 0.8935
#> Epoch 32/500
#> 7/7 - 0s - 3ms/step - loss: 0.8848 - mean_squared_error: 0.8848
#> Epoch 33/500
#> 7/7 - 0s - 3ms/step - loss: 0.8806 - mean_squared_error: 0.8806
#> Epoch 34/500
#> 7/7 - 0s - 3ms/step - loss: 0.8708 - mean_squared_error: 0.8708
#> Epoch 35/500
#> 7/7 - 0s - 3ms/step - loss: 0.8646 - mean_squared_error: 0.8646
#> Epoch 36/500
#> 7/7 - 0s - 3ms/step - loss: 0.8536 - mean_squared_error: 0.8536
#> Epoch 37/500
#> 7/7 - 0s - 3ms/step - loss: 0.8476 - mean_squared_error: 0.8476
#> Epoch 38/500
#> 7/7 - 0s - 3ms/step - loss: 0.8396 - mean_squared_error: 0.8396
#> Epoch 39/500
#> 7/7 - 0s - 3ms/step - loss: 0.8323 - mean_squared_error: 0.8323
#> Epoch 40/500
#> 7/7 - 0s - 3ms/step - loss: 0.8270 - mean_squared_error: 0.8270
#> Epoch 41/500
#> 7/7 - 0s - 3ms/step - loss: 0.8181 - mean_squared_error: 0.8181
#> Epoch 42/500
#> 7/7 - 0s - 3ms/step - loss: 0.8116 - mean_squared_error: 0.8116
#> Epoch 43/500
#> 7/7 - 0s - 3ms/step - loss: 0.8042 - mean_squared_error: 0.8042
#> Epoch 44/500
#> 7/7 - 0s - 3ms/step - loss: 0.7980 - mean_squared_error: 0.7980
#> Epoch 45/500
#> 7/7 - 0s - 3ms/step - loss: 0.7914 - mean_squared_error: 0.7914
#> Epoch 46/500
#> 7/7 - 0s - 3ms/step - loss: 0.7847 - mean_squared_error: 0.7847
#> Epoch 47/500
#> 7/7 - 0s - 4ms/step - loss: 0.7756 - mean_squared_error: 0.7756
#> Epoch 48/500
#> 7/7 - 0s - 3ms/step - loss: 0.7721 - mean_squared_error: 0.7721
#> Epoch 49/500
#> 7/7 - 0s - 3ms/step - loss: 0.7630 - mean_squared_error: 0.7630
#> Epoch 50/500
#> 7/7 - 0s - 3ms/step - loss: 0.7577 - mean_squared_error: 0.7577
#> Epoch 51/500
#> 7/7 - 0s - 3ms/step - loss: 0.7525 - mean_squared_error: 0.7525
#> Epoch 52/500
#> 7/7 - 0s - 3ms/step - loss: 0.7446 - mean_squared_error: 0.7446
#> Epoch 53/500
#> 7/7 - 0s - 3ms/step - loss: 0.7400 - mean_squared_error: 0.7400
#> Epoch 54/500
#> 7/7 - 0s - 3ms/step - loss: 0.7319 - mean_squared_error: 0.7319
#> Epoch 55/500
#> 7/7 - 0s - 3ms/step - loss: 0.7263 - mean_squared_error: 0.7263
#> Epoch 56/500
#> 7/7 - 0s - 3ms/step - loss: 0.7205 - mean_squared_error: 0.7205
#> Epoch 57/500
#> 7/7 - 0s - 3ms/step - loss: 0.7134 - mean_squared_error: 0.7134
#> Epoch 58/500
#> 7/7 - 0s - 3ms/step - loss: 0.7083 - mean_squared_error: 0.7083
#> Epoch 59/500
#> 7/7 - 0s - 3ms/step - loss: 0.7019 - mean_squared_error: 0.7019
#> Epoch 60/500
#> 7/7 - 0s - 3ms/step - loss: 0.6958 - mean_squared_error: 0.6958
#> Epoch 61/500
#> 7/7 - 0s - 3ms/step - loss: 0.6894 - mean_squared_error: 0.6894
#> Epoch 62/500
#> 7/7 - 0s - 3ms/step - loss: 0.6837 - mean_squared_error: 0.6837
#> Epoch 63/500
#> 7/7 - 0s - 3ms/step - loss: 0.6732 - mean_squared_error: 0.6732
#> Epoch 64/500
#> 7/7 - 0s - 3ms/step - loss: 0.6694 - mean_squared_error: 0.6694
#> Epoch 65/500
#> 7/7 - 0s - 3ms/step - loss: 0.6617 - mean_squared_error: 0.6617
#> Epoch 66/500
#> 7/7 - 0s - 3ms/step - loss: 0.6585 - mean_squared_error: 0.6585
#> Epoch 67/500
#> 7/7 - 0s - 3ms/step - loss: 0.6519 - mean_squared_error: 0.6519
#> Epoch 68/500
#> 7/7 - 0s - 3ms/step - loss: 0.6485 - mean_squared_error: 0.6485
#> Epoch 69/500
#> 7/7 - 0s - 3ms/step - loss: 0.6389 - mean_squared_error: 0.6389
#> Epoch 70/500
#> 7/7 - 0s - 3ms/step - loss: 0.6345 - mean_squared_error: 0.6345
#> Epoch 71/500
#> 7/7 - 0s - 3ms/step - loss: 0.6337 - mean_squared_error: 0.6337
#> Epoch 72/500
#> 7/7 - 0s - 3ms/step - loss: 0.6267 - mean_squared_error: 0.6267
#> Epoch 73/500
#> 7/7 - 0s - 3ms/step - loss: 0.6216 - mean_squared_error: 0.6216
#> Epoch 74/500
#> 7/7 - 0s - 3ms/step - loss: 0.6164 - mean_squared_error: 0.6164
#> Epoch 75/500
#> 7/7 - 0s - 3ms/step - loss: 0.6090 - mean_squared_error: 0.6090
#> Epoch 76/500
#> 7/7 - 0s - 3ms/step - loss: 0.6022 - mean_squared_error: 0.6022
#> Epoch 77/500
#> 7/7 - 0s - 3ms/step - loss: 0.6016 - mean_squared_error: 0.6016
#> Epoch 78/500
#> 7/7 - 0s - 3ms/step - loss: 0.5934 - mean_squared_error: 0.5934
#> Epoch 79/500
#> 7/7 - 0s - 3ms/step - loss: 0.5862 - mean_squared_error: 0.5862
#> Epoch 80/500
#> 7/7 - 0s - 3ms/step - loss: 0.5812 - mean_squared_error: 0.5812
#> Epoch 81/500
#> 7/7 - 0s - 3ms/step - loss: 0.5721 - mean_squared_error: 0.5721
#> Epoch 82/500
#> 7/7 - 0s - 3ms/step - loss: 0.5649 - mean_squared_error: 0.5649
#> Epoch 83/500
#> 7/7 - 0s - 3ms/step - loss: 0.5589 - mean_squared_error: 0.5589
#> Epoch 84/500
#> 7/7 - 0s - 3ms/step - loss: 0.5514 - mean_squared_error: 0.5514
#> Epoch 85/500
#> 7/7 - 0s - 3ms/step - loss: 0.5455 - mean_squared_error: 0.5455
#> Epoch 86/500
#> 7/7 - 0s - 4ms/step - loss: 0.5413 - mean_squared_error: 0.5413
#> Epoch 87/500
#> 7/7 - 0s - 3ms/step - loss: 0.5318 - mean_squared_error: 0.5318
#> Epoch 88/500
#> 7/7 - 0s - 3ms/step - loss: 0.5286 - mean_squared_error: 0.5286
#> Epoch 89/500
#> 7/7 - 0s - 3ms/step - loss: 0.5193 - mean_squared_error: 0.5193
#> Epoch 90/500
#> 7/7 - 0s - 3ms/step - loss: 0.5144 - mean_squared_error: 0.5144
#> Epoch 91/500
#> 7/7 - 0s - 3ms/step - loss: 0.5060 - mean_squared_error: 0.5060
#> Epoch 92/500
#> 7/7 - 0s - 3ms/step - loss: 0.5029 - mean_squared_error: 0.5029
#> Epoch 93/500
#> 7/7 - 0s - 3ms/step - loss: 0.4966 - mean_squared_error: 0.4966
#> Epoch 94/500
#> 7/7 - 0s - 3ms/step - loss: 0.4934 - mean_squared_error: 0.4934
#> Epoch 95/500
#> 7/7 - 0s - 3ms/step - loss: 0.4878 - mean_squared_error: 0.4878
#> Epoch 96/500
#> 7/7 - 0s - 3ms/step - loss: 0.4854 - mean_squared_error: 0.4854
#> Epoch 97/500
#> 7/7 - 0s - 3ms/step - loss: 0.4757 - mean_squared_error: 0.4757
#> Epoch 98/500
#> 7/7 - 0s - 3ms/step - loss: 0.4740 - mean_squared_error: 0.4740
#> Epoch 99/500
#> 7/7 - 0s - 3ms/step - loss: 0.4686 - mean_squared_error: 0.4686
#> Epoch 100/500
#> 7/7 - 0s - 3ms/step - loss: 0.4634 - mean_squared_error: 0.4634
#> Epoch 101/500
#> 7/7 - 0s - 3ms/step - loss: 0.4594 - mean_squared_error: 0.4594
#> Epoch 102/500
#> 7/7 - 0s - 3ms/step - loss: 0.4543 - mean_squared_error: 0.4543
#> Epoch 103/500
#> 7/7 - 0s - 3ms/step - loss: 0.4477 - mean_squared_error: 0.4477
#> Epoch 104/500
#> 7/7 - 0s - 3ms/step - loss: 0.4446 - mean_squared_error: 0.4446
#> Epoch 105/500
#> 7/7 - 0s - 3ms/step - loss: 0.4409 - mean_squared_error: 0.4409
#> Epoch 106/500
#> 7/7 - 0s - 3ms/step - loss: 0.4355 - mean_squared_error: 0.4355
#> Epoch 107/500
#> 7/7 - 0s - 3ms/step - loss: 0.4297 - mean_squared_error: 0.4297
#> Epoch 108/500
#> 7/7 - 0s - 3ms/step - loss: 0.4267 - mean_squared_error: 0.4267
#> Epoch 109/500
#> 7/7 - 0s - 3ms/step - loss: 0.4218 - mean_squared_error: 0.4218
#> Epoch 110/500
#> 7/7 - 0s - 3ms/step - loss: 0.4156 - mean_squared_error: 0.4156
#> Epoch 111/500
#> 7/7 - 0s - 3ms/step - loss: 0.4151 - mean_squared_error: 0.4151
#> Epoch 112/500
#> 7/7 - 0s - 3ms/step - loss: 0.4077 - mean_squared_error: 0.4077
#> Epoch 113/500
#> 7/7 - 0s - 3ms/step - loss: 0.4041 - mean_squared_error: 0.4041
#> Epoch 114/500
#> 7/7 - 0s - 3ms/step - loss: 0.3973 - mean_squared_error: 0.3973
#> Epoch 115/500
#> 7/7 - 0s - 3ms/step - loss: 0.3944 - mean_squared_error: 0.3944
#> Epoch 116/500
#> 7/7 - 0s - 3ms/step - loss: 0.3902 - mean_squared_error: 0.3902
#> Epoch 117/500
#> 7/7 - 0s - 3ms/step - loss: 0.3862 - mean_squared_error: 0.3862
#> Epoch 118/500
#> 7/7 - 0s - 3ms/step - loss: 0.3831 - mean_squared_error: 0.3831
#> Epoch 119/500
#> 7/7 - 0s - 3ms/step - loss: 0.3806 - mean_squared_error: 0.3806
#> Epoch 120/500
#> 7/7 - 0s - 3ms/step - loss: 0.3746 - mean_squared_error: 0.3746
#> Epoch 121/500
#> 7/7 - 0s - 3ms/step - loss: 0.3699 - mean_squared_error: 0.3699
#> Epoch 122/500
#> 7/7 - 0s - 3ms/step - loss: 0.3652 - mean_squared_error: 0.3652
#> Epoch 123/500
#> 7/7 - 0s - 3ms/step - loss: 0.3623 - mean_squared_error: 0.3623
#> Epoch 124/500
#> 7/7 - 0s - 3ms/step - loss: 0.3567 - mean_squared_error: 0.3567
#> Epoch 125/500
#> 7/7 - 0s - 3ms/step - loss: 0.3533 - mean_squared_error: 0.3533
#> Epoch 126/500
#> 7/7 - 0s - 3ms/step - loss: 0.3463 - mean_squared_error: 0.3463
#> Epoch 127/500
#> 7/7 - 0s - 3ms/step - loss: 0.3423 - mean_squared_error: 0.3423
#> Epoch 128/500
#> 7/7 - 0s - 3ms/step - loss: 0.3396 - mean_squared_error: 0.3396
#> Epoch 129/500
#> 7/7 - 0s - 3ms/step - loss: 0.3375 - mean_squared_error: 0.3375
#> Epoch 130/500
#> 7/7 - 0s - 3ms/step - loss: 0.3341 - mean_squared_error: 0.3341
#> Epoch 131/500
#> 7/7 - 0s - 3ms/step - loss: 0.3294 - mean_squared_error: 0.3294
#> Epoch 132/500
#> 7/7 - 0s - 3ms/step - loss: 0.3292 - mean_squared_error: 0.3292
#> Epoch 133/500
#> 7/7 - 0s - 3ms/step - loss: 0.3203 - mean_squared_error: 0.3203
#> Epoch 134/500
#> 7/7 - 0s - 3ms/step - loss: 0.3213 - mean_squared_error: 0.3213
#> Epoch 135/500
#> 7/7 - 0s - 3ms/step - loss: 0.3171 - mean_squared_error: 0.3171
#> Epoch 136/500
#> 7/7 - 0s - 3ms/step - loss: 0.3122 - mean_squared_error: 0.3122
#> Epoch 137/500
#> 7/7 - 0s - 3ms/step - loss: 0.3079 - mean_squared_error: 0.3079
#> Epoch 138/500
#> 7/7 - 0s - 3ms/step - loss: 0.3050 - mean_squared_error: 0.3050
#> Epoch 139/500
#> 7/7 - 0s - 3ms/step - loss: 0.2999 - mean_squared_error: 0.2999
#> Epoch 140/500
#> 7/7 - 0s - 3ms/step - loss: 0.2981 - mean_squared_error: 0.2981
#> Epoch 141/500
#> 7/7 - 0s - 4ms/step - loss: 0.2915 - mean_squared_error: 0.2915
#> Epoch 142/500
#> 7/7 - 0s - 3ms/step - loss: 0.2891 - mean_squared_error: 0.2891
#> Epoch 143/500
#> 7/7 - 0s - 3ms/step - loss: 0.2871 - mean_squared_error: 0.2871
#> Epoch 144/500
#> 7/7 - 0s - 3ms/step - loss: 0.2827 - mean_squared_error: 0.2827
#> Epoch 145/500
#> 7/7 - 0s - 3ms/step - loss: 0.2810 - mean_squared_error: 0.2810
#> Epoch 146/500
#> 7/7 - 0s - 3ms/step - loss: 0.2755 - mean_squared_error: 0.2755
#> Epoch 147/500
#> 7/7 - 0s - 3ms/step - loss: 0.2706 - mean_squared_error: 0.2706
#> Epoch 148/500
#> 7/7 - 0s - 3ms/step - loss: 0.2672 - mean_squared_error: 0.2672
#> Epoch 149/500
#> 7/7 - 0s - 3ms/step - loss: 0.2665 - mean_squared_error: 0.2665
#> Epoch 150/500
#> 7/7 - 0s - 3ms/step - loss: 0.2579 - mean_squared_error: 0.2579
#> Epoch 151/500
#> 7/7 - 0s - 3ms/step - loss: 0.2569 - mean_squared_error: 0.2569
#> Epoch 152/500
#> 7/7 - 0s - 3ms/step - loss: 0.2552 - mean_squared_error: 0.2552
#> Epoch 153/500
#> 7/7 - 0s - 3ms/step - loss: 0.2526 - mean_squared_error: 0.2526
#> Epoch 154/500
#> 7/7 - 0s - 3ms/step - loss: 0.2521 - mean_squared_error: 0.2521
#> Epoch 155/500
#> 7/7 - 0s - 3ms/step - loss: 0.2502 - mean_squared_error: 0.2502
#> Epoch 156/500
#> 7/7 - 0s - 3ms/step - loss: 0.2466 - mean_squared_error: 0.2466
#> Epoch 157/500
#> 7/7 - 0s - 3ms/step - loss: 0.2437 - mean_squared_error: 0.2437
#> Epoch 158/500
#> 7/7 - 0s - 3ms/step - loss: 0.2418 - mean_squared_error: 0.2418
#> Epoch 159/500
#> 7/7 - 0s - 3ms/step - loss: 0.2370 - mean_squared_error: 0.2370
#> Epoch 160/500
#> 7/7 - 0s - 3ms/step - loss: 0.2339 - mean_squared_error: 0.2339
#> Epoch 161/500
#> 7/7 - 0s - 3ms/step - loss: 0.2330 - mean_squared_error: 0.2330
#> Epoch 162/500
#> 7/7 - 0s - 3ms/step - loss: 0.2290 - mean_squared_error: 0.2290
#> Epoch 163/500
#> 7/7 - 0s - 3ms/step - loss: 0.2245 - mean_squared_error: 0.2245
#> Epoch 164/500
#> 7/7 - 0s - 3ms/step - loss: 0.2230 - mean_squared_error: 0.2230
#> Epoch 165/500
#> 7/7 - 0s - 3ms/step - loss: 0.2209 - mean_squared_error: 0.2209
#> Epoch 166/500
#> 7/7 - 0s - 3ms/step - loss: 0.2166 - mean_squared_error: 0.2166
#> Epoch 167/500
#> 7/7 - 0s - 4ms/step - loss: 0.2143 - mean_squared_error: 0.2143
#> Epoch 168/500
#> 7/7 - 0s - 3ms/step - loss: 0.2099 - mean_squared_error: 0.2099
#> Epoch 169/500
#> 7/7 - 0s - 3ms/step - loss: 0.2084 - mean_squared_error: 0.2084
#> Epoch 170/500
#> 7/7 - 0s - 3ms/step - loss: 0.2065 - mean_squared_error: 0.2065
#> Epoch 171/500
#> 7/7 - 0s - 3ms/step - loss: 0.2024 - mean_squared_error: 0.2024
#> Epoch 172/500
#> 7/7 - 0s - 3ms/step - loss: 0.2013 - mean_squared_error: 0.2013
#> Epoch 173/500
#> 7/7 - 0s - 3ms/step - loss: 0.2014 - mean_squared_error: 0.2014
#> Epoch 174/500
#> 7/7 - 0s - 3ms/step - loss: 0.1960 - mean_squared_error: 0.1960
#> Epoch 175/500
#> 7/7 - 0s - 4ms/step - loss: 0.1944 - mean_squared_error: 0.1944
#> Epoch 176/500
#> 7/7 - 0s - 3ms/step - loss: 0.1900 - mean_squared_error: 0.1900
#> Epoch 177/500
#> 7/7 - 0s - 4ms/step - loss: 0.1901 - mean_squared_error: 0.1901
#> Epoch 178/500
#> 7/7 - 0s - 3ms/step - loss: 0.1879 - mean_squared_error: 0.1879
#> Epoch 179/500
#> 7/7 - 0s - 3ms/step - loss: 0.1857 - mean_squared_error: 0.1857
#> Epoch 180/500
#> 7/7 - 0s - 3ms/step - loss: 0.1809 - mean_squared_error: 0.1809
#> Epoch 181/500
#> 7/7 - 0s - 3ms/step - loss: 0.1848 - mean_squared_error: 0.1848
#> Epoch 182/500
#> 7/7 - 0s - 3ms/step - loss: 0.1802 - mean_squared_error: 0.1802
#> Epoch 183/500
#> 7/7 - 0s - 3ms/step - loss: 0.1765 - mean_squared_error: 0.1765
#> Epoch 184/500
#> 7/7 - 0s - 3ms/step - loss: 0.1729 - mean_squared_error: 0.1729
#> Epoch 185/500
#> 7/7 - 0s - 3ms/step - loss: 0.1732 - mean_squared_error: 0.1732
#> Epoch 186/500
#> 7/7 - 0s - 3ms/step - loss: 0.1688 - mean_squared_error: 0.1688
#> Epoch 187/500
#> 7/7 - 0s - 3ms/step - loss: 0.1669 - mean_squared_error: 0.1669
#> Epoch 188/500
#> 7/7 - 0s - 3ms/step - loss: 0.1652 - mean_squared_error: 0.1652
#> Epoch 189/500
#> 7/7 - 0s - 4ms/step - loss: 0.1623 - mean_squared_error: 0.1623
#> Epoch 190/500
#> 7/7 - 0s - 3ms/step - loss: 0.1607 - mean_squared_error: 0.1607
#> Epoch 191/500
#> 7/7 - 0s - 3ms/step - loss: 0.1595 - mean_squared_error: 0.1595
#> Epoch 192/500
#> 7/7 - 0s - 3ms/step - loss: 0.1557 - mean_squared_error: 0.1557
#> Epoch 193/500
#> 7/7 - 0s - 3ms/step - loss: 0.1522 - mean_squared_error: 0.1522
#> Epoch 194/500
#> 7/7 - 0s - 3ms/step - loss: 0.1529 - mean_squared_error: 0.1529
#> Epoch 195/500
#> 7/7 - 0s - 3ms/step - loss: 0.1488 - mean_squared_error: 0.1488
#> Epoch 196/500
#> 7/7 - 0s - 3ms/step - loss: 0.1487 - mean_squared_error: 0.1487
#> Epoch 197/500
#> 7/7 - 0s - 3ms/step - loss: 0.1458 - mean_squared_error: 0.1458
#> Epoch 198/500
#> 7/7 - 0s - 3ms/step - loss: 0.1438 - mean_squared_error: 0.1438
#> Epoch 199/500
#> 7/7 - 0s - 3ms/step - loss: 0.1423 - mean_squared_error: 0.1423
#> Epoch 200/500
#> 7/7 - 0s - 3ms/step - loss: 0.1409 - mean_squared_error: 0.1409
#> Epoch 201/500
#> 7/7 - 0s - 3ms/step - loss: 0.1371 - mean_squared_error: 0.1371
#> Epoch 202/500
#> 7/7 - 0s - 3ms/step - loss: 0.1366 - mean_squared_error: 0.1366
#> Epoch 203/500
#> 7/7 - 0s - 3ms/step - loss: 0.1338 - mean_squared_error: 0.1338
#> Epoch 204/500
#> 7/7 - 0s - 3ms/step - loss: 0.1328 - mean_squared_error: 0.1328
#> Epoch 205/500
#> 7/7 - 0s - 3ms/step - loss: 0.1310 - mean_squared_error: 0.1310
#> Epoch 206/500
#> 7/7 - 0s - 3ms/step - loss: 0.1325 - mean_squared_error: 0.1325
#> Epoch 207/500
#> 7/7 - 0s - 3ms/step - loss: 0.1286 - mean_squared_error: 0.1286
#> Epoch 208/500
#> 7/7 - 0s - 3ms/step - loss: 0.1260 - mean_squared_error: 0.1260
#> Epoch 209/500
#> 7/7 - 0s - 3ms/step - loss: 0.1258 - mean_squared_error: 0.1258
#> Epoch 210/500
#> 7/7 - 0s - 3ms/step - loss: 0.1273 - mean_squared_error: 0.1273
#> Epoch 211/500
#> 7/7 - 0s - 3ms/step - loss: 0.1215 - mean_squared_error: 0.1215
#> Epoch 212/500
#> 7/7 - 0s - 3ms/step - loss: 0.1226 - mean_squared_error: 0.1226
#> Epoch 213/500
#> 7/7 - 0s - 3ms/step - loss: 0.1205 - mean_squared_error: 0.1205
#> Epoch 214/500
#> 7/7 - 0s - 3ms/step - loss: 0.1194 - mean_squared_error: 0.1194
#> Epoch 215/500
#> 7/7 - 0s - 4ms/step - loss: 0.1178 - mean_squared_error: 0.1178
#> Epoch 216/500
#> 7/7 - 0s - 3ms/step - loss: 0.1146 - mean_squared_error: 0.1146
#> Epoch 217/500
#> 7/7 - 0s - 3ms/step - loss: 0.1136 - mean_squared_error: 0.1136
#> Epoch 218/500
#> 7/7 - 0s - 3ms/step - loss: 0.1141 - mean_squared_error: 0.1141
#> Epoch 219/500
#> 7/7 - 0s - 3ms/step - loss: 0.1106 - mean_squared_error: 0.1106
#> Epoch 220/500
#> 7/7 - 0s - 3ms/step - loss: 0.1090 - mean_squared_error: 0.1090
#> Epoch 221/500
#> 7/7 - 0s - 3ms/step - loss: 0.1067 - mean_squared_error: 0.1067
#> Epoch 222/500
#> 7/7 - 0s - 3ms/step - loss: 0.1086 - mean_squared_error: 0.1086
#> Epoch 223/500
#> 7/7 - 0s - 3ms/step - loss: 0.1052 - mean_squared_error: 0.1052
#> Epoch 224/500
#> 7/7 - 0s - 3ms/step - loss: 0.1074 - mean_squared_error: 0.1074
#> Epoch 225/500
#> 7/7 - 0s - 3ms/step - loss: 0.1045 - mean_squared_error: 0.1045
#> Epoch 226/500
#> 7/7 - 0s - 4ms/step - loss: 0.1025 - mean_squared_error: 0.1025
#> Epoch 227/500
#> 7/7 - 0s - 3ms/step - loss: 0.1012 - mean_squared_error: 0.1012
#> Epoch 228/500
#> 7/7 - 0s - 3ms/step - loss: 0.1015 - mean_squared_error: 0.1015
#> Epoch 229/500
#> 7/7 - 0s - 3ms/step - loss: 0.0998 - mean_squared_error: 0.0998
#> Epoch 230/500
#> 7/7 - 0s - 3ms/step - loss: 0.0973 - mean_squared_error: 0.0973
#> Epoch 231/500
#> 7/7 - 0s - 3ms/step - loss: 0.0960 - mean_squared_error: 0.0960
#> Epoch 232/500
#> 7/7 - 0s - 3ms/step - loss: 0.0958 - mean_squared_error: 0.0958
#> Epoch 233/500
#> 7/7 - 0s - 3ms/step - loss: 0.0969 - mean_squared_error: 0.0969
#> Epoch 234/500
#> 7/7 - 0s - 3ms/step - loss: 0.0940 - mean_squared_error: 0.0940
#> Epoch 235/500
#> 7/7 - 0s - 3ms/step - loss: 0.0925 - mean_squared_error: 0.0925
#> Epoch 236/500
#> 7/7 - 0s - 3ms/step - loss: 0.0908 - mean_squared_error: 0.0908
#> Epoch 237/500
#> 7/7 - 0s - 3ms/step - loss: 0.0888 - mean_squared_error: 0.0888
#> Epoch 238/500
#> 7/7 - 0s - 3ms/step - loss: 0.0882 - mean_squared_error: 0.0882
#> Epoch 239/500
#> 7/7 - 0s - 3ms/step - loss: 0.0902 - mean_squared_error: 0.0902
#> Epoch 240/500
#> 7/7 - 0s - 3ms/step - loss: 0.0857 - mean_squared_error: 0.0857
#> Epoch 241/500
#> 7/7 - 0s - 3ms/step - loss: 0.0870 - mean_squared_error: 0.0870
#> Epoch 242/500
#> 7/7 - 0s - 3ms/step - loss: 0.0843 - mean_squared_error: 0.0843
#> Epoch 243/500
#> 7/7 - 0s - 3ms/step - loss: 0.0841 - mean_squared_error: 0.0841
#> Epoch 244/500
#> 7/7 - 0s - 3ms/step - loss: 0.0807 - mean_squared_error: 0.0807
#> Epoch 245/500
#> 7/7 - 0s - 3ms/step - loss: 0.0808 - mean_squared_error: 0.0808
#> Epoch 246/500
#> 7/7 - 0s - 3ms/step - loss: 0.0845 - mean_squared_error: 0.0845
#> Epoch 247/500
#> 7/7 - 0s - 3ms/step - loss: 0.0781 - mean_squared_error: 0.0781
#> Epoch 248/500
#> 7/7 - 0s - 3ms/step - loss: 0.0789 - mean_squared_error: 0.0789
#> Epoch 249/500
#> 7/7 - 0s - 3ms/step - loss: 0.0765 - mean_squared_error: 0.0765
#> Epoch 250/500
#> 7/7 - 0s - 3ms/step - loss: 0.0753 - mean_squared_error: 0.0753
#> Epoch 251/500
#> 7/7 - 0s - 3ms/step - loss: 0.0782 - mean_squared_error: 0.0782
#> Epoch 252/500
#> 7/7 - 0s - 3ms/step - loss: 0.0766 - mean_squared_error: 0.0766
#> Epoch 253/500
#> 7/7 - 0s - 4ms/step - loss: 0.0729 - mean_squared_error: 0.0729
#> Epoch 254/500
#> 7/7 - 0s - 3ms/step - loss: 0.0751 - mean_squared_error: 0.0751
#> Epoch 255/500
#> 7/7 - 0s - 3ms/step - loss: 0.0711 - mean_squared_error: 0.0711
#> Epoch 256/500
#> 7/7 - 0s - 3ms/step - loss: 0.0697 - mean_squared_error: 0.0697
#> Epoch 257/500
#> 7/7 - 0s - 3ms/step - loss: 0.0713 - mean_squared_error: 0.0713
#> Epoch 258/500
#> 7/7 - 0s - 3ms/step - loss: 0.0687 - mean_squared_error: 0.0687
#> Epoch 259/500
#> 7/7 - 0s - 3ms/step - loss: 0.0695 - mean_squared_error: 0.0695
#> Epoch 260/500
#> 7/7 - 0s - 3ms/step - loss: 0.0659 - mean_squared_error: 0.0659
#> Epoch 261/500
#> 7/7 - 0s - 3ms/step - loss: 0.0687 - mean_squared_error: 0.0687
#> Epoch 262/500
#> 7/7 - 0s - 4ms/step - loss: 0.0674 - mean_squared_error: 0.0674
#> Epoch 263/500
#> 7/7 - 0s - 3ms/step - loss: 0.0653 - mean_squared_error: 0.0653
#> Epoch 264/500
#> 7/7 - 0s - 3ms/step - loss: 0.0631 - mean_squared_error: 0.0631
#> Epoch 265/500
#> 7/7 - 0s - 3ms/step - loss: 0.0624 - mean_squared_error: 0.0624
#> Epoch 266/500
#> 7/7 - 0s - 3ms/step - loss: 0.0642 - mean_squared_error: 0.0642
#> Epoch 267/500
#> 7/7 - 0s - 3ms/step - loss: 0.0603 - mean_squared_error: 0.0603
#> Epoch 268/500
#> 7/7 - 0s - 3ms/step - loss: 0.0629 - mean_squared_error: 0.0629
#> Epoch 269/500
#> 7/7 - 0s - 3ms/step - loss: 0.0600 - mean_squared_error: 0.0600
#> Epoch 270/500
#> 7/7 - 0s - 3ms/step - loss: 0.0600 - mean_squared_error: 0.0600
#> Epoch 271/500
#> 7/7 - 0s - 3ms/step - loss: 0.0585 - mean_squared_error: 0.0585
#> Epoch 272/500
#> 7/7 - 0s - 3ms/step - loss: 0.0583 - mean_squared_error: 0.0583
#> Epoch 273/500
#> 7/7 - 0s - 3ms/step - loss: 0.0576 - mean_squared_error: 0.0576
#> Epoch 274/500
#> 7/7 - 0s - 3ms/step - loss: 0.0576 - mean_squared_error: 0.0576
#> Epoch 275/500
#> 7/7 - 0s - 3ms/step - loss: 0.0572 - mean_squared_error: 0.0572
#> Epoch 276/500
#> 7/7 - 0s - 3ms/step - loss: 0.0554 - mean_squared_error: 0.0554
#> Epoch 277/500
#> 7/7 - 0s - 3ms/step - loss: 0.0549 - mean_squared_error: 0.0549
#> Epoch 278/500
#> 7/7 - 0s - 3ms/step - loss: 0.0535 - mean_squared_error: 0.0535
#> Epoch 279/500
#> 7/7 - 0s - 3ms/step - loss: 0.0555 - mean_squared_error: 0.0555
#> Epoch 280/500
#> 7/7 - 0s - 3ms/step - loss: 0.0551 - mean_squared_error: 0.0551
#> Epoch 281/500
#> 7/7 - 0s - 3ms/step - loss: 0.0512 - mean_squared_error: 0.0512
#> Epoch 282/500
#> 7/7 - 0s - 3ms/step - loss: 0.0502 - mean_squared_error: 0.0502
#> Epoch 283/500
#> 7/7 - 0s - 3ms/step - loss: 0.0519 - mean_squared_error: 0.0519
#> Epoch 284/500
#> 7/7 - 0s - 3ms/step - loss: 0.0510 - mean_squared_error: 0.0510
#> Epoch 285/500
#> 7/7 - 0s - 3ms/step - loss: 0.0505 - mean_squared_error: 0.0505
#> Epoch 286/500
#> 7/7 - 0s - 3ms/step - loss: 0.0514 - mean_squared_error: 0.0514
#> Epoch 287/500
#> 7/7 - 0s - 3ms/step - loss: 0.0484 - mean_squared_error: 0.0484
#> Epoch 288/500
#> 7/7 - 0s - 3ms/step - loss: 0.0486 - mean_squared_error: 0.0486
#> Epoch 289/500
#> 7/7 - 0s - 3ms/step - loss: 0.0490 - mean_squared_error: 0.0490
#> Epoch 290/500
#> 7/7 - 0s - 3ms/step - loss: 0.0486 - mean_squared_error: 0.0486
#> Epoch 291/500
#> 7/7 - 0s - 3ms/step - loss: 0.0473 - mean_squared_error: 0.0473
#> Epoch 292/500
#> 7/7 - 0s - 3ms/step - loss: 0.0460 - mean_squared_error: 0.0460
#> Epoch 293/500
#> 7/7 - 0s - 3ms/step - loss: 0.0467 - mean_squared_error: 0.0467
#> Epoch 294/500
#> 7/7 - 0s - 3ms/step - loss: 0.0445 - mean_squared_error: 0.0445
#> Epoch 295/500
#> 7/7 - 0s - 3ms/step - loss: 0.0445 - mean_squared_error: 0.0445
#> Epoch 296/500
#> 7/7 - 0s - 3ms/step - loss: 0.0439 - mean_squared_error: 0.0439
#> Epoch 297/500
#> 7/7 - 0s - 3ms/step - loss: 0.0453 - mean_squared_error: 0.0453
#> Epoch 298/500
#> 7/7 - 0s - 3ms/step - loss: 0.0436 - mean_squared_error: 0.0436
#> Epoch 299/500
#> 7/7 - 0s - 3ms/step - loss: 0.0429 - mean_squared_error: 0.0429
#> Epoch 300/500
#> 7/7 - 0s - 3ms/step - loss: 0.0447 - mean_squared_error: 0.0447
#> Epoch 301/500
#> 7/7 - 0s - 3ms/step - loss: 0.0428 - mean_squared_error: 0.0428
#> Epoch 302/500
#> 7/7 - 0s - 3ms/step - loss: 0.0434 - mean_squared_error: 0.0434
#> Epoch 303/500
#> 7/7 - 0s - 3ms/step - loss: 0.0407 - mean_squared_error: 0.0407
#> Epoch 304/500
#> 7/7 - 0s - 3ms/step - loss: 0.0410 - mean_squared_error: 0.0410
#> Epoch 305/500
#> 7/7 - 0s - 3ms/step - loss: 0.0433 - mean_squared_error: 0.0433
#> Epoch 306/500
#> 7/7 - 0s - 3ms/step - loss: 0.0407 - mean_squared_error: 0.0407
#> Epoch 307/500
#> 7/7 - 0s - 3ms/step - loss: 0.0402 - mean_squared_error: 0.0402
#> Epoch 308/500
#> 7/7 - 0s - 3ms/step - loss: 0.0402 - mean_squared_error: 0.0402
#> Epoch 309/500
#> 7/7 - 0s - 3ms/step - loss: 0.0406 - mean_squared_error: 0.0406
#> Epoch 310/500
#> 7/7 - 0s - 3ms/step - loss: 0.0400 - mean_squared_error: 0.0400
#> Epoch 311/500
#> 7/7 - 0s - 3ms/step - loss: 0.0381 - mean_squared_error: 0.0381
#> Epoch 312/500
#> 7/7 - 0s - 3ms/step - loss: 0.0382 - mean_squared_error: 0.0382
#> Epoch 313/500
#> 7/7 - 0s - 3ms/step - loss: 0.0396 - mean_squared_error: 0.0396
#> Epoch 314/500
#> 7/7 - 0s - 3ms/step - loss: 0.0381 - mean_squared_error: 0.0381
#> Epoch 315/500
#> 7/7 - 0s - 3ms/step - loss: 0.0369 - mean_squared_error: 0.0369
#> Epoch 316/500
#> 7/7 - 0s - 3ms/step - loss: 0.0372 - mean_squared_error: 0.0372
#> Epoch 317/500
#> 7/7 - 0s - 3ms/step - loss: 0.0379 - mean_squared_error: 0.0379
#> Epoch 318/500
#> 7/7 - 0s - 3ms/step - loss: 0.0347 - mean_squared_error: 0.0347
#> Epoch 319/500
#> 7/7 - 0s - 3ms/step - loss: 0.0351 - mean_squared_error: 0.0351
#> Epoch 320/500
#> 7/7 - 0s - 3ms/step - loss: 0.0367 - mean_squared_error: 0.0367
#> Epoch 321/500
#> 7/7 - 0s - 3ms/step - loss: 0.0378 - mean_squared_error: 0.0378
#> Epoch 322/500
#> 7/7 - 0s - 3ms/step - loss: 0.0352 - mean_squared_error: 0.0352
#> Epoch 323/500
#> 7/7 - 0s - 3ms/step - loss: 0.0357 - mean_squared_error: 0.0357
#> Epoch 324/500
#> 7/7 - 0s - 3ms/step - loss: 0.0335 - mean_squared_error: 0.0335
#> Epoch 325/500
#> 7/7 - 0s - 3ms/step - loss: 0.0342 - mean_squared_error: 0.0342
#> Epoch 326/500
#> 7/7 - 0s - 3ms/step - loss: 0.0354 - mean_squared_error: 0.0354
#> Epoch 327/500
#> 7/7 - 0s - 3ms/step - loss: 0.0342 - mean_squared_error: 0.0342
#> Epoch 328/500
#> 7/7 - 0s - 3ms/step - loss: 0.0350 - mean_squared_error: 0.0350
#> Epoch 329/500
#> 7/7 - 0s - 3ms/step - loss: 0.0330 - mean_squared_error: 0.0330
#> Epoch 330/500
#> 7/7 - 0s - 3ms/step - loss: 0.0328 - mean_squared_error: 0.0328
#> Epoch 331/500
#> 7/7 - 0s - 3ms/step - loss: 0.0329 - mean_squared_error: 0.0329
#> Epoch 332/500
#> 7/7 - 0s - 3ms/step - loss: 0.0333 - mean_squared_error: 0.0333
#> Epoch 333/500
#> 7/7 - 0s - 3ms/step - loss: 0.0335 - mean_squared_error: 0.0335
#> Epoch 334/500
#> 7/7 - 0s - 3ms/step - loss: 0.0327 - mean_squared_error: 0.0327
#> Epoch 335/500
#> 7/7 - 0s - 3ms/step - loss: 0.0326 - mean_squared_error: 0.0326
#> Epoch 336/500
#> 7/7 - 0s - 4ms/step - loss: 0.0321 - mean_squared_error: 0.0321
#> Epoch 337/500
#> 7/7 - 0s - 3ms/step - loss: 0.0316 - mean_squared_error: 0.0316
#> Epoch 338/500
#> 7/7 - 0s - 3ms/step - loss: 0.0319 - mean_squared_error: 0.0319
#> Epoch 339/500
#> 7/7 - 0s - 3ms/step - loss: 0.0321 - mean_squared_error: 0.0321
#> Epoch 340/500
#> 7/7 - 0s - 3ms/step - loss: 0.0300 - mean_squared_error: 0.0300
#> Epoch 341/500
#> 7/7 - 0s - 3ms/step - loss: 0.0300 - mean_squared_error: 0.0300
#> Epoch 342/500
#> 7/7 - 0s - 3ms/step - loss: 0.0305 - mean_squared_error: 0.0305
#> Epoch 343/500
#> 7/7 - 0s - 3ms/step - loss: 0.0307 - mean_squared_error: 0.0307
#> Epoch 344/500
#> 7/7 - 0s - 3ms/step - loss: 0.0299 - mean_squared_error: 0.0299
#> Epoch 345/500
#> 7/7 - 0s - 3ms/step - loss: 0.0293 - mean_squared_error: 0.0293
#> Epoch 346/500
#> 7/7 - 0s - 3ms/step - loss: 0.0281 - mean_squared_error: 0.0281
#> Epoch 347/500
#> 7/7 - 0s - 3ms/step - loss: 0.0300 - mean_squared_error: 0.0300
#> Epoch 348/500
#> 7/7 - 0s - 3ms/step - loss: 0.0316 - mean_squared_error: 0.0316
#> Epoch 349/500
#> 7/7 - 0s - 3ms/step - loss: 0.0274 - mean_squared_error: 0.0274
#> Epoch 350/500
#> 7/7 - 0s - 3ms/step - loss: 0.0278 - mean_squared_error: 0.0278
#> Epoch 351/500
#> 7/7 - 0s - 3ms/step - loss: 0.0270 - mean_squared_error: 0.0270
#> Epoch 352/500
#> 7/7 - 0s - 4ms/step - loss: 0.0287 - mean_squared_error: 0.0287
#> Epoch 353/500
#> 7/7 - 0s - 3ms/step - loss: 0.0266 - mean_squared_error: 0.0266
#> Epoch 354/500
#> 7/7 - 0s - 3ms/step - loss: 0.0276 - mean_squared_error: 0.0276
#> Epoch 355/500
#> 7/7 - 0s - 3ms/step - loss: 0.0292 - mean_squared_error: 0.0292
#> Epoch 356/500
#> 7/7 - 0s - 3ms/step - loss: 0.0263 - mean_squared_error: 0.0263
#> Epoch 357/500
#> 7/7 - 0s - 3ms/step - loss: 0.0266 - mean_squared_error: 0.0266
#> Epoch 358/500
#> 7/7 - 0s - 3ms/step - loss: 0.0275 - mean_squared_error: 0.0275
#> Epoch 359/500
#> 7/7 - 0s - 3ms/step - loss: 0.0268 - mean_squared_error: 0.0268
#> Epoch 360/500
#> 7/7 - 0s - 3ms/step - loss: 0.0266 - mean_squared_error: 0.0266
#> Epoch 361/500
#> 7/7 - 0s - 3ms/step - loss: 0.0261 - mean_squared_error: 0.0261
#> Epoch 362/500
#> 7/7 - 0s - 3ms/step - loss: 0.0249 - mean_squared_error: 0.0249
#> Epoch 363/500
#> 7/7 - 0s - 3ms/step - loss: 0.0264 - mean_squared_error: 0.0264
#> Epoch 364/500
#> 7/7 - 0s - 3ms/step - loss: 0.0260 - mean_squared_error: 0.0260
#> Epoch 365/500
#> 7/7 - 0s - 3ms/step - loss: 0.0246 - mean_squared_error: 0.0246
#> Epoch 366/500
#> 7/7 - 0s - 3ms/step - loss: 0.0250 - mean_squared_error: 0.0250
#> Epoch 367/500
#> 7/7 - 0s - 3ms/step - loss: 0.0253 - mean_squared_error: 0.0253
#> Epoch 368/500
#> 7/7 - 0s - 3ms/step - loss: 0.0240 - mean_squared_error: 0.0240
#> Epoch 369/500
#> 7/7 - 0s - 3ms/step - loss: 0.0256 - mean_squared_error: 0.0256
#> Epoch 370/500
#> 7/7 - 0s - 3ms/step - loss: 0.0243 - mean_squared_error: 0.0243
#> Epoch 371/500
#> 7/7 - 0s - 4ms/step - loss: 0.0237 - mean_squared_error: 0.0237
#> Epoch 372/500
#> 7/7 - 0s - 3ms/step - loss: 0.0234 - mean_squared_error: 0.0234
#> Epoch 373/500
#> 7/7 - 0s - 3ms/step - loss: 0.0238 - mean_squared_error: 0.0238
#> Epoch 374/500
#> 7/7 - 0s - 3ms/step - loss: 0.0240 - mean_squared_error: 0.0240
#> Epoch 375/500
#> 7/7 - 0s - 4ms/step - loss: 0.0235 - mean_squared_error: 0.0235
#> Epoch 376/500
#> 7/7 - 0s - 3ms/step - loss: 0.0228 - mean_squared_error: 0.0228
#> Epoch 377/500
#> 7/7 - 0s - 3ms/step - loss: 0.0247 - mean_squared_error: 0.0247
#> Epoch 378/500
#> 7/7 - 0s - 3ms/step - loss: 0.0244 - mean_squared_error: 0.0244
#> Epoch 379/500
#> 7/7 - 0s - 3ms/step - loss: 0.0221 - mean_squared_error: 0.0221
#> Epoch 380/500
#> 7/7 - 0s - 3ms/step - loss: 0.0236 - mean_squared_error: 0.0236
#> Epoch 381/500
#> 7/7 - 0s - 3ms/step - loss: 0.0231 - mean_squared_error: 0.0231
#> Epoch 382/500
#> 7/7 - 0s - 3ms/step - loss: 0.0212 - mean_squared_error: 0.0212
#> Epoch 383/500
#> 7/7 - 0s - 3ms/step - loss: 0.0216 - mean_squared_error: 0.0216
#> Epoch 384/500
#> 7/7 - 0s - 3ms/step - loss: 0.0213 - mean_squared_error: 0.0213
#> Epoch 385/500
#> 7/7 - 0s - 3ms/step - loss: 0.0226 - mean_squared_error: 0.0226
#> Epoch 386/500
#> 7/7 - 0s - 3ms/step - loss: 0.0208 - mean_squared_error: 0.0208
#> Epoch 387/500
#> 7/7 - 0s - 3ms/step - loss: 0.0244 - mean_squared_error: 0.0244
#> Epoch 388/500
#> 7/7 - 0s - 3ms/step - loss: 0.0216 - mean_squared_error: 0.0216
#> Epoch 389/500
#> 7/7 - 0s - 3ms/step - loss: 0.0210 - mean_squared_error: 0.0210
#> Epoch 390/500
#> 7/7 - 0s - 3ms/step - loss: 0.0211 - mean_squared_error: 0.0211
#> Epoch 391/500
#> 7/7 - 0s - 3ms/step - loss: 0.0215 - mean_squared_error: 0.0215
#> Epoch 392/500
#> 7/7 - 0s - 4ms/step - loss: 0.0204 - mean_squared_error: 0.0204
#> Epoch 393/500
#> 7/7 - 0s - 3ms/step - loss: 0.0205 - mean_squared_error: 0.0205
#> Epoch 394/500
#> 7/7 - 0s - 3ms/step - loss: 0.0202 - mean_squared_error: 0.0202
#> Epoch 395/500
#> 7/7 - 0s - 3ms/step - loss: 0.0219 - mean_squared_error: 0.0219
#> Epoch 396/500
#> 7/7 - 0s - 4ms/step - loss: 0.0203 - mean_squared_error: 0.0203
#> Epoch 397/500
#> 7/7 - 0s - 3ms/step - loss: 0.0213 - mean_squared_error: 0.0213
#> Epoch 398/500
#> 7/7 - 0s - 3ms/step - loss: 0.0191 - mean_squared_error: 0.0191
#> Epoch 399/500
#> 7/7 - 0s - 4ms/step - loss: 0.0219 - mean_squared_error: 0.0219
#> Epoch 400/500
#> 7/7 - 0s - 3ms/step - loss: 0.0193 - mean_squared_error: 0.0193
#> Epoch 401/500
#> 7/7 - 0s - 3ms/step - loss: 0.0190 - mean_squared_error: 0.0190
#> Epoch 402/500
#> 7/7 - 0s - 4ms/step - loss: 0.0186 - mean_squared_error: 0.0186
#> Epoch 403/500
#> 7/7 - 0s - 3ms/step - loss: 0.0201 - mean_squared_error: 0.0201
#> Epoch 404/500
#> 7/7 - 0s - 3ms/step - loss: 0.0203 - mean_squared_error: 0.0203
#> Epoch 405/500
#> 7/7 - 0s - 3ms/step - loss: 0.0201 - mean_squared_error: 0.0201
#> Epoch 406/500
#> 7/7 - 0s - 3ms/step - loss: 0.0202 - mean_squared_error: 0.0202
#> Epoch 407/500
#> 7/7 - 0s - 3ms/step - loss: 0.0196 - mean_squared_error: 0.0196
#> Epoch 408/500
#> 7/7 - 0s - 3ms/step - loss: 0.0183 - mean_squared_error: 0.0183
#> Epoch 409/500
#> 7/7 - 0s - 3ms/step - loss: 0.0197 - mean_squared_error: 0.0197
#> Epoch 410/500
#> 7/7 - 0s - 3ms/step - loss: 0.0217 - mean_squared_error: 0.0217
#> Epoch 411/500
#> 7/7 - 0s - 3ms/step - loss: 0.0179 - mean_squared_error: 0.0179
#> Epoch 412/500
#> 7/7 - 0s - 3ms/step - loss: 0.0183 - mean_squared_error: 0.0183
#> Epoch 413/500
#> 7/7 - 0s - 3ms/step - loss: 0.0180 - mean_squared_error: 0.0180
#> Epoch 414/500
#> 7/7 - 0s - 3ms/step - loss: 0.0175 - mean_squared_error: 0.0175
#> Epoch 415/500
#> 7/7 - 0s - 3ms/step - loss: 0.0171 - mean_squared_error: 0.0171
#> Epoch 416/500
#> 7/7 - 0s - 3ms/step - loss: 0.0176 - mean_squared_error: 0.0176
#> Epoch 417/500
#> 7/7 - 0s - 3ms/step - loss: 0.0189 - mean_squared_error: 0.0189
#> Epoch 418/500
#> 7/7 - 0s - 3ms/step - loss: 0.0173 - mean_squared_error: 0.0173
#> Epoch 419/500
#> 7/7 - 0s - 3ms/step - loss: 0.0181 - mean_squared_error: 0.0181
#> Epoch 420/500
#> 7/7 - 0s - 3ms/step - loss: 0.0167 - mean_squared_error: 0.0167
#> Epoch 421/500
#> 7/7 - 0s - 3ms/step - loss: 0.0176 - mean_squared_error: 0.0176
#> Epoch 422/500
#> 7/7 - 0s - 3ms/step - loss: 0.0167 - mean_squared_error: 0.0167
#> Epoch 423/500
#> 7/7 - 0s - 3ms/step - loss: 0.0174 - mean_squared_error: 0.0174
#> Epoch 424/500
#> 7/7 - 0s - 3ms/step - loss: 0.0170 - mean_squared_error: 0.0170
#> Epoch 425/500
#> 7/7 - 0s - 3ms/step - loss: 0.0171 - mean_squared_error: 0.0171
#> Epoch 426/500
#> 7/7 - 0s - 3ms/step - loss: 0.0153 - mean_squared_error: 0.0153
#> Epoch 427/500
#> 7/7 - 0s - 3ms/step - loss: 0.0160 - mean_squared_error: 0.0160
#> Epoch 428/500
#> 7/7 - 0s - 3ms/step - loss: 0.0162 - mean_squared_error: 0.0162
#> Epoch 429/500
#> 7/7 - 0s - 3ms/step - loss: 0.0165 - mean_squared_error: 0.0165
#> Epoch 430/500
#> 7/7 - 0s - 3ms/step - loss: 0.0149 - mean_squared_error: 0.0149
#> Epoch 431/500
#> 7/7 - 0s - 3ms/step - loss: 0.0159 - mean_squared_error: 0.0159
#> Epoch 432/500
#> 7/7 - 0s - 9ms/step - loss: 0.0175 - mean_squared_error: 0.0175
#> Epoch 433/500
#> 7/7 - 0s - 3ms/step - loss: 0.0168 - mean_squared_error: 0.0168
#> Epoch 434/500
#> 7/7 - 0s - 3ms/step - loss: 0.0145 - mean_squared_error: 0.0145
#> Epoch 435/500
#> 7/7 - 0s - 3ms/step - loss: 0.0162 - mean_squared_error: 0.0162
#> Epoch 436/500
#> 7/7 - 0s - 3ms/step - loss: 0.0156 - mean_squared_error: 0.0156
#> Epoch 437/500
#> 7/7 - 0s - 4ms/step - loss: 0.0155 - mean_squared_error: 0.0155
#> Epoch 438/500
#> 7/7 - 0s - 4ms/step - loss: 0.0147 - mean_squared_error: 0.0147
#> Epoch 439/500
#> 7/7 - 0s - 3ms/step - loss: 0.0144 - mean_squared_error: 0.0144
#> Epoch 440/500
#> 7/7 - 0s - 3ms/step - loss: 0.0156 - mean_squared_error: 0.0156
#> Epoch 441/500
#> 7/7 - 0s - 3ms/step - loss: 0.0147 - mean_squared_error: 0.0147
#> Epoch 442/500
#> 7/7 - 0s - 3ms/step - loss: 0.0156 - mean_squared_error: 0.0156
#> Epoch 443/500
#> 7/7 - 0s - 3ms/step - loss: 0.0152 - mean_squared_error: 0.0152
#> Epoch 444/500
#> 7/7 - 0s - 3ms/step - loss: 0.0162 - mean_squared_error: 0.0162
#> Epoch 445/500
#> 7/7 - 0s - 3ms/step - loss: 0.0143 - mean_squared_error: 0.0143
#> Epoch 446/500
#> 7/7 - 0s - 4ms/step - loss: 0.0135 - mean_squared_error: 0.0135
#> Epoch 447/500
#> 7/7 - 0s - 3ms/step - loss: 0.0131 - mean_squared_error: 0.0131
#> Epoch 448/500
#> 7/7 - 0s - 3ms/step - loss: 0.0135 - mean_squared_error: 0.0135
#> Epoch 449/500
#> 7/7 - 0s - 3ms/step - loss: 0.0132 - mean_squared_error: 0.0132
#> Epoch 450/500
#> 7/7 - 0s - 3ms/step - loss: 0.0142 - mean_squared_error: 0.0142
#> Epoch 451/500
#> 7/7 - 0s - 3ms/step - loss: 0.0135 - mean_squared_error: 0.0135
#> Epoch 452/500
#> 7/7 - 0s - 3ms/step - loss: 0.0134 - mean_squared_error: 0.0134
#> Epoch 453/500
#> 7/7 - 0s - 3ms/step - loss: 0.0133 - mean_squared_error: 0.0133
#> Epoch 454/500
#> 7/7 - 0s - 3ms/step - loss: 0.0153 - mean_squared_error: 0.0153
#> Epoch 455/500
#> 7/7 - 0s - 3ms/step - loss: 0.0139 - mean_squared_error: 0.0139
#> Epoch 456/500
#> 7/7 - 0s - 3ms/step - loss: 0.0129 - mean_squared_error: 0.0129
#> Epoch 457/500
#> 7/7 - 0s - 3ms/step - loss: 0.0154 - mean_squared_error: 0.0154
#> Epoch 458/500
#> 7/7 - 0s - 3ms/step - loss: 0.0123 - mean_squared_error: 0.0123
#> Epoch 459/500
#> 7/7 - 0s - 3ms/step - loss: 0.0133 - mean_squared_error: 0.0133
#> Epoch 460/500
#> 7/7 - 0s - 4ms/step - loss: 0.0120 - mean_squared_error: 0.0120
#> Epoch 461/500
#> 7/7 - 0s - 3ms/step - loss: 0.0146 - mean_squared_error: 0.0146
#> Epoch 462/500
#> 7/7 - 0s - 3ms/step - loss: 0.0135 - mean_squared_error: 0.0135
#> Epoch 463/500
#> 7/7 - 0s - 3ms/step - loss: 0.0129 - mean_squared_error: 0.0129
#> Epoch 464/500
#> 7/7 - 0s - 3ms/step - loss: 0.0112 - mean_squared_error: 0.0112
#> Epoch 465/500
#> 7/7 - 0s - 3ms/step - loss: 0.0135 - mean_squared_error: 0.0135
#> Epoch 466/500
#> 7/7 - 0s - 3ms/step - loss: 0.0115 - mean_squared_error: 0.0115
#> Epoch 467/500
#> 7/7 - 0s - 3ms/step - loss: 0.0129 - mean_squared_error: 0.0129
#> Epoch 468/500
#> 7/7 - 0s - 3ms/step - loss: 0.0125 - mean_squared_error: 0.0125
#> Epoch 469/500
#> 7/7 - 0s - 3ms/step - loss: 0.0124 - mean_squared_error: 0.0124
#> Epoch 470/500
#> 7/7 - 0s - 4ms/step - loss: 0.0126 - mean_squared_error: 0.0126
#> Epoch 471/500
#> 7/7 - 0s - 3ms/step - loss: 0.0131 - mean_squared_error: 0.0131
#> Epoch 472/500
#> 7/7 - 0s - 3ms/step - loss: 0.0123 - mean_squared_error: 0.0123
#> Epoch 473/500
#> 7/7 - 0s - 3ms/step - loss: 0.0118 - mean_squared_error: 0.0118
#> Epoch 474/500
#> 7/7 - 0s - 3ms/step - loss: 0.0138 - mean_squared_error: 0.0138
#> Epoch 475/500
#> 7/7 - 0s - 3ms/step - loss: 0.0112 - mean_squared_error: 0.0112
#> Epoch 476/500
#> 7/7 - 0s - 3ms/step - loss: 0.0111 - mean_squared_error: 0.0111
#> Epoch 477/500
#> 7/7 - 0s - 3ms/step - loss: 0.0116 - mean_squared_error: 0.0116
#> Epoch 478/500
#> 7/7 - 0s - 3ms/step - loss: 0.0124 - mean_squared_error: 0.0124
#> Epoch 479/500
#> 7/7 - 0s - 3ms/step - loss: 0.0118 - mean_squared_error: 0.0118
#> Epoch 480/500
#> 7/7 - 0s - 3ms/step - loss: 0.0113 - mean_squared_error: 0.0113
#> Epoch 481/500
#> 7/7 - 0s - 4ms/step - loss: 0.0114 - mean_squared_error: 0.0114
#> Epoch 482/500
#> 7/7 - 0s - 4ms/step - loss: 0.0108 - mean_squared_error: 0.0108
#> Epoch 483/500
#> 7/7 - 0s - 3ms/step - loss: 0.0127 - mean_squared_error: 0.0127
#> Epoch 484/500
#> 7/7 - 0s - 3ms/step - loss: 0.0125 - mean_squared_error: 0.0125
#> Epoch 485/500
#> 7/7 - 0s - 3ms/step - loss: 0.0110 - mean_squared_error: 0.0110
#> Epoch 486/500
#> 7/7 - 0s - 3ms/step - loss: 0.0115 - mean_squared_error: 0.0115
#> Epoch 487/500
#> 7/7 - 0s - 3ms/step - loss: 0.0117 - mean_squared_error: 0.0117
#> Epoch 488/500
#> 7/7 - 0s - 3ms/step - loss: 0.0098 - mean_squared_error: 0.0098
#> Epoch 489/500
#> 7/7 - 0s - 3ms/step - loss: 0.0106 - mean_squared_error: 0.0106
#> Epoch 490/500
#> 7/7 - 0s - 3ms/step - loss: 0.0110 - mean_squared_error: 0.0110
#> Epoch 491/500
#> 7/7 - 0s - 3ms/step - loss: 0.0110 - mean_squared_error: 0.0110
#> Epoch 492/500
#> 7/7 - 0s - 4ms/step - loss: 0.0113 - mean_squared_error: 0.0113
#> Epoch 493/500
#> 7/7 - 0s - 3ms/step - loss: 0.0106 - mean_squared_error: 0.0106
#> Epoch 494/500
#> 7/7 - 0s - 3ms/step - loss: 0.0103 - mean_squared_error: 0.0103
#> Epoch 495/500
#> 7/7 - 0s - 3ms/step - loss: 0.0129 - mean_squared_error: 0.0129
#> Epoch 496/500
#> 7/7 - 0s - 3ms/step - loss: 0.0099 - mean_squared_error: 0.0099
#> Epoch 497/500
#> 7/7 - 0s - 3ms/step - loss: 0.0132 - mean_squared_error: 0.0132
#> Epoch 498/500
#> 7/7 - 0s - 3ms/step - loss: 0.0097 - mean_squared_error: 0.0097
#> Epoch 499/500
#> 7/7 - 0s - 3ms/step - loss: 0.0103 - mean_squared_error: 0.0103
#> Epoch 500/500
#> 7/7 - 0s - 3ms/step - loss: 0.0098 - mean_squared_error: 0.0098TensorFlow models are fast because they are compiled: the computer takes the very flexible instructions within R and then turns them into inflexible computer instructions that can’t be changed but can run much more quickly. To complete our model, TensorFlow needs to know how to tell when the model is doing poorly—it needs a definition of error so it can propagate that error back through the network. TensorFlow is much more sophisticated and flexible than the package we were using in our previous session, so we have a lot more options here and it can’t just pick for us. I’ve specified this measured loss of predictive power as the mean squared error (just the same as what we used last time)—this is exactly the same definition of residual error as used in standard linear regression. I then ask TensorFlow to use a built-in optimizer—the way it propagates error throughout the network and trains the model—that is essentially the default option for squared error (optimizer_rmsprop). Notice that this optimizer is a function, and so you can write your own if you wish (…please don’t in this class…), and this flexibility is one of the reasons I’m not going into the details of the fitting process for TensorFlow in this class. Finally, I ask it to report and record metrics on its fit to the training data, in this case the same as the one I’m asking it to use in the fitting process. Finally, I fit the model to data, using \(500\) training iterations, which as we discussed before are called epochs. As ever, let’s validate out model in independent test data.
plot(predict(model, exp[-training,])[,1] ~ resp[-training])
#> 7/7 - 0s - 6ms/step
cor.test(predict(model, exp[-training,])[,1], resp[-training])
#> 7/7 - 0s - 3ms/step
#>
#> Pearson's product-moment correlation
#>
#> data: predict(model, exp[-training, ])[, 1] and resp[-training]
#> t = 17.482, df = 198, p-value < 2.2e-16
#> alternative hypothesis: true correlation is not equal to 0
#> 95 percent confidence interval:
#> 0.7178518 0.8282326
#> sample estimates:
#> cor
#> 0.7790072I would suggest that the model has done quite well. So, let’s finish up our tour of TensorFlow by unpacking some of the options it provides for different activation functions and node types.
10.1.3 Tweaking all the dials
Activation functions are often the special sauce of networks, and so it’s worth knowing a little bit about the options you have available. Some common ones are:
linear. The identity function: what you put in is exactly what you get out. Generally only useful if you want something that is one layer deep and is mathematically identical to a multiple regression (see previous session). Thus, if you find yourself using this, ask yourself whether you need to be using TensorFlow at all.softmax. This is often described online as if it’s some sort of magic function that turns TensorFlow into either a Bayesian or a frequentist approach; this is not true. The softmax function is a generalization of the logistic function to deal with high-dimensional data: it takes multiple inputs and then smoothly rescales them to a number between \(0\) and \(1\). It’s defined as \(\frac{e^{x}}{\sum{e^x_i}}\) for a load of \(i\) input variables (\(x\)). If given ten input variables, it will exponentiate those input variables and divide them all by the sum of the exponents. It’s really useful as the last step in a model where you’re trying to predict whether the input data is one of a number of things (e.g., cute kitten, very cute kitten, or something else) because it will give what you can treat as predicted probabilities of each of those categories.relu. The rectifier function; there are many related functions. It’s formally defined as \(max(0, x)\), and it essentially means “throw away all the negative values and leave everything else unchanged”. There are ‘smooth’ variants of it (\(log(1 + e^x)\)), leaky variants of it (if \(x > 0\) then \(x\), otherwise \(0.01x\)), and they’re all very important because they really do help models train faster. Small differences among these families can matter a great deal—use whichever one is generally recommended for your types of problems, or just experiment and see what happens.
There are also quite a few different options for loss functions:
mean_absolute_error. This one’s the absolute error, which we can’t use so easily in standard statistical models, but is fair game in machine learning. Avoid if you like mapping things onto Normal distributions and the like, but use if you like simplicity.logcosh. \(log(\cosh(error))\), which basically means “the same as the mean squared error but not so affected by outliers”. Much more helpful than you might think, because if you knew what your outliers were ahead of time you might not need an artificial neural network!kullback_leibler_divergence. If you have taken an advanced class in model-averaging and AIC techniques, you might be excited to see this. If not, note that you can almost get a likelihood out of a network this way.poisson. I put this here as a reminder that you can essentially fit Generalized Linear Models in TensorFlow (if you’ve taken my class, note that).categorical_crossentropy. A useful option if you’re doing some categorical models (there are other options for that in Keras). It’s essentially \(-\sum{p_i}\) where \(p_i\) is the probability ascribed to the correct label in your dataset. Thus if you had ten data-points and your model estimate a probability of \(1\) that each was the correct image, then your cross-entropy would be at its lowest possible value (\(-10\)). There is a correction of this for when you don’t know the true value (i.e., you’re applying your model to new data).
We will be covering different kinds of network architecture next time, so I don’t want to dwell on that now, but I do want to show you the very cool training dashboard that comes with Keras. If, while training, you tell Keras where to store log-files, you can visualize them later. Try running the following, for example:
model %>% fit(
exp[training,], resp[training], epochs=500,
callbacks = callback_tensorboard("folder/on/your/computer")
)
#> Epoch 1/500
#> 7/7 - 0s - 6ms/step - loss: 0.0098 - mean_squared_error: 0.0098
#> Epoch 2/500
#> 7/7 - 0s - 3ms/step - loss: 0.0106 - mean_squared_error: 0.0106
#> Epoch 3/500
#> 7/7 - 0s - 4ms/step - loss: 0.0107 - mean_squared_error: 0.0107
#> Epoch 4/500
#> 7/7 - 0s - 4ms/step - loss: 0.0110 - mean_squared_error: 0.0110
#> Epoch 5/500
#> 7/7 - 0s - 3ms/step - loss: 0.0097 - mean_squared_error: 0.0097
#> Epoch 6/500
#> 7/7 - 0s - 4ms/step - loss: 0.0098 - mean_squared_error: 0.0098
#> Epoch 7/500
#> 7/7 - 0s - 3ms/step - loss: 0.0087 - mean_squared_error: 0.0087
#> Epoch 8/500
#> 7/7 - 0s - 4ms/step - loss: 0.0104 - mean_squared_error: 0.0104
#> Epoch 9/500
#> 7/7 - 0s - 4ms/step - loss: 0.0099 - mean_squared_error: 0.0099
#> Epoch 10/500
#> 7/7 - 0s - 3ms/step - loss: 0.0091 - mean_squared_error: 0.0091
#> Epoch 11/500
#> 7/7 - 0s - 3ms/step - loss: 0.0106 - mean_squared_error: 0.0106
#> Epoch 12/500
#> 7/7 - 0s - 3ms/step - loss: 0.0094 - mean_squared_error: 0.0094
#> Epoch 13/500
#> 7/7 - 0s - 3ms/step - loss: 0.0090 - mean_squared_error: 0.0090
#> Epoch 14/500
#> 7/7 - 0s - 3ms/step - loss: 0.0096 - mean_squared_error: 0.0096
#> Epoch 15/500
#> 7/7 - 0s - 3ms/step - loss: 0.0100 - mean_squared_error: 0.0100
#> Epoch 16/500
#> 7/7 - 0s - 3ms/step - loss: 0.0094 - mean_squared_error: 0.0094
#> Epoch 17/500
#> 7/7 - 0s - 3ms/step - loss: 0.0092 - mean_squared_error: 0.0092
#> Epoch 18/500
#> 7/7 - 0s - 3ms/step - loss: 0.0088 - mean_squared_error: 0.0088
#> Epoch 19/500
#> 7/7 - 0s - 3ms/step - loss: 0.0089 - mean_squared_error: 0.0089
#> Epoch 20/500
#> 7/7 - 0s - 4ms/step - loss: 0.0120 - mean_squared_error: 0.0120
#> Epoch 21/500
#> 7/7 - 0s - 3ms/step - loss: 0.0081 - mean_squared_error: 0.0081
#> Epoch 22/500
#> 7/7 - 0s - 4ms/step - loss: 0.0082 - mean_squared_error: 0.0082
#> Epoch 23/500
#> 7/7 - 0s - 4ms/step - loss: 0.0107 - mean_squared_error: 0.0107
#> Epoch 24/500
#> 7/7 - 0s - 4ms/step - loss: 0.0085 - mean_squared_error: 0.0085
#> Epoch 25/500
#> 7/7 - 0s - 3ms/step - loss: 0.0087 - mean_squared_error: 0.0087
#> Epoch 26/500
#> 7/7 - 0s - 4ms/step - loss: 0.0094 - mean_squared_error: 0.0094
#> Epoch 27/500
#> 7/7 - 0s - 3ms/step - loss: 0.0114 - mean_squared_error: 0.0114
#> Epoch 28/500
#> 7/7 - 0s - 3ms/step - loss: 0.0086 - mean_squared_error: 0.0086
#> Epoch 29/500
#> 7/7 - 0s - 4ms/step - loss: 0.0081 - mean_squared_error: 0.0081
#> Epoch 30/500
#> 7/7 - 0s - 4ms/step - loss: 0.0093 - mean_squared_error: 0.0093
#> Epoch 31/500
#> 7/7 - 0s - 3ms/step - loss: 0.0084 - mean_squared_error: 0.0084
#> Epoch 32/500
#> 7/7 - 0s - 4ms/step - loss: 0.0086 - mean_squared_error: 0.0086
#> Epoch 33/500
#> 7/7 - 0s - 3ms/step - loss: 0.0087 - mean_squared_error: 0.0087
#> Epoch 34/500
#> 7/7 - 0s - 3ms/step - loss: 0.0098 - mean_squared_error: 0.0098
#> Epoch 35/500
#> 7/7 - 0s - 3ms/step - loss: 0.0099 - mean_squared_error: 0.0099
#> Epoch 36/500
#> 7/7 - 0s - 3ms/step - loss: 0.0078 - mean_squared_error: 0.0078
#> Epoch 37/500
#> 7/7 - 0s - 3ms/step - loss: 0.0077 - mean_squared_error: 0.0077
#> Epoch 38/500
#> 7/7 - 0s - 4ms/step - loss: 0.0085 - mean_squared_error: 0.0085
#> Epoch 39/500
#> 7/7 - 0s - 3ms/step - loss: 0.0075 - mean_squared_error: 0.0075
#> Epoch 40/500
#> 7/7 - 0s - 3ms/step - loss: 0.0076 - mean_squared_error: 0.0076
#> Epoch 41/500
#> 7/7 - 0s - 3ms/step - loss: 0.0100 - mean_squared_error: 0.0100
#> Epoch 42/500
#> 7/7 - 0s - 3ms/step - loss: 0.0076 - mean_squared_error: 0.0076
#> Epoch 43/500
#> 7/7 - 0s - 3ms/step - loss: 0.0078 - mean_squared_error: 0.0078
#> Epoch 44/500
#> 7/7 - 0s - 3ms/step - loss: 0.0079 - mean_squared_error: 0.0079
#> Epoch 45/500
#> 7/7 - 0s - 3ms/step - loss: 0.0081 - mean_squared_error: 0.0081
#> Epoch 46/500
#> 7/7 - 0s - 3ms/step - loss: 0.0084 - mean_squared_error: 0.0084
#> Epoch 47/500
#> 7/7 - 0s - 4ms/step - loss: 0.0080 - mean_squared_error: 0.0080
#> Epoch 48/500
#> 7/7 - 0s - 3ms/step - loss: 0.0084 - mean_squared_error: 0.0084
#> Epoch 49/500
#> 7/7 - 0s - 3ms/step - loss: 0.0081 - mean_squared_error: 0.0081
#> Epoch 50/500
#> 7/7 - 0s - 3ms/step - loss: 0.0099 - mean_squared_error: 0.0099
#> Epoch 51/500
#> 7/7 - 0s - 4ms/step - loss: 0.0074 - mean_squared_error: 0.0074
#> Epoch 52/500
#> 7/7 - 0s - 3ms/step - loss: 0.0094 - mean_squared_error: 0.0094
#> Epoch 53/500
#> 7/7 - 0s - 4ms/step - loss: 0.0083 - mean_squared_error: 0.0083
#> Epoch 54/500
#> 7/7 - 0s - 3ms/step - loss: 0.0080 - mean_squared_error: 0.0080
#> Epoch 55/500
#> 7/7 - 0s - 3ms/step - loss: 0.0092 - mean_squared_error: 0.0092
#> Epoch 56/500
#> 7/7 - 0s - 3ms/step - loss: 0.0080 - mean_squared_error: 0.0080
#> Epoch 57/500
#> 7/7 - 0s - 3ms/step - loss: 0.0072 - mean_squared_error: 0.0072
#> Epoch 58/500
#> 7/7 - 0s - 3ms/step - loss: 0.0084 - mean_squared_error: 0.0084
#> Epoch 59/500
#> 7/7 - 0s - 3ms/step - loss: 0.0072 - mean_squared_error: 0.0072
#> Epoch 60/500
#> 7/7 - 0s - 3ms/step - loss: 0.0074 - mean_squared_error: 0.0074
#> Epoch 61/500
#> 7/7 - 0s - 3ms/step - loss: 0.0086 - mean_squared_error: 0.0086
#> Epoch 62/500
#> 7/7 - 0s - 4ms/step - loss: 0.0080 - mean_squared_error: 0.0080
#> Epoch 63/500
#> 7/7 - 0s - 3ms/step - loss: 0.0076 - mean_squared_error: 0.0076
#> Epoch 64/500
#> 7/7 - 0s - 3ms/step - loss: 0.0075 - mean_squared_error: 0.0075
#> Epoch 65/500
#> 7/7 - 0s - 3ms/step - loss: 0.0069 - mean_squared_error: 0.0069
#> Epoch 66/500
#> 7/7 - 0s - 3ms/step - loss: 0.0091 - mean_squared_error: 0.0091
#> Epoch 67/500
#> 7/7 - 0s - 3ms/step - loss: 0.0073 - mean_squared_error: 0.0073
#> Epoch 68/500
#> 7/7 - 0s - 3ms/step - loss: 0.0075 - mean_squared_error: 0.0075
#> Epoch 69/500
#> 7/7 - 0s - 3ms/step - loss: 0.0082 - mean_squared_error: 0.0082
#> Epoch 70/500
#> 7/7 - 0s - 3ms/step - loss: 0.0074 - mean_squared_error: 0.0074
#> Epoch 71/500
#> 7/7 - 0s - 3ms/step - loss: 0.0072 - mean_squared_error: 0.0072
#> Epoch 72/500
#> 7/7 - 0s - 6ms/step - loss: 0.0065 - mean_squared_error: 0.0065
#> Epoch 73/500
#> 7/7 - 0s - 3ms/step - loss: 0.0078 - mean_squared_error: 0.0078
#> Epoch 74/500
#> 7/7 - 0s - 3ms/step - loss: 0.0083 - mean_squared_error: 0.0083
#> Epoch 75/500
#> 7/7 - 0s - 3ms/step - loss: 0.0079 - mean_squared_error: 0.0079
#> Epoch 76/500
#> 7/7 - 0s - 4ms/step - loss: 0.0070 - mean_squared_error: 0.0070
#> Epoch 77/500
#> 7/7 - 0s - 3ms/step - loss: 0.0080 - mean_squared_error: 0.0080
#> Epoch 78/500
#> 7/7 - 0s - 3ms/step - loss: 0.0071 - mean_squared_error: 0.0071
#> Epoch 79/500
#> 7/7 - 0s - 3ms/step - loss: 0.0068 - mean_squared_error: 0.0068
#> Epoch 80/500
#> 7/7 - 0s - 3ms/step - loss: 0.0076 - mean_squared_error: 0.0076
#> Epoch 81/500
#> 7/7 - 0s - 3ms/step - loss: 0.0068 - mean_squared_error: 0.0068
#> Epoch 82/500
#> 7/7 - 0s - 3ms/step - loss: 0.0079 - mean_squared_error: 0.0079
#> Epoch 83/500
#> 7/7 - 0s - 3ms/step - loss: 0.0076 - mean_squared_error: 0.0076
#> Epoch 84/500
#> 7/7 - 0s - 4ms/step - loss: 0.0068 - mean_squared_error: 0.0068
#> Epoch 85/500
#> 7/7 - 0s - 3ms/step - loss: 0.0083 - mean_squared_error: 0.0083
#> Epoch 86/500
#> 7/7 - 0s - 3ms/step - loss: 0.0083 - mean_squared_error: 0.0083
#> Epoch 87/500
#> 7/7 - 0s - 3ms/step - loss: 0.0065 - mean_squared_error: 0.0065
#> Epoch 88/500
#> 7/7 - 0s - 3ms/step - loss: 0.0078 - mean_squared_error: 0.0078
#> Epoch 89/500
#> 7/7 - 0s - 3ms/step - loss: 0.0066 - mean_squared_error: 0.0066
#> Epoch 90/500
#> 7/7 - 0s - 3ms/step - loss: 0.0071 - mean_squared_error: 0.0071
#> Epoch 91/500
#> 7/7 - 0s - 4ms/step - loss: 0.0055 - mean_squared_error: 0.0055
#> Epoch 92/500
#> 7/7 - 0s - 3ms/step - loss: 0.0056 - mean_squared_error: 0.0056
#> Epoch 93/500
#> 7/7 - 0s - 3ms/step - loss: 0.0077 - mean_squared_error: 0.0077
#> Epoch 94/500
#> 7/7 - 0s - 3ms/step - loss: 0.0072 - mean_squared_error: 0.0072
#> Epoch 95/500
#> 7/7 - 0s - 3ms/step - loss: 0.0081 - mean_squared_error: 0.0081
#> Epoch 96/500
#> 7/7 - 0s - 3ms/step - loss: 0.0060 - mean_squared_error: 0.0060
#> Epoch 97/500
#> 7/7 - 0s - 4ms/step - loss: 0.0058 - mean_squared_error: 0.0058
#> Epoch 98/500
#> 7/7 - 0s - 3ms/step - loss: 0.0079 - mean_squared_error: 0.0079
#> Epoch 99/500
#> 7/7 - 0s - 3ms/step - loss: 0.0056 - mean_squared_error: 0.0056
#> Epoch 100/500
#> 7/7 - 0s - 3ms/step - loss: 0.0062 - mean_squared_error: 0.0062
#> Epoch 101/500
#> 7/7 - 0s - 3ms/step - loss: 0.0062 - mean_squared_error: 0.0062
#> Epoch 102/500
#> 7/7 - 0s - 3ms/step - loss: 0.0067 - mean_squared_error: 0.0067
#> Epoch 103/500
#> 7/7 - 0s - 3ms/step - loss: 0.0075 - mean_squared_error: 0.0075
#> Epoch 104/500
#> 7/7 - 0s - 3ms/step - loss: 0.0069 - mean_squared_error: 0.0069
#> Epoch 105/500
#> 7/7 - 0s - 4ms/step - loss: 0.0067 - mean_squared_error: 0.0067
#> Epoch 106/500
#> 7/7 - 0s - 3ms/step - loss: 0.0054 - mean_squared_error: 0.0054
#> Epoch 107/500
#> 7/7 - 0s - 3ms/step - loss: 0.0063 - mean_squared_error: 0.0063
#> Epoch 108/500
#> 7/7 - 0s - 3ms/step - loss: 0.0061 - mean_squared_error: 0.0061
#> Epoch 109/500
#> 7/7 - 0s - 3ms/step - loss: 0.0085 - mean_squared_error: 0.0085
#> Epoch 110/500
#> 7/7 - 0s - 3ms/step - loss: 0.0051 - mean_squared_error: 0.0051
#> Epoch 111/500
#> 7/7 - 0s - 3ms/step - loss: 0.0056 - mean_squared_error: 0.0056
#> Epoch 112/500
#> 7/7 - 0s - 3ms/step - loss: 0.0058 - mean_squared_error: 0.0058
#> Epoch 113/500
#> 7/7 - 0s - 3ms/step - loss: 0.0069 - mean_squared_error: 0.0069
#> Epoch 114/500
#> 7/7 - 0s - 3ms/step - loss: 0.0055 - mean_squared_error: 0.0055
#> Epoch 115/500
#> 7/7 - 0s - 3ms/step - loss: 0.0070 - mean_squared_error: 0.0070
#> Epoch 116/500
#> 7/7 - 0s - 4ms/step - loss: 0.0070 - mean_squared_error: 0.0070
#> Epoch 117/500
#> 7/7 - 0s - 3ms/step - loss: 0.0069 - mean_squared_error: 0.0069
#> Epoch 118/500
#> 7/7 - 0s - 3ms/step - loss: 0.0061 - mean_squared_error: 0.0061
#> Epoch 119/500
#> 7/7 - 0s - 4ms/step - loss: 0.0071 - mean_squared_error: 0.0071
#> Epoch 120/500
#> 7/7 - 0s - 3ms/step - loss: 0.0059 - mean_squared_error: 0.0059
#> Epoch 121/500
#> 7/7 - 0s - 3ms/step - loss: 0.0057 - mean_squared_error: 0.0057
#> Epoch 122/500
#> 7/7 - 0s - 3ms/step - loss: 0.0070 - mean_squared_error: 0.0070
#> Epoch 123/500
#> 7/7 - 0s - 3ms/step - loss: 0.0062 - mean_squared_error: 0.0062
#> Epoch 124/500
#> 7/7 - 0s - 3ms/step - loss: 0.0051 - mean_squared_error: 0.0051
#> Epoch 125/500
#> 7/7 - 0s - 3ms/step - loss: 0.0067 - mean_squared_error: 0.0067
#> Epoch 126/500
#> 7/7 - 0s - 4ms/step - loss: 0.0058 - mean_squared_error: 0.0058
#> Epoch 127/500
#> 7/7 - 0s - 3ms/step - loss: 0.0065 - mean_squared_error: 0.0065
#> Epoch 128/500
#> 7/7 - 0s - 3ms/step - loss: 0.0057 - mean_squared_error: 0.0057
#> Epoch 129/500
#> 7/7 - 0s - 3ms/step - loss: 0.0056 - mean_squared_error: 0.0056
#> Epoch 130/500
#> 7/7 - 0s - 3ms/step - loss: 0.0052 - mean_squared_error: 0.0052
#> Epoch 131/500
#> 7/7 - 0s - 3ms/step - loss: 0.0065 - mean_squared_error: 0.0065
#> Epoch 132/500
#> 7/7 - 0s - 3ms/step - loss: 0.0054 - mean_squared_error: 0.0054
#> Epoch 133/500
#> 7/7 - 0s - 3ms/step - loss: 0.0054 - mean_squared_error: 0.0054
#> Epoch 134/500
#> 7/7 - 0s - 3ms/step - loss: 0.0058 - mean_squared_error: 0.0058
#> Epoch 135/500
#> 7/7 - 0s - 4ms/step - loss: 0.0063 - mean_squared_error: 0.0063
#> Epoch 136/500
#> 7/7 - 0s - 3ms/step - loss: 0.0065 - mean_squared_error: 0.0065
#> Epoch 137/500
#> 7/7 - 0s - 3ms/step - loss: 0.0055 - mean_squared_error: 0.0055
#> Epoch 138/500
#> 7/7 - 0s - 3ms/step - loss: 0.0050 - mean_squared_error: 0.0050
#> Epoch 139/500
#> 7/7 - 0s - 3ms/step - loss: 0.0059 - mean_squared_error: 0.0059
#> Epoch 140/500
#> 7/7 - 0s - 3ms/step - loss: 0.0056 - mean_squared_error: 0.0056
#> Epoch 141/500
#> 7/7 - 0s - 3ms/step - loss: 0.0050 - mean_squared_error: 0.0050
#> Epoch 142/500
#> 7/7 - 0s - 3ms/step - loss: 0.0083 - mean_squared_error: 0.0083
#> Epoch 143/500
#> 7/7 - 0s - 3ms/step - loss: 0.0047 - mean_squared_error: 0.0047
#> Epoch 144/500
#> 7/7 - 0s - 3ms/step - loss: 0.0056 - mean_squared_error: 0.0056
#> Epoch 145/500
#> 7/7 - 0s - 3ms/step - loss: 0.0054 - mean_squared_error: 0.0054
#> Epoch 146/500
#> 7/7 - 0s - 3ms/step - loss: 0.0052 - mean_squared_error: 0.0052
#> Epoch 147/500
#> 7/7 - 0s - 3ms/step - loss: 0.0055 - mean_squared_error: 0.0055
#> Epoch 148/500
#> 7/7 - 0s - 4ms/step - loss: 0.0057 - mean_squared_error: 0.0057
#> Epoch 149/500
#> 7/7 - 0s - 3ms/step - loss: 0.0055 - mean_squared_error: 0.0055
#> Epoch 150/500
#> 7/7 - 0s - 3ms/step - loss: 0.0043 - mean_squared_error: 0.0043
#> Epoch 151/500
#> 7/7 - 0s - 4ms/step - loss: 0.0063 - mean_squared_error: 0.0063
#> Epoch 152/500
#> 7/7 - 0s - 3ms/step - loss: 0.0068 - mean_squared_error: 0.0068
#> Epoch 153/500
#> 7/7 - 0s - 3ms/step - loss: 0.0058 - mean_squared_error: 0.0058
#> Epoch 154/500
#> 7/7 - 0s - 3ms/step - loss: 0.0042 - mean_squared_error: 0.0042
#> Epoch 155/500
#> 7/7 - 0s - 3ms/step - loss: 0.0042 - mean_squared_error: 0.0042
#> Epoch 156/500
#> 7/7 - 0s - 3ms/step - loss: 0.0069 - mean_squared_error: 0.0069
#> Epoch 157/500
#> 7/7 - 0s - 3ms/step - loss: 0.0058 - mean_squared_error: 0.0058
#> Epoch 158/500
#> 7/7 - 0s - 3ms/step - loss: 0.0045 - mean_squared_error: 0.0045
#> Epoch 159/500
#> 7/7 - 0s - 4ms/step - loss: 0.0071 - mean_squared_error: 0.0071
#> Epoch 160/500
#> 7/7 - 0s - 3ms/step - loss: 0.0051 - mean_squared_error: 0.0051
#> Epoch 161/500
#> 7/7 - 0s - 4ms/step - loss: 0.0050 - mean_squared_error: 0.0050
#> Epoch 162/500
#> 7/7 - 0s - 3ms/step - loss: 0.0058 - mean_squared_error: 0.0058
#> Epoch 163/500
#> 7/7 - 0s - 3ms/step - loss: 0.0045 - mean_squared_error: 0.0045
#> Epoch 164/500
#> 7/7 - 0s - 3ms/step - loss: 0.0064 - mean_squared_error: 0.0064
#> Epoch 165/500
#> 7/7 - 0s - 3ms/step - loss: 0.0050 - mean_squared_error: 0.0050
#> Epoch 166/500
#> 7/7 - 0s - 3ms/step - loss: 0.0049 - mean_squared_error: 0.0049
#> Epoch 167/500
#> 7/7 - 0s - 4ms/step - loss: 0.0063 - mean_squared_error: 0.0063
#> Epoch 168/500
#> 7/7 - 0s - 4ms/step - loss: 0.0045 - mean_squared_error: 0.0045
#> Epoch 169/500
#> 7/7 - 0s - 3ms/step - loss: 0.0063 - mean_squared_error: 0.0063
#> Epoch 170/500
#> 7/7 - 0s - 3ms/step - loss: 0.0049 - mean_squared_error: 0.0049
#> Epoch 171/500
#> 7/7 - 0s - 3ms/step - loss: 0.0046 - mean_squared_error: 0.0046
#> Epoch 172/500
#> 7/7 - 0s - 3ms/step - loss: 0.0056 - mean_squared_error: 0.0056
#> Epoch 173/500
#> 7/7 - 0s - 4ms/step - loss: 0.0063 - mean_squared_error: 0.0063
#> Epoch 174/500
#> 7/7 - 0s - 4ms/step - loss: 0.0047 - mean_squared_error: 0.0047
#> Epoch 175/500
#> 7/7 - 0s - 3ms/step - loss: 0.0058 - mean_squared_error: 0.0058
#> Epoch 176/500
#> 7/7 - 0s - 3ms/step - loss: 0.0054 - mean_squared_error: 0.0054
#> Epoch 177/500
#> 7/7 - 0s - 3ms/step - loss: 0.0052 - mean_squared_error: 0.0052
#> Epoch 178/500
#> 7/7 - 0s - 3ms/step - loss: 0.0046 - mean_squared_error: 0.0046
#> Epoch 179/500
#> 7/7 - 0s - 3ms/step - loss: 0.0047 - mean_squared_error: 0.0047
#> Epoch 180/500
#> 7/7 - 0s - 3ms/step - loss: 0.0044 - mean_squared_error: 0.0044
#> Epoch 181/500
#> 7/7 - 0s - 3ms/step - loss: 0.0080 - mean_squared_error: 0.0080
#> Epoch 182/500
#> 7/7 - 0s - 3ms/step - loss: 0.0043 - mean_squared_error: 0.0043
#> Epoch 183/500
#> 7/7 - 0s - 3ms/step - loss: 0.0054 - mean_squared_error: 0.0054
#> Epoch 184/500
#> 7/7 - 0s - 3ms/step - loss: 0.0041 - mean_squared_error: 0.0041
#> Epoch 185/500
#> 7/7 - 0s - 3ms/step - loss: 0.0055 - mean_squared_error: 0.0055
#> Epoch 186/500
#> 7/7 - 0s - 3ms/step - loss: 0.0049 - mean_squared_error: 0.0049
#> Epoch 187/500
#> 7/7 - 0s - 3ms/step - loss: 0.0044 - mean_squared_error: 0.0044
#> Epoch 188/500
#> 7/7 - 0s - 3ms/step - loss: 0.0072 - mean_squared_error: 0.0072
#> Epoch 189/500
#> 7/7 - 0s - 3ms/step - loss: 0.0041 - mean_squared_error: 0.0041
#> Epoch 190/500
#> 7/7 - 0s - 3ms/step - loss: 0.0059 - mean_squared_error: 0.0059
#> Epoch 191/500
#> 7/7 - 0s - 3ms/step - loss: 0.0043 - mean_squared_error: 0.0043
#> Epoch 192/500
#> 7/7 - 0s - 3ms/step - loss: 0.0050 - mean_squared_error: 0.0050
#> Epoch 193/500
#> 7/7 - 0s - 3ms/step - loss: 0.0056 - mean_squared_error: 0.0056
#> Epoch 194/500
#> 7/7 - 0s - 3ms/step - loss: 0.0042 - mean_squared_error: 0.0042
#> Epoch 195/500
#> 7/7 - 0s - 3ms/step - loss: 0.0051 - mean_squared_error: 0.0051
#> Epoch 196/500
#> 7/7 - 0s - 3ms/step - loss: 0.0065 - mean_squared_error: 0.0065
#> Epoch 197/500
#> 7/7 - 0s - 3ms/step - loss: 0.0041 - mean_squared_error: 0.0041
#> Epoch 198/500
#> 7/7 - 0s - 4ms/step - loss: 0.0038 - mean_squared_error: 0.0038
#> Epoch 199/500
#> 7/7 - 0s - 3ms/step - loss: 0.0066 - mean_squared_error: 0.0066
#> Epoch 200/500
#> 7/7 - 0s - 4ms/step - loss: 0.0044 - mean_squared_error: 0.0044
#> Epoch 201/500
#> 7/7 - 0s - 3ms/step - loss: 0.0050 - mean_squared_error: 0.0050
#> Epoch 202/500
#> 7/7 - 0s - 3ms/step - loss: 0.0063 - mean_squared_error: 0.0063
#> Epoch 203/500
#> 7/7 - 0s - 3ms/step - loss: 0.0034 - mean_squared_error: 0.0034
#> Epoch 204/500
#> 7/7 - 0s - 3ms/step - loss: 0.0038 - mean_squared_error: 0.0038
#> Epoch 205/500
#> 7/7 - 0s - 3ms/step - loss: 0.0053 - mean_squared_error: 0.0053
#> Epoch 206/500
#> 7/7 - 0s - 3ms/step - loss: 0.0052 - mean_squared_error: 0.0052
#> Epoch 207/500
#> 7/7 - 0s - 4ms/step - loss: 0.0054 - mean_squared_error: 0.0054
#> Epoch 208/500
#> 7/7 - 0s - 3ms/step - loss: 0.0053 - mean_squared_error: 0.0053
#> Epoch 209/500
#> 7/7 - 0s - 3ms/step - loss: 0.0036 - mean_squared_error: 0.0036
#> Epoch 210/500
#> 7/7 - 0s - 3ms/step - loss: 0.0046 - mean_squared_error: 0.0046
#> Epoch 211/500
#> 7/7 - 0s - 3ms/step - loss: 0.0049 - mean_squared_error: 0.0049
#> Epoch 212/500
#> 7/7 - 0s - 3ms/step - loss: 0.0055 - mean_squared_error: 0.0055
#> Epoch 213/500
#> 7/7 - 0s - 4ms/step - loss: 0.0040 - mean_squared_error: 0.0040
#> Epoch 214/500
#> 7/7 - 0s - 4ms/step - loss: 0.0051 - mean_squared_error: 0.0051
#> Epoch 215/500
#> 7/7 - 0s - 3ms/step - loss: 0.0039 - mean_squared_error: 0.0039
#> Epoch 216/500
#> 7/7 - 0s - 3ms/step - loss: 0.0053 - mean_squared_error: 0.0053
#> Epoch 217/500
#> 7/7 - 0s - 3ms/step - loss: 0.0035 - mean_squared_error: 0.0035
#> Epoch 218/500
#> 7/7 - 0s - 3ms/step - loss: 0.0045 - mean_squared_error: 0.0045
#> Epoch 219/500
#> 7/7 - 0s - 3ms/step - loss: 0.0044 - mean_squared_error: 0.0044
#> Epoch 220/500
#> 7/7 - 0s - 3ms/step - loss: 0.0042 - mean_squared_error: 0.0042
#> Epoch 221/500
#> 7/7 - 0s - 3ms/step - loss: 0.0055 - mean_squared_error: 0.0055
#> Epoch 222/500
#> 7/7 - 0s - 3ms/step - loss: 0.0054 - mean_squared_error: 0.0054
#> Epoch 223/500
#> 7/7 - 0s - 3ms/step - loss: 0.0042 - mean_squared_error: 0.0042
#> Epoch 224/500
#> 7/7 - 0s - 3ms/step - loss: 0.0042 - mean_squared_error: 0.0042
#> Epoch 225/500
#> 7/7 - 0s - 3ms/step - loss: 0.0035 - mean_squared_error: 0.0035
#> Epoch 226/500
#> 7/7 - 0s - 3ms/step - loss: 0.0039 - mean_squared_error: 0.0039
#> Epoch 227/500
#> 7/7 - 0s - 3ms/step - loss: 0.0063 - mean_squared_error: 0.0063
#> Epoch 228/500
#> 7/7 - 0s - 3ms/step - loss: 0.0039 - mean_squared_error: 0.0039
#> Epoch 229/500
#> 7/7 - 0s - 3ms/step - loss: 0.0040 - mean_squared_error: 0.0040
#> Epoch 230/500
#> 7/7 - 0s - 3ms/step - loss: 0.0049 - mean_squared_error: 0.0049
#> Epoch 231/500
#> 7/7 - 0s - 3ms/step - loss: 0.0043 - mean_squared_error: 0.0043
#> Epoch 232/500
#> 7/7 - 0s - 3ms/step - loss: 0.0043 - mean_squared_error: 0.0043
#> Epoch 233/500
#> 7/7 - 0s - 3ms/step - loss: 0.0044 - mean_squared_error: 0.0044
#> Epoch 234/500
#> 7/7 - 0s - 3ms/step - loss: 0.0045 - mean_squared_error: 0.0045
#> Epoch 235/500
#> 7/7 - 0s - 3ms/step - loss: 0.0047 - mean_squared_error: 0.0047
#> Epoch 236/500
#> 7/7 - 0s - 3ms/step - loss: 0.0040 - mean_squared_error: 0.0040
#> Epoch 237/500
#> 7/7 - 0s - 3ms/step - loss: 0.0044 - mean_squared_error: 0.0044
#> Epoch 238/500
#> 7/7 - 0s - 3ms/step - loss: 0.0045 - mean_squared_error: 0.0045
#> Epoch 239/500
#> 7/7 - 0s - 3ms/step - loss: 0.0054 - mean_squared_error: 0.0054
#> Epoch 240/500
#> 7/7 - 0s - 3ms/step - loss: 0.0034 - mean_squared_error: 0.0034
#> Epoch 241/500
#> 7/7 - 0s - 3ms/step - loss: 0.0031 - mean_squared_error: 0.0031
#> Epoch 242/500
#> 7/7 - 0s - 3ms/step - loss: 0.0058 - mean_squared_error: 0.0058
#> Epoch 243/500
#> 7/7 - 0s - 3ms/step - loss: 0.0031 - mean_squared_error: 0.0031
#> Epoch 244/500
#> 7/7 - 0s - 3ms/step - loss: 0.0045 - mean_squared_error: 0.0045
#> Epoch 245/500
#> 7/7 - 0s - 3ms/step - loss: 0.0033 - mean_squared_error: 0.0033
#> Epoch 246/500
#> 7/7 - 0s - 3ms/step - loss: 0.0056 - mean_squared_error: 0.0056
#> Epoch 247/500
#> 7/7 - 0s - 3ms/step - loss: 0.0044 - mean_squared_error: 0.0044
#> Epoch 248/500
#> 7/7 - 0s - 3ms/step - loss: 0.0039 - mean_squared_error: 0.0039
#> Epoch 249/500
#> 7/7 - 0s - 3ms/step - loss: 0.0048 - mean_squared_error: 0.0048
#> Epoch 250/500
#> 7/7 - 0s - 3ms/step - loss: 0.0037 - mean_squared_error: 0.0037
#> Epoch 251/500
#> 7/7 - 0s - 3ms/step - loss: 0.0055 - mean_squared_error: 0.0055
#> Epoch 252/500
#> 7/7 - 0s - 3ms/step - loss: 0.0035 - mean_squared_error: 0.0035
#> Epoch 253/500
#> 7/7 - 0s - 3ms/step - loss: 0.0037 - mean_squared_error: 0.0037
#> Epoch 254/500
#> 7/7 - 0s - 3ms/step - loss: 0.0031 - mean_squared_error: 0.0031
#> Epoch 255/500
#> 7/7 - 0s - 3ms/step - loss: 0.0055 - mean_squared_error: 0.0055
#> Epoch 256/500
#> 7/7 - 0s - 3ms/step - loss: 0.0031 - mean_squared_error: 0.0031
#> Epoch 257/500
#> 7/7 - 0s - 3ms/step - loss: 0.0045 - mean_squared_error: 0.0045
#> Epoch 258/500
#> 7/7 - 0s - 3ms/step - loss: 0.0066 - mean_squared_error: 0.0066
#> Epoch 259/500
#> 7/7 - 0s - 4ms/step - loss: 0.0034 - mean_squared_error: 0.0034
#> Epoch 260/500
#> 7/7 - 0s - 3ms/step - loss: 0.0031 - mean_squared_error: 0.0031
#> Epoch 261/500
#> 7/7 - 0s - 3ms/step - loss: 0.0046 - mean_squared_error: 0.0046
#> Epoch 262/500
#> 7/7 - 0s - 3ms/step - loss: 0.0032 - mean_squared_error: 0.0032
#> Epoch 263/500
#> 7/7 - 0s - 3ms/step - loss: 0.0045 - mean_squared_error: 0.0045
#> Epoch 264/500
#> 7/7 - 0s - 3ms/step - loss: 0.0042 - mean_squared_error: 0.0042
#> Epoch 265/500
#> 7/7 - 0s - 3ms/step - loss: 0.0054 - mean_squared_error: 0.0054
#> Epoch 266/500
#> 7/7 - 0s - 3ms/step - loss: 0.0044 - mean_squared_error: 0.0044
#> Epoch 267/500
#> 7/7 - 0s - 3ms/step - loss: 0.0036 - mean_squared_error: 0.0036
#> Epoch 268/500
#> 7/7 - 0s - 3ms/step - loss: 0.0033 - mean_squared_error: 0.0033
#> Epoch 269/500
#> 7/7 - 0s - 4ms/step - loss: 0.0042 - mean_squared_error: 0.0042
#> Epoch 270/500
#> 7/7 - 0s - 3ms/step - loss: 0.0055 - mean_squared_error: 0.0055
#> Epoch 271/500
#> 7/7 - 0s - 3ms/step - loss: 0.0029 - mean_squared_error: 0.0029
#> Epoch 272/500
#> 7/7 - 0s - 3ms/step - loss: 0.0033 - mean_squared_error: 0.0033
#> Epoch 273/500
#> 7/7 - 0s - 3ms/step - loss: 0.0038 - mean_squared_error: 0.0038
#> Epoch 274/500
#> 7/7 - 0s - 3ms/step - loss: 0.0052 - mean_squared_error: 0.0052
#> Epoch 275/500
#> 7/7 - 0s - 4ms/step - loss: 0.0034 - mean_squared_error: 0.0034
#> Epoch 276/500
#> 7/7 - 0s - 3ms/step - loss: 0.0058 - mean_squared_error: 0.0058
#> Epoch 277/500
#> 7/7 - 0s - 4ms/step - loss: 0.0038 - mean_squared_error: 0.0038
#> Epoch 278/500
#> 7/7 - 0s - 4ms/step - loss: 0.0039 - mean_squared_error: 0.0039
#> Epoch 279/500
#> 7/7 - 0s - 4ms/step - loss: 0.0052 - mean_squared_error: 0.0052
#> Epoch 280/500
#> 7/7 - 0s - 3ms/step - loss: 0.0026 - mean_squared_error: 0.0026
#> Epoch 281/500
#> 7/7 - 0s - 3ms/step - loss: 0.0048 - mean_squared_error: 0.0048
#> Epoch 282/500
#> 7/7 - 0s - 3ms/step - loss: 0.0041 - mean_squared_error: 0.0041
#> Epoch 283/500
#> 7/7 - 0s - 3ms/step - loss: 0.0046 - mean_squared_error: 0.0046
#> Epoch 284/500
#> 7/7 - 0s - 3ms/step - loss: 0.0063 - mean_squared_error: 0.0063
#> Epoch 285/500
#> 7/7 - 0s - 3ms/step - loss: 0.0030 - mean_squared_error: 0.0030
#> Epoch 286/500
#> 7/7 - 0s - 3ms/step - loss: 0.0032 - mean_squared_error: 0.0032
#> Epoch 287/500
#> 7/7 - 0s - 3ms/step - loss: 0.0040 - mean_squared_error: 0.0040
#> Epoch 288/500
#> 7/7 - 0s - 4ms/step - loss: 0.0046 - mean_squared_error: 0.0046
#> Epoch 289/500
#> 7/7 - 0s - 3ms/step - loss: 0.0033 - mean_squared_error: 0.0033
#> Epoch 290/500
#> 7/7 - 0s - 3ms/step - loss: 0.0046 - mean_squared_error: 0.0046
#> Epoch 291/500
#> 7/7 - 0s - 3ms/step - loss: 0.0041 - mean_squared_error: 0.0041
#> Epoch 292/500
#> 7/7 - 0s - 3ms/step - loss: 0.0053 - mean_squared_error: 0.0053
#> Epoch 293/500
#> 7/7 - 0s - 4ms/step - loss: 0.0032 - mean_squared_error: 0.0032
#> Epoch 294/500
#> 7/7 - 0s - 3ms/step - loss: 0.0030 - mean_squared_error: 0.0030
#> Epoch 295/500
#> 7/7 - 0s - 3ms/step - loss: 0.0032 - mean_squared_error: 0.0032
#> Epoch 296/500
#> 7/7 - 0s - 3ms/step - loss: 0.0032 - mean_squared_error: 0.0032
#> Epoch 297/500
#> 7/7 - 0s - 3ms/step - loss: 0.0041 - mean_squared_error: 0.0041
#> Epoch 298/500
#> 7/7 - 0s - 3ms/step - loss: 0.0035 - mean_squared_error: 0.0035
#> Epoch 299/500
#> 7/7 - 0s - 4ms/step - loss: 0.0032 - mean_squared_error: 0.0032
#> Epoch 300/500
#> 7/7 - 0s - 3ms/step - loss: 0.0040 - mean_squared_error: 0.0040
#> Epoch 301/500
#> 7/7 - 0s - 3ms/step - loss: 0.0044 - mean_squared_error: 0.0044
#> Epoch 302/500
#> 7/7 - 0s - 3ms/step - loss: 0.0044 - mean_squared_error: 0.0044
#> Epoch 303/500
#> 7/7 - 0s - 3ms/step - loss: 0.0027 - mean_squared_error: 0.0027
#> Epoch 304/500
#> 7/7 - 0s - 3ms/step - loss: 0.0042 - mean_squared_error: 0.0042
#> Epoch 305/500
#> 7/7 - 0s - 3ms/step - loss: 0.0049 - mean_squared_error: 0.0049
#> Epoch 306/500
#> 7/7 - 0s - 3ms/step - loss: 0.0031 - mean_squared_error: 0.0031
#> Epoch 307/500
#> 7/7 - 0s - 3ms/step - loss: 0.0039 - mean_squared_error: 0.0039
#> Epoch 308/500
#> 7/7 - 0s - 3ms/step - loss: 0.0030 - mean_squared_error: 0.0030
#> Epoch 309/500
#> 7/7 - 0s - 3ms/step - loss: 0.0045 - mean_squared_error: 0.0045
#> Epoch 310/500
#> 7/7 - 0s - 4ms/step - loss: 0.0042 - mean_squared_error: 0.0042
#> Epoch 311/500
#> 7/7 - 0s - 4ms/step - loss: 0.0043 - mean_squared_error: 0.0043
#> Epoch 312/500
#> 7/7 - 0s - 4ms/step - loss: 0.0024 - mean_squared_error: 0.0024
#> Epoch 313/500
#> 7/7 - 0s - 3ms/step - loss: 0.0047 - mean_squared_error: 0.0047
#> Epoch 314/500
#> 7/7 - 0s - 4ms/step - loss: 0.0040 - mean_squared_error: 0.0040
#> Epoch 315/500
#> 7/7 - 0s - 4ms/step - loss: 0.0031 - mean_squared_error: 0.0031
#> Epoch 316/500
#> 7/7 - 0s - 4ms/step - loss: 0.0030 - mean_squared_error: 0.0030
#> Epoch 317/500
#> 7/7 - 0s - 4ms/step - loss: 0.0042 - mean_squared_error: 0.0042
#> Epoch 318/500
#> 7/7 - 0s - 4ms/step - loss: 0.0040 - mean_squared_error: 0.0040
#> Epoch 319/500
#> 7/7 - 0s - 4ms/step - loss: 0.0030 - mean_squared_error: 0.0030
#> Epoch 320/500
#> 7/7 - 0s - 3ms/step - loss: 0.0025 - mean_squared_error: 0.0025
#> Epoch 321/500
#> 7/7 - 0s - 3ms/step - loss: 0.0065 - mean_squared_error: 0.0065
#> Epoch 322/500
#> 7/7 - 0s - 3ms/step - loss: 0.0028 - mean_squared_error: 0.0028
#> Epoch 323/500
#> 7/7 - 0s - 3ms/step - loss: 0.0030 - mean_squared_error: 0.0030
#> Epoch 324/500
#> 7/7 - 0s - 3ms/step - loss: 0.0040 - mean_squared_error: 0.0040
#> Epoch 325/500
#> 7/7 - 0s - 3ms/step - loss: 0.0032 - mean_squared_error: 0.0032
#> Epoch 326/500
#> 7/7 - 0s - 4ms/step - loss: 0.0040 - mean_squared_error: 0.0040
#> Epoch 327/500
#> 7/7 - 0s - 3ms/step - loss: 0.0035 - mean_squared_error: 0.0035
#> Epoch 328/500
#> 7/7 - 0s - 4ms/step - loss: 0.0041 - mean_squared_error: 0.0041
#> Epoch 329/500
#> 7/7 - 0s - 3ms/step - loss: 0.0033 - mean_squared_error: 0.0033
#> Epoch 330/500
#> 7/7 - 0s - 3ms/step - loss: 0.0049 - mean_squared_error: 0.0049
#> Epoch 331/500
#> 7/7 - 0s - 3ms/step - loss: 0.0030 - mean_squared_error: 0.0030
#> Epoch 332/500
#> 7/7 - 0s - 3ms/step - loss: 0.0034 - mean_squared_error: 0.0034
#> Epoch 333/500
#> 7/7 - 0s - 3ms/step - loss: 0.0032 - mean_squared_error: 0.0032
#> Epoch 334/500
#> 7/7 - 0s - 3ms/step - loss: 0.0040 - mean_squared_error: 0.0040
#> Epoch 335/500
#> 7/7 - 0s - 3ms/step - loss: 0.0041 - mean_squared_error: 0.0041
#> Epoch 336/500
#> 7/7 - 0s - 4ms/step - loss: 0.0042 - mean_squared_error: 0.0042
#> Epoch 337/500
#> 7/7 - 0s - 3ms/step - loss: 0.0029 - mean_squared_error: 0.0029
#> Epoch 338/500
#> 7/7 - 0s - 4ms/step - loss: 0.0046 - mean_squared_error: 0.0046
#> Epoch 339/500
#> 7/7 - 0s - 3ms/step - loss: 0.0028 - mean_squared_error: 0.0028
#> Epoch 340/500
#> 7/7 - 0s - 3ms/step - loss: 0.0045 - mean_squared_error: 0.0045
#> Epoch 341/500
#> 7/7 - 0s - 3ms/step - loss: 0.0029 - mean_squared_error: 0.0029
#> Epoch 342/500
#> 7/7 - 0s - 3ms/step - loss: 0.0043 - mean_squared_error: 0.0043
#> Epoch 343/500
#> 7/7 - 0s - 3ms/step - loss: 0.0034 - mean_squared_error: 0.0034
#> Epoch 344/500
#> 7/7 - 0s - 3ms/step - loss: 0.0039 - mean_squared_error: 0.0039
#> Epoch 345/500
#> 7/7 - 0s - 3ms/step - loss: 0.0033 - mean_squared_error: 0.0033
#> Epoch 346/500
#> 7/7 - 0s - 4ms/step - loss: 0.0040 - mean_squared_error: 0.0040
#> Epoch 347/500
#> 7/7 - 0s - 3ms/step - loss: 0.0024 - mean_squared_error: 0.0024
#> Epoch 348/500
#> 7/7 - 0s - 3ms/step - loss: 0.0030 - mean_squared_error: 0.0030
#> Epoch 349/500
#> 7/7 - 0s - 3ms/step - loss: 0.0052 - mean_squared_error: 0.0052
#> Epoch 350/500
#> 7/7 - 0s - 3ms/step - loss: 0.0030 - mean_squared_error: 0.0030
#> Epoch 351/500
#> 7/7 - 0s - 3ms/step - loss: 0.0048 - mean_squared_error: 0.0048
#> Epoch 352/500
#> 7/7 - 0s - 3ms/step - loss: 0.0025 - mean_squared_error: 0.0025
#> Epoch 353/500
#> 7/7 - 0s - 3ms/step - loss: 0.0039 - mean_squared_error: 0.0039
#> Epoch 354/500
#> 7/7 - 0s - 3ms/step - loss: 0.0035 - mean_squared_error: 0.0035
#> Epoch 355/500
#> 7/7 - 0s - 3ms/step - loss: 0.0039 - mean_squared_error: 0.0039
#> Epoch 356/500
#> 7/7 - 0s - 3ms/step - loss: 0.0031 - mean_squared_error: 0.0031
#> Epoch 357/500
#> 7/7 - 0s - 3ms/step - loss: 0.0041 - mean_squared_error: 0.0041
#> Epoch 358/500
#> 7/7 - 0s - 4ms/step - loss: 0.0023 - mean_squared_error: 0.0023
#> Epoch 359/500
#> 7/7 - 0s - 3ms/step - loss: 0.0051 - mean_squared_error: 0.0051
#> Epoch 360/500
#> 7/7 - 0s - 3ms/step - loss: 0.0024 - mean_squared_error: 0.0024
#> Epoch 361/500
#> 7/7 - 0s - 3ms/step - loss: 0.0035 - mean_squared_error: 0.0035
#> Epoch 362/500
#> 7/7 - 0s - 3ms/step - loss: 0.0028 - mean_squared_error: 0.0028
#> Epoch 363/500
#> 7/7 - 0s - 3ms/step - loss: 0.0048 - mean_squared_error: 0.0048
#> Epoch 364/500
#> 7/7 - 0s - 3ms/step - loss: 0.0032 - mean_squared_error: 0.0032
#> Epoch 365/500
#> 7/7 - 0s - 3ms/step - loss: 0.0037 - mean_squared_error: 0.0037
#> Epoch 366/500
#> 7/7 - 0s - 3ms/step - loss: 0.0029 - mean_squared_error: 0.0029
#> Epoch 367/500
#> 7/7 - 0s - 3ms/step - loss: 0.0060 - mean_squared_error: 0.0060
#> Epoch 368/500
#> 7/7 - 0s - 4ms/step - loss: 0.0025 - mean_squared_error: 0.0025
#> Epoch 369/500
#> 7/7 - 0s - 4ms/step - loss: 0.0035 - mean_squared_error: 0.0035
#> Epoch 370/500
#> 7/7 - 0s - 3ms/step - loss: 0.0034 - mean_squared_error: 0.0034
#> Epoch 371/500
#> 7/7 - 0s - 3ms/step - loss: 0.0033 - mean_squared_error: 0.0033
#> Epoch 372/500
#> 7/7 - 0s - 3ms/step - loss: 0.0041 - mean_squared_error: 0.0041
#> Epoch 373/500
#> 7/7 - 0s - 4ms/step - loss: 0.0031 - mean_squared_error: 0.0031
#> Epoch 374/500
#> 7/7 - 0s - 3ms/step - loss: 0.0030 - mean_squared_error: 0.0030
#> Epoch 375/500
#> 7/7 - 0s - 3ms/step - loss: 0.0040 - mean_squared_error: 0.0040
#> Epoch 376/500
#> 7/7 - 0s - 3ms/step - loss: 0.0039 - mean_squared_error: 0.0039
#> Epoch 377/500
#> 7/7 - 0s - 3ms/step - loss: 0.0019 - mean_squared_error: 0.0019
#> Epoch 378/500
#> 7/7 - 0s - 3ms/step - loss: 0.0045 - mean_squared_error: 0.0045
#> Epoch 379/500
#> 7/7 - 0s - 3ms/step - loss: 0.0022 - mean_squared_error: 0.0022
#> Epoch 380/500
#> 7/7 - 0s - 3ms/step - loss: 0.0033 - mean_squared_error: 0.0033
#> Epoch 381/500
#> 7/7 - 0s - 3ms/step - loss: 0.0030 - mean_squared_error: 0.0030
#> Epoch 382/500
#> 7/7 - 0s - 4ms/step - loss: 0.0051 - mean_squared_error: 0.0051
#> Epoch 383/500
#> 7/7 - 0s - 4ms/step - loss: 0.0022 - mean_squared_error: 0.0022
#> Epoch 384/500
#> 7/7 - 0s - 4ms/step - loss: 0.0052 - mean_squared_error: 0.0052
#> Epoch 385/500
#> 7/7 - 0s - 4ms/step - loss: 0.0034 - mean_squared_error: 0.0034
#> Epoch 386/500
#> 7/7 - 0s - 3ms/step - loss: 0.0026 - mean_squared_error: 0.0026
#> Epoch 387/500
#> 7/7 - 0s - 4ms/step - loss: 0.0026 - mean_squared_error: 0.0026
#> Epoch 388/500
#> 7/7 - 0s - 3ms/step - loss: 0.0022 - mean_squared_error: 0.0022
#> Epoch 389/500
#> 7/7 - 0s - 3ms/step - loss: 0.0048 - mean_squared_error: 0.0048
#> Epoch 390/500
#> 7/7 - 0s - 3ms/step - loss: 0.0034 - mean_squared_error: 0.0034
#> Epoch 391/500
#> 7/7 - 0s - 3ms/step - loss: 0.0028 - mean_squared_error: 0.0028
#> Epoch 392/500
#> 7/7 - 0s - 3ms/step - loss: 0.0024 - mean_squared_error: 0.0024
#> Epoch 393/500
#> 7/7 - 0s - 3ms/step - loss: 0.0060 - mean_squared_error: 0.0060
#> Epoch 394/500
#> 7/7 - 0s - 3ms/step - loss: 0.0023 - mean_squared_error: 0.0023
#> Epoch 395/500
#> 7/7 - 0s - 3ms/step - loss: 0.0027 - mean_squared_error: 0.0027
#> Epoch 396/500
#> 7/7 - 0s - 3ms/step - loss: 0.0049 - mean_squared_error: 0.0049
#> Epoch 397/500
#> 7/7 - 0s - 3ms/step - loss: 0.0026 - mean_squared_error: 0.0026
#> Epoch 398/500
#> 7/7 - 0s - 4ms/step - loss: 0.0030 - mean_squared_error: 0.0030
#> Epoch 399/500
#> 7/7 - 0s - 4ms/step - loss: 0.0025 - mean_squared_error: 0.0025
#> Epoch 400/500
#> 7/7 - 0s - 4ms/step - loss: 0.0036 - mean_squared_error: 0.0036
#> Epoch 401/500
#> 7/7 - 0s - 4ms/step - loss: 0.0038 - mean_squared_error: 0.0038
#> Epoch 402/500
#> 7/7 - 0s - 4ms/step - loss: 0.0025 - mean_squared_error: 0.0025
#> Epoch 403/500
#> 7/7 - 0s - 4ms/step - loss: 0.0033 - mean_squared_error: 0.0033
#> Epoch 404/500
#> 7/7 - 0s - 3ms/step - loss: 0.0036 - mean_squared_error: 0.0036
#> Epoch 405/500
#> 7/7 - 0s - 4ms/step - loss: 0.0043 - mean_squared_error: 0.0043
#> Epoch 406/500
#> 7/7 - 0s - 3ms/step - loss: 0.0025 - mean_squared_error: 0.0025
#> Epoch 407/500
#> 7/7 - 0s - 3ms/step - loss: 0.0043 - mean_squared_error: 0.0043
#> Epoch 408/500
#> 7/7 - 0s - 3ms/step - loss: 0.0027 - mean_squared_error: 0.0027
#> Epoch 409/500
#> 7/7 - 0s - 4ms/step - loss: 0.0040 - mean_squared_error: 0.0040
#> Epoch 410/500
#> 7/7 - 0s - 3ms/step - loss: 0.0021 - mean_squared_error: 0.0021
#> Epoch 411/500
#> 7/7 - 0s - 3ms/step - loss: 0.0047 - mean_squared_error: 0.0047
#> Epoch 412/500
#> 7/7 - 0s - 3ms/step - loss: 0.0026 - mean_squared_error: 0.0026
#> Epoch 413/500
#> 7/7 - 0s - 4ms/step - loss: 0.0040 - mean_squared_error: 0.0040
#> Epoch 414/500
#> 7/7 - 0s - 3ms/step - loss: 0.0022 - mean_squared_error: 0.0022
#> Epoch 415/500
#> 7/7 - 0s - 3ms/step - loss: 0.0026 - mean_squared_error: 0.0026
#> Epoch 416/500
#> 7/7 - 0s - 3ms/step - loss: 0.0030 - mean_squared_error: 0.0030
#> Epoch 417/500
#> 7/7 - 0s - 4ms/step - loss: 0.0044 - mean_squared_error: 0.0044
#> Epoch 418/500
#> 7/7 - 0s - 3ms/step - loss: 0.0023 - mean_squared_error: 0.0023
#> Epoch 419/500
#> 7/7 - 0s - 3ms/step - loss: 0.0047 - mean_squared_error: 0.0047
#> Epoch 420/500
#> 7/7 - 0s - 4ms/step - loss: 0.0021 - mean_squared_error: 0.0021
#> Epoch 421/500
#> 7/7 - 0s - 3ms/step - loss: 0.0019 - mean_squared_error: 0.0019
#> Epoch 422/500
#> 7/7 - 0s - 4ms/step - loss: 0.0050 - mean_squared_error: 0.0050
#> Epoch 423/500
#> 7/7 - 0s - 3ms/step - loss: 0.0026 - mean_squared_error: 0.0026
#> Epoch 424/500
#> 7/7 - 0s - 4ms/step - loss: 0.0032 - mean_squared_error: 0.0032
#> Epoch 425/500
#> 7/7 - 0s - 3ms/step - loss: 0.0029 - mean_squared_error: 0.0029
#> Epoch 426/500
#> 7/7 - 0s - 3ms/step - loss: 0.0022 - mean_squared_error: 0.0022
#> Epoch 427/500
#> 7/7 - 0s - 3ms/step - loss: 0.0057 - mean_squared_error: 0.0057
#> Epoch 428/500
#> 7/7 - 0s - 3ms/step - loss: 0.0019 - mean_squared_error: 0.0019
#> Epoch 429/500
#> 7/7 - 0s - 3ms/step - loss: 0.0035 - mean_squared_error: 0.0035
#> Epoch 430/500
#> 7/7 - 0s - 3ms/step - loss: 0.0030 - mean_squared_error: 0.0030
#> Epoch 431/500
#> 7/7 - 0s - 4ms/step - loss: 0.0022 - mean_squared_error: 0.0022
#> Epoch 432/500
#> 7/7 - 0s - 3ms/step - loss: 0.0046 - mean_squared_error: 0.0046
#> Epoch 433/500
#> 7/7 - 0s - 4ms/step - loss: 0.0023 - mean_squared_error: 0.0023
#> Epoch 434/500
#> 7/7 - 0s - 4ms/step - loss: 0.0036 - mean_squared_error: 0.0036
#> Epoch 435/500
#> 7/7 - 0s - 3ms/step - loss: 0.0019 - mean_squared_error: 0.0019
#> Epoch 436/500
#> 7/7 - 0s - 3ms/step - loss: 0.0027 - mean_squared_error: 0.0027
#> Epoch 437/500
#> 7/7 - 0s - 3ms/step - loss: 0.0033 - mean_squared_error: 0.0033
#> Epoch 438/500
#> 7/7 - 0s - 3ms/step - loss: 0.0042 - mean_squared_error: 0.0042
#> Epoch 439/500
#> 7/7 - 0s - 3ms/step - loss: 0.0029 - mean_squared_error: 0.0029
#> Epoch 440/500
#> 7/7 - 0s - 4ms/step - loss: 0.0037 - mean_squared_error: 0.0037
#> Epoch 441/500
#> 7/7 - 0s - 3ms/step - loss: 0.0032 - mean_squared_error: 0.0032
#> Epoch 442/500
#> 7/7 - 0s - 3ms/step - loss: 0.0040 - mean_squared_error: 0.0040
#> Epoch 443/500
#> 7/7 - 0s - 4ms/step - loss: 0.0029 - mean_squared_error: 0.0029
#> Epoch 444/500
#> 7/7 - 0s - 3ms/step - loss: 0.0029 - mean_squared_error: 0.0029
#> Epoch 445/500
#> 7/7 - 0s - 3ms/step - loss: 0.0043 - mean_squared_error: 0.0043
#> Epoch 446/500
#> 7/7 - 0s - 4ms/step - loss: 0.0024 - mean_squared_error: 0.0024
#> Epoch 447/500
#> 7/7 - 0s - 4ms/step - loss: 0.0028 - mean_squared_error: 0.0028
#> Epoch 448/500
#> 7/7 - 0s - 4ms/step - loss: 0.0041 - mean_squared_error: 0.0041
#> Epoch 449/500
#> 7/7 - 0s - 3ms/step - loss: 0.0025 - mean_squared_error: 0.0025
#> Epoch 450/500
#> 7/7 - 0s - 3ms/step - loss: 0.0031 - mean_squared_error: 0.0031
#> Epoch 451/500
#> 7/7 - 0s - 3ms/step - loss: 0.0026 - mean_squared_error: 0.0026
#> Epoch 452/500
#> 7/7 - 0s - 3ms/step - loss: 0.0029 - mean_squared_error: 0.0029
#> Epoch 453/500
#> 7/7 - 0s - 3ms/step - loss: 0.0040 - mean_squared_error: 0.0040
#> Epoch 454/500
#> 7/7 - 0s - 4ms/step - loss: 0.0025 - mean_squared_error: 0.0025
#> Epoch 455/500
#> 7/7 - 0s - 4ms/step - loss: 0.0027 - mean_squared_error: 0.0027
#> Epoch 456/500
#> 7/7 - 0s - 4ms/step - loss: 0.0041 - mean_squared_error: 0.0041
#> Epoch 457/500
#> 7/7 - 0s - 3ms/step - loss: 0.0034 - mean_squared_error: 0.0034
#> Epoch 458/500
#> 7/7 - 0s - 4ms/step - loss: 0.0025 - mean_squared_error: 0.0025
#> Epoch 459/500
#> 7/7 - 0s - 3ms/step - loss: 0.0019 - mean_squared_error: 0.0019
#> Epoch 460/500
#> 7/7 - 0s - 4ms/step - loss: 0.0023 - mean_squared_error: 0.0023
#> Epoch 461/500
#> 7/7 - 0s - 3ms/step - loss: 0.0037 - mean_squared_error: 0.0037
#> Epoch 462/500
#> 7/7 - 0s - 4ms/step - loss: 0.0033 - mean_squared_error: 0.0033
#> Epoch 463/500
#> 7/7 - 0s - 4ms/step - loss: 0.0030 - mean_squared_error: 0.0030
#> Epoch 464/500
#> 7/7 - 0s - 4ms/step - loss: 0.0024 - mean_squared_error: 0.0024
#> Epoch 465/500
#> 7/7 - 0s - 3ms/step - loss: 0.0034 - mean_squared_error: 0.0034
#> Epoch 466/500
#> 7/7 - 0s - 3ms/step - loss: 0.0029 - mean_squared_error: 0.0029
#> Epoch 467/500
#> 7/7 - 0s - 4ms/step - loss: 0.0035 - mean_squared_error: 0.0035
#> Epoch 468/500
#> 7/7 - 0s - 3ms/step - loss: 0.0023 - mean_squared_error: 0.0023
#> Epoch 469/500
#> 7/7 - 0s - 4ms/step - loss: 0.0037 - mean_squared_error: 0.0037
#> Epoch 470/500
#> 7/7 - 0s - 4ms/step - loss: 0.0027 - mean_squared_error: 0.0027
#> Epoch 471/500
#> 7/7 - 0s - 4ms/step - loss: 0.0051 - mean_squared_error: 0.0051
#> Epoch 472/500
#> 7/7 - 0s - 4ms/step - loss: 0.0029 - mean_squared_error: 0.0029
#> Epoch 473/500
#> 7/7 - 0s - 3ms/step - loss: 0.0018 - mean_squared_error: 0.0018
#> Epoch 474/500
#> 7/7 - 0s - 3ms/step - loss: 0.0026 - mean_squared_error: 0.0026
#> Epoch 475/500
#> 7/7 - 0s - 4ms/step - loss: 0.0042 - mean_squared_error: 0.0042
#> Epoch 476/500
#> 7/7 - 0s - 3ms/step - loss: 0.0029 - mean_squared_error: 0.0029
#> Epoch 477/500
#> 7/7 - 0s - 4ms/step - loss: 0.0021 - mean_squared_error: 0.0021
#> Epoch 478/500
#> 7/7 - 0s - 4ms/step - loss: 0.0040 - mean_squared_error: 0.0040
#> Epoch 479/500
#> 7/7 - 0s - 4ms/step - loss: 0.0026 - mean_squared_error: 0.0026
#> Epoch 480/500
#> 7/7 - 0s - 4ms/step - loss: 0.0019 - mean_squared_error: 0.0019
#> Epoch 481/500
#> 7/7 - 0s - 3ms/step - loss: 0.0043 - mean_squared_error: 0.0043
#> Epoch 482/500
#> 7/7 - 0s - 4ms/step - loss: 0.0023 - mean_squared_error: 0.0023
#> Epoch 483/500
#> 7/7 - 0s - 3ms/step - loss: 0.0041 - mean_squared_error: 0.0041
#> Epoch 484/500
#> 7/7 - 0s - 3ms/step - loss: 0.0020 - mean_squared_error: 0.0020
#> Epoch 485/500
#> 7/7 - 0s - 4ms/step - loss: 0.0032 - mean_squared_error: 0.0032
#> Epoch 486/500
#> 7/7 - 0s - 3ms/step - loss: 0.0033 - mean_squared_error: 0.0033
#> Epoch 487/500
#> 7/7 - 0s - 3ms/step - loss: 0.0026 - mean_squared_error: 0.0026
#> Epoch 488/500
#> 7/7 - 0s - 3ms/step - loss: 0.0019 - mean_squared_error: 0.0019
#> Epoch 489/500
#> 7/7 - 0s - 3ms/step - loss: 0.0037 - mean_squared_error: 0.0037
#> Epoch 490/500
#> 7/7 - 0s - 4ms/step - loss: 0.0035 - mean_squared_error: 0.0035
#> Epoch 491/500
#> 7/7 - 0s - 3ms/step - loss: 0.0029 - mean_squared_error: 0.0029
#> Epoch 492/500
#> 7/7 - 0s - 3ms/step - loss: 0.0021 - mean_squared_error: 0.0021
#> Epoch 493/500
#> 7/7 - 0s - 4ms/step - loss: 0.0033 - mean_squared_error: 0.0033
#> Epoch 494/500
#> 7/7 - 0s - 3ms/step - loss: 0.0036 - mean_squared_error: 0.0036
#> Epoch 495/500
#> 7/7 - 0s - 3ms/step - loss: 0.0022 - mean_squared_error: 0.0022
#> Epoch 496/500
#> 7/7 - 0s - 4ms/step - loss: 0.0019 - mean_squared_error: 0.0019
#> Epoch 497/500
#> 7/7 - 0s - 4ms/step - loss: 0.0038 - mean_squared_error: 0.0038
#> Epoch 498/500
#> 7/7 - 0s - 3ms/step - loss: 0.0025 - mean_squared_error: 0.0025
#> Epoch 499/500
#> 7/7 - 0s - 3ms/step - loss: 0.0023 - mean_squared_error: 0.0023
#> Epoch 500/500
#> 7/7 - 0s - 4ms/step - loss: 0.0045 - mean_squared_error: 0.0045
tensorboard("folder/on/your/computer")
#> Started TensorBoard at http://127.0.0.1:3584Behold! A very fancy set of diagnostics appears on your computer about your run. Remember that you might need to clean out "folder/on/your/computer" at the end of a load of tests, and you need to replace that text with a folder on your computer. If the folder doesn’t exist, it will be created. I should add that the line above sometimes causes an error on computers configured to be skeptical of programs setting up web servers and then opening them in browsers; if so, open up a browser and trying going to http://127.0.0.1:4298 in it and, with a bit of luck, your dashboard should appear there.
10.2 Convolutional neural networks
This is the fourth of four sessions in your options series, and is itself the last of two related sessions that form an introduction to modern deep learning using artificial neural networks. Today we are going to launch into convolutional neural networks. These are an extremely popular class of neural networks whose topology was designed with the mammalian visual cortex in mind. They’re extremely powerful tools, and have the additional advantage that they can carry out pre-processing of images on our behalf. This means that, with sufficiently well-designed networks, you can simply whack your images into the model and the model will down-scale and clean them up for you. I’m sure you can imagine how useful having the computer do the work for you can be!
10.2.1 Basic image classification
Let’s start off by working with a built-in dataset within Keras and classifying the data using the techniques we’ve already learned. First of all, let’s download and clean up the demonstration data. It’s roughly 100 Mb in size, so this may take a moment.
# Setup
library(keras3)
# Load data
raw.data <- dataset_fashion_mnist()
resp <- raw.data$train$y
exp <- raw.data$train$x
# Do a bit of renaming and scaling, and plot for sense
lookup <- c("T-shirt/top", "Trouser", "Pullover", "Dress",
"Coat", "Sandal", "Shirt", "Sneaker", "Bag", "Ankle boot")
exp <- exp / 255
par(mfrow=c(2,2)) # I've made this smaller for printing - expand to 5,5 to see them all
for(i in seq_len(2*2)) # Same here
image(t(exp[i,28:1,]), main=lookup[resp[i]+1], col=grey.colors(255))
# ... the images are upside-down, hence the 28:1 statemnt,
# ... and R needs matrices rotated to plot, hence the use of *t*ranspose
# ... and keras needs labels (resp) to start at 0, hence +1Hopefully you should now see 25 images of clothing from a standard machine learning dataset on articles of clothing. If it seems odd that this is a standard test dataset, take a minute to consider that almost everyone you have ever met owns at least one pair of clothes, so we have a lot of data and a lot of motivation to make good models of what people wear! Let’s now fit a standard artificial neural network to these data, making use of two new kinds of layer (see if you can spot them before I describe them).
# Define model
model <- keras_model_sequential(input_shape = c(28, 28, 1)) %>%
layer_flatten() %>%
layer_dense(units = 128, activation = 'relu') %>%
layer_dropout(rate = 0.5) %>%
layer_dense(units = 10, activation = 'softmax')
# Compile model
model %>% compile(
optimizer = 'adam',
loss = 'sparse_categorical_crossentropy',
metrics = c('accuracy')
)
# Fit model and independently validate
model %>% fit(exp, resp, epochs = 5)
#> Epoch 1/5
#> 1875/1875 - 3s - 2ms/step - accuracy: 0.7809 - loss: 0.6191
#> Epoch 2/5
#> 1875/1875 - 3s - 2ms/step - accuracy: 0.8292 - loss: 0.4724
#> Epoch 3/5
#> 1875/1875 - 3s - 2ms/step - accuracy: 0.8409 - loss: 0.4381
#> Epoch 4/5
#> 1875/1875 - 3s - 2ms/step - accuracy: 0.8463 - loss: 0.4205
#> Epoch 5/5
#> 1875/1875 - 3s - 2ms/step - accuracy: 0.8527 - loss: 0.4030
test.resp <- raw.data$test$y
test.exp <- raw.data$test$x
test.exp <- test.exp/255
model %>% evaluate(test.exp, test.resp)
#> 313/313 - 0s - 1ms/step - accuracy: 0.8648 - loss: 0.3766
#> $accuracy
#> [1] 0.8648
#>
#> $loss
#> [1] 0.3765551
predictions <- model %>% predict(test.exp)
#> 313/313 - 0s - 1ms/step
table(apply(predictions, 1, which.max)-1, test.resp)
#> test.resp
#> 0 1 2 3 4 5 6 7 8 9
#> 0 769 3 10 14 0 0 100 0 0 0
#> 1 2 958 0 11 0 0 1 0 0 0
#> 2 34 3 780 22 108 0 123 0 8 0
#> 3 31 24 8 855 17 1 25 0 4 0
#> 4 6 5 139 57 834 0 108 0 6 0
#> 5 0 0 0 0 0 951 0 22 1 8
#> 6 147 5 59 37 39 0 629 0 3 1
#> 7 0 0 0 0 0 31 0 951 6 42
#> 8 11 2 4 4 2 2 14 0 972 0
#> 9 0 0 0 0 0 15 0 27 0 949The two new kinds of layer are the flattening layer (layer_flatten) and the dropout layer (layer_dropout). The flattening layer is more of a processing layer than anything: it takes the two-dimensional input data (the images) that we have supplied and turns them into one-dimensional vectors. Notice that it specifies its input shape as a 28-by-28 object (input_shape=c(28,28))—we have flattened two dimensions of data into one. You have only been working with such data in this class so far. The dropout layer is a weird idea that can often work well in neural networks. The idea is to randomly drop nodes from the network during each training epoch with a set probability for each node. Thus the network topology is different during each training session. Then, when it comes to using the network on real (or validation) data, all the nodes are brought back into the network but each node’s ‘downstream’ (subtending) connection weights are multiplied by the probability of that node being in the network during training. This tends to sort of ‘damp down’ the network, and reduces the likelihood of overfitting. Conceptually, this approach works for reasons very similar to the random regression trees we covered a few sessions ago: we’re playing around with the variance within the network. We are using a new optimizer because we are dealing with a different kind of data (it is beyond the scope of this course to explain this optimizer, as I described last time), and the new loss function we covered last time.
10.2.2 Convolutional neural networks
Dealing with images as flat data is fine, but conceptually it feels odd to throw away what might important data as to what pixels are close to one-another. A modern class of networks—convolutional neural networks—are named after the mathematical operation they use (a convolution) to emulate the operation of the visual cortex in mammals. To explain these networks, I’m going to (very briefly!) outline the basics of vision in humans, then describe what a mathematical convolution is, and finally show you the network structure of these models.
We are still learning about the human visual cortex, but it is fair to describe it as a series of layers that take input from the eyes and apply a series of detectors to process that input into a form that is more useful to us. For example, we have layers whose job is to detect edges, and others whose job is to detect movement in particular directions. The ‘Waterfall Illusion’, which was first described by Aristotle, is a direct consequence of these detectors: if you stare at a waterfall for a long time, it tires (saturates) the downward detectors in your visual cortex. When you then look away from the waterfall, because those downward detectors have saturated they fire less intensely than they would normally, and are weaker than the upward detectors they normally counter-balance when we look at something at rest8. Thus when you look away from the waterfall, suddenly everything seems to be rising9. Most detectors work to generate an image from whatever the eye is seeing at that moment in time, and as such take input from several adjacent regions of the retina and combine that information in order to detect a particular kind of feature. For example, edge detectors take input from several regions (let’s call them cells) and compare them to see if some are lighter than others. If they are, then it sends a signal that an edge exists at that point in the visual field.
In biology, neurons are cheap10, but in computing, nodes are not. So when computer scientists wanted to use similar techniques to analyze images in artificial neural networks, they hit a snag: in our brains there are almost as many detectors as there are inputting rods and cones in our eyes. That makes sense: each ‘pixel’ in our vision must be compared, and so each ‘pixel’ needs a receptor. Each layer of our visual cortex is massive because of this. Having that many inputs would become unwieldy in machine learning: we would need thousands of nodes just to model a single reasonably-sized image. The solution is to cheat, and use convolution to set up only a single receptor (a convolution layer) and then apply that convolution layer across all the possible subsets of the image to get the same effect done. Thus, using convolution, it’s possible to have a single detector that examines 5x5 pixel sections across an entire 1000x1000 image, using only 5x5 nodes.
Which means we need a quick note on what, exactly, a convolution is… Technically, if you have two functions you are convolving, called \(f\) and \(g\), their convolution is the integral of the product of the two functions once one has been reversed and shifted by a set amount. In scary equations, therefore, it’s:
\[ (f * g)(x) = \int{f(x)g(-x)dx} \]
Where \(*\) means “convolve”. This ability to essentially multiply things together is what allows the convolution to be performed across so many different inputs in succession: we’re convolving everything together. The confusing part is what \(x\) represents, and why \(f(x)\) is working on the reversed set of data from \(g(-x)\). \(x\) represents being fit across all parts of the image in our case, and so is the application of the detector to the various parts of the image (first the top-left, then the top-middle, then the top-right, then the middle-left, then the…). The reversal is often described as being in order to make the whole operation commutative [\(f(x)*g(x)=g(x)*f(x)\), which is not always the case in math], which is reassuring and everything but doesn’t always seem to make the reasoning much clearer… Remember that \(x\) represents the convolution happening, and the convolution is basically multiplying all the parts of the functions together. Think of the two functions as curves that are being multiplied by each other when they cross over, and then think of the ‘crossing over’ as the two curves being dragged past each other and then the convolution being the multiplication of the points that line up. As \(f\) is ‘dragged’ to the right, we are ‘dragging’ \(g\) to the left, and so by flipping \(g\) we are actually preserving the ordering of the functions. Sadly, this is something that is difficult to describe, but straightforward to draw, so pay attention in my lecture if this doesn’t make much sense to read11.
10.2.3 Hands-on with convolution and network design
OK, so let’s go right ahead and fit a convolutional neural network in Keras.
# Model specification
conv <- keras_model_sequential(input_shape = c(28, 28, 1)) %>%
layer_conv_2d(filters = 20, kernel_size = c(3,3), activation = 'relu') %>%
layer_max_pooling_2d(pool_size = c(2, 2)) %>%
#layer_dropout(rate = 0.25) %>%
layer_flatten() %>%
layer_dense(units = 20, activation = 'relu') %>%
layer_dense(units = 10, activation = 'softmax') %>%
compile(
optimizer = 'adam',
loss = 'sparse_categorical_crossentropy',
metrics = c('accuracy')
)
# Re-arrange data and fit model
array.exp <- array(exp, dim=c(dim(exp), 1))
conv %>% fit(array.exp, resp, epochs = 10)
#> Epoch 1/10
#> 1875/1875 - 4s - 2ms/step - accuracy: 0.8324 - loss: 0.4737
#> Epoch 2/10
#> 1875/1875 - 4s - 2ms/step - accuracy: 0.8871 - loss: 0.3161
#> Epoch 3/10
#> 1875/1875 - 4s - 2ms/step - accuracy: 0.9015 - loss: 0.2773
#> Epoch 4/10
#> 1875/1875 - 4s - 2ms/step - accuracy: 0.9095 - loss: 0.2520
#> Epoch 5/10
#> 1875/1875 - 4s - 2ms/step - accuracy: 0.9151 - loss: 0.2328
#> Epoch 6/10
#> 1875/1875 - 4s - 2ms/step - accuracy: 0.9210 - loss: 0.2186
#> Epoch 7/10
#> 1875/1875 - 4s - 2ms/step - accuracy: 0.9255 - loss: 0.2045
#> Epoch 8/10
#> 1875/1875 - 4s - 2ms/step - accuracy: 0.9298 - loss: 0.1921
#> Epoch 9/10
#> 1875/1875 - 4s - 2ms/step - accuracy: 0.9330 - loss: 0.1816
#> Epoch 10/10
#> 1875/1875 - 4s - 2ms/step - accuracy: 0.9358 - loss: 0.1741
# (If you want to subset the data a bit more, try)
subset.exp <- exp[1:500,,]
s.array.exp <- array(subset.exp, dim=c(dim(subset.exp), 1))
conv %>% fit(s.array.exp, resp[1:500], epochs = 10)
#> Epoch 1/10
#> 16/16 - 0s - 11ms/step - accuracy: 0.9600 - loss: 0.1180
#> Epoch 2/10
#> 16/16 - 0s - 3ms/step - accuracy: 0.9760 - loss: 0.0827
#> Epoch 3/10
#> 16/16 - 0s - 3ms/step - accuracy: 0.9860 - loss: 0.0642
#> Epoch 4/10
#> 16/16 - 0s - 3ms/step - accuracy: 0.9860 - loss: 0.0546
#> Epoch 5/10
#> 16/16 - 0s - 3ms/step - accuracy: 0.9940 - loss: 0.0456
#> Epoch 6/10
#> 16/16 - 0s - 3ms/step - accuracy: 0.9960 - loss: 0.0391
#> Epoch 7/10
#> 16/16 - 0s - 3ms/step - accuracy: 0.9980 - loss: 0.0343
#> Epoch 8/10
#> 16/16 - 0s - 3ms/step - accuracy: 0.9980 - loss: 0.0313
#> Epoch 9/10
#> 16/16 - 0s - 3ms/step - accuracy: 1.0000 - loss: 0.0276
#> Epoch 10/10
#> 16/16 - 0s - 3ms/step - accuracy: 1.0000 - loss: 0.0259
# Much faster!A few things to note before we go into the specifics of convolutional networks. First, you know how to validate models, so I’m not going to bother showing you that again. If you test this, you should find it performs better than the previous model (albeit taking more time to train; more on that in a moment). Second, we’re experienced with Keras now, so I don’t need to split my model specification across multiple lines: you can see that it’s possible to define the model without multiple <- calls, and so I’m doing it all in one. You will see people online writing functions to contain their model definitions; this apparently makes them feel clever but is unnecessary. Thirdly, because we now are dealing with images, I have to restructure the data a little bit. Keras comes with built-in functions to do this (array_reshape) but there’s no need to use these if you’re careful. Critically, however, we have made our data four-dimensional: the fourth dimension is how many channels of color there are in our image. Because ours is one channel (black–grey–white), this extra dimension contains a single column, but you can fit these models with multiple colors if you wish. You will have to do this to your validation data as well (you should be able to do this yourself using my code).
layer_conv_2d does the convolution work for us here. We specify the number of times we want to apply the detector to the image via filters; I’ve picked 20 arbitrarily, and if you pick too many or too few for it to work neatly then Keras will ‘pad’ your output for you. We also have to specify the input shape of the image (28x28, with one channel), and specify how big our detector (‘kernel’ in Keras-speak) is going to be. In this case… 3x3 images. This is it: this is the whole shebang. As the name implies, you can do 3D convolutions (videos; consider the Waterfall Illusion I discussed above), and one-dimensional convolutions which are useful in classic time-series-type analyses.
layer_max_pooling_2d is amazing. It downscales whatever it’s given to reduce its dimensionality: it takes 2-by-2 cells (defined in the layer), and compresses them (averages them) to output a single cell for each of them. This is wonderful, because it allows us to do the processing of images within our neural network. In more advanced cases, you can even have the degree of processing be determined by the network itself, such that your model will figure out what needs to be done to your data in order to get the best result12.
As you might have noticed, I’ve left an additional layers in the code above commented-out. This is to help you with your exercises below; ignore them for now.
10.2.4 Freezing networks and shattering images
A really neat feature of the %>% operator adding on to whatever has happened before is you can continue training your model from wherever you left off last. Try it now, if you wish, by providing additional training data in the form of the reduced subset code I give at the end of the example. This allows you to, for example, train a model in small batches until it’s good enough (defined however you wish) for whatever you’re trying to do. It is even possible to train part of a neural network, then plug those trained components into an even larger network. When doing so, the weights will carry over to the new network, and training will begin with those as starting points.
new <- keras_model_sequential() %>%
conv() %>%
layer_dense(units = 10, activation = "softmax") %>%
compile(
optimizer = 'adam',
loss = 'sparse_categorical_crossentropy',
metrics = c('accuracy')
)
new %>% fit(s.array.exp, resp[1:500], epochs = 10)
#> Epoch 1/10
#> 16/16 - 0s - 28ms/step - accuracy: 0.0060 - loss: 2.3909
#> Epoch 2/10
#> 16/16 - 0s - 3ms/step - accuracy: 0.0120 - loss: 2.2966
#> Epoch 3/10
#> 16/16 - 0s - 3ms/step - accuracy: 0.1680 - loss: 2.1902
#> Epoch 4/10
#> 16/16 - 0s - 3ms/step - accuracy: 0.2220 - loss: 2.1308
#> Epoch 5/10
#> 16/16 - 0s - 3ms/step - accuracy: 0.2160 - loss: 2.1000
#> Epoch 6/10
#> 16/16 - 0s - 3ms/step - accuracy: 0.2480 - loss: 2.0707
#> Epoch 7/10
#> 16/16 - 0s - 3ms/step - accuracy: 0.2920 - loss: 2.0514
#> Epoch 8/10
#> 16/16 - 0s - 3ms/step - accuracy: 0.2980 - loss: 2.0268
#> Epoch 9/10
#> 16/16 - 0s - 3ms/step - accuracy: 0.3780 - loss: 2.0072
#> Epoch 10/10
#> 16/16 - 0s - 3ms/step - accuracy: 0.4000 - loss: 1.9868Re-using things is great and all, but the trouble is the training process doesn’t necessarily get much quicker with these additions because, even though the starting point is a little different, there are still a lot of connections to train and they interact with the connections that are already trained. A better approach is to freeze a neural network, or part of a neural network, and then combine it with something else. That way its weights can’t be updated, making it faster to train. This might seem a bit weird, but it’s super useful if you’re working with some data the kind of which has been seen before. Say, for example, that you are trying to classify some faces. If someone has already built a neural network that does face classification (Google, for example), then you can take their network, leave it as-is, and then bolt some additional nodes at the bottom. It isn’t ideal, but if Google trained a network of over a million nodes on a dataset of thousands of faces, it’s going to be better than anything you could hope to do yourself. You can also, of course, selectively unfreeze certain layers if you want. The example below shows this, although I must emphasize that we’re gaining very little by doing all this freezing and unfreezing in such a trivial example.
# Freeze the second layer (arbitrarily chosen)
layer <- conv$layers[[2]]
layer$trainable <- FALSE
# Re-compile the model (you must do this before use after any freezing or unfreezing)
new %>% compile(
optimizer = 'adam',
loss = 'sparse_categorical_crossentropy',
metrics = c('accuracy')
)
# ...if you've taken my programming class, you might be surprised that we don't have to
# ...re-compile conv here. Remember TensorFlow is a Python library - call by name
# ...is going on here, and so freeze_layers has done the work there for us alreadyI want to leave you with a note on variance. Back in the regression tree exercises, we discussed how a good way to deal with over-fitting is to bootstrap your data. Bagged trees and the like use random subsets of the data to add variance to the training set, and in-so-doing, and rather counter-intuitively, make the training process much more powerful. The same is true of neural networks13, and in image analysis is often given very fancy names like augmenting. Indeed, it’s excellent because it reduces (if you’re clever about it, to zero) the number of times a network is trained with exactly the same data. There are built-in ways of doing this within Keras (look up image_data_generator), but it’s also possible to do it yourself and I think it serves as an excellent demonstration of how you can go a long way with a little thought. Before I show you how below, think for a moment about how, given everything you know, you might write some code that would introduce a little noise in your training subset. Make sure to unfreeze your layers first…
# Let's do 20 training epochs
for(i in 1:20){
# Randomly alter the training data
augmented <- array.exp + rnorm(length(as.numeric(exp)), sd=.1)
# Train once on new data
conv %>% fit(augmented, resp, epochs=1)
}
#> 1875/1875 - 4s - 2ms/step - accuracy: 0.8943 - loss: 0.3084
#> 1875/1875 - 4s - 2ms/step - accuracy: 0.9101 - loss: 0.2398
#> 1875/1875 - 4s - 2ms/step - accuracy: 0.9124 - loss: 0.2315
#> 1875/1875 - 4s - 2ms/step - accuracy: 0.9141 - loss: 0.2295
#> 1875/1875 - 4s - 2ms/step - accuracy: 0.9176 - loss: 0.2226
#> 1875/1875 - 4s - 2ms/step - accuracy: 0.9169 - loss: 0.2203
#> 1875/1875 - 4s - 2ms/step - accuracy: 0.9196 - loss: 0.2150
#> 1875/1875 - 4s - 2ms/step - accuracy: 0.9188 - loss: 0.2126
#> 1875/1875 - 4s - 2ms/step - accuracy: 0.9187 - loss: 0.2137
#> 1875/1875 - 4s - 2ms/step - accuracy: 0.9222 - loss: 0.2079
#> 1875/1875 - 4s - 2ms/step - accuracy: 0.9216 - loss: 0.2075
#> 1875/1875 - 4s - 2ms/step - accuracy: 0.9217 - loss: 0.2058
#> 1875/1875 - 4s - 2ms/step - accuracy: 0.9230 - loss: 0.2017
#> 1875/1875 - 4s - 2ms/step - accuracy: 0.9241 - loss: 0.2005
#> 1875/1875 - 4s - 2ms/step - accuracy: 0.9258 - loss: 0.1966
#> 1875/1875 - 4s - 2ms/step - accuracy: 0.9245 - loss: 0.1976
#> 1875/1875 - 4s - 2ms/step - accuracy: 0.9273 - loss: 0.1923
#> 1875/1875 - 4s - 2ms/step - accuracy: 0.9275 - loss: 0.1921
#> 1875/1875 - 4s - 2ms/step - accuracy: 0.9276 - loss: 0.1901
#> 1875/1875 - 4s - 2ms/step - accuracy: 0.9296 - loss: 0.1864Does your model fit any better now? Is there something I could do a little differently? I might have guessd sdomething wrong, maybe?
10.3 Recurrent neural networks
Up until now, we have covered the kinds of problems that deep learning can tackle that can be formulated in ways that we could imagine traditional statistical models working. Even image analysis is an example of this—we are essentially taking a set of input variables (pixel values within an image) and predicting a binary variable (e.g., cat/dog) on the basis of those explanatory variables. Recurrent neural networks allow us to go further, and consider explanatory ‘variables’ such as input sentences (e.g., “where is the rain in Spain?”) and response ‘variables’ that are themselves sentences (e.g., “mainly on the plain”). To describe such data in traditional statistical terms—time-series data—is technically correct but misses the true magnitude of difference. Today we will be dipping our toes into recurrent neural networks—networks architectures that are capable of feeding back into themselves—in order to be able to fit deep networks to time-series data. These kinds of models are tremendously powerful, but are also tremendously difficult to get right, and so you should view this session as more of your first steps into the ocean of deep networks. It is important to remember, in reading around this literature, that such recurrent architectures are technically ‘Turing complete’ and as such you will read a lot of pontificating on the Internet about what this means for the future of artificial life. I would encourage you to be sceptical of the idea that a model you can write in a few minutes on a laptop can become sentient…
10.3.1 Time-series data and neural networks
A time-series dataset is one where there is an inherent temporal order to the data. Stock prices are a common example: each ‘row’ in such a dataset represent the price of a stock at a given time, and as we move from the top to the bottom of the dataset we are moving forwards in time. It is not always the case that the intervals in time between the entries need to be the same, but for our purposes we will assume they are. This makes it easy to indexed time-series for our convenience; a variable \(x\) is measured at a time \(t\), such that \(x_{t+1}\) was measured one unit of time (an hour, a day, whatever convention we define) after \(x_t\). Such data could be handled in a very natural way with the network architectures you already know: we could have an input node for each time-point in a time-series dataset, such that a stock dataset of 100 days of prices would have 100 input nodes, and then have all that information feed forward into whatever kind of architecture you wanted. You could even design such a network such that the linkages between the layers were incomplete; perhaps every \(x_t\) is linked to nodes directly downstream from \(x_{t+1}\) and \(x_{t+2}\), but \(x_{t+1}\) is not so-linked to \(x_t\) (i.e., information can move forward in time but not backwards).
The problem with such architectures is they become huge very quickly, and are extremely inefficient because there is no obvious way to generalise very simple properties in time-series analysis such as a constant temporal lag across observations. In the example I gave earlier, consider that the network would have to learn weights for each possible temporal lag (\(t+1\), \(t+2\), etc.) for each time-point separately: there is no way to share the weights of connections across spacings. There are many possible solutions to this problem, but the most popular is the recurrent neural network, where nodes are allowed to feed back into the network. An example of this is shown in Figure 10.1, where we see the structure of the network compressed and also ‘rolled out’. In a recurrent neural network the state of the network for a given time-step is allowed to propagate forward to affect the state of variables in later time-step(s). You can think of this as allowing the network to have a form of memory: the signal sent from a node at one point in time can affect future states of nodes.
There are a few practical considerations for such recurrent networks; many of them involve problems that crop up a lot with language translation that I am introducing you to in order to allow you to read the literature widely, not because I am hoping you will become linguists. The first is whether we want them to ‘look’ ahead or to the past. While you might think that only allowing a model to propagate information forward is best (because otherwise you are ‘cheating’ and going backwards in time), remember that not all time-series dataset involve things that are moving through time. For example, sentences are often treated as time-series datasets, and in networks designed to translate from one language to another it is quite common to have separate parts of the network that look forward through the sentence and that look backward. The second problem is how to encode words so that the network can work with them. Perhaps your first thought is to make a factor variable with one level per word; thus, for example, the sentence “Will gave Will a gift” could be encoded as a numeric sequence “1 2 1 3 4” (1=‘Will’, 2=‘gave’, 3=‘a’, 4=‘gift’ 14). Such one-hot encoding works well apart from when you have extremely large vocabularies (consider the problems inherent in “Will” vs. “Bonnie” and “gave” vs. “lent”) and grammar (consider “gave” vs. “will give”), and there is a vast literature on hierarchical data structures to try and account for such problems. Finally, there is also the problem of how to predict when a sentence (or paragraph, or time series more generally) will end. Again, the trivial and obvious solution (a special terminator signal, much like the NULL terminators used in strings) works well, it creates problems when the model must predict forward or backwards from such a terminator. For example, special care must be given to the training of models that incorporate full stops—what should the model output as the most likely word to follow a full stop? What about a question mark?… We will happily ignore such problems by focusing on more traditional time-series datasets!
10.3.2 Hands-on with recurrent neural networks
Alright, let’s just get on with it shall we? Let’s fit a very simple RNN to a single, simulated time series dataset.
# Get Keras ready
library(keras3)
# Simulate a time series (y) with first-order autocorrelation
y <- rep(0, 101)
for(i in seq(2, length(y)))
y[i] <- y[i-1] + rnorm(1)
y <- as.numeric(scale(y))
# ... why are we scaling the variable?
# Make a predictor variable that is the previous time-step (x)
# and then split into training/test data
x <- y[1:100]
x_train <- array(t(matrix(x[1:50], 5, 10)), dim=c(10,5,1))
x_test <- array(t(matrix(x[51:100], 5, 10)), dim=c(10,5,1))
y_train <- y[seq(6,51, by=5)]
y_test <- y[seq(56,101, by=5)]
# ... note we are using arrays, grouping our data into runs of
# 5 sequential points, with 10 training/test replicates,
# and an overall dimension of 1 (i.e., a single variable)
# Build the model itself - spot the RNN layer!...
model <- keras_model_sequential(input_shape = c(5, 1)) %>%
layer_dense(units=5) %>%
layer_simple_rnn(units=5) %>%
layer_dense(units=1)
# Train the model
model %>%
compile(
loss = "mean_squared_error",
optimizer = optimizer_rmsprop(),
metrics = list("mean_squared_error")
)
# Let store the training data so that we can...
history <- model %>% fit(
x_train, y_train, epochs = 500
)
#> Epoch 1/500
#> 1/1 - 0s - 404ms/step - loss: 0.2370 - mean_squared_error: 0.2370
#> Epoch 2/500
#> 1/1 - 0s - 19ms/step - loss: 0.2101 - mean_squared_error: 0.2101
#> Epoch 3/500
#> 1/1 - 0s - 20ms/step - loss: 0.1933 - mean_squared_error: 0.1933
#> Epoch 4/500
#> 1/1 - 0s - 19ms/step - loss: 0.1807 - mean_squared_error: 0.1807
#> Epoch 5/500
#> 1/1 - 0s - 19ms/step - loss: 0.1705 - mean_squared_error: 0.1705
#> Epoch 6/500
#> 1/1 - 0s - 23ms/step - loss: 0.1619 - mean_squared_error: 0.1619
#> Epoch 7/500
#> 1/1 - 0s - 19ms/step - loss: 0.1545 - mean_squared_error: 0.1545
#> Epoch 8/500
#> 1/1 - 0s - 19ms/step - loss: 0.1480 - mean_squared_error: 0.1480
#> Epoch 9/500
#> 1/1 - 0s - 19ms/step - loss: 0.1423 - mean_squared_error: 0.1423
#> Epoch 10/500
#> 1/1 - 0s - 19ms/step - loss: 0.1371 - mean_squared_error: 0.1371
#> Epoch 11/500
#> 1/1 - 0s - 19ms/step - loss: 0.1324 - mean_squared_error: 0.1324
#> Epoch 12/500
#> 1/1 - 0s - 19ms/step - loss: 0.1281 - mean_squared_error: 0.1281
#> Epoch 13/500
#> 1/1 - 0s - 19ms/step - loss: 0.1243 - mean_squared_error: 0.1243
#> Epoch 14/500
#> 1/1 - 0s - 19ms/step - loss: 0.1207 - mean_squared_error: 0.1207
#> Epoch 15/500
#> 1/1 - 0s - 19ms/step - loss: 0.1174 - mean_squared_error: 0.1174
#> Epoch 16/500
#> 1/1 - 0s - 20ms/step - loss: 0.1144 - mean_squared_error: 0.1144
#> Epoch 17/500
#> 1/1 - 0s - 19ms/step - loss: 0.1116 - mean_squared_error: 0.1116
#> Epoch 18/500
#> 1/1 - 0s - 19ms/step - loss: 0.1090 - mean_squared_error: 0.1090
#> Epoch 19/500
#> 1/1 - 0s - 19ms/step - loss: 0.1066 - mean_squared_error: 0.1066
#> Epoch 20/500
#> 1/1 - 0s - 19ms/step - loss: 0.1044 - mean_squared_error: 0.1044
#> Epoch 21/500
#> 1/1 - 0s - 21ms/step - loss: 0.1023 - mean_squared_error: 0.1023
#> Epoch 22/500
#> 1/1 - 0s - 20ms/step - loss: 0.1004 - mean_squared_error: 0.1004
#> Epoch 23/500
#> 1/1 - 0s - 20ms/step - loss: 0.0986 - mean_squared_error: 0.0986
#> Epoch 24/500
#> 1/1 - 0s - 18ms/step - loss: 0.0969 - mean_squared_error: 0.0969
#> Epoch 25/500
#> 1/1 - 0s - 18ms/step - loss: 0.0953 - mean_squared_error: 0.0953
#> Epoch 26/500
#> 1/1 - 0s - 20ms/step - loss: 0.0938 - mean_squared_error: 0.0938
#> Epoch 27/500
#> 1/1 - 0s - 20ms/step - loss: 0.0924 - mean_squared_error: 0.0924
#> Epoch 28/500
#> 1/1 - 0s - 21ms/step - loss: 0.0911 - mean_squared_error: 0.0911
#> Epoch 29/500
#> 1/1 - 0s - 19ms/step - loss: 0.0898 - mean_squared_error: 0.0898
#> Epoch 30/500
#> 1/1 - 0s - 19ms/step - loss: 0.0886 - mean_squared_error: 0.0886
#> Epoch 31/500
#> 1/1 - 0s - 20ms/step - loss: 0.0874 - mean_squared_error: 0.0874
#> Epoch 32/500
#> 1/1 - 0s - 19ms/step - loss: 0.0863 - mean_squared_error: 0.0863
#> Epoch 33/500
#> 1/1 - 0s - 20ms/step - loss: 0.0852 - mean_squared_error: 0.0852
#> Epoch 34/500
#> 1/1 - 0s - 19ms/step - loss: 0.0842 - mean_squared_error: 0.0842
#> Epoch 35/500
#> 1/1 - 0s - 18ms/step - loss: 0.0832 - mean_squared_error: 0.0832
#> Epoch 36/500
#> 1/1 - 0s - 18ms/step - loss: 0.0822 - mean_squared_error: 0.0822
#> Epoch 37/500
#> 1/1 - 0s - 18ms/step - loss: 0.0812 - mean_squared_error: 0.0812
#> Epoch 38/500
#> 1/1 - 0s - 19ms/step - loss: 0.0803 - mean_squared_error: 0.0803
#> Epoch 39/500
#> 1/1 - 0s - 20ms/step - loss: 0.0793 - mean_squared_error: 0.0793
#> Epoch 40/500
#> 1/1 - 0s - 19ms/step - loss: 0.0784 - mean_squared_error: 0.0784
#> Epoch 41/500
#> 1/1 - 0s - 20ms/step - loss: 0.0775 - mean_squared_error: 0.0775
#> Epoch 42/500
#> 1/1 - 0s - 19ms/step - loss: 0.0766 - mean_squared_error: 0.0766
#> Epoch 43/500
#> 1/1 - 0s - 19ms/step - loss: 0.0757 - mean_squared_error: 0.0757
#> Epoch 44/500
#> 1/1 - 0s - 19ms/step - loss: 0.0748 - mean_squared_error: 0.0748
#> Epoch 45/500
#> 1/1 - 0s - 20ms/step - loss: 0.0739 - mean_squared_error: 0.0739
#> Epoch 46/500
#> 1/1 - 0s - 20ms/step - loss: 0.0730 - mean_squared_error: 0.0730
#> Epoch 47/500
#> 1/1 - 0s - 18ms/step - loss: 0.0722 - mean_squared_error: 0.0722
#> Epoch 48/500
#> 1/1 - 0s - 18ms/step - loss: 0.0713 - mean_squared_error: 0.0713
#> Epoch 49/500
#> 1/1 - 0s - 18ms/step - loss: 0.0704 - mean_squared_error: 0.0704
#> Epoch 50/500
#> 1/1 - 0s - 20ms/step - loss: 0.0695 - mean_squared_error: 0.0695
#> Epoch 51/500
#> 1/1 - 0s - 21ms/step - loss: 0.0687 - mean_squared_error: 0.0687
#> Epoch 52/500
#> 1/1 - 0s - 20ms/step - loss: 0.0678 - mean_squared_error: 0.0678
#> Epoch 53/500
#> 1/1 - 0s - 19ms/step - loss: 0.0669 - mean_squared_error: 0.0669
#> Epoch 54/500
#> 1/1 - 0s - 19ms/step - loss: 0.0660 - mean_squared_error: 0.0660
#> Epoch 55/500
#> 1/1 - 0s - 19ms/step - loss: 0.0652 - mean_squared_error: 0.0652
#> Epoch 56/500
#> 1/1 - 0s - 19ms/step - loss: 0.0643 - mean_squared_error: 0.0643
#> Epoch 57/500
#> 1/1 - 0s - 23ms/step - loss: 0.0634 - mean_squared_error: 0.0634
#> Epoch 58/500
#> 1/1 - 0s - 19ms/step - loss: 0.0625 - mean_squared_error: 0.0625
#> Epoch 59/500
#> 1/1 - 0s - 18ms/step - loss: 0.0616 - mean_squared_error: 0.0616
#> Epoch 60/500
#> 1/1 - 0s - 19ms/step - loss: 0.0608 - mean_squared_error: 0.0608
#> Epoch 61/500
#> 1/1 - 0s - 20ms/step - loss: 0.0599 - mean_squared_error: 0.0599
#> Epoch 62/500
#> 1/1 - 0s - 20ms/step - loss: 0.0590 - mean_squared_error: 0.0590
#> Epoch 63/500
#> 1/1 - 0s - 19ms/step - loss: 0.0581 - mean_squared_error: 0.0581
#> Epoch 64/500
#> 1/1 - 0s - 18ms/step - loss: 0.0572 - mean_squared_error: 0.0572
#> Epoch 65/500
#> 1/1 - 0s - 18ms/step - loss: 0.0563 - mean_squared_error: 0.0563
#> Epoch 66/500
#> 1/1 - 0s - 18ms/step - loss: 0.0555 - mean_squared_error: 0.0555
#> Epoch 67/500
#> 1/1 - 0s - 21ms/step - loss: 0.0546 - mean_squared_error: 0.0546
#> Epoch 68/500
#> 1/1 - 0s - 20ms/step - loss: 0.0537 - mean_squared_error: 0.0537
#> Epoch 69/500
#> 1/1 - 0s - 20ms/step - loss: 0.0529 - mean_squared_error: 0.0529
#> Epoch 70/500
#> 1/1 - 0s - 19ms/step - loss: 0.0520 - mean_squared_error: 0.0520
#> Epoch 71/500
#> 1/1 - 0s - 22ms/step - loss: 0.0512 - mean_squared_error: 0.0512
#> Epoch 72/500
#> 1/1 - 0s - 19ms/step - loss: 0.0504 - mean_squared_error: 0.0504
#> Epoch 73/500
#> 1/1 - 0s - 21ms/step - loss: 0.0496 - mean_squared_error: 0.0496
#> Epoch 74/500
#> 1/1 - 0s - 18ms/step - loss: 0.0488 - mean_squared_error: 0.0488
#> Epoch 75/500
#> 1/1 - 0s - 19ms/step - loss: 0.0481 - mean_squared_error: 0.0481
#> Epoch 76/500
#> 1/1 - 0s - 19ms/step - loss: 0.0473 - mean_squared_error: 0.0473
#> Epoch 77/500
#> 1/1 - 0s - 21ms/step - loss: 0.0466 - mean_squared_error: 0.0466
#> Epoch 78/500
#> 1/1 - 0s - 21ms/step - loss: 0.0459 - mean_squared_error: 0.0459
#> Epoch 79/500
#> 1/1 - 0s - 19ms/step - loss: 0.0453 - mean_squared_error: 0.0453
#> Epoch 80/500
#> 1/1 - 0s - 20ms/step - loss: 0.0446 - mean_squared_error: 0.0446
#> Epoch 81/500
#> 1/1 - 0s - 20ms/step - loss: 0.0440 - mean_squared_error: 0.0440
#> Epoch 82/500
#> 1/1 - 0s - 19ms/step - loss: 0.0434 - mean_squared_error: 0.0434
#> Epoch 83/500
#> 1/1 - 0s - 19ms/step - loss: 0.0429 - mean_squared_error: 0.0429
#> Epoch 84/500
#> 1/1 - 0s - 18ms/step - loss: 0.0423 - mean_squared_error: 0.0423
#> Epoch 85/500
#> 1/1 - 0s - 18ms/step - loss: 0.0418 - mean_squared_error: 0.0418
#> Epoch 86/500
#> 1/1 - 0s - 19ms/step - loss: 0.0413 - mean_squared_error: 0.0413
#> Epoch 87/500
#> 1/1 - 0s - 19ms/step - loss: 0.0409 - mean_squared_error: 0.0409
#> Epoch 88/500
#> 1/1 - 0s - 19ms/step - loss: 0.0404 - mean_squared_error: 0.0404
#> Epoch 89/500
#> 1/1 - 0s - 20ms/step - loss: 0.0400 - mean_squared_error: 0.0400
#> Epoch 90/500
#> 1/1 - 0s - 20ms/step - loss: 0.0396 - mean_squared_error: 0.0396
#> Epoch 91/500
#> 1/1 - 0s - 18ms/step - loss: 0.0392 - mean_squared_error: 0.0392
#> Epoch 92/500
#> 1/1 - 0s - 19ms/step - loss: 0.0389 - mean_squared_error: 0.0389
#> Epoch 93/500
#> 1/1 - 0s - 19ms/step - loss: 0.0385 - mean_squared_error: 0.0385
#> Epoch 94/500
#> 1/1 - 0s - 20ms/step - loss: 0.0383 - mean_squared_error: 0.0383
#> Epoch 95/500
#> 1/1 - 0s - 20ms/step - loss: 0.0383 - mean_squared_error: 0.0383
#> Epoch 96/500
#> 1/1 - 0s - 19ms/step - loss: 0.0380 - mean_squared_error: 0.0380
#> Epoch 97/500
#> 1/1 - 0s - 18ms/step - loss: 0.0376 - mean_squared_error: 0.0376
#> Epoch 98/500
#> 1/1 - 0s - 18ms/step - loss: 0.0372 - mean_squared_error: 0.0372
#> Epoch 99/500
#> 1/1 - 0s - 20ms/step - loss: 0.0369 - mean_squared_error: 0.0369
#> Epoch 100/500
#> 1/1 - 0s - 20ms/step - loss: 0.0367 - mean_squared_error: 0.0367
#> Epoch 101/500
#> 1/1 - 0s - 18ms/step - loss: 0.0364 - mean_squared_error: 0.0364
#> Epoch 102/500
#> 1/1 - 0s - 18ms/step - loss: 0.0362 - mean_squared_error: 0.0362
#> Epoch 103/500
#> 1/1 - 0s - 19ms/step - loss: 0.0360 - mean_squared_error: 0.0360
#> Epoch 104/500
#> 1/1 - 0s - 20ms/step - loss: 0.0358 - mean_squared_error: 0.0358
#> Epoch 105/500
#> 1/1 - 0s - 20ms/step - loss: 0.0356 - mean_squared_error: 0.0356
#> Epoch 106/500
#> 1/1 - 0s - 19ms/step - loss: 0.0354 - mean_squared_error: 0.0354
#> Epoch 107/500
#> 1/1 - 0s - 19ms/step - loss: 0.0352 - mean_squared_error: 0.0352
#> Epoch 108/500
#> 1/1 - 0s - 18ms/step - loss: 0.0351 - mean_squared_error: 0.0351
#> Epoch 109/500
#> 1/1 - 0s - 18ms/step - loss: 0.0350 - mean_squared_error: 0.0350
#> Epoch 110/500
#> 1/1 - 0s - 19ms/step - loss: 0.0348 - mean_squared_error: 0.0348
#> Epoch 111/500
#> 1/1 - 0s - 19ms/step - loss: 0.0346 - mean_squared_error: 0.0346
#> Epoch 112/500
#> 1/1 - 0s - 22ms/step - loss: 0.0344 - mean_squared_error: 0.0344
#> Epoch 113/500
#> 1/1 - 0s - 19ms/step - loss: 0.0342 - mean_squared_error: 0.0342
#> Epoch 114/500
#> 1/1 - 0s - 19ms/step - loss: 0.0340 - mean_squared_error: 0.0340
#> Epoch 115/500
#> 1/1 - 0s - 19ms/step - loss: 0.0338 - mean_squared_error: 0.0338
#> Epoch 116/500
#> 1/1 - 0s - 19ms/step - loss: 0.0336 - mean_squared_error: 0.0336
#> Epoch 117/500
#> 1/1 - 0s - 21ms/step - loss: 0.0335 - mean_squared_error: 0.0335
#> Epoch 118/500
#> 1/1 - 0s - 19ms/step - loss: 0.0333 - mean_squared_error: 0.0333
#> Epoch 119/500
#> 1/1 - 0s - 20ms/step - loss: 0.0332 - mean_squared_error: 0.0332
#> Epoch 120/500
#> 1/1 - 0s - 18ms/step - loss: 0.0331 - mean_squared_error: 0.0331
#> Epoch 121/500
#> 1/1 - 0s - 19ms/step - loss: 0.0329 - mean_squared_error: 0.0329
#> Epoch 122/500
#> 1/1 - 0s - 19ms/step - loss: 0.0328 - mean_squared_error: 0.0328
#> Epoch 123/500
#> 1/1 - 0s - 20ms/step - loss: 0.0327 - mean_squared_error: 0.0327
#> Epoch 124/500
#> 1/1 - 0s - 20ms/step - loss: 0.0325 - mean_squared_error: 0.0325
#> Epoch 125/500
#> 1/1 - 0s - 18ms/step - loss: 0.0324 - mean_squared_error: 0.0324
#> Epoch 126/500
#> 1/1 - 0s - 19ms/step - loss: 0.0322 - mean_squared_error: 0.0322
#> Epoch 127/500
#> 1/1 - 0s - 19ms/step - loss: 0.0321 - mean_squared_error: 0.0321
#> Epoch 128/500
#> 1/1 - 0s - 23ms/step - loss: 0.0319 - mean_squared_error: 0.0319
#> Epoch 129/500
#> 1/1 - 0s - 22ms/step - loss: 0.0317 - mean_squared_error: 0.0317
#> Epoch 130/500
#> 1/1 - 0s - 23ms/step - loss: 0.0316 - mean_squared_error: 0.0316
#> Epoch 131/500
#> 1/1 - 0s - 19ms/step - loss: 0.0315 - mean_squared_error: 0.0315
#> Epoch 132/500
#> 1/1 - 0s - 19ms/step - loss: 0.0313 - mean_squared_error: 0.0313
#> Epoch 133/500
#> 1/1 - 0s - 18ms/step - loss: 0.0312 - mean_squared_error: 0.0312
#> Epoch 134/500
#> 1/1 - 0s - 18ms/step - loss: 0.0311 - mean_squared_error: 0.0311
#> Epoch 135/500
#> 1/1 - 0s - 18ms/step - loss: 0.0310 - mean_squared_error: 0.0310
#> Epoch 136/500
#> 1/1 - 0s - 21ms/step - loss: 0.0308 - mean_squared_error: 0.0308
#> Epoch 137/500
#> 1/1 - 0s - 20ms/step - loss: 0.0307 - mean_squared_error: 0.0307
#> Epoch 138/500
#> 1/1 - 0s - 19ms/step - loss: 0.0306 - mean_squared_error: 0.0306
#> Epoch 139/500
#> 1/1 - 0s - 18ms/step - loss: 0.0305 - mean_squared_error: 0.0305
#> Epoch 140/500
#> 1/1 - 0s - 18ms/step - loss: 0.0303 - mean_squared_error: 0.0303
#> Epoch 141/500
#> 1/1 - 0s - 20ms/step - loss: 0.0302 - mean_squared_error: 0.0302
#> Epoch 142/500
#> 1/1 - 0s - 21ms/step - loss: 0.0300 - mean_squared_error: 0.0300
#> Epoch 143/500
#> 1/1 - 0s - 19ms/step - loss: 0.0299 - mean_squared_error: 0.0299
#> Epoch 144/500
#> 1/1 - 0s - 20ms/step - loss: 0.0298 - mean_squared_error: 0.0298
#> Epoch 145/500
#> 1/1 - 0s - 21ms/step - loss: 0.0297 - mean_squared_error: 0.0297
#> Epoch 146/500
#> 1/1 - 0s - 21ms/step - loss: 0.0295 - mean_squared_error: 0.0295
#> Epoch 147/500
#> 1/1 - 0s - 21ms/step - loss: 0.0294 - mean_squared_error: 0.0294
#> Epoch 148/500
#> 1/1 - 0s - 19ms/step - loss: 0.0293 - mean_squared_error: 0.0293
#> Epoch 149/500
#> 1/1 - 0s - 19ms/step - loss: 0.0292 - mean_squared_error: 0.0292
#> Epoch 150/500
#> 1/1 - 0s - 20ms/step - loss: 0.0291 - mean_squared_error: 0.0291
#> Epoch 151/500
#> 1/1 - 0s - 19ms/step - loss: 0.0290 - mean_squared_error: 0.0290
#> Epoch 152/500
#> 1/1 - 0s - 18ms/step - loss: 0.0288 - mean_squared_error: 0.0288
#> Epoch 153/500
#> 1/1 - 0s - 20ms/step - loss: 0.0287 - mean_squared_error: 0.0287
#> Epoch 154/500
#> 1/1 - 0s - 20ms/step - loss: 0.0286 - mean_squared_error: 0.0286
#> Epoch 155/500
#> 1/1 - 0s - 27ms/step - loss: 0.0284 - mean_squared_error: 0.0284
#> Epoch 156/500
#> 1/1 - 0s - 21ms/step - loss: 0.0283 - mean_squared_error: 0.0283
#> Epoch 157/500
#> 1/1 - 0s - 19ms/step - loss: 0.0282 - mean_squared_error: 0.0282
#> Epoch 158/500
#> 1/1 - 0s - 19ms/step - loss: 0.0281 - mean_squared_error: 0.0281
#> Epoch 159/500
#> 1/1 - 0s - 18ms/step - loss: 0.0280 - mean_squared_error: 0.0280
#> Epoch 160/500
#> 1/1 - 0s - 20ms/step - loss: 0.0278 - mean_squared_error: 0.0278
#> Epoch 161/500
#> 1/1 - 0s - 19ms/step - loss: 0.0277 - mean_squared_error: 0.0277
#> Epoch 162/500
#> 1/1 - 0s - 18ms/step - loss: 0.0276 - mean_squared_error: 0.0276
#> Epoch 163/500
#> 1/1 - 0s - 19ms/step - loss: 0.0275 - mean_squared_error: 0.0275
#> Epoch 164/500
#> 1/1 - 0s - 19ms/step - loss: 0.0274 - mean_squared_error: 0.0274
#> Epoch 165/500
#> 1/1 - 0s - 19ms/step - loss: 0.0273 - mean_squared_error: 0.0273
#> Epoch 166/500
#> 1/1 - 0s - 21ms/step - loss: 0.0271 - mean_squared_error: 0.0271
#> Epoch 167/500
#> 1/1 - 0s - 19ms/step - loss: 0.0270 - mean_squared_error: 0.0270
#> Epoch 168/500
#> 1/1 - 0s - 18ms/step - loss: 0.0269 - mean_squared_error: 0.0269
#> Epoch 169/500
#> 1/1 - 0s - 18ms/step - loss: 0.0268 - mean_squared_error: 0.0268
#> Epoch 170/500
#> 1/1 - 0s - 18ms/step - loss: 0.0267 - mean_squared_error: 0.0267
#> Epoch 171/500
#> 1/1 - 0s - 19ms/step - loss: 0.0266 - mean_squared_error: 0.0266
#> Epoch 172/500
#> 1/1 - 0s - 19ms/step - loss: 0.0264 - mean_squared_error: 0.0264
#> Epoch 173/500
#> 1/1 - 0s - 20ms/step - loss: 0.0263 - mean_squared_error: 0.0263
#> Epoch 174/500
#> 1/1 - 0s - 19ms/step - loss: 0.0262 - mean_squared_error: 0.0262
#> Epoch 175/500
#> 1/1 - 0s - 20ms/step - loss: 0.0261 - mean_squared_error: 0.0261
#> Epoch 176/500
#> 1/1 - 0s - 19ms/step - loss: 0.0260 - mean_squared_error: 0.0260
#> Epoch 177/500
#> 1/1 - 0s - 19ms/step - loss: 0.0259 - mean_squared_error: 0.0259
#> Epoch 178/500
#> 1/1 - 0s - 19ms/step - loss: 0.0258 - mean_squared_error: 0.0258
#> Epoch 179/500
#> 1/1 - 0s - 20ms/step - loss: 0.0256 - mean_squared_error: 0.0256
#> Epoch 180/500
#> 1/1 - 0s - 21ms/step - loss: 0.0255 - mean_squared_error: 0.0255
#> Epoch 181/500
#> 1/1 - 0s - 19ms/step - loss: 0.0254 - mean_squared_error: 0.0254
#> Epoch 182/500
#> 1/1 - 0s - 19ms/step - loss: 0.0253 - mean_squared_error: 0.0253
#> Epoch 183/500
#> 1/1 - 0s - 19ms/step - loss: 0.0252 - mean_squared_error: 0.0252
#> Epoch 184/500
#> 1/1 - 0s - 20ms/step - loss: 0.0251 - mean_squared_error: 0.0251
#> Epoch 185/500
#> 1/1 - 0s - 18ms/step - loss: 0.0249 - mean_squared_error: 0.0249
#> Epoch 186/500
#> 1/1 - 0s - 19ms/step - loss: 0.0248 - mean_squared_error: 0.0248
#> Epoch 187/500
#> 1/1 - 0s - 19ms/step - loss: 0.0247 - mean_squared_error: 0.0247
#> Epoch 188/500
#> 1/1 - 0s - 19ms/step - loss: 0.0246 - mean_squared_error: 0.0246
#> Epoch 189/500
#> 1/1 - 0s - 19ms/step - loss: 0.0245 - mean_squared_error: 0.0245
#> Epoch 190/500
#> 1/1 - 0s - 18ms/step - loss: 0.0244 - mean_squared_error: 0.0244
#> Epoch 191/500
#> 1/1 - 0s - 18ms/step - loss: 0.0243 - mean_squared_error: 0.0243
#> Epoch 192/500
#> 1/1 - 0s - 19ms/step - loss: 0.0242 - mean_squared_error: 0.0242
#> Epoch 193/500
#> 1/1 - 0s - 18ms/step - loss: 0.0241 - mean_squared_error: 0.0241
#> Epoch 194/500
#> 1/1 - 0s - 18ms/step - loss: 0.0239 - mean_squared_error: 0.0239
#> Epoch 195/500
#> 1/1 - 0s - 19ms/step - loss: 0.0238 - mean_squared_error: 0.0238
#> Epoch 196/500
#> 1/1 - 0s - 20ms/step - loss: 0.0237 - mean_squared_error: 0.0237
#> Epoch 197/500
#> 1/1 - 0s - 20ms/step - loss: 0.0236 - mean_squared_error: 0.0236
#> Epoch 198/500
#> 1/1 - 0s - 19ms/step - loss: 0.0235 - mean_squared_error: 0.0235
#> Epoch 199/500
#> 1/1 - 0s - 19ms/step - loss: 0.0234 - mean_squared_error: 0.0234
#> Epoch 200/500
#> 1/1 - 0s - 18ms/step - loss: 0.0233 - mean_squared_error: 0.0233
#> Epoch 201/500
#> 1/1 - 0s - 18ms/step - loss: 0.0232 - mean_squared_error: 0.0232
#> Epoch 202/500
#> 1/1 - 0s - 21ms/step - loss: 0.0230 - mean_squared_error: 0.0230
#> Epoch 203/500
#> 1/1 - 0s - 19ms/step - loss: 0.0229 - mean_squared_error: 0.0229
#> Epoch 204/500
#> 1/1 - 0s - 20ms/step - loss: 0.0228 - mean_squared_error: 0.0228
#> Epoch 205/500
#> 1/1 - 0s - 19ms/step - loss: 0.0227 - mean_squared_error: 0.0227
#> Epoch 206/500
#> 1/1 - 0s - 19ms/step - loss: 0.0226 - mean_squared_error: 0.0226
#> Epoch 207/500
#> 1/1 - 0s - 19ms/step - loss: 0.0225 - mean_squared_error: 0.0225
#> Epoch 208/500
#> 1/1 - 0s - 22ms/step - loss: 0.0224 - mean_squared_error: 0.0224
#> Epoch 209/500
#> 1/1 - 0s - 19ms/step - loss: 0.0223 - mean_squared_error: 0.0223
#> Epoch 210/500
#> 1/1 - 0s - 18ms/step - loss: 0.0222 - mean_squared_error: 0.0222
#> Epoch 211/500
#> 1/1 - 0s - 20ms/step - loss: 0.0221 - mean_squared_error: 0.0221
#> Epoch 212/500
#> 1/1 - 0s - 21ms/step - loss: 0.0220 - mean_squared_error: 0.0220
#> Epoch 213/500
#> 1/1 - 0s - 19ms/step - loss: 0.0219 - mean_squared_error: 0.0219
#> Epoch 214/500
#> 1/1 - 0s - 19ms/step - loss: 0.0217 - mean_squared_error: 0.0217
#> Epoch 215/500
#> 1/1 - 0s - 20ms/step - loss: 0.0216 - mean_squared_error: 0.0216
#> Epoch 216/500
#> 1/1 - 0s - 19ms/step - loss: 0.0215 - mean_squared_error: 0.0215
#> Epoch 217/500
#> 1/1 - 0s - 19ms/step - loss: 0.0214 - mean_squared_error: 0.0214
#> Epoch 218/500
#> 1/1 - 0s - 20ms/step - loss: 0.0213 - mean_squared_error: 0.0213
#> Epoch 219/500
#> 1/1 - 0s - 19ms/step - loss: 0.0212 - mean_squared_error: 0.0212
#> Epoch 220/500
#> 1/1 - 0s - 19ms/step - loss: 0.0211 - mean_squared_error: 0.0211
#> Epoch 221/500
#> 1/1 - 0s - 19ms/step - loss: 0.0210 - mean_squared_error: 0.0210
#> Epoch 222/500
#> 1/1 - 0s - 19ms/step - loss: 0.0209 - mean_squared_error: 0.0209
#> Epoch 223/500
#> 1/1 - 0s - 20ms/step - loss: 0.0208 - mean_squared_error: 0.0208
#> Epoch 224/500
#> 1/1 - 0s - 19ms/step - loss: 0.0207 - mean_squared_error: 0.0207
#> Epoch 225/500
#> 1/1 - 0s - 19ms/step - loss: 0.0206 - mean_squared_error: 0.0206
#> Epoch 226/500
#> 1/1 - 0s - 20ms/step - loss: 0.0205 - mean_squared_error: 0.0205
#> Epoch 227/500
#> 1/1 - 0s - 20ms/step - loss: 0.0204 - mean_squared_error: 0.0204
#> Epoch 228/500
#> 1/1 - 0s - 19ms/step - loss: 0.0203 - mean_squared_error: 0.0203
#> Epoch 229/500
#> 1/1 - 0s - 20ms/step - loss: 0.0202 - mean_squared_error: 0.0202
#> Epoch 230/500
#> 1/1 - 0s - 19ms/step - loss: 0.0200 - mean_squared_error: 0.0200
#> Epoch 231/500
#> 1/1 - 0s - 19ms/step - loss: 0.0200 - mean_squared_error: 0.0200
#> Epoch 232/500
#> 1/1 - 0s - 19ms/step - loss: 0.0198 - mean_squared_error: 0.0198
#> Epoch 233/500
#> 1/1 - 0s - 20ms/step - loss: 0.0197 - mean_squared_error: 0.0197
#> Epoch 234/500
#> 1/1 - 0s - 20ms/step - loss: 0.0196 - mean_squared_error: 0.0196
#> Epoch 235/500
#> 1/1 - 0s - 19ms/step - loss: 0.0195 - mean_squared_error: 0.0195
#> Epoch 236/500
#> 1/1 - 0s - 18ms/step - loss: 0.0194 - mean_squared_error: 0.0194
#> Epoch 237/500
#> 1/1 - 0s - 18ms/step - loss: 0.0193 - mean_squared_error: 0.0193
#> Epoch 238/500
#> 1/1 - 0s - 19ms/step - loss: 0.0192 - mean_squared_error: 0.0192
#> Epoch 239/500
#> 1/1 - 0s - 20ms/step - loss: 0.0191 - mean_squared_error: 0.0191
#> Epoch 240/500
#> 1/1 - 0s - 19ms/step - loss: 0.0190 - mean_squared_error: 0.0190
#> Epoch 241/500
#> 1/1 - 0s - 19ms/step - loss: 0.0189 - mean_squared_error: 0.0189
#> Epoch 242/500
#> 1/1 - 0s - 18ms/step - loss: 0.0188 - mean_squared_error: 0.0188
#> Epoch 243/500
#> 1/1 - 0s - 19ms/step - loss: 0.0187 - mean_squared_error: 0.0187
#> Epoch 244/500
#> 1/1 - 0s - 18ms/step - loss: 0.0186 - mean_squared_error: 0.0186
#> Epoch 245/500
#> 1/1 - 0s - 19ms/step - loss: 0.0185 - mean_squared_error: 0.0185
#> Epoch 246/500
#> 1/1 - 0s - 21ms/step - loss: 0.0184 - mean_squared_error: 0.0184
#> Epoch 247/500
#> 1/1 - 0s - 21ms/step - loss: 0.0183 - mean_squared_error: 0.0183
#> Epoch 248/500
#> 1/1 - 0s - 20ms/step - loss: 0.0182 - mean_squared_error: 0.0182
#> Epoch 249/500
#> 1/1 - 0s - 19ms/step - loss: 0.0181 - mean_squared_error: 0.0181
#> Epoch 250/500
#> 1/1 - 0s - 21ms/step - loss: 0.0180 - mean_squared_error: 0.0180
#> Epoch 251/500
#> 1/1 - 0s - 19ms/step - loss: 0.0179 - mean_squared_error: 0.0179
#> Epoch 252/500
#> 1/1 - 0s - 20ms/step - loss: 0.0178 - mean_squared_error: 0.0178
#> Epoch 253/500
#> 1/1 - 0s - 20ms/step - loss: 0.0177 - mean_squared_error: 0.0177
#> Epoch 254/500
#> 1/1 - 0s - 20ms/step - loss: 0.0176 - mean_squared_error: 0.0176
#> Epoch 255/500
#> 1/1 - 0s - 20ms/step - loss: 0.0175 - mean_squared_error: 0.0175
#> Epoch 256/500
#> 1/1 - 0s - 19ms/step - loss: 0.0174 - mean_squared_error: 0.0174
#> Epoch 257/500
#> 1/1 - 0s - 19ms/step - loss: 0.0173 - mean_squared_error: 0.0173
#> Epoch 258/500
#> 1/1 - 0s - 20ms/step - loss: 0.0172 - mean_squared_error: 0.0172
#> Epoch 259/500
#> 1/1 - 0s - 19ms/step - loss: 0.0171 - mean_squared_error: 0.0171
#> Epoch 260/500
#> 1/1 - 0s - 20ms/step - loss: 0.0170 - mean_squared_error: 0.0170
#> Epoch 261/500
#> 1/1 - 0s - 19ms/step - loss: 0.0169 - mean_squared_error: 0.0169
#> Epoch 262/500
#> 1/1 - 0s - 18ms/step - loss: 0.0168 - mean_squared_error: 0.0168
#> Epoch 263/500
#> 1/1 - 0s - 18ms/step - loss: 0.0167 - mean_squared_error: 0.0167
#> Epoch 264/500
#> 1/1 - 0s - 19ms/step - loss: 0.0166 - mean_squared_error: 0.0166
#> Epoch 265/500
#> 1/1 - 0s - 20ms/step - loss: 0.0165 - mean_squared_error: 0.0165
#> Epoch 266/500
#> 1/1 - 0s - 20ms/step - loss: 0.0164 - mean_squared_error: 0.0164
#> Epoch 267/500
#> 1/1 - 0s - 18ms/step - loss: 0.0164 - mean_squared_error: 0.0164
#> Epoch 268/500
#> 1/1 - 0s - 18ms/step - loss: 0.0162 - mean_squared_error: 0.0162
#> Epoch 269/500
#> 1/1 - 0s - 18ms/step - loss: 0.0162 - mean_squared_error: 0.0162
#> Epoch 270/500
#> 1/1 - 0s - 20ms/step - loss: 0.0161 - mean_squared_error: 0.0161
#> Epoch 271/500
#> 1/1 - 0s - 18ms/step - loss: 0.0160 - mean_squared_error: 0.0160
#> Epoch 272/500
#> 1/1 - 0s - 19ms/step - loss: 0.0159 - mean_squared_error: 0.0159
#> Epoch 273/500
#> 1/1 - 0s - 20ms/step - loss: 0.0158 - mean_squared_error: 0.0158
#> Epoch 274/500
#> 1/1 - 0s - 18ms/step - loss: 0.0157 - mean_squared_error: 0.0157
#> Epoch 275/500
#> 1/1 - 0s - 19ms/step - loss: 0.0156 - mean_squared_error: 0.0156
#> Epoch 276/500
#> 1/1 - 0s - 19ms/step - loss: 0.0155 - mean_squared_error: 0.0155
#> Epoch 277/500
#> 1/1 - 0s - 18ms/step - loss: 0.0154 - mean_squared_error: 0.0154
#> Epoch 278/500
#> 1/1 - 0s - 19ms/step - loss: 0.0153 - mean_squared_error: 0.0153
#> Epoch 279/500
#> 1/1 - 0s - 20ms/step - loss: 0.0152 - mean_squared_error: 0.0152
#> Epoch 280/500
#> 1/1 - 0s - 18ms/step - loss: 0.0151 - mean_squared_error: 0.0151
#> Epoch 281/500
#> 1/1 - 0s - 19ms/step - loss: 0.0151 - mean_squared_error: 0.0151
#> Epoch 282/500
#> 1/1 - 0s - 19ms/step - loss: 0.0150 - mean_squared_error: 0.0150
#> Epoch 283/500
#> 1/1 - 0s - 19ms/step - loss: 0.0149 - mean_squared_error: 0.0149
#> Epoch 284/500
#> 1/1 - 0s - 21ms/step - loss: 0.0148 - mean_squared_error: 0.0148
#> Epoch 285/500
#> 1/1 - 0s - 19ms/step - loss: 0.0147 - mean_squared_error: 0.0147
#> Epoch 286/500
#> 1/1 - 0s - 18ms/step - loss: 0.0146 - mean_squared_error: 0.0146
#> Epoch 287/500
#> 1/1 - 0s - 20ms/step - loss: 0.0145 - mean_squared_error: 0.0145
#> Epoch 288/500
#> 1/1 - 0s - 19ms/step - loss: 0.0144 - mean_squared_error: 0.0144
#> Epoch 289/500
#> 1/1 - 0s - 21ms/step - loss: 0.0144 - mean_squared_error: 0.0144
#> Epoch 290/500
#> 1/1 - 0s - 19ms/step - loss: 0.0143 - mean_squared_error: 0.0143
#> Epoch 291/500
#> 1/1 - 0s - 18ms/step - loss: 0.0142 - mean_squared_error: 0.0142
#> Epoch 292/500
#> 1/1 - 0s - 19ms/step - loss: 0.0141 - mean_squared_error: 0.0141
#> Epoch 293/500
#> 1/1 - 0s - 19ms/step - loss: 0.0140 - mean_squared_error: 0.0140
#> Epoch 294/500
#> 1/1 - 0s - 20ms/step - loss: 0.0139 - mean_squared_error: 0.0139
#> Epoch 295/500
#> 1/1 - 0s - 20ms/step - loss: 0.0139 - mean_squared_error: 0.0139
#> Epoch 296/500
#> 1/1 - 0s - 19ms/step - loss: 0.0138 - mean_squared_error: 0.0138
#> Epoch 297/500
#> 1/1 - 0s - 22ms/step - loss: 0.0137 - mean_squared_error: 0.0137
#> Epoch 298/500
#> 1/1 - 0s - 19ms/step - loss: 0.0136 - mean_squared_error: 0.0136
#> Epoch 299/500
#> 1/1 - 0s - 19ms/step - loss: 0.0135 - mean_squared_error: 0.0135
#> Epoch 300/500
#> 1/1 - 0s - 20ms/step - loss: 0.0135 - mean_squared_error: 0.0135
#> Epoch 301/500
#> 1/1 - 0s - 21ms/step - loss: 0.0134 - mean_squared_error: 0.0134
#> Epoch 302/500
#> 1/1 - 0s - 21ms/step - loss: 0.0133 - mean_squared_error: 0.0133
#> Epoch 303/500
#> 1/1 - 0s - 19ms/step - loss: 0.0132 - mean_squared_error: 0.0132
#> Epoch 304/500
#> 1/1 - 0s - 19ms/step - loss: 0.0131 - mean_squared_error: 0.0131
#> Epoch 305/500
#> 1/1 - 0s - 19ms/step - loss: 0.0131 - mean_squared_error: 0.0131
#> Epoch 306/500
#> 1/1 - 0s - 19ms/step - loss: 0.0130 - mean_squared_error: 0.0130
#> Epoch 307/500
#> 1/1 - 0s - 19ms/step - loss: 0.0129 - mean_squared_error: 0.0129
#> Epoch 308/500
#> 1/1 - 0s - 19ms/step - loss: 0.0128 - mean_squared_error: 0.0128
#> Epoch 309/500
#> 1/1 - 0s - 20ms/step - loss: 0.0128 - mean_squared_error: 0.0128
#> Epoch 310/500
#> 1/1 - 0s - 20ms/step - loss: 0.0127 - mean_squared_error: 0.0127
#> Epoch 311/500
#> 1/1 - 0s - 18ms/step - loss: 0.0126 - mean_squared_error: 0.0126
#> Epoch 312/500
#> 1/1 - 0s - 19ms/step - loss: 0.0125 - mean_squared_error: 0.0125
#> Epoch 313/500
#> 1/1 - 0s - 20ms/step - loss: 0.0125 - mean_squared_error: 0.0125
#> Epoch 314/500
#> 1/1 - 0s - 21ms/step - loss: 0.0124 - mean_squared_error: 0.0124
#> Epoch 315/500
#> 1/1 - 0s - 19ms/step - loss: 0.0123 - mean_squared_error: 0.0123
#> Epoch 316/500
#> 1/1 - 0s - 18ms/step - loss: 0.0123 - mean_squared_error: 0.0123
#> Epoch 317/500
#> 1/1 - 0s - 19ms/step - loss: 0.0122 - mean_squared_error: 0.0122
#> Epoch 318/500
#> 1/1 - 0s - 19ms/step - loss: 0.0121 - mean_squared_error: 0.0121
#> Epoch 319/500
#> 1/1 - 0s - 19ms/step - loss: 0.0121 - mean_squared_error: 0.0121
#> Epoch 320/500
#> 1/1 - 0s - 20ms/step - loss: 0.0120 - mean_squared_error: 0.0120
#> Epoch 321/500
#> 1/1 - 0s - 20ms/step - loss: 0.0119 - mean_squared_error: 0.0119
#> Epoch 322/500
#> 1/1 - 0s - 21ms/step - loss: 0.0118 - mean_squared_error: 0.0118
#> Epoch 323/500
#> 1/1 - 0s - 19ms/step - loss: 0.0118 - mean_squared_error: 0.0118
#> Epoch 324/500
#> 1/1 - 0s - 19ms/step - loss: 0.0117 - mean_squared_error: 0.0117
#> Epoch 325/500
#> 1/1 - 0s - 19ms/step - loss: 0.0117 - mean_squared_error: 0.0117
#> Epoch 326/500
#> 1/1 - 0s - 18ms/step - loss: 0.0116 - mean_squared_error: 0.0116
#> Epoch 327/500
#> 1/1 - 0s - 18ms/step - loss: 0.0115 - mean_squared_error: 0.0115
#> Epoch 328/500
#> 1/1 - 0s - 18ms/step - loss: 0.0114 - mean_squared_error: 0.0114
#> Epoch 329/500
#> 1/1 - 0s - 19ms/step - loss: 0.0114 - mean_squared_error: 0.0114
#> Epoch 330/500
#> 1/1 - 0s - 19ms/step - loss: 0.0113 - mean_squared_error: 0.0113
#> Epoch 331/500
#> 1/1 - 0s - 18ms/step - loss: 0.0113 - mean_squared_error: 0.0113
#> Epoch 332/500
#> 1/1 - 0s - 20ms/step - loss: 0.0112 - mean_squared_error: 0.0112
#> Epoch 333/500
#> 1/1 - 0s - 19ms/step - loss: 0.0112 - mean_squared_error: 0.0112
#> Epoch 334/500
#> 1/1 - 0s - 20ms/step - loss: 0.0111 - mean_squared_error: 0.0111
#> Epoch 335/500
#> 1/1 - 0s - 19ms/step - loss: 0.0110 - mean_squared_error: 0.0110
#> Epoch 336/500
#> 1/1 - 0s - 18ms/step - loss: 0.0110 - mean_squared_error: 0.0110
#> Epoch 337/500
#> 1/1 - 0s - 20ms/step - loss: 0.0109 - mean_squared_error: 0.0109
#> Epoch 338/500
#> 1/1 - 0s - 19ms/step - loss: 0.0108 - mean_squared_error: 0.0108
#> Epoch 339/500
#> 1/1 - 0s - 21ms/step - loss: 0.0108 - mean_squared_error: 0.0108
#> Epoch 340/500
#> 1/1 - 0s - 19ms/step - loss: 0.0107 - mean_squared_error: 0.0107
#> Epoch 341/500
#> 1/1 - 0s - 21ms/step - loss: 0.0107 - mean_squared_error: 0.0107
#> Epoch 342/500
#> 1/1 - 0s - 20ms/step - loss: 0.0106 - mean_squared_error: 0.0106
#> Epoch 343/500
#> 1/1 - 0s - 20ms/step - loss: 0.0106 - mean_squared_error: 0.0106
#> Epoch 344/500
#> 1/1 - 0s - 18ms/step - loss: 0.0105 - mean_squared_error: 0.0105
#> Epoch 345/500
#> 1/1 - 0s - 18ms/step - loss: 0.0105 - mean_squared_error: 0.0105
#> Epoch 346/500
#> 1/1 - 0s - 18ms/step - loss: 0.0104 - mean_squared_error: 0.0104
#> Epoch 347/500
#> 1/1 - 0s - 21ms/step - loss: 0.0104 - mean_squared_error: 0.0104
#> Epoch 348/500
#> 1/1 - 0s - 22ms/step - loss: 0.0103 - mean_squared_error: 0.0103
#> Epoch 349/500
#> 1/1 - 0s - 20ms/step - loss: 0.0103 - mean_squared_error: 0.0103
#> Epoch 350/500
#> 1/1 - 0s - 18ms/step - loss: 0.0102 - mean_squared_error: 0.0102
#> Epoch 351/500
#> 1/1 - 0s - 19ms/step - loss: 0.0102 - mean_squared_error: 0.0102
#> Epoch 352/500
#> 1/1 - 0s - 21ms/step - loss: 0.0101 - mean_squared_error: 0.0101
#> Epoch 353/500
#> 1/1 - 0s - 20ms/step - loss: 0.0101 - mean_squared_error: 0.0101
#> Epoch 354/500
#> 1/1 - 0s - 19ms/step - loss: 0.0100 - mean_squared_error: 0.0100
#> Epoch 355/500
#> 1/1 - 0s - 18ms/step - loss: 0.0100 - mean_squared_error: 0.0100
#> Epoch 356/500
#> 1/1 - 0s - 20ms/step - loss: 0.0099 - mean_squared_error: 0.0099
#> Epoch 357/500
#> 1/1 - 0s - 20ms/step - loss: 0.0099 - mean_squared_error: 0.0099
#> Epoch 358/500
#> 1/1 - 0s - 20ms/step - loss: 0.0098 - mean_squared_error: 0.0098
#> Epoch 359/500
#> 1/1 - 0s - 20ms/step - loss: 0.0098 - mean_squared_error: 0.0098
#> Epoch 360/500
#> 1/1 - 0s - 19ms/step - loss: 0.0097 - mean_squared_error: 0.0097
#> Epoch 361/500
#> 1/1 - 0s - 19ms/step - loss: 0.0097 - mean_squared_error: 0.0097
#> Epoch 362/500
#> 1/1 - 0s - 19ms/step - loss: 0.0096 - mean_squared_error: 0.0096
#> Epoch 363/500
#> 1/1 - 0s - 18ms/step - loss: 0.0096 - mean_squared_error: 0.0096
#> Epoch 364/500
#> 1/1 - 0s - 19ms/step - loss: 0.0095 - mean_squared_error: 0.0095
#> Epoch 365/500
#> 1/1 - 0s - 18ms/step - loss: 0.0095 - mean_squared_error: 0.0095
#> Epoch 366/500
#> 1/1 - 0s - 19ms/step - loss: 0.0094 - mean_squared_error: 0.0094
#> Epoch 367/500
#> 1/1 - 0s - 18ms/step - loss: 0.0094 - mean_squared_error: 0.0094
#> Epoch 368/500
#> 1/1 - 0s - 18ms/step - loss: 0.0093 - mean_squared_error: 0.0093
#> Epoch 369/500
#> 1/1 - 0s - 20ms/step - loss: 0.0093 - mean_squared_error: 0.0093
#> Epoch 370/500
#> 1/1 - 0s - 19ms/step - loss: 0.0093 - mean_squared_error: 0.0093
#> Epoch 371/500
#> 1/1 - 0s - 21ms/step - loss: 0.0092 - mean_squared_error: 0.0092
#> Epoch 372/500
#> 1/1 - 0s - 18ms/step - loss: 0.0092 - mean_squared_error: 0.0092
#> Epoch 373/500
#> 1/1 - 0s - 19ms/step - loss: 0.0092 - mean_squared_error: 0.0092
#> Epoch 374/500
#> 1/1 - 0s - 18ms/step - loss: 0.0091 - mean_squared_error: 0.0091
#> Epoch 375/500
#> 1/1 - 0s - 19ms/step - loss: 0.0091 - mean_squared_error: 0.0091
#> Epoch 376/500
#> 1/1 - 0s - 20ms/step - loss: 0.0090 - mean_squared_error: 0.0090
#> Epoch 377/500
#> 1/1 - 0s - 19ms/step - loss: 0.0090 - mean_squared_error: 0.0090
#> Epoch 378/500
#> 1/1 - 0s - 20ms/step - loss: 0.0089 - mean_squared_error: 0.0089
#> Epoch 379/500
#> 1/1 - 0s - 20ms/step - loss: 0.0089 - mean_squared_error: 0.0089
#> Epoch 380/500
#> 1/1 - 0s - 19ms/step - loss: 0.0089 - mean_squared_error: 0.0089
#> Epoch 381/500
#> 1/1 - 0s - 18ms/step - loss: 0.0088 - mean_squared_error: 0.0088
#> Epoch 382/500
#> 1/1 - 0s - 18ms/step - loss: 0.0088 - mean_squared_error: 0.0088
#> Epoch 383/500
#> 1/1 - 0s - 18ms/step - loss: 0.0088 - mean_squared_error: 0.0088
#> Epoch 384/500
#> 1/1 - 0s - 19ms/step - loss: 0.0087 - mean_squared_error: 0.0087
#> Epoch 385/500
#> 1/1 - 0s - 19ms/step - loss: 0.0087 - mean_squared_error: 0.0087
#> Epoch 386/500
#> 1/1 - 0s - 21ms/step - loss: 0.0086 - mean_squared_error: 0.0086
#> Epoch 387/500
#> 1/1 - 0s - 20ms/step - loss: 0.0086 - mean_squared_error: 0.0086
#> Epoch 388/500
#> 1/1 - 0s - 20ms/step - loss: 0.0086 - mean_squared_error: 0.0086
#> Epoch 389/500
#> 1/1 - 0s - 19ms/step - loss: 0.0086 - mean_squared_error: 0.0086
#> Epoch 390/500
#> 1/1 - 0s - 20ms/step - loss: 0.0085 - mean_squared_error: 0.0085
#> Epoch 391/500
#> 1/1 - 0s - 20ms/step - loss: 0.0085 - mean_squared_error: 0.0085
#> Epoch 392/500
#> 1/1 - 0s - 19ms/step - loss: 0.0084 - mean_squared_error: 0.0084
#> Epoch 393/500
#> 1/1 - 0s - 19ms/step - loss: 0.0084 - mean_squared_error: 0.0084
#> Epoch 394/500
#> 1/1 - 0s - 18ms/step - loss: 0.0084 - mean_squared_error: 0.0084
#> Epoch 395/500
#> 1/1 - 0s - 21ms/step - loss: 0.0083 - mean_squared_error: 0.0083
#> Epoch 396/500
#> 1/1 - 0s - 18ms/step - loss: 0.0083 - mean_squared_error: 0.0083
#> Epoch 397/500
#> 1/1 - 0s - 18ms/step - loss: 0.0083 - mean_squared_error: 0.0083
#> Epoch 398/500
#> 1/1 - 0s - 20ms/step - loss: 0.0082 - mean_squared_error: 0.0082
#> Epoch 399/500
#> 1/1 - 0s - 20ms/step - loss: 0.0082 - mean_squared_error: 0.0082
#> Epoch 400/500
#> 1/1 - 0s - 19ms/step - loss: 0.0082 - mean_squared_error: 0.0082
#> Epoch 401/500
#> 1/1 - 0s - 18ms/step - loss: 0.0082 - mean_squared_error: 0.0082
#> Epoch 402/500
#> 1/1 - 0s - 19ms/step - loss: 0.0081 - mean_squared_error: 0.0081
#> Epoch 403/500
#> 1/1 - 0s - 21ms/step - loss: 0.0081 - mean_squared_error: 0.0081
#> Epoch 404/500
#> 1/1 - 0s - 19ms/step - loss: 0.0080 - mean_squared_error: 0.0080
#> Epoch 405/500
#> 1/1 - 0s - 18ms/step - loss: 0.0080 - mean_squared_error: 0.0080
#> Epoch 406/500
#> 1/1 - 0s - 19ms/step - loss: 0.0080 - mean_squared_error: 0.0080
#> Epoch 407/500
#> 1/1 - 0s - 21ms/step - loss: 0.0080 - mean_squared_error: 0.0080
#> Epoch 408/500
#> 1/1 - 0s - 20ms/step - loss: 0.0079 - mean_squared_error: 0.0079
#> Epoch 409/500
#> 1/1 - 0s - 19ms/step - loss: 0.0079 - mean_squared_error: 0.0079
#> Epoch 410/500
#> 1/1 - 0s - 19ms/step - loss: 0.0079 - mean_squared_error: 0.0079
#> Epoch 411/500
#> 1/1 - 0s - 19ms/step - loss: 0.0079 - mean_squared_error: 0.0079
#> Epoch 412/500
#> 1/1 - 0s - 19ms/step - loss: 0.0078 - mean_squared_error: 0.0078
#> Epoch 413/500
#> 1/1 - 0s - 19ms/step - loss: 0.0078 - mean_squared_error: 0.0078
#> Epoch 414/500
#> 1/1 - 0s - 19ms/step - loss: 0.0078 - mean_squared_error: 0.0078
#> Epoch 415/500
#> 1/1 - 0s - 20ms/step - loss: 0.0077 - mean_squared_error: 0.0077
#> Epoch 416/500
#> 1/1 - 0s - 18ms/step - loss: 0.0077 - mean_squared_error: 0.0077
#> Epoch 417/500
#> 1/1 - 0s - 19ms/step - loss: 0.0077 - mean_squared_error: 0.0077
#> Epoch 418/500
#> 1/1 - 0s - 18ms/step - loss: 0.0076 - mean_squared_error: 0.0076
#> Epoch 419/500
#> 1/1 - 0s - 19ms/step - loss: 0.0076 - mean_squared_error: 0.0076
#> Epoch 420/500
#> 1/1 - 0s - 18ms/step - loss: 0.0076 - mean_squared_error: 0.0076
#> Epoch 421/500
#> 1/1 - 0s - 19ms/step - loss: 0.0076 - mean_squared_error: 0.0076
#> Epoch 422/500
#> 1/1 - 0s - 19ms/step - loss: 0.0075 - mean_squared_error: 0.0075
#> Epoch 423/500
#> 1/1 - 0s - 18ms/step - loss: 0.0075 - mean_squared_error: 0.0075
#> Epoch 424/500
#> 1/1 - 0s - 18ms/step - loss: 0.0075 - mean_squared_error: 0.0075
#> Epoch 425/500
#> 1/1 - 0s - 20ms/step - loss: 0.0075 - mean_squared_error: 0.0075
#> Epoch 426/500
#> 1/1 - 0s - 20ms/step - loss: 0.0074 - mean_squared_error: 0.0074
#> Epoch 427/500
#> 1/1 - 0s - 19ms/step - loss: 0.0074 - mean_squared_error: 0.0074
#> Epoch 428/500
#> 1/1 - 0s - 18ms/step - loss: 0.0074 - mean_squared_error: 0.0074
#> Epoch 429/500
#> 1/1 - 0s - 18ms/step - loss: 0.0074 - mean_squared_error: 0.0074
#> Epoch 430/500
#> 1/1 - 0s - 19ms/step - loss: 0.0073 - mean_squared_error: 0.0073
#> Epoch 431/500
#> 1/1 - 0s - 21ms/step - loss: 0.0073 - mean_squared_error: 0.0073
#> Epoch 432/500
#> 1/1 - 0s - 21ms/step - loss: 0.0073 - mean_squared_error: 0.0073
#> Epoch 433/500
#> 1/1 - 0s - 18ms/step - loss: 0.0073 - mean_squared_error: 0.0073
#> Epoch 434/500
#> 1/1 - 0s - 18ms/step - loss: 0.0072 - mean_squared_error: 0.0072
#> Epoch 435/500
#> 1/1 - 0s - 19ms/step - loss: 0.0072 - mean_squared_error: 0.0072
#> Epoch 436/500
#> 1/1 - 0s - 20ms/step - loss: 0.0072 - mean_squared_error: 0.0072
#> Epoch 437/500
#> 1/1 - 0s - 19ms/step - loss: 0.0072 - mean_squared_error: 0.0072
#> Epoch 438/500
#> 1/1 - 0s - 19ms/step - loss: 0.0071 - mean_squared_error: 0.0071
#> Epoch 439/500
#> 1/1 - 0s - 19ms/step - loss: 0.0071 - mean_squared_error: 0.0071
#> Epoch 440/500
#> 1/1 - 0s - 20ms/step - loss: 0.0071 - mean_squared_error: 0.0071
#> Epoch 441/500
#> 1/1 - 0s - 21ms/step - loss: 0.0071 - mean_squared_error: 0.0071
#> Epoch 442/500
#> 1/1 - 0s - 20ms/step - loss: 0.0070 - mean_squared_error: 0.0070
#> Epoch 443/500
#> 1/1 - 0s - 19ms/step - loss: 0.0070 - mean_squared_error: 0.0070
#> Epoch 444/500
#> 1/1 - 0s - 19ms/step - loss: 0.0070 - mean_squared_error: 0.0070
#> Epoch 445/500
#> 1/1 - 0s - 19ms/step - loss: 0.0070 - mean_squared_error: 0.0070
#> Epoch 446/500
#> 1/1 - 0s - 20ms/step - loss: 0.0070 - mean_squared_error: 0.0070
#> Epoch 447/500
#> 1/1 - 0s - 19ms/step - loss: 0.0070 - mean_squared_error: 0.0070
#> Epoch 448/500
#> 1/1 - 0s - 18ms/step - loss: 0.0069 - mean_squared_error: 0.0069
#> Epoch 449/500
#> 1/1 - 0s - 18ms/step - loss: 0.0069 - mean_squared_error: 0.0069
#> Epoch 450/500
#> 1/1 - 0s - 19ms/step - loss: 0.0069 - mean_squared_error: 0.0069
#> Epoch 451/500
#> 1/1 - 0s - 19ms/step - loss: 0.0069 - mean_squared_error: 0.0069
#> Epoch 452/500
#> 1/1 - 0s - 20ms/step - loss: 0.0068 - mean_squared_error: 0.0068
#> Epoch 453/500
#> 1/1 - 0s - 19ms/step - loss: 0.0068 - mean_squared_error: 0.0068
#> Epoch 454/500
#> 1/1 - 0s - 19ms/step - loss: 0.0068 - mean_squared_error: 0.0068
#> Epoch 455/500
#> 1/1 - 0s - 19ms/step - loss: 0.0068 - mean_squared_error: 0.0068
#> Epoch 456/500
#> 1/1 - 0s - 18ms/step - loss: 0.0067 - mean_squared_error: 0.0067
#> Epoch 457/500
#> 1/1 - 0s - 19ms/step - loss: 0.0067 - mean_squared_error: 0.0067
#> Epoch 458/500
#> 1/1 - 0s - 20ms/step - loss: 0.0067 - mean_squared_error: 0.0067
#> Epoch 459/500
#> 1/1 - 0s - 20ms/step - loss: 0.0067 - mean_squared_error: 0.0067
#> Epoch 460/500
#> 1/1 - 0s - 19ms/step - loss: 0.0067 - mean_squared_error: 0.0067
#> Epoch 461/500
#> 1/1 - 0s - 18ms/step - loss: 0.0067 - mean_squared_error: 0.0067
#> Epoch 462/500
#> 1/1 - 0s - 21ms/step - loss: 0.0066 - mean_squared_error: 0.0066
#> Epoch 463/500
#> 1/1 - 0s - 20ms/step - loss: 0.0066 - mean_squared_error: 0.0066
#> Epoch 464/500
#> 1/1 - 0s - 20ms/step - loss: 0.0066 - mean_squared_error: 0.0066
#> Epoch 465/500
#> 1/1 - 0s - 18ms/step - loss: 0.0066 - mean_squared_error: 0.0066
#> Epoch 466/500
#> 1/1 - 0s - 18ms/step - loss: 0.0065 - mean_squared_error: 0.0065
#> Epoch 467/500
#> 1/1 - 0s - 18ms/step - loss: 0.0065 - mean_squared_error: 0.0065
#> Epoch 468/500
#> 1/1 - 0s - 19ms/step - loss: 0.0065 - mean_squared_error: 0.0065
#> Epoch 469/500
#> 1/1 - 0s - 19ms/step - loss: 0.0065 - mean_squared_error: 0.0065
#> Epoch 470/500
#> 1/1 - 0s - 22ms/step - loss: 0.0065 - mean_squared_error: 0.0065
#> Epoch 471/500
#> 1/1 - 0s - 18ms/step - loss: 0.0065 - mean_squared_error: 0.0065
#> Epoch 472/500
#> 1/1 - 0s - 18ms/step - loss: 0.0064 - mean_squared_error: 0.0064
#> Epoch 473/500
#> 1/1 - 0s - 22ms/step - loss: 0.0064 - mean_squared_error: 0.0064
#> Epoch 474/500
#> 1/1 - 0s - 22ms/step - loss: 0.0064 - mean_squared_error: 0.0064
#> Epoch 475/500
#> 1/1 - 0s - 19ms/step - loss: 0.0064 - mean_squared_error: 0.0064
#> Epoch 476/500
#> 1/1 - 0s - 18ms/step - loss: 0.0064 - mean_squared_error: 0.0064
#> Epoch 477/500
#> 1/1 - 0s - 20ms/step - loss: 0.0064 - mean_squared_error: 0.0064
#> Epoch 478/500
#> 1/1 - 0s - 21ms/step - loss: 0.0063 - mean_squared_error: 0.0063
#> Epoch 479/500
#> 1/1 - 0s - 19ms/step - loss: 0.0063 - mean_squared_error: 0.0063
#> Epoch 480/500
#> 1/1 - 0s - 18ms/step - loss: 0.0063 - mean_squared_error: 0.0063
#> Epoch 481/500
#> 1/1 - 0s - 19ms/step - loss: 0.0063 - mean_squared_error: 0.0063
#> Epoch 482/500
#> 1/1 - 0s - 18ms/step - loss: 0.0063 - mean_squared_error: 0.0063
#> Epoch 483/500
#> 1/1 - 0s - 18ms/step - loss: 0.0063 - mean_squared_error: 0.0063
#> Epoch 484/500
#> 1/1 - 0s - 20ms/step - loss: 0.0062 - mean_squared_error: 0.0062
#> Epoch 485/500
#> 1/1 - 0s - 19ms/step - loss: 0.0062 - mean_squared_error: 0.0062
#> Epoch 486/500
#> 1/1 - 0s - 18ms/step - loss: 0.0062 - mean_squared_error: 0.0062
#> Epoch 487/500
#> 1/1 - 0s - 18ms/step - loss: 0.0062 - mean_squared_error: 0.0062
#> Epoch 488/500
#> 1/1 - 0s - 18ms/step - loss: 0.0062 - mean_squared_error: 0.0062
#> Epoch 489/500
#> 1/1 - 0s - 19ms/step - loss: 0.0062 - mean_squared_error: 0.0062
#> Epoch 490/500
#> 1/1 - 0s - 18ms/step - loss: 0.0061 - mean_squared_error: 0.0061
#> Epoch 491/500
#> 1/1 - 0s - 21ms/step - loss: 0.0061 - mean_squared_error: 0.0061
#> Epoch 492/500
#> 1/1 - 0s - 19ms/step - loss: 0.0061 - mean_squared_error: 0.0061
#> Epoch 493/500
#> 1/1 - 0s - 19ms/step - loss: 0.0061 - mean_squared_error: 0.0061
#> Epoch 494/500
#> 1/1 - 0s - 20ms/step - loss: 0.0061 - mean_squared_error: 0.0061
#> Epoch 495/500
#> 1/1 - 0s - 20ms/step - loss: 0.0061 - mean_squared_error: 0.0061
#> Epoch 496/500
#> 1/1 - 0s - 21ms/step - loss: 0.0060 - mean_squared_error: 0.0060
#> Epoch 497/500
#> 1/1 - 0s - 19ms/step - loss: 0.0060 - mean_squared_error: 0.0060
#> Epoch 498/500
#> 1/1 - 0s - 19ms/step - loss: 0.0060 - mean_squared_error: 0.0060
#> Epoch 499/500
#> 1/1 - 0s - 21ms/step - loss: 0.0060 - mean_squared_error: 0.0060
#> Epoch 500/500
#> 1/1 - 0s - 20ms/step - loss: 0.0060 - mean_squared_error: 0.0060
# ...plot it out, along with model predictions
plot(history)
plot(predict(model, x_test)[,1] ~ y_test)
#> 1/1 - 0s - 55ms/step
I don’t want you to focus too much on the problem of how the data were simulated yet, other than to note that we had to scale that data because that will be relevant for the next section. For now, focus on the dimensionality of the data: we have a time series of length 101 (y), which we cut into training/validation subsets in runs of 5 (making 10 such runs in the training and subset data). This allows us to give our data to TensorFlow in a very specific form: an array with the different training runs (10 of them, each of length 5) grouped up. Take a look at dim(x_training) and dim(y_training) until you’re 100% clear on what is going on: our response variable is a 1-dimensional variable of length 10, and our training data is a 3D array with 10 runs, each of length 5. Note the use of 1, 5, and 10 in the neural network structure itself: we have to be crystal clear with TensorFlow what we’re giving it, because otherwise it doesn’t know how to recurse from the explanatory input variables onto the response variables. Once the data is in the right format, the actual recursion itself is trivial—it’s ‘just another layer’ and we specify the dimension of the recursion itself.
10.3.3 Exploding/vanishing gradients (and recurrency)
That’s the end of the session, right? We’re done! Sadly, no. Because recurrent networks involve so much propagation of signal throughout their network, they tend to suffer very badly from the exploding or vanishing gradient problem. You’ve probably not thought about it very much before, but if you keep multiplying numbers by themselves they tend to get very large or very small very quickly (compare \(10^{10}\) with \(0.1^{10}\)). Neural networks with many layers (i.e., deep networks) are basically just lots of multiplications all together, and it means that signals can get very large as they propagate through that network. Things like bias neurons can help with this to some extent 15, but in the face of very deep networks—or recurrent networks where signals can propagate across time-series—they cannot keep up. This creates a real problem because the signals passing through networks become so large that we can’t effectively train them anymore. Because the multiplication of the weights makes everything very large or very small, it becomes effectively impossibly to see the impact that quantitative changes in the weights have on overall model performance 16. Thus we can’t fit the network anymore, and out predictions are uniformly awful.
The band-aid solution to this is to scale your input data, as I did in the last code example, but that can only do so much. Another option, which we will not explore here in detail but is worth knowing about, is to allow inputs to enter at different points within the network. So-called skip layers allow for the outputs from particular nodes to go around other layers in the network. Such structures are actually reasonably common in mammalian brains, where external stimuli (e.g., retinal cells) are allowed to directly innervate neurons deep within our cortex (often in several places). This perhaps somewhat alleviates these drop-out problems by re-introducing the original signal to ensure that it doesn’t get lost in all the noise. Of course it’s not entirely clear that back-propagating artificial neural network problems back onto real neural networks is a valid approach. I mention all of this simply to remind you that the kind of linear, sequential network structures we are exploring in this course are not the only kinds of networks that can be fit.
A popular solution are so-called Long Short-Term Memory layers (LSTM), which are a much better idea than their totally nonsensical name implies. Their goal is to obviate this problem by dampening down the propagation of information throughout the network: creating a form of short-term memory that can be propagated back into the network, modified through time, and of course completely reset. I would advise you to ignore any diagrams you find that attempt to explain what they ‘look like’ because they are really more akin to memory registers in a computer and thus don’t really ‘look’ like conventional nodes in a network. Just as with circuit-boards, they have ‘gates’: input, forget, output, and new cell state gates. Think of these gates as ways of accessing the value (the memory) of the LSTM node itself: depending on how strongly they are activated these gates ‘open’ and perform an operation on the LSTM node itself. If these gates are sufficiently innervated (they received input from other nodes), they perform their particular function: the input gate will add value to the node’s value, the forget gate will wipe the value (memory) clear, and the output gate will cause the node to propagate its value on to its connected nodes (just as if it were a regular node in a regular layer). The ‘new cell state’ gate is somewhat special and gives the node its initial value. In some cases the node’s cell state (its value, the memory itself) is something that you can inspect and derive meaning from. A trivial example is that in networks associated with language production such a node may contain an encoded value that is the subject of a sentence (‘1’ for ‘Will’ in the example I gave earlier in this section). There are simplified versions of the LSTM gate (‘GRU’ gates) but, frankly, they’re not much more simplified so I think you might as well wrestle with the LSTM gate and be done with it.
The good news is that fitting an LSTM layer is very straightforward in TensorFlow. I leave it as an exercise to you, dear reader, to replace layer_simple_rnn with layer_lstm and see what happens. Depending on the parameters you pass to the LSTM layer you may be able to obviate the need for rescaling of your input variables (why not take this opportunity to look at the fantastic documentation the Keras API provides; https://keras.io/api/layers/recurrent_layers/lstm/).
10.4 Autoencoders and the ‘secret’ of deep learning
I thought it would be nice to finish on what, perhaps, we should have started with: why deep neural networks are so powerful. Neural networks are so flexible that, for some architectures, they can be shown mathematically to be identical to classical methods, and the architecture I’m about to show you is mathematically identical to something called a Principal Components Analysis (PCA)17. Autoencoding networks are notable because they use neural networks to build reduced-dimensionality representations of their input data, and as such their hidden layers encode a representation of the input data but using fewer dimensions. They always have the same dimensionality going in as going out, because they are trained to replicate their input data. Below is an example of how to build one, and also how to extract that reduced-dimensionality representation.
# Simulate data and verify they have one major axis
pc1 <- rnorm(100)
xs <- replicate(10, pc1+rnorm(100, sd=.2))
biplot(prcomp(xs, scale=TRUE))
# Build the encoder from scratch
autoencoder <- keras_model_sequential(input_shape = 10) %>%
layer_dense(units = 5, activation = "relu") %>%
layer_dense(units = 1) %>%
layer_dense(units = 5, activation = "relu") %>%
layer_dense(units = 10)
# Train the model
autoencoder %>% compile(
loss = "mean_squared_error",
optimizer = optimizer_rmsprop()
)
autoencoder %>% fit(xs, xs, epochs = 100)
#> Epoch 1/100
#> 4/4 - 0s - 57ms/step - loss: 1.2989
#> Epoch 2/100
#> 4/4 - 0s - 5ms/step - loss: 1.2713
#> Epoch 3/100
#> 4/4 - 0s - 5ms/step - loss: 1.2527
#> Epoch 4/100
#> 4/4 - 0s - 6ms/step - loss: 1.2369
#> Epoch 5/100
#> 4/4 - 0s - 5ms/step - loss: 1.2227
#> Epoch 6/100
#> 4/4 - 0s - 5ms/step - loss: 1.2068
#> Epoch 7/100
#> 4/4 - 0s - 5ms/step - loss: 1.1878
#> Epoch 8/100
#> 4/4 - 0s - 5ms/step - loss: 1.1724
#> Epoch 9/100
#> 4/4 - 0s - 5ms/step - loss: 1.1552
#> Epoch 10/100
#> 4/4 - 0s - 5ms/step - loss: 1.1389
#> Epoch 11/100
#> 4/4 - 0s - 5ms/step - loss: 1.1168
#> Epoch 12/100
#> 4/4 - 0s - 5ms/step - loss: 1.0963
#> Epoch 13/100
#> 4/4 - 0s - 5ms/step - loss: 1.0768
#> Epoch 14/100
#> 4/4 - 0s - 5ms/step - loss: 1.0545
#> Epoch 15/100
#> 4/4 - 0s - 5ms/step - loss: 1.0368
#> Epoch 16/100
#> 4/4 - 0s - 5ms/step - loss: 1.0171
#> Epoch 17/100
#> 4/4 - 0s - 5ms/step - loss: 0.9967
#> Epoch 18/100
#> 4/4 - 0s - 5ms/step - loss: 0.9785
#> Epoch 19/100
#> 4/4 - 0s - 6ms/step - loss: 0.9589
#> Epoch 20/100
#> 4/4 - 0s - 5ms/step - loss: 0.9384
#> Epoch 21/100
#> 4/4 - 0s - 5ms/step - loss: 0.9184
#> Epoch 22/100
#> 4/4 - 0s - 5ms/step - loss: 0.8955
#> Epoch 23/100
#> 4/4 - 0s - 5ms/step - loss: 0.8745
#> Epoch 24/100
#> 4/4 - 0s - 5ms/step - loss: 0.8552
#> Epoch 25/100
#> 4/4 - 0s - 5ms/step - loss: 0.8334
#> Epoch 26/100
#> 4/4 - 0s - 5ms/step - loss: 0.8149
#> Epoch 27/100
#> 4/4 - 0s - 5ms/step - loss: 0.7931
#> Epoch 28/100
#> 4/4 - 0s - 5ms/step - loss: 0.7747
#> Epoch 29/100
#> 4/4 - 0s - 5ms/step - loss: 0.7557
#> Epoch 30/100
#> 4/4 - 0s - 5ms/step - loss: 0.7369
#> Epoch 31/100
#> 4/4 - 0s - 5ms/step - loss: 0.7169
#> Epoch 32/100
#> 4/4 - 0s - 5ms/step - loss: 0.6976
#> Epoch 33/100
#> 4/4 - 0s - 5ms/step - loss: 0.6825
#> Epoch 34/100
#> 4/4 - 0s - 5ms/step - loss: 0.6659
#> Epoch 35/100
#> 4/4 - 0s - 5ms/step - loss: 0.6493
#> Epoch 36/100
#> 4/4 - 0s - 5ms/step - loss: 0.6294
#> Epoch 37/100
#> 4/4 - 0s - 5ms/step - loss: 0.6104
#> Epoch 38/100
#> 4/4 - 0s - 5ms/step - loss: 0.5979
#> Epoch 39/100
#> 4/4 - 0s - 6ms/step - loss: 0.5821
#> Epoch 40/100
#> 4/4 - 0s - 5ms/step - loss: 0.5679
#> Epoch 41/100
#> 4/4 - 0s - 6ms/step - loss: 0.5540
#> Epoch 42/100
#> 4/4 - 0s - 5ms/step - loss: 0.5401
#> Epoch 43/100
#> 4/4 - 0s - 5ms/step - loss: 0.5263
#> Epoch 44/100
#> 4/4 - 0s - 5ms/step - loss: 0.5134
#> Epoch 45/100
#> 4/4 - 0s - 5ms/step - loss: 0.5004
#> Epoch 46/100
#> 4/4 - 0s - 5ms/step - loss: 0.4891
#> Epoch 47/100
#> 4/4 - 0s - 5ms/step - loss: 0.4786
#> Epoch 48/100
#> 4/4 - 0s - 5ms/step - loss: 0.4656
#> Epoch 49/100
#> 4/4 - 0s - 6ms/step - loss: 0.4538
#> Epoch 50/100
#> 4/4 - 0s - 5ms/step - loss: 0.4428
#> Epoch 51/100
#> 4/4 - 0s - 5ms/step - loss: 0.4320
#> Epoch 52/100
#> 4/4 - 0s - 5ms/step - loss: 0.4200
#> Epoch 53/100
#> 4/4 - 0s - 5ms/step - loss: 0.4084
#> Epoch 54/100
#> 4/4 - 0s - 5ms/step - loss: 0.3996
#> Epoch 55/100
#> 4/4 - 0s - 5ms/step - loss: 0.3882
#> Epoch 56/100
#> 4/4 - 0s - 5ms/step - loss: 0.3768
#> Epoch 57/100
#> 4/4 - 0s - 5ms/step - loss: 0.3681
#> Epoch 58/100
#> 4/4 - 0s - 5ms/step - loss: 0.3591
#> Epoch 59/100
#> 4/4 - 0s - 5ms/step - loss: 0.3509
#> Epoch 60/100
#> 4/4 - 0s - 5ms/step - loss: 0.3420
#> Epoch 61/100
#> 4/4 - 0s - 5ms/step - loss: 0.3331
#> Epoch 62/100
#> 4/4 - 0s - 5ms/step - loss: 0.3256
#> Epoch 63/100
#> 4/4 - 0s - 5ms/step - loss: 0.3175
#> Epoch 64/100
#> 4/4 - 0s - 5ms/step - loss: 0.3107
#> Epoch 65/100
#> 4/4 - 0s - 6ms/step - loss: 0.3014
#> Epoch 66/100
#> 4/4 - 0s - 5ms/step - loss: 0.2942
#> Epoch 67/100
#> 4/4 - 0s - 5ms/step - loss: 0.2885
#> Epoch 68/100
#> 4/4 - 0s - 6ms/step - loss: 0.2822
#> Epoch 69/100
#> 4/4 - 0s - 5ms/step - loss: 0.2746
#> Epoch 70/100
#> 4/4 - 0s - 5ms/step - loss: 0.2683
#> Epoch 71/100
#> 4/4 - 0s - 5ms/step - loss: 0.2609
#> Epoch 72/100
#> 4/4 - 0s - 5ms/step - loss: 0.2546
#> Epoch 73/100
#> 4/4 - 0s - 6ms/step - loss: 0.2486
#> Epoch 74/100
#> 4/4 - 0s - 5ms/step - loss: 0.2421
#> Epoch 75/100
#> 4/4 - 0s - 6ms/step - loss: 0.2357
#> Epoch 76/100
#> 4/4 - 0s - 5ms/step - loss: 0.2315
#> Epoch 77/100
#> 4/4 - 0s - 5ms/step - loss: 0.2265
#> Epoch 78/100
#> 4/4 - 0s - 5ms/step - loss: 0.2206
#> Epoch 79/100
#> 4/4 - 0s - 5ms/step - loss: 0.2170
#> Epoch 80/100
#> 4/4 - 0s - 5ms/step - loss: 0.2128
#> Epoch 81/100
#> 4/4 - 0s - 5ms/step - loss: 0.2089
#> Epoch 82/100
#> 4/4 - 0s - 6ms/step - loss: 0.2051
#> Epoch 83/100
#> 4/4 - 0s - 5ms/step - loss: 0.2013
#> Epoch 84/100
#> 4/4 - 0s - 5ms/step - loss: 0.1964
#> Epoch 85/100
#> 4/4 - 0s - 5ms/step - loss: 0.1913
#> Epoch 86/100
#> 4/4 - 0s - 5ms/step - loss: 0.1874
#> Epoch 87/100
#> 4/4 - 0s - 5ms/step - loss: 0.1844
#> Epoch 88/100
#> 4/4 - 0s - 5ms/step - loss: 0.1817
#> Epoch 89/100
#> 4/4 - 0s - 5ms/step - loss: 0.1790
#> Epoch 90/100
#> 4/4 - 0s - 6ms/step - loss: 0.1758
#> Epoch 91/100
#> 4/4 - 0s - 5ms/step - loss: 0.1722
#> Epoch 92/100
#> 4/4 - 0s - 5ms/step - loss: 0.1682
#> Epoch 93/100
#> 4/4 - 0s - 5ms/step - loss: 0.1652
#> Epoch 94/100
#> 4/4 - 0s - 5ms/step - loss: 0.1615
#> Epoch 95/100
#> 4/4 - 0s - 5ms/step - loss: 0.1581
#> Epoch 96/100
#> 4/4 - 0s - 5ms/step - loss: 0.1558
#> Epoch 97/100
#> 4/4 - 0s - 5ms/step - loss: 0.1526
#> Epoch 98/100
#> 4/4 - 0s - 5ms/step - loss: 0.1507
#> Epoch 99/100
#> 4/4 - 0s - 5ms/step - loss: 0.1484
#> Epoch 100/100
#> 4/4 - 0s - 5ms/step - loss: 0.1458
# Now pull out the first layer and run the data through that
est.pc1 <- autoencoder$layers[[1]](xs)
# Now run the output from the above through the second layer
est.pc1 <- autoencoder$layers[[2]](est.pc1)
# That's the same as running the encoder! Get the data back into R format, not tensorflow format
est.pc1 <- as.numeric(est.pc1)
# Confirm that we've got something that looks like the major axis
plot(est.pc1 ~ pc1)
cor.test(est.pc1, pc1)
#>
#> Pearson's product-moment correlation
#>
#> data: est.pc1 and pc1
#> t = -93.332, df = 98, p-value < 2.2e-16
#> alternative hypothesis: true correlation is not equal to 0
#> 95 percent confidence interval:
#> -0.9962500 -0.9917062
#> sample estimates:
#> cor
#> -0.9944219As you can see, by fitting a network that tapers to a single layer, and where we are minimising the difference between the input and an identical output, we have encoded a reduced dimensionality representation of the underlying data (i.e., the underlying major axis of the data) inside the network itself.
What is perhaps the most surprising thing about this network is that it performs well by being simple. Although it is tempting, when first starting to get to grips with deep learning, to fit networks with many layers, each layer dense with nodes, the truth is that doing so misses the true underlying power of deep learning. Deep learning is powerful because successive layers of relatively few nodes force the network to simply encode the properties of your data. Multiple, successive layers are flexible because they represent multiple, successive applications of complex mathematical transformations on data. Yet having fewer nodes within each layer forces the network to compress signal and extract higher-level features of the data within the network. Equally, having too many layers leaves you open to gradient problems in training, making it difficult for the network to learn finer features of your data. Thus the expert user of neural networks knows that the best performance comes from having a handful of layers, each with relatively few nodes. And so, as with everything in statistics, the trick is using just enough complexity and never any more.
10.5 Transformers and attention
Overview
Transformers have (rightly) received a huge amount of attention: they are the ‘T’ in ‘ChatGPT’, and you would have to been living under a rock for the last few years to not think the arrival of ChatGPT was important. In this session, we’re going to cover what transformers are, and how you can make use of existing transformer-based models in your own work. This session is a bit different from the others: I’m not going to show you how to write your own transformer within Keras. The reason for that is simple: there’s no real need. By this point you are just as capable as I am of replacing a CNN layer with a transformer layer (that’s essentially all you need to do!), and there are fantastic instructions online that show you how to build more complex model structures using Transformers18. I am, however, going to introduce you to a few existing tools that have been built using transformers and have the capacity to change the way you do your science. I want to emphasise that the parts of the handout where I show you how to use existing tools are likely to become out-of-date quite quickly. If some code doesn’t work, please flag it to me: it’s quite likely that something has changed since I wrote this!
10.5.1 Pay attention!
It might not seem obvious at first, but CNNs and RNNs are kind of similar. Consider a CNN that, instead of being applied to a two-dimensional dataset (e.g., an image) was applied to a one-dimensional dataset (e.g., a time-series). Such a CNN would have a constant feature detector moving across the time-series, whereas in an RNN the information moves across the time-series while the model remains constant19. You can see this kind of one-dimensional architecture in Figure 10.2 (a) and (b).
An attention layer is like a combination of both an RNN and a CNN. When the model is run, it doesn’t consider each word20 one-at-a-time (like an RNN) or only the words nearby (like a CNN), but rather it considers the entire sentence at once. It returns a set of attention scores: how important each word in the sentence is to every other word. The clever bit of the attention layer is it can do all this very quickly (because it’s built are tensors and their flow; see previous chapters), across the entire sentence at once, and the model can be trained in parallel across all words in the sentence at once. That last part is perhaps the most important bit: it allows absolutely massive models that compare loads of different sentences and words to be trained at-scale. It can do this because the attention scores are all relevant to the same sentence, and also because of some fancy maths involving how you multiply large vectors of numbers together. Doing that, however, requires you to be quite clever about how you tell the attention layer where every word is in the sentence, which brings us to the next part of the puzzle…
10.5.1.1 Transformers: attention-seeking robots in disguise
As we’ve learned by now, machine learning researchers just love to give names to things, and when Google researchers figured out the precise way to make an attention layer to allow for large-language models they just couldn’t help themselves21. Attention layers already existed (albeit not in precisely the form I’ve described above), so they decided to give a name to the artificial neural network architecture they needed to sit within: the transformer. The transformer architecture looks absolutely horrific when printed out (look at Figure 10.3 if you like, which is the first transformer in the original Vaswani et al. paper) but the concepts within it aren’t as bad as they seem22. You already know what the attention layers are, so I won’t cover those other than to say that there are four of them. They’re called ‘multi-head attention’ layers in this diagram because attention layers already existed at that time and were somewhat different; now it’s more common to just say ‘attention layer’ because no one really uses the other kind anymore.
10.5.1.2 Encoding and decoding
Bet you wish you’d paid attention in the auto-encoding section now, huh? The ‘encoder’ half of the diagram (the left-section that’s marked ‘Nx’) takes the sentences and words and then transforms them so that the model can make sense of them. It’s ‘encoding’ them into a format that the model can make use of. The ‘decoder layer’ (the right-section that’s marked ‘Nx’) does the reverse, and gets everything ready so that it can output the model’s predictions in terms of the probabilities of what the next word(s) in the sentence will be.
10.5.1.3 Positional encoding
These bits are actually really important and, even though they seem really small, were one of the most clever bits of the whole original idea. They allow the model to know, regardless of how long the sentence/paragraph being input is, where each word is in the sentence (e.g., the first, second, penultimate, or last word) and allow the model to be trained in parallel across words. That last part is critical: instead of having to sequentially train each word in a single sentence or paragraph, everything can be calculated at the same time (because every word is being compared to every other word). Remember how GPUs are really good at doing small, simple calculations lots of times? This parallelisation allows the model to be trained at-scale on GPUs, and without this innovation you likely would still be asking stack-overflow for help solving your homework23. These are the twirly-sort-of-yin-yang symbols at the bottom of Figure 10.4.
How does it work? Instead of keeping track of which number word you’re looking at in the sentence, they run a series of sine and cosine waves with different frequencies along the sentence. If you’ve ever slide one of those transparent ‘super secret spy decoder’ things along text as a kid then you have played with these: the combination of waves mean that every position has a unique combination the waves, and you can do that without even needing to track how long the sentence is (or the predicted sentence could be). This is important: without knowing where in the sentence a word is, it wouldn’t be able to distinguish between the sentences “Will loves Transformers” and “Transformers love Will”.
10.5.1.4 Other bits
There are a few other layers in there too, but they’re pretty standard and you know about them by now.
Add & Norm. These just add up all the results from the previous attention layers so that the information is shared across the whole sentence with all its words.
Linear. This is just a regular ol’ flat, dense layer of nodes.
Softmax. We’re outputting what we hope are probabilities of words, so we use softmax to make sure those numbers scale between 0 and 1.
Feed Forward. These are just some linear, dense layers with a few dropout nodes chucked in as well. Nothing special about these either.
10.5.2 Transformers, roll out
Transformers are used in a lot of contexts. Below I go through a number of models, all of which make use of transformers in some way. I am not going to explain how each of these models work other than in the briefest of terms: above is an explanation of the first transformer, and everything that follows is a variation and improvement on that theme. By the time I explained each model, I would likely have to re-write that part of the handout anyway because the model would have updated (at the time of writing, I have done that three times: for SAM, YOLO, and DeepSeek). But there are some general principles to bear in mind when working with these models, which I outline below.
10.5.2.1 Install in a dedicated environment
Container. Some of these models come with containers (normally hosted through something called ‘docker’). Once you’ve got the container environment setup, it allows you to essentially run a micro-virtual-machine that runs nothing but the model or piece of code that you want. They’re incredibly useful because, as they’re entirely sandboxed and contain everything they need, once you’re ready to go you can run anything you want in exactly the way the developer intended. The downside is they’re slower than getting everything setup yourself (necessarily, because they contain a stripped-down operating system). Further, all this extra code can end up taking up loads of space on your hard-drive. So I, personally, prefer to use…
Virtual environments. Python people love to make virtual environments and, even though I try to avoid Python wherever possible, even I must admit they’re useful sometimes. Often different models will have conflicting version requirements (e.g., one needs TensorFlow 2.1 and the other 2.2) and it can be essentially impossible to have the two co-existing. A virtual environment can be setup, and then activated, like this from a command line:
# Setup environment
python -m venv .llama-env
# Activate the environment
source .llama-env/bin/activate…then you go ahead installing whatever dependencies you need for your model. When you’re done, close the terminal window, and when you need to run the model again, simply run that last line. Done with the model? Deleting the folder .llama-env will get rid of all the install dependencies (don’t forget the model too).
10.5.2.2 Large-language models need quantising
Ever wondered how those massive large-language models could possible run on your laptop computer? The answer, I’m afraid to say, is that sometimes they can’t. The number of parameters, and even the numerical precision of the calculations, can be so large that they can either be impossible to run on your computer or too slow to be practical. But fear not! There is a solution: quantising. Running this will essentially compress whatever model you have downloaded, allowing you to run a lower-quality of the model. Doing this will essentially introduce rounding errors into your model, and so it will be less accurate (it won’t spout garbage, but the results won’t be as good) but it will run. This process is quite tricky and requires a bit of technical fiddling: my advice is to find someone else’s quantisation and work with that, but otherwise prepare yourself for some fine-tuning!
10.5.3 Image and video models
Each of these models has slightly different features, as well as slightly different features with different speeds, memory sizes, and features. On paper, YOLO has the most features (and notably can identify and classify objects in images and videos), but in my personal experience SAM is the best at object segmentation (pulling out images from their backgrounds).
10.5.3.1 YOLO: You Only Look Once
YOLO (‘You Only Look Once’) was one of the first object-detection-within-image methods to make a sufficiently large splash that it was well-appreciated outside of computer scientists. The original algorithm was notable for passing the entire image through the same CNN architecture in a single pass (hence ‘you only look once’); it achieved this through some clever tricks with how it divided the image into sections and defined bounding-boxes of objects within it. If you saw an image with coloured squares around objects of interest in the mid-2010s, the chances are you were looking at the output from YOLO, and although that style of presentation is now quite common that approach was, at the time, quite innovative.
The model has subsequently gone through numerous iterations, and I’m not entirely sure that the latest version (at the time of writing, version 12) bears much relation to the original approach. My understanding is that it’s not necessarily even maintained by the same people, and that subsequent versions (e.g., 9 and then 10) aren’t developed by the same team. So it’s a bit of a free-for-all, but somewhere, someone seems to know what they’re doing because it’s still very popular software.
YOLO12 is the latest version (it came out 6 days ago) and, as a result, while it claims that it’s faster that YOLO11 I feel confident saying that YOLO11 is still very fast and is still vastly easier to use. YOLO11 has a command-line option that makes it trivial to use; set-up really is intended to be easy (…is it ever not? read https://github.com/ultralytics/ultralytics for details) so something like this:
# Install dependencies
pip install ultralytics
# Get the model
wget https://github.com/ultralytics/assets/releases/download/v8.3.0/yolo11n.pt
# Predict a file
yolo predict model=yolo11n.pt source='path/to/file.png'…the only thing I’d say is that YOLO11 is clearly being run by a company, which means that you need to license its use in certain cases (e.g., making money).
Speaking of vested interested, though, have I mentioned YOLO-Behaviour? It’s a fantastic add-on to YOLO8 that has been trained and tailored for use with animal data in ecological settings. You can read more about it online (https://github.com/alexhang212/YOLO_Behaviour_Repo) including in a fantastic paper in Methods in Ecology and Evolution (Chan et al. in press; DOI: 10.1111/2041-210X.14502). So really, if you were trying to do something to quickly measure animals, why would you use anything else? Conflict of interest? Me? No, not at all! Oh, I suppose the lead author of this was a student in my lab. But no, I don’t see a conflict at all!…
10.5.4 SAM: Segment Anything Model
This model is so good at object segmentation that I shuttered a line of research in my lab (…and opened up another). Developed by Meta (previously known as Facebook), there is no way that any model could be trained on more labelled images than this one. Making use of it comes at a price, therefore: the certainty that it’s been trained on your gran’s photos24, restrictions on the commercial use of the model, and the near-certainty that copyrighted materials have also been used on the sly to train it25.
Meta are never really going to tell you how SAM works, even if they let you use it. When used with single images, it’s essentially just a very large transformer with (given how Meta run other models, and the fact that they don’t talk about this in their paper), some sort of fancy footwork with activation functions. For the videos and the tracking, they claim to make use of a kind of ‘encoder memory’ that allows the model access to where the tracked object was in the previous frame. Notably, they talk about how they can propagate through the coordinates of the tracked object in the paper, but they don’t give much in the way of detail, so presumably they do something fancy there. Are you picking up on the fact that I really, really like using these models but I really, really don’t trust the descriptions of them yet?
Getting SAM1 to work over the command-line is relatively straightforward, and Meta released a tool for it. There isn’t the same level of support for SAM2, but it’s relatively straightforward to get it working in a Python script. I would, therefore, recommend making use of SAM if you’re thinking of doing some kind of image labelling and extraction work. You could, for example: (1) Get a folder of images, (2) run the images through SAM2 and then ‘prompt’ it by telling it what kinds of things you want from images (e.g., click on fifteen pictures of meerkats in camera traps), (3) run it through thousands of camera traps once it knows what it’s looking for and then (4) run those images through some kind of downstream classifier that you build yourself. Steps 1–3 are all on the GitHub README page (https://github.com/facebookresearch/sam2), and step 4 is in this handout in the CNN section. You could, of course, do much the same with YOLO (and perhaps YOLO would even guess which images were the meerkats). Oh, and did I mention YOLO-Behaviour?…
10.5.5 Sound data
I’m unaware of anything as good as YOLO or SAM for sound data. That’s not to say there aren’t world-leading models for specific use-cases: I believe I may have written the world’s first AI birdsong classifier 26 but I stopped working on that the moment I realised the Cornell Lab for Ornithology were doing something similar. Their BirdNET model (https://github.com/kahst/BirdNET-Analyzer) is unbeaten. Equally, nearly-off-the-shelf products exist for bats (https://github.com/macaodha/batdetect2) and you would be a fool not to explore those.
That said, if you are working with sound data, my advice would be to make use of the fantastic products that already exist for images. Making judicious use of Cover’s Theorem, you could take existing neural nets trained for images, sandwich them between input layers that feed in spectrograms of calls, and voilà, c’est une pipe. R has fantastic spectrogram-producing packages (check out seewave), xeno-canto has fantastic download APIs for bird calls27, and you could either make use of the models above or make use of built-in keras routines to load image architectures such as VGG (there is even a VGG function in Keras; at the time of reading there are likely even more).
10.5.6 Language models
Much of what you can do in ChatGPT you can likely do on your laptop too. The advantage of doing it on your laptop is that Sam Altman won’t be reading over your should and then sharing it with everyone; notably, you can request that OpenAI don’t make use of your requests but you can’t demand it. While that might seem an academic issue in the grand scheme of things, as an academic it’s actually quite the issue because it’s not uncommon to find your own words parroted back to you by ChatGPT.
The trick to working with local models is to make use of a standardised framework so that you can easily hot-swap models depending on your needs. HuggingFace has emerged as the de facto place to share and contribute to model development, and lots of frameworks exist that can communicate in useful ways with HuggingFace. It’s kind of like GitHub for models, if GitHub were set up to store and serve gigabytes of models.
I only really dabble in language models, but when I do I use llama.cpp. It used to support images and multi-modal models, which was convenient for making things for my daughter, and while the developer has dropped support for this feature recently the command-line and in-browser support is really quite impressive. What is confusing when setting up llama.cpp is that it’s named for the Meta model LLaMa28 and this can make it a bit confusing to know whether you’ve installed LLaMa, llama.cpp, or both the same time you use it. To install llama.cpp, go to its GitHub page (https://github.com/ggml-org/llama.cpp) and either compile from source or download the binary. Accept no imitations, and if you get something working when you type llama into the command-line it’s likely not llama.cpp but rather some other critter. Once you have, you can use the models over the command line (remember that you’ll have to specify your own prompt: many of these models are a bit… eccentric until you give them quite clear instructions that ChatGPT and the like give them by default) or over the browser (where, in my experience, they are by default a bit better behaved). Once there, you can interact with them in any way you want, and even perform some fine-tuning to give them practice in whatever task you want them to learn. For guidance on prompt selection, the best paper I have read for this is Scheepens et al. (2024; DOI: 10.1111/2041-210X.14341): it’s written clearly, isn’t full of nonsense hype, and within minutes you’ll be a rockstar.
10.5.6.1 LLaMa: Large Language Model Meta AI
You have never read a methods description as lacking in detail as LLaMa. Go ahead, read the section (third page of https://arxiv.org/pdf/2302.13971; I’ll wait): it doesn’t describe what the model does, but rather how it differs (in six sentences, total) from three other models in what it does. As mentioned earlier in this handout, there is good reason to suppose that the listed training data for the model is incomplete, so I’m not even sure I believe that. Anyway, the important thing is it’s a really good model. The 1Tb version is sufficiently lightweight and accurate that I feel comfortable using it for toy demonstrations (if you’re reading this, I’ve likely used it in class) and so I encourage you to give it a whirl. Bear in mind you will have to pre-register to download the models from Meta (the registration doesn’t take long to come through) but the quantisation process for these models is quite painful, so I would recommend making use of one of the many existing quantised versions available on HuggingFace.
10.5.6.2 DeepSeek
DeepSeek caused a big stir on its release for a few reasons. One of them was its price; DeepSeek was trained for “only” $6 million dollars (that’s the price of running the computers for that length of time, not the price even of buying them or researching what to run on them!). The other was its use of other language models, both in terms of to train its models (see ‘distillation’ below) but also potentially by making use of the weights from other models. These matters don’t really concern us here, but are of relevance when thinking about the long-term viability of the AI modelling ecosystem.
Reflection and Rules. When using DeepSeek, it becomes immediately apparent that it has a different conversational ‘feel’ to it. It sounds almost Socratic: asking itself questions, and working through its reasoning out loud29. This was by design: it was trained through a rule-based, reasoning-reinforcement approach whereby models were trained to ‘think out loud’ and fed training data labelled in that way. It’s worth noting that DeepSeek isn’t the only model being trained in this way (OpenAI have been doing this for some time) but it is quite clear that DeepSeek have done it very well.
Mixture of Experts. DeepSeek’s architecture was designed to not be one uber-model that was flexible and could do anything, but rather a series of sub-models that excelled at particular tasks. To take a biological metaphor, DeepSeek is like the mammalian brain: it has specialised regions that are best-suited to particular tasks, while most other language models are a bit more like the human frontal cortex in that they’re just a big blob of powerful cells that self-organise to solve problems30. It seems that architecture is easier to train.
Distillation. Distillation is when you take larger, more complex models, and then use their outputs to train smaller models. Typically you are a bit clever in what you take; making use of, for example, the estimated probabilities of word-choice rather than the words output, so that the smaller model can learn more efficiently from the larger model. A big bone of contention is the extent to which DeepSeek made use of other models in doing this, as they may even have made use of the weights from these models in their training. Ironically, some large tech companies seem to be unhappy that their intellectual property is being used in this way31…
10.6 Exercises
Please remember to scale your data before analysis using something like data <- data.frame(scale(data)) where appropriate (you shouldn’t rescale images, for example, although consider carefully your pre-processing). You can also get terms to include in your models with the following handy line of R code: cat(names(data), sep="+").
- The following exercises will make use of the simulated data from your regression tree practical. The dataset you will be working with will be the plant dataset I simulate for you in that handout. Only fit your models with a training dataset of, at most, 500 samples—this is simply for speed of calculation.
- Fit an ANN with a single layer of 5 neurons to your training data. Validate that model using your validation data.
- Does your ANN perform better or worse than a regression tree on the same data? Do you think this is a fair test?
- Calculate the relative importance of each term in your ANN. Describe the results in, at most, a few short sentences.
- Fit an ANN with two layers of neurons (you pick the number in each layer). Does it perform better, or worse, than your first model?
- The data for this exercise come from a study of red wine quality (Cortez et al. 2009; Decision Support Systems, 47(4):547–553) and are available on your course website in
winequality-red.csv.
- Fit an ANN with a single layer neurons (you decide the number) to the data. Validate the model using independent data.
- What are the most important, if any, predictors of wine quality?
- How useful would this model be to wine producers versus restaurant owners? Why?
- These data come from
boardgamegeek.com32. They describe ratings given to board games by people on the Internet (rating), and information about each board game such as theyearin which they were published, theircomplexity(how difficult they are to learn, rated out of 5), and various other variables that boring people such as myself find interesting. TensorFlow is capable of dealing with discrete/categorical variables, but to keep things simple I’m not including them in today’s exercise.
# Load the data in like this (what does the row.names bit do?)
data <- read.csv("boardgames-continuous.csv", row.names=1, as.is=TRUE)
# Plot the data out like this
with(data, plot(rating ~ year))
# Format the data like this
response <- data$rating
explanatory <- as.matrix(data[,-1])
# Get the dimensions of the data
ncol(explanatory)
#> [1] 11
# Don't forget to scale them!...
# If you have trouble with rownames, you can get rid of them like this
rownames(explanatory) <- NULL- Train a deep learning network with at least two hidden layers on these data. Use the same activation function throughout the network.
- Validate the data using independent data. How well does the model perform?
- Use a different activation function in one, or all, of the layers. Assess, using independent data, whether this difference makes the model perform better.
- These data come from a classic handwriting dataset that comes bundled with Keras. There are many guides online as to how to load this dataset, but I should add that it’s in precisely the same structure as the data I have been using in this session. So please do go ahead and read those guides, because you might just learn something new!
library(keras3)
mnist <- dataset_mnist()
x_train <- mnist$train$x
y_train <- mnist$train$y
x_test <- mnist$test$x
y_test <- mnist$test$y- Train a “flat” deep learning network with at least two hidden layers on these data. These are handwritten digits (the numbers 0–9), so set up your output layer accordingly.
- Fit a convolutional neural network to the data of the same hidden structure as in my example code above.
- Add, subtract, and/or alter hidden layers in order to get your model to fit faster than it does in part (b) above.
- Which of the two models, (b) or (c), do you think is the best fit to your data? Things to think about: if your model fits faster, does it matter if its quality increases more slowly across epochs?…
- Recurrent neural networks are the most advanced topic in the course, and so if you want to develop the ability to work with such data I’m afraid you’re going to have to experiment a little on your own. Below I give an example of a stock-market analysis you could perform using Keras, but I encourage you to follow your own interests. Bear in mind that many of these questions are harder than they seem; do not be dispirited and experiment with different architectures. The code below will load some stock data into
Rfor you.
library(tidyquant)
#> Registered S3 method overwritten by 'quantmod':
#> method from
#> as.zoo.data.frame zoo
#> ── Attaching core tidyquant packages ─────────────────────── tidyquant 1.0.12 ──
#> ✔ PerformanceAnalytics 2.1.0 ✔ TTR 0.24.4
#> ✔ quantmod 0.4.29 ✔ xts 0.14.2
#> ── Conflicts ────────────────────────────────────────── tidyquant_conflicts() ──
#> ✖ zoo::as.Date() masks base::as.Date()
#> ✖ zoo::as.Date.numeric() masks base::as.Date.numeric()
#> ✖ PerformanceAnalytics::legend() masks graphics::legend()
#> ✖ quantmod::summary() masks base::summary()
#> ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
apple <- getSymbols("AAPL", from="2018-01-01", to="2018-12-31")- Train a standard (i.e., not recurrent) neural network to predict a stock’s value on the basis of the stock values for the past five days.
- Fit a recurrent neural network that will do the same task as in (a).
- Fit a recurrent neural network that will predict stock values for the next week on the basis of the last week’s worth of stock values.
- Fit a recurrent neural network that will predict six months of stock values on the basis of the last six months. (Note: this is harder than it sounds and will require careful thought as to how to prepare your training data, as well as needing more training data).
- Fit a recurrent network to multiple stocks over whatever time period you choose. Note that this is a multivariate time-series analysis, and so will require you to use something like the
time_distributedlayer (https://keras.io/api/layers/recurrent_layers/time_distributed/)—read the documentation for this layer carefully and consider this a challenge exercise. - Improve your answer to (e) above using an autoencoder layer, and explore the behaviour of its reduced representation through time. By carefully choosing input stocks, develop an accurate model of the stock market and then send it to me33.
If you know what Newton’s method for finding the minimum of a function is, let me save you some trouble: that’s exactly what this whole thing is. It’s not described like that on the Internet because most people don’t seem to notice it, and academics don’t simply use Newton’s method because the likelihood surface is so strange that it often fails (all the interactions between layers mess things up). Don’t believe everything you read on the Internet about artificial neural networks…↩︎
Alright, that’s the last Terminator reference, I promise.↩︎
Please note that I am deliberately trying to be informal in my definitions for ease of presentation; if this informality is causing you trouble then you likely already know what a tensor is, so feel free to skip these two paragraphs.↩︎
If you’re joining me from skipping the last two paragraphs—what’s the difference between a table and a matrix among friends?…↩︎
Technically a ‘multi-dimensional array’, but the meaning is just the same↩︎
‘High-level’ means you don’t have to worry about how TensorFlow is programmed (unless you want to).
Ris a high-level language because you can use it on a Mac or a Windows computer and it works the same way, despite them both being fundamentally different kinds of computer. The low-level complexity is hidden from you.↩︎Not everyone, of course, and if you know someone who’s an exception then I’m sure they have a great reason to do so.↩︎
All neurons within your brain fire at a background rate; it is deviations from this background rate that convey information.↩︎
You don’t believe me, do you? It is possible to make your vision almost buckle by staring at this video (https://upload.wikimedia.org/wikipedia/commons/d/d3/Illusion_movie.ogv) for about twenty seconds and then looking away. There are versions of these kinds of effects that can alter your vision for some time; this isn’t one of them, but after one very irritating experiment as an undergraduate I perceived musical staves as a very pale light-green for a month or so.↩︎
Well, they’re not super cheap, but let’s ignore that for a moment↩︎
Or read the above, carefully, and then watch the videos in the ‘Visual explanation’ at https://en.wikipedia.org/wiki/Convolution#Visual_explanation↩︎
If this seems somehow ‘naughty’, recall that all modeling really involves is the selection of transformations to your data that best explain it…↩︎
I might argue both artificial and real!↩︎
Perhaps a rather lonely example, but you get my point…↩︎
remember those from the first section?↩︎
(Advanced side-note): This is the neural network equivalent of problems exploring a very fine-grained likelihood space in maximum likelihood-estimation, or ‘divergent transitions’ in Hamiltonian MCMC in Bayesian problems.↩︎
Alright, you got me, technically a factor analysis, but for the purposes of this demonstration it’s easier to lie to you and, also, there is a good chance you’ve encountered PCA before…↩︎
My favourite is on the TensorFlow website: https://www.tensorflow.org/text/tutorials/transformer↩︎
Come to think of it, is there anything stopping you having a two-dimensional RNN?…↩︎
Strictly speaking, it is using ‘tokens’, which could be entire words or (more likely) parts of words, but I’m going to use ‘word’ in order to reduce the number of new terms↩︎
It’s worth noting that this was all founded by Google, and first used in Google Translate. I’m a bit of a Google fan-boy I know, but it is noteworthy that the original articles about ChatGPT were probably translated into other languages by transformers… Vaswani et al. (2017) “Attention is all you need.” NIPS 30.↩︎
Alright, they’re still bad, but they’re not that bad…↩︎
I mean, you still kind of are, just with more steps and no credit being given to the people who figured out the answers in the first place…↩︎
Did she give permission?↩︎
Slander? Well, yes, but… https://www.wired.com/story/new-documents-unredacted-meta-copyright-ai-lawsuit/↩︎
Pearse et al. (2018) Evolution 72(4): 944↩︎
Hint hint nudge nudge https://github.com/willpearse/Xena↩︎
I’m sure they wouldn’t say that, but it clearly is↩︎
I like to think that’s how I sound in lectures, but I have been reliably informed that I just sound like I’m madly ranting at myself↩︎
That is doing a bit of disservice to the human brain but, in many ways, the frontal cortex would be the place to take a massive head-injury because it can re-wire and adapt to it. Your entire personality would change, of course, and so arguably you would die, but still: you would likely still be able to do the sorts of things you ask ChatGPT to do↩︎
It’s easy for me to make cheap jokes, and so I will, but my slight fear is that this will lead to a further crackdown on model architecture, data, and model weight release.↩︎
Downloaded using code from https://github.com/willpearse/boardgamegeek—you can only imagine how exciting I am in my free time.↩︎
This last part is a joke…↩︎