7 The history of Machine Learning (ML)

This book provides an overview of what I view as the most important concepts in machine learning (ML) as used in research today. It is not a catch-all bag for every technique, but rather an attempt to cover the fundamental concepts that form the ‘classical canon’ of methods that I personally think every practising biologists should be aware of (this chapter) and then a very popular new approach (deep learning) that is tremendously powerful but quite a bit more complex. I strongly encourage you to view this section of the course as a sort of pot-luck: you’re going to be learning a lot of methods very quickly, so view this as a quick overview that you can dip into later for more details when applying approaches. Learn the critical concepts (described below) and how they are applied in each approach that I describe. The details of each method, while important, are things you can return to later: the general framework within which everything sits is the most important thing for you to learn (and something that is absent from most other resources you will find).

7.1 A historical note

To the lay audience, the explosion in application of machine learning approaches over the last decade is often viewed as unprecedented, terrifying, or transformative to the way we live and conduct science. Before going through, in practical terms, what machine learning is, I think it’s important to engage with each of these three concepts.

Despite this section of this course covering what I am terming ‘classical’ methods, it is the only part of the course that contains anything that would have been unknown to a statistician in the 1980s. There is essentially nothing in artificial neural networks that we are covering in this course, and we are covering some very advanced topics that have powered a number of headlines, that is younger than I am. Thus the advances we are seeing in machine learning are not, in any sense, unprecedented: they are a consequence of Moore’s law and the emergence of Graphical Processing Units (GPUs). Moore’s law is a prediction (that has, broadly speaking, held until quite recently) made in 1965 that the number of transistors on a computer chip would double every year. The consequence of this is an explosion in the processing power—speed—with which we can carry out work. It is why deep learning methods that, in the 1980s, were essentially impossible to carry out, are possible today. Indeed, it’s why a lot of machine learning research moved on from deep learning in the 1980s, because it was quite clear we wouldn’t be able to fit such methods for some time. Changes in computer memory (size and speed of access) have, of course, also helped a great deal—perhaps more than processor power increases more recently. GPUs, which were originally developed to power computer games, have turned out to be incredibly powerful tools for machine learning. GPUs are essentially stupid CPUs (Central Processing Units); glossing over a lot of technical details, whereas CPUs now often have two (or, in particularly powerful machines, 30) processing cores, GPUs in consumer laptops often have hundreds of cores (and powerful server machines tens of thousands). These GPUs have less memory per core, can’t do fancy things like vector-instructions or quite standard things like follow independent lines of execution (e.g., they can’t execute an ‘if’ statement independently from other cores), but goodness are there a lot of them. GPUs are very good at doing the same calculation over and over again, and are wonderful for training artificial neural networks.

Machine learning came about because attempts at true Artificial Intelligence (AI) failed. In the 1970s there were reasons to suppose that we might truly be able to build a sentient computer within a few decades; barring the Internet being self-aware, I think it’s fair to say we failed at that task. Scientists had the idea that focusing on problems in prediction and statistical learning—which they termed machine learning—might present a lateral move that might one day allow us to move back into true AI. The field you see today results from that lateral move, and as it has expanded it has somehow come to pass that a lot of machine learning methods that seem impressive are termed ‘AI’ in the media again. People trying to sell self-driving cars and hot-dog detectors liked the positive press, and so the old term stuck. There is nothing in a support vector machine that implies, in any sense, sentience or self-awareness. As you will see, supervised methods do have a sense of ‘goals’, but they are defined by us and not by the system itself. Artificial neural networks can be made to be ‘Turing complete’ but this does not change the fact that they are iteratively minimising the sum of squared errors. There is no meaningful sense in which such a network can become self-aware; they are transient extensions of linear models. Anyone who thinks otherwise is ignorant, and anyone who thinks otherwise and continues to fit one is a monster! But there is no need to fear your laptop has begun planning to take over the world.

One thing is for certain, however, and that is that machine learning methods are very powerful. There is, however, a distinction to be drawn between machine learning and big data: Twitter is exerting a huge influence on our lives because of the volume of data and number of people using it, not because of the algorithms it employs. The long-term implications of machine learning on society are likely to be very large indeed, but as with every other piece of technology from stone tools to nuclear reactors the terrifying and transformative aspects of it will be how we choose to use it.

In biology, I have noticed a tendency for researchers to treat machine learning as a black box that can solve a statistical problem. This is a profound waste of the power of ML: it is very difficult to derive general insights from the most complex ML methods, and it is often difficult to assess support for general mechanisms using them. Indeed, at worst it is literally impossible to understand how or why an ML method performs well. What ML is capable of doing, and where I think it may be truly transformative for the life sciences, is in automating tasks that up until recently required human intervention: tasks such as image processing, acoustic recognition, and literature review. Thus, in this class, I will be focusing on methods that are very good at certain classical statistical problems, but also those methods I view as transformative to how we conduct boring tasks in biology. My aim is to give you an ML superpower: to free you from routine drudgery so that you can focus on biological questions and problems.

7.2 Machine Learning (ML) and algorithm definitions

There is a reason I gave you a historical overview of ML before actually defining it, and that is that there is no good single definition of ML that I have ever found. I define machine learning approaches as ‘algorithms that are applied to data with the hope that they solve some particular problem, with no guarantee that they will generalise to other problems or situations’. Let’s unpack that statement, and by the end of it you will hopefully have an intuitive understanding of what ML is, and thus why there will never be a good definition of it.

ML is all about algorithms. Algorithms are well (albeit variously) defined, but broadly they are sequences of computational steps that transform an input into an output. They have no guarantee of meaning; indeed, they are meaning free, and they differ from mathematical proofs in that the are procedural and cannot guarantee anything. Mathematical proofs can be written about algorithms, showing that they will always do certain things, but the algorithms themselves cannot guarantee anything. Algorithms have been developed to minimise the sum of the squared errors between predictions and underlying data in bivariate regression, and as a result there are linear regression algorithms and so linear regression is a machine learning algorithm. However, the derivation of linear regression (and its promised features, like passing through the means of its underlying data) are statistically proved, and not part of the algorithm. In other words, someone has proved that linear regression will do certain things—that its results will generalise to other situations—but the same does not have to be true of a machine learning algorithm. Thus all statistical methods are machine learning algorithms, because they are embedded within concepts such as probability density functions or Bayesian posteriors, but not all machine learning methods need be and so not all machine learning algorithms are statistical methods.

Now perhaps you see why I introduced the concepts of ‘hope’ and ‘no guarantee’ into my definition of ML. By leaving behind the requirement that we can prove something about probabilities or statistics and instead focusing on algorithms, ML frees us to explore weird and wacky things that we couldn’t hope to explain to our friends in the mathematics department¹. But it also means we are left with hopes and dreams and desires, because algorithms are, by definition, free of meaning. We can only hope that they work well, and we must come up with new ways (described in this course!) of assessing whether or not the work well.

And so perhaps you also see why there can never be a good definition of ML. Because ML is a field that is opening its arms, and saying ‘come and have a go I’ll try anything!’ means that almost anything can be classified as ML. An ‘if’ statement in your first programming script could be described as ML if you really wanted it to, just as could generalised linear mixed effects models—all of them involve an algorithm, and all of them produce an output from an input.

7.3 (Un)supervised methods, dimensionality reduction, and prediction

The distinction is often made in ML between supervised and unsupervised methods. A supervised method is on where you (metaphorically) tell the method what it needs to do given some input data. So, for example, you might have a set of 100 images and you have labelled those 100 as being of either a blackbird or a blue tit. If the ML method is to take the input (the images) and predict the output (the label—the bird species) then it is a supervised method. In an unsupervised method, you don’t tell the algorithm whether it has found the right answer or not. So, for example, you might ask it to sort those same 100 images into categories, but you wouldn’t tell it what you thought the categories should be. The way I prefer to think of it is that supervised methods have response variables, and unsupervised methods do not. Note that, in the example of categorisation I just gave, we might choose to ascribe meaning to the categories the algorithm has created. That’s fine, but as that information doesn’t feed back into the ML algorithm this is still an unsupervised process. People occasionally find ways to blur the lines between the two categories; good for them, but don’t worry about such edge cases.

Unsupervised methods often involve some aspect of dimensionality reduction. You are familiar, by now, from your linear algebra about how matrices can have orders and thus datasets can have dimensions. To review, a table (matrix) might have ten columns; if it were a dataset, we might say it has ten variables and thus is a ten-dimensional dataset. Dimensionality reduction seeks to find ways of describing the majority of information within that dataset in lower dimensions: reducing or compressing in the information in those ten dimensions into two dimensions, for example. Sometimes such methods compress the information in columns of continuous numerical data, sometimes into categories. Most of the good methods for this are much older than the field of ML itself (notably principal components analysis).

Supervised methods are often used for prediction. Indeed, the supervised methods we’ll be covering are analogues and complements to classical statistical techniques like regression that you are already familiar with. In more advanced techniques, and in particular those where we have left behind any hope of deriving or working with a likelihood for the model we are fitting, we must use prediction as our measure of goodness of fit. We often run our model to predict some known training data, and then assess its efficiency on some independent test data. We might then go ahead and run that model on even more data to solve our particular problem. This is important because advanced methods have a tendency to overfit data: they fit the training data so well that they cannot generalise to work with other data. To give an example of such a workflow, we might train our bird identifying model with 100 images, then test its predictive accuracy in 200 test images. If it performed well (let’s say it got 98% of the test images right), we could then apply it to $10,000$ additional images about which we didn’t know the species. If, however, the accuracy with the 100 training images was very high (98%) but very poor in the 200 test images (10%), we would say our model was overfitting the test data.

7.4 Variance (reduction) and bagging

If most of academia and life is about managing expectations, most of statistics is about managing variance. Sometimes we want to identify the sources of variance (e.g., in ANOVA), sometimes we want to minimise unexplainable variation (e.g., with careful experimental designs). In ML, variance is a problem because of overfitting: we want to minimise the ability of our models to predict variance that’s unique to our training data. One way, somewhat paradoxically, is to bootstrap your training data: split it up into lots of smaller, random subsets, and then fit your model to those subsets. In so doing you increase the variability in your training data because each subset is different. Because your model has to perform well across all of these random subsets, it paradoxically is less-prone to overfitting because the training process exposes it to more variation. Think of it as like taking the average of lots of groups: we’re averaging out the variation within the training data, and thus allowing our model to see more of the ‘big picture’ variation within the data. We’ll see the power of such approaches when we encounter regression tree models, whose ability to detect complex interactions among variables make them susceptible to overfitting without care and attention.

Such variance-managing techniques are also important for deep learning approaches, which are also very susceptible to overfitting. In such models it’s often common to use augmentation approaches: to expand our training data by adding variance to it, and then running fitting our model to those augmented data. The idea is, once again, to minimise overfitting by introducing variance: in this case, that variance is (by design) erroneous, and so you’re better preparing your model for the ‘real world’ where there is a lot more variation than is otherwise visible in the smaller subsets of training data you have available. What is surprising, to me, about such approaches is how well they work, even when the augmentation methods are very simplistic (e.g., slight, random colour changes in images) and the data very complex (e.g., images of birds).

In what is an emerging and recurring theme of ML, these approaches are neither unique to ML nor were they invented for ML. Bootstrapping and augmentation are approaches to manage variance in other, classical statistical settings. So don’t view these tools as purely ML tools, but rather statistical tools.

7.5 How to choose the right tool for the job

I hate statistical flow-charts because, unlike taxonomic identification, there is rarely a single, unambiguous ‘correct’ statistical approach for a particular problem (that isn’t how statistics works). Regardless, when picking a problem, ask yourself first: do I have a response variable? If so, then you want a supervised approach (e.g., SVM). For supervised problems, ask yourself if you have a lot of explanatory variables (i.e., more than 20). If so, then you want a lasso or least-angle regression. If you have a very complex set of potentially interacting factors, then you probably want some form of regression tree or an artificial neural network. Otherwise, you probably want something else: and do bear in mind that thing you want is likely a classical statistical test. If you don’t have a response variable and you want to simplify your data somehow, you likely want an unsupervised approach. If all your data are continuous, use a principal components analysis, otherwise use a principal coordinate analysis. If you want to split your data into categories, use some kind of clustering algorithm.

The huge variety of supervised machine learning algorithms—approaches that require response variables—is often overlooked because students are so desperate to learn about artificial neural networks. This is a shame, because the ML toolkit contains a number of extremely powerful tools that are capable of solving a number of very common problems. In this section we are going to cover classical tools that are in many ways the ML complement of Generalised Linear Models (GLM). These approaches are capable of either categorising data (e.g., true/false, red/blue/green) or predicting continuous variables. They set themselves apart from approaches like GLM in terms of their flexibility; some are capable of dealing with overwhelmingly large number of potential explanatory variables (e.g., lasso regression), managing complex and multi-faceted interactions naturally (e.g., regression trees), or are so flexible that they can handle almost any classification problem (e.g., SVMs). Critically and importantly, even when these approaches work with categories, they are supervised approaches, and so you are required to give the aclgorithms exemplars of the different categories for which you are building a model. Whereas a clustering algorithm will choose the identity (and potentially number) of groups for you, in supervised methods you must give data on cluster identity.

Note that many of your friends in the mathematics department are working on ML, because they hope to derive new algorithms from statistical and mathematical theory. If you don’t have a friend in the mathematics department, get one: we could all use more friends, and in my experience maths departement parties are pretty crazy.↩︎

# The history of Machine Learning (ML) This book provides an overview of what I view as the most important concepts in machine learning (ML) as used in research today. It is not a catch-all bag for every technique, but rather an attempt to cover the fundamental concepts that form the 'classical canon' of methods that I personally think every practising biologists should be aware of (this chapter) and then a very popular new approach (deep learning) that is tremendously powerful but quite a bit more complex. I strongly encourage you to view this section of the course as a sort of pot-luck: you're going to be learning a lot of methods very quickly, so view this as a quick overview that you can dip into later for more details when applying approaches. Learn the critical concepts (described below) and how they are applied in each approach that I describe. The details of each method, while important, are things you can return to later: the general framework within which everything sits is the most important thing for you to learn (and something that is absent from most other resources you will find). ## A historical note To the lay audience, the explosion in application of machine learning approaches over the last decade is often viewed as unprecedented, terrifying, or transformative to the way we live and conduct science. Before going through, in practical terms, what machine learning is, I think it's important to engage with each of these three concepts. Despite this section of this course covering what I am terming 'classical' methods, it is the only part of the course that contains anything that would have been unknown to a statistician in the 1980s. There is essentially nothing in artificial neural networks that we are covering in this course, and we are covering some very advanced topics that have powered a number of headlines, that is younger than I am. Thus the advances we are seeing in machine learning are not, in any sense, unprecedented: they are a consequence of Moore's law and the emergence of Graphical Processing Units (GPUs). Moore's law is a prediction (that has, broadly speaking, held until quite recently) made in 1965 that the number of transistors on a computer chip would double every year. The consequence of this is an explosion in the processing power—speed—with which we can carry out work. It is why deep learning methods that, in the 1980s, were essentially impossible to carry out, are possible today. Indeed, it's why a lot of machine learning research moved on from deep learning in the 1980s, because it was quite clear we wouldn't be able to fit such methods for some time. Changes in computer memory (size and speed of access) have, of course, also helped a great deal—perhaps more than processor power increases more recently. GPUs, which were originally developed to power computer games, have turned out to be incredibly powerful tools for machine learning. GPUs are essentially stupid CPUs (Central Processing Units); glossing over a lot of technical details, whereas CPUs now often have two (or, in particularly powerful machines, 30) processing cores, GPUs in consumer laptops often have hundreds of cores (and powerful server machines tens of thousands). These GPUs have less memory per core, can't do fancy things like vector-instructions or quite standard things like follow independent lines of execution (*e.g.*, they can't execute an 'if' statement independently from other cores), but goodness are there a lot of them. GPUs are very good at doing the same calculation over and over again, and are wonderful for training artificial neural networks. Machine learning came about because attempts at true Artificial Intelligence (AI) failed. In the 1970s there were reasons to suppose that we might truly be able to build a sentient computer within a few decades; barring the Internet being self-aware, I think it's fair to say we failed at that task. Scientists had the idea that focusing on problems in prediction and statistical learning—which they termed machine learning—might present a lateral move that might one day allow us to move back into true AI. The field you see today results from that lateral move, and as it has expanded it has somehow come to pass that a lot of machine learning methods that seem impressive are termed 'AI' in the media again. People trying to sell self-driving cars and hot-dog detectors liked the positive press, and so the old term stuck. There is nothing in a support vector machine that implies, in any sense, sentience or self-awareness. As you will see, supervised methods do have a sense of 'goals', but they are defined by us and not by the system itself. Artificial neural networks can be made to be 'Turing complete' but this does not change the fact that they are iteratively minimising the sum of squared errors. There is no meaningful sense in which such a network can become self-aware; they are transient extensions of linear models. Anyone who thinks otherwise is ignorant, and anyone who thinks otherwise and continues to fit one is a monster! But there is no need to fear your laptop has begun planning to take over the world. One thing is for certain, however, and that is that machine learning methods are very powerful. There is, however, a distinction to be drawn between machine learning and big data: Twitter is exerting a huge influence on our lives because of the volume of data and number of people using it, not because of the algorithms it employs. The long-term implications of machine learning on society are likely to be very large indeed, but as with every other piece of technology from stone tools to nuclear reactors the terrifying and transformative aspects of it will be how we choose to use it. In biology, I have noticed a tendency for researchers to treat machine learning as a black box that can solve a statistical problem. This is a profound waste of the power of ML: it is very difficult to derive general insights from the most complex ML methods, and it is often difficult to assess support for general mechanisms using them. Indeed, at worst it is literally impossible to understand how or why an ML method performs well. What ML is capable of doing, and where I think it may be truly transformative for the life sciences, is in automating tasks that up until recently required human intervention: tasks such as image processing, acoustic recognition, and literature review. Thus, in this class, I will be focusing on methods that are very good at certain classical statistical problems, but also those methods I view as transformative to how we conduct boring tasks in biology. My aim is to give you an ML superpower: to free you from routine drudgery so that you can focus on biological questions and problems. ## Machine Learning (ML) and algorithm definitions There is a reason I gave you a historical overview of ML before actually defining it, and that is that there is no good single definition of ML that I have ever found. I define machine learning approaches as 'algorithms that are applied to data with the hope that they solve some particular problem, with no guarantee that they will generalise to other problems or situations'. Let's unpack that statement, and by the end of it you will hopefully have an intuitive understanding of what ML is, and thus why there will never be a good definition of it. ML is all about algorithms. Algorithms are well (albeit variously) defined, but broadly they are sequences of computational steps that transform an input into an output. They have no guarantee of meaning; indeed, they are meaning free, and they differ from mathematical proofs in that the are procedural and cannot guarantee anything. Mathematical proofs can be written about algorithms, showing that they will always do certain things, but the algorithms themselves cannot guarantee anything. Algorithms have been developed to minimise the sum of the squared errors between predictions and underlying data in bivariate regression, and as a result there are linear regression algorithms and so linear regression is a machine learning algorithm. However, the derivation of linear regression (and its promised features, like passing through the means of its underlying data) are statistically proved, and not part of the algorithm. In other words, someone has *proved* that linear regression will do certain things—that its results will generalise to other situations—but the same does not have to be true of a machine learning algorithm. Thus all statistical methods are machine learning algorithms, because they are embedded within concepts such as probability density functions or Bayesian posteriors, but not all machine learning methods need be and so not all machine learning algorithms are statistical methods. Now perhaps you see why I introduced the concepts of 'hope' and 'no guarantee' into my definition of ML. By leaving behind the requirement that we can prove something about probabilities or statistics and instead focusing on algorithms, ML frees us to explore weird and wacky things that we couldn't hope to explain to our friends in the mathematics department^[Note that many of your friends in the mathematics department are working on ML, because they hope to derive new algorithms from statistical and mathematical theory. If you don't have a friend in the mathematics department, get one: we could all use more friends, and in my experience maths departement parties are pretty crazy.]. But it also means we are left with hopes and dreams and desires, because algorithms are, by definition, free of meaning. We can only hope that they work well, and we must come up with new ways (described in this course!) of assessing whether or not the work well. And so perhaps you also see why there can never be a good definition of ML. Because ML is a field that is opening its arms, and saying 'come and have a go I'll try anything!' means that almost anything can be classified as ML. An 'if' statement in your first programming script could be described as ML if you really wanted it to, just as could generalised linear mixed effects models—all of them involve an algorithm, and all of them produce an output from an input. ## (Un)supervised methods, dimensionality reduction, and prediction The distinction is often made in ML between *supervised* and *unsupervised* methods. A supervised method is on where you (metaphorically) tell the method what it needs to do given some input data. So, for example, you might have a set of 100 images and you have labelled those 100 as being of either a blackbird or a blue tit. If the ML method is to take the input (the images) and predict the output (the label—the bird species) then it is a supervised method. In an unsupervised method, you don't tell the algorithm whether it has found the right answer or not. So, for example, you might ask it to sort those same 100 images into categories, but you wouldn't tell it what you thought the categories should be. The way I prefer to think of it is that *supervised methods have response variables, and unsupervised methods do not*. Note that, in the example of categorisation I just gave, we might choose to ascribe meaning to the categories the algorithm has created. That's fine, but as that information doesn't feed back into the ML algorithm this is still an unsupervised process. People occasionally find ways to blur the lines between the two categories; good for them, but don't worry about such edge cases. Unsupervised methods often involve some aspect of *dimensionality reduction*. You are familiar, by now, from your linear algebra about how matrices can have orders and thus datasets can have dimensions. To review, a table (matrix) might have ten columns; if it were a dataset, we might say it has ten variables and thus is a ten-dimensional dataset. Dimensionality reduction seeks to find ways of describing the majority of information within that dataset in lower dimensions: reducing or compressing in the information in those ten dimensions into two dimensions, for example. Sometimes such methods compress the information in columns of continuous numerical data, sometimes into categories. Most of the good methods for this are much older than the field of ML itself (notably principal components analysis). Supervised methods are often used for *prediction*. Indeed, the supervised methods we'll be covering are analogues and complements to classical statistical techniques like regression that you are already familiar with. In more advanced techniques, and in particular those where we have left behind any hope of deriving or working with a likelihood for the model we are fitting, we must use prediction as our measure of goodness of fit. We often run our model to predict some known *training* data, and then assess its efficiency on some independent *test* data. We might then go ahead and run that model on even more data to solve our particular problem. This is important because advanced methods have a tendency to *overfit* data: they fit the training data so well that they cannot generalise to work with other data. To give an example of such a workflow, we might train our bird identifying model with 100 images, then test its predictive accuracy in 200 test images. If it performed well (let's say it got 98% of the test images right), we could then apply it to $10,000$ additional images about which we didn't know the species. If, however, the accuracy with the 100 training images was very high (98%) but very poor in the 200 test images (10%), we would say our model was overfitting the test data. ## Variance (reduction) and bagging If most of academia and life is about managing expectations, most of statistics is about managing variance. Sometimes we want to identify the sources of variance (*e.g.*, in ANOVA), sometimes we want to minimise unexplainable variation (*e.g.*, with careful experimental designs). In ML, variance is a problem because of overfitting: we want to minimise the ability of our models to predict variance that's unique to our training data. One way, somewhat paradoxically, is to *bootstrap* your training data: split it up into lots of smaller, random subsets, and then fit your model to those subsets. In so doing you increase the variability in your training data because each subset is different. Because your model has to perform well across all of these random subsets, it paradoxically is less-prone to overfitting because the training process exposes it to more variation. Think of it as like taking the average of lots of groups: we're averaging out the variation within the training data, and thus allowing our model to see more of the 'big picture' variation within the data. We'll see the power of such approaches when we encounter regression tree models, whose ability to detect complex interactions among variables make them susceptible to overfitting without care and attention. Such variance-managing techniques are also important for deep learning approaches, which are also very susceptible to overfitting. In such models it's often common to use *augmentation* approaches: to expand our training data by adding variance to it, and then running fitting our model to those augmented data. The idea is, once again, to minimise overfitting by introducing variance: in this case, that variance is (by design) erroneous, and so you're better preparing your model for the 'real world' where there is a lot more variation than is otherwise visible in the smaller subsets of training data you have available. What is surprising, to me, about such approaches is how well they work, even when the augmentation methods are very simplistic (*e.g.*, slight, random colour changes in images) and the data very complex (*e.g.*, images of birds). In what is an emerging and recurring theme of ML, these approaches are neither unique to ML nor were they invented for ML. Bootstrapping and augmentation are approaches to manage variance in other, classical statistical settings. So don't view these tools as purely ML tools, but rather statistical tools. ## How to choose the right tool for the job I hate statistical flow-charts because, unlike taxonomic identification, there is rarely a single, unambiguous 'correct' statistical approach for a particular problem (that isn't how statistics works). Regardless, when picking a problem, ask yourself first: do I have a response variable? If so, then you want a supervised approach (*e.g.*, SVM). For supervised problems, ask yourself if you have a lot of explanatory variables (*i.e.*, more than 20). If so, then you want a lasso or least-angle regression. If you have a very complex set of potentially interacting factors, then you probably want some form of regression tree or an artificial neural network. Otherwise, you probably want something else: *and do bear in mind that thing you want is likely a classical statistical test*. If you don't have a response variable and you want to simplify your data somehow, you likely want an unsupervised approach. If all your data are continuous, use a principal components analysis, otherwise use a principal coordinate analysis. If you want to split your data into categories, use some kind of clustering algorithm. The huge variety of supervised machine learning algorithms—approaches that require response variables—is often overlooked because students are so desperate to learn about artificial neural networks. This is a shame, because the ML toolkit contains a number of extremely powerful tools that are capable of solving a number of very common problems. In this section we are going to cover classical tools that are in many ways the ML complement of Generalised Linear Models (GLM). These approaches are capable of either categorising data (*e.g.*, true/false, red/blue/green) or predicting continuous variables. They set themselves apart from approaches like GLM in terms of their flexibility; some are capable of dealing with overwhelmingly large number of potential explanatory variables (*e.g.*, lasso regression), managing complex and multi-faceted interactions naturally (*e.g.*, regression trees), or are so flexible that they can handle almost any classification problem (*e.g.*, SVMs). Critically and importantly, even when these approaches work with categories, they are *supervised* approaches, and so you are required to give the aclgorithms exemplars of the different categories for which you are building a model. Whereas a clustering algorithm will choose the identity (and potentially number) of groups for you, in supervised methods you must give data on cluster identity.