Machine Learning: Unsupervised Backpropagation - machine-learning

I'm having trouble with some of the concepts in machine learning through neural networks. One of them is backpropagation. In the weight updating equation,
delta_w = a*(t - y)*g'(h)*x
t is the "target output", which would be your class label, or something, in the case of supervised learning. But what would the "target output" be for unsupervised learning?
Can someone kindly provide an example of how you'd use BP in unsupervised learning, specifically for clustering of classification?
Thanks in advance.

The most common thing to do is train an autoencoder, where the desired outputs are equal to the inputs. This makes the network try to learn a representation that best "compresses" the input distribution.
Here's a patent describing a different approach, where the output labels are assigned randomly and then sometimes flipped based on convergence rates. It seems weird to me, but okay.
I'm not familiar with other methods that use backpropogation for clustering or other unsupervised tasks. Clustering approaches with ANNs seem to use other algorithms (example 1, example 2).

I'm not sure which unsupervised machine learning algorithm uses backpropagation specifically; if there is one I haven't heard of it. Can you point to an example?
Backpropagation is used to compute the derivatives of the error function for training an artificial neural network with respect to the weights in the network. It's named as such because the "errors" are "propagating" through the network "backwards". You need it in this case because the final error with respect to the target depends on a function of functions (of functions ... depending on how many layers in your ANN.) The derivatives allow you to then adjust the values to improve the error function, tempered by the learning rate (this is gradient descent).
In unsupervised algorithms, you don't need to do this. For example, in k-Means, where you are trying to minimize the mean squared error (MSE), you can minimize the error directly at each step given the assignments; no gradients needed. In other clustering models, such as a mixture of Gaussians, the expectation-maximization (EM) algorithm is much more powerful and accurate than any gradient-descent based method.

What you might be asking is about unsupervised feature learning and deep learning.
Feature learning is the only unsupervised method I can think of with respect of NN or its recent variant.(a variant called mixture of RBM's is there analogous to mixture of gaussians but you can build a lot of models based on the two). But basically Two models I am familiar with are RBM's(restricted boltzman machines) and Autoencoders.
Autoencoders(optionally sparse activations can be encoded in optimization function) are just feedforward neural networks which tune its weights in such a way that the output is a reconstructed input. Multiple hidden layers can be used but the weight initialization uses a greedy layer wise training for better starting point. So to answer the question the target function will be input itself.
RBM's are stochastic networks usually interpreted as graphical model which has restrictions on connections. In this setting there is no output layer and the connection between input and latent layer is bidirectional like an undirected graphical model. What it tries to learn is a distribution on inputs(observed and unobserved variables). Here also your answer would be input is the target.
Mixture of RBM's(analogous to mixture of gaussians) can be used for soft clustering or KRBM(analogous to K-means) can be used for hard clustering. Which in effect feels like learning multiple non-linear subspaces.
http://deeplearning.net/tutorial/rbm.html
http://ufldl.stanford.edu/wiki/index.php/UFLDL_Tutorial

An alternative approach is to use something like generative backpropagation. In this scenario, you train a neural network updating the weights AND the input values. The given values are used as the output values since you can compute an error value directly. This approach has been used in dimensionality reduction, matrix completion (missing value imputation) among other applications. For more information, see non-linear principal component analysis (NLPCA) and unsupervised backpropagation (UBP) which uses the idea of generative backpropagation. UBP extends NLPCA by introducing a pre-training stage. An implementation of UBP and NLPCA and unsupervised backpropagation can be found in the waffles machine learning toolkit. The documentation for UBP and NLPCA can be found using the nlpca command.

To use back-propagation for unsupervised learning it is merely necessary to set t, the target output, at each stage of the algorithm to the class for which the average distance to each element of the class before updating is least. In short we always try to train the ANN to place its input into the class whose members are most similar in terms of our input. Because this process is sensitive to input scale it is necessary to first normalize the input data in each dimension by subtracting the average and dividing by the standard deviation for each component in order to calculate the distance in a scale-invariant manner.
The advantage to using a back-prop neural network rather than a simple distance from a center definition of the clusters is that neural networks can allow for more complex and irregular boundaries between clusters.

Related

Why do neural networks work so well?

I understand all the computational steps of training a neural network with gradient descent using forwardprop and backprop, but I'm trying to wrap my head around why they work so much better than logistic regression.
For now all I can think of is:
A) the neural network can learn it's own parameters
B) there are many more weights than simple logistic regression thus allowing for more complex hypotheses
Can someone explain why a neural network works so well in general? I am a relative beginner.
Neural Networks can have a large number of free parameters (the weights and biases between interconnected units) and this gives them the flexibility to fit highly complex data (when trained correctly) that other models are too simple to fit. This model complexity brings with it the problems of training such a complex network and ensuring the resultant model generalises to the examples it’s trained on (typically neural networks require large volumes of training data, that other models don't).
Classically logistic regression has been limited to binary classification using a linear classifier (although multi-class classification can easily be achieved with one-vs-all, one-vs-one approaches etc. and there are kernalised variants of logistic regression that allow for non-linear classification tasks). In general therefore, logistic regression is typically applied to more simple, linearly-separable classification tasks, where small amounts of training data are available.
Models such as logistic regression and linear regression can be thought of as simple multi-layer perceptrons (check out this site for one explanation of how).
To conclude, it’s the model complexity that allows neural nets to solve more complex classification tasks, and to have a broader application (particularly when applied to raw data such as image pixel intensities etc.), but their complexity means that large volumes of training data are required and training them can be a difficult task.
Recently Dr. Naftali Tishby's idea of Information Bottleneck to explain the effectiveness of deep neural networks is making the rounds in the academic circles.
His video explaining the idea (link below) can be rather dense so I'll try to give the distilled/general form of the core idea to help build intuition
https://www.youtube.com/watch?v=XL07WEc2TRI
To ground your thinking, vizualize the MNIST task of classifying the digit in the image. For this, I am only talking about simple fully-connected neural networks (not Convolutional NN as is typically used for MNIST)
The input to a NN contains information about the output hidden inside of it. Some function is needed to transform the input to the output form. Pretty obvious.
The key difference in thinking needed to build better intuition is to think of the input as a signal with "information" in it (I won't go into information theory here). Some of this information is relevant for the task at hand (predicting the output). Think of the output as also a signal with a certain amount of "information". The neural network tries to "successively refine" and compress the input signal's information to match the desired output signal. Think of each layer as cutting away at the unneccessary parts of the input information, and
keeping and/or transforming the output information along the way through the network.
The fully-connected neural network will transform the input information into a form in the final hidden layer, such that it is linearly separable by the output layer.
This is a very high-level and fundamental interpretation of the NN, and I hope it will help you see it clearer. If there are parts you'd like me to clarify, let me know.
There are other essential pieces in Dr.Tishby's work, such as how minibatch noise helps training, and how the weights of a neural network layer can be seen as doing a random walk within the constraints of the problem.
These parts are a little more detailed, and I'd recommend first toying with neural networks and taking a course on Information Theory to help build your understanding.
Consider you have a large dataset and you want to build a binary classification model for that, Now you have two options that you have pointed out
Logistic Regression
Neural Networks ( Consider FFN for now )
Each node in a neural network will be associated with an activation function for example let's choose Sigmoid since Logistic regression also uses sigmoid internally to make decision.
Let's see how the decision of logistic regression looks when applied on the data
See some of the green spots present in the red boundary?
Now let's see the decision boundary of neural network (Forgive me for using a different color)
Why this happens? Why does the decision boundary of neural network is so flexible which gives more accurate results than Logistic regression?
or the question you asked is "Why neural networks works so well ?" is because of it's hidden units or hidden layers and their representation power.
Let me put it this way.
You have a logistic regression model and a Neural network which has say 100 neurons each of Sigmoid activation. Now each neuron will be equivalent to one logistic regression.
Now assume a hundred logistic units trained together to solve one problem versus one logistic regression model. Because of these hidden layers the decision boundary expands and yields better results.
While you are experimenting you can add more number of neurons and see how the decision boundary is changing. A logistic regression is same as a neural network with single neuron.
The above given is just an example. Neural networks can be trained to get very complex decision boundaries
Neural networks allow the person training them to algorithmically discover features, as you pointed out. However, they also allow for very general nonlinearity. If you wish, you can use polynomial terms in logistic regression to achieve some degree of nonlinearity, however, you must decide which terms you will use. That is you must decide a priori which model will work. Neural networks can discover the nonlinear model that is needed.
'Work so well' depends on the concrete scenario. Both of them do essentially the same thing: predicting.
The main difference here is neural network can have hidden nodes for concepts, if it's propperly set up (not easy), using these inputs to make the final decission.
Whereas linear regression is based on more obvious facts, and not side effects. A neural network should de able to make more accurate predictions than linear regression.
Neural networks excel at a variety of tasks, but to get an understanding of exactly why, it may be easier to take a particular task like classification and dive deeper.
In simple terms, machine learning techniques learn a function to predict which class a particular input belongs to, depending on past examples. What sets neural nets apart is their ability to construct these functions that can explain even complex patterns in the data. The heart of a neural network is an activation function like Relu, which allows it to draw some basic classification boundaries like:
Example classification boundaries of Relus
By composing hundreds of such Relus together, neural networks can create arbitrarily complex classification boundaries, for example:
Composing classification boundaries
The following article tries to explain the intuition behind how neural networks work: https://medium.com/machine-intelligence-report/how-do-neural-networks-work-57d1ab5337ce
Before you step into neural network see if you have assessed all aspects of normal regression.
Use this as a guide
and even before you discard normal regression - for curved type of dependencies - you should strongly consider kernels with SVM
Neural networks are defined with an objective and loss function. The only process that happens within a neural net is to optimize for the objective function by reducing the loss function or error. The back propagation helps in finding the optimized objective function and reach our output with an output condition.

Why use a restricted Boltzmann machine rather than a multi-layer perceptron?

I'm trying to understand the difference between a restricted Boltzmann machine (RBM), and a feed-forward neural network (NN). I know that an RBM is a generative model, where the idea is to reconstruct the input, whereas an NN is a discriminative model, where the idea is the predict a label. But what I am unclear about, is why you cannot just use a NN for a generative model? In particular, I am thinking about deep belief networks and multi-layer perceptrons.
Suppose my input to the NN is a set of notes called x, and my output of the NN is a set of nodes y. In a discriminative model, my loss during training would be the difference between y, and the value of y that I want x to produce (e.g. ground truth probabilities for class labels). However, what about if I just made the output have the same number of nodes as the input, and then set the loss to be the difference between x and y? In this way, the network would learn to reconstruct the input, like in an RBM.
So, given that a NN (or a multi-layer perceptron) can be used to train a generative model in this way, why would you use an RBM (or a deep belief network) instead? Or in this case, would they be exactly the same?
You can use a NN for a generative model in exactly the way you describe. This is known as an autoencoder, and these can work quite well. In fact, these are often the building blocks of deep belief networks.
An RBM is a quite different model from a feed-forward neural network. They have connections going both ways (forward and backward) that have a probabilistic / energy interpretation. You'll need to read the details to understand.
A deep belief network (DBN) is just a neural network with many layers. This can be a large NN with layers consisting of a sort of autoencoders, or consist of stacked RBMs. You need special methods, tricks and lots of data for training these deep and large networks. Simple back-propagation suffers from the vanishing gradients problem. But if you do manage to train them, they can be very powerful (encode "higher level" concepts).
Hope this helps to point you in the right directions.

Graphically, how does the non-linear activation function project the input onto the classification space?

I am finding a very hard time to visualize how the activation function actually manages to classify non-linearly separable training data sets.
Why does the activation function (e.g tanh function) work for non-linear cases? What exactly happens mathematically when the activation function projects the input to output? What separates training samples of different classes, and how does this work if one had to plot this process graphically?
I've tried looking for numerous sources, but what exactly makes the activation function actually work for classifying training samples in a neural network, I just cannot grasp easily and would like to be able to picture this in my mind.
Mathematical result behind neural networks is Universal Approximation Theorem. Basically, sigmoidal functions (those which saturate on both ends, like tanh) are smooth almost-piecewise-constant approximators. The more neurons you have – the better your approximation is.
This picture was taked from this article: A visual proof that neural nets can compute any function. Make sure to check that article, it has other examples and interactive applets.
NNs actually, at each level, create new features by distorting input space. Non-linear functions allow you to change "curvature" of target function, so further layers have chance to make it linear-separable. If there were no non-linear functions, any combination of linear function is still linear, thus no benefit from multi-layerness. As a graphical example consider
this animation
This pictures where taken from this article. Also check out that cool visualization applet.
Activation functions have very little to do with classifying non-linearly separable sets of data.
Activation functions are used as a way to normalize signals at every step in your neural network. They typically have an infinite domain and a finite range. Tanh, for example, has a domain of (-∞,∞) and a range of (-1,1). The sigmoid function maps the same domain to (0,1).
You can think of this as a way of enforcing equality across all of your learned features at a given neural layer (a.k.a. feature scaling). Since the input domain is not known before hand it's not as simple as regular feature scaling (for linear regression) and thusly activation functions must be used. The effects of the activation function are compensated for when computing errors during back-propagation.
Back-propagation is a process that applies error to the neural network. You can think of this as a positive reward for the neurons that contributed to the correct classification and a negative reward for the neurons that contributed to an incorrect classification. This contribution is often known as the gradient of the neural network. The gradient is, effectively, a multi-variable derivative.
When back-propagating the error, each individual neuron's contribution to the gradient is the activations function's derivative at the input value for that neuron. Sigmoid is a particularly interesting function because its derivative is extremely cheap to compute. Specifically s'(x) = 1 - s(x); it was designed this way.
Here is an example image (found by google image searching: neural network classification) that demonstrates how a neural network might be superimposed on top of your data set:
I hope that gives you a relatively clear idea of how neural networks might classify non-linearly separable datasets.

Hidden Markov Model Tools : Jahmm

I am a novice to machine learning, I have read about the HMM but I still have a few questions:
When applying the HMM for machine learning, how can the initial, emmission and transition probabilities be obtained?
Currently I have a set of values (consisting the angles of a hand which I would like to classify via an HMM), what should my first step be?
I know that there are three problems in a HMM (ForwardBackward, Baum-Welch, and Viterbi), but what should I do with my data?
In the literature that I have read, I never encountered the use of distribution functions within an HMM, yet the constructor that JaHMM uses for an HMM consists of:
number of states
Probability Distribution Function factory
Constructor Description:
Creates a new HMM. Each state has the same pi value and the transition probabilities are all equal.
Parameters:
nbStates The (strictly positive) number of states of the HMM.
opdfFactory A pdf generator that is used to build the pdfs associated to each state.
What is this used for? And how can I use it?
Thank you
You have to somehow model and learn the initial, emmision, and tranisition probabilities such that they represent your data.
In the case of discrete distributions and not to much variables/states you can obtain them form maximum likelihood fitting or you train a discriminative classifier that can give you a probability estimate like Random Forests or Naive Bayes. For continuous distributions have a look at Gaussian Processes or any other regression method like Gaussian Mixture Models or Regression Forests.
Regarding your 2. and 3. question: they are to general and fuzzy to be answered here. You should kindly refer to the following books: "Pattern Recognition and Machine Learning" by Bishop and "Probabilistic Graphical Models" by Koller/Friedman.

What is the difference between supervised learning and unsupervised learning? [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 2 years ago.
Improve this question
In terms of artificial intelligence and machine learning, what is the difference between supervised and unsupervised learning?
Can you provide a basic, easy explanation with an example?
Since you ask this very basic question, it looks like it's worth specifying what Machine Learning itself is.
Machine Learning is a class of algorithms which is data-driven, i.e. unlike "normal" algorithms it is the data that "tells" what the "good answer" is. Example: a hypothetical non-machine learning algorithm for face detection in images would try to define what a face is (round skin-like-colored disk, with dark area where you expect the eyes etc). A machine learning algorithm would not have such coded definition, but would "learn-by-examples": you'll show several images of faces and not-faces and a good algorithm will eventually learn and be able to predict whether or not an unseen image is a face.
This particular example of face detection is supervised, which means that your examples must be labeled, or explicitly say which ones are faces and which ones aren't.
In an unsupervised algorithm your examples are not labeled, i.e. you don't say anything. Of course, in such a case the algorithm itself cannot "invent" what a face is, but it can try to cluster the data into different groups, e.g. it can distinguish that faces are very different from landscapes, which are very different from horses.
Since another answer mentions it (though, in an incorrect way): there are "intermediate" forms of supervision, i.e. semi-supervised and active learning. Technically, these are supervised methods in which there is some "smart" way to avoid a large number of labeled examples. In active learning, the algorithm itself decides which thing you should label (e.g. it can be pretty sure about a landscape and a horse, but it might ask you to confirm if a gorilla is indeed the picture of a face). In semi-supervised learning, there are two different algorithms which start with the labeled examples, and then "tell" each other the way they think about some large number of unlabeled data. From this "discussion" they learn.
Supervised learning is when the data you feed your algorithm with is "tagged" or "labelled", to help your logic make decisions.
Example: Bayes spam filtering, where you have to flag an item as spam to refine the results.
Unsupervised learning are types of algorithms that try to find correlations without any external inputs other than the raw data.
Example: data mining clustering algorithms.
Supervised learning
Applications in which the training data comprises examples of the input vectors along with their corresponding target vectors are known as supervised learning problems.
Unsupervised learning
In other pattern recognition problems, the training data consists of a set of input vectors x without any corresponding target values. The goal in such unsupervised learning problems may be to discover groups of similar examples within the data, where it is called clustering
Pattern Recognition and Machine Learning (Bishop, 2006)
In supervised learning, the input x is provided with the expected outcome y (i.e., the output the model is supposed to produce when the input is x), which is often called the "class" (or "label") of the corresponding input x.
In unsupervised learning, the "class" of an example x is not provided. So, unsupervised learning can be thought of as finding "hidden structure" in unlabelled data set.
Approaches to supervised learning include:
Classification (1R, Naive Bayes, decision tree learning algorithm, such
as ID3 CART, and so on)
Numeric Value Prediction
Approaches to unsupervised learning include:
Clustering (K-means, hierarchical clustering)
Association Rule Learning
I can tell you an example.
Suppose you need to recognize which vehicle is a car and which one is a motorcycle.
In the supervised learning case, your input (training) dataset needs to be labelled, that is, for each input element in your input (training) dataset, you should specify if it represents a car or a motorcycle.
In the unsupervised learning case, you do not label the inputs. The unsupervised model clusters the input into clusters based e.g. on similar features/properties. So, in this case, there is are no labels like "car".
For instance, very often training a neural network is supervised learning: you're telling the network to which class corresponds the feature vector you're feeding.
Clustering is unsupervised learning: you let the algorithm decide how to group samples into classes that share common properties.
Another example of unsupervised learning is Kohonen's self organizing maps.
I have always found the distinction between unsupervised and supervised learning to be arbitrary and a little confusing. There is no real distinction between the two cases, instead there is a range of situations in which an algorithm can have more or less 'supervision'. The existence of semi-supervised learning is an obvious examples where the line is blurred.
I tend to think of supervision as giving feedback to the algorithm about what solutions should be preferred. For a traditional supervised setting, such as spam detection, you tell the algorithm "don't make any mistakes on the training set"; for a traditional unsupervised setting, such as clustering, you tell the algorithm "points that are close to each other should be in the same cluster". It just so happens that, the first form of feedback is a lot more specific than the latter.
In short, when someone says 'supervised', think classification, when they say 'unsupervised' think clustering and try not to worry too much about it beyond that.
Supervised Learning
Supervised learning is based on training a data sample
from data source with correct classification already assigned.
Such techniques are utilized in feedforward or MultiLayer
Perceptron (MLP) models. These MLP has three distinctive
characteristics:
One or more layers of hidden neurons that are not part of the input
or output layers of the network that enable the network to learn and
solve any complex problems
The nonlinearity reflected in the neuronal activity is
differentiable and,
The interconnection model of the network exhibits a high degree of
connectivity.
These characteristics along with learning through training
solve difficult and diverse problems. Learning through
training in a supervised ANN model also called as error backpropagation algorithm. The error correction-learning
algorithm trains the network based on the input-output
samples and finds error signal, which is the difference of the
output calculated and the desired output and adjusts the
synaptic weights of the neurons that is proportional to the
product of the error signal and the input instance of the
synaptic weight. Based on this principle, error back
propagation learning occurs in two passes:
Forward Pass:
Here, input vector is presented to the network. This input signal propagates forward, neuron by neuron through the network and emerges at the output end of
the network as output signal: y(n) = φ(v(n)) where v(n) is the induced local field of a neuron defined by v(n) =Σ w(n)y(n). The output that is calculated at the output layer o(n) is compared with the desired response d(n) and finds the error e(n) for that neuron. The synaptic weights of the network during this pass are remains same.
Backward Pass:
The error signal that is originated at the output neuron of that layer is propagated backward through network. This calculates the local gradient for each neuron in each layer and allows the synaptic weights of the network to undergo changes in accordance with the delta rule as:
Δw(n) = η * δ(n) * y(n).
This recursive computation is continued, with forward pass followed by the backward pass for each input pattern till the network is converged.
Supervised learning paradigm of an ANN is efficient and finds solutions to several linear and non-linear problems such as classification, plant control, forecasting, prediction, robotics etc.
Unsupervised Learning
Self-Organizing neural networks learn using unsupervised learning algorithm to identify hidden patterns in unlabelled input data. This unsupervised refers to the ability to learn and organize information without providing an error signal to evaluate the potential solution. The lack of direction for the learning algorithm in unsupervised learning can sometime be advantageous, since it lets the algorithm to look back for patterns that have not been previously considered. The main characteristics of Self-Organizing Maps (SOM) are:
It transforms an incoming signal pattern of arbitrary dimension into
one or 2 dimensional map and perform this transformation adaptively
The network represents feedforward structure with a single
computational layer consisting of neurons arranged in rows and
columns. At each stage of representation, each input signal is kept
in its proper context and,
Neurons dealing with closely related pieces of information are close
together and they communicate through synaptic connections.
The computational layer is also called as competitive layer since the neurons in the layer compete with each other to become active. Hence, this learning algorithm is called competitive algorithm. Unsupervised algorithm in SOM
works in three phases:
Competition phase:
for each input pattern x, presented to the network, inner product with synaptic weight w is calculated and the neurons in the competitive layer finds a discriminant function that induce competition among the neurons and the synaptic weight vector that is close to the input vector in the Euclidean distance is announced as winner in the competition. That neuron is called best matching neuron,
i.e. x = arg min ║x - w║.
Cooperative phase:
the winning neuron determines the center of a topological neighborhood h of cooperating neurons. This is performed by the lateral interaction d among the
cooperative neurons. This topological neighborhood reduces its size over a time period.
Adaptive phase:
enables the winning neuron and its neighborhood neurons to increase their individual values of the discriminant function in relation to the input pattern
through suitable synaptic weight adjustments,
Δw = ηh(x)(x –w).
Upon repeated presentation of the training patterns, the synaptic weight vectors tend to follow the distribution of the input patterns due to the neighborhood updating and thus ANN learns without supervisor.
Self-Organizing Model naturally represents the neuro-biological behavior, and hence is used in many real world applications such as clustering, speech recognition, texture segmentation, vector coding etc.
Reference.
There are many answers already which explain the differences in detail. I found these gifs on codeacademy and they often help me explain the differences effectively.
Supervised Learning
Notice that the training images have labels here and that the model is learning the names of the images.
Unsupervised Learning
Notice that what's being done here is just grouping(clustering) and that the model doesn't know anything about any image.
Machine learning:
It explores the study and construction of algorithms that can learn from and make predictions on data.Such algorithms operate by building a model from example inputs in order to make data-driven predictions or decisions expressed as outputs,rather than following strictly static program instructions.
Supervised learning:
It is the machine learning task of inferring a function from labeled training data.The training data consist of a set of training examples. In supervised learning, each example is a pair consisting of an input object (typically a vector) and a desired output value (also called the supervisory signal). A supervised learning algorithm analyzes the training data and produces an inferred function, which can be used for mapping new examples.
The computer is presented with example inputs and their desired outputs, given by a "teacher", and the goal is to learn a general rule that maps inputs to outputs.Specifically, a supervised learning algorithm takes a known set of input data and known responses to the data (output), and trains a model to generate reasonable predictions for the response to new data.
Unsupervised learning:
It is learning without a teacher. One basic
thing that you might want to do with data is to visualize it. It is the machine learning task of inferring a function to describe hidden structure from unlabeled data. Since the examples given to the learner are unlabeled, there is no error or reward signal to evaluate a potential solution. This distinguishes unsupervised learning from supervised learning. Unsupervised learning uses procedures that attempt to find natural partitions
of patterns.
With unsupervised learning there is no feedback based on the prediction results, i.e., there is no teacher to correct you.Under the Unsupervised learning methods no labeled examples are provided and there is no notion of the output during the learning process. As a result, it is up to the learning scheme/model to find patterns or discover the groups of the input data
You should use unsupervised learning methods when you need a large
amount of data to train your models, and the willingness and ability
to experiment and explore, and of course a challenge that isn’t well
solved via more-established methods.With unsupervised learning it is
possible to learn larger and more complex models than with supervised
learning.Here is a good example on it
.
Supervised Learning: You give variously labelled example data as input, along with the correct answers. This algorithm will learn from it, and start predicting correct results based on the inputs thereafter. Example: Email Spam filter
Unsupervised Learning: You just give data and don't tell anything - like labels or correct answers. Algorithm automatically analyses patterns in the data. Example: Google News
Supervised learning:
say a kid goes to kinder-garden. here teacher shows him 3 toys-house,ball and car. now teacher gives him 10 toys.
he will classify them in 3 box of house,ball and car based on his previous experience.
so kid was first supervised by teachers for getting right answers for few sets. then he was tested on unknown toys.
Unsupervised learning:
again kindergarten example.A child is given 10 toys. he is told to segment similar ones.
so based on features like shape,size,color,function etc he will try to make 3 groups say A,B,C and group them.
The word Supervise means you are giving supervision/instruction to machine to help it find answers. Once it learns instructions, it can easily predict for new case.
Unsupervised means there is no supervision or instruction how to find answers/labels and machine will use its intelligence to find some pattern in our data. Here it will not make prediction, it will just try to find clusters which has similar data.
Supervised learning, given the data with an answer.
Given email labeled as spam/not spam, learn a spam filter.
Given a dataset of patients diagnosed as either having diabetes or not, learn to classify new patients as having diabetes or not.
Unsupervised learning, given the data without an answer, let the pc to group things.
Given a set of news articles found on the web, group the into set of articles about the same story.
Given a database of custom data, automatically discover market segments and group customers into different market segments.
Reference
Supervised Learning
In this, every input pattern that is used to train the network is
associated with an output pattern, which is the target or the desired
pattern. A teacher is assumed to be present during the learning
process, when a comparison is made between the network's computed
output and the correct expected output, to determine the error. The
error can then be used to change network parameters, which result in
an improvement in performance.
Unsupervised Learning
In this learning method, the target output is not presented to the
network. It is as if there is no teacher to present the desired
pattern and hence, the system learns of its own by discovering and
adapting to structural features in the input patterns.
I'll try to keep it simple.
Supervised Learning: In this technique of learning, we are given a data set and the system already knows the correct output of the data set. So here, our system learns by predicting a value of its own. Then, it does an accuracy check by using a cost function to check how close its prediction was to the actual output.
Unsupervised Learning: In this approach, we have little or no knowledge of what our result would be. So instead, we derive structure from the data where we don't know effect of variable.
We make structure by clustering the data based on relationship among the variable in data.
Here, we don't have a feedback based on our prediction.
Supervised learning
You have input x and a target output t. So you train the algorithm to generalize to the missing parts. It is supervised because the target is given. You are the supervisor telling the algorithm: For the example x, you should output t!
Unsupervised learning
Although segmentation, clustering and compression are usually counted in this direction, I have a hard time to come up with a good definition for it.
Let's take auto-encoders for compression as an example. While you only have the input x given, it is the human engineer how tells the algorithm that the target is also x. So in some sense, this is not different from supervised learning.
And for clustering and segmentation, I'm not too sure if it really fits the definition of machine learning (see other question).
Supervised Learning: You have labeled data and have to learn from that. e.g house data along with price and then learn to predict price
Unsupervised learning: you have to find the trend and then predict, no prior labels given.
e.g different people in the class and then a new person comes so what group does this new student belong to.
In Supervised Learning we know what the input and output should be. For example , given a set of cars. We have to find out which ones red and which ones blue.
Whereas, Unsupervised learning is where we have to find out the answer with a very little or without any idea about how the output should be. For example, a learner might be able to build a model that detects when people are smiling based on correlation of facial patterns and words such as "what are you smiling about?".
Supervised learning can label a new item into one of the trained labels based on learning during training. You need to provide large numbers of training data set, validation data set and test data set. If you provide say pixel image vectors of digits along with training data with labels, then it can identify the numbers.
Unsupervised learning does not require training data-sets. In unsupervised learning it can group items into different clusters based on the difference in the input vectors. If you provide pixel image vectors of digits and ask it to classify into 10 categories, it may do that. But it does know how to labels it as you have not provided training labels.
Supervised Learning is basically where you have input variables(x) and output variable(y) and use algorithm to learn the mapping function from input to the output. The reason why we called this as supervised is because algorithm learns from the training dataset, the algorithm iteratively makes predictions on the training data.
Supervised have two types-Classification and Regression.
Classification is when the output variable is category like yes/no, true/false.
Regression is when the output is real values like height of person, Temperature etc.
UN supervised learning is where we have only input data(X) and no output variables.
This is called an unsupervised learning because unlike supervised learning above there is no correct answers and there is no teacher. Algorithms are left to their own devises to discover and present the interesting structure in the data.
Types of unsupervised learning are clustering and Association.
Supervised Learning is basically a technique in which the training data from which the machine learns is already labelled that is suppose a simple even odd number classifier where you have already classified the data during training . Therefore it uses "LABELLED" data.
Unsupervised learning on the contrary is a technique in which the machine by itself labels the data . Or you can say its the case when the machine learns by itself from scratch.
In Simple
Supervised learning is type of machine learning problem in which we have some labels and by using that labels we implement algorithm such as regression and classification .Classification is applied where our output is like in the form of
0 or 1 ,true/false,yes/no. and regression is applied where out put a real value such a house of price
Unsupervised Learning is a type of machine learning problem in which we don't have any labels means we have some data only ,unstructured data and we have to cluster the data (grouping of data)using various unsupervised algorithm
Supervised Machine Learning
"The process of an algorithm learning from training dataset and
predict the output. "
Accuracy of predicted output directly proportional to the training data (length)
Supervised learning is where you have input variables (x) (training dataset) and an output variable (Y) (testing dataset) and you use an algorithm to learn the mapping function from the input to the output.
Y = f(X)
Major types:
Classification (discrete y-axis)
Predictive (continuous y-axis)
Algorithms:
Classification Algorithms:
Neural Networks
Naïve Bayes classifiers
Fisher linear discriminant
KNN
Decision Tree
Super Vector Machines
Predictive Algorithms:
Nearest neighbor
Linear Regression,Multi Regression
Application areas:
Classifying emails as spam
Classifying whether patient has
disease or not
Voice Recognition
Predict the HR select particular candidate or not
Predict the stock market price
Supervised learning:
A supervised learning algorithm analyzes the training data and produces an inferred function, which can be used for mapping new examples.
We provide training data and we know correct output for a certain input
We know relation between input and output
Categories of problem:
Regression: Predict results within a continuous output => map input variables to some continuous function.
Example:
Given a picture of a person, predict his age
Classification: Predict results in a discrete output => map input variables into discrete categories
Example:
Is this tumer cancerous?
Unsupervised learning:
Unsupervised learning learns from test data that has not been labeled, classified or categorized. Unsupervised learning identifies commonalities in the data and reacts based on the presence or absence of such commonalities in each new piece of data.
We can derive this structure by clustering the data based on relationships among the variables in the data.
There is no feedback based on the prediction results.
Categories of problem:
Clustering: is the task of grouping a set of objects in such a way that objects in the same group (called a cluster) are more similar (in some sense) to each other than to those in other groups (clusters)
Example:
Take a collection of 1,000,000 different genes, and find a way to automatically group these genes into groups that are somehow similar or related by different variables, such as lifespan, location, roles, and so on.
Popular use cases are listed here.
Difference between classification and clustering in data mining?
References:
Supervised_learning
Unsupervised_learning
machine-learning from coursera
towardsdatascience
Supervised Learning
Unsupervised Learning
Example:
Supervised Learning:
One bag with apple
One bag with orange
=> build model
One mixed bag of apple and orange.
=> Please classify
Unsupervised Learning:
One mixed bag of apple and orange.
=> build model
Another mixed bag
=> Please classify
In simple words.. :) It's my understanding, feel free to correct.
Supervised learning is, we know what we are predicting on the basis of provided data. So we have a column in the dataset which needs to be predicated.
Unsupervised learning is, we try to extract meaning out of the provided dataset. We don't have clarity on what to be predicted. So question is why we do this?.. :) Answer is - the outcome of Unsupervised learning is groups/clusters(similar data together). So if we receive any new data then we associate that with the identified cluster/group and understand it's features.
I hope it will help you.
supervised learning
supervised learning is where we know the output of the raw input, i.e the data is labelled so that during the training of machine learning model it will understand what it need to detect in the give output, and it will guide the system during the training to detect the pre-labelled objects on that basis it will detect the similar objects which we have provided in training.
Here the algorithms will know what's the structure and pattern of data. Supervised learning is used for classification
As an example, we can have a different objects whose shapes are square, circle, trianle our task is to arrange the same types of shapes
the labelled dataset have all the shapes labelled, and we will train the machine learning model on that dataset, on the based of training dateset it will start detecting the shapes.
Un-supervised learning
Unsupervised learning is a unguided learning where the end result is not known, it will cluster the dataset and based on similar properties of the object it will divide the objects on different bunches and detect the objects.
Here algorithms will search for the different pattern in the raw data, and based on that it will cluster the data. Un-supervised learning is used for clustering.
As an example, we can have different objects of multiple shapes square, circle, triangle, so it will make the bunches based on the object properties, if a object has four sides it will consider it square, and if it have three sides triangle and if no sides than circle, here the the data is not labelled, it will learn itself to detect the various shapes
Machine learning is a field where you are trying to make machine to mimic the human behavior.
You train machine just like a baby.The way humans learn, identify features, recognize patterns and train himself, same way you train machine by feeding data with various features. Machine algorithm identify the pattern within the data and classify it into particular category.
Machine learning broadly divided into two category, supervised and unsupervised learning.
Supervised learning is the concept where you have input vector / data with corresponding target value (output).On the other hand unsupervised learning is the concept where you only have input vectors / data without any corresponding target value.
An example of supervised learning is handwritten digits recognition where you have image of digits with corresponding digit [0-9], and an example of unsupervised learning is grouping customers by purchasing behavior.

Resources