I'm new to neural networks/machine learning/genetic algorithms, and for my first implementation I am writing a network that learns to play snake (An example in case you haven't played it before) I have a few questions that I don't fully understand:
Before my questions I just want to make sure I understand the general idea correctly. There is a population of snakes, each with randomly generated DNA. The DNA is the weights used in the neural network. Each time the snake moves, it uses the neural net to decide where to go (using a bias). When the population dies, select some parents (maybe highest fitness), and crossover their DNA with a slight mutation chance.
1) If given the whole board as an input (about 400 spots) enough hidden layers (no idea how many, maybe 256-64-32-2?), and enough time, would it learn to not box itself in?
2) What would be good inputs? Here are some of my ideas:
400 inputs, one for each space on the board. Positive if snake should go there (the apple) and negative if it is a wall/your body. The closer to -1/1 it is the closer it is.
6 inputs: game width, game height, snake x, snake y, apple x, and apple y (may learn to play on different size boards if trained that way, but not sure how to input it's body, since it changes size)
Give it a field of view (maybe 3x3 square in front of head) that can alert the snake of a wall, apple, or it's body. (the snake would only be able to see whats right in front unfortunately, which could hinder it's learning ability)
3) Given the input method, what would be a good starting place for hidden layer sizes (of course plan on tweaking this, just don't know what a good starting place)
4) Finally, the fitness of the snake. Besides time to get the apple, it's length, and it's lifetime, should anything else be factored in? In order to get the snake to learn to not block itself in, is there anything else I could add to the fitness to help that?
Thank you!
In this post, I will advise you of:
How to map navigational instructions to action sequences with an LSTM
neural network
Resources that will help you learn how to use neural
networks to accomplish your task
How to install and configure neural
network libraries based on what I needed to learn the hard way
General opinion of your idea:
I can see what you're trying to do, and I believe that your game idea (of using randomly generated identities of adversaries that control their behavior in a way that randomly alters the way they're using artificial intelligence to behave intelligently) has a lot of potential.
Mapping navigational instructions to action sequences with a neural network
For processing your game board, because it involves dense (as opposed to sparse) data, you could find a Convolutional Neural Network (CNN) to be useful. However, because you need to translate the map to an action sequence, sequence-optimized neural networks (such as Recurrent Neural Networks) will likely be the most useful for you. I did find some studies that use neural networks to map navigational instructions to action sequences, construct the game map, and move a character through a game with many types of inputs:
Mei, H., Bansal, M., & Walter, M. R. (2015). Listen, attend, and walk: Neural mapping of navigational instructions to action sequences. arXiv preprint arXiv:1506.04089. Available at: Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequences
Lample, G., & Chaplot, D. S. (2016). Playing FPS games with deep reinforcement learning. arXiv preprint arXiv:1609.05521. Available at: Super Mario as a String: Platformer Level Generation Via LSTMs
Lample, G., & Chaplot, D. S. (2016). Playing FPS games with deep reinforcement learning. arXiv preprint arXiv:1609.05521. Available at: Playing FPS Games with Deep Reinforcement Learning
Schulz, R., Talbot, B., Lam, O., Dayoub, F., Corke, P., Upcroft, B., & Wyeth, G. (2015, May). Robot navigation using human cues: A robot navigation system for symbolic goal-directed exploration. In Robotics and Automation (ICRA), 2015 IEEE International Conference on (pp. 1100-1105). IEEE. Available at: Robot Navigation Using Human Cues: A robot navigation system for symbolic goal-directed exploration
General opinion of what will help you
It sounds like you're missing some basic understanding of how neural networks work, so my primary recommendation to you is to study more of the underlying mechanics behind neural networks in general. It's important to keep in mind that a neural network is a type of machine learning model. So, it doesn't really make sense to just construct a neural network with random parameters. A neural network is a machine learning model that is trained from sample data, and once it is trained, it can be evaluated on test data (e.g. to perform predictions).
The root of machine learning is largely influenced by Bayesian statistics, so you might benefit from getting a textbook on Bayesian statistics to gain a deeper understanding of how machine-based classification works in general.
It will also be valuable for you to learn the differences between different types of neural networks, such as Long Short Term Memory (LSTM) and Convolutional Neural Networks (CNNs).
If you want to tinker with how neural networks can be used for classification tasks, try this:
Tensorflow Playground
To learn the math:
My professional opinion is that learning the underlying math of neural networks is very important. If it's intimidating, I give you my testimony that I was able to learn all of it on my own. But if you prefer learning in a classroom environment, then I recommend that you try that. A great resource and textbook for learning the mechanics and mathematics of neural networks is:
Neural Networks and Deep Learning
Tutorials for neural network libraries
I recommend that you try working through the tutorials for a neural network library, such as:
TensorFlow tutorials
Deep Learning tutorials with Theano
CNTK tutorials (CNTK 205: Artistic Style Transfer is particularly cool.)
Keras tutorial (Keras is a powerful high-level neural network library that can use either TensorFlow or Theano.)
I saw similar application. Inputs usually were snake coordinates, apple coordinates and some sensory data(is wall next to snake head or no in your case).
Using genetic algorithm is a good idea in this case. You doing only parametric learning(finding set of weights), but structure will be based on your estimation. GA can be also used for structure learning(finding topology of ANN). But using GA for both will be very computational hard.
Professor Floreano did something similar. He use GA for finding weights for neural network controller of robot. Robot was in labyrinth and perform some task. Neural network hidden layer was one neuron with recurrent joints on inputs and one lateral connection on himself. There was two outputs. Outputs were connected on input layer and hidden layer(mentioned one neuron).
But Floreano did something more interesting. He say, We don't born with determined synapses, our synapses change in our lifetime. So he use GA for finding rules for change of synapses. These rules was based on Hebbian learning. He perform node encoding(for all weights connected to neuron will apply same rule). On beginning, he initialized weights on small random values. Finding rules instead of numerical value of synapse leads to better results.
One from Floreno's articles.
And on the and my own experience. In last semester I and my schoolmate get a task finding the rules for synapse with GA but for Spiking neural network. Our SNN was controller for kinematic model of mobile robot and task was lead robot in to the chosen point. We obtained some results but not expected. You can see results here. So I recommend you use "ordinary" ANN instead off SNN because SNN brings new phenomens.
I know about supervised and unsupervised learning but still not getting how Reinforcement machine learning works.
can somebody help me with proper example ? and use cases that how it works ?
Reinforcement machine learning is when the machine learns from experience, where the feedback is "good" or "bad".
A classic example is when training agents for games. You first start training your agent with the data you have (supervised), and when it is exhausted, start training several agents and let the compete each other. Those who win are getting "reinforced", and go on.
This was one of the "tricks" used to train AlphaGo, (and previously in TD-Gammon)
...
The policy networks were therefore
improved by letting them play against each other, using the outcome of
these games as a training signal. This is called reinforcement
learning, or even deep reinforcement learning (because the networks
being trained are deep).
You mentioned about supervised and unsupervised learning.
There is a slight difference in these 3.
Supervised learning: You have label for each tuple of data.
Unsupervised learning: You don't have label for tuples but you want to find relations between inputs
Reinforcement leaning: You have very few labels for sparse entries. that label is rewards.
reinforcement learning is a process how a person learns about a new situation. it takes any random action, observe the behavior of the environment and learns accordingly.
What is a reward.?
a reward is positive or negative feedback from the environment. An action is responsible for all its future rewards. hence it need to take those action which can achieve most positive reward in future.
This can be achieve by Q-learning algorithm. i request you to check about this topic.
I used reinforcement algorithm to train pacman. i hope you know the game. the goal is to take action by which it should not hit the ghosts and also should be able to take all points from the map. it train itself after many iterations and thousands of gameplay. I also used same to train a car to drive on a specific track!
The reinforcement learning can be used to train an AI to learn any game.! Though more complex games require Neural Networks, and that is called Deep learning.
Reinforcement learning is a type of model that is rewarded for doing good (or bad) things. With supervised learning, it is up to some curator to label all the data that the model can learn from. That is the beauty of reinforcement learning, the model obtains direct feedback from its environment and adjusts its behavior automatically. It's how human's learn a lot of our simple life lessons (e.g., avoid things that hurt you, do more of things that make you feel good)
A lot of reinforcement learning is focused around deep learning these days and the biggest examples have been about video games. Reinforcement learning is also a powerful personalization tool. You can think of an amazon recommender as a reinforcement learning algorithm that is rewarded when it recommends the right products by receiving a click or purchase, or a netflix recommender is rewarded when a user starts watching a movie.
Reinforcement learning is often used for robotics, gaming, and navigation.
With reinforcement learning, the algorithm discovers through trial and error which actions yield the greatest rewards.
This type of learning has three primary components: the agent (the learner or decision-maker), the environment (everything the agent interacts with) and actions (what the agent can do).
The objective is for the agent to choose actions that maximize the expected reward over a given amount of time.
The agent will reach the goal much faster by following a good policy. So the goal in reinforcement learning is to learn the best policy.
As a beginner in ML and AI, I have come across ANN, RNN and LSTMs, however I would like to know what is the classification among neural networks ranging from the simplest single perceptron feedforward network to the most complicated one.
Maybe you'll be interested in looking at the neural network zoo. It's not a hierarchical classification of all the different neural network types, but it does graphically show a lot of types and also provides short descriptions of them (and also some other models that are not typically considered to be neural networks). I haven't read through it all in detail, so I can't personally vouch for the page's correctness, but it looks good.
In Wikipedia I found that machine learning is a subsection of neural networks science. So, does it mean that work with machine learning is itself implies working with neural networks or not?
What will be better to use for pattern recognition tasks in terms of efficiency and complexity?
Machine learning is a part of neural network? I'd be surprised because machine learning includes dozen of techniques that have nothing to do with neural network. It's most likely the other way around.
The exact pattern recognition algorithm depends on your requirement and data set. There're many such algorithms, for example, SVM, linear models for classification, HMM, PCA etc. Note that the phase "pattern recognition" is a very general term, there is no algorithm that always work. It all depends on what patterns you are looking and what kind of assumption you can make.
I recommend Dr Bishop's "Pattern Recognition and Machine Learning" book, you'll learn a lot from the book.
I am planning to create a Tetris AI using artificial neural network and train it with genetic algorithm for a project in my high school computer science class. I have a basic understanding of how an ANN works and how to implement it with a genetic algorithm. I have already written a working Neural Network based on this tutorial and I'm currently working on a genetic algorithm.
My questions are:
Which GA model is better for this situation (Tetris), and why?
What should I use for input for the neural network? Because currently, the method I'm using is to simply convert the state of the board (the pieces) into a one dimensional array and feed it into the neural network? Is there a better approach?
What should the size (number of layers, neurons per layer) the neural network be?
Are there any good sources of information that can help me?
Thank you!
Similar task was already solved by Google, but they solved it for all kinds of Atari games - https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf.
Carefully read this article and all of the related articles too
This is a reinforcement learning task, in my opinion the hardest task in ML domain. So there will be no short answer for your questions - except that probably you shouldn't use GA heuristic at all and rely on reinforcements methods.