how to expand spinner time? [closed] - ruby-on-rails

It's difficult to tell what is being asked here. This question is ambiguous, vague, incomplete, overly broad, or rhetorical and cannot be reasonably answered in its current form. For help clarifying this question so that it can be reopened, visit the help center.
Closed 12 years ago.
how to expand spinner time ?

From The Nature of Time:
The Nature of Time
We have all been asked the question
'what is the time?', but this entry
will be addressing the subtly
different question: 'what is time?'
Many people, on being asked this
question, would probably say that they
don't have the time to answer. In this
entry, we will explore many theories
about the nature of time. No one
theory has unquestioning truth about
it, so it will be up to you to decide
on the one you see is best.
The Classical Approach to Time
There is not very much to say on this
theory since it is the one with which
we are most familiar. Traditionally,
time is simply seen as a measure of
the distance between events. It has a
past, present and a future. The past
is considered to have already happened
and to be unchangeable, while the
future is considered to be open to
many possibilities. Humans measure
time using many units, some based on
real events like the rotation of the
Earth, others that are even more
arbitrary.
Isaac Newton's classical description
of time in his highly-regarded work
Principia is that it 'flows equably of
itself', which means that time 'flows'
at a constant rate that is the same
for everybody - it is independent of
the events that take place in it. It
would be untrue to say that this idea
was unchallenged until the twentieth
century - the 18th-Century empiricist
philosopher George Berkeley, for
example, disagreed with Newton and
held that time was 'the succession of
ideas in the mind' - but there was no
serious evidence to suggest that
Newton's elegant and absolute
description was wrong until Einstein
destroyed it.
Unfortunately, the classical view of
time is biased towards the human
perception of the 'flow' of time. We
see events in one direction, and we
assume time to be the same everywhere.
The classical approach to time does
not explain exactly why we perceive
time in this way, and it does not
describe how the effect is achieved.
The other theories of the nature of
time challenge the very roots of this
natural point of view.
Relativity
The Theory of Relativity is the
celebrated discovery of the physicist
Albert Einstein. Originally, it was
two theories: the Special Theory of
Relativity came first in 1905 and
states that the rate at which time
passes is not the same all over the
universe - it is dependent on the
observer (in other words, it is
relative). It is not hard to see that
different people perceive the passing
of time at a different rate to others:
as we get older, less information is
processed about our surroundings per
second, so we perceive time to be
going faster.
But Einstein's theory went further
than this. The relativity of time is
based not on our age, but on our speed
of movement through space. The faster
we travel through space, the slower we
travel through time. Although this
sounds crazy at first, it makes sense
when thought of in a particular way.
The theory of relativity demands that
we view space and time not as separate
entities but as one concept called
space-time. Time becomes a fourth
dimension, just like the other three
dimensions of space that we are used
to (height, width and length). This
view of time is crucial to
understanding most of the other
theories about time's ultimate nature.
Humans only possess two-dimensional
retinae (the light-receptive surface
at the back of our eyes), which means
that we can only see in two
dimensions. Our vision of the third
dimension is a result of perspective
and the existence of our binocular
vision. If we had three-dimensional
retinae, we would be able to see all
of an entire room simultaneously - its
walls, its floor and its ceiling at
the same time! For this reason, it is
very difficult, if not totally
impossible, for humans to visualise a
fourth dimension.
To overcome this impairment, it is
useful to use lower-dimensional
analogies when talking about
dimensions above three, even if we are
talking about time as one of these
dimensions. So in this case, let us
imagine that the universe is shaped
like a cuboid1, and that humans are
two-dimensional and have
one-dimensional retinae. Imagine that
the spatial dimensions are the width
and height of a cross-section of the
cuboid, meaning that humans can move
up, down, left and right at will
within the cuboid. Imagine that the
depth of the cuboid is time.
Right, now imagine that you are a
two-dimensional human within the
cuboid and that you start off being
midway up the cuboid. Then you start
moving upward (ie, through space, but
not time). Eventually you hit the edge
of the cuboid. Now imagine that you
move downwards, but that you also move
through time in a forward direction.
This time it will take you longer to
get back to being mid-way up the
cuboid because you are not taking a
direct route downwards - you are also
moving through time. As you can see,
moving through time slows down your
movement in space.
It works the other way around too. If
you stay still in space and just move
forward in time, then it will take
less time to get to a particular point
in time than if you move upwards and
forwards in time simultaneously. So
movement in space slows down your
movement in time. This is what
relativity states about how time
really is. However, the amount by
which time is slowed down when you
move through space is very small in
everyday situations, and you would
need to move at a speed of a
considerable percentage the speed of
light in order for it to make any
noticeable difference.
Relativity has been proven too. Atomic
clocks2 have been placed in aeroplanes
moving at high speeds and then
compared with clocks that were on
Earth. Slight differences that are
exactly predicted by the mathematical
equations of relativity were indeed
detected.
The general theory of relativity goes
a step further and was published in
1916. Einstein stated that mass curves the 'fabric' of space-time to create
the illusion of the force of gravity.
Again, a lower-dimensional analogy is
best. Imagine putting bowling balls on
a sheet of rubber. They bend the
rubber. Any object coming into the
vicinity of the curve begins to move
around the curve like something
spiralling around a sink basin.
Einstein's picture of gravity is that
simple. And again, this has been
proved. Einstein made predictions
about how light would be taking
technically curved paths around large
masses, and this effect was measured
during a total eclipse of the sun.
Time and Determinism
You will have noticed that the theory
of relativity does not carry any
description of a 'flow' of time, and
in fact, it describes time in almost
exactly the same way that we are used
to thinking about space. Relativity
unifies space and time. All points in
space are in existence simultaneously
- this is common sense; so are all points in time in existence
simultaneously too? This would suggest
that all events in time are already
'here' and that there is no scope for
choosing what happens in the future.
This view of time is called
determinism because events are
pre-determined.
It is worth noting that relativity
does not rule out the idea of free
will, but does not provide any support
for it either. Many people can get
upset about the evidence supporting
determinism because humans like to
think they have a free will to make
independent decisions. Such people
would not feel better if they heard
about the many worlds theory of
quantum mechanics.
Time in the Many Worlds Theory of
Quantum Mechanics
To understand this theory, we need to
go back to our cuboid example. You
will notice that each event in time is
simply a cross-section of the cuboid
(a square). Humans effectively
perceive the dimension of time in this
cuboid to be a succession of these
squares. Like frames in a movie, these
create the illusion of a smooth
passage of time. But why is it that we
see time like this? The answer to this
question will be explored later.
If you think about the world around
you, you will most likely notice that
it seems to have been tailor-made for
life. The universe has the precise
properties that led to the formation
of life on Earth. For example, in the
early universe there was a 'battle'
between matter and anti-matter3. The
particles with the certain quantum
properties that we now characterise as
'matter', for a hitherto inexplicable
reason, won the battle. If this hadn't
happened, we could not exist, or we
would not be the same as we are today.
Many physicists have speculated that
this and other similar events are too
much of a coincidence to be regarded
as just that: a coincidence.
Martin Rees, the Astronomer Royal of
the UK, paints an analogous picture of
going into a clothes shop. If you go
into a clothes shop that only sells
one size of clothing, it would be a
big coincidence if you found it was
your size. However, we get no surprise
when finding our own clothes size in a
clothes shop because good clothes
shops sell a wide range of clothes
sizes. We can now extend this picture
to the universe. It is very unlikely
that the universe should exist because
of how biased it seems to have been
towards gravitational stability and
the creation of diverse life later on.
However, if we see the universe as
providing a wide range of 'universes'
of different properties, it will come
as no surprise if we find one universe
that supports life.
You can think of this theory as
multiple cuboids in a vast universe of
cuboids, all with their own
space-time. Each cuboid represents one
universe that has a different set of
laws of physics, and therefore could
be wildly different from all the other
universes. There may in fact be a
large number of universes that support
life but with small differences, just
as there might be many shirts of your
size in the clothes shop, but perhaps
in different colours.
In this view, there are multiple
timelines. Some people have likened
this view of time to train tracks. We
move along a train track in one
direction, but there are huge numbers
of other train tracks running parallel
to ours. Each train track may be
different in some way (it might have
trains on it, for example). For this
reason, the other universes around us
in this 'multiverse'4 are referred to
as parallel universes.
A multiverse of space-times is not
just a theory that solves the question
of why our environment is so suited to
life; it is also a theory of quantum
mechanics. In the quantum theory there
are many events that take place
because of random chance. In electric
currents, for example, the electrons
that make up the current follow a
random path in the wires that is
influenced by a fields of electrical
forces they pass through, which is why
it always seems that the current is
split 50:50. Many physicists believe
that with each quantum decision like
this, every possibility has a separate
universe in which it is enacted.
Hence, in one universe the electron
goes one way; in another, it goes the
other way.
In this theory - which is called the
many worlds interpretation of quantum
mechanics - every possibility gets
enacted. Since quantum interactions
are the fundamentals of any larger (or
'macroscopic') reaction, we can infer
that everything happens in one
universe or other. So if you have a
decision to make, say whether to take
a holiday to Hawaii or not, there is
one universe where you go, and one
universe where you don't. This also
spells trouble for free will. All
possibilities get played out, so it is
just a matter of which universe you
are in to determine which way you go.
There is a variation of this theory.
For this variation we will need to
think another dimension lower. So,
instead of imagining universes as
cuboids, we need to imagine them as
rectangles. Imagine the length of the
rectangle is time; and its other
dimension, space. The rectangle has no
thickness whatsoever, so if you put
multiple rectangles (ie, multiple
universes) on top of each other, the
whole structure becomes no thicker.
This version of the many worlds
interpretation is slightly easier to
grasp, because otherwise we would have
universes branching off from one
another to eternity, which is rather
difficult to imagine. There is no real
evidence for or against either of the
theories of the multiverse,
unfortunately.
You will have noticed that in all
these theories, time has two
directions just like all the other
dimensions. In theory, there is
nothing to stop us from moving in the
other direction. There is another
slightly different theory of time as
being bi-directional, and you might
also be interested to see how this
could lead to possibilities of
time-travel.
Why Do We Perceive Time the Way We Do?
What, then, is time? If no one asks
me, I know what it is. If I wish to
explain it to him who asks me, I do
not know.
- St Augustine. If time is a dimension just like all the others,
why do we experience it so
differently? This is the question that
interests James Hartle of the
University of California in Santa
Barbara, along with physicists Stephen
Hawking, Murray Gell-Mann and Steven
Weinberg. They believe that the
passage of time is just an illusion.
Hartle thinks that time's arrow is a
product of the way we process
information. Gell-Mann gave creatures
that process time in this way the name
'information gathering and utilising
systems' (IGUSs). Humans are IGUSs.
Because of our two-dimensional
retinae, we can't take in multiple
cross-sections of the 'cuboid' - ie
'frames' of time - simultaneously. We
gather information about this frame -
our surrounding environment - using
our senses, and then we store the
information in an input register. This
does not have an unlimited capacity,
so we have to transfer the information
to our memory registers before we can
input the information about the next
frame. Humans have a short-term and a
long-term memory, as well as our
cerebellums that store 'unforgettable'
information (such as how to swim).
IGUSs also carry something called a
'schema', which is a generalised model
of our perception of our environment.
It holds several rules about what is
best to do and what is not a good idea
to do. The information we receive from
our surroundings is passed to the
schema to determine how we react in
certain situations. The decision is
conscious, but we also do unconscious
computation of information: the schema
is updated unconsciously. The
conscious part of the IGUS in humans
focuses on the input register, which
we call the present. The unconscious
part focuses on information in the
memories, and we call that the past.
This is why we consciously experience
the present and remember the past.
The movement of the information
through the IGUSs registers creates
the illusion of the flow of time. It
is not time itself that flows. Each
IGUS has a different speed for the
flow of its information between
registers. This corresponds to
differences between the perception of
the speed of the flow of time. Flies,
for example, need to process more
information per second in order to fly
so quickly but still avoid common
obstacles; therefore, they perceive
time as going slower. To us, a fly's
perception of time would look like
slow motion. Flies only live for a few
days, or a few weeks as a maximum, and
to a human, this is a very short
lifetime. But to a fly, this feels a
lot longer.
So the reason that we experience a
'flow' of time could just be because
of how we process information. It is a
competitive advantage to us as a
species to process information bits at
a time. It wouldn't make sense for us
to have evolved with the capability to
see all time simultaneously.
Digital Time, or, Is Time Like a
Movie?
You may have noticed a reference to
'frames' of time in the explanations
above. We usually think of time as
continuous - a smooth passage of
events. However, most physical
theories define space and time as
being the opposite of a continuous
passage of events. M-theory and Loop
Quantum Gravity, for example, are both
serious scientific theories (not
proven theories, though) that state
that space and time have minimum
units. There was even a theory of
quantum mechanics to suggest that time
was made of particles called
'chronons'!
The theorised minimum length of time
possible is called the Planck time and
is equivalent to 10-43 seconds. When
space or time is 'digital' like this,
we say that it is 'discrete'.
If this theory is true, then our
perception of time could be like a
movie. Movies are not continuous: if
you slow them down enough, you see
that they are just collections of
still photographs played in quick
succession. We process information
about our surroundings and obtain a
picture just like one frame of a movie
or animation. When 'played' in quick
succession, this creates the illusion
of smooth, continuous movement.
Is Time Really That Much Like Space?
So far, time has been seen as very
much like a dimension of space, and
its passage in one direction has been
seen as an illusion. But there are
some counter-arguments; there are
still some big differences between
time and space that cannot easily be
explained as illusions.
One way of supporting the idea that an
'arrow of time' is irrelevant is by
proving that all processes are the
same if done forwards or backwards. In
quantum mechanics, most interactions
between particles are 'time-symmetric'
- it doesn't matter whether you look at them from past to future or future
to past because they look the same.
But this is not true of macroscopic
objects. Wine glasses shatter, but you
rarely see shards of glass assemble
themselves into wine glasses.
Physicists can explain why shards of
glass do not form wine glasses by
postulating the existence of 'the
thermodynamic arrow of time'.
Thermodynamics is basically a
collection of laws. Here is how the
chemist PW Atkins summarises them:
There are four laws. The third of
them, the Second Law, was recognised
first; the first, the Zeroth law, was
formulated last; the First Law was
second; the Third Law might not even
be a law in the same sense as the
others. The gist of it is that the
universe is always becoming more
disordered. The disorder of the
universe is called 'entropy', so we
say that entropy is always increasing.
Nobody really knows why this is the
case, but we see it all the time in
experiments. This is why heat always
flows into colder areas, but never the
other way round. Heat is simply the
result of giving particles in a given
system more energy; they begin to move
and vibrate randomly, which is a
disordered state. Colder things are
more ordered because their constituent
particles tend to be harder to move.
This in-built arrow explains why
macroscopic objects have irreversible
interactions. This is a clear
difference from space. If you think of
the spatial manifestation of a table,
it does not follow that one end of the
table is more disordered than the
other, but it does follow that the
table will end up more disordered in
the future than when it has just been
made. Hence, there is a very distinct
difference between time and space.
Can Time Be Reversed?
If time's 'flow' in one direction
really is an illusion, what is there
stopping us from reversing it? In
theory, nothing! Lawrence Schulman of
Clarkson University in New York
thoroughly believes that time can run
backwards. In other words, shards of
glass can turn into wine glasses,
people grow younger and younger and
the universe gets smaller and smaller.
In fact, Schulman goes as far as to
say that such reversed-time zones can
exist as spaces within our own
universe. A computer simulation has
shown that regions with opposite time
arrows do not cancel each other out
and do not disturb each other at all.
The great thing about this theory is
that if a civilisation in a
reversed-time region kept records of
events that occur in our future, the
records might have survived to our
past (which is their future). Finding
these records could tell us the
future. This is, of course, a long
shot, but still a physical
possibility.
Another possibility is that the
universe's arrow of time (as far as
thermodynamics is concerned) will
naturally reverse itself at a crucial
point in the history of the universe.
At this point, the universe would
start to get smaller and everybody
would get younger until there was a
big crunch analogous to the big bang.
This creates a perfect symmetry to the
universe.
Again, there is little evidence that
shows us that reversed-time regions
exist, and there is no evidence that
the universe's thermodynamic arrow of
time will naturally reverse itself.
Equally, there is little evidence
against these theories either.
So what is time? Is it a dimension
just like space? Does it flow, or is
that just an illusion? Is time digital
like the frames of a movie, or does it
flow continuously? And can time really
be reversed or manipulated? None of
these questions can be answered with
definite confidence, but next time
somebody asks you what the time is,
perhaps you'll think of the answer
differently.

Kernel::sleep ?

Related

Methods to Find 'Best' Cut-Off Point for a Continuous Target Variable

I am working on a machine learning scenario where the target variable is Duration of power outages.
The distribution of the target variable is severely skewed right (You can imagine most power outages occur and are over with fairly quick, but then there are many, many outliers that can last much longer) A lot of these power outages become less and less 'explainable' by data as the durations get longer and longer. They become more or less, 'unique outages', where events are occurring on site that are not necessarily 'typical' of other outages nor is data recorded on the specifics of those events outside of what's already available for all other 'typical' outages.
This causes a problem when creating models. This unexplainable data mingles in with the explainable parts and skews the models ability to predict as well.
I analyzed some percentiles to decide on a point that I considered to encompass as many outages as possible while I still believed that the duration was going to be mostly explainable. This was somewhere around the 320 minute mark and contained about 90% of the outages.
This was completely subjective to my opinion though and I know there has to be some kind of procedure in order to determine a 'best' cut-off point for this target variable. Ideally, I would like this procedure to be robust enough to consider the trade-off of encompassing as much data as possible and not telling me to make my cut-off 2 hours and thus cutting out a significant amount of customers as the purpose of this is to provide an accurate Estimated Restoration Time to as many customers as possible.
FYI: The methods of modeling I am using that appear to be working the best right now are random forests and conditional random forests. Methods I have used in this scenario include multiple linear regression, decision trees, random forests, and conditional random forests. MLR was by far the least effective. :(
I have exactly the same problem! I hope someone more informed brings his knowledge. I wander to what point is a long duration something that we want to discard or that we want to predict!
Also, I tried treating my data by log transforming it, and the density plot shows a funny artifact on the left side of the distribution ( because I only have durations of integer numbers, not floats). I think this helps, you also should log transform the features that have similar distributions.
I finally thought that the solution should be stratified sampling or giving weights to features, but I don't know exactly how to implement that. My tries didn't produce any good results. Perhaps my data is too stochastic!

direct or indirect for the types of training exprience

I have a question,
in machine learning we define 2 types for the type of training experience:
Direct and indirect.
I searched a lot about the difference but I coud not find. Is anyone familiar with these?
Thank you in advance
In his book "Machine Learning" (1st ed.), Tom Mitchell explains this as follows (see section 1.2.1, page 5):
For example, in learning to play checkers, the system might learn from direct training examples consisting of individual checkers board states and the correct move for each. Alternatively, it might have available only indirect information consisting of the move sequences and final outcomes of various games played. In this later case, information about the correctness of specific moves early in the game must be inferred indirectly from the fact that the game was eventually won or lost.
He further states:
Here [using indirect feedback] the learner faces an additional problem of credit assignment, or determining the degree to which each move in the sequence deserves credit or blame for the final outcome. Credit assignment can be a particularly difficult problem because the game can be lost even when early moves are optimal, if these are followed later by poor moves. Hence, learning from direct training feedback is typically easier than learning from indirect feedback.
Considering the example Playing Chess
Direct experience : Learning the Rules of playing chess. Learning the different moves of different coins. Example : elephant moves only straight, soldier moves only one step at a time etc.
Indirect experience :
Learning from the previous experience, if particular movement of the coin in the game has led the to win, this movement will be assigned some reward or credit, if particular movement has led to loose game, penalty is assigned. This experience of the previous game is used to make coin movement against particular step, so that game can be won.

How to train an artificial neural network to play Diablo 2 using visual input?

I'm currently trying to get an ANN to play a video game and and I was hoping to get some help from the wonderful community here.
I've settled on Diablo 2. Game play is thus in real-time and from an isometric viewpoint, with the player controlling a single avatar whom the camera is centered on.
To make things concrete, the task is to get your character x experience points without having its health drop to 0, where experience point are gained through killing monsters. Here is an example of the gameplay:
Now, since I want the net to operate based solely on the information it gets from the pixels on the screen, it must learn a very rich representation in order to play efficiently, since this would presumably require it to know (implicitly at least) how divide the game world up into objects and how to interact with them.
And all of this information must be taught to the net somehow. I can't for the life of me think of how to train this thing. My only idea is have a separate program visually extract something innately good/bad in the game (e.g. health, gold, experience) from the screen, and then use that stat in a reinforcement learning procedure. I think that will be part of the answer, but I don't think it'll be enough; there are just too many levels of abstraction from raw visual input to goal-oriented behavior for such limited feedback to train a net within my lifetime.
So, my question: what other ways can you think of to train a net to do at least some part of this task? preferably without making thousands of labeled examples.
Just for a little more direction: I'm looking for some other sources of reinforcement learning and/or any unsupervised methods for extracting useful information in this setting. Or a supervised algorithm if you can think of a way of getting labeled data out of a game world without having to manually label it.
UPDATE(04/27/12):
Strangely, I'm still working on this and seem to be making progress. The biggest secret to getting a ANN controller to work is to use the most advanced ANN architectures appropriate to the task. Hence I've been using a deep belief net composed of factored conditional restricted Boltzmann machines that I've trained in an unsupervised manner (on video of me playing the game) before fine tuning with temporal difference back-propagation (i.e. reinforcement learning with standard feed-forward ANNs).
Still looking for more valuable input though, especially on the problem of action selection in real-time and how to encode color images for ANN processing :-)
UPDATE(10/21/15):
Just remembered I asked this question back-in-the-day, and thought I should mention that this is no longer a crazy idea. Since my last update, DeepMind published their nature paper on getting neural networks to play Atari games from visual inputs. Indeed, the only thing preventing me from using their architecture to play, a limited subset, of Diablo 2 is the lack of access to the underlying game engine. Rendering to the screen and then redirecting it to the network is just far too slow to train in a reasonable amount of time. Thus we probably won't see this sort of bot playing Diablo 2 anytime soon, but only because it'll be playing something either open-source or with API access to the rendering target. (Quake perhaps?)
I can see that you are worried about how to train the ANN, but this project hides a complexity that you might not be aware of. Object/character recognition on computer games through image processing it's a highly challenging task (not say crazy for FPS and RPG games). I don't doubt of your skills and I'm also not saying it can't be done, but you can easily spend 10x more time working on recognizing stuff than implementing the ANN itself (assuming you already have experience with digital image processing techniques).
I think your idea is very interesting and also very ambitious. At this point you might want to reconsider it. I sense that this project is something you are planning for the university, so if the focus of the work is really ANN you should probably pick another game, something more simple.
I remember that someone else came looking for tips on a different but somehow similar project not too long ago. It's worth checking it out.
On the other hand, there might be better/easier approaches for identifying objects in-game if you're accepting suggestions. But first, let's call this project for what you want it to be: a smart-bot.
One method for implementing bots accesses the memory of the game client to find relevant information, such as the location of the character on the screen and it's health. Reading computer memory is trivial, but figuring out exactly where in memory to look for is not. Memory scanners like Cheat Engine can be very helpful for this.
Another method, which works under the game, involves manipulating rendering information. All objects of the game must be rendered to the screen. This means that the locations of all 3D objects will eventually be sent to the video card for processing. Be ready for some serious debugging.
In this answer I briefly described 2 methods to accomplish what you want through image processing. If you are interested in them you can find more about them on Exploiting Online Games (chapter 6), an excellent book on the subject.
UPDATE 2018-07-26: That's it! We are now approaching the point where this kind of game will be solvable! Using OpenAI and based on the game DotA 2, a team could make an AI that can beat semi-professional gamers in a 5v5 game. If you know DotA 2, you know this game is quite similar to Diablo-like games in terms of mechanics, but one could argue that it is even more complicated because of the team play.
As expected, this was achieved thanks to the latest advances in reinforcement learning with deep learning, and using open game frameworks like OpenAI which eases the development of an AI since you get a neat API and also because you can accelerate the game (the AI played the equivalent of 180 years of gameplay against itself everyday!).
On the 5th of August 2018 (in 10 days!), it is planned to pit this AI against top DotA 2 gamers. If this works out, expect a big revolution, maybe not as mediatized as the solving of the Go game, but it will nonetheless be a huge milestone for games AI!
UPDATE 2017-01: The field is moving very fast since AlphaGo's success, and there are new frameworks to facilitate the development of machine learning algorithms on games almost every months. Here is a list of the latest ones I've found:
OpenAI's Universe: a platform to play virtually any game using machine learning. The API is in Python, and it runs the games behind a VNC remote desktop environment, so it can capture the images of any game! You can probably use Universe to play Diablo II through a machine learning algorithm!
OpenAI's Gym: Similar to Universe but targeting reinforcement learning algorithms specifically (so it's kind of a generalization of the framework used by AlphaGo but to a lot more games). There is a course on Udemy covering the application of machine learning to games like breakout or Doom using OpenAI Gym.
TorchCraft: a bridge between Torch (machine learning framework) and StarCraft: Brood War.
pyGTA5: a project to build self-driving cars in GTA5 using only screen captures (with lots of videos online).
Very exciting times!
IMPORTANT UPDATE (2016-06): As noted by OP, this problem of training artificial networks to play games using only visual inputs is now being tackled by several serious institutions, with quite promising results, such as DeepMind Deep-Qlearning-Network (DQN).
And now, if you want to get to take on the next level challenge, you can use one of the various AI vision game development platforms such as ViZDoom, a highly optimized platform (7000 fps) to train networks to play Doom using only visual inputs:
ViZDoom allows developing AI bots that play Doom using only the visual information (the screen buffer). It is primarily intended for research in machine visual learning, and deep reinforcement learning, in particular.
ViZDoom is based on ZDoom to provide the game mechanics.
And the results are quite amazing, see the videos on their webpage and the nice tutorial (in Python) here!
There is also a similar project for Quake 3 Arena, called Quagents, which also provides easy API access to underlying game data, but you can scrap it and just use screenshots and the API only to control your agent.
Why is such a platform useful if we only use screenshots? Even if you don't access underlying game data, such a platform provide:
high performance implementation of games (you can generate more data/plays/learning generations with less time so that your learning algorithms can converge faster!).
a simple and responsive API to control your agents (ie, if you try to use human inputs to control a game, some of your commands may be lost, so you'd also deal with unreliability of your outputs...).
easy setup of custom scenarios.
customizable rendering (can be useful to "simplify" the images you get to ease processing)
synchronized ("turn-by-turn") play (so you don't need your algorithm to work in realtime at first, that's a huge complexity reduction).
additional convenience features such as crossplatform compatibility, retrocompatibility (you don't risk your bot not working with the game anymore when there is a new game update), etc.
To summarize, the great thing about these platforms is that they alleviate much of the previous technical issues you had to deal with (how to manipulate game inputs, how to setup scenarios, etc.) so that you just have to deal with the learning algorithm itself.
So now, get to work and make us the best AI visual bot ever ;)
Old post describing the technical issues of developping an AI relying only on visual inputs:
Contrary to some of my colleagues above, I do not think this problem is intractable. But it surely is a hella hard one!
The first problem as pointed out above is that of the representation of the state of the game: you can't represent the full state with just a single image, you need to maintain some kind of memorization (health but also objects equipped and items available to use, quests and goals, etc.). To fetch such informations you have two ways: either by directly accessing the game data, which is the most reliable and easy; or either you can create an abstract representation of these informations by implementing some simple procedures (open inventory, take a screenshot, extract the data). Of course, extracting data from a screenshot will either have you to put in some supervised procedure (that you define completely) or unsupervised (via a machine learning algorithm, but then it'll scale up a lot the complexity...). For unsupervised machine learning, you will need to use a quite recent kind of algorithms called structural learning algorithms (which learn the structure of data rather than how to classify them or predict a value). One such algorithm is the Recursive Neural Network (not to confuse with Recurrent Neural Network) by Richard Socher: http://techtalks.tv/talks/54422/
Then, another problem is that even when you have fetched all the data you need, the game is only partially observable. Thus you need to inject an abstract model of the world and feed it with processed information from the game, for example the location of your avatar, but also the location of quest items, goals and enemies outside the screen. You may maybe look into Mixture Particle Filters by Vermaak 2003 for this.
Also, you need to have an autonomous agent, with goals dynamically generated. A well-known architecture you can try is BDI agent, but you will probably have to tweak it for this architecture to work in your practical case. As an alternative, there is also the Recursive Petri Net, which you can probably combine with all kinds of variations of the petri nets to achieve what you want since it is a very well studied and flexible framework, with great formalization and proofs procedures.
And at last, even if you do all the above, you will need to find a way to emulate the game in accelerated speed (using a video may be nice, but the problem is that your algorithm will only spectate without control, and being able to try for itself is very important for learning). Indeed, it is well-known that current state-of-the-art algorithm takes a lot more time to learn the same thing a human can learn (even more so with reinforcement learning), thus if can't speed up the process (ie, if you can't speed up the game time), your algorithm won't even converge in a single lifetime...
To conclude, what you want to achieve here is at the limit (and maybe a bit beyond) of current state-of-the-art algorithms. I think it may be possible, but even if it is, you are going to spend a hella lot of time, because this is not a theoretical problem but a practical problem you are approaching here, and thus you need to implement and combine a lot of different AI approaches in order to solve it.
Several decades of research with a whole team working on it would may not suffice, so if you are alone and working on it in part-time (as you probably have a job for a living) you may spend a whole lifetime without reaching anywhere near a working solution.
So my most important advice here would be that you lower down your expectations, and try to reduce the complexity of your problem by using all the information you can, and avoid as much as possible relying on screenshots (ie, try to hook directly into the game, look for DLL injection), and simplify some problems by implementing supervised procedures, do not let your algorithm learn everything (ie, drop image processing for now as much as possible and rely on internal game informations, later on if your algorithm works well, you can replace some parts of your AI program with image processing, thus gruadually attaining your full goal, for example if you can get something to work quite well, you can try to complexify your problem and replace supervised procedures and memory game data by unsupervised machine learning algorithms on screenshots).
Good luck, and if it works, make sure to publish an article, you can surely get renowned for solving such a hard practical problem!
The problem you are pursuing is intractable in the way you have defined it. It is usually a mistake to think that a neural network would "magically" learn a rich reprsentation of a problem. A good fact to keep in mind when deciding whether ANN is the right tool for a task is that it is an interpolation method. Think, whether you can frame your problem as finding an approximation of a function, where you have many points from this function and lots of time for designing the network and training it.
The problem you propose does not pass this test. Game control is not a function of the image on the screen. There is a lot of information the player has to keep in memory. For a simple example, it is often true that every time you enter a shop in a game, the screen looks the same. However, what you buy depends on the circumstances. No matter how complicated the network, if the screen pixels are its input, it would always perform the same action upon entering the store.
Besides, there is the problem of scale. The task you propose is simply too complicated to learn in any reasonable amount of time. You should see aigamedev.com for how game AI works. Artitificial Neural Networks have been used successfully in some games, but in very limited manner. Game AI is difficult and often expensive to develop. If there was a general approach of constructing functional neural networks, the industry would have most likely seized on it. I recommend that you begin with much, much simpler examples, like tic-tac-toe.
Seems like the heart of this project is exploring what is possible with an ANN, so I would suggest picking a game where you don't have to deal with image processing (which from other's answers on here, seems like a really difficult task in a real-time game). You could use the Starcraft API to build your bot, they give you access to all relevant game state.
http://code.google.com/p/bwapi/
As a first step you might look at the difference of consecutive frames. You have to distinguish between background and actual monster sprites. I guess the world may also contain animations. In order to find those I would have the character move around and collect everything that moves with the world into a big background image/animation.
You could detect and and identify enemies with correlation (using FFT). However if the animations repeat pixel-exact it will be faster to just look at a few pixel values. Your main task will be to write a robust system that will identify when a new object appears on the screen and will gradually all the frames of the sprite frame to a database. Probably you have to build models for weapon effects as well. Those can should be subtracted so that they don't clutter your opponent database.
Well assuming at any time you could generate a set of 'outcomes' (might involve probabilities) from a set of all possible 'moves', and that there is some notion of consistency in the game (eg you can play level X over and over again), you could start with N neural networks with random weights, and have each of them play the game in the following way:
1) For every possible 'move', generate a list of possible 'outcomes' (with associated probabilities)
2) For each outcome, use your neural network to determine an associated 'worth' (score) of the 'outcome' (eg a number between -1 and 1, 1 being the best possible outcome, -1 being the worst)
3) Choose the 'move' leading to the highest prob * score
4) If the move led to a 'win' or 'lose', stop, otherwise go back to step 1.
After a certain amount of time (or a 'win'/'lose'), evaluate how close the neural network was to the 'goal' (this will probably involve some domain knowledge). Then throw out the 50% (or some other percentage) of NNs that were farthest away from the goal, do crossover/mutation of the top 50%, and run the new set of NNs again. Continue running until a satisfactory NN comes out.
I think your best bet would be a complex architecture involving a few/may networks: i.e. one recognizing and responding to items, one for the shop, one for combat (maybe here you would need one for enemy recognition, one for attacks), etc.
Then try to think of the simplest possible Diablo II gameplay, probably a Barbarian. Then keep it simple at first, like Act I, first area only.
Then I guess valuable 'goals' would be disappearance of enemy objects, and diminution of health bar (scored inversely).
Once you have these separate, 'simpler' tasks taken care of, you can use a 'master' ANN to decide which sub-ANN to activate.
As for training, I see only three options: you could use the evolutionary method described above, but then you need to manually select the 'winners', unless you code a whole separate program for that. You could have the networks 'watch' someone play. Here they will learn to emulate a player or group of player's style. The network tries to predict the player's next action, gets reinforced for a correct guess, etc. If you actually get the ANN you want this could be done with video gameplay, no need for actual live gameplay. Finally you could let the network play the game, having enemy deaths, level ups, regained health, etc. as positive reinforcement and player deaths, lost health, etc. as negative reinforcement. But seeing how even a simple network requires thousands of concrete training steps to learn even simple tasks, you would need a lot of patience for this one.
All in all your project is very ambitious. But I for one think it could 'in theory be done', given enough time.
Hope it helps and good luck!

Game network physics collision

How to simulating two client-controlled vehicles colliding (sensibly) in a typical client/server setup for a network game? I did read this eminent blog post on how to do distributed network physics in general (without traditional client prediction), but this question is specifically on how to handle collisions of owned objects.
Example
Say client A is 20 ms ahead of server, client B 300 ms ahead of server (counting both latency and maximum jitter). This means that when the two vehicles collide, both clients will see the other as 320 ms behind - in the opposite direction of the velocity of the other vehicle. Head-to-head on a Swedish highway means a difference of 16 meters/17.5 yards!
What not to try
It is virtually impossible to extrapolate the positions, since I also have very complex vehicles with joints and bodies all over, which in turn have linear and angular positions, velocities and accelerations, not to mention states from user input.
I don't know of a perfect solution, and I have a feeling that one does not exist. Even if you could accurately predict the future position of the vehicle, you would be unable to predict the way the user will operate the controls. So the problem comes down to minimizing the negative effects of client/server lag. With that in mind, I would approach this from the position of the principle of least astonishment (paraphrased from Wikipedia):
In user interface design, the principle of least astonishment (or surprise) states that, when two elements of an interface conflict, or are ambiguous, the behaviour should be that which will least surprise the human user at the time the conflict arises.
In your example, each user sees two vehicles. Their own, and that of another player. The user expects their own vehicle to behave exactly the way they control it, so we are unable to play with that aspect of the simulation. However, the user can not know exactly how the other user is controlling their vehicle, and I would use this ambiguity to hide the lag from the user.
Here is the basic idea:
The server has to make the decision about an impending collision. The collision detection algorithm doesn't have to be 100% perfect, it just has to be close enough to avoid obvious inconsistencies.
Once the server has determined that two vehicles will collide, it sends each of the two users a message indicating that a collision is imminent.
On client A, the position of vehicle B is adjusted (realistically) to guarantee that the collision occurs.
On client B, the position of vehicle A is adjusted (realistically) to guarantee that the collision occurs.
During the aftermath of the collision, the position of each vehicle can be adjusted, as necessary, so that the end result is in keeping with the rest of the game. This part is exactly what MedicineMan proposed in his answer.
In this way, each user is still in complete control of their own vehicle. When the collision occurs, it will not be unexpected. Each user will see the other vehicle move towards them, and they will still have the feeling of a real-time simulation. The nice thing is that this method reacts well in low-lag conditions. If both clients have low-latency connections to the server, the amount of adjustment will be small. The end result will, of course, get worse as the lag increases, but that is unavoidable. If someone is playing a fast-paced action game over a connection with several seconds worth of lag, they simply aren't going to get the full exeperience.
Perhaps the best thing that you can do is not so show the actual collision real time, but give the illusion that things are happening in real time.
Since the client is behind the server (lag), and the server needs to show the result of the collision, perhaps what you can do, client side, is to show a flash or explosion or some other graphic to distract the user and buy enough time on the server side to calculate the result of the collision.. When you are finished with the prediction, you ship it back to the client side for presentation.
Sorry to answer with "What not to try", but I've never heard of a solution that doesn't involve predicting the outcome on client side. Consider a simplified example:
Client A is stationary, and watching client B's vehicle approach a cliff. Client B's vehicle is capable of reducing speed to 0 instantly, and does so at the last possible moment before going over the cliff.
If Client A is attempting to show Client B's state in real time, Client A has no choice but to predict that Client B fell off the cliff. You see this a lot in MMORPGs designed such that a player's character is capable of stopping immediately when running full-speed. Otherwise, Client A could just show Client B's state as the state updates come in, but this isn't viable, as Client A needs to be able to interact with Client B in real time in your scenario (I assume).
Could you try simplifying the collision models so that extrapolation is possible for real time prediction? Maybe make your "joints and bodies all over" have processor-less-intensive physical models, like a few cubes or spheres. I'm not too familiar with how to improve the efficiency of collision detection, but I assume it's done by detecting collisions of models that are less complex than the visual models.
Regarding "What not to try". You are assuming that you need to predict perfectly, but you are never going to find a perfect solution in a game with complex physics. An approximation is probably the best you can do (for example, most commercial physics engines can cast a shape into the physics scene and return the first point of collision).
For example, I implemented some critical parts of the network physics for Mercenaries 2 under the guidance of Glenn (the blog poster you mentioned). It was impossible to push all of the necessary physics state across the wire for even a single rigid body. Havok physics gradually generates contact points each frame, so the current "contact manifold" is a necessary part of the physics state to keep the simulation deterministic. It's also way too much data. Instead, we sent over the desired transform and velocities and used forces and torques to gently push bodies into place. Errors are inevitable, so you need a good error correction scheme.
What I eventually ended up doing was simply skipping prediction alltogether and simply doing this:
Client has very much say about its own position,
Server (almost) only says anything about the owning client's position when a "high energy" collision has happened with another dynamic object (i.e. not static environment).
Client takes meshoffset=meshpos-physpos when receiving a positional update from the server and then sets meshpos=physpos+meshoffset each frame and gradually decreases meshoffset.
It looks quite good most of the time (in low latency situation), I don't even have to slerp my quaternions to get smooth transitions.
Skipping prediction probably gives high-latency clients an awful experiance, but I don't have time to dwell on this if I'm ever going to ship this indie game. Once in a while it's nice to create a half-ass solution that works good enough but best. ;)
Edit: I eventually ended up adding the "ownership" feature that Glen Fiedler (the blogger mentioned in the question) implemented for Mercenaries 2: each client gets ownership of (dynamic) objects that they collide with for a while. This was necessary since the penetration otherwise becomes deep in high latency and high speed situations. That soluation works just as great as you'd think when you see the GDC video presentation, can definitely recommend it!
Few thoughts.
Peer to peer is better at dealing with latency & high speeds.
So if this is your own engine then switch to peer to peer. You then extrapolate the other peer's vehicle based on their button input to move forward to where it is now. You the set collision such that you collide against the other vehicle as if it's world. I.e. you take the hit.
This means as you collide against the other, you bounce off, on the peer's network they bounce off you, so it looks roughly correct. The lower the latency the better it works.
If you want to go client / server then this will be inferior to p2p
Things to attempt
o) Extrapolate clients forward as in p2p to perform collision detection.
o) Send collision results back to clients and extrapolate forward
Note, this is NEVER going to be as good as p2p. Fundamentally high speed & latency = error so removing latency is the best strategy. P2P does that.
In addition to predicting on the client side where the other user might be and sending the collision information and how you handled it to the server, what most mmo's do to deal with lag is they have the server run "in the past" as it were. Basically they buffer the recent inputs but only react to what happened .1sec in the past. This lets you "peek into the future" when you need to (ie when a collision is about to happen in your time frame, you can look at the buffered input to see what will happen and decide if the collision is real).
Of course, this adds an extra layer of complexity to your program as you have to consider what data to send to your clients and how they should react to it. For example you could send the entire "future" buffer to the clients and let them see which possible collisions will actually happen and which won't.
Ross has a good point. You could simplify the model you use to detect collisions by abstracting it to some simpler volume (i.e. the rough boxy outline of the vehicle). Then you can do the predictions based on the simple volume and the detailed calculations on the exact volumes while you have the user distracted by the "explosion". It may not be perfect but would allow you to speed up your collision detection.

Quality vs. ROI - When is Good Enough, good enough? [closed]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 4 years ago.
Improve this question
UPDATED: I'm asking this from a development perspective, however to illustrate, a canoical non-development example that comes to mind is that if it costs, say, $10,000
to keep a uptime rate of 99%, then it theoretically can cost $100,000 to keep a rate
of 99.9%, and possibly $1,000,000 to keep a rate of 99.99%.
Somewhat like calculus in approaching 0, as we closely approach 100%,
the cost can increase exponentially. Therefore, as a developer or PM, where do you decide
that the deliverable is "good enough" given the time and monetary constraints, e.g.: are you getting a good ROI at 99%, 99.9%,
99.99%?
I'm using a non-development example because I'm not sure of a solid metric for development. Maybe in the above example "uptime" could be replaced with "function point to defect ratio", or some such reasonable measure rate of bugs vs. the complexity of code. I would also welcome input regarding all stages of a software development lifecycle.
Keep the classic Project Triangle constraints in mind (quality vs. speed vs. cost). And let's assume that the customer wants the best quality you can deliver given the original budget.
There's no way to answer this without knowing what happens when your application goes down.
If someone dies when your application goes down, uptime is worth spending millions or even billions of dollars on (aerospace, medical devices).
If someone may be injured if your software goes down, uptime is worth hundreds of thousands or millions of dollars (industrial control systems, auto safety devices)
If someone looses millions of dollars if your software goes down, uptime is worth spending millions on (financial services, large e-commerce apps).
If someone looses thousands of dollars if your software goes down, uptime is worth spending thousands on (retail, small e-commerce apps).
If someone will swear at the computer and looses productivity while it reboots when your software goes down, then uptime is worth spending thousands on (most internal software).
etc.
Basically take (cost of going down) x (number of times the software will go down) and you know how much to spend on uptime.
The Quality vs Good Enough discussion I've seen has a practical ROI at 95% defect fixes. Obviously show stoppers / critical defects are fixed (and always there are the exceptions like air-plane autopilots etc, that need to not have so many defects).
I can't seem to find the reference to the 95% defect fixes, it is either in Rapid Development or in Applied Software Measurement by Caper Jones.
Here is a link to a useful strategy for attacking code quality:
http://www.gamedev.net/reference/articles/article1050.asp
The client, of course, would likely balk at that number and might say no more than 1 hour of downtime per year is acceptable. That's 12 times more stable. Do you tell the customer, sorry, we can't do that for $100,000, or do you make your best attempt, hoping your analysis was conservative?
Flat out tell the customer what they want isn't reasonable. In order to gain that kind of uptime, a massive amount of money would be needed, and realistically, the chances of reaching that percentage of uptime constantly just isn't possible.
I personally would go back to the customer and tell them that you'll provide them with the best setup with 100k and set up an outage report guideline. Something like, for every outage you have, we will complete an investigation as to why this outage happened, and how what we will do to make the chances of it happening again almost non existent.
I think offering SLAs is just a mistake.
I think the answer to this question depends entirely on the individual application.
Software that has an impact on human safety has much different requirements than, say, an RSS feed reader.
The project triangle is a gross simplification. In lots of cases you can actually save time by improving quality. For example by reducing repairs and avoiding costs in maintenance. This is not only true in software development.Toyota lean production proved that this works in manufacturing too.
The whole process of software development is far too complex to make generalizations on cost vs quality. Quality is a fuzzy concept that consists of multiple factors. Is testable code of higher quality than performant code? Is maintainable code of higher quality than testable code? Do you need testable code for an RSS reader or performant code? And for a fly-by-wire F16?
It's more productive to make informed desisions on a case-by-case basis. And don't be afraid to over-invest in quality. It's usually much cheaper and safer than under-investing.
To answer in an equally simplistic way..
..When you stop hearing from the customers (and not because they stopped using your product).. except for enhancement requests and bouquets :)
And its not a triangle, it has 4 corners - Cost Time Quality and Scope.
To expand on what "17 of 26" said, the answer depends on value to the customer. In the case of critical software, like aircrafct controller applications, the value to the customer of a high quality rating by whatever measure they use is quite high. To the user of an RSS feed reader, the value of high quality is considerably lower.
It's all about the customer (notice I didn't say user - sometimes they're the same, and sometimes they're not).
Chasing the word "Quality" is like chasing the horizon. I have never seen anything (in the IT world or outside) that is 100% quality. There's always room for improvement.
Secondly, "quality" is an overly broad term. It means something different to everyone and subjective in it's degree of implementation.
That being said, every effort boils down to what "engineering" means--making the right choices to balance cost, time and key characteristics (ie. speed, size, shape, weight, etc.) These are constraints.

Resources