Avoid echoing with LSTM - machine-learning

In this twitter thread https://twitter.com/jlowin/status/1061085224862433280, Jeremiah says that the author makes the error of echoing in this article https://heartbeat.fritz.ai/using-a-keras-long-shortterm-memory-lstm-model-to-predict-stock-prices-a08c9f69aa74
"3/6 The echo is the result of a model collapsing in a fit of maximum worthlessness. The model is saying, "My best guess is that tomorrow is pretty much going to be like today," or x_t+1 ≈ x_t. You don't need an AI for that. You barely need a calculator."
Whats wrong with the author model? how should he have done to avoid echo?
Thanks

Related

daily quote for each page with latex

I want create calendar planner from 1/1/2019 to 31/2/2019:
In Main.tex: I use \pgfcalendar{cal}{2019-01-01}{2019-02-31} to create pdf.
One page = one day
Now i want add each different quote to each day.
How write code in quote.tex and connect in Main.tex.
Thanks
\pgfcalendar{cal}{2019-01-01}{2019-02-31}
{%
\LARGE\bfseries
\pgfcalendarweekdayname{\pgfcalendarcurrentweekday},
\pgfcalendarcurrentday{}.
\pgfcalendarmonthname{\pgfcalendarcurrentmonth}
\pgfcalendarcurrentyear{}
%}
\pagebreak
==========
I'd suggest setting up a daily quote in Excel with all the bells and whistles of formulas to extract the weekday, day, month and year from a date. I pulled some random quotes from the Internets to compile with each of the first 100 days...
\documentclass{article}
\usepackage{filecontents}
\begin{filecontents*}{daily_quote.csv}
Number,Date,Day,Month,Year,DayOfWeek,Quote,Author
1,2019-01-01,1,January,2019,Tuesday,"We are what we repeatedly do. Excellence, therefore, is not an act but a habit.",Aristotle
2,2019-01-02,2,January,2019,Wednesday,The best way out is always through.,Robert Frost
3,2019-01-03,3,January,2019,Thursday,Do not wait to strike till the iron is hot; but make it hot by striking.,William B Sprague
4,2019-01-04,4,January,2019,Friday,Great spirits have always encountered violent opposition from mediocre minds.,Albert Einstein
5,2019-01-05,5,January,2019,Saturday,"Whether you think you can or think you can't, you're right.",Henry Ford
6,2019-01-06,6,January,2019,Sunday,I know for sure that what we dwell on is who we become.,Oprah Winfrey
7,2019-01-07,7,January,2019,Monday,"I've missed more than 9000 shots in my career. I've lost almost 300 games. 26 times, I've been trusted to take the game winning shot and missed. I've failed over and over and over again in my life. And that is why I succeed.",Michael Jordan
8,2019-01-08,8,January,2019,Tuesday,You must be the change you want to see in the world.,Mahatma Gandhi
9,2019-01-09,9,January,2019,Wednesday,What you get by achieving your goals is not as important as what you become by achieving your goals.,Goethe
10,2019-01-10,10,January,2019,Thursday,You can get everything in life you want if you will just help enough other people get what they want.,Zig Ziglar
11,2019-01-11,11,January,2019,Friday,"Whatever you do will be insignificant, but it is very important that you do it.",Mahatma Gandhi
12,2019-01-12,12,January,2019,Saturday,"Desire is the starting point of all achievement, not a hope, not a wish, but a keen pulsating desire which transcends everything.",Napoleon Hill
13,2019-01-13,13,January,2019,Sunday,Failure is the condiment that gives success its flavor.,Truman Capote
14,2019-01-14,14,January,2019,Monday,Vision without action is daydream. Action without vision is nightmare.,Japanese Proverb
15,2019-01-15,15,January,2019,Tuesday,"In any situation, the best thing you can do is the right thing; the next best thing you can do is the wrong thing; the worst thing you can do is nothing.",Theodore Roosevelt
16,2019-01-16,16,January,2019,Wednesday,"If you keep saying things are going to be bad, you have a chance of being a prophet.",Isaac B Singer
17,2019-01-17,17,January,2019,Thursday,Success consists of doing the common things of life uncommonly well.,Unknown
18,2019-01-18,18,January,2019,Friday,"Keep on going and the chances are you will stumble on something, perhaps when you are least expecting it. I have never heard of anyone stumbling on something sitting down.","Charles F Kettering, Engineer and Inventor"
19,2019-01-19,19,January,2019,Saturday,Twenty years from now you will be more disappointed by the things that you didn't do than by the ones you did do. So throw off the bowlines. Sail away from the safe harbor. Catch the trade winds in your sails. Explore. Dream. Discover.,Mark Twain
20,2019-01-20,20,January,2019,Sunday,Losers visualize the penalties of failure. Winners visualize the rewards of success.,Unknown
21,2019-01-21,21,January,2019,Monday,Some succeed because they are destined. Some succeed because they are determined.,Unknown
22,2019-01-22,22,January,2019,Tuesday,Experience is what you get when you don't get what you want.,Dan Stanford
23,2019-01-23,23,January,2019,Wednesday,Setting an example is not the main means of influencing others; it is the only means.,Albert Einstein
24,2019-01-24,24,January,2019,Thursday,"A happy person is not a person in a certain set of circumstances, but rather a person with a certain set of attitudes.",Hugh Downs
25,2019-01-25,25,January,2019,Friday,"If you're going to be able to look back on something and laugh about it, you might as well laugh about it now.",Marie Osmond
26,2019-01-26,26,January,2019,Saturday,"Remember that happiness is a way of travel, not a destination.",Roy Goodman
27,2019-01-27,27,January,2019,Sunday,"If you want to test your memory, try to recall what you were worrying about one year ago today.",E Joseph Cossman
28,2019-01-28,28,January,2019,Monday,What lies behind us and what lies before us are tiny matters compared to what lies within us.,Ralph Waldo Emerson
29,2019-01-29,29,January,2019,Tuesday,We judge of man's wisdom by his hope.,Ralph Waldo Emerson
30,2019-01-30,30,January,2019,Wednesday,The best way to cheer yourself up is to try to cheer somebody else up.,Mark Twain
31,2019-01-31,31,January,2019,Thursday,"Age is an issue of mind over matter. If you don't mind, it doesn't matter.",Mark Twain
32,2019-02-01,1,February,2019,Friday,"Whenever you find yourself on the side of the majority, it's time to pause and reflect.",Mark Twain
33,2019-02-02,2,February,2019,Saturday,"Keep away from people who try to belittle your ambitions. Small people always do that, but the really great make you feel that you, too, can become great.",Mark Twain
34,2019-02-03,3,February,2019,Sunday,The surest way not to fail is to determine to succeed.,Richard B Sheridan
35,2019-02-04,4,February,2019,Monday,"Take the first step in faith. You don't have to see the whole staircase, just take the first step.",Dr Martin Luther King Jr
36,2019-02-05,5,February,2019,Tuesday,Act or accept.,Anonymous
37,2019-02-06,6,February,2019,Wednesday,"Many great ideas go unexecuted, and many great executioners are without ideas. One without the other is worthless.",Tim Blixseth
38,2019-02-07,7,February,2019,Thursday,The world is more malleable than you think and it's waiting for you to hammer it into shape.,Bono
39,2019-02-08,8,February,2019,Friday,Sometimes you just got to give yourself what you wish someone else would give you.,Dr Phil
40,2019-02-09,9,February,2019,Saturday,"Motivation is a fire from within. If someone else tries to light that fire under you, chances are it will burn very briefly.",Stephen R Covey
41,2019-02-10,10,February,2019,Sunday,People become really quite remarkable when they start thinking that they can do things. When they believe in themselves they have the first secret of success.,Norman Vincent Peale
42,2019-02-11,11,February,2019,Monday,Whenever you find whole world against you just turn around and lead the world.,Anonymous
43,2019-02-12,12,February,2019,Tuesday,Being defeated is only a temporary condition; giving up is what makes it permanent.,"Marilyn vos Savant, Author and Advice Columnist"
44,2019-02-13,13,February,2019,Wednesday,I can't understand why people are frightened by new ideas. I'm frightened by old ones.,John Cage
45,2019-02-14,14,February,2019,Thursday,Setting an example is not the main means of influencing others; it is the only means.,Albert Einstein
46,2019-02-15,15,February,2019,Friday,The difference between ordinary and extraordinary is that little extra.,Unknown
47,2019-02-16,16,February,2019,Saturday,The best way to predict the future is to create it.,Unknown
48,2019-02-17,17,February,2019,Sunday,Anyone can do something when they WANT to do it. Really successful people do things when they don't want to do it.,Dr Phil
49,2019-02-18,18,February,2019,Monday,"There are two primary choices in life: to accept conditions as they exist, or accept the responsibility for changing them.",Dr Denis Waitley
50,2019-02-19,19,February,2019,Tuesday,Success is the ability to go from failure to failure without losing your enthusiasm.,Sir Winston Churchill
51,2019-02-20,20,February,2019,Wednesday,Success seems to be connected with action. Successful people keep moving. They make mistakes but don't quit.,Conrad Hilton
52,2019-02-21,21,February,2019,Thursday,Attitudes are contagious. Make yours worth catching.,Unknown
53,2019-02-22,22,February,2019,Friday,Do not let what you cannot do interfere with what you can do.,John Wooden
54,2019-02-23,23,February,2019,Saturday,"There are only two rules for being successful. One, figure out exactly what you want to do, and two, do it.",Mario Cuomo
55,2019-02-24,24,February,2019,Sunday,"Sooner or later, those who win are those who think they can.",Richard Bach
56,2019-02-25,25,February,2019,Monday,Vision doesn't usually come as a lightening bolt. Rather it comes as a slow crystallization of life challenges that we one day recognize as a beautiful diamond with great value to ourselves and others.,Dr Michael Norwood
57,2019-02-26,26,February,2019,Tuesday,"Success is a state of mind. If you want success, start thinking of yourself as a success.",Dr Joyce Brothers
58,2019-02-27,27,February,2019,Wednesday,Ever tried. Ever failed. No matter. Try Again. Fail again. Fail better.,Samuel Beckett
59,2019-02-28,28,February,2019,Thursday,Flops are a part of life's menu and I've never been a girl to miss out on any of the courses.,Rosalind Russell
60,2019-03-01,1,March,2019,Friday,Cause change and lead. Accept change and survive. Resist change and die.,"Ray Norda, Chairman, Novell"
61,2019-03-02,2,March,2019,Saturday,"Winners lose much more often than losers. So if you keep losing but you're still trying, keep it up! You're right on track.",Matthew Keith Groves
62,2019-03-03,3,March,2019,Sunday,"An idea can turn to dust or magic, depending on the talent that rubs against it.",Bill Bernbach
63,2019-03-04,4,March,2019,Monday,An obstacle is often a stepping stone.,Prescott
64,2019-03-05,5,March,2019,Tuesday,Life is trying things to see if they work,Ray Bradbury
65,2019-03-06,6,March,2019,Wednesday,"If you worry about yesterday's failures, then today's successes will be few.",Anonymous
66,2019-03-07,7,March,2019,Thursday,Life is 10% what happens to us and 90% how we react to it.,Dennis P Kimbro
67,2019-03-08,8,March,2019,Friday,"We are all inventors, each sailing out on a voyage of discovery, guided each by a private chart, of which there is no duplicate. The world is all gates, all opportunities.",Ralph Waldo Emerson
68,2019-03-09,9,March,2019,Saturday,Knowing is not enough; we must apply. Willing is not enough; we must do.,Johann Wolfgang von Goethe
69,2019-03-10,10,March,2019,Sunday,"In matters of style, swim with the current; in matters of principle, stand like a rock.",Thomas Jefferson
70,2019-03-11,11,March,2019,Monday,"I think and think for months and years. Ninety-nine times, the conclusion is false. The hundredth time I am right.",Albert Einstein
71,2019-03-12,12,March,2019,Tuesday,"Where the willingness is great, the difficulties cannot be great.",Machiavelli
72,2019-03-13,13,March,2019,Wednesday,Strength does not come from physical capacity. It comes from an indomitable will.,Mahatma Gandhi
73,2019-03-14,14,March,2019,Thursday,You are what you think about all day long.,Dr Robert Schuller
74,2019-03-15,15,March,2019,Friday,What you do speaks so loudly that I cannot hear what you say.,Ralph Waldo Emerson
75,2019-03-16,16,March,2019,Saturday,"Success is not to be measured by the position someone has reached in life, but the obstacles he has overcome while trying to succeed.",Booker T Washington
76,2019-03-17,17,March,2019,Sunday,"Talent is formed in solitude, character in the bustle of the world.",Johann Wolfgang von Goethe
77,2019-03-18,18,March,2019,Monday,"To avoid criticism do nothing, say nothing, be nothing.",Elbert Hubbard
78,2019-03-19,19,March,2019,Tuesday,"If you want to make your dreams come true, the first thing you have to do is wake up.",JM Power
79,2019-03-20,20,March,2019,Wednesday,By working faithfully eight hours a day you may eventually get to be boss and work twelve hours a day,Robert Frost
80,2019-03-21,21,March,2019,Thursday,"I've learned that no matter what happens, or how bad it seems today, life does go on, and it will be better tomorrow.",Maya Angelou
81,2019-03-22,22,March,2019,Friday,The art of being wise is the art of knowing what to overlook.,William James
82,2019-03-23,23,March,2019,Saturday,"When I hear somebody sigh, ‘Life is hard,' I am always tempted to ask, ‘Compared to what?'",Sydney Harris
83,2019-03-24,24,March,2019,Sunday,Don't let life discourage you; everyone who got where he is had to begin where he was.,Richard L Evans
84,2019-03-25,25,March,2019,Monday,In three words I can sum up everything I've learned about life: It goes on.,Robert Frost
85,2019-03-26,26,March,2019,Tuesday,"You gain strength, courage and confidence by every experience in which you stop to look fear in the face.",Eleanor Roosevelt
86,2019-03-27,27,March,2019,Wednesday,Sometimes even to live is an act of courage.,Seneca
87,2019-03-28,28,March,2019,Thursday,"Do first things first, and second things not at all.",Peter Drucker
88,2019-03-29,29,March,2019,Friday,The only people who find what they are looking for in life are the fault finders.,Foster's Law
89,2019-03-30,30,March,2019,Saturday,Defeat is not bitter unless you swallow it.,Joe Clark
90,2019-03-31,31,March,2019,Sunday,I am an optimist. It does not seem too much use being anything else.,Winston Churchill
91,2019-04-01,1,April,2019,Monday,Positive anything is better than negative thinking.,Elbert Hubbard
92,2019-04-02,2,April,2019,Tuesday,People seem not to see that their opinion of the world is also a confession of character.,Ralph Waldo Emerson
93,2019-04-03,3,April,2019,Wednesday,"Those who wish to sing, always find a song.",Swedish Proverb
94,2019-04-04,4,April,2019,Thursday,"If you're going through hell, keep going.",Winston Churchill
95,2019-04-05,5,April,2019,Friday,"The sun shines and warms and lights us and we have no curiosity to know why this is so; but we ask the reason of all evil, of pain, and hunger, and mosquitoes and silly people.",Ralph Waldo Emerson
96,2019-04-06,6,April,2019,Saturday,Life is a shipwreck but we must not forget to sing in the lifeboats.,Voltaire
97,2019-04-07,7,April,2019,Sunday,"Enduring habits I hate…. Yes, at the very bottom of my soul I feel grateful to all my misery and bouts of sickness and everything about me that is imperfect, because this sort of thing leaves me with a hundred backdoors through which I can escape from enduring habits.","Friedrich Nietzsche, The Gay Science, 1882"
98,2019-04-08,8,April,2019,Monday,There is no education like adversity.,Disraeli
99,2019-04-09,9,April,2019,Tuesday,He who has a why to live can bear almost any how.,Friedrich Nietzsche
100,2019-04-10,10,April,2019,Wednesday,Adversity introduces a man to himself.,Unknown
\end{filecontents*}
\usepackage{datatool}
\begin{document}
\DTLloadrawdb[keys = {
Number,
Date,
Day,
Month,
Year,
DayOfWeek,
Quote,
Author
}]
{DailyQuote} % DB name
{daily_quote.csv} % Filename
\raggedright
\DTLforeach
{DailyQuote}
{\Day = Day,
\Month = Month,
\Year = Year,
\DayOfWeek = DayOfWeek,
\Quote = Quote,
\Author = Author
}{
\clearpage
\section*{\DayOfWeek, \Month~\Day, \Year}
\Quote~\textit{\Author}
}
\end{document}

Does it help to duplicate original data in order to make more data for building model?

I just got an interview question.
"Assume you want to build a statistical or machine learning model, but you have very limited data on hand. Your boss told you can duplicate original data several times, to make more data for building the model" Does it help?
Intuitively, it does not help, because duplicating original data doesn't create more "information" to feed the model.
But is there anyone can explain it more statistically? Thanks
Consider e.g. variance. The data set with the duplicated data will have the exact same variance - you don't have a more precise estimate of the distrbution afterwards.
There are, however, some exceptions. For example bootstrap validation helps when evaluating your model, but you have very little data.
Well, it depends on exactly what one means by "duplicating the data".
If one is exactly duplicating the whole data set a number of times, then methods based on maximum likelihood (as with many models in common use) must find exactly the same result since the log likelihood function of the duplicated data is exactly a multiple of the unduplicated data's log likelihood, and therefore has the same maxima. (This argument doesn't apply to methods which aren't based on the likelihood function; I believe that CART and other tree models, and SVM's, are such models. In that case you'll have to work out a different argument.)
However, if by duplicating, one means duplicating the positive examples in a classification problem (which is common enough, since there are often many more negative examples than positive), then that does make a difference, since the likelihood function is modified.
Also if one means bootstrapping, then that, too, makes a difference.
PS. Probably you'll get more interest in this question on stats.stackexchange.com.

Hyperopt Exploration/Exploitation strategy

What kind of settings Hyperopt provides to adjust balance between exploration with exploitation ? There's something like "bandit" and "bandit_algo" in the code but no explanation.
Could someone provide any code sample.
Thanks a lot for any help!
I just found hyperopt partial() a magical wrapper function for the optimizer algo. It allows to balance between different strategies and then E/E:
Partial returns the result of a randomly-chosen suggest function. For example to search by sometimes using random search, sometimes anneal, and sometimes tpe, type:
fmin(...,
algo=partial(mix.suggest,
p_suggest=[
(.1, rand.suggest),
(.2, anneal.suggest),
(.7, tpe.suggest),]),
)
Parameter "p_suggest": list of (probability, suggest) pairs. Make a suggestion from one of the suggest functions, in proportion to its corresponding probability. sum(probabilities) must be [close to] 1.0.
If you want an even sharper control of algo progression: you can use the fact that hyperopt optimizer algos are stateless and return the trial object which can be provided as an input to a new fmin to continue the process. Then you can call fmin with max_evals at 1 and handle the process in a loop, therefore you could modify "trials" and "suggest algo" between each iteration.
For the best bet, read the papers by Bergstra et. al. 1 2 and 3. I am not 100% clear on what the bandit_algo is, except that one of the papers mentions it as an alternative method to Gaussian Process and Tree of Parzen Estimators - maybe you can use it in the same way as those two?
My guess is that if it not documented, it may not be finished yet. You can try raising an issue on Github - the devs are fairly responsive from what I have seen.
EDIT: Looking at this paper, these bandit algorithms may be the base class that the others inherit from.

Conjoint analysis based on a orthogonal design

I'm having some issues regarding a conjoint analysis. Excuse me if some of the terms I use are wrong, but it has been some time since I last worked with SPSS - and my teacher was Danish.
Task object
I am to make a series of concept travelpackages (attributes and attribute notes/levels).
This far I've got things under control - I've reduced the number of packages from 81 to 9, with the help of 'orthogonal' design.
These 9 packages have been rated by some people (1-10), on a questionnaire.
Then I've been asked to write a syntax which evaluates my conjoint plan:
CONJOINT PLAN= 'C:\Users\MYNAME\DROBBOXFOLDER\Conjoint_cards.SAV'
/DATA='C:\Users\MYNAME\DROBBOXFOLDER\Respondents.SAV'
/SCORE=Card_1 TO Card_9
/SUBJECT=ID
/FACTORS= SMS Minutter Data Tryghed
/PRINT=ALL
/PLOT=ALL.
However I keep getting this error:
SUBJECT SUBCOMMAND -- Subject variable is not on data file.
Execution of this command stops.
At this point I've been to the dark pages of Google and back for an answer to what I am doing wrong, but nothing so far. The answer is probably staring me in the face. But I will appreciate any help or pointers as to what I'm doing wrong.
Problem solved:
So apparently one shouldn't follow a guide to the letter. My datafile didn't contain a ID, so removing this from my syntax solved the problem.

Best way to detect and store path combinations for analysing purpose later

I am searching for ideas/examples on how to store path patterns from users - with the goal of analysing their behaviours and optimizing on "most used path" when we can detect them somehow.
Eg. which action do they do after what, so that we later on can check to see if certain actions are done over and over again - therefore developing a shortcut or assembling some of the actions into a combined multiaction.
My first guess would be some sort of "simple log", perhaps stored in some SQL-manner, where we can keep each action as an index and then just record everything.
Problem is that the path/action might be dynamically changed - even while logging - so we need to be able to take care of this fact too, when looking for patterns later.
Would you log everthing "bigtime" first and then POST-process every bit of details after some time or do you have great experience with other tactics?
My worry is that this is going to take up space, BIG TIME while logging 1000 users each day for a month or more.
Hope this makes sense and I am curious to see if anyone can provide sample code, pseudocode or perhaps links to something usefull.
Our tools will be C#, SQL-database, XML and .NET 3.5 - clients could also get .NET 4.0 if needed.
Patterns examples as we expect them
...
User #1001: A-B-A-A-A-B-C-E-F-G-H-A-A-A-C-B-A
User #1002: B-A-A-B-C-E-F
User #1003: F-B-B-A-E-C-A-A-A
User #1002: C-E-F
...
etc. no real way to know what they do next nor how many they will use, how often they will do it.
A secondary goal, if possible, if we later on add a new "action" called G (just sample to illustrate, there will be hundreds of actions) how could we detect these new behaviours influence on the previous patterns.
To explain it better, my thought here would be some way to detect "patterns within patterns", sort of like how compressions work, so that "repeative patterns" are spottet. We dont know how long these patterns might be, nor how often they might come. How do we break this down into "small bits and pieces" - whats the best approach you think?
I am not sure what you mean by path, but, if you gave every action in a path a unique symbol, you could reduce the problem to longest common substring or subsequence.
Or have a map of paths to the number of times that action occurred. Every time a certain path happens, increment the count for that path. Then sort to find the most common.
Pseudo idea/implementation so far
Log ever users action into a list/series of actions, bulk kinda style (textfiles/SQL - what ever, just store the whole thing for post-processing)
start counting every "1 action", "2 actions", "3 actions" up til a certain amount (lets say 30 levels)
sort them all, by giving values of importants to some of the actions (might be those producing end results)
A usefull result perhaps?
If we count all [A], [A-A], [A-B], [A-C], [A-A-A], [A-A-B] etc. its going to make a LONG and fine list of which actions are used in row frequently, and thats in the right direction, because if some of these results gets too high, we might need a shorter path. Problem is then, whats too few actions to be optimized and whats the longest needed actionlist to search for? My guess is that we need to do this counting first, then examine the numbers.
Problem is that this would be part of an analyzing tool we are developing and we dont have data until implementation, so we dont know what to look for before its actually done. hmm... wondering if there really IS an answer to this one.

Resources