daily quote for each page with latex - latex

I want create calendar planner from 1/1/2019 to 31/2/2019:
In Main.tex: I use \pgfcalendar{cal}{2019-01-01}{2019-02-31} to create pdf.
One page = one day
Now i want add each different quote to each day.
How write code in quote.tex and connect in Main.tex.
Thanks
\pgfcalendar{cal}{2019-01-01}{2019-02-31}
{%
\LARGE\bfseries
\pgfcalendarweekdayname{\pgfcalendarcurrentweekday},
\pgfcalendarcurrentday{}.
\pgfcalendarmonthname{\pgfcalendarcurrentmonth}
\pgfcalendarcurrentyear{}
%}
\pagebreak
==========

I'd suggest setting up a daily quote in Excel with all the bells and whistles of formulas to extract the weekday, day, month and year from a date. I pulled some random quotes from the Internets to compile with each of the first 100 days...
\documentclass{article}
\usepackage{filecontents}
\begin{filecontents*}{daily_quote.csv}
Number,Date,Day,Month,Year,DayOfWeek,Quote,Author
1,2019-01-01,1,January,2019,Tuesday,"We are what we repeatedly do. Excellence, therefore, is not an act but a habit.",Aristotle
2,2019-01-02,2,January,2019,Wednesday,The best way out is always through.,Robert Frost
3,2019-01-03,3,January,2019,Thursday,Do not wait to strike till the iron is hot; but make it hot by striking.,William B Sprague
4,2019-01-04,4,January,2019,Friday,Great spirits have always encountered violent opposition from mediocre minds.,Albert Einstein
5,2019-01-05,5,January,2019,Saturday,"Whether you think you can or think you can't, you're right.",Henry Ford
6,2019-01-06,6,January,2019,Sunday,I know for sure that what we dwell on is who we become.,Oprah Winfrey
7,2019-01-07,7,January,2019,Monday,"I've missed more than 9000 shots in my career. I've lost almost 300 games. 26 times, I've been trusted to take the game winning shot and missed. I've failed over and over and over again in my life. And that is why I succeed.",Michael Jordan
8,2019-01-08,8,January,2019,Tuesday,You must be the change you want to see in the world.,Mahatma Gandhi
9,2019-01-09,9,January,2019,Wednesday,What you get by achieving your goals is not as important as what you become by achieving your goals.,Goethe
10,2019-01-10,10,January,2019,Thursday,You can get everything in life you want if you will just help enough other people get what they want.,Zig Ziglar
11,2019-01-11,11,January,2019,Friday,"Whatever you do will be insignificant, but it is very important that you do it.",Mahatma Gandhi
12,2019-01-12,12,January,2019,Saturday,"Desire is the starting point of all achievement, not a hope, not a wish, but a keen pulsating desire which transcends everything.",Napoleon Hill
13,2019-01-13,13,January,2019,Sunday,Failure is the condiment that gives success its flavor.,Truman Capote
14,2019-01-14,14,January,2019,Monday,Vision without action is daydream. Action without vision is nightmare.,Japanese Proverb
15,2019-01-15,15,January,2019,Tuesday,"In any situation, the best thing you can do is the right thing; the next best thing you can do is the wrong thing; the worst thing you can do is nothing.",Theodore Roosevelt
16,2019-01-16,16,January,2019,Wednesday,"If you keep saying things are going to be bad, you have a chance of being a prophet.",Isaac B Singer
17,2019-01-17,17,January,2019,Thursday,Success consists of doing the common things of life uncommonly well.,Unknown
18,2019-01-18,18,January,2019,Friday,"Keep on going and the chances are you will stumble on something, perhaps when you are least expecting it. I have never heard of anyone stumbling on something sitting down.","Charles F Kettering, Engineer and Inventor"
19,2019-01-19,19,January,2019,Saturday,Twenty years from now you will be more disappointed by the things that you didn't do than by the ones you did do. So throw off the bowlines. Sail away from the safe harbor. Catch the trade winds in your sails. Explore. Dream. Discover.,Mark Twain
20,2019-01-20,20,January,2019,Sunday,Losers visualize the penalties of failure. Winners visualize the rewards of success.,Unknown
21,2019-01-21,21,January,2019,Monday,Some succeed because they are destined. Some succeed because they are determined.,Unknown
22,2019-01-22,22,January,2019,Tuesday,Experience is what you get when you don't get what you want.,Dan Stanford
23,2019-01-23,23,January,2019,Wednesday,Setting an example is not the main means of influencing others; it is the only means.,Albert Einstein
24,2019-01-24,24,January,2019,Thursday,"A happy person is not a person in a certain set of circumstances, but rather a person with a certain set of attitudes.",Hugh Downs
25,2019-01-25,25,January,2019,Friday,"If you're going to be able to look back on something and laugh about it, you might as well laugh about it now.",Marie Osmond
26,2019-01-26,26,January,2019,Saturday,"Remember that happiness is a way of travel, not a destination.",Roy Goodman
27,2019-01-27,27,January,2019,Sunday,"If you want to test your memory, try to recall what you were worrying about one year ago today.",E Joseph Cossman
28,2019-01-28,28,January,2019,Monday,What lies behind us and what lies before us are tiny matters compared to what lies within us.,Ralph Waldo Emerson
29,2019-01-29,29,January,2019,Tuesday,We judge of man's wisdom by his hope.,Ralph Waldo Emerson
30,2019-01-30,30,January,2019,Wednesday,The best way to cheer yourself up is to try to cheer somebody else up.,Mark Twain
31,2019-01-31,31,January,2019,Thursday,"Age is an issue of mind over matter. If you don't mind, it doesn't matter.",Mark Twain
32,2019-02-01,1,February,2019,Friday,"Whenever you find yourself on the side of the majority, it's time to pause and reflect.",Mark Twain
33,2019-02-02,2,February,2019,Saturday,"Keep away from people who try to belittle your ambitions. Small people always do that, but the really great make you feel that you, too, can become great.",Mark Twain
34,2019-02-03,3,February,2019,Sunday,The surest way not to fail is to determine to succeed.,Richard B Sheridan
35,2019-02-04,4,February,2019,Monday,"Take the first step in faith. You don't have to see the whole staircase, just take the first step.",Dr Martin Luther King Jr
36,2019-02-05,5,February,2019,Tuesday,Act or accept.,Anonymous
37,2019-02-06,6,February,2019,Wednesday,"Many great ideas go unexecuted, and many great executioners are without ideas. One without the other is worthless.",Tim Blixseth
38,2019-02-07,7,February,2019,Thursday,The world is more malleable than you think and it's waiting for you to hammer it into shape.,Bono
39,2019-02-08,8,February,2019,Friday,Sometimes you just got to give yourself what you wish someone else would give you.,Dr Phil
40,2019-02-09,9,February,2019,Saturday,"Motivation is a fire from within. If someone else tries to light that fire under you, chances are it will burn very briefly.",Stephen R Covey
41,2019-02-10,10,February,2019,Sunday,People become really quite remarkable when they start thinking that they can do things. When they believe in themselves they have the first secret of success.,Norman Vincent Peale
42,2019-02-11,11,February,2019,Monday,Whenever you find whole world against you just turn around and lead the world.,Anonymous
43,2019-02-12,12,February,2019,Tuesday,Being defeated is only a temporary condition; giving up is what makes it permanent.,"Marilyn vos Savant, Author and Advice Columnist"
44,2019-02-13,13,February,2019,Wednesday,I can't understand why people are frightened by new ideas. I'm frightened by old ones.,John Cage
45,2019-02-14,14,February,2019,Thursday,Setting an example is not the main means of influencing others; it is the only means.,Albert Einstein
46,2019-02-15,15,February,2019,Friday,The difference between ordinary and extraordinary is that little extra.,Unknown
47,2019-02-16,16,February,2019,Saturday,The best way to predict the future is to create it.,Unknown
48,2019-02-17,17,February,2019,Sunday,Anyone can do something when they WANT to do it. Really successful people do things when they don't want to do it.,Dr Phil
49,2019-02-18,18,February,2019,Monday,"There are two primary choices in life: to accept conditions as they exist, or accept the responsibility for changing them.",Dr Denis Waitley
50,2019-02-19,19,February,2019,Tuesday,Success is the ability to go from failure to failure without losing your enthusiasm.,Sir Winston Churchill
51,2019-02-20,20,February,2019,Wednesday,Success seems to be connected with action. Successful people keep moving. They make mistakes but don't quit.,Conrad Hilton
52,2019-02-21,21,February,2019,Thursday,Attitudes are contagious. Make yours worth catching.,Unknown
53,2019-02-22,22,February,2019,Friday,Do not let what you cannot do interfere with what you can do.,John Wooden
54,2019-02-23,23,February,2019,Saturday,"There are only two rules for being successful. One, figure out exactly what you want to do, and two, do it.",Mario Cuomo
55,2019-02-24,24,February,2019,Sunday,"Sooner or later, those who win are those who think they can.",Richard Bach
56,2019-02-25,25,February,2019,Monday,Vision doesn't usually come as a lightening bolt. Rather it comes as a slow crystallization of life challenges that we one day recognize as a beautiful diamond with great value to ourselves and others.,Dr Michael Norwood
57,2019-02-26,26,February,2019,Tuesday,"Success is a state of mind. If you want success, start thinking of yourself as a success.",Dr Joyce Brothers
58,2019-02-27,27,February,2019,Wednesday,Ever tried. Ever failed. No matter. Try Again. Fail again. Fail better.,Samuel Beckett
59,2019-02-28,28,February,2019,Thursday,Flops are a part of life's menu and I've never been a girl to miss out on any of the courses.,Rosalind Russell
60,2019-03-01,1,March,2019,Friday,Cause change and lead. Accept change and survive. Resist change and die.,"Ray Norda, Chairman, Novell"
61,2019-03-02,2,March,2019,Saturday,"Winners lose much more often than losers. So if you keep losing but you're still trying, keep it up! You're right on track.",Matthew Keith Groves
62,2019-03-03,3,March,2019,Sunday,"An idea can turn to dust or magic, depending on the talent that rubs against it.",Bill Bernbach
63,2019-03-04,4,March,2019,Monday,An obstacle is often a stepping stone.,Prescott
64,2019-03-05,5,March,2019,Tuesday,Life is trying things to see if they work,Ray Bradbury
65,2019-03-06,6,March,2019,Wednesday,"If you worry about yesterday's failures, then today's successes will be few.",Anonymous
66,2019-03-07,7,March,2019,Thursday,Life is 10% what happens to us and 90% how we react to it.,Dennis P Kimbro
67,2019-03-08,8,March,2019,Friday,"We are all inventors, each sailing out on a voyage of discovery, guided each by a private chart, of which there is no duplicate. The world is all gates, all opportunities.",Ralph Waldo Emerson
68,2019-03-09,9,March,2019,Saturday,Knowing is not enough; we must apply. Willing is not enough; we must do.,Johann Wolfgang von Goethe
69,2019-03-10,10,March,2019,Sunday,"In matters of style, swim with the current; in matters of principle, stand like a rock.",Thomas Jefferson
70,2019-03-11,11,March,2019,Monday,"I think and think for months and years. Ninety-nine times, the conclusion is false. The hundredth time I am right.",Albert Einstein
71,2019-03-12,12,March,2019,Tuesday,"Where the willingness is great, the difficulties cannot be great.",Machiavelli
72,2019-03-13,13,March,2019,Wednesday,Strength does not come from physical capacity. It comes from an indomitable will.,Mahatma Gandhi
73,2019-03-14,14,March,2019,Thursday,You are what you think about all day long.,Dr Robert Schuller
74,2019-03-15,15,March,2019,Friday,What you do speaks so loudly that I cannot hear what you say.,Ralph Waldo Emerson
75,2019-03-16,16,March,2019,Saturday,"Success is not to be measured by the position someone has reached in life, but the obstacles he has overcome while trying to succeed.",Booker T Washington
76,2019-03-17,17,March,2019,Sunday,"Talent is formed in solitude, character in the bustle of the world.",Johann Wolfgang von Goethe
77,2019-03-18,18,March,2019,Monday,"To avoid criticism do nothing, say nothing, be nothing.",Elbert Hubbard
78,2019-03-19,19,March,2019,Tuesday,"If you want to make your dreams come true, the first thing you have to do is wake up.",JM Power
79,2019-03-20,20,March,2019,Wednesday,By working faithfully eight hours a day you may eventually get to be boss and work twelve hours a day,Robert Frost
80,2019-03-21,21,March,2019,Thursday,"I've learned that no matter what happens, or how bad it seems today, life does go on, and it will be better tomorrow.",Maya Angelou
81,2019-03-22,22,March,2019,Friday,The art of being wise is the art of knowing what to overlook.,William James
82,2019-03-23,23,March,2019,Saturday,"When I hear somebody sigh, ‘Life is hard,' I am always tempted to ask, ‘Compared to what?'",Sydney Harris
83,2019-03-24,24,March,2019,Sunday,Don't let life discourage you; everyone who got where he is had to begin where he was.,Richard L Evans
84,2019-03-25,25,March,2019,Monday,In three words I can sum up everything I've learned about life: It goes on.,Robert Frost
85,2019-03-26,26,March,2019,Tuesday,"You gain strength, courage and confidence by every experience in which you stop to look fear in the face.",Eleanor Roosevelt
86,2019-03-27,27,March,2019,Wednesday,Sometimes even to live is an act of courage.,Seneca
87,2019-03-28,28,March,2019,Thursday,"Do first things first, and second things not at all.",Peter Drucker
88,2019-03-29,29,March,2019,Friday,The only people who find what they are looking for in life are the fault finders.,Foster's Law
89,2019-03-30,30,March,2019,Saturday,Defeat is not bitter unless you swallow it.,Joe Clark
90,2019-03-31,31,March,2019,Sunday,I am an optimist. It does not seem too much use being anything else.,Winston Churchill
91,2019-04-01,1,April,2019,Monday,Positive anything is better than negative thinking.,Elbert Hubbard
92,2019-04-02,2,April,2019,Tuesday,People seem not to see that their opinion of the world is also a confession of character.,Ralph Waldo Emerson
93,2019-04-03,3,April,2019,Wednesday,"Those who wish to sing, always find a song.",Swedish Proverb
94,2019-04-04,4,April,2019,Thursday,"If you're going through hell, keep going.",Winston Churchill
95,2019-04-05,5,April,2019,Friday,"The sun shines and warms and lights us and we have no curiosity to know why this is so; but we ask the reason of all evil, of pain, and hunger, and mosquitoes and silly people.",Ralph Waldo Emerson
96,2019-04-06,6,April,2019,Saturday,Life is a shipwreck but we must not forget to sing in the lifeboats.,Voltaire
97,2019-04-07,7,April,2019,Sunday,"Enduring habits I hate…. Yes, at the very bottom of my soul I feel grateful to all my misery and bouts of sickness and everything about me that is imperfect, because this sort of thing leaves me with a hundred backdoors through which I can escape from enduring habits.","Friedrich Nietzsche, The Gay Science, 1882"
98,2019-04-08,8,April,2019,Monday,There is no education like adversity.,Disraeli
99,2019-04-09,9,April,2019,Tuesday,He who has a why to live can bear almost any how.,Friedrich Nietzsche
100,2019-04-10,10,April,2019,Wednesday,Adversity introduces a man to himself.,Unknown
\end{filecontents*}
\usepackage{datatool}
\begin{document}
\DTLloadrawdb[keys = {
Number,
Date,
Day,
Month,
Year,
DayOfWeek,
Quote,
Author
}]
{DailyQuote} % DB name
{daily_quote.csv} % Filename
\raggedright
\DTLforeach
{DailyQuote}
{\Day = Day,
\Month = Month,
\Year = Year,
\DayOfWeek = DayOfWeek,
\Quote = Quote,
\Author = Author
}{
\clearpage
\section*{\DayOfWeek, \Month~\Day, \Year}
\Quote~\textit{\Author}
}
\end{document}

Related

Have figure appear close to the reference when ithe mages are included at the end of the document in latex

Hi I have a long document such as this wikipedia entry on Alice in wonderland below. I reference the figure on the rabbit in the first line but I import the figure at the very end of the document. I now want the figure to appear close to the reference in the pdf. Does any one have any suggestions? [h] does not work...
The White Rabbit (Figure \ref{fig:rabbit})
Chapter One – Down the Rabbit Hole: Alice is feeling bored and drowsy while
sitting on the riverbank with her older sister, who is reading a book with
no pictures or conversations. She then notices a White Rabbit wearing a
waistcoat and pocket watch, talking to itself as it runs past. She follows
it down a rabbit hole, but suddenly falls a long way to a curious hall with
many locked doors of all sizes. She finds a small key to a door too small
for her to fit through, but through it she sees an attractive garden. She
then discovers a bottle on a table labelled DRINK ME, the contents of which
cause her to shrink too small to reach the key, which she has left on the
table. She eats a cake with EAT ME written on it in currants as the chapter
closes.
Chapter Two – The Pool of Tears: Chapter Two opens with Alice growing to
such a tremendous size that her head hits the ceiling. Alice is unhappy and,
as she cries, her tears flood the hallway. After shrinking down again due to
a fan she had picked up, Alice swims through her own tears and meets a
Mouse, who is swimming as well. She tries to make small talk with him in
elementary French (thinking he may be a French mouse) but her opening gambit
Où est ma chatte? (Where is my cat?) offends the mouse and he tries to
escape her.
Chapter Three – The Caucus Race and a Long Tale: The sea of tears becomes
crowded with other animals and birds that have been swept away by the rising
waters. Alice and the other animals convene on the bank and the question
among them is how to get dry again. The Mouse gives them a very dry lecture
on William the Conqueror. A Dodo decides that the best thing to dry them off
would be a Caucus-Race, which consists of everyone running in a circle with
no clear winner. Alice eventually frightens all the animals away,
unwittingly, by talking about her (moderately ferocious) cat.
Chapter Four – The Rabbit Sends a Little Bill: The White Rabbit appears
again in search of the Duchess's gloves and fan. Mistaking her for his
maidservant, Mary Ann, he orders Alice to go into the house and retrieve
them, but once she gets inside she starts growing. The horrified Rabbit
orders his gardener, Bill the Lizard, to climb on the roof and go down the
chimney. Outside, Alice hears the voices of animals that have gathered to
gawk at her giant arm. The crowd hurls pebbles at her, which turn into
little cakes. Alice eats them, and they make her smaller again. Upon leaving
the house, she comes upon a puppy, with whom she plays for a while before
finding a mushroom with a hookah-smoking blue Caterpillar.
Chapter Five – Advice from a Caterpillar: The Caterpillar questions
Alice and she admits to her current identity crisis, compounded by her
inability to remember a poem. Before crawling away, the caterpillar tells
Alice that one side of the mushroom will make her taller and the other side
will make her shorter. She breaks off two pieces from the mushroom. One side
makes her shrink smaller than ever, while another causes her neck to grow
high into the trees, where a pigeon mistakes her for a serpent. With some
effort, Alice brings herself back to her normal height. She stumbles upon a
small estate and uses the mushroom to reach a more appropriate height.
The Cheshire Cat
Chapter Six – Pig and Pepper: A Fish-Footman has an invitation for the
Duchess of the house, which he delivers to a Frog-Footman. Alice observes
this transaction and, after a perplexing conversation with the frog, lets
herself into the house. The Duchess's Cook is throwing dishes and making a
soup that has too much pepper, which causes Alice, the Duchess, and her baby
(but not the cook or grinning Cheshire Cat) to sneeze violently. Alice is
given the baby by the Duchess and to her surprise, the baby turns into a
pig. The Cheshire Cat appears in a tree, directing her to the March Hare's
house. He disappears, but his grin remains behind to float on its own in the
air, prompting Alice to remark that she has often seen a cat without a grin
but never a grin without a cat.
Chapter Seven – A Mad Tea-Party: Alice becomes a guest at a mad tea party
along with the March Hare, the Hatter, and a very tired Dormouse who falls
asleep frequently, only to be violently woken up moments later by the March
Hare and the Hatter. The characters give Alice many riddles and stories,
including the famous Why is a raven like a writing desk?. The Hatter reveals
that they have tea all day because Time has punished him by eternally
standing
still at 6 pm (tea time). Alice becomes insulted and tired of being
bombarded with riddles and she leaves, claiming that it was the stupidest
tea party that she had ever been to. She makes her way back to the hallway
and, using the pieces of
mushroom, manages to get the key, unlock the door and enter the garden.
\begin{figure}
\centering
\includegraphics[width=0.4\textwidth]{./Figures/rabit.png}
\caption[Rabbit] {Picture of rabbit.}
\label{fig:rabbit}
\end{figure}
I'm afraid this is impossible to do in pure LaTeX. LaTeX's figure placements are restricted to 'here', 'top', 'bottom' and 'page'. None of these are sufficient for your case. 'h' from 'here' means close to where the figure is defined, not referenced.
It is however possible to have your .tex file processed by some other tool to move the figure definition closer to the reference and then have the 'here' placement applied, or to not style the image as a float at all.
Tools that can do this include awk and perl.

Programming can be confusing. Ruby language

I am here because I have recently decided to change careers, and considering both my parents are programmers I have always wanted to try out learning to code.
I want to apply to a little academy in San Francisco named App Academy but in order to move forward, I have to pass an exam for them. Their website is:
http://www.appacademy.io/
I've been reading a lot lately trying to prepare for their exam so to speak but somehow I'm still struggling to get the logic out of this. I feel like there are so many different ways of programming something, there are literally no limits sometimes.
In short, here is my question:
Write a method that will take in a number of minutes, and returns a string that formats the number into hours:minutes.
def time_conversion(minutes)
end
Where I'm struggling is that I'm never sure where to start. Every time I attack a new problem somehow I can't figure out the logic. In this problem, please don't provide answers as I want to try it myself. However I would really appreciate some help on learning a methodology to apply for every problem.
For instance, here the first thing I realize is that every hour has 60 minutes so at some point I will have to include that in my answer. Do I have to create a loop? I think so. Keep in mind I DON'T KNOW how to write yet...Perhaps something like:
def time_conversion(minutes)
i = 0
while i > 60
return hours of integer i % 60
return minutes of remainder * 60
else return minutes of remainder * 60
end
I'm sure I got it all wrong....please HELP! Is the logic in the right direction though?
You're going to have to work backwards here. Formatting a string is easily done with the sprintf method, but how do you know what to put in that? Here's the target:
sprintf("%d:%02d", hours, minutes)
Calculating hours and minutes given seconds just requires math:
hours = minutes / 60
minutes = minutes % 60
Then you can combine those two and get your result.
You might be overthinking this. How about the following approach?
def minutes_timestamp(minutes)
"%d:%.2d" % [minutes/60, minutes % 60]
end
EDIT
tadman's answer is correct as well. The above example is equivalent to:
def minutes_timestamp(minutes)
sprintf("%d:%.2d", minutes/60, minutes % 60)
end
The "%d:%.2d" syntax is a bit cryptic, but it simply ensures the resulting string conforms to the HH:MM format, while also ensuring a leading zero for single digit minutes. There's a great tutorial that can explain sprintf and string formatting better than I can here: https://blog.udemy.com/ruby-sprintf/.
Welcome to Stack Overflow. Learning programming is very difficult task as you have to change how you approach problems. Also you need before starting to learn logic to know the syntax of language you are learning. In your case return statement will exit the function and thus not allowing you to go to minutes part.
I can only recommend this book http://poignant.guide/ for a start in Ruby as it's very funny (at least to me). It will help you understand a language. After it you should try solving the logic/approach problem now that you know what you can use.
For finding solutions I can only recommend you to use pen and paper and find a solution there. After you are sure of human logic, try to pretend that you are a computer with access only to data (written on paper and identified by name) and operations that you know code can do and mark the changes in data after each operation.
I.e. you as human know that 1,3,5,7,9... are odd numbers, but given a number x how can you know is it odd in code? This is simple: is_odd = x % 2 == 1 so you can use that part of code whenever you had to check if something is odd. So my advice is to try to approach the problems more as a machine then as a human.
Temporarily ignore the fact that a computer is involved, and pretend you're giving instructions to someone without any common sense, who will follow your instructions to the letter.
What is it that you want the person to do for you? In this case, turn a single number (of minutes) into two numbers (a number of hours, and then a number of minutes left over), and then put them side-by-side with a colon between them.
Because it isn't immediately obvious how to do this, you then break this down:
How can you turn a number of minutes into a number of hours (ignoring the remainder, the formatting etc.)?
Once you have some of the minutes represented as a number of hours, how can you work out the number of minutes left?
Once you have the two numbers, how can you package them up in the desired output representation?
If these steps are too complex to just write the answer to, break each of them down, and so on. The computer code is merely a final imprint of the design.
minutes = 246
"%02d:%02d" % minutes.divmod(60) #=> "04:06"
Fixnum#divmod don't get no respect.

Why does this API return its response in AFNetworking as a "failure", when it's exactly what it's supposed to be and the status code is 200?

I formulate a POST to an API called Diffbot and it returns exactly what I want; a bunch of parsed articles. (Here's how the API is interacted with.)
I do this with the following code (with the exception of me removing my token):
[AFDiffbotClient sharedClient].operationQueue.maxConcurrentOperationCount = NSOperationQueueDefaultMaxConcurrentOperationCount;
NSMutableArray *individualRequests = [[NSMutableArray alloc] init];
for (NSDictionary *URLAndID in URLsAndIDs) {
NSString *articleURL = [URLAndID objectForKey:#"URL"];
NSString *requestURL = [NSString stringWithFormat:#"/api/article?token=...&fields=text,title,url&fields=text,title,url&url=%#", articleURL];
[individualRequests addObject:#{#"method": #"GET",
#"relative_url": requestURL}];
}
NSError *error;
NSData *individualRequestsJSONData = [NSJSONSerialization dataWithJSONObject:individualRequests options:kNilOptions error:&error];
NSString *individualRequestsJSONString = [[NSString alloc] initWithData:individualRequestsJSONData encoding:NSUTF8StringEncoding];
NSDictionary *parameters = #{#"token": #"...",
#"batch": individualRequestsJSONString};
[[AFDiffbotClient sharedClient] setParameterEncoding:AFFormURLParameterEncoding];
[[AFDiffbotClient sharedClient] postPath:#"http://diffbot.com/api/batch" parameters:parameters success:^(AFHTTPRequestOperation *operation, id responseObject) {
NSLog(#"WIN");
} failure:^(AFHTTPRequestOperation *operation, NSError *error) {
NSLog(#"%#", error);
}];
And here's part of the response I get back (full response on GitHub):
Error Domain=AFNetworkingErrorDomain Code=-1016 "Expected content type {(
"text/json",
"application/json",
"text/javascript"
)}, got text/plain" UserInfo=0xc0a43b0 {NSLocalizedRecoverySuggestion=[
{"headers":[{"name":null,"value":"HTTP/1.1 200 OK"},{"name":"Transfer-Encoding","value":"chunked"},{"name":"Vary","value":"Accept-Encoding"},{"name":"Date","value":"Tue, 21 Jan 2014 02:42:38 GMT"},{"name":"Content-Type","value":"application/json"},{"name":"Server","value":"Apache-Coyote/1.1"}],"body":"{\"category\":\"business_finance\",\"title\":\"A Firsthand Account of Microsoft\\u2019s Employee-Ranking System\",\"text\":\"There were seven or eight of us managers in the conference room, all peers, as well as our own manager. The conference rooms all had large tables, whether heavy varnished oak or cheap plywood. The chairs were the sort that let you lean back with increasing tension, and they had a few levers underneath the seat that raise and lower the seat and adjust the back. I\\u2019m a fidgeter, so I played with them a lot during meetings.\\nWe were mostly white, and all men. Each of us had between three and six \\u201Cdirect reports\\u201D: nonmanager programmers who we oversaw. We were the direct reports of our manager. There were lots and lots of managers at Microsoft\\u2014it was the only path to advancement, so the company structure became more and more steeply vertical. Once or twice a year, we would all get together and decide how good each of our reports was, by ordering them from best to worst.\\nThe system was called the stack rank.\\nFollowing Friday\\u2019s news of Microsoft CEO Steve Ballmer\\u2019s imminent retirement, postmortems of his lackluster 13-year reign have pointed to stack ranking \\u2014which, to be entirely fair, predated him\\u2014as both a cause and a symptom of the corporation\\u2019s decline. As a software developer and later development lead at Microsoft between 1998\\u20132003, I had to evaluate others and be evaluated myself under this system. And I can say that yes, stack ranking is as toxic for innovation and integrity and morale as media reports made it out to be , and then some.\\nEach report\\u2019s name was written on an index card and put on the table. It was a two-step process. First, reports were broadly sorted into four buckets: excellent, good, mediocre, and awful. Then, within each bucket, people were paired for comparison and bubbled up or down. Managers would argue whether a particular report was better or worse than some other manager\\u2019s report in the same bucket. Our manager would adjudicate the debates. Some managers were better fighters than others.\\nA manager\\u2019s goals here are twofold. First, you want to place your best people as high as possible on the ladder. This will help them get big bonuses, promotions, and raises, and thus keep them happy and less likely to leave your group or the company. Second, if you\\u2019re unfortunate enough to have weak reports, you want either to help them by placing them sufficiently high that they don\\u2019t get dinged too badly come the annual review, or to throw them to the wolves and let them get ranked low. If you give up on them, they\\u2019ll be put on a dead-end track that marks them as more or less useless. The object is to get them out from under you and make them someone else\\u2019s problem.\\nI was lucky not to have any weak reports. I had been encouraged to take one at an earlier point, but frenetically talked my way out of it, using all the managerese I could summon to argue that he didn\\u2019t belong on my team.\\nMy reports seemingly did fine. I got them all into the top two buckets. After the stack rank was over, our boss, who\\u2019d overseen the whole meeting, took our stack rank and then went to another, similar meeting one level up, where now we would be stacked and ranked behind our backs, then mixed in with the stack we\\u2019d just created, cruelly shuffled like a deck of human cards.\\nEventually, the vice presidents would have to bargain among themselves for how many bonuses and raises they would get for their entire organizations, then ration them out according to the stack rank. People would then be assigned one of three grades: 4.0 (Above Average), 3.5 (Average), and 3.0 (Below Average). The very rare 4.5 got you a set of steak knives ; a 2.5 meant you were fired (more or less). I\\u2019m not sure if Alan Turing himself could have gotten a 5.0.\\nThe stack rank was harmful. It served as an incentive not to join high-quality groups, because you\\u2019d be that much more likely to fall low in the stack rank. Better to join a weak group where you\\u2019d be the star, and then coast. Maybe the executives thought this would help strong people lift up weak teams. It never worked that way. More often, it just encouraged people to backstab their co-workers, since their loss entailed your profit.\\nThe stack rank was a zero-sum game\\u2014one person could only excel by the amount that others were penalized. And it was applied at every level of the organization. Even if you were in a group of three high performers, it was very likely that one of you would be graded Above Average, one Average, and one Below Average. Unless your manager was a prick or an idiot or both, the ordering would reflect your relative skills, but that never came as too much comfort to the hard-working schlub who just wasn\\u2019t as good as the other two.\\nThis was my problem. I had three reports, A, B, and C, and they neatly fit into three categories: C was good, B was great, and A was fantastic. They were all nice and retiring sorts\\u2014they weren\\u2019t self-promoters, which put them at a disadvantage at Microsoft\\u2014and I did want to do well by them. Based on their position in the stack rank, I thought that this would have been a fair assessment of them relative to the company in general:\\nMy Ideal Distribution\\nA: Above Average\\nB: Above Average\\nC: Average\\nAbove Average would get A and B nice bonuses and raises, while C might get a small raise and a decent bonus with an Average. That didn\\u2019t happen. My manager told me baldly that this was how it would go:\\nThe Actual Distribution\\nA: Above average\\nB: Average\\nC: Below average\\nMy desired rankings were out of the question, since my manager would then have had to steal that extra Above Average from some other manager. I thought that B could live with Average (we were all well-compensated, after all), but rating C as Below Average hurt.\\nSo I argued for C, and my manager said there was exactly one alternative:\\nThe Alternative Distribution\\nA: Average\\nB: Average\\nC: Average\\nBut A had been at the very top of the stack! How could A do worse than people we\\u2019d all agreed were weaker programmers? I gave up and let C take the Below Average. This is the zero-sum game at work.\\nI still feel bad about this.\\nThen I had to explain things to my reports. This illustrated another problem with the system: It destroyed trust between individual contributors and management, because the stack rank required that all lower-level managers systematically lie to their reports. Why? Because for years Microsoft did not admit the existence of the stack rank to nonmanagers . Knowledge of the process gradually leaked out, becoming a recurrent complaint on the much-loathed (by Microsoft) Mini-Microsoft blog , where a high-up Microsoft manager bitterly complained about organizational dysfunction and was joined in by a chorus of hundreds of employees. The stack rank finally made it into a Vanity Fair article in 2012, but for many years it was not common knowledge, inside or outside Microsoft. It was presented to the individual contributors as a system of objective assessment of \\u201Ccore competencies,\\u201D with each person being judged in isolation.\\nWhen review time came, and programmers would fill out a short self-assessment talking about their achievements, strengths, and weaknesses, only some of them knew that their ratings had been more or less already foreordained at the stack rank. The ones who knew could sometimes be recognized by their flip comments on their performance reviews, like the hot-tempered guy who wrote every year in \\u201CAreas to Improve,\\u201D \\u201CI will try to be less of an asshole.\\u201D\\nThey were exceptions, though. If you did know about the stack rank, you weren\\u2019t supposed to admit it. So you went through the pageantry of the performance review anyway, arguing with your manager in the rhetoric of \\u201Ccore competencies.\\u201D The managers would respond in kind. Since the managers had little control over the actual score and attendant bonus and raise (if any), their job was to write a review to justify the stack rank in the language of absolute merit. (\\u201CHigher visibility\\u201D was always a good catch-all: Sure, you may be a great coder and work 80 hours a week, but not enough people have heard of you!)\\nStrangely, this charade would sometimes happen even between managers and their managers, both pretending that they didn\\u2019t know about the stack rank that they had recently participated in. This kind of bad faith is more common than you might think. I saw it most vividly in a certain number of party-liners who seemed wholly oblivious to the dissonance between the performance review and the stack rank, as though the two would always magically line up, even though they never did.\\nThis sort of organizational dissembling skews your psyche. After I left Microsoft, I was left with lingering paranoia for months, always wondering about the agendas of those around me, skeptical that what I was being told was the real story. I didn\\u2019t realize until the nonstacked performance review time at my new job that I\\u2019d become so wary. At the time\\u2014inside Microsoft\\u2014it just seemed the only logical way to be.\",\"date_created\":\"Mon, 20 Jan 2014 18:02:03 PST\",\"categories\":{\"entertainment_culture\":0.014285714285714284,\"hospitality_recreation\":0.03428571428571428,\"other\":0.0,\"business_finance\":0.5157142857142857,\"technology_internet\":0.05999695366444171,\"socialissues\":0.022857142857142854,\"sports\":0.04052786085981821,\"humaninterest\":0.03714285714285714,\"religion_belief\":0.039999999999999994,\"war_conflict\":0.022259647554023695,\"education\":0.011428571428571427,\"health_medical_pharma\":0.031428571428571424,\"labor\":0.028571428571428567,\"law_crime\":0.023333333333333327,\"politics\":0.01714285714285714,\"environment\":0.039596490302668735,\"weather\":0.00857142857142857,\"disaster_accident\":0.052857142857142846},\"supertags\":[{\"id\":501672,\"positions\":[[92,100],[155,162],[647,654],[684,692],[1795,1803],[1921,1928],[1964,1972]],\"scope\":\"baseball\",\"name\":\"Manager (baseball)\",\"score\":0.7,\"contentMatch\":0.47236180904522607,\"categories\":{\"22299804\":\"Baseball strategy\",\"17908961\":\"Sports coaches\",\"846062\":\"Baseball managers\",\"2280640\":\"Baseball coaches\",\"9697509\":\"Baseball occupations\"},\"type\":1,\"senseRank\":1,\"variety\":0.7574257425742574,\"depth\":0.6470588235294117},{\"id\":337689,\"positions\":[[1950,1957]],\"name\":\"Debate\",\"score\":0.6,\"contentMatch\":0.8994974874371859,\"categories\":{\"698270\":\"Debating\"},\"type\":1,\"senseRank\":1,\"variety\":0.5891089108910892,\"depth\":0.6470588235294117},{\"id\":4692150,\"positions\":[[632,639],[887,894],[1479,1486],[1626,1633],[1837,1843],[1890,1896],[2308,2315],[2717,2724],[2909,2916]],\"name\":\"Report\",\"score\":0.6,\"contentMatch\":0.7135678391959799,\"categories\":{\"1222842\":\"Technical communication\",\"694072\":\"Documents\"},\"type\":1,\"senseRank\":1,\"variety\":0.7623762376237624,\"depth\":0.7058823529411764},{\"id\":30873450,\"positions\":[[12,19]],\"name\":\"Deposit account\",\"score\":0.6,\"contentMatch\":0.7135678391959799,\"categories\":{\"38182585\":\"Banking terms\"},\"type\":1,\"senseRank\":2,\"variety\":0.8415841584158416,\"depth\":0.6470588235294117}],\"type\":\"article\",\"cid\":1350985132,\"url\":\"http:\\/\\/mobile.slate.com\\/articles\\/business\\/moneybox\\/2013\\/08\\/microsoft_ceo_steve_ballmer_retires_a_firsthand_account_of_the_company_s.single.html?original_referrer=http:\\/\\/t.co\\/yOO5N2OQxZ\"}\n","method":"GET","code":200,"relative_url":"/api/article?token=...&fields=text,title,url&fields=text,title,url&url=http://mobile.slate.com/articles/business/moneybox/2013/08/microsoft_ceo_steve_ballmer_retires_a_firsthand_account_of_the_company_s.single.html?original_referrer=http%3A%2F%2Ft.co%2FyOO5N2OQxZ&utm_campaign=Buffer&utm_content=buffer47791&utm_medium=twitter&utm_source=buffer"}
{"headers":[{"name":null,"value":"HTTP/1.1 200 OK"},{"name":"Vary","value":"Accept-Encoding"},{"name":"Date","value":"Tue, 21 Jan 2014 02:43:46 GMT"},{"name":"Content-Length","value":"7934"},{"name":"Content-Type","value":"application/json"},{"name":"Server","value":"Apache-Coyote/1.1"}],"body":"{\"icon\":\"http:\\/\\/cdn.macrumors.com\\/images-new\\/apple-touch-icon.png\",\"text\":\"Apple debuted a new television ad for the iPad during the NFL playoffs this weekend. The 90-second spot features a Robin Williams speech from the film Dead Poets Society overlaying the iPad being used in a variety of industries including filmmaking, mountaineering, SCUBA diving, music and more.\\nApple has also introduced a new website called \\\"Your Verse\\\" to share the stories of the iPad users behind the ad.\\nWe don\\u2019t read and write poetry because it\\u2019s cute. We read and write poetry because we are members of the human race. And the human race is filled with passion. And medicine, law, business, engineering \\u2014 these are noble pursuits and necessary to sustain life. But poetry, beauty, romance, love \\u2014 these are what we stay alive for.\\nTo quote from Whitman,\\n\\\"O me, O life of the questions of these recurring.\\nOf the endless trains of the faithless. Of cities filled with the foolish. What good amid these, O me, O life?\\nAnswer: that you are here. That life exists and identity. That the powerful play goes on, and you may contribute a verse.\\\"\\n\\\"That the powerful play goes on, and you may contribute a verse.\\\"\\nWhat will your verse be? Apple is using the spot, along with it's previously existing 'Life on iPad' website , to show the wide variety of ways the iPad can be used in work and life. Back in October, alongside the iPad Air launch, Apple ran a 'Pencil' television spot that touted the iPad's usefulness in \\\"classrooms, boardrooms, expeditions\\\" and in space.\",\"date_created\":\"Sun, 12 Jan 2014 11:08:40 PST\",\"date\":\"Sun, 12 Jan 2014 18:48:00 GMT\",\"type\":\"article\",\"cid\":1331963972,\"human_language\":\"en\",\"url\":\"http:\\/\\/www.macrumors.com\\/2014\\/01\\/12\\/your-verse-ipad-ad\\/\",\"author\":\"Sunday January 12, 2014 10:48 am PST by Jordan Golson\",\"title\":\"Apple Debuts New Poetry-Themed 'Your Verse' iPad TV Ad\",\"html\":\"<div><div class=\\\"image_frame\\\"><img alt=\\\"Advertisement\\\" src=\\\"http:\\/\\/s0.2mdn.net\\/948853\\/Scion_10Series_FR-S_Brand_300x250.jpg\\\"><\\/img><div class=\\\"caption\\\">Advertisement<\\/div><\\/div><div><div>\\n\\t\\t\\t\\t\\t\\tApple debuted a new television ad for the iPad during the NFL playoffs this weekend. The 90-second spot features a Robin Williams speech from the film <a href=\\\"http:\\/\\/www.youtube.com\\/watch?v=aS1esgRV4Rc\\\">Dead Poets Society<\\/a> overlaying the iPad being used in a variety of industries including filmmaking, mountaineering, SCUBA diving, music and more.<br><br>\\nApple has also <a href=\\\"http:\\/\\/www.apple.com\\/your-verse\\/\\\">introduced a new website<\\/a> called "Your Verse" to share the stories of the iPad users behind the ad. <br><br><blockquote>We don’t read and write poetry because it’s cute. We read and write poetry because we are members of the human race. And the human race is filled with passion. And medicine, law, business, engineering — these are noble pursuits and necessary to sustain life. But poetry, beauty, romance, love — these are what we stay alive for.<br><br>\\nTo quote from Whitman,<br><br>\\n"O me, O life of the questions of these recurring.<br>\\nOf the endless trains of the faithless. Of cities filled with the foolish. What good amid these, O me, O life?<br>\\nAnswer: that you are here. That life exists and identity. That the powerful play goes on, and you may contribute a verse."<br><br>\\n"That the powerful play goes on, and you may contribute a verse."<br><br>\\nWhat will your verse be?<\\/blockquote>Apple is using the spot, along with it's previously existing <a href=\\\"http:\\/\\/www.apple.com\\/ipad\\/life-on-ipad\\/\\\">'Life on iPad' website<\\/a>, to show the wide variety of ways the iPad can be used in work and life. Back in October, alongside the iPad Air launch, Apple ran a <a href=\\\"http:\\/\\/www.macrumors.com\\/2013\\/10\\/23\\/apple-posts-ipad-air-introduction-life-on-ipad-and-pencil-tv-ad-videos\\/\\\">'Pencil' television spot<\\/a> that touted the iPad's usefulness in "classrooms, boardrooms, expeditions" and in space.\\n <\\/div><\\/div><div class=\\\"image_frame\\\"><img alt=\\\"Advertisement\\\" src=\\\"http:\\/\\/s0.2mdn.net\\/viewad\\/1361549\\/FY14_Nissan_CPO_728x90_V2.jpg\\\"><\\/img><div class=\\\"caption\\\">Advertisement<\\/div><\\/div><\\/div>\",\"categories\":{\"entertainment_culture\":0.03542857142857143,\"hospitality_recreation\":0.011428571428571429,\"other\":0.0,\"business_finance\":0.002857142857142857,\"technology_internet\":0.0890952380952381,\"socialissues\":0.022857142857142857,\"sports\":0.39933333333333326,\"humaninterest\":0.09185714285714286,\"religion_belief\":0.054285714285714284,\"war_conflict\":0.0,\"education\":0.005714285714285714,\"health_medical_pharma\":0.0619047619047619,\"labor\":0.02,\"law_crime\":0.1142384513306551,\"politics\":0.03142857142857143,\"environment\":0.022428215336011578,\"weather\":0.0,\"disaster_accident\":0.037142857142857144},\"supertags\":[{\"id\":21211,\"positions\":[[114,117]],\"name\":\"National Football League\",\"score\":0.8,\"contentMatch\":0.9236111111111112,\"categories\":{\"27603183\":\"American football leagues in the United States\",\"33170718\":\"Sports leagues established in 1920\",\"31480721\":\"1920 establishments in the United States\",\"841731\":\"National Football League\",\"30960960\":\"Cooperatives in the United States\",\"33866299\":\"Professional sports leagues in the United States\"},\"type\":1,\"senseRank\":1,\"variety\":0.26771653543307083,\"depth\":0.5882352941176471},{\"id\":2861,\"positions\":[[52,54],[87,89],[462,464]],\"name\":\"Advertising\",\"score\":0.7,\"contentMatch\":1,\"categories\":{\"884913\":\"Advertising\",\"1232608\":\"Promotion and marketing communications\",\"999626\":\"Graphic design\",\"7406890\":\"Communication design\",\"718764\":\"Finance\"},\"type\":1,\"senseRank\":2,\"variety\":0.2440944881889764,\"depth\":0.7058823529411764},{\"id\":8221,\"positions\":[[207,211]],\"name\":\"Death\",\"score\":0.7,\"contentMatch\":0.9236111111111112,\"categories\":{\"921954\":\"Death\",\"2389032\":\"Life\",\"3265614\":\"Demography\",\"37107764\":\"Senescence\"},\"type\":1,\"senseRank\":1,\"variety\":0.2755905511811023,\"depth\":0.7647058823529411},{\"id\":53242,\"positions\":[[171,185]],\"name\":\"Robin Williams\",\"score\":0.7,\"contentMatch\":0.6666666666666666,\"categories\":{\"1005370\":\"1951 births\",\"8273176\":\"People from Oakland County, Michigan\",\"1027717\":\"Science fiction fans\",\"5179534\":\"American stand-up comedians\",\"5598416\":\"American Episcopalians\",\"36085024\":\"Detroit Country Day School alumni\",\"25909468\":\"Outstanding Performance by a Cast in a Motion Picture Screen Actors Guild Award winners\",\"2834619\":\"American film actors\",\"8330045\":\"People from Marin County, California\",\"2834699\":\"American television actors\",\"15947829\":\"Cecil B. DeMille Award Golden Globe winners\",\"21357091\":\"Actors from California\",\"38633811\":\"21st-century American actors\",\"4801090\":\"Audio book narrators\",\"8278761\":\"American impressionists (entertainers)\",\"36931540\":\"Grammy Award-winning artists\",\"38608301\":\"20th-century American actors\",\"26497101\":\"American people of German descent\",\"15947643\":\"Best Musical or Comedy Actor Golden Globe (television) winners\",\"2554933\":\"American voice actors\",\"10147153\":\"Juilliard School alumni\",\"15947615\":\"Best Musical or Comedy Actor Golden Globe (film) winners\",\"26294350\":\"Living People\",\"26502512\":\"American people of Scottish descent\",\"3190124\":\"American comedians\",\"6202968\":\"Best Supporting Actor Academy Award winners\",\"26498486\":\"American people of Irish descent\",\"38424647\":\"Outstanding Performance by a Male Actor in a Supporting Role Screen Actors Guild Award winners\",\"22128351\":\"Actors from Chicago, Illinois\"},\"type\":1,\"senseRank\":1,\"variety\":0.5039370078740157,\"depth\":0.6470588235294117}],\"media\":[{\"link\":\"http:\\/\\/s0.2mdn.net\\/948853\\/Scion_10Series_FR-S_Brand_300x250.jpg\",\"primary\":\"true\",\"caption\":\"Advertisement\",\"type\":\"image\"},{\"link\":\"http:\\/\\/s0.2mdn.net\\/viewad\\/1361549\\/FY14_Nissan_CPO_728x90_V2.jpg\",\"caption\":\"Advertisement\",\"type\":\"image\"}]}\n","method":"GET","code":200,"relative_url":"/api/article?token=...&fields=text,title,url&fields=text,title,url&url=http://www.macrumors.com/2014/01/12/your-verse-ipad-ad/"},
Which as you can see at the top indicates it returned an error. Yet... a little further down it returns what the API should: a bunch of JSON objects. It even records it on my Diffbot account as successful API calls.
And I know it indicates it wants JSON to be passed, but if I change AFFormURLParameterEncoding to AFJSONParameterEncoding I just get this:
Error Domain=AFNetworkingErrorDomain Code=-1011 "Expected status code in (200-299), got 401" UserInfo=0xc3f91e0 {NSLocalizedRecoverySuggestion={"error":"Not authorized API token.","errorCode":401}, AFNetworkingOperationFailingURLRequestErrorKey=<NSMutableURLRequest: 0xc2d2bf0> { URL: http://diffbot.com/api/batch }, NSErrorFailingURLKey=http://diffbot.com/api/batch, NSLocalizedDescription=Expected status code in (200-299), got 401, AFNetworkingOperationFailingURLResponseErrorKey=<NSHTTPURLResponse: 0xc0b2e10> { URL: http://diffbot.com/api/batch } { status code: 401, headers {
Connection = "keep-alive";
"Content-Encoding" = gzip;
"Content-Type" = "application/json;charset=UTF-8";
Date = "Tue, 21 Jan 2014 02:55:06 GMT";
Server = "nginx/1.3.4";
"Transfer-Encoding" = Identity;
Vary = "Accept-Encoding";
} }}
Is this an AFNetworking bug (unlikely, but I am using AFNetworking 1.0 still), or something I'm doing foolishly (more likely)?
Looks like you are not alone with this issue. The first error you included seems to be sending more than one server response, hence appearing as "text/html".
You can try the following "hackish" solution and add "text/html" to your acceptable types. See if this helps:
https://stackoverflow.com/a/12726827/1134576
Your second error is a 401 error, with is '401 Unauthorized' which means there is a credentials error.

Reintroducing spaces into a document

Imagine we have some reference text on hand
Four score and seven years ago our fathers brought forth on this
continent a new nation, conceived in liberty, and dedicated to the
proposition that all men are created equal. Now we are engaged in a
great civil war, testing whether that nation, or any nation, so
conceived and so dedicated, can long endure. We are met on a great
battle-field of that war. We have come to dedicate a portion of that
field, as a final resting place for those who here gave their lives
that that nation might live. It is altogether fitting and proper that
we should do this. But, in a larger sense, we can not dedicate, we can
not consecrate, we can not hallow this ground. The brave men, living
and dead, who struggled here, have consecrated it, far above our poor
power to add or detract. The world will little note, nor long remember
what we say here, but it can never forget what they did here. It is
for us the living, rather, to be dedicated here to the unfinished work
which they who fought here have thus far so nobly advanced. It is
rather for us to be here dedicated to the great task remaining before
us—that from these honored dead we take increased devotion to that
cause for which they gave the last full measure of devotion—that we
here highly resolve that these dead shall not have died in vain—that
this nation, under God, shall have a new birth of freedom—and that
government of the people, by the people, for the people, shall not
perish from the earth.
and we receive snippets of that text back to us with no spaces or punctuation, and some characters deleted, inserted, and substituted
ieldasafinalrTstingplaceforwhofoughtheregavetheirliZesthatthatn
Using the reference text what are some tools (in any programming language) we can use to try properly space the words
ield as a final rTsting place for who fought here gave their liZes that that n
correcting errors is not necessary, just spacing
Weird problem you've got there :)
If you can't rely on capitalization for hints, just lowercase everything to start with.
Then get a dictionary of words. Maybe just a wordlist, or you could try Wordnet.
And a corpus of similar, correctly spaced, material. If suitable, download the Wikipedia dump. You'll need to clean it up and break into ngrams. 3 grams will probably suit the task. Or save yourself the time and use Google's ngram data. Either the web ngrams (paid) or the book ngrams (free-ish).
Set a max word length cap. Let's say 20chars.
Take the first char of your mystery string and look it up in the dictionary. Then take the first 2 chars and look them up. Keep doing this until you get to 20. Store all matches you get, but the longest one is probably the best. Move the starting point 1 char at a time, through your string.
You'll end up with an array of valid word matches.
Loop through this new array and pair each value up with the following value, comparing it to the original string, so that you identify all possible valid word combinations that don't overlap. You might end up with 1 output string, or several.
If you've got several, break each output string into 3-grams. Then lookup in your new ngram database to see which combinations are most frequent.
There might also be some time-saving techniques like starting with stop words, checking them in a dictionary combined with incremental letter either side, and adding spaces there first.
... or I'm over-thinging the whole issue and there's an awk one liner that someone will humble me with :)
You can do this using edit distance and finding the minimum edit distance substring of the reference. Check out my answer (PHP implementation) to a similar question here:
Longest Common Substring with wrong character tolerance
Using the shortest_edit_substring() function from the above link, you can add this to do the search after stripping out everything but letters (or whatever you want to keep in: letters, numbers, etc.) and then correctly map the result back to the original version.
// map a stripped down substring back to the original version
function map_substring($haystack_letters,$start,$length,$haystack, $regexp)
{
$r_haystack = str_split($haystack);
$r_haystack_letters = $r_haystack;
foreach($r_haystack as $k => $l)
{
if (preg_match($regexp,$l))
{
unset($r_haystack_letters[$k]);
}
}
$key_map = array_keys($r_haystack_letters);
$real_start = $key_map[$start];
$real_end = $key_map[$start+$length-1];
$real_length = $real_end - $real_start + 1;
return array($real_start,$real_length);
}
$haystack = 'Four score and seven years ago our fathers brought forth on this continent a new nation, conceived in liberty, and dedicated to the proposition that all men are created equal. Now we are engaged in a great civil war, testing whether that nation, or any nation, so conceived and so dedicated, can long endure. We are met on a great battle-field of that war. We have come to dedicate a portion of that field, as a final resting place for those who here gave their lives that that nation might live. It is altogether fitting and proper that we should do this. But, in a larger sense, we can not dedicate, we can not consecrate, we can not hallow this ground. The brave men, living and dead, who struggled here, have consecrated it, far above our poor power to add or detract. The world will little note, nor long remember what we say here, but it can never forget what they did here. It is for us the living, rather, to be dedicated here to the unfinished work which they who fought here have thus far so nobly advanced. It is rather for us to be here dedicated to the great task remaining before us—that from these honored dead we take increased devotion to that cause for which they gave the last full measure of devotion—that we here highly resolve that these dead shall not have died in vain—that this nation, under God, shall have a new birth of freedom—and that government of the people, by the people, for the people, shall not perish from the earth.';
$needle = 'ieldasafinalrTstingplaceforwhofoughtheregavetheirliZesthatthatn';
// strip out all non-letters
$regexp_to_strip_out = '/[^A-Za-z]/';
$haystack_letters = preg_replace($regexp_to_strip_out,'',$haystack);
list($start,$length) = shortest_edit_substring($needle,$haystack_letters);
list($real_start,$real_length) = map_substring($haystack_letters,$start,$length,$haystack,$regexp_to_strip_out);
printf("Found |%s| in |%s|, matching |%s|\n",substr($haystack,$real_start,$real_length),$haystack,$needle);
This will do the error correction as well; it's actually easier to do it than to not do it. The minimum edit distance search is pretty straightforward to implement in other languages if you want something faster than PHP.

Best way to detect and store path combinations for analysing purpose later

I am searching for ideas/examples on how to store path patterns from users - with the goal of analysing their behaviours and optimizing on "most used path" when we can detect them somehow.
Eg. which action do they do after what, so that we later on can check to see if certain actions are done over and over again - therefore developing a shortcut or assembling some of the actions into a combined multiaction.
My first guess would be some sort of "simple log", perhaps stored in some SQL-manner, where we can keep each action as an index and then just record everything.
Problem is that the path/action might be dynamically changed - even while logging - so we need to be able to take care of this fact too, when looking for patterns later.
Would you log everthing "bigtime" first and then POST-process every bit of details after some time or do you have great experience with other tactics?
My worry is that this is going to take up space, BIG TIME while logging 1000 users each day for a month or more.
Hope this makes sense and I am curious to see if anyone can provide sample code, pseudocode or perhaps links to something usefull.
Our tools will be C#, SQL-database, XML and .NET 3.5 - clients could also get .NET 4.0 if needed.
Patterns examples as we expect them
...
User #1001: A-B-A-A-A-B-C-E-F-G-H-A-A-A-C-B-A
User #1002: B-A-A-B-C-E-F
User #1003: F-B-B-A-E-C-A-A-A
User #1002: C-E-F
...
etc. no real way to know what they do next nor how many they will use, how often they will do it.
A secondary goal, if possible, if we later on add a new "action" called G (just sample to illustrate, there will be hundreds of actions) how could we detect these new behaviours influence on the previous patterns.
To explain it better, my thought here would be some way to detect "patterns within patterns", sort of like how compressions work, so that "repeative patterns" are spottet. We dont know how long these patterns might be, nor how often they might come. How do we break this down into "small bits and pieces" - whats the best approach you think?
I am not sure what you mean by path, but, if you gave every action in a path a unique symbol, you could reduce the problem to longest common substring or subsequence.
Or have a map of paths to the number of times that action occurred. Every time a certain path happens, increment the count for that path. Then sort to find the most common.
Pseudo idea/implementation so far
Log ever users action into a list/series of actions, bulk kinda style (textfiles/SQL - what ever, just store the whole thing for post-processing)
start counting every "1 action", "2 actions", "3 actions" up til a certain amount (lets say 30 levels)
sort them all, by giving values of importants to some of the actions (might be those producing end results)
A usefull result perhaps?
If we count all [A], [A-A], [A-B], [A-C], [A-A-A], [A-A-B] etc. its going to make a LONG and fine list of which actions are used in row frequently, and thats in the right direction, because if some of these results gets too high, we might need a shorter path. Problem is then, whats too few actions to be optimized and whats the longest needed actionlist to search for? My guess is that we need to do this counting first, then examine the numbers.
Problem is that this would be part of an analyzing tool we are developing and we dont have data until implementation, so we dont know what to look for before its actually done. hmm... wondering if there really IS an answer to this one.

Resources