So I'm making an iphone application that needs to scrape a website once everyday.
What I'm going to scrape is a table of upcoming games for that same day for a soccer division. Thats why i need the app to scrape from the same page and same table once everyday to keep the upcoming games updated.
I was referred to import.io but they didn't have something like a schedule re-crawl.
I would love to get some ideas and tips to how i should do this since I'm stuck now.
You might take a look at https://www.kimonolabs.com/
I played around with the service a while back and was impressed with how easy it way to set up. They have a "free" option so long as the APIs you create are not private.
Oh, and I agree with Paul, screen scraping is not something the iOS client should be doing. Too fragile, and when (not if) something breaks, you will need to go through an Apple review process to fix it.
This doesn't seem like something an app should do, your server should do it (so that the scraping is only performed once), and your clients can retrieve it from your server. That also means you could send out push notifications for important fixtures etc. Maybe that's what you meant, anyway.
If it's on the server you can just setup a scheduler (in Java, for example) to run once every x hours (probably a smaller number than 24 assuming you don't know when the website is to be updated). Then your app can just get the latest list of fixtures from your server on startup, pull-to-refresh, etc. Presumably someone will open your app, look at the fixtures, then come out of your app - so it doesn't seem like you need to cover the case where someone is in your app all day, but if you did you could use NSTimer to run every x minutes after the initial on-startup server call.
Related
Two Quick questions:
Are the seconds value of the clock/time on all iPhones around the world the same? If not, are they close at least & how close?
Do the seconds value of the clock/time on all iPhones around the world change/increment at the exact same time?
Upon request, I'm editing this post and adding the purpose for asking such questions:
I'm trying to make a corporate app that can play a video on multiple iPhones around the world at the exact same time (or as close as possible, ideally the exact same moment). Could you please guide me on how to do this?
Much thanks in advance!
To answer your actual question per se,
Are the seconds value of the clock/time on all iPhones around the world the same?
The fact is, yes, 99.9999% of iPhone users simply use the "get time from a server" system which is of course built in to any phone now.
(Indeed, this simply applies to any Windows, Mac, Android, iOS, etc etc device.)
Yes, they are all "about the same".
You could rely on it being within a second or two, probably even closer.
You cannot rely on it being closer than that.
It seems that you
I'm trying to make a corporate app that can play a video on multiple iPhones around the world at the exact same time
Synchronization in general and streaming video (is it streaming?) is a well-explored (and rather technical) branch of software engineering. (For example a massive amount of game engineering, which is a huge field, relates to inside-the-frame synchronization.)
This is not something you can learn how to do in five minutes, and it requires a stack of device-cloud stuff. Go ahead and ask new questions about this, or just google to get started! Good luck.
Regarding using push notifications.
With the push notification you would send a time. You would not send a "command to play it".
So say right now it is (example) 09:13:28. You would pick a time in the future by a couple minutes. So let's say 09:14:30.
Then using a push notification you would send that information "09:14:30" (and a video file name) to everyone connected. (You'd be sending a "command" as it were, to play video X at 09:14:30.)
Then every device would in fact play the video at 09:14:30 (simply using the local clock, as asked in your question).
Be aware that sending push notifications is extremely sloppy and slow. It can take any amount of time from 5 seconds to a minute, AND quite often there are delays beyond that (ie ten minutes or the like).
I personally would not even bother starting to experiment with push notifications, for the project you describe.
These days, making apps is entirely about using device-cloud services, such as Firebase. Everything is about "OCC" - occasionally connected computing.
(So, you can't get a job "making apps" anymore - i.e. if you know how to move buttons around on an iPhone screen. You get a job because you can make a total, live, device-cloud system - indeed such as you are making.)
Indeed your example project is the perfect such "demo" project for learning about how to do modern apps.
Simply use Firebase to sync everything up.
You'll essentially put a piece of information on Firebase ("play video X at 09:14:30") and that information will be communicated fairly quickly/reliably to everyone connected.
For the particular task you describe, I personally would use PubNub which is faster than Firebase and basically made for game-like problems precisely like you describe.
http://pubnub.com
If you truly needed performance/reliability better than pubnub, you are really talking major engineering. So, the (buildings of) engineers who make live games at Nintendo, Warcraft etc, would tackle such an issue as "being even faster than PubNub".
So, the answer in brief!
The very short answer then to your question posed is:
Learn to use the various device-cloud services, which are at the heart of all apps today. (Knowing how to make "an Android or iOS app", as such, is of no consequence today.) For your particular problem, you'll want to use PubNub specifically, as it is built for precisely realtime problems such as this. (Firebase more leans towards "OCC" type data problems.)
Really that's it.
Is there a way in Rails to send out a request at a certain time?
I'm using an external credit card charging API, and I want to adjust each monthly subscription based on how many referrals they have (10% each, 10 referrals max). The API has a beta referral system built in, but it doesn't seem to work the way I need it to. Plus, there are just too many unknowns that I'd rather not get into at the moment. I just want to get it up and working, and since my system is fairly simple, I'd rather just do it manually.
There's a billing date for each subscription, and what I want to do is just manually adjust the price of the subscription based on how many active users there are containing the referral code of the user being charged. I'd like to just send out this request to the API just before they're billed. Like sometime around subscription.next_billing_at - 1.minute.
Then just set the subscription.price to price - (price * (User.where(referral_code: current_user_code)).count / 10).
I'm aware this is far from an optimal approach, considering the amount of extra requests being made each month, but since we're small right now, it shouldn't be a problem. Again, it's just a temporary solution so we can get things running now.
There are two options which directly answer your question.
Write a rake task and run it daily with cron via the Whenever gem. If you take this approach, you will have to have the task just load all subscriptions which are due to be billed in the next cycle and update them as required.
Alternatively, use something like Resque-scheduler, which would allow you to run some task at next_billing_at - 1.minute or something.
But if you are small, why not just update the price every time a new referral is created using a callback? Unless there are specific rate or query limits on this API, I doubt a card processor is going to be affected by the traffic you generate. Of course if there are other requirements, like, a referral only applies after a month or something like that, you are going to be stuck with one of the first 2 options, and the Cron + Rake task is probably the best solution in that case.
I'd like to infrequently open a Twitter streaming connection with TweetStream and listen for new statuses for about an hour.
How should I go about opening the connection, keeping it open for an hour, and then closing it gracefully?
Normally for background processes I would use Resque or Sidekiq, but from my understanding those are for completing tasks as quickly as possible, not chilling and keeping a connection open.
I thought about using a global variable like $twitter_client but that wouldn't horizontally scale.
I also thought about building a second application that runs on one box to handle this functionality, but that seems excessive if it can be integrated into the main app somehow.
To clarify, I have no trouble starting a process, capturing tweets, and using them appropriately. I'm just not sure what I should be starting. A new app? A daemon of some sort?
I've never encountered a problem like this, and am completely lost. Any direction would be much appreciated!
Although not a direct fix, this is what I would look at:
Time
You're working with time, so I'd look at what time-centric processes could be used to induce the connection for an hour
Specifically, I'd look at running a some sort of job on the server, which you could fire at specific times (programmatically if required), to open & close the connection. I only have experience with resque, but as you say, it's probably not up to the job. If I find any better solutions, I'll certainly update the answer
Storage
Once you've connected to TweetStream, you'll want to look at how you can capture the tweets for that time period. It seems a waste to create a data table just for the job, so I'd be inclined to use something like Redis to store the tweets that you need
This can then be used to output the tweets you need, allowing you to simulate storing / capturing them, but then delete them after the hour-window has passed
Delivery
I don't know what context you're using this feature in, so I'll just give you as generic process idea as possible
To display the tweets, I'd personally create some sort of record in the DB to show the time you're pinging TweetStream that day (if it changes; if it's constant, just set a constant in an initializer), and then just include some logic to try and get the tweets from Redis. If you're able to collect them, show them as you wish, else don't print anything
Hope that gives you a broader spectrum of ideas?
I'm working on an iOS app for having a list of top 10 of everything.
So the problem I'm facing is how do I update the list every week ?
I thought of linking it to external links but then I was unable to format the webpage according to the interface of my app.
Lastly I want the app not only to display the top 10 list but also to be able to interact. For example if the user enters into top 10 songs then he should be able to play the tracks as well ! So for this I guess I'll be needing a database which I could update weekly. I don't know how to go about this. Please help !
You need a remote server, running with an application that handles the list generation/updates, and you will probably need also to manage a remote database, if you expect that the number of item will be reasonably large.
Your ios application will have to interact with your remote server using an API that you will have to define as well.
I'm sorry to be that generic, but the hard truth is that there is no quick or easy way to do it: you will need to learn and possibly develop every part of your system and then integrate them.
You have nearly an infinite number of options in terms of platforms/languages/databases to choose from, for the backend part.
Over the last few years, there have been services that should make handling the remote backend easier (Stackmob, cloudbase...) but it really depends on much control you want to have on the "lists" you want to manage.
I'm creating an online multiplayer board game using Rails, and I want the game to be able to be played without having to manually refresh the page. I'm planning on using websockets or Server Sent Events's to update the page. I haven't decided which yet, but I don't expect it to make a difference for my question.
The thing I'm confused about is knowing when to send an update to the client. For example, let's say I was making chess, and player1 took a turn. They would send up their move to the server, the server would then save it to the database. How does player2's websocket or SSE know to push the info about player1's move to player2's brower?
I know they could regularly query the database to see if the game has been updated, but that seems like it would be hard to scale since every user would be querying the database very often.
Is there an easy way to wait for an update to the database and be notified of it automatically somehow? Or are there other solutions?
I've looked around online a bunch for solutions to this, but haven't really gotten any concrete ideas. I'm definitely open to using a library or whatever to solve the problem as long as it's free. I've seen some paid solutions but those aren't an option.
I also saw someone suggest somehow using node.js alongside rails to do something with realtime. I don't know much about node.js, but I have heard it's good for asynchronous functionality like that. I'd be interested in learning how to use it if it's better than any rails solution.
I'm using Rails 3.2, but if Rails 4 has something that would work well here, I'd probably be interested in that as well.
As nik suggested, use the FAYE Ruby Server if you're a ruby developer.
In terms of how things could hang together:
Each game is a channel
Each player subscribes to this channel
When a player makes a move, the move is sent to the server
The server saves the move against the game in the db
The server sends a message to the channel.
Both player's ui update with the new move.
This setup allows you to easily replay a whole game or continue where you left off. Just get all moves from db and send them in order to a new channel. This might not be performant, but it should work.
you can also use FAYE which has Ruby Server and Node.js server, you can choose one