I have an idea for a program that will do a particular job for me; a job which I keep forgetting to do. I am looking for some advice on how to get started with it, i.e. language recommendations that will allow me to complete the program.
The chore is as follows:
I open up a web browser, visit the web page, log in, click a tab in my account and "sign on", log out, close browser.
Due to having so much on at the moment with work I forget about this and it is important that I do it before a particular time in the morning.
Aside from alleviating me of the chore, I am keen to learn about the software technologies that will allow me to automate this and work on my novice programming skills some more.
The way I perceive this of working would be:
Write program that runs as a daemon, checks date and time every 30 minutes (to "sign on" half an hour before deadline), either calls another program or methods/functions within its own programming that opens up a socket, port and IP before deadline, automates text entry of username and password fields within the HTML of log-in page's source, automate/action the Javascript "submit" button, accept/store cookie, action the appropriate radio button's associated Javascript function for "sign on", action "log off" Javascript function, nullify cookie/session, end.
I am, unfortunately for me, a little hazy in the technical aspects that this would involve. I run Linux so perhaps the daemon will be a cron job, and the actual main bulk of the program would be methods/functions called up at the appropriate time before the deadline?
Is this better accomplished in Ruby, Perl, Python or Java? Or Perhaps the cron-job associated with a simple script that calls a Java program? I apologise but I am a little naive here as software isn't my subject so some advice would be greatly appreciated. So too would be if you could point me in the direction of the right resources to read as well.
I am aware of Ruby's Mechanize library, but I am only familiar with Java. However, willing to pick up another language.
This sounds like something that, depending on the structure of their website, could easily be handled by curl.
man page: http://linux.die.net/man/1/curl
In essence, you can write a cron job that calls a little bash script. Your bash script will just be a couple curl statements that log you on and request the "sign on" page.
Related
I created an application that uses Watir to automate logging in and perform a couple of functions within a site.
Right now it's written 100% purely in ruby classes that I was just executing in irb, but I want to put it into a Rails application and put it online. I haven't been able to find much information about using something like Capybara or Watir for anything other than testing. Is this because of how slow they are or is it a capabilities issue?
Would I be able to run a background process that opens a browser with Watir and performs a few functions for each user in production?
Another question I have is how to keep the session over a longer period of time. There are two sites that require 2FA that my app logs into. If I wanted to log in and perform a function once an hour with a Watir browser, I could create it as a background process (if that works). But when the process is done the browser closes and when the background process runs again in an hour it requires 2FA again.
My other worry is speed. If I have 50 users that all need to run a Watir browser at the same time I imagine that will be slow. I am not worried as much about speed as long as they run and collect the data and perform the few actions we need, but how it will effect the applications integrity.
WATIR is specifically designed as a testing tool.. WATIR stands for Web Application Testing In Ruby. It's design is centered around interacting with a browser the way a user would, effectively simulating the same actions a user would take when using a site. That it would be sub-optimal for other tasks is to be expected. Due to scraping and testing having very similar activities, there a a number of people that use watir for that task, but it is not designed for that purpose, and it is unlikely that the WATIR developers would ever add features specific to data scraping verses testing.
With the things you are contemplating, you should ask yourself if you are doing the equivalent of using a socket wrench as a hammer, and if there might be a better tool you could use.
If the sites you are interacting with support an API, then that would be the preferred way to interact with them, to get information from the site. If that is not supported, you may want to consider looking at other gems that would let you request the site HTML or parse the HTML directly (e.g. Nokogiri)
You should also inspect the terms of service for the sites you are interacting with (if you don't own them) to ensure that there are not prohibitions against using 'robots' or other automated means to access the site. If so, then using Watir in the way you propose may end up getting you banned from access to the site if the pattern of your access is obviously the result of an automated process.
This is actually done more often than people think. Maybe not specifically with Watir, but running a browser automation task in a job. The jobs should be queued and run asynchronously, preferably in a different process than your main web app.
I wrote about this strategy here: https://blogstephenarifin.wordpress.com/2018/08/23/integrating-a-third-party-without-an-api-using-rails-5-and-heroku/
If you find yourself having to use Watir then the best way is to use it to render the page (for example in headless mode for javascript), save it and then use Nokogiri to process it. Apis are suggested a lot by people who don't and can't find a use for scraping but at times is necessary and perfectly legit (you may even be scraping your own data). Apis are not a universal option.
Second you should probably regulate its use to a background job. If you have end users (and you really shouldn't have many simultaneous users) many services inform customers data will be available in a few hours to a few days
How can I know that my server is working fine i.e in good health condition.
My Requirement is Users are complaining that they can not access the web application (Web site) something like it taking long time to do, some times its not completing the request.
I want to know whether my web site is in good condition or not before users and to get an alert message.
I want to know how we can measure whether the server is very responsive or user is not facing any problem. Some times my site takes long time coz. millions of data records have to be retrieved in that case I can not depend upon response time.
please help me on this
Monitoring response time without any third party software can be done with scripts like webinject. Webinject is a perl script that execute some browsing scenario and tells you if it acceptable or not.
Run a script at a regular interval, say 10min, that will start a webinject scenario. If the scenario is ko (check the return code of your webinject call), your script can send you an email, a sms, start a sound alarm, ... whatever is relevant to you.
You can also add some complexity by running a diagnostic script (check network by pinging relevant hardware, check cpu/ram usage of your servers, check the number of sessions in your database, ...) and send the diagnostic by email. You can also save the response times in a database (like a rdd database) to have a graphical view and be able to do some problem analysis on it.
I have a program which needs to be constantly running on my Windows XP box. When I'm away from home (or even when I'm at home) I'd like to get an email notification if that program were to ever crash, which it does occasionally. If it is a scheduled task that monitors it a 5m interval between checks is OK.
Any suggestions?
You can write simple windows service to monitor your task. But what if your machine is down?
You can go a step further and integrate with AlertGrid service.
Each time your program executes some logic it should send signal to AlertGrid. Than you can set up rule like 'if my program hasn't responded for more than 5 minutes -> send Email/SMS'.
There are some examples in various programming languages how to do this, so you don't need to be a seasoned coder.
This assumes that you have access to your source code, if not you can use AlertGrid Basic monitor which will send list of available processes to AlertGrid and if your program is not on a processes list then you might get email alert.
AlertGrid's Free account is enough for this case.
PS: I am a developer in AlertGrid team. We would be glad to help you in case of any questions.
Hope that helps!
This is my very first question in stack overflow and I am quite excited. Liked a lot the interface of this site and the buoyant community!
I am building a rails app that receives text as an email and creates a post.
I ask or your expert opinion on which is the best option to receive the mail message and process it?
To send the mail to the application:
A1: X accesses a POP/IMAP account periodically (30s cronjob?) and processes the message
A2: the message is piped from the mailserver to the application X
To process the application
B1: rails with MMS2R
B2: PHP that processes the message and sends a POST in rails
B3: PERL that processes the message and sends a POST to rails
Which combination A+B would you recommend for a big volume of mails?
Any other A or B option?
Thank you very much and good luck with all your scripting!
Any of them that doesn't then perform an HTTP POST to the web app, it's an unnecessary middle man and will only slow you down under volume. Having said that, the upside is one location for any business logic if necessary. So, weigh those two points before making a decision.
Now, for this personally, I'd probably go Perl over the others, though PHP would be a closer option. That, of course, is because I don't know Ruby. For you, though, I'd probably suggest Ruby so that your language is consistent across the application. That makes maintenance a lot easier in the long run and it probably allows for sharing of business logic with the web app without needing to specific use the web app.
if you are concerned about volume, I would just use perl + db (bypass rail app).
We currently use Hp SiteScope for monitoring synthetic transactions across some of our web apps. This works pretty well except for the licensing cost for each synthetic transaction makes it prohibitive to ensure adequate coverage across our applications.
So, an alternative would be to use SiteScope's URL monitoring which can basically call a URL and then provide some basic checks for the certain strings. With that approach, I'd like to create a page that either calls a bunch of pages or try to tap into a MSTest group somehow to run tests.
In the end, I'd like a set of test cases that can be used against multiple environments to be used for production verification, uptime, status, etc.
Thanks,
Matt
Have you taken a look at System Center Operations Manager 2007?
I'm just getting started, but it appears to do what you are describing in your question.
We are looking to monitoring our data center and the a web application...from the few things I have found on the web it is going to fit our need.
Update
I've since moved to Application Insights. A great overview can be found here, https://azure.microsoft.com/en-us/documentation/articles/app-insights-monitor-web-app-availability/
There are two methods one can use, a simple ping, or record a multi-step synthetic user "experience". Basically you act as a user, and using IE and a Visual Studio Web Test project you record navigating around your site and upload that file to Azure.
For example, I record logging in, navigating a few pages, and then logging out. As long as all of those events happen in a timely manner the site is in a good operating state.
If the tests fail, take too long to respond for example, I'll get an email alerting me something isn't exactly right.