Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 11 years ago.
Improve this question
I want to search on Google Yahoo, for forums and blog posts limited to a specific country. The results will be saved to a database for sorting and further processing.
From each search result, I need :
the URL itself
date and time
the domain
I am working on a program, that accepts keywords as input, and the program will automatically search on Google and Yahoo and save the results to a database.
function OnLoad() {
// Create a search control
var searchControl = new google.search.SearchControl();
// Add in a full set of searchers
var localSearch = new google.search.LocalSearch();
searchControl.addSearcher(localSearch);
searchControl.addSearcher(new google.search.WebSearch());
searchControl.addSearcher(new google.search.VideoSearch());
searchControl.addSearcher(new google.search.BlogSearch());
searchControl.addSearcher(new google.search.NewsSearch());
searchControl.addSearcher(new google.search.ImageSearch());
searchControl.addSearcher(new google.search.BookSearch());
searchControl.addSearcher(new google.search.PatentSearch());
// Set the Local Search center point
localSearch.setCenterPoint("New York, NY");
// tell the searcher to draw itself and tell it where to attach
searchControl.draw(document.getElementById("searchcontrol"));
// execute an inital search
searchControl.execute("VW GTI");
}
google.setOnLoadCallback(OnLoad);
This code is from the Google AJAX search API, however there seems not to be a way to specify the domain, country, date and time as search criteria. Moreover, it returns the result in HTML, which is hard to slice up and save as search results entries to the DB.
EDITED to describe my specific problem.
Parsing the raw HTML should be your last resort here. If they change the markup, you have to redesign your parser. That is pretty much guaranteed to happen before the "3 years" time period that you have mentioned with Google's AJAX Search API.
Related
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 11 months ago.
Improve this question
Suppose there's a website that has a list of details of some companies, for example, name, HQ area, turnover, etc. How do I scrape that data and fill it into different columns (like name, turnover) with each row having the details of a separate company?
Google Sheets allow you to import html tables or list with the IMPORTHTML(url, query, index) function.
For example, using the Wikipedia page List of largest companies by revenue as an example.
We want the data from the main table, so the first thing that we have to do, is to know what index it occupies in the page. To do this, we can use document.querySelectorAll('table') or $$('table'), as you can see from the result, the table that we want is in the position 5 of the array, so inside our google sheet we can use:
=IMPORTHTML("https://en.wikipedia.org/wiki/List_of_largest_companies_by_revenue","table",5)
From here, you should change the query parameter to list and find what index it occupies within the page using the method described above. In any case, you could always use IMPORTXML(url, xpath_query), and knowing the XPath of the information, you could come up with a similar solution.
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 8 years ago.
Improve this question
I'm learning Ruby and Rails while also trying to lose weight. To that end, I've made my first project a dieting app. Each time you eat something, you note its name, caloric value, and sugar, and then you can check a page showing your ongoing calorie deficit, a graph of your history, and so on.
The problem: Each meal has its own record, but I need to sum every meal from a given day up into a "Tuesday's total calories" value to use in a table. Before I duct-tape together some sloppy solution, I thought I'd ask what the proper way to do something like that is. Is there an ActiveRecord method I can use? If I write my own code to iterate over a month's worth of queries to build up 31 daily values, should I be doing that in my controller, or is there a way to do it in the model?
Iterating over all the meals would be bad, performance wise. Especially for people who do have many meals ;) It would load all the Meal records from the database, and then into memory.
ActiveRecord offers an interface to use the calculation functions provided by your database. It would only take a single query to let the database sum up the calories for a given day, week or month.
You'd do something to this end:
total_calories = Meal.where('DATE(created_at) = ?', Date.today).sum(:calories)
If you want to retrieve all the meals for the last 30 days with a single SQL query, you can do something like the code below. Notice how it still takes advantage of the sum method even with grouping:
Meal.where(created_at: 1.month.ago..Time.now)
.group('DATE(created_at)').order('date_created_at').sum(:calories)
Read about the calculation methods in the Rails documentation.
You can count and group by date in one query:
Meal.select('DATE(created_at) AS ordered_date, SUM(calories) AS calories').
group('DATE(created_at)').
order('ordered_date')
Perhaps you want to at a time range:
where(:created_at => (Time.now.beginning_of_month..Time.now))
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 8 years ago.
Improve this question
I have a simple search query that returns list of buildings based zip codes. The zip codes are stored in address table with a one to one relation on buildings and address. Now I need to search the buildings based on multiple zip codes.
<input type="text"
name ="fs_Zip[]"
class="input-small text-tip"
data-original-title="Enter Zip Code to search."
placeholder ="Zip Code"/>
this is the code in my view. and in controller
buildings = buildings.where('address.zip in (?)', params['fs_Zip'])
if (!params["fs_Zip"].blank? && params["fs_Zip"] != 'zipcode')
But it does not give desired result. Any help.
If I understand correctly, your query should look something more like this:
buildings = Building.joins(:address).where('addresses.zip in (?)', params['fs_Zip'])
But you need to make sure params['fs_Zip'] is in the proper format for that query (i.e. an array of zips).
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking for code must demonstrate a minimal understanding of the problem being solved. Include attempted solutions, why they didn't work, and the expected results. See also: Stack Overflow question checklist
Closed 8 years ago.
Improve this question
Hi I'm new to Ruby on Rails. I created Users with device, on profile creation the user enters a numeric field that's limited to 5 numbers. Now I need to get the most common numbers from all users. How could I do that?
I want to show most common numbers in my admin controller, please help.
You can get the array of numeric fields by doing
array = User.all.map(&:numeric_field) # assuming there is a numeric_field column
Then you can get the most common number (called the "mode") with several different methods, referenced by this post: Ruby: How to find item in array which has the most occurrences?
If you have a lot of users, this approach can hog memory, in which case you can do the operation in batches:
array = []
User.find_in_batches do |users|
array += users.map(&:numeric_field)
end
And then just use the array as before.
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 8 years ago.
Improve this question
I currently have a large set of objects in SAS (>100000) each with about 60 columns of data, including an ID number. There are many duplicate ID numbers in this set of data. My goal is to convert the ID numbers that I currently have into another form of ID number using a piece of software that I have. The problem is that when I input the ID numbers into the software, the converted output comes back without the duplicates, which I need. Is there any way to use the output ID numbers to somehow create a list of output IDs except with the duplicates that the original set of data had. Any language or piece of software would be fine.
Here is a illustration of what I described above.
Original IDs: 086516 677240 449370 677240 941053 449370
Output: 147244 147947 147957 148021
Preferred Output: 147244 147947 147957 147947 148021 147957
You can merge on the ID using a MERGE statement, and it will append the value to each of the records with the same ID value.
data want;
merge have(in=a) newIDs(in=b);
by id;
if a and b;
run;