addons.mozilla.org comes with an XML API that you could fetch any public data from it's database:
For example, information about this Personas:
https://addons.mozilla.org/en-US/firefox/addon/this-blue-planet/
can be accessed as XML:
https://services.addons.mozilla.org/zh-TW/firefox/api/1.5/addon/this-blue-planet
The only problem is, unlike extensions, where the URL of XPI and the file hash is all you need to initiate an off-site install, Personas is represented as a JSON object in data-browsertheme property of the mouseover-able and click-able preview button; the content of JSON object cannot be found within the XML representation.
I've think of fetch the HTML page and parse the content, but I need a robust method that doesn't break with an AMO version update. Anyone can answer my question or point me a person that could answer the question is greatly appreciated.
PS: I am making a Firefox promotion website that work with these data, if you are interested you could check it out (in Traditional Chinese): http://gfx.tw/
The addons.mozilla.org API doesn't currently support Personas. The primary home of Personas is getpersonas.com, and it doesn't have an API.
You can file a bug for Persona support at bugzilla.mozilla.org, but it likely won't be integrated until much later in the year.
Related
I’m trying to tie data from another product with my data inside of Adobe Analytics.
We have Adobe Analytics javascript on our website collecting data and we use a third party tool to track how users interact with certain parts of the website. We’re trying to use the Adobe API to tie the data together.
So far we’ve gone down the path of using the Data Insertion API, but it wasn’t quite right as it’s meant to be used as a replacement for the JS, from what I can tell.
We also explored using the Data Sources API. Now the documentation for this suggests you can use a transaction ID to tie offline data with the data collected from the JS, we’ve tried this and it doesn’t match the data up. We’re now exploring using Visitor ID to tie the sessions together but we’re having problems uploading any rows with the Visitor ID column, Adobe just returns the error “Column header: ‘visitorid’ is not a valid column header”. We’ve tried several different variations of visitor id, such as “visitor_id”, “visitor-id”, “vistor id”, etc and still no luck.
The end goal is for us to be able to upload data to Adobe that will update/add eVars for already existing sessions earlier that day. How would I go about doing this? Is there something I'm missing or doing wrong?
Edit: I managed to solve this problem by using the Adobe SAINT API. When a user arrives at the site, we push an eVar for that user with a unique ID and then the day after we use the SAINT API and the unique ID in the eVar we pushed previously to add the additional data we needed.
It could be a good idea to look back at the Data Insertion API and combine it with the visitorId approach where you tie existing/old visitorID's to new eVars and use the timestamp to "update" the dataset.
Although this is experimental, it might be worth a try.
Best regards,
As the title says, I'm looking for a string to be input in an IOS app and stored in a way that it would be accessible from a web browser.
Both editable and readable from web and the app.
From initial googling, I've come across Dropbox Datastore API but it is being deprecated.
What would be a good alternative for this?
Your intentions are quite scant. However, I assume you want a little more than just reading and writing a string. Inferring a bit more context, you should look at Apple's CloudKit.
It has API for iOS, OS X, and apparently now javascript. See this link for more details.
You need some code running on a server and a database to save the string, with an API to access this string from a client.
FireBase is probably the simplest drop in solution since it gives you a very basic example and a web based UI for testing (plus it updates in realtime)
I am trying to build a web based flash application. I am quite new to flash. I would like to develop it in two forms - demo and paid version. For this application to act as a paid version I need to have some kind of serial key. In order to achieve this I googled and came across something like this
MD5(MD5(thisuri)+thisuri)
I think 'thisuri' points to the current url page but I don't know how to get that url and I don't know whether '+' acts as a character or an operator.
Can you please help me?
It seems that a library exists in AS3.0 : as3corelib
An ActionScript 3 Library that contains a number of classes and utilities for working with ActionScript? 3. These include classes for MD5 and SHA 1 hashing, Image encoders, and JSON serialization as well as general String, Number and Date APIs.
To use it, just download the zip file, decompress it and copy the contents of "src" directory to the root of your project.
Then in your actionscript code, simply do the following :
import com.adobe.crypto.MD5;
var hash:String = MD5.hash(”test”);
source in french
To add to #Julien's recommendation of using as3corelib, you will also need the advice from this post to get the current url: Get Current Browser URL - ActionScript 3
Somehow I think there's a more elegant way to get the url, but I don't remember it.
While this may provide you with some basic check for the paid version, a determined hacker will easily fool this algorithm. For example, I could fool the environment into thinking that its being served from a domain that you've registered as part of the "paid" version. Also, since the client has the flash code, they can decompile the binary and potentially see the algorithm you're using. Depending on what you're offering in the app, this extreme case may or may not be acceptable to you.
Look into more secure authentication mechanisms if you're serious about security.
I want to know if there is a better way of extracting info from a web page than parsing the HTML for what i'm searching. ie: Extracting movie rating from 'imdb.com'
I'm currently using the IndyHttp components to get the page and i'm using strUtils to parse the text but the content is limited.
I found plain simple regex-es to be highly intuitive and simple when dealing with good web-sites, and IMDB is a good web site.
For example the movie rating on the IMDB's movie HTML page is in a <DIV> with class="star-box-giga-star". That's VERY easy to extract using a regular expression. The following regular expression will extract the movie rating from the raw HTML into capture group 1:
star-box-giga-star[^>]*>([^<]*)<
It's not pretty, but it does the job. The regex looks for the "star-box-giga-star" class id, then it looks for the > that terminates the DIV, and then captures everything until the following <. To create a new regex like this you should use a web browser that allows inspecting elements (for example Crome or Opera). With Chrome you can simply look at the web-page, right-click on the element you want to capture and do Inspect element, then look around for easily identifiable elements that can be used to create a good regex. In this case the "star-box-giga-star" class is obviously easily identifiable! You'll usually have no problem finding such identifiable elements on good web sites because good web sites use CSS and CSS requires ID's or class'es to be able to style the elements properly.
Processing RSS feed is more comfortable.
As of the time of posting, the only RSS feeds available on the site are:
Born on this Date
Died on this Date
Daily Poll
Yet, you may make a call for adding a new one by getting in touch with the help desk.
Resources on RSS feed processing:
Relevant post here on SO.
Super Object
Wikipedia.
When scraping websites, you cannot rely on the availability of the information. IMDB may detect your scraping and attempt to block you, or they may frequently change the format to make it more difficult.
Therefore, you should always try to use a supported API Or RSS feed, or at least get permission from the web site to aggregate their data, and ensure that you're abiding by their terms. Often, you will have to pay for this type of access. Scraping a website without permission may open you up to liability on a couple legal fronts (Denial of Service and Intellectual Property).
Here's IMDB's statement:
You may not use data mining, robots, screen scraping, or similar
online data gathering and extraction tools on our website.
To answer your question, the better way is to use the method provided by the website. For non-commercial use, and if you abide by their terms, you can download the IMDB database directly and use the data from there instead of scraping their site. Simply update your database frequently, and it's a better solution than scraping the site. You could even wrap your own web API around it. Ratings are available as a standalone table.
Use HTML Tidy to convert any HTML to valid XML and then use an XML parser, maybe using XPATH or developing your own code (which is what I do).
All the answers posted cover well your generic question. I usually follow an strategy similar to the one detailed by Cosmin. I use wininet and regex for most of my web extraction needs.
But let me add my two cents at the specific subquestion on extracting imdb qualification. IMDBAPI.COM provides a query interface returning json code, which is very handy for this type of searches.
So a very simple command line program for getting a imdb rating would be...
program imdbrating;
{$apptype console}
uses htmlutils;
function ExtractJsonParm(parm,h:string):string;
var r:integer;
begin
r:=pos('"'+Parm+'":',h);
if r<>0 then
result:=copy(h,r+length(Parm)+4,pos(',',copy(h,r+length(Parm)+4,length(h)))-2)
else
result:='N/A';
end;
var h:string;
begin
h:=HttpGet('http://www.imdbapi.com/?t=' + UrlEncode(ParamStr(1)));
writeln(ExtractJsonParm('Rating',h));
end.
If the page you are crawling is valid XML, i use SimpleXML to extract infos. Works pretty well.
Resource:
Download link.
I just have a link to a product page, at amazon. How do I get all the information (photo, price etc), in my ruby program, just using this link?
Here's the list of supported urls as disclosed by amazon for their oembed, product advertising API would come to picture only after parsing through these URLs and getting the ASINs
http://*amazon.*/gp/product/*
http://*amazon.*/*/dp/*
http://*amazon.*/dp/*
http://*amazon.*/o/ASIN/*
http://*amazon.*/gp/offer-listing/*
http://*amazon.*/*/ASIN/*
http://*amazon.*/gp/product/images/*
http://*amazon.*/gp/aw/d/*
http://www.amzn.com/*
http://amzn.com/*
I found this library (I'm using Rails)
amazon-ecs
I'm experimenting with it. Still, I'd require some kind of ID (product id?) to get details of a particular product. For example, consider this link to kindle
http://www.amazon.com/Kindle-Amazons-Wireless-Reading-Generation/dp/B00154JDAI/ref=amb_link_84372271_1?pf_rd_m=ATVPDKIKX0DER&pf_rd_s=center-1&pf_rd_r=06JJGQP9J3BHKPE38SXP&pf_rd_t=101&pf_rd_p=478184871&pf_rd_i=507846
In that link, I noticed ASIN, which is B00154JDAI.
Looks like I can use this ID, to get product information (using amazon-ecs). I just need to parse the URL, to get ASIN.
Is there any other way to do it?
No, I am not going to do screen scraping, that is not a good idea anytime.
If you want to do this, the Nokogiri or hpricot libraries both allow HTML parsing and searching. However, this kind of screen-scraping is notoriously unreliable (as it may break any time Amazon decides to reorganize their HTML), so if you're planning to do this sort of thing for any length of time I'd recommend leveraging the Amazon Product Advertising API instead.
In your program: fetch the page and parse HTML. Filter out the required information. There may be some libraries in Ruby (that I am unaware of), which parse HTML.
hpricot seems to do what you want.
You should use the library Ruby/AWS (google for it, my karma is not high enough to allow external links...). It has been written exactly for that.
You might need to use the built-in Search to find the item you're looking for. After that, the API gives access to pictures, links and all usable information.