I would like to clarify how data exactly flows from on-premis or internet data source to GoodData platform during local execution of the graphs. Case with local data source I think is obvious, but data source outside LAN is not so when executed locally.
Yuya,
This depends on the reader and connection type. If using a REST or SOAP connector it uses that protocol over HTTP to read the data for example. Certain technology ones are hard coded like SFDC using their REST API or ExactTarget using their SOAP API. Besides the HTTP protocol we support FTP/SFTP, WebDAV, and JDBC for connections.
http://developer.gooddata.com/docs/data-loading provides an overview of data loading with a bunch of related articles / tutorials linked to further down the page. There is also a CloudConnect User Guide and Data Loading Admin Guide that might help.
Is there a specific type of data pull you are interested in understanding further?
Related
I have implemented a custom email server and web client. The server is just a REST API (similar to google's gmail API) that uses a 3rd party (sendgrid) for sending and receiving. The emails are stored in a database. The web client just talks to the REST client for sending and receiving.
The problem with this approach is it doesn't implement IMAP anywhere, which makes it impossible for standard clients (outlook, iphone, etc.) to connect to and use our email API. This limits customers to using only our client for email.
What I need is some sort of IMAP Server "facade" that will manage the connections to clients and make calls to my REST API for actually handling the requests (get email, send email, etc.).
How can an IMAP facade be implemented? Is there maybe a way to take an existing MailServer and gut it and point all it's "events" to making calls to my API?
tl:dr; write your gateway in Perl; use Net::IMAP::Server; override Net::IMAP::Server::Mailbox; and use one of the many Perl REST clients to talk to your server.
Your best bet for doing this quickly, while maintaining a reasonable amount of code security, is with Perl. You'll need two Perl modules. The first is Net::IMAP::Server, and here is the Github repository for that module. This is a standards-compliant RFC 3501 server that was purposely designed to have a configurable mail store. You will override the default Net::IMAP::Server::Mailbox implementation with your own code that talks to your custom email backend.
For your second module, choose your favorite Perl module(s) to use to speak to your REST server. Your choice depends on how much fine grained control you want to have over the construction and delivery of the REST messages.
Fortunately, here you have tons of choices. One possibility is Eixo::REST, which has a Github repository here. Eixo::REST seems to deal well with asynchronous vs. synchronous REST API calls, but it doesn't provide a lot of control over X509 key management. Depending on how googley your API is, there's also the REST::Google module. Interestingly, this family also has a REST::Google::Apps::EmailSettings module, specifically for setting Gmail-specific funkiness like labels and languages. Lastly, the REST::Consumer module seems to encapsulate a lot of https-specific things like timeout and authentication as parameters to Perl object instantiation.
If you use these existing frameworks, then about 90% of the necessary code should already be done for you.
Don't do this by hacking Dovecot or any other mail server written in C or C++. If you hack together a mail server quickly using a compiled language, your server will sooner or later experience all the joy of buffer overflows and stack smashing and everything else that the Internet does to fuck over mail servers. Get it working safely first, then optimize later.
(This is basically my comment again, but elaborated quite a bit more.)
Some IMAP servers, most notably Dovecot, are structured such that the file access is in a separate module with a defined interface. Dovecot isn't the only one, but it's by far the most popular and its backend interface is known to be appropriate, so I'd take that absent specific concerns.
There already exist non-file modules such as imapc, which proves that it can be done. When a client opens a mailbox backed by imapc, Dovecot parses IMAP commands, calls message access functions in imapc, imapc issues new IMAP commands, parses the server responses, returns C structs to Dovecot, Dovecot fashions new IMAP responses and returns them to the client.
I suggest that you take the dovecot source, look at src/lib-storage/inbox/index/imapc and the other backends in that directory, and implement one that speaks your REST API as a client.
Since you're familiar with .NET, I would suggest hacking either of the following implementations of IMAPv4 servers to your liking:
Lumisoft Mail Server - a very old project indeed (let's call it "mature", huh?). Don't be too turned off by the decade-old website and the lack of a github link - the source is provided under "other downloads".
McNNTP - also an older project and with a major focus on NNTP (as the name says) but very close to what you're trying to achieve in terms of the IMAP component. Take a look, you'll probably find this a good starting point.
I am having trouble understanding how to hook up a backend that will support relayjs. Does GraphQL/Relay automatically help store queries? I.e. if I implement a REST backend that returns a json blob with everything, will Relay take care of the data query optimizations? I'm not entirely sure of the abstraction between Relay and backend/database.
Take a look at Getting Started in the Relay documentation. Relay requires that you have a GraphQL server that understands GraphQL queries and can respond with valid response payloads.
GraphQL provides an abstraction with which Relay uses to interface with your backend. GraphQL is not a database, but rather a layer that sits on top of your server application code. For example, you could implement a GraphQL server that sits in front of your REST server as demonstrated by the graphql/swapi-graphql repository.
Nick Schrock's presentation of GraphQL at #Scale 2015 also provides a good overview.
what Tin Can API can do other than storing the state of the agent and how can we retrieve the publicly stored statements from Tin Can API
Thanks in advance
You can do a lot with the Tin Can API (Experience API). The point of the xAPI is to store user experiences, anything from I completed a course to I started watching a video. I've seen or worked on things as simple as using the xAPI to send SCORM tracking to an LRS, to support mobile, tracking sensor data from field exercises, to storing information collected in games and simulations. And the Experience API gives you the ability, like you said, to get data back out in a standard way, to support reporting and evaluation of data.
There are groups working with the Experience API to do interesting things. https://groups.google.com/a/adlnet.gov/forum/#!forum/xapi-design
There is also a spec working group forum where you can get more resources and answers: https://groups.google.com/a/adlnet.gov/forum/#!forum/xapi-spec
There are also resources and articles talking about what you can do with the Experience API. http://www.adlnet.gov/tla/experience-api/
and http://en.wikipedia.org/wiki/Tin_Can_API
There are some open source projects on ADL's GitHub page that also show how you can use the Experience API. https://github.com/adlnet
For sending and retrieving info from an LRS in web browsers there's a JavaScript library: https://github.com/adlnet/xAPIWrapper .. it's been built and minified..you can just include the xapiwrapper.min.js in your page and use the readme examples to get started.
For reporting and querying data you can look at the new project: https://github.com/adlnet/xAPI-Dashboard
There's a starting Java library to make talking to an LRS easier in Java, which could be used for regular Java apps or for Android apps: https://github.com/adlnet/jxapi
They're also starting a JQuery Mobile Plugin: https://github.com/adlnet/xapi-jqm
And even an example of using the Experience API with MedBiquitous and Common Core competencies to identify learner's progress toward becoming competent in some aspect: https://github.com/adlnet/xci
As for your question about getting statements from an LRS, you would just do a GET request to the statements endpoint. The spec currently says that requests must include the Experience API version header: https://github.com/adlnet/xAPI-Spec/blob/master/xAPI.md#62-api-versioning . And you will probably need to authenticate as a client using the LRS. This is generally done by registering on the LRS and getting some sort of credentials. This will vary based on the LRS you use, but they all have instructions on how to use and send the credentials. https://github.com/adlnet/xAPI-Spec/blob/master/xAPI.md#64-security
ADL's hosted example LRS opened up the GET statements endpoint so that people new to the Experience API could hit it and see statements without needing to figure out the request rules: https://lrs.adlnet.gov/xapi/statements
What is the best way to create forms (textfields, checkboxes, radio buttons, ...) and handle the data after the user's input?
Just use web components as dart's web ui?
http://www.dartlang.org/articles/web-ui/
http://www.dartlang.org/docs/tutorials/web-ui/
edit: Lets imagine the following example application: I would like to create an online quiz/test.
First the user has to register
Data will be stored in a textfile or in a database
User can log in and play an online quiz or do an online test.
For that quiz/test i need to evaluate the input with the predefined correct answers
Here's a high-level answer to your question.
To handle data on the server side you can use the HttpServer class to start a web server. See this article.
To store data in a flat file you'll need to use the dart:io package to open a file and write to it. See the documentation for File.openWrite().
To store data in a database there are packages available on pub for mysql and postgresql.
There are two different ways to implement the client side. The traditional way is to use templating to generate html with the data in input elements within a form tag, and then handle the form submission in your webserver.
The modern way, that is the focus within the Dart community, is to write a single page app, which uses HttpRequest to read data from and send data to the server (usually using json).
On the client side, you could retrieve data from server (e.g. as JSON) and use that to build a form. This seems like a good fit for a web component as elements can be dynamically added based on received data.
The component would be bound to the model so you can serialize the model object to JSON on submit and send it to server on submit or just send it as standard HTML form.
The server side of the story is less clear, there are no production quality web server libraries that I am aware of, but you could take a look at DartExpress as an example, or Stream, and there are others, more or less complete. Anyway, you would have to extract the POST payload from HttpRequest (if sent as JSON) or use the form data which is also accessible via queryParameters property - please note that this is Dart:io.HttpRequest class, not Dart:html.HttpRequest, and it is available only on server side.
The mentioned server frameworks simplify this part a bit.
Using Web-UI would be a good choice. The todomvc application illustrates nicely how to dynamically capture the input from a user. Processing on the server side is wide open as far as choices go. Dart does have server side capabilities, and you could use some of the existing libraries to accomplish what you want.
Another way that you can process the information server side is to comunicate with the DB directly using a REST based web service like CouchDB. Cloudant offers such a service and allows you to communicate directly to the DB from the client, providing you can overcome the Same-Origin-Policy. There are 2 ways to do this. Enable CORS on the CouchDB instance, or host your application on the server that has the DB, which is also possible with CouchDB.
Dart serverside also supports websockets, so you can easily deliver the user provided data to the server with web sockets, and then do whatever processing you like on the serverside.
One other option I can think of would be to have the information processed and saved in the local browser. You can access the local DB or local browser file system from the Dart client, and keep everything local. For statistics, you can have the client update a web service of your choice.
I'm working on an app that is reading it's data from a customized RSS feed, that contains information about events. The feed is parsed with the GData XML framework, and written to SQLite with the FMDB wrapper.
The feed information is created on the server by a web-app.
I'm now thinking about how I can make the app bullet proof against hacking attacks that might get through the servers validation. I have seen some threads here about SQL Injection, so I have information regarding that.
Are there other attacks/mechanisms to consider?
The information in my feed is not secret, I'm only after a well working app that can not be hacked easily.
[late answer]
You should harden your server. There is not much you can do to secure the feed itself against man in the middle unfortunately, as SSL and/or XML signature is not really defined for RSS.