I am trying to figure out the managed service which can write data to kinesis. with this way i need my message should get at least one delivery to kinesis stream.
is it recommended or good idea to use SQS to write to Kinesis. I am looking the solution which can scale horizontally.
There are multiple options to write to Kinesis, based on your experience and environment.
The most straightforward method is to call the put-record API directly. You can write one record at the time with put-record or batch them together with put-records. These API calls are supported by the various SDKs (Java, .NET, php, ruby, javascript, python…).
Since one of the most useful use cases of Kinesis is to gather information from millions of users on their mobile devices or browsers, there are dedicated mobile SDKs for iOS, Android and JavaScript in the browser. See here: http://aws.amazon.com/mobile/sdk/. You can use these SDKs to remove the need for an ingestion infrastructure beyond Kinesis.
Some other options are to use tools like FluentD that is very popular in shipping logs. See here a connector that can make your life easier: https://github.com/awslabs/aws-fluent-plugin-kinesis
Another recent option is to use Kinesis Producer Library (KPL) that is adding the ability to aggregate many events together into a single event to optimise the shard capacity to the limit. It is also allowing asynchronous writing to Kinesis by the producers, and the KPL will handle the blocking, retires and other latency causing methods.
The log4j appenders are also a popular way to write log events directly to kinesis with minimal effort from the side of the developers. See here for more details: https://github.com/awslabs/kinesis-log4j-appender
If you are getting your resources from HTTP calls, try Amazon API Gateway: http://aws.amazon.com/api-gateway/
Here is a nice post about capabilities: https://aws.amazon.com/blogs/aws/amazon-api-gateway-build-and-run-scalable-application-backends/
you can send post action in http to put record to kinesis:
POST / HTTP/1.1
Host: firehose.<region>.<domain>
Content-Length: <PayloadSizeBytes>
User-Agent: <UserAgentString>
Content-Type: application/x-amz-json-1.1
Authorization: <AuthParams>
Connection: Keep-Alive
X-Amz-Date: <Date>
X-Amz-Target: Firehose_20150804.PutRecord
{
"DeliveryStreamName": "some_delivery_stream",
"Record": {
"Data": "..."
}
}
https://docs.aws.amazon.com/firehose/latest/APIReference/API_PutRecord.html
Related
I have a large JAVA application which connects to hundreds of cloud based systems using their REST API's and fetch the data from those systems.
To connects those different cloud systems we have different modules and each one have different approach to call REST API's like some modules using apache rest client some module using googles rest client.
So there is no centralise place where the REST api is getting called.
I have to track performance of the application e.g. to fetch accounts info from test system takes 1 hour. and this process need
4 api calls for https://test/api/v2/accounts -- (this will return all account id's)
8000 api calls for https://test/api/v2/accounts/{accountId}. --- (this will return deaths of each account)
I need to track what is the time taken by each api to responds and based on that calculate time taken by application to process that data.
Important part here is deatiled api analysis and make graphical data if possible e.g.
4 api calls for https://test/api/v2/accounts --- taken 3 minutes
8000 api calls for https://test/api/v2/accounts/{accountId} -- taken
48 minutes
I need any any pointer how can I achieve that something like intercept all rest api made to https://test/api/v2
As you've probably already discovered, without some extra tweaking, wireshark just shows you the connections at the FQDN level: you can't see which individual endpoint is called (because TLS, by design, hides the content of the connection). You have a few options though:
if you control the APIs that are being connected to, you can load the
TLS keys into wireshark, and it'll let you decrypt the TLS
connection;
if you can force your app to use a proxy, you can use a Man-In-The-Middle (MITM) proxy (like Burp) to intercept the traffic; or
you can instrument your app to log destination and duration for all the API requests.
I was playing around with GZIP compression recently and the way I understand the following:
Client requests some files or data from a Web Server. Client also sends a header that says "Accept-Encoding,gzip"
Web Server retrieves the files or data, compresses them, and sends them back GZIP compressed to the client. The Web Server also sends a header saying "Content-Encoded,gzip" to note to the Client that the data is compressed.
The Client then de-compresses the data/files and loads them for the user.
I understand that this is common practice, and it makes a ton of sense when you need to load a page that requires a ton of HTML, CSS, and JavaScript, which can be relatively large, and add to your browser's loading time.
However, I was trying to look further into this and why is it not common to GZIP compress a request body when doing a POST call? Is it because usually request bodies are small so the time it takes to decompress the file on the web server is longer than it takes to simply send the request? Is there some sort of document or reference I can have about this?
Thanks!
It's uncommon because in a client - server relationship, the server sends all the data to the client, and as you mentioned, the data coming from the client tends to be small and so compression rarely brings any performance gains.
In a REST API, I would say that big request payloads were common, but apparently Spring Framework, known for their REST tools, disagree - they explicitly say in their docs here that you can set the servlet container to do response compression, with no mention of request compression. As Spring Framework's mode of operation is to provide functionality that they think lots of people will use, they obviously didn't feel it worthwhile to provide a ServletFilter implementation that we users could employ to read compressed request bodies.
It would be interesting to trawl the user mailing lists of tomcat, struts, jackson, gson etc for similar discussions.
If you want to write your own decompression filter, try reading this: How to decode Gzip compressed request body in Spring MVC
Alternatively, put your servlet container behind a web server that offers more functionality. People obviously do need request compression enough that web servers such as Apache offer it - this SO answer summarises it well already: HTTP request compression - you'll find the reference to the HTTP spec there too.
Very old question but I decided to resurrect it because it was my first google result and I feel the currently only answer is incomplete.
HTTP request compression is uncommon because the client can't be sure the server supports it.
When the server sends a response, it can use the Accept-Encoding header from the client's request to see if the client would understand a gzipped response.
When the client sends a request, it can be the first HTTP communication so there is nothing to tell the client that the server would understand a gzipped request. The client can still do so, but it's a gamble.
Although very few modern http servers would not know gzip, the configuration to apply it to request bodies is still very uncommon. At least on nginx, it looks like custom Lua scripting is required to get it working.
Don't do it, for no other reason than security. Firewalls have a hard or impossible time dealing with compressed input data.
Does anyone know of an example which has really simple onConnect onMessage onClose structure for use with grails-spring-websocket?
The first problem when trying to implement web sockets in grails 3 is which plugin library to use. It needs to be one which will be supported in grails so it gets new versions, and one which has lots of users, examples and/or documentation.
This article has exactly what i need - the ability for users to connect, to store the list of connected users in a collection, for clients send messages to the server, and have the server send async "replies" back to one or small subset of the connected users. But it uses an obscure socket implementation (javax.websocket:javax.websocket-api:1.1) with grails.
This one: https://plugins.grails.org/plugin/zyro/grails-spring-websocket seems to be more popular, but all the examples I can find only cover publish/subscribe via topics, and don't have any connection to the individual client. While I could create a topic per client, this is far more complicated than it should be. Also, configuration looks to be arcane and overly complex, there should be none.
Someone suggested using the wschat plugin as a tictactoe game was done with it, but really I just need a simple socket implemenation with onConnect onMessage and onClose callbacks - no publish and subscribe, no topics etc.
I implemented this in 5 minutes in node.js using the "ws" plugin, which is really simple, e.g.:
wss.on("connection", myFunction(ws) {..}
ws.on('message', function(message) {
messageHandler(message, ws)
})
ws.on('error', function(er) {
console.log(er)
})
ws.on('close', function() {
console.log('Connection closed')
})
Is anything like this available in grails with an official web socket plugin, or should I stick to node for websockets? no security or other features are required (security is handled by passing user/pass down the secure socket in a "login" message - more or less the same as a web login)
Update
No luck yet finding a way to handle simple websocket events and to send web socket messages back to users. This plugin looked promising, but it only handles topics/queues. The article mentioned earlier which uses javax.websocket-api is ugly but more or less what we need, but unfortunately as it is stand alone code, you cant access the database or services from the handlers, and services can't send messages.
I am trying to monitor calls to an API, and I am trying to do so with wireshark. However, I only see low level packets, I want to see the actual http and https requests and responses. Is this possible with wireshark?
You might consider using something besides wireshark for this.
For instance the Burp proxy will allow you to inspect requests and responses between you and the application, as well as pause a request, edit it, then send it on it's way. It really is a great tool for working with web APIs.
I am working in a Rails application and below is the scenario requiring a solution.
I'm doing some time consuming processes in the background using Sidekiq and saves the related information in the database. Now when each of the process gets completed, we would like to show notifications in a separate area saying that the process has been completed.
So, the notifications area really need to pull things from the back-end (This notification area will be available in every page) and show it dynamically. So, I thought Ajax must be an option. But, I don't know how to trigger it for a particular area only. Or is there any other option by which Client can fetch dynamic content from the server efficiently without creating much traffic.
I know it would be a broad topic to say about. But any relevant info would be greatly appreciated. Thanks :)
You're looking at a perpetual connection (either using SSE's or Websockets), something Rails has started to look at with ActionController::Live
Live
You're looking for "live" connectivity:
"Live" functionality works by keeping a connection open
between your app and the server. Rails is an HTTP request-based
framework, meaning it only sends responses to requests. The way to
send live data is to keep the response open (using a perpetual connection), which allows you to send updated data to your page on its
own timescale
The way to do this is to use a front-end method to keep the connection "live", and a back-end stack to serve the updates. The front-end will need either SSE's or a websocket, which you'll connect with use of JS
The SEE's and websockets basically give you access to the server out of the scope of "normal" requests (they use text/event-stream content / mime type)
Recommendation
We use a service called pusher
This basically creates a third-party websocket service, to which you can push updates. Once the service receives the updates, it will send it to any channels which are connected to it. You can split the channels it broadcasts to using the pub/sub pattern
I'd recommend using this service directly (they have a Rails gem) (I'm not affiliated with them), as well as providing a super simple API
Other than that, you should look at the ActionController::Live functionality of Rails
The answer suggested in the comment by #h0lyalg0rithm is an option to go.
However, primitive options are.
Use setinterval in javascript to perform a task every x seconds. Say polling.
Use jQuery or native ajax to poll for information to a controller/action via route and have the controller push data as JSON.
Use document.getElementById or jQuery to update data on the page.