I have been thinking a lot about scaling out the WAMP router onto multiple machines. Especially because relying on one system without having a backup-system seems a bit tough in a live scenario.
To not have a complicated master node election and all that comes with it, I had the following idea and I would love to get some feedback.
Routers would need to share some information, i.e.:
Authentication
session ids
RPC
check if a client has already registered for an uri when using simple registration
forward calls for pattern based registration
meta api information
PUB/SUB
provide event messages to all clients in the network
meta api information
There must be a connection between all routers. To keep the connections and traffic low, the Idea is to have a ring infrastructure:
+----------------+ +-----------------+ +--------------+
| | | | | |
+--> First Router +----> Second Router +----> Nth Router +--+
| | | | | | | |
| +----------------+ +-----------------+ +--------------+ |
| |
+-----------------------------------------------------------------+
Every router would have it's own client that is connected to the next router. By storing the incoming message id for a CALL or PUBLISH that came from a client the router could identify a round trip.
Incoming calls coming from another router would be handled as is.
Yields would have the correct INVOCATION.Request|id and could be forwarded to the correct router.
LOOSING A ROUTER
This infrastructure forces to have some static and identical configuration for all routers. This means dynamic scaling would not work, but therefor no leader election is necessary. If a routers is killed the next router will be connected from the list of routers. A constant polling could check for the router to be back online.
I have not done any deep research on this. So I would be happy to get some input for that.
nginx can work as a reverse proxy to accomplish this.
ip_hash on the upstream list will make sure a client IP get's the same server every time. You may or may not need this depending on your situation.
Essentially, you want a "Load Balancer" to stand in front of WAMP servers, and delegate connections as needed.
Related
We are currently building a ForwardAuth middleware for Traefik.
We want to add user groups and give different groups access to different routers (/containers). How can we get info about D in FA to verify that the user has the permission to access D? For example, the container ID of D would be very useful to have in FA, but it doesn't seem like this is possible. We would really appreciate some help!
For the way I used and implemented FA before, I never needed information about D (and I thought that was the whole beauty of it).
There are rules defined in FA that would allow a user through or not. Those can come from anywhere needed (internal logic, database, other external systems - just not D in your case). Once the user is allowed through, I attach relevant headers to the request and that is what is forwarded to Traefik to serve. In your example, the user role could become one of the headers.
Now the routes defined in Traefik each accept requests, if certain hearders with certain values are present. And that is done with labels (in docker).
So the FA and D remain decoupled as they should be.
Would this approach work for your case as well?
I have the following use case that I would appreciate any input on. I have a Docker Swarm running with Traefik pointed to it for ingress and routing. Currently I have it as a single service that is defined to have 6 replicas of the same image, so works out to be two containers each on three nodes.
The containers basically host a GraphQL server and my requirement is that depending on which client the request is coming from, it always goes to the same specific container (ie replica). So as an example say I have user1 and user2 at client1 and user3 and user4 at client2, if user1 makes a request and it goes to replica1, then if user2 makes a request it MUST go to replica1 as well. So basically, I could take a numeric hash on the client id (client1) and mod 6 it and decide which replica it goes to and in that way make sure any subsequent calls from any user in that client id goes to the same replica. Additionally, that information of what client the call is coming from is coded in a JWT token that the user sends in their request.
Any idea how I would go about changing my Docker Swarm to implement this? My best guess is to change the swarm to not be 6 replicas and instead define each container as a separate service with its own port. Then I could potentially point Traefik to nginx or something similar which would then receive the request, grab the JWT, decode it to find the client id, take a hash and then internally route it to the appropriate node:port combination.
But I feel like there must be a more elegant and simpler way of doing this. Maybe Traefik could facilitate this directly somehow or Docker Swarm has some configuration that I don't know about, that could be used. Any ideas?
Edit: Just to clarify my usecase, not just looking for the same user to always go to the same container but the same type of user to always go to the same container
For this kind of routing you need to setup Traefik for Sticky Sessions
This is a Traefik middleware that adds a cookie to responses that is used in subsequent requests to route to the same service.
I need to secure a public facing HTTP API where I can not touch the code on the API server.
The HTTP API has multiple end-users that will consume it using non-interactive clients (typically backend services). Just to be clear, the client owns the resources that it will access and as such must provide a user since authorisation logic needs to be tied to a end-user.
I’m toying with the idea of using OAuth2 and the Resource Owner Password Credentials Grant
and then using the access token provided to get a JWT which the client can present to a HTTP proxy that parses the request before passing it to the HTTP API Server.
Here is the flow as I envision it:
+----------+ +---------------+
| |>--(A)---- Resource Owner ------->| |
| | Password Credentials | Authorization |
| Client | | Server |
| |<--(B)---- Access Token ---------<| |
| | (w/Refresh Token) |---------------|
| | | |
| |>—-(C)---- Request JWT ——-------->| JWT Service |
| | (w/Access Token) | |
| | | |
| |<--(D)---- JWT ------------------<| |
| | | |
+----------+ +---------------+
v
|
|
| +---------------+
| | |
| | HTTP |
--(E)---- HTTP Request w/JWT ---------->| Proxy |
| |
| (F) |
| |
+---------------+
v
|
(G)
|
v
+---------------+
| |
| HTTP |
| API |
| |
+---------------+
(A), (B), (C) Get an access token using the Password Grant flow.
(D) Use access token to get a JWT.
(E) Attach JWT to HTTP request and send it to the HTTP Proxy.
(F) Check that JWT is valid.
(G) Pass request to the HTTP API Server.
Has anyone else solved a similar use case and would care to shed some light or have a discussion?
OAuth2 has a number of advantages ...
It has a clear flow and multiple types of grants it can use to cater to different needs.
Another advantage is that there are libraries that deal with the complexities of OAuth2, such as Identity Server : https://identityserver.github.io/Documentation/
Whether it is overkill for your project or not, only you can answer that question. A lot of people who claim OAuth2 is complicated haven't really spent enough time trying to understand it.
What I advise you not to is to not rely on any kind of self baked security model as this is what causes the downfall of a system. OAuth2 libraries have been battle tested by lots of users and companies.
A lot of companies which provide apis do so via OAuth2.
So, bottom line, if you do want to use it, do your research ,understand it and then implement it.
As for your actual question, yes I have built similar systems with lots of users, using various grants and everything worked quite well. There's nothing to be scared about as long as you spend enough time knowing what you get yourself into ...
I'm getting downvoted for this answer, so I better explain myself.
I'm not the one to suggest "just write your own security libs", but I do make an exception with oauth+api clients (and especially oauth2).
Why not Oauth2?
extra hops & extra system components as compared to a traditional authentication scheme,
chances are that whatever you do, someone using some other programming language might not have a client library compatible with whatever you're using, people make money making oauth interoperable and simple
think about this: nobody makes money making ie basic authentication "simple, compatible with 1000s of providers and just working" (to quote oauth.io), that's because basic authentication just works on every "provider" and oauth2 is a bit of a shitty, complex, non-interoprable framework - that's why we call basic authentication a part of a "protocol", we call oauth(1) a protocol, but we call oauth2 a framework
think of the implications of maintaining a non-interactive client:
you have a single bearer token across the entire cluster,
so you will need a distributed key-value store or a DB table to hold it
You will need to capture specific errors that mean the bearer has
expired,
in which case you will want to seamlessly request the
bearer again and retry the request (without losing requests).
Problem here is that this on a busy site can start to happen in
parallel on 100s of threads
So, you need a distributed locking mechanism to do it right - redis mutex is my poison of choice when
someone greets me with an oauth api
That + good luck testing that piece of complex
distributed race condition logic, and when you break your back to do
it (hi webmock) and then you still get random non-deterministic failures from time to time because the gods of concurrency met some combination of conditions that VCR/webmock didn't handle nicely
this, or just SHA512 a secret together with a nonce and HTTP body
According to the lead author of the Oauth2 project: (my emphasis)
All the hard fought compromises on the mailing list, in meetings, in
special design committees, and in back channels resulted in a
specification that fails to deliver its two main goals — security and
interoperability. In fact, one of the compromises was to rename it
from a protocol to a framework, and another to add a disclaimer that
warns that the specification is unlike to produce interoperable
implementations.
When compared with OAuth 1.0, the 2.0 specification is more complex,
less interoperable, less useful, more incomplete, and most
importantly, less secure.
To be clear, OAuth 2.0 at the hand of a developer with deep
understanding of web security will likely result is a secure
implementation. However, at the hands of most developers — as has been
the experience from the past two years — 2.0 is likely to produce
insecure implementations.
...
In the real world, Facebook is still running on draft 12 from a year
and a half ago, with absolutely no reason to update their
implementation. After all, an updated 2.0 client written to work with
Facebook’s implementation is unlikely to be useful with any other
provider and vice-versa. OAuth 2.0 offers little to none code
re-usability.
What 2.0 offers is a blueprint for an authorization protocol. As
defined, it is largely useless and must be profiles into a working
solution — and that is the enterprise way. The WS-* way. 2.0 provides
a whole new frontier to sell consulting services and integration
solutions.
What to do instead?
Oauth is typically overkill unless you're creating a bigger eco-system.
Simpler solution is DIY, define a custom authorization header as:
authentication_id api_client_id nonce digest
ie
FooApp-SHA512-fixed 4iuz43i43uz chc42n8chn823 fshi4z73h438f4h34h348h3f4834h7384
where:
authentication_id is a a fixed string that describes what kind of authentication is being used,
api_client_id is a public piece of information identifying the API client (I assume the API has more than 1 client, or that it will have more than 1 client at some point) - API client ID is there to allow you to match the API client with the API clients' secret
nonce is just a random string
secret is a random string known only to you and the client and the client should treat it as a password (ie not commit it to versioning)
digest is a SHA512 hex/base64 digest of the api_client_id + nonce + secret (you can add also concatenate the HTTP body, I'd add the HTTP body unless the body is big - such as with file uploads)
If the client passes the authentication simply forward the request to the backend API service and return it's response to the client, otherwise render an error.
I have a gen_server speaking to several hardware sensors. This setup works fine. Now I want to extend this system with a real time visualization of the data. And I want to deliver this data to a web client over websockets.
As I might have several "listeners", e.g. several people visualizing this data on different web browsers, I am thinking of something resembling the Observer Pattern. Each time a web client asks to subscribe to a sensor, it should be added to a list of stakeholders for that sensor. As soon as new sensor data arrives, it should be pushed out to the client without delay.
I am using yaws to quickly get websocket functionality. My problem is related to the way that yaws seems to work. On the server side I only "see" the client at connection time through the A#arg.clisock value (e.g. #Port<0.2825>). In the code below I register ws_server to receive the callbacks when new data enters from the client. After this point yaws seems to only allow me to respond to messages entering server side.
out(A) ->
CallbackMod = ws_server,
Opts = [{origin, "http://" ++ (A#arg.headers)#headers.host}],
{websocket, CallbackMod, Opts}.
This is what the callback module looks like:
% handle_message(Incoming)
% Incoming :: {text,Msg} | {binary,Msg} | {close, Status, Reason}
handle_message({Type,Data}) ->
gen_server:cast(?SERVER,{websocket,Data}),
noreply.
Nowhere, it seems, am I able to react to a message such as subscribe <sensor> and (after connection time) dynamically add this stakeholder to a list of observers.
How can I use the yaws server to accomplish asynchronous pushing of data to client and during session add and remove sensors that I want to listen to. Basically the easiest way would be if yaws could call back handle_message/2 with first argument being From. Otherwise I need to add a ref to keep a ref on both sides and send that to server each time, it seems backwards that I need to keep that information.
When the client starts a websocket connection Yaws will start a new gen_server process to handle all communication between client and server.
The gen_server dispatches all client-sent messages to your callback function handle_message/1, so it is not used only for the initial connect.
Also, the gen_server process is used to push data from the server to the client - use the yaws_api:websocket_send(Pid, {Type, Data}) function. The Pid parameter is the pid of the websocket gen_server. Your callback module could be changed to:
handle_message({Type,Data}) ->
gen_server:cast(?SERVER,{self(), Data}),
noreply.
This gives your server the pid of the websocket gen_server to be used with yaws_api:websocket_send/2. The Data parameter contains the client request and needs to be parsed by your server to check for sensor subscription requests and associate the websocket pid with the appropriate sensor.
In a concept proof I'm developing, I've built the folowing scheme:
_ A _ _
/ | \ \
SS S H CC
/ \
C C
In which:
A - Application
SS - Supervisor
CC - "Real" client
S - gen_server
H - gen_event
C - "internal" client
The application works like a multiplexer. I connect on my internal client on my server and request some stuff. Then I pass this request to my real client (witch connects to the real application) and distribute it over any internal client who is demand that kind of stuff.
Everything is working fine except for the event handling: I wanted to make my internal client send messages directly to the gen_event handlers, and he would intermediate the process of requesting to the real server/ reading from cache and replying to the client.
I figured that I need to make a link between the event handler and my client handling process, this is right? Maybe it's the case to make the event handler to be a global process and just pump messages from between clients?
Sorry for the long question.
I am not sure I understand the question but assuming you are after a sort of message switch to enable Clients to exchange messages in a distributed fashion, you could look at one my projects that does just that: Mswitch.
Of course if you register a "global process" (accessible locally and/or remotely), you can always shuttle messages to it and have this process distribute the said messages as you see fit. Be sure not to forget about the common pitfalls of making processes in different Emulator instances talk to each other (cookie sync and registered VM name).
If you register the handler, it will have a globally known name , and then you can send messages to it directly.