I have been successfully evaluating Spring Cloud DataFlow with a typically simple flow: source | processor | sink.
For deployment there will be multiple sources feeding into this pipeline which I can do using data flow labels. All well and good.
Each source is a different rabbitmq instance and because the processor needs to know where the message came from (because it has to call back to the source system to get further information), the strategy I'd thought of was to enrich each message with header details about the source system which is then transparently passed along to the processor.
Now, I'm well-versed in Spring, Spring Boot and Spring Integration but I cannot find out how to enrich each message in a dataflow source component.
The source component is bound to an org.springframework.cloud.stream.app.rabbit.source.RabbitSourceConfiguration. The source uses the default Source.OUTPUT channel. How do I get hold of each message in the source to enrich it?
My processor component uses some Spring Integration DSL to do some of what it needs to do but then this processor component has both an INPUT and OUTPUT channel by definition. Not so with the RabbitSourceConfiguration source.
So, can this be done?
I think you need a custom MessageListener on the MessageListenerContainer in RabbitSourceConfiguration.
In the RabbitSourceConfiguration you can set a custom ChannelAwareMessageListener (You can possibly extend from MessagingMessageListenerAdapter as well) on the MessageListenerContainer that does what you incline to do.
In the end what worked was subclassing org.springframework.cloud.stream.app.rabbit.source.RabbitSourceConfiguration to:
override public SimpleMessageListenerContainer container() so that I could insert a custom health check before calling super.container(). My business logic enriches each message (see next bullet) with details of where the message came from (note, this is the publisher of the messages and not the rabbit queue). There's a health check needed to validate the additional enriching information (which is provided via configuration) to ensure that messages aren't consumed from the queue and enriched with the wrong information. If the validation fails, the source component fails to start and hence no messages are consumed.
override the creation of the AmqpInboundChannelAdapter bean so that a custom subclass of DefaultAmqpHeaderMapper can be set on the adapter. This custom mapper adds the enriched headers in public Map toHeadersFromRequest(final MessageProperties source).
For me, the inability of stream/dataflow to intercept and modify messages in Source components is problematic. I really shouldn't have to fiddle around with the underlying message broker API in the ways I did. I should be able to do it with e.g. Spring Integration. Indeed I can register a global message interceptor but I cannot change the headers of the message.
This ability would go on my WIBNI (wouldn't it be nice if) list. Perhaps I'll raise a request for this.
Related
I am upgrading our old application for Serilog... One of the existing functionality is ... When log level = ERROR, it will log into local file and send 'WCF' request to the remote server, remote server will update database...
Basically it will log into multiple source(local file, remote database by sending wcf request) if it level is 'ERROR'.
I understand using 'rollingfile' appender to logging into local file.
However, i do not know how to configure 'WCF Service' for Serilog... is there any 'WCF SINK' can help me achieve this?
As of this writing there's no generic sink that makes "WCF" calls... You'd have to build your own sink, implementing the calls you need.
You can see a list of documented sinks in the "Provided Sinks" page on Serilog's wiki, and you can also see available sinks in NuGet.org.
I building multitenancy into the unit of work for a set of services. I want to keep the tenancy question out of the way of day-to-day business domain work, and I do not want to touch every existing consumer in the system (I am retrofitting the multitenancy onto a system without any prior concept of a tenant).
Most messages in the system will be contexted by a tenant. However, there will be some infrastructure messages which will not be, particularly for the purpose of automating tenant creation. I need a way of determining whether to use a tenant-contexted unit of work, or a infrastructure unit of work uncontexted by a tenant because the way I interact with the database is different depending on whether I have tenant context. The unit of work is built in the process of spinning up the dependencies of the consumer.
As such I need a way of peeking at the message or its metadata before consuming it, and specifically, I need to be able to peek at it during the dependency building. I was intended to have a tag interface to mark tenant management messages out from normal business domain messages, but any form of identifying the difference could work. If I am in a unit of work resulting from an HTTP request, I can look at WebApi's HttpContext.Current and see the headers of the current request, etc. How do I do something analogous to this if I am in a unit of work resulting from messaging?
I see there is a way to intercept messages with BeforeConsumingMessage() but I need a way of correlating it to the current unit of work I am spinning up and I'm not seeing how that would work for me. Pseudocode for what I am trying to do:
if MessageContext.Message.GetType() = typeof<ITenantInfrastructureMessage>:
database = new Database(...)
else:
tenantId = MessageContext.Headers.TenantId;
database = new TenantDatabase(..., tenantId)
I am working in C#/.NET using MassTransit with RabbitMQ and Autofac with MassTransit's built-in support for both.
Your best option is to override at the IConsumerFactory<T> extension point, and extract the tenant from the message (either via a message header, or some message property) and register that in the container child lifetime scope so that subsequent resolutions from the actual consumer class (and it's dependencies) are properly matched to the tenant in the message.
In our systems, we have a TenantContext that is registered in a newly created LifetimeScope (we're using Autofac), after which we resolve the consume from the child scope, and the dependencies that use the tenant context get the proper value since it's registered as part of building the child container for the message scope.
It works extremely well, we even built up extension methods to make it easy for developers registering consumers to specify "tenant context providers" that go from a message type to the proper tenant id, which is used to build the TenantContext.
You can do similar things with activity factories in Courier routing slips (which are a specialization of a consumer).
I have two separate services communicating using AmqpProxyFactoryBean (the "client") and AmqpInvokerServiceExporter (the "server"). Now, I'd like to include some custom headers on every request made through the AMQP proxy and be able to access them on the "server". Is there any easy way I can achieve this?
Since AmqpClientInterceptor uses AmqpTemplate to send and receive AMQP messages, you can provide for that RabbitTemplate any custom MessageConverter. And populate any additional headers from your toMessage() implementation.
However I'm not sure that you will be able to access to those custom header on the server side. We end up there just with RemoteInvocation.invoke().
So, seems for me you finally come up to the solution with an additional RPC param.
From other side that custom header may be useful for other AQMP routing scenarios when you can route that RPC message not only to the RPC queue.
Consider using Spring Integration AMQP Gateways instead of remoting over rabbitmq; that way you have complete control over the headers passed back and forth.
If you don't want to use Spring Integration, you can use the RabbitTemplate sendAndReceive methods on the client and either the receiveAndSend or a listener container on the server.
Again, this gives you full control over the headers.
Using the sink MQTT source module for Spring XD, I am getting wrong values on the payload. I have the stream subscribed to a certain topic and a normal client in eclipse also subscribed to the same topic. The payload is supposed to be an array of bytes. For the same message I am getting
xxxxx...0000073F, on Spring XD
xxxxx...000007F9, on eclipse Paho client
In reality, this value is supposed to be a counter, and the eclipse paho client behaves perfectly but Spring XD seems to behave a weird behavior when any of the hexadecimal digits reaches F. For same reason it gets stuck on the same number until the counter has increased enough to get rid of any F on the sequence.
My question would be, if there eny pre processing happening on the MQTT client provided on Spring XD that explains why I am getting different values on the payload. I am sure that the second is correct since it is myself who is sending the values.
Thanks.
Spring XD uses Spring Integration which uses the Paho client under the covers.
Unfortunately, it converts the payload to String (UTF-8 by default), which produces results like this with data that is not valid UTF-8.
The adapter can be configured to pass the payload as binary but, unfortunately, that option is not currently exposed in XD.
The work around is to create a sublass of DefaultPahoMessageConverter, override mqttBytesToPayload ...
protected Object mqttBytesToPayload(MqttMessage mqttMessage) throws Exception {
return mqttMessage.getPayload();
}
Put the converter in a jar in the module's lib directory, and update the mqtt.xml to pass an instance of the converter in the converter attribute.
I will open a JIRA issue to make binary a standard option of the module.
I am developing some SOAP web services using Ruby on Rails and considering how to handle generic failures. These generic errors are applicable to all the methods within the service and include the following :-
Missing Request element
Missing Authentication element (Custom)
Invalid Authentication details
I can intercept these errors within my controller before calling the relevant method and respond appropriately. My question is which implementation is easiest to manage from a Client perspective. My options for handling these errors seem to be as follows.
Raise an exception and let the SOAP service generate a SoapFault. This would be fine except I have little (no) control over the structure of the message contained within the SOAP fault.
Return an Http 400 response with an agreed data structure to indicate the error message. This structure would not be defined within the WSDL though.
Include a Status element in all responses, whether successful or not and have that status element include a code and an array of error data (Including error messages).
Option three seems like the best solution but is also the most error prone to implement as the implementation of web services in ROR precludes me from implementing this in a generic way and each method becomes responsible for checking the result of the checks and rendering an appropriate response. Admittedly this would be a single function call and return on failure but it is relying on the developer to remember to do this as we add more options.
I appreciate that most ROR developers will say that this should be implemented as a REST service and I agree, in fact we already have REST services to do this but the spread of SOAP in the corporate world, and its impressive tooling support means that we have to provide SOAP services to remain competitive.
In your experience what would be the easiest implementation for clients to handle and does this differ dependant upon the libraries/language of the client process.
A SoapFault would be the preferred way to signify errors. SoapFaults can contain additional information in their <detail> element.
The advantage of a SoapFault over some status element is that the caller can use standard exception handling, instead of checking for some status field.