How can I use native Flume sinks with fiware-cygnus? - flume

Question

Fiware-cygnus documentation mentions that it is based on Apache Flume. However, it is not clear whether I can use native Flume sinks to persist events arriving from Orion Context Broker. Is this something I can easily do, with little (or ideally zero) coding? If not -- would be good to know why (and whether this can be supported going forward). Thanks!

Answer 1

You can use native Flume sinks by simply configuring them. Nothing has been changed in Cygnus in terms of configuration management, thus you can configure a Orion-like sink or a native one.
Nevertheless, there are differences between Orion-like and native Flume sinks.
The first one is the Orion-like sinks store the relevant data with certain structure, and the Flume native sinks will store the notified raw data. I mean, if you receive a Json-based notification such as:
{
"subscriptionId" : "51c0ac9ed714fb3b37d7d5a8",
"originator" : "localhost",
"contextResponses" : [
{
"contextElement" : {
"attributes" : [
{
"name" : "speed",
"type" : "float",
"value" : "112.9",
"metadatas": []
},
{
"name" : "oil_level",
"type" : "float",
"value" : "74.6",
"metadatas": []
}
],
"type" : "car",
"isPattern" : "false",
"id" : "car1"
},
"statusCode" : {
"code" : "200",
"reasonPhrase" : "OK"
}
]
}
OrionHDFSSink will store something like:
{"recvTimeTs":"1429535775","recvTime":"2015-04-20T12:13:22.41.124Z","fiware-servicePath":"4wheels","entityId":"car1","entityType":"car","attrName":"speed","attrType":"float","attrValue":"112.9","attrMd":[]}
But a native HDFS sink (or any other one) will persist the entire notified json.
The second main difference if the handling of the notified fiware-service and fiware-servicePath. Cygnus's sinks are able to deal with these values in order to map the notified data into specific data structures (folders, databases, tables, resources, queues...). This is very important for multi-tenancy purposes.
Third, Cygnus adds sinks for storages not covered by native Flume, such as CKAN, STH, MongoDB, MySQL or DynamoDB.
There are many other differences:
The usage of the Grouping Rules.
The Management Interface.
OAuth2 authentication, which is the official FIWARE's mechanism.
...