I'd like to preface this with the fact that I am completely new to SAP and SAP HANA, and OData.
I was tasked with pulling changes from a SAP HANA table and transfer those to Kafka.
I noticed there was a Kafka source connector already written, which can be found here.
For this task, I was given a URL, a username and a password.
The URL looks like this:
https://blablabla.companyName.com/companyName/Foo/Bar/Baz/Foo/Table/Resource.xsodata
And this is a sample of the source connector's configs:
# The first few settings are required for all connectors:
# a name, the connector class to run, and the maximum number of
# tasks to create.
name = odatav4-source-connector
connector.class = org.init.ohja.kafka.connect.odatav4.source.OData4SourceConnector
tasks.max = 1
# The remaining configs are specific to the OData v4 source connector.
# OData server host as either DNS or IP
sap.odata.host.address = services.odata.org
# OData server port
sap.odata.host.port = 443
# OData protocol (supported values are http or https)
sap.odata.host.protocol = https
# OData user name for basic authentication
# For services not requiring authentication this can be set to any value
sap.odata.user.name = anonymous
# OData user password for basic authentication
# For services not requiring authentication this can be set to any value
sap.odata.user.pwd = anonymous
# Optional list of service URL query parameters in the form of "param1=value1,param2=value2", e.g. sap-client=200
#sap.odata.query-params=
# none(default): DECIMALs will be mapped to Connect Decimal data type
# primitive: DECIMALs will be mapped to INT64(id scale = 0) anf FLOAT64
#sap.odata.decimal.mapping = none
# maximum amount of retries in case of service connection/communication errors (e.g. HTTP status codes 400-599)
#sap.odata.max.retries = 30
# The backoff strategy applied will select a random number of milliseconds
# to wait between min.retry.backoff.ms and max.retry.backoff.ms before starting
# the next retry.
#sap.odata.min.retry.backoff.ms = 20000
#sap.odata.max.retry.backoff.ms = 180000
# Timeout in milliseconds for establishing http connections
#sap.odata.connection.connect.timeout.ms=3000
# Timeout in milliseconds for reading data from a http connection
#sap.odata.connection.read.timeout.ms=10000
# Individual configurations for each OData v4 service entity.
# service and entityset build up the primary key for each OData configuration.
# OData v4 URL service path
sap.odata#00.service = /V4/Northwind/Northwind.svc/
# OData v4 entity set name
# The entity set name can be queried from the /$metadata service URL
sap.odata#00.entityset = Order_Details
# Kafka topic name the data for this OData service entity set will be pushed to
sap.odata#00.topic = Order_Details
# Execution interval in seconds for the scheduled data extractions
# Set to -1 to process subscription events only
#sap.odata#00.exec-period = 900
# If changes to entities selected by the first query should be tracked and returned as deltas in subsequent polls
# Set to 1 to enable odata delta mode
#sap.odata#00.track-changes = 0
# Paging mode (server or client) determines the type of paging
# server: use HTTP prefer-headers to request a maximum package size from the odata server
# client: use query functions skip and top (not compatible to change tracking)
#sap.odata#00.paging.mode = server
# Packaging size in count of entity set records
#sap.odata#00.paging.size = 50000
# Optional: Hierarchy level up to which recommendations for the expand.list configuration (query option $expand) will
# be shown in the Confluent Control Center
#sap.odata#00.expand.level = 1
# Optional: List of expand query options that will define the deep structure of returned entity messages
#sap.odata#00.expand.list =
# Optional: comma separated list of selected non-key fields to be extracted
#sap.odata#00.projection =
# Optional: filter query options
# Supported logical operations/options are: eq, ne, le, lt, ge, gt, bt, nb, in
#sap.odata#00.select#00.fieldname =
#sap.odata#00.select#00.option =
#sap.odata#00.select#00.low =
#sap.odata#00.select#00.high =
# If set to 1 the connector will subscribe to push-notifications issued by the corresponding OData service entity
#sap.odata#00.subscription.enable = 0
So I tried to create my own, like so:
{
"name": "sap-hana-source-connector",
"config": {
"connector.class": "org.init.ohja.kafka.connect.odatav4.source.OData4SourceConnector",
"sap.odata.user.name": "username",
"sap.odata.host.address": "blablabla.companyName.com",
"sap.odata.host.port": "443",
"sap.odata.host.protocol": "https",
"sap.odata#00.service": "/companyName/Foo/Bar/Baz/Foo/Table/Resource.xsodata",
"sap.odata#00.entityset": "Resource",
"sap.odata.user.pwd": "pwd"
}
}
The issue is that the only error I get is this:
{
"error_code": 400,
"message": "Connector configuration is invalid and contains the following 14 error(s):\nInvalid configuration sap.odata.host.address: No configured service reachable. Maybe invalid destination configuration?\nInvalid configuration sap.odata.host.protocol: No configured service reachable. Maybe invalid destination configuration?\nInvalid configuration sap.odata.host.port: No configured service reachable. Maybe invalid destination configuration?\nInvalid configuration sap.odata.user.name: No configured service reachable. Maybe invalid destination configuration?\nInvalid configuration sap.odata.user.pwd: No configured service reachable. Maybe invalid destination configuration?\nInvalid configuration sap.odata.max.retries: No configured service reachable. Maybe invalid destination configuration?\nInvalid configuration sap.odata.min.retry.backoff.ms: No configured service reachable. Maybe invalid destination configuration?\nInvalid configuration sap.odata.max.retry.backoff.ms: No configured service reachable. Maybe invalid destination configuration?\nInvalid configuration sap.odata.connection.connect.timeout.ms: No configured service reachable. Maybe invalid destination configuration?\nInvalid configuration sap.odata.connection.read.timeout.ms: No configured service reachable. Maybe invalid destination configuration?\nInvalid configuration sap.odata.query-params: No configured service reachable. Maybe invalid destination configuration?\nInvalid configuration sap.odata.trace.mode: No configured service reachable. Maybe invalid destination configuration?\nInvalid configuration sap.odata.trace.path: No configured service reachable. Maybe invalid destination configuration?\nInvalid configuration sap.odata.decimal.mapping: No configured service reachable. Maybe invalid destination configuration?\nYou can also find the above list of errors at the endpoint `/connector-plugins/{connectorType}/config/validate`"
}
As someone who is completely new to OData and SAP, I don't know how I'd debug this.
I noticed that, in the OData's Kafka Source Connector documentation the services end in .svc and not .xsodata; so maybe it's something to do with that?
Also, what am I supposed to be for the sap.odata#00.entityset config?
Is there a way to get a more detailed error message?
Thanks.
Related
I am trying to configure isync to fetch data from home.pl IMAP server. I have the simplest possible configuration file:
# ~/.mbsyncrc
IMAPStore store
Host host
Port 993
User kosciej#host.com.pl
PassCmd "pass poczta.home.pl/kosciej"
SSLType IMAPS
MaildirStore host-local
Path ~/.mail/host/
Channel host-inbox
Master :host-remote:"INBOX"
Slave :host-local:inbox
However it produces the error IMAP error: mailbox name inbox contains server's hierarchy delimiter. I connected to the server with openssl and found there is INBOX mailbox with . as delimiter.
a list "" *
* LIST (\HasNoChildren) "." "INBOX"
...
I managed to fix the problem with Pattern
Master :host-remote:
Slave :host-local:
Patterns inbox
But cannot understand what the error was about, it looks like "INBOX" has . somewhere?
Could anyone explain why the first version doesn't work?
I have a remote AuraDB, but I'm unable to visualize with neovis.js
var config = {
container_id: "viz",
server_url: "bolt://<server_url>:7687",
server_user: <user>,
server_password: <pwd>,
initial_cypher: "match n=(:Person)-[:Knows]->(:Person) return n"
}
The above code segment is from the neovis GitHub documentation.
(https://github.com/neo4j-contrib/neovis.js/)
If I use the "bolt" protocol, the error is :
Neo4jError: WebSocket connection failure. Due to security constraints in your web browser, the reason for the failure is not available to this Neo4j Driver. Please use your browsers development console to determine the root cause of the failure. Common reasons include the database being unavailable, using the wrong connection URL or temporary network problems. If you have enabled encryption, ensure your browser is configured to trust the certificate Neo4j is configured to use. WebSocket readyState is: 3
If I use the "neo4j" protocol, then :
Neo4jError: Could not perform discovery. No routing servers available.
If I use the "neo4j+s" / "neo4j+ssc" protocols, then :
Encryption/trust can only be configured either through URL or config, not both
I have observed neovis plugin for my webapp works well with a local neo4j db and the bolt protocol.
Please help me out with some understanding in the case of visualizing a remote neo4j aura db.
Aura has strict encryption policies. And the driver config used for neovis.js seems half-baked. It doesn't seem to be enough for Aura and doesn't seem to allow turning encryption off completely either.
So the best option here is to enforce encryption in the config and use an unencrypted connection Scheme.
Working config for Aura by using an unencrypted connection URI and enforcing encryption in the config
var config = {
encrypted:"ENCRYPTION_ON",
trust: "TRUST_SYSTEM_CA_SIGNED_CERTIFICATES",
container_id: "viz",
server_url: "neo4j://<dbid>.databases.neo4j.io",
...
Updated Solution for Neovis 2.0.2, from this GitHub issue
var config = {
containerId: "viz",
neo4j: {
serverUrl: "neo4j://<dbid>.databases.neo4j.io",
serverUser: "neo4j",
serverPassword: "secret",
driverConfig: {
encrypted: "ENCRYPTION_ON",
trust: "TRUST_SYSTEM_CA_SIGNED_CERTIFICATES"
}
},
I have been scouring the internet with no luck. I have a basic LUA script for HAProxy, which looks like this:
core.Info("LUA script for parsing request ID element - loaded");
function parseId(txn, salt)
local payload = txn.sf:req_body()
-- parses hex value from element named "ID". Example payload: {"Platform":"xyz.hu","RecipientId":"xyz.hu","Channel":"xyz","CallbackURL":"http://x.x.x.x:123","ID":"5f99453d000000000a0c5164233e0002"}
local value = string.sub(string.match(payload, "\"ID\":\"[0-9a-f]+\""), 8, -2)
core.Info("ID : " .. value)
return value
end
-- register HAProxy "fetch"
core.register_fetches("parseId", parseId)
What it does is what it says: takes a 32 characater long ID from an incoming request. In the HAProxy config file, the result is used for sticky-session handling:
stick-table type string len 32 size 30k expire 30m
stick on "lua.parseId" table gw_back
This produces two lines of log for each request:
ID: xyz which is logged from the LUA script
The detailed request data which is logged from the HAProxy config file using "log-format", e.g.:
Jan 20 22:13:52 localhost haproxy[12991]: Client IP:port = [x.x.x.x:123], Start Time = [20/Jan/2022:22:13:52.069], Frontend Name = [gw_front], Backend Name = [gw_back], Backend Server = [gw1], Time to receive full request = [0 ms], Response time = [449 ms], Status Code = [200], Bytes Read = [308], Request = ["POST /Gateway/init HTTP/1.1"], ID = [""], Request Body = [{"Platform":"xyz.hu","RecipientId":"xyz.hu","Channel":"xyz","CallbackURL":"http://x.x.x.x:123","ID":"61e9d03e000000000a0c5164233e0002"}]
I wanted to extend logging due to some strange issues happening sometimes, so I wanted to one (or both) of below approaches:
Pass the "ID" value back from the LUA script into the HAProxy config as a variable, and log it along with the request details. I can log the full request body, but don't want to due to GDPR and whatnot.
Get some request details in the LUA script itself, and log it along with the ID.
So, basically, to be able to connect the ID with the request details. If multiple requests are coming to same URL very quickly, it is difficult to find which of them belongs to a specific ID. However I couldn't accomplish these.
For the first one, I added this line into the LUA before returning the "value" variable:
txn:set_var("req_id", value)
I was hoping this would create a variable in HAProxy called "req_id", and I can log it with "log-format", but all I got was empty string:
ID = [""]
For the second one, I'm at a complete loss. I'm not able to find ANY documentation on these. I have been scouring the internet with no luck. E.g. the txn.sf:req_body() function which I know is working, I simply cannot find it documented anywhere, so I'm not sure what other functions are available to get some request details.
Any ideas for either or both of my approaches? I'm attaching my full HAProxy config here at the end, just in case:
global
log 127.0.0.1 len 10000 local2 debug
chroot /var/lib/haproxy
user haproxy
group haproxy
daemon
lua-load /opt/LUA/parseId.lua
stats socket /etc/haproxy/haproxysock level admin
defaults
log global
option httplog
option dontlognull
mode http
timeout connect 5000
timeout client 50000
timeout server 50000
# Request body is temporarily logged in test environment
log-format "Client IP:port = [%ci:%cp], Start Time = [%tr], Frontend Name = [%ft], Backend Name = [%b], Backend Server = [%s], Time to receive full request = [%TR ms], Response time = [%Tr ms], Status Code = [%ST], Bytes Read = [%B], Request = [%{+Q}r], ID = [%{+Q}[var(txn.req_id)]], Request Body = [%[capture.req.hdr(0)]]"
frontend gw_front
bind *:8776
option http-buffer-request
declare capture request len 40000
http-request capture req.body id 0
http-request track-sc0 src table gw_back
use_backend gw_back
backend gw_back
balance roundrobin
stick-table type string len 32 size 30k expire 30m
stick on "lua.parseId" table gw_back
# Use HTTP check mode with /ping interface instead of TCP-only check
option httpchk POST /Gateway/ping
server gw1 x.x.x.x:8080 check inter 10s
server gw2 y.y.y.y:8080 check inter 10s
listen stats
bind *:8774 ssl crt /etc/haproxy/haproxy.cer
mode http
maxconn 5
stats enable
stats refresh 10s
stats realm Haproxy\ Statistics
stats uri /stats
stats auth user:password
Maybe (hopefully) I'm missing something very simple, but I can't seem to figure this out.
I have a set of gRPC services that I would like to put behind a nghttpx proxy. For this I need to be able to configure my client with a channel on a non-root url. Eg.
channel = grpc.insecure_channel('localhost:50051/myapp')
stub = MyAppStub(channel)
This wasn't working immediately through the proxy (it just hangs), so I tested with a server on the sub context.
server = grpc.server(executor)
service_pb2.add_MyAppServicer_to_server(
MyAppService(), server)
server.add_insecure_port('{}:{}/myapp'.format(hostname, port))
server.start()
I get the following
E1103 21:00:13.880474000 140735277326336 server_chttp2.c:159]
{"created":"#1478203213.880457000","description":"OS Error",
"errno":8,"file":"src/core/lib/iomgr/resolve_address_posix.c",
"file_line":115,"os_error":"nodename nor servname provided, or not known",
"syscall":"getaddrinfo","target_address":"[::]:50051/myapp"}
So the question is - is it possible to create gRPC channels on non-root urls?
As confirmed here, this is not possible. I will route traffic via subdomains in nghttpx.
Apache fluent API allows simple one-line http calls like:
String content = Request.Get(url).execute().returnContent().asString();
Which is executed by a ...fluent.Executor, whose javadoc says:
A PoolingHttpClientConnectionManager with maximum 100 connections per route and a total maximum of 200 connections is used internally
I would like to change the maximum connection parameters to be used for a specific call, but I can't find a way to access the connection manager used by the above code. I have tried:
Executor.newInstance().execute(Request.Get("")).returnContent().asString();
but there is no way to change these parameters on the Executor returned by Executor.newInstance().
Is there is way to use the fluent API but with custom maximum connection values?
One can bind an instance of the fluent executor to an arbitrary HttpClient instance
PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager();
CloseableHttpClient client = HttpClients.custom()
.setConnectionManager(cm)
.build();
cm.setDefaultMaxPerRoute(15);
Executor.newInstance(client).execute(Request.Get("/")).discardContent();