I have set up an Opentelemetry collector which exports metrics to New Relic, but it is giving me the following error:
exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval.
{
"kind": "exporter",
"name": "otlp",
"error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded",
"interval": "7.202545441s"
}
This is my collector.yaml file:
receivers:
otlp:
protocols:
grpc:
http:
exporters:
logging:
loglevel: debug
awsxray:
awsemf:
namespace: PaymentService
otlp:
endpoint: ${NEW_RELIC_OPENTELEMETRY_ENDPOINT}
headers:
api-key: ${NEW_RELIC_LICENSE_KEY}
service:
pipelines:
traces:
receivers: [otlp]
exporters: [otlp]
metrics:
receivers: [otlp]
exporters: [otlp]
logs:
receivers: [otlp]
exporters: [otlp]
I am not sure if whether this is an Opentelemetry issue or a New Relic one.
Also, these are other logs that are showing up:
warn zapgrpc/zapgrpc.go:191 [transport] transport: http2Server.HandleStreams failed to read frame: read tcp 127.0.0.1:4317->127.0.0.1:33316: read: connection reset by peer {
"grpc_log": true
}
If that message is intermittent, its likely that isn’t any issue with either the collector or new relic. The internet has intermittent issues - the customers network may temporarily have connectivity issues or low throughput causing the request to timeout. As indicated in the error message, the request will be retried after some interval.
If the message is consistent, they might be misconfiguring the otlp exporter or their network may not be configured correctly to connect to the new relic OTLP endpoint.
Related
getting below error while running packer using vsphere-iso builder.
Error waiting for SSH: Packer experienced an authentication error when trying to connect via SSH. This can happen if your username/password are wrong. You may want to double-check your credentials as part of your debugging process. original error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none password keyboard-interactive], no supported methods remain
config.json
"communicator": "ssh",
"ssh_username": "{{user `ssh_username`}}",
"ssh_password": "{{user `ssh_password`}}",
"ssh_timeout": "30m",
username and password is coming from Jenkins at run time. same has beed updated in autounattend.xml , if i hard-code the credential in config.json file then its working fine. dont know what's the issue
packer debug log
2022/05/09 10:16:20 packer.exe plugin: [DEBUG] Detected authentication error. Increasing handshake attempts.
2022/05/09 10:16:27 packer.exe plugin: [INFO] Attempting SSH connection to 172.16.112.59:22...
2022/05/09 10:16:27 packer.exe plugin: [DEBUG] reconnecting to TCP connection for SSH
2022/05/09 10:16:27 packer.exe plugin: [DEBUG] handshaking with SSH
2022/05/09 10:16:28 packer.exe plugin: [DEBUG] SSH handshake err: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none password keyboard-
I got the solution as below
In autounattend.xml file password was given as plain text true , which needs to be change to false.
I am currently working on an already started project, with the current situation (completely new to Caddy, so sorry if asking something basic):
A docker container with postgresSQL -- container called myappdb
A Spring Boot docker application with some microservices -- container called backend
A caddy docker container that reverse proxies to Spring boot container -- container called caddy
The three containers are in a docker network called project_net.
I worked on the spring boot backend and everything worked well. Accidentally I stopped the caddy container and restarted it, and now I cannot make rest calls to https server anymore.
Here the Caddyfile:
https://app.myapp.it {
tls myapp#gmail.com
reverse_proxy /* {
to backend:48795
flush interval -1
}
}
Here the Dockerfile for caddy image:
FROM caddy:2.4.5
COPY Caddyfile /etc/caddy/Caddyfile
ENV ACME_AGREE=true
EXPOSE 443
All is running on an apache application server and I thing everything is set up because everything worked well until yesterday!
Here the log of the caddy container on start:
2022-02-24T00:49:13.077709051Z 2022/02/24 00:49:13.077 INFO using provided configuration {"config_file": "/etc/caddy/Caddyfile", "config_adapter": "caddyfile"}
2022-02-24T00:49:13.080517683Z 2022/02/24 00:49:13.080 WARN input is not formatted with 'caddy fmt' {"adapter": "caddyfile", "file": "/etc/caddy/Caddyfile", "line": 2}
2022-02-24T00:49:13.082483777Z 2022/02/24 00:49:13.082 INFO admin admin endpoint started {"address": "tcp/localhost:2019", "enforce_origin": false, "origins": ["localhost:2019", "[::1]:2019", "127.0.0.1:2019"]}
2022-02-24T00:49:13.083012379Z 2022/02/24 00:49:13.082 INFO http server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS {"server_name": "srv0", "https_port": 443}
2022-02-24T00:49:13.083044007Z 2022/02/24 00:49:13.082 INFO http enabling automatic HTTP->HTTPS redirects {"server_name": "srv0"}
2022-02-24T00:49:13.083262915Z 2022/02/24 00:49:13.082 INFO tls.cache.maintenance started background certificate maintenance {"cache": "0xc0003bdb90"}
2022-02-24T00:49:13.088176927Z 2022/02/24 00:49:13.087 INFO tls cleaning storage unit {"description": "FileStorage:/data/caddy"}
2022-02-24T00:49:13.088214299Z 2022/02/24 00:49:13.087 INFO tls finished cleaning storage units
2022-02-24T00:49:13.088566440Z 2022/02/24 00:49:13.088 INFO http enabling automatic TLS certificate management {"domains": ["app.myapp.it"]}
2022-02-24T00:49:13.089217858Z 2022/02/24 00:49:13.088 INFO autosaved config (load with --resume flag) {"file": "/config/caddy/autosave.json"}
2022-02-24T00:49:13.089255497Z 2022/02/24 00:49:13.088 INFO serving initial configuration
2022-02-24T00:49:13.090255185Z 2022/02/24 00:49:13.089 INFO tls.obtain acquiring lock {"identifier": "app.myapp.it"}
2022-02-24T00:49:13.104037308Z 2022/02/24 00:49:13.103 INFO tls.obtain lock acquired {"identifier": "app.myapp.it"}
2022-02-24T00:49:13.980759033Z 2022/02/24 00:49:13.980 INFO tls.issuance.acme waiting on internal rate limiter {"identifiers": ["app.myapp.it"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": "myapp#gmail.com"}
2022-02-24T00:49:13.980807648Z 2022/02/24 00:49:13.980 INFO tls.issuance.acme done waiting on internal rate limiter {"identifiers": ["app.myapp.it"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": "myapp#gmail.com"}
2022-02-24T00:49:14.538528714Z 2022/02/24 00:49:14.538 INFO tls.issuance.acme.acme_client trying to solve challenge {"identifier": "app.myapp.it", "challenge_type": "tls-alpn-01", "ca": "https://acme-v02.api.letsencrypt.org/directory"}
2022-02-24T00:49:15.976582736Z 2022/02/24 00:49:15.976 ERROR tls.issuance.acme.acme_client challenge failed {"identifier": "app.myapp.it", "challenge_type": "tls-alpn-01", "status_code": 403, "problem_type": "urn:ietf:params:acme:error:unauthorized", "error": "Cannot negotiate ALPN protocol \"acme-tls/1\" for tls-alpn-01 challenge"}
2022-02-24T00:49:15.976692391Z 2022/02/24 00:49:15.976 ERROR tls.issuance.acme.acme_client validating authorization {"identifier": "app.myapp.it", "error": "authorization failed: HTTP 403 urn:ietf:params:acme:error:unauthorized - Cannot negotiate ALPN protocol \"acme-tls/1\" for tls-alpn-01 challenge", "order": "https://acme-v02.api.letsencrypt.org/acme/order/422657490/66417417610", "attempt": 1, "max_attempts": 3}
2022-02-24T00:49:17.508224302Z 2022/02/24 00:49:17.507 INFO tls.issuance.acme.acme_client trying to solve challenge {"identifier": "app.myapp.it", "challenge_type": "http-01", "ca": "https://acme-v02.api.letsencrypt.org/directory"}
2022-02-24T00:49:18.933967989Z 2022/02/24 00:49:18.933 ERROR tls.issuance.acme.acme_client challenge failed {"identifier": "app.myapp.it", "challenge_type": "http-01", "status_code": 403, "problem_type": "urn:ietf:params:acme:error:unauthorized", "error": "Invalid response from http://app.ripapp.it/.well-known/acme-challenge/QG2yr7WcBg8Wbj9evi8oyk1CzaTFM0Y9bkgkmqq5Iww [91.187.200.219]: \"<html lang=\\\"en\\\" xml:lang=\\\"en\\\" xmlns=\\\"http://www.w3.org/1999/xhtml\\\">\\n<head>\\n <title>Connection denied by Geolocation</title>\\n \""}
2022-02-24T00:49:18.934101729Z 2022/02/24 00:49:18.933 ERROR tls.issuance.acme.acme_client validating authorization {"identifier": "app.myapp.it", "error": "authorization failed: HTTP 403 urn:ietf:params:acme:error:unauthorized - Invalid response from http://app.myapp.it/.well-known/acme-challenge/QG2yr7WcBg8Wbj9evi8oyk1CzaTFM0Y9bkgkmqq5Iww [91.187.200.219]: \"<html lang=\\\"en\\\" xml:lang=\\\"en\\\" xmlns=\\\"http://www.w3.org/1999/xhtml\\\">\\n<head>\\n <title>Connection denied by Geolocation</title>\\n \"", "order": "https://acme-v02.api.letsencrypt.org/acme/order/422657490/66417426840", "attempt": 2, "max_attempts": 3}
2022-02-24T00:49:20.696387362Z 2022/02/24 00:49:20.695 ERROR tls.obtain could not get certificate from issuer {"identifier": "app.myapp.it", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "[app.myapp.it] solving challenges: app.myapp.it: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[http-01 dns-01 tls-alpn-01] remaining=[dns-01]) (order=https://acme-v02.api.letsencrypt.org/acme/order/422657490/66417435240) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
2022-02-24T00:49:21.383148322Z 2022/02/24 00:49:21.382 INFO tls.issuance.zerossl generated EAB credentials {"key_id": "fiNQgkXxmfwTdX1q1gFasg"}
2022-02-24T00:49:24.460492479Z 2022/02/24 00:49:24.459 INFO tls.issuance.acme waiting on internal rate limiter {"identifiers": ["app.myapp.it"], "ca": "https://acme.zerossl.com/v2/DV90", "account": "myapp#gmail.com"}
2022-02-24T00:49:24.460580992Z 2022/02/24 00:49:24.460 INFO tls.issuance.acme done waiting on internal rate limiter {"identifiers": ["app.myapp.it"], "ca": "https://acme.zerossl.com/v2/DV90", "account": "myapp#gmail.com"}
I cannot work without resolving this (on http port is listening current active website, so I cannot test anything over http port).
It seems the problem is that letsencrypt refuses someway the connection. What can I do?
Is there something that I can do to solve? (or also if you need some other files and configurations)
Was thinking about changing to traefik, but the ideal thing is to solve and leave the structure of the project as it is.
I am running in Local using Docker an Ocelot API gateway (https://localhost:5010) service that calls a Dot net core API (https://localhost:5003).
I used self signed certificates and both services are secure:
To make it work locally I add in Ocelot.Development.json:
"DownstreamPathTemplate": "/ActivityTypes",
"DownstreamScheme": "https",
"DownstreamHostAndPorts": [
{
"Host": "xplora.api",
"Port": "443"
}
],
"UpstreamPathTemplate": "/ActivityTypes",
"UpstreamHttpMethod": [ "GET" ],
"RateLimitOptions": {
"ClientWhitelist": [],
"EnableRateLimiting": true,
"Period": "3s",
"PeriodTimespan": 1,
"Limit": 1
},
"FileCacheOptions": { "TtlSeconds": 30 }
}
Where xplora.api is the name of the container defined in docker-compose.
container_name: xplora.api
environment:
When I access the Ocelot api gateway I get this error:
xploraproject-xploraapigateway-1 | requestId: 0HMDG3PCRH54C:00000001, previousRequestId: no previous request id, message: Error Code: ConnectionToDownstreamServiceError Message: Error connecting to downstream service, exception: System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
xploraproject-xploraapigateway-1 | ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure: RemoteCertificateNameMismatch, RemoteCertificateChainErrors
xploraproject-xploraapigateway-1 | at System.Net.Security.SslStream.SendAuthResetSignal(ProtocolToken message, ExceptionDispatchInfo exception)
xploraproject-xploraapigateway-1 | at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
xploraproject-xploraapigateway-1 | at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Boolean async, Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
xploraproject-xploraapigateway-1 | --- End of inner exception stack trace ---
xploraproject-xploraapigateway-1 | at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Boolean async, Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
I suspect the problem is that when I try to access this services using localhost it works fine because the certificate is assigned to localhost domain but when Ocelot tries to use https://xplora.api:5003 this certificate is not working. Am I right? If so.. how can I make it work? I used "DangerousAcceptAnyServerCertificateValidator": true in Ocelot.Development.json and it works but I want a real solution, without using DangerousAcceptAnyServerCertificateValidator.
Thanks
I am unable to figure out how to set the tls protocol version. This is part of the code I am using:
let tlsOptions = NWProtocolTLS.Options()
sec_protocol_options_add_pre_shared_key(...)
sec_protocol_options_add_tls_ciphersuite(tlsOptions.securityProtocolOptions, TLS_PSK_WITH_AES_128_GCM_SHA256)
sec_protocol_options_set_min_tls_protocol_version(tlsOptions.securityProtocolOptions, tls_protocol_version_t.DTLSv12)
let parameters = NWParameters(dtls: tlsOptions)
connection = NWConnection(host: ..., port: NWEndpoint.Port(rawValue: 2100)!, using: parameters)
connection.start(queue: .main)
When I run this code I get the following errors:
2019-10-19 14:30:31.628250+0200 MyApp[4906:117290] [BoringSSL] boringssl_helper_tls_protocol_version_from_SSLProtocol(111) [C6:1][0x7fd33fc4d4f0] Unknown SSLProtocol version: 11
2019-10-19 14:30:31.649137+0200 MyApp[4906:117290] [BoringSSL] boringssl_context_handle_fatal_alert(1874) [C6:1][0x7fd33fc4d4f0] read alert, level: fatal, description: bad record mac
2019-10-19 14:30:31.650008+0200 MyApp[4906:117290] [BoringSSL] boringssl_session_handshake_error_print(111) [C6:1][0x7fd33fc4d4f0] 140546626918936:error:100003fc:SSL routines:OPENSSL_internal:SSLV3_ALERT_BAD_RECORD_MAC:/BuildRoot/Library/Caches/com.apple.xbs/Sources/boringssl_Sim/boringssl-283.40.1/ssl/tls_record.cc:587:SSL alert number 20
2019-10-19 14:30:31.650105+0200 MyApp[4906:117290] [BoringSSL] nw_protocol_boringssl_handshake_negotiate_proceed(724) [C6:1][0x7fd33fc4d4f0] handshake failed at state 12288
It appears that the tls protocol version is unknown. This is the function I am using to set the version: https://developer.apple.com/documentation/security/3180218-sec_protocol_options_set_min_tls
I am using the tls_protocol_version_t.DTLSv12 contant so why does this show an error?
Windows 10 setup :
Thingsboard server running as local service on windows
Thingsboard.yml mqtt parameters
MQTT server parameters
mqtt:
bind_address: "${MQTT_BIND_ADDRESS:0.0.0.0}"
bind_port: "${MQTT_BIND_PORT:1883}"
adaptor: "${MQTT_ADAPTOR_NAME:JsonMqttAdaptor}"
timeout: "${MQTT_TIMEOUT:10000}"
Thingsboard gateway service running as local service on windows
> tb-gateway.yml mqtt parameters
> mqtt:
> enabled: true
> configuration: mqtt-config.json
configuration file of mqtt is set by default,below.
mqtt-config.json mqtt parameters
"brokers": [
{
"host": "localhost",
"port": 1883,
"ssl": false,
"retryInterval": 3000,
"credentials": {
"type": "anonymous"
},
These are the only two services running on my laptop, I published a mqtt message as follows, per docs :
mosquitto_pub -h localhost -p 1883 -u "XXXXXXXX" -t "sensors" -m '{"serialNumber":"TB-GW-SN-001","model":"TB-GW-T1000","temperature":35.2}'
I see errors in both the logs.
thingsboard.log
2018-01-10 20:14:56,174 [nioEventLoopGroup-6-11] INFO o.t.s.t.mqtt.MqttTransportHandler - [mqtt815] Processing connect msg for client: efd91958-ba8f-480a-9a56-ad9d5588c8c7!
2018-01-10 20:14:56,177 [nioEventLoopGroup-6-12] INFO o.t.s.t.mqtt.MqttTransportHandler - [127.0.0.1:51192] Invalid message received
2018-01-10 20:14:59,183 [nioEventLoopGroup-6-1] INFO o.t.s.t.mqtt.MqttTransportHandler - [mqtt817] Processing connect msg for client: efd91958-ba8f-480a-9a56-ad9d5588c8c7!
2018-01-10 20:14:59,188 [nioEventLoopGroup-6-2] INFO o.t.s.t.mqtt.MqttTransportHandler - [127.0.0.1:51194] Invalid message received
2018-01-10 20:15:02,193 [nioEventLoopGroup-6-3] INFO o.t.s.t.mqtt.MqttTransportHandler - [mqtt819] Processing connect msg for client: efd91958-ba8f-480a-9a56-ad9d5588c8c7!
2018-01-10 20:15:02,197 [nioEventLoopGroup-6-4] INFO o.t.s.t.mqtt.MqttTransportHandler - [127.0.0.1:51196] Invalid message received
error in thingsboard gateway is strange, rather.
tb-gateway.log
2018-01-10 20:14:59,191 [main] WARN o.t.g.e.m.client.MqttBrokerMonitor - [localhost:1883] MQTT broker connection failed!
org.eclipse.paho.client.mqttv3.MqttException: Connection lost
at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:164)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.EOFException: null
at java.io.DataInputStream.readByte(Unknown Source)
at org.eclipse.paho.client.mqttv3.internal.wire.MqttInputStream.readMqttWireMessage(MqttInputStream.java:92)
at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:116)
... 1 common frames omitted
2018-01-10 20:15:02,198 [main] WARN o.t.g.e.m.client.MqttBrokerMonitor - [localhost:1883] MQTT broker connection failed!
org.eclipse.paho.client.mqttv3.MqttException: Connection lost
at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:164)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.EOFException: null
at java.io.DataInputStream.readByte(Unknown Source)
at org.eclipse.paho.client.mqttv3.internal.wire.MqttInputStream.readMqttWireMessage(MqttInputStream.java:92)
at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:116)
... 1 common frames omitted
What am I doing wrong? I have mosquitto installed locally, hence used the mosquitto_pub tool to publish the messages.
Any clues, folks?
All is well.
As https://stackoverflow.com/users/3203737/andrew pointed out, I wasnt running mosquitto and the ports were the same. I cleaned up the mess and the devices were registered perfectly using TB-gateway, in the dashboard.