I have situation where I want to create large number of entities on orion. I am using docker version of Orion and mongo with this docker-compose.
version: "3"
services:
mongo:
image: mongo:3.4
volumes:
- /data/docker-mongo/db:/data/db
- /data/docker-mongo/log/mongodb.log:/var/log/mongodb/mongod.log
command: --nojournal
orion:
image: fiware/orion
volumes:
- /data/docker-mongo/log/contextBroker.log:/tmp/contextBroker.log
links:
- mongo
ports:
- "1026:1026"
command: -dbhost mongo
Now problems happens when I want to upload 2000 entities (opening new connection for each, I know it can be done different but for now this is request), I successfully create no more than 600 (or less never exact number) of them rest fail to create with error:
"error": {
"errno": "ECONNRESET",
"code": "ECONNRESET",
"syscall": "read"
},
So I assume this issue has something to do with maxConnections, reqPoolSize etc settings in Orion. But in docker I failed to locate Orion config file, I have no way of knowing when I type commands like contextBroker -maxConnections 123456 that setting is being accepted by Orion and docker container.
Also log of Orion is empty, and i cannot determined what is causing this issue when Orion is running on docker.
So main questions:
Can Orion running on docker be used in same manner as Orion running on VM (are there some fallbacks)
And how do I check this problem when Orion is running in docker, because I read a lot of docs/issues but no luck (or I missed something).
If you have some advice/soultion it would really help.
Thanks
{
"orion" : {
"version" : "1.13.0-next",
"uptime" : "2 d, 15 h, 46 m, 34 s",
"git_hash" : "ae72acf9e8eeaacaf4eb138f7de37bfee4514c6b",
"compile_time" : "Fri May 4 10:12:18 UTC 2018",
"compiled_by" : "root",
"compiled_in" : "1901fd6bb51a",
"release_date" : "Fri May 4 10:12:18 UTC 2018",
"doc" : "https://fiware-orion.readthedocs.org/en/master/"
}
}
{ Error: socket hang up
at createHangUpError (_http_client.js:313:15)
at Socket.socketOnEnd (_http_client.js:416:23)
at Socket.emit (events.js:187:15)
at endReadableNT (_stream_readable.js:1090:12)
at process._tickCallback (internal/process/next_tick.js:63:19) code: 'ECONNRESET' }
error:
{ Error: connect ECONNREFUSED ipofvirtualm:1026
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1174:14)
errno: 'ECONNREFUSED',
code: 'ECONNREFUSED',
syscall: 'read',
address: 'ipofvm',
port: 1026 },
options:
{ method: 'POST',
uri: 'http://ip:1026/v2/entities?options=keyValues',
headers:
{ 'Fiware-Service': 'some service',
'Fiware-ServicePath': 'some servicepath' },
body:
{ id: 'F0B935',
type: 'Transaction',
refEmitter: 'F0B935',
refReceiver: '7501JXG',
refCapturer: 'testtdata',
date: '12/12/2017 13:25',
refTransferredResources: 'testtdata',
transferredLoad: 92 },
json: true,
callback: [Function: RP$callback],
transform: undefined,
simple: true,
resolveWithFullResponse: false,
transform2xxOnly: false },
I am using request promise library for making calls, i try others they had same issue. Now since i cannot send u all 2000 responses i will try to describe. So it when i start to send this it behave. It create like 30 entities then next few or more return response saying ECONNRESET then it start creating again and so on.
What confuse me is that it is not failing totally meaning it works but not as intended. Also it seem that Orion close socket or hang it up some period then he is open again and create as normal and so on. If u need any more info ask, and thanks for quick answer.
instead of opening a new connection per entity why don't you use
POST /v2/op/update
and create all entities in just one batch? or a couple of batches
See some code at
https://github.com/Fiware/dataModels/blob/master/Weather/WeatherObserved/harvest/spain_weather_observed_harvest.py#L235
With regards to CLI argument passing to CB running inside docker, use the command line in docker compose file, eg:
command: -dbhost mongo -maxConnections 123456
However, I'm not sure that would help to solve the problem, as Orion should deal with your use case without any special customization. Looking to the error message (which seems to be about some problema at TCP layer) I wonder if docker networking layer is acting as bottleneck in some way...
In addition, the suggestion done by Jose Manuel Cantera about using POST /v2/op/update would be a good idea. It would reduce connection stress at network layer and may help to alleviate the problem.
If you cannot change your update strategy, maybe using an inter-request delay (100-200ms) could also help.
Related
compose artifactory pro v6.9.0 running
In my compose I have two services :
image: docker.bintray.io/jfrog/artifactory-pro:6.9.0
image: docker.io/library/postgres:9.6.11
I was able to upgrade to 6.23.13 without any problem just by changing the version of the image.
When I try the same thing with any 7.X version (after upgrading to at least 6.10 as the doc says), I have errors.
For example, trying 7.21.3, I have these warnings
2022-08-01T08:41:30.343L [tomct] [WARNING] [ ] [org.apache.catalina.startup.HostConfig] [org.apache.catalina.startup.HostConfig deployDescriptor] - A docBase [/opt/jfrog/artifactory/app/artifactory/tomcat/webapps/access.war] inside the host appBase has been specified, and will be ignored
2022-08-01T08:41:30.343L [tomct] [WARNING] [ ] [org.apache.catalina.startup.HostConfig] [org.apache.catalina.startup.HostConfig deployDescriptor] - A docBase [/opt/jfrog/artifactory/app/artifactory/tomcat/webapps/artifactory.war] inside the host appBase has been specified, and will be ignored
...
2022-08-01T08:41:37.344Z [jfrt ] [WARN ] [ce1b2553475da56b] [c.z.h.p.ProxyConnection:182 ] [ocalhost-startStop-2] - HikariCP Main - Connection org.apache.derby.impl.jdbc.EmbedConnection#1597179442 (XID = 24), (SESSIONID = 3), (DATABASE = {db.home}), (DRDAID = null) marked as broken because of SQLSTATE(0A000), ErrorCode(20000)
java.sql.SQLFeatureNotSupportedException: Feature not implemented: No details.
and these errors
08:41:34,803 |-ERROR in ch.qos.logback.core.joran.action.AppenderAction - Could not create an Appender of type [org.artifactory.usage.appender.UsageTrafficTimeBasedRollingFileAppender]. ch.qos.logback.core.util.DynamicClassLoadingException: Failed to instantiate type org.artifactory.usage.appender.UsageTrafficTimeBasedRollingFileAppender
...
2022-08-01T08:41:37.113Z [jfrt ] [ERROR] [ce1b2553475da56b] [d.d.l.DbDistributeLocksDao:506] [ocalhost-startStop-2] - Unable to detect database version Unable to get connection from unique lock data source
2022-08-01T08:41:37.353Z [jfrt ] [ERROR] [ce1b2553475da56b] [tifactoryHomeConfigListener:55] [ocalhost-startStop-2] - Failed initializing Home. Caught exception:
java.lang.IllegalStateException: Could not find database table: db_properties
After reading the docs I do not clearly understand if I have to download the new docker-compose package from jfrog. I've tried but once there, the config.sh ask for external database and no question about reusing existing image directory.
Thx for help
Not sure if this is still relevant for you, but I got the same problem. As far as I see there are new DB connection environment variables, that must be set. This error message is just a symptom, because without the new DB connection env, Artifactory will try to use an in-memory database (Apache Derby). Something like this is needed to fix it (as Docker Compose config of the Artifactory container):
environment:
- JF_SHARED_DATABASE_TYPE=postgresql
- JF_SHARED_DATABASE_USERNAME=${POSTGRESQL_USERNAME}
- JF_SHARED_DATABASE_PASSWORD=${POSTGRESQL_PASSWORD}
- JF_SHARED_DATABASE_URL=jdbc:postgresql://postgresql:5432/artifactory
- JF_SHARED_DATABASE_DRIVER=org.postgresql.Driver
I'm new to Docker and RabbitMQ and I've been trying for 2 days to solve an error in my docker containers which contains: api_client, api_consumer, RabbitMQ. I've done a research and tried to read as many threads with this problem as I found but unfortunately nothing helped.
So here is my code:
compose.yml
services:
api_client:
build:
context: ""
dockerfile: apps/api_client/Dockerfile
env_file:
- ./config/.env.local
restart: always
ports:
- "3000:3000"
depends_on:
- rabbitmq
api_consumer:
build:
context: ""
dockerfile: apps/api_consumer/Dockerfile
env_file:
- ./config/.env.local
restart: always
depends_on:
- rabbitmq
rabbitmq:
image: rabbitmq:3.9.2-management
container_name: rabbitmq
hostname: rabbitmq
volumes:
- /var/lib/rabbitmq
- ./rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf
ports:
- "5672:5672"
- "15672:15672"
main.ts (in api_consumer)
async function bootstrap() {
const app = await NestFactory.createMicroservice<MicroserviceOptions>(
ApiConsumerModule,
{
transport: Transport.RMQ,
options: {
queue: 'test_queue',
urls: ['amqp://guest:guest#rabbitmq:5672'],
queueOptions: {
durable: true
}
}
},
);
const AWSAppConfig = app.get(AwsAppconfigLoaderService);
const Log = new Logger(ApiClientService.name);
await AWSAppConfig.loadAWSAppConfig()
.then((_) => {
Log.log(AWSAppConfig.getAppName());
})
.catch((err) => {
Log.error(
`Error occured while downloading AWS Config: ${JSON.stringify(
err,
)}`,
);
});
await app.listen();
}
bootstrap();
api-client.module.ts (in api_client)
#Module({
imports: [
ConfigModule.forRoot({
isGlobal: true,
load: [AppConfig],
}),
ClientsModule.register([{
name: GET_MATCHED_DEVICES,
transport: Transport.RMQ,
options: {
queue: 'test_queue',
urls: ['amqp://guest:guest#rabbitmq:5672'],
queueOptions: {
durable: true
}
}
},
]),
AwsAppconfigLoaderModule,
],
controllers: [ApiClientController],
providers: [ApiClientService],
})
export class ApiClientModule {}
Functionality is simple- when GET on localhost:3000 (api_client) is called, it calls (in controller) return this.client.send('getSample', "hello") and then in api_consumer it should call (in controller)
#MessagePattern('getSample')
getSample(data): string {
Logger.debug(data)
return "It works!";
}
When all docker services start there is the first error:
Disconnected from RMQ. Trying to reconnect.
{
"err": {
"code": 406,
"classId": 60,
"methodId": 40
}
}
And then when I try to access the localhost:3000, this error always occur:
Error: Channel closed by server: 406 (PRECONDITION-FAILED) with message "PRECONDITION_FAILED - fast reply consumer does not exist"
Both errors come from api_client.
What I've tried and didn't help:
-change durable to false or remove durable options completely
-add noAck
-remove queue in adminer on localhost:15672 (which works fine)
-remove port from urls in both microservices
-as you can see the queue options are the same in both microservices
Now the most absurd thing is that this code did work absolutely fine until I started to work on second compose file (and dockerfiles) for local (faster) development with volumes. Then suddenly these errors have started to occur and even if I undid all my code changes the errors are still there. Because of this I've wiped all my volumes (with docker system prune -a --volumes) many times but still nothing. My OS is Ubuntu 20.04
I am completely out of ideas so I've written it here in hope for some help, please.
The failure occurs because the app can not connect to rabbitmq, the problem resides in your docker compose. Make sure the services are using the same network in docker and are able to communicate.
I know it sounds strange but I have had the same problem yesterday, also using a docker-compose file and without making any changes to it or to the rabbitmq logic, it broke. I tried many things, and when i changed
return this.client.send('getSample', "hello") and #MessagePattern
to
return this.client.emit('getSample', "hello") and #EventPattern
thinking it wouldn't make sense for it to fix the issue, it actually did.
I suggest you try that and tell me if it works, sorry if i can't help you more.
So after 5 days of figuring out what causes the error I've found out the problem lies in Nest itself. The bug has been documented here. I've deleted all the code associated with #nestjs/microservices and tried to use this approach using amqlib only and the request/response functionality finally works.
We have been trying for quite a while to connect our email server to the rocketchat, but we keep getting an authorisation mistake (see log below). It might have to do with the fact that in the form one has to add user and password but it’s not very clear user and password of what. We have tried all possible combinations we could think of, but we are kind of out of ideas. Suggestions?
Thanks in advance.
Server Setup Information
Version of Rocket.Chat Server: CE 3.15.1
Operating System: ubuntu 20.04
Deployment Method: digitalocean IaaS
Number of Running Instances: 1
Log:
Exception in callback of async function: Error: Authentication failed. at Connection._resTagged (/opt/Rocket.Chat/programs/server/npm/node_modules/imap/lib/Connection.js:1502:11) at Parser. (/opt/Rocket.Chat/programs/server/npm/node_modules/imap/lib/Connection.js:194:10) at Parser.emit (events.js:314:20) at Parser.EventEmitter.emit (domain.js:483:12) at Parser._resTagged (/opt/Rocket.Chat/programs/server/npm/node_modules/imap/lib/Parser.js:175:10) at Parser._parse (/opt/Rocket.Chat/programs/server/npm/node_modules/imap/lib/Parser.js:139:16) at Parser._tryread (/opt/Rocket.Chat/programs/server/npm/node_modules/imap/lib/Parser.js:82:15) at TLSSocket.Parser.cbReadable (/opt/Rocket.Chat/programs/server/npm/node_modules/imap/lib/Parser.js:53:12) at TLSSocket.emit (events.js:314:20) at TLSSocket.EventEmitter.emit (domain.js:483:12) at emitReadable (_stream_readable.js:557:12) at processTicksAndRejections (internal/process/task_queues.js:83:21) { type: ‘no’, textCode: ‘AUTHENTICATIONFAILED’, source: ‘authentication’ }
After proceed with the installation of WhatsApp Business API (developer single instance) in docker on windows 10 Enterprise, I´m facing the following msg error when using https://192.168.43.200:8080/v1/health by postman
Error msg:
{
"meta": {
"version": "v2.33.3",
"api_status": "stable"
},
"errors": [
{
"code": 1014,
"title": "Internal error",
"details": "php_network_getaddresses: getaddrinfo failed: Temporary failure in name resolution. Please check if wacore is running: wacore:6252"
}
]
}
Looking in log files it´s seems that the core is listening in one port that is different from expected by web
---> Web log
[2021-02-24 12:46:38.560338] app.INFO: [064af96616514f6f8b41fc530047db4b] Matched route "{route}". {"route":"GET_v1_health","route_parameters":{"_controller":"WhatsApp\Controller\HealthController::getHealth","_route":"GET_v1_health"},"request_uri":"https://192.168.43.200:8080/v1/health","method":"GET"} []
[2021-02-24 12:46:38.587929] app.INFO: [064af96616514f6f8b41fc530047db4b] Guard authentication successful! {"token":"[object] (Symfony\Component\Security\Guard\Token\PostAuthenticationGuardToken: PostAuthenticationGuardToken(user="admin", authenticated=true, roles="ROLE_ADMIN"))","authenticator":"WhatsApp\Security\TokenAuthenticator"} []
[2021-02-24 12:47:14.646964] app.INFO: [064af96616514f6f8b41fc530047db4b] Response: {"meta":{"version":"v2.33.3","api_status":"stable"},"errors":[{"code":1014,"title":"Internal error","details":"php_network_getaddresses: getaddrinfo failed: Temporary failure in name resolution. Please check if wacore is running: wacore:6252"}]} []
[2021-02-24 12:47:14.650236] app.INFO: [064af96616514f6f8b41fc530047db4b] Request GET_/v1/health returns 500 in 36269.15 ms [] []
===================================================================================
Core log
D 2021-02-24 12:10:39.282 UTC 28 apiendpointmanager.cpp:190] Endpoint "healthcheck" is listening on address "0.0.0.0" port 6253 req_id=Main
D 2021-02-24 12:10:39.282 UTC 29 apiendpointmanager.cpp:190] Endpoint "control" is listening on address "0.0.0.0" port 6252 req_id=Main
===================================================================================
No one change was executed in docker-compose.yml. Is the same that is on GitHub (https://github.com/WhatsApp/WhatsApp-Business-API-Setup-Scripts) except network mode was changed "bridge" to "nat" since I´m using windows
$ docker network ls
NETWORK ID NAME DRIVER SCOPE
9d811d5d3283 Default Switch ics local
27dc22b69113 nat nat local
4e2733cd792d none null local
$ docker container ls -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8d7000856b95 docker.whatsapp.biz/web:v2.33.3 "/opt/whatsapp/bin/w…" 17 hours ago Exited (4294967295) 6 minutes ago postgres_waweb_1
909781cdb775 docker.whatsapp.biz/coreapp:v2.33.3 "/opt/whatsapp/bin/w…" 17 hours ago Up 5 minutes 6250-6253/tcp postgres_wacore_1
7d68b7a61cad postgres:10.6 "docker-entrypoint.s…" 17 hours ago Up 6 minutes 5432/tcp, 33060/tcp, 0.0.0.0:33060->3306/tcp postgres_db_1
219b1e393f21 nginx "/docker-entrypoint.…" 42 hours ago Exited (4294967295) 41 hours ago nostalgic_jennings
The current WA_API_VERSION is 2.33.3
Database used is Postgress10.6
Looking at a similar question answered by #WeiyanWang (How to access wacore container using WhatsApp Business API) I tried to execute the same command in Postgres, but no success
Regards,
after some investigation on scenery described, I found some settings mistakes in windows docker.
To fix these problems following the steps:
Change docker settings to original installation
Choose the option "Switch to linux container..."
Reinstall the WhatsApp Business Api following documentation
Note: Is not necessary change the "bridge" to "nat" network settings. I just only changed the "waweb" from 9090:443 to 8080:443
I am facing problems deploying a smart contract on my private blockchain network. I created my blockchain network on three VMs (miners) using puppeth on a fourth VM (controller) by following the steps in this blog: https://medium.com/#collin.cusce/using-puppeth-to-manually-create-an-ethereum-proof-of-authority-clique-network-on-aws-ae0d7c906cce
Afterwards, I installed truffle on one of the miners VMs and i initialized truffle using the command:
truffle init
Then I wrote a simple hello world smart contract, compiled it and deployed it on truffle development blockchain and it worked. However, I tried to deploy it on my private blockchain but I can't connect to the network.
The admin.nodeInfo command in geth console returns the folowing output:
docker exec -it 954cd3955065 geth attach ipc:/root/.ethereum/geth.ipc
Welcome to the Geth JavaScript console!
instance: Geth/v1.9.25-unstable-ead81461-20201123/linux-amd64/go1.15.5
coinbase: 0xe8cc4bea2cfdfd14cddefe1141bedd109576b9a9
at block: 78558 (Tue Dec 01 2020 22:01:02 GMT+0000 (UTC))
datadir: /root/.ethereum
modules: admin:1.0 clique:1.0 debug:1.0 eth:1.0 miner:1.0 net:1.0 personal:1.0 rpc:1.0 txpool:1.0 web3:1.0
To exit, press ctrl-d
> admin.nodeInfo
{
enode: "enode://7206ca3c62f6db47e1230dcf14a765d4c9b4870a66470dbb21fcc5ed2fab2167d6bcc47eec8044c42037b3e6e0017aeb8ddfc3580471da54a6c7274a0c1fe46b#10.100.2.32:30303",
enr: "enr:-Je4QGXlVAESp8r2s1uHBJxoDLWQo8IvZsbe5sX2YRBb0un9Gdlt8nfDKQBR_j0lDPtaoCCuis4cJJlqtEHfa4tLO2EIg2V0aMfGhG5b-B6AgmlkgnY0gmlwhApkAiCJc2VjcDI1NmsxoQNyBso8YvbbR-EjDc8Up2XUybSHCmZHDbsh_MXtL6shZ4N0Y3CCdl-DdWRwgnZf",
id: "027a351994ac1b127df56180b6210310cc0164f17f1b12c167cb167c4ffaa122",
ip: "10.100.2.32",
listenAddr: "[::]:30303",
name: "Geth/v1.9.25-unstable-ead81461-20201123/linux-amd64/go1.15.5",
ports: {
discovery: 30303,
listener: 30303
},
protocols: {
eth: {
config: {
byzantiumBlock: 0,
chainId: 1515,
clique: {...},
constantinopleBlock: 0,
eip150Block: 0,
eip150Hash: "0x0000000000000000000000000000000000000000000000000000000000000000",
eip155Block: 0,
eip158Block: 0,
homesteadBlock: 0,
istanbulBlock: 0,
petersburgBlock: 0
},
difficulty: 98201,
genesis: "0x17f752387c901db617cf0594ecd2cb9811dfcd666318c2e0e7cb0239471da979",
head: "0xf8a37d0390558746901faa55463c127c553f02cf2d23ce0cb469fcd470c810f9",
network: 1515
}
}
}
I tried adding the network configuration in truffle-config.js like this:
devnet2: {
host: "localhost",
port: "30303", //port where the node is
network_id: "*",
from: 0x91cd7b879fefff34259d577a56d290b3315bf9b3 // Treats this network as if it was a public net. (default: false)
}
then, when deploying using the command truffle deploy --network devnet2 i always get this error:
Compiling your contracts...
===========================
> Everything is up to date, there is nothing to compile.
/usr/local/lib/node_modules/truffle/build/webpack:/packages/provider/index.js:56
throw new Error(errorMessage);
^
Error: There was a timeout while attempting to connect to the network.
Check to see that your provider is valid.
If you have a slow internet connection, try configuring a longer timeout in your Truffle config. Use the networks[networkName].networkCheckTimeout property to do this.
at Timeout.setTimeout (/usr/local/lib/node_modules/truffle/build/webpack:/packages/provider/index.js:56:1)
at ontimeout (timers.js:436:11)
at tryOnTimeout (timers.js:300:5)
at listOnTimeout (timers.js:263:5)
at Timer.processTimers (timers.js:223:10)
I tried extending the timeout limit but it didn't work. I also tried using Web3 Providers (HTTPProvider and IPCProvider) but without any luck (i can give more details, if needed).
Any help is well appreciated because i spent a lot of time on it without getting anywhere. Unfortunately, i couldn't find anything on deploying smart contracts to a node that is running on docker. If needed, i can gladly give more details about what i did.
I managed to run smart contracts on a private network, not using docker however. Some things come to mind. did you run a miner on your network? you will need to run a miner so that the contract gets migrated. Did you make sure that the gaslimit is met when running the contract? the miners will wait for the max gas limit to be reached before processing any request.
Did you already deploy the contract? in the migration scripts you either create a new migration script by bumping the version or use the reset flag to run all migration scripts again.