I didn't configure the project and I get this error whenever I run my job 'The network default doesn't have rules that open TCP ports 1-65535 for internal connection with other VMs. Only rules with a target tag 'dataflow' or empty target tags set apply. If you don't specify such a rule, any pipeline with more than one worker that shuffles data will hang. Causes: No firewall rules associated with your network.'
google_cloud_options = p_options.view_as(GoogleCloudOptions)
google_cloud_options.region = 'europe-west1'
google_cloud_options.project = 'my-project'
google_cloud_options.job_name = 'rim'
google_cloud_options.staging_location = 'gs://my-bucket/binaries'
google_cloud_options.temp_location = 'gs://my-bucket/temp'
p_options.view_as(StandardOptions).runner = 'DataflowRunner'
p_options.view_as(SetupOptions).save_main_session = True
p_options.view_as(StandardOptions).streaming = True
p_options.view_as(WorkerOptions).subnetwork = 'regions/europe-west1/subnetworks/test'
p = beam.Pipeline(options=p_options)
I tried to specify --network 'test' in the command line since it is not the default configuration
It looks like your default firewall rules were modified and dataflow detected this and prevented your job from launching. Could you verify your firewall rules were not modified in your project?. Please take a look at the documentation here. You will also find a command here to restore the firewall rules:
gcloud compute firewall-rules create [FIREWALL_RULE_NAME] \
--network [NETWORK] \
--action allow \
--direction ingress \
--target-tags dataflow \
--source-tags dataflow \
--priority 0 \
--rules tcp:1-65535
Pick a name for the firewall, and provide a network name. Then pass in the network name with --network when you launch the dataflow job. If you have a network named 'default' dataflow will try to use that automatically, so you won't need to pass in --network. If you've deleted that network you may wish to recreate it.
As of now, till apache beam version 2.19.0. There is no provision from dataflow to set network tag for its VM.
Instead while creating firewall rule, we should add a tag for dataflow.
gcloud compute firewall-rules create FIREWALL_RULE_NAME \
--network NETWORK \
--action allow \
--direction DIRECTION \
--target-tags dataflow \
--source-tags dataflow \
--priority 0 \
--rules tcp:12345-12346
See this link for more details
https://cloud.google.com/dataflow/docs/guides/routes-firewall
Related
I am trying to settup a sawtooth network like in the following tutorial.
I use the following docker-compose.yaml file as instructed in the tutorial to create a sawtooth network of 5 nodes using the pbft consesus engine.
The problem is that once I try to check whether peering has occurred on the network by submit a peers query to the REST API on the first node from the shell container I get a connection refused answer:
curl: (7) Failed to connect to sawtooth-rest-api-default-0 port 8008: Connection refused
Connectivity among the containers seems to be working fine (I have checked with ping from inside the containers).
I suspect that the problem stems from the following line of the docker-compose.yaml file:
sawtooth-validator -vv \
--endpoint tcp://validator-0:8800 \
--bind component:tcp://eth0:4004 \
--bind consensus:tcp://eth0:5050 \
--bind network:tcp://eth0:8800 \
--scheduler parallel \
--peering static \
--maximum-peer-connectivity 10000
and more specifically the --bind option. I noticed that eth0 is not resolved properly to the IP of the container network, but instead to the loopback:
terminal output for validator 0
Do you believe that this could be the problem or is there something else I might have overlooked?
Thannk you
Looks like the moment I post something here the answer magically reveals itself.
The backslash characters are not interpreted correctly so the --bind option was not taken into account and the default is the loopback.
What I did to fix it is either put the whole command in the same line or use double backslash.
I have a PhpStorm IDE set up with Xdebug, a docker container that is set up with Xdebug included and the browser module applied. However, when I try to 'pick up the phone' and refresh the code contained in the environment, nothing is ever picked up.
I've tried a variety of ports and server names, along with the tutorials I can find. I am not certain if it's security or a bad setup with my docker but I am relatively certain the PhpStorm setup is correct.
I've tried exposing ports in the Dockerfile (9000 & 9001)
My .php file is just echo and some math with break points applied:
echo("TEST 1<br>");
$test = 2;
echo("TEST " . $test . "<br>");
$testArray = xdebug_get_code_coverage();
var_dump($testArray);
phpinfo();
dd("TEST 3");
In my .env file, the following is defined:
PHP_IDE_CONFIG=serverName=jumbledowns-demo
XDEBUG_CONFIG=remote_host=localhost remote_port=9001
And my Dockerfile sets up Xdebug thus:
RUN pecl install xdebug; \
docker-php-ext-enable xdebug; \
echo "error_reporting = E_ALL" >> /usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini; \
echo "display_startup_errors = On" >> /usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini; \
echo "display_errors = On" >> /usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini; \
echo "xdebug.remote_enable=1" >> /usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini;
PHPinfo on the browser side shows the code as expected, and the following settings for Xdebug:
xdebug.remote_host localhost localhost
xdebug.remote_log no value no value
xdebug.remote_mode req req
xdebug.remote_port 9001 9000
I've tried using URLs and assigned IP addresses in the above and can change them as needed.
Ideally, I'd like to be able to have a command line log or output that is showing what Xdebug is trying to call and on what port.
In order to find out what Xdebug is trying to do, you need to set the xdebug.remote_log setting to something non-empty, such as xdebug.remote_log=/tmp/xdebug.log. That file will include all connection attempts, and communication protocol contents if a connection is made.
remote_host=localhost
Is almost certainly not correct though, as PHP and Xdebug, running in your docker container need to open a connection to this host, and localhost, is not going to be the right hostname/IP address. It's more likely that it's host.docker.internal or something like that, or rather, hard code the IP address of the machine where your IDE runs, as is accessible to from your docker container.
I created a service called accountservice and replicated it 3 times after. In my service I get IP address of the producing service instance and populate it in JSON response. The question is everytime I run curl $manager-ip:6767/accounts/10000 the returned IP is the same as before (I tried 100 times)
manager-ip environment variable:
set -x manager-ip (docker-machine ip swarm-manager-1)
Here's my Dockerfile:
FROM iron/base
EXPOSE 6767
ADD accountservice-linux-amd64 /
ADD healthchecker-linux-amd64 /
HEALTHCHECK --interval=3s --timeout=3s CMD ["./healthchecker-linux-amd64", "-port=6767"] || exit 1
ENTRYPOINT ["./accountservice-linux-amd64"]
And here's my automation script to build and run service:
#!/usr/bin/env fish
set -x GOOS linux
set -x CGO_ENABLED 0
set -x GOBIN ""
eval (docker-machine env swarm-manager-1)
go get
go build -o accountservice-linux-amd64 .
pushd ./healthchecker
go get
go build -o ../healthchecker-linux-amd64 .
popd
docker build -t azbshiri/accountservice .
docker service rm accountservice
docker service create \
--name accountservice \
--network my_network \
--replicas=1 \
-p 6767:6767 \
-p 6767:6767/udp \
azbshiri/accountservice
And here's the function I call to get the IP:
package common
import "net"
func GetIP() string {
addrs, err := net.InterfaceAddrs()
if err != nil {
return "error"
}
for _, addr := range addrs {
ipnet, ok := addr.(*net.IPNet)
if ok && !ipnet.IP.IsLoopback() {
if ipnet.IP.To4() != nil {
return ipnet.IP.String()
}
}
}
panic("Unable to determine local IP address (non loopback). Exiting.")
}
And I scale the service using the command below:
docker service scale accountservice=3
A few things:
Your results are normal. By default, a Swarm service has a VIP (virtual IP) in front of the service tasks to act as a load balancer. Trying to reach that service from inside the virtual network will only show that IP.
If you want to use a round-robin approach and skip the VIP, you could create a service with --endpoint-mode=dnsrr that would then return a different service task for each DNS request (but your client might be caching DNS names, causing that to show the same IP, which is why VIP is usually better).
If you wanted to get a list of IP's for task replicas, do a dig tasks.<servicename> inside the service's network.
If you wanted to test something easy, have your service create a random string, or use hostname on startup and return that so you can tell the different replicas when accessing. A easy example is to run one service using image elasticsearch:2 which will return JSON on port 9200 with a different random name per container.
I am setting up a Neo4j HA cluster according to the documentation (https://neo4j.com/docs/operations-manual/current/tutorial/highly-available-cluster) but Neo4j does not seem to be applying the HA configuration.
I can browse to the Neo4j browser and my database is active, but :play sysinfo shows 'High Availability' = 'Not enabled' and Cluster = 'No cluster'.
I also cannot telnet to the clustering port (5001):
ubuntu#ip-172-0-31-71:~$ telnet localhost 5001
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Connection closed by foreign host.
ubuntu#ip-172-0-31-71:~$
The HA configuration in Neo4j is:
# Unique server id for this Neo4j instance
# can not be negative id and must be unique
ha.server_id=2
# List of other known instances in this cluster
ha.initial_hosts=neo4j-prod-1:5001,neo4j-prod-2:5001,neo4j-prod-3:5001
# HA - High Availability
# SINGLE - Single mode, default.
dbms.mode=HA
I'm starting the container with the following command which routes communication on the clustering port (5001) to the container:
/usr/bin/docker run \
--publish=7474:7474 --publish=7687:7687 --publish=5001:5001 \
--volume=/var/lib/neo4j/data:/data \
--volume=/var/lib/neo4j/logs:/logs \
--volume=/var/lib/neo4j/conf:/conf \
neo4j:3.0
It looks like Neo4j is not loading the HA configuration - where should I look next?
HA does not work in Neo4j Community Edition.
To enable HA, run Enterprise Edition with:
/usr/bin/docker run \
--publish=7474:7474 --publish=7687:7687 --publish=5001:5001 \
--volume=/var/lib/neo4j/data:/data \
--volume=/var/lib/neo4j/logs:/logs \
--volume=/var/lib/neo4j/conf:/conf \
neo4j:3.0-enterprise
I'm working on a RabbitMQ distributed POC and I'm stuck at the basics of clustering the nodes.
I'm trying to follow the rabbit's tutorial on clustering so this is my reference.
After installing erlang (R14B04) and rabbit (2.8.2-1) I've copied the .erlang.cookie file contents from one node to the other two.
I wasn't sure about how to get erlang to notice this change to I had to restart the machines themselves (pretty brute force but I don't know erlang at all).
In addtion I opened in iptables 4369 and 5 additional ports for communications and placed under
/usr/lib64/erlang/bin/sys.config the following config:
{kernel,[{inet_dist_listen_min, XX00},{inet_dist_listen_max,XX05}]}]
Then another restart (dumb I know) to verify erlang takes these into consideration but still when I run:
rabbitmqctl cluster rabbit#HostName1
I get:
Clustering node rabbit#HostName2 with [rabbit#HostName1] ...
Error: {no_running_cluster_nodes,[rabbit#HostName1],
[rabbit#HostName1]}
There is a chance my fiddling with the erlang.cookie or with the ports did not succeed but I don't know how to check them. I tried typing erl in the cmd and then erl_epmd:names() or other commands to get more information but I'm probably way off in erlang land.
Would truly appreciate any help
Update:
I tried pinging two erlang nodes manually and got pang back.
I did the following:
Connected to two nodes, stopped rabbitmq (wasn't sure if needed but to be sure), started erlang like so (erl -sname dilbert and erl -sname dilbert2) when the erlang command line started i ran node(). on each of them and got dilbert#HostName1 and dilbert2#HostName2 respectively. I then tried to run net_adm:ping('dilbert'). and net_adm:ping('dilbert#HostName1'). with the single quote and without them from both nodes (changed names of course) and got on all 8 cases pang.
When I ran nodes(). on one of the machines I got back an empty array.
I've also tried to allow all traffic in the firewall (script) and then try to run the above commands (don't worry they're back on now) and still got back pang.
Update2:
For some reason I had cookies mismatch which I needed to resolve (thanks #kjw0188 for the suggestion [I ran erlang:get_cookie(). in the erlang command line]).
This did not help and I needed to stop iptables completely (not sure why but I'll figure it soon) and load the erlang node with -name dilbert#my-ip because my rackspace servers have no dns-name. This finally enabled me to get a pong and see the nodes see each other (nodes(). returns a non-empty array after the ping).
The problem I'm facing now is how to instruct RabbitMQ to use -name instead of -sname when starting erlang.
So I had multiple issues with connecting my two RabbitMQ nodes-
I'll add that my nodes are hosted on rackspace, and so don't have a default exposable hostname, and require iptables since there is no DMZ or built in security group concept like amazon.
Problems:
1. Cookie- Not sure how or why but I had multiple instances of .erlang.cookie (in /root, in my home directory and in /var/lib/rabbitmq/) I kept only the one in rabbitmq and verified all nodes have the same cookie.
2. IPTables- In order for the nodes to communicate I needed to open the epmd port and the range of ports for the actual communication inet_dist_listen_min inet_dist_listen_max.
/sbin/iptables -A INPUT -i eth1 -p tcp --dport ${epmd} -s ${otherNode} -j ACCEPT
/sbin/iptables -A INPUT -i eth1 -p tcp --dport ${inet_dist_listen_min}:${inet_dist_listen_max} -s ${otherNode} -j ACCEPT
empd is the usuall 4369 port and for the other range use whatever range you want.
${otherNode} is the ip of my other node.
I also needed to configure erlang through rabbitmq to use these ports (see config file at end)
3. HostName- Seeing as I don't have a hostname I needed to edit the rabbit scripts to use -name and not -sname (the first tells erlang to take the whole name, the latter stands for short name and thus appends an # symbol and the hostname).
This was accomplished by editing:
/usr/lib/rabbitmq/bin/rabbitmqctl
Added at the beginning the definition of the RABBITMQ_NODE_IP_ADDRESS property
DEFAULT_NODE_IP_ADDRESS=auto
DEFAULT_NODE_PORT=5672
[ "x" = "x$RABBITMQ_NODE_IP_ADDRESS" ] && RABBITMQ_NODE_IP_ADDRESS=${NODE_IP_ADDRESS}
[ "x" = "x$RABBITMQ_NODE_PORT" ] && RABBITMQ_NODE_PORT=${NODE_PORT}
[ "x" = "x$RABBITMQ_NODE_IP_ADDRESS" ] && [ "x" != "x$RABBITMQ_NODE_PORT" ] && RABBITMQ_NODE_IP_ADDRESS=${DEFAULT_NODE_IP_ADDRESS}
[ "x" != "x$RABBITMQ_NODE_IP_ADDRESS" ] && [ "x" = "x$RABBITMQ_NODE_PORT" ] && RABBITMQ_NODE_PORT=${DEFAULT_NODE_PORT}
and in the actual erl command I changed
-sname ${RABBITMQ_NODENAME} \ to
-name ${RABBITMQ_NODENAME}#${RABBITMQ_NODE_IP_ADDRESS}\.
This made rabbitmq listen only on the specified ip address (specified in the config file at the end) and load with that ip instead of the usuall hostname.
edited /usr/lib/rabbitmq/bin/rabbitmq-server
Changed the actual erl command from -sname ${RABBITMQ_NODENAME} \ to -name ${RABBITMQ_NODENAME}#${RABBITMQ_NODE_IP_ADDRESS}\
Added a rabbit conf (/etc/rabbitmq/rabbitmq-env.conf) file with-
#the ip address which rabbit should use, this is to limit rabbit to only use internal rackspace communication and not publicly accessible ports
NODE_IP_ADDRESS=myIpAdress
#had to change the nodename becaue otherwise rabbitmq used rabbit#Hostname and not only rabbit
NODENAME=myCompany
#This instructed rabbit to instruct erlang which ports it should use for its communications with other nodes
export SERVER_ERL_ARGS="$SERVER_ERL_ARGS -kernel inet_dist_listen_min somePort -kernel inet_dist_listen_max someOtherBiggerPort"
Some resources which helped me along the way:
RabbitMQ Clustering Guide
Clustering RabbitMQ servers for High Availability
rabbitmq-env.conf(5) manual page
Node communication by public IP address erlang mailing list (The middle post)
Configuring RabbitMQ Cluster on Cloud
Hope this will help anyone else.
EDIT:
Not sure how I was mistaken but it seemed my erlang-rabbit port instructions were not taken into consideration or were not enough. Ended up having to allow all communications between the two nodes...
One thing to really watch out for is whitespace of any kind in the erlang cookie file, especially line breaks AFTER the contents of the cookie. So long as both are identical, things are okay, but when one has a line break and the other doesn't, thing won't work.
Background: I was facing the same issue while setting up Rabbitmq cluster. I was using 2 docker containers running on my host-machine, which is equivalent to 2 separate nodes and I could not create a cluster of these two.
Solution: 1. Make sure you have same erlang cookie on all your cluster nodes, the default location is /var/lib/rabbitmq/.erlang.cookie. This file is used for authentication, so make sure, you have it same on all the nodes. After changing the .erlang.cookie restart your rabbitmq service.
Make sure that nodes are accessible from one other, use ping or telnet to check the connection.
Check that /etc/hosts have correct entries, for example if rabbit2 wants to join cluster rabbit1, /etc/hosts of rabbit2 should contain.
172.68.1.6 rabbit1
172.68.1.7 rabbit2
Now stop service using $rabbitmqctl stop_app followed by $rabbitmqctl join_cluster rabbit#rabbit1, start your service by rabbitmqctl start_app and check $rabbitmqctl cluster_status to see weather you have joined the cluster or not.
I followed the rabbitmq official documentation to setup the cluster.
to change RabbitMQ sname/name behaviour you can edit the scripts:
rabbitmq-multi
rabbitmq-server
rabbitmqctl
Example
In script rabbitmqctl there is the following piece of code:
exec erl \
-pa "${RABBITMQ_HOME}/ebin" \
-noinput \
-hidden \
${RABBITMQ_CTL_ERL_ARGS} \
-sname rabbitmqctl$$ \
-s rabbit_control \
-nodename $RABBITMQ_NODENAME \
-extra "$#"
You have to change it in:
exec erl \
-pa "${RABBITMQ_HOME}/ebin" \
-noinput \
-hidden \
${RABBITMQ_CTL_ERL_ARGS} \
-name rabbitmqctl$$ \
-s rabbit_control \
-nodename $RABBITMQ_NODENAME \
-extra "$#"
http://pearlin.info/?p=1672
so you need to copy the cookie from the node you trying to connect
example :- rabbit#node1
rabbit#node2
go to rabbit#node1 and copy the cookie from cat /var/lib/rabbitmq/.erlang.cookie
go to rabbit#node2 remove the current cookie and paste the new one.
on same node
/usr/sbin/rabbitmqctl stop_app
/usr/sbin/rabbitmqctl reset
/usr/sbin/rabbitmqctl cluster rabbit#node1
should do it.
same documented here.
http://pearlin.info/?p=1672