Exception (403) Reason: "username or password not allowed" - docker

I am trying to access rabbitmq cluster where TLS is enabled. I have written sample go app which trying to connect to the rabbitmq server using set of client certificates and client keys.
I am facing error -
Error is - Exception (403) Reason: "username or password not allowed"
panic: Exception (403) Reason: "username or password not allowed"
My code snippet
package main
import (
"crypto/tls"
"crypto/x509"
"fmt"
"io/ioutil"
"log"
"github.com/streadway/amqp"
)
func main() {
fmt.Println("Go RabbitMQ Consumer Tutorial")
fmt.Println("Testing ClusterIP service connection over TLS")
cert, err := tls.LoadX509KeyPair("client.crt", "client.key")
if err != nil {
panic(err)
}
//Load CA cert.
caCert, err := ioutil.ReadFile("ca.crt") // The same you configured in your MQ server
if err != nil {
log.Fatal(err)
}
caCertPool := x509.NewCertPool()
caCertPool.AppendCertsFromPEM(caCert)
TlsConfig1 := &tls.Config{
Certificates: []tls.Certificate{cert}, // from tls.LoadX509KeyPair
RootCAs: caCertPool,
InsecureSkipVerify: true,
// ...other options are just the same as yours
}
conn, err := amqp.DialTLS("amqps://<username>:<password>#<rabbitmq-service-name>.<namespace>.svc.cluster.local:5671/", TlsConfig1)
if err != nil {
fmt.Println("Error is -", err)
panic(err)
}
fmt.Println("Connected to consumer successfully")
defer conn.Close()
ch, err := conn.Channel()
if err != nil {
fmt.Println(err)
}
defer ch.Close()
if err != nil {
fmt.Println(err)
}
msgs, err := ch.Consume(
"TestQueue",
"",
true,
false,
false,
false,
nil,
)
forever := make(chan bool)
go func() {
for d := range msgs {
fmt.Printf("Recieved Message: %s\n", d.Body)
}
}()
fmt.Println("Successfully Connected to our RabbitMQ Instance")
fmt.Println(" [*] - Waiting for messages")
<-forever
}
The code snippet is running as pod in the EKS cluster where my rabbitmq cluster is running.

After going through the rabbitmq TLS debugging doc- https://www.rabbitmq.com/access-control.html I figured out that I was giving wrong username and password.
NOTE: if any one who is also getting the same erorr - please confirm that you are using correct username and password.
you can check the username and password for the cluster by logging into the rabbitmq cluster pod.

This happens to me on version 11.6.0 but not on 10.3.9.

Related

Add Tracing to internal methods in Cloud Run

We would like to add tracing to methods used within services deployed on Cloud Run.
Tracing already provided Cloud Run requests:
Let's say we have the following gRPC method:
func (s *myServiceService) SyncTable(ctx context.Context, req *pb.SyncTableRequest) (*longrunning.Operation, error) {
//.... some stuff here...
// making a call to the internal method, which has a tracing span
err := dropRequestOnStorage(ctx, ...)
if err != nil {
return nil, err
}
return op, nil
}
Here is an example of an internal method to which we have added a Trace span and is called by the main gRPC method:
// dropRequestOnStorage loads the requests on the relevant bucket.
func dropRequestOnStorage(ctx context.Context, filename string, operationID string, req *pb.ExtractDataRequest) error {
// add tracing to this method.
ctx, span := otel.Tracer("").Start(ctx, "dropRequestOnStorage")
defer span.End()
// load json object to storage
reqByte, err := protojson.Marshal(req)
if err != nil {
fmt.Println(err)
}
wc := storageClient.Bucket("my-bucket-with-cool-stuff").Object(filename).NewWriter(ctx)
wc.ContentType = "application/json"
_, err = wc.Write(reqByte)
if err != nil {
fmt.Println(err)
}
wc.Close()
fmt.Println(filename)
return nil
}
Looking at Tracing for Google Cloud Run I see traces for the above method:
Despite passing the context from the main gRPC to the internal method, Tracing is not pulled through to the underlying internals. The traces generated by the internal methods does not 'receive' the main gRPC trace as a parent.
Is this because the default tracing provided by Cloud Run is done by the Cloud Run internals? And therefore not available to the context of the gRPC methods?
Tracing using gRPC Interceptors
The only way to get this to work was to add gRPC interceptors to create tracing spans for each gRPC method.
package main
import (
"context"
texporter "github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/trace"
"go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/propagation"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
"google.golang.org/grpc"
"log"
"net"
"os"
)
func init() {
// Pre-declare err to avoid shadowing.
var err error
// initialising tracing exporter
//exporter, err := stdout.NewExporter(stdout.WithPrettyPrint())
exporter, err := texporter.NewExporter(texporter.WithProjectID("alis-de"))
if err != nil {
log.Fatalf("texporter.NewExporter: %v", err)
}
tp := sdktrace.NewTracerProvider(
sdktrace.WithSampler(sdktrace.AlwaysSample()),
sdktrace.WithSyncer(exporter),
)
otel.SetTracerProvider(tp)
otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))
}
func main() {
log.Printf("starting server...")
port := os.Getenv("PORT")
if port == "" {
port = "8080"
log.Printf("Defaulting to port %s", port)
}
listener, err := net.Listen("tcp", ":"+port)
if err != nil {
log.Fatalf("net.Listen: %v", err)
}
// Attaching grpc interceptors to automatically enable tracing at gRCP methods
grpcServer := grpc.NewServer(
grpc.UnaryInterceptor(otelgrpc.UnaryServerInterceptor()),
grpc.StreamInterceptor(otelgrpc.StreamServerInterceptor()),
)
pb.RegisterOperationsServer(grpcServer, &operationsService{})
if err = grpcServer.Serve(listener); err != nil {
log.Fatal(err)
}
}
Tracing now pulls through to the Console:
However, looking at the Traces, there are now (unfortunately??) two trace entries:
The default trace provided by Cloud Run (with no child traces)
The new trace generated by the gRPC interceptors (with child traces reflecting the internally called methods)

Server in Docker container connection refused, should I add time.Sleep(100 * time.Millisecond) to my tests?

I am currently working on a server that is designed to be run in a Docker container.
Here is my setup method for my tests:
func TestMain(m *testing.M) {
schedulerName := "scheduler1"
IP, err := container.StartNewScheduler(schedulerName)
if err != nil {
log.Println("Could not create container.")
log.Fatal(err)
}
serverIP = IP
code := m.Run()
cleanupContainer(schedulerName)
os.Exit(code)
}
The line container.StartNewScheduler(schedulername) boots up a new docker container called "scheduler1" and tells it to run the server inside of it.
Next I run my tests with the container running in the background, right now I only have one test.
func TestNewScheduler(t *testing.T) {
testCodeInput := "THIS IS A TEST"
requestBody, err := json.Marshal(map[string]string{
"Code": fmt.Sprintf("print(\"%s\")", testCodeInput),
})
if err != nil {
t.Fatal(err)
}
url := fmt.Sprintf("http://%s:%d/execute/python", serverIP, 3000)
contentType := "application/json"
body := bytes.NewBuffer(requestBody)
response := post(url, contentType, body, t)
actual := parseOutput(response.Body, t)
response.Body.Close()
expected := fmt.Sprintf("{\"Stdout\":\"%s\\n\"}", testCodeInput)
if actual != expected {
t.Fatalf("Expected %s, but got %s", expected, actual)
}
}
The problem that I am running into is sometimes I get a connection refused and sometimes I don't.
server_container_test.go:51: Post http://172.20.0.2:3000/execute/python: dial tcp 172.20.0.2:3000: connect: connection refused
I noticed that whenever I try and debug the issue everything seems to work fine. My running theory is because when I step through my code the container has more time to start up and get the server running inside it.
In order to test my Hypothesis I added a second post call in my post method with a timer set before I call it.
func post(url string, contentType string, body io.Reader, t *testing.T) *http.Response {
t.Helper()
response, err := http.Post(url, contentType, body)
if err != nil {
//There is an error where the container takes a second to boot up and so
//the scheduler isn't running when the first request is sent, so we try
//one more time here and check again.
time.Sleep(100 * time.Millisecond) <--- Right here
response, err = http.Post(url, contentType, body)
if err != nil {
t.Fatal(err)
}
}
return response
}
Does anyone else have any guesses as to what could be causing me this issue?
If my hypothesis is correct is this the best way to fix this? Is it a bad idea to add a time.Sleep to your tests?
Thanks!
Ok so after some more thought I changed up my source code, please let me know if you think this is a good solution to my problem. I am still learning Go and HTTP servers so any input is appreciated.
Here is my fix/idea:
Previously once the container was created I just returned it's IP address and forgot about it.
Now I create a go routine that repeatedly tries to send a POST request to the server. If it doesn't fail then I send true through a channel and close the function.
IP := info.NetworkSettings.Networks[networkName].IPAddress
works := make(chan bool)
ctx, canelRoutine := context.WithCancel(context.Background())
defer canelRoutine()
go func(ctx context.Context) {
requestBody, _ := json.Marshal(map[string]string{
"Code": "print()",
})
select {
case <-ctx.Done():
return
default:
for {
_, err := http.Post(
fmt.Sprintf("http://%s:%d/execute/python", IP, 3000),
"application/json",
bytes.NewBuffer(requestBody),
)
if err == nil {
works <- true
return
}
}
}
}(ctx)
After sending the goroutine off I create a timer and and wait for either the timer to return or the goroutine.
timer := time.After(500 * time.Millisecond)
select {
case <-works:
return IP, nil
case <-timer:
return IP, &UnreachableContainerError{name: schedulerName}
}
The upside to this solution is I have now introduced an UnreachableContainerError which allows me to be more specific about my error messages and it can be checked on the receiving side. I also send the IP address back either way just in case the client needs it for some other reason.
Here is the full StartNewScheduler method in case you wanted to see it.
//StartNewScheduler starts a new scheduler with the given options.
//returns the IP address for the given scheduler.
func StartNewScheduler(schedulerName string) (string, error) {
///Defaults
dockerfile := "Dockerfile_standard"
networkName := "scheduler-cluster"
imageID := "lkelly93/scheduler_image:latest"
cli, err := client.NewEnvClient()
if err != nil {
return "", err
}
err = createDefaultImageIfNeeded(
cli,
imageID,
dockerfile)
if err != nil {
return "", err
}
err = createSchedulerClusterNetworkIfNeeded(cli, networkName)
if err != nil {
return "", err
}
ctx := context.Background()
resp, err := cli.ContainerCreate(
ctx,
&container.Config{Image: imageID},
&container.HostConfig{
NetworkMode: container.NetworkMode(networkName),
Privileged: true,
},
nil,
schedulerName,
)
if err != nil {
return "", err
}
err = cli.ContainerStart(ctx, resp.ID, types.ContainerStartOptions{})
if err != nil {
return "", err
}
//Get container IP
info, err := cli.ContainerInspect(ctx, resp.ID)
if err != nil {
return "", err
}
IP := info.NetworkSettings.Networks[networkName].IPAddress
works := make(chan bool)
ctx, canelRoutine := context.WithCancel(context.Background())
defer canelRoutine()
go func(ctx context.Context) {
requestBody, _ := json.Marshal(map[string]string{
"Code": "print()",
})
select {
case <-ctx.Done():
return
default:
for {
_, err := http.Post(
fmt.Sprintf("http://%s:%d/execute/python", IP, 3000),
"application/json",
bytes.NewBuffer(requestBody),
)
if err == nil {
works <- true
return
}
}
}
}(ctx)
timer := time.After(500 * time.Millisecond)
select {
case <-works:
return IP, nil
case <-timer:
return IP, &UnreachableContainerError{name: schedulerName}
}
}
You could run the test code inside of a container, started via docker compose with the option depends_on.

http.Post() from a gRPC server to an http server returns EOF error on a docker-compose setup

I have a gRPC server (server) written in Go that a Python gRPC client (client) talks to. The server occasionally sends http post requests to a Go based http server (sigsvc). All of these instances run as docker instances spun up through docker-compose sharing the same docker network.
This is the section of code on server that creates and sends the http request:
b := new(bytes.Buffer)
txbytes, err := json.Marshal(tx)
if err != nil {
log.WithError(err).Error("failed to marshal transaction")
return nil, err
}
b.Write(txbytes)
resp, err := http.Post(sigsvc.signerURL, "application/json; charset=utf-8", b)
if err != nil {
log.WithError(err).Errorf("error signing transaction with signer %s", sigsvc.signerURL)
return nil, err
}
defer resp.Body.Close()
var signedTx types.Transaction
err = json.NewDecoder(resp.Body).Decode(&signedTx)
if err != nil {
log.WithError(err).Error("couldn't decode signed transaction")
return nil, err
}
sigsvc.signerURL maps to something like http://signer:6666/sign which is the endpoint on the http signer service that handles the request.
signer refers to the service name listed on a docker-compose.yml specification.
This is how the handler looks like on sigsvc:
func (sv *SignerSv) handleSignTx() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
log.Info("request received to sign transaction")
dump, err := httputil.DumpRequest(r, true)
if err != nil {
http.Error(w, fmt.Sprint(err), http.StatusInternalServerError)
}
log.Debugf("%q", dump)
if r.Body == nil {
log.Error("request body missing")
http.Error(w, "Please send a request body", 400)
return
}
log.Debugf("request body: %v", r.Body)
var tx types.Transaction
err = json.NewDecoder(r.Body).Decode(&tx)
if err != nil {
log.WithError(err).Error("failed to unmarshal transaction")
http.Error(w, err.Error(), 400)
return
}
log.WithFields(log.Fields{
"txhash": tx.Hash().Hex(),
"nonce": tx.Nonce(),
"to": tx.To().Hex(),
"data": tx.Data(),
"gasLimit": tx.Gas(),
"gasPrice": tx.GasPrice(),
"value": tx.Value(),
}).Debug("Decoded transaction from request body")
Both the request and request body are dumped successfully by the debug logs. However, apparently the line decoding the request body to the transaction type is never executed, since no error or decoded transaction logs are logged.
On server, I keep getting the following error:
error="Post http://signer:6666/sign: EOF"
This is how the request is logged on sigsvc:
msg="\"POST /sign HTTP/1.1\\r\\nHost: signer:6666\\r\\nConnection: close\\r\\nAccept-Encoding: gzip\\r\\nConnection: close\\r\\nContent-Length: 10708\\r\\nUser-Agent: Go-http-client/1.1\\r\\n\\r\\n{\\\"nonce\\\":\\\"0x0\\\",\\\"gasPrice\\\":\\\"0x2540be400\\\",\\\"gas\\\":\\\"0x15c285\\\",\\\"to\\\":null,\\\"value\\\":\\\"0x0\\\",\\\"input\\\":\\\"0x6080604055",\\\"v\\\":\\\"0x0\\\",\\\"r\\\":\\\"0x0\\\",\\\"s\\\":\\\"0x0\\\",\\\"hash\\\":\\\"0xab55920fb3d490fc55ccd76a29dfb380f4f8a9e5d0bda4155a3b114fca26da0a\\\"}\"
I have tried reproducing this error on similar but simplified docker setups, but I have failed at that.
I'm trying to understand the following:
If there is anything wrong with this code that is being exposed due
to a particular setup on docker?
Or, do I need to look at some docker setup specifics to debug the instances.
The problem was in the way the http handler code was logging the to field in this logrus call.
log.WithFields(log.Fields{
"txhash": tx.Hash().Hex(),
"nonce": tx.Nonce(),
"to": tx.To().Hex(),
"data": tx.Data(),
"gasLimit": tx.Gas(),
"gasPrice": tx.GasPrice(),
"value": tx.Value(),
}).Debug("Decoded transaction from request body")
Under specific circumstances, the tx.To() call returns nil, which implies calling tx.To().Hex() would lead to an error on account of trying to make a method call on a nil pointer. On the face of it, one would expect the log.WithFields() call to error out or panic, but instead the handler silently closes connection with the client side getting an EOF response.

Redigo fetches old value from Redis Docker container

I'm developing a golang web project with Redigo to connect to a redis docker container.
While the golang web application is in running state, I've changed the value of a redis key using SET MyFlag true in redis-cli. But this is not reflecting in my webapp. When MyFlag is fetched using flag, err := redis.Bool(conn.Do("GET", "MyFlag")), it gives the old value.
But, after the webapp is restarted, the same command fetches the new value.
conn is retrieved from a redis connection pool. This is the configuration for redis pool
redisPool = &redis.Pool{
MaxIdle: 5,
IdleTimeout: 240 * time.Second,
MaxActive: 10,
Wait: true,
Dial: func() (redis.Conn, error) {
c, err := redis.Dial("tcp", db.GetUrl())
if err != nil {
return nil, err
}
password := db.GetPassword()
if password != "" {
if _, err := c.Do("AUTH", password); err != nil {
_ = c.Close()
return nil, err
}
}
return c, nil
},
}
Is this the issue of caching in Redis/Redigo/Docker ?

Grab output from container in Docker SDK

I'm trying to run a container using Docker SDK for golang and I can't get the output from the container. I'm using the following code for that that actually runs the container, but doesn't sends back stderr and stdout of the application. Can you advice what I'm doing wrong?
type dckr struct {
cli *client.Client
username string
password string
addr string
ctx context.Context
}
func (d *dckr) Run(containername string, image string, command []string, bind []string, stdout io.Writer, stderr io.Writer) error {
log.Printf("[Create] %s -> %s \n", image, containername)
res, err := d.cli.ContainerCreate(
d.ctx,
&container.Config{
User: "root",
AttachStdout: true,
AttachStderr: true,
Image: image,
Cmd: command,
},
&container.HostConfig{
AutoRemove: true,
Binds: bind,
},
&network.NetworkingConfig{},
containername,
)
if err != nil {
log.Println("[Create] Failed. %s", err)
return err
}
defer d.cli.ContainerRemove(d.ctx, res.ID, types.ContainerRemoveOptions{Force: true})
log.Printf("[Create] id: %s \n", res.ID)
for wrn := range res.Warnings {
log.Printf("[Create] %s \n", wrn)
}
rsp, err := d.cli.ContainerAttach(d.ctx, containername, types.ContainerAttachOptions{
Stream: false,
Stdout: true,
Stderr: true,
Logs: true,
})
if err != nil {
log.Printf("[Attach] Fail. %s \n", err)
return err
}
log.Printf("[Attach] %s", res.ID)
defer rsp.Close()
err = d.cli.ContainerStart(d.ctx, res.ID, types.ContainerStartOptions{})
if err != nil {
log.Printf("[Run] Fail. %s \n", err)
return err
}
_, err = stdcopy.StdCopy(stdout, stderr, rsp.Reader)
return err
}
The question was asked in 2017 and I'm answering it in 2022. I understand the APIs might have changed, but I have landed on a similar boat.
Let's not talk about how to start a container as you seem to have already done that. Here is my code to fetch the logs from a given container:
// GetLogs return logs from the container io.ReadCloser. It's the caller duty
// duty to do a stdcopy.StdCopy. Any other method might render unknown
// unicode character as log output has both stdout and stderr. That starting
// has info if that line is stderr or stdout.
func GetLogs(ctx context.Context, cli *client.Client, contName string) (logOutput io.ReadCloser) {
options := types.ContainerLogsOptions{ShowStdout: true}
out, err := cli.ContainerLogs(ctx, contName, options)
if err != nil {
panic(err)
}
return out
}
You can call this GetLogs from another routine. I am saving both of the stream in specific files. But you may want to use os.Stdout and os.Stderr if you just want to see them on your terminal:
func main() {
stdoutLog, _ := os.Create("yourContainerName.log")
defer stdoutLog.Close()
stderrLog, _ := os.Create("yourContainerName.err")
defer stderrLog.Close()
var stdout bytes.Buffer
var stderr bytes.Buffer
containerLog := docker.GetLogs(ctx, dc, "yourContainerName")
stdcopy.StdCopy(&stdout, &stderr, containerLog)
stdoutLog.Write(stdout.Bytes())
stderrLog.Write(stderr.Bytes())
}
Let me know the second part if you still have confusion. I'm happy to help as I had a similar problem.

Resources