I'm trying to build an automation tool to pull docker images using golang.
Here is the simplified version of the script:
package dockermgr
import (
"context"
"github.com/docker/docker/api/types"
"github.com/docker/docker/client"
)
func getClient() (*client.Client, error) {
return client.NewClientWithOpts(client.FromEnv)
}
func ImagePull(image models.DockerImage) error {
// TODO: Is docker daemon running?
cli, err := getClient()
if err != nil {
panic(err)
}
defer cli.Close()
reader, err := cli.ImagePull(
context.Background(),
image.GetNameAndTag(),
types.ImagePullOptions{})
if err != nil {
panic(err)
}
defer reader.Close()
return err
}
It apparently pulls the image, but when I turn back to terminal and execute a docker image list, I can't see the specified image pulled & saved into the local registry.
I wondered if it invokes it's own docker daemon instead of using the local one. But it does not seem to be the case. If I try to pause the local docker daemon, it wakes back up when I execute this.
So what am I missing?
You can add this to print the events produced by the Pull request. There might be an error when pulling the image. For example if you're on M1 and trying to pull MySQL it fails but without this you won' see the error.
nc ImagePull(imageNameAndTag string) error {
// TODO: Is docker daemon running?
cli, err := getClient()
if err != nil {
panic(err)
}
defer cli.Close()
events, err := cli.ImagePull(
context.Background(),
imageNameAndTag,
types.ImagePullOptions{})
if err != nil {
panic(err)
}
d := json.NewDecoder(events)
defer events.Close()
var event interface{}
for {
err = d.Decode(&event)
if err != nil {
// handle EOF error
break
}
fmt.Printf("%+v\n", event)
}
return err
}
Related
I have a docker app which has two containers. One is MySql and the other is some logic code which I have created a custom image of using a Dockerfile. For end to end testing, I wish to store some values in the database and then run the logic code image (Logic in golang). This is the docker-compose file I have currently:
version: '3'
networks:
docker-network:
driver: bridge
services:
database:
image: mysql
env_file:
- ./src/logic/environment-variables.env
ports:
- 3306:3306
healthcheck:
test: "mysql -uroot -p$$MYSQL_ROOT_PASSWORD $$MYSQL_DATABASE -e 'select 1'"
timeout: 20s
retries: 10
network:
docker-network
logic:
container_name: main-logic
build: ./src/logic/.
depends_on:
database:
condition: service_healthy
network:
docker-network
I cannot run this app as a whole as that would run the main as soon as the db is running. Instead, I want to start the db, store some values in it, then run the logic image. How can I do this in a test method?
Approaches considered:
Start up the mysql image separately from the test method and then store values in it.Then start the logic image and check the database for results. Is there a better way or a framework to use for this?
What you need here are database migrations. That should work as follows :
Start DB instance before starting the service.
Connect the service to DB.
Run migrations on DB.
Continue with the service execution.
Consider this : https://github.com/golang-migrate/migrate
You can do exactly what you say in the question: start the database, manually load the seed data, and start the rest of the application. Since your database has published ports: you can connect to it directly from the host without doing anything special.
docker-compose up -d database
mysql -h 127.0.0.1 < seed_data.sql
docker-compose up -d
#advayrajhansa's answer suggests using a database-migration system. If this was built into your image, you could docker-compose run logic migrate ... as the middle step. This runs an alternate command on the container you otherwise have defined in the docker-compose.yml file.
For your approach:
Start MySQL image.
Upload data to the database.
Start the logic image.
Check the database for results.
You can:
Use Makefile
with a sh script inside, that will execute all steps one by one.
Makefile:
start_test:
docker-compose run -d database
# put here your data uploading script
docker-compose run -d logic
# put here your data database checking script
Then execute
$make start_test # execute all steps
Use Testcontainers-Go
Testcontainers GitHub
Testcontainers-Go is a Go package that makes it simple to create and clean up container-based dependencies for automated integration/smoke tests.
It allows you to execute all steps in a go test method.
For your case you will have something like this:
just a draft code to catch up the idea:
package main
import (
"context"
"database/sql"
"fmt"
"github.com/pkg/errors"
"github.com/testcontainers/testcontainers-go"
"github.com/testcontainers/testcontainers-go/wait"
"log"
"testing"
)
var db *sql.DB
func TestIntegration(t *testing.T) {
if testing.Short() {
t.Skip("skipping integration test")
}
err := setupMySql()
if err != nil {
t.Errorf("Test failed with error: %s", err)
}
err = setupData()
if err != nil {
t.Errorf("Test failed with error: %s", err)
}
err = setupLogic()
if err != nil {
t.Errorf("Test failed with error: %s", err)
}
err = checkResult()
if err != nil {
t.Errorf("Test failed with error: %s", err)
}
}
func setupMySql() error {
ctx := context.Background()
req := testcontainers.ContainerRequest{
Image: "mysql:latest",
ExposedPorts: []string{"3306/tcp", "33060/tcp"},
Env: map[string]string{
"MYSQL_ROOT_PASSWORD": "secret",
},
WaitingFor: wait.ForLog("port: 3306 MySQL Community Server - GPL"),
}
mysqlC, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{
ContainerRequest: req,
Started: true,
})
defer func() {
err := mysqlC.Terminate(ctx)
if err != nil {
log.Fatal(err)
}
}()
if err != nil {
return errors.Wrap(err, "Failed to run test container")
}
host, err := mysqlC.Host(ctx)
p, err := mysqlC.MappedPort(ctx, "3306/tcp")
port := p.Int()
connectionString := fmt.Sprintf("%s:%s#tcp(%s:%d)/%s?tls=skip-verify",
"root", "secret", host, port, "database")
db, err = sql.Open("mysql", connectionString)
defer func(db *sql.DB) {
err := db.Close()
if err != nil {
log.Fatal(err)
}
}(db)
if err != nil {
return errors.Wrap(err, "Failed to connect to db")
}
return nil
}
func setupData() error {
// db.Query(), your code with uploading data
return nil
}
func setupLogic() error {
// run your logic container
return nil
}
func checkResult() error {
// db.Query(), your code with checking result
return nil
}
Use Dockertest
Dockertest helps you boot up ephermal docker images for your Go tests with minimal work.
Same as Testcontainers-Go,
just a draft code to catch up the idea:
package main
import (
"database/sql"
"fmt"
"github.com/ory/dockertest/v3"
"github.com/pkg/errors"
"testing"
)
var db *sql.DB
func TestIntegration(t *testing.T) {
if testing.Short() {
t.Skip("skipping integration test")
}
err := setupMySql()
if err != nil {
t.Errorf("Test failed with error: %s", err)
}
err = setupData()
if err != nil {
t.Errorf("Test failed with error: %s", err)
}
err = setupLogic()
if err != nil {
t.Errorf("Test failed with error: %s", err)
}
err = checkResult()
if err != nil {
t.Errorf("Test failed with error: %s", err)
}
}
func setupMySql() error {
// uses a sensible default on windows (tcp/http) and linux/osx (socket)
pool, err := dockertest.NewPool("")
if err != nil {
return errors.Wrap(err, "Could not connect to docker")
}
// pulls an image, creates a container based on it and runs it
resource, err := pool.Run("mysql", "5.7", []string{"MYSQL_ROOT_PASSWORD=secret"})
if err != nil {
return errors.Wrap(err, "Could not start resource")
}
// exponential backoff-retry, because the application in the container might not be ready to accept connections yet
if err := pool.Retry(func() error {
var err error
db, err = sql.Open("mysql", fmt.Sprintf("root:secret#(localhost:%s)/mysql", resource.GetPort("3306/tcp")))
if err != nil {
return err
}
return db.Ping()
}); err != nil {
return errors.Wrap(err, "Could not connect to database")
}
if err := pool.Purge(resource); err != nil {
return errors.Wrap(err, "Could not purge resource")
}
return nil
}
func setupData() error {
// db.Query(), your code with uploading data
return nil
}
func setupLogic() error {
// run your logic container
return nil
}
func checkResult() error {
// db.Query(), your code with checking result
return nil
}
I need to access a private docker registry using the Go SDK. I found "package registry".
I see it has "DefaultSession" object. I can connect to the private registry, but I can't investigate it using the DefaultSession.
Secondly, the registry package contains the Session struct. They wrote it's for the v1 protocol only. Ok, I connect the private repository using the Session:
c := http.Client{}
indexInfo, err := registry.ParseSearchIndexInfo("repo")
if err != nil {
log.Error(err)
return
}
endpoint, err := registry.NewV1Endpoint(indexInfo, "", nil)
if err != nil {
log.Error(err)
return
}
session, err := registry.NewSession(&c, &authConfig, endpoint)
if err != nil {
log.Error(err)
return
}
n, err := reference.ParseNamed("docker.io/repo/image")
if err != nil {
log.Error(err)
return
}
rep, err := session.GetRepositoryData(n)
if err != nil {
log.Error(err)
return
}
But GetRepositoryData returns zero images, but they are in the repository. Why?
Is it right way I do access to a remote repository? Is there a v2 SDK for Go?
I have a need to programmatically (using golang) login to gcr.io docker registry using this package library https://godoc.org/github.com/docker/docker/client
I have tried using it, i can successfully login but upon pushing an image to my gcr.io project registry, it said
{"errorDetail":{"message":"unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication"},"error":"unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication"}
My code looks like this
package client
import (
"context"
"fmt"
"io"
"os"
"github.com/docker/docker/api/types"
dockerClient "github.com/docker/docker/client"
)
type Service struct{
DockerClient *dockerClient.Client
}
type CopyImageOptions struct {
DestRegistryAuth string
}
type DockerImageService interface {
CopyImage(ctx context.Context, source, dest string, option CopyImageOptions)
}
// NewDockerClient returns a client
func NewDockerClient() *Service {
cli, err := dockerClient.NewEnvClient()
if err != nil {
panic(err)
}
return &Service{DockerClient: cli}
}
func (s *Service) CopyImage(ctx context.Context, source, dest string, option CopyImageOptions) error {
rc, err := s.DockerClient.ImagePull(ctx, source, types.ImagePullOptions{})
if err != nil{
return fmt.Errorf("error when pulling source image. err: %v", err)
}
defer rc.Close()
io.Copy(os.Stdout, rc)
destClient := NewDockerClient()
if option.DestRegistryAuth != "" {
//current use case we can assume that the dest is on asia.gcr.io
status, err := destClient.DockerClient.RegistryLogin(ctx, types.AuthConfig{
Username: "oauth2accesstoken",
Password: option.DestRegistryAuth,
ServerAddress: "asia.gcr.io",
})
if err != nil{
return fmt.Errorf("error when login to destination image registry. err: %v", err)
}
fmt.Println(status)
}
err = destClient.DockerClient.ImageTag(ctx, source, dest)
if err != nil {
return fmt.Errorf("error when tagging image. err: %v", err)
}
rc, err = destClient.DockerClient.ImagePush(ctx, dest, types.ImagePushOptions{
RegistryAuth: option.DestRegistryAuth,
})
if err != nil{
return fmt.Errorf("error when pushing image to destionation. err: %v", err)
}
defer rc.Close()
io.Copy(os.Stdout, rc)
return nil
}
You may take a look at the CopyImage method, where the option.DestRegistryAuth is assigned with the output gcloud auth print-access-token. The username is set to "oauth2accesstoken" because I followed this instruction: https://cloud.google.com/container-registry/docs/advanced-authentication
As for the source parameter, it is assumed it's from public registry like docker.io/library/alpine:3.10, so we can pull it without having configuring any auth token. However for the dest parameter, currently it is an image in my private registry such as: asia.gcr.io/<gcp-project-id>/alpine:3.10
Also, the gcloud auth print-access-token is called after I did gcloud auth loginand I already had full permission to access my private asia.gcr.io registry (assigned on bucket level).
Now the weird thing is I can successfully push it using docker push command, right after do docker login described in here https://cloud.google.com/container-registry/docs/advanced-authentication .
Any advice?
Okay I just found out what the mistake is on my code above. I realized this after looking at example code on pulling image from private registry here: https://docs.docker.com/develop/sdk/examples/#pull-an-image-with-authentication
As it turns out, the RegistryAuth arg in types.ImagePush options expect a base64 encoding string.
So with this code, I can successfully push local image to my private registry.
authConfig := types.AuthConfig{
Username: "oauth2accesstoken",
Password: option.DestRegistryAuth,
}
encodedJSON, err := json.Marshal(authConfig)
if err != nil {
return fmt.Errorf("error when encoding authConfig. err: %v", err)
}
authStr := base64.URLEncoding.EncodeToString(encodedJSON)
rc, err = destClient.DockerClient.ImagePush(ctx, dest, types.ImagePushOptions{
RegistryAuth: authStr,
})
I can tweet, and upload media, but I cannot figure out how to tweet with the media using anaconda("github.com/ChimeraCoder/anaconda"). The media_id in the example was from a sucessfull media upload call.
mediaResponse, err := api.UploadMedia("R0lGODlhEAALALMMAOXp8a2503CHtOrt9L3G2+Dl7vL0+J6sy4yew1Jvp/T2+e/y9v///wAAAAAAAAAAACH/C05FVFNDQVBFMi4wAwEAAAAh+QQFCwAMACwAAAAAEAALAAAEK5DJSau91KxlpObepinKIi2kyaAlq7pnCq9p3NZ0aW/47H4dBjAEwhiPlAgAIfkECQsADAAsAAAAAAQACwAABA9QpCQRmhbflPnu4HdJVAQAIfkECQsADAAsAAAAABAACwAABDKQySlSEnOGc4JMCJJk0kEQxxeOpImqIsm4KQPG7VnfbEbDvcnPtpINebJNByiTVS6yCAAh+QQJCwAMACwAAAAAEAALAAAEPpDJSaVISVQWzglSgiAJUBSAdBDEEY5JMQyFyrqMSMq03b67WY2x+uVgvGERp4sJfUyYCQUFJjadj3WzuWQiACH5BAkLAAwALAAAAAAQAAsAAAQ9kMlJq73hnGDWMhJQFIB0EMSxKMoiFcNQmKjKugws0+navrEZ49S7AXfDmg+nExIPnU9oVEqmLpXMBouNAAAh+QQFCwAMACwAAAAAEAALAAAEM5DJSau91KxlpOYSUBTAoiiLZKJSMQzFmjJy+8bnXDMuvO89HIuWs8E+HQYyNAJgntBKBAAh+QQFFAAMACwMAAIABAAHAAAEDNCsJZWaFt+V+ZVUBAA7")
if err != nil {
fmt.Println(err)
}
//v := url.Values{}
//v.Set("media_ids", string(mediaResponse.MediaID))
fmt.Println(mediaResponse)
tweet := `
"media_ids": 612877656984416256,
"status": "hello"
`
result, err := api.PostTweet(tweet, nil)
if err != nil {
fmt.Println(err)
} else {
fmt.Println(result)
}
Can someone assist in telling me how to parse the json or call the PostTweet with the media id? I've also tried adding the media to url.Values without sucess.
Thanks everyone. I see that the json was invalid but the issue was an error passing the media_ids parameter. The response was: "errors":[{"code":44,"message":"media_ids parameter is invalid."}] which i though erroring out on the formatting but it had to do with not converting the media_ids type int64 to a string correctly. Here is the fixed code:
data, err := ioutil.ReadFile(fileName)
if err != nil {
fmt.Println(err)
}
mediaResponse, err := api.UploadMedia(base64.StdEncoding.EncodeToString(data))
if err != nil {
fmt.Println(err)
}
v := url.Values{}
v.Set("media_ids", strconv.FormatInt(mediaResponse.MediaID, 10))
result, err := api.PostTweet(posttitle, v)
if err != nil {
fmt.Println(err)
} else {
fmt.Println(result)
}
This is not valid json:
tweet := `
"media_ids": 612877656984416256,
"status": "hello"
`
Try using this to generate your json:
type Tweet struct {
MediaIds uint64 `json:"media_ids"`
Status string `json:"status"`
}
tweet := Tweet{612877656984416256, "hello"}
b, err := json.Marshal(tweet)
This results in :
{"media_ids":612877656984416256,"status":"hello"}
This has a few benefits over using a raw string.
It is more go centric. The struct can be passed around with values set and read with proper type checking caught at compile time.
The generated json string is more likely to be semantically correct. e.g. Go will also escape certain characters to help ensure they will be parsed properly by the receiver.
This is related to this other question. I'm fetching a URL through a proxy using this simple code:
package main
import (
"fmt"
"io/ioutil"
"net/http"
"net/url"
)
func main() {
proxyUrl, err := url.Parse("87.236.233.92:8080")
httpClient := &http.Client { Transport: &http.Transport { Proxy: http.ProxyURL(proxyUrl) } }
response, err := httpClient.Get("http://stackoverflow.com")
if err != nil {
fmt.Println(err.Error())
} else {
body, _ := ioutil.ReadAll(response.Body)
fmt.Println("OK: ", len(body))
}
}
If I run this code, I am getting this error:
Get http://stackoverflow.com: http: error connecting to proxy 87.236.233.92:8080: GetServByName: The requested name is valid, but no data of the requested type was found.
I know that the proxy address is valid and if I fetch the URL through the proxy by other means it work. Any idea why I'm getting this error?
Specify your proxy with http:// in and it should work, eg
proxyUrl, err := url.Parse("http://87.236.233.92:8080")
if err != nil {
fmt.Println("Bad proxy URL", err)
return
}