Why can I not run a Kafka connector? - docker

Background
Firstly - a bit of background - I am trying to learn a bit more about Kafka and Kafka connect. In that vein I'm following along to an early release book 'Kafka Connect' by Mickael Maison and Kate Stanley.
Run Connectors
Very early on (Chapter 2 - components in a connect data pipeline) they give an example of 'How do you run connectors'. Note that the authors are not using Confluent. Here in the early stages, we are advised to create a file named sink-config.json and then create a topic called topic-to-export with the following line of code:
bin/kafka-topics.sh --bootstrap-server localhost:9092 \
--create --replication-factor 1 --partitions 1 --topic topic-to-export
We are then instructed to "use the Connect REST API to start the connector with the configuration you created"
$ curl -X PUT -H "Content-Type: application/json" \ http://localhost:8083/connectors/file-sink/config --data "#sink-config.json"
The Error
However, when I run this command it brings up the following error:
{"error_code":500,"message":"Cannot deserialize value of type `java.lang.String` from Object value (token `JsonToken.START_OBJECT`)\n at [Source: (org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream); line: 1, column: 36] (through reference chain: java.util.LinkedHashMap[\"config\"])"}
Trying to fix the error
Keeping in mind that I'm still trying to learn Kafka and Kafka Connect I've done a pretty simple search which has brought me to a post on StackOverflow which seemed to suggest maybe this should have been a POST not a PUT. However, changing this to:
curl -d #sink-config.json -H "Content-Type: application/json" -X POST http://localhost:8083/connectors/file-sink/config
simply brings up another error:
{"error_code":405,"message":"HTTP 405 Method Not Allowed"}
I'm really not sure where to go from here as this 'seems' to be the way that you should be able to get a connector to run. For example, this intro to connectors by Baeldung also seems to specify this way of doing things.
Does anyone have any ideas what is going on? I'm not sure where to start...

First, thanks for taking a look at the early access version of our book.
You found a mistake in this example!
To start a connector, the recommended way is to use the PUT /connectors/file-sink/config endpoint, however the example JSON we provided is not correct.
The JSON file should be something like:
{
"name": "file-sink",
"connector.class": "org.apache.kafka.connect.file.FileStreamSinkConnector",
"tasks.max": 1,
"topics": "topic-to-export",
"file": "/tmp/sink.out",
"value.converter": "org.apache.kafka.connect.storage.StringConverter"
}
The mistake comes because there's another endpoint that can be used to start connectors, POST /connectors, and the JSON we provided is for that endpoint.
We recommend you use PUT /connectors/file-sink/config as the same endpoint can also be used to reconfigure connectors. In addition, the same JSON file can also be used with the PUT /connector-plugins/{connector-type}/config/validate endpoint.
Thanks again for spotting the mistake and reporting it, we'll fix the example in the coming weeks. We'll also reply to your emails about the other questions shortly.

Related

ActiveMQ Artemis: Obtain list of acceptors via JMX

How can I retrieve the list of configured acceptors in ActiveMQ Artemis via Jolokia/JMX (and curl)? I need to reload the acceptors after a TLS certificate update but looks like passing the acceptor name is mandatory. Unfortunately, I cannot just pass a static name because we use different acceptors, all using TLS – and don’t want to change the reloading code just because the acceptor config changed.
curl -s -f -u username:password -H 'Origin: localhost' 'http://127.0.0.1:8161/console/jolokia/read/org.apache.activemq.artemis:broker="borker-primary-0"'
shows the connectors, but not the acceptors.
This question is related to a change introduced in v2.18.0, see question on TLS certificate reload.
There is a getConnectors method on the main ActiveMQServerControl MBean which is why Jolokia's read command returns those values. However, there is no corresponding getAcceptors method, but you can use Jolokia's list command to effectively get the same information. Use something like this:
curl -s -f -u username:password -H 'Origin: localhost' 'http://127.0.0.1:8161/console/jolokia/list/org.apache.activemq.artemis:broker="borker-primary-0"'
Then look through the results for component=acceptors and you'll be able to find all the acceptors with their respective names.
This is a bit of a hack but a necessary one at this point given the lack of a management method to get the acceptors. I've opened ARTEMIS-3601 and sent a PR to deal with this use-case so in future versions this won't be necessary. You'll just be able to invoke getAcceptors or inspect them from the output of Jolokia's read command.

Create repo on Bitbucket programmatically

I used to do
curl -k -X POST --user john#outlook.com:doe13 "https://api.bitbucket.org/1.0/repositories" -d "name=logoApp"
and success.
now I got : error
{"type": "error", "error": {"message": "Resource removed", "detail": "This API is no longer supported.\n\nFor information about its removal, please refer to the deprecation notice at: https://developer.atlassian.com/cloud/bitbucket/deprecation-notice-v1-apis/"}}
Does anyone know a know way to do this ?
There's a difference between a success from curl (OK:200) and an error from the service you're trying to use. The error, however, mentions that you're trying to use the Cloud Rest API version 1, which is deprecated effective 30 June 2018.
Read this for more information.
I don't use Bitbucket Server (a local option), and I think that has more features for this sort of thing.
For the public Bitbucket, you can still do it but it isn't documented.
The v1.0 API has been removed, and the new v2.0 API doesn't document a POST to a /repositories. Instead, you have to hit an endpoint that includes the repo that doesn't yet exist: /repositories/workspace/repo_slug
The JSON payload needs to know the project for the repo: look in the slug for a project that already exists. Fill in the user/team and repo name in the URL. And, you can make an application password so you aren't using your account password. This app password can limit the scope of what that access can do.
% curl -X POST --user 'user:app_pass' \
-H "Content-type: application/json" \
-d '{"project":{"key":"PROJ"}}' \
"https://api.bitbucket.org/2.0/repositories/USER/REPO"

Can't get HTTP PATCH to work on Google Cloud Run instance

I have running a webserver called Postgrest which generates a REST API on top of a postgres DB. I have this running in Google Cloud run, and have it working for the most part. The HTTP actions I need to take are POST, GET, DELETE and PATCH.
Everything works correctly except PATCH, which I use to update an existing value in the DB.
When I run the command from curl command prompt, no error is given, but it doesnt'w work.
https://postgrest-q5mmtshbma-uc.a.run.app/notes?noteid=eq.3 -X PATCH -H "Authorization: Bearer $TOKEN" -H "Conte nt-Type: application/json" -d '{"note" : "updated it!"}'
When I run this against same postgrest version running locally, everything works correctly, so it has me thinking there might be an issue with Google Cloud run and not allowing/accepting PATCH requests? Again, POST, DELETE, GET all work fine.
Anyone have any insight what might be happening here?
I ultimately found the issue with this was related to using RLS (Row level security) in the PostGres DB, and I had setup specific policy's for insert, update, delete, and select.
The "update" policy I believe was incorrectly setup, so the update failed, but both postgre DB and the postgrest WebServer did not provide an errors to this effect.
Ultimately, when I re-created the update policy on this table, the PATCH (update) command ran successfully.

How to list Docker images in a Codefresh private registry?

I'm currently using Codefresh's free private registry to store my images. I am using CircleCI for my Docker builds, so from there I use docker login so I can push and pull to the Codefresh registry. This works fine. However, Codefresh only lists images in its web app for which it did the building, so I cannot easily see a catalogue of them. I suspect this is by design, so that users stick with Codefresh for CI. However, if possible, I would like to list the images I have in the registry.
I assume that this registry is a basic v2 version of the standard Docker Registry. So, I have had a hunt around, and found this console utility to manage images. However, it does not seem to work for me. I don't know whether this is because registry management tools are still immature generally (web searches indicate they were added much later, and there are lot of folks for whom this simple task has become a substantial undertaking) or whether there is something unusual with Codefresh.
Here is what I am trying:
reg -d -r r.cfcr.io -u myusername -p cfaccesstoken
(The -d is for debug).
This results in:
2017/10/18 11:24:43 registry.ping url=https://r.cfcr.io/v2/
2017/10/18 11:24:44 registry.catalog url=https://r.cfcr.io/v2/_catalog
2017/10/18 11:25:53 registry.catalog url=https://r.cfcr.iohttps://r.cfcr.io/v2/_catalog?n=1000&last=davigsantos/davigsantos/codeflix
FATA[0075] Get https://r.cfcr.iohttps//r.cfcr.io/v2/_catalog?n=1000&last=davigsantos/davigsantos/codeflix: dial tcp: lookup r.cfcr.iohttps on 127.0.1.1:53: no such host
The penultimate line contains a container name I don't recognise - I hope it is a public one, and not one I should not be seeing!
The last line indicates some sort of fatal error, with all sorts of URL fragments mashed together in ways that definitely won't work.
I have also discovered that dropping the access token makes no difference; the output is exactly the same:
reg -d -r r.cfcr.io -u myusername
What else can I try? I am running Mint Linux and would be fine with swapping to another utility if necessary. Is there something wrong about the way I am issuing this command, or is Codefresh running a non-standard registry that is incompatible with standard API calls?
Update
It looks like Codefresh also has their own API, though it does not seem to be documented as far as I can tell. Running the get operation produces this error:
Failed to authenticate request because no token was provided
That's encouraging, so I will try to find out how to provide a token in a curl call; the Swagger web interface does not seem to permit it. However I am conscious that, if I can get the API working, it may not list my Docker images anyway, since they were not built by Codefresh.
Update 2
I have found some articles on the Codefresh blog that hint how to authenticate on the API, the format is a header thus:
--header "x-access-token: (token)"
However I get this error:
{"status":401,"code":"2401","name":"UNAUTHORIZED_ERROR","message":"Failed to authenticate request because token is not valid","context":{}}
The token I am using is the same one as I use for docker login, which works. I notice I was not specifying my username, so I am also adding this curl option:
-u (user):(token)
As you can see, I am close to trying random things now, as there does not seem to be official help online.
Update 3
Prompted by a comment below, it seems that Docker maintains an access token after login is used, in ~/.docker/config.json.
I therefore tried this:
reg -d -r r.cfcr.io -u myusername -p dockeraccesstoken
(Note how cfaccesstoken has been changed to dockeraccesstoken).
This returns much more quickly now (as opposed to appearing to hang), but returns nothing:
2017/10/24 10:56:16 registry.ping url=https://r.cfcr.io/v2/
2017/10/24 10:56:18 registry.catalog url=https://r.cfcr.io/v2/_catalog
Repositories for r.cfcr.io
You need to first generate a token on CodeFresh User Settings
Next I will show you how to login from terminal
curl -H 'Host: r.cfcr.io' -H 'User-Agent: ItsMe' -H 'Authorization: Basic <AUTH>' --compressed 'https://r.cfcr.io/v2/token?account=xxxx&scope=repository%3A<user>/<name>%3Apush%2Cpull&service=r.cfcr.io'
You can get the <AUTH> by running
echo <account>:<token> | base64
This will return you a big token
{"expires_in":43200,"issued_at":"2017-10-24T03:34:54.118390368-07:00","token":"APMm...P91"}%
Now you can use that token to make a docker API call
$ curl -IH 'Host: r.cfcr.io' -H 'User-Agent: ItsMe' -H 'Authorization: Bearer APMm+...aRL' -X HEAD 'https://r.cfcr.io/v2/<user>/<user>/testci/blobs/sha256:c7fefcc4c54c63f32b5422ede095793eb5ff0f45ad7a22861974ec9b61e7749b'
HTTP/1.1 200 OK
Docker-Distribution-API-Version: registry/2.0
Content-Length: 22488057
Date: Tue, 24 Oct 2017 10:42:23 GMT
Content-Type: text/html
Server: Docker Registry
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
For registry search access below should have worked
curl -H 'Host: r.cfcr.io' -H 'User-Agent: ItsMe' -H 'Authorization: Basic ' --compressed 'https://r.cfcr.io/v2/token?account=xxxx&scope=registry%3Acatalog%3Asearch&service=r.cfcr.io'
But it doesn't and neither does registry:catalog:* for scopes. This is like hitting a target with blind folds and not even knowing in which direction we stand. You best bet is to get them to disclose some information to you
Edit-1: Getting the catalog
So I finally managed to crack it, but the result is a bit unfortunate. I ended up getting catalog/repositories of every single user. I checked, you still can't pull those repo. So their docker images are safe as such. These look like so:
Edit-2: Fetching all repos
After we notified Codefresh of this situation, here is how to do the fetch. One needs to first generate a token using below:
curl -H 'Host: r.cfcr.io' -H 'User-Agent: MyAgent' -H 'Authorization: Basic .....' --compressed 'https://r.cfcr.io/v2/token?account=<account>&service=r.cfcr.io'
And then using the same we can query the complete catalog:
curl -H "User-Agent: ItsMe" -H 'Authorization: Bearer <TOKEN>' 'https://r.cfcr.io/v2/_catalog?n=10' --compressed
I have a partial answer, and I think it is useful enough on its own for others approaching the same difficulty. I got some kind assistance via the chat widget on Codefresh's support page.
Proprietary API
Regarding the Codefresh API, I had not spotted that the domain g.codefresh.io is the same as their control panel. So, to authenticate, all I had to do was log into the control panel - oops! That reveals the call to https://g.codefresh.io/api/images and a much more complicated access token than I've been using - perhaps it is OAuth. It looks like this:
curl \
-X GET \
--header "Accept: application/json" \
--header "x-access-token: (36chars).(143chars).(22chars)-(4chars)-(15chars)" \
"https://g.codefresh.io/api/images"
As I considered in the question, it does look like the /api/images endpoint is for Codefresh builds only. So, this is out for me - I want everything in the registry.
Open API
So, turning to Docker's open standard for accessing registry, my support contact said this:
if you want to connect to cfcr.io through docker CLI, you can.
The username is your username at Codefresh. The password is a token you can generate at user settings -> "GENERATE" button at the Codefresh registry section.
That's what I've been doing so far, and it works for login, push and pull. It does not seem to work for the reg utility in ls mode though. Either I am still doing something wrong, or there are restrictions at Codefresh as to how the private registry may be used.
Unfortunately, it's not possible to use search and catalog commands for Codefresh Registry. The Codefresh registry (r.cfcr.io) is based on Google Container Registry (aka GCR) and Google Registry does not implement v1 Docker registry API and _catalog function.
Once they do, things will work for Codefresh Registry too.

Passing command line arguments through the Remote API

I am using the remote API to create and start containers, but I am not sure how to pass in the command line arguments I normally would when creating from the local machine. Specifically, I am using this image, which requires a bunch of arguments I would normally do when running 'docker run [arguments][image]'. Any ideas?
For argument passing, you can like this
curl -X POST localhost:2375/containers/create -H "Content-Type: application/json" -d '{"Cmd":["ping", "8.8.8.8"], "Image": "ubuntu"}'
Also see: http://blog.flux7.com/blogs/docker/docker-tutorial-series-part-8-docker-remote-api
This depends on which arguments you want to set. Up to port bindings,here you can find how to do that. In general, you have to use the JSON objects passed as body in your create and in your start request.

Resources