How to download Google Compute Engine disk content? - neo4j

I have linked a Persistent Volume to my Kubernetes Neo4j Replication Controller to store the DB data. Now I would like to download that data locally to run the production DB on my system. I can't find the way to download the Disk content. Can someone point me in the right direction?
Updates (Persistent Volume Creation with Kubernetes):
persistent-volume-db.json
{
"kind": "PersistentVolume",
"apiVersion": "v1",
"metadata": {
"name": "pv-db"
},
"spec": {
"capacity": {
"storage": "500Gi"
},
"accessModes": [
"ReadWriteMany"
],
"gcePersistentDisk": {
"pdName": "tuwa-db-data-disk",
"fsType": "ext4"
}
}
}
persistent-volume-claim-db.json
{
"kind": "PersistentVolumeClaim",
"apiVersion": "v1",
"metadata": {
"name": "pvc"
},
"spec": {
"accessModes": [
"ReadWriteMany"
],
"resources": {
"requests": {
"storage": "500Gi"
}
}
}
}
And then the usage:
neo4j-controller.json
{
"kind": "ReplicationController",
"apiVersion": "v1",
"metadata": {
"name": "neo4j-controller",
"labels": {
"name": "neo4j"
}
},
"spec": {
"replicas": 1,
"template": {
"metadata": {
"labels": {
"name": "neo4j"
}
},
"spec": {
"containers": [
{
"name": "neo4j",
"image": "neo4j/neo4j",
"ports": [
{
"name": "neo4j-server",
"containerPort": 7474
}
],
"volumeMounts": [
{
"mountPath": "/data/databases",
"name": "pv-db"
}
]
}
],
"volumes": [
{
"name": "pv-db",
"persistentVolumeClaim": {
"claimName": "pvc-db"
}
}
]
}
}
}
}

GCE's admin panel doesn't have a "download" button for persistent disks, but gcloud makes it easy to copy files from an instance to your local machine:
gcloud compute copy-files example-instance:~/REMOTE-DIR ~/LOCAL-DIR --zone us-central1-a
This will copy ~/REMOTE-DIR from a remote instance into ~/LOCAL-DIR on your machine. Just replace the directory names, example-instance with your instances name, and adjust your zone if necessary. More info here in the docs.

gcloud compute copy-files #has been deprecated
Please use gcloud compute scp instead. Note that gcloud compute scp does not have recursive copy on by default. To turn on recursion, use the --recurse flag.
gcloud compute scp --recurse example-instance:~/instance-1_path ~/locad_path --zone=us-central1-a
This will copy ~/instance-1_path from a remote compute instance into ~/local_path on your personal machine. Just replace the directory names, example-instance with your instances name, and adjust your zone if necessary.
Complete documentation of this SDK can be found here

Related

How to get image upload date/time from Docker Registry

I need to get the image upload date from a Docker Registry
I currently use the following https://registry/v2/repository/manifests/tag which gives me Creation Date. This is stale in most cases. I want to know when something was uploaded.
If I can't get this, is there a way to do docker build and specify and date/time which could be used in place of the Creation Date?
An example of the manifest I'm getting which shows the only dates available are the v1Compatibility/created:
{
"schemaVersion": 1,
"name": "users/jesaremi/baseimage",
"tag": "6a69f60507f029f76ff102aa1b89b562d2d784dfdbfef38cb0ed5c0b61a188ff",
"architecture": "amd64",
"fsLayers": [
{
"blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
},
{
"blobSum": "sha256:7c9d20b9b6cda1c58bc4f9d6c401386786f584437abbe87e58910f8a9a15386b"
}
],
"history": [
{
"v1Compatibility": "{\"architecture\":\"amd64\",\"config\":{\"Hostname\":\"\",\"Domainname\":\"\",\"User\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"],\"Cmd\":[\"sh\"],\"ArgsEscaped\":true,\"Image\":\"sha256:758a17a836a4c09586a291c928d1f0561320e252d07c4749e14338daefe84b18\",\"Volumes\":null,\"WorkingDir\":\"\",\"Entrypoint\":null,\"OnBuild\":null,\"Labels\":null},\"container\":\"e30cd53834b3dfdb989c63cc73f4f31f404c7a6a0c0e9d6b9e3e8451edd72596\",\"container_config\":{\"Hostname\":\"e30cd53834b3\",\"Domainname\":\"\",\"User\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"],\"Cmd\":[\"/bin/sh\",\"-c\",\"#(nop) \",\"CMD [\\\"sh\\\"]\"],\"ArgsEscaped\":true,\"Image\":\"sha256:758a17a836a4c09586a291c928d1f0561320e252d07c4749e14338daefe84b18\",\"Volumes\":null,\"WorkingDir\":\"\",\"Entrypoint\":null,\"OnBuild\":null,\"Labels\":{}},\"created\":\"2019-09-04T19:20:16.230463098Z\",\"docker_version\":\"18.06.1-ce\",\"id\":\"a91ec18e2f45c300f1df0a23ac04c1396d791c6c387dd5e16e44dc96a4fc309d\",\"os\":\"linux\",\"parent\":\"bd5fbbc2870744fe1d37bbd120eebe4c441f8401c54b04b8ae0f9f625936f4c3\",\"throwaway\":true}"
},
{
"v1Compatibility": "{\"id\":\"bd5fbbc2870744fe1d37bbd120eebe4c441f8401c54b04b8ae0f9f625936f4c3\",\"created\":\"2019-09-04T19:20:16.080265634Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) ADD file:9151f4d22f19f41b7a289e87aa9cfba3956ffd27746cb3b171b9bd2cb7e6c313 in / \"]}}"
}
],
"signatures": [
{
"header": {
"jwk": {
"crv": "P-256",
"kid": "3NQM:K5YD:M3XF:EKJD:4S64:3772:BJOT:JIMR:NX4R:2XYS:IDNA:NOKL",
"kty": "EC",
"x": "k6pZfyr-dKYLri5KJCL70UmNLCQnfUh2lAC_nDK9PVw",
"y": "MhrKOUbx1sgsbF0kG9d5bfvkVaxaFWiKlWTwgFyHkbQ"
},
"alg": "ES256"
},
"signature": "klE8-cWOS1GZenBB7CPXYUK8VWmqiVQaFfWGgBQPn_L8iayojGEUc9D_06WCUdAqL7upvNIxcCPXJvZMORLn_Q",
"protected": "eyJmb3JtYXRMZW5ndGgiOjIxOTksImZvcm1hdFRhaWwiOiJDbjAiLCJ0aW1lIjoiMjAxOS0xMC0yM1QyMTo0MTozM1oifQ"
}
]
}

How to directly mount external NFS share/volume in kubernetes(1.10.3)

I am using kubernetes : v1.10.3 , i have one external NFS server which i am able to mount anywhere ( any physical machines). I want to mount this NFS directly to pod/container . I tried but every time i am getting error. don't want to use privileges, kindly help me to fix.
ERROR: MountVolume.SetUp failed for volume "nfs" : mount failed: exit
status 32 Mounting command: systemd-run Mounting arguments:
--description=Kubernetes transient mount for /var/lib/kubelet/pods/d65eb963-68be-11e8-8181-00163eeb9788/volumes/kubernetes.io~nfs/nfs
--scope -- mount -t nfs 10.225.241.137:/stagingfs/alt/ /var/lib/kubelet/pods/d65eb963-68be-11e8-8181-00163eeb9788/volumes/kubernetes.io~nfs/nfs
Output: Running scope as unit run-43393.scope. mount: wrong fs type,
bad option, bad superblock on 10.225.241.137:/stagingfs/alt/, missing
codepage or helper program, or other error (for several filesystems
(e.g. nfs, cifs) you might need a /sbin/mount. helper program)
In some cases useful info is found in syslog - try dmesg | tail or so.
NFS server : mount -t nfs 10.X.X.137:/stagingfs/alt /alt
I added two things for volume here but getting error every time.
first :
"volumeMounts": [
{
"name": "nfs",
"mountPath": "/alt"
}
],
Second :
"volumes": [
{
"name": "nfs",
"nfs": {
"server": "10.X.X.137",
"path": "/stagingfs/alt/"
}
}
],
---------------------complete yaml --------------------------------
{
"kind": "Deployment",
"apiVersion": "extensions/v1beta1",
"metadata": {
"name": "jboss",
"namespace": "staging",
"selfLink": "/apis/extensions/v1beta1/namespaces/staging/deployments/jboss",
"uid": "6a85e235-68b4-11e8-8181-00163eeb9788",
"resourceVersion": "609891",
"generation": 2,
"creationTimestamp": "2018-06-05T11:34:32Z",
"labels": {
"k8s-app": "jboss"
},
"annotations": {
"deployment.kubernetes.io/revision": "2"
}
},
"spec": {
"replicas": 1,
"selector": {
"matchLabels": {
"k8s-app": "jboss"
}
},
"template": {
"metadata": {
"name": "jboss",
"creationTimestamp": null,
"labels": {
"k8s-app": "jboss"
}
},
"spec": {
"volumes": [
{
"name": "nfs",
"nfs": {
"server": "10.X.X.137",
"path": "/stagingfs/alt/"
}
}
],
"containers": [
{
"name": "jboss",
"image": "my.abc.com/alt:7.1_1.1",
"resources": {},
"volumeMounts": [
{
"name": "nfs",
"mountPath": "/alt"
}
],
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"imagePullPolicy": "IfNotPresent",
"securityContext": {
"privileged": true
}
}
],
"restartPolicy": "Always",
"terminationGracePeriodSeconds": 30,
"dnsPolicy": "ClusterFirst",
"securityContext": {},
"schedulerName": "default-scheduler"
}
},
"strategy": {
"type": "RollingUpdate",
"rollingUpdate": {
"maxUnavailable": "25%",
"maxSurge": "25%"
}
},
"revisionHistoryLimit": 10,
"progressDeadlineSeconds": 600
},
"status": {
"observedGeneration": 2,
"replicas": 1,
"updatedReplicas": 1,
"readyReplicas": 1,
"availableReplicas": 1,
"conditions": [
{
"type": "Available",
"status": "True",
"lastUpdateTime": "2018-06-05T11:35:45Z",
"lastTransitionTime": "2018-06-05T11:35:45Z",
"reason": "MinimumReplicasAvailable",
"message": "Deployment has minimum availability."
},
{
"type": "Progressing",
"status": "True",
"lastUpdateTime": "2018-06-05T11:35:46Z",
"lastTransitionTime": "2018-06-05T11:34:32Z",
"reason": "NewReplicaSetAvailable",
"message": "ReplicaSet \"jboss-8674444985\" has successfully progressed."
}
]
}
}
Regards
Anupam Narayan
As stated in the error log:
for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount. helper program
According to this question, you might be missing the nfs-commons package which you can install using sudo apt install nfs-common

Kubernetes create Service in deployment+rbd mode, also same configured to succeed in default namespace, fail under non default namespace?

Kubernetes create Service in deployment+rbd mode, also same configured to succeed in default namespace, fail under non default namespace?
The config(json):
{
"kind": "Deployment",
"spec": {
"replicas": "1",
"template": {
"spec": {
"volumes": [
{
"rbd": {
"secretRef": {
"name": "ceph-secret"
},
"image": "zhaosiyi.24",
"fsType": "ext4",
"readOnly": false,
"user": "admin",
"monitors": [
"xxx.xxx.xxx.6:6789",
"xxx.xxx.xxx.7:6789",
"xxx.xxx.xxx.8:6789"
],
"pool": "rrkd.rbd"
},
"name": "aa"
}
],
"imagePullSecrets": [
{
"name": "registrykey-m3-1"
}
],
"containers": [
{
"image": "ccr.ccs.tencentyun.com/rrkd/rrkd-nginx:1.0",
"volumeMounts": [
{
"readOnly": false,
"mountPath": "/mnt",
"name": "aa"
}
],
"name": "aa",
"ports": [
{
"protocol": "TCP",
"containerPort": 80
}
]
}
]
},
"metadata": {
"labels": {
"name": "aa"
}
}
},
"selector": {
"matchLabels": {
"name": "aa"
}
}
},
"apiVersion": "extensions/v1beta1",
"metadata": {
"labels": {
"name": "aa"
},
"name": "aa"
}
}
{
"kind": "Service",
"spec": {
"type": "NodePort",
"ports": [
{
"targetPort": 80,
"protocol": "TCP",
"port": 80
}
],
"selector": {
"name": "aa"
}
},
"apiVersion": "v1",
"metadata": {
"labels": {
"name": "aa"
},
"name": "aa"
}
}
The strangest thing is that the details of the describe pod show success, without any error information, but the get pod is not actually successful, as below:
enter image description here
enter image description here
The problem has been solved. Non default of namespace is unsuccessful because the new namespace is not have Secret, so the authentication failed at pull image. Need to manually create the Secret for pull image. If it is RBD or PVC mode, you also need to manually create the Secret for mounting storage.
Can you show more detail for the describe of pod?
I think it will show the failed log
it always you don't have secret or the secret not setting well,
and the ceph maybe wrong too

Kubernetes: how to use gitRepo volume?

Can someone give an example of how to use the gitRepo type of volume in Kubernetes?
The doc says it's a plugin, not sure what that means. Could not find an example anywhere and i don't know the proper syntax.
especially is there parameters to pull a specific branch, use credentials (username, password, or SSH key) etc...
EDIT:
Going through the Kubernetes code this is what I figured so far:
- name: data
gitRepo:
repository: "git repo url"
revision: "hash of the commit to use"
But can't seen to make it work, and not sure how to troubleshoot this issue
This is a sample application I used:
{
"kind": "ReplicationController",
"apiVersion": "v1",
"metadata": {
"name": "tess.io",
"labels": {
"name": "tess.io"
}
},
"spec": {
"replicas": 3,
"selector": {
"name": "tess.io"
},
"template": {
"metadata": {
"labels": {
"name": "tess.io"
}
},
"spec": {
"containers": [
{
"image": "tess/tessio:0.0.3",
"name": "tessio",
"ports": [
{
"containerPort": 80,
"protocol": "TCP"
}
],
"volumeMounts": [
{
"mountPath": "/tess",
"name": "tess"
}
]
}
],
"volumes": [
{
"name": "tess",
"gitRepo": {
"repository": "https://<TOKEN>:x-oauth-basic#github.com/tess/tess.io"
}
}
]
}
}
}
}
And you can use the revision too.
PS: The repo above does not exist anymore.
UPDATE:
gitRepo is now deprecated
https://github.com/kubernetes/kubernetes/issues/60999
ORIGINAL ANSWER:
going through the code this is what i figured:
- name: data
gitRepo:
repository: "git repo url"
revision: "hash of the commit to use"
after fixing typos in my mountPath, it works fine.

Service host/port undefined, Kubernetes/Google Container Engine

I have a service with the name mongodb. According to the documentation, the service host and port should be available to other pods in the same cluster through $MONGODB_SERVICE_HOST and $MONGODB_SERVICE_PORT.
However, neither of these are set in my frontend pods. What are the requirements for this to work?
frontend-controller.json
{
"id": "frontend",
"kind": "ReplicationController",
"apiVersion": "v1beta1",
"desiredState": {
"replicas": 1,
"replicaSelector": {"name": "spatula", "role": "frontend"},
"podTemplate": {
"desiredState": {
"manifest": {
"version": "v1beta1",
"id": "frontend",
"containers": [{
"name": "frontend",
"image": "gcr.io/crafty_apex_841/spatula_frontend",
"cpu": 100,
"ports": [{"name": "spatula-server", "containerPort": 80}]
}]
}
},
"labels": { "name": "spatula", "role": "frontend" }
}
},
"labels": { "name": "spatula", "role": "frontend" }
}
frontend-service.json
{
"apiVersion": "v1beta1",
"kind": "Service",
"id": "frontend",
"port": 80,
"containerPort": "spatula-server",
"labels": { "name": "spatula", "role": "frontend" },
"selector": { "name": "spatula", "role": "frontend" },
"createExternalLoadBalancer": true
}
mongodb-service.json
{
"apiVersion": "v1beta1",
"kind": "Service",
"id": "mongodb",
"port": 27017,
"containerPort": "mongodb-server",
"labels": { "name": "spatula", "role": "mongodb" },
"selector": { "name": "spatula", "role": "mongodb" }
}
mongodb-controller.json
{
"id": "mongodb",
"kind": "ReplicationController",
"apiVersion": "v1beta1",
"desiredState": {
"replicas": 1,
"replicaSelector": {"name": "spatula", "role": "mongodb"},
"podTemplate": {
"desiredState": {
"manifest": {
"version": "v1beta1",
"id": "mongodb",
"containers": [{
"name": "mongodb",
"image": "dockerfile/mongodb",
"cpu": 100,
"ports": [{"name": "mongodb-server", "containerPort": 27017}]
}]
}
},
"labels": { "name": "spatula", "role": "mongodb" }
}
},
"labels": { "name": "spatula", "role": "mongodb" }
}
The service:
$ gcloud preview container services list
NAME LABELS SELECTOR IP PORT
mongodb name=spatula,role=mongodb name=spatula,role=mongodb 10.111.240.154 27017
The pod:
$ gcloud preview container pods list
POD IP CONTAINER(S) IMAGE(S) HOST LABELS STATUS
9ffd980f-ab56-11e4-ad76-42010af069b6 10.108.0.11 mongodb dockerfile/mongodb k8s-spatula-node-1.c.crafty-apex-841.internal/104.154.44.77 name=spatula,role=mongodb Running
Because environment variables for pods are only created when the pod is started, the service has to exist before a given pod in order for that pod to see the service's environment variables. You should be able to see them from all new pods you create.
If you'd like to learn more, additional explanation of how services work can be found in the documentation.
Alternatively, all newly created clusters in Container Engine (version 0.9.2 and above) have a SkyDNS service running in the cluster that you can use to access services from pods, even those without the environment variables.

Resources