I have a use case wherein I need to refer to the input dataset in the ACI/AKS which is in a blob (same used for training model). I'm not able to find related resources in the Microsoft official documentation. If anyone suggests to me how to do it, that will be very helpful.
It will be supported in the near future, Running Python scripts on Azure with Azure Container Instances to connect the blob.
https://kohera.be/tutorials-2/running-python-scripts-on-azure-with-azure-container-instances/
Related
I'm trying to create an assignment for students to do that contains the following :
A docker image with issues that have to be scanned and remedied. (using an opensource scanner in kubernetes)
(Maybe) A sample attack scenario that can exploit those vulnerabilities.
The problem arises when I try to find a suitable vulnerable image or create one. I cannot find a base of security issues at all. I really bend my back thinking of a suitable phrase in Google but everything leads merely to some blog posts about how-to scan an image.
I expected a database that might contain multiple sec issues and what causes them. I'd also expect some way to discern which are the most popular ones.
Do you have the source I require ?
Maybe you can just offer me 3-4 common security issues that are good to know and educational when having your first brush with docker ? (And how to create those issues ?)
The whole situation would have been probably easier if I myself would have been an expert in the field, but the thing I do is also my assignment as a student. (So as students we design assignments for each other. )
Looks like you are looking for the Container security hardening and Kubernetes security options maybe.
You can use some tools like
kubesec - Security risk analysis for Kubernetes resources
checkov - Prevent cloud misconfigurations and find vulnerabilities during build-time in infrastructure as code, container images and open-source packages
Trivy - vulnerabilities, misconfigurations, secrets, SBOM in containers, Kubernetes, code repositories, clouds and more
If you are looking for some questions you can set like, this is CKS (Certified Kubernetes Security) exam question
There are a number of pods/container running in the "spectacle" namespace.
Identify and delete the pods which have CRITICAL vulnerabilities.
For this trivy opensource tools comes into the picture to scan the image that you will be using in the deployment of Kubernetes or docker
trivy image --severity CRITICAL nginx:1.16 (Image running in container)
List of few questions you can create lab out of it : https://github.com/moabukar/CKS-Exercises-Certified-Kubernetes-Security-Specialist/tree/main/7-mock-exam-questions
Saving different models with their corresponding config files, tracking the results and parameters, searching among them using customized filters and maybe always having a pointer to the current SOTA can be quite time-saving.
I couldn't even find something similar to TensorFlow Hub on the local server. Right now, closest I could get is Git LFS.
Is there anything better out there?
I found the answer. A few open source projects are trying to do the job. The first one is named Data Science Version Control or DVC. Which according to the docs is:
simple command line Git-like experience. Does not require installing and maintaining any databases. Does not depend on any proprietary online services;
It manages and versions datasets and machine learning models. Data is saved in S3, Google cloud, Azure, Alibaba cloud, SSH server, HDFS or even local HDD RAID;
It makes projects reproducible and shareable, it helps to answer the question: "how the model was built";
It helps manage experiments with Git tags or branches and metrics tracking;
The other possible solution to think of is MinIO which is an object storage server
suited for storing unstructured data such as photos, videos, log files, backups and container / VM images.
Microsoft Azure has a service called Azure Machine Learnig service that does exactly this, but goes much further with governance/model explainability/DevOps etc. We also include free tiers to a lot of services and announced unlimited private repos on GitHub recently.
I understand that it is software shipped in some sort of binary format. In simple terms, what exactly is a docker-image? And what problem is it trying to solve? And how is it better than other distribution formats?
As a student, I don't completely get the picture of how it works and why so many companies are opting for it? Even many open source libraries are shipped as docker images.
To understand the docker images, you should better understand the main element of the Docker mechanism the UnionFS.
Unionfs is a filesystem service for Linux, FreeBSD and NetBSD which
implements a union mount for other file systems. It allows files and
directories of separate file systems, known as branches, to be
transparently overlaid, forming a single coherent file system.
The docker images сonsist of several layers(levels). Each layer is a write-protected filesystem, for every instruction in the Dockerfile created own layer, which has been placed over already created. Then the docker run or docker create is invoked, it make a layer on the top with write persmission(also it has doing a lot of other things). This approach to the distribution of containers is very good in my opinion.
Disclaimer:
It is my opinion which I'm found somewhere, feel free to correct me if I'm wrong.
I am looking for a tool which support sufficient providers list (like EC2 , Azure, Google cloud, and private cloud) with the option to create VMs in the cloud.
Have no luck in finding those, seems like I will need to learn lots of APIs to support my project, which doesn't make any sense, it also weird that nobody has already developed something like this, I have found Docker-machine which has lots of drivers for different cloud providers, but it seems like it wasn't support to be the goal of docker-machine, any idea?
check out packer.io and www.terraform.io
Pakcer helps you build images, and you can deploy them with terraform.
Hope this helps.
Why should I consider using Rocket instead of Docker in our development pipeline. We would like to use docker to create testable containers, but now there is Rocket which pretends to know the same. If we would like to start containerization should we seriously consider Rocket as it seems it is still pretty new?
There is not much information about Rocket, so I'm no clear where it stays now in 2015.
UPDATE: from https://coreos.com/blog/app-container-and-the-open-container-project/
As we participate in OCP, our primary goals are as follows:
Users should be able to package their application once and have it work with any container runtime (like Docker, rkt, Kurma, or Jetpack)
The standard should fulfill the requirements of the most rigorous security and production environments
The standard should be vendor neutral and developed in the open
Rocket is officially dead: https://github.com/rkt/rkt/issues/4024
After acquisition by Red Hot new owner concentrates efforts on https://podman.io/
podman provides rootless containers. Something that Docker strove to get for a long time (according to the below comment, they finally managed).
As with most competitors both have their advantage and disandvantages.
Docker hub offers a public registry where docker images can be pushed and pulled with ease.
There is also now a free registry offered by GitLab! Its really good.
A core issue at the moment is security. Docker now scan their images for security flaws and report on the security status of each image.
With rocket image signatures are cross checked with the signature of the publisher to see if they have been tampered with. This affords a degree of confidence.
For a fuller discussion on security see https://bobcares.com/blog/docker-vs-rkt-rocket/
With regards standards, it seems that OCI (Open Container Initiative) has been adopted by the big players and will pave the way forward for containerisation standatisation.