I am trying to learn both Dart and GCE. I already created a server at GCE, but I don't know how to install Dart since I can only use Linux commands on the Debian server.
This is mostly about Dart on AppEngine:
You should be able to find all information here https://www.dartlang.org/cloud/
I did it using the instructions from this page and the linked pages at the bottom.
The discussions here https://groups.google.com/a/dartlang.org/forum/#!forum/cloud provide some useful bits too.
Dart on Compute engine:
Here is a blog post that covers it pretty good http://financecoding.github.io/blog/2013/09/30/getting-started-with-dart-on-compute-engine/ but some things have changed since this was written.
There are docker images https://github.com/dart-lang/dart_docker ready to execute Dart scripts.
Just search for information how to use Docker images with GCE and you should be fine (there should already be a lot available)
Please ask again if you encounter concrete problems.
Dart on AppEngine runs as Managed VM. Managed VMs work differently than real AppEngine VMs that run 'native' supported languages like Java, Python, and Go. Managed VMs are in fact Compute engines but managed by AppEngine. This means they are launched and shut down depending on the load (and depending on some basic configuration settings in app.yaml and also depending on payment settings) while Compute Engines instances are basically always on. You have to manage yourself when instances should be added/removed depending on the load. There is Kubernetes which is a handy tool to make this easier but you have to actually manage your instances. Besides from that there is not much difference between Managed VMs and Compute Engine instances. A difference to native AppEngine is that you can add any libraries and also binaries to Managed VMs like to CE.
There are pricing differences but I don't know details about this myself yet
Related
We all know situations when you cannot go open source and freely distribute software - and I am in one of these situations.
I have an app that consists of a number of binaries (compiled from C sources) and Python code that wraps it all into a system. This app used to work as a cloud solution so users had access to app functions via network but no chance to touch the actual server where binaries and code are stored.
Now we want to deliver the "local" version of our system. The app will be running on PCs that our users will physically own. We know that everything could be broken, but at least want to protect the app from possible copying and reverse-engineering as much as possible.
I know that Docker is a wonderful deployment tool so I wonder: is it possible to create encrypted Docker containers where no one can see any data stored in the container's filesystem? Is there a known solution to this problem?
Also, maybe there are well known solutions not based on Docker?
The root user on the host machine (where the docker daemon runs) has full access to all the processes running on the host. That means the person who controls the host machine can always get access to the RAM of the application as well as the file system. That makes it impossible to hide a key for decrypting the file system or protecting RAM from debugging.
Using obfuscation on a standard Linux box, you can make it harder to read the file system and RAM, but you can't make it impossible or the container cannot run.
If you can control the hardware running the operating system, then you might want to look at the Trusted Platform Module which starts system verification as soon as the system boots. You could then theoretically do things before the root user has access to the system to hide keys and strongly encrypt file systems. Even then, given physical access to the machine, a determined attacker can always get the decrypted data.
What you are asking about is called obfuscation. It has nothing to do with Docker and is a very language-specific problem; for data you can always do whatever mangling you want, but while you can hope to discourage the attacker it will never be secure. Even state-of-the-art encryption schemes can't help since the program (which you provide) has to contain the key.
C is usually hard enough to reverse engineer, for Python you can try pyobfuscate and similar.
For data, I found this question (keywords: encrypting files game).
If you want a completely secure solution, you're searching for the 'holy grail' of confidentiality: homomorphous encryption. In short, you want to encrypt your application and data, send them to a PC, and have this PC run them without its owner, OS, or anyone else being able to scoop at the data.
Doing so without a massive performance penalty is an active research project. There has been at least one project having managed this, but it still has limitations:
It's windows-only
The CPU has access to the key (ie, you have to trust Intel)
It's optimised for cloud scenarios. If you want to install this to multiple PCs, you need to provide the key in a secure way (ie just go there and type it yourself) to one of the PCs you're going to install your application, and this PC should be able to securely propagate the key to the other PCs.
Andy's suggestion on using the TPM has similar implications to points 2 and 3.
Sounds like Docker is not the right tool, because it was never intended to be used as a full-blown sandbox (at least based on what I've been reading). Why aren't you using a more full-blown VirtualBox approach? At least then you're able to lock up the virtual machine behind logins (as much as a physical installation on someone else's computer can be locked up) and run it isolated, encrypted filesystems and the whole nine yards.
You can either go lightweight and open, or fat and closed. I don't know that there's a "lightweight and closed" option.
I have exactly the same problem. Currently what I was able to discover is bellow.
A. Asylo(https://asylo.dev)
Asylo requires programs/algorithms to be written in C++.
Asylo library is integrated in docker and it seems to be feаsable to create custom dоcker image based on Asylo .
Asylo depends on many not so popular technologies like "proto buffers" and "bazel" etc. To me it seems that learning curve will be steep i.e. the person who is creating docker images/(programs) will need a lot of time to understand how to do it.
Asylo is free of charge
Asylo is bright new with all the advantages and disadvantages of being that.
Asylo is produced by Google but it is NOT an officially supported Google product according to the disclaimer on its page.
Asylo promises that data in trusted environment could be saved even from user with root privileges. However, there is lack of documentation and currently it is not clear how this could be implemented.
B. Scone(https://sconedocs.github.io)
It is binded to INTEL SGX technology but also there is Simulation mode(for development).
It is not free. It has just a small set of functionalities which are not paid.
Seems to support a lot of security functionalities.
Easy for use.
They seems to have more documentation and instructions how to build your own docker image with their technology.
For the Python part, you might consider using Pyinstaller, with appropriate options, it can pack your whole python app in a single executable file, which will not require python installation to be run by end users. It effectively runs a python interpreter on the packaged code, but it has a cipher option, which allows you to encrypt the bytecode.
Yes, the key will be somewhere around the executable, and a very savvy costumer might have the means to extract it, thus unraveling a not so readable code. It's up to you to know if your code contains some big secret you need to hide at all costs. I would probably not do it if I wanted to charge big money for any bug solving in the deployed product. I could use it if client has good compliance standards and is not a potential competitor, nor is expected to pay for more licenses.
While I've done this once, I honestly would avoid doing it again.
Regarding the C code, if you can compile it into executables and/or shared libraries can be included in the executable generated by Pyinstaller.
I need to perform some machine learning tasks using a Tensor Flow based Neuronal network architecture (PointNet https://github.com/charlesq34/pointnet). I would like to use cloud infrastructure to do this, because I do not have the physical resources needed. The demands of the customer are, that they would like to get the whole set up machine I used for the training afterward and not only the final model. This is because they are researchers and would like to use the machine themselves, play around and understand what I did but they do not want to do the setup/installation work on their own. Unfortunately they can not provide a (physical or virtual) machine themselves right now.
The question is: Is it possible/reasonable to set up a machine on a cloud infrastructure provider like google cloud or AWS, install the needed software (which uses Nvidia Cuda) and export this machine after a while when suitable hardware is available, import it to a virtualisation tool (like Virtual Box) and continue the usage on ones own system? Will the installed GPU/Cuda-related software like TensorFlow etc. still work?
I guess it's possible, but it will be needed to configure the specific hardware to make it work on the local environment.
For Google Cloud Platform,
the introduction to Deep Learning Containers, will you allow to create portable environments.
Deep Learning Containers are a set of Docker containers with key data science frameworks, libraries, and tools pre-installed. These containers provide you with performance-optimized, consistent environments that can help you prototype and implement workflows quickly. Learn more.
In addition, please check Running Instances with GPU accelerators
Google provides a seamless experience for users to run their GPU workloads within Docker containers on Container-Optimized OS VM instances so that users can benefit from other Container-Optimized OS features such as security and reliability as well.
To configure Docker with Virtualbox, please check this external blog.
I am in the process of learning ROS and Gazebo for a project and saw that Amazon's Robomaker seems to be a great environment to get something up and working quickly. My concern is if I design and develop my project in Robomaker, that it will be tough to move over to my own servers should the time come.
Has anyone had any experience with this? Is it pretty simple to move over to another solution/infrastructure or are you married to AWS once you go down the rabbit hole?
Do they demand ownership of your IP?
Basically, if it is "normal ROS", there's a robot encoded in an URDF file, there's a bunch of standard ROS libraries (gazebo, controller, robot_localization, amcl, etc.), all strung together by a bunch of launch files (aka xml startup scripts).
They can't stop you from using this on your own system.
What they really offer, if you look at what they're pitching (https://aws.amazon.com/robomaker/), is a bunch of custom software, chiefly useful for running a fleet/multiple of robots in an industrial setting, involving human interaction / voice, data analytics and monitoring.
For most people learning, if you were to even touch this (you can get entirely on your own on a native ubuntu or ubuntu virtualbox), their simulation environment may be nicer out-of-the-box (I haven't played with it). The rest of the benefits, to translate them, is to have robots with data analytics and fancy AI+web tools, and then to do fleet lifecycle management.
You are most certainly not locked in - just make sure you want what they offer before shelling out anything.
In general, is it also a good idea to put your application in containers using docker o AWS ECR. So you can deploy your app in any other environment using a CI/CD pipeline, including the edge computer.
After evaluating existent tools like Ansible Tower, rundeck and others, it seems that no tool can fulfill the needed requirements.
We have complex data-center servers, cluster of DB and web servers, the data-center has a lot of client-systems, +100, and other tools like solr, redis, kafka... deployed there across the physical servers, not to mention that the same data-center servers have different accounts, linux users, (QA,stag,production..etc), for now the meta-data about these environments alongside their web-apps, source code to be used, servers of the cluster are all defined on xml and there is a bash scriptsreads from that XML that operated manually to run any operation/task (like checkout the source, build, deploy, start, stop... and other customized operations)
This system should be done with a developer and DevOps engineers together, but what I want to know, is there any preferable framework(s) that could be used for this system? does the workflows frameworks are usable on this case? e.g, Activiti BPMN? the Ant is an option but we don't need just an automation tool more that scheduler and logging and a lot of other services.
And if this is not the right place, can you please point out where cat I ask such question?
What's required is to create a web-based system as automation tool with:
UI to define the specific operations to be done, like build, deploy across the cluster specific web-app on specific env, start/stop specific web-app on specific machine, or any other customized operation, with multiple selections and flexible and dynamic options choosing way.
The FE should show the executing workflows and operations within them.
Dynamic way to create set of operations as a single workflow, that have dynamic ability to set the dependencies among them.
An interface between the back-end code of this system with the already existent bash scripts that will do the actual tasks across the DC servers.
A scheduler to be able to organize these operations in respect to a defined complicated dependencies between the workflows.
Authentication & authorization services to the users since there will be a lot of customized roles upon the operations,environments, the products...etc
Logging system to save the operations outputs.
Why not use a combination of ansible/Docker and jenkins, Jenkins can do most of the stuff you described using Pipeline projects/MultiProjects and Ansible for your UI and role related details
The task you're describing is basic network orchestration, and there're a bunch of orchestration engines/software out there. However, the fact that you are asking this question means you're just starting out and not ready to invest in a full fledged management product yet.
I strongly suggest that you take a look at Chef for your purposes.
Hope this helps.
I would recommend you to take a look at jenkinsx.io if you are targeting kubernetes and containers (docker). As part of the activiti bpm team we are trying to align with such technologies to make people life easy to integrate more complex workflows with DevOps and operations of your projects.
I recently deployed a tiny Haskell app with docker, using "scratch-haskell" as a base image.
Then I read about Unikernels and HALVM. And I got a little confused.
My docker container is about 6MB large. A Unikernel (with the same haskell app) would be roughly the same size I guess.
The Unikernel runs directly on the Xen hypervisor, whereas the docker Image (or general LXC) runs on a normal Linux distribution, which runs on bare metal.
Now I have the "choice" of running Linux with multiple minimal containers OR a Xen machine with multiple small Unikernels.
But what are the advantages and disadvantages of those two solutions? Is one more secure than the other? And are there any significant performance differences between them?
from http://wiki.xenproject.org/wiki/Unikernels
What do Unikernels Provide?
Unikernels normally generate a singular runtime environment meant to
enable single applications built solely with that environment.
Generally, this environment lacks the ability to spawn subprocesses,
execute shell commands, create multiple threads, or fork processes.
Instead, they provide a pure incarnation of the language runtime
targetted, be it OCaml, Haskell, Java, Erlang, or some other
environment.
Unikernels Versus Linux Containers
Much has been made recently of the advantages of Linux Container
solutions over traditional VMs. It is said by container advocates that
their lightweight memory footprint, quick boot time, and ease of
packaging makes containers the future of virtualization. While these
aspects of containers are certainly notable, they do not spell the end
of the world of the hypervisor. In fact, Unikernels may reduce the
long-term usefulness of containers.
Unikernels facilitate the very same desirable attributes described by
the container proponents, with the addition of an absolutely splendid
security story which few other solutions can match.
So if you want just run Haskell application Unikernels may work for you, and they should have even less overhead than docker (and docker overhead is very small anyway), but if your application will need some prepared environment, need to communicate with non Unikernels software docker is a better choice. I guess it is too early to say will Unikernels be useful or widespread or not, only time will tell.
Unikernals are great for things that are stateless. When you start needing disk access you are better off using Docker.
That's why all the "killer" apps for unikernals are statically configured kernels, like static web pages or software defined networking stacks.
There are many good explations heres a simple one :
Unikernel are VMs but specialized and optimized for the particular application.