I'm looking for a cloud-based (pub, priv or hybrid) solution that allows me to configure every detail about the platform (container, system stack, virtualized hardware, etc.) for my app, but also deploys a templated version of my app on all app server nodes as soon as I run my 1st build. Hence I configure the app/platform, click a button, and boom: I have a WAR deployed and running across a cluster of nodes. (Granted, since I have not written any code at this point, this deployed WAR would have de minimis code inside of it and would consistitute the bare minimal code required to produce a WAR. In the case of Grails, it might just be the code that is generated by running grails create-app myapp.)
I'm calling this "Application-as-a-Service", because it not only is a traditional PaaS, but also goes one step further and deploys packaged WARs using some kind of templated source code repo.
My question: CloudFoundry says they support multiple frameworks (Spring, Grails, etc.); does this mean it can do what I describe above? If not, what does CloudFoundry mean when they say that they "support the Grails framework"?
Using CF you are able to configure platform/OS. Currently used container is warden. Virtualised hardware depends on IaaS used for(under) CF. Then you may 'click a button' and your app will be deployed and running across a cluster of nodes (DEA instances).
CloudFoundry says they support multiple frameworks (Spring, Grails, etc.); does this mean it can do what I describe above?
I don't fully follow what you're trying to do above, but I can tell you about the general work flows for CloudFoundry.
As an administrator, you use Bosh to deploy CloudFoundry to your IaaS. You can control anything you want at the IaaS level (assuming you're an administrator for your IaaS), so long as you meed the requirements of CF like storage, memory and CPU. In addition, you can control the CF deployment by adjusting the various configuration settings (YAML files) for CF. This allows you to tune the amount of resources (memory, CPU, disk space, etc..) that are available for application developers.
As a developer, you take your application and push it to a running CF installation. You may or may not be the administrator of that CF installation, if you are not then you'll be subject to the policies of the administrator.
The push process takes your application (ruby, python, php, go, java, grails, etc...) and uploads it to CF. From there, the application files are turned into a droplet by the build pack. This process is easier than it sounds, the build pack is just adding all the things that are necessary to run your app, like a web server or app container. With the droplet created, CF will run it or possibly even multiple instances of it if you so desire.
A couple further notes / clarifications:
How much memory your application gets is up to the app developer and can be adjusted at the time an app is push or later using cf scale.
CPUs are shared across all apps. You cannot reserve or guarantee usage per app. Application usage is limited though, so one app cannot steal all of the available CPUs.
The amount of disk storage per app is set by the administrator.
Disk storage for applications is ephemeral and is cleared when an application is restarted. Persistent storage should be provided through a service.
As you can see, CF administrators have a good degree of control over the system. Application developers not so much, but that is in fact the point of PaaS. Application developers don't want to waste time playing sys-admin. They just want to run their apps.
If not, what does CloudFoundry mean when they say that they "support the Grails framework"?
What is meant by this is that you can take a WAR file produced by Grails and run it on CloudFoundry without any additional work.
The CloudFoundry system uses build packs to handle the process of taking your application (i.e. your Grails WAR file) and preparing it to run. As I mentioned above, this usually involves installing and configuring some sort of server. In the case of Java / Grails, it involves setting up Apache Tomcat and configuring it to run your application (Note, if you don't like the way it's configured by default, you can customize or create your own build pack to configure it exactly the way you like).
CloudFoundry supports Grails and other JVM based languages because it can take those applications and automatically install, configure and run them.
Related
I am using docker containers and have docker-compose files for both local development and production environment. I want to try Google Cloud Platform for my new app and specifically Google Kubernetes Engine. My tools is Docker for Mac with Kubernetes on local machine.
It is super important for developers to be able to change code and to see changes live for local development.
Use cases:
Backend developer make changes to basic Flask API (or whatever you use) and should see changes on reloaded app immediately.
Frontend developer make changes to HTML layout and should see changes on web page immediately.
At the moment i am using docker-compose files to mount source code to local containers. But Kubernetes does not support relative paths to mount the source code.
Ideally i should be able to set the variable
Deployment.spec.templates.spec.containers.volumes.hostPath
as relative path to my repo. For example, in our team developers clone repo to this folders:
/User/BACKEND_developer/code/project_repo
/User/FRONTEND_developer/code/project_repo
Obviously you can't commit and build the image after every little change to the source code.
So what is the best practice for local development with Kubernetes? Do i need some additional tools to modify .yaml files for every developer?
#tgogos is right.
The best way to achieve your goal is to use Skaffold
It will rebuild container whenever it sees changes in source code.
Skaffold has a pluggable architecture that allows you to choose the tools in developer workflow that work best for you:
A very promising approach for dynamic languages is the hybrid approach recently introduced by Skaffold, allowing to take advantage of the usual auto-reload mechanisms. You can define two set of files:
Changing a file on the first set triggers the full rebuild+push+deploy mechanism.
Changing a file on the second set only syncs the file between the local machine and the container.
Such an hybrid approach is well suited to a large class of technology stacks, like Node.js, React, Angular, Python, where you can use the native hot-reload mechanism for source code changes, and trigger the full rebuild only when it’s needed (for example, adding a dependency). This helps a lot in keeping the latency low.
I spoke about this in my recent talk at All Day Devops. Here there’s an example based on Node.JS.
We create iOS and Android apps that are white-labeled. They all use a single code base (one for iOS and one for Android). Whenever we need to make changes to all of our apps (> 100 live in App Store) we rely on Fastlane. We have a "bulk" command that submits each new build to Apple, changing out config variables first and a few files so each app is unique.
This has worked well for us... but... its getting really slow. We'd love to be able to take advantage of some of the continuous development services out there. It seems like they weren't necessarily made for this use case but it might still work?
Ideally instead of running bulk on a local machine we could spin up 100 instances on something like CircleCI and they all run side by side, using our fastlane script to build, submit, etc.
We started by looking into CircleCI. The problem we are running into is they don't allow injection of variables into a job (https://ideas.circleci.com/ideas/CCI-I-690).
Is there a better service for this goal? Is there a tool that was built to achieve this? Struggling to find an alternative to hacking together a bunch of smaller tools.
I think you already identified your first step: You will have to split your fastlane (and other tooling) configuration, so it is possible to build each app in isolation.
Then you can trigger a job for each app on a CI service like for example Travis CI or Azure Pipelines (both have a simple API you can use to start jobs and give them some parameters that will be available to your job) that builds and releases the app.
All the other things (e.g. one big build vs. many small build steps etc.) are just implementation details and will depend on the individual service or tools you choose.
I am using docker containers and have docker-compose files for both local development and production environment. I want to try Google Cloud Platform for my new app and specifically Google Kubernetes Engine. My tools is Docker for Mac with Kubernetes on local machine.
It is super important for developers to be able to change code and to see changes live for local development.
Use cases:
Backend developer make changes to basic Flask API (or whatever you use) and should see changes on reloaded app immediately.
Frontend developer make changes to HTML layout and should see changes on web page immediately.
At the moment i am using docker-compose files to mount source code to local containers. But Kubernetes does not support relative paths to mount the source code.
Ideally i should be able to set the variable
Deployment.spec.templates.spec.containers.volumes.hostPath
as relative path to my repo. For example, in our team developers clone repo to this folders:
/User/BACKEND_developer/code/project_repo
/User/FRONTEND_developer/code/project_repo
Obviously you can't commit and build the image after every little change to the source code.
So what is the best practice for local development with Kubernetes? Do i need some additional tools to modify .yaml files for every developer?
#tgogos is right.
The best way to achieve your goal is to use Skaffold
It will rebuild container whenever it sees changes in source code.
Skaffold has a pluggable architecture that allows you to choose the tools in developer workflow that work best for you:
A very promising approach for dynamic languages is the hybrid approach recently introduced by Skaffold, allowing to take advantage of the usual auto-reload mechanisms. You can define two set of files:
Changing a file on the first set triggers the full rebuild+push+deploy mechanism.
Changing a file on the second set only syncs the file between the local machine and the container.
Such an hybrid approach is well suited to a large class of technology stacks, like Node.js, React, Angular, Python, where you can use the native hot-reload mechanism for source code changes, and trigger the full rebuild only when it’s needed (for example, adding a dependency). This helps a lot in keeping the latency low.
I spoke about this in my recent talk at All Day Devops. Here there’s an example based on Node.JS.
I am working on a grails app and need regularly to deploy hot fixes to a remote server. I am using jenkins with grails plugin for automation.
My point is the following:
Most of the time i fix a few classes, with no big changes in the app (such as new database schema, new plugins....). However each time i create a patch i have to upload trough ssh a 75M war file, which takes between 15 to 20 min. Most of the data is not needed (ie all the packaged jars). What would be sufficient is to upload only the fresh compiled classes from WEB-INF/classes/ and reload the servlet container (in my case jetty).
Anybody experienced with this, preferably with jenkins?
Check the nojars argument for the war task: http://www.grails.org/doc/1.3.7/ref/Command%20Line/war.html
This way you can place all your .jars (which are usually the biggest files inside a .war) in some other directory on the server and just reference that directory in your Jetty classpath.
Or you could write a shell script to explode the .war file (after all it's just a regular .zip file), add the compiled classes and then re-package it.
You could try using CloudBees to do continuous delivery releases. They also use deltas to upload your changes, and deployments don't affect the user experience at all.
An easy to use plugin is available to make the process seamless from within your Grails app and in a Jenkins build. I've written a blog post about how to get it all working easily.
I remember seeing this subject on the mailing list...
http://grails.1312388.n4.nabble.com/Incremental-Deployment-td3066617.html
...they recommend using rsync or xdelta3 to only transfer updated files. Haven't tried it, but it might help you?
Maybe the Cloudfoundry Micro Cloud is an option, a deployment just transfers the deltas and not the whole war file.
I am evaluating the Hudson build system for use as a centralized, "sterile" build environment for a large company with very distributed development (from both a geographical and managerial perspective). One goal is to ensure that builds are only a function of the contents of a source control tree and a build script (also part of that tree). This way, we can be certain that the code placed into a production environment actually originated from our source control system.
Hudson seems to provide an ant script with the full set of rights assigned to the user invoking the Hudson server itself. Because we want to allow individual development groups to modify their build scripts without administrator intervention, we would like a way to sandbox the build process to (1) limit the potential harm caused by an errant build script, and (2) avoid all the games one might play to insert malicious code into a build.
Here's what I think I want (at least for Ant, we aren't using Maven/Ivy right now):
The Ant build script only has access to its workspace directory
It can only read from the source tree (so that svn updates can be trusted and no other code is inserted).
It could perhaps be allowed read access to certain directories (Ant distribution, JDK, etc.) that are required for the build classpath.
I can think of three ways to implement this:
Write an ant wrapper that uses the Java security model to constrain access
Create a user for each build and assign the rights described above. Launch builds in this user space.
(Updated) Use Linux "Jails" to avoid the burden of creating a new user account for each build process. I know little about these though, but we will be running our builds on a Linux box with a recent RedHatEL distro.
Am I thinking about this problem correctly? What have other people done?
Update: This guy considered the chroot jail idea:
https://www.thebedells.org/blog/2008/02/29/l33t-iphone-c0d1ng-ski1lz
Update 2: Trust is an interesting word. Do we think that any developers might attempt anything malicious? Nope. However, I'd bet that, with 30 projects building over the course of a year with developer-updated build scripts, there will be several instances of (1) accidental clobbering of filesystem areas outside of the project workspace, and (2) build corruptions that take a lot of time to figure out. Do we trust all our developers to not mess up? Nope. I don't trust myself to that level, that's for sure.
With respect to malicious code insertion, the real goal is to be able to eliminate the possibility from consideration if someone thinks that such a thing might have happened.
Also, with controls in place, developers can modify their own build scripts and test them without fear of catastrophe. This will lead to more build "innovation" and higher levels of quality enforced by the build process (unit test execution, etc.)
This may not be something you can change, but if you can't trust the developers then you have a larger problem then what they can or can not do to your build machine.
You could go about this a different way, if you can't trust what is going to be run, you may need a dedicated person(s) to act as build master to verify not only changes to your SCM, but also execute the builds.
Then you have a clear path of responsibilty for builds to not be modified after the build and to only come from that build system.
Another option is to firewall off outbound requests from the build machine to only allow certain resources like your SCM server, and your other operational network resources like e-mail, os updates etc.
This would prevent people from making requests in Ant to off the build system for resources not in source control.
When using Hudson you can setup a Master/Slave configuration and then not allow builds to be performed on the Master. If you configure the Slaves to be in a virtual machine, that can be easily snapshotted and restored, then you don't have to worry about a person messing up the build environment. If you apply a firewall to these Slaves, then it should solve your isolation needs.
I suggest you have 1 Hudson master instance, which is an entry point for everyone to see/configure/build the projects. Then you can set up multiple Hudson slaves, which might very well be virtual machines or (not 100% sure if this is possible) simply unprivileged users on the same machine.
Once you have this set up, you can tie builds to specific nodes, which are not allowed - either by virtual machine boundaries or by Linux filesystem permissions - to modify other workspaces.
How many projects will Hudson be building? Perhaps one Hudson instance would be too big, given the security concerns you are expressing. Have you considered distributing the Hudson instances out - one per team. This avoids the permission issue entirely.