Is it possible to run bentoml build without importing the services.py file during the process?
I'm trying to put the bento build and containarize steps in our CI/CD server. Our model depends on some OS packages installed and some python packages. I thought I could run bentoml build to package the model code and binaries that are present. I'd leave the dependencies especification to the contanairize step.
To my surprise the bentoml build process tried to import the service file during the packaging and the build failed since I didn't have the dependencies installed in my CI/CD machine.
Can I prevent this importing while building/packaging the model? Maybe I should ignore the bento containarize and create my bento container by hand and just execute the bentoml serve inside.
I feel that the need to install by hand the dependencies is doubling the effort to specify them in the bentofile.yaml and preventing the reproducibility of my environment.
This is not currently possible. The community is working on an environment management feature, such that an environment with the necessary dependencies will be automatically created during build.
Related
Absolute beginner in DevOps here. I have a Gitlab repo that I would like to build and run its tests in the Gitlab pipeline CI.
So far, I'm only testing locally on my machine with a specific runner. There's a lot information out there and I'm starting to get lost with what to use and how to use it.
How would I go about creating a container with the tools that I need ? (VS compiler, cmake, git, etc...)
My application contains an SDK that only works on windows, so I'm not sure building on another platform would work at all, so how do I select a windows based container?
How would I use that container in the yml file in gitlab so that I can build my solution and run my tests?
Any specific documentation links or suggestions are welcomed and appreciated.
How would I go about creating a container with the tools that I need ? (VS compiler, cmake, git, etc...)
you can install those tools before the pipeline script runs. I usually do this in before_script.
If there's large-ish packages that need to be installed on every pipeline run, I'd recommend that you make yourown image, with all the required build dependencies, push it to GitLab and then just use it as your job image.
My application contains an SDK that only works on windows, so I'm not sure building on another platform would work at all, so how do I select a windows based container?
If you're using gitlab.com - Windows runners are currently in beta, but available for use.
SaaS runners on Windows are in beta and shouldn’t be used for production workloads.
During this beta period, the shared runner quota for CI/CD minutes applies for groups and projects in the same manner as Linux runners. This may change when the beta period ends, as discussed in this related issue.
If you're self-hosting - setup your own runner on Windows.
How would I use that container in the yml file in gitlab so that I can build my solution and run my tests?
This really depends on:
previous parts (you're using GL.com / self hosted)
how your application is built
what infrastructure you have access to
What I'm trying to say is that I feel like I can't give you a good answer without quite some more information
I am currently testing Xcode Cloud as a member of Apple's private beta program and encountered an issue when trying to Archive / Build (any action) on the cloud for my project.
The project is a fairly basic SwiftUI app with CocoaPods dependencies. I have followed the steps to integrate CocoaPods into my project as described by Apple by simply committing my Pods directory to GitHub. However, I am getting a build error on the cloud for every attempted action:
Command PhaseScriptExecution failed with a nonzero exit code
Here is the ASC log for reference:
This is very strange because the same project builds and archives successfully on my local machine. I have used the same macOS and Xcode versions in the Workflow editor as my local version of Xcode.
How can I resolve this error?
Custom shell scripts restrictions
TL;DR
Apple has locked down the security of their hosted infrastructure by only enabling shell scripts to run as part of ci_post_clone.sh, ci_pre_xcodebuild.sh or ci_post_xcodebuild.sh in the ci_scripts folder. Your Pods project has a custom shell script outside of this folder that is not triggered by one of these CI scripts, so does not have running permissions.
The solution for this specific issue are:
(technically) Refactor your build script phase to inline the shell script file inside the run script.
(recommended, but not always possible) Use the Swift Package version of the CocoaPod if available.
(workaround) Downgrade the CocoaPod to a version without an external shell script.
Reference
From Customize your advanced Xcode Cloud workflows:
If your script doesn't appear to be running when you expect it to,
double-check that you've named it correctly and placed it in a
ci_scripts folder alongside your project.
...
Lastly, it should be
noted that in a test action, multiple environments are used to build
and run your tests. Only the environment that is used for building
your tests will have your source code cloned into it by default. The
environments that run your tests won't have source code cloned into
them. They'll only have the ci_scripts folder made available on
them. As a result, the post-clone script won't run in these
environments and your custom scripts and any of their dependencies,
such as other shell scripts and small tools, must be entirely
contained within the ci_scripts folder.
Build script phases ARE allowed to run user-defined code as part of the build process, however, we can only run inlined custom scripts here. As discussed, external shell scripts have restricted permissions. Running the external shell script file after moving to ci_scripts does NOT work. e.g. "${PODS_ROOT}/../ci_scripts/AppCenter-xcframeworks.sh".
Although not relevant here, note that the environment that tests your project won't have the source code cloned into them.
For tests or other environments to reference custom script files, we need to store additional scripts inside the ci_scripts folder to ensure the action has access to it. Apple only allows 3 scripts to run corresponding to 3 stages of a build:
After cloning the source code into the build environment.
Before running xcodebuild.
After running xcodebuild.
Additional shell scripts can can ONLY run here after delegating from the respective ci_post_clone.sh, ci_pre_xcodebuild.sh or ci_post_xcodebuild.sh files in the ci_scripts folder.
Solution 1
My issue was running an external shell script during the build process. Apple does allow Run Script Build Phases in Xcode Cloud workflows, but they have to be inlined. So, I had to do 4 steps to run a custom Pod shell script as part of a build phase:
Refactor your Build Phases -> Run Script Phase script to inline the shell script file.
Check the project builds locally after clearing DerivedData.
Commit your code to GitHub / VCS.
Trigger the workflow.
Solution 2 (recommended, but not always possible)
Add the Swift Package version of the CocoaPod as a dependency following Apple's documentation.
Solution 3 (workaround)
Downgrade your CocoaPod to a version without external shell scripts.
Notes
As you can tell from the amount of effort required to workaround custom shell script build phases with Xcode Cloud, I suggest raising an issue on the specific CocoaPod repository to migrate away from custom shell script files. These kinds of steps make using Xcode Cloud very painful.
As Xcode Cloud adoption grows it is entirely possible that individual CocoaPods no longer reference custom shell script files. I don't see Apple opening up their infrastructure to enable arbitrary shell script execution because this is a security risk, and to be honest, should have been prevented on other CI providers too.
I can also see how hassles like these to include legacy CocoaPods dependencies could accelerate more projects to migrate to SPM. SPM is already popular, and will likely become more popular as Apple ensures first-class integration.
Disclaimer: Xcode Cloud is in private beta so this issue may be resolved in future versions if shell script permissions are relaxed...
I have an existing Symfony 5 project with a mysql database and a nginx Webserver. I wanna dockerize this project, but on the web I have found different opinions how to do that.
My plan is to write a multi-stage Docker file with at least a dev and a prod stage and let this build with docker-swarm. In my opinion it is useful to install the complete code during the build and having multiple composer.json files (one for every stage). In the web I have found opinions to not install the app new on every build but to copy the vendor and var folder to the container. Another opinion was to start the installation after the build process of the container is ready. But I think with that the service is not ready, when the app is successfully deployed.
What are you thinking is the best practice here?
Build exactly the same image for all environments
Do not build 2 different images for prod and dev. One of the main docker benefits is, that you can provide exactly the same environment for production and dev.
You should control your environment with ENV vars. for example, you can enable Xdebug for dev and disable it for prod.
Composer has option to install dev and production packages. You should use this feature.
If you decide to install some packages to dev. Try to use the same Dockerfile for both environment. Do not use Dockerfile.prod and Dockerfile.dev It will introduce some mess in the future.
Multistage build
You can do multistage build described in the official Docker documentation if your build environment requires much more dependencies than your runtime.
Example of it is compilation of the program. During compilation you need a lot of libraries, and you produce single binary. So your runtime does not need all dev libraries.
First stage can be build in second stage you just copy binary and it is.
Build all packages into the docker image
You should build your application when Docker image is building. All libraries and packages should be copied into image, you should not install them when the application is starting. Reasons:
Application starts faster when everything is installed
Some of the libraries can be changed in future or removed. You will be in troubles and probably you will spend a lot of time to do debugging.
Implement health checks
You should implement Health-Check. Applications require external dependencies, like Passwords, API KEY, some non sensitive data. Usually, We inject data with environment variables.
You should check if all required variables are passed, and have good format before your application is started. You can implement Health-Check or you can check it in the entrypoint.
Test your solution before it is released
You should implement mechanism of testing your images. For example in the CI:
Run your unit tests before image is built
Build the Docker image
Start new application image with dummy data. If you require PostgreSQL DB you can start another container,
Run integration tests.
Publish new version of the image only if all tests pass.
Imagine that I need to build a big Cuba application (it uses Gradle to manage dependencies and in the build produces a .war).
I need to dockerize both the build and the application. The latter is run in a Tomcat image in which the .war is copied.
The most of the dependencies remain actually unchanged between consecutive builds of the project, but the build seems to go over them each time, taking like forever...
I'd like to produce a custom Docker image from gradle:jdk8 (kinda) that imports all the Gradle dependencies.
his image will be used for consecutive builds to produce the .wars and will be rebuilt only when there is a change in the dependencies' versions.
Though, I'm quite new to Gradle and I don't know:
if it is possible to import the dependencies without building the project;
if it is actually possible to use previously imported dependencies to build the project in a shorter time.
Any advice/suggestion? Is this possible?
Hope my question is clear, but I have difficulties in explaining my aim. Ask me for better explication.
Thanks in advance.
You mean that you want to build a Docker image for build runner (or build agent), right?
It's not possible to import the dependencies without building the project, because Gradle resolves dependencies lazily, only when they are needed.
E.g. artifacts to build a CUBA theme are resolved only when web theme is built.
Yes, it's possible to re-use previously downloaded library artifacts (cached in ~/.gradle/caches) to build the project in a shorter time.
So in your case you need to create build runner's docker image by fully building your project once in a Docker container. Dependencies will be downloaded and cached in the file system. Then you can pull that image and use it again for subsequent builds, avoiding re-downloading artifacts.
If you change CUBA platform version in your project, you'll need to re-create the build runner image if you want to avoid downloading cuba-*.jar artifacts for every build.
Using csx scripts in Azure Functions I can use the Project.json file to install nuget packages, but when I'm using fsx scripts the packages aren't installed (the log console never shows the Starting NuGet restore message). The only way I found is installing locally and uploading the dependencies. Am I missing something?
I think that the current execution model for F# in Azure functions does not support project.json. There is a work in progress PR to improve F# support that will enable this.
For now, I think there are two options:
Install the packages locally and upload them to Azure (as you are doing)
If you're deploying via git, then I think the deployment lets you run deployment script (in the same way in which Azure WebSites let you run a deployment script).
I have not tested the second approach with Azure functions, but I think it could work. For example, see the F# Snippets' deployment script which calls a build script that starts by using Paket to restore dependencies. This way, you need just paket.bootstrapper.exe and paket.dependencies with paket.lock to specify your NuGet dependencies.