I'm looking about application deployment with docker containers for production in some server (not hundreds).
I can see some deployment managers like docker-compose who deploy according to YAML service
description file.
Official docker-compose.yml example file:
web:
build: .
ports:
- "5000:5000"
volumes:
- .:/code
links:
- redis
redis:
image: redis
I'm looking about solution to manage/produce these YAML files and communicate with deployment managers like docker-compose.
This solution should permeit to manage Applications templates, deployeds instances of them, configuration of them, etc.
Illustration of it:
Docker
+-------------------+
docker-compose.yml | |
+---------------+ +-------+ | containers |
| APP manager |------->|Mysql_a| | +---------------+ |
| | |Mysql_b+-----------+ | |MySQL_a |Mysq| |
| MySQL Tpl | |Mysql_c| docker-compose | +---------------+ |
| Wordpress tpl | |Wp_a | | | |l_b |Mysql_c | |
| | +---+---+ | | +---------+-----+ |
| Mysql_a | | +------+ |Wp_a | | |
| Mysql_b +----------> | | | +---------+ | |
| Mysql_c | | | | | | |
| Wp_a | | | | | | |
+---------------+ | | | | | |
+---------------+ | +---------------+ |
+-------------------+
My thirst think is for panamax but is it approriate ? Whats other open source solutions exists ?
Related
when I using fastlane to publish ios app in github:
- name: Deploy to TestFlight/PGY
run: |
cd ./ios
bundle exec fastlane beta
env:
FLUTTER_ROOT: ${{ secrets.FLUTTER_ROOT }}
APPLE_ID: ${{ secrets.APPLE_ID }}
GIT_URL: ${{ secrets.GIT_URL }}
PGY_USER_KEY: ${{ secrets.PGY_USER_KEY }}
PGY_API_KEY: ${{ secrets.PGY_API_KEY }}
TEAM_ID: ${{ secrets.TEAM_ID }}
ITC_TEAM_ID: ${{ secrets.ITC_TEAM_ID }}
FASTLANE_USER: ${{ secrets.FASTLANE_USER }}
FASTLANE_PASSWORD: ${{ secrets.FASTLANE_PASSWORD }}
FASTLANE_APPLE_APPLICATION_SPECIFIC_PASSWORD: ${{ secrets.FASTLANE_APPLE_APPLICATION_SPECIFIC_PASSWORD }}
FASTLANE_SESSION: ${{ secrets.FASTLANE_SESSION }}
MATCH_PASSWORD: ${{ secrets.MATCH_PASSWORD }}
MATCH_KEYCHAIN_NAME: ${{ secrets.MATCH_KEYCHAIN_NAME }}
MATCH_KEYCHAIN_PASSWORD: ${{ secrets.MATCH_KEYCHAIN_PASSWORD }}
shows this error:
No code signing identity found and can not create a new one because you enabled `readonly`
this is the part of the log:
+-----------------------+---------+--------+
| Used plugins |
+-----------------------+---------+--------+
| Plugin | Version | Action |
+-----------------------+---------+--------+
| fastlane-plugin-pgyer | 0.2.2 | pgyer |
+-----------------------+---------+--------+
[14:58:29]: Sending anonymous analytics information
[14:58:29]: Learn more at https://docs.fastlane.tools/#metrics
[14:58:29]: No personal or sensitive data is sent.
[14:58:29]: You can disable this by adding `opt_out_usage` at the top of your Fastfile
[14:58:29]: ------------------------------
[14:58:29]: --- Step: default_platform ---
[14:58:29]: ------------------------------
[14:58:29]: Driving the lane 'ios beta' 🚀
[14:58:29]: --------------------------
[14:58:29]: --- Step: xcode_select ---
[14:58:29]: --------------------------
[14:58:29]: Setting Xcode version to /Applications/Xcode_12.4.app for all build steps
[14:58:29]: -----------------------------
[14:58:29]: --- Step: create_keychain ---
[14:58:29]: -----------------------------
[14:58:29]: Found keychain '~/Library/Keychains/***', creation skipped
[14:58:29]: If creating a new Keychain DB is required please set the `require_create` option true to cause the action to fail
[14:58:29]: $ security list-keychains -d user
[14:58:29]: ▸ "/Users/runner/Library/Keychains/***-db"
[14:58:29]: Found keychain '/Users/runner/Library/Keychains/***-db' in list-keychains, adding to search list skipped
[14:58:29]: -------------------
[14:58:29]: --- Step: is_ci ---
[14:58:29]: -------------------
[14:58:30]: -------------------
[14:58:30]: --- Step: match ---
[14:58:30]: -------------------
[14:58:30]: Successfully loaded '/Users/runner/work/flutter-netease-music/flutter-netease-music/ios/fastlane/Matchfile' 📄
+----------------+-----------------------------------------------------------------------------------------------------------------+
| Detected Values from './fastlane/Matchfile' |
+----------------+-----------------------------------------------------------------------------------------------------------------+
| git_url | *** |
| git_branch | master |
| storage_mode | git |
| type | adhoc |
| app_identifier | ["com.reddwarf.musicapp"] |
| username | *** |
+----------------+-----------------------------------------------------------------------------------------------------------------+
+--------------------------------+-----------------------------------------------------------------------------------------------------------------+
| Summary for match 2.191.0 |
+--------------------------------+-----------------------------------------------------------------------------------------------------------------+
| app_identifier | ["com.reddwarf.musicapp"] |
| git_url | *** |
| type | adhoc |
| readonly | true |
| keychain_name | *** |
| generate_apple_certs | true |
| skip_provisioning_profiles | false |
| username | *** |
| team_id | *** |
| storage_mode | git |
| git_branch | master |
| shallow_clone | false |
| clone_branch_directly | false |
| force | false |
| force_for_new_devices | false |
| skip_confirmation | false |
| skip_docs | false |
| platform | ios |
| derive_catalyst_app_identifier | false |
| fail_on_name_taken | false |
| skip_certificate_matching | false |
| skip_set_partition_list | false |
| verbose | false |
+--------------------------------+-----------------------------------------------------------------------------------------------------------------+
[14:58:30]: Cloning remote git repo...
[14:58:30]: If cloning the repo takes too long, you can use the `clone_branch_directly` option in match.
[14:58:30]: Checking out branch master...
[14:58:30]: 🔓 Successfully decrypted certificates repo
[14:58:30]: Couldn't find a valid code signing identity for distribution... creating one for you now
+---------------------------+-----------------------------------------------------+
| Lane Context |
+---------------------------+-----------------------------------------------------+
| DEFAULT_PLATFORM | ios |
| PLATFORM_NAME | ios |
| LANE_NAME | ios beta |
| KEYCHAIN_PATH | ~/Library/Keychains/*** |
| ORIGINAL_DEFAULT_KEYCHAIN | "/Users/runner/Library/Keychains/***-db" |
+---------------------------+-----------------------------------------------------+
[14:58:30]: No code signing identity found and can not create a new one because you enabled `readonly`
+------+------------------+-------------+
| fastlane summary |
+------+------------------+-------------+
| Step | Action | Time (in s) |
+------+------------------+-------------+
| 1 | default_platform | 0 |
| 2 | xcode_select | 0 |
| 3 | create_keychain | 0 |
| 4 | is_ci | 0 |
| 💥 | match | 0 |
+------+------------------+-------------+
[14:58:30]: fastlane finished with errors
what should I do to fix the problem?
I see that is_ci also ran. Does your match command look like this:
match(.., readonly: is_ci, ...) and are you running the command on a CI service like Jenkins or some other one?
If so, run it locally first, that will generate all the relevant certs and provisioning profiles needed. Then run it on your CI service again.
I am trying to run the clara train example, but when I execute the startClaraTrainNoteBooks.sh, the container cannot find the nvidia driver.
I already know that the script executes docker-compose.yml. So I tested whether docker-compose can found the nvidia driver:
services:
test:
image: nvidia/cuda:10.2-base
command: nvidia-smi
deploy:
resources:
reservations:
devices:
- driver: nvidia
capabilities: [gpu]
device_ids: ['0']
Output:
USER#test:~$ docker-compose up
WARNING: Found orphan containers (hp_nvsmi_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Starting hp_test_1 ... done
Attaching to hp_test_1
test_1 | Mon Jun 7 09:01:44 2021
test_1 | +-----------------------------------------------------------------------------+
test_1 | | NVIDIA-SMI 460.27.04 Driver Version: 460.27.04 CUDA Version: 11.2 |
test_1 | |-------------------------------+----------------------+----------------------+
test_1 | | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
test_1 | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
test_1 | | | | MIG M. |
test_1 | |===============================+======================+======================|
test_1 | | 0 GeForce RTX 206... Off | 00000000:01:00.0 Off | N/A |
test_1 | | 0% 34C P8 17W / 215W | 100MiB / 7979MiB | 0% Default |
test_1 | | | | N/A |
test_1 | +-------------------------------+----------------------+----------------------+
test_1 |
test_1 | +-----------------------------------------------------------------------------+
test_1 | | Processes: |
test_1 | | GPU GI CI PID Type Process name GPU Memory |
test_1 | | ID ID Usage |
test_1 | |=============================================================================|
test_1 | +-----------------------------------------------------------------------------+
hp_test_1 exited with code 0
But the startClaraTrainNoteBooks.sh cna not find it.
root#claratrain:/claraDevDay# nvidia-smi
root#claratrain:/claraDevDay#
Actually, startDocker.sh can find the driver.
root#c7c2d5597eb8:/claraDevDay# nvidia-smi
Mon Jun 7 09:11:43 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.27.04 Driver Version: 460.27.04 CUDA Version: 11.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce RTX 206... Off | 00000000:01:00.0 Off | N/A |
| 0% 35C P8 17W / 215W | 100MiB / 7979MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
+-----------------------------------------------------------------------------+
root#c7c2d5597eb8:/claraDevDay#
What should I do?
The docker-compose.yml script need to rewrite like this and working:
# SPDX-License-Identifier: Apache-2.0
version: "3.8"
services:
claratrain:
container_name: claradevday-pt
hostname: claratrain
##### use vanilla clara train docker
#image: nvcr.io/nvidia/clara-train-sdk:v4.0
##### to build image with GPU dashboard inside jupyter lab
build:
context: ./dockerWGPUDashboardPlugin/ # Project root
dockerfile: ./Dockerfile # Relative to context
image: clara-train-nvdashboard:v4.0
depends_on:
- tritonserver
ports:
- "3030:8888" # Jupyter lab port
- "3031:5000" # AIAA port
ipc: host
volumes:
- ${TRAIN_DEV_DAY_ROOT}:/claraDevDay/
- /raid/users/aharouni/data:/data/
command: "jupyter lab /claraDevDay --ip 0.0.0.0 --allow-root --no-browser --config /claraDevDay/scripts/jupyter_notebook_config.py"
# command: tail -f /dev/null
# tty: true
deploy:
resources:
reservations:
devices:
- driver: nvidia
capabilities: [ gpu ]
# To specify certain GPU uncomment line below
#device_ids: ['0,3']
#############################################################
tritonserver:
image: nvcr.io/nvidia/tritonserver:21.02-py3
container_name: aiaa-triton
hostname: tritonserver
restart: unless-stopped
command: >
sh -c "chmod 777 /triton_models &&
/opt/tritonserver/bin/tritonserver \
--model-store /triton_models \
--model-control-mode="poll" \
--repository-poll-secs=5 \
--log-verbose ${TRITON_VERBOSE}"
volumes:
- ${TRAIN_DEV_DAY_ROOT}/AIAA/workspace/triton_models:/triton_models
# shm_size: 1gb
# ulimits:
# memlock: -1
# stack: 67108864
# logging:
# driver: json-file
I have added Spring Security to an existing JEE application to add OAuth to the application.
The security configuration is set to protect the REST API, and that part seems to work fine.
When the UI requests a protected URL, the response contains a redirect to 'oauth2/authorize/keycloak'.
But that's where the story ends, since the request to 'oauth2/authorize/keycloak' itself returns a 404.
I am pretty out of date with spring security (have used it the last time with Spring applications about 8 years ago) and I have no idea where I am supposed to find the implementation of the endpoint 'oauth2/authorize/keycloak' in order to figure out what is missing or wrong in my setup.
The relevant part of my dependency tree looks as follows:
[INFO] | +- com.mycompany.auth:authentication-sso-configuration:jar:1.0.0-SNAPSHOT:compile
[INFO] | | +- org.reactivestreams:reactive-streams:jar:1.0.3:compile
[INFO] | | +- org.springframework.security:spring-security-oauth2-client:jar:5.3.3.RELEASE:compile
[INFO] | | | +- com.nimbusds:oauth2-oidc-sdk:jar:7.5:compile
[INFO] | | | | +- com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:compile
[INFO] | | | | +- com.nimbusds:content-type:jar:2.0:compile
[INFO] | | | | +- net.minidev:json-smart:jar:2.3:compile (version selected from constraint [1.3.1,2.3])
[INFO] | | | | | \- net.minidev:accessors-smart:jar:1.2:compile
[INFO] | | | | | \- org.ow2.asm:asm:jar:5.0.4:compile
[INFO] | | | | \- com.nimbusds:lang-tag:jar:1.4.4:compile
[INFO] | | | +- org.springframework.security:spring-security-oauth2-core:jar:5.3.3.RELEASE:compile
[INFO] | | | \- org.springframework:spring-core:jar:5.2.6.RELEASE:compile
[INFO] | | | \- org.springframework:spring-jcl:jar:5.2.6.RELEASE:compile
[INFO] | | +- org.springframework.security:spring-security-oauth2-jose:jar:5.3.3.RELEASE:compile
[INFO] | | | \- com.nimbusds:nimbus-jose-jwt:jar:8.18.1:compile
[INFO] | | +- org.springframework.security:spring-security-oauth2-resource-server:jar:5.3.3.RELEASE:compile
[INFO] | | +- org.springframework.security:spring-security-core:jar:5.3.3.RELEASE:compile
[INFO] | | | +- org.springframework:spring-aop:jar:5.2.6.RELEASE:compile
[INFO] | | | +- org.springframework:spring-beans:jar:5.2.6.RELEASE:compile
[INFO] | | | +- org.springframework:spring-context:jar:5.2.6.RELEASE:compile
[INFO] | | | \- org.springframework:spring-expression:jar:5.2.6.RELEASE:compile
[INFO] | | +- org.springframework.security:spring-security-web:jar:5.3.3.RELEASE:compile
[INFO] | | | \- org.springframework:spring-web:jar:5.2.6.RELEASE:compile
[INFO] | | +- org.springframework.security:spring-security-config:jar:5.3.3.RELEASE:compile
[INFO] | | +- org.springframework.security:spring-security-saml2-service-provider:jar:5.3.3.RELEASE:compile
[INFO] | | | +- org.opensaml:opensaml-core:jar:3.4.5:compile
[INFO] | | | | +- io.dropwizard.metrics:metrics-core:jar:3.1.2:compile
[INFO] | | | | \- net.shibboleth.utilities:java-support:jar:7.5.1:compile
[INFO] | | | +- org.opensaml:opensaml-saml-api:jar:3.4.5:compile
[INFO] | | | | +- org.opensaml:opensaml-xmlsec-api:jar:3.4.5:compile
[INFO] | | | | | \- org.opensaml:opensaml-security-api:jar:3.4.5:compile
[INFO] | | | | +- org.opensaml:opensaml-soap-api:jar:3.4.5:compile
[INFO] | | | | +- org.opensaml:opensaml-messaging-api:jar:3.4.5:compile
[INFO] | | | | +- org.opensaml:opensaml-profile-api:jar:3.4.5:compile
[INFO] | | | | \- org.opensaml:opensaml-storage-api:jar:3.4.5:compile
[INFO] | | | \- org.opensaml:opensaml-saml-impl:jar:3.4.5:compile
[INFO] | | | +- org.opensaml:opensaml-security-impl:jar:3.4.5:compile
[INFO] | | | +- org.opensaml:opensaml-xmlsec-impl:jar:3.4.5:compile
[INFO] | | | | \- org.apache.santuario:xmlsec:jar:2.0.10:compile
[INFO] | | | | \- com.fasterxml.woodstox:woodstox-core:jar:5.0.3:compile
[INFO] | | | | \- org.codehaus.woodstox:stax2-api:jar:3.1.4:compile
[INFO] | | | +- org.opensaml:opensaml-soap-impl:jar:3.4.5:compile
[INFO] | | | \- org.apache.velocity:velocity:jar:1.7:compile
[INFO] | | +- org.apache.logging.log4j:log4j-api:jar:2.13.3:compile
[INFO] | | +- org.apache.logging.log4j:log4j-core:jar:2.13.3:compile
[INFO] | | +- org.yaml:snakeyaml:jar:1.26:compile
[INFO] | | +- commons-collections:commons-collections:jar:3.2.2:compile
[INFO] | | +- org.bouncycastle:bcprov-jdk15on:jar:1.66:compile
[INFO] | | +- org.cryptacular:cryptacular:jar:1.2.4:compile
[INFO] | | \- org.apache.commons:commons-configuration2:jar:2.7:compile
[INFO] | | \- org.apache.commons:commons-text:jar:1.8:compile
And this is the configuration for OAuth
# OAuth2 login manifest
oauth2Login:
authorizationCode:
authorizationUri: "http://localhost:8180/auth/realms/master/protocol/openid-connect/auth"
scope:
- "openid"
- "finx"
redirectUriTemplate: "{baseUrl}/login/oauth2/code/{registrationId}"
tokenUri: "http://localhost:8180/auth/realms/master/protocol/openid-connect/token"
userInfoUri: "http://localhost:8180/auth/realms/master/protocol/openid-connect/userinfo"
jwkSetKeyUri: "http://localhost:8180/auth/realms/master/protocol/openid-connect/certs"
registrationId: "keycloak"
clientId: "finx_oauth2"
clientSecret:
vaultType: PLAIN_TEXT
secret: "my-secret"
clientName: "FinX"
entryPoints:
- pathMatcher: "/ledger-api/**"
- pathMatcher: "/ledger-api-internal/**"
- pathMatcher: "/ledger-api-ui/**"
# OAuth2 resource server
oauth2ResourceServer:
keySetUri: "http://localhost:8180/auth/realms/master/protocol/openid-connect/certs"
pathMatchers:
- "/api/**"
- "/orchestration-api/**"
I have been digging through the spring source code in order to find the implementation of the endpoint 'oauth2/authorize/keycloak', but this is not an easy task.
So looking for someone who can help me with some pointers on what could be missing/wrong in my configuration.
By default, the OAuth 2.0 Login Page is auto-generated by the DefaultLoginPageGeneratingFilter.
The login page for a client defaults to this: OAuth2AuthorizationRequestRedirectFilter.DEFAULT_AUTHORIZATION_REQUEST_BASE_URI + "/{registrationId}". As per your configuration, registrationId: "keycloak", this means (/oauth2/authorization/keycloak).
Please check your WebSecurityConfigurerAdapter configuration. Try to override the default login page by configuring oauth2Login().loginPage() and (optionally) oauth2Login().authorizationEndpoint().baseUri().
The following listing shows an example:
#Override
protected void configure(HttpSecurity http) throws Exception {
http
.oauth2Login()
.loginPage("/login/oauth2")
...
.authorizationEndpoint()
.baseUri("/login/oauth2/authorization")
....
}
Please check OAuth 2.0 Login - Advanced Configuration for more information.
It looks like heroku pg:push has put all my data into a schema named for my local db, and not in public where they'd be accessible to my app.
How do I fix this?
Schema | Name | Type | Owner
--------------------+------------------------------+-------+----------------
information_schema | sql_features | table | postgres
information_schema | sql_implementation_info | table | postgres
information_schema | sql_languages | table | postgres
information_schema | sql_packages | table | postgres
information_schema | sql_parts | table | postgres
information_schema | sql_sizing | table | postgres
information_schema | sql_sizing_profiles | table | postgres
lorax_development | ac_coa | table | iykrnaofpnlzod
lorax_development | acbkacct | table | iykrnaofpnlzod
lorax_development | accommodation | table | iykrnaofpnlzod
lorax_development | accommodation_copy | table | iykrnaofpnlzod
lorax_development | address | table | iykrnaofpnlzod
lorax_development | advert | table | iykrnaofpnlzod
lorax_development | affiliation | table | iykrnaofpnlzod
lorax_development | agency | table | iykrnaofpnlzod
lorax_development | always | table | iykrnaofpnlzod
lorax_development | answer | table | iykrnaofpnlzod
lorax_development | ar_internal_metadata | table | iykrnaofpnlzod
lorax_development | best_month | table | iykrnaofpnlzod
lorax_development | bklin | table | iykrnaofpnlzod
...
I'm installing a Gitlab instance with docker-compose on a server machine on my local network, and I'd like to access to my Gitlab instance from anywhere in my local network by visiting for example "https://my-hostname"
I follow this.
I'm running:
web:
image: 'gitlab/gitlab-ce:latest'
restart: always
hostname: 'gitlab.example.com'
environment:
GITLAB_OMNIBUS_CONFIG: |
external_url 'https://gitlab.example.com'
# Add any other gitlab.rb configuration here, each on its own line
ports:
- '7780:80'
- '7443:443'
- '7722:22'
volumes:
- '/srv/gitlab/config:/etc/gitlab'
- '/srv/gitlab/logs:/var/log/gitlab'
- '/srv/gitlab/data:/var/opt/gitlab'
Now I have very (very) limited network knowledge, so basically, how do I access to my running gitlab instance ? When I go to the local network IP of my host, my browser tells me that it can't connect.
Here is what I'm hoping to achieve:
LOCAL NETWORK
+--------------------------------------------------------------------------+
| |
| +--------------------+ |
| | My_Server | |
| | | |
| | +----------------+ | |
| | | | | "https://my-hostname" +-------------------+ |
| | | Docker: Gitlab | <------------------------+ My_Client | |
| | | | | +-------------------+ |
| | +----------------+ | |
| | | |
| +--------------------+ |
| |
+--------------------------------------------------------------------------+
The ports part of your configuration maps the host's ports to the container's ports.
So if you have
ports:
- '7780:80'
- '7443:443'
- '7722:22'
that is redirecting port 7780 on your host to port 80 on your container, and so forth. You should be able to access your container's services (via its local IP address, and then its hostname via local DNS) with this knowledge.