Can I ask what is the principle of dynamic config in temporal?
When changing content of dynamic config, how long matching services change config based on it really?
Dynamic config allows you to update parts of server configuration without having to restart your services. It includes a lot of fine-tuning knobs like rps limits, qps limits etc (more info here).
As far as the update interval, you can set it via your static config, see for example in default config template https://github.com/temporalio/temporal/blob/master/docker/config_template.yaml#L379-L381. Min value for update you can set ins 5s.
Related
I am migrating a micro-service system to Helm. The system has roughly 30 distinct deployments depending on an installation context. We are using Helm 3. Our currently layout is a three tier chart/subcharts organised by functionality that may or may not be required in a given context. The subcharts, when grouped in a 2nd-level subchart, usually need to be enabled/disabled together; so this is easy by disabling their parent in the top-level values file. However, there are some scenarios where grand-child charts depend on an uncle chart and I'm having difficulty finding an elegant solution to these situations.
What are strategies that have been used successfully in other charts?
Two scenarios that currently fall into this category for me are:
I would like to have a "feature flag" (global) that allows the installer to decide if a PVC should be create and mounted on applicable pods so that they can log into a central place for retrieval later (ELK, I know, I know...). If the flag is set then the PVC needs creation and the deployments will mount it. If not, then no PVC should be created and an empty dir used.
Some of the deployments use a technical "account" to communicate with each-other. So when these services are enabled, I'd like to create a secret with the username/password and run a Job to create the user in our identity provider. That same secret would then be added to the applicable deployments' environment variables. There are a handful of these technical accounts that are reused my multiple deployments. As such, I'd like to only create their secret and run the user creation job once.
Many thanks for any hints or helpful experience that you can send my way.
The documentation for the ActiveSupport::Cache::RedisCacheStore states:
Take care to use a dedicated Redis cache rather than pointing this at your existing Redis server. It won't cope well with mixed usage patterns and it won't expire cache entries by default.
Is this advice still true in general, especially when talking about custom data caches, not page (fragment) caches?
Or, more specifically, If I'm building a custom cache for specific "costly" backend calls to a slow 3rd-party API and I set an explicit expires_in value on my cache (or all my cached values), does this advice apply to me at all?
TLDR; yes, as long as you set an eviction policy. Which one? Read on.
On the same page, the docs for #new state that:
No expiry is set on cache entries by default. Redis is expected to be configured with an eviction policy that automatically deletes least-recently or -frequently used keys when it reaches max memory. See redis.io/topics/lru-cache for cache server setup.
This is more about memory management and access patterns than what's being cached. The Redis eviction policy documentation has a detailed section for policy choice and mixed usage (whether to use a single instance or not):
Picking the right eviction policy is important depending on the access pattern of your application, however you can reconfigure the policy at runtime while the application is running, and monitor the number of cache misses and hits using the Redis INFO output to tune your setup.
In general as a rule of thumb:
Use the allkeys-lru policy when you expect a power-law distribution in the popularity of your requests. That is, you expect a subset of elements will be accessed far more often than the rest. This is a good pick if you are unsure.
Use the allkeys-random if you have a cyclic access where all the keys are scanned continuously, or when you expect the distribution to be uniform.
Use the volatile-ttl if you want to be able to provide hints to Redis about what are good candidate for expiration by using different TTL values when you create your cache objects.
The volatile-lru and volatile-random policies are mainly useful when you want to use a single instance for both caching and to have a set of persistent keys. However it is usually a better idea to run two Redis instances to solve such a problem.
It is also worth noting that setting an expire value to a key costs memory, so using a policy like allkeys-lru is more memory efficient since there is no need for an expire configuration for the key to be evicted under memory pressure.
You do not have mixed usage. For example, you do not persist Sidekiq jobs in Redis, which have no TTL/expiry by default. So, you can treat your Redis instance as cache-only.
We want to fix all our spring cloud dataflow task and stream deployments to a particular set of nodes.
I have this working manually for a sample task eg
task launch test-timestamp --properties "deployer.*.kubernetes.deployment.nodeSelector=env:development"
(this wasn't obvious as the documentation here https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#configuration-kubernetes-deployer seems to imply the key is just nodeSelector not deployment.nodeSelector)
This correctly adds the node selector into the pod yaml for kubernetes.
But I want this to be set automatically ie using task.platform.kubernetes.accounts.default properties in the SCDF server config.
Ive tried:
task.platform.kubernetes.accounts.default.deployment.nodeSelector: env:development
and
task.platform.kubernetes.accounts.default.nodeSelector: env:development
but neither seem to work. What is the correct way to configure this?
Same question for stream deployments via skipper.
Also how do I set this up for scheduled tasks?
Sorry that you've had to try a few options to get to the bottom of finding the right deployer property that actually works.
In general, from SCDF's Shell/UI, the deployer token is a short-form for spring.cloud.deployer.kubernetes property. It's a repetitive thing to supply when you have more deployer properties to configure in a stream/task, so we have a short-form for that reason.
However, the nodeSelector is not a deployer-level property with a default, though. It is only available as a deployment level property, so that means, it is only available as an option for a per deployment basis.
To put it differently, it is not available as an option for "global" configuration, so that's why task.platform.kubernetes.accounts.default.deployment.nodeSelector: env:development is not taking into account. Same also is true for Streams through Skipper, as well.
It can be improved, though. I created spring-cloud/spring-cloud-deployer-kubernetes#300 for tracking - feel free to subscribe to the notifications. Both Streams and Tasks should then be able to take advantage of it as a global configuration. Once the PR is merged, you should be able to try it with SCDF's 2.2.0.BUILD-SNAPSHOT image.
As for the K8s-scheduler implementation, we do not have support for nodeSelectors yet. I created spring-cloud/spring-cloud-scheduler-kubernetes#25 - we could collaborate on a PR if you want to port the functionality from K8s-deployer.
so I have been building my application mostly as 12 factor app and now looking at the config part.
Right now as it stands I have separate config files for dev and production and through the build process we either build a dev or production image. The code is 100% the same the only thing that changes is the config.
Now I 100% understand that in a 12 factor app the config should come from external source such as: environment variables, or maybe a safe store like vault etc...
So what the various articles and blogs fail to mention about the config is the how is the config stored/processed. If the code is separated in it's own git repo and it has no config stored with it then how do we handle the config?
Do we store the actual config values on a separate git and then some how merge/push/execute those on the target environment (Kubernet config map, marathon JSON config, Vault, etc...) through the build process using some kind of trigger?
There is not a standard but what I've been observing is some common behaviors like:
Sensitive information never gets on versioning system, specially git which is a DCVS (you can clone the repo for other locations). If you don't follow, remember that our existing "security system" is based on the incapacity of read crypto info in a certain time, but in certain point you might be able to read the info. Usually on kubernetes I see operators, managing the service account across multiple namespaces and then other only referring the service account, tools like KMS, Cert manager, Vault and etc. are welcome
Configuration like env vars, endpoints, are stored and versioned with their own "lifecycle".
12factor does not meant to separate the configuration of your app from your repository, instead suggest not to put into your app (like on your container or even binary distribution).
In fact if you want to use a separate repo only for config you can do it, but if you want to put aside your project source code the configuration, you can do it as well. It is more a decision based on the size of the project, complexity, segregation of duties and team context. (IMHO)
On my case of study for instance, makes sense to separate config on a dedicated repository as production environment has more than 50 cluster, which one with their own isolation stack, also there are different teams managing their own services and using common backing services (db, api, streams...). In my opinion as long as things gets more complex and cross-shared, makes more sense to separate config on independent repository, as there are several teams and resources over multiple clusters.
The question is based on configurations explained in http://neo4j.com/docs/ogm-manual/current/reference/#reference:indexing
Though config.setAutoIndex("validate"); seem a good option. An accidental miss in applying config on prod could bring the system down.
So, is there a way to apply index if they are missing with less programmatic effort?
As that doc states, you can just configure ogm.properties with indexes.auto=validate to avoid having to do anything programmatically.
For simple systems where the schema in the Neo4j database is equal to what is declared in the domain model you could use assert.
However this is rarely the case. New update option coming in OGM 3.1 will be more usable (it won't drop existing indexes/constraints), but is still not enough for more complicated deployments (e.g . indexes not defined in the domain, multiple applications accessing the db, ...).
Is such cases you should manage the indexes either manually or using some tool, such as http://www.liquigraph.org/, and use validate option in your OGM application to verify all expected indexes are present.
When doing so you should check that
the schema update is not going to affect the existing load (e.g. plan for time with low load)
existing data match the new schema (e.g. properties for new constraints exist on all nodes)
indexes/constraints no longer needed are dropped