I have a question regarding airflow when running in AKS.
We have deployed airflow in AKS and have mounted Azure Fileshare to the airflow pods. We have using this fileshare for DAG Folder. However, there are huge transactions (at least 20K for every 5 minutes) from airflow to the FileShare which is incurring us lot of costs from Azure. FYI - billing on Azure fileshare is done based on the number of transactions and not the amount of size that we use.
Could you please let me know if this is an issue with having DAG folder in Fileshare? If so, is there any other approach that we can use. Have tried mounting managed disk (instead of fileshare) to the pods. But issue with managed disk is that we cant mount a disk to more than 1 pod.
Appreciate for the help/ideas.
Thanks
I was running into a similar issue, having 8k transactions every 5 minutes for just 3 DAGs. I got it down to about 800 transactions every 5 minutes by setting file_parsing_sort_mode to alphabetical.
https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#file-parsing-sort-mode
The default setting for this, which is modified_time would make the DAG processor retrieve the last modified time of the file from the fileshare on every loop. Weirdly, this action even triggers write operations which are more costly than read operations.
https://github.com/apache/airflow/blob/2d79d730d7ff9d2c10a2e99a4e728eb831194a97/airflow/dag_processing/manager.py#L982-L1008
Related
The project in which I am working develops a Java service that uses MarkLogic 9 in the backend.
We are running a Jenkins build server that executes (amongst others) several tests in MarkLogic written in XQuery.
For those tests MarkLogic is running in a docker container on the Jenkins host (which is running Ubuntu Linux).
The Jenkins host has 12 GB of RAM and 8 GB of swap configured.
Recently I have noticed that the MarkLogic instance running in the container uses a huge amount of RAM (up to 10 GB).
As there are often other build jobs running in parallel, the Jenkins starts to swap, sometimes even eating up all swap
so that MarkLogic reports it cannot get more memory.
Obviously, this situation leads to failed builds quite often.
To analyse this further I made some tests on my PC running Docker for Windows and found out that the MarkLogic tests
can be run successfully with 5-6 GB RAM. The MarkLogic logs show that it sees all the host memory and wants to use everything.
But as we have other build processes running on that host this behaviour is not desirable.
My question: is there any possibility to tell the MarkLogic to not use so much memory?
We are preparing the docker image during the build, so we could modify some configuration, but it has to be scripted somehow.
The issue of the container not detecting memory limit correctly has been identified, and should be addressed in a forthcoming release.
In the meantime, you might be able to mitigate the issue by:
changing the group cache sizing from automatic to manual and setting cache sizes appropriate for the allocated resources. There area variety of ways to set these configs, whether deploying and settings configs from ml-gradle project, making your own Manage API REST calls, or programmatically:
admin:group-set-cache-sizing
admin:group-set-compressed-tree-cache-partitions
admin:group-set-compressed-tree-cache-size
admin:group-set-expanded-tree-cache-partitions
admin:group-set-expanded-tree-cache-size
admin:group-set-list-cache-partitions
admin:group-set-list-cache-size
reducing the in-memory-limit
in memory limit specifies the maximum number of fragments in an in-memory stand. An in-memory stand contains the latest version of any new or changed fragments. Periodically, in-memory stands are written to disk as a new stand in the forest. Also, if a stand accumulates a number of fragments beyond this limit, it is automatically saved to disk by a background thread.
I know there are multiple threads discussing NFS mounted volumes and file modification watch issues. Since most of the discussions are old, some from 8 years ago, my goal here is to compile some and bring them up again to check what are the most recent solutions you guys have been using to handle those issues.
The core problem
Linux relies on inotify, a kernel subsystem, to generate events when files are modified (changed/deleted) and those events are most often used by developer tools to watch files to trigger some task. The core issue is that when you have a volume/folder shared via NFS protocol, it doesn't generate the events, and therefore the tools need to use polling methods instead of trigger based on events.
Polling methods often create multiple issues such as high CPU usage, delay to trigger tasks over file alterations, and so on.
Some of the watching tools:
Webpack watch: https://webpack.js.org/configuration/watch/
Guard: https://github.com/guard/guard
Popular threads
inotify with NFS
Why are inotify events different on an NFS mount?
File update in shared folder does not trigger inotify on Ubuntu
Triggering file changes over VirtualBox shared folders
Nice solution attempts
Vagrant file system notification forwarder plugin - ABANDONED AND NOT RELIABLE
vagrant-fsnotify - ABANDONED AND NOT RELIABLE
My current challenge
We run our dev env using macOS as host, Vagrant (provider Virtualbox) with Alpine Linux as the guest, and Docker containers for the services (Node, NGINX...), the setup is running smoothly for everything aside from when the frontend developers need to watch file modifications using webpack watch feature. It works with polling but with a delay of 3-10 seconds.
Any updates or any recommendations on how you solve this problem?
I've been able to work around this issue by setting the actimeo NFS option to 1. This reduces the length of time that NFS caches the file system attributes which seems to keep the host and guest in near sync. Webpack watch now picks up changes for me almost immediately.
Here is the VagrantFile setting I use to implement this NFS share. Note the mount_options setting
config.vm.synced_folder "./my_host_syncd_folder", "/guest/path", type: 'nfs', mount_options: ['actimeo=1']
So I have a working DASK/SLURM cluster of 4 raspberry Pis with a common NFS share, that I can run Python jobs succesfully.
However, I want to add some more arm devices to my cluster that do not support NFS mounts (Kernel module missing) so I wish to move to fuse based ftp mounts wiht CurlftpFS.
I have setup the mounts sucesfully with anonymous username and without any passwords and the common FTP share can be seen by all the nodes (just as before when it was an NFS share).
I can still run SLURM jobs (since they do not use the share) but when I try to run a DASK job the master node timesout complaining that no worker nodes could be started.
I am not sure what exactly is the problem, since the share it open to anyone for read/write access (e.g. logs and dask queue intermediate files).
Any ideas how I can troubleshoot this?
I don't believe anyone has a cluster like yours!
At a guess, the filesystem access via FUSE, ftp and the pi is much slower than the OS is expecting, and you are seeing the effects of low-level timeouts, i.e., from Dask's point of view it appears that files reads are failing. Dask needs access to storage for configuration and sometimes temporary files. You would want to make sure that these locations are on local storage or tuned off. However, if this is happening during import of modules, which you have on the shared drive by design, there may be no fixing it (python loads many small files during import). Why not use rsync to move the files to the nodes?
Currently I'm researching on how our dockerised microservices could be orchestrated on AWS.
The Fargate option of ECS looks promising eliminating the need of managing EC2 instances.
Although it's a surprisingly long time needed to start a "task" in Fargate, even for a simple one-container setup. A 60 seconds to 90 seconds are typical for our Docker app images. And I heard it may take even more time like minutes or so.
So the question is: while Docker containers typically may start in say seconds what is exactly a reason for such an overhead in Fargate case?
P.S. The search on related questions returns such options:
Docker image load/extract time
Load Balancer influence -
registering, healthchecks grace period etc
But even in simplest possible config with no Load Balancer deployed and assuming the Docker image is not cached in ECS, it is still at least ~2 times slower to start task with single Docker image in Fargate (~ 60 sec) than launch the same Docker image on bare EC2 instance (25 sec)
Yes takes a little longer but we can't generalize the startup time for fargate. You can reduce this time tweaking some settings.
vCPU is directly impacting the start up time, So you have to keep in mind that in bare EC2 instance you have complete vCPU at your disposal , while in cases of fargate you may be assigning portion of it.
Since AWS manages servers for you they have to do few underline things. Assigning the VM into your VPC to docker images download/extract, assigning IPs and running the container can take this much time.
It's a nice blog and at the end of following article you can find good practices.
Analyzing AWS Fargate
We are running our application on a DCOS cluster in Azure Container Service. Docker image of our marathon app is approx 7GB. I know this is against best practices but lets keep that debate aside for this question. We pull latest on worker nodes and it takes around 20 minutes, if no running container currently uses this image on a node, it gets deleted from that node by some cleanup routine job.
Is there a way to prevent this from happening?
Amount of time to wait before Docker containers are removed can be set using this flag (this is agent option)
--docker_remove_delay flag
--docker_remove_delay=VALUE The amount of time to wait before removing docker containers (e.g., 3days, 2weeks, etc). (default: 6hrs)
To prevent accidental deletion (or modification) of a resource. You can create a lock that would prevent users from deleting or modifying the resource while the lock is there (even if they have the permissions to delete\modify the resource).
For more details, refer "Lock resources to prevent unexpected changes".