Permission denied on Jenkins job - jenkins

I'm running pipeline job with this pipeline script
node {
stage ('Prepare Environment') {
checkout scm
sh 'npm install'
}
stage('Build') {
sh 'docker-compose --verbose up -d'
}
}
and there is an error on docker-compose
compose.config.config.find: Using configuration files: ./docker-compose.yml
docker.auth.find_config_file: Trying paths: ['/var/lib/jenkins/.docker/config.json', '/var/lib/jenkins/.dockercfg']
docker.auth.find_config_file: No config file found
Traceback (most recent call last):
File "bin/docker-compose", line 6, in <module>
File "compose/cli/main.py", line 71, in main
File "compose/cli/main.py", line 121, in perform_command
File "compose/cli/command.py", line 40, in project_from_options
File "compose/cli/command.py", line 118, in get_project
File "compose/cli/command.py", line 93, in get_client
File "site-packages/docker/api/daemon.py", line 177, in version
File "site-packages/docker/utils/decorators.py", line 46, in inner
File "site-packages/docker/api/client.py", line 191, in _get
File "site-packages/requests/sessions.py", line 521, in get
File "site-packages/requests/sessions.py", line 508, in request
File "site-packages/requests/sessions.py", line 618, in send
File "site-packages/requests/adapters.py", line 490, in send
requests.exceptions.ConnectionError: ('Connection aborted.', error(13, 'Permission denied'))
Also I've tried to follow this command in same machine with jenkins user in the same directory, it's work fine with no error.

It seems a permission problem, maybe your docker-compose.yml file (or the binary file) is readable only by jenkins user.

It seem there is problem with jenkins job executor don't reload shell permission even I've restart jenkins with /restart multiple times.
But restart jenkins deamon using systemctl restart jenkins will do the job.

Related

frida.TimedOutError: unexpectedly timed out while initializing suspended process

Describe the bug
Timeout occurred on objection explore
To Reproduce
Steps to reproduce the behavior:
Run command objection --gadget "com.apple.AppStore" explore
Evidence / Logs / Screenshots
Using USB device `iPhone`
Traceback (most recent call last):
File "/usr/local/bin/objection", line 8, in <module>
sys.exit(cli())
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/objection/console/cli.py", line 114, in explore
agent.inject()
File "/usr/local/lib/python3.9/site-packages/objection/utils/agent.py", line 202, in inject
session = self.get_session()
File "/usr/local/lib/python3.9/site-packages/objection/utils/agent.py", line 169, in get_session
self.session = self.device.attach(self.spawned_pid)
File "/usr/local/lib/python3.9/site-packages/frida/core.py", line 76, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/frida/core.py", line 800, in attach
return Session(self._impl.attach(self._pid_of(target), **kwargs)) # type: ignore
frida.TimedOutError: unexpectedly timed out while initializing suspended process
Environment (please complete the following information):
Device: iPhone 7
OS: 15.3.1
Frida Version: 16.0.2
Objection Version: 1.11.0
Thanks to #Robert solved with pid choice. There are two important things here:
Don't use Apple-Slicon(m1/m2) MacOS
Use process-id instead of process-name. objection -g pid explore

Beam Dataflow python job fails in dist_proc/dax/shuffle/batch/chunking_shuffle_writer.cc#146

I have a Dataflow job that fails repeatedly with the following log message:
022-03-27T01:39:21.871476411ZAn exception was raised when trying to execute the workitem 1257504293434498471 : Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/dataflow_worker/batchworker.py", line 646, in do_work work_executor.execute() File "/usr/local/lib/python3.7/site-packages/dataflow_worker/executor.py", line 210, in execute op.finish() File "dataflow_worker/shuffle_operations.py", line 171, in dataflow_worker.shuffle_operations.ShuffleWriteOperation.finish File "dataflow_worker/shuffle_operations.py", line 172, in dataflow_worker.shuffle_operations.ShuffleWriteOperation.finish File "dataflow_worker/shuffle_operations.py", line 174, in dataflow_worker.shuffle_operations.ShuffleWriteOperation.finish File "/usr/local/lib/python3.7/site-packages/dataflow_worker/shuffle.py", line 593, in exit self.writer.Close() File "third_party/windmill/shuffle/python/shuffle_client.pyx", line 185, in shuffle_client.PyShuffleWriter.Close OSError: Shuffle close failed: b'FAILED_PRECONDITION: Precondition check failed. [type.googleapis.com/util.MessageSetPayload='[dist_proc.dax.internal.TrailProto] { trail_point { source_file_loc { filepath: "dist_proc/dax/shuffle/batch/chunking_shuffle_writer.cc" line: 146 } } trail_point { source_file_loc { filepath: "dist_proc/dax/shuffle/batch/chunking_shuffle_writer.cc" line: 102 } } }']'
I tried running the job a few times, and it fails in the same place each time. Other than that error and the set of DoFns that were running, I don't have many clues about where to look for problems.
I suspect it's my code that causing a problem. Would love advice on how to diagnose this.
Have you tried using runner v2?

Cannot use sqlite with the LocalExecutor [AIrflow]

I am trying to restart the airflow scheduler using the following command
airflow scheduler
I am using docker. I went inside my docker image for airflow and opened the CLI for my airflow image. That is where I used this command.
It throws an exception
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 25, in <module>
from airflow.configuration import conf
File "/usr/local/lib/python3.6/site-packages/airflow/__init__.py", line 31, in <module>
from airflow.utils.log.logging_mixin import LoggingMixin
File "/usr/local/lib/python3.6/site-packages/airflow/utils/__init__.py", line 24, in <module>
from .decorators import apply_defaults as _apply_defaults
File "/usr/local/lib/python3.6/site-packages/airflow/utils/decorators.py", line 36, in <module>
from airflow import settings
File "/usr/local/lib/python3.6/site-packages/airflow/settings.py", line 37, in <module>
from airflow.configuration import conf, AIRFLOW_HOME, WEBSERVER_CONFIG # NOQA F401
File "/usr/local/lib/python3.6/site-packages/airflow/configuration.py", line 731, in <module>
conf.read(AIRFLOW_CONFIG)
File "/usr/local/lib/python3.6/site-packages/airflow/configuration.py", line 421, in read
self._validate()
File "/usr/local/lib/python3.6/site-packages/airflow/configuration.py", line 213, in _validate
self._validate_config_dependencies()
File "/usr/local/lib/python3.6/site-packages/airflow/configuration.py", line 247, in _validate_config_dependencies
self.get('core', 'executor')))
airflow.exceptions.AirflowConfigException: error: cannot use sqlite with the LocalExecutor
I am looking for any way to restart the airflow scheduler.
This is expected.
Since sqlite doesn’t support multiple connections it can only be used with SequentialExecutor. This is also explained in the docs.
If you want to use LocalExecutor please set MySQL or PostgreSQL as backend.

Superset Oauth Integration config using Ambari error

I am trying to config OAUTH_PROVIDERS using ambari
File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SUPERSET/package/scripts/superset.py", line 184, in <module>
Superset().execute()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute
method(env)
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 971, in restart
self.stop(env, upgrade_type=upgrade_type)
File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SUPERSET/package/scripts/superset.py", line 133, in stop
self.configure(env, upgrade_type=upgrade_type)
File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SUPERSET/package/scripts/superset.py", line 90, in configure
user=params.superset_user)
File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
self.env.run()
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 263, in action_run
returns=self.resource.returns)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner
result = function(command, **kwargs)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call
tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy, returns=returns)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 314, in _call
raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'source /usr/hdp/current/superset/conf/superset-env.sh ; /usr/hdp/current/superset/bin/superset db upgrade' returned 1. Loaded your LOCAL configuration at [/usr/hdp/current/superset/conf/superset_config.py]
Traceback (most recent call last):
File "/usr/hdp/current/superset/bin/superset", line 12, in <module>
from superset.cli import manager
File "/usr/hdp/3.1.4.0-315/superset/lib/python3.6/site-packages/superset/__init__.py", line 180, in <module>
update_perms=utils.get_update_perms_flag(),
File "/usr/hdp/3.1.4.0-315/superset/lib/python3.6/site-packages/flask_appbuilder/base.py", line 135, in __init__
self.init_app(app, session)
File "/usr/hdp/3.1.4.0-315/superset/lib/python3.6/site-packages/flask_appbuilder/base.py", line 156, in init_app
self.sm = self.security_manager_class(self)
File "/usr/hdp/3.1.4.0-315/superset/lib/python3.6/site-packages/flask_appbuilder/security/sqla/manager.py", line 39, in __init__
super(SecurityManager, self).__init__(appbuilder)
File "/usr/hdp/3.1.4.0-315/superset/lib/python3.6/site-packages/flask_appbuilder/security/manager.py", line 199, in __init__
provider_name = _provider['name']
TypeError: string indices must be integers
I am able to setup superset Oauth without Ambari, but struggling to make the config in Ambari because even if make change in superset_cofig.py Ambari is overwriting the superset_cofig.py when we restart the service.
I never used Superset with Ambari, but I am currently struggling with it for standalone usage due to the lack of proper documentation and practical use cases.
To my understanding, in order to read the superset_config.py you need to export the PYTHONPATH and point down to the folder where the config is placed.
For example: export PYTHONPATH=/<folder where the config is placed>/:$PYTHONPATH
If you get this right, you should see in Superset's logs something like this
Loaded your LOCAL configuration at [/<folder where the config is>/superset_config.py]

Pipeline keeps running because of Dataflow runner not closing file system writer. NotImplementedError

I'm running a Dataflow job (Apache Beam 2.12.0) using python on the Google Cloud Platform. The pipeline is not terminating and keeps running.
The issue is the same to
https://issues.apache.org/jira/browse/BEAM-7266
It wasn't solved and says "open when met again". It also says that file writer is not closing.
There's only one error log:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 649, in do_work
work_executor.execute()
File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/executor.py", line 178, in execute
op.finish()
File "dataflow_worker/native_operations.py", line 93, in dataflow_worker.native_operations.NativeWriteOperation.finish
def finish(self):
File "dataflow_worker/native_operations.py", line 94, in dataflow_worker.native_operations.NativeWriteOperation.finish
with self.scoped_finish_state:
File "dataflow_worker/native_operations.py", line 95, in dataflow_worker.native_operations.NativeWriteOperation.finish
self.writer.__exit__(None, None, None)
File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/nativefileio.py", line 465, in __exit__
self.file.close()
File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/filesystemio.py", line 202, in close
self._uploader.finish()
File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/gcsio.py", line 606, in finish
raise self._upload_thread.last_error # pylint: disable=raising-bad-type
NotImplementedError: offset: 0, whence: 0, position: 51518, last: 0

Resources