Hi I have a problem that for some reason I get blocked form a private corporate repo from installing dependencies, it seems that due to the large amount of requests I generate when running bower install I get blocked... this is a known issue, and fixing it could take a couple of days.
I wonder if there is a way to stop - pause and ask for user to touch a key to continue the installation?
this way I would be able to overcome this problem that is out my scope fixing and I need to install the component to work. any hint?
I get this message
ECMDERR Failed to execute "git ls-remote --tags --heads
git#github.company.com:components/something.git", exit code of #128 client_loop:
send disconnect: Connection reset by peer fatal: Could not read from remote
repository. Please make sure you have the correct access rights and the
repository exists.
The repo exists and I do have access rights.
I have confirmed the cause of this issue is that private repo is wrongly cutting my connection due to the large amounts of requests, this triggers a brute force attack protection... the solution could take days to be completed in large company with processes that need to be initilizied with the corresponding teams.
A very tedious workaround is to install manually each package or to cut the bower.json package in parts and run them in separate batches.
Related
For reasons pertaining to storage and git, installing homebrew comes with the issue of:
Error:
homebrew-core is a shallow clone.
homebrew-cask is a shallow clone.
To `brew update`, first run:
git -C /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core fetch --unshallow
git -C /usr/local/Homebrew/Library/Taps/homebrew/homebrew-cask fetch --unshallow
These commands may take a few minutes to run due to the large size of the repositories.
This restriction has been made on GitHub's request because updating shallow
clones is an extremely expensive operation due to the tree layout and traffic of
Homebrew/homebrew-core and Homebrew/homebrew-cask. We don't do this for you
automatically to avoid repeatedly performing an expensive unshallow operation in
CI systems (which should instead be fixed to not use shallow clones). Sorry for
the inconvenience!
It explicitly states what to do next, but I've found that running those git commands is a pretty terrible experience with slow internet (or just in general). The absolute best solution to such an issue would be if there were torrents which contained the entire project folder including git. Barring that, a simple download would be nice. Really, anything but git is in scope of this question.
To reiterate the issues with using git:
requires git
opaque UX by default (tends to hang without any updates for long periods of time)
not great for slow connections
wrong tool for a situation that only involves downloading files
Is there an alternative path to Homebrew that doesn't incorporate git?
As said in the comments, you can use the skip tap cloning feature of Homebrew. Note that this is, for now, a beta feature:
Skip Tap Cloning (beta)
You can instruct Homebrew to skip cloning the
Homebrew/homebrew-core tap during installation by setting the beta
HOMEBREW_INSTALL_FROM_API environment variable with the following:
export HOMEBREW_INSTALL_FROM_API=1
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"
This will make Homebrew install formulae and casks from the
homebrew/core and homebrew/cask taps using Homebrew’s API instead of
local checkouts of these repositories.
I agree with OP 100%.
Homebrew should simply provide a downloadable file. Especially since GitHub has implemented a very unfriendly interface and the default shell in MacOS Ventura is now zsh which Homebrew specifically states will not work with their provided Terminal install command.
If you make it so difficult to install your software, then don't be surprised that fewer people use it.
Edit: I suppose they just don't really care, but I spent a few hours trying to get Homebrew up to date tonight because of GitHub.
I have just installed puppet server enterprise and successfully added a few nodes and got some custom modules running also. I am now wanting to move to Code Manager before we get too deep in it.
I have followed the instructions for creating an empty Bitbucket repo here and initializing it with one single file environment.conf on a production branch as described in that link.
I have then followed the steps here to configure Code Manager but when I get to Test the control repository section to test the connection with puppet-code deploy --dry-run I get the following error:
--dry-run implies --all.
--dry-run implies --wait.
Dry-run deploying all environments.
2021/12/21 20:21:12 ERROR - [POST /deploys][500] Errors while collecting a list of environments to deploy (exit code: 1).
"/opt/puppetlabs/puppet/lib/ruby/gems/2.7.0/gems/rugged-0.27.7/lib/rugged/repository.rb:258: warning: Using the last argument as keyword parameters is deprecated\nERROR\t -\u003e Unable to determine current branches for Git source 'puppet' (/etc/puppetlabs/code-staging/environments)\nOriginal exception:\nFailed to authenticate SSH session: Unable to send userauth-publickey request at /opt/puppetlabs/server/data/code-manager/git/git#git.company.com-1234-in-puppet-control-repo.git\n"
I have added the puppet server's SSH pub key to the bitbucket repo's access tokens.
There are a few things in that error message im not fully understanding.
Unable to determine current branches for Git source 'puppet' - What is meant by source 'puppet' - my repo is called puppet-control-repo...?
Failed to authenticate SSH session: Unable to send userauth-publickey request - My puppet master's SSH keys are in the token list for that repo so confused here also.
Any guidance would be appreciated.
UPDATE (13-01-2022):
I can successfully clone on puppet server using command
git clone ssh://git#git.example.com:1234/project/puppet-control-repo.git --config core.sshCommand="ssh -i /etc/puppetlabs/puppetserver/ssh/id-control_repo.rsa"
Note sure why puppet is still returning:
Failed to authenticate SSH session: Unable to send userauth-publickey request
I don't know if you saw the instructions here https://puppet.com/docs/pe/2021.4/control_repo.html#managing_environments_with_a_control_repository but you can run
puppet infrastructure configure
which makes sure the files have right permissions.
I would also test attempting a clone with keys works outside of code deploy
git clone -i /etc/puppetlabs/puppetserver/ssh/id-control_repo.rsa your_gir_url
If this works it may be worth being aware of an issue we experienced on github https://puppet.com/blog/how-githubs-protocol-changes-impact-your-puppet-code-deployments/ which depending on bitbuckets approach to protocal may be having a similar affect.
We are updating docs to recommend the usage of more secure keys ed25519 creating as per the article.
if a manual clone doesnt work it suggests bitbucket doesn't have your public key correctly
Also a more complete debugging command is
runuser -u pe-puppet -- /opt/puppetlabs/puppet/bin/r10k -c /opt/puppetlabs/server/data/code-manager/r10k.yaml deploy environment production --puppetfile --verbose debug2
FOLLOWUP
On investigation we found https://support.puppet.com/hc/en-us/articles/227829007 which showed ssh:// was required at the start of r10k_remote making an example command of ssh://git#bitbucket.org:davidsandilands/control-repo.git
I have requested updates to https://support.puppet.com/hc/en-us/articles/227829007 to highlight this is not a version confined issue and asked for the puppet code manager configuration docs to be updated to reflect this may be required.
I see that you have a .pub file in the ssh directory. I believe it's expecting a private key there.
Also do you have the master class set up to point to your repo inside of Puppet Enterprise web ui?
You'll want to set the following parameters on that class.
code_manager_auto_configure = true
r10k_private_key = $PRIVATE_KEY_IN_SSH_FOLDER_ABSOLUTE_PATH
r10k_remote = Your git URL
The PE Master can be found in Node Groups on the PE Web UI Node Groups -> PE Infrastructure -> PE Master
Thanks to #david-sandilands for helping me resolve this and guiding me to this article via the puppet community slack. Top guy!
EDIT 1:
The solution was documented here: https://support.puppet.com/hc/en-us/articles/227829007-Fix-your-Bitbucket-Stash-Code-Manager-configuration-in-Puppet-Enterprise-2015-3-to-2017-2
However the documentation was out of date as it affected version 2021.4 also.
In short:
r10k_remote = "ssh://git#git.company.com:1234/project/control-repo.git"
Not
r10k_remote = "git#git.company.com:1234/project/control-repo.git"
When working with Bitbucket Server.
EDIT 2:
Puppet have since updated their documentation:
https://puppet.com/docs/pe/2021.5/code_mgr_config.html#code_mgr_enable
A little confused at the moment. I've got docker on one my servers and as it doesn't have internet access, I'm trying to build a base image for centos7.4. The nice Docker site has a mkimage_yum.sh script for this purpose, but it consistently fails when it tries running:
yum -c /tmp/mkimage_yum.sh.gnagTv/etc/yum.conf --installroot=/tmp/mkimage_yum.sh.gnagTv -y clean all
with a "No enabled repos" error. The thing is, if I enter "yum repolist" I get back 17 entries, and I have manually tried to set several repos to enabled. Yet, this command still fails, and I do not understand what could be missing.
Anybody have some idea of what I can so this succeeds?
Jay
I figured out why this was failing, the docker file for mkimage_yum.sh does not contain the proper code if you're storing your repos in /etc/yum.repos.d, it assumes that everything is in /etc/yum.conf. This is really not correct, and it causes one of the later yum clean operations to fail. I fixed it, but I cannot upload the change as the server has no internet access.
When running npm install within circleci we fetch some node packages from our github repositories through package.json. This operation is happening when building a docker image from a Dockerfile.
This has been working great until last week when without changes in our side, we started to get errors while cloning these packages. To perform this operation, we were using Basic Authentication in the URL providing user credentials in it. For ie:
https://<username>:<password>#github.com/elektron-technogoly/<repository>.git
Now, we get the following errors:
npm ERR! Command failed: git clone ...
npm ERR! fatal: unable to access 'https://<username>:<password>#github.com/elektron-technogoly/<repository>.git':
Could not resolve host: <username>
From the error message it seems like it thinks the username is the host and thus, fails. I checked that password is still valid and it did not expire.
Has recently - around last week - something changed that could cause this error? Has Basic Authentication been disabled?
UPDATE: Playing a bit seems like when you change the base docker image (say from node:4-slim to node:4), the first time it works, subsequent times don't. Unfortunately, logs are not giving me any lead, both look exactly the same but the error appears from the first onwards.
This is my build.
https://travis-ci.org/gogo/protobuf
It intermittently fails for some of the builds.
I think it is struggling with installing a protocol buffer version using wget, but I can't see the logs, since they take forever to load.
It would be great if travis could tell me that it has failed to load the logs instead of just pretending to load them. Sorry I don't know if that is really the case, but that is how it feels.
Also I don't understand why this works some of the time and randomly fails. If the server is overloaded, put me in a queue, please don't fail when there is not something wrong with the code.
Please help I am new to travis, so maybe I am just doing it wrong.
Some of the other builds with the same use of PROTOBUF_VERSION are successful and show some output from the final step of install-protobuf.sh (./configure --prefix=/home/travis && make -j2 && make install). So similar to you, I suspect that the wget step in install-protobuf.sh is what is failing in the failed builds.
I would suggest editing install-protobuf.sh so that you can better see what is going on in the travis-ci logs:
change set command to: set -ex
remove the -q option from your use of wget in:
wget -q https://github.com/google/protobuf/releases/download/v$PROTOBUF_VERSION/$basename.tar.gz