Jenkins Scan Multibranch Pipeline fails Host key verification failed - jenkins

Question

Jenkins is failing with an SSH issue when it tries to scan the multibranch pipeline and ssh is giving "Host key verification failed":
[Wed Sep 21 01:10:09 PDT 2022] Starting branch indexing...
> git --version # timeout=10
> git --version # 'git version 2.31.1'
using GIT_SSH to set credentials jenkins#jenkinsserver SSH private key
[INFO] Currently running in a labeled security context
[INFO] Currently SELinux is 'enforcing' on the host
> /usr/bin/chcon --type=ssh_home_t /tmp/jenkins-gitclient-ssh7000755047282481393.key
Verifying host key using manually-configured host key entries
> git ls-remote --symref -- ssh://jenkins#jenkinsserver.corp.contoso.com/vol/git/cmb.git # timeout=10
ERROR: [Wed Sep 21 01:10:09 PDT 2022] Could not update folder level actions from source blueocean
[Wed Sep 21 01:10:09 PDT 2022] Finished branch indexing. Indexing took 0.13 sec
FATAL: Failed to recompute children of CertAccord
hudson.plugins.git.GitException: Command "git ls-remote --symref -- ssh://jenkins#jenkinsserver.corp.contoso.com/\
vol/git/cmb.git" returned status code 128:
stdout:
stderr: No ECDSA host key is known for jenkinsserver.corp.contoso.com and you have requested strict checking.
Host key verification failed.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2697)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2111)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2009)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2000)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.getRemoteSymbolicReferences(CliGitAPIImpl.java:3675)
at jenkins.plugins.git.AbstractGitSCMSource.retrieveActions(AbstractGitSCMSource.java:1152)
at jenkins.scm.api.SCMSource.fetchActions(SCMSource.java:848)
at jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:598)
at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:278)
at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:166)
at jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1032)
at hudson.model.ResourceController.execute(ResourceController.java:107)
at hudson.model.Executor.run(Executor.java:449)
Finished: FAILURE
I'm aware that this problem indicates the ssh client is receiving an ssh server signature that it doesn't recognize (e.g. it's not in known_hosts).
If I "su - jenkins" on the jenkinsserver I can manually run git and ssh without a problem:
$ git ls-remote --symref -- ssh://jenkins#jenkinsserver.corp.contoso.com/vol/git/cmb.git
ref: refs/heads/master HEAD
f79a54e2233749e0f0a9cf01 HEAD
... snip ...
$ ssh jenkinsserver.corp.contoso.com date
Wed Sep 21 01:45:20 PDT 2022
$ grep jenkinsserver ~/.ssh/known_hosts
jenkinsserver.corp.contoso.com ecdsa-sha2-nistp256 AAAAE2VjZHN....
This means that there is some other known_hosts file that Jenkins is somehow telling ssh to use. I just can't figure out where that file is!
Since this is a branch scan it's running on the jenkin's server, right?
The jenkinsserver is a RHEL 8.6 system. Jenkins version 2.361.1-1.1

Answer 1

Found a solution for this. In Manage Jenkins > Configure Global Security > Git Host Key Verification Configuration I changed it to "Accept first connection". It was set to "Manually provide keys".