Github webhook no longer triggering Jenkins builds - jenkins

Question

We've recently noticed our Jenkins builds stopped getting triggered automatically. After further investigation there were numerous issues.
Github webhooks were unsuccessful with Github reporting "Couldn't connect to the server" in Github's webhook configuration UI. I can confirm our ELB and EC2 instance hosting Jenkins is live and healthy. No DNS changes have been made here.
Jenkins logs report various failures:
Invalid credentials despite having a valid Jenkins username & password credentials (password is a personal API token):
There is no credentials with admin access to manage hooks on GitHubRepositoryName[host=github.com,username=REDACTED,repository=REDACTED]
Failed to delete post-commit hook:
ALPN callback dropped: SPDY and HTTP/2 are disabled. Is alpn-boot on the boot class path?
Apr 12, 2019 6:15:43 PM WARNING org.jenkinsci.plugins.github.webhook.WebhookManager$2 applyNullSafe
Failed to add GitHub webhook for GitHubRepositoryName[host=github.com,username=REDACTED,repository=REDACTED]
java.io.FileNotFoundException: https://api.github.com/repos/REDACTED/REDACTED/hooks/101704125
at com.squareup.okhttp.internal.huc.HttpURLConnectionImpl.getInputStream(HttpURLConnectionImpl.java:243)
at com.squareup.okhttp.internal.huc.DelegatingHttpsURLConnection.getInputStream(DelegatingHttpsURLConnection.java:210)
at com.squareup.okhttp.internal.huc.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:25)
at org.kohsuke.github.Requester.parse(Requester.java:617)
at org.kohsuke.github.Requester.parse(Requester.java:599)
at org.kohsuke.github.Requester._to(Requester.java:277)
Caused: org.kohsuke.github.GHFileNotFoundException: {"message":"Not Found","documentation_url":"https://developer.github.com/v3/repos/hooks/#delete-a-hook"}
at org.kohsuke.github.Requester.handleApiError(Requester.java:691)
at org.kohsuke.github.Requester._to(Requester.java:298)
at org.kohsuke.github.Requester.to(Requester.java:239)
at org.kohsuke.github.Requester.to(Requester.java:227)
at org.kohsuke.github.GHHook.delete(GHHook.java:56)
at org.jenkinsci.plugins.github.webhook.WebhookManager$10.applyNullSafe(WebhookManager.java:344)
Caused: org.kohsuke.github.GHException: Failed to delete post-commit hook
at org.jenkinsci.plugins.github.webhook.WebhookManager$10.applyNullSafe(WebhookManager.java:347)
at org.jenkinsci.plugins.github.webhook.WebhookManager$10.applyNullSafe(WebhookManager.java:341)
at org.jenkinsci.plugins.github.util.misc.NullSafePredicate.apply(NullSafePredicate.java:19)
at com.google.common.collect.Iterators$7.computeNext(Iterators.java:649)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at com.google.common.collect.Iterators$7.computeNext(Iterators.java:647)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at com.google.common.collect.Lists.newArrayList(Lists.java:138)
at com.google.common.collect.Lists.newArrayList(Lists.java:119)
at org.jenkinsci.plugins.github.util.FluentIterableWrapper.toList(FluentIterableWrapper.java:147)
at org.jenkinsci.plugins.github.webhook.WebhookManager$2.applyNullSafe(WebhookManager.java:202)
at org.jenkinsci.plugins.github.webhook.WebhookManager$2.applyNullSafe(WebhookManager.java:175)
at org.jenkinsci.plugins.github.util.misc.NullSafeFunction.apply(NullSafeFunction.java:18)
at com.google.common.collect.Iterators$8.next(Iterators.java:812)
at com.google.common.collect.Iterators$7.computeNext(Iterators.java:648)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at com.google.common.collect.Iterators$7.computeNext(Iterators.java:647)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at com.google.common.collect.Lists.newArrayList(Lists.java:138)
at com.google.common.collect.Lists.newArrayList(Lists.java:119)
at org.jenkinsci.plugins.github.util.FluentIterableWrapper.toList(FluentIterableWrapper.java:147)
at org.jenkinsci.plugins.github.webhook.WebhookManager$1.run(WebhookManager.java:127)
at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:119)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Failed to create hook:
Apr 12, 2019 6:15:44 PM WARNING org.jenkinsci.plugins.github.webhook.WebhookManager$2 applyNullSafe
Failed to add GitHub webhook for GitHubRepositoryName[host=github.com,username=REDACTED,repository=REDACTED]
java.io.FileNotFoundException: https://api.github.com/repos/REDACTED/REDACTED/hooks
at com.squareup.okhttp.internal.huc.HttpURLConnectionImpl.getInputStream(HttpURLConnectionImpl.java:243)
at com.squareup.okhttp.internal.huc.DelegatingHttpsURLConnection.getInputStream(DelegatingHttpsURLConnection.java:210)
at com.squareup.okhttp.internal.huc.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:25)
at org.kohsuke.github.Requester.parse(Requester.java:617)
at org.kohsuke.github.Requester.parse(Requester.java:599)
at org.kohsuke.github.Requester._to(Requester.java:277)
Caused: org.kohsuke.github.GHFileNotFoundException: {"message":"Validation Failed","errors":[{"resource":"Hook","code":"custom","message":"Hook already exists on this repository"}],"documentation_url":"https://developer.github.com/v3/repos/hooks/#create-a-hook"}
at org.kohsuke.github.Requester.handleApiError(Requester.java:691)
at org.kohsuke.github.Requester._to(Requester.java:298)
at org.kohsuke.github.Requester.to(Requester.java:239)
at org.kohsuke.github.GHHooks$Context.createHook(GHHooks.java:49)
at org.kohsuke.github.GHRepository.createHook(GHRepository.java:1206)
at org.jenkinsci.plugins.github.webhook.WebhookManager$9.applyNullSafe(WebhookManager.java:329)
Caused: org.kohsuke.github.GHException: Failed to create hook
at org.jenkinsci.plugins.github.webhook.WebhookManager$9.applyNullSafe(WebhookManager.java:331)
at org.jenkinsci.plugins.github.webhook.WebhookManager$9.applyNullSafe(WebhookManager.java:316)
at org.jenkinsci.plugins.github.util.misc.NullSafeFunction.apply(NullSafeFunction.java:18)
at org.jenkinsci.plugins.github.webhook.WebhookManager$2.applyNullSafe(WebhookManager.java:204)
at org.jenkinsci.plugins.github.webhook.WebhookManager$2.applyNullSafe(WebhookManager.java:175)
at org.jenkinsci.plugins.github.util.misc.NullSafeFunction.apply(NullSafeFunction.java:18)
at com.google.common.collect.Iterators$8.next(Iterators.java:812)
at com.google.common.collect.Iterators$7.computeNext(Iterators.java:648)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at com.google.common.collect.Iterators$7.computeNext(Iterators.java:647)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at com.google.common.collect.Lists.newArrayList(Lists.java:138)
at com.google.common.collect.Lists.newArrayList(Lists.java:119)
at org.jenkinsci.plugins.github.util.FluentIterableWrapper.toList(FluentIterableWrapper.java:147)
at org.jenkinsci.plugins.github.webhook.WebhookManager$1.run(WebhookManager.java:127)
at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:119)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
It's odd because I also see positive logs such as:
GitHub webhooks activated for job REDACTED_ORG_NAME/REDACTED_REPO with [GitHubRepositoryName[host=github.com,username=REDACTED,repository=REDACTED_REPO]] (events: [PULL_REQUEST, PUSH])
Apr 12, 2019 6:15:43 PM INFO org.jenkinsci.plugins.github.webhook.WebhookManager$1 run
GitHub webhooks activated for job REDACTED_ORG_NAME/REDACTED_REPO/develop with [GitHubRepositoryName[host=github.com,username=REDACTED,repository=REDACTED_REPO]] (events: [PULL_REQUEST, PUSH])
Apr 12, 2019 6:15:43 PM INFO org.jenkinsci.plugins.github.webhook.WebhookManager$1 run
GitHub webhooks activated for job REDACTED_ORG_NAME/REDACTED_REPO/master with [] (events: [PULL_REQUEST, PUSH])
Apr 12, 2019 6:15:43 PM INFO org.jenkinsci.plugins.github.webhook.WebhookManager$1 run
GitHub webhooks activated for job REDACTED_ORG_NAME/REDACTED_REPO/release%2F0.x with [] (events: [PULL_REQUEST, PUSH]
We've configured the following Jenkins plugins:
Git Plugin
Github Organizations with Github Branch Source with https://support.cloudbees.com/hc/en-us/articles/224543927-GitHub-webhook-configuration
I've followed all the troubleshooting steps at https://support.cloudbees.com/hc/en-us/articles/224621648-GitHub-webhook-troubleshooting but I'm getting no where. Steps that fail:
A.2. I tried redelivering the webhook payload but get Couldn't connect to server
B.4 Displays lists of Failed to delete post-commit hook:, Failed to create hook
, and There is no credentials with admin access to manage hooks on GitHubRepositoryName[host=github.com,username=REDACTED,repository=REDACTED]
What Does Work
Able to scan organization
Able to trigger builds manually by using Build Now
There seems like there are so many issues I have no idea where to look next. Thank you for help in advance.

Answer 1

Looks like our AWS Security Group was white-listing Github Hook IP addresses (https://api.github.com/meta) and Github recently added or changed an IP address that was not in the Security Group. So our AWS Security Group was rejecting requests. We added refreshed the IP addresses and it now works.