GitHub webhook can't connect to Jenkins inside VPN? - jenkins

Question

I am setting up continuous integration using Jenkins and GitHub Enterprise.
So far I have been able to easily create Jenkins projects and run jobs regularly.
Jenkins is using SSH public/private keys to authenticate and clone GitHub repos.
Jenkins runs on a Linux server that is inside my company's VPN network.
My next step is to configure Jenkins/GitHub so that each git push to GitHub would trigger a Jenkins build job. I have set up the post-commit webhook on GitHub, and enabled the GitHub hook trigger for GITScm polling checkbox in my Jenkins projects.
However, GitHub fails to connect to Jenkins.
Here is the log:
We couldn’t deliver this payload: failed to connect
{
"zen": "Design for failure.",
"hook_id": 287451636,
"hook": {
"type": "Repository",
"id": 287451636,
"name": "web",
"active": true,
"events": [
"push"
],
"config": {
"content_type": "json",
"insecure_ssl": "0",
"url": "http://1.2.3.4/jenkins/github-webhook/"
},
"updated_at": "2021-03-19T07:45:12Z",
"created_at": "2021-03-19T07:45:12Z",
"url": "https://api.github.com/repos/mycompany/my_repo/hooks/287451636",
"test_url": "https://api.github.com/repos/mycompany/my_repo/hooks/287451636/test",
"ping_url": "https://api.github.com/repos/mycompany/my_repo/hooks/287451636/pings",
"last_response": {
"code": null,
"status": "unused",
"message": null
}
},
"repository": {
"id": 347030450,
"node_id": "MDEwOlJlcG9zaXRvcnkzNDcwMzA0NTA=",
"name": "my_repo",
"full_name": "mycompany/my_repo",
"private": true,
"owner": {
"login": "mycompany",
"id": 11980925,
"node_id": "MDEyOk9yZ2FuaXphdGlvbjExOTgwOTI1",
"avatar_url": "https://avatars.githubusercontent.com/u/11980925?v=4",
"gravatar_id": "",
"url": "https://api.github.com/users/mycompany",
"html_url": "https://github.com/mycompany",
"followers_url": "https://api.github.com/users/mycompany/followers",
"following_url": "https://api.github.com/users/mycompany/following{/other_user}",
"gists_url": "https://api.github.com/users/mycompany/gists{/gist_id}",
"starred_url": "https://api.github.com/users/mycompany/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/mycompany/subscriptions",
"organizations_url": "https://api.github.com/users/mycompany/orgs",
"repos_url": "https://api.github.com/users/mycompany/repos",
"events_url": "https://api.github.com/users/mycompany/events{/privacy}",
"received_events_url": "https://api.github.com/users/mycompany/received_events",
"type": "Organization",
"site_admin": false
},
"html_url": "https://github.com/mycompany/my_repo",
"description": "My test repo.",
"fork": false,
"url": "https://api.github.com/repos/mycompany/my_repo",
"forks_url": "https://api.github.com/repos/mycompany/my_repo/forks",
"keys_url": "https://api.github.com/repos/mycompany/my_repo/keys{/key_id}",
"collaborators_url": "https://api.github.com/repos/mycompany/my_repo/collaborators{/collaborator}",
"teams_url": "https://api.github.com/repos/mycompany/my_repo/teams",
"hooks_url": "https://api.github.com/repos/mycompany/my_repo/hooks",
"issue_events_url": "https://api.github.com/repos/mycompany/my_repo/issues/events{/number}",
"events_url": "https://api.github.com/repos/mycompany/my_repo/events",
"assignees_url": "https://api.github.com/repos/mycompany/my_repo/assignees{/user}",
"branches_url": "https://api.github.com/repos/mycompany/my_repo/branches{/branch}",
"tags_url": "https://api.github.com/repos/mycompany/my_repo/tags",
"blobs_url": "https://api.github.com/repos/mycompany/my_repo/git/blobs{/sha}",
"git_tags_url": "https://api.github.com/repos/mycompany/my_repo/git/tags{/sha}",
"git_refs_url": "https://api.github.com/repos/mycompany/my_repo/git/refs{/sha}",
"trees_url": "https://api.github.com/repos/mycompany/my_repo/git/trees{/sha}",
"statuses_url": "https://api.github.com/repos/mycompany/my_repo/statuses/{sha}",
"languages_url": "https://api.github.com/repos/mycompany/my_repo/languages",
"stargazers_url": "https://api.github.com/repos/mycompany/my_repo/stargazers",
"contributors_url": "https://api.github.com/repos/mycompany/my_repo/contributors",
"subscribers_url": "https://api.github.com/repos/mycompany/my_repo/subscribers",
"subscription_url": "https://api.github.com/repos/mycompany/my_repo/subscription",
"commits_url": "https://api.github.com/repos/mycompany/my_repo/commits{/sha}",
"git_commits_url": "https://api.github.com/repos/mycompany/my_repo/git/commits{/sha}",
"comments_url": "https://api.github.com/repos/mycompany/my_repo/comments{/number}",
"issue_comment_url": "https://api.github.com/repos/mycompany/my_repo/issues/comments{/number}",
"contents_url": "https://api.github.com/repos/mycompany/my_repo/contents/{+path}",
"compare_url": "https://api.github.com/repos/mycompany/my_repo/compare/{base}...{head}",
"merges_url": "https://api.github.com/repos/mycompany/my_repo/merges",
"archive_url": "https://api.github.com/repos/mycompany/my_repo/{archive_format}{/ref}",
"downloads_url": "https://api.github.com/repos/mycompany/my_repo/downloads",
"issues_url": "https://api.github.com/repos/mycompany/my_repo/issues{/number}",
"pulls_url": "https://api.github.com/repos/mycompany/my_repo/pulls{/number}",
"milestones_url": "https://api.github.com/repos/mycompany/my_repo/milestones{/number}",
"notifications_url": "https://api.github.com/repos/mycompany/my_repo/notifications{?since,all,participating}",
"labels_url": "https://api.github.com/repos/mycompany/my_repo/labels{/name}",
"releases_url": "https://api.github.com/repos/mycompany/my_repo/releases{/id}",
"deployments_url": "https://api.github.com/repos/mycompany/my_repo/deployments",
"created_at": "2021-03-12T10:40:44Z",
"updated_at": "2021-03-18T13:59:52Z",
"pushed_at": "2021-03-18T13:59:49Z",
"git_url": "git://github.com/mycompany/my_repo.git",
"ssh_url": "git#github.com:mycompany/my_repo.git",
"clone_url": "https://github.com/mycompany/my_repo.git",
"svn_url": "https://github.com/mycompany/my_repo",
"homepage": null,
"size": 3,
"stargazers_count": 0,
"watchers_count": 0,
"language": "Makefile",
"has_issues": true,
"has_projects": true,
"has_downloads": true,
"has_wiki": true,
"has_pages": false,
"forks_count": 0,
"mirror_url": null,
"archived": false,
"disabled": false,
"open_issues_count": 0,
"license": null,
"forks": 0,
"open_issues": 0,
"watchers": 0,
"default_branch": "main"
},
"sender": {
"login": "MyLogin",
"id": 63738097,
"node_id": "MDQ6VXNlcjYzNzM4MDk3",
"avatar_url": "https://avatars.githubusercontent.com/u/13818097?v=4",
"gravatar_id": "",
"url": "https://api.github.com/users/MyLogin",
"html_url": "https://github.com/MyLogin",
"followers_url": "https://api.github.com/users/MyLogin/followers",
"following_url": "https://api.github.com/users/MyLogin/following{/other_user}",
"gists_url": "https://api.github.com/users/MyLogin/gists{/gist_id}",
"starred_url": "https://api.github.com/users/MyLogin/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/MyLogin/subscriptions",
"organizations_url": "https://api.github.com/users/MyLogin/orgs",
"repos_url": "https://api.github.com/users/MyLogin/repos",
"events_url": "https://api.github.com/users/MyLogin/events{/privacy}",
"received_events_url": "https://api.github.com/users/MyLogin/received_events",
"type": "User",
"site_admin": false
}
}
What could be the problem?

Answer 1

Well the problem is exactly because github cannot access your internal network.
There's an article in Jenkins Community Blog where author suggests using smee (released under ISC License) for webhooks forwarding.

Answer 2

If using smee to deliver webhooks from GitHub to your VPNed Jenkins (you set up Github to send events to smee.io, and than you use the Smee client to retrieve the events and use them internally on Jenknins), here is a quick way to set it up.
Go to Smee.io and get yourself a "channel", write down the URL you get (for example https://smee.io/Wua8D46nBvJ63nPP).
Install Smee on your Jenkins: sudo npm install --global smee-client
Configure the internal forwarding: smee -u https://smee.io/Wua8D46nBvJ63nPP --path /github-webhook/ --port 8080
Go to your GitHub repository: Settings->Webhooks->Add webhook, and paste the URL under "Payload URL", choose Content type "application/json".
Setup your Smee to run as a service on Jenkins, to start at boot time. You will need two files:
Init script /etc/init.d/smee
systemctl service description file /etc/systemd/system/smee.service
/etc/init.d/smee:
#! /bin/sh
### BEGIN INIT INFO
# Provides: smee
# Required-Start: $all
# Required-Stop:
# Default-Start: 2 3 4 5
# Default-Stop:
# Short-Description: Start SMEE daemon at boot time
# Description: Forwards GitHub webhook notifications to Jenkins.
### END INIT INFO
PATH=/bin:/usr/bin:/sbin:/usr/sbin
DAEMON=/usr/bin/smee
PIDFILE=/var/run/smee.pid
test -x $DAEMON || exit 0
. /lib/lsb/init-functions
case "$1" in
start)
log_daemon_msg "Starting smee" "smee"
start_daemon -p $PIDFILE $DAEMON
log_end_msg $?
;;
stop)
log_daemon_msg "Stopping smee" "smee"
killproc -p $PIDFILE $DAEMON
log_end_msg $?
;;
force-reload|restart)
$0 stop
$0 start
;;
status)
status_of_proc -p $PIDFILE $DAEMON atd && exit 0 || exit $?
;;
*)
echo "Usage: /etc/init.d/smee {start|stop|restart|force-reload|status}"
exit 1
;;
esac
exit 0
/etc/systemd/system/smee.service:
[Unit]
Description=smee.io webhook delivery from GitHub
After=network.target
StartLimitIntervalSec=0
[Service]
Type=simple
Restart=always
RestartSec=1
User=jenkins
ExecStart=/usr/bin/smee -u https://smee.io/Wua8D46nBvJ63nPP --path /jenkins/github-webhook/ --port 8080
[Install]
WantedBy=multi-user.target
Start the service: sudo /etc/init.d/smee start
Check the status: sudo /etc/init.d/smee status
The notifications from your repository should now be sent to Jenkins and Jenkins should be able to see and handle them. To quickly check if Jenkins handles any Smee comands, open the "Dashboard->System Log" and search for "smee".