kubernetes deployment notification to slack channel using webhook - docker

Question

I have set up the notification for cloud build CI/CD which is pushing notification to a respective slack channel.
After a successful build that image push to kubernetes cluster and rolling update strategy followed by deployment.
So I want to push notification when new pod become ready and old pod terminated so that time gets an idea about new changes applied to deployment.
Note : I am using GKE cluster but not installed Prometheus due to resource limits.

Answer 1

There are multiple ways of doing this, I can think of two ways right now:
Use Prometheus + Alert manager to send you a slack notification when pods became ready.
Use CI/CD pipeline to continuously check for the status of the pods, once they are updated successfully, send a notification.
Hope this answers your question.
EDIT:
If you would like to stick to using stackdriver, then there is a solution for it as well: https://kubernetes.io/docs/tasks/debug-application-cluster/events-stackdriver/

Answer 2

If you cannot afford to Prometheus stack due to resources limitation check kubewatch it has slack support build-in so it should be suitable for your needs.

Answer 3

Using kubernetes api watch can be implemented
https://github.com/kubernetes-client/csharp/tree/master/examples/watch
Basically it will check the not running pod and notify using Microsoft Teams web-hook. It will also notify pod which was initially not running and came back to running status again(recovered pod)
C# code snippet below with Main and Notify function.
You can replace the deployment over pod
static async Task Main(string[] args)
{
// Load from the default kubeconfig on the machine.
var config = KubernetesClientConfiguration.BuildConfigFromConfigFile();
// Use the config object to create a client.
var client = new Kubernetes(config);
try
{
var podlistResp = await client.ListNamespacedPodWithHttpMessagesAsync(Namespace, watch: true);
using (podlistResp.Watch<V1Pod, V1PodList>(async (type, item) =>
{
Console.WriteLine(type);
Console.WriteLine("==on watch event==");
var message = $"Namespace: {Namespace} Pod: {item.Metadata.Name} Type: {type} Phase:{item.Status.Phase}";
var remessage = $"Namespace: {Namespace} Pod: {item.Metadata.Name} Type: {type} back to Phase:{item.Status.Phase}";
Console.WriteLine(message);
if (!item.Status.Phase.Equals("Running") && !item.Status.Phase.Equals("Succeeded"))
{ Console.WriteLine("==on watch event==");
await Notify(message);
Console.WriteLine("==on watch event==");
}
if ( type== WatchEventType.Modified && item.Status.Phase.Equals("Running") )
{ Console.WriteLine("==on watch event==");
await Notify(remessage);
Console.WriteLine("==on watch event==");
}
}))
{
Console.WriteLine("press ctrl + c to stop watching");
var ctrlc = new ManualResetEventSlim(false);
Console.CancelKeyPress += (sender, eventArgs) => ctrlc.Set();
ctrlc.Wait();
}
}
catch (System.Exception ex)
{
Console.Error.WriteLine($"An error happened Message: {ex.Message}", ex);
}
}
private static async Task Notify(string message)
{
using (var client = new HttpClient())
{
client.BaseAddress = new Uri("https://outlook.office.com");
var body = new { text = message };
var content = new StringContent(JsonConvert.SerializeObject(body));
var result = await client.PostAsync("https://outlook.office.com/webhook/xxxx/IncomingWebhook/xxx", content);
result.EnsureSuccessStatusCode();
}
}

Answer 4

For anyone looking for such tool, do try BotKube. It is built to send
Kubernetes events to messengers like Slack, Mattermost and Microsoft Teams. It also has features like custom filters