Kubernetes Troubleshooting
==========================

Throughout this class you have used several first step methods in troubleshooting. Using *get* and *describe* help start the process, but what if they do not provide 
enough detail to lead you to the problem? In this lab we'll show you some additional commands you can use to get information or configure all the objects in your cluster.


Explain
-------

Let's start with the *explain* command. The Kubernetes explain command provides documentation of the resource or specific field of the resource. In past labs, you used 
imperative commands to create pods but we only covered the fields used. You can find reference to all objects right from the cli.

Example:

.. code-block:: bash
   :caption: Explain

   kubectl explain <resource>

Now, let's *walk* some parts of the pod resource. This command will show the fields available to you to configure a pod.

.. code-block:: bash 
   :caption: Explain Pod

   kubectl explain pod

Digging deeper, what details are available under *metadata*?

.. code-block:: bash
   :caption: Pod Meta


   kubectl explain pod.metadata

As you can see from the output, the *name* field is required when declaring a pod.

Get
---

Kubernetes *get* command will fetch the status and name of an object. In the below example, instead of requesting information on a specific object we are seeking information
on all objects within the *test* namespace.

.. code-block:: bash
   :caption: Get

   kubectl get all -n test 

Describe
--------

Kubernetes *describe* shows details of a specific resource or group of resources. This command joins many API calls together to form a detailed description of 
a given resource or group of resources. This is the command you have run several times during this class to find detailed information on your deployed resources.

.. code-block:: bash 
   :caption: Describe

   kubectl describe <resource_type> <resource_name> -n <namespace>

Events
------

Kubernetes *events* can provide valuable insights to events from controllers, schedulers, pods and nodes. You can, and should, filter down events.

.. code-block:: bash 
   :caption: Filter Namespace

   kubectl get events -n test

Get live events:

.. code-block:: bash 
   :caption: Filter Follow

   kubectl get events -n test --watch

Filter events by *namespace* and resource type.

.. code-block:: bash 
   :caption: Filter Pod

   kubectl get events -n test --field-selector involvedObject.kind=Pod

Filter events by namespace, resource type, and pod name.

.. code-block:: bash 
   :caption: Filter Pod

   kubectl get events -n test --field-selector involvedObject.kind=Pod --field-selector involvedObject.name=testpod

To sort events by time you can use the below command. The **--sort-by** command is actually reading the Kubernetes JSON returned data to extract the *lastTimestamp* field.

.. code-block:: bash
   :caption: Time Sort

   kubectl get events -n test --sort-by={.lastTimestamp}

Logs
----

The *logs* command allows you to view logs generated by a pod. You have a run a similar command in the Container lab. In these two steps you'll view logs in a single pod
and all pod logs from the deployment.

.. code-block:: bash 
   :caption: Pod Logs

   kubectl logs testpod -n test

Deployment logs:

.. code-block:: bash
   :caption: Deployment Logs

   kubectl logs deploy/lab-deploy -n test

Execute
-------

You can connect to the shell of a running container by using the below command. 

.. code-block:: bash 
   :caption: Shell Single Container

   kubectl exec -it testpod -n test -- /bin/bash

You should now see a prompt:

.. code-block:: bash
   :caption: Bash
   :emphasize-lines: 2

   lab@k3s-leader:~$ kubectl exec -it testpod -n test -- /bin/bash
   root@testpod:/#


Feel free to run some Linux commands such as

- pwd
- ls -la

To exit the shell, type **exit**

But you don't have to access the shell to run your commands, you can *pass* the command to the shell.

.. code-block:: bash
   :caption: Shell

   kubectl exec -it testpod -n test -- ls -la


If your pod has more than one container you must specify the container you want to connect to with the ``-c`` flag as in the example below.

.. code-block:: bash 
   :caption: Example Shell Multi-Container

   kubectl exec -it <pod_name> -c <container_name> -n <namespace> -- /bin/bash


DNS Utils
---------

For this next troubleshooting exercise, you'll deploy a special *dnsutils* container image. This container has *dnsutils* installed and will allow you to view how services are
registered in CoreDNS.

.. code-block:: bash
   :caption: DNSUTILS

   kubectl run dnsutils --image=registry.k8s.io/e2e-test-images/jessie-dnsutils:1.3 --restart=Always -n test -- /bin/bash -c "sleep infinity"

Once deployed and running, you can execute dig commands from inside the cluster using the dnsutils tools. If this command fails, wait a few seconds then execute again.

.. code-block:: bash
   :caption: DNS dig

   kubectl exec -it dnsutils -n test -- dig lab-deploy-svc.test.svc.cluster.local