Kubernetes

Registry

The registry has a Kubernetes Helm chart that provides a pre-configured Scraper and Topology with some common defaults

The kubernetes scraper collects all of the resources and events in a Kubernetes cluster, and then watches for changes.

kubernetes-scraper.yaml
apiVersion: configs.flanksource.com/v1
kind: ScrapeConfig
metadata:
  name: kubernetes-scraper
spec:
  retention:
    changes:
      - name: PodCrashLooping
        count: 10
        age: 72h
  kubernetes:
    - clusterName: local-kind-cluster
      transform:
        relationship:
          # Link a service to a deployment (adjust the label selector accordingly)
          - filter: config_type == "Kubernetes::Service"
            type:
              value: 'Kubernetes::Deployment'
            name:
              expr: |
                has(config.spec.selector) && has(config.spec.selector.name) ? config.spec.selector.name : ''
          # Link Pods to PVCs
          - filter: config_type == 'Kubernetes::Pod'
            expr: |
              config.spec.volumes.
                filter(item, has(item.persistentVolumeClaim)).
                map(item, {"type": "Kubernetes::PersistentVolumeClaim", "name": item.persistentVolumeClaim.claimName}).
                toJSON()
          # Link Argo Application to the resources
          - filter: config_type == "Kubernetes::Application" && config.apiVersion == "argoproj.io/v1alpha1"
            expr: |
              config.status.resources.map(item, {
                "type": "Kubernetes::" + item.kind,
                "name": item.name,
                "labels": {
                  "namespace": item.namespace,
                },
              }).toJSON()
        mask:
          - selector: |
              has(config.kind) ? config.kind == 'Certificate' : false
            jsonpath: .spec.dnsNames
            value: md5sum
          - selector: 'config_type == "Kubernetes::Certificate"'
            jsonpath: .spec.commonName
            value: md5sum
        exclude:
          - types:
              - Kubernetes::*
            jsonpath: '.metadata.ownerReferences'
          - types:
              - Kubernetes::Pod
            jsonpath: '.metadata.generateName'
        changes:
          mapping:
            - filter: >
                change.change_type == 'diff' && change.summary == "status.containerStatuses" && 
                patch != null && has(patch.status) && has(patch.status.containerStatuses) && 
                patch.status.containerStatuses.size() > 0 &&
                has(patch.status.containerStatuses[0].restartCount)
              type: PodCrashLooping
          exclude:
            - 'config_type == "Kubernetes::Endpoints" && details.message == "metadata.annotations.endpoints.kubernetes.io/last-change-trigger-time"'
            - 'config_type == "Kubernetes::Node" && has(details.message) && details.message == "status.images"'
            - 'details.source.component == "canary-checker" && (change_type == "Failed" || change_type == "Pass")'
            - >
              change_type == "diff" && summary == "status.reconciledAt" && 
              config != null && 
              has(config.apiVersion) && config.apiVersion == "argoproj.io/v1alpha1" && 
              has(config.kind) && config.kind == "Application"
      properties:
        - filter: 'config_type == "Kubernetes::Pod"'
          name: Logs
          icon: opensearch
          links:
            - text: opensearch
              url: https://opensearch.svc/_dashboards/app/discover#/?_a=(query:(language:kuery,query:'kubernetes_pod_id:{{.id}}'))
        - filter: 'config_type == "Kubernetes::Node"'
          name: Grafana
          icon: grafana
          links:
            - text: grafana
              url: https://grafana.svc/d/85a562078cdf77779eaa1add43ccec1e/kubernetes-compute-resources-namespace-pods?var-namespace={{.name}}
      exclusions:
        name:
          - junit*
          - k6-junit*
          - newman-junit*
          - playwright-junit-*
          - hello-world*
        namespace:
          - canaries
          - monitoring
        kind:
          - Secret
          - ReplicaSet
          - APIService
          - PodMetrics
          - NodeMetrics
          - endpoints.discovery.k8s.io
          - endpointslices.discovery.k8s.io
          - leases.coordination.k8s.io
          - podmetrics.metrics.k8s.io
          - nodemetrics.metrics.k8s.io
          - customresourcedefinition
          - controllerrevision
          - certificaterequest
          - orders.acme.cert-manager.io
        labels:
          canary-checker.flanksource.com/generated: 'true'
      relationships:
        - kind:
            expr: "has(spec.claimRef) ? spec.claimRef.kind : ''"
          name:
            expr: "has(spec.claimRef) ? spec.claimRef.name : ''"
          namespace:
            expr: "has(spec.claimRef) ? spec.claimRef.namespace : ''"

        - kind:
            value: Kustomization
          name:
            label: kustomize.toolkit.fluxcd.io/name
          namespace:
            label: kustomize.toolkit.fluxcd.io/namespace

        - kind:
            value: HelmRelease
          name:
            label: helm.toolkit.fluxcd.io/name
          namespace:
            label: helm.toolkit.fluxcd.io/namespace

        # FluxCD Git relationships
        - name:
            expr: "has(spec.sourceRef) ? spec.sourceRef.name : '' "
          namespace:
            expr: "has(spec.sourceRef) && has(spec.sourceRef.namespace)  ? spec.sourceRef.namespace : metadata.namespace "
          kind:
            value: "GitRepository"
      event:
        exclusions:
          reason:
            - SuccessfulCreate
            - Created
            - DNSConfigForming
        severityKeywords:
          error:
            - failed
            - error
          warn:
            - backoff
            - nodeoutofmemory

Field	Description	Scheme	Required
logLevel	Specify the level of logging.	string
schedule	Specify the interval to scrape in cron format. Defaults to every 60 minutes.	string
retention	Settings for retaining changes, analysis and scraped items	Retention
kubernetes	Specifies the list of Kubernetes configurations to scrape.	[]Kubernetes

Kubernetes

Field	Description	Scheme	Required
clusterName	Specify cluster name	string
event	Specify configuration to handle Kubernetes events.	Event
exclusions	Resources to be excluded from scraping	[]string
fieldSelector	Resources to be included e.g status.Phase=Running	string
kubeconfig	Kubeconfig to connect to the cluster	[]EnvVar
namespace	Include resources only from this namespace	string
relationships	Create relationships between kubernetes objects.	[]Relationship
scope	Specify scope for scrape. e.g cluster for scraping at Cluster level	string
selector	Include resources matching this selector only e.g matchLabels	string
since	Set time constraint for scraping resources within the set period	string
properties	Custom properties to be added for each item	[]ConfigProperty
transform	Custom transformations to apply	Transform
tags	Tags to set on each config item. cluster and namespace are set by default	map[string]string

Events

Kubernetes::Event resources are mapped to config changes. Events can be very verbose so they can be excluded or their severity level changed:

Field	Description	Scheme	Required
exclusions	A list of keywords used to exclude event objects based on the reason	[]string
severityKeywords	Specify keywords used to identify the severity of the Kubernetes Event based on the reason	SeverityKeywords

SeverityKeywords

Field	Description	Scheme	Required
warn	A list of keywords used to identify a warning severity from the reason. It could also be a match pattern: e.g. * to match all or !badword to exclude badword	[]string
error	Same as warn but used to map to error severity.	[]string

Relationships

You can create relationships between kubernetes objects on the basis of

info

Relationships can also be defined under transform.relationships, however defining them under kubernetes.relationships is simpler with specific support for kind, name and namespace fields.

kubernetes-relationship.yaml
kubernetes:
  - clusterName: 'eks'
    relationships:
      # If object has spec.claimRef field, use its kind, name and namespace
      - kind:
          expr: "has(spec.claimRef) ? spec.claimRef.kind : ''"
        name:
          expr: "has(spec.claimRef) ? spec.claimRef.name : ''"
        namespace:
          expr: "has(spec.claimRef) ? spec.claimRef.namespace : ''"

      # If object flux kustomize labels, link it to the parent Kustomization object
      - kind:
          value: Kustomization
        name:
          label: kustomize.toolkit.fluxcd.io/name
        namespace:
          label: kustomize.toolkit.fluxcd.io/namespace

      # If object helm kustomize labels, link it to the parent HelmRelease object
      - kind:
          value: HelmRelease
        name:
          label: helm.toolkit.fluxcd.io/name
        namespace:
          label: helm.toolkit.fluxcd.io/namespace

Field	Description	Scheme	Required
kind	kind of Kubernetes Object	Lookup	true
name	name of Kubernetes Object	Lookup	true
namespace	namespace of Kubernetes Object	Lookup	true

Lookup

There are 3 different ways to specify which value to use when finding related configs:

Field	Description	Scheme	Required
expr	Use an expression to get the value	string
value	Specify a static value	string
label	Get the value from a label	string

Special annotations

Kubernetes resources can be annotated with some special annotations that can direct the scraper to certain behaviors.

Field	Description	Scheme
config-db.flanksource.com/ignore	Exclude the object from being scraped along with all of its changes.	boolean
config-db.flanksource.com/ignore-change-severity	Exclude changes by severity for the given object that matches the pattern.	MatchPattern
config-db.flanksource.com/ignore-changes	Exclude changes by type for the given object that matches the pattern.	MatchPattern
config-db.flanksource.com/tags	Attach custom tags to the object. A config can have as many as 5 tags, so keep the custom tags limited.	key1:val1,key2:val2

Examples

Exclude verbose changes from argo application

argo-application.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: sock-shop
  namespace: argo
  annotations:
    config-db.flanksource.com/ignore-changes: ReconciliationSucceeded
    config-db.flanksource.com/ignore-change-severity: low
spec:
  ...

Excluding a particular secret from being scraped

secret.yaml
apiVersion: v1
kind: Secret
metadata:
  annotations:
    config-db.flanksource.com/ignore: true
  name: slack
  namespace: default
type: Opaque
data:
  token: ...