Docs Developer Site
DeveloperDocsKubernetes Operator For XpStableUsage

Using the operator

Contents

This section covers the operator functionality and how to use it, once installed on a Kubernetes cluster.

Custom Resources

The operator introduces a couple of new resources into your Kubernetes cluster.

Xp7Deployment

This is the main custom resource that defines your deployment. It is namespaced, so you have to create a namespace for it. Only 1 Xp7Deployment can reside in each namespace.

apiVersion: enonic.cloud/v1
kind: Xp7Deployment
metadata:
  name: my-deployment
  namespace: my-namespace
spec:
  enabled: true     # Should the pods be running
  xpVersion: 7.13.2  # XP version, must be >=7.13.2

  # List of volumes shared by all nodes (see description of volumes below)
  nodesSharedVolumes:
    - name: sharedVolume
      size: 1Gi

  # List of disks shared by all nodes (see description of disks below)
  nodesSharedDisks:
    - name: blobstore
      volume: sharedVolume
    - name: snapshots
      volume: sharedVolume
    - name: export
      size: 1Gi

  # List of nodegroups
  nodeGroups:
      # Name and number of replicas
    - name: main
      replicas: 1

      # Role of the node
      data: true
      master: true

      initContainers:
        # Preinstall Snapshotter app
        - image: enonic/snapshotter:3.1.0
          name: snapshotter-deploy
          mounts:
            - name: deploy
              mountPath: /deploy

      # Environment variables
      env:
        - name: MY_ENV_VAR
          value: hey

      resources:
        # Cpu and memory allocation
        cpu: "1.5"
        memory: 512Mi
        dirs:
          # Snapshotter app copied to deploy folder
          - name: deploy
            size: 100Mi
            mountReadOnly: true
        # List of volumes private to each and every node
        volumes:
        - name: privateVolume
          size: 1Gi
          class: standard

        # List of disks private to each and every node
        disks:
          - name: index
            volume: privateVolume
          - name: work
            volume: privateVolume

Disks

In the Xp7Deployment resource we have names of disks that you can define to be shared, private or non existant.

blobstore

All blob objects. This should always be shared in a cluster.

snapshots

Index snapshots. This should always be shared in a cluster.

export

Data dumps and other data. This should be shared in a cluster.

index

Private node index. This should never be shared.

work

Private node cache. This should never be shared.

deploy

Locally installed apps on this single node. This should never be shared.

If you define a volume property, then the disk will be mounted in specified volume by subpath equals to the disk name. If you dont define a volume property, then the implicit volume will be created and the disk will be mounted in it to the root path.

For customization, two key attributes can be adjusted: size and storageClass. These attributes pertain to the implicit volume. To specify default values use the values.yaml file during the installation stage.

For custom disks, specify a folder where the disk will be mounted inside the container, specify operator.charts.values.volumes.mounts.<disk_name> property in the values.yaml file during the installation stage. There is no need to specify default disk locations and it’s not recommended to override them without a good reason.

The distinction between shared and private disks is that shared disks are mounted in all nodes, while private disks are mounted only in the node where they are defined.

Volumes

Sometimes, you don’t want to have a separate implicit volume for every disk on the node. In this case, you can define a volume and mount some or all disks in it as subfolders. This could be useful when you have limitations on the minimum size of the volume from your cloud provider, for example.

Similar to disks, volumes can also be private or shared among nodes. Private volumes are used by private disks, while shared volumes are used by shared disks, respectively.

A size and class params can be specified for every volume. If you don’t specify them, the default values will be used. To specify default values use the values.yaml file during the installation stage.

initContainers

In the context of Xp7Deployment, initContainers are containers launched before the XP container itself, as part of the deployment lifecycle. This mechanism can be useful to install applications (add applications to deploy folder of the main container), preinstall custom workloads or set up the settings of the XP container or perform other tasks. One important thing to remember is that, if there are several initContainers defined, they will be launched sequentially, following the order in the configuration. If any of the initContainers fail, Kubernetes will keep retrying until it succeeds.

  ...
      initContainers:
        - name: init-db
          image: db-init-image:latest
          command: ["sh", "-c"]
          args: ["until nslookup mydatabase; do echo waiting for mydatabase; sleep 2; done"]
          env:
            - name: CHECK_INTERVAL
              value: "2"
          mounts:
            - name: init-scripts
              mountPath: /scripts
  ...

An initContainer block in Xp7Deployment has several possible fields:

  • name:The name of the initContainer

  • image: The image to use for the initContainer

  • command: The command to run in the initContainer

  • args: The arguments for the command in initContainer

  • env: Specify environment variables here. See env for more information.

  • ports: A list of port objects with properties name, containerPort, and protocol. This can be used to expose ports

  • mounts: A list of mount dirs, disks or volumes with properties name, mountPath, subPath.

Property Description

name

name of an actual dir, disk or volume name, defined in Xp7Deployment resource.

mountPath

It is the path to mount the dir, disk or volume in the container.

subPath

It is the path inside the dir, disk or volume to mount in the container. If not specified, the root path will be used.

Read more about the init containers.

sidecars

A sidecar is a utility container that is designed to support and enhance the functionality of the main XP application container. All containers within a pod share the same network stack, so sidecar containers can provide additional functionality without modifying the main container.

For instance, a sidecar can be conceived to collecting logs from the main container,collect prometheus metrics or to provide a backup agent for blobstore and snapshots disks.

Here is how you can define a sidecar within the container structure:

  ...
      sidecars:
        - image: my-backup-agent
          name: my-backup-agent
          command: []
          args: []
          mounts:
            - name: restic
              mountPath: /restic
            - name: export
              mountPath: /enonic-xp/home/data
            - name: shared
              subPath: blobstore
              mountPath: /enonic-xp/home/repo/blob
            - name: shared
              subPath: snapshots
              mountPath: /enonic-xp/home/snapshots
          ports:
            - name: backup
              containerPort: 8000
              protocol: TCP
          env:
            - name: AGENT_K8S_CONTAINER
              value: my-backup-agent

  • name:The name of the sidecar

  • image: The image to use for the sidecar

  • command: The command to run in the sidecar

  • args: The arguments for the command in sidecar

  • env: Specify environment variables here. See env for more information.

  • ports: A list of port objects with properties name, containerPort, and protocol. This can be used to expose ports

  • mounts: A list of mount dirs, disks or volumes with properties name, mountPath, subPath.

Property Description

name

name of an actual dir, disk or volume name, defined in Xp7Deployment resource.

mountPath

It is the path to mount the dir, disk or volume in the container.

subPath

It is the path inside the dir, disk or volume to mount in the container. If not specified, the root path will be use

env

In the same way as for initContainers and sidecars, you can specify environment variables for the XP containers. This can be done by adding an env block to the particular nodeGroup of the Xp7Deployment resource.

Every environment variable in the env array is an object with properties:

  • name: A string containing the name of the environment variable.

  • value: A string containing the value of the environment variable. This field is optional.

  • valueFrom: An object that allows setting the value of the environment variable from various sources such as field references (fieldRef), secrets (secretKeyRef) and configMaps (configMapKeyRef). This field is optional.

Simple single environment variable:

      env:
        - name: MY_ENV_VAR
          value: my-value

Secret key reference example:

    env:
      - name: SECRET_USERNAME
        valueFrom:
          secretKeyRef:
            name: mysecret
            key: username
            optional: true

An environment variable SECRET_USERNAME is defined whose value is sourced from a Secret named mysecret and the key username. The secretKeyRef object allows this kind of Secret reference providing a secure way to store sensitive information like usernames, passwords, or keys in Kubernetes, which can then be consumed by your containers.

Field reference example:

      env:
        - name: AGENT_K8S_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace

This configuration means the AGENT_K8S_NAMESPACE environment variable will be set to the namespace in which the XP deployment is running.

ConfigMap key reference example:

      env:
        - name: APP_CONFIG
          valueFrom:
            configMapKeyRef:
              name: app-config
              key: log_level

This configuration means the APP_CONFIG environment variable will be set to the value of the log_level key in the app-config ConfigMap.

Xp7Config

This custom resource manages the configuration of XP nodes. This resource namespaced, and should be created in a namespace that already contains a Xp7Deployment.

apiVersion: enonic.cloud/v1
kind: Xp7Config
metadata:
  name: my-config
  namespace: my-namespace
spec:
  #nodeGroup: all
  #dataBase64: false
  file: my.app.name.cfg
  data: |
    my = custom
    config = file
The nodegroup field is optional and defaults to all nodegroups. If you want to apply you configuration only to a single node group, set the field appropriately.
The dataBase64 field is optional and defaults to false.

You can also create Xp7Config files that hold binary data. To do that you have to base64 encode the data and set the dataBase64 field to true like so:

apiVersion: enonic.cloud/v1
kind: Xp7Config
metadata:
  name: my-config
  namespace: my-namespace
spec:
  #nodeGroup: all
  dataBase64: true
  file: my.app.name.cfg
  data: SGVpISBZb3UgYXJlIG9uZSBub3N5IGZveC4gVGhpcyBpcyB0b3Agc2VjcmV0IGRhdGEuIEdldCBvdXQgb2YgaGVyZS4gU2hvb28uLi4uLi4uLi4uLg==
It can vary how fast XP registers the Xp7Config changes. It can be instant, but it can also take up to a couple of minutes, depending on the Kubernetes cluster setup.

Xp7App

This resource is to manage apps with the operator. While you can manage them with XP, this provides you with the option to create a deployment complete with your custom apps using the operator. This resource namespaced, and should be created in a namespace that already contains a Xp7Deployment.

apiVersion: enonic.cloud/v1
kind: Xp7App
metadata:
  name: contentstudio
  namespace: my-namespace
spec:
  url: https://repo.enonic.com/public/com/enonic/app/contentstudio/5.0.3/contentstudio-5.0.3.jar
  #sha512: ba6b40ebf0808c6fa619ba2a05d07ce3e566eada7e9f3f34c1d280e9d1dcbd1e3c25eff9896d1057c4578ff3f516afa7a905c9e42ddc5e08f1cdf06f7e89774c
The sha512 field is optional, but if provided, XP will validate the sha512 sum of the jar before installing it. This prevents installing of potential malicious apps from the internet.

Added functionality

In addition to new resources, there are also new annotations that add some functionality.

Ingresses

To create virtual hosts for XP you use ingress annotations. These follow the format of:

enonic.cloud/xp7.vhost.mapping.<MAPPING_NAME>.source: /admin
enonic.cloud/xp7.vhost.mapping.<MAPPING_NAME>.target: /admin
enonic.cloud/xp7.vhost.mapping.<MAPPING_NAME>.idproviders: <DEFAULT_IDPROVIDER>,<OTHER_ENABLED_IDPROVIDER>

A very important thing to keep in mind is that the annotation enonic.cloud/xp7.vhost.mapping.<MAPPING_NAME>.source has to match a defined spec.rules[?].http.paths[?].path in the same ingress. That is so the operator knows what node groups it needs to update. That brings us to the second point. The spec.rules[?].http.paths[?].backend.serviceName has to match a node group name defined in your Xp7Deployment.

An example of a valid ingress, assuming you have a nodegroup main, would look something like this.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-domain-com
  namespace: my-namespace
  annotations:
    enonic.cloud/xp7.vhost.mapping.my-mapping-site.source: /
    enonic.cloud/xp7.vhost.mapping.my-mapping-site.target: /site/default/master/homepage

    enonic.cloud/xp7.vhost.mapping.my-mapping-admin.source: /admin
    enonic.cloud/xp7.vhost.mapping.my-mapping-admin.target: /admin
    enonic.cloud/xp7.vhost.mapping.my-mapping-admin.idproviders: system
spec:
  rules:
    - host: my-domain.com
      http:
        paths:
          - path: /
            pathType: ImplementationSpecific
            backend:
              service:
                name: main
                port:
                  number: 8080

          - path: /admin
            pathType: ImplementationSpecific
            backend:
              service:
                name: main
                port:
                  number: 8080
Like Xp7Config, changes to virtual hosts can take a couple of minutes to register in XP.

Namespaces

It can be desireble to delete all created resources that are associated with an Xp7Deployment once its deleted. That is quite easy to do with this namespace annotation:

apiVersion: v1
kind: Namespace
metadata:
  name: my-namespace
  annotations:
    enonic.cloud/remove.with.xp7deployment: my-deployment

Common operations

Get deployment status

$ kubectl -n my-namespace get xp7deployments.enonic.cloud

NAME            ENABLED   VERSION   STATE     MSG
my-deployment   true      7.13.2     RUNNING   OK

Get config status

$ kubectl -n my-namespace get xp7configs.enonic.cloud

NAME                 NODEGROUP   FILE                                 STATE   MSG
all-admin            all         com.enonic.xp.app.main.cfg           READY   OK
all-appstatus        all         com.enonic.app.status.cfg            READY   OK
all-cluster          all         com.enonic.xp.cluster.cfg            READY   OK
all-logback          all         logback.xml                          READY   OK
all-sessionstore     all         com.enonic.xp.web.sessionstore.cfg   READY   OK
all-system           all         system.properties                    READY   OK
main-elasticsearch   main        com.enonic.xp.elasticsearch.cfg      READY   OK
main-vhosts          main        com.enonic.xp.web.vhost.cfg          READY   OK
my-config            all         com.my-app.cfg                       READY   OK

Get app status

$ kubectl -n my-namespace get xp7apps.enonic.cloud

NAME            KEY                            VERSION   STATE     MSG
contentstudio   com.enonic.app.contentstudio   3.2.0     RUNNING   OK

Fetching SU password

$ kubectl -n my-namespace get secret su -o go-template="{{ .data.pass | base64decode }}"

NGDDlGdFYkX8i@#49#Z6N45tfhX6#3Rw

Access admin (bypassing ingress)

$ kubectl -n my-namespace port-forward main-0 8080

Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080

Then open up localhost:8080 in your browser.

View XP logs

$ kubectl -n my-namespace logs -c exp main-0
                         _____
____________________________(_)______   ____  _________
_  _ \_  __ \  __ \_  __ \_  /_  ___/   __  |/_/__  __ \
/  __/  / / / /_/ /  / / /  / / /__     __>  < __  /_/ /
\___//_/ /_/\____//_/ /_//_/  \___/     /_/|_| _  .___/
                                               /_/

# Enonic XP 7.13.2
# Built on 2021-02-02T15:28:02Z (hash = 632195fda1bf0e9ce4a314d70b403ef731955ad0, branch = e4ea190187bd27bc1143a29c4ff2c80e564f58c0)
# OpenJDK 64-Bit Server VM 11.0.10 (AdoptOpenJDK)
# Linux 4.19.157 (amd64)
# Install directory is /enonic-xp
# Home directory is /enonic-xp/home
....

Edit configuration

$ kubectl -n my-namespace edit xp7configs.enonic.cloud my-config

xp7config.enonic.cloud/my-config edited

Watch configuration

$ kubectl -n my-namespace get xp7configs.enonic.cloud -w

NAME                 NODEGROUP   FILE                                 STATE     MSG
all-admin            all         com.enonic.xp.app.main.cfg           READY     OK
all-appstatus        all         com.enonic.app.status.cfg            READY     OK
all-cluster          all         com.enonic.xp.cluster.cfg            READY     OK
all-logback          all         logback.xml                          READY     OK
all-sessionstore     all         com.enonic.xp.web.sessionstore.cfg   READY     OK
all-system           all         system.properties                    READY     OK
main-elasticsearch   main        com.enonic.xp.elasticsearch.cfg      READY     OK
main-vhosts          main        com.enonic.xp.web.vhost.cfg          READY     OK
my-config            all         com.my-app.cfg                       PENDING   Not loaded

Simple example

Let’s deploy a simple example. Create a file called simple.yaml and paste these contents to it:

simple.yaml 
# Create a namespace
apiVersion: v1
kind: Namespace
metadata:
  name: my-namespace
  annotations:
    # Delete this namespace if the deployment is deleted
    enonic.cloud/remove.with.xp7deployment: my-deployment
---
# Create deployment in the namespace
apiVersion: enonic.cloud/v1
kind: Xp7Deployment
metadata:
  name: my-deployment
  namespace: my-namespace
spec:
  enabled: true
  xpVersion: 7.13.4

  # Create one node
  nodeGroups:
    - name: main
      replicas: 1

      data: true
      master: true

      initContainers:
        # Preinstall Snapshotter app
        - image: enonic/snapshotter:3.1.0
          name: snapshotter-deploy
          mounts:
            - name: deploy
              mountPath: /deploy

      resources:
        cpu: "1"
        memory: 1Gi
        dirs:
          # Snapshotter app copied to deploy folder
          - name: deploy
            size: 100Mi
            mountReadOnly: true
        # Disks private to the node
        disks:
          - name: blobstore
            size: 1Gi
          - name: snapshots
            size: 1Gi
          - name: export # Dumps and other data
            size: 1Gi
          - name: index # Node ES index
            size: 1Gi
          - name: work # Node cache
            size: 1Gi
---
# Install content studio
apiVersion: enonic.cloud/v1
kind: Xp7App
metadata:
  name: contentstudio
  namespace: my-namespace
spec:
  url: https://repo.enonic.com/public/com/enonic/app/contentstudio/5.0.3/contentstudio-5.0.3.jar
  sha512: ba6b40ebf0808c6fa619ba2a05d07ce3e566eada7e9f3f34c1d280e9d1dcbd1e3c25eff9896d1057c4578ff3f516afa7a905c9e42ddc5e08f1cdf06f7e89774c
---
# Add your own custom config
apiVersion: enonic.cloud/v1
kind: Xp7Config
metadata:
  name: my-config
  namespace: my-namespace
spec:
  nodeGroup: all
  file: com.my-app.cfg
  data: |
    my = config
---
# Expose XP through an ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-domain-com
  namespace: my-namespace
  annotations:
    enonic.cloud/xp7.vhost.mapping.my-mapping-site.source: /
    enonic.cloud/xp7.vhost.mapping.my-mapping-site.target: /site/default/master/homepage

    enonic.cloud/xp7.vhost.mapping.my-mapping-admin.source: /admin
    enonic.cloud/xp7.vhost.mapping.my-mapping-admin.target: /admin
    enonic.cloud/xp7.vhost.mapping.my-mapping-admin.idproviders: system
spec:
  rules:
    - host: my-domain.com
      http:
        paths:
          - path: /
            pathType: ImplementationSpecific
            backend:
              service:
                name: main
                port:
                  number: 8080

          - path: /admin
            pathType: ImplementationSpecific
            backend:
              service:
                name: main
                port:
                  number: 8080

Deploy this by running:

$ kubectl apply -f simple.yaml

namespace/my-namespace created
xp7deployment.enonic.cloud/my-deployment created
xp7app.enonic.cloud/contentstudio created
xp7config.enonic.cloud/my-config created
ingress.networking.k8s.io/my-domain-com created

Once the XP pods have started you can open up the admin by following the Access admin (bypassing ingress) section or call the ingress controller, if you have one set up.

Simple shared volume example

In case you want to map all your disks to a single volume, you can change the Xp7Deployment resource from the previous example to look like this:

simple-shared-volume.yaml 
# Create deployment in the namespace
apiVersion: enonic.cloud/v1
kind: Xp7Deployment

...

      resources:
        cpu: "1"
        memory: 1Gi
        dirs:
          # Snapshotter app copied to deploy folder
          - name: deploy
            size: 100Mi
            mountReadOnly: true
        # Single inner volume for all disks
        volumes:
          - name: inner
            size: 1Gi

        # Private disks to use the shared private volume
        disks:
          - name: index
            volume: inner
          - name: work
            volume: inner
          - name: export
            volume: inner
          - name: snapshots
            volume: inner
          - name: blobstore
            volume: inner

Cluster example

The values cpu and memory setting in this example are too low for a good cluster setup. They are set this way, so you can try it out on a low resource Kubernetes cluster.

Let’s deploy a cluster example. Create a file called cluster.yaml and paste these contents to it:

cluster.yaml 
# Create a namespace
apiVersion: v1
kind: Namespace
metadata:
  name: my-namespace
  annotations:
    # Delete this namespace it the deployment is deleted
    enonic.cloud/remove.with.xp7deployment: my-deployment
---
# Create deployment in the namespace
apiVersion: enonic.cloud/v1
kind: Xp7Deployment
metadata:
  name: my-deployment
  namespace: my-namespace
spec:
  enabled: true
  xpVersion: 7.13.4

  # Create volumes shared by all nodes in this deployment
  nodesSharedDisks:
    - name: blobstore
      size: 1Gi

    - name: snapshots
      size: 1Gi

    - name: export # Dumps and other data
      size: 1Gi

  # Create nodes
  nodeGroups:
    # 3 master nodes
    - name: master
      replicas: 3

      data: false
      master: true

      resources:
        cpu: "0.5"
        memory: 1Gi

        # Volumes private to the node
        disks:
          - name: index # Node ES index
            size: 1Gi

    # 2 data nodes
    - name: worker
      replicas: 2

      data: true
      master: false

      initContainers:
        # Preinstall Snapshotter app on worker nodes
        - image: enonic/snapshotter:3.1.0
          name: snapshotter-deploy
          mounts:
            - name: deploy
              mountPath: /deploy

      resources:
        cpu: "1"
        memory: 1Gi

        dirs:
          # Snapshotter app copied to deploy folder
          - name: deploy
            size: 100Mi
            mountReadOnly: true
        # Volumes private to the node
        disks:
          - name: index # Node ES index
            size: 1Gi
---
# Install content studio
apiVersion: enonic.cloud/v1
kind: Xp7App
metadata:
  name: contentstudio
  namespace: my-namespace
spec:
  url: https://repo.enonic.com/public/com/enonic/app/contentstudio/5.0.3/contentstudio-5.0.3.jar
  sha512: ba6b40ebf0808c6fa619ba2a05d07ce3e566eada7e9f3f34c1d280e9d1dcbd1e3c25eff9896d1057c4578ff3f516afa7a905c9e42ddc5e08f1cdf06f7e89774c
---
# Add your own custom config
apiVersion: enonic.cloud/v1
kind: Xp7Config
metadata:
  name: my-config
  namespace: my-namespace
spec:
  nodeGroup: all
  file: com.my-app.cfg
  data: |
    my = config
---
# Expose XP site on frontend nodes through an ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-domain-com-site
  namespace: my-namespace
  annotations:
    enonic.cloud/xp7.vhost.mapping.my-mapping-site.source: /
    enonic.cloud/xp7.vhost.mapping.my-mapping-site.target: /site/default/master/homepage
spec:
  rules:
    - host: my-domain.com
      http:
        paths:
          - path: /
            pathType: ImplementationSpecific
            backend:
              service:
                name: worker
                port:
                  number: 8080
---
# Expose XP admin on admin nodes through an ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-domain-com-admin
  namespace: my-namespace
  annotations:
    # Enable sticky sessions with nginx
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/session-cookie-name: "stickyXpAdmin"
    nginx.ingress.kubernetes.io/session-cookie-expires: "129600" # 36 hours
    nginx.ingress.kubernetes.io/session-cookie-max-age: "129600" # 36 hours
    nginx.ingress.kubernetes.io/session-cookie-change-on-failure: "true"

    enonic.cloud/xp7.vhost.mapping.my-mapping-admin.source: /admin
    enonic.cloud/xp7.vhost.mapping.my-mapping-admin.target: /admin
    enonic.cloud/xp7.vhost.mapping.my-mapping-admin.idproviders: system
spec:
  ingressClassName: nginx
  rules:
    - host: my-domain.com
      http:
        paths:
          - path: /admin
            pathType: ImplementationSpecific
            backend:
              service:
                name: worker
                port:
                  number: 8080

Deploy this by running:

$ kubectl apply -f cluster.yaml

namespace/my-namespace created
xp7deployment.enonic.cloud/my-deployment created
xp7app.enonic.cloud/contentstudio created
xp7config.enonic.cloud/my-config created
ingress.networking.k8s.io/my-domain-com-site created
ingress.networking.k8s.io/my-domain-com-admin created

Once the XP pods have started you can open up the admin by following the Access admin (bypassing ingress) section or call the ingress controller, if you have one set up.

Contents

Contents