Elastx Documentation

• 1: AI and Other Services
• 1.1: Announcements
• 2: Cloud Connect
• 3: Data Security Manager (beta)
• 3.1: File System Encryption for Linux
• 4: DBaaS
• 4.1: Announcements
• 4.2: DBaaS Getting Started
• 4.3: FAQ
• 4.4: Managed Service
• 4.5: Overview
• 4.6: Reference
• 4.6.1: Datastore Statuses
• 4.6.2: Glossary
• 4.6.3: Notifications
• 4.6.4: Observability
• 4.6.4.1: Metrics
• 4.6.4.1.1: Introduction
• 4.6.4.1.2: MySQL And MariaDB
• 4.6.4.1.3: PostgreSQL
• 4.6.4.1.4: Redis
• 4.6.4.1.5: System
• 4.6.4.1.6: Valkey
• 4.6.5: Products
• 4.6.5.1: MariaDb
• 4.6.5.1.1: Backup
• 4.6.5.1.2: Configuration
• 4.6.5.1.3: Importing Data
• 4.6.5.1.4: Limitations
• 4.6.5.1.5: Overview
• 4.6.5.1.6: Restore
• 4.6.5.1.7: TLS Connection
• 4.6.5.2: MSSQLServer
• 4.6.5.2.1: Configurations
• 4.6.5.2.2: Limitations
• 4.6.5.2.3: Overview
• 4.6.5.2.4: User Management
• 4.6.5.3: MySQL
• 4.6.5.3.1: Backup
• 4.6.5.3.2: Configuration
• 4.6.5.3.3: Importing Data
• 4.6.5.3.4: Importing Data From AWS RDS
• 4.6.5.3.5: Importing Data From GCP
• 4.6.5.3.6: Limitations
• 4.6.5.3.7: Overview
• 4.6.5.3.8: Restore
• 4.6.5.3.9: TLS Connection
• 4.6.5.3.10: User Management
• 4.6.5.4: PostgreSQL
• 4.6.5.4.1: Backup
• 4.6.5.4.2: Configuration
• 4.6.5.4.3: Extensions
• 4.6.5.4.4: Importing Data
• 4.6.5.4.5: Limitations
• 4.6.5.4.6: Restore
• 4.6.5.5: Redis
• 4.6.5.5.1: Backup
• 4.6.5.5.2: Configuration
• 4.6.5.5.3: User Management
• 4.6.5.6: Valkey
• 4.6.5.6.1: Backup
• 4.6.5.6.2: Configuration
• 4.6.5.6.3: User Management
• 4.6.6: Supported Databases
• 4.7: Changelog
• 4.8: DBaaS Guides
• 4.8.1: Backup and Restore via CLI
• 4.8.2: Backup and Restore via DBaaS UI
• 4.8.3: Config Management
• 4.8.4: Create Datastore From Backup
• 4.8.5: Database Db Management
• 4.8.6: Database User Management
• 4.8.7: Datastore Settings
• 4.8.8: DBaaS with Terraform
• 4.8.9: Deploy A Datastore
• 4.8.10: Event Viewer
• 4.8.11: Firewall
• 4.8.12: Logs Viewer
• 4.8.13: Observability
• 4.8.14: Parameter Group
• 4.8.15: Promote A Replica
• 4.8.16: Reboot A Node
• 4.8.17: Restore Backup
• 4.8.18: Scale A Datastore
• 4.8.19: Terraform Provider
• 4.8.20: TLS For Metrics
• 4.8.21: Upgrade Lifecycle Mgmt
• 4.8.22: Connect Kubernetes with DBaaS
• 5: Elastx Cloud Platform API (beta)
• 5.1: ⚡ Quick Start Guide
• 6: Elastx Identity Provider
• 6.1: Overview
• 7: Kubernetes CaaS
• 7.1: Announcements
• 7.2: Overview
• 7.3: Getting started
• 7.3.1: Accessing your cluster
• 7.3.2: Auto Healing
• 7.3.3: Auto Scaling
• 7.3.4: Cluster configuration
• 7.3.5: Cluster upgrades
• 7.3.6: Kubernetes API whitelist
• 7.3.7: Order a new cluster
• 7.3.8: Recommendations
• 7.4: Guides
• 7.4.1: Cert-manager and Cloudflare demo
• 7.4.2: Change PV StorageClass
• 7.4.3: Ingress and cert-manager
• 7.4.4: Install and upgrade cert-manager
• 7.4.5: Install and upgrade ingress-nginx
• 7.4.6: Load balancers
• 7.4.7: Migration to Kubernetes CaaS v2
• 7.4.8: Persistent volumes
• 7.4.9: Your first deployment
• 7.5: Changelog
• 7.5.1: Changelog for Kubernetes 1.34
• 7.5.2: Changelog for Kubernetes 1.33
• 7.5.3: Changelog for Kubernetes 1.32
• 7.5.4: Changelog for Kubernetes 1.31
• 7.5.5: Changelog for Kubernetes 1.30
• 7.5.6: Changelog for Kubernetes 1.29
• 7.5.7: Changelog for Kubernetes 1.28
• 7.5.8: Changelog for Kubernetes 1.27
• 7.5.9: Changelog for Kubernetes 1.26
• 7.5.10: Changelog for Kubernetes 1.25
• 7.5.11: Changelog for Kubernetes 1.24
• 7.5.12: Changelog for Kubernetes 1.23
• 7.5.13: Changelog for Kubernetes 1.22
• 7.5.14: Changelog for Kubernetes 1.21
• 7.5.15: Changelog for Kubernetes 1.20
• 7.5.16: Changelog for Kubernetes 1.19
• 7.5.17: Changelog for Kubernetes 1.18
• 7.5.18: Changelog for Kubernetes 1.17
• 8: Mail Relay
• 8.1: Overview
• 8.2: Announcements
• 8.3: Email and DNS
• 8.4: FAQ
• 8.5: Getting started
• 9: OpenStack IaaS
• 9.1: Announcements
• 9.2: Changelog
• 9.2.1: Changelog for OpenStack Train
• 9.2.2: Changelog for OpenStack Ussuri
• 9.2.3: Changelog for OpenStack Wallaby
• 9.2.4: Changelog for OpenStack Yoga
• 9.3: Overview
• 9.4: Network
• 9.5: Guides
• 9.5.1: Adjutant
• 9.5.2: Affinity Policy
• 9.5.3: API access
• 9.5.4: Application credentials
• 9.5.5: Application credentials - Access Rules
• 9.5.6: Barbican
• 9.5.7: Billing
• 9.5.8: Detach & Attach interface on a Ubuntu instance
• 9.5.9: EC2 Credentials
• 9.5.10: Getting started with OpenStack
• 9.5.11: Octavia
• 9.5.12: Swift getting started
• 9.5.13: Swift projects
• 9.5.14: Swift S3 compatibility
• 9.5.15: Terraform Backend
• 9.5.16: Volume Attachment Limits
• 9.5.17: Volume Backup & Restore
• 9.5.18: Volume migration
• 9.5.19: Volume Retype
• 9.5.20: Windows volume offline after restart
• 10: The Vault (Technical Preview)
• 10.1: Knowledge base
• 10.1.1: FAQ
• 10.2: Backup solutions
• 11: Varnish CDN
• 12: Virtuozzo PaaS
• 12.1: Announcements
• 12.2: Guides
• 12.2.1: Catch-all VirtualHost with 301 redirect on Apache
• 12.2.2: Change public IP without downtime
• 12.2.3: Copy a SQL database
• 12.2.4: Copy files between environments
• 12.2.5: Enable IPv6
• 12.2.6: Enable X11-Forwarding on VPS
• 12.2.7: FAQ
• 12.2.8: Force HTTPS on Apache behind Nginx load balancer
• 12.2.9: Force HTTPS with Tomcat
• 12.2.10: Log real client IP behind a proxy
• 12.2.11: Nginx LB HTTP to HTTPS redirect
• 12.2.12: Nginx redirect to HTTPS
• 12.2.13: Node.JS NPM Module Problems
• 12.2.14: PHP max upload file size
• 12.2.15: Redirect nginx
• 12.2.16: Restrict phpMyAdmin access
• 12.2.17: SMTP on port 25 not working

1 - AI and Other Services

1.1 - Announcements

2024-06-04 AI services now available

Starting with AI to create business value can seem like a complex process. To help our customers to get started with AI we have launched a few different AI introduction services that offer a structured and proven approach to get started. They allow each business to choose which steps to implement and when, based on their unique situation and needs.

The following AI services are now available.

GPU and Private Language Model

Our Private Language Model as a Service is designed to provide you with a secure, cost-effective way to test and verify our future generative AI services within your system.

AI Introduction

AI Introduction is a collaborative starter package by Algorithma and Elastx, designed to efficiently and swiftly kickstart your AI journey.

Responsible AI Training

Join us at our location or invite us to yours for an in-depth session on information security and responsible AI.

GPU and Private Language Model

Our Private Language Model as a Service is designed to provide you with a secure, cost-effective way to test and verify our future generative AI services within your system.

Customized AI Projects

In collaboration with you, Algorithma and Elastx plan and execute tailored AI projects.

You can find information, specifications and pricing here, https://elastx.se/en/ai-services/.

If you have any general questions or would like to sign-up please contact us at hello@elastx.se. For technical questions please register a support ticket at https://support.elastx.se.

2024-02-01 DDoS protection now included

We are happy to announce that we are now adding our DDoS protection to all Elastx Cloud Platform customers for free. This is yet another step in our mission to provide the best cloud platform for business critical services with sensitive data. Your services will now be protected from L3/L4 volumetric DDoS attacks by our inline protection.

If you already are an Elastx customer you do not need to do anything, the DDoS protection service will be enabled automatically. If you are currently subscribing to our DDoS protection service, your subscription will be updated accordingly.

This will apply from 2024-02-01.

L4 proxy and Geo Fencing

We have also added an anycast L4 proxy service with an optional Geo Fencing function.
You can find information, specifications and pricing here, https://elastx.se/en/ddos-protection.

If you have any general questions or would like to sign-up please contact us at hello@elastx.se.
Any technical questions please register a support ticket at https://support.elastx.se.

2023-05-02 Cloud Colocation

We are happy to announce these platform news that will help you to run applications on Elastx Cloud Platform with enhanced security.

Cloud Colocation

Cloud Colocation is a service that enables customers to host their own hardware in the same data centers as the Elastx platform. This could, for example, be hardware security modules (HSM) for storing and managing secrets if required to manage that on your own.

Customers can acquire individual rack units or a separate section in a rack and will receive a dedicated, fast, and private network connection to connect to the desired network in the Elastx platform via Cloud Connect. Customers can also interconnect with other cloud platforms using Elastx Cloud Exchange.

Physical access to the space is provided, and on-site staff is available to offer assistance if needed. Mounting equipment, including cables and rack screws, is included.

Cloud Colocation has been available since 2023-03-04

2023-01-27 Elastx Cloud Platform pricing adjustment

To Elastx Customers and Partners,
We are trying to avoid a general price increase on all services, even though the current high inflation is affecting us hard. We are investing in new more efficient technology to compensate for the increased cost derived from product vendors, utility services, financial service and internal costs. We have identified a few selected services where we need to perform price adjustments to be able to continue the development in a sustainable way.
The new pricing will apply from 2023-03-01.

We will adjust the pricing on Professional Services and also reduce the number of different types to a minimum to make it easier.

We will adjust the pricing on Application Monitoring in order to be able to keep the high service levels and continue the development of the service.

2 - Cloud Connect

General

Cloud Connect is a service for customers to make a physical connection to Elastx public cloud.

Locations

The service is offered in our availability zones:

• Elastx STO1
• Elastx STO2
• Elastx STO3

Customers can also connect at “Private Peering Facilities” locations on PeeringDB.

Media

The service only allows fiber-based connections. The following physical media is supported. Customer needs to specify which media they want to connect with.

10G ports:

• 1000BASE-LX (1310 nm)
• 10GBASE-LR (1310 nm)

100G ports:

• 100GBASE-LR4 (1310 nm)

Data Plane Protocols

• MTU 1500 is default. This can be increased to MTU 9000.
• VLAN encapsulation (IEEE 802.1Q) is recommended. All (1-4093) VLAN numbers are available.
• Link Aggregation Control Protocol (IEEE 802.3ad) is supported.

Control Plane Procotols

• Border Gateway Protocol (BGP) is recommended.

Border Gateway Protocol (BGP)

Elastx currently supports IPv4 Address Family (AFI 1) Unicast (SAFI 1).

Private peering

Customer can advertise any network, including 0/0 for default routing. Elastx needs to informed of the networks prior to advertisement.

Any AS numbers within 64512–65534, 4200000000–4294967294 (RFC 6996) can be used. With the exception of reservation 4258252000-4258252999.

Public AS numbers can be approved after ownership verification.

Public peering

Customer can only advertise networks assigned by Elastx. AS numbers are assigned by Elastx.

Public AS numbers can be approved after ownership verification.

3 - Data Security Manager (beta)

3.1 - File System Encryption for Linux

File System Encryption for Linux

This is a quick setup guide for File System Encryption (FSE) using Elastx Data Security Manager.

1. Log in to Elastx DSM https://hsm.elastx..cloud and enter the account you want to use.
2. Create a Group and set a name. A group can contain multiple secrets but if you want to use quorum approvals you can only manage one FSE per group.
3. Create an app and set a name.
   1. Select the default API Key and the authentication method.
   2. Assigning the app to the group you just created.
4. Get the API Key, select the app you created, under Info > API Key, press the “VIEW API KEY DETAILS” button.
5. Log in to the Linux machine where you want to encrypt data. (These instructions are made for Ubuntu 24.04)
   1. Install fuse.
      sudo apt install libfuse2
   2. Download and install the FSE agent.
      wget https://download.fortanix.com/clients/FSE/1.10.147/fortanix-dsm-fseagent-1.10.147.deb
sudo apt install ./fortanix-dsm-fseagent-1.10.147.deb
   3. Create a directory where the configuration and the encrypted files will be stored and a mount point
      sudo mkdir /fse /data
   4. Configure the file system
      sudo fortanix-dsm-fseagent -dsm -init /fse
      1. Enter the DSM Endpoint: https://hsm.elastx.cloud
      2. Enter the Api Key: <api key>
         There is no text echo, paste the key and press enter.
   5. Mount the filesystem
      sudo fortanix-dsm-fseagent --allow_other /fse /data
      1. Enter the Api Key: <api key>
         (twice)
6. If you want to automatically mount the filesystem at boot do the following.
   1. Add the API key to file /etc/fse-auto-mount/api_keys/1.conf
   2. Add the mount command to file /etc/fse-auto-mount/mount_cmd/1.conf
   3. Reload systemd to apply the changes
      sudo systemctl daemon-reload
   4. Enable the service
      sudo systemctl enable fse-auto-mount@1.service
7. Done

You can find the full documentation here.

4 - DBaaS

4.1 - Announcements

2026-03-31 Elastx Cloud Platform - Compute, Storage and DBaaS

Pricing adjustment

We have absorbed rising operational costs rather than pass them on to you. Despite significant inflationary pressure across the industry, we have managed to keep our prices stable.

The primary driver for our price adjustment is a sharp increase in hardware costs, which have risen 150–250% in the last 6 months. This is a market-wide development entirely outside our control, and one we have worked hard to shield you from for as long as possible. We have reached a point where continuing to do so would compromise our ability to maintain and develop the platform in a sustainable way.

Effective July 1, we will apply a 10% price adjustment to all Compute, Storage and DBaaS services.

We do not take this decision lightly. It is the first time we have increased prices on most of these services, and it reflects the reality of today’s hardware market rather than any change in our commitment to you.

This change will take effect starting July 1, 2026.

Pricing adjustments

We will adjust the pricing on the following services.

2024-10-14 MSSQL DBaaS now available

We are excited to announce that we now have Microsoft SQL Server available in Elastx Database as a Service.

Elastx DBaaS automatically ensures your databases are reliable, secure, and scalable so that your business continues to run without disruption. You can achieve high availability and disaster protection by configuring replication and backups to protect your data. Backups and multi-node datastores are disaster protected as they are running over multiple availability zones which in our case are geographically separated data centers. Automatic failover makes your database highly available.

The following services are included as standard in our prices: 24x7 support, Threat Intelligence, DDoS protection, encrypted traffic between our availability zones and data encryption at rest.

You can find detailed information, specifications and pricing here, https://elastx.se/en/mssql-dbaas.

If you have any general questions or would like to sign-up please contact us at hello@elastx.se. Any technical questions please register a support ticket at https://support.elastx.se.

2024-04-12 ECP DBaaS Generally Available

We are happy to announce that the following services are now generally available (GA) in Elastx Cloud Platform (ECP).

• MySQL DBaaS
• MariaDB DBaaS
• PostgreSQL DBaaS
• Redis DBaaS

The ECP DataBase as a Service gives you a fully managed database with the possibility to run your database in a high availability and disaster protected environment.

It has been available as a public tech-preview since november 2023. A number of updates and features have been added during this period and the service has now reached a maturity level where we can offer this service to all customers.

You can find information, specifications and pricing here, https://elastx.se/en/database. Service documentation is available here, https://docs.elastx.cloud/docs/dbaas/.

If you have any general questions please contact us at hello@elastx.se and you can sign-up for the service here, https://elastx.se/en/signup. Any technical questions please register a support ticket at https://support.elastx.se.

4.2 - DBaaS Getting Started

CCX is a comprehensive data management and storage solution that offers a range of features including flexible node configurations, scalable storage options, secure networking, and robust monitoring tools. It supports various deployment types to cater to different scalability and redundancy needs, alongside comprehensive management functions for users, databases, nodes, and firewalls. The CCX project provides a versatile platform for efficient data handling, security, and operational management, making it suitable for a wide array of applications and workloads.

Deployment Solutions

Our deployment solutions offer customizable configurations for various node types, designed to support both dynamic and ephemeral storage requirements across multiple cloud environments. This includes comprehensive support for a wide range of cloud regions and instances, ensuring flexibility and scalability.

Database Support

Our platform is compatible with a diverse array of database types, including:

• MariaDB
• MySQL
• PostgreSQL
• Cache 22 (deprecated and will be removed in a future release. It is replaced with Valkey)
• Valkey
• Microsoft SQL Server

Node Configurations

We provide support for various node configurations to meet your database needs:

• Replica nodes for MariaDB, MySQL, PostgreSQL, Redis, and Microsoft SQL Server (Single server and Always-On)
• Galera clusters for MariaDB and MySQL

Monitoring and Management

Our platform features advanced monitoring capabilities, offering detailed performance analysis through extensive charts. It enables efficient management of nodes, including:

• Datastore scaling
• Volume scaling
• Promote replica to primary
• Node repair mechanisms

User and Database Administration

We offer sophisticated tools for managing database users and their permissions, ensuring secure access control.

Network Security

Our firewall configuration options are designed to enhance network security, providing robust protection for your data.

Event Logging

The event viewer tracks and displays a comprehensive history of operations performed on the datastore, enhancing transparency and accountability.

Backup and Recovery

Our backup solutions include:

• Incremental and full backup options for comprehensive data protection
• Point-in-time recovery capabilities
• Automated cloud backup uploads with customizable retention periods
• Restoration from separate volumes to optimize datastore space utilization

Customizable Settings

We offer customizable settings for various operational database parameters, allowing for tailored database management.

Account Management

Our platform facilitates user account creation and management, streamlining the login and registration process.

Billing and Payments

Our billing and payment processing tools are designed to simplify financial transactions, including the management of payments and invoices.

Feature Matrix

Each datastore has different features and are suitable for different use cases. Below is a feature matrix showing what operational feature is supported on each datastore:

† : User management features and scope depends on the underlying datastore. There are datastore specific limitations.

4.3 - FAQ

Does CCX provide a High-availability feature

Yes

Can I change the database configuration?

This is a managed service.

Does CCX support multiple AZs.

CCX supports multiple AZs if the Cloud Provider does.

Can write only instances and read only replicas be created?

Yes, but the write-only instances is read-write. It allows both reads and writes.

Does support use a Proxy or Load Balancer (fora example, if there are 2 or more Read Replica instances, then Read can be load balanced across multiple instances).

DNS is used to facilitate this. However, the user can create his own loadbalancer (such as HAProxy or ProxySQL) and connect to the database service. A load balancer should be placed as close as possible to the user’s application. We recommend that the end-user manages the loadbalancer.

Are backups automatic?

Yes, backups are created automatically. The user can set the frequency.

Is there an auto-upgrade SQL version feature, for minor and major updates?

Only minor upgrades. Major upgrades are not supported in an online operation. See product documentation about upgrades (Life-cycle management)

Can it be backed up externally, for example dumping data?

Yes. See product documentation.

Can external data be restored, if yes, how?

See product documentation.

4.4 - Managed Service

CCX is provided as a managed service for your database engine in the cloud(s) you select. This document aims to outline the responsibilities of CCX and what rests with you, as the user.

CCX does:

• Deploy, secure and configure database engines onto virtual machines (VMs) in your chosen cloud
• Allow you to:
  • Configure firewall rules for access
  • Create new databases and users
  • Configure replication for the chosen topology
  • Ensure connectivity between nodes
  • View metrics for each VM and the datastore
  • View query statistics for your service
  • Configure and set a backup schedule for your service
  • Configure a maintenance window that allows CCX to perform maintenance and provide security patches
  • Scale your service horizontally (up and down)
  • Scale your storage vertically (up)
  • Manage and monitor the database to ensure connectivity

The primary responsibility of CCX is to ensure that your datastore is running at all times, reacting to scenarios to ensure this is true. CCX does not access your data or control how you use the databases within your datastore.

In order to achieve this, CCX does not:

• Provide SSH or other ways to access the underlying infrastructure
• Allow superuser access to the managed services
• Allow the modification of settings that are not suitable for production use
• Allow the installation of untrusted extensions or code

If you do have specific requirements, such as:

1. Temporary modification of configuration
2. Feature requests for extensions or different versions of the databases

Please reach out to Support and we will work with you to find a solution.

4.5 - Overview

Elastx DBaaS automatically ensures your databases are reliable, secure, and scalable so that your business continues to run without disruption. It provides full compatibility with the source database engines while reducing operations costs by automating database provisioning and other time-consuming tasks.

Easy high availability and disaster protection by configuring replication, clusters and backups to protect your data. Backups and multi-node datastores are disaster protected as they are running over multiple availability zones which in our case are geographically separated data centers. Automatic failover makes your database highly available. Your data is encrypted at rest, and ISO 27001, 27017, 27018 as well as 14001 compliant.

A Datastore is a database instance with one or more nodes. In the Datastore you can have one or more databases. A datastore can be created with a single node, three nodes in a active/active cluster or a primary node with one or two read only replicas.

You can manage the Datastores with the web-UI. Authentication to the web-UI is done with the Elastx Identity Provider where MFA with TOTP or Yubikey is required. All datastores owned by the organization will be visible for all users with access to that organization.

To access the datastores you need database user credentials which you get and manage for each individual database. You also need to configure the Datastore firewall to allow access from specific IP addresses. The Connection assistant will help you to get the connection string for common programming languages.

In the web-UI you get graphs on key performance metrics on the database and the nodes that will help you to manage capacity and performance. You can scale the Datastore by adding or removing nodes and also to change the size of a node by replacing a node with a different flavor. Contact Elastx support if you need to increase the storage capacity and we will help you. Please note that ephemeral storage can’t be increased unless you change node flavor.

4.6 - Reference

4.6.1 - Datastore Statuses

When you deploy a Datastore, you will see a Status reported in the CCX Dashboard. This article outlines the statuses and what they mean.

4.6.2 - Glossary

4.6.3 - Notifications

CCX notifies users by email in case of certain events. Recipients can be configured on the Datastore Settings page or in the Datastore wizard.

4.6.4 - Observability

4.6.4.1 - Metrics

4.6.4.1.1 - Introduction

CCX uses Prometheus and exporters for monitoring. The monitoring data is exposed though the exports from each node. This is a controlled under the Firewall tab in the CCX UI.

4.6.4.1.2 - MySQL And MariaDB

• MySQL / MariaDB
  • Handler Stats
    • Statistics for the handler. Shown as:
      • Read Rnd
        • Count of requests to read a row based on a fixed position
      • Read Rnd Next
        • Count of requests to read a subsequent row in a data file
      • Read Next
        • Count of requests to read the next row in key order
      • Read Last
        • Count of requests to read the last key in an index
      • Read Prev
        • Count of requests to read the previous row in key order
      • Read First
        • Count of requests to read a row based on an index key value
      • Read Key
        • Count of requests to read the last key in an index
      • Update
        • Count of requests to update a row
      • Write
        • Count of requests to insert to a table
  • Handler Transaction Stats
  • Database Connections
    • Count of connections to the database. Shown as:
      • Thread Connected
        • Count of clients connected to the database
      • Max Connections
        • Count of max connections allowed to the database
      • Max Used Connections
        • Maximum number of connections in use
      • Aborted Clients
        • Number of connections aborted due to client not closing
      • Aborted Connects
        • Number of failed connection attempts
      • Connections
        • Number of connection attempts
  • Queries
    • Count of queries executed
  • Scan Operations
    • Count of operations for the operations: SELECT, UPDATE and DELETE
  • Table Locking
    • Count of table locks. Shown as:
      • Table locks immediate
        • Count of table locks that could be granted immediately
      • Table locks waited
        • Count of locks that had to be waited due to existing locks or another reason
  • Temporary Tables
    • Count of temporary tables created. Shown as:
      • Temporary tables
        • Count of temporary tables created
      • Temporary tables on Disk
        • Count of temporary tables created on disk rather than in memory
  • Sorting
  • Aborted Connections
    • Count of failed or aborted connections to the database. Shown as:
      • Aborted Clients
        • Number of connections aborted due to client not closing
      • Aborted Connects
        • Number of failed connection attempts
      • Access Denied Errors
        • Count of unsuccessful authentication attempts
    • Memory Utilisation

4.6.4.1.3 - PostgreSQL

• PostgreSQL
  • SELECT (fetched)
    • Count of rows fetched by queries to the database
  • SELECT (returned)
    • Count of rows returned by queries to the database
  • INSERT
    • Count of rows inserted to the database
  • UPDATE
    • Count of rows updated in the database
  • DELETE
    • Count of rows deleted in the database
  • Active Sessions
    • Count of currently running queries
  • Idle Sessions
    • Count of connections to the database that are not currently in use
  • Idle Sessions in transaction
    • Count of connections that have begun a transaction but not yet completed while not actively doing work
  • Idle Sessions in transaction (aborted)
    • Count of connections that have begun a transaction but did not complete and were forcefully aborted before they could complete
  • Lock tables
    • Active locks on the database
  • Checkpoints requested and timed
    • Count of checkpoints requested and scheduled
  • Checkpoint sync time
    • Time synchronising checkpoint files to disk
  • Checkpoint write time
    • Time to write checkpoints to disk

4.6.4.1.4 - Redis

• Redis
  • Blocked Clients
    • Clients blocked while waiting on a command to execute
  • Memory Used
    • Amount of memory used by Redis (in bytes)
  • Connected Clients
    • Count of clients connected to Redis
  • Redis commands per second
    • Count of commands processed per second
  • Total keys
    • The total count of all keys stored by Redis
  • Replica Lag
    • The lag (in seconds) between the primary and the replica(s)

4.6.4.1.5 - System

• System - Hardware level metrics for your Datastore
  • Load Average
    • The overall load on your Datastore within the preset period
  • CPU Usage
    • The breakdown of CPU utilisation for your Datastore, including both System and User processes
  • RAM Usage
    • The amount of RAM (in Gigabytes) used and available within the preset period
  • Network Usage
    • The amount of data (in Kilobits or Megabits per second) received and sent within the preset period
  • Disk Usage
    • The total amount of storage used (in Gigabytes) and what is available within the preset period
  • Disk IO
    • The input and output utilisation for your disk within the preset period
  • Disk IOPS
    • The number of read and write operations within the preset period
  • Disk Throughput
    • The amount of data (in Megabytes per second) that is being read from, or written to, the disk within the preset period

4.6.4.1.6 - Valkey

• Valkey
  • Blocked Clients
    • Clients blocked while waiting on a command to execute
  • Memory Used
    • Amount of memory used by Valkey (in bytes)
  • Connected Clients
    • Count of clients connected to Valkey
  • Valkey commands per second
    • Count of commands processed per second
  • Total keys
    • The total count of all keys stored by Valkey
  • Replica Lag
    • The lag (in seconds) between the primary and the replica(s)

4.6.5 - Products

4.6.5.1 - MariaDb

4.6.5.1.1 - Backup

Mariabackup is used to create backups.

CCX backups the Primary server. In multi-primary setups the node with the highest wsrep_local_index is elected.

Backups are streamed directly to S3 staroge.

Mariabackup blocks DDL operations during the backup using the --lock-ddl flag. Any attempt to CREATE, ALTER, DROP, TRUNCATE a table during backup creation will be locked with the status Waiting for backup lock (see SHOW FULL PROCESSLIST). In this case, wait for the backup to finish and, perform the operation later.

Also see the section ‘Schedule’.

Schedule

The backup schedule can be tuned and backups can be paused

4.6.5.1.2 - Configuration

max_connections

• 75 connections / GB of RAM.
• Example: 4GB of RAM yields 300 connections.
• This setting cannot be changed as it affects system stability.

InnoDB settings

• These setting cannot be changed as it affects system stability.

innodb_buffer_pool_size

• 50% of RAM if total RAM is > 4GB
• 25% of RAM if total RAM is <= 4GB

innodb_log_file_size

• 1024 MB if innodb_buffer_pool_size >= 8192MB
• 512 MB if innodb_buffer_pool_size < 8192MB

innodb_buffer_pool_instances

• 8

InnoDB options

General options

Storage

Recommended storage size

• We recommend a maximum of 100GB storage per GB of RAM.
• Example: 4GB of RAM yields 400GB of storage.
• The recommendation is not enforced by the CCX platform.

4.6.5.1.3 - Importing Data

This procedure describes how to import data to a MariaDB datastore located in CCX.

• The MariaDB Datastore on CCX is denoted as the ‘replica’
• The source of the data is denoted as the ‘source’

note:

If you do not want to setup replication, then you can chose to only apply the sections:

• Create a database dump file
• Apply the dumpfile on the replica

Limitations of MariaDB

MariaDB does not offer as fine grained control over privileges as MySQL. Nor does it have the same level of replication features.

The following properties must be respected in order to comply with the SLA:

• There must be no user management happening on the source, while the data is imported and the replication link is active. This is avoid corruption of the mysql database and possibly other system databases.
• It is recommended to set binlog-ignore-db on the source to ‘mysql, performance_schema, and sys’ during the data import/sync process.

Preparations

Ensure that the source is configured to act as a replication source.

• Binary logging is enabled.
• server_id is set to non 0.

Also, prepare the replica with the databases you wish to replicate from the source to the master:

• Using the CCX UI, go to Databases, and issue a Create Database for each database that will be replicated.

Ensure the CCX Firewall is updated:

• Add the replication source as a Trusted Source in the Firewall section of the CCX UI.

Create a replication user on the source

Create a replication user with sufficient privileges on the source:

CREATE USER 'repluser'@'%' IDENTIFIED BY '<SECRET>';
GRANT REPLICATION SLAVE ON *.* TO 'repluser'@'%';

Prepare the replica to replicate from the source

The replica must be instrucuted to replicate from the source. Make sure to change <SOURCE_IP>, <SOURCE_PORT>, and <SECRET>. Run the following on the source:

CHANGE MASTER TO MASTER_HOST=<SOURCE_IP>, MASTER_PORT=<SOURCE_PORT>, MASTER_USER='repluser', MASTER_PASSWORD='<SECRET>', MASTER_SSL=1;

Create a database dump file of the source

The database dump contains the data that you wish to import into the replica. Only partial dumps are possible. The dump must not contains any mysql or other system databases.

danger: The dump must not contains any mysql or other system databases.

On the source, issue the following command. Change ADMIN, SECRET and DATABASES:

mysqldump -uADMIN -p<SECRET>   --master-data --single-transaction --triggers --routines --events --databases DATABASES > dump.sql`

If your database dump contains SPROCs, triggers or events, then you must replace DEFINER. This may take a while:

sed 's/\sDEFINER=`[^`]*`@`[^`]*`//g' -i dump.sql

Apply the dumpfile on the replica

cat dump.sql | mysql -uccxadmin -p -h<REPLICA_PRIMARY>

Start the replica

On the replica do:

START SLAVE

followed by

SHOW SLAVE STATUS;

And verify that:

             Slave_IO_State: Waiting for source to send event
	     ..
  	     Slave_IO_Running: Yes
             Slave_SQL_Running: Yes

When the migration is ready

STOP SLAVE;
RESET SLAVE ALL;

Troubleshooting

If the replication fails to start then verify:

• All the steps above has been followed.
• Ensure that the replication source is added as a Trusted Source in the Firewall section of the CCX UI.
• Ensure that you have the correct IP/FQDN of the replication source.
• Ensure that users are created correctly and using the correct password.
• Ensure that the dump is fresh.

4.6.5.1.4 - Limitations

Every product has limitations. Here is a list MariaDB limitations:

Permissions

The privilege system is not as flexible as in MySQL.

The ‘ccxadmin’ user has the following privileges:

Global / all databases (.):

• CREATE USER, REPLICATION SLAVE, REPLICATION SLAVE ADMIN, SLAVE MONITOR

On databases created from CCX, the admin user can create new users and grant privileges:

• ALL PRIVILEGES WITH GRANT OPTION

This means that users can only create databases from the CCX UI. Once the database has been created from the CCX UI, then the ccxadmin user can create users and grant user privileges on the database using MariaDB CLI.

4.6.5.1.5 - Overview

CCX supports two types of MariaDB clustering:

• MariaDB Replication (Primary-replica configuration)x
• MariaDB Cluster (Multi-primary configuration)

For general purpose applications we recommend using MariaDB Replication, and we only recommend to use MariaDB Cluster if you are migrating from an existing application that uses MariaDB Cluster.

If you are new to MariaDB Cluster we stronly recommend to read about the MariaDB Cluster 10.x limitations and MariaDB Cluster Overview to understand if your application can benefit from MariaDB Cluster.

MariaDB Replication uses the standard asynchronous replication based on GTIDs.

Scaling

Storage and nodes can be scaled online.

Nodes (horizonal)

• The maximum number of database nodes in a datastore is 5.
• Multi-primary configuration must contain an odd number of nodes (1, 3 and 5).

Nodes (vertical)

A node cannot be scaled vertically currently. To scale to large instance type, then a larger instance must be added and then remove the unwanted smaller instances.

Storage

• Maximum size depends on the service provider and instance size
• Volume type cannot currently be changed

Further Reading

• Limitations of MariaDB
• Configuration
• Importing Data

4.6.5.1.6 - Restore

There are two options to restore a backup:

• Restore a backup on the existing datastore
• Restore a backup on a new datastore

Please note that restoring a backup may be a long running process.

This option allows to restore a backup with point in time recovery. The WAL logs are replayed until the desired PITR.

Warning! Running several restores may change the timelines.

This option allows to restore a backup on a new datastore. This option does not currently support PITR.

4.6.5.1.7 - TLS Connection

SSL Modes

CCX currently supports connections to MariaDB in two SSL modes:

1. REQUIRED: This mode requires an SSL connection. If a client attempts to connect without SSL, the server rejects the connection.
2. VERIFY_CA: This mode requires an SSL connection and the server must verify the client’s certificate against the CA certificates that it has.

CA Certificate

The Certificate Authority (CA) certificate required for VERIFY_CA mode can be downloaded from your datastore on CCX using an API call or through the user interface on page https://{your_ccx_domain}/projects/default/data-stores/{datastore_id}/settings. This certificate is used for the VERIFY_CA SSL mode.

Example Commands

Here are example commands for connecting to the MySQL server using the two supported SSL modes:

1. REQUIRED mode:

   mysql --ssl-mode=REQUIRED -u username -p -h hostname
   

2. VERIFY_CA mode:

   mysql --ssl-mode=VERIFY_CA --ssl-ca=ca.pem -u username -p -h hostname
   

require_secure_transport

This is a MariaDB setting that governs if connections to the datastore are required to use SSL. You can change this setting in CCX in Settings -> DB Parameters

Examples

ssl-mode=DISABLED and require_secure_transport=OFF

mysql -uccxadmin -p -h...  -P3306 ccxdb --ssl-mode=disabled
...
mysql> \s
--------------
...
Connection id:		52
Current database:	ccxdb
Current user:		ccxadmin@...
*SSL:			Not in use*
Current pager:		stdout
...

ssl-mode=PREFERRED and require_secure_transport=OFF

mysql -uccxadmin -p -h...  -P3306 ccxdb --ssl-mode=preferred
...
mysql> \s
--------------
...
Connection id:		52
Current database:	ccxdb
Current user:		ccxadmin@...
SSL:			Cipher in use is TLS_AES_256_GCM_SHA384
Current pager:		stdout
...

ssl-mode=DISABLED and require_secure_transport=ON

mysql -uccxadmin -p -h...  -P3306 ccxdb --ssl-mode=disabled
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 3159 (08004): Connections using insecure transport are prohibited while --require_secure_transport=ON.

ssl-mode=PREFERRED|REQUIRED and require_secure_transport=ON

mysql -uccxadmin -p -h...  -P3306 ccxdb --ssl-mode=preferred|required
mysql> \s
--------------
...
Connection id:		52
Current database:	ccxdb
Current user:		ccxadmin@...
SSL:			Cipher in use is TLS_AES_256_GCM_SHA384
Current pager:		stdout
...

tls_version

The tls_version is set to the following by default:

4.6.5.2 - MSSQLServer

4.6.5.2.1 - Configurations

Important default values

max_connections

• SQL Server has no direct “max connection per GB of RAM” rule. The actual number of user connections allowed depends on the version of SQL Server that you are using, and also the limits of your application(s), and hardware.
• SQL Server allows a maximum of 32,767 user connections.
• User connections is a dynamic (self-configuring) option, SQL Server adjusts the maximum number of user connections automatically as needed, up to the maximum value allowable.
• In most cases, you do not have to change the value for this option. The default is 0, which means that the maximum (32,767) user connections are allowed.
• To determine the maximum number of user connections that your system allows, you can execute sp_configure or query the sys.configuration catalog view.
• For more info: https://learn.microsoft.com/en-us/sql/database-engine/configure-windows/configure-the-user-connections-server-configuration-option?view=sql-server-ver16&viewFallbackFrom=sql-server-ver16.

4.6.5.2.2 - Limitations

Every product has limitations. Below is a list of Microsoft SQL Server limitations:

License

• The standard license is applied.

Configurations

• Single node (no High Availability)
• Always On (2 nodes, asynchronous commit mode, High Availability)

Always On-specific limitations

• Refer to the Microsoft standard license for a complete list of limitations.
• Only asynchronous commit mode is currently supported.
• The ccxdb is currently the only supported Always On enabled database.
• Scaling is not supported as the standard license does not permit more than two nodes.

User-created databases (not Always On) are not transferred to the replica

• In the Always On configuration, only the ccxdb is replicated.
• Data loss may occur for other user-created databases, as they are not transferred to the replica during the add node process. Therefore, they may be lost if a failover, automatic repair, or any other life-cycle management event occurs.

4.6.5.2.3 - Overview

CCX supports two Microsoft SQLServer 2022 configurations:

• Single-node (No high-availability)
• Always-on, two nodes, async-commit mode (high-availability) in an primary-replica configuration.

The ‘standard’ license is applied.

Scaling

Scaling is not supported in SQLServer as of the standard license.

Storage

• Maximum size depends on the service provider and instance size
• Volume type cannot currently be changed

Further Reading

• Limitations

4.6.5.2.4 - User Management

CCX supports creating database users from the web interface.

The database user is created as follows:

CREATE LOGIN username WITH PASSWORD = 'SECRET',  DEFAULT_DATABASE=[master], CHECK_EXPIRATION=OFF,  CHECK_POLICY=OFF
ALTER SERVER ROLE [sysadmin] ADD MEMBER [username]

4.6.5.3 - MySQL

4.6.5.3.1 - Backup

Percona Xtrabackup is used to create backups.

CCX backups the Primary server. In multi-primary setups the node with the highest wsrep_local_index is elected.

Backups are streamed directly to S3 staroge.

Percona Xtrabackup blocks DDL operations during the backup using the --lock-ddl flag. Any attempt to CREATE, ALTER, DROP, TRUNCATE a table during backup creation will be locked with the status Waiting for backup lock (see SHOW FULL PROCESSLIST). In this case, wait for the backup to finish and, perform the operation later.

Also see the section ‘Schedule’.

Schedule

The backup schedule can be tuned and backups can be paused.

4.6.5.3.2 - Configuration

max_connections

• 75 connections / GB of RAM.
• Example: 4GB of RAM yields 300 connections.
• This setting cannot be changed as it affects system stability.

InnoDB settings

• These setting cannot be changed as it affects system stability.

innodb_buffer_pool_size

• 50% of RAM if total RAM is > 4GB
• 25% of RAM if total RAM is <= 4GB

innodb_log_file_size

• 1024 MB if innodb_buffer_pool_size >= 8192MB
• 512 MB if innodb_buffer_pool_size < 8192MB

innodb_buffer_pool_instances

• 8

InnoDB options

General options

Storage

Recommended storage size

• We recommend a maximum of 100GB storage per GB of RAM.
• Example: 4GB of RAM yields 400GB of storage.
• The recommendation is not enforced by the CCX platform.

4.6.5.3.3 - Importing Data

This procedure describes how to import data to a MySQL datastore located in CCX.

• The MySQL Datastore on CCX is denoted as the ‘replica’
• The source of the data is denoted as the ‘source’

note:

If you dont want to setup replication, then you can chose to only apply the sections:

• Create a database dump file
• Apply the dumpfile on the replica

Preparations

Ensure that the source is configured to act as a replication source:

• Binary logging is enabled.
• server_id is set to non 0.

Ensure the CCX Firewall is updated:

• Add the replication source as a Trusted Source in the Firewall section of the CCX UI.

Create a replication user on the source

Create a replication user with sufficient privileges on the source:

CREATE USER 'repluser'@'%' IDENTIFIED BY '<SECRET>';
GRANT REPLICATION SLAVE ON *.* TO  'repluser'@'%';

Prepare the replica to replicate from the source

The replica must be instructed to replicate from the source:

Make sure to change <SOURCE_IP>, <SOURCE_PORT>, and <SECRET>.

CHANGE REPLICATION SOURCE TO SOURCE_HOST=<SOURCE_IP>, SOURCE_PORT=<SOURCE_PORT>, SOURCE_USER='repluser', SOURCE_PASSWORD='<SECRET>', SOURCE_SSL=1;

Create a replication filter on the replica

The replica filter prevents corruption of the datastore.

If the datastore’s system tables are corrupted using replication then the SLA is void and the datastore must be recreated.

CHANGE REPLICATION FILTER REPLICATE_IGNORE_DB=(mysql,sys, performance_schema);

Create a database dump file

The database dump contains the data that you wish to import into the replica. Only partial dumps are possible. The dump must not contains any mysql or other system databases.

On the source, issue the following command. Change USER, SECRET and DATABASES:

mysqldump --set-gtid-purged=OFF -uUSER -pSECRET   --master-data --single-transaction --triggers --routines --events  --databases DATABASES > dump.sql

Important! If your database dump contains SPROCs, triggers or events, then you must replace DEFINER:

sed 's/\sDEFINER=`[^`]*`@`[^`]*`//g' -i dump.sql

Apply the dumpfile on the replica

cat dump.sql | mysql -uccxadmin -p -h<REPLICA_PRIMARY>

Start the replica

On the replica do:

START REPLICA;

followed by

SHOW REPLICA STATUS;

And verify that:

             Replica_IO_State: Waiting for source to send event
	     ..
  	     Replica_IO_Running: Yes
             Replica_SQL_Running: Yes

When the migration is ready

STOP REPLICA;
RESET REPLICA ALL;
CHANGE REPLICATION FILTER REPLICATE_IGNORE_DB=();

Troubleshooting

If the replication fails to start then verify:

• All the steps above has been followed.
• Ensure that the replication source is added as a Trusted Source in the Firewall section of the CCX UI.
• Ensure that you have the correct IP/FQDN of the replication source.
• Ensure that users are created correctly and using the correct password.
• Ensure that the dump is fresh.

4.6.5.3.4 - Importing Data From AWS RDS

This procedure describes how to import data from Amazon RDS to a MySQL datastore located in CCX.

• The MySQL Datastore on CCX is referred to as the ‘CCX Primary’
• The RDS Source of the data is referred to as the ‘RDS Writer’

Schematically, this is what we will set up:

warning:

AWS RDS makes it intentionally difficult to migrate away from. Many procedures on the internet, as well as AWS’s own procedures, will not work.

The migration we suggest here (and is the only one we know works) requires that the RDS Writer instance be blocked for writes until a mysqldump has been completed. However, AWS RDS blocks operations such as FLUSH TABLES WITH READ LOCK: mysqldump: Couldn't execute 'FLUSH TABLES WITH READ LOCK': Access denied for user 'admin'@'%' (using password: YES) (1045) Therefore, the actual applications must be blocked from writing.

Also, some procedures on the internet suggest creating a read-replica. This will not work either, as the AWS RDS Read-replica is crippled and lacks GTID support.

note:

If you don’t want to set up replication, you can choose to only apply the following sections:

• Create a database dump file of the RDS Writer
• Apply the dump file on the CCX replica

Also, practice this a few times before you actually do the migration.

Preparations

• Create a datastore on CCX. Note that you can also replicate from MySQL 8.0 to MySQL 8.4.
• Get the endpoint of the CCX Primary (under the Nodes section). The endpoint in our case is db-9bq15.471ed518-8524-4f37-a3b2-136c68ed3aa6.user-ccx.mydbservice.net.
• Get the endpoint of the RDS Writer. In this example, the endpoint is database-1.cluster-cqc4xehkpymd.eu-north-1.rds.amazonaws.com
• Update the Security group on AWS RDS to allow the IP address of the CCX Primary to connect. To get the IP address of the CCX Primary, run:

  dig db-9bq15.471ed518-8524-4f37-a3b2-136c68ed3aa6.user-ccx.mydbservice.net
  

• Ensure you can connect a MySQL client to both the CCX Primary and the RDS Writer.

Create a Replication User On the RDS Writer Instance

Create a replication user with sufficient privileges on the RDS Writer. In the steps below, we will use repl and replpassword as the credentials when setting up the replica on CCX.

CREATE USER 'repl'@'%' IDENTIFIED BY 'replpassword';
GRANT REPLICATION SLAVE ON *.* TO  'repluser'@'%'; #mysql 8.0
GRANT REPLICATION REPLICATION_SLAVE_ADMIN ON *.* TO  'repluser'@'%';

Block Writes to the RDS Writer Instance

This is the most challenging part. You must ensure your applications cannot write to the Writer instance. Unfortunately, AWS RDS blocks operations like FLUSH TABLES WITH READ LOCK.

Create a Consistent Dump

Assuming that writes are now blocked on the RDS Writer Instance, you must get the binary log file and the position of the RDS Writer instance.

Get the Replication Start Position

The start position (binary log file name and position) is used to tell the replica where to start replicating data from.

MySQL 8.0: SHOW MASTER STATUS\G
MySQL 8.4 and later: SHOW BINARY LOG STATUS\G

It will output:

 *************************** 1. row ***************************
             File: mysql-bin-changelog.000901
         Position: 584
     Binlog_Do_DB:
 Binlog_Ignore_DB:
Executed_Gtid_Set: 796aacf3-24ed-11f0-949d-0605a27ab4b9:1-876
1 row in set (0.02 sec)

Record the File: mysql-bin-changelog.000901 and the Position: 584 as they will be used to set up replication.

Create the mysqldump

Be sure to specify the database you wish to replicate. You must omit any system databases. In this example, we will dump the databases prod and crm.

mysqldump -uadmin -p -hdatabase-1.cluster-cqc4xehkpymd.eu-north-1.rds.amazonaws.com --databases prod crm --triggers --routines --events --set-gtid_purged=OFF --single-transaction  > dump.sql

Wait for it to complete.

Unblock Writes to the RDS Writer Instance

At this stage, it is safe to enable application writes again.

Load the Dump On the Replica

Create a Replication Filter On the Replica

The replica filter prevents corruption of the datastore, and we are not interested in changes logged by AWS RDS to mysql.rds* tables anyway. Also add other databases that you do not wish to replicate to the filter.

note:

If the CCX datastore’s system tables are corrupted using replication, then the datastore must be recreated.

CHANGE REPLICATION FILTER REPLICATE_IGNORE_DB=(mysql, sys, performance_schema);

Important! If your database dump contains stored procedures, triggers, or events, then you must replace DEFINER:

sed 's/\sDEFINER=`[^`]*`@`[^`]*`//g' -i dump.sql

Apply the Dump File On the CCX Primary:

cat dump.sql | mysql -uccxadmin -p -hCCX_PRIMARY

Connect the CCX Primary to the RDS Writer Instance

The CCX Primary must be instructed to replicate from the RDS Writer. We have the binlog file and position from the earlier step:

• mysql-bin-changelog.000901
• 584

CHANGE REPLICATION SOURCE TO SOURCE_HOST='database-1.cluster-cqc4xehkpymd.eu-north-1.rds.amazonaws.com', SOURCE_PORT=3306, SOURCE_USER='repl', SOURCE_PASSWORD='replpassword', SOURCE_SSL=1, SOURCE_LOG_FILE='mysql-bin-changelog.000901', SOURCE_LOG_POS=584;

Start the Replica

On the replica, run:

START REPLICA;

followed by:

SHOW REPLICA STATUS;

And verify that:

             Replica_IO_State: Waiting for source to send event
	            ...
  	       Replica_IO_Running: Yes
          Replica_SQL_Running: Yes

When the Migration is Ready

At some point, you will need to point your applications to the new datastore. Ensure:

• Prevent writes to the RDS Writer
• Make sure the CCX Primary has applied all data (use SHOW REPLICA STATUS)
• Connect the applications to the new datastore

STOP REPLICA;
RESET REPLICA ALL;
CHANGE REPLICATION FILTER REPLICATE_IGNORE_DB=();

Troubleshooting

If the replication fails to start, verify:

• All the steps above have been followed
• Ensure that the IP address of the CCX Primary is added to the security group used by the RDS Writer instance
• Ensure that you have the correct IP/FQDN of the RDS Writer instance
• Ensure that users are created correctly and using the correct password
• Ensure that the dump is fresh

4.6.5.3.5 - Importing Data From GCP

This procedure describes how to import data from Google Cloud SQL to a MySQL datastore located in CCX.

• The MySQL Datastore on CCX is referred to as the ‘CCX Primary’
• The GCP Source of the data is referred to as the ‘GCP Primary’

Schematically, this is what we will set up:

note:

If you don’t want to set up replication, you can choose to only apply the following sections:

• Create a database dump file of the GCP Primary
• Apply the dump file on the CCX replica

Also, practice this a few times before you actually do the migration.

Preparations

• Create a datastore on CCX. Note that you can also replicate from MySQL 8.0 to MySQL 8.4.
• Get the endpoint of the CCX Primary (under the Nodes section). The endpoint in our case is db-9bq15.471ed518-8524-4f37-a3b2-136c68ed3aa6.user-ccx.mydbservice.net.
• The GCP Primary must have a Public IP.
• Get the endpoint of the GCP Primary. In this example, the endpoint is 34.51.xxx.xxx
• Update the Security group on GCP to allow the IP address of the CCX Primary to connect. To get the IP address of the CCX Primary, run:

  dig db-9bq15.471ed518-8524-4f37-a3b2-136c68ed3aa6.user-ccx.mydbservice.net
  

• Ensure you can connect a MySQL client to both the CCX Primary and the GCP Primary.

Create a Replication User on the GCP Primary Instance

Create a replication user with sufficient privileges on the GCP Primary. In the steps below, we will use repl and replpassword as the credentials when setting up the replica on CCX.

CREATE USER 'repl'@'%' IDENTIFIED BY 'replpassword';
GRANT REPLICATION SLAVE ON *.* TO  'repluser'@'%'; #mysql 8.0
GRANT REPLICATION REPLICATION_SLAVE_ADMIN ON *.* TO  'repluser'@'%';

Create the mysqldump

Be sure to specify the database you wish to replicate. You must omit any system databases. In this example, we will dump the databases prod and crm.

mysqldump -uroot -p -h34.51.xxx.xxx --databases prod crm --triggers --routines --events --set-gtid_purged=OFF --source-data --single-transaction > dump.sql

Wait for it to complete.

Load the Dump on the Replica

Create a Replication Filter on the Replica

The replica filter prevents corruption of the datastore, and we are not interested in changes logged by GCP to mysql.rds* tables anyway. Also add other databases that you do not wish to replicate to the filter.

note:

If the CCX datastore’s system tables are corrupted using replication, then the datastore must be recreated.

CHANGE REPLICATION FILTER REPLICATE_IGNORE_DB=(mysql, sys, performance_schema);

Important! If your database dump contains stored procedures, triggers, or events, then you must replace DEFINER:

sed 's/\sDEFINER=`[^`]*`@`[^`]*`//g' -i dump.sql

Apply the Dump File on the CCX Primary:

cat dump.sql | mysql -uccxadmin -p -hCCX_PRIMARY

Connect the CCX Primary to the GCP Primary

Issue the following commands on the CCX Primary:

CHANGE REPLICATION SOURCE TO SOURCE_HOST='34.51.xxx.xxx', SOURCE_PORT=3306, SOURCE_USER='repl', SOURCE_PASSWORD='replpassword', SOURCE_SSL=1;

Start the Replica

On the CCX Primary, run:

START REPLICA;

followed by:

SHOW REPLICA STATUS\G

And verify that:

             Replica_IO_State: Waiting for source to send event
	     ..
  	     Replica_IO_Running: Yes
             Replica_SQL_Running: Yes

When the Migration is Ready

At some point, you will need to point your applications to the new datastore. Ensure:

• There are no application writes to the GCP Primary
• The CCX Primary has applied all data (use SHOW REPLICA STATUS \G, check the Seconds_Behind_Master)
• Connect the applications to the new datastore

Then you can clean up the replication link on the CCX Primary:

STOP REPLICA;
RESET REPLICA ALL;
CHANGE REPLICATION FILTER REPLICATE_IGNORE_DB=();

Troubleshooting

If the replication fails to start, verify:

• All the steps above have been followed
• Ensure that the IP address of the CCX Primary is added to the security group used by the GCP Primary instance
• Ensure that you have the correct IP/FQDN of the GCP Primary instance
• Ensure that users are created correctly and using the correct password
• Ensure that the dump is fresh

4.6.5.3.6 - Limitations

Every product has limitations. Here is a list MySQL limitations:

Permissions

The privilege system in MySQL is offers more capabilties than MariaDB. Hence, the ‘ccxadmin’ user has more privileges in MySQL than in MariaDB.

The ‘ccxadmin’ user has the following privileges:

• Global / all databases (*.*):
  • SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, RELOAD, PROCESS, REFERENCES, INDEX, ALTER, SHOW DATABASES, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, REPLICATION SLAVE, REPLICATION CLIENT, CREATE VIEW, REPLICATION_SLAVE_ADMIN, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, CREATE USER, EVENT, TRIGGER, GRANT

This means that the ‘ccxadmin’ may assign privileges to users on all databases.

Restrictions:

‘ccxadmin’ is not allowed to modify the following databases

• mysql.*
• sys.*

For those database, the following privileges have been revoked from ‘ccxadmin’:

• INSERT, UPDATE, DELETE, CREATE, DROP, REFERENCES, INDEX, ALTER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, EVENT, TRIGGER

4.6.5.3.7 - Overview

CCX supports two types of MySQL clustering:

• MySQL Replication (Primary-replica configuration)
• Percona XtraDb Cluster (Multi-primary configuration)

For general purpose applications we recommend using MySQL Replication, and we only recommend to use Percona XtraDb Cluster if you are migrating from an existing application that uses Percona XtraDb Cluster.

If you are new to Percona XtraDb Cluster we stronly recommend to read about the Percona XtraDb Cluster limitations and Percona XtraDb Cluster Overview to understand if your application can benefit from Percona XtraDb Cluster.

MySQL Replication uses the standard asynchronous replication based on GTIDs.

Scaling

Storage and nodes can be scaled online.

Nodes (horizonal)

• The maximum number of database nodes in a datastore is 5.
• Multi-primary configuration must contain an odd number of nodes (1, 3 and 5).

Nodes (vertical)

A node cannot be scaled vertically currently. To scale to large instance type, then a larger instance must be added and then remove the unwanted smaller instances.

Storage

• Maximum size depends on the service provider and instance size
• Volume type cannot currently be changed

Further Reading

• Limitations
• Configuration
• Importing Data

4.6.5.3.8 - Restore

There are two options to restore a backup:

• Restore a backup on the existing datastore
• Restore a backup on a new datastore

Please note that restoring a backup may be a long running process.

This option allows to restore a backup with point in time recovery. The WAL logs are replayed until the desired PITR. Warning! Running several restores may change the timelines.

This option allows to restore a backup on a new datastore. This option does not currently support PITR.

4.6.5.3.9 - TLS Connection

SSL Modes

CCX currently supports connections to MySQL in two SSL modes:

1. REQUIRED: This mode requires an SSL connection. If a client attempts to connect without SSL, the server rejects the connection.

2. VERIFY_CA: This mode requires an SSL connection and the server must verify the client’s certificate against the CA certificates that it has.

CA Certificate

The Certificate Authority (CA) certificate required for VERIFY_CA mode can be downloaded from your datastore on CCX using an API call or through the user interface on page https://{your_ccx_domain}/projects/default/data-stores/{datastore_id}/settings. This certificate is used for the VERIFY_CA SSL mode.

Example Commands

Here are example commands for connecting to the MySQL server using the two supported SSL modes:

1. REQUIRED mode:

   mysql --ssl-mode=REQUIRED -u username -p -h hostname
   

2. VERIFY_CA mode:

   mysql --ssl-mode=VERIFY_CA --ssl-ca=ca.pem -u username -p -h hostname
   

require_secure_transport

This is a MySQL setting that governs if connections to the datastore are required to use SSL. You can change this setting in CCX in Settings -> DB Parameters:

Examples

ssl-mode=DISABLED and require_secure_transport=OFF

mysql -uccxadmin -p -h...  -P3306 ccxdb --ssl-mode=disabled
...
mysql> \s
--------------
...
Connection id:		52
Current database:	ccxdb
Current user:		ccxadmin@...
*SSL:			Not in use*
Current pager:		stdout
...

ssl-mode=PREFERRED and require_secure_transport=OFF

mysql -uccxadmin -p -h...  -P3306 ccxdb --ssl-mode=preferred
...
mysql> \s
--------------
...
Connection id:		52
Current database:	ccxdb
Current user:		ccxadmin@...
SSL:			Cipher in use is TLS_AES_256_GCM_SHA384
Current pager:		stdout
...

ssl-mode=DISABLED and require_secure_transport=ON

mysql -uccxadmin -p -h...  -P3306 ccxdb --ssl-mode=disabled
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 3159 (08004): Connections using insecure transport are prohibited while --require_secure_transport=ON.

ssl-mode=PREFERRED|REQUIRED and require_secure_transport=ON

mysql -uccxadmin -p -h...  -P3306 ccxdb --ssl-mode=preferred|required
mysql> \s
--------------
...
Connection id:		52
Current database:	ccxdb
Current user:		ccxadmin@...
SSL:			Cipher in use is TLS_AES_256_GCM_SHA384
Current pager:		stdout
...

tls_version

The tls_version is set to the following by default:

4.6.5.3.10 - User Management

CCX supports creating database users from the web interface. The database user has the following privileges:

• Global / all databases (*.*):
  • SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, RELOAD, PROCESS, REFERENCES, INDEX, ALTER, SHOW DATABASES, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, REPLICATION SLAVE, REPLICATION CLIENT, CREATE VIEW, REPLICATION_SLAVE_ADMIN, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, CREATE USER, EVENT, TRIGGER, GRANT

This means that the database user may assign privileges to users on all databases.

Restrictions:

The database user is not allowed to modify the following databases

• mysql.*
• sys.*

For those database, the following privileges have been revoked from ‘ccxadmin’:

• INSERT, UPDATE, DELETE, CREATE, DROP, REFERENCES, INDEX, ALTER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, EVENT, TRIGGER

4.6.5.4 - PostgreSQL

4.6.5.4.1 - Backup

Pg_basebackup is used to create backup. Also see the section ‘Schedule’.

CCX backups the Secondary server.

Backups are streamed directly to S3 staroge.

Schedule

The backup schedule can be tuned and backups can be paused

4.6.5.4.2 - Configuration

These settings cannot be changed as it affects system stability

Important default values

Max connections

The maximum number of connections depends on the instance size. The number of connections can be scaled by adding a new database secondary allowing of a larger instance size. The new replica can then be promoted to the new primary. See Promoting a replica for more information.

Archive mode

All nodes are configured with archive_mode=always.

Auto-vacuum

Auto-vacuum settings are set to default. Please read more about automatic vaccuming here

4.6.5.4.3 - Extensions

Supported extentions

Creating an extension

Connect to PostgreSQL using an admin account (e.g ccxadmin).

CREATE EXTENSION vector;
CREATE EXTENSION

See Postgres documentation for more information.

4.6.5.4.4 - Importing Data

This procedure describes how to import data to a PostgreSQL datastore located in CCX.

• The PostgreSQL Datastore on CCX is denoted as the ‘replica’
• The source of the data is denoted as the ‘source’

Create a database dump file

Dump the database schema from the <DATABASE> you wish to replicate:

pg_dump --no-owner -d<DATABASE> > /tmp/DATABASE.sql

Apply the dumpfile on the replica

postgres=# CREATE DATABASE <DATABASE>;

Copy the DSN from Nodes, Connection Information in the CCX UI. Change ‘ccxdb’ to <DATABASE>:

psql postgres://ccxadmin:.../<DATABASE> <  /tmp/DATABASE.sql

4.6.5.4.5 - Limitations

Every product has limitations. Here is a list PostgreSQL limitations:

Permissions

PostgreSQL users are created with the following permissions:

• NOSUPERUSER, CREATEROLE, LOGIN, CREATEDB

4.6.5.4.6 - Restore

Postgres configures archive_command and archive_mode=always. Morever, during the restore the restore_command is set.

There are two options to restore a backup:

• Restore a backup on the existing datastore
• Restore a backup on a new datastore

Please note that restoring a backup may be a long running process.

This option allows to restore a backup with point in time recovery. The WAL logs are replayed until the desired PITR. Warning! Running several restores may change the timelines.

This option allows to restore a backup on a new datastore. This option does not currently support PITR.

4.6.5.5 - Redis

4.6.5.5.1 - Backup

A backup of Redis consists of both RDB and AOF.

Schedule

The backup schedule can be tuned and backups can be paused

4.6.5.5.2 - Configuration

Volume size

Since Redis is an in-memory database, the storage size is fixed and twice the amount of the RAM. Thus, it is not possible to:

• specify the storage size in the deployment wizard.
• scale the storage.

Persistance

Redis is configured to use both AOF and RDB for persistance. The following configuration parameters are set:

• appendonly yes
• default values for AOF
• default values for RDB

4.6.5.5.3 - User Management

CCX simplifies Redis user management by providing a clear and intuitive user interface for managing privileges and accounts. Below are detailed instructions and explanations for managing Redis users within the CCX environment.

Viewing Existing Users

To view existing Redis users:

1. Navigate to the Users section in your CCX Redis cluster.
2. Here you’ll see a list of existing user accounts along with their associated privileges.

User Information Displayed:

• Account: Username of the Redis user.
• Privileges: Specific privileges granted or filtered out.
• Actions: Options to manage (modify/delete) the user.

Note: By default, the -@admin and -@dangerous privileges are filtered out for security purposes.

Creating a New Redis Admin User

To create a new Redis admin user:

1. Click on the Create Admin user button.

2. Fill in the required fields:

   • Username: Enter the desired username.
   • Password: Enter a secure password for the user.
   • Categories: Enter the privilege categories. By default, using +@all will grant all privileges except those explicitly filtered (like -@admin and -@dangerous).

3. Optionally, you can define more granular restrictions:

   • Commands: Enter commands to explicitly allow (+) or disallow (-). For example:

     • Allow command: +get
     • Disallow command: -get

   • Channels: Specify Redis Pub/Sub channels. You can allow (&channel) or disallow (-&channel).

   • Keys: Specify key access patterns. Use the syntax ~key to allow or ~-key to disallow access to specific keys or patterns.

4. After completing the form, click on the Create button to save the new user.

Default Privilege Filtering

CCX ensures the security of your Redis instance by automatically filtering potentially harmful privileges:

• -@admin: Restricts administrative commands.
• -@dangerous: Restricts commands that could compromise the cluster’s stability.

These privileges cannot be granted through CCX’s standard user interface for security reasons.

Firewall and Access Control

User accounts in CCX Redis clusters are protected by built-in firewall rules:

• Accounts are only allowed to connect from trusted sources defined in the firewall settings.

Ensure your firewall rules are properly configured to maintain secure access control to your Redis users.

4.6.5.6 - Valkey

4.6.5.6.1 - Backup

A backup of Valkey consists of both RDB and AOF.

Schedule

The backup schedule can be tuned and backups can be paused

4.6.5.6.2 - Configuration

Volume size

Since Valkey is an in-memory database, the storage size is fixed and twice the amount of the RAM. Thus, it is not possible to:

• specify the storage size in the deployment wizard.
• scale the storage.

Persistance

Redis is configured to use both AOF and RDB for persistance. The following configuration parameters are set:

• appendonly yes
• default values for AOF
• default values for RDB

4.6.5.6.3 - User Management

CCX simplifies Valkey user management by providing a clear and intuitive user interface for managing privileges and accounts. Below are detailed instructions and explanations for managing Valkey users within the CCX environment.

Viewing Existing Users

To view existing Valkey users:

1. Navigate to the Users section in your CCX Valkey cluster.
2. Here you’ll see a list of existing user accounts along with their associated privileges.

User Information Displayed:

• Account: Username of the Valkey user.
• Privileges: Specific privileges granted or filtered out.
• Actions: Options to manage (modify/delete) the user.

Note: By default, the -@admin and -@dangerous privileges are filtered out for security purposes.

Creating a New Valkey Admin User

To create a new Valkey admin user:

1. Click on the Create Admin user button.

2. Fill in the required fields:

   • Username: Enter the desired username.
   • Password: Enter a secure password for the user.
   • Categories: Enter the privilege categories. By default, using +@all will grant all privileges except those explicitly filtered (like -@admin and -@dangerous).

3. Optionally, you can define more granular restrictions:

   • Commands: Enter commands to explicitly allow (+) or disallow (-). For example:

     • Allow command: +get
     • Disallow command: -get

   • Channels: Specify Valkey Pub/Sub channels. You can allow (&channel) or disallow (-&channel).

   • Keys: Specify key access patterns. Use the syntax ~key to allow or ~-key to disallow access to specific keys or patterns.

4. After completing the form, click on the Create button to save the new user.

Default Privilege Filtering

CCX ensures the security of your Valkey instance by automatically filtering potentially harmful privileges:

• -@admin: Restricts administrative commands.
• -@dangerous: Restricts commands that could compromise the cluster’s stability.

These privileges cannot be granted through CCX’s standard user interface for security reasons.

Firewall and Access Control

User accounts in CCX Valkey clusters are protected by built-in firewall rules:

• Accounts are only allowed to connect from trusted sources defined in the firewall settings.

Ensure your firewall rules are properly configured to maintain secure access control to your Valkey users.

4.6.6 - Supported Databases

4.7 - Changelog

V 1.55

Overview

Changes has been introduced in user onboarding flow. Important security fixes has been deployed.

Changes

• Change in user onboarding flow.

Important fixes

• Fixed CMON config invalid save.
• Single server MSSQL datastore creation fails unless 1 node is manually chosen.
• UI fix for create datastore from backup when changing node configuration.
• Backend using 8.0.6 Valkey resolving security issues (CVE-2025-49844).

V 1.54

Overview

Support for new PostgreSQL and MariaDB versions. Added more Valkey modules.

Changes

• PostgreSQL 17.
• MariaDB 11.8 including MariaDB Vector.
• Improved instance selection.
• Added valkey-search and valkey-json.

Important fixes

• Fixed missing add/extend storage option for MSSQL.
• Fixed issue with PostgreSQL 17 unable to set DB parameters.
• Corrected disk utilization charts (sum vs. max).

V 1.53

Overview

Valkey is now replacing Redis as option for creating new datastores. The monitoring charts have been improved and refreshed.

Changes

• Improved backup handling.
• Customer log UI improvements.
• Terraform functionality extensions.

Important fixes

• Fixed problem with DB parameter acceptance and group synchronisation.
• Corrected a problem with incorrect volume type for expanded volumes.
• Improved datastore status transitions during upgrades.

V 1.51

Overview

A new way to group custom database parameters is introduced. This allows to apply the group to multiple datastores in a more structured way. Autoscaling of volumes has been improved where the actual scaling is less intrusive than it was in 1.50. Added a SUPERUSER checkbox when creating a new PostgreSQL user, with caution prompts.

Changes

• Parameter Groups for Database Management.
• Database Logs in Events Viewer.
• Create Datastore from incremental backup from different storage types.
• Reboot database node.
• Make Postgres SUPERUSER configurable.

Important fixes

• Deployments stuck in deploying status forever
• Corrected state transitions so a failed deployment eventually marks as “failed” instead of hanging.
• Reduced unnecessary Service updates, lowering API calls to Kubernetes.
• Set correct interval to 15/30/60 minutes for incremental backups.
• Disabled volume-editing for ephemeral storage as it was never intended.

V 1.50

Overview

This release offers upward volume autoscaling, new customer database parameters, improved monitoring in terms of mail notifications and more metrics. It is now possible to create (m)TLS-based sessions where the client can prefetch server certificates. The backup management has been improved disalloving concurrent backup race conditions.

Changes

• Auto-scale volumes, enabled by default.
• Send email notifications to end user.
• (m)TLS for Mysql/MariaDB, Postgres and Redis.
• Do all DNS queries through ExternalDNS.
• The Terraform provider has been substantially improved.

Important fixes

• Fixed a problem where multiple concurrent backups were executed.
• There was a problem in removing datastores stuck in creation or modifying state.
• Redis and MSSQL backup restore was not working properly.
• Optimized failover time for Always on.

V 1.48

Overview

In this release we introduce MSSQL, the new Openstack V2 instance flavors and volume types giving even better performance and price efficiency.

Changes

• MSSQL in Standalone and Always On versions.
• Lifecycle management, database upgrades.
• Improved automatic datastore failover handling.
• Change existing datastore volume type and size.
• Account password management.
• Choose new V2 node flavors with improved performance.
• Mobile UI.
• Datastore UI overview page paging and filtering.
• Terraform provider upgrade allowing automated datastore and firewall management.
• Improved documentation with more practical examples for, among other, external backup/restore and Terraform provider usage, and much more.

Important fixes

• Fixed a problem where DNS name for datatore nodes occasionally disappeared.
• Fixed Postgres creation and restore who occasionally could fail.
• Corrected a problem where datastore creation from backup failed.

V 1.47

Overview

The release focus on datastore failure handling, e.g. when nodes are lost and how the failover is managed. It introduces improved general database life cycle management and initial backend support for MSSQL Server 2022.

Changes

• Automatatic datastore failure handling.
• Datastore creation from backups.
• Improved datastore upgrade process.
• Expose monitoring ports for customer prometheus clients.
• Repair and node scaling for MSSQL.
• UI view filtering and list presentation.
• Improved UI guidance tool tips.
• Terraform API for grouped firewall rules.

Important fixes

• Corrected a problem where promotion of new MSSQL primary led to endless loops.
• Fixed problem where DNS records for datastore nodes could sometime disappear after upgrades.
• Corrected a problem with inconsistent logging and presentation of changed cluster and node status.

V 1.46

Overview

This release introduces configuration management and simplified service access. Initial support for life cycle management is introduced.

Changes

• Access to services/failover. This provides the user with a single entrypoint to the datastore.
• Configuration Management. Ability to let the end-user tune certain configuration values.
• Lifecycle Management. Ability to upgrade datastores (OS and database software) using a roll-forward upgrade method.
• Improved customer error interaction for handling nodes.

Important fixes

• Corrected a bug that caused the control plane process to restart occasionally.

4.8 - DBaaS Guides

4.8.1 - Backup and Restore via CLI

Overview

This guide will help you getting started with creating and restoring your own backups using various database CLI tools.
For the built-in backup functionality, please see here.

PostgreSQL

Backup

The client we are using in this guide is pg_dump which is included in the PostgreSQL client package. It is recommended to use the same client version as your server version.

The basic syntax and an example to dump a PostgreSQL database with the official tool pg_dump is shown below. To connect and authenticate with a remote host you can either specify this information either with options, environment variables or a password file.

Usage & example

pg_dump [OPTION]... [DBNAME]
  -h, --host=HOSTNAME      database server host or socket directory (default: "local socket")
  -p, --port=PORT          database server port (default: "5432")
  -U, --username=USERNAME  database user name (default: "$USER")
  -f, --file=FILENAME      output file or directory name
  -d, --dbname=DBNAME      database to dump

pg_dump -h mydatabaseserver -U mydatabaseuser -f dump.sql -d mydatabase

Environment variables

As mentioned, we can also specify the options, connection and authentication information via environment variables, by default the client checks if the below environment variables are set.

For a full list, check out the documentation under PostgreSQL Documentation.

PGDATABASE
PGHOST
PGOPTIONS
PGPORT
PGUSER

It is not recommended to specify the password via the above methods, and thus not listed here. For the password it is better to use a so called password file. By default the client checks the user’s home directory for a file named .pgpass. Read more about the password file by going to the official documentation linked under PostgreSQL Documentation.

Restore

To restore a database we will use the client psql which is also included in the PostgreSQL client package. It is recommended to use the same client version as your server version.

Usage & example

psql [OPTION]... [DBNAME [USERNAME]]
  -h, --host=HOSTNAME      database server host or socket directory (default: "local socket")
  -p, --port=PORT          database server port (default: "5432")
  -U, --username=USERNAME  database user name (default: "$USER")
  -f, --file=FILENAME      execute commands from file, then exit
  -d, --dbname=DBNAME      database name to connect to

psql -h mydatabaseserver -U mydatabaseuser -f dump.sql -d mydatabase

PostgreSQL Documentation

• PostgreSQL 11/14 - pgdump
• PostgreSQL 11/14 - The Password file
• PostgreSQL 11/14 - Environment variables
• PostgreSQL 11/14 - SQL Dump

MariaDB

Backup

The client we are using in this guide is mariadb-dump which is included in the MariaDB client package.

The basic syntax and an example to dump a MariaDB database with the official tool mariadb-dump is shown below together with some of the options we will use.

Usage & example

mariadb-dump [OPTIONS] database [tables]
OR     mariadb-dump [OPTIONS] --databases DB1 [DB2 DB3...]
-h, --host=name       Connect to host.
-B, --databases       Dump several databases...
-q, --quick           Don't buffer query, dump directly to stdout.
--single-transaction  Creates a consistent snapshot by dumping all tables
                      in a single transaction...
--skip-lock-tables    Disable the default setting to lock tables

For a full list of options, check out the documentation under MariaDB Documentation.

Depending on your specific needs and the scope of the backup you might need to use the pre-created database user. This is because any subsequent users created in the portal are set up with permissions to a specific database while the pre-existing admin user have more global permissions that are needed for some of the dump options.

mariadb-dump -h mydatabaseserver -B mydatabase --quick --single-transaction --skip-lock-tables > dump.sql

It is not recommended to specify the password via the command line. Consider using an option file instead, by default the client checks the user’s home directory for a file named .my.cnf. You can read more about option files in the official documentation linked under MariaDB Documentation.

Restore

To restore the database from the dump file we will use the tool mariadb that is also included in the MariaDB client package.

Usage & example

mariadb [OPTIONS] [database]
-h, --host=name     Connect to host

mariadb -h mydatabaseserver mydatabase < dump.sql

MariaDB Documentation

• MariaDB mariadb-dump tool
• MariaDB Option Files
• MariaDB Backup and Restore Overview

MySQL

Backup

The client we are using in this guide is mysqldump which is included in the MySQL client package.

The basic syntax and an example to dump a MySQL database with the official tool mysqldump is shown below together with some of the options we will use.

Usage & example

mysqldump [OPTIONS] database [tables]
OR     mysqldump [OPTIONS] --databases DB1 [DB2 DB3...]
-h, --host=name       Connect to host.
-B, --databases       Dump several databases...
-q, --quick           Don't buffer query, dump directly to stdout.
--single-transaction  Creates a consistent snapshot by dumping all tables
                      in a single transaction...
--skip-lock-tables    Disable the default setting to lock tables
--no-tablespaces      Do not write any CREATE LOGFILE GROUP or 
                      CREATE TABLESPACE statements in output 

For a full list of options, check out the documentation under MySQL Documentation.

Depending on your specific needs and the scope of the backup you might need to use the pre-created database user. This is because any subsequent users created in the portal are set up with permissions to a specific database while the pre-existing admin user have more global permissions that are needed for some of the dump options.

mysqldump -h mydatabaseserver -B mydatabase --quick --single-transaction --skip-lock-tables --no-tablespaces > dump.sql`

It is not recommended to specify the password via the command line. Consider using an option file instead, by default the client checks the user’s home directory for a file named .my.cnf. You can read more about option files in the official documentation linked under MySQL Documentation.

Restore

To restore the database from the dump file we will use the tool mysql that is also included in the MySQL client package.

Usage & example

mysql [OPTIONS] [database]
-h, --host=name     Connect to host

mysql -h mydatabaseserver mydatabase < dump.sql

MySQL Documentation

• MySQL mysqldump tool
• MySQL Option Files
• MySQL Backup and Recovery

4.8.2 - Backup and Restore via DBaaS UI

Overview

All our supported database types comes with built-in backup functionality and is enabled by default. Backups are stored in our object storage, which is encrypted at rest and also utilizes all of our availability zones for highest availability. You can easily set the amount of backups per day, the prefered time of day and the retention period in our DBaaS UI. For MySQL, MariaDB and PostgreSQL we also support creating new datastores from backup, making it easy to create a new database cluster using another cluster as a base.
For backup pricing, you can use our DBaaS price calculator found here: ECP-DBaaS

Good to know

Beaware: Please note that if you delete a datastore, all backups for that datastore will also be deleted. This action cannot be reverted.

• Backups are taken for the whole datastore.
• Maximum backup retention period is 90 days. Default value is 7 days.
• There’s no storage quota for backups.
• Incremental backups are supported and enabled by default on MySQL and MariaDB.
• Backups cannot be downloaded locally. To create an offsite backup, you can use one of the CLI-tools. See here for some examples.
• Creating new datastores from previously taken backups is supported for MySQL, MariaDB and PostgreSQL.

Manage backups

Begin by logging into your Elastx DBaaS account, choose your datastore and go to Backups.
Under this tab you will see all the previously taken backups for the chosen datastore, if you just created this datastore, it might be empty.

Retention Period

To change retention period click on Backup settings at the top right corner, set your prefered retention period and click Save.

Backup schedules

For datastores running MySQL and MariaDB you have the ability to set schedules for both full and incremental backups.
To change how often and when your backups should run, click on Backup Schedules in the left corner.
Select the backup type you want to change and choose edit:

• Incremental backups can be set to run every 15, 30 and 60 minutes.
• Full backups can be set to run hourly or daily. Set your prefered time in UTC.

Restore backup on your running datastore

Beaware: Please note that this process will completely overwrite your current data and all changes since your last backup will be lost.

Go to the Backups tab for the datastore you want to restore. Select the prefered backup and click on the three dots under Actions and choose restore.

Create a new datastore from backup

For MySQL, MariaDB and PostgreSQL you have the ability to use a backup as a base for a new datastore.
Go to backups and click on the three dots under actions for the backup you want to use as a base and select Create Datastore.
A new datastore will be created with the same specification and name (with extension _Copy) as the base datastore.
When it’s finished, you can rename your new datastore by going to Settings > Datastore name.

Disable backups

Beaware: Not recommended. Please note that if you disable full backups, no backups will be taken after this point until you manually enable it again.

Go to the backups tab for the datastore you want to pause backups. Select Backup Schedules, click on the three dots for type of backup you want to disable and choose pause. To re-enable backups again, take the same steps and choose enable.

4.8.3 - Config Management

note: Deprecated in v1.51 in favor of parameter groups Please see to Parameter Groups

In CCX, you have the ability to fine-tune your database performance by adjusting various DB Parameters. These parameters control the behavior of the database server and can impact performance, resource usage, and compatibility.

Available DB Parameters

This is an example, and is subject to change and depends on the configuration of CCX.

1. group_concat_max_len

   • Description: Specifies the maximum allowed result length of the GROUP_CONCAT() function.
   • Max: 104857600 | Min: 1024 | Default: 1024

2. interactive_timeout

   • Description: Sets the number of seconds the server waits for activity on an interactive connection before closing it.
   • Max: 28800 | Min: 3000 | Default: 28800

3. max_allowed_packet

   • Description: Specifies the maximum size of a packet or a generated/intermediate string.
   • Max: 1073741824 | Min: 536870912 | Default: 536870912

4. sql_mode

   • Description: Defines the SQL mode for MySQL, which affects behaviors such as handling of invalid dates and zero values.
   • Default: ONLY_FULL_GROUP_BY, STRICT_TRANS_TABLES, NO_ZERO_IN_DATE, NO_ZERO_DATE, ERROR_FOR_DIVISION_BY_ZERO, NO_ENGINE_SUBSTITUTION

5. table_open_cache

   • Description: Sets the number of open tables for all threads.
   • Max: 10000 | Min: 4000 | Default: 4000

6. wait_timeout

   • Description: Defines the number of seconds the server waits for activity on a non-interactive connection before closing it.
   • Max: 28800 | Min: 3000 | Default: 28800

How to Change Parameters

1. Navigate to the DB Parameters tab within the Settings section.
2. Review the list of available parameters and their current values.
3. Click on the Edit Parameters button in the upper-right corner.
4. Adjust the values as necessary within the defined minimum and maximum limits.
5. Once you’ve made the required changes, save the new configuration.

note: The latest saved settings are applied when adding a node (either as part of Scaling, during Lifecycle management, or during automatic repair).

Best Practices

• Understand the impact: Changing certain parameters can significantly impact the performance and stability of your database. Make sure to test changes in a staging environment if possible.
• Stay within limits: Ensure that your values respect the maximum and minimum bounds defined for each parameter.
• Monitor after changes: After adjusting any parameter, monitor your database performance to ensure the changes have the desired effect.

By properly configuring these parameters, you can optimize your database for your specific workload and operational requirements.

4.8.4 - Create Datastore From Backup

In CCX, it is possible to create a new datastore from a backup Supported databases: MySQL, MariaDb, Postgres.

Select the backup you wish to restore in the Backup tab and select “Create datastore” from the action menu next to the backup. This process may take some time depending on the size of the backup. The new datastore will have the name datastore as the parent but will be suffixed with _Copy.

This allows you to:

• create a datastore from a backup for development and testing purposes.
• Investigate and analyse data without interfering with the production environment.

Limitations

PITR is not supported yet.

4.8.5 - Database Db Management

This guide explains how to create, list, and manage databases within the CCX platform for both PostgreSQL and MySQL systems. Databases is not a concept in Redis, and in Microsoft SQLServer creating databases is not supported.

Listing Existing Databases

Once databases are created, you can view the list of databases in the Databases tab.

• The Database Name column shows the names of the databases.
• The Size column displays the size of the database.
• The Tables column indicates the number of tables within each database.

• For MySQL, the database list will appear similar, with columns for database name, size, and tables.

Creating a New Database

To create a new database in the CCX platform:

note:

• PostgreSQL Database Owner: When creating a database in PostgreSQL, ensure that a valid user is selected as the owner of the database.:**
• MySQL Database Management: MySQL database creation does not require specifying an owner, but all other functions (listing, deleting) remain similar.:**

1. Navigate to the Databases Tab:

   • Click on the Databases section from the main dashboard.

2. Click on Create New Database:

   • A form will appear asking for the following details:
     • Database Name: The name of the new database.
     • DB Owner: The user who will own the database (applicable to PostgreSQL).

   

3. Submit the Form:

   • After filling in the necessary information, click Create to create the new database.

4. MySQL Database Creation:

   • For MySQL, the owner field is not required. You only need to specify the database name.

   

Dropping a Database

note:

• MySQL/MariaDb Database locks / metadata locks: The DROP DATABASE will hang if there is a metadata lock on the database or a table/resource in the database. Use SHOW PROCESSLISTin the mysql client to identify the lock. Either release the lock, KILL the connection, or wait for the lock to be released.:**

To delete or drop a database:

1. Locate the Database:

   • In the Databases tab, find the database you want to delete.

2. Click the Delete Icon:

   • Click on the red delete icon next to the database entry.
   • A confirmation dialog will appear asking if you are sure about dropping the database.

   

3. Confirm Deletion:

   • Click OK to proceed. WARNING: All data in the database will be lost.

Troubleshooting

Drop database hangs, the icon is spinning in the frontend.

Check if there are locks preventing the database from being deleted.

• In MySQL, the DROP DATABASE will hang if there is a metadata lock on the database or a table/resource in the database. Use SHOW PROCESSLIST in the mysql/mariadb client to identify the lock. Either release the lock, KILL the connection, or wait for the lock to be released.

4.8.6 - Database User Management

CCX allows you to create admin users. These users can in turn be used to create database uses with lesser privileges. Privileges and implementation is specific to the type of database. Admin users can be created for the following databases:

• PostgreSQL
• MySQL
• MariaDb
• Valkey
• Cache 22
• Microsoft SQL Server

List database users

To list database users do Navigate to the Users Tab::

Creating an Admin User

To create a new admin user, follow these steps:

1. Navigate to Users Tab:

   • Go to the Users section from the main dashboard.

2. Click on Create Admin User:

   Below is the MySQL interface described, but the interface is similar for the other database types

   • A form will appear prompting you to enter the following details:
   • Username: Specify the username for the new admin.
   • Password: Enter a strong password for the admin user.
   • Database Name: Select or specify the database this user will be associated with.
   • Authentication Plugin: Choose the authentication method for the user. Available options:
     • caching_sha2_password (default)
     • mysql_native_password (for MySQL compatibility)

   

Deleting a database user

Delete User: To delete a user, click on the red delete icon beside the user entry. A confirmation dialog will appear before the user is removed.

Connection assistant

CCX provides a Connection Assistant to help configure connection strings for your database clients.

1. Configure Database User and Database Name:

   • Select the database user and the database name.
   • Choose the Endpoint type (Primary or Replica).

2. Connection String Generation:

   • Based on the selected options, a connection string is generated for various technologies, including:
     • JDBC
     • ODBC
     • Python (psycopg2)
     • Node.js (pg)

3. Example:

   String url = "jdbc:postgresql://<host>:<port>/<dbname>?verifyServerCertificate=true&useSSL=true&requireSSL=true";
   myDbConn = DriverManager.getConnection(url, "<username>", "<password>");
   

4.8.7 - Datastore Settings

In the Settings section of CCX, there are two primary configuration options: General and DB Parameters.

The General settings section allows you to configure high-level settings for your datastore. This may include basic configurations such as system name, storage options, and general system behavior.

The DB Parameters section is used for fine-tuning your database. Here, you can adjust specific database settings such as memory allocation, query behavior, or performance-related parameters. These settings allow for a deeper level of control and optimization of the datastore for your specific workload.

Database Parameters

Please see Configuration management.

Changing the Datastore Name in CCX

The Datastore Name in CCX is an identifier for your datastore instance, and it is important for proper organization and management of multiple datastores. The name can be set when creating a datastore or changed later to better reflect its purpose or environment.

Notifications in CCX

Introduced in v.1.50.

The Notifications feature in CCX allows you to configure email alerts for important system events. These notifications help ensure that you are aware of critical events happening within your environment, such as when the disk space usage exceeds a certain threshold or when important jobs are started on the datastore.

To configure recipients of notification emails, simply enter the email addresses in the provided field. Multiple recipients can be added by separating each email with a semicolon (;).

If no email addresses are added, notifications will be disabled.

Key Notifications:

• Disk Space Alerts: When disk usage exceeds 85%, a notification is sent to the configured recipients.
• Job Alerts: Notifications are sent when significant jobs (such as data processing or backups) are initiated on the datastore.

This feature ensures that system administrators and key stakeholders are always up-to-date with the health and operations of the system, reducing the risk of unexpected issues.

Auto Scaling Storage Size in CCX

Introduced in v.1.50.

CCX provides a convenient Auto Scaling Storage Size feature that ensures your system never runs out of storage capacity unexpectedly. By enabling this feature, users can automatically scale storage based on usage, optimizing space management.

When Auto Scale is turned ON, the system will automatically increase the storage size by 20% when the used space exceeds 85% of the allocated storage. This proactive scaling ensures that your system maintains sufficient space for operations, preventing service interruptions due to storage constraints.

Key Benefits:

• Automatic scaling by 20% when usage exceeds 85%.
• Ensures consistent performance and reliability.
• Eliminates the need for manual storage interventions.

This feature is especially useful for dynamic environments where storage usage can rapidly change, allowing for seamless growth as your data expands.

Authentication in CCX

Introduced in v.1.49.

The Authentication section in CCX allows users to download credentials and CA certificates, which are essential for securing communication between the system and external services or applications.

Credentials

The Credentials download provides the necessary authentication details, such as API keys, tokens, or certificates, that are used to authenticate your system when connecting to external services or accessing certain system resources. These credentials should be securely stored and used only by authorized personnel.

To download the credentials, simply click the Download button.

CA Certificate

The CA Certificate ensures secure communication by verifying the identity of external systems or services through a trusted Certificate Authority (CA). This certificate is critical when establishing secure connections like HTTPS or mutual TLS (mTLS).

To download the CA Certificate, click the Download button next to the CA Certificate section.

Security Considerations:

• Keep credentials secure: After downloading, ensure the credentials and certificates are stored in a secure location and only accessible by authorized personnel.
• Use encryption: Where possible, encrypt your credentials and certificates both at rest and in transit.
• Regularly rotate credentials: To maintain security, periodically rotate your credentials and update any related system configurations.

This Authentication section is vital for maintaining a secure and trustworthy communication environment in your CCX setup.

4.8.8 - DBaaS with Terraform

Overview

This guide will help you getting started with managing datastores in Elastx DBaaS using Terraform.
For this we will be using OAuth2 for authentication and the CCX Terraform provider. You can find more information about the latest CCX provider here.

Good To Know

• Create/Destroy datastores supported.
• Setting firewall rules supported.
• Setting database parameter values supported.
• Scale out/in nodes supported.
• Create users and databases currently not supported.

DBaaS OAuth2 credentials

Before we get started with terraform, we need to create a new set of OAuth2 credentials.
In the DBaaS UI, go to your Account settings, select Authorization and choose Create credentials.

In the Create Credentials window, you can add a description and set an expiration date for your new OAuth2 credential.
Expiration date is based on the number of hours starting from when the credential were created. If left empty, the credential will not have an expiration date. You can however revoke and-/or remove your credentials at any time.
When you’re done select Create.

Copy Client ID and Client Secret. We will be using them to authenticate to DBaaS with Terraform.
Make sure you’ve copied and saved the client secret before closing the popup window. The client secret cannot be obtained later and you will have to create a new one.

Terraform configuration

We’ll start by creating a new, empty file, and adding the Client ID and Secret as variables, which will be exported and used for authenticaton later when we apply our terraform configuration.
Add your Client ID and Client Secret.

#!/usr/bin/env bash

export CCX_BASE_URL="https://dbaas.elastx.cloud"
export CCX_CLIENT_ID="<client-id>"
export CCX_CLIENT_SECRET="<client-secret>"

Source your newly created credentials file.

source /path/to/myfile.sh

Terraform provider

Create a new terraform configuration file. In this example we create provider.tf and add the CCX provider.

terraform {
  required_providers {
    ccx = {
      source = "severalnines/ccx"
      version = "0.3.1"
    }
  }
}

Create your first datastore with Terraform

Create an additional terraform configuration file and add your prefered datastore settings. In this example we create a configuration file named main.tf and specify that his is a single node datastore with MariaDB.

resource "ccx_datastore" "elastx-dbaas" {
  name           = "my-terraform-datastore"
  db_vendor      = "mariadb"
  size           = "1"
  instance_size  = "v2-c2-m8-d80"
  volume_type    = "v2-1k"
  volume_size    = "80"
  cloud_provider = "elastx"
  cloud_region   = "se-sto"
  tags           = ["terraform", "elastx", "mariadb"]
}

Create primary/replica datastores with added firewall rules and database parameter values

This example is built upon the previous MariaDB example. Here we added a second node to create a primary/replica datastore. We’re also adding firewall rules and setting database parameter values. To see all available database parameters for your specific database type, log into the DBaaS UI, go to your specific datastore > Settings > DB Parameters.

resource "ccx_datastore" "elastx-dbaas" {
  name           = "my-terraform-datastore"
  db_vendor      = "mariadb"
  size           = "2"
  instance_size  = "v2-c2-m8-d80"
  volume_type    = "v2-1k"
  volume_size    = "80"
  cloud_provider = "elastx"
  cloud_region   = "se-sto"
  tags           = ["terraform", "elastx", "mariadb"]

# You can add multiple firewall rules here
  firewall {
    source       = "x.x.x.x/32"
    description  = "My Application"
  }

  firewall {
    source      = "x.x.x.x/32"
    description = "My database client"
  }

# Set your specific database parameter values here. Values should be comma-separated without spaces.
  db_params = {
    sql_mode = "STRICT_TRANS_TABLES,NO_AUTO_CREATE_USER"
  }
 
}

Available options

Below you will find a table with available options you can choose from.

4.8.9 - Deploy A Datastore

MySQL or MariaDB

MySQL 8.4 is recommended if you are migrating from an existing MySQL system. MariaDB 11.4 is recommended if you are migrating from an existing MariaDB system. MySQL 8.4 offers a more sophisticated privilege system which makes database administration easier, wheres

High-availability

MySQL and MariaDB offers two configurations for High-availability.

• Primary/replica (asynchronous replication)
• Multi-primary (Galera replication)

Primary/replica is recommended for general purpose.

Scaling

MySQL and MariaDb can be created with one node (no high-availability) and can later be scaled with read-replicas or Primarys (in case of Multi-primary configuration).

PostgreSQL

PostgreSQL 15 and later supports the following extensions by default:

• PostGis
• PgVector

High-availability

High-availability is facilitated with PostgreSQL streaming replication

Scaling

PostgreSQL can be created with one node (no high-availability) but can later be scaled with read-replicas.

Cache22 (aka Redis)

deprecated

Cache22 is an in-memory data structure store.

High-availability

High-availability is facilitated with Redis replication and Redis Sentinels.

Scaling

Redis can be created with one node (no high-availability) but can later be scaled with read-replicas.

Valkey

Valkey is an in-memory data structure store.

High-availability

High-availability is facilitated with Valkey replication and Valkey Sentinels.

Scaling

Valkey can be created with one node (no high-availability) but can later be scaled with read-replicas.

MSSQL Server

Microsoft SQLServer 2022. Special license restrictions apply and this option may not be available in all CCX implementations.

4.8.10 - Event Viewer

The Event Viewer provides a detailed history of actions performed on the datastore. It tracks when changes were made, their status, who initiated the action, and a brief description of the action itself.

• When: Timestamp indicating when the event occurred.
• Status: The current status of the event (e.g., Finished for successfully completed tasks).
• Initiated by: The user or process that initiated the action.
• Description: A summary of the action performed.

Example Events:

The Event Viewer is essential for tracking the progress of tasks such as node scaling, promotions, and configuration updates. Each event is clearly labeled, providing users with transparency and insight into the state of their datastore operations.

4.8.11 - Firewall

This guide explains how to manage trusted sources and open ports within the firewall settings of the CCX platform. Only trusted sources are allowed to connect to the datastore.

A number of ports are open for each trusted source. One port is opened for the database service, but other ports are open for metrics. This makes it possible to connect and scrape the database nodes for metrics from a trusted source. The metrics are served using Prometheus exporters.

List Trusted Sources

Trusted sources can be managed from the Firewall tab. Only trusted sources are allowed to connect to the datastore. Here you can see:

• Source: View the allowed IP addresses or ranges.
• Description: Review the description of the source for identification.
• Actions: Delete the source by clicking on the red trash icon.

Adding a Trusted Source

To allow connections from a specific IP address or range, you need to create a trusted source.

Click on Create Trusted Source:

• A form will appear prompting you to enter the following details:
  • Source IP: Specify the IP address or CIDR range to allow. It is possible to specify a semicolon-separated list of CIDRs. If no CIDR is specified, then /32 is automatically added to the IP address.
  • Description: Add a description to identify the source (e.g., “My office”, “Data Center”).

After filling out the details, click Create to add the trusted source.

Viewing and Managing Trusted Sources

Managing Open Ports for Each Trusted Source.

TLS access to exporter metrics are described in this section.

Each trusted source can have specific ports opened for services. To manage the ports:

1. Expand a Trusted Source:

   • Click the down arrow beside the source IP to view the open ports.

2. Port Configuration:

   • Port Number: The number of the open port (e.g., 9100, 5432).
   • Port Name: The name of the service associated with the port (e.g., node_exporter, postgres_exporter, service).

   The service indicates the listening port of the database server. The ports for the node_exporter and db_exporter allows you to tap in to observability metrics for the database nodes.

3. Actions:

   • Delete a Port: Remove a port by clicking the red trash icon next to the port number.

Example Ports:

• Port 9100: node_exporter

• Port 9187: postgres_exporter

• Port 5432: service

  

Deleting Trusted Sources and Ports

Deleting a Trusted Source:

To remove a trusted source entirely, click on the red trash icon next to the source IP. This will remove the source and all associated ports.

Deleting an Individual Port:

To delete a specific port for a trusted source, click on the red trash icon next to the port number. This action will only remove the specific port.

This documentation covers the basic operations for managing firewall trusted sources and ports within the CCX platform. For further details, refer to the CCX platform’s official user manual or support.

4.8.12 - Logs Viewer

The Logs Viewer provides a comprehensive view of Database logs. The generated logs can be accessed for troubleshooting. It provides real-time access to essential logs, such as error logs, slow query logs though UI.

• Name: The file path or identifier of the log file.
• When: The timestamp indicating the most recent update or entry in the log file.
• Actions: Options to view, download the log file for further analysis.

Example Logs

The Logs Viewer is a critical tool for system administrators, enabling real-time monitoring and investigation of log files. With clear timestamps and actionable options, it ensures efficient identification and resolution of issues to maintain the stability of datastore operations.

4.8.13 - Observability

Overview

DBaaS offers metrics monitoring via the UI and remote.

Via UI there are various metrics for both databases and the nodes are presented under the datastore Monitor tab.

Remotely it is possible to monitor by using Prometheus and different exporters. The monitoring data is exposed though the exports from each node in the datastore. This is controlled under the Firewall tab in the DBaaS UI.

The ports available for the specific datastore configuration can be seen in UI under Firewall tab and the specific IP-address entry (fold the arrow to the left of the IP-address).

Exporter ports

Each exporter has its own port used by prometheus to scrape metrics.

Sample visible metrics

The following tables are excerpts of metrics for the different exporters to quickly get started.

System - Hardware level metrics

MySQL / MariaDB

MySQL metrics reference

• Handler Stats

  Statistic	Description
  Read Rnd	Count of requests to read a row based on a fixed position
  Read Rnd Next	Count of requests to read a subsequent row in a data file
  Read Next	Count of requests to read the next row in key order
  Read Last	Count of requests to read the last key in an index
  Read Prev	Count of requests to read the previous row in key order
  Read First	Count of requests to read a row based on an index key value
  Read Key	Count of requests to read the last key in an index
  Update	Count of requests to update a row
  Write	Count of requests to insert to a table

• Database Connections

  Metric	Description
  Thread Connected	Count of clients connected to the database
  Max Connections	Count of max connections allowed to the database
  Max Used Connections	Maximum number of connections in use
  Aborted Clients	Number of connections aborted due to client not closing
  Aborted Connects	Number of failed connection attempts
  Connections	Number of connection attempts

• Queries
  • Count of queries executed
• Scan Operations
  • Count of operations for the operations: SELECT, UPDATE and DELETE
• Table Locking

  Metric	Description
  Table locks immediate	Count of table locks that could be granted immediately
  Table locks waited	Count of locks that had to be waited due to existing locks or another reason

• Temporary Tables

  Metric	Description
  Temporary tables	Count of temporary tables created
  Temporary tables on Disk	Count of temporary tables created on disk rather than in memory

• Aborted Connections

  Metric	Description
  Aborted Clients	Number of connections aborted due to client not closing
  Aborted Connects	Number of failed connection attempts
  Access Denied Errors	Count of unsuccessful authentication attempts

• Memory Utilisation

  Metric	Description
  SELECT (fetched)	Count of rows fetched by queries to the database
  SELECT (returned)	Count of rows returned by queries to the database
  INSERT	Count of rows inserted to the database
  UPDATE	Count of rows updated in the database
  DELETE	Count of rows deleted in the database
  Active Sessions	Count of currently running queries
  Idle Sessions	Count of connections to the database that are not currently in use
  Idle Sessions in transaction	Count of connections that have begun a transaction but not yet completed while not actively doing work
  Idle Sessions in transaction (aborted)	Count of connections that have begun a transaction but did not complete and were forcefully aborted before they could complete
  Lock tables	Active locks on the database
  Checkpoints requested and timed	Count of checkpoints requested and scheduled
  Checkpoint sync time	Time synchronising checkpoint files to disk
  Checkpoint write time	Time to write checkpoints to disk

Redis

Redis metrics reference

4.8.14 - Parameter Group

Introduced in v.1.51

Parameter Groups is a powerful new feature introduced in version 1.51 of CCX. It enables users to manage and fine-tune database parameters within a group, simplifying configuration and ensuring consistency across datastores.

Overview

With Parameter Groups, users can:

• Create new parameter groups with customized settings.
• Assign parameter groups to specific datastores.
• Edit and update parameters within a group.
• Delete unused parameter groups.
• Automatically synchronize parameter changes with associated datastores.

note:

A datastore can only be associated with one parameter group at a time. Changes to parameters are automatically propagated to all associated datastores.

Features

1. Creating a Parameter Group

Users can create a new parameter group to define custom configurations for their databases.

Steps to Create a New Parameter Group:

1. Navigate to the DB Parameters section.
2. Click on the + Create new group button.
3. Fill in the required details:
   • Group Name: A unique name for the parameter group.
   • Description: A brief description of the group.
   • Vendor: Select the database type (e.g., MySQL, PostgreSQL, Redis).
   • Version: Specify the database version.
   • Configuration: Choose the type of configuration (e.g., Primary/Replica).
4. Customize the parameter values as needed.
5. Click Create to save the new group.

2. Assigning a Parameter Group to a Datastore

Once created, parameter groups can be assigned to datastores to apply the defined settings. The parameter can be assigned to an existing datastore or when a datastore is created.

Steps to Assign a Parameter Group in the Deployment wizard:

1. Open the Create datastore wizard
2. In the Configuration step, press Advanced, and select the parameter group under DB Settings.

note Please note that atleast one parameter group must exist matching the vendor, version and configuration.

Steps to Assign a Parameter Group to an existing datastore:

1. Navigate to the datastore you want to configure.
2. Go to the DB Parameters tab.
3. Click Change group or Assign group.
4. Select the desired parameter group from the dropdown.
5. Click Save to apply the group to the datastore.

The system will display the synchronization status (e.g., Pending or Synced) after assigning the group.

3. Viewing and Managing Parameter Groups

Users can view all parameter groups in the DB Parameters section. For each group, the following details are displayed:

• Group Name
• Vendor and Version
• Datastores: Associated datastores.
• Descriptions

From this view, users can:

• Edit: Modify the group’s parameters.
• Duplicate: Create a copy of the group.
• Delete: Remove the group.

4. Editing a Parameter Group

Parameter groups can be updated to reflect new configurations. Any changes are automatically synchronized with associated datastores.

Steps to Edit a Parameter Group:

1. Navigate to the DB Parameters section.
2. Click on the three-dot menu next to the group you want to edit.
3. Select Edit.
4. Update the parameter values as needed.
5. Click Save.

5. Deleting a Parameter Group

Unused parameter groups can be deleted to maintain a clean configuration environment.

Steps to Delete a Parameter Group:

1. Navigate to the DB Parameters section.
2. Click on the three-dot menu next to the group you want to delete.
3. Select Delete.
4. Confirm the deletion.

note A parameter group cannot be deleted if it is assigned to a datastore.

6. Synchronization

Once a parameter group is assigned to a datastore, the parameters are automatically synchronized. The status of synchronization (e.g., Pending or Synced) is visible in the DB Parameters tab of the datastore, and also in the Event Viewer.

Best Practices

• Use Descriptive Names: Give parameter groups clear, descriptive names to make them easily identifiable.
• Regular Updates: Regularly review and update parameter groups to optimize database performance.
• Monitor Sync Status: Always verify that parameter changes are properly synced to the datastores.

Conclusion

Parameter Groups in CCX provide a centralized and efficient way to manage database configurations. By grouping parameters and syncing them to datastores, users can ensure consistency, reduce manual errors, and improve overall system performance.

4.8.15 - Promote A Replica

You may want to promote a replica to become the new primary. For instance, if you’ve scaled up with a larger instance, you might prefer to designate it as the primary. Alternatively, if you’re scaling down, you may want to switch to a smaller configuration for the primary node.

In the Nodes view, select the Promote Replica action from the action menu next to the replica you wish to promote:

In this example, the replica with an instance size of ‘medium’ will be promoted to the new primary.

A final confirmation screen will appear, detailing the steps that will be performed:

4.8.16 - Reboot A Node

Introduced in v.1.51

The reboot command is found under the action menu of a databse node, on the Nodes page.

Selecting “Reboot” triggers a restart of the chosen replica. Use this option when:

• the replica needs to be refreshed due to performance issues
• for maintenance purposes.
• For some parameters, any change to the parameter value in a parameter group only takes effect after a reboot.

danger:

• Ensure all tasks linked with the node are concluded before initiating a reboot to prevent data loss.
• Only authorized personnel should perform actions within the administration panel to maintain system integrity.

note:

• Please note that rebooting may cause temporary unavailability.
• In Valkey, the primary may failover to a secondary if the reboot takes more than 30 seconds.

4.8.17 - Restore Backup

The Backup and Restore feature provides users with the ability to create, view, and restore backups for their databases. This ensures data safety and allows recovery to previous states if necessary.

Backup List View

In the Backup tab, users can view all the backups that have been created. The table provides essential information about each backup, such as:

• Method: The tool or service used to perform the backup (e.g., mariabackup).
• Type: The type of backup (e.g., full backup).
• Status: The current state of the backup (e.g., Completed).
• Started: The start time of the backup process.
• Duration: How long the backup process took.
• Size: The total size of the backup file.
• Actions: Options to manage or restore backups.

Example Backup Table

Users can manage their backups using the “Actions” menu, where options such as restoring a backup are available.

Backup Schedules View

The Backup Schedules allows users to manage scheduled backups for their datastore. Users can configure automatic backup schedules to ensure data is periodically saved without manual intervention.

Backup Schedule Table

The schedule table shows the details of each scheduled backup, including:

• Method: The tool or service used to perform the backup (e.g., mariabackup).
• Type: The type of backup, such as incremental or full.
• Status: The current state of the scheduled backup (e.g., Active).
• Created: The date and time when the backup schedule was created.
• Recurrence: The schedule’s frequency, showing the cron expression used for the schedule (e.g., TZ=UTC 5 * * *).
• Action: Options to manage the schedule, such as Pause or Edit.

Example Backup Schedule Table:

Managing Backup Schedules

The Action menu next to each schedule allows users to:

• Pause: Temporarily stop the backup schedule.
• Edit: Adjust the backup schedule settings, such as its frequency or time.

Editing a Backup Schedule

When editing a backup schedule, users can specify:

• Frequency: Choose between Hourly or Daily backups.
• Time: Set the exact time when the backup will start (e.g., 05:00 UTC).

For example, in the Edit Full Backup Schedule dialog, you can configure a full backup to run every day at a specified time. Adjust the settings as needed and click Save to apply the changes.

Example Backup Schedule Edit Dialog:

This dialog allows you to easily adjust backup intervals, ensuring that backups align with your operational needs.

note:

Editing or pausing a schedule will not affect the current backups already created. The changes will only apply to future backups.

Restore Backup

To restore a backup, navigate to the Backup tab, find the desired backup, and select the Restore action from the Actions menu. This opens the restore dialog, where the following information is displayed:

• Backup ID: The unique identifier of the backup.
• Type: The type of backup (e.g., full backup).
• Size: The total size of the backup file.

Restore Settings

• Use Point in Time Recovery: Option to enable point-in-time recovery for finer control over the restore process. PITR is only supported by Postgres, MySQL/MariaDb, and MS SQLServer.

By default, this option is turned off, allowing a full restoration from the selected backup.

Confirmation

Before initiating the restore, users are presented with a confirmation dialog:

You are going to restore a backup
You are about to restore a backup created on 03/10/2024 05:00 UTC.
This process will completely overwrite your current data, and all changes since your last backup will be lost.

Users can then choose to either Cancel or proceed with the Restore.

Example Restore Dialog:

This ensures that users are fully aware of the potential data loss before proceeding with the restore operation.

4.8.18 - Scale A Datastore

This section explains how to scale a datastore, including:

• Scaling volumes
• Scaling nodes (out, in, up and down)

A datastore can be scaled out to meet growing demands. Scaling out involves adding:

• One or more replica nodes (for primary/replica configurations). This is useful when you need to scale up and want the primary node to have more resources, such as additional CPU cores and RAM.
• One or more primary nodes (for multi-primary configurations). In multi-primary setups, scaling up or down must maintain an odd number of nodes to preserve quorum and the consensus protocol required by the database.

The instance type of the new nodes may differ from the current ones.

To scale a datastore, navigate to the Nodes page and select Nodes Configuration.

Scaling Up or Down, In or Out

Use the slider to adjust the datastore’s new size. In this example, we have two nodes (one primary and one replica), and we want to scale up to four nodes. You can also specify the availability zones and instance sizes for the new nodes. Later, you might choose to promote one of the replicas to be the new primary. To proceed with scaling, click Save and wait for the scaling job to complete.

Scaling Down

You can also scale down by removing replicas or primary nodes (in a multi-primary configuration). In the Nodes Configuration view, select the nodes you wish to remove, then click Save to begin the scaling process. This allows you to reduce the size of the datastore or remove nodes with unwanted instance sizes.

Scaling Volumes

To scale storage, go to the Nodes tab and select Scale Storage. You can extend the storage size, but it cannot be reduced. All nodes in the datastore will have their storage scaled to the new size.

4.8.19 - Terraform Provider

The CCX Terraform provider allows to create datastores on all supported clouds. The CCX Terraform provider project is hosted on github.

Oauth2 credentials

Oauth2 credentials are used to authenticate the CCX Terraform provider with CCX. You can generate these credentials on the Account page Authorization tab. And then you will see:

Requirement

• Terraform 0.13.x or later

Quick Start

1. Create Oauth2 credentials.
2. Create a terraform.tf
3. Set client_id, client_secret, below is a terraform.tf file:

terraform {
  required_providers {
    ccx = {
      source  = "severalnines/ccx"
      version = "~> 0.4.7"
    }
  }
}

provider "ccx" {
    client_id = `client_id`
    client_secret = `client_secret`
}
```

Now, you can create a datastore using the following terraform code.
Here is an example of a parameter group:

```terraform
resource "ccx_parameter_group" "asteroid" {
    name = "asteroid"
    database_vendor = "mariadb"
    database_version = "10.11"
    database_type = "galera"

    parameters = {
      table_open_cache = 8000
      sql_mode = "STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"
    }
}
```

This group can then be associated with a datastore as follows:

```terraform
resource "ccx_datastore" "luna_mysql" {
	name           = "luna_mysql"
	size           = 3
	type           = "replication"
	db_vendor      = "mysql"
	tags           = ["new", "test"]
	cloud_provider = "CCX_CLOUD"
	cloud_region   = "CCX-REGION-1"
	instance_size  = "MEGALARGE"
	volume_size    = 80
	volume_type    = "MEGAFAST"
	parameter_group = ccx_parameter_group.asteroid.id
}
```

Replace CCX_CLOUD, CCX-REGION-1, MEGALARGE and, MEGAFAST, with actual values depending on the cloud infrastructure available.

For more information and examples, visit the [terraform-provider-ccx](https://github.com/severalnines/terraform-provider-ccx) github page.

## More on parameter groups
Only one parameter group can be used at any give time by a datastore.
Also, you cannot change an existing parameter group from terraform.
If you want to change an existing parameter group, then you need to create a new parameter group:
```terraform
resource "ccx_parameter_group" "asteroid2" {
    name = "asteroid2"
    database_vendor = "mariadb"
    database_version = "10.11"
    database_type = "galera"

    parameters = {
      table_open_cache = 7000
      sql_mode = "NO_ENGINE_SUBSTITUTION"
    }
}
```
And then reference it in:
```terraform
resource "ccx_datastore" "luna_mysql" {
	name           = "luna_mysql"
  ... <same as before>
	parameter_group = ccx_parameter_group.asteroid2.id
}
```
Now you can apply this to terraform. Always test config changes first on a test system to be sure the config change works as expected.

## Features
The following settings can be updated:

- Add and remove nodes
- Volume type
- Volume size
- Notifications
- Maintenance time
- Modify firewall (add/remove) entries. Multiple entries can be specified with a comma-separated list.

### Limitations

- Change the existing parameter group is not possible after initial creation, however you can create a new parameter group and reference that.
- It is not possible to change instance type.
- Changing availability zone is not possible.

4.8.20 - TLS For Metrics

Overview

To enhance security, using TLS for accessing metrics is recommended. This document outlines how the metrics served securely using TLS for each exporter. Each node typically has a Node Exporter and a corresponding database-specific exporter to provide detailed metrics. Access to these metrics is limited to the sources specified in Firewall Management.

Service discovery

There is a service discovery endpoint created for each datastore. Available from CCX v1.53 onwards.

It’s available at https://<ccxFQDN>/metrics/<storeID>/targets and implements Prometheus HTTP SD Endpoint.

note:

<ccxFQDN> is the domain you see in your address bar with CCX UI open, not a datastore URL or a connection string. We’ll use ccx.example.com hereafter.

Here is an example of a scrape config for Prometheus:

scrape_configs:
  - job_name: 'my datastore'
    http_sd_configs:
      - url: 'https://ccx.example.com/metrics/50e4db2a-85cd-4190-b312-e9e263045b5b/targets'

Individual Metrics Endpoints Format

Metrics for each exporter is served on:

https://ccx.example.com/metrics/<storeID>/<nodeName>/<exporterType>

Where nodeName is short name, not full fqdn.

Exporter Type Examples:

1. MSSQL:

   • URL: https://ccx.example.com/metrics/<storeID>/<nodeName>/mssql_exporter

2. Redis:

   • URL: https://ccx.example.com/metrics/<storeID>/<nodeName>/redis_exporter

3. PostgreSQL:

   • URL: https://ccx.example.com/metrics/<storeID>/<nodeName>/postgres_exporter

4. MySQL:

   • URL: https://ccx.example.com/metrics/<storeID>/<nodeName>/mysqld_exporter

5. MariaDB:

   • URL: https://ccx.example.com/metrics/<storeID>/<nodeName>/mysqld_exporter

6. NodeExporter:

   • URL: https://ccx.example.com/metrics/<storeID>/<nodeName>/node_exporter

By serving metrics over HTTPS with TLS, we ensure secure monitoring access for customers.

4.8.21 - Upgrade Lifecycle Mgmt

CCX will keep your system updated with the latest security patches for both the operating system and the database software.

You will be informed when there is a pending update and you have two options:

• Apply the update now
• Schedule a time for the update

The update will be performed using a roll-forward upgrade algorithm:

1. The oldest replica (or primary if no replica exist) will be selected first
2. A new node will be added with the same specification as the oldest node and join the datastore
3. The oldest node will be removed
4. 1-3 continues until all replicas (or primaries in case of a multi-primary setup) are updated.
5. If it is a primary-replica configuration then the primary will be updated last. A new node will be added, the new node will be promoted to become the new primary, and the old primary will be removed.

Upgrade now

This option will start the upgrade now.

Scheduled upgrade

The upgrade will start at a time (in UTC) and on a weekday which suits the application. Please note, that for primary-replica configurations, the update will cause the current primary to be changed.

Upgrade database major version

To upgrade the database major version from e.g MariaDB 10.6 to 10.11, you need to create a new datastore from backup, alternatively take mysqldump or pgdump and apply it to your new datastore.

4.8.22 - Connect Kubernetes with DBaaS

Overview

To connect your Kubernetes cluster with DBaaS, you need to allow the external IP addresses of your worker nodes, including reserved IP adresses, in DBaaS UIs firewall. You can find the reserved IPs in your clusters Openstack project or ask the support for help.

Get your worker nodes external IP with the CLI tool kubectl: kubectl get nodes -o wide

NAME                                            STATUS   ROLES           AGE    VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
company-stage1-control-plane-1701435699-7s27c   Ready    control-plane   153d   v1.28.4   10.128.0.40    <none>        Ubuntu 22.04.3 LTS   5.15.0-88-generic   containerd://1.7.6
company-stage1-control-plane-1701435699-9spjg   Ready    control-plane   153d   v1.28.4   10.128.1.160   <none>        Ubuntu 22.04.3 LTS   5.15.0-88-generic   containerd://1.7.6
company-stage1-control-plane-1701435699-wm8pd   Ready    control-plane   153d   v1.28.4   10.128.3.13    <none>        Ubuntu 22.04.3 LTS   5.15.0-88-generic   containerd://1.7.6
company-stage1-worker-sto1-1701436487-dwr5f     Ready    <none>          153d   v1.28.4   10.128.3.227   1.2.3.5       Ubuntu 22.04.3 LTS   5.15.0-88-generic   containerd://1.7.6
company-stage1-worker-sto2-1701436613-d2wgw     Ready    <none>          153d   v1.28.4   10.128.2.180   1.2.3.6       Ubuntu 22.04.3 LTS   5.15.0-88-generic   containerd://1.7.6
company-stage1-worker-sto3-1701437761-4d9bl     Ready    <none>          153d   v1.28.4   10.128.0.134   1.2.3.7       Ubuntu 22.04.3 LTS   5.15.0-88-generic   containerd://1.7.6

Copy the external IP for each worker node, in this case the three nodes with the ROLE <none>.

In the DBaaS UI, go to Datastores -> Firewall -> Create trusted source, Add the external IP with CIDR notation /32 for each IP address (E.g. 1.2.3.5/32).

5 - Elastx Cloud Platform API (beta)

5.1 - ⚡ Quick Start Guide

1. Introduction

The ECPAPI provides programmatic access to Elastx Cloud services, including resource management and automation capabilities. This guide will walk you through how to generate your API key, explore the API using Swagger UI, and make authenticated requests.

ECPAPI is currently in beta.

2. Authentication

Before accessing the API, you need to generate a Service Account Token from the Elastx Cloud Console (ECC).

This token will inherit the same permissions as your account within the selected organization. Use it in the Authorization header as follows:

Authorization: ServiceAccount <token>

Creating a Service Account Token

1. Log in to https://console.elastx.cloud/account.
2. Navigate to Service Accounts.
3. Select your organization.
4. Optionally set the token expiry (in seconds). Leave blank to use the default (1 year).
5. Click Save.
6. Copy and securely store the generated token. It will be used as a Bearer token in your API requests.

📌 Important: The token will only be shown once after creation. Store it securely.

Curl example using Service Account Token

curl -X 'GET' \
  'https://console.elastx.cloud/ecp-api/apis/v1/organizations' \
  -H 'accept: application/json' \
  -H 'Authorization: ServiceAccount <token>'

3. Explore the API (Swagger UI)

You can explore and test the API interactively using the Swagger UI.

• API Base URL: https://console.elastx.cloud/ecp-api/
• Swagger UI: https://console.elastx.cloud/ecp-api/openapi/

4. Example API Calls Using curl

List your organizations

Since the service account token is organization scoped, when using a Service Account Token, this will only return one organization.

curl -X 'GET' \
  'https://console.elastx.cloud/ecp-api/apis/v1/organizations' \
  -H 'accept: application/json' \
  -H 'Authorization: ServiceAccount <token>'

List Dbaas Projects in organization

curl -X 'GET' \
  'https://console.elastx.cloud/ecp-api/apis/v1/organizations/<org-id>/dbaasprojects' \
  -H 'accept: application/json' \
  -H 'Authorization: ServiceAccount <token>'

List users in organization

curl -X 'GET' \
  'https://console.elastx.cloud/ecp-api/apis/v1/organizations/<org-id>/users' \
  -H 'accept: application/json' \
  -H 'Authorization: ServiceAccount <token>'

Add user to organization

These users will not automatically get any access to neither ECC nor DBaaS projects, but you will need to add them to your organization in order to give them access to a DBaaS project. If the user does not have a prior account with a verified email you will need to trigger an invite to let the user verify their email address.

curl -X 'POST' \
  'https://console.elastx.cloud/ecp-api/apis/v1/organizations/<org-id>/users' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: ServiceAccount <token>'
  -d '{
  "email": "user@example.com",
  "firstName": "Firstname",
  "lastName": "LastName"
}'

response:

{
  "name": "useratexample-com",
  "email": "user@example.com",
  "firstName": "Firstname",
  "lastName": "LastName"
  "emailVerified": false
}

Send invite email

curl -X 'POST' \
  'https://console.elastx.cloud/ecp-api/apis/v1/organizations/<org-id>/users/<user-name>/invites' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: ServiceAccount <token>'

Create a new DbaasProject

curl -X 'POST' \
  'https://console.elastx.cloud/ecp-api/apis/v1/organizations/<org-id>/dbaasprojects' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: ServiceAccount <token>'
  -d '{
"projectName": "test project"
}'

response:

{
  "name": "test-project",
  "organization": "e0db3c07-b6aa-4ec6-8d21-f5c5fa670fbb",
  "projectName": "test project",
  "ccxProjectName": ""
}

Give user access to DbaasProject

Role can be member or admin both grant access to the DbaasProject in the DBaaS console, but only admin will give the user access to manage the DBaaS project in ECC and add additional users to the project.

curl -X 'POST' \
  'http://localhost:9000/apis/v1/organizations/<org-id>/dbaasprojects/<project-name>/members' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: ServiceAccount <token>'
  -d '{
  "role": "member",
  "userRef": "test-ecp-apiatelastx-se"
}'

Remove users access from DbaasProject

curl -X 'DELETE' \
  'http://localhost:9000/apis/v1/organizations/<org-id>/dbaasprojects/<project-name>/members/<user-ref>' \
  -H 'accept: application/json' \
  -H 'Authorization: ServiceAccount <token>'

5. Additional API: DBaaS (Database as a Service)

Elastx also provides a separate API for managing DBaaS environments.

• Base URL: https://dbaas.elastx.cloud/api
• Swagger UI: API Docs

6. Support

Need help with authentication or API usage?

📧 Contact us at support@elastx.se

6 - Elastx Identity Provider

6.1 - Overview

The purpose of Elastx Identity Provider (IDP) is to provide a Single Sign-on experience for access to the different Elastx service offerings. The service provides integration with OpenStack IaaS to enable usage/enforcement of multi-factor authentication.

Enrollment

Once enrolled, all or selected users will receive an email to verify their account and setup multi-factor authentication.

Please review known issues and limitations for further reference.

Update account settings

Users in organizations which have enrolled can use the “My Account” web page to change their password, configure multi-factor authentication and similar account related settings.

Supported MFA methods

In addition to username and password for authentication, the ability to configure multi-factor authentication using either TOTP, supported by mobile applications such as “Google Authenticator” and “FreeOTP”), or the Webauthn standard which enables usage of hardware tokens such as the Yubikey.

TOTP is default method for all accounts (except if explicitly defined otherwise for the organization), but can be changed by the user after initial setup.

OpenStack authentication flow

Once enrolled users can login to the OpenStack web dashboard (Horizon) by selecting “elastx-id” in the “Authenticate using” drop-down menu and pressing “Connect”:

The users are redirected redirected to the login portal, which validates their password/second-factor and redirects them back to the OpenStack dashboard.

Known issues/limitations

The OpenStack platform is configured to support authentication and account management using both platform-specific features (Keystone/Adjutant) and those provided by IDP.

The following sections describe known issues, limitations and relevant workarounds.

Mandatory usage of MFA

All users enrolled are required to configure both a password and a second factor for authentication. Enforcement of MFA for all non-services users is planned.

CLI and Terraform usage

Applications that use the OpenStack APIs directly, such as the official CLI utilities, Terraform and similar automation tools, do not support authentication using the identity provider. For these use-cases, a dedicated service account or usage of application credentials is highly recommended.

Logout functionality in Horizon

Once a user who has authenticated through the IDP logs out of the OpenStack web dashboard (Horizon), their browser session token remains valid for a short period of time. This issue has been mitigated in later versions of OpenStack and the only currently available work-around is to manually clear all browser cookies for the “ops.elastx.cloud” origin.

User settings and project management

Users enrolled can not utilize functionality exposed through the OpenStack web dashboard for account management, such as change/reset password features. These actions may instead be performed through the “My Account” web page.

While the ability for project moderators to invite other users to their projects still exist, it should be noted that new users won’t be automatically enrolled.

Full credential reset

Users can initiate a password reset by clicking on the “Forgot Password?” link below the login form. The password reset flow requires that user authenticate using their second-factor before proceeding.

If the user has lost access to their second-factor (multiple can be configured as a backup), Elastx Support must be contacted for additional identity validation before both credentials are reset. This process is by design for security reasons, but may be reconsidered before the service is generally available.

7 - Kubernetes CaaS

7.1 - Announcements

2024-06-28 New default Kubernetes StorageClass

We are changing the default StorageClass from v1-dynamic-40 to v2-1k ahead of schedule.

Background
The change was already implemented by accident to clusters created on versions after v1.26, which meant we diverged from advertised changes in our changelog.

By committing to doing this change actively for all clusters, we’re catching up to reality and uniforming our clusters.

Impact
Customers who are not specifying a storage class, will not have any impact on creating new volumes. In this scenario, a volume will be created with the v2-1k as the new default. Customers that actively specify the old v1-dynamic-40 will face no impact as this StorageClass is still supported.

General note
To simplify the future necessary migration to v2 storage classes, please consider to stop creating new volumes using StorageClass that are not prefixed with “v2-”.

A list of available StorageClasses and their respective pricing can be found here: https://elastx.se/en/openstack/pricing

A guide on how to migrate volumes between StorageClasses can be found here: https://docs.elastx.cloud/docs/kubernetes/guides/change_storageclass/

2024-03-08 Kubernetes v1.26 upgrade notice

To ECP customers that have not yet upgraded to v1.26, this announcement is valid for you.

We have received, and acted upon, customer feedback regarding the v1.26 upgrade. Considering valuable feedback received from our customers, we are introducing two new options to ensure a suitable upgrade path for your cluster.

Ingress and Certmanager

• We will not require customers to take ownership of the Ingress and Certmanager as advertised previously. We will continue to provide a managed Ingress/Certmanager.

A new cluster free of charge

• You have the option to request a new cluster, in which you can setup your services at your own pace. You can choose the kubernetes version, we support 1.26 or 1.29 (soon 1.30). The cluster is free of charge for 30 days and after that, part of your standard environment.
• We expect you to migrate your workloads from the old cluster to the new one, and then cancel the old cluster via a ZD ticket.

What’s next?
Our team will initiate contact via a Zendesk ticket to discuss your preferences and gather the necessary configuration options. We will initially propose a date and time for the upgrade.

Meanwhile, please have a look at our updated version of the migration guide to v1.26:
https://docs.elastx.cloud/docs/kubernetes/knowledge-base/migration-to-caasv2/.

In case you have any technical inquiries please submit a support ticket at:
https://support.elastx.se.

We are happy to help and guide you through the upgrade process.

2023-12-08 Kubernetes CaaS updates including autoscaling

We are happy to announce our new Kubernetes CaaS lifecycle management with support for both worker node auto scaling and auto healing. We have reworked a great deal of the backend for the service which will speed up changes, allow you to run clusters in a more efficient way as well as being able to handle increased load without manual intervention.

All new clusters will automatically be deployed using our new backend. Existing clusters will need to be running Kubernetes 1.25 in order to be upgraded. We plan to contact all customers during Q1 2024 in order to plan this together with the Kubernetes 1.26 upgrade.

When upgrading, there are a few changes that need immediate action. Most notably the ingress will be migrated to a load balancer setup. We have information on all changes in more detail here: https://docs.elastx.cloud/docs/kubernetes/knowledge-base/migration-to-caasv2/

You can find information, specifications and pricing here, https://elastx.se/en/kubernetes/.

Service documentation is available here, https://docs.elastx.cloud/docs/kubernetes/.

If you have any general questions or would like to sign-up please contact us at hello@elastx.se.

For any technical questions please register a support ticket at https://support.elastx.se.

7.2 - Overview

Elastx Kubernetes CaaS consists of a fully redundant Kubernetes cluster spread over three separate physical locations (availability zones) in Stockholm, Sweden. We offer managed addons and monitoring 24x7, including support.

Features

Elastx Kubernetes CaaS runs on top of our high perfomant OpenStack IaaS platform and we integrate with the features it provides.

• High availability: Cluster nodes are spread over our three availability zones, combined with our great connectivity this creates a great platform for you to build highly available services on.

• Load Balancer: Services that use the type “LoadBalancer” in Kubernetes integrate with OpenStack Octavia. Each service exposed this way gets its own public IP (Floating IP in OpenStack lingo).

• Persistent Storage: When creating a Persistent Volume Claim Kubernetes creates a volume using OpenStack Cinder and then connects the volume on the node where your pod(s) gets scheduled.

• Auto scaling: Starting in CaaS 2.0 we offer node autoscaling. Autoscaling works by checking the resources your workload is requesting. Autoscaling can help you scale your clusters in case you need to run jobs or when yur application scales out due to more traffic or users than normal.

• Standards conformant: Our clusters are certified by the CNCF Conformance Program ensuring interoperability with Cloud Native technologies and minimizing vendor lock-in.

Good to know

Design your Cloud

We expect customers to design their setup to not require access to Openstack Horizon. This is to future proof the product. This means, do not place other instances in the same Openstack project, nor utilize Swift (objectstore) in the same project. We are happy to provide a separate Swiftproject, and a secondary Openstack project for all needs.

Persistent volumes

Cross availability zone mounting of volumes is not supported. Therefore, volumes can only be mounted by nodes in the same availability zone.

Ordering and scaling

Ordering and scaling of clusters is currently a manual process involving contact with either our sales department or our support. This is a known limitation, but we are quick to respond and a cluster is typically delivered within a business day.

Since Elastx Private Kubernetes 2.0 we offer auto scaling of workload nodes. This is based on resource requests, which means it relies on the administator to set realistic requests on the workload. Configuring auto-scaling options is currently a manual process involving contact with either our sales department or our support.

Cluster add-ons

We offer a managed cert-manager and a managed NGINX Ingress Controller.

If you are interested in removing any limitations, we’ve assembled guides with everything you need to install the same IngressController and cert-manager as we provide. This will give you full control. The various resources gives configuration examples, and instructions for lifecycle management. These can be found in the sections Getting Started and Guides.

7.3 - Getting started

7.3.1 - Accessing your cluster

In order to access your cluster there are a couple of things you need to do. First you need to make sure you have the correct tools installed, the default client for interacting with Kubernetes clusters is called kubectl. Instructions for installing it on your system can be found by following the link.

You may of course use any Kubernetes client you wish to access your cluster however setting up other clients is beyond the scope of this documentation.

Credentials (kubeconfig)

Once you have a client you can use to access the cluster you will need to fetch the credentials for you cluster. You can find the credentials for your cluster by logging in to Elastx OpenStack IaaS. When logged in you can find the kubeconfig file for your cluster by clicking on the “Object Storage” menu option in the left-hand side menu. And then click on “Containers”, you should now see a container with the same name as your cluster (clusters are named “customer-cluster_name”). Clicking on the container should reveal a file called admin.conf in the right-hand pane. Click on the `Download" button to the right of the file name to download it to your computer.

NOTE These credentials will be rotated when your cluster is upgraded so you should periodically fetch new credentials to make sure you have a fresh set.

NOTE The kubeconfig you just downloaded has full administrator privileges.

Configuring kubectl to use your credentials

In order for kubectl to be able to use the credentials you just downloaded you need to either place the credentials in the default location or otherwise configure kubectl to utilize them. The official documentation covers this process in detail.

Verify access

To verify you’ve got access to the cluster you can run something like this:

$ kubectl get nodes
NAME                           STATUS   ROLES           AGE   VERSION
hux-lab1-control-plane-c9bmm   Ready    control-plane   14h   v1.27.3
hux-lab1-control-plane-j5p42   Ready    control-plane   14h   v1.27.3
hux-lab1-control-plane-wlwr8   Ready    control-plane   14h   v1.27.3
hux-lab1-worker-447sn          Ready    <none>          13h   v1.27.3
hux-lab1-worker-9ltbp          Ready    <none>          14h   v1.27.3
hux-lab1-worker-vszmc          Ready    <none>          14h   v1.27.3

If your output looks similar then you should be good to go! If it looks very different or contains error messages, don’t hesitate to contact our support if you can’t figure out how to solve it on your own.

Restrict access

Access to the API server is controlled in the loadbalancer in front of the API. Currently, managing the IP-range allowlist requires a support ticket here. All Elastx IP ranges are always included.

Instructions for older versions

Everything under this section is only for clusters running older versions of our private Kubernetes service.

Security groups

Note: This part only applies to clusters not already running Private Kubernetes 2.0 or later.

If your cluster was created prior to Kubernetes 1.26 or when we specifically informed you that this part applies.

If you are not sure if this part applies, you can validate it by checking if there is a security group called cluster-name-master-customer in your openstack project.

To do so, log in to Elastx Openstack IaaS. When logged in click on the “Network” menu option in the left-hand side menu. Then click on “Security Groups”, finally click on the “Manage Rules” button to the right of the security group named cluster-name-master-customer. To add a rule click on the “Add Rule” button.

For example, to allow access from the ip address 1.2.3.4 configure the rule as follows:

Rule: Custom TCP Rule
Direction: Ingress
Open Port: Port
Port: 6443
Remote: CIDR
CIDR: 1.2.3.4/32

Once you’ve set up rules that allow you to access your cluster you are ready to verify access.

7.3.2 - Auto Healing

In our Kubernetes Services, we have implemented a robust auto-healing mechanism to ensure the high availability and reliability of our infrastructure. This system is designed to automatically manage and replace unhealthy nodes, thereby minimizing downtime and maintaining the stability of our services.

Auto-Healing Mechanism

Triggers

1. Unready Node Detection:

   • The auto-healing process is triggered when a node remains in an “not ready” or “unknown” state for 15 minutes.
   • This delay allows for transient issues to resolve themselves without unnecessary node replacements.

2. Node Creation Failure:

   • To ensure new nodes are given adequate time to initialize and join the cluster, we have configured startup timers:
     • Control Plane Nodes:
       • A new control plane node has a maximum startup time of 30 minutes. This extended period accounts for the critical nature and complexity of control plane operations.
     • Worker Nodes:
       • A new worker node has a maximum startup time of 10 minutes, reflecting the relatively simpler setup process compared to control plane nodes.

Actions

1. Unresponsive Node:
   • Once a node is identified as unready for the specified duration, the auto-healing system deletes the old node.
   • Simultaneously, it initiates the creation of a new node to take its place, ensuring the cluster remains properly sized and functional.

Built-in Failsafe

To prevent cascading failures and to handle scenarios where multiple nodes become unresponsive, we have a built-in failsafe mechanism:

• Threshold for Unresponsive Nodes:
  • If more than 35% of the nodes in the cluster become unresponsive simultaneously, the failsafe activates.
  • This failsafe blocks any further changes, as such a widespread issue likely indicates a broader underlying problem, such as network or platform-related issues, rather than isolated node failures.

By integrating these features, our Kubernetes Services can automatically handle node failures and maintain high availability, while also providing safeguards against systemic issues. This auto-healing capability ensures that our infrastructure remains resilient, responsive, and capable of supporting continuous service delivery.

7.3.3 - Auto Scaling

We now offer autoscaling of nodes.

What is a nodegroup?

In order to simplify node management we now have nodegroup.

A nodegroup is a set of nodes, They span over all 3 of our availability zones. All nodes in a nodegroup are using the same flavour. This means if you want to mix flavours in your cluster there will be at least one nodegroup per flavor. We can also create custom nodegroups upon requests meaning you can have 2 nodegroups with the same flavour.

By default clusters are created with one nodegroup called “worker”. When listing nodes by running kubectl get nodes you can see the node group by looking at the node name. All node names begin with clustername - nodegroup.

In the example below we have the cluster hux-lab1 and can see the default workers are located in the nodegroup worker and additionally, the added nodegroup nodegroup2 with a few extra nodes.

❯ kubectl get nodes
NAME                           STATUS   ROLES           AGE     VERSION
hux-lab1-control-plane-c9bmm   Ready    control-plane   2d18h   v1.27.3
hux-lab1-control-plane-j5p42   Ready    control-plane   2d18h   v1.27.3
hux-lab1-control-plane-wlwr8   Ready    control-plane   2d18h   v1.27.3
hux-lab1-worker-447sn          Ready    <none>          2d18h   v1.27.3
hux-lab1-worker-9ltbp          Ready    <none>          2d18h   v1.27.3
hux-lab1-worker-htfbp          Ready    <none>          15h     v1.27.3
hux-lab1-worker-k56hn          Ready    <none>          16h     v1.27.3
hux-lab1-nodegroup2-33hbp      Ready    <none>          15h     v1.27.3
hux-lab1-nodegroup2-54j5k      Ready    <none>          16h     v1.27.3

How to activate autoscaling?

Autoscaling currently needs to be configured by Elastx support.

In order to activate auto scaling we need to know clustername and nodegroup with two values for minimum/maximum number of desired nodes. Currently we have a minimum set to 1 node per AZ.

Nodegroups are by default split into availability zones, if you want 3 nodes you get one in each availability zone. We have the option to lock a node group to a single AZ.

Examples: Default nodegroup in 3 AZ: Minimum of 3 nodes and maximum of 7. This would translate to minimum one node per availability zone and maximum nodes AZ-1: 3, AZ-2: 2, AZ-3: 2. Any number of max nodes is supported.

Opt for foot print nodegroup: 1 Az. Min 1 node. Any number of max nodes is supported. Opt for 2 AZ redundancy: nodegroup with 2 nodes, 2 AZ, can accomodate for min 2 nodes and any number of max nodes.

If you are unsure contact out support and we will help you get the configuration you wish for.

How does autoscaling know when to add additional nodes?

Nodes are added when they are needed. There are two scenarios:

1. You have a pod that fails to be scheduled on existing nodes
2. Scheduled pods requests more then 100% of any resource, this method is smart and senses the amount of resources per node and can therfor add more than one node at a time if required.

When does the autoscaler scale down nodes?

The autoscaler removes nodes when it senses there is enough free resources to accomodate all current workload (based on requests) on fewer nodes. To avoid all nodes having 100% resource requests (and thereby usage), there is also a built-in mechanism to ensure there is always at least 50% of a node available resources to accept additional requests.

Meaning if you have a nodegroup with 3 nodes and all of them have 4 CPU cores you need to have a total of 2 CPU cores that is not requested per any workload.

To refrain from triggering the auto-scaling feature excessively, there is a built in delay of 10 minutes for scale down actions to occur. Scale up events are triggered immediately.

Can I disable auto scaling after activating it?

Yes, just contact Elastx support and we will help you with this.

When disabling auto scaling node count will be locked. Contact support if the number if nodes you wish to keep deviates from current amount od nodes, and we will scale it for you.

7.3.4 - Cluster configuration

There are a lot of options possible for your cluster. Most options have a sane default however could be overridden on request.

A default cluster comes with 3 control plane and 3 worker nodes. To connect all nodes we create a network, default (10.128.0.0/22). We also deploy monitoring to ensure functionality of all cluster components. However most things are just a default and could be overridden.

Common options

Nodes

The standard configuration consists of the following:

• Three control plane nodes, one in each of our availability zones. Flavor: v2-c2-m8-d80
• Three worker nodes, one in each of our availability zones, in a single nodegroup. Flavor: v2-c2-m8-d80

Minimal configuration

• Three control plane nodes, one in each of our availability zones. Flavor: v2-c2-m8-d80

• One worker node, Flavor: v2-c2-m8-d80

  This is the minimal configuration offered. Scaling to larger flavors and adding nodes are supported. Autoscaling is not supported with a single worker node.

  Note: SLA is different for minimal configuration type of cluster. SLA’s can be found here.

Nodegroups and multiple flavors

A nodegroup contains of one or multiple nodes with the same flavor and a list of availability zones to deploy nodes in. Clusters are default delivered with a single nodegroup containing 3 nodes, one in each AZ. Each nodegroup is limited to one flavor.

You could have multiple nodegroups, if you for example want to target workload on separate nodes or in case you wish to consume multiple flavors.

A few examples of nodegroups:

In the examples we could see worker our default nodegroup and an example of having separate nodes for databases and frontend where the database is running on dedicated nodes and the frontend is running on smaller nodes but can autoscale between 3 and 12 nodes based on current cluster request. We also have a jobs nodegroup where we have one node in sto1 but can scale up to 3 nodes where all are placed inside STO1. You can read more about autoscaling here.

Nodegroups can be changed at any time. Please also note that we have auto-healing meaning in case any of your nodes for any reason stops working we will replace them. More about autohealing could be found here.

Worker nodes Floating IPs

By default, our clusters come with nodes that do not have any Floating IPs attached to them. If, for any reason, you require Floating IPs on your workload nodes, please inform us, and we can configure your cluster accordingly. It’s worth noting that the most common use case for Floating IPs is to ensure predictable source IPs. However, please note that enabling or disabling Floating IPs will necessitate the recreation of all your nodes.

Since during upgrades we create a new node prior to removing an old node you would need to have an additional IP adress on standby. Thus, for a 3 worker nodes, with with autoscaling up to 5 nodes, we will allocate 6 IPs.

Network

By default we create a node network (10.128.0.0/22). However we could use another subnet per customer request. The most common scenario is when customer request another subnet is when exposing multiple Kubernetes clusters over a VPN.

Please make sure to inform us if you wish to use a custom subnet during the ordering process since we cannot replace the network after creation, meaning we would then need to recreate your entire cluster.

We currently only support cidr in the 10.0.0.0/8 subnet range and at least a /24. Both nodes and loadbalancers are using IPs for this range meaning you need to have a sizable network from the beginning.

Cluster domain

We default all clusters to “cluster.local”. If you wish to have another cluster domain please let us know during the ordering procedure since it cannot be replaced after cluster creation.

OIDC

If you wish to integrate with your existing OIDC compatible IDP, example Microsoft AD And Google Workspace that is supported directy in the kubernetes api service.

By default we ship clusters with this option disabled however if you wish to make use of OIDC just let us know when order the cluster or afterwards. OIDC can be enabled, disabled or changed at any time.

Kubelet configurations and resource reservations

We make a few adaptations to Kubernetes vanilla settings.

• NodeDrainVolume and NodeDrainTimeout: 5 -> 15min

  • Increased duration to 15 minutes to allow more time for graceful shutdown and controlled startup of workload on new nodes, while respecting PodDisruptionBudgets.

• podPidsLimit: 0 → 4096

  • Added safety net of a maximum of Per-pod PIDs (process IDs), that is limited and enforced by the kubelet. We used to not have any limitation. Setting this to 4096 limits how many PIDs a single pod may create, which helps mitigate runaway processes or fork-bombs.

• serializeImagePulls: true → false

  • Allows the kubelet to pull multiple images in parallel, speeding up startup times.

• maxParallelImagePulls: 0 → 10

  • Controls the maximum number of image pulls the kubelet will perform in parallel.

Resource reservations on worker nodes

To improve stability and predictability of the core Kubernetes functionality during heavy load, we introduce node reservations for CPU, memory, and ephemeral storage.

The reservation model follows proven hyperscaler formulas but is tuned conservatively, ensuring more allocatable resources.

Hyperscalers tend to not make a distinction of systemReserved and kubeReserved, and bundle all reservations into and kubeReserved. We make use of both, but skewed towards kube reservations to align closer with Hyperscalers, but still maintain the reservations of the system. We calculate the reservations settings based on cpu cores, memory and storage of each flavor dynamically.

Here we’ve provided a sample of what to expect:

CPU Reservations Table

Memory Reservations

Ephemeral Disk Reservations

NOTE: We use the default of nodefs.available at 10%.

Cluster add-ons

We currently offer managed cert-manager, NGINX Ingress and elx-nodegroup-controller.

Cert-manager

Cert-manager (link to cert-manager.io) helps you to manage TLS certificates. A common use case is to use lets-encrypt to “automatically” generate certificates for web apps. However the functionality goes much deeper. We also have usage instructions and have a guide if you wish to deploy cert-manager yourself.

Ingress

An ingress controller in a Kubernetes cluster manages how external traffic reaches your services. It routes requests based on rules, handles load balancing, and can integrate with cert-manager to manage TLS certificates. This simplifies traffic handling and improves scalability and security compared to exposing each service individually. We have a usage guide with examples that can be found here.

We have chosen to use ingress-nginx and to support ingress, we limit what custom configurations can be made per cluster. We offer two “modes”. One that we call direct mode, which is the default behavior. This mode is used when end-clients connect directly to your ingress. We also have a proxy mode for when a proxy (e.g., WAF) is used in front of your ingress. When running in proxy mode, we also have the ability to limit traffic from specific IP addresses, which we recommend doing for security reasons. If you are unsure which mode to use or how to handle IP whitelisting, just let us know and we will help you choose the best options for your use case.

If you are interested in removing any limitations, we’ve assembled guides with everything you need to install the same IngressController as we provide. This will give you full control. The various resources give configuration examples and instructions for lifecycle management. These can be found here.

elx-nodegroup-controller

The nodegroup controller is useful when customers want to use custom taints or labels on their nodes. It supports matching nodes based on nodegroup or by name. The controller can be found on Github if you wish to inspect the code or deploy it yourself.

7.3.5 - Cluster upgrades

Introduction

Kubernetes versions are released approximately three times a year, introducing enhancements, security updates, and bug fixes. The planning and initiation of a cluster upgrade is a manual task that requires coordination with our customers.

To schedule the upgrade of your cluster(s), we require a designated point of contact for coordination.
For customers with multiple clusters, please provide your preferred sequence and timeline for upgrades. If you haven’t shared this information yet, kindly submit a support ticket with these details.

Upgrade Planning

Upgrades are scheduled in consultation with the customer and can be done on at the initiative of either Elastx or the customer. If the customer does not initiate the planning of an upgrade, we will reach out to the designated contact in a support ticket at least twice a year with suggested upgrade dates.

NOTE: Upgrades are not performed during our changestop periods:

• In general the full month of July and through the first week of August
• December 23rd to January 2nd

Before scheduling and confirming a time slot, please review the relevant changelog and the Kubernetes Deprecated API Migration guide:

• Elastx PcaaS Changelog: https://docs.elastx.cloud/docs/kubernetes/changelog/
• Kubernetes Deprecated API Migration Guide: https://kubernetes.io/docs/reference/using-api/deprecation-guide/

Upgrade Process

NOTE Please refrain from making any changes while the upgrade is in progress.

The duration of the upgrade typically ranges from 1 to 3 hours, depending on the size of the cluster.
The upgrade starts with the control plane nodes followed by the worker nodes, one nodegroup at a time.

Steps Involved

1. A new node with the newer version is added to the cluster to replace the old node.
2. Once the new node is ready, the old node is drained.
3. Once all transferable loads have been migrated, the old node is removed from the cluster.
4. This process is repeated until all nodes in the cluster have been upgraded.

NOTE When using public IPs on worker nodes to ensure predictable egress IP, a previously unused IP will be assigned to the new worker node. This IP should have been provided to you in a list of all allocated IPs during your request for adding public IPs on the worker nodes.

Support and Communication During Upgrades

The engineer responsible for executing the upgrade will notify you through the support ticket when the upgrade begins and once it is completed. The support ticket serves as the primary channel for communication during the upgrade process. If you have any concerns or questions about the upgrade, please use the support ticket to reach out.

Additional Information

• Upon request, upgrades can be scheduled outside office hours if needed. Upgrades outside of office hours depend on personnel availability and comes at an additional fee, see current price for professional services.
• Our Kubernetes service includes up to four version upgrades per year; additional upgrades can be performed at an extra cost.
• To address critical security vulnerabilities, additional upgrades can be performed and will not count against the four upgrades included per year.
• In a previous Tech-fika, we discussed how to build redundancy and implement autoscaling with our Kubernetes service. You can access the presentation here to help you prepare for a smoother upgrade experience.

7.3.6 - Kubernetes API whitelist

In our Kubernetes Services, we rely on Openstack loadbalancers in front of the control planes to ensure traffic will be sent to a functional node. Whitelisting of access to the API server is now controlled in the loadbalancer in front of the API. Currently, managing the IP-range whitelist requires a support ticket here.

Please submit a ticket with CIDR/ranges for the ip’s you wish to whitelist. We are happy to help you ASAP.

Note: All Elastx IP ranges are always included.

In the future, we expect to have this functionality available self-service style.

7.3.7 - Order a new cluster

How to order or remove a cluster

Ordering and scaling of clusters is currently a manual process involving contact with either our sales department or our support. This is a known limitation, but may change in the future.

7.3.8 - Recommendations

This page describes a list of things that could help you get the best experience out of your cluster.

Note: You do not need to follow this documentation in order to use your cluster

Ingress and cert-manager

To make it easier to expose applications an ingress controller is commonly deployed.

An ingress controller makes sure when you go to a specific webpage you are routed towards the correct application.

There are a lot of different ingress controllers available. We on Elastx are using ingress-nginx and have a guide ready on how to get started. However you can deploy any ingress controller you wish inside your clusters.

To get a single IP-address you can point your DNS towards we recommend to deploy an ingress-controller with a service of type LoadBalancer. More information regarding Load Balancers can be found here.

In order to automatically generate and update TLS certificates cert-manager is commonly deployed side by side with an ingress controller.

We have created a guide on how to get started with ingress-nginx and Cert-manager that can be found here.

Requests and limits

Below we describe requests and limits briefly. For a more detailed description or help setting requests and limits we recommend to check out Kubernetes documentation here.

Requests

Requests and limits are critical to enable Kubernetes to make informed decisions on when and where to schedule and limit your workload.

Requests are important for the scheduler. Requests can be seen as “Amount of resources the pod would utilize during normal operation”. This means that the scheduler will allocate the required amount of resources and make sure they are always available to your pod.

Requests also enables the auto-scaler to make decisions on when to scale a cluster up and down.

Limits

Limits define the maximum allowed resource usage for a pod. This is important to avoid slowdowns in other pods running on the same node.

CPU limit. Your application will be throttled or simply run slower when trying to exceed the limit. Run slower equals to fewer cpu cycles per given time. That is, introducing latency. Memory limit is another beast. If any pod trying to use memory above the the limit, the pod will be Out of memory killed.

Autoscaling

Autoscaling can operate on both node-level and pod-level. To get the absolute best experience we recommend a combination of both.

Scaling nodes

We have built-in support for scaling nodes. To get started with autoscaling we recommend to check the guide here.

Scaling pods

Kubernetes official documentation has a guide on how to accomplish this can be found here.

In short, node autoscaling is only taken into consideration if you have any pods that cannot be scheduled or if you have set requests on your pods. In order to automatically scale an application pod scaling can make sure you get more pods before reaching your pod limit and if more nodes are needed in order to run the new pods nodes will automatically be added and then later removed when no longer needed.

Network policies

Network policies can in short be seen as Kubernetes built in firewalls.

Network policy can be used to limit both incoming and outgoing traffic. This is useful to specify a set of pods that are allowed to communicate with the database.

Kubernetes documentation have an excellent guide on how to get started with network policies here.

Pod Security Standards / Pod Security Admission

Pod Security Admission can be used to limit what your pods can do. For example you can make sure pods are not allowed to run as root.

In order to get to know this more in detail and getting started we recommend to follow the Kubernetes documentation here.

Load Balancers

Load Balancers allow your application to be accessed from the internet. Load Balancers can automatically split traffic to all your nodes to even out load. Load Balancers can also detect if a node is having problems and remove it to avoid displaying errors to end users.

We have a guide on how to get started with Boad Balancers here.

7.4 - Guides

7.4.1 - Cert-manager and Cloudflare demo

In this guide we will use a Cloudflare managed domain and a our own cert-manager to provide LetsEncrypt certificates for a test deployment.

The guide is suitable if you have a domain connected to a single cluster, and would like a to issue/manage certificates from within kubernetes. The setup below becomes Clusterwider, meaning it will deploy certificates to any namespace specifed.

Prerequisites

• DNS managed on Cloudflare
• Cloudflare API token
• Installed cert-manager. See our guide here.
• Installed IngressController. See our guide here.

Setup ClusterIssuer

Create a file to hold the secret of your api token for your Cloudflare DNS. Then create the ClusterIssuer configuration file adapted for Cloudflare.

apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token
  namespace: cert-manager
type: Opaque
stringData:
  api-token: "<your api token>"
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: cloudflare-issuer
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: <your email>
    privateKeySecretRef:
      name: cloudflare-issuer-key
    solvers:
    - dns01:
        cloudflare:
          email: <your email>
          apiTokenSecretRef:
            name: cloudflare-api-token
            key: api-token

kubectl apply -f cloudflare-issuer.yml

The clusterIssuer is soon ready. Example output:

kubectl get clusterissuers.cert-manager.io 
NAME                READY   AGE
cloudflare-issuer   True    6d18h

Expose a workload and secure with Let’s encrypt certificate

In this section we will setup a deployment, with it’s accompanying service and ingress object. The ingress object will request a certificate for test2.domain.ltd, and once fully up and running, should provide https://test2.domain.ltd with a valid letsencrypt certificate.

We’ll use the created ClusterIssuer and let cert-manager request new certificates for any added ingress object. This setup requires the “*” record setup in the DNS provider.

This is how the DNS is setup in this particular example: A A record (“domain.ltd”) points to the loadbalancer IP of the cluster. A CNAME record refers to ("*") and points to the A record above.

This example also specifies the namespace “echo2”.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo2-dep
  namespace: echo2
spec:
  selector:
    matchLabels:
      app: echo2
  replicas: 1
  template:
    metadata:
      labels:
        app: echo2
    spec:
      containers:
      - name: echo2
        image: hashicorp/http-echo
        args:
        - "-text=echo2"
        ports:
        - containerPort: 5678
      securityContext:
        runAsUser: 1001
        fsGroup: 1001
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: echo2
  name: echo2-service
  namespace: echo2
spec:
  ports:
    - protocol: TCP
      port: 5678
      targetPort: 5678
  selector:
    app: echo2
  type: ClusterIP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: echo2-ingress
  namespace: echo2
  annotations:
    cert-manager.io/cluster-issuer: cloudflare-issuer
    kubernetes.io/ingress.class: "nginx"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - test2.domain.ltd
    secretName: test2-domain-tls
  rules:
  - host: test2.domain.ltd
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: echo5-service
            port:
              number: 5678

The DNS challenge and certificate issue process takes a couple of minutes. You can follow the progress by watching:

kubectl events -n cert-manager

Once completed, it shall all be accessible at http://test2.domain.ltd

7.4.2 - Change PV StorageClass

This guide details all steps to change storage class of a volume. The instruction can be used to migrate from one storage class to another, while retaining data. For example from 8kto v2-4k.

Prerequisites

• Access to the kubernetes cluster
• Access to Openstack kubernetes Project

Preparation steps

1. Populate variables

   Complete with relevant names for your setup. Then copy/paste them into the terminal to set them as environment variables that will be used throughout the guide. PVC is the

   PVC=test1
   NAMESPACE=default
   NEWSTORAGECLASS=v2-1k
   

2. Fetch and populate the PV name by running:

   PV=$(kubectl get pvc -n $NAMESPACE $PVC -o go-template='{{.spec.volumeName}}')
   

3. Create backup of PVC and PV configurations

   Fetch the PVC and PV configurations and store in /tmp/ for later use:

   kubectl get pvc -n $NAMESPACE $PVC -o yaml | tee /tmp/pvc.yaml
   kubectl get pv  $PV -o yaml | tee /tmp/pv.yaml
   

4. Change VolumeReclaimPolicy

   To avoid deletion of the PV when deleting the PVC, the volume needs to have VolumeReclaimPolicy set to Retain.

   Patch:

   kubectl patch pv $PV -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
   

5. Stop pods from accessing the mounted volume (ie kill pods/scale statefulset/etc..).

6. Delete the PVC.

   kubectl delete pvc -n "$NAMESPACE" "$PVC"
   

Login to Openstack

1. Navigate to: Volumes -> Volumes

2. Make a backup of the volume From the drop-down to the right, select backup. The backup is good practice, not used in the following steps.

3. Change the storage type to desired type. The volume should now or shortly have status Available. Dropdown to the right, Edit volume -> Change volume type:

   • Select your desired storage type
   • Select Migration policy=Ondemand

   The window will close, and the volume will be updated and migrated (to the v2 storage platform) if necessary, by the backend. The status becomes “Volume retyping”. Wait until completed.

   We have a complementary guide here.

Back to kubernetes

1. Release the tie between PVC and PV

   The PV is still referencing its old PVC, in the claimRef, found under spec.claimRef.uid. This UID needs to be nullified to release the PV, allowing it to be adopted by a PVC with correct storageClass.

   Patch claimRef to null:

   kubectl patch pv "$PV" -p '{"spec":{"claimRef":{"namespace":"'$NAMESPACE'","name":"'$PVC'","uid":null}}}'
   

2. The PV StorageClass in kubernetes does not match to its counterpart in Openstack.

   We need to patch the storageClassName reference in the PV:

   kubectl patch pv "$PV" -p '{"spec":{"storageClassName":"'$NEWSTORAGECLASS'"}}'
   

3. Prepare a new PVC with the updated storageClass

   We need to modify the saved /tmp/pvc.yaml.

   1. Remove “last-applied-configuration”:

      sed -i '/kubectl.kubernetes.io\/last-applied-configuration: |/ { N; d; }' /tmp/pvc.yaml
      

   2. Update existing storageClassName to the new one:

      sed -i 's/storageClassName: .*/storageClassName: '$NEWSTORAGECLASS'/g' /tmp/pvc.yaml
      

4. Apply the updated /tmp/pvc.yaml

   kubectl apply -f /tmp/pvc.yaml
   

5. Update the PV to bind with the new PVC

   We must allow the new PVC to bind correctly to the old PV. We need to first fetch the new PVC UID, then patch the PV with the PVC UID so kubernetes understands what PVC the PV belongs to.

   1. Retrieve the new PVC UID:

      PVCUID=$(kubectl get -n "$NAMESPACE" pvc "$PVC" -o custom-columns=UID:.metadata.uid --no-headers)
      

   2. Patch the PV with the new UID of the PVC:

      kubectl patch pv "$PV" -p '{"spec":{"claimRef":{"uid":"'$PVCUID'"}}}'
      

6. Reset the Reclaim Policy of the volume to Delete:

   kubectl patch pv $PV -p '{"spec":{"persistentVolumeReclaimPolicy":"Delete"}}'
   

7. Completed.

   • Verify the volume works healthily.
   • Update your manifests to reflect the new storageClassName.

7.4.3 - Ingress and cert-manager

Follow along demo

In this piece, we show all steps to expose a web service using an Ingress resource. Additionally, we demonstrate how to enable TLS, by using cert-manager to request a Let’s Encrypt certificate.

Prerequisites

1. A DNS record pointing at the public IP address of your worker nodes. In the examples all references to the domain example.ltd must be replaced by the domain you wish to issue certificates for. Configuring DNS is out of scope for this documentation.
2. For clusters created on or after Kubernetes 1.26 you need to ensure there is a Ingress controller and cert-manager installed.
   • Our guide to ingress-nginx can be found here
   • Our guide to installing cert-manager can be found here
   • Upstream documentation: NGINX Ingress Controller
   • Upstream documentation: cert-manager

Create resources

Create a file called ingress.yaml with the following content:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: my-web-service
  name: my-web-service
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-web-service
  template:
    metadata:
      labels:
        app: my-web-service
    spec:
      securityContext:
        runAsUser: 1001
        fsGroup: 1001
      containers:
      - image: k8s.gcr.io/serve_hostname
        name: servehostname
        ports:
        - containerPort: 9376
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: my-web-service
  name: my-web-service
spec:
  ports:
  - port: 9376
    protocol: TCP
    targetPort: 9376
  selector:
    app: my-web-service
  type: ClusterIP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-web-service-ingress
  annotations:
    cert-manager.io/issuer: letsencrypt-prod
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - example.tld
    secretName: example-tld
  rules:
  - host: example.tld
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: my-web-service
            port:
              number: 9376