This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Admin Guide

Provision your infrastructure

1: Setup Kahuna
2: Provisioning Users
3: Create Your First Region
4: Provisioning Kiwi

1 - Setup Kahuna

Let’s start with the orchestration core

Now let’s suppose that you’ve cloned the Git platform repository template and that your Kahuna instance server has been provisioned with latest Ubuntu LTS distribution. Be sure that it is SSH-accessible with some local user.

Let’s take the following assumptions for the rest of this tutorial:

We only have one single Kahuna instance (no high-availability).
Local bootstrap user with sudo privileges is ubuntu, with key-based SSH authentication.
Kahuna instance is public-Internet exposed through IP address 1.2.3.4, translated to kowabunga.acme.com DNS.
Kahuna instance is private-network exposed through IP address 10.0.0.1.
Kahuna instance hostname is kowabunga-kahuna-1.

Setup DNS

Please ensure that your kowabunga.acme.com domain translates to public IP address 1.2.3.4. Configuration is up to you and your DNS provider and can be done manually.

Being IaC-supporters, we advise using OpenTofu for that purpose. Let’s see how we can do, using Cloudflare DNS provider.

Start by editing the terraform/providers.tf file in your platform’s repository:

terraform {
  required_providers {
    cloudflare = {
      source  = "cloudflare/cloudflare"
      version = "~> 5"
    }
  }
}

provider "cloudflare" {
  api_token = local.secrets.cloudflare_api_token
}

extend the terraform/secrets.tf file with:

locals {
  secrets = {
    cloudflare_api_token = data.sops_file.secrets.data.cloudflare_api_token
  }
}

and add the associated:

cloudflare_api_token: MY_PREVIOUSLY_GENERATED_API_TOKEN

variable in terraform/secrets.yml file thanks to:

$ kobra secrets edit terraform/secrets.yml

Then, simply edit your terraform/main.tf file with the following:

resource "cloudflare_dns_record" "kowabunga" {
  zone_id = "ACME_COM_ZONE_ID"
  name    = "kowabunga"
  ttl     = 3600
  type    = "A"
  content = "1.2.3.4"
  proxied = false
}

initialize OpenTofu (once, or each time you add a new provider):

$ kobra tf init

and apply infrastructure changes:

$ kobra tf apply

Inventory Management

It is now time to declare your Kahuna instance in Ansible’s inventory. Simply extend the ansible/inventories/hosts.txt the following way:

[kahuna]
10.0.0.1 name=kowabunga-kahuna-1 ansible_ssh_user=ubuntu

The instance is now declared to be part of kahuna group and Ansible will use ubuntu local user account to connect through SSH.

Note that doing so, you can now safely:

declare host-specific variables in ansible/host_vars/10.0.0.1.yml file.
declare host-specific sensitive variables in ansible/host_vars/10.0.0.1.sops.yml file.
declare kahuna group-specific variables in ansible/group_vars/kahuna/main.yml file.
declare kahuna group-specific sensitive variables in ansible/group_vars/kahuna.sops.yml file.
declare any other global variables in ansible/group_vars/all/main.yml file.
declare any other global sensitive variables in ansible/group_vars/all.sops.yml file.

Note that Ansible variables precedence will apply:

role defaults < all vars < group vars < host vars < role vars

Ansible Kowabunga Collection

Kowabunga comes with an official Ansible Collection and its associated documentation.

The collection contains:

roles and playbooks to easily deploy the various Kahuna, Koala, Kiwi and Kaktus instances.
actions so you can create your own tasks to interact and manage a previously setup Kowabunga instance.

Check out ansible/requirements.yml file to declare the specific collection version you’d like to use:

---
collections:
  - name: kowabunga.cloud
    version: 0.0.1

By default, your platform is configured to pull a tagged official release from Ansible Galaxy. You may however prefer to pull it directly from Git, using latest commit for instance. This can be accomodated through:

---
collections:
  - name: git@github.com:kowabunga-cloud/ansible-collections-kowabunga
    type: git
    version: master

Once defined, simply pull it into your local machine:

$ kobra ansible pull

Kahuna Settings

Kahuna instance deployment will take care of everything. It’ll take the assumption of running a supported Ubuntu LTS release, enforce some configuration and security settings, install the necessary packages, create local admin user accounts, if required, and setup some form of deny-all filtering policy firewall, so you’re safely exposed.

Admin Accounts

Let’s start by declaring some user admin accounts we’d like to create. We don’t want to keep on using the single nominative ubuntu account for everyone after all.

Simply create/edit the ansible/inventories/group_vars/all/main.yml file the following way:

kowabunga_os_user_admin_accounts_enabled:
  - admin_user_1
  - admin_user_2

kowabunga_os_user_admin_accounts_pubkey_dirs:
  - "{{ playbook_dir }}/../../../../../files/pubkeys"

to declare all your expected admin users, and add their respective SSH public key files in the ansible/files/pubkeys directory, e.g.:

$ tree ansible/files/pubkeys/
ansible/files/pubkeys/
└── admin_user_1
└── admin_user_2

Note

Note that all registered admin accounts will have password-less sudo privileges.

We’d also recommend you to set/update the root account password. By default, Ubuntu comes without any, making it impossible to login. Kowabunga’s playbook make sure that root login is prohibited from SSH for security reasons (e.g. brute-force attacks) but we encourage you setting one, as it’s always useful, especially on public cloud VPS or bare metal servers to get a console/IPMI access to log into.

If you intend to do so, simply edit the secrets file:

$ kobra secrets edit ansible/inventories/group_vars/all.sops.yml

and set the requested password:

secret_kowabunga_os_user_root_password: MY_SUPER_SETRONG_PASSWORD

Firewall

If you Kahuna instance is connected on public Internet, it is more than recommended to enable a network firewall. This can be easily done by extending the ansible/inventories/group_vars/kahuna/main.yml file with:

kowabunga_firewall_enabled: true
kowabunga_firewall_open_tcp_ports:
  - 22
  - 80
  - 443

Note that we’re limited opened ports to SSH and HTTP/HTTPS here, which should be more than enough (HTTP is only used by Caddy server for certificate auto-renewal and will redirect traffic to HTTPS anyway). If you don’t expect your instance to be SSH-accessible on public Internet, you can safely drop this line.

MongoDB

Kahuna comes with a bundled, ready-to-be-used MongoDB deployment. This comes in handy if you only have a unique instance to manage. This remains however optional (default), as you may very well be willing to re-use an existing external production-grade MongoDB cluster, already deployed.

If you intend to go with the bundled one, a few settings must be configured in ansible/inventories/group_vars/kahuna/main.yml file:

kowabunga_mongodb_enabled: true
kowabunga_mongodb_listen_addr: "127.0.0.1,10.0.0.1"
kowabunga_mongodb_rs_key: "{{ secret_kowabunga_mongodb_rs_key }}"
kowabunga_mongodb_rs_name: kowabunga
kowabunga_mongodb_admin_password: "{{ secret_kowabunga_mongodb_admin_password }}"
kowabunga_mongodb_users:
  - base: kowabunga
    username: kowabunga
    password: '{{ secret_kowabunga_mongodb_user_password }}'
    readWrite: true

and their associated secrets in ansible/inventories/group_vars/kahuna.sops.yml

secret_kowabunga_mongodb_rs_key: YOUR_CUSTOM_REPLICA_SET_KEY
secret_kowabunga_mongodb_admin_password: A_STRONG_ADMIN_PASSWORD
secret_kowabunga_mongodb_user_password: A_STRONG_USER_PASSWORD

This will basically instruct Ansible to install MongoDB server, configure it with a replicaset (so it can be part of a future cluster instance, we never know), secure it with admin credentials of your choice and create a kowabunga database/collection and associated service user.

Kahuna Settings

Finally, let’s ensure the Kahuna orchestrator gets everything he needs to operate.

You’ll need to define:

a custom email address (and associated SMTP connection settings) for Kahuna to be able to send email notifications to users.
a randomly generated key to sign JWT tokens (please ensure it is secure enough, not to compromise issued tokens robustness).
a randomly generated admin API key. It’ll be used to provision the admin bits of Kowabunga, until proper user accounts have been created.
a private/public SSH key-pair to be used by platform admins to seamlessly SSH into instantiated Kompute instances. Please ensure that the private key is being stored securely somewhere.

Then simply edit the ansible/inventories/group_vars/kahuna/main.yml file the following way:

kowabunga_public_url: "https://kowabunga.acme.com"

kowabunga_kahuna_http_address: "10.0.0.1"
kowabunga_kahuna_admin_email: kowabunga@acme.com
kowabunga_kahuna_jwt_signature: "{{ secret_kowabunga_kahuna_jwt_signature }}"
kowabunga_kahuna_db_uri: "mongodb://kowabunga:{{ secret_kowabunga_mongodb_user_password }}@10.0.0.1:{{ mongodb_port }}/kowabunga?authSource=kowabunga"
kowabunga_kahuna_api_key: "{{ secret_kowabunga_kahuna_api_key }}"

kowabunga_kahuna_bootstrap_user: kowabunga
kowabunga_kahuna_bootstrap_pubkey: "YOUR_ADMIN_SSH_PUB_KEY"

kowabunga_kahuna_smtp_host: "smtp.acme.com"
kowabunga_kahuna_smtp_port: 587
kowabunga_kahuna_smtp_from: "Kowabunga <{{ kowabunga_kahuna_admin_email }}>"
kowabunga_kahuna_smtp_username: johndoe
kowabunga_kahuna_smtp_password: "{{ secret_kowabunga_kahuna_smtp_password }}"

and add the respective secrets into ansible/inventories/group_vars/kahuna.sops.yml:

secret_kowabunga_kahuna_jwt_signature: A_STRONG_JWT_SGINATURE
secret_kowabunga_kahuna_api_key: A_STRONG_API_KEY
secret_kowabunga_kahuna_smtp_password: A_STRONG_PASSWORD

Ansible Deployment

We’re done with configuration (finally) ! All we need to do now is finally run Ansible to make things live. This is done by invoking the kahuna playbook from the kowabunga.cloud collection:

$ kobra ansible deploy -p kowabunga.cloud.kahuna

Note that, under-the-hood, Ansible will use Ansible Mitogen extension to speed things up. Bear in mind that Ansible’s run is idempotent. Anything’s failing can be re-executed. You can also run it as many times you want, or re-run it in the next 6 months or so, provided you’re using a tagged collection, the end result will always be the same.

After a few minutes, if everything’s went okay, you should have a working Kahuna instance, i.e.:

A Caddy frontal reverse-proxy, taking care of automatic TLS certificate issuance, renewal and traffic termination, forwarding requests back to either Koala Web application or Kahuna backend server.
The Kahuna backend server itself, our core orchestrator.
Optionally, MongoDB database.

We’re now ready for provisionning users and teams !

2 - Provisioning Users

Let’s populate admin users and teams

Your Kahuna instance is now up and running, let’s get things and create a few admin users accounts. At first, we only have the super-admin API key that was previously set through Ansible deployment. We’ll make use of it to provision further users and associated teams. After all, we want a nominative user acount for each contributor, right ?

Back to TF config, let’s edit the terraform/providers.tf file:

terraform {
  required_providers {
    kowabunga = {
      source  = "kowabunga-cloud/kowabunga"
      version = ">=0.55.0"
    }
  }
}

provider "kowabunga" {
  uri   = "https://kowabunga.acme.com"
  token = local.secrets.kowabunga_admin_api_key
}

Make sure to edit the Kowabunga provider’s uri with the associated DNS of your freshly deployed Kahuna instance and edit the terraform/secrets.yml file so match the kowabunga_admin_api_key you’ve picked before. OpenTofu will make use of these parameters to connect to your private Kahuna and apply for resources.

Now declare a few users in your terraform/locals.tf file:

locals {
  admins = {
    // HUMANS
    "John Doe" = {
      email  = "john@acme.com",
      role   = "superAdmin",
      notify = true,
    }
    "Jane Doe" = {
      email  = "jane@acme.com",
      role   = "superAdmin",
      notify = true,
    }

    // BOTS
    "Admin TF Bot" = {
      email = "tf@acme.com",
      role  = "superAdmin",
      bot   = true,
    }
  }
}

and the following resources definition in terraform/main.tf:

resource "kowabunga_user" "admins" {
  for_each      = local.admins
  name          = each.key
  email         = each.value.email
  role          = each.value.role
  notifications = try(each.value.notify, false)
  bot           = try(each.value.bot, false)
}

resource "kowabunga_team" "admin" {
  name  = "admin"
  desc  = "Kowabunga Admins"
  users = sort([for key, user in local.admins : kowabunga_user.users[key].id])
}

Then, simply apply for resources creation:

$ kobra tf apply

What we’ve done here was to register a new admin team, with 3 new associated user accounts: 2 regular ones for human administrators and one bot, which you’ll be able to use its API key instead of the super-admin master one to further provision resources if you’d like.

Better do this way as, shall the key be compromised, you’ll only have to revoke it or destroy the bot account, instead of replacing the master one on Kahuna instance.

Newly registered user will be prompted with 2 emails from Kahuna:

a “Welcome to Kowabunga !” one, simply asking yourself to confirm your account’s creation.
a “Forgot about your Kowabunga password ?” one, prompting for a password reset.

Warning

Account’s creation confirmation is required for the user to proceed further. For security purpose, newly created user accounts are locked-down until properly activated.

With security in mind, Kowabunga will prevent you from setting your own password. Whichever IT policy you’d choose, you will always end up with users having a weak password or finding a way to compromise your system. We don’t want that to happen, nor do we think it’s worth asking a user to generate a random ‘strong-enough’ password by himself, so Kowabunga does it for you.

Once users have been registered and password generated, and provided Koala Web application has been deployed as well, they can connect to (and land on a perfectly empty and so useless dashboard ;-) for now at least ).

Let’s move on and start creating our first region !

3 - Create Your First Region

Let’s setup a new region and its Kiwi and Kaktus instances

Orchestrator being ready, we can now boostrap our first region.

Let’s take the following assumptions for the rest of this tutorial:

The Kowabunga region is to be called eu-west.
The region will have a single zone named eu-west-a.
It’ll feature 2 Kiwi and 3 Kaktus instances.

Back on the TF configuration, let’s use the following:

Region and Zone

locals {
  eu-west = {
    desc = "Europe West"

    zones = {
      "eu-west-a" = {
        id = "A"
      }
    }
  }
}

resource "kowabunga_region" "eu-west" {
  name = "eu-west"
  desc = local.eu-west.desc
}

resource "kowabunga_zone" "eu-west" {
  for_each = local.eu-west.zones
  region   = kowabunga_region.eu-west.id
  name     = each.key
  desc     = "${local.eu-west.desc} - Zone ${each.value.id}"
}

And apply:

$ kobra tf apply

Nothing really complex here to be fair, we’re just using Kahuna’s API to register the region and its associated zone.

Kiwi Instances and Agents

Now, we’ll register the 2 Kiwi instances and 3 Kaktus ones. Please note that:

we’ll extend the TF locals definition for that.
Kiwi is to be associated to the global region.
while Kaktus is ti be associated to the region’s zone.

Let’s start by registering one Kiwi and 2 associated agents:

locals {
  eu-west = {

    agents = {
      "kiwi-eu-west-1" = {
        desc = "Kiwi EU-WEST-1 Agent"
        type = "Kiwi"
      }
      "kiwi-eu-west-2" = {
        desc = "Kiwi EU-WEST-2 Agent"
        type = "Kiwi"
      }
    }

    kiwi = {
      "kiwi-eu-west" = {
        desc   = "Kiwi EU-WEST",
        agents = ["kiwi-eu-west-1", "kiwi-eu-west-2"]
      }
    }
  }
}

resource "kowabunga_agent" "eu-west" {
  for_each = merge(local.eu-west.agents)
  name     = each.key
  desc     = "${local.eu-west.desc} - ${each.value.desc}"
  type     = each.value.type
}

resource "kowabunga_kiwi" "eu-west" {
  for_each = local.eu-west.kiwi
  region   = kowabunga_region.eu-west.id
  name     = each.key
  desc     = "${local.eu-west.desc} - ${each.value.desc}"
  agents   = [for agent in try(each.value.agents, []) : kowabunga_agent.eu-west[agent].id]
}

Warning

Note that, despite have 2 Kiwi instances, from Kowabunga perspective, we’re only registering one. This is because, the 2 instances are only used for high-availability and failover perspective. From service point of view, the region only has one single network gateway.

Despite that, each instance will have its own agent, to establish a WebSocket connection to Kahuna orchestrator.

Kaktus Instances and Agents

Let’s continue with the 3 Kaktus instances declaration and their associated agents. Note that, this time, instances are associated to the zone itself, not the region.

Information

Note that Kaktus instance creaion/update takes 4 specific parameters into account:

cpu_price and memory_price are purely arbitrary values that express how much actual money is worth your metal infrastructure. These are used to compute virtual cost calculation later, when you’ll be spwaning Kompute instances with vCPUs and vGB of RAM. Each server being different, it’s fully okay to have different values here for your fleet.
cpu_overcommit and memory_overcommit define the overcommit ratio you accept your physical hosts to address. As for price, not every server is born equal. Some have hyper-threading, other don’t. You may consider that a value of 3 or 4 is fine, other tend to be stricter and use 2 instead. The more you set the bar, the more virtual resources you’ll be able to create but the less actual physical resources they’ll be able to get.

locals {
  currency           = "EUR"
  cpu_overcommit     = 3
  memory_overcommit  = 2

  eu-west = {
    zones = {
      "eu-west-a" = {
        id = "A"

        agents = {
          "kaktus-eu-west-a-1" = {
            desc = "Kaktus EU-WEST A-1 Agent"
            type = "Kaktus"
          }
          "kaktus-eu-west-a-2" = {
            desc = "Kaktus EU-WEST A-2 Agent"
            type = "Kaktus"
          }
          "kaktus-eu-west-a-3" = {
            desc = "Kaktus EU-WEST A-3 Agent"
            type = "Kaktus"
          }
        }

        kaktuses = {
          "kaktus-eu-west-a-1" = {
            desc        = "Kaktus EU-WEST A-1",
            cpu_cost    = 500
            memory_cost = 200
            agents      = ["kaktus-eu-west-a-1"]
          }
          "kaktus-eu-west-a-2" = {
            desc        = "Kaktus EU-WEST A-2",
            cpu_cost    = 500
            memory_cost = 200
            agents      = ["kaktus-eu-west-a-2"]
          }
          "kaktus-eu-west-a-3" = {
            desc        = "Kaktus A-3",
            cpu_cost    = 500
            memory_cost = 200
            agents      = ["kaktus-eu-west-a-3"]
          }
        }
      }
    }
  }
}

resource "kowabunga_agent" "eu-west-a" {
  for_each = merge(local.eu-west.zones.eu-west-a.agents)
  name     = each.key
  desc     = "${local.eu-west.desc} - ${each.value.desc}"
  type     = each.value.type
}

resource "kowabunga_kaktus" "eu-west-a" {
  for_each          = local.eu-west.zones.eu-west-a.kaktuses
  zone              = kowabunga_zone.eu-west["eu-west-a"].id
  name              = each.key
  desc              = "${local.eu-west.desc} - ${each.value.desc}"
  cpu_price         = each.value.cpu_cost
  memory_price      = each.value.memory_cost
  currency          = local.currency
  cpu_overcommit    = try(each.value.cpu_overcommit, local.cpu_overcommit)
  memory_overcommit = try(each.value.memory_overcommit, local.memory_overcommit)
  agents            = [for agent in try(each.value.agents, []) : kowabunga_agent.eu-west-a[agent].id]
}

And again, apply:

$ kobra tf apply

That done, Kiwi and Kaktus instances have been registered, but more essentially, their associated agents. For each newly created agent, you should have received an email (check the admin one you previously set in Kahuna’s configuration). Keep track of these emails, they contain one-time credentials about the agent identifier and it’s associated API key.

This is the super secret thing that will allow them further to establish secure connection to Kahuna orchestrator. We’re soon going to declare these credentials in Ansible’s secrets so Kiwi and Kaktus instances can be provisioned accordingly.

Warning

There’s no way to recover the agent API key. It’s never printed anywhere but on the email you just received. Even the database doesn’t contain it. If one agent’s API key is lost, you can either request a new one from API or destroy the agent and create a new one in-place.

Virtual Networks

Let’s keep on provisioning Kahuna’s database with the network configuration from our network topology.

We’ll use different VLANs (expressed as VNET or Virtual NETwork in Kowabunga’s terminology) to segregate tenant traffic:

VLAN 0 (i.e. no VLAN) will be used for public subnets (i.e. where to hook public IP addresses).
VLAN 102 will be dedicated to storage backend.
VLANs 201 to 209 will be reserved for tenants/projects (automatically assigned at new project’s creation).

Note

Note that exposing Ceph storage network might come in handy if you intend to run applications which are expected to consume or provision resources directly on the underlying storage.

By default, if you only intend to use plain old Kompute instances, virtual disks are directly mapped by virtualization and you don’t have to care about how. In that case, there’s no specific need to expose VLAN 102.

If you however expect to further use KFS or running your own Kubernetes flavor, with an attempt to directly use Ceph backend to instantiate PVCs, exposing the VLAN 102 is mandatory.

To be on the safe side, and furure-proof, keep it exposed.

So let’s extend our terraform/main.tf with the following VNET resources declaration for the newly registered region.

resource "kowabunga_vnet" "eu-west" {
  for_each  = local.eu-west.vnets
  region    = kowabunga_region.eu-west.id
  name      = each.key
  desc      = try(each.value.desc, "EU-WEST VLAN ${each.value.vlan} Network")
  vlan      = each.value.vlan
  interface = each.value.interface
  private   = each.value.vlan == "0" ? false : true
}

This will iterate over a list of VNET objects that we’ll define in terraform/locals.tf file:

locals {
  eu-west = {
    vnets = {
      // public network
      "eu-west-0" = {
        desc      = "EU-WEST Public Network",
        vlan      = "0",
        interface = "br0",
      },

      // storage network
      "eu-west-102" = {
        desc      = "EU-WEST Ceph Storage Network",
        vlan      = "102",
        interface = "br102",
      },

      // services networks
      "eu-west-201" = {
        vlan      = "201",
        interface = "br201",
      },
      [...]
      "eu-west-209" = {
        vlan      = "209",
        interface = "br209",
      },
    }
  }
}

And again, apply:

$ kobra tf apply

What have we done here ? Simply iterating over VNETs to associate those with VLAN IDs and the name of Linux bridge interfaces which will be created on each Kaktus instance from the zone (see further).

Note

Note that while services instances will have dedicated reserved networks, we’ll (conventionnally) add the VLAN 0 here (which is not really a VLAN at all).

Kaktus instances will be created with a br0 bridge interface, mapped on host private network controller interface(s), where public IP addresses will be bound. This will allow further create virtual machines to be able to bind public IPs through the bridged interface.

Subnets

Now that virtual networks have been registered, it’s time to associate each of them with service subnets. Again, let’s edit our terraform/main.tf to declare resources objects, on which we’ll iterate.

resource "kowabunga_subnet" "eu-west" {
  for_each    = local.eu-west.subnets
  vnet        = kowabunga_vnet.eu-west[each.key].id
  name        = each.key
  desc        = try(each.value.desc, "")
  cidr        = each.value.cidr
  gateway     = each.value.gw
  dns         = try(each.value.dns, each.value.gw)
  reserved    = try(each.value.reserved, [])
  gw_pool     = try(each.value.gw_pool, [])
  routes      = kowabunga_vnet.eu-west[each.key].private ? local.extra_routes : []
  application = try(each.value.app, local.subnet_application)
  default     = try(each.value.default, false)
}

Subnet objects are associated with a given virtual network and usual network settings (such as CIDR, route/rgateway, DNS server) are associated.

Note the use of 2 interesting parameters:

reserved, which is basically a list of IP addresses ranges, which are part of the provided CIDR, but not not to be assigned to further created virtual machines and services. This may come in handy if you have specific use of static IP addresses in your project and want to ensure they’ll never get assigned to anyone programmatically.
gw_pool, which is a range of IP addresses that are to be assigned to each project’s Kawaii instances as virtual IPs. These are fixed IPs (so that router address never changes, even if you do destroy/recreate service instances countless times). You usually need one per zone, not more. But it’s safe to extend the range for future-use (e.g. adding new zones in your region).

Now let’s declare the various subnets in terraform/locals.tf file as well:

locals {
  subnet_application = "user"

  eu-west = {
      "eu-west-0" = {
        desc = "EU-WEST Public Network",
        vnet = "0",
        cidr = "4.5.6.0/26",
        gw   = "4.5.6.62",
        dns  = "9.9.9.9"
        reserved = [
          "4.5.6.0-4.5.6.0",   # network address
          "4.5.6.62-4.5.6.63", # reserved (gateway, broadcast)
        ]
      },

      "eu-west-102" = {
        desc = "EU-WEST Ceph Storage Network",
        vnet = "102",
        cidr = "10.50.102.0/24",
        gw   = "10.50.102.1",
        dns  = "9.9.9.9"
        reserved = [
          "10.50.102.0-10.50.102.69", # currently used by Iris(es) and Kaktus(es) (room for more)
        ]
        app = "ceph"
      },

      # /24 subnets
      "eu-west-201" = {
        vnet = "201",
        cidr = "10.50.201.0/24",
        gw   = "10.50.201.1",
        reserved = [
          "10.50.201.1-10.50.201.5",
        ]
        gw_pool = [
          "10.50.201.252-10.50.201.254",
        ]
      },
      [...]
      "eu-west-209" = {
        vnet = "209",
        cidr = "10.50.209.0/24",
        gw   = "10.50.209.1",
        reserved = [
          "10.50.209.1-10.50.209.5",
        ]
        gw_pool = [
          "10.50.209.252-10.50.209.254",
        ]
      },
    }
  }
}

Note

Note that we arbitrary took multiple decisions here:

Reserve the first 69 IP addresses of the 10.50.102.0/24 subnet for our region growth. Each project’s Kawaii instance (one per zone) will bind an IP from the range. That’s plain enough room for the 10 projects we intend to host. But this saves us some space, shall we need to extend our infrastructure, by adding new Kaktus instances.
Use of /24 subnets. This is really up to each network administrator. You could pick whichever range you need which wouldn’t collapse with what’s currently in place.
Limit virtual network to one single subnet. We could have added as much as needed.
Reserve the first 5 IPs of each subnet. Remember, our 2 Kiwi instances are already configured to bind .2 and .3 (and .1 is the VIP). We’ll save a few exra room for future use (one never knows …).
Reserve the subnet’s last 3 IP addresses for Kawaii gateways virtual IPs. We only have one zone for now, so 1 would have been anough, but again, we never know what the future holds …

Warning

It is now time to read carefully what you wrote, really do. We just wrote down a bloated list of network settings, subnets, CIDRs, IP addresses and mistake have probably happened. While nothing’s written in stone (you can always apply TF config again), better find it now, that wasting hours trying to figure out later why your virtual machine doesn’t get network access ;-)

Once carefully reviewed, again, apply:

$ kobra tf apply

Let’s continue and provision our region’s Kiwi instances !

4 - Provisioning Kiwi

Let’s provision our Kiwi instances

As detailed in network topology, we’ll have 2 Kiwi instances:

kiwi-eu-west-1:
- with VLAN 101 as administrative segment with 10.50.101.2,
- with VLAN 102 as storage segment with 10.50.102.2,
- with VLAN 201 to 209 as service VLANs.
kiwi-eu-west-1:
- with VLAN 101 as administrative segment with 10.50.101.3,
- with VLAN 102 as storage segment with 10.50.102.3,
- with VLAN 201 to 209 as service VLANs.

Note that 10.50.101.1 and 10.50.102.1 will be used as virtual IPs (VIPs).

Inventory Management

It is now time to declare your Kiwi instances in Ansible’s inventory. Simply extend the ansible/inventories/hosts.txt the following way:

[kiwi]
10.50.101.2 name=kiwi-eu-west-1 ansible_ssh_user=ubuntu
10.50.101.3 name=kiwi-eu-west-2 ansible_ssh_user=ubuntu

Important

Note that for the first-time installation, private IPs from the inventory are to replaced by the servers private ones (or anything in place which allows for bootstrapping machines).

The instances are now declared to be part of kiwi group and Ansible will use ubuntu local user account to connect through SSH.

Note that doing so, you can now safely:

declare host-specific variables in ansible/host_vars/10.50.101.{2,3}.yml files.
declare host-specific sensitive variables in ansible/host_vars/10.50.101.{2,3}.sops.yml file.
declare kiwi group-specific variables in ansible/group_vars/kiwi/main.yml file.
declare kiwi group-specific sensitive variables in ansible/group_vars/kiwi.sops.yml file.
declare any other global variables in ansible/group_vars/all/main.yml file.
declare any other global sensitive variables in ansible/group_vars/all.sops.yml file.

Note that Ansible variables precedence will apply:

role defaults < all vars < group vars < host vars < role vars

Network Configuration

We’ll instruct the Ansible collection to provision network settings through Netplan. Note that our example is pretty simple, with only a single network interface to be used for private LAN, no link aggregation being used (recommended for enterprise-grade setups).

Let’s declare the following configuration in ansible/inventories/host_vars/10.50.101.2.yml file:

kowabunga_netplan_config:
  ethernet:
    - name: "{{ wan_dev }}"
      mac: "aa:bb:cc:dd:ee:ff"
      ips:
        - a.b.c.d/24
      routes:
        - to: default
          via: e.c.d.f
    - name: "{{ lan_dev }}"
      mac: "aa:bb:cc:dd:ee:ff"
  vlan:
    # EU-WEST admin network
    - name: vlan101
      id: 101
      link: "{{ lan_dev }}"
      ips:
        - 10.50.101.2/24
    # EU-WEST storage network
    - name: vlan102
      id: 102
      link: "{{ lan_dev }}"
      ips:
        - 10.50.102.2/24
    # EU-WEST services networks
    - name: vlan201
      id: 201
      link: "{{ lan_dev }}"
      ips:
        - 10.50.201.2/24
    [...]
    - name: vlan209
      id: 209
      link: "{{ lan_dev }}"
      ips:
        - 10.50.209.2/24

You’ll need to ensure that the MAC addresses and host and gateway IP addresses are correctly set, depending on your setup. Once done, you can do the same for the alternate Kiwi instance in ansible/inventories/host_vars/10.50.101.2.yml file file.

Extend the ansible/inventories/group_vars/kiwi/main.yml file with the following to ensure generic settings are propagated to all Kiwi instances:

kowabunga_netplan_disable_cloud_init: true
kowabunga_netplan_apply_enabled: true

Information

Note that setting kowabunga_netplan_disable_cloud_init is an optional step. If you’d like to keep whatever configuration cloud-init has previously set, it’s all fine (but it’s always recommended not to have dual sourc eof truth).

Network Failover

Each Kiwi instance configuration is now set to receive host-specific network configuration. But they are meant to work in an HA-cluster, so let’s define some redundancy rules. The two instances respectively bind the .2 and .3 private IPs from each subnet, but our active router will be .1, so let’s define network failover configuration for that.

Again, extend the ansible/inventories/group_vars/kiwi/main.yml file with the following configuration:

kowabunga_kiwi_primary_host: "kiwi-eu-west-1"
kowabunga_network_failover_settings:
  peers: "{{ groups['kiwi'] }}"
  use_unicast: true
  trackers:
    - name: kiwi-eu-west-vip
      configs:
        - vip: 10.50.101.1/24
          vrid: 101
          primary: "{{ kowabunga_kiwi_primary_host }}"
          control_interface: vlan101
          interface: vlan101
          nopreempt: true
        - vip: 10.50.102.1/24
          vrid: 102
          primary: "{{ kowabunga_kiwi_primary_host }}"
          control_interface: vlan102
          interface: vlan102
          nopreempt: true
        - vip: 10.50.201.1/24
          vrid: 201
          primary: "{{ kowabunga_kiwi_primary_host }}"
          control_interface: vlan201
          interface: vlan201
          nopreempt: true
        [...]
        - vip: 10.50.209.1/24
          vrid: 209
          primary: "{{ kowabunga_kiwi_primary_host }}"
          control_interface: vlan209
          interface: vlan209
          nopreempt: true

This will ensure that VRRP packets flows between the 2 peers so one always ends up being the active router for each virtual network interface.