Cloud-init

Figure 1100. Introduction and reference Slide presentation

Figure 1101. In a nutshell Slide presentation
  • Initial system configuration

  • Standardized installation related tasks:

    packages, users, ssh keys, networks, hostnames, custom scripts


Figure 1102. Post server creation steps Slide presentation
  • Individual CRUD file operations

  • Adding users

  • Supplying ssh user and host keys.

  • System hardening: Firewall rules, security policies, logging / monitoring

  • Installing packages

  • Enabling and starting services

  • System upgrade + reboot

  • Arbitrary command execution


Figure 1103. DIY using Bash by Terraform's user_data attribute Slide presentation
# main.tf

resource "hcloud_server" "nginxServer" {           
  name         = "hello"
  image        = "debian-13"
  server_type  = "cx23"
  ssh_keys     = ...

  # Post server creation init script
  user_data    = file("myinit.sh")
}
#!/bin/bash

# myinit.sh

aptitude update && aptitude -y upgrade     
├── main.tf
├── ... 
└── myinit.sh

exercise No. 18

Automatic Nginx installation

Q:

Follow Figure 1103, “DIY using Bash by Terraform's user_data attribute ” and create a bash script which:

  • Installs the Nginx web server by its nginx package.

    Follow How to Install Nginx on Debian 13 installing and configuring the ufw firewall alongside as well. You'll thus do not need a Hetzner firewall. Do not forget to enable ssh access.

  • Uses aptitude for updating the package list and updating all installed packages.

Tip

  • Automated installation clashes with interactive user input: Use the aptitude install --assume-yes option allowing for automated package installation and upgrades.

  • There is a similar option when enabling your ufw firewall.

  • Execute your initialization script beforehand line by line on an existing machine interactively. You'll find errors more quickly thereby avoiding time consuming terraform apply cycles

Figure 1104. Basic Cloud-init facts Slide presentation
  • Server post-creation install steps

  • YAML based syntax

#cloud-config
package_reboot_if_required: true
package_update: true
package_upgrade: true

packages:
  - nginx

runcmd:
  - systemctl enable nginx
  - systemctl start nginx
  - rm -rf /var/www/html/*
  - >
    echo "I'm Nginx @ $(curl -4 https://ifconfig.me) 
    created $(date -u)" > /var/www/html/index.html

Figure 1105. cloud-init file validation Slide presentation

exercise No. 19

Cloud-init for starters

Q:

Follow Hetzner's Cloud-init introduction and assemble a corresponding Cloud-init script. Use the package_reboot_if_required: true directive in favor of an explicit reboot command.

Use the Hetzner WEB Gui for manual server creation. Copy-paste your Cloud-init script into the respective text field.

Tip

Do a syntax validation of your Cloud-init script beforehand. For this purpose install cloud-init on your local machine. Beware: Executing cloud-init requires root privileges, sudo may be your friend here.

Without prior validation you'll likely end up in tedious debugging struggles.

Figure 1106. Terraform triggering Cloud-init Slide presentation
# main.tf
resource "hcloud_server" "web" {
  name         = var.server_name
       ...
  user_data = file("userData.yml")
}
#cloud-config

# userData.yml

packages:
  - nginx
runcmd:
  - systemctl enable nginx
  - rm /var/www/html/*
  - >
    echo "I'm Nginx @ $(curl -4 https://ifconfig.me) 
    created $(date -u)" > /var/www/html/index.html
├── main.tf
├── ... 
└── userData.yml

exercise No. 20

Cloud-init by Terraform

Q:

Re-use your Cloud-init script from Cloud-init for starters . This time follow Figure 1106, “Terraform triggering Cloud-init ” using Terraform for executing Cloud-init rather than Hetzner's WEB GUI.

Figure 1107. Using template file interpolation Slide presentation
# main.tf
resource "local_file" "user_data" {
  content         = templatefile("tpl/userData.yml", {
  loginUser              = "devops"
                    })
  filename        = "gen/userData.yml"
}

resource "hcloud_server" "mySrv" {
  ...
  # Using gen/userData.yml
  user_data    = local_file.user_data.content
}
# tpl/userData.yml
...
users:
  - name: ${loginUser}
    ...
# gen/userData.yml
...
users:
  - name: devops
     ...

Figure 1108. Yaml syntax quirk 1: Missing indentation Slide presentation
resource "tls_private_key" "host" {
  algorithm   = "ED25519"
}
...
resource "local_file" "user_data" {
  content  = templatefile("tpl/userData.yml", {
     host_ed25519_private =
       tls_private_key.host.private_key_openssh
     host_ed25519_public  = ...
                })
  filename = "gen/user_data.yml"
}
...
ssh_keys:
  ed25519_private: |
    -----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAABG5vbmUA ...
   ...
-----END OPENSSH PRIVATE KEY-----
# E1: File gen/user_data.yml is not valid YAML.
  in "<unicode string>", line 19, column 1:
    b3BlbnNzaC1rZXktdjEAAAAABG5vbmUA ...

Figure 1109. Resolving missing Yaml indentation Slide presentation
# main.tf
resource "local_file" "user_data" {
  content  = templatefile("tpl/userData.yml", {
     host_ed25519_private = 
       indent(4, tls....host.private_key_openssh)
     ...})
  filename = "gen/user_data.yml"
}
# gen/usr_data.yml
ssh_keys:
  ed25519_private: |
␣␣␣␣-----BEGIN OPENSSH PRIVATE KEY-----
␣␣␣␣b3BlbnNzaC1 ...
␣␣␣␣...
␣␣␣␣...
␣␣␣␣-----END OPENSSH PRIVATE KEY-----

Figure 1110. Yaml syntax quirk 2: Wrong indentation Slide presentation
# gen/usr_data.yml

ssh_keys:
  ed25519_private: |
␣␣␣␣-----BEGIN OPENSSH PRIVATE KEY-----
␣␣␣␣␣␣b3BlbnNzaC1 ...
␣␣␣␣␣␣...
␣␣␣␣␣␣...
␣␣␣␣␣␣-----END OPENSSH PRIVATE KEY-----
# /etc/ssh/ssh_host_ed25519_key
...
ssh_keys:
  ed25519_private: |
-----BEGIN OPENSSH PRIVATE KEY-----
␣␣b3BlbnNzaC1 ...
␣␣...
␣␣...
␣␣-----END OPENSSH PRIVATE KEY-----

Figure 1111. Cloud-init configuration file validation Slide presentation
resource "local_file" "user_data" {
  content  = templatefile("tpl/userData.yml", {})
  filename = "gen/user_data.yml"
}
#> cloud-init schema --config-file  gen/user_data.yml --annotate

Figure 1112. Watch out for your enemies! Slide presentation
root@hello:~# journalctl -f
May 06 04:41:20 hello cloud-init[898]: Cloud-init v. 22.4.2 finished at Mon, 06 May 2024 04:41:20 +0000. Datasource DataSourceHetzner.  Up 11.78 seconds
   ...
May 06 04:46:16 hello sshd[927]: Invalid user abc from 43.163.218.130 port 33408
May 06 04:46:17 hello sshd[927]: Received disconnect from 43.163.218.130 port 33408:11: Bye Bye [preauth]
May 06 04:46:17 hello sshd[927]: Disconnected from invalid user abc 43.163.218.130 port 33408 [preauth]
   ...
May 06 04:50:54 hello sshd[930]: fatal: Timeout before authentication for 27.128.243.225 port 59866
   ...
May 06 04:52:45 hello sshd[933]: Invalid user cos from 43.163.218.130 port 59776
   ...
May 06 04:53:04 hello sshd[935]: Invalid user admin from 194.169.175.35 port 51128
May 06 04:53:49 hello sshd[937]: User root from 43.163.218.130 not allowed because not listed in AllowUsers
May 06 04:53:49 hello sshd[937]: Disconnected from invalid user root 43.163.218.130 port 50592 [preauth]

exercise No. 21

Enhancing index.html generation

Q:

We extend our exercise Cloud-init by Terraform :

  • We recently cheated by creating an invalid HTML file:

    echo "I'm Nginx @ $(curl -4 https://ifconfig.me) created $(date -u)" > /var/www/html/index.html

    This time start from a proper tpl/index.html template file:

    <!DOCTYPE html>
    <html>
      <head>
        <title>Template demo</title>
      </head>
      <body>                                     <!-- To become e.g. Sunday 08 Nov 2026 at 18:08 --->
        <h1>Terraform template file demo created ${creationDate}.</h1> 
        <p>This is server <b>${serverName}</b> having IP addresses:</p>
         <ul>
           <li>IPV4: <a href="http://${ipv4Address}">${ipv4Address}</a></li>          <!-- To become e.g. 2a01:4f8:c016:9b2::/64 -->
           <li>IPV6: <a href="http://[${ipv6Address}]">${ipv6Address}</a> from subnet ${ipv6Subnet}</li>
         </ul>
      </body>
    </html>
  • Use Terraform's templatefile(...) function to replace all five variables by their respective values using Terraform for generating a gen/index.html file.

    Tip

    Due to resource dependencies (Which ones actually? Document them!) your tasks requires allocating IPV4 and IPV6 addresses before creating your server using resource "hcloud_primary_ip".

  • Use Cloud-init to transfer the generated index.html file to its respective location on your destination server. Nginx should then be happily picking it up.

    Tip

    Use a Json definition for your server's user_data attribute avoiding indentation hassles as in Figure 1110, “Yaml syntax quirk 2: Wrong indentation ”:

    #cloud-config
    
    ${yamlencode(
    {
      write_files = [
        {
          content     = "${indexHtmlContent}"
          path        = "/var/www/html/index.html"
          permissions = "0644"
          owner       = "root:root"
        }
      ]
    
      packages = ["nginx"]
    
      runcmd   = ["systemctl enable nginx", "systemctl start nginx"]
    
    })}
Figure 1113. Configuration change by replacement Slide presentation

Figure 1114. Possible quirks Slide presentation
  • Conflicting edits by multiple packages

  • De-installation of software-z.deb

  • Updates of service-xy.deb


Figure 1115. Configuration change by replacement Slide presentation

exercise No. 22

Hardening your server

Q:

Extending the previous exercise we address login and firewall related issues.

  1. Define a hcloud_firewall in favour of a server local ufw allowing only ssh access.

    You should not be able to access your server using http://<serverIp>.

  2. Add port 80 to the list of allowed incoming connections. Your server should become accessible.

  3. With respect to Figure 1112, “Watch out for your enemies! ” inspect the output of journalctl -f on your server for a while. Can you spot suspicious login attempts?

    Read the sshd_config(5) - Linux man page. Then provide a configuration file /etc/ssh/sshd_config.d/99Local.conf by using Cloud-init again. Use this file for overriding suitable ssh server defaults:

    • Create a user devops eligible for both ssh login and sudo execution.

      Then restrict ssh logins to the devops user account thereby excluding all other users.

    • Disallow ssh password authentication altogether

    On success the following sequence should be possible:

    $ ssh -v devops@95.217.154.104
    ...
    debug1: SSH2_MSG_SERVICE_ACCEPT received
    debug1: Authentications that can continue: publickey ❶
    debug1: Next authentication method: publickey
    ...
    Last login: Sun May  5 19:21:12 2024 from 217.245.243.187
    
    devops@hello:~$ sudo su - ❷
    root@hello:~# hostname
    hello

    ❶

    password is not among the list of allowed authentication methods.

    ❷

    User devops may execute sudo commands by virtue of his membership in the sudo group .

    On contrary without owning an authorized private key ssh root or devops logins must be prohibited:

    $ ssh root@95.217.154.104
    root@95.217.154.104: Permission denied (publickey).
    
    $ ssh devops@95.217.154.104
    devops@95.217.154.104: Permission denied (publickey).
  4. Read the cloud-init documentation and/or related tutorials to:

    • Currently your (most likely outdated) cloud provider supplied distribution does not get upgraded on installation time:

      $ ./bin/ssh 
      ...
      devops@hello:~$ sudo su -
      root@hello:~# apt update
      Hit:1 http://security.debian.org/debian-security bookworm-security InRelease
      Hit:2 http://deb.debian.org/debian bookworm InRelease                                              
      ...
      Reading package lists... Done
      Building dependency tree... Done
      Reading state information... Done
      6 packages can be upgraded. Run 'apt list --upgradable' to see them.
      
      # apt list --upgradable
      Listing... Done
      less/stable-security,stable-security 590-2.1~deb12u2 amd64 [upgradable from: 590-2]
      libc-bin/stable-security,stable-security 2.36-9+deb12u7 amd64 [upgradable from: 2.36-9+deb12u6]
      libc-l10n/stable-security,stable-security 2.36-9+deb12u7 all [upgradable from: 2.36-9+deb12u6]
      libc6/stable-security,stable-security 2.36-9+deb12u7 amd64 [upgradable from: 2.36-9+deb12u6]
      locales-all/stable-security,stable-security 2.36-9+deb12u7 amd64 [upgradable from: 2.36-9+deb12u6]
      locales/stable-security,stable-security 2.36-9+deb12u7 all [upgradable from: 2.36-9+deb12u6]

      Modify your Cloud-init configuration to upgrade your distribution at server creation time. If so required your system should also reboot.

    • Install and configure fail2ban limiting ssh failed connection attempts.

      Tip

    • Install the plocate file indexer package and initialize it.

    On success all packages should be up to date:

    $ ./bin/ssh 
    ...
    devops@hello:~$ sudo su -
    root@hello:~# apt update
    Hit:1 http://security.debian.org/debian-security bookworm-security InRelease
    Hit:2 http://deb.debian.org/debian bookworm InRelease                                              
    ...
    Reading package lists... Done
    Building dependency tree... Done
    Reading state information... Done
    All packages are up to date.

    Failed login attempts should be banned: Keep a second login open in advance when trying to simulate login failures! You should then see a report similar to:

    root@hello:~# fail2ban-client status sshd
    Status for the jail: sshd
    |- Filter
    |  |- Currently failed:	2
    |  |- Total failed:	14
    |  `- Journal matches:	_SYSTEMD_UNIT=sshd.service + _COMM=sshd
    `- Actions
       |- Currently banned:	2
       |- Total banned:	2
       `- Banned IP list:	170.64.133.30 213.136.94.219

    Searching for file name components should work like e.g.:

    root@hello:~# locate ssh_host
    /etc/ssh/ssh_host_ed25519_key
    /etc/ssh/ssh_host_ed25519_key.pub
Figure 1116. Problem: Duplicate known_hosts entry on re-creating server Slide presentation

Problem of repeated terraform apply:

$ ssh root@128.140.108.60
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!

Figure 1117. Solution: Generating known_hosts ... Slide presentation
resource "tls_private_key" "host" { algorithm   = "ED25519"}
   ...
resource "hcloud_server" "mySrv" {...}
   ...
resource "local_file" "known_hosts" {
  content = join(" "
            ,[ hcloud_server.mySrv.ipv4_address
            , tls_private_key.host.public_key_openssh]
            )
  filename        = "gen/known_hosts"
  file_permission = "644"
}
157.180.68.60 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5...XXgAC3eaueHMQ8ePtbnw

Figure 1118. Transferring key pair to server, #1 / 2 Slide presentation
resource "tls_private_key" "host" { algorithm = "ED25519"}
  ...
resource "local_file" "user_data" {
  content  = templatefile("tpl/userData.yml", {
     host_ed25519_private = tls_private_key.host.private_key_openssh
     host_ed25519_public  = tls_private_key.host.public_key_openssh
     devopsUsername       = hcloud_ssh_key.loginUser.name
               ...
  filename = "gen/user_data.yml" ...
}

resource "hcloud_server" "helloServer" {
    ...
   user_data = local_file.user_data.content
}

Figure 1119. Transferring key pair to server, #2 / 2 Slide presentation
#cloud-config

${yamlencode(
  { ...
    ssh_keys = {
      ed25519_private = host_ed25519_private
      ed25519_public  = host_ed25519_public
    } ...
  }
)}
users:
  - name: ${devopsUsername} ...

Figure 1120. Creating a ssh wrapper Slide presentation
# main.tf
resource "local_file" "ssh_script" {
  content = templatefile("tpl/ssh.sh", {
    ip    = hcloud_server.hello.ipv4_address    
  })
  filename        = "bin/ssh"
  file_permission = "755"
  depends_on      = [local_file.known_hosts]
}
#!/bin/bash

# File tpl/ssh.sh

GEN_DIR=$(dirname "$0")/../gen

ssh -o UserKnownHostsFile= \
  "$GEN_DIR/known_hosts" devops@${ip} "$@"    

exercise No. 23

Solving the ~/.ssh/known_hosts quirk

Q:

Extend Hardening your server generating both a bin/ssh wrapper and a corresponding gen/known_hosts file to be used by the wrapper script.

Avoiding ssh host key issues on login we require the creation of a known_hosts file among with each server being created:

157.180.78.16 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJDd+x7b80BM97rTU4RCM/CP+K7u4QAqx8ulDdXm9JDv

We assume this file being generated at gen/known_hosts. A corresponding ssh bash script shall be generated at the same level in bin/ssh:

#!/bin/bash

GEN_DIR=$(dirname "$0")/../gen

ssh -o UserKnownHostsFile="$GEN_DIR/known_hosts" devops@157.180.78.16 "$@"

# end of script

This way invoking ./bin/ssh will instruct the ssh client to read gen/known_hosts and thus avoiding unknown host key issues. The "$@" allows for optional user provided parameters.

For generating this script we start from a corresponding tpl/ssh.sh template file:

#!/usr/bin/env bash

GEN_DIR=$(dirname "$0")/../gen

ssh -o UserKnownHostsFile="$GEN_DIR/known_hosts" ${devopsUsername}@${ip} "$@"

# end of script

We then let Terraform use this template replacing ${devopsUsername} and ${ip} by your deployment's actual values.

As shown in Figure 1129, “Parent module vs. sub module context ” point to your child module's local tpl path by using ${path.module}:

resource "local_file" "ssh_script" {
  content = templatefile("${path.module}/tpl/ssh.sh", {
    ...
}

Likewise create a bin/scp script to be used for file transfer purposes as well:

#!/usr/bin/env bash

GEN_DIR=$(dirname "$0")/../gen

if [ $# -lt 2 ]; then
   echo usage: .../bin/scp ... devops@157.180.78.16 ...
else
   scp -o UserKnownHostsFile="$GEN_DIR/known_hosts" $@
fi

# end of script

Since copying objects requires at least a source and a destination parameter this script checks for the presence of at least two extra user supplied values by means of if [ $# -lt 2 ].

Figure 1121. Failsafe console login Slide presentation
#cloud-config

chpasswd:
  list: |
    root:bingo9wingo
  expire: False

Figure 1122. Avoiding Yaml parsing issues Slide presentation
#cloud-config

write_files:
  - content:
        ${base64encode(private_key_pem)}
    encoding: base64
    path:  /etc/nginx/snippets/cert/private.pem

Note

Congrats to paultyng