Cloud-init

Figure 1089. Introduction and reference Slide presentation

Figure 1090. In a nutshell Slide presentation
  • Distribution image containing pre-installed Cloud Init

  • Script configurable installation options


Figure 1091. Configuration options Slide presentation
  • Individual CRUD file operations

  • Supplying ssh user and host keys.

  • Adding users

  • ...

  • Installing packages

  • System Upgrade + reboot

  • Arbitrary command execution


Figure 1092. Bash DIY Slide presentation
main.tf myinit.sh
resource "hcloud_server" "nginxServer" { 
  name         = "hello"
  image        = "debian-13"
  server_type  = "cx23"
  user_data    = file("myinit.sh")
}
#!/bin/sh

aptitude update && aptitude -y upgrade

exercise No. 14

Automatic Nginx installation

Q:

Use a Terraform user_data included bash script to:

  1. Install the Nginx web server automatically.

  2. Start Nginx using the systemctl command.

  3. Enable the Nginx server permanently (surviving re-boot) using systemctl again.

Tip

You may test your script beforehand on an existing server. This may save unnecessary Terraform apply / destroy cycles

Figure 1093. Terraform interface to Cloud Init Slide presentation
resource "hcloud_server" "web" {
  name         = var.server_name
       ...
  user_data = file("userData.yml")
}
#cloud-config
packages:
  - nginx
runcmd:
  - systemctl enable nginx
  - rm /var/www/html/*
  - >
    echo "I'm Nginx @ $(dig -4 TXT +short o-o.myaddr.l.google.com @ns1.google.com) 
    created $(date -u)" >> /var/www/html/index.html

Figure 1094. Using template files Slide presentation
# main.tf
resource "local_file" "user_data" {
  content         = templatefile("tpl/userData.yml", {
  loginUser              = "devops"
                    })
  filename        = "gen/userData.yml"
}

resource "hcloud_server" "helloServer" {
  ...
  # Will become gen/userData.yml
  user_data    = local_file.user_data.content
}
# tpl/userData.yml
...
users:
  - name: ${loginUser}
    ...
# gen/userData.yml
...
users:
  - name: devops
     ...

Figure 1095. cloud-init file validation Slide presentation
  • # cloud-init schema --system --annotate

    ...
    apt:		# E1
    -   debconf_selections: openssh-server  openssh-server/password-authentication  boolean
            false openssh-server  openssh-server/permit-root-login        boolean false
    ...
    # Errors: -------------
    # E1: [{'debconf_selections': 'openssh-server  ....boolean false'}] is not of type 'object'
  • # cloud-init schema --config-file /var/lib/cloud/instance/user-data.txt

    Valid cloud-config: /var/lib/cloud/instance/user-data.txt

Figure 1096. Yaml syntax quirk 1: Missing indentation Slide presentation
resource "tls_private_key" "host" {
  algorithm   = "ED25519"
}
...
resource "local_file" "user_data" {
  content  = templatefile("tpl/userData.yml", {
     host_ed25519_private =
       tls_private_key.host.private_key_openssh
     host_ed25519_public  = ...
                })
  filename = "gen/user_data.yml"
}
...
ssh_keys:
  ed25519_private: |
    -----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAABG5vbmUA...
-----END OPENSSH PRIVATE KEY-----
# E1: File gen/user_data.yml is not valid YAML.
  in "<unicode string>", line 19, column 1:
    b3BlbnNzaC1rZXktdjEAAAAABG5vbmUA ...

Figure 1097. Yaml missing indentation resolution Slide presentation
# main.tf
resource "local_file" "user_data" {
  content  = templatefile("tpl/userData.yml", {
     host_ed25519_private = 
       indent(4, tls....host.private_key_openssh)
     ...})
  filename = "gen/user_data.yml"
}
# gen/usr_data.yml
ssh_keys:
  ed25519_private: |
␣␣␣␣-----BEGIN OPENSSH PRIVATE KEY-----
␣␣␣␣b3BlbnNzaC1 ...
␣␣␣␣...
␣␣␣␣...
␣␣␣␣-----END OPENSSH PRIVATE KEY-----

Figure 1098. Yaml syntax quirk 2: Wrong indentation Slide presentation
# gen/usr_data.yml

ssh_keys:
  ed25519_private: |
␣␣␣␣-----BEGIN OPENSSH PRIVATE KEY-----
␣␣␣␣␣␣b3BlbnNzaC1 ...
␣␣␣␣␣␣...
␣␣␣␣␣␣...
␣␣␣␣␣␣-----END OPENSSH PRIVATE KEY-----
# /etc/ssh/ssh_host_ed25519_key
...
ssh_keys:
  ed25519_private: |
-----BEGIN OPENSSH PRIVATE KEY-----
␣␣b3BlbnNzaC1 ...
␣␣...
␣␣...
␣␣-----END OPENSSH PRIVATE KEY-----

Figure 1099. Cloud-init configuration file validation Slide presentation
resource "local_file" "user_data" {
  content  = templatefile("tpl/userData.yml", {})
  filename = "gen/user_data.yml"
}
#> cloud-init schema --config-file  gen/user_data.yml

Figure 1100. Watch out for your enemies! Slide presentation
root@hello:~# journalctl -f
May 06 04:41:20 hello cloud-init[898]: Cloud-init v. 22.4.2 finished at Mon, 06 May 2024 04:41:20 +0000. Datasource DataSourceHetzner.  Up 11.78 seconds
   ...
May 06 04:46:16 hello sshd[927]: Invalid user abc from 43.163.218.130 port 33408
May 06 04:46:17 hello sshd[927]: Received disconnect from 43.163.218.130 port 33408:11: Bye Bye [preauth]
May 06 04:46:17 hello sshd[927]: Disconnected from invalid user abc 43.163.218.130 port 33408 [preauth]
   ...
May 06 04:50:54 hello sshd[930]: fatal: Timeout before authentication for 27.128.243.225 port 59866
   ...
May 06 04:52:45 hello sshd[933]: Invalid user cos from 43.163.218.130 port 59776
   ...
May 06 04:53:04 hello sshd[935]: Invalid user admin from 194.169.175.35 port 51128
May 06 04:53:49 hello sshd[937]: User root from 43.163.218.130 not allowed because not listed in AllowUsers
May 06 04:53:49 hello sshd[937]: Disconnected from invalid user root 43.163.218.130 port 50592 [preauth]

exercise No. 15

Working on Cloud-init

Q:

We continue our exercise series Incrementally creating a base system by adding a Cloud-init configuration:

  1. Follow Figure 1093, “Terraform interface to Cloud Init ” and Figure 1093, “Terraform interface to Cloud Init ” to create simple web server.

    Tip

    You will have to extend your current firewall configuration allowing inbound traffic to port 80.

    On success pointing your web browser of choice to http://<your server's IP> should result in something similar to:

    I'm Nginx @ "95.217.154.104" created Sun May 5 06:58:37 PM UTC 2024
  2. With respect to Figure 1100, “Watch out for your enemies! ” inspect the output of journalctl -f on your own server for a while. Then modify your current sshd configuration:

    On success the following sequence should be possible:

    $ ssh -v devops@95.217.154.104
    ...
    debug1: SSH2_MSG_SERVICE_ACCEPT received
    debug1: Authentications that can continue: publickey ❶
    debug1: Next authentication method: publickey
    ...
    Last login: Sun May  5 19:21:12 2024 from 217.245.243.187
    devops@hello:~$ sudo su - ❷
    root@hello:~# hostname
    hello

    ❶

    password is not among the list of allowed authentication methods.

    ❷

    User devops may execute sudo commands by virtue of his membership in group sudo.

    On contrary ssh root login must be prohibited:

    $ ssh root@95.217.154.104
    root@95.217.154.104: Permission denied (publickey).
  3. Read the cloud-init documentation and/or related tutorials to:

    • Currently your (most likely outdated) cloud provider supplied distribution does not get upgraded on installation time:

      $ ./bin/ssh 
      ...
      devops@hello:~$ sudo su -
      root@hello:~# apt update
      Hit:1 http://security.debian.org/debian-security bookworm-security InRelease
      Hit:2 http://deb.debian.org/debian bookworm InRelease                                              
      ...
      Reading package lists... Done
      Building dependency tree... Done
      Reading state information... Done
      6 packages can be upgraded. Run 'apt list --upgradable' to see them.
      
      # apt list --upgradable
      Listing... Done
      less/stable-security,stable-security 590-2.1~deb12u2 amd64 [upgradable from: 590-2]
      libc-bin/stable-security,stable-security 2.36-9+deb12u7 amd64 [upgradable from: 2.36-9+deb12u6]
      libc-l10n/stable-security,stable-security 2.36-9+deb12u7 all [upgradable from: 2.36-9+deb12u6]
      libc6/stable-security,stable-security 2.36-9+deb12u7 amd64 [upgradable from: 2.36-9+deb12u6]
      locales-all/stable-security,stable-security 2.36-9+deb12u7 amd64 [upgradable from: 2.36-9+deb12u6]
      locales/stable-security,stable-security 2.36-9+deb12u7 all [upgradable from: 2.36-9+deb12u6]

      Modify your Cloud-init configuration to upgrade your distribution at server creation time. If so required your system should also reboot.

    • Install and configure fail2ban limiting ssh failed connection attempts.

      Tip

    • Install the plocate file indexer package and initialize it.

    On success all packages should be up to date:

    $ ./bin/ssh 
    ...
    devops@hello:~$ sudo su -
    root@hello:~# apt update
    Hit:1 http://security.debian.org/debian-security bookworm-security InRelease
    Hit:2 http://deb.debian.org/debian bookworm InRelease                                              
    ...
    Reading package lists... Done
    Building dependency tree... Done
    Reading state information... Done
    All packages are up to date.

    Failed login attempts should be banned: Keep a second login open in advance when trying to simulate login failures! You should then see a report similar to:

    root@hello:~# fail2ban-client status sshd
    Status for the jail: sshd
    |- Filter
    |  |- Currently failed:	2
    |  |- Total failed:	14
    |  `- Journal matches:	_SYSTEMD_UNIT=sshd.service + _COMM=sshd
    `- Actions
       |- Currently banned:	2
       |- Total banned:	2
       `- Banned IP list:	170.64.133.30 213.136.94.219

    Searching for file name components should work like e.g.:

    root@hello:~# locate ssh_host
    /etc/ssh/ssh_host_ed25519_key
    /etc/ssh/ssh_host_ed25519_key.pub
Figure 1101. Problem: Duplicate known_hosts entry on re-creating server Slide presentation

Problem of repeated terraform apply:

$ ssh root@128.140.108.60
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!

Figure 1102. Solution: Generating known_hosts ... Slide presentation
resource "local_file" "known_hosts" {
  content         = join(" "
                        ,[ hcloud_server.helloServer.ipv4_address
                        , tls_private_key.host.public_key_openssh ]
                    )
  filename        = "gen/known_hosts"
  file_permission = "644"
}
157.180.68.60 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5...XXgAC3eaueHMQ8ePtbnw

Figure 1103. ... cloud-init config using yamlencode / JSON Slide presentation
resource "local_file" "user_data" {
  content  = templatefile("tpl/userData.yml", {
     host_ed25519_private = tls_private_key.host.private_key_openssh
     host_ed25519_public  = tls_private_key.host.public_key_openssh
     devopsUsername       = hcloud_ssh_key.loginUser.name
               ...
  filename = "gen/user_data.yml" ...
#cloud-config
${yamlencode(
  { ...
    ssh_keys = {
      ed25519_private = host_ed25519_private
      ed25519_public  = host_ed25519_public
    } ...
  }
)}
users:
  - name: ${devopsUsername} ...

Figure 1104. ... and ssh wrapper Slide presentation
main.tf tpl/ssh.sh
resource "local_file" "ssh_script" {
  content = templatefile("tpl/ssh.sh", {
    ip=hcloud_server.hello.ipv4_address
  })
  filename        = "bin/ssh"
  file_permission = "755"
  depends_on      = [local_file.known_hosts]
}
#!/usr/bin/env bash

GEN_DIR=$(dirname "$0")/../gen

ssh -o UserKnownHostsFile= \
  "$GEN_DIR/known_hosts" devops@${ip} "$@"

exercise No. 16

Solving the ~/.ssh/known_hosts quirk

Q:

Extend Working on Cloud-init generating both a bin/ssh wrapper, a bin/scp wrapper and a corresponding gen/known_hosts file to be used by the two former scripts.

Avoiding ssh host key issues on login we require the creation of a known_hosts file among with each server being created:

157.180.78.16 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJDd+x7b80BM97rTU4RCM/CP+K7u4QAqx8ulDdXm9JDv

We assume this file being generated at gen/known_hosts. A corresponding ssh bash script shall be generated at the same level in bin/ssh:

#!/usr/bin/env bash

GEN_DIR=$(dirname "$0")/../gen

ssh -o UserKnownHostsFile="$GEN_DIR/known_hosts" devops@157.180.78.16 "$@"

# end of script

This way invoking ./bin/ssh will instruct the ssh client to read gen/known_hosts and thus avoiding unknown host key issues. The "$@" allows for optional user provided parameters.

For generating this script we start from a corresponding tpl/ssh.sh template file:

#!/usr/bin/env bash

GEN_DIR=$(dirname "$0")/../gen

ssh -o UserKnownHostsFile="$GEN_DIR/known_hosts" ${devopsUsername}@${ip} "$@"

# end of script

We then let Terraform use this template replacing ${devopsUsername} and ${ip} by your deployment's actual values.

As shown in Figure 1113, “Parent module vs. sub module context ” point to your child module's local tpl path by using ${path.module}:

resource "local_file" "ssh_script" {
  content = templatefile("${path.module}/tpl/ssh.sh", {
    ...
}

Likewise create a bin/scp script to be used for file transfer purposes as well:

#!/usr/bin/env bash

GEN_DIR=$(dirname "$0")/../gen

if [ $# -lt 2 ]; then
   echo usage: .../bin/scp ... devops@157.180.78.16 ...
else
   scp -o UserKnownHostsFile="$GEN_DIR/known_hosts" $@
fi

# end of script

Since copying objects requires at least a source and a destination parameter this script checks for the presence of at least two extra user supplied values by means of if [ $# -lt 2 ].

Figure 1105. Failsafe console login Slide presentation
#cloud-config

chpasswd:
  list: |
    root:bingo9wingo
  expire: False

Figure 1106. Avoiding Yaml parsing issues Slide presentation
#cloud-config

write_files:
  - content:
        ${base64encode(private_key_pem)}
    encoding: base64
    path:  /etc/nginx/snippets/cert/private.pem

Note

Congrats to paultyng