Cloud-init

Figure 1020. Introduction and reference Slide presentation

Figure 1021. In a nutshell Slide presentation
  • Distribution image containing pre-installed Cloud Init

  • Script configurable installation options


Figure 1022. Configuration options Slide presentation
  • Individual CRUD file operations

  • Supplying ssh user and host keys.

  • Adding users

  • ...

  • Installing packages

  • System Upgrade + reboot

  • Arbitrary command execution


Figure 1023. Bash DIY Slide presentation
main.tf myinit.sh
resource "hcloud_server" "nginxServer" { 
  name         = "hello"
  image        = "debian-12"
  server_type  = "cx22"
  user_data    = file("myinit.sh")
}
#!/bin/sh

aptitude update && aptitude -y upgrade

exercise No. 12

Automatic Nginx installation

Q:

Use a Terraform user_data included bash script to:

  1. Install the Nginx web server automatically.

  2. Start Nginx using the systemctl command.

  3. Enable the Nginx server permanently (surviving re-boot) using systemctl again.

Tip

You may test your script beforehand on an existing server. This may save unnecessary Terraform apply / destroy cycles

Figure 1024. Terraform interface to Cloud Init Slide presentation
resource "hcloud_server" "web" {
  name         = var.server_name
       ...
  user_data = file("userData.yml")
}
#cloud-config
packages:
  - nginx
runcmd:
  - systemctl enable nginx
  - rm /var/www/html/*
  - >
    echo "I'm Nginx @ $(dig -4 TXT +short o-o.myaddr.l.google.com @ns1.google.com) 
    created $(date -u)" >> /var/www/html/index.html

Figure 1025. Using template files Slide presentation
# main.tf
resource "local_file" "user_data" {
  content         = templatefile("tpl/userData.yml", {
  loginUser              = "devops"
                    })
  filename        = "gen/userData.yml"
}

resource "hcloud_server" "helloServer" {
  ...
  # Will become gen/userData.yml
  user_data    = local_file.user_data.content
}
# tpl/userData.yml
...
users:
  - name: ${loginUser}
    ...
# tpl/userData.yml
...
users:
  - name: devops
     ...

Figure 1026. cloud-init file validation Slide presentation
  • # cloud-init schema --system --annotate

    ...
    apt:		# E1
    -   debconf_selections: openssh-server  openssh-server/password-authentication  boolean
            false openssh-server  openssh-server/permit-root-login        boolean false
    ...
    # Errors: -------------
    # E1: [{'debconf_selections': 'openssh-server  ....boolean false'}] is not of type 'object'
  • # cloud-init schema --config-file /var/lib/cloud/instance/user-data.txt

    Valid cloud-config: /var/lib/cloud/instance/user-data.txt

Figure 1027. Yaml syntax quirk 1: Missing indentation Slide presentation
resource "tls_private_key" "host" {
  algorithm   = "ED25519"
}
...
resource "local_file" "user_data" {
  content  = templatefile("tpl/userData.yml", {
     host_ed25519_private =
       tls_private_key.host.private_key_openssh
     host_ed25519_public  = ...
                })
  filename = "gen/user_data.yml"
}
...
ssh_keys:
  ed25519_private: |
    -----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAABG5vbmUA...
-----END OPENSSH PRIVATE KEY-----
# E1: File gen/user_data.yml is not valid YAML.
  in "<unicode string>", line 19, column 1:
    b3BlbnNzaC1rZXktdjEAAAAABG5vbmUA ...

Figure 1028. Yaml missing indentation resolution Slide presentation
# main.tf
resource "local_file" "user_data" {
  content  = templatefile("tpl/userData.yml", {
     host_ed25519_private = 
       indent(4, tls....host.private_key_openssh)
     ...})
  filename = "gen/user_data.yml"
}
# gen/usr_data.yml
ssh_keys:
  ed25519_private: |
␣␣␣␣-----BEGIN OPENSSH PRIVATE KEY-----
␣␣␣␣b3BlbnNzaC1 ...
␣␣␣␣...
␣␣␣␣...
␣␣␣␣-----END OPENSSH PRIVATE KEY-----

Figure 1029. Yaml syntax quirk 2: Wrong indentation Slide presentation
# gen/usr_data.yml

ssh_keys:
  ed25519_private: |
␣␣␣␣-----BEGIN OPENSSH PRIVATE KEY-----
␣␣␣␣␣␣b3BlbnNzaC1 ...
␣␣␣␣␣␣...
␣␣␣␣␣␣...
␣␣␣␣␣␣-----END OPENSSH PRIVATE KEY-----
# /etc/ssh/ssh_host_ed25519_key
...
ssh_keys:
  ed25519_private: |
-----BEGIN OPENSSH PRIVATE KEY-----
␣␣b3BlbnNzaC1 ...
␣␣...
␣␣...
␣␣-----END OPENSSH PRIVATE KEY-----

Figure 1030. Solving Yaml quirks using yamlencode / JSON Slide presentation
resource "local_file" "user_data" {
  content  = templatefile("tpl/userData.yml", {
     host_ed25519_private = tls_private_key.host.private_key_openssh
     host_ed25519_public  = tls_private_key.host.public_key_openssh
     devopsUsername       = hcloud_ssh_key.loginUser.name
               ...
#cloud-config

${yamlencode(
  {
    ...
    ssh_keys = {
      ed25519_private = host_ed25519_private
      ed25519_public = host_ed25519_public
    } ...
  }
)}

users:
  - name: ${devopsUsername} ...

Figure 1031. Watch out for your enemies! Slide presentation
root@hello:~# journalctl -f
May 06 04:41:20 hello cloud-init[898]: Cloud-init v. 22.4.2 finished at Mon, 06 May 2024 04:41:20 +0000. Datasource DataSourceHetzner.  Up 11.78 seconds
   ...
May 06 04:46:16 hello sshd[927]: Invalid user abc from 43.163.218.130 port 33408
May 06 04:46:17 hello sshd[927]: Received disconnect from 43.163.218.130 port 33408:11: Bye Bye [preauth]
May 06 04:46:17 hello sshd[927]: Disconnected from invalid user abc 43.163.218.130 port 33408 [preauth]
   ...
May 06 04:50:54 hello sshd[930]: fatal: Timeout before authentication for 27.128.243.225 port 59866
   ...
May 06 04:52:45 hello sshd[933]: Invalid user cos from 43.163.218.130 port 59776
   ...
May 06 04:53:04 hello sshd[935]: Invalid user admin from 194.169.175.35 port 51128
May 06 04:53:49 hello sshd[937]: User root from 43.163.218.130 not allowed because not listed in AllowUsers
May 06 04:53:49 hello sshd[937]: Disconnected from invalid user root 43.163.218.130 port 50592 [preauth]

exercise No. 13

Working on Cloud-init

Q:

We continue our exercise series Incrementally creating a base system by adding a Cloud-init configuration:

  1. Follow Figure 1024, “Terraform interface to Cloud Init and Figure 1024, “Terraform interface to Cloud Init to create simple web server.

    Tip

    You will have to extend your current firewall configuration allowing inbound traffic to port 80.

    On success pointing your web browser of choice to http://<your server's IP> should result in something similar to:

    I'm Nginx @ "95.217.154.104" created Sun May 5 06:58:37 PM UTC 2024
  2. With respect to Figure 1031, “Watch out for your enemies! ” inspect the output of journalctl -f on your own server for a while. Then modify your current sshd configuration:

    On success the following sequence should be possible:

    $ ssh -v devops@95.217.154.104
    ...
    debug1: SSH2_MSG_SERVICE_ACCEPT received
    debug1: Authentications that can continue: publickey 
    debug1: Next authentication method: publickey
    ...
    Last login: Sun May  5 19:21:12 2024 from 217.245.243.187
    devops@hello:~$ sudo su - 
    root@hello:~# hostname
    hello

    password is not among the list of allowed authentication methods.

    User devops may execute sudo commands by virtue of his membership in group sudo.

    On contrary ssh root login must be prohibited:

    $ ssh root@95.217.154.104
    root@95.217.154.104: Permission denied (publickey).
  3. Read the cloud-init documentation and/or related tutorials to:

    • Currently your (most likely outdated) cloud provider supplied distribution does not get upgraded on installation time:

      $ ./bin/ssh 
      ...
      devops@hello:~$ sudo su -
      root@hello:~# apt update
      Hit:1 http://security.debian.org/debian-security bookworm-security InRelease
      Hit:2 http://deb.debian.org/debian bookworm InRelease                                              
      ...
      Reading package lists... Done
      Building dependency tree... Done
      Reading state information... Done
      6 packages can be upgraded. Run 'apt list --upgradable' to see them.
      
      # apt list --upgradable
      Listing... Done
      less/stable-security,stable-security 590-2.1~deb12u2 amd64 [upgradable from: 590-2]
      libc-bin/stable-security,stable-security 2.36-9+deb12u7 amd64 [upgradable from: 2.36-9+deb12u6]
      libc-l10n/stable-security,stable-security 2.36-9+deb12u7 all [upgradable from: 2.36-9+deb12u6]
      libc6/stable-security,stable-security 2.36-9+deb12u7 amd64 [upgradable from: 2.36-9+deb12u6]
      locales-all/stable-security,stable-security 2.36-9+deb12u7 amd64 [upgradable from: 2.36-9+deb12u6]
      locales/stable-security,stable-security 2.36-9+deb12u7 all [upgradable from: 2.36-9+deb12u6]

      Modify your Cloud-init configuration to upgrade your distribution at server creation time. If so required your system should also reboot.

    • Install and configure fail2ban limiting ssh failed connection attempts.

      Tip

    • Install the plocate file indexer package and initialize it.

    On success all packages should be up to date:

    $ ./bin/ssh 
    ...
    devops@hello:~$ sudo su -
    root@hello:~# apt update
    Hit:1 http://security.debian.org/debian-security bookworm-security InRelease
    Hit:2 http://deb.debian.org/debian bookworm InRelease                                              
    ...
    Reading package lists... Done
    Building dependency tree... Done
    Reading state information... Done
    All packages are up to date.

    Failed login attempts should be banned: Keep a second login open in advance when trying to simulate login failures! You should then see a report similar to:

    root@hello:~# fail2ban-client status sshd
    Status for the jail: sshd
    |- Filter
    |  |- Currently failed:	2
    |  |- Total failed:	14
    |  `- Journal matches:	_SYSTEMD_UNIT=sshd.service + _COMM=sshd
    `- Actions
       |- Currently banned:	2
       |- Total banned:	2
       `- Banned IP list:	170.64.133.30 213.136.94.219

    Searching for file name components should work like e.g.:

    root@hello:~# locate ssh_host
    /etc/ssh/ssh_host_ed25519_key
    /etc/ssh/ssh_host_ed25519_key.pub
Figure 1032. Problem: Duplicate known_hosts entry on re-creating server Slide presentation

Problem of repeated terraform apply:

$ ssh root@128.140.108.60
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!

Figure 1033. Solution: Generating known_hosts ... Slide presentation
resource "local_file" "known_hosts" {
  content         = join(" "
                        ,[ hcloud_server.helloServer.ipv4_address
                        , tls_private_key.host.public_key_openssh ]
                    )
  filename        = "gen/known_hosts"
  file_permission = "644"
}
157.180.68.60 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5...XXgAC3eaueHMQ8ePtbnw

Figure 1034. ... and ssh wrapper Slide presentation
main.tf tpl/ssh.sh
resource "local_file" "ssh_script" {
  content = templatefile("tpl/ssh.sh", {
    ip=hcloud_server.hello.ipv4_address
  })
  filename        = "bin/ssh"
  file_permission = "755"
  depends_on      = [local_file.known_hosts]
}
#!/usr/bin/env bash

GEN_DIR=$(dirname "$0")/../gen

ssh -o UserKnownHostsFile= \
  "$GEN_DIR/known_hosts" devops@${ip} "$@"

exercise No. 14

Solving the ~/.ssh/known_hosts quirk

Q:

Extend Working on Cloud-init generating both a bin/ssh wrapper, a bin/scp wrapper and a corresponding gen/known_hosts file to be used by the two former scripts.

Avoiding ssh host key issues on login we require the creation of a known_hosts file among with each server being created:

157.180.78.16 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJDd+x7b80BM97rTU4RCM/CP+K7u4QAqx8ulDdXm9JDv

We assume this file being generated at gen/known_hosts. A corresponding ssh bash script shall be generated at the same level in bin/ssh:

#!/usr/bin/env bash

GEN_DIR=$(dirname "$0")/../gen

ssh -o UserKnownHostsFile="$GEN_DIR/known_hosts" devops@157.180.78.16 "$@"

# end of script

This way invoking ./bin/ssh will instruct the ssh client to read gen/known_hosts and thus avoiding unknown host key issues. The "$@" allows for optional user provided parameters.

For generating this script we start from a corresponding tpl/ssh.sh template file:

#!/usr/bin/env bash

GEN_DIR=$(dirname "$0")/../gen

ssh -o UserKnownHostsFile="$GEN_DIR/known_hosts" ${devopsUsername}@${ip} "$@"

# end of script

We then let Terraform use this template replacing ${devopsUsername} and ${ip} by your deployment's actual values.

As shown in Figure 1048, “Parent module vs. sub module context ” point to your child module's local tpl path by using ${path.module}:

resource "local_file" "ssh_script" {
  content = templatefile("${path.module}/tpl/ssh.sh", {
    ...
}

Likewise create a bin/scp script to be used for file transfer purposes as well:

#!/usr/bin/env bash

GEN_DIR=$(dirname "$0")/../gen

if [ $# -lt 2 ]; then
   echo usage: .../bin/scp ... devops@157.180.78.16 ...
else
   scp -o UserKnownHostsFile="$GEN_DIR/known_hosts" $@
fi

# end of script

Since copying objects requires at least a source and a destination parameter this script checks for the presence of at least two extra user supplied values by means of if [ $# -lt 2 ].

Figure 1035. Failsafe console login Slide presentation
#cloud-config

chpasswd:
  list: |
    root:bingo9wingo
  expire: False

Figure 1036. Avoiding Yaml parsing issues Slide presentation
#cloud-config

write_files:
  - content:
        ${base64encode(private_key_pem)}
    encoding: base64
    path:  /etc/nginx/snippets/cert/private.pem

Note

Congrats to paultyng