blog/source/_posts/ssh-certificate-cloudflare-tunnel.md

---
title: SSH certificate using Cloudflare Tunnel
excerpt: A quick quide to SSH certificate without using an identity provider.
date: 2023-02-13
updated: 2023-02-21
tags:
  - cloudflare
---

This article provides a quick-start guide to SSH certificate using Cloudflare Tunnel. More information can be found in the official docs.

- [Public keys are not enough for SSH security](https://blog.cloudflare.com/public-keys-are-not-enough-for-ssh-security/)
- [SSH with short-lived certificates](https://developers.cloudflare.com/cloudflare-one/tutorials/ssh-cert-bastion/)
- [Configure short-lived certificates](https://developers.cloudflare.com/cloudflare-one/identity/users/short-lived-certificates/)
- [Self-hosted applications](https://developers.cloudflare.com/cloudflare-one/applications/configure-apps/self-hosted-apps/)
- [Connect with SSH through Cloudflare Tunnel](https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/use_cases/ssh/)

## Introduction

One unpleasant task I had previously in an enterprise with Linux servers was SSH key management, specifically checking the SSH public keys of departed staff have been removed from the Ansible config. Then I learned from [this article](https://smallstep.com/blog/use-ssh-certificates/) that it is possible to SSH using a short-lived (<1 day) certificate that is only issued to the user after successfully authenticate with the enterprise identity provider's (e.g. Azure AD) single sign-on (SSO). This means once a user is revoked from the identity provider, that user would not be issued with a new certificate to SSH again the next day. At that time, I didn't feel like configuring and integrating an identity provider, so I held off trying the feature.

Recently, I wanted to try out the Cloudflare Zero Trust free tier. While reading through the [SSH configuration](https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/use_cases/ssh/) guide, I found out that Cloudflare support issuing [SSH user certificate](https://developers.cloudflare.com/cloudflare-one/identity/users/short-lived-certificates/). While Cloudflare supports several [SSO integration](https://developers.cloudflare.com/cloudflare-one/identity/idp-integration/), it also supports authenticating using [one-time PIN](https://developers.cloudflare.com/cloudflare-one/identity/one-time-pin/) sent to an email address that does not have to be a Cloudflare account. Cloudflare also supports browser-based shell, just like the AWS [Session Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager.html).

## Prerequisites

- A domain hosted on Cloudflare DNS
- Cloudflare Zero Trust (free for 50 users)
- A VM or cloud instance (optional, easier to clean up)

## Cloudflare Zero Trust

Navigate to **Zero Trust** page shown on the sidebar after you login to [dash.cloudflare.com](https://dash.cloudflare.com). If this is your first time, Cloudflare will ask for billing info in which you can use an existing one or add a new credit card. You won't get charged as long as you stay within the **free tier** (50 users), I will show you how to check later in this article.

The setup will then ask you to name your team domain _team-name_.cloudflareaccess.com. Just create a random name for now, you can always change it later.

## Add an application

Once you're in Zero Trust console, navigate to **Access** -> **Applications**. **Add an application** and choose **Self-hosted**.

**Configure app** tab,

- Application name: any name
- Session duration: 15 minutes.
  - In a corporate environment, "6 hours" is probably more user-friendly.
  - For sensitive server, consider "No duration".
- Application domain: test.yourdomain.com
  - The subdomain should not have an existing website.
  - It may be possible to use an existing website, by specifying test.yourdomain.com/custom-path for SSH, though I haven't try it.
- App Launcher visbility: No
- Accept all available identity providers: No, unless you have integrated an identity provider.
- Select One-time PIN
- Instant Auth: Yes

**Add policies** tab,

- Policy name: any name
- Action: Allow
- Session duration: same
- Configure rules: (Include) Emails = an email address
  - Any of your email is fine, regardless whether it's a Cloudflare account.
  - Cloudflare _will not_ create an account using that email, it will only be used to receive one-time PIN.

**Setup** tab:

- CORS settings: leave it as is
- Cookies settings:
  - SameSite Attribute: blank or Lax
    - Either setting is practically the same, browsers default to Lax when SameSite is not set.
    - "Strict" value cannot be used because Cloudflare will authenticate the user on _team-name_.cloudflareaccess.com and issue a cookie on test.yourdomain.com.
  - HTTP Only: Yes
- Additional settings:
  - Enable automatic cloudflared authentication: Yes
  - Browser rendering: SSH

## Generate a CA certificate

Navigate to **Access** -> **Service Auth** -> **SSH** tab. Select the application you just created and **Generate certificate**.

Copy the generated public key and save it to `/etc/ssh/ca.pub` in your host (the host you're going to SSH into).

```
sudo -e /etc/ssh/ca.pub
```

## Create a tunnel

Navigate to **Access** -> **Tunnels**

- Name: any name

**Install connector** tab, choose the relevant OS and run the installation command. Once installed, you should see "connected" status.

**Route tunnel** tab,

- Public hostname: test.yourdomain.com
  - This is the application domain in the [Add an application](#Add-an-application) step.
- Service
  - SSH type: URL = localhost:22
    - Replace 22 with the custom SSH port you are going to use.

After finishing creating a tunnel, you should have a new CNAME DNS record that points to _tunnel-id_.cfargotunnel.com. If there is no CNAME entry, grab the tunnel ID and create a new DNS record.

## Start SSH server

Install `openssh-server`.

`sudo -e /etc/ssh/sshd_config.d/cf.conf`

```plain /etc/ssh/sshd_config.d/cf.conf
TrustedUserCAKeys /etc/ssh/ca.pub
ListenAddress 127.0.0.1
ListenAddress ::1
PasswordAuthentication no
# Uncomment below line for custom port
# Port 1234
```

`systemctl restart ssh` or `systemctl restart sshd`

## Create a test user

The easiest setup is one where a Unix username matches the email that you configured to receive one-time PIN in previous steps. For example, if you set **loremipsum**@youremail.com, then create a new user **loremipsum**.

`sudo adduser loremipsum`

Set a random password and leave everything else blank.

### Matching email to different username

To match **loremipsum**@youremail.com to **lipsum** user:

```plain /etc/ssh/sshd_config.d/cf.conf
Match user lipsum
  AuthorizedPrincipalsCommand /bin/echo 'loremipsum'
  AuthorizedPrincipalsCommandUser nobody
```

**loremipsum+somealias**@youremail.com also works.

```plain /etc/ssh/sshd_config.d/cf.conf
Match user lipsum
  AuthorizedPrincipalsCommand /bin/echo 'loremipsum+somealias'
  AuthorizedPrincipalsCommandUser nobody
```

### AuthorizedPrincipalsFile

For NixOS user, `AuthorizedPrincipalsCommand` will not work because the command will run within "/nix/store" but it is read-only. Instead, you should use `AuthorizedPrincipalsFile`. This config also enables you to match multiple emails to a username, just separate each email user by newline. This applies to all OpenSSH instances, not just NixOS.

`echo 'loremipsum' | sudo tee /etc/ssh/authorized_principals`

````nix /etc/nixos/configuration.nix
  services.openssh = {
    enable = true;
    permitRootLogin = "no";
    passwordAuthentication = false;
    # ports = [ 1234 ];
    extraConfig =
      ''
        TrustedUserCAKeys /etc/ssh/ca.pub
        Match User lipsum
          AuthorizedPrincipalsFile /etc/ssh/authorized_principals
          # if there is no existing AuthenticationMethods
          AuthenticationMethods publickey
      '';
  };
```

### Other use cases

https://developers.cloudflare.com/cloudflare-one/identity/users/short-lived-certificates/#2-ensure-unix-usernames-match-user-sso-identities

## Initiate SSH connection

Install `cloudflared` on the host that you're going to SSH from.

`cloudflared access ssh-config --hostname test.yourdomain.com --short-lived-cert`

Example output:

```plain ~/.ssh/config
Match host test.yourdomain.com exec "/usr/local/bin/cloudflared access ssh-gen --hostname %h"
    ProxyCommand /usr/local/bin/cloudflared access ssh --hostname %h
    IdentityFile ~/.cloudflared/%h-cf_key
    CertificateFile ~/.cloudflared/%h-cf_key-cert.pub
````

or

```plain ~/.ssh/config
Host test.yourdomain.com
    ProxyCommand bash -c '/usr/local/bin/cloudflared access ssh-gen --hostname %h; ssh -tt %r@cfpipe-test.yourdomain.com >&2 <&1'

Host cfpipe-test.yourdomain.com
    HostName test.yourdomain.com
    ProxyCommand /usr/local/bin/cloudflared access ssh --hostname %h
    IdentityFile ~/.cloudflared/test.yourdomain.com-cf_key
    CertificateFile ~/.cloudflared/test.yourdomain.com-cf_key-cert.pub
```

Save the output to `$HOME/.ssh/config`.

Now, the moment of truth.

`ssh loremipsum@test.yourdomain.com` (replace the username with the one you created in [Create a test user](#Create-a-test-user) step.)

The terminal should launch a website to _team-name_.cloudflareaccess.com. Enter the email you configured in [Add an application](#Add-an-application) step and then enter the received 6-digit PIN.

Back to the terminal, wait for at least 5 seconds and you should see the usual SSH authentication.

> You may wondering why you still see fingerprint warning, I find this article [SSH Best Practices using Certificates, 2FA and Bastions](https://goteleport.com/blog/how-to-ssh-properly/) explains it well.

## Browser-based shell

As a bonus, head to test.yourdomain.com (see [Add an application](#Add-an-application) step) which will redirect you to a login page just the previous step. After login with a 6-digit PIN, you shall see a browser-based shell.

## Usage monitoring

Head to **Settings** -> **Account** to monitor how many users you have, each email address you configured to receive one-time PIN is counted as one user.

To delete user(s), head to **Users**, tick the relevant users, **Update status** and then **Remove**. The seat usage column should show _Inactive_.

## Inspect user certificate

`ssh-keygen -L -f ~/.cloudflared/test.yourdomain.com-cf_key-cert.pub`