grim/hgkeeper

Split the server pieces out of hgweb.go

2022-03-07, Gary Kramlich
334e63a51155
Split the server pieces out of hgweb.go
# HGKeeper
[ ![Issues](https://img.shields.io/badge/Issues-YouTrack-ee3b8b.svg) ](https://issues.imfreedom.org/issues/HGKEEPER?u=1)
HGKeeper is an server for [mercurial](https://www.mercurial-scm.org/)
repositories. It provides access control for SSH access and public HTTP
access via hgweb.
It was originally designed to be run in a container but recently support has
been added to run it from an existing openssh-server.
## License
HGKeeper is licensed under the GNU AFFERO GENERAL PUBLIC LICENSE version 3.
## Status
This project is brand new and only lightly tested. If you find issues, or are
looking for ways to help, please check out our
[issue tracker](https://issues.imfreedom.org/issues/HGKEEPER?u=1).
## Building
You can build hgkeeper with the following commands:
```
$ go build
```
## Getting Started
The initial setup of HGKeeper has a few steps.
### SSH Host Keys
Since HGKeeper provides an SSH server you will need to generate host keys for
it if you are going to use it. You can skip the SSH server by passing
`--disable-ssh` or setting the `HGK_DISABLE_SSH` environment variable to `true`.
This is useful if you just want to provide read only repositories.
You can generate SSH host keys for whatever type you like, but rsa will cover
just about everyone. That said, a lot of people prefer to use ed25519 as well.
By default the SSH host keys will be looked for in the directory `host-keys` in
the current working directory of HGKeeper. This can be changed with the
`--ssh-host-keys-path` or `-H` command line arguments to hgkeeper.
This directory will be read and files in it will attempt to be loaded into the
server.
To generate a host key you can use the following commands, note that you can
create other types via the `-t` option, but you should read the `ssh-keygen`
documentation as other options are available for each type.
```
$ ssh-keygen -t rsa -b 4096 -o host-keys/ssh_host_rsa_key
$ ssh-keygen -t ed25519 -o host-keys/ssh_host_ed25519_key
```
### Create the hgkeeper repo
Before you can run the server we need to create the `hgkeeper` admin
repository. This can be done via `hgkeeper setup`. You will need to pass the
arguments `--admin-username` which is the username of the admin user, as well
as `--admin-pubkey` which is the path to the SSH public key for the new admin
user. By default this will create a new repository under `repos/hgkeeper`.
There are some additional options which you can discover via
`hgkeeper setup --help`.
## Usage
HGKeeper has a couple of modes of operation but `serve` is the main mode.
### setup
The `setup` command is used to bootstrap HGKeeper. It will create the
directory for the repositories as well as the `hgkeeper` repository.
After initial setup, please make sure to read the `README.md` in the `hgkeeper`
admin repository that was created, as it details how access control works.
### serve
The `serve` command is the main mode of operation which is to provide access to
the repositories.
You can optionally disable either one of the HTTP or SSH servers to limit
resources or just disable the portions you don't need.
For example, say you don't need public access or the web interface at all, you
can disable it by passing `--disable-http` to the `serve` command or by setting
the `HGK_DISABLE_HTTP` environment variable to `true`.
Likewise, if you just want to serve up read only repositories over http you can
disable the SSH server by passing `--disable-ssh` to the `serve` command or by
setting the `HGK_DISABLE_SSH` environment variable to `true`.
The `--disable-http` and `--disable-ssh` options are mutually exclusive and
passing both will stop HGKeeper from starting.
The `serve` command also has options to tell HGKeeper how it can be accessed
remotely. This is useful when integrating it with OpenSSH Server. These options
are `--external-hostname` and `--external-port`. `--external-hostname` can be
a hostname or an IP address, but it should be the address that you're using
to access HGKeeper. Likewise, `--external-port` is the external port that will
get you to HGKeeper. It has a default value of `22222`.
### authorized-keys
The `authorized-keys` command is used when itegrating with OpenSSH on the same
host as HGKeeper. It takes a single argument as the SHA256 fingerprint of the
an SSH public key. If that key is known to HGKeeper, it will output an
`authorized_keys` text that OpenSSH can use to continue the HG operation.
### once
The `once` command is used in conjunction with the above mentioned
`authorized-keys` command. It is the command that HGKeeper specifies that
Open SSH should use to continue the SSH operations and is not useful in any
other way.
## Access Control
Access control is defined in the `hgkeeper` admin repository that is created
via the `hgkeeper setup` command. It is implemented via
[casbin](https://casbin.org) using the RBAC with deny-override model as a
base. More information can be found in the [files](setup/resources/)
that are placed in the `hgkeeper` admin repository.
## Running HGKeeper
There are multiple ways to run HGKeeper. It can be ran in a container, as a
standalone server directly on a host, or executed on demand by OpenSSH.
Both running in a container and running as a standalone can also be integrated
with OpenSSH as well.
## Running in a Container
HGKeeper is available on docker hub under
[rwgrim/hgkeeper](https://hub.docker.com/r/rwgrim/hgkeeper) and is updated via
CI.
Just like all modes of operation, running in the container is going to require
an `hgkeeper` admin repository and at least one SSH host key.
First we will create the HGKeeper admin repository. We will use the normal
HGKeeper container with an overridden command to run the setup. An extra step
to this is that you will need to volume mount in a file containing the public
key of the initial administrator of this instance. In the following example, we
assume that the key is in `~/.ssh/id_rsa.pub`.
Also, since this container is just used for initialization of files on the
host, we're passing the `--rm` flag to make sure it's deleted when done.
We can now run the initialization setup that follows.
```
docker run --rm \
-v $(pwd)/repos:/repos \
-v ~/.ssh/id_rsa.pub:/admin-pubkey:ro \
-e HGK_ADMIN_USERNAME=my_username \
-e HGK_ADMIN_PUBKEY=/admin-pubkey \
-e HGK_REPOS_PATH=/repos \
docker.io/rwgrim/hgkeeper:latest \
hgkeeper setup
```
Once this step is done you should now have a `repos` directory in your current
working directory and it should have a brand new `hgkeeper` admin repository
in it. These repositories are HGKeeper's copies of them, you should not modify
them directly and should be using the SSH capabilities of HGKeeper to push/pull
to them.
For the rest of these instructions we are going to assume that you have your
SSH host keys in a directory named `host-keys` in the current working
directory.
The following example uses the same assumptions as the setup container above,
but it is also going to expose the container on the host's network interface.
The following example runs the container in the background, gives it a name of
`hgkeeper`, and sets it to always restart. If you're just doing testing, you
will probably want to remove the `-d` (background), `--name hgkeeper` (name),
and `--restart=always` options.
```
docker run -d --name hgkeeper \
--restart=always \
-v $(pwd)/host-keys:/host-keys:ro \
-v $(pwd)/repos:/repos \
-e HGK_SSH_HOST_KEYS=/host-keys \
-e HGK_REPOS_PATH=/repos \
-p 8080:8080 \
-p 22222:22222 \
docker.io/rwgrim/hgkeeper:latest \
hgkeeper serve
```
If you would like to integrate this container with OpenSSH server, please see
the `Integrating With OpenSSH Server` section below.
If you want to integrate this container with an existing OpenSSH server and
keep HGKeeper only on the loopback device, you can do that by adding the
`HGK_EXTERNAL_HOSTNAME` and `HGK_EXTERNAL_PORT` environment variables. This
will enable an `/ssh/authorized_keys` HTTP endpoint that we can tell OpenSSH
server to use. We'll also change the port forwards from any interface on the
host to just the loopback device. To do this, you'd change the container
command to the following:
```
docker run -d --name hgkeeper \
--restart=always \
-v $(pwd)/host-keys:/host-keys:ro \
-v $(pwd)/repos:/repos \
-e HGK_SSH_HOST_KEYS=/host-keys \
-e HGK_REPOS_PATH=/repos \
-e HGK_EXTERNAL_HOSTNAME=<hostname or ip address> \
-e HGK_EXTERNAL_PORT=22222 \
-p 127.0.0.1:8080:8080 \
-p 127.0.0.1:22222:22222 \
docker.io/rwgrim/hgkeeper:latest \
hgkeeper serve
```
The `<hostname or ip address>` is the hostname or IP address of the host where
the container is running, and the value of `HGK_EXTERNAL_PORT` needs to match
the host port number specified in the `-p` argument for the SSH port which is
22222 in this example.
To configure the OpenSSH server portion you'll need to make sure that the host
has the `ssh` client as well as `curl` installed. Then you can add the
following block to `/etc/ssh/sshd_config` replacing `<hostname>` and `<port>`
with the hostname of the machine running the container and the HTTP port that
was forwarded for the container. In example above, that was port `8080`.
```
Match User hg
AuthorizedKeysCommand curl -q http://<hostname>:<port>/ssh/authorized_keys?fp=%f
AuthorizedKeysCommandUser hg
```
Once this change has been made, you need to reload OpenSSH server, but then you
should be able to access the HGKeeper repositories by using
`ssh://hg@<hostname>:<port>/<repository>`.
And that's it!
Of course, you'll probably want to add some more users. To find out how to do
that, be sure to read the `README.md` in the `hgkeeper` admin repository.
## Running Locally
We do not currently provide any packages or startup scripts for HGKeeper, but
to get started it is recommended that you create an `hg` user and store
everything in that user's home directory.
For most Linux distributions, you can create the `hg` user with the following
command:
```
useradd --home-dir /var/lib/hg --create-home --system --shell /usr/sbin/nologin
```
To make life easier we're going to run the setup command as the `hg` user, but
since the `hg` user doesn't have a shell we have to tell the `su` command which
shell to use. So to get started type the following command:
```
su -s /bin/bash hg
cd ~
whoami
```
You should now see a bash prompt with the output of the `whoami` command saying
that you are the `hg` user. If not, please review these steps and try again.
If you did see the `hg` username output, you can follow the steps in
`SSH Host Keys` and `Create the hgkeeper repo`.
Once the SSH host keys and the `hgkeeper` admin repository are created, you
can run HGKeeper with the `hgkeeper serve` command. There are some other
options that are available so be sure to check out `hgkeeper serve --help`.
If you would like to integrate this with an existing OpenSSH server, see the
`Integrating with OpenSSH Server` section below.
## Integrating with OpenSSH Server
There are a two different ways to integrate HGKeeper with OpenSSH Server and
some of the steps vary across operating systems. If you run into a case where
these instructions do not work for you, please reach out via the issue tracker
so we can help you and fix the documentation.
The two modes of operation are `serve` and `run once`.
`serve` mode can include the normal container setup described above, which
would integrate with OpenSSH Server the same way as running HGKeeper locally.
Running HGKeeper in `serve` mode means that you can also use the HTTP portion
of it as the HTTP portion is required for integrating the serve mode with
OpenSSH.
Run once mode, uses the built-in mechanisms of OpenSSH Server to run HGKeeper
when someone connects. This method means that the HTTP portion of HGKeeper will
not be available.
### Creating the hg user
When it comes to integrating with OpenSSH server you need to create a user that
will run interact with HGKeeper during SSH connections and in `run once` mode
own the repositories on disk. This user can be named whatever you like, but for
the purposes of this documentation we will be naming the user `hg`.
Regardless of your mode of operation, you will need to create this user on the
OpenSSH Server's host operating system. In the documentation for running
locally above, we already created this user.
For most Linux distributions, you can create the `hg` user with the following
command:
```
useradd --home-dir /var/lib/hg --create-home --system --shell /usr/sbin/nologin
```
### Setup HGKeeper
This step is only necessary for `run once` mode.
To get HGKeeper fully running, you will need to run `hgkeeper setup` to create
the `hgkeeper` admin repository as well as the initial admin user. Once this
repository is created, make sure it and it's parent directory is owned by the
`hg` user and that the `hg` user has write permission.
### Installing HGKeeper
This step is only necessary for `run once` mode.
OpenSSH server has some very specific requirements for calling applications
directly. These requirements are that the executable as well as all of the
directories leading up to the executable must be owned by root and not
writeable by the group or other users. To deal with this, we will be
installing HGKeeper into `/usr/local/bin`. This directory should fulfill all
of the those requirements. So just `sudo cp hgkeeper /usr/local/bin/` and
make sure that it is owned by `root` with a file mode of `755`.
### Configuring OpenSSH Server
The OpenSSH configuration is actually quite easy, you just need to drop the
one of the following snippet into `/etc/ssh/sshd_config` depending on your
mode of operation. Of course, if you customized the install location or
username you'll have to adjust that in the snippets below.
You may be able to use `/etc/ssh/sshd_config.d/hgkeeper.conf` but in our
testing on Debian unstable we were unable to get it working properly.
For `run once` mode, you can use the following snippet, but note that the value
for `--repos-path` needs to be the absolute path to your repositories.
```
Match User hg
AuthorizedKeysCommand /usr/local/bin/hgkeeper --repos-path=<path to your repositories> authorized-keys %f
AuthorizedKeysCommandUser hg
```
For serve mode you need to know the external hostname and port number for HTTP
for HGKeeper. Once you have that info, you can use the following snippet.
```
Match User hg
AuthorizedKeysCommand /usr/bin/curl -q http://<external-hostname>:<external-http-port>/ssh/authorized_keys?fp=%f
AuthorizedKeysCommandUser hg
```
Once you've created the file, you'll need to reload OpenSSH Server. This is
usually done via `service ssh reload` but may vary based on your operating
system.
Once you've reloaded OpenSSH Server you should be able to clone the `hgkeeper`
admin repository with `hg clone ssh://hg@yourhostname/hgkeeper` as long as the
machine you're cloning from has the private key that you added as your initial
admin user.
## Creating Repositories
Creating a new repository in HGKeeper is just the same as using Mercurial over
SSH. That is, you can use the `hg init` command with a path to HGKeeper. As
long as you have init permission for the path `HGKeeper` will create an empty
repository for you.
For example, if you wanted to create a new repository named
`myteam/new-project-1`, you would do so with
`hg init ssh://hg@my-hgkeeper-domain/myteam/new-project-1`. If there were no
issues, the command will exit without error and you are now free to clone it
with `hg clone ssh://hg@my-hgkeeper-domain/myteam/new-project-1`.
## Known Errata
### no mutual signature algorithm
Fedora 33 disabled `ssh-rsa` by default in openssh-client. This breaks the
golang `x/crypto/ssh` library. See
[this bug](https://github.com/golang/go/issues/37278) for more details.
The current work around for this is to add a host block in your ssh client
configuration to re-enable `ssh-rsa`. See the following example:
```
Host <hgkeeper hostname>
PubkeyAcceptedKeyTypes ssh-rsa
```