# The prow cluster

[The prow cluster](../prow) is where we run Prow, which currently does a lot of our CI, though
we are trying to [dogfood](dogfooding.md) more and more.

`tektoncd` uses
[`Prow`](https://github.com/kubernetes/test-infra/tree/master/prow)
for CI automation, though we are moving this over to
[use our own dogfooding](dogfooding.md).

- Prow runs in [the tektoncd GCP project](README.md#gcp-projects)
- [Ingress is configured to `prow.tekton.dev`](#ingress)
- Prow results are displayed via [gubernator](../gubernator/README.md)
- [Instructions for creating the Prow cluster](#creating-the-prow-cluster)
- [Instructions for updating Prow](#updating-prow-itself) and [Prow's Tekton Pipelines instance](#tekton-pipelines-with-prow)
- [Instructions for updating Prow configuration](#updating-prow-configuration)

_[See the community docs](../CONTRIBUTING.md#pull-request-process) for more on
Prow and the PR process, and see [Prow's own docs](https://github.com/kubernetes/test-infra/tree/master/prow)._

## Prow secrets

Secrets which have been applied to the prow cluster but are not committed here are:

- `GitHub` personal access tokens:
  - `bot-token-github` in the default namespace
  - `bot-token-github` in the github-admin namespace
  - `hmac-token` for authenticating GitHub
  - `oauth-token` which is a GitHub access token for [`tekton-robot`](https://github.com/tekton-robot),
    used by Prow itself as well as by containers started by Prow via [the Prow config](../prow/config.yaml).
    See [the GitHub secret Prow docs](https://github.com/kubernetes/test-infra/blob/068e83ba2f8e9261c0af4cee598c70b92775945f/prow/getting_started_deploy.md#create-the-github-secrets).
- `GCP` secrets:
  - `test-account` is a token for the service account
    `prow-account@tekton-releases.iam.gserviceaccount.com`. This account can
     interact with GCP resources such as uploading Prow results to GCS
     (which is done directly from the containers started by Prow, configured in [config.yaml](../prow/config.yaml)) and
     [interacting with boskos clusters](../boskos/README.md).
  - Nightly release secret: `nightly-account` a token for the nightly-release GCP service account

## Creating the Prow cluster

If you need to re-create the Prow cluster (which includes [the boskos](../boskos/README.md)
running inside), you will need to:

1. [Create a new cluster](#creating-the-cluster)
2. Create [the necessary secrets](#prow-secrets)
3. [Apply the new Prow and Boskos](#start-it)
4. [Setup ingress](#ingress)
4. [Update GitHub webhook(s)](#update-github-webhook)

### Creating the cluster

To create a cluster of the right size, using [the same GCP project](README.md#gcp-projects):

```bash
export PROJECT_ID=tekton-releases
export CLUSTER_NAME=tekton-plumbing

gcloud container clusters create $CLUSTER_NAME \
 --scopes=cloud-platform \
 --enable-basic-auth \
 --issue-client-certificate \
 --project=$PROJECT_ID \
 --region=us-central1-a \
 --machine-type=n1-standard-4 \
 --image-type=cos \
 --num-nodes=8 \
 --cluster-version=latest
```

### Start it

Apply the Prow and boskos configuration:

```bash
# Deploy boskos
kubectl apply -f boskos/boskos.yaml # Must be applied first to create the namespace
kubectl apply -f boskos/boskos-config.yaml
kubectl apply -f boskos/storage-class.yaml

# Deploy GitHub Proxy
kubectl apply -f prow/gce-ssd-retain_storageclass.yaml
kubectl apply -f prow/ghproxy.yaml

# Deploy Prow
kubectl apply -f prow/prowjob-schemaless_customresourcedefinition.yaml
kubectl apply -f prow/prow.yaml
kubectl apply -f prow/cherrypicker_deployment.yaml
kubectl apply -f prow/cherrypicker_service.yaml

# Deploy daemonset to configure fs.inotify.max_user_[watches,instances] via sysctl.
# This is to deal with kind having issues like https://kind.sigs.k8s.io/docs/user/known-issues/#pod-errors-due-to-too-many-open-files
kubectl apply -f prow/tune-sysctls_daemonset.yaml

# Create Prow's configuration
kubectl create configmap config --from-file=config.yaml=prow/config.yaml
kubectl create configmap plugins --from-file=plugins.yaml=prow/plugins.yaml
```

### Ingress

To get ingress working properly, you must:

- Install and configure [cert-manager](https://github.com/jetstack/cert-manager/).
  `cert-manager` can be installed via `Helm` using this
  [guide](https://docs.cert-manager.io/en/latest/getting-started/)
- Apply the ingress resource and update the `prow.tekton.dev` DNS configuration.

To apply the ingress resource:

```bash
# Apply the ingress resource, configured to use `prow.tekton.dev`
kubectl apply -f prow/ingress.yaml
```

To see the IP of the ingress in the new cluster:

```bash
kubectl get ingress ing
```

You should be able to navigate to this endpoint in your browser and see the Prow landing page.

Then you can update https://prow.tekton.dev to point at the Cluster ingress address.
(Not sure who has access to this domain name registration, someone in the Linux Foundation?
[dlorenc@](http://github.com/dlorenc) can provide more info.)

### Update GitHub webhook

You will need to configure [GitHubs's webhook(s)](https://developer.github.com/webhooks/)
to point at [the ingress](#ingress) of the new Prow cluster. (Or you can use the domain name.)

For `tektoncd` this is configured at the Org level.

* github.com/tektoncd -> Settings -> Webhooks -> `http://some-ingress-ip/hook`

Update the value of the webhook with `http://ingress-address/hook`
(see [kicking the tires](#kicking-the-tires) to get the ingress IP).

### OAuth Setup

OAuth Setup is done following the [official guide](https://github.com/kubernetes/test-infra/blob/master/prow/cmd/deck/github_oauth_setup.md).
The "Prow" OAuth GitHub application is defined in the `tektoncd` GitHub org.

## Updating Prow itself

Prow has been installed by taking the
[starter.yaml](https://github.com/kubernetes/test-infra/blob/master/prow/cluster/starter.yaml)
and modifying it for our needs.

Updating (e.g. bumping the versions of the images being used) requires:

0. If you are feeling cautious and motivated, manually backup the config values by hand
   (see [prow.yaml](../prow/prow.yaml) to see what values will be changed).
1. Manually updating the `image` values and applying any other config changes found in the
   [starter.yaml](https://github.com/kubernetes/test-infra/blob/master/prow/cluster/starter.yaml)
   to our [prow.yaml](../prow/prow.yaml).
2. Updating the `utility_images` in our [config.yaml](../prow/config.yaml) if the version of
   the `plank` component is changed.
3. Applying the new configuration with:

   ```yaml
    # Step 1: Configure kubectl to use the cluster, doesn't have to be via gcloud but gcloud makes it easy
    gcloud container clusters get-credentials prow --zone us-central1-a --project tekton-releases

    # Step 2: Update Prow itself
    kubectl apply -f prow/prow.yaml

    # Step 2: Update the configuration used by Prow
    kubectl create configmap config --from-file=config.yaml=prow/config.yaml --dry-run -o yaml | kubectl replace configmap config -f -

    # Step 3: Remember to configure kubectl to connect to your regular cluster!
    gcloud container clusters get-credentials ...
   ```
4. Verify that the changes are working by opening a PR and **manually looking at the logs of each check**,
   in case Prow has gotten into a state where failures are being reported as successes.

These values have been removed from the original
[starter.yaml](https://github.com/kubernetes/test-infra/blob/master/prow/cluster/starter.yaml):

- The `ConfigMap` values `plugins` and `config` because they are generated from
  [config.yaml](../prow/config.yaml) and [plugin](../prow/plugins.yaml)
- The `Services` which were manually configured with a `ClusterIP` and other routing
  information (`deck`, `tide`, `hook`)
- The `Ingress` `ing` - Configuration for this is in [ingress.yaml](../prow/ingress.yaml)
- The `statusreconciler` Deployment, etc. - Created #54 to investigate adding this.
- The `Role` values give `pod` permissions in the `default` namespace as well as `test-pods` -
  The intention seems to be that `test-pods` be used to run the pods themselves, but we
  don't currently have that configured in our [config.yaml](../prow/config.yaml).

### Tekton Pipelines with Prow

[Tekton Pipelines](https://github.com/tektoncd/pipelines) is also installed in the `prow`
cluster so that Prow can trigger the execution of
[`PipelineRuns`](https://github.com/tektoncd/pipeline/blob/main/docs/pipelineruns.md).

Prow supports pipelines v1alpha1 up to v0.13.1:

```bash
kubectl apply --filename  https://storage.googleapis.com/tekton-releases/pipeline/previous/v0.13.1/release.yaml
```

_See also [Tekton Pipelines installation instructions](https://github.com/tektoncd/pipeline/blob/main/docs/install.md)._

## Updating Prow configuration

Changes to [config.yaml](../prow/config.yaml) are automatically applied to the Prow
cluster via a [tekton task](../tekton/resources/cd/prow-condig-cd.yaml) that
runs in the `dogfooding` cluster.

To apply the configuration "manually":

```bash
# Step 1: Configure kubectl to use the cluster, doesn't have to be via gcloud but gcloud makes it easy
gcloud container clusters get-credentials prow --zone us-central1-a --project tekton-releases

# Step 2: Update the configuration used by Prow
kubectl create configmap config --from-file=config.yaml=prow/config.yaml --dry-run -o yaml | kubectl replace configmap config -f -

# Step 3: Remember to configure kubectl to connect to your regular cluster!
gcloud container clusters get-credentials ...
```
