How to maintain the Jenkins pipeline for OpenWhisk
The OpenWhisk community now supports a multi-node staging environment that consists of three virtual machines that are used as the compute resource to launch and run tests using Jenkins pipeline builds.
This article explains how we created and configured these VMs with the Apache Infrastructure team, as well as how to maintain them and the Jenkins pipeline for OpenWhisk contributors.
Here is the basic information about all the VMs:
The label can be directly used in the script of Jenkinsfile to access each VM. The public domain name or public IP can be used to access each VM anywhere there is an internet connection.
Gain the root access to the VMs
In order to configure these VMs, you need to have the root access to them. Apache allows ASF committers to access VMs by SSH.
First, please add a public key to your account after you login the website https://id.apache.org. The public key can be added in the field SSH Key (authorized_keys line).
Second, you need to reach to the Apache infrastructure team to get the root access. I suggest you contact them by the slack channel asfinfra at https://the-asf.slack.com/, or create a ticket for the Apache infra team at https://issues.apache.org/jira/secure/Dashboard.jspa. The username you can use is your Apache ID.
After they grant you the access, you can access each of the VM, and configure your initial password. Apache applies OPIE to configure and access the backend VMs, when the sudo command is executed. The instruction for the password configuration is available here: https://infra-test.apache.org/reference/committer/opie.html. It is possible that your username has not been added into the sudoer’s list. Apache configures the sudoer’s list on a VM-to-VM basis. When you reach out to the Apache infra team, make sure to list all the VMs you’d like to have the sudo permission.
When you walk through all the steps above, you should be able to access the VMs by SSH. For example, to access VM openwhisk1 you can run the following command in your terminal:
ssh openwhisk-vm1-he-de.apache.org -l <your Apache ID>If the current user of the terminal is different from your Apache ID, you need to add “-l <your Apache ID>” to use your apache ID.
Existing configurations for the VMs
# SSH access among all these three VMS
If we deploy OpenWhisk on top of multiple machines with ansible command, we need to have SSH communication among these machines. I have already created a user called openwhisk for each VM to configure the SSH access among all VMs. This user has been granted sudo permission. The key pairs has been generated under /home/openwhisk/.ssh. In order to access one VM, the public keys of the other two should be stored in openwhisk.pub under /etc/ssh/ssh_keys. The file name ending with “.pub” needs to match the username to be configured. Each VM has a configuration file, indicating where to store the public keys for SSH authentication, available at /etc/ssh/sshd_config. The AuthorizedKeysFile is set to /etc/ssh/ssh_keys/%u.pub. If you need to configure SSH for other usernames, this paragraph is your reference.
# Certificate to launch the private docker registry service
Since we only need to download and build the source code of OpenWhisk on one VM, we need to set up a private docker registry service, so that the docker images we build can be access by other two VMs. The user Jenkins is used by the Jenkins CI to launch the pipeline build. I have already created the certificate and the key to launch the docker registry. The cert domain.crt and the key domain.key can be found under /home/jenkins/certs on VM openwhisk1, since openwhisk1 is used to download and build the source code. If you want to or need to update the certificate and the key, please run the following command:
openssl req -x509 -newkey rsa:2048 -nodes -keyout domain.key -x509 -days 3650 -out domain.crtThis command creates the certificate and key lasting 10 years. You can specify the expiration time as you wish. To resolve the issue of “x509: certificate signed by unknown authority”, you need to register the key and certificate with docker service by adding domain.crt and domain.key into the directory /etc/docker/certs.d/<domain name:port>. You need to create a file called domain.cert. Attention: it ends with “.cert”. The content of this file is the same as domain.crt. Here is what the file structure looks like:
/etc/docker/certs.d/ <-- Certificate directory
└── <domain name:port>
├── domain.cert <-- Client certificate
├── domain.key <-- Client key
└── domain.crt <-- Certificate authority that signed the registry certificateThe only thing left is the domain name and the port. Since openwhisk1 is used to download and build the source, domain name is openwhisk-vm1-he-de.apache.org. By default, the port is 443. However, OpenWhisk runs the Nginx service on that port, I use 444 as the port here. As a recap, the directory is named openwhisk-vm1-he-de.apache.org:444. Remember: you need to make sure it has been configured for each VM.
I have added command to launch the docker registry service in Jenkinsfile, but if you want to launch it separately, here is the command to run, after you switch the current user to Jenkins:
docker run -d — restart=always — name registry -v “$HOME”/certs:/certs -e REGISTRY_HTTP_ADDR=0.0.0.0:444 -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key -p 444:444 registry:2Configure the Jenkins pipeline
I requested to be added into the list of hudson admins for Apache Jenkins. You can check the full list here: https://whimsy.apache.org/roster/group/hudson-jobadmin. Our project mentor should be able to assist any openwhisk committer to gain the access to Jenkins CI. When you gain the access, you will have option to “configure” for the OpenWhisk pipeline available at https://builds.apache.org/job/OpenWhisk-Playground/. This pipeline is kicked-off manually by setting the fork name, the repository name and the branch name.
All pipeline stages have been implemented in the script called Jenkinsfile, you can access here: https://github.com/apache/incubator-openwhisk/tree/master/tools/jenkins/Jenkinsfile.
OpenWhisk has dedicated ansible playbook variables for Jenkins CI to import. You can find them available under https://github.com/apache/incubator-openwhisk/tree/master/ansible/environments/jenkins. The file hosts.j2.ini is the template to generate the hosts file defining where each service is deployed and how it is accessed. The docker_registry is set to “openwhisk-vm1-he-de.apache.org:444/” in the file group_vars/all, which makes sure the docker images are pulled from the private docker registry service.
