Project Atomic

Running Kubernetes and Friends in Containers on CentOS Atomic Host

Thu, 15 Sep 2016 12:00:00 +0000

The atomic hosts from CentOS and Fedora earn their “atomic” namesake by providing for atomic, image-based system updates via rpm-ostree, and atomic, image-based application updates via docker containers.

This “system” vs “application” division isn’t set in stone, however. There’s room for system components to move across from the somewhat rigid world of ostree commits to the freer-flowing container side.

In particular, the key atomic host components involved in orchestrating containers across multiple hosts, such as flannel, etcd and kubernetes, could run instead in containers, making life simpler for those looking to test out newer or different versions of these components, or to swap them out for alternatives.

The devel tree of CentOS Atomic Host, which features a trimmed-down system image that leaves out kubernetes and related system components, is a great place to experiment with alternative methods of running these components, and swapping between them.

System Containers

Running system components in docker containers can be tricky, because these containers aren’t automatically integrated with systemd, like other system services, and because some components, such as flannel, need to modify docker configs, resulting in a bit of a chicken-and-egg situation.

One solution is system containers for atomic, which can be run independently from the docker daemon. Giuseppe Scrivano has built example containers for flannel and for etcd, and in this post, I’ll be using system containers to run flannel and etcd on my atomic hosts.

You need a very recent version of the atomic command. I used a pair of CentOS Atomic Hosts running the “continuous” stream.

The master host needs etcd and flannel:

# atomic install --system gscrivano/etcd

# systemctl start etcd

With etcd running, we can use it to configure flannel:

# runc exec etcd etcdctl set /atomic.io/network/config '{"Network":"172.17.0.0/16"}'

# atomic install --system gscrivano/flannel

# systemctl start flannel

The worker node needs flannel as well:

# export MASTER_IP=YOUR-MASTER-IP

# atomic install --system --set FLANNELD_ETCD_ENDPOINTS=http://$MASTER_IP:2379 gscrivano/flannel

# systemctl start flannel

On both the master and the worker, we need to make docker use flannel:

# echo "/usr/libexec/flannel/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker" | runc exec flannel bash

Also on both hosts, we need this docker tweak (because of this):

# cp /usr/lib/systemd/system/docker.service /etc/systemd/system/

# sed -i s/MountFlags=slave/MountFlags=/g /etc/systemd/system/docker.service

# systemctl daemon-reload

# systemctl restart docker

On both hosts, some context tweaks to make SELinux happy:

# mkdir -p /var/lib/kubelet/

# chcon -R -t svirt_sandbox_file_t /var/lib/kubelet/

# chcon -R -t svirt_sandbox_file_t /var/lib/docker/

Kube in Containers

With etcd and flannel in place, we can proceed with running kubernetes. We can run each of the kubernetes master and worker components in containers, using the rpms available in the CentOS repositories. Here I’m using containers built from these dockerfiles.

On the master

# export MASTER_IP=YOUR-MASTER-IP

# docker run -d --net=host jasonbrooks/kubernetes-apiserver:centos --admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ResourceQuota --address=0.0.0.0 --insecure-bind-address=0.0.0.0

# docker run -d --net=host --privileged jasonbrooks/kubernetes-controller-manager:centos

# docker run -d --net=host jasonbrooks/kubernetes-scheduler:centos

On the worker

# export WORKER_IP=YOUR-WORKER-IP

# atomic run --opt3="--master=http://$MASTER_IP:8080" jasonbrooks/kubernetes-proxy:centos

# atomic run --opt1="-v /etc/kubernetes/manifests:/etc/kubernetes/manifests:ro" --opt3="--address=$WORKER_IP --config=/etc/kubernetes/manifests --hostname_override=$WORKER_IP --api_servers=http://$MASTER_IP:8080 --cluster-dns=10.254.0.10 --cluster-domain=cluster.local" jasonbrooks/kubernetes-kubelet:centos

Get matching kubectl

I like to test things out from my master node. We can grab the version of the kubernetes command line client, kubectl, that matches our rpms by extracting it from the CentOS rpm.

# rpm2cpio http://mirror.centos.org/centos/7/extras/x86_64/Packages/kubernetes-client-1.2.0-0.13.gitec7364b.el7.x86_64.rpm | cpio -iv --to-stdout "./usr/bin/kubectl" > /usr/local/bin/kubectl && chmod +x /usr/local/bin/kubectl

We can then use kubectl to check on the status of our node(s):

# kubectl get nodes

NAME            STATUS    AGE
10.10.171.216   Ready     3h

DNS Addon

The guestbookgo sample app that I like to use for testing requires the dns addon, so I always ensure that it’s installed. Here I’m following the directions from the docker-multinode page in the kube-deploy repository:

# curl -O https://raw.githubusercontent.com/kubernetes/kube-deploy/master/docker-multinode/skydns.yaml

I skipped over setting up certificates for this cluster, so in order for the dns addon to work, the kube2sky container in the dns pod needs the argument --kube-master-url=http://$MASTER_IP:8080. Also, I’m changing the clusterIP to 10.254.0.10 to match the default from the rpms.

# vi skydns.yaml

...

- name: kube2sky
        image: gcr.io/google_containers/kube2sky-ARCH:1.15
        resources:
          limits:
            cpu: 100m
            memory: 50Mi
        args:
        # command = "/kube2sky"
        - --domain=cluster.local
		- --kube-master-url=http://$MASTER_IP:8080
...

spec:
  selector:
    k8s-app: kube-dns
  clusterIP: 10.254.0.10

Now to start up the dns addon:

# kubectl create namespace kube-system

# export ARCH=amd64

# sed -e "s/ARCH/${ARCH}/g;" skydns.yaml | kubectl create -f -

Test it

It takes a few minutes for all the containers to get up and running. Once they are, you can start running kubernetes apps. I typically test with the guestbookgo atomicapp:

# atomic run projectatomic/guestbookgo-atomicapp

Wait a few minutes, until kubectl get pods tells you that your guestbook and redis pods are running, and then:

# kubectl describe service guestbook | grep NodePort

Visiting the NodePort returned above at either my master or worker IP (these kube scripts configure both to serve as workers) gives me this:

A Newer Kube

The kubernetes rpms in the CentOS repositories are currently at version 1.2. To try out the current, 1.3 version of kubernetes, you can swap in some newer rpms, running in Fedora or Rawhide-based containers, or you can use the latest kubernetes containers provided by the upstream project.

Cleaning up

If you’ve already configured kubernetes using the above instructions, and want to start over with a different kubernetes version, you can blow your existing cluster away (while leaving the flannel and etcd pieces in place).

First, remove all the docker containers running on your master and node(s). On each machine, run:

# docker rm -f $(docker ps -a -q)

Run docker ps to see if any containers are left over, if there are, run the command above again until they’re all gone. Since we’re running flannel and etcd using runc, killing all our docker containers won’t affect our system containers.

Next, you can clear out the etcd keys associated with the cluster by running this on your master:

# runc exec -t etcd etcdctl rm --recursive /registry

Then, on your worker node, clean up the /var/lib/kubelet directory:

# rm -rf /var/lib/kubelet/*

## Containers from Rawhide

I mentioned that the CentOS repositories contain a v1.2 kubernetes. However, Fedora’s Rawhide includes v1.3-based kubernetes packages. howI wanted to try out the most recent rpm-packaged kubernetes, so I rebuilt the containers I used above swapping in fedora:rawhide for centos:centos7 in the FROM: lines of the dockerfiles.

You can run the same series of commands listed above, with the container tag changed from :centos to :rawhide.

For instance:

# docker run -d --net=host jasonbrooks/kubernetes-scheduler:centos --master=http://$MASTER_IP:8080

becomes:

# docker run -d --net=host jasonbrooks/kubernetes-scheduler:rawhide --master=http://$MASTER_IP:8080

Containers from Upstream

With flannel and etcd running in system containers, and with docker configured properly, we can start up kubernetes using the containers built by the upstream kubernetes project. I’ve pulled the following docker run commands from the docker-multinode scripts in the kubernetes project’s kube-deploy repository.

On the master:

# docker run -d \
    --net=host \
    --pid=host \
    --privileged \
    --restart="unless-stopped" \
    --name kube_kubelet_$(date | md5sum | cut -c-5) \
    -v /sys:/sys:rw \
    -v /var/run:/var/run:rw \
    -v /run:/run:rw \
    -v /var/lib/docker:/var/lib/docker:rw \
    -v /var/lib/kubelet:/var/lib/kubelet:shared \
    -v /var/log/containers:/var/log/containers:rw \
    gcr.io/google_containers/hyperkube-amd64:$(curl -sSL "https://storage.googleapis.com/kubernetes-release/release/stable.txt") \
    /hyperkube kubelet \
      --allow-privileged \
      --api-servers=http://localhost:8080 \
      --config=/etc/kubernetes/manifests-multi \
      --cluster-dns=10.0.0.10 \
      --cluster-domain=cluster.local \
      --hostname-override=${MASTER_IP} \
      --v=2

On the worker:

# export WORKER_IP=YOUR-WORKER-IP

# docker run -d \
    --net=host \
    --pid=host \
    --privileged \
    --restart="unless-stopped" \
    --name kube_kubelet_$(date | md5sum | cut -c-5) \
    -v /sys:/sys:rw \
    -v /var/run:/var/run:rw \
    -v /run:/run:rw \
    -v /var/lib/docker:/var/lib/docker:rw \
    -v /var/lib/kubelet:/var/lib/kubelet:shared \
    -v /var/log/containers:/var/log/containers:rw \
    gcr.io/google_containers/hyperkube-amd64:$(curl -sSL "https://storage.googleapis.com/kubernetes-release/release/stable.txt") \
    /hyperkube kubelet \
      --allow-privileged \
      --api-servers=http://${MASTER_IP}:8080 \
      --cluster-dns=10.0.0.10 \
      --cluster-domain=cluster.local \
      --hostname-override=${WORKER_IP} \
      --v=2

# docker run -d \
    --net=host \
    --privileged \
    --name kube_proxy_$(date | md5sum | cut -c-5) \
    --restart="unless-stopped" \
    gcr.io/google_containers/hyperkube-amd64:$(curl -sSL "https://storage.googleapis.com/kubernetes-release/release/stable.txt") \
    /hyperkube proxy \
        --master=http://${MASTER_IP}:8080 \
        --v=2

Get current kubectl

I usually test things out from the master node, so I’ll download the newest stable kubectl binary to there:

# curl -sSL https://storage.googleapis.com/kubernetes-release/release/$(curl -sSL "https://storage.googleapis.com/kubernetes-release/release/stable.txt")/bin/linux/amd64/kubectl > /usr/local/bin/kubectl

# chmod +x /usr/local/bin/kubectl

# kubectl get nodes

From here, you an scroll up to the subhead “Test it” to run the guestbookgo app. It’s not necessary to start up the dns addon manually with these upstream containers, because this configuration starts that addon automatically.

Introduction to System Containers

Mon, 12 Sep 2016 13:00:00 +0000

As part of our effort to reduce the number of packages that are shipped with the Atomic Host image, we faced the problem of how to containerize services that are needed before Docker itself is running. The result: “system containers,” a way to run containers in production using read only images.

System containers use different technologies such as OSTree for the storage, Skopeo to pull images from a registry, runC to run the containers and systemd to manage their life cycle.

To use system containers you must have Atomic CLI version 1.12 or later and the ostree utility installed. Currently, this means you must be running the CentOS Continuous Atomic, but updates for Fedora Atomic should be coming soon.

Pull an image

An image must be present in the OSTree system repository before we can use it as a system container. By using skopeo, the atomic tool can pull an image from different locations, a registry, the local Docker engine or a tarball, according to how the image is prefixed:

# atomic pull --storage=ostree gscrivano/etcd
Image gscrivano/etcd is being pulled to ostree ...
Pulling layer e4410b03d7db030dba502fef7bfd1dae56a6c48faae63a80fd82450322def2c5
Pulling layer 2176ad01d5670713218844201dc4edb36d2692fcc79ad7008003227a5f80097b
Pulling layer 9086967f25375e976260ad004a6ac3cc75ba020669042cb431904d2914ac1735
Pulling layer c0ee5e1cf412f1fd511aa1c7427c6fd825dfe4969d9ed7462ff8f989aceded7a
Pulling layer 024037bdea19132da059961b3ec58e2aff329fb2fe8ffd8030a65a27d7e7db5f

# atomic pull --storage=ostree dockertar:/tmp/etcd.tar
# atomic pull --storage=ostree docker:etcd

Each layer in the image is stored as a separate OSTree branch, this takes advantage of the layered model used by Docker images, since atomic pull will download only the layers that are not already available. All the images are stored into the OSTree system repository.

Using OSTree as storage has the advantage that if the same file is present in more layers, it will be stored only once, just like for container image layers. A container is installed through hardlinks, the storage is shared with the OSTree repository “hardlink farm”.

atomic images list shows the list of the available images:

# atomic images list
   REPOSITORY    TAG          IMAGE ID       CREATED            VIRTUAL SIZE   TYPE
gscrivano/etcd   latest       d7c1702506ff   2016-09-08 16:39                  system

atomic images delete deletes one tag and atomic images prune removes the unused layers:

# atomic images delete -f gscrivano/etcd
# atomic images prune
Deleting ociimage/9086967f25375e976260ad004a6ac3cc75ba020669042cb431904d2914ac1735
Deleting ociimage/2176ad01d5670713218844201dc4edb36d2692fcc79ad7008003227a5f80097b
Deleting ociimage/e4410b03d7db030dba502fef7bfd1dae56a6c48faae63a80fd82450322def2c5
Deleting ociimage/c0ee5e1cf412f1fd511aa1c7427c6fd825dfe4969d9ed7462ff8f989aceded7a
Deleting ociimage/024037bdea19132da059961b3ec58e2aff329fb2fe8ffd8030a65a27d7e7db5f

Installation

System images are installed with atomic install --system as:

# atomic install --system gscrivano/etcd
Extracting to /var/lib/containers/atomic/etcd.0
systemctl daemon-reload
systemd-tmpfiles --create /etc/tmpfiles.d/etcd.conf
systemctl enable etcd

# atomic install --system gscrivano/flannel
Extracting to /var/lib/containers/atomic/flannel.0
systemctl daemon-reload
systemd-tmpfiles --create /etc/tmpfiles.d/flannel.conf
systemctl enable flannel

# systemctl start etcd
# runc exec etcd etcdctl set /atomic.io/network/config '{"Network":"10.40.0.0/16"}'
# systemctl start flannel

The template mechanism allows us to configure settings for images. For example, we could use the following command to configure Flannel to use another Etcd endpoint instead of the default http://127.0.0.1:2379:

# atomic install --system --set ETCD_ENDPOINTS=http://192.168.122.2:2379 gscrivano/flannel

The atomic containers verb is used to see containers:

# atomic containers list -a
   CONTAINER ID IMAGE                COMMAND              CREATED          STATUS    RUNTIME
   etcd         gscrivano/etcd       /usr/bin/etcd-env.sh 2016-09-08 14:19 running   runc

Uninstallation

Similarly to atomic install, atomic uninstall is used to uninstall an installed system container.

# atomic uninstall etcd

Structure of a System Image

System images are Docker images with a few extra files that are exported as part of the image itself, under the directory ‘/exports’. In other words, an existing Dockerfile can be converted adding the configuration files needed to run it as a system container (which translate to an additional ADD [files] /exports directive in the Dockerfile).

These files are:

config.json.template - template for the OCI configuration file that will be used to launch the runC container.
manifest.json - used to define default values for configuration variables.
service.template - template unit file for systemD.
tmpfiles.template - template configuration file for systemd-tmpfiles.

Not all of them are necessary for every image.

All the files with a .template suffix are preprocessed and every variable in the form $VARIABLE or ${VARIABLE} is replaced with its value. This allows to define variables that are set at installation time (through the --set option) as we saw with the Flannel example. It is possible to set a default value for these settings using the manifest.json file of the system container image.

If any of these files are missing, atomic will provide a default one. For instance, if config.json.template is not included in the image, the default configuration will launch the run.sh script without any tty.

There are some variables that are always defined by the atomic tool, without the need for an user to specify them via --set. Of those, only RUN_DIRECTORY and STATE_DIRECTORY can be overriden with --set:

DESTDIR - path where the container is installed on the system
NAME - name of the container
EXEC_START - Start directive for the systemD unit file.
EXEC_STOP - Stop directive for the systemD unit file.
HOST_UID - uid of the user installing the container.
HOST_GID - gid of the user installing the container.
RUN_DIRECTORY - run directory. /run for system containers.
STATE_DIRECTORY - path to the storage directory. /var/lib/ for system containers.

We’re excited about the ability of system containers to greatly improve administration and infrastructure service delivery for Atomic clusters. Please give them a try and tell us what you think.

New CentOS Atomic Host with Package Layering Support

Tue, 30 Aug 2016 18:38:04 +0000

Last week, the CentOS Atomic SIG released an updated version of CentOS Atomic Host (tree version 7.20160818), featuring support for rpm-ostree package layering.

CentOS Atomic Host is available as a VirtualBox or libvirt-formatted Vagrant box; or as an installable ISO, qcow2, or Amazon Machine image. Check out the CentOS wiki for download links and installation instructions, or read on to learn more about what’s new in this release.

CentOS Atomic Host includes these core component versions:

docker-1.10.3-46.el7.centos.10.x86_64
kubernetes-1.2.0-0.13.gitec7364b.el7.x86_64
kernel-3.10.0-327.28.2.el7.x86_64
atomic-1.10.5-7.el7.x86_64
flannel-0.5.3-9.el7.x86_64
ostree-2016.7-2.atomic.el7.x86_64
etcd-2.3.7-2.el7.x86_64
cloud-init-0.7.5-10.el7.centos.1.x86_64

Package Layering

Using the command rpm-ostree pkg-add, it’s now possible to layer new packages into an installed image that persist across reboots and upgrades, a topic that Jonathan Lebon covered in some detail in a post last month.

For instance, if I wanted to install ansible on an atomic host:

# rpm-ostree pkg-add epel-release
# reboot
# rpm-ostree pkg-add ansible
# reboot
# ansible --version
ansible 2.1.1.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = Default w/o overrides

I first installed the epel-release package because ansible lives in EPEL. The intermediate reboot was required to boot into the new EPEL-i-fied tree. I could have instead added the repo file for EPEL in my /etc/yum.repos.d/ directory, and skipped the extra install and reboot operations. To learn about the work going on to make package layering more “live,” check out this issue.

There are limitations to package layering. For instance, I’ve written in the past about running oVirt’s guest agent (which is not part of the standard atomic host image) in a docker container. Package layering won’t work for this scenario, because installing packages which contain files owned by users other than root is currently not supported:

# rpm-ostree pkg-add ovirt-guest-agent-common
notice: pkg-add is a preview command and subject to change.

Downloading metadata: [================================================] 100%
Resolving dependencies... done
Will download: 3 packages (209.2 kB)

  Downloading from epel: [=============================================] 100%

  Downloading from base: [=============================================] 100%

Importing: [===================                                        ]  33%
error: Unpacking ovirt-guest-agent-common-1.0.12-3.el7.noarch: Non-root ownership currently unsupported: path "/var/log/ovirt-guest-agent" marked as ovirtagent:ovirtagent)

CentOS Atomic Host Alpha

While it’s not yet possible to pkg-add packages with files owned by users other than root on the current CentOS Atomic Host release, the host’s Alpha stream includes a newer version of rpm-ostree that works just fine with these sorts of packages.

Apart from its newer rpm-ostree version, the Alpha release of CentOS Atomic Host now features a much slimmer package list, as the project begins to move toward containerization or package layering for system components such as kubernetes, flannel, and etcd.

Project Atomic Docker Patches

Thu, 18 Aug 2016 12:00:00 +0000

Project Atomic’s version of the Docker-based container runtime has been carrying a series of patches on the upstream Docker project for a while now. Each time we carry a patch, it adds significant effort as we continue to track upstream, therefore we would prefer to never carry any patches. We always strive to get our patches upstream and do it in the open.

This post, and the accompanying document, will attempt to describe the patches we are currently carrying:

Explanation on types of patches.
Description of patches.
Links to GitHub discussions, and pull requests for upstreaming the patches to docker.

Some people have asserted that our docker repo is a fork of the upstream docker project.

What Does It Mean To Be a Fork?

I have been in open source for a long time, and my definition of a “fork” might be dated. I think of a “fork” as a hostile action taken by one group to get others to use and contribute to their version of an upstream project and ignore the “original” version. For example, LibreOffice forking off of OpenOffice or going way back Xorg forking off of Xfree86.

Nowadays, GitHub has changed the meaning. When a software repository exists on GitHub or a similar platform, everyone who wants to contribute has to hit the “fork” button, and start building their patches. As of this writing, Docker on GitHub has 9,860 forks, including ours. By this definition, however, all packages that distributions ship that include patches are forks. Red Hat ships the Linux Kernel, and I have not heard this referred to as a fork. But it would be considered a “fork” if you’re considering any upstream project shipped with patches a fork.

The Docker upstream even relies on Ubuntu carrying patches for AUFS that were never merged into the upstream kernel. Since Red Hat-based distributions don’t carry the AUFS patches, we contributed the support for Devicemapper, OverlayFS, and Btrfs backends, which are fully supported in the upstream kernel. This is what enterprise distributions should do: attempt to ship packages configured in a way that they can be supported for a long time.

At the end of the day, we continue to track the changes made to the upstream Docker Project and re-apply our patches to that project. We believe this is an important distinction to allow freedom in software to thrive while continually building stronger communities. It’s very different than a hostile fork that divides communities—we are still working very hard to maintain continuity around unified upstreams.

How Can I Find Out About Patches for a Particular Version of Docker?

All of the patches we ship are described in the README.md file on the appropriate branch of our docker repository. If you want to look at the patches for docker-1.12 you would look at the docker-1.12 branch.

You can then look on the docker patches list page for information about these patches.

What Kind of Patches does Project Atomic Include?

Here is a quick overview of the kinds of patches we carry, and then guidance on finding information on specific patches.

Upstream Fixes

The Docker Project upstream tends to fix issues in the next version of Docker. This means if a user finds an issue in docker-1.11 and we provide a fix for this to upstream, the patch gets merged in to the master branch, and it will probably not get back ported to docker-1.11.

Since Docker is releasing at such a rapid rate, they tell users to just install docker-1.12 when it is available. This is fine for people who want to be on the bleeding edge, but in a lot of cases the newer version of Docker comes with new issues along with the fixes.

For example, docker-1.11 split the docker daemon into three parts: docker daemon, containerd, and runc. We did not feel this was stable enough to ship to enterprise customers right when it came out, yet it had multiple fixes for the docker-1.10 version. Many users want to only get new fixes to their existing software and not have to re-certify their apps every two months.

Another issue with supporting stable software with rapidly changing dependencies is that developers on the stable projects must spend time ensuring that their product remains stable every time one of their dependencies is updated. This is an expensive process, dependencies end up being updated only infrequently. This causes us to “cherry-pick” fixes from upstream Docker and to ship these fixes on older versions so that we can get the benefits from the bug fixes without the cost of updating the entire dependency. This is the same approach we take in order to add capabilities to the Linux kernel, a practice that has proven to be very valuable to our users.

Proposed Patches for Upstream

We carry patches that we know our users require right now, but have not yet been merged into the upstream project. Every patch that we add to the Project Atomic repository also gets proposed to the upstream docker repository.

These sorts of patches remain on the Project Atomic repository briefly while they’re being considered upstream, or forever if the upstream community rejects them. If we don’t agree with upstream Docker and feel our users need these patches, we continue to carry them. In some cases we have worked out alternative solutions like building authorization plugins.

For example, users of RHEL images are not supposed to push these image onto public web sites. We wanted a way to prevent users from accidentally pushing RHEL based images to Docker Hub, so we originally created a patch to block the pushing. When authorization plugins were added we then created a plugin to protect users from pushing RHEL content to a public registry like Docker Hub, and no longer had to carry the custom patch.

Detailed List of Patches

Want to know more about specific patches? You can find the current table and list of patches on our new docker patches list page.

Vagrant Service Manager 1.3.0 Released

Tue, 16 Aug 2016 12:54:00 +0000

This version of vagrant-service-manager introduces support for displaying Kubernetes configuration information. This enable users to access the Kubernetes server that runs inside ADB virtual machine from their host machine.

This version also includes binary installation support for Kubernetes. This support is extended to users of the Red Hat Container Development Kit. For information about client binary installation, see the previous release announcement “Client Binary Installation Now Included in the ADB”.

The full list of features from this version are:

Configuration information for Kubernetes provided as part of the env command
Client binary installation support for Kubernetes added to the ADB
Client binary installation support for OpenShift, Kubernetes and Docker in the Red Hat Container Development Kit
Auto-detection of a previously downloaded oc executable binary on Windows operating systems
Unit and acceptance tests for the Kubernetes service
Option to enable Kubernetes from a Vagrantfile with the following command:

  config.servicemanager.services = 'kubernetes'

1. Install the kubernetes client binary

Run the following command to install the kubernetes binary, `kubectl`

$ vagrant service-manager install-cli kubernetes
# Binary now available at /home/budhram/.vagrant.d/data/service-manager/bin/kubernetes/1.2.0/kubectl
# run binary as:
# kubectl <command>
export PATH=/home/budhram/.vagrant.d/data/service-manager/bin/kubernetes/1.2.0:$PATH

# run following command to configure your shell:
# eval "$(VAGRANT_NO_COLOR=1 vagrant service-manager install-cli kubernetes | tr -d '\r')"

Run the following command to configure your shell

$ eval "$(VAGRANT_NO_COLOR=1 vagrant service-manager install-cli kubernetes | tr -d '\r')"

2. Enable access to the kubernetes server that runs inside of the ADB

Run the following command to display environment variable for kubernetes

$ vagrant service-manager env kubernetes
# Set the following environment variables to enable access to the
# kubernetes server running inside of the vagrant virtual machine:
export KUBECONFIG=/home/budhram/.vagrant.d/data/service-manager/kubeconfig

# run following command to configure your shell:
# eval "$(vagrant service-manager env kubernetes)"

Run the following command to configure your shell

eval "$(vagrant service-manager env kubernetes)"

For a full list of changes in version 1.3.0, see the release log.

Creating OCI configurations with the ocitools generate library

Tue, 09 Aug 2016 14:00:00 +0000

OCI runc is a cool new tool for running containers on Linux machines. It follows the OCI container runtime specification. As of docker-1.11 it is the main mechanism that docker uses for launching containers.

The really cool thing is that you can use runc without even using docker. First you create a rootfs on your disk: a directory that includes all of your software and usually follows the basic layout of /. There are several tools that can create a rootfs, including dnf or the atomic command. Once you have a rootfs, you need to create a config.json file which runc will read. config.json has all of the specifications for running a container, things like which namespaces to use, which capabilities to use in your container, and what is the pid 1 of your container. It is somewhat similar to the output of docker inspect.

Creating and editing the config.json is not for the faint of heart, so we developed a command line tool called ocitools generate that can do the hard work of creating the config.json file.

Creating OCI Configurations

This post will guide you through the steps of creating OCI configurations using the ocitools generate library for the go programming language.

There are four steps to create an OCI configuration using the ocitools generate library:

Import the ocitools generate library into your project;
Create an OCI specification generator;
Modify the specification by calling different methods of the specification generator;
Save the specification.

Overview

The ocitools generate library defines a struct type, Generator, to enclose a pointer to an OCI Spec. Different methods are defined on the Generator type to allow the user to customize the specification pointed by the Generator. Once a Generator object is created, different fields of the specification can be modified by calling the corresponding methods of Generator. After finishing modifying the specification, Generator.Save can be called to save the specification into a local file.

Create an OCI configurations using the ocitools generate library

Step 1: import the ocitools generate library in your go project:

import "github.com/opencontainers/ocitools/generate"

Step 2: create a specification Generator:

specgen := generate.New()

generate.New creates a spec Generator with the default spec. You can also create a spec Generator with the spec from a local file using generate.NewFromFile:

specgen := generate.NewFromFile("/data/myspec.json")

Step 3: modify the specification:

specgen.SetRootPath("rootfs")
specgen.SetProcessGID(1000)
specgen.SetProcessTerminal(true)
specgen.SetLinuxResourcesCPUShares(512)
specgen.SetupPrivileged(false)

specgen.ClearAnnotations()
specgen.AddAnnotation("owner", "hmeng")
specgen.RemoveAnnotation("createdat")

specgen.AddLinuxUIDMapping(0, 1000, 50)
specgen.AddPreStartHook("/tmp/install.sh", []string{"--mode", "silent"})
specgen.AddTmpfsMount("/tmp", "ro")
specgen.AddCgroupsMount("rw")
specgen.AddBindMount("/home/test/file1", "/file2", "rw")
specgen.DropProcessCapability("audit_read")

specgen.AddOrReplaceLinuxNamespace("pid", "/proc/28341/ns/pid")

For the fields of the OCI spec which have basic types, such as numbers, strings, and booleans, the methods modifying them are named in the format of SetFieldName. A good example is SetRootPath.

For the fields whose types are slices and maps, the library provides the following three categories of methods: ClearFieldName clears the fields, AddFieldName adds a new data object into the field, and RemoveFieldName removes an existing data object from the field. A good example is ClearAnnotations, AddAnnotation, and RemoveAnnotation.

There are exceptions. For example, Spec.Process.Args is a slice of string, however, the library only provides a single method SetProcessArgs to set the process args, because it makes more sense to set the process args in a bundle than adding each argument incrementally.

Step 4: save the specification:

specgen.SaveToFile("/data/runc/busybox/config.json")

And you’re done! You now have a generator for creating spec files for your runc containers.

Download and Get Involved with Fedora Atomic 24

Fri, 29 Jul 2016 07:00:00 +0000

This week, the Fedora Project released updated images for its Fedora 24-based Atomic Host. Fedora Atomic Host is a leading-edge operating system designed around Kubernetes and Docker containers.

Fedora Atomic Host images are updated roughly every two weeks, rather than on the main six-month Fedora cadence. Because development is moving quickly, only the latest major Fedora release is supported.

Note: Due to an issue with the image-building process, the current Fedora Atomic Host images include an older version of the system tree. Be sure to atomic host upgrade to get the latest set of components. The next two-week media refresh will include an up-to-date tree.

Fedora Atomic Host includes these core component versions:

kernel-4.6.4-301.fc24.x86_64
docker-1.10.3-24.git29066b4.fc24.x86_64
kubernetes-1.2.0-0.24.git4a3f9c5.fc24.x86_64
atomic-1.10.5-1.gitce09e40.fc24.x86_64
rpm-ostree-2016.4-2.fc24.x86_64
flannel-0.5.5-6.fc24.x86_64
etcd-2.2.5-5.fc24.x86_64
cloud-init-0.7.6-8.20150813bzr1137.fc24.noarch

Upgrading

Upgrading from an existing Atomic Host to Fedora Atomic 24 involves replacing the Fedora 23-based fedora-atomic remote with the current one, and then rebasing on the new tree. Due to this issue, it may be necessary to put SELinux into permissive mode for the rebase operation:

$ sudo setenforce 0
$ sudo ostree remote delete fedora-atomic
$ sudo ostree remote add fedora-atomic --set=gpg-verify=false https://dl.fedoraproject.org/pub/fedora/linux/atomic/24
$ sudo rpm-ostree rebase fedora-atomic:fedora-atomic/24/x86_64/docker-host
$ sudo reboot

Atomic Images

Fedora Atomic Host is available as a virtualbox or libvirt vagrant image, as an installable iso image, as a raw or qcow2-formatted cloud image, or as an Amazon AMI.

To bring up Fedora Atomic Host in a vagrant box, issue a command like:

vagrant init fedora/24-atomic-host && vagrant up

If you’ve previously used vagrant to run a Fedora Atomic 24 VM, first run vagrant box update --box=fedora/24-atomic-host to ensure that you have the latest version.

Note: Due to this issue, you’ll need to add a line to your Vagrantfile like config.vm.synced_folder "./", "/vagrant", disabled: 'true' to disable folder sync.

Fedora Atomic Host is available as a qcow2 or raw-formatted image, both of which require a cloud-init data source, be it from your cloud or virtualization provider, or from a local source.

The Fedora Project maintains Atomic Host images for Amazon EC2 in both GP2 (SSD-based) and standard formats. Check out the atomic host download page for AMI IDs specific to your desired region.

There’s also an anaconda-based ISO installer for use with bare metal or as an alternative to configuring cloud-init for virtual machines.

Get Involved

To get involved with Fedora Atomic Host, get in touch with the Fedora Cloud SIG. The SIG meets each week on Wednesdays at 17:00 UTC in the #fedora-meetings-1 channel, and hangs out in the #fedora-cloud channel and on the Fedora Cloud mailing list.

One of the best ways help out with Fedora Atomic is to participate in testing core atomic host components using Fedora’s Bodhi. Following this link will provide a list of key atomic packages currently in need of testing for Fedora 24.

The Fedora Project maintains a version of the Fedora Atomic system tree that includes packages from the updates-testing repo. Rebasing an atomic host to this tree is a handy way to run the latest packages in need of testing:

$ sudo rpm-ostree rebase fedora-atomic:fedora-atomic/24/x86_64/testing/docker-host
$ sudo systemctl reboot

If you have questions about how best to test one of these packages, ask on the Fedora Cloud mailing list or in the #fedora-cloud in irc.

Atomic App 0.6.2 released with new index CLI command

Wed, 27 Jul 2016 16:55:00 +0000

This release of Atomic App introduces the new atomicapp index command.

We add this command in order to give a quick overview of all available featured and tested Nuleculized applications on github.com/projectatomic/nulecule-library. The ability to generate your own list is available as well with the atomicapp index generate command.

The main features of this release are:

Addition of the atomicapp index command
Correct file permissions are now when extracting Nuleculized containers
OpenShift connection issue bugfix

`atomicapp index`

This release adds the addition of the atomicapp index command. By using the atomicapp index list command, Atomic App will retrieve a container containing a valid index.yml and output all available Nulecule containers. This index can also be updated by using atomicapp index update.

atomicapp index list

Outputs the list of available containers located at ~/.atomicapp/index.yml.

▶ atomicapp index list
INFO   :: Atomic App: 0.6.2 - Mode: Index
ID                        VER      PROVIDERS  LOCATION                                             
postgresql-atomicapp      1.0.0    {D,O,K}    docker.io/projectatomic/postgresql-centos7-atomicapp 
flask_redis_nulecule      0.0.1    {D,K}      docker.io/projectatomic/flask-redis-centos7-atomicapp
redis-atomicapp           0.0.1    {D,O,K}    docker.io/projectatomic/redis-centos7-atomicapp      
gocounter                 0.0.1    {D,K}      docker.io/projectatomic/gocounter-scratch-atomicapp  
mariadb-atomicapp         1.0.0    {D,O,K}    docker.io/projectatomic/mariadb-centos7-atomicapp    
helloapache-app           0.0.1    {D,K,M}    docker.io/projectatomic/helloapache                  
mongodb-atomicapp         1.0.0    {D,O,K}    docker.io/projectatomic/mongodb-centos7-atomicapp    
etherpad-app              0.0.1    {D,O,K}    docker.io/projectatomic/etherpad-centos7-atomicapp   
apache-centos7-atomicapp  0.0.1    {D,K,M}    docker.io/projectatomic/apache-centos7-atomicapp     
wordpress-atomicapp       2.0.0    {D,O,K}    docker.io/projectatomic/wordpress-centos7-atomicapp  
skydns-atomicapp          0.0.1    {K}        docker.io/projectatomic/skydns-atomicapp             
guestbookgo-atomicapp     0.0.1    {O,K}      docker.io/projectatomic/guestbookgo-atomicapp        
mariadb-app               0.0.1    {D,K}      docker.io/projectatomic/mariadb-fedora-atomicapp     
gitlab-atomicapp          1.2.0    {D,K}      docker.io/projectatomic/gitlab-centos7-atomicapp

atomicapp index update

Updates the index.yml file.

▶ atomicapp index update
INFO   :: Atomic App: 0.6.2 - Mode: Index
INFO   :: Updating the index list
INFO   :: Pulling latest index image...
INFO   :: Skipping pulling docker image: projectatomic/nulecule-library
INFO   :: Copying files from image projectatomic/nulecule-library:/index.yaml to /home/wikus/.atomicapp/index.yaml
INFO   :: Index updated

atomicapp index generate

Generates a valid index.yml file to use in listing all available containers.

▶ atomicapp index generate ./nulecule-library
INFO   :: Atomic App: 0.6.1 - Mode: Index
INFO   :: Generating index.yaml from ./nulecule-library
INFO   :: index.yaml generated

Want to get started using Atomic App? Have a look at our extensive start guide, or use Atomic App as part of the Atomic CLI on an Atomic Host.

For a full list of changes between 0.6.1 and the 0.6.2 please see the commit log.

Working with Containers' Images Made Easy Part 1: skopeo

Mon, 25 Jul 2016 08:58:20 +0000

This is the first part of a series of posts about containers’ images. In this first part we’re going to focus on skopeo.

Back in March, I published a post about skopeo, a new tiny binary to help people interact with Docker registries. Its job has been limited to inspect (skopeo is greek for looking for, observe) images on remote registries as opposed to docker inspect, which is working for locally pulled images.

The Tool

Since then, we’ve been adding more features to skopeo, such as downloading image layers (via skopeo layers), and eventually we came up with a nice abstraction to the problem of downloading, inspecting, and uploading images (without even the need to have Docker installed on the system). We called this new abstraction copy and here is a straightforward example about how to use it:

# skopeo copy <source> <destination>

$ id -u
1000

 # let's say we want to download the Fedora 24 docker image from the Docker Hub and store it on a local directory
$ mkdir fedora-24
$ skopeo copy docker://fedora:24 dir:fedora-24
$ tree fedora-24
fedora-24
├── 7c91a140e7a1025c3bc3aace4c80c0d9933ac4ee24b8630a6b0b5d8b9ce6b9d4.tar
├── f9873d530588316311ac1d3d15e95487b947f5d8b560e72bdd6eb73a7831b2c4.tar
└── manifest.json

0 directories, 3 files

You can see from the output above that skopeo copy successfully downloaded the Fedora 24 image—as in, it downloaded all its layers plus the image manifest.

You can also notice the whole operation has been done with an unprivileged user—while Docker needs you to be root to even do a docker pull.

What can you do with that fedora-24 directory now? There comes the fun. A nice new addition to the atomic tool has been the so-called system containers—they are containers meant to be working before the Docker daemon comes up and they’re powered by the community project runc which is part of the Open Container Initiative. Basically, those containers are set up with these steps:

Download the image layers and manifest with skopeo
Setup a new ostree repository which will be the root filesystem of our system container
Import downloaded layers in the order they come in the image manifest
Create a systemd unit file to run runc with said filesystem
Spawn the service
Enjoy

If the above sounds rather complex, the atomic tool already provides a --system flag which can be used to create a system container:

$ sudo atomic install --system --name system-container gscrivano/spc-helloworld
Missing layer b037963d9b4419ffe09694c450bd33a06d24945416109aeb2937c7a8595252d9
Missing layer a1b129b466881845cdf628321bf7ed597b3d0cad0b8dd01564f78a4417c750fe
Missing layer a8a1c0600345270e055477e8f282d1318f0cef0debaed032cd1ba1e20eb2a35e
Missing layer 236608c7b546e2f4e7223526c74fc71470ba06d46ec82aeb402e704bfdee02a2
Extracting to /var/lib/containers/atomic/spc.0
systemctl enable spc
Created symlink from /etc/systemd/system/multi-user.target.wants/spc.service to /etc/systemd/system/spc.service.
systemctl start spc

# verify the service is running smoothly
$ sudo systemctl status spc  
● spc.service - Hello World System Container
   Loaded: loaded (/etc/systemd/system/spc.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2016-07-22 09:27:47 CEST; 5s ago
 Main PID: 10405 (runc)
    Tasks: 10 (limit: 512)
   CGroup: /system.slice/spc.service
           ├─10405 /bin/runc start spc
           └─spc
             ├─10416 /bin/sh /usr/bin/run.sh
             └─10435 nc -k -l 8081 --sh-exec /usr/bin/greet.sh

Jul 22 09:27:47 localhost.localdomain systemd[1]: Started Hello World System Container.

# we know our system container is listening on port 8081 so let's test it out!
$ nc localhost 8081                 
HTTP/1.1 200 OK
Connection: Close

Hi World

The new skopeo copy command isn’t just limited to download to local directories. Instead, it can do almost all sort of download/upload between:

docker registries -> local diretories
docker registries -> docker registries (skopeo copy docker://myimage docker://anotherrepo/myimage)
docker registries -> Atomic registry (atomic: prefix)
docker registries -> OCI image-layout directories (oci: prefix)
and viceversa in any combination!

Possibilities seems endless, being able to pull as an unprivileged user also open the doors to work with unprivileged sanboxes like Flatpak, bubblewrap.

As part of this unprileged capability, we’re working on a new feature in the Atomic tool, which will be able to pull images to the calling user home directory and later run any containers from those images (by also remapping files ownership to the one of the calling user).

Supporting the Open Container Initiative by early implementing the image specification had also great advantages as we eventually helped the specification itself where things weren’t totally clear and defined. We’re continuosly adding wider support as the specification moves along also in areas like images signing and layers federation.

The next post will take care of explaining how we extracted some core components from skopeo and moved them to a set of reusable libraries.

Client Binary Installation Now Included in the ADB

Fri, 22 Jul 2016 09:56:00 +0000

As part of the effort to continually improve the developer experience and make getting started easier, the ADB now supports client binary downloads. These downloads are facilitated by a new feature in ‘vagrant-service-manger’, the install-cli command.

The vagrant-service-manager plugin enables easier access to the features and services provided by the Atomic Developer Bundle (ADB). More information can be found in the README of ‘vagrant-service-manager’ repo.

The install-cli command was released as part of ‘vagrant-service-manager’ version 1.2.0. This command installs the client binary for services provided by the ADB. Today it can download client binaries for docker and OpenShift. This feature allows developers to know they have the best client for use with the ADB services they are using.

To use the ‘install-cli’ command, you must have version 1.2.0 or later of ‘vagrant-service-manager’ installed. You can verify the version you have installed with the following command:

$ vagrant plugin list

You can install the plugin with the following command:

$ vagrant plugin install vagrant-service-manager

The usage of the ‘install-cli’ command is very straightforward:

vagrant service-manager install-cli [service]

Where 'service' can be 'docker' and 'openshift'.

Here is how you can use the command. In this example, we will use an ADB set up and running OpenShift Origin. To setup the ADB and start OpenShift, use the following commands:

$ mkdir adb-openshift
$ cd adb-openshift
$ curl -o Vagrantfile https://raw.githubusercontent.com/projectatomic/adb-atomic-developer-bundle/master/components/centos/centos-openshift-setup/Vagrantfile
$ vagrant up

vagrant up will take a few minutes (and longer in the case of a slow network connection) to finish as it has to download the OpenShift Origin images from the Docker Hub. Once everything is ready you will see the information about how to access OpenShift Origin.

Now, the OpenShift origin server is ready and you may need a client to access it and perform your desired operations. You can manually download the client binary from OpenShift repository but we recommend you to use ‘install-cli’ command provided.

To get started, let’s review the help:

$ vagrant service-manager install-cli --help

Now, if you want to install the OpenShift client binary, ‘oc’, run the following command:

$ vagrant service-manager install-cli openshift
# Binary now available at /home/budhram/.vagrant.d/data/service-manager/bin/openshift/1.1.1/oc
# run binary as:
# oc <command>
export PATH=/home/budhram/.vagrant.d/data/service-manager/bin/openshift/1.1.1:$PATH

# run following command to configure your shell:
# eval "$(VAGRANT_NO_COLOR=1 vagrant service-manager install-cli openshift | tr -d '\r')"

Now, the binary has been downloaded and made available in the listed directory. You can configure your shell with the command mentioned in the output. This will make sure that the ‘oc’ binary is in your executable path.

You can verify everything worked by running the oc version command as is shown below:

$ oc version
oc v1.1.1
kubernetes v1.1.0-origin-1107-g4c8e6f4

Great!

Now the OpenShift binary client has been set up and you can play around with it. You may wish to watch OpenShift Origin quickstart with Atomic Developer Bundle as a next step.

Project Atomic

Running Kubernetes and Friends in Containers on CentOS Atomic Host

System Containers

Kube in Containers

On the master

On the worker

Get matching kubectl

DNS Addon

Test it

A Newer Kube

Cleaning up

Containers from Upstream

Get current kubectl

Introduction to System Containers

Pull an image

Installation

Uninstallation

Structure of a System Image

New CentOS Atomic Host with Package Layering Support

Package Layering

CentOS Atomic Host Alpha

Project Atomic Docker Patches

What Does It Mean To Be a Fork?

How Can I Find Out About Patches for a Particular Version of Docker?

What Kind of Patches does Project Atomic Include?

Upstream Fixes

Proposed Patches for Upstream

Detailed List of Patches

Vagrant Service Manager 1.3.0 Released

1. Install the kubernetes client binary

Run the following command to install the kubernetes binary, kubectl

Run the following command to configure your shell

2. Enable access to the kubernetes server that runs inside of the ADB

Run the following command to display environment variable for kubernetes

Run the following command to configure your shell

Creating OCI configurations with the ocitools generate library

Creating OCI Configurations

Overview

Create an OCI configurations using the ocitools generate library

Download and Get Involved with Fedora Atomic 24

Upgrading

Atomic Images

Get Involved

Atomic App 0.6.2 released with new index CLI command

atomicapp index

Working with Containers' Images Made Easy Part 1: skopeo

The Tool

Client Binary Installation Now Included in the ADB

Run the following command to install the kubernetes binary, `kubectl`

`atomicapp index`