Open in app

Sign in

Write

Sign in

How to install registry mirror for Docker Hub Rate Limiting

Thanaphat Nuangjumnong
6 min readDec 16, 2020

This session we talking about how to implement “Registry Mirror (Proxy cache)” used Nginx and Docker-compose

When the pods on Kubenates cluster pull image each container goes out to the internet and fetches an image it doesn’t have locally from dockerhub.io

So we can run a local registry mirror and point all cri-o (CNI) to registry mirror (Proxy cache)

I’m configuration Nginx (Port: 443) with SSL and proxy_pass http://127.0.0.1:5000 going to Docker registry (Port:5000) in my localhost

## install cfsslcurl -s -L -o /bin/cfssl https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
curl -s -L -o /bin/cfssljson https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
curl -s -L -o /bin/cfssl-certinfo https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x /bin/cfssl*
cfssl -h

## mkdir cert

mkdir -p /data/certs
cd /data/certs

## create ca-config

cat > ca-config.json <<EOL
{
  "signing": {
    "default": {
      "expiry": "876682h"
    },
    "profiles": {
      "registry": {
        "usages": ["signing", "key encipherment", "server auth", "client auth"],
        "expiry": "876682h"
      }
    }
  }
}
EOLcat > ca-csr.json <<EOL
{"CA": {
                "expiry": "876682h",
                "pathlen": 0
            },  "CN": "registry",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
  {
    "C": "TH",
    "L": "ABC",
    "O": "Registry",
    "OU": "ABC",
    "ST": "Bangkok"
  }
]
}
EOL

## gencert CA

cfssl gencert -initca ca-csr.json | cfssljson -bare ca
cp -p ca.pem ca.crt
cp -p ca-key.pem ca.key

## Check result

[root@prxhbpr01 certs]# ll
total 28
-rw-r--r-- 1 root root  236 Dec  3 17:03 ca-config.json
-rw-r--r-- 1 root root 1379 Dec  3 17:20 ca.crt
-rw-r--r-- 1 root root 1009 Dec  3 17:20 ca.csr
-rw-r--r-- 1 root root  290 Dec  3 17:20 ca-csr.json
-rw------- 1 root root 1679 Dec  3 17:20 ca.key
-rw------- 1 root root 1679 Dec  3 17:20 ca-key.pem
-rw-r--r-- 1 root root 1379 Dec  3 17:20 ca.pem

## create registry-csr

cat >registry-csr.json <<EOL
{
  "CN": "registry",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
  {
    "C": "TH",
    "L": "ABC",
    "O": "Registry",
    "OU": "ABC",
    "ST": "Bangkok"
  }
]
}
EOL

## cfssl gencert

#Change hostname to your hosts
cfssl gencert \
-ca=ca.pem \
-ca-key=ca-key.pem \
-config=ca-config.json \
-hostname=example-mirror-0.local,127.0.0.1,localhost \
-profile=registry registry-csr.json | \
cfssljson -bare registry

## re-check file

[root@prxhbpr01 certs]# pwd
/data/certs[root@prxhbpr01 certs]# ls -al
total 52
drwxr-xr-x 2 root root 4096 Dec  3 18:20 .
drwxr-xr-x 4 root root 4096 Dec  3 17:03 ..
-rw-r--r-- 1 root root  234 Dec  3 18:19 ca-config.json
-rw-r--r-- 1 root root 1306 Dec  3 18:20 ca.crt
-rw-r--r-- 1 root root 1045 Dec  3 18:20 ca.csr
-rw-r--r-- 1 root root  281 Dec  3 18:20 ca-csr.json
-rw------- 1 root root 1679 Dec  3 18:20 ca.key
-rw------- 1 root root 1679 Dec  3 18:20 ca-key.pem
-rw-r--r-- 1 root root 1306 Dec  3 18:20 ca.pem
-rw-r--r-- 1 root root 1139 Dec  3 18:20 registry.csr
-rw-r--r-- 1 root root  190 Dec  3 18:20 registry-csr.json
-rw------- 1 root root 1679 Dec  3 18:20 registry-key.pem
-rw-r--r-- 1 root root 1464 Dec  3 18:20 registry.pem

## change type of file

cp -rp registry.pem registry.crt
cp -rp registry-key.pem registry-key.key

## install docker

yum install docker -y
yum install docker-compose -y

## Create /etc/docker

mkdir /etc/docker

## Set up the Docker daemon

cat > /etc/docker/daemon.json <<EOF
{
  "bip": "192.168.0.1/24"
}
EOF

## Restart Docker

systemctl daemon-reload
systemctl restart docker

## mkdir

mkdir -p /data/registry

## make docker-compose file without ssl

vi docker-compose.yml
registry:
  restart: always
  image: registry:2
  ports:
    - 127.0.0.1:5000:5000
  #environment:
    #REGISTRY_HTTP_TLS_CERTIFICATE: /certs/registry.crt
    #REGISTRY_HTTP_TLS_KEY: /certs/registry-key.key
    #REGISTRY_AUTH: htpasswd
    #REGISTRY_AUTH_HTPASSWD_PATH: /auth/htpasswd
    #REGISTRY_AUTH_HTPASSWD_REALM: Registry Realm
  volumes:
    - /data/registry:/var/lib/registry
    #- /data/certs:/certs
    - /data/config.yml:/etc/docker/registry/config.yml
    #- /path/auth:/auth

##vi /data/config.yml

version: 0.1
log:
  fields:
    service: registry
storage:
  cache:
    blobdescriptor: inmemory
  filesystem:
    rootdirectory: /var/lib/registry
http:
  addr: 0.0.0.0:5000
  headers:
    X-Content-Type-Options: [nosniff]
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3
proxy:
  remoteurl: https://registry-1.docker.io
delete:
  enabled: true
maintenance:
  uploadpurging:
    enabled: true
    age: 24h
    interval: 1h
    dryrun: false
  readonly:
    enabled: false

## start docker-compose

cd /data
docker-compose up -d

## stop docker-compose

cd /data
docker-compose down

## verify

[root@prxhbpr01 data]# docker ps -aCONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                      PORTS               NAMES
72cb032e2eda        registry:2          "/entrypoint.sh /e..."   3 hours ago         Exited (2) 57 minutes ago                       data_registry_1

## check log

docker logs -f 72cb032e2eda

## make systemd ##vi /etc/systemd/system/proxy-harbor.service

[Unit]
Description=Docker Compose Application Service
Requires=docker.service
After=docker.service[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/data
ExecStart=/usr/bin/docker-compose up -d
ExecStop=/usr/bin/docker-compose down
TimeoutStartSec=0[Install]
WantedBy=multi-user.target

## Enable / Stop / Start / Start Service

systemctl enable proxy-harbor.service
systemctl start proxy-harbor.service
systemctl status proxy-harbor.service

How to install nginx with ssl and proxy_pass to docker-compose

yum install -y nginxcd /etc/nginx/conf.dvi proxy-harbor.conf
server {
  listen       443 ssl http2 default_server;
  listen       [::]:443 ssl http2 default_server;
  server_name  prxhbpr01;
  root         /usr/share/nginx/html;  ssl_certificate "/data/certs/registry.crt";
  ssl_certificate_key "/data/certs/registry-key.key";
  ssl_session_cache shared:SSL:1m;
  ssl_session_timeout  10m;
  ssl_ciphers HIGH:!aNULL:!MD5;
  ssl_prefer_server_ciphers on;  # Load configuration files for the default server block.
  include /etc/nginx/default.d/*.conf;        location / {
         proxy_set_header  X-Real-IP $remote_addr;
          proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header  Host $http_host;        proxy_pass http://127.0.0.1:5000;
        }        error_page 404 /404.html;
        location = /404.html {
        }        error_page 500 502 503 504 /50x.html;
        location = /50x.html {
        }
    }

## Enable / Stop / Start / Start Service

systemctl enable nginx.service
systemctl start nginx.service
systemctl status nginx.service

## How to config cri-o (CNI) using docker.mirror

vi /etc/containers/registries.confunqualified-search-registries = ["docker.io"]

[[registry]]
prefix = "example.com/foo"
insecure = false
blocked = false
location = "internal-registry-for-example.com/bar"

[[registry.mirror]]
location = "example-mirror-0.local/mirror-for-foo"

[[registry.mirror]]
location = "example-mirror-1.local/mirrors/foo"
insecure = true

## Test and Verify

root@k8swkscbvmts1:~# crictl pull nginx

Image is up to date for docker.io/library/nginx@sha256:6b1daa9462046581ac15be20277a7c75476283f969cb3a61c8725ec38d3b01c3

## Verify by nginx log on Docker-mirror Node

##crictl pull nginx
Image is up to date for docker.io/library/nginx@sha256:13e4551010728646aa7e1b1ac5313e04cf75d051fa441396832fcd6d600b5e71##tail -f /var/log/nginx/access.log192.19.163.65 - - [16/Dec/2020:16:53:30 +0700] "GET /v2/ HTTP/1.1" 200 2 "-" "cri-o/1.16.6 go/go1.10.1 os/linux arch/amd64" "-"
192.19.163.65 - - [16/Dec/2020:16:53:35 +0700] "GET /v2/library/nginx/manifests/sha256:13e4551010728646aa7e1b1ac5313e04cf75d051fa441396832fcd6d600b5e71 HTTP/1.1" 200 1362 "-" "cri-o/1.16.6 go/go1.10.1 os/linux arch/amd64" "-"
192.19.163.65 - - [16/Dec/2020:16:53:36 +0700] "GET /v2/library/nginx/manifests/latest HTTP/1.1" 200 1862 "-" "cri-o/1.16.6 go/go1.10.1 os/linux arch/amd64" "-"
192.19.163.65 - - [16/Dec/2020:16:53:39 +0700] "GET /v2/library/nginx/blobs/sha256:ae2feff98a0cc5095d97c6c283dcd33090770c76d63877caa99aefbbe4343bdd HTTP/1.1" 200 7481 "-" "cri-o/1.16.6 go/go1.10.1 os/linux arch/amd64" "-"

## if error please check on error.log

tail -f /var/log/nginx/error.log

## Persistance image on local storage

## ls -al /data/registry/docker/registry/v2/repositories

drwxr-xr-x 61 root root 4096 Dec 15 17:02 .
drwxr-xr-x  4 root root 4096 Dec  4 22:21 ..
drwxr-xr-x 11 root root 4096 Dec 15 16:38 alfresco
drwxr-xr-x  3 root root 4096 Dec 15 17:02 baseron01
drwxr-xr-x  8 root root 4096 Dec 15 15:56 bitnami
drwxr-xr-x  3 root root 4096 Dec 15 15:43 bondxp
drwxr-xr-x  3 root root 4096 Dec 14 16:15 bsdtrue
drwxr-xr-x  3 root root 4096 Dec 15 16:09 budsakol
drwxr-xr-x  6 root root 4096 Dec 15 15:35 buoyantio
drwxr-xr-x  3 root root 4096 Dec 15 14:34 byrnedo
drwxr-xr-x 10 root root 4096 Dec 15 15:18 bytekode
drwxr-xr-x  3 root root 4096 Dec 15 11:26 confluentinc
drwxr-xr-x  3 root root 4096 Dec 13 01:15 curlimages
drwxr-xr-x  3 root root 4096 Dec 15 11:28 dpage
drwxr-xr-x  4 root root 4096 Dec 15 15:14 drone
drwxr-xr-x  4 root root 4096 Dec 15 11:56 envoyproxy
drwxr-xr-x  3 root root 4096 Dec 15 16:26 fortio
drwxr-xr-x  3 root root 4096 Dec 15 09:54 frolvlad
drwxr-xr-x  3 root root 4096 Dec 13 19:16 gethue
drwxr-xr-x  3 root root 4096 Dec 15 15:33 grafana
drwxr-xr-x  3 root root 4096 Dec 14 15:44 greut
drwxr-xr-x  9 root root 4096 Dec 15 15:43 istio

Ref. https://docs.docker.com/registry/deploying/#deploy-your-registry-using-a-compose-file

Ref. https://github.com/containers/image/blob/master/docs/containers-registries.conf.5.md

Ref. https://docs.docker.com/registry/recipes/mirror/

Kubernetes
Docker

--

--

Thanaphat Nuangjumnong

Written by Thanaphat Nuangjumnong

19 Followers

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams