How to caching the docker images?

Question

I try to improve the performance of my pipeline stages in gitlab cicd. The Gitlab is self hosted and the Runner runs as docker with docker executor. I recognized, that for both jobs in my stages (test, build) they begin with pulling the images from docker hub:

Running with gitlab-runner 14.8.0 (565b6c0b)
  on Shared gitlab-runner on ARWS. 3yyScM1M
Preparing the "docker" executor 00:03
Using Docker executor with image python:3.9 ...
Pulling docker image python:3.9 ...
Using docker image sha256:4819be0df94257e1e31cd64dda12d46ff8b2180f9576ad9eaf98dcac9d70d9f9 for python:3.9 with digest python@sha256:57274f3fe86ad3d010025e9019bc41dcf9ee5833cfe4ba7b6773ef49672d855f ...
Preparing environment
.
.
.

This part seems to take most of the pipeline time. At the moment not much is happening in both jobs. I already started to implement caching in the pipelinbe:

image: python:3.9
stages:

test
build

variables:
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"
cache:
  key:
    files:
      - poetry.lock
      - .gitlab-ci.yml
    prefix: ${CI_JOB_NAME}
  paths:
    - .venv
    - .cache/pip
before_script:

python --version  # For debugging
pip install poetry
poetry config virtualenvs.in-project true
poetry install
source poetry env info --path/bin/activate

testing:
  tags: 
    - docker
    - linux
  stage: test
  script:
    - echo "Launching Job 'testing' in stage 'test'"
    - pytest -v --junitxml=report.xml
  artifacts:
    when: always
    reports:
      junit: /builds/$CI_PROJECT_PATH/report.xml
building:
  tags:
    - docker
    - linux
  stage: build
  script:
    - echo "Launching Job 'building' in stage 'build'"

And there is a huge possibility I do some stupid stuff. But the real question bothers me: Is it possible to cache the environment instead of pulling and preparing it at the start of every job?

score 0 · Answer 1 · answered Feb 26 '22 at 19:09

GitLab has some docs on how to speed up builds, and one of the suggestions does involve caching runner images.

https://docs.gitlab.com/runner/configuration/speed_up_job_execution.html#use-a-proxy-for-containers

Use a proxy for containers

You can speed up the time it takes to download Docker images by using:

The GitLab Dependency Proxy or
A mirror of the DockerHub Registry
Other open source solutions

There are other non-container related suggestions too, as well as information on how to use these up at that link.

You can also host your own runners, in which case you can run them on docker or just centos, etc. I assume, in that case, that you could already have the images available on the runner node to avoid new pulls. We run ours in kubernetes and are looking at harbor as an option to speed these things up too, but we haven't tested it yet. I imagine self-hosted k8s and harbor would be similar to GitLab using its own dependency proxy solution.

How to caching the docker images?

1 Answers1

Use a proxy for containers