DocX Github contribution chart
DocX Github Stats
DocX Most Used Languages

Activity

28 Sep 2022

Issue Comment

DocX

Cache miss when using the image that has been built from the another cache

Hello,

I'm using Docker version 20.10.17, build 100c701 on WSL2 with Ubuntu (5.10.102.1-microsoft-standard-WSL2 #1 SMP Wed Mar 2 00:30:59 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux).

I encountered a strange behavior that the cache miss occurs when using the image that has been built from the another cache.

Steps to reproduce:

  • Run a Docker registry for storing images.
  • Build a fresh Docker image (a first one) and push it to the registry.
  • Change Docker context to cause a Docker cache miss in the only one layer.
  • Build a new Docker image (a second one) using cache from the first one and push it to the registry. Only one (the latest) layer has to be rebuilt.
  • Change Docker context to cause a Docker cache miss in the only one layer.
  • Build a new Docker image (a third one) using cache from the second one and push it to the registry. Only one (the latest) layer has to be rebuilt. But it rebuilds the whole image - all layers.

To simplify the issue explanation I've created a small repository with the reproduce.sh script - https://github.com/AlexPykavy/docker-cache-miss-issue.

Checkout the output:

+ echo '************************* Starting Docker registry *************************'
************************* Starting Docker registry *************************
++ pwd
+ registry_mount=/mnt/c/Users/Alexander.Pykavy/Workspace/Playground/docker-cache-experiments/registry
+ docker run -d -p 5000:5000 -v /mnt/c/Users/Alexander.Pykavy/Workspace/Playground/docker-cache-experiments/registry:/var/lib/registry --restart=always --name registry registry:2
7bf2139bb35502398292069fda069b4c76c0705ac2263adb5922ff140f3722cf
+ echo '************************* Building initial image *************************'
************************* Building initial image *************************
+ docker system prune -af
Deleted Images:
untagged: localhost:5000/cache-experiments:3
deleted: sha256:33b4ff8cd367e2483ae4dc1853858d34050b95d52ee7a6cdaabb20b0c76caab3

Deleted build cache objects:
tgzhc2zqjf2wfnqb7tzhgz85t
pedx98is8tj34g129qgpz98ye
q9ub8btj54my6iq3k1pkyk49m
tv961itpyh47y12k0bhkrku2p
bxbfz16zl8vdh0hgludpzvrvk
to93frbzk2em5axqi3b07ew6n
vmyksjeas1dsr18o549egwokv
9rjdjsmbyn79cneaj4rs3ltxg
eciei23i0azt4qkisuv3qgqs3
v77r0jl3o37dthnly80vmrarp
zkdr46ir26lba3vnarjeomgba
n0i6d7kkgdfo0cjkv84sdw3m6

Total reclaimed space: 12MB
+ docker images -a
REPOSITORY   TAG       IMAGE ID       CREATED       SIZE
registry     2         d1fe2eaf6101   2 weeks ago   24.1MB
+ docker build -t localhost:5000/cache-experiments:1 --build-arg BUILDKIT_INLINE_CACHE=1 .
[+] Building 95.6s (11/11) FINISHED
 => [internal] load build definition from Dockerfile                                                                                           0.0s
 => => transferring dockerfile: 229B                                                                                                           0.0s
 => [internal] load .dockerignore                                                                                                              0.0s 
 => => transferring context: 2B                                                                                                                0.0s 
 => [internal] load metadata for docker.io/library/python:3.10.5-slim-bullseye                                                                88.1s 
 => [internal] load build context                                                                                                              0.0s
 => => transferring context: 368B                                                                                                              0.0s 
 => [1/5] FROM docker.io/library/python:3.10.5-slim-bullseye@sha256:f9f03f46267e182193544299504687e711c623e2a085323138f94ed9b01ce641           3.9s 
 => => resolve docker.io/library/python:3.10.5-slim-bullseye@sha256:f9f03f46267e182193544299504687e711c623e2a085323138f94ed9b01ce641           0.0s 
 => => sha256:126aaa6587b8608b62fa39019a891e98460e23bad2d04fb235c4dc8fae2adf3c 1.08MB / 1.08MB                                                 0.7s
 => => sha256:5a96a0b5efbdf4efe9ff1e6aeffa495d7148e0366b5c9958a625329f4beea3f5 11.77MB / 11.77MB                                               1.1s 
 => => sha256:f9f03f46267e182193544299504687e711c623e2a085323138f94ed9b01ce641 1.86kB / 1.86kB                                                 0.0s 
 => => sha256:bdf0079de4094afdb26b94d9f89b716499436282c972461d945a87899c015c23 1.37kB / 1.37kB                                                 0.0s 
 => => sha256:5b7d04484c737c4a2af7f33a85470dab369e922a570411f40ebd72d35167bc49 7.49kB / 7.49kB                                                 0.0s
 => => sha256:1efc276f4ff952c055dea726cfc96ec6a4fdb8b62d9eed816bd2b788f2860ad7 31.37MB / 31.37MB                                               1.5s 
 => => sha256:65462f2b00a4c62e8aa87511a8185d447d6a1d413afe42e905072da4d47f7445 233B / 233B                                                     0.9s 
 => => sha256:098c9e455557d7de690b40c84f2c1afed09d362d7ede72bd32f815ddd21b3c83 3.18MB / 3.18MB                                                 1.4s 
 => => extracting sha256:1efc276f4ff952c055dea726cfc96ec6a4fdb8b62d9eed816bd2b788f2860ad7                                                      1.2s 
 => => extracting sha256:126aaa6587b8608b62fa39019a891e98460e23bad2d04fb235c4dc8fae2adf3c                                                      0.1s 
 => => extracting sha256:5a96a0b5efbdf4efe9ff1e6aeffa495d7148e0366b5c9958a625329f4beea3f5                                                      0.4s 
 => => extracting sha256:65462f2b00a4c62e8aa87511a8185d447d6a1d413afe42e905072da4d47f7445                                                      0.0s 
 => => extracting sha256:098c9e455557d7de690b40c84f2c1afed09d362d7ede72bd32f815ddd21b3c83                                                      0.2s 
 => [2/5] WORKDIR /app                                                                                                                         0.2s 
 => [3/5] COPY requirements.txt ./                                                                                                             0.0s 
 => [4/5] RUN pip install -r requirements.txt                                                                                                  3.1s 
 => [5/5] COPY src ./                                                                                                                          0.0s 
 => exporting to image                                                                                                                         0.0s 
 => => exporting layers                                                                                                                        0.0s 
 => => writing image sha256:e5871e1a72635ddcd169ce3930e91647adedc36cd9c561e308b548fd0a89b8d4                                                   0.0s 
 => => naming to localhost:5000/cache-experiments:1                                                                                            0.0s 
 => exporting cache                                                                                                                            0.0s 
 => => preparing build cache for export                                                                                                        0.0s 

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
+ docker push localhost:5000/cache-experiments:1
The push refers to repository [localhost:5000/cache-experiments]
6bc1a8c56437: Pushed
f30a8f1d517a: Pushed 
10ed798bb875: Pushed
e78c457618a4: Pushed
b4573ffeefec: Pushed
5ebe2abcebb0: Pushed
7287cceb8eb6: Pushed
e7a38ce55a0d: Pushed
92a4e8a3140f: Pushed
1: digest: sha256:9250525ab23e71a3f8ba223555624a54374876f5d60cff98c0ce66f76e4f92a5 size: 2202
+ echo '************************* Changing the Docker context *************************'
************************* Changing the Docker context *************************
+ echo import os
+ echo '************************* Building the second image *************************'
************************* Building the second image *************************
+ docker system prune -af
Deleted Images:
untagged: localhost:5000/cache-experiments:1
untagged: localhost:5000/cache-experiments@sha256:9250525ab23e71a3f8ba223555624a54374876f5d60cff98c0ce66f76e4f92a5
deleted: sha256:e5871e1a72635ddcd169ce3930e91647adedc36cd9c561e308b548fd0a89b8d4

Deleted build cache objects:
3ik09luezfct8zraputleixyj
ssv1ygz4aga5222a66h9fhc8z
komliwdehakp13kawekurkdny
i3epw8wwi4h9abkdf5p06rd3a
rvg3z6021s9baf4abfn98pxjf
mps7fpoaodb6ao3ddtoea50de
z34qhtdiklae00o9nlu6czxt0
wmq6bk2e6knkaopy0c094kg7z
kuqsy5xpwd9z037r58fy0kp5b
t812pl4jyzhx1mt8nox4h244w
oz1c1sojyg0cftpvz0606an9r
jp29eb5sfqzq5uvd09bqly6a0

Total reclaimed space: 12MB
+ docker images -a
REPOSITORY   TAG       IMAGE ID       CREATED       SIZE
registry     2         d1fe2eaf6101   2 weeks ago   24.1MB
+ docker build -t localhost:5000/cache-experiments:2 --cache-from localhost:5000/cache-experiments:1 --build-arg BUILDKIT_INLINE_CACHE=1 .
[+] Building 47.3s (12/12) FINISHED
 => [internal] load build definition from Dockerfile                                                                                           0.0s
 => => transferring dockerfile: 229B                                                                                                           0.0s 
 => [internal] load .dockerignore                                                                                                              0.0s
 => => transferring context: 2B                                                                                                                0.0s 
 => [internal] load metadata for docker.io/library/python:3.10.5-slim-bullseye                                                                43.7s 
 => importing cache manifest from localhost:5000/cache-experiments:1                                                                           0.0s
 => [internal] load build context                                                                                                              0.0s 
 => => transferring context: 418B                                                                                                              0.0s 
 => [1/5] FROM docker.io/library/python:3.10.5-slim-bullseye@sha256:f9f03f46267e182193544299504687e711c623e2a085323138f94ed9b01ce641           0.0s
 => CACHED [2/5] WORKDIR /app                                                                                                                  0.0s 
 => CACHED [3/5] COPY requirements.txt ./                                                                                                      0.0s 
 => CACHED [4/5] RUN pip install -r requirements.txt                                                                                           3.1s 
 => => pulling sha256:126aaa6587b8608b62fa39019a891e98460e23bad2d04fb235c4dc8fae2adf3c                                                         0.0s 
 => => pulling sha256:5a96a0b5efbdf4efe9ff1e6aeffa495d7148e0366b5c9958a625329f4beea3f5                                                         0.1s
 => => pulling sha256:1efc276f4ff952c055dea726cfc96ec6a4fdb8b62d9eed816bd2b788f2860ad7                                                         0.2s 
 => => pulling sha256:65462f2b00a4c62e8aa87511a8185d447d6a1d413afe42e905072da4d47f7445                                                         0.0s 
 => => pulling sha256:098c9e455557d7de690b40c84f2c1afed09d362d7ede72bd32f815ddd21b3c83                                                         0.1s 
 => => pulling sha256:5f71eb90fb301cc7600aa81f89eabc0a135cfe8dfede2488f93abc3f74be80a2                                                         0.0s 
 => => pulling sha256:46ccdbc34fc33a74ae69bbaed8d3a1b339db81a5c231ef5646d414a424bf8fb2                                                         0.0s 
 => => pulling sha256:612fc9be3d770781610df94fe2336cc63851fe9ea7ca2b919299b331b01565ae                                                         0.1s 
 => [5/5] COPY src ./                                                                                                                          0.2s 
 => exporting to image                                                                                                                         0.0s 
 => => exporting layers                                                                                                                        0.0s 
 => => writing image sha256:1ed2f6427ff9596dd94065719b5d9bc7c5be85c703b4ef0c41ecc30c2fd6c1d6                                                   0.0s 
 => => naming to localhost:5000/cache-experiments:2                                                                                            0.0s 
 => exporting cache                                                                                                                            0.0s 
 => => preparing build cache for export                                                                                                        0.0s 

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
+ docker push localhost:5000/cache-experiments:2
The push refers to repository [localhost:5000/cache-experiments]
0730abb3e72a: Pushed
f30a8f1d517a: Layer already exists                                                                                                                  
10ed798bb875: Layer already exists                                                                                                                  
e78c457618a4: Layer already exists                                                                                                                  
b4573ffeefec: Layer already exists                                                                                                                  
5ebe2abcebb0: Layer already exists                                                                                                                  
7287cceb8eb6: Layer already exists                                                                                                                  
e7a38ce55a0d: Layer already exists                                                                                                                  
92a4e8a3140f: Layer already exists                                                                                                                  
2: digest: sha256:ba6ddefecbc8cbc931dfe013dde59c111f9332837592a47f257105cb7f24df82 size: 2202
+ echo '************************* Changing the Docker context *************************'
************************* Changing the Docker context *************************
+ rm -f src/temp.py
+ echo '************************* Building the third image *************************'
************************* Building the third image *************************
+ docker system prune -af
Deleted Images:
untagged: localhost:5000/cache-experiments:2
untagged: localhost:5000/cache-experiments@sha256:ba6ddefecbc8cbc931dfe013dde59c111f9332837592a47f257105cb7f24df82
deleted: sha256:1ed2f6427ff9596dd94065719b5d9bc7c5be85c703b4ef0c41ecc30c2fd6c1d6

Deleted build cache objects:
oiyrsimyuz9rmajllipr1zj67
q1bxe5s514pegv9ov8pbeqnre
bphbx4hwwq14hn9t9yrycgfsx
bfrvsoow2dkmjyycjn0tzd2nj
q498s9fcpgc39hv73b12i6k6i
iw0tuvylaw2s24i4qkhzn4kvg
a2lkp9h02y6v87jg4zb81p6ky
efe0f1jjqdkynhbqvjuo1jddy
hyyh8nmvbey95klz2sm9he3nh
rdoyvdti8xd855tb9p0udas7g
qrg97d6dyu3uhg2kqjwmo6fw5
hcuu0x8vy2cc5ebegozgc3z39

Total reclaimed space: 721B
+ docker images -a
REPOSITORY   TAG       IMAGE ID       CREATED       SIZE
registry     2         d1fe2eaf6101   2 weeks ago   24.1MB
+ docker build -t localhost:5000/cache-experiments:3 --cache-from localhost:5000/cache-experiments:2 --build-arg BUILDKIT_INLINE_CACHE=1 .
[+] Building 51.0s (12/12) FINISHED
 => [internal] load build definition from Dockerfile                                                                                           0.0s
 => => transferring dockerfile: 229B                                                                                                           0.0s 
 => [internal] load .dockerignore                                                                                                              0.0s 
 => => transferring context: 2B                                                                                                                0.0s 
 => [internal] load metadata for docker.io/library/python:3.10.5-slim-bullseye                                                                43.8s
 => importing cache manifest from localhost:5000/cache-experiments:2                                                                           0.0s
 => [1/5] FROM docker.io/library/python:3.10.5-slim-bullseye@sha256:f9f03f46267e182193544299504687e711c623e2a085323138f94ed9b01ce641           3.6s 
 => => resolve docker.io/library/python:3.10.5-slim-bullseye@sha256:f9f03f46267e182193544299504687e711c623e2a085323138f94ed9b01ce641           0.0s 
 => => sha256:bdf0079de4094afdb26b94d9f89b716499436282c972461d945a87899c015c23 1.37kB / 1.37kB                                                 0.0s
 => => sha256:5b7d04484c737c4a2af7f33a85470dab369e922a570411f40ebd72d35167bc49 7.49kB / 7.49kB                                                 0.0s 
 => => sha256:1efc276f4ff952c055dea726cfc96ec6a4fdb8b62d9eed816bd2b788f2860ad7 31.37MB / 31.37MB                                               1.5s 
 => => sha256:126aaa6587b8608b62fa39019a891e98460e23bad2d04fb235c4dc8fae2adf3c 1.08MB / 1.08MB                                                 0.4s 
 => => sha256:5a96a0b5efbdf4efe9ff1e6aeffa495d7148e0366b5c9958a625329f4beea3f5 11.77MB / 11.77MB                                               1.1s 
 => => sha256:f9f03f46267e182193544299504687e711c623e2a085323138f94ed9b01ce641 1.86kB / 1.86kB                                                 0.0s
 => => sha256:65462f2b00a4c62e8aa87511a8185d447d6a1d413afe42e905072da4d47f7445 233B / 233B                                                     0.6s 
 => => sha256:098c9e455557d7de690b40c84f2c1afed09d362d7ede72bd32f815ddd21b3c83 3.18MB / 3.18MB                                                 1.1s 
 => => extracting sha256:1efc276f4ff952c055dea726cfc96ec6a4fdb8b62d9eed816bd2b788f2860ad7                                                      1.2s 
 => => extracting sha256:126aaa6587b8608b62fa39019a891e98460e23bad2d04fb235c4dc8fae2adf3c                                                      0.1s 
 => => extracting sha256:5a96a0b5efbdf4efe9ff1e6aeffa495d7148e0366b5c9958a625329f4beea3f5                                                      0.4s 
 => => extracting sha256:65462f2b00a4c62e8aa87511a8185d447d6a1d413afe42e905072da4d47f7445                                                      0.0s 
 => => extracting sha256:098c9e455557d7de690b40c84f2c1afed09d362d7ede72bd32f815ddd21b3c83                                                      0.2s 
 => [internal] load build context                                                                                                              0.0s 
 => => transferring context: 368B                                                                                                              0.0s 
 => [2/5] WORKDIR /app                                                                                                                         0.2s 
 => [3/5] COPY requirements.txt ./                                                                                                             0.0s 
 => [4/5] RUN pip install -r requirements.txt                                                                                                  3.0s 
 => [5/5] COPY src ./                                                                                                                          0.0s 
 => exporting to image                                                                                                                         0.0s 
 => => exporting layers                                                                                                                        0.0s 
 => => writing image sha256:7749ebdbcd4d4b5f515a41be6aebd3ceca86d316d952216e6bc350dd0b7ef956                                                   0.0s 
 => => naming to localhost:5000/cache-experiments:3                                                                                            0.0s 
 => exporting cache                                                                                                                            0.0s 
 => => preparing build cache for export                                                                                                        0.0s 

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
+ rm -rf /mnt/c/Users/Alexander.Pykavy/Workspace/Playground/docker-cache-experiments/registry
+ docker rm -vf registry
registry 

Is it a known issue or something new?

Forked On 28 Sep 2022 at 10:22:21

DocX

I have exactly the same issue, I posted it in another one: https://github.com/moby/buildkit/issues/3009#issuecomment-1260694663

Commented On 28 Sep 2022 at 10:22:21
Issue Comment

DocX

Reproducibility is broken when re-building the exact same image multiple times because sometimes the `moby.buildkit.cache.v0` entry changes

The problem is that after a few re-builds the inline cache value (base64) changes. Nothing else changes except that the chains object is removed.

I don't know how to explain that but its super easy to reproduce (on latest and master buildkit):

test/Dockerfile that just copies something

FROM busybox@sha256:98de1ad411c6d08e50f26f392f3bc6cd65f686469b7c22a85c7b5fb1b820c154

# do some stupid copy
COPY --link --from=alpine@sha256:9b2a28eb47540823042a2ba401386845089bb7b62a9637d55816132c4c3c36eb /bin/ls /bin/ls 

reproduce.sh

img="ghcr.io/skirsten/tmp:test-loop-4" # Change or remove this image between tests

for j in {1..10}; do
  buildctl prune >/dev/null

  echo "building..."
  buildctl build --frontend dockerfile.v0 --local context=test --local dockerfile=test \
    --import-cache type=registry,ref=$img \
    --export-cache type=inline \
    --output type=image,name=$img,push=true \
    --metadata-file metadata.json 2>/dev/null

  digest=$(jq -r '."containerimage.digest"' metadata.json)
  config_digest=$(jq -r '."containerimage.config.digest"' metadata.json)

  echo "digest: $digest"
  echo "config_digest: $config_digest"

  crane config "$img@$digest" | jq -r '."moby.buildkit.cache.v0"' | base64 -d | jq . >new.json

  diff old.json new.json || true
  mv new.json old.json

  echo

  sleep 1
done 

Output:

building...
digest: sha256:630b2de48a2e51079cec38c002f0c8bc3820b859557963c9fbc79e0d4697ecb1
config_digest: sha256:7231e8e691bad124e4d170998c4b566487a1c61deeffa4a74c5c441138ca640f

building...
digest: sha256:630b2de48a2e51079cec38c002f0c8bc3820b859557963c9fbc79e0d4697ecb1
config_digest: sha256:7231e8e691bad124e4d170998c4b566487a1c61deeffa4a74c5c441138ca640f

building...
digest: sha256:9c703d1e490f1e6dfbfde75e6be0a1ee50ea9de7072e10b422c331820b1c50fe
config_digest: sha256:d915739f055f10545f797706423c52f88b5cddd429cb40640b45a03c0f33b8e5
33,40d32
<     "chains": [
<       {
<         "layers": [
<           1
<         ],
<         "createdAt": "0001-01-01T00:00:00Z"
<       }
<     ],

building...
digest: sha256:9c703d1e490f1e6dfbfde75e6be0a1ee50ea9de7072e10b422c331820b1c50fe
config_digest: sha256:d915739f055f10545f797706423c52f88b5cddd429cb40640b45a03c0f33b8e5

building...
digest: sha256:9c703d1e490f1e6dfbfde75e6be0a1ee50ea9de7072e10b422c331820b1c50fe
config_digest: sha256:d915739f055f10545f797706423c52f88b5cddd429cb40640b45a03c0f33b8e5

building...
digest: sha256:9c703d1e490f1e6dfbfde75e6be0a1ee50ea9de7072e10b422c331820b1c50fe
config_digest: sha256:d915739f055f10545f797706423c52f88b5cddd429cb40640b45a03c0f33b8e5

building...
digest: sha256:9c703d1e490f1e6dfbfde75e6be0a1ee50ea9de7072e10b422c331820b1c50fe
config_digest: sha256:d915739f055f10545f797706423c52f88b5cddd429cb40640b45a03c0f33b8e5

building...
digest: sha256:9c703d1e490f1e6dfbfde75e6be0a1ee50ea9de7072e10b422c331820b1c50fe
config_digest: sha256:d915739f055f10545f797706423c52f88b5cddd429cb40640b45a03c0f33b8e5

building...
digest: sha256:9c703d1e490f1e6dfbfde75e6be0a1ee50ea9de7072e10b422c331820b1c50fe
config_digest: sha256:d915739f055f10545f797706423c52f88b5cddd429cb40640b45a03c0f33b8e5

building...
digest: sha256:9c703d1e490f1e6dfbfde75e6be0a1ee50ea9de7072e10b422c331820b1c50fe
config_digest: sha256:d915739f055f10545f797706423c52f88b5cddd429cb40640b45a03c0f33b8e5 

as can be seen, the third build removes the chains object from the base64'd cache. This causes the config_digest to change which then changes the digest. At which build number it breaks seems random to me... This breaks the reproducibility.

Pretty sure this is a bug. Any help is appreciated, thanks :)

Forked On 28 Sep 2022 at 10:17:34

DocX

I may have related/same issue:

image build for v2 using --cache-from v1 (previsouly built with inline cache) does not reuse the cache. the next build, v3, using cache from v2, uses the cache. and this repeats every second image.

I found the difference in moby.buildkit.cache.v0:

Image that works as --cache-from has:

{
    "layers": [
      {
        "layer": 12,
        "createdAt": "2022-09-28T08:30:41.439375309Z"
      }
    ],
    "digest": "sha256:13c946961fa6795c59a5ec1f3ce23075ea8bd73056c4b837765a536fa85c7a92",
    "inputs": [
      [
        {
          "link": 6
        }
      ],
      [
        {
          "link": 19
        }
      ]
    ]
  },
  ... 

while the next image, that does not work as --cache-from, is missing the layers and have different numbers on the link values:

{
    "digest": "sha256:13c946961fa6795c59a5ec1f3ce23075ea8bd73056c4b837765a536fa85c7a92",
    "inputs": [
      [
        {
          "link": 5
        }
      ],
      [
        {
          "link": 18
        }
      ]
    ]
  }, 

I was trying to reproduce it, and it happens when the build-args changes only. When build args are the same, the caches can be reused correctly.

Commented On 28 Sep 2022 at 10:17:34

DocX

Autoscaling components for Kubernetes

Forked On 06 Sep 2022 at 11:15:25
Pull Request

DocX

fix: update version to 1.3.1-zendesk

Created On 19 Aug 2022 at 10:02:26

DocX

fix: update version to 0.1.0-zendesk

Created On 19 Aug 2022 at 10:02:17

DocX

fix: ruby example of tags submission

Created On 11 Jul 2022 at 03:38:54

DocX

fix: ruby example of tags submission

Pushed On 11 Jul 2022 at 03:37:48