Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DO NOT MERGE: set up a local registry, cache images #22726

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

DO NOT MERGE: set up a local registry, cache images #22726

wants to merge 5 commits into from

Conversation

edsantiago
Copy link
Collaborator

On each test VM:

  • bring up a local registry
  • copy all images used by tests, from quay to this registry
  • change the registries.conf used by tests so it points to this reg

Yes, this is stupid as it stands. It gains us nothing. It's just
a proof of concept. If it works, the registry setup and cache
will be moved to automation_images, so each CI VM will come
preloaded with a cache registry. And if it doesn't work, this
PR is a much much faster way to find out than a constant
flurry of spinning up new images.

Signed-off-by: Ed Santiago santiago@redhat.com

None

@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented May 16, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: edsantiago

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 16, 2024
@packit-as-a-service Packit-as-a-Service
Copy link

Ephemeral COPR build failed. @containers/packit-build please check.

@edsantiago edsantiago force-pushed the pull-from-local-registry branch 5 times, most recently from 684512b to c82b31a Compare May 16, 2024 16:14
Luap99
Luap99 reviewed May 16, 2024
View reviewed changes
test/e2e/pull_test.go Outdated
@@ -38,7 +38,8 @@ var _ = Describe("Podman pull", func() {

session := podmanTest.Podman([]string{"pull", "quay.io/libpod/ibetthisdoesntexist:there"})
session.WaitWithDefaultTimeout()
Expect(session).To(ExitWithError(125, "nitializing source docker://quay.io/libpod/ibetthisdoesntexist:there: reading manifest there in quay.io/libpod/ibetthisdoesntexist: unauthorized: access to the requested resource is not authorized"))
// FIXME: uncomfortable hardcoding of localhost:56789
Expect(session).To(ExitWithError(125, "nitializing source docker://quay.io/libpod/ibetthisdoesntexist:there: reading manifest there in localhost:56789/libpod/ibetthisdoesntexist: manifest unknown"))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

heads up, tests should still pass locally and I don't think we want to setup a local registry there right?
Thus I would think we need a regex or a Or() matcher to match both strings anyway

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is thorny. As tests are currently written, e2e tests hard-force the use of test/registries.conf. No matter where they're run (CI, laptop, anywhere). This may need to be reevaluated, but I'm not bothering with any of that until I find out if this approach is viable.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes sure keep testing, just keep in mind that the end result must still work locally

contrib/cirrus/runner.sh Outdated
set -x
# shellcheck disable=SC2154
exec bin/podman run --rm --privileged --net=host --cgroupns=host \
-v `mktemp -d -p /var/tmp`:/var/tmp:Z \
--tmpfs /tmp:mode=1777 \
--expose 56789 \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this does not do anything, not sure if you were expecting anything with that or if this serves documentation purposes.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need the container to be able to talk to host:56789. --port does the opposite, IIRC: host can talk to container. I'll look at logs and see what happens.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The container uses --network=host so it shares the network namespace with the host so from a network POV there should be no functional difference

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duh. Thanks, I missed that.

@edsantiago edsantiago force-pushed the pull-from-local-registry branch 7 times, most recently from 03825bc to 48e506f Compare May 22, 2024 14:40
Luap99
Luap99 reviewed May 22, 2024
View reviewed changes
contrib/cirrus/local-cache-registry Outdated

# Run the registry container.
must_pass podman run --quiet -d \
-p ${PODMAN_REGISTRY_PORT}:5000 \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I recommend to use --network host and set -e REGISTRY_HTTP_ADDR=0.0.0.0:${PODMAN_REGISTRY_PORT} instead.
Reason by using already a podman container you sort of conflict with later running containers networking wise. The network reload tests that nuke iptables,etc... will cause issues for connectivity one way or another.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ohhhhhh.. thank you. That's probably the cause of the hangs in the 700-kube tests.

@edsantiago edsantiago force-pushed the pull-from-local-registry branch 10 times, most recently from 293a60b to 88fa517 Compare May 28, 2024 13:54
@edsantiago edsantiago force-pushed the pull-from-local-registry branch 2 times, most recently from 6cc76b9 to 3bc7dbf Compare May 29, 2024 21:50
@edsantiago edsantiago force-pushed the pull-from-local-registry branch 6 times, most recently from ab22542 to 7b333a9 Compare June 6, 2024 01:09
@edsantiago edsantiago force-pushed the pull-from-local-registry branch 12 times, most recently from 08e647d to 7a0d1e9 Compare June 10, 2024 18:34
@edsantiago
CI: use local registry, part 1 of 3: setup
5b1c166 
As of containers/automation_images#357
our CI VMs include a local registry preloaded with all(*)
images used in tests.

 * where "all" means "most".

This commit starts that registry as part of VM setup, and
installs a new registries.conf that redirects docker and
quay to the new local registry. The hope is that this will
reduce CI flakes.

Since tests change over time, and new tests may require new
images, this commit also adds a mechanism for pulling in
remote images at test run time. Obviously this negates
the purpose of the cache, since it introduces a flake
pain point. The idea is: DO NOT DO THIS UNLESS ABSOLUTELY
NECESSARY, and then, if we have to do this, hurry up and
spin new CI VMs that include the new image(s).

Signed-off-by: Ed Santiago <santiago@redhat.com>
@edsantiago
CU: use local registry, part 2 of 3: fix tests
f87a915 
This commit gets tests working under the new local-registry system:

  * amend a few image names, mostly just sticking to a consistent
    list of those images in our registry cache. Mostly minor
    tag updates.

  * trickier: pull_test: change some error messages, and remove
    a test that's now a NOP. Basically, with a local (unprotected)
    registry we always get "404 manifest unknown"; with a real
    registry we'll get "403 I can't tell you".

  * trickiest: seccomp_test: build our own images at run time,
    with our desired labels. Until now we've been pulling
    prebuilt images, but those will not copy to the local
    cache registry. Something about v1? Anyhow, I gave up
    trying to cache them, and the workaround is straightforward.

Also took the liberty of strengthening a few error-message checks

Signed-off-by: Ed Santiago <santiago@redhat.com>
@edsantiago
CI: use local registry, part 3 of 3: for developers
743ffb3 
New tool, get-local-registry-script, intended for developers
to get a local registry running in their environment. This is
now necessary for e2e and apiv2 tests, because those use an
immutable registries.conf file.

Signed-off-by: Ed Santiago <santiago@redhat.com>
@edsantiago
keep trying new vms
981ac10 
Signed-off-by: Ed Santiago <santiago@redhat.com>
@edsantiago
fixup: manually start the registry
fabc5f2 
Signed-off-by: Ed Santiago <santiago@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Luap99 Luap99 Luap99 left review comments

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. release-note-none
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@edsantiago @Luap99