Support development on air-gapped environments

[jsoriano]

Users may be developing on restricted environments, where they have to access external resources through mirrors or proxies. Support this use case by allowing to configure the access to these resources. This will translate to allow the configuration of any currently hard-coded external URL.

For convenience with these use cases, the configuration to access these resources should be supported at the profile level. elastic-package should allow the configuration of any hard-coded external URL.

Some identified resources at the moment:

• Package Registry managed by elastic-package uses EPR_PROXY_TO to connect to https://epr.elastic.co.
  • Done in Expose profile setting to configure EPR_PROXY_TO #3113.
• ECS schemas are retrieved from the hard-coded ecsSchemaURL.
• elastic-package status:
  • It looks for packages published in https://epr.elastic.co, this could be parameterizable to use a custom registry.
  • It looks for serverless config in github, this should be probably optional.

[nimarezainia]

@jsoriano do we have a full list of these hard-coded settings? (trying to gauge the extent of this effort)

[jsoriano]

@jsoriano do we have a full list of these hard-coded settings? (trying to gauge the extent of this effort)

The ones in the description are the only ones we have identified, I have added another reference to epr.elastic.co that is probably not so blocker.

There are some other hard-coded URLs used in the serverless stack provider, but we can probably ignore them as it doesn't make sense to use serverless in air-gapped environments.

[nimarezainia]

@jsoriano couldn't users use DNS to get the hard-coded urls to resolve to a local address?

[jsoriano]

@jsoriano couldn't users use DNS to get the hard-coded urls to resolve to a local address?

In theory yes, but it would be quite hacky.

Apart from DNS, they would need to solve SSL, as the registry serves over HTTPS.

For DNS they would need to configure their internal servers (or their /etc/hosts file or equivalent) to resolve epr.elastic.co to their internal registry.

For HTTPS they would need to use insecure connections, what I think is not supported in elastic-package now. Or they would need to use a self-signed CA to generate a certificate for epr.elastic.co.

And imagine the same for Github to retrieve ECS schemas.

Reached this point it would be probably easier for these users to maintain a fork of elastic-package.

[jsoriano]

Took a quick look to this issue:

• Package Registry managed by elastic-package uses EPR_PROXY_TO to connect to https://epr.elastic.co.

This one seems easy to do, PR open for this in #3113.

• ECS schemas are retrieved from the hard-coded ecsSchemaURL.

Ideally this should be obtained from the application configuration, as the ECS schemas cache is global. This is started in #3114, but it shows that the logic retrieving ECS schemas is quite disconnected from the places where we should be reading the configuration.

Alternatively we could allow to configure this using an environment variable, but then it won't be possible to persist this configuration.

• elastic-package status:
  • It looks for packages published in https://epr.elastic.co, this could be parameterizable to use a custom registry.
  • It looks for serverless config in github, this should be probably optional.

These are probably not a blocker for this issue and could be done later.

[nimarezainia]

thanks @jsoriano Would you be able to provide us some small description for documentation as to how the user can deploy EPR in air-gap environments and what configuration to apply? (cc: @jstrassb )

[jsoriano]

thanks @jsoriano Would you be able to provide us some small description for documentation as to how the user can deploy EPR in air-gap environments and what configuration to apply? (cc: @jstrassb )

Do you mean the local package registry managed by elastic-package or a custom package registry for publishing custom packages?

For the one in elastic-package you can follow the test instructions in #3113 (merged but not released yet).

For a custom registry you could put the packages in a directory and use the vanilla docker image to serve them, as described in https://github.com/elastic/package-registry/?tab=readme-ov-file#docker.

[nimarezainia]

@jstrassb could you please review #3113 and see if this addresses the use case?

[nimarezainia]

@jsoriano Thanks for quick turn around on this, appreciate it.

thanks @jsoriano Would you be able to provide us some small description for documentation as to how the user can deploy EPR in air-gap environments and what configuration to apply? (cc: @jstrassb )

Do you mean the local package registry managed by elastic-package or a custom package registry for publishing custom packages?

What I was looking for in your PR is to find what needs to be updated in our docs (probably here) so the user can utilize what you developed here.

Clearly this makes for a better air-gap UX but don't really know which part of it needs to be in our public docs.

(cc: @karenzone we probably need to update the docs with this outcome. thx)