Skip to content

Production Image Building #1954

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
GernotMaier merged 20 commits into from
Jan 5, 2026
Merged

Production Image Building #1954

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
0e0877f
Fix version setting for CORSIKA configuration repo
GernotMaier Dec 17, 2025
62cc589
add simtel versions to image tag
GernotMaier Dec 17, 2025
7a49549
base tags
GernotMaier Dec 17, 2025
921ceb2
Docker files prod improvement and tagging
GernotMaier Dec 17, 2025
b993820
legacy renaming
GernotMaier Dec 17, 2025
eb5d5ca
legacy builds
GernotMaier Dec 17, 2025
6ff8c04
legacy builds
GernotMaier Dec 17, 2025
9ff1e7e
typos
GernotMaier Dec 17, 2025
b2b01bd
production docker file
GernotMaier Dec 17, 2025
a709f20
corsika config release v0.1.0
GernotMaier Dec 17, 2025
0ee4bb2
fix image tag
GernotMaier Dec 17, 2025
665c586
simpler tags
GernotMaier Dec 17, 2025
253145b
naming in changelog
GernotMaier Dec 17, 2025
293b74b
api
GernotMaier Dec 17, 2025
6e956d6
corsika version
GernotMaier Dec 17, 2025
0ce52f2
add gcc
GernotMaier Dec 17, 2025
cc0bcd7
add back prod6
GernotMaier Dec 17, 2025
7a1c8d8
copilot comments
GernotMaier Dec 18, 2025
c56463e
Merge branch 'main' into image-building-prod
GernotMaier Jan 5, 2026
8c783be
tmp fix
GernotMaier Jan 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 14 additions & 13 deletions .github/workflows/build-corsika-simtel.yml → ...workflows/build-corsika-simtel-legacy.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
---
name: build-corsika-simtel
name: build-corsika-simtel-legacy
# **legacy build** - will be deprecated in near future
# Build and push CORSIKA / sim_telarray base images for different CPU architectures.
# The no_opt (non-optimized) images are built as multi-platform images (x86_64 and arm64).
#
Expand Down Expand Up @@ -73,22 +74,22 @@ jobs:
matrix:
version:
# list of avx instructions: 'avx2', 'avx512f', 'sse4', 'avx'
- {simtel_version: '240205', corsika_version: '77500', bernlohr_version: '1.67', hadronic_model: 'qgs2', avx_instruction: 'no_opt'}
- {simtel_version: '240927', corsika_version: '77550', bernlohr_version: '1.68', hadronic_model: 'qgs2', avx_instruction: 'no_opt'}
- {simtel_version: '240927', corsika_version: '77550', bernlohr_version: '1.68', hadronic_model: 'qgs2', avx_instruction: 'avx2'}
- {simtel_version: '240927', corsika_version: '77550', bernlohr_version: '1.68', hadronic_model: 'qgs2', avx_instruction: 'avx512f'}
- {simtel_version: '240927', corsika_version: '77550', bernlohr_version: '1.68', hadronic_model: 'qgs2', avx_instruction: 'sse4'}
- {simtel_version: '250903', corsika_version: '78010', bernlohr_version: '1.70', hadronic_model: 'qgs3', avx_instruction: 'no_opt'}
- {simtel_version: '250903', corsika_version: '78010', bernlohr_version: '1.70', hadronic_model: 'qgs3', avx_instruction: 'avx2'}
- {simtel_version: '250903', corsika_version: '78010', bernlohr_version: '1.70', hadronic_model: 'qgs3', avx_instruction: 'avx512f'}
- {simtel_version: '250903', corsika_version: '78010', bernlohr_version: '1.70', hadronic_model: 'qgs3', avx_instruction: 'sse4'}
- {simtel_version: '240205', corsika_version: '77500', bernlohr_version: '1.67', hadronic_model: 'qgs2', avx_flag: 'no_opt'}
- {simtel_version: '240927', corsika_version: '77550', bernlohr_version: '1.68', hadronic_model: 'qgs2', avx_flag: 'no_opt'}
- {simtel_version: '240927', corsika_version: '77550', bernlohr_version: '1.68', hadronic_model: 'qgs2', avx_flag: 'avx2'}
- {simtel_version: '240927', corsika_version: '77550', bernlohr_version: '1.68', hadronic_model: 'qgs2', avx_flag: 'avx512f'}
- {simtel_version: '240927', corsika_version: '77550', bernlohr_version: '1.68', hadronic_model: 'qgs2', avx_flag: 'sse4'}
- {simtel_version: '250903', corsika_version: '78010', bernlohr_version: '1.70', hadronic_model: 'qgs3', avx_flag: 'no_opt'}
- {simtel_version: '250903', corsika_version: '78010', bernlohr_version: '1.70', hadronic_model: 'qgs3', avx_flag: 'avx2'}
- {simtel_version: '250903', corsika_version: '78010', bernlohr_version: '1.70', hadronic_model: 'qgs3', avx_flag: 'avx512f'}
- {simtel_version: '250903', corsika_version: '78010', bernlohr_version: '1.70', hadronic_model: 'qgs3', avx_flag: 'sse4'}
production:
# list of productions build options: 'prod6-sc', 'prod6-baseline', 'prod5', 'prod5-sc'
# - {build_opt: 'prod6-sc', extra_def: ''} # currently not used
- {build_opt: 'prod6-baseline', extra_def: ''}

env:
BASE_LABELS: "corsika-simtel-${{ matrix.version.simtel_version }}-corsika-${{ matrix.version.corsika_version }}-bernlohr-${{ matrix.version.bernlohr_version }}-${{ matrix.production.build_opt }}-${{ matrix.version.hadronic_model }}-${{ matrix.version.avx_instruction }}"
BASE_LABELS: "corsika-simtel-${{ matrix.version.simtel_version }}-corsika-${{ matrix.version.corsika_version }}-bernlohr-${{ matrix.version.bernlohr_version }}-${{ matrix.production.build_opt }}-${{ matrix.version.hadronic_model }}-${{ matrix.version.avx_flag }}"

steps:
- name: Checkout repository
Expand Down Expand Up @@ -144,12 +145,12 @@ jobs:
with:
context: .
# Build multi-platform only for no_opt (non-optimized) variant, others are x86-specific
platforms: ${{ matrix.version.avx_instruction == 'no_opt' && 'linux/amd64,linux/arm64/v8' || 'linux/amd64' }}
platforms: ${{ matrix.version.avx_flag == 'no_opt' && 'linux/amd64,linux/arm64/v8' || 'linux/amd64' }}
build-args: |
BUILD_OPT=${{ matrix.production.build_opt }}
EXTRA_DEF=${{ matrix.production.extra_def }}
HADRONIC_MODEL=${{ matrix.version.hadronic_model }}
AVX_FLAG=${{ matrix.version.avx_instruction }}
AVX_FLAG=${{ matrix.version.avx_flag }}
CORSIKA_VERSION=${{ matrix.version.corsika_version }}
BERNLOHR_VERSION=${{ matrix.version.bernlohr_version }}
secrets: |
Expand Down
44 changes: 20 additions & 24 deletions .github/workflows/build-corsika7.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ name: build-corsika7
# The no_opt (non-optimized) images are built as multi-platform images (x86_64 and arm64).
#
# The images are built and pushed when manually triggered (workflow_dispatch).
#

on:
workflow_dispatch:
Expand All @@ -19,7 +20,6 @@ env:
CLOUD_URL: "https://syncandshare.desy.de/index.php/s/"

jobs:

download-auxiliary-files:
runs-on: ubuntu-latest

Expand All @@ -44,13 +44,13 @@ jobs:
strategy:
matrix:
version:
- {corsika_version: '78010', avx_instruction: 'no_opt'}
- {corsika_version: '78010', avx_instruction: 'avx2'}
- {corsika_version: '78010', avx_instruction: 'avx512f'}
- {corsika_version: '78010', avx_instruction: 'sse4'}

- {corsika: '78010', avx_flag: 'no_opt', corsika_config: 'v0.1.0'}
- {corsika: '78010', avx_flag: 'avx2', corsika_config: 'v0.1.0'}
- {corsika: '78010', avx_flag: 'avx512f', corsika_config: 'v0.1.0'}
- {corsika: '78010', avx_flag: 'sse4', corsika_config: 'v0.1.0'}
env:
BASE_LABELS: "corsika7-${{ matrix.version.corsika_version }}-${{ matrix.version.avx_instruction }}"
BASE_LABEL: "corsika7"
BASE_TAG: "${{ matrix.version.corsika }}-${{ matrix.version.avx_flag }}"

steps:
- name: Checkout repository
Expand All @@ -73,34 +73,30 @@ jobs:
password: ${{ secrets.GITHUB_TOKEN }}

- name: Extract metadata
id: meta-base
id: meta
uses: docker/metadata-action@v5
with:
tags: |
type=ref,event=pr
type=semver,pattern={{major}}.{{minor}}.{{patch}}
type=schedule,pattern={{date 'YYYYMMDD'}}
type=raw,value={{date 'YYYYMMDD-HHmmss'}}
images: ${{ env.REGISTRY }}/gammasim/${{ env.BASE_LABELS }}
flavor: latest=true
type=ref,event=pr,suffix=-${{ env.BASE_TAG }}
type=semver,pattern={{major}}.{{minor}}.{{patch}}-${{ env.BASE_TAG }}
type=schedule,pattern={{date 'YYYYMMDD'}}-${{ env.BASE_TAG }}
type=raw,value={{date 'YYYYMMDD'}}-${{ env.BASE_TAG }}
images: ${{ env.REGISTRY }}/gammasim/${{ env.BASE_LABEL }}
flavor: latest=false

- name: Build and push
uses: docker/build-push-action@v6
with:
context: .
# Build multi-platform only for no_opt (non-optimized) variant, others are x86-specific
platforms: ${{ matrix.version.avx_instruction == 'no_opt' && 'linux/amd64,linux/arm64/v8' || 'linux/amd64' }}
platforms: ${{ matrix.version.avx_flag == 'no_opt' && 'linux/amd64,linux/arm64/v8' || 'linux/amd64' }}
build-args: |
AVX_FLAG=${{ matrix.version.avx_instruction }}
CORSIKA_VERSION=${{ matrix.version.corsika_version }}
AVX_FLAG=${{ matrix.version.avx_flag }}
CORSIKA_VERSION=${{ matrix.version.corsika }}
CORSIKA_CONFIG_VERSION=${{ matrix.version.corsika_config }}
secrets: |
corsika7_token=${{ secrets.CORSIKA7_KIT_GITLAB }}
push: ${{ github.event_name == 'workflow_dispatch' }}
file: ./docker/Dockerfile-corsika7
tags: ${{ steps.meta-base.outputs.tags }}
labels: ${{ steps.meta-base.outputs.labels }}

- name: Output image information
run: |
echo "Built base image: ${{ env.REGISTRY }}/gammasim/${{ env.BASE_LABELS }}:latest"
echo "Image tags: ${{ steps.meta-base.outputs.tags }}"
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
40 changes: 16 additions & 24 deletions .github/workflows/build-sim_telarray.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ env:
CLOUD_URL: "https://syncandshare.desy.de/index.php/s/"

jobs:
build-base-images:
build-sim_telarray:
runs-on: ubuntu-latest
permissions:
contents: read
Expand All @@ -28,11 +28,9 @@ jobs:
matrix:
version:
- {simtel_version: 'master', hessio_version: 'master', stdtools_version: 'master'}
production:
- {build_opt: 'prod6-baseline', extra_def: '-DMAXIMUM_TELESCOPES=99'}

env:
BASE_LABELS: "simtel-${{ matrix.version.simtel_version }}-${{ matrix.production.build_opt }}"
BASE_LABEL: "sim_telarray"
BASE_TAG: "${{ matrix.version.simtel_version }}"

steps:
- name: Checkout repository
Expand All @@ -48,19 +46,19 @@ jobs:
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Extract metadata for sim_telarray image (tags, labels)
id: meta-base
- name: Extract metadata
id: meta
uses: docker/metadata-action@v5
with:
tags: |
type=ref,event=pr
type=semver,pattern={{major}}.{{minor}}.{{patch}}
type=schedule,pattern={{date 'YYYYMMDD'}}
type=raw,value={{date 'YYYYMMDD-HHmmss'}}
images: ${{ env.REGISTRY }}/gammasim/${{ env.BASE_LABELS }}
type=ref,event=pr,suffix=-${{ env.BASE_TAG }}
type=semver,pattern={{major}}.{{minor}}.{{patch}}-${{ env.BASE_TAG }}
type=schedule,pattern={{date 'YYYYMMDD'}}-${{ env.BASE_TAG }}
type=raw,value={{date 'YYYYMMDD'}}-${{ env.BASE_TAG }}
images: ${{ env.REGISTRY }}/gammasim/${{ env.BASE_LABEL }}
flavor: latest=true

- name: Build /sim_telarray image
- name: Build and push
uses: docker/build-push-action@v6
with:
context: .
Expand All @@ -69,17 +67,11 @@ jobs:
SIMTEL_VERSION=${{ matrix.version.simtel_version }}
HESSIO_VERSION=${{ matrix.version.hessio_version }}
STDTOOLS_VERSION=${{ matrix.version.stdtools_version }}
EXTRA_DEF=${{ matrix.production.extra_def }}
secrets: |
sim_telarray_token=${{ secrets.SIMTELARRAY_CTAO_GITLAB}}
hessio_token=${{ secrets.HESSIO_CTAO_GITLAB}}
stdtools_token=${{ secrets.STDTOOLS_CTAO_GITLAB}}
sim_telarray_token=${{ secrets.SIMTELARRAY_CTAO_GITLAB }}
hessio_token=${{ secrets.HESSIO_CTAO_GITLAB }}
stdtools_token=${{ secrets.STDTOOLS_CTAO_GITLAB }}
push: ${{ github.event_name == 'workflow_dispatch' }}
file: ./docker/Dockerfile-simtel_array
tags: ${{ steps.meta-base.outputs.tags }}
labels: ${{ steps.meta-base.outputs.labels }}

- name: Output image information
run: |
echo "Built base image: ${{ env.REGISTRY }}/gammasim/${{ env.BASE_LABELS }}:latest"
echo "Image tags: ${{ steps.meta-base.outputs.tags }}"
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
44 changes: 15 additions & 29 deletions .github/workflows/build-simtools-dev.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ on:
workflow_dispatch:
pull_request:
paths:
- './docker/Dockerfile-simtools-dev'
- docker/Dockerfile-simtools-dev
release:
types: [published]
schedule:
Expand All @@ -15,7 +15,6 @@ on:
env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}
TEST_TAG: user/app:test
CLOUD_URL: "https://syncandshare.desy.de/index.php/s/"

jobs:
Expand All @@ -34,15 +33,14 @@ jobs:
qgsdat-III.bz2
retention-days: 1

build-simtools-container:
build-simtools-dev:
runs-on: ubuntu-latest
needs: [download-qgsjet-tables]
permissions:
contents: read
packages: write
strategy:
matrix:
type: ['dev']
env:
BASE_LABEL: "simtools-dev"

steps:
- name: Checkout repository
Expand All @@ -54,17 +52,14 @@ jobs:
name: upload-qgsjet-tables
path: .

- name: Set build arguments (correct branch)
- name: Set build branch
run: |
if [[ "${{ github.event_name }}" == 'pull_request' ]]; then
echo "BUILD_BRANCH=${GITHUB_HEAD_REF}" >> "$GITHUB_ENV"
echo "SIMTOOLS_BRANCH=${GITHUB_HEAD_REF}" >> "$GITHUB_ENV"
else
echo "BUILD_BRANCH=${GITHUB_REF#refs/heads/}" >> "$GITHUB_ENV"
echo "SIMTOOLS_BRANCH=${GITHUB_REF#refs/heads/}" >> "$GITHUB_ENV"
fi
echo "BUILD BRANCH ${{ env.BUILD_BRANCH }}"

- name: Set up QEMU
uses: docker/setup-qemu-action@v3
echo "BUILD BRANCH ${{ env.SIMTOOLS_BRANCH }}"

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
Expand All @@ -76,34 +71,25 @@ jobs:
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Extract metadata (tags, labels) for Docker
- name: Extract metadata
id: meta
uses: docker/metadata-action@v5
with:
tags: |
type=ref,event=pr
type=semver,pattern={{major}}.{{minor}}.{{patch}}
type=schedule,pattern={{date 'YYYYMMDD'}}
type=raw,value={{date 'YYYYMMDD-HHmmss'}}
images: ${{ env.REGISTRY }}/gammasim/simtools-${{ matrix.type }}
type=raw,value={{date 'YYYYMMDD'}}
images: ${{ env.REGISTRY }}/gammasim/${{ env.BASE_LABEL }}
flavor: latest=true

- name: Build Docker image
uses: docker/build-push-action@v6
with:
context: .
build-args: BUILD_BRANCH=${{ env.BUILD_BRANCH }}
load: true
file: ./docker/Dockerfile-simtools-${{ matrix.type }}
tags: ${{ env.TEST_TAG }}

- name: Build and push image
- name: Build and push
uses: docker/build-push-action@v6
with:
context: .
platforms: linux/amd64,linux/arm64/v8
build-args: BUILD_BRANCH=${{ env.BUILD_BRANCH }}
build-args: SIMTOOLS_BRANCH=${{ env.SIMTOOLS_BRANCH }}
push: ${{ github.event_name == 'release' || github.ref == 'refs/heads/main' || github.event_name == 'workflow_dispatch' }}
file: ./docker/Dockerfile-simtools-${{ matrix.type }}
file: ./docker/Dockerfile-simtools-dev
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}-${{ matrix.type }}
labels: ${{ steps.meta.outputs.labels }}
Loading