[BE] Cache & Optimize Woodpecker CI #3450
[BE] Cache & Optimize Woodpecker CI #3450

github.com
Cache & Optimize Woodpecker CI by cetra3 · Pull Request #3450 · LemmyNet/lemmy

This is a branch I have been working on to optimise performance of the woodpecker CI using a drone cache plugin & some changes to the clippy step. This branch in its current state will not pass, as there requires some ancillary services.
What this PR does is two things:
- Adds in a cache restore and publish step, using a local minio instance to store/retrieve the cache. Currently the following folders are cached & restored each run:
target/
.cargo/
api_tests/node_modules
- Adjusts the
cargo clippy
step to run once only with all lints present. To acheive this, in test modules theunwrap_used
andindexing_slicing
are turned off (hence the many changes).
Testing on my own woodpecker instance, with a warmed cache, the CI run takes <8mins:
For the cache
there are two methods available for storing/retrieving the build cache between runs:
- Use an S3 compatible object store, i.e, minio or actually amazon S3. I've been using local minio instance to test & have include a
docker-compose.yml
. If this is the path that people want to take, then I would suggest having it fairly close, if not the same host as the woodpecker ci. - Use a volume mount cache. This unfortunately requires that the project settings needs to be set to
trusted
, which I'm not sure the ramifications of it are.
Here's the docker-compose.yml
I've been using:
yml
# docker-compose.yml version: '3' services: woodpecker-server: image: woodpeckerci/woodpecker-server:latest ports: - 8000:8000 volumes: - woodpecker-server-data:/var/lib/woodpecker/ environment: - WOODPECKER_OPEN=true - WOODPECKER_HOST=${WOODPECKER_HOST} - WOODPECKER_GITHUB=true - WOODPECKER_GITHUB_CLIENT=${WOODPECKER_GITHUB_CLIENT} - WOODPECKER_GITHUB_SECRET=${WOODPECKER_GITHUB_SECRET} - WOODPECKER_AGENT_SECRET=${WOODPECKER_AGENT_SECRET} - WOODPECKER_ADMIN=cetra3 woodpecker-agent: image: woodpeckerci/woodpecker-agent:latest command: agent restart: always depends_on: - woodpecker-server volumes: - /var/run/docker.sock:/var/run/docker.sock - agent-cache:/cache/ environment: - WOODPECKER_SERVER=woodpecker-server:9000 - WOODPECKER_AGENT_SECRET=${WOODPECKER_AGENT_SECRET} minio: image: minio/minio ports: - 9000:9000 - 9090:9090 command: - minio - server - --console-address - ":9090" environment: - MINIO_ROOT_USER=testing123 - MINIO_ROOT_PASSWORD=testing123 - MINIO_VOLUMES=/mnt/data volumes: - minio-cache:/mnt/data volumes: woodpecker-server-data: agent-cache: minio-cache: