Dagster docker run launches for truly seperate runtime instance

18-03-2026

• the gRPC server is meant to serve definitions, but the run execution still depends on which run launcher the daemon uses:
  • DefaultRunLauncher — sends StartRun RPC to the gRPC server, which then executes the run in its own process. This requires instance storage access.
  • DockerRunLauncher / K8sRunLauncher — spins up a new container to execute the run. The gRPC server only serves definitions.

Fix:

• Change the daemon’s run launcher to DockerRunLauncher — then your container stays truly standalone (serves definitions only), and runs execute in fresh containers. This is the “proper” architecture for containerised code locations.

Links:

• https://docs.dagster.io/guides/build/projects/workspaces/workspace-yaml#running-your-own-grpc-server
• https://docs.dagster.io/integrations/libraries/docker/dagster-docker#dagster_docker.DockerRunLauncher
• https://docs.dagster.io/guides/build/projects/workspaces/dg-toml#running-your-own-grpc-server

IO managers:

• we would also require a non-default io manager with the container setup, there is no built-in postgres manager, we could define our own but easiest to use in memory manager given each job (+ dependent jobs) run in same process / container, going forward we could look at azure delta lake storage which looks quite neat, basically blob storage with filesystem syntax (which would open up some nice retrieval / search use cases too) - https://docs.dagster.io/guides/build/io-managers#built-in, old one but some nice gotchas that i need to applky - https://www.cigari.co.uk/blog/2023-07-14-a-few-weeks-with-dagster/ (not polluting the public schema, which i have just done lol)
• looks like there is an oss postgres + polars io manager - https://gist.github.com/lorenzkort/ff620515d94aa25b604401d2328ff9fc (and geopandas - https://gist.github.com/hbruch/672b2a020d8bb618784d15c90dc91614))