Run Commands
Read the GLOSSARY series >

Pachyderm Worker

Learn about the concept of a Pachyderm worker.

About #

Pachyderm workers are kubernetes pods that run the docker image (your user code) specified in the pipeline specification. When you create a pipeline, Pachyderm spins up workers that continuously run in the cluster, waiting for new data to process.

Each datum goes through the following processing phases inside a Pachyderm worker pod:

PhaseDescription
DownloadingThe Pachyderm worker pod downloads the datum contents
into Pachyderm.
ProcessingThe Pachyderm worker pod runs the contents of the datum
against your code.
UploadingThe Pachyderm worker pod uploads the results of processing
into an output repository.

Distributed processing internals