Jobflow: Computational Workflows Made Simple

Rosen, Andrew S.;Gallant, Max;George, Janine;Riebesell, Janosh;Ganose, Alex M.;et.al.
(2024) Journal of Open Source Software — Vol. 9, n° 93, p. 5995 (2024)

Files

rosen2024.pdf
  • Open Access
  • Adobe PDF
  • 277.28 KB
Download

Details

Authors
  • Rosen, Andrew S.orcid-logoDepartment of Materials Science and Engineering, University of California, Berkeley, Berkeley, CA, USA
    Author
  • Gallant, Maxorcid-logoDepartment of Materials Science and Engineering, University of California, Berkeley, Berkeley, CA, USA
    Author
  • George, Janineorcid-logoFederal Institute for Materials Research and Testing, Department Materials Chemistry, Berlin, Germany
    Author
  • Riebesell, Janoshorcid-logoMaterials Science Division, Lawrence Berkeley National Laboratory, Berkeley, CA, USA
    Author
  • Evans, Mattheworcid-logoUCLouvain
    Author
  • Petretto, GuidoUCLouvain
    Author
  • Rignanese, Gian-Marcoorcid-logoUCLouvain
    Author
  • Ganose, Alex M.orcid-logoDepartment of Chemistry, Imperial College London, London, UK
    Author
Show more
Abstract
We present Jobflow, a domain-agnostic Python package for writing computational workflows tailored for high-throughput computing applications. With its simple decorator-based approach, functions and class methods can be transformed into compute jobs that can be stitched together into complex workflows. Jobflow fully supports dynamic workflows where the full acyclic graph of compute jobs is not known until runtime, such as compute jobs that launch other jobs based on the results of previous steps in the workflow. The results of all Jobflow compute jobs can be easily stored in a variety of filesystem- and cloud-based databases without the data storage process being part of the underlying workflow logic itself. Jobflow has been intentionally designed to be fully independent of the choice of workflow manager used to dispatch the calculations on remote computing resources. At the time of writing, Jobflow workflows can be executed either locally or across distributed compute environments via an adapter to the FireWorks package, and Jobflow fully supports the integration of additional workflow execution adapters in the future.
Affiliations

Citations

  • APA
  • Chicago
  • FWB

Rosen, A. S., Gallant, M., George, J., Riebesell, J., Sahasrabuddhe, H., Shen, J.-X., Wen, M., Evans, M., Petretto, G., Waroquiers, D., Rignanese, G.-M., Persson, K. A., Jain, A., & Ganose, A. M. (2024). Jobflow: Computational Workflows Made Simple. Journal of Open Source Software, 9(93), 5995. https://doi.org/10.21105/joss.05995 (Original work published 2024)