Executing Simulations¶
Overview¶
Questions¶
How can I execute a multistep workflow on many simulations?
Objectives¶
Introduce workflows.
Demonstrate how to use row to define multiple actions and the dependencies between them.
Execute the workflow to randomize and compress all state points in the data space.
Actions¶
The Introducing HOOMD-blue tutorial describes distinct actions that must be performed in sequence on each state point. These are initialization, randomization, compression, equilibration, and analysis. The previous section in this tutorial initialized every state point in the data space. This section will randomize and compress them and the next section will equilibrate them. Analysis can also be implemented as an action, but this is left as an exercise for the reader.
This tutorial defines each action as a function Python code that takes a list of signac jobs as an argument and breaks the project up into several smaller modules. The first module defines the create_simulation
method which creates a Simulation
object given the parameters in the signac job.
[2]:
%pycat create_simulation.py
import hoomd
def create_simulation(job, communicator):
"""Create a Simulation object based on the signac job."""
cpu = hoomd.device.CPU(communicator=communicator)
# Set the simulation seed from the state point.
simulation = hoomd.Simulation(device=cpu, seed=job.statepoint.seed)
mc = hoomd.hpmc.integrate.ConvexPolyhedron()
mc.shape["octahedron"] = dict(
vertices=[
(-0.5, 0, 0),
(0.5, 0, 0),
(0, -0.5, 0),
(0, 0.5, 0),
(0, 0, -0.5),
(0, 0, 0.5),
]
)
simulation.operations.integrator = mc
return simulation
The randomize action implements the code previously shown in the Introducing HOOMD-blue tutorial with slight adjustments to read the lattice.gsd
file from each state point and write outputs to the job directory.
[3]:
%pycat randomize.py
import hoomd
from create_simulation import create_simulation
def randomize(*jobs):
"""Randomize the particle positions and orientations."""
for job in jobs:
communicator = hoomd.communicator.Communicator()
simulation = create_simulation(job, communicator)
# Read `lattice.gsd` from the signac job's directory.
simulation.create_state_from_gsd(filename=job.fn("lattice.gsd"))
# Apply trial moves to randomize the particle positions and orientations.
simulation.run(10e3)
# Write `random.gsd` to the signac job's directory.
hoomd.write.GSD.write(
state=simulation.state, mode="xb", filename=job.fn("random.gsd")
)
compress.py
similarly implements the compress action where the final box volume is a function of the job’s state point:
[4]:
%pycat compress.py
import math
import hoomd
from create_simulation import create_simulation
def compress(*jobs):
"""Compress the simjlation to the target density."""
for job in jobs:
communicator = hoomd.communicator.Communicator()
simulation = create_simulation(job, communicator)
# Read `random.gsd` from the signac job directory.
simulation.create_state_from_gsd(filename=job.fn("random.gsd"))
a = math.sqrt(2) / 2
V_particle = 1 / 3 * math.sqrt(2) * a**3
initial_box = simulation.state.box
final_box = hoomd.Box.from_box(initial_box)
# Set the final box volume to the volume fraction for this signac job.
final_box.volume = (
simulation.state.N_particles * V_particle / job.statepoint.volume_fraction
)
compress = hoomd.hpmc.update.QuickCompress(
trigger=hoomd.trigger.Periodic(10), target_box=final_box
)
simulation.operations.updaters.append(compress)
# Tune trial move sizes during the compression.
periodic = hoomd.trigger.Periodic(10)
tune = hoomd.hpmc.tune.MoveSize.scale_solver(
moves=["a", "d"],
target=0.2,
trigger=periodic,
max_translation_move=0.2,
max_rotation_move=0.2,
)
simulation.operations.tuners.append(tune)
# Compress the system to the target box size.
while not compress.complete and simulation.timestep < 1e6:
simulation.run(1000)
if not compress.complete:
message = "Compression failed to complete."
raise RuntimeError(message)
# Write `compressed.gsd` to the job's directory
hoomd.write.GSD.write(
state=simulation.state, mode="xb", filename=job.fn("compressed.gsd")
)
# Save the timestep that compression completed
job.document["compressed_step"] = simulation.timestep
project.py
imports these methods and implements a command line argument parser (equilibrate will be described in the next section).
[5]:
%pycat project.py
import argparse
import signac
from compress import compress
from equilibrate import equilibrate
from randomize import randomize
if __name__ == "__main__":
# Parse the command line arguments: python action.py --action <ACTION> [DIRECTORIES]
parser = argparse.ArgumentParser()
parser.add_argument("--action", required=True)
parser.add_argument("directories", nargs="+")
args = parser.parse_args()
# Open the signac jobs
project = signac.get_project()
jobs = [project.open_job(id=directory) for directory in args.directories]
# Call the action
if args.action == "compress":
compress(*jobs)
elif args.action == "equilibrate":
equilibrate(*jobs)
elif args.action == "randomize":
randomize(*jobs)
You can use project.py
to execute any of these actions on the command line For example:
$ python project.py --action randomize 59363805e6f46a715bc154b38dffc4e4
As your project grows, you will soon find it difficult to keep track of which actions you have executed on which directories and what needs to be done next.
Define the workflow¶
Row is a command line tool that helps you manage your workflow. This tutorial demonstrates how to use row with HOOMD-blue. See the row user documentation for full details, including an introductory tutorial.
You describe the workflow actions in the file workflow.toml
. Normally, you would write this file in a text editor. This tutorial is a Jupyter notebook, so it will instead use Python code to write it (workflow.toml
will change from one section to the next).
[6]:
with open("workflow.toml", "w") as workflow:
workflow.write("""
[default.action]
command = "python project.py --action $ACTION_NAME {directories}"
[[action]]
name = "randomize"
products = ["random.gsd"]
resources.walltime.per_directory = "00:05:00"
[[action]]
name = "compress"
previous_actions = ["randomize"]
products = ["compressed.gsd"]
resources.walltime.per_directory = "00:10:00"
""")
workflow.toml
describes an array of tables in action
where each element describes a single action. The keys in default.action
apply to all actions (the default can be overridden). In this example, the default action’s command executes the project.py
file shown above. In the command’s arguments, row will provide concrete values for $ACTION_NAME
and {directories}
.
The two tables in the [[action]]
array describe the two actions in our workflow. Each is given a name (in name
), a list of files it produces (in products
), and an estimate of the maximum walltime needed to execute the action (in resources.walltime
). The compress action names randomize in previous_actions
. This tells row that it should not execute compress on a given directory until randomize has completed.
Run the Workflow¶
Now that you have defined the workflow, you can check its status. Normally, you would execute row show status
in your shell. Because this tutorial is a Jupyter notebook, it must prefix shell commands with !
.
[7]:
! row show status
Action Completed Submitted Eligible Waiting Remaining cost
randomize 0 0 3 0 0 CPU-hours
compress 0 0 0 3 0 CPU-hours
There are 3 directories eligible to run the randomize action. The 3 for compress are waiting for randomize to complete first.
row submit
with no other arguments will submit all eligible jobs. When you are on a workstation (or you set --cluster=none
), row submit
will execute the actions immediately. When you are on a cluster (see the last section in this tutorial), row submit
will submit cluster jobs to the queue. This notebook was executed on a workstation.
[8]:
! row submit
Submitting 1 job that may cost up to 0 CPU-hours.
[1/1] Submitting action 'randomize' on directory 59363805e6f46a715bc154b38dffc4e4 and 2 more (0ms).
Every directory in the data space now has a random.gsd
file produced by randomize:
[9]:
! ls workspace/*
workspace/59363805e6f46a715bc154b38dffc4e4:
lattice.gsd random.gsd signac_statepoint.json
workspace/972b10bd6b308f65f0bc3a06db58cf9d:
lattice.gsd random.gsd signac_statepoint.json
workspace/c1a59a95a0e8b4526b28cf12aa0a689e:
lattice.gsd random.gsd signac_statepoint.json
The status now shows that randomize is complete and compress is eligible:
[10]:
! row show status
Action Completed Submitted Eligible Waiting Remaining cost
randomize 3 0 0 0
compress 0 0 3 0 0 CPU-hours
Execute it:
[11]:
! row submit
Submitting 1 job that may cost up to 0 CPU-hours.
[1/1] Submitting action 'compress' on directory 59363805e6f46a715bc154b38dffc4e4 and 2 more (0ms).
Every directory in the data space now has a compressed.gsd
file produced by compress:
[12]:
! ls workspace/*
workspace/59363805e6f46a715bc154b38dffc4e4:
compressed.gsd random.gsd signac_statepoint.json
lattice.gsd signac_job_document.json
workspace/972b10bd6b308f65f0bc3a06db58cf9d:
compressed.gsd random.gsd signac_statepoint.json
lattice.gsd signac_job_document.json
workspace/c1a59a95a0e8b4526b28cf12aa0a689e:
compressed.gsd random.gsd signac_statepoint.json
lattice.gsd signac_job_document.json
Summary¶
In this section of the tutorial, you defined the actions to randomize and then compress the initial configuration using row. Then you executed these actions on all state points in the data space. The directory for each simulation now contains compressed.gsd
and is ready for equilibration at the target volume fraction.
The next section in this tutorial teaches you how to write an action that can continue itself and complete over several submissions.
This tutorial only teaches the basics of row. Read the row documentation to learn more.