Executing Simulations

Overview

Questions

• How can I execute a multistep workflow on many simulations?

Objectives

• Introduce workflows.

• Demonstrate how to use row to define multiple actions and the dependencies between them.

• Execute the workflow to randomize and compress all state points in the data space.

Actions

The Introducing HOOMD-blue tutorial describes distinct actions that must be performed in sequence on each state point. These are initialization, randomization, compression, equilibration, and analysis. The previous section in this tutorial initialized every state point in the data space. This section will randomize and compress them and the next section will equilibrate them. Analysis can also be implemented as an action, but this is left as an exercise for the reader.

This tutorial defines each action as a function Python code that takes a list of signac jobs as an argument and breaks the project up into several smaller modules. The first module defines the create_simulation method which creates a Simulation object given the parameters in the signac job.

%pycat create_simulation.py

import hoomd


def create_simulation(job, communicator):
    """Create a Simulation object based on the signac job."""
    cpu = hoomd.device.CPU(communicator=communicator)

    # Set the simulation seed from the state point.
    simulation = hoomd.Simulation(device=cpu, seed=job.statepoint.seed)
    mc = hoomd.hpmc.integrate.ConvexPolyhedron()
    mc.shape["octahedron"] = dict(
        vertices=[
            (-0.5, 0, 0),
            (0.5, 0, 0),
            (0, -0.5, 0),
            (0, 0.5, 0),
            (0, 0, -0.5),
            (0, 0, 0.5),
        ]
    )
    simulation.operations.integrator = mc

    return simulation

The randomize action implements the code previously shown in the Introducing HOOMD-blue tutorial with slight adjustments to read the lattice.gsd file from each state point and write outputs to the job directory.

%pycat randomize.py

import hoomd
from create_simulation import create_simulation


def randomize(*jobs):
    """Randomize the particle positions and orientations."""
    for job in jobs:
        communicator = hoomd.communicator.Communicator()
        simulation = create_simulation(job, communicator)

        # Read `lattice.gsd` from the signac job's directory.
        simulation.create_state_from_gsd(filename=job.fn("lattice.gsd"))

        # Apply trial moves to randomize the particle positions and orientations.
        simulation.run(10e3)

        # Write `random.gsd` to the signac job's directory.
        hoomd.write.GSD.write(
            state=simulation.state, mode="xb", filename=job.fn("random.gsd")
        )

compress.py similarly implements the compress action where the final box volume is a function of the job’s state point:

%pycat compress.py

import math

import hoomd
from create_simulation import create_simulation


def compress(*jobs):
    """Compress the simjlation to the target density."""
    for job in jobs:
        communicator = hoomd.communicator.Communicator()
        simulation = create_simulation(job, communicator)

        # Read `random.gsd` from the signac job directory.
        simulation.create_state_from_gsd(filename=job.fn("random.gsd"))

        a = math.sqrt(2) / 2
        V_particle = 1 / 3 * math.sqrt(2) * a**3

        initial_box = simulation.state.box
        final_box = hoomd.Box.from_box(initial_box)

        # Set the final box volume to the volume fraction for this signac job.
        final_box.volume = (
            simulation.state.N_particles * V_particle / job.statepoint.volume_fraction
        )
        compress = hoomd.hpmc.update.QuickCompress(
            trigger=hoomd.trigger.Periodic(10), target_box=final_box
        )
        simulation.operations.updaters.append(compress)

        # Tune trial move sizes during the compression.
        periodic = hoomd.trigger.Periodic(10)
        tune = hoomd.hpmc.tune.MoveSize.scale_solver(
            moves=["a", "d"],
            target=0.2,
            trigger=periodic,
            max_translation_move=0.2,
            max_rotation_move=0.2,
        )
        simulation.operations.tuners.append(tune)

        # Compress the system to the target box size.
        while not compress.complete and simulation.timestep < 1e6:
            simulation.run(1000)

        if not compress.complete:
            message = "Compression failed to complete."
            raise RuntimeError(message)

        # Write `compressed.gsd` to the job's directory
        hoomd.write.GSD.write(
            state=simulation.state, mode="xb", filename=job.fn("compressed.gsd")
        )
        # Save the timestep that compression completed
        job.document["compressed_step"] = simulation.timestep

project.py imports these methods and implements a command line argument parser (equilibrate will be described in the next section).

%pycat project.py

import argparse

import signac
from compress import compress
from equilibrate import equilibrate
from randomize import randomize

if __name__ == "__main__":
    # Parse the command line arguments: python action.py --action <ACTION> [DIRECTORIES]
    parser = argparse.ArgumentParser()
    parser.add_argument("--action", required=True)
    parser.add_argument("directories", nargs="+")
    args = parser.parse_args()

    # Open the signac jobs
    project = signac.get_project()
    jobs = [project.open_job(id=directory) for directory in args.directories]

    # Call the action
    if args.action == "compress":
        compress(*jobs)
    elif args.action == "equilibrate":
        equilibrate(*jobs)
    elif args.action == "randomize":
        randomize(*jobs)

You can use project.py to execute any of these actions on the command line For example:

$ python project.py --action randomize 59363805e6f46a715bc154b38dffc4e4

As your project grows, you will soon find it difficult to keep track of which actions you have executed on which directories and what needs to be done next.

Define the workflow

Row is a command line tool that helps you manage your workflow. This tutorial demonstrates how to use row with HOOMD-blue. See the row user documentation for full details, including an introductory tutorial.

You describe the workflow actions in the file workflow.toml. Normally, you would write this file in a text editor. This tutorial is a Jupyter notebook, so it will instead use Python code to write it (workflow.toml will change from one section to the next).

with open("workflow.toml", "w") as workflow:
    workflow.write("""
[default.action]
command = "python project.py --action $ACTION_NAME {directories}"

[[action]]
name = "randomize"
products = ["random.gsd"]
resources.walltime.per_directory = "00:05:00"

[[action]]
name = "compress"
previous_actions = ["randomize"]
products = ["compressed.gsd"]
resources.walltime.per_directory = "00:10:00"
""")

workflow.toml describes an array of tables in action where each element describes a single action. The keys in default.action apply to all actions (the default can be overridden). In this example, the default action’s command executes the project.py file shown above. In the command’s arguments, row will provide concrete values for $ACTION_NAME and {directories}.

The two tables in the [[action]] array describe the two actions in our workflow. Each is given a name (in name), a list of files it produces (in products), and an estimate of the maximum walltime needed to execute the action (in resources.walltime). The compress action names randomize in previous_actions. This tells row that it should not execute compress on a given directory until randomize has completed.

Run the Workflow

Now that you have defined the workflow, you can check its status. Normally, you would execute row show status in your shell. Because this tutorial is a Jupyter notebook, it must prefix shell commands with !.

! row show status

Action    Completed Submitted Eligible Waiting Remaining cost
randomize         0         0        3       0    0 CPU-hours
compress          0         0        0       3    0 CPU-hours

There are 3 directories eligible to run the randomize action. The 3 for compress are waiting for randomize to complete first.

row submit with no other arguments will submit all eligible jobs. When you are on a workstation (or you set --cluster=none), row submit will execute the actions immediately. When you are on a cluster (see the last section in this tutorial), row submit will submit cluster jobs to the queue. This notebook was executed on a workstation.

! row submit

Submitting 1 job that may cost up to 0 CPU-hours.
[1/1] Submitting action 'randomize' on directory 59363805e6f46a715bc154b38dffc4e4 and 2 more (0ms).

Every directory in the data space now has a random.gsd file produced by randomize:

! ls workspace/*

workspace/59363805e6f46a715bc154b38dffc4e4:
lattice.gsd            random.gsd             signac_statepoint.json

workspace/972b10bd6b308f65f0bc3a06db58cf9d:
lattice.gsd            random.gsd             signac_statepoint.json

workspace/c1a59a95a0e8b4526b28cf12aa0a689e:
lattice.gsd            random.gsd             signac_statepoint.json

The status now shows that randomize is complete and compress is eligible:

! row show status

Action    Completed Submitted Eligible Waiting Remaining cost
randomize         3         0        0       0
compress          0         0        3       0    0 CPU-hours

Execute it:

! row submit

Submitting 1 job that may cost up to 0 CPU-hours.
[1/1] Submitting action 'compress' on directory 59363805e6f46a715bc154b38dffc4e4 and 2 more (0ms).

Every directory in the data space now has a compressed.gsd file produced by compress:

! ls workspace/*

workspace/59363805e6f46a715bc154b38dffc4e4:
compressed.gsd           random.gsd               signac_statepoint.json
lattice.gsd              signac_job_document.json

workspace/972b10bd6b308f65f0bc3a06db58cf9d:
compressed.gsd           random.gsd               signac_statepoint.json
lattice.gsd              signac_job_document.json

workspace/c1a59a95a0e8b4526b28cf12aa0a689e:
compressed.gsd           random.gsd               signac_statepoint.json
lattice.gsd              signac_job_document.json

Summary

In this section of the tutorial, you defined the actions to randomize and then compress the initial configuration using row. Then you executed these actions on all state points in the data space. The directory for each simulation now contains compressed.gsd and is ready for equilibration at the target volume fraction.

The next section in this tutorial teaches you how to write an action that can continue itself and complete over several submissions.

This tutorial only teaches the basics of row. Read the row documentation to learn more.