Ready-set tracker

Hands out the tasks of a graph without cycles as soon as every task they depend on is done, so a slow task holds up only the tasks that wait for it.

Time complexity O(V+E), gograph function dag.NewTracker.

How it works

A Tracker runs a dependency graph the way a build system does. Ready returns the tasks that can start, and Done marks tasks as finished, which makes ready every task that was waiting only on them. An edge from A to B means B depends on A, so Ready returns B once A is done.

Levels split the same graph into rounds, and each round waits for its slowest task. The tracker starts a task the moment its own dependencies finish, so a long task in one round doesn't hold up the tasks of the next round that never waited for it.

The tracker leaves running the tasks to the caller, and with them how many run at once. This page uses two workers that take the ready tasks first in, first out, and each task takes its weight in time. NewTracker copies the graph in O(V+E), and all the Ready and Done calls of a run add up to O(V+E) too. It returns ErrDAGHasCycle for a graph with a cycle, and Done returns ErrNotReady for a task Ready hasn't returned.

Build pipeline

The jobs of a CI pipeline for a web service, and how many minutes each one takes. Each edge points from a job to a job that waits for it.

A directed graph with 7 vertices and 9 edges.

With two runners, each job starting as soon as the jobs it waits for are done, how many minutes does the pipeline take, and how does that compare with running it level by level?

With 2 workers, the last of the 7 tasks finishes at time 17. Running the levels one after another on the same workers takes 18, since each level waits for its slowest task. One task at a time would take 26.

Step by step

The tracker hands out a task once every task it depends on is done, and 2 workers take the ready tasks in the order it returns them. Each task takes its weight in time, and the time it ends shows next to its name once it starts. The first call to Ready returns Checkout, the only task that depends on nothing.
At time 0, Checkout starts and takes 1, so it ends at 1. One worker has nothing to do until another task is ready.
At time 1, Checkout finishes. Done makes Install ready. Install starts and takes 3, so it ends at 4. One worker has nothing to do until another task is ready.
At time 4, Install finishes. Done makes Lint and Compile ready. Lint and Compile start, to end at 10 and 7.
At time 7, Compile finishes. Done makes Unit tests and Docker image ready. Unit tests starts and takes 6, so it ends at 13. Every worker is busy, so 1 ready task waits for one to come free.
At time 10, Lint finishes. Done makes no new task ready. Docker image starts and takes 5, so it ends at 15.

2 more steps lead to the last one:

Done. At time 17, Deploy finishes, the last of the 7 tasks. Running the levels one after another on 2 workers would take 18, and one task at a time 26.

Use it in Go

t, err := dag.NewTracker(g)
if err != nil {
	return err
}
workers := make(chan struct{}, 2)
finished := make(chan string, g.Order())
for t.Remaining() > 0 {
	for _, task := range t.Ready() {
		go func() {
			workers <- struct{}{}
			run(task) // task.Weight() is how long it takes
			<-workers
			finished <- task.Label()
		}()
	}
	if err := t.Done(<-finished); err != nil {
		return err
	}
}

Example graphs

Build pipeline, 7 vertices and 9 edges
Course prerequisites, 14 vertices and 18 edges
Spreadsheet formulas, 25 vertices and 41 edges
Monorepo build, 100 vertices and 266 edges