The Monitor Worker provides a mechanism to trigger TUF signing without requiring any credentials (or the CLI) to be present in the packaging pipeline environment. As with the other Rugger workers, the Monitor Worker is implemented using the Celery distributed task queue.
The Monitor Worker acts as a bridge between the Packaging Pipeline and the rest of the Rugged workers. It mounts two remote filesystems: the regular shared filesystem used by the rest of the Rugged components, and a second “post-to-TUF” directory. The latter is where the packaging pipeline will write specially named sub-directories containing the targets to be signed.
Refer to the Package Release Workflow with Monitor Worker page for a diagram illustrating where and how the Monitor Worker fits into the Rugged TUF Server. The diagram is broken down into four distinct parts.
The Monitor Worker uses a cron-like mechanism (provided by Celery) to schedule periodic tasks. In particular, it schedules a find-new-targets task to run every 5 seconds. The interval is configurable using the scheduler_scan_period option.
This task checks for new targets that are ready to process in the “post-to-tuf” directory. The path to this directory is configurable using the post_to_tuf_path option.
Note that we’re copying files across network-mounted file-system boundaries, and this can take a while to complete. With a high enough volume, a second monitor-worker process could be triggered before the current one ends. This could result in targets.json being written by the first process, while the second one had not yet signed all of its targets. Since we need the targets.json to include all targets for a given package, we need to process packages serially.
As we’ll see in Part 4 (“Prepare inbound targets”), the Monitor Worker renames the directory it’s currently processing to tuf_processing_<TIMESTAMP>. This directory acts as a semaphore to indicate to subsequent scheduled runs that Rugged is already occupied processing a batch of targets.
So, at this point in the periodic scan, monitor-worker looks for a directory starting with tuf_processing_. If it finds one, then it ends the current process. If not, it proceeds to look for a target that is ready for processing, indicated by the tuf_ready_ prefix. We’ll explore this in Part 3 (“Post to TUF”).
This part reflects a typical packaging process. Generally it will result in multiple artifacts, usually consisting of at least an archive of the package source code, and a file containing metadata for use by the package manager and repository.
Nothing has to be modified in this process to accomodate TUF-signing by Rugged.
This part describes the steps necessary for the packaging pipeline to provide its packages to the Monitor Worker.
N.B. The packaging pipeline must mount a network filesystem (typically NFS, or similar) that contains the “post-to-TUF” directory. This filesystem is shared with the Monitor Worker, but not the other workers that make up the Rugged TUF Server.
A script running in the packaging pipeline environment must perform the following steps:
tuf_tmp_<TIMESTAMP>/) in the post-to-TUF directorytuf_ready_<TIMESTAMP>) (note: this must not cross a filesystem boundary)The timestamp uses microseconds in order to ensure the uniqueness of the subsequently created directories.
The Monitor Worker will ignore the presense of the temporary directory. This is intended to ensure consistency in writing to the network filesystem.
Renaming the directory to tuf_ready_<TIMESTAMP> indicates to the Monitor Worker that this package’s targets are ready for processing. Any ready target directories ought to be ordered by the timestamp embedded in the filename. This will ensure that they are processed in order, on a first-in, first-out basis.
As mentioned in Part 1, and elaborated further in Part 4 (below), if another target is being processed, this will be indicated by the presence of a tuf_processing_<TIMESTAMP> directory within the post-to-tuf directory. If such a directory exists, the Monitor worker will not proceed with processing any ready targets.
When the Monitor Worker detects that there are no currently processing targets, it will select the first ready target, based on the oldest timestamp. Immediately upon selecting a directory containing ready targets, it will rename it to tuf_processing_<TIMESTAMP>. As previously noted, this will block any further periodic processes from starting to process additional targets.
Next, the Monitor Worker creates a new directory with the same name (tuf_processing_<TIMESTAMP>), in Rugged’s “inbound” directory, preserving the existing timestamp. This directory (and its contents) will be ignored by the Targets Worker.
It then moves the target files from the post-to-tuf processing directory, to the identically named one in the inbound directory. This may cross a network filesystem boundary, and thus may take a non-trivial amount of time to complete.
In order to prepare the target files for signing, the Monitor Worker then moves the files from the inbound processing directory (typically /var/rugged/inbound_targets/tuf_processing_<TIMESTAMP>/ on the Rugged shared filesystem) to the inbound directory (typically /var/rugged/inbound_targets/ on the Rugged shared filesystem). It then deletes the now-empty inbound processing directory.
Finally, the Monitor worker triggers the signing of target files by posting a task for the Targets Worker to add targets.
At this point, the regular TUF signing process begins. The Targets Worker, having received an add-targets task proceeds to:
targets.jsonThe Monitor worker then proceeds to trigger a Snapshot update, followed by a Timestamp update.
With the TUF metadata now fully updated and consistent, the Monitor Worker deletes the tuf_processing_<TIMESTAMP> directory within the post-to-tuf directory. This allows the subsequent Monitor Worker periodic task to proceed with processing the next ready target (if any).