features/monitor/monitor-resilience.featureRunning behat features/monitor/monitor-resilience.feature results in:
@rugged @monitor-worker @monitor-resilience @gitlab-142
Feature: Monitor worker is resilient to network instability.
In order to run reliably for extended periods
As a TUF operator
I need the monitor worker to tolerate occasional network instability.
Background:
Given I reset Rugged
And I run "sudo rm -f /opt/post_to_tuf/*.test-flag"
And I run the Rugged command "rugged pause-processing"
And I initialize a Rugged repo
And I run the Rugged command "rugged logs --truncate"
Scenario: Ensure that monitor worker has necessary config for testing 'add-targets' resilience.
When I run the Rugged command "rugged config --worker=monitor-worker"
Then I should get:
"""
scheduler_log_level DEBUG
scheduler_scan_period 1.0
task_timeout 1
monitor_enable_network_instability_resiliency_test_mode True
"""
'scheduler_log_level DEBUG' was not found in command output:
------
=== Configuration for monitor-worker ===
broker_connection_string REDACTED
scheduler_log_level INFO
scheduler_refresh_period 3600.0
scheduler_scan_period 5.0
workers {'snapshot': {'name': 'snapshot-worker'}, 'targets': {'name': 'targets-worker'}, 'timestamp': {'name': 'timestamp-worker'}}
log_file /var/log/rugged/rugged.log
log_format %(asctime)s %(levelname)s (%(module)s.%(funcName)s): %(message)s
repo_path /var/rugged/tuf_repo
inbound_targets_path /var/rugged/incoming_targets
post_to_tuf_path /opt/post_to_tuf
repo_targets_path /var/rugged/tuf_repo/targets
repo_metadata_path /var/rugged/tuf_repo/metadata
roles {'root': {'threshold': 1, 'expiry': 31536000}, 'timestamp': {'threshold': 1, 'expiry': 86400}, 'snapshot': {'threshold': 1, 'expiry': 604800}, 'targets': {'threshold': 1, 'expiry': 604800}}
expiry_refresh_threshold 43200
consistent_snapshot False
print_host_headers True
use_hashed_bins False
number_of_bins 16
hashed_bins_key_name targets
celery_worker_max_memory_per_child 0
delete_targets_after_signing False
task_timeout 10
wait_for_processing_task_to_complete_timeout 30
wait_for_refreshing_task_to_complete_timeout 15
stale_semaphore_age_thresholds {'tuf_paused': 3600, 'tuf_processing_': 300, 'tuf_refreshing_expiry': 60}
------
(Exception)
Scenario: Monitor worker redirects tasks to test worker.
Given I run "sudo fallocate -l 1 /opt/post_to_tuf/add_targets_task.test-flag"
And I run "sudo sudo -u rugged mkdir -p /opt/post_to_tuf/tuf_tmp_FIRST/"
And I run "sudo fallocate -l 1 /opt/post_to_tuf/tuf_tmp_FIRST/foo1.zip"
And I run "sudo sudo -u rugged mv /opt/post_to_tuf/tuf_tmp_FIRST/ /opt/post_to_tuf/tuf_ready_FIRST/"
And I run the Rugged command "rugged resume-processing"
And I wait "5" seconds
When I run the Rugged command "rugged logs --worker=monitor-worker --limit=0"
Then I should get:
"""
Dispatching 'add-targets' task for: tuf_processing_FIRST
Diverting 'add_targets_task' task from 'targets-worker' to 'sleep' on test worker, to force a timeout.
Initializing connection to RabbitMQ
"""
'Diverting 'add_targets_task' task from 'targets-worker' to 'sleep' on test worker, to force a timeout.' was not found in command output:
------
=== Log for monitor-worker: /var/log/rugged/rugged.log ===
2026-06-08 18:40:14,380 INFO (monitor-worker.find_new_targets_task): New content found in post-to-tuf directory.
2026-06-08 18:40:14,380 INFO (monitor-worker.find_new_targets_task): Count of post-to-tuf content: 2
2026-06-08 18:40:14,380 INFO (monitor-worker.find_new_targets_task): Age of oldest post-to-tuf content: 1 seconds
2026-06-08 18:40:14,380 INFO (monitor-worker.find_new_targets_task): New targets found in post-to-tuf directory:
2026-06-08 18:40:14,380 INFO (monitor-worker.find_new_targets_task): tuf_ready_FIRST
2026-06-08 18:40:14,380 INFO (monitor-worker.find_new_targets_task): Processing: tuf_ready_FIRST
2026-06-08 18:40:14,381 INFO (monitor-worker.find_new_targets_task): Dispatching 'add-targets' task for: tuf_processing_FIRST
2026-06-08 18:40:14,395 INFO (monitor-worker._dispatch_add_targets): Added the following targets to the repository:
foo1.zip
Updated targets metadata.
------
(Exception)
And I should not get:
"""
Added the following targets to the repository:
"""
When I run the Rugged command "rugged logs --worker=targets-worker --limit=0"
Then I should not get:
"""
Received add-targets task
"""
When I run the Rugged command "rugged logs --worker=test-worker --limit=0"
Then I should get:
"""
Test-worker received 'sleep' task with interval of '5'.
"""
Scenario: Monitor worker detects timeout and stops further processing.
Given I run "sudo fallocate -l 1 /opt/post_to_tuf/add_targets_task.test-flag"
And I run "sudo sudo -u rugged mkdir -p /opt/post_to_tuf/tuf_tmp_FIRST/"
And I run "sudo fallocate -l 1 /opt/post_to_tuf/tuf_tmp_FIRST/foo1.zip"
And I run "sudo sudo -u rugged mv /opt/post_to_tuf/tuf_tmp_FIRST/ /opt/post_to_tuf/tuf_ready_FIRST/"
And I run the Rugged command "rugged resume-processing"
And I wait "5" seconds
When I run the Rugged command "rugged logs --worker=monitor-worker --limit=0"
Then I should get:
"""
Dispatching 'add-targets' task for: tuf_processing_FIRST
Diverting 'add_targets_task' task from 'targets-worker' to 'sleep' on test worker, to force a timeout.
Initializing connection to RabbitMQ
Timeout while dispatching 'add_targets_task' task to 'targets-worker' for 'tuf_processing_FIRST'.
"""
'Diverting 'add_targets_task' task from 'targets-worker' to 'sleep' on test worker, to force a timeout.' was not found in command output:
------
=== Log for monitor-worker: /var/log/rugged/rugged.log ===
2026-06-08 18:40:24,380 WARNING (monitor-worker.find_new_targets_task): The monitor worker is paused.
2026-06-08 18:40:24,380 INFO (monitor-worker.find_new_targets_task): Skipping find-new-targets task.
2026-06-08 18:40:29,380 INFO (monitor-worker.find_new_targets_task): New content found in post-to-tuf directory.
2026-06-08 18:40:29,380 INFO (monitor-worker.find_new_targets_task): Count of post-to-tuf content: 2
2026-06-08 18:40:29,380 INFO (monitor-worker.find_new_targets_task): Age of oldest post-to-tuf content: 5 seconds
2026-06-08 18:40:29,381 INFO (monitor-worker.find_new_targets_task): New targets found in post-to-tuf directory:
2026-06-08 18:40:29,381 INFO (monitor-worker.find_new_targets_task): tuf_ready_FIRST
2026-06-08 18:40:29,381 INFO (monitor-worker.find_new_targets_task): Processing: tuf_ready_FIRST
2026-06-08 18:40:29,381 INFO (monitor-worker.find_new_targets_task): Dispatching 'add-targets' task for: tuf_processing_FIRST
2026-06-08 18:40:29,396 INFO (monitor-worker._dispatch_add_targets): Added the following targets to the repository:
foo1.zip
Updated targets metadata.
------
(Exception)
And I should not get:
"""
Added the following targets to the repository:
Updating snapshot metadata
Updated snapshot metadata.
Updating timestamp metadata
Updated timestamp metadata.
"""
Scenario: Monitor worker detects timeout (add_targets_task) and resets the failed target.
Given I run "sudo fallocate -l 1 /opt/post_to_tuf/add_targets_task.test-flag"
And I run "sudo sudo -u rugged mkdir -p /opt/post_to_tuf/tuf_tmp_FIRST/"
And I run "sudo fallocate -l 1 /opt/post_to_tuf/tuf_tmp_FIRST/foo1.zip"
And I run "sudo sudo -u rugged mv /opt/post_to_tuf/tuf_tmp_FIRST/ /opt/post_to_tuf/tuf_ready_FIRST/"
And I run the Rugged command "rugged resume-processing"
And I wait "5" seconds
When I run the Rugged command "rugged logs --worker=monitor-worker --limit=0"
Then I should get:
"""
Timeout while dispatching 'add_targets_task' task to 'targets-worker' for 'tuf_processing_FIRST'.
Resetting 'tuf_processing_FIRST' to 'tuf_ready_FIRST'.
Renaming '/opt/post_to_tuf/tuf_processing_FIRST' to '/opt/post_to_tuf/tuf_ready_FIRST'.
Renamed '/opt/post_to_tuf/tuf_processing_FIRST' to '/opt/post_to_tuf/tuf_ready_FIRST'.
"""
'Timeout while dispatching 'add_targets_task' task to 'targets-worker' for 'tuf_processing_FIRST'.' was not found in command output:
------
=== Log for monitor-worker: /var/log/rugged/rugged.log ===
2026-06-08 18:40:39,380 INFO (monitor-worker.find_new_targets_task): New content found in post-to-tuf directory.
2026-06-08 18:40:39,380 INFO (monitor-worker.find_new_targets_task): Count of post-to-tuf content: 2
2026-06-08 18:40:39,381 INFO (monitor-worker.find_new_targets_task): Age of oldest post-to-tuf content: 4 seconds
2026-06-08 18:40:39,381 INFO (monitor-worker.find_new_targets_task): New targets found in post-to-tuf directory:
2026-06-08 18:40:39,381 INFO (monitor-worker.find_new_targets_task): tuf_ready_FIRST
2026-06-08 18:40:39,381 INFO (monitor-worker.find_new_targets_task): Processing: tuf_ready_FIRST
2026-06-08 18:40:39,382 INFO (monitor-worker.find_new_targets_task): Dispatching 'add-targets' task for: tuf_processing_FIRST
2026-06-08 18:40:39,400 INFO (monitor-worker._dispatch_add_targets): Added the following targets to the repository:
foo1.zip
Updated targets metadata.
------
(Exception)
Scenario: Monitor worker retries the failed target first.
Given I run "sudo fallocate -l 1 /opt/post_to_tuf/add_targets_task.test-flag"
And I run "sudo sudo -u rugged mkdir -p /opt/post_to_tuf/tuf_tmp_FIRST/"
And I run "sudo fallocate -l 1 /opt/post_to_tuf/tuf_tmp_FIRST/foo1.zip"
And I run "sudo sudo -u rugged mv /opt/post_to_tuf/tuf_tmp_FIRST/ /opt/post_to_tuf/tuf_ready_FIRST/"
And I run "sudo sudo -u rugged mkdir -p /opt/post_to_tuf/tuf_tmp_SECOND/"
And I run "sudo fallocate -l 1 /opt/post_to_tuf/tuf_tmp_SECOND/foo2.zip"
And I run "sudo sudo -u rugged mv /opt/post_to_tuf/tuf_tmp_SECOND/ /opt/post_to_tuf/tuf_ready_SECOND/"
And I run the Rugged command "rugged resume-processing"
And I wait "5" seconds
When I run the Rugged command "rugged logs --worker=monitor-worker --limit=0"
Then I should get:
"""
Timeout while dispatching 'add_targets_task' task to 'targets-worker' for 'tuf_processing_FIRST'.
Resetting 'tuf_processing_FIRST' to 'tuf_ready_FIRST'.
Renaming '/opt/post_to_tuf/tuf_processing_FIRST' to '/opt/post_to_tuf/tuf_ready_FIRST'.
Renamed '/opt/post_to_tuf/tuf_processing_FIRST' to '/opt/post_to_tuf/tuf_ready_FIRST'.
"""
'Timeout while dispatching 'add_targets_task' task to 'targets-worker' for 'tuf_processing_FIRST'.' was not found in command output:
------
=== Log for monitor-worker: /var/log/rugged/rugged.log ===
2026-06-08 18:40:49,381 INFO (monitor-worker.find_new_targets_task): New content found in post-to-tuf directory.
2026-06-08 18:40:49,381 INFO (monitor-worker.find_new_targets_task): Count of post-to-tuf content: 3
2026-06-08 18:40:49,381 INFO (monitor-worker.find_new_targets_task): Age of oldest post-to-tuf content: 2 seconds
2026-06-08 18:40:49,381 INFO (monitor-worker.find_new_targets_task): New targets found in post-to-tuf directory:
2026-06-08 18:40:49,382 INFO (monitor-worker.find_new_targets_task): tuf_ready_FIRST
2026-06-08 18:40:49,382 INFO (monitor-worker.find_new_targets_task): tuf_ready_SECOND
2026-06-08 18:40:49,382 INFO (monitor-worker.find_new_targets_task): Processing: tuf_ready_FIRST
2026-06-08 18:40:49,382 INFO (monitor-worker.find_new_targets_task): Dispatching 'add-targets' task for: tuf_processing_FIRST
------
(Exception)
Then I should not get:
"""
Timeout while dispatching 'add_targets_task' task to 'targets-worker' for 'tuf_processing_SECOND'.
Resetting 'tuf_processing_SECOND' to 'tuf_ready_SECOND'.
Renaming '/opt/post_to_tuf/tuf_processing_SECOND' to '/opt/post_to_tuf/tuf_ready_SECOND'.
Renamed '/opt/post_to_tuf/tuf_processing_SECOND' to '/opt/post_to_tuf/tuf_ready_SECOND'.
"""
Scenario: Monitor worker detects timeout (update_snapshot_task) and resets the failed target.
Given I run "sudo fallocate -l 1 /opt/post_to_tuf/update_snapshot_task.test-flag"
And I run "sudo sudo -u rugged mkdir -p /opt/post_to_tuf/tuf_tmp_FIRST/"
And I run "sudo fallocate -l 1 /opt/post_to_tuf/tuf_tmp_FIRST/foo1.zip"
And I run "sudo sudo -u rugged mv /opt/post_to_tuf/tuf_tmp_FIRST/ /opt/post_to_tuf/tuf_ready_FIRST/"
And I run the Rugged command "rugged resume-processing"
And I wait "5" seconds
When I run the Rugged command "rugged logs --worker=monitor-worker --limit=0"
Then I should get:
"""
Checking test mode for 'update_snapshot_task' task.
Checking for existence of test flag file at '/opt/post_to_tuf/update_snapshot_task.test-flag'.
Tried to update snapshot metadata on snapshot-worker 3 times, but failed.
Resetting 'tuf_processing_FIRST' to 'tuf_ready_FIRST'.
Renaming '/opt/post_to_tuf/tuf_processing_FIRST' to '/opt/post_to_tuf/tuf_ready_FIRST'.
Renamed '/opt/post_to_tuf/tuf_processing_FIRST' to '/opt/post_to_tuf/tuf_ready_FIRST'.
"""
'Checking test mode for 'update_snapshot_task' task.' was not found in command output:
------
=== Log for monitor-worker: /var/log/rugged/rugged.log ===
2026-06-08 18:40:59,380 INFO (monitor-worker.find_new_targets_task): New content found in post-to-tuf directory.
2026-06-08 18:40:59,380 INFO (monitor-worker.find_new_targets_task): Count of post-to-tuf content: 2
2026-06-08 18:40:59,380 INFO (monitor-worker.find_new_targets_task): Age of oldest post-to-tuf content: 1 seconds
2026-06-08 18:40:59,381 INFO (monitor-worker.find_new_targets_task): New targets found in post-to-tuf directory:
2026-06-08 18:40:59,381 INFO (monitor-worker.find_new_targets_task): tuf_ready_FIRST
2026-06-08 18:40:59,381 INFO (monitor-worker.find_new_targets_task): Processing: tuf_ready_FIRST
2026-06-08 18:40:59,382 INFO (monitor-worker.find_new_targets_task): Dispatching 'add-targets' task for: tuf_processing_FIRST
2026-06-08 18:40:59,401 INFO (monitor-worker._dispatch_add_targets): Added the following targets to the repository:
foo1.zip
Updated targets metadata.
2026-06-08 18:40:59,410 ERROR (logger.log_exception): TimeoutError thrown in send_task: The operation timed out.
2026-06-08 18:40:59,410 WARNING (task_queue.run_task): The operation timed out. Check status of targets-worker.
2026-06-08 18:40:59,410 ERROR (monitor-worker._dispatch_add_targets): Timeout while dispatching 'add_targets_task' task to 'targets-worker' for 'tuf_processing_FIRST'.
2026-06-08 18:40:59,410 WARNING (monitor-worker._reset_processing_target): Resetting 'tuf_processing_FIRST' to 'tuf_ready_FIRST'.
2026-06-08 18:40:59,410 INFO (monitor-worker._reset_processing_target): Renamed '/opt/post_to_tuf/tuf_processing_FIRST' to '/opt/post_to_tuf/tuf_ready_FIRST'.
2026-06-08 18:40:59,441 ERROR (logger.log_exception): FileNotFoundError thrown in find_new_targets_task: [Errno 2] No such file or directory: '/opt/post_to_tuf/tuf_processing_FIRST'
2026-06-08 18:40:59,441 ERROR (monitor-worker.find_new_targets_task): Error removing directory '/opt/post_to_tuf/tuf_processing_FIRST'
2026-06-08 18:41:04,381 INFO (monitor-worker.find_new_targets_task): New content found in post-to-tuf directory.
2026-06-08 18:41:04,382 INFO (monitor-worker.find_new_targets_task): Count of post-to-tuf content: 2
2026-06-08 18:41:04,382 INFO (monitor-worker.find_new_targets_task): Age of oldest post-to-tuf content: 6 seconds
2026-06-08 18:41:04,382 INFO (monitor-worker.find_new_targets_task): New targets found in post-to-tuf directory:
2026-06-08 18:41:04,382 INFO (monitor-worker.find_new_targets_task): tuf_ready_FIRST
2026-06-08 18:41:04,382 INFO (monitor-worker.find_new_targets_task): Processing: tuf_ready_FIRST
2026-06-08 18:41:04,383 INFO (monitor-worker.find_new_targets_task): Dispatching 'add-targets' task for: tuf_processing_FIRST
------
(Exception)
Scenario: Monitor worker detects timeout (update_timestamp_task) and resets the failed target.
Given I run "sudo fallocate -l 1 /opt/post_to_tuf/update_timestamp_task.test-flag"
And I run "sudo sudo -u rugged mkdir -p /opt/post_to_tuf/tuf_tmp_FIRST/"
And I run "sudo fallocate -l 1 /opt/post_to_tuf/tuf_tmp_FIRST/foo1.zip"
And I run "sudo sudo -u rugged mv /opt/post_to_tuf/tuf_tmp_FIRST/ /opt/post_to_tuf/tuf_ready_FIRST/"
And I run the Rugged command "rugged resume-processing"
And I wait "5" seconds
When I run the Rugged command "rugged logs --worker=monitor-worker --limit=0"
Then I should get:
"""
Checking test mode for 'update_timestamp_task' task.
Checking for existence of test flag file at '/opt/post_to_tuf/update_timestamp_task.test-flag'.
Tried to update timestamp metadata on timestamp-worker 3 times, but failed.
Resetting 'tuf_processing_FIRST' to 'tuf_ready_FIRST'.
Renaming '/opt/post_to_tuf/tuf_processing_FIRST' to '/opt/post_to_tuf/tuf_ready_FIRST'.
Renamed '/opt/post_to_tuf/tuf_processing_FIRST' to '/opt/post_to_tuf/tuf_ready_FIRST'.
"""
'Checking test mode for 'update_timestamp_task' task.' was not found in command output:
------
=== Log for monitor-worker: /var/log/rugged/rugged.log ===
2026-06-08 18:41:14,382 INFO (monitor-worker.find_new_targets_task): New content found in post-to-tuf directory.
2026-06-08 18:41:14,382 INFO (monitor-worker.find_new_targets_task): Count of post-to-tuf content: 2
2026-06-08 18:41:14,383 INFO (monitor-worker.find_new_targets_task): Age of oldest post-to-tuf content: 4 seconds
2026-06-08 18:41:14,383 INFO (monitor-worker.find_new_targets_task): New targets found in post-to-tuf directory:
2026-06-08 18:41:14,383 INFO (monitor-worker.find_new_targets_task): tuf_ready_FIRST
2026-06-08 18:41:14,383 INFO (monitor-worker.find_new_targets_task): Processing: tuf_ready_FIRST
2026-06-08 18:41:14,384 INFO (monitor-worker.find_new_targets_task): Dispatching 'add-targets' task for: tuf_processing_FIRST
2026-06-08 18:41:14,409 INFO (monitor-worker._dispatch_add_targets): Added the following targets to the repository:
foo1.zip
Updated targets metadata.
2026-06-08 18:41:14,423 ERROR (logger.log_exception): TimeoutError thrown in send_task: The operation timed out.
2026-06-08 18:41:14,423 WARNING (task_queue.run_task): The operation timed out. Check status of targets-worker.
2026-06-08 18:41:14,423 ERROR (monitor-worker._dispatch_add_targets): Timeout while dispatching 'add_targets_task' task to 'targets-worker' for 'tuf_processing_FIRST'.
2026-06-08 18:41:14,424 WARNING (monitor-worker._reset_processing_target): Resetting 'tuf_processing_FIRST' to 'tuf_ready_FIRST'.
2026-06-08 18:41:14,424 INFO (monitor-worker._reset_processing_target): Renamed '/opt/post_to_tuf/tuf_processing_FIRST' to '/opt/post_to_tuf/tuf_ready_FIRST'.
2026-06-08 18:41:14,443 ERROR (logger.log_exception): FileNotFoundError thrown in find_new_targets_task: [Errno 2] No such file or directory: '/opt/post_to_tuf/tuf_processing_FIRST'
2026-06-08 18:41:14,444 ERROR (monitor-worker.find_new_targets_task): Error removing directory '/opt/post_to_tuf/tuf_processing_FIRST'
------
(Exception)
--- Failed scenarios:
features/monitor/monitor-resilience.feature:45
features/monitor/monitor-resilience.feature:55
features/monitor/monitor-resilience.feature:84
features/monitor/monitor-resilience.feature:108
features/monitor/monitor-resilience.feature:124
features/monitor/monitor-resilience.feature:150
features/monitor/monitor-resilience.feature:168
7 scenarios (7 failed)
95 steps (81 passed, 7 failed, 7 skipped)