features/monitor/refresh_expiry_resilience.featureRunning behat features/monitor/refresh_expiry_resilience.feature results in:
@rugged @refresh-expiry-resilience @gitlab-173
Feature: Refresh expiry tasks on the monitor-worker are resilient to network instability.
In order to run reliably for extended periods
As a TUF operator
I need automatic expiry-refresh tasks to tolerate occasional network instability.
Background:
Given I reset Rugged
And I run "sudo rm -rf /opt/post_to_tuf/*"
And I run "sudo cp features/fixtures/config/expiry/short_targets_expiry.yaml /var/rugged/.config/rugged/config.yaml"
And I run the Rugged command "rugged pause-processing"
And I run the Rugged command "rugged generate-keys --local"
And I run the Rugged command "rugged logs --truncate --worker=monitor-worker"
And I run the Rugged command "rugged initialize --local"
Scenario: Ensure that monitor worker has necessary config for testing 'refresh-expiry' resilience.
When I run the Rugged command "rugged config --worker=monitor-worker"
Then I should get:
"""
scheduler_log_level DEBUG
task_timeout 1
monitor_enable_network_instability_resiliency_test_mode True
scheduler_refresh_period 5.0
"""
'scheduler_log_level DEBUG' was not found in command output:
------
=== Configuration for monitor-worker ===
broker_connection_string REDACTED
scheduler_log_level INFO
scheduler_refresh_period 3600.0
scheduler_scan_period 5.0
workers {'snapshot': {'name': 'snapshot-worker'}, 'targets': {'name': 'targets-worker'}, 'timestamp': {'name': 'timestamp-worker'}}
log_file /var/log/rugged/rugged.log
log_format %(asctime)s %(levelname)s (%(module)s.%(funcName)s): %(message)s
repo_path /var/rugged/tuf_repo
inbound_targets_path /var/rugged/incoming_targets
post_to_tuf_path /opt/post_to_tuf
repo_targets_path /var/rugged/tuf_repo/targets
repo_metadata_path /var/rugged/tuf_repo/metadata
roles {'root': {'threshold': 1, 'expiry': 31536000}, 'timestamp': {'threshold': 1, 'expiry': 86400}, 'snapshot': {'threshold': 1, 'expiry': 604800}, 'targets': {'threshold': 1, 'expiry': 604800}}
expiry_refresh_threshold 43200
consistent_snapshot False
print_host_headers True
use_hashed_bins False
number_of_bins 16
hashed_bins_key_name targets
celery_worker_max_memory_per_child 0
delete_targets_after_signing False
task_timeout 10
wait_for_processing_task_to_complete_timeout 30
wait_for_refreshing_task_to_complete_timeout 15
stale_semaphore_age_thresholds {'tuf_paused': 3600, 'tuf_processing_': 300, 'tuf_refreshing_expiry': 60}
------
(Exception)
Scenario Outline: If a worker task times out, it is retried.
Given I run the Rugged command "touch /opt/post_to_tuf/<WORKER_TASK>.test-flag"
And I run the Rugged command "rugged resume-processing"
And I wait "10" seconds
When I run the Rugged command "rugged logs --worker=monitor-worker --limit=0"
And I should get:
"""
Received refresh-expiry task.
Starting <VERB> of <OBJECT> on <WORKER>.
Diverting '<WORKER_TASK>' task from '<WORKER>' to 'sleep' on test worker, to force a timeout.
RuggedTimeoutError: The operation exceeded the given deadline.
Failed to <VERB> <OBJECT> on <WORKER>.
Task timed-out during <VERB> of <OBJECT> on <WORKER>.
Retry 1/2: Retrying <VERB> of <OBJECT> on <WORKER>.
Retry 2/2: Retrying <VERB> of <OBJECT> on <WORKER>.
Tried to <VERB> <OBJECT> on <WORKER> 3 times, but failed.
RuggedMetadataError: Tried to <VERB> <OBJECT> on <WORKER> 3 times, but failed.
"""
And I should not get:
"""
Completed <VERB> of <OBJECT> on <WORKER>.
"""
Examples:
| WORKER_TASK | WORKER | VERB | OBJECT |
| refresh_expiry_task | targets-worker | refresh | expiring metadata |
Failed step: And I should get:
'Received refresh-expiry task.' was not found in command output:
------
=== Log for monitor-worker: /var/log/rugged/rugged.log ===
2026-06-08 18:41:49,382 WARNING (monitor-worker.find_new_targets_task): The monitor worker is paused.
2026-06-08 18:41:49,382 INFO (monitor-worker.find_new_targets_task): Skipping find-new-targets task.
2026-06-08 18:41:54,381 INFO (monitor-worker.find_new_targets_task): New content found in post-to-tuf directory.
2026-06-08 18:41:54,381 INFO (monitor-worker.find_new_targets_task): Count of post-to-tuf content: 1
2026-06-08 18:41:54,381 INFO (monitor-worker.find_new_targets_task): Age of oldest post-to-tuf content: 5 seconds
2026-06-08 18:41:54,381 INFO (monitor-worker.find_new_targets_task): No new targets found.
2026-06-08 18:41:59,381 INFO (monitor-worker.find_new_targets_task): New content found in post-to-tuf directory.
2026-06-08 18:41:59,381 INFO (monitor-worker.find_new_targets_task): Count of post-to-tuf content: 1
2026-06-08 18:41:59,381 INFO (monitor-worker.find_new_targets_task): Age of oldest post-to-tuf content: 10 seconds
2026-06-08 18:41:59,382 INFO (monitor-worker.find_new_targets_task): No new targets found.
------
(Exception)
| get_expiring_metadata_task | targets-worker | fetch | expiring metadata |
Failed step: And I should get:
'Received refresh-expiry task.' was not found in command output:
------
=== Log for monitor-worker: /var/log/rugged/rugged.log ===
2026-06-08 18:42:04,380 WARNING (monitor-worker.find_new_targets_task): The monitor worker is paused.
2026-06-08 18:42:04,381 INFO (monitor-worker.find_new_targets_task): Skipping find-new-targets task.
2026-06-08 18:42:09,382 INFO (monitor-worker.find_new_targets_task): New content found in post-to-tuf directory.
2026-06-08 18:42:09,382 INFO (monitor-worker.find_new_targets_task): Count of post-to-tuf content: 1
2026-06-08 18:42:09,383 INFO (monitor-worker.find_new_targets_task): Age of oldest post-to-tuf content: 5 seconds
2026-06-08 18:42:09,383 INFO (monitor-worker.find_new_targets_task): No new targets found.
2026-06-08 18:42:14,381 INFO (monitor-worker.find_new_targets_task): New content found in post-to-tuf directory.
2026-06-08 18:42:14,382 INFO (monitor-worker.find_new_targets_task): Count of post-to-tuf content: 1
2026-06-08 18:42:14,382 INFO (monitor-worker.find_new_targets_task): Age of oldest post-to-tuf content: 10 seconds
2026-06-08 18:42:14,382 INFO (monitor-worker.find_new_targets_task): No new targets found.
------
(Exception)
| update_snapshot_task | snapshot-worker | update | snapshot metadata |
Failed step: And I should get:
'Received refresh-expiry task.' was not found in command output:
------
=== Log for monitor-worker: /var/log/rugged/rugged.log ===
2026-06-08 18:42:24,384 INFO (monitor-worker.find_new_targets_task): New content found in post-to-tuf directory.
2026-06-08 18:42:24,384 INFO (monitor-worker.find_new_targets_task): Count of post-to-tuf content: 1
2026-06-08 18:42:24,385 INFO (monitor-worker.find_new_targets_task): Age of oldest post-to-tuf content: 4 seconds
2026-06-08 18:42:24,385 INFO (monitor-worker.find_new_targets_task): No new targets found.
2026-06-08 18:42:29,381 INFO (monitor-worker.find_new_targets_task): New content found in post-to-tuf directory.
2026-06-08 18:42:29,381 INFO (monitor-worker.find_new_targets_task): Count of post-to-tuf content: 1
2026-06-08 18:42:29,381 INFO (monitor-worker.find_new_targets_task): Age of oldest post-to-tuf content: 9 seconds
2026-06-08 18:42:29,381 INFO (monitor-worker.find_new_targets_task): No new targets found.
------
(Exception)
| update_timestamp_task | timestamp-worker | update | timestamp metadata |
Failed step: And I should get:
'Received refresh-expiry task.' was not found in command output:
------
=== Log for monitor-worker: /var/log/rugged/rugged.log ===
2026-06-08 18:42:39,381 INFO (monitor-worker.find_new_targets_task): New content found in post-to-tuf directory.
2026-06-08 18:42:39,381 INFO (monitor-worker.find_new_targets_task): Count of post-to-tuf content: 1
2026-06-08 18:42:39,381 INFO (monitor-worker.find_new_targets_task): Age of oldest post-to-tuf content: 3 seconds
2026-06-08 18:42:39,382 INFO (monitor-worker.find_new_targets_task): No new targets found.
2026-06-08 18:42:44,382 INFO (monitor-worker.find_new_targets_task): New content found in post-to-tuf directory.
2026-06-08 18:42:44,382 INFO (monitor-worker.find_new_targets_task): Count of post-to-tuf content: 1
2026-06-08 18:42:44,382 INFO (monitor-worker.find_new_targets_task): Age of oldest post-to-tuf content: 8 seconds
2026-06-08 18:42:44,382 INFO (monitor-worker.find_new_targets_task): No new targets found.
------
(Exception)
--- Failed scenarios:
features/monitor/refresh_expiry_resilience.feature:16
features/monitor/refresh_expiry_resilience.feature:55
features/monitor/refresh_expiry_resilience.feature:56
features/monitor/refresh_expiry_resilience.feature:57
features/monitor/refresh_expiry_resilience.feature:58
5 scenarios (5 failed)
61 steps (52 passed, 5 failed, 4 skipped)