features/monitor/reset_semaphores.featureRunning behat features/monitor/reset_semaphores.feature results in:
@rugged @monitor @scheduler @reset-semaphores @gitlab-204
Feature: Stale sempahores are reset automatically.
In order to keep a TUF systems running smoothly,
As a TUF administrator
I need to ensure that stale semaphores are reset automatically.
Background:
Given I reset Rugged
And I run "sudo rm -rf /opt/post_to_tuf/*"
Scenario: Ensure that monitor worker has necessary config for testing 'reset-semaphores'.
When I run the Rugged command "rugged config --worker=monitor-worker"
Then I should get:
"""
scheduler_log_level DEBUG
task_timeout 3
scheduler_reset_period 5.0
"""
'scheduler_log_level DEBUG' was not found in command output:
------
=== Configuration for monitor-worker ===
broker_connection_string REDACTED
scheduler_log_level INFO
scheduler_refresh_period 3600.0
scheduler_scan_period 5.0
workers {'snapshot': {'name': 'snapshot-worker'}, 'targets': {'name': 'targets-worker'}, 'timestamp': {'name': 'timestamp-worker'}}
log_file /var/log/rugged/rugged.log
log_format %(asctime)s %(levelname)s (%(module)s.%(funcName)s): %(message)s
repo_path /var/rugged/tuf_repo
inbound_targets_path /var/rugged/incoming_targets
post_to_tuf_path /opt/post_to_tuf
repo_targets_path /var/rugged/tuf_repo/targets
repo_metadata_path /var/rugged/tuf_repo/metadata
roles {'root': {'threshold': 1, 'expiry': 31536000}, 'timestamp': {'threshold': 1, 'expiry': 86400}, 'snapshot': {'threshold': 1, 'expiry': 604800}, 'targets': {'threshold': 1, 'expiry': 604800}}
expiry_refresh_threshold 43200
consistent_snapshot False
print_host_headers True
use_hashed_bins False
number_of_bins 16
hashed_bins_key_name targets
celery_worker_max_memory_per_child 0
delete_targets_after_signing False
task_timeout 10
wait_for_processing_task_to_complete_timeout 30
wait_for_refreshing_task_to_complete_timeout 15
stale_semaphore_age_thresholds {'tuf_paused': 3600, 'tuf_processing_': 300, 'tuf_refreshing_expiry': 60}
------
(Exception)
Scenario: Ensure that monitor worker is polling for stale semaphores.
Given I wait "6" seconds
When I run the Rugged command "rugged logs --worker=monitor-worker --limit=0"
Then I should get:
"""
Received reset-semaphores task.
"""
'Received reset-semaphores task.' was not found in command output:
------
=== Log for monitor-worker: /var/log/rugged/rugged.log ===
------
(Exception)
Scenario: Fresh tuf-processing semaphores are not reset automatically.
Given I run the Rugged command "mkdir -p /opt/post_to_tuf/tuf_processing_TIMESTAMP/"
When I wait "6" seconds
Then the "/opt/post_to_tuf/tuf_processing_TIMESTAMP" directory should exist
And the "/opt/post_to_tuf/tuf_ready_TIMESTAMP" directory should not exist
When I run the Rugged command "rugged logs --worker=monitor-worker --limit=0"
Then I should get:
"""
Checking for stale processing directories.
"""
'Checking for stale processing directories.' was not found in command output:
------
=== Log for monitor-worker: /var/log/rugged/rugged.log ===
2026-06-08 18:35:59,378 INFO (monitor-worker.find_new_targets_task): New content found in post-to-tuf directory.
2026-06-08 18:35:59,378 INFO (monitor-worker.find_new_targets_task): Count of post-to-tuf content: 1
2026-06-08 18:35:59,379 INFO (monitor-worker.find_new_targets_task): Age of oldest post-to-tuf content: 5 seconds
2026-06-08 18:35:59,379 INFO (monitor-worker.find_new_targets_task): No new targets found.
------
(Exception)
And I should not get:
"""
Found stale processing directory. Resetting.
"""
Scenario: Fresh refreshing-expiry semaphores are not reset automatically.
Given I run the Rugged command "mkdir -p /opt/post_to_tuf/tuf_refreshing_expiry/"
When I wait "6" seconds
Then the "/opt/post_to_tuf/tuf_refreshing_expiry" directory should exist
When I run the Rugged command "rugged logs --worker=monitor-worker --limit=0"
Then I should get:
"""
Checking for stale expiry-refreshing flag.
"""
'Checking for stale expiry-refreshing flag.' was not found in command output:
------
=== Log for monitor-worker: /var/log/rugged/rugged.log ===
2026-06-08 18:36:04,377 INFO (monitor-worker.find_new_targets_task): New content found in post-to-tuf directory.
2026-06-08 18:36:04,377 INFO (monitor-worker.find_new_targets_task): Count of post-to-tuf content: 1
2026-06-08 18:36:04,378 INFO (monitor-worker.find_new_targets_task): Age of oldest post-to-tuf content: 1 seconds
2026-06-08 18:36:04,378 INFO (monitor-worker.find_new_targets_task): Monitor worker is refreshing expiry periods. Waiting for next scan to continue.
2026-06-08 18:36:09,378 INFO (monitor-worker.find_new_targets_task): New content found in post-to-tuf directory.
2026-06-08 18:36:09,378 INFO (monitor-worker.find_new_targets_task): Count of post-to-tuf content: 1
2026-06-08 18:36:09,378 INFO (monitor-worker.find_new_targets_task): Age of oldest post-to-tuf content: 6 seconds
2026-06-08 18:36:09,378 INFO (monitor-worker.find_new_targets_task): Monitor worker is refreshing expiry periods. Waiting for next scan to continue.
------
(Exception)
And I should not get:
"""
Found stale expiry-refreshing flag. Resetting.
"""
Scenario: Stale tuf-processing semaphores are reset automatically.
Given I run the Rugged command "mkdir -p /opt/post_to_tuf/tuf_processing_TIMESTAMP/"
And I run the Rugged command "touch -d yesterday /opt/post_to_tuf/tuf_processing_TIMESTAMP/"
When I wait "6" seconds
When I run the Rugged command "rugged logs --worker=monitor-worker --limit=0"
Then I should get:
"""
Checking for stale processing directories.
Found stale processing directory. Resetting.
Resetting processing directory ('/opt/post_to_tuf/tuf_processing_TIMESTAMP') to ready state.
Renaming '/opt/post_to_tuf/tuf_processing_TIMESTAMP' to '/opt/post_to_tuf/tuf_ready_TIMESTAMP'.
Renamed '/opt/post_to_tuf/tuf_processing_TIMESTAMP' to '/opt/post_to_tuf/tuf_ready_TIMESTAMP'.
"""
'Checking for stale processing directories.' was not found in command output:
------
=== Log for monitor-worker: /var/log/rugged/rugged.log ===
2026-06-08 18:36:14,378 INFO (monitor-worker.find_new_targets_task): New content found in post-to-tuf directory.
2026-06-08 18:36:14,378 INFO (monitor-worker.find_new_targets_task): Count of post-to-tuf content: 1
2026-06-08 18:36:14,379 INFO (monitor-worker.find_new_targets_task): Age of oldest post-to-tuf content: 86402 seconds
2026-06-08 18:36:14,379 INFO (monitor-worker.find_new_targets_task): No new targets found.
------
(Exception)
Scenario: Stale refreshing-expiry semaphores are reset automatically.
Given I run the Rugged command "mkdir -p /opt/post_to_tuf/tuf_refreshing_expiry/"
And I run the Rugged command "touch -d yesterday /opt/post_to_tuf/tuf_refreshing_expiry/"
When I wait "6" seconds
Then the "/opt/post_to_tuf/tuf_refreshing_expiry" directory should not exist
Unexpected directory '/opt/post_to_tuf/tuf_refreshing_expiry' was found. (RuntimeException)
When I run the Rugged command "rugged logs --worker=monitor-worker --limit=0"
Then I should get:
"""
Checking for stale expiry-refreshing flag.
Found stale expiry-refreshing flag. Resetting.
Deleting 'refresh-expiry' task semaphore directory: /opt/post_to_tuf/tuf_refr
eshing_expiry
Deleted 'refresh-expiry' task semaphore directory: /opt/post_to_tuf/tuf_refres
hing_expiry
"""
--- Failed scenarios:
features/monitor/reset_semaphores.feature:11
features/monitor/reset_semaphores.feature:20
features/monitor/reset_semaphores.feature:29
features/monitor/reset_semaphores.feature:44
features/monitor/reset_semaphores.feature:58
features/monitor/reset_semaphores.feature:72
6 scenarios (6 failed)
41 steps (31 passed, 6 failed, 4 skipped)