Background task issues affecting multiple Hoxhunt components

Incident window (UTC):
Jan 1, 2026 02:30 – Jan 2, 2026 16:30

Summary

Between January 1st and January 2nd, we experienced a disruption in our background processing system. During this period, background jobs were not completing as expected, which resulted in delayed execution of several automated features across the platform.

Customer Impact

During the incident window, background tasks were queued but not processed. This caused delays (not data loss) in the following areas:

• Training and SAT reminders
• Training and benchmark delivery
• Training leaderboards
• Training simulation delivery
• Training automation runs
• Respond threat feedback reporting
• Respond threat evaluation
• Respond incident escalation processing

User access to the platform itself was unaffected, but the above features experienced degraded availability due to delayed background processing.

Root Cause

A large batch of scheduled background tasks triggered a performance issue in our task allocation logic. Under this unusually large workload, the system was unable to distribute tasks within required time limits.

Resolution

On January 2nd at 16:30 UTC, we restored normal background task processing by resolving the backlog that had accumulated.

Preventive Actions

We are actively working on improvements to prevent similar incidents in the future, including:

• Improving background task handling to better support large task bursts
• Introducing pacing and safeguards for large task submissions
• Strengthening monitoring to detect queue degradation earlier