On November 10 2024, as a result of RPC outages on Solana, Circuit’s JLP and JLP (USDC) vaults experienced delayed trade execution. As a result, the vaults experienced a ~20% drawdown due to transaction confirmation delays and the subsequent inaccurate position sizing.
All affected users will be reimbursed for losses incurred prior to November 10.
The trading infrastructure system was designed for market making with "fire and forget" transactions. The system included a default retry mechanism for transactions that fail due to expired blockhashes, primarily implemented for non-time-critical operations. This retry mechanism was intended to ensure transaction completion for less time-sensitive operations, but proved problematic if transactions still landed during congestion periods.
The delayed transaction confirmations led to stale position data, while the trading system continued generating new orders without accounting for in-flight transactions. This situation was then exacerbated by the retry mechanism which resubmitted expired transactions, leading to a compounding effect where many resubmitted transactions were executed significantly later than expected (some up to 700 slots).
Slot latency here is the number of slots between when a transaction was sent and confirmed by the Solana network
While this is normally not a problem since the JLP vaults trades at a lower frequency, the combination of network congestion and the retry mechanism uncovered a bug which lead to a cascade of position misalignments.
Green: size of order sent Blue: current position Similar pattern can be seen across other markets traded by the vault.
The combination of delayed transactions and position mismanagement led to significant negative PnL for the vault.