Ticket #1719780

Hi Team, I wanted to make everyone aware of a pattern we are starting to see across multiple customer sites involving Lenovo ThinkSystem ST550 servers equipped with the RAID 530-8i controller. We are currently working with a customer where the server began experiencing the following symptoms: * System appears to hang or takes an unusually long time during UEFI / DXE initialization * Hardware diagnostics for storage devices run extremely slowly * System may eventually reach BIOS, but storage diagnostics remain very slow * RAID controller suspected to be causing the delay during device initialization While troubleshooting this issue, Ryan who is currently onsite at Raytheon in Texas reported very similar behavior on another ST550 server. After physically inspecting the system and removing the controller, they confirmed that it is also using the RAID 530-8i controller and the symptoms appear consistent. Including an earlier case this year with Astronatics , this would now represent three customer environments where we have seen similar behavior involving the same controller. The controller installed in these systems is: Lenovo ThinkSystem RAID 530-8i Part Numbers: • 02JG102 (FRU) • 00MW544 (Option Kit) Technical Observation The RAID 530-8i is based on the Broadcom / LSI SAS3108 RAID-on-Chip platform. One characteristic of this controller is that when a disk begins responding slowly or starts failing, the controller can enter extended retry loops while attempting to communicate with the drive. During POST this can appear as: * UEFI → DXE INIT → system appears frozen In reality the controller is performing repeated drive communication retries: Drive query → timeout → retry → retry → retry Each retry can take several seconds, and when multiple retries occur the system can appear to hang for several minutes. Because Lenovo hardware diagnostics communicate through the RAID controller, this behavior also explains why storage diagnostics appear extremely slow. This pattern is often associated with: * aging disks beginning to degrade * slow-responding drives * controller retry behavior during initialization Recommended Mitigation As part of troubleshooting we have recommended replacing the controller with the newer: Lenovo ThinkSystem RAID 730-8i Part Numbers: • 01PE852 (FRU) • 7Y37A01083 (Option Kit) • 4Y37A09722 (Option Kit) The RAID 730-8i is also based on the Broadcom MegaRAID architecture and is compatible with the ST550 platform. Because RAID configuration metadata is stored on the disks themselves, the replacement controller should detect the existing array and prompt to: Import Foreign Configuration This will bring the array back online without recreating it. Preventative Practices Given the pattern we are observing, it may also be worth keeping the following in mind for similar systems: 1. Monitor drives for predictive failure indicators (SMART warnings, media errors, command timeouts, slow response warnings) 2. Replace drives once predictive failure warnings appear rather than waiting for complete failure 3. Keep RAID controller firmware and drive firmware updated 4. Consider maintaining a hot spare disk where possible Replacing disks early can prevent the controller from entering long retry loops during boot, which helps avoid the DXE initialization delays we are seeing. At this point we cannot definitively say whether the issue is related to: * controller firmware behavior * hardware degradation of the controller * interaction with aging disks However, the repeated occurrence across multiple ST550 deployments suggests it may be something worth monitoring going forward. If anyone else has encountered similar behavior with ST550 systems using the RAID 530-8i, please share your observations. Also one last note, when we deploy physical servers to customer, we should probably encourage them to lookup their RAID controller card, and buy a backup replacement ahead of time. Thanks, [This is Jeremiah Soliz's card. Their email is [email] Their phone number is +[phone].]