As most of you are likely already aware, the University System of Georgia (USG) is experiencing a critical outage affecting our ERP systems, including financial, HCM, and student information systems. This issue started late Friday afternoon, and the root issue lies with USG’s Storage Area Network (SAN), a Dell product that provides essential storage for both application and data servers. This storage array functions as a single, unified block of storage. When it encounters critical issues, all USG ITS services relying on it can be impacted. It is designed with high availability and has multiple redundant layers in both its hardware and software, and such prolonged disruptions are highly unusual. The last comparable event for ITS was over 15 years ago.
In his latest update, Dr. Tim Chester, Vice Chancellor and Chief Information Officer for the University System of Georgia, reports the following:
“Dell continues to work on this incident, using resources from their R&D team. They believe the root cause is a software bug impacting the core SAN OS, and they are preparing a software fix. At their last report, at approximately 11 a.m. this morning, they reported that they needed another four to six hours to validate their patch and develop a roadmap to resolving the issue and restoring our services.
In the meantime, the ITS and Shared Service Center teams have developed a plan to work today to restore OneUSG Connect, if necessary, from our backup data center at UGA. If necessary, we would implement that plan so that OneUSG Connect would be in production and available by 8 a.m. Sunday morning. Our Georgia Best (Banner) teams are working on a similar plan to have ITS hosted instances of Banner available by 8 a.m. Monday morning. These are contingency plans, and our hope is that Dell will resolve the root cause later today. When that happens, we will restore services as fast as possible.”
I am in direct contact with Dr. Chester and he is aware that we plan to open registration at 7 AM on Monday, October 28th. He is extremely hopeful that this will be resolved later today and that all services will be fully restored in plenty of time for us to resume business as usual before Monday morning.
This is an unfortunate reminder that there are no absolutes when it comes to information technology. At every level, we try to procure quality products, implement reliable solutions and leverage reputable systems. In the end, technology sometimes fails and we have to be well prepared to recover as gracefully as possible with minimal impact to operations.
I continue to monitor the situation and will maintain frequent contact with Dr. Chester. GCSU IT personnel are engaged and stand ready to assist as we can. As significant information becomes available, we will communicate that to the GCSU community. If you care to monitor the situation on your own, you can refer to the USG services status page at: https://status.usg.edu.
We thank you for your patience and understanding. Please reach out to me directly with any specific questions or concerns.
Brian
Brian T Watson, Interim Chief Information Officer
Georgia College & State University