---------------------------------------------------------------------------------------------------------------
EGI BROADCAST TOOL : https://operations-portal.egi.eu/broadcast/send
---------------------------------------------------------------------------------------------------------------
Publication from : Alastair Dewhurst <alastair.dewhurst(a)cern.ch>
----------------------------------------------------------------------------------------------------------------
Dear all,
As of 16:00 UTC, the network failure that caused the loss of the databases services at RAL has been resolved, although we are currently running in a non-resilient manner. We will be running at risk over night.
The GOCDB is back in full production (i.e. both writes and reads).
The RAL Tier-1 Disk storage (Echo) and Tape storage (Antares) are back in production. The batch farm will be brought back online tomorrow morning. Further downtimes will be noted in the GOCDB (rather than more announcements).
Thank you for your patience
Alastair
RAL Tier-1 Manager
----------------------------------------------------------------------------------------------------------------
link to this broadcast : https://operations-portal.egi.eu/broadcast/archive/2946
----------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------
EGI BROADCAST TOOL : https://operations-portal.egi.eu/broadcast/send
---------------------------------------------------------------------------------------------------------------
Publication from : Alastair Dewhurst <alastair.dewhurst(a)cern.ch>
----------------------------------------------------------------------------------------------------------------
Dear all,
The site downtime at RAL caused by the core network failure is still ongoing. We are working on recovering the databases currently.
As mentioned in the previous update, the GOCDB is working (at risk) in read-only mode: https://goc.egi.eu/
Echo (Disk storage) is functioning but in a degraded state.
Antares (Tape storage) is currently down however we can bring the service back quickly once the network has stabilized.
We are expecting the batch farm to remain down until tomorrow morning.
We will provide an update by 14:00 UTC and hope to be in a position to announce the return of some production services before the end of the day.
Thanks,
Alastair
RAL Tier-1 Manager
----------------------------------------------------------------------------------------------------------------
link to this broadcast : https://operations-portal.egi.eu/broadcast/archive/2945
----------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------
EGI BROADCAST TOOL : https://operations-portal.egi.eu/broadcast/send
---------------------------------------------------------------------------------------------------------------
Publication from : Alastair Dewhurst <alastair.dewhurst(a)cern.ch>
----------------------------------------------------------------------------------------------------------------
Dear All
At shortly after 15:00 local time (14:00 UTC) today (17/10/22) there was a major failure of the RAL core network. All services are currently down including the GOCDB which would normally be used to declare outages.
After 6 hours of work we have only been able to partially restore network connectivity and we are thus declaring a whole site outage until at least midday (11:00 UTC) tomorrow (18/10/22).
Alastair
RAL Tier-1 Manager
----------------------------------------------------------------------------------------------------------------
link to this broadcast : https://operations-portal.egi.eu/broadcast/archive/2943
----------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------
EGI BROADCAST TOOL : https://operations-portal.egi.eu/broadcast/send
---------------------------------------------------------------------------------------------------------------
Publication from : Adrian Coveney <gocdb-admins(a)mailman.egi.eu>
----------------------------------------------------------------------------------------------------------------
Due to network issues at RAL, GOCDB remains unavailable. Staff are on site investigating.
We will provide a further update once more information is available, but that may not be until tomorrow. Connectivity to GOCDB may be restored in the meantime.
----------------------------------------------------------------------------------------------------------------
link to this broadcast : https://operations-portal.egi.eu/broadcast/archive/2942
----------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------
EGI BROADCAST TOOL : https://operations-portal.egi.eu/broadcast/send
---------------------------------------------------------------------------------------------------------------
Publication from : Adrian Coveney <gocdb-admins(a)mailman.egi.eu>
----------------------------------------------------------------------------------------------------------------
Due to network issues at RAL, GOCDB is currently unavailable. An update will be provided within the next hour.
----------------------------------------------------------------------------------------------------------------
link to this broadcast : https://operations-portal.egi.eu/broadcast/archive/2941
----------------------------------------------------------------------------------------------------------------