Downtime in the weekend 4/11 - 6/11

A major security vulnerability has been discovered in linux. We need downtime on the entire system this weekend

  • Red Hat 7 machines needs to be booted with the latest kernel
  • Red Hat 6 machines directly connected to the storage system needs to be reinstalled with Red Hat 7
  • Red Hat 6 machines not directly connected to the storage system needs to be booted with the latest kernel
  • The StorNext storage system needs to be upgraded to the latest version.
  • The OS-X Mac Pro's needs to be shut down since StorNext will be unavailable.
  • We will also use this opportunity to get som other upgrades done that requires downtime.

A consequence of this is that owl1-16, nekkar, achernar, papsukal and aramec will no longer have infiniband, as Intel has not been able to provide software that supports the latest kernels.

Home directories, personal web pages, the license servers and the login server will be unavailable.We start shutdown on Friday at 17.00.

More info on the linux security hole can be found here:

http://www.digi.no/artikler/ni-ar-gammel-linux-sarbarhet-kan-vaere-under-angrep/363358
https://dirtycow.ninja/
https://access.redhat.com/security/vulnerabilities/2706661

STATUS:

09.11.16 15:00 Charybdis1-3 and proxima is ready. Working on beehive26-28.

09.11.16 07:20 owl17-24 is ready.

08.11.16 12:05 Samba problems fixed.Screen driver problems fixed.

08.11.16 09:15 owl1 - owl16, achernar, papsukal and nekkar are ready.

07.11.16 14:17 NFS-server problems fixed. Waiting for USIT to fix Samba problems. Some screen driver problems after Red Hat released version 7.3 of the operating systems. Being fixed now. All Mac OS-X machines up. Som work remains on owl2-owl23, charybdis1-3, proxima, beehive26-28, achernar, papsukal, and nekkar.

07.11.16 08:00 Some problems with the nfs server. Seems to be related to file locking (thanks Sigurd for pointing this out). Backing down NFS-server to RHEL6. At the moment workstations mounting home directories over NFS will hang (among others, all student machines). 

06.11.16 19:25 All linux workstations are now up. login.astro.uio.no (tsih) is also up. Some OS-X workstations refuse to boot. Will be checked in the morning. Some work remain on: owl2-owl23, charybdis1-3, proxima, beehive26-28, achernar, papsukal, and nekkar. aramec seems to have a hardware problem.

06.11.16 08:30 Upgrading RHEL7 workstations.

05.11.16 21:20 Metadata controllers upgraded. Halfway through RHEL6 to RHEL7 reinstalls. Enough for tonight. Going home.

05.11.16 20:20 Some metadata switch stack problems. Solved. Upgrading StorNext metadata controllers.

05.11.16 16:25 Storage firmware update complete. Starting StorNext servers. RHEL6 to RHEL7 updates in progress

05.11.16 12:40 Server config in progress. Switch config complete. Firmware upgrade for storage in progress

05.11.16 10:20 Servers down. Firmware upgrades on switches and storage in progress. Reinstall of servers.

05.11.16 09:00 Client shutdown complete, server shutdown in progress

 

 

 

By Torben Leifsen
Published Nov. 1, 2016 11:47 AM - Last modified Nov. 9, 2016 3:04 PM