Category: Maintenance

The RCD team has completed its annual summer maintenance for Palmetto 2 and Indigo. All RCD services are now available.

During the maintenance window, the team completed:

• Linux kernel updates to patch recently identified security vulnerabilities
• Minor NVIDIA driver updates
• Minor InfiniBand driver updates
• Upgrades to the Indigo Data Lake
• Updates to Open OnDemand, including the new Palmetto Desktop experience described below

These updates help maintain the security, reliability, and stability of Clemson’s research computing infrastructure as we prepare for the Fall semester.

We appreciate your patience while our team completed this work. If you experience any issues following maintenance, please submit a support ticket so the RCD team can assist.

Changes to $TMPDIR

The behavior of $TMPDIR for jobs running on Palmetto has changed.

Previously, $TMPDIR pointed to a job-specific directory under /local_scratch:

/local_scratch/slurm.$SLURM_JOBID

Following this maintenance, each user receives a namespaced /tmp directory on the compute node, and $TMPDIR now points to that location.

This change does not change the underlying storage characteristics of $TMPDIR. The directory continues to use the local storage on the compute node and is intended for temporary files created during a job. Its contents are still automatically cleaned up after the job ends.

Most applications and scripts that use the $TMPDIR environment variable should continue to work without modification. However, scripts that explicitly reference /local_scratch/slurm.$SLURM_JOBID should be updated to use $TMPDIR instead.

For more information about local scratch storage and $TMPDIR, see the ⁠Palmetto storage documentation.

New Palmetto Desktop

As part of this maintenance, RCD updated Open OnDemand with a new Palmetto Desktop experience.

The New Palmetto Desktop provides an updated way to launch and use a graphical desktop environment on Palmetto through Open OnDemand. Users can access the new desktop from the Open OnDemand interface alongside other interactive applications.

We encourage users who work with graphical applications or prefer a desktop environment on Palmetto to try the New Palmetto Desktop and share any feedback or issues with the RCD team.

The RCD team has scheduled its annual summer maintenance for Palmetto 2 and Indigo before the start of the Fall semester.

Maintenance will begin on Friday, August 7, 2026, at 9:00 AM. During the maintenance window, all RCD services will be unavailable.

The planned maintenance includes:

• Minor operating system upgrades
• Upgrades to the Indigo Data Lake

These updates are necessary to maintain the security, reliability, and stability of the cluster for the Fall semester.

There are no plans to purge scratch storage during this maintenance. However, users should remember that scratch storage is not backed up. Any important data should be stored in home or project storage.

The summer maintenance is scheduled to conclude on Wednesday, August 12, 2026, at 9:00 AM. If maintenance is completed ahead of schedule, RCD will notify users by email as services become available.

The RCD team has scheduled maintenance on Palmetto 2, the Indigo Data Lake, and other systems at the end of the Spring semester.

This work will begin on Monday, May 4th, 2026 at 9:00 AM. While maintenance work is in progress, all RCD services will be unavailable.

During the maintenance window we plan on completing the following:

• Minor OS upgrades
• Nvidia driver upgrade + CUDA 13 support
• Networking Maintenance
• Upgrade to Indigo Data Lake

There are no plans to purge scratch space during this maintenance, but users should be mindful that scratch space is never backed up and critical files should always be stored on home or project storage.

The Spring Maintenance is scheduled until  Monday, May 11th, 2026, at 9:00 AM. Users should monitor their email for updates from RCD in case the maintenance activities are completed ahead of schedule.

On Monday, December 22, Duke Energy will begin work on the power infrastructure at Clemson’s Data Center. This preventive maintenance is intended to improve power stability and reduce the risk of unscheduled outages in the future.

As a result, Palmetto 2, Indigo, and all other RCD services will be unavailable until the work is completed on Tuesday, December 23.

Any batch jobs that cannot finish before the maintenance window will be held in the queue, but all interactive jobs will be canceled. Data transfers will also be interrupted during this time.

Additionally, during this maintenance window, the K20 and K40 GPUs will be retired after 13 and 11 years of service, respectively. The nodes will remain in the cluster as CPU-only nodes.

The RCD team has scheduled a change window to perform work on the Palmetto Cluster and Indigo Data Lake on August 15th, 2025.

All services will remain online during this time and no impact to jobs is expected.

We plan on making the following changes:

• Disable AD Lockout checking on Indigo
• Removal of legacy PBS/Torque commands – users will need to switch to native Slurm commands.
  • For assistance, users should review the PBS to Slurm migration guide.
  • List of commands that will be removed:
    • mpiexec
    • pbsnodes
    • qalter
    • qdel
    • qhold
    • qrerun
    • qrls
    • qstat
    • qsub
• The default CRAN and Bioconductor location for the R modules will be set to a private mirror hosted by RCD

Please reach out to the RCD team if you have any questions or concerns.

We are excited to announce that the Spring 2025 maintenance work was completed successfully. 

All RCD services have been restored and are ready for users to access.

During the maintenance period, we completed the following:

• Critical updates to network and storage infrastructure were completed.
  • These improvements have improved performance and stability for all users of the cluster.
• Three large-memory nodes were added to the cluster. Two nodes have 3TB of memory and one node has 6TB of memory.
• More AMD Genoa nodes were added to the cluster.
• Slurm was updated to 24.11.5
• Installed AMD-optimized Compiler 5.0.0 and the following dependent modules:
  • amdblis/5.0
  • amdlibflame/5.0
  • amdlibm/5.0
  • openmpi/5.0.5
    • amdscalapack/5.0
    • lammps/20240829.1
    • osu-mircro-benchmarks/7.5
• Installed the following software:
  • gcc/14.2.0
  • intel-oneapi-compilers/2025.1.1
  • cmake/3.30.5
  • intel-oneapi-mkl/2025.1.0
  • perl/5.40.0
  • r/4.5.0
  • gromacs/gromacs-2025.1-cpu
  • gromacs/gromacs-2025.1-gpu
  • comsol/6.3
  • matlab/2025a
• Spack was upgraded to v0.23.1.
  • Users of spack will be able to install newer packages provided with this update.
  • After loading the spack module, make sure to run spack clean -a to remove old temporary build files.
• Open OnDemand was upgraded to 4.0.3
  • MATLAB application was added. This will launch a native web version of MATLAB.
  • PyMOL was added as a GUI application.
• GitLab was upgraded to v17.11.3
• ColdFront was updated to v1.15

This changelog may continue to be updated as our staff finishes their release notes.

We appreciate your patience during the maintenance period and hope that these changes will improve the user experience.

If you have any questions or have encountered post-maintenance issues, please let us know by submitting a support ticket.

The RCD team has scheduled a maintenance window to perform work on the Palmetto Cluster, Indigo Data Lake, and other systems at the end of the Spring semester.

This work will begin on Saturday, May 31st, 2025, at 9:00 AM. While maintenance work is in progress, all RCD services will be unavailable.

During the maintenance window, we plan to complete the following:

• Minor OS Upgrades
• Networking Maintenance
• System Testing and Benchmarking

There are no plans to purge scratch space during this maintenance, but users should be mindful that scratch space is never backed up and critical files should always be stored on home or project storage.

Users should expect that services will be restored no earlier than Friday, June 6th, 2025, at 5:00 PM and should monitor their email for updates from RCD.

Please feel free to reach out to RCD with any questions or concerns that you have about the maintenance work by submitting a support ticket – we would love to hear from you!

There will be a partial outage on February 24, 2025, at 9 AM. We expect the maintenance to take approximately one hour to complete. 

We have identified an unexpected issue with one of the network switches in Palmetto, requiring us to reboot the switch. This will only affect a subset of compute nodes on Palmetto. 

We are preventing new jobs from landing on the affected nodes to minimize disruptions. Jobs currently running on these nodes will be allowed to continue until the maintenance period begins. Users will still be able to log in, submit jobs, and use other RCD services, such as Open OnDemand. However, please keep in mind that you may experience extended wait times due to the affected compute nodes. 

We have chosen to perform this emergency maintenance as soon as possible to avoid a larger, unplanned outage. We apologize for any inconvenience this may cause. 

If you have any questions, please reach out to us by submitting a support ticket.

We are replacing our old Data Transfer nodes with new Data Transfer nodes on Tuesday, 02/11/2025 at 9:00 am. We expect the replacement process to take about 2 hours.

There will be no change to the Data transfer node names and details. Users will not have to update any details on their end.

Active SCP/SFTP transfers will be interrupted during this time. Globus transfers should be able to restart after the new nodes come online.

Please reach out to us if you have any questions or concerns by submitting a support ticket.

RCD and Duke Energy have completed the planned maintenance work for this week. All RCD services are now available to users again.

For details about the completed work, see the RCD maintenance post and the Duke Energy maintenance post.

If you have any questions about this maintenance work or are experiencing issues, please do not hesitate to submit a support ticket to let us know.