Author: Matt Moersen

RCD has become aware of an issue affecting the Palmetto 2 cluster that requires emergency maintenance and downtime. We are actively working to address the issue and will provide additional updates as more information becomes available.

What’s Affected

Starting now:

• New jobs will not be able to run. Jobs submitted to Palmetto 2 will remain in a pending state until service is restored.
• Currently running jobs will continue to run for now.
• Running jobs may need to be interrupted and requeued in the future if required to complete the maintenance. We will provide an update before taking this action whenever possible.

Users do not need to cancel pending jobs at this time.

We apologize for the disruption and appreciate your patience while we work to restore normal operation of Palmetto 2. Please check back for updates on the maintenance and expected restoration of service.

Research Computing and Data (RCD) is excited to announce our Fall 2026 workshop schedule.

You can view the full schedule and register for upcoming sessions here:

More workshops will be posted soon!

Featured this semester: HPC for Genomics Life Scientists 🧬

We are especially excited to offer a new HPC for Genomics Life Scientists workshop series, developed in partnership with the Clemson University Genomics and Bioinformatics Facility (CUGBF). The series is designed to help life sciences researchers understand and apply Clemson’s high-performance computing resources to genomics research.

HPC for Genomics Life Scientists: Foundations
This two-part workshop introduces the foundations of HPC for genomics research and how research computing resources can support common genomics workflows.

• Part 1: Monday, August 31
• Part 2: Wednesday, September 9
• Time: 1:00–4:30 PM
• Location: LSB 142

HPC for Genomics Life Scientists: Practical Demonstrations
This three-part workshop builds on those foundations with practical examples demonstrating how HPC resources can be applied to genomics data and analyses.

• Part 1: Monday, October 5
• Part 2: Tuesday, October 6
• Part 3: Wednesday, October 7
• Time: 1:00–4:30 PM
• Location: LSB 142

We hope to see you at a workshop this semester!

The RCD team has completed its annual summer maintenance for Palmetto 2 and Indigo. All RCD services are now available.

During the maintenance window, the team completed:

• Linux kernel updates to patch recently identified security vulnerabilities
• Minor NVIDIA driver updates
• Minor InfiniBand driver updates
• Upgrades to the Indigo Data Lake
• Updates to Open OnDemand, including the new Palmetto Desktop experience described below

These updates help maintain the security, reliability, and stability of Clemson’s research computing infrastructure as we prepare for the Fall semester.

We appreciate your patience while our team completed this work. If you experience any issues following maintenance, please submit a support ticket so the RCD team can assist.

Changes to $TMPDIR

The behavior of $TMPDIR for jobs running on Palmetto has changed.

Previously, $TMPDIR pointed to a job-specific directory under /local_scratch:

/local_scratch/slurm.$SLURM_JOBID

Following this maintenance, each user receives a namespaced /tmp directory on the compute node, and $TMPDIR now points to that location.

This change does not change the underlying storage characteristics of $TMPDIR. The directory continues to use the local storage on the compute node and is intended for temporary files created during a job. Its contents are still automatically cleaned up after the job ends.

Most applications and scripts that use the $TMPDIR environment variable should continue to work without modification. However, scripts that explicitly reference /local_scratch/slurm.$SLURM_JOBID should be updated to use $TMPDIR instead.

For more information about local scratch storage and $TMPDIR, see the ⁠Palmetto storage documentation.

New Palmetto Desktop

As part of this maintenance, RCD updated Open OnDemand with a new Palmetto Desktop experience.

The New Palmetto Desktop provides an updated way to launch and use a graphical desktop environment on Palmetto through Open OnDemand. Users can access the new desktop from the Open OnDemand interface alongside other interactive applications.

We encourage users who work with graphical applications or prefer a desktop environment on Palmetto to try the New Palmetto Desktop and share any feedback or issues with the RCD team.

The RCD team has scheduled its annual summer maintenance for Palmetto 2 and Indigo before the start of the Fall semester.

Maintenance will begin on Friday, August 7, 2026, at 9:00 AM. During the maintenance window, all RCD services will be unavailable.

The planned maintenance includes:

• Minor operating system upgrades
• Upgrades to the Indigo Data Lake

These updates are necessary to maintain the security, reliability, and stability of the cluster for the Fall semester.

There are no plans to purge scratch storage during this maintenance. However, users should remember that scratch storage is not backed up. Any important data should be stored in home or project storage.

The summer maintenance is scheduled to conclude on Wednesday, August 12, 2026, at 9:00 AM. If maintenance is completed ahead of schedule, RCD will notify users by email as services become available.

We are currently experiencing a chiller outage at Clemson’s datacenter. To help reduce the cooling load and maintain system stability, we have temporarily limited the number of jobs that can run concurrently on the cluster.

As a result, queue wait times will be longer than usual until the issue is resolved.

We apologize for the inconvenience and appreciate your patience while we work through this situation. We will provide updates here as they become available.

The Research Computing and Data (RCD) team is pleased to announce a significant increase in the scratch storage quota on Palmetto 2.

Effective immediately, the default scratch storage allocation has been increased from 5 TiB and 5 million files to 10 TiB and 10 million files per user.

Purge Policy Remains in Effect

This quota increase does not change Palmetto 2’s existing scratch space purge policy. The /scratch filesystem remains a temporary workspace intended for active research and computation, and files that meet the purge criteria remain subject to removal.

Users are responsible for understanding and complying with the scratch purge policy. Attempts to circumvent the policy—including actions intended solely to prevent files from being identified as inactive or eligible for purge—are prohibited. RCD reserves the right to remove data associated with such activity without prior notice in order to maintain the health, performance, and fairness of the shared filesystem.

Researchers should ensure that important data is stored in appropriate long-term storage locations and should not rely on /scratch for archival or permanent storage.

Why the Change?

As research workflows continue to grow in scale and complexity, many users are working with increasingly large datasets, simulation outputs, machine learning training data, and high-throughput analysis pipelines. Based on user feedback and observed usage patterns, RCD has expanded scratch storage capacity to better support these evolving computational needs.

The updated limits provide:

• 2× more scratch storage capacity (10 TiB)
• 2× higher file count allowance (10 million files)
• Greater flexibility for data-intensive workflows
• Reduced need for users to actively manage scratch space during large projects

About Scratch Storage

The /scratch filesystem on Palmetto 2 is designed for temporary, high-performance storage during active computational work. Users should continue to treat scratch space as temporary storage and maintain copies of important data in appropriate long-term storage locations.

While the per-user quota has been increased, users should be aware that scratch storage is a shared resource with finite capacity. The new quota represents the maximum amount of storage and number of files a user may consume, but it does not guarantee that every user can simultaneously utilize their full allocation. Responsible use of scratch space helps ensure that the filesystem remains performant and available for the entire Palmetto community.

Users can monitor their current scratch usage and quota limits at any time using the checkquota command. Additional information about scratch storage, quotas, and purge policies is available in the Palmetto documentation.

As a reminder:

• Scratch storage is intended for active jobs and working datasets.
• Critical research data should be backed up elsewhere.
• Users are encouraged to periodically remove unneeded files and directories.

The Research Computing and Data (RCD) team is excited to announce our Summer 2026 workshop series, available free of charge to all Clemson University students, faculty, and staff.

Our series will cover various high-performance computing, machine learning, and software development topics. Here’s a summary of what we’ll cover:

• Introduction to Linux
• Introduction to Research Computing on Palmetto
• Introduction to Nextflow
• AI Agent Skills and GPTs for Research
• AI Code Assistants for HPC on Palmetto
• Research Computing on Kubernetes
• Inside the Agent: Demystifying LLM Agents in Practice

You can learn more about the details of each workshop, the schedule, and registration links on the upcoming live training sessions page of our documentation site.

We look forward to seeing you this summer and helping you make the most of Clemson’s research computing resources!

We are excited to announce that the Spring 2026 maintenance work was completed successfully.

All RCD services have been restored and are ready for users to access.

The RCD completed the following tasks:

• Critical updates to network and storage infrastructure were completed.
  • These updates have improved performance and stability for all users of the cluster
• Critical Kernel updates to address CVE-2026-3141 (Copy Fail)
• Mitigations to address CVE-2026-43284 and CVE-2026-43500 (Dirty Frag)
• Nvidia driver upgraded to 580 branch (CUDA 13 support)
  • cuda/13.0.2 module added
• Slurm was upgraded to 25.11.5
• Open OnDemand was upgraded to 4.1.5
• GitLab was updated to v18.11.2

We appreciate your patience during the maintenance period and hope that these changes will improve the user experience.

If you have any questions or have encountered post-maintenance issues, please let us know by submitting a support ticket.

The RCD team has scheduled maintenance on Palmetto 2, the Indigo Data Lake, and other systems at the end of the Spring semester.

This work will begin on Monday, May 4th, 2026 at 9:00 AM. While maintenance work is in progress, all RCD services will be unavailable.

During the maintenance window we plan on completing the following:

• Minor OS upgrades
• Nvidia driver upgrade + CUDA 13 support
• Networking Maintenance
• Upgrade to Indigo Data Lake

There are no plans to purge scratch space during this maintenance, but users should be mindful that scratch space is never backed up and critical files should always be stored on home or project storage.

The Spring Maintenance is scheduled until  Monday, May 11th, 2026, at 9:00 AM. Users should monitor their email for updates from RCD in case the maintenance activities are completed ahead of schedule.

On Monday, December 22, Duke Energy will begin work on the power infrastructure at Clemson’s Data Center. This preventive maintenance is intended to improve power stability and reduce the risk of unscheduled outages in the future.

As a result, Palmetto 2, Indigo, and all other RCD services will be unavailable until the work is completed on Tuesday, December 23.

Any batch jobs that cannot finish before the maintenance window will be held in the queue, but all interactive jobs will be canceled. Data transfers will also be interrupted during this time.

Additionally, during this maintenance window, the K20 and K40 GPUs will be retired after 13 and 11 years of service, respectively. The nodes will remain in the cluster as CPU-only nodes.