Northwestern Information Technology is pleased to announce Quest scratch space, a new service available free of charge to all users in active Quest allocations. Quest users can apply for Quest scratch space beginning November 1st.
Background
For the last year, members of Northwestern IT’s Research Computing Services (RCS) and Research Cyberinfrastructure (RCI) teams came together in a Quest scratch project team to bring global scratch space to Quest. Scratch space is one of our initiatives to extend storage options for researchers on Quest without additional costs to users; as part of this initiative, we’ve also increased all Quest General Access Allocation I storage quotas from 500 GB to 1 TB.
To create a Quest scratch space offering that optimizes both usefulness for researchers and supportability by existing staff, the Quest scratch project team researched scratch space offerings at 30 peer institutions to survey their scratch implementation decisions, specifically:
- How much scratch storage is offered
- How long is scratch data kept before it’s deleted
- What policies are in place to keep scratch space from filling up
That last question was of particular interest to the Quest scratch project team because if scratch space fills up completely, any job that writes to it will fail, a situation the team has worked hard to prevent.
Informed by the experiences of peer institutions and by the research and support implications of various options, the Quest scratch project team designed Quest scratch space by creating policies and developing code that maintains Quest scratch space automatically. After extensive internal testing and beta testing in the community, Quest scratch space is now ready for everyone.
What is Quest scratch space?
Quest scratch space is high-speed storage on Quest’s file system for storing intermediate calculations or datasets that can be deleted once they’ve been used to reach a result. Quest scratch space takes its name from scratch paper used to write out the intermediate steps of solving a problem – when the answer is reached, the scratch paper with the intermediate steps is thrown out because it’s not needed anymore. Just like with scratch paper, intermediate data stored in Quest scratch is “thrown out” after it’s used through a regular deletion schedule.
Quest scratch space can be used for:
- storing temporary files created while running jobs,
- downloading data for processing,
- staging data for transfers or jobs to be run,
- short-term storage for large datasets.
Files in Quest scratch are not backed up and can’t be recovered once deleted.
How does it work?
Access to Quest scratch is granted through an application process, and in order to be approved for Quest scratch access you must currently be in an active Quest allocation. You can check the status of your Quest allocation through the checkproject utility on the Quest command line; if you are not in an active Quest allocation and want to apply for Quest scratch access, first apply to join, create a new allocation, or renew an existing allocation.
Once you’ve agreed to the Quest scratch user terms and conditions and your application is approved, you’ll receive an email with the full path name to your new scratch directory. Each Quest scratch user gets their own personal directory in Quest’s /scratch partition, with the pathname /scratch/<netID> . Files in /scratch directories are accessible across all Quest nodes, including the login nodes, Quest compute nodes, and the Quest analytics nodes.
Are there any limits on my scratch space utilization?
Certain limitations are put in place to assure sustainability and create equitable access to this new service. Consequently, each Quest scratch user has an individual quota of 5 TB and 5,000,000 files, and files will be deleted automatically 30-days after the last change to a file. If a Quest scratch user exceeds their 5 TB / 5,000,000 file quota, they won’t be able to write files into their scratch directory and will need to delete files to free up space before they can write files there again. Quest scratch users can monitor how much scratch storage they are occupying and how many files they have in their scratch directory using the checkscratch utility. Note the 5 TB / 5,000,000 file scratch quota is universally applied and cannot be changed for individuals.
Managing Quest scratch
Northwestern IT continuously monitors the total amount of storage in Quest scratch because if Quest’s scratch partition fills up completely, jobs that write data to scratch will fail. To that end:
- If Quest scratch begins to fill up, all Quest scratch users will be sent an email asking them to delete any files in their scratch directory that they no longer need. If this frees up enough storage to return to normal scratch operations, no further action will be taken.
- If Quest scratch becomes critically full, the 30-day retention period for scratch will be temporarily shortened by 5 days to delete more files and increase the available storage in scratch. If this happens, Quest scratch users will be given 5-days notice before this temporary shortening of the Quest scratch retention period. Once the amount of free space in Quest scratch returns to the normal operations range, Quest’s scratch data retention period will be lengthened back to 30-days.
Apply for Quest scratch
After November 1st, we invite everyone with an active Quest allocation to apply by filling out the Quest Scratch application form on the Quest Storage and Data Policy page. Note the application form won’t be available until that date.
Best Practices
When using Quest scratch, we recommend configuring your scripts to write your computation’s final result into your /projects directory, which is storage that is available to your active allocation and won’t be automatically deleted as long as your allocation is active.
If you want to keep files currently in scratch long-term, you must move them out of scratch before they are deleted automatically. We recommend using Globus for file transfers and giving yourself plenty of time to move large files out of scratch – your file transfer will fail if the files being moved from scratch are 30-days old and get automatically deleted during the file transfer. We are unable to make exceptions to the 30-day deletion policy, so please plan accordingly.
If your script writes interim files into your scratch directory, at the end of your script include a command to delete any files you won’t need after the run of the job. By deleting temporary files when you no longer need them, you will have more storage available in your scratch directory and won’t have to clean it up later. You can also use this practice when running jobs out of /projects directories for the same benefit.
Feedback
We want to know what you think of Quest scratch space! If you have feedback about scratch, please fill out the Quest scratch feedback form. Feedback will help inform future planning for Quest scratch space and is much appreciated!
If you have any questions about Quest scratch space, please reach out to quest-help@northwestern.edu.