The cost of success – user overload on DeiC Interactive HPC

Re-posting of an article: https://interactivehpc.dk/?p=1286

While there was never any doubt that DeiC Interactive HPC would be a success, the popularity of the HPC facility has taken the Interactive HPC consortium a little by surprise. The two-year old system reached no less than 5000 users back in December, and while every milestone is celebrated the rapid success also comes with a price.

We’re seeing an average rise in users of 1000 each quarter and we’re very proud of the success. However, with an average utilisation of 135% of resources for containerised applications, we’re also experiencing issues with user overload recurring more and more frequently as more users join.DeiC Interactive HPC Consortium representative, Professor Kristoffer Nielbo, Center for Humanities Computing, Aarhus University

Additional hardware would solve most of the issues but adding more hardware is a time-consuming process, and in the meantime the consortium behind Interactive HPC is working on other solutions to ensure the best user experience.

We are currently working on making operational status available to users to allow them to see when user overload is causing issues and plan their work differently accordingly. The part of Interactive HPC running on SDU already has a solution underway, and the part of the system located at AAU will follow as soon as possible.Professor Kristoffer Nielbo

However, users can also actively do things to avoid the user overload issues. The DeiC Interactive HPC consortium recommends that; Small users should make sure they only use the resources they need; medium users are asked to consider whether their work could be done on other HPC systems; and large users should apply for resources via the national calls.

Applying for national resources may not fix the problem right now but by doing so researchers indicate that there is a need for additional hardware for Interactive HPC, and this can help speed up the expansion process.Professor Kristoffer Nielbo

The consortium also recommends using the new DeiC Integration portal when it makes sense. The portal integrates multiple national HPC systems and allows for users to seamlessly shift to other facilities clearing up space for users whose only option is Interactive HPC.

The consortium will continue work on solving any issues and ensure that necessary resources are available because there’s no doubt that interactive HPC is here to stay as a favourite HPC resource for researchers.