I rarely use cloud-based HPC clusters for scientific simulations because they are just too expensive. I usually have access to clusters at different high-performance computing centers. But those clusters have one serious problem: too many people are waiting in the queue for resources. It’s not uncommon for me to wait a day, or even several days, for my jobs to start after I submit them.
For runs that take several days, waiting a day may be fine. But sometimes I just need a cluster for something like a 10-minute interactive session to debug internode parallel issues or run a quick benchmark. It makes no sense to wait one day just for a 10-minute job.
So I decided to build my own personal HPC cluster with a cloud service provider. It sounded good. I could use a pay-as-you-go model. And because I would only use this cloud cluster for small jobs, it should not be too expensive.
I had used Microsoft Azure at some point in the past. But after comparing prices and convenience, I decided to go with Google Cloud Platform.
Everything went fine during the initial experimentation stage. I was able to build a small cluster with 10 compute nodes using very cheap instances, plus the Slurm scheduler and an NFS server. A Lustre cluster was too complicated, so I passed on that. Google Cloud Platform also doesn’t seem to have InfiniBand. But that was okay for a personal cluster.
However, terrible things happened when I moved into a more production-like stage. When I tried to replace those very cheap compute nodes with high-performance instances (i.e., C2-series instances) and GPUs, I started to regret using Google Cloud Platform.
First, C2-series instances cannot have GPUs. This is ridiculous. Yes, I know machine learning applications usually don’t need powerful CPUs. But I’M NOT RUNNING MACHINE LEARNING APPLICATIONS. Many traditional scientific simulation programs exploit all available heterogeneous hardware. My applications need both powerful CPUs and GPUs.
It is absurd that GPUs cannot be paired with powerful instances. It seems Google Cloud Platform only cares about machine learning applications. All of their web pages and promotions talk about machine learning. They don’t seem to care about other applications that need more powerful machines.
Second, they impose very restrictive quotas on how many CPUs and GPUs you can request. The default quota for C2 CPUs is 8 CPU cores, and the quota for V100 GPUs is 1 GPU. What kind of HPC cluster has only 8 CPU cores and 1 GPU?
Okay, so I can request a quota increase. Fine. I submitted a request to increase the C2 CPU quota from 8 to 256 and the V100 GPU quota from 1 to 16. I thought that at least I could have a cluster with 8 nodes, each with 32 C2 cores and 2 V100 GPUs.
But no. THEY REJECTED MY REQUEST IMMEDIATELY.
I just don’t understand. 256 CPU cores and 16 GPUs are not crazy numbers. And my cluster would not be running 24/7. It would only be up when I needed it. And the rejection came right after I clicked the request submission button. That means no real human had even reviewed my request. This is ridiculous. I’m willing to pay out of my own pocket, but they do not even bother to review the request.