r/HPC • • 8h ago

[Open Source] Canary – Like pytest for HPC

11 Upvotes

I’ve been developing an open-source project at Sandia National Labs called Canary.

Like, pytest, but for scientific applications and HPC. Canary provides a Python-based framework for defining, scheduling, and executing tests and workflows, from your laptop to large HPC clusters.

It started as an application testing framework but has grown into a more general-purpose tool for managing computational workflows and dependent jobs.

It’s open source, and I’d love feedback from others working in scientific computing and HPC.

GitHub: https://github.com/sandialabs/canary


r/HPC • • 1d ago

Does anyone have advice for finding a job in HPC?

27 Upvotes

I'm an undergrad CS student that is about to graduate in May, and I've been working at my university's HPC Center for the past 2 years as a federal work study student. I am located in the US if that is relevant.

I've been applying to HPC jobs, but most jobs are not for entry level experience. I have yet to hear back from any labs. Perhaps it's a resume issue?

I do have experience maintaining my own systems because my supervisors are nice enough to lend me retired hardware, but I am not sure if that's enough... Stuff like GPFS, Infiniband, NCCL, Singularity, Slurm, Pyxis, Enroot, those I have some experience in.

I'm not gonna say I'm seasoned in this, but I feel like it's hard to find a job in HPC without having proper work experience prior...


r/HPC • • 2d ago

Fiber task scheduler: parallelize sequential code without callbacks or state machines

3 Upvotes

An encoder or a compressor has plenty of parallel work: channels, blocks, the stages of consecutive frames. But these pieces keep waiting for each other, often after just a few microseconds of work.

With OS threads every such wait goes through the kernel and costs about 5 µs, before cache effects and time-slice delays. That only pays off for tasks of about 45 µs or more, so most of the fine-grained parallelism is lost. Fibers on one worker thread per core bring the cost of a wait down to about 0.1 µs, and tasks of about 1 µs become worthwhile. Unlike with callbacks or stackless coroutines, the code that waits still reads like the sequential original.

ThinkMeta.ConcurrentTasks is a small scheduler that does this. To test it, I parallelized five encoders (LAME, opusenc, libdeflate, jpegli, cjxl), each on a single file and byte-identical to its reference. Most of them run several times faster than the originals (up to 13.9x on 8 cores / 16 threads). The encoders were written AI-assisted. The API and the encoders are open source; the core is a free binary for non-commercial use.


r/HPC • • 2d ago

Open-source Hugging Face mirror for clusters: one binary, no root, works for air-gapped nodes

0 Upvotes

I built a self-hosted registry that mirrors Hugging Face model repositories, and I’d like feedback from people who run this on real clusters.

The problem I had was batch inference jobs that all start by pulling the same weights from Hugging Face. That made an outside service a dependency of every job, and on nodes with no outbound access it doesn’t work at all.

What it does:

  • Mirror a repo once from a machine that has internet, then serve it inside the cluster. Existing tools just need HF_ENDPOINT set; nothing else changes.
  • One static Go binary with SQLite inside. No root, no container runtime, no database server.
  • Every file is checked against the digest Hugging Face publishes, and stored once by SHA-256, so shared files across models aren’t duplicated.
  • A registry can mirror from another registry, so a site-wide mirror can feed per-cluster ones.

Limits worth knowing: it’s a single server with no replication, there’s no scheduled sync yet, and its data directory has to be on local disk, not a network filesystem (model files can go to S3-compatible storage instead).

Apache 2.0. Repo: https://github.com/model-delivery/registry

What would you need before running something like this on a cluster? Slurm integration, a module file, parallel filesystem support?


r/HPC • • 4d ago

The Fastest Academic Supercomputer in the U.S. | Horizon Supercomputer

20 Upvotes

r/HPC • • 5d ago

Looking for academic/free HPC access for undergraduate research from Pakistan.

6 Upvotes

We are trying to develop a surrogate model for SITVC (Secondary Injection Thrust Vector Control). We currently have around 48 initial sampling points using a maximin Latin hypercube design, with all six input factors varying simultaneously. All the data is supposed to come from CFD simulations.

The two papers we are primarily using as references are Zmijanovich et al. and Abdelwahab et al. Zmijanovich used around 1.9 million cells, with a first cell height near the nozzle wall of approximately (10^{-6}) m, while Abdelwahab used around 5 million cells for their medium mesh.

Now, the problem is computational resources. We are a group of three, and even our best available system has only around 6 CPU cores and 16 GB of RAM. As a result, a single simulation can take several days to complete.

We haven't managed to complete even a single 3D simulation so far because of these limitations, and have only been able to work with 2D steady-state simulations. The problem is that the actual flow physics is inherently unsteady, so we eventually need to move towards 3D transient simulations.

Could you guys please suggest some affordable, preferably free, options for accessing decent computational resources? Based on the mesh sizes and simulations we are aiming for, we have been considering systems with 32–64 CPU cores and sufficient RAM, but we aren't sure whether this is actually necessary or what would be a realistic configuration for our work.

Also, we're based in Pakistan, not the US, which makes things a bit more complicated when it comes to accessing university HPC facilities, research grants, or cloud computing credits.

Any suggestions, especially from people who have dealt with similar computational constraints during their research, would be greatly appreciated!


r/HPC • • 6d ago

What kind of research still makes sense?

27 Upvotes

I’m a student and I was wondering what people from this sector still consider valuable in research. Many things can be done now with the help of LLMs, but what does actually makes senso to tackle? I fill like LLM can help find narrow gaps in the literature and fill them, I don’t view that as really useful, am I wrong? I’m really curios to hear your opinions


r/HPC • • 7d ago

What OpenMP concepts should a beginner practice first?

24 Upvotes

I’ve been learning OpenMP in C and putting together small examples to understand concepts like data sharing, synchronization, tasks, and reductions. For those who’ve used OpenMP, which concepts or common pitfalls would you recommend a beginner focus on?

I’ve collected my practice examples here in case context is useful: https://github.com/GANESHPARDESHI07/Openmp-playground.

Feedback is welcome.


r/HPC • • 10d ago

How important is it to take up a fluid mechanics course?

10 Upvotes

Hey guys, I'm taking the HPC subtrack at KTH for my masters in Comp. Sci. We have the Intro and Methods in HPC courses. I'm wondering if fluid mechanics is also important to take. I remember one master's thesis position explicitly asking it, but I don't know how how common it is.


r/HPC • • 15d ago

Strangely, memory read in GPUs don't work predictably on predictable data

11 Upvotes

Full write-up with the measurements, DRAM/L2 analysis, and barrier experiment: here

I was curious whether the power effect Horace He observed on matmuls (zeros running faster under power limits) would show up the same way on pure memory reads.

I wrote a simple bandwidth-bound read kernel on an A100. Bandwidth stayed at ~1.76 TB/s across patterns, but board power varied considerably:

Zeros: 210 W

Sorted BF16: ~253 W

Shuffled BF16: ~289 W

Random bits: Hits the 300 W power limit

Sorted data was nowhere near as cheap as zeros.

The main reason turned out to be warp drift: thousands of warps start together but gradually drift apart. As a result, the shared wires between L2 and the SMs keep seeing values from distant parts of the sorted array, causing repeated bit flips.

Adding barriers to resynchronize the warps dropped sorted-data power noticeably while still maintaining high bandwidth.

The interesting takeaway: data patterns alone don't determine power consumption. How warps access that data over time matters too.


r/HPC • • 16d ago

Experiences with Ozaki emulation

6 Upvotes

AMD strikes back in the FP64 war: https://www.hpcwire.com/2026/11/02/fp64-performance-and-the-limits-of-ozaki-emulation-on-hpc-systems/?_gl=1*12hq0mt*_up*MQ..*_ga*MjAzOTYwODgzMC4xNzkzOTU4MDI3*_ga_5DXR30MHNK*czE3OTM5NTgwMjckbzEkZzAkdDE3OTM5NTgwMjckajYwJGwwJGgw

I wonder if you have any real-world experience with FP64 emulation on Nvidia GPUs. So far I only know it as a marketing tool by Nvidia to increase the Top500 results, but I also see the value in the emualation.

My biggest concern is how it might be implemented for kernels that you can't map to a BLAS routine. For example various kernels in particle physics have roots, trigonometric functions etc. For a wide-spread adoption of emulation, I would expect that there must be some solution so that it ideally comes down to a simple compiler flag.


r/HPC • • 16d ago

How are universities handling batch LLM inference on their HPC clusters?

38 Upvotes

I've been working on some GPU orchestration and batch inference tooling as a side project, and lately I've been wondering how universities and research institutions actually handle this stuff.

I know Slurm is pretty common in academic HPC environments, but I'm curious how that translates to LLM inference workloads. Like, if a researcher needs to process 100k documents through an LLM, are they spinning up their own vLLM instance on an allocated GPU, or do universities typically provide some kind of shared inference service?

I'm also curious about GPU utilization. Do these clusters tend to be pretty saturated, or is there a decent amount of idle capacity during nights/weekends? And if there is idle capacity, are there mechanisms in place to take advantage of it for lower priority jobs?

I've been exploring the idea of an on-prem batch inference system that could sit alongside existing infrastructure, basically letting researchers submit jobs through an OpenAI-compatible API without having to worry about the underlying GPUs, scheduling, etc. But I honestly have no idea if this is solving a problem that actually exists in academia or if you guys already have this figured out.

Would love to hear how this works in practice, especially from anyone managing GPU clusters at universities. What are the biggest headaches you deal with when it comes to supporting AI workloads?

ETA: Thanks everyone, this has been incredibly helpful. What I’m hearing: clusters are mostly busy, allocations aren’t the same as utilization, and walltimes plus preemption make running your own vLLM painful for big jobs.


r/HPC • • 16d ago

Nvidia maxLPS

1 Upvotes

Has anyone had hands-on experience deploying NVIDIA DSX MaxLPS yet? Given how new it is, real-world data points seem scarce—curious how closely field performance and efficiency numbers align with NVIDIA's published claims. Additionally, for anyone architecting a large-scale Vera Rubin cluster, what critical design considerations, operational trade-offs, or integration nuances should be kept in mind regarding NVIDIA's DPS suite?


r/HPC • • 17d ago

NCCL Pre-check: A GPU Cluster Infrastructure Checklist

10 Upvotes

Before running multi-node NCCL benchmarks, it's important to verify that the underlying GPU infrastructure is properly configured.

I've put together an open-source checklist covering common pre-flight checks for GPU clusters, including:

  • GPU topology, P2P and NVLink
  • PCIe link speed and width
  • GPUDirect RDMA (GDR)
  • InfiniBand / RoCE connectivity
  • ACS, MTU and GPU health

The goal is to help identify potential infrastructure issues before they affect NCCL benchmark results or distributed training performance.

This is a practical checklist based on GPU cluster operations, not an automated diagnostic tool.

GitHub: https://github.com/LiquidFLOPS/aiinfra-precheck

Feedback, suggestions and contributions are welcome! I'd especially appreciate input from people operating large-scale GPU/HPC clusters.


r/HPC • • 17d ago

Opinion of student cluster competition (ISC SCC)

6 Upvotes

I got selected to represent my college (we are regular participants of virtual version). I am curious about will it help in job search or research positions???, love to listen to some past experiences.


r/HPC • • 17d ago

SC - Supercomputing Conference

31 Upvotes

Curious how many people in this HPC subreddit go to big hpc/supercomputing conferences like 'SC'?

A. You attend SC most years

B. You attend SC occasionally

C. You know of SC but don't attend

D. What is SC?

For anyone attending for the first time at SC26, what would it be helpful to know from people who go most years (like me)?


r/HPC • • 18d ago

PSA: LBNL Node Health Check (nhc) has released version 1.5

16 Upvotes

After several years nhc has released a new version 1.5:


r/HPC • • 20d ago

CS grad missing linear algebra, stats, and diff eq. Realistic chances for UT Austin ACSES and EUMaster4HPC?

4 Upvotes

I'm 28, a cloud engineer at a large energy company, with a BA in CS from FIU (3.66, cum laude) and some research experience. I'm applying for Fall 2027 to UT Austin's ACSES track and EUMaster4HPC (also looking at UPC's MIRI-HPC). My transcript has no linear algebra, probability and stats, or differential equations, and I know those are listed as prerequisites.
Will the missing math get me rejected, or can work experience and online coursework make up for it? Any advice, especially from people who got in without the usual math background, would help.


r/HPC • • 23d ago

how to start in HPC career as non-US/non-EU citizen

19 Upvotes

being a first generation grad & post grad, I got not much guidance around myself. so here I am figuring things out. I get to know High performance computing is something that is growing and will stay for large period of time.

could anyone help me where to start in HPC with little to know resources. where to learn from


r/HPC • • 23d ago

Considering a career change. What positions would best suit me?

13 Upvotes

My exposure to HPC is up till this point on the application side. I have a PhD and a background in numerical mathematics, deep learning and physics.

My academic contract is ending and I'm not keen on the location and the work so I'm looking for a breath of fresh air.

Since I have decently good domain knowledge I'd like to work in the HPC industry (or is it called the datacentre industry now?!) in some sort of support role.

To be honest, I'm not keen on sitting being a desk and writing code. I'd rather do something administrative, maybe some role that involves travel and logistics, or maybe sales? I know vendors tend to have many people who travel on location.

Does anyone work in such roles? Do you think my background would be a suitable fit or would vendors not be interested? How do you find the job salary wise and time wise?


r/HPC • • 24d ago

Need for a C++ BLAS Library?

10 Upvotes

I would like to discuss the need (or no-need) for a C++ interface/libary that is intended to be a
- building block for high level linear algebra packages and
- low-level, hardware oriented implementations of special numerical operations.

 My C++ library ulmBLAS is kind of a prototype for this idea, and heavily inspired by BLIS

The “high-level world” …



In the C++ world, you have libraries like Eigen, Blaze, … where you can matrix operations like

C = A * B;


and regarding performance often stops here. Because some experts figured out how to implement such that it’s fast.

Or maybe there is a short discussion why you should write




C.noalias() = A * B;




in Eigen.

And that’s great. In this world you wanna build something great based on Linear Algebra operations.



The “Low-level world” …



The world of BLAS (Basic Linear Algebra Subprograms), with implementations such as Intel MKL, OpenBLAS, BLIS, or Apple Accelerate.

In this subreddit you know this world. Let's look at it from a "high-level world" point of view ...

They provide implementations for the matrix product that are very efficient. But maybe not that user friendly to use. For example they provide a function for the operation



C ← βC + αAB


where A, B and C are matrices. Where elements can all be float, or can be double, or can be complex-float or can be complex-double. But all of the same type.

And A and B can be either not-transposed or transposed (in the complex case actually conjugated-or-transposed).

A call to this function looks like this when you use the Fortran Interface (GEMM is short for General Matrix Matrix Product, D indicates that elements and scalars all have double precision)

:


dgemm("N", "N", &m, &n, &k, &alpha, A, &lda, B, &ldb, &beta, C, &ldc);


Or through the C-Interface:

cblas_dgemm(CblasColMajor, CblasNoTrans, CblasNoTrans,  m, n, k, alpha, A, lda, B, ldb, beta, C, ldc);



So for computing C = A*B you choose alpha=1 and beta = 0.

And you should only call it if A, B, C are not only different names but also refer so separate memory blocks.



So this is a very simple, almost primitive, interface.

But we know: that’s what makes it great.

Because providing an efficient implementation (close to peak performance) is not that trivial. But with a narrow interface (e.g. all elements of the same type) at least feasible. A well-defined problem were you can differentiate into the details.

What you do in the implementation is not some Strassen algorithm. In short:


- You compute the matrix product block-wise (so that you can exploit the cache hierarchy)

- On the smallest block (that fit in the L1 cache) you apply architecture specific micro kernels




Well-known fact here on r/HPC. But look into stack overflow and other communities. You will see articles and posts claiming BLAS is fast because it's implemented in Fortran, because it uses Strassen (or its improved variants), because ...

Two world in one …



Eigen actually does both. It provides matrix/vector classes and implements blocked matrix operations with platform specific micro kernels.

That’s great.



But still, I think it is of value to also have something which is a layer that brings them together.

ulmBLAS, for example, is kind of a prototype for this idea:


 - on top high-level libraries can be build

- at the bottom the architecture specific stuff integrated



Back in 2016 (yes most of the code of ulmBLAS is from around 2014) I made experiments on how higher level libraries can make use of ulmBLAS:


- Boost.uBLAS and
- Blaze.



On the high-level part you can do crazy things, like matrix classes where elements are not stored in memory, but instead, computed when you access elements. And still, you can achieve a pretty good performance.



And other stuff like mixed element types. Where A, B, C, alpha, beta can have different types. So unless you make copies, you can’t do the operation with BLAS. But in the ulmBLAS layer you can deal with that when matrix blocks get packed and do the type promotions there with little overhead.


r/HPC • • 26d ago

Most CUDA tutorials introduce too much too early

52 Upvotes

A lot of CUDA tutorials throw the whole programming model at you almost immediately.

You learn the syntax, then grids and blocks, then memory types, then synchronization, and pretty quickly you’re juggling a bunch of concepts before you have a good feel for what CUDA is enabling you to do.

I wanted to try a slower approach.

Part 1 is mostly about building the basic mental model: what a kernel is, what it means to launch lots of threads, how those threads identify the piece of work they own, and how to start thinking about a problem in GPU terms.

The hardware details come later.

I wrote this mainly for programmers who are curious about CUDA but haven’t spent much time with GPU programming yet.

Would be interested in feedback from people here, especially on whether the progression feels natural or whether I’m introducing something in the wrong order.

Part 1: https://computejunkie.substack.com/p/cuda-programming-for-the-curious
Part 2: https://computejunkie.substack.com/p/cuda-programming-for-the-curious-f8f


r/HPC • • 27d ago

Can a linux kernel engineer transition to an LLM inference/model perf engineer?

6 Upvotes

I guess one can always does the transition, I'm just wondering how to bridge these two areas, if there are any transferrable skills, or if someone ever sees an low-level perf/linux kernel engineer transitions to an LLM inference/model perf engineer?


r/HPC • • Oct 23 '26

HPC wiki info site down

11 Upvotes

Is the website https://hpc-wiki.info down for anyone else as well? Probably something related to the recent php vuln.