jonathan@researchcomputingteams.org
Know What Your CIO Is Thinking about AI. Plus: The fast track tip; Huh as a signal; Prioritization together; Strategy as group choosing; Lessons from building CLIs
Do you know what your CIO is thinking about and planning for AI? We owe it to research at our institutions to know, and to both shape their thinking and align our own efforts.
Universities have three main missions — teaching, research, and community relations. Those functions usually operate in splendid organizational isolation from each other, even if many individuals do freely move between them. Underpinning all of those missions is a fourth function, of a sort; the enterprise side of the operation, which makes sure people get paid on time, runs facilities and wifi and email, etc. IT lives there.
Enterprise IT has two significant advantages in their operations, compared to our teams:
The same things that make them steady mean those teams find it hard to rapidly change, which has made it an awkward home for the constantly shifting needs of the research mission. But under the right pressures, change they can. And when an organization that size does shift direction quickly, it carries with it a fearful amount of momentum.
Even if our teams eventually report up to the CIO, historically our teams have been pretty independent from Enterprise IT. On the systems side, they wouldn’t get deeply involved in HPC operations any more than they would in the operation of a large mass spectrometer. It’s different technologies, “weird” operationally, and has very different quality of service requirements than they’re comfortable providing. Same with research software, which requires very different skills than enterprise software development — more uncertainty, different languages and operating systems, and fuzzier requirements; or domain-dependent data science and curation efforts which require a blending of both research and data skills. Easiest all around to leave it alone, there’s more than enough other work to be done.
Although we’ve typically operated quite independently from enterprise IT, this has changed recently. There’s been six shifts in the last 15 years which have brought us closer to their orbit:
Enterprise adoption of cloud - For most of our institutions, Microsoft Office shifting to Office 365 basically forced a cloud adoption strategy of some sort on the enterprise side of the house. Once you’re contracting with Microsoft or Google Suite for handling your institutionally sensitive emails and data, the barrier to adopting other cloud services starts looking pretty modest. And once you have a corporate cloud accounts with negotiated discounted rates, making that rate available to researchers who need Jupyter notebooks or dropbox-type storage or even just bursty compute starts becomes an obvious next step.
Everyone wants Agile - Enterprise has been hearing nonstop for 15 years about agile methodologies, and has slowly learned how to adapt more quickly and adjust plans more dynamically. Cloud technologies, whether on-prem or commercial cloud, along with web apps as a delivery mechanism, have helped with this.
The explosion of data, funder requirements for data management - Comparatively suddenly, many researchers now have data that they need to do something with afterwards - and with high availability and integrity requirements for that archived data. Libraries, long the traditional stewards of archival knowledge, have stepped up. Libraries (because they also span all University missions, and have high uptime requirements — people get really cross if Library systems are down) tend to be much more aligned with Enterprise IT than RCD. That means Libraries and Enterprise IT have grown significant data storage infrastructure at many research institutions.
The emergence of teaching data science - Sort-of-separately, but hard to disentangle from the above, is the desire for data science education at the Undergraduate and Continuing Ed level as part of the teaching and/or community missions. RCD has long been involved with graduate student education, which is kind-of-research training, and involves much smaller numbers. But RCD can’t and arguably shouldn’t offer data science platforms with the same high availability requirements and volumes needed for for the teaching mission. Further, undergrad (and especially) continuing ed are significant income streams, which opens up funding options. So increasingly, many institutions’ Enterprise IT are getting experience providing (one way or another) other sorts of computing and data expertise that would normally be housed in RCD teams.
Expansion of computing beyond RCD’s traditional fields - Growth in computing power and algorithms for analyzing (say) textual corpora has meant that a lot of researchers who affiliate themselves much more closely with the Library are now doing real research computing, data, and software work. Similarly, experimentalists or field researchers who mostly used Excel and other enterprise IT packages are now needing bigger computing. Many don’t look up research computing, software, and data teams when they first need something bigger - they contact the Libraries or the Enterprise Helpdesk.
Cybersecurity concerns - Every enterprise IT leader can recount in an urgent tone too-close-to-home stories of hacking, bitcoin mining, or ransomware attacks at comparable institutions if not their own. This, plus the increasing amount of sensitive data some humanities or social sciences or medical researchers are working with, has meant that systems, data sets, and sometimes even software of RCD teams are coming under the umbrella of CIO or CISO policies.
To be clear — all of this is fantastic. It’s an amazing success and vindication to see that research computing, software, and data plays such a broad and widespread role across the institution now. Having other groups helping us support research this way is amazing. We have collaborators, not competitors (#142). Advancing research with computing, software, and data resources and expertise is too large a task for any one team or set of teams.
There’s about to be another item for the list above making a change in the corporate IT landscape; the only question is how large. The seventh is going to be adoption AI technologies. You see it all over the place in the agenda of the EDUCAUSE conference this week, for instance. Educause, to their credit, has hosted discussions about AI since the previous decade, but the urgency has ramped up. Ever since last November, every University board is asking their CIO what they’re going to “do about AI”.
You, having seen many technology hype cycles in your career, might be reasonably skeptical about the importance of AI and whether it will go anywhere. But even if (say) Generative AI stalls out where it is today, the enterprise is paying close attention and will do something. Very highly paid consultants do consistently better with GPT-4 than without, for instance. Almost everyone in University administration is a knowledge worker, salary and benefits make up the vast majority of institutional budgets already, so adding more people is hard, and hiring is challenging even if the money was there. The interest in having AI technologies support registrar functions, institutional development, enterprise IT, cyber security …. is extremely high. And those are all areas where a certain amount of customization (fine-tuning, prompt tuning, maybe even custom training) and thus in-house expertise is useful.
Ok, great, so far, so good. But now combine that with the previous shifts and imagine the following scenario.
Again, progress and other groups being involved is good and we should be enthusiastic about it. Our teams shouldn’t be, mustn’t be, and frankly aren’t capable of becoming gatekeepers to technical research support technologies offered by other parts of the institution.
Our teams do, however, have an amazing amount of expertise on technical and research issues. From workflows to software development to data curation to AI/ML to complex compute systems operation. We owe it to Science and to our institution to make sure that expertise has the greatest impact possible on science and scholarship in our communities. And we owe it to the individual members of our teams to make sure they personally have as much impact and ability to grow as they can and desire.
So even (especially!) if our teams report up to the VPR, we owe it to our teams and our institutions to make sure we are talking with the teams “over there” in the libraries and Enterprise IT, and in particular to inform and stay informed about our and their DL/ML/GenAI plans — or other plans, for that matter. The days of splendid isolation between research and enterprise computing, software, and data are coming to an end, and thank heavens. We still need both skills and capabilities! But combining strengths and capabilities is much better than each side going it alone.
What do you think? Are you already being drawn into these conversations? What are researchers asking you for? How about administration? I’d love to hear about it - just hit reply, email me at jonathan@researchcomputingteams.org, or set up a call with me to vent about it.
And now, on to the roundup!
Over at Manager, Ph.D in issue #163, I covered:
The fast track tip - Bite code!
As you know, I’m a big fan of process when used appropriately, and not just process as way of avoiding enforcing expectations (#148). But there should always be a clear reason, and it should be open to updates and easy to change.
This writer goes further and advocates for an explicit “fast track” to make it easy to bypass the full process when the situation warrants, or even just to encourage behaviour you want to see more of (like adding documentation to a repo).
Fast tracks are simplified procedures for specific cases that you know should not warrant the full-blown process.
The context here is tickets or PR reviews for software, but really this is something that applies more widely and is well worth thinking a bit.
Huh! as a signal - Dan Slimmon
The most exciting phrase to hear in science, the one that heralds new discoveries, is not “Eureka!” (I found it!) but “That’s funny …”
— Attributed to Isaac Asimov, but apparently just showed up on Usenet out of nowhere
For what it’s worth, I believe this is true in people systems, too, but here Slimmon is talking about technical systems. The idea is that yes, things fall over or evince unwanted behaviour all the time. But surprising behaviour is interesting in and of itself:
When we go looking at data […] sometimes we see something weird, and we go like, Huh!. That Huh! is a signal. If we follow that Huh! – get to the bottom of it, figure it out, make it not surprising anymore – two things happen. First, we get a chance to correct a latent problem which might some day contribute to a failure. And second, we make our mental model that much better.
This I think is the main benefit of chaos monkey/disasterpiece theatre (#5) approaches - actively finding surprising behaviour.
The Magic Prioritization Trick - John Cutler, The Beautiful Mess
Cutler describes a group activity to get people to make sensible prioritization choices - making people come to some kind of consensus on urgency of possibile activities, then impact, then amount of effort. From there you have a set of crude cost-of-delays for the items, which immediately suggests a prioritization.
The Real Reason Why Business Strategy Fails - NOBL Academy
The first step might be to simply stop calling “strategy,” strategy for the time being.
Ultimately, “strategy” involves making choices as a group. And in fact, if we replaced “strategy” entirely with the term “group choice making,” we’d probably approach it more effectively.
Most of the other items I listed as separate items under the “strategy” umbrella - positioning, prioritization, arguably even problem solving - are about group choice making. There’s still stakeholder engagement, but I’m happy to consider that as data collection for input into the group choice making.
Things I’ve learned about building CLI tools in Python - Simon Willison
Willison writes about his extensive experience writing CLI tools, and distills down some lessons.
I don’t think the “in Python” here limits the utility of this article to that language, because the lessons are pretty general:
--help output ([YES! PLEASE! - ljd])Real Multithreading is Coming to Python - Learn How You Can Use It Now - Martin Heinz
Something that is Python-specific - a good overview of what’s underlying the real, GIL-less multithreading coming to Python as of 3.12.
Still working my way through all the randomness of the past summer…
Early computer art of the 50s & 60s.
Some of you who are my age will be very interested to see to hear that there’s a chance that a whole bunch of old Computer Shoppers will at some point be scanned and available online.
Amazon Science has an article which is a pretty good quick intro into physics-constrained ML for scientific computing, a topic I’m excited about.
Visualizing LLM tokenizers, and a quick-and-dirty way to understand diffusion models.
While a lot of our requests right now are to help researchers training models, eventually the point is to perform inference on them. This article is a year or so old, but it’s still a nice overview and tutorial for back-of-the-envelope estimations of inference of DL models, transformer models in particular. (Here’s a very good deeper overview of transformers).
I’m fascinated by AI town, and this very different take on agent-based simulations from what I’m used to - here’s an open-source stater kit for playing with such things.
A fascinating attempt at a bioinformatics chat bot.
Explaining and visualizing the Python dict.
Using a wok and some computing to observe the neutral hydrogen emission line in the Milky Way.
Visualizing the Fibonacci series as a matrix transformation.
And that’s it for another week. Let me know what you thought, or if you have anything you’d like to share about the newsletter or management. Just email me or reply to this newsletter if you get it in your inbox.
Have a great weekend, and good luck in the coming week with your research computing team,
Jonathan
Research computing - the intertwined streams of software development, systems, data management and analysis - is much more than technology. It’s teams, it’s communities, it’s product management - it’s people. It’s also one of the most important ways we can be supporting science, scholarship, and R&D today.
So research computing teams are too important to research to be managed poorly. But no one teaches us how to be effective managers and leaders in academia. We have an advantage, though - working in research collaborations have taught us the advanced management skills, but not the basics.
This newsletter focusses on providing new and experienced research computing and data managers the tools they need to be good managers without the stress, and to help their teams achieve great results and grow their careers.
(I’m not currently maintaining the job board, but I’m happy to take submitted jobs from the readership…)
Senior Scientific HPC/Storage Consultant - BioTeam, Inc. (US Remote)
As a Senior Scientific HPC/Storage Consultant, you’ll lead the development and implementation of cutting-edge, science-driven, on-premises and hybrid high-performance computing (HPC) and storage technology solutions for our clients. From pre-sales scoping to assessment, design, implementation, testing, and maintenance, you’ll help clients achieve their scientific mission with objective, best-fit infrastructure for science.
To excel in this role, you’ll need to be a subject matter expert in HPC, and scientific storage and computing practices. You’ll work closely with clients to discover requirements and design, deploy, and manage high-performance computing (HPC) and storage systems, scientific networks, processes, and other relevant computing infrastructure. Your deep curiosity about the latest advances in biomedical science and prior experience designing and managing science-focused HPC, storage, and computing systems are key.