Thermodynamic Computing
New hardwares for future AI algorithms
What happens if computation, physics, and information theory converge together? New exciting, fast and energy-efficient computational approaches that take advantage of noise and stochastic fluctuations are emerging. They go under the name of Thermodynamic-based hardwares. This will be the focus of today’s post.
The outline will be:
Why new hardware paradigms are needed?
The startups that are leading Thermodynamic AI
🚀 Job & Research opportunities, talks, and events in AI.
Let’s start!
Thermodynamic Computing
To better understand why new hardware paradigms are needed we must go back on the software side, in particular on Thermodynamic AI algorithms. These are a family of models that heavily rely on randomness, energy-based approaches or ensembles. Among them, the most popular are Generative Diffusion Models, Monte Carlo Sampling, Bayesian Neural Networks, and Simulated Annealing. These algorithms are at the core of many Deep Learning architectures.
The problem is that today’s thermodynamic-based models are trained or run on digital hardware (CPUs or GPUs), thus limiting their scalability and overall potential (not to mention their power consumption!).
This motivates the search for novel, thermodynamic-inspired computing hardware to make Generative AI more capable, faster, and efficient. All together, these innovative approaches takes the name of Thermodynamic Computing.
Two emerging startups are challenging Nvidia's dominance, reinventing the computer chip entirely and paving the way for Thermodynamic Computing.
Normal Computing
Normal Computing is developing a physics-based thermodynamic hardware, named Stochastic Processing Units (SPU). A conventional processor performs calculations by processing information in binary form. A SPU, on the other hand, exploits the thermodynamic properties of oscillators to perform calculations using the random fluctuations that occur within circuits.
This process makes it possible to generate random samples useful for calculations or to solve linear algebra calculations, which are ubiquitous in science and engineering as well as in machine learning.
If you want to get started you can read the following papers:
Thermodynamic AI and the fluctuation frontier where the new computing paradigm of Thermodynamic AI is introduced;
Thermodynamic Linear Algebra where authors show how thermal equilibration of simple electrical circuits can be used to perform fundamental operations in linear algebra widely used in ML, such as matrix inversion;
Thermodynamic Computing System for AI Applications where they introduce the first continuous-variable thermodynamic computer, the stochastic processing unit (SPU), and demonstrate Gaussian sampling and matrix inversion on it;
Error Mitigation for Thermodynamic Computing where they show a method that reduces the overall error from a linear to a quadratic dependence, allowing one to suppress the effects of hardware imprecision by 20%.
Extropic
Extropic mission is to merge generative AI with the physics of the world. Their motivation starts from the consideration that Moore's law is slowing down, given the physical limits of transistor silicon technology. Instead of fighting against nature, Extropic embraces it, drawing inspiration from biology's efficient computation to create hardware that thrives amid noise and randomness.
Digital computers are bad at generating random numbers. Extropic hardware accelerators aim to speed up sampling, a crucial operation in many AI models.
Their engineering challenge is clear:
How can we design a complete AI hardware and software system from the ground up that thrives in an intrinsically noisy environment?
The answer stands in probabilistic approaches, in particular relying on Energy-Based Models (EBMs). Extropic chips, modelled on the principles of Brownian motion, are programmable sources of randomness that leverage EBMs directly as parameterized stochastic analog circuits. Extropic claims that its accelerators will achieve many orders of magnitude of improvement over digital computers in terms of both runtime and energy efficiency for algorithms based on sampling from complex landscapes.
If you want to dive deeper into Extropic and probabilistic Machine Learning:
You can read the recent Extropic lite paper or listen to its founders in this mini-podcast with Garry Tan (President of YCombinator);
Here a Course on Energy-Based Models by Alfredo Canziani and Yann LeCun.
The number of alternative approaches to computing is an ever-increasing trend that will drastically transform the way, speed and efficiency with which we process information.
Opportunities, talks, and events
I share some opportunities from my network that you might find interesting:
🚀 Job opportunities:
Atmosoar.io, an innovative startup in the weather data industry (with applications in the drone, energy, and insurance industry) is looking for a Data Science, Modeling, and Simulation Specialist;
2024 Amazon Summer Research Science Internships in Quantum Computing;
New Research internship in the AI for Climate Impact team @ IBM;
Zephyr AI, a healthcare technology company that reshapes traditional approaches to drug discovery and precision medicine, is looking for an ML Research Scientist;
Are you a Science Educator looking for a new challenge? Join the CERN Science Gateway education team by applying here!
🔬 Research opportunities:
The Human Technopole opened in Milan and at SISSA (Trieste) several PhD positions in AI and Data Science for Neuroscience, Genomics and Biology;
A postdoc position related to Human-AI Interaction and Explainable AI, with a special focus on biomedical informatics and medical decision making is open at Bicocca University through Federico Cabitza;
Vincenzo Lomonaco (Pisa University) is looking two research assistant (only Master Degree required) on Continual Learning!
Junior Data Physicist @ INFN (Florence) for ML technologies applied to Particle Physics (through Matteo Barbetti)
📚 Learning opportunities:
Application are now open for Session #15 of the School of AI @ Pi School. Around 10 grants (8-week hands-on AI programme on real-world challenge) are available (apply here);
Application are now open for several full fellowships at the Master in High-performance Computing (MHPC, SISSA/ICTP, Trieste), a 15-months-program for future experts in HPC, AI and Quantum Computing (Link to apply);
Quantum Machine Learning Master at Ca’ Foscari Challenge School - Master e Alta Formazione (apply here);
Master's Degree in Human-Centered AI @ Statale di Milano (Applications are open!)
You can find me on LinkedIn or Twitter, where I share Science & Tech news. If you wish to book a call or view my lectures and courses (technical and non-technical), you can find me here.