The era of Artificial Collective Intelligence (ACI) is about to start
Ensemble Models & Multi-agent Collective Behaviours
Ensemble models and learning techniques, such as Mixture of Experts, along with Multi-agents systems, are attracting a great deal of interest in the AI world. In this post, I want to explore recent advances in these fields and inspire you towards their scientific interest and potentially powerful applications.
The outline will be:
Memory and Energy-efficient Mixture of Experts;
An overview of Multi-agent LLMs;
🚀 Job & Research opportunities, talks, and events in AI.
Let’s start!
Mixture of Experts
I recently came across the use of Mixtures of Experts (MoEs), which became a hot topic after the release of Mixtral 8x7B.
How do they work? Unlike conventional DeepLearning models, where the entire network is executed to process all inputs, MoE-based systems selectively activate only designated sections, or experts, depending on the input at hand (often this is also referred as sparsity, i.e. when only certain parts of the entire system are run). This selective activation is managed by a relatively simple mapping function learned during training, which identifies the most appropriate expert for each input.
Particularly in Transformers, at the architectural level, this is implemented by replacing the traditional dense feed-forward network layers with sparse MoE layers. These layers include a gating mechanism that directs the input to one of several experts based on the input’s characteristics. This strategy enables LLMs to significantly reduce computational costs during pre-training phases and enhances speed during inference. By engaging only the necessary experts rather than the entire network, MoEs streamline the processing and potentially increase model efficiency. On the other side, in order to improve memory usage of MoEs, an innovative approach, named MixLoRA, aims at constructing a resource-efficient sparse MoE model based on LoRA techniques. MixLoRA enables parallel fine-tuning of multiple mixture-of-experts models on a single 24GB GPU without quantization, thereby reducing GPU memory consumption by 41% and latency during the training process by 17%. Other memory-efficient approaches, such as DS-MoE, employ dense computation across all experts during training and sparse computation during inference.
Lately, I have experimented with various MoEs and explored several libraries to better understand and utilize this technology. For those starting with MoEs, I recommend a couple of blog posts, a detailed notebook, and a useful library that have been instrumental in my experiments.
Intro blog post by HF (Omar Sanseviero et al.);
Intro blog post by IBM (Dave Bergmann);
Library for building MoEs by Arcee.ai
It is also worth mentioning that nowadays there are several open source projects to train MoEs, such as Megablocks, Fairseq, and OpenMoE.
Multi-agent Architectures
I do not think that the near future of AI is AGI, Artificial General Intelligence.
Instead, I think the next few years will be marked by ACI, Artificial Collective Intelligence.
What excites me most is not the capabilities of a single LLM-based AI agent, but the collective intelligence that emerges from the cooperation and interaction of several different agents. Whether it is called Social AI, Sociology of AI, or AI Collective behaviour does not matter, but the truth is that a whole new branch of AI is emerging.
Recently, an interesting article ( 🔗 link below) showed how teams of LLM-based agents are much more effective if there is a team leader among them to orchestrate tasks. Without a team leader, LLM agents spent 50% giving orders to others.
A natural question is: when to use single or multiple agents?
Single agents are generally best suited for tasks with a well-defined tool list and where processes are well defined. They do not work well in scenarios where no examples are provided. Multi-agent architectures are suitable for tasks where feedback from multiple people is useful (e.g. document generation may benefit from a multi-agent architecture where one agent provides clear feedback to another on a written section of the document). Multi-agent systems are also useful when parallelisation between distinct tasks or workflows is required.
If you want to get started with multi-agent LLM-based systems, here is the above mentioned paper, a recent review, and a library:
LLMs Learn to Cooperate in Organized Teams by Mengdi Wang Lab;
A review by Tula Masterman, Sandi Besen, and others;
CrewAI is a library by João Moura, a framework for orchestrating role-playing, autonomous AI agents.
Many other works are gaining attention, showing how multi-agent models based on LLM can cooperate and dynamically adjust their composition as a greater-than-the-sum-of-its-parts system. In particular, I highlight the Self-Organized multi-Agent framework (SoA) and AgentVerse where emergent social behaviors among individual agents within a group are shown during collaborative task accomplishment.
We are only at the beginning of this new era for Social AI and there is still plenty of room for interesting ideas aimed at improving the collaborative potential of multi-agent groups!
Opportunities, talks, and events
I share some opportunities from my network that you might find interesting:
📚 Learning opportunities:
20 full Scholarships (by Horizon Europe) are available for the MAGICA Summer School, deep diving into AgriTech and AI for Climate Transition! (deadline, May 30);
Several full fellowships at the Master in High-performance Computing (MHPC, SISSA/ICTP, Trieste), a 15-months-program for future experts in HPC, AI and Quantum Computing (Link to apply, deadline May 15).
🚀 Job opportunities:
Two Research Scientist positions in AI for Earth Observation (Africa & France One Forest Vision Initiative): Paris and Bordeaux;
Ecosmic, a newly born Italian Space Startup is hiring!
An internship for an Investment Analyst with STEM background in Milan just opened at Primo Space Fund.
🔬 Research opportunities:
PhD opportunity in Deep Learning for Energy Analytics at The Arctic University of Norway;
The MilaNLP Lab at Università Bocconi is looking for a two-year Postdoctoral Researcher.
You can find me on LinkedIn or Twitter, where I share Science & Tech news. If you wish to book a call or view my lectures and courses (technical and non-technical), you can find me here.