Learning from multiple sources in heterogeneous groups of agents

Abstract

The field of Multiagent Systems (MAS) is concerned with software solutions composed of several autonomous elements (agents) that interact and communicate. The scenarios these techniques apply to have special characteristics that bring about new problems, but also provide new tools to develop adequate solutions. One of the research fields that is evolving in parallel and adapting to this new type of paradigm is Machine Learning (ML). Research in ML has been increasingly focused on the development of solutions that can deal with the problems posed by MAS. The contribution of ML to this field is of utmost importance since adaptability and learning are fundamental in increasing agents’ autonomy and flexibility. This work presents a study concerning the relationship between communication and learning in a certain type of MAS. We focus on problems where we have different teams of agents, solving similar problems at different locations. Each of these teams may use different learning algorithms or heuristic solutions. In the past, learning-agents used solely the environment’s feedback as a source of information for learning. MAS provide other sources of information that can increase agents’ learning capabilities. Our goal is to determine how the communication of examples and reward information can affect the learning process. The hypotheses posed in this thesis are: That communication can improve agents’ learning performance for several learning algorithms in a specific type of problems; It is possible to enhance the benefits of communication by: using of hybrid algorithms to integrate information from different sources, using heterogeneous environments and an adequate selection of information sources. These techniques are tested in three application domains: a “toy-problem” (predator-prey), a simulation with synthetic data (load-balancing) and one using real data (traffic-control). During this study several variables that influence the performance of the exchange of information during learning where identified, namely: use of batch or specific of information; online or offline integration; number of advisors; use of heuristic advisors; heterogeneity of the environment; type of algorithm used in the integration of external information. Although not exhaustive, due to the large number of possible combinations, our research tests the effects of several of these possibilities. The initial expectations pointed towards the possibility of increasing the speed of learning and the performance by exchanging information, particularly when using heterogeneous environments. It was verified that exchanging information is beneficial, in terms of speed, performance and reliability. Contrary to our expectations the environments’ heterogeneity and other tested techniques did not show the desired effects. This fact is due, mainly, to the near-optimal performance of agents in most environments where agents are allowed to communicate. Even though there is still a long path to follow in the quest for adequate solutions, this work provides, apart from the above mentioned contributions, a review of the main difficulties found during this research that may be helpful for those that follow this path in the future. The ultimate goal of this line of research is to endow agents with the capability of learning from more sources than just the environments’ feedback in a context where information is abundant. This new perspective of learning in MAS can lead to the development of new learning paradigms, specially suited for MAS, and take us one step further in the construction of autonomous and intelligent agents.