'When We Find Something That Doesn’t Work Well Enough, We Replace It in Order to Develop a More Effective Approach'

Photo by Yandex

The winners of the third annual Ilya Segalovich Award were recently announced in Moscow. Established by Yandex, the award promotes the scientific endeavours of young researchers from Russia, Belarus, and Kazakhstan in the field of Computer Science. Among this year’s winners were three HSE students, including Alexander Grishin, a Doctoral student of the Big Data and Information Retrieval School of the HSE Faculty of Computer Science. Alexander spoke to us about his work, research challenges, and why he was surprised to receive the award.

HSE University and My Research Work

I started doing research work as an undergraduate in the Faculty of Biological and Medical Physics at the Moscow Institute of Physics and Technology (MIPT). In my third and fourth years there, I used machine learning to analyse DNA nucleotide sequences. It wasn’t an easy task—I had never studied machine learning before and I didn’t have enough knowledge of biology. I was keen on the subject, but I felt I lacked hard skills.

I realized that I was more interested in the technical side of things than the biology side, and I started looking for a place where I could gain those skills. That’s how I ended up in HSE University’s ‘Data Science’ Master’s Programme at the School of Data Analysis. The first year required a lot of intensive study, so I had no time for research. Then I met Professor Dmitry Vetrov, and we’ve been doing research together ever since.

My work on Professor Vetrov’s research team started with the biology side of things. I joined Insilico Medicine, a company that synthesizes pharmaceuticals. Alongside my academic advisor Daniil Polykovsky, I worked on forecasting the properties of molecules. This would enable us to generate molecules with the same properties for medicinal purposes.

Solving a difficult task requires solving easier ones first, so we started by trying to determine whether or not a molecule had any specific properties. We never published the method we developed, as two very similar articles were released around that time and the authors of those articles used a slightly better approach. That’s when we knew that our work was sound and up-to-date, just not quick enough.

After that, I began looking for another area of research. I wanted to work with Pavel Shvechikov and luckily he was able to take me on. I wanted to explore a new field, and that ended up being reinforcement learning. Pavel and I worked together for a few years. Our team gave a presentation at the NeurIPS oral workshop and wrote an article for ICML.

Our team doesn’t specialize in reinforcement learning, so we sometimes lack the level of expertise we’d like. But this is common, as not many people work on reinforcement learning at a serious academic level. We decided to put together a team dedicated to the field. As for the commercial sector, companies have always been interested in this area, but it remains underdeveloped.

The Prize-Winning Research

The things we’re doing are quite simple. Rather than focus on specific tasks, we develop general algorithms. We take an academic approach and try to develop more effective algorithms that can cover a wide range of tasks. We also study existing algorithms and the way they work in order to uncover potential problems. To put it simply, we analyze each individual element and when we find something that doesn’t work well enough, we replace it in order to develop a more effective approach.

We received the Ilya Segalovich Award for our article about overestimation bias in reinforcement learning. After performing an action, we try to figure out whether these actions will have good or bad consequences. Previously, we have relied on very primitive tools such as understatement or artificial suppression to reduce overestimation bias. But we’ve developed a more flexible and effective tool to avoid this kind of overestimation. Samsung has recognized our research too, which was a surprise to me. I thought the company would be more interested in practical results, while our work has the academic focus of enhancing the effectiveness of reinforcement learning.

Are there traditional tasks in reinforcement learning? Yes, and they’re used to test algorithms and compare them. One example is Atari games, in which the agent sees a picture, another image of the game, and performs a certain action (controlling the agent). These games vary both visually and mechanically.

We tested our algorithms on ‘locomotor tasks’—a set of tasks in a physical simulator that models different kinds of robots, from simple robots with a couple of joints to complex humanoid ones. The agent receives signals from sensors informing it of the position, angle, and speed of each limb. Every 15 milliseconds, the agent must exert a force to change the position of the body. Then the agent must exert more force to perform a new task, such as running as fast as it can without falling.

Reinforcement learning involves working with states—things the agent sees or does depending on the state it finds itself in, or rewards it receives for performing the right actions or sequences. The great thing about it is that the specifics of these states—where they come from, what they look like, how they change—don’t really matter. This is because learning algorithms are universal and the task is always the same: to teach the agent the best approach that produces the biggest reward. From a theoretical standpoint, the reward itself isn’t important, be it the speed of the robot’s movements, efficient power consumption, a click from a user, or money made at the stock exchange. The methods we used to teach robots how to run (or rather, to learn how to run themselves) could be used to distribute power generated by a nuclear power plant or advertise goods to consumers. Reinforcement learning has limitless applications.

The Yandex Prize and My Hobby

I’m delighted we received the award for a number of reasons. Firstly, personal motivation and external recognition often don’t go hand-in-hand in academia. Everyone has had their articles turned down more than once. So an award is something extraordinary, and it drives you to do better. Another important aspect is that the winners of this award receive substantial financial support, access to Yandex Toloka, and an invite to a conference. These are great motivators too.

Secondly, I didn’t think I had much of a chance of winning because I’ve had very few things published, so the award came as a pleasant surprise. The organizers must have seen me as a researcher capable of creating something interesting and useful. The number of publications wasn’t a deciding factor in their decision.

I was also delighted to tell Professor Vetrov (he wasn’t eligible for the award, although I’m sure he would have won if he had been) and my friends. Even if they don’t know much about science or research, they think it’s cool and they’re happy for me.

The award ceremony took place almost at the same time as the first concert of the ensemble I’m in, which is made up of six cellists and a pianist. We’ve got quite a wide repertoire, so there’s bound to be something for everyone to enjoy. We play everything from Shostakovich waltzes to the Game of Thrones theme. That’s another achievement I’m proud of.