I want in IT: Why this message doesn't work in data markup

I want in IT:

Why this message doesn't work in data markup

Over the past year, Anastasia has encountered unusual job market challenges in hiring managers for data collection and markup. This article looks at finding employees for a data markup company, as well as the challenges recruiters face when reviewing questionnaires and tests of applicants.

The data markup market grew by 70% in 2022, which indicates the rapid development of the AI industry in Russia and makes the direction promising for the development of the labor market. With the emergence of data markup companies, new professions have begun to appear (e.g., data quality control manager, data markup project manager, data markup artist) that job seekers had not previously heard of. Using TrainingData's recruiting experience as an example, let's look at what difficulties this leads to when hiring.

Anastasia Stoyanova is an HR, L&D manager at
TrainingData.Solutions

Who was needed for the position?

In mid-July 2022, the HR department received a request to hire several project managers due to an increase in data markup projects.

Without going into detail, here is the operational vertical of the company where the managers were to be hired:

At that time, the team already had more than 15 project managers, whose regular tasks include:

communication with customers at all stages of project implementation regarding project conditions, deadlines, data uploading, adjustments to the terms of reference (TOR), etc;
drafting and executing pilot projects to determine the metrics and cost of the order;
drawing up detailed terms of reference for data markers (other assessors), their training and certification;
quality control and timing of the project;
transferring data, summarizing and completing the cycle of work with the customer;
maintaining various reports on the project and payroll of his team.

Due to the newness of the direction for the vacancy, there were no requirements to have experience in the data markup project manager position. We focused our attention on project managers in general, emphasizing candidates with managerial skills and the ability to learn new programs on their own.

How did you plan to evaluate applicants?

HR, together with operations managers, developed a strategy for evaluating candidates for project manager positions. The assessment of applicants was planned in 5 stages:

1. CV screening.

2. Motivational questionnaire.
A small survey where applicants were asked to tell about:

their experience as a project manager;
education specialization;
their motivation;
their areas of immediate development;
experience of working remotely;
presentation experience;
salary expectations.

3. Managerial cases.

In parallel with the questionnaire, the applicants were offered to solve three cases aimed at testing the skills of organizing processes, the ability to communicate with customers, to find ways out of difficult stressful situations, and simply to be smart. Here are a few examples of the test task:

You have been given a project with a large amount of data to process — 500,000 photos. The work period is a week. The average speed of one executor is 800 photos per hour. Executors rarely work more than 4 hours per day. The project needs to start tomorrow. How do you organize your work? What will be your steps in the preparation of the team, in the management of the project, in the control and its delivery?
You have completed the project, and give the result to the customer. A few days later, the client sends you feedback, saying that there are a lot of errors in the delivered work. What will you do in such a case?
Until the end of the project 2-3 days, with the existing number of performers, you do not have time to pass the project on time. What actions will you take in order to meet the deadline?

4.Test Assignment.

When passing stages 1 and 2, applicants received a test assignment on CVAT — one of the main data markup programs, where they were asked to familiarize themselves with the algorithms of the program, and in the open version to create a task with 5 photos of cats and on each create a polygon (two eyes and a nose), and then send a backup copy to HR.

5.Interviews with HR and Operations Managers.

According to HR forecasts the average number of responses for this vacancy was expected, because data marking is a narrow specialization, and not all applicants may be suited to the rapid pace of work in the company. I think the reader will agree that if the first three cases can be solved logically (based on experience), then the fourth case even at the stage of reading may not be understood (although for our job «is the base»).

What did we get in reality?

In reality, however, the number of responses was many times greater than HR assumed. During the first 24 hours after the publication of the vacancy at hh.ru there were 308 responses, and after two days there were over 600 responses. Truth be told, the batch of responses for the second day and beyond was not really taken into account, since the first three hundred were able to close the vacancy.

Because of the large number of responses in the first day, we had to change the strategy for evaluating applicants a bit, namely to remove the initial screening of resumes, because, first, it greatly increased the response processing time, and, second, there was a certain fear of losing good candidates because of the focus on specific experience. The resume screening was after the questionnaire and case study analysis. Let's look at the stages of the recruitment process after the first changes in more detail.

The questionnaire was filled out by 295 applicants, which accounted for 95.8% of conversions. It was found that about 20 people, when asked «What are your areas of immediate development?» saw themselves as developers, programmers, IT team leaders. Below are examples of the answers of the applicants

In general, there were not many comments in the style of «Get into IT», but there were, and that was our bad luck, because the job description did not say anything about programming. Immediately a question arose: Why did they respond to the vacancy, if they plan to develop in a completely different direction? Perhaps you, readers, have some suggestions in this area — we can discuss it in the comments.

«In the IT industry, already started, I'm studying to be a Front-end developer»
«This is definitely a development in the IT industry, related to product, project, analytics and programming»
«IT industry»
«I am mostly interested in the IT field and its part, dedicated to development management. I would like to develop as a project manager for IT teams»
«Management of IT projects»
«To get an offer on PM position, to advance in IT field (I'm studying frontend, now somewhere at basic js level), to get experience and new knowledge in real projects»

According to the results of the analysis phase of the questionnaire and cases, all applicants were divided into three categories:

red (unambiguously do not pass on) — 155

yellow (50\50) — 70

green (fit and solved cases, give priority) — 70

No sooner had we sent out the CVAT test job to candidates, than we got a huge wave of feedback that CVAT open access is not working for many due to technical reasons. There were, of course, some candidates who were able to complete the task or were more creative and completed it in another program with similar functionality! When inviting candidates for interviews, recruiters of course took this into account, but since it was not possible to create the same conditions for everyone, we refused to evaluate the job using CVAT.

Resumes were screened to determine the priority of interview invitations. HR colleagues will wonder: «Screening in the penultimate step, not at the very beginning!» This is due to the fact that project manager in data markup is a new profession, so the right applicants are people who may have never worked with markup before, but have great potential due to their unique background and experience in various fields. When evaluating candidates, what mattered more to us was not so much a person's background, but rather their developmental vector and career goals in data markup. And when it came to prioritizing interviews, we paid attention to the candidate's «past» and additional skills.

So, based on the screening results, the first 25 applicants were invited for an interview with HR and operations managers. A series of three days of intensive interviews ensued, focusing on the skills, motivation, growth prospects, and values of the candidates. Also, some applicants again had the «I really want to be in IT and it doesn't matter what I do» story — unfortunately for candidates, these were stop-factors. At the end of this stage 5 offers were put up, all were accepted.

Ultimately, the hiring funnel for the project manager position looked like this:

308 — responded (100%)
295 — filled out the questionnaire (95.8%).
70 — passed the questionnaire (22.7%)
25 — interviews (8.12%)
5 — offers (1.6%)

As you can see, a fairly large cut of the conversion occurred just at the stage of the survey due to the large number of irrelevant responses.

Why did this happen?

Perhaps the applicants saw the adorable cat on the TrainingData website, stopped thinking about anything else, and responded. Or they were looking for all sorts of ways to get there in hopes of realizing themselves in IT, and they saw data markup as a shortcut to the IT world, and even without programming knowledge.

The second assumption seems more realistic. People go into IT from completely different spheres and at completely different ages because of economic necessity. The most obvious fact: the salaries of the IT sector in Russia look many times more attractive than in other professional fields. Also, the transition of people in IT during COVID-19 is due to the demand for specialists to organize new business processes. Thus, in 2020 the staff of Russian software developers grew by 12%, up to 200 thousand people. But in 2022 the picture is not so rosy — the demand for IT specialists in Russia has dropped by 25%. The story is fueled by many grandiose stories of success in IT with zero experience and a huge amount of advice with the cherished answer to the question: «How to get into IT?» Some authors try to debunk the illusions of newcomers to IT, but for the most part people's unfulfilled desires are a «marketing joy» for the many schools selling IT courses from scratch. People even write entire articles explaining IT memes:

Unfortunately, we mostly get a large flow of unskilled supply on the labor market in the IT sector.

On the one hand, it is good that people strive for more, analyze and make a choice in favor of innovative technologies, but on the other hand, there is a question: do people have an understanding of where they are going and for what? Judging by the responses to the position of data markup project manager, it seems as if this understanding is in short supply right now.

Data collection and markup for machine learning tasks is a young and dynamically developing field, which needs active and ambitious specialists. Those who are ready to dive into a complex narrow niche and become a real and, most importantly, in-demand expert. TrainingData is open to communication and training with people from completely different fields who want to make the world more secure and technological, see their development in data markup for machine learning, and want to conquer this new, little known niche.

Ресурсы и дополнительные ссылки: