In 2021, the EIC Accelerator blended financing (formerly SME Instrument Phase 2, grant and equity) has completed its first year under the new application framework (read: Application Process). With 2 cut-offs in 2021 (June and October), it presented a steep learning curve for Small- and Medium-Sized Enterprises (SME), professional writers and the European Innovation Council (EIC).
A new online process for EIC Accelerator applications was rolled out and it was continuously improved in parallel to the first grant submissions which presented unique challenges to the way the EIC and consultants communicated towards potential applicants.
Deadlines were shifted, leaked information was more reliable than official EIC press releases and the comments provided by the evaluators led to some controversy. While more transparency is generally a positive step, especially for a public institution funded through the citizen's taxes, it can backfire if it exposes significant inconsistencies.
This article aims to explore some of these inconsistencies.
The Application Steps
The European Innovation Council and SME's Executive Agency (EISMEA), European Commission (EC) and EIC have come up with a new application process that involves 3 distinct steps (note: these are unrelated to the Phases of 2020). This new process heavily relies on the use of an online submission form and has phased out most of the PDF/Document-type formats that applicants used prior to 2021. In summary, the current steps are:
- Step 1: A mini-application (text, video, pitch deck). At least 2 out of 4 evaluators must approve of the application to succeed.
- Step 2: A long application (text, support documents, pitch deck). At least 3 out of 3 evaluators must approve of the application to succeed.
- Step 3: A remote or in-person interview. All jury members must approve of the application to succeed.
Startups have to successfully pass all three steps in the designated order to receive the EIC Accelerator financing. Each attempted step, successful or not, will likewise receive detailed comments from the evaluators or jury members.
Note: Through the fast-track program implemented by the EIC, some companies can skip certain steps if the respective conditions are met.
Step 1
Step 1 is designed to pique the evaluator's interest as the EIC has stated. It is a very short version of a business plan and provides no detailed information on finances, the planned workpackages or other critical parts of the innovation project. Even the pitch deck is reduced to a 10-slide document that will be read and not actually pitched.
The Step 1 success threshold is very easy to pass since only 2 out of the 4 remote evaluators must provide a favourable review which will allow an applicant to move towards Step 2 (see success rates).
Step 2
Step 2 is a very in-depth presentation of the proposed innovation project since it requires the creation of a business plan which almost exclusively consists of text, provides very little visual data and asks the applying SME's to answer many detailed questions. These include the value chain, product descriptions, technical backgrounds, market analyses, commercial strategies and many more details.
This Step has proven to be the most selective and also the most work-intensive stage of the EIC Accelerator.
Step 3
Step 3 is a remote or in-person interview which consists of a 10 minute pitch and a 35 minute Q&A session. The interview will be based on the submitted Step 2 application and pitch deck but the jurors might not be intimately familiar with all of the provided content.
Linear Progression Between Steps
While the new process for EIC Accelerator applications looks and feels modern, it has added a new layer of problems that is interlinked with its 3-Step structure.
When generating an application process that screens companies over multiple months, it is important to make sure that each evaluation step presents a linear progression from its predecessor. If the assessments of Step 1 and Step 2 are too different then this will inevitably lead to wasted effort for both the applicants and the reviewers.
To be transparent about this fact, the EIC should publish quality control data where the results of all three steps, if available for each applicant, are correlated to identify if a section was evaluated consistently across multiple steps. If all evaluators approve a very detailed business model in Step 2 but the jury members unanimously question its quality in Step 3 then the process would be flawed.
Based on the first applications in 2021, it is clear that the three steps have different degrees of depth, a different focus and they use different evaluator pools which inherently leads to significant limitations. As a result, the process is not fully linear.
Conflicts Between Evaluations
A linear application process would see a project with a perfect score in Step 1 do well in Step 2. A project which has presented dozens of pages on the commercial strategy and has received a perfect score by evaluators in Step 2 should not have this review be reversed in Step 3. While the difference in quantity between Step 1 and Step 2 is significant and can lead to shifts in the perceived quality, the difference between Step 2 and Step 3 should be minuscule.
In a linear process, there should never be a case where a revenue model was graded perfectly in Step 2 only to be rejected with poor reviews in Step 3. But such cases do occur frequently since an approximate 50% of applicants will be rejected in Step 3 with the top reasons being commercial aspects. If the project has not changed in between the two steps then how is it possible that the Step 2 evaluators grade a project so differently from the Step 3 Jury?
The Step 2 application is presenting an unprecedented level of detail compared to earlier years so a lack of content would be a poor reason for the discrepancy. It is also unlikely that an applicant will intentionally submit false information or act fraudulently so how can such a result be explained?
EIC Jurors vs. Remote Evaluators
The final decision-makers for the grant and equity financing are the EIC jury members but these will only be exposed to the project in the very last step. Step 1 and Step 2 use a pool of thousands of remote evaluators but none of them will be included in Step 3 - the most important step. Step 3 jurors consist of venture capitalists, angel investors, heads of local accelerators and other business experts of varying degrees. Since there is only a handful of such business experts available to the EU, the EIC must rely on remote evaluators.
Due to time constraints, the jury members cannot participate in the large scale evaluations in Step 1 and Step 2. One could also argue that it is impossible to assess a project without an in-person pitch but the submitted business plans are extremely detailed and it should be possible to assess a project accurately.
When seen through this lens, changing the evaluation process from a two-step process in 2020 into a three-step process in 2021 might have been premature. What has been achieved is that the most important decision-makers of the EIC Accelerator grant are only part of 33% of the Steps and are only exposed to approx. 10% of all applications.
The current EIC Accelerator process can be a rollercoaster ride for applicants who received high praise from the evaluators in Step 1 and 2 but sharp criticism from the Step 3 jury. The same can be said of companies which would have succeeded in Step 3 but never got through the evaluators criticisms in Step 1 and 2.
The Future of the EIC Accelerator
The challenge to be addressed is significant: How to find a needle in a haystack without too much effort?
The EIC knows that their budget is limited and that there are excellent companies out there - but how can they find them?
Using a more structured and automated evaluation process (i.e. the EIC Platform and AI Tool) is a good first step. Here are additional functions that could be implemented to further improve the process:
The End in Mind
The final evaluations are executed by the jury members in Step 3 who have a very different level of expertise and focus compared to the remote evaluators. These jury members only see a fraction of all the applicants which means that the EIC is allowing an approximate 90% of applicants to be rejected based on questionable assessments.
Since the remote evaluators clearly misjudge even the very detailed Step 2 applications, there should be a mechanism to either include the jury in the early steps or to further enhance the pool of remote evaluators based on performance data. The problem with the former approach is that the number of remote evaluators is highly limited while the latter approach would require significant data analysis.
Changing the Thresholds
Based on the accumulated data from 2021, it can be assessed if the current thresholds are suitable for this type of financing. If successful applications receiving 2 out of 4 positive assessments in Step 1 never make it to Step 3 or pass this step then the threshold could be increased (i.e. 2/3 or 3/4).
This would reduce the workload for Step 2 applications and could make room for jury members to participate to some degree (i.e. only viewing the pitch decks or a special 3-pager). An approach could be to select Step 2 candidates with a 2/3 or 3/4 threshold and to then have jury members perform the final selection based on a 3-pager or pitch deck even before interview invitations are issued. Requiring perfect consensus among remote evaluators who do not exhibit the same level of expertise as the jury members can be avoided with this approach.
Changing the Type of Evaluation
Depending on the feedback received in Step 3, the evaluation criteria in Step 1 and 2 could be changed if the main reason for rejections is highly consistent among failing applicants (i.e. insufficient business plan, lack of disruption). The current Step 1 acts as a teaser for the project, team and innovation but it might be better to focus on i.e. the business case, competition and product-market-fit if that was what most Step 3 rejectees were lacking.
The same is true for Step 2 where applicants have to prepare a large quantity of information but, if 50% of applicants are still rejected in Step 3, it is meaningless to require so much detail. If the funding rate in Step 3 was 90% then it would validate the high information density requested in Step 2 but it clearly does not seem to provide any extra value compared to 2020 applications outside of presenting a work-load barrier for applicants. Reducing the amount of text provided in Step 2 by half could be a sensible approach.
Analyzing the Pool of Evaluators
All EIC Accelerator applicants have been provided with detailed comments regarding their application, including both positive and negative remarks. This has allowed startups to gain a more transparent look at the internal evaluation process but it has also revealed the flawed system that the EIC is utilizing. If, in Step 2, two evalutors praise the excellent business model, the proposal can still be rejected if the third evaluator disagrees. The explanation given by the EIC is that one has to convince all evaluators and it is insinuated that the two positive assessments might have missed what the third assessment criticised.
But there are obvious problems with this approach and reasoning. The perspective of the evaluators highly depends on their background and experience. Clearly, there are both highly knowledgeable evaluators and those who lack expertise in certain areas. If the evaluators were the right judges for an EIC financing decision then there would be no need for a jury since all Step 2 winners are equally suitable for funding. Or the remote evaluators could participate in a pitch event as Jury members but that is not the case.
Presumably, the EIC is aware of the insufficiencies of its evaluator pool but it has not communicated how it will intervene. Unfortunately, this pool of evaluators is handled in a very non-transparent way.
Analyzing the Pool of Evaluators: Step 1 & 2 Evaluator Score
A solution to that could be to use data based on the thousands of evaluations performed in 2021. Since gradings are given for each key section of the application and consensus between evaluations is a key factor, the EIC can segment evalautors based on the accuracy of their results. If one evaluator consistently gives negative grades then that evaluator might not be suitable to conduct such assessments which is also true for those who consistently grade projects in a positive way.
Going further, AI data analysis can be used to identify the general reasons for rejections and contrast them with the other evaluations as well as the background and track record of the respective evaluator. If an evaluator with a scientific and University background criticises the business model while two entrepreneurial evaluators praise it, then the former should not be weighted in the final result and vice versa.
The EIC can ask their evaluators to fill out questionnaires such as:
- Have you founded a company before?
- Have you held a CEO position?
- What is the largest financing you have raised from investors to date?
- How would you rate your scientific knowledge in the field of Chemistry?
- ...
This can then be used to better understand what a rejection or approval of an evaluator means for a project.
Analyzing the Pool of Evaluators: Step 3 Confidence Score
As an extension, the EIC can introduce Confidence Scores for evaluators which indicates how accurately an evaluator assesses a project based on the jury confirmation.
To achieve this, one can use the scoring by Step 3 jury members who are the final decision-makers as a reference and correlate which Step 2 evaluators were unable to identify what a good business model or innovation would be. This method can work in excluding some of the evaluators that did not judge the project accurately but, unfortunately, it will only exclude the evaluators who err on the positive rather than the negative.
The evaluators who reject suitable candidates because of poor judgement will be less likely to be filtered using this approach since the rejected projects do not make it to Step 3 (unless the suggested Jury-involvement is implemented). To supplement this, re-submissions can be used as a data source. If a project has been rejected in Step 2 and subsequently been re-submitted just to successfully reach Step 3 then the original Step 2 evaluators who rejected the project can be filtered out.
A limitation to this approach is that a re-submission to Step 2 might have implemented substantial changes that have cleared previous insufficiencies thus rendering the initial rejection accurate. But which a substantial dataset size, such cases can be identified.
Based on the collected data, which uses Step 3 assessments as a reference, evaluators can be graded and the high-ranking ones can be used in Step 2 while the low-ranking ones can be used in Step 1 or removed from the pool. Applicants spend a significant amount of time on Step 2 submissions and re-submissions which is why the EIC should work on avoiding contradictory assessments in Step 1, 2 and 3.
