Why I refuse to use Codility and so should You | Tymoteusz Paul - X20X Development

Why I refuse to use Codility and so should You

Posted on February 17, 2014


Ah the Codility, project that was born out of necessity and over the time turned into a monster that does exactly the opposite of what was intended. A tool that every recruiter loves, every wanna-be programmer fears and every person who writes software, in a commercial environment, laughs at (if they don’t, they should!). And all this hatred comes from a very simple reason – Codility doesn’t test programmers; it tests ability to google and use notepad. And I am not just blindly ranting, so bare with me, while I explain the top issues I have with Codility.

Let’s start with explaining what Codility is, for those who may be unfamiliar with the concept. It is a website where recruiters can invite a person to solve a programming challenge. Once you follow the invitation, you will be presented with a notepad, a task and some fields in which you can provide test cases to your solution. Everything you type into the cute notepad is recorded on the timeline and can later on be reviewed by a recruiter, together with a score – a measure of how well you did on your test. This is where the first issue arises – the notepad. While there still are some whackos out there who use text editors for heavy duty programming (not just quick script/fix), but they are a minority and a dying breed. The standard today (cheap standard, as you can get modern and robust IDEs for as little as 0$) is to use IDE, which provides you with proper code completion, online code validation and API suggestions. With Codility notepad, you get none of that as all you see is basic code coloring. No completion, no API suggestions, no PEP-8 validation. I frankly cannot even get the print right (especially since I moved to python 3), and my IDE is notoriously correcting me. Does that make me a bad programmer? Maybe, but I have 45$ tool called PyCharm (I got it cheap) that completely resolves the problem with remembering the order of arguments for each function. I don’t know a single programmer who doesn’t ridicule the notion of writing non-pseudocode code on paper; this is no different. Of course, I could ignore it, fire up my visual studio and just develop the proper solution there, but then if a recruiter peeks into the solution then all he will see is one big copy/paste (and most likely will conclude that I am a cheat).

But that brings me right to the next point – all of their questions are old, well known, and they are slow in adding new ones. And when they finally do – Codility is infested with people who apparently pay their subscription fee just to do the solutions and post them all around the web. Seriously, you can find solutions to the new questions just a day or two after they were added. And they usually are good, better than what most coders can come up within 30 minutes, especially without the proper environment.

But let’s just say that you are an honest person and enjoy the tournament of writing in a notepad. This brings us right to the tasks, the bread and butter . So let’s have a quick look at a sample task that is available at hand:

The prime is a positive integer X that has exactly two distinct divisors: 1 and X. The first few prime integers are 2, 3, 5, 7, 11 and 13. The semiprime is a natural number that is the product of two (not necessarily distinct) prime numbers. The first few semiprimes are 4, 6, 9, 10, 14, 15, 21, 22, 25, 26.

You are given two non-empty zero-indexed arrays P and Q, each consisting of M integers. These arrays represent queries about the number of semiprimes within specified ranges.

Query K requires you to find the number of semiprimes within the range (P[K], Q[K]), where 1 ≤ P[K] ≤ Q[K] ≤ N.

For example, consider an integer N = 26 and arrays P, Q such that:

P[0] = 1 Q[0] = 26 P[1] = 4 Q[1] = 10 P[2] = 16 Q[2] = 20

The number of semiprimes within each of these ranges is as follows:

(1, 26) is 10, (4, 10) is 4, (16, 20) is 0. struct Results {

int * A;

int M;


Write a function:

struct Results solution(int N, int P[], int Q[], int M);

that, given an integer N and two non-empty zero-indexed arrays P and Q consisting of M integers, returns an array consisting of M elements specifying the consecutive answers to all the queries.

For example, given an integer N = 26 and arrays P, Q such that:

P[0] = 1 Q[0] = 26 P[1] = 4 Q[1] = 10 P[2] = 16 Q[2] = 20

the function should return the values [10, 4, 0], as explained above.

Assume that:

N is an integer within the range [1..50,000]; M is an integer within the range [1..30,000]; each element of array P is an integer within the range [1..N]; each element of array Q is an integer within the range [1..N]; P[i] ≤ Q[i]. Complexity:

expected worst-case time complexity is O(N*log(log(N))+M); expected worst-case space complexity is O(N+M), beyond input storage (not counting the storage required for input arguments). Elements of input arrays can be modified.

I must say that in my long career in IT, which included AI development, neural networks, trying to beat my PC at rock-paper-scissors, I never had a task like that. And I do have almost complete history of all tasks I have ever resolved (I am an e-hoarder). Most of my tasks begin with “fix..,” “figure out….,” “rework/rewrite for performance….,” not with “for collection X process it to Y without loops”. That is another issue I have with Codility, as it doesn’t test your knowledge of a specific language, or even its libraries but their knowledge of mathematics and algorithms. While most people like to think that programming is mostly about creating algorithms, spending hours in debates over one line of code – it isn’t. It’s a world where you sit down, write some code, run it through tests and pray that it works. It always depends on your knowledge of the language, very often of knowledge of available libraries and very rarely on your knowledge of mathematics. Quite frankly most programmers will never deal with a mathematical issue in their whole career, and still they will be delivering very good code that works and brings money. On top of that not only is your time limited and short, by default you get 30 minutes to solve a challenge, you are also valued on performance of your task. And you don’t need just OK performance; your script is ran against solutions that teams of developers optimized over a long period, which also included peer review. If you think that even an exceptional programmer knows all performance chinks for every part of the language, you are misguided and just plain wrong. It is even more valid in modern programming world, a world in which scaling up and sideways is not only as easy as moving a slider from one side to another, but is also much cheaper than putting extra day or two to squeeze every bit of performance from a routine.

But let’s keep going, you feel like sharpening your math-foo, so you work very hard on the solution in their little notepad, you test it against all the cases you can imagine and it passes, so you submit your work. What happens next? Nothing! As a candidate you do not receive any feedback, you will never know if/what was wrong with your solution, not even something as simple as “your score is 70%,” not to mention details of what use case You’ve failed. This is an awful practice which does not create a better programmer, it only breaks people down when after beating a recruiter into a pulp they hear “your score was so low that we will reject you”. What would make codility half-useful, at least in the self-improving aspect, would be to present candidate with his results, list of failed tests and some articles explaining why your solution may have failed those.

But I guess that if they started doing that the tests would be compromised even sooner than they are now, and in much broader variety of languages. Effect of that would be that they would need to come up with new challenges all the time, which just may break their model of pair-bottomfeeding with so called “recruitment agencies”. And while I do understand that there are some jobs that do require that level of scrutiny that codility expects from candidates, I also know from experience that the high-ticket companies would never entrust such easy-to-cheat tool with any level of screening.

Copyright © x20x.co.uk