Time for the Kdb/Q specialists to quit finance?

If you're a developer specialized in the Kdb database and its associated programming language, Q, the chances are that you work in financial services. After all, the whole Kdb paradigm was engineered by Arthur Whitney, a former Morgan Stanley technologist after he quit banking in 1993. Since then, Kdb has evolved into the de facto system for banks, hedge funds, and high frequency trading houses looking for fast data extraction and analytics technologies. Now, however, finance firms have competition: their army of Kdb talent is wanted elsewhere.

"Other areas are beginning to catch up [with the need to process large amounts of data]" says Paul Bilokon, a Deutsche Bank director and founder and CEO of Thalesians, a think tank for people interested in quantitative finance, economics, computer science and physics. "I suspect Kdb+/Q specialists will be in high demand in other areas - from civil engineering, to bioinformatics, to medicine."

A research paper released last year by Bloor, the IT consulting company, underscores the coming threat. Bloor predicted that Kdb's architecture will increasingly be used across all industries using streaming analytics. What started life as a database system used to underpin banks' electronic trading and analytics platforms now has applications across the internet of things, the retail sector, self-driving cars, smart technology and industrial automation. The more the world runs on real-time analysis of data, the more that today's Kdb specialists will be wanted outside banks.

This has the potential to create problems. Banks remain highly reliant on their pool of Kdb expertise. J.P. Morgan, for example, is currently looking for a Kdb developer and data scientist to join its electronic market making team, Bank of America wants a Kdb/Q developer to work on its algorithmic trading systems, Morgan Stanley wants a Kdb developer for its metrics team in New York. The list goes on.

Of course, Kdb isn't the only database option - there are also Hadoop and SQL and Cassandra and MongoDB waiting in the wings. But the way KX, the company that sells Kdb tells it, the alternatives are all inferior.  "Kdb+ is particularly good (both in terms of performance and functionality) at processing, manipulating and analysing data (especially numeric data) in real-time, alongside the analysis of historical data," says KX, adding that, "The Q language is significantly more efficient than other languages that you might use (both procedural and declarative) for analysis purposes."

You might say that KX would say this, but Bilokon agrees: "In my personal view, nothing has so far approached Kdb+ in usability, flexibility and speed."

Recruiters are already alert to the possibility of pulling Kdb specialists out of banks. "Any organization that needs a lot of data processed very quickly is going to need these people," says one Kdb-focused search consultant in London. "Investment banks have had a monopoly on these people but you're going to see demand from the likes of the NHS." Another recruiter, also speaking off the record, says Kdb specialists have clustered in London and New York City: "Historically you've seen almost none of them in Silicon Valley."

As Kdb specialists eye alternatives to finance, banks may need to hike pay to stop the flow. Recruiters say Kdb experts don't command much of a premium: in London you might get £650 a day as an experienced Kdb/Q programmer, but you can get more than that if you're excellent in Java.

The alternative to pay rises for existing talent is for banks to train new people up. Q isn't an easy language to learn, but Bilokon predicts banks will start training people in-house to meet their needs. One recruiter says this is happening already. Failing that, First Derivatives, a company which partially owns KX Systems (the company founded by Whitney which owns the Kdb architecture). runs a training program of its own. - Elizabeth O'Hanlon, global talent acquisition manager at First Derivatives says they hire all year round (the graduate intake is thought to be around 250) and have increased recruitment to match rising demand.

Contact: sbutcher@efinancialcareers.com

Photo credit: confused perfectionist running away from his perfection by paolobarzman is licensed under CC BY 2.0.