Category: database

The databases have no version control systems. Important fields for identifying individuals were used inconsistently — for example, containing junk data, test data, or null data. Although the tech team used workarounds to compensate for the lack of basic technical standards, they were ad hoc and known by only a handful of individuals, creating business continuity risks should they leave the organization, according to the whistleblower. Despite having been warned of the issues of basic technical practice internally, the NMC failed to acknowledge the problems. Only after exhausting other avenues did the whistleblower raise concern externally with the ICO and The Register. The NMC stores sensitive data on behalf of the professionals that it registers, including gender, sexual orientation, gender identity, ethnicity and nationality, disability details, marital status, as well as other personal information.

The whistleblower’s complaint claims the NMC falls well short of [the standards required under current UK law for data protection and the EU’s General Data Protection Regulation (GDPR)]. The statement alleges that the NMC’s “data management and data retrieval practices were completely unacceptable.” “There is not even much by way of internal structure of the databases for self-documentation, such as primary keys, foreign keys (with a few honorable exceptions), check constraints and table constraints. Even fields that should not be null are nullable. This is frankly astonishing and not the practice of a mature, professional organization,” the statement says. For example, the databases contain a unique ten-digit number (or PRN) to identify individuals registered to the NMC. However, the fields for PRNs sometimes contain individuals’ names, start with a letter or other invalid data, or are simply null. The whistleblower’s complaint says that the PRN problem, and other database design deficiencies, meant that it was nearly impossible to produce “accurate, correct, business critical reports … because frankly no one knows where the correct data is to be found.” A spokesperson for the NMC said the register was “organized and documented” in the SQL Server database. “For clarity, the register of all our nurses, midwives and nursing practitioners is held within Dynamics 365 which is our system of record. This solution and the data held within it, is secure and well documented. It does not rely on any SQL database. The SQL database referenced by the whistleblower relates to our data warehouse which we are in the process of modernizing as previously shared.”

“Google Research maintains a search site for test datasets, too, if you know what you’re looking for,” adds Esther. There’s also, of course, Kaggle.com.

At some point I realized that some queries were taking a few seconds to produce results, so I started to wonder if a more robust database such as PostgreSQL would provide better performance. Having publicly professed my dislike of performance benchmarks, I resisted the urge to look up any comparisons online, and instead embarked on a series of experiments to accurately measure the performance of these two databases for the specific use cases of this application. What follows is a detailed account of my effort, the results of my testing (including a surprising twist!), and my analysis and final decision, which ended up being more involved than I expected. […] If you are going to take one thing away from this article, I hope it is that the only benchmarks that are valuable are those that run on your own platform, with your own stack, with your own data, and with your own software. And even then, you may need to add custom optimizations to get the best performance.