Database For UK Nurse Registration ‘Completely Unacceptable’

Lindsay Clark reports via The Register: The UK Information Commissioner’s Office has received a complaint detailing the mismanagement of personal data at the Nursing and Midwifery Council (NMC), the regulator that oversees worker registration. Employment as a nurse or midwife depends on enrollment with the NMC in the UK. According to whistleblower evidence seen by The Register, the databases on which the personal information is held lack rudimentary technical standards and practices. The NMC said its data was secure with a high level of quality, allowing it to fulfill its regulatory role, although it was on “a journey of improvement.” But without basic documentation, or the primary keys or foreign keys common in database management, the Microsoft SQL Server databases — holding information about 800,000 registered professionals — are difficult to query and manage, making assurances on governance nearly impossible, the whistleblower told us.

The databases have no version control systems. Important fields for identifying individuals were used inconsistently — for example, containing junk data, test data, or null data. Although the tech team used workarounds to compensate for the lack of basic technical standards, they were ad hoc and known by only a handful of individuals, creating business continuity risks should they leave the organization, according to the whistleblower. Despite having been warned of the issues of basic technical practice internally, the NMC failed to acknowledge the problems. Only after exhausting other avenues did the whistleblower raise concern externally with the ICO and The Register. The NMC stores sensitive data on behalf of the professionals that it registers, including gender, sexual orientation, gender identity, ethnicity and nationality, disability details, marital status, as well as other personal information.

The whistleblower’s complaint claims the NMC falls well short of [the standards required under current UK law for data protection and the EU’s General Data Protection Regulation (GDPR)]. The statement alleges that the NMC’s “data management and data retrieval practices were completely unacceptable.” “There is not even much by way of internal structure of the databases for self-documentation, such as primary keys, foreign keys (with a few honorable exceptions), check constraints and table constraints. Even fields that should not be null are nullable. This is frankly astonishing and not the practice of a mature, professional organization,” the statement says. For example, the databases contain a unique ten-digit number (or PRN) to identify individuals registered to the NMC. However, the fields for PRNs sometimes contain individuals’ names, start with a letter or other invalid data, or are simply null. The whistleblower’s complaint says that the PRN problem, and other database design deficiencies, meant that it was nearly impossible to produce “accurate, correct, business critical reports … because frankly no one knows where the correct data is to be found.” A spokesperson for the NMC said the register was “organized and documented” in the SQL Server database. “For clarity, the register of all our nurses, midwives and nursing practitioners is held within Dynamics 365 which is our system of record. This solution and the data held within it, is secure and well documented. It does not rely on any SQL database. The SQL database referenced by the whistleblower relates to our data warehouse which we are in the process of modernizing as previously shared.”

Read more of this story at Slashdot.

SQLite or PostgreSQL? It’s Complicated!

Miguel Grinberg, a Principal Software Engineer for Technical Content at Twilio, writes in a blog post: We take blogging very seriously at Twilio. To help us understand what content works well and what doesn’t on our blog, we have a dashboard that combines the metadata that we maintain for each article such as author, team, product, publication date, etc., with traffic information from Google Analytics. Users can interactively request charts and tables while filtering and grouping the data in many different ways. I chose SQLite for the database that supports this dashboard, which in early 2021 when I built this system, seemed like a perfect choice for what I thought would be a small, niche application that my teammates and I can use to improve our blogging. But almost a year and a half later, this application tracks daily traffic for close to 8000 articles across the Twilio and SendGrid blogs, with about 6.5 million individual daily traffic records, and with a user base that grew to over 200 employees.

At some point I realized that some queries were taking a few seconds to produce results, so I started to wonder if a more robust database such as PostgreSQL would provide better performance. Having publicly professed my dislike of performance benchmarks, I resisted the urge to look up any comparisons online, and instead embarked on a series of experiments to accurately measure the performance of these two databases for the specific use cases of this application. What follows is a detailed account of my effort, the results of my testing (including a surprising twist!), and my analysis and final decision, which ended up being more involved than I expected. […] If you are going to take one thing away from this article, I hope it is that the only benchmarks that are valuable are those that run on your own platform, with your own stack, with your own data, and with your own software. And even then, you may need to add custom optimizations to get the best performance.

Read more of this story at Slashdot.