AI Researcher Warns Data Science Could Face a Reproducibility Crisis

Long-time Slashdot reader theodp shared this warning from a long-time AI researcher arguing that data science “is due” for a reckoning over whether results can be reproduced. “Few technological revolutions came with such a low barrier of entry as Machine Learning…”
Unlike Machine Learning, Data Science is not an academic discipline, with its own set of algorithms and methods… There is an immense diversity, but also disparities in skill, expertise, and knowledge among Data Scientists… In practice, depending on their backgrounds, data scientists may have large knowledge gaps in computer science, software engineering, theory of computation, and even statistics in the context of machine learning, despite those topics being fundamental to any ML project. But it’s ok, because you can just call the API, and Python is easy to learn. Right…?

Building products using Machine Learning and data is still difficult. The tooling infrastructure is still very immature and the non-standard combination of data and software creates unforeseen challenges for engineering teams. But in my views, a lot of the failures come from this explosive cocktail of ritualistic Machine Learning:

– Weak software engineering knowledge and practices compounded by the tools themselves;
– Knowledge gap in mathematical, statistical, and computational methods, encouraged black boxing API;
– Ill-defined range of competence for the role of data scientist, reinforced by a pool of candidates with an unusually wide range of backgrounds;
– A tendency to follow the hype rather than the science. –

What can you do?

– Hold your data scientists accountable using Science.
– At a minimum, any AI/ML project should include an Exploratory Data Analysis, whose results directly support the design choices for feature engineering and model selection.
– Data scientists should be encouraged to think outside-of-the box of ML, which is a very small box
– Data scientists should be trained to use eXplainable AI methods to provide context about the algorithm’s performance beyond the traditional performance metrics like accuracy, FPR, or FNR.
– Data scientists should be held at similar standards than other software engineering specialties, with code review, code documentation, and architectural designs.

The article concludes, “Until such practices are established as the norm, I’ll remain skeptical of Data Science.”

Read more of this story at Slashdot.

ASUS Promises Support Overhaul After YouTube Investigators Allege Dishonesty

ASUS has suddenly agreed “to overhaul its customer support and warranty systems,” writes the hardware review site Gamers Nexus — after a three-video series on its YouTube channel documented bad and “potentially illegal” handling of customer warranties for the channel’s 2.2 million viewers.

The Verge highlights ASUS’s biggest change:
If you’ve ever been denied a warranty repair or charged for a service that was unnecessary or should’ve been free, Asus wants to hear from you at a new email address. It claims those disputes will be processed by Asus’ own staff rather than outsourced customer support agents…. The company is also apologizing today for previous experiences you might have had with repairs. “We’re very sorry to anyone who has had a negative experience with our service team. We appreciate your feedback and giving us a chance to make amends.”

It started five weeks ago when Gamers Nexus requested service for a joystick problem, according to a May 10 video. First they’d received a response wrongly telling them their damage was out of warranty — which also meant Asus could add a $20 shipping charge for the requested repair. “Somehow that turned into ASUS saying the LCD needs to be replaced, even though the joystick is covered under their repair policies,” the investigators say in the video. [They also note this response didn’t even address their original joystick problem — “only that thing that they had decided to find” — and that ASUS later made an out-of-the-blue reference to “liquid damage.”] The repair would ultimately cost $191.47, with ASUS mentioning that otherwise “the unit will be sent back un-repaired and may be disassembled.” ASUS gave them four days to respond, with some legalese adding that an out-of-warranty repair fee is non-refundable, yet still “does not guarantee that repairs can be made.”

Even when ASUS later agreed to do a free “partial” repair (providing the requested in-warranty service), the video’s investigators still received another email warning of “pending service cancellation” and return of the unit unless they spoke to “Invoice Quotation Support” immediately. The video-makers stood firm, and the in-warranty repair was later performed free — but they still concluded that “It felt like ASUS tried to scam us.” ASUS’s response was documented in a second video, with ASUS claiming it had merely been sending a list of “available” repairs (and promising that in the future ASUS would stop automatically including costs for the unrequested repair of “cosmetic imperfections” — and that they’d also change their automatic emails.)
Gamers Nexus eventually created a fourth, hour-long video confronting various company officials at Computex — which finally led to them publishing a list of ASUS’s promised improvements on Friday. Some highlights:

ASUS promises it’s “created a Task Force team to retroactively go back through a long history of customer surveys that were negative to try and fix the issues.” (The third video from Gamers Nexus warned ASUS was already on the government’s radar over its handling of warranty issues.)
ASUS also announced their repairs centers were no longer allowed to claim “customer-induced damage” (which Gamers Nexus believes “will remove some of the financial incentive to fail devices” to speed up workloads).
ASUS is creating a new U.S. support center allowing customers to choose either a refurbished board or a longer repair.

Gamers Nexus says they already have devices at ASUS repair centers — under pseudonyms — and that they “plan to continue sampling them over the next 6-12 months so we can ensure these are permanent improvements.”

And there’s one final improvement, according to Gamers Nexus. “After over a year of refusing to acknowledge the microSD card reader failures on the ROG Ally [handheld gaming console], ASUS will be posting a formal statement next week about the defect.”

Read more of this story at Slashdot.

Researchers Find No Amount of Alcohol is Healthy For You

The New York Times magazine remembers that once upon a time, in the early 1990s, “some prominent researchers were promoting, and the media helped popularize, the idea that moderate drinking…was linked to greater longevity.

“The cause of that association was not clear, but red wine, researchers theorized, might have anti-inflammatory properties that extended life and protected cardiovascular health…”

More recently, though, research has piled up debunking the idea that moderate drinking is good for you. Last year, a major meta-analysis that re-examined 107 studies over 40 years came to the conclusion that no amount of alcohol improves health; and in 2022, a well-designed study found that consuming even a small amount brought some risk to heart health. That same year, Nature published research stating that consuming as little as one or two drinks a day (even less for women) was associated with shrinkage in the brain — a phenomenon normally associated with aging…

[M]ore people are now reporting that they consume cannabis than alcohol on a daily basis. Some governments are responding to the new research by overhauling their messaging. Last year, Ireland became the first country to pass legislation requiring a cancer warning on all alcohol products sold there, similar to those found on cigarettes: “There is a direct link between alcohol and fatal cancers,” the language will read. And in Canada, the government has revised its alcohol guidelines, announcing: “We now know that even a small amount of alcohol can be damaging to health.” The guidelines characterize one to two drinks a week as carrying “low risk” and three to six drinks as carrying “moderate risk.” (Previously the guidelines suggested that women limit themselves to no more than two standard drinks most days, and that men place that limit at three.)

Read more of this story at Slashdot.

Security Lessons from the Change Healthcare Ransomware Catastrophe

The $22 million paid by Change Healthcare’s parent company to unlock its systems “may have emboldened bad actors to further target the vulnerable industry,” writes Axios:
There were 44 attacks against the health care sector in April, the most that [cybersecurity firm] Recorded Future has seen in the four years it’s been collecting data. It was also the second-largest month-over-month jump, after 30 ransomware attacks were recorded in March. There were 32 attacks in February and May.

But an analysis by the security-focused magazine CSO says the “disastrous” incident also “starkly illustrated the fragility of the healthcare sector, prompting calls for regulatory action.”
In response to the attack, US politicians have called for mandated baseline cybersecurity standards in the health sector, as well as better information sharing. They have also raised concerns that industry consolidation is increasing cyber risk.

So what went wrong? The attackers used a set of stolen credentials to remotely access the company’s systems. But the article also notes Change Healthcare’s systems “suffered from a lack of segmentation, which enables easy lateral movement of the attack” — and that the company’s acquisition may have played a role:

Mergers and acquisitions create new cyber threats because they involve the integration of systems, data, and processes from different organizations, each with its own security protocols and potential vulnerabilities. “During this transition, cybercriminals can exploit discrepancies in security measures, gaps in IT governance, and the increased complexity of managing merged IT environments,” Aron Brand, CTO of CTERA told CSOonline. “Additionally, the heightened sharing of sensitive information between parties provides more opportunities for data breaches.”

And “In the end, paying the ransom failed to protect UHG from secondary attempts at extortion.”

In April, cybercriminals from the RansomHub group threatened to leak portions of 6TB of sensitive data stolen from the breach of Change Healthcare, and obtained through Nichy, according to an analysis by security vendor Forescout. An estimated one in three Americans had their sensitive data exposed as a result of the attack. Such secondary scams are becoming increasingly commonplace and healthcare providers are particularly at risk, according to compliance experts… The US Department of Health and Human Services (HHS) is investigating whether a breach of protected health information occurred in assessing whether either UHG or Change Healthcare violated strict healthcare sector privacy regulations.

Thanks to Slashdot reader snydeq for sharing the article.

Read more of this story at Slashdot.

Is C++ More Popular Than C?

Last month TIOBE announced its estimate that the four most popular programming languages were:
1. Python
2. C
3. C++
4. Java
But this month C++ “overtook” C for the first time, TIOBE announced, becoming (according to the same methodology) the #2 most popular programming language, with C dropping to #3. ” C++ has never been that high in the TIOBE index,” says TIOBE Software CEO Paul Jansen in the announcement, “whereas C has never been that low.”

1. Python
2. C++
3. C
4. Java

C++ started a new life as of 2011 with its consistent 3 yearly updates. Although most compilers and most engineers can’t take up with this pace, it is considered a success to see the language evolve.

The main strengths of C++ are its performance and scalability. Its downside is its many ways to get things done, i.e. its rich idiom of features, which is caused by its long history and aim for backward compatibility.

C++ is heavily used in embedded systems, game development and financial trading software, just to name a few domains.

There’s different rankings from the rival PYPL index of programming language popularity. It lumps C and C++ together to award them a collective ranking (#5). But unlike TIOBE, it shows Java [and JavaScript and C#] all being more popular (with Python still the #1 most popular language).

Of course, statistical anomalies could be also skewing the results. Visual Basic also lost two ranks in popularity in the last month, according to TIOBE, dropping from the #7 position to the #9 position (now falling just behind Go and SQL). This becomes the first time that Go has risen as high as #7, according to TIOBE’s announcement — with Rust also reaching an all-time high of #17…

Read more of this story at Slashdot.

GPT-4 Has Passed the Turing Test, Researchers Claim

Drew Turney reports via Live Science: The “Turing test,” first proposed as “the imitation game” by computer scientist Alan Turing in 1950, judges whether a machine’s ability to show intelligence is indistinguishable from a human. For a machine to pass the Turing test, it must be able to talk to somebody and fool them into thinking it is human. Scientists decided to replicate this test by asking 500 people to speak with four respondents, including a human and the 1960s-era AI program ELIZA as well as both GPT-3.5 and GPT-4, the AI that powers ChatGPT. The conversations lasted five minutes — after which participants had to say whether they believed they were talking to a human or an AI. In the study, published May 9 to the pre-print arXiv server, the scientists found that participants judged GPT-4 to be human 54% of the time.

ELIZA, a system pre-programmed with responses but with no large language model (LLM) or neural network architecture, was judged to be human just 22% of the time. GPT-3.5 scored 50% while the human participant scored 67%. “Machines can confabulate, mashing together plausible ex-post-facto justifications for things, as humans do,” Nell Watson, an AI researcher at the Institute of Electrical and Electronics Engineers (IEEE), told Live Science. “They can be subject to cognitive biases, bamboozled and manipulated, and are becoming increasingly deceptive. All these elements mean human-like foibles and quirks are being expressed in AI systems, which makes them more human-like than previous approaches that had little more than a list of canned responses.” Further reading: 1960s Chatbot ELIZA Beat OpenAI’s GPT-3.5 In a Recent Turing Test Study

Read more of this story at Slashdot.