Anthropic Looks To Fund a New, More Comprehensive Generation of AI Benchmarks

AI firm Anthropic launched a funding program Monday to develop new benchmarks for evaluating AI models, including its chatbot Claude. The initiative will pay third-party organizations to create metrics for assessing advanced AI capabilities. Anthropic aims to “elevate the entire field of AI safety” with this investment, according to its blog. TechCrunch adds: As we’ve highlighted before, AI has a benchmarking problem. The most commonly cited benchmarks for AI today do a poor job of capturing how the average person actually uses the systems being tested. There are also questions as to whether some benchmarks, particularly those released before the dawn of modern generative AI, even measure what they purport to measure, given their age.

The very-high-level, harder-than-it-sounds solution Anthropic is proposing is creating challenging benchmarks with a focus on AI security and societal implications via new tools, infrastructure and methods.

Read more of this story at Slashdot.

Alzheimer’s Scientist Indicted For Allegedly Falsifying Data In $16 Million Scheme

“A federal grand jury has indicted an embattled Alzheimer’s researcher for allegedly falsifying data to fraudulently obtain $16 million in federal research funding from the National Institutes of Health for the development of a controversial Alzheimer’s drug and diagnostic test,” writes Beth Mole via Ars Technica. “Wang is charged with one count of major fraud against the United States, two counts of wire fraud, and one count of false statements. If convicted, he faces a maximum penalty of 10 years in prison for the major fraud charge, 20 years in prison for each count of wire fraud, and five years in prison for the count of false statements […].” From the report: Hoau-Yan Wang, 67, a medical professor at the City University of New York, was a paid collaborator with the Austin, Texas-based pharmaceutical company Cassava Sciences. Wang’s research and publications provided scientific underpinnings for Cassava’s Alzheimer’s treatment, Simufilam, which is now in Phase III trials. Simufilam is a small-molecule drug that Cassava claims can restore the structure and function of a scaffolding protein in the brain of people with Alzheimer’s, leading to slowed cognitive decline. But outside researchers have long expressed doubts and concerns about the research.

In 2023, Science magazine obtained a 50-page report from an internal investigation at CUNY that looked into 31 misconduct allegations made against Wang in 2021. According to the report, the investigating committee “found evidence highly suggestive of deliberate scientific misconduct by Wang for 14 of the 31 allegations,” the report states. The allegations largely centered around doctored and fabricated images from Western blotting, an analytical technique used to separate and detect proteins. However, the committee couldn’t conclusively prove the images were falsified “due to the failure of Dr. Wang to provide underlying, original data or research records and the low quality of the published images that had to be examined in their place.” In all, the investigation “revealed long-standing and egregious misconduct in data management and record keeping by Dr. Wang,” and concluded that “the integrity of Dr. Wang’s work remains highly questionable.” The committee also concluded that Cassava’s lead scientist on its Alzheimer’s disease program, Lindsay Burns, who was a frequent co-author with Wang, also likely bears some responsibility for the misconduct.

In March 2022, five of Wang’s articles published in the journal PLOS One were retracted over integrity concerns with images in the papers. Other papers by Wang have also been retracted or had statements of concern attached to them. Further, in September 2022, the Food and Drug Administration conducted an inspection of the analytical work and techniques used by Wang to analyze blood and cerebrospinal fluid from patients in a simufilam trial. The investigation found a slew of egregious problems, which were laid out in a “damning” report (PDF) obtained by Science. In the indictment last week (PDF), federal authorities were explicit about the allegations, claiming that Wang falsified the results of his scientific research to NIH “by, among other things, manipulating data and images of Western blots to artificially add bands [which represent proteins], subtract bands, and change their relative thickness and/or darkness, and then drawing conclusions” based on those false results.

Read more of this story at Slashdot.

Microsoft Tells Yet More Customers Their Emails Have Been Stolen

Microsoft revealed that the Russian hackers who breached its systems earlier this year stole more emails than initially reported. “We are continuing notifications to customers who corresponded with Microsoft corporate email accounts that were exfiltrated by the Midnight Blizzard threat actor, and we are providing the customers the email correspondence that was accessed by this actor,” a Microsoft spokesperson told Bloomberg (paywalled). “This is increased detail for customers who have already been notified and also includes new notifications.” The Register reports: We’ve been aware for some time that the digital Russian break-in at the Windows maker saw Kremlin spies make off with source code, executive emails, and sensitive U.S. government data. Reports last week revealed that the issue was even larger than initially believed and additional customers’ data has been stolen. Along with Russia, Microsoft was also compromised by state actors from China not long ago, and that issue similarly led to the theft of emails and other data belonging to senior U.S. government officials.

Both incidents have led experts to call Microsoft a threat to U.S. national security, and president Brad Smith to issue a less-than-reassuring mea culpa to Congress. All the while, the U.S. government has actually invested more in its Microsoft kit. Bloomberg reported that emails being sent to affected Microsoft customers include a link to a secure environment where customers can visit a site to review messages Microsoft identified as having been compromised. But even that might not have been the most security-conscious way to notify folks: Several thought they were being phished.

Read more of this story at Slashdot.

William Gibson’s ‘Neuromancer’ to Become a Series on Apple TV+

It’s been adapted into a graphic novel, a videogame, a radio play, and an opera, according to Wikipedia — which also describes years of trying to adapt Neuromancer into a movie. “The landmark 1984 cyberpunk novel has been on Hollywood’s wishlist for decades,” writes Gizmodo, “with multiple filmmakers attempting to bring it to the big screen.” (Back in 2010, Slashdot’s CmdrTaco even posted an update with the headline “Neuromancer Movie In Your Future?” with a 2011 story promising the movie deal was “moving forward….”)

But now Deadline reports it’s becoming a 10-episode series on Apple TV+ (co-produced by Apple Studios) starring Callum Turner and Brianna Middleton:
Created for television by Graham Roland and JD Dillard, Neuromancer follows a damaged, top-rung super-hacker named Case (Turner) who is thrust into a web of digital espionage and high stakes crime with his partner Molly (Middleton), a razor-girl assassin with mirrored eyes, aiming to pull a heist on a corporate dynasty with untold secrets.
More from Gizmodo:
“We’re incredibly excited to be bringing this iconic property to Apple TV+,” Roland and Dillard said in a statement. “Since we became friends nearly 10 years ago, we’ve looked for something to team up on, so this collaboration marks a dream come true. Neuromancer has inspired so much of the science fiction that’s come after it and we’re looking forward to bringing television audiences into Gibson’s definitive ‘cyberpunk’ world.”
The novel launched Gibson’s “Sprawl” trilogy of novels (building on the dystopia in his 1982 short story “Burning Chrome”), also resurrecting the “Molly Millions” character from Johnny Mnemonic — an even earlier short story from 1981…

Read more of this story at Slashdot.

Caching Is Key, and SIEVE Is Better Than LRU

USENIX, the long-running OS/networking research group, also publishes a magazine called ;login:. Today the magazine’s editor — security consultant Rik Farrow — stopped by Slashdot to share some new research. rikfarrow writes:
Caching means using faster memory to store frequently requested data, and the most commonly used algorithm for determining which items to discard when the cache is full is Least Recently Used [or “LRU”]. These researchers have come up with a more efficient and scalable method that uses just a few lines of code to convert LRU to SIEVE.
Just like a sieve, it sifts through objects (using a pointer called a “hand”) to “filter out unpopular objects and retain the popular ones,” with popularity based on a single bit that tracks whether a cached object has been visited:

As the “hand” moves from the tail (the oldest object) to the head (the newest object), objects that have not been visited are evicted… During the subsequent rounds of sifting, if objects that survived previous rounds remain popular, they will stay in the cache. In such a case, since most old objects are not evicted, the eviction hand quickly moves past the old popular objects to the queue positions close to the head. This allows newly inserted objects to be quickly assessed and evicted, putting greater eviction pressure on unpopular items (such as “one-hit wonders”) than LRU-based eviction algorithms.
It’s an example of “lazy promotion and quick demotion”. Popular objects get retained with minimal effort, with quick demotion “critical because most objects are not reused before eviction.”

After 1559 traces (of 247,017 million requests to 14,852 million objects), they found SIEVE reduces the miss ratio (when needed data isn’t in the cache) by more than 42% on 10% of the traces with a mean of 21%, when compared to FIFO. (And it was also faster and more scalable than LRU.)

“SIEVE not only achieves better efficiency, higher throughput, and better scalability, but it is also very simple.”

Read more of this story at Slashdot.