Anthropic Looks To Fund a New, More Comprehensive Generation of AI Benchmarks

AI firm Anthropic launched a funding program Monday to develop new benchmarks for evaluating AI models, including its chatbot Claude. The initiative will pay third-party organizations to create metrics for assessing advanced AI capabilities. Anthropic aims to “elevate the entire field of AI safety” with this investment, according to its blog. TechCrunch adds: As we’ve highlighted before, AI has a benchmarking problem. The most commonly cited benchmarks for AI today do a poor job of capturing how the average person actually uses the systems being tested. There are also questions as to whether some benchmarks, particularly those released before the dawn of modern generative AI, even measure what they purport to measure, given their age.

The very-high-level, harder-than-it-sounds solution Anthropic is proposing is creating challenging benchmarks with a focus on AI security and societal implications via new tools, infrastructure and methods.

Read more of this story at Slashdot.

Microsoft Tells Yet More Customers Their Emails Have Been Stolen

Microsoft revealed that the Russian hackers who breached its systems earlier this year stole more emails than initially reported. “We are continuing notifications to customers who corresponded with Microsoft corporate email accounts that were exfiltrated by the Midnight Blizzard threat actor, and we are providing the customers the email correspondence that was accessed by this actor,” a Microsoft spokesperson told Bloomberg (paywalled). “This is increased detail for customers who have already been notified and also includes new notifications.” The Register reports: We’ve been aware for some time that the digital Russian break-in at the Windows maker saw Kremlin spies make off with source code, executive emails, and sensitive U.S. government data. Reports last week revealed that the issue was even larger than initially believed and additional customers’ data has been stolen. Along with Russia, Microsoft was also compromised by state actors from China not long ago, and that issue similarly led to the theft of emails and other data belonging to senior U.S. government officials.

Both incidents have led experts to call Microsoft a threat to U.S. national security, and president Brad Smith to issue a less-than-reassuring mea culpa to Congress. All the while, the U.S. government has actually invested more in its Microsoft kit. Bloomberg reported that emails being sent to affected Microsoft customers include a link to a secure environment where customers can visit a site to review messages Microsoft identified as having been compromised. But even that might not have been the most security-conscious way to notify folks: Several thought they were being phished.

Read more of this story at Slashdot.

Caching Is Key, and SIEVE Is Better Than LRU

USENIX, the long-running OS/networking research group, also publishes a magazine called ;login:. Today the magazine’s editor — security consultant Rik Farrow — stopped by Slashdot to share some new research. rikfarrow writes:
Caching means using faster memory to store frequently requested data, and the most commonly used algorithm for determining which items to discard when the cache is full is Least Recently Used [or “LRU”]. These researchers have come up with a more efficient and scalable method that uses just a few lines of code to convert LRU to SIEVE.
Just like a sieve, it sifts through objects (using a pointer called a “hand”) to “filter out unpopular objects and retain the popular ones,” with popularity based on a single bit that tracks whether a cached object has been visited:

As the “hand” moves from the tail (the oldest object) to the head (the newest object), objects that have not been visited are evicted… During the subsequent rounds of sifting, if objects that survived previous rounds remain popular, they will stay in the cache. In such a case, since most old objects are not evicted, the eviction hand quickly moves past the old popular objects to the queue positions close to the head. This allows newly inserted objects to be quickly assessed and evicted, putting greater eviction pressure on unpopular items (such as “one-hit wonders”) than LRU-based eviction algorithms.
It’s an example of “lazy promotion and quick demotion”. Popular objects get retained with minimal effort, with quick demotion “critical because most objects are not reused before eviction.”

After 1559 traces (of 247,017 million requests to 14,852 million objects), they found SIEVE reduces the miss ratio (when needed data isn’t in the cache) by more than 42% on 10% of the traces with a mean of 21%, when compared to FIFO. (And it was also faster and more scalable than LRU.)

“SIEVE not only achieves better efficiency, higher throughput, and better scalability, but it is also very simple.”

Read more of this story at Slashdot.

William Gibson’s ‘Neuromancer’ to Become a Series on Apple TV+

It’s been adapted into a graphic novel, a videogame, a radio play, and an opera, according to Wikipedia — which also describes years of trying to adapt Neuromancer into a movie. “The landmark 1984 cyberpunk novel has been on Hollywood’s wishlist for decades,” writes Gizmodo, “with multiple filmmakers attempting to bring it to the big screen.” (Back in 2010, Slashdot’s CmdrTaco even posted an update with the headline “Neuromancer Movie In Your Future?” with a 2011 story promising the movie deal was “moving forward….”)

But now Deadline reports it’s becoming a 10-episode series on Apple TV+ (co-produced by Apple Studios) starring Callum Turner and Brianna Middleton:
Created for television by Graham Roland and JD Dillard, Neuromancer follows a damaged, top-rung super-hacker named Case (Turner) who is thrust into a web of digital espionage and high stakes crime with his partner Molly (Middleton), a razor-girl assassin with mirrored eyes, aiming to pull a heist on a corporate dynasty with untold secrets.
More from Gizmodo:
“We’re incredibly excited to be bringing this iconic property to Apple TV+,” Roland and Dillard said in a statement. “Since we became friends nearly 10 years ago, we’ve looked for something to team up on, so this collaboration marks a dream come true. Neuromancer has inspired so much of the science fiction that’s come after it and we’re looking forward to bringing television audiences into Gibson’s definitive ‘cyberpunk’ world.”
The novel launched Gibson’s “Sprawl” trilogy of novels (building on the dystopia in his 1982 short story “Burning Chrome”), also resurrecting the “Molly Millions” character from Johnny Mnemonic — an even earlier short story from 1981…

Read more of this story at Slashdot.