Sojourner Truth, born Isabella Baumfree, (c.1797 1883) was an American abolitionist and activist for African-American civil rights, women's rights, and alcohol temperance. Truth was born into slavery in Swartekill, New York, but escaped with her infant daughter to freedom in 1826. After going to court to recover her son in 1828, she became the first black woman to win such a case against a white man. This cabinet card of Truth was produced around 1864, and is now in the collection of the National Museum of African American History and Culture.Photograph credit: unknown; restored by Adam CuerdenRecently featured: Very Large TelescopeCharles Henry TurnerHamerkopArchiveMore featured pictures

February 5: Constitution Day in Mexico (1917)Painting of the twenty-sixAD62 Pompeii was severely damaged by a strong earthquake, which may have been a precursor to the eruption of Mount Vesuvius that destroyed the town 17 years later.1597 As part of enforcing Toyotomi Hideyoshi's ban on Christianity in Japan, twenty-six Catholics, a mix of European missionaries and Japanese converts, were executed (depicted) near Nagasaki by crucifixion and impalement.1861 In a speech before the U.S. Congress, Representative John Edward Bouligny refused to join his fellow Louisiana congressmen in heeding the state's secession convention and resigning.1967 Cultural Revolution: The January Storm revolt in Shanghai reached its apogee as Maoist rebels proclaimed the establishment of the Shanghai People's Commune, a move the previously supportive Mao Zedong criticized.2000 Second Chechen War: As the Battle of Grozny came to a close, Russian forces summarily executed at least 60 civilians in Grozny's Novye Aldi suburb.Elizabeth Ryan (b.1892)Queen Mary (b.1972)Michalina Wisocka (d.2005)Marisa Del Frate (d.2015)More anniversaries: February 4February 5February 6ArchiveBy emailList of days of the yearAbout

Advanced Micro Devices reported revenue of $7.658 billion for the fourth quarter, up 24% from a year ago.Read More

Electronic Arts reported weak earnings today for the holiday quarter, with results in line with the poor quarter that EA preannounced.Read More

Instead of making Titanfall 3, developer Respawn took the franchise in a completely different direction with Apex Legends. A free-to-play battle royale game that mashes together elements of Overwatch and Fortnite, Apex Legends was a complete surprise when it launched, but it has turned out to be a big hit early on, luring tens of millions of players. Keep up with the game as it continues to develop and grow right here.Today, A minute agoRichard LawlerApex Legends 2 got announced before Titanfall 3?While discussing Q3 2025 earnings results, EA CEO Andrew Wilson laid out the future of Apex Legends for investors. With tens of millions of regular players and a significant update planned after the new Battlefield (due out by next spring) launches, but thats not all:Our expectation is that Apex will be also one of those franchises [that last 10, 20, 30 years], and that some time, on a longer term time horizon, there will be an even bigger more meaningful update to that experience, an Apex 2.0 if you will. This will not be the last incarnation of Apex.Nov 11, 2024Richard LawlerOverwatch 2 and Apex Legends follow Fortnite by going back to the beginning.Last years Fortnite OG event has attracted a new round of free-to-play copycats.Apex Legends responded to revenue that EA execs said fell short of expectations with a Launch Royale revisiting the original map and characters until November 19th.And now, Blizzard announced Overwatch Classic is coming tomorrow with 6v6, no-limits gameplay like Overwatch had at its very beginning.Jul 12, 2024Ash ParrishApex Legends is having a rough weekImage: EA / Respawn EntertainmentApex Legends cant seem to catch a break. The game has been plagued with a pair of high-profile issues for the last few weeks.First up, earlier this week, the hero shooters developer, Respawn Entertainment, announced it was retooling the games battle pass. For season 22 and beyond, the battle pass will now be split into two halves, with 60 levels for each half. Additionally, players will no longer be able to purchase a battle pass with AC or Apexs in-game currency. The decision to move from AC to real world currency is not one that we made lightly, but it does allow us to decrease the price of Premium+ for our community, the announcement read.Read Article >Apr 19, 2024Richard LawlerApex Legends locks things down and brings back Three Strikes.The developers of Apex Legends announced a new collection event, the return of the fast-paced Three Strikes mode that brings bad players (like me) right back into the action, and a new Apex Rumble asynchronous tournament system.But most importantly, TechCrunch reports that the hacker behind last months ALGS incident says the exploit he used to hack streamers mid-game has been patched.Mar 20, 2024Jess WeatherbedApex Legends devs confirm esports hacking incident, respond with layered updatesTwo pro players were forcibly given cheats while livestreaming their finals matches. Image: RespawnApex Legends has postponed the North America finals for its Global Series championship over concerns that hackers have compromised the competitive integrity of the game. Two professional players from major teams were targeted while livestreaming on Sunday by an attack that forcibly applied advantage-granting cheats during competitive gameplay.The cause of these attacks hasnt been officially verified, leading to speculation that the hack may have been delivered via Apex Legends anti-cheat protections. An exchange posted by a cheat monitoring account on X claims that the attacker found an RCE (or remote code execution) exploit to run their code directly on the players computers.Read Article >Feb 13, 2024Richard LawlerOverwatch 2 and Apex Legends hit the reset button.Over the next couple of hours, the two games I play the most are releasing massive updates, with changes to ranked tier systems, character abilities, and either 120Hz refresh rates or new bullet sizes.With games like Baldurs Gate 3 and Spider-Man 2 atop the sales charts, these live service games need something new to keep going and hopefully, with these changes they find it.Apex Legends Breakout season starts at 1PM ET, and Overwatch 2 starts season 9 at 2PM ETImage: EA / BlizzardFeb 5, 2024Richard LawlerThe fifth anniversary of Apex Legends is bringing big changes.The Breakout season of Apex Legends starts Tuesday, February 13th, and after a lot of recent changes that didnt all go over well, the battle royale is undoing some of them while making other, larger shifts detailed in this developer diary video.Theres a new in-match progression system that gives players choices in ways to upgrade their legends abilities, and shields work completely differently. Oh, and after a long wait, 120fps Performance Mode is finally coming for PS5 and Xbox Series consoles.Apex Legends Breakout changes Image: EAOct 27, 2023Richard LawlerCross-progression is finally coming to Apex Legends.Apex Legends has thrived since its surprise launch four years ago, and while it has thrived since then, the fact that players couldnt migrate acquired content and stats across platforms has become ridiculous as competitors like Fortnite, Warzone, and even Overwatch merged accounts across PC, PlayStation, Xbox, and Switch.But now, Apex will finally join them in a phased launch along with rolling out a new season, Ignite, a new legend with a tragic Titanfall-related backstory, and a smaller Storm Point map on October 31st.Aug 3, 2023Richard LawlerApex Legends: Resurrection is another fresh start for EAs battle royaleImage: EAMore than four years after its explosive launch shook up the battle royale market, Apex Legends is looking for yet another fresh start (no, that fresh start doesnt include any updates about cross-progression or 120fps on consoles) when its new season, Resurrection, launches on August 8th at 1PM ET.In the new season, the Apex developers are undertaking a campaign to rework and revisit existing elements, starting with one of the games older characters, Revenant.Read Article >Apr 1, 2023Richard LawlerApril Fools 2023: Apex Legends.Combining last years Nessie-shooting pistol and the Mozambique here! joke thats been present since Apex Legends launched, Respawn has made (what used to be) the games worst gun into an assortment of other weapons.Pinging a door or other weapon turns it into one of several random Mozambiques that have properties of other guns that have been removed from the game, as well as the Nessie variant.Feb 6, 2023Richard LawlerThe new season of Apex Legends wont add a new character, but it does have a new weapon and team deathmatch.Respawn and EAs battle royale shooter celebrates its fourth anniversary with a 16th season, dubbed Revelry. It arrives as its mobile version meets an untimely end, and players ask for changes to keep things fresh.Polygon has more details, including a reclassification of the games characters, a playlist of rotating alternate modes, and bot matches to help new players get their feet wet when it launches on February 14th.Aug 4, 2022Richard LawlerApex Legends new character comes with a cute sidekick and a big sniper rifleApex Legends continues to hold its players attention even after several years on the market, and next week, the battle royale shooter will bring things back to where they started with a new season that revisits the games original map. This year, the game already added two new characters (plus two more in the new mobile version), brought upgrades for next-gen consoles, rotated in a well-received Control game mode, and, according to EAs earnings call this week, notched double-digit income growth again for the last quarter. On the other hand, player complaints over glitches, ranked mode scoring, matchmaking, and practically any other small detail continue to mount.The last season was particularly troubled, with problems that affected one of the games best features its responsive controls and enough bugginess that it even popped up during a professional Apex Legends tournament last month.Read Article >Jul 5, 2022Richard LawlerXbox input lag for Apex Legends is finally fixed, two weeks laterUpdate July 5th, 2:30PM ET: On Tuesday afternoon Respawn and EA released a new patch saying it should address the issue, and after playing a bit I can confirm it has fixed input lag with all the controllers I used on Xbox Series X. Last week, EAs battle royale shooter Apex Legends launched a collection event, which is shorthand for a limited-time setup that is mostly about players buying a bunch of cosmetic items from the in-game store. The only problem is that this time, the massive update has made the game nearly unplayable for many people on consoles.Read Article >May 5, 2022Richard LawlerApex Legends season 13 pushes teamwork with a new kind of hero and ranked mode changesOver the last few seasons, the developers of Apex Legends have worked on opening up the game, both with a massive new map and adding a temporary game mode that swaps out battle royale action for a more traditional team deathmatch. On May 10th, theyre introducing another new legend to the arena with Newcastle and making changes that might help players focus on teamwork.Season 13 of Apex Legends carries the title Saviors, which matches Newcastles kit of abilities that are about keeping his teammates alive longer to help them win fights. While a slew of leaks in the past few months spoiled some of the surprises, seeing him in action is an entirely different thing. He has a passive ability that is unlike any character weve seen before, with the ability to not only shield downed teammates while reviving them, but he can also drag them to safety. The shields power level increases if Newcastle is carrying a higher-tier knockdown shield, which adds another tweak to an often overlooked item.Read Article >Mar 29, 2022Richard LawlerApex Legends next-gen console updates with 4K and HDR launch tomorrowRespawn Entertainment has confirmed that Apex Legends upgrades for the PS5 and Xbox Series X / S consoles will launch tomorrow with the new Warriors collection event. Until now, the battle royale shooter has shared versions across the console generations. But when the update is released usually around 1PM ET console players will finally be able to join those on PC with support for natively rendered 4K (except on the Series S), HDR, better shadow maps, and greater level of detail distances.All versions on consoles are still limited to 60Hz, with 120Hz gameplay listed as an upgrade coming in the future (however, once the PS5 supports 120Hz output, it will be able to force it even on games that dont have support built-in if you want to give it a try). When I talked to the developers before the launch of the current season, Defiance, they said that we would see new next-gen features roll out throughout the year with the new seasons. Other features confirmed in the works are adaptive trigger and haptic support for the PlayStation 5 DualSense controller.Read Article >Feb 4, 2022Richard LawlerApex Legends turns three and prepares to change everything with the launch of DefianceIn 2019, Respawn Entertainment surprised gamers by following up its Titanfall games with something slightly different, instead opting for a take on the (then) new and growing free-to-play battle royale genre with Apex Legends.You know what happened next. After a healthy dose of promoted live streams, the game quickly carved out a unique space and grew rapidly in popularity, racking up 25 million registered players in one week and crossing the 100 million mark in time for its second birthday. Now, Apex Legends is turning three, and in a conversation with The Verge, game director Steven Ferreira said that this year, players should expect to see the title do things that are completely unlike what weve seen so far while still keeping the battle royale as the core of the game.Read Article >Oct 28, 2021Richard LawlerApex Legends: Escape takes the battle royale shooter on vacation at the perfect timeImage: EAIn the years since Respawn and EA launched Apex Legends, the shooter has carved out its place among a crowded field of contenders. Other games have tried to enter the market and failed to make much impact alongside the other battle royale titans like Call of Duty: Warzone, PUBG, and Fortnite. Still, despite a healthy player base and competitive scene, a new update, dubbed Escape, arrives following one of the roughest seasons since the debut, and it might be exactly what the game needs.Last seasons new character, Seer, drew criticism from many players until Respawn nerfed his special abilities. Then that change led to its own controversy, as players accused the developers of intentionally releasing heroes with overpowered abilities to coax more money out of participants in this free game.Read Article >Jul 9, 2019Nick StattApex Legends ranked mode will be the key to keeping it aliveImage: Respawn EntertainmentApex Legends is one week into its second season, which features a drastically changed map, a new character to master, and a pleasantly revamped battle pass with much-needed new rewards. But the most pivotal addition to the battle royale game is its ranked mode. Split into six tiers, this new mode will likely be the lifeblood of Apex Legends going forward, keeping it engaging for competitive players and carving out a community of top-tier streamers and aspiring pros who can form the backbone of a future e-sports scene. The good news is that Apex Legends ranked mode is wonderfully designed. It fixes almost every big complaint Ive had about such modes in other games, giving me a rewarding way to spend my time in a way that never feels like a second job, as so many games-as-a-service titles like Bungies Destiny 2 can feel. Ive only just started playing, but can already see myself spending dozens to hundreds of hours here, so long as Respawn keeps it from growing stale and finds ways to combat the inherent toxicity ranked modes are infamous for. Read Article >May 7, 2019Andrew WebsterEA wants to bring Apex Legends to mobileToday EA released its quarterly earnings, and unsurprisingly Apex Legends was a big hit. According to EA, the Respawn-developed battle royale game is easily the fastest-growing franchise weve ever had. What might be surprising, though, is that EA said its looking to bring the game from PC and console to mobile platforms. We are in advanced negotiations to bring Apex Legends to China and to mobile, the company said during todays call.Unfortunately, thats all the information we have right now. But clearly EA is looking to keep pace with Apexs biggest competitors, Fortnite and PUBG. Both of those games are available on mobile, though they utilize different strategies. Fortnite is essentially the exact same experience across all platforms, including mobile, while PUBG has a separate version called PUBG Mobile. Both games are massive hits; as of last December, the mobile version of PUBG had as many players as all of Fortnite, at around 200 million.Read Article >Apr 25, 2019Andrew WebsterRespawn wont rush Apex Legends updates so that it can avoid crunchImage: EA / Respawn EntertainmentApex Legends was a massive hit out of the gate, but since then, many players have complained about a lack of content updates, especially compared to the constantly changing Fortnite. Today, in a surprising reveal, executive producer Drew McCoy explained that this wont necessarily be changing any time soon.Our long-term goal is to ensure Apex Legends always feels alive and thriving, with a focus on quality of content over novelty or speed of release, McCoy wrote in a blog post. At the same time, we want to maintain our culture as a development team and avoid crunch that can quickly lead to burnout or worse.Read Article >Mar 18, 2019Nick StattThe first Apex Legends battle pass launches tomorrow with new character OctaneImage: Respawn EntertainmentRespawn Entertainments fast-growing battle royale hit Apex Legends is launching its much-anticipated first season tomorrow, the developer announced this morning. The first season, dubbed Wild Frontier, will come with a Fortnite-style battle pass costing $9.50 for three months alongside the games newest character, the legend Octane.Fans of Apex Legends have been anxiously awaiting the battle pass launch since the game first released at the beginning of February. As pioneered first by Fortnite creator Epic Games, a battle pass system lets players pay a small fee, typically around $10, for three months of ongoing in-game rewards and special events.Read Article >Mar 13, 2019Shannon LiaoandNick StattTwitch streamer Ninja reportedly earned $1 million for promoting Apex Legends launchPhoto by Nick Statt / The VergeWhen Electronic Arts battle royale shooter Apex Legends surprise-launched early last month, it seemed like the entire gaming community started paying attention at once. Part of the reason for the immediate surge in interest was that EA paid top Twitch streamers to promote the game. And, in fact, one of those streamers, Fortnite extraordinaire Tyler Ninja Blevins, may have earned as much as $1 million for doing so, according to Reuters.Ninjas live-streaming and social media promotions helped Apex Legends gain traction among his 13.6 million-plus Twitch followers, as well as his Twitter following of over 4 million and Instagram audience of over 13.2 million. Now, an anonymous source tells Reuters that his efforts were reportedly worth the seven-figure sum.Read Article >Mar 10, 2019Andrew WebsterE-sports teams are flocking to Apex LegendsApex Legends is the hottest game of the moment, skyrocketing to 50 million registered players in its first month of existence. Its combination of lightning-quick action and squad-focused gameplay has made it not only a popular game to play, but an exciting one to watch as well. Apex regularly dominates the Twitch charts, often topping mainstay Fortnite. So its no surprise that Apex Legends is primed to become the next big e-sport.Over the last few weeks, established e-sports teams have been slowly assembling Apex Legends squads. The last week, in particular, has been a flurry of announcements. TSM, 100 Thieves, Gen.G, and NRG all added either entire Apex teams or individual players over the last few days. One of the more surprising moves saw venerated e-sports organization Team Liquid move its Call of Duty: Black Ops 4 blackout team over to Apex.Read Article >Feb 27, 2019Andrew WebsterTheres not enough time to play all of these online shootersAnthem. Image: EA/BioWareFor the past few nights, most of my time has been spent exploring the fantastical sci-fi realm of Anthem, fighting off countless alien bad guys and putting far too much energy into making sure my armor is just the right color. During that same period, Ive been getting in regular matches of Apex Legends, Respawns new team-based battle royale thats perfect for people who dont like to talk while they shoot. Meanwhile, Ive also been preparing myself for the impending eighth season of Fornite, wrapping up a final few quests and tracking the earthquake-induced cracks in the landscape.Ive also realized that I dont have time for all of this.Read Article >Feb 18, 2019Andrew WebsterApex Legends is a better battle royale, but Id rather hang out in FortniteEven though Ive been playing Fortnite for the better part of a year, I still get surprised by the butterflies. There I am, running along the edge of a giant crater or sneaking through a dense forest, and some blue, fluttering wings catch my eye. I dont exactly stop and stare thats too dangerous in a battle royale where everyone is out to kill you but its enough to capture my attention for a brief second, before I move on to explore other parts of the island.Theres nothing similar to those charming butterflies in Apex Legends. Respawns Titanfall-themed shooter stormed out of the gates when it launched two weeks ago, racking up 25 million players in seven days and regularly dominating the Twitch charts. Apex isnt the first battle royale game to challenge Fortnite, but none of its contemporaries have had this kind of momentum. Even neon-haired Fortnite star Ninja is playing Respawns game, winning the games first major tournament last week in front of hundreds of thousands of viewers.Read Article >More Stories

Elgato is releasing a new version of its audio stream control software Wave Link that includes new AI-powered voice enhancements for your mic. It also has a new OS-level management system to redirect audio streams to certain apps, split audio to yourself and your audience, add audio effects, and quickly mute channels. The latest version 2.0 update is available to download now and works with Elgato hardware, including the companys Mic and Stream Deck control ecosystem.Elgato and audio machine learning company AIcoustics are including Voice Focus in the Wave Link program, which can remove noise and fix acoustic problems so you can have professional-sounding audio regardless of your room situation. It also supports audio effects, and you can include custom plugins to get your desired sound. However, you do need an Elgato microphone or a Stream Deck Plus to use Wave Link and its low-latency audio mixing magic.Wave Link 2.0 now works more like SteelSeries GG, taking over the entire Windows audio process. It allows you complete control to route audio where you want it to go or group apps into one input channel so certain audio always stays together. Wave Link can create custom Stream Deck control buttons with add to Stream Deck that lets you quickly change the volume for each audio stream you manage. It can even show a volume slider that can span multiple buttons, so you know the current volume levels at a glance. It also works in conjunction with Stream Deck foot pads and the dials of the Stream Deck Plus.

Cloud AI is Rigged Against Startups, and DeepSeek is the Warning Shot 0 like February 4, 2025Share this postAuthor(s): Krishna Chaitanya Chavati Originally published on Towards AI. Meet Alex, a startup entrepreneur pursuing AI-driven efficiency only to encounter rising costs, hidden fees, and unsolvable trade-offsThis member-only story is on us. Upgrade to access all of Medium.Source: Author generated using whiskTrying to start an AI company? Its a mess. On the one hand, theres cloud AI, which is polished and simple to use but way too expensive. It costs more to grow because of API fees, hidden costs, and being locked into one vendor. Open-source AI, on the other hand, is cheap and flexible, but when you try to use it, you find that it needs weeks of fine-tuning, expensive infrastructure, and still isnt as good as paid models.Those were the only options for a long time. Pay or have a hard time. But things are beginning to change. Models like DeepSeek, Qwen and smaller ones that have been simplified are making things better. Not only are they cheap, but they can also compete with well-known closed-source AI while using fewer resources.But how does this actually play out? Thats where Alex comes in. Hes a startup founder who thinks AI will help him cut costs and scale.Alex had a plan. A clear, data-driven, scalable plan. The kind that convinces company founders they are about to change the world.And AI was the solution.Why pay humans when Read the full blog for free on Medium.Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming asponsor. Published via Towards AITowards AI - Medium Share this post

Author(s): Kailash Thiyagarajan Originally published on Towards AI. Accelerating AI: A Deep Dive into Flash Attention and Its ImpactsImage Generated by AuthorIntroductionTransformers, introduced in the groundbreaking paper Attention Is All You Need, have revolutionized artificial intelligence, particularly in natural language processing and image classification. At the core of this success is the attention mechanism, which enables models to dynamically focus on relevant parts of the input. However, as transformers grow larger and deeper, the attention mechanism faces significant computational bottlenecks, especially with long input sequences.ProblemThe self-attention module in transformers has a time and memory complexity that scales quadratically with sequence length, making it challenging to handle long contexts. While methods like sparse and low-rank approximations aim to reduce computational costs, they often overlook memory access overheads, limiting practical speedups. As modern GPUs have advanced in compute speed more than memory speed, memory access remains a critical bottleneck. IO-aware algorithms have optimized memory-bound tasks in other fields, but deep learning frameworks like PyTorch and TensorFlow lack fine-grained memory control.OverviewIn this article, we will explore the FlashAttention mechanism and its approach to addressing these challenges. We will examine GPU requirements and demonstrate its implementation with a short code example.Key-Value Attention: The Backbone of TransformersHaving outlined the challenges faced by traditional attention mechanisms, we now turn our focus to the core component that underpins these models: key-value attention. This mechanism is crucial for enabling transformers to efficiently process and prioritize information within input sequences. Lets delve into how key-value attention operatesThe process involves three matrices:Query (Q): What were looking for.Key (K): Where to find it.Value (V): The information we care about.For simplicity, The attention mechanism can be broken down into three steps:Compute the similarity between queries and keys to generate a score matrix (S).Apply the softmax function to turn scores into probabilities.Multiply the probabilities with the values to get the final output.For a detailed explanation, refer to my previous article on Improving LLM Efficiency. Each step involves reading and writing data to high-bandwidth memory (HBM), which can slow down the attention process.Image Source: HuggingFaceThe GPU Memory Pyramid: A Balancing ActA GPU has three main types of memory, and each has different strengths and weaknesses. Think about them as layers in a pyramid, with very marked trade-offs between speed and capacity:Image Source: Flash Attention arxivImage Source: AuthorThe faster the memory, the smaller it gets, creating a classic computing trade-off. While SRAM is ideal for speed, its limited capacity means we often lean on HBM for larger tasks.The Problem with Moving Everything to SRAMWhy not simply perform all these operations in SRAM, the fastest memory layer? While it sounds like a great idea in theory, there are some significant hurdles:Size Limitations:The score matrix S alone can be massive, especially for long sequences or high-dimensional embeddings. A single computation could easily exceed the 20 MB capacity of SRAM.Complexity of Access:Even if we could split data into smaller chunks to fit into SRAM, the frequent movement of these chunks in and out of SRAM would lead to inefficiencies, defeating the purpose of using fast memory.Energy Costs:SRAM is designed for speed, not for handling the repeated, large-scale memory operations required by attention mechanisms. Constantly managing this flow would drain computational resources.As a result, we rely on HBM for these operations, despite its slower speed and higher latency compared to SRAM.Enter Flash Attention: A Smarter Way to WorkFlash Attention changes the game by rethinking how attention computations are performed. Instead of relying on HBM for storing large intermediate results, Flash Attention cleverly restructures the process to make the most of SRAMs speed.Heres how it works:Tiling the Computation:Tiling is a technique used to optimize matrix multiplication by breaking down large matrices into smaller sub-matrices or tiles. This approach enhances performance by improving cache usage and reducing memory bandwidth requirements.Image Source : Tiled Matrix MultiplicationSteps in TilingDivide Matrices into Tiles: The large matrices (Share A and Share B) are divided into smaller blocks or tiles. In this example, each matrix is divided into four smaller 22 tiles.Multiply Tiles: Each tile from Share A is multiplied with the corresponding tile from Share B. This multiplication is done independently for each pair of tiles, resulting in a temporary result (Temp).Accumulate Results: The temporary results from each tile multiplication are accumulated into the final result matrix. This is done by adding the Temp results to the corresponding positions in the Result matrix.Repeat for All Tiles:The process is repeated for all combinations of tiles until the entire matrix multiplication is complete.In Simple terms, Imagine you have two big grids of numbers (Share A and Share B), and you want to multiply them to get a new grid (Result). Doing this all at once can be slow and use a lot of memory.Tiling is like cutting these big grids into smaller squares (tiles). You multiply each small square from Share A with a matching square from Share B to get a small result (Temp). Then, you add up all these small results to get your final big grid (Result).This method is faster because:It works with small pieces at a time, which fits better in computers fast memory.It can do many small multiplications at the same time if you have multiple processors.Overall, tiling makes the multiplication of large matrices more efficient by breaking the task into smaller, more manageable pieces.2. Online Softmax:The term online in online softmax refers to the process of computing the softmax in a streaming or incremental manner, rather than computing it all at once. This is particularly useful for handling large sequences that might not fit into memory if processed in a single batch.Chunking: The input sequence is divided into smaller chunks that can be processed independently. This reduces memory usage because you only need to keep a small part of the sequence in memory at any given time.Incremental Computation: As each chunk is processed, the softmax is computed for that chunk, and intermediate results are stored. This allows the algorithm to build up the final result incrementally.Numerical Stability: Online softmax can be designed to maintain numerical stability by carefully managing the range of values during the computation. This is crucial because exponentiating large numbers can lead to overflow, and small numbers can lead to underflow.Efficiency: By processing data in chunks and only keeping necessary intermediate results, online softmax can significantly reduce the computational overhead and memory footprint, making it suitable for large-scale applications.In Simple terms, Imagine you have a long list of numbers, and you want to convert them into a list of probabilities. Normally, youd take all the numbers at once, do the math (exponentiation and division), and get your probabilities. But if your list is really long, this can be slow and use a lot of memory.Online softmax is like breaking that long list into smaller, manageable pieces. You do the math on each piece one at a time, and then combine the results. This way, you never have to deal with the whole list at once, which saves time and memory.In the context of FlashAttention, this means you can handle really long sequences of data more efficiently, which is great for tasks like natural language processing where you often deal with long texts.3. Weighted Sum of Values:Apply Attention Weights: Use the probabilities from the online softmax to weight the corresponding value vectors.Sum Weighted Values:For each query, compute the weighted sum of the value vectors. This gives the final attention output for each query.In Simple terms, Think of it like using the probabilities to decide how much of each piece of information (value vectors) to include in the final answer. You combine these weighted pieces to get the result for each part of your input.The result? Faster computations, lower energy costs, and the ability to handle larger models without hitting memory bottlenecks. Putting it all together , here is the representation from the original paperImage Source: Flash Attention arxivWith a clear understanding of Flash Attention, lets now take a closer look at its next evolution: Flash Attention v2.Diving into Flash Attention v2:Flash Attention v2 is an improved version of the original Flash Attention algorithm, designed to further optimize the memory and computational efficiency of transformer models. It introduces advanced techniques for multi-query and grouped attention, making it suitable for both inference and training at scale. By efficiently leveraging SRAM through better tiling and streamlining operations, Flash Attention v2 minimizes memory bottlenecks and improves throughput, especially for large models.Key GPU Requirements of Flash AttentionTensor Core Support: Flash Attention heavily relies on Tensor Cores to perform efficient mixed-precision computations such as FP16 or BF16. Tensor Cores were introduced in the NVIDIA Volta architecture (V100) and have been improved in subsequent generations.Warp-Level Primitives: Flash Attention leverages warp-level parallelism in GPUs, which is optimized in NVIDIA GPUs starting from Turing (T4) and beyond.Image: AuthorBF16 is Generally Optimized for Training/Inference:Larger Dynamic Range: Matches FP32 for handling extreme values, crucial for stable computations in attention mechanisms.Efficiency Without Accuracy Loss: Reduces precision minimally while maintaining model accuracy, enabling faster computations.Hardware Support: Modern GPUs (e.g., NVIDIA A100) are optimized for BF16, enhancing throughput for training and inference.Image FP16 vs BF16FP16 vs. BF16 RepresentationFP16 (16-bit Floating Point): Uses 1 bit for the sign, 5 bits for the exponent, and 10 bits for the mantissa. It has a smaller range and precision but is efficient for neural network computations.BF16 (Brain Floating Point): Uses the same 8-bit exponent as FP32 but reduces the mantissa to 7 bits. It provides a larger range and better compatibility with FP32 while maintaining efficiency.Example: Evaluating Flash Attention with Metas Llama ModelNow that weve covered the fundamentals of Flash Attention and its evolution to Flash Attention v2, lets shift gears to see it in action. To truly understand its performance benefits, well walk through an example using the Meta-LLaMA model on the Orca Math Word Problems dataset. This practical demonstration will highlight how Flash Attention v2 improves efficiency and scalability, especially when leveraging mixed precision (FP16/BF16) for inference. So, lets dive into the code to explore this in more detail.Follow this with the code block for seamless engagement.The dataset is limited to 10,000 entries and pre-trained model for faster experimentation. The evaluation focuses on runtime performance for generating text outputs.Heres the code:The performance gains from using FlashAttention with BF32 compared to other mechanisms are more noticeable with larger model sizes and higher batch sizes. These conditions allow the optimizations in memory access and computation to be fully leveraged. Additionally, factors such as sequence length, input data characteristics, and hardware capabilities play a crucial role in realizing these benefits. By optimizing these parameters, you can better exploit the efficiencies offered by FlashAttention in large-scale applications.I hope you found this article useful, and if you did, consider giving claps. #llm #flash-attn #optimization #kv-cacheReferences:FlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessTransformers are slow and memory-hungry on long sequences, since the time and memory complexity of self-attention arearxiv.orgJoin thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming asponsor. Published via Towards AI

If you're searching for your next new game to play, Humble Choice February is now live, and this is the perfect bundle to discover something new! This month, Immortals of Aveum headlines the bundle, with seven other games like Total War: Pharoah Dynasties, Fabledom, and more included. Instead of paying hundreds of dollars to grab these games individually, you can get all eight games for just $11.99 at Humble Bundle for this month only.Humble Choice Games for February 2025Humble Choice - February 2025See it at Humble ChoiceImmortals of AveumTrepang2Total War: Pharoah DynastiesFabledomGriftlandsTales & TacticsNaheulbeuk's Dungeon MasterMy Little UniverseEach month, Humble Choice curates a selection of games for members to redeem. These games are sent as Steam codes, which are yours to keep forever. You can even send them to a friend if you already have the game or want to gift one! For February, Immortals of Aveum is well worth the price of this package alone, receiving an 8/10 in our review:"Immortals of Aveum is an impressively confident first-person shooter that successfully trades muzzles and magazines for mages and magic. Its fast-paced, spellcasting combat is both satisfying to master and spectacular to look at, and it features a lengthy and hearty campaign packed with secrets to keep us going well after the story has wrapped. As someone who desperately hopes that unapologetically single-player shooters will live forever, these Immortals have done a great job of making sure theyre definitely not dead yet."PlayIn addition to these games, you can also score up to 20% off select games on the Humble Store. The longer that you have an active subscription, the more rewards you can unlock. 5% of each Humble Choice purchase also supports a charity that rotates monthly, with February's supporting United Negro College Fund, an organization that works to increase the total annual number of African American college graduates. Don't miss out on this month's Humble Choice selection!

The latest Capcom Spotlight and the Monster Hunter Wilds Showcase have arrived and they revealed a ton of new details for a wide array of Capcom games. Alongside a new story trailer and Open Beta 2 details for Monster Hunter Wilds, we were also treated to more details on Onimusha: Way of the Sword, an Onimusha 2: Samurai's Destiny remaster, a release date for Capcom Fighting Collection 2, and much more.Onimusha: Way of the Sword Gets a Bunch of New DetailsPlayOnimusha: Way of the Sword is currently in development for a 2026 release date and Capcom gave us a few more exciting details on what fans can expect from the franchise's return. When creating this new entry, the team is focusing on three core aspects, including creating compelling characters, a new protagonist, and engaging enemies. They are doing all they can to recreate the setting of Kyoto, which is "brimming with historical, real-life locations." Lastly, the team wants to create the "ultimate sword fighting action" game with the "visceral satisfaction of slicing through enemies."While we didn't learn much about this new protagonist, it was confirmed Onimusha: Way of the Sword will be set in the Edo Period and will see us fighting off against evil creatures known as Genma. Due to a "twist of fate," our protagonist will wield an Oni Gauntlet and use it to dispatch any foes in his path and feed it souls in return. The game will be challenging, but the team doesn't want it to be "impossibly difficult." Instead, it wants it to be a game that action fans of all skill levels can enjoy to its fullest.Onimusha 2: Samurai's Destiny Is Getting a Remastered Version in 2025Play2002's Onimusha: Samurai's Destiny will be getting a remastered version in 2025, and Capcom hopes it will help with the wait for Onimusha: Way of the Sword in 2026.Monster Hunter Wilds Open Beta 2 to Feature Flagship Monster ArkveldPlayMonster Hunter Wilds' second Open Beta test is almost here and Capcom has pulled back the curtain on what will be different from the first beta. The biggest addition will be that players will be able to take on the new flagship monster, Arkveld, in an advanced quest that will test the skills of all who take it on.There will also be a Gypceros Hunt, a training area to explore, and some online features like Private Lobbies and Online Single Player mode for players to try out. The Private Lobbies won't appear in searches and are perfect for those who want to just play with some friends. Online Single Player will let players jump in without joining a lobby. While they won't see other players in their Base Camp, they can use and S.O.S. Flare to switch to multiplayer when they need a little help.Players will be able to transfer their data from the first Open Beta test for this new one, and also returning will be Character Creator, Story Trial, and the Doshaguma hunt. Monster Hunter Wilds will be released on February 28 and the second Open Beta test, which will feature Cross-Play will take place during the following times;Thursday, February 6 at 7pm PT to Sunday, February 9 at 6:59pm PTThursday, February 13 at 7pm PT to Sunday, February 16 at 6:59pm PTPlayPlayers will be able to transfer their character data to the full game, but no game progression will carry over. Those who try it out will earn a special Pendant as a bonus for participating.Monster Hunter Wilds Reveals Iceshard Cliffs and New Enemies Just Waiting to Be FoughtThe Monster Hunter Wilds Showcase gave fans a look at a new story trailer for the game set to be released on February 28, and it took us all the to frozen locale known as the Iceshard Cliffs. Inhabiting this snowy place will be the quiryky Wudwud known as Rove, the Hirabami - Leviathan, Nerscylla - Temnoceran, and the ferocious Gore Magala. And yes, we do get to learn more about the flagship monster known as the Arkveld.Capcom Fighting Collection 2 Launches on May 16PlayCapcom Fighting Collection 2 will officially launch on May 16, 2025, and will bring with it a collection of fighting games that include Capcom vs. SNK Millennium Fight 2000 Pro, Capcom vs. SNK 2: Mark of the Millennium 2001, Capcom Fighting Evolution, Street Fighter Alpha 3 Upper, Power Stone, Power Stone 2, Project Justice, and Plasma Sword: Night of Bilstein.Street Fighter 6 Adds Fatal Fury's Mai on February 5PlayStreet Fighter 6 is set to add Fatal Fury's Mai to its roster on February 5, and she will be the second-last character Year 2 character following M. Bison and Terry. Elena will be the final character added in this group, and more details on her will be revealed in the future.Have a tip for us? Want to discuss a possible story? Please send an email to newstips@ign.com.Adam Bankhurst is a writer for IGN. You can follow him on X/Twitter @AdamBankhurst and on TikTok.