www.marktechpost.com
Artificial intelligence systems designed for physical settings require more than just perceptual abilitiesthey must also reason about objects, actions, and consequences in dynamic, real-world environments. These systems must understand spatial arrangements, cause-and-effect relationships, and the progression of events over time. In applications like robotics, self-driving vehicles, or assistive technologies, AI must comprehend its surroundings physical constraints and affordances to make intelligent and safe decisions. This fusion of perception with structured reasoning about physical dynamics forms the backbone of Physical AI.A core issue for such systems is their inability to conclude physical environments using integrated visual and contextual information. Although vision-language models have made significant progress, they still struggle to determine whether a task has been completed, what action should follow next, or whether a proposed action is feasible. The gap between perception and decision-making becomes especially critical when AI needs to operate independently and interpret tasks from complex visual scenarios. These systems remain unreliable in high-stakes or fast-changing environments without mechanisms to verify their reasoning.Existing models such as LLaVA, GPT-4o, and Gemini 2.0 Flash are proficient in handling text and visual data but underperform physically grounded reasoning. Tasks like identifying temporal order, spatial continuity, or object permanence are rarely handled effectively. Popular benchmarks often fail to evaluate such scenarios, offering limited insight into a models ability to reason about physical events or agent actions. Moreover, current systems usually rely on textual cues rather than making decisions based on visual evidence, leading to inconsistent or incorrect conclusions when applied to the physical world.Researchers from NVIDIA introduced Cosmos-Reason1, a family of vision-language models developed specifically for reasoning about physical environments. These models were released in two sizes: 8 billion and 56 billion parameters. The models were built with a structured approach that included defining ontologies for physical common sense, constructing specialized training data, and designing a comprehensive suite of evaluation benchmarks. These benchmarks test capabilities such as action prediction, task verification, and judgment of physical feasibility. The research team developed datasets including BridgeData V2, RoboVQA, RoboFail, AgiBot, HoloAssist, and AV to rigorously evaluate the models.Cosmos-Reason1 uses a hybrid Mamba-MLP-Transformer architecture that integrates both vision and language components. The training process was conducted in multiple phases. Initially, a vision encoder and language model were pretrained and fine-tuned using general supervised data. Then, a physical AI-specific supervised fine-tuning (SFT) phase introduced datasets focused on space, time, and object interactions. The final reinforcement learning (RL) phase applied rule-based rewards to improve performance in areas like arrow of time detection, spatial puzzles, and object permanence. The RL setup used a modular framework that leveraged distributed computing to scale training efficiently. The model responses were structured using tags, allowing reward systems to evaluate both correctness and reasoning structure. Each question had up to nine model-generated responses, and RL training continued for 500 iterations using a global batch size of 128 questions.Evaluation of Cosmos-Reason1 showed a substantial performance increase compared to other models. In the physical common sense benchmark, Cosmos-Reason1-56B achieved an average accuracy of 60.2%, outperforming OpenAI o1, which scored 59.9%. The 8B variant also improved, reaching 52.3%. Cosmos-Reason1-56B scored an average of 63.7% for embodied reasoning tasks, up from a 53.5% baseline. Benchmarks like RoboVQA and HoloAssist showed strong gains, with the 56B model scoring 80.0% and 57.8%, respectively. Cosmos-Reason1-8B improved to 68.7% on intuitive physics tasks, showing strong gains in object permanence and spatial puzzle reasoning. However, the model faced challenges on datasets like RoboFail due to a lack of sufficiently diverse training examples.In conclusion, this research introduces a targeted and layered strategy to advance AI systems that reason about physical interactions. The researchers at NVIDIA created a scalable training method combined with a comprehensive evaluation to tackle long-standing gaps in embodied reasoning. Cosmos-Reason1 demonstrates how structured fine-tuning and reinforcement learning can build AI systems more aligned with real-world physical logic and agent behavior.Check outthe Paper and GitHub Page.All credit for this research goes to the researchers of this project. Also,feel free to follow us onTwitterand dont forget to join our85k+ ML SubReddit. NikhilNikhil is an intern consultant at Marktechpost. He is pursuing an integrated dual degree in Materials at the Indian Institute of Technology, Kharagpur. Nikhil is an AI/ML enthusiast who is always researching applications in fields like biomaterials and biomedical science. With a strong background in Material Science, he is exploring new advancements and creating opportunities to contribute.Nikhilhttps://www.marktechpost.com/author/nikhil0980/This AI Paper from UC Berkeley Introduces TULIP: A Unified Contrastive Learning Model for High-Fidelity Vision and Language UnderstandingNikhilhttps://www.marktechpost.com/author/nikhil0980/Meta AI Researchers Introduced SWEET-RL and CollaborativeAgentBench: A Step-Wise Reinforcement Learning Framework to Train Multi-Turn Language Agents for Realistic Human-AI Collaboration TasksNikhilhttps://www.marktechpost.com/author/nikhil0980/OpenAI Introduced Advanced Audio Models gpt-4o-mini-tts, gpt-4o-transcribe, and gpt-4o-mini-transcribe: Enhancing Real-Time Speech Synthesis and Transcription Capabilities for DevelopersNikhilhttps://www.marktechpost.com/author/nikhil0980/How to Use SQL Databases with Python: A Beginner-Friendly Tutorial
