SG-WORLD V2: A Planning-First Framework for Multimodal World Simulation

Piyush Thapliyal; Purva Mundada; Mukthikka V; Gurpreet Singh

doi:10.55524/ijirem.2026.13.3.15

Abstract

Current systems that generate content across multiple modes, like video and audio, often face issues such as missing objects, weak connections between meanings, and poor coordination between visual and sound elements. These problems often arise because visual and audio parts are created separately, leading to mismatches between what is shown and what is heard. To tackle these issues, this paper introduces SG-WORLD V2, a framework that focuses on planning first for semantic reasoning and coordination in structured world simulations. Instead of directly creating video or audio, SG-WORLD V2 builds a semantic blueprint first. It translates natural language into machine-readable forms using reasoning based on the Universal Scene Graph (USG), adds a Quantification Rule to keep track of requested items; and uses Deterministic Pre-Temporal Synchronization (DPTS) to align audio and visual events during planning. The system also includes a self-correcting mechanism to check for semantic consistency before generating outputs. The framework was tested with 54 different prompts that cover object arrangements, spatial connections, actions, and synchronized audio-video situations. The results showed strong performance in keeping track of objects, maintaining relationships, and creating logically aligned plans. Overall, SG-WORLD V2 offers a clear and dependable semantic planning method that can support future systems for generating video and audio while improving consistency and alignment between different modes.

Keywords

Audio-Visual Synchronization, Multimodal Generation, Semantic Planning, Universal Scene Graph.

References

Z. Li et al., “Multimodal Video Generation Models with Audio: Present and Future,” 2026. Available from: http://doi.org/10.13140/RG.2.2.26531.31528.
S. Mo, J. Shi, and Y. Tian, “Text-to-Audio Generation Synchronized with Videos,” arXiv:2403.07938, Mar. 2024. Available from: http://doi.org/10.48550/arXiv.2403.07938.
Z. Gao, W. Huang, J. Zhang, A. Kembhavi, and R. Krishna, “Generate Any Scene: Scene Graph Driven Data Synthesis for Visual Generation Training,” arXiv:2412.08221, Mar. 2026. Available from: http://doi.org/10.48550/arXiv.2412.08221.
S. Wu, H. Fei, and T.-S. Chua, “Universal Scene Graph Generation,” arXiv:2503.15005, Mar. 2025. Available from: http://doi.org/10.48550/arXiv.2503.15005.
Y. Lu, L. Zhu, H. Fan, and Y. Yang, “FlowZero: Zero-Shot Text-to-Video Synthesis with LLM-Driven Dynamic Scene Syntax,” arXiv:2311.15813, Nov. 2023. Available from: http://doi.org/10.48550/arXiv.2311.15813.
Y. Li et al., “Anim-Director: A Large Multimodal Model Powered Agent for Controllable Animation Video Generation,” arXiv:2408.09787, Aug. 2024. Available from: http://doi.org/10.48550/arXiv.2408.09787.
A. Fime, S. Mahmud, A. Das, M. S. Islam, and H.-H. Kim, “Automatic Scene Generation: State-of-the-Art Techniques, Models, Datasets, Challenges, and Future Prospects,” arXiv:2410.01816, Sep. 2024. Available from: http://doi.org/10.48550/arXiv.2410.01816.
Y. Li, M. R. Min, D. Shen, D. Carlson, and L. Carin, “Video Generation From Text,” arXiv:1710.00421, Oct. 2017. Available from: http://doi.org/10.48550/arXiv.1710.00421.
X. Chen et al., “SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction,” arXiv:2310.20700, Nov. 2023. Available from: http://doi.org/10.48550/arXiv.2310.20700.
L. Niu, W. Xie, D. Wang, Z. Cao, and X. Liu, “Audio2AB: Audio-driven collaborative generation of virtual character animation,” Virtual Reality & Intelligent Hardware, vol. 6, no. 1, pp. 56–70, Feb. 2024. Available from: http://doi.org/10.1016/j.vrih.2023.08.006.
Y. Huang et al., “Structure-CLIP: Towards Scene Graph Knowledge to Enhance Multi-modal Structured Representations,” arXiv:2305.06152, Dec. 2023. Available from: http://doi.org/10.48550/arXiv.2305.06152.
Tian, “A Large Language Model-Based System for Semantic Understanding and Automated Scene Generation in Animation Scripts,” in Proceedings of the 2nd International Conference on Machine Intelligence and Digital Applications (MIDA ’25), Jul. 2025, pp. 116–120. Available from: https://doi.org/10.1145/3744464.3744483.
“Multimodal and Large Language Model Approaches in Cybersecurity: A Systematic Review,” International Journal of Engineering, Science and Environment (IJESE), vol. 1, no. 1, 2026. Available from: https://ijese.in/journal/article/view/18.
Singh, A. Naaz, A. Syed, and V. Akhila, “AI-Assisted Storytelling: Enhancing Narrative Creation in Digital Media,” Preprints, 2026. Available from: https://doi.org/10.20944/preprints202601.0330.v1.
Singh, L. Qamar, N. V. Volta, A. Velamuri, and A. Khanyile, “Vision–Language Foundation Models and Multimodal Large Language Models: A Comprehensive Survey of Architectures, Benchmarks, and Open Challenges,” Preprints, 2026. Available from: https://doi.org/10.20944/preprints202602.0467.v2.
Singh, “A Review of Multimodal Vision–Language Models: Foundations, Applications, and Future Directions,” Preprints, 2025. Available from: https://doi.org/10.20944/preprints202510.2511.v1.
Singh and T. Banerjee, “AgroMM-GSF++: Confidence-Adaptive Lightweight Multimodal Fusion for Plant Disease Recognition with Repeated Multi-Seed Cross-Validation and External Benchmarking,” Research Square Preprint, Apr. 2026. Available from: https://doi.org/10.21203/rs.3.rs-9556788/v1.
T. Banerjee, Piyush, Mukthikka V et al., “Multimodal and Large Language Model Approaches in Cybersecurity: A Systematic Review,” Research Square Preprint, Apr. 2026. Available from: https://doi.org/10.21203/rs.3.rs-9527568/v1.
T. Banerjee, Piyush NA, and Mukthikka V et al., “Multi-Modal and CNN-Based Approaches in Cybersecurity: A Comprehensive Review,” ScienceOpen Preprints, 2026. Available from: https://doi.org/10.14293/PR2199.003446.v1.
Singh and P. Mundada, “SignFuse: A Proposed Dual-Stream Cross-Modal Framework for Gloss-Free Sign Language Translation with Large Language Models,” Preprints, 2026. Available from: https://doi.org/10.20944/preprints202604.1065.v1.
Singh, S. Singh, N. Rehmani, P. Kumari, and S. Vedha Varshini, “The Role of Data Analytics in Driving Business Innovation and Economic Growth—A Comparative Study Across Industries,” International Journal of Innovative Research in Engineering and Management, vol. 11, no. 4, pp. 33–38, 2024. Available from: https://doi.org/10.55524/ijirem.2024.11.4.5.
G. Singh, T. Banerjee, and N. Ghosh, “Tracing the Evolution of Artificial Intelligence: A Review of Tools, Frameworks, and Technologies (1950–2025),” Preprints, 2025. Available from: https://doi.org/10.20944/preprints202511.0637.v1.

Cites this article as

P. Thapliyal, P. Mundada, M. V, G. Singh, "SG-WORLD V2: A Planning-First Framework for Multimodal World Simulation", International Journal of Innovative Research in Engineering and Management (IJIREM), Vol-13, Issue-3, Page No-125-137, 2026. Available from: https://doi.org/10.55524/ijirem.2026.13.3.15