Figure 1. Our MaGic-NeRF achieves ultra-high compression ratios, storing a single model in just 0.04 MB. We transform the NeRF into conditionally generative neural planes, employing an LMM to decompose the condition into compact semantic representations.

Abstract

Neural Radiance Fields (NeRFs) offer realistic scene representations but require substantial storage, limiting their deployment in real-world systems. Existing NeRF compression methods inherently face a trade-off between compression efficiency and reconstruction fidelity, and cannot fully exploit the semantic priors naturally present in scenes. We introduce MaGic-NeRF, a novel framework that achieves extremely high compression ratios by leveraging robust flow models and multimodal semantic priors. To reduce storage overhead, we distill a trained NeRF into neural planes and encode them with a flow model conditioned on a rendered image. The conditioning image is hierarchically reasoned by large multimodal models (LMMs), enabling semantic-aware decomposition and intuitively guided reconstruction.

We also introduce 3DFRONT-NeRF, an indoor dataset that combines complex scene-level geometry with controllable object-level semantics, featuring realistic materials and physically-based lighting in complex room layouts. The benchmark supports novel view synthesis, 3D reconstruction, semantic reasoning, and layout generation. Extensive experiments demonstrate that our approach achieves compression ratios of up to 1900× and 4400× relative to TensoRF on our 3DFRONT-NeRF and LLFF, respectively, and 125× relative to the original MLP-based NeRF on object-level datasets. We will release the code and dataset.

Pipeline

Figure 2. Overview of our MaGic-NeRF framework. (a) A pretrained TensoRF is distilled into a triplane, which is then embedded using HDT-Flow conditioned on a rendered image. (b) The conditioning image is decomposed into a compact 2D semantic prior consisting of an ultra-low-resolution image, text, and a mask. (c) The stored semantic prior is passed through a step-wise generation process to recover the condition image. (d) The recovered image is fed to the inverse flow to reconstruct a high-fidelity triplane for direct rendering. Dashed boxes indicate trainable modules.

Results

Quantitative Comparisons on 3DFRONT-NeRF and LLFF

Quantitative comparisons on 3DFRONT-NeRF and LLFF datasets. We mainly evaluate against TensoRF and other TensoRF-based compression methods. Best results are marked in bold. Size is measured in MB.

Quantitative Comparisons on NeRF-Synthetic, Synthetic-NSVF, and Tanks & Temples

Quantitative comparisons on the NeRF-Synthetic, Synthetic-NSVF, and Tanks & Temples datasets. We evaluate against conventional NeRF models and methods tailored for NeRF compression. Size is measured in MB.

Acknowledgements

This work was supported by the Australian Research Council (ARC) under discovery grant project # 240101926. Professor Ajmal Mian was supported by ARC Fellowship Award funded by Australian Government under Project FT210100268.