NeRFtoGSandBack #3150
NeRFtoGSandBack #3150
Conversation
|
Can you update your branch with latest main? |
|
|
||
| This is the implementation of [From NeRFs to Gaussian Splats, and Back](https://arxiv.org/abs/2405.09717); An efficient procedure to convert back and forth between NeRF and GS, and thereby get the best of both approaches. New dataset in the paper can be downloaded from [this google drive link](https://drive.google.com/drive/folders/1xvbONL4EVgHxaHMsV101455l_jNgyaUM?usp=sharing). | ||
|
|
||
| <video id="teaser" muted autoplay playsinline loop controls width="100%"> |
There was a problem hiding this comment.
@Siming-He, really interesting project. I don't fully understand the demo video though, can you briefly describe what the left and right columns depict, mainly the right column. It switches from nerf-sh to nerf-gs (what exactly does that mean)? SH is the nerf variant and then you convert it to the Nerf-GS variant or do you train again to generate the Nerf-GS variant?
There was a problem hiding this comment.
Hi, I updated my branch with the latest main. For the video, NeRF-SH is the nerf variant that predicts spherical harmonic coefficients. I directly convert it to NeRF-GS with conversion time 7s (which is Gaussian splats extracted from NeRF-GS). Then I finetune NeRF-GS on the training data (for 3s, 33s, and 153s) to get better NeRF-GS. Thank you for the question! I'll add the explanations into the video.
|
Interesting project |
Thank you! In the video, 7s is the time to export NeRFGS directly from NeRF-SH. Then, we finetune the NeRFGS for 3s, 33s and 153s on training data for better results. |
|
Any reviewers yet? I am not the developers of Nerfstudio |
|
@Siming-He, just one question. Does it make sense to report the "training for 3s, 33s, or 153s" since doesn't the finetuning and amount of seconds highly depend on your computer specs? I wonder if its more meaningful to report how many iterations you retrain or how many images are seen during re-training (since for example GS is trained per-image at a training iteration, so num iterations is equivalent to num of images seen). |
You are right. In the report, we mentioned that training 100, 1000, 5000 iterations is about 3s, 33s, or 153s on GeForce RTX 4090. I added the number of training iterations in the video. Thank you for pointing this out! |
|
Great, thanks @Siming-He for answering my questions. LGTM, if you update your branch I can merge it! |
It contains the updates to include NeRFtoGSandBack (https://arxiv.org/abs/2405.09717) in nerfstudio. Thanks for your time in advance!