back

stereo panorama with movement
in one cube face with ffmpeg

The idea is to make a 360° stereo panorama video (or timelapse in this case as video format is restricted to 16x9 format on these cameras) where there is movement in only one part. That's in one cubic face of the six cubic faces that makes the sphere stereo panorama. Using only PTGui and a late version of FFMpeg. Here from still image formats to the final movie.

I take 8x2 still images around and from the final fixed position, an extra sequence of 18x2 images when the dancers walk over the stage.

An advancement would be to populate more cubic faces, up to all, but each separately.

There is a lack of open source stereo programs for Mac OS other than ffmpeg, please correct me. The one-line commands should work on Mac and Linux, and on PC with minor changes.

When I upgrade my computer I would love to learn how to do this in Blackmagic Resolve Studio + KartaVR.

So this is more a proof of concept. Also to show that ffmpeg can be a tool in projects like this.



This is the area where I want to have movement.



And here is a b/w anaglyph cubic stereo face from my test.

Here is the test in flat anaglyph HD format, 9 sec at 2fps. You can see where the dancers gets cut off at the edge of the cubic face.

8 photos x 2 round, no zenith or nadir + 18 x 2 for the movement in the final camera position.

Hardware: ( x 2 when needed )
Sony a 6300, APS-C, 1.5 half-frame.
Nikon AF-S Fisheye-Nikkor 10.5mm f/2.8G ED.
Metabones N/F-E Adapter Nikon G -> Sony E-mount.
Sony RM-VPR1 cables from Nodal Ninja (via Pano Society) with a home made release box - with a switch to allow camera settings to be adjusted when connected.
Cameras in portrait format, slightly toe-in, rotated on the left camera's no-parallax point. Old Manfrotto rotator. Smallrig and other Arca Swiss compatible parts.

Software:
(Lightroom 6.9)
PTGui Pro 12 beta 14, following the Zakato_stereo_webinar with Antonio Victor Garcia-Serrano: https://www.youtube.com/watch?v=bOoHpd9T_e4&feature=youtu.be
Next time I will try Thomas Sharples's multipatch technique.
GNU bash, version 5.1.4(1)-release
FFMpeg 4.3.1, command-line tool
BBEdit 12.6.7 or other text editor
NameChanger or other file renamer
MacOS 10.13, on a very modest MacBook Air i5

It really helps (me) to rename/renumber the photos from the two cameras before combining them to stereo, or reset the numbering in the cameras.

In ffmpeg stereo, there is a frame size limit of ≈16000x16000 px over/under (or 32000x8000 side/side). I tried to work with 16384x16384 and it failed. Jpgs can give corrupt files, tif and png and many other picture and video formats works. – I'm learning ffmpeg as I go along – I found the "untile" filter when writing this (December 2020).

Combining a panorama for the left eye and a panorama for the right eye to stereo:
ffmpeg hstack filter
ffmpeg vstack filter
$ ffmpeg -i left.tif -i right.tif -filter_complex "vstack" above_below.tif

$ ffmpeg -i left.tif -i right.tif -filter_complex "hstack" side_by_side.tif

To desaturate the stereo panorama:
ffmpeg hue filter
$ ffmpeg -i stereo.tif -vf "hue=s=0" stereo_black_white.tif

And to make the anaglyph:
ffmpeg stereo3d filter
$ ffmpeg -i above_below.tif -vf "stereo3d=abl:arcd" abl_anaglyph.tif

$ ffmpeg -i side_by_side.tif -vf "stereo3d=sbsl:arcd" sbsl_anaglyph.tif

The new filter "v360" converts 360° videos/images between various formats. Here I'm converting an equirectangular panorama to a cubemap in 6x1 format (horisontal).
ffmpeg v360 filter
v360 filter, cubemap face names: right 'r', left 'l', up 'u', down 'd', forward 'f', back 'b'. Default value is 'rludfb'. To get the forward cubemap face first or last in the cubemap, set order thus:
$ ffmpeg -i anaglyph.tif -vf 'v360=input=e:output=c6x1:out_forder=fbrlud' cubemap_fbrlud.tif

$ ffmpeg -i anaglyph.tif -vf 'v360=e:c6x1:out_forder=brludf' cubemap_brludf.tif

I've added a set of 6 small (240x240 px) cubic faces to play with - if you have ffmpeg installed and have not used the v360 filter, also a text file with a few commands: zipped faces



Example, forward cube face: 5.

Untile the cubemap to it's six faces:
ffmpeg untile filter
$ ffmpeg -i folder/cubmap.tif -vf untile=6x1 folder/%d.tif

or keep the "brlud" part of the cubemap intact, with cubemaps of 12000x2000 px:
$ ffmpeg -i cubemap_brludf.tif -vf "crop=10000:2000:0:0" cubemap_brlud.tif

To extract the forward cube faces, with cubemaps of 12000x2000 px. (Or use the untile filter.)
ffmpeg crop filter
$ ffmpeg -i cubemap.tif -vf "crop=2000:2000:8000:0" face05f.tif

Convert the other five cube faces separately to video with a duration of 9 sec at 2 fps:
$ ffmpeg -loop 1 -framerate 2 -i 1.tif -c:v libx264 -t 9 -pix_fmt yuv420p 1.mp4

Reassembly of the (video) cube frames.
ffmpeg tile filter
v360 filter names cubemap faces: 'r' right, 'l' left, 'u' up, 'd' down, 'f' forward, 'b' back. Default value is 'rludfb'. Rename each cube frame if needed, right to 1.mp4, left to 2.mp4 etc. Then reassemble them with only 'f', 'forward', '5', having actual movement.
 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
This is not the same order that Adobe, Apple, Hugin, Pano2VR, Panotools, Quake II or RealViz has been using (frblud), WHY?!, so if you have cubemaps from them you have to untile and reassemble them to work with this filter. And we can't set v360 to 'frblud' order either. This 'rludfb' seems to be the 'NVIDIA Texture Tools Exporter' order, any more?
 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
$ ffmpeg -i folder/1.mov -i folder/2.mov -i folder/3.mov -i folder/4.mov -i folder/5.mov -i folder/6.mov -filter_complex 'hstack=inputs=6' folder/cubemap123456.mp4
For some reason I don't get "%d" or "*" selection to work for cube faces in video format. cubemap123456.mp4

ffmpeg's v360 filter can convert the cubemaps back to equirectangular after reassembly:
$ ffmpeg -i folder/cubemap123456.mov -vf v360=c6x1:e folder/equirectangular.mp4

Alternatively make 18 equirectangular pngs, as a sequence of 01.png to 18.png, convert to a movie:
$ ffmpeg -framerate 2 -i folder/%02d.png -vf format=yuv420p -c:v libx264 -preset slow -crf 18 folder/studio0202d.mp4

The movie was 6000x3000 px and I wanted it smaller and in hd format for this page:
$ ffmpeg -framerate 2 -i folder/%02d.png -vf "scale=hd1080" format=yuv420p -c:v libx264 -preset slow -crf 18 folder/studio0202d1.mp4
Which you see above.

I notice that the forward cube face gets placed in the center of the equirectangular file.
To offset the forward cube face to the left in the equirectangular file, in degrees:
$ ffmpeg -i equirectangular01.mp4 -vf 'v360=input=e:output=e:yaw=90' equirectangular01_90.mp4
yaw offset pixels from one side to the other, like offset filter with horizontal wrap around in PhotoShop.

You can find examples of v360 filtering (and much more) in:
http://www.astro-electronic.de/FFmpeg_Book.pdf  by Michael Koch. The commands are in PC/DOS form, so for Mac and Linux I make them one-liners and if they don't work, I enclose the filter part (after the -vf) in double quotes "". That works most of the time. To translate the PC batch files to shell scripts this resource helps: https://tldp.org/LDP/abs/html/dosbatch.html

my very similar: "ffmpeg for mono VRs"



The rig is not optimal when changing memory cards or batteries. But to shoot video they should be positioned side by side like here (not one vertically flipped) or the rolling shutter will give problems, fast movement across the frame in normal horisontal video is bad enough.

back