Rendering issues during movement

[smart4654154]

your work is good,thank you
In the NS viewer, when the user's perspective moves from point A to point B, the rendering outcome transitions from clear (at position A) to blurry and then back to clear (at position B). As demonstrated:

The rendering resolution originates from the render_state_machine.py file，calculate_image_res function.
For instance, the resolution at position A is 1000x1000 which is very clear, during the transitional process, the resolution drops to 100x100 which appears blurry, and upon reaching position B, the resolution returns to 1000x1000, thus restoring clarity.

I have 2 question:

1. the image of 100x100 pixels cannot fill the entire screen, yet the rendering result does cover the entire screen. I suspect that the image of 100x100 pixels was stretched to fit. Is this correct? Could you tell me where the specific processing codelocated? If my guess is incorrect, could you please explain the correct processing code?
2. During the transition from position A to position B, there is some lag, and I suspect that a delay function has been set for the movement process. Is this right?
   I am very grateful for your assistance.

[brentyi]

Hello!

the image of 100x100 pixels cannot fill the entire screen, yet the rendering result does cover the entire screen. I suspect that the image of 100x100 pixels was stretched to fit. Is this correct? Could you tell me where the specific processing codelocated? If my guess is incorrect, could you please explain the correct processing code?

Yeah, we just stretch the image. The image is rendered here:

viser/src/viser/client/src/App.tsx

Lines 516 to 624 in a52d9a6

	/* Background image with support for depth compositing. */
	function BackgroundImage() {
	// Create a fragment shader that composites depth using depth and rgb
	const vertShader = `
	varying vec2 vUv;
	void main() {
	vUv = uv;
	gl_Position = projectionMatrix * modelViewMatrix * vec4(position, 1.0);
	}
	`.trim();
	const fragShader = `
	#include <packing>
	precision highp float;
	precision highp int;
	varying vec2 vUv;
	uniform sampler2D colorMap;
	uniform sampler2D depthMap;
	uniform float cameraNear;
	uniform float cameraFar;
	uniform bool enabled;
	uniform bool hasDepth;
	float readDepth(sampler2D depthMap, vec2 coord) {
	vec4 rgbPacked = texture(depthMap, coord);
	// For the k-th channel, coefficients are calculated as: 255 * 1e-5 * 2^(8 * k).
	// Note that: [0, 255] channels are scaled to [0, 1], and we multiply by 1e5 on the server side.
	float depth = rgbPacked.r * 0.00255 + rgbPacked.g * 0.6528 + rgbPacked.b * 167.1168;
	return depth;
	}
	void main() {
	if (!enabled) {
	// discard the pixel if we're not enabled
	discard;
	}
	vec4 color = texture(colorMap, vUv);
	gl_FragColor = vec4(color.rgb, 1.0);
	float bufDepth;
	if(hasDepth){
	float depth = readDepth(depthMap, vUv);
	bufDepth = viewZToPerspectiveDepth(-depth, cameraNear, cameraFar);
	} else {
	// If no depth enabled, set depth to 1.0 (infinity) to treat it like a background image.
	bufDepth = 1.0;
	}
	gl_FragDepth = bufDepth;
	}`.trim();
	// initialize the rgb texture with all white and depth at infinity
	const backgroundMaterial = new THREE.ShaderMaterial({
	fragmentShader: fragShader,
	vertexShader: vertShader,
	uniforms: {
	enabled: { value: false },
	depthMap: { value: null },
	colorMap: { value: null },
	cameraNear: { value: null },
	cameraFar: { value: null },
	hasDepth: { value: false },
	},
	});
	const { backgroundMaterialRef } = React.useContext(ViewerContext)!;
	backgroundMaterialRef.current = backgroundMaterial;
	const backgroundMesh = React.useRef<THREE.Mesh>(null);
	useFrame(({ camera }) => {
	// Logic ahead relies on perspective camera assumption.
	if (!(camera instanceof THREE.PerspectiveCamera)) {
	console.error(
	"Camera is not a perspective camera, cannot render background image",
	);
	return;
	}
	// Update the position of the mesh based on the camera position.
	const lookdir = camera.getWorldDirection(new THREE.Vector3());
	backgroundMesh.current!.position.set(
	camera.position.x,
	camera.position.y,
	camera.position.z,
	);
	backgroundMesh.current!.position.addScaledVector(lookdir, 1.0);
	backgroundMesh.current!.quaternion.copy(camera.quaternion);
	// Resize the mesh based on focal length.
	const f = camera.getFocalLength();
	backgroundMesh.current!.scale.set(
	camera.getFilmWidth() / f,
	camera.getFilmHeight() / f,
	1.0,
	);
	// Set near/far uniforms.
	backgroundMaterial.uniforms.cameraNear.value = camera.near;
	backgroundMaterial.uniforms.cameraFar.value = camera.far;
	});
	return (
	<mesh
	ref={backgroundMesh}
	material={backgroundMaterial}
	matrixWorldAutoUpdate={false}
	>
	<planeGeometry attach="geometry" args={[1, 1]} />
	</mesh>
	);
	}

Where we create a plane whose bounds are calculated to fill the screen.

During the transition from position A to position B, there is some lag, and I suspect that a delay function has been set for the movement process. Is this right?

I'm not sure about this one, but it could just be rendering or communication latency.

[smart4654154]

thank you.
for the question1, Which function determines the resolution? For example, 100 * 100.
I think this is a python function in render_state_machine.py .
def _calculate_image_res(self, aspect_ratio: float) -> Tuple[int, int]:
"""Calculate the maximum image height that can be rendered in the time budget

    Args:
        apect_ratio: the aspect ratio of the current view
    Returns:
        image_height: the maximum image height that can be rendered in the time budget
        image_width: the maximum image width that can be rendered in the time budget
    """
    max_res = self.viewer.control_panel.max_res
    if self.state == "high":
        # high res is always static
        image_height = max_res
        image_width = int(image_height * aspect_ratio)
        if image_width > max_res:
            image_width = max_res
            image_height = int(image_width / aspect_ratio)
    elif self.state in ("low_move", "low_static"):
        if writer.is_initialized() and EventName.VIS_RAYS_PER_SEC.value in GLOBAL_BUFFER["events"]:
            vis_rays_per_sec = GLOBAL_BUFFER["events"][EventName.VIS_RAYS_PER_SEC.value]["avg"]
        else:
            vis_rays_per_sec = 100000
        target_fps = self.target_fps
        num_vis_rays = vis_rays_per_sec / target_fps
        image_height = (num_vis_rays / aspect_ratio) ** 0.5
        image_height = int(round(image_height, -1))
        image_height = max(min(max_res, image_height), 30)
        image_width = int(image_height * aspect_ratio)
        if image_width > max_res:
            image_width = max_res
            image_height = int(image_width / aspect_ratio)
    else:
        raise ValueError(f"Invalid state: {self.state}")

    return image_height, image_width

Am I right? Is there any other function in Viser that determines resolution?
thank you