Using Orbbec Astra for interactive floors

Let's make your motion-interactive playground!

In this guide we describe one of the way to start using Motioncube on the interactive floor. You will see how to run the free Motioncube 3D Fish app on the floor surface using a depth camera and a projector.

The goal for us is to get by you a similar result, as shown on the video below.

Interactive Motioncube 3D Fish projected on the floor - experiments with depth cameras

What you will need:

• a computer with Windows 10 or later (see minimum requirements),
• one of the supported depth cameras: Orbbec Astra (e.g. Pro Plus or just Astra) (read more about Motioncube support for depth cameras),
• a projector (not required to do a quick test of the application),
• Motioncube Player in latest version,
• an account in the Motioncube Store.

1. Prepare a computer with Windows 10 or later and connect with your depth camera

We recommend using Orbbec Astra because, in our tests, it gives better results than Kinect 2 and Intel Realsense. The Orbecc Astra cameras are also under active development and available at a lower price than Kinect and Realsesne. There will be a dedicated post about depth cameras presenting our experiments and observations.

Mount the camera next to the projector. Both devices shall face the same surface where you will be testing. The projector is not required for the first test, as you can observe the experiment on your computer screen.

Use the camera manufacturer's software to check if the camera is working - whether it gives a stable signal without interference. If the camera is not directly connected to a computer, you may use an active cable or USB HUB with the additional power supply. This will provide enough power for the camera.

Set the camera so that it can see the entire image displayed by the projector (it can see more - we will calibrate it later using Motioncube). If you do not have a projector, you can perform a simplified motion test (e.g. by waving your hand in front of the camera) to initially assess whether your move is detected and whether the Motioncube 3D Fish app meets your expectations.

2. Download the latest version of Motioncube Player and test your camera

A Motioncube account is not required at this stage. First, check if the Motioncube handles your camera and working environment correctly.

Download the Motioncube installer.

1. Run the installer and go through the installation process.
2. Launch Motioncube using the shortcut found in the Windows Start Menu.
3. Select a language and accept the license. You may skip logging in this time.
4. Select Projection device, then Interactive Floor, and check Motion interaction. These options will be enough to get you started. You'll be able to change it later.

If a popup appears with the Interactive Projection Wizard, you can skip it because it is dedicated for Motioncube IR Kit and does not yet support depth cameras.

Now you will integrate your camera.

1. Go to Menu > Settings > Advanced and enter the service mode.
2. Then click on the Camera tile and expand the list in the Select device section.
3. Click on Configure supported devices.
4. In the list, select the exact type of the camera you have connected to the computer (you can choose several options if more cameras are connected). For the Orbbec Astra select "Orbbec(R) Astra Depth stream" and close this window.

After clicking OK, Motioncube will refresh the camera list. See if the depth camera you have connected is on the list. If Motioncube does not see your camera, check again that the camera view is visible in the hardware manufacturer's software without interruption.

Select the depth camera you want to test from the list - they have "(Native support)" added. Wait for the camera to initialize and see if the image preview appears. For the Orbbec Astra select the "Orbbec Astra Depth Stream (Native support)".

If you can see the camera preview, you can perform additional experiments.

(a) You can experiment with the range the camera sees. Select Change camera filters and use the sliders to limit the field of view to cut out the visibility of the floor (it will change from grey to black).

As you can see in the preview, for a person sitting on the floor, removing the visibility of the floor also results in partial loss of important information, i.e. parts of that person's legs.

So let's reset these values to the initial state so that the full range of the image is observed at this stage - Motioncube, as we will see later, can handle this.

(b) It is now worth slightly narrowing down the area that will be taken into account by the image processing algorithm. This is because the edges of the depth image are not stable and generate excessive noise.

Select the Color mode option for the time of manual calibration in the Depth Stream properties box. In this mode, you can see much more details, since it is an RGB image from the camera superimposed on the depth image. The operation of calibrating the camera view can be performed on the computer screen. Uncheck the Color mode option.

(c) Go to Full screen mode (in the Projection area section) to test the motion detection. Use the [Delete] key to preview the motion detection.

Step off the interaction surface and remove other elements so the camera only observes the empty floor. Use the up/down arrow keys to set the motion detection sensitivity to the lowest possible value, for which you will see a black image without distortion. Then step on the floor and see if your movement is detected.

If all tests have been passed successfully so far, you can proceed with step 3 below.

3. Create a Motioncube account, install and launch the Motioncube 3D Fish app

In the menu on the motioncube.io website, click the Login button and go to the Motioncube account creation form, which is required to download the free Motioncube 3D Fish application.

After creating an account, restart Motioncube Player and log in to your Motioncube account.

Then click on Menu Application store > Applications. In the search box, type 'motioncube 3d fish'. Install the app that appears. No fees are charged.

After installation, launch the application.

When the interaction area is empty, fish will swim freely, and the water will not be waving.

When you move on the floor (or wave your hand in front of the camera), you will see the waves on the water's surface. The fish will startle and change their direction when your movement is detected.

4. Prepare installation setup

All the tests so far you could do without a projector. However, it's time to connect all the hardware components and perform the integrated test.

Mount the projector so that it faces the same surface as the camera. There are many ways how to do this. The most popular approaches are described in our Help Center on the Working Environment subpage.

Make sure the camera sees the whole image displayed by the projector. The depth cameras have small angles of view and, therefore often must be placed above the projector (1-2m above, depending on the projector throw ratio and size of the displayed image). Top central mounting above the interactive area is often a choice, but you can also work with a side camera mount. Remember that people playing in the interactive area shall be in front of the projector and not have the projector behind their backs due to the shadow effect.

Below is a visualized mobile installation idea for the short throw projector (0.6) and the depth camera. Both devices are mounted using stands/tripods. For the projector use a special mount enabling to face the projector down the floor (i.e. projector plate and 90 degree ball head mount). For the camera use a boom stand.

You can also combine these two devices on one tripod/stand with two mounting points.

You can use instead an ultra-short throw projector and place it just above the floor.

The idea for Motioncube interactive installation with a short throw projector and a depth camera

Below you can see our interactive environment for testing. We used the short throw projector with a throw ratio of 0.58 and mounted it to the ceiling on the side.

Motioncube interactive installation for testing

Below you can see how we installed the cameras to perform our tests. We used a lighting ramp that allows you to mount any elements to the frame. The viewing angle of the cameras is smaller than the projection angle. Cameras are placed much higher than the projector to capture the projection area.

Depth cameras for testing mounted to the frame

5. Calibrate the interaction area on the floor

Calibrate the interaction area directly on the floor. Show the camera where the image from the projector is by adjusting the corners of the interaction area to match the image displayed by the projector. Grab the yellow squares with your mouse cursor and drag them to the right place.

To help yourself (when adjusting the camera view from the PC screen) you may place some objects in four corners of the image on the floor, that will be visible in the camera view.

Calibration of the interaction area

Once you have set the interaction area, check the motion detection by selecting Fullscreen, then press the Delete key. Move in the interaction area and observe the preview. If you do not see your movement detected in some of the areas (focus on edges and corners), please correct the interaction area in the camera options. To leave the Fullscreen mode press Escape key.

Watch a short video where we presented the configuration process for the Orbbec Astra camera.

6. Final test

Now you can test the "Motioncube 3D Fish" app on the floor. You will find the app in Menu > My app library > Applications.

See how it works with us:

We hope the guide was helpful to you. In case of problems with integration, contact us, and we will help you solve them.