Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

RealSense D435 + Neural Compute Stick + MobileNet-SSD + RaspberryPi3

It may already be released by someone, but I made a cooperative sample of RealSense D435 + Neural Compute Stick + MobileNet-SSD + RaspberryPi3.
Distance can also be measured while detecting objects.
I'm glad if you can use it as a reference.
This time, it is single stick + single thread.

By the way, sample of multistick + multithread is here.



  • 35 Comments sorted by Votes Date Added
  • However, at the present time it will not work with NCSDK v 2.00 +

  • PINTO why will it not work with 2.0 +? Is it because of the initialization ?

  • How many fps were able to maintain.

  • edited July 10 Vote Up0Vote Down


    PINTO why will it not work with 2.0 +? Is it because of the initialization ?

    This sample uses programs created for the generation of NCSDK Ver1.
    It seems that the call part of the API has been updated in Ver2.
    I just did not test it with Ver.2.
    I will make it correspond to Ver.2 in the near future.

    How many fps were able to maintain.

    By the way, I have not measured FPS for the following programs yet.
    I will try it today. Please wait for a while.
    -> MobileNet-SSD-RealSense

    Movie playback and detection are performed asynchronously.
    Detection is probably about 8FPS for single threads and about 15FPS for 3 threads.
    FPS changes according to the input resolution of the camera.
    In order to prioritize speed, the input image is now dropped to 320x240, so it is a coarse image.
    -> MobileNet-SSD

  • @chicagobob123
    MobileNet-SSD-RealSense is about 6.5 FPS.(Detection + Screen drawing)

  • 6.5 at 320x240 very impressive for a PI.

  • edited July 11 Vote Up0Vote Down

    Since the installation of NCSDK v2.05.00.02 is now finished, i will modify the Github program.

  • That would be great. Thanks.

  • edited July 14 Vote Up0Vote Down

    It corresponds to NCSDK Ver2.
    Input image is 640x480.
    I do not know whether the measurement method is correct, the performance is still 6.5 FPS.

    It is an article written by me, but combined the following procedure.
    Procedure for installing NCSDK v2.05.00.02 to RaspberryPi3(Raspbian Stretch)
    Procedure for "make examples" NCSDK v2.05.00.02 RaspberryPi3(Raspbian Stretch) / Ubuntu16.04

    Large screen support (over 1024x768) and multistick/multithread correspondence will be done next time.

  • edited July 19 Vote Up0Vote Down

    The following sample program is compatible with NCSDK v + MobileNetSSD(Caffe) + Asynchronous execution.
    It was about 12 FPS with one stick.
    Due to the influence of "Global Interpreter Lock", it does not seem to be able to raise the performance any further unless it corresponds to multiprocess.

    Sample of Multistick + Multithread.

  • Hello. And I'm sorry many times.
    It supports MultiProcessing. (Quad core)
    It is a rough measurement result, but it is a play speed of about 25 FPS and a prediction rate of about 12 FPS.
    It is out of frame because it is asynchronous.
    I hope it will be helpful for everyone.

    I was referring to @sggriset article.
    Thank you very much!!

  • @PINTO Thanks for sharing your efforts and contributions to the Intel Movidius NCS community! Can't wait to test this one out.

  • edited July 23 Vote Up0Vote Down

    It is a soliloquy.
    Supported for USB Camera Mode.
    MultiStick x4 Boosted.
    In Japan, we call it "4 bai KAI-O-KEN".

  • Hello everyone.
    I tried combining Object Detection with MobileNetSSD and Transparent Background.
    By using SSD, I succeeded transparent the background while responding to dynamically changing distances.
    Because the load of image processing is high, although only performance of only 5FPS comes out...
    It may be interesting to look at combination with face detection etc.

  • ok , I finish the same work too on a upsquared board(from intel)

  • edited August 13 Vote Up0Vote Down

    If you have a lot of money, that is the correct answer.
    Performance of UP Board 2 is incomplete, so I would like you to try it on LattePanda Alpha+OpenVINO if you can.
    I am poor so I can not try it.

  • emmm I just think it has nothing to do with it's price, you know intel would send many device for free

  • edited August 25 Vote Up0Vote Down

    I thought that many engineers wanted to know the marginal performance.
    Seriously, I measured the performance for each number of sticks.
    Video playback rate and detection rate are calculated separately.
    In 4 core RaspberryPi 3 use of 3 stick or more seems to have no effect.
    Video device : USB Camera (No RealSense D435)
    Input resolution : 640x480
    Output resolution : 640x480

    1 Stick = 6 FPS

    2 Sticks = 12 FPS

    3 Sticks = 16.5 FPS

    4 Sticks = 16.5 FPS

  • Do you use a powered hub or plug in the sticks direct? If direct, how do you get them to fit.

  • @chicagobob123
    I am sorry I can not meet your expectations.
    I used a self-powered USB HUB.
    With RaspberryPi alone, the voltage is not enough from the second stick.

  • "I am sorry I can not meet your expectations"
    Pinto, just asking a question because the sticks are so fat with the heat sink I can not plug more than one into the Pi. I used a short USB to UBS cable and have had 2 plugged in but I think it blew up the Pi. It no longer starts giving me low voltage warnings. I am trying to rebuild a new one. I love your tests because you are trying some interesting combinations.

  • @chicagobob123
    I thought you were expecting a great suggestion on how to not use self-powered USBHub in MultiStick configuration.
    I was misunderstanding.

    just asking a question because the sticks are so fat with the heat sink I can not plug more than one into the Pi.

    I understood what you said.
    Certainly, Stick is enormously fat.
    I am very happy if it is a little thinner...
    As you say, connecting two directly to a USB port will inevitably make it impossible to connect other USB devices.

    I love your tests because you are trying some interesting combinations.

    Thank you very much!

  • edited September 1 Vote Up0Vote Down

    I corresponded to MultiGraph and FaceDetection.
    USB Camera + MultiGraph(PASCAL VOC + WIDER FACE) + MultiProcessing + MultiStick

    And, After changing the Good quality USB Camera and lowering the resolution, it turned out that the performance got better even with 4 Stick.
    About 25 FPS
    USB Camera = PlayStationEye (Second-hand goods $5)

  • edited September 3 Vote Up0Vote Down

    When I increased the stick to 5, I got about 30 FPS.
    Up to the upper limit of the bus speed of USB 2.0, performance may be improved still more.
    I can not procure any more sticks, so I can not check it...
    USB Camera is PlayStationEye (Second-hand goods $5)
    Because children are asleep, I verified in the dark, and I made many mistakes in label judgment.

  • edited September 8 Vote Up0Vote Down

    I have learned that there is a problem that the movement of the stick becomes unstable due to heat.
    And I noticed that many engineers suffered from that problem.

    To prevent thermal runaway, simple clustering function of stick was implemented.
    However, Error handling is not implemented seriously.

    When a certain cycle or constant temperature is reached, the active cluster switches seamlessly automatically.
    You must turn on the clustering enable flag.
    The default switch period is 10 seconds, the default temperature threshold is 65°C.
    The number, cycle, and temperature of sticks constituting one cluster can be specified by the start parameter.
    However, fine-grained control is not performed automatically.
    Depending on your environment, please tune to the optimum parameters yourself.

  • Hi, very nice work! Can you use the RealSense 435 module as source of depth information AND as RGB Camera at the same time?
    BG, Thomas

  • edited October 5 Vote Up0Vote Down

    of course. :)

    # Wait for a coherent pair of frames: depth and color
    frames = pipeline.wait_for_frames()
    depth_frame = frames.get_depth_frame()←Depth
    color_frame = frames.get_color_frame()←RGB
  • nice, but this is SDK code, so it does not report itself as a normal USB webcam I suppose?

  • edited October 5 Vote Up0Vote Down

    Sorry, I seem to have misunderstood the meaning of your question.
    Which of the following does your question mean?

    1. D435 + RealSenseSDK(Depth) + USB Camera(RGB) ... While acquiring Depth at D435, use the USB camera at the same time.
    2. D435(Depth) + USB Camera(RGB) ... While acquiring Depth at D435, use the USB camera at the same time. However, do not use RealSense SDK.
    3. USB Camera (Depth + RGB) ... Get Depth and RGB with USB camera alone. Do not use RealSense SDK at all.

    If it is 1... It can be realized by modifying the program.
    If it is 2... As long as you do not use SDK, RealSense will not work.
    If it is 3... (1) If you know the height of the object, you can measure the coarse distance from the illuminance information with only one USB camera.
    If it is 3... (2) If you have two USB cameras, you can calculate the distance with a simple trigonometric function.

    Please read the article using Google Translate.

    Distance measurement with one RGB camera (Japanese)簡易距離計測

    Distance measurement with two RGB cameras (Japanese)

  • Thx for your reply!
    What I meant was 1.) I want to measure distance and use the RGB camer at the same time from the D435 so that I only need one imaging sensor and thanks, I will look at your code, I have just ordered the devices.

Sign In or Register to comment.