frame

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Alert: Beginning Tuesday, June 25th, we will be freezing this site and migrating the content and forums to our new home at https://forums.intel.com. Check it out now!

[NCS2, RPi] Running multiple models as a flask app

I have a flask app running multiple models with one NCS2.
Implementation wise, individual models have their own module with initialization and inference functions that's imported to the main flask app.

There's a callback function that releases opencv resources, but it returns this error whenever I try to run a different model afterwards. (this one is called manually after running a face-detection model with flask)

E: [ncAPI] [    196800] resetAll:256     Failed to connect to stalled device, rc: X_LINK_ERROR
    Traceback (most recent call last):
      File "ssd.py", line 48, in <module>
        out = net.forward()
    cv2.error: OpenCV(4.0.1-openvino) /home/jenkins/workspace/OpenCV/OpenVINO/build/opencv/modules/dnn/src/op_inf_engine.cpp:553: error: (-215:Assertion failed) Failed to initialize Inference Engine backend: Can not init USB device: NC_DEVICE_NOT_FOUND in function 'initPlugin'

I tried resetting the USB manually to release the graph in the NCS2; but the moment I unplug the usb I get these messages. (also gonna try software reset with usbreset.c, but I expect something similar)
`
λ ~ python3 server/server
OpenCV(4.0.1-openvino) /home/jenkins/workspace/OpenCV/OpenVINO/build/opencv/modules/imgproc/src/resize.cpp:3784: error: (-215:Assertion failed) !ssize.empty() in function 'resize'

    //Unplugged and replugged NCS2 without calling the face detection function - A okay

    //Unplugged NCS2 after running it and stopping it
    E: [xLink] [    938622] dispatcherEventReceive:308      dispatcherEventReceive() Read failed -1 | event 0x67b01e00 USB_READ_REL_REQ

    E: [xLink] [    938622] eventReader:256 eventReader stopped
    E: [watchdog] [    938711] sendPingMessage:164  Failed send ping message: X_LINK_ERROR
    E: [watchdog] [    939711] sendPingMessage:164  Failed send ping message: X_LINK_ERROR
    E: [watchdog] [    940712] sendPingMessage:164  Failed send ping message: X_LINK_ERROR
    E: [watchdog] [    941712] sendPingMessage:164  Failed send ping message: X_LINK_ERROR
    E: [watchdog] [    942712] sendPingMessage:164  Failed send ping message: X_LINK_ERROR
    E: [watchdog] [    943712] sendPingMessage:164  Failed send ping message: X_LINK_ERROR
    E: [watchdog] [    944713] sendPingMessage:164  Failed send ping message: X_LINK_ERROR
    E: [watchdog] [    945713] sendPingMessage:164  Failed send ping message: X_LINK_ERROR
    E: [watchdog] [    946713] sendPingMessage:164  Failed send ping message: X_LINK_ERROR
    E: [watchdog] [    947713] sendPingMessage:164  Failed send ping message: X_LINK_ERROR
    E: [watchdog] [    948713] sendPingMessage:164  Failed send ping message: X_LINK_ERROR
    E: [watchdog] [    949714] sendPingMessage:164  Failed send ping message: X_LINK_ERROR
    ...
`

Module itself has this structure

import cv2
from importlib import reload

net = ""
cap = ""

def get_frame():
    ...
    return jpeg.tobytes()

def gen():
    ...
    yield (get_frame())

def cam_start():
    global cap, net

    #net = cv2.dnn.readNet('face-detection-adas-0001.xml', 'face-detection-adas-0001.bin')
    net = cv2.dnn.readNet('/home/pi/server/face_detection/face-detection-retail-0004.xml',
                          '/home/pi/server/face_detection/face-detection-retail-0004.bin')

    # Specify target device
    net.setPreferableTarget(cv2.dnn.DNN_TARGET_MYRIAD)
    cap = cv2.VideoCapture(0, cv2.CAP_V4L)
    cap.set(cv2.CAP_PROP_FPS, 15)
    cap.set(cv2.CAP_PROP_FRAME_WIDTH, 480)
    cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 360)
    return "cam started"

def cam_stop():
    global cap, net
    del net
    net = ""
    cap.release()
    reload(cv2)
    #cv2.destroyAllWindows()
    return "cam stopped"

They both run fine individually.
I looked through opencv4.0.1 documentation to look for a way to release the graph or anything relevant in the dnn module section, but I've had no luck so far.

Anything I might've missed or hacky potential solution would be appreciated :)

Thanks,

Comments

  • 8 Comments sorted by Votes Date Added
  • Hi @minswho ,

    Thanks for contacting us. Let me know if I understand correctly, you have two models that run fine individually but when you attempt to use them together it fails, may I ask which release of OpenVINO Toolkit are you using? Also your host OS and any other details that could be helpful (i.e. powered USB hub 2.0/3.0, Ubuntu 16.04, etc.). If you could also share your code to help us reproduce the problem would be great.

    Regards,
    @Luis_at_Intel

  • I believe I'm running R5. And no, models are run one at a time. However, when I try to load the NCS with another model after having already run one, it gives out the "failed to connect to stalled device" error, even with cam_stop() releasing opencv resources. I found a hacky workaround where I intentionally cause a module import error to kind of hard reset (idk, it's a throwaway demo under time crunch), but I'd still like to find out how I can "reset" the NCS2.

    Finally, this is a bit unrelated, but can I ask if movidius plan on eventually supporting RNN with NCS2? If not, could you recommend any similar chipsets that support RNN who'll be in volume manufacturing by Q4 2019?

  • @Luis_at_Intel I encountered exactly the same error message on my UP-BOARD computer (Win10 1709) with NCS2 (through additional power supply USB3.0 Y cable) and OpenVINO 2018 R5.0.1. I believe this issue can be easily reproduced with OpenVino TinyYOLO object detection Python example (async mode) after running the object detection 2~3 hours continuously. I have no idea why it's not stable under torture testing. Hopefully, NCS2 is designed for 24x7 workload, and it's not an overheating issue.

    WinUsb_ReadPipe: System err 2
    [35mE: [xLink] [ 0] handleIncomingEvent:240 handleIncomingEvent() Read failed -2
    [0m
    WinUsb_WritePipe: System err 22
    [33mW: [xLink] [ 0] dispatcherEventReceive:324
    WinUsb_WritePipe failed with error:=22
    [35mE: [xLink] [ 0] dispatcherEventSend:889 Write failed header -2 | event USB_WRITE_REQ
    [0m
    Failed to handle incoming event[0m
    WinUsb_SetPipePolicy: System err 22
    [35mE: [xLink] [ 0] dispatcherEventReceive:308 dispatcherEventReceive() Read failed -2 | event 00000010C2B2FEA0 USB_WRITE_REQ
    [0m
    [35mE: [xLink] [ 0] eventReader:256 eventReader stopped[0m
    [35mE: [ncAPI] [ 0] ncGraphQueueInference:3538 Can't send trigger request[0m
    [35mE: [watchdog] [ 0] sendPingMessage:164 Failed send ping message: X_LINK_ERROR[0m
    [35mE: [watchdog] [ 0] sendPingMessage:164 Failed send ping message: X_LINK_ERROR[0m
    [35mE: [watchdog] [ 0] sendPingMessage:164 Failed send ping message: X_LINK_ERROR[0m
    [35mE: [watchdog] [ 0] sendPingMessage:164 Failed send ping message: X_LINK_ERROR[0m
    [35mE: [watchdog] [ 0] sendPingMessage:164 Failed send ping message: X_LINK_ERROR[0m
    [35mE: [watchdog] [ 0] sendPingMessage:164 Failed send ping message: X_LINK_ERROR[0m
    [35mE: [watchdog] [ 0] sendPingMessage:164 Failed send ping message: X_LINK_ERROR[0m

  • @Luis_at_Intel BTW, the error I encountered seems to be related to https://ncsforum.movidius.com/discussion/1106/ncs-temperature-issue
    I'm not able to reproduce the USB link disconnection issue when NCS2 running under sync mode (using infer() instead of start_async() ) with TinyYOLO Python example. Is it possible to return a meaningful error code rather than USB link disconnection while NCSv2 overheating?

  • @Luis_at_Intel

    Hi Luis, I just wanted to follow up.
    Your input would be really appreciated.

  • Hi @minswho ,

    Apologies for the delay in my response. Unfortunately we cannot comment or share any roadmap information for the NCS/NCS2 devices, but I will provide this feedback to the engineering team. As far as any recommendations go for similar chipsets that support RNN I'm sorry but I don't have any insights for that.

    I am trying to find out if there is a way in the OpenVINO toolkit to reset the device, I will get back to you as I get more information. Thank you for your patience.
    Regards,
    @Luis_at_Intel

  • Hi @minswho ,

    Thank you for your patience. I believe I have found information related on how to reset the NCS device within OpenVINO. There is a parameter KEY_VPU_FORCE_RESET that resets any stalled devices on plugin initialization, it must be used with SetConfig method. I am hoping to find the equivalent configuration parameters in python, but I think the information below should hopefully guide you in the right direction.

    Myriad Plugin, Supported Configuration Parameters
    * https://docs.openvinotoolkit.org/2018_R5/_docs_IE_DG_supported_plugins_MYRIAD.html
    VPU Plugins, VPU Common Configuration Parameters
    * https://docs.openvinotoolkit.org/2018_R5/_docs_IE_DG_supported_plugins_VPU.html

    Regards,
    @Luis_at_Intel

This discussion has been closed.