frame

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Stick timeout, only full power cycle recovers

Hi, we are on our way to deploying a four to five figure number of Movidius NCS. We run two sticks for our product, connected via usb hub, either with Raspberry Pi or intel NUC (Ubuntu 16.04) base systems.
We have encountered the following error with a number of sticks (at least five different sticks), seemingly at random. The sticks only recover after a full power cycle and work fine afterwards.
Environment temperature is usually around 23° with AC.

Here is the error report:
Scenarios:
1)
2 Movidius Sticks on powered hub (4A, more than enough) on Intel Nuc (Ubuntu 16.04)
2)
2 Movidius Sticks directly plugged into Intel Nuc
3)
2 Movidius Sticks on powered hub on Raspberry Pi 3 Model B, Raspian Stretch
4)
2 Movidius Sticks on powered hub on Raspberry Pi 3+ Model B, Raspian Stretch

We tried USB 3.0 Hubs from Raidsonic, Amazon Basics and other providers. We also tried USB 2.0 Hubs. Other conntected devices: USB Camera (One or two), sometimes LTE Modems.

Task:
Running two models on in background threads, one of them is on quite constant load (inference time around 80ms), one only working occasionally (inference time around 300ms).

Problem:

After some varying time, one of the two movidius sticks throws the Timeout Error (Exception: mvncStatus.TIMEOUT).
Closing or opening it then does not work (as seen in the logs below).
Then not even a reboot solves the problem, just a complete power cycle.

Intel Nuc Syslog:
Sep 6 18:46:50 nuc1 systemd[1]: Started cartwatch entrance.
Sep 6 18:46:52 nuc1 start_entrance[13445]: Stream thread started.
Sep 6 18:47:07 nuc1 start_entrance[13445]: Traceback (most recent call last):
Sep 6 18:47:07 nuc1 start_entrance[13445]: File "demos/detect_movi.py", line 116, in
Sep 6 18:47:07 nuc1 start_entrance[13445]: sys.exit(main(parse_args()))
Sep 6 18:47:07 nuc1 start_entrance[13445]: File "demos/detect_movi.py", line 17, in main
Sep 6 18:47:07 nuc1 start_entrance[13445]: checkout = Checkout(FLAGS)
Sep 6 18:47:07 nuc1 start_entrance[13445]: File "/home/hh/workspace/checkout/demos/../modules/checkout.py", line 96, in init
Sep 6 18:47:07 nuc1 start_entrance[13445]: self.detector_process = Detector(self.frames_queue, self.detections_queue, FLAGS.det_graph)
Sep 6 18:47:07 nuc1 start_entrance[13445]: File "/home/hh/workspace/checkout/demos/../modules/movidius_detector.py", line 46, in init
Sep 6 18:47:07 nuc1 start_entrance[13445]: self.device.OpenDevice()
Sep 6 18:47:07 nuc1 start_entrance[13445]: File "/usr/local/lib/python3.5/dist-packages/mvnc/mvncapi.py", line 147, in OpenDevice
Sep 6 18:47:07 nuc1 start_entrance[13445]: raise Exception(Status(status))
Sep 6 18:47:07 nuc1 start_entrance[13445]: Exception: mvncStatus.ERROR
Sep 6 18:47:07 nuc1 start_entrance[13445]: Exception ignored in: >
Sep 6 18:47:07 nuc1 start_entrance[13445]: Traceback (most recent call last):
Sep 6 18:47:07 nuc1 start_entrance[13445]: File "/home/hh/workspace/checkout/demos/../modules/movidius_detector.py", line 144, in del
Sep 6 18:47:07 nuc1 start_entrance[13445]: self.device.CloseDevice()
Sep 6 18:47:07 nuc1 start_entrance[13445]: File "/usr/local/lib/python3.5/dist-packages/mvnc/mvncapi.py", line 153, in CloseDevice
Sep 6 18:47:07 nuc1 start_entrance[13445]: raise Exception(Status(status))
Sep 6 18:47:07 nuc1 start_entrance[13445]: Exception: mvncStatus.INVALID_PARAMETERS
Sep 6 18:47:08 nuc1 systemd[1]: entrance.service: Main process exited, code=exited, status=1/FAILURE
Sep 6 18:47:08 nuc1 systemd[1]: entrance.service: Unit entered failed state.
Sep 6 18:47:08 nuc1 systemd[1]: entrance.service: Failed with result 'exit-code'.
Sep 6 18:47:18 nuc1 systemd[1]: entrance.service: Service hold-off time over, scheduling restart.
Sep 6 18:47:18 nuc1 systemd[1]: Stopped cartwatch entrance.
Sep 6 18:47:18 nuc1 systemd[1]: Started cartwatch entrance.
Sep 6 18:47:20 nuc1 start_entrance[13671]: Stream thread started.
Sep 6 18:47:35 nuc1 start_entrance[13671]: Traceback (most recent call last):
Sep 6 18:47:35 nuc1 start_entrance[13671]: File "demos/detect_movi.py", line 116, in
Sep 6 18:47:35 nuc1 start_entrance[13671]: sys.exit(main(parse_args()))
Sep 6 18:47:35 nuc1 start_entrance[13671]: File "demos/detect_movi.py", line 17, in main
Sep 6 18:47:35 nuc1 start_entrance[13671]: checkout = Checkout(FLAGS)
Sep 6 18:47:35 nuc1 start_entrance[13671]: File "/home/hh/workspace/checkout/demos/../modules/checkout.py", line 96, in init
Sep 6 18:47:35 nuc1 start_entrance[13671]: self.detector_process = Detector(self.frames_queue, self.detections_queue, FLAGS.det_graph)
Sep 6 18:47:35 nuc1 start_entrance[13671]: File "/home/hh/workspace/checkout/demos/../modules/movidius_detector.py", line 46, in init
Sep 6 18:47:35 nuc1 start_entrance[13671]: self.device.OpenDevice()
Sep 6 18:47:35 nuc1 start_entrance[13671]: File "/usr/local/lib/python3.5/dist-packages/mvnc/mvncapi.py", line 147, in OpenDevice
Sep 6 18:47:35 nuc1 start_entrance[13671]: raise Exception(Status(status))
Sep 6 18:47:35 nuc1 start_entrance[13671]: Exception: mvncStatus.ERROR
Sep 6 18:47:35 nuc1 start_entrance[13671]: Exception ignored in: >
Sep 6 18:47:35 nuc1 start_entrance[13671]: Traceback (most recent call last):
Sep 6 18:47:35 nuc1 start_entrance[13671]: File "/home/hh/workspace/checkout/demos/../modules/movidius_detector.py", line 144, in del
Sep 6 18:47:35 nuc1 start_entrance[13671]: self.device.CloseDevice()
Sep 6 18:47:35 nuc1 start_entrance[13671]: File "/usr/local/lib/python3.5/dist-packages/mvnc/mvncapi.py", line 153, in CloseDevice
Sep 6 18:47:35 nuc1 start_entrance[13671]: raise Exception(Status(status))
Sep 6 18:47:35 nuc1 start_entrance[13671]: Exception: mvncStatus.INVALID_PARAMETERS
Sep 6 18:47:36 nuc1 systemd[1]: entrance.service: Main process exited, code=exited, status=1/FAILURE
Sep 6 18:47:36 nuc1 systemd[1]: entrance.service: Unit entered failed state.

And so fourth..

Rebooting..
then running the command lsusb shows:

Bus 002 Device 004: ID 05e3:0612 Genesys Logic, Inc.
Bus 002 Device 002: ID 05e3:0612 Genesys Logic, Inc.
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 004: ID 8087:0a2b Intel Corp.
Bus 001 Device 007: ID 05a3:9422 ARC International
Bus 001 Device 009: ID 12d1:1506 Huawei Technologies Co., Ltd. Modem/Networkcard
Bus 001 Device 008: ID 05a3:9230 ARC International
Bus 001 Device 006: ID 05e3:0610 Genesys Logic, Inc. 4-port hub
Bus 001 Device 005: ID 05a3:9230 ARC International
Bus 001 Device 003: ID 05e3:0610 Genesys Logic, Inc. 4-port hub
Bus 001 Device 018: ID 03e7:2150
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

-> Showing only one movidius stick.

Doing a full power cycle then restores to the initial situation.

Comments

Sign In or Register to comment.