frame

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Resnet-10 and Resnet-50 both hang device - Help would be appreciated

I'm working on pervasive device that had support for the caffe framework and I'm attempting to use resnet-10 from opencv to do face detection (https://github.com/opencv/opencv/tree/master/samples/dnn/face_detector). The graph file compiles fine, and the first pass even executes fine and then the device hangs. The following is the trace I'm getting from the device
[INFO] Get Tensor Descriptor Structures
[INFO] create input and output Fifos
I: [ 0] ncFifoCreate:2126 Init fifo
I: [ 0] ncFifoCreate:2126 Init fifo
I: [ 0] ncFifoAllocate:2321 Creating fifo
I: [ 0] ncFifoAllocate:2321 Creating fifo
[INFO] return input and output Fifos
[INFO] Sending Image to NCS device]
D: [ 0] ncFifoWriteElem:2592 No layout conversion is needed 0

D: [ 0] convertDataTypeAndLayout:170 src data type 1 dst data type 0

D: [ 0] convertDataTypeAndLayout:172 SRC: w 300 h 300 c 3 w_s 12 h_s 3600 c_s 4

D: [ 0] convertDataTypeAndLayout:174 DST: w 300 h 300 c 3 w_s 6 h_s 1800 c_s 2

D: [ 0] ncFifoWriteElem:2617 write count 0 num_elements 2 userparam 0x75a86418

[INFO Sendiing to NCS Queue]
I: [ 0] ncGraphQueueInference:3048 trigger start

I: [ 0] ncGraphQueueInference:3145 trigger end

D: [ 0] ncFifoReadElem:2686 No layout conversion is needed 0

D: [ 0] convertDataTypeAndLayout:170 src data type 0 dst data type 1

D: [ 0] convertDataTypeAndLayout:172 SRC: w 7 h 201 c 1 w_s 2 h_s 14 c_s 2

D: [ 0] convertDataTypeAndLayout:174 DST: w 7 h 201 c 1 w_s 4 h_s 28 c_s 4

D: [ 0] ncFifoReadElem:2714 num_elements 2 userparam 0x75a86558 output length 5628

[INFO] Image prdictions returned from NCS
1
1
[INFO] Sending Image to NCS device]
D: [ 0] ncFifoWriteElem:2592 No layout conversion is needed 0

D: [ 0] convertDataTypeAndLayout:170 src data type 1 dst data type 0

D: [ 0] convertDataTypeAndLayout:172 SRC: w 300 h 300 c 3 w_s 12 h_s 3600 c_s 4

D: [ 0] convertDataTypeAndLayout:174 DST: w 300 h 300 c 3 w_s 6 h_s 1800 c_s 2

D: [ 0] ncFifoWriteElem:2617 write count 1 num_elements 2 userparam 0x75a86418

[INFO Sendiing to NCS Queue]
I: [ 0] ncGraphQueueInference:3048 trigger start

E: [ 0] dispatcherEventReceive:200 dispatcherEventReceive() Read failed -1

I notice in the SDK readme that this model had not been tested but Resnet-50 had, so I tried that Model with the same results. The trace from that run can be found below
[INFO] Get Tensor Descriptor Structures
[INFO] create input and output Fifos
I: [ 0] ncFifoCreate:2126 Init fifo
I: [ 0] ncFifoCreate:2126 Init fifo
I: [ 0] ncFifoAllocate:2321 Creating fifo
I: [ 0] ncFifoAllocate:2321 Creating fifo
[INFO] return input and output Fifos
[INFO] Sending Image to NCS device]
D: [ 0] ncFifoWriteElem:2592 No layout conversion is needed 0

D: [ 0] convertDataTypeAndLayout:170 src data type 1 dst data type 0

D: [ 0] convertDataTypeAndLayout:172 SRC: w 224 h 224 c 3 w_s 12 h_s 2688 c_s 4

D: [ 0] convertDataTypeAndLayout:174 DST: w 224 h 224 c 3 w_s 6 h_s 1344 c_s 2

D: [ 0] ncFifoWriteElem:2617 write count 0 num_elements 2 userparam 0x75ac4468

[INFO Sendiing to NCS Queue]
I: [ 0] ncGraphQueueInference:3048 trigger start

I: [ 0] ncGraphQueueInference:3145 trigger end

D: [ 0] ncFifoReadElem:2686 No layout conversion is needed 1

D: [ 0] convertDataTypeAndLayout:170 src data type 0 dst data type 1

D: [ 0] convertDataTypeAndLayout:172 SRC: w 1 h 1 c 1000 w_s 2000 h_s 2000 c_s 2

D: [ 0] convertDataTypeAndLayout:174 DST: w 1 h 1 c 1000 w_s 4000 h_s 4000 c_s 4

D: [ 0] ncFifoReadElem:2714 num_elements 2 userparam 0x75ac45a8 output length 4000

[INFO] Image prdictions returned from NCS
0
0
[INFO] Sending Image to NCS device]
D: [ 0] ncFifoWriteElem:2592 No layout conversion is needed 0

D: [ 0] convertDataTypeAndLayout:170 src data type 1 dst data type 0

D: [ 0] convertDataTypeAndLayout:172 SRC: w 224 h 224 c 3 w_s 12 h_s 2688 c_s 4

D: [ 0] convertDataTypeAndLayout:174 DST: w 224 h 224 c 3 w_s 6 h_s 1344 c_s 2

D: [ 0] ncFifoWriteElem:2617 write count 1 num_elements 2 userparam 0x75ac4468

[INFO Sendiing to NCS Queue]
I: [ 0] ncGraphQueueInference:3048 trigger start

E: [ 0] dispatcherEventReceive:200 dispatcherEventReceive() Read failed -1

The Device works fine for other SSD models I have tried so I don't think I have a hardware problem. The frustrating thing on this problem is it works the first pass fine. Any help would be greatly appreciated.

Comments

  • 7 Comments sorted by Votes Date Added
  • @sggriset Thanks for reporting this issue. I've also seen these hangs with the same networks and they usually happen during fifo.read_elem(). Can you provide your python code for testing and issue reproduction? Thanks.

  • Thank you for your quick response. The following code is set up for resnet-10 SSD from opencv. If you adjust the dimensions you can test resnet-50 also. Assuming you have the graph files so here you go

    # import necessary libs
    import usb.core  # from pyusb
    import usb.util  # from pyusb
    import mvnc.mvncapi as mvnc
    from threading import Thread
    import numpy as np
    import time
    import cv2
    import os
    
    
    VID = 0x03e7
    PID = 0x2150
    
    dim_prep = (300, 300)
    graph_path = os.path.relpath("graphs/facecvgraph")
    out_fifo = 'faceout'
    in_fifo = 'facein'
    graph_name = 'facegraph'
    
    
    class WebcamVideoStream:
        def __init__(self, src = 0):
            # initialize the video camera stream and read the frist frame
            # from the stream
            self.stream = cv2.VideoCapture(src)
            # camera returns a tuple grabbed = boolean value whether the frame was read 
            # frame = the video frame itself
            (self.grabbed, self.frame) = self.stream.read()
            # initialize the variable used to indicate if the thread should
            # be stopped
            self.stopped = False
    
        def start(self):
            # start the thread to read frames from the video stream
            Thread(target = self.update, args=()).start()
            return self
    
        def update(self):
            # keep looping infinitely until the tread is stopped
            while True:
                # if the thread indicator variable is set, stop the thread
                if self.stopped:
                    return
                # otherwise read the next frame from the stream
                (self.grabbed, self.frame) = self.stream.read()
    
        def read(self):
            # return the frame most recently read
            return self.frame
    
        def stop(self):
            # indicate that the thread should be stopped
            self.stopped = True
    
    def preprocess_image(input_image, dim):
        # preprocess the image
        preprocessed = cv2.resize(input_image, dim)
        preprocessed = preprocessed - 127.5
        preprocessed = preprocessed * 0.007843
        preprocessed = preprocessed.astype(np.float32)
    
        # return the image to the callling function
        return preprocessed
    
    # Load graph files and return graph object
    def load_graph(device, Graphpath, graph_name):
        # Initalize a graph
        print(Graphpath)
        print("[INFO] loading the graph file into NCS memory")
        with open(Graphpath, mode='rb') as f:
            graph_buffer = f.read()
    
        print("[INFO] initialize graph object ")
        # Initalize a graph object
        graph = mvnc.Graph(graph_name)
    
        # Allocate the graph to the device
        print("[INFO] allocating the graph to the NCS")
        graph.allocate(device, graph_buffer)
    
        # return graph object
        return graph
    
    def get_tensors(graph, device, input_name, output_name):
        print("[INFO] Get Tensor Descriptor Structures")
        input_descriptors = graph.get_option(mvnc.GraphOption.RO_INPUT_TENSOR_DESCRIPTORS)
        output_descriptors = graph.get_option(mvnc.GraphOption.RO_OUTPUT_TENSOR_DESCRIPTORS)
    
        print("[INFO] create input and output Fifos")
        input_fifo = mvnc.Fifo(input_name, mvnc.FifoType.HOST_WO)
        output_fifo = mvnc.Fifo(output_name, mvnc.FifoType.HOST_RO)
        input_fifo.allocate(device, input_descriptors[0], 2)
        output_fifo.allocate(device, output_descriptors[0], 2)
    
        print("[INFO] return input and output Fifos")
        return  input_fifo, output_fifo
    
    def predict(image, graph, input_fifo, output_fifo):
        # preprocess the image
        # send the image to the NCS and run a forward-pass to grab the
        # netowrk predictions
        print("[INFO] Sending Image to NCS device]")
        input_fifo.write_elem(image, None)
        print("[INFO Sendiing to NCS Queue]")
        graph.queue_inference(input_fifo, output_fifo)
        (output, _ ) = output_fifo.read_elem()
        print("[INFO] Image predictions returned from NCS")
         # grab the number of valid object predictions from the output
        num_valid_boxes = int(output[0])
        predictions = []
    
        for box_index in range(num_valid_boxes):
    
             # calculate the base index into our array so we can extract
             # bound box information
            base_index = 7 + box_index * 7
    
    
             # boxes with non-finite(inf, nan, etc) numbers must be
             # ignored
            if(not np.isfinite(output[base_index]) or
              not np.isfinite(output[base_index + 1]) or
              not np.isfinite(output[base_index + 2]) or
              not np.isfinite(output[base_index + 3]) or
              not np.isfinite(output[base_index + 4]) or
              not np.isfinite(output[base_index + 5]) or
              not np.isfinite(output[base_index + 6])):
              continue
    
            # extract the image width and height and clip the boxes to the
            # image size in case network returns boxes outside of the image
            # boundaries
            (h, w) = image.shape[:2]
            x1 = max(0, int(output[base_index + 3] * w))
            y1 = max(0, int(output[base_index + 4] * h))
            x2 = min(w, int(output[base_index + 5] * w))
            y2 = min(h, int(output[base_index + 6] * h))
    
            # grab the prediction class label, confidence,
            # and bounding box (x, y) - coordinates
            pred_class = int(output[base_index + 1])
            pred_conf = output[base_index + 2]
            pred_boxpts = ((x1, y1), (x2, y2))
    
            # create prediction tuple and append the prediction to the
            # predictions list
            prediction = (pred_class, pred_conf, pred_boxpts)
            predictions.append(prediction)
        print(len(predictions))
        # return the list of predictions to the calling function
        return predictions
    
    
    
    def main():
        mob_device = usb.core.find(idVendor=VID, idProduct=PID)
        if not mob_device:
            print ("[INFO] Movidius Device Not Found")
            exit(1)
        print ("[INFO] Found Movidius Device")
        # set the logging level for the Movidius device
        mvnc.global_set_option(mvnc.GlobalOption.RW_LOG_LEVEL, 0)
    
        devices = mvnc.enumerate_devices()
        if(len(devices) < 1):
            print("[INFO] Error - no Movidius devices detected")
            exit(1)
    
        # get the first NCS device
        dev = mvnc.Device(devices[0])
    
        try:
            dev.open()
        except:
            print("[INFO] Error - Could not open NCS device.")
            exit(1)
        # load graph file
        my_graph = load_graph(dev, graph_path, graph_name)
    
         # set up Fifos
        inputfifo, outputfifo = get_tensors(my_graph, dev, in_fifo, out_fifo)
    
        vs = WebcamVideoStream(src = 0).start()
        # let camera warm up
        time.sleep(2.0)
        while True:
                frame = vs.read()
                frame = preprocess_image(frame, dim_prep)
                key =  key = cv2.waitKey(1) & 0xFF
                if key == 'q':
                    break
                predictions  = predict(frame, my_graph, inputfifo, outputfifo)
                print(len(predictions))
    
        cv2.destroyAllWindows()
        vs.stop()
        input_fifo.destroy()
        output_fifo.destroy()
        dev.destroy()
        exit(0)
    
    if __name__ == '__main__':
        main()
    
  • Tome_at_Intel

    Any progress on reproduction of problem? Do you have what you need to reproduce the problem? Or do you need to supply you the graph files and model?

    Let me know again thank you for your help

  • @sggriset If you could also provide the model file with the prototxt file, it would be helpful to me. Thanks again.

  • Tome_at_Intel

    You can find the caffe model file at the following url
    https://github.com/opencv/opencv_3rdparty/tree/dnn_samples_face_detector_20170830

    You can find the prototxt file (deploy.prototxt) file at the following url https://github.com/opencv/opencv/tree/master/samples/dnn/face_detector

    Let me know what else I can do for you. Thank you for your help and enjoy your weekend.

    Steve

  • Hello, I'm interested too, this model runs fine on OpenCV.

    Thanks!

  • Any update on this issue would be greatly appreciated

Sign In or Register to comment.