MTCNN ncsdk v2 mvnccompile error

mdsimmo

Hello,

I wish to compile MTCNN graphs from the ncappzoo (sdk v1) for sdk v2. However, when compiling, I get the following error:

making compile
Compiling to create p2838.graph
/usr/local/bin/ncsdk/Controllers/Parsers/TensorFlowParser/Convolution.py:44: SyntaxWarning: assertion is always true, perhaps remove parentheses?
  assert(False, "Layer type not supported by Convolution: " + obj.type)
mvNCCompile v02.00, Copyright @ Intel Corporation 2017

/usr/local/lib/python3.5/dist-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
shape: (1, 28, 38, 3)
Traceback (most recent call last):
  File "/usr/local/bin/mvNCCompile", line 169, in <module>
    create_graph(args.network, args.image, args.inputnode, args.outputnode, args.outfile, args.nshaves, args.inputsize, args.weights, args.explicit_concat, args.ma2480, args.scheduler, args.new_parser, args)
  File "/usr/local/bin/mvNCCompile", line 148, in create_graph
    load_ret = load_network(args, parser, myriad_config)
  File "/usr/local/bin/ncsdk/Controllers/Scheduler.py", line 100, in load_network
    parse_ret = parse_tensor(arguments, myriad_conf)
  File "/usr/local/bin/ncsdk/Controllers/TensorFlowParser.py", line 1168, in parse_tensor
    xyz = (1, 1, int(input_shape[1]))
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/tensor_shape.py", line 521, in __getitem__
    return self._dims[key]
IndexError: list index out of range
***
Error - Could not compile, not able to create p2838.graph 
***

Makefile:112: recipe for target 'compile' failed
make: *** [compile] Error 1

I can run and compile other examples fine and I could compile it using sdk v1, so I don't know what the problem is.

Regards,
Matt

PINTO

@mdsimmo
https://ncsforum.movidius.com/discussion/1095/informal-supported-unsupported-tensorflow-layer-cause-etc-of-out-of-range-exception

mdsimmo

@PINTO
But why can it be compiled in sdk v1? Was support for some layers dropped?

PINTO

@mdsimmo

Was support for some layers dropped?

Probably it is.
Intel seems to be changing the specifications significantly when upgrading from v1 to v2.
Many engineers confirmed that MTCNN operates on v1, but also confirmed that v2 does not work.

https://ncsforum.movidius.com/discussion/comment/3720/
https://ncsforum.movidius.com/discussion/1093/how-to-assign-multiple-graph-files-to-one-stick

mdsimmo

So the code is breaking in a known layer type, so it cannot be unsupported layers. In "/usr/local/bin/ncsdk/Controllers/TensorFlowParser.py" (where the code is breaking) there have been a few changes between sdk v1 and sdk v2. By comparing the runtime between the two files, I've managed to get it to compile but don't know if it actually works yet (I'll test soon - I first have to make the program use only one NCS). Before I forget, here are the changes I had to make:

All changes are in "/usr/local/bin/ncsdk/Controllers/TensorFlowParser.py" (from sdk 2 install). Note: I have no clue what I'm doing and take no responsibility for screwing up your install.

line ~1146:

was:

elif ((node.type == 'Add' or node.type == 'Mul' or node.type == 'Maximum') and
                    get_input(strip_tensor_id(node.inputs[0].name), False) != 0 and
                    get_input(strip_tensor_id(node.inputs[1].name), False) != 0 and

should be:

 elif ((node.type == 'Add' or node.type == 'Mul' or node.type == 'Maximum') and
                    ("alpha" not in node.inputs[0].name and get_input(strip_tensor_id(node.inputs[0].name), False) != 0) and
                    ("alpha" not in node.inputs[1].name and get_input(strip_tensor_id(node.inputs[1].name), False) != 0) and

line ~1126:

was:

elif (node.type == "Concat" or node.type == "ConcatV2") and not node.inputs[0].dtype.is_integer:
                if debug:
                    print("Concat")
                concat_channel_size = 0
                inputs = node.inputs
                for src in inputs:
                    if len(src.get_shape()) >= 4:
                        concat_channel_size += int(src.get_shape()[3])
                a_input = node.inputs[1].get_shape()
                node.outputs[0].set_shape(
                    [a_input[0], a_input[1], a_input[2], concat_channel_size])

should be:

 elif (node.type == "Concat" or node.type == "ConcatV2") and not node.inputs[0].dtype.is_integer:
                if debug:
                    print("Concat")
                concat_channel_size = 0
                inputs = node.inputs
                for src in inputs:
                    dim = len(src.get_shape())-1
                    if dim > 0:
                        concat_channel_size += int(src.get_shape()[dim])
                a_input = node.inputs[1].get_shape()

                if len(a_input) == 2:
                    node.outputs[0].set_shape([a_input[0], concat_channel_size])
                else: 
                    node.outputs[0].set_shape(
                        [a_input[0], a_input[1], a_input[2], concat_channel_size])

mdsimmo

Well, it didn't work. Trying to run the graph produces a mighty error when calling output_pgraph.read_elem(). I can make zero sense of what is wrong, so I don't know where to go from here.

*** Error in `python3': malloc(): memory corruption: 0x0000000001dfd8a0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7fd239edb7e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8213e)[0x7fd239ee613e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x54)[0x7fd239ee8184]
python3(PyObject_Malloc+0x159)[0x5d3c49]
python3(PyBytes_FromStringAndSize+0x3f)[0x5c5adf]
python3[0x4e9798]
python3(_PyObject_GenericGetAttrWithDict+0x11d)[0x593afd]
python3(PyEval_EvalFrameEx+0x44d)[0x53712d]
python3(PyEval_EvalFrameEx+0x4b04)[0x53b7e4]
python3(PyEval_EvalFrameEx+0x4b04)[0x53b7e4]
python3[0x540199]
python3(PyEval_EvalFrameEx+0x50b2)[0x53bd92]
python3[0x540199]
python3(PyEval_EvalCode+0x1f)[0x540e4f]
python3[0x60c272]
python3(PyRun_FileExFlags+0x9a)[0x60e71a]
python3(PyRun_SimpleFileExFlags+0x1bc)[0x60ef0c]
python3(Py_Main+0x456)[0x63fb26]
python3(main+0xe1)[0x4cfeb1]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7fd239e84830]
python3(_start+0x29)[0x5d6049]
======= Memory map: ========
00400000-007a9000 r-xp 00000000 08:01 11541642                           /usr/bin/python3.5
009a9000-009ab000 r--p 003a9000 08:01 11541642                           /usr/bin/python3.5
009ab000-00a42000 rw-p 003ab000 08:01 11541642                           /usr/bin/python3.5
00a42000-00a73000 rw-p 00000000 00:00 0 
015b7000-01e1a000 rw-p 00000000 00:00 0                                  [heap]
7fd1f8000000-7fd1f8021000 rw-p 00000000 00:00 0 
...
7fd224e88000-7fd224e8a000 rw-p 0017c000 08:01 11536513                   /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7fd224e8a000-7fd224e8e000 rw-p 00000000 00:00 0 Aborted (core dumped)

shartoo

So it seems OOM problem?Can you test it on two sticks?