frame

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Alert: This site has migrating to our new home at https://forums.intel.com/s/topic/0TO0P000000PqZDWA0/intel-neural-compute-sticks. Please visit us there!

[Error 5] Toolkit Error: Stage Details Not Supported: Unsupported Mean operation

I am trying to compile Inception-Resenet-V1 (facenet) and I see the mean operation is not supported by the NCS stick yet. Just wanted to see if there plans to support this netowrk architecture and/or the operation.

Here is the network I am using: https://github.com/davidsandberg/facenet

Thanks,
Roman

Comments

  • 19 Comments sorted by Votes Date Added
  • @traderking Hi. To investigate this further, I want to reproduce the problem you encountered. Can you provide the command you used to compile this network? If we don't support the Mean operation, I would like to confirm this so we can add the feature to the NCSDK in the future. Thanks.

  • Ok.

    Here is the direct link to download the .meta file - https://drive.google.com/file/d/0B5MzpY9kBtDVZ2RpVDYwWmxoSUk/edit
    The commend I used: mvNCCompile model-20170512-110547.meta -s 12 -in=input -on=Logits/biases/ExponentialMovingAverage/Assign

    Thanks.

  • Hello!
    I am facing the same problem while trying to parse the FaceNet model in Tensorflow.
    I can see that the Mean operation is supported by NCSDK but unable to trace exactly why and where the dimension mismatch is happening. Can you please help with the same?
    The meta file and checkpoints are the same as what@traderking shared.

  • @PINTO
    @Tome_at_Intel

    I guess you're right.

    I reinstalled ncsdk version 2.0.8 and am trying to figure out the actual error:
    [Error 5] Toolkit Error: Stage Details Not Supported: Top Not Found batch_normalization_1/batchnorm/add_1

    I am currently using tensorflow version 1.9.0 and the first 20 or so operations looks like this:

    (<tf.Tensor 'input_1:0' shape=(1, 150, 150, 3) dtype=float32>,),
    (<tf.Tensor 'batch_normalization_1/beta/Initializer/zeros:0' shape=(3,) dtype=float32>,),
    (<tf.Tensor 'batch_normalization_1/beta:0' shape=(3,) dtype=float32_ref>,),
    (<tf.Tensor 'batch_normalization_1/beta/Assign:0' shape=(3,) dtype=float32_ref>,),
    (<tf.Tensor 'batch_normalization_1/beta/read:0' shape=(3,) dtype=float32>,),
    (<tf.Tensor 'batch_normalization_1/gamma/Initializer/ones:0' shape=(3,) dtype=float32>,),
    (<tf.Tensor 'batch_normalization_1/gamma:0' shape=(3,) dtype=float32_ref>,),
    (<tf.Tensor 'batch_normalization_1/gamma/Assign:0' shape=(3,) dtype=float32_ref>,),
    (<tf.Tensor 'batch_normalization_1/gamma/read:0' shape=(3,) dtype=float32>,),
    (<tf.Tensor 'batch_normalization_1/moving_mean/Initializer/zeros:0' shape=(3,) dtype=float32>,),
    (<tf.Tensor 'batch_normalization_1/moving_mean:0' shape=(3,) dtype=float32_ref>,),
    (<tf.Tensor 'batch_normalization_1/moving_mean/Assign:0' shape=(3,) dtype=float32_ref>,),
    (<tf.Tensor 'batch_normalization_1/moving_mean/read:0' shape=(3,) dtype=float32>,),
    (<tf.Tensor 'batch_normalization_1/moving_variance/Initializer/ones:0' shape=(3,) dtype=float32>,),
    (<tf.Tensor 'batch_normalization_1/moving_variance:0' shape=(3,) dtype=float32_ref>,),
    (<tf.Tensor 'batch_normalization_1/moving_variance/Assign:0' shape=(3,) dtype=float32_ref>,),
    (<tf.Tensor 'batch_normalization_1/moving_variance/read:0' shape=(3,) dtype=float32>,),
    (<tf.Tensor 'batch_normalization_1/batchnorm/add/y:0' shape=() dtype=float32>,),
    (<tf.Tensor 'batch_normalization_1/batchnorm/add:0' shape=(3,) dtype=float32>,),
    (<tf.Tensor 'batch_normalization_1/batchnorm/Rsqrt:0' shape=(3,) dtype=float32>,),
    (<tf.Tensor 'batch_normalization_1/batchnorm/mul:0' shape=(3,) dtype=float32>,),
    (<tf.Tensor 'batch_normalization_1/batchnorm/mul_1:0' shape=(1, 150, 150, 3) dtype=float32>,),
    (<tf.Tensor 'batch_normalization_1/batchnorm/mul_2:0' shape=(3,) dtype=float32>,),
    (<tf.Tensor 'batch_normalization_1/batchnorm/sub:0' shape=(3,) dtype=float32>,),
    ** (<tf.Tensor 'batch_normalization_1/batchnorm/add_1:0' shape=(1, 150, 150, 3) dtype=float32>,),**
    (<tf.Tensor 'block1_conv1/weights/Initializer/random_uniform/shape:0' shape=(4,) dtype=int32>,),

    It seems like the one that is bolded is causing the error and I was wondering if there is a way to get around this.

    Thanks in advance!

  • @traderking It looks like we do have support for the mean operation, but there is a restriction on the shape for the mean operation. If you take a look at the Tensorflow parser located in the /opt/movidius/NCSDK/ncsdk-x86_64/tk/Controllers directory, you can see what I'm talking about.

    elif node.type == "Mean":
                    if debug:
                        print("Mean")
                    inputs = node.inputs[0]
                    input_shape = node.inputs[0].get_shape()
                    dimensions = node.inputs[1].eval()
                    if dimensions[0] != 1 or dimensions[1] != 2:
                        throw_error(
                            ErrorTable.StageDetailsNotSupported,
                            "Unsupported Mean operation")
  • @Tome_at_Intel We have investigated this issue further and identified the shape for our network (inception-resnet-v1) Mean operation is 3 dimensional:
    (160, 160, 3). So, it makes sense why it is breaking as it seems NCS only supports 2-dimensional.

    Do you have plans to support 3 dimensional Mean operations in the future? Or will we need to find a work-around for this?

  • @traderking were you able to find a workaround for this?

  • @traderking Thank you for bringing these types of issues to the forum. We are looking to add as much support to the NCSDK as possible with priority for issues/features that developers request the most. We don't have support for 3 dimensional means at the moment, but we will definitely take this into consideration for a future release.

  • I would like to add an update on this issue.
    This happens because the 3D Mean operation is performed during training time. Remove training related features as per https://movidius.github.io/ncsdk/tf_compile_guidance.html
    FaceNet compiles successfully after this.

  • @suryaprakashmcw Thanks for that update suryaprakashmcw!

  • @Tome_at_Intel thanks for the example. I tried compiling facenet for NCS and running into issues.

    1) I downloaded the model 20170512-110547.zip and placed into facenet root directory.
    2) run make all

    Error I am receiving: [Error 5] Toolkit Error: Stage Details Not Supported: Top Not Found InceptionResnetV1/Bottleneck/BatchNorm/FusedBatchNorm

  • @traderking Just to check, are you running NCSDK version 1.12? It can be found @ https://github.com/movidius/ncsdk/releases/tag/v1.12.00.01

  • @Tome_at_Intel works with 1.2:) Thanks for the support!

  • @traderking Glad to hear it's working for you.

  • @Tome_at_Intel
    Hi, I am actually facing similar error when I try to use mvNCCompile on model with following layers:

    batch_normalization
    vgg19
    dense 50
    dense 3

    [Error 5] Toolkit Error: Stage Details Not Supported: Top Not Found batch_normalization_1/batchnorm/add_1

    I am trying to compile the model on Raspberry Pi 3, and previously I was using NCSDK version 2.0.
    After reading the comments above, I tried uninstalling v2 with 'make uninstall', and reinstall the version provided on the link.

    The installation of v1.12 was successful but the message that pops up after entering the mvNCCompile command still is:
    "mvNCCompile v02.00, Copyright @Movidius Ltd 2016"

    It looks like version 1.12 is not being installed?

    Anyways, my original problem has not been fixed and I was wondering if you could guide me in solving the error.

    Thanks!

  • @KaungB

    The installation of v1.12 was successful but the message that pops up after entering the mvNCCompile command still is:
    "mvNCCompile v02.00, Copyright @Movidius Ltd 2016"
    It looks like version 1.12 is not being installed?

    The version of NCSDK does not seem to be related at all to the above display.

  • Or is the operation itself not supported?

  • edited January 18 Vote Up0Vote Down

    @KaungB

    Since NCSDK is unstable, I recommend using OpenVINO.
    https://github.com/PINTO0309/OpenVINO-YoloV3#1-work-with-laptoppc-ubuntu-1604

This discussion has been closed.