frame

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

[Error 5] Toolkit Error: Stage Details Not Supported: VarHandleOp

I just prototyping a model in tf.keras following the link below:
https://www.dlology.com/blog/how-to-run-keras-model-on-movidius-neural-compute-stick/

I converted my Keras model to tensorflow, but when I tried to compile my tensorflow model to graph, it gives me error below:
[Error 5] Toolkit Error: Stage Details Not Supported: VarHandleOp

my tensorflow version is 1.9.0

my model:

#

model = models.Sequential()
model.add(layers.Flatten(input_shape=(28,28)))
model.add(layers.Dense(128,activation='relu'))
model.add(layers.Dense(128,activation='relu'))
model.add(layers.Dense(10,activation='softmax'))

model.compile(optimizer='adam',loss = 'sparse_categorical_crossentropy',metrics=['accuracy'])

#

using this command to compile:

#

mvNCCompile tf_model.meta -in=flatten_1_input -on=dense_5/Softmax

#

error:

#

mvNCCompile v02.00, Copyright @ Intel Corporation 2017

****** Info: No Weights provided. inferred path: tf_model.data-00000-of-00001******
shape: [1, 28, 28]
[Error 5] Toolkit Error: Stage Details Not Supported: VarHandleOp

#

I have tried another model structure but I ended up facing the same error

Thank you.

Comments

  • 5 Comments sorted by Votes Date Added
  • Hi @ddvoviyum
    Try enabling the debug option in the Tensorflow parser - this could provide more information about the error and which layer/operation is throwing it. You will need to set the debug flag to True in the following file as sudo:
    /usr/local/bin/ncsdk/Controllers/Parsers/TensorFlow.py

    Let me know what you find.

    Sincerely,
    Sahira

  • I am facing the same issue. I have taken inception_v3 and did transfer learning on it by adding last layer as dense. I have used tf.saved_model.simple_save to export the model.

    Following is the error I get:

    pi@raspberrypi:~$ mvNCCompile -s 12 _retrain_checkpoint.meta -in=Placeholder -on=final_result -o first_graph.graph             
    /usr/lib/python3/dist-packages/scipy/_lib/_numpy_compat.py:10: DeprecationWarning: Importing from numpy.testing.nosetester is deprecated since 1.15.0, import 
    from numpy.testing instead.                                                                                                                                   
      from numpy.testing.nosetester import import_nose                                                                                                            
    /usr/lib/python3/dist-packages/scipy/stats/morestats.py:16: DeprecationWarning: Importing from numpy.testing.decorators is deprecated since numpy 1.15.0, impo
    rt from numpy.testing instead.                                                                                                                                
      from numpy.testing.decorators import setastest                                                                                                              
    /usr/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: compiletime version 3.4 of module 'tensorflow.python.framework.fast_tensor_util' does not matc
    h runtime version 3.5                                                                                                                                         
      return f(*args, **kwds)                                                                                                                                     
    /usr/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: builtins.type size changed, may indicate binary incompatibility. Expected 432, got 412        
      return f(*args, **kwds)                                                                                                                                     
    /usr/local/bin/ncsdk/Controllers/Parsers/TensorFlowParser/Convolution.py:46: SyntaxWarning: assertion is always true, perhaps remove parentheses?             
      assert(False, "Layer type not supported by Convolution: " + obj.type)                                                                                       
    /usr/local/bin/ncsdk/Controllers/Parsers/Phases.py:322: SyntaxWarning: assertion is always true, perhaps remove parentheses?                                  
      assert(len(pred) == 1, "Slice not supported to have >1 predecessors")                                                                                       
    mvNCCompile v02.00, Copyright @ Intel Corporation 2017                                                                                                        
    
    ****** Info: No Weights provided. inferred path: _retrain_checkpoint.data-00000-of-00001******                                                                
    WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.ch
    eckpoint_management) is deprecated and will be removed in a future version.                                                                                   
    Instructions for updating:                                                                                                                                    
    Use standard file APIs to check for files with this prefix.                                                                                                   
    shape: [1, 299, 299, 3]                                                                                                                                       
    [Error 5] Toolkit Error: Stage Details Not Supported: VarHandleOp
    

    I have confirmed the name of the in and out layers by sess.graph.get_operations().

    I am thinking may be it is because of the variables defined in the network?

  • Hi @ddvoviyum
    The "Stage Details Not Supported: VarHandleOp" error is given when you're using operations or layers that aren't yet supported. These errors can get tricky sometimes. Can you share your model and exact steps and I will try to reproduce and get back to you with results.

    Best Regards,
    Sahira

  • Thanks for the response @Sahira_at_Intel
    Here are the steps to reproduce:

    Base script used:
    https://github.com/tensorflow/hub/raw/master/examples/image_retraining/retrain.py

    Changes made:
    1. Changed module to https://tfhub.dev/google/imagenet/inception_v3/feature_vector/1
    2. Used custom dataset of 3 classes instead of the default flower example
    3. Extracted final result files (checkpoints, .meta etc)
    4. Ran mvNCCompile -s 12 _retrain_checkpoint.meta -in=Placeholder -on=final_result

  • @Sahira_at_Intel

    Since my last message, I have conducted my own steps as well. Following are the things I have done:

    Renamed the input layer to img_input by modifying https://github.com/tensorflow/hub/blob/master/examples/image_retraining/retrain.py#L309 to: resized_input_tensor = tf.placeholder(tf.float32, [None, height, width, 3], name="img_input")

    To get information about the final model generated, I used TF's summarize_graph which gives the following output:

    Found 1 possible inputs: (name=img_input, type=float(1), shape=[?,299,299,3]) 
    No variables spotted.
    Found 1 possible outputs: (name=final_result, op=Softmax) 
    Found 21824117 (21.82M) const parameters, 0 (0) variable parameters, and 0 control_edges
    Op types used: 490 Const, 378 Identity, 94 Conv2D, 94 FusedBatchNorm, 94 Relu, 15 ConcatV2, 9 AvgPool, 4 MaxPool, 1 Add, 1 Mean, 1 Mul, 1 Placeholder, 1 PlaceholderWithDefault, 1 MatMul, 1 Softmax, 1 Squeeze, 1 Sub
    To use with tensorflow/tools/benchmark:benchmark_model try these arguments:
    bazel run tensorflow/tools/benchmark:benchmark_model -- --graph=model.pb --show_flops --input_layer=img_input --input_layer_type=float --input_layer_shape=-1,299,299,3 --output_layer=final_result
    

    Upon running mvNCCompile model.pb -in=img_input -on=final_result -o firstgraph, following is the output:

    /usr/lib/python3/dist-packages/scipy/_lib/_numpy_compat.py:10: DeprecationWarning: Importing from numpy.testing.nosetester is deprecated since 1.15.0, import from numpy.testing instead.
      from numpy.testing.nosetester import import_nose
    /usr/lib/python3/dist-packages/scipy/stats/morestats.py:16: DeprecationWarning: Importing from numpy.testing.decorators is deprecated since numpy 1.15.0, import from numpy.testing instead.
      from numpy.testing.decorators import setastest
    /usr/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: compiletime version 3.4 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.5
      return f(*args, **kwds)
    /usr/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: builtins.type size changed, may indicate binary incompatibility. Expected 432, got 412
      return f(*args, **kwds)
    /usr/local/bin/ncsdk/Controllers/Parsers/TensorFlowParser/Convolution.py:46: SyntaxWarning: assertion is always true, perhaps remove parentheses?
      assert(False, "Layer type not supported by Convolution: " + obj.type)
    /usr/local/bin/ncsdk/Controllers/Parsers/Phases.py:322: SyntaxWarning: assertion is always true, perhaps remove parentheses?
      assert(len(pred) == 1, "Slice not supported to have >1 predecessors")
    mvNCCompile v02.00, Copyright @ Intel Corporation 2017
    
    shape: [1, 299, 299, 3]
    [Error 5] Toolkit Error: Stage Details Not Supported: Top Not Found module_apply_default/hub_input/Sub
    

    Next, upon reading some of the threads here in the forum, I thought of giving TF's transform_graph a try:

    bazel-bin/tensorflow/tools/graph_transforms/transform_graph \
        --in_graph=model.pb \
        --out_graph=optimized_model.pb \
        --inputs='img_input' \
        --outputs='final_result' \
        --transforms='
    strip_unused_nodes(type=float, shape="1,299,299,3")
    remove_nodes(op=Identity, op=CheckNumerics)
    fold_old_batch_norms
    '
    

    Then I ran the compilation on this optimized model, which gave the following output:

    /usr/lib/python3/dist-packages/scipy/_lib/_numpy_compat.py:10: DeprecationWarning: Importing from numpy.testing.nosetester is deprecated since 1.15.0, import from numpy.testing instead.
      from numpy.testing.nosetester import import_nose
    /usr/lib/python3/dist-packages/scipy/stats/morestats.py:16: DeprecationWarning: Importing from numpy.testing.decorators is deprecated since numpy 1.15.0, import from numpy.testing instead.
      from numpy.testing.decorators import setastest
    /usr/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: compiletime version 3.4 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.5
      return f(*args, **kwds)
    /usr/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: builtins.type size changed, may indicate binary incompatibility. Expected 432, got 412
      return f(*args, **kwds)
    /usr/local/bin/ncsdk/Controllers/Parsers/TensorFlowParser/Convolution.py:46: SyntaxWarning: assertion is always true, perhaps remove parentheses?
      assert(False, "Layer type not supported by Convolution: " + obj.type)
    /usr/local/bin/ncsdk/Controllers/Parsers/Phases.py:322: SyntaxWarning: assertion is always true, perhaps remove parentheses?
      assert(len(pred) == 1, "Slice not supported to have >1 predecessors")
    mvNCCompile v02.00, Copyright @ Intel Corporation 2017
    
    shape: [1, 299, 299, 3]
    [Error 5] Toolkit Error: Stage Details Not Supported: Top Not Found module_apply_default/hub_input/Sub
    

    Another interesting observation is that if I run mvNCCompile model.pb -in=input/BottleneckInputPlaceholder -on=final_result -o second.graph I get the following:

    /usr/lib/python3/dist-packages/scipy/_lib/_numpy_compat.py:10: DeprecationWarning: Importing from numpy.testing.nosetester is deprecated since 1.15.0, import from numpy.testing instead.
      from numpy.testing.nosetester import import_nose
    /usr/lib/python3/dist-packages/scipy/stats/morestats.py:16: DeprecationWarning: Importing from numpy.testing.decorators is deprecated since numpy 1.15.0, import from numpy.testing instead.
      from numpy.testing.decorators import setastest
    /usr/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: compiletime version 3.4 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.5
      return f(*args, **kwds)
    /usr/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: builtins.type size changed, may indicate binary incompatibility. Expected 432, got 412
      return f(*args, **kwds)
    /usr/local/bin/ncsdk/Controllers/Parsers/TensorFlowParser/Convolution.py:46: SyntaxWarning: assertion is always true, perhaps remove parentheses?
      assert(False, "Layer type not supported by Convolution: " + obj.type)
    /usr/local/bin/ncsdk/Controllers/Parsers/Phases.py:322: SyntaxWarning: assertion is always true, perhaps remove parentheses?
      assert(len(pred) == 1, "Slice not supported to have >1 predecessors")
    mvNCCompile v02.00, Copyright @ Intel Corporation 2017
    
    shape: [1, 2048]
    res.shape:  (1, 2)
    TensorFlow output shape:  (1, 1, 2)
    /usr/local/bin/ncsdk/Controllers/FileIO.py:65: UserWarning: You are using a large type. Consider reducing your data sizes for best performance
    Blob generated
    

    A success! However, this input layer is of a different shape, as you can see. Same success for optimized_model.pb is also seen.

    I suspect that I really need to get rid of the training steps from retrain.py. Well, I do not know.

    All these results are with debug = True in TF parser, as you suggested.

    Model: https://drive.google.com/file/d/1wHRUEOiBjx4dHBeikbMjI7A2SxEjpv60/view?usp=sharing

    Thanks!

This discussion has been closed.