CpuFullyConnected Class Reference

Basic function to compute a Fully Connected layer. More...

#include <CpuFullyConnected.h>

Collaboration diagram for CpuFullyConnected:

Public Member Functions
	CpuFullyConnected ()
	Constructor. More...
	~CpuFullyConnected ()
	Destructor. More...
void	configure (const ITensorInfo *src, const ITensorInfo *weights, const ITensorInfo *biases, ITensorInfo *dst, FullyConnectedLayerInfo fc_info=FullyConnectedLayerInfo())
	Set the input and output tensors. More...
void	run (ITensorPack &tensors) override
	Run the kernels contained in the function. More...
void	prepare (ITensorPack &tensors) override
	Prepare the function for executing. More...
experimental::MemoryRequirements	workspace () const override
	Return the memory requirements required by the workspace. More...
Public Member Functions inherited from INEOperator
	INEOperator (IRuntimeContext *ctx=nullptr)
	Constructor. More...
	INEOperator (const INEOperator &)=delete
	Prevent instances of this class from being copied (As this class contains pointers) More...
	INEOperator (INEOperator &&)=default
	Default move constructor. More...
INEOperator &	operator= (const INEOperator &)=delete
	Prevent instances of this class from being copied (As this class contains pointers) More...
INEOperator &	operator= (INEOperator &&)=default
	Default move assignment operator. More...
	~INEOperator ()
	Default destructor. More...
Public Member Functions inherited from IOperator
virtual	~IOperator ()=default
	Destructor. More...

Static Public Member Functions
static Status	validate (const ITensorInfo *src, const ITensorInfo *weights, const ITensorInfo *biases, const ITensorInfo *dst, FullyConnectedLayerInfo fc_info=FullyConnectedLayerInfo())
	Static function to check if given info will lead to a valid configuration of CpuFullyConnected. More...

Detailed Description

Basic function to compute a Fully Connected layer.

This function calls the following kernels:

1. kernels::CpuIm2ColKernel (called when the input comes from a convolutional layer)
2. kernels::CpuTransposeKernel (if are_weights_reshaped is set to false and transpose_weights is set to true ) (called once)
3. CpuGemm or CpuGemmLowpMatrixMultiplyCore (if quantized asymmetric)
4. kernels::CpuGemmMatrixAdditionKernel or CpuGemmLowpOutputStage (if quantized asymmetric) (if biases is not equal to nullptr)

Note

The fully connected layer accepts "weights" tensors only with 2 dimensions.

Definition at line 54 of file CpuFullyConnected.h.

Constructor & Destructor Documentation

CpuFullyConnected()

CpuFullyConnected

(

)

Constructor.

Definition at line 148 of file CpuFullyConnected.cpp.

    : _flatten(nullptr),
      _convert_weights(nullptr),
      _transpose_weights(nullptr),
      _mm_gemm(nullptr),
      _mm_gemmlowp(nullptr),
      _flattened_src(),
      _converted_weights(),
      _reshaped_weights(),
      _trans_weights(),
      _trans_weights_idx(AuxTensorIdx::Count),
      _aux_mem(Count),
      _needs_weights_conversion(false),
      _needs_weights_reshape(false),
      _is_fc_after_conv(false),
      _is_quantized_asymmetric(false),
      _is_prepared(false),
      _enable_fast_math(false)

{
}

~CpuFullyConnected()

~CpuFullyConnected ( )

default

Destructor.

Member Function Documentation

configure()

void configure	(	const ITensorInfo *	src,
		const ITensorInfo *	weights,
		const ITensorInfo *	biases,
		ITensorInfo *	dst,
		FullyConnectedLayerInfo	fc_info = FullyConnectedLayerInfo()
	)

Set the input and output tensors.

Valid data layouts:

• NHWC
• NCHW

Valid data type configurations:

src0	src1	src2	dst
F16	F16	F16	F16
F32	F32	F32	F32
QASYMM8	QASYMM8	S32	QASYMM8
QASYMM8_SIGNED	QASYMM8_SIGNED	S32	QASYMM8_SIGNED

Parameters

[in]	src	Source tensor info. Data type supported: QASYMM8/QASYMM8_SIGNED/F16/F32.
[in]	weights	Weights tensor info. The weights must be 2 dimensional. If this function is called after a Convolution Layer, the (transposed) weights will have as many rows as the product of the first 3 input's dimensions. If it is called after another FullyConnected Layer, the (transposed) weights will have as many rows as the input's first dimension. Data type supported: Same as src.
[in]	biases	Bias tensor info. Can be nullptr. Data type supported: Same as weights, S32 if weights is QASYMM8/QASYMM8_SIGNED.
[out]	dst	Destination tensor info. Its shape should be equal to the output of a matrix multiplication between: The output of im2col on the input and the (transposed) 2D weights, if the function is called after a Convolution Layer The input tensor and the (transposed) 2D weights, if the function is called after another FullyConnected Layer. Data type supported: Same as src.
[in]	fc_info	(Optional) Fully connected layer additional info

Definition at line 231 of file CpuFullyConnected.cpp.

References FullyConnectedLayerInfo::activation_info, ITensorInfo::are_values_constant(), FullyConnectedLayerInfo::are_weights_reshaped, ARM_COMPUTE_ERROR_ON_NULLPTR, ARM_COMPUTE_ERROR_THROW_ON, ARM_COMPUTE_LOG_PARAMS, Dimensions< T >::cbegin(), Dimensions< T >::cend(), ITensorInfo::data_layout(), ITensorInfo::data_type(), ITensorInfo::dimension(), FullyConnectedLayerInfo::enable_fast_math, arm_compute::is_data_type_quantized_asymmetric(), ITensorInfo::num_dimensions(), Dimensions< size_t >::num_max_dimensions, arm_compute::offset_int_vec(), arm_compute::experimental::Prepare, FullyConnectedLayerInfo::retain_internal_weights, ITensorInfo::tensor_shape(), TensorInfo::total_size(), FullyConnectedLayerInfo::transpose_weights, CpuFullyConnected::validate(), and FullyConnectedLayerInfo::weights_trained_layout.

{
    // Perform validate step
    ARM_COMPUTE_ERROR_ON_NULLPTR(src, weights, dst);
    ARM_COMPUTE_ERROR_THROW_ON(CpuFullyConnected::validate(src,
                                                           weights,
                                                           biases != nullptr ? biases : nullptr,
                                                           dst,
                                                           fc_info));
    ARM_COMPUTE_LOG_PARAMS(src, weights, biases, dst, fc_info);

    _needs_weights_conversion = false;
    _needs_weights_reshape    = fc_info.transpose_weights ? !fc_info.are_weights_reshaped : false;
    _needs_weights_reshape    = _needs_weights_reshape && !fc_info.retain_internal_weights;
    _is_fc_after_conv         = true;
    _is_quantized_asymmetric  = is_data_type_quantized_asymmetric(src->data_type());
    _is_prepared              = false;
    _trans_weights_idx        = AuxTensorIdx::Count;
    _enable_fast_math         = fc_info.enable_fast_math;

    // With the Fully Connected layer we can have 4 different cases:
    //  1) Convolution layer -> Fully Connected layer without batches
    //  2) Fully Connected layer -> Fully Connected layer without batches
    //  3) Convolution layer -> Fully Connected layer with batches
    //  4) Fully Connected layer -> Fully Connected layer with batches

    const ITensorInfo *weights_to_use = weights;

    // Check if we have a fully connected layer with batches
    const bool is_batched_fc_layer = dst->dimension(1) > 1;
    if(is_batched_fc_layer)
    {
        _is_fc_after_conv = (TensorShape::num_max_dimensions >= 4) && (std::equal(src->tensor_shape().cbegin() + 3,
                                                                                  src->tensor_shape().cend(),
                                                                                  dst->tensor_shape().cbegin() + 1));
    }
    else
    {
        _is_fc_after_conv = src->num_dimensions() > 1;
    }

    // Reshape weights if needed
    if(_needs_weights_reshape)
    {
        // Reshape the weights
        _transpose_weights = std::make_unique<kernels::CpuTransposeKernel>();
        _transpose_weights->configure(weights, &_reshaped_weights);
        weights_to_use     = &_reshaped_weights;
        _trans_weights_idx = AuxTensorIdx::TransposedWeights;
    }

    // Convert weights if needed
    if(_is_fc_after_conv && (src->data_layout() != fc_info.weights_trained_layout))
    {
        // Convert weights
        _convert_weights = std::make_unique<CpuConvertFullyConnectedWeights>();
        _convert_weights->configure(weights_to_use,
                                    &_converted_weights,
                                    src->tensor_shape(),
                                    fc_info.weights_trained_layout);

        weights_to_use            = &_converted_weights;
        _needs_weights_conversion = true;
        _trans_weights_idx        = AuxTensorIdx::ConvertedWeights;
    }

    if(_is_fc_after_conv)
    {
        // Fully Connected layer after a Convolution Layer without batches
        configure_conv_fc(src, weights_to_use, biases, dst, fc_info.activation_info);
    }
    else
    {
        // Fully Connected layer after a Fully Connected Layer without batches
        configure_fc_fc(src, weights_to_use, biases, dst, fc_info.activation_info);
    }

    // Retain the tensorinfo with the weights to use
    if(_needs_weights_reshape || _needs_weights_conversion)
    {
        _trans_weights = *weights_to_use;
    }

    // Set auxiliary memory requirements
    auto gemm_mem_req = (_is_quantized_asymmetric) ? _mm_gemmlowp->workspace() : _mm_gemm->workspace();
    for(unsigned int i = 0; i < gemm_mem_req.size(); ++i)
    {
        _aux_mem[i] = gemm_mem_req[i];
    }

    if(_aux_mem[Pretranspose].size > 0)
    {
        // Release permuted weights at the end of prepare as they are further transposed by the assembly dispatch
        // Do not release them if biases are dynamic and data type is quantized, since the weights tensor will be used for biases offset calculation
        _aux_mem[TransposedWeights] = MemoryInfo(offset_int_vec(TransposedWeights), (_is_quantized_asymmetric
                                                                                     && biases && !(biases->are_values_constant())) ?
                                                 MemoryLifetime::Persistent :
                                                 MemoryLifetime::Prepare,
                                                 _reshaped_weights.total_size());
        _aux_mem[ConvertedWeights] = MemoryInfo(offset_int_vec(ConvertedWeights), MemoryLifetime::Prepare, _converted_weights.total_size());
    }
    else
    {
        _aux_mem[TransposedWeights] = MemoryInfo(offset_int_vec(TransposedWeights), _needs_weights_conversion ? MemoryLifetime::Prepare : MemoryLifetime::Persistent, _reshaped_weights.total_size());
        _aux_mem[ConvertedWeights]  = MemoryInfo(offset_int_vec(ConvertedWeights), MemoryLifetime::Persistent, _converted_weights.total_size());
    }
    _aux_mem[FlattenedSrc] = MemoryInfo(offset_int_vec(FlattenedSrc), MemoryLifetime::Temporary, _flattened_src.total_size());
}

prepare()

void prepare ( ITensorPack &  constants )

overridevirtual

Prepare the function for executing.

Any one off pre-processing step required by the function is handled here

Parameters

[in]

constants

Vector that contains the constants tensors.

Note

Prepare stage might not need all the function's buffers' backing memory to be available in order to execute

Reimplemented from INEOperator.

Definition at line 467 of file CpuFullyConnected.cpp.

References arm_compute::ACL_DST, arm_compute::ACL_SRC, arm_compute::ACL_SRC_1, ITensorPack::add_const_tensor(), Window::DimY, Scheduler::get(), CpuAuxTensorHandler::get(), ITensorPack::get_const_tensor(), ITensor::mark_as_unused(), arm_compute::offset_int_vec(), and IScheduler::schedule_op().

Referenced by CpuFullyConnected::run().

{
    if(!_is_prepared)
    {
        auto weights = tensors.get_const_tensor(ACL_SRC_1);

        CpuAuxTensorHandler reshaped_weights(offset_int_vec(TransposedWeights), _reshaped_weights, tensors, false);
        CpuAuxTensorHandler converted_weights(offset_int_vec(ConvertedWeights), _converted_weights, tensors, false);

        // Pointer to current weights
        const ITensor *cur_weights = weights;

        // Reshape of the weights (happens only once)
        if(_needs_weights_reshape)
        {
            // Run reshape weights kernel and mark weights as unused
            ITensorPack transpose_pack{ { ACL_SRC, weights }, { ACL_DST, reshaped_weights.get() } };
            NEScheduler::get().schedule_op(_transpose_weights.get(), Window::DimY, _transpose_weights->window(), transpose_pack);

            cur_weights->mark_as_unused();
            cur_weights = reshaped_weights.get();
        }

        // Convert weights if needed (happens only once)
        if(_needs_weights_conversion)
        {
            ITensorPack convert_pack{ { ACL_SRC, cur_weights }, { ACL_DST, converted_weights.get() } };
            _convert_weights->run(convert_pack);

            cur_weights->mark_as_unused();
            cur_weights = converted_weights.get();
        }

        ITensorPack gemm_pack = tensors;
        gemm_pack.add_const_tensor(ACL_SRC_1, cur_weights);

        // Prepare GEMM prepare and release unused weights
        if(!_is_quantized_asymmetric)
        {
            _mm_gemm->prepare(gemm_pack);
        }
        else
        {
            _mm_gemmlowp->prepare(gemm_pack);
        }

        _is_prepared = true;
    }
}

run()

void run ( ITensorPack &  tensors )

overridevirtual

Run the kernels contained in the function.

Parameters

[in]

tensors

Vector that contains the tensors to operate on.

Reimplemented from INEOperator.

Definition at line 433 of file CpuFullyConnected.cpp.

References arm_compute::ACL_DST, arm_compute::ACL_SRC, arm_compute::ACL_SRC_0, arm_compute::ACL_SRC_1, ITensorPack::add_const_tensor(), CpuAuxTensorHandler::get(), ITensorPack::get_const_tensor(), arm_compute::offset_int_vec(), CpuFullyConnected::prepare(), and arm_compute::test::validation::src.

{
    prepare(tensors);

    auto src = tensors.get_const_tensor(ACL_SRC_0);

    CpuAuxTensorHandler flattened_src(offset_int_vec(FlattenedSrc), _flattened_src, tensors, false);
    CpuAuxTensorHandler transformed_wei(offset_int_vec(_trans_weights_idx), _trans_weights, tensors, false);

    // Linearize src if it comes from a convolutional layer
    if(_is_fc_after_conv)
    {
        ITensorPack flatten_pack{ { ACL_SRC, src }, { ACL_DST, flattened_src.get() } };
        _flatten->run(flatten_pack);
    }

    ITensorPack gemm_pack = tensors;
    gemm_pack.add_const_tensor(ACL_SRC_0, (_is_fc_after_conv) ? flattened_src.get() : src);
    if(_needs_weights_reshape || _needs_weights_conversion)
    {
        gemm_pack.add_const_tensor(ACL_SRC_1, transformed_wei.get());
    }

    // Run matrix multiply
    if(_is_quantized_asymmetric)
    {
        _mm_gemmlowp->run(gemm_pack);
    }
    else
    {
        _mm_gemm->run(gemm_pack);
    }
}

validate()

Status validate ( const ITensorInfo *  src, const ITensorInfo *  weights, const ITensorInfo *  biases, const ITensorInfo *  dst, FullyConnectedLayerInfo  fc_info = FullyConnectedLayerInfo()  )

static

Static function to check if given info will lead to a valid configuration of CpuFullyConnected.

Similar to CpuFullyConnected

Returns

a status

Definition at line 341 of file CpuFullyConnected.cpp.

References ActivationLayerInfo::activation(), FullyConnectedLayerInfo::activation_info, ITensorInfo::are_values_constant(), FullyConnectedLayerInfo::are_weights_reshaped, ARM_COMPUTE_RETURN_ERROR_ON, ARM_COMPUTE_RETURN_ERROR_ON_DATA_TYPE_CHANNEL_NOT_IN, ARM_COMPUTE_RETURN_ERROR_ON_MISMATCHING_DATA_TYPES, ARM_COMPUTE_RETURN_ERROR_ON_NULLPTR, ARM_COMPUTE_RETURN_ON_ERROR, ARM_COMPUTE_UNUSED, ActivationLayerInfo::BOUNDED_RELU, Dimensions< T >::cbegin(), Dimensions< T >::cend(), ICloneable< T >::clone(), arm_compute::misc::shape_calculator::compute_flatten_shape(), arm_compute::misc::shape_calculator::compute_transposed_shape(), ITensorInfo::data_layout(), ITensorInfo::data_type(), ITensorInfo::dimension(), FullyConnectedLayerInfo::enable_fast_math, ActivationLayerInfo::enabled(), arm_compute::F16, arm_compute::F32, arm_compute::is_data_type_quantized(), ActivationLayerInfo::LU_BOUNDED_RELU, ITensorInfo::num_dimensions(), Dimensions< size_t >::num_max_dimensions, arm_compute::QASYMM8, arm_compute::QASYMM8_SIGNED, ActivationLayerInfo::RELU, FullyConnectedLayerInfo::retain_internal_weights, arm_compute::S32, arm_compute::test::validation::src, ITensorInfo::tensor_shape(), FullyConnectedLayerInfo::transpose_weights, CpuConvertFullyConnectedWeights::validate(), CpuTransposeKernel::validate(), CpuFlatten::validate(), and FullyConnectedLayerInfo::weights_trained_layout.

Referenced by CpuFullyConnected::configure(), and NEFullyConnectedLayer::validate().

{
    ARM_COMPUTE_UNUSED(fc_info.retain_internal_weights);
    ARM_COMPUTE_RETURN_ERROR_ON_NULLPTR(src, weights, dst);
    ARM_COMPUTE_RETURN_ERROR_ON_DATA_TYPE_CHANNEL_NOT_IN(src, 1, DataType::QASYMM8, DataType::QASYMM8_SIGNED, DataType::F16, DataType::F32);
    ARM_COMPUTE_RETURN_ERROR_ON_MISMATCHING_DATA_TYPES(src, weights, dst);
    ARM_COMPUTE_RETURN_ERROR_ON(weights->num_dimensions() > 2);
    ARM_COMPUTE_RETURN_ERROR_ON(fc_info.activation_info.enabled() && is_data_type_quantized(src->data_type()) && fc_info.activation_info.activation() != ActivationLayerInfo::ActivationFunction::RELU
                                && fc_info.activation_info.activation() != ActivationLayerInfo::ActivationFunction::BOUNDED_RELU && fc_info.activation_info.activation() != ActivationLayerInfo::ActivationFunction::LU_BOUNDED_RELU);
    ARM_COMPUTE_RETURN_ERROR_ON(!weights->are_values_constant() && (!fc_info.are_weights_reshaped || fc_info.transpose_weights));

    bool weights_reshaped = fc_info.transpose_weights ? fc_info.are_weights_reshaped : true;
    bool is_fc_after_conv = true;

    const ITensorInfo &flatten_src       = TensorInfo(src->clone()->set_is_resizable(true).reset_padding().set_tensor_shape(compute_flatten_shape(src)));
    const ITensorInfo &reshaped_weights  = TensorInfo(weights->clone()->set_is_resizable(true).reset_padding().set_tensor_shape(compute_transposed_shape(*weights)));
    const ITensorInfo &converted_weights = weights_reshaped ? TensorInfo(weights->clone()->set_is_resizable(true).reset_padding()) : TensorInfo(*reshaped_weights.clone());

    // With the Fully Connected layer we can have 4 different cases:
    //  1) Convolution layer -> Fully Connected layer without batches
    //  2) Fully Connected layer -> Fully Connected layer without batches
    //  3) Convolution layer -> Fully Connected layer with batches
    //  4) Fully Connected layer -> Fully Connected layer with batches

    const ITensorInfo *src_to_use     = src;
    const ITensorInfo *weights_to_use = weights;

    // Check if we have a fully connected layer with batches
    const bool is_batched_fc_layer = dst->dimension(1) > 1;

    if(biases != nullptr)
    {
        ARM_COMPUTE_RETURN_ERROR_ON(biases->num_dimensions() > 1);
        if(is_data_type_quantized(src->data_type()))
        {
            ARM_COMPUTE_RETURN_ERROR_ON_DATA_TYPE_CHANNEL_NOT_IN(biases, 1, DataType::S32);
        }
        else
        {
            ARM_COMPUTE_RETURN_ERROR_ON_MISMATCHING_DATA_TYPES(src, biases);
        }
    }

    if(is_batched_fc_layer)
    {
        is_fc_after_conv = (TensorShape::num_max_dimensions >= 4) && (std::equal(src->tensor_shape().cbegin() + 3,
                                                                                 src->tensor_shape().cend(),
                                                                                 dst->tensor_shape().cbegin() + 1));
    }
    else
    {
        is_fc_after_conv = src->num_dimensions() > 1;
    }

    if(!weights_reshaped)
    {
        // Validate reshape weights kernel
        ARM_COMPUTE_RETURN_ON_ERROR(kernels::CpuTransposeKernel::validate(weights, &reshaped_weights));
        weights_to_use = &reshaped_weights;
    }

    if(is_fc_after_conv && (src->data_layout() != fc_info.weights_trained_layout))
    {
        // Validate convert weights kernel
        ARM_COMPUTE_RETURN_ON_ERROR(CpuConvertFullyConnectedWeights::validate(weights_to_use,
                                                                              &converted_weights,
                                                                              src->tensor_shape(),
                                                                              fc_info.weights_trained_layout));
        weights_to_use = &converted_weights;
    }

    if(is_fc_after_conv)
    {
        // Fully Connected layer after a Convolution Layer without batches
        ARM_COMPUTE_RETURN_ERROR_ON((weights_to_use->dimension(1) != (src->dimension(0) * src->dimension(1) * src->dimension(2))));

        // Validate flatten kernel
        ARM_COMPUTE_RETURN_ON_ERROR(CpuFlatten::validate(src, &flatten_src));
        src_to_use = &flatten_src;
    }
    else
    {
        // Fully Connected layer after a Fully Connected Layer without batches
        ARM_COMPUTE_RETURN_ERROR_ON(src->dimension(0) != weights_to_use->dimension(1));
    }
    // Validate matrix multiply kernel
    ARM_COMPUTE_RETURN_ON_ERROR(validate_mm(src_to_use, weights_to_use, biases, dst, fc_info.activation_info, fc_info.enable_fast_math));

    return Status{};
}

workspace()

experimental::MemoryRequirements workspace ( ) const

overridevirtual

Return the memory requirements required by the workspace.

Reimplemented from INEOperator.

Definition at line 517 of file CpuFullyConnected.cpp.

{
    return _aux_mem;
}

---

The documentation for this class was generated from the following files:

• src/cpu/operators/CpuFullyConnected.h
• src/cpu/operators/CpuFullyConnected.cpp