This CL adds a check during SetUp that an NNAPI driver service is still
alive by pinging the driver service. If it is not alive, the test will
fail during SetUp. Without this quick-fail, the test case would continue
as if the driver were still active, which would result in multiple
EXPECT_* and ASSERT_* statements failing instead of a single, clear
failure message.
Bug: 197035200
Test: mma
Test: presubmit: VtsHalNeuralnetworks*TargetTest
Change-Id: Ib1b75ed20f764055699590581d5ad4e5aff4baae
Do not run validation on "inputs_as_internal" and
"all_tensors_as_inputs" variants.
Bug: 138149072
Bug: 149840439
Test: mma
Test: VtsHalNeuralnetworksV1_*TargetTest
Change-Id: I0699ed6703e48b6c4bc0e7a392b79c12770f04c9
Merged-In: I0699ed6703e48b6c4bc0e7a392b79c12770f04c9
(cherry picked from commit 678a10600c)
This CL removes a dependency on the VTS test runner by dynamically
discovering all NN HAL service instances in the gtest binary itself,
and runs through all service instances with parameterized tests.
This CL converts TEST_F cases to TEST_P cases, where the test parameter
is the name of the service instance. For existing TEST_P cases (such as
the generated test cases), the service instance name is made to be the
first test parameter.
This CL enables the NN VTS tests to be more portable, e.g., they can
run directly as a presubmit test.
Fixes: 124540002
Test: mma
Test: VtsHalNeuralnetworksV1_*TargetTest (with sample-all)
Test: cd $ANDROID_BUILD_TOP/hardware/interfaces/neuralnetworks && atest
Change-Id: I1e301d7c9f9342bb8f35a267bef180f510944b19
Merged-In: I1e301d7c9f9342bb8f35a267bef180f510944b19
(cherry picked from commit 7076f629b7)
This CL does additional NNAPI VTS test cleanup, including consolidating
duplicate functionality. Specifically, this CL:
* consolidates the createPreparedModel function, removing the duplicate
* consolidates the std::out ErrorStatus and DeviceStatus code into Utils
* changes non-null constant pointers to constant references
* removes redudant leading namespace specifiers (V1_0::, ::testing, etc.)
* makes the Valdiation tests free functions
* renames device to kDevice and mTestModel to kTestModel
Bug: N/A
Test: mma
Test: VtsHalNeuralnetworksV1_*TargetTest (with sample-all)
Change-Id: Ic401bb1f1760cc10384ac0d30c0c93409b63a9c7
Merged-In: Ic401bb1f1760cc10384ac0d30c0c93409b63a9c7
(cherry picked from commit e16af0a44b)
See change I2c0366fb87c96851fa6e0f8fe9ceac012d8e3513
Bug: 136097638
Test: m VtsHalNeuralnetworksV1_0TargetTest
Test: m VtsHalNeuralnetworksV1_1TargetTest
Test: m VtsHalNeuralnetworksV1_2TargetTest
Test: m VtsHalNeuralnetworksV1_2CompatV1_0TargetTest
Change-Id: I6fdede028422145d313d46532b5d2154ef0d40bc
Merged-In: I6fdede028422145d313d46532b5d2154ef0d40bc
(cherry picked from commit 2bcfdc82a0)
To make it easier to create the next version of NNAPI, this change
removes the following nonsensical dependence:
- NNAPI 1.0 VTS depends on NNAPI 1.1 and 1.2
- NNAPI 1.1 VTS depends on NNAPI 1.2
In particular, I made the following changes:
- split GeneratedTestHarness.cpp into three separate implementations,
- created a restricted version of Callbacks.h for 1.0 and 1.1,
- removed the dependency on frameworks/ml/nn/HalInterfaces.h,
- refactored Android.bp files for more autonomy between 1.0, 1.1, and 1.2,
- consolidated some common code into Utils.h,
- created structure for sharing code between VTS versions (VtsHalNeuralNetworksV1_0_utils).
Bug: 74827824
Bug: 124462414
Test: VtsHalNeuralnetworksV1_0TargetTest
Test: VtsHalNeuralnetworksV1_1TargetTest
Test: VtsHalNeuralnetworksV1_1CompatV1_0TargetTest
Test: VtsHalNeuralnetworksV1_2TargetTest
Test: VtsHalNeuralnetworksV1_2CompatV1_0TargetTest
Test: VtsHalNeuralnetworksV1_2CompatV1_1TargetTest
Change-Id: I4243d0b5e574255cef1070850f4d0a284f65f54e
Merged-In: I4243d0b5e574255cef1070850f4d0a284f65f54e
(cherry picked from commit 1d6b465997)
This CL adapts the VTS code to the corresponding changes made in the NN
utility library.
Bug: 128319484
Test: mma
Test: atest VtsHalNeuralnetworksV1_0Target
Test: atest VtsHalNeuralnetworksV1_1Target
Test: atest VtsHalNeuralnetworksV1_2Target
Change-Id: I470e8228cde2b75620ad851e4fe408f8e8329e7c
Merged-In: I470e8228cde2b75620ad851e4fe408f8e8329e7c
(cherry picked from commit 102e0442d8)
This CL adapts the VTS code to the corresponding changes made in the NN
utility library.
Bug: 119570067
Test: mma
Test: atest VtsHalNeuralnetworksV1_0Target
Test: atest VtsHalNeuralnetworksV1_1Target
Test: atest VtsHalNeuralnetworksV1_2Target
Change-Id: I7cbc1d7025c0352aa1ed29d71dc84c2fcfc20a4f
Merged-In: I7cbc1d7025c0352aa1ed29d71dc84c2fcfc20a4f
(cherry picked from commit e68668f65b)
- Instead of isCachingSupport returning a single boolean, switch to
getNumberOfCacheFilesNeeded returning the number of cache files. This
is to support use cases when driver needs more than one cache file for
each type, or when driver does not need data cache.
- Instead of a separate saveToCache, pass cache info along with
prepareModel_1_2 to save into cache as well as perform compilation.
This is to avoid a potential additional copy of cache files.
Bug: 123780248
Test: VtsHalNeuralnetworksV1_xTargetTest with 1.2 sample driver
Test: VtsHalNeuralnetworksV1_xTargetTest with a test driver that can
read and write cache entries
Change-Id: I921b7b8ccc3c66af19f6589f7213c6870d6f07bf
Merged-In: I921b7b8ccc3c66af19f6589f7213c6870d6f07bf
(cherry picked from commit b61ba1ed0b)
This is needed to be able to test DEQUANTIZE after adding
TENSOR_QUANT8_SYMM support.
Test: NeuralNetworksTest_static
Test: VtsHalNeuralnetworksV1_2TargetTest
Change-Id: Iba9b286df70919d7b67cd77c91e625a044bd686c
Merged-In: Iba9b286df70919d7b67cd77c91e625a044bd686c
(cherry picked from commit bf26a9e3d7)
Add the following tests for compilation caching:
- validation tests
- Test isCachingSupported
- Test prepareModelFromCache with invalid numFd and invalid access mode
- Test saveToCache with invalid numFd, invalid access mode,
invalid file size, and invalid fd offset
- execution test
- Save a mobilenet model to cache and then retrieve and run accuracy
evaluation.
- The same test but the file offsets for prepareModelFromCache is not at zero.
- security test
- CompilationCachingSecurityTest.CorruptedSecuritySensitiveCache
Randomly flip one bit of security-sensitive cache.
- CompilationCachingSecurityTest.WrongLengthSecuritySensitiveCache
Randomly append bytes to security-sensitive cache.
- CompilationCachingSecurityTest.WrongToken
Randomly flip one bit of cache token.
Bug: 119616526
Test: VtsHalNeuralnetworksV1_xTargetTest with 1.2 sample driver
Test: VtsHalNeuralnetworksV1_xTargetTest with a test driver that can
read and write cache entries
Change-Id: Iae9211cb28ce972b29572dfedd45d1ade4dfdaf5
Merged-In: Iae9211cb28ce972b29572dfedd45d1ade4dfdaf5
(cherry picked from commit 3405878e5e)
Test dynamic output shape with generated models when
- Dimensions of output operands are fully specified
- Dimensions of output operands are unspecified with sufficient buffer
- Dimensions of output operands are unspecified with insufficient buffer
Test: VTS on 1.2 sample driver
Change-Id: I4d26395ce443687ccbd47445b36e3356d70035cc
Merged-In: I4d26395ce443687ccbd47445b36e3356d70035cc
(cherry picked from commit 929fd21e06)
- Instead of reporting PASS for unsupported tests, use GTEST_SKIP to
skip the tests at runtime.
Bug: 113356629
Test: mm
Test: VTS tests on HVX driver
Change-Id: I6a870b61809e58490e66dd4ea36ddeb64fc68a07
Merged-In: I6a870b61809e58490e66dd4ea36ddeb64fc68a07
(cherry picked from commit bb685a4a97)
FastMessageQueue is a Treble-compliant data structure that enables fast
communication between two processes. The FMQ object itself is an atomic
circular buffer that is optionally synchronized with a futex. However,
FMQ has no notion of ownership or lifetime across processes, so it must
be paired with higher-level constructs to manage the lifetime and
ownership.
The NNAPI is introducing the notion of an "Execution Burst" object (or
more simply a "Burst" object), which is similar to an
ANeuralNetworksExecution, but is intended to be reused across multiple
executions and has lower IPC overheads. It achieves this low IPC
overhead by replacing HIDL HwBinder calls with FMQ messages.
Specifically, it replaces IPreparedModel::executeSynchronously's call
from the client into the service with fmq_sync<FmqRequestDatum> (an FMQ
channel used to pass a serialized Request object) and it replaces
the return from the service into the client with
fmq_sync<FmqResultDatum> (an FMQ channel used to return serialized
result status and OutputShapes information).
Each channel is a unidirectional flow of information with exactly one
producer and exactly one consumer. The channels are created by the NN
runtime and passed to the service via
IPreparedModel::configureExecutionBurst.
This CL tests the Burst in both the execution path and validation path
in the Vendor Test Suite (VTS) in neuralnetworks/1.*/vts/functional/.
The VTS binary--VtsHalNeuralnetworksV1_2TargetTest--can be built and run
as any previous version could.
Bug: 119570067
Test: mma
Test: VtsHalNeuralnetworksV1_2TargetTest
Change-Id: I3a36484eff9565c2d028c07c099804a0289f294a
Merged-In: I3a36484eff9565c2d028c07c099804a0289f294a
(cherry picked from commit 814d8372f3)
Bug: 119274127
Test: all of the following, with the appropriate android.hardware.neuralnetworks@1.${X}::IDevice/sample-all
VtsHalNeuralnetworksV1_0TargetTest
VtsHalNeuralnetworksV1_0TargetTest
VtsHalNeuralnetworksV1_1CompatV1_0TargetTest
VtsHalNeuralnetworksV1_1CompatV1_0TargetTest
VtsHalNeuralnetworksV1_1TargetTest
VtsHalNeuralnetworksV1_1TargetTest
VtsHalNeuralnetworksV1_2CompatV1_0TargetTest
VtsHalNeuralnetworksV1_2CompatV1_0TargetTest
VtsHalNeuralnetworksV1_2CompatV1_1TargetTest
VtsHalNeuralnetworksV1_2CompatV1_1TargetTest
VtsHalNeuralnetworksV1_2TargetTest
VtsHalNeuralnetworksV1_2TargetTest
Change-Id: Iedfa485b4008d9cec3b81ff4c0ce3ebc0b83c823
(cherry picked from commit 49e41678f5)
Create 1.2 version IPreparedModel, IPreparedModelCallback, and
IExecutionCallback.
Currently the new interfaces are created the same as 1.0 version,
but will have more methods introduced in later CLs.
Bug: 73506513
Test: VtsHalNeuralnetworksV1_xTargetTest with 1.2 sample driver
Change-Id: Icf4d04c22f88e825d87562f1489377fdf6bf585d
Merged-In: Icf4d04c22f88e825d87562f1489377fdf6bf585d
(cherry picked from commit b5cb8f7632)
This makes it easier to find all the places that need to be changed
after adding a new type to MixedTyped.
Test: VtsHalNeuralnetworksV1_2TargetTest
Change-Id: I92867de6574ec6dc1a17e30d889c79501ea93063
Merged-In: I92867de6574ec6dc1a17e30d889c79501ea93063
(cherry picked from commit 9b490f4833)
This is a copy the v1.1 tests since we don't have any new ops
implemented in v1.2 yet.
Bug: 114365802
Test: mm
Test: NNAPI VTS
Change-Id: Ida7525fcd3ae0fd6f88ff9591e06aba922bdae64
Merged-In: Ida7525fcd3ae0fd6f88ff9591e06aba922bdae64
(cherry-picked from 871be94770)
Set the acceptable error range based on both absolute tolerance and
relative tolerance.
Currently, absolute tolerance is set to 1e-5 for FP32 and 5 epsilon
(~5e-3) for FP16 relaxed computation. The relative tolerance is set to
5ULP of the corresponding precision. Add a TODO mark for potential
future adjustment on error limit based on testing.
Bug: 111768023
Test: none
Change-Id: Idedcec3e09fd7de9696811b93c81d0f180e896ef
A model can be prepared in different ways to optimize for different
use-cases. This CL propagates the execution preference across the HAL so
that the NN service can better fit the users needs.
Bug: 77864669
Test: mma
Test: NeuralNetworksTest_static
Test: VtsHalNeuralnetworksV1_1TargetTest
Merged-In: Ib928d510d462f36b6a87d5e81010513db7829fa8
Change-Id: Ib928d510d462f36b6a87d5e81010513db7829fa8
(cherry picked from commit 2504c2fe4f)
This CL adds validation tests for all of the existing generated models.
The strategy of this CL is this: given a valid model or request, make a
single change to invalidate the model or request, then verify that the
vendor service driver catches the inconsistency and returns
INVALID_ARGUMENT.
Bug: 67828197
Test: mma
Test: VtsHalNeuralnetworksV1_0TargetTest
Test: VtsHalNeuralnetworksV1_1TargetTest
Merged-In: I8efcdbdccc77aaf78992e52c1eac5c940fc81a03
Change-Id: I8efcdbdccc77aaf78992e52c1eac5c940fc81a03
(cherry picked from commit f76acd0312)
There are some NN VTS tests that assume a service is able to generate a
model consisting only of a floating point add operation. However, some
drivers do not support floating point operations. This CL relaxes the
test requirements to allow a test to be skipped if the service does not
support floating point add.
Bug: 72764145
Test: mma
Test: VtsHalNeuralnetworksV1_0TargetTest
Merged-In: I6b0644432680fc2f8098b5187795dc2953df03f9
Change-Id: I6b0644432680fc2f8098b5187795dc2953df03f9
(cherry picked from commit 4d5bb1097a)
Bug: 63911257
Test: mm
Test: VTS test pass on walleye
Merged-In: I41d9ea782262a894e7dbd00b782803c40b31c430
Change-Id: I41d9ea782262a894e7dbd00b782803c40b31c430
(cherry picked from commit 842e1a0331)
- Create VtsHalNeuralnetworksTest_utils module to be shared between
v1.x tests.
- Split the existing tests into two categories: basic, and generated.
- Created v1.1 VTS tests ensuring no regression in existing ML models.
Bug: 63911257
Test: mm
Test: NNAPI VTS tests pass on v1.0 and v1.1 sample drivers
Merged-In: Ic77c90a3a5bbd96b0ce2acd03764dde4b3034cc9
Change-Id: Ic77c90a3a5bbd96b0ce2acd03764dde4b3034cc9
(cherry picked from commit a2d04c828e)
Address comments in ag/2926388
Test: VtsHalNeuralnetworksV1_0TargetTest on Angler with sample driver
Bug: 67597726
Change-Id: Ie3118dcbfb11c25dbc84d2becdefbbae868ff28f
RequestArgument for optional inputs should have its .hasNoValue set
and other fields left empty. The test harness was incorrectly ignoring
the hasNoValue field and initializing the offset field even for missing
arguments. That inconsistency was caught by the sample driver and caused
an invalid argument error. With this CL the test harness now populates
requests correctly for tests with intentionally missing inputs, and that
fixes lstm* and lsh_projection.
Test: VtsHalNeuralnetworksV1_0TargetTest on Angler
Bug: 63905942
Change-Id: I8729124b720f0d4163f4b9efd452f82b825b0885
Some service drivers won't be able to support all operations. In those
cases, try to prepare the model anyway. If it works, proceed with the
test; if it fails, terminate the test early.
For example, the hvx nn service does not support svdf, so it should
attempt to prepare the model, but the test should not fail if the hvx
driver reports an error.
Bug: 63905942
Test: mm, expected behavior when running on hvx service
Change-Id: I51aa1911145f0d17590906c21314d2bdc24cffea
Some service drivers won't be able to support all operations. In those
cases, run the test as far as it can go, but ignore the output data.
For example, the hvx nn service does not support svdf, so it should
attempt to prepare and execute the model, but the test should not fail
if the hvx driver reports an error. However, hvx indicates it is able to
run MAX_POOL_2D, so the test should fail if that output is incorrect.
Bug: 63905942
Test: mm
Test: run vts binary on hvx service
Change-Id: I3ebc05836c8e070b66daa2e38b107f96637ca812
IEvent was a synchronization primitive which caused some confusion
in the interface. Originally the event object was paired with an
asynchronous task, and the asynchronous task would signal this event
when the corresponding output was ready to be used.
In the case of IDevice::prepareModel, the function call would return an
IPreparedModel object that was not guaranteed to be prepared until the
runtime had returned from waiting on the corresponding event object.
The event object has been changed to two explicit callbacks--
IPreparedModelCallback and IExecutionCallback. Now,
IDevice::prepareModel no longer returns an unfinished IPreparedModel;
instead, it will pass the IPreparedModel object to the runtime through
IPreparedModelCallback::notify. When the runtime retreives the
IPreparedModel object, the asynchronous task has already finished
preparing the model.
The two callbacks are used for different purposes. Each has its own
version of notify to pass the data back to the runtime:
* IPreparedModelCallback::notify(ErrorStatus, IPreparedModel)
* IExecutionCallback::notify(ErrorStatus)
Bug: 63905942
Test: mm, vts, ml/nn/runtime/tests
Change-Id: I0c88cd262ba762e0af15e9da31ebe813a5d150b2
NDK libc++ does not yet support full C++14. This CL replaced use of
std::get with type as well as auto lambda arguments with equivalent
C++11 constructs in the VTS test harness for NNAPI.
Test: VtsHalNeuralnetworksV1_0TargetTest
Bug: 63905942
Change-Id: If75e7c088e9221a70bcc47bc647e0dd7b045bfa1
Filtered out outputs that are not needed for correctness verification
during result validation.
Bug: 63905942
Test: VtsHalNeuralnetworksV1_0TargetTest on Angler
Change-Id: Ic967a821c2a60194c9d9f386601a865f5c409a4f
Tests like RNN needs slightly more relaxed tolerance than 4ULP.
Use 1e-5 from corresponding TFLite tests.
Bug: 63905942
Test: VtsHalNeuralnetworksV1_0TargetTest on Angler
Change-Id: I721711e13575f56e956d87e3ad537467f85e93f4
Created initial test harness for test models and examples from
NNAPI test generator in VtsHalNeuralnetworksV1_0TargetTest. As
an example, also added a test generated from test spec at
frameworks/ml/nn/tools/test_generator/tests/P_vts_full/.
Generated model setup code and examples are from:
frameworks/ml/nn/runtime/test/generated/examples and
frameworks/ml/nn/runtime/test/generated/vts_models respectively.
Bug: 63905942
Bug: 63525563
Test: VtsHalNeuralnetworksV1_0TargetTest with sample driver enabled
by cherry-pick
Change-Id: Ief029eed9718c8724ef0b64fc6a7f6b9a7bc7b7b
