Certain mutation testing -- mutateOperandLifeTimeTest and
mutateOperandInputOutputTest -- can introduce potentially very large
CONSTANT_COPY operands, which can in turn create potentially very
large Models which must be passed across binder. To avoid overflowing
the binder buffer, we estimate the size of the mutated Model, and skip
the test if that size is too high. The old logic recognizes that our
tests only have a single active binder transaction at a time, and
assumes that there are no other clients using the same service at the
same time, and so we should have the binder buffer to ourselves; to be
conservative, we reject any Model whose estimated size exceeds half
the binder buffer size. Unfortunately, sometimes the binder buffer
still overflows, because it unexpectedly contains an allocation from
some other transaction: It appears that binder buffer memory
management is not serialized with respect to transactions from our
tests, and therefore depending on scheduler behavior, there may be a
sizeable allocation still in the buffer when we attempt to pass the
large Model. To fix this problem we become even more conservative,
and instead of limiting the Model to half the binder buffer size, we
limit it to half IBinder.MAX_IPC_SIZE (the recommended transaction
size limit). To confirm that this change does not exclude too many
tests, I checked how may times the size filter function
exceedsBinderSizeLimit is called, how many times it rejects a model
under the new logic (modelsExceedHalfMaxIPCSize), and how many times
it rejects a model under the old logic (modelsExceedHalfMaxIPCSize).
Test: VtsHalNeuralnetworksV1_0TargetTest --gtest_filter=TestGenerated/ValidationTest.Test/*-*dsp*
Test: # models = 3592, modelsExceedHalfMaxIPCSize = 212, modelsExceedHalfBufferSize = 18
Test: VtsHalNeuralnetworksV1_1TargetTest --gtest_filter=TestGenerated/ValidationTest.Test/*-*dsp*
Test: # models = 7228, modelsExceedHalfMaxIPCSize = 330, modelsExceedHalfBufferSize = 28
Test: VtsHalNeuralnetworksV1_2TargetTest --gtest_filter=TestGenerated/ValidationTest.Test/*-*dsp*
Test: # models = 52072, modelsExceedHalfMaxIPCSize = 506, modelsExceedHalfBufferSize = 28
Test: VtsHalNeuralnetworksV1_3TargetTest --gtest_filter=TestGenerated/ValidationTest.Test/*-*dsp*
Test: # models = 73342, modelsExceedHalfMaxIPCSize = 568, modelsExceedHalfBufferSize = 28
Test: VtsHalNeuralnetworksTargetTest
Bug: 227719657
Bug: 227719752
Bug: 231928847
Bug: 238777741
Bug: 242271308
Change-Id: I3f81d71ca3c0ad4c639096b1dc034a8909bc8971
- detect cycle (CycleTest)
- detect bad execution order (mutateExecutionOrderTest)
- detect lifetime inconsistent with whether operand is written (mutateOperandLifeTimeTest)
- detect lifetime inconsistent with Model inputIndexes/outputIndexes (mutateOperandInputOutputTest)
- detect incorrect number of consumers (mutateOperandNumberOfConsumersTest)
- detect operand written multiple times (mutateOperandAddWriterTest)
- detect operand never written (mutateOperationRemoveWriteTest)
Bug: 66478689
Test: VtsHalNeuralnetworksV1_*TargetTest
Change-Id: Id4ba19660bbd31a16f8a675f7b6437f4d779e8da
This CL also cleans up the validation framework code.
Bug: 67828197
Test: mma
Test: VtsHalNeuralnetworksV1_*TargetTest
Change-Id: I84661fb2b8204148788d10425ca0ac986158b15f
This CL does additional NNAPI VTS test cleanup, including consolidating
duplicate functionality. Specifically, this CL:
* consolidates the createPreparedModel function, removing the duplicate
* consolidates the std::out ErrorStatus and DeviceStatus code into Utils
* changes non-null constant pointers to constant references
* removes redudant leading namespace specifiers (V1_0::, ::testing, etc.)
* makes the Valdiation tests free functions
* renames device to kDevice and mTestModel to kTestModel
Bug: N/A
Test: mma
Test: VtsHalNeuralnetworksV1_*TargetTest (with sample-all)
Change-Id: Ic401bb1f1760cc10384ac0d30c0c93409b63a9c7
To make it easier to create the next version of NNAPI, this change
removes the following nonsensical dependence:
- NNAPI 1.0 VTS depends on NNAPI 1.1 and 1.2
- NNAPI 1.1 VTS depends on NNAPI 1.2
In particular, I made the following changes:
- split GeneratedTestHarness.cpp into three separate implementations,
- created a restricted version of Callbacks.h for 1.0 and 1.1,
- removed the dependency on frameworks/ml/nn/HalInterfaces.h,
- refactored Android.bp files for more autonomy between 1.0, 1.1, and 1.2,
- consolidated some common code into Utils.h,
- created structure for sharing code between VTS versions (VtsHalNeuralNetworksV1_0_utils).
Bug: 74827824
Bug: 124462414
Test: VtsHalNeuralnetworksV1_0TargetTest
Test: VtsHalNeuralnetworksV1_1TargetTest
Test: VtsHalNeuralnetworksV1_1CompatV1_0TargetTest
Test: VtsHalNeuralnetworksV1_2TargetTest
Test: VtsHalNeuralnetworksV1_2CompatV1_0TargetTest
Test: VtsHalNeuralnetworksV1_2CompatV1_1TargetTest
Change-Id: I4243d0b5e574255cef1070850f4d0a284f65f54e
Create 1.2 version IPreparedModel, IPreparedModelCallback, and
IExecutionCallback.
Currently the new interfaces are created the same as 1.0 version,
but will have more methods introduced in later CLs.
Bug: 73506513
Test: VtsHalNeuralnetworksV1_xTargetTest with 1.2 sample driver
Change-Id: Icf4d04c22f88e825d87562f1489377fdf6bf585d
A model can be prepared in different ways to optimize for different
use-cases. This CL propagates the execution preference across the HAL so
that the NN service can better fit the users needs.
Bug: 77864669
Test: mma
Test: NeuralNetworksTest_static
Test: VtsHalNeuralnetworksV1_1TargetTest
Merged-In: Ib928d510d462f36b6a87d5e81010513db7829fa8
Change-Id: Ib928d510d462f36b6a87d5e81010513db7829fa8
(cherry picked from commit 2504c2fe4f)
This CL adds validation tests for all of the existing generated models.
The strategy of this CL is this: given a valid model or request, make a
single change to invalidate the model or request, then verify that the
vendor service driver catches the inconsistency and returns
INVALID_ARGUMENT.
Bug: 67828197
Test: mma
Test: VtsHalNeuralnetworksV1_0TargetTest
Test: VtsHalNeuralnetworksV1_1TargetTest
Merged-In: I8efcdbdccc77aaf78992e52c1eac5c940fc81a03
Change-Id: I8efcdbdccc77aaf78992e52c1eac5c940fc81a03
(cherry picked from commit f76acd0312)
