1# Neural Network Runtime
2
3## Introduction
4
5Neural Network Runtime (NNRt) functions as a bridge to connect the upper-layer AI inference framework and bottom-layer acceleration chip, implementing cross-chip inference computing of AI models.
6
7As shown in Figure 1, NNRt opens Native APIs for the AI inference framework to access. Currently, NNRt interconnects with the built-in [MindSpore Lite](https://gitee.com/openharmony/third_party_mindspore) inference framework of the system. In addition, NNRt opens HDI APIs for device-side AI acceleration chips (such as NPUs and DSPs) to access the OpenHarmony hardware ecosystem. AI applications can directly use underlying chips to accelerate inference and computing through the AI inference framework and NNRt.
8
9NNRt and MindSpore Lite use MindIR unified intermediate representation to reduce unnecessary model conversion in the intermediate process, making model transfer more efficient.
10
11Generally, the AI application, AI inference engine, and NNRt are in the same process, and the chip driver runs in another process. The transmission of models and computing data between the two processes should be implemented by IPC. NNRt architecture implements the HDI client based on the HDI APIs. Accordingly, chip vendors need to implement and open the HDI services through HDI APIs.
12
13**Figure 1** NNRt architecture
14!["NNRt architecture"](./figures/neural_network_runtime.png)
15
16## Directory Structure
17
18```text
19/foundation/ai/neural_network_runtime
20├── common                         # Common functions
21├── figures                        # Images referenced by README
22├── example                        # Development samples
23│   ├── deep_learning_framework    # Application/Inference framework development samples
24│   └── drivers                    # Device driver development samples
25├── frameworks
26│   └── native                     # Framework code
27│       └── ops                    # Operator header files and implementation
28├── interfaces                     # APIs
29│   ├── innerkits                  # Internal APIs
30│   └── kits                       # External APIs
31└── test                           # Test cases
32    ├── system_test                # System test cases
33    └── unittest                   #  Unit test cases
34```
35
36## Compilation and Building
37
38In the root directory of the OpenHarmony source code, call the following command to compile NNRt separately:
39```shell
40./build.sh --product-name rk3568 --ccache --build-target neural_network_runtime --jobs 4
41```
42> **Note:**
43--product-name: product name, for example, <b>Hi3516DV300</b> and <b>rk3568</b>.
44--ccache: The cache function is used during compilation.
45--build-target: name of the compiled component.
46--jobs: number of compilation threads, which can accelerate compilation.
47
48## Description
49
50### API Description
51
52- [Native API reference](https://gitee.com/openharmony/docs/tree/master/zh-cn/application-dev/reference/apis-neural-network-runtime-kit)
53- [HDI API reference](https://gitee.com/openharmony/drivers_interface/tree/master/nnrt)
54
55### How to Use
56
57- For details about AI inference engine/application development, see Neural Network Runtime App Development Guide.
58- For details about how to develop AI acceleration chip drivers and devices, see Neural Network Runtime Device Development Guide.
59
60## Repositories Involved
61
62- [neural_network_runtime](https://gitee.com/openharmony/neural_network_runtime)
63- [third_party_mindspore](https://gitee.com/openharmony/third_party_mindspore)
64