first commit

doc/about.md

@@ -0,0 +1,101 @@
+
+here are important instructions for VideoPipe if you want to figure out how it works!
+
+## Core parts in video structured application ## 
+`video structured` is a process which converts unstructure data (video here) into structured data. unstructure data:
+- video
+- image
+- audio
+- nature text
+
+and structure data mainly includes something like json, xml or data table in DataBase which can by processed directly by machine (program).
+
+Specifically in terms of video, the process of structured mainly involves these core parts:
+1. `read stream`. capture video stream from network or local machine.
+2. `decode`. decode byte stream to frames, algorithm/procedure can only act on images.
+3. `inference`. deep learning work on images, detect, classification or feature extraction.
+4. `track`. track on objects in video.
+5. `behaviour analysis` (optional). analysis on objects' tracks.
+6. `osd`. on screen display, show results on images for debug purpose or intuitive effects.
+7. `message broker`. push structured data to external.
+8. `encode`. encode frames which contains results to byte stream for transfer/serialization purpose.
+9. `push stream`. push byte stream to external or save it directly.
+
+<p align="center">
+<img src="./p23.png">
+</p>
+<div align="center">figure 1. core parts in video structured application. </div>
+
+each core part in `video structured` corresponding to one type of plugin in `VideoPipe`, namely **`Node`** in code.
+
+
+## Node in VideoPipe ##
+one `Node` in VideoPipe responsible for single task such as decoding or inference. we put many nodes together to construct a pipe, and let video data flow through the whole pipeline. every `Node` has 2 queues inside, one is for caching data from upstream nodes and another one is for caching data waiting for being pushed to downstream nodes. we can write logic code between the 2 queues, they are typical `producer-consumer` pattern.
+
+<p align="center">
+<img src="./p24.png">
+</p>
+<div align="center">figure 2. what Node looks like? </div>
+
+by default, producer and consumer work with single thread inside node, we need write async code when deal with complex tasks (for example, pushing data is a time-consuming operation in `vp_message_broker_node`) to avoid blocking the pipeline.
+
+<p align="center">
+<img src="./p25.png">
+</p>
+<div align="center">figure 3. async task inside Node. </div>
+
+there are 3 types of `Node` in VideoPipe, namely:
+- `SRC Node`. source node where data was created (only 1 queue inside used for caching data being pushed to downstream nodes).
+- `MID Node`. middle node where data would be handled.
+- `DES Node`. destination node where data disappears (only 1 queue inside used for caching data from upstream nodes).
+
+each `Node` `itself` has the ability to merge multi upstream nodes, and split into multi downstream nodes as well. note that `Node` use shallow-copy and copy Equally when data transfered from one node to other nodes by default, if you need deep-copy or want to transfer data by channel index (just hope data unconfused), add a `vp_split_node` at the point of spliting which would get different behaviour.
+<p align="center">
+<img src="./p28.png">
+</p>
+<div align="center">figure 4. merge & split in VideoPipe. </div>
+
+## Data flow in VideoPipe ##
+video (frame here) is a type of heavyweight data, so deep copying frequently would decrease the performance of pipeline. actually data transfered between 2 nodes in VideoPipe use `smart pointers` by default, once data was created by source nodes, the data content would NOT be copyed later at most time in the whole pipeline (but we can specify deep-copy if we need, using `vp_split_node` for instance).
+
+<p align="center">
+<img src="./p26.png">
+</p>
+<div align="center">figure 5. how data flows in VideoPipe. </div>
+
+video consist of continuous frames, VideoPipe handle these frames One by One, so the `frame index` in frame meta would increase continuously as well.
+
+
+## Hooks in VideoPipe ##
+hook is a mechanism which let host notify listeners when something happens, VideoPipe support hooks as well. pipeline invokes callback functions (via `std::function` object) to communicate with external code, such as export `fps`, `latency` and other status of pipeline itself. we should NOT block the callback functions when writing custom code inside it.
+
+<p align="center">
+<img src="./p27.png">
+</p>
+<div align="center">figure 6. hooks in VideoPipe. </div>
+
+hooks help to debug with our application and quickly find the bottleneck in whole pipe, visualization tool `vp_analysis_board` works depend on hooks.
+
+
+## Implement new Node type in VideoPipe ##
+`vp_node` is the base class for all nodes in VideoPipe. we can define a new node class derived from `vp_node` and override some virtual functions like `handle_frame_meta` and `handle_control_meta`.
+- `handle_frame_meta`. handle frame data flowing current node.
+- `handle_control_meta`. handle control data flowing current node.
+
+<p align="center">
+<img src="./p29.png">
+</p>
+<div align="center">figure 7. override virtual functions in custom Node. </div>
+
+frame data means `vp_frame_meta` in VideoPipe, contains data related to frame such as `frame index`, `data buffer`, `original width`. control data means `vp_control_meta` in VideoPipe, contains data related command such as `record video`, `record image`.
+
+note, NOT all data flowing current node should be handled using new logic, they just pass through if no operations work on them. we just need handle what we are interested in. 
+
+## Hardware Acceleration in VideoPipe ##
+some operations in `video structured` applications can benefit from hardware such as GPUs/NPUs. for example, video encoding on GPUs have higher speed/performace than CPUs. VideoPipe support hardware acceleration for these parts:
+
+- decode/encode. based on HARD decode/encode gstreamer plugins, [look more](https://github.com/sherlockchou86/video_pipe_c/blob/master/doc/env.md#about-hardware-acceleration).
+- inference. no doubt about it.
+- osd. need implement based on hardware acceleration SDKs by yourself.
+
+it is important to note that, although VideoPipe support hardware acceleration for above logic, `they could NOT share memory between each others`. it means that data will be copyed from GPU to CPU or CPU to GPU over and over again, which is the biggest disadvantage compared to other similar SDKs such as DeepStream.

doc/env.md

@@ -0,0 +1,101 @@
+
+
+## Personal Development Environment ##
+
+- VS Code for Windows 11
+- Ubuntu 18.04 x86_64 / C++17  / GCC 7.5 / GTX 1080 GPU
+- GStreamer 1.14.5 / OpenCV 4.6
+---------
+
+Install GStreamer (1.14.5 for Ubuntu 18.04 by default):
+```
+apt-get install libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav gstreamer1.0-tools gstreamer1.0-x gstreamer1.0-alsa gstreamer1.0-gl gstreamer1.0-gtk3 gstreamer1.0-qt5 gstreamer1.0-pulseaudio libgstrtspserver-1.0-dev gstreamer1.0-rtsp
+```
+
+Install OpenCV from source with `gstreamer` ON (CUDA optional). download source code of OpenCV 4.6.0 (with extra contrib modules) from github first, put them at the same directory then run `cmake` and `make` command:
+
+```
+step 1:
+cd `the path of opencv 4.6.0`
+mkdir build && cd build
+```
+
+```
+step 2:
+cmake -D CMAKE_BUILD_TYPE=RELEASE \
+-D CMAKE_INSTALL_PREFIX=/usr/local \
+-D WITH_TBB=ON \
+-D ENABLE_FAST_MATH=1 \
+-D CUDA_FAST_MATH=1 \
+-D WITH_CUBLAS=1 \
+-D WITH_CUDA=ON \
+-D BUILD_opencv_cudacodec=OFF \
+-D WITH_CUDNN=ON \
+-D OPENCV_DNN_CUDA=ON \
+-D CUDA_ARCH_BIN=6.1 \
+-D WITH_V4L=ON \
+-D WITH_QT=OFF \
+-D WITH_OPENGL=ON \
+-D WITH_GSTREAMER=ON \
+-D OPENCV_GENERATE_PKGCONFIG=ON \
+-D OPENCV_PC_FILE_NAME=opencv.pc \
+-D OPENCV_ENABLE_NONFREE=ON \
+-D OPENCV_EXTRA_MODULES_PATH=../../opencv_contrib-4.6.0/modules \
+-D INSTALL_PYTHON_EXAMPLES=OFF \
+-D INSTALL_C_EXAMPLES=OFF \
+-D BUILD_EXAMPLES=OFF ..
+```
+
+```
+step 3:
+make -j8
+```
+
+---------
+`VcXsrv` for screen display from remote machine to local desktop in case of using SSH terminal.
+
+- first install PC client from: https://sourceforge.net/p/vcxsrv/wiki/Home/
+- then run `export DISPLAY=local_ip:0.0` (or add it to ~/.bashrc) on remote machine (linux server or embedded board)
+
+---------
+Maybe you need install nginx with `http-rtmp-module` as rtmp server for debug purpose (other tools such as `ZLMediaKit` works fine **which was highly recommended**). Also, maybe you need a rtsp server from which we can receive rtsp stream for debug purpose.
+
+- [how to install ZLMeidaKit](https://github.com/ZLMediaKit/ZLMediaKit/wiki/vcpkg%E6%96%B9%E5%BC%8F%E5%AE%89%E8%A3%85zlmediakit)
+- [how to push stream to ZLMediaKit](https://github.com/ZLMediaKit/ZLMediaKit/wiki/ZLMediaKit%E6%8E%A8%E6%B5%81%E6%B5%8B%E8%AF%95), a few samples which used `vp_rtmp_des_node` in VidepPipe would push rtmp stream to server.
+- [how to pull/play stream from ZLMediaKit](https://github.com/ZLMediaKit/ZLMediaKit/wiki/%E6%92%AD%E6%94%BEurl%E8%A7%84%E5%88%99), VLC player or ffplay was recommended.
+
+---------
+Please prepare kafka server AND install `librdkafka` client sdk if you want to enable kafka-related codes.
+- [how to install kafka server](https://kafka.apache.org/quickstart)
+- how to install `librdkafka`? run `sudo apt-get install librdkafka-dev`
+
+
+## tips ##
+- Use shared_ptr/make_shared in whole project, do not use new/delete.
+- The pipeline is driven by stream data, if your app is not responding, maybe no stream input.
+
+
+## about Hardware Acceleration ##
+Since decode & encode in VideoPipe depend on gstreamer (encapsulated inside opencv), if you want to use your GPUs/NPUs to accelerate decoding and encoding performace, you need get/install HARD decode or HARD encode `gstreamer plugins` correctly first and modify gst launch string (take `vp_file_des_node` for example):
+```cpp
+appsrc ! videoconvert ! x264enc bitrate=%d ! mp4mux ! filesink location=%s
+```
+to
+```
+appsrc ! videoconvert ! nvv4l2h264enc bitrate=%d ! mp4mux ! filesink location=%s
+```
+the plugin `x264enc` use CPUs to encode video stream, but `nvv4l2h264enc`(comes from DeepStream SDK) use GPUs instread. if you use other platforms other than NVIDIA, you need Corresponding Hardware Acceleration plugins.
+
+**soft/hard decode example**
+```
+gst-launch-1.0 filesrc location=./face.mp4 ! qtdemux ! h264parse ! avdec_h264 ! videoconvert ! autovideosink    // decode by avdec_h264 use CPUs
+gst-launch-1.0 filesrc location=./face.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! videoconvert ! autovideosink // decode by nvv4l2decoder use NVIDIA GPUs
+```
+
+**soft/hard encode example**
+```
+gst-launch-1.0 filesrc location=./face.mp4 ! qtdemux ! h264parse ! avdec_h264 ! x264enc ! h264parse ! flvmux ! filesink location=./new_face.flv    // encode by x264enc use CPUs
+gst-launch-1.0 filesrc location=./face.mp4 ! qtdemux ! h264parse ! avdec_h264 ! nvv4l2h264enc ! h264parse ! flvmux ! filesink location=./new_face.flv  // encode by nvv4l2h264enc use NVIDIA GPUs
+```
+[source code of hard decode/encode gstreamer plugins for NVIDIA](https://gitlab.freedesktop.org/gstreamer/gstreamer/-/tree/main/subprojects/gst-plugins-bad/sys/nvcodec).(developed by community, open source), we could also use decode/encode plugins from [DeepStream SDK](https://docs.nvidia.com/metropolis/deepstream/6.0/dev-guide/text/DS_Quickstart.html) which maintained by NVIDIA but closed source.
+