{"id":863,"date":"2022-11-25T16:23:40","date_gmt":"2022-11-25T09:23:40","guid":{"rendered":"https:\/\/dgway.com\/blog_E\/?p=863"},"modified":"2022-11-25T16:23:42","modified_gmt":"2022-11-25T09:23:42","slug":"super-resolution-on-kv260","status":"publish","type":"post","link":"https:\/\/dgway.com\/blog_E\/2022\/11\/25\/super-resolution-on-kv260\/","title":{"rendered":"Super-resolution on KV260"},"content":{"rendered":"\n<p>The Kria KV260* is one of Xilinx edge devices designed to be suitable for Vision AI based applications.To implement customized AI applications on KV260, Xilinx Vitis AI** is used as a development stack to develop AI inference. It consists of optimized IP, tools, libraries, example designs, and free pre-trained models.&nbsp;<\/p>\n\n\n\n<p>For super resolution application, Vitis AI model zoo provides a pre-trained model of the very deep residual channel attention networks (RCAN***), a popular image super-resolution (SR) architecture. This blog introduces steps to deploy the RCAN model on kv260 which are preparing xmodel and implementing on KV260.<\/p>\n\n\n\n<p>Xilinx Vitis AI is used as a development stack which consists of optimized IP, tools, libraries, models, and example designs to develop AI inference on Xilinx hardware platforms. According to <a rel=\"noreferrer noopener\" href=\"https:\/\/www.xilinx.com\/content\/dam\/xilinx\/support\/documents\/sw_manuals\/vitis_ai\/2_5\/ug1414-vitis-ai.pdf\" target=\"_blank\">Vitis AI User Guide<\/a>, this blog introduces steps to deploy RCAN model on kv260 which are preparing xmodel and implementing on KV260.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Preparing xmodel<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">(0) Test Environment<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Python3.7 and Pytorch 1.13<\/li>\n\n\n\n<li>Docker with Vitis AI 2.5 <br>Installation instructions is provided as follow <a rel=\"noreferrer noopener\" href=\"https:\/\/github.com\/Xilinx\/Vitis-AI#getting-started\" target=\"_blank\">https:\/\/github.com\/Xilinx\/Vitis-AI#getting-started<\/a>.<\/li>\n<\/ul>\n\n\n\n<p>There are some different settings for CPU\/GPU docker in the preparing xmodel step. Please follow these changes according to your docker version.<\/p>\n\n\n\n<p>For Docker CPU version<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>remove CUDA_HOME environment variable setting as follows: <code>unset CUDA_HOME<\/code><\/li>\n\n\n\n<li>Pass argument <code>--cpu<\/code> when running python script<\/li>\n\n\n\n<li>Pass <code>map_location=torch.device('cpu')<\/code> to <code>torch.load()<\/code> in python script<\/li>\n\n\n\n<li>Pass <code>device=torch.device('cpu')<\/code> to <code>QatProcessor()<\/code> in python script<\/li>\n<\/ul>\n\n\n\n<p>For Docker GPU version<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Set <code>export CUDA_HOME=\/usr\/local\/cuda<\/code><\/li>\n\n\n\n<li>Set <code>CUDA_VISIBLE_DEVICES=0<\/code> (according to ID of GPU in use)<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">(1) Prepare training data<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Download model from model zoo : <a rel=\"noreferrer noopener\" href=\"https:\/\/www.xilinx.com\/bin\/public\/openDownload?filename=pt_OFA-rcan_DIV2K_360_640_45.7G_2.5.zip\" target=\"_blank\">pt-OFA-rcan_DIK2K_360_640_45.7G_2.5.zip<\/a><\/li>\n\n\n\n<li>Prepare a dataset<\/li>\n<\/ul>\n\n\n\n<p>Download DIV2K dataset : <a rel=\"noreferrer noopener\" href=\"https:\/\/idealo.github.io\/image-super-resolution\/tutorials\/training\/\" target=\"_blank\">https:\/\/idealo.github.io\/image-super-resolution\/tutorials\/training\/<\/a><br>Download benchmark : <a rel=\"noreferrer noopener\" href=\"https:\/\/cv.snu.ac.kr\/research\/EDSR\/benchmark.tar\" target=\"_blank\">https:\/\/cv.snu.ac.kr\/research\/EDSR\/benchmark.tar<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\">(2) Evaluate floating model<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Launch the docker image<\/li>\n\n\n\n<li>Install vai_q_pytorch as follow <a rel=\"noreferrer noopener\" href=\"https:\/\/docs.xilinx.com\/r\/2.0-English\/ug1414-vitis-ai\/Installing-vai_q_pytorch\" target=\"_blank\">https:\/\/docs.xilinx.com\/r\/2.0-English\/ug1414-vitis-ai\/Installing-vai_q_pytorch<\/a><\/li>\n\n\n\n<li>Run <code>run_test.sh<\/code> shell script<\/li>\n<\/ul>\n\n\n\n<p>Output<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"791\" height=\"244\" src=\"https:\/\/dgway.com\/blog_E\/wp-content\/uploads\/2022\/11\/EvaluationResult.png\" alt=\"\" class=\"wp-image-864\" srcset=\"https:\/\/dgway.com\/blog_E\/wp-content\/uploads\/2022\/11\/EvaluationResult.png 791w, https:\/\/dgway.com\/blog_E\/wp-content\/uploads\/2022\/11\/EvaluationResult-300x93.png 300w, https:\/\/dgway.com\/blog_E\/wp-content\/uploads\/2022\/11\/EvaluationResult-768x237.png 768w\" sizes=\"auto, (max-width: 791px) 100vw, 791px\" \/><figcaption class=\"wp-element-caption\">Figure 1 : Evaluation result<\/figcaption><\/figure>\n<\/div>\n\n\n<h3 class=\"wp-block-heading\">(3) Train model<\/h3>\n\n\n\n<p>To train the model with the DIV2K dataset, run <code>run_train.sh<\/code> shell script.<br>The model parameters can be adjusted by editing the run_train.sh as desired.<br><code>--scale<\/code> : scaling factor that matches your training dataset. In this case, <code>scale<\/code> is set to be 2.<br><code>--n_resgroups<\/code> : The number of residual groups. In this case, <code>n_resgroups<\/code> is set to be 1.<br><code>--epochs<\/code> :The number of epochs can be changed in order to decrease training time or increase PSNR. In this case, <code>epochs<\/code> is set to be 250.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">(4) Quantize<\/h3>\n\n\n\n<p>The floating-point model is quantized and calibrated by using Vitis AI Quantizer, run <code>run_qat.sh<\/code> shell script.<\/p>\n\n\n\n<p>Model parameter<\/p>\n\n\n\n<p><code>--scale<\/code> : In this case, <code>scale<\/code> is set to be 2.<br><code>--n_resgroups<\/code> : In this case, <code>n_resgroups<\/code> is set to be 1.<br><code>--epochs<\/code> : In this case, epochs is set to be 250.<\/p>\n\n\n\n<p>After calibration, <code>Model_0_int.xmodel<\/code> are generated as DPU deployable model.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">(5) Compile the model<\/h3>\n\n\n\n<p>Vitis AI compiler is used to compile the DPU deployable model by calling vai_c_xir as follows, <\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">vai_c_xir \\\n  --xmodel      ..\/snapshot\/OFA_RCAN_QAT\/model\/qat_result\/Model_0_int.xmodel \\\n  --arch        \/opt\/vitis_ai\/compiler\/arch\/DPUCZDX8G\/KV260\/arch.json \\\n  --net_name    OFA_RCAN_QAT \\\n  --output_dir  ..\/snapshot\/OFA_RCAN_QAT\/OFA_RCAN_QAT\n<\/pre>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"354\" src=\"https:\/\/dgway.com\/blog_E\/wp-content\/uploads\/2022\/11\/CompilingResult-1024x354.png\" alt=\"\" class=\"wp-image-865\" srcset=\"https:\/\/dgway.com\/blog_E\/wp-content\/uploads\/2022\/11\/CompilingResult-1024x354.png 1024w, https:\/\/dgway.com\/blog_E\/wp-content\/uploads\/2022\/11\/CompilingResult-300x104.png 300w, https:\/\/dgway.com\/blog_E\/wp-content\/uploads\/2022\/11\/CompilingResult-768x266.png 768w, https:\/\/dgway.com\/blog_E\/wp-content\/uploads\/2022\/11\/CompilingResult.png 1226w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption class=\"wp-element-caption\">Figure 2 : Compiling result<\/figcaption><\/figure>\n<\/div>\n\n\n<p>OFA_RCAN_QAT.xmodel, md5sum.txt and meta.json are saved as the output of the compiling process. These files are used to implement the RCAN model on edge devices.<\/p>\n\n\n\n<p><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Implementing on KV260<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">(1) Set up the board<\/h3>\n\n\n\n<p>Download the SD Card Image (xilinx-kv260-dpu-v2022.1-v2.5.0.img.gz) <br><a rel=\"noreferrer noopener\" href=\"https:\/\/www.xilinx.com\/member\/forms\/download\/design-license-xef.html?filename=xilinx-kv260-dpu-v2022.1-v2.5.0.img.gz\" target=\"_blank\">https:\/\/www.xilinx.com\/member\/forms\/download\/design-license-xef.html?filename=xilinx-kv260-dpu-v2022.1-v2.5.0.img.gz<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\">(2) Move model file to board<\/h3>\n\n\n\n<figure class=\"wp-block-image size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/dgway.com\/blog_E\/wp-content\/uploads\/2022\/11\/Picture2-2-1024x243.png\" alt=\"\" class=\"wp-image-899\" width=\"603\" height=\"143\" srcset=\"https:\/\/dgway.com\/blog_E\/wp-content\/uploads\/2022\/11\/Picture2-2-1024x243.png 1024w, https:\/\/dgway.com\/blog_E\/wp-content\/uploads\/2022\/11\/Picture2-2-300x71.png 300w, https:\/\/dgway.com\/blog_E\/wp-content\/uploads\/2022\/11\/Picture2-2-768x183.png 768w, https:\/\/dgway.com\/blog_E\/wp-content\/uploads\/2022\/11\/Picture2-2.png 1258w\" sizes=\"auto, (max-width: 603px) 100vw, 603px\" \/><\/figure>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Build RCAN model <\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-preformatted\">[edge]# cd .\/Vitis-AI\/examples\/Vitis-AI-Library\/samples\/rcan \n[edge]# bash -x build.sh\n<\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Run <code>test_jpeg_rcan<\/code><br>Output image is saved at .\/Vitis-AI\/examples\/Vitis-AI-Library\/samples\/rcan as 0_DG_sample_result.jpg<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-preformatted\">[edge]# .\/test_jpeg_rcan RCAN_GPU DG_sample.jpg<\/pre>\n\n\n\n<p>Result<\/p>\n\n\n\n<figure class=\"wp-block-table aligncenter is-style-regular\"><table><tbody><tr><td class=\"has-text-align-left\" data-align=\"left\">Input image: DG_sample.jpg<br><img loading=\"lazy\" decoding=\"async\" width=\"300\" height=\"169\" class=\"wp-image-866\" style=\"width: 300px;\" src=\"https:\/\/dgway.com\/blog_E\/wp-content\/uploads\/2022\/11\/DG_sample.jpg\" alt=\"\" srcset=\"https:\/\/dgway.com\/blog_E\/wp-content\/uploads\/2022\/11\/DG_sample.jpg 640w, https:\/\/dgway.com\/blog_E\/wp-content\/uploads\/2022\/11\/DG_sample-300x169.jpg 300w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><br><\/td><td class=\"has-text-align-left\" data-align=\"left\">Result : 0_DG_sample_result.jpg<br><img loading=\"lazy\" decoding=\"async\" width=\"500\" height=\"281\" class=\"wp-image-867\" style=\"width: 500px;\" src=\"https:\/\/dgway.com\/blog_E\/wp-content\/uploads\/2022\/11\/0_DG_sample_result.jpg\" alt=\"\" srcset=\"https:\/\/dgway.com\/blog_E\/wp-content\/uploads\/2022\/11\/0_DG_sample_result.jpg 1280w, https:\/\/dgway.com\/blog_E\/wp-content\/uploads\/2022\/11\/0_DG_sample_result-300x169.jpg 300w, https:\/\/dgway.com\/blog_E\/wp-content\/uploads\/2022\/11\/0_DG_sample_result-1024x576.jpg 1024w, https:\/\/dgway.com\/blog_E\/wp-content\/uploads\/2022\/11\/0_DG_sample_result-768x432.jpg 768w\" sizes=\"auto, (max-width: 500px) 100vw, 500px\" \/><br><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\">(3) Test Performance<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Send images.lst (list of image file names) and images folder to target<\/li>\n\n\n\n<li>Run test performance command<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-preformatted\">[edge]# .\/test_performance_rcan RCAN_GPU images.lst<\/pre>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"212\" src=\"https:\/\/dgway.com\/blog_E\/wp-content\/uploads\/2022\/11\/Performance-1024x212.png\" alt=\"\" class=\"wp-image-868\" srcset=\"https:\/\/dgway.com\/blog_E\/wp-content\/uploads\/2022\/11\/Performance-1024x212.png 1024w, https:\/\/dgway.com\/blog_E\/wp-content\/uploads\/2022\/11\/Performance-300x62.png 300w, https:\/\/dgway.com\/blog_E\/wp-content\/uploads\/2022\/11\/Performance-768x159.png 768w, https:\/\/dgway.com\/blog_E\/wp-content\/uploads\/2022\/11\/Performance.png 1160w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption class=\"wp-element-caption\">Figure 3 : Performance of RCAN_GPU model<\/figcaption><\/figure>\n<\/div>\n\n\n<p>For more information, please refer to document UG1414 and UG1354 from Xilinx.<br><a rel=\"noreferrer noopener\" href=\"https:\/\/docs.xilinx.com\/r\/2.0-English\/ug1414-vitis-ai\/Vitis-AI-Overview\" target=\"_blank\">https:\/\/docs.xilinx.com\/r\/2.0-English\/ug1414-vitis-ai\/Vitis-AI-Overview<\/a><br><a rel=\"noreferrer noopener\" href=\"https:\/\/docs.xilinx.com\/r\/en-US\/ug1354-xilinx-ai-sdk\/Model-Samples\" target=\"_blank\">https:\/\/docs.xilinx.com\/r\/en-US\/ug1354-xilinx-ai-sdk\/Model-Samples<\/a><\/p>\n\n\n\n<p>* For further information about the Kria KV260, please refer to&nbsp;<a rel=\"noreferrer noopener\" href=\"https:\/\/www.xilinx.com\/products\/som\/kria\/kv260-vision-starter-kit.html\" target=\"_blank\">https:\/\/www.xilinx.com\/products\/som\/kria\/kv260-vision-starter-kit.html<\/a><br>** For further information about the RCAN model, please refer to <a rel=\"noreferrer noopener\" href=\"https:\/\/github.com\/yulunzhang\/RCAN\" target=\"_blank\">https:\/\/github.com\/yulunzhang\/RCAN<\/a><br>*** For further information of Vitis AI, Please follow <a rel=\"noreferrer noopener\" href=\"https:\/\/dgway.com\/blog_E\/2022\/10\/03\/accelerator-systems-for-ai\/\" target=\"_blank\">https:\/\/dgway.com\/blog_E\/2022\/10\/03\/accelerator-systems-for-ai\/<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>The Kria KV260* is one of Xilinx edge devices designed to be suitable for Vision AI based applications.To implement customized AI applications on KV260, Xilinx Vitis AI** is used as a development stack to develop AI inference. It consists of optimized IP, tools, libraries, example designs, and free pre-trained models.&nbsp; For super resolution application, Vitis AI model zoo provides a&#46;&#46;&#46;<\/p>\n","protected":false},"author":1,"featured_media":909,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[17],"tags":[],"class_list":["post-863","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-fpga-ai"],"_links":{"self":[{"href":"https:\/\/dgway.com\/blog_E\/wp-json\/wp\/v2\/posts\/863","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dgway.com\/blog_E\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dgway.com\/blog_E\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dgway.com\/blog_E\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dgway.com\/blog_E\/wp-json\/wp\/v2\/comments?post=863"}],"version-history":[{"count":34,"href":"https:\/\/dgway.com\/blog_E\/wp-json\/wp\/v2\/posts\/863\/revisions"}],"predecessor-version":[{"id":910,"href":"https:\/\/dgway.com\/blog_E\/wp-json\/wp\/v2\/posts\/863\/revisions\/910"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/dgway.com\/blog_E\/wp-json\/wp\/v2\/media\/909"}],"wp:attachment":[{"href":"https:\/\/dgway.com\/blog_E\/wp-json\/wp\/v2\/media?parent=863"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dgway.com\/blog_E\/wp-json\/wp\/v2\/categories?post=863"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dgway.com\/blog_E\/wp-json\/wp\/v2\/tags?post=863"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}