Setting the D5M Terasic Camera using Nios II at 1920x1080

Written by Holguer Andres

Requirements:

• Quartus Version Above or equal to 14.1
• D5M Camera.
• RAM (Where we can store the video acquisition).
• HDMI/VGA/SDI, or others video outputs (to stream out the video in order to see the results).

This manual is composed by two sections:

• Hardware description part.
  In this part you can learn how to create a video acquisition system using Qsys.
• Software part.
  In this part you will learn how to communicate with the D5M camera using Nios II through the "Audio and Video config" core.

Hardware

1. Depending on the Quartus Version that you might have, you should download from the following link the “Altera University Program IP Cores”.
   *note: If you are using a version above 16.1 you don’t need to install this.
2. Once you have downloaded the version according to your Quartus version, you must install it on the root directory according the version of Quartus.
3. The core we are going to work with is the “Audio and video config core”, therefore it would be useful to download the PDF manual that is located on the same page from where you have downloaded the Installer of the Altera university program.
4. Now depending on the board you are using, you should use the “System builder” to create the initial template. However for sake of time, in this manual we are going to use the Cyclone V GX starter kit to illustrate the basic steps(Generated file).
   
5. After generating the basic template for this exercise, open Quartus, and open the template.
   
   
6. Now download the following Qsys template (basic_nios.qsys), or use your own in which you have a Nios II.
7. Copy this file to the directory of the template.
   
   
8. Open Qsys, and open the template file<basic_nios.qsys> or you qsys file in which you will find the following:
   
   
   
   
9. Now for the video acquisition it is vital to have a ram in which we can store the video acquisition coming from the camera. As the Cyclone V GX Starter Kit has a LPDDR2 ram we are going to add this ram controller to this Qsys file. Nevertheless, it is important to know that the RAM controller specifications for the video acquisition may vary depending on the board you are using; therefore, you might have to change this one to port the qsys system to other board of your interest.
   
   So let’s start by searching on the IP library the LPDDR2 SDRAM Controller, and add it.
   
   
10. Once it is added, you have to configure the RAM controller with the following settings:
    
    
    
    
    
    
    
    
    
    
    
    
    
11. Once you have configured the RAM controller, you must connect it as follows:
    

    

12. Download the following subQsys System "video_d5m_input.qsys", and paste it on the project directory, which contains all the necessary cores for the camera acquisition.
13. Save the project.
14. Now go to the menu "File->Refresh System"
    
15. After refreshing the system, you can search on the IP library the Subsytem with the name "video_d5m_input5", add it, and rename it with the name "camera"
    
    
    
    
    
16. If you want to explore this subsystem you can press right click on "camera", and press "Drill into system", and to go back to the top system you can press the highlighted button.
     
    
    
    
    as you can see this subsystem has the core "Audio and Video Config", which is used to configure the D5M through I2C, you can double click to explore it, and you will find the following configurations (in the section software, these configurations are properly explained):
    
    
17. Now it is time to add a “clocked video output” , and configure it as follows:
    
    
    
    
    
    
    
18. Now it is time to interconnect all the modules we have added to the qsys design as follows:
    
19. Once everything is connected, we are going to download another Qsys Subsytem "plls_camera_hdmi.qsys", which contains 2 PLLs, one for the camera which has to work at 25MHz, and the other one for the HDMI interface that works at 148.5 MHz.
    You have to copy this subsystem in the directory of the project.
20. Save the Qsys project.
21. Press F5 to refresh the System.
22. Now you can search the module called "plls_camera_hdmi", add it to the Qsys system, and rename it with the name "video_clocks"
    
    
    
    as the subsytem "video_d5m_input", you can drill into this module to explore what is inside it.
    
    
23. Connect the added "video_clocks" as follows
    
24. Now that the Qsys design has been completed, go to the menu System->Assign Base Addresses
    
25. Save the design.
26. Go to the menu Generate-> Generate HDL, and generate the Qsys system
    
    
    
27. For this design, while the Qsys system is generated, we will get this message
    
    
    

    It is important to know that after the synthesize you must execute this tcl file, so that you don’t get any timing issues with the RAM. However, if you are using other kind of RAM controller you might have to avoid this point.

28. Once the generation is complete, you get the following message, you have to press close, and close the Qsys.
    
    
    
    
29. Now go back to Quartus and press on the tap “Files” on the window “Project navigator”, then right click on the folder files, press “add/remove files in project”
    
    
    
    and add the file “basic_nios.qip” to the project
    
    
    
30. Now create a folder on the project directory, and name it “modules”
    
    
    
    
31. Download the following Verilog files and paste them within the folder “Modules”
    
    
    
    
32. Now add them to the Quartus project
    
    
    
33. Now open the file “Line_Buffer”, and make sure that the line 91 looks like this:
    
34. Open the file “CCD_Capture”, and make sure that the line 81 looks like this:
    
35. Now perform double click on the Top module of the hierarchy  “D5M_Example”, and instantiate the added modules.
    
    

    
    //=======================================================
    //  REG/WIRE declarations
    //=======================================================
    
    wire reset_n;
    reg		[11:0]	rCCD_DATA;
    reg				rCCD_LVAL;
    reg				rCCD_FVAL;
    wire	[11:0]	mCCD_DATA;
    wire			mCCD_DVAL;
    wire			mCCD_DVAL_d;
    wire	[15:0]	X_Cont;
    wire	[15:0]	Y_Cont;
    wire	[31:0]	Frame_Cont;
    wire			DLY_RST_0;
    wire			DLY_RST_1;
    wire			DLY_RST_2;
    wire			DLY_RST_3;
    wire			DLY_RST_4;
    wire	[11:0]	sCCD_R;
    wire	[11:0]	sCCD_G;
    wire	[11:0]	sCCD_B;
    wire			sCCD_DVAL;
    wire auto_start;
    //=======================================================
    //  Structural coding
    //=======================================================
    
    assign reset_n = 1'b1;
    assign D5M_RESET_N=DLY_RST_1;
    assign D5M_TRIGGER=1'b1;
     assign	LEDG		=	Y_Cont;
    
    //auto start when power on
    assign auto_start = ((KEY[0])&&(DLY_RST_3)&&(!DLY_RST_4))? 1'b1:1'b0;
    
    //D5M read 
    always@(posedge D5M_PIXCLK)
    begin
    	rCCD_DATA	<=	D5M_D;
    	rCCD_LVAL	<=	D5M_LVAL;
    	rCCD_FVAL	<=	D5M_FVAL;
    end
    
    
    //Reset module
    Reset_Delay			u2	(	.iCLK(CLOCK_50_B5B),
    							.iRST(KEY[0]),
    							.oRST_0(DLY_RST_0),
    							.oRST_1(DLY_RST_1),
    							.oRST_2(DLY_RST_2),
    							.oRST_3(DLY_RST_3),
    							.oRST_4(DLY_RST_4)
    						);
    //D5M image capture
    CCD_Capture			u3	(	
    							.oDATA(mCCD_DATA),
    							.oDVAL(mCCD_DVAL),
    							.oX_Cont(X_Cont),
    							.oY_Cont(Y_Cont),
    							.oFrame_Cont(Frame_Cont),
    							.iDATA(rCCD_DATA),
    							.iFVAL(rCCD_FVAL), 
    							.iLVAL(rCCD_LVAL),
    							.iSTART(KEY[1]|auto_start),
    							.iEND(!KEY[1]),
    							.iCLK(~D5M_PIXCLK),
    							.iRST(DLY_RST_2)
    						);
    //Bayer to RGB
    RAW2RGB				u4	(	
    							.iCLK(~D5M_PIXCLK),
    							.iRST_n(DLY_RST_1),
    							.iData(mCCD_DATA),
    							.iDval(mCCD_DVAL),
    							.oRed(sCCD_R),
    							.oGreen(sCCD_G),
    							.oBlue(sCCD_B),
    							.oDval(sCCD_DVAL),
    							.iX_Cont(X_Cont),
    							.iY_Cont(Y_Cont)
    						);
    // HDMI Config - HDMI I2C
    I2C_HDMI_Config u27 ( 
    .iCLK(CLOCK_50_B5B),
    .iRST_N(reset_n),
    .I2C_SCLK(I2C_SCL),
    .I2C_SDAT(I2C_SDA),
    .HDMI_TX_INT(HDMI_TX_INT) 
    );
    

36. Now it is time to instantiate the Qsys system on the top module, for that you have to go the Project Navigator and select “Files”, and then expand the file “basic_nios.qip”, select the file “basic_nios.v”, perform right click, and press on “Create Verilog Instantiation…”
    
    
37. After this, a new file should appear on the project path, with the name of “basic_nios_inst.v”, open it and copy the instantiation.
    
    
    
38. Once copied, you must paste it on the top Verilog module, below the already instantiated modules() as follows.
    
    

    
    basic_nios basic_nios_inst
    (
    	//system clocks and reset
    	.clk_clk(CLOCK_50_B6A) ,	// input  clk_clk_sig
    	.reset_reset_n(DLY_RST_2), 	// input  reset_reset_n_sig
       .clk_0_clk                  (CLOCK_50_B6A),                                                  //                                clk_0.clk
       .reset_0_reset_n            (DLY_RST_2),                                            //                              reset_0.reset_n
       .clk_1_clk                  (CLOCK_50_B7A),                                                  //                                clk_1.clk
       .reset_1_reset_n            (DLY_RST_2),                                            //                              reset_1.reset_n
       .video_clocks_d5m_clk_clk   (D5M_XCLKIN),                                   //                 video_clocks_d5m_clk.clk
       .video_clocks_hdmi_clk_clk  (HDMI_TX_CLK) ,  
    	
    	
    	//LPDDR2 SDRAM Interface
    	.memory_mem_ca    (DDR2LP_CA),    // memory.mem_ca
       .memory_mem_ck    (DDR2LP_CK_p),    //       .mem_ck
       .memory_mem_ck_n  (DDR2LP_CK_n),  //       .mem_ck_n
       .memory_mem_cke   (DDR2LP_CKE),   //       .mem_cke
       .memory_mem_cs_n  (DDR2LP_CS_n),  //       .mem_cs_n
       .memory_mem_dm    (DDR2LP_DM),    //       .mem_dm
       .memory_mem_dq    (DDR2LP_DQ),    //       .mem_dq
       .memory_mem_dqs   (DDR2LP_DQS_p),   //       .mem_dqs
       .memory_mem_dqs_n (DDR2LP_DQS_n), //       .mem_dqs_n
       .oct_rzqin        (DDR2LP_OCT_RZQ),         //    oct.rzqin
       .mem_if_lpddr2_emif_0_pll_ref_clk_clk          (CLOCK_50_B5B),
       .mem_if_lpddr2_emif_0_status_local_init_done   (lpddr2_local_init_done),   // mem_if_lpddr2_emif_status.local_init_done
       .mem_if_lpddr2_emif_0_status_local_cal_success (lpddr2_local_cal_success), //                          .local_cal_success
       .mem_if_lpddr2_emif_0_status_local_cal_fail    (lpddr2_local_cal_fail),    //                          .local_cal_fail
    
    	
    	 //camera input
       .camera_alt_vip_cti_0_clocked_video_vid_clk                   (D5M_PIXCLK),
       .camera_alt_vip_cti_0_clocked_video_vid_data                  ({sCCD_R[11:4],sCCD_G[11:4],sCCD_B[11:4]}),
       .camera_alt_vip_cti_0_clocked_video_vid_datavalid             (rCCD_LVAL),
       .camera_alt_vip_cti_0_clocked_video_vid_f                     (1'b0),
       .camera_alt_vip_cti_0_clocked_video_vid_h_sync                (rCCD_LVAL),
       .camera_alt_vip_cti_0_clocked_video_vid_locked                (DLY_RST_3),
       .camera_alt_vip_cti_0_clocked_video_vid_v_sync                (rCCD_FVAL),
    	//camera config
    	.camera_d5m_config_external_interface_SDAT (D5M_SDATA) ,	// inout  d5m_camera_config_external_interface_SDAT_sig
    	.camera_d5m_config_external_interface_SCLK (D5M_SCLK) ,	// output  d5m_camera_config_external_interface_SCLK_sig
    	
     
    	//Video output
    	.alt_vip_itc_0_clocked_video_vid_clk		(HDMI_TX_CLK) ,	// input  alt_vip_itc_0_clocked_video_vid_clk_sig
    	.alt_vip_itc_0_clocked_video_vid_data		(HDMI_TX_D[23:0]) ,	// output [23:0] alt_vip_itc_0_clocked_video_vid_data_sig
    	.alt_vip_itc_0_clocked_video_vid_datavalid(HDMI_TX_DE) ,	// output  alt_vip_itc_0_clocked_video_vid_datavalid_sig
    	.alt_vip_itc_0_clocked_video_vid_v_sync	(HDMI_TX_VS) ,	// output  alt_vip_itc_0_clocked_video_vid_v_sync_sig
    	.alt_vip_itc_0_clocked_video_vid_h_sync	(HDMI_TX_HS) 	// output  alt_vip_itc_0_clocked_video_vid_h_sync_sig
    
    
    );
    

    
    
    
39. Now it is time to Synthesize, for that perform double click on window “Tasks”, specifically on the section “Analysis and Synthesis”.
    *Note: DO NOT COMPILE THE ENTIRE PROJECT.
    
    
    
    
40. Once the synthesis is finished, and recalling the point 23 in which the Qsys Console gave us a message that said “Run the tcl script after synthesis and before fitting”, we need to go to the menu “Tools->Tcl Scripts…”, and look for the tcl file we want to run, and then just press on “run”.
    
    
    
    
    
41. The after running the TCL file, we are free to close the TCL scripts window, and then double click on “Assembler(Generate Programing files)” on the window “Tasks”
    
    
    
    
    
    
    
42. Finally we have finished the Hardware part, you can now download the <>.sof file to see the camera working on the HDMI, and then you can continue with the software part in which we are going to configure the camera exposure through the Nios II using the “Audio and Video Config” module.
    
43. In addition, you can download the Hardware version from here, only as a guidance in case you have errors.
    

    The Hardware Solution you can see it running on the following video:
    
     

Software

1. Making sure you have completed the hardware section, and that your system works as you could see on the youtube video, it is time to write C code for the Nios II to change the Camera parameters such as Exposure, Camera color gains, among others.
   For this purpose it is important to Download the Datasheet of the camera that you can find on the following link.
2. As we are using the “Audio and Video Config” Core, it would be useful to download the datasheet of this core as well. However, we are going to pore through the most important parts of the camera datasheet for this manual.
3. Open the camera datasheet, and go to the page 10, and take a look at the table 1.7, and 1.8.
   these tables are divided into different columns, and some of them have below the title a number like this , those number are addresses that have to be set to the respective values to achieve certain resolutions and frame rates.
4. In our particular case, the resolution at we are using the D5M camera is 1920x1080 which you can find on the table 1.8.
   
5. By using the “Audio and Video Config” we can modify these registers to configure the camera as we whish.
   *Note: there are other registers that you can find in detail in the page 11, chapter 2.
6. For this example, we are going to write code to modify the different registers of the D5M camera.
7. Open the “Nios II XX.X Software Build Tools for Eclipse”, and select the workspace “fpgas”.
   
8. Now on the menu go to “File->Nios II application and BSP from template”.
   
9. Select the SOPC (basic_nios. sopcinfo) File that you can find in the hardware project directory and contains the specifications for the Nios II. Then, name the Project as “Camera”, select the Project Template “Blank Project”, and press finish.
   
   
   
10. Once you press finish, on the project explorer there should be two folders, one with the name “Camera”, and the other with the name “Camera_bsp”.
    
11. Perform right click on the folder “Camera_bsp” and go to “Nios II->BSP Editor…”
    
12. On settings go to common, and select “make“ and change the bsp optimization to -O2, then press Generate and exit.
    
    
    
    
13. Now perform right click on the folder “Camera”, press on properties, then select “Nios II Application Properties”, and change the Optimization Level to 2.
    
    
    
14. Now right click on the folder “Camera”, and create a new “Source file”, and name it “main_camera.c”.
    
    
15. Now copy the following code on the created file “main_camera.c”, save it, and analyze the code, comparing it with the D5M Datasheet, and the mentioned registers on the table 1.7, and 1.8.
    

    /*
    Written by Holguer Andres Becerra
    for more go to www.fpgalover.com
    
    */
    #include <stdio.h>
    #include <stdlib.h>
    #include <system.h>
    #include "io.h"
    #include "altera_up_avalon_audio_and_video_config.h"
    
    
    
    #define D5M_COLUMN_SIZE	1919
    #define D5M_ROW_SIZE	1079
    #define D5M_COLUMN_BIN	0x0000
    #define D5M_ROW_BIN		0x0000
    #define EXPOSURE_LIMIT 1079
    
    //Registers Addresses
    unsigned char address[25] = { 0x00, 0x20, 0x09, 0x05, 0x06, 0x0A, 0x2B, 0x2C,
    		0x2D, 0x2E, 0x10, 0x11, 0x12, 0x10, 0x98, 0xA0, 0xA1, 0xA2, 0x01, 0x02,
    		0x03, 0x04, 0x22, 0x23, 0x49 };
    //Registers Values
    unsigned short data[25] = { 0x0000, 0xc000, EXPOSURE_LIMIT, 0x0000, 0x0019, 0x8000,
    		0x000b, 0x000f, 0x000f, 0x000b, 0x0051, 0x1807, 0x0002, 0x0053, 0x0000,
    		0x0000, 0x0000, 0x0FFF, 0x0036, 0x0010, D5M_ROW_SIZE, D5M_COLUMN_SIZE, D5M_ROW_BIN, D5M_COLUMN_BIN,
    		0x01A8 };
    
    
    
    int i;
    
    void init_d5m_camera(alt_up_av_config_dev * d5m) {
    	d5m->type = TRDB_D5M_CONFIG;
    	d5m->type = TRDB_D5M_CONFIG;
    	for (i = 0; i < 25; i++) {
    		alt_up_av_config_write_D5M_cfg_register(d5m, address[i], data[i]);
    	}
    }
    void set_d5m_exposure(alt_up_av_config_dev * d5m,unsigned short exposure){
    	alt_up_av_config_write_D5M_cfg_register(d5m, 0x09, exposure);
    }
    
    int main() {
    	alt_up_av_config_dev * d5m= alt_up_av_config_open_dev("/dev/camera_d5m_config");
    	init_d5m_camera(d5m);
    
    	unsigned short exposure = 0;
    	while (1) {
    		//exposure
    		set_d5m_exposure(d5m,exposure);
    		exposure += 100;
    		if(exposure>EXPOSURE_LIMIT*2){
    			exposure=0;
    		}
    
    		printf("exposure %d\n", exposure);
    		usleep(100000);
    	}
    
    	return 0;
    }
    
    
    ​

16. Right click on the folder “Camera”, and build project.
    
17. If everything was successful, you will have inside the folder “Camera” a new file called “Camera.elf”
    
18. Go to Quartus, open the programmer and download the .sof file to the FPGA.
19. Go back to Eclipse, right click on the folder “Camera”, and press “Run As-> Nios II Hardware”
    
    
20. You can see the results on the following youtube video.
    
    
    

Attachments:

basic_nios	[Qsys Template]	31 kB
Solution hw	[Solution]	949 kB
System builder Template	[System Builder Template]	7 kB
PLLs	[PLLs for camera and HDMI]	26 kB
TRDB-D5M_Hardware specification_V0.2.pdf	[D5M Datasheet]	921 kB
Camera Input Subsystem	[Camera Qsys Input Subsystem]	9 kB