USB-UART control port

As part of my GSoC assignment, I tried to expose the control port via uart, in addition to control port via usb which was already present. I could not get it to work that time. So now I took the job again. Instead on working on my previous code, I decided to start with a clean slate.
My approach was to mux the already available resource which was being used for usb control port. After many hours of understanding the simulation waveforms, I finally managed to get things working, umm partially.
There are two kinds of commands. One which cause some changes for example forcing output video to be in grayscale, changing to HDMI0 etc. These are currently working in my uart control port. Then there are commands which output status back to the terminal. Now these currently don't work and I can get them working. The only problem will be that then the tools/fpga_debug will have to be written again for uart which will be plenty of work to do before sprints starts.

Need for a remote access to FPGA

My mid sems finally got over yesterday and I got some free time in my life. So I am blogging. After completion of GSoC, lot of my juniors are asking me about my  project and a few seem to be interested. Few days back I answered a question on quora : http://www.quora.com/What-are-some-suggested-open-source-projects-in-Verilog-for-intermediates/answer/Ajit-Mathew and the person asking the question PMed me regarding my project. But what is stopping them from contributing is that they don't have an access to FPGA. In India, an Atlys board costs around INR 20K which is a big deal for a student. Also colleges have the old spartan 3 boards or don't give easy access to FPGA board.
I was recently talking to an alumni who is working in a FPGA based startup and he mentioned to me how they have created a hack which allows the developers to remotely access the FPGA, dump the .bit and see results. If we at TimVideos can develop a similar solution then it would improve participation greatly. If the logistics are handled, making this possible is not very challenging. Another solution can be to make a comprehensive test bench which allows black box and white box testing. A good language to create this test bench will be System Verilog as it combine verilog and class structure of C++. But this is a very big project. Creating accurate models for verification will take a lot of time.

The need to have expensive hardware for development is big disadvantage for open source hardware development. 

Thank You TimVideos!

It is slightly late but better late than never. My final evaluation results are out and I have been given a pass grade by my organisation. It felt really good to see the mail of successful completion of GSoC after three months of hard work.
I would be very ungrateful if I don't thank my mentor, Joel "Shenki" Stanley. He is probably the coolest mentor and I have dibs on him if I participate next year. Initially I was worried as I didn't have a clear understanding of what to do and in those times shenki was really cool and motivated me which was a big help. I would also like to than our org admin Tim "mithro" Ansell, whose dedication to this org inspires me. Seriously, I would love to sit and talk with him on how he manages to motivate himself to learn and do so many stuff (BTW he works at Google).
I would like to keep contributing for the organisation as and when I get some free time. I also see a lot of my junior getting inspired and wanting to work for this org (one of them had sent a mail on the mailing list some time ago). I am planning to start a OSDG-Hardware group and introduce them to TimVideos. I see a lot of student interested to contribute but don't have sufficient knowledge about things relates to FPGA. So I will be soon (read "when I am free") posting some material and links which will help a raw student who knows only a HDL to become a "hardware developer". Another problem which I found when I discussed with my junior about open source hardware development is the need to have "the hardware" which for us in out case is the FPGA (very expensive considering a country like India). We can slightly mitigate this problem by developing a test suite which helps students who don't have hardware.
The journey of GSoC was great. I have was voted as the most hardworking GSoC intern among my friends who were also interning at GSoC orgs. My learning was really awesome. I feel smarter than most of my batch mates (which will slowly fade away when mid semester exams start). Also the exposure to open source community has restored my faith in humanity because good people still exist and that too in great number. Overall, this experience will remain green in my mind forever.

GSOC Final Report

Result Summary

Timvideos is trying to develop an opensource video conferencing solution. The organisation has both a hardware component (HDMI2USB) and software component (like Gstreamer, flumotion). I was working on the hardware side of the org specifically on HDMI2USB fpga firmware. HDMI2SUSB is basically a solution based on FPGA to compress HD video stream input from a source like video camera and output the compressed stream via USB, hence the name HDMI2USB.

GOALS

My  project was on "Optimisation of MPEG Core". In the original firmware, the output stream from USB was about 10-11 fps which is very low and not suitable for video recording, the primary purpose of HDMI2USB. The aim of my project was to improve this frame rate. I had proposed to improve it to 60 fps @ 720p. As the reason for a slow frame rate was not fully understood, I built a lot of firmware to get the "status" of the HDMI2USB which later on became my second project, to create a "Debug Infrastructure" for HDMI2USB.

Final Outcome

Finally I was able to improve the frame rate to 30 fps (which is the minimum required for recording) and set a system which allows developers to easily output debug data via CDC or UART Port.

How did the frame rate improve?

The title of my project "Optimisation of MPEG Core" isn't a very apt title. The work I did tried to optimize the whole firmware. The original firmware has an image buffer which stores a frame before being compress by the JPEG core. The state machine of the image buffer was such that no processing was done when a frame was being read and no frames were read when processing was done. This was the bottleneck which was causing a slow frame rate. To remove this bottleneck I pipelined the read and write cycles of the image buffer. This reduced the number of frames which were being dropped and hence improve the frame rate.

Debug Interface

The debug interface is a new feature added to the HDMI2USB firmware. Unlike programming language which can be easily debugged using printf statement, systems on hardware like FPGA require elaborate methods. The debug interface which has been implemented allows data to be output via CDC port which was part of the original firmware and UART port which has been added to allow more options to developers and users incase CDC fails. Also a program has been included which reads bytes from fpga and presents them to users in a human readable format.

Heart Beat Feature

Sometimes when the image is static, it is difficult to tell whether the fpga firmware is working or it has hung. To remove this ambiguity, a heart beat feature was added which is basically a small block of pixel in the bottom right corner of the frame pulsing at a constant rate. This feature can be turned on/off using a hardware switch and via the CDC control port.

Community Outcomes

The Timvideos developers’ community has promising outcomes after completion of this project. The biggest being the HDMI2USB can be used for recording. Now that HDMI2USB firmware streams at 30 fps, it can be used for recording at 30 fps which is the minimum frame rate required for video recording. Also, I may not have been able to meet the 60 fps as I mentioned in my proposal but I have successfully revealed some issues in the HDMI2USB firmware which need to be dealt with, most importantly the bandwidth of FX2 chip.

There are certain other features which can be implemented but have not been implemented due to lack of time like reducing subsampling to 4:2:0 and implementing a triple buffer in the image buffer. This may or may not change the frame rate but it will be interesting to see the outcomes if they are implemented. The debug infrastructure that has been setup will allow future developers to easily debug the hardware and also provide an easy way of troubleshooting for future users. Addition of UART allows multiple debug/control option incase cypress fx2 fails.  As my project dealt more with optimisation which involves getting a good understanding of the system as a whole, I have gained good knowledge about the fpga firmware. This will be useful for future developers as I am in a position to be able to guide them regarding specific issues. Also thanks to the exposure I received due to this project, I have started a Open Source Hardware Developers Group in my university where soon I will be introducing TimVideos. Hopefully this will increase the number of developers for HDMI2USB.

Replicating Results

All the code relevant to the project has been merged into the HDMI2USB repository.

Also check out the how to use section in the following blog posts:

• UART
• Heart Beat
• Debug Interface
• To check the frame rate  

Bugs/To Do

• Support 1080p via USB: I successfully added 1080p test pattern (Hidef snow) to the hdmi2usb. It was streaming out of the HDMI port but not out of USB. Here is the video.
• Add Control Port via UART: I was able to expose some features of control port to UART but not all. I will try to finish after GSOC.
• Automatic Encoding Quality Control: It has been noticed that the encoding quality depends highly on the type of image. So there can be a auto encoding qulaity control like the one in youtube (it changes the resolution though) which allows more compression if the frame rate is slow.

Documentation

Developers Guide has been updated to reflect new changes.

Learnings

This project has been a great learning curve for me. My knowledge about fpga and related stuff has increased exponentially. The mentors are great and supportive. Working for this org was a great introduction to Open Source Development. I would highly recommend the org to anyone who is new and wants to get dirty with FOSS.

Heart Beat Feature

If a static is being displayed by HDMI2USB (for example a slide in a presentation), then it is difficult to tell whether the HDMI2USB is working or it is hung. So a heart beat feature has been added which causes a block of pixel to pulse in the lower right corner of the screen indicating that HDMI2USB is alive.

How to control?

Heart Beat (HB) can be switched on/off using switch SW 0 or using the CDC control port by sending ASCII command "S(s)" and "H(h)". But the control port will work only when the switch is on. So the switch acts as a "hard off" signal.

How to tweak the pulse?

The heart beat module is instantiated in the image_selector module. To change height, width and pulsing rate of the pixels, change the generics HB_length, HB_width and alt_aft_frame in hdl/misc/image_selector.vhd file.

Video

Debug Infrastructure Documentation

Debug infrastructure provides an easy interface for a developer to output data from the FPGA board for debugging. It is also useful for end users in case of troubleshooting.

The BIG PICTURE:

The debug data is collected into the debug_module which then breaks the data into bytes and sends them to UART module and USB_top module. These module send data via CDC/UART to host.

How to use?

The output of UART/CDC are bytes. So to convert them to human readable format and hence allow easy debugging, a program fpga_debug.c has been provided which reads data from the fpga and then outputs data in correct format.

To compile fpga_debug.c

> cd tools

> make

To run debugging program

> ./fpga_debug

How to add new debug data? 

• Route the data into debug_module
• Break it into bytes and add to uart_byte_array
•  Change value of "constant N_BYTES  : integer" appropriately
• In fpga_debug.c add code to parse and output the new data

You are done!

Output Bytes:

Byte	Description
1	Device State
5:2	Resolution of Source
6	Input frame rate in fps
7	Output frame rate in fps
8	Frame write time in ms
9	Frame Processing Time in ms
10	No. of frames dropped for every frame processed
13:11	No. of bytes in a frame

NOTE: To use UART port, exar device driver should be installed. But the driver does not allow CDC port to function along with it. So you can use either UART or CDC but not both.

Video

Control Port Still not working

Fixed some error but status commands are still not working grrrr..