Thank You TimVideos!

It is slightly late but better late than never. My final evaluation results are out and I have been given a pass grade by my organisation. It felt really good to see the mail of successful completion of GSoC after three months of hard work.
I would be very ungrateful if I don't thank my mentor, Joel "Shenki" Stanley. He is probably the coolest mentor and I have dibs on him if I participate next year. Initially I was worried as I didn't have a clear understanding of what to do and in those times shenki was really cool and motivated me which was a big help. I would also like to than our org admin Tim "mithro" Ansell, whose dedication to this org inspires me. Seriously, I would love to sit and talk with him on how he manages to motivate himself to learn and do so many stuff (BTW he works at Google).
I would like to keep contributing for the organisation as and when I get some free time. I also see a lot of my junior getting inspired and wanting to work for this org (one of them had sent a mail on the mailing list some time ago). I am planning to start a OSDG-Hardware group and introduce them to TimVideos. I see a lot of student interested to contribute but don't have sufficient knowledge about things relates to FPGA. So I will be soon (read "when I am free") posting some material and links which will help a raw student who knows only a HDL to become a "hardware developer". Another problem which I found when I discussed with my junior about open source hardware development is the need to have "the hardware" which for us in out case is the FPGA (very expensive considering a country like India). We can slightly mitigate this problem by developing a test suite which helps students who don't have hardware.
The journey of GSoC was great. I have was voted as the most hardworking GSoC intern among my friends who were also interning at GSoC orgs. My learning was really awesome. I feel smarter than most of my batch mates (which will slowly fade away when mid semester exams start). Also the exposure to open source community has restored my faith in humanity because good people still exist and that too in great number. Overall, this experience will remain green in my mind forever.

GSOC Final Report

Result Summary

Timvideos is trying to develop an opensource video conferencing solution. The organisation has both a hardware component (HDMI2USB) and software component (like Gstreamer, flumotion). I was working on the hardware side of the org specifically on HDMI2USB fpga firmware. HDMI2SUSB is basically a solution based on FPGA to compress HD video stream input from a source like video camera and output the compressed stream via USB, hence the name HDMI2USB.

GOALS

My  project was on "Optimisation of MPEG Core". In the original firmware, the output stream from USB was about 10-11 fps which is very low and not suitable for video recording, the primary purpose of HDMI2USB. The aim of my project was to improve this frame rate. I had proposed to improve it to 60 fps @ 720p. As the reason for a slow frame rate was not fully understood, I built a lot of firmware to get the "status" of the HDMI2USB which later on became my second project, to create a "Debug Infrastructure" for HDMI2USB.

Final Outcome

Finally I was able to improve the frame rate to 30 fps (which is the minimum required for recording) and set a system which allows developers to easily output debug data via CDC or UART Port.

How did the frame rate improve?

The title of my project "Optimisation of MPEG Core" isn't a very apt title. The work I did tried to optimize the whole firmware. The original firmware has an image buffer which stores a frame before being compress by the JPEG core. The state machine of the image buffer was such that no processing was done when a frame was being read and no frames were read when processing was done. This was the bottleneck which was causing a slow frame rate. To remove this bottleneck I pipelined the read and write cycles of the image buffer. This reduced the number of frames which were being dropped and hence improve the frame rate.

Debug Interface

The debug interface is a new feature added to the HDMI2USB firmware. Unlike programming language which can be easily debugged using printf statement, systems on hardware like FPGA require elaborate methods. The debug interface which has been implemented allows data to be output via CDC port which was part of the original firmware and UART port which has been added to allow more options to developers and users incase CDC fails. Also a program has been included which reads bytes from fpga and presents them to users in a human readable format.

Heart Beat Feature

Sometimes when the image is static, it is difficult to tell whether the fpga firmware is working or it has hung. To remove this ambiguity, a heart beat feature was added which is basically a small block of pixel in the bottom right corner of the frame pulsing at a constant rate. This feature can be turned on/off using a hardware switch and via the CDC control port.

Community Outcomes

The Timvideos developers’ community has promising outcomes after completion of this project. The biggest being the HDMI2USB can be used for recording. Now that HDMI2USB firmware streams at 30 fps, it can be used for recording at 30 fps which is the minimum frame rate required for video recording. Also, I may not have been able to meet the 60 fps as I mentioned in my proposal but I have successfully revealed some issues in the HDMI2USB firmware which need to be dealt with, most importantly the bandwidth of FX2 chip.

There are certain other features which can be implemented but have not been implemented due to lack of time like reducing subsampling to 4:2:0 and implementing a triple buffer in the image buffer. This may or may not change the frame rate but it will be interesting to see the outcomes if they are implemented. The debug infrastructure that has been setup will allow future developers to easily debug the hardware and also provide an easy way of troubleshooting for future users. Addition of UART allows multiple debug/control option incase cypress fx2 fails.  As my project dealt more with optimisation which involves getting a good understanding of the system as a whole, I have gained good knowledge about the fpga firmware. This will be useful for future developers as I am in a position to be able to guide them regarding specific issues. Also thanks to the exposure I received due to this project, I have started a Open Source Hardware Developers Group in my university where soon I will be introducing TimVideos. Hopefully this will increase the number of developers for HDMI2USB.

Replicating Results

All the code relevant to the project has been merged into the HDMI2USB repository.

Also check out the how to use section in the following blog posts:

• UART
• Heart Beat
• Debug Interface
• To check the frame rate  

Bugs/To Do

• Support 1080p via USB: I successfully added 1080p test pattern (Hidef snow) to the hdmi2usb. It was streaming out of the HDMI port but not out of USB. Here is the video.
• Add Control Port via UART: I was able to expose some features of control port to UART but not all. I will try to finish after GSOC.
• Automatic Encoding Quality Control: It has been noticed that the encoding quality depends highly on the type of image. So there can be a auto encoding qulaity control like the one in youtube (it changes the resolution though) which allows more compression if the frame rate is slow.

Documentation

Developers Guide has been updated to reflect new changes.

Learnings

This project has been a great learning curve for me. My knowledge about fpga and related stuff has increased exponentially. The mentors are great and supportive. Working for this org was a great introduction to Open Source Development. I would highly recommend the org to anyone who is new and wants to get dirty with FOSS.

Heart Beat Feature

If a static is being displayed by HDMI2USB (for example a slide in a presentation), then it is difficult to tell whether the HDMI2USB is working or it is hung. So a heart beat feature has been added which causes a block of pixel to pulse in the lower right corner of the screen indicating that HDMI2USB is alive.

How to control?

Heart Beat (HB) can be switched on/off using switch SW 0 or using the CDC control port by sending ASCII command "S(s)" and "H(h)". But the control port will work only when the switch is on. So the switch acts as a "hard off" signal.

How to tweak the pulse?

The heart beat module is instantiated in the image_selector module. To change height, width and pulsing rate of the pixels, change the generics HB_length, HB_width and alt_aft_frame in hdl/misc/image_selector.vhd file.

Video

Debug Infrastructure Documentation

Debug infrastructure provides an easy interface for a developer to output data from the FPGA board for debugging. It is also useful for end users in case of troubleshooting.

The BIG PICTURE:

The debug data is collected into the debug_module which then breaks the data into bytes and sends them to UART module and USB_top module. These module send data via CDC/UART to host.

How to use?

The output of UART/CDC are bytes. So to convert them to human readable format and hence allow easy debugging, a program fpga_debug.c has been provided which reads data from the fpga and then outputs data in correct format.

To compile fpga_debug.c

> cd tools

> make

To run debugging program

> ./fpga_debug

How to add new debug data? 

• Route the data into debug_module
• Break it into bytes and add to uart_byte_array
•  Change value of "constant N_BYTES  : integer" appropriately
• In fpga_debug.c add code to parse and output the new data

You are done!

Output Bytes:

Byte	Description
1	Device State
5:2	Resolution of Source
6	Input frame rate in fps
7	Output frame rate in fps
8	Frame write time in ms
9	Frame Processing Time in ms
10	No. of frames dropped for every frame processed
13:11	No. of bytes in a frame

NOTE: To use UART port, exar device driver should be installed. But the driver does not allow CDC port to function along with it. So you can use either UART or CDC but not both.

Video

Control Port Still not working

Fixed some error but status commands are still not working grrrr..

Control port via UART

This is turning out to be more difficult than I thought. I was trying to use the same controller module for cdc and UART. But for some reason which I haven't been able to figure out, the video output is null when I am trying to do that. Tried a bunch of stuff but nothing worked. I guess I will have to make a separate controller for UART.

Hidef snow working!

Finally got the hidef snow working through hdmi2usb via HDMI out but not via usb.
I am getting a timeout error. I increased the watchdog timer inside the usb module to absurdly large value but it still did not remove the error. The large size of frame might be the problem.

Unable to get hidefsnow working

I wanted to add a 1080p test pattern to firmware so the performance at  that resolution can be understood. Unfortunately things are not working. I am getting strange output values from debugger. Hopefully I get it done soon.