USB-UART control port

As part of my GSoC assignment, I tried to expose the control port via uart, in addition to control port via usb which was already present. I could not get it to work that time. So now I took the job again. Instead on working on my previous code, I decided to start with a clean slate.
My approach was to mux the already available resource which was being used for usb control port. After many hours of understanding the simulation waveforms, I finally managed to get things working, umm partially.
There are two kinds of commands. One which cause some changes for example forcing output video to be in grayscale, changing to HDMI0 etc. These are currently working in my uart control port. Then there are commands which output status back to the terminal. Now these currently don't work and I can get them working. The only problem will be that then the tools/fpga_debug will have to be written again for uart which will be plenty of work to do before sprints starts.

Need for a remote access to FPGA

My mid sems finally got over yesterday and I got some free time in my life. So I am blogging. After completion of GSoC, lot of my juniors are asking me about my  project and a few seem to be interested. Few days back I answered a question on quora : http://www.quora.com/What-are-some-suggested-open-source-projects-in-Verilog-for-intermediates/answer/Ajit-Mathew and the person asking the question PMed me regarding my project. But what is stopping them from contributing is that they don't have an access to FPGA. In India, an Atlys board costs around INR 20K which is a big deal for a student. Also colleges have the old spartan 3 boards or don't give easy access to FPGA board.
I was recently talking to an alumni who is working in a FPGA based startup and he mentioned to me how they have created a hack which allows the developers to remotely access the FPGA, dump the .bit and see results. If we at TimVideos can develop a similar solution then it would improve participation greatly. If the logistics are handled, making this possible is not very challenging. Another solution can be to make a comprehensive test bench which allows black box and white box testing. A good language to create this test bench will be System Verilog as it combine verilog and class structure of C++. But this is a very big project. Creating accurate models for verification will take a lot of time.

The need to have expensive hardware for development is big disadvantage for open source hardware development. 

Thank You TimVideos!

It is slightly late but better late than never. My final evaluation results are out and I have been given a pass grade by my organisation. It felt really good to see the mail of successful completion of GSoC after three months of hard work.
I would be very ungrateful if I don't thank my mentor, Joel "Shenki" Stanley. He is probably the coolest mentor and I have dibs on him if I participate next year. Initially I was worried as I didn't have a clear understanding of what to do and in those times shenki was really cool and motivated me which was a big help. I would also like to than our org admin Tim "mithro" Ansell, whose dedication to this org inspires me. Seriously, I would love to sit and talk with him on how he manages to motivate himself to learn and do so many stuff (BTW he works at Google).
I would like to keep contributing for the organisation as and when I get some free time. I also see a lot of my junior getting inspired and wanting to work for this org (one of them had sent a mail on the mailing list some time ago). I am planning to start a OSDG-Hardware group and introduce them to TimVideos. I see a lot of student interested to contribute but don't have sufficient knowledge about things relates to FPGA. So I will be soon (read "when I am free") posting some material and links which will help a raw student who knows only a HDL to become a "hardware developer". Another problem which I found when I discussed with my junior about open source hardware development is the need to have "the hardware" which for us in out case is the FPGA (very expensive considering a country like India). We can slightly mitigate this problem by developing a test suite which helps students who don't have hardware.
The journey of GSoC was great. I have was voted as the most hardworking GSoC intern among my friends who were also interning at GSoC orgs. My learning was really awesome. I feel smarter than most of my batch mates (which will slowly fade away when mid semester exams start). Also the exposure to open source community has restored my faith in humanity because good people still exist and that too in great number. Overall, this experience will remain green in my mind forever.

GSOC Final Report

Result Summary

Timvideos is trying to develop an opensource video conferencing solution. The organisation has both a hardware component (HDMI2USB) and software component (like Gstreamer, flumotion). I was working on the hardware side of the org specifically on HDMI2USB fpga firmware. HDMI2SUSB is basically a solution based on FPGA to compress HD video stream input from a source like video camera and output the compressed stream via USB, hence the name HDMI2USB.

GOALS

My  project was on "Optimisation of MPEG Core". In the original firmware, the output stream from USB was about 10-11 fps which is very low and not suitable for video recording, the primary purpose of HDMI2USB. The aim of my project was to improve this frame rate. I had proposed to improve it to 60 fps @ 720p. As the reason for a slow frame rate was not fully understood, I built a lot of firmware to get the "status" of the HDMI2USB which later on became my second project, to create a "Debug Infrastructure" for HDMI2USB.

Final Outcome

Finally I was able to improve the frame rate to 30 fps (which is the minimum required for recording) and set a system which allows developers to easily output debug data via CDC or UART Port.

How did the frame rate improve?

The title of my project "Optimisation of MPEG Core" isn't a very apt title. The work I did tried to optimize the whole firmware. The original firmware has an image buffer which stores a frame before being compress by the JPEG core. The state machine of the image buffer was such that no processing was done when a frame was being read and no frames were read when processing was done. This was the bottleneck which was causing a slow frame rate. To remove this bottleneck I pipelined the read and write cycles of the image buffer. This reduced the number of frames which were being dropped and hence improve the frame rate.

Debug Interface

The debug interface is a new feature added to the HDMI2USB firmware. Unlike programming language which can be easily debugged using printf statement, systems on hardware like FPGA require elaborate methods. The debug interface which has been implemented allows data to be output via CDC port which was part of the original firmware and UART port which has been added to allow more options to developers and users incase CDC fails. Also a program has been included which reads bytes from fpga and presents them to users in a human readable format.

Heart Beat Feature

Sometimes when the image is static, it is difficult to tell whether the fpga firmware is working or it has hung. To remove this ambiguity, a heart beat feature was added which is basically a small block of pixel in the bottom right corner of the frame pulsing at a constant rate. This feature can be turned on/off using a hardware switch and via the CDC control port.

Community Outcomes

The Timvideos developers’ community has promising outcomes after completion of this project. The biggest being the HDMI2USB can be used for recording. Now that HDMI2USB firmware streams at 30 fps, it can be used for recording at 30 fps which is the minimum frame rate required for video recording. Also, I may not have been able to meet the 60 fps as I mentioned in my proposal but I have successfully revealed some issues in the HDMI2USB firmware which need to be dealt with, most importantly the bandwidth of FX2 chip.

There are certain other features which can be implemented but have not been implemented due to lack of time like reducing subsampling to 4:2:0 and implementing a triple buffer in the image buffer. This may or may not change the frame rate but it will be interesting to see the outcomes if they are implemented. The debug infrastructure that has been setup will allow future developers to easily debug the hardware and also provide an easy way of troubleshooting for future users. Addition of UART allows multiple debug/control option incase cypress fx2 fails.  As my project dealt more with optimisation which involves getting a good understanding of the system as a whole, I have gained good knowledge about the fpga firmware. This will be useful for future developers as I am in a position to be able to guide them regarding specific issues. Also thanks to the exposure I received due to this project, I have started a Open Source Hardware Developers Group in my university where soon I will be introducing TimVideos. Hopefully this will increase the number of developers for HDMI2USB.

Replicating Results

All the code relevant to the project has been merged into the HDMI2USB repository.

Also check out the how to use section in the following blog posts:

• UART
• Heart Beat
• Debug Interface
• To check the frame rate  

Bugs/To Do

• Support 1080p via USB: I successfully added 1080p test pattern (Hidef snow) to the hdmi2usb. It was streaming out of the HDMI port but not out of USB. Here is the video.
• Add Control Port via UART: I was able to expose some features of control port to UART but not all. I will try to finish after GSOC.
• Automatic Encoding Quality Control: It has been noticed that the encoding quality depends highly on the type of image. So there can be a auto encoding qulaity control like the one in youtube (it changes the resolution though) which allows more compression if the frame rate is slow.

Documentation

Developers Guide has been updated to reflect new changes.

Learnings

This project has been a great learning curve for me. My knowledge about fpga and related stuff has increased exponentially. The mentors are great and supportive. Working for this org was a great introduction to Open Source Development. I would highly recommend the org to anyone who is new and wants to get dirty with FOSS.

Heart Beat Feature

If a static is being displayed by HDMI2USB (for example a slide in a presentation), then it is difficult to tell whether the HDMI2USB is working or it is hung. So a heart beat feature has been added which causes a block of pixel to pulse in the lower right corner of the screen indicating that HDMI2USB is alive.

How to control?

Heart Beat (HB) can be switched on/off using switch SW 0 or using the CDC control port by sending ASCII command "S(s)" and "H(h)". But the control port will work only when the switch is on. So the switch acts as a "hard off" signal.

How to tweak the pulse?

The heart beat module is instantiated in the image_selector module. To change height, width and pulsing rate of the pixels, change the generics HB_length, HB_width and alt_aft_frame in hdl/misc/image_selector.vhd file.

Video

Debug Infrastructure Documentation

Debug infrastructure provides an easy interface for a developer to output data from the FPGA board for debugging. It is also useful for end users in case of troubleshooting.

The BIG PICTURE:

The debug data is collected into the debug_module which then breaks the data into bytes and sends them to UART module and USB_top module. These module send data via CDC/UART to host.

How to use?

The output of UART/CDC are bytes. So to convert them to human readable format and hence allow easy debugging, a program fpga_debug.c has been provided which reads data from the fpga and then outputs data in correct format.

To compile fpga_debug.c

> cd tools

> make

To run debugging program

> ./fpga_debug

How to add new debug data? 

• Route the data into debug_module
• Break it into bytes and add to uart_byte_array
•  Change value of "constant N_BYTES  : integer" appropriately
• In fpga_debug.c add code to parse and output the new data

You are done!

Output Bytes:

Byte	Description
1	Device State
5:2	Resolution of Source
6	Input frame rate in fps
7	Output frame rate in fps
8	Frame write time in ms
9	Frame Processing Time in ms
10	No. of frames dropped for every frame processed
13:11	No. of bytes in a frame

NOTE: To use UART port, exar device driver should be installed. But the driver does not allow CDC port to function along with it. So you can use either UART or CDC but not both.

Video

Control Port Still not working

Fixed some error but status commands are still not working grrrr..

Control port via UART

This is turning out to be more difficult than I thought. I was trying to use the same controller module for cdc and UART. But for some reason which I haven't been able to figure out, the video output is null when I am trying to do that. Tried a bunch of stuff but nothing worked. I guess I will have to make a separate controller for UART.

Hidef snow working!

Finally got the hidef snow working through hdmi2usb via HDMI out but not via usb.
I am getting a timeout error. I increased the watchdog timer inside the usb module to absurdly large value but it still did not remove the error. The large size of frame might be the problem.

Unable to get hidefsnow working

I wanted to add a 1080p test pattern to firmware so the performance at  that resolution can be understood. Unfortunately things are not working. I am getting strange output values from debugger. Hopefully I get it done soon.

Lot of pushing done

- Debug Infrastructure Pushed
- Pipelined Memory Pushed
- Will do heart beat tomorrow

Daily Snippey

Debugger program with cdc in done
H2u firmware with cdc debug done with timing errors. But it works. Worked really hard on it but did not meet timing.
Memory pipelining done for all encoding quality.
Today (Saturday) we have a Church outing so I will fix everything and commit them on Sunday.
Cheers!

EDIT: The title was a typo because I was using my tab. But it sounds cool.

Daily Snippet

Video Link

Daily snippet

Oops missed yesterday's blog.
- Heart beat feature completed
- Sent debug data out of cdc
- working on debugger program
- major changes to debug module

HDMI2USB Serial Controller Port

When we connect HDMI2USB to host, two CDC are enumerated. One is for UVC(ACM0) and the other is for controller port(ACM1).
The controller port can be used to control certain features of HDMI2USB and also display status of USB, jpeg encoder etc.
To access this use any serial terminal program like gtkterm. The commands send to the control port is in ASCII. The ascii commands are two byte, first is the address(like U or u for USB TOP) and the second is the command you want to send to that address (like S or s for status). So if you send US, it will return the status of the USB Top module. Since the terminal I am using allows only hexadecimal commands, so I have to convert the ASCII commands to their hex equivalent (which is irritating).

While creating the debug infrastructure, I completely neglected that there is already a lot of infrastructure already built in due to the control port. Some feature that I have added like frame rate etc can be exposed via the control port. Then we can have a debugger using the control port. Plus it can be used to control features of firmware which is even more awesome. What say mithro/shenki?

Controller Port

I fixed up Heart Beat module as Karl required it. Mithro asked me to add functionality to turn it off/on using controller port. To do that I dug through the code of how data is send through the second ACM port that is enumerated via HDMI2USB. I was finally able to do it.

Daily Snippet

Instead of two encoder, I tried changing the state machine to drop frames as low as possible but it didn't work. Will try to work on it tomorrow.
Also I changed the code of the heart beat feature I had implemented long ago. I did not have atlys board when I implemented it so came to know about a bug in the code when I ran it on the board. Eventually was able to fix it and improved the code to use lesser number of flops.
Also mithro added me to the TimVideos Hardware hacker team. It felt nice, actually very nice. It actually felt like this:

So can I now update my linkedin profile? But before that let me learn Git coz when shenki talks about stuff like bisecting, rebase I am like this

Debug Program almost done!

The bare bone debug program is ready.
Here is how the program looks like. Things will improve as other developers add stuff to the debug infrastructure like packet count, expansion boards ID etc.
There are two bugs which I couldn't fix even after hours of attempt: 

• UART is suddenly send incorrect data causing wrong output for a moment.
• HDMI 1 is shown connected even though only one is connected.

I will spend some more time on it in the weekend.

Tomorrow I am planning to go back to the real stuff I am working on. Tomorrow I will try to add another encoder. Let's see if it is possible. If not, atleast we will know that this solution is not possible for available fpga resource.

Till then I am going to sleep.

Oops Forgot again

I usually write my blog before going to bed. Yesterday I went to bed and then I realised I forgot to write my snippet. Now I had a very tough choice to make-"Sleep,blopost,sleep,blogpost" Finally I chose sleep. So now I wake up and write my blog.
Daily Snippet:
1) Removed some bugs from the debugger. (Behold the irony)
2) Added non blocking keyboard IO.

Hopefully I complete it today.

Phew, now I can sleep in peace.

GSOC ain't your College Assignment

Today I submitted my first c code and it was a such a disaster that I am looking back and laughing at it. Contributing to open source comes with it's own baggage like license, coding style and git, all of which I never cared about. But these are very important in an collaborative environment and hence important to learn. I got the bare bone version of my debug program working (with errors) and committed it. But before committing I remembered, "Where is the license?" So I copied the license from a VHDL file ( I know I am a smartass) and then happily committed it, feeling proud of my achievement. Only later I realised that it won't compile because to comment in VHDL we use "--" which will throw a compile error in C. Shenki must have been ROFLing on seeing it.

Also I would like to propose daily tweeting instead of daily blogging. I haven't been blogging lately not because I was not working but blog, don't know why, sounds to  me hard work, like the daily cursive writing my Mom used to make me write when I was a kid.

Okay now I am rambling. Daily Snippet:
-[code]Debug program
-Studied Git
-Studied Linux Coding Stlye Guide
-Sat and wondered why mithro and shenki work so hard?

Tomorrow:
-[code]Debug program
-work on comments and *try* to rebase