I’ve been using the Aptina MT9D131 image sensor for a new project. It’s an interesting chip. One of the great features of this chip is its ability to output JPEG images (which includes white balance, bayer interpolation, etc). That saves a great deal of processing from the host CPU. The chip is also readily available from digikey which is nice for prototyping. The documentation for the chip isn’t freely available though – you need to get that from Aptina, most likely through your local Aptina representative. (Please don’t email me asking for the documents – I cannot give them to you – you need to get them through the proper channels because Aptina treats them as confidential.)
In JPG mode the sensor does not output a JPG header – it only outputs the scan data. To create a JPG file you need to add a JPG header to the start, and put an “end of scan” marker (the two bytes 0xFF 0xD9) at the end of the file. The Aptina documentation contains some source code to generate the header file. Just copy & paste it into your favourite compiler and you’re good to go. The only problem is, when I did that, and used the resulting header, I got images with very “fuzzy’ boundaries at edges. Here’s a close-up of the testpattern the image sensor can generate:
You can clearly see the problem along the vertical lines. Other folks over at the Sparkfun forums were having the same problems.
For a long time I thought it was a register settings problem with the image sensor. Eventually I clued in that it might be the header. Finding some jpeg encoder software that Cristi Cuturicu at wotsit.org posted, I re-wrote parts of the Aptina header generation code. The result was an immediate improvement:
You can see the big improvement in the edges of the vertical bars. The problem lay in the function DefineQuantizationTableMarker(), which as its name suggests, generates the quantisation tables (both luminance and chrominance). My new version of that function is as follows:
// Calculate and write the quantisation tables
// qscale is the customised scaling factor - see MT9D131 developer guide page 78
int DefineQuantizationTableMarker (unsigned char *pbuf, int qscale, int format)
{
int i, length, temp;
unsigned char newtbl[64]; // temporary array to store scaled zigzagged quant entries
if (format == FORMAT_MONOCHROME) // monochrome
length = 67;
else
length = 132;
*pbuf++ = 0xFF; // define quantization table marker
*pbuf++ = 0xDB;
*pbuf++ = length>>8; // length field
*pbuf++ = length&0xFF;
*pbuf++ = 0; // quantization table precision | identifier for luminance
// calculate scaled zigzagged luminance quantisation table entries
for (i=0; i<64; i++) {
temp = (JPEG_StdQuantTblY[i] * qscale + 16) / 32;
// limit the values to the valid range
if (temp <= 0)
temp = 1;
if (temp > 255)
temp = 255;
newtbl[zigzag[i]] = (unsigned char) temp;
}
// write the resulting luminance quant table to the output buffer
for (i=0; i<64; i++)
*pbuf++ = newtbl[i];
// if format is monochrome we're finished, otherwise continue on, to do chrominance quant table
if (format == FORMAT_MONOCHROME)
return (length+2);
*pbuf++ = 1; // quantization table precision | identifier for chrominance
// calculate scaled zigzagged chrominance quantisation table entries
for (i=0; i<64; i++) {
temp = (JPEG_StdQuantTblC[i] * qscale + 16) / 32;
// limit the values to the valid range
if (temp <= 0)
temp = 1;
if (temp > 255)
temp = 255;
newtbl[zigzag[i]] = (unsigned char) temp;
}
// write the resulting chrominance quant table to the output buffer
for (i=0; i<64; i++)
*pbuf++ = newtbl[i];
return (length+2);
}
This function requires the following 3 tables (standard luminance quantization, standard chrominance quantization, and zigzag), which are straight out of the JPEG specification.
unsigned char JPEG_StdQuantTblY[64] =
{
16, 11, 10, 16, 24, 40, 51, 61,
12, 12, 14, 19, 26, 58, 60, 55,
14, 13, 16, 24, 40, 57, 69, 56,
14, 17, 22, 29, 51, 87, 80, 62,
18, 22, 37, 56, 68, 109, 103, 77,
24, 35, 55, 64, 81, 104, 113, 92,
49, 64, 78, 87, 103, 121, 120, 101,
72, 92, 95, 98, 112, 100, 103, 99
};
unsigned char JPEG_StdQuantTblC[64] =
{
17, 18, 24, 47, 99, 99, 99, 99,
18, 21, 26, 66, 99, 99, 99, 99,
24, 26, 56, 99, 99, 99, 99, 99,
47, 66, 99, 99, 99, 99, 99, 99,
99, 99, 99, 99, 99, 99, 99, 99,
99, 99, 99, 99, 99, 99, 99, 99,
99, 99, 99, 99, 99, 99, 99, 99,
99, 99, 99, 99, 99, 99, 99, 99
};
// this table is used for regular-position to zigzagged-position lookup
// This is Figure A.6 from the ISO/IEC 10918-1 1993 specification
static unsigned char zigzag[64] =
{
0, 1, 5, 6,14,15,27,28,
2, 4, 7,13,16,26,29,42,
3, 8,12,17,25,30,41,43,
9,11,18,24,31,40,44,53,
10,19,23,32,39,45,52,54,
20,22,33,38,46,51,55,60,
21,34,37,47,50,56,59,61,
35,36,48,49,57,58,62,63
};
This is a drop-in replacement for the Aptina function. If you're seeing similar problems, give this version of that function a try. Some folks have found that the standard Aptina datasheet code works fine; other folks (including me) have found that it doesn't. The way the Aptina code is written could well be sensitive to which C compiler (and compile options) is used. Use my code if you need it. It's a great image sensor - hopefully this information will help others get it up and running just a little bit quicker.
Hi Frank,
just dropping a hello, stumbled over your article while googling up some old imaging sensor details and thought: Wait a second, I’ve ‘talked’ to this guy before. I believe we were speaking JPEG stuff in the good old Surveyor days.
Anyhow, hope you’re still having fun with imaging and electronics,
greetings from switzerland,
– Martin
Thanks Frank, its good to know I dont have to make a different header for every image. These kind of things really help me to narrow down whats going on.
I have the dg, and the full datasheet. The restart int is by default 0, from the registry settings i found on the sparkfun forums the restart int is set to
CAM_WriteReg(0xC6, 0x2908); //16-bit, offset 0x08: restart interval
CAM_WriteReg(0xC8, 0x0002); //write value
2, so i guess in my constructor of the header i’ll put 2. Then for the quality scale there are three “qscales” in the datasheet with default values 6,9 and 12. I’m guessing i’ll use 6 in the constructor of the jpeg header code. I dont understand why there are three qscales. This is what i’m going to use so far
CreateJpegHeader(header, 128, 96, FORMAT_YCBCR422, 0x02, 6);
I still have to make sure i’m getting the right data out of the camera, with this mess of wires i think i’m not getting a good signal somewhere. I set the registry value to only enable pixclk when valid jpeg data, but does that only apply while in “capture” mode. Do I switch it from capture to preview, do i have to set things like jpeg driver id=9 offset 7 bit 3 “host indicates ready for next frame”.
This is the code i was refering to the code to set registers to take a picture, go into capture and accepting of the jpeg.
Thanks again, i hope to be able to figure this out i’ve been working on it for months now trying to get my little robot some eyes !
The MT9D131 Developer Guide talks about restart_int and qscale. If you don’t have that document, you need to contact your Aptina representative or distributor, sign their NDA and get it. It’s not so much a question of what value you give for those things, but what’s critical is that you use the same value for your header generation as you give to the image sensor in its register settings. Qscale is a quality setting – lower quality of course results in smaller image sizes. Restart_int is the “restart interval”, which says how often you restart the encoding within the JPEG compressed image. If you had a perfect error-free environment, you arguably would never need to restart the encoding partway through. But in case of errors, having a few restarts in the image helps recover the image, at least partially. If you play with it, you’ll find that, except for extreme cases, it doesn’t make much difference to image size, so it’s worth turning it on.
I cannot post my entire header code because it’s the exact code that’s in the back of the MT9D131 Developers Guide, which is under NDA. So, get that document and you’ll have the header code. The sole change I made to that code (and some folks have found my change to be unnecessary) was in the quantization tables generation, which is detailed in this post. Once you have generated the header, and its settings are the same as what the image sensor is using, you use that same header at the front of every image. Let me be very clear here. The MT9D131 does not output a complete JPEG file. It outputs the “core portion” of a JPEG file. What you receive from the image sensor needs to have a header stuck on the front, and a little trailer stuck on the end (0xFF 0xD9). That same header is used for every file, every image, until such time as you change the image settings (image dimensions, quality, etc) in which case you’d need to create a new header.
Hey Frank, thanks for all your help on this post and other forums. I’m really confused as to what i’m supposed to pass into the createjpegheader constructor as far as restart_int and qscale ? I’m using areo spaces’s register settings that he/she posted on sparkfun. Is there any way you could share the code you used to create the jpeg header ? Can we generate it once and use that over and over or does it need to be generated every time ?
I originally used a mclk of 10Mhz but increased it to 24MHz thinking that perhaps at 10MHz it wasn’t able to ‘keep up’ with the jpg processing. Didn’t help much though. I will look at the registers you mentioned and indeed if I can slow down just the pxlclk then I’ll try that. I thought though that if pxlclk was too slow then the jpg FIFO may not be able to get the data out fast enough and have overflow errors. But worth a shot. When I read the jpg status flags they all show ok, no errors, overruns so I don’t understand how my data is getting corrupted. It would be less than probably 0.25% of my data that seems to be bad. I’m also having trouble with images greater than 800×600 for some reason. I used the timing settings as per Aptina Devware at 1600×1200 but usually have jpg images that won’t even open. The register wizard from Aptina only covers the SOC2010 (MT9D111) and doesn’t have any settings for jpg so am a bit baffled there too. Surely the Aptina software would cover the MT9D131. In the generic Devware it isn’t specifically covered. Figure that ? Thanks again Frank for your suggestions. I am puzzled that for such a widely available, relatively cheap sensor that can be easily prototyped, with jpg output, there aren’t more people using this sensor.
As you know, there are a very large number of settings in this sensor which can affect output timing. And they’re scattered all over the place – some are in the core registers, others in the JPEG variables, etc. Have you tried slowing down the sensor output clock (while still remaining within the image sensor specs for clocks)? If you have a timing problem (as opposed to a logic problem), slowing things down can often help. I use the registers mode.fifo_conf to reduce the output clock speed to around 4 MHz. (See the developer guide page 21).
Frank
I tried your header code changes. I’m still seeing what looks like noise/errors in the jpg image. I have tried so many things with the camera but still no success. I am thinking that perhaps the header is still giving me trouble.When I run jpeg analysers over the resultant file it shows errors in the jpg data stream which again points to corrupted data. I’ve played with pad slew as you suggested but I think my timing is ok. I have wirewrapped my breadboard so at the clk frequencies, maybe I’m just getting crosstalk and induced noise which is showing up as corrupted images. Before I lay out my custom PCB, I wanted to be sure the hardware was ok, software bugs I can fix later but I am at a standstill. I have the developers guide, datasheet and Aptina register wizard software and believe my settings should work.