MIPI deployment in ultra-low-power streaming sensors

By Brian Lenkowski, Director Product Management, Consumer CMOS Image Sensors, ams OSRAM.
Mahmoud ElBanna, General Manager, Mixel-Egypt.

Streams of data from higher-speed sensors pose throughput and latency challenges for designers. However, optimizing a design for those criteria can come at the expense of increased power consumption if not conceived and executed carefully. A device like a high-resolution, high-frame-rate home security camera in a non-wired application requiring frequent battery changes or recharging will likely register strong dissatisfaction among most users, even if it delivers its stream at full quality when working. Interfacing streaming sensors with MIPI^® helps designers balance performance with power consumption by leveraging various modes of operation: high speed, ultra-low power, or something in between.

The MIPI Alliance was formed to shape data interfaces for mobile devices where extending battery life is a priority. With its wide adoption in smartphone cameras and, to a lesser degree, displays, MIPI became the de facto standard in these applications. MIPI-based cameras are now seeing increased adoption in non-mobile applications, including automotive, IoT, virtual reality (VR), augmented reality (AR), industrial, and medical applications. MIPI D-PHY^SM and MIPI C-PHY^SM modules feature two sets of optimized PHYs, controlled by state machines managing high-speed, low-power, and ultra-low-power states, including shutting down entirely for power savings when data communication is inactive.

This article focuses on low-power scenarios with streaming sensors connected to a processor via MIPI. After a short overview of low-duty cycle sensor principles, we explore how a streaming image sensor leverages those principles, show how one semiconductor company uses MIPI CSI-2^® and D-PHY interfaces in their imaging solution, and touch on use cases for ultra-low-power streaming sensors deploying MIPI.

Dual operating contexts lower the sensor duty cycle

Modern digital semiconductor technology consumes power at the transistor level in two fundamental ways: dynamic power generated when transistors switch, which is proportional to clock speed, and leakage power from supply voltages resulting in small leakage currents. Ultra-low-power chip architects typically mitigate these sources by selecting a fabrication process with fast-switching, low-leakage transistors, lowering bias voltages, and matching clock speeds to the workload.

Attention then turns to power management in a processor architecture. Clock and power gating shut off portions of a chip when those functions are unnecessary, going as far as a sleep mode where only event-monitoring circuitry needed to reawaken the chip remains powered on. Many applications can minimize average power by lowering the processor's duty cycle – sleeping for as long as possible, waking up to work in short bursts when needed, and returning to sleep when done.

Many sensor applications fit this low-duty cycle profile. The bursts of work occur around new sensor samples, which can be random or periodic. Random sampling gives a snapshot of a sensor's status at that moment, such as pressing a button to take a temperature reading. Periodic sampling repeats sensor readings at specific intervals. Architects choose periodic sampling for three reasons: correlating time domain analysis on a known time scale, filtering or analyzing signals in the frequency domain, or digitizing and processing signals before converting them back to human-friendly analog formats.

Streaming image sensors rely on periodic sampling. Human visual perception provides a continuous view of a scene, while digital image sensors define sampled resolution and frame rate. Not enough pixels of resolution or a frame rate of less than 24 frames per second (fps), and humans start having difficulty interpolating motion, seeing an uncomfortable jitter in displayed video. Raising the frame rate is costly – sampling and transmitting all those pixels in less time takes more power with a faster clock rate, even with efficient handling in MIPI interfaces. It's a trap that seemingly leaves designers no choice but to use more energy to maintain acceptable streaming image quality for human viewers.

Challenging one assumption changes the power equation dramatically. Having humans watch surveillance video 24/7 is expensive and mind-numbing. Capturing human-quality video 24/7 is also expensive for a streaming imaging sensor constantly running at full power. If nothing moves from one frame to the next in a streaming image scene, there is nothing for humans or machines to see. However, an imaging sensor with smart processing can watch at a reduced video quality and energy level until the right moment.

^{Click to enlarge}

ams-OSRAM employs a concept for establishing dual imaging sensor operating contexts, lowering the frame rate and the number of pixels using subsampling across the entire region of interest resulting in a dramatically reduced MIPI transfer rate. An ultra-low-power sensor context waits to see an event, such as some object moving. Detecting the event shifts the sensor, MIPI interface, and processor into a full-power context with all the pixels in the region of interest sampled, a high frame rate, a corresponding high-speed data transfer rate, and enhanced object tracking. When the event conditions subside, the streaming chain returns to the low-power context, waiting for another event. The power savings between sensor contexts can be 20x or more, with the lower duty cycle substantially reducing its average power consumption.

MIPI deployed in the Mira050 image sensor

Image sensors from ams-OSRAM are suited for many consumer and industrial applications, such as smart home devices, drones, robotics, and wearable devices. The Mira050 is a 0.5 megapixel, global shutter back-side illuminated CMOS image sensor with 93% quantum efficiency at visible and over 50% in near-infrared wavelengths. Earlier generations of the Mira family used an LVDS interface, whereas the Mira050 now uses Mixel MIPI IP for its off-chip video interface. An immediate benefit of moving to MIPI is pin reduction, transitioning from a parallel to serial data interface, and with a stacked chip design, the Mira050 fits in a smaller 2.83mm x 2.29mm package. Looking inside the concept for the Mira050 MIPI implementation:

• The CSI-2 protocol layer handles packetization, while the D-PHY physical layer handles low-level data transport. CSI-2 provides flexible data transport with three packet formats: RAW8, RAW10, or RAW12. It also offers an optional metadata channel for embedding 8-bit non-image data to help identify sensor modes.
• The D-PHY physical layer allows the data lane to operate in either the high-speed or ultra-low-power context, with fast cycling between modes coordinated with sensor resolution and frame rate by external logic synchronizing packets and clocks.

^{Click to enlarge}

A closer look at the Mixel MIPI IP used in the Mira050 reveals more detail. Mixel customized its D-PHY TX⁺ solution, combining D-PHY v 2.1 and CSI-2 v 1.3 functions in a single IP block. Its high-speed transmission and reception run at 1.5Gbps per lane. In low-power mode, transmission and reception run at 10Mbps. 4-lane D-PHY TX⁺ IP blocks provide more data lanes for higher transfer rates. The D-PHY TX⁺ includes extensive built-in self-test (BIST) logic optimized for testability and production yield, exercising data transfers in high-speed and low-power modes in the hard macro and CIL RTL.

^{Click to enlarge}

D-PHY TX⁺ IP is redesigned from the ground up for exceptional efficiency. It uses 30% less area than the comparable D-PHY Universal configuration yet provides 100% test coverage of its hard macro blocks with its BIST capability. It also reduces leakage power by 40% for ultra-low-power applications like the Mira050.

Coordinating the Mira050 sensor resolution and sampling, frame rate, and MIPI transfer rate, with fast switching between modes, delivers uncompromised streaming performance up to 290 fps yet still conserves energy by lowering resolution and frame rate when appropriate. Mixel MIPI IP is an integral part of the solution, offering performance, power manageability, first-pass silicon success, and testability for production.

Optimizing ultra-low-power streaming sensors with MIPI

Low-area D-PHY TX⁺ IP also allowed ams-OSRAM to get everything for the Mira050 sensor, including the active pixel array, on a single in-house designed chip. Their solution uses stacked layers, one for pixels and another for the readout and other logic. The pixel layer defines an overall footprint for the logic layer, and the more compact MIPI interface helps with a fit. The chip drops into a chip-scale package, not much larger than the die's outer dimensions, optimizing the overall footprint for the sensor.

Image processing takes over the next level of optimization. Unsurprisingly, the sensor has 'full-on' and 'full standby' modes. However, the Mira050 can optimize sensor resolution, sampling, and frame rate to settings between the two extremes, keeping the MIPI transfer rate synchronized accordingly. Effectively, the sensor can adapt to its conditions, using only enough power to detect motion frame-to-frame.

Event detection starts with tiling one or more regions of interest in the sensor's field of view. Within a tile, undersampling reduces the number of pixels by as much as a 1:8 ratio while still covering the region of interest. Frame rates also adjust down, lowering the data transfer rate required. A reference frame sample contains an image with no motion. Processing compares each new incoming frame to the reference, adjusting thresholds to ensure that only larger objects meet the criteria for motion, helping prevent random background motion such as wind or small animals from triggering the sensor.

The idea is to use only as much power as required to detect motion reliably and stay at that reduced power level indefinitely until detection. In some conditions, a single-digit frame rate may suffice. When a tile is flagged as changed, the sensor moves to a higher sampling mode – possibly not full-on, depending on conditions. ams-OSRAM reports that their full-on power is 75mW, full standby power is 60uW, and reliable motion detection is possible using as little as 3mW.

^{Click to enlarge}

Power optimization is not the only achievement of the Mira050, although it is undoubtedly significant. Note the 1.7x reduction shown next is the savings at full-power mode compared to previous generations. Footprint reduction is up to 3x, which translates to smaller device designs. Sensitivity is higher thanks in part to the isolated pixel layer. Visual and near-infrared performance is nearly identical to previous generations. In short, cutting power came without compromises in other metrics.

Use cases for ultra-low-power streaming sensors

Home security cameras are an excellent example of a use case for ultra-low-power streaming sensors with MIPI. Some other use cases include:

• Consumer robotics, where key features include indoor and outdoor mapping, object detection and collision avoidance, and remote image monitoring through a smartphone app
• E-door locks with biometric face authentication, moving from standby to person detection to high-resolution 3D mapping modes smoothly
• AR/VR wearables, with tight constraints on imaging subsystem size and battery life, enhancing eye, head, and face tracking capability

^{Click to enlarge}

Any use case combining the elements of a small physical footprint and reduced power consumption yet requiring quality image processing for object classification and event detection can benefit from MIPI integration. Mixel's MIPI IP goes well beyond delivering high-speed image transfers, supporting low-power operation modes, shrinking device footprints, and increasing testability for better production yields. Mixel continues pushing state-of-the-art MIPI IP for integration in consumer and industrial applications and tracking advances in MIPI specifications and industry-recognized quality frameworks, which IP must meet for customers to achieve system certifications.

Visit the Mixel website for more information on Mixel MIPI IP, including the Mixel D-PHY TX+ solution for ultra-low-power streaming sensors with MIPI.

If you wish to download a copy of this white paper, click here