Camera

From postmarketOS Wiki
Note This page is written to the best (but limited) knowledge. If you have any improvements to the text, feel free to edit the page and correct or expand it. Contributions welcome!

A smartphone normally has multiple cameras (also named sensors) which are made by companies such as Sony, OmniVision or Samsung.

These are normally controlled via an I²C bus and deliver image data via MIPI CSI-2 (Camera Serial Interface). Some sensors are connected via a parallel interface rather than serial (CSI-2) which seems to be called digital video port (DVP). This fact is currently mostly ignored in this article and covers CSI-2 only for now. Many of the same concepts should apply to parallel also.

In many cases the sensor package also comes with additional components:

  • VCM (voice coil motor) actuator which can physically move the lens for focusing, image stabilization or optical zoom.
  • EEPROM which presumably contains some calibration information and some other details

A sensor normally has the following connections to the SoC:

  • A clock input, commonly named MCLK or XVCLK
  • Several power inputs, commonly named e.g. AVDD (analog supply), DVDD (digital supply), DOVDD/IOVCC (I/O supply)
  • One or more GPIOs for reset and/or shutdown purposes. These are not used in some setups when e.g. the reset pin is connected to DOVDD, so when the power supplies are on, the sensor is on.
  • Two pins for the I²C connection (SDA & SCL). Also named SCCB (serial camera control bus) or CCI (camera control instance or camera control interface) or just 2-wire serial communication
  • Between 3 and 9 pins for MIPI CSI-2 connection (1 pin for the clock lane, 2 pins for each of the 1-4 data lanes depending on the sensor and setup)

On SoC side the following components exist:

  • I²C bus (one or more, sometimes shared between multiple camera sensors, sometimes one camera sensor per bus)
  • Receiver of the physical MIPI CSI-2 signals (multiple in most cases)
  • Various other blocks for (efficient) image processing

Some sensors support lane remapping or lane swap for the MIPI CSI-2 connection, when the hardware design requires that. Also the sensor must be configured for the right number of lanes from the driver side (declared in data-lanes property) so that it sends the data correctly.

Camera Sensor Basics

Sensor example

An example for a sensor is below where we provide the before mentioned clocks, supplies, GPIOs. The pinctrl makes sure the MCLK0 pin on the SoC is set to the correct function.

The sensor node can also provide orientation (front or rear camera) and rotation (0°, 90°, 180°, 270°) which define metadata for user space.

You can also link lens-focus and flash-leds to the VCM and flash LED respectively. This information can (or at least could) be used by user space to e.g. trigger the flash when you take a picture with the rear camera.

For more information about these properties, see Documentation/devicetree/bindings/media/video-interface-devices.yaml. 

&cci_i2c0 {
	ov8865: camera-sensor@10 {
		compatible = "ovti,ov8865";
		reg = <0x10>;

		pinctrl-0 = <&mclk0_pin_a>;
		pinctrl-names = "default";

		clocks = <&mmcc CAMSS_MCLK0_CLK>;

		avdd-supply = <&pm8941_l17>;
		dovdd-supply = <&pm8941_lvs3>;
		dvdd-supply = <&pm8941_l3>;

		powerdown-gpios = <&tlmm 89 GPIO_ACTIVE_LOW>;
		reset-gpios = <&tlmm 90 GPIO_ACTIVE_LOW>;

		orientation = <1>; /* Rear camera */
		rotation = <90>;

		lens-focus = <&ad5823>;
		flash-leds = <&pm8941_flash>;

		port {
			ov8865_ep: endpoint {
				data-lanes = <1 2 3 4>;
				bus-type = <MEDIA_BUS_TYPE_CSI2_DPHY>;
				link-frequencies = /bits/ 64 <360000000>;
				remote-endpoint = <&csiphy0_ep>;
			};
		};
	};
};

Sensors status

Sensor Parent Driver Status Found in Notes
Hynix Hi-556 5MP sr556 hynix,sr556 WIP sdm450-samsung-a6plte-r4
OmniVision OV5670 5MP ov5670 ovti,ov5670 Y msm8953-xiaomi-markw
OmniVision OV5675 5MP ov5675 ovti,ov5675 Y msm8953-xiaomi-daisy
msm8953-xiaomi-vince
OmniVision OV8856 8MP ov8856 ovti,ov8856 sm7150-xiaomi-davinci Untested, but the probe is successful
OmniVision OV8865 8MP ov8865 ovti,ov8865 Y msm8916-wingtech-wt88047 Usage in Redmi 2: https://github.com/msm8916-mainline/linux/pull/345
Samsung ISOCELL 3L6 13MP s5k3l6 samsung,s5k3l6xx WIP imx8mq-librem5
sm7150-xiaomi-davinci		 sm7150-xiaomi-davinci is untested, driver probes with changes to allow 19.2MHz clock
Samsung ISOCELL 3L8 13MP s5k2xx samsung,s5k3l8 WIP msm8953-xiaomi-markw
msm8953-xiaomi-mido
Samsung ISOCELL 3P8 16MP s5k2xx samsung,s5k3p8sp WIP msm8953-xiaomi-ysl
Samsung LSI S5C73M3 8MP S5C73M3 samsung,s5c73m3 Y exynos4412-samsung-{m0,m3,t03g,t0lte} TRY syscalls need implementing (see FIXME in s5c73m3-core.c).

Qualcomm

Further information: Qualcomm Camera Subsystem

Qualcomm SoCs contain a block called CAMSS (camera subsystem). For more information about some of the components that are part of it see Qualcomm Camera Subsystem driver - The Linux Kernel documentation.

A Qualcomm SoC commonly contains somewhere between two and four 4-lane CSIPHY receivers, so normally each sensor is connected to a dedicated CSIPHY and do not share them. For bringup you can focus on this CSIPHY and probably ignore the other components of CAMSS.

The I²C bus used for the camera sensors is not a regular QUP/BLSP bus but is the CCI (camera control interface) which has a separate driver in Linux.

Example

The following example shows how the CAMSS side can be configured in dts. Each port here is a CSIPHY receiver, so the part that receives CSI-2 data. In this example CSIPHY0 and CSIPHY2 is used. CSIPHY1 is unused.

On the schematics there's 5 equal lanes per CSIPHY, where normally (e.g. for 4 data lanes) the CSI data lanes are connected to lane 0, 2, 3 and 4, while clock lane is connected to lane 1. This is represented using "clock-lanes" and "data-lanes" properties. Note: Both properties refer to the lane number and is not a count of how many lanes are used for a particular purpose. A sensor with 1 data lane would have data-lanes = <0>; and clock-lanes = <1>; on the SoC side.

For more information about these properties, see Documentation/devicetree/bindings/media/video-interfaces.yaml. 

&camss {
	vdda-supply = <&pm8941_l12>;
	
	status = "okay";

	ports {
		port@0 {
			reg = <0>;
			csiphy0_ep: endpoint {
				clock-lanes = <1>;
				data-lanes = <0 2 3 4>;
				bus-type = <MEDIA_BUS_TYPE_CSI2_DPHY>;
				link-frequencies = /bits/ 64 <360000000>;
				remote-endpoint = <&ov8865_ep>;
			};
		};
		port@2 {
			reg = <2>;
			/* [...] */
		};
	};
};

Downstream devicetree

In most cases, the downstream device tree does not adequately describe the actual hardware setup, but you can still extract some useful information from it.

  • csiphy-sd-index: which CSIPHY the sensor is connected to
  • cci-master: which CCI (I²C) bus the sensor sits on
  • clocks: the clock used for the sensor
  • gpios: various gpios (reset, shutdown, etc). Double-check with the names provided in the dts what function the gpio actually has. For example CAM_VDIG would be a fixed-regulator that provides power to one of the supply rails. The gpios named CAMIF_MCLK you can probably ignore here.

Bringup

Before bringing up the sensor you should make sure the I²C bus (CCI) works correctly. For this you can enable the bus in your dts and run i2cdetect (needs kernel module i2c-dev) command on it. This should pretty much immediately give you the result back, probably with no devices listed since everything's turned off by default. If it hangs and you get some timeout errors in dmesg you need to make sure that the pullup for the I²C bus is enabled. This is in many cases one of the regulators used for a supply for the cameras. If you're not sure, you can try enabling various regulators you think might provide this by putting regulator-always-on; on the regulator node.

Then for actualling getting the sensor up, you should first establish (manual) I2C communication with it. For this you enable the required regulators with regulator-always-on; and also force the clock for the sensor to on using regulator-fixed-clock + regulator-always-on. Also don't forget to set the pinctrl correctly otherwise the clock signal won't actually reach the sensor.

An example of forcing on mclk2: 

/ {
    fake-reg-mclk2 {
        compatible = "regulator-fixed-clock";
        regulator-name = "fake-reg-mclk2";
        pinctrl-names = "default";
        pinctrl-0 = <&cam_mclk2_default>;
        clocks = <&camcc CAM_CC_MCLK2_CLK>;
        regulator-always-on;
    };
};

&tlmm {
    cam_mclk2_default: cam-mclk2-default {
        pins = "gpio31";
        function = "cam_mclk2";
        drive-strength = <2>;
        bias-disable;
    };
};

Then when you run i2cdetect on the bus and see if a sensor appears. It's possible also multiple I2C addresses pop up, only one of them is actually the sensor, the others might be eeprom or VCM or others.

If you have access to a datasheet for the sensor, be careful as with any I2C device the address might be given as 8-bit address while Linux always uses 7-bit addresses. For example if the datasheet says the address is 0x20 it might actually be 0x10 instead (shift one bit to the right).

When you know which I2C address the sensor uses and it appears, you can try reading to the chip ID register to make sure communication really works and you're talking to the correct chip.

Since for some sensors 16-bit registers are used then reading is a bit more tricky manually but still easily possible. For example the IMX582 has register 0x0016 which holds the value 0x0582. In this example the sensor is connected to I2C bus 3 (check i2cdetect -l) and has the I2C address 0x1a. 

$ i2cset -y 3 0x1a 0x00 0x16
$ i2cget -y 3 0x1a
0x05
$ i2cget -y 3 0x1a
0x82

TODO: Expand on CSI-2 stuff and what to do after i2c communication is established. ( + Some drivers in Linux support configuration of the number of CSI-2 lanes (data-lanes) but some do not.)

Testing

TODO: introduce manual exposure and gain with v4l-ctl, "media-ctl -p", cam, bayer format, test patterns in ISP and sensor, v4l-ctl for flash/torch, example for qcam usage that works on FP2

libcamera

To troubleshoot the libcamera stack there are a set of useful utilities: libcamera-tools qcam. Most useful commands include:

  • cam --list Looks for cameras in different locations, printing useful debug output and finish with a list of cameras it found. The number at the beginning of found cameras shall be used to identify cameras in later commands
  • cam --camera #num --list-properties In case a camera is found, this command lists the properties of the specific camera. It is usually common to look for the Location and Rotation properties, as they are often missing from the device trees
  • cam --camera #num --info Lists all modes reported by the camera. This can be useful to troubleshoot missing resolutions on formats
  • qcam Opens a new window that allows selecting the right camera to test a direct video streaming from libcamera. This command needs to be run in a terminal with the whole environment setup

In addition to that, as of July 2024, libcamera requires big contiguous-memory-address (CMA) buffers. Their size can be checked with cat /proc/meminfo | grep Cma. They should usually be over the size of hundred of megabytes. These can be configured in 2 ways:

  • Adding cma=256M to the kernel commandline. For testing, it can be added to the the append variable in /etc/deviceinfo, after which it is needed to run sudo mkinitfs
  • Modifying CONFIG_CMA_SIZE_MBYTES in the kernel config (proposal to do it by default: pmaports#3368)

Routing

Some camera-related drivers (such as the Qualcomm Camera Subsystem) use the kernel media controller API to expose routing controls to userspace. Routing can be done manually using the media-ctl command.

The routing entities have names and pads. All entities can be listed with this information using the command: 

$ media-ctl -p
Media controller API version ...

Media device information
------------------------
...

Device topology
- entity 1: msm_csiphy0 (2 pads, 4 links)
            ...
 	pad0: Sink
		[fmt:UYVY8_2X8/1920x1080 field:none colorspace:srgb]
		<- "imx355 4-001a":0 [ENABLED,IMMUTABLE]
	pad1: Source
		[fmt:UYVY8_2X8/1920x1080 field:none colorspace:srgb]
		-> "msm_csid0":0 []
		-> "msm_csid1":0 []
		-> "msm_csid2":0 []
...

In this example, there is an enabled link between entity "msm_csiphy0", pad 0, and entity "imx355 4-001a", pad 0.

Links can be enabled and disabled with the command media-ctl -l '"ENTITY":PAD->"ENTITY":PAD[ENABLED]'. Here is an example that enables then disables the link between msm_csiphy0 (pad 1) and msm_csid0 (pad 0): 

$ media-ctl -l '"msm_csiphy0":1->"msm_csid0":0[1]'
$ media-ctl -l '"msm_csiphy0":1->"msm_csid0":0[0]'

The formats should be consistent between pads, unless your driver supports format shifting. The format of the pads can be changed with media-ctl -V: 

$ media-ctl -V '"msm_csiphy0":0[fmt:SRGGB10_1X10/3280x2464]'

Apps

There are multiple camera apps available, in various states of development and functionality.

Retrieved from "https://wiki.postmarketos.org/index.php?title=Camera&oldid=67571"