Google’s Astrophotography mode reveals the Milky Way
The Pixel 4’s Astrophotography mode wowed audiences at its October debut; the camera’s unique ability to capture the night sky in stunning detail complete with stars and even the Milky Way seeming almost like magic. Now two of Google’s software engineers, Florian Kainz and Kiran Murthy, have written a detailed Google AI Blog explaining how the new feature works and revealing the key techniques behind the software sorcery.
If you’re more of a pictures person, you can take a look at these examples of Kainz’s stunning Astrophotography taken with the Pixel 4 and if you’d like to have a go for yourself, you can download this handy PDF of tips and tricks.
The Challenge Of Night Photography
Astrophotography is a form of low light photography which comes with significant and very specific technical challenges.
In conventional photography, capturing high-quality photos of a starry sky typically requires large, unwieldy lenses to funnel in as much light as possible coupled to bulky camera bodies with large and highly sensitive imaging sensors to capture them efficiently.
Mobile phones, due to their size, can offer neither of these attributes – and this is where Google’s computational photography skills come to the fore.
Astrophotography Mode is an extension of Google’s Night Sight mode, which leverages numerous techniques, including merging multiple shots with a liberal smattering of AI to capture bright, pleasing images in very low-light situations.
The Unique Challenge Of Astrophotography
When shooting with Night Sight activated, the Pixel phone will automatically engage Astrophotography mode if the phone is mounted on a tripod or other steady surface. In this mode, the camera will boost the amount of light captured by shooting many more individual shots, each of which can last up to 16 seconds in duration. The Pixel 4 will spend up to four minutes capturing an astrophotography image, with the Pixel 3 and 3a both limited to one minute.
Four minutes was chosen deliberately as the maximum length of time users would typically be prepared to wait for each photo, while the maximum individual shot duration is limited by the movement of the stars themselves – any longer than 16 seconds per shot will result in blurred lines rather than points of light.
(If you have more patience than most, you can extend these limits by downloading Cstark27’s modified Google Camera app, which will allow capture times of up to six and a half minutes as well as enabling deliberate motion blur effects.)
Capturing such long exposures has an unwanted side-effect of magnifying flaws within the camera’s sensor. These show up as unwanted colored spots that must be removed. Google’s camera app detects these spots automatically and deletes them by averaging the colors of the surrounding pixels.
The app also enhances the viewfinder experience on the phone’s screen by switching into a new ‘post-shutter’ mode to help with image composition. The traditional real-time ‘video feed’ through the lens becomes unusably dark in extreme low light environments, so Astrophotography mode displays the results of each individual shot after it has been captured enabling you to see more clearly what you are capturing.
Focusing the camera also becomes much more difficult in the dark, so Astrophotography mode captures two extra frames at the start of the process which are used purely to assist in focusing the camera and discarded before creating the final picture.
Enhanced Sky Processing
Night Sight uses AI to determine which pixels in your photo form part of the sky. It then processes these pixels selectively in order to ensure a natural-looking night sky which isn’t too bright. Astrophotography mode adds to this by looking out for features such as clouds and the Milky Way and then tweaking the contrast to make them stand out more. Thereby performing automatically the same sort of edits an Astrophotographer might make in external editing software. This is how Astrophotography mode is able to take such striking pictures of the Milky Way which wouldn’t ordinarily be visible to the human eye.
As mentioned in the Pixel 4 keynote, there’s still further to go. Astrophotography mode currently relies on moonlight to illuminate the landscape when photographing the stars. Future iterations will be able to capture a visible landscape even with a moonless sky.
With any luck, the Google team will soon return with an even more impressive gallery of photos.