Kornia 0.6 — High Level Computer Vision for AI

Kornia is a modern set of Computer Vision libraries built on top of PyTorch that implements curated low level Computer Vision algorithms for the areas of robotics, camera localization and AR.

To learn more about the library and our ecosystem visit our Documentation and dive into our interactive Tutorials to play with our functionality.

Kornia v0.6

In this release we have focused on providing more high level APIs for the most demanded tasks in our community:

Image Registration

We provide the ImageRegistrator API from which you can leverage to align automatically two images by a process of direct optimisation using the PyTorch Autograd differentiability.

Sequence of narrow Depth-Of-Field macro photos of the bee are registered with kornia ImageRegistrator. Then their sharp parts are merged together to obtain whole bee in focus.

Homography Tracking

We added kornia.tracking module, which now contains HomographyTracker. You can use it to perform planar object tracking, which is useful for creating augmented reality apps. We have created an example of such application here:

Book position is tracked with HomographyTracker and the GIF is placed on top of it.

Image Matching

We provide many modules and functions for the image matching: from building blocks like local feature detectors, descriptors, descriptor matching, geometric model estimation

However we recommend to start with high-level API, such as LoFTR or LocalFeatureMatcher you can use to find correspondence between two images which uses a #transformers model.

You also can go through a full tutorial using Colab found here or try with HuggingFace spaces here.

Image correspondences found by LoFTR

Image Stitching

Image stitching is the process of combining multiple images with overlapping fields of view to produce a segmented panorama. Here, we provide ImageStitcher to easily stitch a number of images.

Input images for the panorama, source: https://github.com/daeyun/Image-Stitching

Panorama created with kornia ImageStitcher.

RANSAC

You know what is common between examples of image stitching, image matching and homography tracking in kornia? They use our freshly-introduced RANSAC module for robust estimation of geometric transformations like homography and epipolar geometry. While it is not (yet!) as fancy as the latest OpenCV RANSAC, it does run on GPU in parallel mode and is good enough for many practical applications.

Image Augmentation

Kornia leverages differentiable and GPU image data augmentation through the module kornia.augmentation by implementing the functionality to be easily used with torch.nn.Sequential and other advanced containers such as AugmentationSequential, ImageSequential, PatchSequential and VideoSequential.

Images augmented with PatchSequential module.

One can use the kornia apply for simple augmentations as follows:

Or use the differentiability to optimise the augmentations parameters as follows:

Extras

Additionally, this new release comes with a wide set of new functionalities and state-of-the-art models, operators and loss functions. Below you can find a triaged set of what you will find in the release.

• Experimental Training API with ImageClassification and SemanticSegmentation trainer classes including coding template examples that can be used as a reference to train your models.
• Video Deblurring features via DeFMO,
• Vision transformer and MobileVIT implemented in terms of generic operator to be used across different tasks or applications.
• Hausdorff distance loss for morphological erosion optimization.
• Homography metrics to evaluate or optimise.
• Connected components for classical image segmentation.
• Stereo camera API working with a horizontal stereo camera setup.
• Camera Calibration module, including image undistortion and PnP algorithms.
• Canny edge detection algorithm.
• CLAHE differentiable to equalize image using local optimization.
• Resize with antialias implementation.
• Integration with VPU via OpenCV-Luxonis programmable cameras [here].
• Added support for YUV420 and YUV422 color conversions.
• Implement separable_filter2d to boost x10 filtering operations.
• Improved our morphology package

We recommend to read carefully our release notes for 0.5.x [here]

Next steps

Our current focus is on the following:

• Optimize and speed-up existing algorithms like Homography tracker, RANSAC, connected components and many others.
• Adding new features in terms of high-level algorithms, which can be plugged in user applications off-the-shelf.
• Expand the functionality of existing image matching, tracking and structure-from-motion modules.
• Work towards a Camera API.

Join the community

• Join our social network communities with 1.8k+ members:
• Twitter: we share the recent research and news for out mainstream community.
• Slack: come to us and chat with our engineers and mentors to get support and resolve your questions.
• LibreCV: its our Open Source and Machine Learning community forum. Come and have fun !
• Subscribe to our YouTube channel to get the latest video demos.

Support

Kornia AI is a non-profit organization project without large funding in the background. If you use kornia, consider giving us support with resources, contributions and donations.

You can use our OpenCollective link for financial support: https://opencollective.com/kornia

If you need our help for adopting kornia in your commercial product, or implementing some special feature, courseware and more advanced tiers reach us at contact@kornia.org

For general technical support, please use our slack Slack and Github issues.