learning3d

Learning3D: A Modern Library for Deep Learning on 3D Point Clouds Data.

Documentation | Blog | Demo

Learning3D is an open-source library that supports the development of deep learning algorithms that deal with 3D data. The Learning3D exposes a set of state of art deep neural networks in python. A modular code has been provided for further development. We welcome contributions from the open-source community.

Latest News:

1. [24 Oct, 2023]: MaskNet++ is now a part of learning3d library.
2. [12 May, 2022]: ChamferDistance loss function is incorporated in learning3d. This is a purely pytorch based loss function.
3. [24 Dec. 2020]: MaskNet is now ready to enhance the performance of registration algorithms in learning3d for occluded point clouds.
4. [24 Dec. 2020]: Loss based on the predicted and ground truth correspondences is added in learning3d after consideration of Correspondence Matrices are Underrated paper.
5. [24 Dec. 2020]: PointConv, latent feature estimation using convolutions on point clouds is now available in learning3d.
6. [16 Oct. 2020]: DeepGMR, registration using gaussian mixture models is now available in learning3d
7. [14 Oct. 2020]: Now, use your own data in learning3d. (Check out UserData functionality!)

Available Computer Vision Algorithms in Learning3D

Sr. No.	Tasks	Algorithms
1	Classification	PointNet, DGCNN, PPFNet, PointConv
2	Segmentation	PointNet, DGCNN
3	Reconstruction	Point Completion Network (PCN)
4	Registration	PointNetLK, PCRNet, DCP, PRNet, RPM-Net, DeepGMR
5	Flow Estimation	FlowNet3D
6	Inlier Estimation	MaskNet, MaskNet++

Available Pretrained Models

1. PointNet
2. PCN
3. PointNetLK
4. PCRNet
5. DCP
6. PRNet
7. FlowNet3D
8. RPM-Net (clean-trained.pth, noisy-trained.pth, partial-pretrained.pth)
9. DeepGMR
10. PointConv (Download from this link)
11. MaskNet
12. MaskNet++ / MaskNet2

Available Datasets

1. ModelNet40

Available Loss Functions

1. Classification Loss (Cross Entropy)
2. Registration Losses (FrobeniusNormLoss, RMSEFeaturesLoss)
3. Distance Losses (Chamfer Distance, Earth Mover's Distance)
4. Correspondence Loss (based on this paper)

Technical Details

Supported OS

1. Ubuntu 16.04
2. Ubuntu 18.04
3. Ubuntu 20.04.6
4. Linux Mint

Requirements

1. CUDA 10.0 or higher
2. Pytorch 1.3 or higher
3. Python 3.8

How to use this library?

Important Note: Clone this repository in your project. Please don't add your codes in "learning3d" folder.

1. All networks are defined in the module "models".
2. All loss functions are defined in the module "losses".
3. Data loaders are pre-defined in data_utils/dataloaders.py file.
4. All pretrained models are provided in learning3d/pretrained folder.

Documentation

B: Batch Size, N: No. of points and C: Channels.

Use of Point Embedding Networks:

from learning3d.models import PointNet, DGCNN, PPFNet
pn = PointNet(emb_dims=1024, input_shape='bnc', use_bn=False)
dgcnn = DGCNN(emb_dims=1024, input_shape='bnc')
ppf = PPFNet(features=['ppf', 'dxyz', 'xyz'], emb_dims=96, radius='0.3', num_neighbours=64)

Sr. No.	Variable	Data type	Shape	Choices	Use
1.	emb_dims	Integer	Scalar	1024, 512	Size of feature vector for the each point
2.	input_shape	String	-	'bnc', 'bcn'	Shape of input point cloud
3.	output	tensor	BxCxN	-	High dimensional embeddings for each point
4.	features	List of Strings	-	['ppf', 'dxyz', 'xyz']	Use of various features
5.	radius	Float	Scalar	0.3	Radius of cluster for computing local features
6.	num_neighbours	Integer	Scalar	64	Maximum number of points to consider per cluster

Use of Classification / Segmentation Network:

from learning3d.models import Classifier, PointNet, Segmentation
classifier = Classifier(feature_model=PointNet(), num_classes=40)
seg = Segmentation(feature_model=PointNet(), num_classes=40)

Sr. No.	Variable	Data type	Shape	Choices	Use
1.	feature_model	Object	-	PointNet / DGCNN	Point cloud embedding network
2.	num_classes	Integer	Scalar	10, 40	Number of object categories to be classified
3.	output	tensor	Classification: Bx40, Segmentation: BxNx40	10, 40	Probabilities of each category or each point

Use of Registration Networks:

from learning3d.models import PointNet, PointNetLK, DCP, iPCRNet, PRNet, PPFNet, RPMNet
pnlk = PointNetLK(feature_model=PointNet(), delta=1e-02, xtol=1e-07, p0_zero_mean=True, p1_zero_mean=True, pooling='max')
dcp = DCP(feature_model=PointNet(), pointer_='transformer', head='svd')
pcrnet = iPCRNet(feature_moodel=PointNet(), pooling='max')
rpmnet = RPMNet(feature_model=PPFNet())
deepgmr = DeepGMR(use_rri=True, feature_model=PointNet(), nearest_neighbors=20)

Sr. No.	Variable	Data type	Choices	Use	Algorithm
1.	feature_model	Object	PointNet / DGCNN	Point cloud embedding network	PointNetLK
2.	delta	Float	Scalar	Parameter to calculate approximate jacobian	PointNetLK
3.	xtol	Float	Scalar	Check tolerance to stop iterations	PointNetLK
4.	p0_zero_mean	Boolean	True/False	Subtract mean from template point cloud	PointNetLK
5.	p1_zero_mean	Boolean	True/False	Subtract mean from source point cloud	PointNetLK
6.	pooling	String	'max' / 'avg'	Type of pooling used to get global feature vectror	PointNetLK
7.	pointer_	String	'transformer' / 'identity'	Choice for Transformer/Attention network	DCP
8.	head	String	'svd' / 'mlp'	Choice of module to estimate registration params	DCP
9.	use_rri	Boolean	True/False	Use nearest neighbors to estimate point cloud features.	DeepGMR
10.	nearest_neighbores	Integer	20/any integer	Give number of nearest neighbors used to estimate features	DeepGMR

Use of Inlier Estimation Network (MaskNet):

from learning3d.models import MaskNet, PointNet, MaskNet2
masknet = MaskNet(feature_model=PointNet(), is_training=True) masknet2 = MaskNet2(feature_model=PointNet(), is_training=True)

Sr. No.	Variable	Data type	Choices	Use
1.	feature_model	Object	PointNet / DGCNN	Point cloud embedding network
2.	is_training	Boolean	True / False	Specify if the network will undergo training or testing

Use of Point Completion Network:

from learning3d.models import PCN
pcn = PCN(emb_dims=1024, input_shape='bnc', num_coarse=1024, grid_size=4, detailed_output=True)

Sr. No.	Variable	Data type	Choices	Use
1.	emb_dims	Integer	1024, 512	Size of feature vector for each point
2.	input_shape	String	'bnc' / 'bcn'	Shape of input point cloud
3.	num_coarse	Integer	1024	Shape of output point cloud
4.	grid_size	Integer	4, 8, 16	Size of grid used to produce detailed output
5.	detailed_output	Boolean	True / False	Choice for additional module to create detailed output point cloud

Use of PointConv:

Use the following to create pretrained model provided by authors.

from learning3d.models import create_pointconv
PointConv = create_pointconv(classifier=True, pretrained='path of checkpoint')
ptconv = PointConv(emb_dims=1024, input_shape='bnc', input_channel_dim=6, classifier=True)

OR
Use the following to create your own PointConv model.

PointConv = create_pointconv(classifier=False, pretrained=None)
ptconv = PointConv(emb_dims=1024, input_shape='bnc', input_channel_dim=3, classifier=True)

PointConv variable is a class. Users can use it to create a sub-class to override create_classifier and create_structure methods in order to change PointConv's network architecture.

Sr. No.	Variable	Data type	Choices	Use
1.	emb_dims	Integer	1024, 512	Size of feature vector for each point
2.	input_shape	String	'bnc' / 'bcn'	Shape of input point cloud
3.	input_channel_dim	Integer	3/6	Define if point cloud contains only xyz co-ordinates or normals and colors as well
4.	classifier	Boolean	True / False	Choose if you want to use a classifier with PointConv
5.	pretrained	Boolean	String	Give path of the pretrained classifier model (only use it for weights given by authors)

Use of Flow Estimation Network:

from learning3d.models import FlowNet3D
flownet = FlowNet3D()

Use of Data Loaders:

from learning3d.data_utils import ModelNet40Data, ClassificationData, RegistrationData, FlowData
modelnet40 = ModelNet40Data(train=True, num_points=1024, download=True)
classification_data = ClassificationData(data_class=ModelNet40Data())
registration_data = RegistrationData(algorithm='PointNetLK', data_class=ModelNet40Data(), partial_source=False, partial_template=False, noise=False)
flow_data = FlowData()

Sr. No.	Variable	Data type	Choices	Use
1.	train	Boolean	True / False	Split data as train/test set
2.	num_points	Integer	1024	Number of points in each point cloud
3.	download	Boolean	True / False	If data not available then download it
4.	data_class	Object	-	Specify which dataset to use
5.	algorithm	String	'PointNetLK', 'PCRNet', 'DCP', 'iPCRNet'	Algorithm used for registration
6.	partial_source	Boolean	True / False	Create partial source point cloud
7.	partial_template	Boolean	True / False	Create partial template point cloud
8.	noise	Boolean	True / False	Add noise in source point cloud

Use Your Own Data:

from learning3d.data_utils import UserData
dataset = UserData(application, data_dict)

Sr. No.	Application	Required Key	Respective Value
1.	'classification'	'pcs'	Point Clouds (BxNx3)
		'labels'	Ground Truth Class Labels (BxN)
2.	'registration'	'template'	Template Point Clouds (BxNx3)
		'source'	Source Point Clouds (BxNx3)
		'transformation'	Ground Truth Transformation (Bx4x4)
3.	'flow_estimation'	'frame1'	Point Clouds (BxNx3)
		'frame2'	Point Clouds (BxNx3)
		'flow'	Ground Truth Flow Vector (BxNx3)

Use of Loss Functions:

from learning3d.losses import RMSEFeaturesLoss, FrobeniusNormLoss, ClassificationLoss, EMDLoss, ChamferDistanceLoss, CorrespondenceLoss
rmse = RMSEFeaturesLoss()
fn_loss = FrobeniusNormLoss()
classification_loss = ClassificationLoss()
emd = EMDLoss()
cd = ChamferDistanceLoss()
corr = CorrespondenceLoss()

Sr. No.	Loss Type	Use
1.	RMSEFeaturesLoss	Used to find root mean square value between two global feature vectors of point clouds
2.	FrobeniusNormLoss	Used to find frobenius norm between two transfromation matrices
3.	ClassificationLoss	Used to calculate cross-entropy loss
4.	EMDLoss	Earth Mover's distance between two given point clouds
5.	ChamferDistanceLoss	Chamfer's distance between two given point clouds
6.	CorrespondenceLoss	Computes cross entropy loss using the predicted correspondence and ground truth correspondence for each source point

To run codes from examples:

1. Copy the file from "examples" folder outside of the directory "learning3d"
2. Now, run the file. (ex. python test_pointnet.py)

• Your Directory/Location
  • learning3d
  • test_pointnet.py

References:

1. PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
2. Dynamic Graph CNN for Learning on Point Clouds
3. PPFNet: Global Context Aware Local Features for Robust 3D Point Matching
4. PointConv: Deep Convolutional Networks on 3D Point Clouds
5. PointNetLK: Robust & Efficient Point Cloud Registration using PointNet
6. PCRNet: Point Cloud Registration Network using PointNet Encoding
7. Deep Closest Point: Learning Representations for Point Cloud Registration
8. PRNet: Self-Supervised Learning for Partial-to-Partial Registration
9. FlowNet3D: Learning Scene Flow in 3D Point Clouds
10. PCN: Point Completion Network
11. RPM-Net: Robust Point Matching using Learned Features
12. 3D ShapeNets: A Deep Representation for Volumetric Shapes
13. DeepGMR: Learning Latent Gaussian Mixture Models for Registration
14. CMU: Correspondence Matrices are Underrated
15. MaskNet: A Fully-Convolutional Network to Estimate Inlier Points
16. MaskNet++: Inlier/outlier identification for two point clouds