Abstract

The classical simple gradient descent used in Deep Learning has two drawbacks: the use of the same non-adaptive learning rate for all parameter components, and a non-invariance with respect to parameter re-encoding inducing different learning rates. As the parameter space of multilayer networks forms a Riemannian space equipped with Fisher information metric, instead of the usual gradient descent method, the natural gradient or Riemannian gradient method, which takes account of the geometric structure of the Riemannian space, is more effective for learning. The natural gradient preserves this invariance to be insensitive to the characteristic scale of each parameter direction. The Fisher metric defines a Riemannian metric as the Hessian of two dual potential functions (the Entropy and the Massieu Characteristic Function).

In Souriau’s Lie groups thermodynamics, the invariance by re-parameterization in information geometry has been replaced by invariance with respect to the action of the group. In Souriau model, under the action of the group, the entropy and the Fisher metric are invariant. Souriau defined a Gibbs density that is covariant under the action of the group. The study of exponential densities invariant by a group goes back to the work of Muriel Casalis in her 1990 thesis. The general problem was solved for Lie groups by Jean-Marie Souriau in Geometric Mechanics in 1969, by defining a “Lie groups Thermodynamics” in Statistical Mechanics. These new tools are bedrocks for Lie Group Machine Learning. Souriau introduced a Riemannian metric, linked to a generalization of the Fisher metric for homogeneous Symplectic manifolds. This model considers the KKS 2 -form (Kostant-Kirillov-Souriau) defined on the coadjoint orbits of the Lie group in the non-null cohomology case, with the introduction of a Symplectic cocycle, called “Souriau’s cocycle”, characterizing the non-equivariance of the coadjoint action (action of the Lie group on the moment map).

We will introduce the link between Souriau “Lie Groups Thermodynamics”, Information Geometry and Kirillov representation theory to define probability densities as Souriau covariant Gibbs densities (density of Maximum of Entropy). We will illustrate this case for the matrix Lie group SU (1,1) (case with null cohomology), and the one for the matrix Lie group SE(3) (case with non-null cohomology), through the computation of Souriau’s moment map, and Kirillov’s orbit method.

BIO: F. Barbaresco received his State Engineering degree from the French Grand Ecole CENTRALE -SUPELEC, Paris, France, in 1991. Since then, he has worked for the THALES Group where he is now SENSING Segment Leader of Key Technology Domain PCC (Processing, Control & Cognition). He has been an Emeritus Member of SEE since 2011 and he was awarded the Aymé Poirson Prize (for application of sciences to industry) by the French Academy of Sciences in 2014, the SEE Ampere Medal in 2007, the Thévenin Prize in 2014 and the NATO SET Lecture Award in 2012. He is President of SEE Technical Club ISIC “Engineering of Information and Communications Systems” and a member of the SEE administrative board. He is member of the administrative board of SMAI and GRETSI . He was an invited lecturer for UNESCO on “Advanced School and Workshop on Matrix Geometries and Applications” in Trieste at the ITCP in June 2013. He is the General Co-chairman of the new international conference GSI “Geometric Sciences of Information”. He was co-editor of MDPI Entropy Books “Information, Entropy and Their Geometric Structures” and “Joseph Fourier 250th Birthday: Modern Fourier Analysis and Fourier Heat Equation in Information Sciences for the XXIst century”. He has co-organized the CIRM seminar TGSI ’17 “Topological and Geometrical Structures of Information” and “FGSI’19 Cartan-Koszul-Souriau” in 2019. He was keynote speaker at SOURIAU ’19 event.