Connections between physics and deep learning

You should watch Max Tegmark's talk "Connections between Physics and Deep Learning" if this interests you [1]. Additionally, a paper that has everyone excited about deep connections between the mathematical analysis of physical systems and the hierarchical feature learning paradigm speaks of the connection in terms of the Renormalization Group [2]. Regardless of it's practical utility, the philosophical implications do tickle the intellect. On a dreamy note, I wonder if it would be possible to draw Category-theoretic parallels between some physical theories and statistical learning theory. There is so much to learn, and I am trying my best to teach myself (on the side) the mathematics that they don't teach in my CS grad school. So much to learn, such little time. :) [1] [VIDEO] [2] [LINK]

According to a talk by Max Tegmark[0] (and its associated paper[1]), neural nets (particularly LSTMs) might be inherently better at this sort of thing due to the way they model mutual information. Markov models are best suited to situations where an observation k-steps in the past gives exponentially less information about the present[2] (decaying according to something like λ^k for 0 <= λ < 1). Intuitively, the amount of context imparted by a word or phrase decays somewhat more slowly. That is, if I know the previous five words, I can make a good prediction about the next one, and likely the next one, and slightly less likely the one after that, whereas in a Markovian setting my confidence in my predictions should decay much more quickly. So in answer to the grandparent, such a thing should be reasonably straightforward to build if it doesn't exist already, and it may offer improvements over a similar model based on Markov chains. --- 0. [VIDEO] 1. [LINK] 2. Why is this? Lin & Tegmark offer details in the paper, but it comes from the fact that the singular values of the transition matrix are all less than or equal to one (an aperiodic & ergodic transition matrix has only one singular value equal to one), and so the other singular vectors fall away exponentially quickly, with the exponent's base being their corresponding singular value.

I think the argument for using physics models for constraints in a DL system was a more clear argument [VIDEO]

My random stream-of-consciousness reference reactions ... Tegmark & Lin's discussion of how deep learning maps onto the physical world: Why does deep and cheap learning work so well? [1][2] The work of Aerts et al. more than 10 years ago, including how vector space models of human categorization show QM structure [3]. Lucien Hardy's exquisite teasing out of the difference between classical and quantum probability: Quantum Theory From Five Reasonable Axioms [4] It turns out that the innovation, power and strangeness of QM is related to separating physical processes into: 1. Linear continuous unitary reversible evolution relying on complex amplitudes (wavefunction propagation). 2. Non-linear discontinuous irreversible 'collapse' or 'measurement' or 'entanglement' or 'correlated branching', with probabilities based on the Born Rule (real values from the square of wave function amplitudes). Neural nets also decompose a problem into successive alternating layers of reversible continuous functions and discrete irreversible categorical decisions (e.g. softmax/sigmoid logistic classifiers). An even more obscure tangent is that most of the financial market is now constructed from 'options', which track continuous values with a continuous payoff, but offer a choice of execution to collapse the option and create a real financial result, e.g. see N.N.Taleb's dicussion of optionality [5]. [1] [LINK] [2] [VIDEO] [3] [LINK] [4] [LINK] [5] [LINK]

Here's a related 2016 talk by Max Tegmark (second author) on connections between deep learning and physics: [VIDEO] The gist of it is that physical data tends to have symmetries, and these symmetries make descriptions of the data very compressible into relatively small neural circuits. Random data does not have this property, and cannot be learned easily. Super fascinating.

See Tegmark's "why does deep cheap learning work so well" (well not so cheap anymore...) [VIDEO]