Neural network feature combinatorics - machine-learning

Question

Suppose we have a neural network with a sufficiently high number of hidden layers, hidden units per layer, and iterations, such that these parameters do not affect the network's predictions.
Given features x1, x2, ..., xn, is it possible (to prove) whether or not a range of potential features are redundant given this subset of features (x1 through xn). That is, given these features (x1 through xn), can a neural network discern other features such as:
Differences or additions (x1-x49, or x17+xn)?
Products and ratios (x1*x1, or x47/xn)
Higher order polynomials (or a products of sequences like ∏(x1
through xn))
trigonometric functions based upon original features (sin(x1*xn) +
x17)
logarithmic functions (ln(x2*x4)/x6)
It is in this line of inquiry that I am wondering if there are situations using a neural network where you would need to add higher order, or different, functions for the network to predict accurately.
In general, given some adequate number of features, is it possible for the network to model ANY graph, and if not what functional domains can neural networks not predict?
Furthermore, is there any research someone could point out that references this topic?
Thanks!

Answer 1

Given features x1, x2, ..., xn, is it possible (to prove) whether or not a range of potential features are redundant given this subset of features (x1 through xn). That is, given these features (x1 through xn), can a neural network discern other features
It seems as if you are looking for a dimensionality reduction with neural networks. Autoencoders can do that:
You have inputs x1, x2, ..., xn.
You create a network which gets those inputs (n input nodes). It has some hidden layers, a bottleneck (k nodes, with k < n) and an output layer (n nodes).
The target is to recreate the input.
When it is trained, you discard the layers after the output. If the network was able to restore the inputs from the bottleneck, the later layers are not necessary.
In general, given some adequate number of features, is it possible for the network to model ANY graph, and if not what functional domains can neural networks not predict?
I guess you are looking for the Universal approximation theorem. In short: Neural networks can approximate any continuous functions on compact subsets of R^n arbitrary close as long as you give them enough nodes and at least one hidden layer

Answer 2

Long story short:
No Neural Network for regression-tasks will ever, yes sorry -- NO ANN WILL EVER -- be able to reasonably predict y_target(s) for such problem-domains, that principally do not match the implemented mathematics of the NN-model.
Trying to predict y_target(s) via ( an almost ) just linear-combination of input-layer state-vector's components ( observations of features ) in X[:] ( that, well, do get some non-linear handling down-the-network of summing their respective scalar amplifications ) have to and will fail to remain precise.
Too complex to read?
Just let me put an example.
One may use an ANN to train such a linear-combination of inputs, to best approximate a cubic ( by nature ) problem-domain behaviour. The mathematics of a minimiser-search will yield such ANN-coefficients, that will provide the lowest penalty of all any other coefficient-settings.
So far so good.
But still, such "tuned"-ANN will never be any closer to the cubic-nature of the underlying ( real-world ) phenomenon. Not because I put it here, but because the linear-combination, however the tweaking of the non-linearised factors get incorporated down the road, during re-calculating all the layers till the final sum and output transformation takes place -- all this principally cannot introduce a cubic-behaviour accross the full domain-ranges of the inputs ( which the Mother Nature, natively, in the problem-domain real-world behaviour, does -- and here is the Devil hidden and the reason, why it cannot get any better this way -- not so hard to create an easy simulation of this principal failure to meet the cubic-reality in code ).
A problem-domain-agnostic Universal ANN might be a nice wish,butwould be an overkill to implementandthe worse for an attempt to .fit()
Yes, you are right one can spend some amount of creativity, so as to create and connect a pre-ANN-black-box right in front of the ANN-input-layer, where all the possible mathematics over the native-X[:] feature observations takes place, thus feeding the ANN-input layer with all possible derived-feature-semi-products, so as to allow ANN to learn any-kind-of real-problem-domain behaviour.
This would seem to be a way, until you implement such a magic-Universal-Behaviour-Model-Black-box and realise the scales that it will enforce on the back-to-back connector, so making the input-Layer and all the hidden-layers to grow in static-scales so wide, that the reality of the resulting O( N^k ) scaling will most probably nail such attempt into etheral waiting, independently of any imaginable parallel-computing efforts, right due to [PTIME,PSPACE] complexities and a fact, that none Turing-SEQ-computing complexity-taxonomy-member will get any better even if successfully fully-translated into a PAR-computing domain ( for reasoning behind this C2-boundary problem ref. comments and citations from here ).
Even if one raises a claim, that Universal Quantum Computers ( ref. U-QC-device ) will make this scenario achieve feasible results in [CTIME,CSPACE] I would be reserved to expect such U-QC-device to become reasonably available for practical deployment any soon ( FYI: current biggest published non-U-QC-device CSPACE scales are about 1024 qbits in 2016, about 2048+ qbits in 2017 and if this progress could keep such pace forever, yet the such CSPACE constraints will keep the magic-Universal-Behaviour-Model-Black-box piggy-backed-ANN rather restrictively small to meet your above expressed expectations:
Published constant-rate CSPACE-problem scalingextented until End of this Century
As of published technical details until EoM-2017/07 the current (non)-U-QC-devices available do not seem to allow for a [CSPACE]-constrained problems to have more than just 11-input-Layer neurons, so imagine just having 11 feature-inputs possible in 2017, for such pioneering, attractive and promising technology [CTIME] ANN-answers, yet with just a QUBO-simplification here of the actually R^m-continuous-domain minimisation-problem ( details ware intentionally excluded, due to additional complexities beyond the QC-curtains ).
2011: 128-neurons -- ( from 1x "extended"-input-L,
2015: 1,024 across all many or few hidden-Ls,
2016: 2,048 up to the .. output-L )
2017: 4,096
2019: 8,192
2021: 16,384
2023: 32,768
2025: 65,536
2027: 131,072
2029: 262,144
2031: 524,288
2033: 1,048,576 - neurons - - IN[300] features
2035: 2,097,152 - first able to compute a trivial ANN
2037: 4,194,304 with just an elementary architecture of
2039: 8,388,608 QuantFX.NN_mapper( ( 300, # IN[300]
2041: 16,777,216 1200, # H1[1200]
2043: 33,554,432 600, # H2[600]
2045: 67,108,864 300, # H3[300]
2047: 134,217,728 - neurons - - IN[ 3096] feat. 1 ),# OUT[1]
2049: 268,435,456 .. )
2051: 536,870,912
2053: 1,073,741,824
2055: 2,147,483,648
2057: 4,294,967,296 - neurons - - IN[17520] features
2059: 8,589,934,592
2061: 17,179,869,184
2063: 34,359,738,368
2065: 68,719,476,736
2067: 137,438,953,472 - neurons - - IN[99080] features
2069: 274,877,906,944
2071: 549,755,813,888
2073: 1,099,511,627,776
2075: 2,199,023,255,552
2077: 4,398,046,511,104
2079: 8,796,093,022,208
2081: 17,592,186,044,416
2083: 35,184,372,088,832
2085: 70,368,744,177,664
2087: 140,737,488,355,328
2089: 281,474,976,710,656
2091: 562,949,953,421,312
2093: 1,125,899,906,842,624
2095: 2,251,799,813,685,248
2097: 4,503,599,627,370,496
2099: 9,007,199,254,740,992 - neurons - - IN[25365000] features
Reality-check:
Given the above technology limits ( be it a [PTIME,PSPACE] eternity for an O(N^k) scaling of a .fit(), where k >= 2, or a [CTIME,CSPACE] problem-scale constraints ) there is not much advantage to create such divine-Black-Box super-universal ANN-device ( and then yet have to wait decades, if not centuries, before it could be used to get first answers from ANN-on-steriods ).
The very opposite is closer to reality.
One may and shall introduce all due problem-domain analysis efforts, so as to properly identify the native-reality behaviour ( ref. Technical Cybernetics: System Identification ) for knowing in advance, how to design a just-enough feature-rich input-layer ( for which synthetic-features - higher-order powers and cross-products, sums, products, harmonics, log-/exp-s, complex- / discrete-magics et al will take place, just-where-necessary for meeting ( not exceeding ) the performed system-identification ), as the ANN-model-scaling could this way remain a right-sized structure with a following pair of systematic-certainties:
a) Removing any single part would damage the model ( missing some indispensable feature will principally cause predictions to fail to be able to meet the system-identified diversity of behaviours ).
b) Adding any single part would not improve the model ( adding any feature, that is not incorporated in the diversity of identified system-behaviour adds zero new power to the current prediction capabilities )
The just-enough complex Feature-Engineering + Right-sizingis the way to go:
|>>> nnMAP, thetaVEC, thetaGRAD, stateOfZ, stateOfA, biasIDX = QuantFX.NN_mapper( ( 300, 1200, 600, 300, 1 ), True )
INF: NN_mapper has found: 5 Layers in total ( 3 of which HIDDEN ), 300 INPUTs, 1 OUTPUTs
INF: NN_mapper has found: 300 Neurons in INPUT-Layer
INF: NN_mapper has found: 1200 Neurons in HIDDEN-Layer_1
INF: NN_mapper has found: 600 Neurons in HIDDEN-Layer_2
INF: NN_mapper has found: 300 Neurons in HIDDEN-Layer_3
INF: NN_mapper has found: 1 Neuron in OUTPUT-Layer
INF: NN_mapper : will return a COMMON-block for nn_MAP__VEC
INF: NN_mapper : will return a COMMON-block for ThetaIJ_VEC having 1262401 cells, being all random.
INF: NN_mapper : will return a COMMON-block for ThetaIJGRAD having 1262401 cells,
INF: NN_mapper : will return a COMMON-block for Z_state_VEC having 2405 cells,
INF: NN_mapper : will return a COMMON-block for A_state_VEC having 2405 cells, with BIAS units == +1
INF: NN_mapper : will return BIAS units' linear addresses in biasIDX vector
: for indirect DMA-access to
: {Z|A}_state_VEC[biasIDX[LayerN]]
: cells, representing the LayerN's BIAS unit
So one might become happy with a just-enough ANN, feasible for training and operations in the realm of a classical-computing, without a need to wait the next 20 years, till and if, the Universal-Quantum-Computing-device(s) start to become able to produce and deliver results in a snap, in [CTIME], once the current [CSPACE]-constraints will stop to block such promising services.