# Hype: Compositional Machine Learning and Hyperparameter Optimization

Hype is a proof-of-concept deep learning library, where you can perform optimization on compositional machine learning systems of many components, even when such components themselves internally perform optimization.

This is enabled by nested automatic differentiation (AD) giving you access to the automatic exact derivative of any floating-point value in your code with respect to any other. Underlying computations are run by a BLAS/LAPACK backend (OpenBLAS by default).

### Automatic derivatives

You do not need to worry about supplying gradients (or Hessians) of your models, which are computed exactly and efficiently by AD. The underlying AD functionality is provided by DiffSharp.

"Reverse mode" AD is a generalized form of "backpropagation" and is distinct from numerical or symbolic differentiation.

In addition to reverse AD, Hype makes use of forward AD and nested combinations of forward and reverse AD. The core differentiation API provides gradients, Hessians, Jacobians, directional derivatives, and matrix-free exact Hessian- and Jacobian-vector products.

You can get exact gradients of the training or validation loss with respect to hyperparameters. These hypergradients allow you to do gradient-based optimization of gradient-based optimization, meaning that you can do things like optimizing learning rate and momentum schedules, weight initialization parameters, or step sizes and mass matrices in Hamiltonian Monte Carlo models. (A recent article doing this with Python: Maclaurin, Dougal, David Duvenaud, and Ryan P. Adams. "Gradient-based Hyperparameter Optimization through Reversible Learning." arXiv preprint arXiv:1502.03492 (2015).)

  1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19:  open Hype open Hype.Neural // Train a network with stochastic gradient descent and a learning rate schedule let train (x:DV) = let n = FeedForward() n.Add(Linear(784, 300)) n.Add(tanh) n.Add(Linear(300, 10)) let loss, _ = Layer.Train(n, data, {Params.Default with LearningRate = Schedule x Momentum = Momentum.DefaultNesterov Batch = Minibatch 100 Loss = CrossEntropyOnLinear}) loss // Return the loss at the end of training // Train the training, i.e., optimize the learning schedule vector by using its hypergradient let hypertrain = Optimize.Minimize(train, DV.create 200 (D 1.f), {Params.Default with Epochs = 50}) 

You can also take derivatives with respect to training data, to analyze training sensitivities.

### Compositionality

Nested AD handles higher-order derivatives up to any level, including in complex cases such as

$\mathbf{min} \left(x \; \mapsto \; f(x) + \mathbf{min} \left( y \; \mapsto \; g(x,\,y) \right) \right)\, ,$

where $$\mathbf{min}$$ uses gradient-based optimization. (Note that the inner function has a reference to the argument of the outer function.) This allows you to create complex systems where many components may internally perform optimization.

For example, you can optimize the rules of a multi-player game where the players themselves optimize their own strategy using a simple model of the opponent which they optimize according to their opponent's observed behaviour.

Or you can perform optimization of procedures that are internally using differentiation for purposes other than optimization, such as adaptive control or simulations.

### Complex objective functions

You can use derivatives in the definition of objective functions for training your models. For example, your objective function can take input sensitivities into account, for training models that are invariant under a set of input transformations.

In the current release
• OpenBLAS backend by default
• Regression, feedforward neural networks
• Recurrent neural networks, LSTMs, GRUs
• Hamiltonian Monte Carlo
Upcoming features
• GPU/CUDA backend
• Probabilistic inference
• Convolutional neural networks

Hype is developed by Atılım Güneş Baydin and Barak A. Pearlmutter at the Brain and Computation Lab, Hamilton Institute, National University of Ireland Maynooth.

Hype is released under the MIT license.

namespace DiffSharp
module Float32

namespace Hype
namespace Hype.Neural
val train : x:DV -> D

Full name: Index.train
val x : DV
Multiple items
union case DV.DV: float32 [] -> DV

--------------------
module DV

--------------------
type DV =
| DV of float32 []
| DVF of DV * DV * uint32
| DVR of DV * DV ref * TraceOp * uint32 ref * uint32
member Copy : unit -> DV
member GetForward : t:DV * i:uint32 -> DV
member GetReverse : i:uint32 -> DV
member GetSlice : lower:int option * upper:int option -> DV
member ToArray : unit -> D []
member ToColDM : unit -> DM
member ToMathematicaString : unit -> string
member ToMatlabString : unit -> string
member ToRowDM : unit -> DM
override ToString : unit -> string
member Visualize : unit -> string
member A : DV
member F : uint32
member Item : i:int -> D with get
member Length : int
member P : DV
member PD : DV
member T : DV
member A : DV with set
member F : uint32 with set
static member Abs : a:DV -> DV
static member Acos : a:DV -> DV
static member AddItem : a:DV * i:int * b:D -> DV
static member AddSubVector : a:DV * i:int * b:DV -> DV
static member Append : a:DV * b:DV -> DV
static member Asin : a:DV -> DV
static member Atan : a:DV -> DV
static member Atan2 : a:int * b:DV -> DV
static member Atan2 : a:DV * b:int -> DV
static member Atan2 : a:float32 * b:DV -> DV
static member Atan2 : a:DV * b:float32 -> DV
static member Atan2 : a:D * b:DV -> DV
static member Atan2 : a:DV * b:D -> DV
static member Atan2 : a:DV * b:DV -> DV
static member Ceiling : a:DV -> DV
static member Cos : a:DV -> DV
static member Cosh : a:DV -> DV
static member Exp : a:DV -> DV
static member Floor : a:DV -> DV
static member L1Norm : a:DV -> D
static member L2Norm : a:DV -> D
static member L2NormSq : a:DV -> D
static member Log : a:DV -> DV
static member Log10 : a:DV -> DV
static member LogSumExp : a:DV -> D
static member Max : a:DV -> D
static member Max : a:D * b:DV -> DV
static member Max : a:DV * b:D -> DV
static member Max : a:DV * b:DV -> DV
static member MaxIndex : a:DV -> int
static member Mean : a:DV -> D
static member Min : a:DV -> D
static member Min : a:D * b:DV -> DV
static member Min : a:DV * b:D -> DV
static member Min : a:DV * b:DV -> DV
static member MinIndex : a:DV -> int
static member Normalize : a:DV -> DV
static member OfArray : a:D [] -> DV
static member Op_DV_D : a:DV * ff:(float32 [] -> float32) * fd:(DV -> D) * df:(D * DV * DV -> D) * r:(DV -> TraceOp) -> D
static member Op_DV_DM : a:DV * ff:(float32 [] -> float32 [,]) * fd:(DV -> DM) * df:(DM * DV * DV -> DM) * r:(DV -> TraceOp) -> DM
static member Op_DV_DV : a:DV * ff:(float32 [] -> float32 []) * fd:(DV -> DV) * df:(DV * DV * DV -> DV) * r:(DV -> TraceOp) -> DV
static member Op_DV_DV_D : a:DV * b:DV * ff:(float32 [] * float32 [] -> float32) * fd:(DV * DV -> D) * df_da:(D * DV * DV -> D) * df_db:(D * DV * DV -> D) * df_dab:(D * DV * DV * DV * DV -> D) * r_d_d:(DV * DV -> TraceOp) * r_d_c:(DV * DV -> TraceOp) * r_c_d:(DV * DV -> TraceOp) -> D
static member Op_DV_DV_DM : a:DV * b:DV * ff:(float32 [] * float32 [] -> float32 [,]) * fd:(DV * DV -> DM) * df_da:(DM * DV * DV -> DM) * df_db:(DM * DV * DV -> DM) * df_dab:(DM * DV * DV * DV * DV -> DM) * r_d_d:(DV * DV -> TraceOp) * r_d_c:(DV * DV -> TraceOp) * r_c_d:(DV * DV -> TraceOp) -> DM
static member Op_DV_DV_DV : a:DV * b:DV * ff:(float32 [] * float32 [] -> float32 []) * fd:(DV * DV -> DV) * df_da:(DV * DV * DV -> DV) * df_db:(DV * DV * DV -> DV) * df_dab:(DV * DV * DV * DV * DV -> DV) * r_d_d:(DV * DV -> TraceOp) * r_d_c:(DV * DV -> TraceOp) * r_c_d:(DV * DV -> TraceOp) -> DV
static member Op_DV_D_DV : a:DV * b:D * ff:(float32 [] * float32 -> float32 []) * fd:(DV * D -> DV) * df_da:(DV * DV * DV -> DV) * df_db:(DV * D * D -> DV) * df_dab:(DV * DV * DV * D * D -> DV) * r_d_d:(DV * D -> TraceOp) * r_d_c:(DV * D -> TraceOp) * r_c_d:(DV * D -> TraceOp) -> DV
static member Op_D_DV_DV : a:D * b:DV * ff:(float32 * float32 [] -> float32 []) * fd:(D * DV -> DV) * df_da:(DV * D * D -> DV) * df_db:(DV * DV * DV -> DV) * df_dab:(DV * D * D * DV * DV -> DV) * r_d_d:(D * DV -> TraceOp) * r_d_c:(D * DV -> TraceOp) * r_c_d:(D * DV -> TraceOp) -> DV
static member Pow : a:int * b:DV -> DV
static member Pow : a:DV * b:int -> DV
static member Pow : a:float32 * b:DV -> DV
static member Pow : a:DV * b:float32 -> DV
static member Pow : a:D * b:DV -> DV
static member Pow : a:DV * b:D -> DV
static member Pow : a:DV * b:DV -> DV
static member ReLU : a:DV -> DV
static member ReshapeToDM : m:int * a:DV -> DM
static member Round : a:DV -> DV
static member Sigmoid : a:DV -> DV
static member Sign : a:DV -> DV
static member Sin : a:DV -> DV
static member Sinh : a:DV -> DV
static member SoftMax : a:DV -> DV
static member SoftPlus : a:DV -> DV
static member SoftSign : a:DV -> DV
static member Split : d:DV * n:seq<int> -> seq<DV>
static member Sqrt : a:DV -> DV
static member StandardDev : a:DV -> D
static member Standardize : a:DV -> DV
static member Sum : a:DV -> D
static member Tan : a:DV -> DV
static member Tanh : a:DV -> DV
static member Variance : a:DV -> D
static member ZeroN : n:int -> DV
static member Zero : DV
static member ( + ) : a:int * b:DV -> DV
static member ( + ) : a:DV * b:int -> DV
static member ( + ) : a:float32 * b:DV -> DV
static member ( + ) : a:DV * b:float32 -> DV
static member ( + ) : a:D * b:DV -> DV
static member ( + ) : a:DV * b:D -> DV
static member ( + ) : a:DV * b:DV -> DV
static member ( &* ) : a:DV * b:DV -> DM
static member ( / ) : a:int * b:DV -> DV
static member ( / ) : a:DV * b:int -> DV
static member ( / ) : a:float32 * b:DV -> DV
static member ( / ) : a:DV * b:float32 -> DV
static member ( / ) : a:D * b:DV -> DV
static member ( / ) : a:DV * b:D -> DV
static member ( ./ ) : a:DV * b:DV -> DV
static member ( .* ) : a:DV * b:DV -> DV
static member op_Explicit : d:float32 [] -> DV
static member op_Explicit : d:DV -> float32 []
static member ( * ) : a:int * b:DV -> DV
static member ( * ) : a:DV * b:int -> DV
static member ( * ) : a:float32 * b:DV -> DV
static member ( * ) : a:DV * b:float32 -> DV
static member ( * ) : a:D * b:DV -> DV
static member ( * ) : a:DV * b:D -> DV
static member ( * ) : a:DV * b:DV -> D
static member ( - ) : a:int * b:DV -> DV
static member ( - ) : a:DV * b:int -> DV
static member ( - ) : a:float32 * b:DV -> DV
static member ( - ) : a:DV * b:float32 -> DV
static member ( - ) : a:D * b:DV -> DV
static member ( - ) : a:DV * b:D -> DV
static member ( - ) : a:DV * b:DV -> DV
static member ( ~- ) : a:DV -> DV

val n : FeedForward
Multiple items
type FeedForward =
inherit Layer
new : unit -> FeedForward
member Add : f:(DM -> DM) -> unit
member Add : l:Layer -> unit
override Decode : w:DV -> unit
override Encode : unit -> DV
override Init : unit -> unit
member Insert : i:int * f:(DM -> DM) -> unit
member Insert : i:int * l:Layer -> unit
member Remove : i:int -> unit
...

Full name: Hype.Neural.FeedForward

--------------------
new : unit -> FeedForward
member FeedForward.Add : f:(DM -> DM) -> unit
member FeedForward.Add : l:Layer -> unit
Multiple items
type Linear =
inherit Layer
new : inputs:int * outputs:int -> Linear
new : inputs:int * outputs:int * initializer:Initializer -> Linear
override Decode : w:DV -> unit
override Encode : unit -> DV
override Init : unit -> unit
override Reset : unit -> unit
override Run : x:DM -> DM
override ToString : unit -> string
override ToStringFull : unit -> string
...

Full name: Hype.Neural.Linear

--------------------
new : inputs:int * outputs:int -> Linear
new : inputs:int * outputs:int * initializer:Initializer -> Linear
val tanh : value:'T -> 'T (requires member Tanh)

Full name: Microsoft.FSharp.Core.Operators.tanh
val loss : D
Multiple items
type Layer =
new : unit -> Layer
abstract member Decode : DV -> unit
abstract member Encode : unit -> DV
abstract member Init : unit -> unit
abstract member Reset : unit -> unit
abstract member Run : DM -> DM
abstract member ToStringFull : unit -> string
abstract member Visualize : unit -> string
abstract member EncodeLength : int
member Train : d:Dataset -> D * D []
...

Full name: Hype.Neural.Layer

--------------------
new : unit -> Layer
static member Layer.Train : l:Layer * d:Dataset -> D * D []
static member Layer.Train : l:Layer * d:Dataset * v:Dataset -> D * D []
static member Layer.Train : l:Layer * d:Dataset * par:Params -> D * D []
static member Layer.Train : l:Layer * d:Dataset * v:Dataset * par:Params -> D * D []
Multiple items
module Params

from Hype

--------------------
type Params =
{Epochs: int;
Method: Method;
LearningRate: LearningRate;
Momentum: Momentum;
Loss: Loss;
Regularization: Regularization;
Batch: Batch;
EarlyStopping: EarlyStopping;
ImprovementThreshold: D;
...}

Full name: Hype.Params
val Default : Params

Full name: Hype.Params.Default
type LearningRate =
| Constant of D
| Decay of D * D
| ExpDecay of D * D
| Schedule of DV
| Backtrack of D * D * D
| StrongWolfe of D * D * D
| RMSProp of D * D
override ToString : unit -> string
member Func : (int -> DV -> (DV -> D) -> D -> DV -> DV ref -> DV -> obj)
static member DefaultBacktrack : LearningRate
static member DefaultConstant : LearningRate
static member DefaultDecay : LearningRate
static member DefaultExpDecay : LearningRate
static member DefaultRMSProp : LearningRate
static member DefaultStrongWolfe : LearningRate

Full name: Hype.LearningRate
union case LearningRate.Schedule: DV -> LearningRate
Multiple items
union case Momentum.Momentum: D -> Momentum

--------------------
type Momentum =
| Momentum of D
| Nesterov of D
| NoMomentum
override ToString : unit -> string
member Func : (DV -> DV -> DV)
static member DefaultMomentum : Momentum
static member DefaultNesterov : Momentum

Full name: Hype.Momentum
property Momentum.DefaultNesterov: Momentum
type Batch =
| Full
| Minibatch of int
| Stochastic
override ToString : unit -> string
member Func : (Dataset -> int -> Dataset)

Full name: Hype.Batch
union case Batch.Minibatch: int -> Batch
type Loss =
| L1Loss
| L2Loss
| CrossEntropyOnLinear
| CrossEntropyOnSoftmax
override ToString : unit -> string
member Func : (Dataset -> (DM -> DM) -> D)

Full name: Hype.Loss
union case Loss.CrossEntropyOnLinear: Loss
val hypertrain : DV * D * DV [] * D []

Full name: Index.hypertrain
type Optimize =
static member Minimize : f:(DV -> D) * w0:DV -> DV * D * DV [] * D []
static member Minimize : f:(DV -> D) * w0:DV * par:Params -> DV * D * DV [] * D []
static member Train : f:(DV -> DM -> DM) * w0:DV * d:Dataset -> DV * D * DV [] * D []
static member Train : f:(DV -> DV -> DV) * w0:DV * d:Dataset -> DV * D * DV [] * D []
static member Train : f:(DV -> DV -> D) * w0:DV * d:Dataset -> DV * D * DV [] * D []
static member Train : f:(DV -> DM -> DM) * w0:DV * d:Dataset * v:Dataset -> DV * D * DV [] * D []
static member Train : f:(DV -> DM -> DM) * w0:DV * d:Dataset * par:Params -> DV * D * DV [] * D []
static member Train : f:(DV -> DV -> DV) * w0:DV * d:Dataset * v:Dataset -> DV * D * DV [] * D []
static member Train : f:(DV -> DV -> DV) * w0:DV * d:Dataset * par:Params -> DV * D * DV [] * D []
static member Train : f:(DV -> DV -> D) * w0:DV * d:Dataset * v:Dataset -> DV * D * DV [] * D []
...

Full name: Hype.Optimize
static member Optimize.Minimize : f:(DV -> D) * w0:DV -> DV * D * DV [] * D []
static member Optimize.Minimize : f:(DV -> D) * w0:DV * par:Params -> DV * D * DV [] * D []
val create : n:int -> v:'a -> DV