Monday, February 29, 2016

Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity





We develop a general duality between neural networks and compositional kernels, striving towards a better understanding of deep learning. We show that initial representations generated by common random initializations are sufficiently rich to express all functions in the dual kernel space. Hence, though the training objective is hard to optimize in the worst case, the initial weights form a good starting point for optimization. Our dual view also reveals a pragmatic and aesthetic perspective of neural networks and underscores their expressive power.


Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Sunday, February 28, 2016

The trees of Svalbard

I am up north these days and came upon a review of a TV series called Fortitude by some of the folks of Svalbard, an island beyond the artice circle that is used as a backprop to the series: 
“There has never been a violent crime here.” Maybe that’s because after shooting a guy in the head you’re told to go home and not worry about it by the cop who watched you pull the trigger.
Believe that’s a realistic portrayal of everyday life in Longyearbyen and you’ll be well-prepared for the rest of “Fortitude,” since throughout the 11 episodes (or 12, since the DVD version treats the double-length opener as two) the locals wander about killing and pummeling each other, stealing relics and expensive equipment, going on drunken shooting binges, and generally acting in ways that make viewers think everyone deserves to be locked at some point. And while some are – always the wrong ones, naturally – nobody’s ever charged, let alone convicted of anything.
But we really don’t care much about that, because the far more twisted thing is – WTF is up with all those trees?
 
Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Friday, February 26, 2016

MLHardware: Deep Learning on FPGAs: Past, Present, and Future


Deep Learning on FPGAs: Past, Present, and Future by Griffin Lacey, Graham W. Taylor, Shawki Areibi
The rapid growth of data size and accessibility in recent years has instigated a shift of philosophy in algorithm design for artificial intelligence. Instead of engineering algorithms by hand, the ability to learn composable systems automatically from massive amounts of data has led to ground-breaking performance in important domains such as computer vision, speech recognition, and natural language processing. The most popular class of techniques used in these domains is called deep learning, and is seeing significant attention from industry. However, these models require incredible amounts of data and compute power to train, and are limited by the need for better hardware acceleration to accommodate scaling beyond current data and model sizes. While the current solution has been to use clusters of graphics processing units (GPU) as general purpose processors (GPGPU), the use of field programmable gate arrays (FPGA) provide an interesting alternative. Current trends in design tools for FPGAs have made them more compatible with the high-level software practices typically practiced in the deep learning community, making FPGAs more accessible to those who build and deploy models. Since FPGA architectures are flexible, this could also allow researchers the ability to explore model-level optimizations beyond what is possible on fixed architectures such as GPUs. As well, FPGAs tend to provide high performance per watt of power consumption, which is of particular importance for application scientists interested in large scale server-based deployment or resource-limited embedded applications. This review takes a look at deep learning and FPGAs from a hardware acceleration perspective, identifying trends and innovations that make these technologies a natural fit, and motivates a discussion on how FPGAs may best serve the needs of the deep learning community moving forward.




Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Thursday, February 25, 2016

Bitwise Neural Networks



Based on the assumption that there exists a neural network that efficiently represents a set of Boolean functions between all binary inputs and outputs, we propose a process for developing and deploying neural networks whose weight parameters, bias terms, input, and intermediate hidden layer output signals, are all binary-valued, and require only basic bit logic for the feedforward pass. The proposed Bitwise Neural Network (BNN) is especially suitable for resource-constrained environments, since it replaces either floating or fixed-point arithmetic with significantly more efficient bitwise operations. Hence, the BNN requires for less spatial complexity, less memory bandwidth, and less power consumption in hardware. In order to design such networks, we propose to add a few training schemes, such as weight compression and noisy backpropagation, which result in a bitwise network that performs almost as well as its corresponding real-valued network. We test the proposed network on the MNIST dataset, represented using binary features, and show that BNNs result in competitive performance while offering dramatic computational savings.


Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Parallel Randomized and Matrix-Free Direct Solvers for Large Structured Dense Linear Systems - implementation -





We design efficient and distributed-memory parallel randomized direct solvers for large structured dense linear systems, including a fully matrix-free version based on matrix-vector multiplications and a partially matrix-free one. The dense coefficient matrix A has an off-diagonal low-rank structure, as often encountered in practical applications such as Toeplitz systems and discretized integral and partial differential equations. A distributed-memory parallel framework for randomized structured solution is shown. Scalable adaptive randomized sampling and hierarchical compression algorithms are designed to approximate A by hierarchically semiseparable (HSS) matrices. Systematic process grid storage schemes are given for different HSS forms. Parallel hierarchical algorithms are proposed for the resulting HSS forms. As compared with existing work on parallel HSS methods, our algorithms have several remarkable advantages, including the matrix-free schemes that avoid directly using dense A, a synchronized adaptive numerical rank detection, the integration of additional structures into the HSS generators, and much more flexible choices of the number of processes. Comprehensive analysis is conducted and shows that the communication costs are significantly reduced by up to an order of magnitude. Furthermore, we improve the original matrix-free HSS construction algorithm by avoiding some instability issues and by better revealing the nested rank structures. Tests on large challenging dense discretized matrices related to 3D scattering fully demonstrate the superior efficiency and scalability of the direct solvers. For example, for a 106 × 106 dense discretized matrix, the partially matrix-free HSS construction takes about 4500 seconds with 512 processes, while the solution takes only 0.63 second. The storage saving is over 30 times. The fully matrix-free one takes slightly longer but is more flexible and accurate. 
An implementation is here.

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Wednesday, February 24, 2016

Understanding Deep Convolutional Networks



Deep convolutional networks provide state of the art classifications and regressions results over many high-dimensional problems. We review their architecture, which scatters data with a cascade of linear filter weights and non-linearities. A mathematical framework is introduced to analyze their properties. Computations of invariants involve multiscale contractions, the linearization of hierarchical symmetries, and sparse separations. Applications are discussed.


Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

GitXiv: The most awesomest page on the interweb - implementation -

One of the most awesome resource on the interwebs after Arxiv: GitXiv

From the about page:

GitXiv — Collaborative Open Computer Science
In recent years, a highly interesting pattern has emerged: Computer scientists release new research findings on arXiv and just days later, developers release an open-source implementation on GitHub. This pattern is immensely powerful. One could call it collaborative open computer science (cocs).
GitXiv is a space to share collaborative open computer science projects. Countless Github and arXiv links are floating around the web. Its hard to keep track of these gems. GitXiv attempts to solve this problem by offering a collaboratively curated feed of projects. Each project is conveniently presented as arXiv + Github + Links + Discussion. Members can submit their findings and let the community rank and discuss it. A regular newsletter makes it easy to stay up-to-date on recent advancements. It´s free and open. Read more...The GitXiv website is open-source.
For ideas/bugs/feature requests, open a Issue.
Collaborate & discuss on the Wiki.
GitXiv has a RSS feed, a Twitter and GitHub Account.
GitXiv was created by Samim & maintained with Graphific.

Tuesday, February 23, 2016

Single-shot phase imaging with randomized light (SPIRaL)

 
 
 
We've seen interesting things from these authors before and they continue: Single-shot phase imaging with randomized light (SPIRaL) by Ryoichi Horisaki, Riki Egami, and Jun Tanida by Ryoichi Horisaki, Riki Egami, and Jun Tanida
We present a method for single-shot phase imaging with randomized light (SPIRaL). In SPIRaL, the complex (amplitude and phase) field of an object illuminated with a randomized coherent beam is captured with an image sensor, without the need for any reference light. The object field is retrieved from the single captured intensity image by a compressive sensing-based algorithm with a sparsity constraint. SPIRaL has higher observation speed, light efficiency, and flexibility of the implementation compared with previous methods. We demonstrate SPIRaL numerically and experimentally. 
 
Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Monday, February 22, 2016

Ce soir: Paris Machine Learning Meetup #9 Season 3: Machine Learnings in Music / A First step with TensorFlow

Yes, tonight, we will have the fourth evening Machine Learning Meetup of the month of February 2016 (here is a link the first, the second and the third one). This time Mobiskill is kindly hosting us and sponsoring the networking event afterwards. Thank you to them. 

Our next meetups will take place 9th march, 13th april, 11th may, 8th june. If you are interested in presenting at these meetups, here is the form:http://goo.gl/forms/iHKTj7L2Ow





Au programme de ce meetup:

Inès Al Ardah, présentation de Mobiskill
 

Bob Sturm,   " Your machine learnings may not be learning what you think they are learning: Lessons in music and experimental design"



In machine learning, generalisation is the aim, and overfitting is the bane; but just because one avoids the latter does not guarantee the former. Of particular importance in some applications of machine learning is the “sanity" of the models learnt. I will discuss one discipline in which model sanity is essential -- machine music listening — and how several hundreds of research publications may have unknowingly built, tuned, tested, compared and advertised “horses” instead of solutions. The true cautionary tale of the horse-genius Clever Hans provides the most appropriate illustration, but also ways forward.
sites:


Jiqiong Qiu "First step Deep Learning with Tensorflow"
"Le Deep Learning est une technique en plein essor dans le domaine du machine learning au cours de ces dernières années. De nombreuses applications ont vu le jour telles que Google Brain, Microsoft Cortana, DeepFace de facebook et Siri chez Apple. A l’aide de l’outil d’apprentissage de google, TensorFlow, nous allons présenter le deep learning et à l’aide d’un exemple basé sur un problème de classification d’image comprendre l’utilisation de cette librairie." 
site associé:  https://github.com/Sfeir/demo-tensorflow

Romain Jouin mentioned a hands-on  meetup around TensorFlow and the attendant Udacity course. It is here.

David Marie-Joseph, Pitch éclair (2 minutes)   "La reconnaissance visuelle au service du shopping"

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Friday, February 19, 2016

Generalization Properties of Learning with Random Features

This paper shows that a judicious incremental use of Random Features seems to work as if one were to choose several regularization Thikonov factors. This is interesting: Generalization Properties of Learning with Random Features by Alessandro Rudi, Raffaello Camoriano, Lorenzo Rosasco

We study the generalization properties of regularized learning with random features in the statistical learning theory framework. We show that optimal learning errors can be achieved with a number of features smaller than the number of examples. As a byproduct, we also show that learning with random features can be seen as a form of regularization, rather than only a way to speed up computations.
 
 
 
Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

CfP: IEEE 2016 Information Theory Workshop, Cambridge, UK,

 
 
  Michèle just sent me the following:
 
Dear Igor

Could you please announce IEEE 2016 Information Theory Workshop on your webpage Nuit Blanche. ITW 2016 will be held in Cambridge, UK, and an important focus is on works on the interface between information theory and compressed sensing  and information theory and statistics and machine learning.

We therefore want to bring this conference to the attention of the compressed sensing and the machine learning communities. 

You find the call for papers attached. Below is an announcement purely in text if you prefer. 

Thank you very much in advance and with best regards,
 
Michèle Wigger
Associate Prof. 
Telecom ParisTech

Sure Michèle, here it is:
The 2016 IEEE Information Theory Workshop will take place
from the 11th to the 14th September 2016 at Robinson College, Cambridge, United Kingdom.

The 2016 IEEE Information Theory Workshop welcomes original
technical contributions in all areas of information theory. The agenda
includes both invited and contributed sessions, with a particular
emphasis on the interface between:

Information Theory, Statistics and Machine Learning
Information Theory and Compressed Sensing
Information Theory and Radar

Plenary Speakers
Yonina Eldar, Technion—Israel Institute of Technology
Andrew Blake, Microsoft Research Cambridge
Thomas Strohmer, University of California, Davis

Paper Submission
Authors are invited to submit previously unpublished papers, not
exceeding five pages, according to the directions that will appear on
the conference website: http://sigproc.eng.cam.ac.uk/ITW2016
The ITW proceedings will be published by the IEEE and will be
available on IEEE Xplore.

Schedule
Paper Submission Deadline: 13th March 2016
Acceptance Notification: 12th June 2016
Final Paper Submission: 31st July 2016

Website: http://sigproc.eng.cam.ac.uk/ITW2016

 
 
Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Thursday, February 18, 2016

Pursuits in Structured Non-Convex Matrix Factorizations

Interesting !



Pursuits in Structured Non-Convex Matrix Factorizations by Rajiv Khanna, Michael Tschannen, Martin Jaggi

Efficiently representing real world data in a succinct and parsimonious manner is of central importance in many fields. We present a generalized greedy pursuit framework, allowing us to efficiently solve structured matrix factorization problems, where the factors are allowed to be from arbitrary sets of structured vectors. Such structure may include sparsity, non-negativeness, order, or a combination thereof. The algorithm approximates a given matrix by a linear combination of few rank-1 matrices, each factorized into an outer product of two vector atoms of the desired structure. For the non-convex subproblems of obtaining good rank-1 structured matrix atoms, we employ and analyze a general atomic power method. In addition to the above applications, we prove linear convergence for generalized pursuit variants in Hilbert spaces - for the task of approximation over the linear span of arbitrary dictionaries - which generalizes OMP and is useful beyond matrix problems. Our experiments on real datasets confirm both the efficiency and also the broad applicability of our framework in practice.



Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Wednesday, February 17, 2016

Paris Machine Learning Meetup #8 Season 3: Fair and Ethical Algorithms

Today in Paris, we'll hold the third meetup of the month (here is a link the first and the second):



This time the topic will be Fair and Ethical Algorithms. Dojocrea ( )is kindly hosting us. ELS - Editions Lefebvre Sarrut, It Lab (  ) is sponsoring the networking event. Thank you to them.

 How do we go about designing fair and ethical Algorithms? It looks like the subject is gaining some attention, here are three that just showed up:


The meetup will be streamed (see above) Here is our lineup:

Franck Bardol and Igor Carron, Introduction

Suresh VenkatasubramanianAn Axiomatic treatment of fairness
@geomblog
We propose a mathematical framework for reasoning about fairness in machine learning. 
Here is an Al Jazzera show on the general subject of fair algorithms (with Suresh in it). Suresh is coming from the world of TCS, so his presentation will provide an interesting take on fairness I am sure.

Michael Benesty Application of advanced NLP techniques to French legal decisions: ​​Demonstration of a significant bias of some French court of appeal judges in decisions about the rights of asylum. (pdf)
 
The presentation will start with a brief overview of the French legal system and the legal decisions that have been analyzed. The main part will be dedicated to Word2vec and a custom multi input and multi task learning algorithm based on bi-directional GRU and classical deep learning used for extraction of information from public law decisions. In the last part, some basic descriptive statistics will be used to analyze the extracted information and reveal the apparent bias of some French court of appeal judges. 
Attendant website: http://www.supralegem.fr

Michel Blancard, CMAP / EtaLab, A sunny day in the CDO team of the French gov 
  and @OpenSolarMap
The OpenSolarMap.org project aims to create a roof orientation map of the French territory. I will present how we crowdsourced a training dataset and how we used deep learning on it.
Attendant website:  OpenSolarMap

Pierre Saurel,   "Ethics of Algorithms: still an oxymoron?"
 
Most algorithms especially those from mathematics are considered intrinsically neutral, objective and universal. Ethics is the science of judgments of appreciation. In that sense algorithms seem to have no ability to judge and Ethics of Algorithms seems to be an empty oxymoron. Nevertheless, and beyond this simplistic view, considering certain kinds of algorithms this expression is no more an oxymoron. We propose six categories of algorithms to establish an Ethics of Algorithms. 


 
Let me finally note that one of the Phoenix workshop presenter is Mark Reidl, he will be a speaker of the Paris Machine Learnng meetup in June . His current paper is titled: "Using Stories to Teach Human Values to Artificial Agents"
Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Tuesday, February 16, 2016

Compressive PCA on Graphs

Another matrix factorization on graphs:

Compressive PCA on Graphs by Nauman Shahid, Nathanael Perraudin, Gilles Puy, Pierre Vandergheynst

Randomized algorithms reduce the complexity of low-rank recovery methods only w.r.t dimension p of a big dataset YRp×n. However, the case of large n is cumbersome to tackle without sacrificing the recovery. The recently introduced Fast Robust PCA on Graphs (FRPCAG) approximates a recovery method for matrices which are low-rank on graphs constructed between their rows and columns. In this paper we provide a novel framework, Compressive PCA on Graphs (CPCA) for an approximate recovery of such data matrices from sampled measurements. We introduce a RIP condition for low-rank matrices on graphs which enables efficient sampling of the rows and columns to perform FRPCAG on the sampled matrix. Several efficient, parallel and parameter-free decoders are presented along with their theoretical analysis for the low-rank recovery and clustering applications of PCA. On a single core machine, CPCA gains a speed up of p/k over FRPCAG, where k is much less than p is the subspace dimension. Numerically, CPCA can efficiently cluster 70,000 MNIST digits in less than a minute and recover a low-rank matrix of size 10304 X 1000 in 15 secs, which is 6 and 100 times faster than FRPCAG and exact recovery.

Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Monday, February 15, 2016

Thesis: A Data and Platform-Aware Framework For Large-Scale Machine Learning, Azalia Mirhoseini ( RankMap: A Platform-Aware Framework for Distributed Learning from Dense Datasets - implementation -)

Just found the following: a thesis, an attendant paper and an implementation:


A Data and Platform-Aware Framework For Large-Scale Machine Learning (pdf is here) by Azalia Mirhoseini
Over the last decade, significant strides have been made in parallel research domains to address the challenges of big data, including efficient algorithmic solutions, advanced computing architectures, and high performance computing engines. My thesis bridges these different domains to provide a framework for scalable machine learning that holistically takes into account the data geometry, learning algorithms, and properties of the underlying computing fabrics. Several classes of fast-growing data, including image and video content, contain non-sparse dependencies, i.e., a large number of non-zeros in their data correlation matrix. Such dependencies severely degrade the performance of most contemporary learning algorithms. This is because these algorithms typically require iterative updates on the correlation matrix to converge. The challenges are exacerbated when the iterative updates have to be applied to data that is distributed across multiple processing nodes. In distributed settings, the dense structure leads to increased iterative computations and inter-node communications, which can quickly become untenable in larger datasets. I have developed RankMap, the first domain-specific data and platform-aware solution to enable highly efficient learning on large and non-sparse datasets in distributed settings. The key idea is that, despite the apparent density and dimensionality, in many datasets, the information may lie on a single or a union of lower dimensional subspaces. I use this property to transform the dense but structured data into a new sparse domain where distributed computing becomes much more efficient. The transformed data becomes a product of a low-dimensional dictionary matrix and a large sparse coefficient matrix. The sparsity property is then exploited to create an efficient data partitioning and execution flow for the iterative algorithms. RankMap leverages the degrees of freedom in the sparse decomposition to tailor the transformation for the target computing platform. By defining a platform-specific computing metric that measures the overall computational and communication cost, RankMap is able to fine-tune the transformation to trade off overheads in the computing platform. For example, in a platform with slower links between nodes, a communication-minimizing transformation might be favorable over a transformation that minimizes the number of FLOPs. My framework has enabled graph-parallel distributed engines, that heavily rely on sparse connections, to become applicable to dense but structured data. In particular, I have developed an efficient mapping of the iterative learning algorithms on the transformed data that conforms to the vertex-centric computing format of GraphLab, a highly successful graph-based distributed machine learning engine. We have implemented RankMap in C++/MPI, as well as a GraphLab module, which are now available as open-source APIs. We have also developed a hardware-accelerated high-throughput version of RankMap that is customizable, resource-aware, and supports streaming input data. RankMap is a generic framework than can be applied to a vast range of problems ranging from cone optimization to regularized loss minimization problems, such as support vector machines (SVMs), ridge, or lasso. Evaluations of L1-regularization, power method, and SVM algorithms on datasets that contain billions of non-zeros demonstrate that RankMap can achieve up to two orders of magnitude improvement in runtime, energy, and memory usage simultaneously compared to the state-of-the art methods. 
 


RankMap: A Platform-Aware Framework for Distributed Learning from Dense Datasets by Azalia Mirhoseini, Eva.L. Dyer, Ebrahim.M. Songhori, Richard Baraniuk, Farinaz Koushanfar

This paper introduces RankMap, a platform-aware end-to-end framework for efficient execution of a broad class of iterative learning algorithms for massive and dense datasets. In contrast to the existing dense (iterative) data analysis methods that are oblivious to the platform, for the first time, we introduce novel scalable data transformation and mapping algorithms that enable optimizing for the underlying computing platforms' cost/constraints. The cost is defined by the number of arithmetic and (within-platform) message passing operations incurred by the variable updates in each iteration, while the constraints are set by the available memory resources. RankMap's transformation scalably factorizes data into an ensemble of lower dimensional subspaces, while its mapping schedules the flow of iterative computation on the transformed data onto the pertinent computing machine. We show a trade-off between the desired level of accuracy for the learning algorithm and the achieved efficiency. RankMap provides two APIs, one matrix-based and one graph-based, which facilitate automated adoption of the framework for performing several contemporary iterative learning applications optimized to the platform. To demonstrate the utility of RankMap, we solve sparse recovery and power iteration problems on various real-world datasets with up to 1.8 billion non-zeros. Our evaluations are performed on Amazon EC2 and IBM iDataPlex platforms using up to 244 cores. The results demonstrate up to 2 orders of magnitude improvements in memory usage, execution speed, and bandwidth compared with the best reported prior work.
 RankMap's implementation is here: https://github.com/azalia/RankMap

 
 
Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Saturday, February 13, 2016

Upcoming Paris Machine Learning Meetup on Fair and Ethical Algorithms (Wednesday 17th, 2016)



How do we go about designing fair and ethical Algorithms? It looks like the subject is gaining some traction. Here is an AMA on Reddit by Bart Selman, Moshe Vardi et Wendell Wallach on the societal impact of  AI .Coincidentally, the 2nd International Workshop on the future of AI is taking place right now in Phoenix, AZ (see the lineup at "What is the future of AI? And what should we be doing about it now?")

This week, in Paris, we'll hold the third meetup of the month (here is a link the first and the second): This time the topic will be Fair and Ethical Algorithms. Dojocrea is kindly hosting us (we are still looking for a sponsor for the pizzas and beer). The meetup will be streamed (stay tuned on the details) Here is our lineup:

Suresh VenkatasubramanianAn Axiomatic treatment of fairness
We propose a mathematical framework for reasoning about fairness in machine learning. 
Here is an Al Jazzera show on the general subject of fair algorithms (with Suresh in it). Suresh is coming from the world of TCS, so his presentation will provide an interesting take on fairness I am sure.

Michael Benesty Application of advanced NLP techniques to French legal decisions: ​​Demonstration of a significant bias of some French court of appeal judges in decisions about the rights of asylum.

The presentation will start with a brief overview of the French legal system and the legal decisions that have been analyzed. The main part will be dedicated to Word2vec and a custom multi input and multi task learning algorithm based on bi-directional GRU and classical deep learning used for extraction of information from public law decisions. In the last part, some basic descriptive statistics will be used to analyze the extracted information and reveal the apparent bias of some French court of appeal judges. 
Michel Blancard, CMAP / EtaLab, A sunny day in the CDO team of the French gov

The OpenSolarMap.org project aims to create a roof orientation map of the French territory. I will present how we crowdsourced a training dataset and how we used deep learning on it.n
Pierre Saurel,   "L'éthique des algorithmes"
Let me finally note that one of the Phoenix workshop presenter is Mark Reidl, he will be a speaker of the Paris Machine Learnng meetup in June . His current paper is titled: "Using Stories to Teach Human Values to Artificial Agents"


Image Credit: NASA/JPL-Caltech
This image was taken by Front Hazcam: Right B (FHAZ_RIGHT_B) onboard NASA's Mars rover Curiosity on Sol 1251 (2016-02-12 14:12:12 UTC).

Full Resolution
Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Friday, February 12, 2016

Random Feature Maps via a Layered Random Projection (LaRP) Framework for Object Classification

From the paper:
"...While these state-of-the-art nonlinear random projection methods have been demonstrated to provide significantly improved accuracy and reduced computational costs on large- scale real-world datasets, they have all primarily focused on embedding nonlinear feature spaces into low dimensional spaces to create nonlinear kernels. As such, alternative strategies for achieving low complexity, nonlinear random projection beyond such kernel methods have not been well-explored, and can have strong potential for improved accuracy and reduced complexity. In this work, we propose a novel method for modelling nonlinear kernels using a Layered Random Projection (LaRP) framework. Contrary to existing kernel methods, LaRP models nonlinear kernels as alternating layers of linear kernel ensembles and nonlinearities. This strategy allows the proposed LaRP framework to overcome the curse of dimensionality while producing more compact and discriminative random features...."
Interesting choice of nonlinearity. As it stands it is the one we also used, great work ! Random Feature Maps via a Layered Random Projection (LaRP) Framework for Object Classification by A. G. Chung, M. J. Shafiee, A. Wong

The approximation of nonlinear kernels via linear feature maps has recently gained interest due to their applications in reducing the training and testing time of kernel-based learning algorithms. Current random projection methods avoid the curse of dimensionality by embedding the nonlinear feature space into a low dimensional Euclidean space to create nonlinear kernels. We introduce a Layered Random Projection (LaRP) framework, where we model the linear kernels and nonlinearity separately for increased training efficiency. The proposed LaRP framework was assessed using the MNIST hand-written digits database and the COIL-100 object database, and showed notable improvement in object classification performance relative to other state-of-the-art random projection methods.
 
 
 
Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

BinaryNet: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1 - implementation -

Very interesting in terms of eventual hardware implementation and in line with what we seem to know that usual architectures are redundant:


BinaryNet: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1 by Matthieu Courbariaux, Yoshua Bengio

We introduce BinaryNet, a method which trains DNNs with binary weights and activations when computing parameters' gradient. We show that it is possible to train a Multi Layer Perceptron (MLP) on MNIST and ConvNets on CIFAR-10 and SVHN with BinaryNet and achieve nearly state-of-the-art results. At run-time, BinaryNet drastically reduces memory usage and replaces most multiplications by 1-bit exclusive-not-or (XNOR) operations, which might have a big impact on both general-purpose and dedicated Deep Learning hardware. We wrote a binary matrix multiplication GPU kernel with which it is possible to run our MNIST MLP 7 times faster than with an unoptimized GPU kernel, without suffering any loss in classification accuracy. The code for BinaryNet is available.
 The implementation is here: https://github.com/MatthieuCourbariaux/BinaryNet/tree/master/Train-time

 
Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Printfriendly