Self supervised learning & making use of unlabelled data.

Please log in to watch this conference skillscast.

The general supervised learning problem starts with a labelled dataset. It's common though to additionally have a large collection of unlabelled data also. Self supervision techniques are a way to make use of this data to boost performance. In this talk we'll review some contrastive learning techniques that can either be used to provide weak labelled data or to act as a way of pre training for few-shot learning.

Q&A

Question: Are there any libraries you would recommend for self-supervision?

Answers: not specifically; I see a lot of these techniques as being more around orchestration of moving parts.

Question: Do you think there's a gap there for libraries that could replace the bespoke code or it just is what it is?

Answers: Yeah absolutely! transfer learning went through a similar thing a while back. it used to be tricky and clumsy to do but, as it became more and more common, the libraries abstracted more and more of it. now it's trivial to do transfer learning. The same thing will happen for these techniques I think.

Question: Self-supervision has been applied to unstructured data (NLP, images etc.). i haven't seen much work on tabular data beyond auto-encoding in TabNet (https://arxiv.org/abs/1908.07442), but it's an important data source at most companies.

what kind of techniques can be used on tabular data?

Answer: I think it's the same set of training techniques. the encoding part covers an inductive biases; e.g. modality, things like images vs text, but the second part, the encoding to the ypred it doesn't matter what the input was anymore. I think for the self-supervision problem it's more about how you define the positive/negative pairs; what examples do you want the encoding to be similar for vs not.

Answer: Two interesting papers recently I think are <a href="https://arxiv.org/abs/2006.10029" target="blank">https://arxiv.org/abs/2006.10029 & https://arxiv.org/abs/2006.07733. from these, I’d look back through common references and find what's interesting to you.

Question: On transfer learning, can we use encodings from a general domain to train more specific domains. For example, using an encoder from say a public feed on the internet to train on a highly specialized corpus, e.g. emails in a trading company?

Answer: Yes; as long as the specific domain, in some way, is a subset of the general domain. it's common for the large original model to deal in a wide set of labels and you'll see a good result if your labels, in some way are a subset. The best results are when the input data is similar; e.g. you might not see as good a result if the large model was trained on camera phone images but you're trying to transfer to images taken from the Hubble space telescope.

YOU MAY ALSO LIKE:

• Practical Learning To Learn (SkillsCast recorded in May 2019)
• Taming the Long Tail of Industrial ML Applications (SkillsCast recorded in May 2021)
• Cluster-wide Scaling of Machine Learning with Ray (SkillsCast recorded in June 2020)