Notes on CLIP by OpenAI

Components for AI

• Data (Experience) - acquisition, search, labelling
• Compute (Mind Power) - GPU quality, electricity, speed, etc.
• Algorithm (Logic and Reasoning) - more innovation and testing, knowledge based

Data Specification and Learning Methods

• Labeled data to specify a visual concept
• Learning from raw text on images
  • So like based of description of images than pixels itself?
• Caption image prediction mechanism
• Natural language refers to visual concepts
• zero-shot (identify object with no prior training)
• tasks : OCR, action recog, geo-local
• Current SOTA —> ResNet 50, ImageNet

Data Labeling and Classification

• Nouns and adjectives paired to images as data labelling to be trained on
• Multilabel classif of data through metadata extraction from title, description
• not some magical guessing going behind :
  • Basically a set of of labels captions to images
  • train model on about 1.3 million of such examples
  • model is trying to predict the caption of a given image zero-shot

Softmax and Model Limitations

• Softmax (like a sigmoid function) output not dynamic due to input size increase (dilutes focus as input size increases)
  • More tokens = less decisive decisions by softmax
  • But more details in input to be more specific for more decisive inputs?
  • Also overconfident on unseen data as its ust a function
  • Limits zero shot capabilites because it is so static

CLIP Approach and Innovations

• CLIP moat - needs to process less number of images for more accuracy zero-shot than other models
  • Approach :
    • Natural Language supervision - thoughout text rep of an image instead of plain un-thought of labelling
    • Large ass dataset
      • Crowd labelled dataset
      • Filtering reduces by a lot
    • Pretraining method
      • Has to be efficient due to lots of compute use
      • first was join image CNN and text transformer from scratch
      • Major change is to predict which text as a whole will be paired with image, not the EXACT words though. (Bag of Words)
      • This boosted efficiency

Model Architecture and Training

• Changes in model architecture
• Res Net 50 and ViT
  • Involved adding in a layer of attention mechanism
• Text encoder for inputs
  • It is a transformer 63M params
  • Caped sequence length for efficiency
  • Embeddings are converting image/text into high dimensional vector reps (numerical)
• What was done different
  • prev scaled models by increasing width or length with compute on only one dimension
  • CLIP allocated additional compute across all width, depth and resolution equally
• Training
  • Adam Optimizer - Adam (Adaptive Moment Estimation)
  • Decoupled Weight Decay Regularization - A modification to Adam
  • Gradient Checkpointing
  • Trades computation for memory: Increases runtime by 15-25%

Prompt Engineering and Data Handling

• Interesting prompt engineering
  • “photo of a bird” than “bird” for example was a good default
  • a sentence text label than just a word was better
  • also a category specification, like the species of a bird was helpful
• Data overall cleanup to prevent screw ups in evals as it could be in pre training
• Thoughts - if clip is image and text pair based NN, then what would be better architectures
  • Predictions of the hex code of that individual pixel based on given word - video gen models?
• Apparently racist to black and least racist to east asians
• Classifies Black category into gorilla, chimpanzee, etc.