Transcript

Hi I'm Christoph and I'm a research scientist at Matter. The same free model can now segment all objects with just one concept prompt that can be short text, phrases or visual examples. The model has two separate Transformers for detection and tracking. The detection task works on an image level or frame based in videos and it just outputs all object instances on a single image for the specific prompt and the tracking task has the task of following. Each individual object throughout the video and they are coupled together to perform the final task. So when we did this research, we had certain challenge that we faced. Tracking and detection of fundamentally quite different. For the detection component, the model needs to learn similar or the same representation for different object instances. So if you have one dog and another dog, you need to learn the same dog representation for each dog. However, if you perform tracking in a video. You need to learn different representations for each instance, even if it's the same object category. So one dog needs to have a different representation than another dog than another dog. So we really leaned into prior research by Meta to build Sam Free. We use the same 2 model for the tracking component. We use the detection transformer, which was released by Meta as a base for the detection architecture. And we also use LAMA in our data engine as an AI annotation tool. So there are many applications for Sam Three. It can be used as a standalone tool. It can be used, for example, in a multimodal large language model or in the Instagram edits. A you can use Sam Free to segment people, objects and then put fun effects on top of these segmentations. We believe Sam Free will be an important milestone in the AI field and hope that the research community keeps building innovative solutions on top of it.