Cross-modal Attention
Cross-modal Attention - Related Videos
13:51
Cross-Modal Multivariate Pattern Analysis
0 Views •
Cited by 6 •
2011
Classical multivariate pattern analysis predicts sensory stimuli a subject perceives from neural activity in the corresponding cortices (e.g. visual stimuli from activity in visual cortex). Here, we apply pattern analysis cross-modally and show that sound- and touch-implying visual stimuli can be predicted from activity in auditory and somatosensory cortices, respectively.
07:13
Multimodal Cross-Device and Marker-Free Co-Registration of Preclinical Imaging Modalities
0 Views •
Cited by 3 •
2023
The combination of multiple imaging modalities is often necessary to gain a comprehensive understanding of pathophysiology. This approach utilizes phantoms to generate a differential transformation between the coordinate systems of two modalities, which is then applied for co-registration. This method eliminates the need for fiducials in production scans.
09:37
Measurement of Neurophysiological Signals of Ignoring and Attending Processes in Attention Control
0 Views •
Cited by 5 •
2015
Attention control comprises enhancement of target signals and attenuation of distractor signals. We describe an approach to measure separately but concurrently, the neurophysiology of attending and ignoring in sustained intermodal attention, utilizing a passive control condition during which neither process is continuously engaged.
09:59
Visual Attention: fMRI Investigation of Object-based Attentional Control
0 Views •
2023
Source: Laboratories of Jonas T. Kaplan and Sarah I. Gimbel— University of Southern California The human visual system is incredibly sophisticated and capable of processing large amounts of information very quickly. However, the brain's capacity to process information is not an unlimited resource. Attention, the ability to selectively process information that is relevant to current goals and to ignore information that is not, is therefore an essential part of visual perception. Some aspects of...
08:52
The Attentional Blink
0 Views •
2023
Source: Laboratory of Jonathan Flombaum—Johns Hopkins University In order for recognition of a certain stimulus to take place, visual attention needs to be directed towards said stimulus. To the earliest parts of the visual system, objects are not objects, they are collections of visual features-lines, corners, changes in texture, color, and light. Attention is the resource that is necessary for later processing in order to recognize what a given bundle of features adds up to. This makes...
FAQs
Each modality first produces features that can be projected into a shared representation. Queries from one modality are compared with keys from another to estimate relevance, while the associated value features provide the information that is combined. This separation lets the model select useful cross-modal evidence rather than treating every image, text, audio, or sensor feature equally.
Images, language, audio, and sensor signals encode information in different forms, so their original features cannot be compared directly in a meaningful way. Projection into a shared representation creates a common space for cross-modal similarity calculations. This alignment allows one modality to guide interpretation of another while preserving modality-specific information for later processing.
Attention weights determine which features contribute most strongly to the combined representation. High weights emphasize signals judged relevant across modalities, whereas low weights reduce the influence of less useful information. In complex environments, this selective weighting can highlight complementary evidence and suppress distractions, supporting more informative representations and more reliable downstream decisions.
Equal-weight combination gives every input feature a similar influence, regardless of its relevance to the other modality. Cross-modal attention instead uses query-key similarity to assign data-dependent weights before combining value features. That distinction enables selective fusion, which is particularly useful when different sensors or input types provide complementary evidence or contain signals that should be downweighted.
A practical workflow begins by extracting modality-specific features, projecting them into a shared representation, and forming queries, keys, and values for the relevant inputs. The system then calculates cross-modal similarities, converts them into attention weights, and combines the selected value features. The resulting representation can support retrieval, question answering, sensor fusion, or coordinated perception.
The mechanism supports multimodal sensor fusion, where signals from different sensors are interpreted together, as well as image-text retrieval and visual question answering. It also contributes to coordinated perception in robots and autonomous systems. These applications use relationships between modalities to improve how systems interpret complex inputs and make decisions based on complementary information.
In robotic and autonomous systems, cross-modal attention can coordinate information from visual inputs and other sensor signals so that one source helps interpret another. The model emphasizes relationships that matter for the current perception task and reduces the impact of less relevant signals. This supports a more integrated representation for decision-making in complex environments.