Temporally-Adaptive Models
for Efficient Video Understanding

Abstract

Spatial convolutions are extensively used in numerous deep video models. It fundamentally assumes spatio-temporal invariance, i.e., using shared weights for every location in different frames. This work presents Temporally-Adaptive Convolutions (TAdaConv) for video understanding, which shows that adaptive weight calibration along the temporal dimension is an efficient way to facilitate modeling complex temporal dynamics in videos. Specifically, TAdaConv empowers spatial convolutions with temporal modeling abilities by calibrating the convolution weights for each frame according to its local and global temporal context. Compared to existing operations for temporal modeling, TAdaConv is more efficient as it operates over the convolution kernels instead of the features, whose dimension is an order of magnitude smaller than the spatial resolutions. Further, kernel calibration brings an increased model capacity. Based on this readily plug-in operation TAdaConv as well as its extension, i.e., TAdaConvV2, we construct TAdaBlocks to empower ConvNeXt and Vision Transformer to have strong temporal modeling capabilities. Empirical results show TAdaConvNeXtV2 and TAdaFormer perform competitively against state-of-the-art convolutional and Transformer-based models in various video understanding benchmarks. Our codes and models are released at: https://github.com/alibaba-mmai-research/TAdaConv.

1 Introduction

Convolutions are an indispensable operation in modern deep vision models [1, 2, 3, 4], whose different variants have driven the state-of-the-art performances of convolutional neural networks (CNNs) in many visual tasks [5, 6, 7, 8, 9] and application scenarios [10, 11]. In the video paradigm, compared to the 3D convolutions [12], the combination of 2D spatial convolutions and 1D temporal convolutions is more widely preferred owing to its efficiency [13, 14]. Nevertheless, 1D temporal convolutions introduce non-negligible computation overhead on top of the spatial convolutions. Therefore, we seek to directly equip spatial convolutions with temporal modeling abilities.

One essential property of convolutions is the translation invariance [15, 16], resulting from its local connectivity and shared weights. However, recent works in dynamic filtering have shown that strictly shard weights for all pixels may be sub-optimal for modeling various spatial contents [17, 18].

Given the diverse nature of the temporal dynamics in videos, we hypothesize that temporal modeling could benefit from relaxed invariance along the temporal dimension. This means that convolution weights for different time steps are no longer strictly shared. Existing dynamic filter networks could achieve this but with two drawbacks. (i) it is difficult for most of them [17, 11] to leverage pre-trained weights, which is critical in video applications since training video models from scratch is highly resource demanding [19, 20] and prone to over-fitting on small datasets. (ii) for most dynamic filters, the weights are generated with respect to its spatial context [17, 21] or the global descriptor [22, 11], which is incapable of capturing the fine-grained temporal variations between frames.

Figure 1: Comparisons between TAdaConv and the spatial convolutions in video models. (a) Standard spatial convolutions in videos share the kernel weights between different frames. (b) Our TAdaConv adaptively calibrates the kernel weights for each frame by its temporal context.

Motivated by this, we present Temporally-Adaptive Convolution (TAdaConv) for video understanding, where the convolution weights are no longer fixed across different frames. Specifically, the convolution kernel for the $t$-th frame ${𝐖}_t$ is factorized to the multiplication of the base weight and a calibration weight: ${𝐖}_t={𝜶}_t\cdot {𝐖}_b$, where the base weight ${𝐖}_b$ is learnable and the calibration weight ${𝜶}_t$ is adaptively generated from the input data in the base weight ${𝐖}_b$. For each frame, we generate the calibration weight based on the frame descriptors of its adjacent time steps as well as the global descriptor, which effectively encodes both local and global temporal dynamics in videos. The difference between TAdaConv and standard convolutions is visualized in Fig. 1.

The main advantages of this factorization are three-fold: (i) TAdaConv can be easily plugged into any existing models to enhance temporal modeling, and their pre-trained weights can still be exploited; (ii) the temporal modeling ability can be highly improved with the help of the temporally-adaptive weight; (iii) in comparison with temporal convolutions that often operate on the learned 2D feature maps, TAdaConv is more efficient by directly operating on the convolution kernels.

TAdaConv is proposed as a drop-in replacement for the convolutions in existing models. A preliminary version of this work [23] is published in ICLR 2022, where TAdaConv has demonstrated a strong capability of temporal modeling, introducing notable performance gains to both image-based models as well as existing video models. In this work, we follow the conceptual idea of TAdaConv and present improvements to the preliminary version on both structural designs as well as model and data scaling. In terms of structural designs, we optimize TAdaConv in the following aspects: (i) At the operation level, the calibration factor generation process of TAdaConv is optimized, where multi-head self-attention [24] is introduced for modeling the global information of the videos. (ii) At the block level, we construct stronger TAdaBlocks by introducing efficient temporal feature aggregation, which we use to construct our convolutional model TAdaConvNeXtV2 and transformer TAdaFormer. Our empirical results show a notable improvement brought by our modifications on both scene- and motion-centric benchmarks. Based on the TAdaConvNeXtV2 and TAdaFormer, we further scale up both the model and data scale, which lead to a competitive performance to existing state-of-the-art approaches.

Figure 2: Instantiations of TAdaConv and TAdaConvV2. (a) Standard convolutions used in video models. (b) Our TAdaConv using non-linear weight calibrations with global temporal context. (c) TAdaConvV2 with global temporal context provided by multi-head self-attention (MHSA).

2 Related Work

Convolutional models for video understanding. Early convolutional models obtain spatio-temporal representations by 3D convolutions [25, 12, 20, 26] or two-stream networks [27]. For efficiency, recent ones build upon 2D networks and design additional operations for temporal modeling [28, 29, 30, 13, 31, 32, 33, 34, 35], where the weights of the 2D convolutions are shared between different timestamps. Our preliminary version [23] find removing this constraint leads to stronger temporal modeling ability. In this work, we modernize the convolutional model according to ConvNeXt [36] and construct a stronger convolutional model for video understanding.

Vision Transformers for video understanding. With the great success of Transformers in natural language processing [24, 37, 38], Vision Transformers (ViT) [39] are showing strong performances in various vision tasks [40, 41, 42, 43, 44, 45, 46] including video understanding [47, 48, 49, 50, 51, 52, 53]. The capability of ViTs is further enhanced when it is pre-trained on a large corpus of image [54, 55], video [56, 57, 58] or multi-modal data [59, 60], or when the size of the model is increased [61, 62], or both [63, 64]. Since directly pre-training with video data is both resource- and time-consuming, an alternative is to exploit the models pre-trained on large-scale image data and empower the model thorough additional structures for temporal modeling, such as temporal [50] or 3D windowed self-attention [48], spatio-temporal adapters [65], etc. In our work, we exploit the vanilla Vision Transformer pre-trained on a large corpus of image-text data [59] and equip it with strong temporal modeling ability with our TAdaBlock.

Dynamic networks. Dynamic networks refer to networks with content-adaptive weights or modules, such as dynamic filters/convolutions [21, 11, 66, 17], dynamic activations [67, 68], and dynamic routing [69, 70], etc. They have demonstrated exceeding network capacity and performance compared to static ones in various tasks [71, 72, 73, 74] as well as in video understanding [31, 75, 76, 77]. Some recent spatially-adaptive convolutions [78, 79] show relaxing spatial invariance could help modeling diverse visual contents, and our preliminary version [23] shows video understanding can benefit from relaxing the temporal invariance. This work further exploits the idea and enhance the temporal modeling capability of TAdaConv by introducing multi-head self-attention for global temporal modeling.

3 Temporally-adaptive Convolutions

In this work, we seek to empower the spatial convolutions with temporal modeling abilities. Inspired by the calibration process of temporal convolutions (Sec. 3.1), TAdaConv dynamically calibrates the convolution weights for each frame (Sec. 3.2) according to its temporal context (Sec. 3.3).

3.1 Revisiting temporal convolutions

We first revisit the temporal convolution to show the underlying process and its relation to dynamic filters. We consider depth-wise temporal convolution for simplicity, which is more widely used due to its efficiency [31, 30]. Formally, for a 3$\times$1$\times$1 temporal convolution filter parameterized by $𝜷=[{𝜷}_1,{𝜷}_2,{𝜷}_3]$ and placed (ignoring normalizations) after the 2D convolution parameterized by $𝐖$, the output feature ${\widetilde{𝐱}}_t$ of the $t$-th frame can be obtained by:

$${\widetilde{𝐱}}_t={𝜷}_1\cdot \delta (𝐖\ast {𝐱}_{t-1})+{𝜷}_2\cdot \delta (𝐖\ast {𝐱}_t)+{𝜷}_3\cdot \delta (𝐖\ast {𝐱}_{t+1}),$$ (1)

where the $\cdot$ indicates the element-wise multiplication, $\ast$ denotes the convolution over the spatial dimension and $\delta$ denotes ReLU activation [80]. It can be rewritten as follows:

$${\widetilde{𝐱}}_t={𝐖}_{t-1}\ast {𝐱}_{t-1}+{𝐖}_t\ast {𝐱}_t+{𝐖}_{t+1}\ast {𝐱}_{t+1},$$ (2)

where ${𝐖}_{t-1}^{i,j}={𝐌}_{t-1}^{i,j}\cdot {𝜷}_1\cdot 𝐖,{𝐖}_t^{i,j}={𝐌}_t^{i,j}\cdot {𝜷}_2\cdot 𝐖$ and ${𝐖}_{t+1}^{i,j}={𝐌}_{t+1}^{i,j}\cdot {𝜷}_3\cdot 𝐖$ are spatio-temporal location adaptive convolution weights. ${𝐌}_t\in {ℝ}^{C\times H\times W}$ is a dynamic tensor, with its value dependent on the result of the spatial convolutions (see Appendix for details). Hence, the temporal convolutions in the (2+1)D convolution essentially perform (i) weight calibration on the spatial convolutions and (ii) feature aggregation between adjacent frames. However, if the temporal modeling is achieved by coupling temporal convolutions to spatial convolutions, a non-negligible computation overhead is still introduced (see Table I).

3.2 Formulation of TAdaConv and TAdaConvV2

For efficiency, we set out to directly empower the spatial convolutions with temporal modeling abilities. Inspired by the recent finding that the relaxation of spatial invariance strengthens spatial modeling [17, 78], we hypothesize that temporally adaptive weights can also help temporal modeling. Therefore, the convolution weights in a TAdaConv layer are varied on a frame-by-frame basis. Since we observe that previous dynamic filters can hardly utilize the pretrained weights, we take inspiration from our observation in the temporal convolutions and factorize the weights for the $t$-th frame ${𝐖}_t$ into the multiplication of a base weight ${𝐖}_b$ shared for all frames, and a calibration weight ${𝜶}_t$ that are different for each time step:

$${\widetilde{𝐱}}_t={𝐖}_t\ast {𝐱}_t=({𝜶}_t\cdot {𝐖}_b)\ast {𝐱}_t.$$ (3)

3.3 Calibration weight generation.

To allow for the TAdaConv to model temporal dynamics, it is crucial that the calibration weight ${𝜶}_t$ for the $t$-th frame takes into account not only the current frame, but more importantly, its temporal context, i.e., ${𝜶}_t=𝒢(\mathrm{\dots },{𝐱}_{t-1},{𝐱}_t,{𝐱}_{t+1},\mathrm{\dots })$. Otherwise, TAdaConv would degenerate to a set of unrelated spatial convolutions with different weights applied on different frames. In practice, the calibration generation function can have various structural designs. In Fig. 2(b) and (c), we show two instantiations of the calibration generation function, which respectively correspond to TAdaConv and TAdaConvV2.

TAdaConv. In our design, we aim for efficiency and the ability to capture inter-frame temporal dynamics. For efficiency, we operate on the frame description vectors $𝐯\in {ℝ}^{T\times C}$ obtained by the global average pooling over the spatial dimension ${\text{GAP}}_s$ for each frame, i.e., ${𝐯}_t={\text{GAP}}_s({𝐱}_t)$. For temporal modeling, we apply two-layer 1D convolutions $ℱ$ with a dimension reduction ratio of $r$ on the local temporal context ${𝐯}_t^{adj}=\{{𝐯}_{t-1},{𝐯}_t,{𝐯}_{t+1}\}$:

(4)

where we use ReLU [80] and batch normalizations [81] for activation and normalization. $f$ denotes 1-D convolutions.

In order for a larger inter-frame field of view in complement to the local 1D convolution, we further incorporate global temporal information into the calibration weight generation process. For TAdaConv, we add a global descriptor to the weight generation process $ℱ$ through a linear mapping function FC:

(5)

where ${\text{GAP}}_t({𝐯}_t)$ denotes global average pooling over the temporal dimension on the frame descriptors ${𝐯}_t$. This is equivalent to global average pooling over all spatiotemporal dimensions on the original input $𝐱$. Hence, ${\text{GAP}}_t({𝐯}_t)$ contains the global temporal context in the input videos.

TAdaConvV2. The instantiation of TAdaConvV2 is generally similar to TAdaConv, with two improvements. (i) We alter the combination of ReLU and batch normalizations to GELU and layer normalizations to conform to the structures in ConvNeXt models. (ii) For global temporal context modeling, we take advantage of the powerful global modeling capability of self-attention [24]. Specifically, the calibration weight generation function can be expressed as follows:

(6)

where MHSA denotes the multi-head self-attention [24]. Since the 1D convolution before MHSA essentially provides a dynamic positional embedding for the frame descriptors $𝐯$, we do not add additional positional embeddings before the MHSA operation.

TABLE I: Comparison of (2+1)D convolution and TAdaConv in FLOPs and number of parameters. Example setting for the operation: $C_o=C_i=64$, $K=3$, $T=8$, $H=W=56$ and $r=4$. Example setting for the network: ResNet-50 with input resolution $8\times {224}^2$. Colored numbers denote the extra FLOPs/parameters introduced to 2D convolutions or ResNet-50. Refer to Appendix for model structures.

	(2+1)D Conv	TAdaConv
FLOPs		$C_o\times C_i\times K^2\times THW$
	$C_o\times C_i\times K^2\times THW$	$+C_i\times \left(THW+T\right)$
	$+C_o\times C_i\times K\times THW$	$+C_i\times C_i/r\times \left(2\times K\times T+1\right)$
		$+C_o\times C_i\times K^2\times T$
E.G. Op	1.2331 (+0.308, $\uparrow$33$\%$)	0.9268 (+0.002, $\uparrow$0.2$\%$)
E.G. Net	37.94 (+4.94, $\uparrow$15$\%$)	33.02 (+0.02, $\uparrow$0.06$\%$)
Params.	$C_o\times C_i\times K^2$	$C_o\times C_i\times K^2$
	$+C_o\times C_i\times K$	$+2\times C_i\times C_i/r\times K$
E.G. Op.	49,152 (+12,288, $\uparrow$33$\%$)	43,008 (+6,144, $\uparrow$17$\%$)
E.G. Net	28.1M (+3.8M, $\uparrow$15.6$\%$)	27.5M (+3.2M, $\uparrow$13.1$\%$)

Initialization. The TAdaConv is designed to be readily inserted into existing models by simply replacing the 2D convolutions. For effective use of the pre-trained weights, TAdaConv is initialized to behave exactly the same as the standard convolution. This is achieved by zero-initializing the weight of the last convolution in $ℱ$ and adding a constant vector $\mathrm{𝟏}$ to the formulation:

$${𝜶}_t=𝒢(𝐱)=\mathrm{𝟏}+ℱ({\text{GAP}}_s({𝐱}_t^{adj})).$$ (7)

In this way, at initial state, ${𝐖}_t=\mathrm{𝟏}\cdot {𝐖}_b={𝐖}_b$, where we load ${𝐖}_b$ with the pre-trained weights.

Calibration dimension. The base weight ${𝐖}_b\in {ℝ}^{C_{\text{out}}\times C_{\text{in}}\times K^2}$ can be calibrated in different dimensions. For standard convolutions, we instantiate the calibration on the $C_{\text{in}}$ dimension (${𝜶}_t\in {ℝ}^{1\times C_{\text{in}}\times 1}$), as the weight generation based on the input features yields a more precise estimation for the relation of the input channels than the output channels or spatial structures (empirical analysis in Table LABEL:tab:calibrationdim). For depthwise convolutions, since the convolution kernel does not have a $C_{\text{in}}$ dimension, the calibration is directly applied on the $C_{out}$ dimension of the convolution kernel.

Comparison with temporal convolutions. Table I compares the TAdaConv with R(2+1)D in parameters and FLOPs, which shows most of our additional computation overhead on top of the spatial convolution is an order of magnitude less than the temporal convolution.

TABLE II: Comparison with existing dynamic filters in terms of temporal modeling capability, location adaptiveness and the ability to exploit pre-trained weights in existing models.

	Temporal	Location	Pretrained
Operations	modeling	adaptive	weights
CondConv [11]	✗	✗	✗
DynamicFilter [21]	✗	✗	✗
DDF [17]	✗	✓	✗
TAM [31]	✓	✗	✗
TAdaConv(V2)	✓	✓	✓

Comparison with existing dynamic filters. Table II compares TAdaConv with existing dynamic filters. The main difference between different dynamic filtering approaches lies in the way that the dynamic weights are generated. Mixture-of-experts-based dynamic filters [11] generate content-dependent weights to dynamically aggregate learnable convolution weights. Other types of dynamic filters [21, 17, 31] generate dynamic weights entirely based on the input content. Our TAdaConv is different from existing dynamic filters in the following three aspects: (i) Compared to image-based dynamic filters [21, 17, 11], TAdaConv achieves temporal modeling by generating weights based on the local and global context. (ii) Compared to TANet [31] in the video paradigm, TAdaConv could model more complex temporal dynamics because of the temporally adaptive weights. (iii) Most existing dynamic filters are incapable of exploiting existing pre-trained weights, while TAdaConv could be initialized to generate dynamic weights that are identical to pre-trained ones. This reduces the training difficulty in video applications. More detailed comparisons of dynamic filters are included in Appendix.

Figure 3: TAdaBlock designs for both convolutional and transformer-based models. For convolutional models, we constructed TAda2D block in (a) and TAdaConvNeXt block in (b) in our preliminary version [23]. Inspired by the efficient temporal aggregation in TAda2D, we introduce a similar strided temporal pooling (T-Pool in the figure) and a separate normalization layer to the TAdaConvNeXtV2 block in (c). For transformer-based models, we insert a TAdaBlock before each multi-head self-attention operation in the TAdaFormer block in (d), where the TAdaBlock is composed of two pointwise convolutions, a depth-wise TAdaConvV2, and a temporal feature aggregation module.

4 TAdaBlocks

Based on TAdaConv and TAdaConvV2, we can construct a series of TAdaBlocks for various models, both convolutional and Transformer-based ones. In Fig. 3, we construct TAda2D block, TAdaConvNeXt(V2) block, and TAdaFormer block, respectively for ResNet [1], ConvNeXt [36] and ViT [39].

Apart from TAdaConv and TAdaConvV2, an important component of our TAdaBlocks is an efficient temporal feature aggregation scheme. This corresponds to the second essential step of temporal convolution. Formally, given the output of TAdaConv $\widetilde{𝐱}$, the aggregated feature can be obtained as follows:

$${𝐱}_{aggr}=\text{ReLU}({\text{Norm}}_1(\widetilde{𝐱})+{\text{Norm}}_2({\text{T-Pool}}_k(\widetilde{𝐱}))),$$ (8)

where ${\text{T-Pool}}_k$ represents the strided temporal pooling operation with a kernel size of $k$. We use different normalization parameters for the features extracted by TAdaConv $\widetilde{𝐱}$ and aggregated by strided average pooling ${\text{TempAvgPool}}_k(\widetilde{𝐱})$, as their distributions are essentially different.

During initialization, we load pre-trained weights (if any) to ${\text{Norm}}_1$, and initialize the parameters of ${\text{Norm}}_2$ to zero. Coupled with the initialization of TAdaConv, the initial state of the TAdaBlocks is exactly the same as the base model, while the calibration and the aggregation notably increase the model capacity with training (See Appendix). In experiments, we refer to this structure as the shortcut (Sc.) branch and the separate BN (SepBN.) branch.

In our preliminary version [23], we explored the TAda2D block and TAdaConvNeXt block. Inspired by the improvements brought by the temporal feature aggregation, we present an improved version of the TAdaConvNeXt block in Fig. 3 (c), i.e., TAdaConvNeXtV2 block. To cater to the modernized convolutional block [36], the structure of the aggregation scheme in TAdaConvNeXtV2 block is modified accordingly, where the activation function is removed and the normalization is switched to LayerNorm [82].

For Transformer-based models, we construct a TAdaFormer block, as in Fig. 3 (d), where a ResNet-like convolutional block is inserted before each self-attention layer. Different from ResNet blocks, we use depth-wise TAdaConvV2 between two point-wise convolutions for efficiency. Inspired by the modernized convolutional block [36], some of the normalization and activation layers are removed, as in Fig. 3 (d). Temporal aggregation is similarly performed using the efficient feature aggregation scheme presented above. Empirically, we found batch normalizations work better in TAdaBlock for TAdaFormer.

5 Evaluations on video classification

Model. We construct different variants for TAda2D, TAdaConvNeXtV2, and TAdaFormer, following the structure of the respective base models ResNet [1], ConvNeXt [36], and Vision Transformer [39]. Our model variants are obtained by replacing the residual blocks or the transformer blocks in the original model with our TAdaBlocks. Additionally, for TAdaConvNeXtV2 and TAdaFormer, we follow recent works [48, 49] and use tubelet embedding stem. More details on the model structure is included in Appendix.

Datasets. For video classification, we use Kinetics-400 [83] (K400), Something-Something-V1 and V2 [84] (SSV1 and SSV2), Epic-Kitchens-100 [85] (EK100), and HACS [86]. Further, we employ UCF101 [87] and HMDB51 [88] for multi-modal zero-shot evaluations. K400 is a widely used action classification dataset with 400 categories covered by $\sim$300K videos. SSV1 and SSV2 include 108K and 220K videos with challenging spatio-temporal interactions in 174 classes. EK100 includes 90K segments labelled by 97 verb and 300 noun classes with actions defined by the combination of nouns and verbs. HACS contains 504K videos with a taxonomy of 200 action classes. The latter two datasets are used for evaluation on action localization as well.

In addition, we also construct a large-scale video classification dataset combining Kinetics-400 [83], Kinetics-600 [89], and Kinetics-700 [90] for pre-training our video models, following [91, 64]. This results in a dataset with around 660K videos over 710 action classes, which is referred to as K710 in the following sections.

(a) Benefit of dynamic calibration. *: w/o our init.

	Temporally	SSV2	SSV2
Calibration	Varying	Top-1	Top-1*
None	✗	-	32.0
Learnable	✗	34.3	32.6
	✓	45.4	43.8
Dynamic	✗	51.2	41.7
	✓	53.8	49.8
TAda	✓	59.2	47.8

(b) Calibration dimension.

Cal. dim.	${𝚫}_{\text{Parms.}}$	${𝚫}_{\text{GFLOPs}}$	Top-1
$C_{\text{in}}$	3.16M	0.016	63.8
$C_{\text{out}}$	3.16M	0.016	63.4
$C_{\text{in}}\times C_{\text{out}}$	4.10M	0.024	63.7
$K^2$	2.24M	0.009	62.7

(c) Plug-in evaluation of TAdaConv.

	TAda
Base Model	Conv	#params.	GFLOPs	K400	SSV2
$\bullet$ SlowOnly 8$\times$8⋆ [19]	✗	32.5M	54.52	74.6	60.3
	✓	35.6M	54.53	75.9 (+1.3)	63.3 (+3.0)
$\bullet$ SlowFast 4$\times$16⋆ [19]	✗	34.5M	36.10	75.0	56.7
	✓	37.7M	36.11	76.5 (+1.5)	59.8 (+3.1)
$\bullet$ SlowFast 8$\times$8⋆ [19]	✗	34.5M	65.71	76.2	61.5
	✓	37.7M	65.73	77.4 (+1.2)	63.9 (+2.4)
$\bullet$ R(2+1)D⋆ [13]	✗	28.1M	49.55	73.6	61.1
	✓2d	31.2M	49.57	75.2 (+1.6)	62.9 (+1.8)
	✓(2+1)d	34.4M	49.58	75.4 (+1.8)	63.8 (+2.7)
$\bullet$ R3D⋆ [13]	✗	47.0M	84.23	73.8	59.9
	✓3d	50.1M	84.24	74.9 (+1.1)	62.9 (+3.0)
Notation $\star$ indicates our own implementation.
See Appendix for details on the model structure.

(d) Calibration weight generation. K: kernel size; Lin./Non-Lin.: linear/non-linear weight generation; G: global information $𝐠$.

Model	TAdaConv	K.	G.	Top-1
TSN⋆	-	-	-	32.0
Ours	Lin.	1	✗	37.5
	Lin.	3	✗	56.5
	Non-Lin.	(1, 1)	✗	36.8
	Non-Lin.	(3, 1)	✗	57.1
	Non-Lin.	(1, 3)	✗	57.3
	Non-Lin.	(3, 3)	✗	57.8
	Lin.	1	✓	53.4
	Non-Lin.	(1, 1)	✓	54.4
	Non-Lin.	(3, 3)	✓	59.2

(e) Feature aggregation scheme. FA: feature aggregation; Sc: shortcut for convolution feature; SepBN: separate batch norm.

TAdaConv	FA.	Sc.	SepBN.	Top-1	$𝚫$
✗	-	-	-	32.0	-
✓	-	-	-	59.2	+27.2
✗	Avg.	✗	-	47.9	+15.9
✗	Avg.	✓	✗	49.0	+17.0
✗	Avg.	✓	✓	57.0	+25.0
✓	Avg.	✗	-	60.1	+28.1
✓	Avg.	✓	✗	61.5	+29.5
✓	Avg.	✓	✓	63.8	+31.8
✓	Max.	✓	✓	63.5	+31.5
✓	Mix.	✓	✓	63.7	+31.7

TABLE III: Verification of hypothesis, plug-in evaluation, and in-depth ablative experiments on TAdaConv. For plug-in evaluations, we plug TAdaConv into existing video recognition models and analyze the performance on both K400 [83] and SSV2 [84]. For ablative experiments on TAdaConv, we mainly investigate its performance on SSV2.

Training. We train models initialized with ImageNet pre-training using AdamW [92] for 100/64/50 epochs on K400, SSV1/SSV2, and EK100, respectively. We adopt RandAugment [93] for data augmentation and stochastic depth [94] and label smoothing [95] for model regularization. We do not use Mixup [96] or Cutmix [97] for both models. Exponential Moving Average (EMA) [98] is used for reducing overfitting during traning. For TAdaFormer with CLIP pre-trained weights [59], we shorten the schedule to 30/24/24 epochs respectively. See Appendix for more details.

5.1 Verification of hypothesis

We start our experiments by verifying our hypothesis that relaxing the temporal invariance could lead to stronger temporal modeling capabilities of the video models. To this end, we choose several sources for the calibration weights and compare the action classification performance on SSV2, with and without the relaxation of temporal invariance. The results are shown in Table III(b). It can be observed that both learnable and dynamic calibration can bring a notable improvement to the baseline with no calibration (TSN [99]), with dynamic calibration performing stronger than learnable calibration. On top of the calibrated models, making the weights vary along the temporal dimension can further boost classification accuracy, which means the model shows a better capability of temporal modeling when the temporal variance is relaxed.

5.2 TAdaConv on existing video backbones

TAdaConv is designed as a plug-in substitution for the spatial convolutions in the video models. As in Table III(c), TAdaConv improves the classification performance with negligible computation overhead on a wide range of video models, including SlowFast [19], R3D [100] and R(2+1)D [13], by an average of 1.3% and 2.8% respectively on K400 and SSV2 at an extra computational cost of less than 0.02 GFlops. Further, not only can TAdaConv improve spatial convolutions, it also notably improve 3D and 1D convolutions. For fair comparison, all models are trained using the same training strategy. Further plug-in evaluations for action classification is presented in Appendix.

5.3 Ablative anslysis on TAdaConv

In this section, we thoroughly analyze our design choices and the effectiveness of TAdaConv and TAdaConvV2 in modeling temporal dynamics. We begin with TAdaConv, with SSV2 chosen as our main evaluation benchmark because of its more complex spatio-temporal relations.

Calibration weight initialization. In Table III(b), we show that our initialization strategy for the calibration weight generation plays a critical role in dynamic weight calibration. As in Table III(b), randomly initializing learnable weights slightly degrades the performance, while randomly initializing dynamic calibration weights (by randomly initializing the last layer of the weight generation function) notably degenerates the performance. It is likely that randomly initialized dynamic calibration weights perturb the pre-trained weights more severely than the learnable weights since it is dependent on the input. Further comparisons on the initialization are shown in the Appendix.

Figure 4: The classification accuracy of TAda2D on SSV2 with different channels (C.) and stages (S.) enabled.

Calibration weight generation function. Having established that the temporally adaptive dynamic calibration with appropriate initialization can be an ideal strategy for temporal modeling, we further ablate different ways for generating the calibration weight in Table LABEL:tab:calibrationweightgen. Linear weight generation function (Lin.) applies a single 1D convolution to generate the calibration weight, while non-linear one (Non-Lin.) uses two stacked 1D convolutions with batch normalizations and ReLU activation in between. When no temporal context is considered (K.=1 or (1,1)), TAdaConv can still improve the baseline but with a limited gap. Enlarging the kernel size to cover the temporal context (K.=3, (1,3), (3,1) or (3,3)) effectively yields a boost of over 20% on the accuracy, with K.=(3,3) having the strongest performance. This shows the importance of the local temporal context during calibration weight generation. Finally, for the scope of temporal context, introducing global context to frame descriptors performs similarly to only generating temporally adaptive calibration weights solely on the global context (in Table III(b)). The combination of the global and temporal context yields a better performance for both variants. In Appendix, we also show that this function in our TAdaConv yields a better calibration on the base weight than existing dynamic filters.

Feature aggregation. We ablate the aggregation scheme in TAda2D in Table III(e). The performance is similar for plain aggregation $𝐱=\text{Avg}(𝐱)$ and aggregation with a shortcut (Sc.) branch $𝐱=𝐱+\text{Avg}(𝐱)$, with Sc. being slightly better. Separating the batchnorm (Eq. 8) for the shortcut and the aggregation branch brings notable improvement. Strided max and mix (avg+max) pooling slightly underperform the average pooling variant. Overall, the combination of TAdaConv and our feature aggregation scheme has an advantage over the TSN baseline of 31.8%.

Calibration dimension. Multiple dimensions can be calibrated in the base weight. Table LABEL:tab:calibrationdim shows that calibrating the channel dimension more suitable than the spatial dimension, which means that the spatial structure of the original convolution kernel should be retained. Within channels, the calibration works better on $C_{\text{in}}$ than $C_{\text{out}}$ or both combined. This is probably because the calibration weight generated by the input feature can better adapt to itself.

Different stages employing TAdaConv. Fig 4 shows the stage by stage replacement of the spatial convolutions with TAdaConv in a ResNet. A minimum improvement of 17.55% is observed when TAdaConv is used in Res2. Compared to early stages, later stages contribute more to the final performance, as later stages provide more accurate calibration because of its rich semantics. Overall, TAdaConv is used in all stages for the highest accuracy.

Different proportion of channels calibrated. Here, we calibrate only a proportion of channels using TAdaConv and leave the other channels uncalibrated. The results are shown in Fig. 4. We find TAdaConv can improve the baseline by a large margin even if only 1/64 channels are calibrated, with larger proportion yielding further larger improvements.

TABLE IV: Modernizing and improving TAdaBlocks.

Model	Variant	K400	SSV2
ResNet2D	Baseline	70.4	32.0
$\downarrow$	+ TAdaConv	73.9	59.2
TAda2D	+ T-Pool	76.7	64.0
ConvNeXt	Baseline	76.0	41.4
$\downarrow$	+ TAdaConv	76.9	59.0
TAdaConvNeXt-T	+ T-Down	78.4	64.8
$\downarrow$	+ TAdaConvV2	78.9	66.0
	+ T-Pool	79.3	66.8
TAdaConvNeXtV2-T	+ Stronger Aug	79.6	67.2

5.4 Modernizing and improving TAdaBlocks

We modernize our TAdaBlock following [36] and improve it with TAdaConvV2 and temporal aggregation in Table IV. We observe a 5.6% and 9.4% improvement in the classification accuracy on K400 and SSV2, respectively, when we switch the base model from ResNet [1] to ConvNeXt [36]. Substituting the depth-wise convolution for TAdaConv further brings a 0.9% and 17.6% improvement. Following [49, 48], we employ a tubelet embedding stem (T-Down) in our TAdaConvNeXt, instantiated as a 3D convolution with temporal downsampling and an increased number of frames to keep the overall computation unchanged.

On top of our TAdaConvNeXt model, we improve TAdaBlock by replacing TAdaConv with TAdaConvV2 and introducing the temporal aggregation scheme (T-Pool). The structural modification further leads to a performance gain of 0.9% and 2.0% on K400 and SSV2, respectively. Finally, with stronger augmentation (m7 to m9 for RandAugment [93]), we achieve an accuracy of 79.6% and 67.2% on the two benchmarks with our tiny model.

5.5 Ablative analysis on TAdaConvV2 and TAdaBlocks

TAdaConvV2 and T-Pool in TAdaBlocks. Table V presents the ablative analysis on the TAdaBlock in both TAdaFormer and TAdaConvNeXtV2, specifically with respect to TAdaConvV2 and the temporal aggregation strategy.

The baseline of TAdaFormer pretrained by CLIP [59] demonstrates a strong spatial modeling capability, achieving an impressive accuracy of 83.6% on K400. However, its ability to model complex dynamics is lacked. Introducing TAdaBlock with simple spatial convolution in between and no temporal aggregation brings negligible effect. On top of this, TAdaConvV2 notably improves the model in terms of temporal modeling, improving the performance on scene-related benchmark K400 by 0.9% while bringing a 20% performance gain on the temporal-related benchmark SSV2. On top of this, employing temporal aggregation (T-Pool) and tubelet embedding (Temp. Down.) further enhances the model’s ability to model complex temporal dynamics.

Compared to TAdaFormer, since TAdaConvNeXtV2 is pre-trained on ImageNet, the baseline performance is slightly lower. All three strategies bring notable improvements to both the scene- and temporal-centric benchmarks.

Pre-training. We explore different pre-trained weights as initialization for TAdaConvNeXt and TAdaFormer in Table VI. For TAdaConvNeXtV2, pre-training on K400 benefits SSV2 performance. For TAdaFormer, using pre-trained weights of CLIP [59] outperforms the ImageNet pre-trained ones on both K400 and SSV2. CLIP+K710 initialization further improve the CLIP pre-trained variant by 2.1% on K400, but the effect on SSV2 is less significant (0.1%). For the comparison against the state-of-the-art, we use ImageNet and CLIP as the default pre-training source respectively for TAdaConvNeXtV2 and TAdaFormer.

TABLE V: Ablation study on the TAdaBlock.

	TAdaBlock		Temp.
Model	TAdaConvV2	T-Pool	Down.	K400	SSV2
$\mathbf{\circ }$ ViT-B/16	N/A	N/A	✗	83.6	48.1
$\mathbf{\circ }$ TAdaFormer-B/16	✗	✗	✗	83.6	48.2
$\mathbf{\circ }$ TAdaFormer-B/16	✓	✗	✗	84.5	68.6
$\mathbf{\circ }$ TAdaFormer-B/16	✓	✓	✗	84.5	69.2
$\mathbf{\circ }$ TAdaFormer-B/16	✓	✓	✓	84.5	70.4
$\bullet$ ConvNeXt-T	✗	✗	✗	77.2	46.2
$\bullet$ TAdaConvNeXtV2-T	✓	✗	✗	78.0	63.3
$\bullet$ TAdaConvNeXtV2-T	✓	✓	✗	79.3	66.3
$\bullet$ TAdaConvNeXtV2-T	✓	✓	✓	79.6	67.2

TABLE VI: Ablation study on different pre-training sources.

Model	Pretrain	K400	SSV2
TAdaConvNeXtV2-T	IN1K	79.6	65.2
	IN1K+K400	-	67.2
TAdaFormer-B/16	IN1K	76.3	63.9
	IN21K	81.8	67.5
	CLIP	84.5	70.4
	CLIP+K710	86.6	70.5
TimeSformer [50]	IN21K	78.7	59.5
UniFormerV2-B/16 [91]	IN21K	81.6	67.5
	CLIP	84.4	69.5
	CLIP+K710	85.6	-

TABLE VII: Classification accuracy on Kinetics-400 [83].

Model	#frames	#param.	GFLOPs$\times$views	Top-1
Models without pretraining
$\bullet$ SlowFast 8$\times$8 [19]	8+32	34.5M	66$\times$3$\times$10	77.0
$\mathbf{\circ }$ MViTv2-B [101]	32	51.2M	225$\times$1$\times$5	82.9
ImageNet-1K pretrained models
$\bullet$ TSM [28]	8	24.3M	43$\times$3$\times$10	74.1
$\bullet$ TAda2D [23]	16	27.5M	86$\times$3$\times$10	77.4
$\bullet$ TAdaConvNeXt-T [23]	32	38.6M	94$\times$3$\times$4	79.1
$\bullet$ TANet [31]	16	25.6M	242$\times$3$\times$4	79.3
$\bullet$ TDN-R101 [29]	8+16	-	258$\times$3$\times$10	79.4
$\bullet$ X3D-XXL [20]	-	20.3M	194$\times$3$\times$10	80.4
$\mathbf{\circ }$ Swin-T [48]	32	28.2M	88$\times$3$\times$4	78.8
$\mathbf{\circ }$ Swin-S [48]	32	49.8M	166$\times$3$\times$4	80.6
$\mathbf{\circ }$ Swin-B [48]	32	88.1M	282$\times$3$\times$4	80.6
$\bullet$ MoViNet-A6 [102]	120	31.4M	386$\times$1$\times$1	81.5
$\bullet$ TAdaConvNeXtV2-T	16	45.9M	47$\times$3$\times$4	79.6
$\bullet$ TAdaConvNeXtV2-T	32	45.9M	94$\times$3$\times$4	80.8
$\bullet$ TAdaConvNeXtV2-S	16	82.2M	91$\times$3$\times$4	80.8
$\bullet$ TAdaConvNeXtV2-S	32	82.2M	183$\times$3$\times$4	81.9
$\bullet$ TAdaConvNeXtV2-B	16	145.7M	162$\times$3$\times$4	81.4
$\bullet$ TAdaConvNeXtV2-B	32	145.7M	324$\times$3$\times$4	82.3
ImageNet-21K pretrained models
$\mathbf{\circ }$ X-ViT [103]	16	-	283$\times$3$\times$1	80.2
$\mathbf{\circ }$ TimeSformer [50]	96	121.4M	2380$\times$3$\times$1	80.7
$\mathbf{\circ }$ ViViT-L [49]	16	310.8M	1446$\times$3$\times$4	80.6
$\mathbf{\circ }$ MTV-B$\uparrow {320}^2$ [104]	32	310M	930$\times$3$\times$4	82.4
$\mathbf{\circ }$ Swin-B [48]	32	88.1M	282$\times$3$\times$4	82.7
$\mathbf{\circ }$ Swin-L [48]	32	197.0M	604$\times$3$\times$4	83.1
$\mathbf{\circ }$ MViT-v2-L$\uparrow {312}^2$ [101]	40	217.6M	2828$\times$3$\times$5	86.1
$\bullet$ TAdaConvNeXtV2-S	32	82.2M	183$\times$3$\times$4	82.9
$\bullet$ TAdaConvNeXtV2-B	32	145.7M	324$\times$3$\times$4	83.7

TABLE VIII: Classification accuracy on Kinetics-400 [83] with large-scale pre-training and post-pre-training.

Model	#frames	#param.	GFLOPs$\times$views	Top-1
Other large-scale pretrained models
$\mathbf{\circ }$ MAE-ST [57]	16	632M	1193$\times$3$\times$7	85.1
$\mathbf{\circ }$ MAR [105]	16	311M	276$\times$3$\times$5	85.3
$\mathbf{\circ }$ MaskFeat [106]	40	218M	3790$\times$3$\times$4	87.0
$\mathbf{\circ }$ CoVeR [107] (JFT-3B)	16	-	-	87.2
$\mathbf{\circ }$ MTV-H(WTS)$\uparrow {280}^2$ [104]	32	-	6130$\times$3$\times$4	89.9
$\mathbf{\circ }$ VideoMAE V2-g $\uparrow {266}^2$ [64]	64	-	26716$\times$3$\times$2	90.0
CLIP pretrained models
$\mathbf{\circ }$ UniFormerV2-B/16 [91]	8	115M	$\sim$150$\times$3$\times$4	84.4
$\mathbf{\circ }$ ST-Adapter-B/16 [65]	32	93M	607$\times$3$\times$1	82.0
$\mathbf{\circ }$ EVL ViT-B/16 [108]	32	115M	592$\times$3$\times$1	84.2
$\mathbf{\circ }$ X-CLIP-B/16 [109]	16	-	287$\times$3$\times$4	84.7
$\mathbf{\circ }$ ViFi-CLIP [110]	16	124.7M	281$\times$4$\times$3	83.9
$\mathbf{\circ }$ TAdaFormer-B/16	16	104.1M	153$\times$3$\times$4	84.5
$\mathbf{\circ }$ ST-Adapter-L/14 [65]	32	347M	2749$\times$3$\times$1	87.2
$\mathbf{\circ }$ EVL ViT-L/14 [108]	32	363M	2696$\times$3$\times$1	87.3
$\mathbf{\circ }$ X-CLIP-L/14 [109]	8	-	658$\times$3$\times$4	87.1
$\mathbf{\circ }$ TAdaFormer-L/14	16	364M	703$\times$3$\times$4	87.6
CLIP+K710 post-pretrained models
$\mathbf{\circ }$ UniFormerV2-B/16 [91]	8	115M	$\sim$150$\times$3$\times$4	85.6
$\bullet$ TAdaConvNeXtV2-S	32	82.2M	183$\times$3$\times$4	86.1
$\bullet$ TAdaConvNeXtV2-B	32	145.7M	324$\times$3$\times$4	86.4
$\mathbf{\circ }$ TAdaFormer-B/16	16	104.1M	153$\times$3$\times$4	86.6
$\mathbf{\circ }$ UniFormerV2-L/14 [91]	8	354M	$\sim$667$\times$3$\times$4	88.8
$\mathbf{\circ }$ UniFormerV2-L/14 [91]	16	354M	$\sim$1334$\times$3$\times$4	89.1
$\mathbf{\circ }$ UniFormerV2-L/14 [91]	32	354M	$\sim$2667$\times$3$\times$4	89.5
$\mathbf{\circ }$ TAdaFormer-L/14	16	364M	703$\times$3$\times$4	88.9
$\mathbf{\circ }$ TAdaFormer-L/14	32	364M	1406$\times$3$\times$4	89.5
$\mathbf{\circ }$ TAdaFormer-L/14	64	364M	2812$\times$3$\times$4	89.9

TABLE IX: Classification accuracy on SSV1 and SSV2.

Model	#frames	GFLOPs$\times$views	SSV1	SSV2
$\bullet$ TSM [28]	16	86$\times$3$\times$2	47.2	63.4
$\bullet$ MoViNet-A3 [102]	50	24$\times$1$\times$1	-	64.1
$\bullet$ TANet [31]	16	86$\times$3$\times$2	47.6	64.6
$\bullet$ TEANet [111]	16	86$\times$1$\times$1	48.9	-
$\bullet$ TEANet [111]	16	86$\times$3$\times$10	-	65.1
$\bullet$ TAda2D [23]	16	86$\times$3$\times$2	-	65.6
$\bullet$ TAdaConvNeXt-T [23]	32	94$\times$3$\times$2	-	67.1
$\bullet$ TDN-R101 [29]	8+16	258$\times$1$\times$1	56.8	68.2
$\bullet$ TAdaConvNeXtV2-T	16	47$\times$3$\times$2	54.1	67.2
$\bullet$ TAdaConvNeXtV2-T	32	94$\times$3$\times$2	56.4	69.8
$\bullet$ TAdaConvNeXtV2-S	16	91$\times$3$\times$2	55.6	68.4
$\bullet$ TAdaConvNeXtV2-S	32	183$\times$3$\times$2	58.5	70.0
$\bullet$ TAdaConvNeXtV2-S†	32	183$\times$3$\times$2	59.7	70.6
$\bullet$ TAdaConvNeXtV2-B†	32	324$\times$3$\times$2	60.7	71.1
$\mathbf{\circ }$ ViViT-L/16x2 FE [49]	32	903$\times$3$\times$4	-	65.4
$\mathbf{\circ }$ X-ViT [103]	16	283$\times$3$\times$1	-	67.2
$\mathbf{\circ }$ MTV-B$\uparrow {320}^2$ [104]	32	930$\times$3$\times$4	-	68.5
$\mathbf{\circ }$ Swin-B†[48]	32	321$\times$3$\times$1	-	69.6
$\mathbf{\circ }$ MViTv2-B [101]	32	225$\times$3$\times$1	-	70.5
$\mathbf{\circ }$ ST-Adapter-B/16⋆ [65]	32	651$\times$3$\times$1	-	69.5
$\mathbf{\circ }$ ST-Adapter-L/14⋆ [65]	32	2749$\times$3$\times$1	-	72.3
$\mathbf{\circ }$ UniFormerV2-B/16⋆ [91]	32	$\sim$370$\times$3$\times$2	59.5	71.0
$\mathbf{\circ }$ UniFormerV2-L/14⋆ [91]	32	$\sim$1716$\times$3$\times$2	62.9	73.1
$\mathbf{\circ }$ MViTv2-L$\uparrow {312}^2$ [101]	40	2828$\times$3$\times$1	-	73.3
$\mathbf{\circ }$ TAdaFormer-B/16⋆	16	187$\times$3$\times$2	59.2	70.4
$\mathbf{\circ }$ TAdaFormer-B/16⋆	32	374$\times$3$\times$2	61.2	71.3
$\mathbf{\circ }$ TAdaFormer-L/14⋆	16	858$\times$3$\times$2	62.0	72.4
$\mathbf{\circ }$ TAdaFormer-L/14⋆	32	1716$\times$3$\times$2	63.7	73.6
† indicates initialization with ImageNet21K+K400 pre-training.
⋆ indicates initialization with CLIP-400M pre-training.

5.6 Main results

TABLE X: Classification accuracy on Epic-Kitchens-100 [85]. $\mathbf{\uparrow }$ indicates the main evaluation metric for the dataset.

Model	Act.$\mathbf{\uparrow }$	Verb	Noun
$\bullet$ TSN [99]	33.2	60.2	46.0
$\bullet$ TRN [112]	35.3	65.9	45.4
$\bullet$ TSM [28]	38.3	67.9	49.0
$\bullet$ SlowFast [19]	38.5	65.6	50.0
$\bullet$ TAda2D [23]	41.6	65.1	52.4
$\bullet$ ir-CSN-152 [113]	44.5	68.4	55.9
$\bullet$ MoViNet-A6 [102]	47.7	72.2	57.3
$\bullet$ TAdaConvNeXtV2-T (IN1K)	42.4	67.1	53.7
$\bullet$ TAdaConvNeXtV2-T (K710)	47.4	70.4	58.6
$\bullet$ TAdaConvNeXtV2-S (K710)	48.9	71.0	60.2
$\mathbf{\circ }$ ViViT-L/16x2 FE [49]	44.0	66.4	56.8
$\mathbf{\circ }$ X-ViT [103]	44.3	68.7	56.4
$\mathbf{\circ }$ ViViT-B/16x2 FE $\uparrow 384$ [113]	47.0	67.2	59.0
$\mathbf{\circ }$ ST-Adapter-B/16 [65]	-	67.6	55.0
$\mathbf{\circ }$ MeMViT [114]	48.4	71.4	60.3
$\mathbf{\circ }$ MTV-B$\uparrow {320}^2$ [104]	48.6	68.0	63.1
$\mathbf{\circ }$ MTV-B(WTS)$\uparrow {280}^2$ [104]	50.5	69.9	63.9
$\mathbf{\circ }$ TAdaFormer-B/16 (K710)	49.1	71.0	60.5
$\mathbf{\circ }$ TAdaFormer-L/14 (K710)	51.8	71.7	64.1

TABLE XI: Zero-shot classification on UCF101 [87] and HMDB51 [88].

Model	HMDB-51	UCF-101
$\bullet$ MTE [115]	19.7 $\pm$ 1.6	15.8 $\pm$ 1.3
$\bullet$ ASR [116]	21.8 $\pm$ 0.9	24.4 $\pm$ 1.0
$\bullet$ ER-ZSAR [117]	35.3 $\pm$ 4.6	51.8 $\pm$ 2.9
$\mathbf{\circ }$ CLIP [59]	40.8 $\pm$ 0.3	63.2 $\pm$ 0.2
$\mathbf{\circ }$ ActionCLIP [118]	40.8 $\pm$ 5.4	58.3 $\pm$ 3.4
$\mathbf{\circ }$ X-CLIP-B/16 [109]	44.6 $\pm$ 5.2	72.0 $\pm$ 2.3
$\mathbf{\circ }$ A5 [119]	44.3 $\pm$ 2.2	69.3 $\pm$ 4.2
$\mathbf{\circ }$ ViFi-CLIP [110]	51.3 $\pm$ 0.6	76.8 $\pm$ 0.7
$\mathbf{\circ }$ TAdaFormer-B/16	52.1 $\pm$ 1.4	78.5 $\pm$ 1.2
$\mathbf{\circ }$ TAdaFormer-L/14	57.2 $\pm$ 0.7	81.1 $\pm$ 0.9
$\mathbf{\circ }$ TAdaFormer-B/16 (K710)	55.9 $\pm$ 0.4	79.5 $\pm$ 0.7
$\mathbf{\circ }$ TAdaFormer-L/14 (K710)	59.7 $\pm$ 0.5	83.0 $\pm$ 0.7

TABLE XII: Action localization on HACS [86].

	HACS
Model	@0.5	@0.6	@0.7	@0.8	@0.9	Avg.$\mathbf{\uparrow }$
SSN [120]	28.8	-	-	-	-	19.0
G-TAD [121]	41.1	-	-	-	-	27.5
TadTR [51]	47.1	-	-	-	-	32.1
BMN [122]+
TSN [23]	43.6	37.7	31.9	24.6	15.0	28.6
TAda2D [23]	48.7	42.7	36.2	28.1	17.3	32.3
TAdaFormer-L/14	51.3	44.8	38.0	30.0	18.6	34.1
TAdaConvNeXt-S	53.3	47.0	40.2	32.0	20.2	36.1

TABLE XIII: Action localization on Epic-Kitchens-100 [85].

	Epic-Kitchens-100
Model	Task	@0.1	@0.2	@0.3	@0.4	@0.5	Avg.$\mathbf{\uparrow }$
BMN [122] +TSN	Verb	15.98	15.01	14.09	12.25	10.01	13.47
	Noun	15.11	14.15	12.78	10.94	8.89	12.37
	Act.$\mathbf{\uparrow }$	10.24	9.61	8.94	7.96	6.79	8.71
BMN [122] +TAda2D [23]	Verb	19.70	18.49	17.41	15.50	12.78	16.78
	Noun	20.54	19.32	17.94	15.77	13.39	17.39
	Act.$\mathbf{\uparrow }$	15.15	14.32	13.59	12.18	10.65	13.18
BMN [122] +TAdaFormer-L/14	Verb	20.87	20.09	18.99	16.42	13.81	18.03
	Noun	27.75	26.28	24.51	21.86	17.97	23.67
	Act.$\mathbf{\uparrow }$	20.39	19.35	18.28	16.35	14.51	17.85
BMN [122] +TAdaConvNeXt-S	Verb	17.81	16.94	16.05	14.25	11.89	15.39
	Noun	21.90	20.92	19.33	17.22	14.68	18.81
	Act.$\mathbf{\uparrow }$	15.61	14.80	13.73	12.35	10.90	13.47
ActionFormer [123] +SlowFast	Verb	26.58	25.42	24.15	22.29	19.09	23.51
	Noun	25.21	24.11	22.66	20.47	16.97	21.88
	Act.$\mathbf{\uparrow }$	18.40	17.71	16.80	15.65	13.52	16.42
ActionFormer [124] +SlowFast&ViViT	Verb	26.97	25.90	24.21	21.77	18.47	23.46
	Noun	28.61	27.14	24.92	22.13	18.69	24.30
	Act.$\mathbf{\uparrow }$	23.90	22.98	21.37	19.57	16.94	20.95
ActionFormer +TAdaConvNeXt-S	Verb	29.11	28.37	26.99	24.22	20.64	25.86
	Noun	29.21	27.94	26.22	23.54	18.73	25.13
	Act.$\mathbf{\uparrow }$	20.78	19.75	18.56	17.07	14.54	18.14
ActionFormer +TAdaFormer-L/14	Verb	32.08	31.09	29.40	26.64	22.71	28.38
	Noun	35.00	33.42	30.98	27.32	22.36	29.82
	Act.$\mathbf{\uparrow }$	24.92	23.68	22.33	20.61	18.29	21.97

Kinetics-400. Table VII shows the results on Kinetics-400 without large-scale pre-training. TAdaConvNeXtV2 surpasses most existing approaches with a similar computation budget both when pre-trained on ImageNet-1K and ImageNet-21K. A highlight is observed where our TAdaConvNeXtV2-S with 32 frames outperforms Swin-B by 1.3 using only 57% of the computation.

Table VIII presents the comparison for models with large-scale pre-training. Compared to existing CLIP pre-trained models, TAdaFormer achieves competitive performance. When post-pre-trained on K710, TAdaFormer outperforms UniFormerV2 by a notable margin under similar computation budgets. We also observe better scalability of TAdaFormer when it is compared with TAdaConvNeXtV2.

Something-Something-V1 and V2. We show the performance comparison on temporal-related datasets, i.e., SSV1 and SSV2, in Table IX. TAdaConvNeXt and TAdaFormer achieve a favorable performance against existing convolutional and transformer-based models with identical or similar pre-training sources, respectively. Compared to the best convolutional model TDN-R101, TAdaConvNeXt-B outperforms it by 3.9 and 2.9 on SSV1 and SSV2. Compared to CLIP-pre-trained UniFormerV2-L/14, TAdaFormer-L/14 achieves an improvement of 0.8 and 0.5 on the two datasets.

Epic-Kitchens-100. We compare the performance on ego-centric action recognition in Table X. Compared to existing convolutional models, our TAdaConvNeXtV2-S achieves a favourable performance. Notably, we observe a higher accuracy for TAdaConvNeXt models on noun recognition in ego-centric videos. Transformer-based models are generally stronger than convolutional ones on EK100, where our TAdaFormer achieves a competitive performance with existing Transformers for video understanding.

Zero-shot classification on UCF101 and HMDB51. To more comprehensively evaluate our TAdaFormer, we include the results on zero-shot classification in Table XI. Here, we initialize the model with CLIP pre-trained weights and train our TAdaFormer with the corresponding language model [59]. We observe a notable improvement of TAdaFormer-B/16 on both datasets compared to the fine-tuned CLIP ViFi-CLIP [110]. On top of this, we find scaling up the model and pre-training brings a further boost to the zero-shot performance.

6 Evaluations on action localization

Dataset, pipeline, and evaluation. Action localization is an essential task for understanding untrimmed videos, whose current pipeline makes it heavily dependent on the quality of the video representations. We evaluate our TAdaConvNeXtV2 and TAdaFormer on two large-scale action localization datasets, HACS [86] and Epic-Kitchens-100 [85]. The general pipeline follows [85, 125, 126], which uses Boundary Matching Network (BMN) [122] for generating action boundaries. For evaluation, we use the average mean Average Precision (average mAP) at IoU [0.5:0.05:0.95] for HACS and [0.1:0.1:0.5] for EK100, following the standard protocol. More details are included in the Appendix.

Main results. We present the results on the two datasets in Table XII and Table XIII. On HACS, we found BMN [122] using TAdaFormer and TAdaConvNeXt features yields a favourable performance compared to some recent methods. On Epic-Kitchens-100, we further employ ActionFormer [123] and found TAdaFormer stronger than the ensemble of ViViT and SlowFast. Overall, we found TAdaConvNeXt and TAdaFormer provide strong features for localzing actions in long videos.

7 Conclusions

Based on our preliminary work [23], this work presents TAdaConvV2 in replacement of the convolution operations in existing models for video understanding, and two strong video models, i.e., TAdaConvNeXtV2 and TAdaFormer. With large-scale pre-training and post-pre-training, our video models demonstrate competitive performances to the state-of-the-art approaches, both in the task of action recognition and localization. We hope our work can facilitate further research in video understanding.

Appendix A Overview

In the appendix, we provide detailed analysis on the temporal convolutions (Appendix B), further implementation details (Appendix C) on the action classification and localization, model structures that we used for evaluation (Appendix D), per-category improvement analysis on Something-Something-V2 (Appendix E), further plug-in evaluations on Epic-Kitchens classification (Appendix G) plug-in evaluations on the temporal action localization task (Appendix H), the visualization of the training procedure of TSN and TAda2D (Appendix I), as well as detailed comparisons between TAdaConv and existing dynamic filters (Appendix J).

Appendix B Detailed analysis on temporal convolutions

Here, we provide a detailed analysis to showcase the underlying process of temporal modeling by temporal convolutions. As in Sec. 3.1, we use depth-wise temporal convolutions for simplicity and its wide application. We first analyze the case where temporal convolutions are directly placed after spatial convolutions without non-linear activation in between, before activation functions are inserted in the second part of our analysis.

Without activation. We first consider a simple case with no non-linear activation functions between the temporal convolution and the spatial convolution. Given a 3$\times$1$\times$1 depth-wise temporal convolution parameterized by $𝜷=[{𝜷}_1,{𝜷}_2,{𝜷}_3]$, where ${𝜷}_1,{𝜷}_2,{𝜷}_3\in {ℝ}^{C_o}$, a spatial convolution parameterized by $𝐖\in {ℝ}^{C_o\times C_i\times K^2}$, the output feature ${\widetilde{𝐱}}_{𝐭}$ of the $t$-th frame can be obtained by:

$${\widetilde{𝐱}}_t={𝜷}_1\cdot (𝐖\ast {𝐱}_{t-1})+{𝜷}_2\cdot (𝐖\ast {𝐱}_t)+{𝜷}_3\cdot (𝐖\ast {𝐱}_{t+1}),$$ (9)

where $\cdot$ denotes element-wise multiplication with broadcasting, and $\ast$ denotes convolution over the spatial dimension. In this case, $𝜷$ could be grouped with the spatial convolution weight $𝐖$ and the combination of temporal and spatial convolution can be rewritten as:

$${\widetilde{𝐱}}_t={𝐖}_{t-1}\ast {𝐱}_{t-1}+{𝐖}_t\ast {𝐱}_t+{𝐖}_{t+1}\ast {𝐱}_{t+1},$$ (10)

where ${𝐖}_{t-1}={𝜷}_1\cdot 𝐖$, ${𝐖}_t={𝜷}_2\cdot 𝐖$ and ${𝐖}_{t+1}={𝜷}_3\cdot 𝐖$. This equation shares the same form with the Eq. 2 in the manuscript. In this case, the combination of temporal convolution with spatial convolution can be certainly viewed as the temporal convolution simply performs calibration on spatial convolutions before aggregation, with different weights assigned to different time steps for the calibration.

With activation. Next, we consider a case where activation is in between the temporal convolution and spatial convolution. The output feature ${\widetilde{𝐱}}_t$ are now obtained by:

$${\widetilde{𝐱}}_t={𝜷}_1\cdot \delta (𝐖\ast {𝐱}_{t-1})+{𝜷}_2\cdot \delta (𝐖\ast {𝐱}_t)+{𝜷}_3\cdot \delta (𝐖\ast {𝐱}_{t+1}).$$ (11)

Next, we show that this can be still rewritten in the form of Eq. 2. Here, we consider the case where ReLU [80] is used as the activation function, denoted as $\delta$:

(12)

Hence, the term $\delta (𝐖\ast {𝐱}_t)$ can be easily expressed as:

$$\delta (𝐖\ast {𝐱}_t)={𝐌}_t\cdot 𝐖\ast {𝐱}_t,$$ (13)

where ${𝐌}_t\in {ℝ}^{C\times H\times W}$ is a binary map sharing the same shape as ${𝐱}_t$, indicating whether the corresponding element in $𝐖\ast {𝐱}_t$ is greater than 0 or not. That is:

(14)

where $c,i,j$ are the location index in the tensor. Hence, with activation, temporal convolution can be expressed as:

$${\widetilde{𝐱}}_t={𝜷}_1\cdot {𝐌}_{t-1}\cdot 𝐖\ast {𝐱}_{t-1}+{𝜷}_2\cdot {𝐌}_t\cdot 𝐖\ast {𝐱}_t+{𝜷}_3\cdot {𝐌}_{t+1}\cdot 𝐖\ast {𝐱}_{t+1}.$$ (15)

In this case, we can set ${𝐖}_{t-1}^{(i,j)}={𝜷}_1\cdot {𝐌}_{t-1}^{(i,j)}\cdot 𝐖$, ${𝐖}_t^{(i,j)}={𝜷}_2\cdot {𝐌}_t^{(i,j)}\cdot 𝐖$, and ${𝐖}_{t+1}^{(i,j)}={𝜷}_3\cdot {𝐌}_{t+1}^{(i,j)}\cdot 𝐖$, where $(i,j)$ indicate the spatial location index. In this case, each filter for a specific time step $t$ is composed of $H\times W$ filters and Eq. 1 can be rewritten as Eq. 2. Interestingly, it can be observed that with ReLU activation function, the convolution weights are different for all spatio-temporal locations, since the binary map $𝐌$ depends on the results of the spatial convolutions.

Appendix C Further implementation details

Here, we further describe the implementation details for the action classification and action localization experiments. For fair comparisons, we keep all the training strategies the same for our baseline, the plug-in evaluations as well as our own models.

training config	K710	K400 (K710)	K400 (ImageNet)	SSV1/SSV2	EK100
optimizer	AdamW [92]
learning rate schedule	cosine decay
weight decay	0.02
optimizer momentum	${\beta }_1,{\beta }_2=0.9,0.999$
dropout [127]	0.5
clip grading	None
base learning rate	5e-4
batch size	512
training epochs	100	30	100	64	50
warmup epochs	8	4	8	2.5	5
randaugment [93]	(9, 0.5)
label smoothing [95]	0.0				0.1
stochastic depth	0.2 (T)	0.2 (T)	0.2 (T)	0.3 (T)	0.3 (T)
	0.4 (S)	0.4 (S)	0.4 (S)	0.5 (S)	0.5 (S)
	0.6 (B)	0.6 (B)	0.6 (B)	0.6 (B)	-

TABLE A1: TAdaConvNeXtV2 training settings on K710, K400, SSV1/SSV2, and EK100.

C.1 Action classification with TAdaConvNeXtV2

We evaluate our approach on action classification using four large-scale benchmarks. We list the training configurations for TAdaConvNeXtV2 and TAdaFormer on action classification benchmarks in Table A1 and Table A2, respectively.

training config	K710		K400 (K710)		K400 (CLIP)		SSV1/SSV2		EK100
optimizer	AdamW [92]
learning rate schedule	cosine decay
weight decay	0.05
optimizer momentum	${\beta }_1,{\beta }_2=0.9,0.999$
dropout [127]	0.5
clip grading	None
EMA [98]	0.9996
	Base	Large	Base	Large	Base	Large	Base	Large	Base	Large
base learning rate	1e-4	5e-5	1e-5	5e-6	5e-5	2e-5	5e-4	2.5e-4	2.5e-4	1e-4
batch size	512	256	256	128	256	128	256	128	128	64
training epochs	30	24	15	10	30	24	24	24	24	15
warmup epochs	5	5	2.5	2	5	5	5	5	5	2.5
layer-wise lr decay [43]	0.7	0.8	0.7	0.8	0.7	0.85	0.7	0.85	0.7	0.85
randaugment [93]	(9, 0.5)		(9, 0.5)		(9, 0.5)		(9, 0.5)		(9, 0.5)
label smoothing [95]	0.1		0.1		0.1		0.1		0.1
stochastic depth	-		-		-		-	0.2	-

TABLE A2: TAdaFormer training settings on K710, K400, SSV1/SSV2, and EK100.

C.2 Action Localization

We evaluate our model on the action localization task using two large-scale datasets. The overall pipeline for our action localization evaluation is divided into finetuning the classification models, obtaining action proposals, and classifying the proposals.

Finetuning. On Epic-Kitchens, we simply use the evaluated action classification model. On HACS, following [126], we initialize the model with Kinetics-400 pre-trained weights and train the model with adamW [92] for 30 epochs (8 warmups) using 32 GPUs. The mini-batch size is 16 videos per GPU. The base learning rate is set to 0.0002, with cosine learning rate decay as in Kinetics. In our case, only the segments with action labels are used for training.

Proposal generation. For the action proposals, a boundary matching network (BMN) [122] is trained over the extracted features on the two datasets. On Epic-Kitchens, we extract features with the videos uniformly decoded at 60 FPS. For each clip, we use 8 frames with an interval of 8 to be consistent with finetuning, which means a feature roughly covers a video clip of one seconds. The interval between each clip for feature extraction is 8 frames (i.e., 0.133 sec) as well. The shorter side of the video is resized to 224 and we feed the whole spatial region into the backbone to retain as much information as possible. Following [125], we generate proposals using BMN based on sliding windows. The predictions on the overlapped region of different sliding windows are simply averaged. On HACS, the videos are decoded at 30 FPS, and extend the interval between clips to be 16 (i.e., 0.533 sec) because the actions in HACS last much longer than in Epic-Kitchens. The shorter side is resized to 128 for efficient processing. For the settings in generating proposals, we mainly follow [126], except that the temporal resolution is resized to 100 in our case instead of 200.

Classification. On Epic-Kitchens, we classify the proposals with the fine-tuned model using 6 clips. Spatially, to comply with the feature extraction process, we resize the shorter side to 224 and feed the whole spatial region to the model for classification. On HACS, considering the property of the dataset that only one action category can exist in a video, we obtain the video level classification results by classifying the video level features, following [126].

Action localization with ActionFormer. We follow all the settings in [124, 123] for action localization experiments with ActionFormer.

Evaluation. For evaluation, we follow the standard evaluation protocol used in the respective datasets, i.e., the average mean Average Precision (average mAP) at IoU threshold [0.5:0.05:0.95] for HACS [86] and [0.1:0.1:0.5] for Epic-Kitchens-100 [85].

Appendix D Model structures

TABLE A3: Model structure of R3D, R(2+1)D and R2D that we used in our experiments. Brown and green fonts indicate respectively the default convolution operation and optional operation that can be replaced by TAdaConv. (Better viewed in color.)

Stage	R3D	R(2+1)D	R2D	output sizes
Sampling	interval 8, 1${}^{\text{2}}$	interval 8, 1${}^{\text{2}}$	interval 8, 1${}^{\text{2}}$	8$\times$224$\times$224
conv1	3$\times$7${}^{\text{2}}$, 64	1$\times$7${}^{\text{2}}$, 64	1$\times$7${}^{\text{2}}$, 64	8$\times$112$\times$112
	stride 1, 2${}^{\text{2}}$	stride 1, 2${}^{\text{2}}$	stride 1, 2${}^{\text{2}}$
res2	$\left[\begin{array}{c}\text{1}\times \text{1}{}^{\text{2}}\text{, 64}\\ \text{3}\times \text{3}{}^{\text{2}}\text{, 64}\\ \text{1}\times \text{1}{}^{\text{2}}\text{, 256}\end{array}\right]$$\times$3	$\left[\begin{array}{c}\text{1}\times \text{1}{}^{\text{2}}\text{, 64}\\ \text{1}\times \text{3}{}^{\text{2}}\text{, 64}\\ \text{3}\times \text{1}{}^{\text{2}}\text{,64}\\ \text{1}\times \text{1}{}^{\text{2}}\text{, 256}\end{array}\right]$$\times$3	$\left[\begin{array}{c}\text{1}\times \text{1}{}^{\text{2}}\text{, 64}\\ \text{1}\times \text{3}{}^{\text{2}}\text{, 64}\\ \text{1}\times \text{1}{}^{\text{2}}\text{, 256}\end{array}\right]$$\times$3	8$\times$56$\times$56
res3	$\left[\begin{array}{c}\text{1}\times \text{1}{}^{\text{2}}\text{, 128}\\ \text{3}\times \text{3}{}^{\text{2}}\text{, 128}\\ \text{1}\times \text{1}{}^{\text{2}}\text{, 512}\end{array}\right]$$\times$4	$\left[\begin{array}{c}\text{1}\times \text{1}{}^{\text{2}}\text{, 128}\\ \text{1}\times \text{3}{}^{\text{2}}\text{, 128}\\ \text{3}\times \text{1}{}^{\text{2}}\text{,128}\\ \text{1}\times \text{1}{}^{\text{2}}\text{, 512}\end{array}\right]$$\times$4	$\left[\begin{array}{c}\text{1}\times \text{1}{}^{\text{2}}\text{, 128}\\ \text{1}\times \text{3}{}^{\text{2}}\text{, 128}\\ \text{1}\times \text{1}{}^{\text{2}}\text{, 512}\end{array}\right]$$\times$4	8$\times$28$\times$28
res4	$\left[\begin{array}{c}\text{1}\times \text{1}{}^{\text{2}}\text{, 256}\\ \text{3}\times \text{3}{}^{\text{2}}\text{, 256}\\ \text{1}\times \text{1}{}^{\text{2}}\text{, 1024}\end{array}\right]$$\times$6	$\left[\begin{array}{c}\text{1}\times \text{1}{}^{\text{2}}\text{, 256}\\ \text{1}\times \text{3}{}^{\text{2}}\text{, 256}\\ \text{3}\times \text{1}{}^{\text{2}}\text{,256}\\ \text{1}\times \text{1}{}^{\text{2}}\text{, 1024}\end{array}\right]$$\times$6	$\left[\begin{array}{c}\text{1}\times \text{1}{}^{\text{2}}\text{, 256}\\ \text{1}\times \text{3}{}^{\text{2}}\text{, 256}\\ \text{1}\times \text{1}{}^{\text{2}}\text{, 1024}\end{array}\right]$$\times$6	8$\times$14$\times$14
res5	$\left[\begin{array}{c}\text{1}\times \text{1}{}^{\text{2}}\text{, 512}\\ \text{3}\times \text{3}{}^{\text{2}}\text{, 512}\\ \text{1}\times \text{1}{}^{\text{2}}\text{, 2048}\end{array}\right]$$\times$3	$\left[\begin{array}{c}\text{1}\times \text{1}{}^{\text{2}}\text{, 512}\\ \text{1}\times \text{3}{}^{\text{2}}\text{, 512}\\ \text{3}\times \text{1}{}^{\text{2}}\text{,512}\\ \text{1}\times \text{1}{}^{\text{2}}\text{, 2048}\end{array}\right]$$\times$3	$\left[\begin{array}{c}\text{1}\times \text{1}{}^{\text{2}}\text{, 512}\\ \text{1}\times \text{3}{}^{\text{2}}\text{, 512}\\ \text{1}\times \text{1}{}^{\text{2}}\text{, 2048}\end{array}\right]$$\times$3	8$\times$7$\times$7
global average pool, fc				1$\times$1$\times$1

The detailed model structures for R2D, R(2+1)D and R3D is specified in Table A3. We highlight the convolutions that are replaced by TAdaConv by default or optionally. For all of our models, a small modification is made in that we remove the max pooling layer after the first convolution and set the spatial stride of the second stage to be 2, following [32]. Temporal resolution is kept unchanged following recent works [19, 128, 30]. Our R3D is obtained by simply expanding the R2D baseline in the temporal dimension by a factor of three. We initialize with weights reduced by 3 times, which means the original weight is evenly distributed in adjacent time steps. We construct the R(2+1)D by adding a temporal convolution operation after the spatial convolution. The temporal convolution can also be optionally replaced by TAdaConv, as shown in both the manuscript and Table A5. For its initialization, the temporal convolution weights are randomly initialized, while the others are initialized with the pre-trained weights on ImageNet. For SlowFast models, we keep all the model structures identical to the original work [19].

For TAdaConvNeXt, we keep most of the model architectures as in ConvNeXt [36], except that we use a tubelet embedding similar to [49], with a size of 3$\times$4$\times$4 and stride of 2$\times$4$\times$4. Center initialization is used as in [49]. Based on this, we simply replace the depth-wise convolutions with TAdaConv to construct TAdaConvNeXt. For TAdaConvNeXtV2, we additionally substitute TAdaConv for TAdaConvV2 and introduce the temporal aggregation scheme.

Figure A1: Per-category performance comparison of TAda2D against the baseline TSN. We achieve an average per-category performance improvement of 30.35%.

Figure A2: Per-category performance comparison of TAdaConvNeXtV2 against the baseline ConvNeXt. We achieve an average per-category performance improvement of 21.36%.

TABLE A4: Ablation studies.

Kernel size	Top-1
1	37.5
3	56.5
5	57.3
7	56.5

(a)

	K2=1	K2=3	K2=5	K2=7
K1=1	36.8	57.1	57.8	57.9
K1=3	57.3	57.8	57.9	58.0
K1=5	57.6	57.9	58.2	57.9
K1=7	57.4	57.6	58.0	57.6

(b)

Ratio $r$	Top-1
1	57.79
2	57.83
4	57.78
8	57.66

(c)

TABLE A5: Classification accuracy on Epic-Kitchens-100 [85]. $\mathbf{\uparrow }$ indicates the main evaluation metric for the dataset. For fair comparison, we implement all the baseline models using our own training strategies.

				Top-1			Top-5
Model	Frames	GFLOPs	Params.	Act.$\mathbf{\uparrow }$	Verb	Noun	Act.$\mathbf{\uparrow }$	Verb	Noun
SlowFast 4$\times$16	4+32	36.10	34.5M	38.17	63.54	48.79	58.68	89.75	73.37
SlowFast 4$\times$16 + TAdaConv	4+32	36.11	37.7M	39.14	64.50	49.59	59.21	89.67	73.88
SlowFast 8$\times$8	8+32	65.71	34.5M	40.08	65.05	50.72	60.10	90.04	74.26
SlowFast 8$\times$8 + TAdaConv	8+32	65.73	37.7M	41.35	66.36	52.32	61.68	90.59	75.89
R(2+1)D	8	49.55	28.1M	37.45	62.92	48.27	58.02	89.75	73.60
R(2+1)D + TAdaConv${}_{\text{2d}}$	8	49.57	31.3M	39.72	64.48	50.26	60.22	90.01	75.06
R(2+1)D + TAdaConv${}_{\text{2d+1d}}$	8	49.58	34.4M	40.10	64.77	50.28	60.45	89.99	75.55
R3D	8	84.23	47.0M	36.67	61.92	47.87	57.47	89.02	73.05
R3D + TAdaConv${}_{\text{3d}}$	8	84.24	50.1M	39.30	64.03	49.94	59.67	89.84	74.56

TABLE A6: Plug-in evaluation of TAdaConv on the action localization on HACS and Epic-Kitchens. $\mathbf{\uparrow }$ indicates the main evaluation metric for the dataset. ‘S.F.’ is SlowFast network.

	HACS						Epic-Kitchen-100
Model	@0.5	@0.6	@0.7	@0.8	@0.9	Avg.$\mathbf{\uparrow }$	Task	@0.1	@0.2	@0.3	@0.4	@0.5	Avg.$\mathbf{\uparrow }$
S.F. 8$\times$8	50.0	44.1	37.7	29.6	18.4	33.7	Verb	19.93	18.92	17.90	16.08	13.24	17.21
							Noun	17.93	16.83	15.53	13.68	11.41	15.07
							Act.$\mathbf{\uparrow }$	14.00	13.19	12.37	11.18	9.52	12.04
S.F. 8$\times$8 + TAdaConv	51.7	45.7	39.3	31.0	19.5	35.1	Verb	19.96	18.71	17.65	15.41	13.35	17.01
							Noun	20.17	18.90	17.58	15.83	13.18	17.13
							Act.$\mathbf{\uparrow }$	14.90	14.12	13.32	12.07	10.57	13.00

Appendix E Per-category improvement analysis on SSV2

This section provides a per-category improvement analysis on the Something-Something-V2 dataset in Fig.A1 and Fig. A2. In terms of overall performance, our TAda2D achieves an improvement of 31.7% over the baseline TSN, while TAdaConvNeXtV2 improves over ConvNeXt by 25.4%. Our per-category analysis shows a mean improvement of 30.35% and 21.36% over all the classes, respectively for TAda2D and TAdaConvNeXtV2. Since both TSN and ConvNeXt have no temporal modeling capabilities, and our approach introduce similar modifications to the base model, the difference pattern in the per-category accuracy is similar. Hence, we take TAda2D as an example for analysis. The largest improvement is observed in class 0 (78.5%, Approaching something with your camera), 32 (78.4%, Moving away from something with your camera), 30 (74.3%, Lifting up one end of something without letting it drop down), 44 (66.2%, Moving something towards the camera) and 41 (66.1%, Moving something away from the camera). Most of these categories contain large movements across the whole video, whose improvement benefits from temporal reasoning over the global spatial context. For class 30, most of its actions last a long time (as it needs to be determined whether the end of something is let down or not). The improvements over the baseline mostly benefit from the global temporal context that is included in the weight generation process.

Appendix F Further ablation studies

Here we provide further ablation studies on the kernel size in the calibration weight generation. As shown in Table LABEL:tab:ablationstudieskernelsizelin and Table LABEL:tab:ablationstudieskernelsizenonlin, kernel size does not affect the classification much, as long as the temporal context is considered. Further, Table LABEL:tab:ablationstudiesreductionratio shows the sensitivity analysis on the reduction ratio, which demonstrate the robustness of our approach against different set of hyper-parameters.

Appendix G Further plug-in evaluation for TAdaConv on classification

In complement to the manuscript, we further show in Table A5 the plug-in evaluation on the action classification task on the Epic-Kitchens-100 dataset. As in the plug-in evaluation on Kinetics and Something-Something-V2, we compare performances with and without TAdaConv over three baseline models, SlowFast [19], R(2+1)D [13] and R3D [100] respectively representing three kinds of temporal modeling techniques. The results are in line with our observation in the plug-in evaluation in the manuscript. Over all three kinds of temporal modelling strategies, adding TAdaConv further improves the recognition accuracy of the model.

Figure A3: Training and validation on Kinetics-400 and Something-Something-V2 for TAda2D. On both datasets, TAda2D shows a stronger capability of fitting the data and a better generality to the validation set. Further, TAda2D reduces the overfitting problem in Something-Something-V2.

Figure A4: Training and validation on Kinetics-400 and Something-Something-V2 for TAdaConvNeXt.

Appendix H Plug-in evaluation for TAdaConv on Action Localization

Here, we show the plug-in evaluation on the temporal action localization task. Specifically, we use SlowFast as our baseline, as it is shown to be superior in the localization performance in [129] compared to many early backbones. The result is presented in Table A6. With TAdaConv, the average mAP on HACS is improved by 1.4%, and the average mAP on Epic-Kitchens-100 action localization is improved by 1.0%.

Appendix I Comparison of training procedure

we compare the training procedure of TSN and TAda2D on Kinetics-400 and Something-Something-V2 in Fig. A3, and that of ConvNeXt and TAdaConvNeXtV2 in Fig. A4. Although TAda2D and TAdaConvNeXtV2 are initialized to be identical to TSN and ConvNeXt, both TAda2D and TAdaConvNeXtV2 demonstrates a stronger performance on both training and validation sets.

TABLE A7: Approach comparison between different dynamic filters. The weights column denotes how weights in respective approaches are obtained. The pre-trained weights colmun shows whether the weight generation can exploit pre-trained models such as ResNet [1].

		Temporal	Location	Pretrained
Operations	Weights	Modelling	Adaptive	weights
CondConv	Mixture of experts $𝐖={\sum }_{n}f{\left(𝐱\right)}_n{𝐖}_n$	✗	✗	✗
DynamicFilter	Completely generated $𝐖=g\left(𝐱\right)$	✗	✗	✗
DDF	Completely generated $𝐖=g\left(𝐱\right)$	✗	✓	✗
TAM	Completely generated $𝐖=g\left(𝐱\right)$	✓	✗	✗
TAdaConv	Calibrated from a base weight $𝐖=h\left(𝐱\right){𝐖}_b$	✓	✓	✓

TABLE A8: Performance comparison with other dynamic filters. Our Init. denotes initializing the calibration weights to ones so that the initial calibrated weights is identical to the pre-trained weights. Temp. Varying is short for temporally varying, which indicates different weights for different temporal locations (frames). * denotes that the branch was originally not designed for generating filter or calibration weights, but we slightly modified the structure so that it can be used for calibration weight generation. (Numbers in brackets) show the performance improvement brought by our initialization scheme for calibration weights.

Calibration Generation	Our Init.	Temp. Varying	Generation source	Top-1
DynamicFilter	✗	✗	${\text{GAP}}_{st}\left(𝐱\right)\left(C\times 1\right)$	41.7
DDF-like	✗	✓	${\text{GAP}}_{st}\left(𝐱\right)\left(C\times 1\right)$	49.8
TAM (global branch)	✗	✗	${\text{GAP}}_s\left(𝐱\right)\left(C\times T\right)$	39.7
TAM (local*+global branch)	✗	✓	${\text{GAP}}_s\left(𝐱\right)\left(C\times T\right)$	41.3
DynamicFilter	✓	✗	${\text{GAP}}_{st}\left(𝐱\right)\left(C\times 1\right)$	51.2 (+9.5)
DDF-like	✓	✓	${\text{GAP}}_{st}\left(𝐱\right)\left(C\times 1\right)$	53.8 (+4.0)
TAM (global branch)	✓	✗	${\text{GAP}}_s\left(𝐱\right)\left(C\times T\right)$	52.9 (+13.2)
TAM (local*+global branch)	✓	✓	${\text{GAP}}_s\left(𝐱\right)\left(C\times T\right)$	54.3 (+13.0)
TAdaConv w/o global info $𝐠$	✓	✓	${\text{GAP}}_s\left(𝐱\right)\left(C\times T\right)$	57.9
TAdaConv	✓	✓	both ${\text{GAP}}_{st}\left(𝐱\right)\left(C\times 1\right)$	59.2
			and ${\text{GAP}}_s\left(𝐱\right)\left(C\times T\right)$

Appendix J Comparison with existing dynamic filters

In this section, we compare our TAdaConv with previous dynamic filters in two perspectives, respectively the difference in the methodology and in the performance.

J.1 Comparison in terms of methodology

We compare TAdaConv with several representative dynamic filtering approaches in image and in videos, respectively CondConv [11], DynamicFilter [21], DDF [17] and TAM [31].

The first difference in terms of methodology lies in the source of weights, where previous approaches obtain weights by mixture of experts or generation completely dependent on the input. Mixture of experts denotes $𝐖={\sum }_n{\alpha }_n{𝐖}_n$, where ${\alpha }_n$ is a scalar obtained by a function $f$, i.e., $𝐖={\sum }_{n}f{(𝐱)}_n{𝐖}_n$. Completely generated means the weights are only dependent on the input, i.e., $𝐖=g((𝐱))$, where $g$ generates complete kernel for the convolution. In comparison, the weights in TAdaConv are obtained by calibration, i.e,, $𝐖=𝜶{𝐖}_b$, where $𝜶$ is a vector calibration weight and $𝜶=h((𝐱))$ where $h(.)$ generates the calibration vector for the convolutions. Hence, this fundamental difference in how to obtain the convolution weights makes the previous approaches difficult to exploit pre-trained weights, while TAdaConv can easily load pre-trained weights in ${𝐖}_b$. This ability is essential for video models to speed up the convergence.

The second difference lies in the ability to perform temporal modelling. The ability to perform temporal modelling does not only mean the ability to generate weights according to the whole sequence in dynamic filters for videos, but it also requires the model to generate different weights for the same set of frames with different orders. For example, weights generated by the global descriptor obtained by global average pooling over the whole video ${\text{GAP}}_{st}$ does not have the temporal modelling ability, since they can not generate different weights if the order of the frames in the input sequence are reversed or randomized. Hence, most image based approaches based on global descriptor vectors (such as CondConv and DynamicFilter) or based on adjacent spatial contents (DDF) can not achieve temporal modelling. TAM generates convolution weights for temporal convolutions based on temporally local descriptors obtained by the global average pooling over the spatial dimension ${\text{GAP}}_s$, which yields different weights if the sequence changes. Hence, in this sense, TAM has the temporal modelling abilities. In contrast, TAdaConv exploits both temporally local and global descriptors to utilize not only local but also global temporal contexts. Details on the source of the weight generation process is also shown in Table A8.

The third difference lies in whether the weights generated are shared for different locations. For CondConv, DynamicFilter and TAM, their generated weights are shared for all locations, while for DDF, the weights are varied according to spatial locations. In comparison, TAdaConv generates temporally adaptive weights.

J.2 Comparison in the performance level

Since TAdaConv is fundamentally different from previous approaches in the generation of calibration weights, it is difficult to directly compare the performance on video modelling, especially for those that are not designed for video modelling. However, since the calibration weight in TAdaConv $𝜶$ is completely generated, i.e., $𝜶=f((𝐱))$, we can use other dynamic filters to generate the calibration weights for TAdaConv. Since MoE-based approaches such as CondConv were essentially designed for applications with less memory constraint but high computation requirements, it is not suitable for video applications since it would be too memory-heavy for video models. Hence, we apply approaches that generate complete kernel weights to generate calibration weights and compare them with TAdaConv. The performance is listed in Table A8.

It is worth noting that these approaches originally generate weights that are randomly initialized. However, as is shown in the manuscript, our initialization strategy for the calibration weights are essential for yielding reasonable results, we further apply our initialization on these existing approaches to see whether their generation function is better than the one in TAdaConv. In the following paragraphs, we provide details for applying representative previous dynamic filters in TAdaConv to generate the calibration weight.

For DynamicFilter [21], the calibration weight $𝜶$ is generated using an MLP over the global descriptor that is obtained by performing global average pooling over the whole input ${\text{GAP}}_{st}$, i.e., $𝜶=\text{MLP}({\text{GAP}}_{st}(𝐱))$. In this case, the calibration weights are shared between different time steps.

For DDF [17], we only use the channel branch since it is shown in the manuscript that it is better to leave the spatial structure unchanged for the base kernel. Similarly, the weights in DDF are also generated by applying an MLP over the global descriptor, i.e., $𝜶=\text{MLP}({\text{GAP}}_{st}(𝐱))$. The difference between DDF and DynamicFilter is that for different time step, DDF generates a different calibration weight.

The original structure of TAM [31] only generates kernel weights with its global branch and uses the local branch to generate attention maps over different time steps. In our experiments, we modify the TAM a little bit and further make the local branch generate kernel calibration weights as well. Hence, for the only-global version of TAM, the calibration weights are calculated as follows: $𝜶=𝒢({\text{GAP}}_s(𝐱))$, where ${\text{GAP}}_s$ denotes global average pooling over the spatial dimension and $𝒢$ denotes the global branch in TAM. In this case, calibration weights are shared for all temporal locations. For local+global version of TAM, the calibration weight are calculated by combining the results of the local $ℒ$ and the global branch $𝒢$, i.e., $𝜶=𝒢({\text{GAP}}_s(𝐱))\cdot ℒ({\text{GAP}}_s(𝐱))$, where $\cdot$ denotes element-wise multiplication with broadcasting. This means in this case, the calibration weights are temporally adaptive. Note that this is our modified version of TAM. The original TAM droes not have temporally adaptive convolution weights.

The results in Table A8 show that (a) without our initialization strategy, previous approaches that generate random weights at initialization are not suitable for generating the calibration weights in TAdaConv; (b) our initialization strategy can conveniently change this and make previous approaches yield reasonable performance when they are used for generating calibration weights; and (c) the calibration weight generation function in TAdaConv, which combines the local and global context, outperform all previous approaches for calibration.

Further, when we compare TAdaConv without global information with TAM (local*+global branch), it can be seen that although both approach generates temporally varying weights from the frame descriptors ${\text{GAP}}_s(𝐱)$ with shape $C\times T$, our TAdaConv achieves a notably higer performance. Adding the global information enables TAdaConv to achieve a more notable lead in the comparison with previous dynamic filters.