• Learn to cycle: Time-consistent feature discovery for action recognition (Pattern Recognition Letters 2021) [pdf]
• Multi-Temporal Convolutions for Human Action Recognition in Videos (IJCNN 2021) [pdf]

Ensure that the following packages are installed in your machine:

• coloredlogs (version >= 14.0)
• ffmpeg-python (version >=0.2.0)
• imgaug (version >= 0.4.0)
• opencv-python (version >= 4.2.0.32)
• torch (version >= 1.9.0)
• youtube-dl (version >= 2020.3.24)

You can install all of the packages with the following command:

$ pip install coloredlogs ffmpeg-python imgaug opencv-python torch torchvision youtube-dl

For calculating FLOPs/GMACs yourself it's suggested to use Vladislav Sovrasov's repo [link]

You can also calculate the forward/backward pass latency with the following simple code snippet:

import time
  tmp = torch.rand(1,3,16,224,224).cuda()
  #--- INFERENCE TIME TEST ---
  net = srtg_r3d_101(num_classes=400).cuda()

  tot_forwrd_inf = 0
  tot_backward_inf = 0
  count = 0

  for epoch in range(0, 1000):

      # Forward inference
      torch.cuda.synchronize()
      since = int(round(time.time()*1000))
      output = net(tmp)
      torch.cuda.synchronize()
      time_elapsed = int(round(time.time()*1000)) - since

      loss = torch.mean(output)

      # Backward inference
      torch.cuda.synchronize()
      tsince = int(round(time.time()*1000))
      loss.backward()
      torch.cuda.synchronize()
      ttime_elapsed = int(round(time.time()*1000)) - tsince

      tot_forwrd_inf += time_elapsed
      tot_backward_inf += ttime_elapsed
      count +=1


  print ('Forward  inference time elapsed {}ms'.format(tot_forwrd_inf/count))
  print ('Backward inference time elapsed {}ms'.format(tot_backward_inf/count))

! Disclaimer: This repository is heavily structurally influenced on Yunpeng Chen's MFNet repo [link]

Please also make sure that git is installed in your machine:

$ sudo apt-get update
$ sudo apt-get install git
$ git clone https://github.com/alexandrosstergiou/Squeeze-and-Recursion-Temporal-Gates.git

A custom format is used for all label files for each of the datasets. If using (1) your own data or (2) converting a file please ensure that the data are saved to .csv files with the following format:

We include training/data loading scripts for six action/video recognition datasets:

• Human Action Clips and Segments (HACS) [link] : It includes a total of ~500K clip segments sampled over 50K videos. you can download the dataset from this link, or alternatively visit the project's website.
• Kinetics [link]: Composed of approximately 600K clips over 700 classes (with previously 400 & 600) and each clip having an average duration of 10 seconds. You can download all three sets from this link.
• Moments in Time (MiT) [link]: Is composed of about 800K videos over 339 classes. Video durations are limited to 3 seconds. The labels can be downloaded from the website after competing the form.
• UCF-101 [link]: A dataset of 13320 clips of 2~14 seconds. It includes a total of 101 action classes. The dataset can be downloaded in full from their website.
• HMDB-51 [link]: The dataset includes a bit less than 10K video of 51 categories sampled from movies and series. The dataset can be downloaded directly from their website.
• Diving-48 [link]: Composed of a total 18K videos of diving actions. The dataset can be downloaded from the project's website.

All three HACS, Kinetics and MiT datasets can be dowloaded through the ActivityNet's official Crawler by changing the download script. Direct downloads are not (officially) supported due to the vastness of these datasets.

Instead of extracting video frames to images we opt to use SQL databases for each of the videos. This effectively (1) limits the number of random read operations and (2) The number of inodes used as individual image files (.png/.jpeg/etc.) increase the number of inodes used significantly.

In total, the speed-up in reading times is close to ~1000x of that with conventional random-access image files

Table for loading speed comparisons between JPG images and SQL databases. Rates are denoted as clips/videos loaded per second (clips/sec.)

All of the experiments were done with an AMD Threadripper 2950X (128GB of RAM w/ 2666MHz) over 24 workers. Disks used: 2x Samsung 970 Evo Plus (2TB & 1TB).

You can use the dataset2databse pypi package or repo to convert videos to SQL:

$ pip install dataset2database

We assume a fixed directory formatting for both the data and the labels used. The dataset should be of the following structure at a directory:

<dataset>
  |
  └─── jpg
        │
        └──<class_i>
        │     │
        │     │─── <video_id_j>
        │     │         │
        │     │         │─── frames.db
        │     │         └─── n_frames
        │     │    
        │     │─── <video_id_j+1>
        │     │         │
        │     │         │─── frames.db
       ...   ...        └─── n_frames

In the structure, any items enclosed in angle brackets should be changed to the correct dataset name, class names and video ids. The three standard elements that should remain the same across any dataset are:

• jpg: The the container folder for all the classes in the dataset.

• frames.db: The SQL database specific containing all the video frames in the format of ObjId: which should be a string containing the video filepath alongside the frame number and frames that encapsulate all the data. The SQL table should also be called Images.

• n_frames: A file that should only include the number of frames for the video for quick access.

You can also use your own dataset if you follow the above structure and convert all your videos to SQL databases. The process for doing so should be identical for any of the currently supported datasets.

The main data loading takes place in data/video_iterator.py which you can see for more information. Lines 87~110 handle both connecting to the sql databases and loading. A densely commented version of those lines can be found below for more info:

con = sqlite3.connect('my_video_database.db')# Establishing connection
cur = con.cursor()# Cursor creation
frame_names = ["{}/{}".format(my_path.split('/')[-1],'frame_%05d'%(index+1)) for index in frame_indices]# Frame indices selection
sql = "SELECT Objid, frames FROM Images WHERE ObjId IN ({seq})".format(seq=','.join(['?']*len(frame_names)))# Build SQL request
row = cur.execute(sql,frame_names)# Execute SQL and retrieve frames
ids = []
frames = []
i = 0
row = row.fetchall()# Data fetching
# Video order re-arrangement
for ObjId, item in row:
  #--- Decode blob
  nparr  = np.fromstring(item, np.uint8)
  img = cv2.imdecode(nparr, cv2.IMREAD_COLOR)
  ids.append(ObjId)
  frames.append(img)
  i+=1
# Ensuring correct order of frames
frames = [frame for _, frame in sorted(zip(ids,frames), key=lambda pair: pair[0])]
# (if required) array conversion [frames x height x width x channels]
frames = np.asarray(frames)

To increase generalisation capabilities of the models we further include video augmentations inside our training process. These augmentations are primarily divided to temporal-based augmentations and spatial-based augmentations.

• Temporal-based augmentations are used to perform changes at the temporal extend of the videos. Therefore, any variations based on time include:

  • Temporal sampling interval, which is the maximum number of skipped frames between two frames from which the final video will be composed by.
  • Frame sampling, which can either be done sequentially (e.g. centre-most frames) or randomly through a uniform/normal distribution.

• Spatial-based augmentations are performed in frame-level and primarily change either the width-height or the channels/RGB values. We set a probability of 0.8 for performing any augmentations for each video, while each augmentation type is assigned a 0.4 probability. We make use of the imgaug package for all of our spatial augmentations:

  • Gaussian blur with a sigma { 0.1 , 0.2, 0.3}.
  • Per-channel addition in range of {-5, ..., 15}.
  • Gaussian noise with a scale of 76.5 (i.e 0.3 * 255).
  • Colour enhancement with a factor of (1.2, 1.6).
  • Motion blur with a kernel of { 3, 5, 7}.
  • Hue saturation addition {-16, ..., 16}.
  • Linear contrast with an alpha of {0.85, ..., 1.115}.
  • Perspective transform with scale {0.02,0.05}.
  • Rotation between {-10, ..., 10} degrees.

  Ranges and values were empirically evaluated in order to balance between a reasonable amount of deformation without alleviating the informative features of the video. It's important to note the all video frames should present exactly the same type of spatial augmentations to ensure coherence.

We additionally use a Multigrid training schedule for both improving generalisation and training times. Our implementation is based on the Wu et al. paper. For convenience three Dataloader objects are used for every long cycle that correspond to changes in data for each short cycle.

• In case of RAM constraints we suggest to either not use the Multigrid training implementation or decrease the number of workers at train/model.py.
• Ensure that you have at least 86GB of shared memory in your system.

Training for each of the datasets is done through the homonym scripts.

Training from scratch: Training from scratch can be done by calling the script for corresponding dataset. The two call argument that you should include are the dataset for the directory of the dataset and network for the architecture to train.

python train_hacs.py --network srtg_r3d_34 --dataset /media/usr/m2ssd_vol1/data/HACS_videos/ --clip-size 324 --gpus 4 --lr-base 0.1 --batch-size 48

Resuming training: You can also resume training by loading previous states and specifying the epoch from which to continue after. You can specify the directory of the model to load weights from with pretrained_3d. You can load models from different disks with the absolute path. Secondly, you should specify the epoch from which to resume training from with resume-epoch.

python train_hacs.py --network srtg_r3d_50 --dataset /media/usr/m2ssd_vol1/data/HACS_videos/ --clip-size 256 --gpus 4 --pretrained_3d results/HACS/srtg_r3d_50_gates_False/srtg_r3d_50_ep-0020.pth --resume-epoch 21 --lr-base 0.1 --batch-size 32

Weight initialisation/Transfer learning: It is possible to initialise weights or use pre-trained networks. Similarly to resuming training if the network architecture includes some variations (e.g. the number of class neurones), only the corresponding weights will be loaded. For transfer learning, you can additionally use the fine-tune call argument to decrease the learning rate used for convolution weights while maintaining the same base learning rate for the added layers.

python train_kinetics.py --network srtg_r3d_34 --dataset /media/usr/m2ssd_vol1/data/Kinetics_videos/ --variant 700 --clip-size 224 --gpus 4 --pretrained_3d results/HACS/srtg_r3d_34_gates_False/srtg_r3d_34_ep-0080.pth --fine-tune True --lr-base 0.1 --batch-size 48

The following arguments are used and can be included at the parser of any training script.

Special call arguments

The are some cases were additional arguments are used based on the structures or specific datasets:

(note that models are trained with float16, float32 stability has not been testes). Table headers network, clip-length, clip-size and batch-size correspond to the script parameters used.

Table headers (network, clip-length, clip-size and batch-size) correspond to passed arguments.

We are currently TRAIN models with mixed precision. Exceptions include both SoftMax and Normalisation methods that are kept in float32 instead.

We report the loss, top-1 and top-5 accuracy during each logging interval during training. For validation, we report the average loss updated after each batch. Both top-1 and top-5 accuracies are saved in a .csv file for both train and validation.

We also provide a monitor for speeds in terms of video-reading from disk (CPU), forward pass (GPU) and backprop (GPU). Speeds are reported as clips per second. You can have a look at class SpeedMonitor in train/callbacks.py for more information. Overall the output at each logging interval should look like:

Colours are used in order to give a quick understanding if the speed is in general fast, average or slow. The limits for each of the speeds are defined as:

• reading (r) (red) 100 < (yellow) 3000 < (green) inf

• forward (f) (red) 50 < (yellow) 300 < (green) inf

• backward (r) (red) 20 < (yellow) 200 < (green) inf

Note that the biggest factor for speeds is the video/clip size rather than the video size. Reading speeds will fall at average/slow speeds only during Dataloader initialisations.

Along scores, speed the batch size and learning rate are also monitored at each logging interval. This is especially useful for using cycles.

All experiments were run with 4x NVIDA 2080 Ti GPUs with a shared memory of 128GB. If you are in Ubuntu the easiest way to assign a larger amount of shared memory is to use:

$ df -h

To find the size of /dev/shm . And then edit/insert the line below in /etc/fstab:

none /dev/shm tmpfs defaults,size=86G 0 0 Then exit and run (w/ sudo):

--- If line was now added: ---
$ mount /dev/shm

--- If line was edited: ---
$ mount -o remount /dev/shm

@article{stergiou2021improved,
  title={Learn to cycle: Time-consistent feature discovery for action recognition},
  author={Stergiou, Alexandros and Poppe, Ronald},
  journal = {Pattern Recognition Letters},
  volume = {141},
  pages = {1--7},
  year = {2021},
  issn = {0167-8655},
  doi = {https://doi.org/10.1016/j.patrec.2020.11.012}
}

@inproceedings{stergiou2021right,
  title={Multi-Temporal Convolutions for Human Action Recognition in Videos},
  author={Stergiou, Alexandros and Poppe, Ronald},
  booktitle={International Joint Conference of Neural Networks (IJCNN)},
  year={2021},
  organization={IEEE}
}

MIT

Alexandros Stergiou

Any queries or suggestions are much appreciated!