Custom Video Export/Import Process With Views and Feeds

In the Media Research Center's set of three main Drupal sites, MRCTV serves as our video platform where all videos are created and stored as nodes, and then using Feeds, specific videos are imported into the other two sites (Newsbusters and CNS News) as video nodes. Then, on NB and CNS, we use the Video Embed Field module with a custom VEF provider for MRCTV to play the embedded videos.

There are only specific videos that need to be imported into the destination sites, so a way to map channels between the two sites is needed. All three sites have a Channels vocabulary, a mapping is created between the appropriate channels. This mapping has two parts:

  1. A feed of channels terms on NB and CNS.
  2. A custom admin form that links source channels on MRCTV with target channels on the destination site.

On the receiving site side, in addition to the standard feed content, the following custom elements are needed for the Feeds import:

  1. The nid of the video node on MRCTV. This is used to create the URL that is put into the VEF field.
  2. The taxonomy terms for the Channels vocabulary terms in the destination sites (NB and CNS).

Since these are outside of the standard feed components, they will need to be added custom to the feed items.

I documented my custom Feeds importer on drupal.stackexchange, so you can see the code there.

MRCTV is finally in the process of being updated to D8 from D6 (insert derision here), so both the mapping form and the feed needed to be re-created. The first part of the structure is the channel mapping form. The following file VideoExportForm.php is placed in /modules/custom/video_export/src/Form:

/**
 * @file
 * Contains \Drupal\video_export\Form\VideoExportForm.
 */

namespace Drupal\video_export\Form;

use Drupal\Core\Config\ConfigFactoryInterface;
use Drupal\Core\Form\ConfigFormBase;
use Drupal\Core\Form\FormStateInterface;
use GuzzleHttp\Exception\RequestException;

class VideoExportForm extends ConfigFormBase {
  
  /**
   * {@inheritdoc}.
   */
  public function getFormId() {
    return 'video_export_settings';
  }
    
  /**
   * {@inheritdoc}
   */
  public function buildform(array $form, FormStateInterface $form_state) {
    $form = array();
    $channels = array();
    
    // Get list of channels.
    $terms =\Drupal::entityTypeManager()->getStorage('taxonomy_term')->loadTree('channels');
    foreach ($terms as $term) {
      $channels[$term->tid] = $term->name;
    }
    
    // Get config data from video_export.settings.yml.
    $config = \Drupal::config('video_export.settings');
    $mapping_config = \Drupal::config('video_export.mappings');
    $sites = $config->get('sites');
    
    foreach($sites as $site => $site_data) {
      // Get channels list.
      try {
        $response = \Drupal::httpClient()->get($site_data['channel_url'], array('headers' => array('Accept' => 'text/plain')));
        $data = $response->getBody();
        if (empty($data)) {
          return FALSE;
        }
      }
      catch (RequestException $e) {
        return FALSE;
      }
  
      $channel_data = new \SimpleXMLElement($data);
      foreach ($channel_data->channel as $channel) {
        $channel_name = $channel->name->__toString();
        $channel_tid = $channel->tid->__toString();
        $target_channels[$channel_tid] = $channel_name;
      }
      // Sort array alphabetically by element.
      asort($target_channels, SORT_STRING);
  
      $target_channel_options = array();
      $target_channel_options[0] = "No Channel";
      foreach ($target_channels as $target_tid => $target_name) {
        $target_channel_options[$target_tid] = $target_name;
      }
  
      //Get mappings from mappings conifg.
      $mappings = $mapping_config->get('sites');
      foreach ($mappings[$site]['mappings'] as $mrctv_channel => $target_channel) {
        $mapping_defaults[$mrctv_channel] = $target_channel;
      }
  
      $form[$site] = array(
        '#type' => 'details',
        '#title' => t($site . ' Channel Mappings'),
        '#description' => t('Map MRCTV channels to ' . $site . ' channels'),
        '#collapsible' => TRUE,
        '#collapsed' => TRUE,
        '#tree' => TRUE,
      );
  
      // Loop through all of the categories and create a fieldset for each one.
      foreach ($channels as $id => $title) {
        $form[$site]['channels'][$id] = array(
          '#type' => 'select',
          '#title' => $title,
          '#options' => $target_channel_options,
          '#tree' => TRUE,
        );
        if (in_array($id, array_keys($mapping_defaults))) {
          $form[$site]['channels'][$id]['#default_value'] = intval($mapping_defaults[$id]);
        }
      }
    }
    
    // Get mapping configs.
    $xml = array();
    $mapping_config = \Drupal::config('video_export.mappings');
    $sites = $mapping_config->get('sites');
    $channel_mappings = $sites[$site]['mappings'];
    // Get video nodes that belong to one of the selected channels.
    $query = \Drupal::entityQuery('node')
      ->condition('status', 1)
      ->condition('type', 'video')
      ->condition('changed', REQUEST_TIME - 59200, '>=')
      ->condition('field_channels.entity.tid', array_keys($channel_mappings), 'IN');
    $nids = $query->execute();
    // Load the entities using the nid values. The array keys are the associated vids.
    $video_nodes = \Drupal::entityTypeManager()->getStorage('node')->loadMultiple($nids);

    foreach ($video_nodes as $nid => $node) {
      $host = \Drupal::request()->getSchemeAndHttpHost();
      $url_alias = \Drupal::service('path.alias_manager')->getAliasByPath('/node/' . $nid);
      // Get channels values.
      $channel_tids = array_column($node->field_channels->getValue(), 'target_id');
      $create_date = \Drupal::service('date.formatter')->format($node->getCreatedTime(), 'custom', 'j M Y h:i:s O');
      $item = array(
        'title' => $node->getTitle(),
        'link' => $host . $url_alias,
        'description' => $node->get('body')->value,
        'mrctv-nid' => $nid,
        'guid' => $nid . ' at ' . $host,
        'pubDate' => $create_date
      );
      // Check for short title and add it if it's there.
      if ($node->get('field_short_title')->value) {
        $item['short-title'] = $node->get('field_short_title')->value;
      }
      foreach ($channel_tids as $ctid) {
        $item[$site . '-channel-map'][] = $ctid;
      }
      $xml[] = $item;
    }

    return parent::buildForm($form, $form_state);
  }
  
  /**
   * {@inheritdoc}.
   */
  public function validateForm(array &$form, FormStateInterface $form_state) {
  
  }
  
  protected function getEditableConfigNames() {
    return ['video_export.mappings'];
  }
  
  /**
   * {@inheritdoc}
   */
  public function submitForm(array &$form, FormStateInterface $form_state) {
    $values = $form_state->getValues();
    $config = $this->config('video_export.mappings');
    $sites = array();

    foreach($values as $site => $mappings) {
      if (is_array($mappings)) {
        foreach ($mappings['channels'] as $mrctv_channel => $target_channel) {
          if ($target_channel != 0) {
            $sites[$site]['mappings'][$mrctv_channel] = $target_channel;
          }

        }
        $config->set('sites', $sites);
      }
    }
    $config->save();
  
    parent::submitForm($form, $form_state);
  }
}

The setting for the channels feeds on NB and CNS are stored in /modules/custom/video_export/config/install/video_export.settings.yml:

sites:
  newsbusters:
    channel_url: 'http://www.newsbusters.org/path/to/channels'
  cnsnews:
    channel_url: 'http://www.cnsnews.com/path/to/channels'
list_time: 24
	

Since this is an admin settings form, I extend the ConfigFormBase class. This adds some additional functionality over the standard FormBase class, similar to the way the system_settings_form() function does in D7 and older (see the change record for details).

As mentioned above the form does the following things:

  1. Reads the channels feed from the destination sites
  2. Creates a fieldset for each site with a select list for each MRCTV channel where the user can select the destination channel.
  3. Saves the mappings in config.

The next thing that is needed is the feed of video nodes that are available to be imported. After trying unsuccessfully to create a custom REST API endpoint, I ended up going with a Feeds display in Views. Out of the box I can create my feed, but I still need to add my custom elements. In D6, I used hook_nodeapi($op = 'rss item') to add my custom elements. In other feeds on D7 sites I've been able to use the Views RSS module with its provided hooks to add custom RSS elements, but as of now it is currently unusable for D8 due to one major issue.

Finally, since everything in D8 is based on OOP, I knew there had to be a way to override a Views class at some level, so after some searching, I decided to override the display plugin. I poked around in the Views code and found the RssFields class that is used for the field level display for a Feeds display, so I overrode that.

namespace Drupal\video_export\Plugin\views\row;

use Drupal\views\Plugin\views\row\RssFields;
use Drupal\Core\Form\FormStateInterface;
use Drupal\Core\Url;

/**
 * Renders an RSS item based on fields.
 *
 * @ViewsRow(
 *   id = "mrctv_rss_fields",
 *   title = @Translation("MRCTV Fields"),
 *   help = @Translation("Display fields as RSS items."),
 *   theme = "views_view_row_rss",
 *   display_types = {"feed"}
 * )
 */
class MRCTVRssFields extends RssFields {
  
  /**
   * Override of RssFields::render() with additional fields.
   *
   * @param object $row
   *
   * @return array
   */
  public function render($row) {
    $build = parent:: render();
    $item = $build['#row'];
    
    // Add MRCTV nid
    $item->elements[] = array(
      'key' => 'mrctv-nid',
      'value' => $row->nid,
    );
  
    // Add channels and their target nids. We can get them from $row->_entity.
    $site = $this->view->args[0];
    // Get MRCTV nids from view.
    $channel_tids = array_column($row->_entity->field_channels->getValue(), 'target_id');
    // Now, get destination tids from config.
    $mapping_config = \Drupal::config('video_export.mappings');
    $all_mappings = $mapping_config->get('sites');
  
    foreach($channel_tids as $mrctv_channel) {
      if(in_array($mrctv_channel, array_keys($all_mappings[$site]['mappings']))) {
        $item->elements[] = array(
          'key' => $site . '-channel-map',
          'value' => $all_mappings[$site]['mappings'][$mrctv_channel],
        );
      }
    }
    
    // Re-populate the $build array with the updated row.
    $build['#row'] = $item;
    
    return $build;
  }
}

As you can see, the override is fairly simple; all I needed to do was override the render() method. This method returns a render array, so all I do is get the built array from the parent class, add my custom elements to the #row element in the array, and return it.

One thing that I couldn't do simply in the views UI was select the nodes that should be in the feed based on the associate Channels vocabulary terms. These are dynamic, based on the mappings selected in the admin form, so I can't pre-select them in the view settings. This is where hook_views_query_alter() comes to the rescue.

/**
 * Implements hook_views_query_alter().
 */
function video_export_views_query_alter(Drupal\views\ViewExecutable $view, Drupal\views\Plugin\views\query\Sql $query) {
  if ($view->id() == 'video_export' && $view->getDisplay()->display['id'] == 'feed_1') {
    // First, we need to get the site parameter from the view.
    $site = $view->args[0];
    
    // Next, we need to get the saved config for the channel mapping.
    $mapping_config = \Drupal::config('video_export.mappings');
    $all_mappings = $mapping_config->get('sites');
    $tids = array_keys($all_mappings[$site]['mappings']);
   
    // Modify query to get nodes that have the selected nids, which are the array keys.
    $query->addWhere(NULL, 'node__field_channels.field_channels_target_id', $tids, 'IN');
  }
}

All I do here is get the saved mappings from config and add them to the views query as a WHERE condition to limit the feed items to the appropriate nodes.

One issue I ran into with the results was duplicate records. Since field_channels (the entity reference field for the Channels vocabulary) is multiselect, the query returns multiple records for each node if there are multiple Channels terms selected. There are display settings to show multiple items in one row, but they don't take effect here. I didn't dig far enough into the code to know for sure, but my guess is that the grouping happens at a higher layer in the views rendering process, so they don't take effect in this situation.

To get around this, I implemented hook_views_pre_render(). At this point in the process, the results have been built, so I just loop through them and remove duplicates.

/**
 * Implements hook_views_query_pre_render().
 */
function video_export_views_pre_render(Drupal\views\ViewExecutable $view) {
  $unique_nids = $new_results = array();
  
  // Loop through results and filter out duplicate results.
  foreach($view->result as $index => $result) {
    if(!in_array($result->nid, $unique_nids)) {
      $unique_nids[] = $result->nid;
    }
    else {
      $new_results[] = $result;
    }
  }
  // Replace $view->result with new array. Apparently views requires sequentially keyed
  // array of results instead of skipping keys (e.g. 0, 2, 4, etc), so we can't just
  // unset the duplicates.
  $view->result = $new_results;
}

As noted in the code comment, views seems to require a sequentially numbered array, so you can't just unset the duplicate keys and leave it as is, so I chose to just add each item to a new array. In retrospect, I could have just used PHP functions like array_splice() and array_filter(), but this method works just as well.

It should also be noted that the views hooks need to go in a *.views_execution.inc file, so this one is in /modules/custom/video_export/video_export.views_execution.inc.

All I do at this point is use the Job Scheduler module with Feeds in the destination sites to schedule the import at the desired interval, and the process runs by itself. 

May2