Does migrating content to a new website fill you with dread? Are you tired of wasting weeks manually entering articles into a CMS? Migrating content doesn't have to be a headache. If you're creating a new website with Drupal 8 there are lots of ways to get content from wherever it currently into your new site.
Here we will go over the couple different types of migrations that I'm most familiar with, they are categorized as follows:
- Drupal to Drupal automated Migrations
- Random data migrations using Drupal Migrations
- Manual migrations using lots of painful code
I have created a Drupal 8 git repo that has all of the code and databases used in this tutorial. There are 3 databases, a drupal8.sql, drupal7.sql and random.sql. If you wanted to follow along with my code and my databases feel free to the login to the drupal 8 site is admin/admin.
Drupal to Drupal Migrations
If you have a Drupal 6 or 7 website and are trying to get the content into a new Drupal 8 website, you're in luck as it's not that painful. Inside of this type of migration there are fully automated migrations that are super easy and fast or the more customized but still not too hard handwritten migrations.
Automated / Generated Migrations
This is the easiest migration you can do, even sometimes being able to do it all through the GUI depending on what you want to do. If you want your site to be mostly the same but just on a new version of Drupal the GUI is at least where you should start.
All you have to do is install the module Migrate, Migrate Drupal, and Migrate Drupal UI (all core modules) from there it will walk you through an upgrade. That will bring over all nodes, users and a lot of other content. Some things like Menus, Views, and blocks don't migrate over from Drupal 7 to 8. Once you have installed the module Migrate Drupal UI you can go to /upgrade and follow the steps to connect to the old database and get you migration started. It will look something like below.
A lot of times you need more customization than just bring everything over because it's not just a Drupal update that's wanted but a whole new site just keeping a lot of the content. For that you will want the module Migrate Update & Migrate Tools which will open up the option to handle migrations through drush and can generate all you migrations for you with the following drush command.
drush migrate-upgrade --legacy-db-url=mysql://user:pass@url/default --legacy-root=http://blahblah.docksal --configure-onlyOnce you run that you can go to the migrations page in the admin (/admin/structure/migrate) and there will be a new group of "Import from Drupal 7" will exist, if you view it you will see a bunch of new migrations exist.
Now you can from here execute all migrations importing everything, or this is the time you can export these migrations to config and edit them to fit your needs.
Importing as is
If you're happy with how things are and want to import you can click execute on each migration through the GUI or run the following drush command.
drush migrate:import --group migrate_drupal_7Editing the migrations
Editing the migrations is the path I see most people wanting to go. Here you can take the bulk on the content as is but edit some migrations to merge content types, rename fields, or really do whatever is needed to the old data before it comes across. To edit the migrations you first must export the site config with drush cex, all the config files will go to you config directory and be named migration_plus.migration.* and the names should for the most part make sense so you know where to go to edit.
When editing Migrations when in config form under the process section the keys are the Drupal 8 machine names and the values are how to get the values, sometimes being the Drupal 7 machine names. Below is an example where on the new site we're using Body but it Drupal 7 the field's name was field_should_be_page_body.
body:
-
plugin: get
source: field_should_be_page_bodyHandwritten Migrations
You also can create your own migrations based off the all the migrations that were just generated for us. You might need to create your own migrations if in you Drupal 7 site you had multiple content types that needed to be consolidated or if you want to rename field machine names to be less crazy. Once you have seen a couple migration config files they can start to make sense and you should be able to start writing your own. I have written custom ones to put content into Groups, and to migrate 5+ Drupal 7 content types into a single Drupal 8 content type.
Below is a config I made that takes a content type in Drupal 7 called "Should Have Been Page" and migrates it to be a "Page" in Drupal 8. I created that by copying the normal Page migration and changing the: id, label, node_type, and body. You can also view it in the git repo.
langcode: en
status: true
dependencies: { }
id: upgrade_d7_should_have_been_page
class: Drupal\migrate\Plugin\Migration
field_plugin_method: null
cck_plugin_method: null
migration_tags:
- 'Drupal 7'
- Content
migration_group: migrate_drupal_7
label: 'Nodes (Should Have Been page)'
source:
plugin: d7_node
node_type: should_have_been_page
process:
nid:
-
plugin: get
source: tnid
vid:
-
plugin: get
source: vid
langcode:
-
plugin: default_value
source: language
default_value: und
title:
-
plugin: get
source: title
uid:
-
plugin: get
source: node_uid
status:
-
plugin: get
source: status
created:
-
plugin: get
source: created
changed:
-
plugin: get
source: changed
promote:
-
plugin: get
source: promote
sticky:
-
plugin: get
source: sticky
revision_uid:
-
plugin: get
source: revision_uid
revision_log:
-
plugin: get
source: log
revision_timestamp:
-
plugin: get
source: timestamp
comment_node_page/0/status:
-
plugin: get
source: comment
body:
-
plugin: get
source: field_should_be_page_body
destination:
plugin: 'entity:node'
default_bundle: page
migration_dependencies: null
Once that is saved you run drush cim to get that new config, then if everything works fine you can import it through the GUI on the main migrations listing page.
Random Databases Migrations using Drupal Migration
This is a pretty common migration request, the client has content is some other type of system as long as you can get a dump of the database or access to it you can migrate it with a normal Drupal migration still. This also could be from an API or a CSV or really anyway you can get data and read it with PHP you can make this work. To do this you need to create a new Migrate source that tells Drupal how to connect to query the database, then a config that maps the database query to Drupal fields.
To do that you will need to create a custom module and specify a custom migration source, I suggest doing both those with Drupal Console. To generate the module run drupal generate:module, follow the prompts then generate the migration source drupal generate:plugin:migrate:source. That will create you all the boilerplate code needed to then create you Migrate source. Your migrate source will live under custom_module/src/Plugin/migrate/source/MigrateSourceName.php. If your source is simple like mine you can solve it with just a couple lines doing a normal database query, you can view mine in the git repo and below.
<?php
namespace Drupal\sample_migration\Plugin\migrate\source;
use Drupal\migrate\Plugin\migrate\source\SqlBase;
use Drupal\migrate\Row;
/**
* Provides a 'Random Database' migrate source.
*
* @MigrateSource(
* id = "RandomDatabases"
* )
*/
class RandomDatabases extends SqlBase {
/**
* {@inheritdoc}
*/
public function query() {
$query = $this->select('Content', 'c');
$query->fields('c', [
'ID',
'Title',
'Body',
]);
return $query;
}
/**
* {@inheritdoc}
*/
public function fields() {
$fields = [
'ID' => $this->t('Source ID'),
'Title' => $this->t('Title'),
'Body' => $this->t('Body'),
];
return $fields;
}
/**
* {@inheritdoc}
*/
public function getIds() {
return [
'ID' => [
'type' => 'integer',
'alias'=> 'ID'
]
];
}
Then the next thing we do is create a migrate config that uses that Migrate source to actually import content into the site. To do that I copied the migrate code from the generated page and modified it again. Here the important changes are to source you need to set the key to the name of your database in your settings.php and the plugin to your new plugin you just created. Once created, you will need to import your config and install your custom module. Again you can view my code on the repo or below.
langcode: en
status: true
dependencies:
enforced:
module:
- migrate_plus
- sample_migration
id: random_databases
class: null
field_plugin_method: null
cck_plugin_method: null
migration_tags: null
migration_group: migrate_drupal_7
label: 'Page From Random Databases'
source:
key: random
plugin: RandomDatabases
process:
type:
plugin: default_value
default_value: page
status:
plugin: default_value
default_value: 1
uid:
plugin: default_value
default_value: 1
langcode:
plugin: default_value
default_value: en
title: Title
body: Body
destination:
plugin: 'entity:node'
migration_dependencies: null
Manual Migrations using custom code
This is one that I've unfortunately had a lot of experience with. Sometimes it seems the normal ways of doing migrations just doesn't work or you can manually force it in faster than doing the normal way. An example I've done is getting Field Collections into Paragraphs or Nodes. To do that generally I create a batch process that does a database query against the old database to get all the info I need, then loop over it and create or update the new content.
All this code is available in my Drupal 8 Migrations repo with the majority of the code being in the sample_migrations.module. The code has lots of comments so hopefully it's simple enough to follow but I'll give a brief overview of what's happening.
Function sample_migration_title_body_paragraphs
This is the main batch process function that will loop over all the field collection items we care about in our Drupal 7 site and import them into the Drupal 8 site.
Function sample_migration_get_title_body_field_collections
This connects to the Drupal 7 database and does the queries with joins needed to get all the data we care about for the Field Collection called "Title Body" which needs to be migrated over to the new site.
Function sample_migration_get_test_paragraph_content_map
With Drupal 8 migrations you will get a lot of tables in the database that are mapping tables. They allow you to compare Drupal 7 ID to Drupal 8 ID. This is very important so you know if the field collection was on NID 57 on the old site that's actually NID 541 on the new site. So this function grabs that table and puts the Drupal 7 ID as the key.
Function sample_migration_create_title_body_paragraph($current_result, $paragraph_data)
This function gets passed the $current_result which is the raw database result back on the for the Drupal 7 Field Collection Item and $paragraph_data which is an array to be turned into a paragraph. With that information it checks to see if this paragraph already has been created in Drupal 8, if so edit it, if not create it. Then find the Drupal 8 node and attach this paragraph to it.
/**
* Batch Process to migrate all Title Body Field Collections from Drupal 7
* into the Title Body Paragraphs in Drupal 8
* You can read more about Batches in Drupal 8 here:
* https://api.drupal.org/api/drupal/core%21includes%21form.inc/group/batch/8.5.x
*
* @param [type] $context
* @return void
*/
function sample_migration_title_body_paragraphs(&$context) {
$entity_type_manager = \Drupal::service('entity_type.manager');
/* @var NodeStorage $node_storage */
$node_storage = $entity_type_manager->getStorage('node');
if (empty($context['sandbox'])) {
$context['sandbox'] = [];
}
if (!isset($context['sandbox']['progress'])) {
$context['sandbox']['progress'] = 0;
$context['sandbox']['current_index'] = 0;
// Load all Current Nodes type of Page
$all_pages = $node_storage->loadByProperties([
'type' => 'page'
]);
// Need to set a bunch of data on the sandbox key to keep it on every loop through the batch process
$context['sandbox']['page_content'] = array_values($all_pages);
$context['sandbox']['title_page_fc'] = sample_migration_get_title_body_field_collections();
$context['sandbox']['test_paragraph_map'] = sample_migration_get_test_paragraph_content_map();
$context['sandbox']['index'] = array_values($context['sandbox']['page_content']);
$context['sandbox']['max'] = count($context['sandbox']['page_content']);
}
$limit = 10;
$indexes = range($context['sandbox']['current_index'], $context['sandbox']['current_index'] + $limit);
// here is each loop through the batch process
foreach ($indexes as $index) {
// based on index get the current field collection item I'm on
// use a bunch of ifs to make sure we can proceed without breaking everything
$current_result = $context['sandbox']['title_page_fc'][$index];
if (!is_null($current_result)) {
$d7_nid = $current_result->entity_id;
if (!is_null($context['sandbox']['title_page_fc'][$index])) {
$d8_nid = $context['sandbox']['test_paragraph_map'][$d7_nid]->destid1;
if (!is_null($d8_nid)) {
// if we made it this far it's safe to try and create a paragraph
// Once the paragraph data is setup pass it to our create paragraph function
$paragraph_data = [
'field_title' => $current_result->field_fc_title_value,
'field_body' => [
'value' => $current_result->field_fc_body_value,
'format' => 'full_html',
],
'field_drupal7_item_id' => $current_result->field_test_paragraphs_content_value,
'parent_type' => 'node',
'type' => 'title_body',
'status' => '1',
'parent_field_name' => 'field_content',
'parent_id' => $d8_nid
];
sample_migration_create_title_body_paragraph($current_result, $paragraph_data);
}
}
}
$context['sandbox']['progress']++;
$context['sandbox']['current_index'] = $index;
$context['message'] = t('Now processing Paragraphs');
}
// Update batch on our progress.
if ($context['sandbox']['progress'] != $context['sandbox']['max']) {
$context['finished'] = $context['sandbox']['progress'] / $context['sandbox']['max'];
}
else {
$context['sandbox']['finished'] = 1;
}
/**
* Simple batch process finished function
*
* @param [type] $success
* @param [type] $results
* @param [type] $operations
* @return void
*/
function sample_migration_title_body_paragraphs_finished($success, $results, $operations) {
$messenger = \Drupal::messenger();
if ($success) {
$message = t('@count Reviews were added to Groups', ['@count' => count($results)]);
$messenger->addMessage($message);
}
else {
// Something went wrong
$error_operation = reset($operations);
$messenger->addMessage(
t('An error occurred while processing @operation with arguments : @args',
[
'@operation' => $error_operation[0],
'@args' => print_r($error_operation[0], TRUE)
]
)
);
}
/**
* Undocumented function
*
* @return void
*/
function sample_migration_get_title_body_field_collections() {
Database::setActiveConnection('drupal7');
$drupal7db = Database::getConnection();
$query = $drupal7db->select('field_data_field_test_paragraphs_content', 'fc');
$query->fields('fc', [
'entity_id',
'field_test_paragraphs_content_value',
'delta'
]);
$query->innerJoin('field_collection_item', 'fci', 'fc.entity_id = fci.item_id');
$query->fields('fci', [
'item_id',
'field_name',
'revision_id'
]);
$query->innerJoin('field_data_field_fc_title', 'title', 'title.entity_id = fci.item_id');
$query->fields('title', [
'entity_type',
'bundle',
'entity_id',
'field_fc_title_value'
]);
$query->innerJoin('field_data_field_fc_body', 'body', 'body.entity_id = fci.item_id');
$query->fields('body', [
'entity_type',
'field_fc_body_value',
'field_fc_body_format',
'bundle',
'entity_id'
]);
$results = $query->execute()->fetchAll();
Database::setActiveConnection('default');
return $results;
/**
* Undocumented function
*
* @return void
*/
function sample_migration_get_test_paragraph_content_map() {
$drupal8db = Database::getConnection();
$query = $drupal8db->select('migrate_map_d7_node__test_paragraphs', 'mm');
$query->fields('mm', [
'sourceid1',
'destid1'
]);
$map_results = $query->execute()->fetchAllAssoc('sourceid1');
return $map_results;
/**
* Create / Update a paragraph based on the data passed into it
*
* @param [type] $current_result
* @param [type] $paragraph_data
* @return void
*/
function sample_migration_create_title_body_paragraph($current_result, $paragraph_data) {
$paragraph_storage = \Drupal::entityTypeManager()->getStorage('paragraph');
$node_storage = \Drupal::entityTypeManager()->getStorage('node');
// check if paragraph already exists, if so we need to update it
// we set field_drupal7_item_id on it so we can make sure we aren't duplicating paragraphs
// so we don't build a million paragraphs
$existing_paragraph = $paragraph_storage->loadByProperties([
'field_drupal7_item_id' => $current_result->field_test_paragraphs_content_value
]);
if (!is_null($existing_paragraph) && count($existing_paragraph)) {
$this_paragraph = reset($existing_paragraph);
}
else {
$this_paragraph = $paragraph_storage->create($paragraph_data);
$this_paragraph->save();
}
// attach paragraph to node
$parent_node = $node_storage->load($current_result->entity_id);
if (!is_null($parent_node)) {
// parent field name
$field_name = 'field_content';
// find all paragraphs already on the node so we don't add this one twice
// also so we don't delete a paragraph by adding a new one
$node_paragraphs = $parent_node->get($field_name)->getValue();
$should_add_paragraph = true;
if (count($node_paragraphs)) {
foreach ($node_paragraphs as $node_paragraph) {
if ($node_paragraph['target_id'] === $this_paragraph->id()) {
// if this paragraph is alrady attached, there is no reason to attach it again
$should_add_paragraph = false;
break;
}
}
}
if ($should_add_paragraph) {
// add this paragraph to the paragraphs array on the node and save it
$node_paragraphs[] = [
'target_id' => $this_paragraph->id(),
'target_revision_id' => $this_paragraph->getRevisionId()
];
$parent_node->set($field_name, $node_paragraphs);
$parent_node->save();
}
}
else {
\Drupal::messenger()->addMessage(t("Unable to attach paragraph to content. Paragraph ID: " . $this_paragraph->id()), 'error');
\Drupal::logger('joco_migrations')->error("Unable to attach paragraph to content. Paragraph ID: " . $this_paragraph->id());
}
Wrapping Up
There are a lot of ways to do Migrations in Drupal 8, I'm sure there are more than I mentioned here or that I'm even aware of. Migrations are very powerful, and can be headache inducing. Hopefully I was able to shed the light on what some options for migrations are and you're more confident to go ahead and start migrating a website today. If you have questions on how to get your content migrated feel free to reach out to me on Twitter or however else you can contact people on the internet.