Sorry, no results found for "".
As a DatoCMS developer, you often find yourself in need of importing data from an external source. For example when you are doing a one-time import from another CMS to DatoCMS, or when you just want to clean up messy data from an external API or RESTful web service, or when you want the ability to perform powerful queries on it.
In this guide we will cover how to do a one-time import from an external data source using Node.JS.
Concepts you should be familiar with: knowledge of Node.js and async
/await
.
What are some common external sources? An external data source can come in a wide range of different formats made available on different transport layers. Here's a few examples:
The REST API of your old CMS
A text file with comma separated values (CSV)
A SQL database
A JSON file or newline delimited JSON (NDJSON) file
No matter what kind of source you are reading from, an external import can be split into three discrete steps:
Read data from the external source
Transform the data to DatoCMS records(s) matching your data model
Save the records to your DatoCMS project
We will cover each of these in order
Let's start with a simple example where the external data source is an API endpoint containing an array of breeds of dogs that we want to import into a DatoCMS project.
The quickest way to read from this API in Node.js is to install the node-fetch
package which gives you a window.fetch
-like API that enables you to fetch the data.
const fetch = require('node-fetch');
async function importDogBreeds() { const response = await fetch('https://something.now.sh/dog-breeds'); const dogBreeds = await response.json();
// we now have an array of dogBreeds from the external API}
importDogBreeds();
Now, let's say the following is the DatoCMS schema we want our imported data to adhere to:
ID: 552
API key: category
Fields:
Name (API key: name
): string
ID: 730
API key: dog_breed
Model fields:
Name (API key: name
): string
Category (API key: category
): link to model category
Breed for (API key: breed_for
): string
Description (API key: description
): text
Image (API key: image
): file
If you look carefully, you'll see that the source data doesn't map 1:1 to the schema model. There's a few differences to note here:
The breed
field is called name
in our DatoCMS model
Instead of importing category
directly as text inside the breed, we want to create a separate record for them, and have the category
field be a reference to it instead;
The life_span
field from the external API isn't relevant to us, and we don't want to import it at all;
This can roughly be codified to the following transform function:
function transformDogBreed(externalData) { return { itemType: '730', // <- that's the ID of our dog_breed model name: externalData.breed, category: ???, breed_for: externalData.breed_for, description: externalData.description, image: ???, };}
As you might have guessed, itemType
means "model" in API-land, and you have to fill it in with the ID of your model (in this case, "730"
).
The category
field requires a category record ID, but right now we do not have it. This suggests us that first we have to import the breed categories, and then we can proceed with importing the dog breeds.
To do that, we get all the different dog breed categories, and then we remove any duplicate:
const uniq = require('lodash.uniq');const fetch = require('node-fetch');
async function importDogBreeds() { const response = await fetch('https://something.now.sh/dog-breeds'); const dogBreeds = await response.json();
const categories = dogBreeds.map(dogBreed => dogBreed.category) const uniqueCategories = uniq(categories);}
In the previous steps all we did was fetch and prepare the data to be imported into your DatoCMS project. Now it's time to actually make it become DatoCMS records.
First we need to configure our DatoCMS client with our project's API token. We will need to add datocms-client
as a dependency to our project and create a client instance:
const { SiteClient } = require('datocms-client');
const client = new SiteClient('<YOUR-TOKEN-WITH-WRITE-ACCESS>');
In order to give this client write access, we need to generate an access token. You can generate an access token under the "API token" section of your project's settings.
Now that we have our client configured, the next step is to create our records, using the client.items.create
method:
const categoryNameToRecord = {};
for (let categoryName of uniqueCategories) { categoryNameToRecord[name] = await client.items.create({ itemType: '552', // <- that's the ID of our category model name });}
As you can see, we save the created records in a categoryNameToRecord
object so that it will be easier to access them during the creation of dog breeds, which is obviously the next thing we need to to do in our script:
for (let dogBreed of dogBreeds) { categoryNameToRecord[name] = await client.items.create({ itemType: '730', // <- that's the ID of our dog_breed model name: externalData.breed, category: categoryNameToRecord[dogBreed.category].id, // <- we pick the ID of our category record breed_for: externalData.breed_for, description: externalData.description, image: ???, });}
The last step is uploading the images. To do that, we can simply use the client.uploadFile
method, passing down additional data such as the default alternate text we want for each image:
1for (let dogBreed of dogBreeds) {2
3 const image = await client.uploadFile(4 dogBreed.image_url,5 {6 defaultFieldMetadata: {7 en: {8 alt: `${dogBreed} dog`9 }10 },11 notes: `Imported from external source`,12 }13 );14
15 categoryNameToRecord[name] = await client.items.create({16 // ...17 image: image,18 });19}
And voilĂ ! You've just successfully imported your external data to DatoCMS! Here's the complete script for reference:
1const uniq = require('lodash.uniq');2const fetch = require('node-fetch');3const { SiteClient } = require('datocms-client');4
5const client = new SiteClient('<YOUR-TOKEN-WITH-WRITE-ACCESS>');6
7const data = [8 {9 "id": 1,10 "breed": "Alapaha Blue Blood Bulldog",11 "bred_for": "Guarding",12 "category": "Mixed",13 "description": "The Alapaha Blue Blood Bulldog is a well-developed, exaggerated bulldog with a broad head and...",14 "life_span": "12 - 13 years",15 "image_url": "https://cdn2.thedogapi.com/images/kuvpGHCzm.jpg"16 },17 {18 "id": 2,19 "breed": "Alaskan Husky",20 "bred_for": "Sled pulling",21 "category": "Mixed",22 "life_span": "10 - 13 years",23 "image_url": "https://cdn2.thedogapi.com/images/uEPB98jBS.jpg"24 },25 {26 "id": 3,27 "breed": "Alaskan Malamute",28 "bred_for": "Hauling heavy freight, Sled pulling",29 "category": "Working",30 "life_span": "12 - 15 years",31 "image_url": "https://cdn2.thedogapi.com/images/aREFAmi5H.jpg"32 }33];34
35async function importDogBreeds() {36 const categories = data.map(dogBreed => dogBreed.category)37 const uniqueCategories = uniq(categories);38
39 const categoryNameToRecord = {};40
41 for (let categoryName of uniqueCategories) {42 categoryNameToRecord[name] = await client.items.create({43 itemType: '<CATEGORY-MODEL-ID>',44 name45 });46 }47
48 for (let dogBreed of dogBreeds) {49 const image = await client.uploadFile(50 dogBreed.image_url,51 {52 defaultFieldMetadata: {53 en: {54 alt: `${dogBreed} dog`55 }56 },57 notes: `Imported from external source`,58 }59 );60
61 categoryNameToRecord[name] = await client.items.create({62 itemType: '<DOG-BREED-MODEL-ID>',63 name: externalData.breed,64 category: categoryNameToRecord[dogBreed.category].id,65 breed_for: externalData.breed_for,66 description: externalData.description,67 image,68 });69 }70}71
72importDogBreeds();