Validation

See the previous tutorial.

Another aspect is to have some control over the data received. Out of the box data-migrator offers a wide range of facilities

None column test

The current list of Field types contains Nullables and non Nullable fields. In some cases you might want to exclude those nulls. This is a generic function in the manager and can be set on a model basis in the Meta block

class Result(models.Model):
  id = models.IntField(pos=0) # keep id
  a  = models.StringField(pos=1)
  b  = models.StringField(pos=2, nullable=None)

  class Meta:
    drop_if_none = ['b']

The moment records are saved they are validated and records are dropped if b turns out to be None

Unique columns

Another common pattern is to drop non unique values (last to be dropped), for example when scanning unique emailaddresses. This functionality is also out of the box:

class Result(models.Model):
  id = models.IntField(pos=0) # keep id
  a  = models.StringField(pos=1, unique=True)

  class Meta:
    drop_non_unique = True

This is checked at the parsing of the input, before any records are written. Use fail_non_unique if you do not want to drop, but completely fail the transformation.

Complex Unique columns

A more complex situation when a combination of (input) columns need to be checked. Consider for example the de-duplication of membership records. This is a solved by using a parse function and a hidden column:

def compound(row):
  return row[0] + row[1] + row[2]

class Result(models.Model):
  id  = models.IntField(pos=0) # keep id
  key = models.HiddenField(parse=compound, unique=True)
  a   = models.StringField(pos=1)

  class Meta:
    drop_non_unique = True

In this example key is set to check uniqueness but not output in the end result.