See the previous tutorial.
Another aspect is to have some control over the data received. Out of the box data-migrator offers a wide range of facilities
None column test¶
The current list of Field types contains Nullables and non Nullable fields. In some cases you might want to exclude those nulls. This is a generic function in the manager and can be set on a model basis in the Meta block
class Result(models.Model): id = models.IntField(pos=0) # keep id a = models.StringField(pos=1) b = models.StringField(pos=2, nullable=None) class Meta: drop_if_none = ['b']
The moment records are saved they are validated and records are dropped if b turns out to be
A common pattern is to drop non unique values (last to be dropped), for example when scanning unique email addresses. This functionality is also out of the box:
class Result(models.Model): id = models.IntField(pos=0) # keep id a = models.StringField(pos=1, unique=True) class Meta: drop_non_unique = True
Uniqueness is checked at the parsing of the input, before any records are
fail_non_unique if you do not want to drop, but completely
fail the transformation.
Complex Unique columns¶
A more complex situation is when a combination of (input) columns needs to be checked. Consider for example the de-duplication of membership records. This is a solved by using a parse function and a hidden column:
def compound(row): return row + row + row class Result(models.Model): id = models.IntField(pos=0) # keep id key = models.HiddenField(parse=compound, unique=True) a = models.StringField(pos=1) class Meta: drop_non_unique = True
In this example key is set to check uniqueness but not used as output in the end result.